<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="margin:0px;font-stretch:normal;line-height:normal">For additional reference:</p><p style="margin:0px;font-stretch:normal;line-height:normal"><font color="#000000" face="Menlo"><span style="font-size:11px">Linux snout 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11) x86_64 GNU/Linux</span></font><br></p><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div>on 10-11 I patched libssl,and libc</div><div>on 10-17 I upgraded sudo (about 30 mins after the first worker crashed)<br></div><div><br></div>[Bro] Crash report from worker-1-12 email received at 16:00</div><div dir="ltr"><br><div>Log output from dpkg for reference:</div><div><br></div><div><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"># less /var/log/dpkg.log |grep "status installed"</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-11 14:59:23 status installed telegraf:amd64 1.12.3-1</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-11 14:59:23 status installed libssl1.0.2:amd64 1.0.2t-1~deb9u1</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-11 14:59:23 status installed libc-bin:amd64 2.24-11+deb9u4</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-11 14:59:23 status installed libssl1.1:amd64 1.1.0l-1~deb9u1</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-11 14:59:23 status installed openssl:amd64 1.1.0l-1~deb9u1</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-11 14:59:24 status installed man-db:amd64 2.7.6.1-2</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-11 14:59:24 status installed libssl1.0-dev:amd64 1.0.2t-1~deb9u1</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-11 14:59:24 status installed libc-bin:amd64 2.24-11+deb9u4</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-17 16:25:47 status installed sudo:amd64 1.8.19p1-2.1+deb9u1</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-17 16:25:47 status installed apache2-utils:amd64 2.4.25-3+deb9u9</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-17 16:25:47 status installed apache2-bin:amd64 2.4.25-3+deb9u9</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-17 16:25:47 status installed apache2-data:all 2.4.25-3+deb9u9</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-17 16:25:47 status installed systemd:amd64 232-25+deb9u12</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-17 16:25:47 status installed man-db:amd64 2.7.6.1-2</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-17 16:25:48 status installed apache2:amd64 2.4.25-3+deb9u9</span></p></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Oct 18, 2019 at 11:12 AM Munroe Sollog <<a href="mailto:mus3@lehigh.edu">mus3@lehigh.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Interestingly enough, we started suffering the same problem at the same time.</div><div dir="ltr"><br></div><div>- 1 node with 44 cores, 256GB of RAM</div><div>- Zeek 2.5.5</div><div>- node.cfg:</div><div> <span style="color:rgb(0,0,0);font-family:Menlo;font-size:11px">[worker-1]</span></div>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">type=worker</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">host=localhost<br></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">interface=af_packet::ens4f0</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">lb_method=custom</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">lb_procs=25</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">pin_cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p>- broctl.cfg:<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">MemLimit = 100000000 #100GB</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">setcap.enabled=1</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">
</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><br></p></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Oct 18, 2019 at 10:48 AM Mark Gardner <<a href="mailto:mkg@vt.edu" target="_blank">mkg@vt.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-size:small">We must have crossed some threshold yesterday. Suddenly we are suffering an epidemic of workers dying with "out of memory in new" even though we made no changes. Previously, we would have a few die each day. Now we have had 250 alerts of workers dying and being restarted from 00:00 to 10:00. I have no idea where to start debugging the problem. Any suggestions?</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">What causes a worker to die by running out of memory? The sensors have lots of memory (see below) so I would not expect to have any out of memory deaths. (To monitor the problem, I am in the process of setting up collectd and graphana.)</div><div class="gmail_default" style="font-size:small"></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Some details:<br></div><div class="gmail_default" style="font-size:small">- 5 sensors, each with 16-core, AMD Epyc 7351P, 128 GB RAM, Intel X520-T2</div><div class="gmail_default" style="font-size:small">- Zeek 2.6.1</div><div class="gmail_default" style="font-size:small">- node.cfg: lb_procs=15, pin_cpus=1-15, af_packet_buffer_size=1*1024*1024*1024</div><div class="gmail_default" style="font-size:small">- broctl.cfg: setcap enabled</div><div class="gmail_default" style="font-size:small">- Not shunting any traffic</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Mark</div>-- <br><div dir="ltr">Mark Gardner<br>--</div></div>
_______________________________________________<br>
Zeek mailing list<br>
<a href="mailto:zeek@zeek.org" target="_blank">zeek@zeek.org</a><br>
<a href="http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek" rel="noreferrer" target="_blank">http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr">Munroe Sollog<div>Senior Network Engineer</div><div><a href="mailto:munroe@lehigh.edu" target="_blank">munroe@lehigh.edu</a></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Munroe Sollog<div>Senior Network Engineer</div><div><a href="mailto:munroe@lehigh.edu" target="_blank">munroe@lehigh.edu</a></div></div></div>