<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="margin:0px;font-stretch:normal;line-height:normal">For additional reference:</p><p style="margin:0px;font-stretch:normal;line-height:normal"><font color="#000000" face="Menlo"><span style="font-size:11px">Linux snout 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11) x86_64 GNU/Linux</span></font><br></p><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div>on 10-11 I patched libssl,and libc</div><div>on 10-17 I upgraded sudo (about 30 mins after the first worker crashed)<br></div><div><br></div>[Bro] Crash report from worker-1-12 email received at 16:00</div><div dir="ltr"><br><div>Log output from dpkg for reference:</div><div><br></div><div><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"># less /var/log/dpkg.log |grep &quot;status installed&quot;</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-11 14:59:23 status installed telegraf:amd64 1.12.3-1</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-11 14:59:23 status installed libssl1.0.2:amd64 1.0.2t-1~deb9u1</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-11 14:59:23 status installed libc-bin:amd64 2.24-11+deb9u4</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-11 14:59:23 status installed libssl1.1:amd64 1.1.0l-1~deb9u1</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-11 14:59:23 status installed openssl:amd64 1.1.0l-1~deb9u1</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-11 14:59:24 status installed man-db:amd64 2.7.6.1-2</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-11 14:59:24 status installed libssl1.0-dev:amd64 1.0.2t-1~deb9u1</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-11 14:59:24 status installed libc-bin:amd64 2.24-11+deb9u4</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-17 16:25:47 status installed sudo:amd64 1.8.19p1-2.1+deb9u1</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-17 16:25:47 status installed apache2-utils:amd64 2.4.25-3+deb9u9</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-17 16:25:47 status installed apache2-bin:amd64 2.4.25-3+deb9u9</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-17 16:25:47 status installed apache2-data:all 2.4.25-3+deb9u9</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-17 16:25:47 status installed systemd:amd64 232-25+deb9u12</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-17 16:25:47 status installed man-db:amd64 2.7.6.1-2</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2019-10-17 16:25:48 status installed apache2:amd64 2.4.25-3+deb9u9</span></p></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Oct 18, 2019 at 11:12 AM Munroe Sollog &lt;<a href="mailto:mus3@lehigh.edu">mus3@lehigh.edu</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Interestingly enough, we started suffering the same problem at the same time.</div><div dir="ltr"><br></div><div>- 1 node with 44 cores, 256GB of RAM</div><div>- Zeek 2.5.5</div><div>- node.cfg:</div><div>  <span style="color:rgb(0,0,0);font-family:Menlo;font-size:11px">[worker-1]</span></div>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">type=worker</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">host=localhost<br></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">interface=af_packet::ens4f0</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">lb_method=custom</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">lb_procs=25</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">pin_cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p>- broctl.cfg:<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">MemLimit = 100000000 #100GB</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">setcap.enabled=1</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">

</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><br></p></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Oct 18, 2019 at 10:48 AM Mark Gardner &lt;<a href="mailto:mkg@vt.edu" target="_blank">mkg@vt.edu</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-size:small">We must have crossed some threshold yesterday. Suddenly we are suffering an epidemic of workers dying with &quot;out of memory in new&quot; even though we made no changes. Previously, we would have a few die each day. Now we have had 250 alerts of workers dying and being restarted from 00:00 to 10:00. I have no idea where to start debugging the problem. Any suggestions?</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">What causes a worker to die by running out of memory? The sensors have lots of memory (see below) so I would not expect to have any out of memory deaths. (To monitor the problem, I am in the process of setting up collectd and graphana.)</div><div class="gmail_default" style="font-size:small"></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Some details:<br></div><div class="gmail_default" style="font-size:small">- 5 sensors, each with 16-core, AMD Epyc 7351P, 128 GB RAM, Intel X520-T2</div><div class="gmail_default" style="font-size:small">- Zeek 2.6.1</div><div class="gmail_default" style="font-size:small">- node.cfg: lb_procs=15, pin_cpus=1-15, af_packet_buffer_size=1*1024*1024*1024</div><div class="gmail_default" style="font-size:small">- broctl.cfg: setcap enabled</div><div class="gmail_default" style="font-size:small">- Not shunting any traffic</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Mark</div>-- <br><div dir="ltr">Mark Gardner<br>--</div></div>
_______________________________________________<br>
Zeek mailing list<br>
<a href="mailto:zeek@zeek.org" target="_blank">zeek@zeek.org</a><br>
<a href="http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek" rel="noreferrer" target="_blank">http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr">Munroe Sollog<div>Senior Network Engineer</div><div><a href="mailto:munroe@lehigh.edu" target="_blank">munroe@lehigh.edu</a></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Munroe Sollog<div>Senior Network Engineer</div><div><a href="mailto:munroe@lehigh.edu" target="_blank">munroe@lehigh.edu</a></div></div></div>