<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Interestingly enough, we started suffering the same problem at the same time.</div><div dir="ltr"><br></div><div>- 1 node with 44 cores, 256GB of RAM</div><div>- Zeek 2.5.5</div><div>- node.cfg:</div><div> <span style="color:rgb(0,0,0);font-family:Menlo;font-size:11px">[worker-1]</span></div>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">type=worker</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">host=localhost<br></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">interface=af_packet::ens4f0</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">lb_method=custom</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">lb_procs=25</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">pin_cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p>- broctl.cfg:<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">MemLimit = 100000000 #100GB</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">setcap.enabled=1</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">
</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><br></p></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Oct 18, 2019 at 10:48 AM Mark Gardner <<a href="mailto:mkg@vt.edu">mkg@vt.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-size:small">We must have crossed some threshold yesterday. Suddenly we are suffering an epidemic of workers dying with "out of memory in new" even though we made no changes. Previously, we would have a few die each day. Now we have had 250 alerts of workers dying and being restarted from 00:00 to 10:00. I have no idea where to start debugging the problem. Any suggestions?</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">What causes a worker to die by running out of memory? The sensors have lots of memory (see below) so I would not expect to have any out of memory deaths. (To monitor the problem, I am in the process of setting up collectd and graphana.)</div><div class="gmail_default" style="font-size:small"></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Some details:<br></div><div class="gmail_default" style="font-size:small">- 5 sensors, each with 16-core, AMD Epyc 7351P, 128 GB RAM, Intel X520-T2</div><div class="gmail_default" style="font-size:small">- Zeek 2.6.1</div><div class="gmail_default" style="font-size:small">- node.cfg: lb_procs=15, pin_cpus=1-15, af_packet_buffer_size=1*1024*1024*1024</div><div class="gmail_default" style="font-size:small">- broctl.cfg: setcap enabled</div><div class="gmail_default" style="font-size:small">- Not shunting any traffic</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Mark</div>-- <br><div dir="ltr">Mark Gardner<br>--</div></div>
_______________________________________________<br>
Zeek mailing list<br>
<a href="mailto:zeek@zeek.org" target="_blank">zeek@zeek.org</a><br>
<a href="http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek" rel="noreferrer" target="_blank">http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Munroe Sollog<div>Senior Network Engineer</div><div><a href="mailto:munroe@lehigh.edu" target="_blank">munroe@lehigh.edu</a></div></div></div>