<div dir="ltr">Hmm, I will disable the SMB analyzer in local.bro and see if it helps.. Thanks Jon! :-)<div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Sep 6, 2019 at 12:58 PM Jon Siwek <<a href="mailto:jsiwek@corelight.com">jsiwek@corelight.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Biggest changes from 2.5.x to 2.6.x that I can recall are (1)<br>
switching remote communication to use the new Broker library and (2)<br>
enabling SMB analysis by default.<br>
<br>
Had you manually enabled SMB in your previous 2.5.x deployment? If<br>
not, you could see if disabling it helps:<br>
<br>
redef Analyzer::disabled_analyzers += { Analyzer::ANALYZER_SMB };<br>
<br>
That's my first guess because we've recently seen/suspected (but not<br>
yet fixed) some state management issues in the SMB analysis scripts<br>
that might explain high memory usage.<br>
<br>
- Jon<br>
<br>
On Fri, Sep 6, 2019 at 8:46 AM fatema bannatwala<br>
<<a href="mailto:fatema.bannatwala@gmail.com" target="_blank">fatema.bannatwala@gmail.com</a>> wrote:<br>
><br>
> Hi All,<br>
><br>
> Couple of months ago I upgraded the Zeek cluster from 2.5 to 2.6.1 (compiled with the jemalloc support).<br>
> I have started seeing increased memory usage by the workers.<br>
><br>
> I have two physical sensors, each running 18 Zeek worker processes LB by PF_RING.<br>
> Not loaded any custom scripts, just the basic scripts that are enabled by default in local.bro (also have misc/scan disabled).<br>
><br>
> I just did a top on one of the boxes and here's the output (specially two Zeek processes -13632, 13611 using >10% memory which is ~11G)<br>
> Also, attaching a weekly available free memory graph for the system.<br>
><br>
> Tasks: 455 total, 9 running, 443 sleeping, 0 stopped, 3 zombie<br>
> %Cpu(s): 18.3 us, 1.7 sy, 0.0 ni, 79.5 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st<br>
> KiB Mem : 98783960 total, 32963660 free, 64807572 used, 1012728 buff/cache<br>
> KiB Swap: 4194300 total, 3572200 free, 622100 used. 33221356 avail Mem<br>
><br>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
> 13589 bro 20 0 3662052 3.4g 73340 R 90.4 3.6 1072:47 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-5 local.bro broctl base/frameworks/cluster broctl/auto<br>
> 13533 bro 20 0 1847972 1.6g 73188 S 50.3 1.7 1098:05 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-6 local.bro broctl base/frameworks/cluster broctl/auto<br>
> 13512 bro 20 0 1291260 1.1g 73052 S 49.7 1.1 1080:30 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-1 local.bro broctl base/frameworks/cluster broctl/auto<br>
> 13628 bro 20 0 2347952 2.1g 73328 R 49.0 2.2 1109:31 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-12 local.bro broctl base/frameworks/cluster broctl/auto<br>
> 13516 bro 20 0 973260 799176 72844 R 47.0 0.8 1036:29 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-3 local.bro broctl base/frameworks/cluster broctl/auto<br>
> 13539 bro 20 0 6374956 6.0g 73456 S 46.0 6.3 1147:08 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-2 local.bro broctl base/frameworks/cluster broctl/auto<br>
> 13591 bro 20 0 865952 726516 73020 S 44.7 0.7 1052:29 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-4 local.bro broctl base/frameworks/cluster broctl/auto<br>
> 13632 bro 20 0 12.2g 12.0g 73584 R 43.7 12.8 1068:17 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-15 local.bro broctl base/frameworks/cluster broctl/auto<br>
> 13540 bro 20 0 2146844 1.9g 73348 R 41.4 2.0 1149:38 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-7 local.bro broctl base/frameworks/cluster broctl/auto<br>
> 13611 bro 20 0 17.0g 16.7g 73404 S 39.7 17.8 1172:14 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-9 local.bro broctl base/frameworks/cluster broctl/auto<br>
> 13640 bro 20 0 2624300 2.1g 73328 S 39.7 2.3 1043:50 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-18 local.bro broctl base/frameworks/cluster broctl/auto<br>
> 13586 bro 20 0 3347044 3.1g 73468 S 39.1 3.2 1042:24 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-10 local.bro broctl base/frameworks/cluster broctl/auto<br>
> 13641 bro 20 0 2274788 2.0g 73424 R 39.1 2.2 1029:58 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-17 local.bro broctl base/frameworks/cluster broctl/auto<br>
> 13614 bro 20 0 1954780 1.7g 73188 S 38.4 1.8 995:00.54 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-13 local.bro broctl base/frameworks/cluster broctl/auto<br>
> 13627 bro 20 0 2756520 2.5g 73288 S 38.4 2.6 1035:18 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-14 local.bro broctl base/frameworks/cluster broctl/auto<br>
> 13638 bro 20 0 1206548 853056 72328 R 37.4 0.9 952:10.00 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-16 local.bro broctl base/frameworks/cluster broctl/auto<br>
> 13623 bro 20 0 8998324 2.1g 73284 S 37.1 2.2 1073:31 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-11 local.bro broctl base/frameworks/cluster broctl/auto<br>
> 13575 bro 20 0 871396 706148 73128 R 36.4 0.7 1028:30 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-8 local.bro broctl base/frameworks/cluster broctl/auto<br>
> 13336 bro 20 0 266244 133920 33388 S 12.6 0.1 400:27.62 /usr/local/bro/2.6.1/bin/bro -U .status -p broctl -p broctl-live -p local -p proxy-2 local.bro broctl base/frameworks/cluster broctl/auto<br>
><br>
> Any suggestions?<br>
><br>
> Thanks!<br>
> Fatema<br>
> _______________________________________________<br>
> Zeek mailing list<br>
> <a href="mailto:zeek@zeek.org" target="_blank">zeek@zeek.org</a><br>
> <a href="http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek" rel="noreferrer" target="_blank">http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek</a><br>
</blockquote></div>