[Zeek] Increased memory usage by Zeek..
Jon Siwek
jsiwek at corelight.com
Fri Sep 6 09:58:33 PDT 2019
Biggest changes from 2.5.x to 2.6.x that I can recall are (1)
switching remote communication to use the new Broker library and (2)
enabling SMB analysis by default.
Had you manually enabled SMB in your previous 2.5.x deployment? If
not, you could see if disabling it helps:
redef Analyzer::disabled_analyzers += { Analyzer::ANALYZER_SMB };
That's my first guess because we've recently seen/suspected (but not
yet fixed) some state management issues in the SMB analysis scripts
that might explain high memory usage.
- Jon
On Fri, Sep 6, 2019 at 8:46 AM fatema bannatwala
<fatema.bannatwala at gmail.com> wrote:
>
> Hi All,
>
> Couple of months ago I upgraded the Zeek cluster from 2.5 to 2.6.1 (compiled with the jemalloc support).
> I have started seeing increased memory usage by the workers.
>
> I have two physical sensors, each running 18 Zeek worker processes LB by PF_RING.
> Not loaded any custom scripts, just the basic scripts that are enabled by default in local.bro (also have misc/scan disabled).
>
> I just did a top on one of the boxes and here's the output (specially two Zeek processes -13632, 13611 using >10% memory which is ~11G)
> Also, attaching a weekly available free memory graph for the system.
>
> Tasks: 455 total, 9 running, 443 sleeping, 0 stopped, 3 zombie
> %Cpu(s): 18.3 us, 1.7 sy, 0.0 ni, 79.5 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
> KiB Mem : 98783960 total, 32963660 free, 64807572 used, 1012728 buff/cache
> KiB Swap: 4194300 total, 3572200 free, 622100 used. 33221356 avail Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 13589 bro 20 0 3662052 3.4g 73340 R 90.4 3.6 1072:47 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-5 local.bro broctl base/frameworks/cluster broctl/auto
> 13533 bro 20 0 1847972 1.6g 73188 S 50.3 1.7 1098:05 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-6 local.bro broctl base/frameworks/cluster broctl/auto
> 13512 bro 20 0 1291260 1.1g 73052 S 49.7 1.1 1080:30 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-1 local.bro broctl base/frameworks/cluster broctl/auto
> 13628 bro 20 0 2347952 2.1g 73328 R 49.0 2.2 1109:31 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-12 local.bro broctl base/frameworks/cluster broctl/auto
> 13516 bro 20 0 973260 799176 72844 R 47.0 0.8 1036:29 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-3 local.bro broctl base/frameworks/cluster broctl/auto
> 13539 bro 20 0 6374956 6.0g 73456 S 46.0 6.3 1147:08 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-2 local.bro broctl base/frameworks/cluster broctl/auto
> 13591 bro 20 0 865952 726516 73020 S 44.7 0.7 1052:29 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-4 local.bro broctl base/frameworks/cluster broctl/auto
> 13632 bro 20 0 12.2g 12.0g 73584 R 43.7 12.8 1068:17 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-15 local.bro broctl base/frameworks/cluster broctl/auto
> 13540 bro 20 0 2146844 1.9g 73348 R 41.4 2.0 1149:38 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-7 local.bro broctl base/frameworks/cluster broctl/auto
> 13611 bro 20 0 17.0g 16.7g 73404 S 39.7 17.8 1172:14 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-9 local.bro broctl base/frameworks/cluster broctl/auto
> 13640 bro 20 0 2624300 2.1g 73328 S 39.7 2.3 1043:50 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-18 local.bro broctl base/frameworks/cluster broctl/auto
> 13586 bro 20 0 3347044 3.1g 73468 S 39.1 3.2 1042:24 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-10 local.bro broctl base/frameworks/cluster broctl/auto
> 13641 bro 20 0 2274788 2.0g 73424 R 39.1 2.2 1029:58 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-17 local.bro broctl base/frameworks/cluster broctl/auto
> 13614 bro 20 0 1954780 1.7g 73188 S 38.4 1.8 995:00.54 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-13 local.bro broctl base/frameworks/cluster broctl/auto
> 13627 bro 20 0 2756520 2.5g 73288 S 38.4 2.6 1035:18 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-14 local.bro broctl base/frameworks/cluster broctl/auto
> 13638 bro 20 0 1206548 853056 72328 R 37.4 0.9 952:10.00 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-16 local.bro broctl base/frameworks/cluster broctl/auto
> 13623 bro 20 0 8998324 2.1g 73284 S 37.1 2.2 1073:31 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-11 local.bro broctl base/frameworks/cluster broctl/auto
> 13575 bro 20 0 871396 706148 73128 R 36.4 0.7 1028:30 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-8 local.bro broctl base/frameworks/cluster broctl/auto
> 13336 bro 20 0 266244 133920 33388 S 12.6 0.1 400:27.62 /usr/local/bro/2.6.1/bin/bro -U .status -p broctl -p broctl-live -p local -p proxy-2 local.bro broctl base/frameworks/cluster broctl/auto
>
> Any suggestions?
>
> Thanks!
> Fatema
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
More information about the Zeek
mailing list