[Zeek] Increased memory usage by Zeek..
Justin Azoff
justin at corelight.com
Fri Sep 6 09:48:23 PDT 2019
Hi!
I've been doing a ton of work in this space and have some tooling I've been
working on to help track down things like this. I'm planning to have
things ready for my ZeekWeek presentation, but if you have some time I can
share the work-in-progress stuff with you and go over how to use it (which
will help with the documentation bits that still need to be written).
The good news is I wouldn't be surprised if this issue is already fixed or
drastically better in 3.0 or master.
On Fri, Sep 6, 2019 at 11:47 AM fatema bannatwala <
fatema.bannatwala at gmail.com> wrote:
> Hi All,
>
> Couple of months ago I upgraded the Zeek cluster from 2.5 to 2.6.1
> (compiled with the jemalloc support).
> I have started seeing increased memory usage by the workers.
>
> I have two physical sensors, each running 18 Zeek worker processes LB by
> PF_RING.
> Not loaded any custom scripts, just the basic scripts that are enabled by
> default in local.bro (also have misc/scan disabled).
>
> I just did a top on one of the boxes and here's the output (specially two
> Zeek processes -*13632, **13611* using >10% memory which is ~11G)
> Also, attaching a weekly available free memory graph for the system.
>
> Tasks: 455 total, 9 running, 443 sleeping, 0 stopped, 3 zombie
> %Cpu(s): 18.3 us, 1.7 sy, 0.0 ni, 79.5 id, 0.0 wa, 0.0 hi, 0.4 si,
> 0.0 st
> KiB Mem : 98783960 total, 32963660 free, *64807572* used, 1012728
> buff/cache
> KiB Swap: 4194300 total, 3572200 free, 622100 used. 33221356 avail Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 13589 bro 20 0 3662052 3.4g 73340 R 90.4 3.6 1072:47
> /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-5 local.bro broctl base/frameworks/cluster broctl/auto
> 13533 bro 20 0 1847972 1.6g 73188 S 50.3 1.7 1098:05
> /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-6 local.bro broctl base/frameworks/cluster broctl/auto
> 13512 bro 20 0 1291260 1.1g 73052 S 49.7 1.1 1080:30
> /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-1 local.bro broctl base/frameworks/cluster broctl/auto
> 13628 bro 20 0 2347952 2.1g 73328 R 49.0 2.2 1109:31
> /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-12 local.bro broctl base/frameworks/cluster broctl/auto
> 13516 bro 20 0 973260 799176 72844 R 47.0 0.8 1036:29
> /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-3 local.bro broctl base/frameworks/cluster broctl/auto
> 13539 bro 20 0 6374956 6.0g 73456 S 46.0 6.3 1147:08
> /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-2 local.bro broctl base/frameworks/cluster broctl/auto
> 13591 bro 20 0 865952 726516 73020 S 44.7 0.7 1052:29
> /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-4 local.bro broctl base/frameworks/cluster broctl/auto
> *13632* *bro 20 0 12.2g 12.0g 73584 R 43.7 * *12.8*
> 1068:17 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-15 local.bro broctl
> base/frameworks/cluster broctl/auto
> 13540 bro 20 0 2146844 1.9g 73348 R 41.4 2.0 1149:38
> /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-7 local.bro broctl base/frameworks/cluster broctl/auto
> *13611* bro *20 0 17.0g 16.7g 73404 S 39.7* *17.8*
> 1172:14 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-9 local.bro broctl base/frameworks/cluster
> broctl/auto
> 13640 bro 20 0 2624300 2.1g 73328 S 39.7 2.3 1043:50
> /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-18 local.bro broctl base/frameworks/cluster broctl/auto
> 13586 bro 20 0 3347044 3.1g 73468 S 39.1 3.2 1042:24
> /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-10 local.bro broctl base/frameworks/cluster broctl/auto
> 13641 bro 20 0 2274788 2.0g 73424 R 39.1 2.2 1029:58
> /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-17 local.bro broctl base/frameworks/cluster broctl/auto
> 13614 bro 20 0 1954780 1.7g 73188 S 38.4 1.8
> 995:00.54 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-13 local.bro broctl
> base/frameworks/cluster broctl/auto
> 13627 bro 20 0 2756520 2.5g 73288 S 38.4 2.6
> 1035:18 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-14 local.bro broctl
> base/frameworks/cluster broctl/auto
> 13638 bro 20 0 1206548 853056 72328 R 37.4 0.9 952:10.00
> /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-16 local.bro broctl base/frameworks/cluster broctl/auto
> 13623 bro 20 0 8998324 2.1g 73284 S 37.1 2.2
> 1073:31 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-11 local.bro broctl
> base/frameworks/cluster broctl/auto
> 13575 bro 20 0 871396 706148 73128 R 36.4 0.7 1028:30
> /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-8 local.bro broctl base/frameworks/cluster broctl/auto
> 13336 bro 20 0 266244 133920 33388 S 12.6 0.1 400:27.62
> /usr/local/bro/2.6.1/bin/bro -U .status -p broctl -p broctl-live -p local
> -p proxy-2 local.bro broctl base/frameworks/cluster broctl/auto
>
> Any suggestions?
>
> Thanks!
> Fatema
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
--
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190906/7d906ac3/attachment.html
More information about the Zeek
mailing list