[Zeek] Increased memory usage by Zeek..

Justin Azoff justin at corelight.com
Fri Sep 6 09:48:23 PDT 2019


Hi!

I've been doing a ton of work in this space and have some tooling I've been
working on to help track down things like this.  I'm planning to have
things ready for my ZeekWeek presentation, but if you have some time I can
share the work-in-progress stuff with you and go over how to use it (which
will help with the documentation bits that still need to be written).

The good news is I wouldn't be surprised if this issue is already fixed or
drastically better in 3.0 or master.


On Fri, Sep 6, 2019 at 11:47 AM fatema bannatwala <
fatema.bannatwala at gmail.com> wrote:

> Hi All,
>
> Couple of months ago I upgraded the Zeek cluster from 2.5 to 2.6.1
> (compiled with the jemalloc support).
> I have started seeing increased memory usage by the workers.
>
> I have two physical sensors, each running 18 Zeek worker processes LB by
> PF_RING.
> Not loaded any custom scripts, just the basic scripts that are enabled by
> default in local.bro (also have misc/scan disabled).
>
> I just did a top on one of the boxes and here's the output (specially two
> Zeek processes -*13632, **13611* using >10% memory which is ~11G)
> Also, attaching a weekly available free memory graph for the system.
>
> Tasks: 455 total,   9 running, 443 sleeping,   0 stopped,   3 zombie
> %Cpu(s): 18.3 us,  1.7 sy,  0.0 ni, 79.5 id,  0.0 wa,  0.0 hi,  0.4 si,
>  0.0 st
> KiB Mem : 98783960 total, 32963660 free, *64807572* used,  1012728
> buff/cache
> KiB Swap:  4194300 total,  3572200 free,   622100 used. 33221356 avail Mem
>
>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
> 13589 bro       20   0 3662052   3.4g  73340 R    90.4     3.6    1072:47
>   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-5 local.bro broctl base/frameworks/cluster broctl/auto
> 13533 bro       20   0 1847972   1.6g  73188 S    50.3     1.7    1098:05
>   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-6 local.bro broctl base/frameworks/cluster broctl/auto
> 13512 bro       20   0 1291260   1.1g  73052 S    49.7     1.1    1080:30
>   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-1 local.bro broctl base/frameworks/cluster broctl/auto
> 13628 bro       20   0 2347952   2.1g  73328 R    49.0     2.2    1109:31
>   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-12 local.bro broctl base/frameworks/cluster broctl/auto
> 13516 bro       20   0  973260 799176  72844 R  47.0     0.8    1036:29
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-3 local.bro broctl base/frameworks/cluster broctl/auto
> 13539 bro       20   0 6374956   6.0g  73456 S    46.0     6.3    1147:08
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-2 local.bro broctl base/frameworks/cluster broctl/auto
> 13591 bro       20   0  865952 726516  73020 S  44.7     0.7    1052:29
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-4 local.bro broctl base/frameworks/cluster broctl/auto
> *13632* *bro       20   0   12.2g  12.0g  73584 R      43.7  *  *12.8*
> 1068:17   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-15 local.bro broctl
> base/frameworks/cluster broctl/auto
> 13540 bro       20   0 2146844   1.9g  73348 R    41.4     2.0    1149:38
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-7 local.bro broctl base/frameworks/cluster broctl/auto
> *13611* bro       *20   0   17.0g  16.7g  73404 S      39.7*    *17.8*
> 1172:14   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-9 local.bro broctl base/frameworks/cluster
> broctl/auto
> 13640 bro       20   0 2624300   2.1g  73328 S    39.7     2.3    1043:50
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-18 local.bro broctl base/frameworks/cluster broctl/auto
> 13586 bro       20   0 3347044   3.1g  73468 S    39.1     3.2    1042:24
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-10 local.bro broctl base/frameworks/cluster broctl/auto
> 13641 bro       20   0 2274788   2.0g  73424 R    39.1     2.2    1029:58
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-17 local.bro broctl base/frameworks/cluster broctl/auto
> 13614 bro       20   0 1954780   1.7g  73188 S    38.4     1.8
>  995:00.54   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-13 local.bro broctl
> base/frameworks/cluster broctl/auto
> 13627 bro       20   0 2756520   2.5g  73288 S    38.4     2.6
>  1035:18   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-14 local.bro broctl
> base/frameworks/cluster broctl/auto
> 13638 bro       20   0 1206548 853056  72328 R  37.4    0.9   952:10.00
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-16 local.bro broctl base/frameworks/cluster broctl/auto
> 13623 bro       20   0 8998324   2.1g  73284 S     37.1    2.2
>  1073:31   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-11 local.bro broctl
> base/frameworks/cluster broctl/auto
> 13575 bro       20   0  871396 706148  73128 R    36.4   0.7     1028:30
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-8 local.bro broctl base/frameworks/cluster broctl/auto
> 13336 bro       20   0  266244 133920  33388 S    12.6   0.1   400:27.62
>  /usr/local/bro/2.6.1/bin/bro -U .status -p broctl -p broctl-live -p local
> -p proxy-2 local.bro broctl base/frameworks/cluster broctl/auto
>
> Any suggestions?
>
> Thanks!
> Fatema
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190906/7d906ac3/attachment.html 


More information about the Zeek mailing list