[Bro] Memory Issue with Bro
Sampada Kathare
skathare at solarflare.com
Tue Sep 29 14:03:04 PDT 2015
Hi,
I re-ran the same test case just now to get the data for you - 16 bro workers, each pinned to a core, no cpu pinning for the manager and proxy processes, data sent at 155000 pps -
Before running traffic -
top - 21:48:24 up 2:11, 5 users, load average: 0.84, 1.14, 0.88
Tasks: 20 total, 0 running, 20 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.6 us, 2.1 sy, 0.0 ni, 96.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 32900200 total, 8381556 used, 24518644 free, 55736 buffers
KiB Swap: 1953076 total, 0 used, 1953076 free. 249556 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P
18301 root 25 5 129980 46096 884 S 0.0 0.1 0:00.00 bro 1
18299 root 25 5 130020 46116 892 S 0.0 0.1 0:00.00 bro 11
18294 root 25 5 129988 46116 892 S 0.0 0.1 0:00.00 bro 12
18289 root 25 5 130012 46076 840 S 0.0 0.1 0:00.00 bro 15
18285 root 25 5 130024 46128 896 S 0.0 0.1 0:00.00 bro 10
18284 root 25 5 129960 46132 892 S 0.0 0.1 0:00.00 bro 14
18283 root 25 5 130048 46136 888 S 0.0 0.1 0:00.00 bro 13
17863 root 20 0 387408 51964 5548 S 9.7 0.2 0:31.44 bro 16
17860 root 20 0 387344 51968 5548 S 9.3 0.2 0:29.73 bro 15
17836 root 20 0 387416 51956 5548 S 8.7 0.2 0:29.62 bro 14
17835 root 20 0 387348 51960 5548 S 9.7 0.2 0:30.02 bro 13
17834 root 20 0 387348 51948 5552 S 9.0 0.2 0:29.31 bro 11
17833 root 20 0 387292 51952 5548 S 9.3 0.2 0:29.26 bro 12
17832 root 20 0 387340 51960 5548 S 9.3 0.2 0:29.85 bro 10
17738 root 20 0 387296 51920 5548 S 10.0 0.2 0:30.59 bro 1
17651 root 25 5 145956 75268 960 S 0.0 0.2 0:00.00 bro 2
17650 root 20 0 109988 43080 5096 S 1.3 0.1 0:04.50 bro 0
17613 root 25 5 146096 75396 944 S 0.0 0.2 0:00.01 bro 9
17604 root 20 0 405392 45428 5116 S 0.7 0.1 0:03.71 bro 26
17457 root 20 0 196772 9016 3448 S 0.0 0.0 0:00.02 solar_clusterd 0
The highlighted rows are Bro manager processes (17613 being the child and 17604 being the parent)
After running traffic at 150000 pps for 3 minutes ->
top - 21:53:58 up 2:16, 5 users, load average: 20.90, 11.89, 5.36
Tasks: 20 total, 10 running, 10 sleeping, 0 stopped, 0 zombie
%Cpu(s): 55.3 us, 3.4 sy, 0.8 ni, 37.5 id, 3.0 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem: 32900200 total, 32703156 used, 197044 free, 892 buffers
KiB Swap: 1953076 total, 626996 used, 1326080 free. 331096 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P
18301 root 25 5 134092 46032 676 S 0.3 0.1 0:00.78 bro 1
18299 root 25 5 134132 46052 676 S 0.3 0.1 0:00.79 bro 11
18294 root 25 5 134100 2024 676 S 0.7 0.0 0:00.88 bro 12
18289 root 25 5 134124 46000 676 S 0.3 0.1 0:00.82 bro 15
18285 root 25 5 134136 2016 676 S 0.3 0.0 0:00.87 bro 10
18284 root 25 5 134072 1992 676 S 0.7 0.0 0:00.87 bro 14
18283 root 25 5 134160 46064 676 S 0.7 0.1 0:00.80 bro 13
17863 root 20 0 404636 24628 3540 R 0.0 0.1 4:36.63 bro 16
17860 root 20 0 404668 67472 3544 R 100.0 0.2 4:29.44 bro 15
17836 root 20 0 404612 24580 3540 R 100.0 0.1 4:35.53 bro 14
17835 root 20 0 404608 67412 3544 R 0.0 0.2 4:33.82 bro 13
17834 root 20 0 404608 67388 3544 R 100.0 0.2 4:35.69 bro 11
17833 root 20 0 404636 24648 3540 R 0.0 0.1 4:32.05 bro 12
17832 root 20 0 404632 24588 3540 R 100.0 0.1 4:36.37 bro 10
17738 root 20 0 404624 67368 3544 R 100.0 0.2 4:38.29 bro 1
17651 root 25 5 145956 7488 648 S 0.0 0.0 0:00.00 bro 17
17650 root 20 0 109988 3892 2916 S 1.7 0.0 0:09.39 bro 29
17613 root 25 5 871884 763968 668 R 99.9 2.3 3:39.04 bro 17
17604 root 20 0 22.869g 0.022t 2932 R 195.4 71.2 7:41.59 bro 18
17457 root 20 0 196772 2272 2272 S 0.0 0.0 0:00.02 solar_clusterd 0
As you can see, the manager parent process seems to be using 71% of the memory and it's CPU utilization is also 195%.
-
Sampada
-----Original Message-----
From: Azoff, Justin S [mailto:jazoff at illinois.edu]
Sent: Tuesday, September 29, 2015 12:19 PM
To: Sampada Kathare <skathare at solarflare.com>
Cc: bro at bro.org
Subject: Re: [Bro] Memory Issue with Bro
> On Sep 29, 2015, at 12:29 PM, Sampada Kathare <skathare at solarflare.com<mailto:skathare at solarflare.com>> wrote:
>
> Hi,
>
> These are the bro workers. I haven't shown the memory usage of the Bro manager and proxy processes. I believe the manager is the one that takes up most of the memory as when I stop the manager, the available free memory goes up by almost 10G! I will send out that log shortly.
There's actually a manager parent and child process, knowing which one is using the memory can help figure this out.
> Could the manager in anyway be accumulating per worker or per flow state and not freeing it?
>
> Thanks!
There is sort of a known issue if the manager can't keep up logging the amount of data it is being sent.
What sort of data are you sending bro? Is it something like random data that will cause a LOT of logging?
--
- Justin Azoff
The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error, please notify the sender immediately and delete the message. Unless you are an addressee (or authorized to receive for an addressee), you may not use, copy or disclose to anyone this message or any information contained in this message. The unauthorized use, disclosure, copying or alteration of this message is strictly prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150929/3258b635/attachment-0001.html
More information about the Bro
mailing list