[Bro] Memory Issue with Bro

Sampada Kathare skathare at solarflare.com
Tue Sep 29 14:08:08 PDT 2015


Continuing the previous test case - After another 7-8 minutes, I don't see the BRO manager process running and memory seems to have been restored -

top - 22:04:54 up  2:27,  5 users,  load average: 16.04, 16.06, 11.24
Tasks:  18 total,   8 running,  10 sleeping,   0 stopped,   0 zombie
%Cpu(s): 47.7 us,  0.1 sy,  0.0 ni, 52.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  32900200 total,  8363464 used, 24536736 free,     4152 buffers
KiB Swap:  1953076 total,   743412 used,  1209664 free.   608176 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                         P
18301 root      25   5  132036  19816    540 S   0.0  0.1   0:00.88 bro                             1
18299 root      25   5  132076  25040    468 S   0.0  0.1   0:00.83 bro                            11
18294 root      25   5  132044   4208    480 S   0.0  0.0   0:01.04 bro                            12
18289 root      25   5  132068  22424    548 S   0.0  0.1   0:00.84 bro                            15
18285 root      25   5  132080   2276    468 S   0.0  0.0   0:00.89 bro                            10
18284 root      25   5  132016   2276    444 S   0.0  0.0   0:00.92 bro                            14
18283 root      25   5  132104  20792    520 S   0.0  0.1   0:00.85 bro                            13
17863 root      20   0  404724  24268   1568 R 100.0  0.1  15:23.31 bro                            16
17860 root      20   0  404592  44080   1568 R  91.2  0.1  14:33.31 bro                            15
17836 root      20   0  405076  24604   1592 R  94.5  0.1  15:09.50 bro                            14
17835 root      20   0  404632  42456   1568 R   0.0  0.1  15:06.73 bro                            13
17834 root      20   0  404616  42740   1592 R  95.2  0.1  15:02.40 bro                            11
17833 root      20   0  404612  24252   1592 R  94.9  0.1  15:00.49 bro                            12
17832 root      20   0  404740  24052   1528 R 100.0  0.1  15:20.78 bro                            10
17738 root      20   0  404676  41420   1528 R   0.0  0.1  15:27.26 bro                             1
17651 root      25   5  143900   4820    360 S   0.0  0.0   0:00.01 bro                            19
17650 root      20   0  109988   1916    736 S   1.3  0.0   0:19.00 bro                            19
17457 root      20   0  196772    272    272 S   0.0  0.0   0:00.02 solar_clusterd                  0


PID 17604 and 17613 missing above?

Is this an expected behavior? During the initial few minutes, does the manager do some sort of stabilization to get everything in order, possibly due to the high traffic rate? It is during these few minutes that I saw packets being dropped. I don't see any drops right now.

-
Sampada

From: Sampada Kathare
Sent: Tuesday, September 29, 2015 2:04 PM
To: 'Azoff, Justin S' <jazoff at illinois.edu>
Cc: bro at bro.org
Subject: RE: [Bro] Memory Issue with Bro


Hi,



I re-ran the same test case just now to get the data for you - 16 bro workers, each pinned to a core, no cpu pinning for the manager and proxy processes, data sent at 155000 pps -



Before running traffic -



top - 21:48:24 up  2:11,  5 users,  load average: 0.84, 1.14, 0.88

Tasks:  20 total,   0 running,  20 sleeping,   0 stopped,   0 zombie

%Cpu(s):  1.6 us,  2.1 sy,  0.0 ni, 96.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

KiB Mem:  32900200 total,  8381556 used, 24518644 free,    55736 buffers

KiB Swap:  1953076 total,        0 used,  1953076 free.   249556 cached Mem



  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                         P

18301 root      25   5  129980  46096    884 S   0.0  0.1   0:00.00 bro                             1

18299 root      25   5  130020  46116    892 S   0.0  0.1   0:00.00 bro                            11

18294 root      25   5  129988  46116    892 S   0.0  0.1   0:00.00 bro                            12

18289 root      25   5  130012  46076    840 S   0.0  0.1   0:00.00 bro                            15

18285 root      25   5  130024  46128    896 S   0.0  0.1   0:00.00 bro                            10

18284 root      25   5  129960  46132    892 S   0.0  0.1   0:00.00 bro                            14

18283 root      25   5  130048  46136    888 S   0.0  0.1   0:00.00 bro                            13

17863 root      20   0  387408  51964   5548 S   9.7  0.2   0:31.44 bro                            16

17860 root      20   0  387344  51968   5548 S   9.3  0.2   0:29.73 bro                            15

17836 root      20   0  387416  51956   5548 S   8.7  0.2   0:29.62 bro                            14

17835 root      20   0  387348  51960   5548 S   9.7  0.2   0:30.02 bro                            13

17834 root      20   0  387348  51948   5552 S   9.0  0.2   0:29.31 bro                            11

17833 root      20   0  387292  51952   5548 S   9.3  0.2   0:29.26 bro                            12

17832 root      20   0  387340  51960   5548 S   9.3  0.2   0:29.85 bro                            10

17738 root      20   0  387296  51920   5548 S  10.0  0.2   0:30.59 bro                             1

17651 root      25   5  145956  75268    960 S   0.0  0.2   0:00.00 bro                             2

17650 root      20   0  109988  43080   5096 S   1.3  0.1   0:04.50 bro                             0

17613 root      25   5  146096  75396    944 S   0.0  0.2   0:00.01 bro                             9

17604 root      20   0  405392  45428   5116 S   0.7  0.1   0:03.71 bro                            26

17457 root      20   0  196772   9016   3448 S   0.0  0.0   0:00.02 solar_clusterd                  0



The highlighted rows are Bro manager processes (17613 being the child and 17604 being the parent)



After running traffic at 150000 pps for 3 minutes ->



top - 21:53:58 up  2:16,  5 users,  load average: 20.90, 11.89, 5.36

Tasks:  20 total,  10 running,  10 sleeping,   0 stopped,   0 zombie

%Cpu(s): 55.3 us,  3.4 sy,  0.8 ni, 37.5 id,  3.0 wa,  0.0 hi,  0.1 si,  0.0 st

KiB Mem:  32900200 total, 32703156 used,   197044 free,      892 buffers

KiB Swap:  1953076 total,   626996 used,  1326080 free.   331096 cached Mem



  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                         P

18301 root      25   5  134092  46032    676 S   0.3  0.1   0:00.78 bro                             1

18299 root      25   5  134132  46052    676 S   0.3  0.1   0:00.79 bro                            11

18294 root      25   5  134100   2024    676 S   0.7  0.0   0:00.88 bro                            12

18289 root      25   5  134124  46000    676 S   0.3  0.1   0:00.82 bro                            15

18285 root      25   5  134136   2016    676 S   0.3  0.0   0:00.87 bro                            10

18284 root      25   5  134072   1992    676 S   0.7  0.0   0:00.87 bro                            14

18283 root      25   5  134160  46064    676 S   0.7  0.1   0:00.80 bro                            13

17863 root      20   0  404636  24628   3540 R   0.0  0.1   4:36.63 bro                            16

17860 root      20   0  404668  67472   3544 R 100.0  0.2   4:29.44 bro                            15

17836 root      20   0  404612  24580   3540 R 100.0  0.1   4:35.53 bro                            14

17835 root      20   0  404608  67412   3544 R   0.0  0.2   4:33.82 bro                            13

17834 root      20   0  404608  67388   3544 R 100.0  0.2   4:35.69 bro                            11

17833 root      20   0  404636  24648   3540 R   0.0  0.1   4:32.05 bro                            12

17832 root      20   0  404632  24588   3540 R 100.0  0.1   4:36.37 bro                            10

17738 root      20   0  404624  67368   3544 R 100.0  0.2   4:38.29 bro                             1

17651 root      25   5  145956   7488    648 S   0.0  0.0   0:00.00 bro                            17

17650 root      20   0  109988   3892   2916 S   1.7  0.0   0:09.39 bro                            29

17613 root      25   5  871884 763968    668 R  99.9  2.3   3:39.04 bro                            17

17604 root      20   0 22.869g 0.022t   2932 R 195.4 71.2   7:41.59 bro                            18

17457 root      20   0  196772   2272   2272 S   0.0  0.0   0:00.02 solar_clusterd                  0



As you can see, the manager parent process seems to be using 71% of the memory and it's CPU utilization is also 195%.



-

Sampada



-----Original Message-----
From: Azoff, Justin S [mailto:jazoff at illinois.edu]
Sent: Tuesday, September 29, 2015 12:19 PM
To: Sampada Kathare <skathare at solarflare.com<mailto:skathare at solarflare.com>>
Cc: bro at bro.org<mailto:bro at bro.org>
Subject: Re: [Bro] Memory Issue with Bro





> On Sep 29, 2015, at 12:29 PM, Sampada Kathare <skathare at solarflare.com<mailto:skathare at solarflare.com>> wrote:

>

> Hi,

>

> These are the bro workers. I haven't shown the memory usage of the Bro manager and proxy processes. I believe the manager is the one that takes up most of the memory as when I stop the manager, the available free memory goes up by almost 10G! I will send out that log shortly.



There's actually a manager parent and child process, knowing which one is using the memory can help figure this out.



> Could the manager in anyway be accumulating per worker or per flow state and not freeing it?

>

> Thanks!



There is sort of a known issue if the manager can't keep up logging the amount of data it is being sent.

What sort of data are you sending bro? Is it something like random data that will cause a LOT of logging?



--

- Justin Azoff



The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error, please notify the sender immediately and delete the message. Unless you are an addressee (or authorized to receive for an addressee), you may not use, copy or disclose to anyone this message or any information contained in this message. The unauthorized use, disclosure, copying or alteration of this message is strictly prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150929/187874d9/attachment-0001.html 


More information about the Bro mailing list