[Bro] Memory Issue with Bro

Sampada Kathare skathare at solarflare.com
Mon Sep 28 13:39:24 PDT 2015


In any case, I think I can give it a shot at least for the PF-RING testcase. How do I re-compile PF-RING for the newest kernel?

-
Sampada

From: Sampada Kathare
Sent: Monday, September 28, 2015 1:37 PM
To: 'Mark Buchanan' <mabuchan at gmail.com>
Cc: bro at bro.org
Subject: RE: [Bro] Memory Issue with Bro

Mark,

I have tried Bro both, with and without PF-RING, and saw the memory issue in both cases. When I am not using PF-RING, I use Solar Capture libpcap and flow balancer.

-
Sampada

From: Mark Buchanan [mailto:mabuchan at gmail.com]
Sent: Monday, September 28, 2015 1:28 PM
To: Sampada Kathare <skathare at solarflare.com<mailto:skathare at solarflare.com>>
Cc: bro at bro.org<mailto:bro at bro.org>
Subject: Re: [Bro] Memory Issue with Bro

Sampada,

I've observed Bro utilizing more memory when PF_RING modules haven't been recompiled for the newest kernel.  Once I recompiled for the current kernel, my memory usage got more inline with the numbers I would expect to see.  Could that be a problem for you or is PF_RING not in the equation?

Mark

On Mon, Sep 28, 2015 at 2:59 PM, Sampada Kathare <skathare at solarflare.com<mailto:skathare at solarflare.com>> wrote:
Hello,

I have been using the Bro 2.4 to test the performance of SFC driver. I have observed the following issue because of which I am unable to proceed with any analysis -

There seems to be a memory leak somewhere as there are times when Bro runs out of memory too soon. These are the instances when drops are also seen too soon even at very low packet rates.
When Bro is started, the available free memory keeps going down till a point where the server is extremely sluggish and there are drops being seen –

An instance of Bro running out of memory (with 16 workers, no cpu pinning and having sent 155K pps for 7-8 minutes)–
[root at dellr620c skathare]# free -m
             total       used       free     shared    buffers     cached
Mem:         32129      31917        211          3          1        376      --> 211MB : that’s very low, considering the system started with some 26GB free memory (and this drop happens just within the first 2 minutes of running the      traffic). System becomes very slow at this point and, of course, it has started dropping packets already.
-/+ buffers/cache:      31539        589
Swap:         1907       1687        219
[root at dellr620c skathare]#

Swap:         1907       1764        142


[root at dellr620c skathare]# cat /proc/meminfo
MemTotal:       32900200 kB
MemFree:          193384 kB
MemAvailable:     480956 kB
Buffers:            2464 kB
Cached:           471260 kB
SwapCached:        74860 kB
Active:         23439908 kB
Inactive:        3120012 kB
Active(anon):   23179296 kB
Inactive(anon):  2914628 kB
Active(file):     260612 kB
Inactive(file):   205384 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       1953076 kB
SwapFree:         152264 kB
Dirty:             22548 kB
Writeback:             8 kB
AnonPages:      26017216 kB
Mapped:            15200 kB
Shmem:              4692 kB
Slab:             190556 kB
SReclaimable:      93648 kB
SUnreclaim:        96908 kB
KernelStack:        4288 kB
PageTables:        71380 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    15638376 kB
Committed_AS:   29119672 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      374920 kB
VmallocChunk:   34342144308 kB
HardwareCorrupted:     0 kB
AnonHugePages:    243712 kB
HugePages_Total:    2700
HugePages_Free:       46
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      345024 kB
DirectMap2M:    18483200 kB
DirectMap1G:    16777216 kB


[root at dellr620c skathare]#top

top - 22:48:06 up 70 days, 19:43,  5 users,  load average: 18.25, 13.49, 10.04
Tasks:  17 total,   1 running,  16 sleeping,   0 stopped,   0 zombie
%Cpu(s): 31.5 us,  2.9 sy,  0.7 ni, 43.7 id, 21.1 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  32900200 total, 32731868 used,   168332 free,      336 buffers
KiB Swap:  1953076 total,  1262956 used,   690120 free.    14248 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                  P
20504 root      20   0  404644  32724   1464 D   0.0  0.1   6:32.62 bro                                                     31
20552 root      20   0  404692  50108   1496 D  52.1  0.2   6:37.18 bro                                                     29
20564 root      20   0  404824  50960   1508 D  49.8  0.2   6:37.16 bro                                                     27
20574 root      20   0  404652  48748   1476 D  52.0  0.1   6:36.52 bro                                                     24
20567 root      20   0  404684  49948   1456 D   0.0  0.2   6:33.32 bro                                                     23
20561 root      20   0  421440  66672   1412 D  51.9  0.2   6:37.14 bro                                                     18
20569 root      20   0  404708  31904   1508 D  41.1  0.1   6:34.77 bro                                                     16
20495 root      20   0  404620  49936   1408 D  27.1  0.2   6:34.91 bro                                                     13
20515 root      20   0  404684  46324   1500 D  21.9  0.1   6:33.25 bro                                                     13
20548 root      20   0  404704  50188   1504 D  43.6  0.2   6:35.16 bro                                                     13
20474 root      20   0  404736  32704   1508 D   0.0  0.1   6:32.79 bro                                                     12
20502 root      20   0  404636  29300   1464 D  52.0  0.1   6:36.13 bro                                                     12
20539 root      20   0  404748  32784   1484 D  44.7  0.1   6:34.08 bro                                                     11
20537 root      20   0  404668  29284   1464 D   0.0  0.1   6:32.03 bro                                                      4
20559 root      20   0  404684  32644   1444 R  54.6  0.1   6:38.12 bro                                                      3
20542 root      20   0  404728  32704   1504 D  25.1  0.1   6:33.84 bro                                                      1
20289 root      20   0  196768    412    412 S   0.0  0.0   0:00.03 solar_clusterd                                           0


After stopping the BRO workers (especially after the manager is killed/stopped), memory recovers –

top - 22:53:01 up 70 days, 19:48,  5 users,  load average: 3.06, 8.83, 9.20
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.1 us,  0.3 sy,  2.2 ni, 96.6 id,  0.8 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  32900200 total,  6702252 used, 26197948 free,     8308 buffers           --> This is almost what the system originally started with – 26GB
KiB Swap:  1953076 total,   237216 used,  1715860 free.   528032 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                  P
20289 root      20   0  196768   1028    484 S   0.0  0.0   0:00.03 solar_clusterd



At very high packet rates, the available free memory keeps going down very fast and starts dropping packets. At lower packet rates, the drop in available free memory is comparatively slower, but it is still there and packets are dropped eventually. When the BRO workers are stopped, the available free memory recovers. During the few successful times when I have been able to go till 150Kpps without seeing any packet drops, the available free memory remained a constant at  ~23G. It remained at this for the entire duration of the test (more than an hour ) and no drops were seen.

The above data is a few days old. When I tried running BRO again today, I saw the memory drop from 18G to 4G in just a matter of few seconds after starting BRO (16 workers, each pinned to a CPU). Is it possible that Bro is accumulating some per-flow state and not freeing it? If so, is there any tuning that should be done to avoid this?

Appreciate any help on this!

-
Sampada
The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error, please notify the sender immediately and delete the message. Unless you are an addressee (or authorized to receive for an addressee), you may not use, copy or disclose to anyone this message or any information contained in this message. The unauthorized use, disclosure, copying or alteration of this message is strictly prohibited.

_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



--
Mark Buchanan
The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error, please notify the sender immediately and delete the message. Unless you are an addressee (or authorized to receive for an addressee), you may not use, copy or disclose to anyone this message or any information contained in this message. The unauthorized use, disclosure, copying or alteration of this message is strictly prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150928/04089214/attachment-0001.html 


More information about the Bro mailing list