[Bro] question about intel files

[philosnef]

With  hyperthreading that's actually 40 cores, not 20. Running 20 workers with 40 cores available should be more than sufficient. At the time brotop was run, 355 out of 390 gigs of ram are in use. The only things running on this box are bro, and a splunk forwarder. The splunk forwarder is only using about 15 gigs of ram. This excessive memory consumption is on all of our bro boxes, no matter the input stream. Even on boxes only getting 500Mb/s, we see this memory creep until it is exhausted. At no point is oomkiller called however, so it is not exceeding available memory, just consuming all of the available memory.
brotop---Name         Type    Host             Pid     Proc    VSize  Rss  Cpu   Cmdmanager      manager localhost        67408   parent  884M   343M 136%  bromanager      manager localhost        67442   child   346M   179M  24%  broproxy-1      proxy   localhost        67512   parent  366M   284M   3%  broproxy-1      proxy   localhost        67542   child   201M   114M   3%  broproxy-2      proxy   localhost        67543   child   201M   107M   3%  broproxy-2      proxy   localhost        67513   parent  366M   284M   1%  broworker-1-1   worker  localhost        67683   parent    1G     1G 100%  broworker-1-1   worker  localhost        68236   child   716M   625M   3%  broworker-1-10  worker  localhost        67688   parent    1G     1G  96%  broworker-1-10  worker  localhost        68278   child   716M   629M   1%  broworker-1-11  worker  localhost        67697   parent    2G     2G 100%  broworker-1-11  worker  localhost        68229   child   716M   628M   0%  broworker-1-12  worker  localhost        67712   parent    1G     1G  83%  broworker-1-12  worker  localhost        68264   child   716M   629M   1%  broworker-1-13  worker  localhost        67717   parent    4G     4G 100%  broworker-1-13  worker  localhost        68233   child   716M   627M   1%  broworker-1-14  worker  localhost        67737   parent    1G     1G  98%  broworker-1-14  worker  localhost        68223   child   716M   626M   1%  broworker-1-15  worker  localhost        67752   parent    2G     2G 100%  broworker-1-15  worker  localhost        68269   child   716M   626M   0%  broworker-1-16  worker  localhost        67749   parent    1G     1G  72%  broworker-1-16  worker  localhost        68228   child   716M   630M   0%  broworker-1-17  worker  localhost        67758   parent    2G     2G  87%  broworker-1-17  worker  localhost        68263   child   716M   627M   1%  broworker-1-18  worker  localhost        67764   parent    1G     1G  98%  broworker-1-18  worker  localhost        68254   child   716M   626M   1%  broworker-1-19  worker  localhost        67767   parent    1G     1G  66%  broworker-1-19  worker  localhost        68239   child   716M   629M   0%  broworker-1-2   worker  localhost        67774   parent    1G     1G  98%  broworker-1-2   worker  localhost        68230   child   716M   625M   0%  broworker-1-20  worker  localhost        67794   parent    3G     3G  98%  broworker-1-20  worker  localhost        68245   child   716M   629M   3%  broworker-1-3   worker  localhost        67792   parent    1G     1G  91%  broworker-1-3   worker  localhost        68265   child   716M   627M   3%  broworker-1-4   worker  localhost        67800   parent    1G     1G  83%  broworker-1-4   worker  localhost        68248   child   716M   628M   1%  broworker-1-5   worker  localhost        67799   parent    1G     1G  98%  broworker-1-5   worker  localhost        68277   child   716M   626M   0%  broworker-1-6   worker  localhost        67801   parent    1G     1G  85%  broworker-1-6   worker  localhost        68279   child   716M   626M   1%  broworker-1-7   worker  localhost        67813   parent    1G     1G 100%  broworker-1-7   worker  localhost        68251   child   716M   628M   1%  broworker-1-8   worker  localhost        67812   parent    1G     1G  79%  broworker-1-8   worker  localhost        68244   child   716M   629M   0%  broworker-1-9   worker  localhost        67814   parent    1G     1G  96%  broworker-1-9   worker  localhost        68266   child   716M   626M   1%  bro
 

    On Wednesday, August 3, 2016 9:43 AM, "Azoff, Justin S" <jazoff at illinois.edu> wrote:
 

 
> On Aug 3, 2016, at 7:22 AM, philosnef <philosnef at yahoo.com> wrote:
> 
> We have 2 10 physical core systems with 20 logical cores for a total of 40. Bro has a capture loss of sub .5% across all workers, so it seems unlikely that the box is overloaded. The capture rate of the box, per pfring is about 3.5Gb/s. We reported memory issues in the past, but those were written off as not related to the memory leak recently patched in the 24 branch and the 25 branch.

What process is using memory?  Workers? Proxies? Manager?  If you can include the output of 'broctl top' that would be helpful.  Otherwise it is pretty hard to determine what the issue may even be.

If you have a dual 10 core system and are running 20 workers then that leaves no room for the manager or for any tasks like log rotation.  For a 20 core system I would run at most 18 workers.

-- 
- Justin Azoff


  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160803/dfb40020/attachment-0001.html