[Bro] Bro's limitations with high worker count and memory exhaustion

[Baxter Milliwew]

Nevermind... new box, default nofile limits.  Thanks for the malloc tip.


On Mon, Jun 29, 2015 at 4:03 PM, Baxter Milliwew <baxter.milliwew at gmail.com>
wrote:

> Switching to jemalloc fixed the stability issue but not the worker count
> limitation.
>
> On Sun, Jun 28, 2015 at 7:18 PM, Baxter Milliwew <
> baxter.milliwew at gmail.com> wrote:
>
>> Looks like malloc from glibc, default on Ubuntu.  I will try jemalloc and
>> others.
>>
>>
>>
>> On Sun, Jun 28, 2015 at 1:03 AM, Jan Grashofer <jan.grashofer at cern.ch>
>> wrote:
>>
>>>  I experienced similar problems (memory gets eaten up quickly and
>>> workers crash with segfault) using tcmalloc. Which malloc do you use?
>>>
>>>
>>>
>>> Regards,
>>>
>>> Jan
>>>
>>>
>>>  ------------------------------
>>> *From:* bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Baxter
>>> Milliwew [baxter.milliwew at gmail.com]
>>> *Sent:* Friday, June 26, 2015 23:03
>>> *To:* bro at bro.org
>>> *Subject:* [Bro] Bro's limitations with high worker count and memory
>>> exhaustion
>>>
>>>   There's some sort of association between memory exhaustion and a high
>>> number of workers.  The poor man's fix would be to purchase new servers
>>> with higher CPU speeds as that would reduce the worker count.  Issues with
>>> high worker count and/or memory exhaustion appears to be a well know
>>> problem based on the mailing list archives.
>>>
>>>  In the current version of bro-2.4 my previous configuration
>>> immediately causes the manager to crash: 15 proxies, 155 workers.  To
>>> resolve this I've lowered the count to 10 proxies and 140 workers.  However
>>> even with this configuration the manager process will exhaust all memory
>>> and crash within about 2 hours.
>>>
>>>  The manager is threaded; I think this is an issue with the threading
>>> behavior between manager, proxies, and workers.  Debugging threading
>>> problems is complex and I'm a complete novice.. my current tutorial is
>>> using information from a stack overflow thread:
>>>
>>>
>>> http://stackoverflow.com/questions/981011/c-programming-debugging-with-pthreads
>>>
>>>  Does anyone else have this problem ?  What have you tried and what do
>>> you suggest ?
>>>
>>>  Thanks
>>>
>>>
>>>
>>>
>>>   1435347409.458185       worker-2-18     parent  -       -       -
>>>   info    [#10000/10.1.1.1:36994] peer sent class "control"
>>>
>>> 1435347409.458185       worker-2-18     parent  -       -       -
>>> info    [#10000/10.1.1.1:36994] phase: handshake
>>>
>>> 1435347409.661085       worker-2-18     parent  -       -       -
>>> info    [#10000/10.1.1.1:36994] request for unknown event save_results
>>>
>>> 1435347409.661085       worker-2-18     parent  -       -       -
>>> info    [#10000/10.1.1.1:36994] registered for event
>>> Control::peer_status_response
>>>
>>> 1435347409.694858       worker-2-18     parent  -       -       -
>>> info    [#10000/10.1.1.1:36994] peer does not support 64bit PIDs; using
>>> compatibility mode
>>>
>>> 1435347409.694858       worker-2-18     parent  -       -       -
>>> info    [#10000/10.1.1.1:36994] peer is a Broccoli
>>>
>>> 1435347409.694858       worker-2-18     parent  -       -       -
>>> info    [#10000/10.1.1.1:36994] phase: running
>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150629/7b7e833e/attachment.html