[Bro] Version: 2.0-907 -- Bro manager memory exhaustion

[Tritium Cat]

On Wed, Aug 1, 2012 at 9:09 PM, Martin Holste <mcholste at gmail.com> wrote:

> > You might be right.  I had considered disk I/O and ran the manager on a
> > decent raid-10 array (the elsa server), disk i/o did not appear to be the
> > problem so much as CPU and memory.  That observation carried over to
> > development builds with a threaded manager, but at an accelerated rate;
> > right now I'm watching the disk I/O under threshold and the manager is
> > consuming 100M of memory per second until memory exhaustion.
>
> How much I/O from the manager have you seen thus far?  I'm not yet
> convinced that writing raw text files is the bottleneck.  When you
> think about writing raw pcap, it's orders of magnitude more MB/sec to
> disk than logging.
>


The highest I've seen so far is 443 MB/s with waves of activity between 80
- 390 MB/s.  That's all within a 3 or 4 minute window of time before all
the memory is near exhaustion.  (45-50G for the manager).

I'm using the following to help profile the system:
  people.freebsd.org/~kris/scaling/Help_my_system_is_slow.pdf

--TC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120801/2acd9024/attachment.html