[Bro-Dev] [Bro] Version: 2.0-907 -- Bro manager memory exhaustion

[Vlad Grigorescu]

An average of 9000 logs/second.

  --Vlad

On 7/31/12 11:43 PM, "Martin Holste" <mcholste at gmail.com> wrote:

What was the logging rate for ES and raw ASCII?

On Tue, Jul 31, 2012 at 8:26 PM, Vlad Grigorescu <vladg at cmu.edu> wrote:
> This seems to me to just be an issue of my ElasticSearch server not
> keeping up with the load.
>
> I ran master with just the Ascii logs for a few hours, and saw no
>evidence
> of a memleak. Valgrind also came back cleanly (and actually, insertion
> rate into ElasticSearch was about the same with/without valgrind - I was
> expecting more of a performance hit).
>
>   --Vlad
>
> On 7/31/12 12:59 PM, "Vlad Grigorescu" <vladg at cmu.edu> wrote:
>
> I've been running 2.0-905 for ~25-26 hours. The manager's memory usage
>has
> slowly crept up to 13 GB.
>
> One thing of note - I'm using the ElasticSearch log writer. I see 3
> possible scenarios for this memleak:
>
> 1) There is indeed a leak in master, potentially only triggered by
> specific traffic,
> 2) There is a leak in the ElasticSearch log writer,
> 3) My ElasticSearch server can't keep up with the load, and the manager
>is
> receiving logs faster than it can send them to the writer, so they just
> queue up.
>
> Has anyone else tried the current code over an extended period on live
> traffic? Also, if anyone has any ideas to try to figure out where this
> leak is occurring, please let me know. I'm going to switch back to ASCII
> logs for a bit, and see what that's like.
>
>   --Vlad
>
> On 7/31/12 12:26 PM, "Robin Sommer" <robin at icir.org> wrote:
>
> (Taking to bro-dev).
>
> On Tue, Jul 31, 2012 at 08:11 -0700, I wrote:
>
>> That's not a good sign for the manager ... It's possible that we have
>> a memory leak in there.
>
> I just reran our leak tests and they didn't report anything (which is
> good, but doesn't completely rule out any leaks).
>
> I did see this though from valgrind:
>
>     Object at 0x94e3410 of 68 bytes from an IgnoreObject() has
>disappeared
>
> Does anybody know what valgrind it trying to tell me with that? Is it
> a problem?
>
> Robin
>
> --
> Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
> ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro-ids.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>
>
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro-ids.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>
>
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro-ids.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
_______________________________________________
bro-dev mailing list
bro-dev at bro-ids.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev