[Bro] Version: 2.0-907 -- Bro manager memory exhaustion

[Siwek, Jonathan Luke]

> I have the following in my local.bro file:
> 
> redef SMTP::generate_md5 += /image.*/;
> redef HTTP::generate_md5 += /image.*/;
...
> 
> Using broctl's top and a little trial and error, I can see that these
> lines are the cause of my high CPU usage.  It also causes higher
> memory usage as well, but memory usage always climbs and never gets
> smaller.  I don't know if these lines are responsible for just higher
> memory usage in general, or whether they are also responsible gradual
> climb in memory.  It appears that memory gradually climbs even without
> these lines, but I haven't had enough time to test that idea. 

In general, the digest BiFs don't look like they leak, but if there is not a md5_hash_finish() for each corresponding md5_hash_init(), that could lead to growth of some internal state over time.  The base scripts all attempt to clean up any md5_hash_init()'s with a corresponding md5_hash_finish(), but I'm not confident all edge cases are covered.

If you have any other local changes, you might see if there's a difference running with them rather than just the vanilla bro scripts -- it can be easy to add something which causes too much state to accumulate over time.  Another quick check is to look for any errors in reporter.log -- currently interpreter exceptions due to scripting errors will not abort bro, but do cause a memory leak.  Otherwise, it might be easiest for you to start looking into using a memory profiling tool (e.g. valgrind, gperftools) to try to locate the problem more definitely.

    Jon