[Bro-Dev] Memory leaks on git eds2245

[Gregor Maier]

On 9/7/11 22:37 , Seth Hall wrote:
> On Sep 8, 2011, at 1:25 AM, Gregor Maier wrote:
>
>> (BTW, the memory problems I have/had weren't "real" leaks. One a SSL
>> connection was done Bro would free the memory for it again. The problem
>> is that many SSL connections can live for days and thus they ultimately
>> consume memory like a "real" leak would).
>
>
> I implemented the code yesterday to stop analyzing connections with the skip_further_processing bif and it caused Bro to peak using more memory on the tracefile I was using it with than not stopping analysis of connections.  One thing the SSL scripts are currently doing that I probably need to change is after logging the SSL log, I should probably do "delete c$ssl".  The certificate and certificate chain are stored in there.  Actually, as I think about it more that's probably most of the problem.

skip_further_processing() actually just sets the skip flag which means 
that no further data is delivered to the analyzers, but the analyzers 
aren't removed.
The disable_analyzer() bif will actually remove the analyzer, however it 
needs an analyzer_id, so the SSL analyzer would somehow need to add it's 
analyzer_id to one of it's events.

> We may want to look into the real traffic implications of calling the skip_further_processing bif eventually too though.  I was pretty disheartened to see more memory used from calling that than not calling it.  Perhaps it results in more memory use to remember which connections to ignore?  I suppose I wasn't checking completion time which is probably where the savings should mostly come from.

How much difference in memory usage did you see? And how much memory 
usage do you see in general? My Bros usually need about 100-300MB and 
with SSL I will eventually get to 1GB or more (not on all nodes and it 
will often take hours until it starts get there).
(Disabling SSL altogether will reduce the "baseline" usage by about 
25%--50%)

However, it still puzzles me that skip_further_processing didn't help. 
In my case memory consumption was ramping up over several minutes so 
either data is still delivered to the analyzer during this ramp-up or 
there are additional reasons for the memory consumption?


cu
Gregor
-- 
Gregor Maier
<gregor at icir.org>  <gregor at icsi.berkeley.edu>
Int. Computer Science Institute (ICSI)
1947 Center St., Ste. 600
Berkeley, CA 94704, USA
http://www.icir.org/gregor/