[Bro-Dev] input framework and tuning options

[Jon Siwek]

On Mon, Jan 22, 2018 at 11:31 PM, Scott Campbell <scampbell at lbl.gov> wrote:

> The data source is a log file that grows at ~7-9k records/sec and
> consists of small text lines of < 512 bytes, newline delimited.
>
> The primary symptom here is a steadily growing memory footprint even
> though the back end analyzer seems to be processing the events in near
> real time - i.e. there is obviously some buffering going on but the data
> is being consumed.  The footprint for script side variables is not to
> blame as it is always << 1% of the total.

The main categories of problems to check for that come to mind:

(a) Rate of production exceeds rate of consumption
(b) Unbounded script state accumulation
(c) Unbounded core state accumulation
(d) Memory leak

It sounds like you've ruled out (a) and (b).  For the others, using a
heap profiler/checker is going to help.  There's a brief guide at [1]
on finding memory leaks in Bro that you can try.  Else if you can
provide a simple test case that reproduces the behavior, filing a
bug/ticket with that info would be the best way to get someone to help
look into it with you.

- Jon

[1] https://www.bro.org/development/howtos/leaks.html