[Bro] local.bro causing memory leak

[Jason Holmes]

We're streaming JSON versions of Bro logs into Splunk without an issue. 
Some pointers that may help:

1. Set your initCrcLength to something like 2048 in your monitor 
statement in your inputs.conf for Bro logs.  The default is 256 bytes, 
which can be too small to extend past the headers at the beginning of a 
Bro log for some log types.  If you don't do something like this, Splunk 
will get confused when logs rotate because it will find a log with a 
different name having the same CRC.  This could be why you're having 
issues with file renames on log rotation.

2. If you rotate your logs off to some other server for long term 
storage, keep a day or three local as well and have Splunk monitor those 
directories as well.  If you have the initCrcLength set, Splunk is smart 
enough to recognize that conn.log and conn-datestamp.log are the same 
thing if they have the same initCrcLength and won't reindex the rotated 
log.  On the other hand, if Splunk was down or had a log queued for 
batch processing and didn't get it before it was rotated, it'll pick it 
up from the archive directory.

We accomplish this by rotating to an archive directory on the same 
partition on the Bro manager.  That makes the rotate time almost nothing 
since the move is essentially a rename rather than moving all of those 
bytes of logs.  We then use a cron job with rsync to copy the files over 
to long term storage.  Another cron job removes files that are too old.

Example monitor statements:

[monitor:///path/to/your/bro/spool/manager/]
disabled = 0
sourcetype = json_bro
index = your_bro_index
initCrcLength = 2048
whitelist = (dns|notice|weird)_json.*\.log$

[monitor:///path/to/your/bro/spool/archive/20*/]
disabled = 0
sourcetype = json_bro
index = your_bro_index
initCrcLength = 2048
whitelist = (dns|notice|weird)_json.*\.log$

3. If you're moving a massive amount of Bro logs and are regularly 
falling behind, try a heavy forwarder rather than a universal forwarder 
and bump the number of parallelIngestionPipelines in your server.conf 
for your Bro node up.

Thanks,

--
Jason Holmes

On 3/20/18 4:11 PM, Benjamin Wood wrote:
> Thanks Seth.
> 
> The whole problem I'm trying to solve is steaming data into splunk. 
> Splunk forwarder's don't like it when filenames change, and the 
> artificial delay created by rotating logs adds too much latency. The 
> solution that was proposed was "don't rotate logs", and leave them in 
> place long enough for the forwarders to finish.
> 
> At this point I've got to step back and ask, "Am I doing it wrong?" This 
> problem has to have been solved by others. I'm certain there is a way to 
> stream my data to splunk that is better than this.
> 
> The file rotation and renaming functions give me enough to play with to 
> solve the problem using bro-script.
> 
> Thanks again for the feedback,
> Ben
> 
> On Tue, Mar 20, 2018 at 2:50 PM, Seth Hall <seth at corelight.com 
> <mailto:seth at corelight.com>> wrote:
> 
>     __
> 
>     On 19 Mar 2018, at 15:31, Benjamin Wood wrote:
> 
>         I'm running my cluster with broctl, and rotation is turned off
>         because I'm
>         naming files with a timestamp to begin with.
> 
>     Justin got your problem right. If you turn off file rotation, then
>     Bro is never closing any of these hourly logs. You have to be really
>     careful with how you use $path_func because you can easily get
>     yourself into hot water.
> 
>     Alternately you need to define a rotation interval and post
>     processor. Something like this...
> 
>     <i also trimmed out some of your code>
> 
>     |function my_log_post_processor(info: Log::RotationInfo): bool {
>     local ext = sub(info$fname,
>     /^[^\-]+-[0-9]+-[0-9]+-[0-9]+_[0-9]+\.[0-9]+\.[0-9]+\./, ""); # Move
>     file to name including both opening and closing time. local dst =
>     fmt("%s_%s_%s-%s.%s", info$path, strftime("%Y%m%d", info$open),
>     strftime("%H:%M:%S", info$open), strftime("%H:%M:%S%z", info$close),
>     ext); local cmd = fmt("/bin/mv %s %s/%s", info$fname, "/data/logs",
>     dst); system(cmd); return T; } event bro_init() { for ( id in
>     Log::active_streams ) { local filter = Log::get_filter(id,
>     "default"); filter$interv = 1hr; filter$postprocessor =
>     my_log_post_processor; Log::add_filter(id, filter); } } |
> 
>     Something like that will enable you to turn off log rotation in
>     broctl (but you'll lose some broctl niceties as well).
> 
>     .Seth
> 
>     --
>     Seth Hall * Corelight, Inc * www.corelight.com
>     <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.corelight.com&data=02%7C01%7Cjwh128%40psu.edu%7C791424b7915646c6fdd408d58e9fea7d%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636571739851708847&sdata=3FMDF6dwHwzKkUtdK9WT0fn4W1x37NjZb9YyZIafm3Q%3D&reserved=0>
> 
> 
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.ICSI.Berkeley.EDU%2Fmailman%2Flistinfo%2Fbro&data=02%7C01%7Cjwh128%40psu.edu%7C791424b7915646c6fdd408d58e9fea7d%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636571739851708847&sdata=WPuthzDJXbZFwNUS5fSKYTqHHnNkuk7IqD1dsZFZoP4%3D&reserved=0
>