[Bro] troubleshooting bro memory usage?

[aaron gee-clough]

All,

I think I know what's causing this on the surface, but I'm unsure of the 
deeper cause. When I commented out the SecurityOnion bro scripts, bro's 
memory usage was stable and reasonable. So the problem was clearly 
coming from securityonion's scripts. I then started adding the 
SecurityOnion rules back in one by one, adding a ton of Reporter::warn 
statements, and watching the reporter.log. What I noticed was the 
securityonion hostname.bro script never completed *if* the device's 
hostname had a dash in it ("location-onion", for example). When I 
changed the server's hostname to not have a dash, the hostname script 
completed without issue.

I suspect this means that the "hostname" and "interface" variables from 
the securityonion scripts weren't being initialized properly while 
trying to start up with a dashed hostname, doing who-knows-what when bro 
was told to add those variables to every logged event.

Given that, I have an easy fix in the short term, which is to rename the 
box running securityonion to not have a dash in its hostname. What I'm 
confused by is why this would happen in the first place. (So I'm not 
clear yet on what patch to suggest to the securityonion folks to prevent 
this from coming up again.)

The securityonion hostname.bro file does the following:

    module SecurityOnion;

    @load base/frameworks/input

    export {
             ## Event to capture when the hostname is discovered.
             global SecurityOnion::found_hostname: event(hostname: string);

             ## Hostname for this box.
             global hostname = "";

    type HostnameCmdLine: record { s: string; };

    event SecurityOnion::hostname_line(description:
    Input::EventDescription, tpe: Input::Event, s: string)
             {
             hostname = s;
             system(fmt("rm %s", description$source));
             event SecurityOnion::found_hostname(hostname);
             }



    event add_hostname_reader(name: string)
             {
             Input::add_event([$source=name,
                               $name=name,
                               $reader=Input::READER_RAW,
                               $want_record=F,
                               $fields=HostnameCmdLine,
                               $ev=SecurityOnion::hostname_line]);
             }

    event bro_init() &priority=5
             {
             local tmpfile = "/tmp/bro-hostname-" + unique_id("");
             system(fmt("hostname > %s", tmpfile));
             event add_hostname_reader(tmpfile);
             }


The SecurityOnion::hostname_line event never fires if the hostname has a 
dash in it (for example, if the contents of the tmpfile are 
"location-onion"). I see the add_hostname_reader event fire, but not the 
hostname_line event. Do you all have any idea why that would fail if 
there's a string with a dash in the file? Is bro thinking it's an 
expression rather than a string? Two strings?

Thanks for all the help so far. This has been hard to nail down.

aaron


On 08/12/2013 08:55 AM, aaron gee-clough wrote:
>
> On 08/11/2013 09:02 PM, Vlad Grigorescu wrote:
>> On Aug 11, 2013, at 8:39 AM, Aaron Gee-Clough <lists at g-clef.net> wrote:
>>
>>> I have two vanilla
>>> securityonion installs (no custom .bro scripts added, just the ones that
>>> came with securityonion), watching just traffic to two different DNS
>>> resolvers
>> What traffic rate do you see?
> 95th percentile over a week (according to MRTG): Box 1: 34.6 Mbps. Box
> 2: 28Mbps
>
>>> right now one of the worker parent processes (according to
>>> "broctl top") on each securityonion box grows monotonically in RAM usage
>>> until it gets killed by Linux (and is then restarted by broctl's cron job).
>> How much RAM is in the box?
>>
> 16 GB. Both have 6-core 2.2GHz CPUs, also.
>
> Thanks.
>
> aaron
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130813/f6cfd3b9/attachment.html