[Bro] troubleshooting bro memory usage?

[Doug Burks]

Hi Aaron,

There are definitely some issues with the hostname and interface scripts.

My demo at Bro Exchange last week failed due to the hostname script,
even though I put precautions in place which had always worked in the
past.  My hostname did include a hyphen, but I recorded a video later
with the same VM (and same hostname) and everything worked fine:
http://youtu.be/0a2WDyBsxzk?t=2m36s

I'll also mention that all of my production servers have a hyphen in
the hostname and they work fine.

Another thing I noticed in testing a few weeks ago in a VM was that if
the VM had only a single CPU core the scripts were much more likely to
fail.  Increasing to 2 or more CPU cores resulted in much higher
levels of success.  Perhaps resource contention on Bro startup?

Seth, I know you're going to rewrite these scripts for Bro 2.2, but do
you have any ideas for troubleshooting in the meantime?

Thanks!

Doug



On Tue, Aug 13, 2013 at 10:27 AM, aaron gee-clough <lists at g-clef.net> wrote:
> All,
>
> I think I know what's causing this on the surface, but I'm unsure of the
> deeper cause. When I commented out the SecurityOnion bro scripts, bro's
> memory usage was stable and reasonable. So the problem was clearly coming
> from securityonion's scripts. I then started adding the SecurityOnion rules
> back in one by one, adding a ton of Reporter::warn statements, and watching
> the reporter.log. What I noticed was the securityonion hostname.bro script
> never completed *if* the device's hostname had a dash in it
> ("location-onion", for example). When I changed the server's hostname to not
> have a dash, the hostname script completed without issue.
>
> I suspect this means that the "hostname" and "interface" variables from the
> securityonion scripts weren't being initialized properly while trying to
> start up with a dashed hostname, doing who-knows-what when bro was told to
> add those variables to every logged event.
>
> Given that, I have an easy fix in the short term, which is to rename the box
> running securityonion to not have a dash in its hostname. What I'm confused
> by is why this would happen in the first place. (So I'm not clear yet on
> what patch to suggest to the securityonion folks to prevent this from coming
> up again.)
>
> The securityonion hostname.bro file does the following:
>
> module SecurityOnion;
>
> @load base/frameworks/input
>
> export {
>         ## Event to capture when the hostname is discovered.
>         global SecurityOnion::found_hostname: event(hostname: string);
>
>         ## Hostname for this box.
>         global hostname = "";
>
> type HostnameCmdLine: record { s: string; };
>
> event SecurityOnion::hostname_line(description: Input::EventDescription,
> tpe: Input::Event, s: string)
>         {
>         hostname = s;
>         system(fmt("rm %s", description$source));
>         event SecurityOnion::found_hostname(hostname);
>         }
>
>
>
> event add_hostname_reader(name: string)
>         {
>         Input::add_event([$source=name,
>                           $name=name,
>                           $reader=Input::READER_RAW,
>                           $want_record=F,
>                           $fields=HostnameCmdLine,
>                           $ev=SecurityOnion::hostname_line]);
>         }
>
> event bro_init() &priority=5
>         {
>         local tmpfile = "/tmp/bro-hostname-" + unique_id("");
>         system(fmt("hostname > %s", tmpfile));
>         event add_hostname_reader(tmpfile);
>         }
>
>
> The SecurityOnion::hostname_line event never fires if the hostname has a
> dash in it (for example, if the contents of the tmpfile are
> "location-onion"). I see the add_hostname_reader event fire, but not the
> hostname_line event. Do you all have any idea why that would fail if there's
> a string with a dash in the file? Is bro thinking it's an expression rather
> than a string? Two strings?
>
> Thanks for all the help so far. This has been hard to nail down.
>
> aaron
>
>
>
> On 08/12/2013 08:55 AM, aaron gee-clough wrote:
>
> On 08/11/2013 09:02 PM, Vlad Grigorescu wrote:
>
> On Aug 11, 2013, at 8:39 AM, Aaron Gee-Clough <lists at g-clef.net> wrote:
>
> I have two vanilla
> securityonion installs (no custom .bro scripts added, just the ones that
> came with securityonion), watching just traffic to two different DNS
> resolvers
>
> What traffic rate do you see?
>
> 95th percentile over a week (according to MRTG): Box 1: 34.6 Mbps. Box
> 2: 28Mbps
>
> right now one of the worker parent processes (according to
> "broctl top") on each securityonion box grows monotonically in RAM usage
> until it gets killed by Linux (and is then restarted by broctl's cron job).
>
> How much RAM is in the box?
>
> 16 GB. Both have 6-core 2.2GHz CPUs, also.
>
> Thanks.
>
> aaron
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
Doug Burks
http://securityonion.blogspot.com