[Bro] elastic search / bro questions

Mon Nov 10 17:47:44 PST 2014

So for the record, this is what happens when you configure bro to have a
log rotate interval of 0 within broctl, and still send logs to
elasticsearch.  Most of the logs will end up in the 'bro' index, but some
will still end up being sent to bro-$DATETIME index.  This was the result
of some legacy configs (no logrotate for rsyslog so as not to lose file
handles) which sent the data to a homebrew ES plugin.  I had forgotten to
remove these configs when setting up bro for the more native ES.

Thanks tons for the quick response in the IRC channel.

Cheers,

JB

On Mon, Nov 10, 2014 at 11:48 AM, M K <mkhan04 at gmail.com> wrote:

> Weird... As Seth mentioned, the writer uses the time and the rotation
> interval to name the indexes. It should also create an @ index for
> metadata. I thought the time format was hard coded in the es writer, but
> it's been a while since I read the code ...
>
> Also, in regards to ES restart, there are some tunable elements. For one,
> optimizing indexes should help. Also if you have the bandwidth, you can
> increase the number of concurrent recoveries and the allowed network
> throughput.
> On Nov 10, 2014 11:20 AM, "Joe Blow" <blackhole.em at gmail.com> wrote:
>
>> Nope, i invoke bro using broctl like this:
>>
>> su snort -c "export https_proxy='https://$PROXY:$PROXYPORT';
>> /opt/data/bro/bin/broctl restart --clean"
>>
>> Which usually shows things like this:
>>
>> cleaning up ...
>> cleaning up nodes ...
>> checking configurations...
>> manager scripts are ok.
>> proxy-0 scripts are ok.
>> worker-0-1 scripts are ok.
>> worker-0-2 scripts are ok.
>> worker-0-3 scripts are ok.
>> worker-0-4 scripts are ok.
>> worker-1-1 scripts are ok.
>> worker-1-2 scripts are ok.
>> worker-1-3 scripts are ok.
>> worker-2-1 scripts are ok.
>> worker-2-2 scripts are ok.
>> worker-2-3 scripts are ok.
>> worker-3-1 scripts are ok.
>> worker-3-10 scripts are ok.
>> worker-3-11 scripts are ok.
>> worker-3-12 scripts are ok.
>> worker-3-2 scripts are ok.
>> worker-3-3 scripts are ok.
>> worker-3-4 scripts are ok.
>> worker-3-5 scripts are ok.
>> worker-3-6 scripts are ok.
>> worker-3-7 scripts are ok.
>> worker-3-8 scripts are ok.
>> worker-3-9 scripts are ok.
>> worker-4-1 scripts are ok.
>> worker-4-2 scripts are ok.
>> worker-4-3 scripts are ok.
>> worker-5-1 scripts are ok.
>> worker-5-2 scripts are ok.
>> worker-5-3 scripts are ok.
>> worker-5-4 scripts are ok.
>> installing ...
>> removing old policies in
>> /opt/data/bro/spool/installed-scripts-do-not-touch/site ... done.
>> removing old policies in
>> /opt/data/bro/spool/installed-scripts-do-not-touch/auto ... done.
>> creating policy directories ... done.
>> installing site policies ... done.
>> generating cluster-layout.bro ... done.
>> generating local-networks.bro ... done.
>> generating broctl-config.bro ... done.
>> updating nodes ... done.
>> starting ...
>> starting manager ...
>> starting proxy-0 ...
>> starting worker-0-1 ...
>> starting worker-0-2 ...
>> starting worker-0-3 ...
>> starting worker-0-4 ...
>> starting worker-1-1 ...
>> starting worker-1-2 ...
>> starting worker-1-3 ...
>> starting worker-2-1 ...
>> starting worker-2-2 ...
>> starting worker-2-3 ...
>> starting worker-3-1 ...
>> starting worker-3-10 ...
>> starting worker-3-11 ...
>> starting worker-3-12 ...
>> starting worker-3-2 ...
>> starting worker-3-3 ...
>> starting worker-3-4 ...
>> starting worker-3-5 ...
>> starting worker-3-6 ...
>> starting worker-3-7 ...
>> starting worker-3-8 ...
>> starting worker-3-9 ...
>> starting worker-4-1 ...
>> starting worker-4-2 ...
>> starting worker-4-3 ...
>> starting worker-5-1 ...
>> starting worker-5-2 ...
>> starting worker-5-3 ...
>> starting worker-5-4 ...
>>
>> Our node looks like this:
>>
>> [manager]
>> type=manager
>> host=$IP
>> [proxy-0]
>> type=proxy
>> host=$IP
>> [worker-0]
>> type=worker
>> host=$IP
>> interface=eth2
>> lb_method=pf_ring
>> lb_procs=4
>> pin_cpus=0,1,2,3
>> [worker-1]
>> type=worker
>> host=$IP
>> interface=eth3
>> lb_method=pf_ring
>> lb_procs=3
>> pin_cpus=5,6,7
>> [worker-2]
>> type=worker
>> host=$IP
>> interface=eth4
>> lb_method=pf_ring
>> lb_procs=3
>> pin_cpus=4,8,9
>> [worker-3]
>> type=worker
>> host=$IP
>> interface=eth5
>> lb_method=pf_ring
>> lb_procs=12
>> pin_cpus=10,11,12,13,14,15,23,24,25,26,27,28
>> [worker-4]
>> type=worker
>> host=$IP
>> interface=eth6
>> lb_method=pf_ring
>> lb_procs=3
>> pin_cpus=16,17,18
>> [worker-5]
>> type=worker
>> host=$IP
>> interface=eth7
>> lb_method=pf_ring
>> lb_procs=4
>> pin_cpus=19,20,21,22
>>
>>
>> Logs-to-elasticsearch.bro has this:
>>
>> const rotation_interval = 24hr &redef;
>>
>> We add custom country logging doing stuff like this (this is
>> smtp/savecountry.bro):
>> <snip>
>> redef record SMTP::Info += {
>>         orig_cc: string &log &optional;
>>         resp_cc: string &log &optional;
>>                 };
>>
>> event smtp_reply(c: connection, is_orig: bool, code: count, cmd: string,
>>                  msg: string, cont_resp: bool) &priority=3
>>         {
>>                                 local orig_loc =
>> lookup_location(c$id$orig_h);
>>                 if ( orig_loc?$country_code )
>>                     c$smtp$orig_cc = orig_loc$country_code;
>>                 local resp_loc = lookup_location(c$id$resp_h);
>>                 if ( resp_loc?$country_code )
>>                     c$smtp$resp_cc = resp_loc$country_code;
>>
>> </snip>
>>
>> This shouldn't need to have the redef for log rotation should it?  The
>> only non stock stuff we do is adding countries to conn and smtp.
>> Everything else should be stock.
>>
>> Any ideas?
>>
>> Cheers,
>>
>> JB
>>
>>
>>
>> On Mon, Nov 10, 2014 at 10:57 AM, Seth Hall <seth at icir.org> wrote:
>>
>>>
>>> > On Nov 10, 2014, at 10:20 AM, Joe Blow <blackhole.em at gmail.com> wrote:
>>> >
>>> > I'm not processing offline files, if that's what you mean (still a bit
>>> new to bro, feel free to expand on the tracefiles).
>>>
>>> Ohh, I know what's happening.  You're running Bro directly at the
>>> command line without using broctl aren't you?  Bro doesn't have log
>>> rotation enabled by default and the index name rotation is based on log log
>>> rotation.
>>>
>>> Set this in a script you're loading...
>>>
>>> redef Log::default_rotation_interval = 1hr;
>>>
>>> I haven't double checked and I not sure what that will do to the Ascii
>>> logs, but it should at least give you partitioned index names in ES.
>>>
>>>   .Seth
>>>
>>> --
>>> Seth Hall
>>> International Computer Science Institute
>>> (Bro) because everyone has a network
>>> http://www.bro.org/
>>>
>>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141110/4a106125/attachment.html 