[Bro] elastic search / bro questions

Mon Nov 10 08:48:24 PST 2014

Weird... As Seth mentioned, the writer uses the time and the rotation
interval to name the indexes. It should also create an @ index for
metadata. I thought the time format was hard coded in the es writer, but
it's been a while since I read the code ...

Also, in regards to ES restart, there are some tunable elements. For one,
optimizing indexes should help. Also if you have the bandwidth, you can
increase the number of concurrent recoveries and the allowed network
throughput.
On Nov 10, 2014 11:20 AM, "Joe Blow" <blackhole.em at gmail.com> wrote:

> Nope, i invoke bro using broctl like this:
>
> su snort -c "export https_proxy='https://$PROXY:$PROXYPORT';
> /opt/data/bro/bin/broctl restart --clean"
>
> Which usually shows things like this:
>
> cleaning up ...
> cleaning up nodes ...
> checking configurations...
> manager scripts are ok.
> proxy-0 scripts are ok.
> worker-0-1 scripts are ok.
> worker-0-2 scripts are ok.
> worker-0-3 scripts are ok.
> worker-0-4 scripts are ok.
> worker-1-1 scripts are ok.
> worker-1-2 scripts are ok.
> worker-1-3 scripts are ok.
> worker-2-1 scripts are ok.
> worker-2-2 scripts are ok.
> worker-2-3 scripts are ok.
> worker-3-1 scripts are ok.
> worker-3-10 scripts are ok.
> worker-3-11 scripts are ok.
> worker-3-12 scripts are ok.
> worker-3-2 scripts are ok.
> worker-3-3 scripts are ok.
> worker-3-4 scripts are ok.
> worker-3-5 scripts are ok.
> worker-3-6 scripts are ok.
> worker-3-7 scripts are ok.
> worker-3-8 scripts are ok.
> worker-3-9 scripts are ok.
> worker-4-1 scripts are ok.
> worker-4-2 scripts are ok.
> worker-4-3 scripts are ok.
> worker-5-1 scripts are ok.
> worker-5-2 scripts are ok.
> worker-5-3 scripts are ok.
> worker-5-4 scripts are ok.
> installing ...
> removing old policies in
> /opt/data/bro/spool/installed-scripts-do-not-touch/site ... done.
> removing old policies in
> /opt/data/bro/spool/installed-scripts-do-not-touch/auto ... done.
> creating policy directories ... done.
> installing site policies ... done.
> generating cluster-layout.bro ... done.
> generating local-networks.bro ... done.
> generating broctl-config.bro ... done.
> updating nodes ... done.
> starting ...
> starting manager ...
> starting proxy-0 ...
> starting worker-0-1 ...
> starting worker-0-2 ...
> starting worker-0-3 ...
> starting worker-0-4 ...
> starting worker-1-1 ...
> starting worker-1-2 ...
> starting worker-1-3 ...
> starting worker-2-1 ...
> starting worker-2-2 ...
> starting worker-2-3 ...
> starting worker-3-1 ...
> starting worker-3-10 ...
> starting worker-3-11 ...
> starting worker-3-12 ...
> starting worker-3-2 ...
> starting worker-3-3 ...
> starting worker-3-4 ...
> starting worker-3-5 ...
> starting worker-3-6 ...
> starting worker-3-7 ...
> starting worker-3-8 ...
> starting worker-3-9 ...
> starting worker-4-1 ...
> starting worker-4-2 ...
> starting worker-4-3 ...
> starting worker-5-1 ...
> starting worker-5-2 ...
> starting worker-5-3 ...
> starting worker-5-4 ...
>
> Our node looks like this:
>
> [manager]
> type=manager
> host=$IP
> [proxy-0]
> type=proxy
> host=$IP
> [worker-0]
> type=worker
> host=$IP
> interface=eth2
> lb_method=pf_ring
> lb_procs=4
> pin_cpus=0,1,2,3
> [worker-1]
> type=worker
> host=$IP
> interface=eth3
> lb_method=pf_ring
> lb_procs=3
> pin_cpus=5,6,7
> [worker-2]
> type=worker
> host=$IP
> interface=eth4
> lb_method=pf_ring
> lb_procs=3
> pin_cpus=4,8,9
> [worker-3]
> type=worker
> host=$IP
> interface=eth5
> lb_method=pf_ring
> lb_procs=12
> pin_cpus=10,11,12,13,14,15,23,24,25,26,27,28
> [worker-4]
> type=worker
> host=$IP
> interface=eth6
> lb_method=pf_ring
> lb_procs=3
> pin_cpus=16,17,18
> [worker-5]
> type=worker
> host=$IP
> interface=eth7
> lb_method=pf_ring
> lb_procs=4
> pin_cpus=19,20,21,22
>
>
> Logs-to-elasticsearch.bro has this:
>
> const rotation_interval = 24hr &redef;
>
> We add custom country logging doing stuff like this (this is
> smtp/savecountry.bro):
> <snip>
> redef record SMTP::Info += {
>         orig_cc: string &log &optional;
>         resp_cc: string &log &optional;
>                 };
>
> event smtp_reply(c: connection, is_orig: bool, code: count, cmd: string,
>                  msg: string, cont_resp: bool) &priority=3
>         {
>                                 local orig_loc =
> lookup_location(c$id$orig_h);
>                 if ( orig_loc?$country_code )
>                     c$smtp$orig_cc = orig_loc$country_code;
>                 local resp_loc = lookup_location(c$id$resp_h);
>                 if ( resp_loc?$country_code )
>                     c$smtp$resp_cc = resp_loc$country_code;
>
> </snip>
>
> This shouldn't need to have the redef for log rotation should it?  The
> only non stock stuff we do is adding countries to conn and smtp.
> Everything else should be stock.
>
> Any ideas?
>
> Cheers,
>
> JB
>
>
>
> On Mon, Nov 10, 2014 at 10:57 AM, Seth Hall <seth at icir.org> wrote:
>
>>
>> > On Nov 10, 2014, at 10:20 AM, Joe Blow <blackhole.em at gmail.com> wrote:
>> >
>> > I'm not processing offline files, if that's what you mean (still a bit
>> new to bro, feel free to expand on the tracefiles).
>>
>> Ohh, I know what's happening.  You're running Bro directly at the command
>> line without using broctl aren't you?  Bro doesn't have log rotation
>> enabled by default and the index name rotation is based on log log rotation.
>>
>> Set this in a script you're loading...
>>
>> redef Log::default_rotation_interval = 1hr;
>>
>> I haven't double checked and I not sure what that will do to the Ascii
>> logs, but it should at least give you partitioned index names in ES.
>>
>>   .Seth
>>
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro.org/
>>
>>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141110/e3063487/attachment.html 