[Bro] elastic search / bro questions

[Joe Blow]

Nope, i invoke bro using broctl like this:

su snort -c "export https_proxy='https://$PROXY:$PROXYPORT';
/opt/data/bro/bin/broctl restart --clean"

Which usually shows things like this:

cleaning up ...
cleaning up nodes ...
checking configurations...
manager scripts are ok.
proxy-0 scripts are ok.
worker-0-1 scripts are ok.
worker-0-2 scripts are ok.
worker-0-3 scripts are ok.
worker-0-4 scripts are ok.
worker-1-1 scripts are ok.
worker-1-2 scripts are ok.
worker-1-3 scripts are ok.
worker-2-1 scripts are ok.
worker-2-2 scripts are ok.
worker-2-3 scripts are ok.
worker-3-1 scripts are ok.
worker-3-10 scripts are ok.
worker-3-11 scripts are ok.
worker-3-12 scripts are ok.
worker-3-2 scripts are ok.
worker-3-3 scripts are ok.
worker-3-4 scripts are ok.
worker-3-5 scripts are ok.
worker-3-6 scripts are ok.
worker-3-7 scripts are ok.
worker-3-8 scripts are ok.
worker-3-9 scripts are ok.
worker-4-1 scripts are ok.
worker-4-2 scripts are ok.
worker-4-3 scripts are ok.
worker-5-1 scripts are ok.
worker-5-2 scripts are ok.
worker-5-3 scripts are ok.
worker-5-4 scripts are ok.
installing ...
removing old policies in
/opt/data/bro/spool/installed-scripts-do-not-touch/site ... done.
removing old policies in
/opt/data/bro/spool/installed-scripts-do-not-touch/auto ... done.
creating policy directories ... done.
installing site policies ... done.
generating cluster-layout.bro ... done.
generating local-networks.bro ... done.
generating broctl-config.bro ... done.
updating nodes ... done.
starting ...
starting manager ...
starting proxy-0 ...
starting worker-0-1 ...
starting worker-0-2 ...
starting worker-0-3 ...
starting worker-0-4 ...
starting worker-1-1 ...
starting worker-1-2 ...
starting worker-1-3 ...
starting worker-2-1 ...
starting worker-2-2 ...
starting worker-2-3 ...
starting worker-3-1 ...
starting worker-3-10 ...
starting worker-3-11 ...
starting worker-3-12 ...
starting worker-3-2 ...
starting worker-3-3 ...
starting worker-3-4 ...
starting worker-3-5 ...
starting worker-3-6 ...
starting worker-3-7 ...
starting worker-3-8 ...
starting worker-3-9 ...
starting worker-4-1 ...
starting worker-4-2 ...
starting worker-4-3 ...
starting worker-5-1 ...
starting worker-5-2 ...
starting worker-5-3 ...
starting worker-5-4 ...

Our node looks like this:

[manager]
type=manager
host=$IP
[proxy-0]
type=proxy
host=$IP
[worker-0]
type=worker
host=$IP
interface=eth2
lb_method=pf_ring
lb_procs=4
pin_cpus=0,1,2,3
[worker-1]
type=worker
host=$IP
interface=eth3
lb_method=pf_ring
lb_procs=3
pin_cpus=5,6,7
[worker-2]
type=worker
host=$IP
interface=eth4
lb_method=pf_ring
lb_procs=3
pin_cpus=4,8,9
[worker-3]
type=worker
host=$IP
interface=eth5
lb_method=pf_ring
lb_procs=12
pin_cpus=10,11,12,13,14,15,23,24,25,26,27,28
[worker-4]
type=worker
host=$IP
interface=eth6
lb_method=pf_ring
lb_procs=3
pin_cpus=16,17,18
[worker-5]
type=worker
host=$IP
interface=eth7
lb_method=pf_ring
lb_procs=4
pin_cpus=19,20,21,22


Logs-to-elasticsearch.bro has this:

const rotation_interval = 24hr &redef;

We add custom country logging doing stuff like this (this is
smtp/savecountry.bro):
<snip>
redef record SMTP::Info += {
        orig_cc: string &log &optional;
        resp_cc: string &log &optional;
                };

event smtp_reply(c: connection, is_orig: bool, code: count, cmd: string,
                 msg: string, cont_resp: bool) &priority=3
        {
                                local orig_loc =
lookup_location(c$id$orig_h);
                if ( orig_loc?$country_code )
                    c$smtp$orig_cc = orig_loc$country_code;
                local resp_loc = lookup_location(c$id$resp_h);
                if ( resp_loc?$country_code )
                    c$smtp$resp_cc = resp_loc$country_code;

</snip>

This shouldn't need to have the redef for log rotation should it?  The only
non stock stuff we do is adding countries to conn and smtp.  Everything
else should be stock.

Any ideas?

Cheers,

JB



On Mon, Nov 10, 2014 at 10:57 AM, Seth Hall <seth at icir.org> wrote:

>
> > On Nov 10, 2014, at 10:20 AM, Joe Blow <blackhole.em at gmail.com> wrote:
> >
> > I'm not processing offline files, if that's what you mean (still a bit
> new to bro, feel free to expand on the tracefiles).
>
> Ohh, I know what's happening.  You're running Bro directly at the command
> line without using broctl aren't you?  Bro doesn't have log rotation
> enabled by default and the index name rotation is based on log log rotation.
>
> Set this in a script you're loading...
>
> redef Log::default_rotation_interval = 1hr;
>
> I haven't double checked and I not sure what that will do to the Ascii
> logs, but it should at least give you partitioned index names in ES.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141110/4a7eef4f/attachment.html