[Bro] elastic search / bro questions
Joe Blow
blackhole.em at gmail.com
Mon Nov 10 18:19:27 PST 2014
One more thing i wanted to share... In
'bro/share/bro/base/frameworks/logging/writers/elasticsearch.bro' it says:
##! There is one known memory issue. If your elasticsearch server is
##! running slowly and taking too long to return from bulk insert
##! requests, the message queue to the writer thread will continue
##! growing larger and larger giving the appearance of a memory leak.
Interesting to see this queuing graphed out on a box with 96gb of ram....
It ran into swap pretty quickly... :)
[image: Inline image 1]
All in good fun i suppose...
Cheers,
JB
On Mon, Nov 10, 2014 at 8:47 PM, Joe Blow <blackhole.em at gmail.com> wrote:
> So for the record, this is what happens when you configure bro to have a
> log rotate interval of 0 within broctl, and still send logs to
> elasticsearch. Most of the logs will end up in the 'bro' index, but some
> will still end up being sent to bro-$DATETIME index. This was the result
> of some legacy configs (no logrotate for rsyslog so as not to lose file
> handles) which sent the data to a homebrew ES plugin. I had forgotten to
> remove these configs when setting up bro for the more native ES.
>
> Thanks tons for the quick response in the IRC channel.
>
> Cheers,
>
> JB
>
> On Mon, Nov 10, 2014 at 11:48 AM, M K <mkhan04 at gmail.com> wrote:
>
>> Weird... As Seth mentioned, the writer uses the time and the rotation
>> interval to name the indexes. It should also create an @ index for
>> metadata. I thought the time format was hard coded in the es writer, but
>> it's been a while since I read the code ...
>>
>> Also, in regards to ES restart, there are some tunable elements. For one,
>> optimizing indexes should help. Also if you have the bandwidth, you can
>> increase the number of concurrent recoveries and the allowed network
>> throughput.
>> On Nov 10, 2014 11:20 AM, "Joe Blow" <blackhole.em at gmail.com> wrote:
>>
>>> Nope, i invoke bro using broctl like this:
>>>
>>> su snort -c "export https_proxy='https://$PROXY:$PROXYPORT';
>>> /opt/data/bro/bin/broctl restart --clean"
>>>
>>> Which usually shows things like this:
>>>
>>> cleaning up ...
>>> cleaning up nodes ...
>>> checking configurations...
>>> manager scripts are ok.
>>> proxy-0 scripts are ok.
>>> worker-0-1 scripts are ok.
>>> worker-0-2 scripts are ok.
>>> worker-0-3 scripts are ok.
>>> worker-0-4 scripts are ok.
>>> worker-1-1 scripts are ok.
>>> worker-1-2 scripts are ok.
>>> worker-1-3 scripts are ok.
>>> worker-2-1 scripts are ok.
>>> worker-2-2 scripts are ok.
>>> worker-2-3 scripts are ok.
>>> worker-3-1 scripts are ok.
>>> worker-3-10 scripts are ok.
>>> worker-3-11 scripts are ok.
>>> worker-3-12 scripts are ok.
>>> worker-3-2 scripts are ok.
>>> worker-3-3 scripts are ok.
>>> worker-3-4 scripts are ok.
>>> worker-3-5 scripts are ok.
>>> worker-3-6 scripts are ok.
>>> worker-3-7 scripts are ok.
>>> worker-3-8 scripts are ok.
>>> worker-3-9 scripts are ok.
>>> worker-4-1 scripts are ok.
>>> worker-4-2 scripts are ok.
>>> worker-4-3 scripts are ok.
>>> worker-5-1 scripts are ok.
>>> worker-5-2 scripts are ok.
>>> worker-5-3 scripts are ok.
>>> worker-5-4 scripts are ok.
>>> installing ...
>>> removing old policies in
>>> /opt/data/bro/spool/installed-scripts-do-not-touch/site ... done.
>>> removing old policies in
>>> /opt/data/bro/spool/installed-scripts-do-not-touch/auto ... done.
>>> creating policy directories ... done.
>>> installing site policies ... done.
>>> generating cluster-layout.bro ... done.
>>> generating local-networks.bro ... done.
>>> generating broctl-config.bro ... done.
>>> updating nodes ... done.
>>> starting ...
>>> starting manager ...
>>> starting proxy-0 ...
>>> starting worker-0-1 ...
>>> starting worker-0-2 ...
>>> starting worker-0-3 ...
>>> starting worker-0-4 ...
>>> starting worker-1-1 ...
>>> starting worker-1-2 ...
>>> starting worker-1-3 ...
>>> starting worker-2-1 ...
>>> starting worker-2-2 ...
>>> starting worker-2-3 ...
>>> starting worker-3-1 ...
>>> starting worker-3-10 ...
>>> starting worker-3-11 ...
>>> starting worker-3-12 ...
>>> starting worker-3-2 ...
>>> starting worker-3-3 ...
>>> starting worker-3-4 ...
>>> starting worker-3-5 ...
>>> starting worker-3-6 ...
>>> starting worker-3-7 ...
>>> starting worker-3-8 ...
>>> starting worker-3-9 ...
>>> starting worker-4-1 ...
>>> starting worker-4-2 ...
>>> starting worker-4-3 ...
>>> starting worker-5-1 ...
>>> starting worker-5-2 ...
>>> starting worker-5-3 ...
>>> starting worker-5-4 ...
>>>
>>> Our node looks like this:
>>>
>>> [manager]
>>> type=manager
>>> host=$IP
>>> [proxy-0]
>>> type=proxy
>>> host=$IP
>>> [worker-0]
>>> type=worker
>>> host=$IP
>>> interface=eth2
>>> lb_method=pf_ring
>>> lb_procs=4
>>> pin_cpus=0,1,2,3
>>> [worker-1]
>>> type=worker
>>> host=$IP
>>> interface=eth3
>>> lb_method=pf_ring
>>> lb_procs=3
>>> pin_cpus=5,6,7
>>> [worker-2]
>>> type=worker
>>> host=$IP
>>> interface=eth4
>>> lb_method=pf_ring
>>> lb_procs=3
>>> pin_cpus=4,8,9
>>> [worker-3]
>>> type=worker
>>> host=$IP
>>> interface=eth5
>>> lb_method=pf_ring
>>> lb_procs=12
>>> pin_cpus=10,11,12,13,14,15,23,24,25,26,27,28
>>> [worker-4]
>>> type=worker
>>> host=$IP
>>> interface=eth6
>>> lb_method=pf_ring
>>> lb_procs=3
>>> pin_cpus=16,17,18
>>> [worker-5]
>>> type=worker
>>> host=$IP
>>> interface=eth7
>>> lb_method=pf_ring
>>> lb_procs=4
>>> pin_cpus=19,20,21,22
>>>
>>>
>>> Logs-to-elasticsearch.bro has this:
>>>
>>> const rotation_interval = 24hr &redef;
>>>
>>> We add custom country logging doing stuff like this (this is
>>> smtp/savecountry.bro):
>>> <snip>
>>> redef record SMTP::Info += {
>>> orig_cc: string &log &optional;
>>> resp_cc: string &log &optional;
>>> };
>>>
>>> event smtp_reply(c: connection, is_orig: bool, code: count, cmd: string,
>>> msg: string, cont_resp: bool) &priority=3
>>> {
>>> local orig_loc =
>>> lookup_location(c$id$orig_h);
>>> if ( orig_loc?$country_code )
>>> c$smtp$orig_cc = orig_loc$country_code;
>>> local resp_loc = lookup_location(c$id$resp_h);
>>> if ( resp_loc?$country_code )
>>> c$smtp$resp_cc = resp_loc$country_code;
>>>
>>> </snip>
>>>
>>> This shouldn't need to have the redef for log rotation should it? The
>>> only non stock stuff we do is adding countries to conn and smtp.
>>> Everything else should be stock.
>>>
>>> Any ideas?
>>>
>>> Cheers,
>>>
>>> JB
>>>
>>>
>>>
>>> On Mon, Nov 10, 2014 at 10:57 AM, Seth Hall <seth at icir.org> wrote:
>>>
>>>>
>>>> > On Nov 10, 2014, at 10:20 AM, Joe Blow <blackhole.em at gmail.com>
>>>> wrote:
>>>> >
>>>> > I'm not processing offline files, if that's what you mean (still a
>>>> bit new to bro, feel free to expand on the tracefiles).
>>>>
>>>> Ohh, I know what's happening. You're running Bro directly at the
>>>> command line without using broctl aren't you? Bro doesn't have log
>>>> rotation enabled by default and the index name rotation is based on log log
>>>> rotation.
>>>>
>>>> Set this in a script you're loading...
>>>>
>>>> redef Log::default_rotation_interval = 1hr;
>>>>
>>>> I haven't double checked and I not sure what that will do to the Ascii
>>>> logs, but it should at least give you partitioned index names in ES.
>>>>
>>>> .Seth
>>>>
>>>> --
>>>> Seth Hall
>>>> International Computer Science Institute
>>>> (Bro) because everyone has a network
>>>> http://www.bro.org/
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141110/b08ea457/attachment.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 62031 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141110/b08ea457/attachment.bin
More information about the Bro
mailing list