[Bro] ElasticSearch plugin

[Gary Faulkner]

I've actually had instances where it "appeared" that Bro couldn't 
process conn events fast enough and Bro "appeared" to drop events 
further down the pipleline or outright crash under certain high stress 
situations. I suspected it was related to not being able to write the 
conn.log events to disk fast enough as I'd see a huge spike in calls to 
write the conn.log file and subsequent dips in the counters for all of 
the protocol based logs. I was writing the counters from the workers to 
an external stats server whenever a script called its specific write log 
event, so I could see the spikes and correlate them to drops in other 
areas, but I could never definitively prove that disk-IO for log writes 
were the bottle-neck as opposed to some other processing task as some of 
the system monitors on the master side would drop stats when it was too 
busy. Ultimately I largely addressed the causes of the processing spikes 
(DDOS attacks etc) upstream from Bro, but I could see the potential for 
wanting to directly forward events to an external location and not write 
them locally at all instead of trying to scale a single Bro master to 
handle writing hundreds of thousands or more events per second to disk.

~Gary


On 6/13/16 2:21 PM, Joe Blow wrote:
> What if you *hate* using logstash because liblognormalize is much, much
> faster than a regex engine, like logstash?  I'd much prefer to simply get
> the data into rsyslog (which should be trivial), then use rsyslog queueing
> and batching which is much more flexible, IMO.  Use RELP to reliably
> forward, send to kafka, do whatever, but once you've got it into rsyslog
> you've got pretty solid queueing which you can use, along with a whole host
> of output modules.
>
> I'd just love to see a better pure syslog integration, without having to
> write to disks, then read from the disks.  It starts potentially cause
> problems when you're capturing 10Gb/s+.
>
> Cheers,
>
> JB
>
> On Mon, Jun 13, 2016 at 2:45 PM, Gary Faulkner <gfaulkner.nsm at gmail.com>
> wrote:
>
>> I believe supporting dots in field names again is on the fix list for
>> Elastic Stack v5 which is currently in development, so that part might
>> at least get fixed on the Elastic end. Technically I believe that fix is
>> a plus even for folks using another plugin such as Kafka as those folks
>> still potentially had to do something to rewrite the field names. I
>> can't speak for anything else that might be broken in the plugin.
>>
>> Here is the reference to that bit in the Elastic Blog:
>> https://www.elastic.co/blog/elasticsearch-5-0-0-alpha3-released
>>
>> I could still see cases where someone could have a low volume Bro +
>> local ES where it might not be desirable or necessary to run a bunch of
>> other stuff in between Bro and ES, but maybe it just isn't too big a
>> deal then to just let Logstash do the work of reading local log files
>> assuming that one is OK with still writing normal Bro log output in
>> addition to ES. An example might be if I wanted to do something akin to
>> a stand-alone Security Onion node, but with Bro and Elastic instead of
>> Bro and ELSA.
>>
>> I'm not using that functionality currently and will probably look at
>> something like Kafka, but I already have a log volume where the overhead
>> of running something like Kafka probably makes sense.
>>
>> ~Gary
>>
>>
>> On 6/13/16 10:44 AM, Seth Hall wrote:
>>> Is there anyone here relying on the elasticsearch writer plugin in the
>> bro-plugins repository?  It doesn't appear to work with current versions of
>> elasticsearch anymore and it has always had trouble at sites with high
>> rates of logging.
>>> If we don't get much of a response on this we will be deprecating and/or
>> removing the elasticsearch writer.  There should be more reliable
>> mechanisms available soon anyway by either writing to a Kafka server and
>> then forwarding to ElasticSearch or writing files as JSON and the
>> forwarding to ElasticSearch.
>>> Thanks,
>>>     .Seth
>>>
>>> --
>>> Seth Hall
>>> International Computer Science Institute
>>> (Bro) because everyone has a network
>>> http://www.bro.org/
>>>
>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>