[Bro] connecting to bro with broccoli

[daniel nagar]

>
> Could you give more information about what events you are sending around?
> Are you receiving events from Bro or sending them to Bro?

I'm capturing HTTP events, only receiving, not sending

What configuration have you done in Bro to send or receive these events?

I've used the default settings, same configurations as in the examples that
come with broccoli

I was sending out many HTTP requests which causes raising of many events
per request/response, capstats showed I was at 2~3 Mbps transfer rate on
the interface but when I checked the transfer rate of events between bro
and my broccoli client I was at 600~700Mbps, the events seem to be too
large, even when using compact events, and my broccoli client ended up
using 100% cpu of the core it was on, maybe enabling parallelism of this
section could give better results at events processing.


I've figured out the memory expansion problem, it seems that the
"ChunkQueue" in "ChunkedIO" does not have a limit and I was sending events
at higher speeds than my broccoli client could process so the queue just
kept growing.
I updated the queue so it will drop chunks when it reaches a certain limit
of chunks in the queue and now the memory stays steady at 1.5GB even at
high speeds of events.

This is a temporary fix in my opinion, a more robust communication
framework is needed such as using an external queue (such as ActiveMQ /
ZeroMQ) for transferring events/chunks.



Daniel.




On Sun, Aug 31, 2014 at 7:13 PM, Seth Hall <seth at icir.org> wrote:

>
> On Aug 27, 2014, at 1:29 PM, daniel nagar <dngr7512 at gmail.com> wrote:
>
> > I'm using bro 2.2 and I connect to bro using broccoli to receive events.
> > I can manage connecting to bro-worker and receive events, not sure if
> it's the correct way to receive event from bro but connecting to the
> manager port didn't retrieve any event whatsoever,
>
> Could you give more information about what events you are sending around?
> Are you receiving events from Bro or sending them to Bro?
>
> What configuration have you done in Bro to send or receive these events?
>
> > the problem is that when I receive events at speeds higher than 2Mbps
> the parent of the bro-worker (not the broccoli application) memory expands
> rapidly and can reach 10Gb in a minute.
>
> Another interesting number might be events per second.  I'm even a little
> unclear what you mean by 2Mbps.  Do you mean that the data rate of your
> connection between your broccoli application and Bro is 2Mbps?
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140831/eb2a0829/attachment.html