[Bro-Dev] [JIRA] (BIT-1388) Broker's integration in Bro's main/run loop

Jon Siwek (JIRA) jira at bro-tracker.atlassian.net
Mon May 11 08:46:02 PDT 2015

    [ https://bro-tracker.atlassian.net/browse/BIT-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20700#comment-20700 ] 

Jon Siwek commented on BIT-1388:

Just referencing a bro-dev mailing list thread w/ other discussion and ideas regarding libev or libuv integration:


> Broker's integration in Bro's main/run loop
> -------------------------------------------
>                 Key: BIT-1388
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1388
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro, Broker
>            Reporter: Jon Siwek
>             Fix For: 2.5
> * There's a cost to Broker queues being idle.  Whenever Broker gets a chance to process messages, it looks for updates to all connections/message-queues/data-stores.  That involves sending synchronous messages between actors, and for empty queues, it just gets back an empty deque object it needs to destroy.
> * Broker queues integrate into Bro's run loop by exposing a file descriptor that's ready when the queue is non-empty.  Users have the capability of adding arbitrary numbers of queues at run-time (e.g. they can freely add subscriptions to any amount of logs, events, etc.).  Relying on select() may become a bottleneck if someone has hundreds of Broker queues, or could possibly break on some systems if FD_SETSIZE is limited to 1024.
> Ideas on how to fix:
> * Improve Bro's main run loop and dedicate an IOSource to each Broker queue (instead of sharing a single IOSource like they do now).  There might be several things that could be tweaked in the main run loop, but at a minimum, epoll()/kqueue() could alternatively replace select().  Could also think about using something like libev (http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod) to abstract what particular polling backend is used.  Might even be able to use libev's timers to fix how Bro's timers are currently coupled w/ there being an active IOSource consistently driving time forward.
> * Move the draining of Broker queues completely off to their own threads.  This maybe is adding a bit too much complexity (Broker/CAF uses threads for queues, then Bro would use more threads just to talk to those other threads...).  Since CAF becomes a requirement, it may be simpler to start replacing/allowing some areas of Bro's threading to be done w/ CAF actors.  And then if Broker exposed an optional API to talk directly w/ CAF actors, the integration w/ Bro may actually become more straightforward.
> And those ideas don't have to be mutually exclusive.

This message was sent by Atlassian JIRA

More information about the bro-dev mailing list