[Bro] A more parallel Bro

[Robin Sommer]

On Wed, Feb 25, 2009 at 14:26 -0600, dalbrech at illinois.edu wrote:

> I've been doing some performance profiling on Bro. 

Interesting! Do you have any results you could share?

> I'm wondering what the ICSI folks' position is on threads vs. clustering.

In short: we consider them orthogonal and are pursuing both. :-)

To elaborate a bit: The cluster is the approach that can provide
significantly increased performance right now; ignoring a few
implementation glitches we still need to sort out, it's working and
ready to use[1]. The cluster is also an approach that will work
long-term: there will always be limits to what a single box can do
(multi-core or not) and the cluster offers the capability to go
beyond that. 

That said, we are in parallel (no pun intended :) working on a
multi-threaded Bro implementation. While we have already made quite
a bit of progress on that, that will however still take a bit to get
into any kind of usable state; we need to restructure Bro internally
quite a bit to exploit its concurrency potential. Once finished
though, we expect Bro's performance to scale pretty well with the
number of available cores (not considering other aspects of the
hardware which might turn into bottlenecks at some point). The
Sarnoff paper has some of the basic ideas for the multi-threaded
Bro, and there'll be an extended journal version of that coming out
soon with more details on the parallel execution model (let me know
if you'd to like to get a draft). 

Does that answer your question? If you have any particular thoughts
on either clustering or the multi-threaded Bro, I'd be happy to hear
them. 

Robin

-- 
Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org 
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org