[Bro-Dev] Performance Enhancements

[Azoff, Justin S]

> On Oct 9, 2017, at 4:03 PM, Clark, Gilbert <gc355804 at ohio.edu> wrote:
> 
> If you look in one of the subdirectories or another, in ages past there was a little shell script to incrementally execute bro against a specific trace, loading one script at a time to see the effect each of them had on the overall runtime.  I can't remember what it was called offhand, but it was useful for quick and dirty testing.
> 
> And yes, function calls in bro script land are pretty horrific in terms of overhead.  Line per line, bro script in general isn't terribly efficient just because it doesn't do any of the things a modern interpreter might (e.g. just-in-time compilation).  That's not a criticism, it's only a note - most folks rely on horizontal scaling to deal with faster ingress, which I think makes a whole lot of sense.

>From what my little microbenchmarks have been showing me, bro scripts are mostly fast, but there are some operations that may be slower than they should be.. like for loops or set/table lookups on small or empty tables.

> Just in the event it's useful, I've attached some profiling I did on the script function overhead with the crap I wrote: these are some graphs on which script functions call which other script functions, how many times that happens, and how much time is spent in each function.  
> 
> avggraph is the average time spent per-call, and resgraph is the aggregate time spent in each function across the entire run.  The numbers' formatting needed some fixing, but never made it that far ...

Something like this from an hours worth of bro runtime would be neat.  One potential issue is that if a function is called a lot, it could mean it's either something that needs to be optimized so it is faster, or it could mean it's something that needs to be refactored so it's not called so much.

> I know Robin et. al. were working on different approaches for next-generation scripting kind of stuff, but haven't kept up well enough to really know where those are.

http://www.icir.org/hilti/ I believe.

> One thing I played around with was using plugin hooks to integrate other scripting languages into the bro fast path (luajit was my weapon of choice) and seeing if conversion from bro script to one of those other languages might improve the run time.  Other languages would still be less efficient than C, and anything garbage collected would need to be *really* carefully used, but ... it struck me as an idea that might be worth a look :)

I'm not sure how hard it would be to write a transpiler for bro scripts and convert them completely to something like lua.   Other than maybe ip address and subnets as data types, I think they overlap fairly well.

> And yeah, generally speaking, most of the toolkits I've played with for software-based packet processing absolutely do use memory pools for the fast path.  They also use burst fetch tricks (to amortize the cost of fetching packets over X packets, rather than fetching one packet at a time), and I've also seen quite a bit of prefetch / SIMD to try to keep things moving quickly once the packets make it to the CPU.  
> 
> Things start to get pretty crazy as packet rates increase, though: once you hit about 10 - 15 Gbps, even a *cache miss* on a modern system is enough to force a drop ...

Data rate is just one part of it.. the number of packets per second and new sessions per second has a huge impact as well.  Handling 10Gbps of a few concurrent connections is easy, but 10Gbps of DNS will not work so well.

DNS is one analyzer/script that I think could benefit from being ported to C++.  The script doesn't do much and there aren't many scripts out there that want the lower level dns events.. Having the analyzer merge the request/reply directly and bypass the scripts entirely could boost performance by quite a bit.

— 
Justin Azoff