<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Howdy:</p>
<p><br>
</p>
<p>Not sure about the content of the BroCon talk ... but a few years back, I did a bit of work on this. <span style="font-size: 12pt;">There was a plugin here:</span></p>
<p><br>
</p>
<p><a href="https://github.com/cubic1271/bro-plugin-instrumentation" class="OWAAutoLink" id="LPlnk922433" previewremoved="true">https://github.com/cubic1271/bro-plugin-instrumentation</a></p>
<p><br>
</p>
<p>that allowed me to profile the execution of various bro scripts and figure out what was eating the most time. It also added a pretty braindead mechanism to expose script variables through a REST interface, which I wrapped in an HTML5 UI to get some real-time
statistics ... though I have no idea where that code went.</p>
<p><br>
</p>
<p>I also threw together this:</p>
<p><br>
</p>
<p><a href="https://github.com/cubic1271/pybrig" class="OWAAutoLink" id="LPlnk782322" previewremoved="true">https://github.com/cubic1271/pybrig</a></p>
<p><br>
</p>
<p>which was intended to benchmark Bro on a specific platform, the idea being to get results that were relatively consistent. It could make some pretty pictures, which was kind of neat ... but I'd probably do things a lot differently if I had it to do over
again :)<br>
<br>
I'll note that one of the challenges with profiling is that there are the bro scripts, and then there is the bro engine. The scripting layer has a completely different set of optimizations that might make sense than the engine does: turning off / turning on
/ tweaking different scripts can have a huge impact on Bro's relative performance depending on the frequency with which those script fragments are executed. Thus, one way to look at speeding things up might be to take a look at the scripts that are run most
often and seeing about ways to accelerate core pieces of them ... possibly by moving pieces of those scripts to builtins (as C methods).<br>
<span style="font-size: 12pt;"><br>
</span></p>
<p><span style="font-size: 12pt;">If I had to guess at one engine-related thing that would've sped things up when I was profiling this stuff back in the day, it'd probably be rebuilding the memory allocation strategy / management. From what I remember, Bro
does do some malloc / free</span><span style="font-size: 12pt;"> in the data path, which hurts quite a bit when one is trying to make things go fast. It also means that the selection of a memory allocat</span><span style="font-size: 12pt;">or</span><span style="font-size: 12pt;">
and </span><span style="font-size: 12pt;">NUMA / per-node memory</span><span style="font-size: 12pt;"> management is going to be importan</span><span style="font-size: 12pt;">t. That's probably not going to qualify as something *small*, though ...</span></p>
<p><span style="font-size: 12pt;"></span></p>
<p><span style="font-size: 12pt;"><br>
</span></p>
<p><span style="font-size: 12pt;">On a related note, a fun experiment is always to try running bro with a different allocator and seeing what happens ...</span></p>
<p></p>
<p><br>
</p>
<p>Another thing that (I found) got me a few percentage points for more-or-less free was profile-guided optimization: I ran bro first with profiling enabled against a representative data set, then rebuilt it against the profile I collected. Of course, your
mileage may vary ...</p>
<p><br>
</p>
<p>Anyway, hope something in there was useful. </p>
<p><br>
</p>
<p><span style="font-size: 12pt;">Cheers,</span></p>
<p><span style="font-size: 12pt;">Gilbert Clark</span><br>
</p>
<p><br>
</p>
<p></p>
<p></p>
<p></p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> bro-dev-bounces@bro.org <bro-dev-bounces@bro.org> on behalf of Jim Mellander <jmellander@lbl.gov><br>
<b>Sent:</b> Thursday, October 5, 2017 3:45:21 PM<br>
<b>To:</b> bro-dev@bro.org<br>
<b>Subject:</b> [Bro-Dev] Performance Enhancements</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Hi all:</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">One item of particular interest to me from Brocon was this tidbit from Packetsled's lightning talk:</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
<p class="MsoNormal">"Optimizing core loops (like net_run() ) with preprocessor branch prediction macros likely() and unlikely() for ~3% speedup. We optimize for maximum load."</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">After conversing with Leo Linsky of Packetsled, I wanted to initiate a conversation about easy performance improvements that may be within fairly easy reach:</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">1. Obviously, branch prediction, as mentioned above. 3% speedup for (almost) free is nothing to sneeze at.</p>
<p class="MsoNormal">2. Profiling bro to identify other hot spots that could benefit from optimization.</p>
<p class="MsoNormal">3. Best practices for compiling Bro (compiler options, etc.)<br>
</p>
<p class="MsoNormal">4. Data structure revisit (hash functions, perhaps?)</p>
<p class="MsoNormal">etc.</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">Perhaps the Bro core team is working on some, all, or a lot more in this area. It might be nice to get the Bro community involved too. Is anyone else interested?</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">Jim Mellander</p>
<p class="MsoNormal">ESNet</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal"><br>
</p>
</div>
</div>
</div>
</body>
</html>