<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<meta content="text/html; charset=UTF-8">
<style type="text/css" style="">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>$0.02 USD:</p>
<br>
<p>As I recall, Bro's per-packet processing overhead can vary significantly as a result of timers and triggers that execute on a situational basis. Also, relative overhead of packet ingest is going to vary based on the set of loaded scripts in addition to
the specific trace used to run the tests. That's not trying to argue that these results are not useful / interesting, but instead *only* that the specific percentages might not be representative of the general case (just because I'm convinced that there really
is not a general case to objectively measure).</p>
<br>
<p>Also ... if the overhead of the polling / ingest itself turns out to be a huge problem at high rates, one idea would be to separate that and pass packets (in bulk) through a ring / high-speed IPC to the process that needs to ingest them. That's worked pretty
well for me in DPDK, and has the benefit of being able to distribute packets from one ingest to multiple processors (which is something I've had to do for process-heavy workloads ... which I would argue is something that Bro tends to be).<br>
</p>
<br>
<p>Along those lines, rather than spending much time on packet ingest mechanics in bro (or pieces thereof), one idea might be to instead focus on integrating packet bricks as a standard ingest / distribution mechanic for everything packet-related in the general
case. The idea would be that fetching packets from bro (and its related processes) would become less about calls to epoll and select, and more about high-speed IPC that went out of its way to avoid kernel-space entirely. The nice thing about that is that
it'd be a little easier to standardize on the bro side of things, and would take a step toward separating bro as a scripting / event engine from bro as a (relative) monolith.<br>
</p>
<p><br>
</p>
<p>Of course, the down side is that packet bricks could add some serious (mandatory) complexity to bro, so maybe it's not the right answer ... but maybe a more lightweight, specialized distribution channel might be doable, or maybe there would be a way to embed
packet bricks inside of an application in the event that folks didn't want to run the two separately, or ... etc.<br>
</p>
<p><br>
</p>
<p>As always, just for what it's worth :)</p>
<br>
<p>-Gilbert</p>
<p><br>
</p>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> bro-dev-bounces@bro.org <bro-dev-bounces@bro.org> on behalf of Siwek, Jon <jsiwek@illinois.edu><br>
<b>Sent:</b> Tuesday, April 11, 2017 8:41:23 PM<br>
<b>To:</b> <bro-dev@bro.org><br>
<b>Subject:</b> [Bro-Dev] early performance comparisons of CAF-based run loop</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">I recently got a minimal CAF-based run loop for Bro working, did crude performance comparisons, and wanted to share.<br>
<br>
The approach was to measure average time between calls of net_packet_dispatch() and also the average time it takes to analyze a packet. The former attempts to measure the overhead imposed by the loop implementation and the later just gives an idea of how significant
a chunk of time that is in relation to Bro’s main workload. I found that the overhead of the loop can be ~5-10% of the packet processing time, so it does seem worthwhile to try and keep the run loop overhead low.<br>
<br>
Initial testing of the CAF-based loop showed the overhead increased by ~1.8x, but there was still a major difference in the implementations: the standard Bro loop only invokes its IOSource polling mechanism (select) once every 25 cycles of the loop, while the
CAF implementation’s polling mechanism (actor/thread scheduling + messaging + epoll) is used for every cycle/packet. As one would expect, by just trivially spinning the main process() function in a loop for 25 iterations, the overhead of the CAF-based loop
comes back into line with the standard run loop.<br>
<br>
To try and better measure the actual differences related to the polling mechanism implementation, I quickly hacked Bro’s standard runloop to select() on every packet instead of once every 25th and found that the overhead measures +/- 10% within the 1.8x overhead
increase of the initial CAF-based loop. So is the cost of the extra system call for epoll/select per packet the main thing to avoid? Sort of. I again hacked Bro’s standard loop to be able to use either epoll or poll instead of select and found that those
do better, with the overhead increase being about 1.3x (still doing one “poll” per packet) in relation to the standard run loop. Meaning there is some measurable trend in polling mechanism performance (for sparse # of FDs/sources): poll comes in first, epoll
second, with CAF and select about tied for third.<br>
<br>
Takeaways:<br>
<br>
(1) Regardless of runloop implementation or polling mechanism choices, performing the polling operation once per packet should probably be avoided. In concept, it’s an easy way to get a 2-5% speedup in relation to total packet processing time.<br>
<br>
(2) Related to (1), but not in the sense of performance, is that even w/ a CAF-based loop it still seems somewhat difficult to reason about the reality of how IOSources are prioritized. In the standard loop, the priority of an IOSource is a combination of
its “idle” state, the polling frequency, and a timestamp, which it often chooses arbitrarily as the “time of last packet”, just so that it gets processed with higher priority than subsequent packets. Maybe the topic of making IOSource prioritization more
explicit/well-defined could be another thread of discussion, but my initial thought is that the whole IOSource abstraction may be over-generalized and maybe not even needed.<br>
<br>
(3) The performance overhead of a CAF-based loop doesn’t seem like a showstopper for proceeding with it as a choice for replacing the current loop. It’s not significantly worse than the current loop (provided we still throttle the polling ratio when packet
sources are saturated), and even using the most minimal loop implementation of just poll() would only be about a 1% speedup in relation to the total packet processing workload.<br>
<br>
Just raw data below, for those interested:<br>
<br>
I tested against the pcaps from <a href="http://tcpreplay.appneta.com/wiki/captures.html">
http://tcpreplay.appneta.com/wiki/captures.html</a><br>
(I was initially going to use tcpreplay to test performance against a live interface, but decided reading from a file is easier and just as good for what I wanted to measure).<br>
Numbers are measured in “ticks”, which are equivalent to nanoseconds on the test system.<br>
Bro and CAF are both compiled w/ optimizations.<br>
<br>
bigFlows.pcap, 1 “poll" per packet<br>
--------------------------<br>
poll<br>
('avg overhead', 1018.8868239999998)<br>
('avg process', 11664.4968147)<br>
<br>
epoll<br>
('avg overhead', 1114.2168096999999)<br>
('avg process', 11680.6078816)<br>
<br>
CAF<br>
('avg overhead', 1515.9933343999996)<br>
('avg process', 11914.897109200003)<br>
<br>
select<br>
('avg overhead', 1792.8142910999995)<br>
('avg process', 11863.308550400001)<br>
<br>
bigFlows.pcap, Polling Throttled to 1 per 25 packets<br>
---------------------------<br>
poll<br>
('avg overhead', 772.6118347999999)<br>
('avg process', 11504.2397625)<br>
<br>
epoll<br>
('avg overhead', 814.4771509)<br>
('avg process', 11547.058394900001)<br>
<br>
CAF<br>
('avg overhead', 847.6571822)<br>
('avg process', 11681.377972700002)<br>
<br>
select<br>
('avg overhead', 855.2147494000001)<br>
('avg process', 11585.1111236)<br>
<br>
smallFlows.pcap, 1 “poll" per packet<br>
----------------------------<br>
poll<br>
('avg overhead', 1403.8950280800004)<br>
('avg process', 22202.960570839998)<br>
<br>
epoll<br>
('avg overhead', 1470.0554376)<br>
('avg process', 22210.3240474)<br>
<br>
select<br>
('avg overhead', 2305.6278429200006)<br>
('avg process', 22549.29251384)<br>
<br>
CAF<br>
('avg overhead', 2405.1401093399995)<br>
('avg process', 23401.66596454)<br>
<br>
smallFlows.pcap, Polling Throttled to 1 per 25 packets<br>
-----------------------------<br>
poll<br>
('avg overhead', 1156.0900352)<br>
('avg process', 22113.8645395)<br>
<br>
epoll<br>
('avg overhead', 1192.37176)<br>
('avg process', 22000.2246757)<br>
<br>
select<br>
('avg overhead', 1269.0761219)<br>
('avg process', 22017.891367999997)<br>
<br>
CAF<br>
('avg overhead', 1441.6064868)<br>
('avg process', 22658.534969599998)<br>
<br>
_______________________________________________<br>
bro-dev mailing list<br>
bro-dev@bro.org<br>
<a href="http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev">http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev</a><br>
</div>
</span></font>
</body>
</html>