[Bro] Packet Loss with Myricom Bro Cluster

Azoff, Justin S jazoff at illinois.edu
Mon Apr 24 15:36:56 PDT 2017


> On Apr 24, 2017, at 6:15 PM, Zach Rogers <zach.rogers at oregonstate.edu> wrote:
>
> Hello,
>
> I have a cluster of five machines for our current Bro configuration: One
> is acting as the Master & Proxy node, and the other four are Worker
> nodes. The four worker nodes have Myricom 10g NICs and are using the
> Bro::Myricom plugin.
>
> The workers are getting traffic from the Myricom NICs, though there
> seems to be substantial packet loss. Here is the output of running
> 'broctl netstats all', as you can see the dropped packet count is quite
> high:


> worker-1-12: 1493071534.001465 recvd=17829130 dropped=1827590608
> link=1845419738
>
...

> I am looking for guidance on how to figure out the cause of this packet
> loss, with hopes of fixing the issue.
>
> Any ideas?
>


There's an issue with the myricom interface where the dropped packet counter is per card and it does not reset to 0 when the application restarts.

This causes the dropped packet counter to appear to be (in your case) at least 12x larger than it is.. If not more, depending on the uptime of the server.

Run this to reset the dropped counter and restart bro:

broctl stop
broctl exec /opt/snf/bin/myri_counters -c -p 0
broctl exec /opt/snf/bin/myri_counters -c -p 1
broctl start

That will reset all counters to zero before bro starts and ensure the drop count isn't crazy high before bro even starts.

I wrote the attached script a while ago to compute the proper per card drop percentage.  It should still work.



Some of these issues can possibly be fixed inside of bro/broctl.  I think bro can just grab the drop count at startup and if it's non-zero use it as an offset for any future values.

The drop count being per card is a bit harder.  You could just divide the drop count by the number of workers, but that's really just trading one misleading value for another.

--
- Justin Azoff
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170424/d094332a/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: netstats_sum.py
Type: text/x-python-script
Size: 785 bytes
Desc: netstats_sum.py
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170424/d094332a/attachment.bin 


More information about the Bro mailing list