[Bro] Packet Loss with Myricom Bro Cluster

[Zach Rogers]

Hello,

I have a cluster of five machines for our current Bro configuration: One
is acting as the Master & Proxy node, and the other four are Worker
nodes. The four worker nodes have Myricom 10g NICs and are using the
Bro::Myricom plugin.

The workers are getting traffic from the Myricom NICs, though there
seems to be substantial packet loss. Here is the output of running
'broctl netstats all', as you can see the dropped packet count is quite
high:

    manager: 1493071687.452948 recvd=5089239 dropped=15719766
link=20825230
    proxy-1: 1493071730.166046 recvd=0 dropped=0 link=0
 worker-1-1: 1493071533.397463 recvd=16723822 dropped=1827590608
link=1844314430
worker-1-10: 1493071533.598767 recvd=15599277 dropped=1827590608
link=1843189885
worker-1-11: 1493071533.799606 recvd=15290549 dropped=1827590608
link=1842881157
worker-1-12: 1493071534.001465 recvd=17829130 dropped=1827590608
link=1845419738
 worker-1-2: 1493071534.201668 recvd=16913145 dropped=1827590608
link=1844503753
 worker-1-3: 1493071534.402914 recvd=13317747 dropped=1827590608
link=1840908355
 worker-1-4: 1493071534.603851 recvd=16654854 dropped=1827590608
link=1844245462
 worker-1-5: 1493071534.791954 recvd=15707229 dropped=1827590608
link=1843297837
 worker-1-6: 1493071535.005706 recvd=44924122 dropped=1827590608
link=1872514730
 worker-1-7: 1493071535.206737 recvd=15892468 dropped=1827590608
link=1843483076
 worker-1-8: 1493071535.407748 recvd=15109664 dropped=1827590608
link=1842700272
 worker-1-9: 1493071535.609068 recvd=17725999 dropped=1827590608
link=1845316607
 worker-2-1: 1493071452.728863 recvd=12467133 dropped=1654102950
link=1666570083
worker-2-10: 1493071452.930009 recvd=18175352 dropped=1654102950
link=1672278302
worker-2-11: 1493071453.129614 recvd=14050575 dropped=1654102950
link=1668153525
worker-2-12: 1493071453.330973 recvd=13461021 dropped=1654102950
link=1667563971
 worker-2-2: 1493071453.532331 recvd=18143473 dropped=1654102950
link=1672246423
 worker-2-3: 1493071453.731385 recvd=21263867 dropped=1654102950
link=1675366817
 worker-2-4: 1493071453.933757 recvd=21217186 dropped=1654102950
link=1675320136
 worker-2-5: 1493071454.135406 recvd=15458489 dropped=1654102950
link=1669561439
 worker-2-6: 1493071454.336124 recvd=27904731 dropped=1654102950
link=1682007681
 worker-2-7: 1493071454.535881 recvd=15498799 dropped=1654102950
link=1669601749
 worker-2-8: 1493071454.738093 recvd=15570231 dropped=1654102950
link=1669673181
 worker-2-9: 1493071454.941583 recvd=15546579 dropped=1654102950
link=1669649529
 worker-3-1: 1493071595.728039 recvd=16016236 dropped=1393679752
link=1409695988
worker-3-10: 1493071595.928627 recvd=16806524 dropped=1393679752
link=1410486276
worker-3-11: 1493071596.129858 recvd=16960142 dropped=1393679752
link=1410639894
worker-3-12: 1493071596.331357 recvd=19031806 dropped=1393679752
link=1412711558
 worker-3-2: 1493071596.531622 recvd=16059903 dropped=1393679752
link=1409739655
 worker-3-3: 1493071596.732314 recvd=20528301 dropped=1393679752
link=1414208053
 worker-3-4: 1493071596.933870 recvd=17478728 dropped=1393679752
link=1411158480
 worker-3-5: 1493071597.134664 recvd=15785290 dropped=1393679752
link=1409465042
 worker-3-6: 1493071597.335570 recvd=13051546 dropped=1393679752
link=1406731298
 worker-3-7: 1493071597.529927 recvd=17668246 dropped=1393679752
link=1411347998
 worker-3-8: 1493071597.736828 recvd=21655950 dropped=1393679752
link=1415335702
 worker-3-9: 1493071597.938700 recvd=18237026 dropped=1393679752
link=1411916778
 worker-4-1: 1493071949.358339 recvd=19489052 dropped=1504105858
link=1523594910
worker-4-10: 1493071949.559733 recvd=18684571 dropped=1504105858
link=1522790429
worker-4-11: 1493071949.760921 recvd=18578400 dropped=1504105858
link=1522684258
worker-4-12: 1493071949.961847 recvd=15738360 dropped=1504105858
link=1519844218
 worker-4-2: 1493071950.160646 recvd=17092444 dropped=1504105858
link=1521198302
 worker-4-3: 1493071950.363595 recvd=16890327 dropped=1504105858
link=1520996185
 worker-4-4: 1493071950.564875 recvd=14940510 dropped=1504105858
link=1519046368
 worker-4-5: 1493071950.765612 recvd=17175752 dropped=1504105858
link=1521281610
 worker-4-6: 1493071950.966709 recvd=14854076 dropped=1504105858
link=1518959934
 worker-4-7: 1493071951.167789 recvd=15455452 dropped=1504105858
link=1519561310
 worker-4-8: 1493071951.368743 recvd=14915366 dropped=1504105858
link=1519021224
 worker-4-9: 1493071951.572223 recvd=21105493 dropped=1504105858
link=1525211351

I am looking for guidance on how to figure out the cause of this packet
loss, with hopes of fixing the issue.

Any ideas?

 
All the best,

-- 
Zach Rogers
Security Analyst, Office of Information Security
Information Services | Oregon State University
http://is.oregonstate.edu/ois