[Bro] Issue: load balancer PF_RING drops 25% of incoming packets

[Azoff, Justin S]

> On Jul 26, 2016, at 8:27 PM, Rosinger, Enno (DualStudy) <enno.rosinger at hpe.com> wrote:
> 
> Hi Justin,
> 
> Thank you for the fast reply.
> 
> 21 Million received packets: Bro receives it's traffic on an isolated network (where the traffic is generated another server by TCPreplay). I manually take the stats of received packets of the NIC before and after a replaying by issuing "ifconfig eno2(interface-name)" .
> 16 Million handled packets: I use broctl and issue the command "netstats" to see the number of each worker process' received packets. If you make a sum out of that you will come to 16 Million (NOTE: now 18 Million, as I upgraded to Zero Copy drivers since the last mail).
> 
> ###Ifconfig on Bro system###
> ###Before replaying###
> [root at slinky-3-4 kernel]# ifconfig eno2
> [...]
>        RX packets 25758824  bytes 20353552393 (18.9 GiB)
>        RX errors 0  dropped 0  overruns 0  frame 0
>        TX packets 182  bytes 36558 (35.7 KiB)
>        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 [...]
> 
> ###After replaying###
> [root at slinky-3-4 kernel]# ifconfig eno2
> [...]
>        RX packets 47447181  bytes 37400251832 (34.8 GiB)
>        RX errors 0  dropped 0  overruns 0  frame 0
>        TX packets 268  bytes 54486 (53.2 KiB)
>        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 [...]
> 
> That makes 47447181 - 25758824  = 21.688.357 received packets
> 
> ###netstats in broctl on Bro system###
> ### after replaying ###
> [BroControl] > netstats
> worker-1-1: 1469577816.953862 recvd=5088052 dropped=0 link=5088052
> worker-1-2: 1469577817.153796 recvd=4205599 dropped=0 link=4205599
> worker-1-3: 1469577817.353889 recvd=4562288 dropped=0 link=4562288
> worker-1-4: 1469577817.554795 recvd=4546975 dropped=0 link=4546975
> 
> The sum of this is 18.402.914 packets, which are seen by BRO as "on the link".
> 
> Thanks to your help on the build issue I can also support this number with the stats of pfcount (NOTE: This is another run - slightly different numbers ) ##PFcount result Absolute Stats: [18'416'555 pkts total][0 pkts dropped][0.0% dropped]
> [18'416'555 pkts rcvd][17'225'248'719 bytes rcvd][58'886.73 pkt/sec][440.62 Mbit/sec] ========================= Actual Stats: [0 pkts rcvd][722.14 ms][0.00 pps][0.00 Gbps]

Ah, do I understand that to mean that pfcount is also showing that only 18 million packets are received as well?  If that is the case you should probably reach out to the pf_ring people and see if they have any ideas.

If pfcount and bro both agree on the number of packets received, the problem is probably not within bro.

The one thing I can think of is that you have not disabled offloading with something like

    for i in rx tx sg tso ufo gso gro lro; do ethtool -K en02 $i off; done

So that while the system is receiving 21 million packets, they are being reassembled into only 18 million.


-- 
- Justin Azoff