[Bro] Issue: load balancer PF_RING drops 25% of incoming packets

[Rosinger, Enno (DualStudy)]

Hey Everyone,

With your hint to disable the offloading features, I can now see all my packets as required and Bro is distributing them evenly.
I consider this Issue fixed for now. Thanks for all the help and advices - I appreciate it.

Best,
Enno

-----Original Message-----
From: Rosinger, Enno (DualStudy) 
Sent: Mittwoch, 27. Juli 2016 11:00
To: 'Azoff, Justin S' <jazoff at illinois.edu>
Cc: 'bro at bro.org' <bro at bro.org>
Subject: RE: [Bro] Issue: load balancer PF_RING drops 25% of incoming packets

Hey Justin,

Thanks for these good advices.
I'll test if it helps turn off the offloading, because it could indeed be, that the packets are reassembled.
Do you think the PF_RINGs packet difference could also be caused by a bad configured irq-affinity? Philosnef suggested that in a separate mail.

And I agree that BRO seems to be working fine, since the numbers match pretty well. 
I am looking forward to the pf-ring people's response and see what their opinion is.

Thanks for the support again. I'll let you know if I could fix it.

Best,
Enno


-----Original Message-----
From: Azoff, Justin S [mailto:jazoff at illinois.edu] 
Sent: Mittwoch, 27. Juli 2016 07:06
To: Rosinger, Enno (DualStudy) <enno.rosinger at hpe.com>
Cc: bro at bro.org
Subject: Re: [Bro] Issue: load balancer PF_RING drops 25% of incoming packets


> On Jul 26, 2016, at 8:27 PM, Rosinger, Enno (DualStudy) <enno.rosinger at hpe.com> wrote:
> 
> Hi Justin,
> 
> Thank you for the fast reply.
> 
> 21 Million received packets: Bro receives it's traffic on an isolated network (where the traffic is generated another server by TCPreplay). I manually take the stats of received packets of the NIC before and after a replaying by issuing "ifconfig eno2(interface-name)" .
> 16 Million handled packets: I use broctl and issue the command "netstats" to see the number of each worker process' received packets. If you make a sum out of that you will come to 16 Million (NOTE: now 18 Million, as I upgraded to Zero Copy drivers since the last mail).
> 
> ###Ifconfig on Bro system###
> ###Before replaying###
> [root at slinky-3-4 kernel]# ifconfig eno2 [...]
>        RX packets 25758824  bytes 20353552393 (18.9 GiB)
>        RX errors 0  dropped 0  overruns 0  frame 0
>        TX packets 182  bytes 36558 (35.7 KiB)
>        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 
> [...]
> 
> ###After replaying###
> [root at slinky-3-4 kernel]# ifconfig eno2 [...]
>        RX packets 47447181  bytes 37400251832 (34.8 GiB)
>        RX errors 0  dropped 0  overruns 0  frame 0
>        TX packets 268  bytes 54486 (53.2 KiB)
>        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 
> [...]
> 
> That makes 47447181 - 25758824  = 21.688.357 received packets
> 
> ###netstats in broctl on Bro system### ### after replaying ### 
> [BroControl] > netstats
> worker-1-1: 1469577816.953862 recvd=5088052 dropped=0 link=5088052
> worker-1-2: 1469577817.153796 recvd=4205599 dropped=0 link=4205599
> worker-1-3: 1469577817.353889 recvd=4562288 dropped=0 link=4562288
> worker-1-4: 1469577817.554795 recvd=4546975 dropped=0 link=4546975
> 
> The sum of this is 18.402.914 packets, which are seen by BRO as "on the link".
> 
> Thanks to your help on the build issue I can also support this number 
> with the stats of pfcount (NOTE: This is another run - slightly 
> different numbers ) ##PFcount result Absolute Stats: [18'416'555 pkts 
> total][0 pkts dropped][0.0% dropped]
> [18'416'555 pkts rcvd][17'225'248'719 bytes rcvd][58'886.73 
> pkt/sec][440.62 Mbit/sec] ========================= Actual Stats: [0 
> pkts rcvd][722.14 ms][0.00 pps][0.00 Gbps]

Ah, do I understand that to mean that pfcount is also showing that only 18 million packets are received as well?  If that is the case you should probably reach out to the pf_ring people and see if they have any ideas.

If pfcount and bro both agree on the number of packets received, the problem is probably not within bro.

The one thing I can think of is that you have not disabled offloading with something like

    for i in rx tx sg tso ufo gso gro lro; do ethtool -K en02 $i off; done

So that while the system is receiving 21 million packets, they are being reassembled into only 18 million.


--
- Justin Azoff