[Bro] bro cluster packet loss with pf_ring_zc

Gary Faulkner gfaulkner.nsm at gmail.com
Tue May 10 08:27:13 PDT 2016


I guess the first thing is to ask how much packet loss? You'll never 
fully eliminate it, but a good cluster set up can keep your average loss 
under 1%. I could ask a whole lot of questions about your cluster set 
up, but it is probably easier if you can share a redacted version of 
your node.conf (redact public IPs, DNS names, any sensitive info etc). 
You could also try running the capture loss script and doing some 
analysis on which and how many workers are dropping packets over time. 
Keep in mind if you are doing any kind of partial flow shunting this 
could skew the results. You could also look at stats.log if you have it 
enabled. If one or two workers are really dropping packets during an 
interval of time, but the rest look OK this could be traffic related 
(some large flows). If it is across the board you may need to look more 
closely at your cluster set-up for sub-optimal configuration or 
over-subscription.

~Gary


On 5/9/16 10:50 PM, Bowen Li wrote:
> Dear list,
>
> I’m using Bro 2.4.1 stable and PF_RING_ZC to analysis network traffic. The
> peak flow of the traffic almost close to 1G/bps(the full load of NIC), and
> the number of data packet in traffic may reach 200,000 pps. PF_RING_ZC
> zbalance_ipc shows pf_ring has no packet loss and broctl netstats
> shows that bro cluster have lost most of the packets, but the link number
> is equal to the receive packet number.
>
> When handle the packets, almost all of the cpus are in full load status, so
> I suspect that the processor of the server limits the packet processing
> speed, so the bro-cluster have to drop packets.
>
> So my question is now the performance of the server under 200,000 pps cases
> a packet loss in bro is normal or not.
>
> Here is the CPU info:
> Architecture:          x86_64
> CPU op-mode(s):        32-bit, 64-bit
> Byte Order:            Little Endian
> CPU(s):                32
> On-line CPU(s) list:   0-31
> Thread(s) per core:    2
> Core(s) per socket:    8
> Socket(s):             2
> NUMA node(s):          2
> Vendor ID:             GenuineIntel
> CPU family:            6
> Model:                 45
> Model name:            Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
> Stepping:              6
> CPU MHz:               1317.937
> BogoMIPS:              4419.58
> Virtualization:        VT-x
> L1d cache:             32K
> L1i cache:             32K
> L2 cache:              256K
> L3 cache:              20480K
> NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30
> NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
>
> Here is the memory info:
>                total        used        free      shared  buff/cache
> available
> Mem:       65759080    13018468    31412324      132364    21328288
>   52079776
> Swap:      29241340           0    29241340
>
> Is anyone able to help me?
>
>
>
> Thanks in advance,
>
> Bowen Li
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160510/bc98bba6/attachment.html 


More information about the Bro mailing list