[Bro-Dev] Bro 2.5 Packet Drop Issue

Thu Aug 30 13:11:07 PDT 2018

Hello Everyone,

I am reaching out with the hope that someone will be able to help us with an issue we are having with Bro upgrade from 2.4.1 to 2.5.X.

We have a system with  12 core (3Ghz) ,128GB RAM, and 10G NIC (Intel X520-SR2 10GbE Dual-port), monitoring between 1.5 - 2.5 Gbps traffic.

Bro 2.4.1 is working great and periodically drops 2-5% when traffic peaks at ~ 2.5. However, when we upgrade to Bro 2.5.3/4 on the same exact system the drops go up to 90%.

We are using CentOS-7 and tired installing Bro and Pfring from both rpm and source without any luck. I wonder if anyone has seen this issue and can give some clues to resolve this issue.

Bro Node Conf:

[manager]

type=manager

host=localhost

#

[proxy-1]

type=proxy

host=localhost

#

[worker-1]

type=worker

host=localhost

interface=ens1f1

lb_method=pf_ring

lb_procs=11

pin_cpus=1,2,3,4,5,6,7,8,9,10,11

[root at bro-test ~]# cat /proc/net/pf_ring/info

PF_RING Version          : 7.3.0 (unknown)

Total rings              : 11

Standard (non ZC) Options

Ring slots               : 65534

Slot version             : 17

Capture TX               : No [RX only]

IP Defragment            : No

Socket Mode              : Standard

Cluster Fragment Queue   : 0

Cluster Fragment Discard : 0

[root at bro-test ~]# tailf /opt/bro/logs/current/capture_loss.log

1535647921.339324       60.000005       worker-1-8      318331  425005  74.900531

1535647921.217853       60.000000       worker-1-5      264716  349078  75.832908

1535647921.241244       60.000021       worker-1-9      265863  364089  73.021432

1535647921.312567       60.000002       worker-1-1      239036  315823  75.686698

1535647922.188607       60.000420       worker-1-4      238192  322818  73.785229

1535647922.760560       60.000029       worker-1-11     250678  338188  74.12386

1535647922.864470       60.000075       worker-1-3      232467  314963  73.807717

1535647923.413121       60.000024       worker-1-10     254241  345382  73.611537

1535647923.205954       60.001556       worker-1-2      259932  354980  73.224407

[root at bro-test ~]# less /opt/bro/logs/current/stats.log | bro-cut  ts      peer    mem     pkts_proc       bytes_recv      pkts_dropped

1535644801.328981       worker-1-8      2854    3523252 2214563854      8841163

1535644801.235592       worker-1-9      2833    3422300 2135680645      9083143

1535644801.299138       worker-1-1      2801    3358673 2089659287      9059868

1535644802.177016       worker-1-4      2727    3262089 2027645336      9155838

1535644801.187590       worker-1-5      2640    3336190 2085853940      9332917

1535644802.750617       worker-1-11     2726    3432674 2153405372      9018943

1535644802.853617       worker-1-3      2816    3448836 2161753414      8929662

1535644803.186853       worker-1-2      2659    3387742 2116043509      9176871

1535644803.395256       worker-1-10     2871    3407486 2132043052      9049047

1535644803.403778       worker-1-7      2821    3278503 2023604941      9966347

1535644850.898433       manager 2340    0       0       -

1535644804.257320       proxy-1 73      0       0       -

[root at bro-test logs]# broctl netstats

worker-1-1: 1535651356.794609 recvd=3501813131 dropped=3589205826 link=3501813131

worker-1-2: 1535651358.808626 recvd=4033892471 dropped=3057179730 link=4033892471

worker-1-3: 1535651358.587316 recvd=3930325145 dropped=3160768660 link=3930325145

worker-1-4: 1535651357.702299 recvd=3561053809 dropped=3530086444 link=3561053809

worker-1-5: 1535651357.650359 recvd=3399338460 dropped=3691836209 link=3399338460

worker-1-6: 1535651334.912244 recvd=3714154738 dropped=3376978237 link=3714154738

worker-1-7: 1535651359.119492 recvd=3684804437 dropped=3406432666 link=3684804437

worker-1-8: 1535651359.668621 recvd=4020016563 dropped=3071265083 link=4020016563

worker-1-9: 1535651359.867601 recvd=3807658264 dropped=3283669188 link=3807658264

worker-1-10: 1535651359.749253 recvd=3703077938 dropped=3388277853 link=3703077938

worker-1-11: 1535651359.907420 recvd=4052516305 dropped=3038874387 link=4052516305

nload output for capture NIC:

[cid:image001.png at 01D4407C.0E3A9670]

Jawad Rajput

System Administrator

U.S. Department of Energy

IM-62 /Germantown Building

HQ Network Security Team

Email: Jawad.Rajput at hq.doe.gov<mailto:Jawad.Rajput at hq.doe.gov>

Office: 301-903-2176

Office: 301-903-3895

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180830/0fd1653e/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 20047 bytes
Desc: image001.png
Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180830/0fd1653e/attachment-0001.bin 