[Bro] Bro and myricom woes

[Harry Hoffman]

Hi All,

Thought I’d write in to seek some guidance from the list.

I’ve got bro running on RHEL 6.5 sitting on a box with 20 cores and 64 GB of RAM and a RAID 6 configuration through 1.2TB disks on a LSI raid card. This is a Cisco UCS 1u server.

I’m running with myricom’s sniffer 10G software (v3) in an x16 slot set at GEN II in the BIOS (I don’t have a x8 slot to put it in).

I’ve tried running both bro out of git and bro 2.3.

/usr/local/bro/etc/node.conf looks like:
drop ring full
[manager]
type=manager
host=localhost
#
[proxy-1]
type=proxy
host=localhost
#
[worker-1]
type=worker
host=localhost
interface=eth2
lb_method=myricom
lb_procs=14
pin_cpus=2,3,4,5,6,7,8,9,10,11,12,13,14,15
#

Policies are set to default for what broctl uses.

I’ve been playing around with the myricom environmentals and have the following exported before running broctl start
SNF_DATARING_SIZE = 17179869184
SNF_DESCRING_SIZE = 4294967296

Running tcpdump to output to /dev/null I see no drops in packet capture (either through myri_counters looking at the SNF ring drop full) or from tcpdump itself. Writing full packet capture to disk using tcpdump -i eth2 -s 0 -C 500 -w /usr/local/bro/logs/testing shows roughly a 2% drop via myri_counters SNF drop ring full.

Average traffic on the interface is .5 Mpps. I’m using some of the features of our gigamon to slice packets and only keep up to 32 bytes of certain packets discarding the rest of the payload (I mention this in the event that bro might have some difficulty in dealing with packets that are shorter then their advertised length).

Of all of the workers running 1 worker is always pegged at 100% and the rest of the workers use roughly 50-65% of their CPU at any given point in time.
when running broctl stop all workers stop fine except the one pegging 100% of the cpu. That was is forcibly terminated.

The increase of counters for SNF drop ring full indicates that the application (bro in this case) is not getting to the packets fast enough. broctl seems to be reporting roughly 80% packet loss.

I’ve got 1490 networks defined in /usr/local/bro/etc/networks.conf but I’m under the impression that this only matters for the email reports.

I’m happy to provide any other information but would really like to have bro work smoothly here so am hoping that someone can chime in with experiences in dealing with such problems.


many thanks!

Cheers,
Harry