[Bro] bro cluster with pf ring dna+libzero

[Li, Yee-Ting]

we're deploying a new bro cluster and am a huge newbie on all of this; so please excuse my ignorance. i have yet to actually start capturing on the cluster (awaiting delivery of a front-end device)

on each worker i have the dna+libzero ixgbe driver installed and insmodded. so i run:

$ sudo insmod pf_ring.ko enable_tx_capture=0 min_num_slots=32768
$ sudo insmod ixgbe.ko RSS=1,1,1,1 num_rx_slots=32768 mtu=9000



$ sudo /usr/sbin/setcap cap_net_raw,cap_net_admin=eip /usr/bin/pfdnacluster_master
$ /usr/bin/pfdnacluster_master -d -P /var/run/pfdnacluster-dna0.pid -D bromaint -c 0 -i dna0 -n 10

i do the setcap as i am running bro as non-root user. looks good…

$ cat /proc/net/pf_ring/13979-dna0.1
Bound Device(s) :
Active : 1
Breed : DNA
Sampling Rate : 1
Capture Direction : RX+TX
Socket Mode : RX only
Appl. Name : dna-cluster-0-socket-0
IP Defragment : No
BPF Filtering : Disabled
# Sw Filt. Rules : 0
# Hw Filt. Rules : 0
Poll Pkt Watermark : 128
Num Poll Calls : 0
Channel Id : 0
Num RX Slots : 32768
Num TX Slots : 8192
Tot Memory : 672399360 bytes
Cluster: Tot Recvd : 11
Cluster: Tot Sent : 0



then on my manager i have the following nodes.cfg:

[manager]
type=manager
host=sec-broman

[proxy-0]
type=proxy
host=sec-broman

[proxy-1]
type=proxy
host=sec-broman

[sec-bro01-0]
type=worker
host=sec-bro01
interface=dnacluster:0
lb_method=pf_ring
lb_procs=10



using bro 2.3; so i believe the lb_pf_ring.py script understands the dnacluster interface spec.

so i do an 'broctl install' (as user bromaint) from the manager, then log onto my worker and run

$ sudo /usr/sbin/setcap cap_net_raw,cap_net_admin=eip /opt/bro/bin/capstats
$ sudo /usr/sbin/setcap cap_net_raw,cap_net_admin=eip /opt/bro/bin/bro



then a 'broctl start' on the manager. everything looks fine so far… then i run 'broctl capstats' and i get:

Interface kpps mbps (10s average)
----------------------------------------
sec-bro01-0-9: capstats failed (error: dnacluster:0: No such device exists (SIOCGIFHWADDR: No such device))



looking at proc for the pid of that bro instance, i get:

$ ps aux | grep sec-bro01-0-9
bromaint 14696 0.0 0.0 108128 1496 ? S 15:28 0:00 bash /opt/bro/share/broctl/scripts/run-bro -1 -i dnacluster:0 at 9 -U .status -p broctl -p broctl-live -p local -p sec-bro01-0-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

bromaint 14778 25.1  0.0 157736 56320 ?        S    15:28   0:02 /opt/bro/bin/bro -i dnacluster:0 at 9 -U .status -p broctl -p broctl-live -p local -p sec-bro01-0-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
bromaint 14846 14.3  0.0 161832 52996 ?        SN   15:28   0:01 /opt/bro/bin/bro -i dnacluster:0 at 9 -U .status -p broctl -p broctl-live -p local -p sec-bro01-0-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

$ cat /proc/net/pf_ring/14778-none.7
Bound Device(s) :
Active : 1
Breed : Non-DNA
Sampling Rate : 1
Capture Direction : RX+TX
Socket Mode : RX+TX
Appl. Name : bro-dnacluster:0 at 9
IP Defragment : No
BPF Filtering : Enabled
# Sw Filt. Rules : 0
# Hw Filt. Rules : 0
Poll Pkt Watermark : 1
Num Poll Calls : 2562490



what gives???

if i manually kill the bro process on the worker and rerun capstats, i get:

Interface kpps mbps (10s average)
----------------------------------------
sec-bro01/dnacluster:0 0.0 0.0
Total 0.0 0.0


also, if i were to change the lb_procs to less than that of the pfdnacluster number of workers (-n), everything (seems to) work fine (bear in mind i'm not capturing any traffic at the moment). but would i loose any data? i'm using pf_ring 6.0.1.


thanks,

Yee.