[Bro] bro cluster with pf ring dna+libzero

[Gary Faulkner]

Hello,

Capstats is a separate application as far as pfdnacluster_master is 
concerned. You can tell pfdnacluster_master that you want to send the 
same traffic to another application using the -n flag by using a "," and 
then specifying how many instances of the second app you intend to run. 
When you call pfdnacluster_master try "-n 10,1" instead of "-n 10". You 
actually want to run two applications against the same traffic, but the 
second app, capstats, will only run one process that needs to consume 
all of the traffic instead of having slices of traffic load balanced 
between multiple processes.

Regards,
Gary



On 6/18/2014 6:02 PM, Li, Yee-Ting wrote:
> we're deploying a new bro cluster and am a huge newbie on all of this; so please excuse my ignorance. i have yet to actually start capturing on the cluster (awaiting delivery of a front-end device)
>
> on each worker i have the dna+libzero ixgbe driver installed and insmodded. so i run:
>
> $ sudo insmod pf_ring.ko enable_tx_capture=0 min_num_slots=32768
> $ sudo insmod ixgbe.ko RSS=1,1,1,1 num_rx_slots=32768 mtu=9000
>
>
>
> $ sudo /usr/sbin/setcap cap_net_raw,cap_net_admin=eip /usr/bin/pfdnacluster_master
> $ /usr/bin/pfdnacluster_master -d -P /var/run/pfdnacluster-dna0.pid -D bromaint -c 0 -i dna0 -n 10
>
> i do the setcap as i am running bro as non-root user. looks good…
>
> $ cat /proc/net/pf_ring/13979-dna0.1
> Bound Device(s) :
> Active : 1
> Breed : DNA
> Sampling Rate : 1
> Capture Direction : RX+TX
> Socket Mode : RX only
> Appl. Name : dna-cluster-0-socket-0
> IP Defragment : No
> BPF Filtering : Disabled
> # Sw Filt. Rules : 0
> # Hw Filt. Rules : 0
> Poll Pkt Watermark : 128
> Num Poll Calls : 0
> Channel Id : 0
> Num RX Slots : 32768
> Num TX Slots : 8192
> Tot Memory : 672399360 bytes
> Cluster: Tot Recvd : 11
> Cluster: Tot Sent : 0
>
>
>
> then on my manager i have the following nodes.cfg:
>
> [manager]
> type=manager
> host=sec-broman
>
> [proxy-0]
> type=proxy
> host=sec-broman
>
> [proxy-1]
> type=proxy
> host=sec-broman
>
> [sec-bro01-0]
> type=worker
> host=sec-bro01
> interface=dnacluster:0
> lb_method=pf_ring
> lb_procs=10
>
>
>
> using bro 2.3; so i believe the lb_pf_ring.py script understands the dnacluster interface spec.
>
> so i do an 'broctl install' (as user bromaint) from the manager, then log onto my worker and run
>
> $ sudo /usr/sbin/setcap cap_net_raw,cap_net_admin=eip /opt/bro/bin/capstats
> $ sudo /usr/sbin/setcap cap_net_raw,cap_net_admin=eip /opt/bro/bin/bro
>
>
>
> then a 'broctl start' on the manager. everything looks fine so far… then i run 'broctl capstats' and i get:
>
> Interface kpps mbps (10s average)
> ----------------------------------------
> sec-bro01-0-9: capstats failed (error: dnacluster:0: No such device exists (SIOCGIFHWADDR: No such device))
>
>
>
> looking at proc for the pid of that bro instance, i get:
>
> $ ps aux | grep sec-bro01-0-9
> bromaint 14696 0.0 0.0 108128 1496 ? S 15:28 0:00 bash /opt/bro/share/broctl/scripts/run-bro -1 -i dnacluster:0 at 9 -U .status -p broctl -p broctl-live -p local -p sec-bro01-0-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
>
> bromaint 14778 25.1  0.0 157736 56320 ?        S    15:28   0:02 /opt/bro/bin/bro -i dnacluster:0 at 9 -U .status -p broctl -p broctl-live -p local -p sec-bro01-0-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
> bromaint 14846 14.3  0.0 161832 52996 ?        SN   15:28   0:01 /opt/bro/bin/bro -i dnacluster:0 at 9 -U .status -p broctl -p broctl-live -p local -p sec-bro01-0-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
>
> $ cat /proc/net/pf_ring/14778-none.7
> Bound Device(s) :
> Active : 1
> Breed : Non-DNA
> Sampling Rate : 1
> Capture Direction : RX+TX
> Socket Mode : RX+TX
> Appl. Name : bro-dnacluster:0 at 9
> IP Defragment : No
> BPF Filtering : Enabled
> # Sw Filt. Rules : 0
> # Hw Filt. Rules : 0
> Poll Pkt Watermark : 1
> Num Poll Calls : 2562490
>
>
>
> what gives???
>
> if i manually kill the bro process on the worker and rerun capstats, i get:
>
> Interface kpps mbps (10s average)
> ----------------------------------------
> sec-bro01/dnacluster:0 0.0 0.0
> Total 0.0 0.0
>
>
> also, if i were to change the lb_procs to less than that of the pfdnacluster number of workers (-n), everything (seems to) work fine (bear in mind i'm not capturing any traffic at the moment). but would i loose any data? i'm using pf_ring 6.0.1.
>
>
> thanks,
>
> Yee.
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro