[Bro] 5 node cluster

[Azoff, Justin S]

> On Oct 7, 2016, at 6:27 PM, Darrain Waters <dwaters at bioteam.net> wrote:
> 
> Sorry, yeah I am getting comm logs and stderr on the manager. I do have two NICS enabled on each system, one for management with IP and the other is the myricom with no IP and in sniffer mode.
> 
> Each of the workers do have the spool wirker directories but they are empty.
> 
> I use to be able to run this on the manager
> 
> [bromgr at bromgr etc]$ sudo tcpdump -i eth2
> 
> 
> tcpdump: snf_ring_open_id(ring=-1) failed: Device or resource busy
> 
> 
> 
> [BroControl] > netstats
> 
>  worker-1-1: 1475878452.092051 recvd=1 dropped=17260812 link=17260813
> 
> worker-1-10: 1475878452.292009 recvd=1 dropped=17260812 link=17260813
> 
>  worker-1-2: 1475878452.493003 recvd=1 dropped=17260812 link=17260813

Ah, ok.. so this isn't the firewall issue...  That's when "everything is working but there are no logs" but in your case nothing is working :-)

I'd stop bro and then make sure everything is stopped.  You can use 'broctl ps.bro' to ensure that there are no stray procs lying around.  Then at that point with nothing else running you should be able to run things like 'tcpdump' or 'broctl capstats' and verify that you can capture packets.

You should also be able to run tools like

/opt/snf/bin/myri_nic_info
/opt/snf/bin/myri_counters
/opt/snf/bin/myri_bandwidth
/opt/snf/sbin/myri_license

to ensure that the card+drivers are working properly as well as check dmesg output and check to see if it is complaining about anything

I don't recall every seeing that particular netstats output, but I bet you'll be able to reproduce the problem with regular tcpdump.  Generally speaking if tcpdump -w foo.pcap writes out packets that look ok, and you can use bro -r against foo.pcap, bro it should work in realtime.

The snf issues on the manager may be due to trying to use snf libs against a regular NIC, I've had to use things like

LD_PRELOAD=/usr/lib64/libpcap.so.1 tcpdump ...

to force it to use standard libpcap.

-- 
- Justin Azoff