[Bro] 5 node cluster

[Darrain Waters]

Turns out it was a simple config issue (like most times & RTFM), and
traffic is flowing to snf0. My workers were not using the snf0 interface as
you must if you compiled using the myricom sniffer. Also changed the cpu
pinning so thanks for that info. I also turned off the time source in the
snf driver. Now I need to add Arista time source.  Thanks for your time.

[worker-3]
type=worker
host=10.0.40.16
interface=snf0     use to be eth2
lb_method=myricom
lb_procs=10
pin_cpus=2,3,4,5,6,7,8,9,10,11     the cpu pinning was not right either
#env_vars=LD_LIBRARY_PATH=/opt/snf/lib:$PATH, SNF_FLAGS=0x1,
SNF_DATARING_SIZE=0x100000000, SNF_NUM_RINGS=10

[worker-3]

type=worker

host=10.0.40.16

interface=eth2

lb_method=myricom

lb_procs=10

pin_cpus=7,8,9,10,11,18,19,20,21,22

env_vars=LD_LIBRARY_PATH=/opt/snf/lib:$PATH, SNF_FLAGS=0x1,
SNF_DATARING_SIZE=0x100000000, SNF_NUM_RINGS=10
To:

[worker-3]
type=worker
host=10.0.40.16
interface=snf0
lb_method=myricom
lb_procs=10
pin_cpus=2,3,4,5,6,7,8,9,10,11




On Fri, Oct 7, 2016 at 10:34 PM, Azoff, Justin S <jazoff at illinois.edu>
wrote:

>
> > On Oct 7, 2016, at 6:27 PM, Darrain Waters <dwaters at bioteam.net> wrote:
> >
> > Sorry, yeah I am getting comm logs and stderr on the manager. I do have
> two NICS enabled on each system, one for management with IP and the other
> is the myricom with no IP and in sniffer mode.
> >
> > Each of the workers do have the spool wirker directories but they are
> empty.
> >
> > I use to be able to run this on the manager
> >
> > [bromgr at bromgr etc]$ sudo tcpdump -i eth2
> >
> >
> > tcpdump: snf_ring_open_id(ring=-1) failed: Device or resource busy
> >
> >
> >
> > [BroControl] > netstats
> >
> >  worker-1-1: 1475878452.092051 recvd=1 dropped=17260812 link=17260813
> >
> > worker-1-10: 1475878452.292009 recvd=1 dropped=17260812 link=17260813
> >
> >  worker-1-2: 1475878452.493003 recvd=1 dropped=17260812 link=17260813
>
> Ah, ok.. so this isn't the firewall issue...  That's when "everything is
> working but there are no logs" but in your case nothing is working :-)
>
> I'd stop bro and then make sure everything is stopped.  You can use
> 'broctl ps.bro' to ensure that there are no stray procs lying around.  Then
> at that point with nothing else running you should be able to run things
> like 'tcpdump' or 'broctl capstats' and verify that you can capture packets.
>
> You should also be able to run tools like
>
> /opt/snf/bin/myri_nic_info
> /opt/snf/bin/myri_counters
> /opt/snf/bin/myri_bandwidth
> /opt/snf/sbin/myri_license
>
> to ensure that the card+drivers are working properly as well as check
> dmesg output and check to see if it is complaining about anything
>
> I don't recall every seeing that particular netstats output, but I bet
> you'll be able to reproduce the problem with regular tcpdump.  Generally
> speaking if tcpdump -w foo.pcap writes out packets that look ok, and you
> can use bro -r against foo.pcap, bro it should work in realtime.
>
> The snf issues on the manager may be due to trying to use snf libs against
> a regular NIC, I've had to use things like
>
> LD_PRELOAD=/usr/lib64/libpcap.so.1 tcpdump ...
>
> to force it to use standard libpcap.
>
> --
> - Justin Azoff
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161008/ca0e6f52/attachment.html