[Bro] 5 node cluster

[Jason Carr]

Do you have a time source hooked up?  If you do not, you'll need to start
the kernel module with myri_timesource=0.  I learned this the hard way
because I received a SYNC device when I ordered a regular.

My startup script does this on my SYNC device, unload your module
/opt/snf/sbin/myri_start_stop stop then

/opt/snf/sbin/rebuild.sh
/opt/snf/sbin/myri_start_stop start myri_timesource=0



On Mon, Oct 10, 2016 at 10:52 PM Darrain Waters <dwaters at bioteam.net> wrote:

> I did compile using below. Perhaps that is not enough, as I was following
> the myricom instructions. See attached
>
> On Sat, Oct 8, 2016 at 3:29 PM, Michał Purzyński <
> michalpurzynski1 at gmail.com> wrote:
>
> http://lmgtfy.com/?q=Bro+Myricom
>
> Finds me
>
> https://www.bro.org/sphinx-git/components/bro-plugins/myricom/README.html
>
> You're welcome ;)
>
> Tested on 2.5 master and beta. Haven't tried on 2.4 although by the time
> you will build your cluster 2.5 will have been released.
>
> On 8 Oct 2016, at 19:52, Darrain Waters <dwaters at bioteam.net> wrote:
>
> Great, would you point me in the right direction. Thank you.
>
> On Sat, Oct 8, 2016 at 12:50 PM, Michał Purzyński <
> michalpurzynski1 at gmail.com> wrote:
>
> You don't seem to use native Myricom support, there's a plugin for that.
>
> On 8 Oct 2016, at 19:44, Darrain Waters <dwaters at bioteam.net> wrote:
>
> Turns out it was a simple config issue (like most times & RTFM), and
> traffic is flowing to snf0. My workers were not using the snf0 interface as
> you must if you compiled using the myricom sniffer. Also changed the cpu
> pinning so thanks for that info. I also turned off the time source in the
> snf driver. Now I need to add Arista time source.  Thanks for your time.
>
> [worker-3]
> type=worker
> host=10.0.40.16
> interface=snf0     use to be eth2
> lb_method=myricom
> lb_procs=10
> pin_cpus=2,3,4,5,6,7,8,9,10,11     the cpu pinning was not right either
> #env_vars=LD_LIBRARY_PATH=/opt/snf/lib:$PATH, SNF_FLAGS=0x1,
> SNF_DATARING_SIZE=0x100000000, SNF_NUM_RINGS=10
>
> [worker-3]
>
> type=worker
>
> host=10.0.40.16
>
> interface=eth2
>
> lb_method=myricom
>
> lb_procs=10
>
> pin_cpus=7,8,9,10,11,18,19,20,21,22
>
> env_vars=LD_LIBRARY_PATH=/opt/snf/lib:$PATH, SNF_FLAGS=0x1,
> SNF_DATARING_SIZE=0x100000000, SNF_NUM_RINGS=10
> To:
>
> [worker-3]
> type=worker
> host=10.0.40.16
> interface=snf0
> lb_method=myricom
> lb_procs=10
> pin_cpus=2,3,4,5,6,7,8,9,10,11
>
>
>
>
> On Fri, Oct 7, 2016 at 10:34 PM, Azoff, Justin S <jazoff at illinois.edu>
> wrote:
>
>
> > On Oct 7, 2016, at 6:27 PM, Darrain Waters <dwaters at bioteam.net> wrote:
> >
> > Sorry, yeah I am getting comm logs and stderr on the manager. I do have
> two NICS enabled on each system, one for management with IP and the other
> is the myricom with no IP and in sniffer mode.
> >
> > Each of the workers do have the spool wirker directories but they are
> empty.
> >
> > I use to be able to run this on the manager
> >
> > [bromgr at bromgr etc]$ sudo tcpdump -i eth2
> >
> >
> > tcpdump: snf_ring_open_id(ring=-1) failed: Device or resource busy
> >
> >
> >
> > [BroControl] > netstats
> >
> >  worker-1-1: 1475878452.092051 recvd=1 dropped=17260812 link=17260813
> >
> > worker-1-10: 1475878452.292009 recvd=1 dropped=17260812 link=17260813
> >
> >  worker-1-2: 1475878452.493003 recvd=1 dropped=17260812 link=17260813
>
> Ah, ok.. so this isn't the firewall issue...  That's when "everything is
> working but there are no logs" but in your case nothing is working :-)
>
> I'd stop bro and then make sure everything is stopped.  You can use
> 'broctl ps.bro' to ensure that there are no stray procs lying around.  Then
> at that point with nothing else running you should be able to run things
> like 'tcpdump' or 'broctl capstats' and verify that you can capture packets.
>
> You should also be able to run tools like
>
> /opt/snf/bin/myri_nic_info
> /opt/snf/bin/myri_counters
> /opt/snf/bin/myri_bandwidth
> /opt/snf/sbin/myri_license
>
> to ensure that the card+drivers are working properly as well as check
> dmesg output and check to see if it is complaining about anything
>
> I don't recall every seeing that particular netstats output, but I bet
> you'll be able to reproduce the problem with regular tcpdump.  Generally
> speaking if tcpdump -w foo.pcap writes out packets that look ok, and you
> can use bro -r against foo.pcap, bro it should work in realtime.
>
> The snf issues on the manager may be due to trying to use snf libs against
> a regular NIC, I've had to use things like
>
> LD_PRELOAD=/usr/lib64/libpcap.so.1 tcpdump ...
>
> to force it to use standard libpcap.
>
> --
> - Justin Azoff
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161011/81b76b86/attachment-0001.html