[Bro] 5 node cluster

Darrain Waters dwaters at bioteam.net
Tue Oct 11 08:15:33 PDT 2016


Yeah, I already did that and determined that it was the issue. I responded
as much, and the sysadmin blocked my response because I had myricom
instructions attached to it.

Thanks for your response, it is helpful.

On Tue, Oct 11, 2016 at 10:09 AM, Jason Carr <jason.carr at gmail.com> wrote:

> Do you have a time source hooked up?  If you do not, you'll need to start
> the kernel module with myri_timesource=0.  I learned this the hard way
> because I received a SYNC device when I ordered a regular.
>
> My startup script does this on my SYNC device, unload your module
> /opt/snf/sbin/myri_start_stop stop then
>
> /opt/snf/sbin/rebuild.sh
> /opt/snf/sbin/myri_start_stop start myri_timesource=0
>
>
>
> On Mon, Oct 10, 2016 at 10:52 PM Darrain Waters <dwaters at bioteam.net>
> wrote:
>
>> I did compile using below. Perhaps that is not enough, as I was following
>> the myricom instructions. See attached
>>
>> On Sat, Oct 8, 2016 at 3:29 PM, Michał Purzyński <
>> michalpurzynski1 at gmail.com> wrote:
>>
>>> http://lmgtfy.com/?q=Bro+Myricom
>>>
>>> Finds me
>>>
>>> https://www.bro.org/sphinx-git/components/bro-plugins/
>>> myricom/README.html
>>>
>>> You're welcome ;)
>>>
>>> Tested on 2.5 master and beta. Haven't tried on 2.4 although by the time
>>> you will build your cluster 2.5 will have been released.
>>>
>>> On 8 Oct 2016, at 19:52, Darrain Waters <dwaters at bioteam.net> wrote:
>>>
>>> Great, would you point me in the right direction. Thank you.
>>>
>>> On Sat, Oct 8, 2016 at 12:50 PM, Michał Purzyński <
>>> michalpurzynski1 at gmail.com> wrote:
>>>
>>>> You don't seem to use native Myricom support, there's a plugin for that.
>>>>
>>>> On 8 Oct 2016, at 19:44, Darrain Waters <dwaters at bioteam.net> wrote:
>>>>
>>>> Turns out it was a simple config issue (like most times & RTFM), and
>>>> traffic is flowing to snf0. My workers were not using the snf0 interface as
>>>> you must if you compiled using the myricom sniffer. Also changed the cpu
>>>> pinning so thanks for that info. I also turned off the time source in the
>>>> snf driver. Now I need to add Arista time source.  Thanks for your time.
>>>>
>>>> [worker-3]
>>>> type=worker
>>>> host=10.0.40.16
>>>> interface=snf0     use to be eth2
>>>> lb_method=myricom
>>>> lb_procs=10
>>>> pin_cpus=2,3,4,5,6,7,8,9,10,11     the cpu pinning was not right either
>>>> #env_vars=LD_LIBRARY_PATH=/opt/snf/lib:$PATH, SNF_FLAGS=0x1,
>>>> SNF_DATARING_SIZE=0x100000000, SNF_NUM_RINGS=10
>>>>
>>>> [worker-3]
>>>>
>>>> type=worker
>>>>
>>>> host=10.0.40.16
>>>>
>>>> interface=eth2
>>>>
>>>> lb_method=myricom
>>>>
>>>> lb_procs=10
>>>>
>>>> pin_cpus=7,8,9,10,11,18,19,20,21,22
>>>>
>>>> env_vars=LD_LIBRARY_PATH=/opt/snf/lib:$PATH, SNF_FLAGS=0x1,
>>>> SNF_DATARING_SIZE=0x100000000, SNF_NUM_RINGS=10
>>>> To:
>>>>
>>>> [worker-3]
>>>> type=worker
>>>> host=10.0.40.16
>>>> interface=snf0
>>>> lb_method=myricom
>>>> lb_procs=10
>>>> pin_cpus=2,3,4,5,6,7,8,9,10,11
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Oct 7, 2016 at 10:34 PM, Azoff, Justin S <jazoff at illinois.edu>
>>>> wrote:
>>>>
>>>>>
>>>>> > On Oct 7, 2016, at 6:27 PM, Darrain Waters <dwaters at bioteam.net>
>>>>> wrote:
>>>>> >
>>>>> > Sorry, yeah I am getting comm logs and stderr on the manager. I do
>>>>> have two NICS enabled on each system, one for management with IP and the
>>>>> other is the myricom with no IP and in sniffer mode.
>>>>> >
>>>>> > Each of the workers do have the spool wirker directories but they
>>>>> are empty.
>>>>> >
>>>>> > I use to be able to run this on the manager
>>>>> >
>>>>> > [bromgr at bromgr etc]$ sudo tcpdump -i eth2
>>>>> >
>>>>> >
>>>>> > tcpdump: snf_ring_open_id(ring=-1) failed: Device or resource busy
>>>>> >
>>>>> >
>>>>> >
>>>>> > [BroControl] > netstats
>>>>> >
>>>>> >  worker-1-1: 1475878452.092051 recvd=1 dropped=17260812 link=17260813
>>>>> >
>>>>> > worker-1-10: 1475878452.292009 recvd=1 dropped=17260812 link=17260813
>>>>> >
>>>>> >  worker-1-2: 1475878452.493003 recvd=1 dropped=17260812 link=17260813
>>>>>
>>>>> Ah, ok.. so this isn't the firewall issue...  That's when "everything
>>>>> is working but there are no logs" but in your case nothing is working :-)
>>>>>
>>>>> I'd stop bro and then make sure everything is stopped.  You can use
>>>>> 'broctl ps.bro' to ensure that there are no stray procs lying around.  Then
>>>>> at that point with nothing else running you should be able to run things
>>>>> like 'tcpdump' or 'broctl capstats' and verify that you can capture packets.
>>>>>
>>>>> You should also be able to run tools like
>>>>>
>>>>> /opt/snf/bin/myri_nic_info
>>>>> /opt/snf/bin/myri_counters
>>>>> /opt/snf/bin/myri_bandwidth
>>>>> /opt/snf/sbin/myri_license
>>>>>
>>>>> to ensure that the card+drivers are working properly as well as check
>>>>> dmesg output and check to see if it is complaining about anything
>>>>>
>>>>> I don't recall every seeing that particular netstats output, but I bet
>>>>> you'll be able to reproduce the problem with regular tcpdump.  Generally
>>>>> speaking if tcpdump -w foo.pcap writes out packets that look ok, and you
>>>>> can use bro -r against foo.pcap, bro it should work in realtime.
>>>>>
>>>>> The snf issues on the manager may be due to trying to use snf libs
>>>>> against a regular NIC, I've had to use things like
>>>>>
>>>>> LD_PRELOAD=/usr/lib64/libpcap.so.1 tcpdump ...
>>>>>
>>>>> to force it to use standard libpcap.
>>>>>
>>>>> --
>>>>> - Justin Azoff
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Bro mailing list
>>>> bro at bro-ids.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>>
>>>>
>>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161011/40eda1ec/attachment.html 


More information about the Bro mailing list