[Bro] 5 node cluster

[Azoff, Justin S]

> On Oct 7, 2016, at 4:40 PM, Darrain Waters <dwaters at bioteam.net> wrote:
> 
> Hello
> 
> The myricom cards in my cluster nodes are dropping packets, and I am not getting any log information in prefix/logs. Did I miss something during the setup process ? Please see below for initial info and please let me know what else is needed. Thank you.
> 
> Darrain
> 
> I compiled bro using the option below.
> --with-pcap=/opt/snf/ 
> 
> [bromgr at bromgr ~]$ ldd /usr/local/bro/bin/bro | grep pcap
> 
> libpcap.so.1 => /opt/snf/lib/libpcap.so.1 (0x00007faf9c3d5000)
> 

Looks good

> 
> I get the following when I run capstats
> 
> [BroControl] > capstats
> 
> 
> 
> Interface             kpps       mbps       (10s average)
> 
> ----------------------------------------
> 
> worker-1-1: capstats failed (error: eth2: snf_ring_open_id(ring=-1) failed: Device or resource busy)
> 
> worker-2-1: capstats failed (error: eth2: snf_ring_open_id(ring=-1) failed: Device or resource busy)
> 
> worker-3-1: capstats failed (error: eth2: snf_ring_open_id(ring=-1) failed: Device or resource busy)
> 
> worker-4-1: capstats failed (error: eth2: snf_ring_open_id(ring=-1) failed: Device or resource busy)
> 
> worker-5-1: capstats failed (error: eth2: snf_ring_open_id(ring=-1) failed: Device or resource busy)

This is normal.. capstats for snf never worked right (it could never work with snfv2 and with snfv3 it needs to set a different app id as bro, otherwise it can't capture at the same time as bro.  As long as bro is running and not failing with the same error you're ok.  There are better ways to get data out of a myricom card using the myricom tools as well.

Your node.cfg looks mostly ok.  I would switch to only running 1 or 2 proxies and just run them on the manager node.

Why are you using 7,8,9,10,11,18,19,20,21,22 in particular?  What CPUs do you have?  This is potentially not doing what you intend.  Most likely 7/19 8/20 9/21 10/22 are the same cpu.

Your underlying problem is probably that a firewall is enabled on your hosts and the worker processes can't reach the manager.  Daniel just wrote a good section on this for the manual:


This section summarizes the network communication between Bro and BroControl,
which is useful to understand if you need to reconfigure your firewall.  If
your firewall is preventing Bro communication, then either the "deploy"
command or the "peerstatus" command will fail.

For a cluster setup, BroControl uses ssh to run commands on other hosts in
the cluster, so the manager host needs to connect to TCP port 22 on each
of the other hosts in the cluster.  Note that BroControl never attempts
to ssh to the localhost, so in a standalone setup BroControl does not use ssh.

Each instance of Bro in a cluster needs to communicate directly with other
instances of Bro regardless of whether these instances are running on the same
host or not.  Each proxy and worker needs to connect to the manager,
and each worker needs to connect to one proxy.  If a logger node is defined,
then each of the other nodes needs to connect to the logger.

Note that you can change the port that Bro listens on by changing the value
of the "BroPort" option in your ``broctl.cfg`` file (this should be needed
only if your system has another process that listens on the same port).  By
default, a standalone Bro listens on TCP port 47760.  For a cluster setup,
the logger listens on TCP port 47761, and the manager listens on TCP port 47762
(or 47761 if no logger is defined).  Each proxy is assigned its own port
number, starting with one number greater than the manager's port.  Likewise,
each worker is assigned its own port starting one number greater than the
highest port number assigned to a proxy.

Finally, a few BroControl commands (such as "print" and "peerstatus") rely
on broccoli to communicate with Bro.  This means that for those commands to
function, BroControl needs to connect to each Bro instance.


-- 
- Justin Azoff