[Bro] broctl unable to find peers

Zeolla@GMail.com zeolla at gmail.com
Wed Dec 21 12:53:05 PST 2016


Awesome, thank you.

So, I worked with Justin on IRC and we did find this:

$ ./broctl print foo worker-1-1

  worker-1-1   <error: cannot connect to WORKER:47767>

However, when I ran tcpdump on WORKER I saw a clean connection setup, data
transfer, and teardown from the manager.  I also turned logging on for the
manager's iptables, ran `./broctl status` assuming it would hit the manager
first, and I didn't see any DROPs or REJECTs that would be relevant
(looking at eth0, 127.0.0.1, and 127.0.1.1).


Per Justin's suggestion I'm going to look into enabling debugging in
broccoli tomorrow.

Jon

On Wed, Dec 21, 2016 at 3:19 PM Daniel Thayer <dnthayer at illinois.edu> wrote:

> One simple workaround for the status command being too slow is to
> edit your etc/broctl.cfg file and look for the option
> "StatusCmdShowAll".  Change it to this:
>
> StatusCmdShowAll = 0
>
> However, this doesn't solve the problem of Bro processes
> not being able to communicate with each other.
>
>
> On 12/21/16 1:43 PM, Zeolla at GMail.com wrote:
> > I get a similar failure with broctl peerstatus when the cluster is up.
> > It sits for a few minutes then kills itself.
> >
> > $ time ./broctl peerstatus
> >
> > Killed
> >
> >
> > real6m48.594s
> >
> > user0m0.102s
> >
> > sys0m0.111s
> >
> >
> > I have tried adding a log line to my iptables so it will log right
> > before getting dropped, but after reviewing the log over a 10 minute
> > period I wasn't able to find anything from any members of my bro cluster
> > getting dropped.  While the logging was on I tried multiple ./broctl
> > commands, including directly hitting the server using ./broctl status
> > worker-1-1 and a more general ./broctl status or ./broctl peerstatus.
> >
> > Jon
> >
> > On Wed, Dec 21, 2016 at 1:54 PM Daniel Thayer <dnthayer at illinois.edu
> > <mailto:dnthayer at illinois.edu>> wrote:
> >
> >     What happens if you run "broctl peerstatus"? (after starting
> >     the cluster, of course)
> >
> >
> >     On 12/21/16 11:18 AM, Zeolla at GMail.com wrote:
> >     > I'm seeing an issue using bro 2.4.1 where when I run `./broctl
> status`
> >     > it hangs on 'Getting peer status ...'.  When I run the same command
> >     > specifying manager, any of the proxies, or any of the individual
> >     workers
> >     > it has no issue.  Has anybody seen this before?
> >     >
> >     > This is a 5 node cluster (1 manager, 4 sensors) running on Ubuntu
> >     > 14.04.  I am in the process of upgrading to 2.5, but before I do
> >     so I'm
> >     > adding 2 additional sensor machines (bringing it to 7 nodes) to the
> >     > cluster because we sorely need the additional processing power.
> After
> >     > the upgrade to 2.5 I will be adding another node and splitting the
> >     > logger function onto it, making it an 8 node cluster.
> >     >
> >     > Here's an example of me running `./broctl status` and it failing
> >     after 3
> >     > 1/2 minutes, then it goes on to successfully get the status for
> every
> >     > component/instance specifically, however the Peers section returns
> >     "???".
> >     >
> >     > $ time ./broctl status || time ./broctl status manager;time for
> >     proxy in
> >     > {1..5}; do ./broctl status proxy-${proxy}; done;for svr in {1..4};
> do
> >     > for instance in {1..20}; do ./broctl status
> worker-${svr}-${instance};
> >     > done; done
> >     >
> >     > removing stale lock
> >     >
> >     > Getting process status ...
> >     >
> >     > Getting peer status ...
> >     >
> >     > Killed
> >     >
> >     >
> >     > real3m35.233s
> >     >
> >     > user0m0.126s
> >     >
> >     > sys0m0.119s
> >     >
> >     > waiting for lock (owned by PID 22222) ...
> >     >
> >     > Getting process status ...
> >     >
> >     > Getting peer status ...
> >     >
> >     > Name         Type    Host             Status    Pid    Peers
> Started
> >     >
> >     > manager      manager A.B.C.D   running   11111  ???    18 Dec
> 03:24:38
> >     >
> >     > <snip>
> >     >
> >     >
> >     > Jon
> >     > --
> >     >
> >     > Jon
> >     >
> >     > Sent from my mobile device
> >     >
> >     >
> >     >
> >     > _______________________________________________
> >     > Bro mailing list
> >     > bro at bro-ids.org <mailto:bro at bro-ids.org>
> >     > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.ICSI.Berkeley.EDU_mailman_listinfo_bro&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Bi5qPBnY0NmYPqnRTPj_AfXQKpfQTZUpCzpfFBcawv0&m=jpt8TXKljrs0LwDVNY1QHlYBJ0kWtZsyM3QUo0ee46M&s=DLU_e8vfR1vSmBwUN8TMkF012iVQWkEVPZXC6elvBLE&e=
> >
> >     >
> >
> > --
> >
> > Jon
> >
> > Sent from my mobile device
> >
>
-- 

Jon

Sent from my mobile device
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161221/5eb8aa8b/attachment.html 


More information about the Bro mailing list