[Bro] Troubleshooting crashes

[Seth Hall]

On Aug 30, 2012, at 5:46 PM, Tritium Cat <tritium.cat at gmail.com> wrote:

> What's the best way to disable Bro in a systematic way to isolate crashes ?

Sending us the diag output from broctl is best since it will include a back trace.

> I disabled all the protocols except SSH and a few default scripts that utilize it.

How were you disabling protocols? (nevermind, i see the answer later and i'll comment there)

>  After ~12 hours I returned to find many of the worker nodes had crashed.  I forgot to look at the diag for the crashed workers before stopping the cluster.

Do you have the cron command setup correctly?  The workers should have been restart automatically after they crashed and a diagnostic email sent to you.
	Mentioned in this section: http://bro-ids.org/documentation/quickstart.html#a-minimal-starting-configuration

> base/init-default.bro configuration 
> =======================================
> bro at bc : [9:20pm] : bro : grep -v "^#" base/init-default.bro  | grep "[a-z]"

I see that you were modifying scripts in the base directory to disable analyzers and I just wanted to point out that we don't support directly making changes to scripts there at all and it's possible that you could get into trouble if you try to update or reinstall with changes in that area.

That said, nice job figuring that out since we don't provide a lot of support in that area right now. :)  I'm hoping to introduce an analyzer framework which will provide an API for doing that with the 2.2 release.

> Total rings         : 10

How many CPU cores do you have?

> -rw-r--r--  1 bro  bro     10323 Aug 30 21:15 reporter.log
> -rw-r--r--  1 bro  bro  52846117 Aug 30 21:27 weird.log

I'm curious about what's in reporter.log, normally that shouldn't have too much in it.  Also, that's an astonishingly large weird.log.  Is there anything that stands out in those two?

Could you show me your node.cfg configuration too?

Oh, and one last thing, have you made sure to disable all of special NIC features?
	http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not-full.html

  .Seth


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/