[Bro] Troubleshooting crashes

[Tritium Cat]

On Tue, Sep 11, 2012 at 8:34 PM, Tritium Cat <tritium.cat at gmail.com> wrote:

>
> On Tue, Sep 11, 2012 at 1:16 PM, Seth Hall <seth at icir.org> wrote:
>
>>
>> On Sep 10, 2012, at 8:51 PM, Tritium Cat <tritium.cat at gmail.com> wrote:
>>
>
>  > bro at bc : [12:33am] : 2012-08-30 : gzcat weird.23:00:00-00:00:00.log.gz
>> | awk '{print $7}' | sort | uniq -c | sort -rn | head -10
>> > 614589 data_before_established
>> > 585445 possible_split_routing
>>
>> I'm a little curious about these two.  Normally lots of these lines
>> indicates that something is wrong with how Bro is collecting packets.  I'm
>> interested to find out if these go away when you adapt the snap length.  Is
>> the MTU of your network actually 9600 or did you just set that MTU for the
>> interface to the maximum it would allow?
>
>
> The MTU in use is actually 9600.  I made the snaplen change you
> recommended but I'm still seeing more or less the same results in
> weird.log.  I checked the source to see what "possible_split_routing"
> represents and that leads me to think our tap/aggregation setup may be
> incorrectly load balancing traffic; I'll get back to you on that later.
>

The front-end setup is working ok.  I was missing PFRINGClusterID in
broctl.conf; fixing that seems to have helped with memory and cpu usage.

I still have lots of entries in weird.log with split_routing being the most
common.  broctl's "netstats" command does not show any drops, nor does the
PF_RING info from /proc/net/pf_ring/<pid>.  The count of "split_routing"
events is about equal across all workers so I think it's something to do
with the load-balancing via PF_RING.  The traffic is 802.1Q tagged so maybe
pf_ring is using 6-tuple load balancing for the cluster.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120913/365ad0e6/attachment.html