[Bro] Bro cluster requirements and manager logging backlog bug

[Hovsep Levi]

> Your weird.log is 1/5 the size of your conn.log and larger than your http
> log.. You should look into which weird name is showing up so much, you may
> have a serious problem with your tap configuration.  That is not normal at
> all.



I'm going to guess that the messages relate to packet loss on the workers
which is currently 3%-10% per worker process.  Adding another worker node
should fix it.

AFAIK the tap is working fine.  In my experience the only weird message
that corresponded to tap issues was "possible_split_routing".


[bro at mgr /opt/bro_data/logs/current]$ awk '{print $7}' weird.log | sort |
uniq -c | sort -rn | head -25
243077 dns_unmatched_msg
205444 SYN_seq_jump
183857 bad_HTTP_request
54153 window_recision
42960 bad_UDP_checksum
20681 dns_unmatched_reply
14280 data_before_established
13048 possible_split_routing
9717 DNS_RR_unknown_type
4310 line_terminated_with_single_CR
4121 active_connection_reuse
1676 TCP_ack_underflow_or_misorder
1490 connection_originator_SYN_ack
1226 SYN_with_data
 925 TCP_seq_underflow_or_misorder
 862 above_hole_data_without_any_acks
 727 Teredo_bubble_with_payload
 706 HTTP_version_mismatch
 685 data_after_reset
 677 unknown_HTTP_method
 626 unexpected_multiple_HTTP_requests
 584 inappropriate_FIN
 566 empty_http_request
 511 bad_TCP_checksum
 373 inflate_failed





>
> It looks like the bulk of your logging load is coming from files+conn and
> weird though, so what you can do is cut down the volume of those logs to
> get your cpu to be happy.
>
> >
> > What model/count CPU does your manager have?
> >
> >
> > Four of these with 32 GB per NUMA node.
> >
> > Processor Information
> >         Socket Designation: CPU1
> >         Type: Central Processor
> >         Family: Opteron 6200
>
> Ah!!! This is part of your problem.  Every site we have worked with in the
> past year or so that was having serious manager performance issues was
> using the crazy high core count AMD systems.  While they perform well when
> you have a heavily threaded task (And I bet they do, we have an entire
> supercomputer filled with 40,000 of them) the bro logger only has few
> heavyweight threads and just does not work well on these processors.
>
> That said, you can probably get this working acceptably though.  There are
> two options for this:
>
> * filter some noisy log lines, which will cause them to not be logged at
> all.
> * split heavy streams into multiple log files, which will let the logger
> process dedicate a logging thread to each part.
>
> I would start by trying some of these config fragments that split log
> files apart:
>
>

Thanks for the help, I'm going to give your suggestions a try.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161220/ba1f2beb/attachment.html