[Bro] Help resolving proxy crash

inetjunkmail inetjunkmail at gmail.com
Thu Sep 25 11:10:46 PDT 2014


All:

I'm having an issue with the proxy service crashing.  I was having it on
2.3 and I'm having it on 2.3.1 too.  It generally occurs a few minutes
after restarting.  I have a _little_ evidence that suggests that it's
stable at lower traffic rates.  This is a single node cluster with 16 cores
(32 counting hyper threading).  Below is some hopefully relevant
information.  Can anyone provide some tips at what to look at next to
correct the issue?

Thanks

--------------------------------------------------

[e at b3 ~]$ sudo broctl status proxy-3
Name         Type    Host             Status    Pid    Peers  Started
proxy-3      proxy   biggsanalyzer3   running   1810   ???    25 Sep
13:09:07

[e at b3 ~]$ sudo broctl status proxy-3
[sudo] password for eric:
Name         Type    Host             Status    Pid    Peers  Started
proxy-3      proxy   biggsanalyzer3   crashed

[e at b3 ~]$ sudo broctl netstats
 worker-3-1: 1411666855.350645 recvd=23952666 dropped=0 link=23952666
worker-3-10: 1411666855.550643 recvd=26529426 dropped=0 link=26529426
worker-3-11: 1411666855.750069 recvd=25799879 dropped=0 link=25799879
worker-3-12: 1411666855.952250 recvd=27786138 dropped=0 link=27786138
worker-3-13: 1411666856.152395 recvd=33072225 dropped=0 link=33072225
worker-3-14: 1411666856.352869 recvd=26334798 dropped=0 link=26334798
 worker-3-2: 1411666856.554573 recvd=26726716 dropped=0 link=26726716
 worker-3-3: 1411666856.754446 recvd=32427073 dropped=0 link=32427073
 worker-3-4: 1411666856.955059 recvd=26646497 dropped=0 link=26646497
 worker-3-5: 1411666857.156298 recvd=27240324 dropped=0 link=27240324
 worker-3-6: 1411666857.356603 recvd=24139487 dropped=0 link=24139487
 worker-3-7: 1411666857.555774 recvd=28722053 dropped=0 link=28722053
 worker-3-8: 1411666857.757538 recvd=27019501 dropped=0 link=27019501
 worker-3-9: 1411666857.126295 recvd=25049180 dropped=0 link=25049180

[e at b3 ~]$ sudo broctl capstats
Interface             kpps       mbps       (10s average)
----------------------------------------
b3/em1             331.7      1146.1

Total                 331.7      1146.1

[e at b3 ~]$ sudo broctl diag proxy-3
[proxy-3]

Bro 2.3.1
Linux 3.10.0-123.6.3.el7.x86_64

core.1810
[New LWP 1810]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/bro/bin/bro -U .status -p broctl -p
broctl-live -p local -p proxy-3'.
Program terminated with signal 6, Aborted.
#0  0x00007fce6a5015c9 in raise () from /lib64/libc.so.6

Thread 1 (Thread 0x7fce6c86c840 (LWP 1810)):
#0  0x00007fce6a5015c9 in raise () from /lib64/libc.so.6
#1  0x00007fce6a502cd8 in abort () from /lib64/libc.so.6
#2  0x000000000059dae1 in Reporter::InternalError (this=<optimized out>,
fmt=fmt at entry=0x7f209b "%s") at /home/e/bro-2.3.1/src/Reporter.cc:137
#3  0x00000000005bc85a in InternalCommError (msg=<optimized out>,
this=0x1915530) at /home/e/bro-2.3.1/src/RemoteSerializer.cc:3231
#4  RemoteSerializer::Poll (this=0x1915530, may_block=may_block at entry=false)
at /home/e/bro-2.3.1/src/RemoteSerializer.cc:1576
#5  0x00000000005bc9df in Poll (may_block=false, this=0x1915530) at
/home/e/bro-2.3.1/src/RemoteSerializer.cc:1413
#6  RemoteSerializer::NextTimestamp (this=0x1915530,
local_network_time=0x7fffa1562040) at
/home/e/bro-2.3.1/src/RemoteSerializer.cc:1380
#7  0x00000000005965fb in IOSourceRegistry::FindSoonest (this=0xaea0d0
<io_sources>, ts=ts at entry=0x7fffa1562108) at
/home/e/bro-2.3.1/src/IOSource.cc:61
#8  0x000000000059fd82 in net_run () at /home/e/bro-2.3.1/src/Net.cc:370
#9  0x0000000000503df8 in main (argc=<optimized out>, argv=<optimized out>)
at /home/e/bro-2.3.1/src/main.cc:1165

==== No reporter.log

==== stderr.log
internal error: unknown msg type 115 in Poll()
/usr/local/bro/share/broctl/scripts/run-bro: line 85:  1810 Aborted
        (core dumped) nohup $mybro "$@"

==== stdout.log
max memory size         (kbytes, -m) unlimited
data seg size           (kbytes, -d) unlimited
virtual memory          (kbytes, -v) unlimited
core file size          (blocks, -c) unlimited

==== .cmdline
-U .status -p broctl -p broctl-live -p local -p proxy-3 local.bro broctl
base/frameworks/cluster local-proxy broctl/auto

==== .env_vars
PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/home/e/perl5/bin:/usr/local/bro/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/e/.local/bin:/home/e/bin
BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site
CLUSTER_NODE=proxy-3

==== .status
TERMINATED [internal_error]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log
[e at b3 ~]$ cat /usr/local/bro/etc/node.cfg
# Example BroControl node configuration.
#
# Example BroControl node configuration.
#
# This example has a standalone node ready to go except for possibly
changing
# the sniffing interface.

# This is a complete standalone configuration.  Most likely you will
# only need to change the interface.
#[bro]
#type=standalone
#host=localhost
#interface=eth0

## Below is an example clustered configuration. If you use this,
## remove the [bro] node above.

[manager]
type=manager
host=b3

[proxy-3]
type=proxy
host=b3

[worker-3]
type=worker
host=b3
interface=em1
lb_method=pf_ring
lb_procs=14
#pin_cpus=4,6,8,10,12,14,16,18,20,22,24,26,28,30
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140925/ba102485/attachment.html 


More information about the Bro mailing list