[Bro] Bro2 Random Crashes

[Chris Crawford]

When initially I set up Bro2, it ran for a few days with no problems.

I recently noticed recently that Bro2 has started to crash randomly.
Typically it will take Bro2 5 or 6 hours before a crash, but sometimes
it crashes immediately.

I installed Bro2 on Ubuntu 10.04 LTS.  Ubuntu is a xen VM on a Citrix
Xen server.  The throughput on the network is < 400 Mb/s.

After taking a look at one of my crash reports, Seth suggested that
I'm running out of memory.

The VM I have initially started out with 4GB of RAM, but I bumped it
up to 8GB of RAM, only to get the same results.  There was no time
difference, and there is no pattern as to why or when Bro2 crashes.

The only thing I've been able to key in on, is that over time Bro2
eventually causes all free memory in Ubuntu to change to cached
memory.  From the Citrix Xen Console, that cached memory shows up as
used memory.  So, maybe Xen interprets cached memory as being used?
Also -- when Xen senses that most of the memory is "used" (but,
really, it's cached inside Ubuntu), the percent utilization in one of
the CPUs in the VM spikes.  After Bro2 crashes, CPU utilization
returns to normal, but memory is never freed -- it remains cached
forever.

I can dump the cache using

echo 1 > /proc/sys/vm/drop_caches

Which converts all the memory allocated as cached to free.  Any chance
that this is related?

Anyone have ideas on what is causing Bro2 to crash?


-Chris



I have included some additional information below.

Here are the steps I used to install Bro2:

sudo aptitude -y install swig libmagic-dev libgeoip-dev cmake
build-essential flex bison libpcap-dev libssl-dev python-dev gawk
cd /tmp
wget http://www.bro-ids.org/downloads/release/bro-2.0.tar.gz
tar xvzf bro-2.0.tar.gz
cd bro-2.0
./configure
make
sudo make install
sudo chmod a+w /etc/bash.bashrc
sudo echo '' >> /etc/bash.bashrc
sudo echo 'export PATH=/usr/local/bro/bin:$PATH' >>  /etc/bash.bashrc
sudo chmod go-w /etc/bash.bashrc

# Add to /usr/local/bro/etc/networks.cfg:
[...]

broctl install
broctl start

### End Installation ###


Here is a recent crash report:

core
[New Thread 5101]
Core was generated by `/usr/local/bro/bin/bro -i eth1 -U .status -p
broctl -p broctl-live -p standalon'.
Program terminated with signal 6, Aborted.
#0  0x00007f72aefcaa75 in raise () from /lib/libc.so.6

==== reporter.log
#separator \x09
#set_separator  ,
#empty_field    (empty)
#unset_field    -
#path   reporter
#fields ts      level   message location
#types  time    enum    string  string
1330462476.662662       Reporter::ERROR bro wasn't compiled with IPv6
support   (empty)

==== stderr.log
listening on eth1, capture length 8192 bytes

terminate called after throwing an instance of 'std::bad_alloc'
what():  std::bad_alloc
/usr/local/bro/share/broctl/scripts/run-bro: line 60:  5101 Aborted
            (core dumped) nohup $mybro $@

==== stdout.log
unlimited
unlimited
unlimited

==== .cmdline
-i eth1 -U .status -p broctl -p broctl-live -p standalone -p local -p
bro local broctl broctl/standalone broctl/auto

==== .env_vars
PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
BROPATH=/logs/bro/spool/policy/site::/logs/bro/spool/policy/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site
CLUSTER_NODE=

==== .status
RUNNING [net_run]

==== No prof.log

==== packet_filter.log
#separator \x09
#set_separator  ,
#empty_field    (empty)
#unset_field    -
#path   packet_filter
#fields ts      node    filter  init    success
#types  time    string  string  bool    bool
1330462468.693819       -       not ip6 T       T

==== loaded_scripts.log
#separator \x09
#set_separator  ,
#empty_field    (empty)
#unset_field    -
#path   loaded_scripts
#fields name
#types  string
/usr/local/bro/share/bro/base/init-bare.bro
/usr/local/bro/share/bro/base/const.bif.bro
/usr/local/bro/share/bro/base/types.bif.bro
/usr/local/bro/share/bro/base/strings.bif.bro
/usr/local/bro/share/bro/base/bro.bif.bro
/usr/local/bro/share/bro/base/reporter.bif.bro
/usr/local/bro/share/bro/base/event.bif.bro
/usr/local/bro/share/bro/base/frameworks/logging/__load__.bro
/usr/local/bro/share/bro/base/frameworks/logging/./main.bro
/usr/local/bro/share/bro/base/logging.bif.bro
/usr/local/bro/share/bro/base/frameworks/logging/./postprocessors/__load__.bro
/usr/local/bro/share/bro/base/frameworks/logging/./postprocessors/./scp.bro
/usr/local/bro/share/bro/base/frameworks/logging/./postprocessors/./sftp.bro
/usr/local/bro/share/bro/base/frameworks/logging/./writers/ascii.bro
/usr/local/bro/share/bro/base/init-default.bro
/usr/local/bro/share/bro/base/utils/site.bro
/usr/local/bro/share/bro/base/utils/./patterns.bro
/usr/local/bro/share/bro/base/utils/addrs.bro
/usr/local/bro/share/bro/base/utils/conn-ids.bro
/usr/local/bro/share/bro/base/utils/directions-and-hosts.bro
/usr/local/bro/share/bro/base/utils/files.bro
/usr/local/bro/share/bro/base/utils/numbers.bro
/usr/local/bro/share/bro/base/utils/paths.bro
/usr/local/bro/share/bro/base/utils/strings.bro
/usr/local/bro/share/bro/base/utils/thresholds.bro
/usr/local/bro/share/bro/base/frameworks/notice/__load__.bro
/usr/local/bro/share/bro/base/frameworks/notice/./main.bro
/usr/local/bro/share/bro/base/frameworks/notice/./weird.bro
/usr/local/bro/share/bro/base/frameworks/notice/./actions/drop.bro
/usr/local/bro/share/bro/base/frameworks/notice/./actions/email_admin.bro
/usr/local/bro/share/bro/base/frameworks/notice/./actions/page.bro
/usr/local/bro/share/bro/base/frameworks/notice/./actions/add-geodata.bro
/usr/local/bro/share/bro/base/frameworks/notice/./extend-email/hostnames.bro
/usr/local/bro/share/bro/base/frameworks/cluster/__load__.bro
/usr/local/bro/share/bro/base/frameworks/cluster/./main.bro
/usr/local/bro/share/bro/base/frameworks/control/__load__.bro
/usr/local/bro/share/bro/base/frameworks/control/./main.bro
/usr/local/bro/share/bro/base/frameworks/notice/./actions/pp-alarms.bro
/usr/local/bro/share/bro/base/frameworks/dpd/__load__.bro
/usr/local/bro/share/bro/base/frameworks/dpd/./main.bro
/usr/local/bro/share/bro/base/frameworks/signatures/__load__.bro
/usr/local/bro/share/bro/base/frameworks/signatures/./main.bro
/usr/local/bro/share/bro/base/frameworks/packet-filter/__load__.bro
/usr/local/bro/share/bro/base/frameworks/packet-filter/./main.bro
/usr/local/bro/share/bro/base/frameworks/packet-filter/./netstats.bro
/usr/local/bro/share/bro/base/frameworks/software/__load__.bro
/usr/local/bro/share/bro/base/frameworks/software/./main.bro
/usr/local/bro/share/bro/base/frameworks/communication/__load__.bro
/usr/local/bro/share/bro/base/frameworks/communication/./main.bro
/usr/local/bro/share/bro/base/frameworks/metrics/__load__.bro
/usr/local/bro/share/bro/base/frameworks/metrics/./main.bro
/usr/local/bro/share/bro/base/frameworks/metrics/./non-cluster.bro
/usr/local/bro/share/bro/base/frameworks/intel/__load__.bro
/usr/local/bro/share/bro/base/frameworks/intel/./main.bro
/usr/local/bro/share/bro/base/frameworks/reporter/__load__.bro
/usr/local/bro/share/bro/base/frameworks/reporter/./main.bro
/usr/local/bro/share/bro/base/protocols/conn/__load__.bro
/usr/local/bro/share/bro/base/protocols/conn/./main.bro
/usr/local/bro/share/bro/base/protocols/conn/./contents.bro
/usr/local/bro/share/bro/base/protocols/conn/./inactivity.bro
/usr/local/bro/share/bro/base/protocols/dns/__load__.bro
/usr/local/bro/share/bro/base/protocols/dns/./consts.bro
/usr/local/bro/share/bro/base/protocols/dns/./main.bro
/usr/local/bro/share/bro/base/protocols/ftp/__load__.bro
/usr/local/bro/share/bro/base/protocols/ftp/./utils-commands.bro
/usr/local/bro/share/bro/base/protocols/ftp/./main.bro
/usr/local/bro/share/bro/base/protocols/ftp/./file-extract.bro
/usr/local/bro/share/bro/base/protocols/http/__load__.bro
/usr/local/bro/share/bro/base/protocols/http/./main.bro
/usr/local/bro/share/bro/base/protocols/http/./utils.bro
/usr/local/bro/share/bro/base/protocols/http/./file-ident.bro
/usr/local/bro/share/bro/base/protocols/http/./file-hash.bro
/usr/local/bro/share/bro/base/protocols/http/./file-extract.bro
/usr/local/bro/share/bro/base/protocols/irc/__load__.bro
/usr/local/bro/share/bro/base/protocols/irc/./main.bro
/usr/local/bro/share/bro/base/protocols/irc/./dcc-send.bro
/usr/local/bro/share/bro/base/protocols/smtp/__load__.bro
/usr/local/bro/share/bro/base/protocols/smtp/./main.bro
/usr/local/bro/share/bro/base/protocols/smtp/./entities.bro
/usr/local/bro/share/bro/base/protocols/smtp/./entities-excerpt.bro
/usr/local/bro/share/bro/base/protocols/ssh/__load__.bro
/usr/local/bro/share/bro/base/protocols/ssh/./main.bro
/usr/local/bro/share/bro/base/protocols/ssl/__load__.bro
/usr/local/bro/share/bro/base/protocols/ssl/./consts.bro
/usr/local/bro/share/bro/base/protocols/ssl/./main.bro
/usr/local/bro/share/bro/base/protocols/ssl/./mozilla-ca-list.bro
/usr/local/bro/share/bro/base/protocols/syslog/__load__.bro
/usr/local/bro/share/bro/base/protocols/syslog/./consts.bro
/usr/local/bro/share/bro/base/protocols/syslog/./main.bro
/logs/bro/spool/policy/site/local.bro
/usr/local/bro/share/bro/policy/misc/loaded-scripts.bro
/usr/local/bro/share/bro/policy/tuning/defaults/__load__.bro
/usr/local/bro/share/bro/policy/tuning/defaults/./packet-fragments.bro
/usr/local/bro/share/bro/policy/tuning/defaults/./warnings.bro
/usr/local/bro/share/bro/policy/frameworks/software/vulnerable.bro
/usr/local/bro/share/bro/policy/frameworks/software/version-changes.bro
/usr/local/bro/share/bro/policy/protocols/ftp/software.bro
/usr/local/bro/share/bro/policy/protocols/smtp/software.bro
/usr/local/bro/share/bro/policy/protocols/ssh/software.bro
/usr/local/bro/share/bro/policy/protocols/http/software.bro
/usr/local/bro/share/bro/policy/protocols/dns/detect-external-names.bro
/usr/local/bro/share/bro/policy/protocols/ftp/detect.bro
/usr/local/bro/share/bro/policy/protocols/conn/known-hosts.bro
/usr/local/bro/share/bro/policy/protocols/conn/known-services.bro
/usr/local/bro/share/bro/policy/protocols/ssl/known-certs.bro
/usr/local/bro/share/bro/policy/protocols/ssl/cert-hash.bro
/usr/local/bro/share/bro/policy/protocols/ssl/validate-certs.bro
/usr/local/bro/share/bro/policy/protocols/ssh/geo-data.bro
/usr/local/bro/share/bro/policy/protocols/ssh/detect-bruteforcing.bro
/usr/local/bro/share/bro/policy/protocols/ssh/interesting-hostnames.bro
/usr/local/bro/share/bro/policy/protocols/http/detect-MHR.bro
/usr/local/bro/share/bro/policy/protocols/http/detect-sqli.bro
/usr/local/bro/share/bro/broctl/__load__.bro
/usr/local/bro/share/bro/broctl/./main.bro
/usr/local/bro/share/bro/policy/frameworks/control/controllee.bro
/usr/local/bro/share/bro/policy/frameworks/communication/listen.bro
/usr/local/bro/share/bro/broctl/standalone.bro
/logs/bro/spool/policy/auto/standalone-layout.bro
/usr/local/bro/share/bro/policy/misc/trim-trace-file.bro
/usr/local/bro/share/bro/broctl/auto.bro
/logs/bro/spool/policy/auto/local-networks.bro
/logs/bro/spool/policy/auto/broctl-config.bro