[Bro] Logger crash with Bro 2.5

Raj Srinivasan raj at bivio.net
Tue Aug 8 12:17:33 PDT 2017


[Note to Moderator: Resending with less data since the previous one was too long]

Hi,

We have been seeing this logger crash with Bro 2.5 running on a PPC based system. I have included the diag output from broctl below. It seems to be related to the rate at which logs are generated - the higher the rate (corresponding to more incoming traffic) the sooner the crash occurs. The CPU on which the logger runs has only about 4GB of memory, out of which approximately 1.5GB is available to the logger. As the logs below show, tc_malloc() does fail to allocate memory a few times, and I am guessing that the disks are not able to keep up with the logging rate.

Is the logger capable of recovering from memory allocation failure? If so, does the trace below indicate any other issue? Would appreciate any suggestions and/or insight into the problem.

Thanks,
Raj



[BroControl] > diag logger
[logger]

Bro 2.5
Linux 2.6.31-45bbv51npc3

Bro plugins: (none found)

Core file: core.14433
[New Thread 14454]
[New Thread 14455]
[New Thread 14713]
[New Thread 14714]
[New Thread 14717]
[New Thread 15060]
[New Thread 15061]
Core was generated by `/var/tmp/bro/spool/tmp/bro -U .status -p broctl -p broctl -live -p local -p logg'.
Program terminated with signal 11, Segmentation fault.
#0  0x0f72b598 in *__GI__IO_default_xsputn (f=0xbf8935a8, data=0x1057a43c,
    n=17) at genops.c:447
        in genops.c

Thread 8 (Thread 15061):
#0  0x1027c89c in threading::formatter::Ascii::Describe(ODesc*, int, threading::Field const* const*, threading::Value**) const ()
#1  0x1027c7ec in threading::formatter::Ascii::Describe(ODesc*, int, threading::Field const* const*, threading::Value**) const ()
#2  0x10503a8c in logging::writer::Ascii::DoWrite(int, threading::Field const* const*, threading::Value**) ()
#3  0x10520a3c in logging::WriterBackend::Write(int, int, threading::Value***)
    ()
Backtrace stopped: frame did not save the PC
Current language:  auto; currently minimal

Thread 7 (Thread 15060):
#0  0x0fb3591c in __pthread_cond_timedwait (cond=0x12a2e570, mutex=0x12a2e468,
    abstime=0x4b03fbc8) at pthread_cond_timedwait.c:199
#1  0x1027a000 in threading::MsgThread::Run() ()
Backtrace stopped: frame did not save the PC

Thread 6 (Thread 14717):
#0  0x0fb3591c in __pthread_cond_timedwait (cond=0x12a2ef30, mutex=0x12a2edc8,
    abstime=0x4a83fbc8) at pthread_cond_timedwait.c:199
#1  0x1027a000 in threading::MsgThread::Run() ()
Backtrace stopped: frame did not save the PC

Thread 5 (Thread 14714):
#0  0x0fb3591c in __pthread_cond_timedwait (cond=0x12a2f800, mutex=0x12a2f6b0,
    abstime=0x4a03fbc8) at pthread_cond_timedwait.c:199
#1  0x1027a000 in threading::MsgThread::Run() ()
Backtrace stopped: frame did not save the PC

Thread 4 (Thread 14713):
#0  0x0fb3591c in __pthread_cond_timedwait (cond=0x12a2c1a0, mutex=0x12a2c080,
    abstime=0x4983fbc8) at pthread_cond_timedwait.c:199
#1  0x1027a000 in threading::MsgThread::Run() ()
Backtrace stopped: frame did not save the PC

Thread 3 (Thread 14455):
#0  0x0fb3591c in __pthread_cond_timedwait (cond=0x12a2cad0, mutex=0x12a2c998,
    abstime=0x4903fbc8) at pthread_cond_timedwait.c:199
#1  0x1027a000 in threading::MsgThread::Run() ()
Backtrace stopped: frame did not save the PC

Thread 2 (Thread 14454):
#0  0x0fb3591c in __pthread_cond_timedwait (cond=0x12a2d3d0, mutex=0x12a2d298,
    abstime=0x4883fbc8) at pthread_cond_timedwait.c:199
#1  0x1027a000 in threading::MsgThread::Run() ()
Backtrace stopped: frame did not save the PC

Thread 1 (Thread 14433):

==== reporter.log
1501635254.338084       Reporter::WARNING       Your interface is likely receivi                        ng invalid TCP checksums, most likely from NIC checksum offloading.  By default,                         packets with invalid checksums are discarded by Bro unless using the -C command                        -line option or toggling the 'ignore_checksums' variable.  Alternatively, disabl                        e checksum offloading by the network adapter to ensure Bro analyzes the actual c                        hecksums that are transmitted.  /usr/local/bro/share/bro/base/misc/find-checksum                        -offloading.bro, line 54
1501635254.016895       Reporter::WARNING       Your interface is likely receivi                        ng invalid TCP checksums, most likely from NIC checksum offloading.  By default,                         packets with invalid checksums are discarded by Bro unless using the -C command                        -line option or toggling the 'ignore_checksums' variable.  Alternatively, disabl                        e checksum offloading by the network adapter to ensure Bro analyzes the actual c                        hecksums that are transmitted.  /usr/local/bro/share/bro/base/misc/find-checksum                        -offloading.bro, line 54
1501635254.486632       Reporter::WARNING       Your interface is likely receivi                        ng invalid TCP checksums, most likely from NIC checksum offloading.  By default,                         packets with invalid checksums are discarded by Bro unless using the -C command                        -line option or toggling the 'ignore_checksums' variable.  Alternatively, disabl                        e checksum offloading by the network adapter to ensure Bro analyzes the actual c                        hecksums that are transmitted.  /usr/local/bro/share/bro/base/misc/find-checksum                        -offloading.bro, line 54
1501635254.857459       Reporter::WARNING       Your interface is likely receivi                        ng invalid TCP checksums, most likely from NIC checksum offloading.  By default,                         packets with invalid checksums are discarded by Bro unless using the -C command                        -line option or toggling the 'ignore_checksums' variable.  Alternatively, disabl                        e checksum offloading by the network adapter to ensure Bro analyzes the actual c                        hecksums that are transmitted.  /usr/local/bro/share/bro/base/misc/find-checksum                        -offloading.bro, line 54
1501635254.017679       Reporter::WARNING       Your interface is likely receivi                        ng invalid TCP checksums, most likely from NIC checksum offloading.  By default,                         packets with invalid checksums are discarded by Bro unless using the -C command                        -line option or toggling the 'ignore_checksums' variable.  Alternatively, disabl                        e checksum offloading by the network adapter to ensure Bro analyzes the actual c                        hecksums that are transmitted.  /usr/local/bro/share/bro/base/misc/find-checksum                        -offloading.bro, line 54
1501635254.365704       Reporter::WARNING       Your interface is likely receivi                        ng invalid TCP checksums, most likely from NIC checksum offloading.  By default,                         packets with invalid checksums are discarded by Bro unless using the -C command                        -line option or toggling the 'ignore_checksums' variable.  Alternatively, disabl                        e checksum offloading by the network adapter to ensure Bro analyzes the actual c                        hecksums that are transmitted.  /usr/local/bro/share/bro/base/misc/find-checksum                        -offloading.bro, line 54
1501635254.300063       Reporter::WARNING       Your interface is likely receivi                        ng invalid TCP checksums, most likely from NIC checksum offloading.  By default,                         packets with invalid checksums are discarded by Bro unless using the -C command                        -line option or toggling the 'ignore_checksums' variable.  Alternatively, disabl                        e checksum offloading by the network adapter to ensure Bro analyzes the actual c                        hecksums that are transmitted.  /usr/local/bro/share/bro/base/misc/find-checksum                        -offloading.bro, line 54
1501635254.486147       Reporter::WARNING       Your interface is likely receivi                        ng invalid TCP checksums, most likely from NIC checksum offloading.  By default,                         packets with invalid checksums are discarded by Bro unless using the -C command                        -line option or toggling the 'ignore_checksums' variable.  Alternatively, disabl                        e checksum offloading by the network adapter to ensure Bro analyzes the actual c                        hecksums that are transmitted.  /usr/local/bro/share/bro/base/misc/find-checksum                        -offloading.bro, line 54
1501635254.686000       Reporter::WARNING       Your interface is likely receivi                        ng invalid TCP checksums, most likely from NIC checksum offloading.  By default,                         packets with invalid checksums are discarded by Bro unless using the -C command                        -line option or toggling the 'ignore_checksums' variable.  Alternatively, disabl                        e checksum offloading by the network adapter to ensure Bro analyzes the actual c                        hecksums that are transmitted.  /usr/local/bro/share/bro/base/misc/find-checksum                        -offloading.bro, line 54
1501635254.733590       Reporter::WARNING       Your interface is likely receivi                        ng invalid TCP checksums, most likely from NIC checksum offloading.  By default,                         packets with invalid checksums are discarded by Bro unless using the -C command                        -line option or toggling the 'ignore_checksums' variable.  Alternatively, disabl                        e checksum offloading by the network adapter to ensure Bro analyzes the actual c                        hecksums that are transmitted.  /usr/local/bro/share/bro/base/misc/find-checksum                        -offloading.bro, line 54

==== stderr.log
src/central_freelist.cc:322] tcmalloc: allocation failed 16384
out of memory in new.
src/central_freelist.cc:322] tcmalloc: allocation failed 16384
out of memory in new.
src/central_freelist.cc:322] tcmalloc: allocation failed 16384
out of memory in new.
src/central_freelist.cc:322] tcmalloc: allocation failed 16384
out of memory in new.
src/central_freelist.cc:322] tcmalloc: allocation failed 16384
/usr/local/bro/share/broctl/scripts/run-bro: line 107: 14433 Segmentation fault                              (core dumped) nohup "$mybro" "$@"

==== stdout.log
max memory size         (kbytes, -m) 750000
data seg size           (kbytes, -d) 750000
virtual memory          (kbytes, -v) 750000
core file size          (blocks, -c) unlimited

==== .cmdline
-U .status -p broctl -p broctl-live -p local -p logger local.bro broctl base/fra                        meworks/cluster local-logger.bro broctl/auto

==== .env_vars
PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/sbin:/usr                        /local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/root/bin:/usr/local/bro/bin
BROPATH=/var/tmp/bro/spool/installed-scripts-do-not-touch/site::/var/tmp/bro/spo                        ol/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/s                        hare/bro/policy:/usr/local/bro/share/bro/site
CLUSTER_NODE=logger

==== .status
RUNNING [net_run]

==== No prof.log

==== packet_filter.log
#separator \x09
#set_separator  ,
#empty_field    (empty)
#unset_field    -
#path   packet_filter
#open   2017-08-01-17-52-24
#fields ts      node    filter  init    success
#types  time    string  string  bool    bool

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170808/b743347d/attachment-0001.html 


More information about the Bro mailing list