[Bro] Worker OOM crashes

[Dave Crawford]

Has anyone experienced this crash scenario that I’m in the process of debugging? This just started in the last couple of days on a cluster that has been in production for just shy of two years without issue.

tcmalloc: large alloc 18446744072956297216 bytes == (nil) @  0x7f08e85b5dcb 0x7f08e85b5f1b 0x7f08e85b6965 0x7f08e85e89c5 0x7f1fff 0x867016 0x86781e 0x7f1a96 0x7f1db6 0x7f121a 0x7efbbe 0x7ed334 0x866eb5 0x56441c 0x600327 0x60124e 0x5cfa14 0x837db3 0x5cfecf 0x52ecb0 0x7f08e759fb45 0x5373ad (nil) out of memory in new.
1477631586.955885 fatal error: out of memory in new.

The server itself still has some ceiling height:

$ free -h
             total       used       free     shared    buffers     cached
Mem:          126G        23G       102G        16M        67M       5.1G
-/+ buffers/cache:        18G       107G
Swap:          33G         0B        33G

But I’m wondering if the processes themselves are hitting a maximum size for memory allocation, as some are nearing 4GB.

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
24925 bro        20   0 3964M 3262M  269M S 78.1  2.5  2h47:55 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-5 loc
  618 bro        20   0 1217M 1160M  269M S 77.0  0.9  1h57:12 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-2 loc
26073 bro        20   0 3989M 3156M  269M S 76.4  2.4  2h41:38 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-7 loc
23938 bro        20   0 2610M 2318M  269M R 75.9  1.8  2h45:00 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-9 loc
27816 bro        20   0 1554M 1455M  269M S 74.8  1.1  2h37:40 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-1 loc
27817 bro        20   0 1532M 1435M  269M S 72.1  1.1  2h33:42 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-4 loc
26677 bro        20   0 1421M 1372M  269M R 70.5  1.1  2h40:42 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-6 loc
  622 bro        20   0 1165M 1108M  269M R 70.5  0.9  1h54:24 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-8 loc
  613 bro        20   0 1231M 1166M  269M R 69.9  0.9  1h55:45 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-10 lo
  621 bro        20   0 1242M 1177M  269M R 68.9  0.9  1h53:02 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-3 loc
 1175 bro        20   0  234M  178M 10776 S  2.2  0.1  2h42:07 /opt/bro/bin/bro -U .status -p broctl -p broctl-live -p local -p WIN_EXT_PXY_1 local.b


-Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161028/de57c224/attachment.html