[Bro] Worker OOM crashes

Azoff, Justin S jazoff at illinois.edu
Fri Oct 28 09:05:30 PDT 2016


Yes.. there seems to be an issue where something goes wrong and tries to allocate too much.  I don't think it has anything to do with 4G

Can you apply this patch and see if the next time it happens you get a core dump + backtrace?

diff --git a/src/util.cc b/src/util.cc
index acfcb19..661f8ac 100644
--- a/src/util.cc
+++ b/src/util.cc
@@ -1624,7 +1624,7 @@ extern "C" void out_of_memory(const char* where)

        if ( reporter )
                // Guess that might fail here if memory is really tight ...
-               reporter->FatalError("out of memory in %s.\n", where);
+               reporter->FatalErrorWithCore("out of memory in %s.\n", where);

        abort();
        }


-- 
- Justin Azoff

> On Oct 28, 2016, at 11:43 AM, Dave Crawford <bro at pingtrip.com> wrote:
> 
> Has anyone experienced this crash scenario that I’m in the process of debugging? This just started in the last couple of days on a cluster that has been in production for just shy of two years without issue.
> 
> tcmalloc: large alloc 18446744072956297216 bytes == (nil) @  0x7f08e85b5dcb 0x7f08e85b5f1b 0x7f08e85b6965 0x7f08e85e89c5 0x7f1fff 0x867016 0x86781e 0x7f1a96 0x7f1db6 0x7f121a 0x7efbbe 0x7ed334 0x866eb5 0x56441c 0x600327 0x60124e 0x5cfa14 0x837db3 0x5cfecf 0x52ecb0 0x7f08e759fb45 0x5373ad (nil) out of memory in new.
> 1477631586.955885 fatal error: out of memory in new.
> 
> The server itself still has some ceiling height:
> 
> $ free -h
>              total       used       free     shared    buffers     cached
> Mem:          126G        23G       102G        16M        67M       5.1G
> -/+ buffers/cache:        18G       107G
> Swap:          33G         0B        33G
> 
> But I’m wondering if the processes themselves are hitting a maximum size for memory allocation, as some are nearing 4GB.
> 
>   PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
> 24925 bro        20   0 3964M 3262M  269M S 78.1  2.5  2h47:55 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-5 loc
>   618 bro        20   0 1217M 1160M  269M S 77.0  0.9  1h57:12 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-2 loc
> 26073 bro        20   0 3989M 3156M  269M S 76.4  2.4  2h41:38 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-7 loc
> 23938 bro        20   0 2610M 2318M  269M R 75.9  1.8  2h45:00 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-9 loc
> 27816 bro        20   0 1554M 1455M  269M S 74.8  1.1  2h37:40 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-1 loc
> 27817 bro        20   0 1532M 1435M  269M S 72.1  1.1  2h33:42 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-4 loc
> 26677 bro        20   0 1421M 1372M  269M R 70.5  1.1  2h40:42 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-6 loc
>   622 bro        20   0 1165M 1108M  269M R 70.5  0.9  1h54:24 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-8 loc
>   613 bro        20   0 1231M 1166M  269M R 69.9  0.9  1h55:45 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-10 lo
>   621 bro        20   0 1242M 1177M  269M R 68.9  0.9  1h53:02 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-3 loc
>  1175 bro        20   0  234M  178M 10776 S  2.2  0.1  2h42:07 /opt/bro/bin/bro -U .status -p broctl -p broctl-live -p local -p WIN_EXT_PXY_1 local.b
> 
> 
> -Dave
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




More information about the Bro mailing list