[Bro] Worker OOM crashes
Azoff, Justin S
jazoff at illinois.edu
Fri Oct 28 09:05:30 PDT 2016
Yes.. there seems to be an issue where something goes wrong and tries to allocate too much. I don't think it has anything to do with 4G
Can you apply this patch and see if the next time it happens you get a core dump + backtrace?
diff --git a/src/util.cc b/src/util.cc
index acfcb19..661f8ac 100644
--- a/src/util.cc
+++ b/src/util.cc
@@ -1624,7 +1624,7 @@ extern "C" void out_of_memory(const char* where)
if ( reporter )
// Guess that might fail here if memory is really tight ...
- reporter->FatalError("out of memory in %s.\n", where);
+ reporter->FatalErrorWithCore("out of memory in %s.\n", where);
abort();
}
--
- Justin Azoff
> On Oct 28, 2016, at 11:43 AM, Dave Crawford <bro at pingtrip.com> wrote:
>
> Has anyone experienced this crash scenario that I’m in the process of debugging? This just started in the last couple of days on a cluster that has been in production for just shy of two years without issue.
>
> tcmalloc: large alloc 18446744072956297216 bytes == (nil) @ 0x7f08e85b5dcb 0x7f08e85b5f1b 0x7f08e85b6965 0x7f08e85e89c5 0x7f1fff 0x867016 0x86781e 0x7f1a96 0x7f1db6 0x7f121a 0x7efbbe 0x7ed334 0x866eb5 0x56441c 0x600327 0x60124e 0x5cfa14 0x837db3 0x5cfecf 0x52ecb0 0x7f08e759fb45 0x5373ad (nil) out of memory in new.
> 1477631586.955885 fatal error: out of memory in new.
>
> The server itself still has some ceiling height:
>
> $ free -h
> total used free shared buffers cached
> Mem: 126G 23G 102G 16M 67M 5.1G
> -/+ buffers/cache: 18G 107G
> Swap: 33G 0B 33G
>
> But I’m wondering if the processes themselves are hitting a maximum size for memory allocation, as some are nearing 4GB.
>
> PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
> 24925 bro 20 0 3964M 3262M 269M S 78.1 2.5 2h47:55 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-5 loc
> 618 bro 20 0 1217M 1160M 269M S 77.0 0.9 1h57:12 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-2 loc
> 26073 bro 20 0 3989M 3156M 269M S 76.4 2.4 2h41:38 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-7 loc
> 23938 bro 20 0 2610M 2318M 269M R 75.9 1.8 2h45:00 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-9 loc
> 27816 bro 20 0 1554M 1455M 269M S 74.8 1.1 2h37:40 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-1 loc
> 27817 bro 20 0 1532M 1435M 269M S 72.1 1.1 2h33:42 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-4 loc
> 26677 bro 20 0 1421M 1372M 269M R 70.5 1.1 2h40:42 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-6 loc
> 622 bro 20 0 1165M 1108M 269M R 70.5 0.9 1h54:24 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-8 loc
> 613 bro 20 0 1231M 1166M 269M R 69.9 0.9 1h55:45 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-10 lo
> 621 bro 20 0 1242M 1177M 269M R 68.9 0.9 1h53:02 /opt/bro/bin/bro -i eth6 -U .status -p broctl -p broctl-live -p local -p WIN_EXT-3 loc
> 1175 bro 20 0 234M 178M 10776 S 2.2 0.1 2h42:07 /opt/bro/bin/bro -U .status -p broctl -p broctl-live -p local -p WIN_EXT_PXY_1 local.b
>
>
> -Dave
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
More information about the Bro
mailing list