<div dir="ltr">Hi<div><br></div><div>Try using the None writer instead of the ASCII one.</div><div>In local.bro add :</div><div>redef Log::default_writer=Log::WRITER_NONE;<br></div><div><br></div><div>If the logger instance still crashes then the issue is not related to an IO bottleneck.</div><div><br></div><div>B</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 24, 2019 at 7:49 PM Kayode Enwerem <<a href="mailto:Kayode_Enwerem@ao.uscourts.gov">Kayode_Enwerem@ao.uscourts.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Thanks for your response.<br>
<br>
I do see the following OOM message in my system logs on the logger process ID:<br>
Sep 23 18:48:00 kernel: Out of memory: Kill process 10439 (bro) score 787 or sacrifice child<br>
Sep 23 18:48:00 kernel: Killed process 10439 (bro), UID 0, total-vm:301983900kB, anon-rss:195261772kB, file-rss:2592kB, shmem-rss:0kB<br>
<br>
Wonder why its taking so much memory, I have 251G and 99G swap on this server.<br>
total used free shared buff/cache available<br>
Mem: 251G 66G 185G 4.2M 488M 184G<br>
Swap: 99G 1.1G 98G<br>
<br>
Below is the output of "broctl diag logger", ran after the logger crashed. <br>
<br>
[logger]<br>
<br>
No core file found.<br>
<br>
Bro 2.6.3<br>
Linux 3.10.0-1062.1.1.el7.x86_64<br>
<br>
Bro plugins:<br>
Bro::AF_Packet - Packet acquisition via AF_Packet (dynamic, version 1.4)<br>
<br>
==== No reporter.log<br>
<br>
==== stderr.log<br>
/usr/local/bro/share/broctl/scripts/run-bro: line 110: 10439 Killed nohup "$mybro" "$@"<br>
<br>
==== stdout.log<br>
max memory size (kbytes, -m) unlimited<br>
data seg size (kbytes, -d) unlimited<br>
virtual memory (kbytes, -v) unlimited<br>
core file size (blocks, -c) unlimited<br>
<br>
==== .cmdline<br>
-U .status -p broctl -p broctl-live -p local -p logger local.bro broctl base/frameworks/cluster broctl/auto<br>
<br>
==== .env_vars<br>
PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bro/bin<br>
BROPATH=/logs/bro/spool/installed-scripts-do-not-touch/site::/logs/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site<br>
CLUSTER_NODE=logger<br>
<br>
==== .status<br>
RUNNING [net_run]<br>
<br>
==== No prof.log<br>
<br>
==== No packet_filter.log<br>
<br>
==== No loaded_scripts.log<br>
<br>
Thoughts? Any suggestions.<br>
<br>
-----Original Message-----<br>
From: Vlad Grigorescu <<a href="mailto:vlad@es.net" target="_blank">vlad@es.net</a>> <br>
Sent: Monday, September 23, 2019 10:20 AM<br>
To: Kayode Enwerem <<a href="mailto:Kayode_Enwerem@ao.uscourts.gov" target="_blank">Kayode_Enwerem@ao.uscourts.gov</a>><br>
Cc: william de ping <<a href="mailto:bill.de.ping@gmail.com" target="_blank">bill.de.ping@gmail.com</a>>; <a href="mailto:zeek@zeek.org" target="_blank">zeek@zeek.org</a><br>
Subject: Re: [Zeek] Why does my logger keep crashing - bro version 2.6.3<br>
<br>
The logger is threaded, so seeing CPU > 100% is not necessarily a problem.<br>
<br>
Have you tried running "broctl diag logger" to see why the logger is crashing? Do you have any messages in your system logs about processing being killed for out of memory (OOM)?<br>
<br>
--Vlad<br>
<br>
On Mon, Sep 23, 2019 at 1:32 PM Kayode Enwerem <<a href="mailto:Kayode_Enwerem@ao.uscourts.gov" target="_blank">Kayode_Enwerem@ao.uscourts.gov</a>> wrote:<br>
><br>
> Thanks for your response. The CPU usage for the logger is at 311%. (look below).<br>
><br>
><br>
><br>
> broctl top<br>
><br>
> Name Type Host Pid VSize Rss Cpu Cmd<br>
><br>
> logger logger localhost 22867 12G 9G 311% bro<br>
><br>
><br>
><br>
> I wasn’t aware that you could set up multiple loggers, I tried checking the docs to see if that was an option. Does anyone know how to do this?<br>
><br>
><br>
><br>
> From: william de ping <<a href="mailto:bill.de.ping@gmail.com" target="_blank">bill.de.ping@gmail.com</a>><br>
> Sent: Sunday, September 22, 2019 6:42 AM<br>
> To: Kayode Enwerem <<a href="mailto:Kayode_Enwerem@ao.uscourts.gov" target="_blank">Kayode_Enwerem@ao.uscourts.gov</a>><br>
> Cc: <a href="mailto:zeek@zeek.org" target="_blank">zeek@zeek.org</a><br>
> Subject: Re: [Zeek] Why does my logger keep crashing - bro version <br>
> 2.6.3<br>
><br>
><br>
><br>
> Hi,<br>
><br>
><br>
><br>
> I would try to monitor the cpu \ mem usage of the logger instance.<br>
><br>
> Try running broctl top, my guess is that you will see that the logger process will have a very high cpu usage.<br>
><br>
><br>
><br>
> I know of an option to have multiple loggers but I am not sure how to set it up.<br>
><br>
><br>
><br>
> Are you writing to a file ?<br>
><br>
><br>
><br>
> B<br>
><br>
><br>
><br>
> On Thu, Sep 19, 2019 at 7:14 PM Kayode Enwerem <<a href="mailto:Kayode_Enwerem@ao.uscourts.gov" target="_blank">Kayode_Enwerem@ao.uscourts.gov</a>> wrote:<br>
><br>
> Hello,<br>
><br>
><br>
><br>
> Why does my logger keep crashing? Can someone please help me with this. I have provided some system information below:<br>
><br>
><br>
><br>
> I am running bro version 2.6.3<br>
><br>
><br>
><br>
> Output of broctl status. The logger is crashed but the manager, proxy and workers are still running.<br>
><br>
> broctl status<br>
><br>
> Name Type Host Status Pid Started<br>
><br>
> logger logger localhost crashed<br>
><br>
> manager manager localhost running 17356 09 Sep 15:42:24<br>
><br>
> proxy-1 proxy localhost running 17401 09 Sep 15:42:25<br>
><br>
> worker-1-1 worker localhost running 17573 09 Sep 15:42:27<br>
><br>
> worker-1-2 worker localhost running 17569 09 Sep 15:42:27<br>
><br>
> worker-1-3 worker localhost running 17572 09 Sep 15:42:27<br>
><br>
> worker-1-4 worker localhost running 17587 09 Sep 15:42:27<br>
><br>
> worker-1-5 worker localhost running 17619 09 Sep 15:42:27<br>
><br>
> worker-1-6 worker localhost running 17614 09 Sep 15:42:27<br>
><br>
> worker-1-7 worker localhost running 17625 09 Sep 15:42:27<br>
><br>
> worker-1-8 worker localhost running 17646 09 Sep 15:42:27<br>
><br>
> worker-1-9 worker localhost running 17671 09 Sep 15:42:27<br>
><br>
> worker-1-10 worker localhost running 17663 09 Sep 15:42:27<br>
><br>
> worker-1-11 worker localhost running 17679 09 Sep 15:42:27<br>
><br>
> worker-1-12 worker localhost running 17685 09 Sep 15:42:27<br>
><br>
> worker-1-13 worker localhost running 17698 09 Sep 15:42:27<br>
><br>
> worker-1-14 worker localhost running 17703 09 Sep 15:42:27<br>
><br>
> worker-1-15 worker localhost running 17710 09 Sep 15:42:27<br>
><br>
> worker-1-16 worker localhost running 17717 09 Sep 15:42:27<br>
><br>
> worker-1-17 worker localhost running 17720 09 Sep 15:42:27<br>
><br>
> worker-1-18 worker localhost running 17727 09 Sep 15:42:27<br>
><br>
> worker-1-19 worker localhost running 17728 09 Sep 15:42:27<br>
><br>
> worker-1-20 worker localhost running 17731 09 Sep 15:42:27<br>
><br>
><br>
><br>
> Here’s my node.cfg settings<br>
><br>
> [logger]<br>
><br>
> type=logger<br>
><br>
> host=localhost<br>
><br>
><br>
><br>
> [manager]<br>
><br>
> type=manager<br>
><br>
> host=localhost<br>
><br>
><br>
><br>
> [proxy-1]<br>
><br>
> type=proxy<br>
><br>
> host=localhost<br>
><br>
><br>
><br>
> [worker-1]<br>
><br>
> type=worker<br>
><br>
> host=localhost<br>
><br>
> interface=af_packet::ens2f0<br>
><br>
> lb_method=custom<br>
><br>
> #lb_method=pf_ring<br>
><br>
> lb_procs=20<br>
><br>
> pin_cpus=6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25<br>
><br>
> af_packet_fanout_id=25<br>
><br>
> af_packet_fanout_mode=AF_Packet::FANOUT_HASH<br>
><br>
><br>
><br>
> Heres more information on my CPU. 32 CPUs, model name – AMD, CPU max <br>
> MHz is 2800.0000<br>
><br>
> Architecture: x86_64<br>
><br>
> CPU op-mode(s): 32-bit, 64-bit<br>
><br>
> Byte Order: Little Endian<br>
><br>
> CPU(s): 32<br>
><br>
> On-line CPU(s) list: 0-31<br>
><br>
> Thread(s) per core: 2<br>
><br>
> Core(s) per socket: 8<br>
><br>
> Socket(s): 2<br>
><br>
> NUMA node(s): 4<br>
><br>
> Vendor ID: AuthenticAMD<br>
><br>
> CPU family: 21<br>
><br>
> Model: 2<br>
><br>
> Model name: AMD Opteron(tm) Processor 6386 SE<br>
><br>
> Stepping: 0<br>
><br>
> CPU MHz: 1960.000<br>
><br>
> CPU max MHz: 2800.0000<br>
><br>
> CPU min MHz: 1400.0000<br>
><br>
> BogoMIPS: 5585.93<br>
><br>
> Virtualization: AMD-V<br>
><br>
> L1d cache: 16K<br>
><br>
> L1i cache: 64K<br>
><br>
> L2 cache: 2048K<br>
><br>
> L3 cache: 6144K<br>
><br>
> NUMA node0 CPU(s): 0,2,4,6,8,10,12,14<br>
><br>
> NUMA node1 CPU(s): 16,18,20,22,24,26,28,30<br>
><br>
> NUMA node2 CPU(s): 1,3,5,7,9,11,13,15<br>
><br>
> NUMA node3 CPU(s): 17,19,21,23,25,27,29,31<br>
><br>
><br>
><br>
> Would also like to know how I can reduce my packet loss. Below is the <br>
> output of broctl netstats<br>
><br>
> broctl netstats<br>
><br>
> worker-1-1: 1568908277.861813 recvd=12248845422 dropped=5171188999 <br>
> link=17420313882<br>
><br>
> worker-1-2: 1568908298.313954 recvd=8636707266 dropped=971489 <br>
> link=8637678939<br>
><br>
> worker-1-3: 1568908278.425888 recvd=11684808853 dropped=5617381647 <br>
> link=17302473791<br>
><br>
> worker-1-4: 1568908285.731130 recvd=12567242226 dropped=4339688288 <br>
> link=16907212802<br>
><br>
> worker-1-5: 1568908298.363911 recvd=8620499351 dropped=24595149 <br>
> link=8645095758<br>
><br>
> worker-1-6: 1568908298.372892 recvd=8710565757 dropped=1731022 <br>
> link=8712297432<br>
><br>
> worker-1-7: 1568908298.266010 recvd=9065207444 dropped=53523232 <br>
> link=9118737229<br>
><br>
> worker-1-8: 1568908286.935607 recvd=11377790124 dropped=3680887247 <br>
> link=15058934491<br>
><br>
> worker-1-9: 1568908298.419657 recvd=8931903322 dropped=39696184 <br>
> link=8971604219<br>
><br>
> worker-1-10: 1568908298.478576 recvd=8842874030 dropped=2501252 <br>
> link=8845376352<br>
><br>
> worker-1-11: 1568908298.506649 recvd=8692769329 dropped=2253413 <br>
> link=8695025626<br>
><br>
> worker-1-12: 1568908298.520830 recvd=8749977028 dropped=2314733 <br>
> link=8752293714<br>
><br>
> worker-1-13: 1568908298.544573 recvd=9101243757 dropped=1779460 <br>
> link=9103025399<br>
><br>
> worker-1-14: 1568908291.370011 recvd=10876925726 dropped=775722632 <br>
> link=11652810353<br>
><br>
> worker-1-15: 1568908298.579721 recvd=8503097394 dropped=1420699 <br>
> link=8504520066<br>
><br>
> worker-1-16: 1568908298.594942 recvd=8515164266 dropped=1840977 <br>
> link=8517006779<br>
><br>
> worker-1-17: 1568908298.646966 recvd=10666567717 dropped=466489754 <br>
> link=11133059283<br>
><br>
> worker-1-18: 1568908298.671246 recvd=9023603573 dropped=2037607 <br>
> link=9025642263<br>
><br>
> worker-1-19: 1568908298.704675 recvd=8907784186 dropped=1164594 <br>
> link=8908950238<br>
><br>
> worker-1-20: 1568908298.718084 recvd=9140525444 dropped=2028593 <br>
> link=9142555259<br>
><br>
><br>
><br>
> Thanks,<br>
><br>
> _______________________________________________<br>
> Zeek mailing list<br>
> <a href="mailto:zeek@zeek.org" target="_blank">zeek@zeek.org</a><br>
> <a href="http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek" rel="noreferrer" target="_blank">http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek</a><br>
><br>
> _______________________________________________<br>
> Zeek mailing list<br>
> <a href="mailto:zeek@zeek.org" target="_blank">zeek@zeek.org</a><br>
> <a href="http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek" rel="noreferrer" target="_blank">http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek</a><br>
</blockquote></div>