<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Looks like setting up 2 loggers resolved the issue of my logger crashing but my Dropped packets are pretty high on my workers. Can someone assist me with how I can reduce my dropped packets.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">cat capture_loss.log<o:p></o:p></p>
<p class="MsoNormal">#separator \x09<o:p></o:p></p>
<p class="MsoNormal">#set_separator ,<o:p></o:p></p>
<p class="MsoNormal">#empty_field (empty)<o:p></o:p></p>
<p class="MsoNormal">#unset_field -<o:p></o:p></p>
<p class="MsoNormal">#path capture_loss<o:p></o:p></p>
<p class="MsoNormal">#open 2019-09-27-12-05-05<o:p></o:p></p>
<p class="MsoNormal">#fields ts ts_delta peer gaps acks percent_lost<o:p></o:p></p>
<p class="MsoNormal">#types time interval string count count double<o:p></o:p></p>
<p class="MsoNormal">1569600304.774215 900.000013 worker-1-1 126463 3246542 3.895314<o:p></o:p></p>
<p class="MsoNormal">1569600304.783703 900.000064 worker-1-3 106904 4465333 2.394088<o:p></o:p></p>
<p class="MsoNormal">1569600304.785983 900.000212 worker-1-11 123729 3768503 3.28324<o:p></o:p></p>
<p class="MsoNormal">1569600304.802244 900.000098 worker-1-14 144154 3584013 4.022139<o:p></o:p></p>
<p class="MsoNormal">1569600304.823378 900.000095 worker-1-18 137507 3503583 3.924754<o:p></o:p></p>
<p class="MsoNormal">1569600304.892559 900.000470 worker-1-13 148904 3448544 4.31788<o:p></o:p></p>
<p class="MsoNormal">1569600305.010986 900.000030 worker-1-8 174213 3409819 5.109157<o:p></o:p></p>
<p class="MsoNormal">1569600305.938686 901.043465 worker-1-15 509268 1072199 47.497526<o:p></o:p></p>
<p class="MsoNormal">1569600304.806850 900.000047 worker-1-22 591232 1234893 47.877185<o:p></o:p></p>
<p class="MsoNormal">1569601204.762382 900.000786 worker-1-16 120086 4491072 2.673883<o:p></o:p></p>
<p class="MsoNormal">1569601204.774220 900.000005 worker-1-1 127257 3461349 3.676515<o:p></o:p></p>
<p class="MsoNormal">1569601204.802447 900.000203 worker-1-14 125481 3171663 3.956316<o:p></o:p></p>
<p class="MsoNormal">1569601204.884438 900.000029 worker-1-19 125037 3566663 3.505714<o:p></o:p></p>
<p class="MsoNormal">1569601204.891746 900.000015 worker-1-23 120553 3078889 3.915471<o:p></o:p></p>
<p class="MsoNormal">1569601205.110098 900.000139 worker-1-10 108016 3442813 3.137434<o:p></o:p></p>
<p class="MsoNormal">1569601205.938906 900.000220 worker-1-15 565536 1156759 48.8897<o:p></o:p></p>
<p class="MsoNormal">1569601218.120290 900.000047 worker-1-6 456312 753749 60.538986<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Below are some of my settings:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have 23 workers defined and I pinned CPU.<o:p></o:p></p>
<p class="MsoNormal">[worker-1]<o:p></o:p></p>
<p class="MsoNormal">type=worker<o:p></o:p></p>
<p class="MsoNormal">host=localhost<o:p></o:p></p>
<p class="MsoNormal">interface=af_packet::ens2f0<o:p></o:p></p>
<p class="MsoNormal">lb_method=custom<o:p></o:p></p>
<p class="MsoNormal">#lb_method=pf_ring<o:p></o:p></p>
<p class="MsoNormal">lb_procs=23<o:p></o:p></p>
<p class="MsoNormal">pin_cpus=5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27<o:p></o:p></p>
<p class="MsoNormal">af_packet_fanout_id=25<o:p></o:p></p>
<p class="MsoNormal">af_packet_fanout_mode=AF_Packet::FANOUT_HASH<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Can someone assist me with this.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>From:</b> william de ping <bill.de.ping@gmail.com> <br>
<b>Sent:</b> Wednesday, September 25, 2019 4:00 AM<br>
<b>To:</b> Kayode Enwerem <Kayode_Enwerem@ao.uscourts.gov><br>
<b>Cc:</b> zeek@zeek.org<br>
<b>Subject:</b> Re: [Zeek] Why does my logger keep crashing - bro version 2.6.3<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Try using the None writer instead of the ASCII one.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">In local.bro add :<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">redef Log::default_writer=Log::WRITER_NONE;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">If the logger instance still crashes then the issue is not related to an IO bottleneck.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">B<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Sep 24, 2019 at 7:49 PM Kayode Enwerem <<a href="mailto:Kayode_Enwerem@ao.uscourts.gov">Kayode_Enwerem@ao.uscourts.gov</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Thanks for your response.<br>
<br>
I do see the following OOM message in my system logs on the logger process ID:<br>
Sep 23 18:48:00 kernel: Out of memory: Kill process 10439 (bro) score 787 or sacrifice child<br>
Sep 23 18:48:00 kernel: Killed process 10439 (bro), UID 0, total-vm:301983900kB, anon-rss:195261772kB, file-rss:2592kB, shmem-rss:0kB<br>
<br>
Wonder why its taking so much memory, I have 251G and 99G swap on this server.<br>
total used free shared buff/cache available<br>
Mem: 251G 66G 185G 4.2M 488M 184G<br>
Swap: 99G 1.1G 98G<br>
<br>
Below is the output of "broctl diag logger", ran after the logger crashed. <br>
<br>
[logger]<br>
<br>
No core file found.<br>
<br>
Bro 2.6.3<br>
Linux 3.10.0-1062.1.1.el7.x86_64<br>
<br>
Bro plugins:<br>
Bro::AF_Packet - Packet acquisition via AF_Packet (dynamic, version 1.4)<br>
<br>
==== No reporter.log<br>
<br>
==== stderr.log<br>
/usr/local/bro/share/broctl/scripts/run-bro: line 110: 10439 Killed nohup "$mybro" "$@"<br>
<br>
==== stdout.log<br>
max memory size (kbytes, -m) unlimited<br>
data seg size (kbytes, -d) unlimited<br>
virtual memory (kbytes, -v) unlimited<br>
core file size (blocks, -c) unlimited<br>
<br>
==== .cmdline<br>
-U .status -p broctl -p broctl-live -p local -p logger local.bro broctl base/frameworks/cluster broctl/auto<br>
<br>
==== .env_vars<br>
PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bro/bin<br>
BROPATH=/logs/bro/spool/installed-scripts-do-not-touch/site::/logs/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site<br>
CLUSTER_NODE=logger<br>
<br>
==== .status<br>
RUNNING [net_run]<br>
<br>
==== No prof.log<br>
<br>
==== No packet_filter.log<br>
<br>
==== No loaded_scripts.log<br>
<br>
Thoughts? Any suggestions.<br>
<br>
-----Original Message-----<br>
From: Vlad Grigorescu <<a href="mailto:vlad@es.net" target="_blank">vlad@es.net</a>>
<br>
Sent: Monday, September 23, 2019 10:20 AM<br>
To: Kayode Enwerem <<a href="mailto:Kayode_Enwerem@ao.uscourts.gov" target="_blank">Kayode_Enwerem@ao.uscourts.gov</a>><br>
Cc: william de ping <<a href="mailto:bill.de.ping@gmail.com" target="_blank">bill.de.ping@gmail.com</a>>;
<a href="mailto:zeek@zeek.org" target="_blank">zeek@zeek.org</a><br>
Subject: Re: [Zeek] Why does my logger keep crashing - bro version 2.6.3<br>
<br>
The logger is threaded, so seeing CPU > 100% is not necessarily a problem.<br>
<br>
Have you tried running "broctl diag logger" to see why the logger is crashing? Do you have any messages in your system logs about processing being killed for out of memory (OOM)?<br>
<br>
--Vlad<br>
<br>
On Mon, Sep 23, 2019 at 1:32 PM Kayode Enwerem <<a href="mailto:Kayode_Enwerem@ao.uscourts.gov" target="_blank">Kayode_Enwerem@ao.uscourts.gov</a>> wrote:<br>
><br>
> Thanks for your response. The CPU usage for the logger is at 311%. (look below).<br>
><br>
><br>
><br>
> broctl top<br>
><br>
> Name Type Host Pid VSize Rss Cpu Cmd<br>
><br>
> logger logger localhost 22867 12G 9G 311% bro<br>
><br>
><br>
><br>
> I wasn’t aware that you could set up multiple loggers, I tried checking the docs to see if that was an option. Does anyone know how to do this?<br>
><br>
><br>
><br>
> From: william de ping <<a href="mailto:bill.de.ping@gmail.com" target="_blank">bill.de.ping@gmail.com</a>><br>
> Sent: Sunday, September 22, 2019 6:42 AM<br>
> To: Kayode Enwerem <<a href="mailto:Kayode_Enwerem@ao.uscourts.gov" target="_blank">Kayode_Enwerem@ao.uscourts.gov</a>><br>
> Cc: <a href="mailto:zeek@zeek.org" target="_blank">zeek@zeek.org</a><br>
> Subject: Re: [Zeek] Why does my logger keep crashing - bro version <br>
> 2.6.3<br>
><br>
><br>
><br>
> Hi,<br>
><br>
><br>
><br>
> I would try to monitor the cpu \ mem usage of the logger instance.<br>
><br>
> Try running broctl top, my guess is that you will see that the logger process will have a very high cpu usage.<br>
><br>
><br>
><br>
> I know of an option to have multiple loggers but I am not sure how to set it up.<br>
><br>
><br>
><br>
> Are you writing to a file ?<br>
><br>
><br>
><br>
> B<br>
><br>
><br>
><br>
> On Thu, Sep 19, 2019 at 7:14 PM Kayode Enwerem <<a href="mailto:Kayode_Enwerem@ao.uscourts.gov" target="_blank">Kayode_Enwerem@ao.uscourts.gov</a>> wrote:<br>
><br>
> Hello,<br>
><br>
><br>
><br>
> Why does my logger keep crashing? Can someone please help me with this. I have provided some system information below:<br>
><br>
><br>
><br>
> I am running bro version 2.6.3<br>
><br>
><br>
><br>
> Output of broctl status. The logger is crashed but the manager, proxy and workers are still running.<br>
><br>
> broctl status<br>
><br>
> Name Type Host Status Pid Started<br>
><br>
> logger logger localhost crashed<br>
><br>
> manager manager localhost running 17356 09 Sep 15:42:24<br>
><br>
> proxy-1 proxy localhost running 17401 09 Sep 15:42:25<br>
><br>
> worker-1-1 worker localhost running 17573 09 Sep 15:42:27<br>
><br>
> worker-1-2 worker localhost running 17569 09 Sep 15:42:27<br>
><br>
> worker-1-3 worker localhost running 17572 09 Sep 15:42:27<br>
><br>
> worker-1-4 worker localhost running 17587 09 Sep 15:42:27<br>
><br>
> worker-1-5 worker localhost running 17619 09 Sep 15:42:27<br>
><br>
> worker-1-6 worker localhost running 17614 09 Sep 15:42:27<br>
><br>
> worker-1-7 worker localhost running 17625 09 Sep 15:42:27<br>
><br>
> worker-1-8 worker localhost running 17646 09 Sep 15:42:27<br>
><br>
> worker-1-9 worker localhost running 17671 09 Sep 15:42:27<br>
><br>
> worker-1-10 worker localhost running 17663 09 Sep 15:42:27<br>
><br>
> worker-1-11 worker localhost running 17679 09 Sep 15:42:27<br>
><br>
> worker-1-12 worker localhost running 17685 09 Sep 15:42:27<br>
><br>
> worker-1-13 worker localhost running 17698 09 Sep 15:42:27<br>
><br>
> worker-1-14 worker localhost running 17703 09 Sep 15:42:27<br>
><br>
> worker-1-15 worker localhost running 17710 09 Sep 15:42:27<br>
><br>
> worker-1-16 worker localhost running 17717 09 Sep 15:42:27<br>
><br>
> worker-1-17 worker localhost running 17720 09 Sep 15:42:27<br>
><br>
> worker-1-18 worker localhost running 17727 09 Sep 15:42:27<br>
><br>
> worker-1-19 worker localhost running 17728 09 Sep 15:42:27<br>
><br>
> worker-1-20 worker localhost running 17731 09 Sep 15:42:27<br>
><br>
><br>
><br>
> Here’s my node.cfg settings<br>
><br>
> [logger]<br>
><br>
> type=logger<br>
><br>
> host=localhost<br>
><br>
><br>
><br>
> [manager]<br>
><br>
> type=manager<br>
><br>
> host=localhost<br>
><br>
><br>
><br>
> [proxy-1]<br>
><br>
> type=proxy<br>
><br>
> host=localhost<br>
><br>
><br>
><br>
> [worker-1]<br>
><br>
> type=worker<br>
><br>
> host=localhost<br>
><br>
> interface=af_packet::ens2f0<br>
><br>
> lb_method=custom<br>
><br>
> #lb_method=pf_ring<br>
><br>
> lb_procs=20<br>
><br>
> pin_cpus=6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25<br>
><br>
> af_packet_fanout_id=25<br>
><br>
> af_packet_fanout_mode=AF_Packet::FANOUT_HASH<br>
><br>
><br>
><br>
> Heres more information on my CPU. 32 CPUs, model name – AMD, CPU max <br>
> MHz is 2800.0000<br>
><br>
> Architecture: x86_64<br>
><br>
> CPU op-mode(s): 32-bit, 64-bit<br>
><br>
> Byte Order: Little Endian<br>
><br>
> CPU(s): 32<br>
><br>
> On-line CPU(s) list: 0-31<br>
><br>
> Thread(s) per core: 2<br>
><br>
> Core(s) per socket: 8<br>
><br>
> Socket(s): 2<br>
><br>
> NUMA node(s): 4<br>
><br>
> Vendor ID: AuthenticAMD<br>
><br>
> CPU family: 21<br>
><br>
> Model: 2<br>
><br>
> Model name: AMD Opteron(tm) Processor 6386 SE<br>
><br>
> Stepping: 0<br>
><br>
> CPU MHz: 1960.000<br>
><br>
> CPU max MHz: 2800.0000<br>
><br>
> CPU min MHz: 1400.0000<br>
><br>
> BogoMIPS: 5585.93<br>
><br>
> Virtualization: AMD-V<br>
><br>
> L1d cache: 16K<br>
><br>
> L1i cache: 64K<br>
><br>
> L2 cache: 2048K<br>
><br>
> L3 cache: 6144K<br>
><br>
> NUMA node0 CPU(s): 0,2,4,6,8,10,12,14<br>
><br>
> NUMA node1 CPU(s): 16,18,20,22,24,26,28,30<br>
><br>
> NUMA node2 CPU(s): 1,3,5,7,9,11,13,15<br>
><br>
> NUMA node3 CPU(s): 17,19,21,23,25,27,29,31<br>
><br>
><br>
><br>
> Would also like to know how I can reduce my packet loss. Below is the <br>
> output of broctl netstats<br>
><br>
> broctl netstats<br>
><br>
> worker-1-1: 1568908277.861813 recvd=12248845422 dropped=5171188999 <br>
> link=17420313882<br>
><br>
> worker-1-2: 1568908298.313954 recvd=8636707266 dropped=971489 <br>
> link=8637678939<br>
><br>
> worker-1-3: 1568908278.425888 recvd=11684808853 dropped=5617381647 <br>
> link=17302473791<br>
><br>
> worker-1-4: 1568908285.731130 recvd=12567242226 dropped=4339688288 <br>
> link=16907212802<br>
><br>
> worker-1-5: 1568908298.363911 recvd=8620499351 dropped=24595149 <br>
> link=8645095758<br>
><br>
> worker-1-6: 1568908298.372892 recvd=8710565757 dropped=1731022 <br>
> link=8712297432<br>
><br>
> worker-1-7: 1568908298.266010 recvd=9065207444 dropped=53523232 <br>
> link=9118737229<br>
><br>
> worker-1-8: 1568908286.935607 recvd=11377790124 dropped=3680887247 <br>
> link=15058934491<br>
><br>
> worker-1-9: 1568908298.419657 recvd=8931903322 dropped=39696184 <br>
> link=8971604219<br>
><br>
> worker-1-10: 1568908298.478576 recvd=8842874030 dropped=2501252 <br>
> link=8845376352<br>
><br>
> worker-1-11: 1568908298.506649 recvd=8692769329 dropped=2253413 <br>
> link=8695025626<br>
><br>
> worker-1-12: 1568908298.520830 recvd=8749977028 dropped=2314733 <br>
> link=8752293714<br>
><br>
> worker-1-13: 1568908298.544573 recvd=9101243757 dropped=1779460 <br>
> link=9103025399<br>
><br>
> worker-1-14: 1568908291.370011 recvd=10876925726 dropped=775722632 <br>
> link=11652810353<br>
><br>
> worker-1-15: 1568908298.579721 recvd=8503097394 dropped=1420699 <br>
> link=8504520066<br>
><br>
> worker-1-16: 1568908298.594942 recvd=8515164266 dropped=1840977 <br>
> link=8517006779<br>
><br>
> worker-1-17: 1568908298.646966 recvd=10666567717 dropped=466489754 <br>
> link=11133059283<br>
><br>
> worker-1-18: 1568908298.671246 recvd=9023603573 dropped=2037607 <br>
> link=9025642263<br>
><br>
> worker-1-19: 1568908298.704675 recvd=8907784186 dropped=1164594 <br>
> link=8908950238<br>
><br>
> worker-1-20: 1568908298.718084 recvd=9140525444 dropped=2028593 <br>
> link=9142555259<br>
><br>
><br>
><br>
> Thanks,<br>
><br>
> _______________________________________________<br>
> Zeek mailing list<br>
> <a href="mailto:zeek@zeek.org" target="_blank">zeek@zeek.org</a><br>
> <a href="http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek" target="_blank">
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek</a><br>
><br>
> _______________________________________________<br>
> Zeek mailing list<br>
> <a href="mailto:zeek@zeek.org" target="_blank">zeek@zeek.org</a><br>
> <a href="http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek" target="_blank">
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</body>
</html>