[Zeek] Dropping packets

Tue Feb 18 05:41:30 PST 2020

Can’t clear the counters, I get an “Operation not permitted”: 

>             Lanai uptime (seconds):                59830

>          Counters uptime (seconds):                59830

>                    Net send KBytes:                    0

>                    Net recv KBytes:          16291892294

>                      Ethernet send:                    8

>                Ethernet Small recv:             12077187

>                  Ethernet Big recv:                    0

>                 Ethernet recv down:                    0

>              Ethernet recv overrun:                  236

>                      SNF send pkts:                    0

>                      SNF recv pkts:          21273184129

>                 SNF drop ring full:                73133

>                         Interrupts:               361990

>             Net bad PHY/CRC32 drop:                    8

>                  Net overflow drop:                10266

>                    Net Recv PAUSEs:                    0

>     Ethernet Multicast filter drop:               153131

>       Ethernet Unicast filter drop:            616646672

> Cannot clear counters: Operation not permitted

For some reason with the workers named as you suggested, everything starts up fine but after ~15 minutes, zeekctl status reports the workers as “stopped”, though all the processes are still running on them and the logger is still receiving data.  We reverted back to the worker-[1234] notation that we had before and they start and stay running.

Any time we try and pin_cpu’s the workers crash.  I was able to get things to start and stay running with 10 lb_procs each (unpinned), though I’m going to try and figure out how to get these counters to clear.

Note, we’ve disabled hyperthreading and included the environment variables that you suggested.  Those are the main changes that we’ve made so far.

From: Justin Azoff <justin at corelight.com> 
Sent: Monday, February 17, 2020 4:45 PM
To: Joseph Fischetti <Joseph.Fischetti at marist.edu>
Cc: zeek at zeek.org
Subject: Re: [Zeek] Dropping packets

[EXTERNAL EMAIL]

On Mon, Feb 17, 2020 at 2:31 PM Joseph Fischetti <Joseph.Fischetti at marist.edu <mailto:Joseph.Fischetti at marist.edu> > wrote:

Thanks for the response Justin.

I’ll start with disabling HT on the workers, and will reconfigure the workers with 8 threads each.  I’ll also pin the processes.

The myricom configuration you quoted included environment variables:

env_vars=SNF_APP_ID=1,SNF_DATARING_SIZE=16384MB,SNF_DESCRING_SIZE=4096MB

What are those for/where can I find documentation/Should I include any of those?

It's in the myricom documentation.. searching for those settings should find you a PDF of it.

You also said the packets dropped are inclusive for the entire worker, but that the packets recvd are per thread:

worker-1-1: 1581949346.194441 recvd=2178149468 dropped=2260820124 link=15063051356

worker-1-2: 1581949346.194473 recvd=274557259 dropped=2260820124 link=13159459147

worker-1-3: 1581949346.168558 recvd=1888926901 dropped=2260820124 link=14773828789

worker-1-4: 1581949346.081130 recvd=2110377092 dropped=2260820124 link=14995278980

worker-1-5: 1581949346.234478 recvd=1032618510 dropped=2260820124 link=13917520398

Those add up to ~7.5B recvd with 2.2B dropped.  That’s 30%, but your script said 3.  Am I looking at some numbers wrong?

Oh, yes!  I forgot one another detail that is specific to the myricom nics.  The link and drop counters don't reset when you restart zeek... possibly this could be worked around in the plugin to snapshot them at startup and report the differences instead of the absolute numbers the card reports...

Try running this:

zeekctl stop

zeekctl exec /opt/snf/bin/myri_counters -c -p 0 >/dev/null

zeekctl exec /opt/snf/bin/myri_counters -c -p 1 >/dev/null 

zeekctl start

that will clear the counters and give you clean numbers.

-- 

Justin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200218/9c00fe7f/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5561 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200218/9c00fe7f/attachment-0001.bin 