[Zeek] Dropping packets

[Joseph Fischetti]

Hmm.. are you running zeek as root or a regular user? may need to use sudo, or tweak the permissions on the /dev/myri.. (i think?) files.

-          We’re running zeek as an unprivileged user.  I was able to go onto each worker as root and issue the commands to clear the counters.

 

That doesn't make a lot of sense.. It's just the name, it doesn't affect anything else... Maybe it was just a coincidence?

-          When we named the workers “worker-1-A”, etc, the processes that were started up were “worker-1-A-1” through “worker-1-A-N”.  Is it possible that zeekctl doesn’t parse something correctly when it checks to make sure the processes are running?  Zeekctl stop failed to stop the workers, zeeksctl status showed them as stopped.  They were still running and the logger was still working.  The only way to kill them was to do a “killall zeek” from the workers.

 

That also doesn't make a lot of sense.   Pinning the workers just causes them to be started via cpuset locked to a core, everything else is the same.

How exactly are the workers crashing?  Are you getting a crash report from zeekctl? or in dmesg?

-          No crash reports that I recall.  I can try and force it to happen again tomorrow.  I’d like to let things go for 24 hours and see where we end up.  We’ve been running for 5 hours and our stats currently looks like this [1].  Note, I modified the script that you provided to work with more than 9 threads.  (host = re.sub(r'\-[0-9]*$','', node))

-          We’re currently running 10 threads per worker *unpinned*

-          Medium load average on both boxes is around 6

-          Memory usage on both boxes is around 75G.

-          I’m going to find out if there’s something we can do in the Aristas so they send some more traffic to the secondary interface.  I need to find out how they’re configured and why.

 

 

[1]

bro at bro-master-1:~$ zeekctl netstats  | ~/calcDrop.py

 

Warning: ZeekControl plugin uses legacy BroControl API. Use

'import ZeekControl.plugin' instead of 'import BroControl.plugin'

 

worker-1 dropped=9051079 rx=6537521948 0.14%

worker-2 dropped=0 rx=6766088683 0.00%

worker-3 dropped=0 rx=437828006 0.00%

worker-4 dropped=0 rx=466874444 0.00%

 

Totals dropped=9051079 rx=14208313081 0.06%

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200218/d39f0f94/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5561 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200218/d39f0f94/attachment-0001.bin