From clopmz at outlook.com  Sun Feb  2 00:22:46 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Sun, 2 Feb 2020 08:22:46 +0000
Subject: [Zeek] Error when trying to use netmap
Message-ID: <F94200AE-5F0A-4B01-BF7E-70DE200DD959@outlook.com>

Hi all,

When I try to use netmap under a FreeBSD 12.1 host and Zeek 3.0.1 (installed from source), returns the following error:

root at fbsdzeek01:/nsm/zeek/spool/worker-1 # /opt/zeek/bin/zeek -i netmap::vtnet2
624.635003 nm_open [920] NIOCREGIF failed: Invalid argument vtnet2
fatal error: problem with interface netmap::vtnet2 (Invalid argument)

bro-netmap plugin is installed. Any idea?
--
Regards,
C. L. Martinez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200202/d7f920c0/attachment.html 

From clopmz at outlook.com  Sun Feb  2 00:45:29 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Sun, 2 Feb 2020 08:45:29 +0000
Subject: [Zeek] Error when trying to use netmap
In-Reply-To: <F94200AE-5F0A-4B01-BF7E-70DE200DD959@outlook.com>
References: <F94200AE-5F0A-4B01-BF7E-70DE200DD959@outlook.com>
Message-ID: <A9FED3ED-AF71-4B14-A27A-32FB4270CAE8@outlook.com>

Another test. I have tried to change network interface model to e1000, and error is the same.

--
Regards,
C. L. Martinez

From: Carlos Lopez <clopmz at outlook.com>
Date: Sunday, 2 February 2020 at 09:22
To: "zeek at zeek.org" <zeek at zeek.org>
Subject: Error when trying to use netmap

Hi all,

When I try to use netmap under a FreeBSD 12.1 host and Zeek 3.0.1 (installed from source), returns the following error:

root at fbsdzeek01:/nsm/zeek/spool/worker-1 # /opt/zeek/bin/zeek -i netmap::vtnet2
624.635003 nm_open [920] NIOCREGIF failed: Invalid argument vtnet2
fatal error: problem with interface netmap::vtnet2 (Invalid argument)

bro-netmap plugin is installed. Any idea?
--
Regards,
C. L. Martinez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200202/dbaf649f/attachment.html 

From akgraner at corelight.com  Mon Feb  3 07:17:14 2020
From: akgraner at corelight.com (Amber Graner)
Date: Mon, 3 Feb 2020 10:17:14 -0500
Subject: [Zeek] Agenda and Reminder - Zeek Community Call - Friday, 7 Feb 20
Message-ID: <CAJhOzuqpcZ4XJNVD2-PofM9NWe4hOhPXUSQ0+kP7_3=E7hYBFg@mail.gmail.com>

Hi all,

Just a quick reminder that the next Zeek Community call will be Friday, 7
Feb 2020 at 3:00-3:45pm ET/12-12:45pm PT.   The Calendar invite has also
been updated.

You can find out what it will be in your timezone at timeanddate.com -
https://www.timeanddate.com/worldclock/converter.html

AGENDA:
========

* Slack Channel
* Newsletter
* ZeekWeek 2020
* Ask The Zeeksperts
* Zeek Blog Posts

Please let me know if there is anything else that you'd like to see added
to the agenda.

JOINING THE CALL
===============

Join Zoom Meeting
https://corelight.zoom.us/j/898658920

Meeting ID: 898 658 920

One tap mobile
+16465588656,,898658920# US (New York)
+16699006833,,898658920# US (San Jose)

Dial by your location
+1 646 558 8656 US (New York)
+1 669 900 6833 US (San Jose)
877 853 5257 US Toll-free
888 475 4499 US Toll-free
Meeting ID: 898 658 920
Find your local number: https://corelight.zoom.us/u/acY5L1LN7

Please let me know if you have any questions.

Thanks,
~Amber
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200203/7166074a/attachment.html 

From monahbaki at gmail.com  Mon Feb  3 12:36:10 2020
From: monahbaki at gmail.com (Monah Baki)
Date: Mon, 3 Feb 2020 15:36:10 -0500
Subject: [Zeek] Detecting md5
Message-ID: <CALP3=x-yr65+Qo15HLsHYt27W+ib_s7VsUbjK1X8jTqDjcLW0w@mail.gmail.com>

Hi all,

Is there a way to detect certain md5 file hashes and send them to a
specific file.

Thanks
Monah
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200203/22821cf8/attachment.html 

From akgraner at corelight.com  Mon Feb  3 12:42:12 2020
From: akgraner at corelight.com (Amber Graner)
Date: Mon, 3 Feb 2020 15:42:12 -0500
Subject: [Zeek] Zeek Related Jobs or Events for Issue 2 of the Zeek
	Newsletter
Message-ID: <CAJhOzupNTXJJrXJqQnveO1yzzGm6iyV=BwXQDFp0eoovn7YnBw@mail.gmail.com>

Hi all,

I'm working on Issue 2 of the newsletter.

I'm looking for more information on Zeek related jobs and events that are
happening in the community.

If you know of any Zeek related jobs or events, please send the links to
news at zeek.org?

Thanks in advance!

With gratitude,
~Amber
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200203/4bef9f6d/attachment.html 

From johanna at corelight.com  Mon Feb  3 13:08:26 2020
From: johanna at corelight.com (Johanna Amann)
Date: Mon, 03 Feb 2020 13:08:26 -0800
Subject: [Zeek] Detecting md5
In-Reply-To: <CALP3=x-yr65+Qo15HLsHYt27W+ib_s7VsUbjK1X8jTqDjcLW0w@mail.gmail.com>
References: <CALP3=x-yr65+Qo15HLsHYt27W+ib_s7VsUbjK1X8jTqDjcLW0w@mail.gmail.com>
Message-ID: <30DD8DE5-12C9-43D3-8700-8A87FDE5F0AB@corelight.com>

On 3 Feb 2020, at 12:36, Monah Baki wrote:

> Hi all,
>
> Is there a way to detect certain md5 file hashes and send them to a
> specific file.

You can load the MD5 hashes into the intelligence framework. 
https://docs.zeek.org/en/stable/frameworks/intel.html

If you enable file-hashing (which is enabled by default in a cluster 
installation) and load your MD5 sums as Intel::FILE_HASH indicators, 
hits will be written into intel.log.

Johanna

From jacartwright at g.hmc.edu  Mon Feb  3 14:48:45 2020
From: jacartwright at g.hmc.edu (Jonah Cartwright)
Date: Mon, 3 Feb 2020 14:48:45 -0800
Subject: [Zeek] Signature for IoT Devices
Message-ID: <CABuu9kYLa-iEhrKuDFvRNUB5JguQBO9ccMR46him5MR5GbgV4Q@mail.gmail.com>

Hi Zeek Community,

I am working on a project to identify IoT devices on a network. We are
primarily working with the signatures framework. We would like to write
signatures for different device types (i.e. smart plug, smart speaker,
etc.). Does anyone have any advice on how to start going about this in
terms of unique identifiers or protocols these IoT devices may be using
that other devices may not use?

Thanks,
Jonah
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200203/955bdecb/attachment.html 

From richard at corelight.com  Mon Feb  3 15:27:38 2020
From: richard at corelight.com (Richard Bejtlich)
Date: Mon, 3 Feb 2020 18:27:38 -0500
Subject: [Zeek] Signature for IoT Devices
In-Reply-To: <CABuu9kYLa-iEhrKuDFvRNUB5JguQBO9ccMR46him5MR5GbgV4Q@mail.gmail.com>
References: <CABuu9kYLa-iEhrKuDFvRNUB5JguQBO9ccMR46him5MR5GbgV4Q@mail.gmail.com>
Message-ID: <CAOtSMjb2ORmQkLLZ52vQGCRi57vNUpKDCgsLAzocebQEb3TaFw@mail.gmail.com>

Just curious ? if you prefer signatures, why choose Zeek over Suricata?

Sincerely,

Richard

On Mon, Feb 3, 2020 at 5:51 PM Jonah Cartwright <jacartwright at g.hmc.edu>
wrote:

> Hi Zeek Community,
>
> I am working on a project to identify IoT devices on a network. We are
> primarily working with the signatures framework. We would like to write
> signatures for different device types (i.e. smart plug, smart speaker,
> etc.). Does anyone have any advice on how to start going about this in
> terms of unique identifiers or protocols these IoT devices may be using
> that other devices may not use?
>
> Thanks,
> Jonah
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

-- 
Richard Bejtlich
Principal Security Strategist, Corelight
https://corelight.blog/author/richardbejtlich/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200203/adfc6f16/attachment.html 

From jacartwright at g.hmc.edu  Mon Feb  3 15:39:04 2020
From: jacartwright at g.hmc.edu (Jonah Cartwright)
Date: Mon, 3 Feb 2020 15:39:04 -0800
Subject: [Zeek] Signature for IoT Devices
In-Reply-To: <CAOtSMjb2ORmQkLLZ52vQGCRi57vNUpKDCgsLAzocebQEb3TaFw@mail.gmail.com>
References: <CABuu9kYLa-iEhrKuDFvRNUB5JguQBO9ccMR46him5MR5GbgV4Q@mail.gmail.com>
	<CAOtSMjb2ORmQkLLZ52vQGCRi57vNUpKDCgsLAzocebQEb3TaFw@mail.gmail.com>
Message-ID: <CABuu9kZKbtXp+b5fOQhMtcPyVWymLx=S56r1G8JGQ-4OPi822g@mail.gmail.com>

Not any particular reason, we were asked to use Zeek for the project, and
figured signatures was the best method to use in Zeek.

On Mon, Feb 3, 2020 at 3:27 PM Richard Bejtlich <richard at corelight.com>
wrote:

> Just curious ? if you prefer signatures, why choose Zeek over Suricata?
>
> Sincerely,
>
> Richard
>
> On Mon, Feb 3, 2020 at 5:51 PM Jonah Cartwright <jacartwright at g.hmc.edu>
> wrote:
>
>> Hi Zeek Community,
>>
>> I am working on a project to identify IoT devices on a network. We are
>> primarily working with the signatures framework. We would like to write
>> signatures for different device types (i.e. smart plug, smart speaker,
>> etc.). Does anyone have any advice on how to start going about this in
>> terms of unique identifiers or protocols these IoT devices may be using
>> that other devices may not use?
>>
>> Thanks,
>> Jonah
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
> --
> Richard Bejtlich
> Principal Security Strategist, Corelight
> https://corelight.blog/author/richardbejtlich/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200203/8e636267/attachment.html 

From michalpurzynski1 at gmail.com  Mon Feb  3 15:45:04 2020
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Mon, 3 Feb 2020 15:45:04 -0800
Subject: [Zeek] Signature for IoT Devices
In-Reply-To: <CABuu9kZKbtXp+b5fOQhMtcPyVWymLx=S56r1G8JGQ-4OPi822g@mail.gmail.com>
References: <CABuu9kZKbtXp+b5fOQhMtcPyVWymLx=S56r1G8JGQ-4OPi822g@mail.gmail.com>
Message-ID: <4B134821-2D57-4657-88BE-A589D5CA2804@gmail.com>

It?s actually the other way. Signatures are the last use case for Zeek. Gathering metadata, writing scripts and writing protocol analyzers - that?s where Zeek shines.

Simple signatures with a way better support, shaped by a huge community that deals with signatures on a daily basis, is what Suricata does best.

> On Feb 3, 2020, at 3:41 PM, Jonah Cartwright <jacartwright at g.hmc.edu> wrote:
> 
> ?
> Not any particular reason, we were asked to use Zeek for the project, and figured signatures was the best method to use in Zeek.
> 
>> On Mon, Feb 3, 2020 at 3:27 PM Richard Bejtlich <richard at corelight.com> wrote:
>> Just curious ? if you prefer signatures, why choose Zeek over Suricata?
>> 
>> Sincerely,
>> 
>> Richard 
>> 
>>> On Mon, Feb 3, 2020 at 5:51 PM Jonah Cartwright <jacartwright at g.hmc.edu> wrote:
>>> Hi Zeek Community,
>>> 
>>> I am working on a project to identify IoT devices on a network. We are primarily working with the signatures framework. We would like to write signatures for different device types (i.e. smart plug, smart speaker, etc.). Does anyone have any advice on how to start going about this in terms of unique identifiers or protocols these IoT devices may be using that other devices may not use?
>>> 
>>> Thanks,
>>> Jonah
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>> -- 
>> Richard Bejtlich
>> Principal Security Strategist, Corelight
>> https://corelight.blog/author/richardbejtlich/
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200203/0b0095ee/attachment-0001.html 

From neslog at gmail.com  Mon Feb  3 15:54:50 2020
From: neslog at gmail.com (Neslog)
Date: Mon, 3 Feb 2020 18:54:50 -0500
Subject: [Zeek] Signature for IoT Devices
In-Reply-To: <CABuu9kZKbtXp+b5fOQhMtcPyVWymLx=S56r1G8JGQ-4OPi822g@mail.gmail.com>
References: <CABuu9kYLa-iEhrKuDFvRNUB5JguQBO9ccMR46him5MR5GbgV4Q@mail.gmail.com>
	<CAOtSMjb2ORmQkLLZ52vQGCRi57vNUpKDCgsLAzocebQEb3TaFw@mail.gmail.com>
	<CABuu9kZKbtXp+b5fOQhMtcPyVWymLx=S56r1G8JGQ-4OPi822g@mail.gmail.com>
Message-ID: <CALgfRG+medZp9f2+VMFHpxuGKv31adPaT2s32+GMPXPxd9nuag@mail.gmail.com>

I?d be happy to help. I don?t have any signatures off hand but happy to
analyze pcaps.

I have a few devices on home WiFi. I?ll dump some traffic and see what I
can come up with.

On Mon, Feb 3, 2020 at 6:41 PM Jonah Cartwright <jacartwright at g.hmc.edu>
wrote:

> Not any particular reason, we were asked to use Zeek for the project, and
> figured signatures was the best method to use in Zeek.
>
> On Mon, Feb 3, 2020 at 3:27 PM Richard Bejtlich <richard at corelight.com>
> wrote:
>
>> Just curious ? if you prefer signatures, why choose Zeek over Suricata?
>>
>> Sincerely,
>>
>> Richard
>>
>> On Mon, Feb 3, 2020 at 5:51 PM Jonah Cartwright <jacartwright at g.hmc.edu>
>> wrote:
>>
>>> Hi Zeek Community,
>>>
>>> I am working on a project to identify IoT devices on a network. We are
>>> primarily working with the signatures framework. We would like to write
>>> signatures for different device types (i.e. smart plug, smart speaker,
>>> etc.). Does anyone have any advice on how to start going about this in
>>> terms of unique identifiers or protocols these IoT devices may be using
>>> that other devices may not use?
>>>
>>> Thanks,
>>> Jonah
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>> --
>> Richard Bejtlich
>> Principal Security Strategist, Corelight
>> https://corelight.blog/author/richardbejtlich/
>>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200203/edfe6c3f/attachment.html 

From akgraner at corelight.com  Mon Feb  3 18:07:17 2020
From: akgraner at corelight.com (akgraner at corelight.com)
Date: Tue, 04 Feb 2020 02:07:17 +0000
Subject: [Zeek] Invitation: Ask The Zeeksperts - Aashish Sharma @ Thu Feb 13,
 2020 3:30pm - 4:30pm (EST) (zeek@zeek.org)
Message-ID: <000000000000d6c796059db67ff0@google.com>

You have been invited to the following event.

Title: Ask The Zeeksperts - Aashish Sharma
This is a bi-monthly call for the open-source Zeek (formerly Bro) community  
to interface directly with leading contributors to the open-source project  
and ask questions live to better understand, expand or troubleshoot  
deployments of the network security monitoring software. Registration Link  
-&nbsp;https://attendee.gotowebinar.com/register/7036660417675395595
When: Thu Feb 13, 2020 3:30pm ? 4:30pm Eastern Time - New York
Where: https://attendee.gotowebinar.com/register/7036660417675395595
Calendar: zeek at zeek.org
Who:
     * akgraner at corelight.com - organizer
     * Aashish Sharma
     * zeek at zeek.org

Event details:  
https://www.google.com/calendar/event?action=VIEW&eid=MDA5dmE2NnBkN2htZ3NvODVnZTJobXZsZHFfMjAyMDAyMTNUMjAzMDAwWiB6ZWVrQHplZWsub3Jn&tok=MjIjYWtncmFuZXJAY29yZWxpZ2h0LmNvbTgzMDY0MTIzNmY2NjYzZTEzNTQzZDBhMWYwODc5MmVmMWIzYTA0ZjE&ctz=America%2FNew_York&hl=en&es=0

Invitation from Google Calendar: https://www.google.com/calendar/

You are receiving this courtesy email at the account zeek at zeek.org because  
you are an attendee of this event.

To stop receiving future updates for this event, decline this event.  
Alternatively you can sign up for a Google account at  
https://www.google.com/calendar/ and control your notification settings for  
your entire calendar.

Forwarding this invitation could allow any recipient to send a response to  
the organizer and be added to the guest list, or invite others regardless  
of their own invitation status, or to modify your RSVP. Learn more at  
https://support.google.com/calendar/answer/37135#forwarding
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200204/aa73001d/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/calendar
Size: 2467 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200204/aa73001d/attachment-0002.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: invite.ics
Type: application/ics
Size: 2526 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200204/aa73001d/attachment-0003.bin 

From akgraner at corelight.com  Mon Feb  3 18:07:37 2020
From: akgraner at corelight.com (akgraner at corelight.com)
Date: Tue, 04 Feb 2020 02:07:37 +0000
Subject: [Zeek] Invitation: Ask The Zeeksperts @ Thu Feb 27,
 2020 3:30pm - 4:30pm (EST) (zeek@zeek.org)
Message-ID: <00000000000003cc60059db681ad@google.com>

You have been invited to the following event.

Title: Ask The Zeeksperts
This is a bi-monthly call for the open-source Zeek (formerly Bro) community  
to interface directly with leading contributors to the open-source project  
and ask questions live to better understand, expand or troubleshoot  
deployments of the network security monitoring software. Registration  
Link&nbsp;  
-&nbsp;https://attendee.gotowebinar.com/register/3466243796592822796
When: Thu Feb 27, 2020 3:30pm ? 4:30pm Eastern Time - New York
Where: https://attendee.gotowebinar.com/register/3466243796592822796
Calendar: zeek at zeek.org
Who:
     * akgraner at corelight.com - organizer
     * zeek at zeek.org

Event details:  
https://www.google.com/calendar/event?action=VIEW&eid=N2gzMTFzdnM1c2NlOWlnMjY3bmQ1ZThsOGNfMjAyMDAyMjdUMjAzMDAwWiB6ZWVrQHplZWsub3Jn&tok=MjIjYWtncmFuZXJAY29yZWxpZ2h0LmNvbWM0OWYxOGI3N2I3YTEyMDlhNGViMzc3M2RmN2E3MzI2ZTAzZjVkYTk&ctz=America%2FNew_York&hl=en&es=0

Invitation from Google Calendar: https://www.google.com/calendar/

You are receiving this courtesy email at the account zeek at zeek.org because  
you are an attendee of this event.

To stop receiving future updates for this event, decline this event.  
Alternatively you can sign up for a Google account at  
https://www.google.com/calendar/ and control your notification settings for  
your entire calendar.

Forwarding this invitation could allow any recipient to send a response to  
the organizer and be added to the guest list, or invite others regardless  
of their own invitation status, or to modify your RSVP. Learn more at  
https://support.google.com/calendar/answer/37135#forwarding
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200204/1b1e7158/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/calendar
Size: 2311 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200204/1b1e7158/attachment.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: invite.ics
Type: application/ics
Size: 2368 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200204/1b1e7158/attachment-0001.bin 

From Paul.Sibley at canarie.ca  Wed Feb  5 09:35:12 2020
From: Paul.Sibley at canarie.ca (Paul Sibley)
Date: Wed, 5 Feb 2020 17:35:12 +0000
Subject: [Zeek] "bro-cluster-in-a-box-setup" to
	"zeek-cluster-in-a-box-setup"?
Message-ID: <DM6PR12MB34983FF065BE194427DECA868B020@DM6PR12MB3498.namprd12.prod.outlook.com>

Hello Zeek Community,

 

I am working on a project where Zeek has been deployed in two phases.
During the first phase, some participants used
"https://github.com/ncsa/bro-cluster-in-a-box-setup" script to assist in,
and automate a lot of the installation process.

Since then we have entered the phase in our project where more participants
have been added, CentOS 8 is preferred, and we are using Zeek 3.0.1.

I wonder if any consideration, or work has been done, in updating the
bro-cluster-in-a-box script to work with the updated OS and Zeek version.
Any information would be appreciated.

 

Thanks in advance,

Paul Sibley

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200205/3949f5f0/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4909 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200205/3949f5f0/attachment.bin 

From justin at corelight.com  Wed Feb  5 12:48:00 2020
From: justin at corelight.com (Justin Azoff)
Date: Wed, 5 Feb 2020 15:48:00 -0500
Subject: [Zeek] "bro-cluster-in-a-box-setup" to
	"zeek-cluster-in-a-box-setup"?
In-Reply-To: <DM6PR12MB34983FF065BE194427DECA868B020@DM6PR12MB3498.namprd12.prod.outlook.com>
References: <DM6PR12MB34983FF065BE194427DECA868B020@DM6PR12MB3498.namprd12.prod.outlook.com>
Message-ID: <CAPfnCugKM6hnRFD_5BzOfS9hUSARvmJ21jd5y4okGgaKDNCW_A@mail.gmail.com>

Hi!

It shouldn't be that hard to update to 3.x..

- bro-pkg should be swapped out with the renamed zkg
- the python2 references can likely be changed to 3
- caf no longer needs to be installed separately
- geoip and databases needs to be swapped out with maxminddb versions,
might need a license
- probably worth it to switch to af_packet from pf_ring.. pf_ring was only
used initially to easily support capturing directly from both halves of a
tap, which might not be a requirement anymore.

My schedule is a bit crazy for the next week, but once I have some time to
work on it I should be able to get things updated pretty quickly.. There's
really not much to it.



On Wed, Feb 5, 2020 at 12:38 PM Paul Sibley <Paul.Sibley at canarie.ca> wrote:

> Hello Zeek Community,
>
>
>
> I am working on a project where Zeek has been deployed in two phases.
> During the first phase, some participants used ?
> https://github.com/ncsa/bro-cluster-in-a-box-setup? script to assist in,
> and automate a lot of the installation process.
>
> Since then we have entered the phase in our project where more
> participants have been added, CentOS 8 is preferred, and we are using Zeek
> 3.0.1.
>
> I wonder if any consideration, or work has been done, in updating the
> bro-cluster-in-a-box script to work with the updated OS and Zeek version.
> Any information would be appreciated.
>
>
>
> Thanks in advance,
>
> Paul Sibley
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200205/8c28d1bf/attachment.html 

From scwang+bro at sfu.ca  Wed Feb  5 13:55:44 2020
From: scwang+bro at sfu.ca (Scott Wang)
Date: Wed, 5 Feb 2020 13:55:44 -0800
Subject: [Zeek] "bro-cluster-in-a-box-setup" to
 "zeek-cluster-in-a-box-setup"?
In-Reply-To: <CAPfnCugKM6hnRFD_5BzOfS9hUSARvmJ21jd5y4okGgaKDNCW_A@mail.gmail.com>
References: <DM6PR12MB34983FF065BE194427DECA868B020@DM6PR12MB3498.namprd12.prod.outlook.com>
	<CAPfnCugKM6hnRFD_5BzOfS9hUSARvmJ21jd5y4okGgaKDNCW_A@mail.gmail.com>
Message-ID: <54871A8A-B529-467A-A4A1-C446C5DC51F3@sfu.ca>

At the Canarie workshop, Steve Smoot from Corelight suggested using pf_ring still. Any thoughts/comments on switching to af_packet? Advantages vs Disadvantages?

Regards,
Scott

> On Feb 05, 2020, at 12:48, Justin Azoff <justin at corelight.com> wrote:
> 
> Hi!
> 
> It shouldn't be that hard to update to 3.x.. 
> 
> - bro-pkg should be swapped out with the renamed zkg
> - the python2 references can likely be changed to 3
> - caf no longer needs to be installed separately
> - geoip and databases needs to be swapped out with maxminddb versions, might need a license
> - probably worth it to switch to af_packet from pf_ring.. pf_ring was only used initially to easily support capturing directly from both halves of a tap, which might not be a requirement anymore.
> 
> My schedule is a bit crazy for the next week, but once I have some time to work on it I should be able to get things updated pretty quickly.. There's really not much to it.
> 
> 
> 
> On Wed, Feb 5, 2020 at 12:38 PM Paul Sibley <Paul.Sibley at canarie.ca <mailto:Paul.Sibley at canarie.ca>> wrote:
> Hello Zeek Community,
> 
>  
> 
> I am working on a project where Zeek has been deployed in two phases.  During the first phase, some participants used ?https://github.com/ncsa/bro-cluster-in-a-box-setup <https://github.com/ncsa/bro-cluster-in-a-box-setup>? script to assist in, and automate a lot of the installation process.
> 
> Since then we have entered the phase in our project where more participants have been added, CentOS 8 is preferred, and we are using Zeek 3.0.1.
> 
> I wonder if any consideration, or work has been done, in updating the bro-cluster-in-a-box script to work with the updated OS and Zeek version.  Any information would be appreciated.
> 
>  
> 
> Thanks in advance,
> 
> Paul Sibley
> 
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org <mailto:zeek at zeek.org>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek <http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek>
> 
> -- 
> Justin
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200205/d3903756/attachment.html 

From smoot at corelight.com  Wed Feb  5 14:09:01 2020
From: smoot at corelight.com (Steve Smoot)
Date: Wed, 5 Feb 2020 22:09:01 +0000
Subject: [Zeek] "bro-cluster-in-a-box-setup" to
	"zeek-cluster-in-a-box-setup"?
In-Reply-To: <54871A8A-B529-467A-A4A1-C446C5DC51F3@sfu.ca>
References: <DM6PR12MB34983FF065BE194427DECA868B020@DM6PR12MB3498.namprd12.prod.outlook.com>
	<CAPfnCugKM6hnRFD_5BzOfS9hUSARvmJ21jd5y4okGgaKDNCW_A@mail.gmail.com>
	<54871A8A-B529-467A-A4A1-C446C5DC51F3@sfu.ca>
Message-ID: <CADOXQ8Y8_dF=TVYvCtRJqfy+FaiLiQ3+iBcVtj+D+dW2ZpxXEg@mail.gmail.com>

Well, it was more nuanced.  Or should have been.  I think AFpacket is
generally better but if you want 2 taps, then PF_Ring.
But always happy to hear other people's experience!
-s

On Wed, Feb 5, 2020 at 10:04 PM Scott Wang <scwang+bro at sfu.ca> wrote:

> At the Canarie workshop, Steve Smoot from Corelight suggested using
> pf_ring still. Any thoughts/comments on switching to af_packet? Advantages
> vs Disadvantages?
>
> Regards,
> Scott
>
> On Feb 05, 2020, at 12:48, Justin Azoff <justin at corelight.com> wrote:
>
> Hi!
>
> It shouldn't be that hard to update to 3.x..
>
> - bro-pkg should be swapped out with the renamed zkg
> - the python2 references can likely be changed to 3
> - caf no longer needs to be installed separately
> - geoip and databases needs to be swapped out with maxminddb versions,
> might need a license
> - probably worth it to switch to af_packet from pf_ring.. pf_ring was only
> used initially to easily support capturing directly from both halves of a
> tap, which might not be a requirement anymore.
>
> My schedule is a bit crazy for the next week, but once I have some time to
> work on it I should be able to get things updated pretty quickly.. There's
> really not much to it.
>
>
>
> On Wed, Feb 5, 2020 at 12:38 PM Paul Sibley <Paul.Sibley at canarie.ca>
> wrote:
>
>> Hello Zeek Community,
>>
>>
>>
>> I am working on a project where Zeek has been deployed in two phases.
>> During the first phase, some participants used ?
>> https://github.com/ncsa/bro-cluster-in-a-box-setup? script to assist in,
>> and automate a lot of the installation process.
>>
>> Since then we have entered the phase in our project where more
>> participants have been added, CentOS 8 is preferred, and we are using Zeek
>> 3.0.1.
>>
>> I wonder if any consideration, or work has been done, in updating the
>> bro-cluster-in-a-box script to work with the updated OS and Zeek version.
>> Any information would be appreciated.
>>
>>
>>
>> Thanks in advance,
>>
>> Paul Sibley
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>> <http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek>
>
>
>
> --
> Justin
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
*Stephen R. Smoot, PhD*
VP, Customer Success
Corelight
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200205/fca2d4ba/attachment.html 

From michalpurzynski1 at gmail.com  Wed Feb  5 14:12:53 2020
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Wed, 5 Feb 2020 14:12:53 -0800
Subject: [Zeek] "bro-cluster-in-a-box-setup" to
	"zeek-cluster-in-a-box-setup"?
In-Reply-To: <54871A8A-B529-467A-A4A1-C446C5DC51F3@sfu.ca>
References: <DM6PR12MB34983FF065BE194427DECA868B020@DM6PR12MB3498.namprd12.prod.outlook.com>
	<CAPfnCugKM6hnRFD_5BzOfS9hUSARvmJ21jd5y4okGgaKDNCW_A@mail.gmail.com>
	<54871A8A-B529-467A-A4A1-C446C5DC51F3@sfu.ca>
Message-ID: <CAJ6bFK3V3FBZr8W84zj2C9y+4HxuSQ3kweEyO+fYxMfcOZ0t5Q@mail.gmail.com>

There's a law that if you say pf_ring and af_packet 3 times, Michal shows
up.

I don't see many (any?) reasons for using pf_ring, TBH, if you have a
modern kernel or a decent network card (Mellanox, Intel, etc). And I still
owe the community the article to show how to use the af_packet correctly :/

The case where one has inputs from multiple taps, to multiple network ports
will be handled the same way by af_packet, if interfaces are bonded or
bridged and by pf_ring. None of them buffers data and processes them at L4
and deals with out of order, etc.

On Wed, Feb 5, 2020 at 2:04 PM Scott Wang <scwang+bro at sfu.ca> wrote:

> At the Canarie workshop, Steve Smoot from Corelight suggested using
> pf_ring still. Any thoughts/comments on switching to af_packet? Advantages
> vs Disadvantages?
>
> Regards,
> Scott
>
> On Feb 05, 2020, at 12:48, Justin Azoff <justin at corelight.com> wrote:
>
> Hi!
>
> It shouldn't be that hard to update to 3.x..
>
> - bro-pkg should be swapped out with the renamed zkg
> - the python2 references can likely be changed to 3
> - caf no longer needs to be installed separately
> - geoip and databases needs to be swapped out with maxminddb versions,
> might need a license
> - probably worth it to switch to af_packet from pf_ring.. pf_ring was only
> used initially to easily support capturing directly from both halves of a
> tap, which might not be a requirement anymore.
>
> My schedule is a bit crazy for the next week, but once I have some time to
> work on it I should be able to get things updated pretty quickly.. There's
> really not much to it.
>
>
>
> On Wed, Feb 5, 2020 at 12:38 PM Paul Sibley <Paul.Sibley at canarie.ca>
> wrote:
>
>> Hello Zeek Community,
>>
>>
>>
>> I am working on a project where Zeek has been deployed in two phases.
>> During the first phase, some participants used ?
>> https://github.com/ncsa/bro-cluster-in-a-box-setup? script to assist in,
>> and automate a lot of the installation process.
>>
>> Since then we have entered the phase in our project where more
>> participants have been added, CentOS 8 is preferred, and we are using Zeek
>> 3.0.1.
>>
>> I wonder if any consideration, or work has been done, in updating the
>> bro-cluster-in-a-box script to work with the updated OS and Zeek version.
>> Any information would be appreciated.
>>
>>
>>
>> Thanks in advance,
>>
>> Paul Sibley
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>> <http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek>
>
>
>
> --
> Justin
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200205/18e567a5/attachment-0001.html 

From justin at corelight.com  Wed Feb  5 14:23:50 2020
From: justin at corelight.com (Justin Azoff)
Date: Wed, 5 Feb 2020 17:23:50 -0500
Subject: [Zeek] "bro-cluster-in-a-box-setup" to
	"zeek-cluster-in-a-box-setup"?
In-Reply-To: <CAJ6bFK3V3FBZr8W84zj2C9y+4HxuSQ3kweEyO+fYxMfcOZ0t5Q@mail.gmail.com>
References: <DM6PR12MB34983FF065BE194427DECA868B020@DM6PR12MB3498.namprd12.prod.outlook.com>
	<CAPfnCugKM6hnRFD_5BzOfS9hUSARvmJ21jd5y4okGgaKDNCW_A@mail.gmail.com>
	<54871A8A-B529-467A-A4A1-C446C5DC51F3@sfu.ca>
	<CAJ6bFK3V3FBZr8W84zj2C9y+4HxuSQ3kweEyO+fYxMfcOZ0t5Q@mail.gmail.com>
Message-ID: <CAPfnCuiUP0UwuSGrDkJusCGzm8wruzhw1FYnYiUjFRcgwPgv-w@mail.gmail.com>

OOOH!  You can bond two interfaces together and run af_packet on the bond0
interface? that works?!?

On Wed, Feb 5, 2020 at 5:13 PM Micha? Purzy?ski <michalpurzynski1 at gmail.com>
wrote:

> There's a law that if you say pf_ring and af_packet 3 times, Michal shows
> up.
>
> I don't see many (any?) reasons for using pf_ring, TBH, if you have a
> modern kernel or a decent network card (Mellanox, Intel, etc). And I still
> owe the community the article to show how to use the af_packet correctly :/
>
> The case where one has inputs from multiple taps, to multiple network
> ports will be handled the same way by af_packet, if interfaces are bonded
> or bridged and by pf_ring. None of them buffers data and processes them at
> L4 and deals with out of order, etc.
>
> On Wed, Feb 5, 2020 at 2:04 PM Scott Wang <scwang+bro at sfu.ca> wrote:
>
>> At the Canarie workshop, Steve Smoot from Corelight suggested using
>> pf_ring still. Any thoughts/comments on switching to af_packet? Advantages
>> vs Disadvantages?
>>
>> Regards,
>> Scott
>>
>> On Feb 05, 2020, at 12:48, Justin Azoff <justin at corelight.com> wrote:
>>
>> Hi!
>>
>> It shouldn't be that hard to update to 3.x..
>>
>> - bro-pkg should be swapped out with the renamed zkg
>> - the python2 references can likely be changed to 3
>> - caf no longer needs to be installed separately
>> - geoip and databases needs to be swapped out with maxminddb versions,
>> might need a license
>> - probably worth it to switch to af_packet from pf_ring.. pf_ring was
>> only used initially to easily support capturing directly from both
>> halves of a tap, which might not be a requirement anymore.
>>
>> My schedule is a bit crazy for the next week, but once I have some time
>> to work on it I should be able to get things updated pretty quickly..
>> There's really not much to it.
>>
>>
>>
>> On Wed, Feb 5, 2020 at 12:38 PM Paul Sibley <Paul.Sibley at canarie.ca>
>> wrote:
>>
>>> Hello Zeek Community,
>>>
>>>
>>>
>>> I am working on a project where Zeek has been deployed in two phases.
>>> During the first phase, some participants used ?
>>> https://github.com/ncsa/bro-cluster-in-a-box-setup? script to assist
>>> in, and automate a lot of the installation process.
>>>
>>> Since then we have entered the phase in our project where more
>>> participants have been added, CentOS 8 is preferred, and we are using Zeek
>>> 3.0.1.
>>>
>>> I wonder if any consideration, or work has been done, in updating the
>>> bro-cluster-in-a-box script to work with the updated OS and Zeek version.
>>> Any information would be appreciated.
>>>
>>>
>>>
>>> Thanks in advance,
>>>
>>> Paul Sibley
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>> <http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek>
>>
>>
>>
>> --
>> Justin
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>>
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>

-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200205/b8452779/attachment.html 

From mabuchan at gmail.com  Wed Feb  5 14:38:45 2020
From: mabuchan at gmail.com (Mark Buchanan)
Date: Wed, 5 Feb 2020 16:38:45 -0600
Subject: [Zeek] "bro-cluster-in-a-box-setup" to
	"zeek-cluster-in-a-box-setup"?
In-Reply-To: <CAPfnCuiUP0UwuSGrDkJusCGzm8wruzhw1FYnYiUjFRcgwPgv-w@mail.gmail.com>
References: <CAPfnCuiUP0UwuSGrDkJusCGzm8wruzhw1FYnYiUjFRcgwPgv-w@mail.gmail.com>
Message-ID: <5F516BC7-C67D-4CA1-8E10-07D8FBCD5033@gmail.com>

Somewhat of a tangent, but did af_packet in CentOS/RHEL 7 kernels ever solve the distribution of packets across multiple bro/zeek processes when observing IPv6 traffic?

I observed an issue a while back where when watching traffic on an interface (bonded or not) with multiple bro/zeek processes, that all processes would see the IPv6 traffic, vice only one process.  IPv4 worked properly, but any network with IPv6 had some nasty logs because of duplication.

--
Mark Buchanan

> On Feb 5, 2020, at 16:26, Justin Azoff <justin at corelight.com> wrote:
> 
> ?
> OOOH!  You can bond two interfaces together and run af_packet on the bond0 interface? that works?!?
> 
>> On Wed, Feb 5, 2020 at 5:13 PM Micha? Purzy?ski <michalpurzynski1 at gmail.com> wrote:
>> There's a law that if you say pf_ring and af_packet 3 times, Michal shows up.
>> 
>> I don't see many (any?) reasons for using pf_ring, TBH, if you have a modern kernel or a decent network card (Mellanox, Intel, etc). And I still owe the community the article to show how to use the af_packet correctly :/
>> 
>> The case where one has inputs from multiple taps, to multiple network ports will be handled the same way by af_packet, if interfaces are bonded or bridged and by pf_ring. None of them buffers data and processes them at L4 and deals with out of order, etc.
>> 
>>> On Wed, Feb 5, 2020 at 2:04 PM Scott Wang <scwang+bro at sfu.ca> wrote:
>>> At the Canarie workshop, Steve Smoot from Corelight suggested using pf_ring still. Any thoughts/comments on switching to af_packet? Advantages vs Disadvantages?
>>> 
>>> Regards,
>>> Scott
>>> 
>>>> On Feb 05, 2020, at 12:48, Justin Azoff <justin at corelight.com> wrote:
>>>> 
>>>> Hi!
>>>> 
>>>> It shouldn't be that hard to update to 3.x.. 
>>>> 
>>>> - bro-pkg should be swapped out with the renamed zkg
>>>> - the python2 references can likely be changed to 3
>>>> - caf no longer needs to be installed separately
>>>> - geoip and databases needs to be swapped out with maxminddb versions, might need a license
>>>> - probably worth it to switch to af_packet from pf_ring.. pf_ring was only used initially to easily support capturing directly from both halves of a tap, which might not be a requirement anymore.
>>>> 
>>>> My schedule is a bit crazy for the next week, but once I have some time to work on it I should be able to get things updated pretty quickly.. There's really not much to it.
>>>> 
>>>> 
>>>> 
>>>>> On Wed, Feb 5, 2020 at 12:38 PM Paul Sibley <Paul.Sibley at canarie.ca> wrote:
>>>>> Hello Zeek Community,
>>>>> 
>>>>>  
>>>>> 
>>>>> I am working on a project where Zeek has been deployed in two phases.  During the first phase, some participants used ?https://github.com/ncsa/bro-cluster-in-a-box-setup? script to assist in, and automate a lot of the installation process.
>>>>> 
>>>>> Since then we have entered the phase in our project where more participants have been added, CentOS 8 is preferred, and we are using Zeek 3.0.1.
>>>>> 
>>>>> I wonder if any consideration, or work has been done, in updating the bro-cluster-in-a-box script to work with the updated OS and Zeek version.  Any information would be appreciated.
>>>>> 
>>>>>  
>>>>> 
>>>>> Thanks in advance,
>>>>> 
>>>>> Paul Sibley
>>>>> 
>>>>> _______________________________________________
>>>>> Zeek mailing list
>>>>> zeek at zeek.org
>>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>>> 
>>>> 
>>>> -- 
>>>> Justin
>>>> _______________________________________________
>>>> Zeek mailing list
>>>> zeek at zeek.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>> 
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> 
> 
> -- 
> Justin
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200205/722b8b51/attachment.html 

From andrew at aklaus.ca  Wed Feb  5 15:49:11 2020
From: andrew at aklaus.ca (Andrew Klaus)
Date: Wed, 5 Feb 2020 16:49:11 -0700
Subject: [Zeek] "bro-cluster-in-a-box-setup" to
	"zeek-cluster-in-a-box-setup"?
In-Reply-To: <mailman.5802.1580940812.827.zeek@zeek.org>
References: <mailman.5802.1580940812.827.zeek@zeek.org>
Message-ID: <CAGENE8u_qaAbr26bkqfO01bgx=C=9AOHkDUxV3R3mtgM=axc5g@mail.gmail.com>

Hi Michal,

We're working with Zeek on the same project as Paul and have opted to use
Zeek within a Docker container since it works better for our workflow. It's
my first time using AF_PACKET, so it would be nice to have a second set of
eyes from you on it if you don't mind :)

I've bridged all interfaces that need to be analyzed:
https://github.com/cybera/jsp-zeek/blob/master/host/60-zeek-bridge.yaml

Then I've disabled hardware features on the NIC, which is done on each
boot: https://github.com/cybera/jsp-zeek/blob/master/host/ethtool.sh

Using "interface=af_packet::br0", and pinning CPUs for the workers,
manager, and proxy:
https://github.com/cybera/jsp-zeek/blob/master/docker/files/etc/node.cfg

We don't have a ton of traffic being analyzed yet, but want to make sure we
have a decent setup for when we start ingesting more data. I've pieced this
together from various Zeek articles I've read, so hopefully it's not too
much of a Frankenstein's Monster ;)

Any help would be appreciated!

Cheers,
Andrew


> Date: Wed, 5 Feb 2020 14:12:53 -0800
> From: Micha? Purzy?ski <michalpurzynski1 at gmail.com>
> Subject: Re: [Zeek] "bro-cluster-in-a-box-setup" to
>         "zeek-cluster-in-a-box-setup"?
> To: Scott Wang <scwang+bro at sfu.ca>
> Cc: Paul Sibley <Paul.Sibley at canarie.ca>, "zeek at zeek.org"
>         <zeek at zeek.org>
> Message-ID:
>         <
> CAJ6bFK3V3FBZr8W84zj2C9y+4HxuSQ3kweEyO+fYxMfcOZ0t5Q at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> There's a law that if you say pf_ring and af_packet 3 times, Michal shows
> up.
>
> I don't see many (any?) reasons for using pf_ring, TBH, if you have a
> modern kernel or a decent network card (Mellanox, Intel, etc). And I still
> owe the community the article to show how to use the af_packet correctly :/
>
> The case where one has inputs from multiple taps, to multiple network ports
> will be handled the same way by af_packet, if interfaces are bonded or
> bridged and by pf_ring. None of them buffers data and processes them at L4
> and deals with out of order, etc.
>
> On Wed, Feb 5, 2020 at 2:04 PM Scott Wang <scwang+bro at sfu.ca> wrote:
>
> > At the Canarie workshop, Steve Smoot from Corelight suggested using
> > pf_ring still. Any thoughts/comments on switching to af_packet?
> Advantages
> > vs Disadvantages?
> >
> > Regards,
> > Scott
> >
> > On Feb 05, 2020, at 12:48, Justin Azoff <justin at corelight.com> wrote:
> >
> > Hi!
> >
> > It shouldn't be that hard to update to 3.x..
> >
> > - bro-pkg should be swapped out with the renamed zkg
> > - the python2 references can likely be changed to 3
> > - caf no longer needs to be installed separately
> > - geoip and databases needs to be swapped out with maxminddb versions,
> > might need a license
> > - probably worth it to switch to af_packet from pf_ring.. pf_ring was
> only
> > used initially to easily support capturing directly from both halves of a
> > tap, which might not be a requirement anymore.
> >
> > My schedule is a bit crazy for the next week, but once I have some time
> to
> > work on it I should be able to get things updated pretty quickly..
> There's
> > really not much to it.
> >
> >
> >
> > On Wed, Feb 5, 2020 at 12:38 PM Paul Sibley <Paul.Sibley at canarie.ca>
> > wrote:
> >
> >> Hello Zeek Community,
> >>
> >>
> >>
> >> I am working on a project where Zeek has been deployed in two phases.
> >> During the first phase, some participants used ?
> >> https://github.com/ncsa/bro-cluster-in-a-box-setup? script to assist
> in,
> >> and automate a lot of the installation process.
> >>
> >> Since then we have entered the phase in our project where more
> >> participants have been added, CentOS 8 is preferred, and we are using
> Zeek
> >> 3.0.1.
> >>
> >> I wonder if any consideration, or work has been done, in updating the
> >> bro-cluster-in-a-box script to work with the updated OS and Zeek
> version.
> >> Any information would be appreciated.
> >>
> >>
> >>
> >> Thanks in advance,
> >>
> >> Paul Sibley
> >> _______________________________________________
> >> Zeek mailing list
> >> zeek at zeek.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> >> <http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek>
> >
> >
> >
> > --
> > Justin
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> >
> >
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200205/18e567a5/attachment.html
>
> ------------------------------
>
> _______________________________________________
> Zeek mailing list
> Zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
> End of Zeek Digest, Vol 166, Issue 8
> ************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200205/aa84fc09/attachment-0001.html 

From blackhole.em at gmail.com  Thu Feb  6 05:04:06 2020
From: blackhole.em at gmail.com (Joe Blow)
Date: Thu, 06 Feb 2020 08:04:06 -0500
Subject: [Zeek] "bro-cluster-in-a-box-setup" to
 "zeek-cluster-in-a-box-setup"?
In-Reply-To: <CAPfnCuiUP0UwuSGrDkJusCGzm8wruzhw1FYnYiUjFRcgwPgv-w@mail.gmail.com>
Message-ID: <ejd4t21slslicoebvb8oui8r.1580994246894@gmail.com>

An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200206/b601c098/attachment.html 

From bill.de.ping at gmail.com  Thu Feb  6 06:17:17 2020
From: bill.de.ping at gmail.com (william de ping)
Date: Thu, 6 Feb 2020 16:17:17 +0200
Subject: [Zeek]  - difference between plugin and package
Message-ID: <CAKZA8xhPsqMy=hi4E-ugftdjB-zu8M4r_h-WO4e3Y9J_-7c_OQ@mail.gmail.com>

Hi all,

I would appreciate an explanation on the differences between plugins and
packages.

Are plugins deprecated ?
Should I convert my code to  package or  plugin ?

Thank you
B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200206/2459b3bc/attachment.html 

From johanna at icir.org  Thu Feb  6 09:16:59 2020
From: johanna at icir.org (Johanna Amann)
Date: Thu, 06 Feb 2020 09:16:59 -0800
Subject: [Zeek] - difference between plugin and package
In-Reply-To: <CAKZA8xhPsqMy=hi4E-ugftdjB-zu8M4r_h-WO4e3Y9J_-7c_OQ@mail.gmail.com>
References: <CAKZA8xhPsqMy=hi4E-ugftdjB-zu8M4r_h-WO4e3Y9J_-7c_OQ@mail.gmail.com>
Message-ID: <5AB97D16-731C-49EE-BBDF-A81A9442974F@icir.org>

Hi William,

> I would appreciate an explanation on the differences between plugins 
> and
> packages.

They describe different concepts.

Plugins are extensions to Zeek written in C++ using the plugin API. This 
can be used to create bifs, log-writers, etc.

Packages are sets of scripts or plugins distributed in a format that the 
zeek package manager can understand.

So - a package can (but does not have to) contain a plugin; it also can 
just contain script-files.

Johanna

From japhughe at ucsc.edu  Thu Feb  6 14:14:43 2020
From: japhughe at ucsc.edu (James Hughes)
Date: Thu, 6 Feb 2020 14:14:43 -0800
Subject: [Zeek] Zeek and Broker versions.
Message-ID: <BC34BB65-3FFF-4B23-B898-BCAAE019642B@ucsc.edu>

Hello Zeek community:

I am new to this list and would like help debugging a connection from Zeek 3.0.1 to a machine running a broker python application. Broker is locally compiled version 1.2.4. This setup used to work, but now we have upgraded to 3.0.1 and it is no longer working. 

The symptoms are that Zeek is connecting and disconnecting from the machine running Broker. I am guessing it is a version miss match but I do not know how to diagnose this. (If there are logs, I don't know where they might be.) Any hints would be appreciated. 

Thanks in advance. 

Sincerely

Jim
'



From jdhayek at protonmail.com  Thu Feb  6 14:30:36 2020
From: jdhayek at protonmail.com (Justin Hayek)
Date: Thu, 06 Feb 2020 22:30:36 +0000
Subject: [Zeek] "bro-cluster-in-a-box-setup" to
	"zeek-cluster-in-a-box-setup"?
In-Reply-To: <ejd4t21slslicoebvb8oui8r.1580994246894@gmail.com>
References: <ejd4t21slslicoebvb8oui8r.1580994246894@gmail.com>
Message-ID: <GVUfXwBNh1UMXaM7u9AQ0VIIF6eukO3bqxAUWRFnCZzYYZxOQLEtaQSSeJsQHda2pGP6Fy7FYohylUuHiwRmrwq9uxlwXupUt470sTENgZw=@protonmail.com>

You can absolutely do this. We are using af_packet and bonded interfaces throughout the majority of our deployments (approximately 1800 sensors).

We decided on af_packet as it was included in recent (at the time 2yrs ago) kernels. I can't speak to non-Debian based distro's, but we haven't seen any issues related to the use of af_packet.

-Justin

??????? Original Message ???????
On Thursday, February 6, 2020 7:04 AM, Joe Blow <blackhole.em at gmail.com> wrote:

> Would love to hear this confirmed with no performance issues.
>
> Cheers,
>
> JB
>
> Sent via [BlackBerry Hub+ Inbox for Android](http://play.google.com/store/apps/details?id=com.blackberry.hub)
>
> From: justin at corelight.com
> Sent: February 5, 2020 5:26 PM
> To: michalpurzynski1 at gmail.com
> Cc: Paul.Sibley at canarie.ca; zeek at zeek.org
> Subject: Re: [Zeek] "bro-cluster-in-a-box-setup" to "zeek-cluster-in-a-box-setup"?
> OOOH!  You can bond two interfaces together and run af_packet on the bond0 interface? that works?!?
>
> On Wed, Feb 5, 2020 at 5:13 PM Micha? Purzy?ski <michalpurzynski1 at gmail.com> wrote:
>
>> There's a law that if you say pf_ring and af_packet 3 times, Michal shows up.
>>
>> I don't see many (any?) reasons for using pf_ring, TBH, if you have a modern kernel or a decent network card (Mellanox, Intel, etc). And I still owe the community the article to show how to use the af_packet correctly :/
>>
>> The case where one has inputs from multiple taps, to multiple network ports will be handled the same way by af_packet, if interfaces are bonded or bridged and by pf_ring. None of them buffers data and processes them at L4 and deals with out of order, etc.
>>
>> On Wed, Feb 5, 2020 at 2:04 PM Scott Wang <[scwang+bro at sfu.ca](mailto:scwang%2Bbro at sfu.ca)> wrote:
>>
>>> At the Canarie workshop, Steve Smoot from Corelight suggested using pf_ring still. Any thoughts/comments on switching to af_packet? Advantages vs Disadvantages?
>>>
>>> Regards,
>>> Scott
>>>
>>>> On Feb 05, 2020, at 12:48, Justin Azoff <justin at corelight.com> wrote:
>>>>
>>>> Hi!
>>>>
>>>> It shouldn't be that hard to update to 3.x..
>>>>
>>>> - bro-pkg should be swapped out with the renamed zkg
>>>> - the python2 references can likely be changed to 3
>>>> - caf no longer needs to be installed separately
>>>> - geoip and databases needs to be swapped out with maxminddb versions, might need a license
>>>> - probably worth it to switch to af_packet from pf_ring.. pf_ring was only used initially to easily support capturing directly from both halves of a tap, which might not be a requirement anymore.
>>>>
>>>> My schedule is a bit crazy for the next week, but once I have some time to work on it I should be able to get things updated pretty quickly.. There's really not much to it.
>>>>
>>>> On Wed, Feb 5, 2020 at 12:38 PM Paul Sibley <Paul.Sibley at canarie.ca> wrote:
>>>>
>>>>> Hello Zeek Community,
>>>>>
>>>>> I am working on a project where Zeek has been deployed in two phases.  During the first phase, some participants used ?https://github.com/ncsa/bro-cluster-in-a-box-setup? script to assist in, and automate a lot of the installation process.
>>>>>
>>>>> Since then we have entered the phase in our project where more participants have been added, CentOS 8 is preferred, and we are using Zeek 3.0.1.
>>>>>
>>>>> I wonder if any consideration, or work has been done, in updating the bro-cluster-in-a-box script to work with the updated OS and Zeek version.  Any information would be appreciated.
>>>>>
>>>>> Thanks in advance,
>>>>>
>>>>> Paul Sibley
>>>>>
>>>>> _______________________________________________
>>>>> Zeek mailing list
>>>>> zeek at zeek.org
>>>>> [http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek](http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek)
>>>>
>>>> --
>>>> Justin
>>>> _______________________________________________
>>>> Zeek mailing list
>>>> zeek at zeek.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>>
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
> --
> Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200206/2d6e3450/attachment-0001.html 

From dopheide at gmail.com  Thu Feb  6 14:30:42 2020
From: dopheide at gmail.com (Mike Dopheide)
Date: Thu, 6 Feb 2020 16:30:42 -0600
Subject: [Zeek] Zeek and Broker versions.
In-Reply-To: <BC34BB65-3FFF-4B23-B898-BCAAE019642B@ucsc.edu>
References: <BC34BB65-3FFF-4B23-B898-BCAAE019642B@ucsc.edu>
Message-ID: <CAPy2kFaK-AF9z+bRs_Qn+mqeKNhnuHa_zr0i5QFqxhxyDQXmNA@mail.gmail.com>

James,

While I haven't specifically tried to debug a situation such as yours, this
may be an avenue to try if you're building zeek from source:

1) Add  *--enable-debug to your configure flags.*
*2) Then you can run zeek with "-B broker" either on the command line or by
adding that to ZeekArgs in zeekctl.cfg.*
*3) Now you should see debug.log getting created.*


*-Dop*


On Thu, Feb 6, 2020 at 4:17 PM James Hughes <japhughe at ucsc.edu> wrote:

> Hello Zeek community:
>
> I am new to this list and would like help debugging a connection from Zeek
> 3.0.1 to a machine running a broker python application. Broker is locally
> compiled version 1.2.4. This setup used to work, but now we have upgraded
> to 3.0.1 and it is no longer working.
>
> The symptoms are that Zeek is connecting and disconnecting from the
> machine running Broker. I am guessing it is a version miss match but I do
> not know how to diagnose this. (If there are logs, I don't know where they
> might be.) Any hints would be appreciated.
>
> Thanks in advance.
>
> Sincerely
>
> Jim
> '
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200206/289de457/attachment-0001.html 

From japhughe at ucsc.edu  Thu Feb  6 14:32:16 2020
From: japhughe at ucsc.edu (James Hughes)
Date: Thu, 6 Feb 2020 14:32:16 -0800
Subject: [Zeek] Zeek and Broker versions.
In-Reply-To: <CAPy2kFaK-AF9z+bRs_Qn+mqeKNhnuHa_zr0i5QFqxhxyDQXmNA@mail.gmail.com>
References: <BC34BB65-3FFF-4B23-B898-BCAAE019642B@ucsc.edu>
	<CAPy2kFaK-AF9z+bRs_Qn+mqeKNhnuHa_zr0i5QFqxhxyDQXmNA@mail.gmail.com>
Message-ID: <26280206-D0CB-426C-BE26-5700C7C0D862@ucsc.edu>

Thanks! Will give that a try! 

Jim

> On Feb 6, 2020, at 2:30 PM, Mike Dopheide <dopheide at gmail.com> wrote:
> 
> James,
> 
> While I haven't specifically tried to debug a situation such as yours, this may be an avenue to try if you're building zeek from source:
> 
> 1) Add  --enable-debug to your configure flags.
> 2) Then you can run zeek with "-B broker" either on the command line or by adding that to ZeekArgs in zeekctl.cfg.
> 3) Now you should see debug.log getting created.
> 
> -Dop
> 
> 
> On Thu, Feb 6, 2020 at 4:17 PM James Hughes <japhughe at ucsc.edu <mailto:japhughe at ucsc.edu>> wrote:
> Hello Zeek community:
> 
> I am new to this list and would like help debugging a connection from Zeek 3.0.1 to a machine running a broker python application. Broker is locally compiled version 1.2.4. This setup used to work, but now we have upgraded to 3.0.1 and it is no longer working. 
> 
> The symptoms are that Zeek is connecting and disconnecting from the machine running Broker. I am guessing it is a version miss match but I do not know how to diagnose this. (If there are logs, I don't know where they might be.) Any hints would be appreciated. 
> 
> Thanks in advance. 
> 
> Sincerely
> 
> Jim
> '
> 
> 
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org <mailto:zeek at zeek.org>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek <http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200206/f5f26610/attachment.html 

From michalpurzynski1 at gmail.com  Thu Feb  6 14:53:51 2020
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Thu, 6 Feb 2020 14:53:51 -0800
Subject: [Zeek] "bro-cluster-in-a-box-setup" to
	"zeek-cluster-in-a-box-setup"?
In-Reply-To: <GVUfXwBNh1UMXaM7u9AQ0VIIF6eukO3bqxAUWRFnCZzYYZxOQLEtaQSSeJsQHda2pGP6Fy7FYohylUuHiwRmrwq9uxlwXupUt470sTENgZw=@protonmail.com>
References: <ejd4t21slslicoebvb8oui8r.1580994246894@gmail.com>
	<GVUfXwBNh1UMXaM7u9AQ0VIIF6eukO3bqxAUWRFnCZzYYZxOQLEtaQSSeJsQHda2pGP6Fy7FYohylUuHiwRmrwq9uxlwXupUt470sTENgZw=@protonmail.com>
Message-ID: <CAJ6bFK2+tm_9AeAjsSmSfpaaCqRmAJAkoZr_tTsjbYSMTO+L_A@mail.gmail.com>

Sure, you can run af_packet on any device, including
device-made-of-devices, any virtual and physical interface and a
combination thereof. The whole af_packet mechanism (they call it "taps"
internally) works on a higher level.

Now let's address the elephant in the room, shall we.

IPv4 is correctly hashed on relatively modern kernels (I believe RHEL 7.4
has a fix for that) - so you can use the cluster_flow mode.
IPv6 seems to have problems, sometimes - I can see it correctly hashed most
of the time (but not always).

What we do on production, is we let card hash packets by src + dst IP
address (and never ports, because fragments don't have port numbers), with
the symmetric key, offloading disabled, correct number of queues set and
cluster_qm.

If the community is interested I can have an article out in a week - just
need to know if there's someone who wants that?

On Thu, Feb 6, 2020 at 2:30 PM Justin Hayek <jdhayek at protonmail.com> wrote:

> You can absolutely do this. We are using af_packet and bonded interfaces
> throughout the majority of our deployments (approximately 1800 sensors).
>
> We decided on af_packet as it was included in recent (at the time 2yrs
> ago) kernels. I can't speak to non-Debian based distro's, but we haven't
> seen any issues related to the use of af_packet.
>
> -Justin
>
>
>
> ??????? Original Message ???????
> On Thursday, February 6, 2020 7:04 AM, Joe Blow <blackhole.em at gmail.com>
> wrote:
>
> Would love to hear this confirmed with no performance issues.
>
> Cheers,
>
> JB
>
> Sent via BlackBerry Hub+ Inbox for Android
> <http://play.google.com/store/apps/details?id=com.blackberry.hub>
> *From:* justin at corelight.com
> *Sent:* February 5, 2020 5:26 PM
> *To:* michalpurzynski1 at gmail.com
> *Cc:* Paul.Sibley at canarie.ca; zeek at zeek.org
> *Subject:* Re: [Zeek] "bro-cluster-in-a-box-setup" to
> "zeek-cluster-in-a-box-setup"?
> OOOH!  You can bond two interfaces together and run af_packet on the bond0
> interface? that works?!?
>
> On Wed, Feb 5, 2020 at 5:13 PM Micha? Purzy?ski <
> michalpurzynski1 at gmail.com> wrote:
>
>> There's a law that if you say pf_ring and af_packet 3 times, Michal shows
>> up.
>>
>> I don't see many (any?) reasons for using pf_ring, TBH, if you have a
>> modern kernel or a decent network card (Mellanox, Intel, etc). And I still
>> owe the community the article to show how to use the af_packet correctly :/
>>
>> The case where one has inputs from multiple taps, to multiple network
>> ports will be handled the same way by af_packet, if interfaces are bonded
>> or bridged and by pf_ring. None of them buffers data and processes them at
>> L4 and deals with out of order, etc.
>>
>> On Wed, Feb 5, 2020 at 2:04 PM Scott Wang <scwang+bro at sfu.ca> wrote:
>>
>>> At the Canarie workshop, Steve Smoot from Corelight suggested using
>>> pf_ring still. Any thoughts/comments on switching to af_packet? Advantages
>>> vs Disadvantages?
>>>
>>> Regards,
>>> Scott
>>>
>>> On Feb 05, 2020, at 12:48, Justin Azoff <justin at corelight.com> wrote:
>>>
>>> Hi!
>>>
>>> It shouldn't be that hard to update to 3.x..
>>>
>>> - bro-pkg should be swapped out with the renamed zkg
>>> - the python2 references can likely be changed to 3
>>> - caf no longer needs to be installed separately
>>> - geoip and databases needs to be swapped out with maxminddb versions,
>>> might need a license
>>> - probably worth it to switch to af_packet from pf_ring.. pf_ring was
>>> only used initially to easily support capturing directly from both
>>> halves of a tap, which might not be a requirement anymore.
>>>
>>> My schedule is a bit crazy for the next week, but once I have some time
>>> to work on it I should be able to get things updated pretty quickly..
>>> There's really not much to it.
>>>
>>>
>>>
>>> On Wed, Feb 5, 2020 at 12:38 PM Paul Sibley <Paul.Sibley at canarie.ca>
>>> wrote:
>>>
>>>> Hello Zeek Community,
>>>>
>>>>
>>>>
>>>> I am working on a project where Zeek has been deployed in two phases.
>>>> During the first phase, some participants used ?
>>>> https://github.com/ncsa/bro-cluster-in-a-box-setup? script to assist
>>>> in, and automate a lot of the installation process.
>>>>
>>>> Since then we have entered the phase in our project where more
>>>> participants have been added, CentOS 8 is preferred, and we are using Zeek
>>>> 3.0.1.
>>>>
>>>> I wonder if any consideration, or work has been done, in updating the
>>>> bro-cluster-in-a-box script to work with the updated OS and Zeek version.
>>>> Any information would be appreciated.
>>>>
>>>>
>>>>
>>>> Thanks in advance,
>>>>
>>>> Paul Sibley
>>>>
>>>> _______________________________________________
>>>> Zeek mailing list
>>>> zeek at zeek.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>>> <http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek>
>>>>
>>>
>>>
>>> --
>>> Justin
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>>
>>>
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>>
>>
>
> --
> Justin
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200206/5af130c1/attachment-0001.html 

From jdhayek at protonmail.com  Thu Feb  6 15:09:06 2020
From: jdhayek at protonmail.com (Justin Hayek)
Date: Thu, 06 Feb 2020 23:09:06 +0000
Subject: [Zeek] "bro-cluster-in-a-box-setup" to
	"zeek-cluster-in-a-box-setup"?
In-Reply-To: <CAJ6bFK2+tm_9AeAjsSmSfpaaCqRmAJAkoZr_tTsjbYSMTO+L_A@mail.gmail.com>
References: <ejd4t21slslicoebvb8oui8r.1580994246894@gmail.com>
	<GVUfXwBNh1UMXaM7u9AQ0VIIF6eukO3bqxAUWRFnCZzYYZxOQLEtaQSSeJsQHda2pGP6Fy7FYohylUuHiwRmrwq9uxlwXupUt470sTENgZw=@protonmail.com>
	<CAJ6bFK2+tm_9AeAjsSmSfpaaCqRmAJAkoZr_tTsjbYSMTO+L_A@mail.gmail.com>
Message-ID: <hQMSAZE38wQ2ThG3p7oRNrgp1Gtkz6NAEfGYoO2PEqVyhPqKGdRp2uRDQ6114meNRCUfZuyq3N7Oo-fReEziPE7k2nUWKtnMSlvMGibahf4=@protonmail.com>

+1 on the article.

-Justin

??????? Original Message ???????
On Thursday, February 6, 2020 4:53 PM, Micha? Purzy?ski <michalpurzynski1 at gmail.com> wrote:

> Sure, you can run af_packet on any device, including device-made-of-devices, any virtual and physical interface and a combination thereof. The whole af_packet mechanism (they call it "taps" internally) works on a higher level.
>
> Now let's address the elephant in the room, shall we.
>
> IPv4 is correctly hashed on relatively modern kernels (I believe RHEL 7.4 has a fix for that) - so you can use the cluster_flow mode.
> IPv6 seems to have problems, sometimes - I can see it correctly hashed most of the time (but not always).
>
> What we do on production, is we let card hash packets by src + dst IP address (and never ports, because fragments don't have port numbers), with the symmetric key, offloading disabled, correct number of queues set and cluster_qm.
>
> If the community is interested I can have an article out in a week - just need to know if there's someone who wants that?
>
> On Thu, Feb 6, 2020 at 2:30 PM Justin Hayek <jdhayek at protonmail.com> wrote:
>
>> You can absolutely do this. We are using af_packet and bonded interfaces throughout the majority of our deployments (approximately 1800 sensors).
>>
>> We decided on af_packet as it was included in recent (at the time 2yrs ago) kernels. I can't speak to non-Debian based distro's, but we haven't seen any issues related to the use of af_packet.
>>
>> -Justin
>>
>> ??????? Original Message ???????
>> On Thursday, February 6, 2020 7:04 AM, Joe Blow <blackhole.em at gmail.com> wrote:
>>
>>> Would love to hear this confirmed with no performance issues.
>>>
>>> Cheers,
>>>
>>> JB
>>>
>>> Sent via [BlackBerry Hub+ Inbox for Android](http://play.google.com/store/apps/details?id=com.blackberry.hub)
>>>
>>> From: justin at corelight.com
>>> Sent: February 5, 2020 5:26 PM
>>> To: michalpurzynski1 at gmail.com
>>> Cc: Paul.Sibley at canarie.ca; zeek at zeek.org
>>> Subject: Re: [Zeek] "bro-cluster-in-a-box-setup" to "zeek-cluster-in-a-box-setup"?
>>> OOOH!  You can bond two interfaces together and run af_packet on the bond0 interface? that works?!?
>>>
>>> On Wed, Feb 5, 2020 at 5:13 PM Micha? Purzy?ski <michalpurzynski1 at gmail.com> wrote:
>>>
>>>> There's a law that if you say pf_ring and af_packet 3 times, Michal shows up.
>>>>
>>>> I don't see many (any?) reasons for using pf_ring, TBH, if you have a modern kernel or a decent network card (Mellanox, Intel, etc). And I still owe the community the article to show how to use the af_packet correctly :/
>>>>
>>>> The case where one has inputs from multiple taps, to multiple network ports will be handled the same way by af_packet, if interfaces are bonded or bridged and by pf_ring. None of them buffers data and processes them at L4 and deals with out of order, etc.
>>>>
>>>> On Wed, Feb 5, 2020 at 2:04 PM Scott Wang <[scwang+bro at sfu.ca](mailto:scwang%2Bbro at sfu.ca)> wrote:
>>>>
>>>>> At the Canarie workshop, Steve Smoot from Corelight suggested using pf_ring still. Any thoughts/comments on switching to af_packet? Advantages vs Disadvantages?
>>>>>
>>>>> Regards,
>>>>> Scott
>>>>>
>>>>>> On Feb 05, 2020, at 12:48, Justin Azoff <justin at corelight.com> wrote:
>>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> It shouldn't be that hard to update to 3.x..
>>>>>>
>>>>>> - bro-pkg should be swapped out with the renamed zkg
>>>>>> - the python2 references can likely be changed to 3
>>>>>> - caf no longer needs to be installed separately
>>>>>> - geoip and databases needs to be swapped out with maxminddb versions, might need a license
>>>>>> - probably worth it to switch to af_packet from pf_ring.. pf_ring was only used initially to easily support capturing directly from both halves of a tap, which might not be a requirement anymore.
>>>>>>
>>>>>> My schedule is a bit crazy for the next week, but once I have some time to work on it I should be able to get things updated pretty quickly.. There's really not much to it.
>>>>>>
>>>>>> On Wed, Feb 5, 2020 at 12:38 PM Paul Sibley <Paul.Sibley at canarie.ca> wrote:
>>>>>>
>>>>>>> Hello Zeek Community,
>>>>>>>
>>>>>>> I am working on a project where Zeek has been deployed in two phases.  During the first phase, some participants used ?https://github.com/ncsa/bro-cluster-in-a-box-setup? script to assist in, and automate a lot of the installation process.
>>>>>>>
>>>>>>> Since then we have entered the phase in our project where more participants have been added, CentOS 8 is preferred, and we are using Zeek 3.0.1.
>>>>>>>
>>>>>>> I wonder if any consideration, or work has been done, in updating the bro-cluster-in-a-box script to work with the updated OS and Zeek version.  Any information would be appreciated.
>>>>>>>
>>>>>>> Thanks in advance,
>>>>>>>
>>>>>>> Paul Sibley
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Zeek mailing list
>>>>>>> zeek at zeek.org
>>>>>>> [http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek](http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek)
>>>>>>
>>>>>> --
>>>>>> Justin
>>>>>> _______________________________________________
>>>>>> Zeek mailing list
>>>>>> zeek at zeek.org
>>>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>>>>
>>>>> _______________________________________________
>>>>> Zeek mailing list
>>>>> zeek at zeek.org
>>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>>
>>> --
>>> Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200206/7dbded4f/attachment.html 

From jsiwek at corelight.com  Thu Feb  6 18:43:35 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 6 Feb 2020 18:43:35 -0800
Subject: [Zeek] Zeek and Broker versions.
In-Reply-To: <BC34BB65-3FFF-4B23-B898-BCAAE019642B@ucsc.edu>
References: <BC34BB65-3FFF-4B23-B898-BCAAE019642B@ucsc.edu>
Message-ID: <CAMzgZ0+WJRP=9+cbEdDAe4HR3udhMMqiHhgB3F=Yp=so7k=GCw@mail.gmail.com>

On Thu, Feb 6, 2020 at 2:16 PM James Hughes <japhughe at ucsc.edu> wrote:

> I am new to this list and would like help debugging a connection from Zeek 3.0.1 to a machine running a broker python application. Broker is locally compiled version 1.2.4. This setup used to work, but now we have upgraded to 3.0.1 and it is no longer working.

>From what version are you upgrading and are both hosts being upgraded
to matching versions?  Zeek 3.0.1 does use Broker 1.2.4 and if the
Python application is also on Broker 1.2.4, that should work.  There
is indeed potential for version mismatches and only recent development
versions help report debug logs and that situation better, so you
might need to do some more crude troubleshooting steps, like starting
from scratch with some smaller toy scripts like this one:

https://docs.zeek.org/projects/broker/en/stable/python.html#exchanging-zeek-events

See if you get it working within a single host, then both hosts, then
add more complexity to make it more similar to the actual scripts that
you found are broken.  If it breaks along the way, you've at least
narrowed it down to something we can try to reproduce and troubleshoot
with you.

Also was thinking if you suspect a version mismatch, it could matter
how you are building Zeek/Broker.  If you use the git repository, and
forget to update git submodules you could accidentally end up with
mismatched Zeek, Broker, or CAF (internal Broker dependency) versions.

- Jon


From clopmz at outlook.com  Thu Feb  6 23:27:08 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Fri, 7 Feb 2020 07:27:08 +0000
Subject: [Zeek] "bro-cluster-in-a-box-setup"
	to	"zeek-cluster-in-a-box-setup"?
In-Reply-To: <CAJ6bFK2+tm_9AeAjsSmSfpaaCqRmAJAkoZr_tTsjbYSMTO+L_A@mail.gmail.com>
References: <ejd4t21slslicoebvb8oui8r.1580994246894@gmail.com>
	<GVUfXwBNh1UMXaM7u9AQ0VIIF6eukO3bqxAUWRFnCZzYYZxOQLEtaQSSeJsQHda2pGP6Fy7FYohylUuHiwRmrwq9uxlwXupUt470sTENgZw=@protonmail.com>,
	<CAJ6bFK2+tm_9AeAjsSmSfpaaCqRmAJAkoZr_tTsjbYSMTO+L_A@mail.gmail.com>
Message-ID: <DB8PR10MB3595C74421ADC7BF40D4DB9CDB1C0@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>

+1.

Regards,
C. L. Martinez


________________________________________
From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> on behalf of Micha? Purzy?ski <michalpurzynski1 at gmail.com>
Sent: 06 February 2020 23:53
To: Justin Hayek
Cc: Paul Sibley; zeek
Subject: Re: [Zeek] "bro-cluster-in-a-box-setup" to     "zeek-cluster-in-a-box-setup"?

Sure, you can run af_packet on any device, including device-made-of-devices, any virtual and physical interface and a combination thereof. The whole af_packet mechanism (they call it "taps" internally) works on a higher level.

Now let's address the elephant in the room, shall we.

IPv4 is correctly hashed on relatively modern kernels (I believe RHEL 7.4 has a fix for that) - so you can use the cluster_flow mode.
IPv6 seems to have problems, sometimes - I can see it correctly hashed most of the time (but not always).

What we do on production, is we let card hash packets by src + dst IP address (and never ports, because fragments don't have port numbers), with the symmetric key, offloading disabled, correct number of queues set and cluster_qm.

If the community is interested I can have an article out in a week - just need to know if there's someone who wants that?

On Thu, Feb 6, 2020 at 2:30 PM Justin Hayek <jdhayek at protonmail.com<mailto:jdhayek at protonmail.com>> wrote:
You can absolutely do this. We are using af_packet and bonded interfaces throughout the majority of our deployments (approximately 1800 sensors).

We decided on af_packet as it was included in recent (at the time 2yrs ago) kernels. I can't speak to non-Debian based distro's, but we haven't seen any issues related to the use of af_packet.

-Justin



??????? Original Message ???????
On Thursday, February 6, 2020 7:04 AM, Joe Blow <blackhole.em at gmail.com<mailto:blackhole.em at gmail.com>> wrote:

Would love to hear this confirmed with no performance issues.

Cheers,

JB

Sent via BlackBerry Hub+ Inbox for Android<http://play.google.com/store/apps/details?id=com.blackberry.hub>
From: justin at corelight.com<mailto:justin at corelight.com>
Sent: February 5, 2020 5:26 PM
To: michalpurzynski1 at gmail.com<mailto:michalpurzynski1 at gmail.com>
Cc: Paul.Sibley at canarie.ca<mailto:Paul.Sibley at canarie.ca>; zeek at zeek.org<mailto:zeek at zeek.org>
Subject: Re: [Zeek] "bro-cluster-in-a-box-setup" to "zeek-cluster-in-a-box-setup"?
OOOH!  You can bond two interfaces together and run af_packet on the bond0 interface? that works?!?

On Wed, Feb 5, 2020 at 5:13 PM Micha? Purzy?ski <michalpurzynski1 at gmail.com<mailto:michalpurzynski1 at gmail.com>> wrote:
There's a law that if you say pf_ring and af_packet 3 times, Michal shows up.

I don't see many (any?) reasons for using pf_ring, TBH, if you have a modern kernel or a decent network card (Mellanox, Intel, etc). And I still owe the community the article to show how to use the af_packet correctly :/

The case where one has inputs from multiple taps, to multiple network ports will be handled the same way by af_packet, if interfaces are bonded or bridged and by pf_ring. None of them buffers data and processes them at L4 and deals with out of order, etc.

On Wed, Feb 5, 2020 at 2:04 PM Scott Wang <scwang+bro at sfu.ca<mailto:scwang%2Bbro at sfu.ca>> wrote:
At the Canarie workshop, Steve Smoot from Corelight suggested using pf_ring still. Any thoughts/comments on switching to af_packet? Advantages vs Disadvantages?

Regards,
Scott

On Feb 05, 2020, at 12:48, Justin Azoff <justin at corelight.com<mailto:justin at corelight.com>> wrote:

Hi!

It shouldn't be that hard to update to 3.x..

- bro-pkg should be swapped out with the renamed zkg
- the python2 references can likely be changed to 3
- caf no longer needs to be installed separately
- geoip and databases needs to be swapped out with maxminddb versions, might need a license
- probably worth it to switch to af_packet from pf_ring.. pf_ring was only used initially to easily support capturing directly from both halves of a tap, which might not be a requirement anymore.

My schedule is a bit crazy for the next week, but once I have some time to work on it I should be able to get things updated pretty quickly.. There's really not much to it.



On Wed, Feb 5, 2020 at 12:38 PM Paul Sibley <Paul.Sibley at canarie.ca<mailto:Paul.Sibley at canarie.ca>> wrote:

Hello Zeek Community,



I am working on a project where Zeek has been deployed in two phases.  During the first phase, some participants used ?https://github.com/ncsa/bro-cluster-in-a-box-setup? script to assist in, and automate a lot of the installation process.

Since then we have entered the phase in our project where more participants have been added, CentOS 8 is preferred, and we are using Zeek 3.0.1.

I wonder if any consideration, or work has been done, in updating the bro-cluster-in-a-box script to work with the updated OS and Zeek version.  Any information would be appreciated.



Thanks in advance,

Paul Sibley

_______________________________________________
Zeek mailing list
zeek at zeek.org<mailto:zeek at zeek.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek<http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek>


--
Justin
_______________________________________________
Zeek mailing list
zeek at zeek.org<mailto:zeek at zeek.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

_______________________________________________
Zeek mailing list
zeek at zeek.org<mailto:zeek at zeek.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


--
Justin



From craig.edgmand at okstate.edu  Fri Feb  7 05:23:45 2020
From: craig.edgmand at okstate.edu (Edgmand, Craig)
Date: Fri, 7 Feb 2020 13:23:45 +0000
Subject: [Zeek] "bro-cluster-in-a-box-setup"
	to	"zeek-cluster-in-a-box-setup"?
In-Reply-To: <CAJ6bFK2+tm_9AeAjsSmSfpaaCqRmAJAkoZr_tTsjbYSMTO+L_A@mail.gmail.com>
References: <ejd4t21slslicoebvb8oui8r.1580994246894@gmail.com>
	<GVUfXwBNh1UMXaM7u9AQ0VIIF6eukO3bqxAUWRFnCZzYYZxOQLEtaQSSeJsQHda2pGP6Fy7FYohylUuHiwRmrwq9uxlwXupUt470sTENgZw=@protonmail.com>
	<CAJ6bFK2+tm_9AeAjsSmSfpaaCqRmAJAkoZr_tTsjbYSMTO+L_A@mail.gmail.com>
Message-ID: <SN6PR03MB3629EA3846DB93C43957010BE31C0@SN6PR03MB3629.namprd03.prod.outlook.com>

+1 on the article.

From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> On Behalf Of Michal Purzynski
Sent: Thursday, February 6, 2020 4:54 PM
To: Justin Hayek <jdhayek at protonmail.com>
Cc: Paul Sibley <Paul.Sibley at canarie.ca>; zeek <zeek at zeek.org>
Subject: Re: [Zeek] "bro-cluster-in-a-box-setup" to "zeek-cluster-in-a-box-setup"?

**External Email - Please verify sender email address before responding.**
Sure, you can run af_packet on any device, including device-made-of-devices, any virtual and physical interface and a combination thereof. The whole af_packet mechanism (they call it "taps" internally) works on a higher level.

Now let's address the elephant in the room, shall we.

IPv4 is correctly hashed on relatively modern kernels (I believe RHEL 7.4 has a fix for that) - so you can use the cluster_flow mode.
IPv6 seems to have problems, sometimes - I can see it correctly hashed most of the time (but not always).

What we do on production, is we let card hash packets by src + dst IP address (and never ports, because fragments don't have port numbers), with the symmetric key, offloading disabled, correct number of queues set and cluster_qm.

If the community is interested I can have an article out in a week - just need to know if there's someone who wants that?

On Thu, Feb 6, 2020 at 2:30 PM Justin Hayek <jdhayek at protonmail.com<mailto:jdhayek at protonmail.com>> wrote:
You can absolutely do this. We are using af_packet and bonded interfaces throughout the majority of our deployments (approximately 1800 sensors).

We decided on af_packet as it was included in recent (at the time 2yrs ago) kernels. I can't speak to non-Debian based distro's, but we haven't seen any issues related to the use of af_packet.

-Justin



??????? Original Message ???????
On Thursday, February 6, 2020 7:04 AM, Joe Blow <blackhole.em at gmail.com<mailto:blackhole.em at gmail.com>> wrote:

Would love to hear this confirmed with no performance issues.

Cheers,

JB

Sent via BlackBerry Hub+ Inbox for Android<https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.blackberry.hub&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828081603&sdata=M%2F1JueQnIyKgRX7jnBU9gH02YALsQaTDDn0Z%2FAbvIk8%3D&reserved=0>
From: justin at corelight.com<mailto:justin at corelight.com>
Sent: February 5, 2020 5:26 PM
To: michalpurzynski1 at gmail.com<mailto:michalpurzynski1 at gmail.com>
Cc: Paul.Sibley at canarie.ca<mailto:Paul.Sibley at canarie.ca>; zeek at zeek.org<mailto:zeek at zeek.org>
Subject: Re: [Zeek] "bro-cluster-in-a-box-setup" to "zeek-cluster-in-a-box-setup"?
OOOH!  You can bond two interfaces together and run af_packet on the bond0 interface? that works?!?

On Wed, Feb 5, 2020 at 5:13 PM Micha? Purzy?ski <michalpurzynski1 at gmail.com<mailto:michalpurzynski1 at gmail.com>> wrote:
There's a law that if you say pf_ring and af_packet 3 times, Michal shows up.

I don't see many (any?) reasons for using pf_ring, TBH, if you have a modern kernel or a decent network card (Mellanox, Intel, etc). And I still owe the community the article to show how to use the af_packet correctly :/

The case where one has inputs from multiple taps, to multiple network ports will be handled the same way by af_packet, if interfaces are bonded or bridged and by pf_ring. None of them buffers data and processes them at L4 and deals with out of order, etc.

On Wed, Feb 5, 2020 at 2:04 PM Scott Wang <scwang+bro at sfu.ca<mailto:scwang%2Bbro at sfu.ca>> wrote:
At the Canarie workshop, Steve Smoot from Corelight suggested using pf_ring still. Any thoughts/comments on switching to af_packet? Advantages vs Disadvantages?

Regards,
Scott

On Feb 05, 2020, at 12:48, Justin Azoff <justin at corelight.com<mailto:justin at corelight.com>> wrote:

Hi!

It shouldn't be that hard to update to 3.x..

- bro-pkg should be swapped out with the renamed zkg
- the python2 references can likely be changed to 3
- caf no longer needs to be installed separately
- geoip and databases needs to be swapped out with maxminddb versions, might need a license
- probably worth it to switch to af_packet from pf_ring.. pf_ring was only used initially to easily support capturing directly from both halves of a tap, which might not be a requirement anymore.

My schedule is a bit crazy for the next week, but once I have some time to work on it I should be able to get things updated pretty quickly.. There's really not much to it.



On Wed, Feb 5, 2020 at 12:38 PM Paul Sibley <Paul.Sibley at canarie.ca<mailto:Paul.Sibley at canarie.ca>> wrote:

Hello Zeek Community,


I am working on a project where Zeek has been deployed in two phases.  During the first phase, some participants used ?https://github.com/ncsa/bro-cluster-in-a-box-setup<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fncsa%2Fbro-cluster-in-a-box-setup&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828091568&sdata=fQfU7TwSzNFS71cpv9lJX%2BgJsBMdjMxBlBKsGumxjM0%3D&reserved=0>? script to assist in, and automate a lot of the installation process.

Since then we have entered the phase in our project where more participants have been added, CentOS 8 is preferred, and we are using Zeek 3.0.1.

I wonder if any consideration, or work has been done, in updating the bro-cluster-in-a-box script to work with the updated OS and Zeek version.  Any information would be appreciated.


Thanks in advance,

Paul Sibley
_______________________________________________
Zeek mailing list
zeek at zeek.org<mailto:zeek at zeek.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek<https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.icsi.berkeley.edu%2Fmailman%2Flistinfo%2Fzeek&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828091568&sdata=GJLJCcUpjnPuIhkuOGUVKOj8pbkiOJjyKbAbgq0H2Nc%3D&reserved=0>


--
Justin
_______________________________________________
Zeek mailing list
zeek at zeek.org<mailto:zeek at zeek.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek<https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.icsi.berkeley.edu%2Fmailman%2Flistinfo%2Fzeek&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828091568&sdata=GJLJCcUpjnPuIhkuOGUVKOj8pbkiOJjyKbAbgq0H2Nc%3D&reserved=0>

_______________________________________________
Zeek mailing list
zeek at zeek.org<mailto:zeek at zeek.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek<https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.icsi.berkeley.edu%2Fmailman%2Flistinfo%2Fzeek&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828101517&sdata=ZSD4CJbQpeb9pgmH6T8WUB8WpyLcEMrNtMmbVqggq1g%3D&reserved=0>


--
Justin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200207/f9c867e9/attachment-0001.html 

From johanna at corelight.com  Fri Feb  7 11:18:06 2020
From: johanna at corelight.com (Johanna Amann)
Date: Fri, 07 Feb 2020 11:18:06 -0800
Subject: [Zeek] Binary packages and 3.1
Message-ID: <E734636A-53E3-4115-8E9B-6D519C187CE6@corelight.com>

Hi Everyone,

with the Zeek 3.1 release around the corner, I just wanted to outline my 
current plan for the binary packages.

As we outlined in 
https://blog.zeek.org/2019/04/new-zeek-release-schedule.html, 3.1 will 
be the first ?feature release? which will exist alongside Zeek 3.0 
(which sill still get patches).

I currently plan to update the ?zeek? package to 3.1, and to 
introduce a new zeek-lts package for people who want to stay on 3.0. The 
zeek-lts package will continue to track 3.0 until zeek 4.0 is released - 
at which point zeek-lts will be updated to 4.0. This means with the 4.0 
release zeek-lts and zeek will essentially be the same package - until 
the release of 4.1 when they will diverge again.

Please let me know if you have any feelings about this - if I don?t 
hear back anything I will create the zeek-lts package in the next few 
days - and write another message about it to this thread.

Johanna

From scwang+bro at sfu.ca  Fri Feb  7 11:29:28 2020
From: scwang+bro at sfu.ca (Scott Wang)
Date: Fri, 7 Feb 2020 11:29:28 -0800
Subject: [Zeek] "bro-cluster-in-a-box-setup"
	to	"zeek-cluster-in-a-box-setup"?
In-Reply-To: <SN6PR03MB3629EA3846DB93C43957010BE31C0@SN6PR03MB3629.namprd03.prod.outlook.com>
References: <ejd4t21slslicoebvb8oui8r.1580994246894@gmail.com>
	<GVUfXwBNh1UMXaM7u9AQ0VIIF6eukO3bqxAUWRFnCZzYYZxOQLEtaQSSeJsQHda2pGP6Fy7FYohylUuHiwRmrwq9uxlwXupUt470sTENgZw=@protonmail.com>
	<CAJ6bFK2+tm_9AeAjsSmSfpaaCqRmAJAkoZr_tTsjbYSMTO+L_A@mail.gmail.com>
	<SN6PR03MB3629EA3846DB93C43957010BE31C0@SN6PR03MB3629.namprd03.prod.outlook.com>
Message-ID: <D1D0D6EB-5B62-4E7B-A833-225BDEFE3E7F@sfu.ca>

+1 on the article (or is it +4 now?)


> On Feb 07, 2020, at 05:23, Edgmand, Craig <craig.edgmand at okstate.edu> wrote:
> 
> +1 on the article.
>  
> From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> On Behalf Of Michal Purzynski
> Sent: Thursday, February 6, 2020 4:54 PM
> To: Justin Hayek <jdhayek at protonmail.com>
> Cc: Paul Sibley <Paul.Sibley at canarie.ca>; zeek <zeek at zeek.org>
> Subject: Re: [Zeek] "bro-cluster-in-a-box-setup" to "zeek-cluster-in-a-box-setup"?
>  
> **External Email - Please verify sender email address before responding.**
> Sure, you can run af_packet on any device, including device-made-of-devices, any virtual and physical interface and a combination thereof. The whole af_packet mechanism (they call it "taps" internally) works on a higher level.
>  
> Now let's address the elephant in the room, shall we.
>  
> IPv4 is correctly hashed on relatively modern kernels (I believe RHEL 7.4 has a fix for that) - so you can use the cluster_flow mode.
> IPv6 seems to have problems, sometimes - I can see it correctly hashed most of the time (but not always).
>  
> What we do on production, is we let card hash packets by src + dst IP address (and never ports, because fragments don't have port numbers), with the symmetric key, offloading disabled, correct number of queues set and cluster_qm.
>  
> If the community is interested I can have an article out in a week - just need to know if there's someone who wants that?
>  
> On Thu, Feb 6, 2020 at 2:30 PM Justin Hayek <jdhayek at protonmail.com <mailto:jdhayek at protonmail.com>> wrote:
> You can absolutely do this. We are using af_packet and bonded interfaces throughout the majority of our deployments (approximately 1800 sensors).
>  
> We decided on af_packet as it was included in recent (at the time 2yrs ago) kernels. I can't speak to non-Debian based distro's, but we haven't seen any issues related to the use of af_packet.
>  
> -Justin
>  
>  
>  
> ??????? Original Message ???????
> On Thursday, February 6, 2020 7:04 AM, Joe Blow <blackhole.em at gmail.com <mailto:blackhole.em at gmail.com>> wrote:
>  
> Would love to hear this confirmed with no performance issues.
>  
> Cheers,
>  
> JB
>  
> Sent via BlackBerry Hub+ Inbox for Android <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.blackberry.hub&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828081603&sdata=M%2F1JueQnIyKgRX7jnBU9gH02YALsQaTDDn0Z%2FAbvIk8%3D&reserved=0>
> From: justin at corelight.com <mailto:justin at corelight.com>
> Sent: February 5, 2020 5:26 PM
> To: michalpurzynski1 at gmail.com <mailto:michalpurzynski1 at gmail.com>
> Cc: Paul.Sibley at canarie.ca <mailto:Paul.Sibley at canarie.ca>; zeek at zeek.org <mailto:zeek at zeek.org>
> Subject: Re: [Zeek] "bro-cluster-in-a-box-setup" to "zeek-cluster-in-a-box-setup"?
> OOOH!  You can bond two interfaces together and run af_packet on the bond0 interface? that works?!?
>  
> On Wed, Feb 5, 2020 at 5:13 PM Micha? Purzy?ski <michalpurzynski1 at gmail.com <mailto:michalpurzynski1 at gmail.com>> wrote:
> There's a law that if you say pf_ring and af_packet 3 times, Michal shows up.
>  
> I don't see many (any?) reasons for using pf_ring, TBH, if you have a modern kernel or a decent network card (Mellanox, Intel, etc). And I still owe the community the article to show how to use the af_packet correctly :/
>  
> The case where one has inputs from multiple taps, to multiple network ports will be handled the same way by af_packet, if interfaces are bonded or bridged and by pf_ring. None of them buffers data and processes them at L4 and deals with out of order, etc.
>  
> On Wed, Feb 5, 2020 at 2:04 PM Scott Wang <scwang+bro at sfu.ca <mailto:scwang%2Bbro at sfu.ca>> wrote:
> At the Canarie workshop, Steve Smoot from Corelight suggested using pf_ring still. Any thoughts/comments on switching to af_packet? Advantages vs Disadvantages?
>  
> Regards,
> Scott
>  
> On Feb 05, 2020, at 12:48, Justin Azoff <justin at corelight.com <mailto:justin at corelight.com>> wrote:
>  
> Hi!
>  
> It shouldn't be that hard to update to 3.x.. 
>  
> - bro-pkg should be swapped out with the renamed zkg
> - the python2 references can likely be changed to 3
> - caf no longer needs to be installed separately
> - geoip and databases needs to be swapped out with maxminddb versions, might need a license
> - probably worth it to switch to af_packet from pf_ring.. pf_ring was only used initially to easily support capturing directly from both halves of a tap, which might not be a requirement anymore.
>  
> My schedule is a bit crazy for the next week, but once I have some time to work on it I should be able to get things updated pretty quickly.. There's really not much to it.
>  
>  
>  
> On Wed, Feb 5, 2020 at 12:38 PM Paul Sibley <Paul.Sibley at canarie.ca <mailto:Paul.Sibley at canarie.ca>> wrote:
> Hello Zeek Community,
> 
>  
> 
> I am working on a project where Zeek has been deployed in two phases.  During the first phase, some participants used ?https://github.com/ncsa/bro-cluster-in-a-box-setup <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fncsa%2Fbro-cluster-in-a-box-setup&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828091568&sdata=fQfU7TwSzNFS71cpv9lJX%2BgJsBMdjMxBlBKsGumxjM0%3D&reserved=0>? script to assist in, and automate a lot of the installation process.
>  
> Since then we have entered the phase in our project where more participants have been added, CentOS 8 is preferred, and we are using Zeek 3.0.1.
>  
> I wonder if any consideration, or work has been done, in updating the bro-cluster-in-a-box script to work with the updated OS and Zeek version.  Any information would be appreciated.
>  
> 
> Thanks in advance,
>  
> Paul Sibley
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org <mailto:zeek at zeek.org>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.icsi.berkeley.edu%2Fmailman%2Flistinfo%2Fzeek&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828091568&sdata=GJLJCcUpjnPuIhkuOGUVKOj8pbkiOJjyKbAbgq0H2Nc%3D&reserved=0>
>  
>  
> --
> Justin
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org <mailto:zeek at zeek.org>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.icsi.berkeley.edu%2Fmailman%2Flistinfo%2Fzeek&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828091568&sdata=GJLJCcUpjnPuIhkuOGUVKOj8pbkiOJjyKbAbgq0H2Nc%3D&reserved=0>
>  
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org <mailto:zeek at zeek.org>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.icsi.berkeley.edu%2Fmailman%2Flistinfo%2Fzeek&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828101517&sdata=ZSD4CJbQpeb9pgmH6T8WUB8WpyLcEMrNtMmbVqggq1g%3D&reserved=0>
>  
>  
> --
> Justin
>  
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200207/5e94f313/attachment-0001.html 

From don.thomas.cissp at gmail.com  Fri Feb  7 12:27:00 2020
From: don.thomas.cissp at gmail.com (Don Thomas)
Date: Fri, 7 Feb 2020 12:27:00 -0800
Subject: [Zeek] Link to the Webcast on using Zeek on a Raspberry Pi
Message-ID: <CABYfsedTnqi4m=6_fgrFtfiW5vp_JEUg5A3YM2mtpa_QwwriKQ@mail.gmail.com>

https://www.youtube.com/watch?time_continue=18&v=vja_H59fh1I

Enjoy,
-dt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200207/993ad834/attachment.html 

From al.kefallonitis at gmail.com  Mon Feb 10 08:17:06 2020
From: al.kefallonitis at gmail.com (Alex Kefallonitis)
Date: Mon, 10 Feb 2020 18:17:06 +0200
Subject: [Zeek] LLMNR/NBT-NS Poisoning and Relay Attacks
Message-ID: <CAHv=MuoJ9NrW=k3Sj_yaC_gSxgW=NjyPpz-PPzJ4SvADjScZcw@mail.gmail.com>

Hi All,

 Any script that can log LLMNR/NBT-NS Poisoning and Relay Attacks ?

Thanks in advanced.

Kind Regards,
Alex Kefallonitis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200210/c2ce8bdb/attachment.html 

From terry.leach at astrolytes.com  Mon Feb 10 09:57:58 2020
From: terry.leach at astrolytes.com (Terry Leach)
Date: Mon, 10 Feb 2020 12:57:58 -0500
Subject: [Zeek] Using Zeek with SIGMA
Message-ID: <CAFyduWgGNuXLTN9Csok_0XPy8wcm41z4XtBtjjmV=hNxLbh50g@mail.gmail.com>

I'm interested in using Zeek for NSM and SIGMA generated rulesets for SIEMs
together.  I'd like to hear from anyone about their experience using both
together for detection. Any feedback welcomed!


Thanks,
-- 
Terry Leach
Astrolytes
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200210/52dba8be/attachment.html 

From jsiwek at corelight.com  Mon Feb 10 11:40:15 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Mon, 10 Feb 2020 11:40:15 -0800
Subject: [Zeek] Zeek 3.1.0 RC1 available
Message-ID: <CAMzgZ0+6mirbz-6nOecZAUX2Vs7T1zemE2QBiGsCJSiDKTp73Q@mail.gmail.com>

A Release Candidate for Zeek 3.1.0 is now available for testing:

  https://www.zeek.org/downloads/zeek-3.1.0-rc1.tar.gz
  https://www.zeek.org/downloads/zeek-3.1.0-rc1.tar.gz.asc

This release primarily adds new features: configuration options,
scripting language functionality, and a new "supervisor" deployment
mode (alternative to ZeekControl). Performance is improved, especially
the handling of SYN-scans and also JSON logging. The main I/O loop of
Zeek is rewritten with better idle behavior and reduced CPU load. It
also includes many smaller bug fixes and improvements.

Further release notes can be read in the NEWS file or here:

  https://github.com/zeek/zeek/releases/tag/v3.1.0-rc1

Bugs found while testing can be reported on GitHub:

  https://github.com/zeek/zeek/issues

Reminder: Zeek 3.0.x is the current Long-Term Support release,
receiving bug fixes until at least October 2020.

From domjevert at gmail.com  Mon Feb 10 17:59:30 2020
From: domjevert at gmail.com (Dominic Evert)
Date: Mon, 10 Feb 2020 15:59:30 -1000
Subject: [Zeek] Parsing Gigamon Source Interface tagging into logs
Message-ID: <CALayXoSNWoa+t6ybVNvtrpXOcYadapbU5JV6Ec3axfezAUxHAQ@mail.gmail.com>

Hello, and good afternoon all,

My team and I currently have several different inline taps on a single
gigamon, due to hardware limitations we're unable to map them
separately and parse the difference in tap locations based on nic in our
sensor stack, so we're currently tagging our traffic using the gigamon,
which appends a 21 byte "header" to the end of each packet varying
dependent on the collection interface. We were hoping someone has run into
a similar issue before, and would be willing to share, before we have to
build/modify a plugin to handle it.
Thank you for your time!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200210/bae608c0/attachment.html 

From phil at brimsecurity.com  Tue Feb 11 15:42:24 2020
From: phil at brimsecurity.com (Phil Rzewski)
Date: Tue, 11 Feb 2020 15:42:24 -0800
Subject: [Zeek] "zq" command-line processor for Zeek logs
Message-ID: <4D066D96-61FD-406A-AE49-ED1CFF63D352@brimsecurity.com>

Zeek community,

We?re writing to let you know about zq <https://github.com/brimsec/zq>, an open source command-line processor for structured logs, built for Zeek. (In fact, we?ve been told zq is ?like zeek-cut on steroids?.)

Those of you who were on the ?Ask the Zeeksperts? call on January 16th saw Seth Hall and Justin Azoff give an early peek of zq (thanks guys!), so this is just an ?official? announcement. Come one, come all!

You can get involved by:
	? Checking out the zq GitHub repo <https://github.com/brimsec/zq> for install info, code, and docs
	? Joining our public Slack <https://join.slack.com/t/brimsec/shared_invite/enQtOTMwMDczODg2ODgyLTk1NTdjOTQxNmI0OGYwOThiYzNlNDc5OWI5NjczZDljNDdmZGZjNGI3NTNiOWRiNzJkMzg4OTEwZWM0Y2NiYWQ> workspace for announcements, Q&A, and to trade query ideas
	? Contacting us directly via email <mailto:info at brimsecurity.com> to schedule a Zoom videoconference

All you need is some Zeek logs (and there?s sample logs <https://github.com/brimsec/zq-sample-data> to help you get started). Here?s just a taste of what?s possible:

- A table of top hosts in a subnet that are experiencing the most SYNs-without-ACK:
   zq -f table "10.164.94.0/24 conn_state=S0 | count() by id.orig_h | sort -r" *

- A regex search for certain HTTP methods, with full events output as NDJSON:
    zq -f ndjson "method=/^(PUT|PATCH|UPDATE)$/" *

- Connections open a long time with low traffic, printed as a Zeek TSV log:
   zq -f zeek "duration>1000 orig_bytes<10 resp_bytes<10" *

Of course, that?s just scratching the surface. Please try it out and let us know what you think on GitHub <https://github.com/brimsec/zq> or Slack <https://join.slack.com/t/brimsec/shared_invite/enQtOTMwMDczODg2ODgyLTk1NTdjOTQxNmI0OGYwOThiYzNlNDc5OWI5NjczZDljNDdmZGZjNGI3NTNiOWRiNzJkMzg4OTEwZWM0Y2NiYWQ>.

Happy hunting, Zeeking, & zq?ing!

--
The Brim team

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200211/55272603/attachment.html 

From michalpurzynski1 at gmail.com  Tue Feb 11 18:47:26 2020
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Tue, 11 Feb 2020 18:47:26 -0800
Subject: [Zeek] "bro-cluster-in-a-box-setup" to
	"zeek-cluster-in-a-box-setup"?
In-Reply-To: <D1D0D6EB-5B62-4E7B-A833-225BDEFE3E7F@sfu.ca>
References: <ejd4t21slslicoebvb8oui8r.1580994246894@gmail.com>
	<GVUfXwBNh1UMXaM7u9AQ0VIIF6eukO3bqxAUWRFnCZzYYZxOQLEtaQSSeJsQHda2pGP6Fy7FYohylUuHiwRmrwq9uxlwXupUt470sTENgZw=@protonmail.com>
	<CAJ6bFK2+tm_9AeAjsSmSfpaaCqRmAJAkoZr_tTsjbYSMTO+L_A@mail.gmail.com>
	<SN6PR03MB3629EA3846DB93C43957010BE31C0@SN6PR03MB3629.namprd03.prod.outlook.com>
	<D1D0D6EB-5B62-4E7B-A833-225BDEFE3E7F@sfu.ca>
Message-ID: <CAJ6bFK2XJyR+NQuGRN6gLRnGN5uu=aj9UCT0a-VqQYaqqEjuYQ@mail.gmail.com>

Give me a week, I already started working on it. I'll be in touch with
Amber. to post it to the official Zeek blog (and only there).


On Fri, Feb 7, 2020 at 11:29 AM Scott Wang <scwang+bro at sfu.ca> wrote:

> +1 on the article (or is it +4 now?)
>
>
> On Feb 07, 2020, at 05:23, Edgmand, Craig <craig.edgmand at okstate.edu>
> wrote:
>
> +1 on the article.
>
>
>
> *From:* zeek-bounces at zeek.org <zeek-bounces at zeek.org> *On Behalf Of *Michal
> Purzynski
> *Sent:* Thursday, February 6, 2020 4:54 PM
> *To:* Justin Hayek <jdhayek at protonmail.com>
> *Cc:* Paul Sibley <Paul.Sibley at canarie.ca>; zeek <zeek at zeek.org>
> *Subject:* Re: [Zeek] "bro-cluster-in-a-box-setup" to
> "zeek-cluster-in-a-box-setup"?
>
>
>
> **External Email - Please verify sender email address before responding.**
>
> Sure, you can run af_packet on any device, including
> device-made-of-devices, any virtual and physical interface and a
> combination thereof. The whole af_packet mechanism (they call it "taps"
> internally) works on a higher level.
>
>
>
> Now let's address the elephant in the room, shall we.
>
>
>
> IPv4 is correctly hashed on relatively modern kernels (I believe RHEL 7.4
> has a fix for that) - so you can use the cluster_flow mode.
>
> IPv6 seems to have problems, sometimes - I can see it correctly hashed
> most of the time (but not always).
>
>
>
> What we do on production, is we let card hash packets by src + dst IP
> address (and never ports, because fragments don't have port numbers), with
> the symmetric key, offloading disabled, correct number of queues set and
> cluster_qm.
>
>
>
> If the community is interested I can have an article out in a week - just
> need to know if there's someone who wants that?
>
>
>
> On Thu, Feb 6, 2020 at 2:30 PM Justin Hayek <jdhayek at protonmail.com>
> wrote:
>
> You can absolutely do this. We are using af_packet and bonded interfaces
> throughout the majority of our deployments (approximately 1800 sensors).
>
>
>
> We decided on af_packet as it was included in recent (at the time 2yrs
> ago) kernels. I can't speak to non-Debian based distro's, but we haven't
> seen any issues related to the use of af_packet.
>
>
>
> -Justin
>
>
>
>
>
>
>
> ??????? Original Message ???????
>
> On Thursday, February 6, 2020 7:04 AM, Joe Blow <blackhole.em at gmail.com>
> wrote:
>
>
>
> Would love to hear this confirmed with no performance issues.
>
>
>
> Cheers,
>
>
>
> JB
>
>
>
> Sent via BlackBerry Hub+ Inbox for Android
> <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.blackberry.hub&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828081603&sdata=M%2F1JueQnIyKgRX7jnBU9gH02YALsQaTDDn0Z%2FAbvIk8%3D&reserved=0>
>
> *From:* justin at corelight.com
>
> *Sent:* February 5, 2020 5:26 PM
>
> *To:* michalpurzynski1 at gmail.com
>
> *Cc:* Paul.Sibley at canarie.ca; zeek at zeek.org
>
> *Subject:* Re: [Zeek] "bro-cluster-in-a-box-setup" to
> "zeek-cluster-in-a-box-setup"?
>
> OOOH!  You can bond two interfaces together and run af_packet on the bond0
> interface? that works?!?
>
>
>
> On Wed, Feb 5, 2020 at 5:13 PM Micha? Purzy?ski <
> michalpurzynski1 at gmail.com> wrote:
>
> There's a law that if you say pf_ring and af_packet 3 times, Michal shows
> up.
>
>
>
> I don't see many (any?) reasons for using pf_ring, TBH, if you have a
> modern kernel or a decent network card (Mellanox, Intel, etc). And I still
> owe the community the article to show how to use the af_packet correctly :/
>
>
>
> The case where one has inputs from multiple taps, to multiple network
> ports will be handled the same way by af_packet, if interfaces are bonded
> or bridged and by pf_ring. None of them buffers data and processes them at
> L4 and deals with out of order, etc.
>
>
>
> On Wed, Feb 5, 2020 at 2:04 PM Scott Wang <scwang+bro at sfu.ca> wrote:
>
> At the Canarie workshop, Steve Smoot from Corelight suggested using
> pf_ring still. Any thoughts/comments on switching to af_packet? Advantages
> vs Disadvantages?
>
>
>
> Regards,
>
> Scott
>
>
>
> On Feb 05, 2020, at 12:48, Justin Azoff <justin at corelight.com> wrote:
>
>
>
> Hi!
>
>
>
> It shouldn't be that hard to update to 3.x..
>
>
>
> - bro-pkg should be swapped out with the renamed zkg
>
> - the python2 references can likely be changed to 3
>
> - caf no longer needs to be installed separately
>
> - geoip and databases needs to be swapped out with maxminddb versions,
> might need a license
>
> - probably worth it to switch to af_packet from pf_ring.. pf_ring was only
> used initially to easily support capturing directly from both halves of a
> tap, which might not be a requirement anymore.
>
>
>
> My schedule is a bit crazy for the next week, but once I have some time to
> work on it I should be able to get things updated pretty quickly.. There's
> really not much to it.
>
>
>
>
>
>
>
> On Wed, Feb 5, 2020 at 12:38 PM Paul Sibley <Paul.Sibley at canarie.ca>
> wrote:
>
> Hello Zeek Community,
>
>
>
> I am working on a project where Zeek has been deployed in two phases.
> During the first phase, some participants used ?
> https://github.com/ncsa/bro-cluster-in-a-box-setup
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fncsa%2Fbro-cluster-in-a-box-setup&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828091568&sdata=fQfU7TwSzNFS71cpv9lJX%2BgJsBMdjMxBlBKsGumxjM0%3D&reserved=0>?
> script to assist in, and automate a lot of the installation process.
>
>
>
> Since then we have entered the phase in our project where more
> participants have been added, CentOS 8 is preferred, and we are using Zeek
> 3.0.1.
>
>
>
> I wonder if any consideration, or work has been done, in updating the
> bro-cluster-in-a-box script to work with the updated OS and Zeek version.
> Any information would be appreciated.
>
>
>
> Thanks in advance,
>
>
>
> Paul Sibley
>
> _______________________________________________
>
> Zeek mailing list
>
> zeek at zeek.org
>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.icsi.berkeley.edu%2Fmailman%2Flistinfo%2Fzeek&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828091568&sdata=GJLJCcUpjnPuIhkuOGUVKOj8pbkiOJjyKbAbgq0H2Nc%3D&reserved=0>
>
>
>
>
>
> --
>
> Justin
>
> _______________________________________________
>
> Zeek mailing list
>
> zeek at zeek.org
>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.icsi.berkeley.edu%2Fmailman%2Flistinfo%2Fzeek&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828091568&sdata=GJLJCcUpjnPuIhkuOGUVKOj8pbkiOJjyKbAbgq0H2Nc%3D&reserved=0>
>
>
>
> _______________________________________________
>
> Zeek mailing list
>
> zeek at zeek.org
>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.icsi.berkeley.edu%2Fmailman%2Flistinfo%2Fzeek&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828101517&sdata=ZSD4CJbQpeb9pgmH6T8WUB8WpyLcEMrNtMmbVqggq1g%3D&reserved=0>
>
>
>
>
>
> --
>
> Justin
>
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200211/8f7892c8/attachment-0001.html 

From jdickenson at gmail.com  Tue Feb 11 21:32:56 2020
From: jdickenson at gmail.com (James Dickenson)
Date: Tue, 11 Feb 2020 21:32:56 -0800
Subject: [Zeek] Using Zeek with SIGMA
In-Reply-To: <CAFyduWgGNuXLTN9Csok_0XPy8wcm41z4XtBtjjmV=hNxLbh50g@mail.gmail.com>
References: <CAFyduWgGNuXLTN9Csok_0XPy8wcm41z4XtBtjjmV=hNxLbh50g@mail.gmail.com>
Message-ID: <CACovUX+uwYjga87tvu0k7RzdZR35QeVVQ1bsBNnmuPdmWt=Scw@mail.gmail.com>

Sigma is awesome to use and works well with Zeek logs in my opinion.  I've
only written a few sigma detections for Zeek but it's basically the same
process as creating any other sigma detection.  Identify what fields/values
that are of interest in the log and add those as selection criteria in the
sigma rule. Additionally you may want to write a sigma log source config to
map Zeek to the appropriate fields for the target SIEM.  There are some
good writes up on how to write sigma rules if you haven't done so before, I
would also add that you will save yourself a lot of
head-banging/frustration if you use a text editor that supports a yaml
linter like VS code or Atom.


-James

On Mon, Feb 10, 2020 at 10:04 AM Terry Leach <terry.leach at astrolytes.com>
wrote:

> I'm interested in using Zeek for NSM and SIGMA generated rulesets for
> SIEMs together.  I'd like to hear from anyone about their experience using
> both together for detection. Any feedback welcomed!
>
>
> Thanks,
> --
> Terry Leach
> Astrolytes
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200211/78cf2b13/attachment.html 

From neslog at gmail.com  Wed Feb 12 10:26:33 2020
From: neslog at gmail.com (Neslog)
Date: Wed, 12 Feb 2020 13:26:33 -0500
Subject: [Zeek] Noise Protocol Framework.
Message-ID: <CALgfRGJed=Qfm92iTasRAODpm6_VMG+27hSTyGv77y965j0_gA@mail.gmail.com>

Hi Zeekers!

Has anyone done work on identifying the Noise Protocol Framework with
Zeek?  I have an edge case I'd like to pursue and wanted to see what has
been done already.

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200212/3ace124e/attachment.html 

From christopher.hobbs at corelight.com  Wed Feb 12 14:14:01 2020
From: christopher.hobbs at corelight.com (Christopher Hobbs)
Date: Wed, 12 Feb 2020 14:14:01 -0800
Subject: [Zeek] Noise Protocol Framework.
In-Reply-To: <CALgfRGJed=Qfm92iTasRAODpm6_VMG+27hSTyGv77y965j0_gA@mail.gmail.com>
References: <CALgfRGJed=Qfm92iTasRAODpm6_VMG+27hSTyGv77y965j0_gA@mail.gmail.com>
Message-ID: <CAN6ZtinLgOMz3HHzGF2cY2oVOXZwnvCcsYCgFwHdbHwJ3oVtUw@mail.gmail.com>

I'm not aware of any offhand but it's certainly traffic and scripting
that I would find interesting.  Are there any public pcaps available?
Please send the list updates you find anything useful.

Thank you!
cmh

On Wed, Feb 12, 2020 at 10:37 AM Neslog <neslog at gmail.com> wrote:
>
> Hi Zeekers!
>
> Has anyone done work on identifying the Noise Protocol Framework with Zeek?  I have an edge case I'd like to pursue and wanted to see what has been done already.
>
> Thanks!
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

From brian at corelight.com  Wed Feb 12 14:33:12 2020
From: brian at corelight.com (Brian Dye)
Date: Wed, 12 Feb 2020 14:33:12 -0800
Subject: [Zeek] Using Zeek with SIGMA
In-Reply-To: <CACovUX+uwYjga87tvu0k7RzdZR35QeVVQ1bsBNnmuPdmWt=Scw@mail.gmail.com>
References: <CAFyduWgGNuXLTN9Csok_0XPy8wcm41z4XtBtjjmV=hNxLbh50g@mail.gmail.com>
	<CACovUX+uwYjga87tvu0k7RzdZR35QeVVQ1bsBNnmuPdmWt=Scw@mail.gmail.com>
Message-ID: <CAEC=iOuGueAT1V7aNHYKMWVO64mkpauUL5fPRf2ix=Uw-9=QNw@mail.gmail.com>

As a quick add to this, we've got work in flight to map the Zeek fields in
to the Sigma sources. Will be contributing that, so while it isn't ready
yet looking forward to sharing when ready (no ETA yet, sorry - but work is
in flight at least).

On Tue, Feb 11, 2020 at 9:34 PM James Dickenson <jdickenson at gmail.com>
wrote:

> Sigma is awesome to use and works well with Zeek logs in my opinion.  I've
> only written a few sigma detections for Zeek but it's basically the same
> process as creating any other sigma detection.  Identify what fields/values
> that are of interest in the log and add those as selection criteria in the
> sigma rule. Additionally you may want to write a sigma log source config to
> map Zeek to the appropriate fields for the target SIEM.  There are some
> good writes up on how to write sigma rules if you haven't done so before, I
> would also add that you will save yourself a lot of
> head-banging/frustration if you use a text editor that supports a yaml
> linter like VS code or Atom.
>
>
> -James
>
> On Mon, Feb 10, 2020 at 10:04 AM Terry Leach <terry.leach at astrolytes.com>
> wrote:
>
>> I'm interested in using Zeek for NSM and SIGMA generated rulesets for
>> SIEMs together.  I'd like to hear from anyone about their experience using
>> both together for detection. Any feedback welcomed!
>>
>>
>> Thanks,
>> --
>> Terry Leach
>> Astrolytes
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200212/fca20415/attachment.html 

From terry.leach at astrolytes.com  Thu Feb 13 08:02:15 2020
From: terry.leach at astrolytes.com (Terry Leach)
Date: Thu, 13 Feb 2020 11:02:15 -0500
Subject: [Zeek] Using Zeek with SIGMA
In-Reply-To: <CAEC=iOuGueAT1V7aNHYKMWVO64mkpauUL5fPRf2ix=Uw-9=QNw@mail.gmail.com>
References: <CAFyduWgGNuXLTN9Csok_0XPy8wcm41z4XtBtjjmV=hNxLbh50g@mail.gmail.com>
	<CACovUX+uwYjga87tvu0k7RzdZR35QeVVQ1bsBNnmuPdmWt=Scw@mail.gmail.com>
	<CAEC=iOuGueAT1V7aNHYKMWVO64mkpauUL5fPRf2ix=Uw-9=QNw@mail.gmail.com>
Message-ID: <CAFyduWjiMMJxcOQvyEsUFv3VHEPnHcN9sVrSQe2HkAYuXZOsLQ@mail.gmail.com>

WOW! Thank you both for the update.

On Wed, Feb 12, 2020 at 5:33 PM Brian Dye <brian at corelight.com> wrote:

> As a quick add to this, we've got work in flight to map the Zeek fields in
> to the Sigma sources. Will be contributing that, so while it isn't ready
> yet looking forward to sharing when ready (no ETA yet, sorry - but work is
> in flight at least).
>
> On Tue, Feb 11, 2020 at 9:34 PM James Dickenson <jdickenson at gmail.com>
> wrote:
>
>> Sigma is awesome to use and works well with Zeek logs in my opinion.
>> I've only written a few sigma detections for Zeek but it's basically the
>> same process as creating any other sigma detection.  Identify what
>> fields/values that are of interest in the log and add those as selection
>> criteria in the sigma rule. Additionally you may want to write a sigma log
>> source config to map Zeek to the appropriate fields for the target SIEM.
>> There are some good writes up on how to write sigma rules if you haven't
>> done so before, I would also add that you will save yourself a lot of
>> head-banging/frustration if you use a text editor that supports a yaml
>> linter like VS code or Atom.
>>
>>
>> -James
>>
>> On Mon, Feb 10, 2020 at 10:04 AM Terry Leach <terry.leach at astrolytes.com>
>> wrote:
>>
>>> I'm interested in using Zeek for NSM and SIGMA generated rulesets for
>>> SIEMs together.  I'd like to hear from anyone about their experience using
>>> both together for detection. Any feedback welcomed!
>>>
>>>
>>> Thanks,
>>> --
>>> Terry Leach
>>> Astrolytes
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>

-- 
Terry Leach
Astrolytes
202-670-0882
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200213/e742518f/attachment.html 

From jdickenson at gmail.com  Thu Feb 13 08:45:39 2020
From: jdickenson at gmail.com (James Dickenson)
Date: Thu, 13 Feb 2020 08:45:39 -0800
Subject: [Zeek] Using Zeek with SIGMA
In-Reply-To: <CAFyduWjiMMJxcOQvyEsUFv3VHEPnHcN9sVrSQe2HkAYuXZOsLQ@mail.gmail.com>
References: <CAFyduWgGNuXLTN9Csok_0XPy8wcm41z4XtBtjjmV=hNxLbh50g@mail.gmail.com>
	<CACovUX+uwYjga87tvu0k7RzdZR35QeVVQ1bsBNnmuPdmWt=Scw@mail.gmail.com>
	<CAEC=iOuGueAT1V7aNHYKMWVO64mkpauUL5fPRf2ix=Uw-9=QNw@mail.gmail.com>
	<CAFyduWjiMMJxcOQvyEsUFv3VHEPnHcN9sVrSQe2HkAYuXZOsLQ@mail.gmail.com>
Message-ID: <CACovUX+c=2nDyeRSt3cYOL8+cJVoXofA7yDEUiWBcjQDtpsTCQ@mail.gmail.com>

I felt bad that there wasn't any rules yet in Sigma rule repository for
Zeek so I added a rule for Kerberos TGS requests  with rc4-hmac cipher
yesterday that looks like it got merged.  Hopefully you find it helpful.

I'm looking forward to the Corelight team's contributions to Sigma as well!

-James

On Thu, Feb 13, 2020 at 8:02 AM Terry Leach <terry.leach at astrolytes.com>
wrote:

> WOW! Thank you both for the update.
>
> On Wed, Feb 12, 2020 at 5:33 PM Brian Dye <brian at corelight.com> wrote:
>
>> As a quick add to this, we've got work in flight to map the Zeek fields
>> in to the Sigma sources. Will be contributing that, so while it isn't ready
>> yet looking forward to sharing when ready (no ETA yet, sorry - but work is
>> in flight at least).
>>
>> On Tue, Feb 11, 2020 at 9:34 PM James Dickenson <jdickenson at gmail.com>
>> wrote:
>>
>>> Sigma is awesome to use and works well with Zeek logs in my opinion.
>>> I've only written a few sigma detections for Zeek but it's basically the
>>> same process as creating any other sigma detection.  Identify what
>>> fields/values that are of interest in the log and add those as selection
>>> criteria in the sigma rule. Additionally you may want to write a sigma log
>>> source config to map Zeek to the appropriate fields for the target SIEM.
>>> There are some good writes up on how to write sigma rules if you haven't
>>> done so before, I would also add that you will save yourself a lot of
>>> head-banging/frustration if you use a text editor that supports a yaml
>>> linter like VS code or Atom.
>>>
>>>
>>> -James
>>>
>>> On Mon, Feb 10, 2020 at 10:04 AM Terry Leach <terry.leach at astrolytes.com>
>>> wrote:
>>>
>>>> I'm interested in using Zeek for NSM and SIGMA generated rulesets for
>>>> SIEMs together.  I'd like to hear from anyone about their experience using
>>>> both together for detection. Any feedback welcomed!
>>>>
>>>>
>>>> Thanks,
>>>> --
>>>> Terry Leach
>>>> Astrolytes
>>>> _______________________________________________
>>>> Zeek mailing list
>>>> zeek at zeek.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>>
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>>
>
> --
> Terry Leach
> Astrolytes
> 202-670-0882
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200213/9f211ada/attachment-0001.html 

From akgraner at corelight.com  Thu Feb 13 10:47:28 2020
From: akgraner at corelight.com (Amber Graner)
Date: Thu, 13 Feb 2020 10:47:28 -0800
Subject: [Zeek] [Reminder] - Ask The Zeeksperts - 13 Feb 2020 - 12:30pm PST
Message-ID: <CAJhOzurYo6YtPR8wOLfvYene_U=6c+oQAOJKKFdC61U=SEMNbw@mail.gmail.com>

Hi all,

Aashish Sharma will be hosting today's ask the Zeeksperts call. If you
your're a Threat Hunter or Incident Responder who uses Zeek, Aashish is a
great person for you to drop into the call and speak with.

To join the webinar you must register - here's the link for today's call:
https://register.gotowebinar.com/register/7036660417675395595

Hope you'll join us!

Thanks,
~Amber

-- 

*Amber Graner*
Director of Community
Corelight, Inc

828.582.9469

Schedule time on my calendar here. <https://calendly.com/amber_graner>



 * Ask me about how you can participate in the Zeek (formerly Bro)
community.
 * Remember - ZEEK AND YOU SHALL FIND!!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200213/8d879b5d/attachment.html 

From glwallum at gmail.com  Thu Feb 13 11:20:45 2020
From: glwallum at gmail.com (Gordon Wallum)
Date: Thu, 13 Feb 2020 12:20:45 -0700
Subject: [Zeek] scan.zeek question - exclude IP addresses
Message-ID: <CAHs9D0sdM0yk4pYWp_E=K-kp1w5ptcVtUJpWmJoRH1RaPQBxSA@mail.gmail.com>

Hello!

I am new with Zeek and looking to learn more. I am currently using the
scan.zeek script (
https://github.com/zeek/zeek/blob/master/scripts/policy/misc/scan.zeek) for
port scanning detection.

I want to exclude certain source IP addresses from this script but I am not
sure the best way to do so. It seems like a comparison with the key$host
variable, but not sure where or how to do this logic in Zeek.

Any advice would be appreciated

Thank you
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200213/bbfb678f/attachment.html 

From christopher.hobbs at corelight.com  Thu Feb 13 11:46:00 2020
From: christopher.hobbs at corelight.com (Christopher Hobbs)
Date: Thu, 13 Feb 2020 11:46:00 -0800
Subject: [Zeek] scan.zeek question - exclude IP addresses
In-Reply-To: <CAHs9D0sdM0yk4pYWp_E=K-kp1w5ptcVtUJpWmJoRH1RaPQBxSA@mail.gmail.com>
References: <CAHs9D0sdM0yk4pYWp_E=K-kp1w5ptcVtUJpWmJoRH1RaPQBxSA@mail.gmail.com>
Message-ID: <CAN6ZtimT4B32iHzsZhO8bXeQzKvE=0si2QyjkSD43OVS3i+t0g@mail.gmail.com>

I don't believe it has that functionality at the moment but I have a
patch that can provide those options.  I'll put it on GH when I get a
spare moment.

It's worth noting that scan.zeek can perform poorly under heavy load
so maybe have a look at bro-simple-scan as well?
https://github.com/ncsa/bro-simple-scan

cmh

On Thu, Feb 13, 2020 at 11:23 AM Gordon Wallum <glwallum at gmail.com> wrote:
>
> Hello!
>
> I am new with Zeek and looking to learn more. I am currently using the scan.zeek script (https://github.com/zeek/zeek/blob/master/scripts/policy/misc/scan.zeek) for port scanning detection.
>
> I want to exclude certain source IP addresses from this script but I am not sure the best way to do so. It seems like a comparison with the key$host variable, but not sure where or how to do this logic in Zeek.
>
> Any advice would be appreciated
>
> Thank you
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

From glwallum at gmail.com  Thu Feb 13 12:10:44 2020
From: glwallum at gmail.com (Gordon Wallum)
Date: Thu, 13 Feb 2020 13:10:44 -0700
Subject: [Zeek] scan.zeek question - exclude IP addresses
In-Reply-To: <CAN6ZtimT4B32iHzsZhO8bXeQzKvE=0si2QyjkSD43OVS3i+t0g@mail.gmail.com>
References: <CAHs9D0sdM0yk4pYWp_E=K-kp1w5ptcVtUJpWmJoRH1RaPQBxSA@mail.gmail.com>
	<CAN6ZtimT4B32iHzsZhO8bXeQzKvE=0si2QyjkSD43OVS3i+t0g@mail.gmail.com>
Message-ID: <CAHs9D0s3GkSSXRrK2Z6iD5KQgeGwDd0S4pgYsnYkB9AUpeSvrQ@mail.gmail.com>

This Christopher, would something like this work for the scan.zeek exclude?
I'll look at the bro simple scan now

At line 71:

local message=fmt("%s scanned at least %d unique hosts on port %s in %s",
key$host, r$unique, key$str, dur);
local exclude_ips: set[addr] = { xxx.xxx.xxx.xxx, xxx.xxx.xxx.yyy,
xxx.xxx.xxx.zzz } ;
if (key$host !in exclude_ips)
NOTICE([$note=Address_Scan,
$src=key$host,
$p=to_port(key$str),
$sub=side,
$msg=message,
$identifier=cat(key$host)]);


On Thu, Feb 13, 2020 at 12:46 PM Christopher Hobbs <
christopher.hobbs at corelight.com> wrote:

> I don't believe it has that functionality at the moment but I have a
> patch that can provide those options.  I'll put it on GH when I get a
> spare moment.
>
> It's worth noting that scan.zeek can perform poorly under heavy load
> so maybe have a look at bro-simple-scan as well?
> https://github.com/ncsa/bro-simple-scan
>
> cmh
>
> On Thu, Feb 13, 2020 at 11:23 AM Gordon Wallum <glwallum at gmail.com> wrote:
> >
> > Hello!
> >
> > I am new with Zeek and looking to learn more. I am currently using the
> scan.zeek script (
> https://github.com/zeek/zeek/blob/master/scripts/policy/misc/scan.zeek)
> for port scanning detection.
> >
> > I want to exclude certain source IP addresses from this script but I am
> not sure the best way to do so. It seems like a comparison with the
> key$host variable, but not sure where or how to do this logic in Zeek.
> >
> > Any advice would be appreciated
> >
> > Thank you
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200213/e29fec9d/attachment.html 

From terry.leach at astrolytes.com  Thu Feb 13 12:30:06 2020
From: terry.leach at astrolytes.com (Terry Leach)
Date: Thu, 13 Feb 2020 15:30:06 -0500
Subject: [Zeek] Using Zeek with SIGMA
In-Reply-To: <CACovUX+c=2nDyeRSt3cYOL8+cJVoXofA7yDEUiWBcjQDtpsTCQ@mail.gmail.com>
References: <CAFyduWgGNuXLTN9Csok_0XPy8wcm41z4XtBtjjmV=hNxLbh50g@mail.gmail.com>
	<CACovUX+uwYjga87tvu0k7RzdZR35QeVVQ1bsBNnmuPdmWt=Scw@mail.gmail.com>
	<CAEC=iOuGueAT1V7aNHYKMWVO64mkpauUL5fPRf2ix=Uw-9=QNw@mail.gmail.com>
	<CAFyduWjiMMJxcOQvyEsUFv3VHEPnHcN9sVrSQe2HkAYuXZOsLQ@mail.gmail.com>
	<CACovUX+c=2nDyeRSt3cYOL8+cJVoXofA7yDEUiWBcjQDtpsTCQ@mail.gmail.com>
Message-ID: <CAFyduWh23xh4+qq+a+99ffXEK+yDY-3cHF=nErbWABnEjFf2RA@mail.gmail.com>

Awesome, thanks for the update!

On Thu, Feb 13, 2020, 11:45 James Dickenson <jdickenson at gmail.com> wrote:

> I felt bad that there wasn't any rules yet in Sigma rule repository for
> Zeek so I added a rule for Kerberos TGS requests  with rc4-hmac cipher
> yesterday that looks like it got merged.  Hopefully you find it helpful.
>
> I'm looking forward to the Corelight team's contributions to Sigma as
> well!
>
> -James
>
> On Thu, Feb 13, 2020 at 8:02 AM Terry Leach <terry.leach at astrolytes.com>
> wrote:
>
>> WOW! Thank you both for the update.
>>
>> On Wed, Feb 12, 2020 at 5:33 PM Brian Dye <brian at corelight.com> wrote:
>>
>>> As a quick add to this, we've got work in flight to map the Zeek fields
>>> in to the Sigma sources. Will be contributing that, so while it isn't ready
>>> yet looking forward to sharing when ready (no ETA yet, sorry - but work is
>>> in flight at least).
>>>
>>> On Tue, Feb 11, 2020 at 9:34 PM James Dickenson <jdickenson at gmail.com>
>>> wrote:
>>>
>>>> Sigma is awesome to use and works well with Zeek logs in my opinion.
>>>> I've only written a few sigma detections for Zeek but it's basically the
>>>> same process as creating any other sigma detection.  Identify what
>>>> fields/values that are of interest in the log and add those as selection
>>>> criteria in the sigma rule. Additionally you may want to write a sigma log
>>>> source config to map Zeek to the appropriate fields for the target SIEM.
>>>> There are some good writes up on how to write sigma rules if you haven't
>>>> done so before, I would also add that you will save yourself a lot of
>>>> head-banging/frustration if you use a text editor that supports a yaml
>>>> linter like VS code or Atom.
>>>>
>>>>
>>>> -James
>>>>
>>>> On Mon, Feb 10, 2020 at 10:04 AM Terry Leach <
>>>> terry.leach at astrolytes.com> wrote:
>>>>
>>>>> I'm interested in using Zeek for NSM and SIGMA generated rulesets for
>>>>> SIEMs together.  I'd like to hear from anyone about their experience using
>>>>> both together for detection. Any feedback welcomed!
>>>>>
>>>>>
>>>>> Thanks,
>>>>> --
>>>>> Terry Leach
>>>>> Astrolytes
>>>>> _______________________________________________
>>>>> Zeek mailing list
>>>>> zeek at zeek.org
>>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>>>
>>>> _______________________________________________
>>>> Zeek mailing list
>>>> zeek at zeek.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>>
>>>
>>
>> --
>> Terry Leach
>> Astrolytes
>> 202-670-0882
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200213/f4f862e7/attachment.html 

From michalpurzynski1 at gmail.com  Thu Feb 13 13:49:09 2020
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Thu, 13 Feb 2020 13:49:09 -0800
Subject: [Zeek] scan.zeek question - exclude IP addresses
In-Reply-To: <CAHs9D0sdM0yk4pYWp_E=K-kp1w5ptcVtUJpWmJoRH1RaPQBxSA@mail.gmail.com>
References: <CAHs9D0sdM0yk4pYWp_E=K-kp1w5ptcVtUJpWmJoRH1RaPQBxSA@mail.gmail.com>
Message-ID: <CAJ6bFK3p1HWyJePwpMhZJm4-fFVWeqr4GnCYffGJBEo6cjn6HA@mail.gmail.com>

A couple of things.

First, you should use Justin's simple-scan. As others have pointed out, the
stock scanning detection script can behave poorly and it's hardly
extensible.

https://github.com/ncsa/bro-simple-scan

(it's also packaged)

Second - you can either ignore connections so the detection algorithm won't
count them (with the hook from the simple-scan code), or you can write a
notice policy and ignore some notices. Up to you - we just ignore some
connections.

Inside Justin's package, you will find a hook - this is what we use to
ignore a set of destination and source IP addresses and some destination
ports

https://github.com/ncsa/bro-simple-scan/blob/master/scripts/scan.bro#L87

Here's how we use that hook

https://gist.github.com/mpurzynski/96a26c42874898447554531b6df9a4bb

The input framework is what we use to update the list runtime. Nowadays you
could use the configuration framework instead.

https://corelight.blog/2018/02/13/runtime-options-the-bro-configuration-framework/


Either way, you do not have to modify any upstream package.


On Thu, Feb 13, 2020 at 11:23 AM Gordon Wallum <glwallum at gmail.com> wrote:

> Hello!
>
> I am new with Zeek and looking to learn more. I am currently using the
> scan.zeek script (
> https://github.com/zeek/zeek/blob/master/scripts/policy/misc/scan.zeek)
> for port scanning detection.
>
> I want to exclude certain source IP addresses from this script but I am
> not sure the best way to do so. It seems like a comparison with the
> key$host variable, but not sure where or how to do this logic in Zeek.
>
> Any advice would be appreciated
>
> Thank you
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200213/adc26d56/attachment.html 

From franky.meier.1 at gmx.de  Thu Feb 13 23:33:04 2020
From: franky.meier.1 at gmx.de (Frank Meier)
Date: Fri, 14 Feb 2020 08:33:04 +0100
Subject: [Zeek] add uid of ftp connection to conn_uids in files.log
Message-ID: <trinity-a4ab458d-89f7-4012-b51b-c2758c6b7950-1581665584841@3c-app-gmx-bap19>

An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200214/d18227ca/attachment-0001.html 

From slantin at cegepbc.ca  Fri Feb 14 05:51:40 2020
From: slantin at cegepbc.ca (=?iso-8859-1?Q?Lantin=2C_St=E9phane?=)
Date: Fri, 14 Feb 2020 13:51:40 +0000
Subject: [Zeek] Zeek 3 / Vmware ESXi
Message-ID: <YT1PR01MB3227F8235871985C478833C4C0150@YT1PR01MB3227.CANPRD01.PROD.OUTLOOK.COM>

Hello,

I have installed Zeek on ESXi and the hardware is dedicated for Zeek.
Everything works fine, Zeek logs the network.

My question is, what difference is for Zeek to be run on virtual vs physical ?
Our highest throughput is 125mbps for our ISP and about 1Gpbs internal.

Some suggests it might impact the performance, but where it could struggle on VM?

Thank you for your time,

_____________________________
St?phane Lantin
Technicien en informatique - C.P
Service des technologies de l'information
C?gep de Baie-Comeau
(418) 589-5707 poste 178
slantin at cegepbc.ca<mailto:slantin at cegepbc.ca>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200214/2949af9c/attachment.html 

From glwallum at gmail.com  Fri Feb 14 08:20:21 2020
From: glwallum at gmail.com (Gordon Wallum)
Date: Fri, 14 Feb 2020 09:20:21 -0700
Subject: [Zeek] scan.zeek question - exclude IP addresses
In-Reply-To: <CAJ6bFK3p1HWyJePwpMhZJm4-fFVWeqr4GnCYffGJBEo6cjn6HA@mail.gmail.com>
References: <CAHs9D0sdM0yk4pYWp_E=K-kp1w5ptcVtUJpWmJoRH1RaPQBxSA@mail.gmail.com>
	<CAJ6bFK3p1HWyJePwpMhZJm4-fFVWeqr4GnCYffGJBEo6cjn6HA@mail.gmail.com>
Message-ID: <CAHs9D0tcJFHgAMwFig54Aouj3J1+2kHUpuMkk-Ke3nfVx_8jVA@mail.gmail.com>

Thank you, this makes sense logically but I can't figure out how to use the
hook to exclude. The code below throws an error

 ## Override this hook to ignore particular scan connections
    global Scan::scan_policy: hook(scanner: addr, victim: addr,
scanned_port: port)
{
            if (( victim in exvictim_ips) || ( scanner in exscanner_ips )
|| ( scanned_port in exscanned_ports))
break;
}

On Thu, Feb 13, 2020 at 2:49 PM Micha? Purzy?ski <michalpurzynski1 at gmail.com>
wrote:

> A couple of things.
>
> First, you should use Justin's simple-scan. As others have pointed out,
> the stock scanning detection script can behave poorly and it's hardly
> extensible.
>
> https://github.com/ncsa/bro-simple-scan
>
> (it's also packaged)
>
> Second - you can either ignore connections so the detection algorithm
> won't count them (with the hook from the simple-scan code), or you can
> write a notice policy and ignore some notices. Up to you - we just ignore
> some connections.
>
> Inside Justin's package, you will find a hook - this is what we use to
> ignore a set of destination and source IP addresses and some destination
> ports
>
> https://github.com/ncsa/bro-simple-scan/blob/master/scripts/scan.bro#L87
>
> Here's how we use that hook
>
> https://gist.github.com/mpurzynski/96a26c42874898447554531b6df9a4bb
>
> The input framework is what we use to update the list runtime. Nowadays
> you could use the configuration framework instead.
>
>
> https://corelight.blog/2018/02/13/runtime-options-the-bro-configuration-framework/
>
>
> Either way, you do not have to modify any upstream package.
>
>
> On Thu, Feb 13, 2020 at 11:23 AM Gordon Wallum <glwallum at gmail.com> wrote:
>
>> Hello!
>>
>> I am new with Zeek and looking to learn more. I am currently using the
>> scan.zeek script (
>> https://github.com/zeek/zeek/blob/master/scripts/policy/misc/scan.zeek)
>> for port scanning detection.
>>
>> I want to exclude certain source IP addresses from this script but I am
>> not sure the best way to do so. It seems like a comparison with the
>> key$host variable, but not sure where or how to do this logic in Zeek.
>>
>> Any advice would be appreciated
>>
>> Thank you
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200214/5751b5bc/attachment.html 

From glwallum at gmail.com  Fri Feb 14 08:26:41 2020
From: glwallum at gmail.com (Gordon Wallum)
Date: Fri, 14 Feb 2020 09:26:41 -0700
Subject: [Zeek] scan.zeek question - exclude IP addresses
In-Reply-To: <CAHs9D0tcJFHgAMwFig54Aouj3J1+2kHUpuMkk-Ke3nfVx_8jVA@mail.gmail.com>
References: <CAHs9D0sdM0yk4pYWp_E=K-kp1w5ptcVtUJpWmJoRH1RaPQBxSA@mail.gmail.com>
	<CAJ6bFK3p1HWyJePwpMhZJm4-fFVWeqr4GnCYffGJBEo6cjn6HA@mail.gmail.com>
	<CAHs9D0tcJFHgAMwFig54Aouj3J1+2kHUpuMkk-Ke3nfVx_8jVA@mail.gmail.com>
Message-ID: <CAHs9D0sA2O=n+6t9BBhAmZGP1QOimqu+DP3Vnn43=q7SSZrjvw@mail.gmail.com>

Also the @load packages/bro-is-darknet is erroring since it is not
installed on my Zeek environment, do I need to use the zeek package manager
to install it?

On Fri, Feb 14, 2020 at 9:20 AM Gordon Wallum <glwallum at gmail.com> wrote:

> Thank you, this makes sense logically but I can't figure out how to use
> the hook to exclude. The code below throws an error
>
>  ## Override this hook to ignore particular scan connections
>     global Scan::scan_policy: hook(scanner: addr, victim: addr,
> scanned_port: port)
> {
>             if (( victim in exvictim_ips) || ( scanner in exscanner_ips )
> || ( scanned_port in exscanned_ports))
> break;
> }
>
> On Thu, Feb 13, 2020 at 2:49 PM Micha? Purzy?ski <
> michalpurzynski1 at gmail.com> wrote:
>
>> A couple of things.
>>
>> First, you should use Justin's simple-scan. As others have pointed out,
>> the stock scanning detection script can behave poorly and it's hardly
>> extensible.
>>
>> https://github.com/ncsa/bro-simple-scan
>>
>> (it's also packaged)
>>
>> Second - you can either ignore connections so the detection algorithm
>> won't count them (with the hook from the simple-scan code), or you can
>> write a notice policy and ignore some notices. Up to you - we just ignore
>> some connections.
>>
>> Inside Justin's package, you will find a hook - this is what we use to
>> ignore a set of destination and source IP addresses and some destination
>> ports
>>
>> https://github.com/ncsa/bro-simple-scan/blob/master/scripts/scan.bro#L87
>>
>> Here's how we use that hook
>>
>> https://gist.github.com/mpurzynski/96a26c42874898447554531b6df9a4bb
>>
>> The input framework is what we use to update the list runtime. Nowadays
>> you could use the configuration framework instead.
>>
>>
>> https://corelight.blog/2018/02/13/runtime-options-the-bro-configuration-framework/
>>
>>
>> Either way, you do not have to modify any upstream package.
>>
>>
>> On Thu, Feb 13, 2020 at 11:23 AM Gordon Wallum <glwallum at gmail.com>
>> wrote:
>>
>>> Hello!
>>>
>>> I am new with Zeek and looking to learn more. I am currently using the
>>> scan.zeek script (
>>> https://github.com/zeek/zeek/blob/master/scripts/policy/misc/scan.zeek)
>>> for port scanning detection.
>>>
>>> I want to exclude certain source IP addresses from this script but I am
>>> not sure the best way to do so. It seems like a comparison with the
>>> key$host variable, but not sure where or how to do this logic in Zeek.
>>>
>>> Any advice would be appreciated
>>>
>>> Thank you
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200214/36c5106a/attachment-0001.html 

From patrick.kelley at criticalpathsecurity.com  Fri Feb 14 11:09:43 2020
From: patrick.kelley at criticalpathsecurity.com (Patrick Kelley)
Date: Fri, 14 Feb 2020 14:09:43 -0500
Subject: [Zeek] Zeek 3 / Vmware ESXi
In-Reply-To: <YT1PR01MB3227F8235871985C478833C4C0150@YT1PR01MB3227.CANPRD01.PROD.OUTLOOK.COM>
References: <YT1PR01MB3227F8235871985C478833C4C0150@YT1PR01MB3227.CANPRD01.PROD.OUTLOOK.COM>
Message-ID: <CA+WAcdQYmfY2RnAO+gHZR_pyK8DzSJfNq-oa6UaPbTjyME6EUA@mail.gmail.com>

St?phane,

I've spent considerable time performing strength tests with Zeek, Suricata,
and Snort with BreakingPoint chassis and similar testing platforms.

The biggest issue you will face is unpredictable resources. Where possible,
isolate the resources for the Zeek instance from other guests on the
fabric.  Zeek isn't really going to know the difference between being
virtual or physical.  As other guests pull from the same source, you will
see an impact. PF_Ring and AF_Packet can be used.  In the past, I've stuck
with PF_Ring as it was a bit more predictable.  This isn't a requirement,
just a preference I have based on previous experience.

For comparison, when I run a 10 Gbps test against physical instances of
Zeek, I will see an average of 7.6 Gbps of actual throughput with a +/- 5%
variation.  With virtual instances on VMware with nothing else running, I
will see closer to 6.4 Gbps with the same traffic replay, showing a +/- 15%
variation. It's just a bit harder to nail down your performance baseline.

I would recommend that you enable jumbo packet support as it will help with
latency and keep an eye on things. Mileage may vary.

Obviously, some traffic is more costly than others. FTP, DNS. etc... is
going to be less of an impact as SMB.  Some detections are more costly than
others.

Hope this helps and as you say, "Mon milieu de vie!".

-PK

On Fri, Feb 14, 2020 at 8:54 AM Lantin, St?phane <slantin at cegepbc.ca> wrote:

> Hello,
>
>
>
> I have installed Zeek on ESXi and the hardware is dedicated for Zeek.
>
> Everything works fine, Zeek logs the network.
>
>
>
> My question is, what difference is for Zeek to be run on virtual vs
> physical ?
>
> Our highest throughput is 125mbps for our ISP and about 1Gpbs internal.
>
>
>
> Some suggests it might impact the performance, but where it could struggle
> on VM?
>
>
>
> Thank you for your time,
>
>
>
> _____________________________
>
> *St?phane Lantin*
>
> *Technicien en informatique - C.P*
>
> *Service des technologies de l?information*
>
> *C?gep de Baie-Comeau*
>
> *(418) 589-5707 poste 178*
>
> *slantin at cegepbc.ca <slantin at cegepbc.ca>*
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 

*Patrick Kelley, CISSP, C|EH, ITIL*
*CTO*
patrick.kelley at criticalpathsecurity.com
(o) 770-224-6482
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200214/acd5bd95/attachment.html 

From glwallum at gmail.com  Fri Feb 14 11:54:37 2020
From: glwallum at gmail.com (Gordon Wallum)
Date: Fri, 14 Feb 2020 12:54:37 -0700
Subject: [Zeek] bro-simple-scan exclude IP addresses
Message-ID: <CAHs9D0v5FvSNk46ZrMQhCoqCmbdgjy4PzrW=VcZvn=D0iw_7Aw@mail.gmail.com>

I'm new to Zeek and looking for help with bro-simple-scan to exclude Ip
addresses. I am trying to use runtime options and if breaks to the script
to accomplish this.

After running i get an error for my options variables. I don't know if I
should move my excludes to a different part of the script or if my options
are just not working right.

Any help would be greatly appreciated

*Error*
error in /opt/bro/share/zeek/policy/custom-scripts/./bro-simple-scan2, line
276: unknown identifier exvictim_ips, at or near "exvictim_ips"


*My config steps: *

Edit local.bro to include the config file:
redef Config::config_files += { "/path/to/config.dat" };

*Create config file with variables: *
PortScanning::exvictim_ips
PortScanning::exscanner_ips xxx.xxx.xxx.xxx,yyy.yyy.yyy.yyy
PortScanning::exscanned_ports


Edit the bro-simple-scan script:

*Added module and export variable options (after @loads)*
module PortScanning;
export {
    option exvictim_ips: set[addr] = {};
    option exscanner_ips: set[addr] = {};
    option exscanned_ports: set[port] = {};
}

*Added if break (in the cluster hook Scan::scan_policy)*

if ( hook Scan::scan_policy(scanner, victim, scanned_port) )
        {
if (( victim in exvictim_ips) || ( scanner in exscanner_ips ) || (
scanned_port in exscanned_ports))
break;

*Add an if break (in the standalone hook Scan::scan_policy)*

if ( hook Scan::scan_policy(scanner, victim, scanned_port) )
        {
if (( victim in exvictim_ips) || ( scanner in exscanner_ips ) || (
scanned_port in exscanned_ports))
break;
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200214/eeb4c5b6/attachment.html 

From smitra at ucn.ca  Fri Feb 14 13:13:20 2020
From: smitra at ucn.ca (Mitra, Shaibal)
Date: Fri, 14 Feb 2020 21:13:20 +0000
Subject: [Zeek] Zeek ELK integration
Message-ID: <d3ea333250d94aaa92f93e7d5c4aac65@ucn.ca>

I am running the latest zeek version.I have set up ELK stack on a different Ubuntu server and installed filebeat(all version 7) and enabled zeek plugin.I am porting zeek JSON logs to Ubuntu server and pointing filebeat to the log location.I am not getting any display of top dns top url etc.There is only a number of sessions overtime display which never changes.Is the filebeat zeek plugin only for zeek 2.6 version?

Thanks

[cid:image001.png at 01D5E349.4B35EA40]

[signature]
IT Network Systems Administrator
The Pas Campus
Ph:204-627-8593(Office)
Ph:204-620-1221(Cell)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200214/44d64afe/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 58042 bytes
Desc: image001.png
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200214/44d64afe/attachment-0002.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 12170 bytes
Desc: image002.png
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200214/44d64afe/attachment-0003.bin 

From clopmz at outlook.com  Sat Feb 15 08:43:28 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Sat, 15 Feb 2020 16:43:28 +0000
Subject: [Zeek] zeekctl netstats returns time out
Message-ID: <028E6FD5-11D2-46FA-A5C0-5A32BC4A839F@outlook.com>

Hi all,

Every time I run ?zeekctl netstats? returns time out under FreeBSD 12.1 hosts using netmap:

root at fbsdzeek01:/nsm/zeek/logs/current # zeekctl netstats

Warning: ZeekControl plugin uses legacy BroControl API. Use
'import ZeekControl.plugin' instead of 'import BroControl.plugin'

       zeek: <error: time-out>

This behavior occurs in both standalone and cluster configurations. Any idea? Maybe is it a bug?

--
Regards,
C. L. Martinez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200215/46378f5a/attachment.html 

From jsiwek at corelight.com  Sat Feb 15 09:57:27 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Sat, 15 Feb 2020 09:57:27 -0800
Subject: [Zeek] zeekctl netstats returns time out
In-Reply-To: <028E6FD5-11D2-46FA-A5C0-5A32BC4A839F@outlook.com>
References: <028E6FD5-11D2-46FA-A5C0-5A32BC4A839F@outlook.com>
Message-ID: <CAMzgZ0+1sO4UK3+SHBGZeipszjgeA3ZZokTmTuHHy3GMGSDrrw@mail.gmail.com>

Zeek 3.0.1's `zeekctl netstats` is working for me in FreeBSD 12.1.
TCP connectivity is required for that command to work and you can read
more about the ports involved for further troubleshooting here:

    https://github.com/zeek/zeekctl#zeek-communication

If the Zeek processes are particularly busy, that could also be a
reason for timing out.  The `CommTimeout` (default 10 seconds) can be
increased in `zeekctl.cfg` in that case.

- Jon

On Sat, Feb 15, 2020 at 8:46 AM Carlos Lopez <clopmz at outlook.com> wrote:
>
> Hi all,
>
>
>
> Every time I run ?zeekctl netstats? returns time out under FreeBSD 12.1 hosts using netmap:
>
>
>
> root at fbsdzeek01:/nsm/zeek/logs/current # zeekctl netstats
>
>
>
> Warning: ZeekControl plugin uses legacy BroControl API. Use
>
> 'import ZeekControl.plugin' instead of 'import BroControl.plugin'
>
>
>
>        zeek: <error: time-out>
>
>
>
> This behavior occurs in both standalone and cluster configurations. Any idea? Maybe is it a bug?
>
>
>
> --
>
> Regards,
>
> C. L. Martinez
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From clopmz at outlook.com  Sat Feb 15 14:21:20 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Sat, 15 Feb 2020 22:21:20 +0000
Subject: [Zeek] zeekctl netstats returns time out
In-Reply-To: <CAMzgZ0+1sO4UK3+SHBGZeipszjgeA3ZZokTmTuHHy3GMGSDrrw@mail.gmail.com>
References: <028E6FD5-11D2-46FA-A5C0-5A32BC4A839F@outlook.com>
	<CAMzgZ0+1sO4UK3+SHBGZeipszjgeA3ZZokTmTuHHy3GMGSDrrw@mail.gmail.com>
Message-ID: <1A1F98C3-8152-4568-AB43-FECB4B7C1455@outlook.com>

Many thanks Jon. Regarding TCP connectivity, I have neither ipfw nor pf enabled between manager and workers. And respecting to "busy" system, shouldn't be the problem either. For example, my top output in standalone config:

last pid:  6492;  load averages:  0.16,  0.22,  0.22                                                                                                                                up 0+06:21:48  22:20:43
44 threads:    1 running, 43 sleeping
CPU:  0.0% user,  0.0% nice,  1.9% system,  0.0% interrupt, 98.1% idle
Mem: 51M Active, 58M Inact, 679M Wired, 271M Buf, 5137M Free
Swap: 4096M Total, 4096M Free

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
45091 root         22    0   460M   111M select   1  18:29   4.71% zeek{zeek}
 6492 root         20    0  1044M  4144K CPU0     0   0:00   0.05% top
45091 root         20    0   460M   111M uwait    0   0:22   0.02% zeek{caf.clock}
39952 _ntp         20  -20  1038M  4000K select   1   0:03   0.01% ntpd
45407 root         20    0  1044M  9912K select   1   0:00   0.01% sshd
45091 root         20    0   460M   111M uwait    1   0:09   0.01% zeek{caf.multiplexer}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.ntp/Log::WRITER_}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.files/Log::WRITE}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.capture_loss/Log}
45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.dns/Log::WRITER_}
45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.ssl/Log::WRITER_}
45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.http/Log::WRITER}
45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.loaded_scripts/L}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.packet_filter/Lo}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.stats/Log::WRITE}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.conn/Log::WRITER}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.software/Log::WR}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.known_services/L}
45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.x509/Log::WRITER}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.notice/Log::WRIT}
45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.ssh/Log::WRITER_}
45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.kerberos/Log::WR}
45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.broker/Log::WRIT}
45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.weird/Log::WRITE}
45091 root         20    0   460M   111M uwait    0   0:00   0.00% zeek{zk.dhcp/Log::WRITER}
45091 root         20    0   460M   111M uwait    1   0:00   0.00% zeek{zk.known_certs/Log:}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.known_hosts/Log:}
96485 root         20    0    17M  6920K select   0   0:00   0.00% sendmail
45091 root         20    0   460M   111M select   0   0:00   0.00% zeek{caf.multiplexer}


-- 
Regards,
C. L. Martinez

?On 15/02/2020, 18:57, "Jon Siwek" <jsiwek at corelight.com> wrote:

    Zeek 3.0.1's `zeekctl netstats` is working for me in FreeBSD 12.1.
    TCP connectivity is required for that command to work and you can read
    more about the ports involved for further troubleshooting here:
    
        https://github.com/zeek/zeekctl#zeek-communication
    
    If the Zeek processes are particularly busy, that could also be a
    reason for timing out.  The `CommTimeout` (default 10 seconds) can be
    increased in `zeekctl.cfg` in that case.
    
    - Jon
    
    On Sat, Feb 15, 2020 at 8:46 AM Carlos Lopez <clopmz at outlook.com> wrote:
    >
    > Hi all,
    >
    >
    >
    > Every time I run ?zeekctl netstats? returns time out under FreeBSD 12.1 hosts using netmap:
    >
    >
    >
    > root at fbsdzeek01:/nsm/zeek/logs/current # zeekctl netstats
    >
    >
    >
    > Warning: ZeekControl plugin uses legacy BroControl API. Use
    >
    > 'import ZeekControl.plugin' instead of 'import BroControl.plugin'
    >
    >
    >
    >        zeek: <error: time-out>
    >
    >
    >
    > This behavior occurs in both standalone and cluster configurations. Any idea? Maybe is it a bug?
    >
    >
    >
    > --
    >
    > Regards,
    >
    > C. L. Martinez
    >
    > _______________________________________________
    > Zeek mailing list
    > zeek at zeek.org
    > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
    



From Joseph.Fischetti at marist.edu  Mon Feb 17 06:41:16 2020
From: Joseph.Fischetti at marist.edu (Joseph Fischetti)
Date: Mon, 17 Feb 2020 14:41:16 +0000
Subject: [Zeek] Dropping packets
Message-ID: <BN8PR20MB2532BBAADDFF169CAD68A740EE160@BN8PR20MB2532.namprd20.prod.outlook.com>

Good morning,

I'm new to the list, and have been working on inheriting an existing zeek
deployment that we have here.  I'm trying to track down some (to me)
excessive packet dropping.

 

We had an older version of zeek (bro) installed and mostly functional,
though as I recall they were having issues with workers occasionally
crashing.

 

Before I started looking into things, a new version of zeek was deployed
(from an binary) and is mostly vanilla.  We've included the bhr and myricom
plugin, but that's about it.

 

Zeek master and workers run bare metal on 3 pretty big Intel hosts (192GB
memory, 2x Xeon E5-2690 with 14cores/socket, Debian 9).  The workers have
myricom interfaces.  There's span ports at the edge that feed into Arista
switches that feed the Myricom interfaces in the workers.

 

We have a few issues:

1)      If I try to start up the workers with any more than ~8 threads,
packet drop and memory usage goes through the roof in pretty short order.
If I try to pin them, the first "worker" cpu's get pegged pretty high and
the others stay more or less idle (though that could be due to the amount of
traffic the second worker interface is receiving).

2)      If I try to start up "1" worker (per worker node), using the
"myricom::*" interface, the worker node goes unresponsive and needs to be
hardware bounced.  (Driver issue?)

3)      I can start workers nodes with multiple workers and ~5 threads each
(currently "unpinned"), but after a few days, Packet drop is still
excessive.

 

My current node.cfg is below [1].  Output from 'zeekctl netstats' is also
below [2].  It's been up since Friday ~2:00pm Eastern.  Load average is
higher than I would think it should be (given how much cpu these workers
actually have, and how idle most of the cpu's actually are).  Htop output
included [3].

 

I understand we should probably be pinning the worker threads, but the
output of 'lstopo-no-graphics  --of txt' is terrible to try and trace with
56 threads available.  Also, do I want to use the "P" or the "L" listings?
I can include that as a follow up if necessary.

 

Please help!

 

[1]

================== 

[manager]

type=manager

host=THE MASTER

 

[logger]

type=logger

host= THE MASTER

 

[proxy-1]

type=proxy

host= THE MASTER

 

[worker-1]

type=worker

host=WORKER 1

lb_method=custom

lb_procs=5

interface=myricom::eth4

 

[worker-2]

type=worker

host=WORKER 2

lb_method=custom

lb_procs=5

interface=myricom::eth4

 

[worker-3]

type=worker

host=WORKER 1

lb_method=custom

lb_procs=5

interface=myricom::eth5

 

[worker-4]

type=worker

host=WORKER 2

lb_method=custom

lb_procs=5

interface=myricom::eth5

 

=================================================

 

[2]

================

bro at bro-master-1:~$ zeekctl netstats

 

Warning: ZeekControl plugin uses legacy BroControl API. Use

'import ZeekControl.plugin' instead of 'import BroControl.plugin'

 

worker-1-1: 1581949346.194441 recvd=2178149468 dropped=2260820124
link=15063051356

worker-1-2: 1581949346.194473 recvd=274557259 dropped=2260820124
link=13159459147

worker-1-3: 1581949346.168558 recvd=1888926901 dropped=2260820124
link=14773828789

worker-1-4: 1581949346.081130 recvd=2110377092 dropped=2260820124
link=14995278980

worker-1-5: 1581949346.234478 recvd=1032618510 dropped=2260820124
link=13917520398

worker-2-1: 1581949346.269794 recvd=1551167612 dropped=640636540
link=14436069500

worker-2-2: 1581949346.271224 recvd=2811566586 dropped=640636540
link=15696468474

worker-2-3: 1581949346.292474 recvd=3295536154 dropped=640636540
link=16180438042

worker-2-4: 1581949346.314556 recvd=2505663441 dropped=640636540
link=15390565329

worker-2-5: 1581949343.011855 recvd=3459004896 dropped=640636540
link=20638874080

worker-3-1: 1581949346.239424 recvd=938819819 dropped=0 link=938819819

worker-3-2: 1581949346.249540 recvd=890104345 dropped=0 link=890104345

worker-3-3: 1581949346.259501 recvd=894787204 dropped=0 link=894787204

worker-3-4: 1581949346.269501 recvd=895479546 dropped=0 link=895479546

worker-3-5: 1581949346.274490 recvd=878546610 dropped=0 link=878546610

worker-4-1: 1581949346.329587 recvd=892356780 dropped=0 link=892356780

worker-4-2: 1581949346.344510 recvd=922981664 dropped=0 link=922981664

worker-4-3: 1581949346.349568 recvd=855515132 dropped=0 link=855515132

worker-4-4: 1581949346.359652 recvd=931447757 dropped=0 link=931447757

worker-4-5: 1581949346.368349 recvd=876976485 dropped=0 link=876976485

 

===========================================================

 

[3]

===================

  1  [||     3.3%]    15 [       0.0%]   29 [||     6.1%]    43
[||||||91.6%]

  2  [||     7.9%]    16 [       0.0%]   30 [|||   14.2%]    44 [
0.0%]

  3  [|      3.3%]    17 [|      1.4%]   31 [||||  20.2%]    45 [||
1.9%]

  4  [||     3.3%]    18 [       0.0%]   32 [||     4.7%]    46 [|
0.5%]

  5  [||     3.7%]    19 [||||||76.3%]   33 [||     4.2%]    47 [||
9.1%]

  6  [||     5.2%]    20 [       0.0%]   34 [||||||39.5%]    48 [||
3.3%]

  7  [||     2.8%]    21 [       0.0%]   35 [||     5.2%]    49 [
0.0%]

  8  [||     5.6%]    22 [||     1.4%]   36 [||     3.7%]    50 [||
3.3%]

  9  [||     6.0%]    23 [|      0.5%]   37 [|||   16.7%]    51 [
0.0%]

  10 [||     1.9%]    24 [|      0.5%]   38 [||||||56.5%]    52 [||
7.1%]

  11 [||     2.8%]    25 [||||||88.4%]   39 [||     6.6%]    53 [
0.0%]

  12 [|||   13.6%]    26 [||     1.4%]   40 [||||| 30.7%]    54 [||
0.9%]

  13 [|||   15.2%]    27 [       0.0%]   41 [||     3.3%]    55 [||
0.9%]

  14 [||     4.8%]    28 [       0.0%]   42 [||     8.1%]    56 [||
2.3%]

 

  Mem[||||||||||||||||||||||119G/188G]   

  Swp[                        0K/191G]   

 

Tasks: 58, 107 thr; 3 running

Load average: 6.79 6.10 5.83

Uptime: 2 days, 18:59:33

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200217/a0b3166f/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5561 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200217/a0b3166f/attachment-0001.bin 

From justin at corelight.com  Mon Feb 17 08:06:07 2020
From: justin at corelight.com (Justin Azoff)
Date: Mon, 17 Feb 2020 11:06:07 -0500
Subject: [Zeek] Dropping packets
In-Reply-To: <BN8PR20MB2532BBAADDFF169CAD68A740EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
References: <BN8PR20MB2532BBAADDFF169CAD68A740EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
Message-ID: <CAPfnCuiNCi+PpSGgPynbOcSYYFq0N1RNw-mZS1CYP-YQpwwBOw@mail.gmail.com>

On Mon, Feb 17, 2020 at 9:44 AM Joseph Fischetti <
Joseph.Fischetti at marist.edu> wrote:

> 1)      If I try to start up the workers with any more than ~8 threads,
> packet drop and memory usage goes through the roof in pretty short order.
> If I try to pin them, the first ?worker? cpu?s get pegged pretty high and
> the others stay more or less idle (though that could be due to the amount
> of traffic the second worker interface is receiving).
>

8 workers on each card should work fine.  Based on your netstats output
your load balancing might not be working that well.  The total received by
interface is:

worker-1 7484629230
worker-2 13622938689
worker-3 4497737524
worker-4 4479277818

which is a bit skewed.

> 2)      If I try to start up ?1? worker (per worker node), using the
> ?myricom::*? interface, the worker node goes unresponsive and needs to be
> hardware bounced.  (Driver issue?)
>
Could be.. I never used that feature.

> 3)      I can start workers nodes with multiple workers and ~5 threads
> each (currently ?unpinned?), but after a few days, Packet drop is still
> excessive.
>
Not super surprising with only 5 workers per interface.. with those cpus
I'd run 10-12


> My current node.cfg is below [1].  Output from ?zeekctl netstats? is also
> below [2].  It?s been up since Friday ~2:00pm Eastern.  Load average is
> higher than I would think it should be (given how much cpu these workers
> actually have, and how idle most of the cpu?s actually are).  Htop output
> included [3].
>
>
>
> I understand we should probably be pinning the worker threads, but the
> output of ?lstopo-no-graphics  --of txt? is terrible to try and trace with
> 56 threads available.  Also, do I want to use the ?P? or the ?L? listings?
> I can include that as a follow up if necessary.
>

Turning off HT will simplify that quite a bit.  You want the P cores I
believe.  In htop you'll end up with cores 1-28  busy and 29-56 idle since
those are the fake ones.  Using lstopo-no-graphics --of ascii (make font
really small) may make things easier to understand.

I'd change your node.cfg to be more like this, the output will make more
sense:
(remember to stop the cluster before changing worker names)

[worker-1-a]
>
> type=worker
>
> host=WORKER 1
>
> lb_method=custom
>
> lb_procs=5
>
> interface=myricom::eth4
>
>
>
> [worker-1-b]
>
> type=worker
>
> host=WORKER 1
>
> lb_method=custom
>
> lb_procs=5
>
> interface=myricom::eth5
>

>
> [worker-2-a]
>
> type=worker
>
> host=WORKER 2
>
> lb_method=custom
>
> lb_procs=5
>
> interface=myricom::eth4
>

>
> [worker-2-b]
>
> type=worker
>
> host=WORKER 2
>
> lb_method=custom
>
> lb_procs=5
>
> interface=myricom::eth5
>

The last time I worked out a node.cfg for a myricom based cluster this is
what I ended up with:

[foo-a]
type=worker
host=foo
interface=myricom::p1p1
lb_method=custom
lb_procs=8
pin_cpus=3,5,7,9,11,13,15,17
env_vars=SNF_APP_ID=1,SNF_DATARING_SIZE=16384MB,SNF_DESCRING_SIZE=4096MB

[foo-b]
type=worker
host=fooo
interface=myricom::p2p1
lb_method=custom
lb_procs=8
pin_cpus=2,4,6,8,10,12,14,16
env_vars=SNF_APP_ID=2,SNF_DATARING_SIZE=16384MB,SNF_DESCRING_SIZE=4096MB

I'm about 90% sure I did the pinning right :-)


> [2]
>
> ================
>
> bro at bro-master-1:~$ zeekctl netstats
>
>
>
> worker-1-1: 1581949346.194441 recvd=2178149468 dropped=2260820124
> link=15063051356
>
> worker-1-2: 1581949346.194473 recvd=274557259 dropped=2260820124
> link=13159459147
>
> worker-1-3: 1581949346.168558 recvd=1888926901 dropped=2260820124
> link=14773828789
>
> worker-1-4: 1581949346.081130 recvd=2110377092 dropped=2260820124
> link=14995278980
>
> worker-1-5: 1581949346.234478 recvd=1032618510 dropped=2260820124
> link=13917520398
>
> worker-2-1: 1581949346.269794 recvd=1551167612 dropped=640636540
> link=14436069500
>
> worker-2-2: 1581949346.271224 recvd=2811566586 dropped=640636540
> link=15696468474
>
> worker-2-3: 1581949346.292474 recvd=3295536154 dropped=640636540
> link=16180438042
>
> worker-2-4: 1581949346.314556 recvd=2505663441 dropped=640636540
> link=15390565329
>
> worker-2-5: 1581949343.011855 recvd=3459004896 dropped=640636540
> link=20638874080
>
>
>
One thing to keep in mind with the myrcom driver is the drops are shared..
so you aren't dropping 2260820124 per worker, you dropped 2260820124 total.
The number is right, but to work out the total drop % you need to sum recvd
but not the dropped number.

Here's a script i wrote a while ago that does that.. can pipe netstats
output to this:

#!/usr/bin/env python
from __future__ import  print_function
import sys
import re
from collections import defaultdict

totals = defaultdict(int)
host_dropped = {}

total_rx = total_drop = 0

for line in sys.stdin:
    parts = re.split('[ =:]+', line.strip())
    node, time, _, recvd, _, dropped, _, link = parts
    host = node[:-2]
    totals[host] += int(link)

    total_rx += int(link)
    if host not in host_dropped:
        total_drop += int(dropped)
        host_dropped[host] = int(dropped)

for host, total in sorted(totals.items()):
    if not total: continue
    d = host_dropped[host]
    print("%s dropped=%d rx=%d %0.2f%%" % (host, d, total, 100.0*d/total))

print()
print("Totals dropped=%d rx=%d %0.2f%%" % (total_drop, total_rx,
100.0*total_drop/total_rx))

may need to tweak the host = node[:-2] line if you run more than 9 workers

running that on your output I get

worker-1 dropped=2260820124 rx=71909138670 3.14%
worker-2 dropped=640636540 rx=82342415425 0.78%

which isn't so bad.. without accounting for the shared drops it looks more
like 15% and 4%.  Double the number of workers and fix the load balancing
and you should be able to get that to zero.

-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200217/da92cf32/attachment.html 

From mfernandez at mitre.org  Mon Feb 17 09:45:50 2020
From: mfernandez at mitre.org (Fernandez, Mark I)
Date: Mon, 17 Feb 2020 17:45:50 +0000
Subject: [Zeek] BZAR Update - Config Options for Detection, Reporting,
 and Whitelisting
Message-ID: <DM6PR09MB3129D16A7271A1970152E095CF160@DM6PR09MB3129.namprd09.prod.outlook.com>

All,



New update to BZAR is available.  As presented at ZeekWeek 2019, we improved the whitelisting capability to ignore activity based on IP address, IP subnet, or hostname., and we added configuration options to toggle on/off detection and reporting of each ATT&CK indicator.  These new features allow for very granular control of the whitelists and toggle switches.  As a result, we split some of the script files to make the code more manageable.  See the CHANGES file for more information.



For the new version, use the Zeek package manager or download from the following URL:

*       https://github.com/mitre-attack/bzar



Please let me know if you encounter any errors. BZAR still uses the .bro file extension for the scripts, so you may see some deprecation warnings, but it should run as expected.  We'll make BZAR fully compliant with Zeek 3.0 soon.



Mark I. Fernandez

The MITRE Corporation

mfernandez at mitre.org<mailto:mfernandez at mitre.org>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200217/21b93a4e/attachment-0001.html 

From Joseph.Fischetti at marist.edu  Mon Feb 17 11:31:08 2020
From: Joseph.Fischetti at marist.edu (Joseph Fischetti)
Date: Mon, 17 Feb 2020 19:31:08 +0000
Subject: [Zeek] Dropping packets
In-Reply-To: <CAPfnCuiNCi+PpSGgPynbOcSYYFq0N1RNw-mZS1CYP-YQpwwBOw@mail.gmail.com>
References: <BN8PR20MB2532BBAADDFF169CAD68A740EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuiNCi+PpSGgPynbOcSYYFq0N1RNw-mZS1CYP-YQpwwBOw@mail.gmail.com>
Message-ID: <BN8PR20MB2532DC39DCD467639251A761EE160@BN8PR20MB2532.namprd20.prod.outlook.com>

Thanks for the response Justin.

 

I?ll start with disabling HT on the workers, and will reconfigure the workers with 8 threads each.  I?ll also pin the processes.

The myricom configuration you quoted included environment variables:

env_vars=SNF_APP_ID=1,SNF_DATARING_SIZE=16384MB,SNF_DESCRING_SIZE=4096MB

What are those for/where can I find documentation/Should I include any of those?

 

You also said the packets dropped are inclusive for the entire worker, but that the packets recvd are per thread:

worker-1-1: 1581949346.194441 recvd=2178149468 dropped=2260820124 link=15063051356

worker-1-2: 1581949346.194473 recvd=274557259 dropped=2260820124 link=13159459147

worker-1-3: 1581949346.168558 recvd=1888926901 dropped=2260820124 link=14773828789

worker-1-4: 1581949346.081130 recvd=2110377092 dropped=2260820124 link=14995278980

worker-1-5: 1581949346.234478 recvd=1032618510 dropped=2260820124 link=13917520398

 

Those add up to ~7.5B recvd with 2.2B dropped.  That?s 30%, but your script said 3.  Am I looking at some numbers wrong?

 

Joe

 

From: Justin Azoff <justin at corelight.com> 
Sent: Monday, February 17, 2020 11:06 AM
To: Joseph Fischetti <Joseph.Fischetti at marist.edu>
Cc: zeek at zeek.org
Subject: Re: [Zeek] Dropping packets

 

[EXTERNAL EMAIL]

On Mon, Feb 17, 2020 at 9:44 AM Joseph Fischetti <Joseph.Fischetti at marist.edu <mailto:Joseph.Fischetti at marist.edu> > wrote:

1)      If I try to start up the workers with any more than ~8 threads, packet drop and memory usage goes through the roof in pretty short order.  If I try to pin them, the first ?worker? cpu?s get pegged pretty high and the others stay more or less idle (though that could be due to the amount of traffic the second worker interface is receiving).

 

8 workers on each card should work fine.  Based on your netstats output your load balancing might not be working that well.  The total received by interface is:

 

worker-1 7484629230
worker-2 13622938689
worker-3 4497737524

worker-4 4479277818

 

which is a bit skewed.     

2)      If I try to start up ?1? worker (per worker node), using the ?myricom::*? interface, the worker node goes unresponsive and needs to be hardware bounced.  (Driver issue?)

Could be.. I never used that feature. 

3)      I can start workers nodes with multiple workers and ~5 threads each (currently ?unpinned?), but after a few days, Packet drop is still excessive.

Not super surprising with only 5 workers per interface.. with those cpus I'd run 10-12 

 

My current node.cfg is below [1].  Output from ?zeekctl netstats? is also below [2].  It?s been up since Friday ~2:00pm Eastern.  Load average is higher than I would think it should be (given how much cpu these workers actually have, and how idle most of the cpu?s actually are).  Htop output included [3].

 

I understand we should probably be pinning the worker threads, but the output of ?lstopo-no-graphics  --of txt? is terrible to try and trace with 56 threads available.  Also, do I want to use the ?P? or the ?L? listings?  I can include that as a follow up if necessary.

 

Turning off HT will simplify that quite a bit.  You want the P cores I believe.  In htop you'll end up with cores 1-28  busy and 29-56 idle since those are the fake ones.  Using lstopo-no-graphics --of ascii (make font really small) may make things easier to understand.

 

I'd change your node.cfg to be more like this, the output will make more sense: 

(remember to stop the cluster before changing worker names)

 

[worker-1-a]

type=worker

host=WORKER 1

lb_method=custom

lb_procs=5

interface=myricom::eth4

 

[worker-1-b]

type=worker

host=WORKER 1

lb_method=custom

lb_procs=5

interface=myricom::eth5

 

[worker-2-a]

type=worker

host=WORKER 2

lb_method=custom

lb_procs=5

interface=myricom::eth4 

 

[worker-2-b]

type=worker

host=WORKER 2

lb_method=custom

lb_procs=5

interface=myricom::eth5

 

The last time I worked out a node.cfg for a myricom based cluster this is what I ended up with:

 

[foo-a]
type=worker
host=foo
interface=myricom::p1p1
lb_method=custom
lb_procs=8
pin_cpus=3,5,7,9,11,13,15,17
env_vars=SNF_APP_ID=1,SNF_DATARING_SIZE=16384MB,SNF_DESCRING_SIZE=4096MB

[foo-b]
type=worker
host=fooo
interface=myricom::p2p1
lb_method=custom
lb_procs=8
pin_cpus=2,4,6,8,10,12,14,16
env_vars=SNF_APP_ID=2,SNF_DATARING_SIZE=16384MB,SNF_DESCRING_SIZE=4096MB

 

I'm about 90% sure I did the pinning right :-)

 

[2]

================

bro at bro-master-1:~$ zeekctl netstats

 

worker-1-1: 1581949346.194441 recvd=2178149468 dropped=2260820124 link=15063051356

worker-1-2: 1581949346.194473 recvd=274557259 dropped=2260820124 link=13159459147

worker-1-3: 1581949346.168558 recvd=1888926901 dropped=2260820124 link=14773828789

worker-1-4: 1581949346.081130 recvd=2110377092 dropped=2260820124 link=14995278980

worker-1-5: 1581949346.234478 recvd=1032618510 dropped=2260820124 link=13917520398

worker-2-1: 1581949346.269794 recvd=1551167612 dropped=640636540 link=14436069500

worker-2-2: 1581949346.271224 recvd=2811566586 dropped=640636540 link=15696468474

worker-2-3: 1581949346.292474 recvd=3295536154 dropped=640636540 link=16180438042

worker-2-4: 1581949346.314556 recvd=2505663441 dropped=640636540 link=15390565329

worker-2-5: 1581949343.011855 recvd=3459004896 dropped=640636540 link=20638874080

 

 

One thing to keep in mind with the myrcom driver is the drops are shared.. so you aren't dropping 2260820124 per worker, you dropped 2260820124 total.

The number is right, but to work out the total drop % you need to sum recvd but not the dropped number.

 

Here's a script i wrote a while ago that does that.. can pipe netstats output to this:

 

#!/usr/bin/env python
from __future__ import  print_function
import sys
import re
from collections import defaultdict

totals = defaultdict(int)
host_dropped = {}

total_rx = total_drop = 0

for line in sys.stdin:
    parts = re.split('[ =:]+', line.strip())
    node, time, _, recvd, _, dropped, _, link = parts
    host = node[:-2]
    totals[host] += int(link)

    total_rx += int(link)
    if host not in host_dropped:
        total_drop += int(dropped)
        host_dropped[host] = int(dropped)

for host, total in sorted(totals.items()):
    if not total: continue
    d = host_dropped[host]
    print("%s dropped=%d rx=%d %0.2f%%" % (host, d, total, 100.0*d/total))

print()
print("Totals dropped=%d rx=%d %0.2f%%" % (total_drop, total_rx, 100.0*total_drop/total_rx))

 

may need to tweak the host = node[:-2] line if you run more than 9 workers

 

running that on your output I get

 

worker-1 dropped=2260820124 rx=71909138670 3.14%
worker-2 dropped=640636540 rx=82342415425 0.78%

 

which isn't so bad.. without accounting for the shared drops it looks more like 15% and 4%.  Double the number of workers and fix the load balancing and you should be able to get that to zero.

 

-- 

Justin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200217/3305333d/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5561 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200217/3305333d/attachment-0001.bin 

From justin at corelight.com  Mon Feb 17 13:44:43 2020
From: justin at corelight.com (Justin Azoff)
Date: Mon, 17 Feb 2020 16:44:43 -0500
Subject: [Zeek] Dropping packets
In-Reply-To: <BN8PR20MB2532DC39DCD467639251A761EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
References: <BN8PR20MB2532BBAADDFF169CAD68A740EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuiNCi+PpSGgPynbOcSYYFq0N1RNw-mZS1CYP-YQpwwBOw@mail.gmail.com>
	<BN8PR20MB2532DC39DCD467639251A761EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
Message-ID: <CAPfnCuhmr7NEb4+CVDxkYJgHrg0oTVEH4hixuE7sbtj05Y16AA@mail.gmail.com>

On Mon, Feb 17, 2020 at 2:31 PM Joseph Fischetti <
Joseph.Fischetti at marist.edu> wrote:

> Thanks for the response Justin.
>
>
>
> I?ll start with disabling HT on the workers, and will reconfigure the
> workers with 8 threads each.  I?ll also pin the processes.
>
> The myricom configuration you quoted included environment variables:
>
> env_vars=SNF_APP_ID=1,SNF_DATARING_SIZE=16384MB,SNF_DESCRING_SIZE=4096MB
>
> What are those for/where can I find documentation/Should I include any of
> those?
>

It's in the myricom documentation.. searching for those settings should
find you a PDF of it.


> You also said the packets dropped are inclusive for the entire worker, but
> that the packets recvd are per thread:
>
> worker-1-1: 1581949346.194441 recvd=2178149468 dropped=2260820124
> link=15063051356
>
> worker-1-2: 1581949346.194473 recvd=274557259 dropped=2260820124
> link=13159459147
>
> worker-1-3: 1581949346.168558 recvd=1888926901 dropped=2260820124
> link=14773828789
>
> worker-1-4: 1581949346.081130 recvd=2110377092 dropped=2260820124
> link=14995278980
>
> worker-1-5: 1581949346.234478 recvd=1032618510 dropped=2260820124
> link=13917520398
>
>
>
> Those add up to ~7.5B recvd with 2.2B dropped.  That?s 30%, but your
> script said 3.  Am I looking at some numbers wrong?
>

Oh, yes!  I forgot one another detail that is specific to the myricom
nics.  The link and drop counters don't reset when you restart zeek...
possibly this could be worked around in the plugin to snapshot them at
startup and report the differences instead of the absolute numbers the card
reports...

Try running this:

zeekctl stop
zeekctl exec /opt/snf/bin/myri_counters -c -p 0 >/dev/null
zeekctl exec /opt/snf/bin/myri_counters -c -p 1 >/dev/null
zeekctl start

that will clear the counters and give you clean numbers.

-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200217/9951fe14/attachment.html 

From ito.satosi at jp.panasonic.com  Mon Feb 17 17:39:55 2020
From: ito.satosi at jp.panasonic.com (ito.satosi at jp.panasonic.com)
Date: Tue, 18 Feb 2020 01:39:55 +0000
Subject: [Zeek] uninstalling of bro(version2.6.1)
Message-ID: <0f0d40d5d5c045a1a394d317f03c3be1@JPA000SECMN13.palet.jp.panasonic.com>

Hi

I am deploying after bro / 2.6.6.1 / configure && make && sudo make install.
I am trying to change from bro(version:2.6.1) to Zeek (version:3.0.0) under ubuntu environment. 
For that I want to uninstall bro2.6.1, can you please tell me the procedure.
For example, is it the procedure to stop with broctl and then delete the folder of /usr/local/bro?

Best Regards,
Satoshi Ito


From Joseph.Fischetti at marist.edu  Tue Feb 18 05:41:30 2020
From: Joseph.Fischetti at marist.edu (Joseph Fischetti)
Date: Tue, 18 Feb 2020 13:41:30 +0000
Subject: [Zeek] Dropping packets
In-Reply-To: <CAPfnCuhmr7NEb4+CVDxkYJgHrg0oTVEH4hixuE7sbtj05Y16AA@mail.gmail.com>
References: <BN8PR20MB2532BBAADDFF169CAD68A740EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuiNCi+PpSGgPynbOcSYYFq0N1RNw-mZS1CYP-YQpwwBOw@mail.gmail.com>
	<BN8PR20MB2532DC39DCD467639251A761EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuhmr7NEb4+CVDxkYJgHrg0oTVEH4hixuE7sbtj05Y16AA@mail.gmail.com>
Message-ID: <BN8PR20MB2532B6D79A9986EC52B4A7C3EE110@BN8PR20MB2532.namprd20.prod.outlook.com>

Can?t clear the counters, I get an ?Operation not permitted?: 

>             Lanai uptime (seconds):                59830

>          Counters uptime (seconds):                59830

>                    Net send KBytes:                    0

>                    Net recv KBytes:          16291892294

>                      Ethernet send:                    8

>                Ethernet Small recv:             12077187

>                  Ethernet Big recv:                    0

>                 Ethernet recv down:                    0

>              Ethernet recv overrun:                  236

>                      SNF send pkts:                    0

>                      SNF recv pkts:          21273184129

>                 SNF drop ring full:                73133

>                         Interrupts:               361990

>             Net bad PHY/CRC32 drop:                    8

>                  Net overflow drop:                10266

>                    Net Recv PAUSEs:                    0

>     Ethernet Multicast filter drop:               153131

>       Ethernet Unicast filter drop:            616646672

> Cannot clear counters: Operation not permitted

 

For some reason with the workers named as you suggested, everything starts up fine but after ~15 minutes, zeekctl status reports the workers as ?stopped?, though all the processes are still running on them and the logger is still receiving data.  We reverted back to the worker-[1234] notation that we had before and they start and stay running.

 

Any time we try and pin_cpu?s the workers crash.  I was able to get things to start and stay running with 10 lb_procs each (unpinned), though I?m going to try and figure out how to get these counters to clear.

 

Note, we?ve disabled hyperthreading and included the environment variables that you suggested.  Those are the main changes that we?ve made so far.

 

From: Justin Azoff <justin at corelight.com> 
Sent: Monday, February 17, 2020 4:45 PM
To: Joseph Fischetti <Joseph.Fischetti at marist.edu>
Cc: zeek at zeek.org
Subject: Re: [Zeek] Dropping packets

 

[EXTERNAL EMAIL]

On Mon, Feb 17, 2020 at 2:31 PM Joseph Fischetti <Joseph.Fischetti at marist.edu <mailto:Joseph.Fischetti at marist.edu> > wrote:

Thanks for the response Justin.

 

I?ll start with disabling HT on the workers, and will reconfigure the workers with 8 threads each.  I?ll also pin the processes.

The myricom configuration you quoted included environment variables:

env_vars=SNF_APP_ID=1,SNF_DATARING_SIZE=16384MB,SNF_DESCRING_SIZE=4096MB

What are those for/where can I find documentation/Should I include any of those?

 

It's in the myricom documentation.. searching for those settings should find you a PDF of it.

 

You also said the packets dropped are inclusive for the entire worker, but that the packets recvd are per thread:

worker-1-1: 1581949346.194441 recvd=2178149468 dropped=2260820124 link=15063051356

worker-1-2: 1581949346.194473 recvd=274557259 dropped=2260820124 link=13159459147

worker-1-3: 1581949346.168558 recvd=1888926901 dropped=2260820124 link=14773828789

worker-1-4: 1581949346.081130 recvd=2110377092 dropped=2260820124 link=14995278980

worker-1-5: 1581949346.234478 recvd=1032618510 dropped=2260820124 link=13917520398

 

Those add up to ~7.5B recvd with 2.2B dropped.  That?s 30%, but your script said 3.  Am I looking at some numbers wrong?

 

Oh, yes!  I forgot one another detail that is specific to the myricom nics.  The link and drop counters don't reset when you restart zeek... possibly this could be worked around in the plugin to snapshot them at startup and report the differences instead of the absolute numbers the card reports...

 

Try running this:

 

zeekctl stop

zeekctl exec /opt/snf/bin/myri_counters -c -p 0 >/dev/null

zeekctl exec /opt/snf/bin/myri_counters -c -p 1 >/dev/null 

zeekctl start

 

that will clear the counters and give you clean numbers.

 

-- 

Justin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200218/9c00fe7f/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5561 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200218/9c00fe7f/attachment-0001.bin 

From clopmz at outlook.com  Tue Feb 18 05:53:28 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Tue, 18 Feb 2020 13:53:28 +0000
Subject: [Zeek] zeekctl netstats returns time out
In-Reply-To: <1A1F98C3-8152-4568-AB43-FECB4B7C1455@outlook.com>
References: <028E6FD5-11D2-46FA-A5C0-5A32BC4A839F@outlook.com>
	<CAMzgZ0+1sO4UK3+SHBGZeipszjgeA3ZZokTmTuHHy3GMGSDrrw@mail.gmail.com>,
	<1A1F98C3-8152-4568-AB43-FECB4B7C1455@outlook.com>
Message-ID: <DB8PR10MB3595DA499A7D9320983429E8DB110@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>

Any idea about how to debug this error?


Regards,
C. L. Martinez


________________________________________
From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> on behalf of Carlos Lopez <clopmz at outlook.com>
Sent: 15 February 2020 23:21
To: Jon Siwek
Cc: zeek at zeek.org
Subject: Re: [Zeek] zeekctl netstats returns time out

Many thanks Jon. Regarding TCP connectivity, I have neither ipfw nor pf enabled between manager and workers. And respecting to "busy" system, shouldn't be the problem either. For example, my top output in standalone config:

last pid:  6492;  load averages:  0.16,  0.22,  0.22                                                                                                                                up 0+06:21:48  22:20:43
44 threads:    1 running, 43 sleeping
CPU:  0.0% user,  0.0% nice,  1.9% system,  0.0% interrupt, 98.1% idle
Mem: 51M Active, 58M Inact, 679M Wired, 271M Buf, 5137M Free
Swap: 4096M Total, 4096M Free

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
45091 root         22    0   460M   111M select   1  18:29   4.71% zeek{zeek}
 6492 root         20    0  1044M  4144K CPU0     0   0:00   0.05% top
45091 root         20    0   460M   111M uwait    0   0:22   0.02% zeek{caf.clock}
39952 _ntp         20  -20  1038M  4000K select   1   0:03   0.01% ntpd
45407 root         20    0  1044M  9912K select   1   0:00   0.01% sshd
45091 root         20    0   460M   111M uwait    1   0:09   0.01% zeek{caf.multiplexer}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.ntp/Log::WRITER_}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.files/Log::WRITE}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.capture_loss/Log}
45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.dns/Log::WRITER_}
45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.ssl/Log::WRITER_}
45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.http/Log::WRITER}
45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.loaded_scripts/L}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.packet_filter/Lo}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.stats/Log::WRITE}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.conn/Log::WRITER}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.software/Log::WR}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.known_services/L}
45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.x509/Log::WRITER}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.notice/Log::WRIT}
45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.ssh/Log::WRITER_}
45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.kerberos/Log::WR}
45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.broker/Log::WRIT}
45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.weird/Log::WRITE}
45091 root         20    0   460M   111M uwait    0   0:00   0.00% zeek{zk.dhcp/Log::WRITER}
45091 root         20    0   460M   111M uwait    1   0:00   0.00% zeek{zk.known_certs/Log:}
45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.known_hosts/Log:}
96485 root         20    0    17M  6920K select   0   0:00   0.00% sendmail
45091 root         20    0   460M   111M select   0   0:00   0.00% zeek{caf.multiplexer}


--
Regards,
C. L. Martinez

?On 15/02/2020, 18:57, "Jon Siwek" <jsiwek at corelight.com> wrote:

    Zeek 3.0.1's `zeekctl netstats` is working for me in FreeBSD 12.1.
    TCP connectivity is required for that command to work and you can read
    more about the ports involved for further troubleshooting here:

        https://github.com/zeek/zeekctl#zeek-communication

    If the Zeek processes are particularly busy, that could also be a
    reason for timing out.  The `CommTimeout` (default 10 seconds) can be
    increased in `zeekctl.cfg` in that case.

    - Jon

    On Sat, Feb 15, 2020 at 8:46 AM Carlos Lopez <clopmz at outlook.com> wrote:
    >
    > Hi all,
    >
    >
    >
    > Every time I run ?zeekctl netstats? returns time out under FreeBSD 12.1 hosts using netmap:
    >
    >
    >
    > root at fbsdzeek01:/nsm/zeek/logs/current # zeekctl netstats
    >
    >
    >
    > Warning: ZeekControl plugin uses legacy BroControl API. Use
    >
    > 'import ZeekControl.plugin' instead of 'import BroControl.plugin'
    >
    >
    >
    >        zeek: <error: time-out>
    >
    >
    >
    > This behavior occurs in both standalone and cluster configurations. Any idea? Maybe is it a bug?
    >
    >
    >
    > --
    >
    > Regards,
    >
    > C. L. Martinez
    >
    > _______________________________________________
    > Zeek mailing list
    > zeek at zeek.org
    > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



_______________________________________________
Zeek mailing list
zeek at zeek.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From justin at corelight.com  Tue Feb 18 09:27:14 2020
From: justin at corelight.com (Justin Azoff)
Date: Tue, 18 Feb 2020 12:27:14 -0500
Subject: [Zeek] Dropping packets
In-Reply-To: <BN8PR20MB2532B6D79A9986EC52B4A7C3EE110@BN8PR20MB2532.namprd20.prod.outlook.com>
References: <BN8PR20MB2532BBAADDFF169CAD68A740EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuiNCi+PpSGgPynbOcSYYFq0N1RNw-mZS1CYP-YQpwwBOw@mail.gmail.com>
	<BN8PR20MB2532DC39DCD467639251A761EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuhmr7NEb4+CVDxkYJgHrg0oTVEH4hixuE7sbtj05Y16AA@mail.gmail.com>
	<BN8PR20MB2532B6D79A9986EC52B4A7C3EE110@BN8PR20MB2532.namprd20.prod.outlook.com>
Message-ID: <CAPfnCugK8ksejTN=r8AeiEStbQYJjOk4y1dyggsusaAop8nx8g@mail.gmail.com>

On Tue, Feb 18, 2020 at 8:41 AM Joseph Fischetti <
Joseph.Fischetti at marist.edu> wrote:

> Can?t clear the counters, I get an ?Operation not permitted?:
>

Hmm.. are you running zeek as root or a regular user? may need to use sudo,
or tweak the permissions on the /dev/myri.. (i think?) files.


> For some reason with the workers named as you suggested, everything starts
> up fine but after ~15 minutes, zeekctl status reports the workers as
> ?stopped?, though all the processes are still running on them and the
> logger is still receiving data.  We reverted back to the worker-[1234]
> notation that we had before and they start and stay running.
>

That doesn't make a lot of sense.. It's just the name, it doesn't affect
anything else... Maybe it was just a coincidence?


> Any time we try and pin_cpu?s the workers crash.  I was able to get things
> to start and stay running with 10 lb_procs each (unpinned), though I?m
> going to try and figure out how to get these counters to clear.
>

That also doesn't make a lot of sense.   Pinning the workers just causes
them to be started via cpuset locked to a core, everything else is the same.

How exactly are the workers crashing?  Are you getting a crash report from
zeekctl? or in dmesg?

-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200218/abd28ca2/attachment.html 

From christopher.hobbs at corelight.com  Tue Feb 18 10:16:13 2020
From: christopher.hobbs at corelight.com (Christopher Hobbs)
Date: Tue, 18 Feb 2020 10:16:13 -0800
Subject: [Zeek] scan.zeek question - exclude IP addresses
In-Reply-To: <CAHs9D0sA2O=n+6t9BBhAmZGP1QOimqu+DP3Vnn43=q7SSZrjvw@mail.gmail.com>
References: <CAHs9D0sdM0yk4pYWp_E=K-kp1w5ptcVtUJpWmJoRH1RaPQBxSA@mail.gmail.com>
	<CAJ6bFK3p1HWyJePwpMhZJm4-fFVWeqr4GnCYffGJBEo6cjn6HA@mail.gmail.com>
	<CAHs9D0tcJFHgAMwFig54Aouj3J1+2kHUpuMkk-Ke3nfVx_8jVA@mail.gmail.com>
	<CAHs9D0sA2O=n+6t9BBhAmZGP1QOimqu+DP3Vnn43=q7SSZrjvw@mail.gmail.com>
Message-ID: <CAN6Ztik7sAUd1a6Cwgq9fEE5EN8FUFA74sfyaW69Ea56M+LScw@mail.gmail.com>

I've heard about the darknet error before but I'm not sure I have a
relevant fix for it.  I'm sorry.  I've only touched these scripts in
passing.

Here are the changes I made to scan.zeek if it's helpful at all to
you.  I'm not going to make a PR for this because I certainly don't
recommend the use of scan.zeek under heavy load but maybe it'll help
you with some ideas:

https://github.com/corelight-chris/zeek/blob/64331b1ace775ee86442ccae79e62d20e79ce0e5/scripts/policy/misc/scan.zeek

Note that I just dredged this out of some of my local tinkering and
didn't bring any btests with it.  I think it's functional, though.

cmh

On Fri, Feb 14, 2020 at 8:28 AM Gordon Wallum <glwallum at gmail.com> wrote:
>
> Also the @load packages/bro-is-darknet is erroring since it is not installed on my Zeek environment, do I need to use the zeek package manager to install it?
>
> On Fri, Feb 14, 2020 at 9:20 AM Gordon Wallum <glwallum at gmail.com> wrote:
>>
>> Thank you, this makes sense logically but I can't figure out how to use the hook to exclude. The code below throws an error
>>
>>  ## Override this hook to ignore particular scan connections
>>     global Scan::scan_policy: hook(scanner: addr, victim: addr, scanned_port: port)
>> {
>>             if (( victim in exvictim_ips) || ( scanner in exscanner_ips ) || ( scanned_port in exscanned_ports))
>> break;
>> }
>>
>> On Thu, Feb 13, 2020 at 2:49 PM Micha? Purzy?ski <michalpurzynski1 at gmail.com> wrote:
>>>
>>> A couple of things.
>>>
>>> First, you should use Justin's simple-scan. As others have pointed out, the stock scanning detection script can behave poorly and it's hardly extensible.
>>>
>>> https://github.com/ncsa/bro-simple-scan
>>>
>>> (it's also packaged)
>>>
>>> Second - you can either ignore connections so the detection algorithm won't count them (with the hook from the simple-scan code), or you can write a notice policy and ignore some notices. Up to you - we just ignore some connections.
>>>
>>> Inside Justin's package, you will find a hook - this is what we use to ignore a set of destination and source IP addresses and some destination ports
>>>
>>> https://github.com/ncsa/bro-simple-scan/blob/master/scripts/scan.bro#L87
>>>
>>> Here's how we use that hook
>>>
>>> https://gist.github.com/mpurzynski/96a26c42874898447554531b6df9a4bb
>>>
>>> The input framework is what we use to update the list runtime. Nowadays you could use the configuration framework instead.
>>>
>>> https://corelight.blog/2018/02/13/runtime-options-the-bro-configuration-framework/
>>>
>>>
>>> Either way, you do not have to modify any upstream package.
>>>
>>>
>>> On Thu, Feb 13, 2020 at 11:23 AM Gordon Wallum <glwallum at gmail.com> wrote:
>>>>
>>>> Hello!
>>>>
>>>> I am new with Zeek and looking to learn more. I am currently using the scan.zeek script (https://github.com/zeek/zeek/blob/master/scripts/policy/misc/scan.zeek) for port scanning detection.
>>>>
>>>> I want to exclude certain source IP addresses from this script but I am not sure the best way to do so. It seems like a comparison with the key$host variable, but not sure where or how to do this logic in Zeek.
>>>>
>>>> Any advice would be appreciated
>>>>
>>>> Thank you
>>>> _______________________________________________
>>>> Zeek mailing list
>>>> zeek at zeek.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From jsiwek at corelight.com  Tue Feb 18 10:35:20 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Tue, 18 Feb 2020 10:35:20 -0800
Subject: [Zeek] uninstalling of bro(version2.6.1)
In-Reply-To: <0f0d40d5d5c045a1a394d317f03c3be1@JPA000SECMN13.palet.jp.panasonic.com>
References: <0f0d40d5d5c045a1a394d317f03c3be1@JPA000SECMN13.palet.jp.panasonic.com>
Message-ID: <CAMzgZ0KwwBor8fE_Q94GCvZF7p9+FauC8BHV52j_41WpoVQ3=w@mail.gmail.com>

Yeah, that's basically the uninstall process:

* broctl stop
* if you made any customizations to /usr/local/bro/etc/* or
/usr/local/bro/share/bro/site/local.bro, you may want to save those to
port to Zeek 3.0.x installation
* by default, all files live under /usr/local/bro/ prefix so deleting
that should be everything

- Jon

On Mon, Feb 17, 2020 at 5:42 PM <ito.satosi at jp.panasonic.com> wrote:
>
> Hi
>
> I am deploying after bro / 2.6.6.1 / configure && make && sudo make install.
> I am trying to change from bro(version:2.6.1) to Zeek (version:3.0.0) under ubuntu environment.
> For that I want to uninstall bro2.6.1, can you please tell me the procedure.
> For example, is it the procedure to stop with broctl and then delete the folder of /usr/local/bro?
>
> Best Regards,
> Satoshi Ito
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

From justin at corelight.com  Tue Feb 18 10:43:59 2020
From: justin at corelight.com (Justin Azoff)
Date: Tue, 18 Feb 2020 13:43:59 -0500
Subject: [Zeek] bro-simple-scan exclude IP addresses
In-Reply-To: <CAHs9D0v5FvSNk46ZrMQhCoqCmbdgjy4PzrW=VcZvn=D0iw_7Aw@mail.gmail.com>
References: <CAHs9D0v5FvSNk46ZrMQhCoqCmbdgjy4PzrW=VcZvn=D0iw_7Aw@mail.gmail.com>
Message-ID: <CAPfnCui+EcBYJULuhce9ZOcmLXjcyhxuZcmehvADwVnz9LCb_g@mail.gmail.com>

Ah.. a bit of confusion here.. but nothing too hard to fix.  So what
Micha? showed here:
https://gist.github.com/mpurzynski/96a26c42874898447554531b6df9a4bb was
almost exactly what you needed.

Undo any changes you made to the scripts (or just reinstall them i guess).
The scan policy hook itself is already there for this exact purpose, so you
don't need to change anything.

Just make a new file called scan-policy.zeek that contains

redef Config::config_files += { "/path/to/config.dat" };
module PortScanning;
export {
    option exvictim_ips: set[addr] = {};
    option exscanner_ips: set[addr] = {};
    option exscanned_ports: set[port] = {};
}

hook Scan::scan_policy(scanner: addr, victim: addr, scanned_port: port)
{
    if (( victim in exvictim_ips) || ( scanner in exscanner_ips ) || (
scanned_port in exscanned_ports))
        break;
}

done! that's all you need.


On Fri, Feb 14, 2020 at 2:57 PM Gordon Wallum <glwallum at gmail.com> wrote:

> I'm new to Zeek and looking for help with bro-simple-scan to exclude Ip
> addresses. I am trying to use runtime options and if breaks to the script
> to accomplish this.
>
> After running i get an error for my options variables. I don't know if I
> should move my excludes to a different part of the script or if my options
> are just not working right.
>
> Any help would be greatly appreciated
>
> *Error*
> error in /opt/bro/share/zeek/policy/custom-scripts/./bro-simple-scan2,
> line 276: unknown identifier exvictim_ips, at or near "exvictim_ips"
>
>
> *My config steps: *
>
> Edit local.bro to include the config file:
> redef Config::config_files += { "/path/to/config.dat" };
>
> *Create config file with variables: *
> PortScanning::exvictim_ips
> PortScanning::exscanner_ips xxx.xxx.xxx.xxx,yyy.yyy.yyy.yyy
> PortScanning::exscanned_ports
>
>
> Edit the bro-simple-scan script:
>
> *Added module and export variable options (after @loads)*
> module PortScanning;
> export {
>     option exvictim_ips: set[addr] = {};
>     option exscanner_ips: set[addr] = {};
>     option exscanned_ports: set[port] = {};
> }
>
> *Added if break (in the cluster hook Scan::scan_policy)*
>
> if ( hook Scan::scan_policy(scanner, victim, scanned_port) )
>         {
> if (( victim in exvictim_ips) || ( scanner in exscanner_ips ) || (
> scanned_port in exscanned_ports))
> break;
>
> *Add an if break (in the standalone hook Scan::scan_policy)*
>
> if ( hook Scan::scan_policy(scanner, victim, scanned_port) )
>         {
> if (( victim in exvictim_ips) || ( scanner in exscanner_ips ) || (
> scanned_port in exscanned_ports))
> break;
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200218/6e8cbacd/attachment.html 

From Joseph.Fischetti at marist.edu  Tue Feb 18 10:57:39 2020
From: Joseph.Fischetti at marist.edu (Joseph Fischetti)
Date: Tue, 18 Feb 2020 18:57:39 +0000
Subject: [Zeek] Dropping packets
In-Reply-To: <CAPfnCugK8ksejTN=r8AeiEStbQYJjOk4y1dyggsusaAop8nx8g@mail.gmail.com>
References: <BN8PR20MB2532BBAADDFF169CAD68A740EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuiNCi+PpSGgPynbOcSYYFq0N1RNw-mZS1CYP-YQpwwBOw@mail.gmail.com>
	<BN8PR20MB2532DC39DCD467639251A761EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuhmr7NEb4+CVDxkYJgHrg0oTVEH4hixuE7sbtj05Y16AA@mail.gmail.com>
	<BN8PR20MB2532B6D79A9986EC52B4A7C3EE110@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCugK8ksejTN=r8AeiEStbQYJjOk4y1dyggsusaAop8nx8g@mail.gmail.com>
Message-ID: <BN8PR20MB25327DC8E048614BA3E69DFAEE110@BN8PR20MB2532.namprd20.prod.outlook.com>

Hmm.. are you running zeek as root or a regular user? may need to use sudo, or tweak the permissions on the /dev/myri.. (i think?) files.

-          We?re running zeek as an unprivileged user.  I was able to go onto each worker as root and issue the commands to clear the counters.

 

That doesn't make a lot of sense.. It's just the name, it doesn't affect anything else... Maybe it was just a coincidence?

-          When we named the workers ?worker-1-A?, etc, the processes that were started up were ?worker-1-A-1? through ?worker-1-A-N?.  Is it possible that zeekctl doesn?t parse something correctly when it checks to make sure the processes are running?  Zeekctl stop failed to stop the workers, zeeksctl status showed them as stopped.  They were still running and the logger was still working.  The only way to kill them was to do a ?killall zeek? from the workers.

 

That also doesn't make a lot of sense.   Pinning the workers just causes them to be started via cpuset locked to a core, everything else is the same.

How exactly are the workers crashing?  Are you getting a crash report from zeekctl? or in dmesg?

-          No crash reports that I recall.  I can try and force it to happen again tomorrow.  I?d like to let things go for 24 hours and see where we end up.  We?ve been running for 5 hours and our stats currently looks like this [1].  Note, I modified the script that you provided to work with more than 9 threads.  (host = re.sub(r'\-[0-9]*$','', node))

-          We?re currently running 10 threads per worker *unpinned*

-          Medium load average on both boxes is around 6

-          Memory usage on both boxes is around 75G.

-          I?m going to find out if there?s something we can do in the Aristas so they send some more traffic to the secondary interface.  I need to find out how they?re configured and why.

 

 

[1]

bro at bro-master-1:~$ zeekctl netstats  | ~/calcDrop.py

 

Warning: ZeekControl plugin uses legacy BroControl API. Use

'import ZeekControl.plugin' instead of 'import BroControl.plugin'

 

worker-1 dropped=9051079 rx=6537521948 0.14%

worker-2 dropped=0 rx=6766088683 0.00%

worker-3 dropped=0 rx=437828006 0.00%

worker-4 dropped=0 rx=466874444 0.00%

 

Totals dropped=9051079 rx=14208313081 0.06%

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200218/d39f0f94/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5561 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200218/d39f0f94/attachment-0001.bin 

From justin at corelight.com  Tue Feb 18 11:07:05 2020
From: justin at corelight.com (Justin Azoff)
Date: Tue, 18 Feb 2020 14:07:05 -0500
Subject: [Zeek] Dropping packets
In-Reply-To: <BN8PR20MB25327DC8E048614BA3E69DFAEE110@BN8PR20MB2532.namprd20.prod.outlook.com>
References: <BN8PR20MB2532BBAADDFF169CAD68A740EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuiNCi+PpSGgPynbOcSYYFq0N1RNw-mZS1CYP-YQpwwBOw@mail.gmail.com>
	<BN8PR20MB2532DC39DCD467639251A761EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuhmr7NEb4+CVDxkYJgHrg0oTVEH4hixuE7sbtj05Y16AA@mail.gmail.com>
	<BN8PR20MB2532B6D79A9986EC52B4A7C3EE110@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCugK8ksejTN=r8AeiEStbQYJjOk4y1dyggsusaAop8nx8g@mail.gmail.com>
	<BN8PR20MB25327DC8E048614BA3E69DFAEE110@BN8PR20MB2532.namprd20.prod.outlook.com>
Message-ID: <CAPfnCuj598V_bzsEq7V1Z4z_wA2tHSNxRJzMDD1gi8y70bMxXw@mail.gmail.com>

On Tue, Feb 18, 2020 at 1:57 PM Joseph Fischetti <
Joseph.Fischetti at marist.edu> wrote:

> Hmm.. are you running zeek as root or a regular user? may need to use
> sudo, or tweak the permissions on the /dev/myri.. (i think?) files.
>
> -          We?re running zeek as an unprivileged user.  I was able to go
> onto each worker as root and issue the commands to clear the counters.
>

I see.. I bet we could fix the myricom plugin to workaround this problem..
basically just treat the initial drop/link numbers as zero.  I had worked
around it so long ago just by always clearing the counters when the cluster
is restarted.. but never looked into a permanent fix.


> That doesn't make a lot of sense.. It's just the name, it doesn't affect
> anything else... Maybe it was just a coincidence?
>
> -          When we named the workers ?worker-1-A?, etc, the processes
> that were started up were ?worker-1-A-1? through ?worker-1-A-N?.  Is it
> possible that zeekctl doesn?t parse something correctly when it checks to
> make sure the processes are running?  Zeekctl stop failed to stop the
> workers, zeeksctl status showed them as stopped.  They were still running
> and the logger was still working.  The only way to kill them was to do a
> ?killall zeek? from the workers.
>
Yeah.. that's the weird part.. it doesn't parse anything or really care
about the names at all, it just keeps track of the pid.  You stopped the
old workers before renaming and starting the new ones right?  not doing so
can cause similar sounding problems,  but I'm not sure if that's what you
ran into.

That also doesn't make a lot of sense.   Pinning the workers just causes
> them to be started via cpuset locked to a core, everything else is the same.
>
> How exactly are the workers crashing?  Are you getting a crash report from
> zeekctl? or in dmesg?
>
> -          No crash reports that I recall.  I can try and force it to
> happen again tomorrow.  I?d like to let things go for 24 hours and see
> where we end up.  We?ve been running for 5 hours and our stats currently
> looks like this [1].  Note, I modified the script that you provided to work
> with more than 9 threads.  (host = re.sub(r'\-[0-9]*$','', node))
>
Nice :-)  Looks like your overall stats are pretty good with the cleared
counters and the extra workers.

> -          We?re currently running 10 threads per worker **unpinned**
>
> -          Medium load average on both boxes is around 6
>
> -          Memory usage on both boxes is around 75G.
>

https://github.com/corelight/zeek-ssl-clear-state and
https://github.com/corelight/zeek-smb-clear-state may help a bit with the
memory usage.. 75G is a bit high for only 10 workers.. .though some of that
reported number is  the myricom buffer I think.


> -          I?m going to find out if there?s something we can do in the
> Aristas so they send some more traffic to the secondary interface.  I need
> to find out how they?re configured and why.
>

It does look like something is setup weirdly if the traffic is not evenly
balanced.. Maybe you have 2 different capture groups setup instead of just
dumping everything into one group split across 4 nics? The closer you look
the more problems you find :-)

-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200218/b7ce0fc2/attachment.html 

From jsiwek at corelight.com  Tue Feb 18 11:16:57 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Tue, 18 Feb 2020 11:16:57 -0800
Subject: [Zeek] zeekctl netstats returns time out
In-Reply-To: <DB8PR10MB3595DA499A7D9320983429E8DB110@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>
References: <028E6FD5-11D2-46FA-A5C0-5A32BC4A839F@outlook.com>
	<CAMzgZ0+1sO4UK3+SHBGZeipszjgeA3ZZokTmTuHHy3GMGSDrrw@mail.gmail.com>
	<1A1F98C3-8152-4568-AB43-FECB4B7C1455@outlook.com>
	<DB8PR10MB3595DA499A7D9320983429E8DB110@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>
Message-ID: <CAMzgZ0LxJqVL3NrwN_dfYLZHZpVDuBi+4xgnvVrVbWmhe7XAFQ@mail.gmail.com>

Since this is working in my own environment, we could maybe compare
configs until we find the differences.  What's the node.cfg you use?
If it's all just a single node using localhost, these are some of the
first things that come to mind for troubleshooting:

Confirm TCP connectivity:

# nc -zv 127.0.0.1 47761
Connection to localhost 47761 port [tcp/*] succeeded!

There's also the other 47761+ ports to try, but likely all get the
same result as the first one.  An IPv4 vs. IPv6 config issue might
also be a problem and can try variations of "::1" and "localhost" in
place of "127.0.0.1" if it's all one node.  To really get all IPv4,
think you can even set 127.0.0.1 in node.cfg and run like this:

    ZEEK_DEFAULT_LISTEN_ADDRESS=127.0.0.1 /usr/local/zeek/bin/zeekctl deploy

The high-level connection attempts are also logged here:

    /usr/local/zeek/logs/current/broker.log

See anything interesting there?  It should have several initial
"peer-added" and "handshake successful" entries for the initial
cluster setup and then for each time you try something like `zeekctl
netstats worker-1` it will have a pair of "peer-added" and
"connection-terminated" entries.

- Jon

On Tue, Feb 18, 2020 at 5:53 AM Carlos Lopez <clopmz at outlook.com> wrote:
>
> Any idea about how to debug this error?
>
>
> Regards,
> C. L. Martinez
>
>
> ________________________________________
> From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> on behalf of Carlos Lopez <clopmz at outlook.com>
> Sent: 15 February 2020 23:21
> To: Jon Siwek
> Cc: zeek at zeek.org
> Subject: Re: [Zeek] zeekctl netstats returns time out
>
> Many thanks Jon. Regarding TCP connectivity, I have neither ipfw nor pf enabled between manager and workers. And respecting to "busy" system, shouldn't be the problem either. For example, my top output in standalone config:
>
> last pid:  6492;  load averages:  0.16,  0.22,  0.22                                                                                                                                up 0+06:21:48  22:20:43
> 44 threads:    1 running, 43 sleeping
> CPU:  0.0% user,  0.0% nice,  1.9% system,  0.0% interrupt, 98.1% idle
> Mem: 51M Active, 58M Inact, 679M Wired, 271M Buf, 5137M Free
> Swap: 4096M Total, 4096M Free
>
>   PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
> 45091 root         22    0   460M   111M select   1  18:29   4.71% zeek{zeek}
>  6492 root         20    0  1044M  4144K CPU0     0   0:00   0.05% top
> 45091 root         20    0   460M   111M uwait    0   0:22   0.02% zeek{caf.clock}
> 39952 _ntp         20  -20  1038M  4000K select   1   0:03   0.01% ntpd
> 45407 root         20    0  1044M  9912K select   1   0:00   0.01% sshd
> 45091 root         20    0   460M   111M uwait    1   0:09   0.01% zeek{caf.multiplexer}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.ntp/Log::WRITER_}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.files/Log::WRITE}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.capture_loss/Log}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.dns/Log::WRITER_}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.ssl/Log::WRITER_}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.http/Log::WRITER}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.loaded_scripts/L}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.packet_filter/Lo}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.stats/Log::WRITE}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.conn/Log::WRITER}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.software/Log::WR}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.known_services/L}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.x509/Log::WRITER}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.notice/Log::WRIT}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.ssh/Log::WRITER_}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.kerberos/Log::WR}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.broker/Log::WRIT}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.weird/Log::WRITE}
> 45091 root         20    0   460M   111M uwait    0   0:00   0.00% zeek{zk.dhcp/Log::WRITER}
> 45091 root         20    0   460M   111M uwait    1   0:00   0.00% zeek{zk.known_certs/Log:}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.known_hosts/Log:}
> 96485 root         20    0    17M  6920K select   0   0:00   0.00% sendmail
> 45091 root         20    0   460M   111M select   0   0:00   0.00% zeek{caf.multiplexer}
>
>
> --
> Regards,
> C. L. Martinez
>
> ?On 15/02/2020, 18:57, "Jon Siwek" <jsiwek at corelight.com> wrote:
>
>     Zeek 3.0.1's `zeekctl netstats` is working for me in FreeBSD 12.1.
>     TCP connectivity is required for that command to work and you can read
>     more about the ports involved for further troubleshooting here:
>
>         https://github.com/zeek/zeekctl#zeek-communication
>
>     If the Zeek processes are particularly busy, that could also be a
>     reason for timing out.  The `CommTimeout` (default 10 seconds) can be
>     increased in `zeekctl.cfg` in that case.
>
>     - Jon
>
>     On Sat, Feb 15, 2020 at 8:46 AM Carlos Lopez <clopmz at outlook.com> wrote:
>     >
>     > Hi all,
>     >
>     >
>     >
>     > Every time I run ?zeekctl netstats? returns time out under FreeBSD 12.1 hosts using netmap:
>     >
>     >
>     >
>     > root at fbsdzeek01:/nsm/zeek/logs/current # zeekctl netstats
>     >
>     >
>     >
>     > Warning: ZeekControl plugin uses legacy BroControl API. Use
>     >
>     > 'import ZeekControl.plugin' instead of 'import BroControl.plugin'
>     >
>     >
>     >
>     >        zeek: <error: time-out>
>     >
>     >
>     >
>     > This behavior occurs in both standalone and cluster configurations. Any idea? Maybe is it a bug?
>     >
>     >
>     >
>     > --
>     >
>     > Regards,
>     >
>     > C. L. Martinez
>     >
>     > _______________________________________________
>     > Zeek mailing list
>     > zeek at zeek.org
>     > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From jpl at didconcept.com  Tue Feb 18 16:26:18 2020
From: jpl at didconcept.com (Jean-Philippe Luiggi)
Date: Tue, 18 Feb 2020 19:26:18 -0500
Subject: [Zeek] zeekctl netstats returns time out
In-Reply-To: <DB8PR10MB3595DA499A7D9320983429E8DB110@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>
References: <028E6FD5-11D2-46FA-A5C0-5A32BC4A839F@outlook.com>
	<CAMzgZ0+1sO4UK3+SHBGZeipszjgeA3ZZokTmTuHHy3GMGSDrrw@mail.gmail.com>
	<1A1F98C3-8152-4568-AB43-FECB4B7C1455@outlook.com>
	<DB8PR10MB3595DA499A7D9320983429E8DB110@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>
Message-ID: <20200219002618.gc53kk42ojhdtfqc@didconcept.com>

On Tue, Feb 18, 2020 at 01:53:28PM +0000, Carlos Lopez wrote:
> Any idea about how to debug this error?
> 
> 
> Regards,
> C. L. Martinez

Hi Carlos,

As Jon said, let's check network things.

Could you confirm the ip/port used on your side ?

# netstat -an
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address          Foreign Address
(state)    
tcp4       0      0 10.0.1.1.47760         *.*                    LISTEN




Cheers.

Jean-Philippe.

From jpl at didconcept.com  Tue Feb 18 16:37:46 2020
From: jpl at didconcept.com (Jean-Philippe Luiggi)
Date: Tue, 18 Feb 2020 19:37:46 -0500
Subject: [Zeek] uninstalling of bro(version2.6.1)
In-Reply-To: <0f0d40d5d5c045a1a394d317f03c3be1@JPA000SECMN13.palet.jp.panasonic.com>
References: <0f0d40d5d5c045a1a394d317f03c3be1@JPA000SECMN13.palet.jp.panasonic.com>
Message-ID: <20200219003746.iwxi4wswqs6vb2cp@didconcept.com>

On Tue, Feb 18, 2020 at 01:39:55AM +0000, ito.satosi at jp.panasonic.com wrote:
> Hi
> 
> I am deploying after bro / 2.6.6.1 / configure && make && sudo make install.
> I am trying to change from bro(version:2.6.1) to Zeek (version:3.0.0) under ubuntu environment. 
> For that I want to uninstall bro2.6.1, can you please tell me the procedure.
> For example, is it the procedure to stop with broctl and then delete the folder of /usr/local/bro?
> 
> Best Regards,
> Satoshi Ito

Hi,

May I suggest a tip I use ? 
I always install "Zeek" into a specific directory then use a logical link.
Please note that you should use the logical link in your config files.

Example:

(Today).

# ./configure --prefix=/op/zeek-3.0.1; make; make install
# cd /opt/
# ln -s zeek-3.0.1 zeek


(Tomorrow).

# ./configure --prefix=/op/zeek-3.0.2; make; make install
# cd /opt/
# rm zeek
# ln -s zeek-3.0.2 zeek


With such setup, you just have to copy *.cfg and the local policies
you use into the new directory.
If your setup refers to /opt/zeek, it's easy to switch back to old
version if you need it.

Cheers.

Jean-Philippe.


From pkelley at hyperionavenue.com  Tue Feb 18 17:14:47 2020
From: pkelley at hyperionavenue.com (Patrick Kelley)
Date: Tue, 18 Feb 2020 20:14:47 -0500
Subject: [Zeek] uninstalling of bro(version2.6.1)
In-Reply-To: <20200219003746.iwxi4wswqs6vb2cp@didconcept.com>
References: <20200219003746.iwxi4wswqs6vb2cp@didconcept.com>
Message-ID: <65E812B9-A3B5-4742-99BD-4C66E24B5486@hyperionavenue.com>

That?s a great tip!

Jean-Philippe, I forgot that I used to build that into my scripts. Appreciate the reminder.

Patrick Kelley, CISSP, C|EH, ITIL
CTO
patrick.kelley at criticalpathsecurity.com


> On Feb 18, 2020, at 7:45 PM, Jean-Philippe Luiggi <jpl at didconcept.com> wrote:
> 
> ?On Tue, Feb 18, 2020 at 01:39:55AM +0000, ito.satosi at jp.panasonic.com wrote:
>> Hi
>> 
>> I am deploying after bro / 2.6.6.1 / configure && make && sudo make install.
>> I am trying to change from bro(version:2.6.1) to Zeek (version:3.0.0) under ubuntu environment. 
>> For that I want to uninstall bro2.6.1, can you please tell me the procedure.
>> For example, is it the procedure to stop with broctl and then delete the folder of /usr/local/bro?
>> 
>> Best Regards,
>> Satoshi Ito
> 
> Hi,
> 
> May I suggest a tip I use ? 
> I always install "Zeek" into a specific directory then use a logical link.
> Please note that you should use the logical link in your config files.
> 
> Example:
> 
> (Today).
> 
> # ./configure --prefix=/op/zeek-3.0.1; make; make install
> # cd /opt/
> # ln -s zeek-3.0.1 zeek
> 
> 
> (Tomorrow).
> 
> # ./configure --prefix=/op/zeek-3.0.2; make; make install
> # cd /opt/
> # rm zeek
> # ln -s zeek-3.0.2 zeek
> 
> 
> With such setup, you just have to copy *.cfg and the local policies
> you use into the new directory.
> If your setup refers to /opt/zeek, it's easy to switch back to old
> version if you need it.
> 
> Cheers.
> 
> Jean-Philippe.
> 
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200218/98ff1e36/attachment.html 

From michalpurzynski1 at gmail.com  Tue Feb 18 17:58:15 2020
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Tue, 18 Feb 2020 17:58:15 -0800
Subject: [Zeek] uninstalling of bro(version2.6.1)
In-Reply-To: <65E812B9-A3B5-4742-99BD-4C66E24B5486@hyperionavenue.com>
References: <20200219003746.iwxi4wswqs6vb2cp@didconcept.com>
	<65E812B9-A3B5-4742-99BD-4C66E24B5486@hyperionavenue.com>
Message-ID: <CAJ6bFK3vrK-gdxsoNGOa45ctaHZhzV5kMK6RS+HWf5uvU6HJ0w@mail.gmail.com>

May I suggest building packages with Zeek? Just a good sysadmin's practice
;)
Or using Johanna's excellent packages
https://www.zeek.org/download/packages.html


On Tue, Feb 18, 2020 at 5:17 PM Patrick Kelley <pkelley at hyperionavenue.com>
wrote:

> That?s a great tip!
>
> Jean-Philippe, I forgot that I used to build that into my scripts.
> Appreciate the reminder.
>
> *Patrick Kelley, CISSP, C|EH, ITIL*
> *CTO*
> patrick.kelley at criticalpathsecurity.com
>
>
> On Feb 18, 2020, at 7:45 PM, Jean-Philippe Luiggi <jpl at didconcept.com>
> wrote:
>
> ?On Tue, Feb 18, 2020 at 01:39:55AM +0000, ito.satosi at jp.panasonic.com
> wrote:
>
> Hi
>
>
> I am deploying after bro / 2.6.6.1 / configure && make && sudo make
> install.
>
> I am trying to change from bro(version:2.6.1) to Zeek (version:3.0.0)
> under ubuntu environment.
>
> For that I want to uninstall bro2.6.1, can you please tell me the
> procedure.
>
> For example, is it the procedure to stop with broctl and then delete the
> folder of /usr/local/bro?
>
>
> Best Regards,
>
> Satoshi Ito
>
>
> Hi,
>
> May I suggest a tip I use ?
> I always install "Zeek" into a specific directory then use a logical link.
> Please note that you should use the logical link in your config files.
>
> Example:
>
> (Today).
>
> # ./configure --prefix=/op/zeek-3.0.1; make; make install
> # cd /opt/
> # ln -s zeek-3.0.1 zeek
>
>
> (Tomorrow).
>
> # ./configure --prefix=/op/zeek-3.0.2; make; make install
> # cd /opt/
> # rm zeek
> # ln -s zeek-3.0.2 zeek
>
>
> With such setup, you just have to copy *.cfg and the local policies
> you use into the new directory.
> If your setup refers to /opt/zeek, it's easy to switch back to old
> version if you need it.
>
> Cheers.
>
> Jean-Philippe.
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200218/0bc9a349/attachment.html 

From al.kefallonitis at gmail.com  Tue Feb 18 23:05:35 2020
From: al.kefallonitis at gmail.com (Alex Kefallonitis)
Date: Wed, 19 Feb 2020 09:05:35 +0200
Subject: [Zeek] LLMNR/NBT-NS Poisoning and Relay Attacks
In-Reply-To: <CAHv=MuoJ9NrW=k3Sj_yaC_gSxgW=NjyPpz-PPzJ4SvADjScZcw@mail.gmail.com>
References: <CAHv=MuoJ9NrW=k3Sj_yaC_gSxgW=NjyPpz-PPzJ4SvADjScZcw@mail.gmail.com>
Message-ID: <CAHv=MupBy+up9edz-F8fSr57v_R+Xk3AX4Ad_-0rk02zxHwVvw@mail.gmail.com>

I guess using the suggestion on corelight site could help built a script
based on the port used in llmnr



*Attackers with the ability to poison or intercept DNS queries can
strengthen their foothold into a targeted network by inserting or
overwriting records for sensitive hosts. For example, if an attacker can
generate a response for "wpad," they can redirect users' web traffic
through a man-in-the-middle of their choosing. LLMNR may be disabled in an
enterprise network, in which case any LLMNR (UDP 5355) traffic would be
immediately actionable based on events within Zeek's conn.log file.*


Kind regards,

Alex Kefallonitis


???? ???, 10 ??? 2020 ???? 6:17 ?.?., ?/? Alex Kefallonitis <
al.kefallonitis at gmail.com> ??????:

> Hi All,
>
>  Any script that can log LLMNR/NBT-NS Poisoning and Relay Attacks ?
>
> Thanks in advanced.
>
> Kind Regards,
> Alex Kefallonitis
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200219/3f70f27c/attachment-0001.html 

From clopmz at outlook.com  Wed Feb 19 00:24:06 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Wed, 19 Feb 2020 08:24:06 +0000
Subject: [Zeek] zeekctl netstats returns time out
In-Reply-To: <CAMzgZ0LxJqVL3NrwN_dfYLZHZpVDuBi+4xgnvVrVbWmhe7XAFQ@mail.gmail.com>
References: <028E6FD5-11D2-46FA-A5C0-5A32BC4A839F@outlook.com>
	<CAMzgZ0+1sO4UK3+SHBGZeipszjgeA3ZZokTmTuHHy3GMGSDrrw@mail.gmail.com>
	<1A1F98C3-8152-4568-AB43-FECB4B7C1455@outlook.com>
	<DB8PR10MB3595DA499A7D9320983429E8DB110@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>,
	<CAMzgZ0LxJqVL3NrwN_dfYLZHZpVDuBi+4xgnvVrVbWmhe7XAFQ@mail.gmail.com>
Message-ID: <DB8PR10MB359502666256C8ABD731C1E9DB100@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>

Good morning,

 Many thanks for your help Jon. All my config that you have requested.

- node.cfg:

[manager]
type=manager
host=localhost

[logger]
type=logger
host=localhost

[proxy-1]
type=proxy
host=localhost

[worker-1]
type=worker
host=localhost
interface=netmap:vtnet2

[worker-2]
type=worker
host=localhost
interface=netmap:vtnet3

- sockstat -l4:
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     sendmail   50520 3  tcp4   127.0.0.1:25          *:*
root     sshd       31667 3  tcp4   *:22                  *:*
root     zeek       27934 17 tcp46  *:47765               *:*
root     zeek       20657 17 tcp46  *:47764               *:*
root     zeek       84818 16 tcp46  *:47763               *:*
root     zeek       91782 16 tcp46  *:47762               *:*
root     zeek       94252 17 tcp46  *:47761               *:*
root     owlhnode   334   6  tcp46  *:50002               *:*
root     nfsd       46617 5  tcp4   *:2049                *:*
root     mountd     37746 8  udp4   *:650                 *:*
root     mountd     37746 9  tcp4   *:650                 *:*
root     rpcbind    52182 9  udp4   *:111                 *:*
root     rpcbind    52182 10 udp4   *:947                 *:*
root     rpcbind    52182 11 tcp4   *:111                 *:*
?        ?          ?     ?  udp4   *:2049                *:*

- nc command (also ipv6 works):

root at fbsdzeek01:~ # nc -zv 127.0.0.1 47761
Connection to 127.0.0.1 47761 port [tcp/*] succeeded!

 - broker.log:

{"ts":"2020-02-19T08:13:35.215215Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10007,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:35.214435Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47761,"message":"received handshake from remote core"}
{"ts":"2020-02-19T08:13:37.198510Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10008,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:37.198165Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10009,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:36.965614Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47761,"message":"received handshake from remote core"}
{"ts":"2020-02-19T08:13:41.269695Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10010,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:41.275816Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10011,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:36.965614Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47762,"message":"received handshake from remote core"}
{"ts":"2020-02-19T08:13:41.271616Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10012,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:41.505503Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10013,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:41.579196Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47761,"message":"received handshake from remote core"}
{"ts":"2020-02-19T08:13:41.964270Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47761,"message":"received handshake from remote core"}

 As you can see, nothing strange here ... As you said, I have changed the definition of "localhost" in the node.cfg file to IP 127.0.0.1 ... and it works!

 Problem solved. Many thanks Jon...

Regards,
C. L. Martinez


________________________________________
From: Jon Siwek <jsiwek at corelight.com>
Sent: 18 February 2020 20:16
To: Carlos Lopez
Cc: zeek at zeek.org
Subject: Re: [Zeek] zeekctl netstats returns time out

Since this is working in my own environment, we could maybe compare
configs until we find the differences.  What's the node.cfg you use?
If it's all just a single node using localhost, these are some of the
first things that come to mind for troubleshooting:

Confirm TCP connectivity:

# nc -zv 127.0.0.1 47761
Connection to localhost 47761 port [tcp/*] succeeded!

There's also the other 47761+ ports to try, but likely all get the
same result as the first one.  An IPv4 vs. IPv6 config issue might
also be a problem and can try variations of "::1" and "localhost" in
place of "127.0.0.1" if it's all one node.  To really get all IPv4,
think you can even set 127.0.0.1 in node.cfg and run like this:

    ZEEK_DEFAULT_LISTEN_ADDRESS=127.0.0.1 /usr/local/zeek/bin/zeekctl deploy

The high-level connection attempts are also logged here:

    /usr/local/zeek/logs/current/broker.log

See anything interesting there?  It should have several initial
"peer-added" and "handshake successful" entries for the initial
cluster setup and then for each time you try something like `zeekctl
netstats worker-1` it will have a pair of "peer-added" and
"connection-terminated" entries.

- Jon

On Tue, Feb 18, 2020 at 5:53 AM Carlos Lopez <clopmz at outlook.com> wrote:
>
> Any idea about how to debug this error?
>
>
> Regards,
> C. L. Martinez
>
>
> ________________________________________
> From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> on behalf of Carlos Lopez <clopmz at outlook.com>
> Sent: 15 February 2020 23:21
> To: Jon Siwek
> Cc: zeek at zeek.org
> Subject: Re: [Zeek] zeekctl netstats returns time out
>
> Many thanks Jon. Regarding TCP connectivity, I have neither ipfw nor pf enabled between manager and workers. And respecting to "busy" system, shouldn't be the problem either. For example, my top output in standalone config:
>
> last pid:  6492;  load averages:  0.16,  0.22,  0.22                                                                                                                                up 0+06:21:48  22:20:43
> 44 threads:    1 running, 43 sleeping
> CPU:  0.0% user,  0.0% nice,  1.9% system,  0.0% interrupt, 98.1% idle
> Mem: 51M Active, 58M Inact, 679M Wired, 271M Buf, 5137M Free
> Swap: 4096M Total, 4096M Free
>
>   PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
> 45091 root         22    0   460M   111M select   1  18:29   4.71% zeek{zeek}
>  6492 root         20    0  1044M  4144K CPU0     0   0:00   0.05% top
> 45091 root         20    0   460M   111M uwait    0   0:22   0.02% zeek{caf.clock}
> 39952 _ntp         20  -20  1038M  4000K select   1   0:03   0.01% ntpd
> 45407 root         20    0  1044M  9912K select   1   0:00   0.01% sshd
> 45091 root         20    0   460M   111M uwait    1   0:09   0.01% zeek{caf.multiplexer}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.ntp/Log::WRITER_}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.files/Log::WRITE}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.capture_loss/Log}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.dns/Log::WRITER_}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.ssl/Log::WRITER_}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.http/Log::WRITER}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.loaded_scripts/L}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.packet_filter/Lo}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.stats/Log::WRITE}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.conn/Log::WRITER}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.software/Log::WR}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.known_services/L}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.x509/Log::WRITER}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.notice/Log::WRIT}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.ssh/Log::WRITER_}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.kerberos/Log::WR}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.broker/Log::WRIT}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.weird/Log::WRITE}
> 45091 root         20    0   460M   111M uwait    0   0:00   0.00% zeek{zk.dhcp/Log::WRITER}
> 45091 root         20    0   460M   111M uwait    1   0:00   0.00% zeek{zk.known_certs/Log:}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.known_hosts/Log:}
> 96485 root         20    0    17M  6920K select   0   0:00   0.00% sendmail
> 45091 root         20    0   460M   111M select   0   0:00   0.00% zeek{caf.multiplexer}
>
>
> --
> Regards,
> C. L. Martinez
>
> ?On 15/02/2020, 18:57, "Jon Siwek" <jsiwek at corelight.com> wrote:
>
>     Zeek 3.0.1's `zeekctl netstats` is working for me in FreeBSD 12.1.
>     TCP connectivity is required for that command to work and you can read
>     more about the ports involved for further troubleshooting here:
>
>         https://github.com/zeek/zeekctl#zeek-communication
>
>     If the Zeek processes are particularly busy, that could also be a
>     reason for timing out.  The `CommTimeout` (default 10 seconds) can be
>     increased in `zeekctl.cfg` in that case.
>
>     - Jon
>
>     On Sat, Feb 15, 2020 at 8:46 AM Carlos Lopez <clopmz at outlook.com> wrote:
>     >
>     > Hi all,
>     >
>     >
>     >
>     > Every time I run ?zeekctl netstats? returns time out under FreeBSD 12.1 hosts using netmap:
>     >
>     >
>     >
>     > root at fbsdzeek01:/nsm/zeek/logs/current # zeekctl netstats
>     >
>     >
>     >
>     > Warning: ZeekControl plugin uses legacy BroControl API. Use
>     >
>     > 'import ZeekControl.plugin' instead of 'import BroControl.plugin'
>     >
>     >
>     >
>     >        zeek: <error: time-out>
>     >
>     >
>     >
>     > This behavior occurs in both standalone and cluster configurations. Any idea? Maybe is it a bug?
>     >
>     >
>     >
>     > --
>     >
>     > Regards,
>     >
>     > C. L. Martinez
>     >
>     > _______________________________________________
>     > Zeek mailing list
>     > zeek at zeek.org
>     > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From bill.de.ping at gmail.com  Wed Feb 19 08:13:53 2020
From: bill.de.ping at gmail.com (william de ping)
Date: Wed, 19 Feb 2020 18:13:53 +0200
Subject: [Zeek]  - new writer plugin
Message-ID: <CAKZA8xirpmhNuGa_+vxzZNRjLVNDBx5zcOwyHoE8qRJNRm_L-A@mail.gmail.com>

Hi all,

I want to make a permanent change into ASCII.cc file of the Ascii_writer.
After making the changes, I've created a dummy plugin and updated the src
and script dirs accordingly.

compiling the plugin works, however, running Zeek with the new
Ascii2_writer as the default writer fails with undefined symbols in the bif
references area.

This issue seems to be related to a linkage issue, since Ascii2_writer code
placed in <zeek_src>/src/logging/writers works properly.

Any ideas on what to do?

Simply copying the Ascii writer code, creating a plugin and using this
plugin under a different namespace fails for the same reason.

Thanks
B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200219/4f7ee4f1/attachment.html 

From justin at corelight.com  Wed Feb 19 08:34:19 2020
From: justin at corelight.com (Justin Azoff)
Date: Wed, 19 Feb 2020 11:34:19 -0500
Subject: [Zeek] - new writer plugin
In-Reply-To: <CAKZA8xirpmhNuGa_+vxzZNRjLVNDBx5zcOwyHoE8qRJNRm_L-A@mail.gmail.com>
References: <CAKZA8xirpmhNuGa_+vxzZNRjLVNDBx5zcOwyHoE8qRJNRm_L-A@mail.gmail.com>
Message-ID: <CAPfnCuiZZKbpNDN+7K4_FzKMi794-BwwADjTBZG1ZtKj_OqLuA@mail.gmail.com>

maybe take a peek at how https://github.com/ncsa/bro-zeromq-writer is put
together.. that's probably one of the simpler standalone log writer plugins.

On Wed, Feb 19, 2020 at 11:17 AM william de ping <bill.de.ping at gmail.com>
wrote:

> Hi all,
>
> I want to make a permanent change into ASCII.cc file of the Ascii_writer.
> After making the changes, I've created a dummy plugin and updated the src
> and script dirs accordingly.
>
> compiling the plugin works, however, running Zeek with the new
> Ascii2_writer as the default writer fails with undefined symbols in the bif
> references area.
>
> This issue seems to be related to a linkage issue, since Ascii2_writer
> code placed in <zeek_src>/src/logging/writers works properly.
>
> Any ideas on what to do?
>
> Simply copying the Ascii writer code, creating a plugin and using this
> plugin under a different namespace fails for the same reason.
>
> Thanks
> B
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200219/cc2466f1/attachment.html 

From bill.de.ping at gmail.com  Thu Feb 20 07:06:42 2020
From: bill.de.ping at gmail.com (william de ping)
Date: Thu, 20 Feb 2020 17:06:42 +0200
Subject: [Zeek] - new writer plugin
In-Reply-To: <CAPfnCuiZZKbpNDN+7K4_FzKMi794-BwwADjTBZG1ZtKj_OqLuA@mail.gmail.com>
References: <CAKZA8xirpmhNuGa_+vxzZNRjLVNDBx5zcOwyHoE8qRJNRm_L-A@mail.gmail.com>
	<CAPfnCuiZZKbpNDN+7K4_FzKMi794-BwwADjTBZG1ZtKj_OqLuA@mail.gmail.com>
Message-ID: <CAKZA8xg+fGHZ21MN9dm93+DtKYw6WtRYVhKua+AB2r6+kQHKSg@mail.gmail.com>

No luck with that.

I see that the Cmakefile and Makefile in the zeromq writer are the same as
mine.
It really seems to be related to a linkage issue.
Running the plugin as a shared object fails due to undefined symbols, yet
running the same code inside zeek/src works.

Any thoughts ?

Thanks
B

On Wed, Feb 19, 2020 at 6:34 PM Justin Azoff <justin at corelight.com> wrote:

> maybe take a peek at how https://github.com/ncsa/bro-zeromq-writer is put
> together.. that's probably one of the simpler standalone log writer plugins.
>
> On Wed, Feb 19, 2020 at 11:17 AM william de ping <bill.de.ping at gmail.com>
> wrote:
>
>> Hi all,
>>
>> I want to make a permanent change into ASCII.cc file of the Ascii_writer.
>> After making the changes, I've created a dummy plugin and updated the src
>> and script dirs accordingly.
>>
>> compiling the plugin works, however, running Zeek with the new
>> Ascii2_writer as the default writer fails with undefined symbols in the bif
>> references area.
>>
>> This issue seems to be related to a linkage issue, since Ascii2_writer
>> code placed in <zeek_src>/src/logging/writers works properly.
>>
>> Any ideas on what to do?
>>
>> Simply copying the Ascii writer code, creating a plugin and using this
>> plugin under a different namespace fails for the same reason.
>>
>> Thanks
>> B
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
>
> --
> Justin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200220/4ab635da/attachment.html 

From justin at corelight.com  Thu Feb 20 07:12:02 2020
From: justin at corelight.com (Justin Azoff)
Date: Thu, 20 Feb 2020 10:12:02 -0500
Subject: [Zeek] - new writer plugin
In-Reply-To: <CAKZA8xg+fGHZ21MN9dm93+DtKYw6WtRYVhKua+AB2r6+kQHKSg@mail.gmail.com>
References: <CAKZA8xirpmhNuGa_+vxzZNRjLVNDBx5zcOwyHoE8qRJNRm_L-A@mail.gmail.com>
	<CAPfnCuiZZKbpNDN+7K4_FzKMi794-BwwADjTBZG1ZtKj_OqLuA@mail.gmail.com>
	<CAKZA8xg+fGHZ21MN9dm93+DtKYw6WtRYVhKua+AB2r6+kQHKSg@mail.gmail.com>
Message-ID: <CAPfnCugxg2JXLZHAgNETxr8fmPtWKcr6ckKaEb=Sbyi2b8jQYg@mail.gmail.com>

On Thu, Feb 20, 2020 at 10:06 AM william de ping <bill.de.ping at gmail.com>
wrote:

> No luck with that.
>
> I see that the Cmakefile and Makefile in the zeromq writer are the same as
> mine.
> It really seems to be related to a linkage issue.
> Running the plugin as a shared object fails due to undefined symbols, yet
> running the same code inside zeek/src works.
>
> Any thoughts ?
>

what are the undefined symbols?

-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200220/206300e4/attachment.html 

From bill.de.ping at gmail.com  Thu Feb 20 07:24:57 2020
From: bill.de.ping at gmail.com (william de ping)
Date: Thu, 20 Feb 2020 17:24:57 +0200
Subject: [Zeek] - new writer plugin
In-Reply-To: <CAPfnCugxg2JXLZHAgNETxr8fmPtWKcr6ckKaEb=Sbyi2b8jQYg@mail.gmail.com>
References: <CAKZA8xirpmhNuGa_+vxzZNRjLVNDBx5zcOwyHoE8qRJNRm_L-A@mail.gmail.com>
	<CAPfnCuiZZKbpNDN+7K4_FzKMi794-BwwADjTBZG1ZtKj_OqLuA@mail.gmail.com>
	<CAKZA8xg+fGHZ21MN9dm93+DtKYw6WtRYVhKua+AB2r6+kQHKSg@mail.gmail.com>
	<CAPfnCugxg2JXLZHAgNETxr8fmPtWKcr6ckKaEb=Sbyi2b8jQYg@mail.gmail.com>
Message-ID: <CAKZA8xii4Lb6-hsF-BCQz2H=vSM9sEVyRfid4tmYevf-wHrQ-Q@mail.gmail.com>

Bifconst json_timestamps
WriterFrontend

and other related functions that are in the included h files (as in the
Ascii writer plugin)

Thanks
B

On Thu, Feb 20, 2020 at 5:12 PM Justin Azoff <justin at corelight.com> wrote:

> On Thu, Feb 20, 2020 at 10:06 AM william de ping <bill.de.ping at gmail.com>
> wrote:
>
>> No luck with that.
>>
>> I see that the Cmakefile and Makefile in the zeromq writer are the same
>> as mine.
>> It really seems to be related to a linkage issue.
>> Running the plugin as a shared object fails due to undefined symbols, yet
>> running the same code inside zeek/src works.
>>
>> Any thoughts ?
>>
>
> what are the undefined symbols?
>
> --
> Justin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200220/cfa9fea5/attachment.html 

From petiepooo at gmail.com  Thu Feb 20 14:03:52 2020
From: petiepooo at gmail.com (Pete Nelson)
Date: Thu, 20 Feb 2020 22:03:52 +0000
Subject: [Zeek] Workers occasionally using 102% CPU
Message-ID: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>

I'm seeing an interesting problem on zeek 3.0.1 (running stock
SecurityOnion sensor setup) where the main thread suddenly spikes to 100%
CPU and stays there.

The base setup: Ubuntu 16.04 kernel 4.15.0-88-generic, zeek is using
AF_PACKET to distribute packets across 8 worker processes.  Baseline load
for each worker is around 18-22% CPU.  Running strace for 1 second and
filtering with 'sort strace.out | uniq -c' on a normal worker looks like
this:

     31 futex(0x7f9586384a38, FUTEX_WAKE_PRIVATE, 1) = 1
     33 futex(0x7f9586384a64, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f9586384a60,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
      1 futex(0x7f9586d03a0c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f9586d03a08,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
   6671 nanosleep({0, 1000}, NULL)              = 0
      1 read(12, "@", 1)                        = 1
      1 read(14, "@", 1)                        = 1
     26 read(16, "@", 1)                        = 1
   6699 select(17, [6 8 10 11 12 14 16], [0], [0], {0, 0}) = 1 (out [0],
left {0, 0})
    198 select(17, [6 8 10 11 12 14 16], [0], [0], {0, 0}) = 2 (in [11],
out [0], left {0, 0})
      1 select(17, [6 8 10 11 12 14 16], [0], [0], {0, 0}) = 2 (in [12],
out [0], left {0, 0})
      1 select(17, [6 8 10 11 12 14 16], [0], [0], {0, 0}) = 2 (in [14],
out [0], left {0, 0})
     21 select(17, [6 8 10 11 12 14 16], [0], [0], {0, 0}) = 2 (in [16],
out [0], left {0, 0})
      1 select(17, [6 8 10 11 12 14 16], [0], [0], {0, 0}) = 3 (in [11 16],
out [0], left {0, 0})
      1 select(17, [6 8 10 11 12 14 16], [0], [0], {0, 0} <detached ...>
    101 select(17, [6 8 10 12 14 16], [], [], {0, 0}) = 0 (Timeout)
     92 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=127, ...}) = 0

Notice there are close to the same number of nanosleep calls as there are
select calls.

After some time - i've seen it happen within 4 minutes of start to several
hours afterwards - the usage suddenly spikes to 100%.  Packets continue to
be processed, and the load on the remaining workers stays about the same,
as does the load on logger, manager, and proxy.  Changing the number of
worker processes doesn't seem to prevent it.  There is no degredation in
output logging, since I have enough cores to compensate for that single hot
process.  Running strace for 1 second and filtering with 'sort strace.out |
uniq -c' looks like this:

      1 futex(0x7f270ef7ea38, FUTEX_WAKE_PRIVATE, 1) = 0
     30 futex(0x7f270ef7ea38, FUTEX_WAKE_PRIVATE, 1) = 1
     35 futex(0x7f270ef7ea64, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f270ef7ea60,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
      1 futex(0x7f270f900a6c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f270f900a68,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
      1 poll([{fd=16, events=POLLIN}], 1, -1)   = 1 ([{fd=16,
revents=POLLIN}])
      1 read(12, "@", 1)                        = 1
      1 read(14, "@", 1)                        = 1
     28 read(16, "@", 1)                        = 1
      1 select(12, [10 11], [0], [0], {0, 0})   = 1 (out [0], left {0, 0})
  21703 select(17, [6 8 10 11 12 14 16], [0], [0], {0, 0}) = 2 (in [16],
out [0], left {0, 0})
    141 select(17, [6 8 10 11 12 14 16], [0], [0], {0, 0}) = 3 (in [11 16],
out [0], left {0, 0})
      1 select(17, [6 8 10 11 12 14 16], [0], [0], {0, 0}) = 3 (in [12 16],
out [0], left {0, 0})
    109 select(17, [6 8 10 12 14 16], [], [], {0, 0}) = 1 (in [16], left
{0, 0})
    106 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=127, ...}) = 0

Note the complete lack of nanosleeps and the jump in the number of calls to
select.  Also notice that fds 12, 14, and 16 continue to be read, but
select always returns with data available on fd16.  procfs shows this (also
showing fd17 since that appears to be the other end of the pipe):

root:/proc/32327# ls -l fd/16 fd/17
lr-x------ 1 root root 64 Feb 20 20:43 fd/16 -> pipe:[48070104]
l-wx------ 1 root root 64 Feb 20 20:43 fd/17 -> pipe:[48070104]
root:/proc/32327# cat fdinfo/16
pos: 0
flags: 02004000
mnt_id: 12
root:/proc/32327# cat fdinfo/17
pos: 0
flags: 02000001
mnt_id: 12

lsof output shows these file descriptors in use on the main thread (and
thus on all):

zeek    32327 sguil    3u  a_inode     0,13        0      12847 [eventpoll]
zeek    32327 sguil    4r     FIFO     0,12      0t0   48059893 pipe
zeek    32327 sguil    5w     FIFO     0,12      0t0   48059893 pipe
zeek    32327 sguil    6r     FIFO     0,12      0t0   48059895 pipe
zeek    32327 sguil    7w     FIFO     0,12      0t0   48059895 pipe
zeek    32327 sguil    8r     FIFO     0,12      0t0   48059896 pipe
zeek    32327 sguil    9w     FIFO     0,12      0t0   48059896 pipe
zeek    32327 sguil   10u     IPv4 48059899      0t0        UDP
localhost:55072->localhost:domain
zeek    32327 sguil   11u     pack 48059900      0t0        ALL
type=SOCK_RAW
zeek    32327 sguil   12r     FIFO     0,12      0t0   48048835 pipe
zeek    32327 sguil   13w     FIFO     0,12      0t0   48048835 pipe
zeek    32327 sguil   14r     FIFO     0,12      0t0   48048836 pipe
zeek    32327 sguil   15w     FIFO     0,12      0t0   48048836 pipe
zeek    32327 sguil   16r     FIFO     0,12      0t0   48070104 pipe
zeek    32327 sguil   17w     FIFO     0,12      0t0   48070104 pipe
zeek    32327 sguil   18u     IPv6 48038789      0t0        TCP *:47770
(LISTEN)
zeek    32327 sguil   19u     IPv4 48038790      0t0        TCP
localhost:47996->localhost:47761 (ESTABLISHED)
zeek    32327 sguil   20u     IPv4 48038791      0t0        TCP
localhost:42076->localhost:47763 (ESTABLISHED)
zeek    32327 sguil   21u     IPv4 48038792      0t0        TCP
localhost:34920->localhost:47762 (ESTABLISHED)

Of the 7 threads created, the 4th (judging by the thread's pid) is calling
'write(17, "@", 1)' 10x per second and getting a return value of 1.

Any ideas what might be wrong?  Any suggestions for further diagnosis?
These are in production, so I can't do too much other than a restart and an
occasional strace.  I cannot reproduce in lab conditions.

I've just upgraded 2 of my sensors from 2.6.4 to 3.0.1, and this is
happening on one only, but its the more heavily loaded one.  I'm hoping to
resolve this before upgrading the remaining 10 sensors, as I don't want to
see it on others...
--
Pete
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200220/c9304a72/attachment.html 

From justin at corelight.com  Thu Feb 20 15:19:12 2020
From: justin at corelight.com (Justin Azoff)
Date: Thu, 20 Feb 2020 18:19:12 -0500
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
Message-ID: <CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>

On Thu, Feb 20, 2020 at 5:06 PM Pete Nelson <petiepooo at gmail.com> wrote:

> I'm seeing an interesting problem on zeek 3.0.1 (running stock
> SecurityOnion sensor setup) where the main thread suddenly spikes to 100%
> CPU and stays there.
>

Could be an elephant flow...


> Any ideas what might be wrong?  Any suggestions for further diagnosis?
> These are in production, so I can't do too much other than a restart and an
> occasional strace.  I cannot reproduce in lab conditions.
>

 Use perf!  Start with this:

    sudo perf record -C 3 -g -F199 --call-graph dwarf sleep 30

where 3 is the cpu core that worker is running on (zero indexed)

Then run

    sudo perf script |c++filt  > perf.out

Then process that with https://github.com/brendangregg/FlameGraph or even
better https://github.com/Netflix/flamescope

You may be able to skip all that and just run

    sudo perf top -C 3 -F199

to see what is going on, but recording helps if it's hard to catch or if
you want to automate it.

-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200220/e197fdd0/attachment.html 

From petiepooo at gmail.com  Thu Feb 20 15:43:55 2020
From: petiepooo at gmail.com (Pete Nelson)
Date: Thu, 20 Feb 2020 23:43:55 +0000
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
Message-ID: <CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>

Thanks, Justin, but I don't think it's an elephant flow.  I'm running snort
on the same host with the same global BPF rules (it still uses pf-ring for
clustering), which is even more sensitive to elephant flows, and it's not
spiking CPU.  Besides, it's fd16 that's triggering select, not fd11, which
is the raw socket.  The count of read calls on fd11 are pretty similar
between the good and bad strace summaries.

I don't have the workers pinned to a CPU, and there are other heavy-duty
processes on that host, so I'm not sure if perf will work for me.  I'll
read up on it and see if I can get some sort of results from it.  I'll also
strace some more and try to catch the right moment to at least figure out
what's happening around trigger time..  The load on that worker isn't so
high that strace interferes, and streaming its output through gzip sips
diskspace.

Also, it's now happening on the other node where I upgraded to v3.0.1.  I
only have 3 workers there, but load is light enough I could get by with 1
or 2.  When I kill the malfunctioning worker, it appears to respawn within
a couple minutes, which is good.  I didn't realize Zeek did that!

Any other ideas?  I can't be the only one seeing this, can I?
--
Pete
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200220/72a2594e/attachment-0001.html 

From justin at corelight.com  Thu Feb 20 16:32:17 2020
From: justin at corelight.com (Justin Azoff)
Date: Thu, 20 Feb 2020 19:32:17 -0500
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
Message-ID: <CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>

On Thu, Feb 20, 2020 at 6:44 PM Pete Nelson <petiepooo at gmail.com> wrote:

>  I don't have the workers pinned to a CPU, and there are other heavy-duty
> processes on that host, so I'm not sure if perf will work for me.  I'll
> read up on it and see if I can get some sort of results from it.  I'll also
> strace some more and try to catch the right moment to at least figure out
> what's happening around trigger time..  The load on that worker isn't so
> high that strace interferes, and streaming its output through gzip sips
> diskspace.
>

Oh.. you can use -p with perf and give it a pid.  strace is really not
helpful at all for this sort of thing.


> Also, it's now happening on the other node where I upgraded to v3.0.1.  I
> only have 3 workers there, but load is light enough I could get by with 1
> or 2.  When I kill the malfunctioning worker, it appears to respawn within
> a couple minutes, which is good.  I didn't realize Zeek did that!
>

zeekctl cron does that when it runs.   The new supervisor stuff
https://docs.zeek.org/en/master/frameworks/supervisor.html aims to have
things restarted immediately.


> Any other ideas?  I can't be the only one seeing this, can I?
>

Hard to say until we know what the 'this' is :-)

-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200220/040ef5b6/attachment.html 

From mfernandez at mitre.org  Fri Feb 21 05:48:32 2020
From: mfernandez at mitre.org (Fernandez, Mark I)
Date: Fri, 21 Feb 2020 13:48:32 +0000
Subject: [Zeek] ICAP Protocol Analyzer, by MITRE
Message-ID: <SN6PR09MB3134794E94908381D595A45ACF120@SN6PR09MB3134.namprd09.prod.outlook.com>

All,

 

MITRE developed a Bro/Zeek analyzer plugin for the Internet Content Adaptation Protocol (ICAP), per RFC 3507.  It provides a novel means by which to inspect Hyper-Text Transfer Protocol Secure (HTTPS) traffic in plain-text, via a web proxy (for more information, see the background section at the bottom of this email). The ICAP analyzer code is publicly released as open source, under MITRE case number 16-3871.

 

Download.  The ICAP analyzer is available for download via the Zeek package manager (pending) and at the following URL:

*	https://github.com/mitre/icap

 

Caveats.

*	The ICAP analyzer was originally developed for Bro v2.4.x and v2.5.x.
*	The plugin seems to build correctly on Bro v2.6.x and Zeek v3.0.x.
*	The ICAP dynamic protocol detection signature (dpd.sig) file is available as part of the plugin, but disabled by default.  The analyzer registers via port 1344/tcp, so it should not require dpd.sig.
*	The ICAP analyzer still uses the .bro file extension for the scripts, so you may see some deprecation warnings, but it should run as expected.  We'll make the ICAP analyzer fully compliant with Zeek 3.0 soon.
*	I do not have any ICAP data or packet capture files to share.  If anyone has ICAP data they can share, please let me know.  It would be great to add the btest feature to the plugin package.

 

Please let me know if you encounter any errors.

 

Mark I. Fernandez

The MITRE Corporation

mfernandez at mitre.org

 

 

Background:

MITRE presented the ICAP analyzer at BroCon 2016.  You can find links to the conference abstract, slides, and video at https://www.zeek.org/community/brocon2016.html.

 

The BroCon 2016 abstract is included below for your convenience...

 

This presentation describes the Internet Content Adaptation Protocol (ICAP) analyzer for the Bro Network Security Monitor tool as a novel means by which to inspect Hyper-Text Transfer Protocol Secure (HTTPS) traffic in plain-text.  It contains an overview of the ICAP specification, an overview of the Bro ICAP analyzer and how it interfaces with the HTTP analyzer and other Bro analyzers.

 

ICAP is defined by Internet Engineering Task Force (IETF) Request for Comments (RFC) 3507.  It is commonly implemented by web proxy devices to modify content of HTTP messages based on anti-virus (AV), data loss prevention (DLP), or other content inspection services.  Either the web client's original HTTP request and/or a web server's original HTTP response are encapsulated within the ICAP payload that is sent from the web proxy to the AV/DLP proxy.  The AV/DLP proxy inspects the ICAP payload to determine whether or not the content should be modified, according to security policy.  For example, if the web page originating from an external HTTP server contains malicious content that triggers an AV signature, then the AV proxy would modify or replace the content with an error or notification message.

 

The objectives of the Bro ICAP analyzer are (a) to monitor the link between the web proxy and AV/DLP proxy; (b) to extract the original HTTP message from the ICAP payload; and (c) to invoke the Bro HTTP analyzer, fully utilizing Bro's built-in analysis capabilities for HTTP inspection, file extraction, etc. 

 

While this may appear to be a convoluted method to monitor HTTP traffic, the true benefit of the Bro ICAP analyzer is achieved if the web proxy is capable of intercepting encrypted HTTPS traffic.  In such a case, the ICAP payload would contain a decrypted copy of the HTTPS message because the AV/DLP proxy would require the content to be plain text in order to inspect it appropriately.  The Bro ICAP analyzer takes advantage of this.   By extracting the decrypted copy of the HTTPS message from the ICAP payload and injecting it into the Bro HTTP analyzer, the Bro ICAP analyzer provides a novel means by which to inspect encrypted web traffic in plain-text.

 

ICAP Abstract for BroCon 2016. Approved for public release. Distribution unlimited. Case number 16-2621.

(c) 2016 The MITRE Corporation. All rights reserved.



From akgraner at corelight.com  Fri Feb 21 05:53:33 2020
From: akgraner at corelight.com (Amber Graner)
Date: Fri, 21 Feb 2020 05:53:33 -0800
Subject: [Zeek] ICAP Protocol Analyzer, by MITRE
In-Reply-To: <SN6PR09MB3134794E94908381D595A45ACF120@SN6PR09MB3134.namprd09.prod.outlook.com>
References: <SN6PR09MB3134794E94908381D595A45ACF120@SN6PR09MB3134.namprd09.prod.outlook.com>
Message-ID: <CAJhOzuq-Vpnq5TOgS+XLAZ+goCF=HXy5mKQw9+vvEb65pfK0VA@mail.gmail.com>

Thanks, Mark!!

~Amber

On Fri, Feb 21, 2020 at 5:52 AM Fernandez, Mark I <mfernandez at mitre.org>
wrote:

> All,
>
>
>
> MITRE developed a Bro/Zeek analyzer plugin for the Internet Content
> Adaptation Protocol (ICAP), per RFC 3507.  It provides a novel means by
> which to inspect Hyper-Text Transfer Protocol Secure (HTTPS) traffic in
> plain-text, via a web proxy (for more information, see the background
> section at the bottom of this email). The ICAP analyzer code is publicly
> released as open source, under MITRE case number 16-3871.
>
>
>
> Download.  The ICAP analyzer is available for download via the Zeek
> package manager (pending) and at the following URL:
>
> *       https://github.com/mitre/icap
>
>
>
> Caveats.
>
> *       The ICAP analyzer was originally developed for Bro v2.4.x and
> v2.5.x.
> *       The plugin seems to build correctly on Bro v2.6.x and Zeek v3.0.x.
> *       The ICAP dynamic protocol detection signature (dpd.sig) file is
> available as part of the plugin, but disabled by default.  The analyzer
> registers via port 1344/tcp, so it should not require dpd.sig.
> *       The ICAP analyzer still uses the .bro file extension for the
> scripts, so you may see some deprecation warnings, but it should run as
> expected.  We'll make the ICAP analyzer fully compliant with Zeek 3.0 soon.
> *       I do not have any ICAP data or packet capture files to share.  If
> anyone has ICAP data they can share, please let me know.  It would be great
> to add the btest feature to the plugin package.
>
>
>
> Please let me know if you encounter any errors.
>
>
>
> Mark I. Fernandez
>
> The MITRE Corporation
>
> mfernandez at mitre.org
>
>
>
>
>
> Background:
>
> MITRE presented the ICAP analyzer at BroCon 2016.  You can find links to
> the conference abstract, slides, and video at
> https://www.zeek.org/community/brocon2016.html.
>
>
>
> The BroCon 2016 abstract is included below for your convenience...
>
>
>
> This presentation describes the Internet Content Adaptation Protocol
> (ICAP) analyzer for the Bro Network Security Monitor tool as a novel means
> by which to inspect Hyper-Text Transfer Protocol Secure (HTTPS) traffic in
> plain-text.  It contains an overview of the ICAP specification, an overview
> of the Bro ICAP analyzer and how it interfaces with the HTTP analyzer and
> other Bro analyzers.
>
>
>
> ICAP is defined by Internet Engineering Task Force (IETF) Request for
> Comments (RFC) 3507.  It is commonly implemented by web proxy devices to
> modify content of HTTP messages based on anti-virus (AV), data loss
> prevention (DLP), or other content inspection services.  Either the web
> client's original HTTP request and/or a web server's original HTTP response
> are encapsulated within the ICAP payload that is sent from the web proxy to
> the AV/DLP proxy.  The AV/DLP proxy inspects the ICAP payload to determine
> whether or not the content should be modified, according to security
> policy.  For example, if the web page originating from an external HTTP
> server contains malicious content that triggers an AV signature, then the
> AV proxy would modify or replace the content with an error or notification
> message.
>
>
>
> The objectives of the Bro ICAP analyzer are (a) to monitor the link
> between the web proxy and AV/DLP proxy; (b) to extract the original HTTP
> message from the ICAP payload; and (c) to invoke the Bro HTTP analyzer,
> fully utilizing Bro's built-in analysis capabilities for HTTP inspection,
> file extraction, etc.
>
>
>
> While this may appear to be a convoluted method to monitor HTTP traffic,
> the true benefit of the Bro ICAP analyzer is achieved if the web proxy is
> capable of intercepting encrypted HTTPS traffic.  In such a case, the ICAP
> payload would contain a decrypted copy of the HTTPS message because the
> AV/DLP proxy would require the content to be plain text in order to inspect
> it appropriately.  The Bro ICAP analyzer takes advantage of this.   By
> extracting the decrypted copy of the HTTPS message from the ICAP payload
> and injecting it into the Bro HTTP analyzer, the Bro ICAP analyzer provides
> a novel means by which to inspect encrypted web traffic in plain-text.
>
>
>
> ICAP Abstract for BroCon 2016. Approved for public release. Distribution
> unlimited. Case number 16-2621.
>
> (c) 2016 The MITRE Corporation. All rights reserved.
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
-- 

*Amber Graner*
Director of Community
Corelight, Inc

828.582.9469

Schedule time on my calendar here. <https://calendly.com/amber_graner>



 * Ask me about how you can participate in the Zeek (formerly Bro)
community.
 * Remember - ZEEK AND YOU SHALL FIND!!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200221/b8e9bc9a/attachment-0001.html 

From petiepooo at gmail.com  Fri Feb 21 07:41:32 2020
From: petiepooo at gmail.com (Pete Nelson)
Date: Fri, 21 Feb 2020 15:41:32 +0000
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
Message-ID: <CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>

On Fri, Feb 21, 2020 at 12:32 AM Justin Azoff <justin at corelight.com> wrote:
>
> Oh.. you can use -p with perf and give it a pid.  strace is really not helpful at all for this sort of thing.

Ah, thanks.  I'm not familiar with perf, so had to research some.
Here's a sample of running perf-top against it for a bit:

Samples: 2K of event 'cycles:ppp', Event count (approx.): 23397706264
  Children      Self  Shared Object                  Symbol
+   36.85%     0.72%  libc-2.23.so                   [.] __select
+   34.78%     1.18%  [kernel]                       [k] do_syscall_64
+   34.53%     0.83%  [kernel]                       [k]
entry_SYSCALL_64_after_hwframe
+   32.88%     0.56%  [kernel]                       [k] sys_select
+   29.65%     1.15%  [kernel]                       [k] core_sys_select
+   20.18%     9.99%  [kernel]                       [k] do_select
+   10.12%     0.00%  [unknown]                      [k] 0x00007fe5752e0b20
+    9.14%     8.91%  libjemalloc.so.1               [.] free
+    7.60%     7.43%  libjemalloc.so.1               [.] malloc
+    7.05%     6.86%  zeek                           [.]
iosource::Manager::FindSoonest
+    5.67%     1.20%  [kernel]                       [k] __fdget
+    5.42%     5.42%  zeek                           [.]
std::_Rb_tree<int, int, std::_Identity<int>, std::less<int>,
std::allocator<int> >::_M_insert_unique<in
+    4.44%     0.93%  [kernel]                       [k] __fget_light
+    4.08%     3.94%  [kernel]                       [k] __fget

I also captured call-graph data as you requested earlier, but I don't
know what I'm looking at... how should I get that to you?  Direct
email, or attach to reply and let the list manager strip and provide a
link?  It's only 350k once gzipped.
--
Pete

From shadowx787 at gmail.com  Fri Feb 21 08:07:56 2020
From: shadowx787 at gmail.com (Justin Mullins)
Date: Fri, 21 Feb 2020 11:07:56 -0500
Subject: [Zeek] Zeek Script Checking
Message-ID: <CAFY8ZinymkyqhD84A7Vn5vdO2T5JhYMGewbVnf5ScTVf3qRQgQ@mail.gmail.com>

For a new organization using Zeek is there a good system/methodology to
test new scripts before putting them into production? Best practices that
should be followed to avoid large performance hits?

I'm curious what different organizations do besides the obvious of just
deploying in a test environment and identifying any performance issues that
way.

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200221/6b71a152/attachment.html 

From tim at tsqrd.net  Fri Feb 21 13:33:31 2020
From: tim at tsqrd.net (Tim Thompson)
Date: Fri, 21 Feb 2020 15:33:31 -0600
Subject: [Zeek] zeek erros with zbalance_ipc
Message-ID: <CADOmza5moz=TZHbF5An-xfOuEzUx6Fr5gs-JcwP6xyzfakVM1Q@mail.gmail.com>

Version Info:
Zeek 3.2.0-dev.53
pfring-7.5.0-2838.x86_64

We currently utilize zbalance_ipc to load balance our traffic on our zeek
boxes.  I built a new box the other day, and now zeek refuses to work with
any interface that zbalance_ipc creates.  This works fine on all of our
existing boxes which are currently running Zeek 3.1.0-dev.292.

Example:
zbalance_ipc -i zc:eth1 -n 1 -m 1 -c 10 -g 1 -a (creates zc:10 at 0)

[root at server opt]# zeek -i zc:10 at 0
listening on zc:10 at 0

error: Failed to register fd 22 from PktSrc: Bad address
fatal error: Failed to register pktsrc fd with iosource_mgr

Has anyone ran into this issue?

Thanks,
Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200221/643d513b/attachment.html 

From tim at corelight.com  Fri Feb 21 13:40:46 2020
From: tim at corelight.com (Tim Wojtulewicz)
Date: Fri, 21 Feb 2020 14:40:46 -0700
Subject: [Zeek] zeek erros with zbalance_ipc
In-Reply-To: <CADOmza5moz=TZHbF5An-xfOuEzUx6Fr5gs-JcwP6xyzfakVM1Q@mail.gmail.com>
References: <CADOmza5moz=TZHbF5An-xfOuEzUx6Fr5gs-JcwP6xyzfakVM1Q@mail.gmail.com>
Message-ID: <F7D8C9F2-E18B-4FC3-8341-1F4DE0A9EB86@corelight.com>

That looks like an issue with the new IOSource changes in 3.1. Is the code for that plugin public and something I could go look at? We tested it pretty heavily with the pcap, af_packet, and myricom plugins so it might be something with that plugin specifically.

Tim

> On Feb 21, 2020, at 2:33 PM, Tim Thompson <tim at tsqrd.net> wrote:
> 
> Version Info:
> Zeek 3.2.0-dev.53
> pfring-7.5.0-2838.x86_64
> 
> We currently utilize zbalance_ipc to load balance our traffic on our zeek boxes.  I built a new box the other day, and now zeek refuses to work with any interface that zbalance_ipc creates.  This works fine on all of our existing boxes which are currently running Zeek 3.1.0-dev.292.
> 
> Example:
> zbalance_ipc -i zc:eth1 -n 1 -m 1 -c 10 -g 1 -a (creates zc:10 at 0)
> 
> [root at server opt]# zeek -i zc:10 at 0
> listening on zc:10 at 0
> 
> error: Failed to register fd 22 from PktSrc: Bad address
> fatal error: Failed to register pktsrc fd with iosource_mgr
> 
> Has anyone ran into this issue?  
> 
> Thanks,
> Tim
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



From tim at tsqrd.net  Fri Feb 21 14:41:01 2020
From: tim at tsqrd.net (Tim Thompson)
Date: Fri, 21 Feb 2020 16:41:01 -0600
Subject: [Zeek] zeek erros with zbalance_ipc
In-Reply-To: <F7D8C9F2-E18B-4FC3-8341-1F4DE0A9EB86@corelight.com>
References: <CADOmza5moz=TZHbF5An-xfOuEzUx6Fr5gs-JcwP6xyzfakVM1Q@mail.gmail.com>
	<F7D8C9F2-E18B-4FC3-8341-1F4DE0A9EB86@corelight.com>
Message-ID: <CADOmza521B+qW2zr56EhK7DGUd=TQSP=VOgAf4euFaeDhbJKNg@mail.gmail.com>

Does this help?

https://github.com/ntop/PF_RING/blob/dev/userland/examples_zc/zbalance_ipc.c


Thanks,
Tim

On Fri, Feb 21, 2020 at 3:40 PM Tim Wojtulewicz <tim at corelight.com> wrote:

> That looks like an issue with the new IOSource changes in 3.1. Is the code
> for that plugin public and something I could go look at? We tested it
> pretty heavily with the pcap, af_packet, and myricom plugins so it might be
> something with that plugin specifically.
>
> Tim
>
> > On Feb 21, 2020, at 2:33 PM, Tim Thompson <tim at tsqrd.net> wrote:
> >
> > Version Info:
> > Zeek 3.2.0-dev.53
> > pfring-7.5.0-2838.x86_64
> >
> > We currently utilize zbalance_ipc to load balance our traffic on our
> zeek boxes.  I built a new box the other day, and now zeek refuses to work
> with any interface that zbalance_ipc creates.  This works fine on all of
> our existing boxes which are currently running Zeek 3.1.0-dev.292.
> >
> > Example:
> > zbalance_ipc -i zc:eth1 -n 1 -m 1 -c 10 -g 1 -a (creates zc:10 at 0)
> >
> > [root at server opt]# zeek -i zc:10 at 0
> > listening on zc:10 at 0
> >
> > error: Failed to register fd 22 from PktSrc: Bad address
> > fatal error: Failed to register pktsrc fd with iosource_mgr
> >
> > Has anyone ran into this issue?
> >
> > Thanks,
> > Tim
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200221/738749c2/attachment.html 

From justin at corelight.com  Fri Feb 21 15:20:14 2020
From: justin at corelight.com (Justin Azoff)
Date: Fri, 21 Feb 2020 18:20:14 -0500
Subject: [Zeek] zeek erros with zbalance_ipc
In-Reply-To: <CADOmza5moz=TZHbF5An-xfOuEzUx6Fr5gs-JcwP6xyzfakVM1Q@mail.gmail.com>
References: <CADOmza5moz=TZHbF5An-xfOuEzUx6Fr5gs-JcwP6xyzfakVM1Q@mail.gmail.com>
Message-ID: <CAPfnCuhFoBTSo-1onegPGGmQYTbd3-Lfzhfc8SkOKa4+DafP-Q@mail.gmail.com>

On Fri, Feb 21, 2020 at 4:36 PM Tim Thompson <tim at tsqrd.net> wrote:

> Example:
> zbalance_ipc -i zc:eth1 -n 1 -m 1 -c 10 -g 1 -a (creates zc:10 at 0)
>
> [root at server opt]# zeek -i zc:10 at 0
> listening on zc:10 at 0
>

ah, you're using pf_ring through the libpcap wrapper and their wrapper is
doing something weird.  This problem should go away using
https://github.com/ntop/bro-pf_ring (and likely better performance too
since you won't need the wrapper)

That package may need a few updates to the cmake bits to compile properly
against zeek.. as well as the configure changes that make it possible to
build plugins without the zeek source code.

-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200221/7aa00647/attachment.html 

From tim at corelight.com  Fri Feb 21 15:28:26 2020
From: tim at corelight.com (Tim Wojtulewicz)
Date: Fri, 21 Feb 2020 16:28:26 -0700
Subject: [Zeek] zeek erros with zbalance_ipc
In-Reply-To: <CAPfnCuhFoBTSo-1onegPGGmQYTbd3-Lfzhfc8SkOKa4+DafP-Q@mail.gmail.com>
References: <CADOmza5moz=TZHbF5An-xfOuEzUx6Fr5gs-JcwP6xyzfakVM1Q@mail.gmail.com>
	<CAPfnCuhFoBTSo-1onegPGGmQYTbd3-Lfzhfc8SkOKa4+DafP-Q@mail.gmail.com>
Message-ID: <3EFAD170-8897-495B-8FDF-EFAB4D22DDDE@corelight.com>

Yah, that would definitely explain it. Libpcap is probably setting the file descriptor to something weird after trying to open ?zc:10? and that?s causing Zeek to throw an error. I?ll see if we can make that a little more friendly.

I recently fixed the af_packet plugin to build correctly with Zeek 3.1 and the process is pretty simple. If you want to look at those changes as a roadmap, go for it. We should have a blog post coming out pretty soon on zeek.org <http://zeek.org/> about it. Otherwise I?ll add it to my list for next week.

Tim

> On Feb 21, 2020, at 4:20 PM, Justin Azoff <justin at corelight.com> wrote:
> 
> On Fri, Feb 21, 2020 at 4:36 PM Tim Thompson <tim at tsqrd.net <mailto:tim at tsqrd.net>> wrote:
> Example:
> zbalance_ipc -i zc:eth1 -n 1 -m 1 -c 10 -g 1 -a (creates zc:10 at 0)
> 
> [root at server opt]# zeek -i zc:10 at 0
> listening on zc:10 at 0
> 
> ah, you're using pf_ring through the libpcap wrapper and their wrapper is doing something weird.  This problem should go away using https://github.com/ntop/bro-pf_ring <https://github.com/ntop/bro-pf_ring> (and likely better performance too since you won't need the wrapper) 
> 
> That package may need a few updates to the cmake bits to compile properly against zeek.. as well as the configure changes that make it possible to build plugins without the zeek source code.
> 
> -- 
> Justin
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200221/6c8196f8/attachment-0001.html 

From petiepooo at gmail.com  Sat Feb 22 07:20:05 2020
From: petiepooo at gmail.com (Pete Nelson)
Date: Sat, 22 Feb 2020 15:20:05 +0000
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
	<CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
Message-ID: <CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>

Perf output mailed to you, Justin.  I'm checking to see if other
SecurityOnion users are seeing this, but haven't heard anything yet.
The good news is that it doesn't break things, as it just consumes
more CPU than usual.

I appreciate you taking a look at this.  Please let me know if you
need anything else.
--
Pete

From shahin203 at hotmail.com  Sat Feb 22 15:55:21 2020
From: shahin203 at hotmail.com (Shahin Khan)
Date: Sat, 22 Feb 2020 23:55:21 +0000
Subject: [Zeek] Zeek Install instruction
Message-ID: <CH2PR14MB4118FDCBD731B77F2C29BA25EBEE0@CH2PR14MB4118.namprd14.prod.outlook.com>

Hello,
I am investigating my time install zeek on Ubuntu desktop but I could not successfully installed. Anybody know step by step instruction
install Zeek with picture on Ubuntu? Would you send me instruction how to install zeek easy way? Thanks

Best,
Shahin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200222/5cb51a40/attachment.html 

From christopher.hobbs at corelight.com  Sun Feb 23 17:34:51 2020
From: christopher.hobbs at corelight.com (Christopher Hobbs)
Date: Sun, 23 Feb 2020 17:34:51 -0800
Subject: [Zeek] Zeek Install instruction
In-Reply-To: <CH2PR14MB4118FDCBD731B77F2C29BA25EBEE0@CH2PR14MB4118.namprd14.prod.outlook.com>
References: <CH2PR14MB4118FDCBD731B77F2C29BA25EBEE0@CH2PR14MB4118.namprd14.prod.outlook.com>
Message-ID: <CAN6ZtinWLep1Uid9KR+iiyGX7FJWWxPHdhELqg6-gd3XtF5erA@mail.gmail.com>

On Sat, Feb 22, 2020 at 3:57 PM Shahin Khan <shahin203 at hotmail.com> wrote:
>
> Hello,
> I am investigating my time install zeek on Ubuntu desktop but I could not successfully installed. Anybody know step by step instruction
> install Zeek with picture on Ubuntu? Would you send me instruction how to install zeek easy way? Thanks
>
> Best,
> Shahin
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

What trouble did you have installing it?  Did you receive any error
messages?  How were you trying to install it?  Were you using a
precompiled binary or were you trying to build zeek yourself?

Ubuntu has a bro meta package but it may be a bit out-dated (looks
like it might be 2.5.5).  You can install that with `sudo apt install
bro`.  Building from source is pretty straightforward:

- Grab a release here:  https://www.zeek.org/download/index.html
- Follow the installation instructions here:
https://docs.zeek.org/en/current/install/install.html
  - Basically extract the archive, install some dependencies with apt,
then do the usual "./configure && make && make install" dance

Hope that helps.

cmh

From Joseph.Fischetti at marist.edu  Mon Feb 24 05:35:19 2020
From: Joseph.Fischetti at marist.edu (Joseph Fischetti)
Date: Mon, 24 Feb 2020 13:35:19 +0000
Subject: [Zeek] Dropping packets
In-Reply-To: <CAPfnCuj598V_bzsEq7V1Z4z_wA2tHSNxRJzMDD1gi8y70bMxXw@mail.gmail.com>
References: <BN8PR20MB2532BBAADDFF169CAD68A740EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuiNCi+PpSGgPynbOcSYYFq0N1RNw-mZS1CYP-YQpwwBOw@mail.gmail.com>
	<BN8PR20MB2532DC39DCD467639251A761EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuhmr7NEb4+CVDxkYJgHrg0oTVEH4hixuE7sbtj05Y16AA@mail.gmail.com>
	<BN8PR20MB2532B6D79A9986EC52B4A7C3EE110@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCugK8ksejTN=r8AeiEStbQYJjOk4y1dyggsusaAop8nx8g@mail.gmail.com>
	<BN8PR20MB25327DC8E048614BA3E69DFAEE110@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuj598V_bzsEq7V1Z4z_wA2tHSNxRJzMDD1gi8y70bMxXw@mail.gmail.com>
Message-ID: <MN2PR20MB25412A07758796665E70DC15EEEC0@MN2PR20MB2541.namprd20.prod.outlook.com>

I contacted myricom support and the first thing they suggested was turning debug logging on (via the environment variable in the node.cfg) so they could get some info. I did that last Wednesday and we?ve had none of the memory issues that we were experiencing before that.  So at least that part of the situation is in check.

 

Here?s the packet drop stats over the last ~4.5 days:

 

worker-1 dropped=534746334 rx=138087967089 0.39%

worker-2 dropped=340526849 rx=147729484064 0.23%

worker-3 dropped=0 rx=9064403485 0.00%

worker-4 dropped=0 rx=9183660589 0.00%

 

Totals dropped=875273183 rx=304065515227 0.29%

 

Just to refresh.  We?re running unpinned with 10 lb_procs per worker.

 

Should I? give it more processes or try to pin the CPU?s?  Load average on the workers is hovering around 5.5 and since we?re unpinned we?re using all 28 cores (but they?re only around 15-20% load).

 

I?ve read through a good portion on the Berkley 100G doc and I?m wondering if I should start looking at shunting as well.  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200224/b8ca4371/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5561 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200224/b8ca4371/attachment-0001.bin 

From justin at corelight.com  Mon Feb 24 07:55:21 2020
From: justin at corelight.com (Justin Azoff)
Date: Mon, 24 Feb 2020 10:55:21 -0500
Subject: [Zeek] Dropping packets
In-Reply-To: <MN2PR20MB25412A07758796665E70DC15EEEC0@MN2PR20MB2541.namprd20.prod.outlook.com>
References: <BN8PR20MB2532BBAADDFF169CAD68A740EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuiNCi+PpSGgPynbOcSYYFq0N1RNw-mZS1CYP-YQpwwBOw@mail.gmail.com>
	<BN8PR20MB2532DC39DCD467639251A761EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuhmr7NEb4+CVDxkYJgHrg0oTVEH4hixuE7sbtj05Y16AA@mail.gmail.com>
	<BN8PR20MB2532B6D79A9986EC52B4A7C3EE110@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCugK8ksejTN=r8AeiEStbQYJjOk4y1dyggsusaAop8nx8g@mail.gmail.com>
	<BN8PR20MB25327DC8E048614BA3E69DFAEE110@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuj598V_bzsEq7V1Z4z_wA2tHSNxRJzMDD1gi8y70bMxXw@mail.gmail.com>
	<MN2PR20MB25412A07758796665E70DC15EEEC0@MN2PR20MB2541.namprd20.prod.outlook.com>
Message-ID: <CAPfnCuh5E1AU50zct6ssopfpdvXEwF97Taw7VZLoC-JxY0CRww@mail.gmail.com>

On Mon, Feb 24, 2020 at 8:35 AM Joseph Fischetti <
Joseph.Fischetti at marist.edu> wrote:

> I contacted myricom support and the first thing they suggested was turning
> debug logging on (via the environment variable in the node.cfg) so they
> could get some info. I did that last Wednesday and we?ve had none of the
> memory issues that we were experiencing before that.  So at least that part
> of the situation is in check.
>

I wouldn't have expected that to be the cause or the fix.. Did you load
those packages I had suggested?


> Here?s the packet drop stats over the last ~4.5 days:
>
>
>
> worker-1 dropped=534746334 rx=138087967089 0.39%
>
> worker-2 dropped=340526849 rx=147729484064 0.23%
>
> worker-3 dropped=0 rx=9064403485 0.00%
>
> worker-4 dropped=0 rx=9183660589 0.00%
>
>
>
> Totals dropped=875273183 rx=304065515227 0.29%
>
>
>
> Just to refresh.  We?re running unpinned with 10 lb_procs per worker.
>
>
>
> Should I? give it more processes or try to pin the CPU?s?  Load average on
> the workers is hovering around 5.5 and since we?re unpinned we?re using all
> 28 cores (but they?re only around 15-20% load).
>

Load average isn't a terribly useful metric.. what really helps is having
per cpu utilization graphs.
Those drop numbers are already looking a lot better, but the rx packets per
worker is still really skewed and points to something weird with your load
balancing.  The first 2 nics are seeing 15 times the number of packets as
the other 2.


> I?ve read through a good portion on the Berkley 100G doc and I?m wondering
> if I should start looking at shunting as well.
>

If you have enough elephant flows, it couldn't hurt.. but likely want to
fix the load balancing first.
-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200224/e0800c1f/attachment.html 

From justin at corelight.com  Mon Feb 24 08:16:30 2020
From: justin at corelight.com (Justin Azoff)
Date: Mon, 24 Feb 2020 11:16:30 -0500
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
	<CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
	<CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>
Message-ID: <CAPfnCuizcGsP1jhAafqmkRn-Mq=uMNq2s3+jQf8vkkQHeLg3fQ@mail.gmail.com>

Ah, sorry.  The profiling confirms it's spending most of its time in
iosource::Manager::FindSoonest.  As you noticed this code is completely
gone in the next version of zeek, from the NEWS file:

- Replace old ``select``-based IO loop with a new architecture that doesn't
  spin checking for active IO sources. The new architecture now waits for
the
  the sources to actively notify it when activity occurs and only processes
  data once it's ready. This helps heavily reduce the CPU usage on idle
  network connections.

I'm not sure if what you are seeing is a bug, or just an instance of the
older select loop not performing well.  Since you said it seems to be
working fine otherwise and not dropping packets, I'm leaning towards just a
performance issue.

The new IO loop should perform much better, especially on lower utilized
links.


On Sat, Feb 22, 2020 at 10:20 AM Pete Nelson <petiepooo at gmail.com> wrote:

> Perf output mailed to you, Justin.  I'm checking to see if other
> SecurityOnion users are seeing this, but haven't heard anything yet.
> The good news is that it doesn't break things, as it just consumes
> more CPU than usual.
>
> I appreciate you taking a look at this.  Please let me know if you
> need anything else.
> --
> Pete
>


-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200224/a2e0bc8d/attachment.html 

From joseph.fischetti at marist.edu  Mon Feb 24 08:57:38 2020
From: joseph.fischetti at marist.edu (Joseph K Fischetti)
Date: Mon, 24 Feb 2020 11:57:38 -0500
Subject: [Zeek] Dropping packets
In-Reply-To: <CAPfnCuh5E1AU50zct6ssopfpdvXEwF97Taw7VZLoC-JxY0CRww@mail.gmail.com>
References: <BN8PR20MB2532BBAADDFF169CAD68A740EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuiNCi+PpSGgPynbOcSYYFq0N1RNw-mZS1CYP-YQpwwBOw@mail.gmail.com>
	<BN8PR20MB2532DC39DCD467639251A761EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuhmr7NEb4+CVDxkYJgHrg0oTVEH4hixuE7sbtj05Y16AA@mail.gmail.com>
	<BN8PR20MB2532B6D79A9986EC52B4A7C3EE110@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCugK8ksejTN=r8AeiEStbQYJjOk4y1dyggsusaAop8nx8g@mail.gmail.com>
	<BN8PR20MB25327DC8E048614BA3E69DFAEE110@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuj598V_bzsEq7V1Z4z_wA2tHSNxRJzMDD1gi8y70bMxXw@mail.gmail.com>
	<MN2PR20MB25412A07758796665E70DC15EEEC0@MN2PR20MB2541.namprd20.prod.outlook.com>
	<CAPfnCuh5E1AU50zct6ssopfpdvXEwF97Taw7VZLoC-JxY0CRww@mail.gmail.com>
Message-ID: <574c4041-8fbb-de6f-1c8d-33da17072d7d@marist.edu>


>     I contacted myricom support and the first thing they suggested was
>     turning debug logging on (via the environment variable in the
>     node.cfg) so they could get some info. I did that last Wednesday
>     and we?ve had none of the memory issues that we were experiencing
>     before that.? So at least that part of the situation is in check.
>
>
> I wouldn't have expected that to be the cause or the fix.. Did you
> load those packages I had suggested???

I didn't.? I was trying to keep it as close to a vanilla configuration
as possible (especially after looping support in).? I probably will this
week though.

> ?
>
>     Here?s the packet drop stats over the last ~4.5 days:
>
>     ?
>
>     worker-1 dropped=534746334 rx=138087967089 0.39%
>
>     worker-2 dropped=340526849 rx=147729484064 0.23%
>
>     worker-3 dropped=0 rx=9064403485 0.00%
>
>     worker-4 dropped=0 rx=9183660589 0.00%
>
>     ?
>
>     Totals dropped=875273183 rx=304065515227 0.29%
>
>     ?
>
>     Just to refresh.? We?re running unpinned with 10 lb_procs per worker.
>
>     ?
>
>     Should I? give it more processes or try to pin the CPU?s?? Load
>     average on the workers is hovering around 5.5 and since we?re
>     unpinned we?re using all 28 cores (but they?re only around 15-20%
>     load).
>
>
> Load average isn't a terribly useful metric.. what really helps is
> having per cpu utilization graphs.
> Those drop numbers are already looking a lot better, but the rx
> packets per worker is still really skewed and points to something
> weird with your load balancing.? The first 2 nics are seeing 15 times
> the number of packets as the other 2.
> ?

I would think the load average should at least indicate how the
processors are keeping up with the traffic.? That said, I know there's
an imbalance.? I need access to the Arista switches.? I guess it's
possible that if we can balance them properly I can get away without
doing *any* more CPU/pinning work.


>     I?ve read through a good portion on the Berkley 100G doc and I?m
>     wondering if I should start looking at shunting as well.?
>
>
> If you have enough elephant flows, it couldn't hurt.. but likely want
> to fix the load balancing first.

Will do.

Thank you

Joe

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200224/e57ec173/attachment-0001.html 

From Paul.Sibley at canarie.ca  Mon Feb 24 13:55:13 2020
From: Paul.Sibley at canarie.ca (Paul Sibley)
Date: Mon, 24 Feb 2020 21:55:13 +0000
Subject: [Zeek] plugin: af_packet 3.0 not working with zkg
Message-ID: <DM6PR12MB349865B19FF95986D768ED9E8BEC0@DM6PR12MB3498.namprd12.prod.outlook.com>

Hello Community,

Some of the participants in our project are attempting to install the
af_packet plugin via the zeek package manager and are receiving errors since
the master was updated from 1.4 to 2.0 ~4 hours ago.

While we can manually download and continue with version 1.4, I am wondering
if there is something in particular that needs to be considered.

 

OS:  CentOS8
Zeek:  3.0.1
Command: zkg install zeek/j-gras/gro-af_packet-plugin
Error(s): 

*	"Failed to initialize: Bad git executable"
*	"zeek/j-gras/bro-af_packet-plugin" tests failed

 

Thanks in advance,

Paul

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200224/e7e11056/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4909 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200224/e7e11056/attachment.bin 

From justin at corelight.com  Mon Feb 24 14:16:30 2020
From: justin at corelight.com (Justin Azoff)
Date: Mon, 24 Feb 2020 17:16:30 -0500
Subject: [Zeek] plugin: af_packet 3.0 not working with zkg
In-Reply-To: <DM6PR12MB349865B19FF95986D768ED9E8BEC0@DM6PR12MB3498.namprd12.prod.outlook.com>
References: <DM6PR12MB349865B19FF95986D768ED9E8BEC0@DM6PR12MB3498.namprd12.prod.outlook.com>
Message-ID: <CAPfnCuhfU5kZpLnz1JPj5yUmjeQZTsD1nnhw9kwQs+FiFJrhPA@mail.gmail.com>

On Mon, Feb 24, 2020 at 4:58 PM Paul Sibley <Paul.Sibley at canarie.ca> wrote:

> Hello Community,
>
> Some of the participants in our project are attempting to install the
> af_packet plugin via the zeek package manager and are receiving errors
> since the master was updated from 1.4 to 2.0 ~4 hours ago.
>
> While we can manually download and continue with version 1.4, I am
> wondering if there is something in particular that needs to be considered.
>
>
>
> *OS*:  CentOS8
> *Zeek*:  3.0.1
> *Command*: zkg install zeek/j-gras/gro-af_packet-plugin
> *Error(s)*:
>
>    - ?Failed to initialize: Bad git executable?
>    - ?zeek/j-gras/bro-af_packet-plugin" tests failed
>
> Looks like that error is from not having git installed.

2.0 may need zeek from master anyway.. but in any case you don't need to
manually install it, you can do

zkg install j-gras/bro-af_packet-plugin --version 1.4.0

-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200224/c0273622/attachment.html 

From ito.satosi at jp.panasonic.com  Mon Feb 24 16:33:16 2020
From: ito.satosi at jp.panasonic.com (ito.satosi at jp.panasonic.com)
Date: Tue, 25 Feb 2020 00:33:16 +0000
Subject: [Zeek] uninstalling of bro(version2.6.1)
In-Reply-To: <CAMzgZ0KwwBor8fE_Q94GCvZF7p9+FauC8BHV52j_41WpoVQ3=w@mail.gmail.com>
References: <0f0d40d5d5c045a1a394d317f03c3be1@JPA000SECMN13.palet.jp.panasonic.com>
	<CAMzgZ0KwwBor8fE_Q94GCvZF7p9+FauC8BHV52j_41WpoVQ3=w@mail.gmail.com>
Message-ID: <9da51e842f9b47cbb1aa9fe5cf9468e2@JPA000SECMN13.palet.jp.panasonic.com>

Awesome, thanks for the information!

> -----Original Message-----
> From: Jon Siwek <jsiwek at corelight.com>
> Sent: Wednesday, February 19, 2020 3:35 AM
> To: Ito Satoshi (?? ??) <ito.satosi at jp.panasonic.com>
> Cc: zeek <zeek at zeek.org>
> Subject: Re: [Zeek] uninstalling of bro(version2.6.1)
> 
> Yeah, that's basically the uninstall process:
> 
> * broctl stop
> * if you made any customizations to /usr/local/bro/etc/* or
> /usr/local/bro/share/bro/site/local.bro, you may want to save those to port to Zeek
> 3.0.x installation
> * by default, all files live under /usr/local/bro/ prefix so deleting that should be
> everything
> 
> - Jon
> 
> On Mon, Feb 17, 2020 at 5:42 PM <ito.satosi at jp.panasonic.com> wrote:
> >
> > Hi
> >
> > I am deploying after bro / 2.6.6.1 / configure && make && sudo make install.
> > I am trying to change from bro(version:2.6.1) to Zeek (version:3.0.0) under
> ubuntu environment.
> > For that I want to uninstall bro2.6.1, can you please tell me the procedure.
> > For example, is it the procedure to stop with broctl and then delete the folder of
> /usr/local/bro?
> >
> > Best Regards,
> > Satoshi Ito
> >
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From jan.grashoefer at gmail.com  Tue Feb 25 02:09:01 2020
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Tue, 25 Feb 2020 11:09:01 +0100
Subject: [Zeek] plugin: af_packet 3.0 not working with zkg
In-Reply-To: <CAPfnCuhfU5kZpLnz1JPj5yUmjeQZTsD1nnhw9kwQs+FiFJrhPA@mail.gmail.com>
References: <DM6PR12MB349865B19FF95986D768ED9E8BEC0@DM6PR12MB3498.namprd12.prod.outlook.com>
	<CAPfnCuhfU5kZpLnz1JPj5yUmjeQZTsD1nnhw9kwQs+FiFJrhPA@mail.gmail.com>
Message-ID: <a893e0df-ba86-5233-7ab9-cd99f2bc8c6b@gmail.com>

The new version of the plugin builds with zeek 3.0. The latest master is 
not required.

Once the package source is updated, the plugin is also available under 
zeek-af_packet-plugin.

Jan

On 24/02/2020 23:16, Justin Azoff wrote:
> On Mon, Feb 24, 2020 at 4:58 PM Paul Sibley <Paul.Sibley at canarie.ca 
> <mailto:Paul.Sibley at canarie.ca>> wrote:
> 
>     Hello Community,
> 
>     Some of the participants in our project are attempting to install
>     the af_packet plugin via the zeek package manager and are receiving
>     errors since the master was updated from 1.4 to 2.0 ~4 hours ago.
> 
>     While we can manually download and continue with version 1.4, I am
>     wondering if there is something in particular that needs to be
>     considered.____
> 
>     __ __
> 
>     _OS_:? CentOS8
>     _Zeek_:? 3.0.1
>     _Command_: zkg install zeek/j-gras/gro-af_packet-plugin
>     _Error(s)_: ____
> 
>       * ?Failed to initialize: Bad git executable?____
>       * ?zeek/j-gras/bro-af_packet-plugin" tests failed
> 
> Looks like that error is from not having git installed.
> 
> 2.0 may need zeek from master anyway.. but in any case you don't need to 
> manually install it, you can do
> 
> zkg install j-gras/bro-af_packet-plugin --version 1.4.0
> 
> -- 
> Justin
> 
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> 

From Joseph.Fischetti at marist.edu  Tue Feb 25 05:48:09 2020
From: Joseph.Fischetti at marist.edu (Joseph Fischetti)
Date: Tue, 25 Feb 2020 13:48:09 +0000
Subject: [Zeek] Dropping packets
In-Reply-To: <574c4041-8fbb-de6f-1c8d-33da17072d7d@marist.edu>
References: <BN8PR20MB2532BBAADDFF169CAD68A740EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuiNCi+PpSGgPynbOcSYYFq0N1RNw-mZS1CYP-YQpwwBOw@mail.gmail.com>
	<BN8PR20MB2532DC39DCD467639251A761EE160@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuhmr7NEb4+CVDxkYJgHrg0oTVEH4hixuE7sbtj05Y16AA@mail.gmail.com>
	<BN8PR20MB2532B6D79A9986EC52B4A7C3EE110@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCugK8ksejTN=r8AeiEStbQYJjOk4y1dyggsusaAop8nx8g@mail.gmail.com>
	<BN8PR20MB25327DC8E048614BA3E69DFAEE110@BN8PR20MB2532.namprd20.prod.outlook.com>
	<CAPfnCuj598V_bzsEq7V1Z4z_wA2tHSNxRJzMDD1gi8y70bMxXw@mail.gmail.com>
	<MN2PR20MB25412A07758796665E70DC15EEEC0@MN2PR20MB2541.namprd20.prod.outlook.com>
	<CAPfnCuh5E1AU50zct6ssopfpdvXEwF97Taw7VZLoC-JxY0CRww@mail.gmail.com>
	<574c4041-8fbb-de6f-1c8d-33da17072d7d@marist.edu>
Message-ID: <MN2PR20MB254118B3E022876DD882824BEEED0@MN2PR20MB2541.namprd20.prod.outlook.com>

The imbalance in traffic on the interfaces is apparently due to the way the Arista switches are physically connected.  Networking is going to change that:


We have span ports on 2 different links feeding 2 different Arista switches.  One Arista switch goes to one set of interfaces (eth4 on both boxes).  And the other Arista switch goes to the other set of interfaces (eth5 on both boxes).  One link is used as primary so it sees a lot more traffic than the other.  In addition to that, the Arista switch with the heavy link has span ports from the core.

 

We?re going to move the core spans over to the Arista with the light link? that should hopefully work to rebalance things.

 

That said? Will the Arista switches do any dedup on the packets that they?re getting?  I was uninvolved on the design/setup of the system.  It would seem to me that if we have some vlans spanning to the Aristas, and ports from the edge spanning to the Aristas?. Packets that go through the edge and through those vlans in the core will both be spanned to the Arista (and sent to zeek).

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200225/b2c1aefa/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5561 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200225/b2c1aefa/attachment.bin 

From petiepooo at gmail.com  Tue Feb 25 08:39:02 2020
From: petiepooo at gmail.com (Pete Nelson)
Date: Tue, 25 Feb 2020 16:39:02 +0000
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAPfnCuizcGsP1jhAafqmkRn-Mq=uMNq2s3+jQf8vkkQHeLg3fQ@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
	<CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
	<CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>
	<CAPfnCuizcGsP1jhAafqmkRn-Mq=uMNq2s3+jQf8vkkQHeLg3fQ@mail.gmail.com>
Message-ID: <CAAYFXLmwMZUep3o-N02fWhx6ybcgbT6htfbdyRKoNd__fdz2hQ@mail.gmail.com>

Justtin,

We agree that a worker process of Zeek using 10-20% of the CPU when
essentially idle is a performance issue.  Thank you for your work on
that.  I'm looking forward to the day that it's rolled out in
SecurityOnion.

That said...

That a worker process of Zeek is taking 100% of the CPU when
essentially idle, but only sometimes and only after running for an
indeterminate amount of time, is a bug.  It's not just not performing
well, as is demonstrated by the strace output; select() is always
returning with a file descriptor ready for IO, which is not cleared by
reading it in the main thread, and thus the main thread never sleeps.
Its behavior is very different from the baseline with just the
performance issue.

To bolster that conclusion, I was doing my best to understand the
strace output I'd captured earlier, and sifting through the series of
select(), epoll_ctl(), futex(), rt_sigprocmask() and rt_sigtimedwait()
syscalls reminded me just how HARD multi-threaded programming can be.
And race conditions can be subtle with no discernible pattern in when
they manifest.  So I tried something: I tried pinning each worker
process to a CPU.

None of the threads have jumped up to 100% since.

FWIW, I've found another SecurityOnion user that is seeing the same
issue.  He reported that load averages skyrocketed after the latest
upgrade, and saw that 9 out of his 11 Zeek worker processes were
pegged at 100% CPU until he restarted it.  I'm going to suggest he pin
workers to specific CPUs to see if it goes away for him too.

>From strace output, it appears that the main thread and one of the
child threads get out of sync, in that there's an extra character in
the pipe used for intra-thread signalling that the main thread never
clears with a read.  Even when a new packet comes in, I think I see
that there's something else (eg. a signal being unmasked?) that
triggers the main thread to read from the pipe, but it still only
reads what it thinks are the right number of characters, rather than
all that are available in the pipe, leaving extra in the pipe select()
always returns that fd as ready for IO.

This may be specific to SecurityOnion's configuration or set of
ZeekScripts, or it may be happening on all instances where Zeek
workers have no cpu affinity.  Perf wouldn't help there, as we need
the specific sequence that triggers the race condition rather than a
heatmap of function calls.  I'd still like to get a trace that shows
the function call sequence at the moment this bug is triggered, but
haven't had any luck so far.

Does this make sense to you?  Trigger any memories of changes that
went into 3.0 that might have altered behavior in that area?  Do you
run your tests without pinning cpus?

I saw in a ZeekWeek2019 slide that 3.0.x is considered a long-term
stable release.  Please let me know if there's anything I can do to
help diagnose this further and make sure it's stable for everyone.
--
Pete

From bart.hermans at os3.nl  Tue Feb 25 10:17:05 2020
From: bart.hermans at os3.nl (Bart Hermans)
Date: Tue, 25 Feb 2020 19:17:05 +0100
Subject: [Zeek] Development of layer 4 protocol parser (ESP)
Message-ID: <b5f6311c-bbed-c2a5-fbd4-909977e47865@os3.nl>

Recently I got into Zeek and started to play around with BinPAC plugin
development. BinPAC allowed me to pretty easily write a protocol parser
for IKE messages. However, I stumbled upon a problem. As I already read
on the mailing list, BinPAC is aimed at parsing protocols which run on
top of UDP or TCP. I also read that to parse protocols on lower layers
(let's say the transport layer), BinPAC won't be able to help you
anymore. The solution that was proposed in a few messages that I read
was to modify the source code of Zeek to support layer 4 protocols other
than TCP, UDP and ICMP.

First and foremost; before posting this message, that's exactly what I
did. My approach was to look at the implementation of ICMP and UDP in
Zeek (which are also layer 4 protocols). Based on this I tried my best
at writing a protocol analyzer alongside these protocols. However, after
spending a good amount of hours trying to write a protocol parser for
ESP-messages (protocol number 50) I came to the conclusion that the code
had become quite messy. Most importantly I didn't get the ESP-parser to
work properly. Even if I would have got it working, the code wouldn't be
patch safe anymore from future versions of Zeek.

My issue is as follows; I only want to be able to detect that a protocol
number 50 packet has been seen with the parsing of the very first field.
Is the only way to get this working to give another shot at modifying
the source code or is there a more cleaner/patch friendly path to
travel? Even a gentle push in the right direction would very much be
appreciated.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200225/0f034bce/attachment.bin 

From jsiwek at corelight.com  Tue Feb 25 10:45:08 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Tue, 25 Feb 2020 10:45:08 -0800
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAAYFXLmwMZUep3o-N02fWhx6ybcgbT6htfbdyRKoNd__fdz2hQ@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
	<CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
	<CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>
	<CAPfnCuizcGsP1jhAafqmkRn-Mq=uMNq2s3+jQf8vkkQHeLg3fQ@mail.gmail.com>
	<CAAYFXLmwMZUep3o-N02fWhx6ybcgbT6htfbdyRKoNd__fdz2hQ@mail.gmail.com>
Message-ID: <CAMzgZ0KoVGu2KmA7u06B9FmbXbhA2h0toxgNDUVGHbz-yc1UCQ@mail.gmail.com>

On Tue, Feb 25, 2020 at 8:41 AM Pete Nelson <petiepooo at gmail.com> wrote:

> Does this make sense to you?  Trigger any memories of changes that
> went into 3.0 that might have altered behavior in that area?

These are some differences between 2.6 and 3.0 that come to mind as
possibly related:

* https://github.com/zeek/zeek/issues/346 and
https://github.com/zeek/zeek/pull/358
* https://github.com/zeek/broker/commit/0abe5277

Your descriptions makes me generally suspect the Broker flare_actor
code, either it being introduced in that later commit or just the
patch that was attempted to fix similar behavior to what you notice is
incomplete (and possibly wasn't an issue in 2.6 because underlying CAF
library was just different there, too).  And if there's still a bug
there, then I'd expect it would also break the 3.1.0 IO loop despite
it generally being re-written.

If you have more ideas to narrow it down or can pinpoint which pipe
isn't getting fully drained, that will be helpful.

- Jon

From jsiwek at corelight.com  Tue Feb 25 14:07:42 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Tue, 25 Feb 2020 14:07:42 -0800
Subject: [Zeek] Zeek 3.0.2 (security update) and 3.1.0 (new features)
	released
Message-ID: <CAMzgZ0J9VhwPVT=rYFEtsDrD5Q3LW8a4KXC_jcVfQkp+u0Otyg@mail.gmail.com>

Downloads for either 3.0.2 or 3.1.0 are now available:

    https://www.zeek.org/download/index.html

Zeek 3.0.2 addresses several important security issues and bug fixes
as part of the long-term supported 3.0.x branch.  Release notes:

    https://github.com/zeek/zeek/releases/tag/v3.0.2

Zeek 3.1.0 contains the same bug fixes as were addressed in 3.0.2, but
also has several new features and enhancements.  This release also
further advances the Bro-to-Zeek renaming process by removing some
backwards-compatibility mechanisms and deprecations.  Release notes:

    https://github.com/zeek/zeek/releases/tag/v3.1.0

This blog post also has further information regarding the releases:

    https://blog.zeek.org/2020/02/zeek-31-released.html

From petiepooo at gmail.com  Tue Feb 25 15:38:40 2020
From: petiepooo at gmail.com (Pete Nelson)
Date: Tue, 25 Feb 2020 23:38:40 +0000
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAMzgZ0KoVGu2KmA7u06B9FmbXbhA2h0toxgNDUVGHbz-yc1UCQ@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
	<CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
	<CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>
	<CAPfnCuizcGsP1jhAafqmkRn-Mq=uMNq2s3+jQf8vkkQHeLg3fQ@mail.gmail.com>
	<CAAYFXLmwMZUep3o-N02fWhx6ybcgbT6htfbdyRKoNd__fdz2hQ@mail.gmail.com>
	<CAMzgZ0KoVGu2KmA7u06B9FmbXbhA2h0toxgNDUVGHbz-yc1UCQ@mail.gmail.com>
Message-ID: <CAAYFXL=bZYmFCiRKHswDY4xBJ5en-kBkWoOoc8Pn4kn=XdLpfQ@mail.gmail.com>

Thanks, Jon.

I'll try to digest those links and dig into the code.  Unfortunately,
it seems running strace on the process keeps it from occurring...  I
may try to get dtrace working in place, but I need to improve my lab
setup first before I go too crazy.
--
Pete

From jan.grashoefer at gmail.com  Wed Feb 26 04:09:35 2020
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Wed, 26 Feb 2020 13:09:35 +0100
Subject: [Zeek] Development of layer 4 protocol parser (ESP)
In-Reply-To: <b5f6311c-bbed-c2a5-fbd4-909977e47865@os3.nl>
References: <b5f6311c-bbed-c2a5-fbd4-909977e47865@os3.nl>
Message-ID: <36926fb5-2198-251a-2bbf-1506ef8779c5@gmail.com>

Hi Bart,

Regarding patch safety, support for pluggable low-lever analyzers would 
help. This is actually a long-standing request: 
https://github.com/zeek/zeek/issues/248 There is a first approach that 
needs some more improvements and reviews. We are working on it.

Jan

On 25/02/2020 19:17, Bart Hermans wrote:
> Recently I got into Zeek and started to play around with BinPAC plugin
> development. BinPAC allowed me to pretty easily write a protocol parser
> for IKE messages. However, I stumbled upon a problem. As I already read
> on the mailing list, BinPAC is aimed at parsing protocols which run on
> top of UDP or TCP. I also read that to parse protocols on lower layers
> (let's say the transport layer), BinPAC won't be able to help you
> anymore. The solution that was proposed in a few messages that I read
> was to modify the source code of Zeek to support layer 4 protocols other
> than TCP, UDP and ICMP.
> 
> First and foremost; before posting this message, that's exactly what I
> did. My approach was to look at the implementation of ICMP and UDP in
> Zeek (which are also layer 4 protocols). Based on this I tried my best
> at writing a protocol analyzer alongside these protocols. However, after
> spending a good amount of hours trying to write a protocol parser for
> ESP-messages (protocol number 50) I came to the conclusion that the code
> had become quite messy. Most importantly I didn't get the ESP-parser to
> work properly. Even if I would have got it working, the code wouldn't be
> patch safe anymore from future versions of Zeek.
> 
> My issue is as follows; I only want to be able to detect that a protocol
> number 50 packet has been seen with the parsing of the very first field.
> Is the only way to get this working to give another shot at modifying
> the source code or is there a more cleaner/patch friendly path to
> travel? Even a gentle push in the right direction would very much be
> appreciated.
> 
> 
> 
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> 

From vlad at es.net  Wed Feb 26 07:58:34 2020
From: vlad at es.net (Vlad Grigorescu)
Date: Wed, 26 Feb 2020 07:58:34 -0800
Subject: [Zeek] Development of layer 4 protocol parser (ESP)
In-Reply-To: <36926fb5-2198-251a-2bbf-1506ef8779c5@gmail.com>
References: <b5f6311c-bbed-c2a5-fbd4-909977e47865@os3.nl>
	<36926fb5-2198-251a-2bbf-1506ef8779c5@gmail.com>
Message-ID: <CAPqbkwu-0-AxNJ_moV0wcKDbMmw1p7R86ZS_GvQVi=zDRPeEHw@mail.gmail.com>

Jan,

Is that branch publicly available somewhere? Thanks!

  ?Vlad

On Wed, Feb 26, 2020 at 04:18 Jan Grash?fer <jan.grashoefer at gmail.com>
wrote:

> Hi Bart,
>
> Regarding patch safety, support for pluggable low-lever analyzers would
> help. This is actually a long-standing request:
> https://github.com/zeek/zeek/issues/248 There is a first approach that
> needs some more improvements and reviews. We are working on it.
>
> Jan
>
> On 25/02/2020 19:17, Bart Hermans wrote:
> > Recently I got into Zeek and started to play around with BinPAC plugin
> > development. BinPAC allowed me to pretty easily write a protocol parser
> > for IKE messages. However, I stumbled upon a problem. As I already read
> > on the mailing list, BinPAC is aimed at parsing protocols which run on
> > top of UDP or TCP. I also read that to parse protocols on lower layers
> > (let's say the transport layer), BinPAC won't be able to help you
> > anymore. The solution that was proposed in a few messages that I read
> > was to modify the source code of Zeek to support layer 4 protocols other
> > than TCP, UDP and ICMP.
> >
> > First and foremost; before posting this message, that's exactly what I
> > did. My approach was to look at the implementation of ICMP and UDP in
> > Zeek (which are also layer 4 protocols). Based on this I tried my best
> > at writing a protocol analyzer alongside these protocols. However, after
> > spending a good amount of hours trying to write a protocol parser for
> > ESP-messages (protocol number 50) I came to the conclusion that the code
> > had become quite messy. Most importantly I didn't get the ESP-parser to
> > work properly. Even if I would have got it working, the code wouldn't be
> > patch safe anymore from future versions of Zeek.
> >
> > My issue is as follows; I only want to be able to detect that a protocol
> > number 50 packet has been seen with the parsing of the very first field.
> > Is the only way to get this working to give another shot at modifying
> > the source code or is there a more cleaner/patch friendly path to
> > travel? Even a gentle push in the right direction would very much be
> > appreciated.
> >
> >
> >
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> >
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200226/2cc3713c/attachment.html 

From akgraner at corelight.com  Wed Feb 26 13:46:54 2020
From: akgraner at corelight.com (Amber Graner)
Date: Wed, 26 Feb 2020 16:46:54 -0500
Subject: [Zeek] ASK THE ZEEKSPERTS with Seth Hall - 27 February 2020
Message-ID: <CAJhOzur4ZH=eHw5K+daz9h-reV7YOdGqEn20tzcd3mYMt6ENaw@mail.gmail.com>

Hi all,

Join us tomorrow for another ASK THE ZEEKSPERTS webinar scheduled for  27
February 2020 at 12:30pm PST/3:30pm EST.

This is a bi-monthly call for the open-source Zeek (formerly Bro) community
to interface directly with leading contributors to the open-source project
and ask questions live to better understand, expand or troubleshoot
deployments of the network security monitoring software.

FEATURED SPEAKER for Thursday February 27th - Seth Hall (
https://www.linkedin.com/in/sethh/)

You can register for the webinar at:
https://attendee.gotowebinar.com/register/3466243796592822796

Thanks,
~Amber
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200226/1b8f0c4b/attachment.html 

From clopmz at outlook.com  Thu Feb 27 00:48:24 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Thu, 27 Feb 2020 08:48:24 +0000
Subject: [Zeek] No emails are received
Message-ID: <DB8PR10MB35954CED6EC1568C0B8F5611DBEB0@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>

Hi all,

 After re-installing my Zeek hosts to version 3.0.2 in my home lab, I haven't received any mail from cron task or any process/alert related to Zeek. But I see some emails queued in /var/zeek/spool/tmp directory like this:

-rw-r--r--. 1 zeek idps 296 Feb 27 07:30 mail.1493.tmp

 With the following content:

From: admin.zeek at domain.org
Subject: [Zeek] cron: expire-logs failed
To: myadmin at otherdomain.org
User-Agent: ZeekControl 2.0.0

expire-logs failed
expire-logs: directory not found: /var/zeek/logs/stats

creating directory for stats file: /var/zeek/logs/stats

--
[Automatically generated.]

 In /var/log/maillog, there is an error when Zeek tries to send any email:

Feb 26 09:20:07 aberdeen postfix/sendmail[21353]: fatal: zeek(994): No recipient addresses found in message header
Feb 26 13:10:07 aberdeen postfix/sendmail[27852]: fatal: zeek(994): No recipient addresses found in message header
Feb 26 13:20:08 aberdeen postfix/sendmail[27999]: fatal: zeek(994): No recipient addresses found in message header
Feb 26 16:40:08 aberdeen postfix/sendmail[718]: fatal: zeek(994): No recipient addresses found in message header
Feb 26 17:00:08 aberdeen postfix/sendmail[1529]: fatal: zeek(994): No recipient addresses found in message header
Feb 26 17:30:07 aberdeen postfix/sendmail[1968]: fatal: zeek(994): No recipient addresses found in message header
Feb 26 17:50:07 aberdeen postfix/sendmail[2261]: fatal: zeek(994): No recipient addresses found in message header

 Options MailTo = and  MailFrom = contain the values indicated in the mail I have shown.

 Is this a bug? In previous versions I didn't have this problem.

Regards,
C. L. Martinez


From bill.de.ping at gmail.com  Thu Feb 27 03:49:19 2020
From: bill.de.ping at gmail.com (william de ping)
Date: Thu, 27 Feb 2020 13:49:19 +0200
Subject: [Zeek] Recording the webinar today
Message-ID: <CAKZA8xjSK5mjMmqRCeOx1W1go5ycULqNs_A-V=E4fGM7cWoXgg@mail.gmail.com>

Hi all,

Anyone want to volunteer to record and publish the webinar today ?

Thanks
B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200227/26114844/attachment.html 

From akgraner at corelight.com  Thu Feb 27 03:55:26 2020
From: akgraner at corelight.com (Amber Graner)
Date: Thu, 27 Feb 2020 06:55:26 -0500
Subject: [Zeek] Recording the webinar today
In-Reply-To: <CAKZA8xjSK5mjMmqRCeOx1W1go5ycULqNs_A-V=E4fGM7cWoXgg@mail.gmail.com>
References: <CAKZA8xjSK5mjMmqRCeOx1W1go5ycULqNs_A-V=E4fGM7cWoXgg@mail.gmail.com>
Message-ID: <CAJhOzuqS4P29cvC0iso1Fz9EavcJpy8aaueaWNQWQoOqEYcL5g@mail.gmail.com>

We purposely don?t record them to encourage people to ask questions freely.

On Thu, Feb 27, 2020 at 6:52 AM william de ping <bill.de.ping at gmail.com>
wrote:

> Hi all,
>
> Anyone want to volunteer to record and publish the webinar today ?
>
> Thanks
> B
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

-- 

*Amber Graner*
Director of Community
Corelight, Inc

828.582.9469

Schedule time on my calendar here. <https://calendly.com/amber_graner>



 * Ask me about how you can participate in the Zeek (formerly Bro)
community.
 * Remember - ZEEK AND YOU SHALL FIND!!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200227/0de24641/attachment.html 

From akgraner at corelight.com  Thu Feb 27 04:02:20 2020
From: akgraner at corelight.com (Amber Graner)
Date: Thu, 27 Feb 2020 07:02:20 -0500
Subject: [Zeek] Recording the webinar today
In-Reply-To: <CAJhOzuqS4P29cvC0iso1Fz9EavcJpy8aaueaWNQWQoOqEYcL5g@mail.gmail.com>
References: <CAKZA8xjSK5mjMmqRCeOx1W1go5ycULqNs_A-V=E4fGM7cWoXgg@mail.gmail.com>
	<CAJhOzuqS4P29cvC0iso1Fz9EavcJpy8aaueaWNQWQoOqEYcL5g@mail.gmail.com>
Message-ID: <CAJhOzuraS5J32CBBNG_f-VXiUMRKJh-4=9hychWi24rN6tYp=w@mail.gmail.com>

Bill,

Since we don?t record the Ask The Zeeksperts webinars and those joining
have the expectation of not being recorded, please feel free to post your
questions on the mailing list if you?re unable to attend the webinar.

Seth and other Zeeksperts are on the list and will see your questions here
as well.

Thanks,
~Amber

On Thu, Feb 27, 2020 at 6:55 AM Amber Graner <akgraner at corelight.com> wrote:

> We purposely don?t record them to encourage people to ask questions
> freely.
>
> On Thu, Feb 27, 2020 at 6:52 AM william de ping <bill.de.ping at gmail.com>
> wrote:
>
>> Hi all,
>>
>> Anyone want to volunteer to record and publish the webinar today ?
>>
>> Thanks
>> B
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
> --
>
> *Amber Graner*
> Director of Community
> Corelight, Inc
>
> 828.582.9469
>
> Schedule time on my calendar here. <https://calendly.com/amber_graner>
>
>
>
>  * Ask me about how you can participate in the Zeek (formerly Bro)
> community.
>  * Remember - ZEEK AND YOU SHALL FIND!!
>
>
> --

*Amber Graner*
Director of Community
Corelight, Inc

828.582.9469

Schedule time on my calendar here. <https://calendly.com/amber_graner>



 * Ask me about how you can participate in the Zeek (formerly Bro)
community.
 * Remember - ZEEK AND YOU SHALL FIND!!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200227/99e2af56/attachment-0001.html 

From karel.kuchar at dardas.cz  Thu Feb 27 09:24:45 2020
From: karel.kuchar at dardas.cz (=?iso-8859-2?Q?Karel_Kucha=F8?=)
Date: Thu, 27 Feb 2020 17:24:45 +0000
Subject: [Zeek] 802.11 frames
Message-ID: <VI1PR02MB43502F246806DA79DD3D7E5295EB0@VI1PR02MB4350.eurprd02.prod.outlook.com>

Dear Zeek Community,

I'm new to zeek but now I'm working on project and I need to solve problem with anomaly detection on Wi-Fi. Is there any possibility how to detect frames specific for 802.11 like EAPOL frame?

Thanks in advance,

Karel K.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200227/e63bf678/attachment.html 

From jan.grashoefer at gmail.com  Thu Feb 27 09:55:06 2020
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Thu, 27 Feb 2020 18:55:06 +0100
Subject: [Zeek] Development of layer 4 protocol parser (ESP)
In-Reply-To: <CAPqbkwu-0-AxNJ_moV0wcKDbMmw1p7R86ZS_GvQVi=zDRPeEHw@mail.gmail.com>
References: <b5f6311c-bbed-c2a5-fbd4-909977e47865@os3.nl>
	<36926fb5-2198-251a-2bbf-1506ef8779c5@gmail.com>
	<CAPqbkwu-0-AxNJ_moV0wcKDbMmw1p7R86ZS_GvQVi=zDRPeEHw@mail.gmail.com>
Message-ID: <b6cba486-d002-0a96-c8d0-c0749b566e53@gmail.com>

Hi Vlad,

as the code was written in context of a thesis, we were not able to 
publish it yet. I'll keep you posted.

Jan

On 26/02/2020 16:58, Vlad Grigorescu wrote:
> Jan,
> 
> Is that branch publicly available somewhere? Thanks!
> 
>  ? ?Vlad
> 
> On Wed, Feb 26, 2020 at 04:18 Jan Grash?fer <jan.grashoefer at gmail.com 
> <mailto:jan.grashoefer at gmail.com>> wrote:
> 
>     Hi Bart,
> 
>     Regarding patch safety, support for pluggable low-lever analyzers would
>     help. This is actually a long-standing request:
>     https://github.com/zeek/zeek/issues/248 There is a first approach that
>     needs some more improvements and reviews. We are working on it.
> 
>     Jan
> 
>     On 25/02/2020 19:17, Bart Hermans wrote:
>      > Recently I got into Zeek and started to play around with BinPAC
>     plugin
>      > development. BinPAC allowed me to pretty easily write a protocol
>     parser
>      > for IKE messages. However, I stumbled upon a problem. As I
>     already read
>      > on the mailing list, BinPAC is aimed at parsing protocols which
>     run on
>      > top of UDP or TCP. I also read that to parse protocols on lower
>     layers
>      > (let's say the transport layer), BinPAC won't be able to help you
>      > anymore. The solution that was proposed in a few messages that I read
>      > was to modify the source code of Zeek to support layer 4
>     protocols other
>      > than TCP, UDP and ICMP.
>      >
>      > First and foremost; before posting this message, that's exactly
>     what I
>      > did. My approach was to look at the implementation of ICMP and UDP in
>      > Zeek (which are also layer 4 protocols). Based on this I tried my
>     best
>      > at writing a protocol analyzer alongside these protocols.
>     However, after
>      > spending a good amount of hours trying to write a protocol parser for
>      > ESP-messages (protocol number 50) I came to the conclusion that
>     the code
>      > had become quite messy. Most importantly I didn't get the
>     ESP-parser to
>      > work properly. Even if I would have got it working, the code
>     wouldn't be
>      > patch safe anymore from future versions of Zeek.
>      >
>      > My issue is as follows; I only want to be able to detect that a
>     protocol
>      > number 50 packet has been seen with the parsing of the very first
>     field.
>      > Is the only way to get this working to give another shot at modifying
>      > the source code or is there a more cleaner/patch friendly path to
>      > travel? Even a gentle push in the right direction would very much be
>      > appreciated.
>      >
>      >
>      >
>      > _______________________________________________
>      > Zeek mailing list
>      > zeek at zeek.org <mailto:zeek at zeek.org>
>      > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>      >
>     _______________________________________________
>     Zeek mailing list
>     zeek at zeek.org <mailto:zeek at zeek.org>
>     http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> 

From clopmz at outlook.com  Fri Feb 28 05:00:59 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Fri, 28 Feb 2020 13:00:59 +0000
Subject: [Zeek] No emails are received
In-Reply-To: <DB8PR10MB35954CED6EC1568C0B8F5611DBEB0@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>
References: <DB8PR10MB35954CED6EC1568C0B8F5611DBEB0@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>
Message-ID: <DB8PR10MB3595771BB88645C8AA85CD7FDBE80@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>

Hi all,

 Is it a bug or a misconfigruation?

Regards,
C. L. Martinez


________________________________________
From: Carlos Lopez <clopmz at outlook.com>
Sent: 27 February 2020 09:48
To: zeek at zeek.org
Subject: No emails are received

Hi all,

 After re-installing my Zeek hosts to version 3.0.2 in my home lab, I haven't received any mail from cron task or any process/alert related to Zeek. But I see some emails queued in /var/zeek/spool/tmp directory like this:

-rw-r--r--. 1 zeek idps 296 Feb 27 07:30 mail.1493.tmp

 With the following content:

From: admin.zeek at domain.org
Subject: [Zeek] cron: expire-logs failed
To: myadmin at otherdomain.org
User-Agent: ZeekControl 2.0.0

expire-logs failed
expire-logs: directory not found: /var/zeek/logs/stats

creating directory for stats file: /var/zeek/logs/stats

--
[Automatically generated.]

 In /var/log/maillog, there is an error when Zeek tries to send any email:

Feb 26 09:20:07 aberdeen postfix/sendmail[21353]: fatal: zeek(994): No recipient addresses found in message header
Feb 26 13:10:07 aberdeen postfix/sendmail[27852]: fatal: zeek(994): No recipient addresses found in message header
Feb 26 13:20:08 aberdeen postfix/sendmail[27999]: fatal: zeek(994): No recipient addresses found in message header
Feb 26 16:40:08 aberdeen postfix/sendmail[718]: fatal: zeek(994): No recipient addresses found in message header
Feb 26 17:00:08 aberdeen postfix/sendmail[1529]: fatal: zeek(994): No recipient addresses found in message header
Feb 26 17:30:07 aberdeen postfix/sendmail[1968]: fatal: zeek(994): No recipient addresses found in message header
Feb 26 17:50:07 aberdeen postfix/sendmail[2261]: fatal: zeek(994): No recipient addresses found in message header

 Options MailTo = and  MailFrom = contain the values indicated in the mail I have shown.

 Is this a bug? In previous versions I didn't have this problem.

Regards,
C. L. Martinez


From justin at corelight.com  Fri Feb 28 08:53:52 2020
From: justin at corelight.com (Justin Azoff)
Date: Fri, 28 Feb 2020 11:53:52 -0500
Subject: [Zeek] No emails are received
In-Reply-To: <DB8PR10MB35954CED6EC1568C0B8F5611DBEB0@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>
References: <DB8PR10MB35954CED6EC1568C0B8F5611DBEB0@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>
Message-ID: <CAPfnCuiNmTrqBu6+TEL41bZyAYnkRk0Os5c5bcCvOUvZL51T6A@mail.gmail.com>

On Thu, Feb 27, 2020 at 3:57 AM Carlos Lopez <clopmz at outlook.com> wrote:

> Hi all,
>
>  After re-installing my Zeek hosts to version 3.0.2 in my home lab, I
> haven't received any mail from cron task or any process/alert related to
> Zeek. But I see some emails queued in /var/zeek/spool/tmp directory like
> this:
>
> -rw-r--r--. 1 zeek idps 296 Feb 27 07:30 mail.1493.tmp
>
>  With the following content:
>
> From: admin.zeek at domain.org
> Subject: [Zeek] cron: expire-logs failed
> To: myadmin at otherdomain.org
> User-Agent: ZeekControl 2.0.0
>
> expire-logs failed
> expire-logs: directory not found: /var/zeek/logs/stats
>
> creating directory for stats file: /var/zeek/logs/stats
>
> --
> [Automatically generated.]
>

what output if any do you get if you run

    sendmail -t -oi /var/zeek/spool/tmp/mail.1493.tmp

or whatever filename exists there.
the "To:" line in there is what it looks for, so that should be working..

-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200228/4e42193a/attachment.html 

From marlitt at ucalgary.ca  Fri Feb 28 15:33:38 2020
From: marlitt at ucalgary.ca (Martin Arlitt)
Date: Fri, 28 Feb 2020 23:33:38 +0000
Subject: [Zeek] 802.11 frames
In-Reply-To: <VI1PR02MB43502F246806DA79DD3D7E5295EB0@VI1PR02MB4350.eurprd02.prod.outlook.com>
References: <VI1PR02MB43502F246806DA79DD3D7E5295EB0@VI1PR02MB4350.eurprd02.prod.outlook.com>
Message-ID: <QB1PR01MB3780EC5F7961C680A79CC4ECBBE80@QB1PR01MB3780.CANPRD01.PROD.OUTLOOK.COM>

hi Karel

The ethertype in an EAPOL frame should be 0x888e (https://www.vocal.com/secure-communication/eapol-extensible-authentication-protocol-over-lan/).

In a pcap file it would be possible to distinguish EAPOL frames from other frames.

I'm not sure if zeek will process EAPOL frames (however, I'm not an expert on this matter). In the past I had to modify the source code in order to process frames that weren't IPv4, IPv6 or ARP ethertypes.

Martin

________________________________
From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> on behalf of Karel Kucha? <karel.kuchar at dardas.cz>
Sent: Thursday, February 27, 2020 10:24 AM
To: zeek at zeek.org <zeek at zeek.org>
Subject: [Zeek] 802.11 frames

Dear Zeek Community,

I'm new to zeek but now I'm working on project and I need to solve problem with anomaly detection on Wi-Fi. Is there any possibility how to detect frames specific for 802.11 like EAPOL frame?

Thanks in advance,

Karel K.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200228/c633c78f/attachment.html 

From clopmz at outlook.com  Sat Feb 29 03:21:51 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Sat, 29 Feb 2020 11:21:51 +0000
Subject: [Zeek] No emails are received
In-Reply-To: <CAPfnCuiNmTrqBu6+TEL41bZyAYnkRk0Os5c5bcCvOUvZL51T6A@mail.gmail.com>
References: <DB8PR10MB35954CED6EC1568C0B8F5611DBEB0@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>
	<CAPfnCuiNmTrqBu6+TEL41bZyAYnkRk0Os5c5bcCvOUvZL51T6A@mail.gmail.com>
Message-ID: <16A6508C-5089-4C8E-B416-D2419C760379@outlook.com>

Thanks for your answer Justin. All my zeek hosts are installed with postfix (they are under RHEL 8.1). I am searching about an equivalent command with postfix but I can't find any?.

--
Regards,
C. L. Martinez

From: Justin Azoff <justin at corelight.com>
Date: Friday, 28 February 2020 at 17:54
To: Carlos Lopez <clopmz at outlook.com>
Cc: "zeek at zeek.org" <zeek at zeek.org>
Subject: Re: [Zeek] No emails are received

On Thu, Feb 27, 2020 at 3:57 AM Carlos Lopez <clopmz at outlook.com<mailto:clopmz at outlook.com>> wrote:
Hi all,

 After re-installing my Zeek hosts to version 3.0.2 in my home lab, I haven't received any mail from cron task or any process/alert related to Zeek. But I see some emails queued in /var/zeek/spool/tmp directory like this:

-rw-r--r--. 1 zeek idps 296 Feb 27 07:30 mail.1493.tmp

 With the following content:

From: admin.zeek at domain.org<mailto:admin.zeek at domain.org>
Subject: [Zeek] cron: expire-logs failed
To: myadmin at otherdomain.org<mailto:myadmin at otherdomain.org>
User-Agent: ZeekControl 2.0.0

expire-logs failed
expire-logs: directory not found: /var/zeek/logs/stats

creating directory for stats file: /var/zeek/logs/stats

--
[Automatically generated.]

what output if any do you get if you run

    sendmail -t -oi /var/zeek/spool/tmp/mail.1493.tmp

or whatever filename exists there.
the "To:" line in there is what it looks for, so that should be working..

--
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200229/12554dbd/attachment-0001.html 

From karel.kuchar at dardas.cz  Sat Feb 29 04:23:14 2020
From: karel.kuchar at dardas.cz (=?iso-8859-2?Q?Karel_Kucha=F8?=)
Date: Sat, 29 Feb 2020 12:23:14 +0000
Subject: [Zeek] 802.11 frames
In-Reply-To: <QB1PR01MB3780EC5F7961C680A79CC4ECBBE80@QB1PR01MB3780.CANPRD01.PROD.OUTLOOK.COM>
References: <VI1PR02MB43502F246806DA79DD3D7E5295EB0@VI1PR02MB4350.eurprd02.prod.outlook.com>,
	<QB1PR01MB3780EC5F7961C680A79CC4ECBBE80@QB1PR01MB3780.CANPRD01.PROD.OUTLOOK.COM>
Message-ID: <VI1PR02MB43505803D68ED8A92091D80B95E90@VI1PR02MB4350.eurprd02.prod.outlook.com>

hi Martin,

thank you for your help. I have already tried to work with wireshark and there is easy to select only eapol frames. But I need to find theese frames within Zeek and to make some action when specific condiciton occures.  I was looking for any possibility to work with ether proto and then specify 0x888e.

thank you

Karel Kucha?

________________________________
Od: Martin Arlitt <marlitt at ucalgary.ca>
Odesl?no: p?tek 28. ?nora 2020 23:33
Komu: Karel Kucha? <karel.kuchar at dardas.cz>; zeek at zeek.org <zeek at zeek.org>
P?edm?t: Re: 802.11 frames

hi Karel

The ethertype in an EAPOL frame should be 0x888e (https://www.vocal.com/secure-communication/eapol-extensible-authentication-protocol-over-lan/).

In a pcap file it would be possible to distinguish EAPOL frames from other frames.

I'm not sure if zeek will process EAPOL frames (however, I'm not an expert on this matter). In the past I had to modify the source code in order to process frames that weren't IPv4, IPv6 or ARP ethertypes.

Martin

________________________________
From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> on behalf of Karel Kucha? <karel.kuchar at dardas.cz>
Sent: Thursday, February 27, 2020 10:24 AM
To: zeek at zeek.org <zeek at zeek.org>
Subject: [Zeek] 802.11 frames

Dear Zeek Community,

I'm new to zeek but now I'm working on project and I need to solve problem with anomaly detection on Wi-Fi. Is there any possibility how to detect frames specific for 802.11 like EAPOL frame?

Thanks in advance,

Karel K.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200229/8e2138c0/attachment.html 

From justin at corelight.com  Sat Feb 29 05:11:55 2020
From: justin at corelight.com (Justin Azoff)
Date: Sat, 29 Feb 2020 08:11:55 -0500
Subject: [Zeek] No emails are received
In-Reply-To: <16A6508C-5089-4C8E-B416-D2419C760379@outlook.com>
References: <DB8PR10MB35954CED6EC1568C0B8F5611DBEB0@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>
	<CAPfnCuiNmTrqBu6+TEL41bZyAYnkRk0Os5c5bcCvOUvZL51T6A@mail.gmail.com>
	<16A6508C-5089-4C8E-B416-D2419C760379@outlook.com>
Message-ID: <CAPfnCuhhZM9NfuxAg09SFZWEY+TsVtFP7wO4oD-HbR9oq7c-uA@mail.gmail.com>

That is the equivalent command for postfix.  All MTAs include a sendmail
compatible binary.

On Saturday, February 29, 2020, Carlos Lopez <clopmz at outlook.com> wrote:

> Thanks for your answer Justin. All my zeek hosts are installed with
> postfix (they are under RHEL 8.1). I am searching about an equivalent
> command with postfix but I can't find any?.
>
>
>
> --
>
> Regards,
>
> C. L. Martinez
>
>
>
> *From: *Justin Azoff <justin at corelight.com>
> *Date: *Friday, 28 February 2020 at 17:54
> *To: *Carlos Lopez <clopmz at outlook.com>
> *Cc: *"zeek at zeek.org" <zeek at zeek.org>
> *Subject: *Re: [Zeek] No emails are received
>
>
>
> On Thu, Feb 27, 2020 at 3:57 AM Carlos Lopez <clopmz at outlook.com> wrote:
>
> Hi all,
>
>  After re-installing my Zeek hosts to version 3.0.2 in my home lab, I
> haven't received any mail from cron task or any process/alert related to
> Zeek. But I see some emails queued in /var/zeek/spool/tmp directory like
> this:
>
> -rw-r--r--. 1 zeek idps 296 Feb 27 07:30 mail.1493.tmp
>
>  With the following content:
>
> From: admin.zeek at domain.org
> Subject: [Zeek] cron: expire-logs failed
> To: myadmin at otherdomain.org
> User-Agent: ZeekControl 2.0.0
>
> expire-logs failed
> expire-logs: directory not found: /var/zeek/logs/stats
>
> creating directory for stats file: /var/zeek/logs/stats
>
> --
> [Automatically generated.]
>
>
>
> what output if any do you get if you run
>
>
>
>     sendmail -t -oi /var/zeek/spool/tmp/mail.1493.tmp
>
>
>
> or whatever filename exists there.
>
> the "To:" line in there is what it looks for, so that should be working..
>
>
>
> --
>
> Justin
>


-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200229/e65573df/attachment.html 

From jlay at slave-tothe-box.net  Sat Feb 29 06:52:58 2020
From: jlay at slave-tothe-box.net (James Lay)
Date: Sat, 29 Feb 2020 07:52:58 -0700
Subject: [Zeek] No emails are received
In-Reply-To: <CAPfnCuhhZM9NfuxAg09SFZWEY+TsVtFP7wO4oD-HbR9oq7c-uA@mail.gmail.com>
References: <DB8PR10MB35954CED6EC1568C0B8F5611DBEB0@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>
	<CAPfnCuiNmTrqBu6+TEL41bZyAYnkRk0Os5c5bcCvOUvZL51T6A@mail.gmail.com>
	<16A6508C-5089-4C8E-B416-D2419C760379@outlook.com>
	<CAPfnCuhhZM9NfuxAg09SFZWEY+TsVtFP7wO4oD-HbR9oq7c-uA@mail.gmail.com>
Message-ID: <ad3225a14b2771f13b5ce86b0071147ce312c60d.camel@slave-tothe-box.net>

The aptly named "sendemail" works well:

http://caspian.dotconf.net/menu/Software/SendEmail/

I know it's in ubuntu's list, not sure on Redhat.

James
On Sat, 2020-02-29 at 08:11 -0500, Justin Azoff wrote:
> That is the equivalent command for postfix.  All MTAs include a
> sendmail compatible binary.
> 
> On Saturday, February 29, 2020, Carlos Lopez <clopmz at outlook.com>
> wrote:
> > 
> > 
> > 
> > 
> > 
> > 
> > Thanks for your answer Justin. All my zeek hosts are installed with
> > postfix (they are under RHEL 8.1). I am searching about an
> > equivalent command with postfix but I can't find any?.
> >  
> > 
> > -- 
> > 
> > Regards,
> > 
> > 
> > C. L. Martinez
> >  
> > 
> > From: Justin Azoff <justin at corelight.com>
> > 
> > Date: Friday, 28 February 2020 at 17:54
> > 
> > To: Carlos Lopez <clopmz at outlook.com>
> > 
> > Cc: "zeek at zeek.org" <zeek at zeek.org>
> > 
> > Subject: Re: [Zeek] No emails are received
> > 
> > 
> >  
> > 
> > 
> > 
> > On Thu, Feb 27, 2020 at 3:57 AM Carlos Lopez <clopmz at outlook.com>
> > wrote:
> > 
> > 
> > > Hi all,
> > > 
> > > 
> > > 
> > >  After re-installing my Zeek hosts to version 3.0.2 in my home
> > > lab, I haven't received any mail from cron task or any
> > > process/alert related to Zeek. But I see some emails queued in
> > > /var/zeek/spool/tmp directory like this:
> > > 
> > > 
> > > 
> > > -rw-r--r--. 1 zeek idps 296 Feb 27 07:30 mail.1493.tmp
> > > 
> > > 
> > > 
> > >  With the following content:
> > > 
> > > 
> > > 
> > > From: admin.zeek at domain.org
> > > 
> > > Subject: [Zeek] cron: expire-logs failed
> > > 
> > > To: myadmin at otherdomain.org
> > > 
> > > User-Agent: ZeekControl 2.0.0
> > > 
> > > 
> > > 
> > > expire-logs failed
> > > 
> > > expire-logs: directory not found: /var/zeek/logs/stats
> > > 
> > > 
> > > 
> > > creating directory for stats file: /var/zeek/logs/stats
> > > 
> > > 
> > > 
> > > --
> > > 
> > > [Automatically generated.]
> > 
> >  
> > 
> > 
> > what output if any do you get if you run
> > 
> > 
> >  
> > 
> > 
> >     sendmail -t -oi /var/zeek/spool/tmp/mail.1493.tmp
> > 
> > 
> > 
> >  
> > 
> > 
> > or whatever filename exists there.
> > 
> > 
> > the "To:" line in there is what it looks for, so that should be
> > working..
> > 
> > 
> >  
> > 
> > -- 
> > 
> > 
> > Justin
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> 
> _______________________________________________Zeek mailing
> listzeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200229/b9c94f40/attachment-0001.html 

From clopmz at outlook.com  Sat Feb 29 07:25:00 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Sat, 29 Feb 2020 15:25:00 +0000
Subject: [Zeek] No emails are received
In-Reply-To: <ad3225a14b2771f13b5ce86b0071147ce312c60d.camel@slave-tothe-box.net>
References: <DB8PR10MB35954CED6EC1568C0B8F5611DBEB0@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>
	<CAPfnCuiNmTrqBu6+TEL41bZyAYnkRk0Os5c5bcCvOUvZL51T6A@mail.gmail.com>
	<16A6508C-5089-4C8E-B416-D2419C760379@outlook.com>
	<CAPfnCuhhZM9NfuxAg09SFZWEY+TsVtFP7wO4oD-HbR9oq7c-uA@mail.gmail.com>
	<ad3225a14b2771f13b5ce86b0071147ce312c60d.camel@slave-tothe-box.net>
Message-ID: <0E61F299-A6C5-44E0-AE84-5FE6D297551A@outlook.com>

Ok, running ?sendmail -t -oi /var/zeek/spool/tmp/mail.1493.tmp?, does not return anything and email is not sent? and sendemail requires ?from? and ?to? options. Putting these options, it works ?

--
Regards,
C. L. Martinez

From: <zeek-bounces at zeek.org> on behalf of James Lay <jlay at slave-tothe-box.net>
Reply to: "jlay at slave-tothe-box.net" <jlay at slave-tothe-box.net>
Date: Saturday, 29 February 2020 at 15:55
To: "zeek at zeek.org" <zeek at zeek.org>
Subject: Re: [Zeek] No emails are received

The aptly named "sendemail" works well:

http://caspian.dotconf.net/menu/Software/SendEmail/

I know it's in ubuntu's list, not sure on Redhat.

James

On Sat, 2020-02-29 at 08:11 -0500, Justin Azoff wrote:
That is the equivalent command for postfix.  All MTAs include a sendmail compatible binary.

On Saturday, February 29, 2020, Carlos Lopez <clopmz at outlook.com<mailto:clopmz at outlook.com>> wrote:

Thanks for your answer Justin. All my zeek hosts are installed with postfix (they are under RHEL 8.1). I am searching about an equivalent command with postfix but I can't find any?.

--
Regards,
C. L. Martinez

From: Justin Azoff <justin at corelight.com<mailto:justin at corelight.com>>
Date: Friday, 28 February 2020 at 17:54
To: Carlos Lopez <clopmz at outlook.com<mailto:clopmz at outlook.com>>
Cc: "zeek at zeek.org<mailto:zeek at zeek.org>" <zeek at zeek.org<mailto:zeek at zeek.org>>
Subject: Re: [Zeek] No emails are received

On Thu, Feb 27, 2020 at 3:57 AM Carlos Lopez <clopmz at outlook.com<mailto:clopmz at outlook.com>> wrote:
Hi all,

 After re-installing my Zeek hosts to version 3.0.2 in my home lab, I haven't received any mail from cron task or any process/alert related to Zeek. But I see some emails queued in /var/zeek/spool/tmp directory like this:

-rw-r--r--. 1 zeek idps 296 Feb 27 07:30 mail.1493.tmp

 With the following content:

From: admin.zeek at domain.org<mailto:admin.zeek at domain.org>
Subject: [Zeek] cron: expire-logs failed
To: myadmin at otherdomain.org<mailto:myadmin at otherdomain.org>
User-Agent: ZeekControl 2.0.0

expire-logs failed
expire-logs: directory not found: /var/zeek/logs/stats

creating directory for stats file: /var/zeek/logs/stats

--
[Automatically generated.]

what output if any do you get if you run

    sendmail -t -oi /var/zeek/spool/tmp/mail.1493.tmp

or whatever filename exists there.
the "To:" line in there is what it looks for, so that should be working..

--
Justin



_______________________________________________

Zeek mailing list

zeek at zeek.org<mailto:zeek at zeek.org>

http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200229/e0e1dc6d/attachment.html 

From michalpurzynski1 at gmail.com  Sat Feb 29 20:56:55 2020
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Sat, 29 Feb 2020 20:56:55 -0800
Subject: [Zeek] Dropping packets
In-Reply-To: <MN2PR20MB254118B3E022876DD882824BEEED0@MN2PR20MB2541.namprd20.prod.outlook.com>
References: <MN2PR20MB254118B3E022876DD882824BEEED0@MN2PR20MB2541.namprd20.prod.outlook.com>
Message-ID: <336B28E0-F862-4F8D-A15E-DB60328DE705@gmail.com>

Arista doesn?t dedup anything automatically. But it?s like 10x cheaper than solutions that do.

You can (and I did) write
- ACLs and attach them as ingress to tap ports ( hardware supported)
- ACLs and attach them as egrees to tool ports (hardware supported)
- drop packets by creating classes and policies and send them nowhere (software only, I did that when Arista was fighting for life with Cisco and had to ban hardware ACLs for a while)

Some people also double-tag  traffic, with the outer-most tag identifying the tap port. You can then use that when writing policies and ACLs.

> On Feb 25, 2020, at 5:50 AM, Joseph Fischetti <Joseph.Fischetti at marist.edu> wrote:
> 
> ?
> The imbalance in traffic on the interfaces is apparently due to the way the Arista switches are physically connected.  Networking is going to change that:
> 
> We have span ports on 2 different links feeding 2 different Arista switches.  One Arista switch goes to one set of interfaces (eth4 on both boxes).  And the other Arista switch goes to the other set of interfaces (eth5 on both boxes).  One link is used as primary so it sees a lot more traffic than the other.  In addition to that, the Arista switch with the heavy link has span ports from the core.
>  
> We?re going to move the core spans over to the Arista with the light link? that should hopefully work to rebalance things.
>  
> That said? Will the Arista switches do any dedup on the packets that they?re getting?  I was uninvolved on the design/setup of the system.  It would seem to me that if we have some vlans spanning to the Aristas, and ports from the edge spanning to the Aristas?. Packets that go through the edge and through those vlans in the core will both be spanned to the Arista (and sent to zeek).
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200229/bdb47628/attachment.html