From manyiant at 163.com  Sun Mar  1 22:33:13 2020
From: manyiant at 163.com (my)
Date: Mon, 2 Mar 2020 14:33:13 +0800 (CST)
Subject: [Zeek] =?gbk?q?Help=A3=ACAbout_Packet_Filter?=
In-Reply-To: <CAPfnCuihqSEtoYZeUbHN5e=RvTxRi3-yPKO5_UaXbKC0gK-o0w@mail.gmail.com>
References: <593e71f.1ac0.16fcb21578c.Coremail.manyiant@163.com>
	<CAPfnCuihqSEtoYZeUbHN5e=RvTxRi3-yPKO5_UaXbKC0gK-o0w@mail.gmail.com>
Message-ID: <6b425659.571a.17099f2d4ed.Coremail.manyiant@163.com>

Hi,
I tested the config you provided, but it didn't work. I read the source code and used the following config for traffic filtering. It's not perfect.


Iredef packet_filter_default = F;
event zeek_init()
{
    install_src_addr_filter($ip=123.2.15.75, $tcp_flags=63, $prob=1.0);
    install_src_addr_filter($ip=123.2.15.75, $tcp_flags=1, $prob=1.0);
    install_src_addr_filter($ip=123.2.15.75, $tcp_flags=2, $prob=1.0);
    install_src_addr_filter($ip=123.2.15.75, $tcp_flags=4, $prob=1.0);
...
}











At 2020-01-24 00:15:22, "Justin Azoff" <justin at corelight.com> wrote:

Is your traffic encapsulated with vlan tags?  Does changing the filter to
    vlan and host 123.2.15.75
work any better?


On Tue, Jan 21, 2020 at 9:44 PM my <manyiant at 163.com> wrote:

Hi?friends:
I use restrict_filters to filter the traffic. but the settings did not take effect, all of the traffic was filtered. What should I do?
My script is as follows:
redef restrict_filters += {
      ["unmonitored host"] = "host 123.2.15.75"
};


I am looking forwoard to your replay. Thakns.




 

_______________________________________________
Zeek mailing list
zeek at zeek.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek




--

Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200302/8d51761c/attachment.html 

From clopmz at outlook.com  Mon Mar  2 03:43:48 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Mon, 2 Mar 2020 11:43:48 +0000
Subject: [Zeek] Some troubles with Corelight's Splunk app using Zeek 3.0.2
Message-ID: <DB8PR10MB359541FB0897CFEB574C6B07DBE70@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>

Good morning,

 I've been using the Corelight's Splunk application for several days now with only one sensor and everything works fine except for the "Data Exploration/Connections" view. It never shows me any results, it always shows "No results found". I have been debugging the searches with the option "inspect" and it is correct: none of those searches can return results.

 An example: for Top Services view, Corelight's app performs the following search:

search (NOT sensor_name!="*" id_orig_h="*" id_orig_p="*" id_resp_h="*" id_resp_p="*" NOT is_broadcast="true" service="*" (eventtype=bro_conn OR eventtype=corelight_conn)) | top service limit=15

 and without result. But If I use the following search in Splunk's general view:

 search (NOT is_broadcast="true" (eventtype=corelight_conn)) | top service limit=15

 I get results as you can see in the screenshot attached.

 Am I doing something wrong or is it a bug?

 Many thaks



Regards,
C. L. Martinez
-------------- next part --------------
A non-text attachment was scrubbed...
Name: corelight.png
Type: image/png
Size: 36459 bytes
Desc: corelight.png
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200302/0d2bc211/attachment-0001.bin 

From akgraner at corelight.com  Mon Mar  2 07:40:48 2020
From: akgraner at corelight.com (Amber Graner)
Date: Mon, 2 Mar 2020 10:40:48 -0500
Subject: [Zeek] Reminder - Monthly Zeek Community Call - 6 March 2020
 12:00pm PST/3:00pm EST
Message-ID: <CAJhOzurqd=OsRzRmGDWbSNKeOsVHCCb7Tvc_qkF1Xx6q_Nk=Vg@mail.gmail.com>

Hi all and Happy Monday!!

Just a reminder that the next monthly Zeek Community Call will be Friday 6
March 2020 at 12:00pm PST/3:00pm EST. The calendar invite has been updated.

You can find out what it will be in your timezone at timeanddate.com
-https://www.timeanddate.com/worldclock/converter.html


Agenda
======

  - ZeekWeek 2020
  - Slack Channel

If you have any topics you'd like to have added to the call, please let me
know and I'll update the invite and the agenda.

Joining the call
============

Join Zoom Meeting
https://corelight.zoom.us/j/898658920

Meeting ID: 898 658 920

One tap mobile
+16465588656,,898658920# US (New York)
+16699006833,,898658920# US (San Jose)

Dial by your location
+1 646 558 8656 US (New York)
+1 669 900 6833 US (San Jose)
877 853 5257 US Toll-free
888 475 4499 US Toll-free
Meeting ID: 898 658 920
Find your local number: https://corelight.zoom.us/u/acY5L1LN7

Please let me know if you have any questions.

Thanks,
~Amber
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200302/ac06478a/attachment.html 

From akgraner at corelight.com  Mon Mar  2 14:33:47 2020
From: akgraner at corelight.com (Amber Graner)
Date: Mon, 2 Mar 2020 17:33:47 -0500
Subject: [Zeek] Issue 2 - Zeek Monthly Newsletter - Now Available
Message-ID: <CAJhOzuruap6b8jHGqX7GpcXa3wdpKDJebJBDxhxJDAmzQPgnGQ@mail.gmail.com>

Issue 2 - Zeek Monthly Newsletter - Now Available -
https://blog.zeek.org/2020/03/zeek-monthly-newsletter-issue-2-march.html

In this Issue:

* General Community News/Updates
* Development Updates
* Zeek In the Community
* Threat of the Month
* Upcoming Events
* Contribution/Contributor of the Month
* Zeek Related Issues
* Publication Schedule
* Get Involved

A .PDF version of this issue can be downloaded at:
http://bit.ly/ZMN_Issue_2_March_2020

We're currently working on Issue 3 - the draft placeholder can be found at:
https://docs.google.com/document/d/1SI2YC13JVj5AlEZK9sdxBjwUO8QwWZ-0o2uXKgz9_6s/edit

If you would like to help with the Newsletter or if you have links that you
would like for us to include, please let me know.

Thanks,
~Amber
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200302/f8be44fe/attachment.html 

From kczarnec at pppl.gov  Tue Mar  3 09:52:18 2020
From: kczarnec at pppl.gov (Kevin Czarnecki)
Date: Tue, 3 Mar 2020 12:52:18 -0500
Subject: [Zeek] [External] Some troubles with Corelight's Splunk app
 using Zeek 3.0.2
In-Reply-To: <DB8PR10MB359541FB0897CFEB574C6B07DBE70@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>
References: <DB8PR10MB359541FB0897CFEB574C6B07DBE70@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>
Message-ID: <CAF3H6E0CUp2E20yLvjnCvbo6vAUEnCQfzuv5vGLEiOFMryC6Dw@mail.gmail.com>

Are you using a Corelight sensor or Zeek? I've found some field names are
different on different devices. (id_orig_h vs. id.orig_h).

I had to do a fair bit of "massaging" of the searches to get them working
with different field and index names.

On Mon, Mar 2, 2020 at 6:47 AM Carlos Lopez <clopmz at outlook.com> wrote:

> Good morning,
>
>  I've been using the Corelight's Splunk application for several days now
> with only one sensor and everything works fine except for the "Data
> Exploration/Connections" view. It never shows me any results, it always
> shows "No results found". I have been debugging the searches with the
> option "inspect" and it is correct: none of those searches can return
> results.
>
>  An example: for Top Services view, Corelight's app performs the following
> search:
>
> search (NOT sensor_name!="*" id_orig_h="*" id_orig_p="*" id_resp_h="*"
> id_resp_p="*" NOT is_broadcast="true" service="*" (eventtype=bro_conn OR
> eventtype=corelight_conn)) | top service limit=15
>
>  and without result. But If I use the following search in Splunk's general
> view:
>
>  search (NOT is_broadcast="true" (eventtype=corelight_conn)) | top service
> limit=15
>
>  I get results as you can see in the screenshot attached.
>
>  Am I doing something wrong or is it a bug?
>
>  Many thaks
>
>
>
> Regards,
> C. L. Martinez
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200303/406730a5/attachment.html 

From jlay at slave-tothe-box.net  Tue Mar  3 11:03:16 2020
From: jlay at slave-tothe-box.net (James Lay)
Date: Tue, 03 Mar 2020 12:03:16 -0700
Subject: [Zeek] First attempt to upgrade to 3:  Multiple interfaces
Message-ID: <c8cb17b05da1d98434bf6ec14e5a035b@slave-tothe-box.net>

Welp...out of luck so far:

/opt/zeek/bin/zeek -C -i eth0 -i eth1 --filter '<redacted>' local 
"Site::local_nets += { 192.168.1.0/24 }"

gets me:

ERROR: Only a single interface option (-i) is allowed.

I didn't have this issue with 2.  Any reason why only one interface is 
allowed now?  Unless something radical has changed with the resources 
that zeekctl uses I have no desire to use it.  I'm dead in the water 
with Zeek as of now.  Thank you.

James

From tim at corelight.com  Tue Mar  3 11:28:47 2020
From: tim at corelight.com (Tim Wojtulewicz)
Date: Tue, 3 Mar 2020 12:28:47 -0700
Subject: [Zeek] First attempt to upgrade to 3:  Multiple interfaces
In-Reply-To: <c8cb17b05da1d98434bf6ec14e5a035b@slave-tothe-box.net>
References: <c8cb17b05da1d98434bf6ec14e5a035b@slave-tothe-box.net>
Message-ID: <5C741D97-A6B8-4ECA-915B-3A6BA4D99E13@corelight.com>

If you don?t really need the latest and greatest cutting edge changes to 3.1, version 3.0.x still supports multiple interfaces. That feature was removed in 3.1 due to the wide changes to the IO Loop architecture, and you?re honestly the first user I?ve heard from that has noticed it missing. It was removed to make that work easier to accomplish, but we can certainly investigate bringing it back if there?s enough of a use case for it.

Tim

> On Mar 3, 2020, at 12:03 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> 
> Welp...out of luck so far:
> 
> /opt/zeek/bin/zeek -C -i eth0 -i eth1 --filter '<redacted>' local 
> "Site::local_nets += { 192.168.1.0/24 }"
> 
> gets me:
> 
> ERROR: Only a single interface option (-i) is allowed.
> 
> I didn't have this issue with 2.  Any reason why only one interface is 
> allowed now?  Unless something radical has changed with the resources 
> that zeekctl uses I have no desire to use it.  I'm dead in the water 
> with Zeek as of now.  Thank you.
> 
> James
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



From smoot at corelight.com  Tue Mar  3 11:42:14 2020
From: smoot at corelight.com (Steve Smoot)
Date: Tue, 3 Mar 2020 11:42:14 -0800
Subject: [Zeek] First attempt to upgrade to 3: Multiple interfaces
In-Reply-To: <5C741D97-A6B8-4ECA-915B-3A6BA4D99E13@corelight.com>
References: <c8cb17b05da1d98434bf6ec14e5a035b@slave-tothe-box.net>
	<5C741D97-A6B8-4ECA-915B-3A6BA4D99E13@corelight.com>
Message-ID: <CADOXQ8YYS2QtOyzoguUQMrGA5Bwo9CUX-kZ-XqvF_P=oAKvo0g@mail.gmail.com>

On Tue, Mar 3, 2020 at 11:30 AM Tim Wojtulewicz <tim at corelight.com> wrote:

> If you don?t really need the latest and greatest cutting edge changes to
> 3.1, version 3.0.x still supports multiple interfaces. That feature was
> removed in 3.1 due to the wide changes to the IO Loop architecture, and
> you?re honestly the first user I?ve heard from that has noticed it missing.
> It was removed to make that work easier to accomplish, but we can certainly
> investigate bringing it back if there?s enough of a use case for it.
>

Another option, I think would be to bond/bridge the interfaces and listen
on that.  If that would work for you,

-s


> Tim
>
> > On Mar 3, 2020, at 12:03 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> >
> > Welp...out of luck so far:
> >
> > /opt/zeek/bin/zeek -C -i eth0 -i eth1 --filter '<redacted>' local
> > "Site::local_nets += { 192.168.1.0/24 }"
> >
> > gets me:
> >
> > ERROR: Only a single interface option (-i) is allowed.
> >
> > I didn't have this issue with 2.  Any reason why only one interface is
> > allowed now?  Unless something radical has changed with the resources
> > that zeekctl uses I have no desire to use it.  I'm dead in the water
> > with Zeek as of now.  Thank you.
> >
> > James
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
*Stephen R. Smoot, PhD*
VP, Customer Success
Corelight
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200303/219edd5d/attachment-0001.html 

From jlay at slave-tothe-box.net  Tue Mar  3 12:10:38 2020
From: jlay at slave-tothe-box.net (James Lay)
Date: Tue, 03 Mar 2020 13:10:38 -0700
Subject: [Zeek] First attempt to upgrade to 3: Multiple interfaces
In-Reply-To: <CADOXQ8YYS2QtOyzoguUQMrGA5Bwo9CUX-kZ-XqvF_P=oAKvo0g@mail.gmail.com>
References: <c8cb17b05da1d98434bf6ec14e5a035b@slave-tothe-box.net>
	<5C741D97-A6B8-4ECA-915B-3A6BA4D99E13@corelight.com>
	<CADOXQ8YYS2QtOyzoguUQMrGA5Bwo9CUX-kZ-XqvF_P=oAKvo0g@mail.gmail.com>
Message-ID: <b6c9aa292de9654b57da9966d87a3ced@slave-tothe-box.net>

Appreciate the responses.  These interfaces are an external on the 
internet, and an internal with a localnet (this devices is classified as 
router), so bonding them isn't an option.  The only reason I'm running 
in this manner as apposed to just letting zeekctl handle it all is the 
process count and memory usage.  I guess I'll test out zeekctl and see 
where I sit....might have to fallback to 3.0.  Thank you.

James

On 2020-03-03 12:42, Steve Smoot wrote:
> On Tue, Mar 3, 2020 at 11:30 AM Tim Wojtulewicz <tim at corelight.com>
> wrote:
> 
>> If you don?t really need the latest and greatest cutting edge
>> changes to 3.1, version 3.0.x still supports multiple interfaces.
>> That feature was removed in 3.1 due to the wide changes to the IO
>> Loop architecture, and you?re honestly the first user I?ve heard
>> from that has noticed it missing. It was removed to make that work
>> easier to accomplish, but we can certainly investigate bringing it
>> back if there?s enough of a use case for it.
> 
> Another option, I think would be to bond/bridge the interfaces and
> listen on that.  If that would work for you,
> 
> -s
> 
>> Tim
>> 
>>> On Mar 3, 2020, at 12:03 PM, James Lay <jlay at slave-tothe-box.net>
>> wrote:
>>> 
>>> Welp...out of luck so far:
>>> 
>>> /opt/zeek/bin/zeek -C -i eth0 -i eth1 --filter '<redacted>' local
>>> "Site::local_nets += { 192.168.1.0/24 [1] }"
>>> 
>>> gets me:
>>> 
>>> ERROR: Only a single interface option (-i) is allowed.
>>> 
>>> I didn't have this issue with 2.  Any reason why only one
>> interface is
>>> allowed now?  Unless something radical has changed with the
>> resources
>>> that zeekctl uses I have no desire to use it.  I'm dead in the
>> water
>>> with Zeek as of now.  Thank you.
>>> 
>>> James
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>> 
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> 
> --
> 
> STEPHEN R. SMOOT, PHD
> VP, Customer Success
> Corelight
> 
> 
> Links:
> ------
> [1] http://192.168.1.0/24

From jlay at slave-tothe-box.net  Tue Mar  3 17:50:51 2020
From: jlay at slave-tothe-box.net (James Lay)
Date: Tue, 03 Mar 2020 18:50:51 -0700
Subject: [Zeek] First attempt to upgrade to 3: Multiple interfaces
In-Reply-To: <b6c9aa292de9654b57da9966d87a3ced@slave-tothe-box.net>
References: <c8cb17b05da1d98434bf6ec14e5a035b@slave-tothe-box.net>
	<5C741D97-A6B8-4ECA-915B-3A6BA4D99E13@corelight.com>
	<CADOXQ8YYS2QtOyzoguUQMrGA5Bwo9CUX-kZ-XqvF_P=oAKvo0g@mail.gmail.com>
	<b6c9aa292de9654b57da9966d87a3ced@slave-tothe-box.net>
Message-ID: <43959c9db49e46e8ea5dbcaaec46d2e73a494424.camel@slave-tothe-box.net>

All,

I'm please to report that using zeekctl method of running is so far
working well...resource usage is so far manageable.  Thanks for the
assistance.

James

On Tue, 2020-03-03 at 13:10 -0700, James Lay wrote:
> Appreciate the responses.  These interfaces are an external on the
> internet, and an internal with a localnet (this devices is classified
> as router), so bonding them isn't an option.  The only reason I'm
> running in this manner as apposed to just letting zeekctl handle it
> all is the process count and memory usage.  I guess I'll test out
> zeekctl and see where I sit....might have to fallback to 3.0.  Thank
> you.
> James
> On 2020-03-03 12:42, Steve Smoot wrote:
> On Tue, Mar 3, 2020 at 11:30 AM Tim Wojtulewicz <tim at corelight.com>wr
> ote:
> If you don?t really need the latest and greatest cutting edgechanges
> to 3.1, version 3.0.x still supports multiple interfaces.That feature
> was removed in 3.1 due to the wide changes to the IOLoop
> architecture, and you?re honestly the first user I?ve heardfrom that
> has noticed it missing. It was removed to make that workeasier to
> accomplish, but we can certainly investigate bringing itback if
> there?s enough of a use case for it.
> Another option, I think would be to bond/bridge the interfaces
> andlisten on that.  If that would work for you,
> -s
> Tim
> On Mar 3, 2020, at 12:03 PM, James Lay <jlay at slave-tothe-box.net>wrot
> e:
> 
> Welp...out of luck so far:
> /opt/zeek/bin/zeek -C -i eth0 -i eth1 --filter '<redacted>'
> local"Site::local_nets += { 192.168.1.0/24 [1] }"
> gets me:
> ERROR: Only a single interface option (-i) is allowed.
> I didn't have this issue with 2.  Any reason why only oneinterface is
> allowed now?  Unless something radical has changed with theresources
> that zeekctl uses I have no desire to use it.  I'm dead in thewater
> with Zeek as of now.  Thank you.
> James_______________________________________________Zeek mailing
> listzeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> _______________________________________________Zeek mailing
> listzeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> --
> STEPHEN R. SMOOT, PHDVP, Customer SuccessCorelight
> 
> Links:------[1] 
> http://192.168.1.0/24_______________________________________________Zeek
>  mailing listzeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200303/e3fb8ac8/attachment.html 

From michalpurzynski1 at gmail.com  Tue Mar  3 18:37:49 2020
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Tue, 3 Mar 2020 18:37:49 -0800
Subject: [Zeek] First attempt to upgrade to 3: Multiple interfaces
In-Reply-To: <43959c9db49e46e8ea5dbcaaec46d2e73a494424.camel@slave-tothe-box.net>
References: <43959c9db49e46e8ea5dbcaaec46d2e73a494424.camel@slave-tothe-box.net>
Message-ID: <29313A40-9D2A-4843-B9C0-2739CDB6822E@gmail.com>

From what you?re describing, you?re running Zeek on a resource constraints device.

You might want to actually use the newest version - not sure where Justin fixes for the packet processing loop went in, but they should help a lot.

There are also some scripts to avoid, like the scan.zeek one and basically anything using SumStats. These can eat a lot of memory.

BTW kill all unnecessary processes. I can take a look if you send a process tree.

Good luck!

> On Mar 3, 2020, at 5:59 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> 
> ?
> All,
> 
> I'm please to report that using zeekctl method of running is so far working well...resource usage is so far manageable. Thanks for the assistance.
> 
> James
> 
>> On Tue, 2020-03-03 at 13:10 -0700, James Lay wrote:
>> Appreciate the responses.  These interfaces are an external on the 
>> internet, and an internal with a localnet (this devices is classified as 
>> router), so bonding them isn't an option.  The only reason I'm running 
>> in this manner as apposed to just letting zeekctl handle it all is the 
>> process count and memory usage.  I guess I'll test out zeekctl and see 
>> where I sit....might have to fallback to 3.0.  Thank you.
>> 
>> James
>> 
>>> On 2020-03-03 12:42, Steve Smoot wrote:
>> 
>> On Tue, Mar 3, 2020 at 11:30 AM Tim Wojtulewicz <tim at corelight.com>
>> wrote:
>> 
>> 
>> If you don?t really need the latest and greatest cutting edge
>> changes to 3.1, version 3.0.x still supports multiple interfaces.
>> That feature was removed in 3.1 due to the wide changes to the IO
>> Loop architecture, and you?re honestly the first user I?ve heard
>> from that has noticed it missing. It was removed to make that work
>> easier to accomplish, but we can certainly investigate bringing it
>> back if there?s enough of a use case for it.
>> 
>> Another option, I think would be to bond/bridge the interfaces and
>> listen on that.  If that would work for you,
>> 
>> -s
>> 
>> 
>> Tim
>> 
>> 
>> On Mar 3, 2020, at 12:03 PM, James Lay <jlay at slave-tothe-box.net>
>> wrote:
>> 
>> 
>> Welp...out of luck so far:
>> 
>> /opt/zeek/bin/zeek -C -i eth0 -i eth1 --filter '<redacted>' local
>> "Site::local_nets += { 192.168.1.0/24 [1] }"
>> 
>> gets me:
>> 
>> ERROR: Only a single interface option (-i) is allowed.
>> 
>> I didn't have this issue with 2.  Any reason why only one
>> interface is
>> 
>> allowed now?  Unless something radical has changed with the
>> resources
>> 
>> that zeekctl uses I have no desire to use it.  I'm dead in the
>> water
>> 
>> with Zeek as of now.  Thank you.
>> 
>> James
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>> 
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>> 
>> --
>> 
>> STEPHEN R. SMOOT, PHD
>> VP, Customer Success
>> Corelight
>> 
>> 
>> Links:
>> ------
>> [1] http://192.168.1.0/24
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> 
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200303/aad611a2/attachment-0001.html 

From justin at corelight.com  Tue Mar  3 18:50:05 2020
From: justin at corelight.com (Justin Azoff)
Date: Tue, 3 Mar 2020 21:50:05 -0500
Subject: [Zeek] First attempt to upgrade to 3: Multiple interfaces
In-Reply-To: <29313A40-9D2A-4843-B9C0-2739CDB6822E@gmail.com>
References: <43959c9db49e46e8ea5dbcaaec46d2e73a494424.camel@slave-tothe-box.net>
	<29313A40-9D2A-4843-B9C0-2739CDB6822E@gmail.com>
Message-ID: <CAPfnCujeMWeykk651-ELeoN_WAtEC85vzpaZhW3LrYg=+799KA@mail.gmail.com>

On Tue, Mar 3, 2020 at 9:39 PM Micha? Purzy?ski <michalpurzynski1 at gmail.com>
wrote:

> From what you?re describing, you?re running Zeek on a resource constraints
> device.
>
> You might want to actually use the newest version - not sure where Justin
> fixes for the packet processing loop went in, but they should help a lot.
>

Oh hah, I don't need any credit for all of that work.  I had an old patch
against 2.4/5/6 that would relax the polling loop interval so it wouldn't
eat up a lot of cpu on smaller devices, but it wasn't a fix.  Tim
completely rewrote the IO loop so now zeek on a raspberry pi should use 0%
cpu when there's no traffic.

-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200303/4d139aeb/attachment.html 

From doug.burks at gmail.com  Wed Mar  4 08:35:54 2020
From: doug.burks at gmail.com (Doug Burks)
Date: Wed, 4 Mar 2020 11:35:54 -0500
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAAYFXL=bZYmFCiRKHswDY4xBJ5en-kBkWoOoc8Pn4kn=XdLpfQ@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
	<CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
	<CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>
	<CAPfnCuizcGsP1jhAafqmkRn-Mq=uMNq2s3+jQf8vkkQHeLg3fQ@mail.gmail.com>
	<CAAYFXLmwMZUep3o-N02fWhx6ybcgbT6htfbdyRKoNd__fdz2hQ@mail.gmail.com>
	<CAMzgZ0KoVGu2KmA7u06B9FmbXbhA2h0toxgNDUVGHbz-yc1UCQ@mail.gmail.com>
	<CAAYFXL=bZYmFCiRKHswDY4xBJ5en-kBkWoOoc8Pn4kn=XdLpfQ@mail.gmail.com>
Message-ID: <CAK8kjrCF_AFZOboTCPZ21x8hGDcwPGGa_KboMA9Bm85pzV_3-A@mail.gmail.com>

I've been able to duplicate this issue so I'm passing along some notes in
the hope that others are able to duplicate the issue as well and perhaps
pinpoint what's going on.

- I've seen this issue on both physical boxes and virtual machines and on
both single CPU socket and multiple CPU socket systems

- I've been able to trigger this issue fairly consistently using VMware
Workstation with the VM set to 4 processors (seems easier to duplicate when
using processors rather than cores)

- 8GB RAM and 2 NICs (one set to NAT for management and the other set to a
custom network and configured for sniffing)

- running our latest Security Onion ISO image which contains Zeek 3.0.1
(I've also duplicated this behavior using Zeek 3.0.2 compiled manually):
https://blog.securityonion.net/2020/02/security-onion-160464-iso-image-now.html

- run sosetup-minimal and choose Evaluation Mode

- once Setup is complete, create some traffic on the sniffing interface:
while :; do sudo so-replay; done

- on my box, Zeek normally runs at about 10% to 20% CPU usage when running
so-replay but after a certain period of time (seems inconsistent, could be
minutes or over an hour), Zeek will go to 100% CPU usage and remain there
even if you kill the so-replay while loop from above

- you can restart Zeek with "sudo so-zeek-restart" and it will go back to
normal operation and normal CPU usage, but after a while of processing
traffic it will go back to 100% CPU usage

- as mentioned above, you can also download Zeek 3.0.2 and compile it
manually according to https://docs.zeek.org/en/v3.0.2/install/install.html
and duplicate the issue there, so this would seem to rule out any possible
issues with our Zeek package or scripts

Please let me know if I can provide any further information to assist in
duplicating and pinpointing this issue.

Thanks!


On Tue, Feb 25, 2020 at 6:41 PM Pete Nelson <petiepooo at gmail.com> wrote:

> Thanks, Jon.
>
> I'll try to digest those links and dig into the code.  Unfortunately,
> it seems running strace on the process keeps it from occurring...  I
> may try to get dtrace working in place, but I need to improve my lab
> setup first before I go too crazy.
> --
> Pete
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>


-- 
Doug Burks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200304/1c3ea34e/attachment.html 

From akgraner at corelight.com  Wed Mar  4 08:37:42 2020
From: akgraner at corelight.com (Amber Graner)
Date: Wed, 4 Mar 2020 11:37:42 -0500
Subject: [Zeek] Zeek Slack Workspace Now available!!
Message-ID: <CAJhOzurQ4k_Vjd-2GzU=C+EfatLuj-OrTZM60E_ySqG_0uZ7xQ@mail.gmail.com>

Hi all,

Thank you so much for your patience as we set up the Workspace and the
administrative items that go with these spaces in an open source community.

* The space is called: zeekorg.slack.com

* You can join using the following invitation link (There is also an invite
link on the blog post):
h
ttps://join.slack.com/t/zeekorg/shared_invite/enQtOTc3MzMxNDI1NDYxLTA1NzhhMTgxNWI1OTk2NjlkMTdjNzY1Nzk5NDk2ZDY1MDBkYWIxOWNjNDE2NDc2MGI5OWM3ZDllYzBmZmNhNDM

In addition to this space we are also introducing a Code of Conduct and
Slack Channel Guidelines, links to both are listed below and on the blog
post (*this links will also be listed on the website soon*).  We'd also
like to the Kubernetes Community as these are based on and modified from
theirs.

* Code of Conduct -
https://www.dropbox.com/s/yh3xku3sflhs4ry/Final-%20%20Zeek%20Community%20Code%20of%20Conduct.pdf?dl=0

* Slack Channel Guidelines  -
https://www.dropbox.com/s/f47mb0ywvilu41g/Final%20-%20Zeek%20Community%20Slack%20Guidelines%20%281%29.pdf?dl=0

Below is the link to the blog post:
https://blog.zeek.org/2020/03/zeek-slack-channel-announced.html

As mentioned in the blog post, "We?re thrilled that more than 20 people
have volunteered to admin the workspace. And we would like to thank
everyone who has offered to help. The Zeek Leadership Team (LT) is
currently working on the policy for adding admins."  Those who volunteered
should hear more about those roles and responsibilities by the end of the
month.

Please let me know if you have any questions.

Thanks,
~Amber
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200304/abe7bfe8/attachment.html 

From tlarson.hiscorp at gmail.com  Wed Mar  4 09:32:25 2020
From: tlarson.hiscorp at gmail.com (Tim Larson)
Date: Wed, 4 Mar 2020 11:32:25 -0600
Subject: [Zeek] Authentication data available in zeek logs
Message-ID: <CAAQcfED92ugv23N8i22Y8vgbnMZz_sHEZ5sz_zySC2XFY9jarQ@mail.gmail.com>

I am not aware of Zeek having an authoritative list of all authentication
data so from your experience which Zeek logs would provide *useful event
data* when authenticating to each of the following:

1) web applications

2) API's

3) SSO

4) SSH

5) Windows authentication

6) FTP


Kind regards,

tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200304/315123f6/attachment-0001.html 

From petr.medonos at etnetera.cz  Wed Mar  4 11:57:03 2020
From: petr.medonos at etnetera.cz (Petr Medonos)
Date: Wed, 4 Mar 2020 20:57:03 +0100
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAK8kjrCF_AFZOboTCPZ21x8hGDcwPGGa_KboMA9Bm85pzV_3-A@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
	<CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
	<CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>
	<CAPfnCuizcGsP1jhAafqmkRn-Mq=uMNq2s3+jQf8vkkQHeLg3fQ@mail.gmail.com>
	<CAAYFXLmwMZUep3o-N02fWhx6ybcgbT6htfbdyRKoNd__fdz2hQ@mail.gmail.com>
	<CAMzgZ0KoVGu2KmA7u06B9FmbXbhA2h0toxgNDUVGHbz-yc1UCQ@mail.gmail.com>
	<CAAYFXL=bZYmFCiRKHswDY4xBJ5en-kBkWoOoc8Pn4kn=XdLpfQ@mail.gmail.com>
	<CAK8kjrCF_AFZOboTCPZ21x8hGDcwPGGa_KboMA9Bm85pzV_3-A@mail.gmail.com>
Message-ID: <95f56d8d-9791-c86f-716c-098ad2423d7a@etnetera.cz>

Hi all,
not sure yet, but looks very similar to issue I started here
https://github.com/zeek/zeek/issues/838. I will take a look what Zeek
scripts are enabled by default in SO to correlate these issues.


Petr


On 04. 03. 20 17:35, Doug Burks wrote:
> I've been able to duplicate this issue so I'm passing along some notes
> in the hope that others are able to duplicate the issue as well and
> perhaps pinpoint what's going on.
> 
> - I've seen this issue on both physical boxes and virtual machines and
> on both single CPU socket and multiple CPU socket systems
> 
> - I've been able to trigger this issue fairly consistently using VMware
> Workstation with the VM set to 4 processors (seems easier to duplicate
> when using processors rather than cores)
> 
> - 8GB RAM and 2 NICs (one set to NAT for management and the other set to
> a custom network and configured for sniffing)
> 
> - running our latest Security Onion ISO image which contains Zeek 3.0.1
> (I've also duplicated this behavior using Zeek 3.0.2 compiled manually):
> https://blog.securityonion.net/2020/02/security-onion-160464-iso-image-now.html
> 
> - run sosetup-minimal and choose Evaluation Mode
> 
> - once Setup is complete, create some traffic on the sniffing interface:
> while :; do sudo so-replay; done
> 
> - on my box, Zeek normally runs at about 10% to 20% CPU usage when
> running so-replay but after a certain period of time (seems
> inconsistent, could be minutes or over an hour), Zeek will go to 100%
> CPU usage and remain there even if you kill the so-replay while loop
> from above
> 
> - you can restart Zeek with "sudo so-zeek-restart" and it will go back
> to normal operation and normal CPU usage, but after a while of
> processing traffic it will go back to 100% CPU usage
> 
> - as mentioned above, you can also download Zeek 3.0.2 and compile it
> manually according
> to?https://docs.zeek.org/en/v3.0.2/install/install.html and duplicate
> the issue there, so this would seem to rule out any possible issues with
> our Zeek package or scripts
> 
> Please let me know if I can provide any further information to assist in
> duplicating and pinpointing this issue.
> 
> Thanks!
> 
> 
> On Tue, Feb 25, 2020 at 6:41 PM Pete Nelson <petiepooo at gmail.com
> <mailto:petiepooo at gmail.com>> wrote:
> 
>     Thanks, Jon.
> 
>     I'll try to digest those links and dig into the code.? Unfortunately,
>     it seems running strace on the process keeps it from occurring...? I
>     may try to get dtrace working in place, but I need to improve my lab
>     setup first before I go too crazy.
>     --
>     Pete
>     _______________________________________________
>     Zeek mailing list
>     zeek at zeek.org <mailto:zeek at zeek.org>
>     http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> 
> 
> 
> -- 
> Doug Burks
> 
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200304/a1dec9f5/attachment.bin 

From doug.burks at gmail.com  Wed Mar  4 12:03:40 2020
From: doug.burks at gmail.com (Doug Burks)
Date: Wed, 4 Mar 2020 15:03:40 -0500
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <95f56d8d-9791-c86f-716c-098ad2423d7a@etnetera.cz>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
	<CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
	<CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>
	<CAPfnCuizcGsP1jhAafqmkRn-Mq=uMNq2s3+jQf8vkkQHeLg3fQ@mail.gmail.com>
	<CAAYFXLmwMZUep3o-N02fWhx6ybcgbT6htfbdyRKoNd__fdz2hQ@mail.gmail.com>
	<CAMzgZ0KoVGu2KmA7u06B9FmbXbhA2h0toxgNDUVGHbz-yc1UCQ@mail.gmail.com>
	<CAAYFXL=bZYmFCiRKHswDY4xBJ5en-kBkWoOoc8Pn4kn=XdLpfQ@mail.gmail.com>
	<CAK8kjrCF_AFZOboTCPZ21x8hGDcwPGGa_KboMA9Bm85pzV_3-A@mail.gmail.com>
	<95f56d8d-9791-c86f-716c-098ad2423d7a@etnetera.cz>
Message-ID: <CAK8kjrAbAo2h=aWpuMgghj8OC2Qtw06HUOg_sQ-rnK+AVq+OZA@mail.gmail.com>

Hi Petr,

It does sound similar to issue 838.  However, please note that I was able
to duplicate this issue on a manual compile of Zeek 3.0.2 with no
additional scripts loaded.

On Wed, Mar 4, 2020 at 2:57 PM Petr Medonos <petr.medonos at etnetera.cz>
wrote:

> Hi all,
> not sure yet, but looks very similar to issue I started here
> https://github.com/zeek/zeek/issues/838. I will take a look what Zeek
> scripts are enabled by default in SO to correlate these issues.
>
>
> Petr
>
>
> On 04. 03. 20 17:35, Doug Burks wrote:
> > I've been able to duplicate this issue so I'm passing along some notes
> > in the hope that others are able to duplicate the issue as well and
> > perhaps pinpoint what's going on.
> >
> > - I've seen this issue on both physical boxes and virtual machines and
> > on both single CPU socket and multiple CPU socket systems
> >
> > - I've been able to trigger this issue fairly consistently using VMware
> > Workstation with the VM set to 4 processors (seems easier to duplicate
> > when using processors rather than cores)
> >
> > - 8GB RAM and 2 NICs (one set to NAT for management and the other set to
> > a custom network and configured for sniffing)
> >
> > - running our latest Security Onion ISO image which contains Zeek 3.0.1
> > (I've also duplicated this behavior using Zeek 3.0.2 compiled manually):
> >
> https://blog.securityonion.net/2020/02/security-onion-160464-iso-image-now.html
> >
> > - run sosetup-minimal and choose Evaluation Mode
> >
> > - once Setup is complete, create some traffic on the sniffing interface:
> > while :; do sudo so-replay; done
> >
> > - on my box, Zeek normally runs at about 10% to 20% CPU usage when
> > running so-replay but after a certain period of time (seems
> > inconsistent, could be minutes or over an hour), Zeek will go to 100%
> > CPU usage and remain there even if you kill the so-replay while loop
> > from above
> >
> > - you can restart Zeek with "sudo so-zeek-restart" and it will go back
> > to normal operation and normal CPU usage, but after a while of
> > processing traffic it will go back to 100% CPU usage
> >
> > - as mentioned above, you can also download Zeek 3.0.2 and compile it
> > manually according
> > to https://docs.zeek.org/en/v3.0.2/install/install.html and duplicate
> > the issue there, so this would seem to rule out any possible issues with
> > our Zeek package or scripts
> >
> > Please let me know if I can provide any further information to assist in
> > duplicating and pinpointing this issue.
> >
> > Thanks!
> >
> >
> > On Tue, Feb 25, 2020 at 6:41 PM Pete Nelson <petiepooo at gmail.com
> > <mailto:petiepooo at gmail.com>> wrote:
> >
> >     Thanks, Jon.
> >
> >     I'll try to digest those links and dig into the code.  Unfortunately,
> >     it seems running strace on the process keeps it from occurring...  I
> >     may try to get dtrace working in place, but I need to improve my lab
> >     setup first before I go too crazy.
> >     --
> >     Pete
> >     _______________________________________________
> >     Zeek mailing list
> >     zeek at zeek.org <mailto:zeek at zeek.org>
> >     http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> >
> >
> >
> > --
> > Doug Burks
> >
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> >
>
>

-- 
Doug Burks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200304/f16dbf0d/attachment.html 

From jsiwek at corelight.com  Wed Mar  4 12:06:11 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Wed, 4 Mar 2020 12:06:11 -0800
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAK8kjrCF_AFZOboTCPZ21x8hGDcwPGGa_KboMA9Bm85pzV_3-A@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
	<CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
	<CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>
	<CAPfnCuizcGsP1jhAafqmkRn-Mq=uMNq2s3+jQf8vkkQHeLg3fQ@mail.gmail.com>
	<CAAYFXLmwMZUep3o-N02fWhx6ybcgbT6htfbdyRKoNd__fdz2hQ@mail.gmail.com>
	<CAMzgZ0KoVGu2KmA7u06B9FmbXbhA2h0toxgNDUVGHbz-yc1UCQ@mail.gmail.com>
	<CAAYFXL=bZYmFCiRKHswDY4xBJ5en-kBkWoOoc8Pn4kn=XdLpfQ@mail.gmail.com>
	<CAK8kjrCF_AFZOboTCPZ21x8hGDcwPGGa_KboMA9Bm85pzV_3-A@mail.gmail.com>
Message-ID: <CAMzgZ0Lb43TfNZkdXUEVLvM0_F=xMJ-1FXZfrsfe61LEAycPVg@mail.gmail.com>

On Wed, Mar 4, 2020 at 8:36 AM Doug Burks <doug.burks at gmail.com> wrote:

> - I've been able to trigger this issue fairly consistently using VMware Workstation with the VM set to 4 processors (seems easier to duplicate when using processors rather than cores)

Thanks, I've reproduced with a similar configuration.  First thing I
notice is `perf top` showing much time spent in
`threading::Manager::NextTimestamp`, so now off to try and understand
how it can get into that state.

> (I've also duplicated this behavior using Zeek 3.0.2 compiled manually):

I'm expecting to have to go this way either to add more
instrumentation or test potential patches, so for consistency, can you
provide the exact commands used from the point of ./configure until
point of running/restarting zeek ?


- Jon

From doug.burks at gmail.com  Wed Mar  4 12:15:49 2020
From: doug.burks at gmail.com (Doug Burks)
Date: Wed, 4 Mar 2020 15:15:49 -0500
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAMzgZ0Lb43TfNZkdXUEVLvM0_F=xMJ-1FXZfrsfe61LEAycPVg@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
	<CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
	<CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>
	<CAPfnCuizcGsP1jhAafqmkRn-Mq=uMNq2s3+jQf8vkkQHeLg3fQ@mail.gmail.com>
	<CAAYFXLmwMZUep3o-N02fWhx6ybcgbT6htfbdyRKoNd__fdz2hQ@mail.gmail.com>
	<CAMzgZ0KoVGu2KmA7u06B9FmbXbhA2h0toxgNDUVGHbz-yc1UCQ@mail.gmail.com>
	<CAAYFXL=bZYmFCiRKHswDY4xBJ5en-kBkWoOoc8Pn4kn=XdLpfQ@mail.gmail.com>
	<CAK8kjrCF_AFZOboTCPZ21x8hGDcwPGGa_KboMA9Bm85pzV_3-A@mail.gmail.com>
	<CAMzgZ0Lb43TfNZkdXUEVLvM0_F=xMJ-1FXZfrsfe61LEAycPVg@mail.gmail.com>
Message-ID: <CAK8kjrDtba0PfhH1Q-XM+2OiREGRCJirv5y-oUzdr8wBOo-VDw@mail.gmail.com>

Hi Jon,

Replies inline.

On Wed, Mar 4, 2020 at 3:06 PM Jon Siwek <jsiwek at corelight.com> wrote:

> On Wed, Mar 4, 2020 at 8:36 AM Doug Burks <doug.burks at gmail.com> wrote:
>
> > - I've been able to trigger this issue fairly consistently using VMware
> Workstation with the VM set to 4 processors (seems easier to duplicate when
> using processors rather than cores)
>
> Thanks, I've reproduced with a similar configuration.  First thing I
> notice is `perf top` showing much time spent in
> `threading::Manager::NextTimestamp`, so now off to try and understand
> how it can get into that state.
>

I'm glad you were able to reproduce this!


>
> > (I've also duplicated this behavior using Zeek 3.0.2 compiled manually):
>
> I'm expecting to have to go this way either to add more
> instrumentation or test potential patches, so for consistency, can you
> provide the exact commands used from the point of ./configure until
> point of running/restarting zeek ?
>
>
> - Jon
>

Going off of memory, but it should have been rather standard:

sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev
python-dev swig zlib1g-dev
./configure
make
sudo make install
modify /usr/local/zeek/etc/node.cfg and configure the standalone stanza to
sniff from the sniffing interface
sudo /usr/local/zeek/bin/zeekctl deploy

Please let me know if you need anything else.

Thanks!

-- 
Doug Burks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200304/1965c949/attachment.html 

From doug.burks at gmail.com  Wed Mar  4 12:43:52 2020
From: doug.burks at gmail.com (Doug Burks)
Date: Wed, 4 Mar 2020 15:43:52 -0500
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAK8kjrDtba0PfhH1Q-XM+2OiREGRCJirv5y-oUzdr8wBOo-VDw@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
	<CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
	<CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>
	<CAPfnCuizcGsP1jhAafqmkRn-Mq=uMNq2s3+jQf8vkkQHeLg3fQ@mail.gmail.com>
	<CAAYFXLmwMZUep3o-N02fWhx6ybcgbT6htfbdyRKoNd__fdz2hQ@mail.gmail.com>
	<CAMzgZ0KoVGu2KmA7u06B9FmbXbhA2h0toxgNDUVGHbz-yc1UCQ@mail.gmail.com>
	<CAAYFXL=bZYmFCiRKHswDY4xBJ5en-kBkWoOoc8Pn4kn=XdLpfQ@mail.gmail.com>
	<CAK8kjrCF_AFZOboTCPZ21x8hGDcwPGGa_KboMA9Bm85pzV_3-A@mail.gmail.com>
	<CAMzgZ0Lb43TfNZkdXUEVLvM0_F=xMJ-1FXZfrsfe61LEAycPVg@mail.gmail.com>
	<CAK8kjrDtba0PfhH1Q-XM+2OiREGRCJirv5y-oUzdr8wBOo-VDw@mail.gmail.com>
Message-ID: <CAK8kjrCOEm-XTu_JzQ70qcbXeayzVVeahw5ENHuXWoZWG5+T9w@mail.gmail.com>

On Wed, Mar 4, 2020 at 3:15 PM Doug Burks <doug.burks at gmail.com> wrote:

> Hi Jon,
>
> Replies inline.
>
> On Wed, Mar 4, 2020 at 3:06 PM Jon Siwek <jsiwek at corelight.com> wrote:
>
>> On Wed, Mar 4, 2020 at 8:36 AM Doug Burks <doug.burks at gmail.com> wrote:
>>
>> > - I've been able to trigger this issue fairly consistently using VMware
>> Workstation with the VM set to 4 processors (seems easier to duplicate when
>> using processors rather than cores)
>>
>> Thanks, I've reproduced with a similar configuration.  First thing I
>> notice is `perf top` showing much time spent in
>> `threading::Manager::NextTimestamp`, so now off to try and understand
>> how it can get into that state.
>
>
Just a quick note to confirm that I'm seeing much time spent in
`threading::Manager::NextTimestamp` here as well.

-- 
Doug Burks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200304/96f3b136/attachment-0001.html 

From michalpurzynski1 at gmail.com  Wed Mar  4 13:48:19 2020
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Wed, 4 Mar 2020 13:48:19 -0800
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAK8kjrCOEm-XTu_JzQ70qcbXeayzVVeahw5ENHuXWoZWG5+T9w@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
	<CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
	<CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>
	<CAPfnCuizcGsP1jhAafqmkRn-Mq=uMNq2s3+jQf8vkkQHeLg3fQ@mail.gmail.com>
	<CAAYFXLmwMZUep3o-N02fWhx6ybcgbT6htfbdyRKoNd__fdz2hQ@mail.gmail.com>
	<CAMzgZ0KoVGu2KmA7u06B9FmbXbhA2h0toxgNDUVGHbz-yc1UCQ@mail.gmail.com>
	<CAAYFXL=bZYmFCiRKHswDY4xBJ5en-kBkWoOoc8Pn4kn=XdLpfQ@mail.gmail.com>
	<CAK8kjrCF_AFZOboTCPZ21x8hGDcwPGGa_KboMA9Bm85pzV_3-A@mail.gmail.com>
	<CAMzgZ0Lb43TfNZkdXUEVLvM0_F=xMJ-1FXZfrsfe61LEAycPVg@mail.gmail.com>
	<CAK8kjrDtba0PfhH1Q-XM+2OiREGRCJirv5y-oUzdr8wBOo-VDw@mail.gmail.com>
	<CAK8kjrCOEm-XTu_JzQ70qcbXeayzVVeahw5ENHuXWoZWG5+T9w@mail.gmail.com>
Message-ID: <CAJ6bFK1aoVRA93q0dE8HWBG+pu9aAR1PjWV7qxvrvdKYEtZgQw@mail.gmail.com>

Can we disable scripts and run Zeek with a minimalist configuration and see
if result changes?

The timestamp thing looks like a packet timestamp?

Do use you libpcap by chance?

On Wed, Mar 4, 2020 at 12:46 PM Doug Burks <doug.burks at gmail.com> wrote:

> On Wed, Mar 4, 2020 at 3:15 PM Doug Burks <doug.burks at gmail.com> wrote:
>
>> Hi Jon,
>>
>> Replies inline.
>>
>> On Wed, Mar 4, 2020 at 3:06 PM Jon Siwek <jsiwek at corelight.com> wrote:
>>
>>> On Wed, Mar 4, 2020 at 8:36 AM Doug Burks <doug.burks at gmail.com> wrote:
>>>
>>> > - I've been able to trigger this issue fairly consistently using
>>> VMware Workstation with the VM set to 4 processors (seems easier to
>>> duplicate when using processors rather than cores)
>>>
>>> Thanks, I've reproduced with a similar configuration.  First thing I
>>> notice is `perf top` showing much time spent in
>>> `threading::Manager::NextTimestamp`, so now off to try and understand
>>> how it can get into that state.
>>
>>
> Just a quick note to confirm that I'm seeing much time spent in
> `threading::Manager::NextTimestamp` here as well.
>
> --
> Doug Burks
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200304/311834f7/attachment.html 

From jlay at slave-tothe-box.net  Wed Mar  4 14:04:06 2020
From: jlay at slave-tothe-box.net (James Lay)
Date: Wed, 04 Mar 2020 15:04:06 -0700
Subject: [Zeek] First attempt to upgrade to 3: Multiple interfaces
In-Reply-To: <CAPfnCujeMWeykk651-ELeoN_WAtEC85vzpaZhW3LrYg=+799KA@mail.gmail.com>
References: <43959c9db49e46e8ea5dbcaaec46d2e73a494424.camel@slave-tothe-box.net>
	<29313A40-9D2A-4843-B9C0-2739CDB6822E@gmail.com>
	<CAPfnCujeMWeykk651-ELeoN_WAtEC85vzpaZhW3LrYg=+799KA@mail.gmail.com>
Message-ID: <1bb195d7377c3de42abc148360ff20e0@slave-tothe-box.net>

Thanks again...resources look great:

PID   USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ 
COMMAND
11360 root      20   0  830360 258148 151156 S   0.3  3.2   7:42.42 zeek
11362 root      20   0  827852 254540 150568 S   0.3  3.2  10:19.57 zeek

Also pleased to note that the trickle of traffic cpu usage has been 
fixed so thumbs up to that.

James

On 2020-03-03 19:50, Justin Azoff wrote:
> On Tue, Mar 3, 2020 at 9:39 PM Micha? Purzy?ski
> <michalpurzynski1 at gmail.com> wrote:
> 
>> From what you?re describing, you?re running Zeek on a resource
>> constraints device.
>> 
>> You might want to actually use the newest version - not sure where
>> Justin fixes for the packet processing loop went in, but they should
>> help a lot.
> 
> Oh hah, I don't need any credit for all of that work.  I had an old
> patch against 2.4/5/6 that would relax the polling loop interval so it
> wouldn't eat up a lot of cpu on smaller devices, but it wasn't a fix.
> Tim completely rewrote the IO loop so now zeek on a raspberry pi
> should use 0% cpu when there's no traffic.
>  --
> 
> Justin

From don.thomas.cissp at gmail.com  Wed Mar  4 16:08:53 2020
From: don.thomas.cissp at gmail.com (Don Thomas)
Date: Wed, 4 Mar 2020 16:08:53 -0800
Subject: [Zeek] First attempt to upgrade to 3: Multiple interfaces
In-Reply-To: <5C741D97-A6B8-4ECA-915B-3A6BA4D99E13@corelight.com>
References: <c8cb17b05da1d98434bf6ec14e5a035b@slave-tothe-box.net>
	<5C741D97-A6B8-4ECA-915B-3A6BA4D99E13@corelight.com>
Message-ID: <CABYfsedv3=70W6=1rsFVPEW3LgG9Hd_GGpkSr5_L_bc+QBTe9Q@mail.gmail.com>

Please bring back the multiple interface option.   I have two Gigamon's
(each has a 10 G interface ) Feeding Zeek on our IDS.

Or else Zeek is a No Go for me as well.

Yes... I need the multiple interface option !

Thank you,
*Don Thomas, CISSP, CISA*



On Tue, Mar 3, 2020 at 11:30 AM Tim Wojtulewicz <tim at corelight.com> wrote:

> If you don?t really need the latest and greatest cutting edge changes to
> 3.1, version 3.0.x still supports multiple interfaces. That feature was
> removed in 3.1 due to the wide changes to the IO Loop architecture, and
> you?re honestly the first user I?ve heard from that has noticed it missing.
> It was removed to make that work easier to accomplish, but we can certainly
> investigate bringing it back if there?s enough of a use case for it.
>
> Tim
>
> > On Mar 3, 2020, at 12:03 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> >
> > Welp...out of luck so far:
> >
> > /opt/zeek/bin/zeek -C -i eth0 -i eth1 --filter '<redacted>' local
> > "Site::local_nets += { 192.168.1.0/24 }"
> >
> > gets me:
> >
> > ERROR: Only a single interface option (-i) is allowed.
> >
> > I didn't have this issue with 2.  Any reason why only one interface is
> > allowed now?  Unless something radical has changed with the resources
> > that zeekctl uses I have no desire to use it.  I'm dead in the water
> > with Zeek as of now.  Thank you.
> >
> > James
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200304/d11dd862/attachment.html 

From johanna at icir.org  Wed Mar  4 17:18:21 2020
From: johanna at icir.org (Johanna Amann)
Date: Wed, 04 Mar 2020 17:18:21 -0800
Subject: [Zeek] First attempt to upgrade to 3: Multiple interfaces
In-Reply-To: <CABYfsedv3=70W6=1rsFVPEW3LgG9Hd_GGpkSr5_L_bc+QBTe9Q@mail.gmail.com>
References: <c8cb17b05da1d98434bf6ec14e5a035b@slave-tothe-box.net>
	<5C741D97-A6B8-4ECA-915B-3A6BA4D99E13@corelight.com>
	<CABYfsedv3=70W6=1rsFVPEW3LgG9Hd_GGpkSr5_L_bc+QBTe9Q@mail.gmail.com>
Message-ID: <1837831E-946D-41D1-8FC1-B2E6D0C164F6@icir.org>

Hi Don,

just to pull over the questions from Slack here:

could you please detail a little bit how you are currently running Zeek.

Since it sounds like you are handling a lot of traffic - you typically 
should already have to run zeek in cluster mode using zeekctl/broctl. At 
which point this problem does not apply anymore - since in a cluster you 
can have a zeek process (or - typically - several zeek processes) for 
each interface.

If you are currently running a single standalone zeek process on several 
10G interfaces - did you check your packet loss? Because it seems 
unlikely to me that this setup is not loosing a ton of traffic.

Thank you :)
  Johanna

On 4 Mar 2020, at 16:08, Don Thomas wrote:

> Please bring back the multiple interface option.   I have two 
> Gigamon's
> (each has a 10 G interface ) Feeding Zeek on our IDS.
>
> Or else Zeek is a No Go for me as well.
>
> Yes... I need the multiple interface option !
>
> Thank you,
> *Don Thomas, CISSP, CISA*
>
>
>
> On Tue, Mar 3, 2020 at 11:30 AM Tim Wojtulewicz <tim at corelight.com> 
> wrote:
>
>> If you don?t really need the latest and greatest cutting edge 
>> changes to
>> 3.1, version 3.0.x still supports multiple interfaces. That feature 
>> was
>> removed in 3.1 due to the wide changes to the IO Loop architecture, 
>> and
>> you?re honestly the first user I?ve heard from that has noticed 
>> it missing.
>> It was removed to make that work easier to accomplish, but we can 
>> certainly
>> investigate bringing it back if there?s enough of a use case for 
>> it.
>>
>> Tim
>>
>>> On Mar 3, 2020, at 12:03 PM, James Lay <jlay at slave-tothe-box.net> 
>>> wrote:
>>>
>>> Welp...out of luck so far:
>>>
>>> /opt/zeek/bin/zeek -C -i eth0 -i eth1 --filter '<redacted>' local
>>> "Site::local_nets += { 192.168.1.0/24 }"
>>>
>>> gets me:
>>>
>>> ERROR: Only a single interface option (-i) is allowed.
>>>
>>> I didn't have this issue with 2.  Any reason why only one interface 
>>> is
>>> allowed now?  Unless something radical has changed with the 
>>> resources
>>> that zeekctl uses I have no desire to use it.  I'm dead in the water
>>> with Zeek as of now.  Thank you.
>>>
>>> James
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>>
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

From jsiwek at corelight.com  Wed Mar  4 20:48:15 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Wed, 4 Mar 2020 20:48:15 -0800
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAK8kjrCOEm-XTu_JzQ70qcbXeayzVVeahw5ENHuXWoZWG5+T9w@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
	<CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
	<CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>
	<CAPfnCuizcGsP1jhAafqmkRn-Mq=uMNq2s3+jQf8vkkQHeLg3fQ@mail.gmail.com>
	<CAAYFXLmwMZUep3o-N02fWhx6ybcgbT6htfbdyRKoNd__fdz2hQ@mail.gmail.com>
	<CAMzgZ0KoVGu2KmA7u06B9FmbXbhA2h0toxgNDUVGHbz-yc1UCQ@mail.gmail.com>
	<CAAYFXL=bZYmFCiRKHswDY4xBJ5en-kBkWoOoc8Pn4kn=XdLpfQ@mail.gmail.com>
	<CAK8kjrCF_AFZOboTCPZ21x8hGDcwPGGa_KboMA9Bm85pzV_3-A@mail.gmail.com>
	<CAMzgZ0Lb43TfNZkdXUEVLvM0_F=xMJ-1FXZfrsfe61LEAycPVg@mail.gmail.com>
	<CAK8kjrDtba0PfhH1Q-XM+2OiREGRCJirv5y-oUzdr8wBOo-VDw@mail.gmail.com>
	<CAK8kjrCOEm-XTu_JzQ70qcbXeayzVVeahw5ENHuXWoZWG5+T9w@mail.gmail.com>
Message-ID: <CAMzgZ0+Fsj6v02oPL-tksQuB00NrnZy8shbY9j1Fm5NZm16HWw@mail.gmail.com>

On Wed, Mar 4, 2020 at 12:44 PM Doug Burks <doug.burks at gmail.com> wrote:

> Just a quick note to confirm that I'm seeing much time spent in `threading::Manager::NextTimestamp` here as well.

That didn't turn out to be directly related, but I did ultimately hunt
down a race condition in Broker as explanation for it getting stuck at
full CPU load.  This is the incoming patch:

    https://github.com/zeek/broker/pull/97

Thanks for the assists in tracking this down.  Any further help in
verifying the patch would also be great.

- Jon

From robin at corelight.com  Thu Mar  5 00:10:25 2020
From: robin at corelight.com (Robin Sommer)
Date: Thu, 5 Mar 2020 08:10:25 +0000
Subject: [Zeek] First attempt to upgrade to 3: Multiple interfaces
In-Reply-To: <1837831E-946D-41D1-8FC1-B2E6D0C164F6@icir.org>
References: <c8cb17b05da1d98434bf6ec14e5a035b@slave-tothe-box.net>
	<5C741D97-A6B8-4ECA-915B-3A6BA4D99E13@corelight.com>
	<CABYfsedv3=70W6=1rsFVPEW3LgG9Hd_GGpkSr5_L_bc+QBTe9Q@mail.gmail.com>
	<1837831E-946D-41D1-8FC1-B2E6D0C164F6@icir.org>
Message-ID: <20200305081025.GA16330@corelight.com>

Btw, just for the record, before doing such breaking changes we tend
to ask here on mailing list for feedback. In this case, see here:
http://mailman.icsi.berkeley.edu/pipermail/zeek/2019-August/014484.html.
I don't think I heard any concerns in response to that.

Robin

On Wed, Mar 04, 2020 at 17:18 -0800, Johanna Amann wrote:

> Hi Don,
> 
> just to pull over the questions from Slack here:
> 
> could you please detail a little bit how you are currently running Zeek.
> 
> Since it sounds like you are handling a lot of traffic - you typically 
> should already have to run zeek in cluster mode using zeekctl/broctl. At 
> which point this problem does not apply anymore - since in a cluster you 
> can have a zeek process (or - typically - several zeek processes) for 
> each interface.
> 
> If you are currently running a single standalone zeek process on several 
> 10G interfaces - did you check your packet loss? Because it seems 
> unlikely to me that this setup is not loosing a ton of traffic.
> 
> Thank you :)
>   Johanna
> 
> On 4 Mar 2020, at 16:08, Don Thomas wrote:
> 
> > Please bring back the multiple interface option.   I have two 
> > Gigamon's
> > (each has a 10 G interface ) Feeding Zeek on our IDS.
> >
> > Or else Zeek is a No Go for me as well.
> >
> > Yes... I need the multiple interface option !
> >
> > Thank you,
> > *Don Thomas, CISSP, CISA*
> >
> >
> >
> > On Tue, Mar 3, 2020 at 11:30 AM Tim Wojtulewicz <tim at corelight.com> 
> > wrote:
> >
> >> If you don?t really need the latest and greatest cutting edge 
> >> changes to
> >> 3.1, version 3.0.x still supports multiple interfaces. That feature 
> >> was
> >> removed in 3.1 due to the wide changes to the IO Loop architecture, 
> >> and
> >> you?re honestly the first user I?ve heard from that has noticed 
> >> it missing.
> >> It was removed to make that work easier to accomplish, but we can 
> >> certainly
> >> investigate bringing it back if there?s enough of a use case for 
> >> it.
> >>
> >> Tim
> >>
> >>> On Mar 3, 2020, at 12:03 PM, James Lay <jlay at slave-tothe-box.net> 
> >>> wrote:
> >>>
> >>> Welp...out of luck so far:
> >>>
> >>> /opt/zeek/bin/zeek -C -i eth0 -i eth1 --filter '<redacted>' local
> >>> "Site::local_nets += { 192.168.1.0/24 }"
> >>>
> >>> gets me:
> >>>
> >>> ERROR: Only a single interface option (-i) is allowed.
> >>>
> >>> I didn't have this issue with 2.  Any reason why only one interface 
> >>> is
> >>> allowed now?  Unless something radical has changed with the 
> >>> resources
> >>> that zeekctl uses I have no desire to use it.  I'm dead in the water
> >>> with Zeek as of now.  Thank you.
> >>>
> >>> James
> >>> _______________________________________________
> >>> Zeek mailing list
> >>> zeek at zeek.org
> >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> >>
> >>
> >> _______________________________________________
> >> Zeek mailing list
> >> zeek at zeek.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


-- 
Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com

From doug.burks at gmail.com  Thu Mar  5 06:44:04 2020
From: doug.burks at gmail.com (Doug Burks)
Date: Thu, 5 Mar 2020 09:44:04 -0500
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAMzgZ0+Fsj6v02oPL-tksQuB00NrnZy8shbY9j1Fm5NZm16HWw@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
	<CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
	<CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>
	<CAPfnCuizcGsP1jhAafqmkRn-Mq=uMNq2s3+jQf8vkkQHeLg3fQ@mail.gmail.com>
	<CAAYFXLmwMZUep3o-N02fWhx6ybcgbT6htfbdyRKoNd__fdz2hQ@mail.gmail.com>
	<CAMzgZ0KoVGu2KmA7u06B9FmbXbhA2h0toxgNDUVGHbz-yc1UCQ@mail.gmail.com>
	<CAAYFXL=bZYmFCiRKHswDY4xBJ5en-kBkWoOoc8Pn4kn=XdLpfQ@mail.gmail.com>
	<CAK8kjrCF_AFZOboTCPZ21x8hGDcwPGGa_KboMA9Bm85pzV_3-A@mail.gmail.com>
	<CAMzgZ0Lb43TfNZkdXUEVLvM0_F=xMJ-1FXZfrsfe61LEAycPVg@mail.gmail.com>
	<CAK8kjrDtba0PfhH1Q-XM+2OiREGRCJirv5y-oUzdr8wBOo-VDw@mail.gmail.com>
	<CAK8kjrCOEm-XTu_JzQ70qcbXeayzVVeahw5ENHuXWoZWG5+T9w@mail.gmail.com>
	<CAMzgZ0+Fsj6v02oPL-tksQuB00NrnZy8shbY9j1Fm5NZm16HWw@mail.gmail.com>
Message-ID: <CAK8kjrAxt7ce8WbteQS2bG2etc5fTgepQWHXRtx=WnOtv8JsCQ@mail.gmail.com>

Good morning Jon,

I applied the relevant patch to the Zeek 3.0.2 tarball, compiled, and have
been running for the last 3 hours or so with the same pcap replay loop
described previously.  Zeek still seems to be averaging 10% to 20% CPU
usage, so the patch is looking good so far.

Would you recommend that we apply the patch to our Zeek 3.0.2 package OR
hold off on 3.0.2 and wait for 3.0.3 assuming it will be included there?

Thanks!

On Wed, Mar 4, 2020 at 11:48 PM Jon Siwek <jsiwek at corelight.com> wrote:

> On Wed, Mar 4, 2020 at 12:44 PM Doug Burks <doug.burks at gmail.com> wrote:
>
> > Just a quick note to confirm that I'm seeing much time spent in
> `threading::Manager::NextTimestamp` here as well.
>
> That didn't turn out to be directly related, but I did ultimately hunt
> down a race condition in Broker as explanation for it getting stuck at
> full CPU load.  This is the incoming patch:
>
>     https://github.com/zeek/broker/pull/97
>
> Thanks for the assists in tracking this down.  Any further help in
> verifying the patch would also be great.
>
> - Jon
>


-- 
Doug Burks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200305/448be63e/attachment.html 

From akgraner at corelight.com  Thu Mar  5 08:31:30 2020
From: akgraner at corelight.com (Amber Graner)
Date: Thu, 5 Mar 2020 11:31:30 -0500
Subject: [Zeek] Zeek Events Calendar Invites and Reminders Opt-In
Message-ID: <CAJhOzupzsmrdd73-tjb+w2zMvdL_ft04NGmcUAydoBL_W6pZVQ@mail.gmail.com>

Hi all,

Since we will no longer be adding the Zeek mailing list to the calendar
invites, I've created a form for folks to opt-in to receiving calendar
invites and reminders.

If you would like to continue to receive the calendar invites and reminders
for the ASK THE ZEEKSPERTS webinars and/or the Monthly Community calls
please take a moment to add your information to the form (Link below)

https://www.surveymonkey.com/r/X5W5YQZ

If you've already accepted the current invites then you'll continue to get
them.  I'll remove everyone else.

Thanks in advance.

With gratitude,
~Amber
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200305/766800b4/attachment.html 

From jsiwek at corelight.com  Thu Mar  5 10:23:27 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 5 Mar 2020 10:23:27 -0800
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAK8kjrAxt7ce8WbteQS2bG2etc5fTgepQWHXRtx=WnOtv8JsCQ@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
	<CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
	<CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>
	<CAPfnCuizcGsP1jhAafqmkRn-Mq=uMNq2s3+jQf8vkkQHeLg3fQ@mail.gmail.com>
	<CAAYFXLmwMZUep3o-N02fWhx6ybcgbT6htfbdyRKoNd__fdz2hQ@mail.gmail.com>
	<CAMzgZ0KoVGu2KmA7u06B9FmbXbhA2h0toxgNDUVGHbz-yc1UCQ@mail.gmail.com>
	<CAAYFXL=bZYmFCiRKHswDY4xBJ5en-kBkWoOoc8Pn4kn=XdLpfQ@mail.gmail.com>
	<CAK8kjrCF_AFZOboTCPZ21x8hGDcwPGGa_KboMA9Bm85pzV_3-A@mail.gmail.com>
	<CAMzgZ0Lb43TfNZkdXUEVLvM0_F=xMJ-1FXZfrsfe61LEAycPVg@mail.gmail.com>
	<CAK8kjrDtba0PfhH1Q-XM+2OiREGRCJirv5y-oUzdr8wBOo-VDw@mail.gmail.com>
	<CAK8kjrCOEm-XTu_JzQ70qcbXeayzVVeahw5ENHuXWoZWG5+T9w@mail.gmail.com>
	<CAMzgZ0+Fsj6v02oPL-tksQuB00NrnZy8shbY9j1Fm5NZm16HWw@mail.gmail.com>
	<CAK8kjrAxt7ce8WbteQS2bG2etc5fTgepQWHXRtx=WnOtv8JsCQ@mail.gmail.com>
Message-ID: <CAMzgZ0LqV7cEMCFa-8yArjXKxPGLaThJ6nLvwrx3LH=zaGvxhg@mail.gmail.com>

On Thu, Mar 5, 2020 at 6:44 AM Doug Burks <doug.burks at gmail.com> wrote:

> Would you recommend that we apply the patch to our Zeek 3.0.2 package OR hold off on 3.0.2 and wait for 3.0.3 assuming it will be included there?

Hi Doug, I do now have this on the schedule for fixing in a Zeek 3.0.3
(and 3.1.1), but not sure what time frame to expect those releases.
I'll suggest to the team to aim at doing one in ~2-3 weeks.  If you'd
like to patch ASAP, you may still want to wait at least until that
Pull Request gets a code review and merged into the master branch.

- Jon

From Kayode_Enwerem at ao.uscourts.gov  Thu Mar  5 10:43:16 2020
From: Kayode_Enwerem at ao.uscourts.gov (Kayode Enwerem)
Date: Thu, 5 Mar 2020 18:43:16 +0000
Subject: [Zeek] Errors trying to implement the detection on CVE-2020-0601
Message-ID: <BY5PR09MB53771FEE30588889A23845CBA1E20@BY5PR09MB5377.namprd09.prod.outlook.com>

Hello,

I am trying to implement the detection of CVE-2020-0601 with zeek (https://blog.zeek.org/2020/01/detecting-cve-2020-0601-with-zeek.html) using the first package (https://github.com/0xxon/cve-2020-0601) but I keep encountering some errors.

Version for bro in my environment: bro version 2.5.5

First thing I did was add this to our local.bro file: redef CVE_2020_0601::log_certs = T;

But when I ran "broctl check" I got the following error message: error in /usr/local/bro/share/bro/site/local.bro, line 13: "redef" used but not previously defined (CVE_             2020_0601::log_certs)

So I created the following file in "share/bro/base/frameworks/notice/cve-2020-0601.bro" and added the script from: https://github.com/0xxon/cve-2020-0601/blob/master/scripts/cve-2020-0601.bro

And also edited the following file "share/bro/base/frameworks/notice/__load__.bro" and added: @load ./cve-2020-0601

Now when I run "broctl check" I am getting the following error message:
error in /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro, line 5: syntax error, at or near "option"

When I comment out line 5 line I get:
error in /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro, line 26: unknown identifier Version::at_least, at or near "Version::at_least"

When I comment out line 26 I get:
error in /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro, line 35: unknown identifier f, at or near "f"

Can someone please help me with this? Am I setting it up right?

Thanks in advance.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200305/bff62660/attachment.html 

From doug.burks at gmail.com  Thu Mar  5 10:57:17 2020
From: doug.burks at gmail.com (Doug Burks)
Date: Thu, 5 Mar 2020 13:57:17 -0500
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAMzgZ0LqV7cEMCFa-8yArjXKxPGLaThJ6nLvwrx3LH=zaGvxhg@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
	<CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
	<CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>
	<CAPfnCuizcGsP1jhAafqmkRn-Mq=uMNq2s3+jQf8vkkQHeLg3fQ@mail.gmail.com>
	<CAAYFXLmwMZUep3o-N02fWhx6ybcgbT6htfbdyRKoNd__fdz2hQ@mail.gmail.com>
	<CAMzgZ0KoVGu2KmA7u06B9FmbXbhA2h0toxgNDUVGHbz-yc1UCQ@mail.gmail.com>
	<CAAYFXL=bZYmFCiRKHswDY4xBJ5en-kBkWoOoc8Pn4kn=XdLpfQ@mail.gmail.com>
	<CAK8kjrCF_AFZOboTCPZ21x8hGDcwPGGa_KboMA9Bm85pzV_3-A@mail.gmail.com>
	<CAMzgZ0Lb43TfNZkdXUEVLvM0_F=xMJ-1FXZfrsfe61LEAycPVg@mail.gmail.com>
	<CAK8kjrDtba0PfhH1Q-XM+2OiREGRCJirv5y-oUzdr8wBOo-VDw@mail.gmail.com>
	<CAK8kjrCOEm-XTu_JzQ70qcbXeayzVVeahw5ENHuXWoZWG5+T9w@mail.gmail.com>
	<CAMzgZ0+Fsj6v02oPL-tksQuB00NrnZy8shbY9j1Fm5NZm16HWw@mail.gmail.com>
	<CAK8kjrAxt7ce8WbteQS2bG2etc5fTgepQWHXRtx=WnOtv8JsCQ@mail.gmail.com>
	<CAMzgZ0LqV7cEMCFa-8yArjXKxPGLaThJ6nLvwrx3LH=zaGvxhg@mail.gmail.com>
Message-ID: <CAK8kjrBb86bDmaN6EFcH8qy9QA=bGrw_hAFh1eYwvEmG+ZxL8A@mail.gmail.com>

OK, thanks Jon.

On Thu, Mar 5, 2020 at 1:23 PM Jon Siwek <jsiwek at corelight.com> wrote:

> On Thu, Mar 5, 2020 at 6:44 AM Doug Burks <doug.burks at gmail.com> wrote:
>
> > Would you recommend that we apply the patch to our Zeek 3.0.2 package OR
> hold off on 3.0.2 and wait for 3.0.3 assuming it will be included there?
>
> Hi Doug, I do now have this on the schedule for fixing in a Zeek 3.0.3
> (and 3.1.1), but not sure what time frame to expect those releases.
> I'll suggest to the team to aim at doing one in ~2-3 weeks.  If you'd
> like to patch ASAP, you may still want to wait at least until that
> Pull Request gets a code review and merged into the master branch.
>
> - Jon
>


-- 
Doug Burks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200305/25e04eb4/attachment.html 

From johanna at icir.org  Thu Mar  5 14:50:42 2020
From: johanna at icir.org (Johanna Amann)
Date: Thu, 05 Mar 2020 14:50:42 -0800
Subject: [Zeek] Errors trying to implement the detection on CVE-2020-0601
In-Reply-To: <BY5PR09MB53771FEE30588889A23845CBA1E20@BY5PR09MB5377.namprd09.prod.outlook.com>
References: <BY5PR09MB53771FEE30588889A23845CBA1E20@BY5PR09MB5377.namprd09.prod.outlook.com>
Message-ID: <D7D210AA-9321-43AA-9C20-54392285153D@icir.org>

Hi Kayode,

the script does, out of the box, not support anything below bro 2.6.

You can probably make it run by changing the option to a ?const 
log_certs = F &redef? and changing the @if (Version::) to @if ( 0 ). 
However, note that while it should work it has not been tested on these 
systems.

Also - please consider updating your Zeek installation. You are missing 
important security and performance fixes.

Johanna

On 5 Mar 2020, at 10:43, Kayode Enwerem wrote:

> Hello,
>
> I am trying to implement the detection of CVE-2020-0601 with zeek 
> (https://blog.zeek.org/2020/01/detecting-cve-2020-0601-with-zeek.html) 
> using the first package (https://github.com/0xxon/cve-2020-0601) but I 
> keep encountering some errors.
>
> Version for bro in my environment: bro version 2.5.5
>
> First thing I did was add this to our local.bro file: redef 
> CVE_2020_0601::log_certs = T;
>
> But when I ran "broctl check" I got the following error message: error 
> in /usr/local/bro/share/bro/site/local.bro, line 13: "redef" used but 
> not previously defined (CVE_             2020_0601::log_certs)
>
> So I created the following file in 
> "share/bro/base/frameworks/notice/cve-2020-0601.bro" and added the 
> script from: 
> https://github.com/0xxon/cve-2020-0601/blob/master/scripts/cve-2020-0601.bro
>
> And also edited the following file 
> "share/bro/base/frameworks/notice/__load__.bro" and added: @load 
> ./cve-2020-0601
>
> Now when I run "broctl check" I am getting the following error 
> message:
> error in 
> /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro, 
> line 5: syntax error, at or near "option"
>
> When I comment out line 5 line I get:
> error in 
> /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro, 
> line 26: unknown identifier Version::at_least, at or near 
> "Version::at_least"
>
> When I comment out line 26 I get:
> error in 
> /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro, 
> line 35: unknown identifier f, at or near "f"
>
> Can someone please help me with this? Am I setting it up right?
>
> Thanks in advance.
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

From undicizeri at gmail.com  Fri Mar  6 02:15:47 2020
From: undicizeri at gmail.com (Federico Foschini)
Date: Fri, 6 Mar 2020 11:15:47 +0100
Subject: [Zeek] Capture packet loss discrepancy
Message-ID: <CAFRU1XuuAzoa8CgxmTTYfX=jqyfBzHyhfKpLhuhkFbiL_m6xWA@mail.gmail.com>

Hello,
I?m using Zeek 3.0.1 and I?m seeing very high zeek capture loss even if the
system load is very low (I?m analyzing 50-100mbps of traffic with a Xeon
8C-16T and 32GiB of ram, the system load is barely rearching 1).

This is what
https://docs.zeek.org/en/stable/scripts/policy/misc/capture-loss.zeek.html
is reporting:

{"_path":"capture_loss","_write_ts":"2020-03-06T10:00:28.080623Z","ts":"2020-03-06T10:00:28.080623Z","ts_delta":900.0000138282776,"peer":"worker-1-1","gaps":980763,"acks":3451717,"percent_lost":28.4137720444
6367}
{"_path":"capture_loss","_write_ts":"2020-03-06T09:45:28.080609Z","ts":"2020-03-06T09:45:28.080609Z","ts_delta":900.0000011920929,"peer":"worker-1-1","gaps":775832,"acks":3944662,"percent_lost":19.6678955002
98886}

But in Zeek stats logs I cannot see any drops:

{"_path":"stats","_write_ts":"2020-03-06T10:00:25.184307Z","ts":"2020-03-06T10:00:25.184307Z","peer":"manager","mem":99,"pkts_proc":0,"bytes_recv":0,"events_proc":2171,"events_queued":2175,"active_tcp_conns"
:0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":1097,"active_timers":77,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size
":0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
{"_path":"stats","_write_ts":"2020-03-06T10:00:26.676726Z","ts":"2020-03-06T10:00:26.676726Z","peer":"proxy-1","mem":95,"pkts_proc":0,"bytes_recv":0,"events_proc":1579,"events_queued":1579,"active_tcp_conns"
:0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":939,"active_timers":39,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size"
:0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
{"_path":"stats","_write_ts":"2020-03-06T10:00:28.087807Z","ts":"2020-03-06T10:00:28.087807Z","peer":"worker-1-1","mem":494,"pkts_proc":4342283,"bytes_recv":3395272841,"pkts_dropped":0,"pkts_link":4347989,"p
kt_lag":0.010818004608154297,"events_proc":894955,"events_queued":894955,"active_tcp_conns":2944,"active_udp_conns":473,"active_icmp_conns":70,"tcp_conns":9036,"udp_conns":7376,"icmp_conns":383,"timers":2168
06,"active_timers":11468,"files":10693,"active_files":10,"dns_requests":2,"active_dns_requests":0,"reassem_tcp_size":500680,"reassem_file_size":451064,"reassem_frag_size":0,"reassem_unknown_size":0}
{"_path":"stats","_write_ts":"2020-03-06T10:05:25.185129Z","ts":"2020-03-06T10:05:25.185129Z","peer":"manager","mem":99,"pkts_proc":0,"bytes_recv":0,"events_proc":3573,"events_queued":3569,"active_tcp_conns"
:0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":1063,"active_timers":77,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size
":0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
{"_path":"stats","_write_ts":"2020-03-06T10:05:26.677296Z","ts":"2020-03-06T10:05:26.677296Z","peer":"proxy-1","mem":95,"pkts_proc":0,"bytes_recv":0,"events_proc":2101,"events_queued":2101,"active_tcp_conns"
:0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":933,"active_timers":39,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size"
:0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
{"_path":"stats","_write_ts":"2020-03-06T10:05:28.087831Z","ts":"2020-03-06T10:05:28.087831Z","peer":"worker-1-1","mem":494,"pkts_proc":4024429,"bytes_recv":3075761558,"pkts_dropped":0,"pkts_link":4029762,"p
kt_lag":0.012360095977783203,"events_proc":860917,"events_queued":860919,"active_tcp_conns":3043,"active_udp_conns":505,"active_icmp_conns":149,"tcp_conns":10006,"udp_conns":7298,"icmp_conns":483,"timers":23
2394,"active_timers":12492,"files":11763,"active_files":13,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size":60920,"reassem_file_size":955592,"reassem_frag_size":0,"reassem_unknown_size":0}

My network interface reports 0 drops:

NIC statistics:
     rx_packets: 57185146109
     tx_packets: 118236
     rx_bytes: 51106679706383
     tx_bytes: 12060072
     rx_broadcast: 28116614
     tx_broadcast: 0
     rx_multicast: 430062675
     tx_multicast: 0
     multicast: 430062675
     collisions: 0
     rx_crc_errors: 0
     rx_no_buffer_count: 0
     rx_missed_errors: 0
     tx_aborted_errors: 0
     tx_carrier_errors: 0
     tx_window_errors: 0
     tx_abort_late_coll: 0
     tx_deferred_ok: 0
     tx_single_coll_ok: 0
     tx_multi_coll_ok: 0
     tx_timeout_count: 0
     rx_long_length_errors: 0
     rx_short_length_errors: 0
     rx_align_errors: 0
     tx_tcp_seg_good: 0
     tx_tcp_seg_failed: 0
     rx_flow_control_xon: 0
     rx_flow_control_xoff: 0
     tx_flow_control_xon: 0
     tx_flow_control_xoff: 0
     rx_long_byte_count: 51106679706383
     tx_dma_out_of_sync: 0
     tx_smbus: 0
     rx_smbus: 0
     dropped_smbus: 0
     os2bmc_rx_by_bmc: 0
     os2bmc_tx_by_bmc: 0
     os2bmc_tx_by_host: 0
     os2bmc_rx_by_host: 0
     tx_hwtstamp_timeouts: 0
     rx_hwtstamp_cleared: 0
     rx_errors: 0
     tx_errors: 0
     tx_dropped: 0
     rx_length_errors: 0
     rx_over_errors: 0
     rx_frame_errors: 0
     rx_fifo_errors: 0
     tx_fifo_errors: 0
     tx_heartbeat_errors: 0
     tx_queue_0_packets: 0
tx_queue_0_restart: 0
     tx_queue_1_packets: 118236
     tx_queue_1_bytes: 11587128
     tx_queue_1_restart: 0
     tx_queue_2_packets: 0
     tx_queue_2_bytes: 0
     tx_queue_2_restart: 0
     tx_queue_3_packets: 0
     tx_queue_3_bytes: 0
     tx_queue_3_restart: 0
     tx_queue_4_packets: 0
     tx_queue_4_bytes: 0
     tx_queue_4_restart: 0
     tx_queue_5_packets: 0
     tx_queue_5_bytes: 0
     tx_queue_5_restart: 0
     tx_queue_6_packets: 0
     tx_queue_6_bytes: 0
     tx_queue_6_restart: 0
     tx_queue_7_packets: 0
     tx_queue_7_bytes: 0
     tx_queue_7_restart: 0
     rx_queue_0_packets: 7309311690
     rx_queue_0_bytes: 6672057827542
     rx_queue_0_drops: 0
     rx_queue_0_csum_err: 0
     rx_queue_0_alloc_failed: 0
     rx_queue_1_packets: 7067404359
     rx_queue_1_bytes: 5978548722708
     rx_queue_1_drops: 0
     rx_queue_1_csum_err: 0
     rx_queue_1_alloc_failed: 0
     rx_queue_2_packets: 6936589456
     rx_queue_2_bytes: 5816850955623
     rx_queue_2_drops: 0
     rx_queue_2_csum_err: 0
     rx_queue_2_alloc_failed: 0
     rx_queue_3_packets: 7560820836
     rx_queue_3_bytes: 7177372551363
     rx_queue_3_drops: 0
     rx_queue_3_csum_err: 0
     rx_queue_3_alloc_failed: 0
     rx_queue_4_packets: 6665690657
     rx_queue_4_bytes: 5815406197188
     rx_queue_4_drops: 0
     rx_queue_4_csum_err: 0
     rx_queue_4_alloc_failed: 0
     rx_queue_5_packets: 7245905157
     rx_queue_5_bytes: 6640952714842
     rx_queue_5_drops: 0
     rx_queue_5_csum_err: 0
tx_queue_5_restart: 0
     tx_queue_6_packets: 0
     tx_queue_6_bytes: 0
     tx_queue_6_restart: 0
     tx_queue_7_packets: 0
     tx_queue_7_bytes: 0
     tx_queue_7_restart: 0
     rx_queue_0_packets: 7309311690
     rx_queue_0_bytes: 6672057827542
     rx_queue_0_drops: 0
     rx_queue_0_csum_err: 0
     rx_queue_0_alloc_failed: 0
     rx_queue_1_packets: 7067404359
     rx_queue_1_bytes: 5978548722708
     rx_queue_1_drops: 0
     rx_queue_1_csum_err: 0
     rx_queue_1_alloc_failed: 0
     rx_queue_2_packets: 6936589456
     rx_queue_2_bytes: 5816850955623
     rx_queue_2_drops: 0
     rx_queue_2_csum_err: 0
     rx_queue_2_alloc_failed: 0
     rx_queue_3_packets: 7560820836
     rx_queue_3_bytes: 7177372551363
     rx_queue_3_drops: 0
     rx_queue_3_csum_err: 0
     rx_queue_3_alloc_failed: 0
     rx_queue_4_packets: 6665690657
     rx_queue_4_bytes: 5815406197188
     rx_queue_4_drops: 0
     rx_queue_4_csum_err: 0
     rx_queue_4_alloc_failed: 0
     rx_queue_5_packets: 7245905157
     rx_queue_5_bytes: 6640952714842
     rx_queue_5_drops: 0
     rx_queue_5_csum_err: 0
     rx_queue_5_alloc_failed: 0
     rx_queue_6_packets: 7693400503
     rx_queue_6_bytes: 6803443308105
     rx_queue_6_drops: 0
     rx_queue_6_csum_err: 0
     rx_queue_6_alloc_failed: 0
     rx_queue_7_packets: 6706023451
     rx_queue_7_bytes: 5768995611822
     rx_queue_7_drops: 0
     rx_queue_7_csum_err: 0
     rx_queue_7_alloc_failed: 0

Is there something am I missing? Is it a way to further analyze the
problem? By looking in zeek logs everything looks fine.
-- 
Federico Foschini.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200306/4b786dae/attachment-0001.html 

From michalpurzynski1 at gmail.com  Fri Mar  6 02:39:07 2020
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Fri, 6 Mar 2020 02:39:07 -0800
Subject: [Zeek] Capture packet loss discrepancy
In-Reply-To: <CAFRU1XuuAzoa8CgxmTTYfX=jqyfBzHyhfKpLhuhkFbiL_m6xWA@mail.gmail.com>
References: <CAFRU1XuuAzoa8CgxmTTYfX=jqyfBzHyhfKpLhuhkFbiL_m6xWA@mail.gmail.com>
Message-ID: <CAJ6bFK344M2ufq_u=9oYzKU2PnB5Tnp2nH2XUzoDKnhxy2x9ZA@mail.gmail.com>

You either drop data before it reaches Zeek sensors or you have an
offloading missconfiguration on the card or you have a load-balancing
problem. That might be a good news.

The capture.log file is based on heuristics "I have seen some ACKs without
matching data packets" - but won't tell you where the problem is. Justin
calls it a "check engine light".

You don't seem to have drops in stats.log and in ethtool stats. That's
great. Stats.log meaning is different for each capture method and it's
never enough without some ethtool stats.

Can you tell us more about your setup?
- how are you feeding your sensor (taps, span ports, etc)
- what's your capture mechanism?
- can you share all commands for interface's configuration?
- can you share the node.cfg
- output of

ethtool -c <interface>
ethtool -l <interface>
ethtool -x <interface>
ethtool -a <interface>
ethtool -g <interface>

On Fri, Mar 6, 2020 at 2:19 AM Federico Foschini <undicizeri at gmail.com>
wrote:

> Hello,
> I?m using Zeek 3.0.1 and I?m seeing very high zeek capture loss even if
> the system load is very low (I?m analyzing 50-100mbps of traffic with a
> Xeon 8C-16T and 32GiB of ram, the system load is barely rearching 1).
>
> This is what
> https://docs.zeek.org/en/stable/scripts/policy/misc/capture-loss.zeek.html
> is reporting:
>
> {"_path":"capture_loss","_write_ts":"2020-03-06T10:00:28.080623Z","ts":"2020-03-06T10:00:28.080623Z","ts_delta":900.0000138282776,"peer":"worker-1-1","gaps":980763,"acks":3451717,"percent_lost":28.4137720444
> 6367}
> {"_path":"capture_loss","_write_ts":"2020-03-06T09:45:28.080609Z","ts":"2020-03-06T09:45:28.080609Z","ts_delta":900.0000011920929,"peer":"worker-1-1","gaps":775832,"acks":3944662,"percent_lost":19.6678955002
> 98886}
>
> But in Zeek stats logs I cannot see any drops:
>
> {"_path":"stats","_write_ts":"2020-03-06T10:00:25.184307Z","ts":"2020-03-06T10:00:25.184307Z","peer":"manager","mem":99,"pkts_proc":0,"bytes_recv":0,"events_proc":2171,"events_queued":2175,"active_tcp_conns"
> :0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":1097,"active_timers":77,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size
> ":0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
> {"_path":"stats","_write_ts":"2020-03-06T10:00:26.676726Z","ts":"2020-03-06T10:00:26.676726Z","peer":"proxy-1","mem":95,"pkts_proc":0,"bytes_recv":0,"events_proc":1579,"events_queued":1579,"active_tcp_conns"
> :0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":939,"active_timers":39,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size"
> :0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
> {"_path":"stats","_write_ts":"2020-03-06T10:00:28.087807Z","ts":"2020-03-06T10:00:28.087807Z","peer":"worker-1-1","mem":494,"pkts_proc":4342283,"bytes_recv":3395272841,"pkts_dropped":0,"pkts_link":4347989,"p
> kt_lag":0.010818004608154297,"events_proc":894955,"events_queued":894955,"active_tcp_conns":2944,"active_udp_conns":473,"active_icmp_conns":70,"tcp_conns":9036,"udp_conns":7376,"icmp_conns":383,"timers":2168
> 06,"active_timers":11468,"files":10693,"active_files":10,"dns_requests":2,"active_dns_requests":0,"reassem_tcp_size":500680,"reassem_file_size":451064,"reassem_frag_size":0,"reassem_unknown_size":0}
> {"_path":"stats","_write_ts":"2020-03-06T10:05:25.185129Z","ts":"2020-03-06T10:05:25.185129Z","peer":"manager","mem":99,"pkts_proc":0,"bytes_recv":0,"events_proc":3573,"events_queued":3569,"active_tcp_conns"
> :0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":1063,"active_timers":77,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size
> ":0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
> {"_path":"stats","_write_ts":"2020-03-06T10:05:26.677296Z","ts":"2020-03-06T10:05:26.677296Z","peer":"proxy-1","mem":95,"pkts_proc":0,"bytes_recv":0,"events_proc":2101,"events_queued":2101,"active_tcp_conns"
> :0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":933,"active_timers":39,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size"
> :0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
> {"_path":"stats","_write_ts":"2020-03-06T10:05:28.087831Z","ts":"2020-03-06T10:05:28.087831Z","peer":"worker-1-1","mem":494,"pkts_proc":4024429,"bytes_recv":3075761558,"pkts_dropped":0,"pkts_link":4029762,"p
> kt_lag":0.012360095977783203,"events_proc":860917,"events_queued":860919,"active_tcp_conns":3043,"active_udp_conns":505,"active_icmp_conns":149,"tcp_conns":10006,"udp_conns":7298,"icmp_conns":483,"timers":23
> 2394,"active_timers":12492,"files":11763,"active_files":13,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size":60920,"reassem_file_size":955592,"reassem_frag_size":0,"reassem_unknown_size":0}
>
> My network interface reports 0 drops:
>
> NIC statistics:
>      rx_packets: 57185146109
>      tx_packets: 118236
>      rx_bytes: 51106679706383
>      tx_bytes: 12060072
>      rx_broadcast: 28116614
>      tx_broadcast: 0
>      rx_multicast: 430062675
>      tx_multicast: 0
>      multicast: 430062675
>      collisions: 0
>      rx_crc_errors: 0
>      rx_no_buffer_count: 0
>      rx_missed_errors: 0
>      tx_aborted_errors: 0
>      tx_carrier_errors: 0
>      tx_window_errors: 0
>      tx_abort_late_coll: 0
>      tx_deferred_ok: 0
>      tx_single_coll_ok: 0
>      tx_multi_coll_ok: 0
>      tx_timeout_count: 0
>      rx_long_length_errors: 0
>      rx_short_length_errors: 0
>      rx_align_errors: 0
>      tx_tcp_seg_good: 0
>      tx_tcp_seg_failed: 0
>      rx_flow_control_xon: 0
>      rx_flow_control_xoff: 0
>      tx_flow_control_xon: 0
>      tx_flow_control_xoff: 0
>      rx_long_byte_count: 51106679706383
>      tx_dma_out_of_sync: 0
>      tx_smbus: 0
>      rx_smbus: 0
>      dropped_smbus: 0
>      os2bmc_rx_by_bmc: 0
>      os2bmc_tx_by_bmc: 0
>      os2bmc_tx_by_host: 0
>      os2bmc_rx_by_host: 0
>      tx_hwtstamp_timeouts: 0
>      rx_hwtstamp_cleared: 0
>      rx_errors: 0
>      tx_errors: 0
>      tx_dropped: 0
>      rx_length_errors: 0
>      rx_over_errors: 0
>      rx_frame_errors: 0
>      rx_fifo_errors: 0
>      tx_fifo_errors: 0
>      tx_heartbeat_errors: 0
>      tx_queue_0_packets: 0
> tx_queue_0_restart: 0
>      tx_queue_1_packets: 118236
>      tx_queue_1_bytes: 11587128
>      tx_queue_1_restart: 0
>      tx_queue_2_packets: 0
>      tx_queue_2_bytes: 0
>      tx_queue_2_restart: 0
>      tx_queue_3_packets: 0
>      tx_queue_3_bytes: 0
>      tx_queue_3_restart: 0
>      tx_queue_4_packets: 0
>      tx_queue_4_bytes: 0
>      tx_queue_4_restart: 0
>      tx_queue_5_packets: 0
>      tx_queue_5_bytes: 0
>      tx_queue_5_restart: 0
>      tx_queue_6_packets: 0
>      tx_queue_6_bytes: 0
>      tx_queue_6_restart: 0
>      tx_queue_7_packets: 0
>      tx_queue_7_bytes: 0
>      tx_queue_7_restart: 0
>      rx_queue_0_packets: 7309311690
>      rx_queue_0_bytes: 6672057827542
>      rx_queue_0_drops: 0
>      rx_queue_0_csum_err: 0
>      rx_queue_0_alloc_failed: 0
>      rx_queue_1_packets: 7067404359
>      rx_queue_1_bytes: 5978548722708
>      rx_queue_1_drops: 0
>      rx_queue_1_csum_err: 0
>      rx_queue_1_alloc_failed: 0
>      rx_queue_2_packets: 6936589456
>      rx_queue_2_bytes: 5816850955623
>      rx_queue_2_drops: 0
>      rx_queue_2_csum_err: 0
>      rx_queue_2_alloc_failed: 0
>      rx_queue_3_packets: 7560820836
>      rx_queue_3_bytes: 7177372551363
>      rx_queue_3_drops: 0
>      rx_queue_3_csum_err: 0
>      rx_queue_3_alloc_failed: 0
>      rx_queue_4_packets: 6665690657
>      rx_queue_4_bytes: 5815406197188
>      rx_queue_4_drops: 0
>      rx_queue_4_csum_err: 0
>      rx_queue_4_alloc_failed: 0
>      rx_queue_5_packets: 7245905157
>      rx_queue_5_bytes: 6640952714842
>      rx_queue_5_drops: 0
>      rx_queue_5_csum_err: 0
> tx_queue_5_restart: 0
>      tx_queue_6_packets: 0
>      tx_queue_6_bytes: 0
>      tx_queue_6_restart: 0
>      tx_queue_7_packets: 0
>      tx_queue_7_bytes: 0
>      tx_queue_7_restart: 0
>      rx_queue_0_packets: 7309311690
>      rx_queue_0_bytes: 6672057827542
>      rx_queue_0_drops: 0
>      rx_queue_0_csum_err: 0
>      rx_queue_0_alloc_failed: 0
>      rx_queue_1_packets: 7067404359
>      rx_queue_1_bytes: 5978548722708
>      rx_queue_1_drops: 0
>      rx_queue_1_csum_err: 0
>      rx_queue_1_alloc_failed: 0
>      rx_queue_2_packets: 6936589456
>      rx_queue_2_bytes: 5816850955623
>      rx_queue_2_drops: 0
>      rx_queue_2_csum_err: 0
>      rx_queue_2_alloc_failed: 0
>      rx_queue_3_packets: 7560820836
>      rx_queue_3_bytes: 7177372551363
>      rx_queue_3_drops: 0
>      rx_queue_3_csum_err: 0
>      rx_queue_3_alloc_failed: 0
>      rx_queue_4_packets: 6665690657
>      rx_queue_4_bytes: 5815406197188
>      rx_queue_4_drops: 0
>      rx_queue_4_csum_err: 0
>      rx_queue_4_alloc_failed: 0
>      rx_queue_5_packets: 7245905157
>      rx_queue_5_bytes: 6640952714842
>      rx_queue_5_drops: 0
>      rx_queue_5_csum_err: 0
>      rx_queue_5_alloc_failed: 0
>      rx_queue_6_packets: 7693400503
>      rx_queue_6_bytes: 6803443308105
>      rx_queue_6_drops: 0
>      rx_queue_6_csum_err: 0
>      rx_queue_6_alloc_failed: 0
>      rx_queue_7_packets: 6706023451
>      rx_queue_7_bytes: 5768995611822
>      rx_queue_7_drops: 0
>      rx_queue_7_csum_err: 0
>      rx_queue_7_alloc_failed: 0
>
> Is there something am I missing? Is it a way to further analyze the
> problem? By looking in zeek logs everything looks fine.
> --
> Federico Foschini.
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200306/2cbaa6be/attachment-0001.html 

From undicizeri at gmail.com  Fri Mar  6 02:47:33 2020
From: undicizeri at gmail.com (Federico Foschini)
Date: Fri, 6 Mar 2020 11:47:33 +0100
Subject: [Zeek] Capture packet loss discrepancy
In-Reply-To: <CAJ6bFK344M2ufq_u=9oYzKU2PnB5Tnp2nH2XUzoDKnhxy2x9ZA@mail.gmail.com>
References: <CAFRU1XuuAzoa8CgxmTTYfX=jqyfBzHyhfKpLhuhkFbiL_m6xWA@mail.gmail.com>
	<CAJ6bFK344M2ufq_u=9oYzKU2PnB5Tnp2nH2XUzoDKnhxy2x9ZA@mail.gmail.com>
Message-ID: <CAFRU1XtdigWe8UY=fWJU+KF_+2znoD5i-gpLSYqAtCpOacAaGw@mail.gmail.com>

Hello,
thank you for your help.
I?m configured a span port on the switch.

In Zeek I?ve configured the AF_Packet plugin.

This is my node.cfg:

[manager]
type=manager
host=localhost

[proxy-1]
type=proxy
host=localhost

[worker-1]
type=worker
host=localhost
interface=bond0
lb_method=custom
lb_procs=1
af_packet_fanout_id=21
af_packet_fanout_mode=AF_Packet::FANOUT_HASH
af_packet_buffer_size=128*1024*1024

This is the ethtool output:

ethtool -c eno2
Coalesce parameters for eno2:
Adaptive RX: off  TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 3
rx-frames: 0
rx-usecs-irq: 0
rx-frames-irq: 0

tx-usecs: 0
tx-frames: 0
tx-usecs-irq: 0
tx-frames-irq: 0

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

ethtool -l eno2
Channel parameters for eno2:
Pre-set maximums:
RX:             0
TX:             0
Other:          1
Combined:       8
Current hardware settings:
RX:             0
TX:             0
Other:          1
Combined:       8

ethtool -x eno2
RX flow hash indirection table for eno2 with 8 RX ring(s):
    0:      0     0     0     0     0     0     0     0
    8:      0     0     0     0     0     0     0     0
   16:      1     1     1     1     1     1     1     1
   24:      1     1     1     1     1     1     1     1
   32:      2     2     2     2     2     2     2     2
   40:      2     2     2     2     2     2     2     2
   48:      3     3     3     3     3     3     3     3
   56:      3     3     3     3     3     3     3     3
   64:      4     4     4     4     4     4     4     4
   72:      4     4     4     4     4     4     4     4
   80:      5     5     5     5     5     5     5     5
   88:      5     5     5     5     5     5     5     5
   96:      6     6     6     6     6     6     6     6
  104:      6     6     6     6     6     6     6     6
  112:      7     7     7     7     7     7     7     7
  120:      7     7     7     7     7     7     7     7
RSS hash key:
Operation not supported

ethtool -a eno2
Pause parameters for eno2:
Autonegotiate:  on
RX:             on
TX:             off

ethtool -g eno2
Ring parameters for eno2:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             256


Il giorno ven 6 mar 2020 alle ore 11:39 Micha? Purzy?ski <
michalpurzynski1 at gmail.com> ha scritto:

> You either drop data before it reaches Zeek sensors or you have an
> offloading missconfiguration on the card or you have a load-balancing
> problem. That might be a good news.
>
> The capture.log file is based on heuristics "I have seen some ACKs without
> matching data packets" - but won't tell you where the problem is. Justin
> calls it a "check engine light".
>
> You don't seem to have drops in stats.log and in ethtool stats. That's
> great. Stats.log meaning is different for each capture method and it's
> never enough without some ethtool stats.
>
> Can you tell us more about your setup?
> - how are you feeding your sensor (taps, span ports, etc)
> - what's your capture mechanism?
> - can you share all commands for interface's configuration?
> - can you share the node.cfg
> - output of
>
> ethtool -c <interface>
> ethtool -l <interface>
> ethtool -x <interface>
> ethtool -a <interface>
> ethtool -g <interface>
>
> On Fri, Mar 6, 2020 at 2:19 AM Federico Foschini <undicizeri at gmail.com>
> wrote:
>
>> Hello,
>> I?m using Zeek 3.0.1 and I?m seeing very high zeek capture loss even if
>> the system load is very low (I?m analyzing 50-100mbps of traffic with a
>> Xeon 8C-16T and 32GiB of ram, the system load is barely rearching 1).
>>
>> This is what
>> https://docs.zeek.org/en/stable/scripts/policy/misc/capture-loss.zeek.html
>> is reporting:
>>
>> {"_path":"capture_loss","_write_ts":"2020-03-06T10:00:28.080623Z","ts":"2020-03-06T10:00:28.080623Z","ts_delta":900.0000138282776,"peer":"worker-1-1","gaps":980763,"acks":3451717,"percent_lost":28.4137720444
>> 6367}
>> {"_path":"capture_loss","_write_ts":"2020-03-06T09:45:28.080609Z","ts":"2020-03-06T09:45:28.080609Z","ts_delta":900.0000011920929,"peer":"worker-1-1","gaps":775832,"acks":3944662,"percent_lost":19.6678955002
>> 98886}
>>
>> But in Zeek stats logs I cannot see any drops:
>>
>> {"_path":"stats","_write_ts":"2020-03-06T10:00:25.184307Z","ts":"2020-03-06T10:00:25.184307Z","peer":"manager","mem":99,"pkts_proc":0,"bytes_recv":0,"events_proc":2171,"events_queued":2175,"active_tcp_conns"
>> :0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":1097,"active_timers":77,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size
>> ":0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
>> {"_path":"stats","_write_ts":"2020-03-06T10:00:26.676726Z","ts":"2020-03-06T10:00:26.676726Z","peer":"proxy-1","mem":95,"pkts_proc":0,"bytes_recv":0,"events_proc":1579,"events_queued":1579,"active_tcp_conns"
>> :0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":939,"active_timers":39,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size"
>> :0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
>> {"_path":"stats","_write_ts":"2020-03-06T10:00:28.087807Z","ts":"2020-03-06T10:00:28.087807Z","peer":"worker-1-1","mem":494,"pkts_proc":4342283,"bytes_recv":3395272841,"pkts_dropped":0,"pkts_link":4347989,"p
>> kt_lag":0.010818004608154297,"events_proc":894955,"events_queued":894955,"active_tcp_conns":2944,"active_udp_conns":473,"active_icmp_conns":70,"tcp_conns":9036,"udp_conns":7376,"icmp_conns":383,"timers":2168
>> 06,"active_timers":11468,"files":10693,"active_files":10,"dns_requests":2,"active_dns_requests":0,"reassem_tcp_size":500680,"reassem_file_size":451064,"reassem_frag_size":0,"reassem_unknown_size":0}
>> {"_path":"stats","_write_ts":"2020-03-06T10:05:25.185129Z","ts":"2020-03-06T10:05:25.185129Z","peer":"manager","mem":99,"pkts_proc":0,"bytes_recv":0,"events_proc":3573,"events_queued":3569,"active_tcp_conns"
>> :0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":1063,"active_timers":77,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size
>> ":0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
>> {"_path":"stats","_write_ts":"2020-03-06T10:05:26.677296Z","ts":"2020-03-06T10:05:26.677296Z","peer":"proxy-1","mem":95,"pkts_proc":0,"bytes_recv":0,"events_proc":2101,"events_queued":2101,"active_tcp_conns"
>> :0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":933,"active_timers":39,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size"
>> :0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
>> {"_path":"stats","_write_ts":"2020-03-06T10:05:28.087831Z","ts":"2020-03-06T10:05:28.087831Z","peer":"worker-1-1","mem":494,"pkts_proc":4024429,"bytes_recv":3075761558,"pkts_dropped":0,"pkts_link":4029762,"p
>> kt_lag":0.012360095977783203,"events_proc":860917,"events_queued":860919,"active_tcp_conns":3043,"active_udp_conns":505,"active_icmp_conns":149,"tcp_conns":10006,"udp_conns":7298,"icmp_conns":483,"timers":23
>> 2394,"active_timers":12492,"files":11763,"active_files":13,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size":60920,"reassem_file_size":955592,"reassem_frag_size":0,"reassem_unknown_size":0}
>>
>> My network interface reports 0 drops:
>>
>> NIC statistics:
>>      rx_packets: 57185146109
>>      tx_packets: 118236
>>      rx_bytes: 51106679706383
>>      tx_bytes: 12060072
>>      rx_broadcast: 28116614
>>      tx_broadcast: 0
>>      rx_multicast: 430062675
>>      tx_multicast: 0
>>      multicast: 430062675
>>      collisions: 0
>>      rx_crc_errors: 0
>>      rx_no_buffer_count: 0
>>      rx_missed_errors: 0
>>      tx_aborted_errors: 0
>>      tx_carrier_errors: 0
>>      tx_window_errors: 0
>>      tx_abort_late_coll: 0
>>      tx_deferred_ok: 0
>>      tx_single_coll_ok: 0
>>      tx_multi_coll_ok: 0
>>      tx_timeout_count: 0
>>      rx_long_length_errors: 0
>>      rx_short_length_errors: 0
>>      rx_align_errors: 0
>>      tx_tcp_seg_good: 0
>>      tx_tcp_seg_failed: 0
>>      rx_flow_control_xon: 0
>>      rx_flow_control_xoff: 0
>>      tx_flow_control_xon: 0
>>      tx_flow_control_xoff: 0
>>      rx_long_byte_count: 51106679706383
>>      tx_dma_out_of_sync: 0
>>      tx_smbus: 0
>>      rx_smbus: 0
>>      dropped_smbus: 0
>>      os2bmc_rx_by_bmc: 0
>>      os2bmc_tx_by_bmc: 0
>>      os2bmc_tx_by_host: 0
>>      os2bmc_rx_by_host: 0
>>      tx_hwtstamp_timeouts: 0
>>      rx_hwtstamp_cleared: 0
>>      rx_errors: 0
>>      tx_errors: 0
>>      tx_dropped: 0
>>      rx_length_errors: 0
>>      rx_over_errors: 0
>>      rx_frame_errors: 0
>>      rx_fifo_errors: 0
>>      tx_fifo_errors: 0
>>      tx_heartbeat_errors: 0
>>      tx_queue_0_packets: 0
>> tx_queue_0_restart: 0
>>      tx_queue_1_packets: 118236
>>      tx_queue_1_bytes: 11587128
>>      tx_queue_1_restart: 0
>>      tx_queue_2_packets: 0
>>      tx_queue_2_bytes: 0
>>      tx_queue_2_restart: 0
>>      tx_queue_3_packets: 0
>>      tx_queue_3_bytes: 0
>>      tx_queue_3_restart: 0
>>      tx_queue_4_packets: 0
>>      tx_queue_4_bytes: 0
>>      tx_queue_4_restart: 0
>>      tx_queue_5_packets: 0
>>      tx_queue_5_bytes: 0
>>      tx_queue_5_restart: 0
>>      tx_queue_6_packets: 0
>>      tx_queue_6_bytes: 0
>>      tx_queue_6_restart: 0
>>      tx_queue_7_packets: 0
>>      tx_queue_7_bytes: 0
>>      tx_queue_7_restart: 0
>>      rx_queue_0_packets: 7309311690
>>      rx_queue_0_bytes: 6672057827542
>>      rx_queue_0_drops: 0
>>      rx_queue_0_csum_err: 0
>>      rx_queue_0_alloc_failed: 0
>>      rx_queue_1_packets: 7067404359
>>      rx_queue_1_bytes: 5978548722708
>>      rx_queue_1_drops: 0
>>      rx_queue_1_csum_err: 0
>>      rx_queue_1_alloc_failed: 0
>>      rx_queue_2_packets: 6936589456
>>      rx_queue_2_bytes: 5816850955623
>>      rx_queue_2_drops: 0
>>      rx_queue_2_csum_err: 0
>>      rx_queue_2_alloc_failed: 0
>>      rx_queue_3_packets: 7560820836
>>      rx_queue_3_bytes: 7177372551363
>>      rx_queue_3_drops: 0
>>      rx_queue_3_csum_err: 0
>>      rx_queue_3_alloc_failed: 0
>>      rx_queue_4_packets: 6665690657
>>      rx_queue_4_bytes: 5815406197188
>>      rx_queue_4_drops: 0
>>      rx_queue_4_csum_err: 0
>>      rx_queue_4_alloc_failed: 0
>>      rx_queue_5_packets: 7245905157
>>      rx_queue_5_bytes: 6640952714842
>>      rx_queue_5_drops: 0
>>      rx_queue_5_csum_err: 0
>> tx_queue_5_restart: 0
>>      tx_queue_6_packets: 0
>>      tx_queue_6_bytes: 0
>>      tx_queue_6_restart: 0
>>      tx_queue_7_packets: 0
>>      tx_queue_7_bytes: 0
>>      tx_queue_7_restart: 0
>>      rx_queue_0_packets: 7309311690
>>      rx_queue_0_bytes: 6672057827542
>>      rx_queue_0_drops: 0
>>      rx_queue_0_csum_err: 0
>>      rx_queue_0_alloc_failed: 0
>>      rx_queue_1_packets: 7067404359
>>      rx_queue_1_bytes: 5978548722708
>>      rx_queue_1_drops: 0
>>      rx_queue_1_csum_err: 0
>>      rx_queue_1_alloc_failed: 0
>>      rx_queue_2_packets: 6936589456
>>      rx_queue_2_bytes: 5816850955623
>>      rx_queue_2_drops: 0
>>      rx_queue_2_csum_err: 0
>>      rx_queue_2_alloc_failed: 0
>>      rx_queue_3_packets: 7560820836
>>      rx_queue_3_bytes: 7177372551363
>>      rx_queue_3_drops: 0
>>      rx_queue_3_csum_err: 0
>>      rx_queue_3_alloc_failed: 0
>>      rx_queue_4_packets: 6665690657
>>      rx_queue_4_bytes: 5815406197188
>>      rx_queue_4_drops: 0
>>      rx_queue_4_csum_err: 0
>>      rx_queue_4_alloc_failed: 0
>>      rx_queue_5_packets: 7245905157
>>      rx_queue_5_bytes: 6640952714842
>>      rx_queue_5_drops: 0
>>      rx_queue_5_csum_err: 0
>>      rx_queue_5_alloc_failed: 0
>>      rx_queue_6_packets: 7693400503
>>      rx_queue_6_bytes: 6803443308105
>>      rx_queue_6_drops: 0
>>      rx_queue_6_csum_err: 0
>>      rx_queue_6_alloc_failed: 0
>>      rx_queue_7_packets: 6706023451
>>      rx_queue_7_bytes: 5768995611822
>>      rx_queue_7_drops: 0
>>      rx_queue_7_csum_err: 0
>>      rx_queue_7_alloc_failed: 0
>>
>> Is there something am I missing? Is it a way to further analyze the
>> problem? By looking in zeek logs everything looks fine.
>> --
>> Federico Foschini.
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>

-- 
Federico Foschini.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200306/f1d18a36/attachment-0001.html 

From undicizeri at gmail.com  Fri Mar  6 02:53:42 2020
From: undicizeri at gmail.com (Federico Foschini)
Date: Fri, 6 Mar 2020 11:53:42 +0100
Subject: [Zeek] Capture packet loss discrepancy
In-Reply-To: <CAFRU1XtdigWe8UY=fWJU+KF_+2znoD5i-gpLSYqAtCpOacAaGw@mail.gmail.com>
References: <CAFRU1XuuAzoa8CgxmTTYfX=jqyfBzHyhfKpLhuhkFbiL_m6xWA@mail.gmail.com>
	<CAJ6bFK344M2ufq_u=9oYzKU2PnB5Tnp2nH2XUzoDKnhxy2x9ZA@mail.gmail.com>
	<CAFRU1XtdigWe8UY=fWJU+KF_+2znoD5i-gpLSYqAtCpOacAaGw@mail.gmail.com>
Message-ID: <CAFRU1Xs7Pqzwfhv2xEHxatWYX7AW=5WEUVCu8H1x0EQaVti5pw@mail.gmail.com>

I?m also adding my /etc/network/interfaces:

auto eno2
iface eno2 inet manual
  up ip link set $IFACE promisc on arp off up
  down ip link set $IFACE promisc off down
  post-up ethtool -G $IFACE rx 4096; for i in rx tx sg tso ufo gso gro
lro; do ethtool -K $IFACE $i off; done
  post-up echo 1 > /proc/sys/net/ipv6/conf/$IFACE/disable_ipv6
  bond-master bond0

auto bond0
iface bond0 inet manual
  bond-mode balance-rr
  bond-slaves eno2
  mtu 1500
  up ip link set bond0 promisc on arp off up
  down ip link set bond0 promisc off down
  post-up ethtool -G bond0 rx 4096; for i in rx tx sg tso ufo gso gro
lro; do ethtool -K bond0 $i off; done
  post-up echo 1 > /proc/sys/net/ipv6/conf/bond0/disable_ipv6

I?ve configured a bond interface because I?m planning to add another span
port. I?ve copied this configuration from SecurityOnion github:
https://github.com/Security-Onion-Solutions/securityonion-saltstack/blob/d3826bc605a22e090bc7cf07309bce76a2f12623/setup/so-functions#L298

Il giorno ven 6 mar 2020 alle ore 11:47 Federico Foschini <
undicizeri at gmail.com> ha scritto:

> Hello,
> thank you for your help.
> I?m configured a span port on the switch.
>
> In Zeek I?ve configured the AF_Packet plugin.
>
> This is my node.cfg:
>
> [manager]
> type=manager
> host=localhost
>
> [proxy-1]
> type=proxy
> host=localhost
>
> [worker-1]
> type=worker
> host=localhost
> interface=bond0
> lb_method=custom
> lb_procs=1
> af_packet_fanout_id=21
> af_packet_fanout_mode=AF_Packet::FANOUT_HASH
> af_packet_buffer_size=128*1024*1024
>
> This is the ethtool output:
>
> ethtool -c eno2
> Coalesce parameters for eno2:
> Adaptive RX: off  TX: off
> stats-block-usecs: 0
> sample-interval: 0
> pkt-rate-low: 0
> pkt-rate-high: 0
>
> rx-usecs: 3
> rx-frames: 0
> rx-usecs-irq: 0
> rx-frames-irq: 0
>
> tx-usecs: 0
> tx-frames: 0
> tx-usecs-irq: 0
> tx-frames-irq: 0
>
> rx-usecs-low: 0
> rx-frame-low: 0
> tx-usecs-low: 0
> tx-frame-low: 0
>
> rx-usecs-high: 0
> rx-frame-high: 0
> tx-usecs-high: 0
> tx-frame-high: 0
>
> ethtool -l eno2
> Channel parameters for eno2:
> Pre-set maximums:
> RX:             0
> TX:             0
> Other:          1
> Combined:       8
> Current hardware settings:
> RX:             0
> TX:             0
> Other:          1
> Combined:       8
>
> ethtool -x eno2
> RX flow hash indirection table for eno2 with 8 RX ring(s):
>     0:      0     0     0     0     0     0     0     0
>     8:      0     0     0     0     0     0     0     0
>    16:      1     1     1     1     1     1     1     1
>    24:      1     1     1     1     1     1     1     1
>    32:      2     2     2     2     2     2     2     2
>    40:      2     2     2     2     2     2     2     2
>    48:      3     3     3     3     3     3     3     3
>    56:      3     3     3     3     3     3     3     3
>    64:      4     4     4     4     4     4     4     4
>    72:      4     4     4     4     4     4     4     4
>    80:      5     5     5     5     5     5     5     5
>    88:      5     5     5     5     5     5     5     5
>    96:      6     6     6     6     6     6     6     6
>   104:      6     6     6     6     6     6     6     6
>   112:      7     7     7     7     7     7     7     7
>   120:      7     7     7     7     7     7     7     7
> RSS hash key:
> Operation not supported
>
> ethtool -a eno2
> Pause parameters for eno2:
> Autonegotiate:  on
> RX:             on
> TX:             off
>
> ethtool -g eno2
> Ring parameters for eno2:
> Pre-set maximums:
> RX:             4096
> RX Mini:        0
> RX Jumbo:       0
> TX:             4096
> Current hardware settings:
> RX:             4096
> RX Mini:        0
> RX Jumbo:       0
> TX:             256
>
>
> Il giorno ven 6 mar 2020 alle ore 11:39 Micha? Purzy?ski <
> michalpurzynski1 at gmail.com> ha scritto:
>
>> You either drop data before it reaches Zeek sensors or you have an
>> offloading missconfiguration on the card or you have a load-balancing
>> problem. That might be a good news.
>>
>> The capture.log file is based on heuristics "I have seen some ACKs
>> without matching data packets" - but won't tell you where the problem is.
>> Justin calls it a "check engine light".
>>
>> You don't seem to have drops in stats.log and in ethtool stats. That's
>> great. Stats.log meaning is different for each capture method and it's
>> never enough without some ethtool stats.
>>
>> Can you tell us more about your setup?
>> - how are you feeding your sensor (taps, span ports, etc)
>> - what's your capture mechanism?
>> - can you share all commands for interface's configuration?
>> - can you share the node.cfg
>> - output of
>>
>> ethtool -c <interface>
>> ethtool -l <interface>
>> ethtool -x <interface>
>> ethtool -a <interface>
>> ethtool -g <interface>
>>
>> On Fri, Mar 6, 2020 at 2:19 AM Federico Foschini <undicizeri at gmail.com>
>> wrote:
>>
>>> Hello,
>>> I?m using Zeek 3.0.1 and I?m seeing very high zeek capture loss even if
>>> the system load is very low (I?m analyzing 50-100mbps of traffic with a
>>> Xeon 8C-16T and 32GiB of ram, the system load is barely rearching 1).
>>>
>>> This is what
>>> https://docs.zeek.org/en/stable/scripts/policy/misc/capture-loss.zeek.html
>>> is reporting:
>>>
>>> {"_path":"capture_loss","_write_ts":"2020-03-06T10:00:28.080623Z","ts":"2020-03-06T10:00:28.080623Z","ts_delta":900.0000138282776,"peer":"worker-1-1","gaps":980763,"acks":3451717,"percent_lost":28.4137720444
>>> 6367}
>>> {"_path":"capture_loss","_write_ts":"2020-03-06T09:45:28.080609Z","ts":"2020-03-06T09:45:28.080609Z","ts_delta":900.0000011920929,"peer":"worker-1-1","gaps":775832,"acks":3944662,"percent_lost":19.6678955002
>>> 98886}
>>>
>>> But in Zeek stats logs I cannot see any drops:
>>>
>>> {"_path":"stats","_write_ts":"2020-03-06T10:00:25.184307Z","ts":"2020-03-06T10:00:25.184307Z","peer":"manager","mem":99,"pkts_proc":0,"bytes_recv":0,"events_proc":2171,"events_queued":2175,"active_tcp_conns"
>>> :0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":1097,"active_timers":77,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size
>>> ":0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
>>> {"_path":"stats","_write_ts":"2020-03-06T10:00:26.676726Z","ts":"2020-03-06T10:00:26.676726Z","peer":"proxy-1","mem":95,"pkts_proc":0,"bytes_recv":0,"events_proc":1579,"events_queued":1579,"active_tcp_conns"
>>> :0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":939,"active_timers":39,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size"
>>> :0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
>>> {"_path":"stats","_write_ts":"2020-03-06T10:00:28.087807Z","ts":"2020-03-06T10:00:28.087807Z","peer":"worker-1-1","mem":494,"pkts_proc":4342283,"bytes_recv":3395272841,"pkts_dropped":0,"pkts_link":4347989,"p
>>> kt_lag":0.010818004608154297,"events_proc":894955,"events_queued":894955,"active_tcp_conns":2944,"active_udp_conns":473,"active_icmp_conns":70,"tcp_conns":9036,"udp_conns":7376,"icmp_conns":383,"timers":2168
>>> 06,"active_timers":11468,"files":10693,"active_files":10,"dns_requests":2,"active_dns_requests":0,"reassem_tcp_size":500680,"reassem_file_size":451064,"reassem_frag_size":0,"reassem_unknown_size":0}
>>> {"_path":"stats","_write_ts":"2020-03-06T10:05:25.185129Z","ts":"2020-03-06T10:05:25.185129Z","peer":"manager","mem":99,"pkts_proc":0,"bytes_recv":0,"events_proc":3573,"events_queued":3569,"active_tcp_conns"
>>> :0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":1063,"active_timers":77,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size
>>> ":0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
>>> {"_path":"stats","_write_ts":"2020-03-06T10:05:26.677296Z","ts":"2020-03-06T10:05:26.677296Z","peer":"proxy-1","mem":95,"pkts_proc":0,"bytes_recv":0,"events_proc":2101,"events_queued":2101,"active_tcp_conns"
>>> :0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":933,"active_timers":39,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size"
>>> :0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
>>> {"_path":"stats","_write_ts":"2020-03-06T10:05:28.087831Z","ts":"2020-03-06T10:05:28.087831Z","peer":"worker-1-1","mem":494,"pkts_proc":4024429,"bytes_recv":3075761558,"pkts_dropped":0,"pkts_link":4029762,"p
>>> kt_lag":0.012360095977783203,"events_proc":860917,"events_queued":860919,"active_tcp_conns":3043,"active_udp_conns":505,"active_icmp_conns":149,"tcp_conns":10006,"udp_conns":7298,"icmp_conns":483,"timers":23
>>> 2394,"active_timers":12492,"files":11763,"active_files":13,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size":60920,"reassem_file_size":955592,"reassem_frag_size":0,"reassem_unknown_size":0}
>>>
>>> My network interface reports 0 drops:
>>>
>>> NIC statistics:
>>>      rx_packets: 57185146109
>>>      tx_packets: 118236
>>>      rx_bytes: 51106679706383
>>>      tx_bytes: 12060072
>>>      rx_broadcast: 28116614
>>>      tx_broadcast: 0
>>>      rx_multicast: 430062675
>>>      tx_multicast: 0
>>>      multicast: 430062675
>>>      collisions: 0
>>>      rx_crc_errors: 0
>>>      rx_no_buffer_count: 0
>>>      rx_missed_errors: 0
>>>      tx_aborted_errors: 0
>>>      tx_carrier_errors: 0
>>>      tx_window_errors: 0
>>>      tx_abort_late_coll: 0
>>>      tx_deferred_ok: 0
>>>      tx_single_coll_ok: 0
>>>      tx_multi_coll_ok: 0
>>>      tx_timeout_count: 0
>>>      rx_long_length_errors: 0
>>>      rx_short_length_errors: 0
>>>      rx_align_errors: 0
>>>      tx_tcp_seg_good: 0
>>>      tx_tcp_seg_failed: 0
>>>      rx_flow_control_xon: 0
>>>      rx_flow_control_xoff: 0
>>>      tx_flow_control_xon: 0
>>>      tx_flow_control_xoff: 0
>>>      rx_long_byte_count: 51106679706383
>>>      tx_dma_out_of_sync: 0
>>>      tx_smbus: 0
>>>      rx_smbus: 0
>>>      dropped_smbus: 0
>>>      os2bmc_rx_by_bmc: 0
>>>      os2bmc_tx_by_bmc: 0
>>>      os2bmc_tx_by_host: 0
>>>      os2bmc_rx_by_host: 0
>>>      tx_hwtstamp_timeouts: 0
>>>      rx_hwtstamp_cleared: 0
>>>      rx_errors: 0
>>>      tx_errors: 0
>>>      tx_dropped: 0
>>>      rx_length_errors: 0
>>>      rx_over_errors: 0
>>>      rx_frame_errors: 0
>>>      rx_fifo_errors: 0
>>>      tx_fifo_errors: 0
>>>      tx_heartbeat_errors: 0
>>>      tx_queue_0_packets: 0
>>> tx_queue_0_restart: 0
>>>      tx_queue_1_packets: 118236
>>>      tx_queue_1_bytes: 11587128
>>>      tx_queue_1_restart: 0
>>>      tx_queue_2_packets: 0
>>>      tx_queue_2_bytes: 0
>>>      tx_queue_2_restart: 0
>>>      tx_queue_3_packets: 0
>>>      tx_queue_3_bytes: 0
>>>      tx_queue_3_restart: 0
>>>      tx_queue_4_packets: 0
>>>      tx_queue_4_bytes: 0
>>>      tx_queue_4_restart: 0
>>>      tx_queue_5_packets: 0
>>>      tx_queue_5_bytes: 0
>>>      tx_queue_5_restart: 0
>>>      tx_queue_6_packets: 0
>>>      tx_queue_6_bytes: 0
>>>      tx_queue_6_restart: 0
>>>      tx_queue_7_packets: 0
>>>      tx_queue_7_bytes: 0
>>>      tx_queue_7_restart: 0
>>>      rx_queue_0_packets: 7309311690
>>>      rx_queue_0_bytes: 6672057827542
>>>      rx_queue_0_drops: 0
>>>      rx_queue_0_csum_err: 0
>>>      rx_queue_0_alloc_failed: 0
>>>      rx_queue_1_packets: 7067404359
>>>      rx_queue_1_bytes: 5978548722708
>>>      rx_queue_1_drops: 0
>>>      rx_queue_1_csum_err: 0
>>>      rx_queue_1_alloc_failed: 0
>>>      rx_queue_2_packets: 6936589456
>>>      rx_queue_2_bytes: 5816850955623
>>>      rx_queue_2_drops: 0
>>>      rx_queue_2_csum_err: 0
>>>      rx_queue_2_alloc_failed: 0
>>>      rx_queue_3_packets: 7560820836
>>>      rx_queue_3_bytes: 7177372551363
>>>      rx_queue_3_drops: 0
>>>      rx_queue_3_csum_err: 0
>>>      rx_queue_3_alloc_failed: 0
>>>      rx_queue_4_packets: 6665690657
>>>      rx_queue_4_bytes: 5815406197188
>>>      rx_queue_4_drops: 0
>>>      rx_queue_4_csum_err: 0
>>>      rx_queue_4_alloc_failed: 0
>>>      rx_queue_5_packets: 7245905157
>>>      rx_queue_5_bytes: 6640952714842
>>>      rx_queue_5_drops: 0
>>>      rx_queue_5_csum_err: 0
>>> tx_queue_5_restart: 0
>>>      tx_queue_6_packets: 0
>>>      tx_queue_6_bytes: 0
>>>      tx_queue_6_restart: 0
>>>      tx_queue_7_packets: 0
>>>      tx_queue_7_bytes: 0
>>>      tx_queue_7_restart: 0
>>>      rx_queue_0_packets: 7309311690
>>>      rx_queue_0_bytes: 6672057827542
>>>      rx_queue_0_drops: 0
>>>      rx_queue_0_csum_err: 0
>>>      rx_queue_0_alloc_failed: 0
>>>      rx_queue_1_packets: 7067404359
>>>      rx_queue_1_bytes: 5978548722708
>>>      rx_queue_1_drops: 0
>>>      rx_queue_1_csum_err: 0
>>>      rx_queue_1_alloc_failed: 0
>>>      rx_queue_2_packets: 6936589456
>>>      rx_queue_2_bytes: 5816850955623
>>>      rx_queue_2_drops: 0
>>>      rx_queue_2_csum_err: 0
>>>      rx_queue_2_alloc_failed: 0
>>>      rx_queue_3_packets: 7560820836
>>>      rx_queue_3_bytes: 7177372551363
>>>      rx_queue_3_drops: 0
>>>      rx_queue_3_csum_err: 0
>>>      rx_queue_3_alloc_failed: 0
>>>      rx_queue_4_packets: 6665690657
>>>      rx_queue_4_bytes: 5815406197188
>>>      rx_queue_4_drops: 0
>>>      rx_queue_4_csum_err: 0
>>>      rx_queue_4_alloc_failed: 0
>>>      rx_queue_5_packets: 7245905157
>>>      rx_queue_5_bytes: 6640952714842
>>>      rx_queue_5_drops: 0
>>>      rx_queue_5_csum_err: 0
>>>      rx_queue_5_alloc_failed: 0
>>>      rx_queue_6_packets: 7693400503
>>>      rx_queue_6_bytes: 6803443308105
>>>      rx_queue_6_drops: 0
>>>      rx_queue_6_csum_err: 0
>>>      rx_queue_6_alloc_failed: 0
>>>      rx_queue_7_packets: 6706023451
>>>      rx_queue_7_bytes: 5768995611822
>>>      rx_queue_7_drops: 0
>>>      rx_queue_7_csum_err: 0
>>>      rx_queue_7_alloc_failed: 0
>>>
>>> Is there something am I missing? Is it a way to further analyze the
>>> problem? By looking in zeek logs everything looks fine.
>>> --
>>> Federico Foschini.
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>>
>
> --
> Federico Foschini.
>


-- 
Federico Foschini.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200306/6638b46c/attachment-0001.html 

From Kayode_Enwerem at ao.uscourts.gov  Fri Mar  6 05:13:03 2020
From: Kayode_Enwerem at ao.uscourts.gov (Kayode Enwerem)
Date: Fri, 6 Mar 2020 13:13:03 +0000
Subject: [Zeek] Errors trying to implement the detection on CVE-2020-0601
In-Reply-To: <D7D210AA-9321-43AA-9C20-54392285153D@icir.org>
References: <BY5PR09MB53771FEE30588889A23845CBA1E20@BY5PR09MB5377.namprd09.prod.outlook.com>
	<D7D210AA-9321-43AA-9C20-54392285153D@icir.org>
Message-ID: <BY5PR09MB5377C43ABFCA25B34EB997B0A1E30@BY5PR09MB5377.namprd09.prod.outlook.com>

Thanks Johanna for your response. 

I set it up and tested it out on another bro instance we have running bro version 2.6.3 and got this same error message:

error in /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro, line 26: unknown identifier Version::at_least, at or near "Version::at_least"


-----Original Message-----
From: Johanna Amann <johanna at icir.org> 
Sent: Thursday, March 5, 2020 5:51 PM
To: Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov>
Cc: zeek at zeek.org
Subject: Re: [Zeek] Errors trying to implement the detection on CVE-2020-0601

Hi Kayode,

the script does, out of the box, not support anything below bro 2.6.

You can probably make it run by changing the option to a ?const log_certs = F &redef? and changing the @if (Version::) to @if ( 0 ). 
However, note that while it should work it has not been tested on these systems.

Also - please consider updating your Zeek installation. You are missing important security and performance fixes.

Johanna

On 5 Mar 2020, at 10:43, Kayode Enwerem wrote:

> Hello,
>
> I am trying to implement the detection of CVE-2020-0601 with zeek
> (https://blog.zeek.org/2020/01/detecting-cve-2020-0601-with-zeek.html)
> using the first package (https://github.com/0xxon/cve-2020-0601) but I 
> keep encountering some errors.
>
> Version for bro in my environment: bro version 2.5.5
>
> First thing I did was add this to our local.bro file: redef 
> CVE_2020_0601::log_certs = T;
>
> But when I ran "broctl check" I got the following error message: error 
> in /usr/local/bro/share/bro/site/local.bro, line 13: "redef" used but
> not previously defined (CVE_             2020_0601::log_certs)
>
> So I created the following file in
> "share/bro/base/frameworks/notice/cve-2020-0601.bro" and added the 
> script from:
> https://github.com/0xxon/cve-2020-0601/blob/master/scripts/cve-2020-06
> 01.bro
>
> And also edited the following file
> "share/bro/base/frameworks/notice/__load__.bro" and added: @load
> ./cve-2020-0601
>
> Now when I run "broctl check" I am getting the following error
> message:
> error in
> /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro,
> line 5: syntax error, at or near "option"
>
> When I comment out line 5 line I get:
> error in
> /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro,
> line 26: unknown identifier Version::at_least, at or near 
> "Version::at_least"
>
> When I comment out line 26 I get:
> error in
> /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro,
> line 35: unknown identifier f, at or near "f"
>
> Can someone please help me with this? Am I setting it up right?
>
> Thanks in advance.
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From johanna at icir.org  Fri Mar  6 08:52:58 2020
From: johanna at icir.org (Johanna Amann)
Date: Fri, 06 Mar 2020 08:52:58 -0800
Subject: [Zeek] Errors trying to implement the detection on CVE-2020-0601
In-Reply-To: <BY5PR09MB5377C43ABFCA25B34EB997B0A1E30@BY5PR09MB5377.namprd09.prod.outlook.com>
References: <BY5PR09MB53771FEE30588889A23845CBA1E20@BY5PR09MB5377.namprd09.prod.outlook.com>
	<D7D210AA-9321-43AA-9C20-54392285153D@icir.org>
	<BY5PR09MB5377C43ABFCA25B34EB997B0A1E30@BY5PR09MB5377.namprd09.prod.outlook.com>
Message-ID: <BA584585-EB21-4B4E-8094-967BC0B62FC5@icir.org>

Oh, sorry - I did not quite thoroughly enough parse all of your first 
email.

The reason for this is load ordering. You added the script to somewhere 
in /share/bro/base. You should never add scripts to base (or change 
scripts in base). We always assume that things in base are untouched - 
they will be overwritten on updates/upgrades. And if you change things 
in base you will also have to make sure that you don?t break things 
because of ordering issues.

In any case - just move the script to share/bro/site and @load it from 
your local.bro, before the line in which you perform the redef - and 
everything should work :)

Alternatively you can also install it via the package manager.

I hope this helps,
  Johanna

On 6 Mar 2020, at 5:13, Kayode Enwerem wrote:

> Thanks Johanna for your response.
>
> I set it up and tested it out on another bro instance we have running 
> bro version 2.6.3 and got this same error message:
>
> error in 
> /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro, 
> line 26: unknown identifier Version::at_least, at or near 
> "Version::at_least"
>
>
> -----Original Message-----
> From: Johanna Amann <johanna at icir.org>
> Sent: Thursday, March 5, 2020 5:51 PM
> To: Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov>
> Cc: zeek at zeek.org
> Subject: Re: [Zeek] Errors trying to implement the detection on 
> CVE-2020-0601
>
> Hi Kayode,
>
> the script does, out of the box, not support anything below bro 2.6.
>
> You can probably make it run by changing the option to a ?const 
> log_certs = F &redef? and changing the @if (Version::) to @if ( 0 ).
> However, note that while it should work it has not been tested on 
> these systems.
>
> Also - please consider updating your Zeek installation. You are 
> missing important security and performance fixes.
>
> Johanna
>
> On 5 Mar 2020, at 10:43, Kayode Enwerem wrote:
>
>> Hello,
>>
>> I am trying to implement the detection of CVE-2020-0601 with zeek
>> (https://blog.zeek.org/2020/01/detecting-cve-2020-0601-with-zeek.html)
>> using the first package (https://github.com/0xxon/cve-2020-0601) but 
>> I
>> keep encountering some errors.
>>
>> Version for bro in my environment: bro version 2.5.5
>>
>> First thing I did was add this to our local.bro file: redef
>> CVE_2020_0601::log_certs = T;
>>
>> But when I ran "broctl check" I got the following error message: 
>> error
>> in /usr/local/bro/share/bro/site/local.bro, line 13: "redef" used but
>> not previously defined (CVE_             2020_0601::log_certs)
>>
>> So I created the following file in
>> "share/bro/base/frameworks/notice/cve-2020-0601.bro" and added the
>> script from:
>> https://github.com/0xxon/cve-2020-0601/blob/master/scripts/cve-2020-06
>> 01.bro
>>
>> And also edited the following file
>> "share/bro/base/frameworks/notice/__load__.bro" and added: @load
>> ./cve-2020-0601
>>
>> Now when I run "broctl check" I am getting the following error
>> message:
>> error in
>> /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro,
>> line 5: syntax error, at or near "option"
>>
>> When I comment out line 5 line I get:
>> error in
>> /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro,
>> line 26: unknown identifier Version::at_least, at or near
>> "Version::at_least"
>>
>> When I comment out line 26 I get:
>> error in
>> /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro,
>> line 35: unknown identifier f, at or near "f"
>>
>> Can someone please help me with this? Am I setting it up right?
>>
>> Thanks in advance.
>>
>>
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

From Kayode_Enwerem at ao.uscourts.gov  Fri Mar  6 10:29:58 2020
From: Kayode_Enwerem at ao.uscourts.gov (Kayode Enwerem)
Date: Fri, 6 Mar 2020 18:29:58 +0000
Subject: [Zeek] Errors trying to implement the detection on CVE-2020-0601
In-Reply-To: <BA584585-EB21-4B4E-8094-967BC0B62FC5@icir.org>
References: <BY5PR09MB53771FEE30588889A23845CBA1E20@BY5PR09MB5377.namprd09.prod.outlook.com>
	<D7D210AA-9321-43AA-9C20-54392285153D@icir.org>
	<BY5PR09MB5377C43ABFCA25B34EB997B0A1E30@BY5PR09MB5377.namprd09.prod.outlook.com>
	<BA584585-EB21-4B4E-8094-967BC0B62FC5@icir.org>
Message-ID: <BY5PR09MB5377A7B71A6B2F36C75E463FA1E30@BY5PR09MB5377.namprd09.prod.outlook.com>

Thanks again for your response. 

How do I test that the openssl version we have installed automatically converts explicit curves to names while the certificate is parsed?

We currently have this version of openssl installed: openssl-1.0.2k-19.el7.x86_64

Thanks.

Kayode Enwerem (CTR)
Security Tools Linux Admin 
ITSO/SOC
Administrative Office of the U.S. Courts
(202) 227-1530

-----Original Message-----
From: Johanna Amann <johanna at icir.org> 
Sent: Friday, March 6, 2020 11:53 AM
To: Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov>
Cc: zeek at zeek.org
Subject: Re: [Zeek] Errors trying to implement the detection on CVE-2020-0601

Oh, sorry - I did not quite thoroughly enough parse all of your first email.

The reason for this is load ordering. You added the script to somewhere in /share/bro/base. You should never add scripts to base (or change scripts in base). We always assume that things in base are untouched - they will be overwritten on updates/upgrades. And if you change things in base you will also have to make sure that you don?t break things because of ordering issues.

In any case - just move the script to share/bro/site and @load it from your local.bro, before the line in which you perform the redef - and everything should work :)

Alternatively you can also install it via the package manager.

I hope this helps,
  Johanna

On 6 Mar 2020, at 5:13, Kayode Enwerem wrote:

> Thanks Johanna for your response.
>
> I set it up and tested it out on another bro instance we have running 
> bro version 2.6.3 and got this same error message:
>
> error in
> /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro,
> line 26: unknown identifier Version::at_least, at or near 
> "Version::at_least"
>
>
> -----Original Message-----
> From: Johanna Amann <johanna at icir.org>
> Sent: Thursday, March 5, 2020 5:51 PM
> To: Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov>
> Cc: zeek at zeek.org
> Subject: Re: [Zeek] Errors trying to implement the detection on 
> CVE-2020-0601
>
> Hi Kayode,
>
> the script does, out of the box, not support anything below bro 2.6.
>
> You can probably make it run by changing the option to a ?const 
> log_certs = F &redef? and changing the @if (Version::) to @if ( 0 ).
> However, note that while it should work it has not been tested on 
> these systems.
>
> Also - please consider updating your Zeek installation. You are 
> missing important security and performance fixes.
>
> Johanna
>
> On 5 Mar 2020, at 10:43, Kayode Enwerem wrote:
>
>> Hello,
>>
>> I am trying to implement the detection of CVE-2020-0601 with zeek
>> (https://blog.zeek.org/2020/01/detecting-cve-2020-0601-with-zeek.html)
>> using the first package (https://github.com/0xxon/cve-2020-0601) but 
>> I
>> keep encountering some errors.
>>
>> Version for bro in my environment: bro version 2.5.5
>>
>> First thing I did was add this to our local.bro file: redef
>> CVE_2020_0601::log_certs = T;
>>
>> But when I ran "broctl check" I got the following error message: 
>> error
>> in /usr/local/bro/share/bro/site/local.bro, line 13: "redef" used but
>> not previously defined (CVE_             2020_0601::log_certs)
>>
>> So I created the following file in
>> "share/bro/base/frameworks/notice/cve-2020-0601.bro" and added the
>> script from:
>> https://github.com/0xxon/cve-2020-0601/blob/master/scripts/cve-2020-06
>> 01.bro
>>
>> And also edited the following file
>> "share/bro/base/frameworks/notice/__load__.bro" and added: @load
>> ./cve-2020-0601
>>
>> Now when I run "broctl check" I am getting the following error
>> message:
>> error in
>> /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro,
>> line 5: syntax error, at or near "option"
>>
>> When I comment out line 5 line I get:
>> error in
>> /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro,
>> line 26: unknown identifier Version::at_least, at or near
>> "Version::at_least"
>>
>> When I comment out line 26 I get:
>> error in
>> /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro,
>> line 35: unknown identifier f, at or near "f"
>>
>> Can someone please help me with this? Am I setting it up right?
>>
>> Thanks in advance.
>>
>>
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From clopmz at outlook.com  Sat Mar  7 13:57:04 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Sat, 7 Mar 2020 21:57:04 +0000
Subject: [Zeek] No packets captured by Zeek under OpenBSD
Message-ID: <E32DF904-61B2-4924-B536-8B7ADFFCABFF@outlook.com>

Hi all,

I just installed Zeek 3.0.3-dev.3 under two OpenBSD 6.6 amd64 vms (one as worker and another as a manager). All seems to work ok but no packet is captured by Zeek worker. In logs directory, there are only the following files:

total 100
drwxr-xr-x  2 root  wheel    512 Mar  7 21:50 ./
drwxr-xr-x  7 root  wheel    512 Mar  7 21:50 ../
-rw-r--r--  1 root  wheel    137 Mar  7 21:42 .cmdline
-rw-r--r--  1 root  wheel    350 Mar  7 21:42 .env_vars
-rw-r--r--  1 root  wheel      6 Mar  7 21:42 .pid
-rw-r--r--  1 root  wheel     58 Mar  7 21:42 .startup
-rwx------  1 root  wheel     18 Mar  7 21:42 .status*
-rw-r--r--  1 root  wheel    401 Mar  7 21:43 cluster.log
-rw-r--r--  1 root  wheel  30276 Mar  7 21:43 loaded_scripts.log
-rw-r--r--  1 root  wheel    856 Mar  7 21:53 stats.log
-rw-r--r--  1 root  wheel      0 Mar  7 21:42 stderr.log
-rw-r--r--  1 root  wheel    140 Mar  7 21:43 stdout.log

No one shows any error. Same for the spool directory ? Running tcpdump in worker node works without problem and I can see all the traffic ?

Any idea?

--
Regards,
C. L. Martinez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200307/8f781c8c/attachment.html 

From clopmz at outlook.com  Sun Mar  8 04:48:58 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Sun, 8 Mar 2020 11:48:58 +0000
Subject: [Zeek] No packets captured by Zeek under OpenBSD
Message-ID: <5C6645FC-453D-495C-A228-B3BCC243C569@outlook.com>

This problem only appears when Zeek is configured as a cluster and  using a distrusted installation or configuring several network interfaces like for example:

[manager]
type=manager
host=127.0.0.1

[logger]
type=logger
host=127.0.0.1

[proxy]
type=proxy
host=127.0.0.1

[worker-1]
type=worker
host=127.0.0.1
interface=vio2

[worker-2]
type=worker
host=127.0.0.1
interface=vio3

When Zeek is configured in standalone mode everything works correctly.

Among the hosts I'm testing, network communications are working perfectly between them  and PF is disabled. Maybe is it a bug? I am using Zeek 3.0.3-dev.3 under OpenBSD 6.6 (fully patched).

--
Regards,
C. L. Martinez

From: <zeek-bounces at zeek.org> on behalf of Carlos Lopez <clopmz at outlook.com>
Date: Saturday, 7 March 2020 at 23:00
To: "zeek at zeek.org" <zeek at zeek.org>
Subject: [Zeek] No packets captured by Zeek under OpenBSD

Hi all,

I just installed Zeek 3.0.3-dev.3 under two OpenBSD 6.6 amd64 vms (one as worker and another as a manager). All seems to work ok but no packet is captured by Zeek worker. In logs directory, there are only the following files:

total 100
drwxr-xr-x  2 root  wheel    512 Mar  7 21:50 ./
drwxr-xr-x  7 root  wheel    512 Mar  7 21:50 ../
-rw-r--r--  1 root  wheel    137 Mar  7 21:42 .cmdline
-rw-r--r--  1 root  wheel    350 Mar  7 21:42 .env_vars
-rw-r--r--  1 root  wheel      6 Mar  7 21:42 .pid
-rw-r--r--  1 root  wheel     58 Mar  7 21:42 .startup
-rwx------  1 root  wheel     18 Mar  7 21:42 .status*
-rw-r--r--  1 root  wheel    401 Mar  7 21:43 cluster.log
-rw-r--r--  1 root  wheel  30276 Mar  7 21:43 loaded_scripts.log
-rw-r--r--  1 root  wheel    856 Mar  7 21:53 stats.log
-rw-r--r--  1 root  wheel      0 Mar  7 21:42 stderr.log
-rw-r--r--  1 root  wheel    140 Mar  7 21:43 stdout.log

No one shows any error. Same for the spool directory ? Running tcpdump in worker node works without problem and I can see all the traffic ?

Any idea?

--
Regards,
C. L. Martinez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200308/27603a1e/attachment-0001.html 

From johanna at icir.org  Sun Mar  8 08:55:11 2020
From: johanna at icir.org (Johanna Amann)
Date: Sun, 08 Mar 2020 08:55:11 -0700
Subject: [Zeek] Errors trying to implement the detection on CVE-2020-0601
In-Reply-To: <BY5PR09MB5377A7B71A6B2F36C75E463FA1E30@BY5PR09MB5377.namprd09.prod.outlook.com>
References: <BY5PR09MB53771FEE30588889A23845CBA1E20@BY5PR09MB5377.namprd09.prod.outlook.com>
	<D7D210AA-9321-43AA-9C20-54392285153D@icir.org>
	<BY5PR09MB5377C43ABFCA25B34EB997B0A1E30@BY5PR09MB5377.namprd09.prod.outlook.com>
	<BA584585-EB21-4B4E-8094-967BC0B62FC5@icir.org>
	<BY5PR09MB5377A7B71A6B2F36C75E463FA1E30@BY5PR09MB5377.namprd09.prod.outlook.com>
Message-ID: <023F3ACD-1CEA-404D-83FD-3BD3B876A7F9@icir.org>

Hi Kayode,

you can test it by running the test suite included with the package (git 
clone it and then run btest in the testing directory - or try installing 
it with the zeek package manager). That will run it automatically.

Also - it is not really all that important - you just increase your 
probability of false positives a bit. That being said - I have not 
actually heard of anyone encountering a false positives - certificates 
with explicit curves are exceedingly rare.

Johanna

On 6 Mar 2020, at 10:29, Kayode Enwerem wrote:

> Thanks again for your response.
>
> How do I test that the openssl version we have installed automatically 
> converts explicit curves to names while the certificate is parsed?
>
> We currently have this version of openssl installed: 
> openssl-1.0.2k-19.el7.x86_64
>
> Thanks.
>
> Kayode Enwerem (CTR)
> Security Tools Linux Admin
> ITSO/SOC
> Administrative Office of the U.S. Courts
> (202) 227-1530
>
> -----Original Message-----
> From: Johanna Amann <johanna at icir.org>
> Sent: Friday, March 6, 2020 11:53 AM
> To: Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov>
> Cc: zeek at zeek.org
> Subject: Re: [Zeek] Errors trying to implement the detection on 
> CVE-2020-0601
>
> Oh, sorry - I did not quite thoroughly enough parse all of your first 
> email.
>
> The reason for this is load ordering. You added the script to 
> somewhere in /share/bro/base. You should never add scripts to base (or 
> change scripts in base). We always assume that things in base are 
> untouched - they will be overwritten on updates/upgrades. And if you 
> change things in base you will also have to make sure that you don?t 
> break things because of ordering issues.
>
> In any case - just move the script to share/bro/site and @load it from 
> your local.bro, before the line in which you perform the redef - and 
> everything should work :)
>
> Alternatively you can also install it via the package manager.
>
> I hope this helps,
>   Johanna
>
> On 6 Mar 2020, at 5:13, Kayode Enwerem wrote:
>
>> Thanks Johanna for your response.
>>
>> I set it up and tested it out on another bro instance we have running
>> bro version 2.6.3 and got this same error message:
>>
>> error in
>> /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro,
>> line 26: unknown identifier Version::at_least, at or near
>> "Version::at_least"
>>
>>
>> -----Original Message-----
>> From: Johanna Amann <johanna at icir.org>
>> Sent: Thursday, March 5, 2020 5:51 PM
>> To: Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov>
>> Cc: zeek at zeek.org
>> Subject: Re: [Zeek] Errors trying to implement the detection on
>> CVE-2020-0601
>>
>> Hi Kayode,
>>
>> the script does, out of the box, not support anything below bro 2.6.
>>
>> You can probably make it run by changing the option to a ?const
>> log_certs = F &redef? and changing the @if (Version::) to @if ( 0 
>> ).
>> However, note that while it should work it has not been tested on
>> these systems.
>>
>> Also - please consider updating your Zeek installation. You are
>> missing important security and performance fixes.
>>
>> Johanna
>>
>> On 5 Mar 2020, at 10:43, Kayode Enwerem wrote:
>>
>>> Hello,
>>>
>>> I am trying to implement the detection of CVE-2020-0601 with zeek
>>> (https://blog.zeek.org/2020/01/detecting-cve-2020-0601-with-zeek.html)
>>> using the first package (https://github.com/0xxon/cve-2020-0601) but
>>> I
>>> keep encountering some errors.
>>>
>>> Version for bro in my environment: bro version 2.5.5
>>>
>>> First thing I did was add this to our local.bro file: redef
>>> CVE_2020_0601::log_certs = T;
>>>
>>> But when I ran "broctl check" I got the following error message:
>>> error
>>> in /usr/local/bro/share/bro/site/local.bro, line 13: "redef" used 
>>> but
>>> not previously defined (CVE_             2020_0601::log_certs)
>>>
>>> So I created the following file in
>>> "share/bro/base/frameworks/notice/cve-2020-0601.bro" and added the
>>> script from:
>>> https://github.com/0xxon/cve-2020-0601/blob/master/scripts/cve-2020-06
>>> 01.bro
>>>
>>> And also edited the following file
>>> "share/bro/base/frameworks/notice/__load__.bro" and added: @load
>>> ./cve-2020-0601
>>>
>>> Now when I run "broctl check" I am getting the following error
>>> message:
>>> error in
>>> /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro,
>>> line 5: syntax error, at or near "option"
>>>
>>> When I comment out line 5 line I get:
>>> error in
>>> /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro,
>>> line 26: unknown identifier Version::at_least, at or near
>>> "Version::at_least"
>>>
>>> When I comment out line 26 I get:
>>> error in
>>> /usr/local/bro/share/bro/base/frameworks/notice/./cve-2020-0601.bro,
>>> line 35: unknown identifier f, at or near "f"
>>>
>>> Can someone please help me with this? Am I setting it up right?
>>>
>>> Thanks in advance.
>>>
>>>
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

From humantargetjoe at hotmail.com  Mon Mar  9 11:12:36 2020
From: humantargetjoe at hotmail.com (Joseph C Marion)
Date: Mon, 9 Mar 2020 18:12:36 +0000
Subject: [Zeek] Avro Schema Availability for Zeek Logs
Message-ID: <MN2PR10MB331167ECE1192011605792B2C1FE0@MN2PR10MB3311.namprd10.prod.outlook.com>

In an effort to save myself a lot of typing, and potentially missing fields, does anyone have a current set of Avro schemas that map to the current built-in logs in Zeek version (3.0.0+)?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200309/3d3938c4/attachment.html 

From jsiwek at corelight.com  Mon Mar  9 11:56:34 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Mon, 9 Mar 2020 11:56:34 -0700
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAK8kjrBb86bDmaN6EFcH8qy9QA=bGrw_hAFh1eYwvEmG+ZxL8A@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
	<CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
	<CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>
	<CAPfnCuizcGsP1jhAafqmkRn-Mq=uMNq2s3+jQf8vkkQHeLg3fQ@mail.gmail.com>
	<CAAYFXLmwMZUep3o-N02fWhx6ybcgbT6htfbdyRKoNd__fdz2hQ@mail.gmail.com>
	<CAMzgZ0KoVGu2KmA7u06B9FmbXbhA2h0toxgNDUVGHbz-yc1UCQ@mail.gmail.com>
	<CAAYFXL=bZYmFCiRKHswDY4xBJ5en-kBkWoOoc8Pn4kn=XdLpfQ@mail.gmail.com>
	<CAK8kjrCF_AFZOboTCPZ21x8hGDcwPGGa_KboMA9Bm85pzV_3-A@mail.gmail.com>
	<CAMzgZ0Lb43TfNZkdXUEVLvM0_F=xMJ-1FXZfrsfe61LEAycPVg@mail.gmail.com>
	<CAK8kjrDtba0PfhH1Q-XM+2OiREGRCJirv5y-oUzdr8wBOo-VDw@mail.gmail.com>
	<CAK8kjrCOEm-XTu_JzQ70qcbXeayzVVeahw5ENHuXWoZWG5+T9w@mail.gmail.com>
	<CAMzgZ0+Fsj6v02oPL-tksQuB00NrnZy8shbY9j1Fm5NZm16HWw@mail.gmail.com>
	<CAK8kjrAxt7ce8WbteQS2bG2etc5fTgepQWHXRtx=WnOtv8JsCQ@mail.gmail.com>
	<CAMzgZ0LqV7cEMCFa-8yArjXKxPGLaThJ6nLvwrx3LH=zaGvxhg@mail.gmail.com>
	<CAK8kjrBb86bDmaN6EFcH8qy9QA=bGrw_hAFh1eYwvEmG+ZxL8A@mail.gmail.com>
Message-ID: <CAMzgZ0JysdrjP_CRxTr-oOSQ5GjS9ynx+4TEk+VgHWiXtiyqdA@mail.gmail.com>

Just a FYI / heads-up: I'm planning to make 3.0.3 and 3.1.1 releases
tomorrow that will include the patch for this issue.

- Jon

On Thu, Mar 5, 2020 at 10:57 AM Doug Burks <doug.burks at gmail.com> wrote:
>
> OK, thanks Jon.
>
> On Thu, Mar 5, 2020 at 1:23 PM Jon Siwek <jsiwek at corelight.com> wrote:
>>
>> On Thu, Mar 5, 2020 at 6:44 AM Doug Burks <doug.burks at gmail.com> wrote:
>>
>> > Would you recommend that we apply the patch to our Zeek 3.0.2 package OR hold off on 3.0.2 and wait for 3.0.3 assuming it will be included there?
>>
>> Hi Doug, I do now have this on the schedule for fixing in a Zeek 3.0.3
>> (and 3.1.1), but not sure what time frame to expect those releases.
>> I'll suggest to the team to aim at doing one in ~2-3 weeks.  If you'd
>> like to patch ASAP, you may still want to wait at least until that
>> Pull Request gets a code review and merged into the master branch.
>>
>> - Jon
>
>
>
> --
> Doug Burks

From doug.burks at gmail.com  Mon Mar  9 12:00:58 2020
From: doug.burks at gmail.com (Doug Burks)
Date: Mon, 9 Mar 2020 15:00:58 -0400
Subject: [Zeek] Workers occasionally using 102% CPU
In-Reply-To: <CAMzgZ0JysdrjP_CRxTr-oOSQ5GjS9ynx+4TEk+VgHWiXtiyqdA@mail.gmail.com>
References: <CAAYFXL=q8g2O34B9UWwJ+DPP+Hgc49WiXND0s72TUVL5ocvt=A@mail.gmail.com>
	<CAPfnCuiRvC-BUvo=fC4FwRO01ne9t0VvogDrFh=YuXvEofN92w@mail.gmail.com>
	<CAAYFXL=cG4ioiZEdXgn-LRz1XPfAHWbs_6KVuzadaz-k-hhptw@mail.gmail.com>
	<CAPfnCui-ko2-T_Z1W4_dv1k83=DEjFjaCKQ3sE8eV5jpLEm=xA@mail.gmail.com>
	<CAAYFXLmUKEU2wQ0=rw1qqWDMcf8UqxRJNufmg4_E94XNtYKfTw@mail.gmail.com>
	<CAAYFXLna5unAUcmkQAsyxc_G47M+rKufpXz8+7QKYmpoQOOcOg@mail.gmail.com>
	<CAPfnCuizcGsP1jhAafqmkRn-Mq=uMNq2s3+jQf8vkkQHeLg3fQ@mail.gmail.com>
	<CAAYFXLmwMZUep3o-N02fWhx6ybcgbT6htfbdyRKoNd__fdz2hQ@mail.gmail.com>
	<CAMzgZ0KoVGu2KmA7u06B9FmbXbhA2h0toxgNDUVGHbz-yc1UCQ@mail.gmail.com>
	<CAAYFXL=bZYmFCiRKHswDY4xBJ5en-kBkWoOoc8Pn4kn=XdLpfQ@mail.gmail.com>
	<CAK8kjrCF_AFZOboTCPZ21x8hGDcwPGGa_KboMA9Bm85pzV_3-A@mail.gmail.com>
	<CAMzgZ0Lb43TfNZkdXUEVLvM0_F=xMJ-1FXZfrsfe61LEAycPVg@mail.gmail.com>
	<CAK8kjrDtba0PfhH1Q-XM+2OiREGRCJirv5y-oUzdr8wBOo-VDw@mail.gmail.com>
	<CAK8kjrCOEm-XTu_JzQ70qcbXeayzVVeahw5ENHuXWoZWG5+T9w@mail.gmail.com>
	<CAMzgZ0+Fsj6v02oPL-tksQuB00NrnZy8shbY9j1Fm5NZm16HWw@mail.gmail.com>
	<CAK8kjrAxt7ce8WbteQS2bG2etc5fTgepQWHXRtx=WnOtv8JsCQ@mail.gmail.com>
	<CAMzgZ0LqV7cEMCFa-8yArjXKxPGLaThJ6nLvwrx3LH=zaGvxhg@mail.gmail.com>
	<CAK8kjrBb86bDmaN6EFcH8qy9QA=bGrw_hAFh1eYwvEmG+ZxL8A@mail.gmail.com>
	<CAMzgZ0JysdrjP_CRxTr-oOSQ5GjS9ynx+4TEk+VgHWiXtiyqdA@mail.gmail.com>
Message-ID: <CAK8kjrCm02Hmp=e9cDhO3RNLBK0xWWvKKJY46tW1RpCL6=XpuQ@mail.gmail.com>

Sounds great, thanks Jon!

On Mon, Mar 9, 2020 at 2:56 PM Jon Siwek <jsiwek at corelight.com> wrote:

> Just a FYI / heads-up: I'm planning to make 3.0.3 and 3.1.1 releases
> tomorrow that will include the patch for this issue.
>
> - Jon
>
> On Thu, Mar 5, 2020 at 10:57 AM Doug Burks <doug.burks at gmail.com> wrote:
> >
> > OK, thanks Jon.
> >
> > On Thu, Mar 5, 2020 at 1:23 PM Jon Siwek <jsiwek at corelight.com> wrote:
> >>
> >> On Thu, Mar 5, 2020 at 6:44 AM Doug Burks <doug.burks at gmail.com> wrote:
> >>
> >> > Would you recommend that we apply the patch to our Zeek 3.0.2 package
> OR hold off on 3.0.2 and wait for 3.0.3 assuming it will be included there?
> >>
> >> Hi Doug, I do now have this on the schedule for fixing in a Zeek 3.0.3
> >> (and 3.1.1), but not sure what time frame to expect those releases.
> >> I'll suggest to the team to aim at doing one in ~2-3 weeks.  If you'd
> >> like to patch ASAP, you may still want to wait at least until that
> >> Pull Request gets a code review and merged into the master branch.
> >>
> >> - Jon
> >
> >
> >
> > --
> > Doug Burks
>


-- 
Doug Burks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200309/89578fe7/attachment.html 

From smitra at ucn.ca  Mon Mar  9 13:19:59 2020
From: smitra at ucn.ca (Mitra, Shaibal)
Date: Mon, 9 Mar 2020 20:19:59 +0000
Subject: [Zeek] SMTP
Message-ID: <7ce89d08b8dc4124b6f5de06627b2e75@ucn.ca>

Is there any package that detects SMTP MITM certificate attacks?We are not yet configured for DANE.

Thanks

[signature]
IT Network Systems Administrator
The Pas Campus
Ph:204-627-8593(Office)
Ph:204-620-1221(Cell)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200309/0d00c9d0/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 12170 bytes
Desc: image001.png
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200309/0d00c9d0/attachment-0001.bin 

From jsiwek at corelight.com  Mon Mar  9 20:25:28 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Mon, 9 Mar 2020 20:25:28 -0700
Subject: [Zeek] No packets captured by Zeek under OpenBSD
In-Reply-To: <5C6645FC-453D-495C-A228-B3BCC243C569@outlook.com>
References: <5C6645FC-453D-495C-A228-B3BCC243C569@outlook.com>
Message-ID: <CAMzgZ0JrTLLgrW98EqiWC5t+KDbrEeYn-d-2cxbGM6EuyfYdjA@mail.gmail.com>

A cluster seems to be working for me on OpenBSD after the following
couple tricks:

* echo "redef Broker::disable_ssl=T;" >>
/usr/local/zeek/share/zeek/site/local.zeek
* ZEEK_DEFAULT_LISTEN_ADDRESS=127.0.0.1 zeekctl deploy

The first one is because SSL between Zeek processes isn't currently
working on OpenBSD (likely due to use some particularity of libressl
usage instead of openssl).

The second one may be a bug, or just general configuration issue with
Zeek processes trying to listen/connect to each other over IPv6 on
OpenBSD.  You can verify what it's trying to listen on via `netstat
-lp tcp` (should find things around 47761+ using "tcp" rather than
"tcp6").

Generally, OpenBSD is not an officially supported platform: doesn't
receive as much testing as others and so more prone to bugs/breakages,
but patches that make things work better on OpenBSD are welcome.

- Jon

On Sun, Mar 8, 2020 at 4:57 AM Carlos Lopez <clopmz at outlook.com> wrote:
>
> This problem only appears when Zeek is configured as a cluster and  using a distrusted installation or configuring several network interfaces like for example:
>
>
>
> [manager]
>
> type=manager
>
> host=127.0.0.1
>
>
>
> [logger]
>
> type=logger
>
> host=127.0.0.1
>
>
>
> [proxy]
>
> type=proxy
>
> host=127.0.0.1
>
>
>
> [worker-1]
>
> type=worker
>
> host=127.0.0.1
>
> interface=vio2
>
>
>
> [worker-2]
>
> type=worker
>
> host=127.0.0.1
>
> interface=vio3
>
>
>
> When Zeek is configured in standalone mode everything works correctly.
>
>
>
> Among the hosts I'm testing, network communications are working perfectly between them  and PF is disabled. Maybe is it a bug? I am using Zeek 3.0.3-dev.3 under OpenBSD 6.6 (fully patched).
>
>
>
> --
>
> Regards,
>
> C. L. Martinez
>
>
>
> From: <zeek-bounces at zeek.org> on behalf of Carlos Lopez <clopmz at outlook.com>
> Date: Saturday, 7 March 2020 at 23:00
> To: "zeek at zeek.org" <zeek at zeek.org>
> Subject: [Zeek] No packets captured by Zeek under OpenBSD
>
>
>
> Hi all,
>
>
>
> I just installed Zeek 3.0.3-dev.3 under two OpenBSD 6.6 amd64 vms (one as worker and another as a manager). All seems to work ok but no packet is captured by Zeek worker. In logs directory, there are only the following files:
>
>
>
> total 100
>
> drwxr-xr-x  2 root  wheel    512 Mar  7 21:50 ./
>
> drwxr-xr-x  7 root  wheel    512 Mar  7 21:50 ../
>
> -rw-r--r--  1 root  wheel    137 Mar  7 21:42 .cmdline
>
> -rw-r--r--  1 root  wheel    350 Mar  7 21:42 .env_vars
>
> -rw-r--r--  1 root  wheel      6 Mar  7 21:42 .pid
>
> -rw-r--r--  1 root  wheel     58 Mar  7 21:42 .startup
>
> -rwx------  1 root  wheel     18 Mar  7 21:42 .status*
>
> -rw-r--r--  1 root  wheel    401 Mar  7 21:43 cluster.log
>
> -rw-r--r--  1 root  wheel  30276 Mar  7 21:43 loaded_scripts.log
>
> -rw-r--r--  1 root  wheel    856 Mar  7 21:53 stats.log
>
> -rw-r--r--  1 root  wheel      0 Mar  7 21:42 stderr.log
>
> -rw-r--r--  1 root  wheel    140 Mar  7 21:43 stdout.log
>
>
>
> No one shows any error. Same for the spool directory ? Running tcpdump in worker node works without problem and I can see all the traffic ?
>
>
>
> Any idea?
>
>
>
> --
>
> Regards,
>
> C. L. Martinez
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From smitra at ucn.ca  Tue Mar 10 07:47:08 2020
From: smitra at ucn.ca (Mitra, Shaibal)
Date: Tue, 10 Mar 2020 14:47:08 +0000
Subject: [Zeek] DNS
Message-ID: <095fcc9427d9485bb54be6407f00365d@ucn.ca>

Now that firefox has adopted dns over https will this require changes to the zeek dns and http modules?

Thanks

[signature]
IT Network Systems Administrator
The Pas Campus
Ph:204-627-8593(Office)
Ph:204-620-1221(Cell)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200310/99cb1f9b/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 12170 bytes
Desc: image001.png
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200310/99cb1f9b/attachment-0001.bin 

From mail at staging.zeek.org  Tue Mar 10 08:24:44 2020
From: mail at staging.zeek.org (Amber Graner)
Date: Tue, 10 Mar 2020 15:24:44 +0000
Subject: [Zeek] New Message From The Zeek Network Security Monitor
Message-ID: <102135976ee0bae6d68e4bd96a64e3aa@staging.zeek.org>

This is a test and only a test.  If this had been an actual emergency....

From akgraner at corelight.com  Tue Mar 10 08:34:27 2020
From: akgraner at corelight.com (Amber Graner)
Date: Tue, 10 Mar 2020 11:34:27 -0400
Subject: [Zeek] New Message From The Zeek Network Security Monitor
In-Reply-To: <102135976ee0bae6d68e4bd96a64e3aa@staging.zeek.org>
References: <102135976ee0bae6d68e4bd96a64e3aa@staging.zeek.org>
Message-ID: <CAJhOzursvOqR3f-vk5tS_H5mBTOsgxLyEqfO0KnhODu-pyph_w@mail.gmail.com>

Hi all,

Just testing contact boxes.

~Amber

On Tue, Mar 10, 2020 at 11:27 AM Amber Graner <mail at staging.zeek.org> wrote:

> This is a test and only a test.  If this had been an actual emergency....
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>


-- 

*Amber Graner*
Director of Community
Corelight, Inc

828.582.9469

Schedule time on my calendar here. <https://calendly.com/amber_graner>



 * Ask me about how you can participate in the Zeek (formerly Bro)
community.
 * Remember - ZEEK AND YOU SHALL FIND!!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200310/01cf0b76/attachment.html 

From jsiwek at corelight.com  Tue Mar 10 12:21:14 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Tue, 10 Mar 2020 12:21:14 -0700
Subject: [Zeek] Zeek 3.0.3 and 3.1.1 released (bug fixes)
Message-ID: <CAMzgZ0+mVRkYt=SArPm_Z65_jYMAky06Ctk2fhGzxBhOAGWvSg@mail.gmail.com>

Downloads for Zeek 3.0.3 and 3.1.1 are available:

    https://www.zeek.org/download/index.html

They address a couple bugs that may commonly lead to cpu or memory
overuse.  Release notes with further details are here:

    https://github.com/zeek/zeek/releases/tag/v3.0.3
    https://github.com/zeek/zeek/releases/tag/v3.1.1

From jawren at cisco.com  Tue Mar 10 12:31:15 2020
From: jawren at cisco.com (Jay Wren (jawren))
Date: Tue, 10 Mar 2020 19:31:15 +0000
Subject: [Zeek] DNS
In-Reply-To: <095fcc9427d9485bb54be6407f00365d@ucn.ca>
References: <095fcc9427d9485bb54be6407f00365d@ucn.ca>
Message-ID: <BL0PR11MB29462B200535139E9A3D38A4C5FF0@BL0PR11MB2946.namprd11.prod.outlook.com>

AFAIK, there isn't anything zeek can do to peek into those dns over https requests because it is encrypted in a TLS session. I suppose something could be updated with a list of known DNS over HTTPS providers and traffic to those IP addresses somehow flagged as such.

I don't trust the DNS over HTTPS providers any more than I trust my own DNS servers and so I've blocked them on my network.
________________________________
From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> on behalf of Mitra, Shaibal <smitra at ucn.ca>
Sent: Tuesday, March 10, 2020 10:47 AM
To: zeek at zeek.org <zeek at zeek.org>
Subject: [Zeek] DNS


Now that firefox has adopted dns over https will this require changes to the zeek dns and http modules?



Thanks



[signature]

IT Network Systems Administrator

The Pas Campus

Ph:204-627-8593(Office)

Ph:204-620-1221(Cell)


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200310/f21a20f3/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 12170 bytes
Desc: image001.png
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200310/f21a20f3/attachment-0001.bin 

From michalpurzynski1 at gmail.com  Tue Mar 10 13:27:12 2020
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Tue, 10 Mar 2020 13:27:12 -0700
Subject: [Zeek] DNS
In-Reply-To: <BL0PR11MB29462B200535139E9A3D38A4C5FF0@BL0PR11MB2946.namprd11.prod.outlook.com>
References: <095fcc9427d9485bb54be6407f00365d@ucn.ca>
	<BL0PR11MB29462B200535139E9A3D38A4C5FF0@BL0PR11MB2946.namprd11.prod.outlook.com>
Message-ID: <CAJ6bFK18HBudVwDur1w9AsrskgH8sRiOS27+THCNz7Op_bmpig@mail.gmail.com>

The best thing to do is to disable the whole thing, at the network level.

https://support.mozilla.org/en-US/kb/canary-domain-use-application-dnsnet

Or on every Firefox, in network.trr.mode=5

How sending all of my DNS data by default to cloudflare is good for privacy
is beyond me.

On Tue, Mar 10, 2020 at 12:33 PM Jay Wren (jawren) <jawren at cisco.com> wrote:

> AFAIK, there isn't anything zeek can do to peek into those dns over https
> requests because it is encrypted in a TLS session. I suppose something
> could be updated with a list of known DNS over HTTPS providers and traffic
> to those IP addresses somehow flagged as such.
>
> I don't trust the DNS over HTTPS providers any more than I trust my own
> DNS servers and so I've blocked them on my network.
> ------------------------------
> *From:* zeek-bounces at zeek.org <zeek-bounces at zeek.org> on behalf of Mitra,
> Shaibal <smitra at ucn.ca>
> *Sent:* Tuesday, March 10, 2020 10:47 AM
> *To:* zeek at zeek.org <zeek at zeek.org>
> *Subject:* [Zeek] DNS
>
>
> Now that firefox has adopted dns over https will this require changes to
> the zeek dns and http modules?
>
>
>
> Thanks
>
>
>
> [image: signature]
>
> IT Network Systems Administrator
>
> The Pas Campus
>
> Ph:204-627-8593(Office)
>
> Ph:204-620-1221(Cell)
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200310/f7351990/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 12170 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200310/f7351990/attachment.bin 

From michalpurzynski1 at gmail.com  Tue Mar 10 16:14:05 2020
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Tue, 10 Mar 2020 16:14:05 -0700
Subject: [Zeek] Capture packet loss discrepancy
In-Reply-To: <CAFRU1Xs7Pqzwfhv2xEHxatWYX7AW=5WEUVCu8H1x0EQaVti5pw@mail.gmail.com>
References: <CAFRU1XuuAzoa8CgxmTTYfX=jqyfBzHyhfKpLhuhkFbiL_m6xWA@mail.gmail.com>
	<CAJ6bFK344M2ufq_u=9oYzKU2PnB5Tnp2nH2XUzoDKnhxy2x9ZA@mail.gmail.com>
	<CAFRU1XtdigWe8UY=fWJU+KF_+2znoD5i-gpLSYqAtCpOacAaGw@mail.gmail.com>
	<CAFRU1Xs7Pqzwfhv2xEHxatWYX7AW=5WEUVCu8H1x0EQaVti5pw@mail.gmail.com>
Message-ID: <CAJ6bFK0SBAWYucDpuciqpwKyBV_fUWi65dkN7=WnOCOdwNgiug@mail.gmail.com>

So it does not look like that's Zeek dropping traffic here. Something else
must be going on.

What's your data rate? What's the MTU?

Something that worries me -

     rx_long_byte_count: 51106679706383

Not sure what these are, as it pretty much always requires reading network
card's driver source (these fields are by no means standarized and even if
they happen to have the same name accross vendors they can mean something
else).

Can you share statistics from the switch? Something like show int <span
port here> stats or similar.

On Fri, Mar 6, 2020 at 2:53 AM Federico Foschini <undicizeri at gmail.com>
wrote:

> I?m also adding my /etc/network/interfaces:
>
> auto eno2
> iface eno2 inet manual
>   up ip link set $IFACE promisc on arp off up
>   down ip link set $IFACE promisc off down
>   post-up ethtool -G $IFACE rx 4096; for i in rx tx sg tso ufo gso gro lro; do ethtool -K $IFACE $i off; done
>   post-up echo 1 > /proc/sys/net/ipv6/conf/$IFACE/disable_ipv6
>   bond-master bond0
>
> auto bond0
> iface bond0 inet manual
>   bond-mode balance-rr
>   bond-slaves eno2
>   mtu 1500
>   up ip link set bond0 promisc on arp off up
>   down ip link set bond0 promisc off down
>   post-up ethtool -G bond0 rx 4096; for i in rx tx sg tso ufo gso gro lro; do ethtool -K bond0 $i off; done
>   post-up echo 1 > /proc/sys/net/ipv6/conf/bond0/disable_ipv6
>
> I?ve configured a bond interface because I?m planning to add another span
> port. I?ve copied this configuration from SecurityOnion github:
> https://github.com/Security-Onion-Solutions/securityonion-saltstack/blob/d3826bc605a22e090bc7cf07309bce76a2f12623/setup/so-functions#L298
>
> Il giorno ven 6 mar 2020 alle ore 11:47 Federico Foschini <
> undicizeri at gmail.com> ha scritto:
>
>> Hello,
>> thank you for your help.
>> I?m configured a span port on the switch.
>>
>> In Zeek I?ve configured the AF_Packet plugin.
>>
>> This is my node.cfg:
>>
>> [manager]
>> type=manager
>> host=localhost
>>
>> [proxy-1]
>> type=proxy
>> host=localhost
>>
>> [worker-1]
>> type=worker
>> host=localhost
>> interface=bond0
>> lb_method=custom
>> lb_procs=1
>> af_packet_fanout_id=21
>> af_packet_fanout_mode=AF_Packet::FANOUT_HASH
>> af_packet_buffer_size=128*1024*1024
>>
>> This is the ethtool output:
>>
>> ethtool -c eno2
>> Coalesce parameters for eno2:
>> Adaptive RX: off  TX: off
>> stats-block-usecs: 0
>> sample-interval: 0
>> pkt-rate-low: 0
>> pkt-rate-high: 0
>>
>> rx-usecs: 3
>> rx-frames: 0
>> rx-usecs-irq: 0
>> rx-frames-irq: 0
>>
>> tx-usecs: 0
>> tx-frames: 0
>> tx-usecs-irq: 0
>> tx-frames-irq: 0
>>
>> rx-usecs-low: 0
>> rx-frame-low: 0
>> tx-usecs-low: 0
>> tx-frame-low: 0
>>
>> rx-usecs-high: 0
>> rx-frame-high: 0
>> tx-usecs-high: 0
>> tx-frame-high: 0
>>
>> ethtool -l eno2
>> Channel parameters for eno2:
>> Pre-set maximums:
>> RX:             0
>> TX:             0
>> Other:          1
>> Combined:       8
>> Current hardware settings:
>> RX:             0
>> TX:             0
>> Other:          1
>> Combined:       8
>>
>> ethtool -x eno2
>> RX flow hash indirection table for eno2 with 8 RX ring(s):
>>     0:      0     0     0     0     0     0     0     0
>>     8:      0     0     0     0     0     0     0     0
>>    16:      1     1     1     1     1     1     1     1
>>    24:      1     1     1     1     1     1     1     1
>>    32:      2     2     2     2     2     2     2     2
>>    40:      2     2     2     2     2     2     2     2
>>    48:      3     3     3     3     3     3     3     3
>>    56:      3     3     3     3     3     3     3     3
>>    64:      4     4     4     4     4     4     4     4
>>    72:      4     4     4     4     4     4     4     4
>>    80:      5     5     5     5     5     5     5     5
>>    88:      5     5     5     5     5     5     5     5
>>    96:      6     6     6     6     6     6     6     6
>>   104:      6     6     6     6     6     6     6     6
>>   112:      7     7     7     7     7     7     7     7
>>   120:      7     7     7     7     7     7     7     7
>> RSS hash key:
>> Operation not supported
>>
>> ethtool -a eno2
>> Pause parameters for eno2:
>> Autonegotiate:  on
>> RX:             on
>> TX:             off
>>
>> ethtool -g eno2
>> Ring parameters for eno2:
>> Pre-set maximums:
>> RX:             4096
>> RX Mini:        0
>> RX Jumbo:       0
>> TX:             4096
>> Current hardware settings:
>> RX:             4096
>> RX Mini:        0
>> RX Jumbo:       0
>> TX:             256
>>
>>
>> Il giorno ven 6 mar 2020 alle ore 11:39 Micha? Purzy?ski <
>> michalpurzynski1 at gmail.com> ha scritto:
>>
>>> You either drop data before it reaches Zeek sensors or you have an
>>> offloading missconfiguration on the card or you have a load-balancing
>>> problem. That might be a good news.
>>>
>>> The capture.log file is based on heuristics "I have seen some ACKs
>>> without matching data packets" - but won't tell you where the problem is.
>>> Justin calls it a "check engine light".
>>>
>>> You don't seem to have drops in stats.log and in ethtool stats. That's
>>> great. Stats.log meaning is different for each capture method and it's
>>> never enough without some ethtool stats.
>>>
>>> Can you tell us more about your setup?
>>> - how are you feeding your sensor (taps, span ports, etc)
>>> - what's your capture mechanism?
>>> - can you share all commands for interface's configuration?
>>> - can you share the node.cfg
>>> - output of
>>>
>>> ethtool -c <interface>
>>> ethtool -l <interface>
>>> ethtool -x <interface>
>>> ethtool -a <interface>
>>> ethtool -g <interface>
>>>
>>> On Fri, Mar 6, 2020 at 2:19 AM Federico Foschini <undicizeri at gmail.com>
>>> wrote:
>>>
>>>> Hello,
>>>> I?m using Zeek 3.0.1 and I?m seeing very high zeek capture loss even if
>>>> the system load is very low (I?m analyzing 50-100mbps of traffic with a
>>>> Xeon 8C-16T and 32GiB of ram, the system load is barely rearching 1).
>>>>
>>>> This is what
>>>> https://docs.zeek.org/en/stable/scripts/policy/misc/capture-loss.zeek.html
>>>> is reporting:
>>>>
>>>> {"_path":"capture_loss","_write_ts":"2020-03-06T10:00:28.080623Z","ts":"2020-03-06T10:00:28.080623Z","ts_delta":900.0000138282776,"peer":"worker-1-1","gaps":980763,"acks":3451717,"percent_lost":28.4137720444
>>>> 6367}
>>>> {"_path":"capture_loss","_write_ts":"2020-03-06T09:45:28.080609Z","ts":"2020-03-06T09:45:28.080609Z","ts_delta":900.0000011920929,"peer":"worker-1-1","gaps":775832,"acks":3944662,"percent_lost":19.6678955002
>>>> 98886}
>>>>
>>>> But in Zeek stats logs I cannot see any drops:
>>>>
>>>> {"_path":"stats","_write_ts":"2020-03-06T10:00:25.184307Z","ts":"2020-03-06T10:00:25.184307Z","peer":"manager","mem":99,"pkts_proc":0,"bytes_recv":0,"events_proc":2171,"events_queued":2175,"active_tcp_conns"
>>>> :0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":1097,"active_timers":77,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size
>>>> ":0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
>>>> {"_path":"stats","_write_ts":"2020-03-06T10:00:26.676726Z","ts":"2020-03-06T10:00:26.676726Z","peer":"proxy-1","mem":95,"pkts_proc":0,"bytes_recv":0,"events_proc":1579,"events_queued":1579,"active_tcp_conns"
>>>> :0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":939,"active_timers":39,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size"
>>>> :0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
>>>> {"_path":"stats","_write_ts":"2020-03-06T10:00:28.087807Z","ts":"2020-03-06T10:00:28.087807Z","peer":"worker-1-1","mem":494,"pkts_proc":4342283,"bytes_recv":3395272841,"pkts_dropped":0,"pkts_link":4347989,"p
>>>> kt_lag":0.010818004608154297,"events_proc":894955,"events_queued":894955,"active_tcp_conns":2944,"active_udp_conns":473,"active_icmp_conns":70,"tcp_conns":9036,"udp_conns":7376,"icmp_conns":383,"timers":2168
>>>> 06,"active_timers":11468,"files":10693,"active_files":10,"dns_requests":2,"active_dns_requests":0,"reassem_tcp_size":500680,"reassem_file_size":451064,"reassem_frag_size":0,"reassem_unknown_size":0}
>>>> {"_path":"stats","_write_ts":"2020-03-06T10:05:25.185129Z","ts":"2020-03-06T10:05:25.185129Z","peer":"manager","mem":99,"pkts_proc":0,"bytes_recv":0,"events_proc":3573,"events_queued":3569,"active_tcp_conns"
>>>> :0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":1063,"active_timers":77,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size
>>>> ":0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
>>>> {"_path":"stats","_write_ts":"2020-03-06T10:05:26.677296Z","ts":"2020-03-06T10:05:26.677296Z","peer":"proxy-1","mem":95,"pkts_proc":0,"bytes_recv":0,"events_proc":2101,"events_queued":2101,"active_tcp_conns"
>>>> :0,"active_udp_conns":0,"active_icmp_conns":0,"tcp_conns":0,"udp_conns":0,"icmp_conns":0,"timers":933,"active_timers":39,"files":0,"active_files":0,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size"
>>>> :0,"reassem_file_size":0,"reassem_frag_size":0,"reassem_unknown_size":0}
>>>> {"_path":"stats","_write_ts":"2020-03-06T10:05:28.087831Z","ts":"2020-03-06T10:05:28.087831Z","peer":"worker-1-1","mem":494,"pkts_proc":4024429,"bytes_recv":3075761558,"pkts_dropped":0,"pkts_link":4029762,"p
>>>> kt_lag":0.012360095977783203,"events_proc":860917,"events_queued":860919,"active_tcp_conns":3043,"active_udp_conns":505,"active_icmp_conns":149,"tcp_conns":10006,"udp_conns":7298,"icmp_conns":483,"timers":23
>>>> 2394,"active_timers":12492,"files":11763,"active_files":13,"dns_requests":0,"active_dns_requests":0,"reassem_tcp_size":60920,"reassem_file_size":955592,"reassem_frag_size":0,"reassem_unknown_size":0}
>>>>
>>>> My network interface reports 0 drops:
>>>>
>>>> NIC statistics:
>>>>      rx_packets: 57185146109
>>>>      tx_packets: 118236
>>>>      rx_bytes: 51106679706383
>>>>      tx_bytes: 12060072
>>>>      rx_broadcast: 28116614
>>>>      tx_broadcast: 0
>>>>      rx_multicast: 430062675
>>>>      tx_multicast: 0
>>>>      multicast: 430062675
>>>>      collisions: 0
>>>>      rx_crc_errors: 0
>>>>      rx_no_buffer_count: 0
>>>>      rx_missed_errors: 0
>>>>      tx_aborted_errors: 0
>>>>      tx_carrier_errors: 0
>>>>      tx_window_errors: 0
>>>>      tx_abort_late_coll: 0
>>>>      tx_deferred_ok: 0
>>>>      tx_single_coll_ok: 0
>>>>      tx_multi_coll_ok: 0
>>>>      tx_timeout_count: 0
>>>>      rx_long_length_errors: 0
>>>>      rx_short_length_errors: 0
>>>>      rx_align_errors: 0
>>>>      tx_tcp_seg_good: 0
>>>>      tx_tcp_seg_failed: 0
>>>>      rx_flow_control_xon: 0
>>>>      rx_flow_control_xoff: 0
>>>>      tx_flow_control_xon: 0
>>>>      tx_flow_control_xoff: 0
>>>>      rx_long_byte_count: 51106679706383
>>>>      tx_dma_out_of_sync: 0
>>>>      tx_smbus: 0
>>>>      rx_smbus: 0
>>>>      dropped_smbus: 0
>>>>      os2bmc_rx_by_bmc: 0
>>>>      os2bmc_tx_by_bmc: 0
>>>>      os2bmc_tx_by_host: 0
>>>>      os2bmc_rx_by_host: 0
>>>>      tx_hwtstamp_timeouts: 0
>>>>      rx_hwtstamp_cleared: 0
>>>>      rx_errors: 0
>>>>      tx_errors: 0
>>>>      tx_dropped: 0
>>>>      rx_length_errors: 0
>>>>      rx_over_errors: 0
>>>>      rx_frame_errors: 0
>>>>      rx_fifo_errors: 0
>>>>      tx_fifo_errors: 0
>>>>      tx_heartbeat_errors: 0
>>>>      tx_queue_0_packets: 0
>>>> tx_queue_0_restart: 0
>>>>      tx_queue_1_packets: 118236
>>>>      tx_queue_1_bytes: 11587128
>>>>      tx_queue_1_restart: 0
>>>>      tx_queue_2_packets: 0
>>>>      tx_queue_2_bytes: 0
>>>>      tx_queue_2_restart: 0
>>>>      tx_queue_3_packets: 0
>>>>      tx_queue_3_bytes: 0
>>>>      tx_queue_3_restart: 0
>>>>      tx_queue_4_packets: 0
>>>>      tx_queue_4_bytes: 0
>>>>      tx_queue_4_restart: 0
>>>>      tx_queue_5_packets: 0
>>>>      tx_queue_5_bytes: 0
>>>>      tx_queue_5_restart: 0
>>>>      tx_queue_6_packets: 0
>>>>      tx_queue_6_bytes: 0
>>>>      tx_queue_6_restart: 0
>>>>      tx_queue_7_packets: 0
>>>>      tx_queue_7_bytes: 0
>>>>      tx_queue_7_restart: 0
>>>>      rx_queue_0_packets: 7309311690
>>>>      rx_queue_0_bytes: 6672057827542
>>>>      rx_queue_0_drops: 0
>>>>      rx_queue_0_csum_err: 0
>>>>      rx_queue_0_alloc_failed: 0
>>>>      rx_queue_1_packets: 7067404359
>>>>      rx_queue_1_bytes: 5978548722708
>>>>      rx_queue_1_drops: 0
>>>>      rx_queue_1_csum_err: 0
>>>>      rx_queue_1_alloc_failed: 0
>>>>      rx_queue_2_packets: 6936589456
>>>>      rx_queue_2_bytes: 5816850955623
>>>>      rx_queue_2_drops: 0
>>>>      rx_queue_2_csum_err: 0
>>>>      rx_queue_2_alloc_failed: 0
>>>>      rx_queue_3_packets: 7560820836
>>>>      rx_queue_3_bytes: 7177372551363
>>>>      rx_queue_3_drops: 0
>>>>      rx_queue_3_csum_err: 0
>>>>      rx_queue_3_alloc_failed: 0
>>>>      rx_queue_4_packets: 6665690657
>>>>      rx_queue_4_bytes: 5815406197188
>>>>      rx_queue_4_drops: 0
>>>>      rx_queue_4_csum_err: 0
>>>>      rx_queue_4_alloc_failed: 0
>>>>      rx_queue_5_packets: 7245905157
>>>>      rx_queue_5_bytes: 6640952714842
>>>>      rx_queue_5_drops: 0
>>>>      rx_queue_5_csum_err: 0
>>>> tx_queue_5_restart: 0
>>>>      tx_queue_6_packets: 0
>>>>      tx_queue_6_bytes: 0
>>>>      tx_queue_6_restart: 0
>>>>      tx_queue_7_packets: 0
>>>>      tx_queue_7_bytes: 0
>>>>      tx_queue_7_restart: 0
>>>>      rx_queue_0_packets: 7309311690
>>>>      rx_queue_0_bytes: 6672057827542
>>>>      rx_queue_0_drops: 0
>>>>      rx_queue_0_csum_err: 0
>>>>      rx_queue_0_alloc_failed: 0
>>>>      rx_queue_1_packets: 7067404359
>>>>      rx_queue_1_bytes: 5978548722708
>>>>      rx_queue_1_drops: 0
>>>>      rx_queue_1_csum_err: 0
>>>>      rx_queue_1_alloc_failed: 0
>>>>      rx_queue_2_packets: 6936589456
>>>>      rx_queue_2_bytes: 5816850955623
>>>>      rx_queue_2_drops: 0
>>>>      rx_queue_2_csum_err: 0
>>>>      rx_queue_2_alloc_failed: 0
>>>>      rx_queue_3_packets: 7560820836
>>>>      rx_queue_3_bytes: 7177372551363
>>>>      rx_queue_3_drops: 0
>>>>      rx_queue_3_csum_err: 0
>>>>      rx_queue_3_alloc_failed: 0
>>>>      rx_queue_4_packets: 6665690657
>>>>      rx_queue_4_bytes: 5815406197188
>>>>      rx_queue_4_drops: 0
>>>>      rx_queue_4_csum_err: 0
>>>>      rx_queue_4_alloc_failed: 0
>>>>      rx_queue_5_packets: 7245905157
>>>>      rx_queue_5_bytes: 6640952714842
>>>>      rx_queue_5_drops: 0
>>>>      rx_queue_5_csum_err: 0
>>>>      rx_queue_5_alloc_failed: 0
>>>>      rx_queue_6_packets: 7693400503
>>>>      rx_queue_6_bytes: 6803443308105
>>>>      rx_queue_6_drops: 0
>>>>      rx_queue_6_csum_err: 0
>>>>      rx_queue_6_alloc_failed: 0
>>>>      rx_queue_7_packets: 6706023451
>>>>      rx_queue_7_bytes: 5768995611822
>>>>      rx_queue_7_drops: 0
>>>>      rx_queue_7_csum_err: 0
>>>>      rx_queue_7_alloc_failed: 0
>>>>
>>>> Is there something am I missing? Is it a way to further analyze the
>>>> problem? By looking in zeek logs everything looks fine.
>>>> --
>>>> Federico Foschini.
>>>> _______________________________________________
>>>> Zeek mailing list
>>>> zeek at zeek.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>>
>>>
>>
>> --
>> Federico Foschini.
>>
>
>
> --
> Federico Foschini.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200310/edc96229/attachment-0001.html 

From bill.de.ping at gmail.com  Wed Mar 11 03:44:21 2020
From: bill.de.ping at gmail.com (william de ping)
Date: Wed, 11 Mar 2020 12:44:21 +0200
Subject: [Zeek]  - HTTP extract files from wget
Message-ID: <CAKZA8xjhrmn=unpKQ6jhAXgdry09EbsTb8NjA4Xzpz6tWW4p9Q@mail.gmail.com>

Hi everyone,

I've stumbled upon a weird issue using Zeek 3.0.
Parsing traffic that has a file transfer over http using wget does not
produce any file analysis.

I do see the get_file_handle event were it says ANALYZER::ANALYZER_HTTP but
no files.log is created and extract-all-files.zeek script does not produce
the transferred file.

am I missing something here ?

Thanks
B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200311/04d29108/attachment.html 

From justin at corelight.com  Wed Mar 11 06:40:44 2020
From: justin at corelight.com (Justin Azoff)
Date: Wed, 11 Mar 2020 09:40:44 -0400
Subject: [Zeek] - HTTP extract files from wget
In-Reply-To: <CAKZA8xjhrmn=unpKQ6jhAXgdry09EbsTb8NjA4Xzpz6tWW4p9Q@mail.gmail.com>
References: <CAKZA8xjhrmn=unpKQ6jhAXgdry09EbsTb8NjA4Xzpz6tWW4p9Q@mail.gmail.com>
Message-ID: <CAPfnCuhoSEL2WouzW3_qOVS9a=VnbMQVEXkT_r8DcJg2cWa7+A@mail.gmail.com>

What does the conn log entry look like?

On Wed, Mar 11, 2020 at 6:48 AM william de ping <bill.de.ping at gmail.com> wrote:
>
> Hi everyone,
>
> I've stumbled upon a weird issue using Zeek 3.0.
> Parsing traffic that has a file transfer over http using wget does not produce any file analysis.
>
> I do see the get_file_handle event were it says ANALYZER::ANALYZER_HTTP but no files.log is created and extract-all-files.zeek script does not produce the transferred file.
>
> am I missing something here ?
>
> Thanks
> B
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
Justin

From akgraner at corelight.com  Wed Mar 11 12:56:10 2020
From: akgraner at corelight.com (Amber Graner)
Date: Wed, 11 Mar 2020 15:56:10 -0400
Subject: [Zeek] [Reminder] ASK THE ZEEKSPERTS - 12 March (Justin Azoff and
	Seth Hall)
Message-ID: <CAJhOzuo++sF63Uupw8QdcpkGz=5833yHkyN5_2=TmnJg_Bbm=g@mail.gmail.com>

Hi all,

For this edition of ASK THE ZEEKSPERTS both Justin Azoff and Seth Hall will
be hosts/panelists.

This is a bi-monthly call for the open-source Zeek (formerly Bro) community
to interface directly with leading contributors to the open-source project
and ask questions live to better understand, expand or troubleshoot
deployments of the network security monitoring software.

Gather your questions and we hope to "see" you on the call!

Date: Thursday 12 March 2020

Time: 12:30pm PST/3:30pm EST

Registration Link:
https://attendee.gotowebinar.com/register/3677991044928808972

Thanks,
~Amber
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200311/9e4794ce/attachment.html 

From toabhisheksingh at gmail.com  Thu Mar 12 00:46:31 2020
From: toabhisheksingh at gmail.com (Abhishek Singh)
Date: Thu, 12 Mar 2020 13:16:31 +0530
Subject: [Zeek] Effect of TLS traffic on inspection
Message-ID: <CAGRS=rHHFxHZfCyc9zg2ooqN+CV=1auonoPMpNJEB=cY9QXMzg@mail.gmail.com>

Experts,

I was wondering what effect the rise in TLS traffic has on IDS applications
like Zeek.
Since Zeek (or other IDS applications like Snort and Suricata) will not be
able to inspect the content of majority of the connections as they will be
encrypted, will this make IDS less useful going forward?
If yes, what are the ways being considered to overcome this challenge? Is
becoming an inline device with man in the middle capabilities an option? Or
is TLS offloading to a different device that can send us a copy of the
decrypted traffic for inspection the preferred option?

Please pardon me if these are naive questions. I am new to the world of IDS
and am trying to learn more about them.

- Abhi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200312/35370b68/attachment.html 

From mail at zeekdotorg.wpcomstaging.com  Thu Mar 12 01:03:58 2020
From: mail at zeekdotorg.wpcomstaging.com (Amber Graner)
Date: Thu, 12 Mar 2020 08:03:58 +0000
Subject: [Zeek] New Message From Zeek
Message-ID: <3667077b79c237001ca2571f71ae9082@zeekdotorg.wpcomstaging.com>

Test

From akgraner at corelight.com  Thu Mar 12 03:37:32 2020
From: akgraner at corelight.com (Amber Graner)
Date: Thu, 12 Mar 2020 06:37:32 -0400
Subject: [Zeek] Announcing the NEW Zeek Website
Message-ID: <CAJhOzuqu-oVfMqM4GWFGRsUQAWgVLoZY4d8eoNaK5W0iyWWcvA@mail.gmail.com>

Hi all,

We're super excited to announce the NEW Zeek Website.

https://zeek.org/2020/03/11/announcing-the-new-zeek-website/

Please let us know if you have any questions, comments or feedback.

Thanks,
~Amber
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200312/fd9632ab/attachment.html 

From blackhole.em at gmail.com  Thu Mar 12 04:05:45 2020
From: blackhole.em at gmail.com (Joe Blow)
Date: Thu, 12 Mar 2020 07:05:45 -0400
Subject: [Zeek] Effect of TLS traffic on inspection
In-Reply-To: <CAGRS=rHHFxHZfCyc9zg2ooqN+CV=1auonoPMpNJEB=cY9QXMzg@mail.gmail.com>
Message-ID: <n44btc666j4p4dqoil3dpp3b.1584011145499@gmail.com>

An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200312/607ec09e/attachment-0001.html 

From richard at corelight.com  Thu Mar 12 05:27:32 2020
From: richard at corelight.com (Richard Bejtlich)
Date: Thu, 12 Mar 2020 08:27:32 -0400
Subject: [Zeek] Effect of TLS traffic on inspection
In-Reply-To: <CAGRS=rHHFxHZfCyc9zg2ooqN+CV=1auonoPMpNJEB=cY9QXMzg@mail.gmail.com>
References: <CAGRS=rHHFxHZfCyc9zg2ooqN+CV=1auonoPMpNJEB=cY9QXMzg@mail.gmail.com>
Message-ID: <CAOtSMjaONq74F0KhW-s6wzaTKqJD=jRERJpi4QvzQMCn56Lrxw@mail.gmail.com>

This is NOT intended as an advertisement, but if you want to check it out,
we've been blogging about how to use Zeek vs encrypted traffic on the
Corelight blog.

https://corelight.blog/tag/encryption/

Sincerely,

Richard

On Thu, Mar 12, 2020 at 3:48 AM Abhishek Singh <toabhisheksingh at gmail.com>
wrote:

> Experts,
>
> I was wondering what effect the rise in TLS traffic has on IDS
> applications like Zeek.
> Since Zeek (or other IDS applications like Snort and Suricata) will not be
> able to inspect the content of majority of the connections as they will be
> encrypted, will this make IDS less useful going forward?
> If yes, what are the ways being considered to overcome this challenge? Is
> becoming an inline device with man in the middle capabilities an option? Or
> is TLS offloading to a different device that can send us a copy of the
> decrypted traffic for inspection the preferred option?
>
> Please pardon me if these are naive questions. I am new to the world of
> IDS and am trying to learn more about them.
>
> - Abhi
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
Richard Bejtlich
Principal Security Strategist, Corelight
https://corelight.blog/author/richardbejtlich/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200312/a25170bf/attachment.html 

From ktheroux at att.net  Thu Mar 12 08:36:08 2020
From: ktheroux at att.net (Kristina Theroux)
Date: Thu, 12 Mar 2020 15:36:08 +0000 (UTC)
Subject: [Zeek] Debugging zeek
References: <1335078017.8373997.1584027368221.ref@mail.yahoo.com>
Message-ID: <1335078017.8373997.1584027368221@mail.yahoo.com>

Hi,
I'm trying to get a better understanding of zeek specifically the publisher/subscriber portion of the manager.? Is there a guide anywhere with information about debugging zeek?? I was trying to follow the dev instructions to use the local zeek that is created using make.? This doesn't seem to log or publish at all from what I could tell.? So how do people usually go about debugging this normally when it comes to using the publisher/subscriber functionality?
Thank you!Kristina
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200312/f2c7a32a/attachment.html 

From kevross33 at googlemail.com  Thu Mar 12 09:01:34 2020
From: kevross33 at googlemail.com (Kevin Ross)
Date: Thu, 12 Mar 2020 16:01:34 +0000
Subject: [Zeek] Effect of TLS traffic on inspection
In-Reply-To: <CAGRS=rHHFxHZfCyc9zg2ooqN+CV=1auonoPMpNJEB=cY9QXMzg@mail.gmail.com>
References: <CAGRS=rHHFxHZfCyc9zg2ooqN+CV=1auonoPMpNJEB=cY9QXMzg@mail.gmail.com>
Message-ID: <CAM_5znu0c7ja7Qtf-dhxJhDUpxvc9fkQZ7=FD1b6XYScAmKHKw@mail.gmail.com>

These are helpful talks on the subject on using zeek style metadata


JA3 asking for a friend (2019):
https://www.youtube.com/watch?v=HrP6Ep3xgQM&t=684s
Network forensics in an encrypted world (2017 but covers a lot of
indicators you can hunt on)
https://www.youtube.com/watch?v=APHlvFaUEKE&t=1930s
Encrypted things: Network detection and response in an encrypted world
https://www.youtube.com/watch?v=HPvIGP2mgbI&t=2667s
Security Onion Conference 2019: Finding traffic anomalies using SSL
certificates https://www.youtube.com/watch?v=-WD9BWlENwc&t=762s

On Thu, 12 Mar 2020, 07:49 Abhishek Singh, <toabhisheksingh at gmail.com>
wrote:

> Experts,
>
> I was wondering what effect the rise in TLS traffic has on IDS
> applications like Zeek.
> Since Zeek (or other IDS applications like Snort and Suricata) will not be
> able to inspect the content of majority of the connections as they will be
> encrypted, will this make IDS less useful going forward?
> If yes, what are the ways being considered to overcome this challenge? Is
> becoming an inline device with man in the middle capabilities an option? Or
> is TLS offloading to a different device that can send us a copy of the
> decrypted traffic for inspection the preferred option?
>
> Please pardon me if these are naive questions. I am new to the world of
> IDS and am trying to learn more about them.
>
> - Abhi
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200312/e39a78fe/attachment.html 

From johanna at corelight.com  Thu Mar 12 10:21:35 2020
From: johanna at corelight.com (Johanna Amann)
Date: Thu, 12 Mar 2020 10:21:35 -0700
Subject: [Zeek] Binary packages and 3.1
In-Reply-To: <E734636A-53E3-4115-8E9B-6D519C187CE6@corelight.com>
References: <E734636A-53E3-4115-8E9B-6D519C187CE6@corelight.com>
Message-ID: <FC49B9F0-88AF-4342-90F5-0EC478B5FDF2@corelight.com>

Hello Everyone,

I have a few small updates on the migration of the binary packages to 
3.1.

As announced in the last email, there is now a ?zeek-lts? package. 
This package will track the LTS releases of Zeek - and currently 
contains Zeek 3.0.3.

As of today, the ?zeek? package also is still on 3.0.3. Sometime 
next week I will switch the zeek package over to 3.1.

If you want are using our current binary packages and want to remain on 
the LTS release of zeek, the only thing you have to do is to perform a

?apt install zeek-lts? on debian-based systems, or a

?yum/dnf install zeek-lts --allowerasing?

This will install zeek-lts and remove the conflicting zeek package. The 
install locations for zeek-lts and zeek are exactly the same 
(/opt/zeek), so nothing else will change.

Please let me know if you encounter any problems,
  Johanna

On 7 Feb 2020, at 11:18, Johanna Amann wrote:

> Hi Everyone,
>
> with the Zeek 3.1 release around the corner, I just wanted to outline 
> my
> current plan for the binary packages.
>
> As we outlined in
> https://blog.zeek.org/2019/04/new-zeek-release-schedule.html, 3.1 will
> be the first ?feature release? which will exist alongside Zeek 3.0
> (which sill still get patches).
>
> I currently plan to update the ?zeek? package to 3.1, and to
> introduce a new zeek-lts package for people who want to stay on 3.0. 
> The
> zeek-lts package will continue to track 3.0 until zeek 4.0 is released 
> -
> at which point zeek-lts will be updated to 4.0. This means with the 
> 4.0
> release zeek-lts and zeek will essentially be the same package - until
> the release of 4.1 when they will diverge again.
>
> Please let me know if you have any feelings about this - if I don?t
> hear back anything I will create the zeek-lts package in the next few
> days - and write another message about it to this thread.
>
> Johanna
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

From akgraner at corelight.com  Thu Mar 12 11:58:47 2020
From: akgraner at corelight.com (Amber Graner)
Date: Thu, 12 Mar 2020 14:58:47 -0400
Subject: [Zeek] Zeek Events Calendar Invites and Reminders Opt-In
In-Reply-To: <CAJhOzupzsmrdd73-tjb+w2zMvdL_ft04NGmcUAydoBL_W6pZVQ@mail.gmail.com>
References: <CAJhOzupzsmrdd73-tjb+w2zMvdL_ft04NGmcUAydoBL_W6pZVQ@mail.gmail.com>
Message-ID: <CAJhOzuotOx=iSR8aYHocdxZTpovF8et4QFa07M-JgwznR7k6vQ@mail.gmail.com>

Hi all,

If you've opt'd in then you should start getting calendar reminders for the
ASK The Zeeksperts and the Monthly Community Calls!

Thanks,
~Amber

On Thu, Mar 5, 2020 at 11:31 AM Amber Graner <akgraner at corelight.com> wrote:

> Hi all,
>
> Since we will no longer be adding the Zeek mailing list to the calendar
> invites, I've created a form for folks to opt-in to receiving calendar
> invites and reminders.
>
> If you would like to continue to receive the calendar invites and
> reminders for the ASK THE ZEEKSPERTS webinars and/or the Monthly Community
> calls please take a moment to add your information to the form (Link below)
>
> https://www.surveymonkey.com/r/X5W5YQZ
>
> If you've already accepted the current invites then you'll continue to get
> them.  I'll remove everyone else.
>
> Thanks in advance.
>
> With gratitude,
> ~Amber
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200312/792a3570/attachment.html 

From jgarciar at sia.es  Wed Mar 18 02:57:15 2020
From: jgarciar at sia.es (Jorge Garcia Rodriguez)
Date: Wed, 18 Mar 2020 09:57:15 +0000
Subject: [Zeek] PF_Ring plugin Issue
Message-ID: <630d0ba4353a438c85d86b4ad2f171b6@sia.es>

Hi guys,

Recently we have been trying different ways to balance the traffic between workers. The thing is that we tried installing the pf_ring plugin, and we installed right. We changed the node.cfg interface to pf_ring::X and it worked but now that we uninstalled the plugin we are having an issue that its like the packets are still going to the pf_ring::X interface but like the plugin is not installed anymore if we put the pf_ring::X interface in the node.cfg, Zeek workers don't start. So, now the only way that Zeek workers start is with the normal interface name but zeek don't see any traffic in this interface.

So it seems that we did something wrong when we uninstalled the plugin. Can someone help me to solve this issue please?

Thank you all.

Best Regards!

Jorge Garc?a Rodr?guez
Technical Consultant
Security Infrastructures
jgarciar at sia.es<mailto:jgarciar at sia.es>
Grupo SIA
Avda.Europa,2 - Alcor Plaza, Edificio B - Parque Oeste Alcorc?n
28922 Alcorc?n - Madrid
Tlf: +34 902 480 580<nxphone:+34%20902%20480%20580>   Fax: +34 91 307 79 80<nxphone:+34%2091%20307%2079%2080>
www.siainternational.com<http://www.siainternational.com/>
delivering value
This e-mail and any attached files are intended solely for the addresse/s identified herein. It may contain confidential and/or legally privileged information and may not necessarily represent the opinion of SIA.
No legally binding commitments will be created by this E-mail message. Where we intend to create legally binding commitments these will be made through hard copy correspondence or documents. If you receive this message by mistake, please immediately notify the sender and delete it since you are not authorized to use, disclose, distribute, print or copy all or part of the contained information Thank you. It is understood that the message was sent to you accidentally, although you appear as the addressee, you can see from the frame of existing relations that you were not the final addressee.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200318/3f7e8682/attachment.html 

From phil at brimsecurity.com  Wed Mar 18 10:32:22 2020
From: phil at brimsecurity.com (Phil Rzewski)
Date: Wed, 18 Mar 2020 10:32:22 -0700
Subject: [Zeek] Avro Schema Availability for Zeek Logs
In-Reply-To: <MN2PR10MB331167ECE1192011605792B2C1FE0@MN2PR10MB3311.namprd10.prod.outlook.com>
References: <MN2PR10MB331167ECE1192011605792B2C1FE0@MN2PR10MB3311.namprd10.prod.outlook.com>
Message-ID: <9998BBE3-03EC-4820-9023-D214A900A65E@brimsecurity.com>

Joseph,

Just chiming in late here with a thought...

If you're using Zeek and Avro with Kafka and Schema Registry <https://github.com/confluentinc/schema-registry> (as I suspect many are), my org put up an open source project up that might be of interest. It's called "zinger" (https://github.com/brimsec/zinger <https://github.com/brimsec/zinger>) and is closely related to "zq" (https://github.com/brimsec/zq <https://github.com/brimsec/zq>), which is another open source project we announced here recently (link <http://mailman.icsi.berkeley.edu/pipermail/zeek/2020-February/015012.html>).

What zinger does is post Avro schemas to the registry dynamically based on what it sees in the Zeek data, in addition to putting Avro-encoded Zeek events onto a Kafka queue. In this regard it's even an improvement over just having a current "stock" Zeek v3 schema, since it will dynamically reflect extra fields/logs it sees in your Zeek data that might have come about as a result of customizations, scripts, and/or plugins you've installed.

If you're in test mode with this stuff, we've got yet another repo called "kavro-demo" (https://github.com/brimsec/kavro-demo <https://github.com/brimsec/kavro-demo>) which creates a simple end-to-end setup with Kafka & Schema Registry along with test producer/consumer scripts, which might be helpful for when you're getting acquainted with zinger.

The zinger project is effectively just a prototype at this point, but we're open to working with people that are testing or starting to use this Kafka/Avro stuff in production. If you or anyone else reading this starts tinkering with it, feel free to ping me via email with questions... or better yet, come find us on our public Slack system <https://join.slack.com/t/brimsec/shared_invite/enQtOTMwMDczODg2ODgyLTk1NTdjOTQxNmI0OGYwOThiYzNlNDc5OWI5NjczZDljNDdmZGZjNGI3NTNiOWRiNzJkMzg4OTEwZWM0Y2NiYWQ> where we're also fielding inquiries about zq.

Good luck!

--
Phil Rzewski
Brim Security


> On Mar 9, 2020, at 11:12 AM, Joseph C Marion <humantargetjoe at hotmail.com> wrote:
> 
> In an effort to save myself a lot of typing, and potentially missing fields, does anyone have a current set of Avro schemas that map to the current built-in logs in Zeek version (3.0.0+)?
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org <mailto:zeek at zeek.org>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek <http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200318/e57309e8/attachment-0001.html 

From justin at corelight.com  Wed Mar 18 12:52:16 2020
From: justin at corelight.com (Justin Azoff)
Date: Wed, 18 Mar 2020 15:52:16 -0400
Subject: [Zeek] PF_Ring plugin Issue
In-Reply-To: <630d0ba4353a438c85d86b4ad2f171b6@sia.es>
References: <630d0ba4353a438c85d86b4ad2f171b6@sia.es>
Message-ID: <CAPfnCujeMGMB_0J=GJesuFvbO2xOrW=aGiAGjqUsrWLfVCgr3g@mail.gmail.com>

If you installed pf_ring and then  built zeek, it's possible that you
ended up compiling zeek against the libpcap from pf_ring, instead of
the standard libpcap.   Try a

    ldd $(which zeek)

and see which libpcap it is using.

In any case, you should try the af_packet plugin.

On Wed, Mar 18, 2020 at 6:06 AM Jorge Garcia Rodriguez <jgarciar at sia.es> wrote:
>
> Hi guys,
>
>
>
> Recently we have been trying different ways to balance the traffic between workers. The thing is that we tried installing the pf_ring plugin, and we installed right. We changed the node.cfg interface to pf_ring::X and it worked but now that we uninstalled the plugin we are having an issue that its like the packets are still going to the pf_ring::X interface but like the plugin is not installed anymore if we put the pf_ring::X interface in the node.cfg, Zeek workers don?t start. So, now the only way that Zeek workers start is with the normal interface name but zeek don?t see any traffic in this interface.
>
>
>
> So it seems that we did something wrong when we uninstalled the plugin. Can someone help me to solve this issue please?
>
>
>
> Thank you all.
>
>
>
> Best Regards!
>
>
>
> Jorge Garc?a Rodr?guez
> Technical Consultant
> Security Infrastructures
> jgarciar at sia.es
>
> Grupo SIA
> Avda.Europa,2 - Alcor Plaza, Edificio B - Parque Oeste Alcorc?n
> 28922 Alcorc?n - Madrid
> Tlf: +34 902 480 580   Fax: +34 91 307 79 80
> www.siainternational.com
>
> delivering value
>
> This e-mail and any attached files are intended solely for the addresse/s identified herein. It may contain confidential and/or legally privileged information and may not necessarily represent the opinion of SIA.
>
> No legally binding commitments will be created by this E-mail message. Where we intend to create legally binding commitments these will be made through hard copy correspondence or documents. If you receive this message by mistake, please immediately notify the sender and delete it since you are not authorized to use, disclose, distribute, print or copy all or part of the contained information Thank you. It is understood that the message was sent to you accidentally, although you appear as the addressee, you can see from the frame of existing relations that you were not the final addressee.
>
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
Justin


From michalpurzynski1 at gmail.com  Wed Mar 18 14:50:24 2020
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Wed, 18 Mar 2020 14:50:24 -0700
Subject: [Zeek] PF_Ring plugin Issue
In-Reply-To: <CAPfnCujeMGMB_0J=GJesuFvbO2xOrW=aGiAGjqUsrWLfVCgr3g@mail.gmail.com>
References: <630d0ba4353a438c85d86b4ad2f171b6@sia.es>
	<CAPfnCujeMGMB_0J=GJesuFvbO2xOrW=aGiAGjqUsrWLfVCgr3g@mail.gmail.com>
Message-ID: <CAJ6bFK2DB=X7BA1O26YqmcJStJA4JmJcMbs3J7FMZKrXW_wRGQ@mail.gmail.com>

Is there any part of pf_ring functionality that you cannot find in the
af_packet capture mechanism?

On Wed, Mar 18, 2020 at 12:56 PM Justin Azoff <justin at corelight.com> wrote:

> If you installed pf_ring and then  built zeek, it's possible that you
> ended up compiling zeek against the libpcap from pf_ring, instead of
> the standard libpcap.   Try a
>
>     ldd $(which zeek)
>
> and see which libpcap it is using.
>
> In any case, you should try the af_packet plugin.
>
> On Wed, Mar 18, 2020 at 6:06 AM Jorge Garcia Rodriguez <jgarciar at sia.es>
> wrote:
> >
> > Hi guys,
> >
> >
> >
> > Recently we have been trying different ways to balance the traffic
> between workers. The thing is that we tried installing the pf_ring plugin,
> and we installed right. We changed the node.cfg interface to pf_ring::X and
> it worked but now that we uninstalled the plugin we are having an issue
> that its like the packets are still going to the pf_ring::X interface but
> like the plugin is not installed anymore if we put the pf_ring::X interface
> in the node.cfg, Zeek workers don?t start. So, now the only way that Zeek
> workers start is with the normal interface name but zeek don?t see any
> traffic in this interface.
> >
> >
> >
> > So it seems that we did something wrong when we uninstalled the plugin.
> Can someone help me to solve this issue please?
> >
> >
> >
> > Thank you all.
> >
> >
> >
> > Best Regards!
> >
> >
> >
> > Jorge Garc?a Rodr?guez
> > Technical Consultant
> > Security Infrastructures
> > jgarciar at sia.es
> >
> > Grupo SIA
> > Avda.Europa,2 - Alcor Plaza, Edificio B - Parque Oeste Alcorc?n
> > 28922 Alcorc?n - Madrid
> > Tlf: +34 902 480 580   Fax: +34 91 307 79 80
> > www.siainternational.com
> >
> > delivering value
> >
> > This e-mail and any attached files are intended solely for the
> addresse/s identified herein. It may contain confidential and/or legally
> privileged information and may not necessarily represent the opinion of SIA.
> >
> > No legally binding commitments will be created by this E-mail message.
> Where we intend to create legally binding commitments these will be made
> through hard copy correspondence or documents. If you receive this message
> by mistake, please immediately notify the sender and delete it since you
> are not authorized to use, disclose, distribute, print or copy all or part
> of the contained information Thank you. It is understood that the message
> was sent to you accidentally, although you appear as the addressee, you can
> see from the frame of existing relations that you were not the final
> addressee.
> >
> >
> >
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
>
> --
> Justin
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200318/1463475f/attachment.html 

From johanna at corelight.com  Thu Mar 19 13:03:56 2020
From: johanna at corelight.com (Johanna Amann)
Date: Thu, 19 Mar 2020 13:03:56 -0700
Subject: [Zeek] Binary packages and 3.1
In-Reply-To: <FC49B9F0-88AF-4342-90F5-0EC478B5FDF2@corelight.com>
References: <E734636A-53E3-4115-8E9B-6D519C187CE6@corelight.com>
	<FC49B9F0-88AF-4342-90F5-0EC478B5FDF2@corelight.com>
Message-ID: <D32C4380-5DE1-4A85-B581-92191C0AE852@corelight.com>

Hello Everyone,

this changeover happened now; the zeek package now is at version 3.1.1; 
zeek 3.0.3 is still available in the zeek-lts package.

Due to the fact that Zeek 3.1 requires newer C++ compilers, we also 
dropped support for a number of distributions. The zeek package is now 
no longer available for:

CentOS 7, Debian 9, Raspbian 9, SLE 12 SP4, SLE 12 SP5, Ubuntu 14.04

(zeek-lts is still and will continue to be available for these 
distributions).

For reference, the current list of distributions that we build the zeek 
package for are:

Debian 10, Debian Testing, Fedora 20, Fedora 30, Fedora 31, Raspbian 10, 
SLE 15, SLE 15 SP1, OpenSUSE Lep 15.1, OpenSUSE Leap 15.2, OpenSUSE 
Tumbleweeb, Ubuntu 18.04, Ubuntu 18.10, Ubuntu 19.04, Ubuntu 19.10.

Johanna

On 12 Mar 2020, at 10:21, Johanna Amann wrote:

> Hello Everyone,
>
> I have a few small updates on the migration of the binary packages to
> 3.1.
>
> As announced in the last email, there is now a ?zeek-lts? package.
> This package will track the LTS releases of Zeek - and currently
> contains Zeek 3.0.3.
>
> As of today, the ?zeek? package also is still on 3.0.3. Sometime
> next week I will switch the zeek package over to 3.1.
>
> If you want are using our current binary packages and want to remain 
> on
> the LTS release of zeek, the only thing you have to do is to perform a
>
> ?apt install zeek-lts? on debian-based systems, or a
>
> ?yum/dnf install zeek-lts --allowerasing?
>
> This will install zeek-lts and remove the conflicting zeek package. 
> The
> install locations for zeek-lts and zeek are exactly the same
> (/opt/zeek), so nothing else will change.
>
> Please let me know if you encounter any problems,
>   Johanna
>
> On 7 Feb 2020, at 11:18, Johanna Amann wrote:
>
>> Hi Everyone,
>>
>> with the Zeek 3.1 release around the corner, I just wanted to outline
>> my
>> current plan for the binary packages.
>>
>> As we outlined in
>> https://blog.zeek.org/2019/04/new-zeek-release-schedule.html, 3.1 
>> will
>> be the first ?feature release? which will exist alongside Zeek 
>> 3.0
>> (which sill still get patches).
>>
>> I currently plan to update the ?zeek? package to 3.1, and to
>> introduce a new zeek-lts package for people who want to stay on 3.0.
>> The
>> zeek-lts package will continue to track 3.0 until zeek 4.0 is 
>> released
>> -
>> at which point zeek-lts will be updated to 4.0. This means with the
>> 4.0
>> release zeek-lts and zeek will essentially be the same package - 
>> until
>> the release of 4.1 when they will diverge again.
>>
>> Please let me know if you have any feelings about this - if I don?t
>> hear back anything I will create the zeek-lts package in the next few
>> days - and write another message about it to this thread.
>>
>> Johanna
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

From johanna at corelight.com  Thu Mar 19 13:26:24 2020
From: johanna at corelight.com (Johanna Amann)
Date: Thu, 19 Mar 2020 13:26:24 -0700
Subject: [Zeek] Binary packages and 3.1
In-Reply-To: <D32C4380-5DE1-4A85-B581-92191C0AE852@corelight.com>
References: <E734636A-53E3-4115-8E9B-6D519C187CE6@corelight.com>
	<FC49B9F0-88AF-4342-90F5-0EC478B5FDF2@corelight.com>
	<D32C4380-5DE1-4A85-B581-92191C0AE852@corelight.com>
Message-ID: <D865ACF0-F416-454A-A1F9-E4C309D5E269@corelight.com>

> Fedora 20

I meant Fedora 29 here, 20 was a while ago :)

Johanna

From jsbarber60 at gmail.com  Thu Mar 19 13:37:01 2020
From: jsbarber60 at gmail.com (Jeff Barber)
Date: Thu, 19 Mar 2020 14:37:01 -0600
Subject: [Zeek] binpac with PDUs in multiple segments
Message-ID: <CABLqKGKAnadK+9eCMwhhvWnDdUrysEphh=TTP2gxBa3odmUHqw@mail.gmail.com>

Hi all,

I'm attempting to use binpac to create an analyzer that works with packets
where the application protocol data unit might be broken across multiple
TCP segments. Specifically, trying to handle TPKT/COTP in a reasonably
robust way, and it's not working as I expected. Trying to figure out if I'm
doing it wrong, or if my expectation is simply wrong, or if it's broken.
I've pulled the key bits out into a separate skeleton analyzer.

Here's my .pac:

type HDR = record {
    version:    uint8;
    reserved:   uint8;
    len:        uint16;
} &byteorder=bigendian;

type FOO_PDU(is_orig: bool) = record {
    hdr:        HDR;
    plen:       uint8;
    ptype:      uint8;
    something:  bytestring &restofdata;
} &byteorder=bigendian, &length=hdr.len;

flow FOO_Flow(is_orig: bool) {
   flowunit = FOO_PDU(is_orig) withcontext(connection, this);
   # datagram = FOO_PDU(is_orig) withcontext(connection, this);
};


The trace file I have has the HDR part (4 bytes: actually a TPKT) in the
first TCP segment, and the remainder in the next packet. If I run it, it
throws an ExceptionOutOfBound with the message:

FOO_PDU:ptype: 6 > 4


Accurate enough, but my expectation was the flowunit together with the
&length clause should cause the analyzer to first accumulate enough data
for the HDR part, then dereference the length embedded there, then
accumulate more data until the &length provided by the header field is
fulfilled in order to fill out the rest of the structure. But that doesn't
happen. (Indeed you can see in the generated .cc that hdr.len is never
evaluated.)

I tried putting a support analyzer in front of it that first accumulates
the correct number of bytes before passing to the binpac code. If I do
that, then remove the &length clause and change the flow to use the datagram
= FOO_PDU clause, it works correctly. But isn't that what the flowunit is
supposed to do? That's what the documentation page implies:
https://github.com/zeek/binpac#id4

So, really the question is: should this work? And if so, what am I doing
wrong?

You can pull and build my code from here if you like:
https://github.com/jsbarber/binpac-fail
The rest of the code is pretty much straight from the binpac quickstart.
There is a pcap in the traces directory that invokes the analyzer.

Any help in understanding this greatly appreciated.
Jeff
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200319/73ca7388/attachment-0001.html 

From jsiwek at corelight.com  Thu Mar 19 22:25:27 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 19 Mar 2020 22:25:27 -0700
Subject: [Zeek] binpac with PDUs in multiple segments
In-Reply-To: <CABLqKGKAnadK+9eCMwhhvWnDdUrysEphh=TTP2gxBa3odmUHqw@mail.gmail.com>
References: <CABLqKGKAnadK+9eCMwhhvWnDdUrysEphh=TTP2gxBa3odmUHqw@mail.gmail.com>
Message-ID: <CAMzgZ0LEjBORU9g8Rz7kTwkcG2w+Z3HnuxcvU6Li13R4B4C4pA@mail.gmail.com>

On Thu, Mar 19, 2020 at 1:39 PM Jeff Barber <jsbarber60 at gmail.com> wrote:

> So, really the question is: should this work? And if so, what am I doing wrong?

I didn't notice anything wrong with what you were trying, but I did
find the parsing logic generated by binpac was wrong.  I made a PR
with a fix (and using your test case) here:

    https://github.com/zeek/zeek/pull/873

Specifically this binpac patch in case you want to use it and work
ahead until the bugfix gets reviewed/merged into Zeek.

    https://github.com/zeek/binpac/commit/31203a3458baa6bdfc355e260a102f9402fbe3e2

- Jon

From johanna at corelight.com  Fri Mar 20 13:13:22 2020
From: johanna at corelight.com (Johanna Amann)
Date: Fri, 20 Mar 2020 13:13:22 -0700
Subject: [Zeek] Binary packages and 3.1
In-Reply-To: <D32C4380-5DE1-4A85-B581-92191C0AE852@corelight.com>
References: <E734636A-53E3-4115-8E9B-6D519C187CE6@corelight.com>
	<FC49B9F0-88AF-4342-90F5-0EC478B5FDF2@corelight.com>
	<D32C4380-5DE1-4A85-B581-92191C0AE852@corelight.com>
Message-ID: <81015B17-CE83-465E-9C3D-9F69AF35FA1D@corelight.com>

In additional news - after a request and a bit of digging there now are 
CentOS 8 builds.

And since I did not include it before - all download links are listed at 
https://old.zeek.org/download/packages.html

Johanna

On 19 Mar 2020, at 13:03, Johanna Amann wrote:

> Hello Everyone,
>
> this changeover happened now; the zeek package now is at version 
> 3.1.1;
> zeek 3.0.3 is still available in the zeek-lts package.
>
> Due to the fact that Zeek 3.1 requires newer C++ compilers, we also
> dropped support for a number of distributions. The zeek package is now
> no longer available for:
>
> CentOS 7, Debian 9, Raspbian 9, SLE 12 SP4, SLE 12 SP5, Ubuntu 14.04
>
> (zeek-lts is still and will continue to be available for these
> distributions).
>
> For reference, the current list of distributions that we build the 
> zeek
> package for are:
>
> Debian 10, Debian Testing, Fedora 20, Fedora 30, Fedora 31, Raspbian 
> 10,
> SLE 15, SLE 15 SP1, OpenSUSE Lep 15.1, OpenSUSE Leap 15.2, OpenSUSE
> Tumbleweeb, Ubuntu 18.04, Ubuntu 18.10, Ubuntu 19.04, Ubuntu 19.10.
>
> Johanna
>
> On 12 Mar 2020, at 10:21, Johanna Amann wrote:
>
>> Hello Everyone,
>>
>> I have a few small updates on the migration of the binary packages to
>> 3.1.
>>
>> As announced in the last email, there is now a ?zeek-lts? 
>> package.
>> This package will track the LTS releases of Zeek - and currently
>> contains Zeek 3.0.3.
>>
>> As of today, the ?zeek? package also is still on 3.0.3. Sometime
>> next week I will switch the zeek package over to 3.1.
>>
>> If you want are using our current binary packages and want to remain
>> on
>> the LTS release of zeek, the only thing you have to do is to perform 
>> a
>>
>> ?apt install zeek-lts? on debian-based systems, or a
>>
>> ?yum/dnf install zeek-lts --allowerasing?
>>
>> This will install zeek-lts and remove the conflicting zeek package.
>> The
>> install locations for zeek-lts and zeek are exactly the same
>> (/opt/zeek), so nothing else will change.
>>
>> Please let me know if you encounter any problems,
>>   Johanna
>>
>> On 7 Feb 2020, at 11:18, Johanna Amann wrote:
>>
>>> Hi Everyone,
>>>
>>> with the Zeek 3.1 release around the corner, I just wanted to 
>>> outline
>>> my
>>> current plan for the binary packages.
>>>
>>> As we outlined in
>>> https://blog.zeek.org/2019/04/new-zeek-release-schedule.html, 3.1
>>> will
>>> be the first ?feature release? which will exist alongside Zeek
>>> 3.0
>>> (which sill still get patches).
>>>
>>> I currently plan to update the ?zeek? package to 3.1, and to
>>> introduce a new zeek-lts package for people who want to stay on 3.0.
>>> The
>>> zeek-lts package will continue to track 3.0 until zeek 4.0 is
>>> released
>>> -
>>> at which point zeek-lts will be updated to 4.0. This means with the
>>> 4.0
>>> release zeek-lts and zeek will essentially be the same package -
>>> until
>>> the release of 4.1 when they will diverge again.
>>>
>>> Please let me know if you have any feelings about this - if I 
>>> don?t
>>> hear back anything I will create the zeek-lts package in the next 
>>> few
>>> days - and write another message about it to this thread.
>>>
>>> Johanna
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

From Kayode_Enwerem at ao.uscourts.gov  Fri Mar 20 15:19:54 2020
From: Kayode_Enwerem at ao.uscourts.gov (Kayode Enwerem)
Date: Fri, 20 Mar 2020 22:19:54 +0000
Subject: [Zeek] Issue installing af_packet plugin
Message-ID: <BY5PR09MB537783D976B37FEBF77601CEA1F50@BY5PR09MB5377.namprd09.prod.outlook.com>

Hello,

I just finished installing the latest version on zeek (zeek version 3.1.1). Now I am trying install af_packet plugin to use with zeek but I am encountering the following issues:

When I run:
zkg install zeek/j-gras/bro-af_packet-plugin

I get the following error message:

The following packages will be INSTALLED:
  zeek/j-gras/bro-af_packet-plugin (2.0.0)

Proceed? [Y/n] Y
Running unit tests for "zeek/j-gras/bro-af_packet-plugin"

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/bin/zkg", line 355, in run
    self.package_name, self.package_version)
  File "/usr/lib/python2.7/site-packages/zeekpkg/manager.py", line 2203, in install
    return self._install(matches[0], version)
  File "/usr/lib/python2.7/site-packages/zeekpkg/manager.py", line 2265, in _install
    status.current_hash = clone.head.object.hexsha
  File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 190, in _get_object
    return Object.new_from_sha(self.repo, hex_to_bin(self.dereference_recursive(self.repo, self.path)))
  File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 132, in dereference_recursive
    hexsha, ref_path = cls._get_ref_info(repo, ref_path)
  File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 181, in _get_ref_info
    return cls._get_ref_info_helper(repo, ref_path)
  File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 145, in _get_ref_info_helper
    with open(osp.join(repodir, ref_path), 'rt', encoding='UTF-8') as fp:
TypeError: 'encoding' is an invalid keyword argument for this function

Traceback (most recent call last):
  File "/bin/zkg", line 2243, in <module>
    main()
  File "/bin/zkg", line 2239, in main
    args.run_cmd(manager, args, config, configfile)
  File "/bin/zkg", line 599, in cmd_install
    print('Installed "{}" ({})'.format(name, ipkg.status.current_version))
AttributeError: 'NoneType' object has no attribute 'status'

Can someone please help me with this?

Thanks in advance
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200320/fc4dbc9e/attachment-0001.html 

From jan.grashoefer at gmail.com  Mon Mar 23 02:27:08 2020
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Mon, 23 Mar 2020 10:27:08 +0100
Subject: [Zeek] Issue installing af_packet plugin
In-Reply-To: <BY5PR09MB537783D976B37FEBF77601CEA1F50@BY5PR09MB5377.namprd09.prod.outlook.com>
References: <BY5PR09MB537783D976B37FEBF77601CEA1F50@BY5PR09MB5377.namprd09.prod.outlook.com>
Message-ID: <b46daf01-85ca-0f26-4ae5-ca2f6c0df2b1@gmail.com>

To me this looks like an zkg related issue. It seems to require python 3 
but is run with 2.7 in your setup.

Jan

On 20/03/2020 23:19, Kayode Enwerem wrote:
> Hello,
> 
> I just finished installing the latest version on zeek (zeek version 
> 3.1.1). Now I am trying install af_packet plugin to use with zeek but I 
> am encountering the following issues:
> 
> When I run:
> 
> *zkg install zeek/j-gras/bro-af_packet-plugin*
> 
> I get the following error message:
> 
> The following packages will be INSTALLED:
> 
>  ? zeek/j-gras/bro-af_packet-plugin (2.0.0)
> 
> Proceed? [Y/n] Y
> 
> Running unit tests for "zeek/j-gras/bro-af_packet-plugin"
> 
> Exception in thread Thread-1:
> 
> Traceback (most recent call last):
> 
>  ? File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
> 
>  ??? self.run()
> 
>  ? File "/bin/zkg", line 355, in run
> 
>  ??? self.package_name, self.package_version)
> 
>  ? File "/usr/lib/python2.7/site-packages/zeekpkg/manager.py", line 
> 2203, in install
> 
>  ??? return self._install(matches[0], version)
> 
>  ? File "/usr/lib/python2.7/site-packages/zeekpkg/manager.py", line 
> 2265, in _install
> 
>  ?? ?status.current_hash = clone.head.object.hexsha
> 
>  ? File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 
> 190, in _get_object
> 
>  ??? return Object.new_from_sha(self.repo, 
> hex_to_bin(self.dereference_recursive(self.repo, self.path)))
> 
>  ? File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 
> 132, in dereference_recursive
> 
>  ??? hexsha, ref_path = cls._get_ref_info(repo, ref_path)
> 
>  ? File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 
> 181, in _get_ref_info
> 
>  ??? return cls._get_ref_info_helper(repo, ref_path)
> 
>  ? File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 
> 145, in _get_ref_info_helper
> 
>  ??? with open(osp.join(repodir, ref_path), 'rt', encoding='UTF-8') as fp:
> 
> TypeError: 'encoding' is an invalid keyword argument for this function
> 
> Traceback (most recent call last):
> 
>  ? File "/bin/zkg", line 2243, in <module>
> 
>  ??? main()
> 
>  ? File "/bin/zkg", line 2239, in main
> 
>  ??? args.run_cmd(manager, args, config, configfile)
> 
>  ? File "/bin/zkg", line 599, in cmd_install
> 
>  ??? print('Installed "{}" ({})'.format(name, ipkg.status.current_version))
> 
> AttributeError: 'NoneType' object has no attribute 'status'
> 
> Can someone please help me with this?
> 
> Thanks in advance
> 
> 
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> 

From robin at corelight.com  Mon Mar 23 04:26:33 2020
From: robin at corelight.com (Robin Sommer)
Date: Mon, 23 Mar 2020 11:26:33 +0000
Subject: [Zeek] Announcing the Zeek Agent
Message-ID: <20200323112633.GA71036@corelight.com>

We are happy to announce the new "Zeek Agent", an open source endpoint
agent that turns host activity into Zeek events as it happens:

    https://zeek.org/2020/03/23/announcing-the-zeek-agent

The agent currently supports Linux and macOS, with Windows being in
the works; and it integrates with osquery so that Zeek gains access to
all of the osquery tables.

This remains an early version still, and we're interested in feedback.
We've been developing the agent on GitHub for a while already, and I
know some of you folks have already been trying it out. If you haven't
yet, let us know how it goes when you get a chance. 

Robin

-- 
Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com

From Kayode_Enwerem at ao.uscourts.gov  Mon Mar 23 06:06:02 2020
From: Kayode_Enwerem at ao.uscourts.gov (Kayode Enwerem)
Date: Mon, 23 Mar 2020 13:06:02 +0000
Subject: [Zeek] Issue installing af_packet plugin
In-Reply-To: <b46daf01-85ca-0f26-4ae5-ca2f6c0df2b1@gmail.com>
References: <BY5PR09MB537783D976B37FEBF77601CEA1F50@BY5PR09MB5377.namprd09.prod.outlook.com>
	<b46daf01-85ca-0f26-4ae5-ca2f6c0df2b1@gmail.com>
Message-ID: <BY5PR09MB5377CBB6E929492E97AB8324A1F00@BY5PR09MB5377.namprd09.prod.outlook.com>

Any idea how I can resolve the issue?

I used alternatives to switch to using "Python 3" but still getting the same error message.


-----Original Message-----
From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> On Behalf Of Jan Grash?fer
Sent: Monday, March 23, 2020 5:27 AM
To: zeek at zeek.org
Subject: Re: [Zeek] Issue installing af_packet plugin

To me this looks like an zkg related issue. It seems to require python 3 but is run with 2.7 in your setup.

Jan

On 20/03/2020 23:19, Kayode Enwerem wrote:
> Hello,
> 
> I just finished installing the latest version on zeek (zeek version 
> 3.1.1). Now I am trying install af_packet plugin to use with zeek but 
> I am encountering the following issues:
> 
> When I run:
> 
> *zkg install zeek/j-gras/bro-af_packet-plugin*
> 
> I get the following error message:
> 
> The following packages will be INSTALLED:
> 
>  ? zeek/j-gras/bro-af_packet-plugin (2.0.0)
> 
> Proceed? [Y/n] Y
> 
> Running unit tests for "zeek/j-gras/bro-af_packet-plugin"
> 
> Exception in thread Thread-1:
> 
> Traceback (most recent call last):
> 
>  ? File "/usr/lib64/python2.7/threading.py", line 812, in 
> __bootstrap_inner
> 
>  ??? self.run()
> 
>  ? File "/bin/zkg", line 355, in run
> 
>  ??? self.package_name, self.package_version)
> 
>  ? File "/usr/lib/python2.7/site-packages/zeekpkg/manager.py", line 
> 2203, in install
> 
>  ??? return self._install(matches[0], version)
> 
>  ? File "/usr/lib/python2.7/site-packages/zeekpkg/manager.py", line 
> 2265, in _install
> 
>  ?? ?status.current_hash = clone.head.object.hexsha
> 
>  ? File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 
> 190, in _get_object
> 
>  ??? return Object.new_from_sha(self.repo, 
> hex_to_bin(self.dereference_recursive(self.repo, self.path)))
> 
>  ? File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 
> 132, in dereference_recursive
> 
>  ??? hexsha, ref_path = cls._get_ref_info(repo, ref_path)
> 
>  ? File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 
> 181, in _get_ref_info
> 
>  ??? return cls._get_ref_info_helper(repo, ref_path)
> 
>  ? File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 
> 145, in _get_ref_info_helper
> 
>  ??? with open(osp.join(repodir, ref_path), 'rt', encoding='UTF-8') as fp:
> 
> TypeError: 'encoding' is an invalid keyword argument for this function
> 
> Traceback (most recent call last):
> 
>  ? File "/bin/zkg", line 2243, in <module>
> 
>  ??? main()
> 
>  ? File "/bin/zkg", line 2239, in main
> 
>  ??? args.run_cmd(manager, args, config, configfile)
> 
>  ? File "/bin/zkg", line 599, in cmd_install
> 
>  ??? print('Installed "{}" ({})'.format(name, 
> ipkg.status.current_version))
> 
> AttributeError: 'NoneType' object has no attribute 'status'
> 
> Can someone please help me with this?
> 
> Thanks in advance
> 
> 
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> 
_______________________________________________
Zeek mailing list
zeek at zeek.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From Kayode_Enwerem at ao.uscourts.gov  Mon Mar 23 09:38:22 2020
From: Kayode_Enwerem at ao.uscourts.gov (Kayode Enwerem)
Date: Mon, 23 Mar 2020 16:38:22 +0000
Subject: [Zeek] Issue installing af_packet plugin
In-Reply-To: <BY5PR09MB5377CBB6E929492E97AB8324A1F00@BY5PR09MB5377.namprd09.prod.outlook.com>
References: <BY5PR09MB537783D976B37FEBF77601CEA1F50@BY5PR09MB5377.namprd09.prod.outlook.com>
	<b46daf01-85ca-0f26-4ae5-ca2f6c0df2b1@gmail.com>
	<BY5PR09MB5377CBB6E929492E97AB8324A1F00@BY5PR09MB5377.namprd09.prod.outlook.com>
Message-ID: <BY5PR09MB5377B9023896DB8D2989F36BA1F00@BY5PR09MB5377.namprd09.prod.outlook.com>

RESOLVED. 

I was able to resolve this by using python3.

Thanks

-----Original Message-----
From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> On Behalf Of Kayode Enwerem
Sent: Monday, March 23, 2020 9:06 AM
To: Jan Grash?fer <jan.grashoefer at gmail.com>; zeek at zeek.org
Subject: Re: [Zeek] Issue installing af_packet plugin

Any idea how I can resolve the issue?

I used alternatives to switch to using "Python 3" but still getting the same error message.


-----Original Message-----
From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> On Behalf Of Jan Grash?fer
Sent: Monday, March 23, 2020 5:27 AM
To: zeek at zeek.org
Subject: Re: [Zeek] Issue installing af_packet plugin

To me this looks like an zkg related issue. It seems to require python 3 but is run with 2.7 in your setup.

Jan

On 20/03/2020 23:19, Kayode Enwerem wrote:
> Hello,
> 
> I just finished installing the latest version on zeek (zeek version 
> 3.1.1). Now I am trying install af_packet plugin to use with zeek but 
> I am encountering the following issues:
> 
> When I run:
> 
> *zkg install zeek/j-gras/bro-af_packet-plugin*
> 
> I get the following error message:
> 
> The following packages will be INSTALLED:
> 
>  ? zeek/j-gras/bro-af_packet-plugin (2.0.0)
> 
> Proceed? [Y/n] Y
> 
> Running unit tests for "zeek/j-gras/bro-af_packet-plugin"
> 
> Exception in thread Thread-1:
> 
> Traceback (most recent call last):
> 
>  ? File "/usr/lib64/python2.7/threading.py", line 812, in 
> __bootstrap_inner
> 
>  ??? self.run()
> 
>  ? File "/bin/zkg", line 355, in run
> 
>  ??? self.package_name, self.package_version)
> 
>  ? File "/usr/lib/python2.7/site-packages/zeekpkg/manager.py", line 
> 2203, in install
> 
>  ??? return self._install(matches[0], version)
> 
>  ? File "/usr/lib/python2.7/site-packages/zeekpkg/manager.py", line 
> 2265, in _install
> 
>  ?? ?status.current_hash = clone.head.object.hexsha
> 
>  ? File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 
> 190, in _get_object
> 
>  ??? return Object.new_from_sha(self.repo, 
> hex_to_bin(self.dereference_recursive(self.repo, self.path)))
> 
>  ? File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 
> 132, in dereference_recursive
> 
>  ??? hexsha, ref_path = cls._get_ref_info(repo, ref_path)
> 
>  ? File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 
> 181, in _get_ref_info
> 
>  ??? return cls._get_ref_info_helper(repo, ref_path)
> 
>  ? File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 
> 145, in _get_ref_info_helper
> 
>  ??? with open(osp.join(repodir, ref_path), 'rt', encoding='UTF-8') as fp:
> 
> TypeError: 'encoding' is an invalid keyword argument for this function
> 
> Traceback (most recent call last):
> 
>  ? File "/bin/zkg", line 2243, in <module>
> 
>  ??? main()
> 
>  ? File "/bin/zkg", line 2239, in main
> 
>  ??? args.run_cmd(manager, args, config, configfile)
> 
>  ? File "/bin/zkg", line 599, in cmd_install
> 
>  ??? print('Installed "{}" ({})'.format(name,
> ipkg.status.current_version))
> 
> AttributeError: 'NoneType' object has no attribute 'status'
> 
> Can someone please help me with this?
> 
> Thanks in advance
> 
> 
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> 
_______________________________________________
Zeek mailing list
zeek at zeek.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

_______________________________________________
Zeek mailing list
zeek at zeek.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From ttomek.koziak at gmail.com  Mon Mar 23 15:09:59 2020
From: ttomek.koziak at gmail.com (Tomek Koziak)
Date: Mon, 23 Mar 2020 23:09:59 +0100
Subject: [Zeek] Is it possible to inspect TCP reserved bits with Zeek?
Message-ID: <CAF=fF7pOWFcUMM+iy7UwVSHG8bwwHK6goOT+kh9YWJe0XC_ArA@mail.gmail.com>

Hi All,

I'm testing Zeek/Bro capabilities in terms of detecting different types of
steganography. After working with the ICMP protocol now I am trying to
inspect the TCP protocol. I want to detect if the reserved bits in TCP are
changed with help of TCP events. Unfortunately without success.

Is it possible to inspect TCP reserved bits with Zeek events? If not is
there any other possible way to detect wheter those bits where changed?

Best regards,
Tomasz Koziak
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200323/9d93767a/attachment.html 

From clopmz at outlook.com  Tue Mar 24 02:45:17 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Tue, 24 Mar 2020 09:45:17 +0000
Subject: [Zeek] Anyone using Bro doctor plugin?
Message-ID: <4D39916F-959B-477A-B4A1-46BC506AE755@outlook.com>

Hi all,

I have enable bro doctor plugin in my Zeek 3.0.3 cluster and I see the following error:

###################################################################
# Checking if connections are unevenly distributed across workers #
###################################################################
error: Traceback (most recent call last):
  File "/usr/lib64/python3.6/cmd.py", line 214, in onecmd
    func = getattr(self, 'do_' + cmd)
AttributeError: 'ZeekCtlCmdLoop' object has no attribute 'do_doctor'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 596, in cmd_custom
    results.ok = f() and results.ok
  File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 457, in check_connection_distribution
    variance = reduce(lambda var, cnt: var + (cnt - mean)**2, nodes.values(), 0) / len(nodes)
NameError: name 'reduce' is not defined

All other doctor options works ok, but not this one ? Is it a bug? Do I need to install some other python module? Zeek is running as unprivileged user ?

--
Regards,
C. L. Martinez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200324/ed987b8e/attachment.html 

From pierre at droids-corp.org  Tue Mar 24 04:13:04 2020
From: pierre at droids-corp.org (Pierre LALET)
Date: Tue, 24 Mar 2020 12:13:04 +0100
Subject: [Zeek] Anyone using Bro doctor plugin?
In-Reply-To: <4D39916F-959B-477A-B4A1-46BC506AE755@outlook.com>
References: <4D39916F-959B-477A-B4A1-46BC506AE755@outlook.com>
Message-ID: <20200324111304.GA11896@droids-corp.org>

Hi there,

> Traceback (most recent call last):
>   File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 596, in cmd_custom
>     results.ok = f() and results.ok
>   File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 457, in check_connection_distribution
>     variance = reduce(lambda var, cnt: var + (cnt - mean)**2, nodes.values(), 0) / len(nodes)
> NameError: name 'reduce' is not defined

This particular error may be related to a Python 2 / Python 3 issue:
reduce() was a built-in in Python 2 and is part of the functools
module in Python3.

Hope this helps,

Pierre

From akgraner at corelight.com  Tue Mar 24 06:28:51 2020
From: akgraner at corelight.com (Amber Graner)
Date: Tue, 24 Mar 2020 09:28:51 -0400
Subject: [Zeek] Ask The Zeeksperts - Reminders - Next Webinar is 26 March
	2020
Message-ID: <CAJhOzuqM6+6wwPC6Z=xpZT=o4xew0n2CjcRfCLSRY4jY1-U8Sg@mail.gmail.com>

Hi all,

 Just a reminder that this week's ASK THE ZEEKSPERTS webinar will be hosted
by Seth Hall on Thursday 26 March at 12:30pm PST/3:30pm EST.

==========================
What is ASK THE ZEEKSPERTS?
==========================

This is an hour long bi-monthly webinarl for the open-source Zeek (formerly
Bro) community to interface directly with leading contributors to the
open-source project and ask questions live to better understand, expand or
troubleshoot deployments of the network security monitoring software.

===================
How to join the Webinar
===================

These webinars are free, but registration is required.  You can register
at:

https://attendee.gotowebinar.com/register/5207290772886395917

=====================
Calendar Reminder Opt-In
=====================

If you would like to receive calendar reminders and you haven't opt'd in,
but would like to do so, please fill out the following form:

https://www.surveymonkey.com/r/X5W5YQZ

===============================
Zeek ask-the zeeksperts Slack Channel
===============================

If you'd like to ask questions ahead of the webinar, so that the host will
have time to prepare for your questions you can do so by replying to this
email or joining the #ask-the-zeeksperts slack channel.

If you haven't joined the Slack channel yet you can do so with the link
below:

http://bit.ly/ZeekOrgSlackInvite

=================================
Future ASK THE ZEEKSPERTS Webinars
=================================

* April 9th - Host Fatema Bannat Wala -
https://attendee.gotowebinar.com/register/2632319203581363981

* April 23 - Host TBD
- https://attendee.gotowebinar.com/register/1763308093940786957

* May 14 - Host TBD
- https://attendee.gotowebinar.com/register/328979782344155149

We look forward to your questions and connecting with you on the ASK THE
ZEEKSPERTS calls.

Please let us know if you have any questions.

Thanks,
~Amber
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200324/c49d02b9/attachment.html 

From jsiwek at corelight.com  Tue Mar 24 10:57:26 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Tue, 24 Mar 2020 10:57:26 -0700
Subject: [Zeek] Is it possible to inspect TCP reserved bits with Zeek?
In-Reply-To: <CAF=fF7pOWFcUMM+iy7UwVSHG8bwwHK6goOT+kh9YWJe0XC_ArA@mail.gmail.com>
References: <CAF=fF7pOWFcUMM+iy7UwVSHG8bwwHK6goOT+kh9YWJe0XC_ArA@mail.gmail.com>
Message-ID: <CAMzgZ0JH84vz9enBay=JFo19Ad62Vy6jRvpfQYQTT_ArYEVfHA@mail.gmail.com>

On Mon, Mar 23, 2020 at 3:11 PM Tomek Koziak <ttomek.koziak at gmail.com> wrote:

> Is it possible to inspect TCP reserved bits with Zeek events? If not is there any other possible way to detect wheter those bits where changed?

I didn't see any events that currently carry the reserved bits, but it
would be simple to extend existing ones like `new_packet` and
`raw_packet`.  You can find an example patch for that in the
`topic/jsiwek/tcp-hdr-reserved-bits` branch here:

    https://github.com/zeek/zeek/compare/topic/jsiwek/tcp-hdr-reserved-bits

Let me know if that works for your purposes and I'll turn it into a
pull request.

- Jon

From phil at brimsecurity.com  Tue Mar 24 17:10:33 2020
From: phil at brimsecurity.com (Phil Rzewski)
Date: Tue, 24 Mar 2020 17:10:33 -0700
Subject: [Zeek] Brim application for Zeek logs & packet captures
In-Reply-To: <4D066D96-61FD-406A-AE49-ED1CFF63D352@brimsecurity.com>
References: <4D066D96-61FD-406A-AE49-ED1CFF63D352@brimsecurity.com>
Message-ID: <76C52729-DA8F-49F4-BEAB-78968CF7D8AC@brimsecurity.com>

Zeek community,

I'm reaching out to announce another open source project... specifically the Brim <https://www.brimsecurity.com/download/> desktop application.

In its first version, the Brim workflow is tuned for starting from a packet capture (even a big one), which the app turns into Zeek logs for you. Then you've got an intuitive UI experience for querying those Zeek logs using the same ZQL <https://github.com/brimsec/zq/tree/master/zql/docs> language you may already know from zq <https://github.com/brimsec/zq> (see prior announcement below). And should your Zeek explorations lead you to a flow for which you want to see the packets, a single click in the app quickly extracts the flow from the big pcap and opens it immediately in Wireshark.

For more details, here's some links for Brim:
Download page <https://www.brimsecurity.com/download/> for the application
GitHub repo <https://github.com/brimsec/brim> for the project
Brim's YouTube channel <https://www.youtube.com/channel/UC0ju7Esmh13oLS8FTS-B3Eg> with a complete video on how to use Brim
Join our public Slack workspace <https://join.slack.com/t/brimsec/shared_invite/zt-cy34xoxg-hZiTKUT~1KdGjlaBIuUUdg> for announcements, Q&A, feedback, and to trade ideas
...or contact us via email <mailto:info at brimsecurity.com> 

There's more coming soon, so keep your eye on the repo for updates.

Happy hunting!

--
The Brim team


> On Feb 11, 2020, at 3:42 PM, Phil Rzewski <phil at brimsecurity.com> wrote:
> 
> Zeek community,
> 
> We?re writing to let you know about zq <https://github.com/brimsec/zq>, an open source command-line processor for structured logs, built for Zeek. (In fact, we?ve been told zq is ?like zeek-cut on steroids?.)
> 
> Those of you who were on the ?Ask the Zeeksperts? call on January 16th saw Seth Hall and Justin Azoff give an early peek of zq (thanks guys!), so this is just an ?official? announcement. Come one, come all!
> 
> You can get involved by:
> 	? Checking out the zq GitHub repo <https://github.com/brimsec/zq> for install info, code, and docs
> 	? Joining our public Slack <https://join.slack.com/t/brimsec/shared_invite/enQtOTMwMDczODg2ODgyLTk1NTdjOTQxNmI0OGYwOThiYzNlNDc5OWI5NjczZDljNDdmZGZjNGI3NTNiOWRiNzJkMzg4OTEwZWM0Y2NiYWQ> workspace for announcements, Q&A, and to trade query ideas
> 	? Contacting us directly via email <mailto:info at brimsecurity.com> to schedule a Zoom videoconference
> 
> All you need is some Zeek logs (and there?s sample logs <https://github.com/brimsec/zq-sample-data> to help you get started). Here?s just a taste of what?s possible:
> 
> - A table of top hosts in a subnet that are experiencing the most SYNs-without-ACK:
>    zq -f table "10.164.94.0/24 conn_state=S0 | count() by id.orig_h | sort -r" *
> 
> - A regex search for certain HTTP methods, with full events output as NDJSON:
>     zq -f ndjson "method=/^(PUT|PATCH|UPDATE)$/" *
> 
> - Connections open a long time with low traffic, printed as a Zeek TSV log:
>    zq -f zeek "duration>1000 orig_bytes<10 resp_bytes<10" *
> 
> Of course, that?s just scratching the surface. Please try it out and let us know what you think on GitHub <https://github.com/brimsec/zq> or Slack <https://join.slack.com/t/brimsec/shared_invite/enQtOTMwMDczODg2ODgyLTk1NTdjOTQxNmI0OGYwOThiYzNlNDc5OWI5NjczZDljNDdmZGZjNGI3NTNiOWRiNzJkMzg4OTEwZWM0Y2NiYWQ>.
> 
> Happy hunting, Zeeking, & zq?ing!
> 
> --
> The Brim team
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200324/555bc118/attachment.html 

From ttomek.koziak at gmail.com  Wed Mar 25 02:53:22 2020
From: ttomek.koziak at gmail.com (Tomek Koziak)
Date: Wed, 25 Mar 2020 10:53:22 +0100
Subject: [Zeek] Is it possible to inspect TCP reserved bits with Zeek?
In-Reply-To: <CAMzgZ0JH84vz9enBay=JFo19Ad62Vy6jRvpfQYQTT_ArYEVfHA@mail.gmail.com>
References: <CAF=fF7pOWFcUMM+iy7UwVSHG8bwwHK6goOT+kh9YWJe0XC_ArA@mail.gmail.com>
	<CAMzgZ0JH84vz9enBay=JFo19Ad62Vy6jRvpfQYQTT_ArYEVfHA@mail.gmail.com>
Message-ID: <CAF=fF7r-iePQJZaZm4MKHqhqyRrrkur1TZSdQn8nf9TjNcgrnQ@mail.gmail.com>

Hi Jon.

Thank you, it's working properly.
In the first place, I have modified the TCP_Flags.h to catch those bits,
but your solution seems to be better.

Tomasz

wt., 24 mar 2020 o 18:57 Jon Siwek <jsiwek at corelight.com> napisa?(a):

> On Mon, Mar 23, 2020 at 3:11 PM Tomek Koziak <ttomek.koziak at gmail.com>
> wrote:
>
> > Is it possible to inspect TCP reserved bits with Zeek events? If not is
> there any other possible way to detect wheter those bits where changed?
>
> I didn't see any events that currently carry the reserved bits, but it
> would be simple to extend existing ones like `new_packet` and
> `raw_packet`.  You can find an example patch for that in the
> `topic/jsiwek/tcp-hdr-reserved-bits` branch here:
>
>
> https://github.com/zeek/zeek/compare/topic/jsiwek/tcp-hdr-reserved-bits
>
> Let me know if that works for your purposes and I'll turn it into a
> pull request.
>
> - Jon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200325/10e2a7a8/attachment.html 

From vincyforce at gmail.com  Wed Mar 25 09:26:31 2020
From: vincyforce at gmail.com (Vincenzo)
Date: Wed, 25 Mar 2020 17:26:31 +0100
Subject: [Zeek] Subject: Drop packet by signature event
Message-ID: <CAPPrz2Ys8G4T5ghQmUc-2YjY7LA2EMiXWXixH67hoXxCfOXjoA@mail.gmail.com>

I have a configuration of FreeBSD with Zeek, my goal is to analyze network
traffic on one network interface and block (IPS) the packet to the other
interface, if this falls within my list of signatures that I have defined
in my signatures.sig.

I have searched far and wide for a solution, but I have not come up with
feasible solutions for this purpose (since Zeek was not born as IPS, as
snort and suricata), do you have any advice?

Zeek 3.0.3
FreeBSD 11
bro-netmap installed

Thanks very much
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200325/c68fcf0f/attachment-0001.html 

From asharma at lbl.gov  Wed Mar 25 09:41:00 2020
From: asharma at lbl.gov (Aashish Sharma)
Date: Wed, 25 Mar 2020 09:41:00 -0700
Subject: [Zeek] Subject: Drop packet by signature event
In-Reply-To: <CAPPrz2Ys8G4T5ghQmUc-2YjY7LA2EMiXWXixH67hoXxCfOXjoA@mail.gmail.com>
References: <CAPPrz2Ys8G4T5ghQmUc-2YjY7LA2EMiXWXixH67hoXxCfOXjoA@mail.gmail.com>
Message-ID: <20200325164059.GD81403@MacPro.local>

> feasible solutions for this purpose (since Zeek was not born as IPS, as
> snort and suricata), do you have any advice?

Umm, zeek's been doing IPS before snort or surcata were born  ;)

I am not sure of specifics on your end (ie how you want to implement it) but You
should look at netcontrol-framework and ACLD 
(https://ee.lbl.gov/downloads/acld/) in case you want to expand and work with
cisco/juniper routers. 

Aashish 

On Wed, Mar 25, 2020 at 05:26:31PM +0100, Vincenzo wrote:
> I have a configuration of FreeBSD with Zeek, my goal is to analyze network
> traffic on one network interface and block (IPS) the packet to the other
> interface, if this falls within my list of signatures that I have defined
> in my signatures.sig.
> 
> I have searched far and wide for a solution, but I have not come up with
> feasible solutions for this purpose (since Zeek was not born as IPS, as
> snort and suricata), do you have any advice?
> 
> Zeek 3.0.3
> FreeBSD 11
> bro-netmap installed
> 
> Thanks very much

> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From justin at corelight.com  Fri Mar 27 08:32:02 2020
From: justin at corelight.com (Justin Azoff)
Date: Fri, 27 Mar 2020 11:32:02 -0400
Subject: [Zeek] Anyone using Bro doctor plugin?
In-Reply-To: <4D39916F-959B-477A-B4A1-46BC506AE755@outlook.com>
References: <4D39916F-959B-477A-B4A1-46BC506AE755@outlook.com>
Message-ID: <CAPfnCuj3_L924wixSj2jaHN17a-Vk_Hre=VY3sGbUCf9fhiPPg@mail.gmail.com>

Sorry about that,  I just pushed 2.0.3 that fixes that issue.

On Tue, Mar 24, 2020 at 5:47 AM Carlos Lopez <clopmz at outlook.com> wrote:
>
> Hi all,
>
>
>
> I have enable bro doctor plugin in my Zeek 3.0.3 cluster and I see the following error:
>
>
>
> ###################################################################
>
> # Checking if connections are unevenly distributed across workers #
>
> ###################################################################
>
> error: Traceback (most recent call last):
>
>   File "/usr/lib64/python3.6/cmd.py", line 214, in onecmd
>
>     func = getattr(self, 'do_' + cmd)
>
> AttributeError: 'ZeekCtlCmdLoop' object has no attribute 'do_doctor'
>
>
>
> During handling of the above exception, another exception occurred:
>
>
>
> Traceback (most recent call last):
>
>   File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 596, in cmd_custom
>
>     results.ok = f() and results.ok
>
>   File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 457, in check_connection_distribution
>
>     variance = reduce(lambda var, cnt: var + (cnt - mean)**2, nodes.values(), 0) / len(nodes)
>
> NameError: name 'reduce' is not defined
>
>
>
> All other doctor options works ok, but not this one ? Is it a bug? Do I need to install some other python module? Zeek is running as unprivileged user ?
>
>
>
> --
>
> Regards,
>
> C. L. Martinez
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
Justin


From clopmz at outlook.com  Fri Mar 27 08:58:43 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Fri, 27 Mar 2020 15:58:43 +0000
Subject: [Zeek] Anyone using Bro doctor plugin?
In-Reply-To: <CAPfnCuj3_L924wixSj2jaHN17a-Vk_Hre=VY3sGbUCf9fhiPPg@mail.gmail.com>
References: <4D39916F-959B-477A-B4A1-46BC506AE755@outlook.com>
	<CAPfnCuj3_L924wixSj2jaHN17a-Vk_Hre=VY3sGbUCf9fhiPPg@mail.gmail.com>
Message-ID: <60859035-D851-4528-BD2F-B85569183F53@outlook.com>

Many thanks Justin ... I will try asap.

-- 
Regards,
C. L. Martinez

?On 27/03/2020, 16:32, "Justin Azoff" <justin at corelight.com> wrote:

    Sorry about that,  I just pushed 2.0.3 that fixes that issue.
    
    On Tue, Mar 24, 2020 at 5:47 AM Carlos Lopez <clopmz at outlook.com> wrote:
    >
    > Hi all,
    >
    >
    >
    > I have enable bro doctor plugin in my Zeek 3.0.3 cluster and I see the following error:
    >
    >
    >
    > ###################################################################
    >
    > # Checking if connections are unevenly distributed across workers #
    >
    > ###################################################################
    >
    > error: Traceback (most recent call last):
    >
    >   File "/usr/lib64/python3.6/cmd.py", line 214, in onecmd
    >
    >     func = getattr(self, 'do_' + cmd)
    >
    > AttributeError: 'ZeekCtlCmdLoop' object has no attribute 'do_doctor'
    >
    >
    >
    > During handling of the above exception, another exception occurred:
    >
    >
    >
    > Traceback (most recent call last):
    >
    >   File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 596, in cmd_custom
    >
    >     results.ok = f() and results.ok
    >
    >   File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 457, in check_connection_distribution
    >
    >     variance = reduce(lambda var, cnt: var + (cnt - mean)**2, nodes.values(), 0) / len(nodes)
    >
    > NameError: name 'reduce' is not defined
    >
    >
    >
    > All other doctor options works ok, but not this one ? Is it a bug? Do I need to install some other python module? Zeek is running as unprivileged user ?
    >
    >
    >
    > --
    >
    > Regards,
    >
    > C. L. Martinez
    >
    > _______________________________________________
    > Zeek mailing list
    > zeek at zeek.org
    > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
    
    
    
    -- 
    Justin
    



From mtrz at google.com  Fri Mar 27 10:11:13 2020
From: mtrz at google.com (Matteo Rizzo)
Date: Fri, 27 Mar 2020 18:11:13 +0100
Subject: [Zeek] Zeek and CLA
Message-ID: <CAHKB1wLXXRXgZwotdJ7vDLJ639xtWmDRCbjM_-NjUN6yvpyzXQ@mail.gmail.com>

Hello,

I wanted to ask whether contributing code to Zeek requires signing a
Contributor License Agreement (CLA). I have not seen any mentions of
this in Zeek's documentation but I'm asking just to make sure.

Thanks in advance,
Matteo Rizzo

From clopmz at outlook.com  Fri Mar 27 11:02:08 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Fri, 27 Mar 2020 18:02:08 +0000
Subject: [Zeek] Anyone using Bro doctor plugin?
In-Reply-To: <CAPfnCuj3_L924wixSj2jaHN17a-Vk_Hre=VY3sGbUCf9fhiPPg@mail.gmail.com>
References: <4D39916F-959B-477A-B4A1-46BC506AE755@outlook.com>
	<CAPfnCuj3_L924wixSj2jaHN17a-Vk_Hre=VY3sGbUCf9fhiPPg@mail.gmail.com>
Message-ID: <5C92E3D3-D6C0-408C-BCCB-15075888CA95@outlook.com>

Hi Justin,

 Same problem:

################################################
# Checking for recent capture_loss.log entries #
################################################
error: Traceback (most recent call last):
  File "/usr/lib64/python3.6/cmd.py", line 214, in onecmd
    func = getattr(self, 'do_' + cmd)
AttributeError: 'ZeekCtlCmdLoop' object has no attribute 'do_doctor'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 596, in cmd_custom
    results.ok = f() and results.ok
  File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 274, in check_capture_loss
    for rec in read_bro_logs_with_line_limit(reversed(files), 10000):
  File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 141, in read_bro_logs_with_line_limit
    for rec in read_bro_log(f):
  File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 131, in read_bro_log
    raise Exception("Unknown bro log type for file {}, first line: {!r}".format(filename, f.readline().strip()))
Exception: Unknown bro log type for file /nsm/zeek/logs/2020-03-26/capture_loss.16:00:00-16:06:17.log.gz, first line: b'"ts":"2020-03-26T16:06:11.983538Z","ts_delta":529.7351248264313,"peer":"worker-2","gaps":0,"acks":7,"percent_lost":0.0}'

My installed packages are:

zeek/corelight/bro-community-id (installed: 1.2) - "Community ID" flow hash support in conn.log
zeek/j-gras/add-node-names (installed: 2.0.0) - Adds cluster node name to logs.
zeek/j-gras/zeek-af_packet-plugin (installed: 2.0.0) - This plugin provides native AF_Packet support for Zeek.
zeek/ncsa/bro-doctor (installed: 2.0.3) - A broctl plugin that helps you troubleshoot common problems For cluster-related checks, the package "add-node-names" is recommended.
zeek/salesforce/hassh (installed: master) - HASSH is used to identify specific Client and Server SSH implementations.
zeek/salesforce/ja3 (installed: master) - JA3 creates 32 character SSL client fingerprints and logs them as a field in ssl.log.

-- 
Regards,
C. L. Martinez

?On 27/03/2020, 16:32, "Justin Azoff" <justin at corelight.com> wrote:

    Sorry about that,  I just pushed 2.0.3 that fixes that issue.
    
    On Tue, Mar 24, 2020 at 5:47 AM Carlos Lopez <clopmz at outlook.com> wrote:
    >
    > Hi all,
    >
    >
    >
    > I have enable bro doctor plugin in my Zeek 3.0.3 cluster and I see the following error:
    >
    >
    >
    > ###################################################################
    >
    > # Checking if connections are unevenly distributed across workers #
    >
    > ###################################################################
    >
    > error: Traceback (most recent call last):
    >
    >   File "/usr/lib64/python3.6/cmd.py", line 214, in onecmd
    >
    >     func = getattr(self, 'do_' + cmd)
    >
    > AttributeError: 'ZeekCtlCmdLoop' object has no attribute 'do_doctor'
    >
    >
    >
    > During handling of the above exception, another exception occurred:
    >
    >
    >
    > Traceback (most recent call last):
    >
    >   File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 596, in cmd_custom
    >
    >     results.ok = f() and results.ok
    >
    >   File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 457, in check_connection_distribution
    >
    >     variance = reduce(lambda var, cnt: var + (cnt - mean)**2, nodes.values(), 0) / len(nodes)
    >
    > NameError: name 'reduce' is not defined
    >
    >
    >
    > All other doctor options works ok, but not this one ? Is it a bug? Do I need to install some other python module? Zeek is running as unprivileged user ?
    >
    >
    >
    > --
    >
    > Regards,
    >
    > C. L. Martinez
    >
    > _______________________________________________
    > Zeek mailing list
    > zeek at zeek.org
    > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
    
    
    
    -- 
    Justin
    



From clopmz at outlook.com  Fri Mar 27 11:08:04 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Fri, 27 Mar 2020 18:08:04 +0000
Subject: [Zeek] Anyone using Bro doctor plugin?
In-Reply-To: <5C92E3D3-D6C0-408C-BCCB-15075888CA95@outlook.com>
References: <4D39916F-959B-477A-B4A1-46BC506AE755@outlook.com>
	<CAPfnCuj3_L924wixSj2jaHN17a-Vk_Hre=VY3sGbUCf9fhiPPg@mail.gmail.com>
	<5C92E3D3-D6C0-408C-BCCB-15075888CA95@outlook.com>
Message-ID: <981F9D00-A719-454B-BA4E-BE3E1DDF1D9B@outlook.com>

And errors appears with reporter also:

############################################
# Checking for recent reporter.log entries #
############################################
error: Found 2 reporter log files in the past 7 days
Recent reporter.log messages:
error: Traceback (most recent call last):
  File "/usr/lib64/python3.6/cmd.py", line 214, in onecmd
    func = getattr(self, 'do_' + cmd)
AttributeError: 'ZeekCtlCmdLoop' object has no attribute 'do_doctor'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 596, in cmd_custom
    results.ok = f() and results.ok
  File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 242, in check_reporter
    for rec in read_bro_logs_with_line_limit(reversed(files), 1000):
  File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 141, in read_bro_logs_with_line_limit
    for rec in read_bro_log(f):
  File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 131, in read_bro_log
    raise Exception("Unknown bro log type for file {}, first line: {!r}".format(filename, f.readline().strip()))
Exception: Unknown bro log type for file /nsm/zeek/logs/2020-03-26/reporter.16:06:11-16:06:17.log.gz, first line: b'"ts":"2020-03-26T16:06:11.983538Z","level":"Reporter::INFO","message":"received termination signal","location":""}'

-- 
Regards,
C. L. Martinez

?On 27/03/2020, 19:05, "zeek-bounces at zeek.org on behalf of Carlos Lopez" <zeek-bounces at zeek.org on behalf of clopmz at outlook.com> wrote:

    Hi Justin,
    
     Same problem:
    
    ################################################
    # Checking for recent capture_loss.log entries #
    ################################################
    error: Traceback (most recent call last):
      File "/usr/lib64/python3.6/cmd.py", line 214, in onecmd
        func = getattr(self, 'do_' + cmd)
    AttributeError: 'ZeekCtlCmdLoop' object has no attribute 'do_doctor'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 596, in cmd_custom
        results.ok = f() and results.ok
      File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 274, in check_capture_loss
        for rec in read_bro_logs_with_line_limit(reversed(files), 10000):
      File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 141, in read_bro_logs_with_line_limit
        for rec in read_bro_log(f):
      File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 131, in read_bro_log
        raise Exception("Unknown bro log type for file {}, first line: {!r}".format(filename, f.readline().strip()))
    Exception: Unknown bro log type for file /nsm/zeek/logs/2020-03-26/capture_loss.16:00:00-16:06:17.log.gz, first line: b'"ts":"2020-03-26T16:06:11.983538Z","ts_delta":529.7351248264313,"peer":"worker-2","gaps":0,"acks":7,"percent_lost":0.0}'
    
    My installed packages are:
    
    zeek/corelight/bro-community-id (installed: 1.2) - "Community ID" flow hash support in conn.log
    zeek/j-gras/add-node-names (installed: 2.0.0) - Adds cluster node name to logs.
    zeek/j-gras/zeek-af_packet-plugin (installed: 2.0.0) - This plugin provides native AF_Packet support for Zeek.
    zeek/ncsa/bro-doctor (installed: 2.0.3) - A broctl plugin that helps you troubleshoot common problems For cluster-related checks, the package "add-node-names" is recommended.
    zeek/salesforce/hassh (installed: master) - HASSH is used to identify specific Client and Server SSH implementations.
    zeek/salesforce/ja3 (installed: master) - JA3 creates 32 character SSL client fingerprints and logs them as a field in ssl.log.
    
    -- 
    Regards,
    C. L. Martinez
    
    On 27/03/2020, 16:32, "Justin Azoff" <justin at corelight.com> wrote:
    
        Sorry about that,  I just pushed 2.0.3 that fixes that issue.
        
        On Tue, Mar 24, 2020 at 5:47 AM Carlos Lopez <clopmz at outlook.com> wrote:
        >
        > Hi all,
        >
        >
        >
        > I have enable bro doctor plugin in my Zeek 3.0.3 cluster and I see the following error:
        >
        >
        >
        > ###################################################################
        >
        > # Checking if connections are unevenly distributed across workers #
        >
        > ###################################################################
        >
        > error: Traceback (most recent call last):
        >
        >   File "/usr/lib64/python3.6/cmd.py", line 214, in onecmd
        >
        >     func = getattr(self, 'do_' + cmd)
        >
        > AttributeError: 'ZeekCtlCmdLoop' object has no attribute 'do_doctor'
        >
        >
        >
        > During handling of the above exception, another exception occurred:
        >
        >
        >
        > Traceback (most recent call last):
        >
        >   File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 596, in cmd_custom
        >
        >     results.ok = f() and results.ok
        >
        >   File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line 457, in check_connection_distribution
        >
        >     variance = reduce(lambda var, cnt: var + (cnt - mean)**2, nodes.values(), 0) / len(nodes)
        >
        > NameError: name 'reduce' is not defined
        >
        >
        >
        > All other doctor options works ok, but not this one ? Is it a bug? Do I need to install some other python module? Zeek is running as unprivileged user ?
        >
        >
        >
        > --
        >
        > Regards,
        >
        > C. L. Martinez
        >
        > _______________________________________________
        > Zeek mailing list
        > zeek at zeek.org
        > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
        
        
        
        -- 
        Justin
        
    
    
    _______________________________________________
    Zeek mailing list
    zeek at zeek.org
    http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



From justin at corelight.com  Sat Mar 28 10:17:22 2020
From: justin at corelight.com (Justin Azoff)
Date: Sat, 28 Mar 2020 13:17:22 -0400
Subject: [Zeek] Anyone using Bro doctor plugin?
In-Reply-To: <981F9D00-A719-454B-BA4E-BE3E1DDF1D9B@outlook.com>
References: <4D39916F-959B-477A-B4A1-46BC506AE755@outlook.com>
	<CAPfnCuj3_L924wixSj2jaHN17a-Vk_Hre=VY3sGbUCf9fhiPPg@mail.gmail.com>
	<5C92E3D3-D6C0-408C-BCCB-15075888CA95@outlook.com>
	<981F9D00-A719-454B-BA4E-BE3E1DDF1D9B@outlook.com>
Message-ID: <CAPfnCuhS9x-xBZ-Yz0kUzAKbm=uzrMnq4rtobD4OZ7_j0V0OSg@mail.gmail.com>

Gah, looks like there are still some stupid python3 issues.  I could have
sworn I fixed all of those a while back.. Must have been in a branch I
never finished.

Changing the "{" to b"{" in read_bro_log should fix that immediate issue.

I'll see about getting this fixed and tested better this weekend.

On Friday, March 27, 2020, Carlos Lopez <clopmz at outlook.com> wrote:

> And errors appears with reporter also:
>
> ############################################
> # Checking for recent reporter.log entries #
> ############################################
> error: Found 2 reporter log files in the past 7 days
> Recent reporter.log messages:
> error: Traceback (most recent call last):
>   File "/usr/lib64/python3.6/cmd.py", line 214, in onecmd
>     func = getattr(self, 'do_' + cmd)
> AttributeError: 'ZeekCtlCmdLoop' object has no attribute 'do_doctor'
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
>   File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line
> 596, in cmd_custom
>     results.ok = f() and results.ok
>   File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line
> 242, in check_reporter
>     for rec in read_bro_logs_with_line_limit(reversed(files), 1000):
>   File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line
> 141, in read_bro_logs_with_line_limit
>     for rec in read_bro_log(f):
>   File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py", line
> 131, in read_bro_log
>     raise Exception("Unknown bro log type for file {}, first line:
> {!r}".format(filename, f.readline().strip()))
> Exception: Unknown bro log type for file /nsm/zeek/logs/2020-03-26/
> reporter.16:06:11-16:06:17.log.gz, first line:
> b'"ts":"2020-03-26T16:06:11.983538Z","level":"Reporter::INFO","message":"received
> termination signal","location":""}'
>
> --
> Regards,
> C. L. Martinez
>
> ?On 27/03/2020, 19:05, "zeek-bounces at zeek.org on behalf of Carlos Lopez" <
> zeek-bounces at zeek.org on behalf of clopmz at outlook.com> wrote:
>
>     Hi Justin,
>
>      Same problem:
>
>     ################################################
>     # Checking for recent capture_loss.log entries #
>     ################################################
>     error: Traceback (most recent call last):
>       File "/usr/lib64/python3.6/cmd.py", line 214, in onecmd
>         func = getattr(self, 'do_' + cmd)
>     AttributeError: 'ZeekCtlCmdLoop' object has no attribute 'do_doctor'
>
>     During handling of the above exception, another exception occurred:
>
>     Traceback (most recent call last):
>       File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py",
> line 596, in cmd_custom
>         results.ok = f() and results.ok
>       File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py",
> line 274, in check_capture_loss
>         for rec in read_bro_logs_with_line_limit(reversed(files), 10000):
>       File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py",
> line 141, in read_bro_logs_with_line_limit
>         for rec in read_bro_log(f):
>       File "/opt/zeek/lib/zeek/plugins/packages/bro-doctor/doctor.py",
> line 131, in read_bro_log
>         raise Exception("Unknown bro log type for file {}, first line:
> {!r}".format(filename, f.readline().strip()))
>     Exception: Unknown bro log type for file /nsm/zeek/logs/2020-03-26/
> capture_loss.16:00:00-16:06:17.log.gz, first line:
> b'"ts":"2020-03-26T16:06:11.983538Z","ts_delta":529.
> 7351248264313,"peer":"worker-2","gaps":0,"acks":7,"percent_lost":0.0}'
>
>     My installed packages are:
>
>     zeek/corelight/bro-community-id (installed: 1.2) - "Community ID"
> flow hash support in conn.log
>     zeek/j-gras/add-node-names (installed: 2.0.0) - Adds cluster node name
> to logs.
>     zeek/j-gras/zeek-af_packet-plugin (installed: 2.0.0) - This plugin
> provides native AF_Packet support for Zeek.
>     zeek/ncsa/bro-doctor (installed: 2.0.3) - A broctl plugin that helps
> you troubleshoot common problems For cluster-related checks, the package
> "add-node-names" is recommended.
>     zeek/salesforce/hassh (installed: master) - HASSH is used to identify
> specific Client and Server SSH implementations.
>     zeek/salesforce/ja3 (installed: master) - JA3 creates 32 character SSL
> client fingerprints and logs them as a field in ssl.log.
>
>     --
>     Regards,
>     C. L. Martinez
>
>     On 27/03/2020, 16:32, "Justin Azoff" <justin at corelight.com> wrote:
>
>         Sorry about that,  I just pushed 2.0.3 that fixes that issue.
>
>         On Tue, Mar 24, 2020 at 5:47 AM Carlos Lopez <clopmz at outlook.com>
> wrote:
>         >
>         > Hi all,
>         >
>         >
>         >
>         > I have enable bro doctor plugin in my Zeek 3.0.3 cluster and I
> see the following error:
>         >
>         >
>         >
>         > ############################################################
> #######
>         >
>         > # Checking if connections are unevenly distributed across
> workers #
>         >
>         > ############################################################
> #######
>         >
>         > error: Traceback (most recent call last):
>         >
>         >   File "/usr/lib64/python3.6/cmd.py", line 214, in onecmd
>         >
>         >     func = getattr(self, 'do_' + cmd)
>         >
>         > AttributeError: 'ZeekCtlCmdLoop' object has no attribute
> 'do_doctor'
>         >
>         >
>         >
>         > During handling of the above exception, another exception
> occurred:
>         >
>         >
>         >
>         > Traceback (most recent call last):
>         >
>         >   File "/opt/zeek/lib/zeek/plugins/
> packages/bro-doctor/doctor.py", line 596, in cmd_custom
>         >
>         >     results.ok = f() and results.ok
>         >
>         >   File "/opt/zeek/lib/zeek/plugins/
> packages/bro-doctor/doctor.py", line 457, in check_connection_distribution
>         >
>         >     variance = reduce(lambda var, cnt: var + (cnt - mean)**2,
> nodes.values(), 0) / len(nodes)
>         >
>         > NameError: name 'reduce' is not defined
>         >
>         >
>         >
>         > All other doctor options works ok, but not this one ? Is it a
> bug? Do I need to install some other python module? Zeek is running as
> unprivileged user ?
>         >
>         >
>         >
>         > --
>         >
>         > Regards,
>         >
>         > C. L. Martinez
>         >
>         > _______________________________________________
>         > Zeek mailing list
>         > zeek at zeek.org
>         > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
>
>         --
>         Justin
>
>
>
>     _______________________________________________
>     Zeek mailing list
>     zeek at zeek.org
>     http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>

-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200328/8bae3835/attachment.html 

From justin at corelight.com  Sun Mar 29 08:25:03 2020
From: justin at corelight.com (Justin Azoff)
Date: Sun, 29 Mar 2020 11:25:03 -0400
Subject: [Zeek] Anyone using Bro doctor plugin?
In-Reply-To: <CAPfnCuhS9x-xBZ-Yz0kUzAKbm=uzrMnq4rtobD4OZ7_j0V0OSg@mail.gmail.com>
References: <4D39916F-959B-477A-B4A1-46BC506AE755@outlook.com>
	<CAPfnCuj3_L924wixSj2jaHN17a-Vk_Hre=VY3sGbUCf9fhiPPg@mail.gmail.com>
	<5C92E3D3-D6C0-408C-BCCB-15075888CA95@outlook.com>
	<981F9D00-A719-454B-BA4E-BE3E1DDF1D9B@outlook.com>
	<CAPfnCuhS9x-xBZ-Yz0kUzAKbm=uzrMnq4rtobD4OZ7_j0V0OSg@mail.gmail.com>
Message-ID: <CAPfnCuh_j=uVvvZhZyiEfiJX6000PQfZ2qQqeTHYVamwh-pq8w@mail.gmail.com>

On Sat, Mar 28, 2020 at 1:17 PM Justin Azoff <justin at corelight.com> wrote:
>
> Gah, looks like there are still some stupid python3 issues.  I could have sworn I fixed all of those a while back.. Must have been in a branch I never finished.

Looks like I did finish it, but I might have missed the final git
push, oops :-)  I merged in the recent changes and pushed a new tag
2.0.4 that has the python3 fixes from December, so everything should
be working like it's supposed to now.

-- 
Justin

From clopmz at outlook.com  Sun Mar 29 09:35:24 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Sun, 29 Mar 2020 16:35:24 +0000
Subject: [Zeek] Anyone using Bro doctor plugin?
In-Reply-To: <CAPfnCuh_j=uVvvZhZyiEfiJX6000PQfZ2qQqeTHYVamwh-pq8w@mail.gmail.com>
References: <4D39916F-959B-477A-B4A1-46BC506AE755@outlook.com>
	<CAPfnCuj3_L924wixSj2jaHN17a-Vk_Hre=VY3sGbUCf9fhiPPg@mail.gmail.com>
	<5C92E3D3-D6C0-408C-BCCB-15075888CA95@outlook.com>
	<981F9D00-A719-454B-BA4E-BE3E1DDF1D9B@outlook.com>
	<CAPfnCuhS9x-xBZ-Yz0kUzAKbm=uzrMnq4rtobD4OZ7_j0V0OSg@mail.gmail.com>
	<CAPfnCuh_j=uVvvZhZyiEfiJX6000PQfZ2qQqeTHYVamwh-pq8w@mail.gmail.com>
Message-ID: <07B2F90F-6D41-4B66-95D9-4A95A9B01A9E@outlook.com>

Many thanks Justin ... It is working ok now ...

-- 
Regards,
C. L. Martinez

?On 29/03/2020, 17:25, "Justin Azoff" <justin at corelight.com> wrote:

    On Sat, Mar 28, 2020 at 1:17 PM Justin Azoff <justin at corelight.com> wrote:
    >
    > Gah, looks like there are still some stupid python3 issues.  I could have sworn I fixed all of those a while back.. Must have been in a branch I never finished.
    
    Looks like I did finish it, but I might have missed the final git
    push, oops :-)  I merged in the recent changes and pushed a new tag
    2.0.4 that has the python3 fixes from December, so everything should
    be working like it's supposed to now.
    
    -- 
    Justin
    



From klehigh at iu.edu  Tue Mar 31 08:50:53 2020
From: klehigh at iu.edu (Keith Lehigh)
Date: Tue, 31 Mar 2020 11:50:53 -0400
Subject: [Zeek] ZeekWeek 2020 Austin - Cancelled - Open Letter to the
	Community
Message-ID: <49DFF651-137E-4F72-8BD4-AE402760CD7B@iu.edu>

Dear Zeek Community,

It is our hope that all of you are staying safe and healthy during this
uncertain time.  We?re all navigating unfamiliar territory together, as the
COVID 19 crisis affects every aspect of our lives both personally and
professionally.

We all wish we knew what the future might hold for 1 month, 3 months, 6
months or more from now. It would make planning for events easier and more
efficient; however, that is not the case.  While we sincerely hope the
world and our daily activities will be back to normal in the fall, we can?t
be assured of that.

Given the uncertainty, we?ve made the difficult decision to cancel ZeekWeek
2020 in Austin.  Rest assured that we are looking at other options to bring
the community together as things improve and become more predictable. Those
options include a virtual event during the same time frame, and if it?s
safe to bring people together, then we will look at holding a smaller event
in a different location.  However, we won?t know until we get closer to
October.

Don?t file away those ZeekWeek talks you were going to pitch!! We will be
announcing some new initiatives for you to share your knowledge and bring
the community together.

In the meantime, there are many ways for you to connect with the Zeek
Community - Mailing Lists <https://zeek.org/mailing-lists/>, Slack Workspace
<https://join.slack.com/t/zeekorg/shared_invite/enQtOTc3MzMxNDI1NDYxLTA1NzhhMTgxNWI1OTk2NjlkMTdjNzY1Nzk5NDk2ZDY1MDBkYWIxOWNjNDE2NDc2MGI5OWM3ZDllYzBmZmNhNDM>,
Webinars
<http://mailman.icsi.berkeley.edu/pipermail/zeek/2020-March/015193.html>
and more.

We look forward to seeing you in-person once it is safe to do so.  Until
then, remember, without you there is no community! Together we are better,
whether in-person or online.

Stay Safe and Zeek on!!

The Zeek Leadership Team


Blog Post of this message -
https://zeek.org/2020/03/31/zeekweek-2020-austin-cancelled-open-letter-to-the-community/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3700 bytes
Desc: S/MIME digital signature
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200331/0b5e1352/attachment.bin 

From raubvogel at gmail.com  Tue Mar 31 10:13:25 2020
From: raubvogel at gmail.com (Mauricio Tavares)
Date: Tue, 31 Mar 2020 13:13:25 -0400
Subject: [Zeek] How to turn verbose on (Can't load packages)?
Message-ID: <CAHEKYV57s9pcEiv6k_e47rqptjPk_F6PJ6OWo+Vjrf-+D89Qbg@mail.gmail.com>

So in my local.bro file I have

@load packages  <---- line 119

When I run zeekctl check I get

manager scripts failed.
fatal error in /usr/local/bro/site/local.bro, line 119: can't find packages

which makes me think there is be a package it cannot load either
because of corruption or said package is just not there. Is there a
way to crank up the verbose level so I can see what it was doing when
it went boink?

From justin at corelight.com  Tue Mar 31 10:36:34 2020
From: justin at corelight.com (Justin Azoff)
Date: Tue, 31 Mar 2020 13:36:34 -0400
Subject: [Zeek] How to turn verbose on (Can't load packages)?
In-Reply-To: <CAHEKYV57s9pcEiv6k_e47rqptjPk_F6PJ6OWo+Vjrf-+D89Qbg@mail.gmail.com>
References: <CAHEKYV57s9pcEiv6k_e47rqptjPk_F6PJ6OWo+Vjrf-+D89Qbg@mail.gmail.com>
Message-ID: <CAPfnCugMcruVq1D3T=Esy4Gq+NwY8JYWvv=vgo9AMJG3g5CcMA@mail.gmail.com>

'packages' will exist if you've used bro-pkg/zkg to install any
packages.  If you haven't used the package manager to install any
packages then you don't have a packages directory and trying to load
it will fail.

On Tue, Mar 31, 2020 at 1:15 PM Mauricio Tavares <raubvogel at gmail.com> wrote:
>
> So in my local.bro file I have
>
> @load packages  <---- line 119
>
> When I run zeekctl check I get
>
> manager scripts failed.
> fatal error in /usr/local/bro/site/local.bro, line 119: can't find packages
>
> which makes me think there is be a package it cannot load either
> because of corruption or said package is just not there. Is there a
> way to crank up the verbose level so I can see what it was doing when
> it went boink?
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
Justin

From raubvogel at gmail.com  Tue Mar 31 11:30:30 2020
From: raubvogel at gmail.com (Mauricio Tavares)
Date: Tue, 31 Mar 2020 14:30:30 -0400
Subject: [Zeek] How to turn verbose on (Can't load packages)?
In-Reply-To: <CAPfnCugMcruVq1D3T=Esy4Gq+NwY8JYWvv=vgo9AMJG3g5CcMA@mail.gmail.com>
References: <CAHEKYV57s9pcEiv6k_e47rqptjPk_F6PJ6OWo+Vjrf-+D89Qbg@mail.gmail.com>
	<CAPfnCugMcruVq1D3T=Esy4Gq+NwY8JYWvv=vgo9AMJG3g5CcMA@mail.gmail.com>
Message-ID: <CAHEKYV46v9-r+rBzDeU3aYo=zgEkWS6Wezaqtntt+24LBswDNw@mail.gmail.com>

On Tue, Mar 31, 2020 at 1:37 PM Justin Azoff <justin at corelight.com> wrote:
>
> 'packages' will exist if you've used bro-pkg/zkg to install any
> packages.  If you haven't used the package manager to install any
> packages then you don't have a packages directory and trying to load
> it will fail.
>
      In that case, I think I am in trouble because I did use zkg to
install a few packages:

zkg install --force --nodeps domain-tld top-dns \
        bro-shellshock \
        venom \
        file-extraction \
        add-node-names \
        zeek-cryptomining \
        bro-doctor \
        bro-interface-setup \
        credit-card-exposure \
        ssn-exposure

> On Tue, Mar 31, 2020 at 1:15 PM Mauricio Tavares <raubvogel at gmail.com> wrote:
> >
> > So in my local.bro file I have
> >
> > @load packages  <---- line 119
> >
> > When I run zeekctl check I get
> >
> > manager scripts failed.
> > fatal error in /usr/local/bro/site/local.bro, line 119: can't find packages
> >
> > which makes me think there is be a package it cannot load either
> > because of corruption or said package is just not there. Is there a
> > way to crank up the verbose level so I can see what it was doing when
> > it went boink?
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
>
> --
> Justin

From justin at corelight.com  Tue Mar 31 11:45:33 2020
From: justin at corelight.com (Justin Azoff)
Date: Tue, 31 Mar 2020 14:45:33 -0400
Subject: [Zeek] How to turn verbose on (Can't load packages)?
In-Reply-To: <CAHEKYV46v9-r+rBzDeU3aYo=zgEkWS6Wezaqtntt+24LBswDNw@mail.gmail.com>
References: <CAHEKYV57s9pcEiv6k_e47rqptjPk_F6PJ6OWo+Vjrf-+D89Qbg@mail.gmail.com>
	<CAPfnCugMcruVq1D3T=Esy4Gq+NwY8JYWvv=vgo9AMJG3g5CcMA@mail.gmail.com>
	<CAHEKYV46v9-r+rBzDeU3aYo=zgEkWS6Wezaqtntt+24LBswDNw@mail.gmail.com>
Message-ID: <CAPfnCuhycBo5TSrV7fs+PD_NDwWh+MPJ48HyKmyaiumRJ=Jbmg@mail.gmail.com>

Ah, ok, in that case the zkg config might be wrong.  A step people
often miss is running

    zkg autoconfig

to have zkg discover where zeek is installed and where it should
install things too.  check your ~/.zkg/config and verify if
script_dir points to your zeek installation.  If not, running zkg
autoconfig and reinstalling the packages should fix things.

On Tue, Mar 31, 2020 at 2:30 PM Mauricio Tavares <raubvogel at gmail.com> wrote:
>
> On Tue, Mar 31, 2020 at 1:37 PM Justin Azoff <justin at corelight.com> wrote:
> >
> > 'packages' will exist if you've used bro-pkg/zkg to install any
> > packages.  If you haven't used the package manager to install any
> > packages then you don't have a packages directory and trying to load
> > it will fail.
> >
>       In that case, I think I am in trouble because I did use zkg to
> install a few packages:
>
> zkg install --force --nodeps domain-tld top-dns \
>         bro-shellshock \
>         venom \
>         file-extraction \
>         add-node-names \
>         zeek-cryptomining \
>         bro-doctor \
>         bro-interface-setup \
>         credit-card-exposure \
>         ssn-exposure
>
> > On Tue, Mar 31, 2020 at 1:15 PM Mauricio Tavares <raubvogel at gmail.com> wrote:
> > >
> > > So in my local.bro file I have
> > >
> > > @load packages  <---- line 119
> > >
> > > When I run zeekctl check I get
> > >
> > > manager scripts failed.
> > > fatal error in /usr/local/bro/site/local.bro, line 119: can't find packages
> > >
> > > which makes me think there is be a package it cannot load either
> > > because of corruption or said package is just not there. Is there a
> > > way to crank up the verbose level so I can see what it was doing when
> > > it went boink?
> > > _______________________________________________
> > > Zeek mailing list
> > > zeek at zeek.org
> > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> >
> >
> >
> > --
> > Justin



-- 
Justin

From jsiwek at corelight.com  Tue Mar 31 11:51:49 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Tue, 31 Mar 2020 11:51:49 -0700
Subject: [Zeek] How to turn verbose on (Can't load packages)?
In-Reply-To: <CAHEKYV46v9-r+rBzDeU3aYo=zgEkWS6Wezaqtntt+24LBswDNw@mail.gmail.com>
References: <CAHEKYV57s9pcEiv6k_e47rqptjPk_F6PJ6OWo+Vjrf-+D89Qbg@mail.gmail.com>
	<CAPfnCugMcruVq1D3T=Esy4Gq+NwY8JYWvv=vgo9AMJG3g5CcMA@mail.gmail.com>
	<CAHEKYV46v9-r+rBzDeU3aYo=zgEkWS6Wezaqtntt+24LBswDNw@mail.gmail.com>
Message-ID: <CAMzgZ0LKbGFDZ_Le4qU+cX65j-ovB-63YhXdkbd6s0J3Bw-+Sg@mail.gmail.com>

On Tue, Mar 31, 2020 at 11:39 AM Mauricio Tavares <raubvogel at gmail.com> wrote:

>       In that case, I think I am in trouble because I did use zkg

What's the output of `zkg config` ?

Did you previously run `zkg autoconfig` ?

If you don't do any configuration, the default location for zkg to
install packages is in $HOME/.zkg rather than inside your Zeek install
prefix and could explain why your `@load packages` doesn't find
anything.

- Jon

From akgraner at corelight.com  Tue Mar 31 12:09:35 2020
From: akgraner at corelight.com (Amber Graner)
Date: Tue, 31 Mar 2020 15:09:35 -0400
Subject: [Zeek] Announcing Zeek From Home Webinar Series
Message-ID: <CAJhOzuphnDbSDv9SEKMsZ=0iO6r9fffKbskoGCP0aF_KG-v1fQ@mail.gmail.com>

Hi all,

Since we won?t be holding any in-person Zeek events for the foreseeable
future, we?d like to invite you to be part of a new weekly ?Zeek From Home?
webinar series to kick off in April. The schedule will be announced once we
have a few submissions queued up.

These presentations will be recorded and shared via the Zeek mailing list,
blog and Twitter account.

Please take a look at the blog post (link below) and consider submitting a
presentation for the Zeek From Home series.

https://zeek.org/2020/03/31/zeek-from-home/

If you have any questions you can email me, reply to this thread or email
info at zeek.org.

With gratitude,
~Amber
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200331/497b0397/attachment.html 

From petr.medonos at etnetera.cz  Tue Mar 31 12:23:39 2020
From: petr.medonos at etnetera.cz (Petr Medonos)
Date: Tue, 31 Mar 2020 21:23:39 +0200
Subject: [Zeek] Long running connection using threshold
Message-ID: <38442acf-7430-2e23-0528-e85a098437b7@etnetera.cz>

Hi,
I tried to write simple script to detect long running connection using
zeek (3.0) threshold. I set duration in connection established event and
then using duration_threshold_crossed logged connection above the limit.
But Notice log is then flooded with every new established connection.
Simple PoC bellow. Did I missed something? Is there any better way to
detect long running connection? I tried Corelight bro-long-connections
but there is lot overhead in my environment. Thanks for pointing me the
right way!


--
Petr


PoC:

@load base/protocols/conn

module LongConnection;

export {
        redef enum Log::ID += { LOG };

        redef enum Notice::Type += {
                LongConnection::found
        };

        const duration: interval = 12hr &redef;
}

event connection_established(c: connection)
        {
        ConnThreshold::set_duration_threshold(c, duration);
        }

event ConnThreshold::duration_threshold_crossed(c: connection,
threshold: interval, is_orig: bool)
  {
        local message = fmt("%s:%s -> %s:%s remained alive for longer
than %s", c$id$orig_h, c$id$orig_p, c$id$resp_h, c$id$resp_p, threshold);

        NOTICE([$note=LongConnection::found,
                        $msg=message,
                        $sub=fmt("%.2f", threshold),
                        $conn=c]);

  }


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200331/600edf89/attachment.bin