From zeolla at gmail.com  Mon May  4 10:01:39 2020
From: zeolla at gmail.com (Zeolla@GMail.com)
Date: Mon, 4 May 2020 13:01:39 -0400
Subject: [Zeek] Zeek 3.1.2 and Kafka - No data flow
In-Reply-To: <CAM1mTCoNK8UbSUJH3OgqSjGi4=KSk15Zxcwd9aUDkzcxaEDoLw@mail.gmail.com>
References: <CAJ19UZCKWL+OuOS2XcWGbaYexyPmeGJ1L3XSW4nFf89Xz_k1WQ@mail.gmail.com>
	<CAM1mTCqZ4kpsD6KqN1P9+8zcE9PX_-_X_VwTTOzCWaOkhQT5_Q@mail.gmail.com>
	<CAM1mTCoNK8UbSUJH3OgqSjGi4=KSk15Zxcwd9aUDkzcxaEDoLw@mail.gmail.com>
Message-ID: <CAM1mTCoP_cCfw7QhxMUxi+iNB1Y3Oo+1A-SEkXqNDMzSfqEM0Q@mail.gmail.com>

I have the plugin working with 3.1.2 here
<https://github.com/apache/metron-bro-plugin-kafka/pull/44> - feedback is
welcome.

- Jon Zeolla
Zeolla at GMail.Com


On Wed, Apr 29, 2020 at 3:35 PM Zeolla at GMail.com <zeolla at gmail.com> wrote:

> Were you able to get this working?  I'm planning to work on the bro to
> zeek cutover for the plugin soon.
>
> - Jon Zeolla
> Zeolla at GMail.Com
>
>
> On Mon, Apr 27, 2020 at 6:39 AM Zeolla at GMail.com <zeolla at gmail.com> wrote:
>
>> I have not run it on 3.1.2 yet but I recommend making your changes to the
>> plugin and running the end to end testing script at
>> https://github.com/apache/metron-bro-plugin-kafka/blob/master/docker/run_end_to_end.sh
>>
>> It was meant to help isolate issues when making changes to the plugin.
>> Also, we welcome PRs against the project so please feel free to
>> contribute.  Thanks,
>>
>> Jon Zeolla
>>
>> On Sun, Apr 26, 2020, 9:12 PM Hovsep Levi <hovsep.sanjay.levi at gmail.com>
>> wrote:
>>
>>> Hello Zeeks
>>>
>>>
>>> Has anyone succeeded to enable Kafka plugin with Zeek 3.1.2 ?  I am
>>> trying to modernize the metron-kafka plugin and have partial success.  My
>>> problem seems to be with script-land referencing.
>>>
>>> The logger node is loading the plugin OK and connects to the Kafka
>>> broker.  The broker IP is redef information found from site/local.zeek.
>>>
>>> $ bin/zeekctl diag logger-1
>>> [logger-1]
>>>
>>> No core file found.
>>>
>>> Zeek 3.1.2-debug
>>>
>>> Zeek plugins:
>>> Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)
>>>
>>> ==== No reporter.log
>>>
>>> ==== stderr.log
>>> %7|1587948661.341|RECV|rdkafka#producer-3| [..kafka messages..] ..
>>>
>>>
>>>
>>>
>>> But the worker node has a problem referencing existing variable
>>> declaration.  The logs-to-kafka.bro script expects it.  There is also
>>> suspicion with the Zeek plugins info that is different from the logger node
>>> and maybe the problem.
>>>
>>> $ bin/zeekctl diag worker-1-1
>>> [worker-1-1]
>>>
>>> No core file found.
>>>
>>> Zeek 3.1.2-debug
>>>
>>> Zeek plugins: (none found)   <<< ??? Normal for worker node ???
>>>
>>> ==== No reporter.log
>>>
>>> ==== stderr.log
>>>
>>> error in
>>> /opt/zeek/spool/installed-scripts-do-not-touch/site/custom_plugins/APACHE_KAFKA/scripts/Apache/Kafka/./logs-to-kafka.bro,
>>> line 24: unknown identifier logs_to_send, at or near "logs_to_send"
>>>
>>>
>>>
>>> The configuration is not default and explained below:
>>>
>>>
>>> The Kafka logger was installed to site/custom_plugins/APACHE_KAFKA
>>>
>>>
>>> share/zeek/site/local.zeek uses:
>>>
>>> @load custom_plugins/APACHE_KAFKA/scripts/Apache/Kafka
>>>
>>>
>>>
>>> lib/zeek/plugins/custom_plugins is a symlink to
>>> share/zeek/site/custom_plugins
>>>
>>>
>>> Using the lib symlink seems to be the only way to load the plugin, then
>>> the @load statement brings redef customizations and scripts.  This works ok
>>> for the logger node but not the worker who cannot interface with the plugin
>>> ?
>>>
>>> Another idea is have non-logger nodes bypass loading logs-to-kafka.bro
>>> but this isn't fully understood.
>>>
>>>
>>> TIA
>>>
>>> /hovsep
>>>
>>>
>>>
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200504/f519feed/attachment.html 

From jradih20 at gmail.com  Tue May  5 14:25:14 2020
From: jradih20 at gmail.com (Hank Duo)
Date: Wed, 6 May 2020 00:25:14 +0300
Subject: [Zeek] Zeek won't extract exe and office files
In-Reply-To: <CAMh3h1x+W9_HKtb_w9T1OkPS-hV7mQyguN32sk-B16Ow-2kDAw@mail.gmail.com>
References: <CAMh3h1x+W9_HKtb_w9T1OkPS-hV7mQyguN32sk-B16Ow-2kDAw@mail.gmail.com>
Message-ID: <CAMh3h1wB=iio6u6-27ty+rUKeXw=8kgXJn2e+Q4eEQis6Yx1hQ@mail.gmail.com>

Hi,
Following my previous email, Zeek started extracting some .exe files but
not all. If for example I download twenty .exe files over http from a
certain website, Zeek extracts like 2 or 3 out of 20. Is there a reason why
Zeek is not recognizing and extracting all.exe files? Also, I added Binary
.bin files to be extracted, however it is not extracting them.
Note: I am downloading all files over http protocol only and not SSL.
Thank you for your help
Regards,
Hank

On Wed, 29 Apr 2020 at 23:42, Hank Duo <jradih20 at gmail.com> wrote:

> Hi all,
>
> I would like to extract .exe and office files for static and dynamic
> malware analysis purpose. I used the attached script however .exe or .docs
> files are not extracted except for html, txt or zip files.
>
> Note that I modified the main.zeek file which is located in
> /usr/local/zeek/share/zeek/zeekctl/main.bro by adding @load
> <https://github.com/load> /frameworks/files/extract-myfiles (which is the
> script file name) and commented the default one and the script was applied
> properly.
>
> Also, is there a way to extract files only from http or smb protocols
> while excluding https?
> Thank you guys
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200506/dec7c34e/attachment.html 

From justin at corelight.com  Tue May  5 15:16:13 2020
From: justin at corelight.com (Justin Azoff)
Date: Tue, 5 May 2020 18:16:13 -0400
Subject: [Zeek] Zeek won't extract exe and office files
In-Reply-To: <CAMh3h1wB=iio6u6-27ty+rUKeXw=8kgXJn2e+Q4eEQis6Yx1hQ@mail.gmail.com>
References: <CAMh3h1x+W9_HKtb_w9T1OkPS-hV7mQyguN32sk-B16Ow-2kDAw@mail.gmail.com>
	<CAMh3h1wB=iio6u6-27ty+rUKeXw=8kgXJn2e+Q4eEQis6Yx1hQ@mail.gmail.com>
Message-ID: <CAPfnCuiKbruJX7NEpYo_exzUDBe6n4V=781gvp50Tzs3nT_tBw@mail.gmail.com>

What does the conn and http log entry look like for the file transfers
that are not  being extracted?

On Tue, May 5, 2020 at 5:33 PM Hank Duo <jradih20 at gmail.com> wrote:
>
> Hi,
> Following my previous email, Zeek started extracting some .exe files but not all. If for example I download twenty .exe files over http from a certain website, Zeek extracts like 2 or 3 out of 20. Is there a reason why Zeek is not recognizing and extracting all.exe files? Also, I added Binary .bin files to be extracted, however it is not extracting them.
> Note: I am downloading all files over http protocol only and not SSL.
> Thank you for your help
> Regards,
> Hank
>
> On Wed, 29 Apr 2020 at 23:42, Hank Duo <jradih20 at gmail.com> wrote:
>>
>> Hi all,
>>
>> I would like to extract .exe and office files for static and dynamic malware analysis purpose. I used the attached script however .exe or .docs files are not extracted except for html, txt or zip files.
>>
>> Note that I modified the main.zeek file which is located in /usr/local/zeek/share/zeek/zeekctl/main.bro by adding @load /frameworks/files/extract-myfiles (which is the script file name) and commented the default one and the script was applied properly.
>>
>> Also, is there a way to extract files only from http or smb protocols while excluding https?
>> Thank you guys
>>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
Justin


From maware at ucsc.edu  Tue May  5 15:32:34 2020
From: maware at ucsc.edu (Mike Ware)
Date: Tue, 5 May 2020 15:32:34 -0700
Subject: [Zeek] Zeek won't extract exe and office files
In-Reply-To: <CAPfnCuiKbruJX7NEpYo_exzUDBe6n4V=781gvp50Tzs3nT_tBw@mail.gmail.com>
References: <CAMh3h1x+W9_HKtb_w9T1OkPS-hV7mQyguN32sk-B16Ow-2kDAw@mail.gmail.com>
	<CAMh3h1wB=iio6u6-27ty+rUKeXw=8kgXJn2e+Q4eEQis6Yx1hQ@mail.gmail.com>
	<CAPfnCuiKbruJX7NEpYo_exzUDBe6n4V=781gvp50Tzs3nT_tBw@mail.gmail.com>
Message-ID: <CAHncJ-H=0tuKQGYuCN0S3Rr51036KQ7CyouSTatgDmhbg==LDg@mail.gmail.com>

Could this be caused by capture loss? If you don't have all the packets you
can't reconstruct.

On Tue, May 5, 2020, 15:18 Justin Azoff <justin at corelight.com> wrote:

> What does the conn and http log entry look like for the file transfers
> that are not  being extracted?
>
> On Tue, May 5, 2020 at 5:33 PM Hank Duo <jradih20 at gmail.com> wrote:
> >
> > Hi,
> > Following my previous email, Zeek started extracting some .exe files but
> not all. If for example I download twenty .exe files over http from a
> certain website, Zeek extracts like 2 or 3 out of 20. Is there a reason why
> Zeek is not recognizing and extracting all.exe files? Also, I added Binary
> .bin files to be extracted, however it is not extracting them.
> > Note: I am downloading all files over http protocol only and not SSL.
> > Thank you for your help
> > Regards,
> > Hank
> >
> > On Wed, 29 Apr 2020 at 23:42, Hank Duo <jradih20 at gmail.com> wrote:
> >>
> >> Hi all,
> >>
> >> I would like to extract .exe and office files for static and dynamic
> malware analysis purpose. I used the attached script however .exe or .docs
> files are not extracted except for html, txt or zip files.
> >>
> >> Note that I modified the main.zeek file which is located in
> /usr/local/zeek/share/zeek/zeekctl/main.bro by adding @load
> /frameworks/files/extract-myfiles (which is the script file name) and
> commented the default one and the script was applied properly.
> >>
> >> Also, is there a way to extract files only from http or smb protocols
> while excluding https?
> >> Thank you guys
> >>
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
>
> --
> Justin
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200505/007290af/attachment.html 

From jradih20 at gmail.com  Tue May  5 19:45:09 2020
From: jradih20 at gmail.com (Hank Duo)
Date: Wed, 6 May 2020 05:45:09 +0300
Subject: [Zeek] Zeek won't extract exe and office files
In-Reply-To: <CAPfnCuiKbruJX7NEpYo_exzUDBe6n4V=781gvp50Tzs3nT_tBw@mail.gmail.com>
References: <CAMh3h1x+W9_HKtb_w9T1OkPS-hV7mQyguN32sk-B16Ow-2kDAw@mail.gmail.com>
	<CAMh3h1wB=iio6u6-27ty+rUKeXw=8kgXJn2e+Q4eEQis6Yx1hQ@mail.gmail.com>
	<CAPfnCuiKbruJX7NEpYo_exzUDBe6n4V=781gvp50Tzs3nT_tBw@mail.gmail.com>
Message-ID: <CAMh3h1yLrRongke-dzJdoCQ+F2K-8MFwGDErhKqWR5gnca3o=w@mail.gmail.com>

Looking into http and conn files, I can see that the downloaded .exe files
appear in http.log however most of the time .exe files are not recognized
as application/x-dosexec files. For example, I tried downloading same .exe
file several times until it got recognized only once as x-dosexec file.
Also, there's a delay to present the traffic log in http or conn files.

Note: Due to lack of resources, the lab is made up of a single HP server
that has Windows 10 where three VMware VMs using VMware Workstation Pro.
The Zeek VM works as an IP Forwarder with two interfaces: one is connected
to the client PC (Internally) and the second interface is connected to the
internet. The client PC is a Windows 7 that has a Gateway IP of the
internal interface on Zeek Machine and get internet through Zeek VM.
The third machine is a web server with a single interface that is in the
same subnet as the Zeek second interface (Connected to the internet).

I configured zeek to monitor the internal interface and the subnet of the
client PC.

On Wed, 6 May 2020 at 01:16, Justin Azoff <justin at corelight.com> wrote:

> What does the conn and http log entry look like for the file transfers
> that are not  being extracted?
>
> On Tue, May 5, 2020 at 5:33 PM Hank Duo <jradih20 at gmail.com> wrote:
> >
> > Hi,
> > Following my previous email, Zeek started extracting some .exe files but
> not all. If for example I download twenty .exe files over http from a
> certain website, Zeek extracts like 2 or 3 out of 20. Is there a reason why
> Zeek is not recognizing and extracting all.exe files? Also, I added Binary
> .bin files to be extracted, however it is not extracting them.
> > Note: I am downloading all files over http protocol only and not SSL.
> > Thank you for your help
> > Regards,
> > Hank
> >
> > On Wed, 29 Apr 2020 at 23:42, Hank Duo <jradih20 at gmail.com> wrote:
> >>
> >> Hi all,
> >>
> >> I would like to extract .exe and office files for static and dynamic
> malware analysis purpose. I used the attached script however .exe or .docs
> files are not extracted except for html, txt or zip files.
> >>
> >> Note that I modified the main.zeek file which is located in
> /usr/local/zeek/share/zeek/zeekctl/main.bro by adding @load
> /frameworks/files/extract-myfiles (which is the script file name) and
> commented the default one and the script was applied properly.
> >>
> >> Also, is there a way to extract files only from http or smb protocols
> while excluding https?
> >> Thank you guys
> >>
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
>
> --
> Justin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200506/c341f24f/attachment-0001.html 

From jradih20 at gmail.com  Wed May  6 04:51:14 2020
From: jradih20 at gmail.com (Hank Duo)
Date: Wed, 6 May 2020 14:51:14 +0300
Subject: [Zeek] Zeek won't extract exe and office files
In-Reply-To: <CAHncJ-H=0tuKQGYuCN0S3Rr51036KQ7CyouSTatgDmhbg==LDg@mail.gmail.com>
References: <CAMh3h1x+W9_HKtb_w9T1OkPS-hV7mQyguN32sk-B16Ow-2kDAw@mail.gmail.com>
	<CAMh3h1wB=iio6u6-27ty+rUKeXw=8kgXJn2e+Q4eEQis6Yx1hQ@mail.gmail.com>
	<CAPfnCuiKbruJX7NEpYo_exzUDBe6n4V=781gvp50Tzs3nT_tBw@mail.gmail.com>
	<CAHncJ-H=0tuKQGYuCN0S3Rr51036KQ7CyouSTatgDmhbg==LDg@mail.gmail.com>
Message-ID: <CAMh3h1zEyhc1mT5YsKYmd0COCbO9Ptgm5x63_+voYAgmijZf1w@mail.gmail.com>

Hi Mike, I checked the capture_loss log and found several records there. I
analyzed the traffic using Wireshark and observed packets loss when
downloading the .exe or .bin files. I am not sure what is causing the
problem so I am trying to figure out. Is there any thing to do on Zeek in
such case?


On Wed, 6 May 2020 at 01:32, Mike Ware <maware at ucsc.edu> wrote:

> Could this be caused by capture loss? If you don't have all the packets
> you can't reconstruct.
>
> On Tue, May 5, 2020, 15:18 Justin Azoff <justin at corelight.com> wrote:
>
>> What does the conn and http log entry look like for the file transfers
>> that are not  being extracted?
>>
>> On Tue, May 5, 2020 at 5:33 PM Hank Duo <jradih20 at gmail.com> wrote:
>> >
>> > Hi,
>> > Following my previous email, Zeek started extracting some .exe files
>> but not all. If for example I download twenty .exe files over http from a
>> certain website, Zeek extracts like 2 or 3 out of 20. Is there a reason why
>> Zeek is not recognizing and extracting all.exe files? Also, I added Binary
>> .bin files to be extracted, however it is not extracting them.
>> > Note: I am downloading all files over http protocol only and not SSL.
>> > Thank you for your help
>> > Regards,
>> > Hank
>> >
>> > On Wed, 29 Apr 2020 at 23:42, Hank Duo <jradih20 at gmail.com> wrote:
>> >>
>> >> Hi all,
>> >>
>> >> I would like to extract .exe and office files for static and dynamic
>> malware analysis purpose. I used the attached script however .exe or .docs
>> files are not extracted except for html, txt or zip files.
>> >>
>> >> Note that I modified the main.zeek file which is located in
>> /usr/local/zeek/share/zeek/zeekctl/main.bro by adding @load
>> /frameworks/files/extract-myfiles (which is the script file name) and
>> commented the default one and the script was applied properly.
>> >>
>> >> Also, is there a way to extract files only from http or smb protocols
>> while excluding https?
>> >> Thank you guys
>> >>
>> > _______________________________________________
>> > Zeek mailing list
>> > zeek at zeek.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>>
>>
>> --
>> Justin
>>
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200506/0ee79b92/attachment.html 

From akgraner at corelight.com  Wed May  6 07:04:30 2020
From: akgraner at corelight.com (Amber Graner)
Date: Wed, 6 May 2020 10:04:30 -0400
Subject: [Zeek] 1 May 2020 - Community Call Notes and Recording
Message-ID: <CAJhOzurp74BPKCJ9jrreY0dLQwbHhYd0w5ge89tW+P5G0ZvZDw@mail.gmail.com>

Hi all,

Thank you so much for all those who attended the Community Call on Friday 1
May.

Below are the links to the recordings of the call:

Video Recording -
https://www.dropbox.com/s/lsitmuhgfrbauai/1May20_ZeekCommunityCall_Video.mp4?dl=0
Audio only -
https://www.dropbox.com/s/bpndd9f1foymnph/1May20_ZeekCommunityCall_Audio.m4a?dl=0

These monthly calls occur on the 1st Friday of each month and are open to
anyone in the community. On these calls we look at ways to help the
community get the most out of Zeek Project. This call is for discussion
around non-code contributions, participation, suggestions, problems and
feedback.

If you have questions, ideas, suggestion, feedback or would like to help
with any of the below listed topics/ideas please let me know.

Thanks,
~Amber

******* Notes and links from call below *******

We had 14 people on the call Friday.  The agenda was an open agenda with a
goal of how to get the most out of these monthly calls.  What would make
the calls better and what would the community like to see more of? The
following is a summary of the discussion and do not always follow the order
of the conversation:

  - That the call is a great place to bring up Issues, Problems,
Suggestions, Ideas as well as the areas below:
    * the mailing list [0] and slack [1] are good places to start
    * Issue tracker on GitHub is also a good place to file tickets against
the Zeek Release

 - Zeek Package Contest
    * ZPC-2  [2] - Reminder that it is still underway and that everyone can
still participate and have the opportunity to win prizes.
    * Idea brought up by the community have a contest that matches people
with package ideas but may not know how to write the packages with
developers who know how.  (Think Google's Summer of Code [3] and Season of
Doc's [4] style events, but around Zeek Packages) - An idea registry to
start - have someone keep it organized by skill level and classification of
ideas.  Build in some incentives.

- Spicy[5] has been released and there seems to be a lot of activity on
Slack around using Spicy to write parsers. Check out this and more on
slack.

 - Information Sharing
   * Encouraging people and organizations to share the cool stuff they are
doing with Zeek.  What are some ways the community can encourage one
another to do that.  Some folks volunteered to talk more about what they
were doing.  We do have the SIEM slack Channel where people are sharing
queries, but is that enough?  Should we have a "use cases" channel or
should the SIEM channel be repurposed for "use cases".

  * Sigma [6] discussion and explanation - (Sigma is a generic and open
signature format that allows you to describe relevant log events in a
straightforward manner.) Nick also mention the uncoder.io[7] site by SOC
Prime[8]

  * Add a space and encourage discussion about threat hunting principles,
threat modeling, best practices. (Documentation and Training Sessions)

  * Folks on the call were asking about getting more information and
tutorials around scripting at all levels.

  * Encourage organizations who are using Zeek and have written packages to
open source those packages[9] and share with the community.

  * Easily searchable Knowledge Base for those getting started is needed. -
this would be in addition to Read The Docs [10] and try.zeek.org [11] -
things like a list of Packages that people would like to see written,
Howtos, List of PCAPs people can use to test Packages, HowTo webinars etc.


   * Best Practice/guides to analyzing the Zeek Logs with Elastic[12] and
Kibana [13] to start.

  * Feedback - It was brought up that someone had filed a ticket [14] and
hadn't gotten an answer or a response in a couple days.  We told them we'd
look into it, but it is an open source project, most everyone working on
the Zeek Project is a volunteer and to also try bringing it up on the
mailing list and the slack channel.

  * Corelight's Support of the Zeek Project - Greg Bell, CEO of Corelight
volunteered to give a report to the community on how Corelight [15]
allocates resources in support of the Zeek Project.  (We'll get this
scheduled for a later date and give plenty of notice to the community as it
is a topic that comes up often)



[0] - Zeek Mailing lists - https://zeek.org/mailing-lists
<https://zeek.org/mailing-lists>
[1] - Zeek Slack Space -
https://join.slack.com/t/zeekorg/shared_invite/enQtOTc3MzMxNDI1NDYxLTA1NzhhMTgxNWI1OTk2NjlkMTdjNzY1Nzk5NDk2ZDY1MDBkYWIxOWNjNDE2NDc2MGI5OWM3ZDllYzBmZmNhNDM
[2} - ZPC-2 - https://zeek.org/2020/04/06/zeek-package-contest-zpc-2/
[3]  - Google Summer of Code - https://summerofcode.withgoogle.com/
[4] - Google Season of Docs - https://developers.google.com/season-of-docs
[5] - Spicy - https://docs.zeek.org/projects/spicy/en/latest/
[6] - Sigma - https://github.com/Neo23x0/sigma
[7] - uncoder.io - https://uncoder.io/
[8] - SOC Prime - https://socprime.com/en/
[9] - Open Source Zeek Packages - https://packages.zeek.org/
[10] - Read the Docs - https://packages.zeek.org/
[11] - Try.zeek.org - https://try.zeek.org/#/?example=hello
[12] - Elastic - https://www.elastic.co/
[13] - Kibana - https://www.elastic.co/guide/en/kibana/current/index.html
[14] - Issue tracker - https://github.com/zeek/zeek/issues
[15] - Corelight - https://www.corelight.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200506/07312d06/attachment-0001.html 

From phil at brimsecurity.com  Wed May  6 12:22:55 2020
From: phil at brimsecurity.com (Phil Rzewski)
Date: Wed, 6 May 2020 12:22:55 -0700
Subject: [Zeek] Brim application for Zeek logs & packet captures (update)
In-Reply-To: <76C52729-DA8F-49F4-BEAB-78968CF7D8AC@brimsecurity.com>
References: <4D066D96-61FD-406A-AE49-ED1CFF63D352@brimsecurity.com>
	<76C52729-DA8F-49F4-BEAB-78968CF7D8AC@brimsecurity.com>
Message-ID: <AD18D6DD-8453-4D78-BDA8-FA5E8D44D8CB@brimsecurity.com>

Zeek community,

It's been a while since we first notified folks here when we initially released our Brim desktop app. Here's an update that we expect will interest the Zeek community.

Whereas the first version of Brim was all about starting from packet captures and turning those into Zeek logs, the new version v0.9.1 we just released introduces direct import of Zeek logs (default TSV format or JSON <https://github.com/brimsec/brim/wiki/Zeek-JSON-Import>). So you can now have all the querying, workflows, etc. for working with your Zeek data in Brim even if you don't have pcaps.

For more details, here's some links for Brim:
Download page <https://www.brimsecurity.com/download/> for the application
GitHub repo <https://github.com/brimsec/brim> for the project
Brim's YouTube channel <https://www.youtube.com/channel/UC0ju7Esmh13oLS8FTS-B3Eg> with a complete video on how to use Brim (which covers the pcap-centric workflow, but includes plenty of coverage for working with Zeek data)
Join our public Slack workspace <https://join.slack.com/t/brimsec/shared_invite/zt-cy34xoxg-hZiTKUT~1KdGjlaBIuUUdg> for announcements, Q&A, feedback, and to trade ideas
...or contact us via email <mailto:info at brimsecurity.com> 

On a separate-but-related topic, while it's not visible in the Brim app yet, our related project zq <https://github.com/brimsec/zq> includes an experimental prototype for working with archived Zeek logs called "zar" that's referenced in a new README <https://github.com/brimsec/zq/blob/master/cmd/zar/README.md>. If this topic interests you, check it out and come talk to us on our Slack <https://join.slack.com/t/brimsec/shared_invite/zt-cy34xoxg-hZiTKUT~1KdGjlaBIuUUdg> in the #zar channel.

Happy hunting!

--
The Brim team


> On Mar 24, 2020, at 5:10 PM, Phil Rzewski <phil at brimsecurity.com> wrote:
> 
> Zeek community,
> 
> I'm reaching out to announce another open source project... specifically the Brim <https://www.brimsecurity.com/download/> desktop application.
> 
> In its first version, the Brim workflow is tuned for starting from a packet capture (even a big one), which the app turns into Zeek logs for you. Then you've got an intuitive UI experience for querying those Zeek logs using the same ZQL <https://github.com/brimsec/zq/tree/master/zql/docs> language you may already know from zq <https://github.com/brimsec/zq> (see prior announcement below). And should your Zeek explorations lead you to a flow for which you want to see the packets, a single click in the app quickly extracts the flow from the big pcap and opens it immediately in Wireshark.
> 
> For more details, here's some links for Brim:
> Download page <https://www.brimsecurity.com/download/> for the application
> GitHub repo <https://github.com/brimsec/brim> for the project
> Brim's YouTube channel <https://www.youtube.com/channel/UC0ju7Esmh13oLS8FTS-B3Eg> with a complete video on how to use Brim
> Join our public Slack workspace <https://join.slack.com/t/brimsec/shared_invite/zt-cy34xoxg-hZiTKUT~1KdGjlaBIuUUdg> for announcements, Q&A, feedback, and to trade ideas
> ...or contact us via email <mailto:info at brimsecurity.com> 
> 
> There's more coming soon, so keep your eye on the repo for updates.
> 
> Happy hunting!
> 
> --
> The Brim team
> 
> 
>> On Feb 11, 2020, at 3:42 PM, Phil Rzewski <phil at brimsecurity.com <mailto:phil at brimsecurity.com>> wrote:
>> 
>> Zeek community,
>> 
>> We?re writing to let you know about zq <https://github.com/brimsec/zq>, an open source command-line processor for structured logs, built for Zeek. (In fact, we?ve been told zq is ?like zeek-cut on steroids?.)
>> 
>> Those of you who were on the ?Ask the Zeeksperts? call on January 16th saw Seth Hall and Justin Azoff give an early peek of zq (thanks guys!), so this is just an ?official? announcement. Come one, come all!
>> 
>> You can get involved by:
>> 	? Checking out the zq GitHub repo <https://github.com/brimsec/zq> for install info, code, and docs
>> 	? Joining our public Slack <https://join.slack.com/t/brimsec/shared_invite/enQtOTMwMDczODg2ODgyLTk1NTdjOTQxNmI0OGYwOThiYzNlNDc5OWI5NjczZDljNDdmZGZjNGI3NTNiOWRiNzJkMzg4OTEwZWM0Y2NiYWQ> workspace for announcements, Q&A, and to trade query ideas
>> 	? Contacting us directly via email <mailto:info at brimsecurity.com> to schedule a Zoom videoconference
>> 
>> All you need is some Zeek logs (and there?s sample logs <https://github.com/brimsec/zq-sample-data> to help you get started). Here?s just a taste of what?s possible:
>> 
>> - A table of top hosts in a subnet that are experiencing the most SYNs-without-ACK:
>>    zq -f table "10.164.94.0/24 conn_state=S0 | count() by id.orig_h | sort -r" *
>> 
>> - A regex search for certain HTTP methods, with full events output as NDJSON:
>>     zq -f ndjson "method=/^(PUT|PATCH|UPDATE)$/" *
>> 
>> - Connections open a long time with low traffic, printed as a Zeek TSV log:
>>    zq -f zeek "duration>1000 orig_bytes<10 resp_bytes<10" *
>> 
>> Of course, that?s just scratching the surface. Please try it out and let us know what you think on GitHub <https://github.com/brimsec/zq> or Slack <https://join.slack.com/t/brimsec/shared_invite/enQtOTMwMDczODg2ODgyLTk1NTdjOTQxNmI0OGYwOThiYzNlNDc5OWI5NjczZDljNDdmZGZjNGI3NTNiOWRiNzJkMzg4OTEwZWM0Y2NiYWQ>.
>> 
>> Happy hunting, Zeeking, & zq?ing!
>> 
>> --
>> The Brim team
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200506/2963f952/attachment.html 

From jsiwek at corelight.com  Wed May  6 13:19:11 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Wed, 6 May 2020 13:19:11 -0700
Subject: [Zeek] Zeek 3.0.6 and 3.1.3 release (security + bug fixes)
Message-ID: <CAMzgZ0L=vE4=7rz06W2UHpeZ+DxADQZL2og3F78a2FbCLAp2CA@mail.gmail.com>

Downloads for Zeek 3.0.6 (LTS) and Zeek 3.1.3 are available:

    https://zeek.org/get-zeek

See the release notes for details of the security/bug fixes:

    https://github.com/zeek/zeek/releases/tag/v3.0.6
    https://github.com/zeek/zeek/releases/tag/v3.1.3

From clopmz at outlook.com  Thu May  7 01:44:50 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Thu, 7 May 2020 08:44:50 +0000
Subject: [Zeek] Anyone using DoveHawk under Zeek 3.0.6
Message-ID: <6B45875C-AA49-45BA-AEDD-D21BDF6E516A@outlook.com>

Hi all,

Today I have updated my Zeek cluster to release 3.0.6. I have installed dovehawk package also, but it is not downloading IOC from my MISP instance. Errors are:

{"ts":"2020-05-07T08:40:39.980846Z","level":"Reporter::ERROR","message":"curl --header \"Authorization: \"AFz6lL1d2PMLPQ9O1V7OVoCOhawbXEZ9S01wu5GS\"\" -s -g -o \"\"/tmp/zeek-activehttp-E4Z8Vauqaq3_body\"\
" -D \"\"/tmp/zeek-activehttp-E4Z8Vauqaq3_headers\"\" -X \"\"GET\"\" -m 60 \"\"https://stonehaven.lab.uxdom.org/attributes/bro/download/all\"\" && touch \"/tmp/zeek-activehttp-E4Z8Vauqaq3_body\" |/Input::
READER_RAW: Child process exited with non-zero return code 60","location":""}
{"ts":"2020-05-07T08:40:39.980846Z","level":"Reporter::ERROR","message":"curl --header \"Authorization: \"AFz6lL1d2PMLPQ9O1V7OVoCOhawbXEZ9S01wu5GS\"\" -s -g -o \"\"/tmp/zeek-activehttp-FRYEeOTxgol_body\"\
" -D \"\"/tmp/zeek-activehttp-FRYEeOTxgol_headers\"\" -X \"\"GET\"\" -m 60 \"\"https://stonehaven.lab.uxdom.org/attributes/text/download/zeek\"\" && touch \"/tmp/zeek-activehttp-FRYEeOTxgol_body\" |/Input
::READER_RAW: Child process exited with non-zero return code 60","location":""}
{"ts":"2020-05-07T08:40:39.982698Z","level":"Reporter::ERROR","message":"/tmp/zeek-activehttp-E4Z8Vauqaq3_body/Input::READER_RAW: Init: cannot open /tmp/zeek-activehttp-E4Z8Vauqaq3_body","location":""}
{"ts":"2020-05-07T08:40:39.982698Z","level":"Reporter::ERROR","message":"/tmp/zeek-activehttp-E4Z8Vauqaq3_body/Input::READER_RAW: Init failed","location":""}
{"ts":"2020-05-07T08:40:39.982698Z","level":"Reporter::ERROR","message":"/tmp/zeek-activehttp-E4Z8Vauqaq3_body/Input::READER_RAW: terminating thread","location":""}
{"ts":"2020-05-07T08:40:39.982698Z","level":"Reporter::ERROR","message":"/tmp/zeek-activehttp-FRYEeOTxgol_body/Input::READER_RAW: Init: cannot open /tmp/zeek-activehttp-FRYEeOTxgol_body","location":""}
{"ts":"2020-05-07T08:40:39.982698Z","level":"Reporter::ERROR","message":"/tmp/zeek-activehttp-FRYEeOTxgol_body/Input::READER_RAW: Init failed","location":""}
{"ts":"2020-05-07T08:40:39.982698Z","level":"Reporter::ERROR","message":"/tmp/zeek-activehttp-FRYEeOTxgol_body/Input::READER_RAW: terminating thread","location":""}

Any light on this?

Regards,
C. L. Martinez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200507/d45b3fa1/attachment.html 

From justin at corelight.com  Thu May  7 06:08:44 2020
From: justin at corelight.com (Justin Azoff)
Date: Thu, 7 May 2020 09:08:44 -0400
Subject: [Zeek] Anyone using DoveHawk under Zeek 3.0.6
In-Reply-To: <6B45875C-AA49-45BA-AEDD-D21BDF6E516A@outlook.com>
References: <6B45875C-AA49-45BA-AEDD-D21BDF6E516A@outlook.com>
Message-ID: <CAPfnCuje+-tbLdkkDeHh45Wh0v+jaN4U2V=y+dy-HodTGfQPUg@mail.gmail.com>

> Child process exited with non-zero return code 60

CURLE_PEER_FAILED_VERIFICATION (60)

The remote server's SSL certificate or SSH md5 fingerprint was deemed
not OK. This error code has been unified with CURLE_SSL_CACERT since
7.62.0. Its previous value was 51.

The sites SSL certificate isn't trusted by your client.

If you run a

curl -v https://stonehaven.lab.uxdom.org/

you should see the same problem.

On Thu, May 7, 2020 at 4:47 AM Carlos Lopez <clopmz at outlook.com> wrote:
>
> Hi all,
>
>
>
> Today I have updated my Zeek cluster to release 3.0.6. I have installed dovehawk package also, but it is not downloading IOC from my MISP instance. Errors are:
>
>
>
> {"ts":"2020-05-07T08:40:39.980846Z","level":"Reporter::ERROR","message":"curl --header \"Authorization: \"AFz6lL1d2PMLPQ9O1V7OVoCOhawbXEZ9S01wu5GS\"\" -s -g -o \"\"/tmp/zeek-activehttp-E4Z8Vauqaq3_body\"\
>
> " -D \"\"/tmp/zeek-activehttp-E4Z8Vauqaq3_headers\"\" -X \"\"GET\"\" -m 60 \"\"https://stonehaven.lab.uxdom.org/attributes/bro/download/all\"\" && touch \"/tmp/zeek-activehttp-E4Z8Vauqaq3_body\" |/Input::
>
> READER_RAW: Child process exited with non-zero return code 60","location":""}
>
> {"ts":"2020-05-07T08:40:39.980846Z","level":"Reporter::ERROR","message":"curl --header \"Authorization: \"AFz6lL1d2PMLPQ9O1V7OVoCOhawbXEZ9S01wu5GS\"\" -s -g -o \"\"/tmp/zeek-activehttp-FRYEeOTxgol_body\"\
>
> " -D \"\"/tmp/zeek-activehttp-FRYEeOTxgol_headers\"\" -X \"\"GET\"\" -m 60 \"\"https://stonehaven.lab.uxdom.org/attributes/text/download/zeek\"\" && touch \"/tmp/zeek-activehttp-FRYEeOTxgol_body\" |/Input
>
> ::READER_RAW: Child process exited with non-zero return code 60","location":""}
>
> {"ts":"2020-05-07T08:40:39.982698Z","level":"Reporter::ERROR","message":"/tmp/zeek-activehttp-E4Z8Vauqaq3_body/Input::READER_RAW: Init: cannot open /tmp/zeek-activehttp-E4Z8Vauqaq3_body","location":""}
>
> {"ts":"2020-05-07T08:40:39.982698Z","level":"Reporter::ERROR","message":"/tmp/zeek-activehttp-E4Z8Vauqaq3_body/Input::READER_RAW: Init failed","location":""}
>
> {"ts":"2020-05-07T08:40:39.982698Z","level":"Reporter::ERROR","message":"/tmp/zeek-activehttp-E4Z8Vauqaq3_body/Input::READER_RAW: terminating thread","location":""}
>
> {"ts":"2020-05-07T08:40:39.982698Z","level":"Reporter::ERROR","message":"/tmp/zeek-activehttp-FRYEeOTxgol_body/Input::READER_RAW: Init: cannot open /tmp/zeek-activehttp-FRYEeOTxgol_body","location":""}
>
> {"ts":"2020-05-07T08:40:39.982698Z","level":"Reporter::ERROR","message":"/tmp/zeek-activehttp-FRYEeOTxgol_body/Input::READER_RAW: Init failed","location":""}
>
> {"ts":"2020-05-07T08:40:39.982698Z","level":"Reporter::ERROR","message":"/tmp/zeek-activehttp-FRYEeOTxgol_body/Input::READER_RAW: terminating thread","location":""}
>
>
>
> Any light on this?
>
>
>
> Regards,
>
> C. L. Martinez
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
Justin

From clopmz at outlook.com  Thu May  7 06:16:28 2020
From: clopmz at outlook.com (Carlos Lopez)
Date: Thu, 7 May 2020 13:16:28 +0000
Subject: [Zeek] Anyone using DoveHawk under Zeek 3.0.6
In-Reply-To: <CAPfnCuje+-tbLdkkDeHh45Wh0v+jaN4U2V=y+dy-HodTGfQPUg@mail.gmail.com>
References: <6B45875C-AA49-45BA-AEDD-D21BDF6E516A@outlook.com>
	<CAPfnCuje+-tbLdkkDeHh45Wh0v+jaN4U2V=y+dy-HodTGfQPUg@mail.gmail.com>
Message-ID: <13A4BA66-6920-455A-9753-8A38B2F45039@outlook.com>

Yep ... It was the certificate ... I have added "-k" flag to curl and all works ok now ...

Thanks.

?On 07/05/2020, 15:09, "Justin Azoff" <justin at corelight.com> wrote:

    > Child process exited with non-zero return code 60

    CURLE_PEER_FAILED_VERIFICATION (60)

    The remote server's SSL certificate or SSH md5 fingerprint was deemed
    not OK. This error code has been unified with CURLE_SSL_CACERT since
    7.62.0. Its previous value was 51.

    The sites SSL certificate isn't trusted by your client.

    If you run a

    curl -v https://stonehaven.lab.uxdom.org/

    you should see the same problem.

    On Thu, May 7, 2020 at 4:47 AM Carlos Lopez <clopmz at outlook.com> wrote:
    >
    > Hi all,
    >
    >
    >
    > Today I have updated my Zeek cluster to release 3.0.6. I have installed dovehawk package also, but it is not downloading IOC from my MISP instance. Errors are:
    >
    >
    >
    > {"ts":"2020-05-07T08:40:39.980846Z","level":"Reporter::ERROR","message":"curl --header \"Authorization: \"AFz6lL1d2PMLPQ9O1V7OVoCOhawbXEZ9S01wu5GS\"\" -s -g -o \"\"/tmp/zeek-activehttp-E4Z8Vauqaq3_body\"\
    >
    > " -D \"\"/tmp/zeek-activehttp-E4Z8Vauqaq3_headers\"\" -X \"\"GET\"\" -m 60 \"\"https://stonehaven.lab.uxdom.org/attributes/bro/download/all\"\" && touch \"/tmp/zeek-activehttp-E4Z8Vauqaq3_body\" |/Input::
    >
    > READER_RAW: Child process exited with non-zero return code 60","location":""}
    >
    > {"ts":"2020-05-07T08:40:39.980846Z","level":"Reporter::ERROR","message":"curl --header \"Authorization: \"AFz6lL1d2PMLPQ9O1V7OVoCOhawbXEZ9S01wu5GS\"\" -s -g -o \"\"/tmp/zeek-activehttp-FRYEeOTxgol_body\"\
    >
    > " -D \"\"/tmp/zeek-activehttp-FRYEeOTxgol_headers\"\" -X \"\"GET\"\" -m 60 \"\"https://stonehaven.lab.uxdom.org/attributes/text/download/zeek\"\" && touch \"/tmp/zeek-activehttp-FRYEeOTxgol_body\" |/Input
    >
    > ::READER_RAW: Child process exited with non-zero return code 60","location":""}
    >
    > {"ts":"2020-05-07T08:40:39.982698Z","level":"Reporter::ERROR","message":"/tmp/zeek-activehttp-E4Z8Vauqaq3_body/Input::READER_RAW: Init: cannot open /tmp/zeek-activehttp-E4Z8Vauqaq3_body","location":""}
    >
    > {"ts":"2020-05-07T08:40:39.982698Z","level":"Reporter::ERROR","message":"/tmp/zeek-activehttp-E4Z8Vauqaq3_body/Input::READER_RAW: Init failed","location":""}
    >
    > {"ts":"2020-05-07T08:40:39.982698Z","level":"Reporter::ERROR","message":"/tmp/zeek-activehttp-E4Z8Vauqaq3_body/Input::READER_RAW: terminating thread","location":""}
    >
    > {"ts":"2020-05-07T08:40:39.982698Z","level":"Reporter::ERROR","message":"/tmp/zeek-activehttp-FRYEeOTxgol_body/Input::READER_RAW: Init: cannot open /tmp/zeek-activehttp-FRYEeOTxgol_body","location":""}
    >
    > {"ts":"2020-05-07T08:40:39.982698Z","level":"Reporter::ERROR","message":"/tmp/zeek-activehttp-FRYEeOTxgol_body/Input::READER_RAW: Init failed","location":""}
    >
    > {"ts":"2020-05-07T08:40:39.982698Z","level":"Reporter::ERROR","message":"/tmp/zeek-activehttp-FRYEeOTxgol_body/Input::READER_RAW: terminating thread","location":""}
    >
    >
    >
    > Any light on this?
    >
    >
    >
    > Regards,
    >
    > C. L. Martinez
    >
    > _______________________________________________
    > Zeek mailing list
    > zeek at zeek.org
    > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



    -- 
    Justin



From antho.arnaudisce at gmail.com  Thu May  7 09:24:03 2020
From: antho.arnaudisce at gmail.com (Anthony Arnaud)
Date: Thu, 7 May 2020 18:24:03 +0200
Subject: [Zeek] Zeek Vs. FreeBSD
Message-ID: <CAGCUUMFpjv5oz-GsQHX-j3fQYVJ8nst8pMtV__62LLpXdvFygQ@mail.gmail.com>

Hi All,
I tried to install Zeek on my FreeBSD server with netmap support.
But VirtIO Ethernet driver is not working properly, there are performance
problems that should be solved in the latest Netmap release, ref to:

https://reviews.freebsd.org/D17916

Unfortunately the bro-netmap plugin does not work with that.
It seems that Zeek is unusable in FreeBSD env, the developments of the
bro-netmap plugin are closed and it is impossible to parallelize network
traffic on multiple zeek workers.
Does anyone know if updates are currently planned?
Or if someone using this plugin with the Netmap last version?
Or, finally, are there other BSD loadbalancing solutions ?
Thank y'all

Anthon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200507/845fad32/attachment.html 

From hovsep.sanjay.levi at gmail.com  Thu May  7 10:12:51 2020
From: hovsep.sanjay.levi at gmail.com (Hovsep Levi)
Date: Thu, 7 May 2020 17:12:51 +0000
Subject: [Zeek] Zeek 3.1.2 and Kafka - No data flow
In-Reply-To: <CAM1mTCoP_cCfw7QhxMUxi+iNB1Y3Oo+1A-SEkXqNDMzSfqEM0Q@mail.gmail.com>
References: <CAJ19UZCKWL+OuOS2XcWGbaYexyPmeGJ1L3XSW4nFf89Xz_k1WQ@mail.gmail.com>
	<CAM1mTCqZ4kpsD6KqN1P9+8zcE9PX_-_X_VwTTOzCWaOkhQT5_Q@mail.gmail.com>
	<CAM1mTCoNK8UbSUJH3OgqSjGi4=KSk15Zxcwd9aUDkzcxaEDoLw@mail.gmail.com>
	<CAM1mTCoP_cCfw7QhxMUxi+iNB1Y3Oo+1A-SEkXqNDMzSfqEM0Q@mail.gmail.com>
Message-ID: <CAJ19UZBKbwEe4eAhxFbSiHzLG6SKD=ZR32APe3nFcCKaoB4qKA@mail.gmail.com>

It is not working yet for me and was set aside to fix another time.

Very glad to hear about pull 44, I will test !

/Hovsep


On Mon, May 4, 2020 at 5:01 PM Zeolla at GMail.com <zeolla at gmail.com> wrote:

> I have the plugin working with 3.1.2 here
> <https://github.com/apache/metron-bro-plugin-kafka/pull/44> - feedback is
> welcome.
>
> - Jon Zeolla
> Zeolla at GMail.Com
>
>
> On Wed, Apr 29, 2020 at 3:35 PM Zeolla at GMail.com <zeolla at gmail.com> wrote:
>
>> Were you able to get this working?  I'm planning to work on the bro to
>> zeek cutover for the plugin soon.
>>
>> - Jon Zeolla
>> Zeolla at GMail.Com
>>
>>
>> On Mon, Apr 27, 2020 at 6:39 AM Zeolla at GMail.com <zeolla at gmail.com>
>> wrote:
>>
>>> I have not run it on 3.1.2 yet but I recommend making your changes to
>>> the plugin and running the end to end testing script at
>>> https://github.com/apache/metron-bro-plugin-kafka/blob/master/docker/run_end_to_end.sh
>>>
>>> It was meant to help isolate issues when making changes to the plugin.
>>> Also, we welcome PRs against the project so please feel free to
>>> contribute.  Thanks,
>>>
>>> Jon Zeolla
>>>
>>> On Sun, Apr 26, 2020, 9:12 PM Hovsep Levi <hovsep.sanjay.levi at gmail.com>
>>> wrote:
>>>
>>>> Hello Zeeks
>>>>
>>>>
>>>> Has anyone succeeded to enable Kafka plugin with Zeek 3.1.2 ?  I am
>>>> trying to modernize the metron-kafka plugin and have partial success.  My
>>>> problem seems to be with script-land referencing.
>>>>
>>>> The logger node is loading the plugin OK and connects to the Kafka
>>>> broker.  The broker IP is redef information found from site/local.zeek.
>>>>
>>>> $ bin/zeekctl diag logger-1
>>>> [logger-1]
>>>>
>>>> No core file found.
>>>>
>>>> Zeek 3.1.2-debug
>>>>
>>>> Zeek plugins:
>>>> Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)
>>>>
>>>> ==== No reporter.log
>>>>
>>>> ==== stderr.log
>>>> %7|1587948661.341|RECV|rdkafka#producer-3| [..kafka messages..] ..
>>>>
>>>>
>>>>
>>>>
>>>> But the worker node has a problem referencing existing variable
>>>> declaration.  The logs-to-kafka.bro script expects it.  There is also
>>>> suspicion with the Zeek plugins info that is different from the logger node
>>>> and maybe the problem.
>>>>
>>>> $ bin/zeekctl diag worker-1-1
>>>> [worker-1-1]
>>>>
>>>> No core file found.
>>>>
>>>> Zeek 3.1.2-debug
>>>>
>>>> Zeek plugins: (none found)   <<< ??? Normal for worker node ???
>>>>
>>>> ==== No reporter.log
>>>>
>>>> ==== stderr.log
>>>>
>>>> error in
>>>> /opt/zeek/spool/installed-scripts-do-not-touch/site/custom_plugins/APACHE_KAFKA/scripts/Apache/Kafka/./logs-to-kafka.bro,
>>>> line 24: unknown identifier logs_to_send, at or near "logs_to_send"
>>>>
>>>>
>>>>
>>>> The configuration is not default and explained below:
>>>>
>>>>
>>>> The Kafka logger was installed to site/custom_plugins/APACHE_KAFKA
>>>>
>>>>
>>>> share/zeek/site/local.zeek uses:
>>>>
>>>> @load custom_plugins/APACHE_KAFKA/scripts/Apache/Kafka
>>>>
>>>>
>>>>
>>>> lib/zeek/plugins/custom_plugins is a symlink to
>>>> share/zeek/site/custom_plugins
>>>>
>>>>
>>>> Using the lib symlink seems to be the only way to load the plugin, then
>>>> the @load statement brings redef customizations and scripts.  This works ok
>>>> for the logger node but not the worker who cannot interface with the plugin
>>>> ?
>>>>
>>>> Another idea is have non-logger nodes bypass loading logs-to-kafka.bro
>>>> but this isn't fully understood.
>>>>
>>>>
>>>> TIA
>>>>
>>>> /hovsep
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Zeek mailing list
>>>> zeek at zeek.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200507/5454da5a/attachment-0001.html 

From zeolla at gmail.com  Thu May  7 11:17:15 2020
From: zeolla at gmail.com (Zeolla@GMail.com)
Date: Thu, 7 May 2020 14:17:15 -0400
Subject: [Zeek] Zeek 3.1.2 and Kafka - No data flow
In-Reply-To: <CAJ19UZBKbwEe4eAhxFbSiHzLG6SKD=ZR32APe3nFcCKaoB4qKA@mail.gmail.com>
References: <CAJ19UZCKWL+OuOS2XcWGbaYexyPmeGJ1L3XSW4nFf89Xz_k1WQ@mail.gmail.com>
	<CAM1mTCqZ4kpsD6KqN1P9+8zcE9PX_-_X_VwTTOzCWaOkhQT5_Q@mail.gmail.com>
	<CAM1mTCoNK8UbSUJH3OgqSjGi4=KSk15Zxcwd9aUDkzcxaEDoLw@mail.gmail.com>
	<CAM1mTCoP_cCfw7QhxMUxi+iNB1Y3Oo+1A-SEkXqNDMzSfqEM0Q@mail.gmail.com>
	<CAJ19UZBKbwEe4eAhxFbSiHzLG6SKD=ZR32APe3nFcCKaoB4qKA@mail.gmail.com>
Message-ID: <CAM1mTCqHujMD71W6GH=4_Jug15gXNKyQJP1qGb9Z-dzD7p_qHQ@mail.gmail.com>

Master officially supports Zeek 3.1 now so please let me know if you
run into any issues.

- Jon Zeolla
Zeolla at GMail.Com


On Thu, May 7, 2020 at 1:15 PM Hovsep Levi <hovsep.sanjay.levi at gmail.com>
wrote:

> It is not working yet for me and was set aside to fix another time.
>
> Very glad to hear about pull 44, I will test !
>
> /Hovsep
>
>
> On Mon, May 4, 2020 at 5:01 PM Zeolla at GMail.com <zeolla at gmail.com> wrote:
>
>> I have the plugin working with 3.1.2 here
>> <https://github.com/apache/metron-bro-plugin-kafka/pull/44> - feedback
>> is welcome.
>>
>> - Jon Zeolla
>> Zeolla at GMail.Com
>>
>>
>> On Wed, Apr 29, 2020 at 3:35 PM Zeolla at GMail.com <zeolla at gmail.com>
>> wrote:
>>
>>> Were you able to get this working?  I'm planning to work on the bro to
>>> zeek cutover for the plugin soon.
>>>
>>> - Jon Zeolla
>>> Zeolla at GMail.Com
>>>
>>>
>>> On Mon, Apr 27, 2020 at 6:39 AM Zeolla at GMail.com <zeolla at gmail.com>
>>> wrote:
>>>
>>>> I have not run it on 3.1.2 yet but I recommend making your changes to
>>>> the plugin and running the end to end testing script at
>>>> https://github.com/apache/metron-bro-plugin-kafka/blob/master/docker/run_end_to_end.sh
>>>>
>>>> It was meant to help isolate issues when making changes to the plugin.
>>>> Also, we welcome PRs against the project so please feel free to
>>>> contribute.  Thanks,
>>>>
>>>> Jon Zeolla
>>>>
>>>> On Sun, Apr 26, 2020, 9:12 PM Hovsep Levi <hovsep.sanjay.levi at gmail.com>
>>>> wrote:
>>>>
>>>>> Hello Zeeks
>>>>>
>>>>>
>>>>> Has anyone succeeded to enable Kafka plugin with Zeek 3.1.2 ?  I am
>>>>> trying to modernize the metron-kafka plugin and have partial success.  My
>>>>> problem seems to be with script-land referencing.
>>>>>
>>>>> The logger node is loading the plugin OK and connects to the Kafka
>>>>> broker.  The broker IP is redef information found from site/local.zeek.
>>>>>
>>>>> $ bin/zeekctl diag logger-1
>>>>> [logger-1]
>>>>>
>>>>> No core file found.
>>>>>
>>>>> Zeek 3.1.2-debug
>>>>>
>>>>> Zeek plugins:
>>>>> Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)
>>>>>
>>>>> ==== No reporter.log
>>>>>
>>>>> ==== stderr.log
>>>>> %7|1587948661.341|RECV|rdkafka#producer-3| [..kafka messages..] ..
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> But the worker node has a problem referencing existing variable
>>>>> declaration.  The logs-to-kafka.bro script expects it.  There is also
>>>>> suspicion with the Zeek plugins info that is different from the logger node
>>>>> and maybe the problem.
>>>>>
>>>>> $ bin/zeekctl diag worker-1-1
>>>>> [worker-1-1]
>>>>>
>>>>> No core file found.
>>>>>
>>>>> Zeek 3.1.2-debug
>>>>>
>>>>> Zeek plugins: (none found)   <<< ??? Normal for worker node ???
>>>>>
>>>>> ==== No reporter.log
>>>>>
>>>>> ==== stderr.log
>>>>>
>>>>> error in
>>>>> /opt/zeek/spool/installed-scripts-do-not-touch/site/custom_plugins/APACHE_KAFKA/scripts/Apache/Kafka/./logs-to-kafka.bro,
>>>>> line 24: unknown identifier logs_to_send, at or near "logs_to_send"
>>>>>
>>>>>
>>>>>
>>>>> The configuration is not default and explained below:
>>>>>
>>>>>
>>>>> The Kafka logger was installed to site/custom_plugins/APACHE_KAFKA
>>>>>
>>>>>
>>>>> share/zeek/site/local.zeek uses:
>>>>>
>>>>> @load custom_plugins/APACHE_KAFKA/scripts/Apache/Kafka
>>>>>
>>>>>
>>>>>
>>>>> lib/zeek/plugins/custom_plugins is a symlink to
>>>>> share/zeek/site/custom_plugins
>>>>>
>>>>>
>>>>> Using the lib symlink seems to be the only way to load the plugin,
>>>>> then the @load statement brings redef customizations and scripts.  This
>>>>> works ok for the logger node but not the worker who cannot interface with
>>>>> the plugin ?
>>>>>
>>>>> Another idea is have non-logger nodes bypass loading logs-to-kafka.bro
>>>>> but this isn't fully understood.
>>>>>
>>>>>
>>>>> TIA
>>>>>
>>>>> /hovsep
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Zeek mailing list
>>>>> zeek at zeek.org
>>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>>>
>>>> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200507/88d2c84e/attachment.html 

From jwelcher at gmail.com  Thu May  7 11:29:30 2020
From: jwelcher at gmail.com (James Welcher)
Date: Thu, 7 May 2020 11:29:30 -0700
Subject: [Zeek] Zeek Vs. FreeBSD
In-Reply-To: <CAGCUUMFpjv5oz-GsQHX-j3fQYVJ8nst8pMtV__62LLpXdvFygQ@mail.gmail.com>
References: <CAGCUUMFpjv5oz-GsQHX-j3fQYVJ8nst8pMtV__62LLpXdvFygQ@mail.gmail.com>
Message-ID: <CAPS1T28vAUvtGhGg2y1tb6LXEncn6kesksXrt_iQLLULxKKdmA@mail.gmail.com>

We're using FreeBSD with Zeek with Myricom cards and their custom driver to
parallelize. We had to pester ARIA a bit for current 11 and 12 builds but
they produced them and are working fine.

On Thu, May 7, 2020 at 9:32 AM Anthony Arnaud <antho.arnaudisce at gmail.com>
wrote:

> Hi All,
> I tried to install Zeek on my FreeBSD server with netmap support.
> But VirtIO Ethernet driver is not working properly, there are performance
> problems that should be solved in the latest Netmap release, ref to:
>
> https://reviews.freebsd.org/D17916
>
> Unfortunately the bro-netmap plugin does not work with that.
> It seems that Zeek is unusable in FreeBSD env, the developments of the
> bro-netmap plugin are closed and it is impossible to parallelize network
> traffic on multiple zeek workers.
> Does anyone know if updates are currently planned?
> Or if someone using this plugin with the Netmap last version?
> Or, finally, are there other BSD loadbalancing solutions ?
> Thank y'all
>
> Anthon
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
James Welcher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200507/2ba32035/attachment.html 

From shirkdog.bsd at gmail.com  Thu May  7 11:30:37 2020
From: shirkdog.bsd at gmail.com (Michael Shirk)
Date: Thu, 7 May 2020 14:30:37 -0400
Subject: [Zeek] Zeek Vs. FreeBSD
In-Reply-To: <CAGCUUMFpjv5oz-GsQHX-j3fQYVJ8nst8pMtV__62LLpXdvFygQ@mail.gmail.com>
References: <CAGCUUMFpjv5oz-GsQHX-j3fQYVJ8nst8pMtV__62LLpXdvFygQ@mail.gmail.com>
Message-ID: <CAL8PkUUpgnH3z_cEV5EFRvbpuztfP1LTeVFSZ5SW-3iyCDi=cQ@mail.gmail.com>

Some questions to get started:
Which version of FreeBSD are you using?
Which network card are you using?

The biggest issue is parity between the netmap and FreeBSD source
trees, you really need to run FreeBSD-CURRENT to ensure you have all
of the latest changes. There were some issues in the past that
affected even Intel network cards from working correctly, so the types
of cards you are using are very important.

I pushed to get the netmap tools added to the source tree, so you can
build "lb" from the following location and use it:
/usr/src/tools/tools/netmap/lb.c

I am updating a FreeBSD system to see if this still builds correctly
as I have not used LB in a while.

On Thu, May 7, 2020 at 12:32 PM Anthony Arnaud
<antho.arnaudisce at gmail.com> wrote:
>
> Hi All,
> I tried to install Zeek on my FreeBSD server with netmap support.
> But VirtIO Ethernet driver is not working properly, there are performance problems that should be solved in the latest Netmap release, ref to:
>
> https://reviews.freebsd.org/D17916
>
> Unfortunately the bro-netmap plugin does not work with that.
> It seems that Zeek is unusable in FreeBSD env, the developments of the bro-netmap plugin are closed and it is impossible to parallelize network traffic on multiple zeek workers.
> Does anyone know if updates are currently planned?
> Or if someone using this plugin with the Netmap last version?
> Or, finally, are there other BSD loadbalancing solutions ?
> Thank y'all
>
> Anthon
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
Michael Shirk
Daemon Security, Inc.
https://www.daemon-security.com

From antho.arnaudisce at gmail.com  Fri May  8 08:33:42 2020
From: antho.arnaudisce at gmail.com (Anthony Arnaud)
Date: Fri, 8 May 2020 17:33:42 +0200
Subject: [Zeek] Zeek Vs. FreeBSD
In-Reply-To: <CAL8PkUUpgnH3z_cEV5EFRvbpuztfP1LTeVFSZ5SW-3iyCDi=cQ@mail.gmail.com>
References: <CAGCUUMFpjv5oz-GsQHX-j3fQYVJ8nst8pMtV__62LLpXdvFygQ@mail.gmail.com>
	<CAL8PkUUpgnH3z_cEV5EFRvbpuztfP1LTeVFSZ5SW-3iyCDi=cQ@mail.gmail.com>
Message-ID: <CAGCUUMHQizDufq+6djMvXy-O1WuxC81fxn1qwd3JV-TG-nEX7Q@mail.gmail.com>

Hi Michael,
I'm using FreeBSD 12.1 and Zeek 3.0.5 (ver. 3.0.6 available in ports has
compilation problems), with 2 NICs without ip, em0 and vtnet1 (em0 is Intel
e1000) but the problem is that zeek plugin is not updated. (API mismatch)

#ifconfig
vtnet1: flags = 8802 <BROADCAST, SIMPLEX, MULTICAST> metric 0 mtu 1500
        options = c00b8 <VLAN_MTU, VLAN_HWTAGGING, JUMBO_MTU, VLAN_HWCSUM,
VLAN_HWTSO, link-state routing protocol>
        ether something
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
        nd6 options = 29 <PERFORMNUD, IFDISABLED, AUTO_LINKLOCAL>

em0: flags = 8802 <BROADCAST, SIMPLEX, MULTICAST> metric 0 mtu 1500
        options = 812,098 <VLAN_MTU, VLAN_HWTAGGING, VLAN_HWCSUM,
WOL_MAGIC, VLAN_HWFILTER>
        ether something
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options = 29 <PERFORMNUD, IFDISABLED, AUTO_LINKLOCAL>

# on zeekctl start
starting workers ...
Error: worker-1-1 terminated immediately after starting; check output with
"diag"
Error: worker-1-2 terminated immediately after starting; check output with
"diag"

#in dmesg
173.973686 [376] netmap_ioctl_legacy Minimum supported API is 14 (requested
11)
173.973712 [376] netmap_ioctl_legacy Minimum supported API is 14 (requested
11)

*#from zeekctl diag*
Zeek 3.0.5
FreeBSD 12.1-RELEASE

Zeek plugins:
Bro :: Netmap - Packet acquisition via Netmap (dynamic, version 1.0.0)

==== stderr.log
292.768100 nm_open [920] NIOCREGIF failed: Invalid argument vtnet1} 1
fatal error: problem with interface netmap :: vtnet1} 1 (Invalid argument)

The netmap tools in kernel sources seems ok, lb start and the network
interface switch in netmap mode.
I think the latest working version of  plugin is compatible with netmap
release available in FreeBSD 11.2, but there are performance issues with
vtnets.
Also tcpreplay doesn't work when i try to send traffic in netmap mode to a
NIC sniffed by zeek (in FBSD 11.2)

Thanks,
Anthon

Il giorno gio 7 mag 2020 alle ore 20:30 Michael Shirk <
shirkdog.bsd at gmail.com> ha scritto:

> Some questions to get started:
> Which version of FreeBSD are you using?
> Which network card are you using?
>
> The biggest issue is parity between the netmap and FreeBSD source
> trees, you really need to run FreeBSD-CURRENT to ensure you have all
> of the latest changes. There were some issues in the past that
> affected even Intel network cards from working correctly, so the types
> of cards you are using are very important.
>
> I pushed to get the netmap tools added to the source tree, so you can
> build "lb" from the following location and use it:
> /usr/src/tools/tools/netmap/lb.c
>
> I am updating a FreeBSD system to see if this still builds correctly
> as I have not used LB in a while.
>
> On Thu, May 7, 2020 at 12:32 PM Anthony Arnaud
> <antho.arnaudisce at gmail.com> wrote:
> >
> > Hi All,
> > I tried to install Zeek on my FreeBSD server with netmap support.
> > But VirtIO Ethernet driver is not working properly, there are
> performance problems that should be solved in the latest Netmap release,
> ref to:
> >
> > https://reviews.freebsd.org/D17916
> >
> > Unfortunately the bro-netmap plugin does not work with that.
> > It seems that Zeek is unusable in FreeBSD env, the developments of the
> bro-netmap plugin are closed and it is impossible to parallelize network
> traffic on multiple zeek workers.
> > Does anyone know if updates are currently planned?
> > Or if someone using this plugin with the Netmap last version?
> > Or, finally, are there other BSD loadbalancing solutions ?
> > Thank y'all
> >
> > Anthon
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
>
> --
> Michael Shirk
> Daemon Security, Inc.
> https://www.daemon-security.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200508/593623e7/attachment.html 

From akgraner at corelight.com  Fri May  8 12:21:36 2020
From: akgraner at corelight.com (Amber Graner)
Date: Fri, 8 May 2020 15:21:36 -0400
Subject: [Zeek] Upcoming Events For May 2020
Message-ID: <CAJhOzur5PvX=MyMYqESzGgOCHg7FMwuj5MEATtDPmo=FyOFETQ@mail.gmail.com>

Hi all,

I'm thrilled to tell you about the lineup of Zeek online events for May.
There's so many that we needed a separate thread about them.  May is filled
with many new voices that include complementing technologies and tools.

You'll hear from Suricata, Security Onion, Brim and more. In addition there
will be a Community CTF offered this month.

These events and registrations links can be found on the events calendar on
Zeek.org at: https://zeek.org/events/#calendar

--------------------

*13 May* - Zeek From Home - Topic and Host TBD - 2-3pm Eastern

Registration:
https://corelight.zoom.us/webinar/register/WN_Hbp2Xm-mSbSRTgbwRMqtPA

--------------------

*14 May* - Ask The Zeeksperts - Suricata - Jason Ish, Suricata Senior
Developer and Peter Manev, Lead QA for Suricata - 3:30-4:30pm Eastern

Registration:
https://corelight.zoom.us/webinar/register/WN_KN8qo9ZDTfKL1nKl1inmQA

--------------------

*15 May* - Community CTF - 4-6pm Eastern  - Players will compete
head-to-head on dozens of security challenges using Zeek data in both
Splunk and Elastic. Players can also use open-source Zeek tools on a CLI.

Registration:
https://www.eventbrite.com/e/zeek-community-ctf-capture-the-flag-tickets-104776368940

--------------------

*20 May* - Zeek From Home - Suricata - Victor Julien, OISF Founder and
Suricata's Lead Developer and Josh Stroschein, Ph.D., Director of Training
and Academic Initiatives - 2-3pm Eastern

Registration: https://corelight.zoom.us/webinar/register/WN_9haXhmcKR7aSEhKyzT9ICA


--------------------

*27 May* - Zeek From Home - Security Onion  - Doug Burks - 2-3pm Eastern

Registration: https://corelight.zoom.us/webinar/register/WN_5t5TdekCQYSkYp_b2K5Ngw


--------------------

*28 May* - Ask The Zeeksperts - Brim Security - Phil Rzewski - 3:30 -
4:30pm Eastern

Registration:
https://corelight.zoom.us/webinar/register/WN_lXJb4F5WTRSQ1BQasln9HA
<https://corelight.zoom.us/webinar/register/WN_lXJb4F5WTRSQ1BQasln9HA>

We hope to see you online.

If you or your organization are hosting any Zeek related events in May,
please let me know.

Please let me know if you have any questions.

Thanks,
~Amber
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200508/cf1db195/attachment.html 

From akgraner at corelight.com  Mon May 11 13:03:14 2020
From: akgraner at corelight.com (Amber Graner)
Date: Mon, 11 May 2020 16:03:14 -0400
Subject: [Zeek] =?utf-8?q?Zeek_Monthly_Newsletter_=E2=80=93_Issue_4_?=
	=?utf-8?b?4oCTIE1heSAyMDIw?=
Message-ID: <CAJhOzur3aM7Vzojyn1Rzdk4bRZaeBUR-Le+c3WKmVMDzpo0YbA@mail.gmail.com>

Below is Issue 4 of the Zeek Monthly Newsletter. You can also find it at:
https://zeek.org/2020/05/11/zeek-monthly-newsletter-issue-4-may-2020/

==Issue 4 - May 2020==

Welcome to the Zeek Monthly Newsletter, Issue 4 covers April  2020 as well
as upcoming events.

===In this Issue:===

* General Community News/Updates
* Development Updates
* Zeek in the News
* Zeek In, Near and Around then Community
* Interviews/Blog Posts
* Threat of the Month
* Upcoming Events
* New Zeek Related Packages
* Publication Schedule
* Get Involved

===General Community News/Updates===

* The Zeek Package Contest Is Still OPEN - ZPC-2 - The ZPC contest series
is intended to inspire Zeek users to demonstrate their creativity and
ingenuity while winning the admiration of their peers, and giving back to
the community. The ZPC-2 contest will focus on the MITRE ATT&CK? Framework,
more specifically packages that help detect C2 Techniques. Find out more
about how you can participate in ZPC-2 at:
https://zeek.org/2020/04/06/zeek-package-contest-zpc-2/

* Check out the Virtual Events this month!! - We have a full line up of
events in May. Presentations for Zeek From Home include Looking Deeper into
the Zeek 3.0 - Major Changes, Point Releases and more; Suricate and
Security Onion.  Ask the Zeeksperts will be hosted by Suricate and Brim and
new for this month is a virtual Zeek community CTF (Capture the Flag)
event.  You can find out more about how to register for these events below
in the events section.

===Development Updates===

* Zeek 3.0.4 and 3.1.2 release (security + bug fixes) - These releases fix
several bugs, including one potential security issue due to a stack
overflow in the POP3 analyzer (thanks to Matteo Rizzo for the report). -
http://mailman.icsi.berkeley.edu/pipermail/zeek/2020-April/015262.html

* The New IO Loop in Zeek 3.1 - This blog post describes the new
architecture for the IO loop and changes made to IO sources to support the
new architecture. - https://zeek.org/2020/04/03/the-new-io-loop-in-zeek-3-1/

* Issue Tracker:  If you would like to see the issues currently being
tracked, help resolve a few or file an issue you can do so at: :
https://github.com/zeek/zeek/issues

===Zeek In, Near and Around The Community===

* Zeek 3.0.5 now available for Security Onion! - More details,
documentation and release notes can be found at:
https://blog.securityonion.net/2020/04/zeek-305-now-available-for-security.html

* Brim?s Open Source Desktop application which was first announced in
March, but still being seen in Twitter feeds and mailing lists around the
community.  You can find out more about it at:
https://github.com/brimsec/brim

* New Research: Open Source Tools! -  By Augusto Barros  -  In this Gartner
blog post, author Augusto?s Barros is looking for some input on some
research that he is doing. ?The intent is to look at the most popular open
source tools used by security operations teams out there. Things like the
ELK stack, Osquery, MISP and Zeek.?  If you?d like to learn more what he?s
looking for or event lend a hand, check out:
https://blogs.gartner.com/augusto-barros/2020/04/17/new-research-open-source-tools/

* Four Key Elements for Comprehensive Network Threat Detection - This
article by Bricata looks at the following key elements for a better
understanding of network threat detection: Deep Packet Inspection
(Signature-Based) Detection, Behavioral Anomaly-Based (Stateful) Detection,
File Hashing and Detection, Artificial Intelligence and Machine Learning
Detection and more.
https://securityboulevard.com/2020/04/four-key-elements-for-comprehensive-network-threat-detection/

* COVID-19 CTI LEAGUE and CRITICAL PATH SECURITY Intel feed - CTI League
and Critical Path Security has shared an updated COVID-19 threat feed for
Zeek.  It includes COVID-19 CTI public data, Critical Path Security data
collection from dns.log, as well as data from PREDICT. Find out more at:
https://github.com/CriticalPathSecurity/COVID-THREAT-INTEL-PUBLIC-ZEEK/blob/master/README.md

===Interviews/Blog Posts===

* Zeek From Home ? Episode 1 ? Zeek-Agent ? Recording Now Available -
Zeek-Agent is an endpoint monitoring agent that provides host activity to
Zeek. More information about Zeek-Agent can be found on the Zeek blog and
Github
These webinars are recorded and if  you were unable to attend the
Zeek-Agent Zeek From Home episode we have made the following available:
video, audio only and slides.
Many thanks to all those who participated!! Keep those questions and
feedback coming!!
Find out more at:
https://zeek.org/2020/04/17/zeek-from-home-episode-1-zeek-agent-recording-now-available/

* Writing My First Protocol Analyzer - Anthony Kasza from Corelight walks
you through his experience with writing his first protocol analyzer for
Zeek. - https://zeek.org/2020/04/16/writing-my-first-protocol-analyzer/

* Got Zoom? - This may be helpful for some out there. It's a simple package
that works on Zoom TLS traffic. - https://zeek.org/2020/04/14/got-zoom/

* Zeek Package Contest ? ZPC-2 - Announcing a new Zeek Package Contest
(ZPC-2).  This contest will focus on the MITRE ATT&CK? Framework, more
specifically packages that help detect C2 Techniques. $2500.00 USD to the
first prize winner. (Some restrictions apply) See Blog post for more
details. - https://zeek.org/2020/04/06/zeek-package-contest-zpc-2/

* 2019 Zeek Package Contest Summary & Winners - In case you weren't at
ZeekWeek last year, here's the list of winning submissions and a summary of
each Package contributed to the first Zeek Package Contest (ZPC-1) Many
thanks to all those who made it a success! -
https://zeek.org/2020/04/06/2019-zeek-package-contest-summary-and-winners-zpc-1/


===Threat of the Month===

Do you have a threat you?d like to share with the community and how using
Zeek in your security stack helped you identify that threat? Please email
news at zeek.org and we?ll work with you to get it written up and shared in
the next newsletter.

===Upcoming Events===

The following is a list of Zeek Related online/virtual events for May 2020.

====Ask the Zeeksperts====

Ask the Zeeksperts  is a one hour bi-weekly call that is hosted by various
?Zeeksperts? in the community.  This is where you can drop by and ask your
Zeek Related questions.  The webinars are free to attend, but registration
is required.

* 14 May  2020  - 12:30pm PST/3:30pm EST - Suricata - Jason Ish, Suricata
Senior Developer and Peter Manev, Lead QA for Suricata - Bring those
Suricate related questions and ask the experts!
Registration:
https://corelight.zoom.us/webinar/register/WN_KN8qo9ZDTfKL1nKl1inmQA

* 28 May  2020 - 12:30pm PST/3:30pm EST - Brim Security - Phil Rzewski -
3:30 - Brim experts will be on hand to answer all your questions about
their latest open source desktop application release.
Registration:
https://corelight.zoom.us/webinar/register/WN_lXJb4F5WTRSQ1BQasln9HA

====Zeek From Home====

This is a new weekly webinar series, where the community can share their
Zeek Related presentations (scripts, use cases, how to?s, unique usages,
lessons learned etc).  These will be recorded.

* 12 May 2020 - 2pm EST/11am PST - Looking Deeper into the Zeek 3.0 - Major
Changes, Point Releases and more with Tim Wojtulewicz.  If you have
questions about the Zeek 3.0 release then this is the presentation for you.

Registration:
https://corelight.zoom.us/webinar/register/WN_Hbp2Xm-mSbSRTgbwRMqtPA

* 20  May 2020 - 2pm EST/11am PST - Suricata - Victor Julien, OISF Founder
and Suricata's Lead Developer and Josh Stroschein, Ph.D., Director of
Training and Academic Initiatives
Registration:
https://corelight.zoom.us/webinar/register/WN_9haXhmcKR7aSEhKyzT9ICA

* 27 May 2020 - 2pm EST/11am PST - Security Onion  - Doug Burks
Registration:
https://corelight.zoom.us/webinar/register/WN_5t5TdekCQYSkYp_b2K5Ngw


====Capture the Flag Events====

These events are free but registration is required. See links below for
more information.

* 15 May 2020 4-6pm Eastern - Zeek Community CTF (Capture the Flag) -
Players will compete head-to-head on dozens of security challenges using
Zeek data in both Splunk and Elastic. Players can also use open-source Zeek
tools on a CLI.
Registration:
https://www.eventbrite.com/e/zeek-community-ctf-capture-the-flag-tickets-10477636894

* Corelight Virtual Hunt from Home (Every Tuesday and Thursday)  - A free,
2-hour Virtual Capture the Flag event hosted by Corelight, where players
compete to answer security challenges using Zeek data in Splunk and
Elastic. The security challenges model realistic IR and hunting queries and
can help you uplevel your Zeek log proficiency. Corelight experts will be
on hand during the game to guide players of all skill levels through two
exciting hunt scenarios. Sign up for one of eight virtual CTF spots in May.
Game winners will take home bragging rights and a $100 Amazon Gift Card.
https://www3.corelight.com/ctf/hunt-from-home

If you know of any Zeek related events that you would like to share with
the community in the monthly newsletter, please email news at zeek.org or
share on the Zeek mailing list (zeek at zeek.org).

====Zeek Related Packages/New Packages Added to packages.zeek.org====

* SPL-SPT - Sequence of Payload Lengths/Sequence of Payload Times -
https://packages.zeek.org/packages/view/6b874e00-7ece-11ea-9321-0a645a3f3086

* Got Zoom ? -
https://packages.zeek.org/packages/view/bb1d635f-8060-11ea-9321-0a645a3f3086


====Publication Schedule (Updated)====

Issue 1 - January 2020 (Covers December 2019) - 14 January 2020 -
https://zeek.org/2020/01/14/zeek-monthly-newsletter-issue-1-january-2020/
Issue 2 - March 2020 (Covers January and February 2020) - 2 March 2020 -
https://zeek.org/2020/03/02/zeek-monthly-newsletter-issue-2-march-2020/
Issue 3 - April 2020 (Covers March 2020) - 7 April 2020 -
https://zeek.org/2020/04/07/zeek-monthly-newsletter-issue-3-april-2020/
Issue 4 - May 2020 (Covers April 2020) - 8 May 2020 -
https://zeek.org/2020/05/11/zeek-monthly-newsletter-issue-4-may-2020/
Issue 5 - June 2020 (Covers May 2020) - 1 June 2020
Issue 6 - July 2020 (Covers June 2020) - 6 July 2020
Issue 7 - August 2020 (Covers July 2020) - 3 August 2020
Issue 8 - September 2020 (Covers August 2020) - 7 September 2020
Issue 9 - October 2020 (Covers September 2020) - 5 October 2020
Issue 10 - November 2020 (Covers October 2020) - 2 November 2020
Issue 11 - December 2020 (Covers November 2020)  - 7 December 2020
Issue 12 - Special Issue - (Year End Review) - 21 December 2020


====Get Involved====

If you are interested in getting involved with the Zeek Newsletter, please
email news at zeek.org.

Join the News Slack Channel at:
https://join.slack.com/t/zeekorg/shared_invite/enQtOTc3MzMxNDI1NDYxLTA1NzhhMTgxNWI1OTk2NjlkMTdjNzY1Nzk5NDk2ZDY1MDBkYWIxOWNjNDE2NDc2MGI5OWM3ZDllYzBmZmNhNDM

Follow us on Twitter at: https://twitter.com/Zeekurity
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200511/6115d70f/attachment-0001.html 

From akgraner at corelight.com  Tue May 12 11:13:53 2020
From: akgraner at corelight.com (Amber Graner)
Date: Tue, 12 May 2020 14:13:53 -0400
Subject: [Zeek] Zeek Package Contest - ZPC-2 - CLOSES SOON- Don't miss out
 on your chance to win $$$
Message-ID: <CAJhOzuqcA6v6gvn79Lo1wSq1wbAY_iU09TC0vMu4HBqKH3enmw@mail.gmail.com>

Hi all,

The Zeek Package Contest ends this Friday (15 May)  and as of now we have
more prizes than we have submissions.  You've still got time to submit your
C2 technique detection Packages!.

$$$ Prizes $$$ include:

 - 1st place wins $2000.00 USD cash
 - 2nd place wins $1000.00 USD cash
 - 3rd place wins $500.00 USD cash

All submitters will receive a Zeek Package Contest Challenge Coin.
* some restrictions apply - see blog post for more details

There's still time to submit!!!  Don't wait! If you need help getting your
script into Github reach out and we'll get someone to help you.

You can reply to this thread or ping someone in the #packages channel on
the Zeek Slack workspace.

Find out more information at:
https://zeek.org/2020/04/06/zeek-package-contest-zpc-2/

Thanks,
~Amber
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200512/a7853fa8/attachment.html 

From Kayode_Enwerem at ao.uscourts.gov  Tue May 12 15:16:36 2020
From: Kayode_Enwerem at ao.uscourts.gov (Kayode Enwerem)
Date: Tue, 12 May 2020 22:16:36 +0000
Subject: [Zeek] issue with file extraction and splunk
Message-ID: <BY5PR09MB5377EE076EF5621D65C7254BA1BE0@BY5PR09MB5377.namprd09.prod.outlook.com>

Hello,

We installed the file extraction package; "zeek/hosom/file-extraction" on our bro server.

We noticed that whenever we load the file extraction package some of the larger logs like conn.log and dns.log stop showing up in splunk (the smaller logs continue to show up in splunk).  Once we unload the file extraction package and restart zeek, those logs start flowing into splunk as normal.

We have 128 CPUs on that server. Pinned a total of 90 workers (45 - worker1 and 45 to worker 2).

Is anyone else experiencing any similar issue? Any ideas or thought?

Zeek version we are running is 3.0.3.

Thanks in advance.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200512/7bed679d/attachment.html 

From akgraner at corelight.com  Wed May 13 08:34:26 2020
From: akgraner at corelight.com (Amber Graner)
Date: Wed, 13 May 2020 11:34:26 -0400
Subject: [Zeek] Calendar reminders for Zeek Events
Message-ID: <CAJhOzuqw5+onuoaQQdfuwiSHhG+QJKyVUP1FvBpmrmgLNg2cPQ@mail.gmail.com>

For those who opt'd in to be added to Calendar reminders for Zeek events,
I've just added you all to the May events.

If you received an invite and would rather not get these or be removed from
one type of event but not another in the future please let me know.

Events are:

Monthly - Community Call
Monthly - Community CTF (Capture the Flag)
Bi-weekly - Ask the Zeeksperts
Weekly - Ask The Zeeksperts

If you'd like to receive Calendar reminders and currently don't please let
me know and I can get you added.

Thanks,
~Amber
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200513/666e697c/attachment.html 

From akgraner at corelight.com  Thu May 14 03:01:01 2020
From: akgraner at corelight.com (Amber Graner)
Date: Thu, 14 May 2020 06:01:01 -0400
Subject: [Zeek] Addition of Jobs Channel on Slack
Message-ID: <CAJhOzurs=ar4WMVXdLu6BXHZtHqw8ZDvT3z2hv8MVxXEKTwHeg@mail.gmail.com>

Hi all,

We've added a new #jobs channel to our Zeek Slack workspace.  If you or
someone you know has a Zeek related job opportunity please feel free to add
it to that channel.

Not on Slack?  Feel free to send me the link or if you'd like to join the
workspace below is the invitation link.

https://join.slack.com/t/zeekorg/shared_invite/enQtOTc3MzMxNDI1NDYxLTA1NzhhMTgxNWI1OTk2NjlkMTdjNzY1Nzk5NDk2ZDY1MDBkYWIxOWNjNDE2NDc2MGI5OWM3ZDllYzBmZmNhNDM

++++++++++++++++++++++++++++++
+ Other Zeek Slack Channels include: +
++++++++++++++++++++++++++++++

#development - Anything about working on the Zeek code base

#general - Workspace-wide communication and announcements

#jobs - Zeek related job postings

#news - Sharing zeek related news for possible inclusion on Zeek Blog or
Newsletter

#packages - Discuss Zeek packages and the Zeek package manager

#random - A place for non-project-related flimflam, faffing, hodge-podge or
jibber-jabber you'd prefer to keep out of more focused project-related
channels.

#siem -  Sharing and collaboration of Zeek queries for usage in SIEMs and
logging systems (Humio, ELK, Splunk)

#spicy - Spicy discussion (https://github.com/zeek/spicy)

#webinars - Discussion and questions about or for any of the Zeek Webinars.
(https://zeek.org/events/)

#zeek-agent - Zeek-agent discussion (https://github.com/zeek/zeek-agent)

Please let me know if you have any questions.

Thanks,
~Amber
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200514/ed75f890/attachment.html 

From greg.grasmehr at caltech.edu  Thu May 14 10:12:18 2020
From: greg.grasmehr at caltech.edu (Greg Grasmehr)
Date: Thu, 14 May 2020 10:12:18 -0700
Subject: [Zeek] Roll over logs at specific time
Message-ID: <20200514171218.GD3043@dakine>

Hello,

Is there a method to roll certain Zeek logs at a particular time instead
of a count of seconds from 0000?  I was hoping setting a log to roll
after 86404 seconds, and then restarting Zeek at the time I wanted the
log to roll, would roll the log then and then persist as the time to
roll, but it also rolled at midnight, which makes sense of course for
basic syslogging.

Thanks in advance for any advice.

-- 
Sincerely,

Greg Grasmehr
Lead Information Security Analyst
California Institute of Technology (Caltech)
GPGMe: 38E2 F9BD A95E 9824 20AB  331A 9E29 D1A1 AAEE 5F42
http://keys.gnupg.net/pks/lookup?search=0x9E29D1A1AAEE5F42

From raubvogel at gmail.com  Thu May 14 10:31:08 2020
From: raubvogel at gmail.com (Mauricio Tavares)
Date: Thu, 14 May 2020 13:31:08 -0400
Subject: [Zeek] Roll over logs at specific time
In-Reply-To: <20200514171218.GD3043@dakine>
References: <20200514171218.GD3043@dakine>
Message-ID: <CAHEKYV7VadehbMmjW8ONNBWKvEKN46KqVtT2N4taoku7ZkzSTg@mail.gmail.com>

On Thu, May 14, 2020 at 1:21 PM Greg Grasmehr <greg.grasmehr at caltech.edu> wrote:
>
> Hello,
>
> Is there a method to roll certain Zeek logs at a particular time instead
> of a count of seconds from 0000?  I was hoping setting a log to roll
> after 86404 seconds, and then restarting Zeek at the time I wanted the
> log to roll, would roll the log then and then persist as the time to
> roll, but it also rolled at midnight, which makes sense of course for
> basic syslogging.
>
      I think the main issue (you need to decide how much of an issue
it is) is that the informational emails depend on the same cronjob
(for a lack of a better term) that rotates the logs. In my case I did
not care much about that, hoping warning emails will be sent
regardless, and I delegated log rotating to the system (rsyslog +
logrotate in my case).

> Thanks in advance for any advice.
>
> --
> Sincerely,
>
> Greg Grasmehr
> Lead Information Security Analyst
> California Institute of Technology (Caltech)
> GPGMe: 38E2 F9BD A95E 9824 20AB  331A 9E29 D1A1 AAEE 5F42
> http://keys.gnupg.net/pks/lookup?search=0x9E29D1A1AAEE5F42
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

From akgraner at corelight.com  Fri May 15 11:00:01 2020
From: akgraner at corelight.com (Amber Graner)
Date: Fri, 15 May 2020 14:00:01 -0400
Subject: [Zeek] =?utf-8?q?Zeek_From_Home_=E2=80=93_Episode_2-_Looking_Deep?=
	=?utf-8?q?er_into_the_Zeek_3=2E0_-_Recording_now_available!?=
Message-ID: <CAJhOzupgwxJa9R8EY__TiSN79icJLYjj1eCfkZ8-LNPGKqmC3w@mail.gmail.com>

Hi all,

Happy Friday!!

The recording is now available  for Zeek From Home ? Episode 2- Looking
Deeper into the Zeek 3.0 ? Major Changes, Point Releases and more.

Below are the links:

* Video -
https://www.dropbox.com/s/vlutaz92i847yn0/13_May_Zeek_From_Home_Zeek_3_0_Video.mp4?dl=0
* Audio only -
https://www.dropbox.com/s/zoryrqvfzk4ydfl/13_May_Zeek_From_Home_Zeek_3_0_Audio.m4a?dl=0
* Slides -
https://www.dropbox.com/s/zdtpm0q2cmxqr6i/Zeek%20From%20Home%20-%20Looking%20Deeper%20into%20the%20Zeek%203.0.pdf?dl=0

The full blog post is available at:
https://zeek.org/2020/05/15/zeek-from-home-episode-2-looking-deeper-into-the-zeek-3-0-major-changes-point-releases-and-more-recording-now-available/

Upcoming online events can be found at: https://zeek.org/events/

If you would like to be included in calendar reminders for these events
please let me know.

Thanks and have a great weekend!

~Amber
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200515/81ec97f3/attachment.html 

From ttomek.koziak at gmail.com  Sat May 16 11:55:13 2020
From: ttomek.koziak at gmail.com (Tomek Koziak)
Date: Sat, 16 May 2020 20:55:13 +0200
Subject: [Zeek] Sorting a vector of intervals.
Message-ID: <CAF=fF7qrcxp3kQ1oU=nOmGZut5DBR514L1cES30dRRkgoV1pVg@mail.gmail.com>

Hi All,
When I try to run the following code

event zeek_init()
{
local v: vector of interval = vector(1min,2min,1min,4min,6min,1min);
local vo : vector of interval = sort(v);
    for (i in vo){
    print vo[i];
    }
}

I get the expected output by as well I obtain the following error:
line 4: comparison function required for sort() with non-integral types
(sort(v)) fatal error: errors occurred while initializing

I have followed the documentation.
https://docs.zeek.org/en/current/scripts/base/bif/zeek.bif.zeek.html#sort
But I quite don't understand the part about the comparison function as a
second argument. What parameter should be placed as a second argument?
<https://docs.zeek.org/en/current/scripts/base/bif/zeek.bif.zeek.html#sort>

Tomasz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200516/7fa6edb2/attachment.html 

From anthony.kasza at gmail.com  Sat May 16 13:48:31 2020
From: anthony.kasza at gmail.com (anthony kasza)
Date: Sat, 16 May 2020 14:48:31 -0600
Subject: [Zeek] Sorting a vector of intervals.
In-Reply-To: <CAF=fF7qrcxp3kQ1oU=nOmGZut5DBR514L1cES30dRRkgoV1pVg@mail.gmail.com>
References: <CAF=fF7qrcxp3kQ1oU=nOmGZut5DBR514L1cES30dRRkgoV1pVg@mail.gmail.com>
Message-ID: <CAEZw2bybg9hfZtnX2uBZS0UaSoOVfcTmqpdAsJ8D4eXZCYdG8A@mail.gmail.com>

The sort function's second argument must be a function. Below, I use an
anonymous function but you could also define and declare it. Think of this
second function as a map function which is applied to the vector you're
sorting.
Sorting is done in place, too. So you may want to add a copy() function to
your definition of vo.


event zeek_init() {
  local v: vector of interval = vector(1min,2min,1min,4min,6min,1min);
  local vo : vector of interval = sort(copy(v), function(a: interval, b:
interval): int {return a > b ? 1 : -1;} );
  for (i in vo) {
    print vo[i];
  }
}

-AK


On Sat, May 16, 2020, 13:03 Tomek Koziak <ttomek.koziak at gmail.com> wrote:

> Hi All,
> When I try to run the following code
>
> event zeek_init()
> {
> local v: vector of interval = vector(1min,2min,1min,4min,6min,1min);
> local vo : vector of interval = sort(v);
>     for (i in vo){
>     print vo[i];
>     }
> }
>
> I get the expected output by as well I obtain the following error:
> line 4: comparison function required for sort() with non-integral types
> (sort(v)) fatal error: errors occurred while initializing
>
> I have followed the documentation.
> https://docs.zeek.org/en/current/scripts/base/bif/zeek.bif.zeek.html#sort
> But I quite don't understand the part about the comparison function as a
> second argument. What parameter should be placed as a second argument?
> <https://docs.zeek.org/en/current/scripts/base/bif/zeek.bif.zeek.html#sort>
>
> Tomasz
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200516/7df752ee/attachment.html 

From akgraner at corelight.com  Mon May 18 07:34:44 2020
From: akgraner at corelight.com (Amber Graner)
Date: Mon, 18 May 2020 10:34:44 -0400
Subject: [Zeek] Announcing the (New) Spicy Parser Generator
Message-ID: <CAJhOzuradryT8tv5Z9RB3mFdWzUGJB9dQ39itvJ6EDw8NGjQdw@mail.gmail.com>

Hi all,

We just published the Spicy Parser Generator announcement.  Thanks, Robin
for posting this.

If you'd like to learn more please go to:
https://zeek.org/2020/05/18/announcing-the-new-spicy-parser-generator/

Thanks,
~Amber
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200518/ea39252d/attachment.html 

From Kayode_Enwerem at ao.uscourts.gov  Mon May 18 13:36:10 2020
From: Kayode_Enwerem at ao.uscourts.gov (Kayode Enwerem)
Date: Mon, 18 May 2020 20:36:10 +0000
Subject: [Zeek] Logs not rotating
Message-ID: <BY5PR09MB537786EBF4BB5C1A0CDF25AAA1B80@BY5PR09MB5377.namprd09.prod.outlook.com>

Hello,

We are experiencing some issues with log rotation. The .log files in current directory are not getting rotated and compressed causing the .log files to grow really big. It starts working again after we restart zeek but after a few hours it stops again.

We are using pigz to zip the files and rotate interval is set to 3600(1hr).
logrotationinterval = 3600
compresscmd = pigz

Is anyone else experiencing any similar issue? Any ideas or thought?

Zeek version we are running is 3.0.3.

Thanks in advance.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200518/54bd9025/attachment.html 

From raubvogel at gmail.com  Tue May 19 05:09:06 2020
From: raubvogel at gmail.com (Mauricio Tavares)
Date: Tue, 19 May 2020 08:09:06 -0400
Subject: [Zeek] Testing modules/policies
Message-ID: <CAHEKYV7atXpQ5OXGwe7J-Ss3CvJSr2__NmeoYSuUcFJRYBWWwg@mail.gmail.com>

1. I understand that if a module is manually tested (by calling it and
feeding a pcap), its log entries will be saved on the same directory
said module was called from. But, what if the module is also supposed
to mail out? Would it be able to do the deed?

2. Is there a verbose option (I am thinking on the -v[v[v]] in
ansible/ssh) when you call

zeek -r pcap policy

I do not mean the -d option, as it seems to behave like gdb.

From vlad at es.net  Tue May 19 06:28:17 2020
From: vlad at es.net (Vlad Grigorescu)
Date: Tue, 19 May 2020 08:28:17 -0500
Subject: [Zeek] Testing modules/policies
In-Reply-To: <CAHEKYV7atXpQ5OXGwe7J-Ss3CvJSr2__NmeoYSuUcFJRYBWWwg@mail.gmail.com>
References: <CAHEKYV7atXpQ5OXGwe7J-Ss3CvJSr2__NmeoYSuUcFJRYBWWwg@mail.gmail.com>
Message-ID: <CAPqbkwtvF3uoUZwWT7RTh8f=qMOQ5kT_tyTFqEbm5iQtWMjMFQ@mail.gmail.com>

On Tue, May 19, 2020 at 7:10 AM Mauricio Tavares <raubvogel at gmail.com>
wrote:

> 2. Is there a verbose option (I am thinking on the -v[v[v]] in
> ansible/ssh) when you call
>
> zeek -r pcap policy
>
> I do not mean the -d option, as it seems to behave like gdb.
>

 One option I use is `zeek -r pcap misc/dump-events my-test-policy.zeek`.
See:
https://docs.zeek.org/en/current/scripts/policy/misc/dump-events.zeek.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200519/f48d627a/attachment-0001.html 

From shirkdog.bsd at gmail.com  Wed May 20 09:01:46 2020
From: shirkdog.bsd at gmail.com (Michael Shirk)
Date: Wed, 20 May 2020 12:01:46 -0400
Subject: [Zeek] Zeek Vs. FreeBSD
In-Reply-To: <CAGCUUMHQizDufq+6djMvXy-O1WuxC81fxn1qwd3JV-TG-nEX7Q@mail.gmail.com>
References: <CAGCUUMFpjv5oz-GsQHX-j3fQYVJ8nst8pMtV__62LLpXdvFygQ@mail.gmail.com>
	<CAL8PkUUpgnH3z_cEV5EFRvbpuztfP1LTeVFSZ5SW-3iyCDi=cQ@mail.gmail.com>
	<CAGCUUMHQizDufq+6djMvXy-O1WuxC81fxn1qwd3JV-TG-nEX7Q@mail.gmail.com>
Message-ID: <CAL8PkUUTQ0bUeF1Jv45gtuxn1D7fPrWQD1QKQE15i6CT=g-yCA@mail.gmail.com>

On Fri, May 8, 2020 at 11:33 AM Anthony Arnaud
<antho.arnaudisce at gmail.com> wrote:
>
> Hi Michael,
> I'm using FreeBSD 12.1 and Zeek 3.0.5 (ver. 3.0.6 available in ports has compilation problems), with 2 NICs without ip, em0 and vtnet1 (em0 is Intel e1000) but the problem is that zeek plugin is not updated. (API mismatch)
>
> #ifconfig
> vtnet1: flags = 8802 <BROADCAST, SIMPLEX, MULTICAST> metric 0 mtu 1500
>         options = c00b8 <VLAN_MTU, VLAN_HWTAGGING, JUMBO_MTU, VLAN_HWCSUM, VLAN_HWTSO, link-state routing protocol>
>         ether something
>         media: Ethernet 10Gbase-T <full-duplex>
>         status: active
>         nd6 options = 29 <PERFORMNUD, IFDISABLED, AUTO_LINKLOCAL>
>
> em0: flags = 8802 <BROADCAST, SIMPLEX, MULTICAST> metric 0 mtu 1500
>         options = 812,098 <VLAN_MTU, VLAN_HWTAGGING, VLAN_HWCSUM, WOL_MAGIC, VLAN_HWFILTER>
>         ether something
>         media: Ethernet autoselect (1000baseT <full-duplex>)
>         status: active
>         nd6 options = 29 <PERFORMNUD, IFDISABLED, AUTO_LINKLOCAL>
>
> # on zeekctl start
> starting workers ...
> Error: worker-1-1 terminated immediately after starting; check output with "diag"
> Error: worker-1-2 terminated immediately after starting; check output with "diag"
>
> #in dmesg
> 173.973686 [376] netmap_ioctl_legacy Minimum supported API is 14 (requested 11)
> 173.973712 [376] netmap_ioctl_legacy Minimum supported API is 14 (requested 11)
>
> #from zeekctl diag
> Zeek 3.0.5
> FreeBSD 12.1-RELEASE
>
> Zeek plugins:
> Bro :: Netmap - Packet acquisition via Netmap (dynamic, version 1.0.0)
>
> ==== stderr.log
> 292.768100 nm_open [920] NIOCREGIF failed: Invalid argument vtnet1} 1
> fatal error: problem with interface netmap :: vtnet1} 1 (Invalid argument)
>
> The netmap tools in kernel sources seems ok, lb start and the network interface switch in netmap mode.
> I think the latest working version of  plugin is compatible with netmap release available in FreeBSD 11.2, but there are performance issues with vtnets.
> Also tcpreplay doesn't work when i try to send traffic in netmap mode to a NIC sniffed by zeek (in FBSD 11.2)
>
> Thanks,
> Anthon
>
> Il giorno gio 7 mag 2020 alle ore 20:30 Michael Shirk <shirkdog.bsd at gmail.com> ha scritto:
>>
>> Some questions to get started:
>> Which version of FreeBSD are you using?
>> Which network card are you using?
>>
>> The biggest issue is parity between the netmap and FreeBSD source
>> trees, you really need to run FreeBSD-CURRENT to ensure you have all
>> of the latest changes. There were some issues in the past that
>> affected even Intel network cards from working correctly, so the types
>> of cards you are using are very important.
>>
>> I pushed to get the netmap tools added to the source tree, so you can
>> build "lb" from the following location and use it:
>> /usr/src/tools/tools/netmap/lb.c
>>
>> I am updating a FreeBSD system to see if this still builds correctly
>> as I have not used LB in a while.
>>
>> On Thu, May 7, 2020 at 12:32 PM Anthony Arnaud
>> <antho.arnaudisce at gmail.com> wrote:
>> >
>> > Hi All,
>> > I tried to install Zeek on my FreeBSD server with netmap support.
>> > But VirtIO Ethernet driver is not working properly, there are performance problems that should be solved in the latest Netmap release, ref to:
>> >
>> > https://reviews.freebsd.org/D17916
>> >
>> > Unfortunately the bro-netmap plugin does not work with that.
>> > It seems that Zeek is unusable in FreeBSD env, the developments of the bro-netmap plugin are closed and it is impossible to parallelize network traffic on multiple zeek workers.
>> > Does anyone know if updates are currently planned?
>> > Or if someone using this plugin with the Netmap last version?
>> > Or, finally, are there other BSD loadbalancing solutions ?
>> > Thank y'all
>> >
>> > Anthon
>> > _______________________________________________
>> > Zeek mailing list
>> > zeek at zeek.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>>
>>
>> --
>> Michael Shirk
>> Daemon Security, Inc.
>> https://www.daemon-security.com

I had a user error on my part, the following if run from a FreeBSD
CURRENT system with the kernel source will build the tools and you can
use them for packet brokering:

cd /usr/src/tools/tools/netmap
make all

The binaries will be located here:
/usr/obj/usr/src/amd64.amd64/tools/tools/netmap

then you can run LB to setup the packet brokering, in this case just
two pipes setup on em0.
./lb -i em0 -p ids:2 -o 1
143.318741 main [588] interface is em0
143.529856 main [702] successfully opened netmap:em0 (tx rings: 1024)
143.529865 main [774] opening pipe named netmap:ids{0/xT at 1
143.530027 nm_mmap [990] do not mmap, inherit from parent
143.530037 main [789] successfully opened pipe #1 netmap:ids{0/xT at 1
(tx slots: 1024)
143.530039 main [794] zerocopy enabled
143.530042 main [774] opening pipe named netmap:ids{1/xT at 1
143.530131 nm_mmap [990] do not mmap, inherit from parent
143.530137 main [789] successfully opened pipe #2 netmap:ids{1/xT at 1
(tx slots: 1024)
143.530140 main [794] zerocopy enabled

Now to see about the netmap plugin working.
-- 
Michael Shirk
Daemon Security, Inc.
https://www.daemon-security.com

From justin at corelight.com  Wed May 20 09:08:07 2020
From: justin at corelight.com (Justin Azoff)
Date: Wed, 20 May 2020 12:08:07 -0400
Subject: [Zeek] Zeek Vs. FreeBSD
In-Reply-To: <CAL8PkUUTQ0bUeF1Jv45gtuxn1D7fPrWQD1QKQE15i6CT=g-yCA@mail.gmail.com>
References: <CAGCUUMFpjv5oz-GsQHX-j3fQYVJ8nst8pMtV__62LLpXdvFygQ@mail.gmail.com>
	<CAL8PkUUpgnH3z_cEV5EFRvbpuztfP1LTeVFSZ5SW-3iyCDi=cQ@mail.gmail.com>
	<CAGCUUMHQizDufq+6djMvXy-O1WuxC81fxn1qwd3JV-TG-nEX7Q@mail.gmail.com>
	<CAL8PkUUTQ0bUeF1Jv45gtuxn1D7fPrWQD1QKQE15i6CT=g-yCA@mail.gmail.com>
Message-ID: <CAPfnCuiV9+VB8oUM2AuAhBJx32NmFDetw=GHqw85w7ahaxwDGw@mail.gmail.com>

> Now to see about the netmap plugin working.

Someone recently made a copy of it with updates to work on zeek:
https://github.com/WqyJh/zeek-netmap

It's hard to tell what changed since it doesn't have the old repo as a
starting point, but it doesn't look like much more than the bro ->
zeek bits.

-- 
Justin

From don.thomas.cissp at gmail.com  Wed May 20 18:04:45 2020
From: don.thomas.cissp at gmail.com (Don Thomas)
Date: Wed, 20 May 2020 18:04:45 -0700
Subject: [Zeek] Zeek From Home - Suricata - With hosts - Victor Julien, OISF
Message-ID: <CABYfsedrJ-ALW31QdkEzdJAFwPC+Uqy+HTez=Py_goHosgk0mQ@mail.gmail.com>

Where can I find the slides from the Suricata presentation today?


Thank you,

*Don Thomas, CISSP, CISA*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200520/860e4fb9/attachment.html 

From akgraner at corelight.com  Wed May 20 18:39:52 2020
From: akgraner at corelight.com (Amber Graner)
Date: Wed, 20 May 2020 21:39:52 -0400
Subject: [Zeek] Zeek From Home - Suricata - With hosts - Victor Julien,
	OISF
In-Reply-To: <CABYfsedrJ-ALW31QdkEzdJAFwPC+Uqy+HTez=Py_goHosgk0mQ@mail.gmail.com>
References: <CABYfsedrJ-ALW31QdkEzdJAFwPC+Uqy+HTez=Py_goHosgk0mQ@mail.gmail.com>
Message-ID: <CAJhOzupCgY4oS=BgzSLfU=PBR0nryhjxP8+728vB_3Lu=hmOoQ@mail.gmail.com>

We?ll put out a blog post and share everything over the next few days.

Thanks,
~Amber

On Wed, May 20, 2020 at 9:06 PM Don Thomas <don.thomas.cissp at gmail.com>
wrote:

>
> Where can I find the slides from the Suricata presentation today?
>
>
> Thank you,
>
> *Don Thomas, CISSP, CISA*
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

-- 

*Amber Graner*
Director of Community
Corelight, Inc

828.582.9469

Schedule time on my calendar here. <https://calendly.com/amber_graner>



 * Ask me about how you can participate in the Zeek (formerly Bro)
community.
 * Remember - ZEEK AND YOU SHALL FIND!!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200520/b737d2a6/attachment.html 

From nabilmemon.ec at gmail.com  Fri May 22 03:16:37 2020
From: nabilmemon.ec at gmail.com (Nabil Memon)
Date: Fri, 22 May 2020 15:46:37 +0530
Subject: [Zeek] Input framework
Message-ID: <CADhcAaerv1k=ocGkHYLLj=uvXA4U=1i3pSLfRNto4Zf7=PKYgQ@mail.gmail.com>

Hi Zeek,

Hope you're all doing well.

I am using an Input framework to provide some dynamic input to bro.
Here's how the infra looks like,

*cat* */usr/local/bro/share/bro/base/protocols/file_port_list*
#fields port_num
60000/tcp
8080/tcp
49154/tcp
55907/tcp
49152/tcp
49153/tcp
8000/tcp
5357/tcp


type Portsx: record {
   port_num: port;
};

global file_port_list: set[port] = set();

redef Communication::nodes += {
     ["python"] = [$host = 127.0.0.1, $events = /*config_update*/,
$connect=F, $ssl=F]
};

event bro_init()
{
   Input::add_table([$source=file_port_list_loc, $name="file_port_list",
$idx=Portsx, $destination=file_port_list]);
}

event bro_done()
{
   Input::remove("file_port_list");
}


event config_update()
{
   Input::force_update("file_port_list");
}

It works really well. But when I terminate bro process, it goes into
defunct state and throws an error on the console.
*Error: received signal while waiting for thread
/usr/local/bro/share/bro/base/protocols/file_port_list/Input::READER_ASCII,
aborting all ...*

Anything am I doing wrong??

Thanks a lot for looking..!!


Regards,
Nabil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200522/022192cc/attachment.html 

From justin at corelight.com  Fri May 22 07:06:00 2020
From: justin at corelight.com (Justin Azoff)
Date: Fri, 22 May 2020 10:06:00 -0400
Subject: [Zeek] Input framework
In-Reply-To: <CADhcAaerv1k=ocGkHYLLj=uvXA4U=1i3pSLfRNto4Zf7=PKYgQ@mail.gmail.com>
References: <CADhcAaerv1k=ocGkHYLLj=uvXA4U=1i3pSLfRNto4Zf7=PKYgQ@mail.gmail.com>
Message-ID: <CAPfnCujTCm-nswHUd89xsuc8srhrGr=veU=z7J1fB1bJp3bFjw@mail.gmail.com>

Try deleting this part:

event bro_done()
{
   Input::remove("file_port_list");
}

On Fri, May 22, 2020 at 6:19 AM Nabil Memon <nabilmemon.ec at gmail.com> wrote:
>
> Hi Zeek,
>
> Hope you're all doing well.
>
> I am using an Input framework to provide some dynamic input to bro.
> Here's how the infra looks like,
>
> cat /usr/local/bro/share/bro/base/protocols/file_port_list
> #fields port_num
> 60000/tcp
> 8080/tcp
> 49154/tcp
> 55907/tcp
> 49152/tcp
> 49153/tcp
> 8000/tcp
> 5357/tcp
>
>
> type Portsx: record {
>    port_num: port;
> };
>
> global file_port_list: set[port] = set();
>
> redef Communication::nodes += {
>      ["python"] = [$host = 127.0.0.1, $events = /config_update/, $connect=F, $ssl=F]
> };
>
> event bro_init()
> {
>    Input::add_table([$source=file_port_list_loc, $name="file_port_list", $idx=Portsx, $destination=file_port_list]);
> }
>
> event bro_done()
> {
>    Input::remove("file_port_list");
> }
>
>
> event config_update()
> {
>    Input::force_update("file_port_list");
> }
>
> It works really well. But when I terminate bro process, it goes into defunct state and throws an error on the console.
> Error: received signal while waiting for thread /usr/local/bro/share/bro/base/protocols/file_port_list/Input::READER_ASCII, aborting all ...
>
> Anything am I doing wrong??
>
> Thanks a lot for looking..!!
>
>
> Regards,
> Nabil
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
Justin

From Kayode_Enwerem at ao.uscourts.gov  Fri May 22 11:25:48 2020
From: Kayode_Enwerem at ao.uscourts.gov (Kayode Enwerem)
Date: Fri, 22 May 2020 18:25:48 +0000
Subject: [Zeek] Logs not rotating
In-Reply-To: <BY5PR09MB537786EBF4BB5C1A0CDF25AAA1B80@BY5PR09MB5377.namprd09.prod.outlook.com>
References: <BY5PR09MB537786EBF4BB5C1A0CDF25AAA1B80@BY5PR09MB5377.namprd09.prod.outlook.com>
Message-ID: <BY5PR09MB5377E08C294B3139A21B19CCA1B40@BY5PR09MB5377.namprd09.prod.outlook.com>

Can anyone please assist with this?

From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> On Behalf Of Kayode Enwerem
Sent: Monday, May 18, 2020 4:36 PM
To: zeek at zeek.org
Subject: [Zeek] Logs not rotating

Hello,

We are experiencing some issues with log rotation. The .log files in current directory are not getting rotated and compressed causing the .log files to grow really big. It starts working again after we restart zeek but after a few hours it stops again.

We are using pigz to zip the files and rotate interval is set to 3600(1hr).
logrotationinterval = 3600
compresscmd = pigz

Is anyone else experiencing any similar issue? Any ideas or thought?

Zeek version we are running is 3.0.3.

Thanks in advance.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200522/fb302263/attachment-0001.html 

From raubvogel at gmail.com  Fri May 22 12:02:53 2020
From: raubvogel at gmail.com (Mauricio Tavares)
Date: Fri, 22 May 2020 15:02:53 -0400
Subject: [Zeek] Logs not rotating
In-Reply-To: <BY5PR09MB5377E08C294B3139A21B19CCA1B40@BY5PR09MB5377.namprd09.prod.outlook.com>
References: <BY5PR09MB537786EBF4BB5C1A0CDF25AAA1B80@BY5PR09MB5377.namprd09.prod.outlook.com>
	<BY5PR09MB5377E08C294B3139A21B19CCA1B40@BY5PR09MB5377.namprd09.prod.outlook.com>
Message-ID: <CAHEKYV752bUYMFkBO6YgGRNW6qmWrinREqecORPcDZRjaON2Pg@mail.gmail.com>

On Fri, May 22, 2020 at 2:27 PM Kayode Enwerem
<Kayode_Enwerem at ao.uscourts.gov> wrote:
>
> Can anyone please assist with this?
>
>
>
> From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> On Behalf Of Kayode Enwerem
> Sent: Monday, May 18, 2020 4:36 PM
> To: zeek at zeek.org
> Subject: [Zeek] Logs not rotating
>
      Is it sending emails out (such as the Connection summary)?
Reason I ask is the log rotation is tied up with the mail sending. If
it is sending emails, or you do not care about that, I can give you a
logrotate config file to buy you time to solve the issue.

>
> Hello,
>
>
>
> We are experiencing some issues with log rotation. The .log files in current directory are not getting rotated and compressed causing the .log files to grow really big. It starts working again after we restart zeek but after a few hours it stops again.
>
>
>
> We are using pigz to zip the files and rotate interval is set to 3600(1hr).
>
> logrotationinterval = 3600
>
> compresscmd = pigz
>
>
>
> Is anyone else experiencing any similar issue? Any ideas or thought?
>
>
>
> Zeek version we are running is 3.0.3.
>
>
>
> Thanks in advance.
>
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

From Kayode_Enwerem at ao.uscourts.gov  Fri May 22 13:00:41 2020
From: Kayode_Enwerem at ao.uscourts.gov (Kayode Enwerem)
Date: Fri, 22 May 2020 20:00:41 +0000
Subject: [Zeek] Logs not rotating
In-Reply-To: <CAHEKYV752bUYMFkBO6YgGRNW6qmWrinREqecORPcDZRjaON2Pg@mail.gmail.com>
References: <BY5PR09MB537786EBF4BB5C1A0CDF25AAA1B80@BY5PR09MB5377.namprd09.prod.outlook.com>
	<BY5PR09MB5377E08C294B3139A21B19CCA1B40@BY5PR09MB5377.namprd09.prod.outlook.com>
	<CAHEKYV752bUYMFkBO6YgGRNW6qmWrinREqecORPcDZRjaON2Pg@mail.gmail.com>
Message-ID: <BY5PR09MB5377E565E0DA6FCF05789FB6A1B40@BY5PR09MB5377.namprd09.prod.outlook.com>

Yes it is sending out connection summary emails.. Please send the config file.

Thanks for the response.

-----Original Message-----
From: Mauricio Tavares <raubvogel at gmail.com> 
Sent: Friday, May 22, 2020 3:03 PM
To: Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov>
Cc: zeek at zeek.org
Subject: Re: [Zeek] Logs not rotating

On Fri, May 22, 2020 at 2:27 PM Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov> wrote:
>
> Can anyone please assist with this?
>
>
>
> From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> On Behalf Of 
> Kayode Enwerem
> Sent: Monday, May 18, 2020 4:36 PM
> To: zeek at zeek.org
> Subject: [Zeek] Logs not rotating
>
      Is it sending emails out (such as the Connection summary)?
Reason I ask is the log rotation is tied up with the mail sending. If it is sending emails, or you do not care about that, I can give you a logrotate config file to buy you time to solve the issue.

>
> Hello,
>
>
>
> We are experiencing some issues with log rotation. The .log files in current directory are not getting rotated and compressed causing the .log files to grow really big. It starts working again after we restart zeek but after a few hours it stops again.
>
>
>
> We are using pigz to zip the files and rotate interval is set to 3600(1hr).
>
> logrotationinterval = 3600
>
> compresscmd = pigz
>
>
>
> Is anyone else experiencing any similar issue? Any ideas or thought?
>
>
>
> Zeek version we are running is 3.0.3.
>
>
>
> Thanks in advance.
>
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From raubvogel at gmail.com  Sun May 24 04:09:27 2020
From: raubvogel at gmail.com (Mauricio Tavares)
Date: Sun, 24 May 2020 07:09:27 -0400
Subject: [Zeek] Logs not rotating
In-Reply-To: <BY5PR09MB5377E565E0DA6FCF05789FB6A1B40@BY5PR09MB5377.namprd09.prod.outlook.com>
References: <BY5PR09MB537786EBF4BB5C1A0CDF25AAA1B80@BY5PR09MB5377.namprd09.prod.outlook.com>
	<BY5PR09MB5377E08C294B3139A21B19CCA1B40@BY5PR09MB5377.namprd09.prod.outlook.com>
	<CAHEKYV752bUYMFkBO6YgGRNW6qmWrinREqecORPcDZRjaON2Pg@mail.gmail.com>
	<BY5PR09MB5377E565E0DA6FCF05789FB6A1B40@BY5PR09MB5377.namprd09.prod.outlook.com>
Message-ID: <CAHEKYV4JFd0+sUZaeeAOZu_s1=GoHvknXs00-n5u7dSZBTmvcQ@mail.gmail.com>

On Fri, May 22, 2020 at 4:00 PM Kayode Enwerem
<Kayode_Enwerem at ao.uscourts.gov> wrote:
>
> Yes it is sending out connection summary emails.. Please send the config file.
>
> Thanks for the response.
>

Well. it took longer to find it than I expected, but here it is:

cat > /etc/logrotate.d/zeeklog << EOF
# This file is offered as-is, without warranty of any kind.
# (20200420) raubvogel at gmail.com
#
# Rotate only the logs which have anything in them daily,
# compress them, and then keep old ones for 5 days.
#
# NOTE:
# - in my install zeek is in /opt/zeek; adjust as needed
# - /opt/zeek/logs/current is an alias to /opt/zeek/spool/manager

/opt/zeek/spool/manager/*.log {
    rotate 5
    daily
    olddir /opt/zeek/logs
    compress
    dateext
    delaycompress
    copytruncate
    sharedscripts
    missingok
    notifempty
}
EOF

> -----Original Message-----
> From: Mauricio Tavares <raubvogel at gmail.com>
> Sent: Friday, May 22, 2020 3:03 PM
> To: Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov>
> Cc: zeek at zeek.org
> Subject: Re: [Zeek] Logs not rotating
>
> On Fri, May 22, 2020 at 2:27 PM Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov> wrote:
> >
> > Can anyone please assist with this?
> >
> >
> >
> > From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> On Behalf Of
> > Kayode Enwerem
> > Sent: Monday, May 18, 2020 4:36 PM
> > To: zeek at zeek.org
> > Subject: [Zeek] Logs not rotating
> >
>       Is it sending emails out (such as the Connection summary)?
> Reason I ask is the log rotation is tied up with the mail sending. If it is sending emails, or you do not care about that, I can give you a logrotate config file to buy you time to solve the issue.
>
> >
> > Hello,
> >
> >
> >
> > We are experiencing some issues with log rotation. The .log files in current directory are not getting rotated and compressed causing the .log files to grow really big. It starts working again after we restart zeek but after a few hours it stops again.
> >
> >
> >
> > We are using pigz to zip the files and rotate interval is set to 3600(1hr).
> >
> > logrotationinterval = 3600
> >
> > compresscmd = pigz
> >
> >
> >
> > Is anyone else experiencing any similar issue? Any ideas or thought?
> >
> >
> >
> > Zeek version we are running is 3.0.3.
> >
> >
> >
> > Thanks in advance.
> >
> >
> >
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

From nabilmemon.ec at gmail.com  Tue May 26 02:42:41 2020
From: nabilmemon.ec at gmail.com (Nabil Memon)
Date: Tue, 26 May 2020 15:12:41 +0530
Subject: [Zeek] CPU usage reaches 100% when in suspend state
Message-ID: <CADhcAad+0TvsoiyHUo5VqStNL50djCWpZe7LeKq+OwB_Bt=yeg@mail.gmail.com>

Hi Zeek,

I am observing that bro's CPU reaches 100% when it is in suspended state by
default.
Is it a normal behavior?
If yes, any tricks I can do to avoid this?

*/usr/local/bro/bin/bro -i eth1 test.bro -C*

*cat test.bro*
event bro_init()
{
suspend_processing();
}

Thanks & Regards,
Nabil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200526/8e497597/attachment.html 

From hovsep.sanjay.levi at gmail.com  Tue May 26 03:14:39 2020
From: hovsep.sanjay.levi at gmail.com (Hovsep Levi)
Date: Tue, 26 May 2020 10:14:39 +0000
Subject: [Zeek] Zeek 3.1.2 and Kafka - No data flow
In-Reply-To: <CAM1mTCqHujMD71W6GH=4_Jug15gXNKyQJP1qGb9Z-dzD7p_qHQ@mail.gmail.com>
References: <CAJ19UZCKWL+OuOS2XcWGbaYexyPmeGJ1L3XSW4nFf89Xz_k1WQ@mail.gmail.com>
	<CAM1mTCqZ4kpsD6KqN1P9+8zcE9PX_-_X_VwTTOzCWaOkhQT5_Q@mail.gmail.com>
	<CAM1mTCoNK8UbSUJH3OgqSjGi4=KSk15Zxcwd9aUDkzcxaEDoLw@mail.gmail.com>
	<CAM1mTCoP_cCfw7QhxMUxi+iNB1Y3Oo+1A-SEkXqNDMzSfqEM0Q@mail.gmail.com>
	<CAJ19UZBKbwEe4eAhxFbSiHzLG6SKD=ZR32APe3nFcCKaoB4qKA@mail.gmail.com>
	<CAM1mTCqHujMD71W6GH=4_Jug15gXNKyQJP1qGb9Z-dzD7p_qHQ@mail.gmail.com>
Message-ID: <CAJ19UZAy7-N+wzJeGmRibafVZjRu5RR9GFt0dOyOTWAtuQ+pGg@mail.gmail.com>

I tested this successfully today.  Thank you again for your work and
assistance.

/hovsep
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200526/78da0f72/attachment.html 

From nabilmemon.ec at gmail.com  Tue May 26 08:45:30 2020
From: nabilmemon.ec at gmail.com (Nabil Memon)
Date: Tue, 26 May 2020 21:15:30 +0530
Subject: [Zeek] Input framework
In-Reply-To: <CAPfnCujTCm-nswHUd89xsuc8srhrGr=veU=z7J1fB1bJp3bFjw@mail.gmail.com>
References: <CADhcAaerv1k=ocGkHYLLj=uvXA4U=1i3pSLfRNto4Zf7=PKYgQ@mail.gmail.com>
	<CAPfnCujTCm-nswHUd89xsuc8srhrGr=veU=z7J1fB1bJp3bFjw@mail.gmail.com>
Message-ID: <CADhcAaeRT7StrWCkONP0hcVAHjjv9vgwaUmqa6sxoaJR2w72cg@mail.gmail.com>

Hi Justin,

I didn't try that yet.
I got caught up in some other side effect. Bro's CPU usage goes 100% when I
use input framework(or maybe my understanding is incorrect). I have no idea
what triggered this.

I tested with and without two lines(integrating bro's input framework) in
the bro script.

*test.bro (On which I observe 100% CPU usage):*
event bro_init()
{
----------
----------
Input::add_table([$source=file_port_list_loc, $name="file_port_list",
$idx=Portsx, $destination=file_port_list]);
Input::remove("file_port_list");
}

*test.bro (On which I CPU usage is normal):*
event bro_init()
{
----------
----------
*#* Input::add_table([$source=file_port_list_loc, $name="file_port_list",
$idx=Portsx, $destination=file_port_list]);
*#* Input::remove("file_port_list");
}

As you can see, I have commented the actual IF's calls in the latter file.

I did strace of the both the PIDs and below is the output.

*strace -p <PID> (PID of Bro reaching 100% CPU usage):*
select(25, [9 13 15 18 20 22 24], [0 13 15], [0 13 15], {0, 0}) = 1 (out
[0], left {0, 0})
select(25, [9 13 15 18 20 22 24], [0 13 15], [0 13 15], {0, 0}) = 1 (out
[0], left {0, 0})
select(25, [9 13 15 18 20 22 24], [0 13 15], [0 13 15], {0, 0}) = 1 (out
[0], left {0, 0})
select(25, [9 13 15 18 20 22 24], [0 13 15], [0 13 15], {0, 0}) = 1 (out
[0], left {0, 0})

*strace -p <PID> (PID of a nice Bro)::*
select(24, [9 13 15 18 19 21 23], [0 13 15], [0 13 15], {0, 0}) = 1 (out
[0], left {0, 0})
select(0, NULL, NULL, NULL, {0, 20})    = 0 (Timeout)
select(24, [9 13 15 18 19 21 23], [0 13 15], [0 13 15], {0, 0}) = 1 (out
[0], left {0, 0})
select(0, NULL, NULL, NULL, {0, 20})    = 0 (Timeout)

As you can notice, difference between these two outputs, is that the nice
process has one more select of timeout of 20 microseconds(which is I
suppose sleep for 20 microseconds). But the same select call is not being
made with the bad process. Because of which the process takes up 100% CPU.

Am I missing anything here?? I am not able to identify what exactly is
causing this behavior.

Thanks a lot for reading !!

Regards,
Nabil


On Fri, May 22, 2020 at 7:36 PM Justin Azoff <justin at corelight.com> wrote:

> Try deleting this part:
>
> event bro_done()
> {
>    Input::remove("file_port_list");
> }
>
> On Fri, May 22, 2020 at 6:19 AM Nabil Memon <nabilmemon.ec at gmail.com>
> wrote:
> >
> > Hi Zeek,
> >
> > Hope you're all doing well.
> >
> > I am using an Input framework to provide some dynamic input to bro.
> > Here's how the infra looks like,
> >
> > cat /usr/local/bro/share/bro/base/protocols/file_port_list
> > #fields port_num
> > 60000/tcp
> > 8080/tcp
> > 49154/tcp
> > 55907/tcp
> > 49152/tcp
> > 49153/tcp
> > 8000/tcp
> > 5357/tcp
> >
> >
> > type Portsx: record {
> >    port_num: port;
> > };
> >
> > global file_port_list: set[port] = set();
> >
> > redef Communication::nodes += {
> >      ["python"] = [$host = 127.0.0.1, $events = /config_update/,
> $connect=F, $ssl=F]
> > };
> >
> > event bro_init()
> > {
> >    Input::add_table([$source=file_port_list_loc, $name="file_port_list",
> $idx=Portsx, $destination=file_port_list]);
> > }
> >
> > event bro_done()
> > {
> >    Input::remove("file_port_list");
> > }
> >
> >
> > event config_update()
> > {
> >    Input::force_update("file_port_list");
> > }
> >
> > It works really well. But when I terminate bro process, it goes into
> defunct state and throws an error on the console.
> > Error: received signal while waiting for thread
> /usr/local/bro/share/bro/base/protocols/file_port_list/Input::READER_ASCII,
> aborting all ...
> >
> > Anything am I doing wrong??
> >
> > Thanks a lot for looking..!!
> >
> >
> > Regards,
> > Nabil
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
>
> --
> Justin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200526/87407ba4/attachment-0001.html 

From jsiwek at corelight.com  Tue May 26 10:23:09 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Tue, 26 May 2020 10:23:09 -0700
Subject: [Zeek] Input framework
In-Reply-To: <CADhcAaeRT7StrWCkONP0hcVAHjjv9vgwaUmqa6sxoaJR2w72cg@mail.gmail.com>
References: <CADhcAaerv1k=ocGkHYLLj=uvXA4U=1i3pSLfRNto4Zf7=PKYgQ@mail.gmail.com>
	<CAPfnCujTCm-nswHUd89xsuc8srhrGr=veU=z7J1fB1bJp3bFjw@mail.gmail.com>
	<CADhcAaeRT7StrWCkONP0hcVAHjjv9vgwaUmqa6sxoaJR2w72cg@mail.gmail.com>
Message-ID: <CAMzgZ0+y6VD_Y6n8r1+w1fLVbHfbbrPGa_QOzsWnYhg0gORVWw@mail.gmail.com>

On Tue, May 26, 2020 at 8:47 AM Nabil Memon <nabilmemon.ec at gmail.com> wrote:

> I got caught up in some other side effect. Bro's CPU usage goes 100% when I use input framework

What version are you using?  You should try comparing against one of
the latest 3.0.x or 3.1.x releases since there's specifically things
addressed in them that might explain that behavior.  E.g.
https://github.com/zeek/broker/pull/97

- Jon

From nabilmemon.ec at gmail.com  Tue May 26 10:35:45 2020
From: nabilmemon.ec at gmail.com (Nabil Memon)
Date: Tue, 26 May 2020 23:05:45 +0530
Subject: [Zeek] Input framework
In-Reply-To: <CAMzgZ0+y6VD_Y6n8r1+w1fLVbHfbbrPGa_QOzsWnYhg0gORVWw@mail.gmail.com>
References: <CADhcAaerv1k=ocGkHYLLj=uvXA4U=1i3pSLfRNto4Zf7=PKYgQ@mail.gmail.com>
	<CAPfnCujTCm-nswHUd89xsuc8srhrGr=veU=z7J1fB1bJp3bFjw@mail.gmail.com>
	<CADhcAaeRT7StrWCkONP0hcVAHjjv9vgwaUmqa6sxoaJR2w72cg@mail.gmail.com>
	<CAMzgZ0+y6VD_Y6n8r1+w1fLVbHfbbrPGa_QOzsWnYhg0gORVWw@mail.gmail.com>
Message-ID: <CADhcAadLvBptqEzfC16Q7eByAn7jWPmswEimrmhJnC=LG88xeg@mail.gmail.com>

I am using 2.6.x version.

On Tue, 26 May, 2020, 10:53 pm Jon Siwek, <jsiwek at corelight.com> wrote:

> On Tue, May 26, 2020 at 8:47 AM Nabil Memon <nabilmemon.ec at gmail.com>
> wrote:
>
> > I got caught up in some other side effect. Bro's CPU usage goes 100%
> when I use input framework
>
> What version are you using?  You should try comparing against one of
> the latest 3.0.x or 3.1.x releases since there's specifically things
> addressed in them that might explain that behavior.  E.g.
> https://github.com/zeek/broker/pull/97
>
> - Jon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200526/25b378e1/attachment.html 

From holgrain at protonmail.com  Wed May 27 05:23:06 2020
From: holgrain at protonmail.com (Elena Bykovchenko)
Date: Wed, 27 May 2020 12:23:06 +0000
Subject: [Zeek] Persistent fuzzing implementation
Message-ID: <BzFerLyM_R3myukQNR9Lq9879jFtpHXyvliLbyPps4BuJWHrj55yOlr-FRLX8iLNuFIPCnpIEUoGwDjTzdDi2Jx26cIJOUEllna7UhAfsww=@protonmail.com>

Hello. We are trying to implement Zeek fuzzing to find possible bugs in custom protocol analyzer. It seems like a good idea to make it persistent, i.e. start Zeek once and feed it with inputs from fuzzer - it should save a lot of time on initialization, scripts parsing, etc. Persistent fuzzing is usually implemented as a function with input buffer as an argument, like this: https://releases.llvm.org/5.0.0/docs/LibFuzzer.html#fuzz-target
I'm not quite sure how to make it work with Zeek. Current design allows to use either PCAP file or network interface as a packet source, PCAP mode being non-persistent. Does anyone have a guideline for making Zeek able to process input from PCAP files in persistent way? Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200527/2dd5a9f2/attachment.html 

From tim at corelight.com  Wed May 27 09:04:37 2020
From: tim at corelight.com (Tim Wojtulewicz)
Date: Wed, 27 May 2020 09:04:37 -0700
Subject: [Zeek] Persistent fuzzing implementation
In-Reply-To: <BzFerLyM_R3myukQNR9Lq9879jFtpHXyvliLbyPps4BuJWHrj55yOlr-FRLX8iLNuFIPCnpIEUoGwDjTzdDi2Jx26cIJOUEllna7UhAfsww=@protonmail.com>
References: <BzFerLyM_R3myukQNR9Lq9879jFtpHXyvliLbyPps4BuJWHrj55yOlr-FRLX8iLNuFIPCnpIEUoGwDjTzdDi2Jx26cIJOUEllna7UhAfsww=@protonmail.com>
Message-ID: <D8EFD381-6C3B-4DD2-8568-FD54212338BC@corelight.com>

We actually just added an implementation of fuzzing for use with OSS-Fuzz at the request of Google, including an implementation using LibFuzzer. If you want take a look at it, it?s mostly contained within the src/fuzzer directory. It currently only supports the POP3 analyzer and a basic packet fuzzer. If you want to add more, please feel free!

Tim

> On May 27, 2020, at 5:23 AM, Elena Bykovchenko <holgrain at protonmail.com> wrote:
> 
> Hello. We are trying to implement Zeek fuzzing to find possible bugs in custom protocol analyzer. It seems like a good idea to make it persistent, i.e. start Zeek once and feed it with inputs from fuzzer - it should save a lot of time on initialization, scripts parsing, etc. Persistent fuzzing is usually implemented as a function with input buffer as an argument, like this: https://releases.llvm.org/5.0.0/docs/LibFuzzer.html#fuzz-target <https://releases.llvm.org/5.0.0/docs/LibFuzzer.html#fuzz-target>
> I'm not quite sure how to make it work with Zeek. Current design allows to use either PCAP file or network interface as a packet source, PCAP mode being non-persistent. Does anyone have a guideline for making Zeek able to process input from PCAP files in persistent way? Thanks.
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200527/02e6ee10/attachment.html 

From akgraner at corelight.com  Wed May 27 10:24:25 2020
From: akgraner at corelight.com (Amber Graner)
Date: Wed, 27 May 2020 13:24:25 -0400
Subject: [Zeek] Zeek Webinars and Virtual Events for this week 25-29 May 2020
Message-ID: <CAJhOzuq4w+yT7mo47SbS44t1RE1wPcg-6ZMSvPeGCYkniLn32A@mail.gmail.com>

Hi all.

If you missed last weeks Zeek From Home Presentation on Suricata the
recording and slides are now available at:
https://zeek.org/2020/05/27/zeek-from-home-episode-3-suricata/

+++++++++++++++++++++++++++++++++++++
+ EVENTS FOR THIS WEEK  25-29 May 2020 +
++++++++ +++++++++++++++++++++++++++++

*27 May 2020* - ZEEK FROM HOME - 11am PST/2pm EST  - Security Onion - Doug
Burks. This webinar is free but registration is required.  You can register
for this webinar at:
https://corelight.zoom.us/webinar/register/WN_5t5TdekCQYSkYp_b2K5Ngw


*28 May  2020 *? ASK THE ZEEKSPERTS ? 12:30pm PST/3:30pm EST ? Brim
Security ? Phil Rzewski ? 3:30 ? Brim experts will be on hand to answer all
your questions about their latest open source desktop application release.
Registration Link:
https://corelight.zoom.us/webinar/register/WN_lXJb4F5WTRSQ1BQasln9HA

*28 May 2020* - HUNT FROM HOME CTF - 1-3pm PST/4-6pm EST - These CTF events
are sponsored by Corelight, free and open to the public.  If you'd like to
join in the fun and compete in the CTF you can register at:
https://www3.corelight.com/ctf/hunt-from-home-australia

++++++++++++++++++++++++
+   ABOUT THESE EVENTS   +
++++++++ ++++++++++++++++

* ZEEK FROM HOME -  A weekly webinar presentation series where Zeek users,
developers and invited guests can present on Zeek related topics.  These
presentations are recorded and shared with the community.  You can find out
more about Zeek From Home at: https://zeek.org/2020/03/31/zeek-from-home/

* ASK THE ZEEKSPERTS - Is a bi weekly webinar series where Zeek users,
developers and invited guests can answer technical questions about
adopting, implementing and using Zeek data.  The community is invited to
"drop in" to  these calls and ask your questions.   These Ask The
Zeeksperts webinars are NOT recorded.

* HUNT FROM HOME - Is a weekly Capture The Flag (CTF) event sponsored by
Corelight.  These are held every Tuesday from 7-9am PST/10-Noon EST and
Thursday from 1-3pm PST/4-6pm EST.  You can find out more at:
https://www3.corelight.com/ctf/hunt-from-home

If you would like to be added to the calendar reminders please let me know.

Please let me know if you have any questions.

With gratitude,
~Amber
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200527/56bed0c0/attachment-0001.html 

From nabilmemon.ec at gmail.com  Thu May 28 02:27:58 2020
From: nabilmemon.ec at gmail.com (Nabil Memon)
Date: Thu, 28 May 2020 14:57:58 +0530
Subject: [Zeek] Input framework
In-Reply-To: <CADhcAadLvBptqEzfC16Q7eByAn7jWPmswEimrmhJnC=LG88xeg@mail.gmail.com>
References: <CADhcAaerv1k=ocGkHYLLj=uvXA4U=1i3pSLfRNto4Zf7=PKYgQ@mail.gmail.com>
	<CAPfnCujTCm-nswHUd89xsuc8srhrGr=veU=z7J1fB1bJp3bFjw@mail.gmail.com>
	<CADhcAaeRT7StrWCkONP0hcVAHjjv9vgwaUmqa6sxoaJR2w72cg@mail.gmail.com>
	<CAMzgZ0+y6VD_Y6n8r1+w1fLVbHfbbrPGa_QOzsWnYhg0gORVWw@mail.gmail.com>
	<CADhcAadLvBptqEzfC16Q7eByAn7jWPmswEimrmhJnC=LG88xeg@mail.gmail.com>
Message-ID: <CADhcAae9iZQHhCozT4+wD0oti3uvY2Xrc4B4QnLmAEBaC2TXUA@mail.gmail.com>

Hi Jon,

Even in the latest zeek version which is 3.1.3, it reaches 100% when I
integrate Input framework,

*cat  /tmp/file_port_list *
#fields port_num
60000/tcp
8080/tcp
49154/tcp
55907/tcp
49152/tcp
49153/tcp
8000/tcp
5357/tcp
80/tcp

*cat test.zeek*
global file_port_list: set[port] = set();
global file_port_list_loc: string = "/tmp/file_port_list";
type Portsx: record {
  port_num: port;
};
event zeek_init() &priority=5
{
  suspend_processing();
  Input::add_table([$source=file_port_list_loc, $name="file_port_list",
$idx=Portsx, $destination=file_port_list]);
  Input::remove("file_port_list");
}

Regards,
Nabil


On Tue, May 26, 2020 at 11:05 PM Nabil Memon <nabilmemon.ec at gmail.com>
wrote:

> I am using 2.6.x version.
>
> On Tue, 26 May, 2020, 10:53 pm Jon Siwek, <jsiwek at corelight.com> wrote:
>
>> On Tue, May 26, 2020 at 8:47 AM Nabil Memon <nabilmemon.ec at gmail.com>
>> wrote:
>>
>> > I got caught up in some other side effect. Bro's CPU usage goes 100%
>> when I use input framework
>>
>> What version are you using?  You should try comparing against one of
>> the latest 3.0.x or 3.1.x releases since there's specifically things
>> addressed in them that might explain that behavior.  E.g.
>> https://github.com/zeek/broker/pull/97
>>
>> - Jon
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200528/737d061d/attachment.html 

From jsiwek at corelight.com  Thu May 28 09:34:35 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 28 May 2020 09:34:35 -0700
Subject: [Zeek] Input framework
In-Reply-To: <CADhcAae9iZQHhCozT4+wD0oti3uvY2Xrc4B4QnLmAEBaC2TXUA@mail.gmail.com>
References: <CADhcAaerv1k=ocGkHYLLj=uvXA4U=1i3pSLfRNto4Zf7=PKYgQ@mail.gmail.com>
	<CAPfnCujTCm-nswHUd89xsuc8srhrGr=veU=z7J1fB1bJp3bFjw@mail.gmail.com>
	<CADhcAaeRT7StrWCkONP0hcVAHjjv9vgwaUmqa6sxoaJR2w72cg@mail.gmail.com>
	<CAMzgZ0+y6VD_Y6n8r1+w1fLVbHfbbrPGa_QOzsWnYhg0gORVWw@mail.gmail.com>
	<CADhcAadLvBptqEzfC16Q7eByAn7jWPmswEimrmhJnC=LG88xeg@mail.gmail.com>
	<CADhcAae9iZQHhCozT4+wD0oti3uvY2Xrc4B4QnLmAEBaC2TXUA@mail.gmail.com>
Message-ID: <CAMzgZ0+_Fe-NpxC-urykAn5_sUnn9_Rha+=y5FsgR7XMjzRs_g@mail.gmail.com>

Are you missing a matching call to `continue_processing()` ?

event Input::end_of_data(name: string, source: string)
    {
    if ( name == "file_port_list" )
        continue_processing();
    }

- Jon

On Thu, May 28, 2020 at 2:28 AM Nabil Memon <nabilmemon.ec at gmail.com> wrote:
>
> Hi Jon,
>
> Even in the latest zeek version which is 3.1.3, it reaches 100% when I integrate Input framework,
>
> cat  /tmp/file_port_list
> #fields port_num
> 60000/tcp
> 8080/tcp
> 49154/tcp
> 55907/tcp
> 49152/tcp
> 49153/tcp
> 8000/tcp
> 5357/tcp
> 80/tcp
>
> cat test.zeek
> global file_port_list: set[port] = set();
> global file_port_list_loc: string = "/tmp/file_port_list";
> type Portsx: record {
>   port_num: port;
> };
> event zeek_init() &priority=5
> {
>   suspend_processing();
>   Input::add_table([$source=file_port_list_loc, $name="file_port_list", $idx=Portsx, $destination=file_port_list]);
>   Input::remove("file_port_list");
> }
>
> Regards,
> Nabil
>
>
> On Tue, May 26, 2020 at 11:05 PM Nabil Memon <nabilmemon.ec at gmail.com> wrote:
>>
>> I am using 2.6.x version.
>>
>> On Tue, 26 May, 2020, 10:53 pm Jon Siwek, <jsiwek at corelight.com> wrote:
>>>
>>> On Tue, May 26, 2020 at 8:47 AM Nabil Memon <nabilmemon.ec at gmail.com> wrote:
>>>
>>> > I got caught up in some other side effect. Bro's CPU usage goes 100% when I use input framework
>>>
>>> What version are you using?  You should try comparing against one of
>>> the latest 3.0.x or 3.1.x releases since there's specifically things
>>> addressed in them that might explain that behavior.  E.g.
>>> https://github.com/zeek/broker/pull/97
>>>
>>> - Jon

From nabilmemon.ec at gmail.com  Thu May 28 21:55:28 2020
From: nabilmemon.ec at gmail.com (Nabil Memon)
Date: Fri, 29 May 2020 10:25:28 +0530
Subject: [Zeek] Input framework
In-Reply-To: <CAMzgZ0+_Fe-NpxC-urykAn5_sUnn9_Rha+=y5FsgR7XMjzRs_g@mail.gmail.com>
References: <CADhcAaerv1k=ocGkHYLLj=uvXA4U=1i3pSLfRNto4Zf7=PKYgQ@mail.gmail.com>
	<CAPfnCujTCm-nswHUd89xsuc8srhrGr=veU=z7J1fB1bJp3bFjw@mail.gmail.com>
	<CADhcAaeRT7StrWCkONP0hcVAHjjv9vgwaUmqa6sxoaJR2w72cg@mail.gmail.com>
	<CAMzgZ0+y6VD_Y6n8r1+w1fLVbHfbbrPGa_QOzsWnYhg0gORVWw@mail.gmail.com>
	<CADhcAadLvBptqEzfC16Q7eByAn7jWPmswEimrmhJnC=LG88xeg@mail.gmail.com>
	<CADhcAae9iZQHhCozT4+wD0oti3uvY2Xrc4B4QnLmAEBaC2TXUA@mail.gmail.com>
	<CAMzgZ0+_Fe-NpxC-urykAn5_sUnn9_Rha+=y5FsgR7XMjzRs_g@mail.gmail.com>
Message-ID: <CADhcAaf=Lm-fy-HVy3FFiQ+R_uJzyEuCjhoVnD3BKOtguK4Pow@mail.gmail.com>

Hi Jon,

I want to keep bro in suspended mode until a signal through broccoli
framework kicks in.
I see bro utilizing 100% CPU when in suspended state. And normal
percentage CPU(roughly 15-20%) usage when in continued state.

Why bro is taking up so much CPU when in suspended state?

*cat test.bro*
*=======================================================================*
global file_port_list: set[port] = set();
global file_port_list_loc: string = "/tmp/file_port_list";
type Portsx: record {
  port_num: port;
};
event bro_init()
{
    suspend_processing();
    Input::add_table([$source=file_port_list_loc, $name="file_port_list",
$idx=Portsx, $destination=file_port_list]);
    Input::remove("file_port_list");
}

#event Input::end_of_data(name: string, source: string)
#{
#  if ( name == "file_port_list"  ) {}
#    #continue_processing();
#}
*======================================================================= *

*bro -i eth1 test.bro -C*
CPU usage using ps and top command shows 100% usage.
>From the source I could just figure out that, when all the sources are
idle, bro takes some rest of 20 usec. But it seems sources are not getting
idle for some reason.

Below is the strace output.

*strace -p <PID of bro>*
select(17, [4 7 9 11 12 14 16], [0 7 9], [0 7 9], {0, 0}) = 1 (out [0],
left {0, 0})
poll([{fd=11, events=POLLIN}], 1, 0)    = 0 (Timeout)
select(17, [4 7 9 11 12 14 16], [0 7 9], [0 7 9], {0, 0}) = 1 (out [0],
left {0, 0})
poll([{fd=11, events=POLLIN}], 1, 0)    = 0 (Timeout)
select(17, [4 7 9 11 12 14 16], [0 7 9], [0 7 9], {0, 0}) = 1 (out [0],
left {0, 0})
poll([{fd=11, events=POLLIN}], 1, 0)    = 0 (Timeout)


- Nabil

On Thu, May 28, 2020 at 10:04 PM Jon Siwek <jsiwek at corelight.com> wrote:

> Are you missing a matching call to `continue_processing()` ?
>
> event Input::end_of_data(name: string, source: string)
>     {
>     if ( name == "file_port_list" )
>         continue_processing();
>     }
>
> - Jon
>
> On Thu, May 28, 2020 at 2:28 AM Nabil Memon <nabilmemon.ec at gmail.com>
> wrote:
> >
> > Hi Jon,
> >
> > Even in the latest zeek version which is 3.1.3, it reaches 100% when I
> integrate Input framework,
> >
> > cat  /tmp/file_port_list
> > #fields port_num
> > 60000/tcp
> > 8080/tcp
> > 49154/tcp
> > 55907/tcp
> > 49152/tcp
> > 49153/tcp
> > 8000/tcp
> > 5357/tcp
> > 80/tcp
> >
> > cat test.zeek
> > global file_port_list: set[port] = set();
> > global file_port_list_loc: string = "/tmp/file_port_list";
> > type Portsx: record {
> >   port_num: port;
> > };
> > event zeek_init() &priority=5
> > {
> >   suspend_processing();
> >   Input::add_table([$source=file_port_list_loc, $name="file_port_list",
> $idx=Portsx, $destination=file_port_list]);
> >   Input::remove("file_port_list");
> > }
> >
> > Regards,
> > Nabil
> >
> >
> > On Tue, May 26, 2020 at 11:05 PM Nabil Memon <nabilmemon.ec at gmail.com>
> wrote:
> >>
> >> I am using 2.6.x version.
> >>
> >> On Tue, 26 May, 2020, 10:53 pm Jon Siwek, <jsiwek at corelight.com> wrote:
> >>>
> >>> On Tue, May 26, 2020 at 8:47 AM Nabil Memon <nabilmemon.ec at gmail.com>
> wrote:
> >>>
> >>> > I got caught up in some other side effect. Bro's CPU usage goes 100%
> when I use input framework
> >>>
> >>> What version are you using?  You should try comparing against one of
> >>> the latest 3.0.x or 3.1.x releases since there's specifically things
> >>> addressed in them that might explain that behavior.  E.g.
> >>> https://github.com/zeek/broker/pull/97
> >>>
> >>> - Jon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200529/ebacb819/attachment.html 

From jsiwek at corelight.com  Fri May 29 09:42:30 2020
From: jsiwek at corelight.com (Jon Siwek)
Date: Fri, 29 May 2020 09:42:30 -0700
Subject: [Zeek] Input framework
In-Reply-To: <CADhcAaf=Lm-fy-HVy3FFiQ+R_uJzyEuCjhoVnD3BKOtguK4Pow@mail.gmail.com>
References: <CADhcAaerv1k=ocGkHYLLj=uvXA4U=1i3pSLfRNto4Zf7=PKYgQ@mail.gmail.com>
	<CAPfnCujTCm-nswHUd89xsuc8srhrGr=veU=z7J1fB1bJp3bFjw@mail.gmail.com>
	<CADhcAaeRT7StrWCkONP0hcVAHjjv9vgwaUmqa6sxoaJR2w72cg@mail.gmail.com>
	<CAMzgZ0+y6VD_Y6n8r1+w1fLVbHfbbrPGa_QOzsWnYhg0gORVWw@mail.gmail.com>
	<CADhcAadLvBptqEzfC16Q7eByAn7jWPmswEimrmhJnC=LG88xeg@mail.gmail.com>
	<CADhcAae9iZQHhCozT4+wD0oti3uvY2Xrc4B4QnLmAEBaC2TXUA@mail.gmail.com>
	<CAMzgZ0+_Fe-NpxC-urykAn5_sUnn9_Rha+=y5FsgR7XMjzRs_g@mail.gmail.com>
	<CADhcAaf=Lm-fy-HVy3FFiQ+R_uJzyEuCjhoVnD3BKOtguK4Pow@mail.gmail.com>
Message-ID: <CAMzgZ0Kcxy-BtsbJz_hT9CuXdMKoUtHQg1O94jPUXmY=1AZ0ow@mail.gmail.com>

On Thu, May 28, 2020 at 9:55 PM Nabil Memon <nabilmemon.ec at gmail.com> wrote:

> I see bro utilizing 100% CPU when in suspended state. And normal percentage CPU(roughly 15-20%) usage when in continued state.
>
> Why bro is taking up so much CPU when in suspended state?

>From a historical explanation: I don't know, but maybe because it
wasn't meant for general usage or only for some niche use-cases where
some short-term high CPU usage was acceptable (the main use-case I
know is currently in Zeek's test suite to cause determinism in some
input/IPC tests that otherwise have subtle races).

>From a technical explanation: likely it's because the suspend/continue
functions don't remove the PktSrc file descriptor from the IO/polling
loop so it's always considered "ready" but also never processes
anything to drive it forward out of that "ready" state due to being
"suspended".

The technicalities of suspend/continue performing an
unregister/register the PktSrc FD are changeable if you want to make a
GitHub issue for that enhancement (also may help if you provide more
explanation of your use-case and the importance of this to it).
Though might not expect any changes related to this happening until
Zeek 3.2 or later -- 3.0.x has a different IO loop that's trickier to
change, 3.1.x is a bit easier to change, but this isn't a clear-cut
"regression" that suits a patch release, and 2.6.x or before are no
longer supported.

- Jon

From nabilmemon.ec at gmail.com  Sun May 31 00:20:02 2020
From: nabilmemon.ec at gmail.com (Nabil Memon)
Date: Sun, 31 May 2020 12:50:02 +0530
Subject: [Zeek] Input framework
In-Reply-To: <CAMzgZ0Kcxy-BtsbJz_hT9CuXdMKoUtHQg1O94jPUXmY=1AZ0ow@mail.gmail.com>
References: <CADhcAaerv1k=ocGkHYLLj=uvXA4U=1i3pSLfRNto4Zf7=PKYgQ@mail.gmail.com>
	<CAPfnCujTCm-nswHUd89xsuc8srhrGr=veU=z7J1fB1bJp3bFjw@mail.gmail.com>
	<CADhcAaeRT7StrWCkONP0hcVAHjjv9vgwaUmqa6sxoaJR2w72cg@mail.gmail.com>
	<CAMzgZ0+y6VD_Y6n8r1+w1fLVbHfbbrPGa_QOzsWnYhg0gORVWw@mail.gmail.com>
	<CADhcAadLvBptqEzfC16Q7eByAn7jWPmswEimrmhJnC=LG88xeg@mail.gmail.com>
	<CADhcAae9iZQHhCozT4+wD0oti3uvY2Xrc4B4QnLmAEBaC2TXUA@mail.gmail.com>
	<CAMzgZ0+_Fe-NpxC-urykAn5_sUnn9_Rha+=y5FsgR7XMjzRs_g@mail.gmail.com>
	<CADhcAaf=Lm-fy-HVy3FFiQ+R_uJzyEuCjhoVnD3BKOtguK4Pow@mail.gmail.com>
	<CAMzgZ0Kcxy-BtsbJz_hT9CuXdMKoUtHQg1O94jPUXmY=1AZ0ow@mail.gmail.com>
Message-ID: <CADhcAafzGcLJJghN27Y=aP7NbDwfvv58A-RvUqnmo9w4N74yrQ@mail.gmail.com>

Thanks a lot for the thorough explanation.
I really appreciate it.

>From the above explanation, it means that, bro is expecting to be
continued soon after the suspend call to tackle subtle race cases as you
mentioned and there's no rest in between(suspend & continue) which can
stabilize the CPU load(basically it polls for it to be continued again). I
hope it cleared my understanding of how suspend/continue_processing() calls
works.

My use case is I want bro not to process any packets from any source(netmap
pipe or libpcap) until I somehow sends a signal for bro to be continued.
The use case is achieved even now but at a higher cost of CPU being loaded.

>> if you want to make a GitHub issue for that enhancement (also may help
if you provide more explanation of your use-case and the importance of this
to it).
Okay...

Thanks a lot again !!

Regards,
Nabil

On Fri, May 29, 2020 at 10:12 PM Jon Siwek <jsiwek at corelight.com> wrote:

> On Thu, May 28, 2020 at 9:55 PM Nabil Memon <nabilmemon.ec at gmail.com>
> wrote:
>
> > I see bro utilizing 100% CPU when in suspended state. And normal
> percentage CPU(roughly 15-20%) usage when in continued state.
> >
> > Why bro is taking up so much CPU when in suspended state?
>
> From a historical explanation: I don't know, but maybe because it
> wasn't meant for general usage or only for some niche use-cases where
> some short-term high CPU usage was acceptable (the main use-case I
> know is currently in Zeek's test suite to cause determinism in some
> input/IPC tests that otherwise have subtle races).
>
> From a technical explanation: likely it's because the suspend/continue
> functions don't remove the PktSrc file descriptor from the IO/polling
> loop so it's always considered "ready" but also never processes
> anything to drive it forward out of that "ready" state due to being
> "suspended".
>
> The technicalities of suspend/continue performing an
> unregister/register the PktSrc FD are changeable if you want to make a
> GitHub issue for that enhancement (also may help if you provide more
> explanation of your use-case and the importance of this to it).
> Though might not expect any changes related to this happening until
> Zeek 3.2 or later -- 3.0.x has a different IO loop that's trickier to
> change, 3.1.x is a bit easier to change, but this isn't a clear-cut
> "regression" that suits a patch release, and 2.6.x or before are no
> longer supported.
>
> - Jon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200531/4a9bb235/attachment.html