From austin522 at gmail.com  Fri Nov  1 02:43:07 2019
From: austin522 at gmail.com (venkatesh bandari)
Date: Fri, 1 Nov 2019 02:43:07 -0700
Subject: [Zeek] zeek package manager
Message-ID: <CAKPzng-FH+Kc558QNTLYncM5ktH3uX2mDSatg0XUuFKDBAdGbQ@mail.gmail.com>

Hello team,

iam trying to install brp-community-id via package manager and it is
failing on the below error.anyone can help

i tried installing from the source and it is also failing

root at ubuntu:/usr/local/zeek/share/zeek/site/packages# zkg install
zeek/corelight/bro-community-id
The following packages will be INSTALLED:
  zeek/corelight/bro-community-id (1.2)

Proceed? [Y/n] y
Running unit tests for "zeek/corelight/bro-community-id"
error: failed to run tests for zeek/corelight/bro-community-id: package
build_command failed, see log in /root/.zkg/logs/bro-community-id-build.log
Proceed to install anyway? [N/y] y

errorlog:
CMakeFiles/Corelight-CommunityID.linux-x86_64.dir/build.make:105: recipe
for target
'CMakeFiles/Corelight-CommunityID.linux-x86_64.dir/communityid.bif.cc.o'
failed
make[3]: ***
[CMakeFiles/Corelight-CommunityID.linux-x86_64.dir/communityid.bif.cc.o]
Error 1
make[3]: Leaving directory '/root/bro-community-id/build'
CMakeFiles/Makefile2:204: recipe for target
'CMakeFiles/Corelight-CommunityID.linux-x86_64.dir/all' failed
make[2]: *** [CMakeFiles/Corelight-CommunityID.linux-x86_64.dir/all] Error 2
make[2]: Leaving directory '/root/bro-community-id/build'
Makefile:151: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/root/bro-community-id/build'
Makefile:11: recipe for target 'build-it' failed
make: *** [build-it] Error 2
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191101/1eb339d2/attachment-0001.html 

From austin522 at gmail.com  Fri Nov  1 03:45:32 2019
From: austin522 at gmail.com (venkatesh bandari)
Date: Fri, 1 Nov 2019 03:45:32 -0700
Subject: [Zeek] smtp-url-extraction
Message-ID: <CAKPzng-wwMGtDvHKQ3h4A4JM30h1ik8Q3xpLphfabf+6VKDMjQ@mail.gmail.com>

Hello team,

i loaded the smtp-url-extarction script in local.zeek.how do i check if the
script is working and where do i see the results.

# smtp url-extraction

@load
/usr/local/zeek/share/zeek/policy/frameworks/intel/seen/smtp-url-extraction.zeek

#where-locations
@load
/usr/local/zeek/share/zeek/policy/frameworks/intel/seen/where-locations.zeek

thanks
Venkatesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191101/dda466a4/attachment.html 

From henridf at gmail.com  Fri Nov  1 04:11:28 2019
From: henridf at gmail.com (Henri Dubois-Ferriere)
Date: Fri, 1 Nov 2019 12:11:28 +0100
Subject: [Zeek] printing stream columns
In-Reply-To: <CABcA1ruH2qu-3R96UVQNbCJxN+FqdycNze13g2kRg7rSpZ_VbQ@mail.gmail.com>
References: <CABcA1ruzNDefM16QHP5yJnOqXL4h=q0=LQwDLatCLaKcHS2aAg@mail.gmail.com>
	<CAMzgZ0KkEk--tNxm7f2dBWuWqCYo8hddfE4DFLgzcOXcV+nAyQ@mail.gmail.com>
	<CABcA1ruH2qu-3R96UVQNbCJxN+FqdycNze13g2kRg7rSpZ_VbQ@mail.gmail.com>
Message-ID: <CABcA1rsvz9kmeWRBwxW5BKungSDE20bjkL53e9UbAtoGwuURuQ@mail.gmail.com>

I've been playing around with Jon's script and am getting close to what I
want, but still have one outstanding issue related to nested records.
Currently they show up as a single feed with a type "record foo" (such
as "record conn_id" or "record FTP::ExpectedDataChannel").

I'd like to be able to peek into nested records to get the inner fields
that will show up in the logs. It doesn't seem like there's a way to do
record introspection given a string representation of the record type name,
but if I'd be delighted to be told I'm missing something.

Thanks for any pointers!

On Wed, 16 Oct 2019 at 22:47, Henri Dubois-Ferriere <henridf at gmail.com>
wrote:

> Thanks Jon and Anthony for the quick responses! print-log-info.bro looks
> promising for what I'm trying to do.
>
> On Wed, 16 Oct 2019 at 22:37, Jon Siwek <jsiwek at corelight.com> wrote:
>
>> On Wed, Oct 16, 2019 at 12:48 PM Henri Dubois-Ferriere
>> <henridf at gmail.com> wrote:
>> >
>> > I'm trying to print the record type for each log stream at startup.
>> Something like:
>> >
>> >  for ( id in Log::active_streams ) {
>> >                  local stream = Log::active_streams[id];
>> >                  print stream$path, stream$columns;
>> > }
>> >
>> > doesn't work because $columns is a record type, and gets stringified
>> "<no value description>".
>>
>> Zeek 3.0 should give better descriptions for types.  This was the
>> relevant patch which is not in any 2.6.x version:
>>
>>
>> https://github.com/bro/bro/commit/1f450c05102be6dd7ebcc2c5901d5a3a231cd675
>>
>> This script may also help demonstrate things related to what you're
>> trying to do:
>>
>>   https://gist.github.com/jsiwek/f843b3321f4227b6ec32d110424ebf70
>>
>> It prints field descriptions of all logs either to stdout or a CSV
>> file.  Example command:
>>
>>   ZEEK_ALLOW_INIT_ERRORS=1 zeek print-log-info.bro PrintLogs::csv=F
>>
>> Sample of output:
>>
>> known_hosts.log | Hosts with complete TCP handshakes
>>   ts: time - The timestamp at which the host was detected.
>>   host: addr - The address that was detected originating or responding
>> to a TCP connection.
>>
>> - Jon
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191101/56dd4b7f/attachment.html 

From austin522 at gmail.com  Fri Nov  1 04:17:35 2019
From: austin522 at gmail.com (venkatesh bandari)
Date: Fri, 1 Nov 2019 19:17:35 +0800
Subject: [Zeek] zeek ts conversion
In-Reply-To: <988A8876-2DCD-46A5-95D8-1A4A5D6196A8@corelight.com>
References: <CAKPzng_ps2YZzJqUy_2ZGWpJHcRrmcixqmRg8sb9RoqfRa0TJg@mail.gmail.com>
	<988A8876-2DCD-46A5-95D8-1A4A5D6196A8@corelight.com>
Message-ID: <CAKPzng_SgtjqFZXH+2Jy3MWP1HkhADCm15WnHX9gkxrmuin3dg@mail.gmail.com>

Thank you Seth

On Thu, 31 Oct 2019 at 8:08 PM, Seth Hall <seth at corelight.com> wrote:

> In local.bro, add the following line...
>
> redef LogAscii::json_timestamps = JSON::TS_ISO8601;
>
> That should make your log have timestamps in ISO8601 time format which
> most systems natively recognize and understand.
>
> .Seth
>
> On 29 Oct 2019, at 23:31, venkatesh bandari wrote:
>
> Hello team,
>
> we are doing a zeek poc.iam doing the integration with splunk.in the spunk
> logs i see the ts value which is not in human readable
> format.zeek-cut/bro-cut on the box can be used to convert ts to human
> readable format using -d
>
> the question is how can i do this before sending the json logs to
> splunk.is
> there a way
>
> Thanks
> Venkatesh
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
> --
> Seth Hall * Corelight, Inc * www.corelight.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191101/f5b2d9e9/attachment.html 

From austin522 at gmail.com  Fri Nov  1 04:45:18 2019
From: austin522 at gmail.com (venkatesh bandari)
Date: Fri, 1 Nov 2019 19:45:18 +0800
Subject: [Zeek] zeek package manager on host not connected to internet
Message-ID: <CAKPzng9L-=5KwGy_Dd2UST8NEdYiCeCC--H5hGLoBUti4-Ke7g@mail.gmail.com>

Hello team,

i have  zeek package manager installed on one system.iam trying to
transfer packages to another system using bundle,unbundle.

do i need to install zkg on another system( lf yes,iam having a problem
installing zkg using pip)

is there anyother way to transfer zeek packages from one system to another
system and load.

thanks
Venkatesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191101/8a4d2216/attachment.html 

From austin522 at gmail.com  Fri Nov  1 08:29:34 2019
From: austin522 at gmail.com (venkatesh bandari)
Date: Fri, 1 Nov 2019 23:29:34 +0800
Subject: [Zeek] smtp url extraction logs
Message-ID: <CAKPzng_oVGC34fXUjbg+K0mmJvRh4tx5g=YuwahmSwFRiUW+rQ@mail.gmail.com>

hello team

iam have included the below line in my local.zeek.started zeek and i dont
see the logs

@load
/opt/zeek/share/zeek/policy/frameworks/intel/seen/smtp-url-extraction.zeek


could someone help to tell where i can see the logs or iam doing
something wrong in loading the scripts
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191101/e53ddb81/attachment.html 

From pumphrey.adam at gmail.com  Fri Nov  1 09:37:54 2019
From: pumphrey.adam at gmail.com (ap)
Date: Fri, 1 Nov 2019 12:37:54 -0400
Subject: [Zeek] smtp url extraction logs
In-Reply-To: <CAKPzng_oVGC34fXUjbg+K0mmJvRh4tx5g=YuwahmSwFRiUW+rQ@mail.gmail.com>
References: <CAKPzng_oVGC34fXUjbg+K0mmJvRh4tx5g=YuwahmSwFRiUW+rQ@mail.gmail.com>
Message-ID: <ED309989-F798-4468-923F-EE7E8B19633B@gmail.com>

Hi Venkatesh, 

The scripts you've loaded make the Intel Framework "aware" of URL's discovered in SMTP messages.  This is specifically for matching URL threat intel indicators in SMTP Email.  Without the indicators, they won't produce any noticeable output.  So now that you've loaded the scripts, your next step is to feed the Intel Framework URL indicators.  You can read more about loading indicators here:  https://docs.zeek.org/en/stable/frameworks/intel.html.  

An important thing to know/remember wrt URL indicators is that they should not include the scheme (the "http://" part).  For example:

use this... 

www.badsite.com/path/to/exploitkit

not this...

http://www.badsite.com/path/to/exploitkit

If you're set up so your Zeek sensor can see SMTP traffic you generate, you can load a URL indicator then send a test email containing it.  You should see an entry appear in the intel.log file.  

That's one way to find known-bad URLs in Email.  If you're interested in other Email analysis techniques that don't exclusively rely on indicators, check out the smtp-url-analysis package.  You can read more about it here: https://github.com/initconf/smtp-url-analysis.  It uses some other great techniques for finding phishing activity, like identifying when a link in an email was clicked by a recipient.  

Hope that helps. 

Adam 

> On Nov 1, 2019, at 11:29 AM, venkatesh bandari <austin522 at gmail.com> wrote:
> 
> hello team
> 
> iam have included the below line in my local.zeek.started zeek and i dont see the logs
> 
> @load /opt/zeek/share/zeek/policy/frameworks/intel/seen/smtp-url-extraction.zeek
> 
> 
> could someone help to tell where i can see the logs or iam doing something wrong in loading the scripts
> 
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



From jsiwek at corelight.com  Fri Nov  1 12:53:13 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Fri, 1 Nov 2019 12:53:13 -0700
Subject: [Zeek] printing stream columns
In-Reply-To: <CABcA1rsvz9kmeWRBwxW5BKungSDE20bjkL53e9UbAtoGwuURuQ@mail.gmail.com>
References: <CABcA1ruzNDefM16QHP5yJnOqXL4h=q0=LQwDLatCLaKcHS2aAg@mail.gmail.com>
	<CAMzgZ0KkEk--tNxm7f2dBWuWqCYo8hddfE4DFLgzcOXcV+nAyQ@mail.gmail.com>
	<CABcA1ruH2qu-3R96UVQNbCJxN+FqdycNze13g2kRg7rSpZ_VbQ@mail.gmail.com>
	<CABcA1rsvz9kmeWRBwxW5BKungSDE20bjkL53e9UbAtoGwuURuQ@mail.gmail.com>
Message-ID: <CAMzgZ0Kh_owe1Ze25BOYGbn2sstvhq6dcv=UW5gmXt41dW4o6A@mail.gmail.com>

On Fri, Nov 1, 2019 at 4:11 AM Henri Dubois-Ferriere <henridf at gmail.com> wrote:

> I'd like to be able to peek into nested records to get the inner fields that will show up in the logs. It doesn't seem like there's a way to do record introspection given a string representation of the record type name, but if I'd be delighted to be told I'm missing something.

No, didn't look like there was a way to do that, but I've made a
PR/patch that should make recursive introspection possible via
something like `record_fields("conn_id")` for any arbitrary record
type name:

https://github.com/zeek/zeek/pull/675

- Jon


From henridf at gmail.com  Fri Nov  1 12:55:07 2019
From: henridf at gmail.com (Henri Dubois-Ferriere)
Date: Fri, 1 Nov 2019 20:55:07 +0100
Subject: [Zeek] printing stream columns
In-Reply-To: <CAMzgZ0Kh_owe1Ze25BOYGbn2sstvhq6dcv=UW5gmXt41dW4o6A@mail.gmail.com>
References: <CABcA1ruzNDefM16QHP5yJnOqXL4h=q0=LQwDLatCLaKcHS2aAg@mail.gmail.com>
	<CAMzgZ0KkEk--tNxm7f2dBWuWqCYo8hddfE4DFLgzcOXcV+nAyQ@mail.gmail.com>
	<CABcA1ruH2qu-3R96UVQNbCJxN+FqdycNze13g2kRg7rSpZ_VbQ@mail.gmail.com>
	<CABcA1rsvz9kmeWRBwxW5BKungSDE20bjkL53e9UbAtoGwuURuQ@mail.gmail.com>
	<CAMzgZ0Kh_owe1Ze25BOYGbn2sstvhq6dcv=UW5gmXt41dW4o6A@mail.gmail.com>
Message-ID: <CABcA1rs7yAbdhiZTjBaRk0yK__R6zqWGR-mfPvxFgebkt3WPdw@mail.gmail.com>

Ooh that looks great. Thanks!

On Fri, 1 Nov 2019 at 20:53, Jon Siwek <jsiwek at corelight.com> wrote:

> On Fri, Nov 1, 2019 at 4:11 AM Henri Dubois-Ferriere <henridf at gmail.com>
> wrote:
>
> > I'd like to be able to peek into nested records to get the inner fields
> that will show up in the logs. It doesn't seem like there's a way to do
> record introspection given a string representation of the record type name,
> but if I'd be delighted to be told I'm missing something.
>
> No, didn't look like there was a way to do that, but I've made a
> PR/patch that should make recursive introspection possible via
> something like `record_fields("conn_id")` for any arbitrary record
> type name:
>
> https://github.com/zeek/zeek/pull/675
>
> - Jon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191101/bcd11507/attachment.html 

From jsiwek at corelight.com  Fri Nov  1 13:06:10 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Fri, 1 Nov 2019 13:06:10 -0700
Subject: [Zeek] zeek package manager on host not connected to internet
In-Reply-To: <CAKPzng9L-=5KwGy_Dd2UST8NEdYiCeCC--H5hGLoBUti4-Ke7g@mail.gmail.com>
References: <CAKPzng9L-=5KwGy_Dd2UST8NEdYiCeCC--H5hGLoBUti4-Ke7g@mail.gmail.com>
Message-ID: <CAMzgZ0Jyg4TOaqJEX54z_Qe_PNgW9iKfVFMxiwUaWy74wW9b=Q@mail.gmail.com>

On Fri, Nov 1, 2019 at 4:46 AM venkatesh bandari <austin522 at gmail.com> wrote:

> do i need to install zkg on another system( lf yes,iam having a problem installing zkg using pip)

Yes, the easiest way to unbundle would be to have `zkg` do it.  Also,
`pip` itself doesn't necessarily need internet to install `zkg` if you
want to just download the source directly and use it.  For example:

  wget https://github.com/zeek/package-manager/archive/v2.0.7.tar.gz
  # Transport the tarfile however you need to get it where it has to be.
  tar -xzf v2.0.7.tar.gz
  pip install package-manager-2.0.7/

Or if the issue is using `pip` at all, you can install manually in the
usual setuptools way.  E.g. `python setup.py install`.

- Jon

From jsiwek at corelight.com  Fri Nov  1 17:23:35 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Fri, 1 Nov 2019 17:23:35 -0700
Subject: [Zeek] zeek package manager
In-Reply-To: <CAKPzng-FH+Kc558QNTLYncM5ktH3uX2mDSatg0XUuFKDBAdGbQ@mail.gmail.com>
References: <CAKPzng-FH+Kc558QNTLYncM5ktH3uX2mDSatg0XUuFKDBAdGbQ@mail.gmail.com>
Message-ID: <CAMzgZ0L=FQQGrV0UK0xYZWPYY7ygbGwEaHuCsNMwR+tEydp=fw@mail.gmail.com>

Thanks for reporting that, there's a recent change in Zeek's git
`master` branch which breaks that plugin.  I've created an issue
related to that for you to follow along here if you'd like:

    https://github.com/zeek/zeek/issues/676

You can also choose to use the Zeek 3.0.0 stable release in the
meantime which doesn't have this issue.  Can download it here:

    https://www.zeek.org/downloads/zeek-3.0.0.tar.gz

- Jon

On Fri, Nov 1, 2019 at 2:45 AM venkatesh bandari <austin522 at gmail.com> wrote:
>
> Hello team,
>
> iam trying to install brp-community-id via package manager and it is failing on the below error.anyone can help
>
> i tried installing from the source and it is also failing
>
> root at ubuntu:/usr/local/zeek/share/zeek/site/packages# zkg install zeek/corelight/bro-community-id
> The following packages will be INSTALLED:
>   zeek/corelight/bro-community-id (1.2)
>
> Proceed? [Y/n] y
> Running unit tests for "zeek/corelight/bro-community-id"
> error: failed to run tests for zeek/corelight/bro-community-id: package build_command failed, see log in /root/.zkg/logs/bro-community-id-build.log
> Proceed to install anyway? [N/y] y
>
> errorlog:
> CMakeFiles/Corelight-CommunityID.linux-x86_64.dir/build.make:105: recipe for target 'CMakeFiles/Corelight-CommunityID.linux-x86_64.dir/communityid.bif.cc.o' failed
> make[3]: *** [CMakeFiles/Corelight-CommunityID.linux-x86_64.dir/communityid.bif.cc.o] Error 1
> make[3]: Leaving directory '/root/bro-community-id/build'
> CMakeFiles/Makefile2:204: recipe for target 'CMakeFiles/Corelight-CommunityID.linux-x86_64.dir/all' failed
> make[2]: *** [CMakeFiles/Corelight-CommunityID.linux-x86_64.dir/all] Error 2
> make[2]: Leaving directory '/root/bro-community-id/build'
> Makefile:151: recipe for target 'all' failed
> make[1]: *** [all] Error 2
> make[1]: Leaving directory '/root/bro-community-id/build'
> Makefile:11: recipe for target 'build-it' failed
> make: *** [build-it] Error 2
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

From austin522 at gmail.com  Fri Nov  1 17:29:12 2019
From: austin522 at gmail.com (venkatesh bandari)
Date: Sat, 2 Nov 2019 08:29:12 +0800
Subject: [Zeek] zeek package manager
In-Reply-To: <CAMzgZ0L=FQQGrV0UK0xYZWPYY7ygbGwEaHuCsNMwR+tEydp=fw@mail.gmail.com>
References: <CAKPzng-FH+Kc558QNTLYncM5ktH3uX2mDSatg0XUuFKDBAdGbQ@mail.gmail.com>
	<CAMzgZ0L=FQQGrV0UK0xYZWPYY7ygbGwEaHuCsNMwR+tEydp=fw@mail.gmail.com>
Message-ID: <CAKPzng8aDZepJ_bGM21n39wsUAxWGaZTg=+08Ox17hF2OSQcbw@mail.gmail.com>

Thank you John for looking into this,I already installed Zeek..I am trying
to install Zeek package manger


On Sat, 2 Nov 2019 at 8:23 AM, Jon Siwek <jsiwek at corelight.com> wrote:

> Thanks for reporting that, there's a recent change in Zeek's git
> `master` branch which breaks that plugin.  I've created an issue
> related to that for you to follow along here if you'd like:
>
>     https://github.com/zeek/zeek/issues/676
>
> You can also choose to use the Zeek 3.0.0 stable release in the
> meantime which doesn't have this issue.  Can download it here:
>
>     https://www.zeek.org/downloads/zeek-3.0.0.tar.gz
>
> - Jon
>
> On Fri, Nov 1, 2019 at 2:45 AM venkatesh bandari <austin522 at gmail.com>
> wrote:
> >
> > Hello team,
> >
> > iam trying to install brp-community-id via package manager and it is
> failing on the below error.anyone can help
> >
> > i tried installing from the source and it is also failing
> >
> > root at ubuntu:/usr/local/zeek/share/zeek/site/packages# zkg install
> zeek/corelight/bro-community-id
> > The following packages will be INSTALLED:
> >   zeek/corelight/bro-community-id (1.2)
> >
> > Proceed? [Y/n] y
> > Running unit tests for "zeek/corelight/bro-community-id"
> > error: failed to run tests for zeek/corelight/bro-community-id: package
> build_command failed, see log in /root/.zkg/logs/bro-community-id-build.log
> > Proceed to install anyway? [N/y] y
> >
> > errorlog:
> > CMakeFiles/Corelight-CommunityID.linux-x86_64.dir/build.make:105: recipe
> for target
> 'CMakeFiles/Corelight-CommunityID.linux-x86_64.dir/communityid.bif.cc.o'
> failed
> > make[3]: ***
> [CMakeFiles/Corelight-CommunityID.linux-x86_64.dir/communityid.bif.cc.o]
> Error 1
> > make[3]: Leaving directory '/root/bro-community-id/build'
> > CMakeFiles/Makefile2:204: recipe for target
> 'CMakeFiles/Corelight-CommunityID.linux-x86_64.dir/all' failed
> > make[2]: *** [CMakeFiles/Corelight-CommunityID.linux-x86_64.dir/all]
> Error 2
> > make[2]: Leaving directory '/root/bro-community-id/build'
> > Makefile:151: recipe for target 'all' failed
> > make[1]: *** [all] Error 2
> > make[1]: Leaving directory '/root/bro-community-id/build'
> > Makefile:11: recipe for target 'build-it' failed
> > make: *** [build-it] Error 2
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191102/0e13e982/attachment.html 

From ralph.rye at gmail.com  Mon Nov  4 07:04:03 2019
From: ralph.rye at gmail.com (Ralph R. Rye)
Date: Mon, 4 Nov 2019 10:04:03 -0500
Subject: [Zeek] ERSPAN / GRE - weird log
Message-ID: <CANyHVQDDUawbmeNxTrRN+v2gSQg_79bcrO+es6LqRLn33PbGwQ@mail.gmail.com>

Hoping to see if someone has gotten Zeek to work with ERSPAN span sessions.

I am doing ERSPAN from a Cisco Nexus switch to a VMware host.  I can see
the traffic at the host and do tcpdump captures without any problems.

When attempting to use Zeek (3.0 or 2.6.3) all I get is entries in the
weird log for the ERSPAN traffic.

I noticed someone previously posting about it may be a GRE type issue, and
that it appears someone modified a source file to get things to work.

Here is the frame/packet header info from the ERSPAN traffic from the Nexus
9k.

As you can see it is type 0x88be

[image: image.png]

I have used Zeek quite a bit in the past with regular SPAN sessions and
TAPs, but having the capability to use ERSPAN would be a great benefit of
being able to pull in traffic from many sections of the network without
having to worry about the physical device requirements of regular SPAN and
TAPS.

I utilize ERSPAN quite a bit with tshark/wireshark for being able to
capture just the traffic I care about in a datacenter.

-Ralph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191104/e08c90b7/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 45590 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191104/e08c90b7/attachment-0001.bin 

From justin at corelight.com  Mon Nov  4 07:13:05 2019
From: justin at corelight.com (Justin Azoff)
Date: Mon, 4 Nov 2019 10:13:05 -0500
Subject: [Zeek] ERSPAN / GRE - weird log
In-Reply-To: <CANyHVQDDUawbmeNxTrRN+v2gSQg_79bcrO+es6LqRLn33PbGwQ@mail.gmail.com>
References: <CANyHVQDDUawbmeNxTrRN+v2gSQg_79bcrO+es6LqRLn33PbGwQ@mail.gmail.com>
Message-ID: <CAPfnCuj7=4J3fqYdyxkSq6k_3conikeBPhy5OFiM5XQbRpEnJg@mail.gmail.com>

On Mon, Nov 4, 2019 at 10:07 AM Ralph R. Rye <ralph.rye at gmail.com> wrote:

> Hoping to see if someone has gotten Zeek to work with ERSPAN span sessions.
>
> I am doing ERSPAN from a Cisco Nexus switch to a VMware host.  I can see
> the traffic at the host and do tcpdump captures without any problems.
>
> When attempting to use Zeek (3.0 or 2.6.3) all I get is entries in the
> weird log for the ERSPAN traffic.
>

2.6 would definitely not work, but 3.0 has support for this:

https://github.com/zeek/zeek/commit/d9533e9616c5e9e34e811b6db57700be8ab61544

What exactly are you getting in the weird.log on 3.0 ?


-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191104/dc49f2fb/attachment.html 

From jan.grashoefer at gmail.com  Tue Nov  5 08:56:15 2019
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Tue, 5 Nov 2019 17:56:15 +0100
Subject: [Zeek] Status of 3 plugins
In-Reply-To: <CAPfnCugK8gvpT=KjkdQHj=Fr6gOLR+iFPaqO0potzZ738=ucZQ@mail.gmail.com>
References: <d44d529d10c95a4a4e982157151cff05@slave-tothe-box.net>
	<CAPfnCugK8gvpT=KjkdQHj=Fr6gOLR+iFPaqO0potzZ738=ucZQ@mail.gmail.com>
Message-ID: <d3641cf3-466a-ba9f-af8b-36308f84c610@gmail.com>

On 31/10/2019 23:30, Justin Azoff wrote:> 
https://github.com/J-Gras/intel-seen-more
> 
> 
> Looks like this hits the issue where it depends on bro/something which 
> are now all zeek/something? I fixed this in one of my packages by just 
> deleting the "bro/" part of the dependency but I think this is more a 
> migration issue that zkg could help resolve.

The issue is already fixed in master since June. I just forgot to tag a 
new release.

> Well, 2 out of the 3 already work, and one just needs a minor update 
> that I'm sure Jan would be happy to make if someone had just told him 
> about it.

True :)

Jan

From stu.h at live.com  Tue Nov  5 10:44:08 2019
From: stu.h at live.com (Stuart H)
Date: Tue, 5 Nov 2019 18:44:08 +0000
Subject: [Zeek] ERSPAN / GRE - weird log
In-Reply-To: <CANyHVQDDUawbmeNxTrRN+v2gSQg_79bcrO+es6LqRLn33PbGwQ@mail.gmail.com>
References: <CANyHVQDDUawbmeNxTrRN+v2gSQg_79bcrO+es6LqRLn33PbGwQ@mail.gmail.com>
Message-ID: <2FDE50E1-4BB5-404F-A1CC-71DAFEEE8A22@live.com>

I added support for ERSPAN type II and III and have it working fine using VMware ERSPAN.

You?re definitely using Zeek 3.0+ right?

From: <zeek-bounces at zeek.org> on behalf of "Ralph R. Rye" <ralph.rye at gmail.com>
Date: Monday, 4 November 2019 at 15:08
To: "zeek at zeek.org" <zeek at zeek.org>
Subject: [Zeek] ERSPAN / GRE - weird log

Hoping to see if someone has gotten Zeek to work with ERSPAN span sessions.

I am doing ERSPAN from a Cisco Nexus switch to a VMware host.  I can see the traffic at the host and do tcpdump captures without any problems.

When attempting to use Zeek (3.0 or 2.6.3) all I get is entries in the weird log for the ERSPAN traffic.

I noticed someone previously posting about it may be a GRE type issue, and that it appears someone modified a source file to get things to work.

Here is the frame/packet header info from the ERSPAN traffic from the Nexus 9k.

As you can see it is type 0x88be

[cid:image001.png at 01D59409.015DD200]

I have used Zeek quite a bit in the past with regular SPAN sessions and TAPs, but having the capability to use ERSPAN would be a great benefit of being able to pull in traffic from many sections of the network without having to worry about the physical device requirements of regular SPAN and TAPS.

I utilize ERSPAN quite a bit with tshark/wireshark for being able to capture just the traffic I care about in a datacenter.

-Ralph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191105/e329f233/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 45591 bytes
Desc: image001.png
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191105/e329f233/attachment-0001.bin 

From austin522 at gmail.com  Tue Nov  5 19:56:40 2019
From: austin522 at gmail.com (venkatesh bandari)
Date: Wed, 6 Nov 2019 11:56:40 +0800
Subject: [Zeek] pf_ring
Message-ID: <CAKPzng-Qa_B-6r1zL1kfxSaFT0ECar9tcKVXLY4hu8CtCBTO4g@mail.gmail.com>

Hello Team,

i installed zeek and using standard libcap
ldd /opt/zeek/bin/zeek | grep pcap
        libpcap.so.1 => /lib64/libpcap.so.1 (0x00007f1bad618000)

iam seeing the box is dropping 40% packets.iam planning to have 2 more
workers with pf_ring.Installed pf_ring

in the link https://www.zeek.org/documentation/load-balancing.html
it says zeek to point to below libcap.how doi change the to pfring sware
libcap

ldd /usr/local/bro/bin/bro | grep pcap
        libpcap.so.1 => /opt/pfring/lib/libpcap.so.1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/e826f4ce/attachment.html 

From 14beemabdullah at seecs.edu.pk  Tue Nov  5 22:13:54 2019
From: 14beemabdullah at seecs.edu.pk (Muhammad Abdullah)
Date: Wed, 6 Nov 2019 11:13:54 +0500
Subject: [Zeek]  DNS logs Anonymization
Message-ID: <CACSt-sxG7w8a5xcO_v-TwOFWfCEiGoYmBStgSSZKu+27vMrTzg@mail.gmail.com>

Hi,

I've just started using Zeek to collect DNS logs by running it from the
CLI. I want to anonymize the IPs contained in these logs. How should I go
about it?

Thanks,

Muhammad.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/642d2f0e/attachment.html 

From mauro.palumbo at aizoon.it  Wed Nov  6 02:08:41 2019
From: mauro.palumbo at aizoon.it (Palumbo Mauro)
Date: Wed, 6 Nov 2019 10:08:41 +0000
Subject: [Zeek] uid in files logs
Message-ID: <b474e395ba664b5fb7a6072d42ed6c18@SRVEX03.aizoon.local>

Hi everybody,
   it would be useful for us to have the conn uids in the logs from file analyzers (pe.log, x509.log,...). I know this information can be gathered by cross-cehcking different bro logs, but it will save some time to have it already in pe.log, etc. I believe this data is available in the record fa_file.conns, available in events in the file framework, so it seems not difficult to add.
Is there any reason why it is not added by default?

Thanks,
Mauro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/2e8ca0e8/attachment.html 

From jlay at slave-tothe-box.net  Wed Nov  6 03:42:39 2019
From: jlay at slave-tothe-box.net (James Lay)
Date: Wed, 06 Nov 2019 04:42:39 -0700
Subject: [Zeek] Status of 3 plugins
In-Reply-To: <d3641cf3-466a-ba9f-af8b-36308f84c610@gmail.com>
References: <d44d529d10c95a4a4e982157151cff05@slave-tothe-box.net>
	<CAPfnCugK8gvpT=KjkdQHj=Fr6gOLR+iFPaqO0potzZ738=ucZQ@mail.gmail.com>
	<d3641cf3-466a-ba9f-af8b-36308f84c610@gmail.com>
Message-ID: <8c19207fdeeb7e0f2e480970d67c885a6f96964c.camel@slave-tothe-box.net>

Thank you!

On Tue, 2019-11-05 at 17:56 +0100, Jan Grash?fer wrote:
> On 31/10/2019 23:30, Justin Azoff wrote:> 
> https://github.com/J-Gras/intel-seen-more
> 
> Looks like this hits the issue where it depends on bro/something
> which are now all zeek/something  I fixed this in one of my packages
> by just deleting the "bro/" part of the dependency but I think this
> is more a migration issue that zkg could help resolve.
> The issue is already fixed in master since June. I just forgot to tag
> a new release.
> Well, 2 out of the 3 already work, and one just needs a minor update
> that I'm sure Jan would be happy to make if someone had just told him
> about it.
> True :)
> Jan_______________________________________________Zeek mailing
> listzeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/87c75729/attachment.html 

From austin522 at gmail.com  Wed Nov  6 04:09:33 2019
From: austin522 at gmail.com (venkatesh bandari)
Date: Wed, 6 Nov 2019 20:09:33 +0800
Subject: [Zeek] zeek worker nodes
Message-ID: <CAKPzng8Q-1pD_Yw9TM8rti+Xh52a-78J01nS1Rzc-VDCXg9oZA@mail.gmail.com>

Hello Team,

could you please share the details if anyone of you deployed worker nodes
.a little background of what iam doing
         1.installed zeek
          2.we are seeing 40% packet loss on interface
          3.installed  pf_ring but i was not able to  do the below step as
i already installed zeek(not compiled from source).It is a yum install
                              ldd /usr/local/bro/bin/bro | grep pcap
libpcap.so.1 => /opt/pfring/lib/libpcap.so.1

           4.i went a ahead and spawn 2 worker nodes
                          [worker-1]
type=worker
host=localhost
interface=eth0
lb_method=pf_ring
lb_procs=10
pin_cpus=2,3,4,5,6,7,8,9,10,11

[worker-2]
type=worker
host=localhost
interface=eth0
lb_method=pf_ring
lb_procs=10
pin_cpus=12,13,14,15,16,17,18,19,20,21
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/fa60fa0d/attachment.html 

From austin522 at gmail.com  Wed Nov  6 04:10:55 2019
From: austin522 at gmail.com (venkatesh bandari)
Date: Wed, 6 Nov 2019 20:10:55 +0800
Subject: [Zeek] zeek worker nodes
In-Reply-To: <CAKPzng8Q-1pD_Yw9TM8rti+Xh52a-78J01nS1Rzc-VDCXg9oZA@mail.gmail.com>
References: <CAKPzng8Q-1pD_Yw9TM8rti+Xh52a-78J01nS1Rzc-VDCXg9oZA@mail.gmail.com>
Message-ID: <CAKPzng_1UbbQmPsD-H-X4GVN+oFbDvnYdyXZA5FMkeTTXprJRA@mail.gmail.com>

Hello Team,

could you please share the details if anyone of you deployed worker nodes
.a little background of what iam doing
         1.installed zeek
          2.we are seeing 40% packet loss on interface
          3.installed  pf_ring but i was not able to  do the below step as
i already installed zeek(not compiled from source).It is a yum install
                              ldd /usr/local/bro/bin/bro | grep pcap
libpcap.so.1 => /opt/pfring/lib/libpcap.so.1

           4.i went a ahead and spawn 2 worker nodes
                          [worker-1]
type=worker
host=localhost
interface=eth0
lb_method=pf_ring
lb_procs=10
pin_cpus=2,3,4,5,6,7,8,9,10,11

[worker-2]
type=worker
host=localhost
interface=eth0
lb_method=pf_ring
lb_procs=10
pin_cpus=12,13,14,15,16,17,18,19,20,21

5.iam still seeing packet drops after spinning 2 worker nodes

my question is did any one implement worker nodes and saw any improvements

any help is much appreciated

thanks
Venkatesh


On Wed, Nov 6, 2019 at 8:09 PM venkatesh bandari <austin522 at gmail.com>
wrote:

> Hello Team,
>
> could you please share the details if anyone of you deployed worker nodes
> .a little background of what iam doing
>          1.installed zeek
>           2.we are seeing 40% packet loss on interface
>           3.installed  pf_ring but i was not able to  do the below step as
> i already installed zeek(not compiled from source).It is a yum install
>                               ldd /usr/local/bro/bin/bro | grep pcap
> libpcap.so.1 => /opt/pfring/lib/libpcap.so.1
>
>            4.i went a ahead and spawn 2 worker nodes
>                           [worker-1]
> type=worker
> host=localhost
> interface=eth0
> lb_method=pf_ring
> lb_procs=10
> pin_cpus=2,3,4,5,6,7,8,9,10,11
>
> [worker-2]
> type=worker
> host=localhost
> interface=eth0
> lb_method=pf_ring
> lb_procs=10
> pin_cpus=12,13,14,15,16,17,18,19,20,21
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/ea51b7ab/attachment-0001.html 

From mkg at vt.edu  Wed Nov  6 05:03:14 2019
From: mkg at vt.edu (Mark Gardner)
Date: Wed, 6 Nov 2019 08:03:14 -0500
Subject: [Zeek] Worker being "killed nohup"
Message-ID: <CAG=_CgA4txhcGmfppTnkOwU8vKWjoEkEfLNTrOtPeLvC6dXxNg@mail.gmail.com>

I'm running Zeek 2.6.4 and I have been seeing occasional error messages of
the form:

run-bro: line 110: 42574 Killed                  nohup ${pin_command}
$pin_cpu "$mybro" "$@"

The workers seem to be restarted fine and other than the error message, I
haven't noticed any ill behavior. What should I do about the error messages?

Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/134fcf31/attachment.html 

From mkg at vt.edu  Wed Nov  6 05:13:55 2019
From: mkg at vt.edu (Mark Gardner)
Date: Wed, 6 Nov 2019 08:13:55 -0500
Subject: [Zeek] DNS logs Anonymization
In-Reply-To: <CACSt-sxG7w8a5xcO_v-TwOFWfCEiGoYmBStgSSZKu+27vMrTzg@mail.gmail.com>
References: <CACSt-sxG7w8a5xcO_v-TwOFWfCEiGoYmBStgSSZKu+27vMrTzg@mail.gmail.com>
Message-ID: <CAG=_CgDeAi9dh88e=VT3K18QaohT8hR4ikHei7kiE-nFU+Zg8A@mail.gmail.com>

On Wed, Nov 6, 2019 at 1:23 AM Muhammad Abdullah <
14beemabdullah at seecs.edu.pk> wrote:

> I've just started using Zeek to collect DNS logs by running it from the
> CLI. I want to anonymize the IPs contained in these logs. How should I go
> about it?
>

We wrote a C++ program that uses the CryptoPANT library (
https://ant.isi.edu/software/cryptopANT/index.html). We first considered
using CryptoPAN (
https://www.cc.gatech.edu/computing/Telecomm/projects/cryptopan/ ; seems to
be unavailable at the moment) but found the other library much easier to
use and it is actively maintained.

Mark
-- 
Mark Gardner
--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/9ad1b46e/attachment.html 

From hkshin98 at gmail.com  Wed Nov  6 06:31:12 2019
From: hkshin98 at gmail.com (Raphael Shin)
Date: Wed, 6 Nov 2019 23:31:12 +0900
Subject: [Zeek] zeekweek 2019 materials
Message-ID: <CAC41-JUPkvO0p1AtvcOQLVLA_V8cV17M-UbMbRAizCaoHM3oHw@mail.gmail.com>

Hi.

I wanted to attend zeekweek 2019 but could not attend it.

Where can I get the presentation materials of zeekweek 2019?

Thanks,

Raphael.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/c1f49ab3/attachment.html 

From akgraner at corelight.com  Wed Nov  6 06:46:35 2019
From: akgraner at corelight.com (Amber Graner)
Date: Wed, 6 Nov 2019 09:46:35 -0500
Subject: [Zeek] zeekweek 2019 materials
In-Reply-To: <CAC41-JUPkvO0p1AtvcOQLVLA_V8cV17M-UbMbRAizCaoHM3oHw@mail.gmail.com>
References: <CAC41-JUPkvO0p1AtvcOQLVLA_V8cV17M-UbMbRAizCaoHM3oHw@mail.gmail.com>
Message-ID: <CAJhOzupW5kpfK_EZ9y4-nkLUSbqBYmsZbZ_a4n36u7VsLJfcDQ@mail.gmail.com>

Hi Raphael.

We'll be making the slides public this week and the videos will follow.

Thanks,
~Amber

On Wed, Nov 6, 2019 at 9:35 AM Raphael Shin <hkshin98 at gmail.com> wrote:

> Hi.
>
> I wanted to attend zeekweek 2019 but could not attend it.
>
> Where can I get the presentation materials of zeekweek 2019?
>
> Thanks,
>
> Raphael.
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 

*Amber Graner*
Director of Community
Corelight, Inc

828.582.9469



 * Ask me about how you can participate in the Zeek (formerly Bro)
community.
 * Remember - ZEEK AND YOU SHALL FIND!!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/f190e235/attachment.html 

From akgraner at corelight.com  Wed Nov  6 07:18:55 2019
From: akgraner at corelight.com (Amber Graner)
Date: Wed, 6 Nov 2019 10:18:55 -0500
Subject: [Zeek] Results of the Slack Poll and Next Steps
Message-ID: <CAJhOzuowTRmWKUBfZeoOjGkx0x5ymRY_kmZ73hcseZq0_MY0wg@mail.gmail.com>

Hi all,

I wanted to give you all an update from the slack poll results and say,
"THANK YOU" to  everyone who took time to participate.

As most of you are aware once the poll started several folks came forward
with an alternate suggestions, because of that, there will be a series of
presentations on Slack, Matrix, and Discourse.  (I'll go back through the
email thread and make sure I haven't missed another solution.)

We'll do this in a public video call that will be recorded and shared for
those who can't watch in real time.  Then we'll have one more poll.  This
is to ensure that everyone has a voice and gets a chance to present their
ideas to the community.

We're working out the date for the public presentations and will post that
as soon as possible.

For those who answered "yes' to question 3 and shared your information on
the Slack Poll, I'll be following up with you all next week.

Thank you so much for patience and input as we work through some new
processes and new opportunities to participate.

Below are the results of the Slack only poll:

======Begin Results=========
Slack Poll Results: 92 responses:

Q1: Would you use an Official Zeek Community Slack Channel?
Answers: Skipped: 1
                     Yes: 73
*                      No: 18*
                   Total: 92

Q2: Would having an Official Zeek Community Slack Channel enable or
encourage more Zeek Community participation?
Answers: Skipped: 1
                     Yes: 75
*                      No: 16*
                   Total: 92

Q3: If an Official Zeek Community Slack channel was made available would
you be willing to be an admin for the channel?
Answers: Skipped: 1
                     Yes: 23
*                      No: 68*
                   Total: 92

Q4: If you answered, yes, to being willing to admin an Official Zeek
Community Slack Channel, please give your contact information.  Amber
Graner, Zeek Community Director will be reaching out.
Answers: Skipped: 68
*             Answered: 24*
                     Total: 92

======End Results=========

Feel free to reach out if you have any questions.

Thanks,
~Amber

-- 

*Amber Graner*
Director of Community
Corelight, Inc

828.582.9469



 * Ask me about how you can participate in the Zeek (formerly Bro)
community.
 * Remember - ZEEK AND YOU SHALL FIND!!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/ffbff153/attachment-0001.html 

From jsbarber60 at gmail.com  Wed Nov  6 10:13:33 2019
From: jsbarber60 at gmail.com (Jeff Barber)
Date: Wed, 6 Nov 2019 13:13:33 -0500
Subject: [Zeek] Worker being "killed nohup"
In-Reply-To: <CAG=_CgA4txhcGmfppTnkOwU8vKWjoEkEfLNTrOtPeLvC6dXxNg@mail.gmail.com>
References: <CAG=_CgA4txhcGmfppTnkOwU8vKWjoEkEfLNTrOtPeLvC6dXxNg@mail.gmail.com>
Message-ID: <CABLqKGLvnu_7FH-6qa3oxOq1ehctX-4VW1n2aX=DffseAiix9w@mail.gmail.com>

> I'm running Zeek 2.6.4 and I have been seeing occasional error messages
of the form:
>
> run-bro: line 110: 42574 Killed                  nohup ${pin_command}
$pin_cpu "$mybro" "$@"
>
> The workers seem to be restarted fine and other than the error message, I
haven't noticed any ill behavior. What should I do about the error messages?

I would check your syslog. Assuming you are running linux, if your system
runs out of memory, the kernel will go find the biggest process and kill
it. This can often be a zeek process as they tend to grow large as more
connections are tracked (depending on many factors: what scripts you are
running, what you're logging, what kind of traffic is being seen, etc.). If
that's happening, you should see something in syslog containing the
string "invoked
oom-killer:" If you look at the surrounding lines, there should be some
info on process sizes showing why it was selected.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/d7bb0b8b/attachment.html 

From mkg at vt.edu  Wed Nov  6 11:10:06 2019
From: mkg at vt.edu (Mark Gardner)
Date: Wed, 6 Nov 2019 14:10:06 -0500
Subject: [Zeek] Worker being "killed nohup"
In-Reply-To: <CABLqKGLvnu_7FH-6qa3oxOq1ehctX-4VW1n2aX=DffseAiix9w@mail.gmail.com>
References: <CAG=_CgA4txhcGmfppTnkOwU8vKWjoEkEfLNTrOtPeLvC6dXxNg@mail.gmail.com>
	<CABLqKGLvnu_7FH-6qa3oxOq1ehctX-4VW1n2aX=DffseAiix9w@mail.gmail.com>
Message-ID: <CAG=_CgDOL1QabA83NuQTjjyDpPndcCf7wquCOrBMJuTd75JMrg@mail.gmail.com>

On Wed, Nov 6, 2019 at 1:13 PM Jeff Barber <jsbarber60 at gmail.com> wrote:

> > I'm running Zeek 2.6.4 and I have been seeing occasional error messages
> of the form:
> >
> > run-bro: line 110: 42574 Killed                  nohup ${pin_command}
> $pin_cpu "$mybro" "$@"
> >
> > The workers seem to be restarted fine and other than the error message,
> I haven't noticed any ill behavior. What should I do about the error
> messages?
>
> I would check your syslog. Assuming you are running linux, if your system
> runs out of memory, the kernel will go find the biggest process and kill
> it. This can often be a zeek process as they tend to grow large as more
> connections are tracked (depending on many factors: what scripts you are
> running, what you're logging, what kind of traffic is being seen, etc.). If
> that's happening, you should see something in syslog containing the string "invoked
> oom-killer:" If you look at the surrounding lines, there should be some
> info on process sizes showing why it was selected.
>

Good catch Jeff.  A crash email arrived at 10:10 and the journal entry
starting around 10:07 contained the following:

Nov 06 10:07:08 sensor1 kernel: oom_reaper: reaped process 43437 (bro), now
anon-rss:0kB, file-rss:1048576kB, shmem-rss:0kB
Nov 06 10:07:05 sensor1 kernel: Killed process 43437 (bro)
total-vm:67116888kB, anon-rss:57339476kB, file-rss:1042296kB, shmem-rss:0kB
Nov 06 10:07:05 sensor1 kernel: Out of memory: Kill process 43437 (bro)
score 444 or sacrifice child
Nov 06 10:07:05 sensor1 kernel: [  54824]    47 54824  1516502  1221530
11284480       80             0 bro
Nov 06 10:07:05 sensor1 kernel: [  54817]    47 54817     1661       56
 49152       16             0 run-bro
Nov 06 10:07:05 sensor1 kernel: [  51543]    47 51543  1516502  1281056
11886592     3616             0 bro
Nov 06 10:07:05 sensor1 kernel: [  51542]    47 51542  1516502  1285841
11964416     2296             0 bro
Nov 06 10:07:05 sensor1 kernel: [  51539]    47 51539  1516502  1277957
11890688     2817             0 bro
Nov 06 10:07:05 sensor1 kernel: [  51537]    47 51537  1516502  1282107
11829248     2543             0 bro
Nov 06 10:07:05 sensor1 kernel: [  51533]    47 51533  1516502  1276417
11939840     7591             0 bro
Nov 06 10:07:05 sensor1 kernel: [  51532]    47 51532  1516502  1290264
12042240     8006             0 bro
Nov 06 10:07:05 sensor1 kernel: [  51523]    47 51523  1516502  1281160
11894784     7479             0 bro
Nov 06 10:07:05 sensor1 kernel: [  51521]    47 51521  1516502  1285423
11984896     7957             0 bro
Nov 06 10:07:05 sensor1 kernel: [  51520]    47 51520  1745878  1370574
12668928     6908             0 bro
Nov 06 10:07:05 sensor1 kernel: [  51501]    47 51501  1516502  1286290
11788288     6967             0 bro
Nov 06 10:07:05 sensor1 kernel: [  51499]    47 51499     1661       22
 49152       49             0 run-bro
Nov 06 10:07:05 sensor1 kernel: [  51498]    47 51498  1516502  1276891
11882496     7233             0 bro
Nov 06 10:07:05 sensor1 kernel: [  51496]    47 51496  1516502  1287064
11948032     7685             0 bro
Nov 06 10:07:05 sensor1 kernel: [  51491]    47 51491     1661       18
 53248       55             0 run-bro
Nov 06 10:07:05 sensor1 kernel: [  51487]    47 51487     1661       18
 57344       54             0 run-bro
Nov 06 10:07:05 sensor1 kernel: [  51474]    47 51474     1661       21
 49152       52             0 run-bro
Nov 06 10:07:05 sensor1 kernel: [  51471]    47 51471  1516502  1277820
11890688     7964             0 bro
Nov 06 10:07:05 sensor1 kernel: [  51465]    47 51465     1661       20
 53248       53             0 run-bro
Nov 06 10:07:05 sensor1 kernel: [  51455]    47 51455     1661       23
 57344       49             0 run-bro
Nov 06 10:07:05 sensor1 kernel: [  51440]    47 51440     1661       18
 49152       54             0 run-bro
Nov 06 10:07:05 sensor1 kernel: [  51438]    47 51438     1661       23
 49152       49             0 run-bro
Nov 06 10:07:05 sensor1 kernel: [  51428]    47 51428     1661       18
 49152       54             0 run-bro
Nov 06 10:07:05 sensor1 kernel: [  51414]    47 51414     1661       16
 53248       56             0 run-bro
Nov 06 10:07:05 sensor1 kernel: [  51405]    47 51405     1661       20
 53248       52             0 run-bro
Nov 06 10:07:05 sensor1 kernel: [  51397]    47 51397     1661       14
 45056       59             0 run-bro
Nov 06 10:07:05 sensor1 kernel: [  51375]    47 51375     1661       15
 49152       58             0 run-bro
Nov 06 10:07:05 sensor1 kernel: [  43437]    47 43437 16779222 14595446
122212352   129972             0 bro
Nov 06 10:07:05 sensor1 kernel: [  43343]    47 43343     1661        2
 49152       70             0 run-bro
Nov 06 10:07:05 sensor1 kernel: [  43183]    47 43183    42625        4
106496      608             0 (sd-pam)
Nov 06 10:07:05 sensor1 kernel: [  43181]    47 43181     5290        0
 77824      342             0 systemd
Nov 06 10:07:05 sensor1 kernel: [    996]     0   996     1346        0
 49152       33             0 agetty
Nov 06 10:07:05 sensor1 kernel: [    995]     0   995     1403        0
 45056       32             0 agetty
Nov 06 10:07:05 sensor1 kernel: [    994]     0   994     3963       38
 69632      178         -1000 sshd
Nov 06 10:07:05 sensor1 kernel: [    991]     0   991     2119       20
 53248       35             0 cron
Nov 06 10:07:05 sensor1 kernel: [    990]   105   990     2210       97
 57344       55          -900 dbus-daemon
Nov 06 10:07:05 sensor1 kernel: [    977]     0   977    56456       90
 86016      378             0 rsyslogd
Nov 06 10:07:05 sensor1 kernel: [    968]     0   968     4875       78
 77824      189             0 systemd-logind
Nov 06 10:07:05 sensor1 kernel: [    944]     0   944     2021        1
 57344      799             0 haveged
Nov 06 10:07:05 sensor1 kernel: [    931]   100   931    23270       26
 86016      207             0 systemd-timesyn
Nov 06 10:07:05 sensor1 kernel: [    640]     0   640     5579       14
 69632      309         -1000 systemd-udevd
Nov 06 10:07:05 sensor1 kernel: [    615]     0   615    74225     2624
638976      568             0 systemd-journal
Nov 06 10:07:05 sensor1 kernel: [  pid  ]   uid  tgid total_vm      rss
pgtables_bytes swapents oom_score_adj name
Nov 06 10:07:05 sensor1 kernel: Tasks state (memory values in pages):
Nov 06 10:07:05 sensor1 kernel: 0 pages hwpoisoned
Nov 06 10:07:05 sensor1 kernel: 551979 pages reserved
Nov 06 10:07:05 sensor1 kernel: 0 pages HighMem/MovableOnly
Nov 06 10:07:05 sensor1 kernel: 33528566 pages RAM
Nov 06 10:07:05 sensor1 kernel: Total swap = 976892kB
Nov 06 10:07:05 sensor1 kernel: Free swap  = 0kB
Nov 06 10:07:05 sensor1 kernel: Swap cache stats: add 759161, delete
711025, find 25951/48832
Nov 06 10:07:05 sensor1 kernel: 48128 pages in swap cache
Nov 06 10:07:05 sensor1 kernel: 52774 total pagecache pages
Nov 06 10:07:05 sensor1 kernel: Node 3 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
Nov 06 10:07:05 sensor1 kernel: Node 3 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=1048576kB
Nov 06 10:07:05 sensor1 kernel: Node 1 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
Nov 06 10:07:05 sensor1 kernel: Node 1 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=1048576kB
Nov 06 10:07:05 sensor1 kernel: Node 3 Normal: 73*4kB (U) 150*8kB (UH)
59*16kB (UH) 1325*32kB (UH) 5*64kB (H) 2*128kB (H) 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 45412kB
Nov 06 10:07:05 sensor1 kernel: Node 1 Normal: 78*4kB (UMEH) 213*8kB (UEH)
70*16kB (UH) 1232*32kB (UMH) 7*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 43008kB
Nov 06 10:07:05 sensor1 kernel: Node 1 DMA32: 697*4kB (UME) 647*8kB (UME)
392*16kB (UME) 190*32kB (UME) 94*64kB (UE) 51*128kB (UE) 63*256kB (UME)
240*512kB (UME) 72*1024kB (UME) 0*2048kB 0*4096kB = 245596kB
Nov 06 10:07:05 sensor1 kernel: Node 1 DMA: 1*4kB (U) 1*8kB (U) 2*16kB (U)
1*32kB (U) 3*64kB (U) 0*128kB 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M)
3*4096kB (M) = 15884kB
Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 06 10:07:05 sensor1 kernel: Node 3 Normal free:44844kB min:45116kB
low:111144kB high:177172kB active_anon:48629496kB inactive_anon:3250728kB
active_file:64kB inactive_file:668kB unevictable:0kB writepending:0kB
present:67094528kB managed:66031300kB mlocked:0kB kernel_stack:5352kB
pagetables:146976kB bounce
Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 06 10:07:05 sensor1 kernel: Node 1 Normal free:41932kB min:42604kB
low:104956kB high:167308kB active_anon:55633904kB inactive_anon:4082748kB
active_file:560kB inactive_file:0kB unevictable:0kB writepending:0kB
present:63438848kB managed:62359848kB mlocked:0kB kernel_stack:4776kB
pagetables:133008kB bounce:
Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 0 60894 60894 60894
Nov 06 10:07:05 sensor1 kernel: Node 1 DMA32 free:245356kB min:2372kB
low:5844kB high:9316kB active_anon:3199852kB inactive_anon:44556kB
active_file:100kB inactive_file:0kB unevictable:0kB writepending:0kB
present:3564892kB managed:3499316kB mlocked:0kB kernel_stack:48kB
pagetables:3092kB bounce:0kB free_pcp:4
Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 3392 64286 64286 64286
Nov 06 10:07:05 sensor1 kernel: Node 1 DMA free:15884kB min:8kB low:20kB
high:32kB active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB
managed:15884kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB
free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 06 10:07:05 sensor1 kernel: Node 3 active_anon:48629496kB
inactive_anon:3250728kB active_file:108kB inactive_file:700kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:13636088kB
dirty:0kB writeback:0kB shmem:9996kB shmem_thp: 0kB shmem_pmdmapped: 0kB
anon_thp: 1337344kB writeback_tmp:0kB unst
Nov 06 10:07:05 sensor1 kernel: Node 1 active_anon:58833756kB
inactive_anon:4127304kB active_file:548kB inactive_file:0kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB mapped:2101152kB dirty:0kB
writeback:0kB shmem:7828kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp:
13608960kB writeback_tmp:0kB unstab
Nov 06 10:07:05 sensor1 kernel: active_anon:26865813 inactive_anon:1844508
isolated_anon:0
                                 active_file:163 inactive_file:0
isolated_file:0
                                 unevictable:0 dirty:0 writeback:0
unstable:0
                                 slab_reclaimable:25057
slab_unreclaimable:65745
                                 mapped:3934289 shmem:4456 pagetables:70769
bounce:0
                                 free:87073 free_pcp:3080 free_cma:0
Nov 06 10:07:05 sensor1 kernel: Mem-Info:
Nov 06 10:07:05 sensor1 kernel: R13: 0000000000001800 R14: 00007f375bd4f780
R15: 0000000000000022
Nov 06 10:07:05 sensor1 kernel: R10: 00007f3618971000 R11: 00007f36189727e0
R12: 00007f375bd4f940
Nov 06 10:07:05 sensor1 kernel: RBP: 00007f361424b000 R08: 0000000000000000
R09: 00007f36189727e0
Nov 06 10:07:05 sensor1 kernel: RDX: 0000000000001780 RSI: 00007f361424b000
RDI: 00007f3618971000
Nov 06 10:07:05 sensor1 kernel: RAX: 00007f3618971000 RBX: 0000000000001800
RCX: 00007f361424c760
Nov 06 10:07:05 sensor1 kernel: RSP: 002b:00007ffd609fbc58 EFLAGS: 00010202
Nov 06 10:07:05 sensor1 kernel: Code: Bad RIP value.
Nov 06 10:07:05 sensor1 kernel: RIP: 0033:0x7f375bf1f925
Nov 06 10:07:05 sensor1 kernel:  page_fault+0x1e/0x30
Nov 06 10:07:05 sensor1 kernel:  ? page_fault+0x8/0x30
Nov 06 10:07:05 sensor1 kernel:  __do_page_fault+0x249/0x4f0
Nov 06 10:07:05 sensor1 kernel:  handle_mm_fault+0xd6/0x200
Nov 06 10:07:05 sensor1 kernel:  ? __switch_to_asm+0x41/0x70
Nov 06 10:07:05 sensor1 kernel:  __handle_mm_fault+0x958/0x1270
Nov 06 10:07:05 sensor1 kernel:  alloc_pages_vma+0x74/0x1c0
Nov 06 10:07:05 sensor1 kernel:  __alloc_pages_nodemask+0x28b/0x2b0
Nov 06 10:07:05 sensor1 kernel:  __alloc_pages_slowpath+0xbd8/0xcb0
Nov 06 10:07:05 sensor1 kernel:  out_of_memory+0x1a5/0x430
Nov 06 10:07:05 sensor1 kernel:  oom_kill_process.cold.30+0xb/0x1cf
Nov 06 10:07:05 sensor1 kernel:  dump_header+0x6b/0x283
Nov 06 10:07:05 sensor1 kernel:  dump_stack+0x5c/0x80
Nov 06 10:07:05 sensor1 kernel: Call Trace:
Nov 06 10:07:05 sensor1 kernel: Hardware name: Supermicro Super
Server/H11SSL-i, BIOS 1.0b 04/27/2018
Nov 06 10:07:05 sensor1 kernel: CPU: 3 PID: 51496 Comm: bro Not tainted
4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u1
Nov 06 10:07:05 sensor1 kernel: bro cpuset=/ mems_allowed=1,3
Nov 06 10:07:05 sensor1 kernel: bro invoked oom-killer:
gfp_mask=0x6280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null),
order=0, oom_score_adj=0

It looks like bro was using 64 GB of RAM (summation of even though the
system has 128 GB installed. (It also looks like there was no free swap.)
Searching the Internet for ways to solve OOM, people suggested turning off
over-commit, and configuring the overcommit ratio a bit higher. Currently,
the values are:

# sysctl vm.overcommit_memory
vm.overcommit_memory = 0
# sysctl vm.overcommit_ratio
vm.overcommit_ratio = 50

In one case the suggestion was to change those to vm.overcommit_memory = 1
and vm.overcommit_ratio = 80 which seems reasonable based on the
description at
https://www.kernel.org/doc/Documentation/vm/overcommit-accounting.

Or is there a better way to handle OOM?

Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/0410889d/attachment-0001.html 

From pumphrey.adam at gmail.com  Wed Nov  6 11:41:47 2019
From: pumphrey.adam at gmail.com (ap)
Date: Wed, 6 Nov 2019 14:41:47 -0500
Subject: [Zeek] DNS logs Anonymization
In-Reply-To: <CACSt-sxG7w8a5xcO_v-TwOFWfCEiGoYmBStgSSZKu+27vMrTzg@mail.gmail.com>
References: <CACSt-sxG7w8a5xcO_v-TwOFWfCEiGoYmBStgSSZKu+27vMrTzg@mail.gmail.com>
Message-ID: <6FD0E930-9A90-49B9-B147-6BFA249C285B@gmail.com>

Muhammad, 

There are multiple ways to go about it.  One consideration is whether or not you want to be able to map back to the original IP from the anonymized one.  Another consideration is if you intend to anonymize every IP in the log, or just specific IPs/subnets.  

You could post-process the logs like Mark suggested with something like CryptopANT.  In addition to the lib they provide an example binary called scramble_ips that might do what you need.  

You could also do this (sort of) in Zeek script within a DNS::log_dns event handler.  There is a BIF called remask_addr (https://docs.zeek.org/en/stable/scripts/base/bif/zeek.bif.zeek.html#id-remask_addr <https://docs.zeek.org/en/stable/scripts/base/bif/zeek.bif.zeek.html#id-remask_addr>) that allows you take subnet bits from one address and host bits of another and combine them to create a new address.   Here?s an example of its usage:  http://try.zeek.org/#/tryzeek/saved/364768 <http://try.zeek.org/#/tryzeek/saved/364768>.  I say "sort of" because the original IP?s aren?t truly anonymized, they are mapped into a new subnet.  But that might obscure the real addresses enough - it depends on your requirements.  

Adam 

> On Nov 6, 2019, at 1:13 AM, Muhammad Abdullah <14beemabdullah at seecs.edu.pk> wrote:
> 
> Hi,
> 
> I've just started using Zeek to collect DNS logs by running it from the CLI. I want to anonymize the IPs contained in these logs. How should I go about it? 
> 
> Thanks,
> 
> Muhammad.
>  _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/66fc678c/attachment.html 

From konrad.weglowski at gmail.com  Wed Nov  6 13:14:39 2019
From: konrad.weglowski at gmail.com (Konrad Weglowski)
Date: Wed, 6 Nov 2019 16:14:39 -0500
Subject: [Zeek] monitoring proxied web traffic
Message-ID: <CA+fGMJRb6kDVcn-1BFZ=hgwLfvM4MWaKdw=jR_BkrB=ymQz8DQ@mail.gmail.com>

Hello,

I need to monitor web traffic with Zeek that is going through an implicit
web proxy. I would like to be able to see real client IPs in which case tap
would be placed before the proxy, however I will not be able to see the
real destination IP as it will always be proxy IP. I can also tap on the
other side of the proxy where source will always be proxy IP at the same
time.

Is there a way with Zeek to correlate the two sessions (before and after
proxy) somehow?

I realize that UID or Community ID are probably not going do it since
different source/destinations pairs for each for each session, but maybe
based on the timestamp/URI/SNI/byte or packet counts/etc?

I was also thinking "X-Forwarded-For" header but I guess that would only
work for HTTP and not HTTPS connections.

Thank You,

Konrad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/538eff1f/attachment.html 

From michalpurzynski1 at gmail.com  Wed Nov  6 15:41:28 2019
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Wed, 6 Nov 2019 15:41:28 -0800
Subject: [Zeek] pf_ring
In-Reply-To: <CAKPzng-Qa_B-6r1zL1kfxSaFT0ECar9tcKVXLY4hu8CtCBTO4g@mail.gmail.com>
References: <CAKPzng-Qa_B-6r1zL1kfxSaFT0ECar9tcKVXLY4hu8CtCBTO4g@mail.gmail.com>
Message-ID: <CAJ6bFK0C=uDp2KZte0K-uyi06nx9tnvqvuwcPSaTJfQ+JNRhrw@mail.gmail.com>

Why not give AF_Packet a try? We have an official zpkg package and a
plugin, with some instructions here
https://github.com/J-Gras/bro-af_packet-plugin

On Tue, Nov 5, 2019 at 8:05 PM venkatesh bandari <austin522 at gmail.com>
wrote:

> Hello Team,
>
> i installed zeek and using standard libcap
> ldd /opt/zeek/bin/zeek | grep pcap
>         libpcap.so.1 => /lib64/libpcap.so.1 (0x00007f1bad618000)
>
> iam seeing the box is dropping 40% packets.iam planning to have 2 more
> workers with pf_ring.Installed pf_ring
>
> in the link https://www.zeek.org/documentation/load-balancing.html
> it says zeek to point to below libcap.how doi change the to pfring sware
> libcap
>
> ldd /usr/local/bro/bin/bro | grep pcap
>         libpcap.so.1 => /opt/pfring/lib/libpcap.so.1
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/7647a60a/attachment.html 

From michalpurzynski1 at gmail.com  Wed Nov  6 15:45:36 2019
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Wed, 6 Nov 2019 15:45:36 -0800
Subject: [Zeek] uid in files logs
In-Reply-To: <b474e395ba664b5fb7a6072d42ed6c18@SRVEX03.aizoon.local>
References: <b474e395ba664b5fb7a6072d42ed6c18@SRVEX03.aizoon.local>
Message-ID: <CAJ6bFK0A50BBxToy0dqqc85=XmyR-BrjSFDP6EWguQ-+PpL=wg@mail.gmail.com>

While I have no idea why it's not default, I'll share a piece of code to
achieve something similar, so you can adopt it to your needs

Here we wanted to kill logging X509 certificates into both files.log and
x509.log - and by doing that we saved like 20% of our SIEM intake, globally
(!!). Should be easy enough to extend x509.log to include data from
conn.log, etc.

@load base/frameworks/files
@load base/files/hash

module X509;

export {
    redef record X509::Info += {
        fuid: string &log &optional;
        md5: string &log &optional;
    };
}

event file_state_remove(f: fa_file) &priority=40
{
    if ( ! f$info?$x509 )
        return;

    f$info$x509$fuid = f$info$fuid;
    f$info$x509$md5 = f$info$md5;
}

On Wed, Nov 6, 2019 at 2:17 AM Palumbo Mauro <mauro.palumbo at aizoon.it>
wrote:

> Hi everybody,
>
>    it would be useful for us to have the conn uids in the logs from file
> analyzers (pe.log, x509.log,?). I know this information can be gathered by
> cross-cehcking different bro logs, but it will save some time to have it
> already in pe.log, etc. I believe this data is available in the record
> fa_file.conns, available in events in the file framework, so it seems not
> difficult to add.
>
> Is there any reason why it is not added by default?
>
>
>
> Thanks,
>
> Mauro
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/93a217a9/attachment.html 

From michalpurzynski1 at gmail.com  Wed Nov  6 15:50:47 2019
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Wed, 6 Nov 2019 15:50:47 -0800
Subject: [Zeek] Worker being "killed nohup"
In-Reply-To: <CAG=_CgDOL1QabA83NuQTjjyDpPndcCf7wquCOrBMJuTd75JMrg@mail.gmail.com>
References: <CAG=_CgA4txhcGmfppTnkOwU8vKWjoEkEfLNTrOtPeLvC6dXxNg@mail.gmail.com>
	<CABLqKGLvnu_7FH-6qa3oxOq1ehctX-4VW1n2aX=DffseAiix9w@mail.gmail.com>
	<CAG=_CgDOL1QabA83NuQTjjyDpPndcCf7wquCOrBMJuTd75JMrg@mail.gmail.com>
Message-ID: <CAJ6bFK2FZWODtTeUk1UuCJ-QxKED7Z0ckRA2Xxb_U3AthtoJtQ@mail.gmail.com>

Let's dig into the root cause.
Your OOM most likely happens because either
- you have tons of state and hit something that Justin neatly described as
"unbound state growth"
- or your logger threads cannot write messages quickly enough

Given the OOM output it looks to me like workers are to blame. Do you have
any custom scripts? Do you use the stock scan.bro or any other SumStats
scripts?

Can you observe the system for a while, in something like htop, or top with
threads, to see what kind of processes eat most of memory and report back?


On Wed, Nov 6, 2019 at 11:13 AM Mark Gardner <mkg at vt.edu> wrote:

> On Wed, Nov 6, 2019 at 1:13 PM Jeff Barber <jsbarber60 at gmail.com> wrote:
>
>> > I'm running Zeek 2.6.4 and I have been seeing occasional error
>> messages of the form:
>> >
>> > run-bro: line 110: 42574 Killed                  nohup ${pin_command}
>> $pin_cpu "$mybro" "$@"
>> >
>> > The workers seem to be restarted fine and other than the error
>> message, I haven't noticed any ill behavior. What should I do about the
>> error messages?
>>
>> I would check your syslog. Assuming you are running linux, if your
>> system runs out of memory, the kernel will go find the biggest process
>> and kill it. This can often be a zeek process as they tend to grow large as
>> more connections are tracked (depending on many factors: what scripts you
>> are running, what you're logging, what kind of traffic is being seen,
>> etc.). If that's happening, you should see something in syslog containing
>> the string "invoked oom-killer:" If you look at the surrounding lines,
>> there should be some info on process sizes showing why it was selected.
>>
>
> Good catch Jeff.  A crash email arrived at 10:10 and the journal entry
> starting around 10:07 contained the following:
>
> Nov 06 10:07:08 sensor1 kernel: oom_reaper: reaped process 43437 (bro),
> now anon-rss:0kB, file-rss:1048576kB, shmem-rss:0kB
> Nov 06 10:07:05 sensor1 kernel: Killed process 43437 (bro)
> total-vm:67116888kB, anon-rss:57339476kB, file-rss:1042296kB, shmem-rss:0kB
> Nov 06 10:07:05 sensor1 kernel: Out of memory: Kill process 43437 (bro)
> score 444 or sacrifice child
> Nov 06 10:07:05 sensor1 kernel: [  54824]    47 54824  1516502  1221530
> 11284480       80             0 bro
> Nov 06 10:07:05 sensor1 kernel: [  54817]    47 54817     1661       56
>  49152       16             0 run-bro
> Nov 06 10:07:05 sensor1 kernel: [  51543]    47 51543  1516502  1281056
> 11886592     3616             0 bro
> Nov 06 10:07:05 sensor1 kernel: [  51542]    47 51542  1516502  1285841
> 11964416     2296             0 bro
> Nov 06 10:07:05 sensor1 kernel: [  51539]    47 51539  1516502  1277957
> 11890688     2817             0 bro
> Nov 06 10:07:05 sensor1 kernel: [  51537]    47 51537  1516502  1282107
> 11829248     2543             0 bro
> Nov 06 10:07:05 sensor1 kernel: [  51533]    47 51533  1516502  1276417
> 11939840     7591             0 bro
> Nov 06 10:07:05 sensor1 kernel: [  51532]    47 51532  1516502  1290264
> 12042240     8006             0 bro
> Nov 06 10:07:05 sensor1 kernel: [  51523]    47 51523  1516502  1281160
> 11894784     7479             0 bro
> Nov 06 10:07:05 sensor1 kernel: [  51521]    47 51521  1516502  1285423
> 11984896     7957             0 bro
> Nov 06 10:07:05 sensor1 kernel: [  51520]    47 51520  1745878  1370574
> 12668928     6908             0 bro
> Nov 06 10:07:05 sensor1 kernel: [  51501]    47 51501  1516502  1286290
> 11788288     6967             0 bro
> Nov 06 10:07:05 sensor1 kernel: [  51499]    47 51499     1661       22
>  49152       49             0 run-bro
> Nov 06 10:07:05 sensor1 kernel: [  51498]    47 51498  1516502  1276891
> 11882496     7233             0 bro
> Nov 06 10:07:05 sensor1 kernel: [  51496]    47 51496  1516502  1287064
> 11948032     7685             0 bro
> Nov 06 10:07:05 sensor1 kernel: [  51491]    47 51491     1661       18
>  53248       55             0 run-bro
> Nov 06 10:07:05 sensor1 kernel: [  51487]    47 51487     1661       18
>  57344       54             0 run-bro
> Nov 06 10:07:05 sensor1 kernel: [  51474]    47 51474     1661       21
>  49152       52             0 run-bro
> Nov 06 10:07:05 sensor1 kernel: [  51471]    47 51471  1516502  1277820
> 11890688     7964             0 bro
> Nov 06 10:07:05 sensor1 kernel: [  51465]    47 51465     1661       20
>  53248       53             0 run-bro
> Nov 06 10:07:05 sensor1 kernel: [  51455]    47 51455     1661       23
>  57344       49             0 run-bro
> Nov 06 10:07:05 sensor1 kernel: [  51440]    47 51440     1661       18
>  49152       54             0 run-bro
> Nov 06 10:07:05 sensor1 kernel: [  51438]    47 51438     1661       23
>  49152       49             0 run-bro
> Nov 06 10:07:05 sensor1 kernel: [  51428]    47 51428     1661       18
>  49152       54             0 run-bro
> Nov 06 10:07:05 sensor1 kernel: [  51414]    47 51414     1661       16
>  53248       56             0 run-bro
> Nov 06 10:07:05 sensor1 kernel: [  51405]    47 51405     1661       20
>  53248       52             0 run-bro
> Nov 06 10:07:05 sensor1 kernel: [  51397]    47 51397     1661       14
>  45056       59             0 run-bro
> Nov 06 10:07:05 sensor1 kernel: [  51375]    47 51375     1661       15
>  49152       58             0 run-bro
> Nov 06 10:07:05 sensor1 kernel: [  43437]    47 43437 16779222 14595446
> 122212352   129972             0 bro
> Nov 06 10:07:05 sensor1 kernel: [  43343]    47 43343     1661        2
>  49152       70             0 run-bro
> Nov 06 10:07:05 sensor1 kernel: [  43183]    47 43183    42625        4
> 106496      608             0 (sd-pam)
> Nov 06 10:07:05 sensor1 kernel: [  43181]    47 43181     5290        0
>  77824      342             0 systemd
> Nov 06 10:07:05 sensor1 kernel: [    996]     0   996     1346        0
>  49152       33             0 agetty
> Nov 06 10:07:05 sensor1 kernel: [    995]     0   995     1403        0
>  45056       32             0 agetty
> Nov 06 10:07:05 sensor1 kernel: [    994]     0   994     3963       38
>  69632      178         -1000 sshd
> Nov 06 10:07:05 sensor1 kernel: [    991]     0   991     2119       20
>  53248       35             0 cron
> Nov 06 10:07:05 sensor1 kernel: [    990]   105   990     2210       97
>  57344       55          -900 dbus-daemon
> Nov 06 10:07:05 sensor1 kernel: [    977]     0   977    56456       90
>  86016      378             0 rsyslogd
> Nov 06 10:07:05 sensor1 kernel: [    968]     0   968     4875       78
>  77824      189             0 systemd-logind
> Nov 06 10:07:05 sensor1 kernel: [    944]     0   944     2021        1
>  57344      799             0 haveged
> Nov 06 10:07:05 sensor1 kernel: [    931]   100   931    23270       26
>  86016      207             0 systemd-timesyn
> Nov 06 10:07:05 sensor1 kernel: [    640]     0   640     5579       14
>  69632      309         -1000 systemd-udevd
> Nov 06 10:07:05 sensor1 kernel: [    615]     0   615    74225     2624
> 638976      568             0 systemd-journal
> Nov 06 10:07:05 sensor1 kernel: [  pid  ]   uid  tgid total_vm      rss
> pgtables_bytes swapents oom_score_adj name
> Nov 06 10:07:05 sensor1 kernel: Tasks state (memory values in pages):
> Nov 06 10:07:05 sensor1 kernel: 0 pages hwpoisoned
> Nov 06 10:07:05 sensor1 kernel: 551979 pages reserved
> Nov 06 10:07:05 sensor1 kernel: 0 pages HighMem/MovableOnly
> Nov 06 10:07:05 sensor1 kernel: 33528566 pages RAM
> Nov 06 10:07:05 sensor1 kernel: Total swap = 976892kB
> Nov 06 10:07:05 sensor1 kernel: Free swap  = 0kB
> Nov 06 10:07:05 sensor1 kernel: Swap cache stats: add 759161, delete
> 711025, find 25951/48832
> Nov 06 10:07:05 sensor1 kernel: 48128 pages in swap cache
> Nov 06 10:07:05 sensor1 kernel: 52774 total pagecache pages
> Nov 06 10:07:05 sensor1 kernel: Node 3 hugepages_total=0 hugepages_free=0
> hugepages_surp=0 hugepages_size=2048kB
> Nov 06 10:07:05 sensor1 kernel: Node 3 hugepages_total=0 hugepages_free=0
> hugepages_surp=0 hugepages_size=1048576kB
> Nov 06 10:07:05 sensor1 kernel: Node 1 hugepages_total=0 hugepages_free=0
> hugepages_surp=0 hugepages_size=2048kB
> Nov 06 10:07:05 sensor1 kernel: Node 1 hugepages_total=0 hugepages_free=0
> hugepages_surp=0 hugepages_size=1048576kB
> Nov 06 10:07:05 sensor1 kernel: Node 3 Normal: 73*4kB (U) 150*8kB (UH)
> 59*16kB (UH) 1325*32kB (UH) 5*64kB (H) 2*128kB (H) 0*256kB 0*512kB 0*1024kB
> 0*2048kB 0*4096kB = 45412kB
> Nov 06 10:07:05 sensor1 kernel: Node 1 Normal: 78*4kB (UMEH) 213*8kB (UEH)
> 70*16kB (UH) 1232*32kB (UMH) 7*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB
> 0*2048kB 0*4096kB = 43008kB
> Nov 06 10:07:05 sensor1 kernel: Node 1 DMA32: 697*4kB (UME) 647*8kB (UME)
> 392*16kB (UME) 190*32kB (UME) 94*64kB (UE) 51*128kB (UE) 63*256kB (UME)
> 240*512kB (UME) 72*1024kB (UME) 0*2048kB 0*4096kB = 245596kB
> Nov 06 10:07:05 sensor1 kernel: Node 1 DMA: 1*4kB (U) 1*8kB (U) 2*16kB (U)
> 1*32kB (U) 3*64kB (U) 0*128kB 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M)
> 3*4096kB (M) = 15884kB
> Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 0 0 0 0
> Nov 06 10:07:05 sensor1 kernel: Node 3 Normal free:44844kB min:45116kB
> low:111144kB high:177172kB active_anon:48629496kB inactive_anon:3250728kB
> active_file:64kB inactive_file:668kB unevictable:0kB writepending:0kB
> present:67094528kB managed:66031300kB mlocked:0kB kernel_stack:5352kB
> pagetables:146976kB bounce
> Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 0 0 0 0
> Nov 06 10:07:05 sensor1 kernel: Node 1 Normal free:41932kB min:42604kB
> low:104956kB high:167308kB active_anon:55633904kB inactive_anon:4082748kB
> active_file:560kB inactive_file:0kB unevictable:0kB writepending:0kB
> present:63438848kB managed:62359848kB mlocked:0kB kernel_stack:4776kB
> pagetables:133008kB bounce:
> Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 0 60894 60894 60894
> Nov 06 10:07:05 sensor1 kernel: Node 1 DMA32 free:245356kB min:2372kB
> low:5844kB high:9316kB active_anon:3199852kB inactive_anon:44556kB
> active_file:100kB inactive_file:0kB unevictable:0kB writepending:0kB
> present:3564892kB managed:3499316kB mlocked:0kB kernel_stack:48kB
> pagetables:3092kB bounce:0kB free_pcp:4
> Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 3392 64286 64286 64286
> Nov 06 10:07:05 sensor1 kernel: Node 1 DMA free:15884kB min:8kB low:20kB
> high:32kB active_anon:0kB inactive_anon:0kB active_file:0kB
> inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB
> managed:15884kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB
> free_pcp:0kB local_pcp:0kB free_cma:0kB
> Nov 06 10:07:05 sensor1 kernel: Node 3 active_anon:48629496kB
> inactive_anon:3250728kB active_file:108kB inactive_file:700kB
> unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:13636088kB
> dirty:0kB writeback:0kB shmem:9996kB shmem_thp: 0kB shmem_pmdmapped: 0kB
> anon_thp: 1337344kB writeback_tmp:0kB unst
> Nov 06 10:07:05 sensor1 kernel: Node 1 active_anon:58833756kB
> inactive_anon:4127304kB active_file:548kB inactive_file:0kB unevictable:0kB
> isolated(anon):0kB isolated(file):0kB mapped:2101152kB dirty:0kB
> writeback:0kB shmem:7828kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp:
> 13608960kB writeback_tmp:0kB unstab
> Nov 06 10:07:05 sensor1 kernel: active_anon:26865813 inactive_anon:1844508
> isolated_anon:0
>                                  active_file:163 inactive_file:0
> isolated_file:0
>                                  unevictable:0 dirty:0 writeback:0
> unstable:0
>                                  slab_reclaimable:25057
> slab_unreclaimable:65745
>                                  mapped:3934289 shmem:4456
> pagetables:70769 bounce:0
>                                  free:87073 free_pcp:3080 free_cma:0
> Nov 06 10:07:05 sensor1 kernel: Mem-Info:
> Nov 06 10:07:05 sensor1 kernel: R13: 0000000000001800 R14:
> 00007f375bd4f780 R15: 0000000000000022
> Nov 06 10:07:05 sensor1 kernel: R10: 00007f3618971000 R11:
> 00007f36189727e0 R12: 00007f375bd4f940
> Nov 06 10:07:05 sensor1 kernel: RBP: 00007f361424b000 R08:
> 0000000000000000 R09: 00007f36189727e0
> Nov 06 10:07:05 sensor1 kernel: RDX: 0000000000001780 RSI:
> 00007f361424b000 RDI: 00007f3618971000
> Nov 06 10:07:05 sensor1 kernel: RAX: 00007f3618971000 RBX:
> 0000000000001800 RCX: 00007f361424c760
> Nov 06 10:07:05 sensor1 kernel: RSP: 002b:00007ffd609fbc58 EFLAGS: 00010202
> Nov 06 10:07:05 sensor1 kernel: Code: Bad RIP value.
> Nov 06 10:07:05 sensor1 kernel: RIP: 0033:0x7f375bf1f925
> Nov 06 10:07:05 sensor1 kernel:  page_fault+0x1e/0x30
> Nov 06 10:07:05 sensor1 kernel:  ? page_fault+0x8/0x30
> Nov 06 10:07:05 sensor1 kernel:  __do_page_fault+0x249/0x4f0
> Nov 06 10:07:05 sensor1 kernel:  handle_mm_fault+0xd6/0x200
> Nov 06 10:07:05 sensor1 kernel:  ? __switch_to_asm+0x41/0x70
> Nov 06 10:07:05 sensor1 kernel:  __handle_mm_fault+0x958/0x1270
> Nov 06 10:07:05 sensor1 kernel:  alloc_pages_vma+0x74/0x1c0
> Nov 06 10:07:05 sensor1 kernel:  __alloc_pages_nodemask+0x28b/0x2b0
> Nov 06 10:07:05 sensor1 kernel:  __alloc_pages_slowpath+0xbd8/0xcb0
> Nov 06 10:07:05 sensor1 kernel:  out_of_memory+0x1a5/0x430
> Nov 06 10:07:05 sensor1 kernel:  oom_kill_process.cold.30+0xb/0x1cf
> Nov 06 10:07:05 sensor1 kernel:  dump_header+0x6b/0x283
> Nov 06 10:07:05 sensor1 kernel:  dump_stack+0x5c/0x80
> Nov 06 10:07:05 sensor1 kernel: Call Trace:
> Nov 06 10:07:05 sensor1 kernel: Hardware name: Supermicro Super
> Server/H11SSL-i, BIOS 1.0b 04/27/2018
> Nov 06 10:07:05 sensor1 kernel: CPU: 3 PID: 51496 Comm: bro Not tainted
> 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u1
> Nov 06 10:07:05 sensor1 kernel: bro cpuset=/ mems_allowed=1,3
> Nov 06 10:07:05 sensor1 kernel: bro invoked oom-killer:
> gfp_mask=0x6280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null),
> order=0, oom_score_adj=0
>
> It looks like bro was using 64 GB of RAM (summation of even though the
> system has 128 GB installed. (It also looks like there was no free swap.)
> Searching the Internet for ways to solve OOM, people suggested turning off
> over-commit, and configuring the overcommit ratio a bit higher. Currently,
> the values are:
>
> # sysctl vm.overcommit_memory
> vm.overcommit_memory = 0
> # sysctl vm.overcommit_ratio
> vm.overcommit_ratio = 50
>
> In one case the suggestion was to change those to vm.overcommit_memory = 1
> and vm.overcommit_ratio = 80 which seems reasonable based on the
> description at
> https://www.kernel.org/doc/Documentation/vm/overcommit-accounting.
>
> Or is there a better way to handle OOM?
>
> Mark
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/3c7d5506/attachment-0001.html 

From michalpurzynski1 at gmail.com  Wed Nov  6 15:58:58 2019
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Wed, 6 Nov 2019 15:58:58 -0800
Subject: [Zeek] monitoring proxied web traffic
In-Reply-To: <CA+fGMJRb6kDVcn-1BFZ=hgwLfvM4MWaKdw=jR_BkrB=ymQz8DQ@mail.gmail.com>
References: <CA+fGMJRb6kDVcn-1BFZ=hgwLfvM4MWaKdw=jR_BkrB=ymQz8DQ@mail.gmail.com>
Message-ID: <CAJ6bFK2DghGOc=o+mcUmgpW7CfxN651TSTuY8tJ+6XPR+OZpNA@mail.gmail.com>

There is no way for Zeek to really correlate if the proxy does everything
at the application level, like Squid. The proxy
 terminates the connection and then it's free to start the second leg, to
the destination host, as it wishes to.

Is your traffic from clients to proxy encrypted as well?

If the traffic between clients are the proxy is not encrypted (as it's
usually the case) then even if the client will establish a TLS session
_through_ the proxy you will see a destination with the "service" field
http,ssl and all the conn + ssl + http logs.

We have that + proxy logs and that's pretty much a full visibility.



On Wed, Nov 6, 2019 at 1:11 PM Konrad Weglowski <konrad.weglowski at gmail.com>
wrote:

> Hello,
>
> I need to monitor web traffic with Zeek that is going through an implicit
> web proxy. I would like to be able to see real client IPs in which case tap
> would be placed before the proxy, however I will not be able to see the
> real destination IP as it will always be proxy IP. I can also tap on the
> other side of the proxy where source will always be proxy IP at the same
> time.
>
> Is there a way with Zeek to correlate the two sessions (before and after
> proxy) somehow?
>
> I realize that UID or Community ID are probably not going do it since
> different source/destinations pairs for each for each session, but maybe
> based on the timestamp/URI/SNI/byte or packet counts/etc?
>
> I was also thinking "X-Forwarded-For" header but I guess that would only
> work for HTTP and not HTTPS connections.
>
> Thank You,
>
> Konrad
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/7545d509/attachment.html 

From jefflang at vt.edu  Wed Nov  6 16:23:14 2019
From: jefflang at vt.edu (Jeffry Lang)
Date: Wed, 6 Nov 2019 19:23:14 -0500
Subject: [Zeek] Worker being "killed nohup"
In-Reply-To: <CAJ6bFK2FZWODtTeUk1UuCJ-QxKED7Z0ckRA2Xxb_U3AthtoJtQ@mail.gmail.com>
References: <CAG=_CgA4txhcGmfppTnkOwU8vKWjoEkEfLNTrOtPeLvC6dXxNg@mail.gmail.com>
	<CABLqKGLvnu_7FH-6qa3oxOq1ehctX-4VW1n2aX=DffseAiix9w@mail.gmail.com>
	<CAG=_CgDOL1QabA83NuQTjjyDpPndcCf7wquCOrBMJuTd75JMrg@mail.gmail.com>
	<CAJ6bFK2FZWODtTeUk1UuCJ-QxKED7Z0ckRA2Xxb_U3AthtoJtQ@mail.gmail.com>
Message-ID: <CAEzr=8OeKVoRZFE+HSwFdQ3wMtW3nuKGjVZ9YY=hfKTywoFd=g@mail.gmail.com>

Can you expand on "tons of state"? It sounds like something I'm familiar
with but want to make sure. I work with Mark and we've been talking a lot
about this.

On Wed, Nov 6, 2019, 6:52 PM Micha? Purzy?ski <michalpurzynski1 at gmail.com>
wrote:

> Let's dig into the root cause.
> Your OOM most likely happens because either
> - you have tons of state and hit something that Justin neatly described as
> "unbound state growth"
> - or your logger threads cannot write messages quickly enough
>
> Given the OOM output it looks to me like workers are to blame. Do you have
> any custom scripts? Do you use the stock scan.bro or any other SumStats
> scripts?
>
> Can you observe the system for a while, in something like htop, or top
> with threads, to see what kind of processes eat most of memory and report
> back?
>
>
> On Wed, Nov 6, 2019 at 11:13 AM Mark Gardner <mkg at vt.edu> wrote:
>
>> On Wed, Nov 6, 2019 at 1:13 PM Jeff Barber <jsbarber60 at gmail.com> wrote:
>>
>>> > I'm running Zeek 2.6.4 and I have been seeing occasional error
>>> messages of the form:
>>> >
>>> > run-bro: line 110: 42574 Killed                  nohup ${pin_command}
>>> $pin_cpu "$mybro" "$@"
>>> >
>>> > The workers seem to be restarted fine and other than the error
>>> message, I haven't noticed any ill behavior. What should I do about the
>>> error messages?
>>>
>>> I would check your syslog. Assuming you are running linux, if your
>>> system runs out of memory, the kernel will go find the biggest process
>>> and kill it. This can often be a zeek process as they tend to grow large as
>>> more connections are tracked (depending on many factors: what scripts you
>>> are running, what you're logging, what kind of traffic is being seen,
>>> etc.). If that's happening, you should see something in syslog containing
>>> the string "invoked oom-killer:" If you look at the surrounding lines,
>>> there should be some info on process sizes showing why it was selected.
>>>
>>
>> Good catch Jeff.  A crash email arrived at 10:10 and the journal entry
>> starting around 10:07 contained the following:
>>
>> Nov 06 10:07:08 sensor1 kernel: oom_reaper: reaped process 43437 (bro),
>> now anon-rss:0kB, file-rss:1048576kB, shmem-rss:0kB
>> Nov 06 10:07:05 sensor1 kernel: Killed process 43437 (bro)
>> total-vm:67116888kB, anon-rss:57339476kB, file-rss:1042296kB, shmem-rss:0kB
>> Nov 06 10:07:05 sensor1 kernel: Out of memory: Kill process 43437 (bro)
>> score 444 or sacrifice child
>> Nov 06 10:07:05 sensor1 kernel: [  54824]    47 54824  1516502  1221530
>> 11284480       80             0 bro
>> Nov 06 10:07:05 sensor1 kernel: [  54817]    47 54817     1661       56
>>  49152       16             0 run-bro
>> Nov 06 10:07:05 sensor1 kernel: [  51543]    47 51543  1516502  1281056
>> 11886592     3616             0 bro
>> Nov 06 10:07:05 sensor1 kernel: [  51542]    47 51542  1516502  1285841
>> 11964416     2296             0 bro
>> Nov 06 10:07:05 sensor1 kernel: [  51539]    47 51539  1516502  1277957
>> 11890688     2817             0 bro
>> Nov 06 10:07:05 sensor1 kernel: [  51537]    47 51537  1516502  1282107
>> 11829248     2543             0 bro
>> Nov 06 10:07:05 sensor1 kernel: [  51533]    47 51533  1516502  1276417
>> 11939840     7591             0 bro
>> Nov 06 10:07:05 sensor1 kernel: [  51532]    47 51532  1516502  1290264
>> 12042240     8006             0 bro
>> Nov 06 10:07:05 sensor1 kernel: [  51523]    47 51523  1516502  1281160
>> 11894784     7479             0 bro
>> Nov 06 10:07:05 sensor1 kernel: [  51521]    47 51521  1516502  1285423
>> 11984896     7957             0 bro
>> Nov 06 10:07:05 sensor1 kernel: [  51520]    47 51520  1745878  1370574
>> 12668928     6908             0 bro
>> Nov 06 10:07:05 sensor1 kernel: [  51501]    47 51501  1516502  1286290
>> 11788288     6967             0 bro
>> Nov 06 10:07:05 sensor1 kernel: [  51499]    47 51499     1661       22
>>  49152       49             0 run-bro
>> Nov 06 10:07:05 sensor1 kernel: [  51498]    47 51498  1516502  1276891
>> 11882496     7233             0 bro
>> Nov 06 10:07:05 sensor1 kernel: [  51496]    47 51496  1516502  1287064
>> 11948032     7685             0 bro
>> Nov 06 10:07:05 sensor1 kernel: [  51491]    47 51491     1661       18
>>  53248       55             0 run-bro
>> Nov 06 10:07:05 sensor1 kernel: [  51487]    47 51487     1661       18
>>  57344       54             0 run-bro
>> Nov 06 10:07:05 sensor1 kernel: [  51474]    47 51474     1661       21
>>  49152       52             0 run-bro
>> Nov 06 10:07:05 sensor1 kernel: [  51471]    47 51471  1516502  1277820
>> 11890688     7964             0 bro
>> Nov 06 10:07:05 sensor1 kernel: [  51465]    47 51465     1661       20
>>  53248       53             0 run-bro
>> Nov 06 10:07:05 sensor1 kernel: [  51455]    47 51455     1661       23
>>  57344       49             0 run-bro
>> Nov 06 10:07:05 sensor1 kernel: [  51440]    47 51440     1661       18
>>  49152       54             0 run-bro
>> Nov 06 10:07:05 sensor1 kernel: [  51438]    47 51438     1661       23
>>  49152       49             0 run-bro
>> Nov 06 10:07:05 sensor1 kernel: [  51428]    47 51428     1661       18
>>  49152       54             0 run-bro
>> Nov 06 10:07:05 sensor1 kernel: [  51414]    47 51414     1661       16
>>  53248       56             0 run-bro
>> Nov 06 10:07:05 sensor1 kernel: [  51405]    47 51405     1661       20
>>  53248       52             0 run-bro
>> Nov 06 10:07:05 sensor1 kernel: [  51397]    47 51397     1661       14
>>  45056       59             0 run-bro
>> Nov 06 10:07:05 sensor1 kernel: [  51375]    47 51375     1661       15
>>  49152       58             0 run-bro
>> Nov 06 10:07:05 sensor1 kernel: [  43437]    47 43437 16779222 14595446
>> 122212352   129972             0 bro
>> Nov 06 10:07:05 sensor1 kernel: [  43343]    47 43343     1661        2
>>  49152       70             0 run-bro
>> Nov 06 10:07:05 sensor1 kernel: [  43183]    47 43183    42625        4
>> 106496      608             0 (sd-pam)
>> Nov 06 10:07:05 sensor1 kernel: [  43181]    47 43181     5290        0
>>  77824      342             0 systemd
>> Nov 06 10:07:05 sensor1 kernel: [    996]     0   996     1346        0
>>  49152       33             0 agetty
>> Nov 06 10:07:05 sensor1 kernel: [    995]     0   995     1403        0
>>  45056       32             0 agetty
>> Nov 06 10:07:05 sensor1 kernel: [    994]     0   994     3963       38
>>  69632      178         -1000 sshd
>> Nov 06 10:07:05 sensor1 kernel: [    991]     0   991     2119       20
>>  53248       35             0 cron
>> Nov 06 10:07:05 sensor1 kernel: [    990]   105   990     2210       97
>>  57344       55          -900 dbus-daemon
>> Nov 06 10:07:05 sensor1 kernel: [    977]     0   977    56456       90
>>  86016      378             0 rsyslogd
>> Nov 06 10:07:05 sensor1 kernel: [    968]     0   968     4875       78
>>  77824      189             0 systemd-logind
>> Nov 06 10:07:05 sensor1 kernel: [    944]     0   944     2021        1
>>  57344      799             0 haveged
>> Nov 06 10:07:05 sensor1 kernel: [    931]   100   931    23270       26
>>  86016      207             0 systemd-timesyn
>> Nov 06 10:07:05 sensor1 kernel: [    640]     0   640     5579       14
>>  69632      309         -1000 systemd-udevd
>> Nov 06 10:07:05 sensor1 kernel: [    615]     0   615    74225     2624
>> 638976      568             0 systemd-journal
>> Nov 06 10:07:05 sensor1 kernel: [  pid  ]   uid  tgid total_vm      rss
>> pgtables_bytes swapents oom_score_adj name
>> Nov 06 10:07:05 sensor1 kernel: Tasks state (memory values in pages):
>> Nov 06 10:07:05 sensor1 kernel: 0 pages hwpoisoned
>> Nov 06 10:07:05 sensor1 kernel: 551979 pages reserved
>> Nov 06 10:07:05 sensor1 kernel: 0 pages HighMem/MovableOnly
>> Nov 06 10:07:05 sensor1 kernel: 33528566 pages RAM
>> Nov 06 10:07:05 sensor1 kernel: Total swap = 976892kB
>> Nov 06 10:07:05 sensor1 kernel: Free swap  = 0kB
>> Nov 06 10:07:05 sensor1 kernel: Swap cache stats: add 759161, delete
>> 711025, find 25951/48832
>> Nov 06 10:07:05 sensor1 kernel: 48128 pages in swap cache
>> Nov 06 10:07:05 sensor1 kernel: 52774 total pagecache pages
>> Nov 06 10:07:05 sensor1 kernel: Node 3 hugepages_total=0 hugepages_free=0
>> hugepages_surp=0 hugepages_size=2048kB
>> Nov 06 10:07:05 sensor1 kernel: Node 3 hugepages_total=0 hugepages_free=0
>> hugepages_surp=0 hugepages_size=1048576kB
>> Nov 06 10:07:05 sensor1 kernel: Node 1 hugepages_total=0 hugepages_free=0
>> hugepages_surp=0 hugepages_size=2048kB
>> Nov 06 10:07:05 sensor1 kernel: Node 1 hugepages_total=0 hugepages_free=0
>> hugepages_surp=0 hugepages_size=1048576kB
>> Nov 06 10:07:05 sensor1 kernel: Node 3 Normal: 73*4kB (U) 150*8kB (UH)
>> 59*16kB (UH) 1325*32kB (UH) 5*64kB (H) 2*128kB (H) 0*256kB 0*512kB 0*1024kB
>> 0*2048kB 0*4096kB = 45412kB
>> Nov 06 10:07:05 sensor1 kernel: Node 1 Normal: 78*4kB (UMEH) 213*8kB
>> (UEH) 70*16kB (UH) 1232*32kB (UMH) 7*64kB (H) 0*128kB 0*256kB 0*512kB
>> 0*1024kB 0*2048kB 0*4096kB = 43008kB
>> Nov 06 10:07:05 sensor1 kernel: Node 1 DMA32: 697*4kB (UME) 647*8kB (UME)
>> 392*16kB (UME) 190*32kB (UME) 94*64kB (UE) 51*128kB (UE) 63*256kB (UME)
>> 240*512kB (UME) 72*1024kB (UME) 0*2048kB 0*4096kB = 245596kB
>> Nov 06 10:07:05 sensor1 kernel: Node 1 DMA: 1*4kB (U) 1*8kB (U) 2*16kB
>> (U) 1*32kB (U) 3*64kB (U) 0*128kB 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB
>> (M) 3*4096kB (M) = 15884kB
>> Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 0 0 0 0
>> Nov 06 10:07:05 sensor1 kernel: Node 3 Normal free:44844kB min:45116kB
>> low:111144kB high:177172kB active_anon:48629496kB inactive_anon:3250728kB
>> active_file:64kB inactive_file:668kB unevictable:0kB writepending:0kB
>> present:67094528kB managed:66031300kB mlocked:0kB kernel_stack:5352kB
>> pagetables:146976kB bounce
>> Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 0 0 0 0
>> Nov 06 10:07:05 sensor1 kernel: Node 1 Normal free:41932kB min:42604kB
>> low:104956kB high:167308kB active_anon:55633904kB inactive_anon:4082748kB
>> active_file:560kB inactive_file:0kB unevictable:0kB writepending:0kB
>> present:63438848kB managed:62359848kB mlocked:0kB kernel_stack:4776kB
>> pagetables:133008kB bounce:
>> Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 0 60894 60894 60894
>> Nov 06 10:07:05 sensor1 kernel: Node 1 DMA32 free:245356kB min:2372kB
>> low:5844kB high:9316kB active_anon:3199852kB inactive_anon:44556kB
>> active_file:100kB inactive_file:0kB unevictable:0kB writepending:0kB
>> present:3564892kB managed:3499316kB mlocked:0kB kernel_stack:48kB
>> pagetables:3092kB bounce:0kB free_pcp:4
>> Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 3392 64286 64286 64286
>> Nov 06 10:07:05 sensor1 kernel: Node 1 DMA free:15884kB min:8kB low:20kB
>> high:32kB active_anon:0kB inactive_anon:0kB active_file:0kB
>> inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB
>> managed:15884kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB
>> free_pcp:0kB local_pcp:0kB free_cma:0kB
>> Nov 06 10:07:05 sensor1 kernel: Node 3 active_anon:48629496kB
>> inactive_anon:3250728kB active_file:108kB inactive_file:700kB
>> unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:13636088kB
>> dirty:0kB writeback:0kB shmem:9996kB shmem_thp: 0kB shmem_pmdmapped: 0kB
>> anon_thp: 1337344kB writeback_tmp:0kB unst
>> Nov 06 10:07:05 sensor1 kernel: Node 1 active_anon:58833756kB
>> inactive_anon:4127304kB active_file:548kB inactive_file:0kB unevictable:0kB
>> isolated(anon):0kB isolated(file):0kB mapped:2101152kB dirty:0kB
>> writeback:0kB shmem:7828kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp:
>> 13608960kB writeback_tmp:0kB unstab
>> Nov 06 10:07:05 sensor1 kernel: active_anon:26865813
>> inactive_anon:1844508 isolated_anon:0
>>                                  active_file:163 inactive_file:0
>> isolated_file:0
>>                                  unevictable:0 dirty:0 writeback:0
>> unstable:0
>>                                  slab_reclaimable:25057
>> slab_unreclaimable:65745
>>                                  mapped:3934289 shmem:4456
>> pagetables:70769 bounce:0
>>                                  free:87073 free_pcp:3080 free_cma:0
>> Nov 06 10:07:05 sensor1 kernel: Mem-Info:
>> Nov 06 10:07:05 sensor1 kernel: R13: 0000000000001800 R14:
>> 00007f375bd4f780 R15: 0000000000000022
>> Nov 06 10:07:05 sensor1 kernel: R10: 00007f3618971000 R11:
>> 00007f36189727e0 R12: 00007f375bd4f940
>> Nov 06 10:07:05 sensor1 kernel: RBP: 00007f361424b000 R08:
>> 0000000000000000 R09: 00007f36189727e0
>> Nov 06 10:07:05 sensor1 kernel: RDX: 0000000000001780 RSI:
>> 00007f361424b000 RDI: 00007f3618971000
>> Nov 06 10:07:05 sensor1 kernel: RAX: 00007f3618971000 RBX:
>> 0000000000001800 RCX: 00007f361424c760
>> Nov 06 10:07:05 sensor1 kernel: RSP: 002b:00007ffd609fbc58 EFLAGS:
>> 00010202
>> Nov 06 10:07:05 sensor1 kernel: Code: Bad RIP value.
>> Nov 06 10:07:05 sensor1 kernel: RIP: 0033:0x7f375bf1f925
>> Nov 06 10:07:05 sensor1 kernel:  page_fault+0x1e/0x30
>> Nov 06 10:07:05 sensor1 kernel:  ? page_fault+0x8/0x30
>> Nov 06 10:07:05 sensor1 kernel:  __do_page_fault+0x249/0x4f0
>> Nov 06 10:07:05 sensor1 kernel:  handle_mm_fault+0xd6/0x200
>> Nov 06 10:07:05 sensor1 kernel:  ? __switch_to_asm+0x41/0x70
>> Nov 06 10:07:05 sensor1 kernel:  __handle_mm_fault+0x958/0x1270
>> Nov 06 10:07:05 sensor1 kernel:  alloc_pages_vma+0x74/0x1c0
>> Nov 06 10:07:05 sensor1 kernel:  __alloc_pages_nodemask+0x28b/0x2b0
>> Nov 06 10:07:05 sensor1 kernel:  __alloc_pages_slowpath+0xbd8/0xcb0
>> Nov 06 10:07:05 sensor1 kernel:  out_of_memory+0x1a5/0x430
>> Nov 06 10:07:05 sensor1 kernel:  oom_kill_process.cold.30+0xb/0x1cf
>> Nov 06 10:07:05 sensor1 kernel:  dump_header+0x6b/0x283
>> Nov 06 10:07:05 sensor1 kernel:  dump_stack+0x5c/0x80
>> Nov 06 10:07:05 sensor1 kernel: Call Trace:
>> Nov 06 10:07:05 sensor1 kernel: Hardware name: Supermicro Super
>> Server/H11SSL-i, BIOS 1.0b 04/27/2018
>> Nov 06 10:07:05 sensor1 kernel: CPU: 3 PID: 51496 Comm: bro Not tainted
>> 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u1
>> Nov 06 10:07:05 sensor1 kernel: bro cpuset=/ mems_allowed=1,3
>> Nov 06 10:07:05 sensor1 kernel: bro invoked oom-killer:
>> gfp_mask=0x6280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null),
>> order=0, oom_score_adj=0
>>
>> It looks like bro was using 64 GB of RAM (summation of even though the
>> system has 128 GB installed. (It also looks like there was no free swap.)
>> Searching the Internet for ways to solve OOM, people suggested turning off
>> over-commit, and configuring the overcommit ratio a bit higher. Currently,
>> the values are:
>>
>> # sysctl vm.overcommit_memory
>> vm.overcommit_memory = 0
>> # sysctl vm.overcommit_ratio
>> vm.overcommit_ratio = 50
>>
>> In one case the suggestion was to change those to vm.overcommit_memory =
>> 1 and vm.overcommit_ratio = 80 which seems reasonable based on the
>> description at
>> https://www.kernel.org/doc/Documentation/vm/overcommit-accounting.
>>
>> Or is there a better way to handle OOM?
>>
>> Mark
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/5ddc4550/attachment-0001.html 

From michalpurzynski1 at gmail.com  Wed Nov  6 19:18:12 2019
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Wed, 6 Nov 2019 19:18:12 -0800
Subject: [Zeek] Worker being "killed nohup"
In-Reply-To: <CAEzr=8OeKVoRZFE+HSwFdQ3wMtW3nuKGjVZ9YY=hfKTywoFd=g@mail.gmail.com>
References: <CAG=_CgA4txhcGmfppTnkOwU8vKWjoEkEfLNTrOtPeLvC6dXxNg@mail.gmail.com>
	<CABLqKGLvnu_7FH-6qa3oxOq1ehctX-4VW1n2aX=DffseAiix9w@mail.gmail.com>
	<CAG=_CgDOL1QabA83NuQTjjyDpPndcCf7wquCOrBMJuTd75JMrg@mail.gmail.com>
	<CAJ6bFK2FZWODtTeUk1UuCJ-QxKED7Z0ckRA2Xxb_U3AthtoJtQ@mail.gmail.com>
	<CAEzr=8OeKVoRZFE+HSwFdQ3wMtW3nuKGjVZ9YY=hfKTywoFd=g@mail.gmail.com>
Message-ID: <CAJ6bFK2_5nQODU57jRTrnHtU3ZoJw5ZnROhAexj83Lq+RHcDpw@mail.gmail.com>

Sure!

Zeek internally keeps a state for each connection - and quite possibly
information in other data structures that your scripts created. If you have
lots of events and each of them adds somehow data into those data
structures, it might eventually eat all your RAM.

To give you an example. An intern of mine, a while ago, designed a
brilliant script

event http_reply(c: connection, version: string, code: count,
                 reason: string) &priority=3

(...)

if (code >= 400) {
            add c$http$tags[HTTP_ERROR];

            SumStats::observe("http.excessive_errors.attacker_badhits",
[$host=c$id$orig_h],
                              [$str=fmt("%d %s%s", code, c$http$host,
c$http$uri)]);
            SumStats::observe("http.excessive_errors.victim_badhits",
[$host=c$id$resp_h],
                              [$str=fmt("%d %s%s", code, c$http$host,
c$http$uri)]);
        }
        else if (code < 400) {
            SumStats::observe("http.excessive_errors.attacker_goodhits",
[$host=c$id$orig_h], []);
            SumStats::observe("http.excessive_errors.victim_goodhits",
[$host=c$id$resp_h], []);


This script will store, per inbound connection, a ratio of good vs bad HTTP
transactions. That can be used to do a behaviour profiling of a client. A
scanner would easily have mostly "bad hits" with return code >400.

This all worked well for 15 minutes and then 11 nodes x 64GB RAM each
cluster ran out of memory. See, this cluster monitored (among other things)
the Firefox update system, with 500 000 000 clients talking to it.

Why did it crash? Because the SumStats framework was adding data, into
internal data structures, per connection.

Notice how there is technically no memory leak. Given 2TB of RAM this maybe
would have worked.

Basically, if you have tons of connections (not simply bandwidth or
packets) the amount of memory necessary to keep all of them might be simply
too much.

Now, this data expires (unless you have a script that never does that), but
it might be the amount of state grows too quickly and the expiration is not
quick enough, to free up some memory.


My quick suspect would be the scan.bro / scan.zeek old script that comes
bundled with Zeek. If you have it enabled, disable and see if you're still
crashing.

You can then take a look at your scripts and see if there is some data
structure that will grow per connection, over time - and how quickly you
purge data from it.


On Wed, Nov 6, 2019 at 4:23 PM Jeffry Lang <jefflang at vt.edu> wrote:

> Can you expand on "tons of state"? It sounds like something I'm familiar
> with but want to make sure. I work with Mark and we've been talking a lot
> about this.
>
> On Wed, Nov 6, 2019, 6:52 PM Micha? Purzy?ski <michalpurzynski1 at gmail.com>
> wrote:
>
>> Let's dig into the root cause.
>> Your OOM most likely happens because either
>> - you have tons of state and hit something that Justin neatly described
>> as "unbound state growth"
>> - or your logger threads cannot write messages quickly enough
>>
>> Given the OOM output it looks to me like workers are to blame. Do you
>> have any custom scripts? Do you use the stock scan.bro or any other
>> SumStats scripts?
>>
>> Can you observe the system for a while, in something like htop, or top
>> with threads, to see what kind of processes eat most of memory and report
>> back?
>>
>>
>> On Wed, Nov 6, 2019 at 11:13 AM Mark Gardner <mkg at vt.edu> wrote:
>>
>>> On Wed, Nov 6, 2019 at 1:13 PM Jeff Barber <jsbarber60 at gmail.com> wrote:
>>>
>>>> > I'm running Zeek 2.6.4 and I have been seeing occasional error
>>>> messages of the form:
>>>> >
>>>> > run-bro: line 110: 42574 Killed                  nohup
>>>> ${pin_command} $pin_cpu "$mybro" "$@"
>>>> >
>>>> > The workers seem to be restarted fine and other than the error
>>>> message, I haven't noticed any ill behavior. What should I do about the
>>>> error messages?
>>>>
>>>> I would check your syslog. Assuming you are running linux, if your
>>>> system runs out of memory, the kernel will go find the biggest process
>>>> and kill it. This can often be a zeek process as they tend to grow large as
>>>> more connections are tracked (depending on many factors: what scripts you
>>>> are running, what you're logging, what kind of traffic is being seen,
>>>> etc.). If that's happening, you should see something in syslog containing
>>>> the string "invoked oom-killer:" If you look at the surrounding lines,
>>>> there should be some info on process sizes showing why it was selected.
>>>>
>>>
>>> Good catch Jeff.  A crash email arrived at 10:10 and the journal entry
>>> starting around 10:07 contained the following:
>>>
>>> Nov 06 10:07:08 sensor1 kernel: oom_reaper: reaped process 43437 (bro),
>>> now anon-rss:0kB, file-rss:1048576kB, shmem-rss:0kB
>>> Nov 06 10:07:05 sensor1 kernel: Killed process 43437 (bro)
>>> total-vm:67116888kB, anon-rss:57339476kB, file-rss:1042296kB, shmem-rss:0kB
>>> Nov 06 10:07:05 sensor1 kernel: Out of memory: Kill process 43437 (bro)
>>> score 444 or sacrifice child
>>> Nov 06 10:07:05 sensor1 kernel: [  54824]    47 54824  1516502  1221530
>>> 11284480       80             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  54817]    47 54817     1661       56
>>>    49152       16             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51543]    47 51543  1516502  1281056
>>> 11886592     3616             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51542]    47 51542  1516502  1285841
>>> 11964416     2296             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51539]    47 51539  1516502  1277957
>>> 11890688     2817             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51537]    47 51537  1516502  1282107
>>> 11829248     2543             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51533]    47 51533  1516502  1276417
>>> 11939840     7591             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51532]    47 51532  1516502  1290264
>>> 12042240     8006             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51523]    47 51523  1516502  1281160
>>> 11894784     7479             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51521]    47 51521  1516502  1285423
>>> 11984896     7957             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51520]    47 51520  1745878  1370574
>>> 12668928     6908             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51501]    47 51501  1516502  1286290
>>> 11788288     6967             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51499]    47 51499     1661       22
>>>    49152       49             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51498]    47 51498  1516502  1276891
>>> 11882496     7233             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51496]    47 51496  1516502  1287064
>>> 11948032     7685             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51491]    47 51491     1661       18
>>>    53248       55             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51487]    47 51487     1661       18
>>>    57344       54             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51474]    47 51474     1661       21
>>>    49152       52             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51471]    47 51471  1516502  1277820
>>> 11890688     7964             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51465]    47 51465     1661       20
>>>    53248       53             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51455]    47 51455     1661       23
>>>    57344       49             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51440]    47 51440     1661       18
>>>    49152       54             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51438]    47 51438     1661       23
>>>    49152       49             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51428]    47 51428     1661       18
>>>    49152       54             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51414]    47 51414     1661       16
>>>    53248       56             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51405]    47 51405     1661       20
>>>    53248       52             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51397]    47 51397     1661       14
>>>    45056       59             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51375]    47 51375     1661       15
>>>    49152       58             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  43437]    47 43437 16779222 14595446
>>> 122212352   129972             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  43343]    47 43343     1661        2
>>>    49152       70             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  43183]    47 43183    42625        4
>>>   106496      608             0 (sd-pam)
>>> Nov 06 10:07:05 sensor1 kernel: [  43181]    47 43181     5290        0
>>>    77824      342             0 systemd
>>> Nov 06 10:07:05 sensor1 kernel: [    996]     0   996     1346        0
>>>    49152       33             0 agetty
>>> Nov 06 10:07:05 sensor1 kernel: [    995]     0   995     1403        0
>>>    45056       32             0 agetty
>>> Nov 06 10:07:05 sensor1 kernel: [    994]     0   994     3963       38
>>>    69632      178         -1000 sshd
>>> Nov 06 10:07:05 sensor1 kernel: [    991]     0   991     2119       20
>>>    53248       35             0 cron
>>> Nov 06 10:07:05 sensor1 kernel: [    990]   105   990     2210       97
>>>    57344       55          -900 dbus-daemon
>>> Nov 06 10:07:05 sensor1 kernel: [    977]     0   977    56456       90
>>>    86016      378             0 rsyslogd
>>> Nov 06 10:07:05 sensor1 kernel: [    968]     0   968     4875       78
>>>    77824      189             0 systemd-logind
>>> Nov 06 10:07:05 sensor1 kernel: [    944]     0   944     2021        1
>>>    57344      799             0 haveged
>>> Nov 06 10:07:05 sensor1 kernel: [    931]   100   931    23270       26
>>>    86016      207             0 systemd-timesyn
>>> Nov 06 10:07:05 sensor1 kernel: [    640]     0   640     5579       14
>>>    69632      309         -1000 systemd-udevd
>>> Nov 06 10:07:05 sensor1 kernel: [    615]     0   615    74225     2624
>>>   638976      568             0 systemd-journal
>>> Nov 06 10:07:05 sensor1 kernel: [  pid  ]   uid  tgid total_vm      rss
>>> pgtables_bytes swapents oom_score_adj name
>>> Nov 06 10:07:05 sensor1 kernel: Tasks state (memory values in pages):
>>> Nov 06 10:07:05 sensor1 kernel: 0 pages hwpoisoned
>>> Nov 06 10:07:05 sensor1 kernel: 551979 pages reserved
>>> Nov 06 10:07:05 sensor1 kernel: 0 pages HighMem/MovableOnly
>>> Nov 06 10:07:05 sensor1 kernel: 33528566 pages RAM
>>> Nov 06 10:07:05 sensor1 kernel: Total swap = 976892kB
>>> Nov 06 10:07:05 sensor1 kernel: Free swap  = 0kB
>>> Nov 06 10:07:05 sensor1 kernel: Swap cache stats: add 759161, delete
>>> 711025, find 25951/48832
>>> Nov 06 10:07:05 sensor1 kernel: 48128 pages in swap cache
>>> Nov 06 10:07:05 sensor1 kernel: 52774 total pagecache pages
>>> Nov 06 10:07:05 sensor1 kernel: Node 3 hugepages_total=0
>>> hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
>>> Nov 06 10:07:05 sensor1 kernel: Node 3 hugepages_total=0
>>> hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
>>> Nov 06 10:07:05 sensor1 kernel: Node 1 hugepages_total=0
>>> hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
>>> Nov 06 10:07:05 sensor1 kernel: Node 1 hugepages_total=0
>>> hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
>>> Nov 06 10:07:05 sensor1 kernel: Node 3 Normal: 73*4kB (U) 150*8kB (UH)
>>> 59*16kB (UH) 1325*32kB (UH) 5*64kB (H) 2*128kB (H) 0*256kB 0*512kB 0*1024kB
>>> 0*2048kB 0*4096kB = 45412kB
>>> Nov 06 10:07:05 sensor1 kernel: Node 1 Normal: 78*4kB (UMEH) 213*8kB
>>> (UEH) 70*16kB (UH) 1232*32kB (UMH) 7*64kB (H) 0*128kB 0*256kB 0*512kB
>>> 0*1024kB 0*2048kB 0*4096kB = 43008kB
>>> Nov 06 10:07:05 sensor1 kernel: Node 1 DMA32: 697*4kB (UME) 647*8kB
>>> (UME) 392*16kB (UME) 190*32kB (UME) 94*64kB (UE) 51*128kB (UE) 63*256kB
>>> (UME) 240*512kB (UME) 72*1024kB (UME) 0*2048kB 0*4096kB = 245596kB
>>> Nov 06 10:07:05 sensor1 kernel: Node 1 DMA: 1*4kB (U) 1*8kB (U) 2*16kB
>>> (U) 1*32kB (U) 3*64kB (U) 0*128kB 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB
>>> (M) 3*4096kB (M) = 15884kB
>>> Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 0 0 0 0
>>> Nov 06 10:07:05 sensor1 kernel: Node 3 Normal free:44844kB min:45116kB
>>> low:111144kB high:177172kB active_anon:48629496kB inactive_anon:3250728kB
>>> active_file:64kB inactive_file:668kB unevictable:0kB writepending:0kB
>>> present:67094528kB managed:66031300kB mlocked:0kB kernel_stack:5352kB
>>> pagetables:146976kB bounce
>>> Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 0 0 0 0
>>> Nov 06 10:07:05 sensor1 kernel: Node 1 Normal free:41932kB min:42604kB
>>> low:104956kB high:167308kB active_anon:55633904kB inactive_anon:4082748kB
>>> active_file:560kB inactive_file:0kB unevictable:0kB writepending:0kB
>>> present:63438848kB managed:62359848kB mlocked:0kB kernel_stack:4776kB
>>> pagetables:133008kB bounce:
>>> Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 0 60894 60894 60894
>>> Nov 06 10:07:05 sensor1 kernel: Node 1 DMA32 free:245356kB min:2372kB
>>> low:5844kB high:9316kB active_anon:3199852kB inactive_anon:44556kB
>>> active_file:100kB inactive_file:0kB unevictable:0kB writepending:0kB
>>> present:3564892kB managed:3499316kB mlocked:0kB kernel_stack:48kB
>>> pagetables:3092kB bounce:0kB free_pcp:4
>>> Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 3392 64286 64286
>>> 64286
>>> Nov 06 10:07:05 sensor1 kernel: Node 1 DMA free:15884kB min:8kB low:20kB
>>> high:32kB active_anon:0kB inactive_anon:0kB active_file:0kB
>>> inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB
>>> managed:15884kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB
>>> free_pcp:0kB local_pcp:0kB free_cma:0kB
>>> Nov 06 10:07:05 sensor1 kernel: Node 3 active_anon:48629496kB
>>> inactive_anon:3250728kB active_file:108kB inactive_file:700kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:13636088kB
>>> dirty:0kB writeback:0kB shmem:9996kB shmem_thp: 0kB shmem_pmdmapped: 0kB
>>> anon_thp: 1337344kB writeback_tmp:0kB unst
>>> Nov 06 10:07:05 sensor1 kernel: Node 1 active_anon:58833756kB
>>> inactive_anon:4127304kB active_file:548kB inactive_file:0kB unevictable:0kB
>>> isolated(anon):0kB isolated(file):0kB mapped:2101152kB dirty:0kB
>>> writeback:0kB shmem:7828kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp:
>>> 13608960kB writeback_tmp:0kB unstab
>>> Nov 06 10:07:05 sensor1 kernel: active_anon:26865813
>>> inactive_anon:1844508 isolated_anon:0
>>>                                  active_file:163 inactive_file:0
>>> isolated_file:0
>>>                                  unevictable:0 dirty:0 writeback:0
>>> unstable:0
>>>                                  slab_reclaimable:25057
>>> slab_unreclaimable:65745
>>>                                  mapped:3934289 shmem:4456
>>> pagetables:70769 bounce:0
>>>                                  free:87073 free_pcp:3080 free_cma:0
>>> Nov 06 10:07:05 sensor1 kernel: Mem-Info:
>>> Nov 06 10:07:05 sensor1 kernel: R13: 0000000000001800 R14:
>>> 00007f375bd4f780 R15: 0000000000000022
>>> Nov 06 10:07:05 sensor1 kernel: R10: 00007f3618971000 R11:
>>> 00007f36189727e0 R12: 00007f375bd4f940
>>> Nov 06 10:07:05 sensor1 kernel: RBP: 00007f361424b000 R08:
>>> 0000000000000000 R09: 00007f36189727e0
>>> Nov 06 10:07:05 sensor1 kernel: RDX: 0000000000001780 RSI:
>>> 00007f361424b000 RDI: 00007f3618971000
>>> Nov 06 10:07:05 sensor1 kernel: RAX: 00007f3618971000 RBX:
>>> 0000000000001800 RCX: 00007f361424c760
>>> Nov 06 10:07:05 sensor1 kernel: RSP: 002b:00007ffd609fbc58 EFLAGS:
>>> 00010202
>>> Nov 06 10:07:05 sensor1 kernel: Code: Bad RIP value.
>>> Nov 06 10:07:05 sensor1 kernel: RIP: 0033:0x7f375bf1f925
>>> Nov 06 10:07:05 sensor1 kernel:  page_fault+0x1e/0x30
>>> Nov 06 10:07:05 sensor1 kernel:  ? page_fault+0x8/0x30
>>> Nov 06 10:07:05 sensor1 kernel:  __do_page_fault+0x249/0x4f0
>>> Nov 06 10:07:05 sensor1 kernel:  handle_mm_fault+0xd6/0x200
>>> Nov 06 10:07:05 sensor1 kernel:  ? __switch_to_asm+0x41/0x70
>>> Nov 06 10:07:05 sensor1 kernel:  __handle_mm_fault+0x958/0x1270
>>> Nov 06 10:07:05 sensor1 kernel:  alloc_pages_vma+0x74/0x1c0
>>> Nov 06 10:07:05 sensor1 kernel:  __alloc_pages_nodemask+0x28b/0x2b0
>>> Nov 06 10:07:05 sensor1 kernel:  __alloc_pages_slowpath+0xbd8/0xcb0
>>> Nov 06 10:07:05 sensor1 kernel:  out_of_memory+0x1a5/0x430
>>> Nov 06 10:07:05 sensor1 kernel:  oom_kill_process.cold.30+0xb/0x1cf
>>> Nov 06 10:07:05 sensor1 kernel:  dump_header+0x6b/0x283
>>> Nov 06 10:07:05 sensor1 kernel:  dump_stack+0x5c/0x80
>>> Nov 06 10:07:05 sensor1 kernel: Call Trace:
>>> Nov 06 10:07:05 sensor1 kernel: Hardware name: Supermicro Super
>>> Server/H11SSL-i, BIOS 1.0b 04/27/2018
>>> Nov 06 10:07:05 sensor1 kernel: CPU: 3 PID: 51496 Comm: bro Not tainted
>>> 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u1
>>> Nov 06 10:07:05 sensor1 kernel: bro cpuset=/ mems_allowed=1,3
>>> Nov 06 10:07:05 sensor1 kernel: bro invoked oom-killer:
>>> gfp_mask=0x6280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null),
>>> order=0, oom_score_adj=0
>>>
>>> It looks like bro was using 64 GB of RAM (summation of even though the
>>> system has 128 GB installed. (It also looks like there was no free swap.)
>>> Searching the Internet for ways to solve OOM, people suggested turning off
>>> over-commit, and configuring the overcommit ratio a bit higher. Currently,
>>> the values are:
>>>
>>> # sysctl vm.overcommit_memory
>>> vm.overcommit_memory = 0
>>> # sysctl vm.overcommit_ratio
>>> vm.overcommit_ratio = 50
>>>
>>> In one case the suggestion was to change those to vm.overcommit_memory
>>> = 1 and vm.overcommit_ratio = 80 which seems reasonable based on the
>>> description at
>>> https://www.kernel.org/doc/Documentation/vm/overcommit-accounting.
>>>
>>> Or is there a better way to handle OOM?
>>>
>>> Mark
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/10313a63/attachment-0001.html 

From michalpurzynski1 at gmail.com  Wed Nov  6 19:21:39 2019
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Wed, 6 Nov 2019 19:21:39 -0800
Subject: [Zeek] Worker being "killed nohup"
In-Reply-To: <CAEzr=8OeKVoRZFE+HSwFdQ3wMtW3nuKGjVZ9YY=hfKTywoFd=g@mail.gmail.com>
References: <CAG=_CgA4txhcGmfppTnkOwU8vKWjoEkEfLNTrOtPeLvC6dXxNg@mail.gmail.com>
	<CABLqKGLvnu_7FH-6qa3oxOq1ehctX-4VW1n2aX=DffseAiix9w@mail.gmail.com>
	<CAG=_CgDOL1QabA83NuQTjjyDpPndcCf7wquCOrBMJuTd75JMrg@mail.gmail.com>
	<CAJ6bFK2FZWODtTeUk1UuCJ-QxKED7Z0ckRA2Xxb_U3AthtoJtQ@mail.gmail.com>
	<CAEzr=8OeKVoRZFE+HSwFdQ3wMtW3nuKGjVZ9YY=hfKTywoFd=g@mail.gmail.com>
Message-ID: <CAJ6bFK0k9XY4uaEgWD-2tGdd094Gajyu7GH+7vTKxhgo+ifvuA@mail.gmail.com>

BTW is your Zeek built with tcmalloc? I heard horror stories about that now
I just use stock glibc malloc or jemalloc.

On Wed, Nov 6, 2019 at 4:23 PM Jeffry Lang <jefflang at vt.edu> wrote:

> Can you expand on "tons of state"? It sounds like something I'm familiar
> with but want to make sure. I work with Mark and we've been talking a lot
> about this.
>
> On Wed, Nov 6, 2019, 6:52 PM Micha? Purzy?ski <michalpurzynski1 at gmail.com>
> wrote:
>
>> Let's dig into the root cause.
>> Your OOM most likely happens because either
>> - you have tons of state and hit something that Justin neatly described
>> as "unbound state growth"
>> - or your logger threads cannot write messages quickly enough
>>
>> Given the OOM output it looks to me like workers are to blame. Do you
>> have any custom scripts? Do you use the stock scan.bro or any other
>> SumStats scripts?
>>
>> Can you observe the system for a while, in something like htop, or top
>> with threads, to see what kind of processes eat most of memory and report
>> back?
>>
>>
>> On Wed, Nov 6, 2019 at 11:13 AM Mark Gardner <mkg at vt.edu> wrote:
>>
>>> On Wed, Nov 6, 2019 at 1:13 PM Jeff Barber <jsbarber60 at gmail.com> wrote:
>>>
>>>> > I'm running Zeek 2.6.4 and I have been seeing occasional error
>>>> messages of the form:
>>>> >
>>>> > run-bro: line 110: 42574 Killed                  nohup
>>>> ${pin_command} $pin_cpu "$mybro" "$@"
>>>> >
>>>> > The workers seem to be restarted fine and other than the error
>>>> message, I haven't noticed any ill behavior. What should I do about the
>>>> error messages?
>>>>
>>>> I would check your syslog. Assuming you are running linux, if your
>>>> system runs out of memory, the kernel will go find the biggest process
>>>> and kill it. This can often be a zeek process as they tend to grow large as
>>>> more connections are tracked (depending on many factors: what scripts you
>>>> are running, what you're logging, what kind of traffic is being seen,
>>>> etc.). If that's happening, you should see something in syslog containing
>>>> the string "invoked oom-killer:" If you look at the surrounding lines,
>>>> there should be some info on process sizes showing why it was selected.
>>>>
>>>
>>> Good catch Jeff.  A crash email arrived at 10:10 and the journal entry
>>> starting around 10:07 contained the following:
>>>
>>> Nov 06 10:07:08 sensor1 kernel: oom_reaper: reaped process 43437 (bro),
>>> now anon-rss:0kB, file-rss:1048576kB, shmem-rss:0kB
>>> Nov 06 10:07:05 sensor1 kernel: Killed process 43437 (bro)
>>> total-vm:67116888kB, anon-rss:57339476kB, file-rss:1042296kB, shmem-rss:0kB
>>> Nov 06 10:07:05 sensor1 kernel: Out of memory: Kill process 43437 (bro)
>>> score 444 or sacrifice child
>>> Nov 06 10:07:05 sensor1 kernel: [  54824]    47 54824  1516502  1221530
>>> 11284480       80             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  54817]    47 54817     1661       56
>>>    49152       16             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51543]    47 51543  1516502  1281056
>>> 11886592     3616             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51542]    47 51542  1516502  1285841
>>> 11964416     2296             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51539]    47 51539  1516502  1277957
>>> 11890688     2817             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51537]    47 51537  1516502  1282107
>>> 11829248     2543             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51533]    47 51533  1516502  1276417
>>> 11939840     7591             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51532]    47 51532  1516502  1290264
>>> 12042240     8006             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51523]    47 51523  1516502  1281160
>>> 11894784     7479             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51521]    47 51521  1516502  1285423
>>> 11984896     7957             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51520]    47 51520  1745878  1370574
>>> 12668928     6908             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51501]    47 51501  1516502  1286290
>>> 11788288     6967             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51499]    47 51499     1661       22
>>>    49152       49             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51498]    47 51498  1516502  1276891
>>> 11882496     7233             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51496]    47 51496  1516502  1287064
>>> 11948032     7685             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51491]    47 51491     1661       18
>>>    53248       55             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51487]    47 51487     1661       18
>>>    57344       54             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51474]    47 51474     1661       21
>>>    49152       52             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51471]    47 51471  1516502  1277820
>>> 11890688     7964             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51465]    47 51465     1661       20
>>>    53248       53             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51455]    47 51455     1661       23
>>>    57344       49             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51440]    47 51440     1661       18
>>>    49152       54             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51438]    47 51438     1661       23
>>>    49152       49             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51428]    47 51428     1661       18
>>>    49152       54             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51414]    47 51414     1661       16
>>>    53248       56             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51405]    47 51405     1661       20
>>>    53248       52             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51397]    47 51397     1661       14
>>>    45056       59             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  51375]    47 51375     1661       15
>>>    49152       58             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  43437]    47 43437 16779222 14595446
>>> 122212352   129972             0 bro
>>> Nov 06 10:07:05 sensor1 kernel: [  43343]    47 43343     1661        2
>>>    49152       70             0 run-bro
>>> Nov 06 10:07:05 sensor1 kernel: [  43183]    47 43183    42625        4
>>>   106496      608             0 (sd-pam)
>>> Nov 06 10:07:05 sensor1 kernel: [  43181]    47 43181     5290        0
>>>    77824      342             0 systemd
>>> Nov 06 10:07:05 sensor1 kernel: [    996]     0   996     1346        0
>>>    49152       33             0 agetty
>>> Nov 06 10:07:05 sensor1 kernel: [    995]     0   995     1403        0
>>>    45056       32             0 agetty
>>> Nov 06 10:07:05 sensor1 kernel: [    994]     0   994     3963       38
>>>    69632      178         -1000 sshd
>>> Nov 06 10:07:05 sensor1 kernel: [    991]     0   991     2119       20
>>>    53248       35             0 cron
>>> Nov 06 10:07:05 sensor1 kernel: [    990]   105   990     2210       97
>>>    57344       55          -900 dbus-daemon
>>> Nov 06 10:07:05 sensor1 kernel: [    977]     0   977    56456       90
>>>    86016      378             0 rsyslogd
>>> Nov 06 10:07:05 sensor1 kernel: [    968]     0   968     4875       78
>>>    77824      189             0 systemd-logind
>>> Nov 06 10:07:05 sensor1 kernel: [    944]     0   944     2021        1
>>>    57344      799             0 haveged
>>> Nov 06 10:07:05 sensor1 kernel: [    931]   100   931    23270       26
>>>    86016      207             0 systemd-timesyn
>>> Nov 06 10:07:05 sensor1 kernel: [    640]     0   640     5579       14
>>>    69632      309         -1000 systemd-udevd
>>> Nov 06 10:07:05 sensor1 kernel: [    615]     0   615    74225     2624
>>>   638976      568             0 systemd-journal
>>> Nov 06 10:07:05 sensor1 kernel: [  pid  ]   uid  tgid total_vm      rss
>>> pgtables_bytes swapents oom_score_adj name
>>> Nov 06 10:07:05 sensor1 kernel: Tasks state (memory values in pages):
>>> Nov 06 10:07:05 sensor1 kernel: 0 pages hwpoisoned
>>> Nov 06 10:07:05 sensor1 kernel: 551979 pages reserved
>>> Nov 06 10:07:05 sensor1 kernel: 0 pages HighMem/MovableOnly
>>> Nov 06 10:07:05 sensor1 kernel: 33528566 pages RAM
>>> Nov 06 10:07:05 sensor1 kernel: Total swap = 976892kB
>>> Nov 06 10:07:05 sensor1 kernel: Free swap  = 0kB
>>> Nov 06 10:07:05 sensor1 kernel: Swap cache stats: add 759161, delete
>>> 711025, find 25951/48832
>>> Nov 06 10:07:05 sensor1 kernel: 48128 pages in swap cache
>>> Nov 06 10:07:05 sensor1 kernel: 52774 total pagecache pages
>>> Nov 06 10:07:05 sensor1 kernel: Node 3 hugepages_total=0
>>> hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
>>> Nov 06 10:07:05 sensor1 kernel: Node 3 hugepages_total=0
>>> hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
>>> Nov 06 10:07:05 sensor1 kernel: Node 1 hugepages_total=0
>>> hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
>>> Nov 06 10:07:05 sensor1 kernel: Node 1 hugepages_total=0
>>> hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
>>> Nov 06 10:07:05 sensor1 kernel: Node 3 Normal: 73*4kB (U) 150*8kB (UH)
>>> 59*16kB (UH) 1325*32kB (UH) 5*64kB (H) 2*128kB (H) 0*256kB 0*512kB 0*1024kB
>>> 0*2048kB 0*4096kB = 45412kB
>>> Nov 06 10:07:05 sensor1 kernel: Node 1 Normal: 78*4kB (UMEH) 213*8kB
>>> (UEH) 70*16kB (UH) 1232*32kB (UMH) 7*64kB (H) 0*128kB 0*256kB 0*512kB
>>> 0*1024kB 0*2048kB 0*4096kB = 43008kB
>>> Nov 06 10:07:05 sensor1 kernel: Node 1 DMA32: 697*4kB (UME) 647*8kB
>>> (UME) 392*16kB (UME) 190*32kB (UME) 94*64kB (UE) 51*128kB (UE) 63*256kB
>>> (UME) 240*512kB (UME) 72*1024kB (UME) 0*2048kB 0*4096kB = 245596kB
>>> Nov 06 10:07:05 sensor1 kernel: Node 1 DMA: 1*4kB (U) 1*8kB (U) 2*16kB
>>> (U) 1*32kB (U) 3*64kB (U) 0*128kB 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB
>>> (M) 3*4096kB (M) = 15884kB
>>> Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 0 0 0 0
>>> Nov 06 10:07:05 sensor1 kernel: Node 3 Normal free:44844kB min:45116kB
>>> low:111144kB high:177172kB active_anon:48629496kB inactive_anon:3250728kB
>>> active_file:64kB inactive_file:668kB unevictable:0kB writepending:0kB
>>> present:67094528kB managed:66031300kB mlocked:0kB kernel_stack:5352kB
>>> pagetables:146976kB bounce
>>> Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 0 0 0 0
>>> Nov 06 10:07:05 sensor1 kernel: Node 1 Normal free:41932kB min:42604kB
>>> low:104956kB high:167308kB active_anon:55633904kB inactive_anon:4082748kB
>>> active_file:560kB inactive_file:0kB unevictable:0kB writepending:0kB
>>> present:63438848kB managed:62359848kB mlocked:0kB kernel_stack:4776kB
>>> pagetables:133008kB bounce:
>>> Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 0 60894 60894 60894
>>> Nov 06 10:07:05 sensor1 kernel: Node 1 DMA32 free:245356kB min:2372kB
>>> low:5844kB high:9316kB active_anon:3199852kB inactive_anon:44556kB
>>> active_file:100kB inactive_file:0kB unevictable:0kB writepending:0kB
>>> present:3564892kB managed:3499316kB mlocked:0kB kernel_stack:48kB
>>> pagetables:3092kB bounce:0kB free_pcp:4
>>> Nov 06 10:07:05 sensor1 kernel: lowmem_reserve[]: 0 3392 64286 64286
>>> 64286
>>> Nov 06 10:07:05 sensor1 kernel: Node 1 DMA free:15884kB min:8kB low:20kB
>>> high:32kB active_anon:0kB inactive_anon:0kB active_file:0kB
>>> inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB
>>> managed:15884kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB
>>> free_pcp:0kB local_pcp:0kB free_cma:0kB
>>> Nov 06 10:07:05 sensor1 kernel: Node 3 active_anon:48629496kB
>>> inactive_anon:3250728kB active_file:108kB inactive_file:700kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:13636088kB
>>> dirty:0kB writeback:0kB shmem:9996kB shmem_thp: 0kB shmem_pmdmapped: 0kB
>>> anon_thp: 1337344kB writeback_tmp:0kB unst
>>> Nov 06 10:07:05 sensor1 kernel: Node 1 active_anon:58833756kB
>>> inactive_anon:4127304kB active_file:548kB inactive_file:0kB unevictable:0kB
>>> isolated(anon):0kB isolated(file):0kB mapped:2101152kB dirty:0kB
>>> writeback:0kB shmem:7828kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp:
>>> 13608960kB writeback_tmp:0kB unstab
>>> Nov 06 10:07:05 sensor1 kernel: active_anon:26865813
>>> inactive_anon:1844508 isolated_anon:0
>>>                                  active_file:163 inactive_file:0
>>> isolated_file:0
>>>                                  unevictable:0 dirty:0 writeback:0
>>> unstable:0
>>>                                  slab_reclaimable:25057
>>> slab_unreclaimable:65745
>>>                                  mapped:3934289 shmem:4456
>>> pagetables:70769 bounce:0
>>>                                  free:87073 free_pcp:3080 free_cma:0
>>> Nov 06 10:07:05 sensor1 kernel: Mem-Info:
>>> Nov 06 10:07:05 sensor1 kernel: R13: 0000000000001800 R14:
>>> 00007f375bd4f780 R15: 0000000000000022
>>> Nov 06 10:07:05 sensor1 kernel: R10: 00007f3618971000 R11:
>>> 00007f36189727e0 R12: 00007f375bd4f940
>>> Nov 06 10:07:05 sensor1 kernel: RBP: 00007f361424b000 R08:
>>> 0000000000000000 R09: 00007f36189727e0
>>> Nov 06 10:07:05 sensor1 kernel: RDX: 0000000000001780 RSI:
>>> 00007f361424b000 RDI: 00007f3618971000
>>> Nov 06 10:07:05 sensor1 kernel: RAX: 00007f3618971000 RBX:
>>> 0000000000001800 RCX: 00007f361424c760
>>> Nov 06 10:07:05 sensor1 kernel: RSP: 002b:00007ffd609fbc58 EFLAGS:
>>> 00010202
>>> Nov 06 10:07:05 sensor1 kernel: Code: Bad RIP value.
>>> Nov 06 10:07:05 sensor1 kernel: RIP: 0033:0x7f375bf1f925
>>> Nov 06 10:07:05 sensor1 kernel:  page_fault+0x1e/0x30
>>> Nov 06 10:07:05 sensor1 kernel:  ? page_fault+0x8/0x30
>>> Nov 06 10:07:05 sensor1 kernel:  __do_page_fault+0x249/0x4f0
>>> Nov 06 10:07:05 sensor1 kernel:  handle_mm_fault+0xd6/0x200
>>> Nov 06 10:07:05 sensor1 kernel:  ? __switch_to_asm+0x41/0x70
>>> Nov 06 10:07:05 sensor1 kernel:  __handle_mm_fault+0x958/0x1270
>>> Nov 06 10:07:05 sensor1 kernel:  alloc_pages_vma+0x74/0x1c0
>>> Nov 06 10:07:05 sensor1 kernel:  __alloc_pages_nodemask+0x28b/0x2b0
>>> Nov 06 10:07:05 sensor1 kernel:  __alloc_pages_slowpath+0xbd8/0xcb0
>>> Nov 06 10:07:05 sensor1 kernel:  out_of_memory+0x1a5/0x430
>>> Nov 06 10:07:05 sensor1 kernel:  oom_kill_process.cold.30+0xb/0x1cf
>>> Nov 06 10:07:05 sensor1 kernel:  dump_header+0x6b/0x283
>>> Nov 06 10:07:05 sensor1 kernel:  dump_stack+0x5c/0x80
>>> Nov 06 10:07:05 sensor1 kernel: Call Trace:
>>> Nov 06 10:07:05 sensor1 kernel: Hardware name: Supermicro Super
>>> Server/H11SSL-i, BIOS 1.0b 04/27/2018
>>> Nov 06 10:07:05 sensor1 kernel: CPU: 3 PID: 51496 Comm: bro Not tainted
>>> 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u1
>>> Nov 06 10:07:05 sensor1 kernel: bro cpuset=/ mems_allowed=1,3
>>> Nov 06 10:07:05 sensor1 kernel: bro invoked oom-killer:
>>> gfp_mask=0x6280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null),
>>> order=0, oom_score_adj=0
>>>
>>> It looks like bro was using 64 GB of RAM (summation of even though the
>>> system has 128 GB installed. (It also looks like there was no free swap.)
>>> Searching the Internet for ways to solve OOM, people suggested turning off
>>> over-commit, and configuring the overcommit ratio a bit higher. Currently,
>>> the values are:
>>>
>>> # sysctl vm.overcommit_memory
>>> vm.overcommit_memory = 0
>>> # sysctl vm.overcommit_ratio
>>> vm.overcommit_ratio = 50
>>>
>>> In one case the suggestion was to change those to vm.overcommit_memory
>>> = 1 and vm.overcommit_ratio = 80 which seems reasonable based on the
>>> description at
>>> https://www.kernel.org/doc/Documentation/vm/overcommit-accounting.
>>>
>>> Or is there a better way to handle OOM?
>>>
>>> Mark
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191106/d53434e2/attachment-0001.html 

From mauro.palumbo at aizoon.it  Thu Nov  7 00:46:11 2019
From: mauro.palumbo at aizoon.it (Palumbo Mauro)
Date: Thu, 7 Nov 2019 08:46:11 +0000
Subject: [Zeek] R:  uid in files logs
In-Reply-To: <CAJ6bFK0A50BBxToy0dqqc85=XmyR-BrjSFDP6EWguQ-+PpL=wg@mail.gmail.com>
References: <b474e395ba664b5fb7a6072d42ed6c18@SRVEX03.aizoon.local>
	<CAJ6bFK0A50BBxToy0dqqc85=XmyR-BrjSFDP6EWguQ-+PpL=wg@mail.gmail.com>
Message-ID: <eef37ab57bc9451eb296478f20850714@SRVEX03.aizoon.local>

Hi Michal,
   thanks, it seems to me pretty easy to implement and corresponding uids for each file are already stored in the record fa_file.conns.

I believe the only reason not to include these uids in pe.log, x5009.log, etc. is that it is already available elsewhere and in general it is better to avoid duplicates. It is one extra columns, which will take some memory/disk space, etc. Unless there would be a significant performance hit I can?t see.

For us, adding this extra column will just to make our lives a bit easier in everyday work?

Mauro

Da: Micha? Purzy?ski [mailto:michalpurzynski1 at gmail.com]
Inviato: gioved? 7 novembre 2019 00:46
A: Palumbo Mauro <mauro.palumbo at aizoon.it>
Cc: zeek <zeek at zeek.org>
Oggetto: Re: [Zeek] uid in files logs

While I have no idea why it's not default, I'll share a piece of code to achieve something similar, so you can adopt it to your needs

Here we wanted to kill logging X509 certificates into both files.log and x509.log - and by doing that we saved like 20% of our SIEM intake, globally (!!). Should be easy enough to extend x509.log to include data from conn.log, etc.

@load base/frameworks/files
@load base/files/hash

module X509;

export {
    redef record X509::Info += {
        fuid: string &log &optional;
        md5: string &log &optional;
    };
}

event file_state_remove(f: fa_file) &priority=40
{
    if ( ! f$info?$x509 )
        return;

    f$info$x509$fuid = f$info$fuid;
    f$info$x509$md5 = f$info$md5;
}

On Wed, Nov 6, 2019 at 2:17 AM Palumbo Mauro <mauro.palumbo at aizoon.it<mailto:mauro.palumbo at aizoon.it>> wrote:
Hi everybody,
   it would be useful for us to have the conn uids in the logs from file analyzers (pe.log, x509.log,?). I know this information can be gathered by cross-cehcking different bro logs, but it will save some time to have it already in pe.log, etc. I believe this data is available in the record fa_file.conns, available in events in the file framework, so it seems not difficult to add.
Is there any reason why it is not added by default?

Thanks,
Mauro
_______________________________________________
Zeek mailing list
zeek at zeek.org<mailto:zeek at zeek.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191107/73e477c2/attachment.html 

From mfernandez at mitre.org  Thu Nov  7 04:22:25 2019
From: mfernandez at mitre.org (Fernandez, Mark I)
Date: Thu, 7 Nov 2019 12:22:25 +0000
Subject: [Zeek] monitoring proxied web traffic
Message-ID: <MN2PR09MB404633E90CBAEA14B5EC6BD3CF780@MN2PR09MB4046.namprd09.prod.outlook.com>

Konrad,

Does your proxy also communicate with a content-inspection device, like for 
anti-virus inspection of web content?  If so, there may be a way to correlate. 
The web proxy would use the Internet Content Adaptation Protocol (ICAP) to 
encapsulate the HTTP/HTTPS traffic to send to the anti-virus server for 
inspection.  I wrote a protocol analyzer for ICAP.  This protocol is very 
similar in syntax to HTTP, and it contains header fields (supported by most 
web proxy vendors) called "X-Client-IP" and "X-Server-IP" which correspond to 
the original IP addresses of the local web client and the remote web server, 
respectively.  Please see my presentation from BroCon 2016, perhaps it 
applies:

https://www.zeek.org/community/brocon2016.html

Mark

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5100 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191107/3d12d0db/attachment-0001.bin 

From clopmz at outlook.com  Thu Nov  7 06:26:33 2019
From: clopmz at outlook.com (Carlos Lopez)
Date: Thu, 7 Nov 2019 14:26:33 +0000
Subject: [Zeek] monitoring proxied web traffic
In-Reply-To: <MN2PR09MB404633E90CBAEA14B5EC6BD3CF780@MN2PR09MB4046.namprd09.prod.outlook.com>
References: <MN2PR09MB404633E90CBAEA14B5EC6BD3CF780@MN2PR09MB4046.namprd09.prod.outlook.com>
Message-ID: <DB8PR10MB359519358F7640E6178F55E7DB780@DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM>

Mark,

 From where can we download the source code (ICAP analyzer)?



Regards,
C. L. Martinez


________________________________________
From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> on behalf of Fernandez, Mark I <mfernandez at mitre.org>
Sent: 07 November 2019 13:22
To: Konrad Weglowski; zeek
Subject: Re: [Zeek] monitoring proxied web traffic

Konrad,

Does your proxy also communicate with a content-inspection device, like for
anti-virus inspection of web content?  If so, there may be a way to correlate.
The web proxy would use the Internet Content Adaptation Protocol (ICAP) to
encapsulate the HTTP/HTTPS traffic to send to the anti-virus server for
inspection.  I wrote a protocol analyzer for ICAP.  This protocol is very
similar in syntax to HTTP, and it contains header fields (supported by most
web proxy vendors) called "X-Client-IP" and "X-Server-IP" which correspond to
the original IP addresses of the local web client and the remote web server,
respectively.  Please see my presentation from BroCon 2016, perhaps it
applies:

https://www.zeek.org/community/brocon2016.html

Mark



From mkg at vt.edu  Thu Nov  7 06:27:03 2019
From: mkg at vt.edu (Mark Gardner)
Date: Thu, 7 Nov 2019 09:27:03 -0500
Subject: [Zeek] Worker being "killed nohup"
In-Reply-To: <CAJ6bFK0k9XY4uaEgWD-2tGdd094Gajyu7GH+7vTKxhgo+ifvuA@mail.gmail.com>
References: <CAG=_CgA4txhcGmfppTnkOwU8vKWjoEkEfLNTrOtPeLvC6dXxNg@mail.gmail.com>
	<CABLqKGLvnu_7FH-6qa3oxOq1ehctX-4VW1n2aX=DffseAiix9w@mail.gmail.com>
	<CAG=_CgDOL1QabA83NuQTjjyDpPndcCf7wquCOrBMJuTd75JMrg@mail.gmail.com>
	<CAJ6bFK2FZWODtTeUk1UuCJ-QxKED7Z0ckRA2Xxb_U3AthtoJtQ@mail.gmail.com>
	<CAEzr=8OeKVoRZFE+HSwFdQ3wMtW3nuKGjVZ9YY=hfKTywoFd=g@mail.gmail.com>
	<CAJ6bFK0k9XY4uaEgWD-2tGdd094Gajyu7GH+7vTKxhgo+ifvuA@mail.gmail.com>
Message-ID: <CAG=_CgBv1ASy1shVThdJMm51x4X4RL9pmUTsXsQb9aTtkwCt8A@mail.gmail.com>

On Wed, Nov 6, 2019 at 10:22 PM Micha? Purzy?ski <michalpurzynski1 at gmail.com>
wrote:

> BTW is your Zeek built with tcmalloc? I heard horror stories about that
> now I just use stock glibc malloc or jemalloc.
>

I've heard those stories too. I build with jemalloc.

Mark
-- 
Mark Gardner
--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191107/431cf1a5/attachment.html 

From mkg at vt.edu  Thu Nov  7 07:37:18 2019
From: mkg at vt.edu (Mark Gardner)
Date: Thu, 7 Nov 2019 10:37:18 -0500
Subject: [Zeek] Worker being "killed nohup"
In-Reply-To: <CAJ6bFK2_5nQODU57jRTrnHtU3ZoJw5ZnROhAexj83Lq+RHcDpw@mail.gmail.com>
References: <CAG=_CgA4txhcGmfppTnkOwU8vKWjoEkEfLNTrOtPeLvC6dXxNg@mail.gmail.com>
	<CABLqKGLvnu_7FH-6qa3oxOq1ehctX-4VW1n2aX=DffseAiix9w@mail.gmail.com>
	<CAG=_CgDOL1QabA83NuQTjjyDpPndcCf7wquCOrBMJuTd75JMrg@mail.gmail.com>
	<CAJ6bFK2FZWODtTeUk1UuCJ-QxKED7Z0ckRA2Xxb_U3AthtoJtQ@mail.gmail.com>
	<CAEzr=8OeKVoRZFE+HSwFdQ3wMtW3nuKGjVZ9YY=hfKTywoFd=g@mail.gmail.com>
	<CAJ6bFK2_5nQODU57jRTrnHtU3ZoJw5ZnROhAexj83Lq+RHcDpw@mail.gmail.com>
Message-ID: <CAG=_CgAxvHvVjH8_qRkwztQR871CNUJwxj1adePE6v7RMsDfKQ@mail.gmail.com>

Your comments have been most helpful, Micha?. I am very appreciative.

On Wed, Nov 6, 2019 at 10:18 PM Micha? Purzy?ski <michalpurzynski1 at gmail.com>
wrote:

> Now, this data expires (unless you have a script that never does that),
> but it might be the amount of state grows too quickly and the expiration is
> not quick enough, to free up some memory.
>

One hypothesis is that we have lots of very long running connections.
Looking through the first 1M connections in the largest 1 hour conn.log for
2019-10-21 (the biggest log in the past month), the min, max, and average
duration of connections are 0.0, 16750, and 37.4s (with stdev of 285.5)
respectively. 66.9% of the connections last less than a second and the
percentage of connections lasting to the mean duration is 86.4%. So the
number of long running connections seems small.

So it doesn't seem like there are tons of long running connections. We just
have many connections. (73,403,294 during the single hour mentioned above.)

> My quick suspect would be the scan.bro / scan.zeek old script that comes
> bundled with Zeek. If you have it enabled, disable and see if you're still
> crashing.
>

We commented out @load misc/scan in bro-2.6.4/share/bro/site/local.bro. Is
that what you meant?

You can then take a look at your scripts and see if there is some data
> structure that will grow per connection, over time - and how quickly you
> purge data from it.
>

The only extra configuration we added to local.bro was @load
tuning/json-logs. But I wouldn't think that would cause a large increase in
memory use (even though it does cause an increase in written file size).

One of the things I did yesterday was add 128 GB of swap file on NVMe to
the workers to augment the 1 GB swap partition already in place. That seems
to be helping. The one sensor I checked this morning was using 6.5 GB of
swap. Yesterday it would have crash exceeding 1 GB swap. So maybe I don't
need to worry about long running connections if I have enough swap.

I need to set up monitoring on the cluster to make it easier to diagnose
these kinds of problems.

Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191107/a9bcec0b/attachment.html 

From konrad.weglowski at gmail.com  Thu Nov  7 10:05:48 2019
From: konrad.weglowski at gmail.com (Konrad Weglowski)
Date: Thu, 7 Nov 2019 13:05:48 -0500
Subject: [Zeek] monitoring proxied web traffic
In-Reply-To: <CAJ6bFK2DghGOc=o+mcUmgpW7CfxN651TSTuY8tJ+6XPR+OZpNA@mail.gmail.com>
References: <CA+fGMJRb6kDVcn-1BFZ=hgwLfvM4MWaKdw=jR_BkrB=ymQz8DQ@mail.gmail.com>
	<CAJ6bFK2DghGOc=o+mcUmgpW7CfxN651TSTuY8tJ+6XPR+OZpNA@mail.gmail.com>
Message-ID: <CA+fGMJTuGcQuJAUw3tqvr786FU9SRSaRCDo7aoQUWZipHu-+HA@mail.gmail.com>

Thank You Micha?

Client to Proxy would not be encrypted. So based on that I will have all
the logs, however my destination IP for proxied connections would be proxy
IP (that was my understanding). I guess ingesting proxy logs would then
provide full visibility as you mentioned. I am thinking maybe if for some
reason I cannot get the proxy logs ingested and also tap after the proxy, I
could possibly correlate both ends of the connections based on URI/SNI/etc
and same timestamps in my analytics platform. Anyways I will give that a
try as well. Thanks for your help again.

Konrad



On Wed, Nov 6, 2019 at 6:59 PM Micha? Purzy?ski <michalpurzynski1 at gmail.com>
wrote:

> There is no way for Zeek to really correlate if the proxy does everything
> at the application level, like Squid. The proxy
>  terminates the connection and then it's free to start the second leg, to
> the destination host, as it wishes to.
>
> Is your traffic from clients to proxy encrypted as well?
>
> If the traffic between clients are the proxy is not encrypted (as it's
> usually the case) then even if the client will establish a TLS session
> _through_ the proxy you will see a destination with the "service" field
> http,ssl and all the conn + ssl + http logs.
>
> We have that + proxy logs and that's pretty much a full visibility.
>
>
>
> On Wed, Nov 6, 2019 at 1:11 PM Konrad Weglowski <
> konrad.weglowski at gmail.com> wrote:
>
>> Hello,
>>
>> I need to monitor web traffic with Zeek that is going through an implicit
>> web proxy. I would like to be able to see real client IPs in which case tap
>> would be placed before the proxy, however I will not be able to see the
>> real destination IP as it will always be proxy IP. I can also tap on the
>> other side of the proxy where source will always be proxy IP at the same
>> time.
>>
>> Is there a way with Zeek to correlate the two sessions (before and after
>> proxy) somehow?
>>
>> I realize that UID or Community ID are probably not going do it since
>> different source/destinations pairs for each for each session, but maybe
>> based on the timestamp/URI/SNI/byte or packet counts/etc?
>>
>> I was also thinking "X-Forwarded-For" header but I guess that would only
>> work for HTTP and not HTTPS connections.
>>
>> Thank You,
>>
>> Konrad
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191107/64d76aa4/attachment-0001.html 

From konrad.weglowski at gmail.com  Thu Nov  7 10:07:29 2019
From: konrad.weglowski at gmail.com (Konrad Weglowski)
Date: Thu, 7 Nov 2019 13:07:29 -0500
Subject: [Zeek] monitoring proxied web traffic
In-Reply-To: <MN2PR09MB404633E90CBAEA14B5EC6BD3CF780@MN2PR09MB4046.namprd09.prod.outlook.com>
References: <MN2PR09MB404633E90CBAEA14B5EC6BD3CF780@MN2PR09MB4046.namprd09.prod.outlook.com>
Message-ID: <CA+fGMJTqLR7Zb3wwMQR3Y+PWpvXEF5tp3NFV-gYUTdR43zjLdg@mail.gmail.com>

Thank you Mark

Proxy itself is doing content inspection/etc so I won't be able to capture
it that way.

On Thu, Nov 7, 2019 at 7:23 AM Fernandez, Mark I <mfernandez at mitre.org>
wrote:

> Konrad,
>
> Does your proxy also communicate with a content-inspection device, like
> for
> anti-virus inspection of web content?  If so, there may be a way to
> correlate.
> The web proxy would use the Internet Content Adaptation Protocol (ICAP) to
> encapsulate the HTTP/HTTPS traffic to send to the anti-virus server for
> inspection.  I wrote a protocol analyzer for ICAP.  This protocol is very
> similar in syntax to HTTP, and it contains header fields (supported by
> most
> web proxy vendors) called "X-Client-IP" and "X-Server-IP" which correspond
> to
> the original IP addresses of the local web client and the remote web
> server,
> respectively.  Please see my presentation from BroCon 2016, perhaps it
> applies:
>
> https://www.zeek.org/community/brocon2016.html
>
> Mark
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191107/d47df169/attachment.html 

From michalpurzynski1 at gmail.com  Fri Nov  8 00:21:05 2019
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Fri, 8 Nov 2019 00:21:05 -0800
Subject: [Zeek] monitoring proxied web traffic
In-Reply-To: <CA+fGMJTuGcQuJAUw3tqvr786FU9SRSaRCDo7aoQUWZipHu-+HA@mail.gmail.com>
References: <CA+fGMJRb6kDVcn-1BFZ=hgwLfvM4MWaKdw=jR_BkrB=ymQz8DQ@mail.gmail.com>
	<CAJ6bFK2DghGOc=o+mcUmgpW7CfxN651TSTuY8tJ+6XPR+OZpNA@mail.gmail.com>
	<CA+fGMJTuGcQuJAUw3tqvr786FU9SRSaRCDo7aoQUWZipHu-+HA@mail.gmail.com>
Message-ID: <CAJ6bFK13smRKz-qdJYF08pTR8wTeJAaE01qBfL7r4tkunCeJtg@mail.gmail.com>

You're welcome!!

If the traffic from the client to the proxy is not encrypted and the proxy
is not "transparent" (meaning you configured it on the client) then you
have the same case as I do here.

So there's this host
source IP 10.48.75.81

trying to talk to clr.telemetry.intel.com

via our proxy at 10.48.122.10 / 10.48.74.16 / 10.48.74.17

*(for anyone working at a place where disclosing an internal IP or an FQDN
equal incident - you know a lot of great places are hiring, right?)*

I used the "uid" to correlate this connection across three types of logs -
conn.log, http.log and ssl.log. You're gonna get them due to the way
proxies operate.

Please ignore tons of metadata from our SIEM (MozDef = custom Python + pure
ES), I'm going to C&P three logs as they are without any redactions. This
post is HUGE!!

*conn.log* - host -> proxy

{
  "_index": "events-20191108",
  "_type": "_doc",
  "_id": "JmLRSW4BiWsbe3JQtC8S",
  "_version": 1,
  "_score": null,
  "_source": {
    "type": "nsm",
    "details": {
      "ts": 1573196540.178897,
*      "uid": "C9PFt711Odfwpbr0Ti",*
*      "proto": "tcp",*
*      "service": "http,ssl",*
      "duration": 0.183212,
      "orig_bytes": 1604,
      "resp_bytes": 7180,
      "conn_state": "SF",
      "local_orig": true,
      "local_resp": true,
      "missed_bytes": 0,
*      "history": "ShADadFf",*
*      "orig_pkts": 14,*
*      "resp_pkts": 13,*
      "peer": "nsm2-af_packet-enp18s0f0-8",
*      "orig_l2_addr": "b4:0c:25:e0:40:10",*
*      "resp_l2_addr": "00:50:56:a1:48:64",*
*      "sourceipaddress": "10.48.75.81",*
*      "sourceport": 55330,*
*      "destinationipaddress": "10.48.122.10",*
*      "destinationport": 3128,*
*      "originipbytes": 2340,*
*      "responseipbytes": 7864,*
      "sourceipv4address": "10.48.75.81",
      "destinationipv4address": "10.48.122.10"
    },
    "customendpoint": "bro",
    "hostname": "nsm2.private.mdc1.xxxxxxxxxx.com",
    "tags": "bro",
    "category": "bro",
*    "source": "conn",*
*    "utctimestamp": "2019-11-08T07:02:20.178897+00:00",*
    "timestamp": "2019-11-08T07:02:20.178897+00:00",
    "receivedtimestamp": "2019-11-08T07:02:27.686061+00:00",
    "eventsource": "nsm",
    "severity": "INFO",
    "mozdefhostname": "mozdef5.private.mdc1.xxxxxxxxxx.com",
    "summary": "10.48.75.81:55330 -> 10.48.122.10:3128 ShADadFf 2340 bytes
/ 7864 bytes",
    "plugins": [
      "customDocType",
      "broFixup",
      "ipFixup",
      "geoip"
    ],
    "processid": "UNKNOWN",
    "processname": "UNKNOWN"
  },
  "fields": {
    "receivedtimestamp": [
      "2019-11-08T07:02:27.686Z"
    ],
    "timestamp": [
      "2019-11-08T07:02:20.178Z"
    ],
    "utctimestamp": [
      "2019-11-08T07:02:20.178Z"
    ]
  },
  "highlight": {
    "summary": [
      "@kibana-highlighted-field at 10.48.75.81@/kibana-highlighted-field@:55330
-> 10.48.122.10:3128 ShADadFf 2340 bytes / 7864 bytes"
    ],
    "details.sourceipv4address": [
      "@kibana-highlighted-field at 10.48.75.81@/kibana-highlighted-field@"
    ],
    "details.uid": [
      "@kibana-highlighted-field at C9PFt711Odfwpbr0Ti
@/kibana-highlighted-field@"
    ],
    "category": [
      "@kibana-highlighted-field at bro@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1573196540178
  ]
}

*http.log* (because the connection from the client to the proxy was
technically a http connection)

{
  "_index": "events-20191108",
  "_type": "_doc",
  "_id": "jPDRSW4BC2j90LU9UdgL",
  "_version": 1,
  "_score": null,
  "_source": {
    "type": "nsm",
    "details": {
      "ts": 1573196540.202888,
*      "uid": "C9PFt711Odfwpbr0Ti",*
      "trans_depth": 1,
      "method": "CONNECT",

*"host": "clr.telemetry.intel.com <http://clr.telemetry.intel.com>",
"uri": "clr.telemetry.intel.com:443 <http://clr.telemetry.intel.com:443>",*
      "version": "1.1",
      "request_body_len": 0,
      "response_body_len": 0,
*      "status_code": 200,*
*      "status_msg": "Connection established",*
      "tags": [],
      "proxied": [
        "PROXY-CONNECTION -> Keep-Alive"
      ],



*"sourceipaddress": "10.48.75.81",      "sourceport": 55330,
"destinationipaddress": "10.48.122.10",      "destinationport": 3128,*
      "sourceipv4address": "10.48.75.81",
      "destinationipv4address": "10.48.122.10"
    },
    "customendpoint": "bro",
    "hostname": "nsm2.private.mdc1.xxxxxxxxxx.com",
    "tags": "bro",
    "category": "bro",
*    "source": "http",*
*    "utctimestamp": "2019-11-08T07:02:20.202888+00:00",*
    "timestamp": "2019-11-08T07:02:20.202888+00:00",
    "receivedtimestamp": "2019-11-08T07:02:21.427537+00:00",
    "eventsource": "nsm",
    "severity": "INFO",
    "mozdefhostname": "mozdef7.private.mdc1.xxxxxxxxxx.com",
    "summary": "HTTP CONNECT 10.48.75.81 -> 10.48.122.10:3128",
    "plugins": [
      "customDocType",
      "broFixup",
      "ipFixup",
      "geoip"
    ],
    "processid": "UNKNOWN",
    "processname": "UNKNOWN"
  },
  "fields": {
    "receivedtimestamp": [
      "2019-11-08T07:02:21.427Z"
    ],
    "timestamp": [
      "2019-11-08T07:02:20.202Z"
    ],
    "utctimestamp": [
      "2019-11-08T07:02:20.202Z"
    ]
  },
  "highlight": {
    "summary": [
      "HTTP CONNECT @kibana-highlighted-field at 10.48.75.81
@/kibana-highlighted-field@ -> 10.48.122.10:3128"
    ],
    "details.sourceipv4address": [
      "@kibana-highlighted-field at 10.48.75.81@/kibana-highlighted-field@"
    ],
    "details.uid": [
      "@kibana-highlighted-field at C9PFt711Odfwpbr0Ti
@/kibana-highlighted-field@"
    ],
    "category": [
      "@kibana-highlighted-field at bro@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1573196540202
  ]
}

And the *ssl.log* - over the previously established http connection the
client told proxy to establish a tunnel to the destination host - and Zeek
parsed that as TLS (find me another IDS/NSM that can do that)

{
  "_index": "events-20191108",
  "_type": "_doc",
  "_id": "sfDRSW4BC2j90LU9S8_Q",
  "_version": 1,
  "_score": null,
  "_source": {
    "type": "nsm",
    "details": {
      "ts": 1573196540.253057,













* "uid": "C9PFt711Odfwpbr0Ti",      "version": "TLSv12",      "cipher":
"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",      "curve": "secp256r1",
"server_name": "clr.telemetry.intel.com <http://clr.telemetry.intel.com>",
    "resumed": false,      "established": true,      "cert_chain_fuids": [
      "FknnGH2L2UIKGERW3b",        "FGtbhK1HoQMWdgyKW",
"F8JbGM30JvqFVi41A6",        "Fsc2fRRTyTDQrT1Gc"      ],
"client_cert_chain_fuids": [],*












*"subject": "CN=c-all.telemetry.intel.com
<http://c-all.telemetry.intel.com>,OU=Multi-Domain SSL,OU=Hosted by Intel
Corporation,OU=SSG,O=Intel Corporation,street=2200 Mission College
Blvd,street=FM1-110,L=Santa
Clara,ST=California,postalCode=95054-1537,C=US",      "issuer": "CN=COMODO
RSA Organization Validation Secure Server CA,O=COMODO CA
Limited,L=Salford,ST=Greater Manchester,C=GB",      "validation_status":
"ok",      "server_cert_md5": "27ba7ede8ceab7c896ee4b6bcb509f63",
"server_cert_sha1": "9077d0ce724c0f7bf1866d7cf8c4ec1303dc6c21",      "pfs":
true,      "ja3": "f436b9416f37d134cadd04886327d3e8",      "ja3s":
"303951d4c50efb2e991652225a6f02b1",      "sourceipaddress": "10.48.75.81",
    "sourceport": 55330,      "destinationipaddress": "10.48.122.10",
"destinationport": 3128,*      "sourceipv4address": "10.48.75.81",
      "destinationipv4address": "10.48.122.10"
    },
    "customendpoint": "bro",
    "hostname": "nsm2.private.mdc1.xxxxxxxxxx.com",
    "tags": "bro",
    "category": "bro",
    "source": "ssl",
    "utctimestamp": "2019-11-08T07:02:20.253057+00:00",
    "timestamp": "2019-11-08T07:02:20.253057+00:00",
    "receivedtimestamp": "2019-11-08T07:02:21.464921+00:00",
    "eventsource": "nsm",
    "severity": "INFO",
    "mozdefhostname": "mozdef6.private.mdc1.xxxxxxxxxx.com",
    "summary": "SSL: 10.48.75.81 -> 10.48.122.10:3128",
    "plugins": [
      "customDocType",
      "broFixup",
      "ipFixup",
      "geoip"
    ],
    "processid": "UNKNOWN",
    "processname": "UNKNOWN"
  },
  "fields": {
    "receivedtimestamp": [
      "2019-11-08T07:02:21.464Z"
    ],
    "timestamp": [
      "2019-11-08T07:02:20.253Z"
    ],
    "utctimestamp": [
      "2019-11-08T07:02:20.253Z"
    ]
  },
  "highlight": {
    "summary": [
      "SSL: @kibana-highlighted-field at 10.48.75.81@/kibana-highlighted-field@
-> 10.48.122.10:3128"
    ],
    "details.sourceipv4address": [
      "@kibana-highlighted-field at 10.48.75.81@/kibana-highlighted-field@"
    ],
    "details.uid": [
      "@kibana-highlighted-field at C9PFt711Odfwpbr0Ti
@/kibana-highlighted-field@"
    ],
    "category": [
      "@kibana-highlighted-field at bro@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1573196540253
  ]
}


Not enough? Let's look for the certificate's MD5

{
  "_index": "events-20191108",
  "_type": "_doc",
  "_id": "_CwBSm4BC2j90LU9vfNt",
  "_version": 1,
  "_score": null,
  "_source": {
    "type": "nsm",
    "details": {
      "ts": 1573199701.992707,
      "id": "FdE5qt1b93z9QOJnz1",
      "san.dns": [
        "c-all.telemetry.intel.com",
        "c.telemetry.intel.com",
        "c1.telemetry.intel.com",
        "c2.telemetry.intel.com",
        "c3.telemetry.intel.com",
        "c4.telemetry.intel.com",
        "c5.telemetry.intel.com",
        "c6.telemetry.intel.com",
        "c7.telemetry.intel.com",
        "c8.telemetry.intel.com",
        "c9.telemetry.intel.com",
        "clr.telemetry.intel.com",
        "sur.telemetry.intel.com"
      ],
      "md5": "27ba7ede8ceab7c896ee4b6bcb509f63",
      "sha1": "9077d0ce724c0f7bf1866d7cf8c4ec1303dc6c21",
      "certificate": {
        "basic_constraints_ca": false,
        "exponent": "65537",
        "issuer": "CN=COMODO RSA Organization Validation Secure Server
CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB",
        "key_alg": "rsaEncryption",
        "key_length": 2048,
        "key_type": "rsa",
        "not_valid_after": "2020-08-02T23:59:59+00:00",
        "not_valid_before": "2018-08-03T00:00:00+00:00",
        "sig_alg": "sha256WithRSAEncryption",
        "subject": "CN=c-all.telemetry.intel.com,OU=Multi-Domain
SSL,OU=Hosted by Intel Corporation,OU=SSG,O=Intel Corporation,street=2200
Mission College Blvd,street=FM1-110,L=Santa
Clara,ST=California,postalCode=95054-1537,C=US",
        "version": 3,
        "serial": "1406C0A906762390B7252C9598F2DB3C"
      }
    },
    "customendpoint": "bro",
    "hostname": "nsm1-ber3.private.ber3.xxxxxxxxxx.com",
    "tags": "bro",
    "category": "bro",
    "source": "x509",
    "utctimestamp": "2019-11-08T07:55:01.992707+00:00",
    "timestamp": "2019-11-08T07:55:01.992707+00:00",
    "receivedtimestamp": "2019-11-08T07:55:17.040196+00:00",
    "eventsource": "nsm",
    "severity": "INFO",
    "mozdefhostname": "mozdef1.private.mdc1.xxxxxxxxxx.com",
    "summary": "X509 certificate seen",
    "plugins": [
      "customDocType",
      "broFixup"
    ],
    "processid": "UNKNOWN",
    "processname": "UNKNOWN"
  },
  "fields": {
    "details.certificate.not_valid_after": [
      "2020-08-02T23:59:59.000Z"
    ],
    "receivedtimestamp": [
      "2019-11-08T07:55:17.040Z"
    ],
    "details.certificate.not_valid_before": [
      "2018-08-03T00:00:00.000Z"
    ],
    "timestamp": [
      "2019-11-08T07:55:01.992Z"
    ],
    "utctimestamp": [
      "2019-11-08T07:55:01.992Z"
    ]
  },
  "highlight": {
    "details.md5": [
      "@kibana-highlighted-field at 27ba7ede8ceab7c896ee4b6bcb509f63
@/kibana-highlighted-field@"
    ],
    "source": [
      "@kibana-highlighted-field at x509@/kibana-highlighted-field@"
    ],
    "category": [
      "@kibana-highlighted-field at bro@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1573199701992
  ]
}


Let's try to splice the first and the second half of the connection

category:bro AND source:dns AND details.query:clr.telemetry.intel.com

clr.ats.infra-host.com, 52.37.21.110, 34.208.200.27

Back to looking at conn.log with destination IP like above and source =
proxies (from there you can again do the crazy dance with ssl.log but
there's no need to)

{
  "_index": "events-20191108",
  "_type": "_doc",
  "_id": "jvDRSW4BC2j90LU9UdgL",
  "_version": 1,
  "_score": null,
  "_source": {
    "type": "nsm",
    "details": {
      "ts": 1573196540.253057,
      "uid": "CrcIhj4lIsSLpPwPe5",
      "version": "TLSv12",
      "cipher": "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
      "curve": "secp256r1",
      "server_name": "clr.telemetry.intel.com",
      "resumed": false,
      "established": true,
      "cert_chain_fuids": [
        "FpfU1u4yMxOBNKQJk2",
        "FYiimM3T6A5AKcTeze",
        "FfAQxKi177b5fl95a",
        "FWHqmB2emMx082lxO1"
      ],
      "client_cert_chain_fuids": [],
      "subject": "CN=c-all.telemetry.intel.com,OU=Multi-Domain
SSL,OU=Hosted by Intel Corporation,OU=SSG,O=Intel Corporation,street=2200
Mission College Blvd,street=FM1-110,L=Santa
Clara,ST=California,postalCode=95054-1537,C=US",
      "issuer": "CN=COMODO RSA Organization Validation Secure Server
CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB",
      "validation_status": "ok",
      "server_cert_md5": "27ba7ede8ceab7c896ee4b6bcb509f63",
      "server_cert_sha1": "9077d0ce724c0f7bf1866d7cf8c4ec1303dc6c21",
      "pfs": true,
      "ja3": "f436b9416f37d134cadd04886327d3e8",
      "ja3s": "303951d4c50efb2e991652225a6f02b1",
      "sourceipaddress": "10.48.74.16",
      "sourceport": 58860,
      "destinationipaddress": "52.37.21.110",
      "destinationport": 443,
      "sourceipv4address": "10.48.74.16",
      "destinationipv4address": "52.37.21.110",
      "destinationipgeolocation": {
        "city": "Boardman",
        "continent": "NA",
        "country_code": "US",
        "country_name": "United States",
        "dma_code": 810,
        "latitude": 45.8491,
        "longitude": -119.7143,
        "metro_code": "Boardman, OR",
        "postal_code": "97818",
        "region_code": "OR",
        "time_zone": "America/Los_Angeles"
      },
      "destinationipgeopoint": "45.8491,-119.7143"
    },
    "customendpoint": "bro",
    "hostname": "nsm2.private.mdc1.xxxxxxxxxx.com",
    "tags": "bro",
    "category": "bro",
    "source": "ssl",
    "utctimestamp": "2019-11-08T07:02:20.253057+00:00",
    "timestamp": "2019-11-08T07:02:20.253057+00:00",
    "receivedtimestamp": "2019-11-08T07:02:21.465325+00:00",
    "eventsource": "nsm",
    "severity": "INFO",
    "mozdefhostname": "mozdef7.private.mdc1.xxxxxxxxxxxxxxxxxxxx.com",
    "summary": "SSL: 10.48.74.16 -> 52.37.21.110:443",
    "plugins": [
      "customDocType",
      "broFixup",
      "ipFixup",
      "geoip"
    ],
    "processid": "UNKNOWN",
    "processname": "UNKNOWN"
  },
  "fields": {
    "receivedtimestamp": [
      "2019-11-08T07:02:21.465Z"
    ],
    "timestamp": [
      "2019-11-08T07:02:20.253Z"
    ],
    "utctimestamp": [
      "2019-11-08T07:02:20.253Z"
    ]
  },
  "highlight": {
    "category": [
      "@kibana-highlighted-field at bro@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1573196540253
  ]
}

I could now paste the third (?!) leg of this connection - after SNAT, but
you get the idea.

Unfortunately, this sort of visibility partially goes away soon, due to
some pseudo-privacy mechanisms AKA a classic shoot in the foot, while
trying to kill a mosquito. From a shotgun. <- my private opinion BTW. I'm
well
after hours ;)

On Thu, Nov 7, 2019 at 10:00 AM Konrad Weglowski <konrad.weglowski at gmail.com>
wrote:

> Thank You Micha?
>
> Client to Proxy would not be encrypted. So based on that I will have all
> the logs, however my destination IP for proxied connections would be proxy
> IP (that was my understanding). I guess ingesting proxy logs would then
> provide full visibility as you mentioned. I am thinking maybe if for some
> reason I cannot get the proxy logs ingested and also tap after the proxy, I
> could possibly correlate both ends of the connections based on URI/SNI/etc
> and same timestamps in my analytics platform. Anyways I will give that a
> try as well. Thanks for your help again.
>
> Konrad
>
>
>
> On Wed, Nov 6, 2019 at 6:59 PM Micha? Purzy?ski <
> michalpurzynski1 at gmail.com> wrote:
>
>> There is no way for Zeek to really correlate if the proxy does everything
>> at the application level, like Squid. The proxy
>>  terminates the connection and then it's free to start the second leg, to
>> the destination host, as it wishes to.
>>
>> Is your traffic from clients to proxy encrypted as well?
>>
>> If the traffic between clients are the proxy is not encrypted (as it's
>> usually the case) then even if the client will establish a TLS session
>> _through_ the proxy you will see a destination with the "service" field
>> http,ssl and all the conn + ssl + http logs.
>>
>> We have that + proxy logs and that's pretty much a full visibility.
>>
>>
>>
>> On Wed, Nov 6, 2019 at 1:11 PM Konrad Weglowski <
>> konrad.weglowski at gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I need to monitor web traffic with Zeek that is going through an
>>> implicit web proxy. I would like to be able to see real client IPs in which
>>> case tap would be placed before the proxy, however I will not be able to
>>> see the real destination IP as it will always be proxy IP. I can also tap
>>> on the other side of the proxy where source will always be proxy IP at the
>>> same time.
>>>
>>> Is there a way with Zeek to correlate the two sessions (before and after
>>> proxy) somehow?
>>>
>>> I realize that UID or Community ID are probably not going do it since
>>> different source/destinations pairs for each for each session, but maybe
>>> based on the timestamp/URI/SNI/byte or packet counts/etc?
>>>
>>> I was also thinking "X-Forwarded-For" header but I guess that would only
>>> work for HTTP and not HTTPS connections.
>>>
>>> Thank You,
>>>
>>> Konrad
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191108/b3c98ee7/attachment-0001.html 

From seth at corelight.com  Mon Nov 11 09:53:01 2019
From: seth at corelight.com (Seth Hall)
Date: Mon, 11 Nov 2019 12:53:01 -0500
Subject: [Zeek] uid in files logs
In-Reply-To: <CAJ6bFK0A50BBxToy0dqqc85=XmyR-BrjSFDP6EWguQ-+PpL=wg@mail.gmail.com>
References: <b474e395ba664b5fb7a6072d42ed6c18@SRVEX03.aizoon.local>
	<CAJ6bFK0A50BBxToy0dqqc85=XmyR-BrjSFDP6EWguQ-+PpL=wg@mail.gmail.com>
Message-ID: <4A2352EC-AECD-4171-80E1-2656D19CC0BB@corelight.com>

On 6 Nov 2019, at 18:45, Micha? Purzy?ski wrote:

> Here we wanted to kill logging X509 certificates into both files.log 
> and x509.log - and by doing that we saved like 20% of our SIEM intake, 
> globally (!!). Should be easy enough to extend x509.log to include 
> data from conn.log, etc.

You could release a package. :P

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com

From henridf at gmail.com  Mon Nov 11 11:37:30 2019
From: henridf at gmail.com (Henri Dubois-Ferriere)
Date: Mon, 11 Nov 2019 20:37:30 +0100
Subject: [Zeek] printing stream columns
In-Reply-To: <CAMzgZ0Kh_owe1Ze25BOYGbn2sstvhq6dcv=UW5gmXt41dW4o6A@mail.gmail.com>
References: <CABcA1ruzNDefM16QHP5yJnOqXL4h=q0=LQwDLatCLaKcHS2aAg@mail.gmail.com>
	<CAMzgZ0KkEk--tNxm7f2dBWuWqCYo8hddfE4DFLgzcOXcV+nAyQ@mail.gmail.com>
	<CABcA1ruH2qu-3R96UVQNbCJxN+FqdycNze13g2kRg7rSpZ_VbQ@mail.gmail.com>
	<CABcA1rsvz9kmeWRBwxW5BKungSDE20bjkL53e9UbAtoGwuURuQ@mail.gmail.com>
	<CAMzgZ0Kh_owe1Ze25BOYGbn2sstvhq6dcv=UW5gmXt41dW4o6A@mail.gmail.com>
Message-ID: <CABcA1rsLcDhtFMLxxaQF1a9+3jBajE_mMLEPGnwBiApyZDBjaw@mail.gmail.com>

Now that this patch is merged (thanks again) I've upgraded my Zeek script
and the record_fields changes work great.

I still have one outstanding issue which is that for a container type,
record_field$type_name is just the container name (such as "vector" or
"set"). I don't see a way to get the type of the container elements from
zeek script, but once again would be delighted to be corrected.

And if there's currently no way, I'm happy to put up a PR, but I could use
some guidance on how to expose this in Zeek (e.g. a new field on
record_field?).

Thanks,
Henri

On Fri, 1 Nov 2019 at 20:53, Jon Siwek <jsiwek at corelight.com> wrote:

> On Fri, Nov 1, 2019 at 4:11 AM Henri Dubois-Ferriere <henridf at gmail.com>
> wrote:
>
> > I'd like to be able to peek into nested records to get the inner fields
> that will show up in the logs. It doesn't seem like there's a way to do
> record introspection given a string representation of the record type name,
> but if I'd be delighted to be told I'm missing something.
>
> No, didn't look like there was a way to do that, but I've made a
> PR/patch that should make recursive introspection possible via
> something like `record_fields("conn_id")` for any arbitrary record
> type name:
>
> https://github.com/zeek/zeek/pull/675
>
> - Jon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191111/f302ab73/attachment.html 

From jsiwek at corelight.com  Mon Nov 11 13:17:05 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Mon, 11 Nov 2019 13:17:05 -0800
Subject: [Zeek] printing stream columns
In-Reply-To: <CABcA1rsLcDhtFMLxxaQF1a9+3jBajE_mMLEPGnwBiApyZDBjaw@mail.gmail.com>
References: <CABcA1ruzNDefM16QHP5yJnOqXL4h=q0=LQwDLatCLaKcHS2aAg@mail.gmail.com>
	<CAMzgZ0KkEk--tNxm7f2dBWuWqCYo8hddfE4DFLgzcOXcV+nAyQ@mail.gmail.com>
	<CABcA1ruH2qu-3R96UVQNbCJxN+FqdycNze13g2kRg7rSpZ_VbQ@mail.gmail.com>
	<CABcA1rsvz9kmeWRBwxW5BKungSDE20bjkL53e9UbAtoGwuURuQ@mail.gmail.com>
	<CAMzgZ0Kh_owe1Ze25BOYGbn2sstvhq6dcv=UW5gmXt41dW4o6A@mail.gmail.com>
	<CABcA1rsLcDhtFMLxxaQF1a9+3jBajE_mMLEPGnwBiApyZDBjaw@mail.gmail.com>
Message-ID: <CAMzgZ0LjfAMg_ZZvnUzgoNswCubOytb=_v2C5qga0WzhDtik8w@mail.gmail.com>

On Mon, Nov 11, 2019 at 11:37 AM Henri Dubois-Ferriere
<henridf at gmail.com> wrote:

> I still have one outstanding issue which is that for a container type,  record_field$type_name is just the container name (such as "vector" or "set"). I don't see a way to get the type of the container elements from zeek script, but once again would be delighted to be corrected.
>
> And if there's currently no way, I'm happy to put up a PR, but I could use some guidance on how to expose this in Zeek (e.g. a new field on record_field?).

Would be great if you want to try making a PR.  The first way to do it
that comes to mind is just alter that "record_field$type_name" to
better describe containers in a format like "vector of XXX",
"set[XXX]" or "table[XXX] of YYY".  This should be the relevant code
to modify:

https://github.com/zeek/zeek/blob/b86a8acc2b84089efbbe51216a4f4d1d57a4f430/src/Type.cc#L845-L850

- Jon


From henridf at gmail.com  Mon Nov 11 13:24:09 2019
From: henridf at gmail.com (Henri Dubois-Ferriere)
Date: Mon, 11 Nov 2019 22:24:09 +0100
Subject: [Zeek] printing stream columns
In-Reply-To: <CAMzgZ0LjfAMg_ZZvnUzgoNswCubOytb=_v2C5qga0WzhDtik8w@mail.gmail.com>
References: <CABcA1ruzNDefM16QHP5yJnOqXL4h=q0=LQwDLatCLaKcHS2aAg@mail.gmail.com>
	<CAMzgZ0KkEk--tNxm7f2dBWuWqCYo8hddfE4DFLgzcOXcV+nAyQ@mail.gmail.com>
	<CABcA1ruH2qu-3R96UVQNbCJxN+FqdycNze13g2kRg7rSpZ_VbQ@mail.gmail.com>
	<CABcA1rsvz9kmeWRBwxW5BKungSDE20bjkL53e9UbAtoGwuURuQ@mail.gmail.com>
	<CAMzgZ0Kh_owe1Ze25BOYGbn2sstvhq6dcv=UW5gmXt41dW4o6A@mail.gmail.com>
	<CABcA1rsLcDhtFMLxxaQF1a9+3jBajE_mMLEPGnwBiApyZDBjaw@mail.gmail.com>
	<CAMzgZ0LjfAMg_ZZvnUzgoNswCubOytb=_v2C5qga0WzhDtik8w@mail.gmail.com>
Message-ID: <CABcA1rscFQadc3typixnV+TdebQWQyNb+bV7+s6ZuxvAJ4v4rQ@mail.gmail.com>

Cool, that's exactly the place i was looking (wasn't sure if changing this
might break existing scripts... but since this is all quite new, probably
best to make the change soon). I'll get the PR up soon.

On Mon, 11 Nov 2019 at 22:17, Jon Siwek <jsiwek at corelight.com> wrote:

> On Mon, Nov 11, 2019 at 11:37 AM Henri Dubois-Ferriere
> <henridf at gmail.com> wrote:
>
> > I still have one outstanding issue which is that for a container type,
> record_field$type_name is just the container name (such as "vector" or
> "set"). I don't see a way to get the type of the container elements from
> zeek script, but once again would be delighted to be corrected.
> >
> > And if there's currently no way, I'm happy to put up a PR, but I could
> use some guidance on how to expose this in Zeek (e.g. a new field on
> record_field?).
>
> Would be great if you want to try making a PR.  The first way to do it
> that comes to mind is just alter that "record_field$type_name" to
> better describe containers in a format like "vector of XXX",
> "set[XXX]" or "table[XXX] of YYY".  This should be the relevant code
> to modify:
>
>
> https://github.com/zeek/zeek/blob/b86a8acc2b84089efbbe51216a4f4d1d57a4f430/src/Type.cc#L845-L850
>
> - Jon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191111/4acc3716/attachment.html 

From konrad.weglowski at gmail.com  Tue Nov 12 07:54:24 2019
From: konrad.weglowski at gmail.com (Konrad Weglowski)
Date: Tue, 12 Nov 2019 10:54:24 -0500
Subject: [Zeek] monitoring proxied web traffic
In-Reply-To: <CAJ6bFK13smRKz-qdJYF08pTR8wTeJAaE01qBfL7r4tkunCeJtg@mail.gmail.com>
References: <CA+fGMJRb6kDVcn-1BFZ=hgwLfvM4MWaKdw=jR_BkrB=ymQz8DQ@mail.gmail.com>
	<CAJ6bFK2DghGOc=o+mcUmgpW7CfxN651TSTuY8tJ+6XPR+OZpNA@mail.gmail.com>
	<CA+fGMJTuGcQuJAUw3tqvr786FU9SRSaRCDo7aoQUWZipHu-+HA@mail.gmail.com>
	<CAJ6bFK13smRKz-qdJYF08pTR8wTeJAaE01qBfL7r4tkunCeJtg@mail.gmail.com>
Message-ID: <CA+fGMJSWrgWGiy71bi5tik_reYpuHGU=e5WPFu29p97qHweCcA@mail.gmail.com>

Thanks again Micha? for the breakdown. I have the full picture now :)

Take care

Konrad




On Fri, Nov 8, 2019 at 3:21 AM Micha? Purzy?ski <michalpurzynski1 at gmail.com>
wrote:

> You're welcome!!
>
> If the traffic from the client to the proxy is not encrypted and the proxy
> is not "transparent" (meaning you configured it on the client) then you
> have the same case as I do here.
>
> So there's this host
> source IP 10.48.75.81
>
> trying to talk to clr.telemetry.intel.com
>
> via our proxy at 10.48.122.10 / 10.48.74.16 / 10.48.74.17
>
> *(for anyone working at a place where disclosing an internal IP or an FQDN
> equal incident - you know a lot of great places are hiring, right?)*
>
> I used the "uid" to correlate this connection across three types of logs -
> conn.log, http.log and ssl.log. You're gonna get them due to the way
> proxies operate.
>
> Please ignore tons of metadata from our SIEM (MozDef = custom Python +
> pure ES), I'm going to C&P three logs as they are without any redactions.
> This post is HUGE!!
>
> *conn.log* - host -> proxy
>
> {
>   "_index": "events-20191108",
>   "_type": "_doc",
>   "_id": "JmLRSW4BiWsbe3JQtC8S",
>   "_version": 1,
>   "_score": null,
>   "_source": {
>     "type": "nsm",
>     "details": {
>       "ts": 1573196540.178897,
> *      "uid": "C9PFt711Odfwpbr0Ti",*
> *      "proto": "tcp",*
> *      "service": "http,ssl",*
>       "duration": 0.183212,
>       "orig_bytes": 1604,
>       "resp_bytes": 7180,
>       "conn_state": "SF",
>       "local_orig": true,
>       "local_resp": true,
>       "missed_bytes": 0,
> *      "history": "ShADadFf",*
> *      "orig_pkts": 14,*
> *      "resp_pkts": 13,*
>       "peer": "nsm2-af_packet-enp18s0f0-8",
> *      "orig_l2_addr": "b4:0c:25:e0:40:10",*
> *      "resp_l2_addr": "00:50:56:a1:48:64",*
> *      "sourceipaddress": "10.48.75.81",*
> *      "sourceport": 55330,*
> *      "destinationipaddress": "10.48.122.10",*
> *      "destinationport": 3128,*
> *      "originipbytes": 2340,*
> *      "responseipbytes": 7864,*
>       "sourceipv4address": "10.48.75.81",
>       "destinationipv4address": "10.48.122.10"
>     },
>     "customendpoint": "bro",
>     "hostname": "nsm2.private.mdc1.xxxxxxxxxx.com",
>     "tags": "bro",
>     "category": "bro",
> *    "source": "conn",*
> *    "utctimestamp": "2019-11-08T07:02:20.178897+00:00",*
>     "timestamp": "2019-11-08T07:02:20.178897+00:00",
>     "receivedtimestamp": "2019-11-08T07:02:27.686061+00:00",
>     "eventsource": "nsm",
>     "severity": "INFO",
>     "mozdefhostname": "mozdef5.private.mdc1.xxxxxxxxxx.com",
>     "summary": "10.48.75.81:55330 -> 10.48.122.10:3128 ShADadFf 2340
> bytes / 7864 bytes",
>     "plugins": [
>       "customDocType",
>       "broFixup",
>       "ipFixup",
>       "geoip"
>     ],
>     "processid": "UNKNOWN",
>     "processname": "UNKNOWN"
>   },
>   "fields": {
>     "receivedtimestamp": [
>       "2019-11-08T07:02:27.686Z"
>     ],
>     "timestamp": [
>       "2019-11-08T07:02:20.178Z"
>     ],
>     "utctimestamp": [
>       "2019-11-08T07:02:20.178Z"
>     ]
>   },
>   "highlight": {
>     "summary": [
>       "@kibana-highlighted-field at 10.48.75.81@/kibana-highlighted-field@:55330
> -> 10.48.122.10:3128 ShADadFf 2340 bytes / 7864 bytes"
>     ],
>     "details.sourceipv4address": [
>       "@kibana-highlighted-field at 10.48.75.81@/kibana-highlighted-field@"
>     ],
>     "details.uid": [
>       "@kibana-highlighted-field at C9PFt711Odfwpbr0Ti
> @/kibana-highlighted-field@"
>     ],
>     "category": [
>       "@kibana-highlighted-field at bro@/kibana-highlighted-field@"
>     ]
>   },
>   "sort": [
>     1573196540178
>   ]
> }
>
> *http.log* (because the connection from the client to the proxy was
> technically a http connection)
>
> {
>   "_index": "events-20191108",
>   "_type": "_doc",
>   "_id": "jPDRSW4BC2j90LU9UdgL",
>   "_version": 1,
>   "_score": null,
>   "_source": {
>     "type": "nsm",
>     "details": {
>       "ts": 1573196540.202888,
> *      "uid": "C9PFt711Odfwpbr0Ti",*
>       "trans_depth": 1,
>       "method": "CONNECT",
>
> *"host": "clr.telemetry.intel.com <http://clr.telemetry.intel.com>",
> "uri": "clr.telemetry.intel.com:443 <http://clr.telemetry.intel.com:443>",*
>       "version": "1.1",
>       "request_body_len": 0,
>       "response_body_len": 0,
> *      "status_code": 200,*
> *      "status_msg": "Connection established",*
>       "tags": [],
>       "proxied": [
>         "PROXY-CONNECTION -> Keep-Alive"
>       ],
>
>
>
> *"sourceipaddress": "10.48.75.81",      "sourceport": 55330,
> "destinationipaddress": "10.48.122.10",      "destinationport": 3128,*
>       "sourceipv4address": "10.48.75.81",
>       "destinationipv4address": "10.48.122.10"
>     },
>     "customendpoint": "bro",
>     "hostname": "nsm2.private.mdc1.xxxxxxxxxx.com",
>     "tags": "bro",
>     "category": "bro",
> *    "source": "http",*
> *    "utctimestamp": "2019-11-08T07:02:20.202888+00:00",*
>     "timestamp": "2019-11-08T07:02:20.202888+00:00",
>     "receivedtimestamp": "2019-11-08T07:02:21.427537+00:00",
>     "eventsource": "nsm",
>     "severity": "INFO",
>     "mozdefhostname": "mozdef7.private.mdc1.xxxxxxxxxx.com",
>     "summary": "HTTP CONNECT 10.48.75.81 -> 10.48.122.10:3128",
>     "plugins": [
>       "customDocType",
>       "broFixup",
>       "ipFixup",
>       "geoip"
>     ],
>     "processid": "UNKNOWN",
>     "processname": "UNKNOWN"
>   },
>   "fields": {
>     "receivedtimestamp": [
>       "2019-11-08T07:02:21.427Z"
>     ],
>     "timestamp": [
>       "2019-11-08T07:02:20.202Z"
>     ],
>     "utctimestamp": [
>       "2019-11-08T07:02:20.202Z"
>     ]
>   },
>   "highlight": {
>     "summary": [
>       "HTTP CONNECT @kibana-highlighted-field at 10.48.75.81
> @/kibana-highlighted-field@ -> 10.48.122.10:3128"
>     ],
>     "details.sourceipv4address": [
>       "@kibana-highlighted-field at 10.48.75.81@/kibana-highlighted-field@"
>     ],
>     "details.uid": [
>       "@kibana-highlighted-field at C9PFt711Odfwpbr0Ti
> @/kibana-highlighted-field@"
>     ],
>     "category": [
>       "@kibana-highlighted-field at bro@/kibana-highlighted-field@"
>     ]
>   },
>   "sort": [
>     1573196540202
>   ]
> }
>
> And the *ssl.log* - over the previously established http connection the
> client told proxy to establish a tunnel to the destination host - and Zeek
> parsed that as TLS (find me another IDS/NSM that can do that)
>
> {
>   "_index": "events-20191108",
>   "_type": "_doc",
>   "_id": "sfDRSW4BC2j90LU9S8_Q",
>   "_version": 1,
>   "_score": null,
>   "_source": {
>     "type": "nsm",
>     "details": {
>       "ts": 1573196540.253057,
>
>
>
>
>
>
>
>
>
>
>
>
>
> * "uid": "C9PFt711Odfwpbr0Ti",      "version": "TLSv12",      "cipher":
> "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",      "curve": "secp256r1",
> "server_name": "clr.telemetry.intel.com <http://clr.telemetry.intel.com>",
>     "resumed": false,      "established": true,      "cert_chain_fuids": [
>       "FknnGH2L2UIKGERW3b",        "FGtbhK1HoQMWdgyKW",
> "F8JbGM30JvqFVi41A6",        "Fsc2fRRTyTDQrT1Gc"      ],
> "client_cert_chain_fuids": [],*
>
>
>
>
>
>
>
>
>
>
>
>
> *"subject": "CN=c-all.telemetry.intel.com
> <http://c-all.telemetry.intel.com>,OU=Multi-Domain SSL,OU=Hosted by Intel
> Corporation,OU=SSG,O=Intel Corporation,street=2200 Mission College
> Blvd,street=FM1-110,L=Santa
> Clara,ST=California,postalCode=95054-1537,C=US",      "issuer": "CN=COMODO
> RSA Organization Validation Secure Server CA,O=COMODO CA
> Limited,L=Salford,ST=Greater Manchester,C=GB",      "validation_status":
> "ok",      "server_cert_md5": "27ba7ede8ceab7c896ee4b6bcb509f63",
> "server_cert_sha1": "9077d0ce724c0f7bf1866d7cf8c4ec1303dc6c21",      "pfs":
> true,      "ja3": "f436b9416f37d134cadd04886327d3e8",      "ja3s":
> "303951d4c50efb2e991652225a6f02b1",      "sourceipaddress": "10.48.75.81",
>     "sourceport": 55330,      "destinationipaddress": "10.48.122.10",
> "destinationport": 3128,*      "sourceipv4address": "10.48.75.81",
>       "destinationipv4address": "10.48.122.10"
>     },
>     "customendpoint": "bro",
>     "hostname": "nsm2.private.mdc1.xxxxxxxxxx.com",
>     "tags": "bro",
>     "category": "bro",
>     "source": "ssl",
>     "utctimestamp": "2019-11-08T07:02:20.253057+00:00",
>     "timestamp": "2019-11-08T07:02:20.253057+00:00",
>     "receivedtimestamp": "2019-11-08T07:02:21.464921+00:00",
>     "eventsource": "nsm",
>     "severity": "INFO",
>     "mozdefhostname": "mozdef6.private.mdc1.xxxxxxxxxx.com",
>     "summary": "SSL: 10.48.75.81 -> 10.48.122.10:3128",
>     "plugins": [
>       "customDocType",
>       "broFixup",
>       "ipFixup",
>       "geoip"
>     ],
>     "processid": "UNKNOWN",
>     "processname": "UNKNOWN"
>   },
>   "fields": {
>     "receivedtimestamp": [
>       "2019-11-08T07:02:21.464Z"
>     ],
>     "timestamp": [
>       "2019-11-08T07:02:20.253Z"
>     ],
>     "utctimestamp": [
>       "2019-11-08T07:02:20.253Z"
>     ]
>   },
>   "highlight": {
>     "summary": [
>       "SSL: @kibana-highlighted-field at 10.48.75.81
> @/kibana-highlighted-field@ -> 10.48.122.10:3128"
>     ],
>     "details.sourceipv4address": [
>       "@kibana-highlighted-field at 10.48.75.81@/kibana-highlighted-field@"
>     ],
>     "details.uid": [
>       "@kibana-highlighted-field at C9PFt711Odfwpbr0Ti
> @/kibana-highlighted-field@"
>     ],
>     "category": [
>       "@kibana-highlighted-field at bro@/kibana-highlighted-field@"
>     ]
>   },
>   "sort": [
>     1573196540253
>   ]
> }
>
>
> Not enough? Let's look for the certificate's MD5
>
> {
>   "_index": "events-20191108",
>   "_type": "_doc",
>   "_id": "_CwBSm4BC2j90LU9vfNt",
>   "_version": 1,
>   "_score": null,
>   "_source": {
>     "type": "nsm",
>     "details": {
>       "ts": 1573199701.992707,
>       "id": "FdE5qt1b93z9QOJnz1",
>       "san.dns": [
>         "c-all.telemetry.intel.com",
>         "c.telemetry.intel.com",
>         "c1.telemetry.intel.com",
>         "c2.telemetry.intel.com",
>         "c3.telemetry.intel.com",
>         "c4.telemetry.intel.com",
>         "c5.telemetry.intel.com",
>         "c6.telemetry.intel.com",
>         "c7.telemetry.intel.com",
>         "c8.telemetry.intel.com",
>         "c9.telemetry.intel.com",
>         "clr.telemetry.intel.com",
>         "sur.telemetry.intel.com"
>       ],
>       "md5": "27ba7ede8ceab7c896ee4b6bcb509f63",
>       "sha1": "9077d0ce724c0f7bf1866d7cf8c4ec1303dc6c21",
>       "certificate": {
>         "basic_constraints_ca": false,
>         "exponent": "65537",
>         "issuer": "CN=COMODO RSA Organization Validation Secure Server
> CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB",
>         "key_alg": "rsaEncryption",
>         "key_length": 2048,
>         "key_type": "rsa",
>         "not_valid_after": "2020-08-02T23:59:59+00:00",
>         "not_valid_before": "2018-08-03T00:00:00+00:00",
>         "sig_alg": "sha256WithRSAEncryption",
>         "subject": "CN=c-all.telemetry.intel.com,OU=Multi-Domain
> SSL,OU=Hosted by Intel Corporation,OU=SSG,O=Intel Corporation,street=2200
> Mission College Blvd,street=FM1-110,L=Santa
> Clara,ST=California,postalCode=95054-1537,C=US",
>         "version": 3,
>         "serial": "1406C0A906762390B7252C9598F2DB3C"
>       }
>     },
>     "customendpoint": "bro",
>     "hostname": "nsm1-ber3.private.ber3.xxxxxxxxxx.com",
>     "tags": "bro",
>     "category": "bro",
>     "source": "x509",
>     "utctimestamp": "2019-11-08T07:55:01.992707+00:00",
>     "timestamp": "2019-11-08T07:55:01.992707+00:00",
>     "receivedtimestamp": "2019-11-08T07:55:17.040196+00:00",
>     "eventsource": "nsm",
>     "severity": "INFO",
>     "mozdefhostname": "mozdef1.private.mdc1.xxxxxxxxxx.com",
>     "summary": "X509 certificate seen",
>     "plugins": [
>       "customDocType",
>       "broFixup"
>     ],
>     "processid": "UNKNOWN",
>     "processname": "UNKNOWN"
>   },
>   "fields": {
>     "details.certificate.not_valid_after": [
>       "2020-08-02T23:59:59.000Z"
>     ],
>     "receivedtimestamp": [
>       "2019-11-08T07:55:17.040Z"
>     ],
>     "details.certificate.not_valid_before": [
>       "2018-08-03T00:00:00.000Z"
>     ],
>     "timestamp": [
>       "2019-11-08T07:55:01.992Z"
>     ],
>     "utctimestamp": [
>       "2019-11-08T07:55:01.992Z"
>     ]
>   },
>   "highlight": {
>     "details.md5": [
>       "@kibana-highlighted-field at 27ba7ede8ceab7c896ee4b6bcb509f63
> @/kibana-highlighted-field@"
>     ],
>     "source": [
>       "@kibana-highlighted-field at x509@/kibana-highlighted-field@"
>     ],
>     "category": [
>       "@kibana-highlighted-field at bro@/kibana-highlighted-field@"
>     ]
>   },
>   "sort": [
>     1573199701992
>   ]
> }
>
>
> Let's try to splice the first and the second half of the connection
>
> category:bro AND source:dns AND details.query:clr.telemetry.intel.com
>
> clr.ats.infra-host.com, 52.37.21.110, 34.208.200.27
>
> Back to looking at conn.log with destination IP like above and source =
> proxies (from there you can again do the crazy dance with ssl.log but
> there's no need to)
>
> {
>   "_index": "events-20191108",
>   "_type": "_doc",
>   "_id": "jvDRSW4BC2j90LU9UdgL",
>   "_version": 1,
>   "_score": null,
>   "_source": {
>     "type": "nsm",
>     "details": {
>       "ts": 1573196540.253057,
>       "uid": "CrcIhj4lIsSLpPwPe5",
>       "version": "TLSv12",
>       "cipher": "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
>       "curve": "secp256r1",
>       "server_name": "clr.telemetry.intel.com",
>       "resumed": false,
>       "established": true,
>       "cert_chain_fuids": [
>         "FpfU1u4yMxOBNKQJk2",
>         "FYiimM3T6A5AKcTeze",
>         "FfAQxKi177b5fl95a",
>         "FWHqmB2emMx082lxO1"
>       ],
>       "client_cert_chain_fuids": [],
>       "subject": "CN=c-all.telemetry.intel.com,OU=Multi-Domain
> SSL,OU=Hosted by Intel Corporation,OU=SSG,O=Intel Corporation,street=2200
> Mission College Blvd,street=FM1-110,L=Santa
> Clara,ST=California,postalCode=95054-1537,C=US",
>       "issuer": "CN=COMODO RSA Organization Validation Secure Server
> CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB",
>       "validation_status": "ok",
>       "server_cert_md5": "27ba7ede8ceab7c896ee4b6bcb509f63",
>       "server_cert_sha1": "9077d0ce724c0f7bf1866d7cf8c4ec1303dc6c21",
>       "pfs": true,
>       "ja3": "f436b9416f37d134cadd04886327d3e8",
>       "ja3s": "303951d4c50efb2e991652225a6f02b1",
>       "sourceipaddress": "10.48.74.16",
>       "sourceport": 58860,
>       "destinationipaddress": "52.37.21.110",
>       "destinationport": 443,
>       "sourceipv4address": "10.48.74.16",
>       "destinationipv4address": "52.37.21.110",
>       "destinationipgeolocation": {
>         "city": "Boardman",
>         "continent": "NA",
>         "country_code": "US",
>         "country_name": "United States",
>         "dma_code": 810,
>         "latitude": 45.8491,
>         "longitude": -119.7143,
>         "metro_code": "Boardman, OR",
>         "postal_code": "97818",
>         "region_code": "OR",
>         "time_zone": "America/Los_Angeles"
>       },
>       "destinationipgeopoint": "45.8491,-119.7143"
>     },
>     "customendpoint": "bro",
>     "hostname": "nsm2.private.mdc1.xxxxxxxxxx.com",
>     "tags": "bro",
>     "category": "bro",
>     "source": "ssl",
>     "utctimestamp": "2019-11-08T07:02:20.253057+00:00",
>     "timestamp": "2019-11-08T07:02:20.253057+00:00",
>     "receivedtimestamp": "2019-11-08T07:02:21.465325+00:00",
>     "eventsource": "nsm",
>     "severity": "INFO",
>     "mozdefhostname": "mozdef7.private.mdc1.xxxxxxxxxxxxxxxxxxxx.com",
>     "summary": "SSL: 10.48.74.16 -> 52.37.21.110:443",
>     "plugins": [
>       "customDocType",
>       "broFixup",
>       "ipFixup",
>       "geoip"
>     ],
>     "processid": "UNKNOWN",
>     "processname": "UNKNOWN"
>   },
>   "fields": {
>     "receivedtimestamp": [
>       "2019-11-08T07:02:21.465Z"
>     ],
>     "timestamp": [
>       "2019-11-08T07:02:20.253Z"
>     ],
>     "utctimestamp": [
>       "2019-11-08T07:02:20.253Z"
>     ]
>   },
>   "highlight": {
>     "category": [
>       "@kibana-highlighted-field at bro@/kibana-highlighted-field@"
>     ]
>   },
>   "sort": [
>     1573196540253
>   ]
> }
>
> I could now paste the third (?!) leg of this connection - after SNAT, but
> you get the idea.
>
> Unfortunately, this sort of visibility partially goes away soon, due to
> some pseudo-privacy mechanisms AKA a classic shoot in the foot, while
> trying to kill a mosquito. From a shotgun. <- my private opinion BTW. I'm
> well
> after hours ;)
>
> On Thu, Nov 7, 2019 at 10:00 AM Konrad Weglowski <
> konrad.weglowski at gmail.com> wrote:
>
>> Thank You Micha?
>>
>> Client to Proxy would not be encrypted. So based on that I will have all
>> the logs, however my destination IP for proxied connections would be proxy
>> IP (that was my understanding). I guess ingesting proxy logs would then
>> provide full visibility as you mentioned. I am thinking maybe if for some
>> reason I cannot get the proxy logs ingested and also tap after the proxy, I
>> could possibly correlate both ends of the connections based on URI/SNI/etc
>> and same timestamps in my analytics platform. Anyways I will give that a
>> try as well. Thanks for your help again.
>>
>> Konrad
>>
>>
>>
>> On Wed, Nov 6, 2019 at 6:59 PM Micha? Purzy?ski <
>> michalpurzynski1 at gmail.com> wrote:
>>
>>> There is no way for Zeek to really correlate if the proxy does
>>> everything at the application level, like Squid. The proxy
>>>  terminates the connection and then it's free to start the second leg,
>>> to the destination host, as it wishes to.
>>>
>>> Is your traffic from clients to proxy encrypted as well?
>>>
>>> If the traffic between clients are the proxy is not encrypted (as it's
>>> usually the case) then even if the client will establish a TLS session
>>> _through_ the proxy you will see a destination with the "service" field
>>> http,ssl and all the conn + ssl + http logs.
>>>
>>> We have that + proxy logs and that's pretty much a full visibility.
>>>
>>>
>>>
>>> On Wed, Nov 6, 2019 at 1:11 PM Konrad Weglowski <
>>> konrad.weglowski at gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I need to monitor web traffic with Zeek that is going through an
>>>> implicit web proxy. I would like to be able to see real client IPs in which
>>>> case tap would be placed before the proxy, however I will not be able to
>>>> see the real destination IP as it will always be proxy IP. I can also tap
>>>> on the other side of the proxy where source will always be proxy IP at the
>>>> same time.
>>>>
>>>> Is there a way with Zeek to correlate the two sessions (before and
>>>> after proxy) somehow?
>>>>
>>>> I realize that UID or Community ID are probably not going do it since
>>>> different source/destinations pairs for each for each session, but maybe
>>>> based on the timestamp/URI/SNI/byte or packet counts/etc?
>>>>
>>>> I was also thinking "X-Forwarded-For" header but I guess that would
>>>> only work for HTTP and not HTTPS connections.
>>>>
>>>> Thank You,
>>>>
>>>> Konrad
>>>> _______________________________________________
>>>> Zeek mailing list
>>>> zeek at zeek.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191112/63377ea2/attachment-0001.html 

From fadelkam at gmail.com  Tue Nov 12 13:05:22 2019
From: fadelkam at gmail.com (Fadel)
Date: Tue, 12 Nov 2019 16:05:22 -0500
Subject: [Zeek] Cannot determine Bro source directory, use --bro-dist=DIR.
Message-ID: <CABFMW7VNSQd9LP1g4Cbc-_cbzWbiBa+ubrww1Gn+mEFkMjhJjw@mail.gmail.com>

Hi, I've been trying to install this plugin and it seems to have
some issues or confusion between bro and zeek for that plugin.
Anyone encountered this issue?
Thanks!


~]# zkg install apache/metron-bro-plugin-kafka
The following packages will be INSTALLED:
 zeek/apache/metron-bro-plugin-kafka (0.3)

Verify the following REQUIRED external dependencies:
(Ensure their installation on all relevant systems before proceeding):
 from zeek/apache/metron-bro-plugin-kafka (0.3):
   librdkafka ~0.11.5

Proceed? [Y/n]
zeek/apache/metron-bro-plugin-kafka asks for LIBRDKAFKA_ROOT (Path to
librdkafka installation tree) ? [/usr/local/lib]
Saved answers to config file: /root/.zkg/config
Running unit tests for "zeek/apache/metron-bro-plugin-kafka"
error: failed to run tests for zeek/apache/metron-bro-plugin-kafka: package
build_command failed, see log in
/root/.zkg/logs/metron-bro-plugin-kafka-build.log
Proceed to install anyway? [N/y]

~]# cat /root/.zkg/logs/metron-bro-plugin-kafka-build.log
=== STDERR ===
=== STDOUT ===
Cannot determine Bro source directory, use --bro-dist=DIR.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191112/febebd8f/attachment.html 

From akgraner at corelight.com  Tue Nov 12 18:15:15 2019
From: akgraner at corelight.com (Amber Graner)
Date: Tue, 12 Nov 2019 21:15:15 -0500
Subject: [Zeek] ZeekWeek2019 Summary and Slides
Message-ID: <CAJhOzupRLiw6eaYv8JVPzaBH01PDjfC7gU6gBiixMJnPrsw-QQ@mail.gmail.com>

Hi all,

Here's the link to the ZeekWeek 2019 Summary and Slides.

We'll add the videos as soon as they are ready.

https://blog.zeek.org/2019/11/zeekweek-2019-summary-and-slides.html

Thanks,
~Amber

-- 

*Amber Graner*
Director of Community
Corelight, Inc

828.582.9469



 * Ask me about how you can participate in the Zeek (formerly Bro)
community.
 * Remember - ZEEK AND YOU SHALL FIND!!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191112/b755f7f6/attachment.html 

From mus3 at lehigh.edu  Wed Nov 13 10:55:15 2019
From: mus3 at lehigh.edu (Munroe Sollog)
Date: Wed, 13 Nov 2019 13:55:15 -0500
Subject: [Zeek] intel framework, disabling certain feeds to certain workers
Message-ID: <CAN3u96-yFyh9PVTJ1nUAQceDvoNXX84ATJRg62wAKSkT5bKeZw@mail.gmail.com>

Is there a way to select which intel "files" are sent to particular
workers?  Perhaps using the aux-script parameter in nodes.cfg?

We are running Bro 2.6.4.  We are using two remote workers to collect
specific data on much smaller machines.  On our main cluster we are trying
to load a +2GB file using the intel framework.  The main cluster seems to
handle this file with no problem.  However, it is causing the "specialized"
remote workers to crash "out of memory".

I realize I'm not necessarily using the cluster feature as intended, but it
has, thus far, been extremely convenient.  I would like to not have to
create a separate cluster.

Any advice?

-- 
Munroe Sollog
Senior Network Engineer
munroe at lehigh.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191113/9e3b6de2/attachment.html 

From jan.grashoefer at gmail.com  Wed Nov 13 11:06:37 2019
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Wed, 13 Nov 2019 20:06:37 +0100
Subject: [Zeek] intel framework,
 disabling certain feeds to certain workers
In-Reply-To: <CAN3u96-yFyh9PVTJ1nUAQceDvoNXX84ATJRg62wAKSkT5bKeZw@mail.gmail.com>
References: <CAN3u96-yFyh9PVTJ1nUAQceDvoNXX84ATJRg62wAKSkT5bKeZw@mail.gmail.com>
Message-ID: <f2c5a866-df8a-b64a-6bd2-2b65d3d19b48@gmail.com>

Unfortunately, the short answer is no.

Jan

On 13/11/2019 19:55, Munroe Sollog wrote:
> Is there a way to select which intel "files" are sent to particular 
> workers?? Perhaps using the aux-script parameter in nodes.cfg?
> 
> We are running Bro 2.6.4.? We are using two remote workers to collect 
> specific data on much smaller machines.? On our main cluster we are 
> trying to load a +2GB file using the intel framework.? The main cluster 
> seems to handle this file with no problem.? However, it is causing the 
> "specialized" remote workers to crash "out of memory".
> 
> I realize I'm not necessarily using the cluster feature as intended, but 
> it has, thus far, been extremely convenient.? I would like to not have 
> to create a separate cluster.
> 
> Any advice?
> 
> -- 
> Munroe Sollog
> Senior Network Engineer
> munroe at lehigh.edu <mailto:munroe at lehigh.edu>
> 
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> 

From akgraner at corelight.com  Wed Nov 13 11:25:32 2019
From: akgraner at corelight.com (Amber Graner)
Date: Wed, 13 Nov 2019 14:25:32 -0500
Subject: [Zeek] =?utf-8?b?TmV3IEJsb2cgUG9zdDogV2hhdCBpcyDigJhXZWlyZOKAmSBp?=
	=?utf-8?q?n_Zeek=3F?=
Message-ID: <CAJhOzup40rHEzxZn93_-XK+Ggs71rUa3RV1qQfts4F-E6v5FbA@mail.gmail.com>

Here's the first of several posts on Weird by Fatema Bannat Wala, Security
Engineer, University of Delaware. -
https://blog.zeek.org/2019/11/what-is-weird-in-zeek.html

If you or someone you know would like to contribute content for the blog,
please let me know.

Thanks,
~Amber
-- 

*Amber Graner*
Director of Community
Corelight, Inc

828.582.9469



 * Ask me about how you can participate in the Zeek (formerly Bro)
community.
 * Remember - ZEEK AND YOU SHALL FIND!!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191113/6db1cfa4/attachment.html 

From michalpurzynski1 at gmail.com  Wed Nov 13 11:55:02 2019
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Wed, 13 Nov 2019 11:55:02 -0800
Subject: [Zeek] intel framework,
	disabling certain feeds to certain workers
In-Reply-To: <CAN3u96-yFyh9PVTJ1nUAQceDvoNXX84ATJRg62wAKSkT5bKeZw@mail.gmail.com>
References: <CAN3u96-yFyh9PVTJ1nUAQceDvoNXX84ATJRg62wAKSkT5bKeZw@mail.gmail.com>
Message-ID: <62CB949A-C86F-49DA-A9B8-CD5690A68B2C@gmail.com>

Maybe if you describe your use case we can find something out. I don?t think you have 2GB or actionable intel ;)

There?s also the input framework and the config framework, that can load data. It?s way easier to use the config framework on the 3.x line, as you don?t have to do the cluster magic yourself.

They just give you data so the matching would be on you and your scripts.

>> On Nov 13, 2019, at 10:57 AM, Munroe Sollog <mus3 at lehigh.edu> wrote:
> ?
> Is there a way to select which intel "files" are sent to particular workers?  Perhaps using the aux-script parameter in nodes.cfg?
> 
> We are running Bro 2.6.4.  We are using two remote workers to collect specific data on much smaller machines.  On our main cluster we are trying to load a +2GB file using the intel framework.  The main cluster seems to handle this file with no problem.  However, it is causing the "specialized" remote workers to crash "out of memory".  
> 
> I realize I'm not necessarily using the cluster feature as intended, but it has, thus far, been extremely convenient.  I would like to not have to create a separate cluster.  
> 
> Any advice?
> 
> -- 
> Munroe Sollog
> Senior Network Engineer
> munroe at lehigh.edu
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191113/39a5067b/attachment-0001.html 

From mus3 at lehigh.edu  Thu Nov 14 07:00:01 2019
From: mus3 at lehigh.edu (Munroe Sollog)
Date: Thu, 14 Nov 2019 10:00:01 -0500
Subject: [Zeek] intel framework,
	disabling certain feeds to certain workers
In-Reply-To: <62CB949A-C86F-49DA-A9B8-CD5690A68B2C@gmail.com>
References: <CAN3u96-yFyh9PVTJ1nUAQceDvoNXX84ATJRg62wAKSkT5bKeZw@mail.gmail.com>
	<62CB949A-C86F-49DA-A9B8-CD5690A68B2C@gmail.com>
Message-ID: <CAN3u968_hQ5erfH84zaN2MnG7VHmgymPRLGRQpk2Ugv3Gb09TQ@mail.gmail.com>

We are extending the concept of a bro/zeek cluster.  In addition to our
traditional cluster that analyzes large bandwidth taps, we started
evaluating using additional 'workers' as sensors on servers to
collect targeted data.  For example, on a web proxy we collect web traffic,
on DNS server we collect DNS queries, etc...  We utilize the 'aux_scripts'
feature in nodes.cfg of broctl to define capture filters appropriate for
each service, which reduces load required to run those "sensors".  This
concept has allowed us centrally manage all workers and aggregate data from
many sources to one main pipeline.

In addition, we are ingesting many "security feeds" from many sources.
Currently the cumulative size of all intel data files exceeds 3GB.  The
"traditional" cluster has no problem loading that intel.  However, these
small "sensors" do. A capture filter of, for example:

redef capture_filters += {

    ["dns"] = "port 53"

};

will never match any intel with types: Intel::FILE_NAME, Intel::FILE_HASH,
INTEL::URL.  Allowing a bit more fine-grained control of how workers
operate would allow us to maintain the centralized collection and control
and scale our concept out to other applications without exploding resource
requirements.

Hope this clarifies our use case.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191114/81b5d632/attachment.html 

From austin522 at gmail.com  Fri Nov 15 04:46:04 2019
From: austin522 at gmail.com (venkatesh bandari)
Date: Fri, 15 Nov 2019 20:46:04 +0800
Subject: [Zeek] Zeek Compling
Message-ID: <CAKPzng_KCH-KaPed1JZ33FD8krnJfZE7OQ5axV=o+zS27eFyZg@mail.gmail.com>

Hello Team,

could someone help me on the issue iam facing

yum install cmake make gcc gcc-c++ flex bison libpcap-devel openssl-devel
python-devel swig zlib-devel

the above command on centos is not installing latest packages of
cmake,gcc,gcc-c++ which is required to compile zeek on centos

Did anyone recently complied zeek on centos 7.any help is much appreciated
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191115/bab006d0/attachment.html 

From dominik.charousset at corelight.com  Fri Nov 15 07:03:58 2019
From: dominik.charousset at corelight.com (Dominik Charousset)
Date: Fri, 15 Nov 2019 16:03:58 +0100
Subject: [Zeek] Zeek Compling
In-Reply-To: <CAKPzng_KCH-KaPed1JZ33FD8krnJfZE7OQ5axV=o+zS27eFyZg@mail.gmail.com>
References: <CAKPzng_KCH-KaPed1JZ33FD8krnJfZE7OQ5axV=o+zS27eFyZg@mail.gmail.com>
Message-ID: <7A023640-E464-4B9E-A82C-3001F3D44D5E@corelight.com>

Hi,

you?ll need to install cmake3 and devtoolset-7.

The most recent documentation has a small section for the steps necessary on CentOS: https://docs.zeek.org/en/latest/install/install.html#required-dependencies.

Hope that helps.

    Dominik


> On 15. Nov 2019, at 13:46, venkatesh bandari <austin522 at gmail.com> wrote:
> 
> Hello Team,
> 
> could someone help me on the issue iam facing
> 
> yum install cmake make gcc gcc-c++ flex bison libpcap-devel openssl-devel python-devel swig zlib-devel
> 
> the above command on centos is not installing latest packages of cmake,gcc,gcc-c++ which is required to compile zeek on centos
> 
> Did anyone recently complied zeek on centos 7.any help is much appreciated
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



From austin522 at gmail.com  Fri Nov 15 07:07:48 2019
From: austin522 at gmail.com (venkatesh bandari)
Date: Fri, 15 Nov 2019 23:07:48 +0800
Subject: [Zeek] Zeek Compling
In-Reply-To: <7A023640-E464-4B9E-A82C-3001F3D44D5E@corelight.com>
References: <CAKPzng_KCH-KaPed1JZ33FD8krnJfZE7OQ5axV=o+zS27eFyZg@mail.gmail.com>
	<7A023640-E464-4B9E-A82C-3001F3D44D5E@corelight.com>
Message-ID: <CAKPzng-7B7ZJL__CnmV6LYCuGhs49fwJRfXGq_XZ-kFFLDhvqA@mail.gmail.com>

Thanks Dominik.. I installed cmake3 but that has a dependency of gcc7.2.0
above which I installed but when I compile Zeek it is complaining gcc
should be 7.0 above

On Fri, 15 Nov 2019 at 11:04 PM, Dominik Charousset <
dominik.charousset at corelight.com> wrote:

> Hi,
>
> you?ll need to install cmake3 and devtoolset-7.
>
> The most recent documentation has a small section for the steps necessary
> on CentOS:
> https://docs.zeek.org/en/latest/install/install.html#required-dependencies
> .
>
> Hope that helps.
>
>     Dominik
>
>
> > On 15. Nov 2019, at 13:46, venkatesh bandari <austin522 at gmail.com>
> wrote:
> >
> > Hello Team,
> >
> > could someone help me on the issue iam facing
> >
> > yum install cmake make gcc gcc-c++ flex bison libpcap-devel
> openssl-devel python-devel swig zlib-devel
> >
> > the above command on centos is not installing latest packages of
> cmake,gcc,gcc-c++ which is required to compile zeek on centos
> >
> > Did anyone recently complied zeek on centos 7.any help is much
> appreciated
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191115/cbcfe3d6/attachment.html 

From dominik.charousset at corelight.com  Fri Nov 15 07:12:41 2019
From: dominik.charousset at corelight.com (Dominik Charousset)
Date: Fri, 15 Nov 2019 16:12:41 +0100
Subject: [Zeek] Zeek Compling
In-Reply-To: <CAKPzng-7B7ZJL__CnmV6LYCuGhs49fwJRfXGq_XZ-kFFLDhvqA@mail.gmail.com>
References: <CAKPzng_KCH-KaPed1JZ33FD8krnJfZE7OQ5axV=o+zS27eFyZg@mail.gmail.com>
	<7A023640-E464-4B9E-A82C-3001F3D44D5E@corelight.com>
	<CAKPzng-7B7ZJL__CnmV6LYCuGhs49fwJRfXGq_XZ-kFFLDhvqA@mail.gmail.com>
Message-ID: <72C9A8AA-4BD9-4684-B93E-2B8373EA6023@corelight.com>

> Thanks Dominik.. I installed cmake3 but that has a dependency of gcc7.2.0 above which I installed but when I compile Zeek it is complaining gcc should be 7.0 above 

By default, CMake picks up the system compiler, which is too old. If you?ve installed the devtoolkit then you can either enable it using "scl enable devtoolset-7 bash? or you can point CMake to the right compiler, e.g., by setting the CXX environment variable (for the configure script) or invoking CMake manually with -DCMAKE_CXX_COMPILER=<path>.


> On Fri, 15 Nov 2019 at 11:04 PM, Dominik Charousset <dominik.charousset at corelight.com> wrote:
> Hi,
> 
> you?ll need to install cmake3 and devtoolset-7.
> 
> The most recent documentation has a small section for the steps necessary on CentOS: https://docs.zeek.org/en/latest/install/install.html#required-dependencies.
> 
> Hope that helps.
> 
>     Dominik
> 
> 
> > On 15. Nov 2019, at 13:46, venkatesh bandari <austin522 at gmail.com> wrote:
> > 
> > Hello Team,
> > 
> > could someone help me on the issue iam facing
> > 
> > yum install cmake make gcc gcc-c++ flex bison libpcap-devel openssl-devel python-devel swig zlib-devel
> > 
> > the above command on centos is not installing latest packages of cmake,gcc,gcc-c++ which is required to compile zeek on centos
> > 
> > Did anyone recently complied zeek on centos 7.any help is much appreciated
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> 



From jayf at wheeling-nisshin.com  Fri Nov 15 12:29:02 2019
From: jayf at wheeling-nisshin.com (jayf at wheeling-nisshin.com)
Date: Fri, 15 Nov 2019 15:29:02 -0500
Subject: [Zeek] Certificate questions
Message-ID: <OF7E1D7963.AF54444C-ON852584B3.006F369B-852584B3.007085FB@wheeling-nisshin.com>



Greetings Zeek community,

I'm very new to Zeek, but really like what I'm see so far.  I need some
help or perhaps a bit of education though. I have it setup in a Security
Onion VM.

I see a lot of messages about SSL including "unable to get local issuer
certificate", which I understand COULD be self-signed certs.

I also see many, many SSL::Invalid_Server_Cert notices in Kibana.  Many
others say "SSL certificate validation failed with (self signed certificate
in certificate chain).

These would all be of interest, however they ALL point back to very
legitimate sources like Apple and Microsoft.  I find it hard to believe
that these major companies have problems with that many certificates and
servers.  Could this really be the case???

I could find very little information on Google regarding this.  One article
said something about Zeek not being able to match them up with root cert
servers or something like that.

Is it possible that Zeek is missing something like a list of root CAs or
something?  Is this just garbage caused by something else.  This will leave
me scratching my head until I come back on Monday.  I appreciate the help.

Jay Fluharty
Network Analyst
NS Wheeling-Nisshin Inc.
PO Box 635
Follansbee, WV 26037
jayf at wheeling-nisshin.com
1-304-527-4819
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191115/d3b33976/attachment.html 

From michalpurzynski1 at gmail.com  Sun Nov 17 02:44:45 2019
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Sun, 17 Nov 2019 02:44:45 -0800
Subject: [Zeek] Certificate questions
In-Reply-To: <OF7E1D7963.AF54444C-ON852584B3.006F369B-852584B3.007085FB@wheeling-nisshin.com>
References: <OF7E1D7963.AF54444C-ON852584B3.006F369B-852584B3.007085FB@wheeling-nisshin.com>
Message-ID: <CAJ6bFK101J74nBw3e_SS=VwrXiePqGnqkYBvwwKVF-k0n9h9Ew@mail.gmail.com>

Excellent question.

The reason you see those errors is the lack of the Root CA in Zeek's
certificate store.

Zeek, by default, uses Mozilla certificate store - the same one your
Firefox uses. Try going to one of these pages, like
https://slscr.update.microsoft.com in FF and you will see certificate
errors as well. You will not see them in Edge. Why's that?

For Microsoft, those certificates chain to a CA that has the root CA
certificate present in the windows certificate store, but nowhere else. For
Apple, the situation is similar - these root CA certificates are present on
the system level but no where else.

Since those are for services not accessed by general public, but things
like iCloud and software updates, these have never been submitted to us for
inclusion into Mozilla root CA program - and hence never landed in Zeek's
land.

An example right here here

subject
CN=slscr.update.microsoft.com,OU=DSP,O=Microsoft,L=Redmond,ST=WA,C=US

issued by
CN=Microsoft ECC Update Secure Server CA 2.1,O=Microsoft
Corporation,L=Redmond,ST=Washington,C=US

issued by
CN=Microsoft ECC Product Root Certificate Authority 2018,O=Microsoft
Corporation,L=Redmond,ST=Washington,C=US

Present in MS root store

There is a fix for that - you have to fetch those certificates with tools
like openssl or the latest Firefox (it's got this nice thing where you can
download the full chain), transform them into Zeek's scripts and include.

I think Justin wrote a nice script for that.

https://gist.github.com/JustinAzoff/7a1b92c976a2fa6e8601

mkdir tmp && cd tmp

openssl s_client -host slscr.update.microsoft.com -port 443 -showcerts <
/dev/null | sed -n '/BEGIN/,/END/p' | openssl x509 -outform DER > o.der

(do that for each CA - ignore the "verify error")

python ../gen_certs.py . cacert.zeek

And then you can @load the cacert.zeek in a script or in a local.zeek

On Fri, Nov 15, 2019 at 12:37 PM <jayf at wheeling-nisshin.com> wrote:

> Greetings Zeek community,
>
> I'm very new to Zeek, but really like what I'm see so far.  I need some
> help or perhaps a bit of education though. I have it setup in a Security
> Onion VM.
>
> I see a lot of messages about SSL including "unable to get local issuer
> certificate", which I understand COULD be self-signed certs.
>
> I also see many, many SSL::Invalid_Server_Cert notices in Kibana.  Many
> others say "SSL certificate validation failed with (self signed certificate
> in certificate chain).
>
> These would all be of interest, however they ALL point back to very
> legitimate sources like Apple and Microsoft.  I find it hard to believe
> that these major companies have problems with that many certificates and
> servers.  Could this really be the case???
>
> I could find very little information on Google regarding this.  One
> article said something about Zeek not being able to match them up with root
> cert servers or something like that.
>
> Is it possible that Zeek is missing something like a list of root CAs or
> something?  Is this just garbage caused by something else.  This will leave
> me scratching my head until I come back on Monday.  I appreciate the help.
>
> Jay Fluharty
> Network Analyst
> NS Wheeling-Nisshin Inc.
> PO Box 635
> Follansbee, WV 26037
> jayf at wheeling-nisshin.com
> 1-304-527-4819
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191117/a17b7d16/attachment.html 

From jayf at wheeling-nisshin.com  Mon Nov 18 10:44:21 2019
From: jayf at wheeling-nisshin.com (jayf at wheeling-nisshin.com)
Date: Mon, 18 Nov 2019 13:44:21 -0500
Subject: [Zeek] Certificate questions
In-Reply-To: <CAJ6bFK101J74nBw3e_SS=VwrXiePqGnqkYBvwwKVF-k0n9h9Ew@mail.gmail.com>
References: <OF7E1D7963.AF54444C-ON852584B3.006F369B-852584B3.007085FB@wheeling-nisshin.com>
	<CAJ6bFK101J74nBw3e_SS=VwrXiePqGnqkYBvwwKVF-k0n9h9Ew@mail.gmail.com>
Message-ID: <OF47F42F91.DE0F7445-ON852584B6.00661702-852584B6.0066F07B@wheeling-nisshin.com>


Thanks for the answer.  I need a bit of clarification though.

Your instructions said, openssl s_client -host slscr.update.microsoft.com
-port 443 -showcerts < /dev/null | sed -n '/BEGIN/,/END/p' | openssl x509
-outform DER > o.der
(do that for each CA - ignore the "verify error")'

Wouldn't fetching each certificate overwrite "o.der".  Should that be a
">>"?  Or do need to modify "o.der" for each certificate I fetch, then do a
"python ../gen_certs.py . cacert.zeek"?  Would that grab all the .der
files?  Do I need to make a seperate, unique "cacert.zeek" to put in my
"@load" statement.

Also, I put "@/path/to/cacert.zeek" file in my
"/opt/bro/share/bro/site/local.bro" file.  I assume that's where the
statement goes?

Sorry for all the questions, but I've not found this information elsewhere.
It really is appreciated.

Jay Fluharty
Network Analyst
NS Wheeling-Nisshin Inc.
PO Box 635
Follansbee, WV 26037
jayf at wheeling-nisshin.com
1-304-527-4819



From:	Micha? Purzy?ski <michalpurzynski1 at gmail.com>
To:	jayf at wheeling-nisshin.com
Cc:	zeek <zeek at zeek.org>
Date:	11/17/2019 05:45 AM
Subject:	Re: [Zeek] Certificate questions



Excellent question.

The reason you see those errors is the lack of the Root CA in Zeek's
certificate store.

Zeek, by default, uses Mozilla certificate store - the same one your
Firefox uses. Try going to one of these pages, like
https://slscr.update.microsoft.com in FF and you will see certificate
errors as well. You will not see them in Edge. Why's that?

For Microsoft, those certificates chain to a CA that has the root CA
certificate present in the windows certificate store, but nowhere else. For
Apple, the situation is similar - these root CA certificates are present on
the system level but no where else.

Since those are for services not accessed by general public, but things
like iCloud and software updates, these have never been submitted to us for
inclusion into Mozilla root CA program - and hence never landed in Zeek's
land.

An example right here here

subject
CN=slscr.update.microsoft.com,OU=DSP,O=Microsoft,L=Redmond,ST=WA,C=US

issued by
CN=Microsoft ECC Update Secure Server CA 2.1,O=Microsoft
Corporation,L=Redmond,ST=Washington,C=US

issued by
CN=Microsoft ECC Product Root Certificate Authority 2018,O=Microsoft
Corporation,L=Redmond,ST=Washington,C=US

Present in MS root store

There is a fix for that - you have to fetch those certificates with tools
like openssl or the latest Firefox (it's got this nice thing where you can
download the full chain), transform them into Zeek's scripts and include.

I think Justin wrote a nice script for that.

https://gist.github.com/JustinAzoff/7a1b92c976a2fa6e8601

mkdir tmp && cd tmp

openssl s_client -host slscr.update.microsoft.com -port 443 -showcerts
< /dev/null | sed -n '/BEGIN/,/END/p' | openssl x509 -outform DER > o.der

(do that for each CA - ignore the "verify error")

python ../gen_certs.py . cacert.zeek

And then you can @load the cacert.zeek in a script or in a local.zeek

On Fri, Nov 15, 2019 at 12:37 PM <jayf at wheeling-nisshin.com> wrote:
  Greetings Zeek community,

  I'm very new to Zeek, but really like what I'm see so far.? I need some
  help or perhaps a bit of education though. I have it setup in a Security
  Onion VM.

  I see a lot of messages about SSL including "unable to get local issuer
  certificate", which I understand COULD be self-signed certs.

  I also see many, many SSL::Invalid_Server_Cert notices in Kibana.? Many
  others say "SSL certificate validation failed with (self signed
  certificate in certificate chain).

  These would all be of interest, however they ALL point back to very
  legitimate sources like Apple and Microsoft.? I find it hard to believe
  that these major companies have problems with that many certificates and
  servers.? Could this really be the case???

  I could find very little information on Google regarding this.? One
  article said something about Zeek not being able to match them up with
  root cert servers or something like that.

  Is it possible that Zeek is missing something like a list of root CAs or
  something?? Is this just garbage caused by something else.? This will
  leave me scratching my head until I come back on Monday.? I appreciate
  the help.

  Jay Fluharty
  Network Analyst
  NS Wheeling-Nisshin Inc.
  PO Box 635
  Follansbee, WV 26037
  jayf at wheeling-nisshin.com
  1-304-527-4819
  _______________________________________________
  Zeek mailing list
  zeek at zeek.org
  http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

_____________________________________________________________________________

Scanned by IBM Email Security Management Services powered by
Symantec.Cloud. For more information please visit
http://www-935.ibm.com/services/us/index.wss/offerfamily/iss/a1026954
_____________________________________________________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191118/af6df3ac/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191118/af6df3ac/attachment.gif 

From michalpurzynski1 at gmail.com  Mon Nov 18 16:36:22 2019
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Mon, 18 Nov 2019 16:36:22 -0800
Subject: [Zeek] Certificate questions
In-Reply-To: <OF47F42F91.DE0F7445-ON852584B6.00661702-852584B6.0066F07B@wheeling-nisshin.com>
References: <OF7E1D7963.AF54444C-ON852584B3.006F369B-852584B3.007085FB@wheeling-nisshin.com>
	<CAJ6bFK101J74nBw3e_SS=VwrXiePqGnqkYBvwwKVF-k0n9h9Ew@mail.gmail.com>
	<OF47F42F91.DE0F7445-ON852584B6.00661702-852584B6.0066F07B@wheeling-nisshin.com>
Message-ID: <CAJ6bFK0qaF5+L3m4xoSTJMxrDzNFNpU5caySZxCjb21FLSsuAg@mail.gmail.com>

Answers inline. Keep asking, this is tricky to get right.

On Mon, Nov 18, 2019 at 10:44 AM <jayf at wheeling-nisshin.com> wrote:

> Thanks for the answer.  I need a bit of clarification though.
>
> Your instructions said, openssl s_client -host slscr.update.microsoft.com
> -port 443 -showcerts < /dev/null | sed -n '/BEGIN/,/END/p' | openssl x509
> -outform DER > o.der
> (do that for each CA - ignore the "verify error")'
>
> Wouldn't fetching each certificate overwrite "o.der".  Should that be a
> ">>"?
>
It will and that's why you want to redirect output for each server you're
fetching certificate chain from, to a new file.
gen_certs.py can then consume entire directory full of "der" files and
combined that into a hex representation for a Zeek script.

>  Or do need to modify "o.der" for each certificate I fetch, then do a
> "python ../gen_certs.py . cacert.zeek"?  Would that grab all the .der
> files?  Do I need to make a seperate, unique "cacert.zeek" to put in my
> "@load" statement.
>
So I just tested it a bit more and I need to modify my original instructions

openssl s_client -host init.push.apple.com -port 443 -showcerts < /dev/null

For each server inspect the output carefully - you want to grab the "CA"
and that's usually the last one. Ignore the sed, etc here, it won't,
unfortunately, work as I wanted it to.

For example, Apple returns

CONNECTED(00000005)
---
Certificate chain
 0 s:/CN=init.push.apple.com/O=Apple Inc./ST=California/C=US
   i:/CN=Apple Server Authentication CA/OU=Certification Authority/O=Apple
Inc./C=US
-----BEGIN CERTIFICATE-----
(...) <- omitted the cert for brevity - Apple's server certificate is here,
i.e. one end of the chain
-----END CERTIFICATE-----

 1 s:/CN=Apple Server Authentication CA/OU=Certification Authority/O=Apple
Inc./C=US
   i:/C=US/O=Apple Inc./OU=Apple Certification Authority/CN=Apple Root CA
-----BEGIN CERTIFICATE-----
(...) <- omitted the cert for brevity - an intermediate cert is here, no
need to worry about that
-----END CERTIFICATE-----

 2 s:/C=US/O=Apple Inc./OU=Apple Certification Authority/CN=Apple Root CA
   i:/C=US/O=Apple Inc./OU=Apple Certification Authority/CN=Apple Root CA
-----BEGIN CERTIFICATE-----
(...) HERE you have the Apple's Root CA certificate that you want to copy
and paste, together with the begin/end lines into a new file and convert
that into DER

openssl x509 -outform DER < appleca1.pem > appleca1.der

Repeat that for each server. This will take a couple of hours or even more
than a day ;)

Now having all "der" files you run

gen_certs . siteca.zeek

Now you can copy your new script (the siteca.zeek or whatever you want to
call that) to your "site" directory and @load siteca.zeek it from the
site/local.zeek file

That should take care of all these "private" and OS-specific services.


There's one more thing...

Johanna found out that Zeek by default won't correctly chain up
certificates if the server is not configured correctly. By correctly I mean
- any server should send a full certificate chain, but some don't and we
end up in a situation where all Zeek can see is the root CA certificate
(from its database), the server's certificate not none of the intermediate
certificates.

Browsers deal with it by caching a lot of intermediate certificates and
then doing insane heuristics trying to verify. Well, they have been doing
it for years and for billions of people ;)

I would advise to first deal with SSL::Invalid_Server_Cert by finding out
who generates the "unable to get local issuer certificate" first, i.e. you
want to find the root CA cert and convert it and add to Zeek.

Next, you can decide what do you want to do with self-signed certificates.
You could add them in the same way, or alert on them, depending on your
organization policy.


>
> Also, I put "@/path/to/cacert.zeek" file in my
> "/opt/bro/share/bro/site/local.bro" file.  I assume that's where the
> statement goes?
>
> Sorry for all the questions, but I've not found this information
> elsewhere.  It really is appreciated.
>
> Jay Fluharty
> Network Analyst
> NS Wheeling-Nisshin Inc.
> PO Box 635
> Follansbee, WV 26037
> jayf at wheeling-nisshin.com
> 1-304-527-4819
>
> [image: Inactive hide details for Micha? Purzy?ski ---11/17/2019 05:45:24
> AM---Excellent question. The reason you see those errors is]Micha?
> Purzy?ski ---11/17/2019 05:45:24 AM---Excellent question. The reason you
> see those errors is the lack of the Root CA in Zeek's
>
> From: Micha? Purzy?ski <michalpurzynski1 at gmail.com>
> To: jayf at wheeling-nisshin.com
> Cc: zeek <zeek at zeek.org>
> Date: 11/17/2019 05:45 AM
> Subject: Re: [Zeek] Certificate questions
> ------------------------------
>
>
>
> Excellent question.
>
> The reason you see those errors is the lack of the Root CA in Zeek's
> certificate store.
>
> Zeek, by default, uses Mozilla certificate store - the same one your
> Firefox uses. Try going to one of these pages, like
> *https://slscr.update.microsoft.com* <https://slscr.update.microsoft.com/> in
> FF and you will see certificate errors as well. You will not see them in
> Edge. Why's that?
>
> For Microsoft, those certificates chain to a CA that has the root CA
> certificate present in the windows certificate store, but nowhere else. For
> Apple, the situation is similar - these root CA certificates are present on
> the system level but no where else.
>
> Since those are for services not accessed by general public, but things
> like iCloud and software updates, these have never been submitted to us for
> inclusion into Mozilla root CA program - and hence never landed in Zeek's
> land.
>
> An example right here here
>
> subject
> CN=*slscr.update.microsoft.com* <http://slscr.update.microsoft.com/>
> ,OU=DSP,O=Microsoft,L=Redmond,ST=WA,C=US
>
> issued by
> CN=Microsoft ECC Update Secure Server CA 2.1,O=Microsoft
> Corporation,L=Redmond,ST=Washington,C=US
>
> issued by
> CN=Microsoft ECC Product Root Certificate Authority 2018,O=Microsoft
> Corporation,L=Redmond,ST=Washington,C=US
>
> Present in MS root store
>
> There is a fix for that - you have to fetch those certificates with tools
> like openssl or the latest Firefox (it's got this nice thing where you can
> download the full chain), transform them into Zeek's scripts and include.
>
> I think Justin wrote a nice script for that.
>
> *https://gist.github.com/JustinAzoff/7a1b92c976a2fa6e8601*
> <https://gist.github.com/JustinAzoff/7a1b92c976a2fa6e8601>
>
> mkdir tmp && cd tmp
>
> openssl s_client -host *slscr.update.microsoft.com*
> <http://slscr.update.microsoft.com/> -port 443 -showcerts < /dev/null |
> sed -n '/BEGIN/,/END/p' | openssl x509 -outform DER > o.der
>
> (do that for each CA - ignore the "verify error")
>
> python ../gen_certs.py . cacert.zeek
>
> And then you can @load the cacert.zeek in a script or in a local.zeek
>
> On Fri, Nov 15, 2019 at 12:37 PM <*jayf at wheeling-nisshin.com*
> <jayf at wheeling-nisshin.com>> wrote:
>
>    Greetings Zeek community,
>
>    I'm very new to Zeek, but really like what I'm see so far.  I need
>    some help or perhaps a bit of education though. I have it setup in a
>    Security Onion VM.
>
>    I see a lot of messages about SSL including "unable to get local
>    issuer certificate", which I understand COULD be self-signed certs.
>
>    I also see many, many SSL::Invalid_Server_Cert notices in Kibana.
>    Many others say "SSL certificate validation failed with (self signed
>    certificate in certificate chain).
>
>    These would all be of interest, however they ALL point back to very
>    legitimate sources like Apple and Microsoft.  I find it hard to believe
>    that these major companies have problems with that many certificates and
>    servers.  Could this really be the case???
>
>    I could find very little information on Google regarding this.  One
>    article said something about Zeek not being able to match them up with root
>    cert servers or something like that.
>
>    Is it possible that Zeek is missing something like a list of root CAs
>    or something?  Is this just garbage caused by something else.  This will
>    leave me scratching my head until I come back on Monday.  I appreciate the
>    help.
>
>    Jay Fluharty
>    Network Analyst
>    NS Wheeling-Nisshin Inc.
>    PO Box 635
>    Follansbee, WV 26037
> *jayf at wheeling-nisshin.com* <jayf at wheeling-nisshin.com>
>    1-304-527-4819
>    _______________________________________________
>    Zeek mailing list
> *zeek at zeek.org* <zeek at zeek.org>
> *http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek*
>    <http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek>
>
>
>
> _____________________________________________________________________________
> Scanned by IBM Email Security Management Services powered by
> Symantec.Cloud. For more information please visit
> http://www-935.ibm.com/services/us/index.wss/offerfamily/iss/a1026954
>
> _____________________________________________________________________________
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191118/0dd76aed/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191118/0dd76aed/attachment.gif 

From rahulbroids at gmail.com  Tue Nov 19 04:45:15 2019
From: rahulbroids at gmail.com (rahul rakesh)
Date: Tue, 19 Nov 2019 18:15:15 +0530
Subject: [Zeek] how to specify ip address through variables
Message-ID: <CADJmW1=RwPwLvFASEyKQdJtt=AHKe0PLAQV4COMY8mhw5dPz3A@mail.gmail.com>

Hi all,

First Question:
------------------
How I can specify the ip address of src-ip/dst-ip <cmp> <address-list>
through variable like $External_Net like in snort signatures.

Suppose same ip address is there in many signatures.How can I define it at
one place.

Second Q
-------------
after event is generated,then how can i call either c++ code or python code
from bro scripts.If it is possible ,can you provide me link.
(Some where i read that through bro scripts, we can call python or c++ code)

Third Q
-----------
Currently Zeek is used at which super computing facility.Is it used at
NERSC center. Just  to know that. You can provide me link also.

Thank you so much.

Ravi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191119/65624c01/attachment-0001.html 

From brittany.donowho at nrl.navy.mil  Tue Nov 19 06:26:52 2019
From: brittany.donowho at nrl.navy.mil (Brittany Donowho)
Date: Tue, 19 Nov 2019 09:26:52 -0500
Subject: [Zeek] Possible bug in Stats framework?
Message-ID: <C5F3B53D-D04D-48DE-B168-ABD99AB57ADA@nrl.navy.mil>

Hi,

I?ve been trying to use the stats framework on PCAP but the logs were always off by several packets (depending on the size of the trace). The numbers match up when I moved the code  

if (zeek_is_terminating() )
                return;

To the end of stats.zeek as follows:

Log::write(Stats::LOG, info);

if ( zeek_is_terminating() )
        return;

schedule report_interval { check_stats(nettime, ns, cs, ps, es, rs, ts, fs, ds) };

Can someone verify that this is correct? I?m wondering if this edit will mess with live traffic stats.

Thanks!

Brittany Donowho






-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191119/1f6d84de/attachment.html 

From mauro.palumbo at aizoon.it  Tue Nov 19 07:44:37 2019
From: mauro.palumbo at aizoon.it (Palumbo Mauro)
Date: Tue, 19 Nov 2019 15:44:37 +0000
Subject: [Zeek] R:  how to specify ip address through variables
In-Reply-To: <CADJmW1=RwPwLvFASEyKQdJtt=AHKe0PLAQV4COMY8mhw5dPz3A@mail.gmail.com>
References: <CADJmW1=RwPwLvFASEyKQdJtt=AHKe0PLAQV4COMY8mhw5dPz3A@mail.gmail.com>
Message-ID: <0057b7bc7d904ed5adf81e7d4b13fc8b@SRVEX03.aizoon.local>

Hi Ravi,

For the 1Q, you can easily define variables for ip addresses in zeek using the type ?addr?. However, I don?t think you can do it in a signature file.

For the 2Q, have a look at https://docs.zeek.org/en/stable/scripts/base/utils/exec.zeek.html#id-Exec::run

For the 3Q, I think they use it at CERN in Geneva too.

Mauro

Da: zeek-bounces at zeek.org [mailto:zeek-bounces at zeek.org] Per conto di rahul rakesh
Inviato: marted? 19 novembre 2019 13:45
A: zeek at zeek.org
Oggetto: [Zeek] how to specify ip address through variables

Hi all,

First Question:
------------------
How I can specify the ip address of src-ip/dst-ip <cmp> <address-list>
through variable like $External_Net like in snort signatures.

Suppose same ip address is there in many signatures.How can I define it at one place.

Second Q
-------------
after event is generated,then how can i call either c++ code or python code from bro scripts.If it is possible ,can you provide me link.
(Some where i read that through bro scripts, we can call python or c++ code)

Third Q
-----------
Currently Zeek is used at which super computing facility.Is it used at NERSC center. Just  to know that. You can provide me link also.

Thank you so much.

Ravi


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191119/bfb3d294/attachment.html 

From monahbaki at gmail.com  Wed Nov 20 08:49:41 2019
From: monahbaki at gmail.com (Monah Baki)
Date: Wed, 20 Nov 2019 11:49:41 -0500
Subject: [Zeek] Detecting software
Message-ID: <CALP3=x9rDu_+gKMGcriKgR4yNq+=NsmePtwrWSGvoOCdS=mvgw@mail.gmail.com>

Hi all,

We have a server that bro detected with port 4545 in listening mode. Is
there a way to find what software had that port opened or any specific
details about it?

Thanks
Monah
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191120/30287ca3/attachment.html 

From vern at corelight.com  Wed Nov 20 10:54:07 2019
From: vern at corelight.com (Vern Paxson)
Date: Wed, 20 Nov 2019 10:54:07 -0800
Subject: [Zeek] Detecting software
In-Reply-To: <CALP3=x9rDu_+gKMGcriKgR4yNq+=NsmePtwrWSGvoOCdS=mvgw@mail.gmail.com>
	(Wed, 20 Nov 2019 11:49:41 EST).
Message-ID: <20191120185407.63DE72C400E@rock.ICSI.Berkeley.EDU>

> We have a server that bro detected with port 4545 in listening mode. Is
> there a way to find what software had that port opened or any specific
> details about it?

Zeek doesn't provide additional insight into servers running protocols for
applications unknown to Zeek.  In practical terms, you could try capturing
a pcap of the traffic and then inspecting it using say Wireshark to see
if you can figure out what it is.

		Vern

From rlabaza at gmail.com  Wed Nov 20 11:28:45 2019
From: rlabaza at gmail.com (Randy Labaza)
Date: Wed, 20 Nov 2019 14:28:45 -0500
Subject: [Zeek] Detecting software
In-Reply-To: <20191120185407.63DE72C400E@rock.ICSI.Berkeley.EDU>
References: <CALP3=x9rDu_+gKMGcriKgR4yNq+=NsmePtwrWSGvoOCdS=mvgw@mail.gmail.com>
	<20191120185407.63DE72C400E@rock.ICSI.Berkeley.EDU>
Message-ID: <CAAsJwLXmxF5nJ_h3SfbtniZXma2uEOxnJvgq4w0zWttv2zpg_A@mail.gmail.com>

You might try some combination of lsof -i:4545, get the PID, then use ps to
find the process...
Regards.
rl.


On Wed, Nov 20, 2019 at 2:02 PM Vern Paxson <vern at corelight.com> wrote:

> > We have a server that bro detected with port 4545 in listening mode. Is
> > there a way to find what software had that port opened or any specific
> > details about it?
>
> Zeek doesn't provide additional insight into servers running protocols for
> applications unknown to Zeek.  In practical terms, you could try capturing
> a pcap of the traffic and then inspecting it using say Wireshark to see
> if you can figure out what it is.
>
>                 Vern
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191120/246e0403/attachment.html 

From jmellander at lbl.gov  Wed Nov 20 11:54:13 2019
From: jmellander at lbl.gov (Jim Mellander)
Date: Wed, 20 Nov 2019 11:54:13 -0800
Subject: [Zeek] Detecting software
In-Reply-To: <CAAsJwLXmxF5nJ_h3SfbtniZXma2uEOxnJvgq4w0zWttv2zpg_A@mail.gmail.com>
References: <CALP3=x9rDu_+gKMGcriKgR4yNq+=NsmePtwrWSGvoOCdS=mvgw@mail.gmail.com>
	<20191120185407.63DE72C400E@rock.ICSI.Berkeley.EDU>
	<CAAsJwLXmxF5nJ_h3SfbtniZXma2uEOxnJvgq4w0zWttv2zpg_A@mail.gmail.com>
Message-ID: <CADju=b5WdNx+=dM1EhddoDSiCCOc61QPAD572mozJi+-=MGW5w@mail.gmail.com>

Just to be clear, Randy's suggestions should be executed on the server
listening on port 4545, not the bro/zeek system.

On Wed, Nov 20, 2019 at 11:31 AM Randy Labaza <rlabaza at gmail.com> wrote:

> You might try some combination of lsof -i:4545, get the PID, then use ps
> to find the process...
> Regards.
> rl.
>
>
> On Wed, Nov 20, 2019 at 2:02 PM Vern Paxson <vern at corelight.com> wrote:
>
>> > We have a server that bro detected with port 4545 in listening mode. Is
>> > there a way to find what software had that port opened or any specific
>> > details about it?
>>
>> Zeek doesn't provide additional insight into servers running protocols for
>> applications unknown to Zeek.  In practical terms, you could try capturing
>> a pcap of the traffic and then inspecting it using say Wireshark to see
>> if you can figure out what it is.
>>
>>                 Vern
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191120/a2f72e6d/attachment.html 

From jerome.grandvalet at hotmail.fr  Thu Nov 21 04:49:26 2019
From: jerome.grandvalet at hotmail.fr (=?utf-8?B?asOpcsO0bWUgZ3JhbmR2YWxldA==?=)
Date: Thu, 21 Nov 2019 12:49:26 +0000
Subject: [Zeek] [zeek] readfile cluster mode
Message-ID: <AM0PR06MB6162D6076B9785777E6088339A4E0@AM0PR06MB6162.eurprd06.prod.outlook.com>

Hello,

Is there a mean to use readfile option (-r) with cluster mode ?
Maybe with some scripts ?

Many thanks.

J?r?me


From austin522 at gmail.com  Thu Nov 21 07:25:53 2019
From: austin522 at gmail.com (venkatesh bandari)
Date: Thu, 21 Nov 2019 23:25:53 +0800
Subject: [Zeek] af_packet
Message-ID: <CAKPzng9V-6kytfByXjv2qxm+cD_3cObK+hj0KC_zz4Yy0LS+gg@mail.gmail.com>

Hello Team,

Iam trying to install af_packet and i see we can only install using
package manager.

could you please help to answer if there is a alternative way  to install
af_packet

Thanks
Venkatesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191121/87a3d08a/attachment.html 

From ericooi at gmail.com  Thu Nov 21 07:35:13 2019
From: ericooi at gmail.com (ericooi at gmail.com)
Date: Thu, 21 Nov 2019 09:35:13 -0600
Subject: [Zeek] af_packet
In-Reply-To: <CAKPzng9V-6kytfByXjv2qxm+cD_3cObK+hj0KC_zz4Yy0LS+gg@mail.gmail.com>
References: <CAKPzng9V-6kytfByXjv2qxm+cD_3cObK+hj0KC_zz4Yy0LS+gg@mail.gmail.com>
Message-ID: <37094E51-DD4E-438E-A4CA-9E9862CF2AEE@gmail.com>

Any particular reason you don?t want to use the package manager?  It?s designed to simplify the process.

I wrote a how-to guide for installation on CentOS 8 that covers using the package manager: https://www.ericooi.com/zeekurity-zen-part-ii-zeek-package-manager/ <https://www.ericooi.com/zeekurity-zen-part-ii-zeek-package-manager/>

> On Nov 21, 2019, at 9:25 AM, venkatesh bandari <austin522 at gmail.com> wrote:
> 
> Hello Team,
> 
> Iam trying to install af_packet and i see we can only install using package manager.
> 
> could you please help to answer if there is a alternative way  to install af_packet
> 
> Thanks
> Venkatesh
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191121/e75667cd/attachment.html 

From austin522 at gmail.com  Thu Nov 21 07:37:08 2019
From: austin522 at gmail.com (venkatesh bandari)
Date: Thu, 21 Nov 2019 23:37:08 +0800
Subject: [Zeek] af_packet
In-Reply-To: <37094E51-DD4E-438E-A4CA-9E9862CF2AEE@gmail.com>
References: <CAKPzng9V-6kytfByXjv2qxm+cD_3cObK+hj0KC_zz4Yy0LS+gg@mail.gmail.com>
	<37094E51-DD4E-438E-A4CA-9E9862CF2AEE@gmail.com>
Message-ID: <CAKPzng_9=LWJgEWYN7nMrWnfSK03bQR1OoTFVUDiWmcRNben4g@mail.gmail.com>

Thank you Eric.i have gone through that article.iam  having hard time to
install zkg via pip(ssl cert error)

So thought of Checking if there is alternative way



On Thu, 21 Nov 2019 at 11:35 PM, ericooi at gmail.com <ericooi at gmail.com>
wrote:

> Any particular reason you don?t want to use the package manager?  It?s
> designed to simplify the process.
>
> I wrote a how-to guide for installation on CentOS 8 that covers using the
> package manager:
> https://www.ericooi.com/zeekurity-zen-part-ii-zeek-package-manager/
>
> On Nov 21, 2019, at 9:25 AM, venkatesh bandari <austin522 at gmail.com>
> wrote:
>
> Hello Team,
>
> Iam trying to install af_packet and i see we can only install using
> package manager.
>
> could you please help to answer if there is a alternative way  to install
> af_packet
>
> Thanks
> Venkatesh
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191121/06a68d3a/attachment.html 

From jlay at slave-tothe-box.net  Thu Nov 21 07:39:07 2019
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 21 Nov 2019 08:39:07 -0700
Subject: [Zeek] af_packet
In-Reply-To: <CAKPzng9V-6kytfByXjv2qxm+cD_3cObK+hj0KC_zz4Yy0LS+gg@mail.gmail.com>
References: <CAKPzng9V-6kytfByXjv2qxm+cD_3cObK+hj0KC_zz4Yy0LS+gg@mail.gmail.com>
Message-ID: <ed6a1761855e4f89901ea91acededc05@slave-tothe-box.net>

On 2019-11-21 08:25, venkatesh bandari wrote:
> Hello Team,
> 
> Iam trying to install af_packet and i see we can only install using
> package manager.
> 
> could you please help to answer if there is a alternative way  to
> install af_packet
> 
> Thanks
> Venkatesh
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

Info is here:

https://github.com/J-Gras/bro-af_packet-plugin

Make sure you tell configure where to go like so:

./configure --bro-dist=../bro-3.0.0 
--install-root=/opt/bro/lib/bro/plugins

James

From michalpurzynski1 at gmail.com  Thu Nov 21 15:09:57 2019
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Thu, 21 Nov 2019 15:09:57 -0800
Subject: [Zeek] af_packet
In-Reply-To: <37094E51-DD4E-438E-A4CA-9E9862CF2AEE@gmail.com>
References: <CAKPzng9V-6kytfByXjv2qxm+cD_3cObK+hj0KC_zz4Yy0LS+gg@mail.gmail.com>
	<37094E51-DD4E-438E-A4CA-9E9862CF2AEE@gmail.com>
Message-ID: <CAJ6bFK3oJfd7HPfycNLmouhSbbRz2KEfSMXnTn8nZTLkVmhSBw@mail.gmail.com>

TBH I see quite a few reasons why plugins and Zeek should not be installed
like that article describes.

- everything should be packaged. This includes Zeek and all plugins, each
as a separate package. Running zkg to install plugin on production will
result in building code on the server (and we don't even have compilers
there).
- the sudo + pip install means you now have python packages installed for
the entire OS but outside of the OS control and packaging, making
vulnerability management difficult (most likely never detected, forgotten
and left there)
- looks like zeek user can sudo to root. There's no need for that dangerous
capability.
- cap_net_admin is not needed and dangerous
- assigning any capabilities to zeekctl is not needed and also dangerous
- and those 3 affiliation links at the bottom...

On Thu, Nov 21, 2019 at 7:46 AM ericooi at gmail.com <ericooi at gmail.com> wrote:

> Any particular reason you don?t want to use the package manager?  It?s
> designed to simplify the process.
>
> I wrote a how-to guide for installation on CentOS 8 that covers using the
> package manager:
> https://www.ericooi.com/zeekurity-zen-part-ii-zeek-package-manager/
>
> On Nov 21, 2019, at 9:25 AM, venkatesh bandari <austin522 at gmail.com>
> wrote:
>
> Hello Team,
>
> Iam trying to install af_packet and i see we can only install using
> package manager.
>
> could you please help to answer if there is a alternative way  to install
> af_packet
>
> Thanks
> Venkatesh
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191121/2a448b7c/attachment.html 

From ericooi at gmail.com  Thu Nov 21 15:13:26 2019
From: ericooi at gmail.com (Eric Ooi)
Date: Thu, 21 Nov 2019 23:13:26 +0000
Subject: [Zeek] af_packet
In-Reply-To: <CAJ6bFK3oJfd7HPfycNLmouhSbbRz2KEfSMXnTn8nZTLkVmhSBw@mail.gmail.com>
References: <CAKPzng9V-6kytfByXjv2qxm+cD_3cObK+hj0KC_zz4Yy0LS+gg@mail.gmail.com>
	<37094E51-DD4E-438E-A4CA-9E9862CF2AEE@gmail.com>,
	<CAJ6bFK3oJfd7HPfycNLmouhSbbRz2KEfSMXnTn8nZTLkVmhSBw@mail.gmail.com>
Message-ID: <DM6PR04MB6251F48E2E333028B1578BE3A14E0@DM6PR04MB6251.namprd04.prod.outlook.com>

Ouch.  Can?t please everyone I guess.


________________________________
From: Micha? Purzy?ski <michalpurzynski1 at gmail.com>
Sent: Thursday, November 21, 2019 5:10 PM
To: ericooi at gmail.com
Cc: venkatesh bandari; zeek
Subject: Re: [Zeek] af_packet

TBH I see quite a few reasons why plugins and Zeek should not be installed like that article describes.

- everything should be packaged. This includes Zeek and all plugins, each as a separate package. Running zkg to install plugin on production will result in building code on the server (and we don't even have compilers there).
- the sudo + pip install means you now have python packages installed for the entire OS but outside of the OS control and packaging, making vulnerability management difficult (most likely never detected, forgotten and left there)
- looks like zeek user can sudo to root. There's no need for that dangerous capability.
- cap_net_admin is not needed and dangerous
- assigning any capabilities to zeekctl is not needed and also dangerous
- and those 3 affiliation links at the bottom...

On Thu, Nov 21, 2019 at 7:46 AM ericooi at gmail.com<mailto:ericooi at gmail.com> <ericooi at gmail.com<mailto:ericooi at gmail.com>> wrote:
Any particular reason you don?t want to use the package manager?  It?s designed to simplify the process.

I wrote a how-to guide for installation on CentOS 8 that covers using the package manager: https://www.ericooi.com/zeekurity-zen-part-ii-zeek-package-manager/

On Nov 21, 2019, at 9:25 AM, venkatesh bandari <austin522 at gmail.com<mailto:austin522 at gmail.com>> wrote:

Hello Team,

Iam trying to install af_packet and i see we can only install using package manager.

could you please help to answer if there is a alternative way  to install af_packet

Thanks
Venkatesh
_______________________________________________
Zeek mailing list
zeek at zeek.org<mailto:zeek at zeek.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

_______________________________________________
Zeek mailing list
zeek at zeek.org<mailto:zeek at zeek.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191121/58a39f41/attachment.html 

From austin522 at gmail.com  Thu Nov 21 16:54:56 2019
From: austin522 at gmail.com (venkatesh bandari)
Date: Fri, 22 Nov 2019 08:54:56 +0800
Subject: [Zeek] workers
Message-ID: <CAKPzng8wo=zfNWYQsQgeRQgrka0RX2xLpG2-GCB+Dt9o3x8R4A@mail.gmail.com>

Hi Team,

I added two workers and pinned the CPU's to solve the packet drops
issue.When i started zeek,all the pinned CPU's are hitting 100%

just want to check if iam doing something wrong

[worker-1]

type=worker

host=localhost

interface=eno49

lb_method=pf_ring

lb_procs=16

pin_cpus=2,3,4,5,6,7,8,9,22,23,24,25,26,27,28,29



[worker-2]

type=worker

host=localhost

interface=eno49

lb_method=pf_ring

lb_procs=17

pin_cpus=11,12,13,14,15,16,17,18,19,30,31,32,33,34,35,36,37
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191122/f1eaa3ff/attachment.html 

From austin522 at gmail.com  Fri Nov 22 06:14:23 2019
From: austin522 at gmail.com (venkatesh bandari)
Date: Fri, 22 Nov 2019 22:14:23 +0800
Subject: [Zeek] af_packet vs pfring
Message-ID: <CAKPzng-_xXKWL=-+_ztg95mfpLgA0mzq1vjjC23AqPcoxSCRvg@mail.gmail.com>

Hello team,

could anyone share ideas/thoughts which one to use either pf_ring or
af_packet to load balance.

any recommendations.

Thanks
Venkatesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191122/12b9ca75/attachment.html 

From craig.edgmand at okstate.edu  Fri Nov 22 06:39:05 2019
From: craig.edgmand at okstate.edu (Edgmand, Craig)
Date: Fri, 22 Nov 2019 14:39:05 +0000
Subject: [Zeek] af_packet vs pfring
In-Reply-To: <CAKPzng-_xXKWL=-+_ztg95mfpLgA0mzq1vjjC23AqPcoxSCRvg@mail.gmail.com>
References: <CAKPzng-_xXKWL=-+_ztg95mfpLgA0mzq1vjjC23AqPcoxSCRvg@mail.gmail.com>
Message-ID: <CY4PR03MB2615CC9D37682DE52B9B018DE3490@CY4PR03MB2615.namprd03.prod.outlook.com>

If your kernel supports it you should use af_packet.  It is built in and pf_ring is a giant pain in the ass.  ?


From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> On Behalf Of venkatesh bandari
Sent: Friday, November 22, 2019 8:14 AM
To: zeek <zeek at zeek.org>
Subject: [Zeek] af_packet vs pfring

**External Email - Please verify sender email address before responding.**
Hello team,

could anyone share ideas/thoughts which one to use either pf_ring or af_packet to load balance.

any recommendations.

Thanks
Venkatesh

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191122/3cdcf5b8/attachment-0001.html 

From ericooi at gmail.com  Fri Nov 22 09:30:32 2019
From: ericooi at gmail.com (ericooi at gmail.com)
Date: Fri, 22 Nov 2019 11:30:32 -0600
Subject: [Zeek] af_packet
In-Reply-To: <DM6PR04MB6251F48E2E333028B1578BE3A14E0@DM6PR04MB6251.namprd04.prod.outlook.com>
References: <CAKPzng9V-6kytfByXjv2qxm+cD_3cObK+hj0KC_zz4Yy0LS+gg@mail.gmail.com>
	<37094E51-DD4E-438E-A4CA-9E9862CF2AEE@gmail.com>
	<CAJ6bFK3oJfd7HPfycNLmouhSbbRz2KEfSMXnTn8nZTLkVmhSBw@mail.gmail.com>
	<DM6PR04MB6251F48E2E333028B1578BE3A14E0@DM6PR04MB6251.namprd04.prod.outlook.com>
Message-ID: <491473DC-15CA-4B0B-9DF0-E0B5C8112B09@gmail.com>

Ok, I updated the articles based on Michal?s feedback.  I?m still learning and appreciate the comments.  

I understand the reasoning around packaging everything.  That?s a bit beyond me so I?ll have to dig more into that.

The 3 links, well, hey, it?s my blog. Don?t click if you don?t want. :P


> On Nov 21, 2019, at 5:13 PM, Eric Ooi <ericooi at gmail.com> wrote:
> 
> Ouch.  Can?t please everyone I guess.
> 
>  
> From: Micha? Purzy?ski <michalpurzynski1 at gmail.com>
> Sent: Thursday, November 21, 2019 5:10 PM
> To: ericooi at gmail.com
> Cc: venkatesh bandari; zeek
> Subject: Re: [Zeek] af_packet
>  
> TBH I see quite a few reasons why plugins and Zeek should not be installed like that article describes.
> 
> - everything should be packaged. This includes Zeek and all plugins, each as a separate package. Running zkg to install plugin on production will result in building code on the server (and we don't even have compilers there).
> - the sudo + pip install means you now have python packages installed for the entire OS but outside of the OS control and packaging, making vulnerability management difficult (most likely never detected, forgotten and left there)
> - looks like zeek user can sudo to root. There's no need for that dangerous capability.
> - cap_net_admin is not needed and dangerous
> - assigning any capabilities to zeekctl is not needed and also dangerous
> - and those 3 affiliation links at the bottom...
> 
> On Thu, Nov 21, 2019 at 7:46 AM ericooi at gmail.com <mailto:ericooi at gmail.com> <ericooi at gmail.com <mailto:ericooi at gmail.com>> wrote:
> Any particular reason you don?t want to use the package manager?  It?s designed to simplify the process.
> 
> I wrote a how-to guide for installation on CentOS 8 that covers using the package manager: https://www.ericooi.com/zeekurity-zen-part-ii-zeek-package-manager/ <https://www.ericooi.com/zeekurity-zen-part-ii-zeek-package-manager/>
> 
>> On Nov 21, 2019, at 9:25 AM, venkatesh bandari <austin522 at gmail.com <mailto:austin522 at gmail.com>> wrote:
>> 
>> Hello Team,
>> 
>> Iam trying to install af_packet and i see we can only install using package manager.
>> 
>> could you please help to answer if there is a alternative way  to install af_packet
>> 
>> Thanks
>> Venkatesh
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org <mailto:zeek at zeek.org>
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek <http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org <mailto:zeek at zeek.org>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek <http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191122/62a9065f/attachment.html 

From michalpurzynski1 at gmail.com  Fri Nov 22 11:59:34 2019
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Fri, 22 Nov 2019 11:59:34 -0800
Subject: [Zeek] af_packet
In-Reply-To: <491473DC-15CA-4B0B-9DF0-E0B5C8112B09@gmail.com>
References: <491473DC-15CA-4B0B-9DF0-E0B5C8112B09@gmail.com>
Message-ID: <B65CAC3A-4DAB-4FF8-B706-E423643A9A5D@gmail.com>

Thanks!

> On Nov 22, 2019, at 9:30 AM, "ericooi at gmail.com" <ericooi at gmail.com> wrote:
> 
> ?Ok, I updated the articles based on Michal?s feedback.  I?m still learning and appreciate the comments.  
> 
> I understand the reasoning around packaging everything.  That?s a bit beyond me so I?ll have to dig more into that.
> 
> The 3 links, well, hey, it?s my blog. Don?t click if you don?t want. :P
> 
> 
>> On Nov 21, 2019, at 5:13 PM, Eric Ooi <ericooi at gmail.com> wrote:
>> 
>> Ouch.  Can?t please everyone I guess.
>> 
>>  
>> From: Micha? Purzy?ski <michalpurzynski1 at gmail.com>
>> Sent: Thursday, November 21, 2019 5:10 PM
>> To: ericooi at gmail.com
>> Cc: venkatesh bandari; zeek
>> Subject: Re: [Zeek] af_packet
>>  
>> TBH I see quite a few reasons why plugins and Zeek should not be installed like that article describes.
>> 
>> - everything should be packaged. This includes Zeek and all plugins, each as a separate package. Running zkg to install plugin on production will result in building code on the server (and we don't even have compilers there).
>> - the sudo + pip install means you now have python packages installed for the entire OS but outside of the OS control and packaging, making vulnerability management difficult (most likely never detected, forgotten and left there)
>> - looks like zeek user can sudo to root. There's no need for that dangerous capability.
>> - cap_net_admin is not needed and dangerous
>> - assigning any capabilities to zeekctl is not needed and also dangerous
>> - and those 3 affiliation links at the bottom...
>> 
>>> On Thu, Nov 21, 2019 at 7:46 AM ericooi at gmail.com <ericooi at gmail.com> wrote:
>>> Any particular reason you don?t want to use the package manager?  It?s designed to simplify the process.
>>> 
>>> I wrote a how-to guide for installation on CentOS 8 that covers using the package manager: https://www.ericooi.com/zeekurity-zen-part-ii-zeek-package-manager/
>>> 
>>>> On Nov 21, 2019, at 9:25 AM, venkatesh bandari <austin522 at gmail.com> wrote:
>>>> 
>>>> Hello Team,
>>>> 
>>>> Iam trying to install af_packet and i see we can only install using package manager.
>>>> 
>>>> could you please help to answer if there is a alternative way  to install af_packet
>>>> 
>>>> Thanks
>>>> Venkatesh
>>>> _______________________________________________
>>>> Zeek mailing list
>>>> zeek at zeek.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>> 
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191122/79c5d4e1/attachment.html 

From andrew at aklaus.ca  Mon Nov 25 10:45:34 2019
From: andrew at aklaus.ca (Andrew Klaus)
Date: Mon, 25 Nov 2019 11:45:34 -0700
Subject: [Zeek] Making Broker Optional in Script
Message-ID: <CAGENE8tCKOsrGc2d_sm1ZDedoPbnfWYNA6Mwxqd1sPo-G49=kQ@mail.gmail.com>

Hello,

I have a Zeek script that I would like to add optional Broker functionality
to. I don't want it to be a requirement, so I'm adding run-time redef
options that can be toggled to enable/disable it.

Something like:
-------
export {
    const broker_enable = F &redef;
}

@if (MODULE::broker_enable)
event bro_init()
{
  Broker::listen("127.0.0.1", 9999/tcp);
}
@endif
-------

When I attempt to add this line to my local.bro/zeek file after the @load
package:
---
redef MODULE::broker_enable = T;
---

It won't override broker_enable and thus doesn't listen on the socket. I
assume that it's because the @load happens before the redef in local.bro,
and thus doesn't override when it checks for the @if ?

Is there a better way that I can do this? Like checking if Broker is
actually available?  I tried this, but it doesn't work either:
---
@ifdef (Broker)
print "Broker Enabled";
@endif
---

Thanks in advance,

Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191125/303e2958/attachment.html 

From nahum at us.ibm.com  Mon Nov 25 11:49:11 2019
From: nahum at us.ibm.com (Erich M Nahum)
Date: Mon, 25 Nov 2019 19:49:11 +0000
Subject: [Zeek] Two instances of bro  on the same host?
Message-ID: <OFD37C3963.6C0E292F-ON002584BD.006B8235-002584BD.006CDFBD@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191125/f180a1ad/attachment.html 

From jlay at slave-tothe-box.net  Mon Nov 25 12:55:24 2019
From: jlay at slave-tothe-box.net (James Lay)
Date: Mon, 25 Nov 2019 13:55:24 -0700
Subject: [Zeek] Two instances of bro  on the same host?
In-Reply-To: <OFD37C3963.6C0E292F-ON002584BD.006B8235-002584BD.006CDFBD@notes.na.collabserv.com>
References: <OFD37C3963.6C0E292F-ON002584BD.006B8235-002584BD.006CDFBD@notes.na.collabserv.com>
Message-ID: <1441ea822f91fff36967a4c4c3f46da6@slave-tothe-box.net>

On 2019-11-25 12:49, Erich M Nahum wrote:
> Howdy,
> 
> I have a multi-core machine listening to 8 interfaces with zeek.  I'm
> using the kafka plugin to send logs to individual topics (conn, dns,
> http, etc).
> 
> I've recently gotten a tap outside the firewall and want to send the
> equivalent logs to different comparable topics (conn-firewall,
> dns-firewall, etc).
> 
> I'm currently using zeekctl with multiple workers.  What I'm wondering
> is can I use two instances of zeekctl on the same machine, one for
> inside the FW and one for outside.
> 
> It's not an option right now to do the outside the FW on a separate
> machine.
> 
> Any suggestions?
> 
> Thanks,
> 
> -Erich
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


I use one bro instance to listen to both internal and external.  Set 
your filters in your SIEM to internal vs. external networks and you 
should be fine.

James

From andrew at aklaus.ca  Mon Nov 25 14:18:11 2019
From: andrew at aklaus.ca (Andrew Klaus)
Date: Mon, 25 Nov 2019 15:18:11 -0700
Subject: [Zeek] Making Broker Optional in Script
In-Reply-To: <CAGENE8tCKOsrGc2d_sm1ZDedoPbnfWYNA6Mwxqd1sPo-G49=kQ@mail.gmail.com>
References: <CAGENE8tCKOsrGc2d_sm1ZDedoPbnfWYNA6Mwxqd1sPo-G49=kQ@mail.gmail.com>
Message-ID: <CAGENE8tMx_tUu=LgvvuDbt1N=wEsNL=RvcaqJG5CkYw7jzJEFA@mail.gmail.com>

This is the exact script I'm trying to load Broker with at run-time:
https://raw.githubusercontent.com/cybera/zeek-sniffpass/broker/scripts/main.bro


When I run the default script without Broker, everything runs fine since
the default is disabled (const broker_enable = F &redef;).

However, when I try to override this on Zeek 3.0.0 with Broker 1.2.3 in my
local.zeek file:
----
# local.zeek
redef SNIFFPASS::broker_enable = T;
----

The `@if (SNIFFPASS::broker_enable)`  doesn't pass after I deploy+restart
Zeek with the default overridden. It does work when I hardcode it to True
in the script however:
---
# main.zeek

export {
...
const broker_enable = T &redef;
}
...
----

So I know my logic is okay and it's just an issue trying to override
broker_enable.

Thanks!
Andrew

On Mon, Nov 25, 2019 at 11:45 AM Andrew Klaus <andrew at aklaus.ca> wrote:

> Hello,
>
> I have a Zeek script that I would like to add optional Broker
> functionality to. I don't want it to be a requirement, so I'm adding
> run-time redef options that can be toggled to enable/disable it.
>
> Something like:
> -------
> export {
>     const broker_enable = F &redef;
> }
>
> @if (MODULE::broker_enable)
> event bro_init()
> {
>   Broker::listen("127.0.0.1", 9999/tcp);
> }
> @endif
> -------
>
> When I attempt to add this line to my local.bro/zeek file after the @load
> package:
> ---
> redef MODULE::broker_enable = T;
> ---
>
> It won't override broker_enable and thus doesn't listen on the socket. I
> assume that it's because the @load happens before the redef in local.bro,
> and thus doesn't override when it checks for the @if ?
>
> Is there a better way that I can do this? Like checking if Broker is
> actually available?  I tried this, but it doesn't work either:
> ---
> @ifdef (Broker)
> print "Broker Enabled";
> @endif
> ---
>
> Thanks in advance,
>
> Andrew
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191125/7df95601/attachment.html 

From justin at corelight.com  Mon Nov 25 16:09:51 2019
From: justin at corelight.com (Justin Azoff)
Date: Mon, 25 Nov 2019 19:09:51 -0500
Subject: [Zeek] Making Broker Optional in Script
In-Reply-To: <CAGENE8tMx_tUu=LgvvuDbt1N=wEsNL=RvcaqJG5CkYw7jzJEFA@mail.gmail.com>
References: <CAGENE8tCKOsrGc2d_sm1ZDedoPbnfWYNA6Mwxqd1sPo-G49=kQ@mail.gmail.com>
	<CAGENE8tMx_tUu=LgvvuDbt1N=wEsNL=RvcaqJG5CkYw7jzJEFA@mail.gmail.com>
Message-ID: <CAPfnCujO6xry+92rzx=VjvFk7juX05P_oB6PB=vQvvvZFMxV3Q@mail.gmail.com>

@if is only evaluated once when scripts are parsed.  If you are changing
broker_enable you have to do it before your script is loaded.

Also, instead of

@if( /^3\./ in bro_version() )

you can just do

@ifdef(zeek_init)



On Mon, Nov 25, 2019 at 5:21 PM Andrew Klaus <andrew at aklaus.ca> wrote:

> This is the exact script I'm trying to load Broker with at run-time:
> https://raw.githubusercontent.com/cybera/zeek-sniffpass/broker/scripts/main.bro
>
>
> When I run the default script without Broker, everything runs fine since
> the default is disabled (const broker_enable = F &redef;).
>
> However, when I try to override this on Zeek 3.0.0 with Broker 1.2.3 in my
> local.zeek file:
> ----
> # local.zeek
> redef SNIFFPASS::broker_enable = T;
> ----
>
> The `@if (SNIFFPASS::broker_enable)`  doesn't pass after I deploy+restart
> Zeek with the default overridden. It does work when I hardcode it to True
> in the script however:
> ---
> # main.zeek
>
> export {
> ...
> const broker_enable = T &redef;
> }
> ...
> ----
>
> So I know my logic is okay and it's just an issue trying to override
> broker_enable.
>
> Thanks!
> Andrew
>
> On Mon, Nov 25, 2019 at 11:45 AM Andrew Klaus <andrew at aklaus.ca> wrote:
>
>> Hello,
>>
>> I have a Zeek script that I would like to add optional Broker
>> functionality to. I don't want it to be a requirement, so I'm adding
>> run-time redef options that can be toggled to enable/disable it.
>>
>> Something like:
>> -------
>> export {
>>     const broker_enable = F &redef;
>> }
>>
>> @if (MODULE::broker_enable)
>> event bro_init()
>> {
>>   Broker::listen("127.0.0.1", 9999/tcp);
>> }
>> @endif
>> -------
>>
>> When I attempt to add this line to my local.bro/zeek file after the @load
>> package:
>> ---
>> redef MODULE::broker_enable = T;
>> ---
>>
>> It won't override broker_enable and thus doesn't listen on the socket. I
>> assume that it's because the @load happens before the redef in local.bro,
>> and thus doesn't override when it checks for the @if ?
>>
>> Is there a better way that I can do this? Like checking if Broker is
>> actually available?  I tried this, but it doesn't work either:
>> ---
>> @ifdef (Broker)
>> print "Broker Enabled";
>> @endif
>> ---
>>
>> Thanks in advance,
>>
>> Andrew
>>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191125/f42b8505/attachment.html 

From gantapritham4 at gmail.com  Mon Nov 25 20:37:22 2019
From: gantapritham4 at gmail.com (Priyatham Ganta)
Date: Mon, 25 Nov 2019 20:37:22 -0800
Subject: [Zeek] Ryu Controller
Message-ID: <CABXPuZ8j4MvC+1JC4CGRtbwne=SHMZFx70bpD=+tgUmHU00pxA@mail.gmail.com>

Hi People,

I want to integrate Ryu controller with Zeek IDS for a project and I need
help to do this. Can anyone help me with it?

Thanks
Priyatham
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191125/916dc4c0/attachment.html 

From johanna at corelight.com  Mon Nov 25 21:45:54 2019
From: johanna at corelight.com (Johanna Amann)
Date: Mon, 25 Nov 2019 21:45:54 -0800
Subject: [Zeek] Ryu Controller
In-Reply-To: <CABXPuZ8j4MvC+1JC4CGRtbwne=SHMZFx70bpD=+tgUmHU00pxA@mail.gmail.com>
References: <CABXPuZ8j4MvC+1JC4CGRtbwne=SHMZFx70bpD=+tgUmHU00pxA@mail.gmail.com>
Message-ID: <4DB260B1-4D40-4AF4-B9D6-720C8A02FFF2@corelight.com>

Hi,

> I want to integrate Ryu controller with Zeek IDS for a project and I 
> need
> help to do this. Can anyone help me with it?

if you just want send commands to Ryu from Zeek - use the netcontrol 
framework. There actually is a Ryu plugin for it, although that might 
have bitrotted a bit by now (so I won?t guarantee that it just works 
out of the box anymore).

In any case - it might be worth taking a look at the netcontrol 
documentation that highlights how netcontrol operates: 
https://docs.zeek.org/en/stable/frameworks/netcontrol.html

It also shows how to instantiate everything. To make things a bit 
complicated, there are two ways to interface with Ryu. The first one 
uses the Ryu REST API directly from Zeek. This does not scale very well 
- but is pretty simple and should still work unless they changed the 
API. That plugin ships with Zeek and is at 
https://github.com/zeek/zeek/blob/master/scripts/base/frameworks/openflow/plugins/ryu.zeek.

The second way is to use the generic broker plugin on the Zeek side - 
and write a Ryu controller that can interact with that. A Ryu controller 
implementing this is in the zeek-netcontrol repository (which is 
contained in aux if you download the distribution). 
https://github.com/zeek/zeek-netcontrol/tree/master/openflow contains 
the source code as well as an example script that ties everything 
together.

I hope this helps a bit to get started :)
  Johanna


From mostafaammar at aast.edu  Tue Nov 26 00:36:26 2019
From: mostafaammar at aast.edu (Dr. Mostafa Abdallah. Ammar)
Date: Tue, 26 Nov 2019 08:36:26 +0000
Subject: [Zeek] Ryu Controller
In-Reply-To: <4DB260B1-4D40-4AF4-B9D6-720C8A02FFF2@corelight.com>
References: <CABXPuZ8j4MvC+1JC4CGRtbwne=SHMZFx70bpD=+tgUmHU00pxA@mail.gmail.com>,
	<4DB260B1-4D40-4AF4-B9D6-720C8A02FFF2@corelight.com>
Message-ID: <279a3bdf07e141e4b6b18457ef797450@KeerMBX03.Egypt.AAST.edu>

Hi,

I made a similar research on how to integrate BRO and snort IDS with SDN controller

Best Regards,

Mostafa Abdallah Ammar, PhD.
Head of Information Security Department
CCIE security #23971
Arab Academy For Science And Technology & maritime Transport
Computer Networks & Data Center (CNDC)
Mobile: 002 01001983674

________________________________________
From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> on behalf of Johanna Amann <johanna at corelight.com>
Sent: Tuesday, November 26, 2019 7:45 AM
To: Priyatham Ganta
Cc: zeek at zeek.org
Subject: Re: [Zeek] Ryu Controller

Hi,

> I want to integrate Ryu controller with Zeek IDS for a project and I
> need
> help to do this. Can anyone help me with it?

if you just want send commands to Ryu from Zeek - use the netcontrol
framework. There actually is a Ryu plugin for it, although that might
have bitrotted a bit by now (so I won?t guarantee that it just works
out of the box anymore).

In any case - it might be worth taking a look at the netcontrol
documentation that highlights how netcontrol operates:
https://docs.zeek.org/en/stable/frameworks/netcontrol.html

It also shows how to instantiate everything. To make things a bit
complicated, there are two ways to interface with Ryu. The first one
uses the Ryu REST API directly from Zeek. This does not scale very well
- but is pretty simple and should still work unless they changed the
API. That plugin ships with Zeek and is at
https://github.com/zeek/zeek/blob/master/scripts/base/frameworks/openflow/plugins/ryu.zeek.

The second way is to use the generic broker plugin on the Zeek side -
and write a Ryu controller that can interact with that. A Ryu controller
implementing this is in the zeek-netcontrol repository (which is
contained in aux if you download the distribution).
https://github.com/zeek/zeek-netcontrol/tree/master/openflow contains
the source code as well as an example script that ties everything
together.

I hope this helps a bit to get started :)
  Johanna

_______________________________________________
Zeek mailing list
zeek at zeek.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From mostafaammar at aast.edu  Tue Nov 26 00:44:39 2019
From: mostafaammar at aast.edu (Dr. Mostafa Abdallah. Ammar)
Date: Tue, 26 Nov 2019 08:44:39 +0000
Subject: [Zeek] Ryu Controller
In-Reply-To: <279a3bdf07e141e4b6b18457ef797450@KeerMBX03.Egypt.AAST.edu>
References: <CABXPuZ8j4MvC+1JC4CGRtbwne=SHMZFx70bpD=+tgUmHU00pxA@mail.gmail.com>,
	<4DB260B1-4D40-4AF4-B9D6-720C8A02FFF2@corelight.com>,
	<279a3bdf07e141e4b6b18457ef797450@KeerMBX03.Egypt.AAST.edu>
Message-ID: <1630b8f1e0a948b8bfff40932ab6cb1e@KeerMBX03.Egypt.AAST.edu>

Hi,

we made a similar research on how to integrate BRO and snort IDS with SDN controller
https://ieeexplore.ieee.org/document/7792427

we used floodlight SDN controller with IDS

In this case we created port mirror on switch to send traffic to IDS for matching attacks and if there is a detected attacker we contact the floodlight controller through rest api to get the host switch port and send a blocking flow to switch to block attacker.

Another integration if we want to search for a specific traffic we forward traffic to controller and create a module on controller , this module matches the traffic against defined database and sends a blocking flow also if match is found.

I dont know if this provides help in your case.

Best Regards,

Mostafa Abdallah Ammar, PhD.
Head of Information Security Department
CCIE security #23971
Arab Academy For Science And Technology & maritime Transport
Computer Networks & Data Center (CNDC)
Mobile: 002 01001983674

________________________________________
From: Dr. Mostafa Abdallah. Ammar
Sent: Tuesday, November 26, 2019 10:36 AM
To: Johanna Amann; Priyatham Ganta
Cc: zeek at zeek.org
Subject: RE: [Zeek] Ryu Controller

Hi,

I made a similar research on how to integrate BRO and snort IDS with SDN controller

Best Regards,

Mostafa Abdallah Ammar, PhD.
Head of Information Security Department
CCIE security #23971
Arab Academy For Science And Technology & maritime Transport
Computer Networks & Data Center (CNDC)
Mobile: 002 01001983674

________________________________________
From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> on behalf of Johanna Amann <johanna at corelight.com>
Sent: Tuesday, November 26, 2019 7:45 AM
To: Priyatham Ganta
Cc: zeek at zeek.org
Subject: Re: [Zeek] Ryu Controller

Hi,

> I want to integrate Ryu controller with Zeek IDS for a project and I
> need
> help to do this. Can anyone help me with it?

if you just want send commands to Ryu from Zeek - use the netcontrol
framework. There actually is a Ryu plugin for it, although that might
have bitrotted a bit by now (so I won?t guarantee that it just works
out of the box anymore).

In any case - it might be worth taking a look at the netcontrol
documentation that highlights how netcontrol operates:
https://docs.zeek.org/en/stable/frameworks/netcontrol.html

It also shows how to instantiate everything. To make things a bit
complicated, there are two ways to interface with Ryu. The first one
uses the Ryu REST API directly from Zeek. This does not scale very well
- but is pretty simple and should still work unless they changed the
API. That plugin ships with Zeek and is at
https://github.com/zeek/zeek/blob/master/scripts/base/frameworks/openflow/plugins/ryu.zeek.

The second way is to use the generic broker plugin on the Zeek side -
and write a Ryu controller that can interact with that. A Ryu controller
implementing this is in the zeek-netcontrol repository (which is
contained in aux if you download the distribution).
https://github.com/zeek/zeek-netcontrol/tree/master/openflow contains
the source code as well as an example script that ties everything
together.

I hope this helps a bit to get started :)
  Johanna

_______________________________________________
Zeek mailing list
zeek at zeek.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From austin522 at gmail.com  Tue Nov 26 03:24:18 2019
From: austin522 at gmail.com (venkatesh bandari)
Date: Tue, 26 Nov 2019 19:24:18 +0800
Subject: [Zeek] (no subject)
Message-ID: <CAKPzng-AN3phZ0g6NRtJzq17DwZgXYBHoikoUHSmGVEcS9Cy+A@mail.gmail.com>

Hello team,

How to measure the packet loss on open source zeek

I see capstats which has nic drops .is this using if config rx_dropped
counter or different

Thanks
Venkatesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191126/0e15604a/attachment.html 

From austin522 at gmail.com  Tue Nov 26 03:33:50 2019
From: austin522 at gmail.com (venkatesh bandari)
Date: Tue, 26 Nov 2019 19:33:50 +0800
Subject: [Zeek] Zeek packet drops
Message-ID: <CAKPzng-tv9_r+GyosvD8L7z53QODiiCBj_tCyTR87VK7nCgJxA@mail.gmail.com>

Hello team,

How to measure the packet loss on open source zeek

I see capstats which has nic drops .is this using if config rx_dropped
counter or different

Thanks
Venkatesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191126/a3ac143b/attachment.html 

From vlad at es.net  Tue Nov 26 06:41:20 2019
From: vlad at es.net (Vlad Grigorescu)
Date: Tue, 26 Nov 2019 14:41:20 +0000
Subject: [Zeek] Zeek packet drops
In-Reply-To: <CAKPzng-tv9_r+GyosvD8L7z53QODiiCBj_tCyTR87VK7nCgJxA@mail.gmail.com>
References: <CAKPzng-tv9_r+GyosvD8L7z53QODiiCBj_tCyTR87VK7nCgJxA@mail.gmail.com>
Message-ID: <CAPqbkwuhZiAKmfVxfej2PmR5kmAnCGSr_E6A5GaLknMqZ9YApA@mail.gmail.com>

Try capture-loss:
https://docs.zeek.org/en/stable/scripts/policy/misc/capture-loss.zeek.html

On Tue, Nov 26, 2019 at 11:35 AM venkatesh bandari <austin522 at gmail.com>
wrote:

> Hello team,
>
> How to measure the packet loss on open source zeek
>
> I see capstats which has nic drops .is this using if config rx_dropped
> counter or different
>
> Thanks
> Venkatesh
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191126/f1f5a8ec/attachment.html 

From gantapritham4 at gmail.com  Tue Nov 26 09:24:16 2019
From: gantapritham4 at gmail.com (Priyatham Ganta)
Date: Tue, 26 Nov 2019 09:24:16 -0800
Subject: [Zeek] Ryu Controller
In-Reply-To: <1630b8f1e0a948b8bfff40932ab6cb1e@KeerMBX03.Egypt.AAST.edu>
References: <CABXPuZ8j4MvC+1JC4CGRtbwne=SHMZFx70bpD=+tgUmHU00pxA@mail.gmail.com>
	<4DB260B1-4D40-4AF4-B9D6-720C8A02FFF2@corelight.com>
	<279a3bdf07e141e4b6b18457ef797450@KeerMBX03.Egypt.AAST.edu>
	<1630b8f1e0a948b8bfff40932ab6cb1e@KeerMBX03.Egypt.AAST.edu>
Message-ID: <CABXPuZ8+aFGuBcdWvUxvGPWO8Mw1bQf7Tkw5HnrLayUCr4SYeQ@mail.gmail.com>

Hi,

How can I run bro for the current traffic and show the alerts on a console
instead of logs?

Also where can I check the policies that are configured to Bro for IDS?

Also what is the difference between the broctl binary and bro binary?

Thanks
Priyatham

On Tue, Nov 26, 2019 at 12:47 AM Dr. Mostafa Abdallah. Ammar <
mostafaammar at aast.edu> wrote:

> Hi,
>
> we made a similar research on how to integrate BRO and snort IDS with SDN
> controller
> https://ieeexplore.ieee.org/document/7792427
>
> we used floodlight SDN controller with IDS
>
> In this case we created port mirror on switch to send traffic to IDS for
> matching attacks and if there is a detected attacker we contact the
> floodlight controller through rest api to get the host switch port and send
> a blocking flow to switch to block attacker.
>
> Another integration if we want to search for a specific traffic we forward
> traffic to controller and create a module on controller , this module
> matches the traffic against defined database and sends a blocking flow also
> if match is found.
>
> I dont know if this provides help in your case.
>
> Best Regards,
>
> Mostafa Abdallah Ammar, PhD.
> Head of Information Security Department
> CCIE security #23971
> Arab Academy For Science And Technology & maritime Transport
> Computer Networks & Data Center (CNDC)
> Mobile: 002 01001983674
>
> ________________________________________
> From: Dr. Mostafa Abdallah. Ammar
> Sent: Tuesday, November 26, 2019 10:36 AM
> To: Johanna Amann; Priyatham Ganta
> Cc: zeek at zeek.org
> Subject: RE: [Zeek] Ryu Controller
>
> Hi,
>
> I made a similar research on how to integrate BRO and snort IDS with SDN
> controller
>
> Best Regards,
>
> Mostafa Abdallah Ammar, PhD.
> Head of Information Security Department
> CCIE security #23971
> Arab Academy For Science And Technology & maritime Transport
> Computer Networks & Data Center (CNDC)
> Mobile: 002 01001983674
>
> ________________________________________
> From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> on behalf of Johanna
> Amann <johanna at corelight.com>
> Sent: Tuesday, November 26, 2019 7:45 AM
> To: Priyatham Ganta
> Cc: zeek at zeek.org
> Subject: Re: [Zeek] Ryu Controller
>
> Hi,
>
> > I want to integrate Ryu controller with Zeek IDS for a project and I
> > need
> > help to do this. Can anyone help me with it?
>
> if you just want send commands to Ryu from Zeek - use the netcontrol
> framework. There actually is a Ryu plugin for it, although that might
> have bitrotted a bit by now (so I won?t guarantee that it just works
> out of the box anymore).
>
> In any case - it might be worth taking a look at the netcontrol
> documentation that highlights how netcontrol operates:
> https://docs.zeek.org/en/stable/frameworks/netcontrol.html
>
> It also shows how to instantiate everything. To make things a bit
> complicated, there are two ways to interface with Ryu. The first one
> uses the Ryu REST API directly from Zeek. This does not scale very well
> - but is pretty simple and should still work unless they changed the
> API. That plugin ships with Zeek and is at
>
> https://github.com/zeek/zeek/blob/master/scripts/base/frameworks/openflow/plugins/ryu.zeek
> .
>
> The second way is to use the generic broker plugin on the Zeek side -
> and write a Ryu controller that can interact with that. A Ryu controller
> implementing this is in the zeek-netcontrol repository (which is
> contained in aux if you download the distribution).
> https://github.com/zeek/zeek-netcontrol/tree/master/openflow contains
> the source code as well as an example script that ties everything
> together.
>
> I hope this helps a bit to get started :)
>   Johanna
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191126/679d17f0/attachment-0001.html 

From johanna at icir.org  Tue Nov 26 10:54:38 2019
From: johanna at icir.org (Johanna Amann)
Date: Tue, 26 Nov 2019 10:54:38 -0800
Subject: [Zeek] Ryu Controller
In-Reply-To: <CABXPuZ8+aFGuBcdWvUxvGPWO8Mw1bQf7Tkw5HnrLayUCr4SYeQ@mail.gmail.com>
References: <CABXPuZ8j4MvC+1JC4CGRtbwne=SHMZFx70bpD=+tgUmHU00pxA@mail.gmail.com>
	<4DB260B1-4D40-4AF4-B9D6-720C8A02FFF2@corelight.com>
	<279a3bdf07e141e4b6b18457ef797450@KeerMBX03.Egypt.AAST.edu>
	<1630b8f1e0a948b8bfff40932ab6cb1e@KeerMBX03.Egypt.AAST.edu>
	<CABXPuZ8+aFGuBcdWvUxvGPWO8Mw1bQf7Tkw5HnrLayUCr4SYeQ@mail.gmail.com>
Message-ID: <C612A80A-E744-48D1-870C-65B6702DD388@icir.org>

Hi,

> How can I run bro for the current traffic and show the alerts on a 
> console
> instead of logs?

you can run it on the command line without using zeekctl/broctl using 
zeek (or bro) -i [interfacename]. However, logs will always written to 
files - it does not really make sense to write them to the console, 
which would make it hard to distinguish between the different log 
streams.

Note - most Zeek logs are policy neutral and not really alerts?

> Also where can I check the policies that are configured to Bro for 
> IDS?

I don?t 100% get the questions. If you load misc/loaded-scripts in 
your configuration, you will get a loaded-scripts.log which will show 
you all script files that are loaded. The default configuration of Zeek 
loads most protocol analyzers and writes their log-files.

> Also what is the difference between the broctl binary and bro binary?

zeekctl/broctl is the management application to start zeek cluster 
setups. See e.g. https://github.com/zeek/zeekctl - or 
https://docs.zeek.org/en/stable/quickstart/ for a getting started guide 
that mentions this.

Johanna

From nahum at us.ibm.com  Tue Nov 26 12:02:16 2019
From: nahum at us.ibm.com (Erich M Nahum)
Date: Tue, 26 Nov 2019 20:02:16 +0000
Subject: [Zeek] Two instances of bro  on the same host?
In-Reply-To: <OFD37C3963.6C0E292F-ON002584BD.006B8235-002584BD.006CDFBD@LocalDomain>
References: <OFD37C3963.6C0E292F-ON002584BD.006B8235-002584BD.006CDFBD@LocalDomain>
Message-ID: <OF7F385FBE.9901AE15-ON002584BE.006DB3D7-002584BE.006E126C@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191126/b0825245/attachment.html 

From gantapritham4 at gmail.com  Tue Nov 26 13:08:31 2019
From: gantapritham4 at gmail.com (Priyatham Ganta)
Date: Tue, 26 Nov 2019 13:08:31 -0800
Subject: [Zeek] Ryu Controller
In-Reply-To: <C612A80A-E744-48D1-870C-65B6702DD388@icir.org>
References: <CABXPuZ8j4MvC+1JC4CGRtbwne=SHMZFx70bpD=+tgUmHU00pxA@mail.gmail.com>
	<4DB260B1-4D40-4AF4-B9D6-720C8A02FFF2@corelight.com>
	<279a3bdf07e141e4b6b18457ef797450@KeerMBX03.Egypt.AAST.edu>
	<1630b8f1e0a948b8bfff40932ab6cb1e@KeerMBX03.Egypt.AAST.edu>
	<CABXPuZ8+aFGuBcdWvUxvGPWO8Mw1bQf7Tkw5HnrLayUCr4SYeQ@mail.gmail.com>
	<C612A80A-E744-48D1-870C-65B6702DD388@icir.org>
Message-ID: <CABXPuZ93G5x_wvWGAi-Rvi6YY8XGfbyJ4APBr2zMG_DQYFynxA@mail.gmail.com>

Hi,

I'm trying to run Bro as IDS. Hence, I don't want to show all the logs on
the console.I just want to look at the alerts generated by Bro if there are
any attacks on the network. That's the reason I want to print only the
alerts and not logs.
How do I run Bro in IDS mode?

For Bro to run as IDS, there should be some policies configured with which
this application will differentiate between normal traffic and malicious
traffic. I want to look at those policies.

Can you help me with this?

Thanks

On Tue, 26 Nov 2019 at 10:54, Johanna Amann <johanna at icir.org> wrote:

> Hi,
>
> > How can I run bro for the current traffic and show the alerts on a
> > console
> > instead of logs?
>
> you can run it on the command line without using zeekctl/broctl using
> zeek (or bro) -i [interfacename]. However, logs will always written to
> files - it does not really make sense to write them to the console,
> which would make it hard to distinguish between the different log
> streams.
>
> Note - most Zeek logs are policy neutral and not really alerts?
>
> > Also where can I check the policies that are configured to Bro for
> > IDS?
>
> I don?t 100% get the questions. If you load misc/loaded-scripts in
> your configuration, you will get a loaded-scripts.log which will show
> you all script files that are loaded. The default configuration of Zeek
> loads most protocol analyzers and writes their log-files.
>
> > Also what is the difference between the broctl binary and bro binary?
>
> zeekctl/broctl is the management application to start zeek cluster
> setups. See e.g. https://github.com/zeek/zeekctl - or
> https://docs.zeek.org/en/stable/quickstart/ for a getting started guide
> that mentions this.
>
> Johanna
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191126/182d9b3d/attachment.html 

From richard at corelight.com  Tue Nov 26 14:20:23 2019
From: richard at corelight.com (Richard Bejtlich)
Date: Tue, 26 Nov 2019 17:20:23 -0500
Subject: [Zeek] Ryu Controller
In-Reply-To: <CABXPuZ93G5x_wvWGAi-Rvi6YY8XGfbyJ4APBr2zMG_DQYFynxA@mail.gmail.com>
References: <CABXPuZ8j4MvC+1JC4CGRtbwne=SHMZFx70bpD=+tgUmHU00pxA@mail.gmail.com>
	<4DB260B1-4D40-4AF4-B9D6-720C8A02FFF2@corelight.com>
	<279a3bdf07e141e4b6b18457ef797450@KeerMBX03.Egypt.AAST.edu>
	<1630b8f1e0a948b8bfff40932ab6cb1e@KeerMBX03.Egypt.AAST.edu>
	<CABXPuZ8+aFGuBcdWvUxvGPWO8Mw1bQf7Tkw5HnrLayUCr4SYeQ@mail.gmail.com>
	<C612A80A-E744-48D1-870C-65B6702DD388@icir.org>
	<CABXPuZ93G5x_wvWGAi-Rvi6YY8XGfbyJ4APBr2zMG_DQYFynxA@mail.gmail.com>
Message-ID: <CAOtSMjb89d-FwFijNvbOuG8pn35e_n7+ySoAGkq=u12zUczAnw@mail.gmail.com>

Why are you interested in this approach? Is it a school project?

Zeek isn?t designed to be an intrusion detection system that creates
alerts, although it does produce notices. You might be better off with
Suricata if you want alerts.

Sincerely,

Richard

On Tue, Nov 26, 2019 at 4:17 PM Priyatham Ganta <gantapritham4 at gmail.com>
wrote:

> Hi,
>
> I'm trying to run Bro as IDS. Hence, I don't want to show all the logs on
> the console.I just want to look at the alerts generated by Bro if there are
> any attacks on the network. That's the reason I want to print only the
> alerts and not logs.
> How do I run Bro in IDS mode?
>
> For Bro to run as IDS, there should be some policies configured with which
> this application will differentiate between normal traffic and malicious
> traffic. I want to look at those policies.
>
> Can you help me with this?
>
> Thanks
>
> On Tue, 26 Nov 2019 at 10:54, Johanna Amann <johanna at icir.org> wrote:
>
>> Hi,
>>
>> > How can I run bro for the current traffic and show the alerts on a
>> > console
>> > instead of logs?
>>
>> you can run it on the command line without using zeekctl/broctl using
>> zeek (or bro) -i [interfacename]. However, logs will always written to
>> files - it does not really make sense to write them to the console,
>> which would make it hard to distinguish between the different log
>> streams.
>>
>> Note - most Zeek logs are policy neutral and not really alerts?
>>
>> > Also where can I check the policies that are configured to Bro for
>> > IDS?
>>
>> I don?t 100% get the questions. If you load misc/loaded-scripts in
>> your configuration, you will get a loaded-scripts.log which will show
>> you all script files that are loaded. The default configuration of Zeek
>> loads most protocol analyzers and writes their log-files.
>>
>> > Also what is the difference between the broctl binary and bro binary?
>>
>> zeekctl/broctl is the management application to start zeek cluster
>> setups. See e.g. https://github.com/zeek/zeekctl - or
>> https://docs.zeek.org/en/stable/quickstart/ for a getting started guide
>> that mentions this.
>>
>> Johanna
>>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

-- 
Richard Bejtlich
Principal Security Strategist, Corelight
https://corelight.blog/author/richardbejtlich/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191126/28e30cf1/attachment.html 

From gantapritham4 at gmail.com  Tue Nov 26 15:59:47 2019
From: gantapritham4 at gmail.com (Priyatham Ganta)
Date: Tue, 26 Nov 2019 15:59:47 -0800
Subject: [Zeek] Ryu Controller
In-Reply-To: <CAOtSMjb89d-FwFijNvbOuG8pn35e_n7+ySoAGkq=u12zUczAnw@mail.gmail.com>
References: <CABXPuZ8j4MvC+1JC4CGRtbwne=SHMZFx70bpD=+tgUmHU00pxA@mail.gmail.com>
	<4DB260B1-4D40-4AF4-B9D6-720C8A02FFF2@corelight.com>
	<279a3bdf07e141e4b6b18457ef797450@KeerMBX03.Egypt.AAST.edu>
	<1630b8f1e0a948b8bfff40932ab6cb1e@KeerMBX03.Egypt.AAST.edu>
	<CABXPuZ8+aFGuBcdWvUxvGPWO8Mw1bQf7Tkw5HnrLayUCr4SYeQ@mail.gmail.com>
	<C612A80A-E744-48D1-870C-65B6702DD388@icir.org>
	<CABXPuZ93G5x_wvWGAi-Rvi6YY8XGfbyJ4APBr2zMG_DQYFynxA@mail.gmail.com>
	<CAOtSMjb89d-FwFijNvbOuG8pn35e_n7+ySoAGkq=u12zUczAnw@mail.gmail.com>
Message-ID: <CABXPuZ_ieFcu=YOzoNcQGsHMeez8kZrCu7J50rvHCssjQWmd5Q@mail.gmail.com>

Hi,

Yes, it is for a school project and would like to use Bro as IDS. And I
would like Bro to generate active alerts for the incoming traffic. How can
I do it?

Thanks

On Tue, 26 Nov 2019 at 14:20, Richard Bejtlich <richard at corelight.com>
wrote:

> Why are you interested in this approach? Is it a school project?
>
> Zeek isn?t designed to be an intrusion detection system that creates
> alerts, although it does produce notices. You might be better off with
> Suricata if you want alerts.
>
> Sincerely,
>
> Richard
>
> On Tue, Nov 26, 2019 at 4:17 PM Priyatham Ganta <gantapritham4 at gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm trying to run Bro as IDS. Hence, I don't want to show all the logs on
>> the console.I just want to look at the alerts generated by Bro if there are
>> any attacks on the network. That's the reason I want to print only the
>> alerts and not logs.
>> How do I run Bro in IDS mode?
>>
>> For Bro to run as IDS, there should be some policies configured with
>> which this application will differentiate between normal traffic and
>> malicious traffic. I want to look at those policies.
>>
>> Can you help me with this?
>>
>> Thanks
>>
>> On Tue, 26 Nov 2019 at 10:54, Johanna Amann <johanna at icir.org> wrote:
>>
>>> Hi,
>>>
>>> > How can I run bro for the current traffic and show the alerts on a
>>> > console
>>> > instead of logs?
>>>
>>> you can run it on the command line without using zeekctl/broctl using
>>> zeek (or bro) -i [interfacename]. However, logs will always written to
>>> files - it does not really make sense to write them to the console,
>>> which would make it hard to distinguish between the different log
>>> streams.
>>>
>>> Note - most Zeek logs are policy neutral and not really alerts?
>>>
>>> > Also where can I check the policies that are configured to Bro for
>>> > IDS?
>>>
>>> I don?t 100% get the questions. If you load misc/loaded-scripts in
>>> your configuration, you will get a loaded-scripts.log which will show
>>> you all script files that are loaded. The default configuration of Zeek
>>> loads most protocol analyzers and writes their log-files.
>>>
>>> > Also what is the difference between the broctl binary and bro binary?
>>>
>>> zeekctl/broctl is the management application to start zeek cluster
>>> setups. See e.g. https://github.com/zeek/zeekctl - or
>>> https://docs.zeek.org/en/stable/quickstart/ for a getting started guide
>>> that mentions this.
>>>
>>> Johanna
>>>
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
> --
> Richard Bejtlich
> Principal Security Strategist, Corelight
> https://corelight.blog/author/richardbejtlich/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191126/7278cab2/attachment-0001.html 

From kilotao at gmail.com  Tue Nov 26 16:11:31 2019
From: kilotao at gmail.com (kilotao at gmail.com)
Date: Tue, 26 Nov 2019 19:11:31 -0500
Subject: [Zeek] Two instances of bro on the same host?
In-Reply-To: <OF7F385FBE.9901AE15-ON002584BE.006DB3D7-002584BE.006E126C@notes.na.collabserv.com>
References: <OFD37C3963.6C0E292F-ON002584BD.006B8235-002584BD.006CDFBD@LocalDomain>
	<OF7F385FBE.9901AE15-ON002584BE.006DB3D7-002584BE.006E126C@notes.na.collabserv.com>
Message-ID: <CAC_rUhL3VL83kXkfCxgd2R2DNZP8UkcQkq1CZpF3dpMwQtkvaA@mail.gmail.com>

Dockorize bro, you can have as many as you want.

On Tue, Nov 26, 2019, 3:05 PM Erich M Nahum <nahum at us.ibm.com> wrote:

> I figured it out.  Two zeek installations (/opt/zeek1, /opt/zeek2), two
> node.cfg files, and most importantly, the second zeekctl.cfg with the port
> changed:
>
> ZeekPort = 48760
>
> instead of the default 47760 port.
>
> I don't know if this is supported, but it seems to work.
>
> -Erich
>
>
> ----- Original message -----
> From: Erich M Nahum/Watson/IBM
> To: zeek at zeek.org
> Cc:
> Subject: Two instances of bro on the same host?
> Date: Mon, Nov 25, 2019 2:49 PM
>
> Howdy,
>
> I have a multi-core machine listening to 8 interfaces with zeek.  I'm
> using the kafka plugin to send logs to individual topics (conn, dns, http,
> etc).
>
> I've recently gotten a tap outside the firewall and want to send the
> equivalent logs to different comparable topics (conn-firewall,
> dns-firewall, etc).
>
> I'm currently using zeekctl with multiple workers.  What I'm wondering is
> can I use two instances of zeekctl on the same machine, one for inside the
> FW and one for outside.
>
> It's not an option right now to do the outside the FW on a separate
> machine.
>
> Any suggestions?
>
> Thanks,
>
> -Erich
>
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191126/e0257883/attachment.html 

From fadelkam at gmail.com  Wed Nov 27 06:48:29 2019
From: fadelkam at gmail.com (Fadel)
Date: Wed, 27 Nov 2019 09:48:29 -0500
Subject: [Zeek] Cannot determine Bro source directory,
	use --bro-dist=DIR.
In-Reply-To: <CABFMW7VNSQd9LP1g4Cbc-_cbzWbiBa+ubrww1Gn+mEFkMjhJjw@mail.gmail.com>
References: <CABFMW7VNSQd9LP1g4Cbc-_cbzWbiBa+ubrww1Gn+mEFkMjhJjw@mail.gmail.com>
Message-ID: <CABFMW7VTjfv6cuWsXadzJWNtWK92YBT1TTyJHkTx=d8BOSvFEA@mail.gmail.com>

Hello zeek community,
Could anyone please help me solve this issue?
I'm not able to move forward...

Thanks very much!

On Tue, Nov 12, 2019, 4:05 PM Fadel <fadelkam at gmail.com> wrote:

> Hi, I've been trying to install this plugin and it seems to have
> some issues or confusion between bro and zeek for that plugin.
> Anyone encountered this issue?
> Thanks!
>
>
> ~]# zkg install apache/metron-bro-plugin-kafka
> The following packages will be INSTALLED:
>  zeek/apache/metron-bro-plugin-kafka (0.3)
>
> Verify the following REQUIRED external dependencies:
> (Ensure their installation on all relevant systems before proceeding):
>  from zeek/apache/metron-bro-plugin-kafka (0.3):
>    librdkafka ~0.11.5
>
> Proceed? [Y/n]
> zeek/apache/metron-bro-plugin-kafka asks for LIBRDKAFKA_ROOT (Path to
> librdkafka installation tree) ? [/usr/local/lib]
> Saved answers to config file: /root/.zkg/config
> Running unit tests for "zeek/apache/metron-bro-plugin-kafka"
> error: failed to run tests for zeek/apache/metron-bro-plugin-kafka:
> package build_command failed, see log in
> /root/.zkg/logs/metron-bro-plugin-kafka-build.log
> Proceed to install anyway? [N/y]
>
> ~]# cat /root/.zkg/logs/metron-bro-plugin-kafka-build.log
> === STDERR ===
> === STDOUT ===
> Cannot determine Bro source directory, use --bro-dist=DIR.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191127/647af7ae/attachment.html 

From jefflang at vt.edu  Wed Nov 27 07:04:54 2019
From: jefflang at vt.edu (Jeffry Lang)
Date: Wed, 27 Nov 2019 10:04:54 -0500
Subject: [Zeek] Cannot determine Bro source directory,
	use --bro-dist=DIR.
In-Reply-To: <CABFMW7VTjfv6cuWsXadzJWNtWK92YBT1TTyJHkTx=d8BOSvFEA@mail.gmail.com>
References: <CABFMW7VNSQd9LP1g4Cbc-_cbzWbiBa+ubrww1Gn+mEFkMjhJjw@mail.gmail.com>
	<CABFMW7VTjfv6cuWsXadzJWNtWK92YBT1TTyJHkTx=d8BOSvFEA@mail.gmail.com>
Message-ID: <CAEzr=8O0bb3-1aAq0AZZ82jWvMGDqnTa8Q-doUud_XuURFW_Sw@mail.gmail.com>

I've not attempted to install that plugin but looking at the github for it,
https://github.com/apache/metron-bro-plugin-kafka, it has this statement
"bro-pkg is the preferred mechanism for installing this plugin, as it will
dynamically retrieve, build, test, and load the plugin."  That leads me to
the zkg config file documentation at
https://docs.zeek.org/projects/package-manager/en/stable/zkg.html#zkg-config-file

# The directory containing Zeek distribution source code.  This is
only# needed when installing packages that contain Zeek plugins that
are# not pre-built.  The legacy name of this option is
"bro_dist".zeek_dist =

It doesn't sound like the plugin is prebuilt, so zkg is trying to do it.
Do you have the source code anywhere on the system?  Have you defined that
varialbe in the config file to point to that location?

Thank you,

Jeff Lang

---------------------------------------------------

Jeffry Lang
IT Security Operations (0284)
1300 Torgersen Hall, Virginia Tech
620 Drillfield Dr.
Blacksburg VA 24061
540-231-4117
jefflang at vt.edu


On Wed, Nov 27, 2019 at 9:49 AM Fadel <fadelkam at gmail.com> wrote:

> Hello zeek community,
> Could anyone please help me solve this issue?
> I'm not able to move forward...
>
> Thanks very much!
>
> On Tue, Nov 12, 2019, 4:05 PM Fadel <fadelkam at gmail.com> wrote:
>
>> Hi, I've been trying to install this plugin and it seems to have
>> some issues or confusion between bro and zeek for that plugin.
>> Anyone encountered this issue?
>> Thanks!
>>
>>
>> ~]# zkg install apache/metron-bro-plugin-kafka
>> The following packages will be INSTALLED:
>>  zeek/apache/metron-bro-plugin-kafka (0.3)
>>
>> Verify the following REQUIRED external dependencies:
>> (Ensure their installation on all relevant systems before proceeding):
>>  from zeek/apache/metron-bro-plugin-kafka (0.3):
>>    librdkafka ~0.11.5
>>
>> Proceed? [Y/n]
>> zeek/apache/metron-bro-plugin-kafka asks for LIBRDKAFKA_ROOT (Path to
>> librdkafka installation tree) ? [/usr/local/lib]
>> Saved answers to config file: /root/.zkg/config
>> Running unit tests for "zeek/apache/metron-bro-plugin-kafka"
>> error: failed to run tests for zeek/apache/metron-bro-plugin-kafka:
>> package build_command failed, see log in
>> /root/.zkg/logs/metron-bro-plugin-kafka-build.log
>> Proceed to install anyway? [N/y]
>>
>> ~]# cat /root/.zkg/logs/metron-bro-plugin-kafka-build.log
>> === STDERR ===
>> === STDOUT ===
>> Cannot determine Bro source directory, use --bro-dist=DIR.
>>
>>
>> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191127/6fae7744/attachment.html 

From mauro.palumbo at aizoon.it  Thu Nov 28 07:02:38 2019
From: mauro.palumbo at aizoon.it (Palumbo Mauro)
Date: Thu, 28 Nov 2019 15:02:38 +0000
Subject: [Zeek] tcp partial connections
Message-ID: <09a8e3bfc8a34d56ae8439e193d219c4@SRVEX03.aizoon.local>

Hi everybody,
   I have noticed that in many app level analyzers there is a sort of "safety check" in the method DeliverStream which essentially returns when the tcp session is partial, i.e.

        if ( TCP() && TCP()->IsPartial() )
                return;

This is true for example for the HTTP, SSH, SSL analyzers and more. My understanding is that this is to prevent app layer analyzers or scripts relying on them from breaking down or missing some information when processing packets with possible missing bytes.
Am I right? How much reliable is this check TCP()->IsPartial()  for partial tcp sessions in the tcp analyzer?

Thanks,
Mauro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191128/13e48532/attachment.html 

From jsiwek at corelight.com  Thu Nov 28 08:06:24 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 28 Nov 2019 08:06:24 -0800
Subject: [Zeek] tcp partial connections
In-Reply-To: <09a8e3bfc8a34d56ae8439e193d219c4@SRVEX03.aizoon.local>
References: <09a8e3bfc8a34d56ae8439e193d219c4@SRVEX03.aizoon.local>
Message-ID: <CAMzgZ0JjaVnKmcioHf=0ob3p+crYujHLOr0jgsHps6CtJp1VQg@mail.gmail.com>

On Thu, Nov 28, 2019 at 7:06 AM Palumbo Mauro <mauro.palumbo at aizoon.it> wrote:

>         if ( TCP() && TCP()->IsPartial() )
>                 return;
>
> This is true for example for the HTTP, SSH, SSL analyzers and more. My understanding is that this is to prevent app layer analyzers or scripts relying on them from breaking down or missing some information when processing packets with possible missing bytes.

Mre related to the "breaking down" part: current protocol parsers
don't have any type of "re-synchronization" mechanism so particularly
if we miss the TCP handshake and assume we may be starting in the
middle of the app-layer protocol stream (or else have a content gap),
the parser won't know what to do with the incoming data and so the
IsPartial() checks just exit early, before attempting to parse
further.

> How much reliable is this check TCP()->IsPartial()  for partial tcp sessions in the tcp analyzer?

Should be reliable in detecting the problematic scenario AFAIK, but in
the case where just the TCP handshake packets are missing and not any
segment data, analyzers that exit early like that are skipping streams
they actually should be able to parse.

- Jon


From hlin33 at illinois.edu  Thu Nov 28 11:36:15 2019
From: hlin33 at illinois.edu (Hui Lin (Hugo))
Date: Thu, 28 Nov 2019 11:36:15 -0800
Subject: [Zeek] Absolute ack and seq number of tcp packet
Message-ID: <CAKq214mCaG+gWWP8zBY4UEJHWRcRoikaSBw+4OibqR5nbK0Z9Q@mail.gmail.com>

Hi

In the tcp_packet event, how can I obtain the *absolute values* (found in
the tcp header), not the relative values of ack and seq numbers.

Best regards,

Hui Lin
-- 
Hui Lin
Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/)
DEPEND (http://depend.csl.illinois.edu/)
ECE, Uni. of Illinois at Urbana-Champaign
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191128/6bd7d9d7/attachment.html