From hkshin98 at gmail.com  Mon Sep  2 18:30:10 2019
From: hkshin98 at gmail.com (Raphael Shin)
Date: Tue, 3 Sep 2019 10:30:10 +0900
Subject: [Zeek] How to configure multiple interfaces
Message-ID: <CAC41-JVcm_KLu=1ZUfLCZe9-j6yEq9euRRYXzKGBbJrbFZ-59w@mail.gmail.com>

Hi,

I am installing Bro on Redhat OS.

My Bro machine has two interfaces.
 - Interface#1(p1p1) : Server farm *inbound* traffic
 - Interface#2(p1p2) : Server farm *outbound* traffic

I configured two interfaces with pf_ring.

node.cfg file is as follows.

----------------------------
[logger]
type=logger
host=localhost

[manager]
type=manager
host=localhost

[proxy-1]
type=proxy
host=localhost

[worker-1]
type=worker
host=localhost
interface=*p1p1*
lb_method=pf_ring
lb_procs=2
pin_cpus=8,9

[worker-2]
type=worker
host=localhost
interface=*p1p2*
lb_method=pf_ring
lb_procs=2
pin_cpus=10,11
----------------------------


but, I had wrong connection information.

Most conn_state is SH or SHR in the conn.log file.

How can I configure the node.cfg file?

Thanks,
Raphael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190903/aee43c77/attachment.html 

From justin at corelight.com  Mon Sep  2 18:39:30 2019
From: justin at corelight.com (Justin Azoff)
Date: Mon, 2 Sep 2019 21:39:30 -0400
Subject: [Zeek] How to configure multiple interfaces
In-Reply-To: <CAC41-JVcm_KLu=1ZUfLCZe9-j6yEq9euRRYXzKGBbJrbFZ-59w@mail.gmail.com>
References: <CAC41-JVcm_KLu=1ZUfLCZe9-j6yEq9euRRYXzKGBbJrbFZ-59w@mail.gmail.com>
Message-ID: <CAPfnCugcdqULz051RokUK2-DmLrYCWSVqjjECAVdXDtB+Vixdg@mail.gmail.com>

Install https://github.com/ntop/bro-pf_ring in general for best results.

Use

interface=pf_ring::p1p1,p1p2

On Mon, Sep 2, 2019 at 9:32 PM Raphael Shin <hkshin98 at gmail.com> wrote:

> Hi,
>
> I am installing Bro on Redhat OS.
>
> My Bro machine has two interfaces.
>  - Interface#1(p1p1) : Server farm *inbound* traffic
>  - Interface#2(p1p2) : Server farm *outbound* traffic
>
> I configured two interfaces with pf_ring.
>
> node.cfg file is as follows.
>
> ----------------------------
> [logger]
> type=logger
> host=localhost
>
> [manager]
> type=manager
> host=localhost
>
> [proxy-1]
> type=proxy
> host=localhost
>
> [worker-1]
> type=worker
> host=localhost
> interface=*p1p1*
> lb_method=pf_ring
> lb_procs=2
> pin_cpus=8,9
>
> [worker-2]
> type=worker
> host=localhost
> interface=*p1p2*
> lb_method=pf_ring
> lb_procs=2
> pin_cpus=10,11
> ----------------------------
>
>
> but, I had wrong connection information.
>
> Most conn_state is SH or SHR in the conn.log file.
>
> How can I configure the node.cfg file?
>
> Thanks,
> Raphael
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190902/dc18ea36/attachment.html 

From hkshin98 at gmail.com  Mon Sep  2 22:50:34 2019
From: hkshin98 at gmail.com (Raphael Shin)
Date: Tue, 3 Sep 2019 14:50:34 +0900
Subject: [Zeek] How to configure multiple interfaces
In-Reply-To: <CAPfnCugcdqULz051RokUK2-DmLrYCWSVqjjECAVdXDtB+Vixdg@mail.gmail.com>
References: <CAC41-JVcm_KLu=1ZUfLCZe9-j6yEq9euRRYXzKGBbJrbFZ-59w@mail.gmail.com>
	<CAPfnCugcdqULz051RokUK2-DmLrYCWSVqjjECAVdXDtB+Vixdg@mail.gmail.com>
Message-ID: <CAC41-JXOApSykz9kx9o5N16=PEcXEVUeuasWw0v0CZorQr_r5g@mail.gmail.com>

Thanks.

Your words helped me a lot.


2019? 9? 3? (?) ?? 10:40, Justin Azoff <justin at corelight.com>?? ??:

> Install https://github.com/ntop/bro-pf_ring in general for best results.
>
> Use
>
> interface=pf_ring::p1p1,p1p2
>
> On Mon, Sep 2, 2019 at 9:32 PM Raphael Shin <hkshin98 at gmail.com> wrote:
>
>> Hi,
>>
>> I am installing Bro on Redhat OS.
>>
>> My Bro machine has two interfaces.
>>  - Interface#1(p1p1) : Server farm *inbound* traffic
>>  - Interface#2(p1p2) : Server farm *outbound* traffic
>>
>> I configured two interfaces with pf_ring.
>>
>> node.cfg file is as follows.
>>
>> ----------------------------
>> [logger]
>> type=logger
>> host=localhost
>>
>> [manager]
>> type=manager
>> host=localhost
>>
>> [proxy-1]
>> type=proxy
>> host=localhost
>>
>> [worker-1]
>> type=worker
>> host=localhost
>> interface=*p1p1*
>> lb_method=pf_ring
>> lb_procs=2
>> pin_cpus=8,9
>>
>> [worker-2]
>> type=worker
>> host=localhost
>> interface=*p1p2*
>> lb_method=pf_ring
>> lb_procs=2
>> pin_cpus=10,11
>> ----------------------------
>>
>>
>> but, I had wrong connection information.
>>
>> Most conn_state is SH or SHR in the conn.log file.
>>
>> How can I configure the node.cfg file?
>>
>> Thanks,
>> Raphael
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
>
> --
> Justin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190903/8399345d/attachment.html 

From xavi at opennac.org  Tue Sep  3 01:20:57 2019
From: xavi at opennac.org (Xavier Gonzalez)
Date: Tue, 3 Sep 2019 08:20:57 +0000
Subject: [Zeek] How to configure multiple interfaces
In-Reply-To: <CAPfnCugcdqULz051RokUK2-DmLrYCWSVqjjECAVdXDtB+Vixdg@mail.gmail.com>
References: <CAC41-JVcm_KLu=1ZUfLCZe9-j6yEq9euRRYXzKGBbJrbFZ-59w@mail.gmail.com>
	<CAPfnCugcdqULz051RokUK2-DmLrYCWSVqjjECAVdXDtB+Vixdg@mail.gmail.com>
Message-ID: <AM0PR06MB445197DEEE89DB9136B8F306A0B90@AM0PR06MB4451.eurprd06.prod.outlook.com>

Hi Justin and Raphael,

Very good point and very interesting issue. Some questions?.

What does exactly do bro-pf_ring?
What does ?best results? mean?
Is it mandatory to use bro + pf_ring?
If not what features are added?

Thanks in advance,

Xavier Gonzalez
CTO & Co-founder
http://opennac.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__opencloudfactory.com_&d=DwMGaQ&c=57MOrb6NxZaYVa4uifS2UQ&r=7STsfyeMyac_1IFIgIS-WjdEirYO1GtwQ7VrUHTtpro&m=T67RemaYW12KUf2SFwdenG64ItIBu5RK6K_Iw51jv2U&s=-eQ9-r22Kp9xiDrpGjPtHV9AfU7tpjiF3EVDFkpQTm8&e=>
http://opencloudfactory.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__opencloudfactory.com_&d=DwMGaQ&c=57MOrb6NxZaYVa4uifS2UQ&r=7STsfyeMyac_1IFIgIS-WjdEirYO1GtwQ7VrUHTtpro&m=T67RemaYW12KUf2SFwdenG64ItIBu5RK6K_Iw51jv2U&s=-eQ9-r22Kp9xiDrpGjPtHV9AfU7tpjiF3EVDFkpQTm8&e=>
follow us: @opennac<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_opennac&d=DwMGaQ&c=57MOrb6NxZaYVa4uifS2UQ&r=7STsfyeMyac_1IFIgIS-WjdEirYO1GtwQ7VrUHTtpro&m=T67RemaYW12KUf2SFwdenG64ItIBu5RK6K_Iw51jv2U&s=Ue3qSgS7Wi6D-MiqHN_1d9FvO2A8XphTHeO9wdDIWwI&e=> @viapps<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_viapps&d=DwMGaQ&c=57MOrb6NxZaYVa4uifS2UQ&r=7STsfyeMyac_1IFIgIS-WjdEirYO1GtwQ7VrUHTtpro&m=T67RemaYW12KUf2SFwdenG64ItIBu5RK6K_Iw51jv2U&s=8TMKueanTMJrW8fDMeZ89hkHjOkTQAJTL0hWy8A9rN0&e=>

From: "zeek-bounces at zeek.org" <zeek-bounces at zeek.org> on behalf of Justin Azoff <justin at corelight.com>
Date: Tuesday, 3 September 2019 at 03:48
To: Raphael Shin <hkshin98 at gmail.com>
Cc: zeek <zeek at zeek.org>
Subject: Re: [Zeek] How to configure multiple interfaces

Install https://github.com/ntop/bro-pf_ring in general for best results.

Use

interface=pf_ring::p1p1,p1p2

On Mon, Sep 2, 2019 at 9:32 PM Raphael Shin <hkshin98 at gmail.com<mailto:hkshin98 at gmail.com>> wrote:
Hi,

I am installing Bro on Redhat OS.

My Bro machine has two interfaces.
 - Interface#1(p1p1) : Server farm inbound traffic
 - Interface#2(p1p2) : Server farm outbound traffic

I configured two interfaces with pf_ring.

node.cfg file is as follows.

----------------------------
[logger]
type=logger
host=localhost

[manager]
type=manager
host=localhost

[proxy-1]
type=proxy
host=localhost

[worker-1]
type=worker
host=localhost
interface=p1p1
lb_method=pf_ring
lb_procs=2
pin_cpus=8,9

[worker-2]
type=worker
host=localhost
interface=p1p2
lb_method=pf_ring
lb_procs=2
pin_cpus=10,11
----------------------------


but, I had wrong connection information.

Most conn_state is SH or SHR in the conn.log file.

How can I configure the node.cfg file?

Thanks,
Raphael
_______________________________________________
Zeek mailing list
zeek at zeek.org<mailto:zeek at zeek.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


--
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190903/cd2a1b7c/attachment-0001.html 

From adh.2234 at gmail.com  Tue Sep  3 01:42:16 2019
From: adh.2234 at gmail.com (=?utf-8?Q?=D0=98=D0=B2=D0=B0=D0=BD_=D0=A0=D0=B0=D1=82=D0=BA=D0=B8=D0=BD?=)
Date: Tue, 3 Sep 2019 11:42:16 +0300
Subject: [Zeek] How to configure multiple interfaces
In-Reply-To: <CAC41-JVcm_KLu=1ZUfLCZe9-j6yEq9euRRYXzKGBbJrbFZ-59w@mail.gmail.com>
References: <CAC41-JVcm_KLu=1ZUfLCZe9-j6yEq9euRRYXzKGBbJrbFZ-59w@mail.gmail.com>
Message-ID: <dc8564ad-34d9-4e49-900e-7cfa3aa9147e@Spark>

Guys, remove me from this, please. IDK how to stop recieving your emails about Bro.
Thanks.
3 ????. 2019 ?., 4:32 +0300, Raphael Shin <hkshin98 at gmail.com>, ?????:
> Hi,
>
> I am installing Bro on Redhat OS.
>
> My Bro machine has two interfaces.
> ?- Interface#1(p1p1) : Server farm inbound traffic
> ?- Interface#2(p1p2) : Server farm outbound traffic
>
> I configured two interfaces with pf_ring.
>
> node.cfg file is as follows.
>
> ----------------------------
> [logger]
> type=logger
> host=localhost
>
> [manager]
> type=manager
> host=localhost
>
> [proxy-1]
> type=proxy
> host=localhost
>
> [worker-1]
> type=worker
> host=localhost
> interface=p1p1
> lb_method=pf_ring
> lb_procs=2
> pin_cpus=8,9
>
> [worker-2]
> type=worker
> host=localhost
> interface=p1p2
> lb_method=pf_ring
> lb_procs=2
> pin_cpus=10,11
> ----------------------------
>
>
> but, I had wrong connection information.
>
> Most conn_state is SH or SHR in the conn.log file.
>
> How can I configure the node.cfg file?
>
> Thanks,
> Raphael
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190903/700801f8/attachment.html 

From ericooi at gmail.com  Tue Sep  3 07:48:16 2019
From: ericooi at gmail.com (Eric Ooi)
Date: Tue, 3 Sep 2019 09:48:16 -0500
Subject: [Zeek] How to configure multiple interfaces
In-Reply-To: <dc8564ad-34d9-4e49-900e-7cfa3aa9147e@Spark>
References: <CAC41-JVcm_KLu=1ZUfLCZe9-j6yEq9euRRYXzKGBbJrbFZ-59w@mail.gmail.com>
	<dc8564ad-34d9-4e49-900e-7cfa3aa9147e@Spark>
Message-ID: <8AFE2B5F-8072-4E8D-8E4C-244E2C1A13FF@gmail.com>

You can unsubscribe here: http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek <http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek>

> On Sep 3, 2019, at 3:42 AM, ???? ?????? <adh.2234 at gmail.com> wrote:
> 
> Guys, remove me from this, please. IDK how to stop recieving your emails about Bro. 
> Thanks.
> 3 ????. 2019 ?., 4:32 +0300, Raphael Shin <hkshin98 at gmail.com>, ?????:
>> Hi,
>> 
>> I am installing Bro on Redhat OS.
>> 
>> My Bro machine has two interfaces.
>>  - Interface#1(p1p1) : Server farm inbound traffic
>>  - Interface#2(p1p2) : Server farm outbound traffic
>> 
>> I configured two interfaces with pf_ring.
>> 
>> node.cfg file is as follows.
>> 
>> ----------------------------
>> [logger]
>> type=logger
>> host=localhost
>> 
>> [manager]
>> type=manager
>> host=localhost
>> 
>> [proxy-1]
>> type=proxy
>> host=localhost
>> 
>> [worker-1]
>> type=worker
>> host=localhost
>> interface=p1p1
>> lb_method=pf_ring
>> lb_procs=2
>> pin_cpus=8,9
>> 
>> [worker-2]
>> type=worker
>> host=localhost
>> interface=p1p2
>> lb_method=pf_ring
>> lb_procs=2
>> pin_cpus=10,11
>> ----------------------------  
>> 
>> 
>> but, I had wrong connection information.
>> 
>> Most conn_state is SH or SHR in the conn.log file.
>> 
>> How can I configure the node.cfg file?
>> 
>> Thanks,
>> Raphael
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190903/465e59b0/attachment.html 

From mgezz66 at gmail.com  Wed Sep  4 08:03:20 2019
From: mgezz66 at gmail.com (Michael Gez)
Date: Wed, 4 Sep 2019 11:03:20 -0400
Subject: [Zeek] Decreasing Log Cycle Time
Message-ID: <CALuqTtXunmUDgGpmQjgJ-FR-xt8-YFtvvVSB88uK_o2nxLVPVw@mail.gmail.com>

Hi all,

If I am not mistaken, Bro/Zeek cycles logs hourly.
This cycle is causing some unpredictable behavior in my tailing algorithm.

If i could set it to cycle every 5 minutes rather than on the hour, it
would be very beneficial to testing and resolving issues.
Is there a way I can reduce the amount of time Bro takes in between log
cycles?

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190904/43f0c26f/attachment.html 

From peter.hallin at ldc.lu.se  Wed Sep  4 08:35:46 2019
From: peter.hallin at ldc.lu.se (Peter Hallin)
Date: Wed, 4 Sep 2019 17:35:46 +0200
Subject: [Zeek] Decreasing Log Cycle Time
In-Reply-To: <CALuqTtXunmUDgGpmQjgJ-FR-xt8-YFtvvVSB88uK_o2nxLVPVw@mail.gmail.com>
References: <CALuqTtXunmUDgGpmQjgJ-FR-xt8-YFtvvVSB88uK_o2nxLVPVw@mail.gmail.com>
Message-ID: <20190904153546.mqhdplnwzm4u2m2k@mini.local>

Hello Michael,

In your broctl.cfg file, you can set the rotation interval to 300
seconds with:

LogRotationInterval = 300

Also, if you want to disable gzip on rotation, use this:

CompressLogs = 0

Brgds,

Peter


On 2019-09-04 11:03, Michael Gez wrote:
> Hi all,
> 
> If I am not mistaken, Bro/Zeek cycles logs hourly.
> This cycle is causing some unpredictable behavior in my tailing algorithm.
> 
> If i could set it to cycle every 5 minutes rather than on the hour, it
> would be very beneficial to testing and resolving issues.
> Is there a way I can reduce the amount of time Bro takes in between log
> cycles?
> 
> Thanks!

> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From ericooi at gmail.com  Wed Sep  4 08:36:52 2019
From: ericooi at gmail.com (Eric Ooi)
Date: Wed, 4 Sep 2019 10:36:52 -0500
Subject: [Zeek] Decreasing Log Cycle Time
In-Reply-To: <CALuqTtXunmUDgGpmQjgJ-FR-xt8-YFtvvVSB88uK_o2nxLVPVw@mail.gmail.com>
References: <CALuqTtXunmUDgGpmQjgJ-FR-xt8-YFtvvVSB88uK_o2nxLVPVw@mail.gmail.com>
Message-ID: <0DA6DD98-F9C6-40A1-9D92-0FDC1350BA67@gmail.com>

Hi Michael,

You should be able to do this by changing the ?LogRotationInterval? variable in /opt/bro/etc/broctl.cfg (assuming you installed bro in /opt/).  By default it?s set to 3600 seconds (1 hour), so setting this to 300 should change this to 5 minutes.  You?ll then have to stop Bro (broctl stop) and redeploy (broctl deploy) for the changes to take effect.

Hope that helps!
Eric
ericooi.com

> On Sep 4, 2019, at 10:03 AM, Michael Gez <mgezz66 at gmail.com> wrote:
> 
> Hi all,
> 
> If I am not mistaken, Bro/Zeek cycles logs hourly. 
> This cycle is causing some unpredictable behavior in my tailing algorithm.
> 
> If i could set it to cycle every 5 minutes rather than on the hour, it would be very beneficial to testing and resolving issues. 
> Is there a way I can reduce the amount of time Bro takes in between log cycles?
> 
> Thanks!
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



From asharma at lbl.gov  Wed Sep  4 09:11:39 2019
From: asharma at lbl.gov (Aashish Sharma)
Date: Wed, 4 Sep 2019 09:11:39 -0700
Subject: [Zeek] Decreasing Log Cycle Time
In-Reply-To: <CALuqTtXunmUDgGpmQjgJ-FR-xt8-YFtvvVSB88uK_o2nxLVPVw@mail.gmail.com>
References: <CALuqTtXunmUDgGpmQjgJ-FR-xt8-YFtvvVSB88uK_o2nxLVPVw@mail.gmail.com>
Message-ID: <20190904161138.GB42937@MacPro-2331.local>

In broctl.cfg, you can set

LogRotationInterval = 300 



On Wed, Sep 04, 2019 at 11:03:20AM -0400, Michael Gez wrote:
> Hi all,
> 
> If I am not mistaken, Bro/Zeek cycles logs hourly.
> This cycle is causing some unpredictable behavior in my tailing algorithm.
> 
> If i could set it to cycle every 5 minutes rather than on the hour, it
> would be very beneficial to testing and resolving issues.
> Is there a way I can reduce the amount of time Bro takes in between log
> cycles?
> 
> Thanks!

> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From greg.grasmehr at caltech.edu  Thu Sep  5 12:23:04 2019
From: greg.grasmehr at caltech.edu (Greg Grasmehr)
Date: Thu, 5 Sep 2019 12:23:04 -0700
Subject: [Zeek] Zeek 2.6.4 undefined symbol: bro_version_2_6_1_plugin_6
Message-ID: <20190905192304.GC17641@dakine>

Greetings,

Compiled Zeek 2.6.4 and when executing deploy I get the errors below,
anyone know the fix?

[BroControl] > deploy
checking configurations ...
logger scripts failed.
fatal error in /usr/local/bro/share/bro/base/init-bare.bro, line 1: cannot load plugin library /usr/local/bro/lib/bro/plugins/packages/bro-myricom//lib/Bro-Myricom.linux-x86_64.so: /usr/local/bro/lib/bro/plugins/packages/bro-myricom//lib/Bro-Myricom.linux-x86_64.so: undefined symbol: bro_version_2_6_1_plugin_6

manager scripts failed.
fatal error in /usr/local/bro/share/bro/base/init-bare.bro, line 1: cannot load plugin library /usr/local/bro/lib/bro/plugins/packages/bro-myricom//lib/Bro-Myricom.linux-x86_64.so: /usr/local/bro/lib/bro/plugins/packages/bro-myricom//lib/Bro-Myricom.linux-x86_64.so: undefined symbol: bro_version_2_6_1_plugin_6

proxy-1 scripts failed.
fatal error in /usr/local/bro/share/bro/base/init-bare.bro, line 1: cannot load plugin library /usr/local/bro/lib/bro/plugins/packages/bro-myricom//lib/Bro-Myricom.linux-x86_64.so: /usr/local/bro/lib/bro/plugins/packages/bro-myricom//lib/Bro-Myricom.linux-x86_64.so: undefined symbol: bro_version_2_6_1_plugin_6

worker-1-1 scripts failed.
fatal error in /usr/local/bro/share/bro/base/init-bare.bro, line 1: cannot load plugin library /usr/local/bro/lib/bro/plugins/packages/bro-myricom//lib/Bro-Myricom.linux-x86_64.so: /usr/local/bro/lib/bro/plugins/packages/bro-myricom//lib/Bro-Myricom.linux-x86_64.so: undefined symbol: bro_version_2_6_1_plugin_6

Thanks!

Greg

From dopheide at gmail.com  Thu Sep  5 12:36:02 2019
From: dopheide at gmail.com (Mike Dopheide)
Date: Thu, 5 Sep 2019 14:36:02 -0500
Subject: [Zeek] Zeek 2.6.4 undefined symbol: bro_version_2_6_1_plugin_6
In-Reply-To: <20190905192304.GC17641@dakine>
References: <20190905192304.GC17641@dakine>
Message-ID: <CAPy2kFbL9q=Md0NDNy3QmALu2V69Vt0zzOSe0m180mS_SOWBZw@mail.gmail.com>

Greg,

It looks like you'll want to remove and reinstall the bro-myricom package
which will rebuild the plugin against the new source tree.

-Dop


On Thu, Sep 5, 2019 at 2:31 PM Greg Grasmehr <greg.grasmehr at caltech.edu>
wrote:

> Greetings,
>
> Compiled Zeek 2.6.4 and when executing deploy I get the errors below,
> anyone know the fix?
>
> [BroControl] > deploy
> checking configurations ...
> logger scripts failed.
> fatal error in /usr/local/bro/share/bro/base/init-bare.bro, line 1: cannot
> load plugin library
> /usr/local/bro/lib/bro/plugins/packages/bro-myricom//lib/
> Bro-Myricom.linux-x86_64.so:
> /usr/local/bro/lib/bro/plugins/packages/bro-myricom//lib/
> Bro-Myricom.linux-x86_64.so: undefined symbol: bro_version_2_6_1_plugin_6
>
> manager scripts failed.
> fatal error in /usr/local/bro/share/bro/base/init-bare.bro, line 1: cannot
> load plugin library
> /usr/local/bro/lib/bro/plugins/packages/bro-myricom//lib/
> Bro-Myricom.linux-x86_64.so:
> /usr/local/bro/lib/bro/plugins/packages/bro-myricom//lib/
> Bro-Myricom.linux-x86_64.so: undefined symbol: bro_version_2_6_1_plugin_6
>
> proxy-1 scripts failed.
> fatal error in /usr/local/bro/share/bro/base/init-bare.bro, line 1: cannot
> load plugin library
> /usr/local/bro/lib/bro/plugins/packages/bro-myricom//lib/
> Bro-Myricom.linux-x86_64.so:
> /usr/local/bro/lib/bro/plugins/packages/bro-myricom//lib/
> Bro-Myricom.linux-x86_64.so: undefined symbol: bro_version_2_6_1_plugin_6
>
> worker-1-1 scripts failed.
> fatal error in /usr/local/bro/share/bro/base/init-bare.bro, line 1: cannot
> load plugin library
> /usr/local/bro/lib/bro/plugins/packages/bro-myricom//lib/
> Bro-Myricom.linux-x86_64.so:
> /usr/local/bro/lib/bro/plugins/packages/bro-myricom//lib/
> Bro-Myricom.linux-x86_64.so: undefined symbol: bro_version_2_6_1_plugin_6
>
> Thanks!
>
> Greg
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190905/a0753e6b/attachment.html 

From greg.grasmehr at caltech.edu  Thu Sep  5 12:53:35 2019
From: greg.grasmehr at caltech.edu (Greg Grasmehr)
Date: Thu, 5 Sep 2019 12:53:35 -0700
Subject: [Zeek] Zeek 2.6.4 undefined symbol: bro_version_2_6_1_plugin_6
In-Reply-To: <CAPy2kFbL9q=Md0NDNy3QmALu2V69Vt0zzOSe0m180mS_SOWBZw@mail.gmail.com>
References: <20190905192304.GC17641@dakine>
	<CAPy2kFbL9q=Md0NDNy3QmALu2V69Vt0zzOSe0m180mS_SOWBZw@mail.gmail.com>
Message-ID: <20190905195335.GD17641@dakine>

Hi,

Yes I did that - didn't work.  In fact there is no bro-myricom; there is
only bro/sethhall/bro-myricom found when searching with bro-pkg

Greg

On 09/05/19 14:36:02, Mike Dopheide wrote:
> Greg,
> 
> It looks like you'll want to remove and reinstall the bro-myricom package
> which will rebuild the plugin against the new source tree.
> 
> -Dop

From jlay at slave-tothe-box.net  Thu Sep  5 13:24:46 2019
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 05 Sep 2019 14:24:46 -0600
Subject: [Zeek] Zeek 2.6.4 undefined symbol: bro_version_2_6_1_plugin_6
In-Reply-To: <20190905195335.GD17641@dakine>
References: <20190905192304.GC17641@dakine>
	<CAPy2kFbL9q=Md0NDNy3QmALu2V69Vt0zzOSe0m180mS_SOWBZw@mail.gmail.com>
	<20190905195335.GD17641@dakine>
Message-ID: <8bb70b3662b1c7afba11ae55794c1243@slave-tothe-box.net>

Do:

git clone https://github.com/sethhall/bro-myricom.git
cd bro-myricom
./configure --bro-dist=<where bro-2.6.4 source is>
make
sudo make install

I've always removed the package prior to doing this.

James

On 2019-09-05 13:53, Greg Grasmehr wrote:
> Hi,
> 
> Yes I did that - didn't work.  In fact there is no bro-myricom; there 
> is
> only bro/sethhall/bro-myricom found when searching with bro-pkg
> 
> Greg
> 
> On 09/05/19 14:36:02, Mike Dopheide wrote:
>> Greg,
>> 
>> It looks like you'll want to remove and reinstall the bro-myricom 
>> package
>> which will rebuild the plugin against the new source tree.
>> 
>> -Dop
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

From dopheide at gmail.com  Thu Sep  5 13:43:20 2019
From: dopheide at gmail.com (Mike Dopheide)
Date: Thu, 5 Sep 2019 15:43:20 -0500
Subject: [Zeek] Zeek 2.6.4 undefined symbol: bro_version_2_6_1_plugin_6
In-Reply-To: <8bb70b3662b1c7afba11ae55794c1243@slave-tothe-box.net>
References: <20190905192304.GC17641@dakine>
	<CAPy2kFbL9q=Md0NDNy3QmALu2V69Vt0zzOSe0m180mS_SOWBZw@mail.gmail.com>
	<20190905195335.GD17641@dakine>
	<8bb70b3662b1c7afba11ae55794c1243@slave-tothe-box.net>
Message-ID: <CAPy2kFZ7=LRAG-+9ZOiuj6yR0hgNzR7rjQUfCfy650yCDARn1g@mail.gmail.com>

Yeah, what James said.  Doing it manually will help track down the issue.
This could be an issue with which zeek_dist is pointed to in your
.zkg/config.  I've had that happen A LOT.

-Dop


On Thu, Sep 5, 2019 at 3:36 PM James Lay <jlay at slave-tothe-box.net> wrote:

> Do:
>
> git clone https://github.com/sethhall/bro-myricom.git
> cd bro-myricom
> ./configure --bro-dist=<where bro-2.6.4 source is>
> make
> sudo make install
>
> I've always removed the package prior to doing this.
>
> James
>
> On 2019-09-05 13:53, Greg Grasmehr wrote:
> > Hi,
> >
> > Yes I did that - didn't work.  In fact there is no bro-myricom; there
> > is
> > only bro/sethhall/bro-myricom found when searching with bro-pkg
> >
> > Greg
> >
> > On 09/05/19 14:36:02, Mike Dopheide wrote:
> >> Greg,
> >>
> >> It looks like you'll want to remove and reinstall the bro-myricom
> >> package
> >> which will rebuild the plugin against the new source tree.
> >>
> >> -Dop
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190905/e4046d3e/attachment.html 

From greg.grasmehr at caltech.edu  Thu Sep  5 15:02:21 2019
From: greg.grasmehr at caltech.edu (Greg Grasmehr)
Date: Thu, 5 Sep 2019 15:02:21 -0700
Subject: [Zeek] Zeek 2.6.4 undefined symbol: bro_version_2_6_1_plugin_6
In-Reply-To: <CAPy2kFZ7=LRAG-+9ZOiuj6yR0hgNzR7rjQUfCfy650yCDARn1g@mail.gmail.com>
References: <20190905192304.GC17641@dakine>
	<CAPy2kFbL9q=Md0NDNy3QmALu2V69Vt0zzOSe0m180mS_SOWBZw@mail.gmail.com>
	<20190905195335.GD17641@dakine>
	<8bb70b3662b1c7afba11ae55794c1243@slave-tothe-box.net>
	<CAPy2kFZ7=LRAG-+9ZOiuj6yR0hgNzR7rjQUfCfy650yCDARn1g@mail.gmail.com>
Message-ID: <20190905220221.GE17641@dakine>

So I ran zkg autoconfig - that fixed that problem, thanks for the hints.

Greg

On 09/05/19 15:43:20, Mike Dopheide wrote:
> Yeah, what James said.  Doing it manually will help track down the issue.
> This could be an issue with which zeek_dist is pointed to in your
> .zkg/config.  I've had that happen A LOT.
> 
> -Dop

From greg.grasmehr at caltech.edu  Thu Sep  5 15:04:18 2019
From: greg.grasmehr at caltech.edu (Greg Grasmehr)
Date: Thu, 5 Sep 2019 15:04:18 -0700
Subject: [Zeek] Please add NCSA/Dumbno to zkg management
Message-ID: <20190905220418.GF17641@dakine>

Hey Justin,

Please add NCSA/Dumbno to zkg package management; what a huge hassle
every time there's upgrade.  :(

Greg

From justin at corelight.com  Thu Sep  5 16:13:14 2019
From: justin at corelight.com (Justin Azoff)
Date: Thu, 5 Sep 2019 19:13:14 -0400
Subject: [Zeek] Please add NCSA/Dumbno to zkg management
In-Reply-To: <20190905220418.GF17641@dakine>
References: <20190905220418.GF17641@dakine>
Message-ID: <CAPfnCugHuPmeBB7eCgZxXxQmaGS8ku7t3s0YwvCPVt9+VwA1Aw@mail.gmail.com>

Oh.. I forgot to submit it to the packages repo, but fyi you can just run

zkg install https://github.com/ncsa/bro-dumbno-client

and it will install the normal way.. you just can't install it with the
short name because it's not in the package index.

On Thu, Sep 5, 2019 at 6:04 PM Greg Grasmehr <greg.grasmehr at caltech.edu>
wrote:

> Hey Justin,
>
> Please add NCSA/Dumbno to zkg package management; what a huge hassle
> every time there's upgrade.  :(
>
> Greg
>


-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190905/e1b16b1b/attachment.html 

From greg.grasmehr at caltech.edu  Thu Sep  5 17:07:00 2019
From: greg.grasmehr at caltech.edu (Greg Grasmehr)
Date: Thu, 5 Sep 2019 17:07:00 -0700
Subject: [Zeek] Please add NCSA/Dumbno to zkg management
In-Reply-To: <CAPfnCugHuPmeBB7eCgZxXxQmaGS8ku7t3s0YwvCPVt9+VwA1Aw@mail.gmail.com>
References: <20190905220418.GF17641@dakine>
	<CAPfnCugHuPmeBB7eCgZxXxQmaGS8ku7t3s0YwvCPVt9+VwA1Aw@mail.gmail.com>
Message-ID: <20190906000659.GI17641@dakine>

Oh excellent news!  Thank you that is a big help.

Greg

On 09/05/19 19:13:14, Justin Azoff wrote:
> Oh.. I forgot to submit it to the packages repo, but fyi you can just run
> 
> zkg install https://github.com/ncsa/bro-dumbno-client
> 
> and it will install the normal way.. you just can't install it with the
> short name because it's not in the package index.
> 
> On Thu, Sep 5, 2019 at 6:04 PM Greg Grasmehr <greg.grasmehr at caltech.edu>
> wrote:
> 
> > Hey Justin,
> >
> > Please add NCSA/Dumbno to zkg package management; what a huge hassle
> > every time there's upgrade.  :(
> >
> > Greg
> >
> 
> 
> -- 
> Justin

From fatema.bannatwala at gmail.com  Fri Sep  6 08:37:21 2019
From: fatema.bannatwala at gmail.com (fatema bannatwala)
Date: Fri, 6 Sep 2019 11:37:21 -0400
Subject: [Zeek] Increased memory usage by Zeek..
Message-ID: <CACX0rUQ4eYH7Pc-pg+fbQ-0rzqhHk4674z2dXoQBy5Sr6EZ4kw@mail.gmail.com>

Hi All,

Couple of months ago I upgraded the Zeek cluster from 2.5 to 2.6.1
(compiled with the jemalloc support).
I have started seeing increased memory usage by the workers.

I have two physical sensors, each running 18 Zeek worker processes LB by
PF_RING.
Not loaded any custom scripts, just the basic scripts that are enabled by
default in local.bro (also have misc/scan disabled).

I just did a top on one of the boxes and here's the output (specially two
Zeek processes -*13632, **13611* using >10% memory which is ~11G)
Also, attaching a weekly available free memory graph for the system.

Tasks: 455 total,   9 running, 443 sleeping,   0 stopped,   3 zombie
%Cpu(s): 18.3 us,  1.7 sy,  0.0 ni, 79.5 id,  0.0 wa,  0.0 hi,  0.4 si,
 0.0 st
KiB Mem : 98783960 total, 32963660 free, *64807572* used,  1012728
buff/cache
KiB Swap:  4194300 total,  3572200 free,   622100 used. 33221356 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
13589 bro       20   0 3662052   3.4g  73340 R    90.4     3.6    1072:47
  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
-p local -p worker-2-5 local.bro broctl base/frameworks/cluster broctl/auto
13533 bro       20   0 1847972   1.6g  73188 S    50.3     1.7    1098:05
  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
-p local -p worker-2-6 local.bro broctl base/frameworks/cluster broctl/auto
13512 bro       20   0 1291260   1.1g  73052 S    49.7     1.1    1080:30
  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
-p local -p worker-2-1 local.bro broctl base/frameworks/cluster broctl/auto
13628 bro       20   0 2347952   2.1g  73328 R    49.0     2.2    1109:31
  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
-p local -p worker-2-12 local.bro broctl base/frameworks/cluster broctl/auto
13516 bro       20   0  973260 799176  72844 R  47.0     0.8    1036:29
 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
-p local -p worker-2-3 local.bro broctl base/frameworks/cluster broctl/auto
13539 bro       20   0 6374956   6.0g  73456 S    46.0     6.3    1147:08
 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
-p local -p worker-2-2 local.bro broctl base/frameworks/cluster broctl/auto
13591 bro       20   0  865952 726516  73020 S  44.7     0.7    1052:29
 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
-p local -p worker-2-4 local.bro broctl base/frameworks/cluster broctl/auto
*13632* *bro       20   0   12.2g  12.0g  73584 R      43.7  *  *12.8*
1068:17   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
broctl-live -p local -p worker-2-15 local.bro broctl
base/frameworks/cluster broctl/auto
13540 bro       20   0 2146844   1.9g  73348 R    41.4     2.0    1149:38
 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
-p local -p worker-2-7 local.bro broctl base/frameworks/cluster broctl/auto
*13611* bro       *20   0   17.0g  16.7g  73404 S      39.7*    *17.8*
1172:14   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
broctl-live -p local -p worker-2-9 local.bro broctl base/frameworks/cluster
broctl/auto
13640 bro       20   0 2624300   2.1g  73328 S    39.7     2.3    1043:50
 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
-p local -p worker-2-18 local.bro broctl base/frameworks/cluster broctl/auto
13586 bro       20   0 3347044   3.1g  73468 S    39.1     3.2    1042:24
 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
-p local -p worker-2-10 local.bro broctl base/frameworks/cluster broctl/auto
13641 bro       20   0 2274788   2.0g  73424 R    39.1     2.2    1029:58
 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
-p local -p worker-2-17 local.bro broctl base/frameworks/cluster broctl/auto
13614 bro       20   0 1954780   1.7g  73188 S    38.4     1.8   995:00.54
 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
-p local -p worker-2-13 local.bro broctl base/frameworks/cluster broctl/auto
13627 bro       20   0 2756520   2.5g  73288 S    38.4     2.6     1035:18
 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
-p local -p worker-2-14 local.bro broctl base/frameworks/cluster broctl/auto
13638 bro       20   0 1206548 853056  72328 R  37.4    0.9   952:10.00
 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
-p local -p worker-2-16 local.bro broctl base/frameworks/cluster broctl/auto
13623 bro       20   0 8998324   2.1g  73284 S     37.1    2.2     1073:31
 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
-p local -p worker-2-11 local.bro broctl base/frameworks/cluster broctl/auto
13575 bro       20   0  871396 706148  73128 R    36.4   0.7     1028:30
 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
-p local -p worker-2-8 local.bro broctl base/frameworks/cluster broctl/auto
13336 bro       20   0  266244 133920  33388 S    12.6   0.1   400:27.62
 /usr/local/bro/2.6.1/bin/bro -U .status -p broctl -p broctl-live -p local
-p proxy-2 local.bro broctl base/frameworks/cluster broctl/auto

Any suggestions?

Thanks!
Fatema
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190906/2c64ba2a/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: weekly-mem-use.PNG
Type: image/png
Size: 15940 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190906/2c64ba2a/attachment-0001.bin 

From justin at corelight.com  Fri Sep  6 09:48:23 2019
From: justin at corelight.com (Justin Azoff)
Date: Fri, 6 Sep 2019 12:48:23 -0400
Subject: [Zeek] Increased memory usage by Zeek..
In-Reply-To: <CACX0rUQ4eYH7Pc-pg+fbQ-0rzqhHk4674z2dXoQBy5Sr6EZ4kw@mail.gmail.com>
References: <CACX0rUQ4eYH7Pc-pg+fbQ-0rzqhHk4674z2dXoQBy5Sr6EZ4kw@mail.gmail.com>
Message-ID: <CAPfnCug=b2arfebNOpVXTgxbwk8Bw_FYuzaOdsGfCM42aYXcKQ@mail.gmail.com>

Hi!

I've been doing a ton of work in this space and have some tooling I've been
working on to help track down things like this.  I'm planning to have
things ready for my ZeekWeek presentation, but if you have some time I can
share the work-in-progress stuff with you and go over how to use it (which
will help with the documentation bits that still need to be written).

The good news is I wouldn't be surprised if this issue is already fixed or
drastically better in 3.0 or master.


On Fri, Sep 6, 2019 at 11:47 AM fatema bannatwala <
fatema.bannatwala at gmail.com> wrote:

> Hi All,
>
> Couple of months ago I upgraded the Zeek cluster from 2.5 to 2.6.1
> (compiled with the jemalloc support).
> I have started seeing increased memory usage by the workers.
>
> I have two physical sensors, each running 18 Zeek worker processes LB by
> PF_RING.
> Not loaded any custom scripts, just the basic scripts that are enabled by
> default in local.bro (also have misc/scan disabled).
>
> I just did a top on one of the boxes and here's the output (specially two
> Zeek processes -*13632, **13611* using >10% memory which is ~11G)
> Also, attaching a weekly available free memory graph for the system.
>
> Tasks: 455 total,   9 running, 443 sleeping,   0 stopped,   3 zombie
> %Cpu(s): 18.3 us,  1.7 sy,  0.0 ni, 79.5 id,  0.0 wa,  0.0 hi,  0.4 si,
>  0.0 st
> KiB Mem : 98783960 total, 32963660 free, *64807572* used,  1012728
> buff/cache
> KiB Swap:  4194300 total,  3572200 free,   622100 used. 33221356 avail Mem
>
>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
> 13589 bro       20   0 3662052   3.4g  73340 R    90.4     3.6    1072:47
>   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-5 local.bro broctl base/frameworks/cluster broctl/auto
> 13533 bro       20   0 1847972   1.6g  73188 S    50.3     1.7    1098:05
>   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-6 local.bro broctl base/frameworks/cluster broctl/auto
> 13512 bro       20   0 1291260   1.1g  73052 S    49.7     1.1    1080:30
>   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-1 local.bro broctl base/frameworks/cluster broctl/auto
> 13628 bro       20   0 2347952   2.1g  73328 R    49.0     2.2    1109:31
>   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-12 local.bro broctl base/frameworks/cluster broctl/auto
> 13516 bro       20   0  973260 799176  72844 R  47.0     0.8    1036:29
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-3 local.bro broctl base/frameworks/cluster broctl/auto
> 13539 bro       20   0 6374956   6.0g  73456 S    46.0     6.3    1147:08
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-2 local.bro broctl base/frameworks/cluster broctl/auto
> 13591 bro       20   0  865952 726516  73020 S  44.7     0.7    1052:29
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-4 local.bro broctl base/frameworks/cluster broctl/auto
> *13632* *bro       20   0   12.2g  12.0g  73584 R      43.7  *  *12.8*
> 1068:17   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-15 local.bro broctl
> base/frameworks/cluster broctl/auto
> 13540 bro       20   0 2146844   1.9g  73348 R    41.4     2.0    1149:38
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-7 local.bro broctl base/frameworks/cluster broctl/auto
> *13611* bro       *20   0   17.0g  16.7g  73404 S      39.7*    *17.8*
> 1172:14   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-9 local.bro broctl base/frameworks/cluster
> broctl/auto
> 13640 bro       20   0 2624300   2.1g  73328 S    39.7     2.3    1043:50
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-18 local.bro broctl base/frameworks/cluster broctl/auto
> 13586 bro       20   0 3347044   3.1g  73468 S    39.1     3.2    1042:24
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-10 local.bro broctl base/frameworks/cluster broctl/auto
> 13641 bro       20   0 2274788   2.0g  73424 R    39.1     2.2    1029:58
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-17 local.bro broctl base/frameworks/cluster broctl/auto
> 13614 bro       20   0 1954780   1.7g  73188 S    38.4     1.8
>  995:00.54   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-13 local.bro broctl
> base/frameworks/cluster broctl/auto
> 13627 bro       20   0 2756520   2.5g  73288 S    38.4     2.6
>  1035:18   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-14 local.bro broctl
> base/frameworks/cluster broctl/auto
> 13638 bro       20   0 1206548 853056  72328 R  37.4    0.9   952:10.00
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-16 local.bro broctl base/frameworks/cluster broctl/auto
> 13623 bro       20   0 8998324   2.1g  73284 S     37.1    2.2
>  1073:31   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-11 local.bro broctl
> base/frameworks/cluster broctl/auto
> 13575 bro       20   0  871396 706148  73128 R    36.4   0.7     1028:30
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-8 local.bro broctl base/frameworks/cluster broctl/auto
> 13336 bro       20   0  266244 133920  33388 S    12.6   0.1   400:27.62
>  /usr/local/bro/2.6.1/bin/bro -U .status -p broctl -p broctl-live -p local
> -p proxy-2 local.bro broctl base/frameworks/cluster broctl/auto
>
> Any suggestions?
>
> Thanks!
> Fatema
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190906/7d906ac3/attachment.html 

From jsiwek at corelight.com  Fri Sep  6 09:58:33 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Fri, 6 Sep 2019 09:58:33 -0700
Subject: [Zeek] Increased memory usage by Zeek..
In-Reply-To: <CACX0rUQ4eYH7Pc-pg+fbQ-0rzqhHk4674z2dXoQBy5Sr6EZ4kw@mail.gmail.com>
References: <CACX0rUQ4eYH7Pc-pg+fbQ-0rzqhHk4674z2dXoQBy5Sr6EZ4kw@mail.gmail.com>
Message-ID: <CAMzgZ0++AmH70SP1TAvmimJtungtahyoq=DwLMqDxzQeDWuXpA@mail.gmail.com>

Biggest changes from 2.5.x to 2.6.x that I can recall are (1)
switching remote communication to use the new Broker library and (2)
enabling SMB analysis by default.

Had you manually enabled SMB in your previous 2.5.x deployment?  If
not, you could see if disabling it helps:

    redef Analyzer::disabled_analyzers += { Analyzer::ANALYZER_SMB };

That's my first guess because we've recently seen/suspected (but not
yet fixed) some state management issues in the SMB analysis scripts
that might explain high memory usage.

- Jon

On Fri, Sep 6, 2019 at 8:46 AM fatema bannatwala
<fatema.bannatwala at gmail.com> wrote:
>
> Hi All,
>
> Couple of months ago I upgraded the Zeek cluster from 2.5 to 2.6.1 (compiled with the jemalloc support).
> I have started seeing increased memory usage by the workers.
>
> I have two physical sensors, each running 18 Zeek worker processes LB by PF_RING.
> Not loaded any custom scripts, just the basic scripts that are enabled by default in local.bro (also have misc/scan disabled).
>
> I just did a top on one of the boxes and here's the output (specially two Zeek processes -13632, 13611 using >10% memory which is ~11G)
> Also, attaching a weekly available free memory graph for the system.
>
> Tasks: 455 total,   9 running, 443 sleeping,   0 stopped,   3 zombie
> %Cpu(s): 18.3 us,  1.7 sy,  0.0 ni, 79.5 id,  0.0 wa,  0.0 hi,  0.4 si,  0.0 st
> KiB Mem : 98783960 total, 32963660 free, 64807572 used,  1012728 buff/cache
> KiB Swap:  4194300 total,  3572200 free,   622100 used. 33221356 avail Mem
>
>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
> 13589 bro       20   0 3662052   3.4g  73340 R    90.4     3.6    1072:47    /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-5 local.bro broctl base/frameworks/cluster broctl/auto
> 13533 bro       20   0 1847972   1.6g  73188 S    50.3     1.7    1098:05    /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-6 local.bro broctl base/frameworks/cluster broctl/auto
> 13512 bro       20   0 1291260   1.1g  73052 S    49.7     1.1    1080:30    /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-1 local.bro broctl base/frameworks/cluster broctl/auto
> 13628 bro       20   0 2347952   2.1g  73328 R    49.0     2.2    1109:31    /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-12 local.bro broctl base/frameworks/cluster broctl/auto
> 13516 bro       20   0  973260 799176  72844 R  47.0     0.8    1036:29   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-3 local.bro broctl base/frameworks/cluster broctl/auto
> 13539 bro       20   0 6374956   6.0g  73456 S    46.0     6.3    1147:08   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-2 local.bro broctl base/frameworks/cluster broctl/auto
> 13591 bro       20   0  865952 726516  73020 S  44.7     0.7    1052:29   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-4 local.bro broctl base/frameworks/cluster broctl/auto
> 13632 bro       20   0   12.2g  12.0g  73584 R      43.7    12.8    1068:17   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-15 local.bro broctl base/frameworks/cluster broctl/auto
> 13540 bro       20   0 2146844   1.9g  73348 R    41.4     2.0    1149:38   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-7 local.bro broctl base/frameworks/cluster broctl/auto
> 13611 bro       20   0   17.0g  16.7g  73404 S      39.7    17.8    1172:14   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-9 local.bro broctl base/frameworks/cluster broctl/auto
> 13640 bro       20   0 2624300   2.1g  73328 S    39.7     2.3    1043:50   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-18 local.bro broctl base/frameworks/cluster broctl/auto
> 13586 bro       20   0 3347044   3.1g  73468 S    39.1     3.2    1042:24   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-10 local.bro broctl base/frameworks/cluster broctl/auto
> 13641 bro       20   0 2274788   2.0g  73424 R    39.1     2.2    1029:58   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-17 local.bro broctl base/frameworks/cluster broctl/auto
> 13614 bro       20   0 1954780   1.7g  73188 S    38.4     1.8   995:00.54   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-13 local.bro broctl base/frameworks/cluster broctl/auto
> 13627 bro       20   0 2756520   2.5g  73288 S    38.4     2.6     1035:18   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-14 local.bro broctl base/frameworks/cluster broctl/auto
> 13638 bro       20   0 1206548 853056  72328 R  37.4    0.9   952:10.00   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-16 local.bro broctl base/frameworks/cluster broctl/auto
> 13623 bro       20   0 8998324   2.1g  73284 S     37.1    2.2     1073:31   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-11 local.bro broctl base/frameworks/cluster broctl/auto
> 13575 bro       20   0  871396 706148  73128 R    36.4   0.7     1028:30   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-8 local.bro broctl base/frameworks/cluster broctl/auto
> 13336 bro       20   0  266244 133920  33388 S    12.6   0.1   400:27.62   /usr/local/bro/2.6.1/bin/bro -U .status -p broctl -p broctl-live -p local -p proxy-2 local.bro broctl base/frameworks/cluster broctl/auto
>
> Any suggestions?
>
> Thanks!
> Fatema
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

From fatema.bannatwala at gmail.com  Fri Sep  6 10:19:15 2019
From: fatema.bannatwala at gmail.com (fatema bannatwala)
Date: Fri, 6 Sep 2019 13:19:15 -0400
Subject: [Zeek] Increased memory usage by Zeek..
In-Reply-To: <CAMzgZ0++AmH70SP1TAvmimJtungtahyoq=DwLMqDxzQeDWuXpA@mail.gmail.com>
References: <CACX0rUQ4eYH7Pc-pg+fbQ-0rzqhHk4674z2dXoQBy5Sr6EZ4kw@mail.gmail.com>
	<CAMzgZ0++AmH70SP1TAvmimJtungtahyoq=DwLMqDxzQeDWuXpA@mail.gmail.com>
Message-ID: <CACX0rUQ189HgRKX3T-FE7NCwg3CmHkAfGOEjbUKip6j7ndcS_g@mail.gmail.com>

Hmm, I will disable the SMB analyzer in local.bro and see if it helps..
Thanks Jon! :-)


On Fri, Sep 6, 2019 at 12:58 PM Jon Siwek <jsiwek at corelight.com> wrote:

> Biggest changes from 2.5.x to 2.6.x that I can recall are (1)
> switching remote communication to use the new Broker library and (2)
> enabling SMB analysis by default.
>
> Had you manually enabled SMB in your previous 2.5.x deployment?  If
> not, you could see if disabling it helps:
>
>     redef Analyzer::disabled_analyzers += { Analyzer::ANALYZER_SMB };
>
> That's my first guess because we've recently seen/suspected (but not
> yet fixed) some state management issues in the SMB analysis scripts
> that might explain high memory usage.
>
> - Jon
>
> On Fri, Sep 6, 2019 at 8:46 AM fatema bannatwala
> <fatema.bannatwala at gmail.com> wrote:
> >
> > Hi All,
> >
> > Couple of months ago I upgraded the Zeek cluster from 2.5 to 2.6.1
> (compiled with the jemalloc support).
> > I have started seeing increased memory usage by the workers.
> >
> > I have two physical sensors, each running 18 Zeek worker processes LB by
> PF_RING.
> > Not loaded any custom scripts, just the basic scripts that are enabled
> by default in local.bro (also have misc/scan disabled).
> >
> > I just did a top on one of the boxes and here's the output (specially
> two Zeek processes -13632, 13611 using >10% memory which is ~11G)
> > Also, attaching a weekly available free memory graph for the system.
> >
> > Tasks: 455 total,   9 running, 443 sleeping,   0 stopped,   3 zombie
> > %Cpu(s): 18.3 us,  1.7 sy,  0.0 ni, 79.5 id,  0.0 wa,  0.0 hi,  0.4 si,
> 0.0 st
> > KiB Mem : 98783960 total, 32963660 free, 64807572 used,  1012728
> buff/cache
> > KiB Swap:  4194300 total,  3572200 free,   622100 used. 33221356 avail
> Mem
> >
> >   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+
> COMMAND
> > 13589 bro       20   0 3662052   3.4g  73340 R    90.4     3.6
> 1072:47    /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-5 local.bro broctl base/frameworks/cluster
> broctl/auto
> > 13533 bro       20   0 1847972   1.6g  73188 S    50.3     1.7
> 1098:05    /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-6 local.bro broctl base/frameworks/cluster
> broctl/auto
> > 13512 bro       20   0 1291260   1.1g  73052 S    49.7     1.1
> 1080:30    /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-1 local.bro broctl base/frameworks/cluster
> broctl/auto
> > 13628 bro       20   0 2347952   2.1g  73328 R    49.0     2.2
> 1109:31    /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-12 local.bro broctl
> base/frameworks/cluster broctl/auto
> > 13516 bro       20   0  973260 799176  72844 R  47.0     0.8    1036:29
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-3 local.bro broctl base/frameworks/cluster broctl/auto
> > 13539 bro       20   0 6374956   6.0g  73456 S    46.0     6.3
> 1147:08   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-2 local.bro broctl base/frameworks/cluster
> broctl/auto
> > 13591 bro       20   0  865952 726516  73020 S  44.7     0.7    1052:29
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-4 local.bro broctl base/frameworks/cluster broctl/auto
> > 13632 bro       20   0   12.2g  12.0g  73584 R      43.7    12.8
> 1068:17   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-15 local.bro broctl
> base/frameworks/cluster broctl/auto
> > 13540 bro       20   0 2146844   1.9g  73348 R    41.4     2.0
> 1149:38   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-7 local.bro broctl base/frameworks/cluster
> broctl/auto
> > 13611 bro       20   0   17.0g  16.7g  73404 S      39.7    17.8
> 1172:14   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-9 local.bro broctl base/frameworks/cluster
> broctl/auto
> > 13640 bro       20   0 2624300   2.1g  73328 S    39.7     2.3
> 1043:50   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-18 local.bro broctl
> base/frameworks/cluster broctl/auto
> > 13586 bro       20   0 3347044   3.1g  73468 S    39.1     3.2
> 1042:24   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-10 local.bro broctl
> base/frameworks/cluster broctl/auto
> > 13641 bro       20   0 2274788   2.0g  73424 R    39.1     2.2
> 1029:58   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-17 local.bro broctl
> base/frameworks/cluster broctl/auto
> > 13614 bro       20   0 1954780   1.7g  73188 S    38.4     1.8
>  995:00.54   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-13 local.bro broctl
> base/frameworks/cluster broctl/auto
> > 13627 bro       20   0 2756520   2.5g  73288 S    38.4     2.6
>  1035:18   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-14 local.bro broctl
> base/frameworks/cluster broctl/auto
> > 13638 bro       20   0 1206548 853056  72328 R  37.4    0.9   952:10.00
>  /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live
> -p local -p worker-2-16 local.bro broctl base/frameworks/cluster broctl/auto
> > 13623 bro       20   0 8998324   2.1g  73284 S     37.1    2.2
>  1073:31   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-11 local.bro broctl
> base/frameworks/cluster broctl/auto
> > 13575 bro       20   0  871396 706148  73128 R    36.4   0.7
>  1028:30   /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p
> broctl-live -p local -p worker-2-8 local.bro broctl base/frameworks/cluster
> broctl/auto
> > 13336 bro       20   0  266244 133920  33388 S    12.6   0.1
>  400:27.62   /usr/local/bro/2.6.1/bin/bro -U .status -p broctl -p
> broctl-live -p local -p proxy-2 local.bro broctl base/frameworks/cluster
> broctl/auto
> >
> > Any suggestions?
> >
> > Thanks!
> > Fatema
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190906/699cd724/attachment-0001.html 

From greg.grasmehr at caltech.edu  Fri Sep  6 12:03:53 2019
From: greg.grasmehr at caltech.edu (Greg Grasmehr)
Date: Fri, 6 Sep 2019 12:03:53 -0700
Subject: [Zeek] Anyone using 40G Intel NIC and PF_RING ZC
Message-ID: <20190906190353.GB12681@dakine>

Hello,

I'm curious if anyone is utilizing Intel 40G NICs and PF_RING ZC and if
you'd be willing to share your experience?  I'm interested in learning
about which NIC you chose and how well it is working out.

Thanks in advance for any info and Happy Friday!

-- 
Sincerely,

Greg Grasmehr
Lead Information Security Analyst
California Institute of Technology (Caltech)
GPGMe: 38E2 F9BD A95E 9824 20AB  331A 9E29 D1A1 AAEE 5F42
pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x9E29D1A1AAEE5F42

From michalpurzynski1 at gmail.com  Fri Sep  6 13:30:26 2019
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Fri, 6 Sep 2019 13:30:26 -0700
Subject: [Zeek] Anyone using 40G Intel NIC and PF_RING ZC
In-Reply-To: <20190906190353.GB12681@dakine>
References: <20190906190353.GB12681@dakine>
Message-ID: <CAJ6bFK3EksrvDjoio2aMdDi=vMY-sHt7JmvKPmcyq3H_eY1Jsw@mail.gmail.com>

There is no need to use pf_ring on any modern operating Linux version.
Linux has the af_packet packet capture mechanism that does the job really
well.

Our production is based on either Intel's X720 or Mellanox ConnectX-4 Lx
cards (any modern Mellanox will do) and the af_packet in a QM mode. That
means card's hardware can do the filtering (and Mellanox is way more
flexible than Intel here) and symmetric hashing, working with af_packet.
BTW Mellanox is cool in yet another way - they happily work with flexoptics
SFP modules, further saving us $$$.

I'm happy to answer all your questions about this setup - so feel free to
ask. You cannot get me tired talking about this ;)

BTW, a while ago, while working with the Suricata's project developer Peter
Manev we wrote these documents. They are slightly outdated but the basics
they describe haven't changed much.

https://github.com/pevma/SEPTun
https://github.com/pevma/SEPTun-Mark-II <- our production is based on this
one
<https://github.com/pevma/SEPTun-Mark-II>

There will also be a talk (shameless self-promotion mode on) on the Zeek
week where I'll present our setup in details and hopefully answer all
questions people might have.


On Fri, Sep 6, 2019 at 12:12 PM Greg Grasmehr <greg.grasmehr at caltech.edu>
wrote:

> Hello,
>
> I'm curious if anyone is utilizing Intel 40G NICs and PF_RING ZC and if
> you'd be willing to share your experience?  I'm interested in learning
> about which NIC you chose and how well it is working out.
>
> Thanks in advance for any info and Happy Friday!
>
> --
> Sincerely,
>
> Greg Grasmehr
> Lead Information Security Analyst
> California Institute of Technology (Caltech)
> GPGMe: 38E2 F9BD A95E 9824 20AB  331A 9E29 D1A1 AAEE 5F42
> pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x9E29D1A1AAEE5F42
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190906/d9a52ead/attachment.html 

From anandtarun2 at gmail.com  Mon Sep  9 04:12:47 2019
From: anandtarun2 at gmail.com (Tarun Anand)
Date: Mon, 9 Sep 2019 16:42:47 +0530
Subject: [Zeek] Is there any implementation of Zeek/Bro with DPDK
Message-ID: <CAGwKEEngeqZd88GqUS7nRXamfKxbRiK4gYKQdOUL3QFbkTT6yQ@mail.gmail.com>

Hello All

I would like to know if there is any prior/ ongoing work to implement Zeek
on top of DPDK?

Thank You

Regards
Tarun Anand
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190909/eb2b72a6/attachment.html 

From gc355804 at ohio.edu  Mon Sep  9 05:44:03 2019
From: gc355804 at ohio.edu (Clark, Gilbert)
Date: Mon, 9 Sep 2019 12:44:03 +0000
Subject: [Zeek] Is there any implementation of Zeek/Bro with DPDK
In-Reply-To: <CAGwKEEngeqZd88GqUS7nRXamfKxbRiK4gYKQdOUL3QFbkTT6yQ@mail.gmail.com>
References: <CAGwKEEngeqZd88GqUS7nRXamfKxbRiK4gYKQdOUL3QFbkTT6yQ@mail.gmail.com>
Message-ID: <SN6PR01MB42392A712611F80D34C796B8AAB70@SN6PR01MB4239.prod.exchangelabs.com>

Hi,

Might be other efforts out there, but I'll note that I messed with this a (large number of) years ago on a small zeek cluster setup.

I didn't find the results of a straightforward (e.g. rewriting a packet driver) implementation to be terribly encouraging: there was an existing implementation that supported ingest from Netmap and PF_RING which did pretty well already, and the vast majority of zeek's time was spent in script processing anyway.  Thus I found the results to be somewhat ... underwhelming, given the work / likely maintenance effort involved.  A port of PacketBricks [1] might've been an interesting alternative approach, but was outside of the scope of the academic work I was doing at the time.

One thing I did have some success with was using DPDK to implement a very limited version of a sensor in C, and forwarding events from that to the larger zeek cluster through broccoli.  That had utility in cases where a large percentage of the traffic was a specific type (and thus would take a well-known path through zeek script), and the number of events generated was relatively limited in relation to the traffic volume.  This is, however, likely only suited for some pretty niche use-cases.

Good luck,
Gilbert Clark

[1] https://github.com/zeek/packet-bricks
________________________________
From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> on behalf of Tarun Anand <anandtarun2 at gmail.com>
Sent: Monday, September 9, 2019 7:12 AM
To: zeek at zeek.org <zeek at zeek.org>
Subject: [Zeek] Is there any implementation of Zeek/Bro with DPDK

Hello All

I would like to know if there is any prior/ ongoing work to implement Zeek on top of DPDK?

Thank You

Regards
Tarun Anand
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190909/8a4da3e3/attachment.html 

From anandtarun2 at gmail.com  Mon Sep  9 07:49:07 2019
From: anandtarun2 at gmail.com (Tarun Anand)
Date: Mon, 9 Sep 2019 20:19:07 +0530
Subject: [Zeek] Is there any implementation of Zeek/Bro with DPDK
In-Reply-To: <SN6PR01MB42392A712611F80D34C796B8AAB70@SN6PR01MB4239.prod.exchangelabs.com>
References: <CAGwKEEngeqZd88GqUS7nRXamfKxbRiK4gYKQdOUL3QFbkTT6yQ@mail.gmail.com>
	<SN6PR01MB42392A712611F80D34C796B8AAB70@SN6PR01MB4239.prod.exchangelabs.com>
Message-ID: <CAGwKEE=4Q3XKmNdTJtewCXGPfnX4SyfNJgjQ7_6Ei3oR_JY5ag@mail.gmail.com>

Thanks for the feedback!


Regards
Tarun Anand

On Mon, 9 Sep 2019 at 6:14 PM, Clark, Gilbert <gc355804 at ohio.edu> wrote:

> Hi,
>
> Might be other efforts out there, but I'll note that I messed with this a
> (large number of) years ago on a small zeek cluster setup.
>
> I didn't find the results of a straightforward (e.g. rewriting a packet
> driver) implementation to be terribly encouraging: there was an existing
> implementation that supported ingest from Netmap and PF_RING which did
> pretty well already, and the vast majority of zeek's time was spent in
> script processing anyway.  Thus I found the results to be somewhat ...
> underwhelming, given the work / likely maintenance effort involved.  A port
> of PacketBricks [1] might've been an interesting alternative approach, but
> was outside of the scope of the academic work I was doing at the time.
>
> One thing I did have some success with was using DPDK to implement a very
> limited version of a sensor in C, and forwarding events from that to the
> larger zeek cluster through broccoli.  That had utility in cases where a
> large percentage of the traffic was a specific type (and thus would take a
> well-known path through zeek script), and the number of events generated
> was relatively limited in relation to the traffic volume.  This is,
> however, likely only suited for some pretty niche use-cases.
>
> Good luck,
> Gilbert Clark
>
> [1] https://github.com/zeek/packet-bricks
> ------------------------------
> *From:* zeek-bounces at zeek.org <zeek-bounces at zeek.org> on behalf of Tarun
> Anand <anandtarun2 at gmail.com>
> *Sent:* Monday, September 9, 2019 7:12 AM
> *To:* zeek at zeek.org <zeek at zeek.org>
> *Subject:* [Zeek] Is there any implementation of Zeek/Bro with DPDK
>
> Hello All
>
> I would like to know if there is any prior/ ongoing work to implement Zeek
> on top of DPDK?
>
> Thank You
>
> Regards
> Tarun Anand
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190909/bae488b6/attachment.html 

From mauro.palumbo at aizoon.it  Wed Sep 11 03:43:13 2019
From: mauro.palumbo at aizoon.it (Palumbo Mauro)
Date: Wed, 11 Sep 2019 10:43:13 +0000
Subject: [Zeek] Zeek and myricom NICs
Message-ID: <e2dc52c1f6964d2aa8c904f64f793c64@SRVEX03.aizoon.local>

Hi everybody,
  quick question: is the bro-myricon plugin (by Seth) still necessary when using myricom nics with Zeek?  I know with pf_ring this is not the case anymore since bro can be directly linked to a modified pf_ring libpcap and I was wondering if this is the case for myricom too.

Thanks,
Mauro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190911/bafb39c2/attachment.html 

From seth at corelight.com  Wed Sep 11 06:36:22 2019
From: seth at corelight.com (Seth Hall)
Date: Wed, 11 Sep 2019 09:36:22 -0400
Subject: [Zeek] Zeek and myricom NICs
In-Reply-To: <e2dc52c1f6964d2aa8c904f64f793c64@SRVEX03.aizoon.local>
References: <e2dc52c1f6964d2aa8c904f64f793c64@SRVEX03.aizoon.local>
Message-ID: <7C075814-3FA9-4926-840E-D7099F500EB9@corelight.com>


On 11 Sep 2019, at 6:43, Palumbo Mauro wrote:

>   quick question: is the bro-myricon plugin (by Seth) still necessary 
> when using myricom nics with Zeek?  I know with pf_ring this is not 
> the case anymore since bro can be directly linked to a modified 
> pf_ring libpcap and I was wondering if this is the case for myricom 
> too.

There are some advantages to using the Myricom plugin directly.  
Generally in my opinion I've been trying to avoid libpcap wrappers for 
quite a few years now because of various quality issues associated with 
several of them that I've experienced.  There tends to be API 
functionality that you don't have an opportunity to take advantage of 
with a pcap wrapper too.

To some degree this is personal preference though.

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com

From mauro.palumbo at aizoon.it  Wed Sep 11 06:52:19 2019
From: mauro.palumbo at aizoon.it (Palumbo Mauro)
Date: Wed, 11 Sep 2019 13:52:19 +0000
Subject: [Zeek] R:  Zeek and myricom NICs
In-Reply-To: <7C075814-3FA9-4926-840E-D7099F500EB9@corelight.com>
References: <e2dc52c1f6964d2aa8c904f64f793c64@SRVEX03.aizoon.local>
	<7C075814-3FA9-4926-840E-D7099F500EB9@corelight.com>
Message-ID: <0aa60503b4724dbba45bbfb38c674bfb@SRVEX03.aizoon.local>

Hi Seth,
   thanks for your prompt reply.

Looking at the myricom software API, I see that they have both a libpcap wrapper and more advanced functionalities in snf.h. Not all source code is open, however, and I am not sure which functionalities are implemented in the libpcap wrappers.  In your plugin you are using snf_open to open the NIC device. I would like to open a Myricom NIC with both aggregation and load_balancing, i.e. 

  int flags = SNF_F_PSHARED;
  flags |= SNF_F_AGGREGATE_PORTMASK;

  struct snf_rss_params rssp;
  rssp.mode = SNF_RSS_FLAGS;
  rssp.params.rss_flags = SNF_RSS_IP | SNF_RSS_SRC_PORT | SNF_RSS_DST_PORT;

   rc = snf_open(portnum, 2, &rssp, dataring_sz, flags, &hsnf);

but I am not sure this works. Did someone ever try it? Myricom documentation is a bit ambiguous on this point...

Mauro


-----Messaggio originale-----
Da: Seth Hall [mailto:seth at corelight.com] 
Inviato: mercoled? 11 settembre 2019 15:36
A: Palumbo Mauro <mauro.palumbo at aizoon.it>
Cc: zeek at zeek.org
Oggetto: Re: [Zeek] Zeek and myricom NICs


On 11 Sep 2019, at 6:43, Palumbo Mauro wrote:

>   quick question: is the bro-myricon plugin (by Seth) still necessary 
> when using myricom nics with Zeek?  I know with pf_ring this is not 
> the case anymore since bro can be directly linked to a modified 
> pf_ring libpcap and I was wondering if this is the case for myricom 
> too.

There are some advantages to using the Myricom plugin directly.  
Generally in my opinion I've been trying to avoid libpcap wrappers for quite a few years now because of various quality issues associated with several of them that I've experienced.  There tends to be API functionality that you don't have an opportunity to take advantage of with a pcap wrapper too.

To some degree this is personal preference though.

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com


From justin at corelight.com  Wed Sep 11 07:03:49 2019
From: justin at corelight.com (Justin Azoff)
Date: Wed, 11 Sep 2019 10:03:49 -0400
Subject: [Zeek] Zeek and myricom NICs
In-Reply-To: <7C075814-3FA9-4926-840E-D7099F500EB9@corelight.com>
References: <e2dc52c1f6964d2aa8c904f64f793c64@SRVEX03.aizoon.local>
	<7C075814-3FA9-4926-840E-D7099F500EB9@corelight.com>
Message-ID: <CAPfnCug4u70N9Jf8250xi344ZOdz5nvEv17UU_g1dYo=6PBBXw@mail.gmail.com>

On Wed, Sep 11, 2019 at 9:38 AM Seth Hall <seth at corelight.com> wrote:

There are some advantages to using the Myricom plugin directly.
> Generally in my opinion I've been trying to avoid libpcap wrappers for
> quite a few years now because of various quality issues associated with
> several of them that I've experienced.
>

A few years ago I found a bug in the snfv3 shipped libpcap where pcap_next
would return the previous packet when no packets were available instead of
returning NULL.  As far as I know it's still not fixed.

-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190911/3c73a3c8/attachment.html 

From seth at corelight.com  Wed Sep 11 13:37:14 2019
From: seth at corelight.com (Seth Hall)
Date: Wed, 11 Sep 2019 16:37:14 -0400
Subject: [Zeek] Zeek and myricom NICs
In-Reply-To: <CAPfnCug4u70N9Jf8250xi344ZOdz5nvEv17UU_g1dYo=6PBBXw@mail.gmail.com>
References: <e2dc52c1f6964d2aa8c904f64f793c64@SRVEX03.aizoon.local>
	<7C075814-3FA9-4926-840E-D7099F500EB9@corelight.com>
	<CAPfnCug4u70N9Jf8250xi344ZOdz5nvEv17UU_g1dYo=6PBBXw@mail.gmail.com>
Message-ID: <BEBBB0B4-FC1C-4326-852F-29EEFB15543C@corelight.com>



On 11 Sep 2019, at 10:03, Justin Azoff wrote:

> A few years ago I found a bug in the snfv3 shipped libpcap 
> where?pcap_next would return the previous packet when no packets were 
> available instead of returning NULL.? As far as I know it's still not 
> fixed.

Hah!  I feel like I've seen little problems in every libpcap wrapper 
I've ever worked with.  Never the same problem. :)

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com

From mauro.palumbo at aizoon.it  Thu Sep 12 00:21:59 2019
From: mauro.palumbo at aizoon.it (Palumbo Mauro)
Date: Thu, 12 Sep 2019 07:21:59 +0000
Subject: [Zeek] R:  Zeek and myricom NICs
In-Reply-To: <BEBBB0B4-FC1C-4326-852F-29EEFB15543C@corelight.com>
References: <e2dc52c1f6964d2aa8c904f64f793c64@SRVEX03.aizoon.local>
	<7C075814-3FA9-4926-840E-D7099F500EB9@corelight.com>
	<CAPfnCug4u70N9Jf8250xi344ZOdz5nvEv17UU_g1dYo=6PBBXw@mail.gmail.com>
	<BEBBB0B4-FC1C-4326-852F-29EEFB15543C@corelight.com>
Message-ID: <9b9d8d9894d444279ad3e1364bb5b584@SRVEX03.aizoon.local>

Is anyone aware of other bugs in libpcap? I think this is valuable information to share to the community...

Mauro

-----Messaggio originale-----
Da: Seth Hall [mailto:seth at corelight.com] 
Inviato: mercoled? 11 settembre 2019 22:37
A: Justin Azoff <justin at corelight.com>
Cc: Palumbo Mauro <mauro.palumbo at aizoon.it>; zeek <zeek at zeek.org>
Oggetto: Re: [Zeek] Zeek and myricom NICs



On 11 Sep 2019, at 10:03, Justin Azoff wrote:

> A few years ago I found a bug in the snfv3 shipped libpcap where?
> pcap_next would return the previous packet when no packets were 
> available instead of returning NULL.? As far as I know it's still not 
> fixed.

Hah!  I feel like I've seen little problems in every libpcap wrapper I've ever worked with.  Never the same problem. :)

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com


From michalpurzynski1 at gmail.com  Thu Sep 12 00:41:19 2019
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Thu, 12 Sep 2019 00:41:19 -0700
Subject: [Zeek] R:  Zeek and myricom NICs
In-Reply-To: <9b9d8d9894d444279ad3e1364bb5b584@SRVEX03.aizoon.local>
References: <e2dc52c1f6964d2aa8c904f64f793c64@SRVEX03.aizoon.local>
	<7C075814-3FA9-4926-840E-D7099F500EB9@corelight.com>
	<CAPfnCug4u70N9Jf8250xi344ZOdz5nvEv17UU_g1dYo=6PBBXw@mail.gmail.com>
	<BEBBB0B4-FC1C-4326-852F-29EEFB15543C@corelight.com>
	<9b9d8d9894d444279ad3e1364bb5b584@SRVEX03.aizoon.local>
Message-ID: <C700CFDB-20BB-4789-B326-695B6B511E18@gmail.com>

Libpcap also makes (with some capture technologies) two calls per packet - one to get the packet and another to get the time stamp. That kills the performance.

When I was developing the early version of the myricom Zeek plugin, I didn?t really notice much, of any, performance improvement.

Btw you can use upstream libpcap and build it yourself against SNF. But why. Get the plugin. It?s easier.

If there are some missing pieces in the plugin I ought be able to help. We not longer have myricoms in production but I keep them in stage servers, for the community ;)

> On Sep 12, 2019, at 12:21 AM, Palumbo Mauro <mauro.palumbo at aizoon.it> wrote:
> 
> Is anyone aware of other bugs in libpcap? I think this is valuable information to share to the community...
> 
> Mauro
> 
> -----Messaggio originale-----
> Da: Seth Hall [mailto:seth at corelight.com] 
> Inviato: mercoled? 11 settembre 2019 22:37
> A: Justin Azoff <justin at corelight.com>
> Cc: Palumbo Mauro <mauro.palumbo at aizoon.it>; zeek <zeek at zeek.org>
> Oggetto: Re: [Zeek] Zeek and myricom NICs
> 
> 
> 
>> On 11 Sep 2019, at 10:03, Justin Azoff wrote:
>> 
>> A few years ago I found a bug in the snfv3 shipped libpcap where 
>> pcap_next would return the previous packet when no packets were 
>> available instead of returning NULL.  As far as I know it's still not 
>> fixed.
> 
> Hah!  I feel like I've seen little problems in every libpcap wrapper I've ever worked with.  Never the same problem. :)
> 
>   .Seth
> 
> --
> Seth Hall * Corelight, Inc * www.corelight.com
> 
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From seth at corelight.com  Thu Sep 12 03:47:45 2019
From: seth at corelight.com (Seth Hall)
Date: Thu, 12 Sep 2019 06:47:45 -0400
Subject: [Zeek] Zeek and myricom NICs
In-Reply-To: <9b9d8d9894d444279ad3e1364bb5b584@SRVEX03.aizoon.local>
References: <e2dc52c1f6964d2aa8c904f64f793c64@SRVEX03.aizoon.local>
	<7C075814-3FA9-4926-840E-D7099F500EB9@corelight.com>
	<CAPfnCug4u70N9Jf8250xi344ZOdz5nvEv17UU_g1dYo=6PBBXw@mail.gmail.com>
	<BEBBB0B4-FC1C-4326-852F-29EEFB15543C@corelight.com>
	<9b9d8d9894d444279ad3e1364bb5b584@SRVEX03.aizoon.local>
Message-ID: <1AF4D451-E9D1-45E3-9CC2-F031A7E91E24@corelight.com>



On 12 Sep 2019, at 3:21, Palumbo Mauro wrote:

> Is anyone aware of other bugs in libpcap? I think this is valuable 
> information to share to the community...

I wasn't referring to bugs in libpcap.  It's the libpcap wrappers (which 
typically aren't libpcap, but rather reimplementations of some or all of 
the libpcap api).

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com

From mauro.palumbo at aizoon.it  Thu Sep 12 04:57:00 2019
From: mauro.palumbo at aizoon.it (Palumbo Mauro)
Date: Thu, 12 Sep 2019 11:57:00 +0000
Subject: [Zeek] R:  Zeek and myricom NICs
In-Reply-To: <1AF4D451-E9D1-45E3-9CC2-F031A7E91E24@corelight.com>
References: <e2dc52c1f6964d2aa8c904f64f793c64@SRVEX03.aizoon.local>
	<7C075814-3FA9-4926-840E-D7099F500EB9@corelight.com>
	<CAPfnCug4u70N9Jf8250xi344ZOdz5nvEv17UU_g1dYo=6PBBXw@mail.gmail.com>
	<BEBBB0B4-FC1C-4326-852F-29EEFB15543C@corelight.com>
	<9b9d8d9894d444279ad3e1364bb5b584@SRVEX03.aizoon.local>
	<1AF4D451-E9D1-45E3-9CC2-F031A7E91E24@corelight.com>
Message-ID: <ae09282f9efc4872b8d68592e09be2a3@SRVEX03.aizoon.local>

Sure, sorry I wrote too quickly. What I meant is if someone is aware of other bugs in other libpcap wrappers.

Mauro

-----Messaggio originale-----
Da: Seth Hall [mailto:seth at corelight.com] 
Inviato: gioved? 12 settembre 2019 12:48
A: Palumbo Mauro <mauro.palumbo at aizoon.it>
Cc: Justin Azoff <justin at corelight.com>; zeek <zeek at zeek.org>
Oggetto: Re: [Zeek] Zeek and myricom NICs



On 12 Sep 2019, at 3:21, Palumbo Mauro wrote:

> Is anyone aware of other bugs in libpcap? I think this is valuable 
> information to share to the community...

I wasn't referring to bugs in libpcap.  It's the libpcap wrappers (which typically aren't libpcap, but rather reimplementations of some or all of the libpcap api).

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com


From akgraner at corelight.com  Thu Sep 12 08:48:32 2019
From: akgraner at corelight.com (Amber Graner)
Date: Thu, 12 Sep 2019 10:48:32 -0500
Subject: [Zeek] ZeekWeek 19 - The leading event for open-source Zeek network
 security monitor comes to Seattle
Message-ID: <CAJhOzuoqv2zQ811OgdA5FNrQXrcD3EsctSHjXtdZKAjtuY_bLA@mail.gmail.com>

Zeek Week to Gather Expert Users and Developers from Around the World to
Showcase New Zeek Technology Innovations and Enhancements -
https://blog.zeek.org/2019/09/zeek-week-to-gather-expert-users-and.html

Discount Registration has been extended in midnight PT on Sunday 15 Sept
2019 - Register today.

Training slots still available for 8 October, 2019.

Check out ZeekWeek.com for more information - https://www.zeekweek.com

Please let me know if you have any questions.

Thanks,
~Amber


-- 

*Amber Graner*
Director of Community
Corelight, Inc

828.582.9469


 * Ask me about how you can participate in the Zeek (formerly Bro)
community.
 * Remember - ZEEK AND YOU SHALL FIND!!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190912/7b949677/attachment.html 

From utoncyr at gmail.com  Thu Sep 12 12:40:12 2019
From: utoncyr at gmail.com (Uton Cyr)
Date: Thu, 12 Sep 2019 21:40:12 +0200
Subject: [Zeek] Zeek - Usecase based File Extraction
Message-ID: <CAHJMt6pqsu1jnDsjWGBKHsnDGT+g8o1qdm0FBgWPRKpYP3nOtQ@mail.gmail.com>

Hi all,

I've recently been working on file carving/extraction based on a few
usecases.

Namely:
During a match with the Intel Framework on a FILE_HASH, I want to extract
the file.
During a match with the Intel Framework on a DOMAIN and ADDR, I want to
extract the file.

*See code below.*

Yet everytime I'll get the error message:
*Analyzer Files::ANALYZER_EXTRACT not added successfully to file ......*

This occurs when you try to extract from the event: file_hash.
However, within events such as file_new and file_sniff, files can be
extracted.
I'd like to hand over the hash within the event of file_hash to
Intel::Seen($indicator=hash)

A few questions:
- Is it possible to extract a file during an Intel::match event?
- If yes, how would I go about this?
- Is there a simple way to hand over the hash, originating tx_host and
domain to the Intel framework and extract a file after a match?

Looking forward to your reply.

Kind regards,
Bart

{CODE}

@load base/frameworks/intel
@load base/files/extract

##Redefine to path desired.
global path = "/home/zintern/EXTRACTED/temp/";

##Redefine to desired IoC .dat file
redef Intel::read_files += {fmt("%s/otx.dat", @DIR)};

## When a new file is seen:
event file_new(f: fa_file)
{
        Files::add_analyzer(f, Files::ANALYZER_MD5);
        Files::add_analyzer(f, Files::ANALYZER_SHA1);
        Files::add_analyzer(f, Files::ANALYZER_SHA256);
}

## When a file_hash has been seen
event file_hash(f: fa_file, kind: string, hash: string)
        {
        local seen = Intel::Seen($indicator=hash,
                                 $indicator_type=Intel::FILE_HASH,
                                 $f=f,
                                 $where=Files::IN_HASH);

        Intel::seen(seen);
        }

## When a match has been found between the seen traffic and the otx.dat
file indicators.
event Intel::match(s: Intel::Seen, items:set[Intel::Item])
{
                if(s$indicator_type == Intel::FILE_HASH)
                {
                local fname = fmt("%s%s-%s", path, s$f$source, s$f$id);
                Files::add_analyzer(s$f,
Files::ANALYZER_EXTRACT,[$extract_filename = fname]);
                }

}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190912/93e65edb/attachment.html 

From jan.grashoefer at gmail.com  Fri Sep 13 08:54:59 2019
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Fri, 13 Sep 2019 17:54:59 +0200
Subject: [Zeek] Zeek - Usecase based File Extraction
In-Reply-To: <CAHJMt6pqsu1jnDsjWGBKHsnDGT+g8o1qdm0FBgWPRKpYP3nOtQ@mail.gmail.com>
References: <CAHJMt6pqsu1jnDsjWGBKHsnDGT+g8o1qdm0FBgWPRKpYP3nOtQ@mail.gmail.com>
Message-ID: <cd260c68-0458-e56c-445d-d5c88a6a4c98@gmail.com>

Hi Bart,

On 12/09/2019 21:40, Uton Cyr wrote:
> A few questions:
> - Is it possible to extract a file during an Intel::match event?
> ...

usually the match is too late to attach the file analyzer that handles 
extraction. Furthermore, in a cluster setup its triggered on the 
manager. The simplest way to get files for intel hits is to extract all 
files and just preserve the ones that triggered a hit (for the poor 
man's approach see 
https://github.com/J-Gras/intel-extensions/blob/master/scripts/preserve_files.bro).

Jan

From utoncyr at gmail.com  Sat Sep 14 08:36:38 2019
From: utoncyr at gmail.com (Uton Cyr)
Date: Sat, 14 Sep 2019 17:36:38 +0200
Subject: [Zeek] Zeek - Usecase based File Extraction
In-Reply-To: <cd260c68-0458-e56c-445d-d5c88a6a4c98@gmail.com>
References: <CAHJMt6pqsu1jnDsjWGBKHsnDGT+g8o1qdm0FBgWPRKpYP3nOtQ@mail.gmail.com>
	<cd260c68-0458-e56c-445d-d5c88a6a4c98@gmail.com>
Message-ID: <CAHJMt6rmE-a6_vYXpidYjZw2iT8FZj9sBbGcz9MUGxNV-PQL8g@mail.gmail.com>

Hi Jan,

Thank you for the clarification!
I should've known a file cannot be extracted "after" the hash of the file
has been calculated.
To calculate the hash of a file in the first place you'd need to analyse
the file in its entirety.
Meaning after the hash has been analysed of the file it's likely at the END
bit of the data stream.

The partial solution to extract first and verify later might be overkill on
a network where thousands of files are downloaded.
Restricting it to particular data protocols such as HTTP 'only' will have
less of an impact on the computational load.
I'll have to try your suggested method, thank you for the link!

I was wondering if the usecase of extracting after getting an intel hit on
INTEL::DOMAIN and INTEL::ADDR might still work.
My assumption here is that the time between the event file_new and
intel::match might be small enough to not make a difference.
As long as the function Intel::seen is called immediately during a file_new
event (this might cause some dataloss).

I have a one more questions if you or anyone has time:
- I'd like to compare the tx_hosts seen of a file with the INTEL::ADDR, how
would I go about this? (since tx_hosts is a set (still learning bro)).

Kind regards,
Bart


Op vr 13 sep. 2019 om 18:03 schreef Jan Grash?fer <jan.grashoefer at gmail.com
>:

> Hi Bart,
>
> On 12/09/2019 21:40, Uton Cyr wrote:
> > A few questions:
> > - Is it possible to extract a file during an Intel::match event?
> > ...
>
> usually the match is too late to attach the file analyzer that handles
> extraction. Furthermore, in a cluster setup its triggered on the
> manager. The simplest way to get files for intel hits is to extract all
> files and just preserve the ones that triggered a hit (for the poor
> man's approach see
>
> https://github.com/J-Gras/intel-extensions/blob/master/scripts/preserve_files.bro
> ).
>
> Jan
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190914/6ef9bf70/attachment.html 

From seth at corelight.com  Mon Sep 16 02:01:59 2019
From: seth at corelight.com (Seth Hall)
Date: Mon, 16 Sep 2019 05:01:59 -0400
Subject: [Zeek] Zeek and myricom NICs
In-Reply-To: <ae09282f9efc4872b8d68592e09be2a3@SRVEX03.aizoon.local>
References: <e2dc52c1f6964d2aa8c904f64f793c64@SRVEX03.aizoon.local>
	<7C075814-3FA9-4926-840E-D7099F500EB9@corelight.com>
	<CAPfnCug4u70N9Jf8250xi344ZOdz5nvEv17UU_g1dYo=6PBBXw@mail.gmail.com>
	<BEBBB0B4-FC1C-4326-852F-29EEFB15543C@corelight.com>
	<9b9d8d9894d444279ad3e1364bb5b584@SRVEX03.aizoon.local>
	<1AF4D451-E9D1-45E3-9CC2-F031A7E91E24@corelight.com>
	<ae09282f9efc4872b8d68592e09be2a3@SRVEX03.aizoon.local>
Message-ID: <F84D3CE2-18B0-4F78-8003-78C19630F956@corelight.com>

Ahh, it's a little hard to track these bugs because they tend to come 
and go without much documentation because vendors will just fix them in 
their SDKs and not make a big deal about it.

   .Seth

On 12 Sep 2019, at 7:57, Palumbo Mauro wrote:

> Sure, sorry I wrote too quickly. What I meant is if someone is aware 
> of other bugs in other libpcap wrappers.
>
> Mauro
>
> -----Messaggio originale-----
> Da: Seth Hall [mailto:seth at corelight.com]
> Inviato: gioved? 12 settembre 2019 12:48
> A: Palumbo Mauro <mauro.palumbo at aizoon.it>
> Cc: Justin Azoff <justin at corelight.com>; zeek <zeek at zeek.org>
> Oggetto: Re: [Zeek] Zeek and myricom NICs
>
>
>
> On 12 Sep 2019, at 3:21, Palumbo Mauro wrote:
>
>> Is anyone aware of other bugs in libpcap? I think this is valuable
>> information to share to the community...
>
> I wasn't referring to bugs in libpcap.  It's the libpcap wrappers 
> (which typically aren't libpcap, but rather reimplementations of some 
> or all of the libpcap api).
>
>    .Seth
>
> --
> Seth Hall * Corelight, Inc * www.corelight.com

--
Seth Hall * Corelight, Inc * www.corelight.com

From jsiwek at corelight.com  Mon Sep 16 11:13:22 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Mon, 16 Sep 2019 11:13:22 -0700
Subject: [Zeek] Zeek 3.0.0 RC2 available
In-Reply-To: <CAMzgZ0Lzv4PJj9mAcHYVFawu2WD10VefyAYabNEW4SE5ufP25Q@mail.gmail.com>
References: <CAMzgZ0Lzv4PJj9mAcHYVFawu2WD10VefyAYabNEW4SE5ufP25Q@mail.gmail.com>
Message-ID: <CAMzgZ0KNA+Nq6DHgt8VXv6MDeuz9rLismgFA9U-Rm+Fp2QzRZQ@mail.gmail.com>

Hi everyone, we're planning to release final Zeek 3.0.0 next week on
9/23, so please take the opportunity to report any bugs/issues you've
encountered.  Afterward, the Bro 2.6.x release series won't see
significant maintenance or updates.

- Jon

On Wed, Aug 28, 2019 at 5:17 PM Jon Siwek <jsiwek at corelight.com> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Release Candidate 2 for Zeek 3.0.0 is now available for
> testing:
>
>   https://www.zeek.org/downloads/zeek-3.0.0-rc2.tar.gz
>   https://www.zeek.org/downloads/zeek-3.0.0-rc2.tar.gz.asc
>
> See the CHANGES file for a list of changes since RC1.
>
> This major release will have many additions and changes, the
> most prominent being a comprehensive adaptation to use Zeek
> instead of Bro.  See the NEWS file for the full list of
> important differences to be aware of when upgrading and testing.
> Our blog also describes the upcoming release and potential
> issues when upgrading:
>
>   https://blog.zeek.org
>
> Please report bugs at our GitHub project:
>
>   https://github.com/zeek/zeek/issues
>
> Or feel free to give feedback directly on the Zeek mailing list.
>
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBAgAdFiEE6WkLK32KwaGfkhxKxotJTfVqzH4FAl1nE94ACgkQxotJTfVq
> zH4i9g//bPeNtIckaFzdRJqevLqqj4H4TU6CWPnKaELVV0GljFcmuFzfMA4W35yx
> xFDUKOxUcIgLoQ79mhOm2x3VOaSpKmSz/8BXII5fvSshQ70CkNeTfOr79SQZ+Lvb
> wPTmq96y2UxSanPH4NanUO7AnI3o7rw9Fu8QGB0MgE0a9Cn1iPaE4dBA4ivAjrI1
> JhLqMcuA7hLYwJSkPG3XjJTLumtELsiXxL8LLmbCKQDPYLm6gLSMTKq4p9n8+zo0
> GJ/ltwPwmsSYcgmhiifEcVns/HpU7qLEI4uP5XnHQ5Fcgvmu7BPvxA5eV6ZwafxP
> 5u2rYiPyC6n5qOOiS/mvMP0Y39H8XDC2Oa6TJ+xy0fC5BHYPCBhRcBNlz31Fp8UR
> 2k1AMAMh+9pSEBz5c7F18H38zblt+swxbp/wN7D+Mg4gwX0qMP1ZUwuGzcYiT5mf
> Of5rUh2kZa1emrjBMqBe85hpd2Yfn6kvSjwqVeoYoMqgMBb3yhmQPH/itqBq/T1M
> G9ULuLB8rYRGvwD5DEnPRqzaXP8T0GGAP+1WNTEZxIL8vD6Ksw/oon1h+odTCtT8
> zu68Jl/2nDCk7Y6kiHr6x5cOVOT0yEPvc5JlRgb9ZWWuWvvqujJI8aHqzLwiz9Wo
> XYKwpgroPGijax95pr8Y7Jzgqmcm66GPyBnRaNWXg2bohGOMycg=
> =8xcQ
> -----END PGP SIGNATURE-----

From jwc3f at virginia.edu  Mon Sep 16 11:14:32 2019
From: jwc3f at virginia.edu (Collyer, Jeffrey W (jwc3f))
Date: Mon, 16 Sep 2019 18:14:32 +0000
Subject: [Zeek] SSH auth_success state true set, but admin claims no logins
Message-ID: <C34FDF4E-8E46-4746-98E0-AB15720B2057@virginia.edu>

So recently I saw an SSH login to a device from outside the US.  I reported it to the end system admin.  The Zeek log set the auth_success state to true, but the admin of the box claims no successful login and is pushing back that it is a false positive.  

Have other Zeek users ever seen this?  Is the SSH auth state detection mistaken here?
I don?t have pcaps to verify one way to the other, sadly.


{"_path":"ssh","_system_name":"corelight","_write_ts":"2019-09-12T22:26:32.106142Z","ts":"2019-09-12T22:26:31.226136Z","uid":"C95i0o2Jl77LXHb2R9","id.orig_h?:?x.x.x.x","id.orig_p":49670,"id.resp_h?:?x.x.x.x","id.resp_p":22,"version":2,"auth_success":true,"auth_attempts":1,"direction":"INBOUND","client":"SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4","server":"SSH-2.0-OpenSSH_7.4","cipher_alg":"chacha20-poly1305 at openssh.com","mac_alg":"umac-64-etm at openssh.com","compression_alg":"none","kex_alg":"curve25519-sha256 at libssh.org","host_key_alg":"ecdsa-sha2-nistp256","host_key":"68:1e:68:89:5e:e5:20:72:f7:e6:bf:21:de:07:3a:b1?}

Can anyone shed light on this?

Thanks
Jeff


Jeffrey Collyer
Information Security Engineer
University of Virginia
jwc3f at virginia.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190916/83f26948/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2436 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190916/83f26948/attachment.bin 

From tet68mt at gmail.com  Mon Sep 16 12:05:22 2019
From: tet68mt at gmail.com (Matt Trostel)
Date: Mon, 16 Sep 2019 14:05:22 -0500
Subject: [Zeek] SSH auth_success state true set,
	but admin claims no logins
In-Reply-To: <C34FDF4E-8E46-4746-98E0-AB15720B2057@virginia.edu>
References: <C34FDF4E-8E46-4746-98E0-AB15720B2057@virginia.edu>
Message-ID: <B76A3454-4C3E-494F-8745-8061C587E278@gmail.com>

We have seen this same behavior. I didn?t do much digging on it at the time, but will keep an eye out for the next occurrence.

-- Matt Trostel

> On Sep 16, 2019, at 13:14, Collyer, Jeffrey W (jwc3f) <jwc3f at virginia.edu> wrote:
> 
> So recently I saw an SSH login to a device from outside the US.  I reported it to the end system admin.  The Zeek log set the auth_success state to true, but the admin of the box claims no successful login and is pushing back that it is a false positive.  
> 
> Have other Zeek users ever seen this?  Is the SSH auth state detection mistaken here?
> I don?t have pcaps to verify one way to the other, sadly.
> 
> 
> {"_path":"ssh","_system_name":"corelight","_write_ts":"2019-09-12T22:26:32.106142Z","ts":"2019-09-12T22:26:31.226136Z","uid":"C95i0o2Jl77LXHb2R9","id.orig_h?:?x.x.x.x","id.orig_p":49670,"id.resp_h?:?x.x.x.x","id.resp_p":22,"version":2,"auth_success":true,"auth_attempts":1,"direction":"INBOUND","client":"SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4","server":"SSH-2.0-OpenSSH_7.4","cipher_alg":"chacha20-poly1305 at openssh.com","mac_alg":"umac-64-etm at openssh.com","compression_alg":"none","kex_alg":"curve25519-sha256 at libssh.org","host_key_alg":"ecdsa-sha2-nistp256","host_key":"68:1e:68:89:5e:e5:20:72:f7:e6:bf:21:de:07:3a:b1?}
> 
> Can anyone shed light on this?
> 
> Thanks
> Jeff
> 
> 
> Jeffrey Collyer
> Information Security Engineer
> University of Virginia
> jwc3f at virginia.edu
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190916/7b549e47/attachment.html 

From jmellander at lbl.gov  Mon Sep 16 12:08:21 2019
From: jmellander at lbl.gov (Jim Mellander)
Date: Mon, 16 Sep 2019 12:08:21 -0700
Subject: [Zeek] SSH auth_success state true set,
	but admin claims no logins
In-Reply-To: <C34FDF4E-8E46-4746-98E0-AB15720B2057@virginia.edu>
References: <C34FDF4E-8E46-4746-98E0-AB15720B2057@virginia.edu>
Message-ID: <CADju=b7huukzXZenC1a=5S6y5tDk5aD7db+1XFG_YF2bZxFVvQ@mail.gmail.com>

Since Zeek only sees the encrypted traffic of an ssh session, it can only
make a best-guess based on packet-size analysis, which is not necessarily
going to be 100% accurate.

On Mon, Sep 16, 2019 at 11:24 AM Collyer, Jeffrey W (jwc3f) <
jwc3f at virginia.edu> wrote:

> So recently I saw an SSH login to a device from outside the US.  I
> reported it to the end system admin.  The Zeek log set the auth_success
> state to true, but the admin of the box claims no successful login and is
> pushing back that it is a false positive.
>
> Have other Zeek users ever seen this?  Is the SSH auth state detection
> mistaken here?
> I don?t have pcaps to verify one way to the other, sadly.
>
>
> {"_path":"ssh","_system_name":"corelight","_write_ts":"2019-09-12T22:26:32.106142Z","ts":"2019-09-12T22:26:31.226136Z","uid":"C95i0o2Jl77LXHb2R9","id.orig_h?:?x.x.x.x","id.orig_p":49670,"id.resp_h?:?x.x.x.x","id.resp_p":22,"version":2,"auth_success":true,"auth_attempts":1,"direction":"INBOUND","client":"SSH-2.0-OpenSSH_7.2p2
> Ubuntu-4ubuntu2.4","server":"SSH-2.0-OpenSSH_7.4","cipher_alg":"
> chacha20-poly1305 at openssh.com","mac_alg":"umac-64-etm at openssh.com
> ","compression_alg":"none","kex_alg":"curve25519-sha256 at libssh.org
> ","host_key_alg":"ecdsa-sha2-nistp256","host_key":"68:1e:68:89:5e:e5:20:72:f7:e6:bf:21:de:07:3a:b1?}
>
> Can anyone shed light on this?
>
> Thanks
> Jeff
>
>
> Jeffrey Collyer
> Information Security Engineer
> University of Virginia
> jwc3f at virginia.edu
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190916/44786cb8/attachment.html 

From jlay at slave-tothe-box.net  Mon Sep 16 14:04:20 2019
From: jlay at slave-tothe-box.net (James Lay)
Date: Mon, 16 Sep 2019 15:04:20 -0600
Subject: [Zeek] 2.6.4 to 3.0.0
Message-ID: <32d5e6cf3092dbddce9da1a49279e3c5@slave-tothe-box.net>

So to mentally prepare myself here....is there a list of breaking 
changes that we'll need to address before attempting an install on this? 
  I have several scripts that I really don't want to see go boom.  Thank 
you!

James

From vlad at es.net  Mon Sep 16 14:39:54 2019
From: vlad at es.net (Vlad Grigorescu)
Date: Mon, 16 Sep 2019 21:39:54 +0000
Subject: [Zeek] SSH auth_success state true set,
	but admin claims no logins
In-Reply-To: <CADju=b7huukzXZenC1a=5S6y5tDk5aD7db+1XFG_YF2bZxFVvQ@mail.gmail.com>
References: <C34FDF4E-8E46-4746-98E0-AB15720B2057@virginia.edu>
	<CADju=b7huukzXZenC1a=5S6y5tDk5aD7db+1XFG_YF2bZxFVvQ@mail.gmail.com>
Message-ID: <CAPqbkwuF+mWrYQriqbv5D8W3+ECBPOLj2DnS3V8X75n-hG-ovA@mail.gmail.com>

Hi Jeffrey,

The SSH detection /should/ be fairly solid. I really tried to err on the
side of caution, and to not make a determination if there was some room for
doubt.

I haven't heard any reports about what specifically might cause a false
positive, but I would guess: some uncommon SSH option (e.g. a large
banner?) or some aggressive TCP settings.

If you can duplicate this by trying to login against this server, and could
share an anonymized PCAP, I'll work updating the analyzer.

Thanks,

  --Vlad

On Mon, Sep 16, 2019 at 7:17 PM Jim Mellander <jmellander at lbl.gov> wrote:

> Since Zeek only sees the encrypted traffic of an ssh session, it can only
> make a best-guess based on packet-size analysis, which is not necessarily
> going to be 100% accurate.
>
> On Mon, Sep 16, 2019 at 11:24 AM Collyer, Jeffrey W (jwc3f) <
> jwc3f at virginia.edu> wrote:
>
>> So recently I saw an SSH login to a device from outside the US.  I
>> reported it to the end system admin.  The Zeek log set the auth_success
>> state to true, but the admin of the box claims no successful login and is
>> pushing back that it is a false positive.
>>
>> Have other Zeek users ever seen this?  Is the SSH auth state detection
>> mistaken here?
>> I don?t have pcaps to verify one way to the other, sadly.
>>
>>
>> {"_path":"ssh","_system_name":"corelight","_write_ts":"2019-09-12T22:26:32.106142Z","ts":"2019-09-12T22:26:31.226136Z","uid":"C95i0o2Jl77LXHb2R9","id.orig_h?:?x.x.x.x","id.orig_p":49670,"id.resp_h?:?x.x.x.x","id.resp_p":22,"version":2,"auth_success":true,"auth_attempts":1,"direction":"INBOUND","client":"SSH-2.0-OpenSSH_7.2p2
>> Ubuntu-4ubuntu2.4","server":"SSH-2.0-OpenSSH_7.4","cipher_alg":"
>> chacha20-poly1305 at openssh.com","mac_alg":"umac-64-etm at openssh.com
>> ","compression_alg":"none","kex_alg":"curve25519-sha256 at libssh.org
>> ","host_key_alg":"ecdsa-sha2-nistp256","host_key":"68:1e:68:89:5e:e5:20:72:f7:e6:bf:21:de:07:3a:b1?}
>>
>> Can anyone shed light on this?
>>
>> Thanks
>> Jeff
>>
>>
>> Jeffrey Collyer
>> Information Security Engineer
>> University of Virginia
>> jwc3f at virginia.edu
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190916/d5410c1c/attachment-0001.html 

From dopheide at gmail.com  Mon Sep 16 14:44:15 2019
From: dopheide at gmail.com (Mike Dopheide)
Date: Mon, 16 Sep 2019 16:44:15 -0500
Subject: [Zeek] 2.6.4 to 3.0.0
In-Reply-To: <32d5e6cf3092dbddce9da1a49279e3c5@slave-tothe-box.net>
References: <32d5e6cf3092dbddce9da1a49279e3c5@slave-tothe-box.net>
Message-ID: <CAPy2kFZRc9KxTQdn0AfBnRw7ZZk44oiiUUfpfmF9Sm7FFQrO_g@mail.gmail.com>

There's a blog post that covers a lot on this topic:
https://blog.zeek.org/2019/08/zeek-300-rc1-released.html

-Dop

On Mon, Sep 16, 2019 at 4:12 PM James Lay <jlay at slave-tothe-box.net> wrote:

> So to mentally prepare myself here....is there a list of breaking
> changes that we'll need to address before attempting an install on this?
>   I have several scripts that I really don't want to see go boom.  Thank
> you!
>
> James
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190916/8842b195/attachment.html 

From soehlert at es.net  Mon Sep 16 14:56:47 2019
From: soehlert at es.net (Samuel Oehlert)
Date: Mon, 16 Sep 2019 16:56:47 -0500
Subject: [Zeek] 2.6.4 to 3.0.0
In-Reply-To: <32d5e6cf3092dbddce9da1a49279e3c5@slave-tothe-box.net>
References: <32d5e6cf3092dbddce9da1a49279e3c5@slave-tothe-box.net>
Message-ID: <CAMLD0uJO4zdFezrv4YAmEhhj0X+fjSTT2dgqj0fLfEBUjMZFfw@mail.gmail.com>

Dop from ESnet worked heavily with Jon Siwek to figure out some places to
watch out for. It was in the announcement blog post located:
https://blog.zeek.org/2019/08/zeek-300-rc1-released.html

On Mon, Sep 16, 2019 at 4:13 PM James Lay <jlay at slave-tothe-box.net> wrote:

> So to mentally prepare myself here....is there a list of breaking
> changes that we'll need to address before attempting an install on this?
>   I have several scripts that I really don't want to see go boom.  Thank
> you!
>
> James
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190916/b3d10ee3/attachment.html 

From mgezz66 at gmail.com  Tue Sep 17 07:18:37 2019
From: mgezz66 at gmail.com (Michael Gez)
Date: Tue, 17 Sep 2019 10:18:37 -0400
Subject: [Zeek] Zeek 3.0 DNS, RDP and SMB Analyzer Changes
Message-ID: <CALuqTtVhO9yTGhqgg5_+eaWY+1nGjXHfAcRQyFjOXn=u2oan7g@mail.gmail.com>

Hi all,

Could anyone provide more information about the changes being made to DNS,
RDP and SMB analyzers in the shift to Zeek 3.0?
Are there new fields being added?
If anyone has tried it out and has any insight it would be appreciated.
I won't get a chance to test 3.0 out myself for a few weeks, so I'm hoping
to have an idea of what to expect when making the switch.

Any information would be greatly appreciated,
Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190917/ffa9daf8/attachment.html 

From jsiwek at corelight.com  Tue Sep 17 09:24:37 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Tue, 17 Sep 2019 09:24:37 -0700
Subject: [Zeek] Zeek 3.0 DNS, RDP and SMB Analyzer Changes
In-Reply-To: <CALuqTtVhO9yTGhqgg5_+eaWY+1nGjXHfAcRQyFjOXn=u2oan7g@mail.gmail.com>
References: <CALuqTtVhO9yTGhqgg5_+eaWY+1nGjXHfAcRQyFjOXn=u2oan7g@mail.gmail.com>
Message-ID: <CAMzgZ0J2Ck5ZuYbt7nxiO_RW93Y07a+4AQSjNBAsysFsfzE1Tw@mail.gmail.com>

On Tue, Sep 17, 2019 at 7:21 AM Michael Gez <mgezz66 at gmail.com> wrote:

> Could anyone provide more information about the changes being made to DNS, RDP and SMB analyzers in the shift to Zeek 3.0?

I'd suggest reading the NEWS file, which calls out all the most
important additions/changes:

https://github.com/zeek/zeek/blob/release/3.0/NEWS

To summarize what I see for those specific analyzers:

* DNS added events for SPF and DNSSEC resource records
* RDP added new events and a "client_channels" field in the rdp.log
* SMB adds support for some 3.x features (new event and new fields in
the `SMB2::NegotiateResponse` record)

- Jon



> Are there new fields being added?
> If anyone has tried it out and has any insight it would be appreciated.
> I won't get a chance to test 3.0 out myself for a few weeks, so I'm hoping to have an idea of what to expect when making the switch.
>
> Any information would be greatly appreciated,
> Thanks!
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek

From jgarciar at sia.es  Wed Sep 18 02:53:35 2019
From: jgarciar at sia.es (Jorge Garcia Rodriguez)
Date: Wed, 18 Sep 2019 09:53:35 +0000
Subject: [Zeek]  Specs needed for 10GBps
Message-ID: <D6E3E7F0075CEF4CB16D0A627912409A35DF5BBC@Newton.sia.es>

Hi, We are new in Zeek and we want to use it to monitor traffic with 10 GBps peaks. We need to know the necessary specifications. How much Ram would be necessary? How much CPU?

We checked the documentation but found nothing about this, only that we need 1 core for every 250 Mbp/s. Which seems that we need 40 cores or so.

Thank you for your help.

Regards.

Jorge Garc?a Rodr?guez
Technical Consultant
Security Infrastructures
jgarciar at sia.es<mailto:jgarciar at sia.es>
Grupo SIA




-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190918/7179d749/attachment.html 

From fatema.bannatwala at gmail.com  Wed Sep 18 12:29:11 2019
From: fatema.bannatwala at gmail.com (fatema bannatwala)
Date: Wed, 18 Sep 2019 15:29:11 -0400
Subject: [Zeek] Specs needed for 10GBps
Message-ID: <CACX0rUTqTdLft-Z=7axU9w0U7XK2x4ktY1UNgBnx45htoB1bkg@mail.gmail.com>

There is this amazing document available for the 100G Intrusion Detection,
where the specs of 10Gbps per system are mentioned with a lot of good
information to consider while spec-ing out the hardware and designing an
efficient monitoring solution.
Here's the link to the whitepaper:
https://www.cspi.com/wp-content/uploads/2016/09/Berkeley-100GIntrusionDetection.pdf


Hope this helps.

Fatema
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190918/e24c7dc2/attachment.html 

From Kayode_Enwerem at ao.uscourts.gov  Thu Sep 19 09:00:38 2019
From: Kayode_Enwerem at ao.uscourts.gov (Kayode Enwerem)
Date: Thu, 19 Sep 2019 16:00:38 +0000
Subject: [Zeek] Why does my logger keep crashing - bro version 2.6.3
In-Reply-To: <BN6PR09MB11542B9F128BD9B7413F842BA1890@BN6PR09MB1154.namprd09.prod.outlook.com>
References: <BN6PR09MB11542B9F128BD9B7413F842BA1890@BN6PR09MB1154.namprd09.prod.outlook.com>
Message-ID: <BN6PR09MB115443C06D491DA302F640F3A1890@BN6PR09MB1154.namprd09.prod.outlook.com>

Hello,

Why does my logger keep crashing? Can someone please help me with this. I have provided some system information below:

I am running bro version 2.6.3


  1.  Output of broctl status. The logger is crashed but the manager, proxy and workers are still running.
broctl status
Name         Type    Host             Status    Pid    Started
logger       logger  localhost        crashed
manager      manager localhost        running   17356  09 Sep 15:42:24
proxy-1      proxy   localhost        running   17401  09 Sep 15:42:25
worker-1-1   worker  localhost        running   17573  09 Sep 15:42:27
worker-1-2   worker  localhost        running   17569  09 Sep 15:42:27
worker-1-3   worker  localhost        running   17572  09 Sep 15:42:27
worker-1-4   worker  localhost        running   17587  09 Sep 15:42:27
worker-1-5   worker  localhost        running   17619  09 Sep 15:42:27
worker-1-6   worker  localhost        running   17614  09 Sep 15:42:27
worker-1-7   worker  localhost        running   17625  09 Sep 15:42:27
worker-1-8   worker  localhost        running   17646  09 Sep 15:42:27
worker-1-9   worker  localhost        running   17671  09 Sep 15:42:27
worker-1-10  worker  localhost        running   17663  09 Sep 15:42:27
worker-1-11  worker  localhost        running   17679  09 Sep 15:42:27
worker-1-12  worker  localhost        running   17685  09 Sep 15:42:27
worker-1-13  worker  localhost        running   17698  09 Sep 15:42:27
worker-1-14  worker  localhost        running   17703  09 Sep 15:42:27
worker-1-15  worker  localhost        running   17710  09 Sep 15:42:27
worker-1-16  worker  localhost        running   17717  09 Sep 15:42:27
worker-1-17  worker  localhost        running   17720  09 Sep 15:42:27
worker-1-18  worker  localhost        running   17727  09 Sep 15:42:27
worker-1-19  worker  localhost        running   17728  09 Sep 15:42:27
worker-1-20  worker  localhost        running   17731  09 Sep 15:42:27


  1.  Here's my node.cfg settings
[logger]
type=logger
host=localhost

[manager]
type=manager
host=localhost

[proxy-1]
type=proxy
host=localhost

[worker-1]
type=worker
host=localhost
interface=af_packet::ens2f0
lb_method=custom
#lb_method=pf_ring
lb_procs=20
pin_cpus=6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25
af_packet_fanout_id=25
af_packet_fanout_mode=AF_Packet::FANOUT_HASH


  1.  Heres more information on my CPU. 32 CPUs, model name - AMD, CPU max MHz is 2800.0000
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                32
On-line CPU(s) list:   0-31
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             2
NUMA node(s):          4
Vendor ID:             AuthenticAMD
CPU family:            21
Model:                 2
Model name:            AMD Opteron(tm) Processor 6386 SE
Stepping:              0
CPU MHz:               1960.000
CPU max MHz:           2800.0000
CPU min MHz:           1400.0000
BogoMIPS:              5585.93
Virtualization:        AMD-V
L1d cache:             16K
L1i cache:             64K
L2 cache:              2048K
L3 cache:              6144K
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14
NUMA node1 CPU(s):     16,18,20,22,24,26,28,30
NUMA node2 CPU(s):     1,3,5,7,9,11,13,15
NUMA node3 CPU(s):     17,19,21,23,25,27,29,31


  1.  Would also like to know how I can reduce my packet loss. Below is the output of broctl netstats
broctl netstats
worker-1-1: 1568908277.861813 recvd=12248845422 dropped=5171188999 link=17420313882
worker-1-2: 1568908298.313954 recvd=8636707266 dropped=971489 link=8637678939
worker-1-3: 1568908278.425888 recvd=11684808853 dropped=5617381647 link=17302473791
worker-1-4: 1568908285.731130 recvd=12567242226 dropped=4339688288 link=16907212802
worker-1-5: 1568908298.363911 recvd=8620499351 dropped=24595149 link=8645095758
worker-1-6: 1568908298.372892 recvd=8710565757 dropped=1731022 link=8712297432
worker-1-7: 1568908298.266010 recvd=9065207444 dropped=53523232 link=9118737229
worker-1-8: 1568908286.935607 recvd=11377790124 dropped=3680887247 link=15058934491
worker-1-9: 1568908298.419657 recvd=8931903322 dropped=39696184 link=8971604219
worker-1-10: 1568908298.478576 recvd=8842874030 dropped=2501252 link=8845376352
worker-1-11: 1568908298.506649 recvd=8692769329 dropped=2253413 link=8695025626
worker-1-12: 1568908298.520830 recvd=8749977028 dropped=2314733 link=8752293714
worker-1-13: 1568908298.544573 recvd=9101243757 dropped=1779460 link=9103025399
worker-1-14: 1568908291.370011 recvd=10876925726 dropped=775722632 link=11652810353
worker-1-15: 1568908298.579721 recvd=8503097394 dropped=1420699 link=8504520066
worker-1-16: 1568908298.594942 recvd=8515164266 dropped=1840977 link=8517006779
worker-1-17: 1568908298.646966 recvd=10666567717 dropped=466489754 link=11133059283
worker-1-18: 1568908298.671246 recvd=9023603573 dropped=2037607 link=9025642263
worker-1-19: 1568908298.704675 recvd=8907784186 dropped=1164594 link=8908950238
worker-1-20: 1568908298.718084 recvd=9140525444 dropped=2028593 link=9142555259

Thanks,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190919/0f10bbe4/attachment-0001.html 

From bill.de.ping at gmail.com  Sun Sep 22 03:41:50 2019
From: bill.de.ping at gmail.com (william de ping)
Date: Sun, 22 Sep 2019 13:41:50 +0300
Subject: [Zeek] Why does my logger keep crashing - bro version 2.6.3
In-Reply-To: <BN6PR09MB115443C06D491DA302F640F3A1890@BN6PR09MB1154.namprd09.prod.outlook.com>
References: <BN6PR09MB11542B9F128BD9B7413F842BA1890@BN6PR09MB1154.namprd09.prod.outlook.com>
	<BN6PR09MB115443C06D491DA302F640F3A1890@BN6PR09MB1154.namprd09.prod.outlook.com>
Message-ID: <CAKZA8xjFZpcsZQ-JVXyT5ZrAaZ0bL7VObV+69NRejn6Wyy4AzQ@mail.gmail.com>

Hi,

I would try to monitor the cpu \ mem usage of the logger instance.
Try running broctl top, my guess is that you will see that the logger
process will have a very high cpu usage.

I know of an option to have multiple loggers but I am not sure how to set
it up.

Are you writing to a file ?

B

On Thu, Sep 19, 2019 at 7:14 PM Kayode Enwerem <
Kayode_Enwerem at ao.uscourts.gov> wrote:

> Hello,
>
>
>
> Why does my logger keep crashing? Can someone please help me with this. I
> have provided some system information below:
>
>
>
> I am running bro version 2.6.3
>
>
>
>    1. Output of broctl status. The logger is crashed but the manager,
>    proxy and workers are still running.
>
> broctl status
>
> *Name         Type    Host             Status    Pid    Started*
>
> logger       logger  localhost        crashed
>
> manager      manager localhost        running   17356  09 Sep 15:42:24
>
> proxy-1      proxy   localhost        running   17401  09 Sep 15:42:25
>
> worker-1-1   worker  localhost        running   17573  09 Sep 15:42:27
>
> worker-1-2   worker  localhost        running   17569  09 Sep 15:42:27
>
> worker-1-3   worker  localhost        running   17572  09 Sep 15:42:27
>
> worker-1-4   worker  localhost        running   17587  09 Sep 15:42:27
>
> worker-1-5   worker  localhost        running   17619  09 Sep 15:42:27
>
> worker-1-6   worker  localhost        running   17614  09 Sep 15:42:27
>
> worker-1-7   worker  localhost        running   17625  09 Sep 15:42:27
>
> worker-1-8   worker  localhost        running   17646  09 Sep 15:42:27
>
> worker-1-9   worker  localhost        running   17671  09 Sep 15:42:27
>
> worker-1-10  worker  localhost        running   17663  09 Sep 15:42:27
>
> worker-1-11  worker  localhost        running   17679  09 Sep 15:42:27
>
> worker-1-12  worker  localhost        running   17685  09 Sep 15:42:27
>
> worker-1-13  worker  localhost        running   17698  09 Sep 15:42:27
>
> worker-1-14  worker  localhost        running   17703  09 Sep 15:42:27
>
> worker-1-15  worker  localhost        running   17710  09 Sep 15:42:27
>
> worker-1-16  worker  localhost        running   17717  09 Sep 15:42:27
>
> worker-1-17  worker  localhost        running   17720  09 Sep 15:42:27
>
> worker-1-18  worker  localhost        running   17727  09 Sep 15:42:27
>
> worker-1-19  worker  localhost        running   17728  09 Sep 15:42:27
>
> worker-1-20  worker  localhost        running   17731  09 Sep 15:42:27
>
>
>
>    1. Here?s my node.cfg settings
>
> [logger]
>
> type=logger
>
> host=localhost
>
>
>
> [manager]
>
> type=manager
>
> host=localhost
>
>
>
> [proxy-1]
>
> type=proxy
>
> host=localhost
>
>
>
> [worker-1]
>
> type=worker
>
> host=localhost
>
> interface=af_packet::ens2f0
>
> lb_method=custom
>
> #lb_method=pf_ring
>
> lb_procs=20
>
> pin_cpus=6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25
>
> af_packet_fanout_id=25
>
> af_packet_fanout_mode=AF_Packet::FANOUT_HASH
>
>
>
>    1. Heres more information on my CPU. 32 CPUs, model name ? AMD, CPU
>    max MHz is 2800.0000
>
> Architecture:          x86_64
>
> CPU op-mode(s):        32-bit, 64-bit
>
> Byte Order:            Little Endian
>
> CPU(s):                32
>
> On-line CPU(s) list:   0-31
>
> Thread(s) per core:    2
>
> Core(s) per socket:    8
>
> Socket(s):             2
>
> NUMA node(s):          4
>
> Vendor ID:             AuthenticAMD
>
> CPU family:            21
>
> Model:                 2
>
> Model name:            AMD Opteron(tm) Processor 6386 SE
>
> Stepping:              0
>
> CPU MHz:               1960.000
>
> CPU max MHz:           2800.0000
>
> CPU min MHz:           1400.0000
>
> BogoMIPS:              5585.93
>
> Virtualization:        AMD-V
>
> L1d cache:             16K
>
> L1i cache:             64K
>
> L2 cache:              2048K
>
> L3 cache:              6144K
>
> NUMA node0 CPU(s):     0,2,4,6,8,10,12,14
>
> NUMA node1 CPU(s):     16,18,20,22,24,26,28,30
>
> NUMA node2 CPU(s):     1,3,5,7,9,11,13,15
>
> NUMA node3 CPU(s):     17,19,21,23,25,27,29,31
>
>
>
>    1. Would also like to know how I can reduce my packet loss. Below is
>    the output of broctl netstats
>
> broctl netstats
>
> worker-1-1: 1568908277.861813 recvd=12248845422 dropped=5171188999
> link=17420313882
>
> worker-1-2: 1568908298.313954 recvd=8636707266 dropped=971489
> link=8637678939
>
> worker-1-3: 1568908278.425888 recvd=11684808853 dropped=5617381647
> link=17302473791
>
> worker-1-4: 1568908285.731130 recvd=12567242226 dropped=4339688288
> link=16907212802
>
> worker-1-5: 1568908298.363911 recvd=8620499351 dropped=24595149
> link=8645095758
>
> worker-1-6: 1568908298.372892 recvd=8710565757 dropped=1731022
> link=8712297432
>
> worker-1-7: 1568908298.266010 recvd=9065207444 dropped=53523232
> link=9118737229
>
> worker-1-8: 1568908286.935607 recvd=11377790124 dropped=3680887247
> link=15058934491
>
> worker-1-9: 1568908298.419657 recvd=8931903322 dropped=39696184
> link=8971604219
>
> worker-1-10: 1568908298.478576 recvd=8842874030 dropped=2501252
> link=8845376352
>
> worker-1-11: 1568908298.506649 recvd=8692769329 dropped=2253413
> link=8695025626
>
> worker-1-12: 1568908298.520830 recvd=8749977028 dropped=2314733
> link=8752293714
>
> worker-1-13: 1568908298.544573 recvd=9101243757 dropped=1779460
> link=9103025399
>
> worker-1-14: 1568908291.370011 recvd=10876925726 dropped=775722632
> link=11652810353
>
> worker-1-15: 1568908298.579721 recvd=8503097394 dropped=1420699
> link=8504520066
>
> worker-1-16: 1568908298.594942 recvd=8515164266 dropped=1840977
> link=8517006779
>
> worker-1-17: 1568908298.646966 recvd=10666567717 dropped=466489754
> link=11133059283
>
> worker-1-18: 1568908298.671246 recvd=9023603573 dropped=2037607
> link=9025642263
>
> worker-1-19: 1568908298.704675 recvd=8907784186 dropped=1164594
> link=8908950238
>
> worker-1-20: 1568908298.718084 recvd=9140525444 dropped=2028593
> link=9142555259
>
>
>
> Thanks,
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190922/5201d19a/attachment.html 

From Kayode_Enwerem at ao.uscourts.gov  Mon Sep 23 06:18:21 2019
From: Kayode_Enwerem at ao.uscourts.gov (Kayode Enwerem)
Date: Mon, 23 Sep 2019 13:18:21 +0000
Subject: [Zeek] Why does my logger keep crashing - bro version 2.6.3
In-Reply-To: <CAKZA8xjFZpcsZQ-JVXyT5ZrAaZ0bL7VObV+69NRejn6Wyy4AzQ@mail.gmail.com>
References: <BN6PR09MB11542B9F128BD9B7413F842BA1890@BN6PR09MB1154.namprd09.prod.outlook.com>
	<BN6PR09MB115443C06D491DA302F640F3A1890@BN6PR09MB1154.namprd09.prod.outlook.com>
	<CAKZA8xjFZpcsZQ-JVXyT5ZrAaZ0bL7VObV+69NRejn6Wyy4AzQ@mail.gmail.com>
Message-ID: <BN6PR09MB11542783738CE7EFBDEF8324A1850@BN6PR09MB1154.namprd09.prod.outlook.com>

Thanks for your response. The CPU usage for the logger is at 311%. (look below).

broctl top
Name         Type    Host             Pid     VSize  Rss  Cpu   Cmd
logger       logger  localhost        22867    12G     9G 311%  bro

I wasn?t aware that you could set up multiple loggers, I tried checking the docs to see if that was an option. Does anyone know how to do this?

From: william de ping <bill.de.ping at gmail.com>
Sent: Sunday, September 22, 2019 6:42 AM
To: Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov>
Cc: zeek at zeek.org
Subject: Re: [Zeek] Why does my logger keep crashing - bro version 2.6.3

Hi,

I would try to monitor the cpu \ mem usage of the logger instance.
Try running broctl top, my guess is that you will see that the logger process will have a very high cpu usage.

I know of an option to have multiple loggers but I am not sure how to set it up.

Are you writing to a file ?

B

On Thu, Sep 19, 2019 at 7:14 PM Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov<mailto:Kayode_Enwerem at ao.uscourts.gov>> wrote:
Hello,

Why does my logger keep crashing? Can someone please help me with this. I have provided some system information below:

I am running bro version 2.6.3


  1.  Output of broctl status. The logger is crashed but the manager, proxy and workers are still running.
broctl status
Name         Type    Host             Status    Pid    Started
logger       logger  localhost        crashed
manager      manager localhost        running   17356  09 Sep 15:42:24
proxy-1      proxy   localhost        running   17401  09 Sep 15:42:25
worker-1-1   worker  localhost        running   17573  09 Sep 15:42:27
worker-1-2   worker  localhost        running   17569  09 Sep 15:42:27
worker-1-3   worker  localhost        running   17572  09 Sep 15:42:27
worker-1-4   worker  localhost        running   17587  09 Sep 15:42:27
worker-1-5   worker  localhost        running   17619  09 Sep 15:42:27
worker-1-6   worker  localhost        running   17614  09 Sep 15:42:27
worker-1-7   worker  localhost        running   17625  09 Sep 15:42:27
worker-1-8   worker  localhost        running   17646  09 Sep 15:42:27
worker-1-9   worker  localhost        running   17671  09 Sep 15:42:27
worker-1-10  worker  localhost        running   17663  09 Sep 15:42:27
worker-1-11  worker  localhost        running   17679  09 Sep 15:42:27
worker-1-12  worker  localhost        running   17685  09 Sep 15:42:27
worker-1-13  worker  localhost        running   17698  09 Sep 15:42:27
worker-1-14  worker  localhost        running   17703  09 Sep 15:42:27
worker-1-15  worker  localhost        running   17710  09 Sep 15:42:27
worker-1-16  worker  localhost        running   17717  09 Sep 15:42:27
worker-1-17  worker  localhost        running   17720  09 Sep 15:42:27
worker-1-18  worker  localhost        running   17727  09 Sep 15:42:27
worker-1-19  worker  localhost        running   17728  09 Sep 15:42:27
worker-1-20  worker  localhost        running   17731  09 Sep 15:42:27


  1.  Here?s my node.cfg settings
[logger]
type=logger
host=localhost

[manager]
type=manager
host=localhost

[proxy-1]
type=proxy
host=localhost

[worker-1]
type=worker
host=localhost
interface=af_packet::ens2f0
lb_method=custom
#lb_method=pf_ring
lb_procs=20
pin_cpus=6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25
af_packet_fanout_id=25
af_packet_fanout_mode=AF_Packet::FANOUT_HASH


  1.  Heres more information on my CPU. 32 CPUs, model name ? AMD, CPU max MHz is 2800.0000
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                32
On-line CPU(s) list:   0-31
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             2
NUMA node(s):          4
Vendor ID:             AuthenticAMD
CPU family:            21
Model:                 2
Model name:            AMD Opteron(tm) Processor 6386 SE
Stepping:              0
CPU MHz:               1960.000
CPU max MHz:           2800.0000
CPU min MHz:           1400.0000
BogoMIPS:              5585.93
Virtualization:        AMD-V
L1d cache:             16K
L1i cache:             64K
L2 cache:              2048K
L3 cache:              6144K
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14
NUMA node1 CPU(s):     16,18,20,22,24,26,28,30
NUMA node2 CPU(s):     1,3,5,7,9,11,13,15
NUMA node3 CPU(s):     17,19,21,23,25,27,29,31


  1.  Would also like to know how I can reduce my packet loss. Below is the output of broctl netstats
broctl netstats
worker-1-1: 1568908277.861813 recvd=12248845422 dropped=5171188999 link=17420313882
worker-1-2: 1568908298.313954 recvd=8636707266 dropped=971489 link=8637678939
worker-1-3: 1568908278.425888 recvd=11684808853 dropped=5617381647 link=17302473791
worker-1-4: 1568908285.731130 recvd=12567242226 dropped=4339688288 link=16907212802
worker-1-5: 1568908298.363911 recvd=8620499351 dropped=24595149 link=8645095758
worker-1-6: 1568908298.372892 recvd=8710565757 dropped=1731022 link=8712297432
worker-1-7: 1568908298.266010 recvd=9065207444 dropped=53523232 link=9118737229
worker-1-8: 1568908286.935607 recvd=11377790124 dropped=3680887247 link=15058934491
worker-1-9: 1568908298.419657 recvd=8931903322 dropped=39696184 link=8971604219
worker-1-10: 1568908298.478576 recvd=8842874030 dropped=2501252 link=8845376352
worker-1-11: 1568908298.506649 recvd=8692769329 dropped=2253413 link=8695025626
worker-1-12: 1568908298.520830 recvd=8749977028 dropped=2314733 link=8752293714
worker-1-13: 1568908298.544573 recvd=9101243757 dropped=1779460 link=9103025399
worker-1-14: 1568908291.370011 recvd=10876925726 dropped=775722632 link=11652810353
worker-1-15: 1568908298.579721 recvd=8503097394 dropped=1420699 link=8504520066
worker-1-16: 1568908298.594942 recvd=8515164266 dropped=1840977 link=8517006779
worker-1-17: 1568908298.646966 recvd=10666567717 dropped=466489754 link=11133059283
worker-1-18: 1568908298.671246 recvd=9023603573 dropped=2037607 link=9025642263
worker-1-19: 1568908298.704675 recvd=8907784186 dropped=1164594 link=8908950238
worker-1-20: 1568908298.718084 recvd=9140525444 dropped=2028593 link=9142555259

Thanks,
_______________________________________________
Zeek mailing list
zeek at zeek.org<mailto:zeek at zeek.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190923/b663be64/attachment-0001.html 

From vlad at es.net  Mon Sep 23 07:20:22 2019
From: vlad at es.net (Vlad Grigorescu)
Date: Mon, 23 Sep 2019 14:20:22 +0000
Subject: [Zeek] Why does my logger keep crashing - bro version 2.6.3
In-Reply-To: <BN6PR09MB11542783738CE7EFBDEF8324A1850@BN6PR09MB1154.namprd09.prod.outlook.com>
References: <BN6PR09MB11542B9F128BD9B7413F842BA1890@BN6PR09MB1154.namprd09.prod.outlook.com>
	<BN6PR09MB115443C06D491DA302F640F3A1890@BN6PR09MB1154.namprd09.prod.outlook.com>
	<CAKZA8xjFZpcsZQ-JVXyT5ZrAaZ0bL7VObV+69NRejn6Wyy4AzQ@mail.gmail.com>
	<BN6PR09MB11542783738CE7EFBDEF8324A1850@BN6PR09MB1154.namprd09.prod.outlook.com>
Message-ID: <CAPqbkwtZSuqUSTLJFDGfpWegyA=gucOZmGbdqvqxoSboLs55jQ@mail.gmail.com>

The logger is threaded, so seeing CPU > 100% is not necessarily a problem.

Have you tried running "broctl diag logger" to see why the logger is
crashing? Do you have any messages in your system logs about
processing being killed for out of memory (OOM)?

  --Vlad

On Mon, Sep 23, 2019 at 1:32 PM Kayode Enwerem
<Kayode_Enwerem at ao.uscourts.gov> wrote:
>
> Thanks for your response. The CPU usage for the logger is at 311%. (look below).
>
>
>
> broctl top
>
> Name         Type    Host             Pid     VSize  Rss  Cpu   Cmd
>
> logger       logger  localhost        22867    12G     9G 311%  bro
>
>
>
> I wasn?t aware that you could set up multiple loggers, I tried checking the docs to see if that was an option. Does anyone know how to do this?
>
>
>
> From: william de ping <bill.de.ping at gmail.com>
> Sent: Sunday, September 22, 2019 6:42 AM
> To: Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov>
> Cc: zeek at zeek.org
> Subject: Re: [Zeek] Why does my logger keep crashing - bro version 2.6.3
>
>
>
> Hi,
>
>
>
> I would try to monitor the cpu \ mem usage of the logger instance.
>
> Try running broctl top, my guess is that you will see that the logger process will have a very high cpu usage.
>
>
>
> I know of an option to have multiple loggers but I am not sure how to set it up.
>
>
>
> Are you writing to a file ?
>
>
>
> B
>
>
>
> On Thu, Sep 19, 2019 at 7:14 PM Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov> wrote:
>
> Hello,
>
>
>
> Why does my logger keep crashing? Can someone please help me with this. I have provided some system information below:
>
>
>
> I am running bro version 2.6.3
>
>
>
> Output of broctl status. The logger is crashed but the manager, proxy and workers are still running.
>
> broctl status
>
> Name         Type    Host             Status    Pid    Started
>
> logger       logger  localhost        crashed
>
> manager      manager localhost        running   17356  09 Sep 15:42:24
>
> proxy-1      proxy   localhost        running   17401  09 Sep 15:42:25
>
> worker-1-1   worker  localhost        running   17573  09 Sep 15:42:27
>
> worker-1-2   worker  localhost        running   17569  09 Sep 15:42:27
>
> worker-1-3   worker  localhost        running   17572  09 Sep 15:42:27
>
> worker-1-4   worker  localhost        running   17587  09 Sep 15:42:27
>
> worker-1-5   worker  localhost        running   17619  09 Sep 15:42:27
>
> worker-1-6   worker  localhost        running   17614  09 Sep 15:42:27
>
> worker-1-7   worker  localhost        running   17625  09 Sep 15:42:27
>
> worker-1-8   worker  localhost        running   17646  09 Sep 15:42:27
>
> worker-1-9   worker  localhost        running   17671  09 Sep 15:42:27
>
> worker-1-10  worker  localhost        running   17663  09 Sep 15:42:27
>
> worker-1-11  worker  localhost        running   17679  09 Sep 15:42:27
>
> worker-1-12  worker  localhost        running   17685  09 Sep 15:42:27
>
> worker-1-13  worker  localhost        running   17698  09 Sep 15:42:27
>
> worker-1-14  worker  localhost        running   17703  09 Sep 15:42:27
>
> worker-1-15  worker  localhost        running   17710  09 Sep 15:42:27
>
> worker-1-16  worker  localhost        running   17717  09 Sep 15:42:27
>
> worker-1-17  worker  localhost        running   17720  09 Sep 15:42:27
>
> worker-1-18  worker  localhost        running   17727  09 Sep 15:42:27
>
> worker-1-19  worker  localhost        running   17728  09 Sep 15:42:27
>
> worker-1-20  worker  localhost        running   17731  09 Sep 15:42:27
>
>
>
> Here?s my node.cfg settings
>
> [logger]
>
> type=logger
>
> host=localhost
>
>
>
> [manager]
>
> type=manager
>
> host=localhost
>
>
>
> [proxy-1]
>
> type=proxy
>
> host=localhost
>
>
>
> [worker-1]
>
> type=worker
>
> host=localhost
>
> interface=af_packet::ens2f0
>
> lb_method=custom
>
> #lb_method=pf_ring
>
> lb_procs=20
>
> pin_cpus=6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25
>
> af_packet_fanout_id=25
>
> af_packet_fanout_mode=AF_Packet::FANOUT_HASH
>
>
>
> Heres more information on my CPU. 32 CPUs, model name ? AMD, CPU max MHz is 2800.0000
>
> Architecture:          x86_64
>
> CPU op-mode(s):        32-bit, 64-bit
>
> Byte Order:            Little Endian
>
> CPU(s):                32
>
> On-line CPU(s) list:   0-31
>
> Thread(s) per core:    2
>
> Core(s) per socket:    8
>
> Socket(s):             2
>
> NUMA node(s):          4
>
> Vendor ID:             AuthenticAMD
>
> CPU family:            21
>
> Model:                 2
>
> Model name:            AMD Opteron(tm) Processor 6386 SE
>
> Stepping:              0
>
> CPU MHz:               1960.000
>
> CPU max MHz:           2800.0000
>
> CPU min MHz:           1400.0000
>
> BogoMIPS:              5585.93
>
> Virtualization:        AMD-V
>
> L1d cache:             16K
>
> L1i cache:             64K
>
> L2 cache:              2048K
>
> L3 cache:              6144K
>
> NUMA node0 CPU(s):     0,2,4,6,8,10,12,14
>
> NUMA node1 CPU(s):     16,18,20,22,24,26,28,30
>
> NUMA node2 CPU(s):     1,3,5,7,9,11,13,15
>
> NUMA node3 CPU(s):     17,19,21,23,25,27,29,31
>
>
>
> Would also like to know how I can reduce my packet loss. Below is the output of broctl netstats
>
> broctl netstats
>
> worker-1-1: 1568908277.861813 recvd=12248845422 dropped=5171188999 link=17420313882
>
> worker-1-2: 1568908298.313954 recvd=8636707266 dropped=971489 link=8637678939
>
> worker-1-3: 1568908278.425888 recvd=11684808853 dropped=5617381647 link=17302473791
>
> worker-1-4: 1568908285.731130 recvd=12567242226 dropped=4339688288 link=16907212802
>
> worker-1-5: 1568908298.363911 recvd=8620499351 dropped=24595149 link=8645095758
>
> worker-1-6: 1568908298.372892 recvd=8710565757 dropped=1731022 link=8712297432
>
> worker-1-7: 1568908298.266010 recvd=9065207444 dropped=53523232 link=9118737229
>
> worker-1-8: 1568908286.935607 recvd=11377790124 dropped=3680887247 link=15058934491
>
> worker-1-9: 1568908298.419657 recvd=8931903322 dropped=39696184 link=8971604219
>
> worker-1-10: 1568908298.478576 recvd=8842874030 dropped=2501252 link=8845376352
>
> worker-1-11: 1568908298.506649 recvd=8692769329 dropped=2253413 link=8695025626
>
> worker-1-12: 1568908298.520830 recvd=8749977028 dropped=2314733 link=8752293714
>
> worker-1-13: 1568908298.544573 recvd=9101243757 dropped=1779460 link=9103025399
>
> worker-1-14: 1568908291.370011 recvd=10876925726 dropped=775722632 link=11652810353
>
> worker-1-15: 1568908298.579721 recvd=8503097394 dropped=1420699 link=8504520066
>
> worker-1-16: 1568908298.594942 recvd=8515164266 dropped=1840977 link=8517006779
>
> worker-1-17: 1568908298.646966 recvd=10666567717 dropped=466489754 link=11133059283
>
> worker-1-18: 1568908298.671246 recvd=9023603573 dropped=2037607 link=9025642263
>
> worker-1-19: 1568908298.704675 recvd=8907784186 dropped=1164594 link=8908950238
>
> worker-1-20: 1568908298.718084 recvd=9140525444 dropped=2028593 link=9142555259
>
>
>
> Thanks,
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From jsiwek at corelight.com  Mon Sep 23 12:16:00 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Mon, 23 Sep 2019 12:16:00 -0700
Subject: [Zeek] Zeek 3.0.0 release available
Message-ID: <CAMzgZ0+rsZJ5_5fP2JC5bprFAye=GZ4JNxGbi3L4woOU2g5uPQ@mail.gmail.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Zeek release 3.0.0 is now available:

  https://www.zeek.org/downloads/zeek-3.0.0.tar.gz
  https://www.zeek.org/downloads/zeek-3.0.0.tar.gz.asc

This major release has many additions and changes, the most
prominent being a comprehensive switch to use the name Zeek
instead of Bro.  Please read the release notes for a full
description of new features and changes:

  https://github.com/zeek/zeek/blob/v3.0.0/NEWS

The Zeek blog also describes the upcoming release and potential
pitfalls to be aware of when upgrading:

  https://blog.zeek.org/2019/09/zeek-300.html

-----BEGIN PGP SIGNATURE-----

iQIzBAEBAgAdFiEE6WkLK32KwaGfkhxKxotJTfVqzH4FAl2JGSkACgkQxotJTfVq
zH4Nsg/+Kdmvlfe/3OfTQnLm9gjAU6ZZ8Zihwusdv1dttG+Bil7yhvBKCTE5PF2W
Ve5tNA6Z/nfCRiBYL26IUP2xPXGlDTWaCwU0uUiOxURDZBD0YXerjNIyBJds4P3l
gD+14GjJaEIWh/2Y0iM8nntTKfdqUmfpMF4laXns3leNj/M0KIgHWJGwvxriVAMu
UhU87m84/l1+AuoUqscnVf1j5qyX06lQET6v06w8xd5eyrI0C5U8eWXWMolPnzoC
oQ5yeuur7o102tNzp5rYS/Pnmn+WQx2HMumB/v6U/iTh8P2cR6n1uzD6d5w35TXN
9zMGss6v5/92SKR7umaUOo5TWM3kS6ieXuEgUwsf77252sE3TpoOPPWQXnzwYdLT
NgqJvyYspWkhNlY2cSJ8LFAu6cobIlBdcGFUtwundLH7to8wFaFS3a9WgCWPV7vk
K/5b47sJy1p2F8rHJljdqcxUw3LAq57lnob6bhsKtKm5ZZBPgFei8d9S8PUrtade
u+mXunraQbLAsyzTdKUhI7hW7gIXF6dRo5NGmYL+Fh1COnAGmhvIrCjOnBftw6LR
FkbLwIAWLS1VD/8tyFm1+klVgQItylbvc/UDsy709mWa0anZ/bmyv9s+pW+2Evbu
g7CTrS7eb+MoEPgtEnmuR+sem34hkEPsAuwYwqYrcLwwm4leREU=
=2Ql2
-----END PGP SIGNATURE-----

From Kayode_Enwerem at ao.uscourts.gov  Tue Sep 24 09:49:07 2019
From: Kayode_Enwerem at ao.uscourts.gov (Kayode Enwerem)
Date: Tue, 24 Sep 2019 16:49:07 +0000
Subject: [Zeek] Why does my logger keep crashing - bro version 2.6.3
In-Reply-To: <CAPqbkwtZSuqUSTLJFDGfpWegyA=gucOZmGbdqvqxoSboLs55jQ@mail.gmail.com>
References: <BN6PR09MB11542B9F128BD9B7413F842BA1890@BN6PR09MB1154.namprd09.prod.outlook.com>
	<BN6PR09MB115443C06D491DA302F640F3A1890@BN6PR09MB1154.namprd09.prod.outlook.com>
	<CAKZA8xjFZpcsZQ-JVXyT5ZrAaZ0bL7VObV+69NRejn6Wyy4AzQ@mail.gmail.com>
	<BN6PR09MB11542783738CE7EFBDEF8324A1850@BN6PR09MB1154.namprd09.prod.outlook.com>
	<CAPqbkwtZSuqUSTLJFDGfpWegyA=gucOZmGbdqvqxoSboLs55jQ@mail.gmail.com>
Message-ID: <BN6PR09MB1154540BEAE66A7A8991C63EA1840@BN6PR09MB1154.namprd09.prod.outlook.com>

Thanks for your response.

I do see the following OOM message in my system logs on the logger process ID:
Sep 23 18:48:00 kernel: Out of memory: Kill process 10439 (bro) score 787 or sacrifice child
Sep 23 18:48:00 kernel: Killed process 10439 (bro), UID 0, total-vm:301983900kB, anon-rss:195261772kB, file-rss:2592kB, shmem-rss:0kB

Wonder why its taking so much memory, I have 251G and 99G swap on this server.
total        used        free      shared  buff/cache   available
Mem:           251G         66G        185G        4.2M        488M        184G
Swap:           99G        1.1G         98G

Below is the output of "broctl diag logger", ran after the logger crashed. 

 [logger]

No core file found.

Bro 2.6.3
Linux 3.10.0-1062.1.1.el7.x86_64

Bro plugins:
Bro::AF_Packet - Packet acquisition via AF_Packet (dynamic, version 1.4)

==== No reporter.log

==== stderr.log
/usr/local/bro/share/broctl/scripts/run-bro: line 110: 10439 Killed                  nohup "$mybro" "$@"

==== stdout.log
max memory size         (kbytes, -m) unlimited
data seg size           (kbytes, -d) unlimited
virtual memory          (kbytes, -v) unlimited
core file size          (blocks, -c) unlimited

==== .cmdline
-U .status -p broctl -p broctl-live -p local -p logger local.bro broctl base/frameworks/cluster broctl/auto

==== .env_vars
PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bro/bin
BROPATH=/logs/bro/spool/installed-scripts-do-not-touch/site::/logs/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site
CLUSTER_NODE=logger

==== .status
RUNNING [net_run]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log

Thoughts? Any suggestions.

-----Original Message-----
From: Vlad Grigorescu <vlad at es.net> 
Sent: Monday, September 23, 2019 10:20 AM
To: Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov>
Cc: william de ping <bill.de.ping at gmail.com>; zeek at zeek.org
Subject: Re: [Zeek] Why does my logger keep crashing - bro version 2.6.3

The logger is threaded, so seeing CPU > 100% is not necessarily a problem.

Have you tried running "broctl diag logger" to see why the logger is crashing? Do you have any messages in your system logs about processing being killed for out of memory (OOM)?

  --Vlad

On Mon, Sep 23, 2019 at 1:32 PM Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov> wrote:
>
> Thanks for your response. The CPU usage for the logger is at 311%. (look below).
>
>
>
> broctl top
>
> Name         Type    Host             Pid     VSize  Rss  Cpu   Cmd
>
> logger       logger  localhost        22867    12G     9G 311%  bro
>
>
>
> I wasn?t aware that you could set up multiple loggers, I tried checking the docs to see if that was an option. Does anyone know how to do this?
>
>
>
> From: william de ping <bill.de.ping at gmail.com>
> Sent: Sunday, September 22, 2019 6:42 AM
> To: Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov>
> Cc: zeek at zeek.org
> Subject: Re: [Zeek] Why does my logger keep crashing - bro version 
> 2.6.3
>
>
>
> Hi,
>
>
>
> I would try to monitor the cpu \ mem usage of the logger instance.
>
> Try running broctl top, my guess is that you will see that the logger process will have a very high cpu usage.
>
>
>
> I know of an option to have multiple loggers but I am not sure how to set it up.
>
>
>
> Are you writing to a file ?
>
>
>
> B
>
>
>
> On Thu, Sep 19, 2019 at 7:14 PM Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov> wrote:
>
> Hello,
>
>
>
> Why does my logger keep crashing? Can someone please help me with this. I have provided some system information below:
>
>
>
> I am running bro version 2.6.3
>
>
>
> Output of broctl status. The logger is crashed but the manager, proxy and workers are still running.
>
> broctl status
>
> Name         Type    Host             Status    Pid    Started
>
> logger       logger  localhost        crashed
>
> manager      manager localhost        running   17356  09 Sep 15:42:24
>
> proxy-1      proxy   localhost        running   17401  09 Sep 15:42:25
>
> worker-1-1   worker  localhost        running   17573  09 Sep 15:42:27
>
> worker-1-2   worker  localhost        running   17569  09 Sep 15:42:27
>
> worker-1-3   worker  localhost        running   17572  09 Sep 15:42:27
>
> worker-1-4   worker  localhost        running   17587  09 Sep 15:42:27
>
> worker-1-5   worker  localhost        running   17619  09 Sep 15:42:27
>
> worker-1-6   worker  localhost        running   17614  09 Sep 15:42:27
>
> worker-1-7   worker  localhost        running   17625  09 Sep 15:42:27
>
> worker-1-8   worker  localhost        running   17646  09 Sep 15:42:27
>
> worker-1-9   worker  localhost        running   17671  09 Sep 15:42:27
>
> worker-1-10  worker  localhost        running   17663  09 Sep 15:42:27
>
> worker-1-11  worker  localhost        running   17679  09 Sep 15:42:27
>
> worker-1-12  worker  localhost        running   17685  09 Sep 15:42:27
>
> worker-1-13  worker  localhost        running   17698  09 Sep 15:42:27
>
> worker-1-14  worker  localhost        running   17703  09 Sep 15:42:27
>
> worker-1-15  worker  localhost        running   17710  09 Sep 15:42:27
>
> worker-1-16  worker  localhost        running   17717  09 Sep 15:42:27
>
> worker-1-17  worker  localhost        running   17720  09 Sep 15:42:27
>
> worker-1-18  worker  localhost        running   17727  09 Sep 15:42:27
>
> worker-1-19  worker  localhost        running   17728  09 Sep 15:42:27
>
> worker-1-20  worker  localhost        running   17731  09 Sep 15:42:27
>
>
>
> Here?s my node.cfg settings
>
> [logger]
>
> type=logger
>
> host=localhost
>
>
>
> [manager]
>
> type=manager
>
> host=localhost
>
>
>
> [proxy-1]
>
> type=proxy
>
> host=localhost
>
>
>
> [worker-1]
>
> type=worker
>
> host=localhost
>
> interface=af_packet::ens2f0
>
> lb_method=custom
>
> #lb_method=pf_ring
>
> lb_procs=20
>
> pin_cpus=6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25
>
> af_packet_fanout_id=25
>
> af_packet_fanout_mode=AF_Packet::FANOUT_HASH
>
>
>
> Heres more information on my CPU. 32 CPUs, model name ? AMD, CPU max 
> MHz is 2800.0000
>
> Architecture:          x86_64
>
> CPU op-mode(s):        32-bit, 64-bit
>
> Byte Order:            Little Endian
>
> CPU(s):                32
>
> On-line CPU(s) list:   0-31
>
> Thread(s) per core:    2
>
> Core(s) per socket:    8
>
> Socket(s):             2
>
> NUMA node(s):          4
>
> Vendor ID:             AuthenticAMD
>
> CPU family:            21
>
> Model:                 2
>
> Model name:            AMD Opteron(tm) Processor 6386 SE
>
> Stepping:              0
>
> CPU MHz:               1960.000
>
> CPU max MHz:           2800.0000
>
> CPU min MHz:           1400.0000
>
> BogoMIPS:              5585.93
>
> Virtualization:        AMD-V
>
> L1d cache:             16K
>
> L1i cache:             64K
>
> L2 cache:              2048K
>
> L3 cache:              6144K
>
> NUMA node0 CPU(s):     0,2,4,6,8,10,12,14
>
> NUMA node1 CPU(s):     16,18,20,22,24,26,28,30
>
> NUMA node2 CPU(s):     1,3,5,7,9,11,13,15
>
> NUMA node3 CPU(s):     17,19,21,23,25,27,29,31
>
>
>
> Would also like to know how I can reduce my packet loss. Below is the 
> output of broctl netstats
>
> broctl netstats
>
> worker-1-1: 1568908277.861813 recvd=12248845422 dropped=5171188999 
> link=17420313882
>
> worker-1-2: 1568908298.313954 recvd=8636707266 dropped=971489 
> link=8637678939
>
> worker-1-3: 1568908278.425888 recvd=11684808853 dropped=5617381647 
> link=17302473791
>
> worker-1-4: 1568908285.731130 recvd=12567242226 dropped=4339688288 
> link=16907212802
>
> worker-1-5: 1568908298.363911 recvd=8620499351 dropped=24595149 
> link=8645095758
>
> worker-1-6: 1568908298.372892 recvd=8710565757 dropped=1731022 
> link=8712297432
>
> worker-1-7: 1568908298.266010 recvd=9065207444 dropped=53523232 
> link=9118737229
>
> worker-1-8: 1568908286.935607 recvd=11377790124 dropped=3680887247 
> link=15058934491
>
> worker-1-9: 1568908298.419657 recvd=8931903322 dropped=39696184 
> link=8971604219
>
> worker-1-10: 1568908298.478576 recvd=8842874030 dropped=2501252 
> link=8845376352
>
> worker-1-11: 1568908298.506649 recvd=8692769329 dropped=2253413 
> link=8695025626
>
> worker-1-12: 1568908298.520830 recvd=8749977028 dropped=2314733 
> link=8752293714
>
> worker-1-13: 1568908298.544573 recvd=9101243757 dropped=1779460 
> link=9103025399
>
> worker-1-14: 1568908291.370011 recvd=10876925726 dropped=775722632 
> link=11652810353
>
> worker-1-15: 1568908298.579721 recvd=8503097394 dropped=1420699 
> link=8504520066
>
> worker-1-16: 1568908298.594942 recvd=8515164266 dropped=1840977 
> link=8517006779
>
> worker-1-17: 1568908298.646966 recvd=10666567717 dropped=466489754 
> link=11133059283
>
> worker-1-18: 1568908298.671246 recvd=9023603573 dropped=2037607 
> link=9025642263
>
> worker-1-19: 1568908298.704675 recvd=8907784186 dropped=1164594 
> link=8908950238
>
> worker-1-20: 1568908298.718084 recvd=9140525444 dropped=2028593 
> link=9142555259
>
>
>
> Thanks,
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From bill.de.ping at gmail.com  Wed Sep 25 00:59:52 2019
From: bill.de.ping at gmail.com (william de ping)
Date: Wed, 25 Sep 2019 10:59:52 +0300
Subject: [Zeek] Why does my logger keep crashing - bro version 2.6.3
In-Reply-To: <BN6PR09MB1154540BEAE66A7A8991C63EA1840@BN6PR09MB1154.namprd09.prod.outlook.com>
References: <BN6PR09MB11542B9F128BD9B7413F842BA1890@BN6PR09MB1154.namprd09.prod.outlook.com>
	<BN6PR09MB115443C06D491DA302F640F3A1890@BN6PR09MB1154.namprd09.prod.outlook.com>
	<CAKZA8xjFZpcsZQ-JVXyT5ZrAaZ0bL7VObV+69NRejn6Wyy4AzQ@mail.gmail.com>
	<BN6PR09MB11542783738CE7EFBDEF8324A1850@BN6PR09MB1154.namprd09.prod.outlook.com>
	<CAPqbkwtZSuqUSTLJFDGfpWegyA=gucOZmGbdqvqxoSboLs55jQ@mail.gmail.com>
	<BN6PR09MB1154540BEAE66A7A8991C63EA1840@BN6PR09MB1154.namprd09.prod.outlook.com>
Message-ID: <CAKZA8xih9SnfgCBODS0cqKj=mh8jtKdFxR96j8rNMw8E6A+p=w@mail.gmail.com>

Hi

Try using the None writer instead of the ASCII one.
In local.bro add :
redef Log::default_writer=Log::WRITER_NONE;

If the logger instance still crashes then the issue is not related to an IO
bottleneck.

B

On Tue, Sep 24, 2019 at 7:49 PM Kayode Enwerem <
Kayode_Enwerem at ao.uscourts.gov> wrote:

> Thanks for your response.
>
> I do see the following OOM message in my system logs on the logger process
> ID:
> Sep 23 18:48:00 kernel: Out of memory: Kill process 10439 (bro) score 787
> or sacrifice child
> Sep 23 18:48:00 kernel: Killed process 10439 (bro), UID 0,
> total-vm:301983900kB, anon-rss:195261772kB, file-rss:2592kB, shmem-rss:0kB
>
> Wonder why its taking so much memory, I have 251G and 99G swap on this
> server.
> total        used        free      shared  buff/cache   available
> Mem:           251G         66G        185G        4.2M        488M
> 184G
> Swap:           99G        1.1G         98G
>
> Below is the output of "broctl diag logger", ran after the logger crashed.
>
>  [logger]
>
> No core file found.
>
> Bro 2.6.3
> Linux 3.10.0-1062.1.1.el7.x86_64
>
> Bro plugins:
> Bro::AF_Packet - Packet acquisition via AF_Packet (dynamic, version 1.4)
>
> ==== No reporter.log
>
> ==== stderr.log
> /usr/local/bro/share/broctl/scripts/run-bro: line 110: 10439 Killed
>           nohup "$mybro" "$@"
>
> ==== stdout.log
> max memory size         (kbytes, -m) unlimited
> data seg size           (kbytes, -d) unlimited
> virtual memory          (kbytes, -v) unlimited
> core file size          (blocks, -c) unlimited
>
> ==== .cmdline
> -U .status -p broctl -p broctl-live -p local -p logger local.bro broctl
> base/frameworks/cluster broctl/auto
>
> ==== .env_vars
>
> PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bro/bin
>
> BROPATH=/logs/bro/spool/installed-scripts-do-not-touch/site::/logs/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site
> CLUSTER_NODE=logger
>
> ==== .status
> RUNNING [net_run]
>
> ==== No prof.log
>
> ==== No packet_filter.log
>
> ==== No loaded_scripts.log
>
> Thoughts? Any suggestions.
>
> -----Original Message-----
> From: Vlad Grigorescu <vlad at es.net>
> Sent: Monday, September 23, 2019 10:20 AM
> To: Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov>
> Cc: william de ping <bill.de.ping at gmail.com>; zeek at zeek.org
> Subject: Re: [Zeek] Why does my logger keep crashing - bro version 2.6.3
>
> The logger is threaded, so seeing CPU > 100% is not necessarily a problem.
>
> Have you tried running "broctl diag logger" to see why the logger is
> crashing? Do you have any messages in your system logs about processing
> being killed for out of memory (OOM)?
>
>   --Vlad
>
> On Mon, Sep 23, 2019 at 1:32 PM Kayode Enwerem <
> Kayode_Enwerem at ao.uscourts.gov> wrote:
> >
> > Thanks for your response. The CPU usage for the logger is at 311%. (look
> below).
> >
> >
> >
> > broctl top
> >
> > Name         Type    Host             Pid     VSize  Rss  Cpu   Cmd
> >
> > logger       logger  localhost        22867    12G     9G 311%  bro
> >
> >
> >
> > I wasn?t aware that you could set up multiple loggers, I tried checking
> the docs to see if that was an option. Does anyone know how to do this?
> >
> >
> >
> > From: william de ping <bill.de.ping at gmail.com>
> > Sent: Sunday, September 22, 2019 6:42 AM
> > To: Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov>
> > Cc: zeek at zeek.org
> > Subject: Re: [Zeek] Why does my logger keep crashing - bro version
> > 2.6.3
> >
> >
> >
> > Hi,
> >
> >
> >
> > I would try to monitor the cpu \ mem usage of the logger instance.
> >
> > Try running broctl top, my guess is that you will see that the logger
> process will have a very high cpu usage.
> >
> >
> >
> > I know of an option to have multiple loggers but I am not sure how to
> set it up.
> >
> >
> >
> > Are you writing to a file ?
> >
> >
> >
> > B
> >
> >
> >
> > On Thu, Sep 19, 2019 at 7:14 PM Kayode Enwerem <
> Kayode_Enwerem at ao.uscourts.gov> wrote:
> >
> > Hello,
> >
> >
> >
> > Why does my logger keep crashing? Can someone please help me with this.
> I have provided some system information below:
> >
> >
> >
> > I am running bro version 2.6.3
> >
> >
> >
> > Output of broctl status. The logger is crashed but the manager, proxy
> and workers are still running.
> >
> > broctl status
> >
> > Name         Type    Host             Status    Pid    Started
> >
> > logger       logger  localhost        crashed
> >
> > manager      manager localhost        running   17356  09 Sep 15:42:24
> >
> > proxy-1      proxy   localhost        running   17401  09 Sep 15:42:25
> >
> > worker-1-1   worker  localhost        running   17573  09 Sep 15:42:27
> >
> > worker-1-2   worker  localhost        running   17569  09 Sep 15:42:27
> >
> > worker-1-3   worker  localhost        running   17572  09 Sep 15:42:27
> >
> > worker-1-4   worker  localhost        running   17587  09 Sep 15:42:27
> >
> > worker-1-5   worker  localhost        running   17619  09 Sep 15:42:27
> >
> > worker-1-6   worker  localhost        running   17614  09 Sep 15:42:27
> >
> > worker-1-7   worker  localhost        running   17625  09 Sep 15:42:27
> >
> > worker-1-8   worker  localhost        running   17646  09 Sep 15:42:27
> >
> > worker-1-9   worker  localhost        running   17671  09 Sep 15:42:27
> >
> > worker-1-10  worker  localhost        running   17663  09 Sep 15:42:27
> >
> > worker-1-11  worker  localhost        running   17679  09 Sep 15:42:27
> >
> > worker-1-12  worker  localhost        running   17685  09 Sep 15:42:27
> >
> > worker-1-13  worker  localhost        running   17698  09 Sep 15:42:27
> >
> > worker-1-14  worker  localhost        running   17703  09 Sep 15:42:27
> >
> > worker-1-15  worker  localhost        running   17710  09 Sep 15:42:27
> >
> > worker-1-16  worker  localhost        running   17717  09 Sep 15:42:27
> >
> > worker-1-17  worker  localhost        running   17720  09 Sep 15:42:27
> >
> > worker-1-18  worker  localhost        running   17727  09 Sep 15:42:27
> >
> > worker-1-19  worker  localhost        running   17728  09 Sep 15:42:27
> >
> > worker-1-20  worker  localhost        running   17731  09 Sep 15:42:27
> >
> >
> >
> > Here?s my node.cfg settings
> >
> > [logger]
> >
> > type=logger
> >
> > host=localhost
> >
> >
> >
> > [manager]
> >
> > type=manager
> >
> > host=localhost
> >
> >
> >
> > [proxy-1]
> >
> > type=proxy
> >
> > host=localhost
> >
> >
> >
> > [worker-1]
> >
> > type=worker
> >
> > host=localhost
> >
> > interface=af_packet::ens2f0
> >
> > lb_method=custom
> >
> > #lb_method=pf_ring
> >
> > lb_procs=20
> >
> > pin_cpus=6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25
> >
> > af_packet_fanout_id=25
> >
> > af_packet_fanout_mode=AF_Packet::FANOUT_HASH
> >
> >
> >
> > Heres more information on my CPU. 32 CPUs, model name ? AMD, CPU max
> > MHz is 2800.0000
> >
> > Architecture:          x86_64
> >
> > CPU op-mode(s):        32-bit, 64-bit
> >
> > Byte Order:            Little Endian
> >
> > CPU(s):                32
> >
> > On-line CPU(s) list:   0-31
> >
> > Thread(s) per core:    2
> >
> > Core(s) per socket:    8
> >
> > Socket(s):             2
> >
> > NUMA node(s):          4
> >
> > Vendor ID:             AuthenticAMD
> >
> > CPU family:            21
> >
> > Model:                 2
> >
> > Model name:            AMD Opteron(tm) Processor 6386 SE
> >
> > Stepping:              0
> >
> > CPU MHz:               1960.000
> >
> > CPU max MHz:           2800.0000
> >
> > CPU min MHz:           1400.0000
> >
> > BogoMIPS:              5585.93
> >
> > Virtualization:        AMD-V
> >
> > L1d cache:             16K
> >
> > L1i cache:             64K
> >
> > L2 cache:              2048K
> >
> > L3 cache:              6144K
> >
> > NUMA node0 CPU(s):     0,2,4,6,8,10,12,14
> >
> > NUMA node1 CPU(s):     16,18,20,22,24,26,28,30
> >
> > NUMA node2 CPU(s):     1,3,5,7,9,11,13,15
> >
> > NUMA node3 CPU(s):     17,19,21,23,25,27,29,31
> >
> >
> >
> > Would also like to know how I can reduce my packet loss. Below is the
> > output of broctl netstats
> >
> > broctl netstats
> >
> > worker-1-1: 1568908277.861813 recvd=12248845422 dropped=5171188999
> > link=17420313882
> >
> > worker-1-2: 1568908298.313954 recvd=8636707266 dropped=971489
> > link=8637678939
> >
> > worker-1-3: 1568908278.425888 recvd=11684808853 dropped=5617381647
> > link=17302473791
> >
> > worker-1-4: 1568908285.731130 recvd=12567242226 dropped=4339688288
> > link=16907212802
> >
> > worker-1-5: 1568908298.363911 recvd=8620499351 dropped=24595149
> > link=8645095758
> >
> > worker-1-6: 1568908298.372892 recvd=8710565757 dropped=1731022
> > link=8712297432
> >
> > worker-1-7: 1568908298.266010 recvd=9065207444 dropped=53523232
> > link=9118737229
> >
> > worker-1-8: 1568908286.935607 recvd=11377790124 dropped=3680887247
> > link=15058934491
> >
> > worker-1-9: 1568908298.419657 recvd=8931903322 dropped=39696184
> > link=8971604219
> >
> > worker-1-10: 1568908298.478576 recvd=8842874030 dropped=2501252
> > link=8845376352
> >
> > worker-1-11: 1568908298.506649 recvd=8692769329 dropped=2253413
> > link=8695025626
> >
> > worker-1-12: 1568908298.520830 recvd=8749977028 dropped=2314733
> > link=8752293714
> >
> > worker-1-13: 1568908298.544573 recvd=9101243757 dropped=1779460
> > link=9103025399
> >
> > worker-1-14: 1568908291.370011 recvd=10876925726 dropped=775722632
> > link=11652810353
> >
> > worker-1-15: 1568908298.579721 recvd=8503097394 dropped=1420699
> > link=8504520066
> >
> > worker-1-16: 1568908298.594942 recvd=8515164266 dropped=1840977
> > link=8517006779
> >
> > worker-1-17: 1568908298.646966 recvd=10666567717 dropped=466489754
> > link=11133059283
> >
> > worker-1-18: 1568908298.671246 recvd=9023603573 dropped=2037607
> > link=9025642263
> >
> > worker-1-19: 1568908298.704675 recvd=8907784186 dropped=1164594
> > link=8908950238
> >
> > worker-1-20: 1568908298.718084 recvd=9140525444 dropped=2028593
> > link=9142555259
> >
> >
> >
> > Thanks,
> >
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> >
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190925/3f79cb4e/attachment-0001.html 

From jasonchen-work at outlook.com  Wed Sep 25 07:27:02 2019
From: jasonchen-work at outlook.com (=?gb2312?B?s8Igwfo=?=)
Date: Wed, 25 Sep 2019 14:27:02 +0000
Subject: [Zeek] Outdated "starting points" page
Message-ID: <SG2PR06MB33229BB4BD16899CF0C8BE6989870@SG2PR06MB3322.apcprd06.prod.outlook.com>

Hi,

recently I'm staring looking at Zeek's source code for learning. But I notice that the doc in zeek.org<http://zeek.org> seems to be outdated, for example, the page "starting points? (https://www.zeek.org/development/howtos/starting-points.html). On this page, it mentioned DPM.cc/.h<http://DPM.cc/.h> and AnalyzerTags.h. But I find them in release 2.1rather than release 2.6.

It will be more friendly for newbie like me if it be updated or noted with version number.

Regards.

-- Jason

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190925/05a2d429/attachment.html 

From vlad at es.net  Wed Sep 25 09:26:20 2019
From: vlad at es.net (Vlad Grigorescu)
Date: Wed, 25 Sep 2019 16:26:20 +0000
Subject: [Zeek] SSH auth_success state true set,
	but admin claims no logins
In-Reply-To: <CAPqbkwuF+mWrYQriqbv5D8W3+ECBPOLj2DnS3V8X75n-hG-ovA@mail.gmail.com>
References: <C34FDF4E-8E46-4746-98E0-AB15720B2057@virginia.edu>
	<CADju=b7huukzXZenC1a=5S6y5tDk5aD7db+1XFG_YF2bZxFVvQ@mail.gmail.com>
	<CAPqbkwuF+mWrYQriqbv5D8W3+ECBPOLj2DnS3V8X75n-hG-ovA@mail.gmail.com>
Message-ID: <CAPqbkwtersWwemvwz5N5F8zxn6V7XcTW+Vg9PPUS4Bqj_pXy_w@mail.gmail.com>

Jeffrey,

I wanted to follow up on this and see if you were able to determine
any additional information.

Thanks!

  --Vlad

On Mon, Sep 16, 2019 at 9:39 PM Vlad Grigorescu <vlad at es.net> wrote:
>
> Hi Jeffrey,
>
> The SSH detection /should/ be fairly solid. I really tried to err on the side of caution, and to not make a determination if there was some room for doubt.
>
> I haven't heard any reports about what specifically might cause a false positive, but I would guess: some uncommon SSH option (e.g. a large banner?) or some aggressive TCP settings.
>
> If you can duplicate this by trying to login against this server, and could share an anonymized PCAP, I'll work updating the analyzer.
>
> Thanks,
>
>   --Vlad
>
> On Mon, Sep 16, 2019 at 7:17 PM Jim Mellander <jmellander at lbl.gov> wrote:
>>
>> Since Zeek only sees the encrypted traffic of an ssh session, it can only make a best-guess based on packet-size analysis, which is not necessarily going to be 100% accurate.
>>
>> On Mon, Sep 16, 2019 at 11:24 AM Collyer, Jeffrey W (jwc3f) <jwc3f at virginia.edu> wrote:
>>>
>>> So recently I saw an SSH login to a device from outside the US.  I reported it to the end system admin.  The Zeek log set the auth_success state to true, but the admin of the box claims no successful login and is pushing back that it is a false positive.
>>>
>>> Have other Zeek users ever seen this?  Is the SSH auth state detection mistaken here?
>>> I don?t have pcaps to verify one way to the other, sadly.
>>>
>>>
>>> {"_path":"ssh","_system_name":"corelight","_write_ts":"2019-09-12T22:26:32.106142Z","ts":"2019-09-12T22:26:31.226136Z","uid":"C95i0o2Jl77LXHb2R9","id.orig_h?:?x.x.x.x","id.orig_p":49670,"id.resp_h?:?x.x.x.x","id.resp_p":22,"version":2,"auth_success":true,"auth_attempts":1,"direction":"INBOUND","client":"SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4","server":"SSH-2.0-OpenSSH_7.4","cipher_alg":"chacha20-poly1305 at openssh.com","mac_alg":"umac-64-etm at openssh.com","compression_alg":"none","kex_alg":"curve25519-sha256 at libssh.org","host_key_alg":"ecdsa-sha2-nistp256","host_key":"68:1e:68:89:5e:e5:20:72:f7:e6:bf:21:de:07:3a:b1?}
>>>
>>> Can anyone shed light on this?
>>>
>>> Thanks
>>> Jeff
>>>
>>>
>>> Jeffrey Collyer
>>> Information Security Engineer
>>> University of Virginia
>>> jwc3f at virginia.edu
>>> _______________________________________________
>>> Zeek mailing list
>>> zeek at zeek.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>> _______________________________________________
>> Zeek mailing list
>> zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek


From muthm at isc.upenn.edu  Wed Sep 25 10:17:18 2019
From: muthm at isc.upenn.edu (Muth, Melissa R)
Date: Wed, 25 Sep 2019 17:17:18 +0000
Subject: [Zeek] SSH auth_success state true set,
 but admin claims no logins
In-Reply-To: <CADju=b7huukzXZenC1a=5S6y5tDk5aD7db+1XFG_YF2bZxFVvQ@mail.gmail.com>
References: <C34FDF4E-8E46-4746-98E0-AB15720B2057@virginia.edu>
	<CADju=b7huukzXZenC1a=5S6y5tDk5aD7db+1XFG_YF2bZxFVvQ@mail.gmail.com>
Message-ID: <0E2BDBA0-1AE1-41F1-8CFF-1E5DD3B8F95A@upenn.edu>

> The Zeek log set the auth_success state to true, but the admin of the box claims no successful login and is pushing back that it is a false positive.

> Have other Zeek users ever seen this?  Is the SSH auth state detection mistaken here?

It?s been my experience that auth_success isn?t reliable enough to be actionable.

Melissa
- -

Melissa Muth

IT Architect, Office of Information Security

Information Systems & Computing

University of Pennsylvania

muthm at isc.upenn.edu   215-573-6798

From: <zeek-bounces at zeek.org> on behalf of Jim Mellander <jmellander at lbl.gov>
Date: Monday, September 16, 2019 at 3:09 PM
To: "Collyer, Jeffrey W (jwc3f)" <jwc3f at virginia.edu>
Cc: Bro <bro at bro.org>
Subject: Re: [Zeek] SSH auth_success state true set, but admin claims no logins

Since Zeek only sees the encrypted traffic of an ssh session, it can only make a best-guess based on packet-size analysis, which is not necessarily going to be 100% accurate.

On Mon, Sep 16, 2019 at 11:24 AM Collyer, Jeffrey W (jwc3f) <jwc3f at virginia.edu<mailto:jwc3f at virginia.edu>> wrote:
So recently I saw an SSH login to a device from outside the US.  I reported it to the end system admin.  The Zeek log set the auth_success state to true, but the admin of the box claims no successful login and is pushing back that it is a false positive.

Have other Zeek users ever seen this?  Is the SSH auth state detection mistaken here?
I don?t have pcaps to verify one way to the other, sadly.


{"_path":"ssh","_system_name":"corelight","_write_ts":"2019-09-12T22:26:32.106142Z","ts":"2019-09-12T22:26:31.226136Z","uid":"C95i0o2Jl77LXHb2R9","id.orig_h?:?x.x.x.x","id.orig_p":49670,"id.resp_h?:?x.x.x.x","id.resp_p":22,"version":2,"auth_success":true,"auth_attempts":1,"direction":"INBOUND","client":"SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4","server":"SSH-2.0-OpenSSH_7.4","cipher_alg":"chacha20-poly1305 at openssh.com<mailto:chacha20-poly1305 at openssh.com>","mac_alg":"umac-64-etm at openssh.com<mailto:umac-64-etm at openssh.com>","compression_alg":"none","kex_alg":"curve25519-sha256 at libssh.org<mailto:curve25519-sha256 at libssh.org>","host_key_alg":"ecdsa-sha2-nistp256","host_key":"68:1e:68:89:5e:e5:20:72:f7:e6:bf:21:de:07:3a:b1?}

Can anyone shed light on this?

Thanks
Jeff


Jeffrey Collyer
Information Security Engineer
University of Virginia
jwc3f at virginia.edu<mailto:jwc3f at virginia.edu>
_______________________________________________
Zeek mailing list
zeek at zeek.org<mailto:zeek at zeek.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190925/a1902e42/attachment-0001.html 

From Kayode_Enwerem at ao.uscourts.gov  Fri Sep 27 09:27:19 2019
From: Kayode_Enwerem at ao.uscourts.gov (Kayode Enwerem)
Date: Fri, 27 Sep 2019 16:27:19 +0000
Subject: [Zeek] Why does my logger keep crashing - bro version 2.6.3
In-Reply-To: <CAKZA8xih9SnfgCBODS0cqKj=mh8jtKdFxR96j8rNMw8E6A+p=w@mail.gmail.com>
References: <BN6PR09MB11542B9F128BD9B7413F842BA1890@BN6PR09MB1154.namprd09.prod.outlook.com>
	<BN6PR09MB115443C06D491DA302F640F3A1890@BN6PR09MB1154.namprd09.prod.outlook.com>
	<CAKZA8xjFZpcsZQ-JVXyT5ZrAaZ0bL7VObV+69NRejn6Wyy4AzQ@mail.gmail.com>
	<BN6PR09MB11542783738CE7EFBDEF8324A1850@BN6PR09MB1154.namprd09.prod.outlook.com>
	<CAPqbkwtZSuqUSTLJFDGfpWegyA=gucOZmGbdqvqxoSboLs55jQ@mail.gmail.com>
	<BN6PR09MB1154540BEAE66A7A8991C63EA1840@BN6PR09MB1154.namprd09.prod.outlook.com>
	<CAKZA8xih9SnfgCBODS0cqKj=mh8jtKdFxR96j8rNMw8E6A+p=w@mail.gmail.com>
Message-ID: <BN6PR09MB11544ACBD5A65A117708FABDA1810@BN6PR09MB1154.namprd09.prod.outlook.com>

Looks like setting up 2 loggers resolved the issue of my logger crashing but my Dropped packets are pretty high on my workers. Can someone assist me with how I can reduce my dropped packets.

cat capture_loss.log
#separator \x09
#set_separator  ,
#empty_field    (empty)
#unset_field    -
#path   capture_loss
#open   2019-09-27-12-05-05
#fields ts      ts_delta        peer    gaps    acks    percent_lost
#types  time    interval        string  count   count   double
1569600304.774215       900.000013      worker-1-1      126463  3246542 3.895314
1569600304.783703       900.000064      worker-1-3      106904  4465333 2.394088
1569600304.785983       900.000212      worker-1-11     123729  3768503 3.28324
1569600304.802244       900.000098      worker-1-14     144154  3584013 4.022139
1569600304.823378       900.000095      worker-1-18     137507  3503583 3.924754
1569600304.892559       900.000470      worker-1-13     148904  3448544 4.31788
1569600305.010986       900.000030      worker-1-8      174213  3409819 5.109157
1569600305.938686       901.043465      worker-1-15     509268  1072199 47.497526
1569600304.806850       900.000047      worker-1-22     591232  1234893 47.877185
1569601204.762382       900.000786      worker-1-16     120086  4491072 2.673883
1569601204.774220       900.000005      worker-1-1      127257  3461349 3.676515
1569601204.802447       900.000203      worker-1-14     125481  3171663 3.956316
1569601204.884438       900.000029      worker-1-19     125037  3566663 3.505714
1569601204.891746       900.000015      worker-1-23     120553  3078889 3.915471
1569601205.110098       900.000139      worker-1-10     108016  3442813 3.137434
1569601205.938906       900.000220      worker-1-15     565536  1156759 48.8897
1569601218.120290       900.000047      worker-1-6      456312  753749  60.538986

Below are some of my settings:

I have 23 workers defined and I pinned CPU.
[worker-1]
type=worker
host=localhost
interface=af_packet::ens2f0
lb_method=custom
#lb_method=pf_ring
lb_procs=23
pin_cpus=5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27
af_packet_fanout_id=25
af_packet_fanout_mode=AF_Packet::FANOUT_HASH

Can someone assist me with this.

Thanks.


From: william de ping <bill.de.ping at gmail.com>
Sent: Wednesday, September 25, 2019 4:00 AM
To: Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov>
Cc: zeek at zeek.org
Subject: Re: [Zeek] Why does my logger keep crashing - bro version 2.6.3

Hi

Try using the None writer instead of the ASCII one.
In local.bro add :
redef Log::default_writer=Log::WRITER_NONE;

If the logger instance still crashes then the issue is not related to an IO bottleneck.

B

On Tue, Sep 24, 2019 at 7:49 PM Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov<mailto:Kayode_Enwerem at ao.uscourts.gov>> wrote:
Thanks for your response.

I do see the following OOM message in my system logs on the logger process ID:
Sep 23 18:48:00 kernel: Out of memory: Kill process 10439 (bro) score 787 or sacrifice child
Sep 23 18:48:00 kernel: Killed process 10439 (bro), UID 0, total-vm:301983900kB, anon-rss:195261772kB, file-rss:2592kB, shmem-rss:0kB

Wonder why its taking so much memory, I have 251G and 99G swap on this server.
total        used        free      shared  buff/cache   available
Mem:           251G         66G        185G        4.2M        488M        184G
Swap:           99G        1.1G         98G

Below is the output of "broctl diag logger", ran after the logger crashed.

 [logger]

No core file found.

Bro 2.6.3
Linux 3.10.0-1062.1.1.el7.x86_64

Bro plugins:
Bro::AF_Packet - Packet acquisition via AF_Packet (dynamic, version 1.4)

==== No reporter.log

==== stderr.log
/usr/local/bro/share/broctl/scripts/run-bro: line 110: 10439 Killed                  nohup "$mybro" "$@"

==== stdout.log
max memory size         (kbytes, -m) unlimited
data seg size           (kbytes, -d) unlimited
virtual memory          (kbytes, -v) unlimited
core file size          (blocks, -c) unlimited

==== .cmdline
-U .status -p broctl -p broctl-live -p local -p logger local.bro broctl base/frameworks/cluster broctl/auto

==== .env_vars
PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bro/bin
BROPATH=/logs/bro/spool/installed-scripts-do-not-touch/site::/logs/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site
CLUSTER_NODE=logger

==== .status
RUNNING [net_run]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log

Thoughts? Any suggestions.

-----Original Message-----
From: Vlad Grigorescu <vlad at es.net<mailto:vlad at es.net>>
Sent: Monday, September 23, 2019 10:20 AM
To: Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov<mailto:Kayode_Enwerem at ao.uscourts.gov>>
Cc: william de ping <bill.de.ping at gmail.com<mailto:bill.de.ping at gmail.com>>; zeek at zeek.org<mailto:zeek at zeek.org>
Subject: Re: [Zeek] Why does my logger keep crashing - bro version 2.6.3

The logger is threaded, so seeing CPU > 100% is not necessarily a problem.

Have you tried running "broctl diag logger" to see why the logger is crashing? Do you have any messages in your system logs about processing being killed for out of memory (OOM)?

  --Vlad

On Mon, Sep 23, 2019 at 1:32 PM Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov<mailto:Kayode_Enwerem at ao.uscourts.gov>> wrote:
>
> Thanks for your response. The CPU usage for the logger is at 311%. (look below).
>
>
>
> broctl top
>
> Name         Type    Host             Pid     VSize  Rss  Cpu   Cmd
>
> logger       logger  localhost        22867    12G     9G 311%  bro
>
>
>
> I wasn?t aware that you could set up multiple loggers, I tried checking the docs to see if that was an option. Does anyone know how to do this?
>
>
>
> From: william de ping <bill.de.ping at gmail.com<mailto:bill.de.ping at gmail.com>>
> Sent: Sunday, September 22, 2019 6:42 AM
> To: Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov<mailto:Kayode_Enwerem at ao.uscourts.gov>>
> Cc: zeek at zeek.org<mailto:zeek at zeek.org>
> Subject: Re: [Zeek] Why does my logger keep crashing - bro version
> 2.6.3
>
>
>
> Hi,
>
>
>
> I would try to monitor the cpu \ mem usage of the logger instance.
>
> Try running broctl top, my guess is that you will see that the logger process will have a very high cpu usage.
>
>
>
> I know of an option to have multiple loggers but I am not sure how to set it up.
>
>
>
> Are you writing to a file ?
>
>
>
> B
>
>
>
> On Thu, Sep 19, 2019 at 7:14 PM Kayode Enwerem <Kayode_Enwerem at ao.uscourts.gov<mailto:Kayode_Enwerem at ao.uscourts.gov>> wrote:
>
> Hello,
>
>
>
> Why does my logger keep crashing? Can someone please help me with this. I have provided some system information below:
>
>
>
> I am running bro version 2.6.3
>
>
>
> Output of broctl status. The logger is crashed but the manager, proxy and workers are still running.
>
> broctl status
>
> Name         Type    Host             Status    Pid    Started
>
> logger       logger  localhost        crashed
>
> manager      manager localhost        running   17356  09 Sep 15:42:24
>
> proxy-1      proxy   localhost        running   17401  09 Sep 15:42:25
>
> worker-1-1   worker  localhost        running   17573  09 Sep 15:42:27
>
> worker-1-2   worker  localhost        running   17569  09 Sep 15:42:27
>
> worker-1-3   worker  localhost        running   17572  09 Sep 15:42:27
>
> worker-1-4   worker  localhost        running   17587  09 Sep 15:42:27
>
> worker-1-5   worker  localhost        running   17619  09 Sep 15:42:27
>
> worker-1-6   worker  localhost        running   17614  09 Sep 15:42:27
>
> worker-1-7   worker  localhost        running   17625  09 Sep 15:42:27
>
> worker-1-8   worker  localhost        running   17646  09 Sep 15:42:27
>
> worker-1-9   worker  localhost        running   17671  09 Sep 15:42:27
>
> worker-1-10  worker  localhost        running   17663  09 Sep 15:42:27
>
> worker-1-11  worker  localhost        running   17679  09 Sep 15:42:27
>
> worker-1-12  worker  localhost        running   17685  09 Sep 15:42:27
>
> worker-1-13  worker  localhost        running   17698  09 Sep 15:42:27
>
> worker-1-14  worker  localhost        running   17703  09 Sep 15:42:27
>
> worker-1-15  worker  localhost        running   17710  09 Sep 15:42:27
>
> worker-1-16  worker  localhost        running   17717  09 Sep 15:42:27
>
> worker-1-17  worker  localhost        running   17720  09 Sep 15:42:27
>
> worker-1-18  worker  localhost        running   17727  09 Sep 15:42:27
>
> worker-1-19  worker  localhost        running   17728  09 Sep 15:42:27
>
> worker-1-20  worker  localhost        running   17731  09 Sep 15:42:27
>
>
>
> Here?s my node.cfg settings
>
> [logger]
>
> type=logger
>
> host=localhost
>
>
>
> [manager]
>
> type=manager
>
> host=localhost
>
>
>
> [proxy-1]
>
> type=proxy
>
> host=localhost
>
>
>
> [worker-1]
>
> type=worker
>
> host=localhost
>
> interface=af_packet::ens2f0
>
> lb_method=custom
>
> #lb_method=pf_ring
>
> lb_procs=20
>
> pin_cpus=6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25
>
> af_packet_fanout_id=25
>
> af_packet_fanout_mode=AF_Packet::FANOUT_HASH
>
>
>
> Heres more information on my CPU. 32 CPUs, model name ? AMD, CPU max
> MHz is 2800.0000
>
> Architecture:          x86_64
>
> CPU op-mode(s):        32-bit, 64-bit
>
> Byte Order:            Little Endian
>
> CPU(s):                32
>
> On-line CPU(s) list:   0-31
>
> Thread(s) per core:    2
>
> Core(s) per socket:    8
>
> Socket(s):             2
>
> NUMA node(s):          4
>
> Vendor ID:             AuthenticAMD
>
> CPU family:            21
>
> Model:                 2
>
> Model name:            AMD Opteron(tm) Processor 6386 SE
>
> Stepping:              0
>
> CPU MHz:               1960.000
>
> CPU max MHz:           2800.0000
>
> CPU min MHz:           1400.0000
>
> BogoMIPS:              5585.93
>
> Virtualization:        AMD-V
>
> L1d cache:             16K
>
> L1i cache:             64K
>
> L2 cache:              2048K
>
> L3 cache:              6144K
>
> NUMA node0 CPU(s):     0,2,4,6,8,10,12,14
>
> NUMA node1 CPU(s):     16,18,20,22,24,26,28,30
>
> NUMA node2 CPU(s):     1,3,5,7,9,11,13,15
>
> NUMA node3 CPU(s):     17,19,21,23,25,27,29,31
>
>
>
> Would also like to know how I can reduce my packet loss. Below is the
> output of broctl netstats
>
> broctl netstats
>
> worker-1-1: 1568908277.861813 recvd=12248845422 dropped=5171188999
> link=17420313882
>
> worker-1-2: 1568908298.313954 recvd=8636707266 dropped=971489
> link=8637678939
>
> worker-1-3: 1568908278.425888 recvd=11684808853 dropped=5617381647
> link=17302473791
>
> worker-1-4: 1568908285.731130 recvd=12567242226 dropped=4339688288
> link=16907212802
>
> worker-1-5: 1568908298.363911 recvd=8620499351 dropped=24595149
> link=8645095758
>
> worker-1-6: 1568908298.372892 recvd=8710565757 dropped=1731022
> link=8712297432
>
> worker-1-7: 1568908298.266010 recvd=9065207444 dropped=53523232
> link=9118737229
>
> worker-1-8: 1568908286.935607 recvd=11377790124 dropped=3680887247
> link=15058934491
>
> worker-1-9: 1568908298.419657 recvd=8931903322 dropped=39696184
> link=8971604219
>
> worker-1-10: 1568908298.478576 recvd=8842874030 dropped=2501252
> link=8845376352
>
> worker-1-11: 1568908298.506649 recvd=8692769329 dropped=2253413
> link=8695025626
>
> worker-1-12: 1568908298.520830 recvd=8749977028 dropped=2314733
> link=8752293714
>
> worker-1-13: 1568908298.544573 recvd=9101243757 dropped=1779460
> link=9103025399
>
> worker-1-14: 1568908291.370011 recvd=10876925726 dropped=775722632
> link=11652810353
>
> worker-1-15: 1568908298.579721 recvd=8503097394 dropped=1420699
> link=8504520066
>
> worker-1-16: 1568908298.594942 recvd=8515164266 dropped=1840977
> link=8517006779
>
> worker-1-17: 1568908298.646966 recvd=10666567717 dropped=466489754
> link=11133059283
>
> worker-1-18: 1568908298.671246 recvd=9023603573 dropped=2037607
> link=9025642263
>
> worker-1-19: 1568908298.704675 recvd=8907784186 dropped=1164594
> link=8908950238
>
> worker-1-20: 1568908298.718084 recvd=9140525444 dropped=2028593
> link=9142555259
>
>
>
> Thanks,
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org<mailto:zeek at zeek.org>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org<mailto:zeek at zeek.org>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190927/b6bd6a71/attachment-0001.html 

From nothinrandom at gmail.com  Fri Sep 27 09:33:34 2019
From: nothinrandom at gmail.com (TQ)
Date: Fri, 27 Sep 2019 09:33:34 -0700
Subject: [Zeek] Segmentation Fault on Zeek 3.0.0
Message-ID: <CAM2MKSpcxdzScdfW8pOuRjaPaxrmFzOPgWX=Y32P728dQz5YKg@mail.gmail.com>

Hello Zeekers,

I'm currently in the process of migrating from Bro 2.6.2 to Zeek 3.0.0, and
I'm experiencing a small headache with segmentation fault in my plugins.  I
didn't have this issue with Bro 2.6.2, so I'm not 100% sure what happened
here.

After making name changes from Bro to Zeek, I was able to successfully
compile all of the plugins.  When I ran them against pcaps that are
specified for the plugin, I noticed that some of the plugins threw a
segmentation fault ("Segmentation fault (core dumped)").  I was replaying a
pcap file like what I usually do by running:
cd ~/Desktop/logs/ && sudo rm -f *.log && zeek -C -t
~/Desktop/logs/output.log -r ~/Desktop/pcap/ testPlugin1_pcap_1.pcapng

After some troubleshooting, I noticed that only the ones that had a switch
case statement inside a while loop inside main.zeek were affected by this.
I do have checks to prevent resource exhaustion, so I'm not sure why the
new version is not happy.  Anyway, I was able to verify by cd into
"/usr/local/zeek/lib/zeek/plugins/Zeek_testPlugin1/scripts" and commenting
out the affected section in main.zeek.  Even something as simple as this
throws segmentation fault:

while (index < payload_length) {
    header = bytestring_to_count(data[index]);
    len = 0;
    index += 1;
    switch (header) {
        default: ##! test
            break;
    }
    #  dummy check as example
    if (index > 10) {
        break;
    }
}

I've been looking at this for the last 8 hours, so more eyes would be
appreciated.

Thanks,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190927/aed8cb15/attachment.html 

From jsiwek at corelight.com  Fri Sep 27 17:15:58 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Fri, 27 Sep 2019 17:15:58 -0700
Subject: [Zeek] Segmentation Fault on Zeek 3.0.0
In-Reply-To: <CAM2MKSpcxdzScdfW8pOuRjaPaxrmFzOPgWX=Y32P728dQz5YKg@mail.gmail.com>
References: <CAM2MKSpcxdzScdfW8pOuRjaPaxrmFzOPgWX=Y32P728dQz5YKg@mail.gmail.com>
Message-ID: <CAMzgZ0JQoH8n5xu0K=OTXHvAHGkHEibZLzR3vZEQ7obDBhZeBg@mail.gmail.com>

On Fri, Sep 27, 2019 at 9:47 AM TQ <nothinrandom at gmail.com> wrote:

> cd ~/Desktop/logs/ && sudo rm -f *.log && zeek -C -t ~/Desktop/logs/output.log -r ~/Desktop/pcap/ testPlugin1_pcap_1.pcapng

The `-t` option isn't commonly used and could see it accidentally
breaking without anyone noticing.  It does still seem to work for me,
but you might try removing it to see if it makes a difference.

But the best thing would be if you can provide the full directions to
be able to reproduce the segfault -- e.g. the plugin/script code along
with pcap and command-line you're using.

If you can't share those, then next best thing would be if you can run
in a debugger (gdb, lldb) and share a stack trace of the segfault.

- Jon

From nothinrandom at gmail.com  Fri Sep 27 19:56:55 2019
From: nothinrandom at gmail.com (TQ)
Date: Fri, 27 Sep 2019 19:56:55 -0700
Subject: [Zeek] Segmentation Fault on Zeek 3.0.0
In-Reply-To: <CAMzgZ0JQoH8n5xu0K=OTXHvAHGkHEibZLzR3vZEQ7obDBhZeBg@mail.gmail.com>
References: <CAM2MKSpcxdzScdfW8pOuRjaPaxrmFzOPgWX=Y32P728dQz5YKg@mail.gmail.com>
	<CAMzgZ0JQoH8n5xu0K=OTXHvAHGkHEibZLzR3vZEQ7obDBhZeBg@mail.gmail.com>
Message-ID: <CAM2MKSoXv+hOBi1bg=Jz-ULE4=_Uke9=NdE53aNJ2_dfy7D-fQ@mail.gmail.com>

Hey Jon,

Thanks for guidance on this!  You are absolutely right.  If I remove "-t
~/Desktop/logs/output.log", then that segmentation fault goes away.  I have
not a clue why as it works fine for 2.6.2.  I thought something was wrong
with the actual code.  Again, thanks for helping out with this!

Thanks,

On Fri, Sep 27, 2019 at 5:16 PM Jon Siwek <jsiwek at corelight.com> wrote:

> On Fri, Sep 27, 2019 at 9:47 AM TQ <nothinrandom at gmail.com> wrote:
>
> > cd ~/Desktop/logs/ && sudo rm -f *.log && zeek -C -t
> ~/Desktop/logs/output.log -r ~/Desktop/pcap/ testPlugin1_pcap_1.pcapng
>
> The `-t` option isn't commonly used and could see it accidentally
> breaking without anyone noticing.  It does still seem to work for me,
> but you might try removing it to see if it makes a difference.
>
> But the best thing would be if you can provide the full directions to
> be able to reproduce the segfault -- e.g. the plugin/script code along
> with pcap and command-line you're using.
>
> If you can't share those, then next best thing would be if you can run
> in a debugger (gdb, lldb) and share a stack trace of the segfault.
>
> - Jon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190927/f7e56b14/attachment.html 

From jsiwek at corelight.com  Mon Sep 30 09:37:07 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Mon, 30 Sep 2019 09:37:07 -0700
Subject: [Zeek] Segmentation Fault on Zeek 3.0.0
In-Reply-To: <CAM2MKSoXv+hOBi1bg=Jz-ULE4=_Uke9=NdE53aNJ2_dfy7D-fQ@mail.gmail.com>
References: <CAM2MKSpcxdzScdfW8pOuRjaPaxrmFzOPgWX=Y32P728dQz5YKg@mail.gmail.com>
	<CAMzgZ0JQoH8n5xu0K=OTXHvAHGkHEibZLzR3vZEQ7obDBhZeBg@mail.gmail.com>
	<CAM2MKSoXv+hOBi1bg=Jz-ULE4=_Uke9=NdE53aNJ2_dfy7D-fQ@mail.gmail.com>
Message-ID: <CAMzgZ0LiyNmCcwS1Xz=v4Uu4YYQwFVy=uDGKrhHMB3qMUVN=LA@mail.gmail.com>

Can you provide more information on how to reproduce the issue (exact
scripts/plugins/pcaps that crash every time) ?  There's still a bug in
Zeek to fix here, but just adding `-t` and trying a few things hasn't
triggered it for me.

- Jon

On Fri, Sep 27, 2019 at 7:57 PM TQ <nothinrandom at gmail.com> wrote:
>
> Hey Jon,
>
> Thanks for guidance on this!  You are absolutely right.  If I remove "-t ~/Desktop/logs/output.log", then that segmentation fault goes away.  I have not a clue why as it works fine for 2.6.2.  I thought something was wrong with the actual code.  Again, thanks for helping out with this!
>
> Thanks,
>
> On Fri, Sep 27, 2019 at 5:16 PM Jon Siwek <jsiwek at corelight.com> wrote:
>>
>> On Fri, Sep 27, 2019 at 9:47 AM TQ <nothinrandom at gmail.com> wrote:
>>
>> > cd ~/Desktop/logs/ && sudo rm -f *.log && zeek -C -t ~/Desktop/logs/output.log -r ~/Desktop/pcap/ testPlugin1_pcap_1.pcapng
>>
>> The `-t` option isn't commonly used and could see it accidentally
>> breaking without anyone noticing.  It does still seem to work for me,
>> but you might try removing it to see if it makes a difference.
>>
>> But the best thing would be if you can provide the full directions to
>> be able to reproduce the segfault -- e.g. the plugin/script code along
>> with pcap and command-line you're using.
>>
>> If you can't share those, then next best thing would be if you can run
>> in a debugger (gdb, lldb) and share a stack trace of the segfault.
>>
>> - Jon


From Kayode_Enwerem at ao.uscourts.gov  Mon Sep 30 10:43:19 2019
From: Kayode_Enwerem at ao.uscourts.gov (Kayode Enwerem)
Date: Mon, 30 Sep 2019 17:43:19 +0000
Subject: [Zeek] How can I reduce my packet loss - bro version 2.6.3
Message-ID: <BN6PR09MB1154190F1B49CAC7C4DF594DA1820@BN6PR09MB1154.namprd09.prod.outlook.com>

Hello,

My packet loss on my workers is pretty high. I have done things like CPU pinning but its still high. Can you please assist me in how I can reduce this to under 1%. Below are some of my settings.

cat capture_loss.log - Percent lost ranges from 2.7 to about 67.69
#fields ts      ts_delta        peer    gaps    acks    percent_lost
#types  time    interval        string  count   count   double
1569863104.831317       900.000030      worker-1-3      114384  3799935 3.010157
1569863104.851327       900.000002      worker-1-1      162671  3677320 4.423629
1569863104.841705       900.000062      worker-1-9      100444  3393374 2.960004
1569863104.843460       900.000058      worker-1-11     148576  4171807 3.56143
1569863104.855034       900.000116      worker-1-16     165242  3769560 4.383589
1569863104.937666       900.000094      worker-1-23     124377  3891351 3.196242
1569863104.811991       900.000040      worker-1-12     339309  3176448 10.682026
1569863104.853635       900.000036      worker-1-7      304519  3266968 9.32115
1569863105.107706       900.000013      worker-1-8      296921  3475658 8.542872
1569863117.781723       900.739890      worker-1-15     635032  1385280 45.841418
1569863114.375886       900.000010      worker-1-17     631085  2596009 24.309816
1569863118.295945       900.001869      worker-1-6      369130  545290  67.694254
1569863160.141238       900.000229      worker-1-22     774134  1785146 43.365305
1569864004.845052       900.001592      worker-1-11     108257  3871564 2.796208
1569864004.860404       900.000040      worker-1-14     111798  3327087 3.360237
1569864004.937679       900.000013      worker-1-23     148738  3568913 4.167599
1569864004.951235       900.000218      worker-1-19     96672   3509661 2.754454
1569864004.976291       900.000009      worker-1-21     152430  3736550 4.079432
1569864005.211316       900.000025      worker-1-4      176180  3226673 5.460113
1569864005.193565       900.000005      worker-1-10     148535  3986455 3.725992
1569864004.811996       900.000005      worker-1-12     289760  3487997 8.307347
1569864005.107761       900.000055      worker-1-8      270405  3340848 8.093903
1569864014.375894       900.000008      worker-1-17     517282  2462898 21.002981
1569864018.295953       900.000008      worker-1-6      362966  571162  63.548695
1569864060.141335       900.000097      worker-1-22     601802  1520920 39.568288

cat node.cfg (below is the worker config in node.cfg. As you can see I pinned 23 CPUs)
[worker-1]
type=worker
host=localhost
interface=af_packet::ens2f0
lb_method=custom
#lb_method=pf_ring
lb_procs=23
pin_cpus=5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27
af_packet_fanout_id=25
af_packet_fanout_mode=AF_Packet::FANOUT_HASH

I have 32 CPUs on this server and CPU model name is - AMD Opteron(tm) Processor 6386 SE

CPU MHz:               2800.000
CPU max MHz:           2800.0000
CPU min MHz:           1400.0000

Please assist. Thanks.

Thanks,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190930/e5aaeba2/attachment.html 

From justin at corelight.com  Mon Sep 30 11:32:42 2019
From: justin at corelight.com (Justin Azoff)
Date: Mon, 30 Sep 2019 14:32:42 -0400
Subject: [Zeek] How can I reduce my packet loss - bro version 2.6.3
In-Reply-To: <BN6PR09MB1154190F1B49CAC7C4DF594DA1820@BN6PR09MB1154.namprd09.prod.outlook.com>
References: <BN6PR09MB1154190F1B49CAC7C4DF594DA1820@BN6PR09MB1154.namprd09.prod.outlook.com>
Message-ID: <CAPfnCuhLD-KiQg7Vgn+w7C8WFXWjZ+=unZBr48MMemo__hpoxQ@mail.gmail.com>

On Mon, Sep 30, 2019 at 1:45 PM Kayode Enwerem <
Kayode_Enwerem at ao.uscourts.gov> wrote:

>
> I have 32 CPUs on this server and CPU model name is - AMD Opteron(tm)
> Processor 6386 SE
>

People have been having issues with older opterons like that for a long
time.  They have a lot of cores, but the single core performance is about
half that of a more recent CPU.

With 32  real cores (assuming this is a dual socket system) I'd try running
closer to 28 workers which gives you 20% more capacity over  23.

After that, you need to look at the conn.log to determine where your
capture loss is coming from by looking at the missed_bytes column.  You may
have some elephant flows that are accounting for the majority of that loss.

-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190930/ba862004/attachment-0001.html