From clopmz at outlook.com  Mon Oct  1 01:47:35 2018
From: clopmz at outlook.com (Carlos Lopez)
Date: Mon, 1 Oct 2018 08:47:35 +0000
Subject: [Bro] OT: SuperMicro hardware recommendations
Message-ID: <94D13236-BB53-4807-93D0-F4A1BF0B369F@outlook.com>

Hi all,

 In the following days, we will setup a Bro-IDS distributed architecture using 10 sensors and one manager. Network bandwidth will be between 2GiB-4GiB. Until now we used HP servers, but we are looking at SuperMicro servers as well. Can anyone recommend series or model?
-- 
Regards,
C. L. Martinez



From michalpurzynski1 at gmail.com  Mon Oct  1 03:47:47 2018
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Mon, 1 Oct 2018 12:47:47 +0200
Subject: [Bro] OT: SuperMicro hardware recommendations
In-Reply-To: <94D13236-BB53-4807-93D0-F4A1BF0B369F@outlook.com>
References: <94D13236-BB53-4807-93D0-F4A1BF0B369F@outlook.com>
Message-ID: <151C7D23-39E9-4170-8954-CDFF566E4ED7@gmail.com>

I?d recommended everything but super micro.  

A box of matches to shine the light on the dark corners of your network will behave better than anything built on super micro.

HP and other vendors may seem more expensive to buy, but the low quality of super micro servers, apparent lack of support, arbitrary cut support ?we do not support this anymore? and ridiculous replacement policy makes it a non-starter for me.

Example: they wanted me to send all dead hard drives back before they can send me a replacement.
And that would take days or a week or two weeks.

Raid battery problems?

SM requires you to send entire server back and in a short 6 weeks - 3 months you will see it back. Maybe.

Mechanically, chassis are a joke, too.

OOB? No OOB hardware monitoring. And Java 1.6 required. If you have to use any kind of proxies, socks, tunneling - forget.


> On Oct 1, 2018, at 10:47 AM, Carlos Lopez <clopmz at outlook.com> wrote:
> 
> Hi all,
> 
> In the following days, we will setup a Bro-IDS distributed architecture using 10 sensors and one manager. Network bandwidth will be between 2GiB-4GiB. Until now we used HP servers, but we are looking at SuperMicro servers as well. Can anyone recommend series or model?
> -- 
> Regards,
> C. L. Martinez
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From zeolla at gmail.com  Mon Oct  1 05:48:51 2018
From: zeolla at gmail.com (Zeolla@GMail.com)
Date: Mon, 1 Oct 2018 08:48:51 -0400
Subject: [Bro] OT: SuperMicro hardware recommendations
In-Reply-To: <151C7D23-39E9-4170-8954-CDFF566E4ED7@gmail.com>
References: <94D13236-BB53-4807-93D0-F4A1BF0B369F@outlook.com>
	<151C7D23-39E9-4170-8954-CDFF566E4ED7@gmail.com>
Message-ID: <CAM1mTCramNX9gZst-OxFRALqvS2vZzTDa6QU-CVXR5LvYNzNFw@mail.gmail.com>

I think they released their HTML5 interface for OOB.  That said, I'm also
not a huge fan.

Jon

On Mon, Oct 1, 2018 at 7:02 AM Micha? Purzy?ski <michalpurzynski1 at gmail.com>
wrote:

> I?d recommended everything but super micro.
>
> A box of matches to shine the light on the dark corners of your network
> will behave better than anything built on super micro.
>
> HP and other vendors may seem more expensive to buy, but the low quality
> of super micro servers, apparent lack of support, arbitrary cut support ?we
> do not support this anymore? and ridiculous replacement policy makes it a
> non-starter for me.
>
> Example: they wanted me to send all dead hard drives back before they can
> send me a replacement.
> And that would take days or a week or two weeks.
>
> Raid battery problems?
>
> SM requires you to send entire server back and in a short 6 weeks - 3
> months you will see it back. Maybe.
>
> Mechanically, chassis are a joke, too.
>
> OOB? No OOB hardware monitoring. And Java 1.6 required. If you have to use
> any kind of proxies, socks, tunneling - forget.
>
>
> > On Oct 1, 2018, at 10:47 AM, Carlos Lopez <clopmz at outlook.com> wrote:
> >
> > Hi all,
> >
> > In the following days, we will setup a Bro-IDS distributed architecture
> using 10 sensors and one manager. Network bandwidth will be between
> 2GiB-4GiB. Until now we used HP servers, but we are looking at SuperMicro
> servers as well. Can anyone recommend series or model?
> > --
> > Regards,
> > C. L. Martinez
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-- 

Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181001/734b17e7/attachment.html 

From markjx at gmail.com  Mon Oct  1 06:02:30 2018
From: markjx at gmail.com (Mark W. Jeanmougin)
Date: Mon, 1 Oct 2018 09:02:30 -0400
Subject: [Bro] OT: SuperMicro hardware recommendations
In-Reply-To: <CAM1mTCramNX9gZst-OxFRALqvS2vZzTDa6QU-CVXR5LvYNzNFw@mail.gmail.com>
References: <94D13236-BB53-4807-93D0-F4A1BF0B369F@outlook.com>
	<151C7D23-39E9-4170-8954-CDFF566E4ED7@gmail.com>
	<CAM1mTCramNX9gZst-OxFRALqvS2vZzTDa6QU-CVXR5LvYNzNFw@mail.gmail.com>
Message-ID: <CAMMgxRXdDMj5JEt116Ne5dJ2a+ZGWbdrcbdLG33RAqEXRYWvgg@mail.gmail.com>

I've only worked with a few dozen SuperMicro boxes; probably less than
100. I don't consider myself an expert.

I found the OOB management sufficient for what we needed. Maybe other
vendors are significantly better?

I found hard drive support to be exceptional. I could keep a stack of
spare drives; if something failed, I'd replace it. If I didn't happen
to have the exact model, I could swing by a Micro Center (or use
Amazon's fast delivery options) and drop in something similar. Doing
things like this was MUCH cheaper (and gave me faster time-to-repair)
than the other vendors.

Dropping in "weird" SSD's, eBay RAID controllers, HBA's from Amazon,
or network cards from NewEgg were all solid. Everything just worked.
My experience is that buying small hardware like that through a
company's "expense" program is much easier and faster (and cheaper!)
than getting a PO through a vendor to get it from an "enterprise" IT
vendor.

I *love* their 45 bay 4U boxes that I can slam off the shelf SATA
drive into. Great way to get LOTS of storage for cheap.

I never had mechanical issues with their chassis or rails. I had some
boxes in 4 post data center racks. I had some "short depth" boxes
mounted in 2post telecom racks out in my branch offices.

I know that Palo Alto Networks, Cisco, Nortel, and others used
SuperMicro to make a fair bit of their gear. Not sure if they still
do.

I've worked with hundreds of HP servers in the $250k-1m range where we
had on site personnel for hardware and software support. The annual
maintenance for a tape drive was more expensive than just buying a
spare to keep on the shelf. We had hundreds of tape drives. It was
great support, but we paid out the nose for it.

I'm just another opinion out on the Internet. :)

MJ

On Mon, Oct 1, 2018 at 8:51 AM Zeolla at GMail.com <zeolla at gmail.com> wrote:
>
> I think they released their HTML5 interface for OOB.  That said, I'm also not a huge fan.
>
> Jon
>
> On Mon, Oct 1, 2018 at 7:02 AM Micha? Purzy?ski <michalpurzynski1 at gmail.com> wrote:
>>
>> I?d recommended everything but super micro.
>>
>> A box of matches to shine the light on the dark corners of your network will behave better than anything built on super micro.
>>
>> HP and other vendors may seem more expensive to buy, but the low quality of super micro servers, apparent lack of support, arbitrary cut support ?we do not support this anymore? and ridiculous replacement policy makes it a non-starter for me.
>>
>> Example: they wanted me to send all dead hard drives back before they can send me a replacement.
>> And that would take days or a week or two weeks.
>>
>> Raid battery problems?
>>
>> SM requires you to send entire server back and in a short 6 weeks - 3 months you will see it back. Maybe.
>>
>> Mechanically, chassis are a joke, too.
>>
>> OOB? No OOB hardware monitoring. And Java 1.6 required. If you have to use any kind of proxies, socks, tunneling - forget.
>>
>>
>> > On Oct 1, 2018, at 10:47 AM, Carlos Lopez <clopmz at outlook.com> wrote:
>> >
>> > Hi all,
>> >
>> > In the following days, we will setup a Bro-IDS distributed architecture using 10 sensors and one manager. Network bandwidth will be between 2GiB-4GiB. Until now we used HP servers, but we are looking at SuperMicro servers as well. Can anyone recommend series or model?
>> > --
>> > Regards,
>> > C. L. Martinez
>> >
>> >
>> > _______________________________________________
>> > Bro mailing list
>> > bro at bro-ids.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> --
>
> Jon
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From clopmz at outlook.com  Mon Oct  1 06:16:41 2018
From: clopmz at outlook.com (Carlos Lopez)
Date: Mon, 1 Oct 2018 13:16:41 +0000
Subject: [Bro] OT: SuperMicro hardware recommendations
In-Reply-To: <151C7D23-39E9-4170-8954-CDFF566E4ED7@gmail.com>
References: <94D13236-BB53-4807-93D0-F4A1BF0B369F@outlook.com>
	<151C7D23-39E9-4170-8954-CDFF566E4ED7@gmail.com>
Message-ID: <EFAF45EB-9512-42EF-825F-A6F0DA0A5FCE@outlook.com>

Thanks Michal. I didn't know that bad support from SuperMicro.
Then I will continue with HP. Do you think the HP DL360 Gen10 are suitable for IDS deployment?

-- 
Regards,
C. L. Martinez

?On 01/10/2018, 12:47, "Micha? Purzy?ski" <michalpurzynski1 at gmail.com> wrote:

    I?d recommended everything but super micro.  
    
    A box of matches to shine the light on the dark corners of your network will behave better than anything built on super micro.
    
    HP and other vendors may seem more expensive to buy, but the low quality of super micro servers, apparent lack of support, arbitrary cut support ?we do not support this anymore? and ridiculous replacement policy makes it a non-starter for me.
    
    Example: they wanted me to send all dead hard drives back before they can send me a replacement.
    And that would take days or a week or two weeks.
    
    Raid battery problems?
    
    SM requires you to send entire server back and in a short 6 weeks - 3 months you will see it back. Maybe.
    
    Mechanically, chassis are a joke, too.
    
    OOB? No OOB hardware monitoring. And Java 1.6 required. If you have to use any kind of proxies, socks, tunneling - forget.
    
    
    > On Oct 1, 2018, at 10:47 AM, Carlos Lopez <clopmz at outlook.com> wrote:
    > 
    > Hi all,
    > 
    > In the following days, we will setup a Bro-IDS distributed architecture using 10 sensors and one manager. Network bandwidth will be between 2GiB-4GiB. Until now we used HP servers, but we are looking at SuperMicro servers as well. Can anyone recommend series or model?
    > -- 
    > Regards,
    > C. L. Martinez
    > 
    > 
    > _______________________________________________
    > Bro mailing list
    > bro at bro-ids.org
    > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
    



From alexwis at gmail.com  Mon Oct  1 06:26:06 2018
From: alexwis at gmail.com (Alex Waher)
Date: Mon, 1 Oct 2018 15:26:06 +0200
Subject: [Bro] OT: SuperMicro hardware recommendations
In-Reply-To: <EFAF45EB-9512-42EF-825F-A6F0DA0A5FCE@outlook.com>
References: <94D13236-BB53-4807-93D0-F4A1BF0B369F@outlook.com>
	<151C7D23-39E9-4170-8954-CDFF566E4ED7@gmail.com>
	<EFAF45EB-9512-42EF-825F-A6F0DA0A5FCE@outlook.com>
Message-ID: <CAGURB2cCMcg2rmxWGJLzXn4LxPoy4VkPCgeS08CoZgr6azW2Ng@mail.gmail.com>

Can also checkout the offerings from the OpenCompute project;
https://www.opencompute.org/products
Can even stick with HP if you like, their 'Cloudline' gear supports (some
aspects) of OCP.

On Mon, Oct 1, 2018 at 3:19 PM Carlos Lopez <clopmz at outlook.com> wrote:

> Thanks Michal. I didn't know that bad support from SuperMicro.
> Then I will continue with HP. Do you think the HP DL360 Gen10 are suitable
> for IDS deployment?
>
> --
> Regards,
> C. L. Martinez
>
> ?On 01/10/2018, 12:47, "Micha? Purzy?ski" <michalpurzynski1 at gmail.com>
> wrote:
>
>     I?d recommended everything but super micro.
>
>     A box of matches to shine the light on the dark corners of your
> network will behave better than anything built on super micro.
>
>     HP and other vendors may seem more expensive to buy, but the low
> quality of super micro servers, apparent lack of support, arbitrary cut
> support ?we do not support this anymore? and ridiculous replacement policy
> makes it a non-starter for me.
>
>     Example: they wanted me to send all dead hard drives back before they
> can send me a replacement.
>     And that would take days or a week or two weeks.
>
>     Raid battery problems?
>
>     SM requires you to send entire server back and in a short 6 weeks - 3
> months you will see it back. Maybe.
>
>     Mechanically, chassis are a joke, too.
>
>     OOB? No OOB hardware monitoring. And Java 1.6 required. If you have to
> use any kind of proxies, socks, tunneling - forget.
>
>
>     > On Oct 1, 2018, at 10:47 AM, Carlos Lopez <clopmz at outlook.com>
> wrote:
>     >
>     > Hi all,
>     >
>     > In the following days, we will setup a Bro-IDS distributed
> architecture using 10 sensors and one manager. Network bandwidth will be
> between 2GiB-4GiB. Until now we used HP servers, but we are looking at
> SuperMicro servers as well. Can anyone recommend series or model?
>     > --
>     > Regards,
>     > C. L. Martinez
>     >
>     >
>     > _______________________________________________
>     > Bro mailing list
>     > bro at bro-ids.org
>     > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181001/cc2a59dc/attachment.html 

From openshift.ninja at gmail.com  Mon Oct  1 07:47:07 2018
From: openshift.ninja at gmail.com (Jeffrey Poore)
Date: Mon, 01 Oct 2018 14:47:07 +0000
Subject: [Bro] Bro logger not starting with mounted directories
Message-ID: <em5a4c3f81-8ea0-4b39-ac66-9e8ebf33d284@starkiller>

So I have a Bro cluster running in some containers, with a single 
instance of each node type, manager, logger, proxy and worker, all 
running on different servers. Log files get written to 
/usr/local/bro-2.5.4/spool/logger on the logger host, and then they get 
moved to folders under /usr/local/bro-2.5.4/logs. Everything is working 
ok, but I want to be able to write the logs to a mounted volume so that 
they can be seen outside the container. I tried mounting folders for 
both the bro/spool/logger folder and then also the bro/logs folder, but 
then the logger won't start (I checked that the permissions for the 
folder allow for reading and writing of any user inside the container, 
although currently the process is running as root). I was able to mount 
the bro/logs folder and start everything ok, but obviously the files 
written to the bro/spool/logger folder are only rotated over to the 
bro/logs folder periodically.

Does anyone have any ideas why the logger wouldn't start? The command to 
start the logger doesn't seem to output any obvious message that would 
indicate why it failed, and the bro process itself isn't running after 
the execution.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181001/afd891e5/attachment.html 

From michalpurzynski1 at gmail.com  Mon Oct  1 10:27:43 2018
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Mon, 1 Oct 2018 19:27:43 +0200
Subject: [Bro] OT: SuperMicro hardware recommendations
In-Reply-To: <CAGURB2cCMcg2rmxWGJLzXn4LxPoy4VkPCgeS08CoZgr6azW2Ng@mail.gmail.com>
References: <94D13236-BB53-4807-93D0-F4A1BF0B369F@outlook.com>
	<151C7D23-39E9-4170-8954-CDFF566E4ED7@gmail.com>
	<EFAF45EB-9512-42EF-825F-A6F0DA0A5FCE@outlook.com>
	<CAGURB2cCMcg2rmxWGJLzXn4LxPoy4VkPCgeS08CoZgr6azW2Ng@mail.gmail.com>
Message-ID: <E33DA773-F6EF-4A8E-9FB2-1EA54FD73389@gmail.com>

One other non-HP brand I?ve always been recommending is Fujitsu.
 People are always surprised that ?Fujitsu makes servers?? but my experience has always been great with both the hardware side and the support.

They were cheaper than HP, last time I was buying them.


> On Oct 1, 2018, at 3:26 PM, Alex Waher <alexwis at gmail.com> wrote:
> 
> Can also checkout the offerings from the OpenCompute project; https://www.opencompute.org/products
> Can even stick with HP if you like, their 'Cloudline' gear supports (some aspects) of OCP.
> 
>> On Mon, Oct 1, 2018 at 3:19 PM Carlos Lopez <clopmz at outlook.com> wrote:
>> Thanks Michal. I didn't know that bad support from SuperMicro.
>> Then I will continue with HP. Do you think the HP DL360 Gen10 are suitable for IDS deployment?
>> 
>> -- 
>> Regards,
>> C. L. Martinez
>> 
>> ?On 01/10/2018, 12:47, "Micha? Purzy?ski" <michalpurzynski1 at gmail.com> wrote:
>> 
>>     I?d recommended everything but super micro.  
>> 
>>     A box of matches to shine the light on the dark corners of your network will behave better than anything built on super micro.
>> 
>>     HP and other vendors may seem more expensive to buy, but the low quality of super micro servers, apparent lack of support, arbitrary cut support ?we do not support this anymore? and ridiculous replacement policy makes it a non-starter for me.
>> 
>>     Example: they wanted me to send all dead hard drives back before they can send me a replacement.
>>     And that would take days or a week or two weeks.
>> 
>>     Raid battery problems?
>> 
>>     SM requires you to send entire server back and in a short 6 weeks - 3 months you will see it back. Maybe.
>> 
>>     Mechanically, chassis are a joke, too.
>> 
>>     OOB? No OOB hardware monitoring. And Java 1.6 required. If you have to use any kind of proxies, socks, tunneling - forget.
>> 
>> 
>>     > On Oct 1, 2018, at 10:47 AM, Carlos Lopez <clopmz at outlook.com> wrote:
>>     > 
>>     > Hi all,
>>     > 
>>     > In the following days, we will setup a Bro-IDS distributed architecture using 10 sensors and one manager. Network bandwidth will be between 2GiB-4GiB. Until now we used HP servers, but we are looking at SuperMicro servers as well. Can anyone recommend series or model?
>>     > -- 
>>     > Regards,
>>     > C. L. Martinez
>>     > 
>>     > 
>>     > _______________________________________________
>>     > Bro mailing list
>>     > bro at bro-ids.org
>>     > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> 
>> 
>> 
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181001/9b9b62bd/attachment.html 

From openshift.ninja at gmail.com  Mon Oct  1 17:45:25 2018
From: openshift.ninja at gmail.com (OpenShift Ninja)
Date: Mon, 1 Oct 2018 20:45:25 -0400
Subject: [Bro] Bro logger not starting with mounted directories
Message-ID: <CAJ8Dx3S6ywRZ+ZpqpinqF2vrDc-cpsLWQihsxh=U_-8H=4s=Cw@mail.gmail.com>

The issue turned out to be with our filesystem that was getting mounted. I
was able to get the logger started and working.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181001/0b8f5e1f/attachment.html 

From seth at corelight.com  Tue Oct  2 06:37:23 2018
From: seth at corelight.com (Seth Hall)
Date: Tue, 02 Oct 2018 09:37:23 -0400
Subject: [Bro] Bro 2.6 beta feedback?
Message-ID: <DC0EED9B-D7A3-44EB-B35C-42082774D4F9@corelight.com>

I've seen very little feedback from the Bro 2.6 beta that was posted 
recently.  I'm wondering how many people have tried it and if there is 
any feedback about it?

Thanks!
   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com

From shirkdog.bsd at gmail.com  Tue Oct  2 07:06:02 2018
From: shirkdog.bsd at gmail.com (Michael Shirk)
Date: Tue, 2 Oct 2018 10:06:02 -0400
Subject: [Bro] Bro 2.6 beta feedback?
In-Reply-To: <DC0EED9B-D7A3-44EB-B35C-42082774D4F9@corelight.com>
References: <DC0EED9B-D7A3-44EB-B35C-42082774D4F9@corelight.com>
Message-ID: <CAL8PkUV2muvCE_WwX=ePD4kaRSGf1MXubgedSYMJ5NoWHY5yNw@mail.gmail.com>

I can report bro "version 2.6-beta2-3" built and started on 9/21/2018
on FreeBSD 11.2-RELEASE-p2 is still working...no issues for me (no
plugins/packages)
On Tue, Oct 2, 2018 at 9:52 AM Seth Hall <seth at corelight.com> wrote:
>
> I've seen very little feedback from the Bro 2.6 beta that was posted
> recently.  I'm wondering how many people have tried it and if there is
> any feedback about it?
>
> Thanks!
>    .Seth
>
> --
> Seth Hall * Corelight, Inc * www.corelight.com
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
Michael Shirk
Daemon Security, Inc.
https://www.daemon-security.com

From dopheide at gmail.com  Tue Oct  2 07:58:43 2018
From: dopheide at gmail.com (Mike Dopheide)
Date: Tue, 2 Oct 2018 09:58:43 -0500
Subject: [Bro] Bro 2.6 beta feedback?
In-Reply-To: <CAL8PkUV2muvCE_WwX=ePD4kaRSGf1MXubgedSYMJ5NoWHY5yNw@mail.gmail.com>
References: <DC0EED9B-D7A3-44EB-B35C-42082774D4F9@corelight.com>
	<CAL8PkUV2muvCE_WwX=ePD4kaRSGf1MXubgedSYMJ5NoWHY5yNw@mail.gmail.com>
Message-ID: <CAPy2kFag6+28r8bY0BMqRK5CD4CDyB5WVgjX9qyT-CjV6XUYaA@mail.gmail.com>

Adding to the data points and hopefully confidence of people who may be
waiting...

ESnet is running dev/2.7 (which includes all of the 2.6 changes) in
production with no issues.  CentOS 7 and MyricomSNF.

-Dop

On Tue, Oct 2, 2018 at 9:06 AM, Michael Shirk <shirkdog.bsd at gmail.com>
wrote:

> I can report bro "version 2.6-beta2-3" built and started on 9/21/2018
> on FreeBSD 11.2-RELEASE-p2 is still working...no issues for me (no
> plugins/packages)
> On Tue, Oct 2, 2018 at 9:52 AM Seth Hall <seth at corelight.com> wrote:
> >
> > I've seen very little feedback from the Bro 2.6 beta that was posted
> > recently.  I'm wondering how many people have tried it and if there is
> > any feedback about it?
> >
> > Thanks!
> >    .Seth
> >
> > --
> > Seth Hall * Corelight, Inc * www.corelight.com
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> --
> Michael Shirk
> Daemon Security, Inc.
> https://www.daemon-security.com
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181002/4955ceca/attachment.html 

From dhoelzer at enclaveforensics.com  Tue Oct  2 13:06:20 2018
From: dhoelzer at enclaveforensics.com (=?UTF-8?Q?David_Hoelzer?=)
Date: Tue, 2 Oct 2018 20:06:20 +0000
Subject: [Bro] Bro 2.6 beta feedback?
In-Reply-To: <DC0EED9B-D7A3-44EB-B35C-42082774D4F9@corelight.com>
References: <DC0EED9B-D7A3-44EB-B35C-42082774D4F9@corelight.com> 
	<9AE95CCE-F5AB-4C64-ADF4-B07E55D123EA@enclaveforensics.com>
Message-ID: <01000166366376ef-ee87387b-108e-4fb7-97b3-e36cffc097da-000000@email.amazonses.com>

Working great for us.  We've had to rewrite some scripts and such, but otherwise, no issues. __

Thanks for your good (hard) work!

?On 10/2/18, 9:37 AM, "Seth Hall" <seth at corelight.com> wrote:

    I've seen very little feedback from the Bro 2.6 beta that was posted 
    recently.  I'm wondering how many people have tried it and if there is 
    any feedback about it?
    
    Thanks!
       .Seth
    
    --
    Seth Hall * Corelight, Inc * www.corelight.com
    
    



From seth at corelight.com  Tue Oct  2 13:38:53 2018
From: seth at corelight.com (Seth Hall)
Date: Tue, 02 Oct 2018 16:38:53 -0400
Subject: [Bro] Bro 2.6 beta feedback?
In-Reply-To: <01000166366376ef-ee87387b-108e-4fb7-97b3-e36cffc097da-000000@email.amazonses.com>
References: <DC0EED9B-D7A3-44EB-B35C-42082774D4F9@corelight.com>
	<9AE95CCE-F5AB-4C64-ADF4-B07E55D123EA@enclaveforensics.com>
	<01000166366376ef-ee87387b-108e-4fb7-97b3-e36cffc097da-000000@email.amazonses.com>
Message-ID: <59E8B650-F4A4-42CC-A4DC-49765C1088D1@corelight.com>



On 2 Oct 2018, at 16:06, David Hoelzer wrote:

> Working great for us.  We've had to rewrite some scripts and such, but 
> otherwise, no issues. __

Any comments about what scripts you had to rewrite or the reasons for 
why they needed rewritten?

> Thanks for your good (hard) work!

I can't take much credit for this release.  I had remarkably little code 
it, but I think it's a pretty nice release!  A ton of work over multiple 
years has gone into it.

  .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com

From asharma at lbl.gov  Tue Oct  2 15:03:17 2018
From: asharma at lbl.gov (Aashish Sharma)
Date: Tue, 2 Oct 2018 15:03:17 -0700
Subject: [Bro] Bro 2.6 beta feedback?
In-Reply-To: <59E8B650-F4A4-42CC-A4DC-49765C1088D1@corelight.com>
References: <DC0EED9B-D7A3-44EB-B35C-42082774D4F9@corelight.com>
	<9AE95CCE-F5AB-4C64-ADF4-B07E55D123EA@enclaveforensics.com>
	<01000166366376ef-ee87387b-108e-4fb7-97b3-e36cffc097da-000000@email.amazonses.com>
	<59E8B650-F4A4-42CC-A4DC-49765C1088D1@corelight.com>
Message-ID: <20181002220316.GK91054@MacPro-2331.local>

So, I tried running this on FreeBSD 10.3-STABLE and I see the following, builds
fine but when trying to run: 

$ ./broctl install
Error: running "bro -v" failed with output:
Undefined symbol "_ZN6broker5topic8reservedE" referenced from COPY relocation in /home/bro/master/bin/bro


I do see that bro builds against its own libcaf:

$ ldd ./bro
./bro:
        libpcap.so.1 => /usr/local/opt/snf/lib/libpcap.so.1 (0x400ef1000)
        libssl.so.9 => /usr/local/lib/libssl.so.9 (0x40113a000)
        libcrypto.so.9 => /usr/local/lib/libcrypto.so.9 (0x401400000)
        libz.so.6 => /lib/libz.so.6 (0x401852000)
        libbroker.so.0 => /usr/local/lib/libbroker.so.0 (0x401c00000)
        libcaf_core.so.0.16.0 => /home/bro/master/lib/libcaf_core.so.0.16.0 (0x402200000)
        libcaf_io.so.0.16.0 => /home/bro/master/lib/libcaf_io.so.0.16.0 (0x402800000)
        libcaf_openssl.so.0.16.0 => /home/bro/master/lib/libcaf_openssl.so.0.16.0 (0x402c86000)
        libc++.so.1 => /usr/local/lib/libc++.so.1 (0x402ee0000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x4031a0000)
        libm.so.5 => /lib/libm.so.5 (0x4033bd000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x4035e6000)
        libthr.so.3 => /lib/libthr.so.3 (0x4037f4000)
        libc.so.7 => /lib/libc.so.7 (0x403a18000)
        libsnf.so.0 => /usr/local/opt/snf/lib/libsnf.so.0 (0x403dc6000)
        libcaf_core.so.0.14.1 => /usr/local/lib/libcaf_core.so.0.14.1 (0x403fd6000)
        libcaf_io.so.0.14.1 => /usr/local/lib/libcaf_io.so.0.14.1 (0x4042c4000)
        libcxxrt.so => /usr/local/lib/libcxxrt.so (0x404574000)
        librt.so.1 => /usr/lib/librt.so.1 (0x404790000)


Aashish 


On Tue, Oct 02, 2018 at 04:38:53PM -0400, Seth Hall wrote:
> 
> 
> On 2 Oct 2018, at 16:06, David Hoelzer wrote:
> 
> > Working great for us.  We've had to rewrite some scripts and such, but 
> > otherwise, no issues. __
> 
> Any comments about what scripts you had to rewrite or the reasons for 
> why they needed rewritten?
> 
> > Thanks for your good (hard) work!
> 
> I can't take much credit for this release.  I had remarkably little code 
> it, but I think it's a pretty nice release!  A ton of work over multiple 
> years has gone into it.
> 
>   .Seth
> 
> --
> Seth Hall * Corelight, Inc * www.corelight.com
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

From jazoff at illinois.edu  Tue Oct  2 15:08:51 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Tue, 2 Oct 2018 22:08:51 +0000
Subject: [Bro] Bro 2.6 beta feedback?
In-Reply-To: <20181002220316.GK91054@MacPro-2331.local>
References: <DC0EED9B-D7A3-44EB-B35C-42082774D4F9@corelight.com>
	<9AE95CCE-F5AB-4C64-ADF4-B07E55D123EA@enclaveforensics.com>
	<01000166366376ef-ee87387b-108e-4fb7-97b3-e36cffc097da-000000@email.amazonses.com>
	<59E8B650-F4A4-42CC-A4DC-49765C1088D1@corelight.com>
	<20181002220316.GK91054@MacPro-2331.local>
Message-ID: <5D23B64A-F9F2-4FD6-82BA-2F3F8707BC00@illinois.edu>


> On Oct 2, 2018, at 6:03 PM, Aashish Sharma <asharma at lbl.gov> wrote:
>        libcaf_core.so.0.16.0 => /home/bro/master/lib/libcaf_core.so.0.16.0 (0x402200000)
>        libcaf_io.so.0.16.0 => /home/bro/master/lib/libcaf_io.so.0.16.0 (0x402800000)
..
>        libcaf_core.so.0.14.1 => /usr/local/lib/libcaf_core.so.0.14.1 (0x403fd6000)
>        libcaf_io.so.0.14.1 => /usr/local/lib/libcaf_io.so.0.14.1 (0x4042c4000)

Looks like you have an older version of caf installed in /usr/local that may be conflicting with things.

I helped someone on IRC with a similar problem and removing any old caf libraries and header files in /usr/local fixed things.

? 
Justin Azoff




From shirkdog.bsd at gmail.com  Tue Oct  2 15:14:34 2018
From: shirkdog.bsd at gmail.com (Michael Shirk)
Date: Tue, 2 Oct 2018 18:14:34 -0400
Subject: [Bro] Bro 2.6 beta feedback?
In-Reply-To: <20181002220316.GK91054@MacPro-2331.local>
References: <DC0EED9B-D7A3-44EB-B35C-42082774D4F9@corelight.com>
	<9AE95CCE-F5AB-4C64-ADF4-B07E55D123EA@enclaveforensics.com>
	<01000166366376ef-ee87387b-108e-4fb7-97b3-e36cffc097da-000000@email.amazonses.com>
	<59E8B650-F4A4-42CC-A4DC-49765C1088D1@corelight.com>
	<20181002220316.GK91054@MacPro-2331.local>
Message-ID: <CAL8PkUXddcmvRTnybOTS=nFLSEkLsBG96P+KAyb161NqOicQfw@mail.gmail.com>

I had that issue where I had to remove the OS installed libcaf on FreeBSD.

--
Michael Shirk
Daemon Security, Inc.
https://www.daemon-security.com

On Tue, Oct 2, 2018, 18:12 Aashish Sharma <asharma at lbl.gov> wrote:

> So, I tried running this on FreeBSD 10.3-STABLE and I see the following,
> builds
> fine but when trying to run:
>
> $ ./broctl install
> Error: running "bro -v" failed with output:
> Undefined symbol "_ZN6broker5topic8reservedE" referenced from COPY
> relocation in /home/bro/master/bin/bro
>
>
> I do see that bro builds against its own libcaf:
>
> $ ldd ./bro
> ./bro:
>         libpcap.so.1 => /usr/local/opt/snf/lib/libpcap.so.1 (0x400ef1000)
>         libssl.so.9 => /usr/local/lib/libssl.so.9 (0x40113a000)
>         libcrypto.so.9 => /usr/local/lib/libcrypto.so.9 (0x401400000)
>         libz.so.6 => /lib/libz.so.6 (0x401852000)
>         libbroker.so.0 => /usr/local/lib/libbroker.so.0 (0x401c00000)
>         libcaf_core.so.0.16.0 =>
> /home/bro/master/lib/libcaf_core.so.0.16.0 (0x402200000)
>         libcaf_io.so.0.16.0 => /home/bro/master/lib/libcaf_io.so.0.16.0
> (0x402800000)
>         libcaf_openssl.so.0.16.0 =>
> /home/bro/master/lib/libcaf_openssl.so.0.16.0 (0x402c86000)
>         libc++.so.1 => /usr/local/lib/libc++.so.1 (0x402ee0000)
>         libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x4031a0000)
>         libm.so.5 => /lib/libm.so.5 (0x4033bd000)
>         libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x4035e6000)
>         libthr.so.3 => /lib/libthr.so.3 (0x4037f4000)
>         libc.so.7 => /lib/libc.so.7 (0x403a18000)
>         libsnf.so.0 => /usr/local/opt/snf/lib/libsnf.so.0 (0x403dc6000)
>         libcaf_core.so.0.14.1 => /usr/local/lib/libcaf_core.so.0.14.1
> (0x403fd6000)
>         libcaf_io.so.0.14.1 => /usr/local/lib/libcaf_io.so.0.14.1
> (0x4042c4000)
>         libcxxrt.so => /usr/local/lib/libcxxrt.so (0x404574000)
>         librt.so.1 => /usr/lib/librt.so.1 (0x404790000)
>
>
> Aashish
>
>
> On Tue, Oct 02, 2018 at 04:38:53PM -0400, Seth Hall wrote:
> >
> >
> > On 2 Oct 2018, at 16:06, David Hoelzer wrote:
> >
> > > Working great for us.  We've had to rewrite some scripts and such, but
> > > otherwise, no issues. __
> >
> > Any comments about what scripts you had to rewrite or the reasons for
> > why they needed rewritten?
> >
> > > Thanks for your good (hard) work!
> >
> > I can't take much credit for this release.  I had remarkably little code
> > it, but I think it's a pretty nice release!  A ton of work over multiple
> > years has gone into it.
> >
> >   .Seth
> >
> > --
> > Seth Hall * Corelight, Inc * www.corelight.com
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181002/d596061e/attachment.html 

From dhoelzer at enclaveforensics.com  Tue Oct  2 15:31:24 2018
From: dhoelzer at enclaveforensics.com (=?UTF-8?Q?David_Hoelzer?=)
Date: Tue, 2 Oct 2018 22:31:24 +0000
Subject: [Bro] Bro 2.6 beta feedback?
In-Reply-To: <59E8B650-F4A4-42CC-A4DC-49765C1088D1@corelight.com>
References: <DC0EED9B-D7A3-44EB-B35C-42082774D4F9@corelight.com> 
	<9AE95CCE-F5AB-4C64-ADF4-B07E55D123EA@enclaveforensics.com> 
	<01000166366376ef-ee87387b-108e-4fb7-97b3-e36cffc097da-000000@email.amazonses.com>
	<59E8B650-F4A4-42CC-A4DC-49765C1088D1@corelight.com> 
	<E816C694-DA7F-4B2D-8890-119040821CC4@enclaveforensics.com>
Message-ID: <0100016636e84870-7cce0f1e-bd7e-46ae-a427-f43c5ac6f08c-000000@email.amazonses.com>

I'll get back to you on script rewrites.

?On 10/2/18, 4:38 PM, "Seth Hall" <seth at corelight.com> wrote:

    
    
    On 2 Oct 2018, at 16:06, David Hoelzer wrote:
    
    > Working great for us.  We've had to rewrite some scripts and such, but 
    > otherwise, no issues. __
    
    Any comments about what scripts you had to rewrite or the reasons for 
    why they needed rewritten?
    
    > Thanks for your good (hard) work!
    
    I can't take much credit for this release.  I had remarkably little code 
    it, but I think it's a pretty nice release!  A ton of work over multiple 
    years has gone into it.
    
      .Seth
    
    --
    Seth Hall * Corelight, Inc * www.corelight.com
    




From seth at corelight.com  Tue Oct  2 16:00:41 2018
From: seth at corelight.com (Seth Hall)
Date: Tue, 2 Oct 2018 19:00:41 -0400
Subject: [Bro] Bro 2.6 beta feedback?
In-Reply-To: <5D23B64A-F9F2-4FD6-82BA-2F3F8707BC00@illinois.edu>
References: <DC0EED9B-D7A3-44EB-B35C-42082774D4F9@corelight.com>
	<9AE95CCE-F5AB-4C64-ADF4-B07E55D123EA@enclaveforensics.com>
	<01000166366376ef-ee87387b-108e-4fb7-97b3-e36cffc097da-000000@email.amazonses.com>
	<59E8B650-F4A4-42CC-A4DC-49765C1088D1@corelight.com>
	<20181002220316.GK91054@MacPro-2331.local>
	<5D23B64A-F9F2-4FD6-82BA-2F3F8707BC00@illinois.edu>
Message-ID: <A35EB860-E786-42A3-AB4A-D28698A4D2BA@corelight.com>

Hm, that makes me wonder if we should do something like set an RPATH to make sure the correct libcaf is used.

Jon, does that make sense?  It seems that this problem will only come up more frequently as caf is packaged by more systems. 

  .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com


> On Oct 2, 2018, at 6:08 PM, Azoff, Justin S <jazoff at illinois.edu> wrote:
> 
> 
>> On Oct 2, 2018, at 6:03 PM, Aashish Sharma <asharma at lbl.gov> wrote:
>>       libcaf_core.so.0.16.0 => /home/bro/master/lib/libcaf_core.so.0.16.0 (0x402200000)
>>       libcaf_io.so.0.16.0 => /home/bro/master/lib/libcaf_io.so.0.16.0 (0x402800000)
> ..
>>       libcaf_core.so.0.14.1 => /usr/local/lib/libcaf_core.so.0.14.1 (0x403fd6000)
>>       libcaf_io.so.0.14.1 => /usr/local/lib/libcaf_io.so.0.14.1 (0x4042c4000)
> 
> Looks like you have an older version of caf installed in /usr/local that may be conflicting with things.
> 
> I helped someone on IRC with a similar problem and removing any old caf libraries and header files in /usr/local fixed things.
> 
> ? 
> Justin Azoff
> 
> 


From johanna at icir.org  Tue Oct  2 17:17:09 2018
From: johanna at icir.org (Johanna Amann)
Date: Tue, 02 Oct 2018 17:17:09 -0700
Subject: [Bro] Bro 2.6 beta feedback?
In-Reply-To: <A35EB860-E786-42A3-AB4A-D28698A4D2BA@corelight.com>
References: <DC0EED9B-D7A3-44EB-B35C-42082774D4F9@corelight.com>
	<9AE95CCE-F5AB-4C64-ADF4-B07E55D123EA@enclaveforensics.com>
	<01000166366376ef-ee87387b-108e-4fb7-97b3-e36cffc097da-000000@email.amazonses.com>
	<59E8B650-F4A4-42CC-A4DC-49765C1088D1@corelight.com>
	<20181002220316.GK91054@MacPro-2331.local>
	<5D23B64A-F9F2-4FD6-82BA-2F3F8707BC00@illinois.edu>
	<A35EB860-E786-42A3-AB4A-D28698A4D2BA@corelight.com>
Message-ID: <0D4FC046-CDBA-46B3-BEF6-91FEB16D4B9E@icir.org>

I am not sure that that will help - setting an rpath in packaged 
binaries is typically at least discouraged. We actually also don?t do 
it in the packages that we do (or will) ship.

For example, see https://wiki.debian.org/RpathIssue, 
https://www.debian.org/doc/manuals/maint-guide/advanced.en.html, 
https://fedoraproject.org/wiki/Packaging_tricks#Acceptable_use_of_rpath 
(rpmlint raises it as an error; it can be considered ok in fedora for 
internal libraries), https://en.opensuse.org/openSUSE:Packaging_checks 
(this seems to outright prohibit its use).

But - perhaps we should recognize such situations and refuse to 
compile/warn users that they mightt have to re-set their dynamic link 
paths?

Johanna

On 2 Oct 2018, at 16:00, Seth Hall wrote:

> Hm, that makes me wonder if we should do something like set an RPATH 
> to make sure the correct libcaf is used.
>
> Jon, does that make sense?  It seems that this problem will only come 
> up more frequently as caf is packaged by more systems.
>
>   .Seth
>
> --
> Seth Hall * Corelight, Inc * www.corelight.com
>
>
>> On Oct 2, 2018, at 6:08 PM, Azoff, Justin S <jazoff at illinois.edu> 
>> wrote:
>>
>>
>>> On Oct 2, 2018, at 6:03 PM, Aashish Sharma <asharma at lbl.gov> wrote:
>>>       libcaf_core.so.0.16.0 => 
>>> /home/bro/master/lib/libcaf_core.so.0.16.0 (0x402200000)
>>>       libcaf_io.so.0.16.0 => 
>>> /home/bro/master/lib/libcaf_io.so.0.16.0 (0x402800000)
>> ..
>>>       libcaf_core.so.0.14.1 => /usr/local/lib/libcaf_core.so.0.14.1 
>>> (0x403fd6000)
>>>       libcaf_io.so.0.14.1 => /usr/local/lib/libcaf_io.so.0.14.1 
>>> (0x4042c4000)
>>
>> Looks like you have an older version of caf installed in /usr/local 
>> that may be conflicting with things.
>>
>> I helped someone on IRC with a similar problem and removing any old 
>> caf libraries and header files in /usr/local fixed things.
>>
>> ?
>> Justin Azoff
>>
>>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

From jsiwek at corelight.com  Tue Oct  2 17:46:10 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Tue, 2 Oct 2018 19:46:10 -0500
Subject: [Bro] Bro 2.6 beta feedback?
In-Reply-To: <A35EB860-E786-42A3-AB4A-D28698A4D2BA@corelight.com>
References: <DC0EED9B-D7A3-44EB-B35C-42082774D4F9@corelight.com>
	<9AE95CCE-F5AB-4C64-ADF4-B07E55D123EA@enclaveforensics.com>
	<01000166366376ef-ee87387b-108e-4fb7-97b3-e36cffc097da-000000@email.amazonses.com>
	<59E8B650-F4A4-42CC-A4DC-49765C1088D1@corelight.com>
	<20181002220316.GK91054@MacPro-2331.local>
	<5D23B64A-F9F2-4FD6-82BA-2F3F8707BC00@illinois.edu>
	<A35EB860-E786-42A3-AB4A-D28698A4D2BA@corelight.com>
Message-ID: <CAMzgZ0+PEbk7Fg0+aA_Rou4NSswgYgrD1-M=xsZ4sW=pTqHy8A@mail.gmail.com>

On Tue, Oct 2, 2018 at 6:15 PM Seth Hall <seth at corelight.com> wrote:
>
> Hm, that makes me wonder if we should do something like set an RPATH to make sure the correct libcaf is used.
>
> Jon, does that make sense?  It seems that this problem will only come up more frequently as caf is packaged by more systems.

An RPATH is already set by default (regardless of platform):

$ readelf -d /usr/local/bro/bin/bro | grep RPATH
 0x000000000000000f (RPATH)              Library rpath: [/usr/local/bro/lib]
$ readelf -d /usr/local/bro/lib/libbroker.so | grep RPATH
 0x000000000000000f (RPATH)              Library rpath: [/usr/local/bro/lib]
$ freebsd-version
10.3-RELEASE-p29

I tried reproducing the problem by faking it out with CAF libs
installed in /usr/local:

$ ls /usr/local/lib/libcaf_*
/usr/local/lib/libcaf_core.so
/usr/local/lib/libcaf_core.so.0.15.3
/usr/local/lib/libcaf_io.so
/usr/local/lib/libcaf_io.so.0.15.3

But that still works fine for me:

$ /usr/bin/ldd /usr/local/bro/bin/bro
/usr/local/bro/bin/bro:
libpcap.so.8 => /lib/libpcap.so.8 (0x80162b000)
libssl.so.7 => /usr/lib/libssl.so.7 (0x801870000)
libcrypto.so.7 => /lib/libcrypto.so.7 (0x801adc000)
libz.so.6 => /lib/libz.so.6 (0x801ed1000)
libbroker.so.0 => /usr/local/bro/lib/libbroker.so.0 (0x802200000)
libthr.so.3 => /lib/libthr.so.3 (0x802e71000)
libcaf_core.so.0.16.0 => /usr/local/bro/lib/libcaf_core.so.0.16.0 (0x803200000)
libcaf_io.so.0.16.0 => /usr/local/bro/lib/libcaf_io.so.0.16.0 (0x803c00000)
libcaf_openssl.so.0.16.0 =>
/usr/local/bro/lib/libcaf_openssl.so.0.16.0 (0x80453e000)
libc++.so.1 => /usr/lib/libc++.so.1 (0x804844000)
libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x804b03000)
libm.so.5 => /lib/libm.so.5 (0x804d20000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x804f49000)
libc.so.7 => /lib/libc.so.7 (0x805157000)

So not sure why Aashish or others had problems with alternate CAF installations.

- Jon

From jsiwek at corelight.com  Tue Oct  2 17:52:00 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Tue, 2 Oct 2018 19:52:00 -0500
Subject: [Bro] Bro 2.6 beta feedback?
In-Reply-To: <0D4FC046-CDBA-46B3-BEF6-91FEB16D4B9E@icir.org>
References: <DC0EED9B-D7A3-44EB-B35C-42082774D4F9@corelight.com>
	<9AE95CCE-F5AB-4C64-ADF4-B07E55D123EA@enclaveforensics.com>
	<01000166366376ef-ee87387b-108e-4fb7-97b3-e36cffc097da-000000@email.amazonses.com>
	<59E8B650-F4A4-42CC-A4DC-49765C1088D1@corelight.com>
	<20181002220316.GK91054@MacPro-2331.local>
	<5D23B64A-F9F2-4FD6-82BA-2F3F8707BC00@illinois.edu>
	<A35EB860-E786-42A3-AB4A-D28698A4D2BA@corelight.com>
	<0D4FC046-CDBA-46B3-BEF6-91FEB16D4B9E@icir.org>
Message-ID: <CAMzgZ0+t3ex+bNwPnKYDFsVwQ7KhcOH+ducAoEpsdD=yjvJ99Q@mail.gmail.com>

On Tue, Oct 2, 2018 at 7:28 PM Johanna Amann <johanna at icir.org> wrote:
>
> I am not sure that that will help - setting an rpath in packaged
> binaries is typically at least discouraged. We actually also don?t do
> it in the packages that we do (or will) ship.

For local installs where we only ever intend to load libraries that
were a part of the same build, it seems reasonable to have an RPATH.

For binary packages, I agree that it's not as good an idea, and that's
what the `configure --binary-package` flag was meant to help with: it
disables setting of RPATH instead of doing what normally happens in
cmake/SetupRPATH.cmake

- Jon


From shutchison at cert.org  Wed Oct  3 11:46:00 2018
From: shutchison at cert.org (Sean Hutchison)
Date: Wed, 3 Oct 2018 18:46:00 +0000
Subject: [Bro] Broctl segmentation fault
Message-ID: <CD76AAEFD1531643AB6A33368B516D770B600D85@marathon>

Hello,

After any build of Bro with Broctl 1.7, I'm experiencing the below error when broctl/scripts/check-config is run...

/opt/bro/share/broctl/scripts/check-config: line 50:  4463 Segmentation fault      "${bro}" $check_option "$@"

Anyone encountered this before? Cannot bypass doing broctl check - broctl start results in failed/crashed processes.

This is on RHEL7.5, after building Bro-2.5.5 (I've tried other minor versions since 2.5 - same issue).

Existing Bro cluster on RHEL7.5 boxes with Bro-2.5 and Broctl 1.5 works fine.

Any help would be greatly appreciated.

V/R
Sean Hutchison

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181003/fdb33a3d/attachment.html 

From jazoff at illinois.edu  Wed Oct  3 12:00:40 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Wed, 3 Oct 2018 19:00:40 +0000
Subject: [Bro] Broctl segmentation fault
In-Reply-To: <CD76AAEFD1531643AB6A33368B516D770B600D85@marathon>
References: <CD76AAEFD1531643AB6A33368B516D770B600D85@marathon>
Message-ID: <83656C44-986A-49F5-BC68-0DABBC006A29@illinois.edu>


> On Oct 3, 2018, at 2:46 PM, Sean Hutchison <shutchison at cert.org> wrote:
> 
> Hello,
>  
> After any build of Bro with Broctl 1.7, I?m experiencing the below error when broctl/scripts/check-config is run?
>  
> /opt/bro/share/broctl/scripts/check-config: line 50:  4463 Segmentation fault      "${bro}" $check_option "$@"
>  
> Anyone encountered this before? Cannot bypass doing broctl check ? broctl start results in failed/crashed processes.
>  
> This is on RHEL7.5, after building Bro-2.5.5 (I?ve tried other minor versions since 2.5 ? same issue).
>  
> Existing Bro cluster on RHEL7.5 boxes with Bro-2.5 and Broctl 1.5 works fine.
>  
> Any help would be greatly appreciated.
>  

check runs bro with the current configuration to see if it can start, so that's bro segfaulting there.. that's why start also fails..

What do you get if you try each of the following?

    bro -v
    bro -NN # just see if this runs or crashes
    bro -b -i lo
    bro -i lo
    bro -i lo local

You can hit control-c if any of those start successfully to get your prompt back.

I'm not aware of any issues like this, so it could be something with your configuration.

Do you have a customized local.bro at all?
Are you building bro against a particular libpcap or malloc implementation?
What does ldd /opt/bro/bin/bro output?

? 
Justin Azoff



From maanamen at hotmail.com  Thu Oct  4 03:06:22 2018
From: maanamen at hotmail.com (=?iso-8859-1?Q?MA=C1N_ABU_SHAQRA?=)
Date: Thu, 4 Oct 2018 10:06:22 +0000
Subject: [Bro] is there a bro script to ignore duplicated logs?
Message-ID: <AM4PR0202MB2850779D68BC367E36A2D657CBEA0@AM4PR0202MB2850.eurprd02.prod.outlook.com>

were facing this issue with bro whereby its duplicating entries see below:

1536746459.586520 CbxxYF1uTyqC499HDe 192.168.20.15 137 10.190.129.26 137 udp 39011 - maanpc 1 C_INTERNET 32 NB F

1536746460.343566 CbxxYF1uTyqC499HDe 192.168.20.15 137 10.190.129.26 137 udp 39011 - maanpc 1 C_INTERNET 32 NB F

1536746461.107930 CbxxYF1uTyqC499HDe 192.168.20.15 137 10.190.129.26 137 udp 39011 - maanpc 1 C_INTERNET 32 NB F

1536746466.418528 CbxxYF1uTyqC499HDe 192.168.20.15 137 10.190.129.26 137 udp 39013 - maanpc 1 C_INTERNET 32 NB F

1536746467.176333 CbxxYF1uTyqC499HDe 192.168.20.15 137 10.190.129.26 137 udp 39013 - maanpc 1 C_INTERNET 32 NB F

1536746467.940695 CbxxYF1uTyqC499HDe 192.168.20.15 137 10.190.129.26 137 udp 39013 - maanpc 1 C_INTERNET 32 NB F

1536746473.250630 CbxxYF1uTyqC499HDe 192.168.20.15 137 10.190.129.26 137 udp 39017 - maanpc 1 C_INTERNET 32 NB F

1536746474.010337 CbxxYF1uTyqC499HDe 192.168.20.15 137 10.190.129.26 137 udp 39017 - maanpc 1 C_INTERNET 32 NB F

1536746474.773560 CbxxYF1uTyqC499HDe 192.168.20.15 137 10.190.129.26 137 udp 39017 - maanpc 1 C_INTERNET 32 NB F

1536746452.751762 CbxxYF1uTyqC499HDe 192.168.20.15 137 10.190.129.26 137 udp 39009 - maanpc 1 C_INTERNET 32 NB F

1536746453.510702 CbxxYF1uTyqC499HDe 192.168.20.15 137 10.190.129.26 137 udp 39009 - maanpc 1 C_INTERNET 32 NB F

1536746454.275116 CbxxYF1uTyqC499HDe 192.168.20.15 137 10.190.129.26 137 udp 39009 - maanpc 1 C_INTERNET 32 NB F


pf_ring / af packet didnt help.


thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181004/2dab4128/attachment.html 

From michalpurzynski1 at gmail.com  Thu Oct  4 03:20:40 2018
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Thu, 4 Oct 2018 11:20:40 +0100
Subject: [Bro] is there a bro script to ignore duplicated logs?
In-Reply-To: <AM4PR0202MB2850779D68BC367E36A2D657CBEA0@AM4PR0202MB2850.eurprd02.prod.outlook.com>
References: <AM4PR0202MB2850779D68BC367E36A2D657CBEA0@AM4PR0202MB2850.eurprd02.prod.outlook.com>
Message-ID: <F106E798-7EF5-40F6-8FA4-97706B83E31A@gmail.com>

These duplicated logs make it apparent that you?re having some packet capture problems.

What?s your packet capture setup? Do you use a span port? Optical taps? Packet brokers?

How do you run bro? 

> On Oct 4, 2018, at 11:06 AM, MA?N ABU SHAQRA <maanamen at hotmail.com> wrote:
> 
> were facing this issue with bro whereby its duplicating entries see below:
> 
> 1536746459.586520 CbxxYF1uTyqC499HDe
> 192.168.20.15 137
> 10.190.129.26 137
> udp 39011
> - maanpc
> 1 C_INTERNET
> 32 NB
> F
> 
> 1536746460.343566 CbxxYF1uTyqC499HDe
> 192.168.20.15 137
> 10.190.129.26 137
> udp 39011
> - maanpc
> 1 C_INTERNET
> 32 NB
> F
> 
> 1536746461.107930 CbxxYF1uTyqC499HDe
> 192.168.20.15 137
> 10.190.129.26 137
> udp 39011
> - maanpc
> 1 C_INTERNET
> 32 NB
> F
> 
> 1536746466.418528 CbxxYF1uTyqC499HDe
> 192.168.20.15 137
> 10.190.129.26 137
> udp 39013
> - maanpc
> 1 C_INTERNET
> 32 NB
> F
> 
> 1536746467.176333 CbxxYF1uTyqC499HDe
> 192.168.20.15 137
> 10.190.129.26 137
> udp 39013
> - maanpc
> 1 C_INTERNET
> 32 NB
> F
> 
> 1536746467.940695 CbxxYF1uTyqC499HDe
> 192.168.20.15 137
> 10.190.129.26 137
> udp 39013
> - maanpc
> 1 C_INTERNET
> 32 NB
> F
> 
> 1536746473.250630 CbxxYF1uTyqC499HDe
> 192.168.20.15 137
> 10.190.129.26 137
> udp 39017
> - maanpc
> 1 C_INTERNET
> 32 NB
> F
> 
> 1536746474.010337 CbxxYF1uTyqC499HDe
> 192.168.20.15 137
> 10.190.129.26 137
> udp 39017
> - maanpc
> 1 C_INTERNET
> 32 NB
> F
> 
> 1536746474.773560 CbxxYF1uTyqC499HDe
> 192.168.20.15 137
> 10.190.129.26 137
> udp 39017
> - maanpc
> 1 C_INTERNET
> 32 NB
> F
> 
> 1536746452.751762 CbxxYF1uTyqC499HDe
> 192.168.20.15 137
> 10.190.129.26 137
> udp 39009
> - maanpc
> 1 C_INTERNET
> 32 NB
> F
> 
> 1536746453.510702 CbxxYF1uTyqC499HDe
> 192.168.20.15 137
> 10.190.129.26 137
> udp 39009
> - maanpc
> 1 C_INTERNET
> 32 NB
> F
> 
> 1536746454.275116 CbxxYF1uTyqC499HDe
> 192.168.20.15 137
> 10.190.129.26 137
> udp 39009
> - maanpc
> 1 C_INTERNET
> 32 NB
> F
> 
> 
> pf_ring / af packet didnt help. 
> 
> thanks
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181004/d0df2851/attachment-0001.html 

From shutchison at cert.org  Thu Oct  4 05:01:02 2018
From: shutchison at cert.org (Sean Hutchison)
Date: Thu, 4 Oct 2018 12:01:02 +0000
Subject: [Bro] Broctl segmentation fault
In-Reply-To: <83656C44-986A-49F5-BC68-0DABBC006A29@illinois.edu>
References: <CD76AAEFD1531643AB6A33368B516D770B600D85@marathon>
	<83656C44-986A-49F5-BC68-0DABBC006A29@illinois.edu>
Message-ID: <CD76AAEFD1531643AB6A33368B516D770B601132@marathon>

# bro -v
bro version 2.5.5

# bro -NN
Segmentation fault

# bro -b -i lo
listening on lo

^C1538653437.070325 received termination signal
1538653437.070325 208 packets received on interface lo, 0 dropped

# bro -i lo
Segmentation fault

# bro -i lo local
Segmentation fault

# ldd /opt/bro/bin/bro
        linux-vdso.so.1 =>  (0x00007fff99dfd000)
        libpcap.so.1 => /lib64/libpcap.so.1 (0x00007f148eec1000)
        libssl.so.10 => /lib64/libssl.so.10 (0x00007f148ec50000)
        libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f148e7ef000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f148e5d6000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f148e3c0000)
        libGeoIP.so.1 => /lib64/libGeoIP.so.1 (0x00007f148e190000)
        libtcmalloc.so.4 => /lib64/libtcmalloc.so.4 (0x00007f148dd9b000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f148db7f000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f148d97b000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f148d674000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f148d372000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f148d15c000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f148cd8f000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f148cb42000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f148c85a000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f148c656000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f148c423000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f148f102000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f148c215000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f148c011000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f148bdea000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f148bb88000)

No custom scripts being loaded via local.bro
Nothing in particular - did yum install/update of RedHat-based dependencies according to https://www.bro.org/sphinx/install/install.html#required-dependencies
Although I did build it against pfring first, using yum package from ntop repo - same issue, have since removed that and did regular build

Only configure switch was --prefix.

V/R
Sean

-----Original Message-----
From: Azoff, Justin S [mailto:jazoff at illinois.edu] 
Sent: Wednesday, October 03, 2018 3:01 PM
To: Sean Hutchison <shutchison at cert.org>
Cc: bro at bro.org
Subject: Re: [Bro] Broctl segmentation fault


> On Oct 3, 2018, at 2:46 PM, Sean Hutchison <shutchison at cert.org> wrote:
> 
> Hello,
>  
> After any build of Bro with Broctl 1.7, I?m experiencing the below error when broctl/scripts/check-config is run?
>  
> /opt/bro/share/broctl/scripts/check-config: line 50:  4463 Segmentation fault      "${bro}" $check_option "$@"
>  
> Anyone encountered this before? Cannot bypass doing broctl check ? broctl start results in failed/crashed processes.
>  
> This is on RHEL7.5, after building Bro-2.5.5 (I?ve tried other minor versions since 2.5 ? same issue).
>  
> Existing Bro cluster on RHEL7.5 boxes with Bro-2.5 and Broctl 1.5 works fine.
>  
> Any help would be greatly appreciated.
>  

check runs bro with the current configuration to see if it can start, so that's bro segfaulting there.. that's why start also fails..

What do you get if you try each of the following?

    bro -v
    bro -NN # just see if this runs or crashes
    bro -b -i lo
    bro -i lo
    bro -i lo local

You can hit control-c if any of those start successfully to get your prompt back.

I'm not aware of any issues like this, so it could be something with your configuration.

Do you have a customized local.bro at all?
Are you building bro against a particular libpcap or malloc implementation?
What does ldd /opt/bro/bin/bro output?

? 
Justin Azoff



From fatema.bannatwala at gmail.com  Thu Oct  4 05:57:14 2018
From: fatema.bannatwala at gmail.com (fatema bannatwala)
Date: Thu, 4 Oct 2018 08:57:14 -0400
Subject: [Bro] is there a bro script to ignore duplicated logs?
Message-ID: <CACX0rUSyEUGQdxY9hKYDWY6zq7J-zUci_ZVFhpmY4cP+Xdj67g@mail.gmail.com>

How many workers do you have in your cluster?
Many months ago, we had split-ed connections issue, where Seth provided a
script to add the worker node to conn.log to see where exactly packets are
being processed,i.e. which nodes.

You can run this script and see if the duplicate connections are happening
on which workers and go from there:

$ cat add-node-to-conn.bro
##! Add the name of the current node to conn.log

@load base/protocols/conn

export {
        redef record Conn::Info += {
                ## The name of the node where this connection was analyzed.
                node: string &log &optional;
        };
}

event connection_state_remove(c: connection) &priority=2
        {
        c$conn$node = peer_description;
        }
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181004/e2dcd517/attachment.html 

From michalpurzynski1 at gmail.com  Thu Oct  4 06:26:03 2018
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Thu, 4 Oct 2018 14:26:03 +0100
Subject: [Bro] is there a bro script to ignore duplicated logs?
In-Reply-To: <CACX0rUSyEUGQdxY9hKYDWY6zq7J-zUci_ZVFhpmY4cP+Xdj67g@mail.gmail.com>
References: <CACX0rUSyEUGQdxY9hKYDWY6zq7J-zUci_ZVFhpmY4cP+Xdj67g@mail.gmail.com>
Message-ID: <DC1A96F9-26D5-494B-AA40-2A5ACCB86208@gmail.com>

Most likely you have separate workers parsing the same traffic.

Can you load the script fatema told you about and also a capture loss script and report results for the hour or so?

If each worker sees the same duplicate traffic then the amount of packets processed will be very similar.

> On Oct 4, 2018, at 1:57 PM, fatema bannatwala <fatema.bannatwala at gmail.com> wrote:
> 
> How many workers do you have in your cluster?
> Many months ago, we had split-ed connections issue, where Seth provided a script to add the worker node to conn.log to see where exactly packets are being processed,i.e. which nodes.
> 
> You can run this script and see if the duplicate connections are happening on which workers and go from there:
> 
> $ cat add-node-to-conn.bro
> ##! Add the name of the current node to conn.log
> 
> @load base/protocols/conn
> 
> export {
>         redef record Conn::Info += {
>                 ## The name of the node where this connection was analyzed.
>                 node: string &log &optional;
>         };
> }
> 
> event connection_state_remove(c: connection) &priority=2
>         {
>         c$conn$node = peer_description;
>         }
> 
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181004/5e1d422d/attachment.html 

From jan.grashoefer at gmail.com  Thu Oct  4 06:33:00 2018
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Thu, 4 Oct 2018 15:33:00 +0200
Subject: [Bro] is there a bro script to ignore duplicated logs?
In-Reply-To: <CACX0rUSyEUGQdxY9hKYDWY6zq7J-zUci_ZVFhpmY4cP+Xdj67g@mail.gmail.com>
References: <CACX0rUSyEUGQdxY9hKYDWY6zq7J-zUci_ZVFhpmY4cP+Xdj67g@mail.gmail.com>
Message-ID: <de5e6fc0-bf16-0dc7-d31f-4bb29c2037b0@gmail.com>

On 04/10/2018 14:57, fatema bannatwala wrote:
> You can run this script and see if the duplicate connections are happening
> on which workers and go from there:

This can be further automated by using Justins bro-doctor script 
available as a package: https://github.com/ncsa/bro-doctor

Jan

From ericooi at gmail.com  Thu Oct  4 06:42:12 2018
From: ericooi at gmail.com (Eric Ooi)
Date: Thu, 4 Oct 2018 09:42:12 -0400
Subject: [Bro] is there a bro script to ignore duplicated logs?
In-Reply-To: <DC1A96F9-26D5-494B-AA40-2A5ACCB86208@gmail.com>
References: <CACX0rUSyEUGQdxY9hKYDWY6zq7J-zUci_ZVFhpmY4cP+Xdj67g@mail.gmail.com>
	<DC1A96F9-26D5-494B-AA40-2A5ACCB86208@gmail.com>
Message-ID: <CAEuvLSYTqAkPRS9VDo62+oZkOXBW8EOv_m192Q=ZGm-dy2pjug@mail.gmail.com>

We had a similar issue and it turned out that the SPAN the network engineer
configured was capturing a trunk such that any inter-VLAN traffic was being
analyzed multiple times.

On Thu, Oct 4, 2018 at 9:37 AM Micha? Purzy?ski <michalpurzynski1 at gmail.com>
wrote:

> Most likely you have separate workers parsing the same traffic.
>
> Can you load the script fatema told you about and also a capture loss
> script and report results for the hour or so?
>
> If each worker sees the same duplicate traffic then the amount of packets
> processed will be very similar.
>
> On Oct 4, 2018, at 1:57 PM, fatema bannatwala <fatema.bannatwala at gmail.com>
> wrote:
>
> How many workers do you have in your cluster?
> Many months ago, we had split-ed connections issue, where Seth provided a
> script to add the worker node to conn.log to see where exactly packets are
> being processed,i.e. which nodes.
>
> You can run this script and see if the duplicate connections are happening
> on which workers and go from there:
>
> $ cat add-node-to-conn.bro
> ##! Add the name of the current node to conn.log
>
> @load base/protocols/conn
>
> export {
>         redef record Conn::Info += {
>                 ## The name of the node where this connection was analyzed.
>                 node: string &log &optional;
>         };
> }
>
> event connection_state_remove(c: connection) &priority=2
>         {
>         c$conn$node = peer_description;
>         }
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181004/50a6a7ba/attachment-0001.html 

From jazoff at illinois.edu  Thu Oct  4 07:21:55 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 4 Oct 2018 14:21:55 +0000
Subject: [Bro] is there a bro script to ignore duplicated logs?
In-Reply-To: <CAEuvLSYTqAkPRS9VDo62+oZkOXBW8EOv_m192Q=ZGm-dy2pjug@mail.gmail.com>
References: <CACX0rUSyEUGQdxY9hKYDWY6zq7J-zUci_ZVFhpmY4cP+Xdj67g@mail.gmail.com>
	<DC1A96F9-26D5-494B-AA40-2A5ACCB86208@gmail.com>
	<CAEuvLSYTqAkPRS9VDo62+oZkOXBW8EOv_m192Q=ZGm-dy2pjug@mail.gmail.com>
Message-ID: <7977D733-513C-450C-B79B-ACF9D8C33E3C@illinois.edu>


> On Oct 4, 2018, at 9:42 AM, Eric Ooi <ericooi at gmail.com> wrote:
> 
> We had a similar issue and it turned out that the SPAN the network engineer configured was capturing a trunk such that any inter-VLAN traffic was being analyzed multiple times.  

Yeah, if the traffic was duplicated 2 (maybe 3) times this could be the problem.  In this case it's duplicated 12 times which almost definitely points to an lb_procs=12 in node.cfg and load balancing not working properly.


? 
Justin Azoff



From craig.edgmand at okstate.edu  Thu Oct  4 08:35:56 2018
From: craig.edgmand at okstate.edu (Edgmand, Craig)
Date: Thu, 4 Oct 2018 15:35:56 +0000
Subject: [Bro] OT: SuperMicro hardware recommendations
In-Reply-To: <E33DA773-F6EF-4A8E-9FB2-1EA54FD73389@gmail.com>
References: <94D13236-BB53-4807-93D0-F4A1BF0B369F@outlook.com>
	<151C7D23-39E9-4170-8954-CDFF566E4ED7@gmail.com>
	<EFAF45EB-9512-42EF-825F-A6F0DA0A5FCE@outlook.com>
	<CAGURB2cCMcg2rmxWGJLzXn4LxPoy4VkPCgeS08CoZgr6azW2Ng@mail.gmail.com>
	<E33DA773-F6EF-4A8E-9FB2-1EA54FD73389@gmail.com>
Message-ID: <DM5PR03MB261908806D09ED5F6DC03049E3EA0@DM5PR03MB2619.namprd03.prod.outlook.com>

With today?s scandal Super Micro might not be around/available for much longer.

https://www.cnbc.com/2018/10/04/super-micro-shares-plummet-amid-china-tech-spying-scandal.html

From: bro-bounces at bro.org <bro-bounces at bro.org> On Behalf Of Michal Purzynski
Sent: Monday, October 1, 2018 12:28 PM
To: Alex Waher <alexwis at gmail.com>
Cc: bro at bro.org List <bro at bro.org>
Subject: Re: [Bro] OT: SuperMicro hardware recommendations

One other non-HP brand I?ve always been recommending is Fujitsu.
 People are always surprised that ?Fujitsu makes servers?? but my experience has always been great with both the hardware side and the support.

They were cheaper than HP, last time I was buying them.


On Oct 1, 2018, at 3:26 PM, Alex Waher <alexwis at gmail.com<mailto:alexwis at gmail.com>> wrote:
Can also checkout the offerings from the OpenCompute project; https://www.opencompute.org/products<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.opencompute.org%2Fproducts&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7Cd78c5eba7c414f93711a08d627c37887%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C636740117834325239&sdata=aa7psklUAiD0HLA%2F3Q2MNXosOl%2B4996%2BOJrtK8fOrUw%3D&reserved=0>
Can even stick with HP if you like, their 'Cloudline' gear supports (some aspects) of OCP.

On Mon, Oct 1, 2018 at 3:19 PM Carlos Lopez <clopmz at outlook.com<mailto:clopmz at outlook.com>> wrote:
Thanks Michal. I didn't know that bad support from SuperMicro.
Then I will continue with HP. Do you think the HP DL360 Gen10 are suitable for IDS deployment?

--
Regards,
C. L. Martinez

?On 01/10/2018, 12:47, "Micha? Purzy?ski" <michalpurzynski1 at gmail.com<mailto:michalpurzynski1 at gmail.com>> wrote:

    I?d recommended everything but super micro.

    A box of matches to shine the light on the dark corners of your network will behave better than anything built on super micro.

    HP and other vendors may seem more expensive to buy, but the low quality of super micro servers, apparent lack of support, arbitrary cut support ?we do not support this anymore? and ridiculous replacement policy makes it a non-starter for me.

    Example: they wanted me to send all dead hard drives back before they can send me a replacement.
    And that would take days or a week or two weeks.

    Raid battery problems?

    SM requires you to send entire server back and in a short 6 weeks - 3 months you will see it back. Maybe.

    Mechanically, chassis are a joke, too.

    OOB? No OOB hardware monitoring. And Java 1.6 required. If you have to use any kind of proxies, socks, tunneling - forget.


    > On Oct 1, 2018, at 10:47 AM, Carlos Lopez <clopmz at outlook.com<mailto:clopmz at outlook.com>> wrote:
    >
    > Hi all,
    >
    > In the following days, we will setup a Bro-IDS distributed architecture using 10 sensors and one manager. Network bandwidth will be between 2GiB-4GiB. Until now we used HP servers, but we are looking at SuperMicro servers as well. Can anyone recommend series or model?
    > --
    > Regards,
    > C. L. Martinez
    >
    >
    > _______________________________________________
    > Bro mailing list
    > bro at bro-ids.org<mailto:bro at bro-ids.org>
    > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.ICSI.Berkeley.EDU%2Fmailman%2Flistinfo%2Fbro&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7Cd78c5eba7c414f93711a08d627c37887%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C636740117834335243&sdata=huyI0JVnJtMIcMFhUBAQ8bVjrvm4J3nTIZ0BeqOKODM%3D&reserved=0>



_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.ICSI.Berkeley.EDU%2Fmailman%2Flistinfo%2Fbro&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7Cd78c5eba7c414f93711a08d627c37887%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C636740117834335243&sdata=huyI0JVnJtMIcMFhUBAQ8bVjrvm4J3nTIZ0BeqOKODM%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181004/773c8c8e/attachment.html 

From johanna at icir.org  Thu Oct  4 08:36:20 2018
From: johanna at icir.org (Johanna Amann)
Date: Thu, 04 Oct 2018 08:36:20 -0700
Subject: [Bro] Broctl segmentation fault
In-Reply-To: <CD76AAEFD1531643AB6A33368B516D770B601132@marathon>
References: <CD76AAEFD1531643AB6A33368B516D770B600D85@marathon>
	<83656C44-986A-49F5-BC68-0DABBC006A29@illinois.edu>
	<CD76AAEFD1531643AB6A33368B516D770B601132@marathon>
Message-ID: <1A3F6849-ED1D-4363-9B54-B7F5D3294275@icir.org>

Hi,

Is there a change that you have binary plugins installed (netmap plugin, 
a few bro-pkg ones)?

They can cause crashes exactly like this. This behavior is fixed with 
Bro 2.6 (it will output an error message instead).

If that is the case - either recompiling or removing the binary plugins 
will fix this.

Johanna

On 4 Oct 2018, at 5:01, Sean Hutchison wrote:

> # bro -v
> bro version 2.5.5
>
> # bro -NN
> Segmentation fault
>
> # bro -b -i lo
> listening on lo
>
> ^C1538653437.070325 received termination signal
> 1538653437.070325 208 packets received on interface lo, 0 dropped
>
> # bro -i lo
> Segmentation fault
>
> # bro -i lo local
> Segmentation fault
>
> # ldd /opt/bro/bin/bro
>         linux-vdso.so.1 =>  (0x00007fff99dfd000)
>         libpcap.so.1 => /lib64/libpcap.so.1 (0x00007f148eec1000)
>         libssl.so.10 => /lib64/libssl.so.10 (0x00007f148ec50000)
>         libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f148e7ef000)
>         libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f148e5d6000)
>         libz.so.1 => /lib64/libz.so.1 (0x00007f148e3c0000)
>         libGeoIP.so.1 => /lib64/libGeoIP.so.1 (0x00007f148e190000)
>         libtcmalloc.so.4 => /lib64/libtcmalloc.so.4 
> (0x00007f148dd9b000)
>         libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f148db7f000)
>         libdl.so.2 => /lib64/libdl.so.2 (0x00007f148d97b000)
>         libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f148d674000)
>         libm.so.6 => /lib64/libm.so.6 (0x00007f148d372000)
>         libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f148d15c000)
>         libc.so.6 => /lib64/libc.so.6 (0x00007f148cd8f000)
>         libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 
> (0x00007f148cb42000)
>         libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f148c85a000)
>         libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f148c656000)
>         libk5crypto.so.3 => /lib64/libk5crypto.so.3 
> (0x00007f148c423000)
>         /lib64/ld-linux-x86-64.so.2 (0x00007f148f102000)
>         libkrb5support.so.0 => /lib64/libkrb5support.so.0 
> (0x00007f148c215000)
>         libkeyutils.so.1 => /lib64/libkeyutils.so.1 
> (0x00007f148c011000)
>         libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f148bdea000)
>         libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f148bb88000)
>
> No custom scripts being loaded via local.bro
> Nothing in particular - did yum install/update of RedHat-based 
> dependencies according to 
> https://www.bro.org/sphinx/install/install.html#required-dependencies
> Although I did build it against pfring first, using yum package from 
> ntop repo - same issue, have since removed that and did regular build
>
> Only configure switch was --prefix.
>
> V/R
> Sean
>
> -----Original Message-----
> From: Azoff, Justin S [mailto:jazoff at illinois.edu]
> Sent: Wednesday, October 03, 2018 3:01 PM
> To: Sean Hutchison <shutchison at cert.org>
> Cc: bro at bro.org
> Subject: Re: [Bro] Broctl segmentation fault
>
>
>> On Oct 3, 2018, at 2:46 PM, Sean Hutchison <shutchison at cert.org> 
>> wrote:
>>
>> Hello,
>>
>> After any build of Bro with Broctl 1.7, I?m experiencing the below 
>> error when broctl/scripts/check-config is run?
>>
>> /opt/bro/share/broctl/scripts/check-config: line 50:  4463 
>> Segmentation fault      "${bro}" $check_option "$@"
>>
>> Anyone encountered this before? Cannot bypass doing broctl check ? 
>> broctl start results in failed/crashed processes.
>>
>> This is on RHEL7.5, after building Bro-2.5.5 (I?ve tried other 
>> minor versions since 2.5 ? same issue).
>>
>> Existing Bro cluster on RHEL7.5 boxes with Bro-2.5 and Broctl 1.5 
>> works fine.
>>
>> Any help would be greatly appreciated.
>>
>
> check runs bro with the current configuration to see if it can start, 
> so that's bro segfaulting there.. that's why start also fails..
>
> What do you get if you try each of the following?
>
>     bro -v
>     bro -NN # just see if this runs or crashes
>     bro -b -i lo
>     bro -i lo
>     bro -i lo local
>
> You can hit control-c if any of those start successfully to get your 
> prompt back.
>
> I'm not aware of any issues like this, so it could be something with 
> your configuration.
>
> Do you have a customized local.bro at all?
> Are you building bro against a particular libpcap or malloc 
> implementation?
> What does ldd /opt/bro/bin/bro output?
>
> ?
> Justin Azoff
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

From perry at degy.com  Thu Oct  4 09:02:27 2018
From: perry at degy.com (Perry Lucas)
Date: Thu, 4 Oct 2018 16:02:27 +0000
Subject: [Bro] OT: SuperMicro hardware recommendations
In-Reply-To: <DM5PR03MB261908806D09ED5F6DC03049E3EA0@DM5PR03MB2619.namprd03.prod.outlook.com>
References: <94D13236-BB53-4807-93D0-F4A1BF0B369F@outlook.com>
	<151C7D23-39E9-4170-8954-CDFF566E4ED7@gmail.com>
	<EFAF45EB-9512-42EF-825F-A6F0DA0A5FCE@outlook.com>
	<CAGURB2cCMcg2rmxWGJLzXn4LxPoy4VkPCgeS08CoZgr6azW2Ng@mail.gmail.com>
	<E33DA773-F6EF-4A8E-9FB2-1EA54FD73389@gmail.com>
	<DM5PR03MB261908806D09ED5F6DC03049E3EA0@DM5PR03MB2619.namprd03.prod.outlook.com>
Message-ID: <SN6PR14MB2206825978D55D692C06573CC6EA0@SN6PR14MB2206.namprd14.prod.outlook.com>

I wouldn?t be surprised if other supply chains have been compromised as well. This is likely just the first.

From: bro-bounces at bro.org <bro-bounces at bro.org> On Behalf Of Edgmand, Craig
Sent: Thursday, October 4, 2018 11:36 AM
To: Micha? Purzy?ski <michalpurzynski1 at gmail.com>; Alex Waher <alexwis at gmail.com>
Cc: bro at bro.org List <bro at bro.org>
Subject: Re: [Bro] OT: SuperMicro hardware recommendations

With today?s scandal Super Micro might not be around/available for much longer.

https://www.cnbc.com/2018/10/04/super-micro-shares-plummet-amid-china-tech-spying-scandal.html

From: bro-bounces at bro.org<mailto:bro-bounces at bro.org> <bro-bounces at bro.org<mailto:bro-bounces at bro.org>> On Behalf Of Michal Purzynski
Sent: Monday, October 1, 2018 12:28 PM
To: Alex Waher <alexwis at gmail.com<mailto:alexwis at gmail.com>>
Cc: bro at bro.org<mailto:bro at bro.org> List <bro at bro.org<mailto:bro at bro.org>>
Subject: Re: [Bro] OT: SuperMicro hardware recommendations

One other non-HP brand I?ve always been recommending is Fujitsu.
 People are always surprised that ?Fujitsu makes servers?? but my experience has always been great with both the hardware side and the support.

They were cheaper than HP, last time I was buying them.


On Oct 1, 2018, at 3:26 PM, Alex Waher <alexwis at gmail.com<mailto:alexwis at gmail.com>> wrote:
Can also checkout the offerings from the OpenCompute project; https://www.opencompute.org/products<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.opencompute.org%2Fproducts&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7Cd78c5eba7c414f93711a08d627c37887%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C636740117834325239&sdata=aa7psklUAiD0HLA%2F3Q2MNXosOl%2B4996%2BOJrtK8fOrUw%3D&reserved=0>
Can even stick with HP if you like, their 'Cloudline' gear supports (some aspects) of OCP.

On Mon, Oct 1, 2018 at 3:19 PM Carlos Lopez <clopmz at outlook.com<mailto:clopmz at outlook.com>> wrote:
Thanks Michal. I didn't know that bad support from SuperMicro.
Then I will continue with HP. Do you think the HP DL360 Gen10 are suitable for IDS deployment?

--
Regards,
C. L. Martinez

?On 01/10/2018, 12:47, "Micha? Purzy?ski" <michalpurzynski1 at gmail.com<mailto:michalpurzynski1 at gmail.com>> wrote:

    I?d recommended everything but super micro.

    A box of matches to shine the light on the dark corners of your network will behave better than anything built on super micro.

    HP and other vendors may seem more expensive to buy, but the low quality of super micro servers, apparent lack of support, arbitrary cut support ?we do not support this anymore? and ridiculous replacement policy makes it a non-starter for me.

    Example: they wanted me to send all dead hard drives back before they can send me a replacement.
    And that would take days or a week or two weeks.

    Raid battery problems?

    SM requires you to send entire server back and in a short 6 weeks - 3 months you will see it back. Maybe.

    Mechanically, chassis are a joke, too.

    OOB? No OOB hardware monitoring. And Java 1.6 required. If you have to use any kind of proxies, socks, tunneling - forget.


    > On Oct 1, 2018, at 10:47 AM, Carlos Lopez <clopmz at outlook.com<mailto:clopmz at outlook.com>> wrote:
    >
    > Hi all,
    >
    > In the following days, we will setup a Bro-IDS distributed architecture using 10 sensors and one manager. Network bandwidth will be between 2GiB-4GiB. Until now we used HP servers, but we are looking at SuperMicro servers as well. Can anyone recommend series or model?
    > --
    > Regards,
    > C. L. Martinez
    >
    >
    > _______________________________________________
    > Bro mailing list
    > bro at bro-ids.org<mailto:bro at bro-ids.org>
    > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.ICSI.Berkeley.EDU%2Fmailman%2Flistinfo%2Fbro&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7Cd78c5eba7c414f93711a08d627c37887%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C636740117834335243&sdata=huyI0JVnJtMIcMFhUBAQ8bVjrvm4J3nTIZ0BeqOKODM%3D&reserved=0>



_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.ICSI.Berkeley.EDU%2Fmailman%2Flistinfo%2Fbro&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7Cd78c5eba7c414f93711a08d627c37887%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C636740117834335243&sdata=huyI0JVnJtMIcMFhUBAQ8bVjrvm4J3nTIZ0BeqOKODM%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181004/1a7ab72e/attachment.html 

From michalpurzynski1 at gmail.com  Thu Oct  4 09:15:00 2018
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Thu, 4 Oct 2018 17:15:00 +0100
Subject: [Bro] OT: SuperMicro hardware recommendations
In-Reply-To: <SN6PR14MB2206825978D55D692C06573CC6EA0@SN6PR14MB2206.namprd14.prod.outlook.com>
References: <94D13236-BB53-4807-93D0-F4A1BF0B369F@outlook.com>
	<151C7D23-39E9-4170-8954-CDFF566E4ED7@gmail.com>
	<EFAF45EB-9512-42EF-825F-A6F0DA0A5FCE@outlook.com>
	<CAGURB2cCMcg2rmxWGJLzXn4LxPoy4VkPCgeS08CoZgr6azW2Ng@mail.gmail.com>
	<E33DA773-F6EF-4A8E-9FB2-1EA54FD73389@gmail.com>
	<DM5PR03MB261908806D09ED5F6DC03049E3EA0@DM5PR03MB2619.namprd03.prod.outlook.com>
	<SN6PR14MB2206825978D55D692C06573CC6EA0@SN6PR14MB2206.namprd14.prod.outlook.com>
Message-ID: <985DB870-0B14-4A41-AFD0-4A0F99CD939A@gmail.com>

Didn?t Apple remove super micro back in 2016?
Aren?t aws servers custom anyway?

So yeah, poor Palo Alto ;)

With those allegations true or not, if supermicro goes out of business there?s no support anymore. 


> On Oct 4, 2018, at 5:02 PM, Perry Lucas <perry at degy.com> wrote:
> 
> I wouldn?t be surprised if other supply chains have been compromised as well. This is likely just the first.
>  
> From: bro-bounces at bro.org <bro-bounces at bro.org> On Behalf Of Edgmand, Craig
> Sent: Thursday, October 4, 2018 11:36 AM
> To: Micha? Purzy?ski <michalpurzynski1 at gmail.com>; Alex Waher <alexwis at gmail.com>
> Cc: bro at bro.org List <bro at bro.org>
> Subject: Re: [Bro] OT: SuperMicro hardware recommendations
>  
> With today?s scandal Super Micro might not be around/available for much longer.
>  
> https://www.cnbc.com/2018/10/04/super-micro-shares-plummet-amid-china-tech-spying-scandal.html
>  
> From: bro-bounces at bro.org <bro-bounces at bro.org> On Behalf Of Michal Purzynski
> Sent: Monday, October 1, 2018 12:28 PM
> To: Alex Waher <alexwis at gmail.com>
> Cc: bro at bro.org List <bro at bro.org>
> Subject: Re: [Bro] OT: SuperMicro hardware recommendations
>  
> One other non-HP brand I?ve always been recommending is Fujitsu.
>  People are always surprised that ?Fujitsu makes servers?? but my experience has always been great with both the hardware side and the support.
>  
> They were cheaper than HP, last time I was buying them.
>  
> 
> On Oct 1, 2018, at 3:26 PM, Alex Waher <alexwis at gmail.com> wrote:
> 
> Can also checkout the offerings from the OpenCompute project; https://www.opencompute.org/products
> Can even stick with HP if you like, their 'Cloudline' gear supports (some aspects) of OCP.
>  
> On Mon, Oct 1, 2018 at 3:19 PM Carlos Lopez <clopmz at outlook.com> wrote:
> Thanks Michal. I didn't know that bad support from SuperMicro.
> Then I will continue with HP. Do you think the HP DL360 Gen10 are suitable for IDS deployment?
> 
> -- 
> Regards,
> C. L. Martinez
> 
> ?On 01/10/2018, 12:47, "Micha? Purzy?ski" <michalpurzynski1 at gmail.com> wrote:
> 
>     I?d recommended everything but super micro.  
> 
>     A box of matches to shine the light on the dark corners of your network will behave better than anything built on super micro.
> 
>     HP and other vendors may seem more expensive to buy, but the low quality of super micro servers, apparent lack of support, arbitrary cut support ?we do not support this anymore? and ridiculous replacement policy makes it a non-starter for me.
> 
>     Example: they wanted me to send all dead hard drives back before they can send me a replacement.
>     And that would take days or a week or two weeks.
> 
>     Raid battery problems?
> 
>     SM requires you to send entire server back and in a short 6 weeks - 3 months you will see it back. Maybe.
> 
>     Mechanically, chassis are a joke, too.
> 
>     OOB? No OOB hardware monitoring. And Java 1.6 required. If you have to use any kind of proxies, socks, tunneling - forget.
> 
> 
>     > On Oct 1, 2018, at 10:47 AM, Carlos Lopez <clopmz at outlook.com> wrote:
>     > 
>     > Hi all,
>     > 
>     > In the following days, we will setup a Bro-IDS distributed architecture using 10 sensors and one manager. Network bandwidth will be between 2GiB-4GiB. Until now we used HP servers, but we are looking at SuperMicro servers as well. Can anyone recommend series or model?
>     > -- 
>     > Regards,
>     > C. L. Martinez
>     > 
>     > 
>     > _______________________________________________
>     > Bro mailing list
>     > bro at bro-ids.org
>     > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181004/55b23af7/attachment-0001.html 

From shutchison at cert.org  Thu Oct  4 10:14:09 2018
From: shutchison at cert.org (Sean Hutchison)
Date: Thu, 4 Oct 2018 17:14:09 +0000
Subject: [Bro] Broctl segmentation fault
In-Reply-To: <1A3F6849-ED1D-4363-9B54-B7F5D3294275@icir.org>
References: <CD76AAEFD1531643AB6A33368B516D770B600D85@marathon>
	<83656C44-986A-49F5-BC68-0DABBC006A29@illinois.edu>
	<CD76AAEFD1531643AB6A33368B516D770B601132@marathon>
	<1A3F6849-ED1D-4363-9B54-B7F5D3294275@icir.org>
Message-ID: <CD76AAEFD1531643AB6A33368B516D770B601387@marathon>

Yes, and I just removed the Bro Kafka plugin and no more error!

Thank you so much.

V/R
Sean

-----Original Message-----
From: Johanna Amann [mailto:johanna at icir.org] 
Sent: Thursday, October 04, 2018 11:36 AM
To: Sean Hutchison <shutchison at cert.org>
Cc: Azoff, Justin S <jazoff at illinois.edu>; bro at bro.org
Subject: Re: [Bro] Broctl segmentation fault

Hi,

Is there a change that you have binary plugins installed (netmap plugin, a few bro-pkg ones)?

They can cause crashes exactly like this. This behavior is fixed with Bro 2.6 (it will output an error message instead).

If that is the case - either recompiling or removing the binary plugins will fix this.

Johanna

On 4 Oct 2018, at 5:01, Sean Hutchison wrote:

> # bro -v
> bro version 2.5.5
>
> # bro -NN
> Segmentation fault
>
> # bro -b -i lo
> listening on lo
>
> ^C1538653437.070325 received termination signal
> 1538653437.070325 208 packets received on interface lo, 0 dropped
>
> # bro -i lo
> Segmentation fault
>
> # bro -i lo local
> Segmentation fault
>
> # ldd /opt/bro/bin/bro
>         linux-vdso.so.1 =>  (0x00007fff99dfd000)
>         libpcap.so.1 => /lib64/libpcap.so.1 (0x00007f148eec1000)
>         libssl.so.10 => /lib64/libssl.so.10 (0x00007f148ec50000)
>         libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f148e7ef000)
>         libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f148e5d6000)
>         libz.so.1 => /lib64/libz.so.1 (0x00007f148e3c0000)
>         libGeoIP.so.1 => /lib64/libGeoIP.so.1 (0x00007f148e190000)
>         libtcmalloc.so.4 => /lib64/libtcmalloc.so.4
> (0x00007f148dd9b000)
>         libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f148db7f000)
>         libdl.so.2 => /lib64/libdl.so.2 (0x00007f148d97b000)
>         libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f148d674000)
>         libm.so.6 => /lib64/libm.so.6 (0x00007f148d372000)
>         libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f148d15c000)
>         libc.so.6 => /lib64/libc.so.6 (0x00007f148cd8f000)
>         libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2
> (0x00007f148cb42000)
>         libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f148c85a000)
>         libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f148c656000)
>         libk5crypto.so.3 => /lib64/libk5crypto.so.3
> (0x00007f148c423000)
>         /lib64/ld-linux-x86-64.so.2 (0x00007f148f102000)
>         libkrb5support.so.0 => /lib64/libkrb5support.so.0
> (0x00007f148c215000)
>         libkeyutils.so.1 => /lib64/libkeyutils.so.1
> (0x00007f148c011000)
>         libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f148bdea000)
>         libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f148bb88000)
>
> No custom scripts being loaded via local.bro Nothing in particular - 
> did yum install/update of RedHat-based dependencies according to 
> https://www.bro.org/sphinx/install/install.html#required-dependencies
> Although I did build it against pfring first, using yum package from 
> ntop repo - same issue, have since removed that and did regular build
>
> Only configure switch was --prefix.
>
> V/R
> Sean
>
> -----Original Message-----
> From: Azoff, Justin S [mailto:jazoff at illinois.edu]
> Sent: Wednesday, October 03, 2018 3:01 PM
> To: Sean Hutchison <shutchison at cert.org>
> Cc: bro at bro.org
> Subject: Re: [Bro] Broctl segmentation fault
>
>
>> On Oct 3, 2018, at 2:46 PM, Sean Hutchison <shutchison at cert.org>
>> wrote:
>>
>> Hello,
>>
>> After any build of Bro with Broctl 1.7, I?m experiencing the below 
>> error when broctl/scripts/check-config is run?
>>
>> /opt/bro/share/broctl/scripts/check-config: line 50:  4463 
>> Segmentation fault      "${bro}" $check_option "$@"
>>
>> Anyone encountered this before? Cannot bypass doing broctl check ? 
>> broctl start results in failed/crashed processes.
>>
>> This is on RHEL7.5, after building Bro-2.5.5 (I?ve tried other minor 
>> versions since 2.5 ? same issue).
>>
>> Existing Bro cluster on RHEL7.5 boxes with Bro-2.5 and Broctl 1.5 
>> works fine.
>>
>> Any help would be greatly appreciated.
>>
>
> check runs bro with the current configuration to see if it can start, 
> so that's bro segfaulting there.. that's why start also fails..
>
> What do you get if you try each of the following?
>
>     bro -v
>     bro -NN # just see if this runs or crashes
>     bro -b -i lo
>     bro -i lo
>     bro -i lo local
>
> You can hit control-c if any of those start successfully to get your 
> prompt back.
>
> I'm not aware of any issues like this, so it could be something with 
> your configuration.
>
> Do you have a customized local.bro at all?
> Are you building bro against a particular libpcap or malloc 
> implementation?
> What does ldd /opt/bro/bin/bro output?
>
> ?
> Justin Azoff
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From zeolla at gmail.com  Thu Oct  4 14:06:06 2018
From: zeolla at gmail.com (Zeolla@GMail.com)
Date: Thu, 4 Oct 2018 17:06:06 -0400
Subject: [Bro] Broctl segmentation fault
In-Reply-To: <CD76AAEFD1531643AB6A33368B516D770B601387@marathon>
References: <CD76AAEFD1531643AB6A33368B516D770B600D85@marathon>
	<83656C44-986A-49F5-BC68-0DABBC006A29@illinois.edu>
	<CD76AAEFD1531643AB6A33368B516D770B601132@marathon>
	<1A3F6849-ED1D-4363-9B54-B7F5D3294275@icir.org>
	<CD76AAEFD1531643AB6A33368B516D770B601387@marathon>
Message-ID: <CAM1mTCr4fbm=890zN+ngfO0nbb9r4Ee6X5GC7bkP4MN-LtA7Vg@mail.gmail.com>

If you don't mind, can you share the steps you took to build and install
the plug-in?  What version?

Jon

On Thu, Oct 4, 2018, 13:23 Sean Hutchison <shutchison at cert.org> wrote:

> Yes, and I just removed the Bro Kafka plugin and no more error!
>
> Thank you so much.
>
> V/R
> Sean
>
> -----Original Message-----
> From: Johanna Amann [mailto:johanna at icir.org]
> Sent: Thursday, October 04, 2018 11:36 AM
> To: Sean Hutchison <shutchison at cert.org>
> Cc: Azoff, Justin S <jazoff at illinois.edu>; bro at bro.org
> Subject: Re: [Bro] Broctl segmentation fault
>
> Hi,
>
> Is there a change that you have binary plugins installed (netmap plugin, a
> few bro-pkg ones)?
>
> They can cause crashes exactly like this. This behavior is fixed with Bro
> 2.6 (it will output an error message instead).
>
> If that is the case - either recompiling or removing the binary plugins
> will fix this.
>
> Johanna
>
> On 4 Oct 2018, at 5:01, Sean Hutchison wrote:
>
> > # bro -v
> > bro version 2.5.5
> >
> > # bro -NN
> > Segmentation fault
> >
> > # bro -b -i lo
> > listening on lo
> >
> > ^C1538653437.070325 received termination signal
> > 1538653437.070325 208 packets received on interface lo, 0 dropped
> >
> > # bro -i lo
> > Segmentation fault
> >
> > # bro -i lo local
> > Segmentation fault
> >
> > # ldd /opt/bro/bin/bro
> >         linux-vdso.so.1 =>  (0x00007fff99dfd000)
> >         libpcap.so.1 => /lib64/libpcap.so.1 (0x00007f148eec1000)
> >         libssl.so.10 => /lib64/libssl.so.10 (0x00007f148ec50000)
> >         libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f148e7ef000)
> >         libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f148e5d6000)
> >         libz.so.1 => /lib64/libz.so.1 (0x00007f148e3c0000)
> >         libGeoIP.so.1 => /lib64/libGeoIP.so.1 (0x00007f148e190000)
> >         libtcmalloc.so.4 => /lib64/libtcmalloc.so.4
> > (0x00007f148dd9b000)
> >         libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f148db7f000)
> >         libdl.so.2 => /lib64/libdl.so.2 (0x00007f148d97b000)
> >         libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f148d674000)
> >         libm.so.6 => /lib64/libm.so.6 (0x00007f148d372000)
> >         libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f148d15c000)
> >         libc.so.6 => /lib64/libc.so.6 (0x00007f148cd8f000)
> >         libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2
> > (0x00007f148cb42000)
> >         libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f148c85a000)
> >         libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f148c656000)
> >         libk5crypto.so.3 => /lib64/libk5crypto.so.3
> > (0x00007f148c423000)
> >         /lib64/ld-linux-x86-64.so.2 (0x00007f148f102000)
> >         libkrb5support.so.0 => /lib64/libkrb5support.so.0
> > (0x00007f148c215000)
> >         libkeyutils.so.1 => /lib64/libkeyutils.so.1
> > (0x00007f148c011000)
> >         libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f148bdea000)
> >         libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f148bb88000)
> >
> > No custom scripts being loaded via local.bro Nothing in particular -
> > did yum install/update of RedHat-based dependencies according to
> > https://www.bro.org/sphinx/install/install.html#required-dependencies
> > Although I did build it against pfring first, using yum package from
> > ntop repo - same issue, have since removed that and did regular build
> >
> > Only configure switch was --prefix.
> >
> > V/R
> > Sean
> >
> > -----Original Message-----
> > From: Azoff, Justin S [mailto:jazoff at illinois.edu]
> > Sent: Wednesday, October 03, 2018 3:01 PM
> > To: Sean Hutchison <shutchison at cert.org>
> > Cc: bro at bro.org
> > Subject: Re: [Bro] Broctl segmentation fault
> >
> >
> >> On Oct 3, 2018, at 2:46 PM, Sean Hutchison <shutchison at cert.org>
> >> wrote:
> >>
> >> Hello,
> >>
> >> After any build of Bro with Broctl 1.7, I?m experiencing the below
> >> error when broctl/scripts/check-config is run?
> >>
> >> /opt/bro/share/broctl/scripts/check-config: line 50:  4463
> >> Segmentation fault      "${bro}" $check_option "$@"
> >>
> >> Anyone encountered this before? Cannot bypass doing broctl check ?
> >> broctl start results in failed/crashed processes.
> >>
> >> This is on RHEL7.5, after building Bro-2.5.5 (I?ve tried other minor
> >> versions since 2.5 ? same issue).
> >>
> >> Existing Bro cluster on RHEL7.5 boxes with Bro-2.5 and Broctl 1.5
> >> works fine.
> >>
> >> Any help would be greatly appreciated.
> >>
> >
> > check runs bro with the current configuration to see if it can start,
> > so that's bro segfaulting there.. that's why start also fails..
> >
> > What do you get if you try each of the following?
> >
> >     bro -v
> >     bro -NN # just see if this runs or crashes
> >     bro -b -i lo
> >     bro -i lo
> >     bro -i lo local
> >
> > You can hit control-c if any of those start successfully to get your
> > prompt back.
> >
> > I'm not aware of any issues like this, so it could be something with
> > your configuration.
> >
> > Do you have a customized local.bro at all?
> > Are you building bro against a particular libpcap or malloc
> > implementation?
> > What does ldd /opt/bro/bin/bro output?
> >
> > ?
> > Justin Azoff
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-- 

Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181004/6f3ab12b/attachment.html 

From klehigh at iu.edu  Fri Oct  5 03:52:44 2018
From: klehigh at iu.edu (Keith Lehigh)
Date: Fri, 05 Oct 2018 06:52:44 -0400
Subject: [Bro] BroCon 2018 Lightning talks
Message-ID: <64508E55-0AEB-4A02-81CC-4E11DA36602C@iu.edu>

Hi Folks,
    We would like to have lightning talks again this year.  If we have more interest than we have slots available, we?ll review them via the review committee.

Google form link : https://goo.gl/forms/qJsc9fG24smSyLUQ2

We ask for your name, email address, organization, talk title and a short description of your talk.

Lightning talk abstract:
Have you contributed a package to the new Bro Package Manager? Or do you have something interesting to share related to Bro that doesn?t fit into a traditional presentation? Great! We?d love to hear from you. We have scheduled a 45-minute session for lightning talks. Be prepared to quickly identify your point, demonstrate it, and provide a link or contact info for later follow-up. We?d like to accommodate as many talks as possible so please limit your talk to less than 5 minutes. No commercially incentivized presentations, please.

- Keith
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3740 bytes
Desc: S/MIME digital signature
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181005/7ada2bdf/attachment.bin 

From shutchison at cert.org  Fri Oct  5 04:55:30 2018
From: shutchison at cert.org (Sean Hutchison)
Date: Fri, 5 Oct 2018 11:55:30 +0000
Subject: [Bro] Broctl segmentation fault
In-Reply-To: <CAM1mTCr4fbm=890zN+ngfO0nbb9r4Ee6X5GC7bkP4MN-LtA7Vg@mail.gmail.com>
References: <CD76AAEFD1531643AB6A33368B516D770B600D85@marathon>
	<83656C44-986A-49F5-BC68-0DABBC006A29@illinois.edu>
	<CD76AAEFD1531643AB6A33368B516D770B601132@marathon>
	<1A3F6849-ED1D-4363-9B54-B7F5D3294275@icir.org>
	<CD76AAEFD1531643AB6A33368B516D770B601387@marathon>
	<CAM1mTCr4fbm=890zN+ngfO0nbb9r4Ee6X5GC7bkP4MN-LtA7Vg@mail.gmail.com>
Message-ID: <CD76AAEFD1531643AB6A33368B516D770B601A4D@marathon>

Ya, first install librdkafka (there?s probably a newer version ? make sure it supports your Kafka broker version) ?
curl --silent -L -k https://github.com/edenhill/librdkafka/archive/v0.9.5.tar.gz | tar xz
cd librdkafka-0.9.5
./configure
make
make install

Then get bro-plugins repo and build kafka plugin against version of Bro you?re using by pointing it to where you extracted bro source?
git clone https://github.com/bro/bro-plugins.git
cd bro-plugins/kafka/
./configure --bro-dist=/path/to/bro-2.#.#
make && make install

Confirm with?
bro -N Bro::Kafka

See https://archive.apache.org/dist/metron/0.4.0/site-book/metron-sensors/bro-plugin-kafka/index.html for example configurations.

V/R
Sean

From: Zeolla at GMail.com [mailto:zeolla at gmail.com]
Sent: Thursday, October 04, 2018 5:06 PM
To: Sean Hutchison <shutchison at cert.org>
Cc: Johanna Amann <johanna at icir.org>; bro at bro.org
Subject: Re: [Bro] Broctl segmentation fault

If you don't mind, can you share the steps you took to build and install the plug-in?  What version?

Jon
On Thu, Oct 4, 2018, 13:23 Sean Hutchison <shutchison at cert.org<mailto:shutchison at cert.org>> wrote:
Yes, and I just removed the Bro Kafka plugin and no more error!

Thank you so much.

V/R
Sean

-----Original Message-----
From: Johanna Amann [mailto:johanna at icir.org<mailto:johanna at icir.org>]
Sent: Thursday, October 04, 2018 11:36 AM
To: Sean Hutchison <shutchison at cert.org<mailto:shutchison at cert.org>>
Cc: Azoff, Justin S <jazoff at illinois.edu<mailto:jazoff at illinois.edu>>; bro at bro.org<mailto:bro at bro.org>
Subject: Re: [Bro] Broctl segmentation fault

Hi,

Is there a change that you have binary plugins installed (netmap plugin, a few bro-pkg ones)?

They can cause crashes exactly like this. This behavior is fixed with Bro 2.6 (it will output an error message instead).

If that is the case - either recompiling or removing the binary plugins will fix this.

Johanna

On 4 Oct 2018, at 5:01, Sean Hutchison wrote:

> # bro -v
> bro version 2.5.5
>
> # bro -NN
> Segmentation fault
>
> # bro -b -i lo
> listening on lo
>
> ^C1538653437.070325 received termination signal
> 1538653437.070325 208 packets received on interface lo, 0 dropped
>
> # bro -i lo
> Segmentation fault
>
> # bro -i lo local
> Segmentation fault
>
> # ldd /opt/bro/bin/bro
>         linux-vdso.so.1 =>  (0x00007fff99dfd000)
>         libpcap.so.1 => /lib64/libpcap.so.1 (0x00007f148eec1000)
>         libssl.so.10 => /lib64/libssl.so.10 (0x00007f148ec50000)
>         libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f148e7ef000)
>         libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f148e5d6000)
>         libz.so.1 => /lib64/libz.so.1 (0x00007f148e3c0000)
>         libGeoIP.so.1 => /lib64/libGeoIP.so.1 (0x00007f148e190000)
>         libtcmalloc.so.4 => /lib64/libtcmalloc.so.4
> (0x00007f148dd9b000)
>         libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f148db7f000)
>         libdl.so.2 => /lib64/libdl.so.2 (0x00007f148d97b000)
>         libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f148d674000)
>         libm.so.6 => /lib64/libm.so.6 (0x00007f148d372000)
>         libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f148d15c000)
>         libc.so.6 => /lib64/libc.so.6 (0x00007f148cd8f000)
>         libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2
> (0x00007f148cb42000)
>         libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f148c85a000)
>         libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f148c656000)
>         libk5crypto.so.3 => /lib64/libk5crypto.so.3
> (0x00007f148c423000)
>         /lib64/ld-linux-x86-64.so.2 (0x00007f148f102000)
>         libkrb5support.so.0 => /lib64/libkrb5support.so.0
> (0x00007f148c215000)
>         libkeyutils.so.1 => /lib64/libkeyutils.so.1
> (0x00007f148c011000)
>         libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f148bdea000)
>         libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f148bb88000)
>
> No custom scripts being loaded via local.bro Nothing in particular -
> did yum install/update of RedHat-based dependencies according to
> https://www.bro.org/sphinx/install/install.html#required-dependencies
> Although I did build it against pfring first, using yum package from
> ntop repo - same issue, have since removed that and did regular build
>
> Only configure switch was --prefix.
>
> V/R
> Sean
>
> -----Original Message-----
> From: Azoff, Justin S [mailto:jazoff at illinois.edu<mailto:jazoff at illinois.edu>]
> Sent: Wednesday, October 03, 2018 3:01 PM
> To: Sean Hutchison <shutchison at cert.org<mailto:shutchison at cert.org>>
> Cc: bro at bro.org<mailto:bro at bro.org>
> Subject: Re: [Bro] Broctl segmentation fault
>
>
>> On Oct 3, 2018, at 2:46 PM, Sean Hutchison <shutchison at cert.org<mailto:shutchison at cert.org>>
>> wrote:
>>
>> Hello,
>>
>> After any build of Bro with Broctl 1.7, I?m experiencing the below
>> error when broctl/scripts/check-config is run?
>>
>> /opt/bro/share/broctl/scripts/check-config: line 50:  4463
>> Segmentation fault      "${bro}" $check_option "$@"
>>
>> Anyone encountered this before? Cannot bypass doing broctl check ?
>> broctl start results in failed/crashed processes.
>>
>> This is on RHEL7.5, after building Bro-2.5.5 (I?ve tried other minor
>> versions since 2.5 ? same issue).
>>
>> Existing Bro cluster on RHEL7.5 boxes with Bro-2.5 and Broctl 1.5
>> works fine.
>>
>> Any help would be greatly appreciated.
>>
>
> check runs bro with the current configuration to see if it can start,
> so that's bro segfaulting there.. that's why start also fails..
>
> What do you get if you try each of the following?
>
>     bro -v
>     bro -NN # just see if this runs or crashes
>     bro -b -i lo
>     bro -i lo
>     bro -i lo local
>
> You can hit control-c if any of those start successfully to get your
> prompt back.
>
> I'm not aware of any issues like this, so it could be something with
> your configuration.
>
> Do you have a customized local.bro at all?
> Are you building bro against a particular libpcap or malloc
> implementation?
> What does ldd /opt/bro/bin/bro output?
>
> ?
> Justin Azoff
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org<mailto:bro at bro-ids.org>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
--

Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181005/8b314469/attachment-0001.html 

From zeolla at gmail.com  Fri Oct  5 09:36:00 2018
From: zeolla at gmail.com (Zeolla@GMail.com)
Date: Fri, 5 Oct 2018 12:36:00 -0400
Subject: [Bro] Broctl segmentation fault
In-Reply-To: <CD76AAEFD1531643AB6A33368B516D770B601A4D@marathon>
References: <CD76AAEFD1531643AB6A33368B516D770B600D85@marathon>
	<83656C44-986A-49F5-BC68-0DABBC006A29@illinois.edu>
	<CD76AAEFD1531643AB6A33368B516D770B601132@marathon>
	<1A3F6849-ED1D-4363-9B54-B7F5D3294275@icir.org>
	<CD76AAEFD1531643AB6A33368B516D770B601387@marathon>
	<CAM1mTCr4fbm=890zN+ngfO0nbb9r4Ee6X5GC7bkP4MN-LtA7Vg@mail.gmail.com>
	<CD76AAEFD1531643AB6A33368B516D770B601A4D@marathon>
Message-ID: <CAM1mTCqyn94BAE+pgAsUX7jvZpXZXd0dsA=tEbcxutxOCq4EPQ@mail.gmail.com>

Sounds like you are looking at a very old version of the plugin, since
bro/bro-plugins has been completely deprecated at this point.  Can you use
bro-pkg to install apache/metron-bro-plugin-kafka?  It should be a bit more
robust and up to date.  Let me know if you have any issues when taking this
approach.

Jon

On Fri, Oct 5, 2018, 07:55 Sean Hutchison <shutchison at cert.org> wrote:

> Ya, first install librdkafka (there?s probably a newer version ? make sure
> it supports your Kafka broker version) ?
>
> curl --silent -L -k
> https://github.com/edenhill/librdkafka/archive/v0.9.5.tar.gz | tar xz
>
> cd librdkafka-0.9.5
>
> ./configure
>
> make
>
> make install
>
>
>
> Then get bro-plugins repo and build kafka plugin against version of Bro
> you?re using by pointing it to where you extracted bro source?
>
> git clone https://github.com/bro/bro-plugins.git
>
> cd bro-plugins/kafka/
>
> ./configure --bro-dist=/path/to/bro-2.#.#
>
> make && make install
>
>
>
> Confirm with?
>
> bro -N Bro::Kafka
>
>
>
> See
> https://archive.apache.org/dist/metron/0.4.0/site-book/metron-sensors/bro-plugin-kafka/index.html
> for example configurations.
>
>
>
> V/R
>
> Sean
>
>
>
> *From:* Zeolla at GMail.com [mailto:zeolla at gmail.com]
> *Sent:* Thursday, October 04, 2018 5:06 PM
> *To:* Sean Hutchison <shutchison at cert.org>
> *Cc:* Johanna Amann <johanna at icir.org>; bro at bro.org
>
>
> *Subject:* Re: [Bro] Broctl segmentation fault
>
>
>
> If you don't mind, can you share the steps you took to build and install
> the plug-in?  What version?
>
>
>
> Jon
>
> On Thu, Oct 4, 2018, 13:23 Sean Hutchison <shutchison at cert.org> wrote:
>
> Yes, and I just removed the Bro Kafka plugin and no more error!
>
> Thank you so much.
>
> V/R
> Sean
>
> -----Original Message-----
> From: Johanna Amann [mailto:johanna at icir.org]
> Sent: Thursday, October 04, 2018 11:36 AM
> To: Sean Hutchison <shutchison at cert.org>
> Cc: Azoff, Justin S <jazoff at illinois.edu>; bro at bro.org
> Subject: Re: [Bro] Broctl segmentation fault
>
> Hi,
>
> Is there a change that you have binary plugins installed (netmap plugin, a
> few bro-pkg ones)?
>
> They can cause crashes exactly like this. This behavior is fixed with Bro
> 2.6 (it will output an error message instead).
>
> If that is the case - either recompiling or removing the binary plugins
> will fix this.
>
> Johanna
>
> On 4 Oct 2018, at 5:01, Sean Hutchison wrote:
>
> > # bro -v
> > bro version 2.5.5
> >
> > # bro -NN
> > Segmentation fault
> >
> > # bro -b -i lo
> > listening on lo
> >
> > ^C1538653437.070325 received termination signal
> > 1538653437.070325 208 packets received on interface lo, 0 dropped
> >
> > # bro -i lo
> > Segmentation fault
> >
> > # bro -i lo local
> > Segmentation fault
> >
> > # ldd /opt/bro/bin/bro
> >         linux-vdso.so.1 =>  (0x00007fff99dfd000)
> >         libpcap.so.1 => /lib64/libpcap.so.1 (0x00007f148eec1000)
> >         libssl.so.10 => /lib64/libssl.so.10 (0x00007f148ec50000)
> >         libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f148e7ef000)
> >         libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f148e5d6000)
> >         libz.so.1 => /lib64/libz.so.1 (0x00007f148e3c0000)
> >         libGeoIP.so.1 => /lib64/libGeoIP.so.1 (0x00007f148e190000)
> >         libtcmalloc.so.4 => /lib64/libtcmalloc.so.4
> > (0x00007f148dd9b000)
> >         libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f148db7f000)
> >         libdl.so.2 => /lib64/libdl.so.2 (0x00007f148d97b000)
> >         libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f148d674000)
> >         libm.so.6 => /lib64/libm.so.6 (0x00007f148d372000)
> >         libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f148d15c000)
> >         libc.so.6 => /lib64/libc.so.6 (0x00007f148cd8f000)
> >         libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2
> > (0x00007f148cb42000)
> >         libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f148c85a000)
> >         libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f148c656000)
> >         libk5crypto.so.3 => /lib64/libk5crypto.so.3
> > (0x00007f148c423000)
> >         /lib64/ld-linux-x86-64.so.2 (0x00007f148f102000)
> >         libkrb5support.so.0 => /lib64/libkrb5support.so.0
> > (0x00007f148c215000)
> >         libkeyutils.so.1 => /lib64/libkeyutils.so.1
> > (0x00007f148c011000)
> >         libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f148bdea000)
> >         libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f148bb88000)
> >
> > No custom scripts being loaded via local.bro Nothing in particular -
> > did yum install/update of RedHat-based dependencies according to
> > https://www.bro.org/sphinx/install/install.html#required-dependencies
> > Although I did build it against pfring first, using yum package from
> > ntop repo - same issue, have since removed that and did regular build
> >
> > Only configure switch was --prefix.
> >
> > V/R
> > Sean
> >
> > -----Original Message-----
> > From: Azoff, Justin S [mailto:jazoff at illinois.edu]
> > Sent: Wednesday, October 03, 2018 3:01 PM
> > To: Sean Hutchison <shutchison at cert.org>
> > Cc: bro at bro.org
> > Subject: Re: [Bro] Broctl segmentation fault
> >
> >
> >> On Oct 3, 2018, at 2:46 PM, Sean Hutchison <shutchison at cert.org>
> >> wrote:
> >>
> >> Hello,
> >>
> >> After any build of Bro with Broctl 1.7, I?m experiencing the below
> >> error when broctl/scripts/check-config is run?
> >>
> >> /opt/bro/share/broctl/scripts/check-config: line 50:  4463
> >> Segmentation fault      "${bro}" $check_option "$@"
> >>
> >> Anyone encountered this before? Cannot bypass doing broctl check ?
> >> broctl start results in failed/crashed processes.
> >>
> >> This is on RHEL7.5, after building Bro-2.5.5 (I?ve tried other minor
> >> versions since 2.5 ? same issue).
> >>
> >> Existing Bro cluster on RHEL7.5 boxes with Bro-2.5 and Broctl 1.5
> >> works fine.
> >>
> >> Any help would be greatly appreciated.
> >>
> >
> > check runs bro with the current configuration to see if it can start,
> > so that's bro segfaulting there.. that's why start also fails..
> >
> > What do you get if you try each of the following?
> >
> >     bro -v
> >     bro -NN # just see if this runs or crashes
> >     bro -b -i lo
> >     bro -i lo
> >     bro -i lo local
> >
> > You can hit control-c if any of those start successfully to get your
> > prompt back.
> >
> > I'm not aware of any issues like this, so it could be something with
> > your configuration.
> >
> > Do you have a customized local.bro at all?
> > Are you building bro against a particular libpcap or malloc
> > implementation?
> > What does ldd /opt/bro/bin/bro output?
> >
> > ?
> > Justin Azoff
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> --
>
> Jon
>
-- 

Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181005/922835d5/attachment.html 

From shutchison at cert.org  Fri Oct  5 10:19:14 2018
From: shutchison at cert.org (Sean Hutchison)
Date: Fri, 5 Oct 2018 17:19:14 +0000
Subject: [Bro] Broctl segmentation fault
In-Reply-To: <CAM1mTCqyn94BAE+pgAsUX7jvZpXZXd0dsA=tEbcxutxOCq4EPQ@mail.gmail.com>
References: <CD76AAEFD1531643AB6A33368B516D770B600D85@marathon>
	<83656C44-986A-49F5-BC68-0DABBC006A29@illinois.edu>
	<CD76AAEFD1531643AB6A33368B516D770B601132@marathon>
	<1A3F6849-ED1D-4363-9B54-B7F5D3294275@icir.org>
	<CD76AAEFD1531643AB6A33368B516D770B601387@marathon>
	<CAM1mTCr4fbm=890zN+ngfO0nbb9r4Ee6X5GC7bkP4MN-LtA7Vg@mail.gmail.com>
	<CD76AAEFD1531643AB6A33368B516D770B601A4D@marathon>
	<CAM1mTCqyn94BAE+pgAsUX7jvZpXZXd0dsA=tEbcxutxOCq4EPQ@mail.gmail.com>
Message-ID: <CD76AAEFD1531643AB6A33368B516D770B601C21@marathon>

Well, we don?t really have a need to use the bro kafka plugin currently ? it was just for a bit of testing previously ? but I can use bro-pkg in the future or for other plugins/scripts.

V/R
Sean

From: Zeolla at GMail.com [mailto:zeolla at gmail.com]
Sent: Friday, October 05, 2018 12:36 PM
To: Sean Hutchison <shutchison at cert.org>
Cc: Johanna Amann <johanna at icir.org>; bro at bro.org
Subject: Re: [Bro] Broctl segmentation fault

Sounds like you are looking at a very old version of the plugin, since bro/bro-plugins has been completely deprecated at this point.  Can you use bro-pkg to install apache/metron-bro-plugin-kafka?  It should be a bit more robust and up to date.  Let me know if you have any issues when taking this approach.

Jon
On Fri, Oct 5, 2018, 07:55 Sean Hutchison <shutchison at cert.org<mailto:shutchison at cert.org>> wrote:
Ya, first install librdkafka (there?s probably a newer version ? make sure it supports your Kafka broker version) ?
curl --silent -L -k https://github.com/edenhill/librdkafka/archive/v0.9.5.tar.gz | tar xz
cd librdkafka-0.9.5
./configure
make
make install

Then get bro-plugins repo and build kafka plugin against version of Bro you?re using by pointing it to where you extracted bro source?
git clone https://github.com/bro/bro-plugins.git
cd bro-plugins/kafka/
./configure --bro-dist=/path/to/bro-2.#.#
make && make install

Confirm with?
bro -N Bro::Kafka

See https://archive.apache.org/dist/metron/0.4.0/site-book/metron-sensors/bro-plugin-kafka/index.html for example configurations.

V/R
Sean

From: Zeolla at GMail.com<mailto:Zeolla at GMail.com> [mailto:zeolla at gmail.com<mailto:zeolla at gmail.com>]
Sent: Thursday, October 04, 2018 5:06 PM
To: Sean Hutchison <shutchison at cert.org<mailto:shutchison at cert.org>>
Cc: Johanna Amann <johanna at icir.org<mailto:johanna at icir.org>>; bro at bro.org<mailto:bro at bro.org>

Subject: Re: [Bro] Broctl segmentation fault

If you don't mind, can you share the steps you took to build and install the plug-in?  What version?

Jon
On Thu, Oct 4, 2018, 13:23 Sean Hutchison <shutchison at cert.org<mailto:shutchison at cert.org>> wrote:
Yes, and I just removed the Bro Kafka plugin and no more error!

Thank you so much.

V/R
Sean

-----Original Message-----
From: Johanna Amann [mailto:johanna at icir.org<mailto:johanna at icir.org>]
Sent: Thursday, October 04, 2018 11:36 AM
To: Sean Hutchison <shutchison at cert.org<mailto:shutchison at cert.org>>
Cc: Azoff, Justin S <jazoff at illinois.edu<mailto:jazoff at illinois.edu>>; bro at bro.org<mailto:bro at bro.org>
Subject: Re: [Bro] Broctl segmentation fault

Hi,

Is there a change that you have binary plugins installed (netmap plugin, a few bro-pkg ones)?

They can cause crashes exactly like this. This behavior is fixed with Bro 2.6 (it will output an error message instead).

If that is the case - either recompiling or removing the binary plugins will fix this.

Johanna

On 4 Oct 2018, at 5:01, Sean Hutchison wrote:

> # bro -v
> bro version 2.5.5
>
> # bro -NN
> Segmentation fault
>
> # bro -b -i lo
> listening on lo
>
> ^C1538653437.070325 received termination signal
> 1538653437.070325 208 packets received on interface lo, 0 dropped
>
> # bro -i lo
> Segmentation fault
>
> # bro -i lo local
> Segmentation fault
>
> # ldd /opt/bro/bin/bro
>         linux-vdso.so.1 =>  (0x00007fff99dfd000)
>         libpcap.so.1 => /lib64/libpcap.so.1 (0x00007f148eec1000)
>         libssl.so.10 => /lib64/libssl.so.10 (0x00007f148ec50000)
>         libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f148e7ef000)
>         libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f148e5d6000)
>         libz.so.1 => /lib64/libz.so.1 (0x00007f148e3c0000)
>         libGeoIP.so.1 => /lib64/libGeoIP.so.1 (0x00007f148e190000)
>         libtcmalloc.so.4 => /lib64/libtcmalloc.so.4
> (0x00007f148dd9b000)
>         libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f148db7f000)
>         libdl.so.2 => /lib64/libdl.so.2 (0x00007f148d97b000)
>         libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f148d674000)
>         libm.so.6 => /lib64/libm.so.6 (0x00007f148d372000)
>         libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f148d15c000)
>         libc.so.6 => /lib64/libc.so.6 (0x00007f148cd8f000)
>         libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2
> (0x00007f148cb42000)
>         libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f148c85a000)
>         libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f148c656000)
>         libk5crypto.so.3 => /lib64/libk5crypto.so.3
> (0x00007f148c423000)
>         /lib64/ld-linux-x86-64.so.2 (0x00007f148f102000)
>         libkrb5support.so.0 => /lib64/libkrb5support.so.0
> (0x00007f148c215000)
>         libkeyutils.so.1 => /lib64/libkeyutils.so.1
> (0x00007f148c011000)
>         libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f148bdea000)
>         libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f148bb88000)
>
> No custom scripts being loaded via local.bro Nothing in particular -
> did yum install/update of RedHat-based dependencies according to
> https://www.bro.org/sphinx/install/install.html#required-dependencies
> Although I did build it against pfring first, using yum package from
> ntop repo - same issue, have since removed that and did regular build
>
> Only configure switch was --prefix.
>
> V/R
> Sean
>
> -----Original Message-----
> From: Azoff, Justin S [mailto:jazoff at illinois.edu<mailto:jazoff at illinois.edu>]
> Sent: Wednesday, October 03, 2018 3:01 PM
> To: Sean Hutchison <shutchison at cert.org<mailto:shutchison at cert.org>>
> Cc: bro at bro.org<mailto:bro at bro.org>
> Subject: Re: [Bro] Broctl segmentation fault
>
>
>> On Oct 3, 2018, at 2:46 PM, Sean Hutchison <shutchison at cert.org<mailto:shutchison at cert.org>>
>> wrote:
>>
>> Hello,
>>
>> After any build of Bro with Broctl 1.7, I?m experiencing the below
>> error when broctl/scripts/check-config is run?
>>
>> /opt/bro/share/broctl/scripts/check-config: line 50:  4463
>> Segmentation fault      "${bro}" $check_option "$@"
>>
>> Anyone encountered this before? Cannot bypass doing broctl check ?
>> broctl start results in failed/crashed processes.
>>
>> This is on RHEL7.5, after building Bro-2.5.5 (I?ve tried other minor
>> versions since 2.5 ? same issue).
>>
>> Existing Bro cluster on RHEL7.5 boxes with Bro-2.5 and Broctl 1.5
>> works fine.
>>
>> Any help would be greatly appreciated.
>>
>
> check runs bro with the current configuration to see if it can start,
> so that's bro segfaulting there.. that's why start also fails..
>
> What do you get if you try each of the following?
>
>     bro -v
>     bro -NN # just see if this runs or crashes
>     bro -b -i lo
>     bro -i lo
>     bro -i lo local
>
> You can hit control-c if any of those start successfully to get your
> prompt back.
>
> I'm not aware of any issues like this, so it could be something with
> your configuration.
>
> Do you have a customized local.bro at all?
> Are you building bro against a particular libpcap or malloc
> implementation?
> What does ldd /opt/bro/bin/bro output?
>
> ?
> Justin Azoff
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org<mailto:bro at bro-ids.org>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
--

Jon
--

Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181005/10e9bc76/attachment-0001.html 

From zeolla at gmail.com  Fri Oct  5 11:58:30 2018
From: zeolla at gmail.com (Zeolla@GMail.com)
Date: Fri, 5 Oct 2018 14:58:30 -0400
Subject: [Bro] Broctl segmentation fault
In-Reply-To: <CD76AAEFD1531643AB6A33368B516D770B601C21@marathon>
References: <CD76AAEFD1531643AB6A33368B516D770B600D85@marathon>
	<83656C44-986A-49F5-BC68-0DABBC006A29@illinois.edu>
	<CD76AAEFD1531643AB6A33368B516D770B601132@marathon>
	<1A3F6849-ED1D-4363-9B54-B7F5D3294275@icir.org>
	<CD76AAEFD1531643AB6A33368B516D770B601387@marathon>
	<CAM1mTCr4fbm=890zN+ngfO0nbb9r4Ee6X5GC7bkP4MN-LtA7Vg@mail.gmail.com>
	<CD76AAEFD1531643AB6A33368B516D770B601A4D@marathon>
	<CAM1mTCqyn94BAE+pgAsUX7jvZpXZXd0dsA=tEbcxutxOCq4EPQ@mail.gmail.com>
	<CD76AAEFD1531643AB6A33368B516D770B601C21@marathon>
Message-ID: <CAM1mTCqA2mGWO5jXR9PzrG+-erM1stMRazax=dj2hUYZv8MmUw@mail.gmail.com>

Gotcha, yeah please do as that is the most updated code and includes some
baseline btests (with more coming in the near future). Feel free to reach
out to me via the list if you have any questions/concerns/issues on it -
thanks!

Jon

On Fri, Oct 5, 2018 at 1:19 PM Sean Hutchison <shutchison at cert.org> wrote:

> Well, we don?t really have a need to use the bro kafka plugin currently ?
> it was just for a bit of testing previously ? but I can use bro-pkg in the
> future or for other plugins/scripts.
>
>
>
> V/R
>
> Sean
>
>
>
> *From:* Zeolla at GMail.com [mailto:zeolla at gmail.com]
> *Sent:* Friday, October 05, 2018 12:36 PM
>
>
> *To:* Sean Hutchison <shutchison at cert.org>
> *Cc:* Johanna Amann <johanna at icir.org>; bro at bro.org
> *Subject:* Re: [Bro] Broctl segmentation fault
>
>
>
> Sounds like you are looking at a very old version of the plugin, since
> bro/bro-plugins has been completely deprecated at this point.  Can you use
> bro-pkg to install apache/metron-bro-plugin-kafka?  It should be a bit more
> robust and up to date.  Let me know if you have any issues when taking this
> approach.
>
>
>
> Jon
>
> On Fri, Oct 5, 2018, 07:55 Sean Hutchison <shutchison at cert.org> wrote:
>
> Ya, first install librdkafka (there?s probably a newer version ? make sure
> it supports your Kafka broker version) ?
>
> curl --silent -L -k
> https://github.com/edenhill/librdkafka/archive/v0.9.5.tar.gz | tar xz
>
> cd librdkafka-0.9.5
>
> ./configure
>
> make
>
> make install
>
>
>
> Then get bro-plugins repo and build kafka plugin against version of Bro
> you?re using by pointing it to where you extracted bro source?
>
> git clone https://github.com/bro/bro-plugins.git
>
> cd bro-plugins/kafka/
>
> ./configure --bro-dist=/path/to/bro-2.#.#
>
> make && make install
>
>
>
> Confirm with?
>
> bro -N Bro::Kafka
>
>
>
> See
> https://archive.apache.org/dist/metron/0.4.0/site-book/metron-sensors/bro-plugin-kafka/index.html
> for example configurations.
>
>
>
> V/R
>
> Sean
>
>
>
> *From:* Zeolla at GMail.com [mailto:zeolla at gmail.com]
> *Sent:* Thursday, October 04, 2018 5:06 PM
> *To:* Sean Hutchison <shutchison at cert.org>
> *Cc:* Johanna Amann <johanna at icir.org>; bro at bro.org
>
>
> *Subject:* Re: [Bro] Broctl segmentation fault
>
>
>
> If you don't mind, can you share the steps you took to build and install
> the plug-in?  What version?
>
>
>
> Jon
>
> On Thu, Oct 4, 2018, 13:23 Sean Hutchison <shutchison at cert.org> wrote:
>
> Yes, and I just removed the Bro Kafka plugin and no more error!
>
> Thank you so much.
>
> V/R
> Sean
>
> -----Original Message-----
> From: Johanna Amann [mailto:johanna at icir.org]
> Sent: Thursday, October 04, 2018 11:36 AM
> To: Sean Hutchison <shutchison at cert.org>
> Cc: Azoff, Justin S <jazoff at illinois.edu>; bro at bro.org
> Subject: Re: [Bro] Broctl segmentation fault
>
> Hi,
>
> Is there a change that you have binary plugins installed (netmap plugin, a
> few bro-pkg ones)?
>
> They can cause crashes exactly like this. This behavior is fixed with Bro
> 2.6 (it will output an error message instead).
>
> If that is the case - either recompiling or removing the binary plugins
> will fix this.
>
> Johanna
>
> On 4 Oct 2018, at 5:01, Sean Hutchison wrote:
>
> > # bro -v
> > bro version 2.5.5
> >
> > # bro -NN
> > Segmentation fault
> >
> > # bro -b -i lo
> > listening on lo
> >
> > ^C1538653437.070325 received termination signal
> > 1538653437.070325 208 packets received on interface lo, 0 dropped
> >
> > # bro -i lo
> > Segmentation fault
> >
> > # bro -i lo local
> > Segmentation fault
> >
> > # ldd /opt/bro/bin/bro
> >         linux-vdso.so.1 =>  (0x00007fff99dfd000)
> >         libpcap.so.1 => /lib64/libpcap.so.1 (0x00007f148eec1000)
> >         libssl.so.10 => /lib64/libssl.so.10 (0x00007f148ec50000)
> >         libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f148e7ef000)
> >         libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f148e5d6000)
> >         libz.so.1 => /lib64/libz.so.1 (0x00007f148e3c0000)
> >         libGeoIP.so.1 => /lib64/libGeoIP.so.1 (0x00007f148e190000)
> >         libtcmalloc.so.4 => /lib64/libtcmalloc.so.4
> > (0x00007f148dd9b000)
> >         libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f148db7f000)
> >         libdl.so.2 => /lib64/libdl.so.2 (0x00007f148d97b000)
> >         libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f148d674000)
> >         libm.so.6 => /lib64/libm.so.6 (0x00007f148d372000)
> >         libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f148d15c000)
> >         libc.so.6 => /lib64/libc.so.6 (0x00007f148cd8f000)
> >         libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2
> > (0x00007f148cb42000)
> >         libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f148c85a000)
> >         libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f148c656000)
> >         libk5crypto.so.3 => /lib64/libk5crypto.so.3
> > (0x00007f148c423000)
> >         /lib64/ld-linux-x86-64.so.2 (0x00007f148f102000)
> >         libkrb5support.so.0 => /lib64/libkrb5support.so.0
> > (0x00007f148c215000)
> >         libkeyutils.so.1 => /lib64/libkeyutils.so.1
> > (0x00007f148c011000)
> >         libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f148bdea000)
> >         libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f148bb88000)
> >
> > No custom scripts being loaded via local.bro Nothing in particular -
> > did yum install/update of RedHat-based dependencies according to
> > https://www.bro.org/sphinx/install/install.html#required-dependencies
> > Although I did build it against pfring first, using yum package from
> > ntop repo - same issue, have since removed that and did regular build
> >
> > Only configure switch was --prefix.
> >
> > V/R
> > Sean
> >
> > -----Original Message-----
> > From: Azoff, Justin S [mailto:jazoff at illinois.edu]
> > Sent: Wednesday, October 03, 2018 3:01 PM
> > To: Sean Hutchison <shutchison at cert.org>
> > Cc: bro at bro.org
> > Subject: Re: [Bro] Broctl segmentation fault
> >
> >
> >> On Oct 3, 2018, at 2:46 PM, Sean Hutchison <shutchison at cert.org>
> >> wrote:
> >>
> >> Hello,
> >>
> >> After any build of Bro with Broctl 1.7, I?m experiencing the below
> >> error when broctl/scripts/check-config is run?
> >>
> >> /opt/bro/share/broctl/scripts/check-config: line 50:  4463
> >> Segmentation fault      "${bro}" $check_option "$@"
> >>
> >> Anyone encountered this before? Cannot bypass doing broctl check ?
> >> broctl start results in failed/crashed processes.
> >>
> >> This is on RHEL7.5, after building Bro-2.5.5 (I?ve tried other minor
> >> versions since 2.5 ? same issue).
> >>
> >> Existing Bro cluster on RHEL7.5 boxes with Bro-2.5 and Broctl 1.5
> >> works fine.
> >>
> >> Any help would be greatly appreciated.
> >>
> >
> > check runs bro with the current configuration to see if it can start,
> > so that's bro segfaulting there.. that's why start also fails..
> >
> > What do you get if you try each of the following?
> >
> >     bro -v
> >     bro -NN # just see if this runs or crashes
> >     bro -b -i lo
> >     bro -i lo
> >     bro -i lo local
> >
> > You can hit control-c if any of those start successfully to get your
> > prompt back.
> >
> > I'm not aware of any issues like this, so it could be something with
> > your configuration.
> >
> > Do you have a customized local.bro at all?
> > Are you building bro against a particular libpcap or malloc
> > implementation?
> > What does ldd /opt/bro/bin/bro output?
> >
> > ?
> > Justin Azoff
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> --
>
> Jon
>
> --
>
> Jon
>
-- 

Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181005/68ddcb2c/attachment.html 

From assaf.morami at gmail.com  Mon Oct  8 08:15:10 2018
From: assaf.morami at gmail.com (Assaf)
Date: Mon, 8 Oct 2018 18:15:10 +0300
Subject: [Bro] Monitor progress and ETA while running bro
Message-ID: <CADsFwicap2p7rV1+DAdRrHLL5hVwZOLzdDQkHcfzAvApp_7M6A@mail.gmail.com>

Hi, I just wanted to share how I monitor progress and ETA while running bro
from a pcap file.

If I have only one pcap I use pipe viewer (the pv command) like this:

pv x.pcap | bro -r -

If I have more than one pcap, e.g. from a big tcpdump run, I merge all of
them on the fly using joincap ( https://github.com/assafmo/joincap ) like
this:

joincap *.pcap | pv -s $(du -bc *.pcap | awk '/total/{print $1}') | bro -r -

This way pv print progress and ETA information while bro is running. :-)

Shameless plug - I wrote joincap specifically for these kind of situations,
because mergecap and tcpslice does not handle errors very well.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181008/b13ae9d7/attachment.html 

From sudesh.barwe at gmail.com  Mon Oct  8 14:24:44 2018
From: sudesh.barwe at gmail.com (Sang Dange)
Date: Mon, 8 Oct 2018 14:24:44 -0700
Subject: [Bro] Bro Digest, Vol 150, Issue 14
In-Reply-To: <mailman.1.1539025201.23889.bro@bro.org>
References: <mailman.1.1539025201.23889.bro@bro.org>
Message-ID: <CAK=qtRqJCgk=eKCH7pFBAtPPUAmhi=b7AeT271Z-CjTbEJ80vA@mail.gmail.com>

Hello all,

Can anyone point me to any documentation on support of Industrial IOT
protocols by Bro?

Thanks,
Sang

On Mon, Oct 8, 2018 at 12:00 PM <bro-request at bro.org> wrote:

> Send Bro mailing list submissions to
>         bro at bro.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> or, via email, send a message with subject or body 'help' to
>         bro-request at bro.org
>
> You can reach the person managing the list at
>         bro-owner at bro.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Bro digest..."
>
>
> Today's Topics:
>
>    1. Monitor progress and ETA while running bro (Assaf)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 8 Oct 2018 18:15:10 +0300
> From: Assaf <assaf.morami at gmail.com>
> Subject: [Bro] Monitor progress and ETA while running bro
> To: "bro at bro.org List" <bro at bro.org>
> Message-ID:
>         <
> CADsFwicap2p7rV1+DAdRrHLL5hVwZOLzdDQkHcfzAvApp_7M6A at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi, I just wanted to share how I monitor progress and ETA while running bro
> from a pcap file.
>
> If I have only one pcap I use pipe viewer (the pv command) like this:
>
> pv x.pcap | bro -r -
>
> If I have more than one pcap, e.g. from a big tcpdump run, I merge all of
> them on the fly using joincap ( https://github.com/assafmo/joincap ) like
> this:
>
> joincap *.pcap | pv -s $(du -bc *.pcap | awk '/total/{print $1}') | bro -r
> -
>
> This way pv print progress and ETA information while bro is running. :-)
>
> Shameless plug - I wrote joincap specifically for these kind of situations,
> because mergecap and tcpslice does not handle errors very well.
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181008/b13ae9d7/attachment-0001.html
>
> ------------------------------
>
> _______________________________________________
> Bro mailing list
> Bro at bro.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> End of Bro Digest, Vol 150, Issue 14
> ************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181008/e305f0d6/attachment.html 

From dnj0496 at gmail.com  Tue Oct  9 22:45:59 2018
From: dnj0496 at gmail.com (Dk Jack)
Date: Tue, 9 Oct 2018 22:45:59 -0700
Subject: [Bro] x509 error
Message-ID: <CAGATen5bCM6WosqM6Jasvg6YKp4xqm3Bru9UDYsL85KT2DUP7w@mail.gmail.com>

Hi,
I am seeing very large number of  'Could not parse X509 certificate' errors
in my weird.log. The ssl cert is sent by the server. Could someone chime in
on the reasons for this error. Thanks.

Dk.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181009/83b47278/attachment.html 

From johanna at icir.org  Wed Oct 10 05:15:23 2018
From: johanna at icir.org (Johanna Amann)
Date: Wed, 10 Oct 2018 08:15:23 -0400
Subject: [Bro] x509 error
In-Reply-To: <CAGATen5bCM6WosqM6Jasvg6YKp4xqm3Bru9UDYsL85KT2DUP7w@mail.gmail.com>
References: <CAGATen5bCM6WosqM6Jasvg6YKp4xqm3Bru9UDYsL85KT2DUP7w@mail.gmail.com>
Message-ID: <3AE6E77F-5A1B-4C39-B408-C130B952609B@icir.org>

Hi,

could you potentially provide a pcap that shows this and send it to me?

Without that it sadly is quite difficult to say what is going on. The 
error comes more or less directly from OpenSSL - so it does not like 
something in the data structure sent in the certificate message.

Johanna

On 10 Oct 2018, at 1:45, Dk Jack wrote:

> Hi,
> I am seeing very large number of  'Could not parse X509 certificate' 
> errors
> in my weird.log. The ssl cert is sent by the server. Could someone 
> chime in
> on the reasons for this error. Thanks.
>
> Dk.
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

From brucekao at heliosdata.com  Thu Oct 11 17:36:41 2018
From: brucekao at heliosdata.com (Bruce Kao)
Date: Fri, 12 Oct 2018 00:36:41 +0000
Subject: [Bro] Long lived MySQL session stops generating events
Message-ID: <MWHPR08MB347055D338D745311110BA29B7E20@MWHPR08MB3470.namprd08.prod.outlook.com>

Hi All,


I am running Bro version 2.5.2, and encountering an issue where an established MySQL session after idling for some time (approximately 10 minutes), the MySQL analyzer events are no longer generated.


The MySQL session itself is still alive and still responding to queries, but the analyzer event mysql_command_request stop generating.  I've also check packet capture and seen no obvious oddities and the packets are the same as when it was working.  I am only issuing a 'Use DB' command for testing.


I suspect there is a timeout somewhere or a limit somewhere that I'm not aware of, as I am still fairly new to Bro itself.


Does anyone have any ideas on how I can continue to debug this issue?


Thanks in advance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181012/4fc80c96/attachment.html 

From jsiwek at corelight.com  Fri Oct 12 15:03:55 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Fri, 12 Oct 2018 17:03:55 -0500
Subject: [Bro] Long lived MySQL session stops generating events
In-Reply-To: <MWHPR08MB347055D338D745311110BA29B7E20@MWHPR08MB3470.namprd08.prod.outlook.com>
References: <MWHPR08MB347055D338D745311110BA29B7E20@MWHPR08MB3470.namprd08.prod.outlook.com>
Message-ID: <CAMzgZ0+ufdb_PPSjBi=tjEUJ1f-bt1ogLT296KS3_ACRtGK4VQ@mail.gmail.com>

On Thu, Oct 11, 2018 at 7:52 PM Bruce Kao <brucekao at heliosdata.com> wrote:

> I am running Bro version 2.5.2, and encountering an issue where an established MySQL session after idling for some time (approximately 10 minutes), the MySQL analyzer events are no longer generated.

I'd first suggest trying with the latest release of Bro (2.5.5) if you
can because the MySQL analyzer had some general changes/fixes in
2.5.4.

Otherwise, it would be most helpful if you could provide a pcap that
reproduces the issue.

- Jon

From fatema.bannatwala at gmail.com  Sat Oct 13 10:23:00 2018
From: fatema.bannatwala at gmail.com (fatema bannatwala)
Date: Sat, 13 Oct 2018 13:23:00 -0400
Subject: [Bro] DNSSEC Support
Message-ID: <CACX0rUQkmfpAFec_C+CWHnbWqeoSDhDVQds6M2m3aO=AXbG=fw@mail.gmail.com>

>Date: Wed, 27 Apr 2016 20:21:39 -0400
>From: Dave Crawford <bro at pingtrip.com>
>Subject: [Bro] DNSSEC Support
>To: bro <bro at bro.org>
>Message-ID: <D82261C5-A89B-4861-A12C-B37D8AED1ED4 at pingtrip.com>
>Content-Type: text/plain; charset=us-ascii
>
>It doesn't appear that there is full support for DNSSEC RR types in the
current release and I'm  >looking for the best option in the meantime.
>
>  For example, answers that include RRSIG's will produce a vector similar
to ["192.168.1.1","  >   <unknown type=46>"] with a corresponding event in
weird.log of "DNS_RR_unknown_type".
>
>  In protocols/dns/consts.bro I see type 46 is included in the query_type
map but based on the  >variable name I assume its not applied to answers?
>
>  -Dave

Hi Dave,

There were some recent commits done to support these DNSSEC RR types
parsing in Bro: RRSIG, DNSKEY, DS, NSEC, NSEC3.

If you want to give it a try, it's available in dev/2.7 branch or a forked
branch from 2.5.4 at following:
https://github.com/fatemabw/bro/tree/master   (bro 2.5.4 with dnssec)
https://github.com/bro/bro/tree/dev/2.7

Apologies for the delay.

Fatema.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181013/516f50aa/attachment.html 

From brucekao at heliosdata.com  Mon Oct 15 10:07:51 2018
From: brucekao at heliosdata.com (Bruce Kao)
Date: Mon, 15 Oct 2018 17:07:51 +0000
Subject: [Bro] Long lived MySQL session stops generating events
In-Reply-To: <CAMzgZ0+ufdb_PPSjBi=tjEUJ1f-bt1ogLT296KS3_ACRtGK4VQ@mail.gmail.com>
References: <MWHPR08MB347055D338D745311110BA29B7E20@MWHPR08MB3470.namprd08.prod.outlook.com>,
	<CAMzgZ0+ufdb_PPSjBi=tjEUJ1f-bt1ogLT296KS3_ACRtGK4VQ@mail.gmail.com>
Message-ID: <MWHPR08MB34702B2C360FE9FAE9935090B7FD0@MWHPR08MB3470.namprd08.prod.outlook.com>

Hi Jon,

I tried the same scenario with Bro 2.5.5 and the same problem with MySQL long session is still reproducible.  I am attaching the pcap here which should show the same issue.


It is a straight forward MySQL session using mysql command.  There is a long period of idling between packet 23 and packet 24, and no mysql events can be generated after packet 24.


The last working command with MySQL event generated is packet 21.


Any suggestions on how to debug this issue would be greatly appreciated.  I am not very familiar with how Bro passes received data into the analyzers for the message parsing, but I would like to find out whether it is that the parsing is failing or if Bro is not picking up the data at all.


Thanks in advance for any guidance anyone can provide.

________________________________
From: Jon Siwek <jsiwek at corelight.com>
Sent: Friday, October 12, 2018 3:03:55 PM
To: Bruce Kao
Cc: bro
Subject: Re: [Bro] Long lived MySQL session stops generating events

On Thu, Oct 11, 2018 at 7:52 PM Bruce Kao <brucekao at heliosdata.com> wrote:

> I am running Bro version 2.5.2, and encountering an issue where an established MySQL session after idling for some time (approximately 10 minutes), the MySQL analyzer events are no longer generated.

I'd first suggest trying with the latest release of Bro (2.5.5) if you
can because the MySQL analyzer had some general changes/fixes in
2.5.4.

Otherwise, it would be most helpful if you could provide a pcap that
reproduces the issue.

- Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181015/8193e7aa/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: out.pcap
Type: application/vnd.tcpdump.pcap
Size: 4046 bytes
Desc: out.pcap
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181015/8193e7aa/attachment.bin 

From jsiwek at corelight.com  Mon Oct 15 12:08:59 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Mon, 15 Oct 2018 14:08:59 -0500
Subject: [Bro] Long lived MySQL session stops generating events
In-Reply-To: <MWHPR08MB34702B2C360FE9FAE9935090B7FD0@MWHPR08MB3470.namprd08.prod.outlook.com>
References: <MWHPR08MB347055D338D745311110BA29B7E20@MWHPR08MB3470.namprd08.prod.outlook.com>
	<CAMzgZ0+ufdb_PPSjBi=tjEUJ1f-bt1ogLT296KS3_ACRtGK4VQ@mail.gmail.com>
	<MWHPR08MB34702B2C360FE9FAE9935090B7FD0@MWHPR08MB3470.namprd08.prod.outlook.com>
Message-ID: <CAMzgZ0J+w+MArXd3doDB=ZmDb+qcSt-RD+TSA25EmYD-QHCQeQ@mail.gmail.com>

You can get a bit more in to the mysql.log if you ran like:

  bro -r out.pcap tcp_inactivity_timeout=10min
"DPD::ignore_violations+={Analyzer::ANALYZER_MYSQL}"

There were a couple potential issues that's working around when I
tested with Bro's git/master branch.

The first was that the responses may be malformed -- Wireshark appears
to report that it's malformed MySQL and Bro also indicates a parsing
problem in dpd.log, showing that the LengthEncodedIntegerLookahead
field was expecting to read 8 bytes, but only 6 were available/sent.
The default behavior is for Bro to disable analyzers that encounter
such protocol violations, but the DPD::ignore_violations customization
prevents that.  Someone may ultimately want to check into whether the
packet is legit following the MySQL protocol, because then the parser
in Bro should be updated to understand it better.

The second issue was related to the long idle period that you
mentioned.  The default timeout for TCP sessions in Bro is 5mins, so
when eventually new MySQL data is seen, it instantiates a new
session/analyzer, however, the MySQL analyzer currently does not
handle "partial connections" and everything from then on is just
skipped.  You can look into tuning the tcp_inactivity_timeout setting
to potentially workaround that, or else it would be better if anyone
wanted to investigate how much it would take to get the MySQL
parser/analyzer able to pick up and analyze sessions already mid-way
in progress.

- Jon

On Mon, Oct 15, 2018 at 12:07 PM Bruce Kao <brucekao at heliosdata.com> wrote:
>
> Hi Jon,
>
> I tried the same scenario with Bro 2.5.5 and the same problem with MySQL long session is still reproducible.  I am attaching the pcap here which should show the same issue.
>
>
> It is a straight forward MySQL session using mysql command.  There is a long period of idling between packet 23 and packet 24, and no mysql events can be generated after packet 24.
>
>
> The last working command with MySQL event generated is packet 21.
>
>
> Any suggestions on how to debug this issue would be greatly appreciated.  I am not very familiar with how Bro passes received data into the analyzers for the message parsing, but I would like to find out whether it is that the parsing is failing or if Bro is not picking up the data at all.
>
>
> Thanks in advance for any guidance anyone can provide.
>
> ________________________________
> From: Jon Siwek <jsiwek at corelight.com>
> Sent: Friday, October 12, 2018 3:03:55 PM
> To: Bruce Kao
> Cc: bro
> Subject: Re: [Bro] Long lived MySQL session stops generating events
>
> On Thu, Oct 11, 2018 at 7:52 PM Bruce Kao <brucekao at heliosdata.com> wrote:
>
> > I am running Bro version 2.5.2, and encountering an issue where an established MySQL session after idling for some time (approximately 10 minutes), the MySQL analyzer events are no longer generated.
>
> I'd first suggest trying with the latest release of Bro (2.5.5) if you
> can because the MySQL analyzer had some general changes/fixes in
> 2.5.4.
>
> Otherwise, it would be most helpful if you could provide a pcap that
> reproduces the issue.
>
> - Jon


From brucekao at heliosdata.com  Mon Oct 15 13:37:34 2018
From: brucekao at heliosdata.com (Bruce Kao)
Date: Mon, 15 Oct 2018 20:37:34 +0000
Subject: [Bro] Long lived MySQL session stops generating events
In-Reply-To: <CAMzgZ0J+w+MArXd3doDB=ZmDb+qcSt-RD+TSA25EmYD-QHCQeQ@mail.gmail.com>
References: <MWHPR08MB347055D338D745311110BA29B7E20@MWHPR08MB3470.namprd08.prod.outlook.com>
	<CAMzgZ0+ufdb_PPSjBi=tjEUJ1f-bt1ogLT296KS3_ACRtGK4VQ@mail.gmail.com>
	<MWHPR08MB34702B2C360FE9FAE9935090B7FD0@MWHPR08MB3470.namprd08.prod.outlook.com>,
	<CAMzgZ0J+w+MArXd3doDB=ZmDb+qcSt-RD+TSA25EmYD-QHCQeQ@mail.gmail.com>
Message-ID: <MWHPR08MB3470E67CDF4EB78B7202EA4EB7FD0@MWHPR08MB3470.namprd08.prod.outlook.com>

Thanks Jon for the tips.  I did see that Wireshark is reporting malformed response data, but since Bro was able to produce events initially I did not dig into it too much.  My MySQL was installed through Ubuntu which is


mysqld  Ver 5.7.23-0ubuntu0.16.04.1 for Linux on x86_64 ((Ubuntu))


I will follow your tips and see if I can find more information and report back.


Thanks

________________________________
From: Jon Siwek <jsiwek at corelight.com>
Sent: Monday, October 15, 2018 12:08:59 PM
To: Bruce Kao
Cc: bro
Subject: Re: [Bro] Long lived MySQL session stops generating events

You can get a bit more in to the mysql.log if you ran like:

  bro -r out.pcap tcp_inactivity_timeout=10min
"DPD::ignore_violations+={Analyzer::ANALYZER_MYSQL}"

There were a couple potential issues that's working around when I
tested with Bro's git/master branch.

The first was that the responses may be malformed -- Wireshark appears
to report that it's malformed MySQL and Bro also indicates a parsing
problem in dpd.log, showing that the LengthEncodedIntegerLookahead
field was expecting to read 8 bytes, but only 6 were available/sent.
The default behavior is for Bro to disable analyzers that encounter
such protocol violations, but the DPD::ignore_violations customization
prevents that.  Someone may ultimately want to check into whether the
packet is legit following the MySQL protocol, because then the parser
in Bro should be updated to understand it better.

The second issue was related to the long idle period that you
mentioned.  The default timeout for TCP sessions in Bro is 5mins, so
when eventually new MySQL data is seen, it instantiates a new
session/analyzer, however, the MySQL analyzer currently does not
handle "partial connections" and everything from then on is just
skipped.  You can look into tuning the tcp_inactivity_timeout setting
to potentially workaround that, or else it would be better if anyone
wanted to investigate how much it would take to get the MySQL
parser/analyzer able to pick up and analyze sessions already mid-way
in progress.

- Jon

On Mon, Oct 15, 2018 at 12:07 PM Bruce Kao <brucekao at heliosdata.com> wrote:
>
> Hi Jon,
>
> I tried the same scenario with Bro 2.5.5 and the same problem with MySQL long session is still reproducible.  I am attaching the pcap here which should show the same issue.
>
>
> It is a straight forward MySQL session using mysql command.  There is a long period of idling between packet 23 and packet 24, and no mysql events can be generated after packet 24.
>
>
> The last working command with MySQL event generated is packet 21.
>
>
> Any suggestions on how to debug this issue would be greatly appreciated.  I am not very familiar with how Bro passes received data into the analyzers for the message parsing, but I would like to find out whether it is that the parsing is failing or if Bro is not picking up the data at all.
>
>
> Thanks in advance for any guidance anyone can provide.
>
> ________________________________
> From: Jon Siwek <jsiwek at corelight.com>
> Sent: Friday, October 12, 2018 3:03:55 PM
> To: Bruce Kao
> Cc: bro
> Subject: Re: [Bro] Long lived MySQL session stops generating events
>
> On Thu, Oct 11, 2018 at 7:52 PM Bruce Kao <brucekao at heliosdata.com> wrote:
>
> > I am running Bro version 2.5.2, and encountering an issue where an established MySQL session after idling for some time (approximately 10 minutes), the MySQL analyzer events are no longer generated.
>
> I'd first suggest trying with the latest release of Bro (2.5.5) if you
> can because the MySQL analyzer had some general changes/fixes in
> 2.5.4.
>
> Otherwise, it would be most helpful if you could provide a pcap that
> reproduces the issue.
>
> - Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181015/b8d21cf5/attachment.html 

From nothinrandom at gmail.com  Mon Oct 15 22:37:35 2018
From: nothinrandom at gmail.com (TQ)
Date: Mon, 15 Oct 2018 22:37:35 -0700
Subject: [Bro] Syntax Question - Nested Switch Case
Message-ID: <CAM2MKSqymHV-JEeE6m829cKj2jwcqPGyucP6yo+x=CbMZjif=w@mail.gmail.com>

Hello Bro Members,

I apologize if this is not the right mailing list... I do not wish to spam
everyone. I just picked up Bro a few days ago, to still learning the ropes.
I have a syntax question that I can't seem to find anywhere. How do you do
a nested switch case inside a record? I have some data
0xAABBCCDD01020304 or 0xAABBCCDD01020405 that I need to verify that the
header is 0xAABBCCDD and switch based on the last two bytes, either 0x0304
or 0x0405. Is this a good practice of switch record since data length will
change based on the command. The nested case I have below is incorrect and
is throwing error "make[3]: *** [test_pac.h] Segmentation fault (core
dumped)"

Currently, I have:

enum cmd_codes {
NOP = 0x00000000,
DEVICE_HEADER = 0x AABBCCDD,
DEVICE_CMD2_1 = 0x0304,
        DEVICE_CMD2_2 = 0x0405
};

type Header = record {
header: uint32; # header
cmd1: uint16; # 0x0102
cmd2: uint16; # 0x0304 or 0x0405
} &byteorder=bigendian;

type Device_Response = record {
header: Device_Header;
data: case(header.header) of {
DEVICE_HEADER -> head: case(header.cmd2) of {
DEVICE_CMD2_1  -> info1: Record_A;

     DEVICE_CMD2_2  -> info2: Record_B;

};
# All the rest
default -> unknown: bytestring &restofdata;
};
} &byteorder=littleendian;

type Record_A = record {
# some data goes here
}

type Record_B = record {
# some data goes here
}

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181015/14e47c29/attachment-0001.html 

From undicizeri at gmail.com  Tue Oct 16 02:55:42 2018
From: undicizeri at gmail.com (Federico Foschini)
Date: Tue, 16 Oct 2018 11:55:42 +0200
Subject: [Bro] /misc/capture_loss percent_lost vs /misc/stats pkts dropped
 and missed bytes in bro_conn
Message-ID: <CAFRU1XuWq7C-fd9Eibgrf7BZMwa6pvz7vEPUveWJTmqPz-3YAA@mail.gmail.com>

Hello,

In one of our bro deployments we are logging some missed byets on bro_conn
logs. This is an example of a conn log with missing bytes:

"local_resp": false,
    "tunnel_parents": [],
    "local_orig": true,
    "dst_addr": "211.115.118.190",
    "src_port": 57786,
    "dst_port": 443,
    "service": "ssl",
    "duration": 0.717725,
    "resp_pkts": 28,
    "src_addr": "10.16.0.115",
    "uid": "C7H1Jb1qJhHVg05wq8",
    "history": "ShADadfF",
    "orig_pkts": 16,
    "host": "logstash",
    "conn_state": "SF",
    "orig_bytes": 2883,
    "path": "/var/log/bro/logs/current/conn.log",
    "@timestamp": "2018-10-16T09:42:14.074Z",
    "times_created": "2018-10-16T09:42:13.357Z",
    "tags": [
      "bro",
      "bro_conn"
    ],
    "proto": "tcp",
    "@version": "1",
    "resp_ip_bytes": 23649,
    "orig_ip_bytes": 3535,
    "missed_bytes": 2920,
    "resp_bytes": 22517,
    "resp_cc": "IT"
  }

I?m running both /policy/misc/capture_loss and /policy/misc/stats scripts
and this is the result:
/misc/stats:

"_source": {
    "files": 40386,
    "mem": 820,
    "active_icmp_conns": 341,
    "dns_requests": 0,
    "active_tcp_conns": 6641,
    "timers": 542182,
    "peer": "worker-1-1",
    "reassem_file_size": 1040104,
    "events_proc": 2285899,
    "active_timers": 33245,
    "host": "logstash",
    "reassem_frag_size": 10528,
    "active_files": 208,
    "icmp_conns": 877,
    "events_queued": 2285898,
    "pkts_dropped": 0,
    "pkts_proc": 10232397,
    "path": "/var/log/bro/logs/current/stats.log",
    "pkts_link": 10232664,
    "udp_conns": 21084,
    "reassem_unknown_size": 0,
    "@timestamp": "2018-10-16T09:15:32.648Z",
    "pkt_lag": 0.007681,
    "active_dns_requests": 0,
    "reassem_tcp_size": 863992,
    "tags": [
      "bro",
      "bro_stats"
    ],
    "active_udp_conns": 2207,
    "tcp_conns": 27070,
    "@version": "1",
    "bytes_recv": 6580937768
  }

/misc/capture_loss:

"_source": {
    "gaps": 92247,
    "peer": "worker-1-1",
    "path": "/var/log/bro/logs/current/capture_loss.log",
    "ts_delta": 900.000031,
    "@timestamp": "2018-10-16T09:15:32.632Z",
    "percent_lost": 2.053046,
    "tags": [
      "bro",
      "bro_stats",
      "bro_capture_loss"
    ],
    "@version": "1",
    "host": "logstash",
    "acks": 4493178
  }

By reading the documentation It looks like the switch SPAN port or the
network interface is dropping packets since bro stats doesn?t register any
packet drops.
I?ve checked on the switch and it doesn?t report any dropped traffic.

Is this possible that the network interface of our server is dropping? Is
there a way to analyze the problem further?
-- 
Federico Foschini.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181016/e630cfef/attachment.html 

From jsiwek at corelight.com  Tue Oct 16 09:42:35 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Tue, 16 Oct 2018 11:42:35 -0500
Subject: [Bro] Syntax Question - Nested Switch Case
In-Reply-To: <CAM2MKSqymHV-JEeE6m829cKj2jwcqPGyucP6yo+x=CbMZjif=w@mail.gmail.com>
References: <CAM2MKSqymHV-JEeE6m829cKj2jwcqPGyucP6yo+x=CbMZjif=w@mail.gmail.com>
Message-ID: <CAMzgZ0+eVgODC-LEKY71kF+12VFkyU1W_WiQmxeJzhumgKjXbA@mail.gmail.com>

On Tue, Oct 16, 2018 at 12:46 AM TQ <nothinrandom at gmail.com> wrote:

> How do you do a nested switch case inside a record? I have some data 0xAABBCCDD01020304 or 0xAABBCCDD01020405 that I need to verify that the header is 0xAABBCCDD and switch based on the last two bytes, either 0x0304 or 0x0405. Is this a good practice of switch record since data length will change based on the command. The nested case I have below is incorrect and is throwing error "make[3]: *** [test_pac.h] Segmentation fault (core dumped)"

It's possible that you've just run into a binpac bug related to nested
records and you can file a bug/issue for that on GitHub, but you may
also be able to organize things differently.  For example, instead of
nesting switch/case you can do:

type HeaderCmd(cmd: uint16) = case cmd of {
  DEVICE_CMD2_1  -> info1: Record_A;
  DEVICE_CMD2_2  -> info2: Record_B;
};

type Device_Response = record {
  header: Header;
  data: case(header.header) of {
    DEVICE_HEADER -> head_cmd: HeaderCmd(header.cmd2);
    default       -> unknown: bytestring &restofdata;
  };
} &byteorder=littleendian;

Also, if you only ever expect to see that DEVICE_HEADER value and not
any other header values for this protocol, you might just use an
&enforce attribute at the top level instead of a switch/case.

- Jon


From michalpurzynski1 at gmail.com  Tue Oct 16 10:29:35 2018
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Tue, 16 Oct 2018 10:29:35 -0700
Subject: [Bro] /misc/capture_loss percent_lost vs /misc/stats pkts
	dropped and missed bytes in bro_conn
In-Reply-To: <CAFRU1XuWq7C-fd9Eibgrf7BZMwa6pvz7vEPUveWJTmqPz-3YAA@mail.gmail.com>
References: <CAFRU1XuWq7C-fd9Eibgrf7BZMwa6pvz7vEPUveWJTmqPz-3YAA@mail.gmail.com>
Message-ID: <9D40876E-E32A-4B02-98EB-5B8CA3D7BF32@gmail.com>

Tell us what kind of capture method you use and we will take it from here.


> On Oct 16, 2018, at 2:55 AM, Federico Foschini <undicizeri at gmail.com> wrote:
> 
> Hello,
> 
> In one of our bro deployments we are logging some missed byets on bro_conn logs. This is an example of a conn log with missing bytes:
> 
> "local_resp": false,
>     "tunnel_parents": [],
>     "local_orig": true,
>     "dst_addr": "211.115.118.190",
>     "src_port": 57786,
>     "dst_port": 443,
>     "service": "ssl",
>     "duration": 0.717725,
>     "resp_pkts": 28,
>     "src_addr": "10.16.0.115",
>     "uid": "C7H1Jb1qJhHVg05wq8",
>     "history": "ShADadfF",
>     "orig_pkts": 16,
>     "host": "logstash",
>     "conn_state": "SF",
>     "orig_bytes": 2883,
>     "path": "/var/log/bro/logs/current/conn.log",
>     "@timestamp": "2018-10-16T09:42:14.074Z",
>     "times_created": "2018-10-16T09:42:13.357Z",
>     "tags": [
>       "bro",
>       "bro_conn"
>     ],
>     "proto": "tcp",
>     "@version": "1",
>     "resp_ip_bytes": 23649,
>     "orig_ip_bytes": 3535,
>     "missed_bytes": 2920,
>     "resp_bytes": 22517,
>     "resp_cc": "IT"
>   }
> I?m running both /policy/misc/capture_loss and /policy/misc/stats scripts and this is the result:
> /misc/stats:
> 
> "_source": {
>     "files": 40386,
>     "mem": 820,
>     "active_icmp_conns": 341,
>     "dns_requests": 0,
>     "active_tcp_conns": 6641,
>     "timers": 542182,
>     "peer": "worker-1-1",
>     "reassem_file_size": 1040104,
>     "events_proc": 2285899,
>     "active_timers": 33245,
>     "host": "logstash",
>     "reassem_frag_size": 10528,
>     "active_files": 208,
>     "icmp_conns": 877,
>     "events_queued": 2285898,
>     "pkts_dropped": 0,
>     "pkts_proc": 10232397,
>     "path": "/var/log/bro/logs/current/stats.log",
>     "pkts_link": 10232664,
>     "udp_conns": 21084,
>     "reassem_unknown_size": 0,
>     "@timestamp": "2018-10-16T09:15:32.648Z",
>     "pkt_lag": 0.007681,
>     "active_dns_requests": 0,
>     "reassem_tcp_size": 863992,
>     "tags": [
>       "bro",
>       "bro_stats"
>     ],
>     "active_udp_conns": 2207,
>     "tcp_conns": 27070,
>     "@version": "1",
>     "bytes_recv": 6580937768
>   }
> /misc/capture_loss:
> 
> "_source": {
>     "gaps": 92247,
>     "peer": "worker-1-1",
>     "path": "/var/log/bro/logs/current/capture_loss.log",
>     "ts_delta": 900.000031,
>     "@timestamp": "2018-10-16T09:15:32.632Z",
>     "percent_lost": 2.053046,
>     "tags": [
>       "bro",
>       "bro_stats",
>       "bro_capture_loss"
>     ],
>     "@version": "1",
>     "host": "logstash",
>     "acks": 4493178
>   }
> By reading the documentation It looks like the switch SPAN port or the network interface is dropping packets since bro stats doesn?t register any packet drops.
> I?ve checked on the switch and it doesn?t report any dropped traffic.
> 
> Is this possible that the network interface of our server is dropping? Is there a way to analyze the problem further?
> 
> -- 
> Federico Foschini.
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181016/643808c5/attachment-0001.html 

From openshift.ninja at gmail.com  Tue Oct 16 13:11:32 2018
From: openshift.ninja at gmail.com (Jeffrey Poore)
Date: Tue, 16 Oct 2018 20:11:32 +0000
Subject: [Bro] getting bro 2.5.5 source
Message-ID: <emfb7a3de2-7d91-457e-9ebc-b75b2a2eeeb6@starkiller>

Hello community. I have been building Bro from source, but I realized 
that I don't want to be downloading the tip source but rather the 2.5.5 
source. I tried grabbing the source from GitHub at 
https://github.com/bro/bro/archive/v2.5.5.tar.gz, but when I do a 
configure inside this directory, it fails because the cmake directory 
inside is empty.

I was doing a recursive clone of the git repo:

$ git clone --recursive https://github.com/bro/bro.git

I noticed that there is a 2.5 release branch, so I checkout that:

$ git checkout release/2.5

But then when I try to run things, it breaks when I run check on the 
scripts (it's like the Bro interpreter is 2.5 but the scripts are 2.6?). 
I even tried doing this, without the recursive clone, but it still 
didn't work:

$ git clone https://github.com/bro/bro.git
$ git checkout release/2.5
$ git submodule init
$ git submodule update
$ ./configure <fails>

Unfortunately, our proxy doesn't let us get to bro.org currently, so I'm 
trying to make this work from the github repo. Any ideas?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181016/81907afa/attachment.html 

From jazoff at illinois.edu  Tue Oct 16 13:23:47 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Tue, 16 Oct 2018 20:23:47 +0000
Subject: [Bro] getting bro 2.5.5 source
In-Reply-To: <emfb7a3de2-7d91-457e-9ebc-b75b2a2eeeb6@starkiller>
References: <emfb7a3de2-7d91-457e-9ebc-b75b2a2eeeb6@starkiller>
Message-ID: <04692792-F03A-4621-8483-572333E2F1E9@illinois.edu>

> On Oct 16, 2018, at 4:11 PM, Jeffrey Poore <openshift.ninja at gmail.com> wrote:
> 
> Hello community. I have been building Bro from source, but I realized that I don't want to be downloading the tip source but rather the 2.5.5 source. I tried grabbing the source from GitHub at https://github.com/bro/bro/archive/v2.5.5.tar.gz, but when I do a configure inside this directory, it fails because the cmake directory inside is empty. 

The file you are probably looking for is https://www.bro.org/downloads/bro-2.5.5.tar.gz

The auto archives from github unfortunately don't include the submodules.

> I was doing a recursive clone of the git repo:
> 
> $ git clone --recursive https://github.com/bro/bro.git
> 
> I noticed that there is a 2.5 release branch, so I checkout that:
> 
> $ git checkout release/2.5
> 
> But then when I try to run things, it breaks when I run check on the scripts (it's like the Bro interpreter is 2.5 but the scripts are 2.6?). I even tried doing this, without the recursive clone, but it still didn't work:
> 
> $ git clone https://github.com/bro/bro.git
> $ git checkout release/2.5
> $ git submodule init
> $ git submodule update
> $ ./configure <fails>
> 
> Unfortunately, our proxy doesn't let us get to bro.org currently, so I'm trying to make this work from the github repo. Any ideas?

I worked this out a while ago,  I think your problem is that updating the submodules only updates the top level, but bro has submodules inside of submodules and you need to do it recursively.

These are the commands I used to use when building an arbitrary bro commit:

    git checkout -f $VERSION
    git submodule foreach --recursive git submodule init
    git submodule update --recursive -f || git submodule update --recursive

Also, I believe that if you did

    git clone --branch v2.5.5 --recursive https://github.com/bro/bro.git

you would end up with the correct checkout as well.

? 
Justin Azoff





From openshift.ninja at gmail.com  Tue Oct 16 13:37:02 2018
From: openshift.ninja at gmail.com (OpenShift Ninja)
Date: Tue, 16 Oct 2018 16:37:02 -0400
Subject: [Bro] getting bro 2.5.5 source
In-Reply-To: <04692792-F03A-4621-8483-572333E2F1E9@illinois.edu>
References: <emfb7a3de2-7d91-457e-9ebc-b75b2a2eeeb6@starkiller>
	<04692792-F03A-4621-8483-572333E2F1E9@illinois.edu>
Message-ID: <CAJ8Dx3T2LnbQhMEHOJARy8a2oSk96xcG6kHMCgkfJo8XB1q=8A@mail.gmail.com>

thank you. that helps a lot! :)
On Tue, Oct 16, 2018 at 4:24 PM Azoff, Justin S <jazoff at illinois.edu> wrote:

> > On Oct 16, 2018, at 4:11 PM, Jeffrey Poore <openshift.ninja at gmail.com>
> wrote:
> >
> > Hello community. I have been building Bro from source, but I realized
> that I don't want to be downloading the tip source but rather the 2.5.5
> source. I tried grabbing the source from GitHub at
> https://github.com/bro/bro/archive/v2.5.5.tar.gz, but when I do a
> configure inside this directory, it fails because the cmake directory
> inside is empty.
>
> The file you are probably looking for is
> https://www.bro.org/downloads/bro-2.5.5.tar.gz
>
> The auto archives from github unfortunately don't include the submodules.
>
> > I was doing a recursive clone of the git repo:
> >
> > $ git clone --recursive https://github.com/bro/bro.git
> >
> > I noticed that there is a 2.5 release branch, so I checkout that:
> >
> > $ git checkout release/2.5
> >
> > But then when I try to run things, it breaks when I run check on the
> scripts (it's like the Bro interpreter is 2.5 but the scripts are 2.6?). I
> even tried doing this, without the recursive clone, but it still didn't
> work:
> >
> > $ git clone https://github.com/bro/bro.git
> > $ git checkout release/2.5
> > $ git submodule init
> > $ git submodule update
> > $ ./configure <fails>
> >
> > Unfortunately, our proxy doesn't let us get to bro.org currently, so
> I'm trying to make this work from the github repo. Any ideas?
>
> I worked this out a while ago,  I think your problem is that updating the
> submodules only updates the top level, but bro has submodules inside of
> submodules and you need to do it recursively.
>
> These are the commands I used to use when building an arbitrary bro commit:
>
>     git checkout -f $VERSION
>     git submodule foreach --recursive git submodule init
>     git submodule update --recursive -f || git submodule update --recursive
>
> Also, I believe that if you did
>
>     git clone --branch v2.5.5 --recursive https://github.com/bro/bro.git
>
> you would end up with the correct checkout as well.
>
> ?
> Justin Azoff
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181016/6d4d6c3f/attachment.html 

From undicizeri at gmail.com  Tue Oct 16 23:49:41 2018
From: undicizeri at gmail.com (Federico Foschini)
Date: Wed, 17 Oct 2018 08:49:41 +0200
Subject: [Bro] /misc/capture_loss percent_lost vs /misc/stats pkts
 dropped and missed bytes in bro_conn
In-Reply-To: <9D40876E-E32A-4B02-98EB-5B8CA3D7BF32@gmail.com>
References: <CAFRU1XuWq7C-fd9Eibgrf7BZMwa6pvz7vEPUveWJTmqPz-3YAA@mail.gmail.com>
	<9D40876E-E32A-4B02-98EB-5B8CA3D7BF32@gmail.com>
Message-ID: <CAFRU1XvJc2N9GPUa-Z8EC=9b-=0Zt4hDboSq4ymGLqmoAJ7f7Q@mail.gmail.com>

Hi, I?m using af_packet. This is my broctl.cfg file:

LogRotationInterval = 3600
LogExpireInterval = 5day
StatsLogEnable = 1
StatsLogExpireInterval = 14
StatusCmdShowAll = 0
CrashExpireInterval = 0
SitePolicyScripts = local.bro
LogDir = /var/log/bro/logs
SpoolDir = /var/log/bro/spool
CfgDir = /opt/bro/etc
lb_custom.InterfacePrefix=af_packet::

And this is my node.cfg file:

[manager]
type=manager
host=localhost

[proxy-1]
type=proxy
host=localhost

[worker-1]
type=worker
host=localhost
interface=enp2s0f1
lb_method=custom
lb_procs=2
af_packet_fanout_id=21
af_packet_fanout_mode=AF_Packet::FANOUT_HASH
af_packet_buffer_size=128*1024*1024

I hope this helps. Thanks for your help!

Il giorno mar 16 ott 2018 alle ore 19:29 Micha? Purzy?ski <
michalpurzynski1 at gmail.com> ha scritto:

> Tell us what kind of capture method you use and we will take it from here.
>
>
> On Oct 16, 2018, at 2:55 AM, Federico Foschini <undicizeri at gmail.com>
> wrote:
>
> Hello,
>
> In one of our bro deployments we are logging some missed byets on
> bro_conn logs. This is an example of a conn log with missing bytes:
>
> "local_resp": false,
>     "tunnel_parents": [],
>     "local_orig": true,
>     "dst_addr": "211.115.118.190",
>     "src_port": 57786,
>     "dst_port": 443,
>     "service": "ssl",
>     "duration": 0.717725,
>     "resp_pkts": 28,
>     "src_addr": "10.16.0.115",
>     "uid": "C7H1Jb1qJhHVg05wq8",
>     "history": "ShADadfF",
>     "orig_pkts": 16,
>     "host": "logstash",
>     "conn_state": "SF",
>     "orig_bytes": 2883,
>     "path": "/var/log/bro/logs/current/conn.log",
>     "@timestamp": "2018-10-16T09:42:14.074Z",
>     "times_created": "2018-10-16T09:42:13.357Z",
>     "tags": [
>       "bro",
>       "bro_conn"
>     ],
>     "proto": "tcp",
>     "@version": "1",
>     "resp_ip_bytes": 23649,
>     "orig_ip_bytes": 3535,
>     "missed_bytes": 2920,
>     "resp_bytes": 22517,
>     "resp_cc": "IT"
>   }
>
> I?m running both /policy/misc/capture_loss and /policy/misc/stats scripts
> and this is the result:
> /misc/stats:
>
> "_source": {
>     "files": 40386,
>     "mem": 820,
>     "active_icmp_conns": 341,
>     "dns_requests": 0,
>     "active_tcp_conns": 6641,
>     "timers": 542182,
>     "peer": "worker-1-1",
>     "reassem_file_size": 1040104,
>     "events_proc": 2285899,
>     "active_timers": 33245,
>     "host": "logstash",
>     "reassem_frag_size": 10528,
>     "active_files": 208,
>     "icmp_conns": 877,
>     "events_queued": 2285898,
>     "pkts_dropped": 0,
>     "pkts_proc": 10232397,
>     "path": "/var/log/bro/logs/current/stats.log",
>     "pkts_link": 10232664,
>     "udp_conns": 21084,
>     "reassem_unknown_size": 0,
>     "@timestamp": "2018-10-16T09:15:32.648Z",
>     "pkt_lag": 0.007681,
>     "active_dns_requests": 0,
>     "reassem_tcp_size": 863992,
>     "tags": [
>       "bro",
>       "bro_stats"
>     ],
>     "active_udp_conns": 2207,
>     "tcp_conns": 27070,
>     "@version": "1",
>     "bytes_recv": 6580937768
>   }
>
> /misc/capture_loss:
>
> "_source": {
>     "gaps": 92247,
>     "peer": "worker-1-1",
>     "path": "/var/log/bro/logs/current/capture_loss.log",
>     "ts_delta": 900.000031,
>     "@timestamp": "2018-10-16T09:15:32.632Z",
>     "percent_lost": 2.053046,
>     "tags": [
>       "bro",
>       "bro_stats",
>       "bro_capture_loss"
>     ],
>     "@version": "1",
>     "host": "logstash",
>     "acks": 4493178
>   }
>
> By reading the documentation It looks like the switch SPAN port or the
> network interface is dropping packets since bro stats doesn?t register any
> packet drops.
> I?ve checked on the switch and it doesn?t report any dropped traffic.
>
> Is this possible that the network interface of our server is dropping? Is
> there a way to analyze the problem further?
> --
> Federico Foschini.
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>

-- 
Federico Foschini.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181017/831c3d35/attachment-0001.html 

From michalpurzynski1 at gmail.com  Wed Oct 17 01:40:57 2018
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Wed, 17 Oct 2018 01:40:57 -0700
Subject: [Bro] /misc/capture_loss percent_lost vs /misc/stats pkts
	dropped and missed bytes in bro_conn
In-Reply-To: <CAFRU1XvJc2N9GPUa-Z8EC=9b-=0Zt4hDboSq4ymGLqmoAJ7f7Q@mail.gmail.com>
References: <CAFRU1XuWq7C-fd9Eibgrf7BZMwa6pvz7vEPUveWJTmqPz-3YAA@mail.gmail.com>
	<9D40876E-E32A-4B02-98EB-5B8CA3D7BF32@gmail.com>
	<CAFRU1XvJc2N9GPUa-Z8EC=9b-=0Zt4hDboSq4ymGLqmoAJ7f7Q@mail.gmail.com>
Message-ID: <9DB248A2-75C3-4E98-A3A4-43754831619F@gmail.com>

Excellent choice!

We created (with Peter Manev, a Suricata developer) a tuning guide that also applies to Bro.

Pay special attention to these two sections
- life of a packet
- packet drops (near the end)

There are a few places packets can be dropped, so it?s important to know all of them.

https://github.com/pevma/SEPTun


> On Oct 16, 2018, at 11:49 PM, Federico Foschini <undicizeri at gmail.com> wrote:
> 
> Hi, I?m using af_packet. This is my broctl.cfg file:
> 
> LogRotationInterval = 3600
> LogExpireInterval = 5day
> StatsLogEnable = 1
> StatsLogExpireInterval = 14
> StatusCmdShowAll = 0
> CrashExpireInterval = 0
> SitePolicyScripts = local.bro
> LogDir = /var/log/bro/logs
> SpoolDir = /var/log/bro/spool
> CfgDir = /opt/bro/etc
> lb_custom.InterfacePrefix=af_packet::
> And this is my node.cfg file:
> 
> [manager]
> type=manager
> host=localhost
> 
> [proxy-1]
> type=proxy
> host=localhost
> 
> [worker-1]
> type=worker
> host=localhost
> interface=enp2s0f1
> lb_method=custom
> lb_procs=2
> af_packet_fanout_id=21
> af_packet_fanout_mode=AF_Packet::FANOUT_HASH
> af_packet_buffer_size=128*1024*1024
> I hope this helps. Thanks for your help!
> 
> 
>> Il giorno mar 16 ott 2018 alle ore 19:29 Micha? Purzy?ski <michalpurzynski1 at gmail.com> ha scritto:
>> Tell us what kind of capture method you use and we will take it from here.
>> 
>> 
>>> On Oct 16, 2018, at 2:55 AM, Federico Foschini <undicizeri at gmail.com> wrote:
>>> 
>>> Hello,
>>> 
>>> In one of our bro deployments we are logging some missed byets on bro_conn logs. This is an example of a conn log with missing bytes:
>>> 
>>> "local_resp": false,
>>>     "tunnel_parents": [],
>>>     "local_orig": true,
>>>     "dst_addr": "211.115.118.190",
>>>     "src_port": 57786,
>>>     "dst_port": 443,
>>>     "service": "ssl",
>>>     "duration": 0.717725,
>>>     "resp_pkts": 28,
>>>     "src_addr": "10.16.0.115",
>>>     "uid": "C7H1Jb1qJhHVg05wq8",
>>>     "history": "ShADadfF",
>>>     "orig_pkts": 16,
>>>     "host": "logstash",
>>>     "conn_state": "SF",
>>>     "orig_bytes": 2883,
>>>     "path": "/var/log/bro/logs/current/conn.log",
>>>     "@timestamp": "2018-10-16T09:42:14.074Z",
>>>     "times_created": "2018-10-16T09:42:13.357Z",
>>>     "tags": [
>>>       "bro",
>>>       "bro_conn"
>>>     ],
>>>     "proto": "tcp",
>>>     "@version": "1",
>>>     "resp_ip_bytes": 23649,
>>>     "orig_ip_bytes": 3535,
>>>     "missed_bytes": 2920,
>>>     "resp_bytes": 22517,
>>>     "resp_cc": "IT"
>>>   }
>>> I?m running both /policy/misc/capture_loss and /policy/misc/stats scripts and this is the result:
>>> /misc/stats:
>>> 
>>> "_source": {
>>>     "files": 40386,
>>>     "mem": 820,
>>>     "active_icmp_conns": 341,
>>>     "dns_requests": 0,
>>>     "active_tcp_conns": 6641,
>>>     "timers": 542182,
>>>     "peer": "worker-1-1",
>>>     "reassem_file_size": 1040104,
>>>     "events_proc": 2285899,
>>>     "active_timers": 33245,
>>>     "host": "logstash",
>>>     "reassem_frag_size": 10528,
>>>     "active_files": 208,
>>>     "icmp_conns": 877,
>>>     "events_queued": 2285898,
>>>     "pkts_dropped": 0,
>>>     "pkts_proc": 10232397,
>>>     "path": "/var/log/bro/logs/current/stats.log",
>>>     "pkts_link": 10232664,
>>>     "udp_conns": 21084,
>>>     "reassem_unknown_size": 0,
>>>     "@timestamp": "2018-10-16T09:15:32.648Z",
>>>     "pkt_lag": 0.007681,
>>>     "active_dns_requests": 0,
>>>     "reassem_tcp_size": 863992,
>>>     "tags": [
>>>       "bro",
>>>       "bro_stats"
>>>     ],
>>>     "active_udp_conns": 2207,
>>>     "tcp_conns": 27070,
>>>     "@version": "1",
>>>     "bytes_recv": 6580937768
>>>   }
>>> /misc/capture_loss:
>>> 
>>> "_source": {
>>>     "gaps": 92247,
>>>     "peer": "worker-1-1",
>>>     "path": "/var/log/bro/logs/current/capture_loss.log",
>>>     "ts_delta": 900.000031,
>>>     "@timestamp": "2018-10-16T09:15:32.632Z",
>>>     "percent_lost": 2.053046,
>>>     "tags": [
>>>       "bro",
>>>       "bro_stats",
>>>       "bro_capture_loss"
>>>     ],
>>>     "@version": "1",
>>>     "host": "logstash",
>>>     "acks": 4493178
>>>   }
>>> By reading the documentation It looks like the switch SPAN port or the network interface is dropping packets since bro stats doesn?t register any packet drops.
>>> I?ve checked on the switch and it doesn?t report any dropped traffic.
>>> 
>>> Is this possible that the network interface of our server is dropping? Is there a way to analyze the problem further?
>>> 
>>> -- 
>>> Federico Foschini.
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 
> -- 
> Federico Foschini.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181017/e042107f/attachment.html 

From seth at corelight.com  Wed Oct 17 08:25:52 2018
From: seth at corelight.com (Seth Hall)
Date: Wed, 17 Oct 2018 11:25:52 -0400
Subject: [Bro] Long lived MySQL session stops generating events
In-Reply-To: <CAMzgZ0J+w+MArXd3doDB=ZmDb+qcSt-RD+TSA25EmYD-QHCQeQ@mail.gmail.com>
References: <MWHPR08MB347055D338D745311110BA29B7E20@MWHPR08MB3470.namprd08.prod.outlook.com>
	<CAMzgZ0+ufdb_PPSjBi=tjEUJ1f-bt1ogLT296KS3_ACRtGK4VQ@mail.gmail.com>
	<MWHPR08MB34702B2C360FE9FAE9935090B7FD0@MWHPR08MB3470.namprd08.prod.outlook.com>
	<CAMzgZ0J+w+MArXd3doDB=ZmDb+qcSt-RD+TSA25EmYD-QHCQeQ@mail.gmail.com>
Message-ID: <42B1526D-34A7-4F26-A2B6-939825DA8B43@corelight.com>



On 15 Oct 2018, at 15:08, Jon Siwek wrote:

> else it would be better if anyone
> wanted to investigate how much it would take to get the MySQL
> parser/analyzer able to pick up and analyze sessions already mid-way
> in progress.

I don't know how difficult this would be, but personally I've been 
putting off too much work into these sorts of efforts with the existing 
binpac analyzers because Spicy should have a mechanism to make analyzers 
"re-synching".  I did it with SMB and it was a little tricky to get 
right.

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com

From vern at icir.org  Wed Oct 17 21:51:43 2018
From: vern at icir.org (Vern Paxson)
Date: Wed, 17 Oct 2018 21:51:43 -0700
Subject: [Bro] renaming the Bro project
Message-ID: <201810180451.w9I4phrm001254@fruitcake.ICSI.Berkeley.EDU>

As many of you no doubt already know, the Leadership Team has chosen a new
name for the project, "Zeek".  To read about the change, see my blog post at

http://blog.bro.org/2018/10/renaming-bro-project_11.html

or, for the media-hungry, the corresponding video from BroCon last week at

https://www.youtube.com/watch?v=L88ZYfjPzyk&feature=youtu.be

We don't yet have a roadmap/timeline for converting the project over to
the new name (though zeek.org is already live, albeit with essentially
unmigrated content).  The 2.6 release hopefully out within a couple of
weeks will still (solely) use the name "Bro".

		Vern

From maanamen at hotmail.com  Thu Oct 18 00:07:38 2018
From: maanamen at hotmail.com (=?iso-8859-1?Q?MA=C1N_ABU_SHAQRA?=)
Date: Thu, 18 Oct 2018 07:07:38 +0000
Subject: [Bro] is there a bro script to ignore duplicated logs?
In-Reply-To: <de5e6fc0-bf16-0dc7-d31f-4bb29c2037b0@gmail.com>
References: <CACX0rUSyEUGQdxY9hKYDWY6zq7J-zUci_ZVFhpmY4cP+Xdj67g@mail.gmail.com>,
	<de5e6fc0-bf16-0dc7-d31f-4bb29c2037b0@gmail.com>
Message-ID: <AM4PR0202MB285065815054CFBAC04F704ACBF80@AM4PR0202MB2850.eurprd02.prod.outlook.com>

Hi All,


@fatema bannatwala<mailto:fatema.bannatwala at gmail.com> im having 2 different interfaces each with 6 workers using af_packet as a load balancer, ive tried the script provided and it showed that both interfaces are streaming same dns/http logs. so i disabled one interface and reduced workers to 3  and kept monitoring one interface only. on conn.log im getting below:


1536746523.249570 CbQGeN3yuYTKQd6xE 10.1.196.178 52851 10.190.129.250 53 udp dns 11.342418 264 88 SF T T 0 Dd 6 432 2 144 (empty) worker-em1-1

on dns.log


1536746526.252543 CbQGeN3yuYTKQd6xE 10.1.196.178 52851 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746526.252634 CbQGeN3yuYTKQd6xE 10.1.196.178 52851 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283534 CbQGeN3yuYTKQd6xE 10.1.196.178 52851 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283625 CbQGeN3yuYTKQd6xE 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746526.252543 CbQGeN3yuYTKQd6xE 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746526.252634 CbQGeN3yuYTKQd6xE 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283534 CbQGeN3yuYTKQd6xE 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283625 CbQGeN3yuYTKQd6xE 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283625 CbQGeN3yuYTKQd6xE 10.1.196.178 52853 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283625 CbQGeN3yuYTKQd6xE 10.1.196.178 52853 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283625 CbQGeN3yuYTKQd6xE 10.1.196.178 52853 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F


ive also checked the http://try.bro.org/<http://try.bro.org/#/?example=basics-switches-solution-switch-exercise> and ran the exercies , and found the same issue.

Try Bro<http://try.bro.org/#/?example=basics-switches-solution-switch-exercise>
try.bro.org
Hello World. Welcome to our interactive Bro tutorial. Click run and see the Bro magic happen. You may need to scroll down a bit to get to the output.

the problem is im using the apapche kafka plugin with apache metron and seeing huge amount of duplicate DNS events and UIDs. i have tried to filter out duplicates in the local.bro using below script.


 Log::add_filter(DNS::LOG, [
        $name = "kafka-dns",
        $writer = Log::WRITER_KAFKAWRITER,
        $pred(rec: DNS::Info) = { return ! (( |rec$uid| == |rec$uid| )); },
        $config = table(
            ["metadata.broker.list"] = "localhost:9092"
        )
    ]);

 but got nothing because all dns entries are duplicated. can anyone help me with the syntax of the above code.


thanks



________________________________
From: bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Jan Grash?fer <jan.grashoefer at gmail.com>
Sent: Thursday, October 4, 2018 4:33 PM
To: bro at bro.org
Subject: Re: [Bro] is there a bro script to ignore duplicated logs?

On 04/10/2018 14:57, fatema bannatwala wrote:
> You can run this script and see if the duplicate connections are happening
> on which workers and go from there:

This can be further automated by using Justins bro-doctor script
available as a package: https://github.com/ncsa/bro-doctor
[https://avatars0.githubusercontent.com/u/7528333?s=400&v=4]<https://github.com/ncsa/bro-doctor>

GitHub - ncsa/bro-doctor<https://github.com/ncsa/bro-doctor>
github.com
Bro Doctor. This plugin provides a "doctor.bro" command for broctl that will help to troubleshoot various common cluster problems. This plugin runs the following checks:




Jan
_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181018/febc9c2e/attachment-0001.html 

From bill.de.ping at gmail.com  Thu Oct 18 02:36:34 2018
From: bill.de.ping at gmail.com (william de ping)
Date: Thu, 18 Oct 2018 12:36:34 +0300
Subject: [Bro]  - time diff between bro logs and host clock
Message-ID: <CAKZA8xhJzUEdc2OC81WAK_=seshqE5k4o7nmW9_DJa=wi4rRmQ@mail.gmail.com>

Hi all,

Any idea why new bro logs epoch timestamp is about 1 hour earlier than the
actual clock of the host ?

I am running a cluster and I was wondering how to fix this issue.

Thank you,
B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181018/f06eefd3/attachment.html 

From jazoff at illinois.edu  Thu Oct 18 06:54:36 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 18 Oct 2018 13:54:36 +0000
Subject: [Bro] is there a bro script to ignore duplicated logs?
In-Reply-To: <AM4PR0202MB285065815054CFBAC04F704ACBF80@AM4PR0202MB2850.eurprd02.prod.outlook.com>
References: <CACX0rUSyEUGQdxY9hKYDWY6zq7J-zUci_ZVFhpmY4cP+Xdj67g@mail.gmail.com>
	<de5e6fc0-bf16-0dc7-d31f-4bb29c2037b0@gmail.com>
	<AM4PR0202MB285065815054CFBAC04F704ACBF80@AM4PR0202MB2850.eurprd02.prod.outlook.com>
Message-ID: <3C502349-3446-4ADA-9556-9CC876CBE0A2@illinois.edu>

Can you post your node.cfg?  It looks like you may have told bro to load balance, but you are not actually using the af_packet plugin.

as in, you have

interface=em1
lb_method=custom
lb_procs=6

instead of

interface=af_packet::em1
lb_method=custom
lb_procs=6

? 
Justin Azoff

> On Oct 18, 2018, at 3:07 AM, MA?N ABU SHAQRA <maanamen at hotmail.com> wrote:
> 
> Hi All,
> 
> @fatema bannatwala im having 2 different interfaces each with 6 workers using af_packet as a load balancer, ive tried the script provided and it showed that both interfaces are streaming same dns/http logs. so i disabled one interface and reduced workers to 3  and kept monitoring one interface only. on conn.log im getting below:
> 
> 1536746523.249570
> CbQGeN3yuYTKQd6xE
> 10.1.196.178 52851
> 10.190.129.250 53
> udp dns
> 11.342418 264
> 88 SF
> T T
> 0 Dd
> 6 432
> 2 144
> (empty) worker-em1-1
> on dns.log
> 
> 1536746526.252543
> CbQGeN3yuYTKQd6xE
> 10.1.196.178
> 52851 10.190.129.250
> 53 udp
> 59985 -
> 100.2.116.192.in-addr.arpa 1
> C_INTERNET 12
> PTR -
> - F
> 1536746526.252634
> CbQGeN3yuYTKQd6xE
> 10.1.196.178
> 52851 10.190.129.250
> 53 udp
> 59985 -
> 100.2.116.192.in-addr.arpa 1
> C_INTERNET 12
> PTR -
> - F
> 1536746530.283534
> CbQGeN3yuYTKQd6xE
> 10.1.196.178
> 52851 10.190.129.250
> 53 udp
> 59985 -
> 100.2.116.192.in-addr.arpa 1
> C_INTERNET 12
> PTR -
> - F
> 1536746530.283625
> CbQGeN3yuYTKQd6xE
> 10.1.196.178
> 52852 10.190.129.250
> 53 udp
> 59985 -
> 100.2.116.192.in-addr.arpa 1
> C_INTERNET 12
> PTR -
> - F
> 1536746526.252543
> CbQGeN3yuYTKQd6xE
> 10.1.196.178 52852
> 10.190.129.250 53
> udp 59985
> - 100.2.116.192.in-addr.arpa
> 1 C_INTERNET
> 12 PTR
> - -
> F
> 1536746526.252634
> CbQGeN3yuYTKQd6xE
> 10.1.196.178 52852
> 10.190.129.250 53
> udp 59985
> - 100.2.116.192.in-addr.arpa
> 1 C_INTERNET
> 12 PTR
> - -
> F
> 1536746530.283534
> CbQGeN3yuYTKQd6xE
> 10.1.196.178 52852
> 10.190.129.250 53
> udp 59985
> - 100.2.116.192.in-addr.arpa
> 1 C_INTERNET
> 12 PTR
> - -
> F
> 1536746530.283625
> CbQGeN3yuYTKQd6xE
> 10.1.196.178 52852
> 10.190.129.250 53
> udp 59985
> - 100.2.116.192.in-addr.arpa
> 1 C_INTERNET
> 12 PTR
> - -
> F
> 1536746530.283625
> CbQGeN3yuYTKQd6xE
> 10.1.196.178
> 52853
> 10.190.129.250
> 53
> udp
> 59985
> -
> 100.2.116.192.in-addr.arpa
> 1
> C_INTERNET
> 12
> PTR
> -
> -
> F
> 1536746530.283625
> CbQGeN3yuYTKQd6xE
> 10.1.196.178
> 52853
> 10.190.129.250
> 53
> udp
> 59985
> -
> 100.2.116.192.in-addr.arpa
> 1
> C_INTERNET
> 12
> PTR
> -
> -
> F
> 1536746530.283625
> CbQGeN3yuYTKQd6xE
> 10.1.196.178
> 52853
> 10.190.129.250
> 53
> udp
> 59985
> -
> 100.2.116.192.in-addr.arpa
> 1
> C_INTERNET
> 12
> PTR
> -
> -
> F
> 
> ive also checked the http://try.bro.org/ and ran the exercies , and found the same issue.  
> Try Bro
> try.bro.org
> Hello World. Welcome to our interactive Bro tutorial. Click run and see the Bro magic happen. You may need to scroll down a bit to get to the output.
> the problem is im using the apapche kafka plugin with apache metron and seeing huge amount of duplicate DNS events and UIDs. i have tried to filter out duplicates in the local.bro using below script.
> 
>  Log::add_filter(DNS::LOG, [
>         $name = "kafka-dns",
>         $writer = Log::WRITER_KAFKAWRITER,
>         $pred(rec: DNS::Info) = { return ! (( |rec$uid| == 
> |rec$uid| )); },
>         $config = table(
>             ["metadata.broker.list"] = "localhost:9092"
>         )
>     ]);
> 
>  but got nothing because all dns entries are duplicated. can anyone help me with the syntax of the above code.
> 
> thanks
> 
> 
> 
> From: bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Jan Grash?fer <jan.grashoefer at gmail.com>
> Sent: Thursday, October 4, 2018 4:33 PM
> To: bro at bro.org
> Subject: Re: [Bro] is there a bro script to ignore duplicated logs?
>  
> On 04/10/2018 14:57, fatema bannatwala wrote:
> > You can run this script and see if the duplicate connections are happening
> > on which workers and go from there:
> 
> This can be further automated by using Justins bro-doctor script 
> available as a package: https://github.com/ncsa/bro-doctor
> 
> GitHub - ncsa/bro-doctor
> github.com
> Bro Doctor. This plugin provides a "doctor.bro" command for broctl that will help to troubleshoot various common cluster problems. This plugin runs the following checks:
> 
> 
> 
> Jan
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From maanamen at hotmail.com  Thu Oct 18 06:58:20 2018
From: maanamen at hotmail.com (=?utf-8?B?TUHDgU4gQUJVIFNIQVFSQQ==?=)
Date: Thu, 18 Oct 2018 13:58:20 +0000
Subject: [Bro] is there a bro script to ignore duplicated logs?
In-Reply-To: <3C502349-3446-4ADA-9556-9CC876CBE0A2@illinois.edu>
References: <CACX0rUSyEUGQdxY9hKYDWY6zq7J-zUci_ZVFhpmY4cP+Xdj67g@mail.gmail.com>
	<de5e6fc0-bf16-0dc7-d31f-4bb29c2037b0@gmail.com>
	<AM4PR0202MB285065815054CFBAC04F704ACBF80@AM4PR0202MB2850.eurprd02.prod.outlook.com>,
	<3C502349-3446-4ADA-9556-9CC876CBE0A2@illinois.edu>
Message-ID: <AM4PR0202MB285045A0AC3CA384DC8B531DCBF80@AM4PR0202MB2850.eurprd02.prod.outlook.com>

Hi Justin,

It is actually set as af_packet::em1

MA?AN ABUSHAQRA 
Dubai, UAE
+971501201752

> On Oct 18, 2018, at 5:54 PM, Azoff, Justin S <jazoff at illinois.edu> wrote:
> 
> Can you post your node.cfg?  It looks like you may have told bro to load balance, but you are not actually using the af_packet plugin.
> 
> as in, you have
> 
> interface=em1
> lb_method=custom
> lb_procs=6
> 
> instead of
> 
> interface=af_packet::em1
> lb_method=custom
> lb_procs=6
> 
> ? 
> Justin Azoff
> 
>> On Oct 18, 2018, at 3:07 AM, MA?N ABU SHAQRA <maanamen at hotmail.com> wrote:
>> 
>> Hi All,
>> 
>> @fatema bannatwala im having 2 different interfaces each with 6 workers using af_packet as a load balancer, ive tried the script provided and it showed that both interfaces are streaming same dns/http logs. so i disabled one interface and reduced workers to 3  and kept monitoring one interface only. on conn.log im getting below:
>> 
>> 1536746523.249570
>> CbQGeN3yuYTKQd6xE
>> 10.1.196.178 52851
>> 10.190.129.250 53
>> udp dns
>> 11.342418 264
>> 88 SF
>> T T
>> 0 Dd
>> 6 432
>> 2 144
>> (empty) worker-em1-1
>> on dns.log
>> 
>> 1536746526.252543
>> CbQGeN3yuYTKQd6xE
>> 10.1.196.178
>> 52851 10.190.129.250
>> 53 udp
>> 59985 -
>> 100.2.116.192.in-addr.arpa 1
>> C_INTERNET 12
>> PTR -
>> - F
>> 1536746526.252634
>> CbQGeN3yuYTKQd6xE
>> 10.1.196.178
>> 52851 10.190.129.250
>> 53 udp
>> 59985 -
>> 100.2.116.192.in-addr.arpa 1
>> C_INTERNET 12
>> PTR -
>> - F
>> 1536746530.283534
>> CbQGeN3yuYTKQd6xE
>> 10.1.196.178
>> 52851 10.190.129.250
>> 53 udp
>> 59985 -
>> 100.2.116.192.in-addr.arpa 1
>> C_INTERNET 12
>> PTR -
>> - F
>> 1536746530.283625
>> CbQGeN3yuYTKQd6xE
>> 10.1.196.178
>> 52852 10.190.129.250
>> 53 udp
>> 59985 -
>> 100.2.116.192.in-addr.arpa 1
>> C_INTERNET 12
>> PTR -
>> - F
>> 1536746526.252543
>> CbQGeN3yuYTKQd6xE
>> 10.1.196.178 52852
>> 10.190.129.250 53
>> udp 59985
>> - 100.2.116.192.in-addr.arpa
>> 1 C_INTERNET
>> 12 PTR
>> - -
>> F
>> 1536746526.252634
>> CbQGeN3yuYTKQd6xE
>> 10.1.196.178 52852
>> 10.190.129.250 53
>> udp 59985
>> - 100.2.116.192.in-addr.arpa
>> 1 C_INTERNET
>> 12 PTR
>> - -
>> F
>> 1536746530.283534
>> CbQGeN3yuYTKQd6xE
>> 10.1.196.178 52852
>> 10.190.129.250 53
>> udp 59985
>> - 100.2.116.192.in-addr.arpa
>> 1 C_INTERNET
>> 12 PTR
>> - -
>> F
>> 1536746530.283625
>> CbQGeN3yuYTKQd6xE
>> 10.1.196.178 52852
>> 10.190.129.250 53
>> udp 59985
>> - 100.2.116.192.in-addr.arpa
>> 1 C_INTERNET
>> 12 PTR
>> - -
>> F
>> 1536746530.283625
>> CbQGeN3yuYTKQd6xE
>> 10.1.196.178
>> 52853
>> 10.190.129.250
>> 53
>> udp
>> 59985
>> -
>> 100.2.116.192.in-addr.arpa
>> 1
>> C_INTERNET
>> 12
>> PTR
>> -
>> -
>> F
>> 1536746530.283625
>> CbQGeN3yuYTKQd6xE
>> 10.1.196.178
>> 52853
>> 10.190.129.250
>> 53
>> udp
>> 59985
>> -
>> 100.2.116.192.in-addr.arpa
>> 1
>> C_INTERNET
>> 12
>> PTR
>> -
>> -
>> F
>> 1536746530.283625
>> CbQGeN3yuYTKQd6xE
>> 10.1.196.178
>> 52853
>> 10.190.129.250
>> 53
>> udp
>> 59985
>> -
>> 100.2.116.192.in-addr.arpa
>> 1
>> C_INTERNET
>> 12
>> PTR
>> -
>> -
>> F
>> 
>> ive also checked the http://try.bro.org/ and ran the exercies , and found the same issue.  
>> Try Bro
>> try.bro.org
>> Hello World. Welcome to our interactive Bro tutorial. Click run and see the Bro magic happen. You may need to scroll down a bit to get to the output.
>> the problem is im using the apapche kafka plugin with apache metron and seeing huge amount of duplicate DNS events and UIDs. i have tried to filter out duplicates in the local.bro using below script.
>> 
>> Log::add_filter(DNS::LOG, [
>>        $name = "kafka-dns",
>>        $writer = Log::WRITER_KAFKAWRITER,
>>        $pred(rec: DNS::Info) = { return ! (( |rec$uid| == 
>> |rec$uid| )); },
>>        $config = table(
>>            ["metadata.broker.list"] = "localhost:9092"
>>        )
>>    ]);
>> 
>> but got nothing because all dns entries are duplicated. can anyone help me with the syntax of the above code.
>> 
>> thanks
>> 
>> 
>> 
>> From: bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Jan Grash?fer <jan.grashoefer at gmail.com>
>> Sent: Thursday, October 4, 2018 4:33 PM
>> To: bro at bro.org
>> Subject: Re: [Bro] is there a bro script to ignore duplicated logs?
>> 
>>> On 04/10/2018 14:57, fatema bannatwala wrote:
>>> You can run this script and see if the duplicate connections are happening
>>> on which workers and go from there:
>> 
>> This can be further automated by using Justins bro-doctor script 
>> available as a package: https://github.com/ncsa/bro-doctor
>> 
>> GitHub - ncsa/bro-doctor
>> github.com
>> Bro Doctor. This plugin provides a "doctor.bro" command for broctl that will help to troubleshoot various common cluster problems. This plugin runs the following checks:
>> 
>> 
>> 
>> Jan
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 


From jazoff at illinois.edu  Thu Oct 18 07:00:24 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 18 Oct 2018 14:00:24 +0000
Subject: [Bro] is there a bro script to ignore duplicated logs?
In-Reply-To: <AM4PR0202MB285045A0AC3CA384DC8B531DCBF80@AM4PR0202MB2850.eurprd02.prod.outlook.com>
References: <CACX0rUSyEUGQdxY9hKYDWY6zq7J-zUci_ZVFhpmY4cP+Xdj67g@mail.gmail.com>
	<de5e6fc0-bf16-0dc7-d31f-4bb29c2037b0@gmail.com>
	<AM4PR0202MB285065815054CFBAC04F704ACBF80@AM4PR0202MB2850.eurprd02.prod.outlook.com>
	<3C502349-3446-4ADA-9556-9CC876CBE0A2@illinois.edu>
	<AM4PR0202MB285045A0AC3CA384DC8B531DCBF80@AM4PR0202MB2850.eurprd02.prod.outlook.com>
Message-ID: <013CBD06-4299-4515-A941-705A8CCAFD6A@illinois.edu>


> On Oct 18, 2018, at 9:58 AM, MA?N ABU SHAQRA <maanamen at hotmail.com> wrote:
> 
> Hi Justin,
> 
> It is actually set as af_packet::em1


Oh :(

What did you mean by 

> ive also checked the http://try.bro.org/ and ran the exercies , and found the same issue.  

Do you mean you uploaded a pcap file and that showed the same duplicated logs?
If so, how did you obtain that pcap?


? 
Justin Azoff



From maanamen at hotmail.com  Thu Oct 18 07:05:31 2018
From: maanamen at hotmail.com (=?utf-8?B?TUHDgU4gQUJVIFNIQVFSQQ==?=)
Date: Thu, 18 Oct 2018 14:05:31 +0000
Subject: [Bro] is there a bro script to ignore duplicated logs?
In-Reply-To: <013CBD06-4299-4515-A941-705A8CCAFD6A@illinois.edu>
References: <CACX0rUSyEUGQdxY9hKYDWY6zq7J-zUci_ZVFhpmY4cP+Xdj67g@mail.gmail.com>
	<de5e6fc0-bf16-0dc7-d31f-4bb29c2037b0@gmail.com>
	<AM4PR0202MB285065815054CFBAC04F704ACBF80@AM4PR0202MB2850.eurprd02.prod.outlook.com>
	<3C502349-3446-4ADA-9556-9CC876CBE0A2@illinois.edu>
	<AM4PR0202MB285045A0AC3CA384DC8B531DCBF80@AM4PR0202MB2850.eurprd02.prod.outlook.com>,
	<013CBD06-4299-4515-A941-705A8CCAFD6A@illinois.edu>
Message-ID: <AM4PR0202MB28502F01150E5C65BEBCDA6FCBF80@AM4PR0202MB2850.eurprd02.prod.outlook.com>

No I didn?t upload a pcap, the provided pcaps on the website show duplicate dns UIDs. I suspect that it?s a duplicated packets issue as I?ve analyzed some traffic on wireshark and it had no duplicates. 

I?d appreciate it if anyone can assist with this,

Thanks

MA?AN ABUSHAQRA 
Dubai, UAE


> On Oct 18, 2018, at 6:00 PM, Azoff, Justin S <jazoff at illinois.edu> wrote:
> 
> 
>> On Oct 18, 2018, at 9:58 AM, MA?N ABU SHAQRA <maanamen at hotmail.com> wrote:
>> 
>> Hi Justin,
>> 
>> It is actually set as af_packet::em1
> 
> 
> Oh :(
> 
> What did you mean by 
> 
>> ive also checked the http://try.bro.org/ and ran the exercies , and found the same issue.  
> 
> Do you mean you uploaded a pcap file and that showed the same duplicated logs?
> If so, how did you obtain that pcap?
> 
> 
> ? 
> Justin Azoff
> 


From jazoff at illinois.edu  Thu Oct 18 07:29:59 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 18 Oct 2018 14:29:59 +0000
Subject: [Bro] is there a bro script to ignore duplicated logs?
In-Reply-To: <AM4PR0202MB28502F01150E5C65BEBCDA6FCBF80@AM4PR0202MB2850.eurprd02.prod.outlook.com>
References: <CACX0rUSyEUGQdxY9hKYDWY6zq7J-zUci_ZVFhpmY4cP+Xdj67g@mail.gmail.com>
	<de5e6fc0-bf16-0dc7-d31f-4bb29c2037b0@gmail.com>
	<AM4PR0202MB285065815054CFBAC04F704ACBF80@AM4PR0202MB2850.eurprd02.prod.outlook.com>
	<3C502349-3446-4ADA-9556-9CC876CBE0A2@illinois.edu>
	<AM4PR0202MB285045A0AC3CA384DC8B531DCBF80@AM4PR0202MB2850.eurprd02.prod.outlook.com>
	<013CBD06-4299-4515-A941-705A8CCAFD6A@illinois.edu>
	<AM4PR0202MB28502F01150E5C65BEBCDA6FCBF80@AM4PR0202MB2850.eurprd02.prod.outlook.com>
Message-ID: <1E3C7662-1EF5-4306-87A7-8041DB7153EC@illinois.edu>


> On Oct 18, 2018, at 10:05 AM, MA?N ABU SHAQRA <maanamen at hotmail.com> wrote:
> 
> No I didn?t upload a pcap, the provided pcaps on the website show duplicate dns UIDs. I suspect that it?s a duplicated packets issue as I?ve analyzed some traffic on wireshark and it had no duplicates. 
> 
> I?d appreciate it if anyone can assist with this,
> 
> Thanks
> 
> MA?AN ABUSHAQRA 
> Dubai, UAE
> 

Oh, yes, I see what you are saying now.

The repeated entries i see on the pcap on try.bro.org are from netbios queries that are all broadcast queries for WORKGROUP.  I believe those are actually just repeated broadcasts with the same 5 tuple which is what causes the duplicates.

At this point with your traffic I would stop bro and run a simple tcpdump to generate a pcap file while you generate some known traffic (like a small file download over HTTP) and then inspect the resulting pcap file.

You could also set lb_procs=1 to ensure that there is only one bro process running to rule out any issue with af_packet load balancing.

One weird thing I see is that for your conn.log entry:

1536746523.249570
CbQGeN3yuYTKQd6xE
10.1.196.178 52851
10.190.129.250 53
udp dns
11.342418 264
88 SF
T T
0 Dd
6 432
2 144
(empty) worker-em1-1

The 6 and 2 are orig_pkts and resp_pkts.  For a simple DNS lookup, you would expect orig_pkts and resp_pkts to both be one.. one packet for the query and then one packet for the response.  But your bro worker somehow saw 6 packets for the query and 2 packets for the response.

? 
Justin Azoff



From fatema.bannatwala at gmail.com  Thu Oct 18 07:41:13 2018
From: fatema.bannatwala at gmail.com (fatema bannatwala)
Date: Thu, 18 Oct 2018 10:41:13 -0400
Subject: [Bro] Bro load with no traffic
Message-ID: <CACX0rUQwWvDzZfFV3pGjPFeAT1j1qF4xcwo=Kh7dRB9YC8kshg@mail.gmail.com>

Does anyone know why Bro would be using resources when no traffic flowing
to the sensor?
Recently we  were having some ECC errors on one of our sensors and turned
off the traffic to that sensor for troubleshooting purposes.
Noticed that the load was pretty high (~7) on that sensor, and was
wondering what Bro must be doing that would cause that load, shouldn't it
be just waiting for the packets without using much cpu/memory resources on
the box?

Stats when no traffic flowing to the sensor, bro processes running because
of cron on manager kicking the bro processes on the workers:

$ top
top - 12:18:17 up 13 days, 19:12,  2 users, * load average: 6.72, 7.05,
7.34*
Tasks: 555 total,   9 running, 546 sleeping,   0 stopped,   0 zombie
%Cpu(s):  9.7 us,  5.7 sy,  0.0 ni, 84.5 id,  0.0 wa,  0.0 hi,  0.1 si,
0.0 st
KiB Mem : 13191564+total, 95957600 free, 32708392 used,  3249652 buff/cache
KiB Swap:  8388600 total,  8388600 free,        0 used. 98285016 avail Mem


When the traffic was turned back on, load average:
$ top
top - 10:39:52 up 1 day, 19:02,  2 users,  load average: 12.89, 12.89, 12.82
Tasks: 551 total,  11 running, 540 sleeping,   0 stopped,   0 zombie
%Cpu(s): 20.9 us,  6.1 sy,  0.1 ni, 72.4 id,  0.0 wa,  0.0 hi,  0.5 si,
0.0 st
KiB Mem : 11540057+total, 59135456 free, 52346920 used,  3918204 buff/cache
KiB Swap:  8388600 total,  8388600 free,        0 used. 62253548 avail Mem

Any thoughts? :)

Thanks,
Fatema
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181018/b291fb2f/attachment.html 

From seth at corelight.com  Thu Oct 18 09:38:16 2018
From: seth at corelight.com (Seth Hall)
Date: Thu, 18 Oct 2018 12:38:16 -0400
Subject: [Bro] - time diff between bro logs and host clock
In-Reply-To: <CAKZA8xhJzUEdc2OC81WAK_=seshqE5k4o7nmW9_DJa=wi4rRmQ@mail.gmail.com>
References: <CAKZA8xhJzUEdc2OC81WAK_=seshqE5k4o7nmW9_DJa=wi4rRmQ@mail.gmail.com>
Message-ID: <6BD28710-BA74-40A4-9F78-853D54EF875A@corelight.com>

On 18 Oct 2018, at 5:36, william de ping wrote:

> Any idea why new bro logs epoch timestamp is about 1 hour earlier than 
> the actual clock of the host ?

Is it possible that you are converting timestamps into something 
readable on a system where the timezone is set differently than you 
expect? One hour off seems suspicious to me as though it might be a 
timezone issue (although the unix epoch timestamp doesn't have a 
timezone built into it so the application of timezone only happens when 
you do the conversion for viewing).

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com

From jsiwek at corelight.com  Thu Oct 18 15:26:40 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 18 Oct 2018 17:26:40 -0500
Subject: [Bro] Bro load with no traffic
In-Reply-To: <CACX0rUQwWvDzZfFV3pGjPFeAT1j1qF4xcwo=Kh7dRB9YC8kshg@mail.gmail.com>
References: <CACX0rUQwWvDzZfFV3pGjPFeAT1j1qF4xcwo=Kh7dRB9YC8kshg@mail.gmail.com>
Message-ID: <CAMzgZ0JEOzaUtRyFs65eXcdSPF-DpBrs_d3zcSqCeLhOuoE4sA@mail.gmail.com>

On Thu, Oct 18, 2018 at 10:01 AM fatema bannatwala
<fatema.bannatwala at gmail.com> wrote:
>
> Does anyone know why Bro would be using resources when no traffic flowing to the sensor?

Currently, Bro's main loop never completely idles in absence of input,
so something on the order of 5% cpu usage in absence of network
traffic might still be "normal".  Also note that that packets aren't
the only input source.  As an example, if you shut off traffic
suddenly, but had a large backlog of Broker messages or continues to
send/recv remote messages, that could be processing resources that Bro
continues to use for some time.  The event engine also continues on
with any scheduled events, etc.

So not particularly unexpected to hear there's some load in absence of
packets, but hard to say specifically what causes the load in this
case -- you may need to profile/trace if you're really interested.

- Jon

From fatema.bannatwala at gmail.com  Fri Oct 19 12:13:09 2018
From: fatema.bannatwala at gmail.com (fatema bannatwala)
Date: Fri, 19 Oct 2018 15:13:09 -0400
Subject: [Bro] Bro load with no traffic
In-Reply-To: <CAMzgZ0JEOzaUtRyFs65eXcdSPF-DpBrs_d3zcSqCeLhOuoE4sA@mail.gmail.com>
References: <CACX0rUQwWvDzZfFV3pGjPFeAT1j1qF4xcwo=Kh7dRB9YC8kshg@mail.gmail.com>
	<CAMzgZ0JEOzaUtRyFs65eXcdSPF-DpBrs_d3zcSqCeLhOuoE4sA@mail.gmail.com>
Message-ID: <CACX0rUTpCeob6WfY2W+LU=yfPa8gVgBuQmCCVNCvBjy_4RedDQ@mail.gmail.com>

Thanks Jon, makes sense now.
I will see if we would want to deep dive into finding out what exactly
causing the load. :)

Fatema.

On Thu, Oct 18, 2018 at 6:26 PM Jon Siwek <jsiwek at corelight.com> wrote:

> On Thu, Oct 18, 2018 at 10:01 AM fatema bannatwala
> <fatema.bannatwala at gmail.com> wrote:
> >
> > Does anyone know why Bro would be using resources when no traffic
> flowing to the sensor?
>
> Currently, Bro's main loop never completely idles in absence of input,
> so something on the order of 5% cpu usage in absence of network
> traffic might still be "normal".  Also note that that packets aren't
> the only input source.  As an example, if you shut off traffic
> suddenly, but had a large backlog of Broker messages or continues to
> send/recv remote messages, that could be processing resources that Bro
> continues to use for some time.  The event engine also continues on
> with any scheduled events, etc.
>
> So not particularly unexpected to hear there's some load in absence of
> packets, but hard to say specifically what causes the load in this
> case -- you may need to profile/trace if you're really interested.
>
> - Jon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181019/97ce1700/attachment.html 

From jazoff at illinois.edu  Fri Oct 19 12:18:41 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Fri, 19 Oct 2018 19:18:41 +0000
Subject: [Bro] Bro load with no traffic
In-Reply-To: <CACX0rUTpCeob6WfY2W+LU=yfPa8gVgBuQmCCVNCvBjy_4RedDQ@mail.gmail.com>
References: <CACX0rUQwWvDzZfFV3pGjPFeAT1j1qF4xcwo=Kh7dRB9YC8kshg@mail.gmail.com>
	<CAMzgZ0JEOzaUtRyFs65eXcdSPF-DpBrs_d3zcSqCeLhOuoE4sA@mail.gmail.com>
	<CACX0rUTpCeob6WfY2W+LU=yfPa8gVgBuQmCCVNCvBjy_4RedDQ@mail.gmail.com>
Message-ID: <D40C0FFF-D476-4086-AA35-5DDB2148ECC1@illinois.edu>

If you are on 2.5.x and not master, this should still work:

http://mailman.icsi.berkeley.edu/pipermail/bro/2016-November/011010.html

I wouldn't bother using it on a production system though.

? 
Justin Azoff

> On Oct 19, 2018, at 3:13 PM, fatema bannatwala <fatema.bannatwala at gmail.com> wrote:
> 
> Thanks Jon, makes sense now.
> I will see if we would want to deep dive into finding out what exactly causing the load. :)
> 
> Fatema.
> 
> On Thu, Oct 18, 2018 at 6:26 PM Jon Siwek <jsiwek at corelight.com> wrote:
> On Thu, Oct 18, 2018 at 10:01 AM fatema bannatwala
> <fatema.bannatwala at gmail.com> wrote:
> >
> > Does anyone know why Bro would be using resources when no traffic flowing to the sensor?
> 
> Currently, Bro's main loop never completely idles in absence of input,
> so something on the order of 5% cpu usage in absence of network
> traffic might still be "normal".  Also note that that packets aren't
> the only input source.  As an example, if you shut off traffic
> suddenly, but had a large backlog of Broker messages or continues to
> send/recv remote messages, that could be processing resources that Bro
> continues to use for some time.  The event engine also continues on
> with any scheduled events, etc.
> 
> So not particularly unexpected to hear there's some load in absence of
> packets, but hard to say specifically what causes the load in this
> case -- you may need to profile/trace if you're really interested.
> 
> - Jon
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From michalpurzynski1 at gmail.com  Fri Oct 19 13:03:05 2018
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Fri, 19 Oct 2018 13:03:05 -0700
Subject: [Bro] Bro load with no traffic
In-Reply-To: <CACX0rUTpCeob6WfY2W+LU=yfPa8gVgBuQmCCVNCvBjy_4RedDQ@mail.gmail.com>
References: <CACX0rUQwWvDzZfFV3pGjPFeAT1j1qF4xcwo=Kh7dRB9YC8kshg@mail.gmail.com>
	<CAMzgZ0JEOzaUtRyFs65eXcdSPF-DpBrs_d3zcSqCeLhOuoE4sA@mail.gmail.com>
	<CACX0rUTpCeob6WfY2W+LU=yfPa8gVgBuQmCCVNCvBjy_4RedDQ@mail.gmail.com>
Message-ID: <C3923F2B-0CC3-4082-A138-FC6ECBD79F0B@gmail.com>

What Jon said.

There was a patch from Justin that lowered the load for embedded systems. It?s not really an issue on most / any production systems I?ve seen. 

> On Oct 19, 2018, at 12:13 PM, fatema bannatwala <fatema.bannatwala at gmail.com> wrote:
> 
> Thanks Jon, makes sense now.
> I will see if we would want to deep dive into finding out what exactly causing the load. :)
> 
> Fatema.
> 
>> On Thu, Oct 18, 2018 at 6:26 PM Jon Siwek <jsiwek at corelight.com> wrote:
>> On Thu, Oct 18, 2018 at 10:01 AM fatema bannatwala
>> <fatema.bannatwala at gmail.com> wrote:
>> >
>> > Does anyone know why Bro would be using resources when no traffic flowing to the sensor?
>> 
>> Currently, Bro's main loop never completely idles in absence of input,
>> so something on the order of 5% cpu usage in absence of network
>> traffic might still be "normal".  Also note that that packets aren't
>> the only input source.  As an example, if you shut off traffic
>> suddenly, but had a large backlog of Broker messages or continues to
>> send/recv remote messages, that could be processing resources that Bro
>> continues to use for some time.  The event engine also continues on
>> with any scheduled events, etc.
>> 
>> So not particularly unexpected to hear there's some load in absence of
>> packets, but hard to say specifically what causes the load in this
>> case -- you may need to profile/trace if you're really interested.
>> 
>> - Jon
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181019/7ec7ded1/attachment.html 

From fatema.bannatwala at gmail.com  Sat Oct 20 08:56:45 2018
From: fatema.bannatwala at gmail.com (fatema bannatwala)
Date: Sat, 20 Oct 2018 11:56:45 -0400
Subject: [Bro] Bro load with no traffic
In-Reply-To: <C3923F2B-0CC3-4082-A138-FC6ECBD79F0B@gmail.com>
References: <CACX0rUQwWvDzZfFV3pGjPFeAT1j1qF4xcwo=Kh7dRB9YC8kshg@mail.gmail.com>
	<CAMzgZ0JEOzaUtRyFs65eXcdSPF-DpBrs_d3zcSqCeLhOuoE4sA@mail.gmail.com>
	<CACX0rUTpCeob6WfY2W+LU=yfPa8gVgBuQmCCVNCvBjy_4RedDQ@mail.gmail.com>
	<C3923F2B-0CC3-4082-A138-FC6ECBD79F0B@gmail.com>
Message-ID: <CACX0rUSNd2p6UiMMz45jSSKRwFz0txE6LpbbwTdK-cHfL96vLw@mail.gmail.com>

Sweet.. the patch looks interesting to deploy. Thanks!
The load is really not an issue in production for us, as the boxes are
pretty beefy with good amount of resources, which never get utilized to
it's full capacity,
but was just curious to know the reason for the high load when the bro
workers are idle (just wanted to make sure it's not some hardware, ex. ECC
errors of memory, that's going bad) :)

Thanks!
Fatema.



On Fri, Oct 19, 2018 at 4:03 PM Micha? Purzy?ski <michalpurzynski1 at gmail.com>
wrote:

> What Jon said.
>
> There was a patch from Justin that lowered the load for embedded systems.
> It?s not really an issue on most / any production systems I?ve seen.
>
> On Oct 19, 2018, at 12:13 PM, fatema bannatwala <
> fatema.bannatwala at gmail.com> wrote:
>
> Thanks Jon, makes sense now.
> I will see if we would want to deep dive into finding out what exactly
> causing the load. :)
>
> Fatema.
>
> On Thu, Oct 18, 2018 at 6:26 PM Jon Siwek <jsiwek at corelight.com> wrote:
>
>> On Thu, Oct 18, 2018 at 10:01 AM fatema bannatwala
>> <fatema.bannatwala at gmail.com> wrote:
>> >
>> > Does anyone know why Bro would be using resources when no traffic
>> flowing to the sensor?
>>
>> Currently, Bro's main loop never completely idles in absence of input,
>> so something on the order of 5% cpu usage in absence of network
>> traffic might still be "normal".  Also note that that packets aren't
>> the only input source.  As an example, if you shut off traffic
>> suddenly, but had a large backlog of Broker messages or continues to
>> send/recv remote messages, that could be processing resources that Bro
>> continues to use for some time.  The event engine also continues on
>> with any scheduled events, etc.
>>
>> So not particularly unexpected to hear there's some load in absence of
>> packets, but hard to say specifically what causes the load in this
>> case -- you may need to profile/trace if you're really interested.
>>
>> - Jon
>>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181020/6f421ace/attachment-0001.html 

From holgrain at protonmail.com  Mon Oct 22 08:09:24 2018
From: holgrain at protonmail.com (Elena Bykovchenko)
Date: Mon, 22 Oct 2018 15:09:24 +0000
Subject: [Bro] Custom event handler script generates heavy CPU load with Bro
	2.5.5 (PF_RING)
Message-ID: <z9FOcdkccodOrDRBCDE2FswhdupRvYaDMmcmBaIIVJQuZulEdNKJAxtUcB4ODmCgs5AKytWjaKgQulcULlbc92L5_ISJcqJbW2p60Gz8VOs=@protonmail.com>

Hello. I have a script which defines a custom handler on mime_data event:

event mime_all_data (c: connection, length: count, data: string)
{
 // do stuff
}

When this script is ran with capturing traffic in PF_RING mode using lb_procs=2, Bro processes consume 100% of both pinned CPU cores. This is not the case when capturing without PF_RING in single process mode though. What are possible reasons for this? Can it be optimized on the script side? What can be done to lower the CPU usage?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181022/9b05d75b/attachment.html 

From seth at corelight.com  Wed Oct 24 07:38:25 2018
From: seth at corelight.com (Seth Hall)
Date: Wed, 24 Oct 2018 10:38:25 -0400
Subject: [Bro] Custom event handler script generates heavy CPU load with
 Bro 2.5.5 (PF_RING)
In-Reply-To: <z9FOcdkccodOrDRBCDE2FswhdupRvYaDMmcmBaIIVJQuZulEdNKJAxtUcB4ODmCgs5AKytWjaKgQulcULlbc92L5_ISJcqJbW2p60Gz8VOs=@protonmail.com>
References: <z9FOcdkccodOrDRBCDE2FswhdupRvYaDMmcmBaIIVJQuZulEdNKJAxtUcB4ODmCgs5AKytWjaKgQulcULlbc92L5_ISJcqJbW2p60Gz8VOs=@protonmail.com>
Message-ID: <AAC8ECFE-D696-44F0-BD1B-D0371FC2E182@corelight.com>

Hi Elena,

I think you're conflating two potential problems.

  - Is handling the mime_all_data event causing overload?
  - Is PF_Ring working correctly?

If you take your script out of the picture are you still seeing 100% cpu 
utilization?

   .Seth

On 22 Oct 2018, at 11:09, Elena Bykovchenko wrote:

> Hello. I have a script which defines a custom handler on mime_data 
> event:
>
> event mime_all_data (c: connection, length: count, data: string)
> {
>  // do stuff
> }
>
> When this script is ran with capturing traffic in PF_RING mode using 
> lb_procs=2, Bro processes consume 100% of both pinned CPU cores. This 
> is not the case when capturing without PF_RING in single process mode 
> though. What are possible reasons for this? Can it be optimized on the 
> script side? What can be done to lower the CPU 
> usage?_______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

--
Seth Hall * Corelight, Inc * www.corelight.com

From changqy10 at gmail.com  Thu Oct 25 23:42:07 2018
From: changqy10 at gmail.com (qingyu chang)
Date: Fri, 26 Oct 2018 14:42:07 +0800
Subject: [Bro] bro, telnet.log
Message-ID: <CAFkth8-5ZeJ7shfgA03+Rto_8L+w=bmQmR2acxer+00w7dcDEw@mail.gmail.com>

Hi, i'm writing a bro script to generate telnet.log, as showing in
attachment. But when i simulate several remote telnet login actions,  bro
could't record all the login actions completely, lost nearly half. And if
i run this script with PCAPs, it tuns out to be normal.   Thanks for read
my letter.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181026/d92fb79d/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: main.bro
Type: application/octet-stream
Size: 3016 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181026/d92fb79d/attachment.obj 

From krasinski at cines.fr  Fri Oct 26 06:39:48 2018
From: krasinski at cines.fr (Nicolas KRASINSKI)
Date: Fri, 26 Oct 2018 15:39:48 +0200 (CEST)
Subject: [Bro] unknown identifier for Notice::ACTION_EMAIL
In-Reply-To: <CAFkth8-5ZeJ7shfgA03+Rto_8L+w=bmQmR2acxer+00w7dcDEw@mail.gmail.com>
References: <CAFkth8-5ZeJ7shfgA03+Rto_8L+w=bmQmR2acxer+00w7dcDEw@mail.gmail.com>
Message-ID: <1744765978.80353466.1540561188334.JavaMail.zimbra@cines.fr>

Hello,

I want bro to send email when a note is seen. 
I try adding these to local.bro :

redef Notice::emailed_types += {
 Address_Scan,
 Port_Scan,
};
hook Notice::policy(n: Notice::Info)
   {
if (n$note in Notice::emailed_types)
add n$actions[Notice::ACTION_EMAIL];
    }

Or this 


hook Notice::policy(n: Notice::Info)
   {
if (n$note == Address_Scan)
add n$actions[Notice::ACTION_EMAIL];
   }
 

But when I do broctl check I have this error : 

error in /usr/local/bro/share/bro/site/local.bro, line 13: unknown identifier Address_Scan, at or near "Address_Scan"

The script policy/misc/scan.bro is well loaded. 

Thanks a lot ! 

Nicolas. 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181026/ad3300f0/attachment.html 

From hosom at battelle.org  Fri Oct 26 07:25:07 2018
From: hosom at battelle.org (Hosom, Stephen M)
Date: Fri, 26 Oct 2018 14:25:07 +0000
Subject: [Bro] unknown identifier for Notice::ACTION_EMAIL
In-Reply-To: <1744765978.80353466.1540561188334.JavaMail.zimbra@cines.fr>
References: <CAFkth8-5ZeJ7shfgA03+Rto_8L+w=bmQmR2acxer+00w7dcDEw@mail.gmail.com>,
	<1744765978.80353466.1540561188334.JavaMail.zimbra@cines.fr>
Message-ID: <b0cdcd092e87484ea8f460dbbed9be2c@battelle.org>

Replace Address_Scan with Scan::Address_Scan and Port_Scan with Scan::Port_Scan and try again. You have to identify the module that those names are in.

________________________________
From: bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Nicolas KRASINSKI <krasinski at cines.fr>
Sent: Friday, October 26, 2018 9:39:48 AM
To: bro at bro.org
Subject: [Bro] unknown identifier for Notice::ACTION_EMAIL

Message received from outside the Battelle network. Carefully examine it before you open any links or attachments.


Hello,

I want bro to send email when a note is seen.
I try adding these to local.bro :

redef Notice::emailed_types += {
 Address_Scan,
 Port_Scan,
};
hook Notice::policy(n: Notice::Info)
   {
if (n$note in Notice::emailed_types)
add n$actions[Notice::ACTION_EMAIL];
    }

Or this


hook Notice::policy(n: Notice::Info)
   {
if (n$note == Address_Scan)
add n$actions[Notice::ACTION_EMAIL];
   }


But when I do broctl check I have this error :

error in /usr/local/bro/share/bro/site/local.bro, line 13: unknown identifier Address_Scan, at or near "Address_Scan"

The script policy/misc/scan.bro is well loaded.

Thanks a lot !

Nicolas.




From krasinski at cines.fr  Fri Oct 26 07:36:34 2018
From: krasinski at cines.fr (Nicolas KRASINSKI)
Date: Fri, 26 Oct 2018 16:36:34 +0200 (CEST)
Subject: [Bro] unknown identifier for Notice::ACTION_EMAIL
In-Reply-To: <b0cdcd092e87484ea8f460dbbed9be2c@battelle.org>
References: <CAFkth8-5ZeJ7shfgA03+Rto_8L+w=bmQmR2acxer+00w7dcDEw@mail.gmail.com>
	<1744765978.80353466.1540561188334.JavaMail.zimbra@cines.fr>
	<b0cdcd092e87484ea8f460dbbed9be2c@battelle.org>
Message-ID: <1126745251.80482472.1540564594547.JavaMail.zimbra@cines.fr>


Thanks, but what are you meaning by identify the module that those names are in ? 
I changed the Address_Scan by Scan::Address_Scan, but same.. 
I added "module Scan;" in local.bro but same error. 

When I add this directly in scan.bro, it works, but it is not advisable... 

hook Notice::policy(n: Notice::Info) 
{ 
if (n$note == Address_Scan) 
add n$actions[Notice::ACTION_EMAIL]; 
} 



De: "Hosom, Stephen M" <hosom at battelle.org> 
?: "krasinski" <krasinski at cines.fr>, "bro" <bro at bro.org> 
Envoy?: Vendredi 26 Octobre 2018 16:25:07 
Objet: Re: unknown identifier for Notice::ACTION_EMAIL 

Replace Address_Scan with Scan::Address_Scan and Port_Scan with Scan::Port_Scan and try again. You have to identify the module that those names are in. 

________________________________ 
From: bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Nicolas KRASINSKI <krasinski at cines.fr> 
Sent: Friday, October 26, 2018 9:39:48 AM 
To: bro at bro.org 
Subject: [Bro] unknown identifier for Notice::ACTION_EMAIL 

Message received from outside the Battelle network. Carefully examine it before you open any links or attachments. 


Hello, 

I want bro to send email when a note is seen. 
I try adding these to local.bro : 

redef Notice::emailed_types += { 
Address_Scan, 
Port_Scan, 
}; 
hook Notice::policy(n: Notice::Info) 
{ 
if (n$note in Notice::emailed_types) 
add n$actions[Notice::ACTION_EMAIL]; 
} 

Or this 


hook Notice::policy(n: Notice::Info) 
{ 
if (n$note == Address_Scan) 
add n$actions[Notice::ACTION_EMAIL]; 
} 


But when I do broctl check I have this error : 

error in /usr/local/bro/share/bro/site/local.bro, line 13: unknown identifier Address_Scan, at or near "Address_Scan" 

The script policy/misc/scan.bro is well loaded. 

Thanks a lot ! 

Nicolas. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181026/184a2742/attachment-0001.html 

From krasinski at cines.fr  Fri Oct 26 09:35:14 2018
From: krasinski at cines.fr (Nicolas KRASINSKI)
Date: Fri, 26 Oct 2018 18:35:14 +0200 (CEST)
Subject: [Bro] unknown identifier for Notice::ACTION_EMAIL
In-Reply-To: <1126745251.80482472.1540564594547.JavaMail.zimbra@cines.fr>
References: <CAFkth8-5ZeJ7shfgA03+Rto_8L+w=bmQmR2acxer+00w7dcDEw@mail.gmail.com>
	<1744765978.80353466.1540561188334.JavaMail.zimbra@cines.fr>
	<b0cdcd092e87484ea8f460dbbed9be2c@battelle.org>
	<1126745251.80482472.1540564594547.JavaMail.zimbra@cines.fr>
Message-ID: <1702350284.80765739.1540571714048.JavaMail.zimbra@cines.fr>

Ok, I found the solution, I added in my local.bro : 

@load misc/scan 

with the code : 

hook Notice::policy(n: Notice::Info) 
{ 
if ( n$note == Scan::Address_Scan || n$note == Scan::Port_Scan ) 
add n$actions[Notice::ACTION_EMAIL]; 
} 

and it works fine. 
Thank you, 


Nicolas. 



De: "krasinski" <krasinski at cines.fr> 
?: "Stephen M Hosom" <hosom at battelle.org> 
Cc: "bro" <bro at bro.org> 
Envoy?: Vendredi 26 Octobre 2018 16:36:34 
Objet: Re: [Bro] unknown identifier for Notice::ACTION_EMAIL 


Thanks, but what are you meaning by identify the module that those names are in ? 
I changed the Address_Scan by Scan::Address_Scan, but same.. 
I added "module Scan;" in local.bro but same error. 

When I add this directly in scan.bro, it works, but it is not advisable... 

hook Notice::policy(n: Notice::Info) 
{ 
if (n$note == Address_Scan) 
add n$actions[Notice::ACTION_EMAIL]; 
} 



De: "Hosom, Stephen M" <hosom at battelle.org> 
?: "krasinski" <krasinski at cines.fr>, "bro" <bro at bro.org> 
Envoy?: Vendredi 26 Octobre 2018 16:25:07 
Objet: Re: unknown identifier for Notice::ACTION_EMAIL 

Replace Address_Scan with Scan::Address_Scan and Port_Scan with Scan::Port_Scan and try again. You have to identify the module that those names are in. 

________________________________ 
From: bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Nicolas KRASINSKI <krasinski at cines.fr> 
Sent: Friday, October 26, 2018 9:39:48 AM 
To: bro at bro.org 
Subject: [Bro] unknown identifier for Notice::ACTION_EMAIL 

Message received from outside the Battelle network. Carefully examine it before you open any links or attachments. 


Hello, 

I want bro to send email when a note is seen. 
I try adding these to local.bro : 

redef Notice::emailed_types += { 
Address_Scan, 
Port_Scan, 
}; 
hook Notice::policy(n: Notice::Info) 
{ 
if (n$note in Notice::emailed_types) 
add n$actions[Notice::ACTION_EMAIL]; 
} 

Or this 


hook Notice::policy(n: Notice::Info) 
{ 
if (n$note == Address_Scan) 
add n$actions[Notice::ACTION_EMAIL]; 
} 


But when I do broctl check I have this error : 

error in /usr/local/bro/share/bro/site/local.bro, line 13: unknown identifier Address_Scan, at or near "Address_Scan" 

The script policy/misc/scan.bro is well loaded. 

Thanks a lot ! 

Nicolas. 

_______________________________________________ 
Bro mailing list 
bro at bro-ids.org 
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181026/08ba8a4e/attachment.html 

From evasionalert at yandex.com  Sun Oct 28 05:25:33 2018
From: evasionalert at yandex.com (Evasion alert)
Date: Sun, 28 Oct 2018 13:25:33 +0100
Subject: [Bro] Fwd: bro-postgresql Configuration example
In-Reply-To: <16074161540729359@sas1-19a94364928d.qloud-c.yandex.net>
Message-ID: <17803401540729533@myt5-b94652e6921b.qloud-c.yandex.net>

An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181028/f6023121/attachment.html 

From nothinrandom at gmail.com  Sun Oct 28 20:40:39 2018
From: nothinrandom at gmail.com (TQ)
Date: Sun, 28 Oct 2018 20:40:39 -0700
Subject: [Bro] Syntax Question - Nested Switch Case
In-Reply-To: <CAMzgZ0+eVgODC-LEKY71kF+12VFkyU1W_WiQmxeJzhumgKjXbA@mail.gmail.com>
References: <CAM2MKSqymHV-JEeE6m829cKj2jwcqPGyucP6yo+x=CbMZjif=w@mail.gmail.com>
	<CAMzgZ0+eVgODC-LEKY71kF+12VFkyU1W_WiQmxeJzhumgKjXbA@mail.gmail.com>
Message-ID: <CAM2MKSo0ucf6w30e-1frHcK+bfmwKunAd-KbEyMHUp0JG6dDcg@mail.gmail.com>

Hey Jon,

Thanks for the reply. Works great. However, what's the best way to continue
to keep on switching inside Record_A/B ? With your incorporated suggestions
previously:

enum cmd_codes {
NOP = 0x00000000,
DEVICE_HEADER = 0x AABBCCDD,
DEVICE_CMD2_1 = 0x0304,
        DEVICE_CMD2_2 = 0x0405
};

type Header = record {
header: uint32; # header
cmd1: uint16; # 0x0102
cmd2: uint16; # 0x0304 or 0x0405
} &byteorder=bigendian;

type Device_Response = record {
header: Device_Header;
data: case(header.header) of {
DEVICE_HEADER -> head:  HeaderCmd(header.cmd2);
# All the rest
default -> unknown: bytestring &restofdata;
};
} &byteorder=littleendian;

type HeaderCmd(cmd: uint16) = case cmd of {
  DEVICE_CMD2_1  -> info1: Record_A;
  DEVICE_CMD2_2  -> info2: Record_B;
};

type Record_A = record {
title: uint16;
author: uint32;
text: uint64;
ending: uint8;
}

type Record_B = record {
title: uint16;
author: uint32;
text: uint64;
ending: uint8;
}

Based on the ending for each Record, I'd switch to another set. Would I set
it up like Device_Response from above if Record_A and Record_B share some
data before the switch occurs?
type Record_Response = record {
header:  Record_A  ;
data: case(header.ending) of {
HAPPY -> happy:   EndingCmd(header.ending);
SAD  -> sad:   EndingCmd(header.ending);
# All the rest
default -> unknown: bytestring &restofdata;
};
} &byteorder=littleendian;

type EndingCmd(ending : uint16) = case  ending of {
  HAPPY_1  -> info1: Record_A;
  HAPPY_2  -> info2: Record_B;
};

type Ending = record {
ending: bytestring;
}

All of the above is under protocol.pac. How would one access information
from analyzer.pac? I could send you some files if none of this makes sense.

Thanks!

On Tue, Oct 16, 2018 at 9:42 AM Jon Siwek <jsiwek at corelight.com> wrote:

> On Tue, Oct 16, 2018 at 12:46 AM TQ <nothinrandom at gmail.com> wrote:
>
> > How do you do a nested switch case inside a record? I have some data
> 0xAABBCCDD01020304 or 0xAABBCCDD01020405 that I need to verify that the
> header is 0xAABBCCDD and switch based on the last two bytes, either 0x0304
> or 0x0405. Is this a good practice of switch record since data length will
> change based on the command. The nested case I have below is incorrect and
> is throwing error "make[3]: *** [test_pac.h] Segmentation fault (core
> dumped)"
>
> It's possible that you've just run into a binpac bug related to nested
> records and you can file a bug/issue for that on GitHub, but you may
> also be able to organize things differently.  For example, instead of
> nesting switch/case you can do:
>
> type HeaderCmd(cmd: uint16) = case cmd of {
>   DEVICE_CMD2_1  -> info1: Record_A;
>   DEVICE_CMD2_2  -> info2: Record_B;
> };
>
> type Device_Response = record {
>   header: Header;
>   data: case(header.header) of {
>     DEVICE_HEADER -> head_cmd: HeaderCmd(header.cmd2);
>     default       -> unknown: bytestring &restofdata;
>   };
> } &byteorder=littleendian;
>
> Also, if you only ever expect to see that DEVICE_HEADER value and not
> any other header values for this protocol, you might just use an
> &enforce attribute at the top level instead of a switch/case.
>
> - Jon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181028/37a43508/attachment.html 

From jsiwek at corelight.com  Mon Oct 29 08:13:37 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Mon, 29 Oct 2018 10:13:37 -0500
Subject: [Bro] Syntax Question - Nested Switch Case
In-Reply-To: <CAM2MKSo0ucf6w30e-1frHcK+bfmwKunAd-KbEyMHUp0JG6dDcg@mail.gmail.com>
References: <CAM2MKSqymHV-JEeE6m829cKj2jwcqPGyucP6yo+x=CbMZjif=w@mail.gmail.com>
	<CAMzgZ0+eVgODC-LEKY71kF+12VFkyU1W_WiQmxeJzhumgKjXbA@mail.gmail.com>
	<CAM2MKSo0ucf6w30e-1frHcK+bfmwKunAd-KbEyMHUp0JG6dDcg@mail.gmail.com>
Message-ID: <CAMzgZ0K2ccuWPinmj8_JFR7smPKgH8qsNhns1FoV+DRXKKCqrw@mail.gmail.com>

On Sun, Oct 28, 2018 at 10:41 PM TQ <nothinrandom at gmail.com> wrote:

> type HeaderCmd(cmd: uint16) = case cmd of {
>   DEVICE_CMD2_1  -> info1: Record_A;
>   DEVICE_CMD2_2  -> info2: Record_B;
> };
>
> type Record_A = record {
> title: uint16;
> author: uint32;
> text: uint64;
> ending: uint8;
> }
>
> type Record_B = record {
> title: uint16;
> author: uint32;
> text: uint64;
> ending: uint8;
> }
>
> Based on the ending for each Record, I'd switch to another set. Would I set it up like Device_Response from above if Record_A and Record_B share some data before the switch occurs?

It's up to you, but if Record_{A,B} share a common format, you might
change HeaderCmd to be a record that contains the common fields plus
an extra switch.  Something like:

type HeaderCmd(cmd: uint16) = record {
  title: uint16;
  author: uint32;
  text: uint64;
  ending: uint8;
  endcmd: case ending of {
    ...
  };
};

> All of the above is under protocol.pac. How would one access information from analyzer.pac?

You can see most analyzers in the Bro source tree for examples, but a
typical pattern is refining the type to add some post-processing
function.  e.g.:

refine connection My_Conn += {
  function proc_header_cmd(rec: HeaderCmd): bool
    %{
    // Add your code here
    return true;
    %}
};

refine typeattr HeaderCmd += &let {
  proc: bool = $context.connection.proc_header_cmd(this);
};

There's nothing special about the filenames -analyzer.pac vs.
-protocol.pac, you can access the same type definitions wherever as
long as the %include setup is correct, but the convention that Bro
tends to use is to put protocol parsing logic in -protocol.pac and
additional protocol analysis/validation logic in -analyzer.pac.

- Jon

From dopheide at gmail.com  Mon Oct 29 09:48:12 2018
From: dopheide at gmail.com (Mike Dopheide)
Date: Mon, 29 Oct 2018 11:48:12 -0500
Subject: [Bro] bro, telnet.log
In-Reply-To: <CAFkth8-5ZeJ7shfgA03+Rto_8L+w=bmQmR2acxer+00w7dcDEw@mail.gmail.com>
References: <CAFkth8-5ZeJ7shfgA03+Rto_8L+w=bmQmR2acxer+00w7dcDEw@mail.gmail.com>
Message-ID: <CAPy2kFaioep4y+=tnjX8rZWmXduu2ESdtz_DDo_g1OnEt7ntzw@mail.gmail.com>

I can't speak to why you're only seeing some of the messages unless you're
using a pcap.  However, I'm curious what version of Bro you're running.  I
believe the telnet analyzer was removed prior to 2.x precisely because it
was inaccurate.

-Dop

On Fri, Oct 26, 2018 at 1:57 AM qingyu chang <changqy10 at gmail.com> wrote:

> Hi, i'm writing a bro script to generate telnet.log, as showing in
> attachment. But when i simulate several remote telnet login actions,  bro
> could't record all the login actions completely, lost nearly half. And if
> i run this script with PCAPs, it tuns out to be normal.   Thanks for read
> my letter.
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181029/143c8228/attachment.html 

From jsiwek at corelight.com  Mon Oct 29 12:26:37 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Mon, 29 Oct 2018 14:26:37 -0500
Subject: [Bro] git submodule URL change
Message-ID: <CAMzgZ0+_zGdfPCUxtdkdoteKW3Eu5ayiRAq8SP0QAC9wUPj4Ow@mail.gmail.com>

FYI if you have an existing Bro git repo clone and want to update/pull
the master branch and all submodules recursively, it will require
syncing one time:

  git submodule sync --recursive

That will update the URL for a git submodule that just changed.
Afterwards, the usual update command will work:

  git submodule update --recursive --init

New git clones Just Work.

- Jon

From vern at icir.org  Mon Oct 29 13:10:24 2018
From: vern at icir.org (Vern Paxson)
Date: Mon, 29 Oct 2018 13:10:24 -0700
Subject: [Bro] Bro/Zeek project leadership update - welcome, Keith Lehigh!
Message-ID: <201810292010.w9TKAOZu025710@fruitcake.ICSI.Berkeley.EDU>

Adam Slagell decided earlier this year that it was time to step down from
his service as the chair of the Bro/Zeek project's Leadership Team[*].
(He'll be continuing as a member of the LT.)  The LT has enthusiastically
selected Keith Lehigh of Indiana University to serve as its new chair.
Many of you will know Keith from his service as this year's program committee
chair of BroCon 2018.

Our thanks to Adam for his industrious and invaluable service!

		Vern


[*] https://www.bro.org/team.html

From ambros.novak.89 at gmail.com  Mon Oct 29 16:12:44 2018
From: ambros.novak.89 at gmail.com (Ambros Novak)
Date: Mon, 29 Oct 2018 19:12:44 -0400
Subject: [Bro] large notice.log
Message-ID: <CAHifxyCGfhp1Sne9=HbLVNVU0CiVvrGZ9SFs7ukLGtvyC17ewg@mail.gmail.com>

Holla!

notice.log is extremely large before it rotates, sometimes 140G+. At times
it rotates to another log with a timestamp added to it's name. This
happened after turning on other analyzers.

Is there a way to suppress notice.log or minimize the events written to it.
The events in the other logs are more important.

There are also other logs that are extremely large as well, and I'm trying
to balance processing and space vs the visibility.

Any advice appreciated.

Merci!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181029/8fcdf49d/attachment.html 

From hosom at battelle.org  Tue Oct 30 09:11:51 2018
From: hosom at battelle.org (Hosom, Stephen M)
Date: Tue, 30 Oct 2018 16:11:51 +0000
Subject: [Bro] large notice.log
In-Reply-To: <CAHifxyCGfhp1Sne9=HbLVNVU0CiVvrGZ9SFs7ukLGtvyC17ewg@mail.gmail.com>
References: <CAHifxyCGfhp1Sne9=HbLVNVU0CiVvrGZ9SFs7ukLGtvyC17ewg@mail.gmail.com>
Message-ID: <b83f3e67349f4d90b9bb46efa2e89d9f@battelle.org>

I could directly answer your question, but I don't think that's necessarily what you want.


> notice.log rotates at 140GB+


Wow! That's a big notice.log. Do you load any custom scripts that generate notices to the notice framework? If you aren't monitoring an extremely large network, this could indicate that something is wrong. Which notes are most common in your notice.log?


> Is there a way to suppress notice.log or minimize the events written to it?


Yes! You can suppress individual notices or the entire log. To suppress notices, you can take a look at the documentation on the Notice framework here: https://www.bro.org/sphinx/frameworks/notice.html


To suppress the entire log, or parts of it... you'd have to use the logging framework: https://www.bro.org/sphinx-git/frameworks/logging.html


I'd recommend looking into why your notice.log is so big though. If you have many logs that are extremely large, it is possible that you are simply monitoring a LOT of traffic. You may have to pick and choose what Bro sees if you can't keep up with the logs, or you could apply some logging filters to cut down on the information that you don't care about.


Additionally, you can disable common notices that you don't care about. Consider adding some notices to Notice::ignored_types.


-Stephen

________________________________
From: bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Ambros Novak <ambros.novak.89 at gmail.com>
Sent: Monday, October 29, 2018 7:12:44 PM
To: bro at bro.org
Subject: [Bro] large notice.log

Message received from outside the Battelle network. Carefully examine it before you open any links or attachments.

Holla!

notice.log is extremely large before it rotates, sometimes 140G+. At times it rotates to another log with a timestamp added to it's name. This happened after turning on other analyzers.

Is there a way to suppress notice.log or minimize the events written to it. The events in the other logs are more important.

There are also other logs that are extremely large as well, and I'm trying to balance processing and space vs the visibility.

Any advice appreciated.

Merci!




From seth at corelight.com  Tue Oct 30 09:14:52 2018
From: seth at corelight.com (Seth Hall)
Date: Tue, 30 Oct 2018 12:14:52 -0400
Subject: [Bro] large notice.log
In-Reply-To: <CAHifxyCGfhp1Sne9=HbLVNVU0CiVvrGZ9SFs7ukLGtvyC17ewg@mail.gmail.com>
References: <CAHifxyCGfhp1Sne9=HbLVNVU0CiVvrGZ9SFs7ukLGtvyC17ewg@mail.gmail.com>
Message-ID: <25399D6E-AF3A-4D7F-B7E7-2AFBA00E2020@corelight.com>

On 29 Oct 2018, at 19:12, Ambros Novak wrote:

> notice.log is extremely large before it rotates, sometimes 140G+.

Wow! What notices do you have? It sounds like you may have a notice that 
is getting out of control and it might make more sense figuring out 
what's going on rather than just trying to muffle the creation of these 
notices.

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com

From matt.thoreson at summitinfosec.com  Wed Oct 31 10:37:32 2018
From: matt.thoreson at summitinfosec.com (Matt Thoreson)
Date: Wed, 31 Oct 2018 10:37:32 -0700
Subject: [Bro] Bro decapsulating ERSPAN (GRE)
Message-ID: <CAKM8Hq21sGefmL+guVVaO7Dm65ME89bzSZwD2Bn1e+8nL4Yz+g@mail.gmail.com>

Hello,

I've got a vmware instance of Ubuntu running Bro 2.6-beta2.   I want bro to
monitor the eth0 interface that is directly receiving ERSPAN (gre tunneled)
data from a Cisco switch.   I've tried a few different scenarios.   I
thought Bro could by default recognize and decapsulate the real traffic
from the GRE tunnel (according to the bro notes it should be able to do
this) but so far when bro runs it just sees the gre traffic in it's
weird.log.  I've also tried creating another tunnel interface tun0 set up
as GRE on the Ubuntu instance and have the traffic forwarded from eth0 to
tun0 and have linux decapsulate it.  That is not working either.

Has anyone gotten something similar to work reading cisco ERSPAN traffic
into bro?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181031/8487bd8d/attachment.html 

From jsiwek at corelight.com  Wed Oct 31 11:07:31 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Wed, 31 Oct 2018 13:07:31 -0500
Subject: [Bro] Bro decapsulating ERSPAN (GRE)
In-Reply-To: <CAKM8Hq21sGefmL+guVVaO7Dm65ME89bzSZwD2Bn1e+8nL4Yz+g@mail.gmail.com>
References: <CAKM8Hq21sGefmL+guVVaO7Dm65ME89bzSZwD2Bn1e+8nL4Yz+g@mail.gmail.com>
Message-ID: <CAMzgZ0Kb0LhoD1w85m4qkU3L2YqzcTNnu5vCtDZJ0W11aB+YPQ@mail.gmail.com>

On Wed, Oct 31, 2018 at 12:40 PM Matt Thoreson
<matt.thoreson at summitinfosec.com> wrote:

> I thought Bro could by default recognize and decapsulate the real traffic from the GRE tunnel (according to the bro notes it should be able to do this) but so far when bro runs it just sees the gre traffic in it's weird.log.

It currently only handles a few GRE protocol types, and doesn't seem
the ERSPAN ones are among them.

- Jon

From jsiwek at corelight.com  Wed Oct 31 14:32:32 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Wed, 31 Oct 2018 16:32:32 -0500
Subject: [Bro] Bro decapsulating ERSPAN (GRE)
In-Reply-To: <CAMzgZ0Kb0LhoD1w85m4qkU3L2YqzcTNnu5vCtDZJ0W11aB+YPQ@mail.gmail.com>
References: <CAKM8Hq21sGefmL+guVVaO7Dm65ME89bzSZwD2Bn1e+8nL4Yz+g@mail.gmail.com>
	<CAMzgZ0Kb0LhoD1w85m4qkU3L2YqzcTNnu5vCtDZJ0W11aB+YPQ@mail.gmail.com>
Message-ID: <CAMzgZ0++PDRDfR5aYVxqwB0BcaCo=Wun6=b2dweo0Mo-pTSYqA@mail.gmail.com>

On Wed, Oct 31, 2018 at 1:07 PM Jon Siwek <jsiwek at corelight.com> wrote:
>
> On Wed, Oct 31, 2018 at 12:40 PM Matt Thoreson
> <matt.thoreson at summitinfosec.com> wrote:
>
> > I thought Bro could by default recognize and decapsulate the real traffic from the GRE tunnel (according to the bro notes it should be able to do this) but so far when bro runs it just sees the gre traffic in it's weird.log.
>
> It currently only handles a few GRE protocol types, and doesn't seem
> the ERSPAN ones are among them.

To clarify that further: I totally missed that the changelog does say
ERSPAN support was implemented, but I was just looking at the actual
code, which does not seem to handle ERSPAN Type II or III (protocol
types 0x88BE, 0x22EB).  The associated commit seems to instead handle
Transparent Ethernet Bridging (protocol type 0x6558).  Not sure if I'm
missing something.  Or if you can give a pcap to test against, that
could help to verify what's going and also serve as test case for
fixing anything that's broken/unimplemented in Bro.

- Jon