From ali at ashemery.com  Mon Jun  1 03:07:19 2015
From: ali at ashemery.com (Ali Hadi)
Date: Mon, 1 Jun 2015 13:07:19 +0300
Subject: [Bro] tx_hosts and rx_hosts in files.log
In-Reply-To: <CANUJ_F=On06e98=zPJt3mALCFDSP23Aw4ZF=Hjaf9-d8YKzXYA@mail.gmail.com>
References: <CAGG-kC+yWP0cg+V+40MabFo3WQdNpauMNSLn-_Qxm4gHWxXzuQ@mail.gmail.com>
	<CANUJ_F=On06e98=zPJt3mALCFDSP23Aw4ZF=Hjaf9-d8YKzXYA@mail.gmail.com>
Message-ID: <CAGG-kCLpHvA85Tweeg=y2y_y_2B2=sqXastbBffhOSSC-mMHKQ@mail.gmail.com>

You're welcome. Hope it will be corrected soon.

?Ali

On Mon, Jun 1, 2015 at 12:35 AM, Vlad Grigorescu <vlad at grigorescu.org>
wrote:

> Thanks for the bug report. Looks like this comes from the assumption made
> here:
>
>
> https://github.com/bro/bro/blob/master/src/analyzer/protocol/mime/MIME.cc#L1459
>
>   --Vlad
>
> On Sat, May 30, 2015 at 2:16 PM, Ali Hadi <ali at ashemery.com> wrote:
>
>> Hi,
>>
>> If you use the PCAP below and analyze it using Bro:
>> https://www.bro.org/static/traces/email.pcap
>>
>> Then when checking the files.log, the tx_hosts is supposed to show the
>> host who transmitted the file, and rx_hosts is for the host who received
>> the file based on Bro's documentation:
>> https://www.bro.org/sphinx-git/scripts/base/frameworks/files/main.bro.html
>>
>> If you do the following:
>> cat files.log | bro-cut fuid tx_hosts rx_hosts | grep <ID OF THE LEAKED
>> PDF FILE>
>>
>> You'll get that the TX Host IP (SrcIP) is 192.168.121.176 and
>> not 192.168.121.179 !!!
>>
>> Is there something I'm doing wrong, or has bro switched their positions
>> in the output?
>>
>> ?Thanks in advance,
>> *Ali*
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150601/b2fc1105/attachment.html 

From robin at icir.org  Mon Jun  1 07:25:02 2015
From: robin at icir.org (Robin Sommer)
Date: Mon, 1 Jun 2015 07:25:02 -0700
Subject: [Bro] tx_hosts and rx_hosts in files.log
In-Reply-To: <CAGG-kCLpHvA85Tweeg=y2y_y_2B2=sqXastbBffhOSSC-mMHKQ@mail.gmail.com>
References: <CAGG-kC+yWP0cg+V+40MabFo3WQdNpauMNSLn-_Qxm4gHWxXzuQ@mail.gmail.com>
	<CANUJ_F=On06e98=zPJt3mALCFDSP23Aw4ZF=Hjaf9-d8YKzXYA@mail.gmail.com>
	<CAGG-kCLpHvA85Tweeg=y2y_y_2B2=sqXastbBffhOSSC-mMHKQ@mail.gmail.com>
Message-ID: <20150601142502.GG72788@icir.org>

On Mon, Jun 01, 2015 at 13:07 +0300, Ali Hadi wrote:

> You're welcome. Hope it will be corrected soon.

Please file a ticket with our tracker.

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From ali at ashemery.com  Mon Jun  1 11:44:01 2015
From: ali at ashemery.com (Ali Hadi)
Date: Mon, 1 Jun 2015 21:44:01 +0300
Subject: [Bro] tx_hosts and rx_hosts in files.log
In-Reply-To: <20150601142502.GG72788@icir.org>
References: <CAGG-kC+yWP0cg+V+40MabFo3WQdNpauMNSLn-_Qxm4gHWxXzuQ@mail.gmail.com>
	<CANUJ_F=On06e98=zPJt3mALCFDSP23Aw4ZF=Hjaf9-d8YKzXYA@mail.gmail.com>
	<CAGG-kCLpHvA85Tweeg=y2y_y_2B2=sqXastbBffhOSSC-mMHKQ@mail.gmail.com>
	<20150601142502.GG72788@icir.org>
Message-ID: <CAGG-kC+pu8nbSSPxfh74EuToA9f3Ac8sCMjWP3G-joGtzfO1_Q@mail.gmail.com>

Okay Robin, it's done.


Best regards,

*Ali *

On Mon, Jun 1, 2015 at 5:25 PM, Robin Sommer <robin at icir.org> wrote:

> On Mon, Jun 01, 2015 at 13:07 +0300, Ali Hadi wrote:
>
> > You're welcome. Hope it will be corrected soon.
>
> Please file a ticket with our tracker.
>
> Robin
>
> --
> Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150601/bae3a3e8/attachment-0001.html 

From usmanshafique68 at hotmail.com  Mon Jun  1 21:09:44 2015
From: usmanshafique68 at hotmail.com (usman shafique)
Date: Tue, 2 Jun 2015 09:09:44 +0500
Subject: [Bro] tx_hosts and rx_hosts in files.log
In-Reply-To: <CAGG-kC+pu8nbSSPxfh74EuToA9f3Ac8sCMjWP3G-joGtzfO1_Q@mail.gmail.com>
References: <CAGG-kC+yWP0cg+V+40MabFo3WQdNpauMNSLn-_Qxm4gHWxXzuQ@mail.gmail.com>,
	<CANUJ_F=On06e98=zPJt3mALCFDSP23Aw4ZF=Hjaf9-d8YKzXYA@mail.gmail.com>,
	<CAGG-kCLpHvA85Tweeg=y2y_y_2B2=sqXastbBffhOSSC-mMHKQ@mail.gmail.com>,
	<20150601142502.GG72788@icir.org>,
	<CAGG-kC+pu8nbSSPxfh74EuToA9f3Ac8sCMjWP3G-joGtzfO1_Q@mail.gmail.com>
Message-ID: <SNT147-W32291BB9E387FA1D466179B1B50@phx.gbl>

can any one tell me how to analyze  dynamic protocol in bro i am beginner i just understand bro scripting and how to run please give me simple example thanks
RegardsUSman 
Date: Mon, 1 Jun 2015 21:44:01 +0300
From: ali at ashemery.com
To: robin at icir.org
CC: bro at bro.org
Subject: Re: [Bro] tx_hosts and rx_hosts in files.log

Okay Robin, it's done.
Best regards,
Ali 

On Mon, Jun 1, 2015 at 5:25 PM, Robin Sommer <robin at icir.org> wrote:
On Mon, Jun 01, 2015 at 13:07 +0300, Ali Hadi wrote:



> You're welcome. Hope it will be corrected soon.



Please file a ticket with our tracker.



Robin



--

Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin




_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150602/c92205c0/attachment.html 

From giedrius.ramas at gmail.com  Mon Jun  1 23:09:07 2015
From: giedrius.ramas at gmail.com (Giedrius Ramas)
Date: Tue, 2 Jun 2015 09:09:07 +0300
Subject: [Bro] BRO 2.3.2 Intel email indicator do not work
Message-ID: <CAHr6V5pNOmSW55+MHruMtgFDe44pZNpMGxa8EtjgQj8-HSNM3A@mail.gmail.com>

Hi,

I found that BRO 2.3.4 Intel do not work with email's indicators. I have
played on my infrastructure to get BRO intel work and found that email
indicator won't work.

I also tested it on  try.bro.org/ the same results . However BRO 2.2
version works well with Intel email's indicators .

Please let me know if more details needed to troubleshoot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150602/e07ec7ad/attachment.html 

From close at ou.edu  Tue Jun  2 07:29:39 2015
From: close at ou.edu (Close, Jason M.)
Date: Tue, 2 Jun 2015 14:29:39 +0000
Subject: [Bro] Multiple masters to ease the workload
Message-ID: <D1932800.4751%close@ou.edu>

Our current configuration is showing a lot of heavy use by the master node.  We currently run around 6 worker nodes that feed data to the master, and while the master is keeping up in terms of CPU, it is consistently teetering on using all available RAM we can throw at it (128GB at the moment).  There are plans in place to increase our available bandwidth 10-fold, so the traffic coming to Bro will ramp up as well.

We could piece apart the subnets and create multiple Bro clusters.  But it would be nice to have a single cluster, and be able to continue to throw more workers and managers at it.  But I have not seen any documentation about configurations using multiple managers.  If that does exist, can someone point me in the right direction?

And if that doesn?t exist, can I get some suggestions about mitigations to this problem?  I know there are a lot of cool things being done with Bro, especially using scripts and APIs where Bro can help reduce traffic being thrown to it.  But due to the taps we have in place, and the manpower availability, right now, spinning up a little more hardware would be a much easier and more economical investment of our time.

Thanks.

Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150602/f466883d/attachment.html 

From seth at icir.org  Tue Jun  2 11:38:33 2015
From: seth at icir.org (Seth Hall)
Date: Tue, 2 Jun 2015 14:38:33 -0400
Subject: [Bro] Multiple masters to ease the workload
In-Reply-To: <D1932800.4751%close@ou.edu>
References: <D1932800.4751%close@ou.edu>
Message-ID: <283D2D69-2DA6-47D5-8C7E-5922867AB779@icir.org>


> On Jun 2, 2015, at 10:29 AM, Close, Jason M. <close at ou.edu> wrote:
> 
> Our current configuration is showing a lot of heavy use by the master node.  We currently run around 6 worker nodes that feed data to the master, and while the master is keeping up in terms of CPU, it is consistently teetering on using all available RAM we can throw at it (128GB at the moment).

That?s indicating a problem.  I?m going to send you a script off-list that you can run and we?ll see if we can nail down what?s causing that.

> We could piece apart the subnets and create multiple Bro clusters.  But it would be nice to have a single cluster, and be able to continue to throw more workers and managers at it.  But I have not seen any documentation about configurations using multiple managers.  If that does exist, can someone point me in the right direction?

You can only run a single manager.

> But due to the taps we have in place, and the manpower availability, right now, spinning up a little more hardware would be a much easier and more economical investment of our time.

Unfortunately in this case you need to fix the problem and can?t really just throw more hardware at it.

  .Seht

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150602/3b25ae60/attachment.bin 

From bilal.comsian09 at gmail.com  Wed Jun  3 07:00:18 2015
From: bilal.comsian09 at gmail.com (bilal Ahmed)
Date: Wed, 03 Jun 2015 19:00:18 +0500
Subject: [Bro] Broccoli
Message-ID: <l7dn5s5xk8y3ruu80mc6byy1.1433340018028@email.android.com>

Hi, I have problem in broccoli it is already configured but I want to test broping.c
If anybody know how to test and use that broccoli is now working 


From earl.eiland at root9b.com  Thu Jun  4 12:16:15 2015
From: earl.eiland at root9b.com (Earl Eiland)
Date: Thu, 4 Jun 2015 19:16:15 +0000
Subject: [Bro] problem with known-services.bro
Message-ID: <CY1PR0401MB1387239619368010AECFDD13D7B30@CY1PR0401MB1387.namprd04.prod.outlook.com>

Hello.  I'm running known-services.bro on a pcap file.  I'm having two problems:  no known-services.log file is being generated, and some packets trigger "internal warning: Unexpected IP version in FragReassembler".


If I run dpd, the logs generated seem normal.


What can I do to debug these issues?


Best Regards,

Earl Eiland,
Sr. Cyber Security Engineer,
Emerging Technologies, root9B,
San Antonio, Texas

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity named.  If you are not the named addressee you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.  Please notify the sender immediately by email if you received this email in error and delete this email from your system. Any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of root9B LLC.?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150604/5bddd5c7/attachment.html 

From bro at pingtrip.com  Thu Jun  4 18:37:29 2015
From: bro at pingtrip.com (Dave Crawford)
Date: Thu, 4 Jun 2015 21:37:29 -0400
Subject: [Bro] Multiple masters to ease the workload
In-Reply-To: <D1932800.4751%close@ou.edu>
References: <D1932800.4751%close@ou.edu>
Message-ID: <F109D630-CBDB-4893-8649-050D0EB0F393@pingtrip.com>

Is it actually 100% RAM usage by applications? Since the manager can be performing a significant amount of disk writes the kernel will allocate ?free? memory as ?cached? to increase file performance. The cached memory is released when applications demand more memory.

Below is the current memory usage on one of my mangers that is handling 25 workers and 2 proxies. At first glance it appears that all the memory has been consumed, but notice how 122G is cached.

             total       used       free     shared    buffers     cached
Mem:          126G       125G       384M         0B       329M       122G
-/+ buffers/cache:       2.6G       123G
Swap:          33G         0B        33G


-Dave

> On Jun 2, 2015, at 10:29 AM, Close, Jason M. <close at ou.edu> wrote:
> 
> Our current configuration is showing a lot of heavy use by the master node.  We currently run around 6 worker nodes that feed data to the master, and while the master is keeping up in terms of CPU, it is consistently teetering on using all available RAM we can throw at it (128GB at the moment).  There are plans in place to increase our available bandwidth 10-fold, so the traffic coming to Bro will ramp up as well. 
> 
> We could piece apart the subnets and create multiple Bro clusters.  But it would be nice to have a single cluster, and be able to continue to throw more workers and managers at it.  But I have not seen any documentation about configurations using multiple managers.  If that does exist, can someone point me in the right direction?
> 
> And if that doesn?t exist, can I get some suggestions about mitigations to this problem?  I know there are a lot of cool things being done with Bro, especially using scripts and APIs where Bro can help reduce traffic being thrown to it.  But due to the taps we have in place, and the manpower availability, right now, spinning up a little more hardware would be a much easier and more economical investment of our time.
> 
> Thanks.
> 
> Jason
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150604/ec600a77/attachment.html 

From seth at icir.org  Thu Jun  4 19:23:16 2015
From: seth at icir.org (Seth Hall)
Date: Thu, 4 Jun 2015 22:23:16 -0400
Subject: [Bro] problem with known-services.bro
In-Reply-To: <CY1PR0401MB1387239619368010AECFDD13D7B30@CY1PR0401MB1387.namprd04.prod.outlook.com>
References: <CY1PR0401MB1387239619368010AECFDD13D7B30@CY1PR0401MB1387.namprd04.prod.outlook.com>
Message-ID: <23066CB0-ECB4-4083-A2C4-E6FBAFDDEBC4@icir.org>

> On Jun 4, 2015, at 3:16 PM, Earl Eiland <earl.eiland at root9b.com> wrote:
> 
> Hello.  I'm running known-services.bro on a pcap file.  I'm having two problems:  no known-services.log file is being generated, and some packets trigger "internal warning: Unexpected IP version in FragReassembler?.

By default the known-hosts script only records hosts in your Site::local_nets.  You can set that with networks.cfg if you are running broctl or you can set it directly in Bro scripts like this...

redef Site::local_nets += { 1.2.3.0/24, 5.6.7.0/24 };

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150604/ccdad951/attachment.bin 

From bilal.comsian09 at gmail.com  Thu Jun  4 22:06:53 2015
From: bilal.comsian09 at gmail.com (bilal Ahmed)
Date: Fri, 05 Jun 2015 10:06:53 +0500
Subject: [Bro] Broccoli
Message-ID: <2yv5sexq7e7upw310a7thdaf.1433480813217@email.android.com>

How to configure broccoli I have installed it but don't know from where I start ??
Regards
Bilal

 Seth Hall <seth at icir.org> wrote:

>
>> On Jun 2, 2015, at 10:29 AM, Close, Jason M. <close at ou.edu> wrote:
>> 
>> Our current configuration is showing a lot of heavy use by the master node.  We currently run around 6 worker nodes that feed data to the master, and while the master is keeping up in terms of CPU, it is consistently teetering on using all available RAM we can throw at it (128GB at the moment).
>
>That?s indicating a problem.  I?m going to send you a script off-list that you can run and we?ll see if we can nail down what?s causing that.
>
>> We could piece apart the subnets and create multiple Bro clusters.  But it would be nice to have a single cluster, and be able to continue to throw more workers and managers at it.  But I have not seen any documentation about configurations using multiple managers.  If that does exist, can someone point me in the right direction?
>
>You can only run a single manager.
>
>> But due to the taps we have in place, and the manpower availability, right now, spinning up a little more hardware would be a much easier and more economical investment of our time.
>
>Unfortunately in this case you need to fix the problem and can?t really just throw more hardware at it.
>
>  .Seht
>
>--
>Seth Hall
>International Computer Science Institute
>(Bro) because everyone has a network
>http://www.bro.org/
>
>
>_______________________________________________
>Bro mailing list
>bro at bro-ids.org
>http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From ali at ashemery.com  Fri Jun  5 00:25:32 2015
From: ali at ashemery.com (Ali Hadi)
Date: Fri, 5 Jun 2015 10:25:32 +0300
Subject: [Bro] problem with known-services.bro
In-Reply-To: <23066CB0-ECB4-4083-A2C4-E6FBAFDDEBC4@icir.org>
References: <CY1PR0401MB1387239619368010AECFDD13D7B30@CY1PR0401MB1387.namprd04.prod.outlook.com>
	<23066CB0-ECB4-4083-A2C4-E6FBAFDDEBC4@icir.org>
Message-ID: <CAGG-kCLk7roRW8CP6n3CbBgOW9-XSDn4HXf40--7moAHE+79Xw@mail.gmail.com>

You could also get the file generated by adding the local and the
Site::local_nets  that Seth mentioned; like this:

bro -r file.pcap local "Site::local_nets +={172.16.0.0/16}"
knwon-services.bro

Where 172.16.0.0 is your local network.


Best regards,

*Ali *

On Fri, Jun 5, 2015 at 5:23 AM, Seth Hall <seth at icir.org> wrote:

> > On Jun 4, 2015, at 3:16 PM, Earl Eiland <earl.eiland at root9b.com> wrote:
> >
> > Hello.  I'm running known-services.bro on a pcap file.  I'm having two
> problems:  no known-services.log file is being generated, and some packets
> trigger "internal warning: Unexpected IP version in FragReassembler?.
>
>  By default the known-hosts script only records hosts in your
> Site::local_nets.  You can set that with networks.cfg if you are running
> broctl or you can set it directly in Bro scripts like this...
>
> redef Site::local_nets += { 1.2.3.0/24, 5.6.7.0/24 };
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150605/5eebe9a4/attachment-0001.html 

From close at ou.edu  Fri Jun  5 05:08:05 2015
From: close at ou.edu (Close, Jason M.)
Date: Fri, 5 Jun 2015 12:08:05 +0000
Subject: [Bro] Multiple masters to ease the workload
In-Reply-To: <D90BEFE4-6EE8-42BA-A5C5-7B48A2A24233@pingtrip.com>
References: <D1932800.4751%close@ou.edu>
	<D90BEFE4-6EE8-42BA-A5C5-7B48A2A24233@pingtrip.com>
Message-ID: <D196F8C3.4921%close@ou.edu>

Thanks.

I went ahead and rebooted the cluster, and that cleared things up (as well as sent out a LOT of emails?).

Has anyone else noticed a memory leak in the sensors?  We slowly see memory usage grow, maybe by about 10GB a month, even when our total traffic has gone down.  I attached an image from our Zabbix monitor.  You can see that once we reboot the box, memory drops down, and then slowly creeps up.  And traffic isn?t increasing (in fact, it decreases by half over the summer).

Jason Close
Information Security Analyst
OU Information Technology
Office: 405.325.8661  Cell: 405.421.1096


From: Dave Crawford <dave at pingtrip.com<mailto:dave at pingtrip.com>>
Date: Thursday, June 4, 2015 at 8:35 PM
To: Jason Close <close at ou.edu<mailto:close at ou.edu>>
Cc: "bro at bro.org<mailto:bro at bro.org>" <bro at bro.org<mailto:bro at bro.org>>
Subject: Re: [Bro] Multiple masters to ease the workload

Is it actually 100% RAM usage by applications? Since the manager can be performing a significant amount of disk writes the kernel will allocate ?free? memory as ?cached? to increase file performance. The cached memory is released when applications demand more memory.

Below is the current memory usage on one of my mangers that is handling 25 workers and 2 proxies. At first glance it appears that all the memory has been consumed, but notice how 122G is cached.

             total       used       free     shared    buffers     cached
Mem:          126G       125G       384M         0B       329M       122G
-/+ buffers/cache:       2.6G       123G
Swap:          33G         0B        33G


-Dave

On Jun 2, 2015, at 10:29 AM, Close, Jason M. <close at ou.edu<mailto:close at ou.edu>> wrote:

Our current configuration is showing a lot of heavy use by the master node.  We currently run around 6 worker nodes that feed data to the master, and while the master is keeping up in terms of CPU, it is consistently teetering on using all available RAM we can throw at it (128GB at the moment).  There are plans in place to increase our available bandwidth 10-fold, so the traffic coming to Bro will ramp up as well.

We could piece apart the subnets and create multiple Bro clusters.  But it would be nice to have a single cluster, and be able to continue to throw more workers and managers at it.  But I have not seen any documentation about configurations using multiple managers.  If that does exist, can someone point me in the right direction?

And if that doesn?t exist, can I get some suggestions about mitigations to this problem?  I know there are a lot of cool things being done with Bro, especially using scripts and APIs where Bro can help reduce traffic being thrown to it.  But due to the taps we have in place, and the manpower availability, right now, spinning up a little more hardware would be a much easier and more economical investment of our time.

Thanks.

Jason
_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150605/8d1855ea/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sensor2.png
Type: image/png
Size: 22950 bytes
Desc: sensor2.png
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150605/8d1855ea/attachment-0001.bin 

From dnthayer at illinois.edu  Fri Jun  5 08:05:55 2015
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Fri, 5 Jun 2015 10:05:55 -0500
Subject: [Bro] Broccoli
In-Reply-To: <l7dn5s5xk8y3ruu80mc6byy1.1433340018028@email.android.com>
References: <l7dn5s5xk8y3ruu80mc6byy1.1433340018028@email.android.com>
Message-ID: <5571BAD3.6040303@illinois.edu>

In the broccoli documentation, there is a section "Test programs"
that describes broping specifically:
https://www.bro.org/sphinx/components/broccoli/broccoli-manual.html


On 06/03/2015 09:00 AM, bilal Ahmed wrote:
> Hi, I have problem in broccoli it is already configured but I want to test broping.c
> If anybody know how to test and use that broccoli is now working
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>

From earl.eiland at root9b.com  Fri Jun  5 08:25:14 2015
From: earl.eiland at root9b.com (Earl Eiland)
Date: Fri, 5 Jun 2015 15:25:14 +0000
Subject: [Bro] "services" variable referenced in known-services.bro
Message-ID: <BY1PR0401MB1382B39D5ED7BE31025F8F4FD7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>


Hello.


I'm using known-services to build a list of observed network conversations and the protocols being used.  Known-services detects the TCP conversations, but I want to include UDP conversations as well.  Known-services.bro seems to use a global variable, "services"; services, however, does not hold UDP conversation protocol labels.  I've searched for the script that populates "services", to no avail.


Where do I find that script?  Alternatively, is there a better way to build an all-inclusive conversation vs. protocol mapping?


Best Regards,

Earl Eiland,
Sr. Cyber Security Engineer,
Emerging Technologies, root9B,
San Antonio, Texas

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity named.  If you are not the named addressee you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.  Please notify the sender immediately by email if you received this email in error and delete this email from your system. Any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of root9B LLC.?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150605/d49de371/attachment.html 

From mabuchan at gmail.com  Fri Jun  5 09:09:02 2015
From: mabuchan at gmail.com (Mark Buchanan)
Date: Fri, 5 Jun 2015 11:09:02 -0500
Subject: [Bro] Multiple masters to ease the workload
In-Reply-To: <D196F8C3.4921%close@ou.edu>
References: <D1932800.4751%close@ou.edu>
	<D90BEFE4-6EE8-42BA-A5C5-7B48A2A24233@pingtrip.com>
	<D196F8C3.4921%close@ou.edu>
Message-ID: <CANQDeBMtrx92x9mCnD1-exrt8_AQF-5tpEb-2hGrTp3HY1srsw@mail.gmail.com>

I too have noticed memory complete memory exhaustion in Bro 2.3.2 (not sure
what version Jason is running).  If the workers are not restarted every few
days or at least once a week, I run out of usable memory on a few sensors
I'm testing.

I have found that just doing a restart within broctl will free the memory
consumed up - but I regularly have to perform restarts to keep the sensors
I am testing running smoothly.

Mark

On Fri, Jun 5, 2015 at 7:08 AM, Close, Jason M. <close at ou.edu> wrote:

>  Thanks.
>
>  I went ahead and rebooted the cluster, and that cleared things up (as
> well as sent out a LOT of emails?).
>
>  Has anyone else noticed a memory leak in the sensors?  We slowly see
> memory usage grow, maybe by about 10GB a month, even when our total traffic
> has gone down.  I attached an image from our Zabbix monitor.  You can see
> that once we reboot the box, memory drops down, and then slowly creeps up.
> And traffic isn?t increasing (in fact, it decreases by half over the
> summer).
>
>   *Jason Close*
> *Information Security Analyst*
> OU Information Technology
> Office: 405.325.8661  Cell: 405.421.1096
>
>
>   From: Dave Crawford <dave at pingtrip.com>
> Date: Thursday, June 4, 2015 at 8:35 PM
> To: Jason Close <close at ou.edu>
> Cc: "bro at bro.org" <bro at bro.org>
> Subject: Re: [Bro] Multiple masters to ease the workload
>
>   Is it actually 100% RAM usage by applications? Since the manager can be
> performing a significant amount of disk writes the kernel will allocate
> ?free? memory as ?cached? to increase file performance. The cached memory
> is released when applications demand more memory.
>
>  Below is the current memory usage on one of my mangers that is handling
> 25 workers and 2 proxies. At first glance it appears that all the memory
> has been consumed, but notice how 122G is cached.
>
>               total       used       free     shared    buffers     cached
> Mem:          126G       125G       384M         0B       329M       122G
> -/+ buffers/cache:       2.6G       123G
> Swap:          33G         0B        33G
>
>
>  -Dave
>
>  On Jun 2, 2015, at 10:29 AM, Close, Jason M. <close at ou.edu> wrote:
>
>    Our current configuration is showing a lot of heavy use by the master
> node.  We currently run around 6 worker nodes that feed data to the master,
> and while the master is keeping up in terms of CPU, it is consistently
> teetering on using all available RAM we can throw at it (128GB at the
> moment).  There are plans in place to increase our available bandwidth
> 10-fold, so the traffic coming to Bro will ramp up as well.
>
>  We could piece apart the subnets and create multiple Bro clusters.  But
> it would be nice to have a single cluster, and be able to continue to throw
> more workers and managers at it.  But I have not seen any documentation
> about configurations using multiple managers.  If that does exist, can
> someone point me in the right direction?
>
>  And if that doesn?t exist, can I get some suggestions about mitigations
> to this problem?  I know there are a lot of cool things being done with
> Bro, especially using scripts and APIs where Bro can help reduce traffic
> being thrown to it.  But due to the taps we have in place, and the manpower
> availability, right now, spinning up a little more hardware would be a much
> easier and more economical investment of our time.
>
>  Thanks.
>
>  Jason
>   _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 
Mark Buchanan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150605/a6669b56/attachment.html 

From hosom at battelle.org  Fri Jun  5 09:40:23 2015
From: hosom at battelle.org (Hosom, Stephen M)
Date: Fri, 5 Jun 2015 16:40:23 +0000
Subject: [Bro] Multiple masters to ease the workload
In-Reply-To: <CANQDeBMtrx92x9mCnD1-exrt8_AQF-5tpEb-2hGrTp3HY1srsw@mail.gmail.com>
References: <D1932800.4751%close@ou.edu>
	<D90BEFE4-6EE8-42BA-A5C5-7B48A2A24233@pingtrip.com>
	<D196F8C3.4921%close@ou.edu>
	<CANQDeBMtrx92x9mCnD1-exrt8_AQF-5tpEb-2hGrTp3HY1srsw@mail.gmail.com>
Message-ID: <E5A2071C6FFFF640B714C224139C9B3301A9A599@WP-MBX2B.milky-way.battelle.org>

You?ll both want to check reporter.log. In most cases memory leaks are introduced due to scripts (either built-in or custom) that error. Errors within bro script land can result in memory leaks, so you want to do your best to avoid those. If you?re willing to share your reporter.log, I could possibly help you fix some of the errors that you?re running into.

From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Mark Buchanan
Sent: Friday, June 05, 2015 12:09 PM
To: Close, Jason M.
Cc: bro at bro.org; Dave Crawford
Subject: Re: [Bro] Multiple masters to ease the workload

I too have noticed memory complete memory exhaustion in Bro 2.3.2 (not sure what version Jason is running).  If the workers are not restarted every few days or at least once a week, I run out of usable memory on a few sensors I'm testing.

I have found that just doing a restart within broctl will free the memory consumed up - but I regularly have to perform restarts to keep the sensors I am testing running smoothly.

Mark

On Fri, Jun 5, 2015 at 7:08 AM, Close, Jason M. <close at ou.edu<mailto:close at ou.edu>> wrote:
Thanks.

I went ahead and rebooted the cluster, and that cleared things up (as well as sent out a LOT of emails?).

Has anyone else noticed a memory leak in the sensors?  We slowly see memory usage grow, maybe by about 10GB a month, even when our total traffic has gone down.  I attached an image from our Zabbix monitor.  You can see that once we reboot the box, memory drops down, and then slowly creeps up.  And traffic isn?t increasing (in fact, it decreases by half over the summer).

Jason Close
Information Security Analyst
OU Information Technology
Office: 405.325.8661<tel:405.325.8661>  Cell: 405.421.1096<tel:405.421.1096>


From: Dave Crawford <dave at pingtrip.com<mailto:dave at pingtrip.com>>
Date: Thursday, June 4, 2015 at 8:35 PM
To: Jason Close <close at ou.edu<mailto:close at ou.edu>>
Cc: "bro at bro.org<mailto:bro at bro.org>" <bro at bro.org<mailto:bro at bro.org>>
Subject: Re: [Bro] Multiple masters to ease the workload

Is it actually 100% RAM usage by applications? Since the manager can be performing a significant amount of disk writes the kernel will allocate ?free? memory as ?cached? to increase file performance. The cached memory is released when applications demand more memory.

Below is the current memory usage on one of my mangers that is handling 25 workers and 2 proxies. At first glance it appears that all the memory has been consumed, but notice how 122G is cached.

             total       used       free     shared    buffers     cached
Mem:          126G       125G       384M         0B       329M       122G
-/+ buffers/cache:       2.6G       123G
Swap:          33G         0B        33G


-Dave

On Jun 2, 2015, at 10:29 AM, Close, Jason M. <close at ou.edu<mailto:close at ou.edu>> wrote:

Our current configuration is showing a lot of heavy use by the master node.  We currently run around 6 worker nodes that feed data to the master, and while the master is keeping up in terms of CPU, it is consistently teetering on using all available RAM we can throw at it (128GB at the moment).  There are plans in place to increase our available bandwidth 10-fold, so the traffic coming to Bro will ramp up as well.

We could piece apart the subnets and create multiple Bro clusters.  But it would be nice to have a single cluster, and be able to continue to throw more workers and managers at it.  But I have not seen any documentation about configurations using multiple managers.  If that does exist, can someone point me in the right direction?

And if that doesn?t exist, can I get some suggestions about mitigations to this problem?  I know there are a lot of cool things being done with Bro, especially using scripts and APIs where Bro can help reduce traffic being thrown to it.  But due to the taps we have in place, and the manpower availability, right now, spinning up a little more hardware would be a much easier and more economical investment of our time.

Thanks.

Jason
_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



--
Mark Buchanan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150605/3d440db2/attachment-0001.html 

From mabuchan at gmail.com  Fri Jun  5 10:26:34 2015
From: mabuchan at gmail.com (Mark Buchanan)
Date: Fri, 5 Jun 2015 12:26:34 -0500
Subject: [Bro] Multiple masters to ease the workload
In-Reply-To: <E5A2071C6FFFF640B714C224139C9B3301A9A599@WP-MBX2B.milky-way.battelle.org>
References: <D1932800.4751%close@ou.edu>
	<D90BEFE4-6EE8-42BA-A5C5-7B48A2A24233@pingtrip.com>
	<D196F8C3.4921%close@ou.edu>
	<CANQDeBMtrx92x9mCnD1-exrt8_AQF-5tpEb-2hGrTp3HY1srsw@mail.gmail.com>
	<E5A2071C6FFFF640B714C224139C9B3301A9A599@WP-MBX2B.milky-way.battelle.org>
Message-ID: <CANQDeBOq1Da39=c_WWr-O26J+T-bF-8sF71hkft1k50rTpw3wA@mail.gmail.com>

I'm using all stock bro scripts in the test I have going - but adding some
intel indicators.

The most recurring message I have is for "NB-DNS error in DNS_Mgr::Process
(recvfrom(): Connection refused)".  This machine does not have DNS access
and the build we used put a DNS server that is not in service into the
/etc/resolv.conf.  This error is about 90% of what is in my reporter.log.
I tried to comment out the /etc/resolv.conf entry and restart bro through
broctl, but am still seeing the issues.

The other significant percentage are misc base64 messages:

"incomplete base64 group, padding with <n> bits of 0" - ~5%
"extra base64 groups after '=' padding are ignored" - ~4%
"character <n> ignored by Base64 decoding" - ~< 1%

Mark






On Fri, Jun 5, 2015 at 11:40 AM, Hosom, Stephen M <hosom at battelle.org>
wrote:

>  You?ll both want to check reporter.log. In most cases memory leaks are
> introduced due to scripts (either built-in or custom) that error. Errors
> within bro script land can result in memory leaks, so you want to do your
> best to avoid those. If you?re willing to share your reporter.log, I could
> possibly help you fix some of the errors that you?re running into.
>
>
>
> *From:* bro-bounces at bro.org [mailto:bro-bounces at bro.org] *On Behalf Of *Mark
> Buchanan
> *Sent:* Friday, June 05, 2015 12:09 PM
> *To:* Close, Jason M.
> *Cc:* bro at bro.org; Dave Crawford
>
> *Subject:* Re: [Bro] Multiple masters to ease the workload
>
>
>
> I too have noticed memory complete memory exhaustion in Bro 2.3.2 (not
> sure what version Jason is running).  If the workers are not restarted
> every few days or at least once a week, I run out of usable memory on a few
> sensors I'm testing.
>
>
>
> I have found that just doing a restart within broctl will free the memory
> consumed up - but I regularly have to perform restarts to keep the sensors
> I am testing running smoothly.
>
>
>
> Mark
>
>
>
> On Fri, Jun 5, 2015 at 7:08 AM, Close, Jason M. <close at ou.edu> wrote:
>
>   Thanks.
>
>
>
> I went ahead and rebooted the cluster, and that cleared things up (as well
> as sent out a LOT of emails?).
>
>
>
> Has anyone else noticed a memory leak in the sensors?  We slowly see
> memory usage grow, maybe by about 10GB a month, even when our total traffic
> has gone down.  I attached an image from our Zabbix monitor.  You can see
> that once we reboot the box, memory drops down, and then slowly creeps up.
> And traffic isn?t increasing (in fact, it decreases by half over the
> summer).
>
>
>
> *Jason Close*
>
> *I**nformation Security Analyst*
>
> OU Information Technology
>
> Office: 405.325.8661  Cell: 405.421.1096
>
>
>
>
>
> *From: *Dave Crawford <dave at pingtrip.com>
> *Date: *Thursday, June 4, 2015 at 8:35 PM
> *To: *Jason Close <close at ou.edu>
> *Cc: *"bro at bro.org" <bro at bro.org>
> *Subject: *Re: [Bro] Multiple masters to ease the workload
>
>
>
> Is it actually 100% RAM usage by applications? Since the manager can be
> performing a significant amount of disk writes the kernel will allocate
> ?free? memory as ?cached? to increase file performance. The cached memory
> is released when applications demand more memory.
>
>
>
> Below is the current memory usage on one of my mangers that is handling 25
> workers and 2 proxies. At first glance it appears that all the memory has
> been consumed, but notice how 122G is cached.
>
>
>
>              total       used       free     shared    buffers     cached
>
> Mem:          126G       125G       384M         0B       329M       122G
>
> -/+ buffers/cache:       2.6G       123G
>
> Swap:          33G         0B        33G
>
>
>
>
>
> -Dave
>
>
>
>  On Jun 2, 2015, at 10:29 AM, Close, Jason M. <close at ou.edu> wrote:
>
>
>
> Our current configuration is showing a lot of heavy use by the master
> node.  We currently run around 6 worker nodes that feed data to the master,
> and while the master is keeping up in terms of CPU, it is consistently
> teetering on using all available RAM we can throw at it (128GB at the
> moment).  There are plans in place to increase our available bandwidth
> 10-fold, so the traffic coming to Bro will ramp up as well.
>
>
>
> We could piece apart the subnets and create multiple Bro clusters.  But it
> would be nice to have a single cluster, and be able to continue to throw
> more workers and managers at it.  But I have not seen any documentation
> about configurations using multiple managers.  If that does exist, can
> someone point me in the right direction?
>
>
>
> And if that doesn?t exist, can I get some suggestions about mitigations to
> this problem?  I know there are a lot of cool things being done with Bro,
> especially using scripts and APIs where Bro can help reduce traffic being
> thrown to it.  But due to the taps we have in place, and the manpower
> availability, right now, spinning up a little more hardware would be a much
> easier and more economical investment of our time.
>
>
>
> Thanks.
>
>
>
> Jason
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
>
> --
>
> Mark Buchanan
>



-- 
Mark Buchanan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150605/92f6835d/attachment.html 

From bro at pingtrip.com  Fri Jun  5 10:40:07 2015
From: bro at pingtrip.com (Dave Crawford)
Date: Fri, 5 Jun 2015 13:40:07 -0400
Subject: [Bro] Multiple masters to ease the workload
In-Reply-To: <CANQDeBMtrx92x9mCnD1-exrt8_AQF-5tpEb-2hGrTp3HY1srsw@mail.gmail.com>
References: <D1932800.4751%close@ou.edu>
	<D90BEFE4-6EE8-42BA-A5C5-7B48A2A24233@pingtrip.com>
	<D196F8C3.4921%close@ou.edu>
	<CANQDeBMtrx92x9mCnD1-exrt8_AQF-5tpEb-2hGrTp3HY1srsw@mail.gmail.com>
Message-ID: <920C4BDB-B538-40C0-8402-6E4E63072AE6@pingtrip.com>


When I recently debugged memory exhaustion on my workers the root cause was related to the software detection scripts, specifically /protocols/http/software. If that script is running on a sensor that is monitoring the inside interface of a web proxy it tracks all the remote software as being on your local network.


> On Jun 5, 2015, at 12:09 PM, Mark Buchanan <mabuchan at gmail.com> wrote:
> 
> I too have noticed memory complete memory exhaustion in Bro 2.3.2 (not sure what version Jason is running).  If the workers are not restarted every few days or at least once a week, I run out of usable memory on a few sensors I'm testing.
>  
> I have found that just doing a restart within broctl will free the memory consumed up - but I regularly have to perform restarts to keep the sensors I am testing running smoothly.
>  
> Mark
> 
>> On Fri, Jun 5, 2015 at 7:08 AM, Close, Jason M. <close at ou.edu> wrote:
>> Thanks. 
>> 
>> I went ahead and rebooted the cluster, and that cleared things up (as well as sent out a LOT of emails?).
>> 
>> Has anyone else noticed a memory leak in the sensors?  We slowly see memory usage grow, maybe by about 10GB a month, even when our total traffic has gone down.  I attached an image from our Zabbix monitor.  You can see that once we reboot the box, memory drops down, and then slowly creeps up.  And traffic isn?t increasing (in fact, it decreases by half over the summer).
>> 
>> Jason Close
>> Information Security Analyst
>> OU Information Technology
>> Office: 405.325.8661  Cell: 405.421.1096
>> 
>> 
>> From: Dave Crawford <dave at pingtrip.com>
>> Date: Thursday, June 4, 2015 at 8:35 PM
>> To: Jason Close <close at ou.edu>
>> Cc: "bro at bro.org" <bro at bro.org>
>> Subject: Re: [Bro] Multiple masters to ease the workload
>> 
>> Is it actually 100% RAM usage by applications? Since the manager can be performing a significant amount of disk writes the kernel will allocate ?free? memory as ?cached? to increase file performance. The cached memory is released when applications demand more memory.
>> 
>> Below is the current memory usage on one of my mangers that is handling 25 workers and 2 proxies. At first glance it appears that all the memory has been consumed, but notice how 122G is cached.
>> 
>>              total       used       free     shared    buffers     cached
>> Mem:          126G       125G       384M         0B       329M       122G
>> -/+ buffers/cache:       2.6G       123G
>> Swap:          33G         0B        33G
>> 
>> 
>> -Dave
>> 
>>> On Jun 2, 2015, at 10:29 AM, Close, Jason M. <close at ou.edu> wrote:
>>> 
>>> Our current configuration is showing a lot of heavy use by the master node.  We currently run around 6 worker nodes that feed data to the master, and while the master is keeping up in terms of CPU, it is consistently teetering on using all available RAM we can throw at it (128GB at the moment).  There are plans in place to increase our available bandwidth 10-fold, so the traffic coming to Bro will ramp up as well. 
>>> 
>>> We could piece apart the subnets and create multiple Bro clusters.  But it would be nice to have a single cluster, and be able to continue to throw more workers and managers at it.  But I have not seen any documentation about configurations using multiple managers.  If that does exist, can someone point me in the right direction?
>>> 
>>> And if that doesn?t exist, can I get some suggestions about mitigations to this problem?  I know there are a lot of cool things being done with Bro, especially using scripts and APIs where Bro can help reduce traffic being thrown to it.  But due to the taps we have in place, and the manpower availability, right now, spinning up a little more hardware would be a much easier and more economical investment of our time.
>>> 
>>> Thanks.
>>> 
>>> Jason
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> 
>> 
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 
> 
> -- 
> Mark Buchanan
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150605/33e2c93b/attachment-0001.html 

From seth at icir.org  Fri Jun  5 11:20:38 2015
From: seth at icir.org (Seth Hall)
Date: Fri, 5 Jun 2015 14:20:38 -0400
Subject: [Bro] "services" variable referenced in known-services.bro
In-Reply-To: <BY1PR0401MB1382B39D5ED7BE31025F8F4FD7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>
References: <BY1PR0401MB1382B39D5ED7BE31025F8F4FD7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>
Message-ID: <00D227E5-A97D-4E29-84F1-D220890524B0@icir.org>


> On Jun 5, 2015, at 11:25 AM, Earl Eiland <earl.eiland at root9b.com> wrote:
> 
> 
> I'm using known-services to build a list of observed network conversations and the protocols being used.  Known-services detects the TCP conversations, but I want to include UDP conversations as well.  Known-services.bro seems to use a global variable, "services?;

The service field is a component of DPD (dynamic protocol detection) and the analyzer code in general.  You can find the script that actually populates that field here though:
	https://github.com/bro/bro/blob/master/scripts/base/protocols/conn/main.bro#L182

That?s going to be a little misleading though because it?s just pulling data from deeper in the connection record into the log.  The real story is that this is done as part of DPD where protocols are guessed at with signatures and then a parser is attached.  Once the parser positively confirms that the protocol is in fact that protocol that the signature matched then it will indicate the service.

The right way to think about the service field is to think of it as an indicator that a connection was successfully analyzed by a particular analyzer.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150605/32e14f16/attachment.bin 

From gfaulkner.nsm at gmail.com  Fri Jun  5 11:21:04 2015
From: gfaulkner.nsm at gmail.com (Gary Faulkner)
Date: Fri, 05 Jun 2015 13:21:04 -0500
Subject: [Bro] Multiple masters to ease the workload
In-Reply-To: <920C4BDB-B538-40C0-8402-6E4E63072AE6@pingtrip.com>
References: <D1932800.4751%close@ou.edu>	<D90BEFE4-6EE8-42BA-A5C5-7B48A2A24233@pingtrip.com>	<D196F8C3.4921%close@ou.edu>	<CANQDeBMtrx92x9mCnD1-exrt8_AQF-5tpEb-2hGrTp3HY1srsw@mail.gmail.com>
	<920C4BDB-B538-40C0-8402-6E4E63072AE6@pingtrip.com>
Message-ID: <5571E890.9000707@gmail.com>

I used to see some stability issues on a reasonably large network (100K 
plus hosts) that appeared related to software asset tracking, coupled 
with a lot of IP churn due to wireless devices and short lease times, 
which resulted in the manager being oom-killed after some number of 
hours or days. It was suggested, I think by Seth, that I experiment with 
disabling it in local.bro:

I'm still on 2.3-419, so not the latest build by any means, but was able 
to test this, and have seen greatly improved stability in my particular 
environment, with the below line in local.bro:

redef Software::asset_tracking=NO_HOSTS;

It might be worth experimenting with if you think you may be having 
software asset tracking related issues. Be mindful if you rely on that 
data for other scripts.

Regards,
Gary

On 6/5/2015 12:40 PM, Dave Crawford wrote:
>
> When I recently debugged memory exhaustion on my workers the root 
> cause was related to the software detection scripts, specifically 
> /protocols/http/software. If that script is running on a sensor that 
> is monitoring the inside interface of a web proxy it tracks all the 
> remote software as being on your local network.
>
>
> On Jun 5, 2015, at 12:09 PM, Mark Buchanan <mabuchan at gmail.com 
> <mailto:mabuchan at gmail.com>> wrote:
>
>> I too have noticed memory complete memory exhaustion in Bro 2.3.2 
>> (not sure what version Jason is running).  If the workers are not 
>> restarted every few days or at least once a week, I run out of usable 
>> memory on a few sensors I'm testing.
>> I have found that just doing a restart within broctl will free the 
>> memory consumed up - but I regularly have to perform restarts to keep 
>> the sensors I am testing running smoothly.
>> Mark
>>
>> On Fri, Jun 5, 2015 at 7:08 AM, Close, Jason M. <close at ou.edu 
>> <mailto:close at ou.edu>> wrote:
>>
>>     Thanks.
>>
>>     I went ahead and rebooted the cluster, and that cleared things up
>>     (as well as sent out a LOT of emails?).
>>
>>     Has anyone else noticed a memory leak in the sensors?  We slowly
>>     see memory usage grow, maybe by about 10GB a month, even when our
>>     total traffic has gone down.  I attached an image from our Zabbix
>>     monitor.  You can see that once we reboot the box, memory drops
>>     down, and then slowly creeps up.  And traffic isn?t increasing
>>     (in fact, it decreases by half over the summer).
>>
>>     *Jason Close*
>>     *Information Security Analyst*
>>     OU Information Technology
>>     Office: 405.325.8661 <tel:405.325.8661> Cell: 405.421.1096
>>     <tel:405.421.1096>
>>
>>
>>     From: Dave Crawford <dave at pingtrip.com <mailto:dave at pingtrip.com>>
>>     Date: Thursday, June 4, 2015 at 8:35 PM
>>     To: Jason Close <close at ou.edu <mailto:close at ou.edu>>
>>     Cc: "bro at bro.org <mailto:bro at bro.org>" <bro at bro.org
>>     <mailto:bro at bro.org>>
>>     Subject: Re: [Bro] Multiple masters to ease the workload
>>
>>     Is it actually 100% RAM usage by applications? Since the manager
>>     can be performing a significant amount of disk writes the kernel
>>     will allocate ?free? memory as ?cached? to increase file
>>     performance. The cached memory is released when applications
>>     demand more memory.
>>
>>     Below is the current memory usage on one of my mangers that is
>>     handling 25 workers and 2 proxies. At first glance it appears
>>     that all the memory has been consumed, but notice how 122G is cached.
>>
>>                  total     used       free     shared    buffers    
>>     cached
>>     Mem:          126G     125G       384M         0B       329M    
>>       122G
>>     -/+ buffers/cache:     2.6G       123G
>>     Swap:          33G       0B        33G
>>
>>
>>     -Dave
>>
>>>     On Jun 2, 2015, at 10:29 AM, Close, Jason M. <close at ou.edu
>>>     <mailto:close at ou.edu>> wrote:
>>>
>>>     Our current configuration is showing a lot of heavy use by the
>>>     master node.  We currently run around 6 worker nodes that feed
>>>     data to the master, and while the master is keeping up in terms
>>>     of CPU, it is consistently teetering on using all available RAM
>>>     we can throw at it (128GB at the moment).  There are plans in
>>>     place to increase our available bandwidth 10-fold, so the
>>>     traffic coming to Bro will ramp up as well.
>>>
>>>     We could piece apart the subnets and create multiple Bro
>>>     clusters. But it would be nice to have a single cluster, and be
>>>     able to continue to throw more workers and managers at it.  But
>>>     I have not seen any documentation about configurations using
>>>     multiple managers.  If that does exist, can someone point me in
>>>     the right direction?
>>>
>>>     And if that doesn?t exist, can I get some suggestions about
>>>     mitigations to this problem?  I know there are a lot of cool
>>>     things being done with Bro, especially using scripts and APIs
>>>     where Bro can help reduce traffic being thrown to it.  But due
>>>     to the taps we have in place, and the manpower availability,
>>>     right now, spinning up a little more hardware would be a much
>>>     easier and more economical investment of our time.
>>>
>>>     Thanks.
>>>
>>>     Jason
>>>     _______________________________________________
>>>     Bro mailing list
>>>     bro at bro-ids.org <mailto:bro at bro-ids.org>
>>>     http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>>     _______________________________________________
>>     Bro mailing list
>>     bro at bro-ids.org <mailto:bro at bro-ids.org>
>>     http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>>
>>
>> -- 
>> Mark Buchanan
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org <mailto:bro at bro-ids.org>
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150605/8d7b6cf9/attachment-0001.html 

From earl.eiland at root9b.com  Fri Jun  5 11:46:25 2015
From: earl.eiland at root9b.com (Earl Eiland)
Date: Fri, 5 Jun 2015 18:46:25 +0000
Subject: [Bro] "services" variable referenced in known-services.bro
In-Reply-To: <00D227E5-A97D-4E29-84F1-D220890524B0@icir.org>
References: <BY1PR0401MB1382B39D5ED7BE31025F8F4FD7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>,
	<00D227E5-A97D-4E29-84F1-D220890524B0@icir.org>
Message-ID: <BY1PR0401MB1382045CA4AFD4197AE3113DD7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>

Thanks, Seth.

That helps a lot.  When I run DPD, the various logs show that traffic is being correctly parsed.  It seems that the information should appear in conn.log's service column, particularly when DPD is invoked from the command line.  This, however, is not the case.   What am I overlooking?

Best Regards,

Earl Eiland,
Sr. Cyber Security Engineer,
Emerging Technologies, root9B,
San Antonio, Texas

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity named.  If you are not the named addressee you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.  Please notify the sender immediately by email if you received this email in error and delete this email from your system. Any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of root9B LLC.?

________________________________________
From: Seth Hall <seth at icir.org>
Sent: Friday, June 5, 2015 1:20 PM
To: Earl Eiland
Cc: bro at bro.org
Subject: Re: [Bro] "services" variable referenced in known-services.bro

> On Jun 5, 2015, at 11:25 AM, Earl Eiland <earl.eiland at root9b.com> wrote:
>
>
> I'm using known-services to build a list of observed network conversations and the protocols being used.  Known-services detects the TCP conversations, but I want to include UDP conversations as well.  Known-services.bro seems to use a global variable, "services?;

The service field is a component of DPD (dynamic protocol detection) and the analyzer code in general.  You can find the script that actually populates that field here though:
        https://github.com/bro/bro/blob/master/scripts/base/protocols/conn/main.bro#L182

That?s going to be a little misleading though because it?s just pulling data from deeper in the connection record into the log.  The real story is that this is done as part of DPD where protocols are guessed at with signatures and then a parser is attached.  Once the parser positively confirms that the protocol is in fact that protocol that the signature matched then it will indicate the service.

The right way to think about the service field is to think of it as an indicator that a connection was successfully analyzed by a particular analyzer.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/


From seth at icir.org  Fri Jun  5 11:59:36 2015
From: seth at icir.org (Seth Hall)
Date: Fri, 5 Jun 2015 14:59:36 -0400
Subject: [Bro] "services" variable referenced in known-services.bro
In-Reply-To: <BY1PR0401MB1382045CA4AFD4197AE3113DD7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>
References: <BY1PR0401MB1382B39D5ED7BE31025F8F4FD7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>
	<00D227E5-A97D-4E29-84F1-D220890524B0@icir.org>
	<BY1PR0401MB1382045CA4AFD4197AE3113DD7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>
Message-ID: <B2946DBB-2E62-489D-9362-3500CC8AEA1B@icir.org>


> On Jun 5, 2015, at 2:46 PM, Earl Eiland <earl.eiland at root9b.com> wrote:
> 
> That helps a lot.  When I run DPD, the various logs show that traffic is being correctly parsed.  It seems that the information should appear in conn.log's service column, particularly when DPD is invoked from the command line.  This, however, is not the case.   What am I overlooking?

Could you show a little more concretely how you?re running Bro?  Ideally you could provide a pcap that shows what you?re seeing although I understand if you?re unable to do that.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150605/54bd6bb1/attachment.bin 

From nb.nospam at gmail.com  Fri Jun  5 14:53:28 2015
From: nb.nospam at gmail.com (N B)
Date: Fri, 5 Jun 2015 14:53:28 -0700
Subject: [Bro] HTTPS Analyzer
Message-ID: <CA+rfBVw3-nAYzJu4Ovi37HE9K1TmxrnGsYRRyWnOV8R7vTnyHQ@mail.gmail.com>

Hello,

I am quite new to Bro and need some help. I did go through some of the
documentation and some source code but still not clear whether its possible
to achieve what we are trying to do.

In a nutshell, we are trying to write an HTTPS analyzer for on the fly
decryption of the SSL stream and then feed it to the built in HTTP
Analyzer. We will use a crypto library + server keys to achieve the
decryption. Is it possible at all do this in Bro?

The high level idea is to derive the HTTPS_Analyzer from the current
HTTP_Analyzer, feed the stream from TCP_Analyzer into the HTTPS_Analyzer
and utilize the HTTP_Analyzer calls for the remainder of the functionality.

Thanks for your help,
NB
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150605/11f2f60f/attachment.html 

From johanna at icir.org  Fri Jun  5 15:46:47 2015
From: johanna at icir.org (Johanna Amann)
Date: Fri, 5 Jun 2015 15:46:47 -0700
Subject: [Bro] HTTPS Analyzer
In-Reply-To: <CA+rfBVw3-nAYzJu4Ovi37HE9K1TmxrnGsYRRyWnOV8R7vTnyHQ@mail.gmail.com>
References: <CA+rfBVw3-nAYzJu4Ovi37HE9K1TmxrnGsYRRyWnOV8R7vTnyHQ@mail.gmail.com>
Message-ID: <20150605224647.GA27348@Beezling.local>

Hello,

> In a nutshell, we are trying to write an HTTPS analyzer for on the fly
> decryption of the SSL stream and then feed it to the built in HTTP
> Analyzer. We will use a crypto library + server keys to achieve the
> decryption. Is it possible at all do this in Bro?

Sure, in theory it is possible to do that. You would have to extend the
current SSL analyzer and start decrypting the packets at the right point
of time. You should not even have to implement an HTTPS analyzer; you
basically can just shove the decrypted data back into the Bro processing
pipeline.

The best example for this happening might potentially be one of the tunnel
analyzers -- SMTP also does it by attaching SSL as a sub-analyzer in case
STARTTLS is used.

The biggest problem will probably be to get the SSL analyzer changed to
decrypt the data. You also will have to get your encryption keys into Bro
somehow before the first encrypted data packet is parsed by the SSL
analyzer.

Johanna

From bilal.comsian09 at gmail.com  Fri Jun  5 23:19:01 2015
From: bilal.comsian09 at gmail.com (bilal Ahmed)
Date: Sat, 06 Jun 2015 11:19:01 +0500
Subject: [Bro] Broccoli
Message-ID: <u0trkkj7ufkwpdx15xxi3s0f.1433571541532@email.android.com>

I followed that documentation but when I run broping.c file foe making object file it show error related to missing of functions like bro_init() and like that
Regards
Bilal

 Daniel Thayer <dnthayer at illinois.edu> wrote:

>In the broccoli documentation, there is a section "Test programs"
>that describes broping specifically:
>https://www.bro.org/sphinx/components/broccoli/broccoli-manual.html
>
>
>On 06/03/2015 09:00 AM, bilal Ahmed wrote:
>> Hi, I have problem in broccoli it is already configured but I want to test broping.c
>> If anybody know how to test and use that broccoli is now working
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>


From f.eskandary2009 at gmail.com  Sat Jun  6 14:03:54 2015
From: f.eskandary2009 at gmail.com (Fateme Eskandari)
Date: Sun, 7 Jun 2015 01:33:54 +0430
Subject: [Bro] Fwd: saving raw packet payload to text file
In-Reply-To: <mailman.15776.1433624430.949.bro@bro.org>
References: <mailman.15776.1433624430.949.bro@bro.org>
Message-ID: <CADsr06y50w-TVGc_LQAGdAZDPnmcrQd138P4LGW_Ba8hfLSa2w@mail.gmail.com>

dear all
i have a pcap file that contain data about some protocols. i want to have a
text file for every protocol from my pcap file that contains all raws of
packet payload in asccii format. just like this:
which command could i use?
please guide me
thanks a lot

GET / HTTP/1.1
Host: www.google.com
Connection: keep-alive
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/40.0.2311.90 Safari/537.36
Referer: https://www.google.com/
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8,fa;q=0.6,nl;q=0.4,de;q=0.2

HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Date: Tue, 21 Apr 2010 2:38:54 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
X-Powered-By: PHP/5.5.9-1ubuntu4.9
Set-Cookie: PHPSESSID=agg92fl57l5si815a03tr269h1; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0,
pre-check=0
Pragma: no-cache
Set-Cookie: testcookie=1
Content-Encoding: gzip


-- 

Fateme Eskandari

Research Assistant at Network Security Research Group

Iran University Of Science and Technology
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150607/7d3c53c9/attachment.html 

From krkhan at inspirated.com  Sun Jun  7 12:15:31 2015
From: krkhan at inspirated.com (Kamran Khan)
Date: Sun, 7 Jun 2015 12:15:31 -0700
Subject: [Bro] Bro port on OpenWRT
Message-ID: <CAN0j2-doQ8eJR+=R69BZOTgSYoKF1anb8Tik6n1U9Jk6s=fRtQ@mail.gmail.com>

Hi folks,

I just updated my port of Bro 2.3.1 on the OpenWRT router and am now
maintaining it in a GitHub repository along with compiled ipk packages.

Repo: https://github.com/krkhan/openwrt-bro
Compiled ipk Packages: https://github.com/krkhan/openwrt-bro/releases

Thanks,
--
Kamran Riaz Khan.

http://inspirated.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150607/e372aac8/attachment.html 

From earl.eiland at root9b.com  Mon Jun  8 04:36:45 2015
From: earl.eiland at root9b.com (Earl Eiland)
Date: Mon, 8 Jun 2015 11:36:45 +0000
Subject: [Bro] Fw:  "services" variable referenced in known-services.bro
In-Reply-To: <BY1PR0401MB138299A9831350134E74EC10D7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>
References: <BY1PR0401MB1382B39D5ED7BE31025F8F4FD7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>
	<00D227E5-A97D-4E29-84F1-D220890524B0@icir.org>
	<BY1PR0401MB1382045CA4AFD4197AE3113DD7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>,
	<B2946DBB-2E62-489D-9362-3500CC8AEA1B@icir.org>,
	<BY1PR0401MB138299A9831350134E74EC10D7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>
Message-ID: <BY1PR0401MB1382684B1759CFACEAB36B90D7BF0@BY1PR0401MB1382.namprd04.prod.outlook.com>


Unfortunately, we don't have permission to share our test data.  However, it is an industrial control system (ICS).  Most of the traffic is MODBUS, although we expect to be deploying the detector on ICSs using DNP3 and IEC 61850.  In addition to MODBUS, our test data also has some TDS and HTTP traffic, and of course, the usual network management traffic (DNS, ICMP, DHCP, etc.).

ICS traffic tends to be quite regular.  Our goal is to develop a detector that can flag a conversation using an anomalous communication protocol.

Best Regards,

Earl Eiland,
Sr. Cyber Security Engineer,
Emerging Technologies, root9B,
San Antonio, Texas

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity named.  If you are not the named addressee you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.  Please notify the sender immediately by email if you received this email in error and delete this email from your system. Any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of root9B LLC.?

________________________________________
From: Seth Hall <seth at icir.org>
Sent: Friday, June 5, 2015 1:59 PM
To: Earl Eiland
Cc: bro at bro.org
Subject: Re: [Bro] "services" variable referenced in known-services.bro

> On Jun 5, 2015, at 2:46 PM, Earl Eiland <earl.eiland at root9b.com> wrote:
>
> That helps a lot.  When I run DPD, the various logs show that traffic is being correctly parsed.  It seems that the information should appear in conn.log's service column, particularly when DPD is invoked from the command line.  This, however, is not the case.   What am I overlooking?

Could you show a little more concretely how you?re running Bro?  Ideally you could provide a pcap that shows what you?re seeing although I understand if you?re unable to do that.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/


From hohoangphi at gmail.com  Mon Jun  8 05:09:35 2015
From: hohoangphi at gmail.com (Phiho Hoang)
Date: Mon, 8 Jun 2015 08:09:35 -0400
Subject: [Bro] Bro port on OpenWRT
In-Reply-To: <CAN0j2-doQ8eJR+=R69BZOTgSYoKF1anb8Tik6n1U9Jk6s=fRtQ@mail.gmail.com>
References: <CAN0j2-doQ8eJR+=R69BZOTgSYoKF1anb8Tik6n1U9Jk6s=fRtQ@mail.gmail.com>
Message-ID: <CALZQg1a41xcODfj39QmNd-UMdzyUOEiR4g-+YoYGQ-MUYbVx3w@mail.gmail.com>

Greetings,

Thank you for releasing and maintaining Bro 2.3.1 on the OpenWRT.

Do you have any recommendation for a suitable (best bang for the buck ;-)
device to use with these packages.

Best regards,

PhiHo


On Sun, Jun 7, 2015 at 3:15 PM, Kamran Khan <krkhan at inspirated.com> wrote:

> Hi folks,
>
> I just updated my port of Bro 2.3.1 on the OpenWRT router and am now
> maintaining it in a GitHub repository along with compiled ipk packages.
>
> Repo: https://github.com/krkhan/openwrt-bro
> Compiled ipk Packages: https://github.com/krkhan/openwrt-bro/releases
>
> Thanks,
> --
> Kamran Riaz Khan.
>
> http://inspirated.com/
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150608/414e94a8/attachment.html 

From abhall1 at yahoo.com  Mon Jun  8 07:01:53 2015
From: abhall1 at yahoo.com (Adam Hall)
Date: Mon, 8 Jun 2015 14:01:53 +0000 (UTC)
Subject: [Bro] disable_stream vs remove_filter
In-Reply-To: <mailman.1.1433703602.9830.bro@bro.org>
References: <mailman.1.1433703602.9830.bro@bro.org>
Message-ID: <1459783901.8638993.1433772113445.JavaMail.yahoo@mail.yahoo.com>

I am trying to figure out if there are any pro's or con's using disable_stream vs remove_filter.
>From the reading they appear as if they are interchangeable... but I want to make sure that one of them doesn't have negative interplay.
Background of the event is:
I have Log::remove_filter(Files::LOG,"default"); and created my own log where I only store certain mime_types.I decided also that I wanted to remove other log files (weird, dpd, modbus, communication, known_*, PacketFilter) that aren't being used when research is performed.I have commented out some in the local.bro but need to either disable_stream or remove_filter the rest.Also, I currently have 2.2 and 2.3 running. ?I am using 2.4 for testing and then figuring backwards compatibility.
I am looking for:
1) Will one give me a performance gain over the other?2) Will one cause problems for other calls being made (If I disable_stream and something calls that stream will it break)?3) If I disable a stream and later decide to add a new filter, will that work?
I am still testing some of this, but any help would greatly appreciated!
Thanks,
Adam "RedLight" Hall

  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150608/41f1f244/attachment.html 

From earl.eiland at root9b.com  Mon Jun  8 08:06:12 2015
From: earl.eiland at root9b.com (Earl Eiland)
Date: Mon, 8 Jun 2015 15:06:12 +0000
Subject: [Bro] "services" variable referenced in known-services.bro
In-Reply-To: <BY1PR0401MB138299A9831350134E74EC10D7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>
References: <BY1PR0401MB1382B39D5ED7BE31025F8F4FD7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>
	<00D227E5-A97D-4E29-84F1-D220890524B0@icir.org>
	<BY1PR0401MB1382045CA4AFD4197AE3113DD7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>,
	<B2946DBB-2E62-489D-9362-3500CC8AEA1B@icir.org>,
	<BY1PR0401MB138299A9831350134E74EC10D7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>
Message-ID: <BY1PR0401MB13820727864EFAFCB5A12B59D7BF0@BY1PR0401MB1382.namprd04.prod.outlook.com>

Hello, Seth.

the services field in conn.log does report http and dns, so I looked in their sripts for references to the services variable.  There doesn't appear to be one, so how is the services variable populated?

Best Regards,

Earl Eiland,
Sr. Cyber Security Engineer,
Emerging Technologies, root9B,
San Antonio, Texas

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity named.  If you are not the named addressee you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.  Please notify the sender immediately by email if you received this email in error and delete this email from your system. Any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of root9B LLC.?

________________________________________
From: Earl Eiland
Sent: Friday, June 5, 2015 2:21 PM
To: Seth Hall
Subject: Re: [Bro] "services" variable referenced in known-services.bro

Unfortunately, we don't have permission to share our test data.  However, it is an industrial control system (ICS).  Most of the traffic is MODBUS, although we expect to be deploying the detector on ICSs using DNP3 and IEC 61850.  In addition to MODBUS, our test data also has some TDS and HTTP traffic, and of course, the usual network management traffic (DNS, ICMP, DHCP, etc.).

ICS traffic tends to be quite regular.  Our goal is to develop a detector that can flag a conversation using an anomalous communication protocol.

Best Regards,

Earl Eiland,
Sr. Cyber Security Engineer,
Emerging Technologies, root9B,
San Antonio, Texas

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity named.  If you are not the named addressee you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.  Please notify the sender immediately by email if you received this email in error and delete this email from your system. Any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of root9B LLC.?

________________________________________
From: Seth Hall <seth at icir.org>
Sent: Friday, June 5, 2015 1:59 PM
To: Earl Eiland
Cc: bro at bro.org
Subject: Re: [Bro] "services" variable referenced in known-services.bro

> On Jun 5, 2015, at 2:46 PM, Earl Eiland <earl.eiland at root9b.com> wrote:
>
> That helps a lot.  When I run DPD, the various logs show that traffic is being correctly parsed.  It seems that the information should appear in conn.log's service column, particularly when DPD is invoked from the command line.  This, however, is not the case.   What am I overlooking?

Could you show a little more concretely how you?re running Bro?  Ideally you could provide a pcap that shows what you?re seeing although I understand if you?re unable to do that.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/


From charles.fair at mac.com  Mon Jun  8 10:31:12 2015
From: charles.fair at mac.com (Charles A. Fair)
Date: Mon, 08 Jun 2015 13:31:12 -0400
Subject: [Bro] "services" variable referenced in known-services.bro
 [email signature block legal disclaimer]
In-Reply-To: <BY1PR0401MB13820727864EFAFCB5A12B59D7BF0@BY1PR0401MB1382.namprd04.prod.outlook.com>
References: <BY1PR0401MB1382B39D5ED7BE31025F8F4FD7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>
	<00D227E5-A97D-4E29-84F1-D220890524B0@icir.org>
	<BY1PR0401MB1382045CA4AFD4197AE3113DD7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>
	<B2946DBB-2E62-489D-9362-3500CC8AEA1B@icir.org>
	<BY1PR0401MB138299A9831350134E74EC10D7B20@BY1PR0401MB1382.namprd04.prod.outlook.com>
	<BY1PR0401MB13820727864EFAFCB5A12B59D7BF0@BY1PR0401MB1382.namprd04.prod.outlook.com>
Message-ID: <E48DD77E-E8AD-492B-B22A-33443A96AD0A@mac.com>


> 
> Earl Eiland,
> Sr. Cyber Security Engineer,
> Emerging Technologies, root9B,
> San Antonio, Texas
> 
> This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity named.  If you are not the named addressee you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.  Please notify the sender immediately by email if you received this email in error and delete this email from your system. Any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of root9B LLC.?
> 

The use of email signature block legal disclaimers is a bit disturbing on a public mailing list.  

Chuck

From jdopheid at illinois.edu  Mon Jun  8 11:14:39 2015
From: jdopheid at illinois.edu (Dopheide, Jeannette M)
Date: Mon, 8 Jun 2015 18:14:39 +0000
Subject: [Bro] BroCon '15: Agenda updates
Message-ID: <D19B45BF.16965%jdopheid@illinois.edu>

Bro Community,

The BroCon agenda has been updated to include the approved CFPs:
https://www.bro.org/community/brocon2015.html#agenda


Speaker abstracts are linked to the presentation names.

If you haven't registered yet, you may do so here:
https://www.regonline.com/brocon2015

This year's conference marks a noteworthy balance in the number of
community and developer presentations. Thank you for your continued
support and enthusiasm for the Bro Project.


See you in August,


The Bro Team



------

Jeannette Dopheide
Bro Outreach Coordinator
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign






From nb.nospam at gmail.com  Mon Jun  8 13:30:12 2015
From: nb.nospam at gmail.com (N B)
Date: Mon, 8 Jun 2015 13:30:12 -0700
Subject: [Bro] HTTPS Analyzer
In-Reply-To: <20150605224647.GA27348@Beezling.local>
References: <CA+rfBVw3-nAYzJu4Ovi37HE9K1TmxrnGsYRRyWnOV8R7vTnyHQ@mail.gmail.com>
	<20150605224647.GA27348@Beezling.local>
Message-ID: <CA+rfBVx2ktWudBYEQzuoyriOgbe9j1gUQaXw2H=DRwvJ6MHTog@mail.gmail.com>

Thanks Johanna. Much appreciated for the suggestion of extending the SSL
analyzer.

> "you basically can just shove the decrypted data back into the Bro
processing pipeline."

I am assuming that by above you mean to just call the "ForwardStream()"
method? Please confirm if that's the case.

> "The biggest problem will probably be to get the SSL analyzer changed to
> decrypt the data. You also will have to get your encryption keys into Bro
> somehow before the first encrypted data packet is parsed by the SSL
> analyzer."

Getting the key loaded via the new class's constructor or as a static
initialized value won't be enough? Maybe I missed something important here.
Can you please clarify?

Thanks
Nikunj



On Fri, Jun 5, 2015 at 3:46 PM, Johanna Amann <johanna at icir.org> wrote:

> Hello,
>
> > In a nutshell, we are trying to write an HTTPS analyzer for on the fly
> > decryption of the SSL stream and then feed it to the built in HTTP
> > Analyzer. We will use a crypto library + server keys to achieve the
> > decryption. Is it possible at all do this in Bro?
>
> Sure, in theory it is possible to do that. You would have to extend the
> current SSL analyzer and start decrypting the packets at the right point
> of time. You should not even have to implement an HTTPS analyzer; you
> basically can just shove the decrypted data back into the Bro processing
> pipeline.
>
> The best example for this happening might potentially be one of the tunnel
> analyzers -- SMTP also does it by attaching SSL as a sub-analyzer in case
> STARTTLS is used.
>
> The biggest problem will probably be to get the SSL analyzer changed to
> decrypt the data. You also will have to get your encryption keys into Bro
> somehow before the first encrypted data packet is parsed by the SSL
> analyzer.
>
> Johanna
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150608/b1aff896/attachment.html 

From giedrius.ramas at gmail.com  Mon Jun  8 22:27:43 2015
From: giedrius.ramas at gmail.com (Giedrius Ramas)
Date: Tue, 9 Jun 2015 08:27:43 +0300
Subject: [Bro] BRO 2.3.2 Intel email indicator do not work
In-Reply-To: <CAHr6V5pNOmSW55+MHruMtgFDe44pZNpMGxa8EtjgQj8-HSNM3A@mail.gmail.com>
References: <CAHr6V5pNOmSW55+MHruMtgFDe44pZNpMGxa8EtjgQj8-HSNM3A@mail.gmail.com>
Message-ID: <CAHr6V5qDUCuH9aEGp3RzurEesEexFgC5w0+6zKZTWv+9MEV9Hw@mail.gmail.com>

anyone faced the same issue ?

On Tue, Jun 2, 2015 at 9:09 AM, Giedrius Ramas <giedrius.ramas at gmail.com>
wrote:

> Hi,
>
> I found that BRO 2.3.4 Intel do not work with email's indicators. I have
> played on my infrastructure to get BRO intel work and found that email
> indicator won't work.
>
> I also tested it on  try.bro.org/ the same results . However BRO 2.2
> version works well with Intel email's indicators .
>
> Please let me know if more details needed to troubleshoot
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150609/663d92ec/attachment.html 

From johanna at icir.org  Tue Jun  9 11:24:30 2015
From: johanna at icir.org (Johanna Amann)
Date: Tue, 9 Jun 2015 11:24:30 -0700
Subject: [Bro] Bro v2.4 release
Message-ID: <20150609182422.GA87375@Beezling.dhcp.lbnl.us>

Hello,

Bro v2.4 just has been released. For more information, see the blog post
at http://blog.bro.org/2015/06/bro-24-released.html as well as the release
notes at https://www.bro.org/download/NEWS.bro.html

Bro is available for download at https://www.bro.org/download/

Thanks to everyone who helped make this release possible.  We extend
special thanks to the community for their feedback and support.

The Bro Team

From earl.eiland at root9b.com  Tue Jun  9 16:05:15 2015
From: earl.eiland at root9b.com (Earl Eiland)
Date: Tue, 9 Jun 2015 23:05:15 +0000
Subject: [Bro] monitoring node conversations vs. communications protocols
Message-ID: <BY1PR0401MB1382180831A1D79BD2E8068BD7BE0@BY1PR0401MB1382.namprd04.prod.outlook.com>

I've been scouring the bro scripts, technical papers, etc., to determine how to map and monitor node conversations vs. communications (service) protocols on a network.   I could use information in conn.log, if the services column was fully populated.  Unfortunately, it doesn't appear that the services variable entered in the conn.log comes from the protocol detection scripts.  I'm new to bro, so it's quite possible I've missed something, but it is looking like I may have to modify the bro source code. Please advise!


Best Regards,

Earl Eiland,
Sr. Cyber Security Engineer,
Emerging Technologies, root9B,
San Antonio, Texas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150609/ced10c4f/attachment.html 

From bilal.comsian09 at gmail.com  Wed Jun 10 00:46:42 2015
From: bilal.comsian09 at gmail.com (bilal ahmed)
Date: Wed, 10 Jun 2015 12:46:42 +0500
Subject: [Bro] Broccoli connection with bro
Message-ID: <CANxtSkbV8ktFvB-6ELBAdn08ghEmb0He5mUQLn7aQAh+YxURAQ@mail.gmail.com>

Hi, I have configured the bro in /nsm/bro/bin and broccoli is also
installed and I did changes in the broccoli.conf file but I'm not able to
connect to bro when I test the broconn or broping it is not able connect to
bro on 127.0.0.1:47758.
I also checked the port 47757 but not able to ping bro.
Then I checked the broping.bro by running it through
 /nsm/bro/bin/bro broping.bro
but it always and didnt show any error or any pong response.
where I'm wrong?? Did I missed something??

Regards
Bilal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150610/958b9925/attachment.html 

From vitologrillo at gmail.com  Wed Jun 10 02:07:37 2015
From: vitologrillo at gmail.com (Vito Logrillo)
Date: Wed, 10 Jun 2015 11:07:37 +0200
Subject: [Bro] Problems with our hardware
Message-ID: <CANdYiFGTfySLo24saA2fiijcoB1f2GWFb5PRAHMNYNYtFAn5sg@mail.gmail.com>

Hi all,
i've made some tests using an hardware with these features:

* 2x Intel(R) Xeon(R) CPU E5320  @ 1.86GHz
* 8 GB RAM
* 8x 72GB disks 10000 rpm scsi
* Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)

Bro was configured in this way:

* Pf_ring aware drivers
* Bro 1.4 beta
* 2 workers

  [manager]
  type=manager
  host=localhost
  #
  [proxy-1]
  type=proxy
  host=localhost
  #
  [worker-1]
  type=worker
  host=localhost
  interface=eth1
  lb_method=pf_ring
  lb_procs=4
  #
  [worker-2]
  type=worker
  host=localhost
  interface=eth1
  lb_method=pf_ring
  lb_procs=4

I've used tcpreplay to send for 1 minute a known stream at full
bandwith:all data sent was fully analyzed by Bro only after 2hours!!

On this link
https://www.bro.org/sphinx-git/cluster/index.html
you have reported this rule of thumb

"The rule of thumb we have followed recently is to allocate
approximately 1 core for every 80Mbps of traffic that is being
analyzed. However, this estimate could be extremely traffic
mix-specific. It has generally worked for mixed traffic with many
users and servers. For example, if your traffic peaks around 2Gbps
(combined) and you want to handle traffic at peak load, you may want
to have 26 cores available (2048 / 80 == 25.6). If the 80Mbps estimate
works for your traffic, this could be handled by 3 physical hosts
dedicated to being workers with each one containing dual 6-core
processors."

Using your rule of thumb, Bro should complete the process in 2 minutes
(more or less) and not two hours: what's wrong? Any suggestion?
Regards,
Vito

From dopheide at gmail.com  Wed Jun 10 02:53:34 2015
From: dopheide at gmail.com (Mike Dopheide)
Date: Wed, 10 Jun 2015 04:53:34 -0500
Subject: [Bro] BRO 2.3.2 Intel email indicator do not work
In-Reply-To: <CAHr6V5qDUCuH9aEGp3RzurEesEexFgC5w0+6zKZTWv+9MEV9Hw@mail.gmail.com>
References: <CAHr6V5pNOmSW55+MHruMtgFDe44pZNpMGxa8EtjgQj8-HSNM3A@mail.gmail.com>
	<CAHr6V5qDUCuH9aEGp3RzurEesEexFgC5w0+6zKZTWv+9MEV9Hw@mail.gmail.com>
Message-ID: <CAPy2kFbxneNuzrRB3xoHNQZOte2vFo7UKt6TPiaKfm0YcAndeQ@mail.gmail.com>

If nobody gets back to you sooner, I'll have time to test later this week
if you hit me up then.  In the meantime, I'd suggest testing with 2.4 that
was just released.

Dop

On Tuesday, June 9, 2015, Giedrius Ramas <giedrius.ramas at gmail.com> wrote:

> anyone faced the same issue ?
>
> On Tue, Jun 2, 2015 at 9:09 AM, Giedrius Ramas <giedrius.ramas at gmail.com
> <javascript:_e(%7B%7D,'cvml','giedrius.ramas at gmail.com');>> wrote:
>
>> Hi,
>>
>> I found that BRO 2.3.4 Intel do not work with email's indicators. I have
>> played on my infrastructure to get BRO intel work and found that email
>> indicator won't work.
>>
>> I also tested it on  try.bro.org/ the same results . However BRO 2.2
>> version works well with Intel email's indicators .
>>
>> Please let me know if more details needed to troubleshoot
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150610/0bc60bc1/attachment.html 

From pch66 at cornell.edu  Wed Jun 10 14:27:18 2015
From: pch66 at cornell.edu (Peter Hansen)
Date: Wed, 10 Jun 2015 17:27:18 -0400
Subject: [Bro] DPD with BinPAC++
Message-ID: <CACPag=3poQ1U+BA1yqei-joDYNNwao1BCc6NeUWDj5HDq9rJAQ@mail.gmail.com>

Hello,

I am currently working with BinPAC++ to write detectors for various
protocols, and I am attempting to use Dynamic Protocol Detection in them,
but I cannot find documentation on how to implement it. I can however, find
references to being able to use DPD analyzers with BinPAC++, so I know it
is possible. Does anyone have insight onto how to use this?

Thanks,
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150610/dbc0c56a/attachment.html 

From robin at icir.org  Wed Jun 10 14:53:09 2015
From: robin at icir.org (Robin Sommer)
Date: Wed, 10 Jun 2015 14:53:09 -0700
Subject: [Bro] DPD with BinPAC++
In-Reply-To: <CACPag=3poQ1U+BA1yqei-joDYNNwao1BCc6NeUWDj5HDq9rJAQ@mail.gmail.com>
References: <CACPag=3poQ1U+BA1yqei-joDYNNwao1BCc6NeUWDj5HDq9rJAQ@mail.gmail.com>
Message-ID: <20150610215309.GD68577@icir.org>



On Wed, Jun 10, 2015 at 17:27 -0400, Peter Hansen wrote:

> I am currently working with BinPAC++ to write detectors for various
> protocols, and I am attempting to use Dynamic Protocol Detection in them,
> but I cannot find documentation on how to implement it.

There's a function to call at the time you consider the protocol
detected: Bro::dpd_confirm(). See bro/pac2/http.pac2 for an example.

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From jlay at slave-tothe-box.net  Wed Jun 10 17:02:39 2015
From: jlay at slave-tothe-box.net (James Lay)
Date: Wed, 10 Jun 2015 18:02:39 -0600
Subject: [Bro] Fixing deprecated scripts
Message-ID: <1433980959.11347.3.camel@JamesiMac>

Topic says it...I have the excellent smtp-embedded-url-bloom.bro which I
use almost every day for looking up links.  Upon upgrading to 2.4 I see:

error in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro,
line 40: value used but not set (bloomfilter_basic_init)
warning in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro,
line 73: deprecated (split)

I read in the release notes that split was going away.  So now, what do
I use to fix it?  I'm including this in the email as I'm not able to
find it online.  Thank you.

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150610/5f4dd69c/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/octet-stream
Size: 7265 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150610/5f4dd69c/attachment.obj 

From nb.nospam at gmail.com  Wed Jun 10 17:03:29 2015
From: nb.nospam at gmail.com (N B)
Date: Wed, 10 Jun 2015 17:03:29 -0700
Subject: [Bro] HTTPS Analyzer
In-Reply-To: <CA+rfBVx2ktWudBYEQzuoyriOgbe9j1gUQaXw2H=DRwvJ6MHTog@mail.gmail.com>
References: <CA+rfBVw3-nAYzJu4Ovi37HE9K1TmxrnGsYRRyWnOV8R7vTnyHQ@mail.gmail.com>
	<20150605224647.GA27348@Beezling.local>
	<CA+rfBVx2ktWudBYEQzuoyriOgbe9j1gUQaXw2H=DRwvJ6MHTog@mail.gmail.com>
Message-ID: <CA+rfBVxujGH1femD6DdrZxHG8FvmaC0jGWF0S4x---dhMZct_A@mail.gmail.com>

Hi Johanna (and everyone else on the list),

I am currently struggling with this as to how to put the decrypted data
back into the Bro pipeline? I am able to get the data decrypted (actually
its just a test with a simple xor data into it and xor it back in the
analyzer) in my analyzer and calling ForwardStream() with the new data and
length. I have checked and double checked that everything looks like it
should be i.e. the resulting stream is HTTP data (headers, content etc) but
for some reason the HTTP analyzer does not get invoked. Please help.

Thanks
Nikunj


On Mon, Jun 8, 2015 at 1:30 PM, N B <nb.nospam at gmail.com> wrote:

> Thanks Johanna. Much appreciated for the suggestion of extending the SSL
> analyzer.
>
> > "you basically can just shove the decrypted data back into the Bro
> processing pipeline."
>
> I am assuming that by above you mean to just call the "ForwardStream()"
> method? Please confirm if that's the case.
>
> > "The biggest problem will probably be to get the SSL analyzer changed to
> > decrypt the data. You also will have to get your encryption keys into Bro
> > somehow before the first encrypted data packet is parsed by the SSL
> > analyzer."
>
> Getting the key loaded via the new class's constructor or as a static
> initialized value won't be enough? Maybe I missed something important here.
> Can you please clarify?
>
> Thanks
> Nikunj
>
>
>
> On Fri, Jun 5, 2015 at 3:46 PM, Johanna Amann <johanna at icir.org> wrote:
>
>> Hello,
>>
>> > In a nutshell, we are trying to write an HTTPS analyzer for on the fly
>> > decryption of the SSL stream and then feed it to the built in HTTP
>> > Analyzer. We will use a crypto library + server keys to achieve the
>> > decryption. Is it possible at all do this in Bro?
>>
>> Sure, in theory it is possible to do that. You would have to extend the
>> current SSL analyzer and start decrypting the packets at the right point
>> of time. You should not even have to implement an HTTPS analyzer; you
>> basically can just shove the decrypted data back into the Bro processing
>> pipeline.
>>
>> The best example for this happening might potentially be one of the tunnel
>> analyzers -- SMTP also does it by attaching SSL as a sub-analyzer in case
>> STARTTLS is used.
>>
>> The biggest problem will probably be to get the SSL analyzer changed to
>> decrypt the data. You also will have to get your encryption keys into Bro
>> somehow before the first encrypted data packet is parsed by the SSL
>> analyzer.
>>
>> Johanna
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150610/4e4ed21c/attachment.html 

From vlad at grigorescu.org  Wed Jun 10 18:56:46 2015
From: vlad at grigorescu.org (Vlad Grigorescu)
Date: Wed, 10 Jun 2015 20:56:46 -0500
Subject: [Bro] Fixing deprecated scripts
In-Reply-To: <1433980959.11347.3.camel@JamesiMac>
References: <1433980959.11347.3.camel@JamesiMac>
Message-ID: <CANUJ_Fk5i3VZfAEny1+1-7GssyPNkyjA=eHwnaFY2NoZT6OPJg@mail.gmail.com>

See: https://github.com/bro/bro/blob/v2.4/NEWS#L238

On Wed, Jun 10, 2015 at 7:02 PM, James Lay <jlay at slave-tothe-box.net> wrote:

>  Topic says it...I have the excellent smtp-embedded-url-bloom.bro which I
> use almost every day for looking up links.  Upon upgrading to 2.4 I see:
>
> error in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro, line
> 40: value used but not set (bloomfilter_basic_init)
> warning in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro,
> line 73: deprecated (split)
>
> I read in the release notes that split was going away.  So now, what do I
> use to fix it?  I'm including this in the email as I'm not able to find it
> online.  Thank you.
>
> James
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150610/8c706eac/attachment-0001.html 

From jlay at slave-tothe-box.net  Wed Jun 10 19:55:38 2015
From: jlay at slave-tothe-box.net (James Lay)
Date: Wed, 10 Jun 2015 20:55:38 -0600
Subject: [Bro] Fixing deprecated scripts
In-Reply-To: <B26E9991-DFDF-4170-AEB4-0B7003129DCC@cwalsh.org>
References: <1433980959.11347.3.camel@JamesiMac>
	<B26E9991-DFDF-4170-AEB4-0B7003129DCC@cwalsh.org>
Message-ID: <1433991338.3690.2.camel@JamesiMac>

On Wed, 2015-06-10 at 20:07 -0500, Chris Walsh wrote:

> From https://www.bro.org/documentation/beta/NEWS.bro.html, under Deprecated Functionality:
> 
> 
> 	? split: use split_string instead.
> 
> I just did this in another script  2 minutes before your mail hit my mailbox :^)
> 
> cw
> On Jun 10, 2015, at 7:02 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> 
> > Topic says it...I have the excellent smtp-embedded-url-bloom.bro which I use almost every day for looking up links.  Upon upgrading to 2.4 I see:
> > 
> > error in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro, line 40: value used but not set (bloomfilter_basic_init)
> > warning in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro, line 73: deprecated (split)
> > 
> > I read in the release notes that split was going away.  So now, what do I use to fix it?  I'm including this in the email as I'm not able to find it online.  Thank you.


Thanks Chris and Vlad, changing split_string fixed the warning, but did
not fix the error at line 40 which halts bro from running.  Any reason
why:

global mail_links = bloomfilter_basic_init(0.00000001, 10000000) ;

causes this issue?  This worked just fine with 2.3.2.  Thank you.

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150610/b8371d43/attachment.html 

From hosom at battelle.org  Thu Jun 11 04:35:14 2015
From: hosom at battelle.org (Hosom, Stephen M)
Date: Thu, 11 Jun 2015 11:35:14 +0000
Subject: [Bro] Fixing deprecated scripts
In-Reply-To: <1433980959.11347.3.camel@JamesiMac>
References: <1433980959.11347.3.camel@JamesiMac>
Message-ID: <E5A2071C6FFFF640B714C224139C9B3301A9D19B@WP-MBX2B.milky-way.battelle.org>

You most likely want the bif split_string

https://www.bro.org/sphinx/scripts/base/bif/strings.bif.bro.html#id-split_string


From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of James Lay
Sent: Wednesday, June 10, 2015 8:03 PM
To: Bro-IDS
Subject: [Bro] Fixing deprecated scripts

Topic says it...I have the excellent smtp-embedded-url-bloom.bro which I use almost every day for looking up links.  Upon upgrading to 2.4 I see:

error in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro, line 40: value used but not set (bloomfilter_basic_init)
warning in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro, line 73: deprecated (split)

I read in the release notes that split was going away.  So now, what do I use to fix it?  I'm including this in the email as I'm not able to find it online.  Thank you.

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150611/14757867/attachment.html 

From jlay at slave-tothe-box.net  Thu Jun 11 05:41:51 2015
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 11 Jun 2015 06:41:51 -0600
Subject: [Bro] Fixing deprecated scripts
In-Reply-To: <1433991338.3690.2.camel@JamesiMac>
References: <1433980959.11347.3.camel@JamesiMac>
	<B26E9991-DFDF-4170-AEB4-0B7003129DCC@cwalsh.org>
	<1433991338.3690.2.camel@JamesiMac>
Message-ID: <1434026511.3690.13.camel@JamesiMac>

On Wed, 2015-06-10 at 20:55 -0600, James Lay wrote:

> On Wed, 2015-06-10 at 20:07 -0500, Chris Walsh wrote: 
> 
> > >From https://www.bro.org/documentation/beta/NEWS.bro.html, under Deprecated Functionality:
> > 
> > 
> > 	? split: use split_string instead.
> > 
> > I just did this in another script  2 minutes before your mail hit my mailbox :^)
> > 
> > cw
> > On Jun 10, 2015, at 7:02 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> > 
> > > Topic says it...I have the excellent smtp-embedded-url-bloom.bro which I use almost every day for looking up links.  Upon upgrading to 2.4 I see:
> > > 
> > > error in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro, line 40: value used but not set (bloomfilter_basic_init)
> > > warning in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro, line 73: deprecated (split)
> > > 
> > > I read in the release notes that split was going away.  So now, what do I use to fix it?  I'm including this in the email as I'm not able to find it online.  Thank you.
> 
> 
> Thanks Chris and Vlad, changing split_string fixed the warning, but
> did not fix the error at line 40 which halts bro from running.  Any
> reason why:
> 
> global mail_links = bloomfilter_basic_init(0.00000001, 10000000) ;
> 
> causes this issue?  This worked just fine with 2.3.2.  Thank you.
> 
> James 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


Looks like this showed up last year:

http://mailman.icsi.berkeley.edu/pipermail/bro/2014-October/007662.html

Is there something I need to do on my end?  Thank you.,

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150611/9724ea2d/attachment.html 

From lysemose at gmail.com  Thu Jun 11 06:06:00 2015
From: lysemose at gmail.com (Heine Lysemose)
Date: Thu, 11 Jun 2015 15:06:00 +0200
Subject: [Bro] Documentation for bro-cut...
Message-ID: <CAN4C-Dm2683XODgYer-0EUWUkfU_sVb-9wJBo6aPuCb19OYpjA@mail.gmail.com>

Hi

I noticed that the documentation looks a bit weird...
https://www.bro.org/sphinx/logs/index.html

[image: Inline image 1]

Regards,
Lysemsoe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150611/13ef44b8/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 37769 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150611/13ef44b8/attachment-0001.bin 

From dnthayer at illinois.edu  Thu Jun 11 08:05:47 2015
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Thu, 11 Jun 2015 10:05:47 -0500
Subject: [Bro] Documentation for bro-cut...
In-Reply-To: <CAN4C-Dm2683XODgYer-0EUWUkfU_sVb-9wJBo6aPuCb19OYpjA@mail.gmail.com>
References: <CAN4C-Dm2683XODgYer-0EUWUkfU_sVb-9wJBo6aPuCb19OYpjA@mail.gmail.com>
Message-ID: <5579A3CB.9050705@illinois.edu>

Thanks for reporting this.  Not sure how this happened, but
for now you can just look at the "Development" edition
of our documentation:
https://www.bro.org/sphinx-git/logs/index.html



On 06/11/2015 08:06 AM, Heine Lysemose wrote:
> Hi
>
> I noticed that the documentation looks a bit weird...
> https://www.bro.org/sphinx/logs/index.html
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.bro.org_sphinx_logs_index.html&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Bi5qPBnY0NmYPqnRTPj_AfXQKpfQTZUpCzpfFBcawv0&m=8A71gDqh1QKiBoZMgCkJnTB1R6XTVR25t9YDpLJQbBs&s=IRfFt67fPklTMDvWAiO50zljmGwFshWcRsTUAZbcDgk&e=>
>
> Inline image 1
>
> Regards,
> Lysemsoe

From lysemose at gmail.com  Thu Jun 11 09:36:53 2015
From: lysemose at gmail.com (Heine Lysemose)
Date: Thu, 11 Jun 2015 18:36:53 +0200
Subject: [Bro] Documentation for bro-cut...
In-Reply-To: <5579A3CB.9050705@illinois.edu>
References: <CAN4C-Dm2683XODgYer-0EUWUkfU_sVb-9wJBo6aPuCb19OYpjA@mail.gmail.com>
	<5579A3CB.9050705@illinois.edu>
Message-ID: <CAN4C-D=xupvMneu+GBwtqzRg7H7R=cXQwR7Y+8F-k_f8vDLY7Q@mail.gmail.com>

Hi Daniel

No problem.

Thanks,
Lysemose
On Jun 11, 2015 5:08 PM, "Daniel Thayer" <dnthayer at illinois.edu> wrote:

> Thanks for reporting this.  Not sure how this happened, but
> for now you can just look at the "Development" edition
> of our documentation:
> https://www.bro.org/sphinx-git/logs/index.html
>
>
>
> On 06/11/2015 08:06 AM, Heine Lysemose wrote:
>
>> Hi
>>
>> I noticed that the documentation looks a bit weird...
>> https://www.bro.org/sphinx/logs/index.html
>> <
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.bro.org_sphinx_logs_index.html&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Bi5qPBnY0NmYPqnRTPj_AfXQKpfQTZUpCzpfFBcawv0&m=8A71gDqh1QKiBoZMgCkJnTB1R6XTVR25t9YDpLJQbBs&s=IRfFt67fPklTMDvWAiO50zljmGwFshWcRsTUAZbcDgk&e=
>> >
>>
>> Inline image 1
>>
>> Regards,
>> Lysemsoe
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150611/31785f3c/attachment.html 

From donaldson8 at llnl.gov  Fri Jun 12 07:56:49 2015
From: donaldson8 at llnl.gov (Donaldson, John)
Date: Fri, 12 Jun 2015 14:56:49 +0000
Subject: [Bro] Disabling interface packet messages?
Message-ID: <D1A05D68.6661%donaldson8@llnl.gov>

Is there any way to disable the ?X is not seeing any packets on interface
Y?/?X is seeing packets again on interface Y? messages that run with the
scheduled Cron jobs? I?m monitoring a link which, naturally, has extended
periods without any traffic, and these alerts clutter up my inbox and
don?t bring me much value.

John Donaldson






From pch66 at cornell.edu  Fri Jun 12 08:12:03 2015
From: pch66 at cornell.edu (Peter Hansen)
Date: Fri, 12 Jun 2015 11:12:03 -0400
Subject: [Bro] DPD with BinPAC++
In-Reply-To: <20150610215309.GD68577@icir.org>
References: <CACPag=3poQ1U+BA1yqei-joDYNNwao1BCc6NeUWDj5HDq9rJAQ@mail.gmail.com>
	<20150610215309.GD68577@icir.org>
Message-ID: <CACPag=2NEAcOxZZ4LLYNKQAJVqU2dwUM+4P=N=CVq7_OsPt8wg@mail.gmail.com>

Hello, and thank you for your answer.

I think I have gotten it working except for the fact that my detector only
triggers on the specific type of traffic I am attempting to track, even if
the different types of data are on the same port, but for some reason it
only works when I specify one or more ports, and when I leave the port
blank, it doesn't detect it at all. Is there a way to specify that it
should listen on all ports?

Thanks,
Peter

On Wed, Jun 10, 2015 at 5:53 PM, Robin Sommer <robin at icir.org> wrote:

>
>
> On Wed, Jun 10, 2015 at 17:27 -0400, Peter Hansen wrote:
>
> > I am currently working with BinPAC++ to write detectors for various
> > protocols, and I am attempting to use Dynamic Protocol Detection in them,
> > but I cannot find documentation on how to implement it.
>
> There's a function to call at the time you consider the protocol
> detected: Bro::dpd_confirm(). See bro/pac2/http.pac2 for an example.
>
> Robin
>
> --
> Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150612/e0476c3c/attachment.html 

From dnthayer at illinois.edu  Fri Jun 12 09:19:59 2015
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Fri, 12 Jun 2015 11:19:59 -0500
Subject: [Bro] Disabling interface packet messages?
In-Reply-To: <D1A05D68.6661%donaldson8@llnl.gov>
References: <D1A05D68.6661%donaldson8@llnl.gov>
Message-ID: <557B06AF.5050907@illinois.edu>

You can set this in your broctl.cfg file:

StatsLogEnable=0

That will turn off those emails, but it will also turn off
the functionality related to this (such as writing to the
spool/stats.log file).



On 06/12/2015 09:56 AM, Donaldson, John wrote:
> Is there any way to disable the ?X is not seeing any packets on interface
> Y?/?X is seeing packets again on interface Y? messages that run with the
> scheduled Cron jobs? I?m monitoring a link which, naturally, has extended
> periods without any traffic, and these alerts clutter up my inbox and
> don?t bring me much value.
>
> John Donaldson
>
>
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>

From asharma at lbl.gov  Fri Jun 12 09:58:01 2015
From: asharma at lbl.gov (Aashish Sharma)
Date: Fri, 12 Jun 2015 09:58:01 -0700
Subject: [Bro] Fixing deprecated scripts
In-Reply-To: <1434026511.3690.13.camel@JamesiMac>
References: <1433980959.11347.3.camel@JamesiMac>
	<B26E9991-DFDF-4170-AEB4-0B7003129DCC@cwalsh.org>
	<1433991338.3690.2.camel@JamesiMac>
	<1434026511.3690.13.camel@JamesiMac>
Message-ID: <20150612165800.GC3273@yaksha.lbl.gov>

Sorry for catching up on this late. 

I take you got script working now. If not, let me know, I will send a revised version. 

Thanks, 
Aashish 

On Thu, Jun 11, 2015 at 06:41:51AM -0600, James Lay wrote:
> 
>    On Wed, 2015-06-10 at 20:55 -0600, James Lay wrote:
> 
>      On Wed, 2015-06-10 at 20:07 -0500, Chris Walsh wrote:
> 
> >From [1]https://www.bro.org/documentation/beta/NEWS.bro.html, under Deprecated
>  Functionality:
> 
> 
>         ? split: use split_string instead.
> 
> I just did this in another script  2 minutes before your mail hit my mailbox :^
> )
> 
> cw
> On Jun 10, 2015, at 7:02 PM, James Lay <[2]jlay at slave-tothe-box.net> wrote:
> 
> > Topic says it...I have the excellent smtp-embedded-url-bloom.bro which I use
> almost every day for looking up links.  Upon upgrading to 2.4 I see:
> >
> > error in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro, line 40
> : value used but not set (bloomfilter_basic_init)
> > warning in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro, line
> 73: deprecated (split)
> >
> > I read in the release notes that split was going away.  So now, what do I use
>  to fix it?  I'm including this in the email as I'm not able to find it online.
>   Thank you.
> 
>      Thanks Chris and Vlad, changing split_string fixed the warning, but did
>      not fix the error at line 40 which halts bro from running.  Any reason
>      why:
>      global mail_links = bloomfilter_basic_init(0.00000001, 10000000) ;
>      causes this issue?  This worked just fine with 2.3.2.  Thank you.
>      James
> _______________________________________________
> Bro mailing list
> [3]bro at bro-ids.org
> [4]http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
>    Looks like this showed up last year:
>    [5]http://mailman.icsi.berkeley.edu/pipermail/bro/2014-October/007662.html
>    Is there something I need to do on my end?  Thank you.,
>    James
> 
> References
> 
>    1. https://www.bro.org/documentation/beta/NEWS.bro.html
>    2. mailto:jlay at slave-tothe-box.net
>    3. mailto:bro at bro-ids.org
>    4. http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>    5. http://mailman.icsi.berkeley.edu/pipermail/bro/2014-October/007662.html

> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


-- 
Aashish Sharma	(asharma at lbl.gov) 				 
Cyber Security, 
Lawrence Berkeley National Laboratory  
http://go.lbl.gov/pgp-aashish 
Office: (510)-495-2680  Cell: (510)-612-7971

From jlay at slave-tothe-box.net  Fri Jun 12 10:05:28 2015
From: jlay at slave-tothe-box.net (James Lay)
Date: Fri, 12 Jun 2015 11:05:28 -0600
Subject: [Bro] Fixing deprecated scripts
In-Reply-To: <20150612165800.GC3273@yaksha.lbl.gov>
References: <1433980959.11347.3.camel@JamesiMac>
	<B26E9991-DFDF-4170-AEB4-0B7003129DCC@cwalsh.org>
	<1433991338.3690.2.camel@JamesiMac>
	<1434026511.3690.13.camel@JamesiMac>
	<20150612165800.GC3273@yaksha.lbl.gov>
Message-ID: <1434128728.3700.2.camel@JamesiMac>

On Fri, 2015-06-12 at 09:58 -0700, Aashish Sharma wrote:

> Sorry for catching up on this late. 
> 
> I take you got script working now. If not, let me know, I will send a revised version. 
> 
> Thanks, 
> Aashish 
> 
> On Thu, Jun 11, 2015 at 06:41:51AM -0600, James Lay wrote:
> > 
> >    On Wed, 2015-06-10 at 20:55 -0600, James Lay wrote:
> > 
> >      On Wed, 2015-06-10 at 20:07 -0500, Chris Walsh wrote:
> > 
> > >From [1]https://www.bro.org/documentation/beta/NEWS.bro.html, under Deprecated
> >  Functionality:
> > 
> > 
> >         ? split: use split_string instead.
> > 
> > I just did this in another script  2 minutes before your mail hit my mailbox :^
> > )
> > 
> > cw
> > On Jun 10, 2015, at 7:02 PM, James Lay <[2]jlay at slave-tothe-box.net> wrote:
> > 
> > > Topic says it...I have the excellent smtp-embedded-url-bloom.bro which I use
> > almost every day for looking up links.  Upon upgrading to 2.4 I see:
> > >
> > > error in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro, line 40
> > : value used but not set (bloomfilter_basic_init)
> > > warning in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro, line
> > 73: deprecated (split)
> > >
> > > I read in the release notes that split was going away.  So now, what do I use
> >  to fix it?  I'm including this in the email as I'm not able to find it online.
> >   Thank you.
> > 
> >      Thanks Chris and Vlad, changing split_string fixed the warning, but did
> >      not fix the error at line 40 which halts bro from running.  Any reason
> >      why:
> >      global mail_links = bloomfilter_basic_init(0.00000001, 10000000) ;
> >      causes this issue?  This worked just fine with 2.3.2.  Thank you.
> >      James
> > _______________________________________________
> > Bro mailing list
> > [3]bro at bro-ids.org
> > [4]http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> > 
> >    Looks like this showed up last year:
> >    [5]http://mailman.icsi.berkeley.edu/pipermail/bro/2014-October/007662.html
> >    Is there something I need to do on my end?  Thank you.,
> >    James
> > 
> > References
> > 
> >    1. https://www.bro.org/documentation/beta/NEWS.bro.html
> >    2. mailto:jlay at slave-tothe-box.net
> >    3. mailto:bro at bro-ids.org
> >    4. http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >    5. http://mailman.icsi.berkeley.edu/pipermail/bro/2014-October/007662.html
> 
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 


I have not...the split issue is fine, the
error in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro,
line 40 : value used but not set (bloomfilter_basic_init)

is not.  Thank you.

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150612/de60ddf1/attachment-0001.html 

From asharma at lbl.gov  Fri Jun 12 10:27:40 2015
From: asharma at lbl.gov (Aashish Sharma)
Date: Fri, 12 Jun 2015 10:27:40 -0700
Subject: [Bro] Fixing deprecated scripts
In-Reply-To: <1434128728.3700.2.camel@JamesiMac>
References: <1433980959.11347.3.camel@JamesiMac>
	<B26E9991-DFDF-4170-AEB4-0B7003129DCC@cwalsh.org>
	<1433991338.3690.2.camel@JamesiMac>
	<1434026511.3690.13.camel@JamesiMac>
	<20150612165800.GC3273@yaksha.lbl.gov>
	<1434128728.3700.2.camel@JamesiMac>
Message-ID: <20150612172738.GD3273@yaksha.lbl.gov>

OK, I have updated the script on git-hub. Should work now. 

Let me know if you still see problems:

https://github.com/initconf/smtp-analysis/blob/master/smtp-embedded-url-bloom.bro

Thanks, 
Aashish 

On Fri, Jun 12, 2015 at 11:05:28AM -0600, James Lay wrote:
> 
>    On Fri, 2015-06-12 at 09:58 -0700, Aashish Sharma wrote:
> 
> Sorry for catching up on this late. 
> 
> I take you got script working now. If not, let me know, I will send a revised v
> ersion. 
> 
> Thanks, 
> Aashish 
> 
> On Thu, Jun 11, 2015 at 06:41:51AM -0600, James Lay wrote:
> > 
> >    On Wed, 2015-06-10 at 20:55 -0600, James Lay wrote:
> > 
> >      On Wed, 2015-06-10 at 20:07 -0500, Chris Walsh wrote:
> > 
> > >From [1][1]https://www.bro.org/documentation/beta/NEWS.bro.html, under Depre
> cated
> >  Functionality:
> >
> >
> >         ? split: use split_string instead.
> >
> > I just did this in another script  2 minutes before your mail hit my mailbox
> :^
> > )
> >
> > cw
> > On Jun 10, 2015, at 7:02 PM, James Lay <[2][2]jlay at slave-tothe-box.net> wrote
> :
> >
> > > Topic says it...I have the excellent smtp-embedded-url-bloom.bro which I us
> e
> > almost every day for looking up links.  Upon upgrading to 2.4 I see:
> > >
> > > error in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro, line
> 40
> > : value used but not set (bloomfilter_basic_init)
> > > warning in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro, lin
> e
> > 73: deprecated (split)
> > >
> > > I read in the release notes that split was going away.  So now, what do I u
> se
> >  to fix it?  I'm including this in the email as I'm not able to find it onlin
> e.
> >   Thank you.
> >
> >      Thanks Chris and Vlad, changing split_string fixed the warning, but did
> >      not fix the error at line 40 which halts bro from running.  Any reason
> >      why:
> >      global mail_links = bloomfilter_basic_init(0.00000001, 10000000) ;
> >      causes this issue?  This worked just fine with 2.3.2.  Thank you.
> >      James
> > _______________________________________________
> > Bro mailing list
> > [3][3]bro at bro-ids.org
> > [4][4]http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >
> >    Looks like this showed up last year:
> >    [5][5]http://mailman.icsi.berkeley.edu/pipermail/bro/2014-October/007662.h
> tml
> >    Is there something I need to do on my end?  Thank you.,
> >    James
> >
> > References
> >
> >    1. [6]https://www.bro.org/documentation/beta/NEWS.bro.html
> >    2. [7]mailto:jlay at slave-tothe-box.net
> >    3. [8]mailto:bro at bro-ids.org
> >    4. [9]http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >    5. [10]http://mailman.icsi.berkeley.edu/pipermail/bro/2014-October/007662.
> html
> 
> > _______________________________________________
> > Bro mailing list
> > [11]bro at bro-ids.org
> > [12]http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 
>    I have not...the split issue is fine, the
>    error in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro, line
>    40 : value used but not set (bloomfilter_basic_init)
>    is not.  Thank you.
>    James
> 
> References
> 
>    1. https://www.bro.org/documentation/beta/NEWS.bro.html
>    2. mailto:jlay at slave-tothe-box.net
>    3. mailto:bro at bro-ids.org
>    4. http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>    5. http://mailman.icsi.berkeley.edu/pipermail/bro/2014-October/007662.html
>    6. https://www.bro.org/documentation/beta/NEWS.bro.html
>    7. mailto:jlay at slave-tothe-box.net
>    8. mailto:bro at bro-ids.org
>    9. http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>   10. http://mailman.icsi.berkeley.edu/pipermail/bro/2014-October/007662.html
>   11. mailto:bro at bro-ids.org
>   12. http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-- 
Aashish Sharma	(asharma at lbl.gov) 				 
Cyber Security, 
Lawrence Berkeley National Laboratory  
http://go.lbl.gov/pgp-aashish 
Office: (510)-495-2680  Cell: (510)-612-7971

From jlay at slave-tothe-box.net  Fri Jun 12 10:58:17 2015
From: jlay at slave-tothe-box.net (James Lay)
Date: Fri, 12 Jun 2015 11:58:17 -0600
Subject: [Bro] Fixing deprecated scripts
In-Reply-To: <20150612172738.GD3273@yaksha.lbl.gov>
References: <1433980959.11347.3.camel@JamesiMac>
	<B26E9991-DFDF-4170-AEB4-0B7003129DCC@cwalsh.org>
	<1433991338.3690.2.camel@JamesiMac>
	<1434026511.3690.13.camel@JamesiMac>
	<20150612165800.GC3273@yaksha.lbl.gov>
	<1434128728.3700.2.camel@JamesiMac>
	<20150612172738.GD3273@yaksha.lbl.gov>
Message-ID: <1434131897.3700.3.camel@JamesiMac>

On Fri, 2015-06-12 at 10:27 -0700, Aashish Sharma wrote:

> OK, I have updated the script on git-hub. Should work now. 
> 
> Let me know if you still see problems:
> 
> https://github.com/initconf/smtp-analysis/blob/master/smtp-embedded-url-bloom.bro
> 
> Thanks, 
> Aashish 
> 
> On Fri, Jun 12, 2015 at 11:05:28AM -0600, James Lay wrote:
> > 
> >    On Fri, 2015-06-12 at 09:58 -0700, Aashish Sharma wrote:
> > 
> > Sorry for catching up on this late. 
> > 
> > I take you got script working now. If not, let me know, I will send a revised v
> > ersion. 
> > 
> > Thanks, 
> > Aashish 
> > 
> > On Thu, Jun 11, 2015 at 06:41:51AM -0600, James Lay wrote:
> > > 
> > >    On Wed, 2015-06-10 at 20:55 -0600, James Lay wrote:
> > > 
> > >      On Wed, 2015-06-10 at 20:07 -0500, Chris Walsh wrote:
> > > 
> > > >From [1][1]https://www.bro.org/documentation/beta/NEWS.bro.html, under Depre
> > cated
> > >  Functionality:
> > >
> > >
> > >         ? split: use split_string instead.
> > >
> > > I just did this in another script  2 minutes before your mail hit my mailbox
> > :^
> > > )
> > >
> > > cw
> > > On Jun 10, 2015, at 7:02 PM, James Lay <[2][2]jlay at slave-tothe-box.net> wrote
> > :
> > >
> > > > Topic says it...I have the excellent smtp-embedded-url-bloom.bro which I us
> > e
> > > almost every day for looking up links.  Upon upgrading to 2.4 I see:
> > > >
> > > > error in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro, line
> > 40
> > > : value used but not set (bloomfilter_basic_init)
> > > > warning in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro, lin
> > e
> > > 73: deprecated (split)
> > > >
> > > > I read in the release notes that split was going away.  So now, what do I u
> > se
> > >  to fix it?  I'm including this in the email as I'm not able to find it onlin
> > e.
> > >   Thank you.
> > >
> > >      Thanks Chris and Vlad, changing split_string fixed the warning, but did
> > >      not fix the error at line 40 which halts bro from running.  Any reason
> > >      why:
> > >      global mail_links = bloomfilter_basic_init(0.00000001, 10000000) ;
> > >      causes this issue?  This worked just fine with 2.3.2.  Thank you.
> > >      James
> > > _______________________________________________
> > > Bro mailing list
> > > [3][3]bro at bro-ids.org
> > > [4][4]http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> > >
> > >    Looks like this showed up last year:
> > >    [5][5]http://mailman.icsi.berkeley.edu/pipermail/bro/2014-October/007662.h
> > tml
> > >    Is there something I need to do on my end?  Thank you.,
> > >    James
> > >
> > > References
> > >
> > >    1. [6]https://www.bro.org/documentation/beta/NEWS.bro.html
> > >    2. [7]mailto:jlay at slave-tothe-box.net
> > >    3. [8]mailto:bro at bro-ids.org
> > >    4. [9]http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> > >    5. [10]http://mailman.icsi.berkeley.edu/pipermail/bro/2014-October/007662.
> > html
> > 
> > > _______________________________________________
> > > Bro mailing list
> > > [11]bro at bro-ids.org
> > > [12]http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> > 
> > 
> >    I have not...the split issue is fine, the
> >    error in /usr/local/bro/share/bro/site/./smtp-embedded-url-bloom.bro, line
> >    40 : value used but not set (bloomfilter_basic_init)
> >    is not.  Thank you.
> >    James
> > 
> > References
> > 
> >    1. https://www.bro.org/documentation/beta/NEWS.bro.html
> >    2. mailto:jlay at slave-tothe-box.net
> >    3. mailto:bro at bro-ids.org
> >    4. http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >    5. http://mailman.icsi.berkeley.edu/pipermail/bro/2014-October/007662.html
> >    6. https://www.bro.org/documentation/beta/NEWS.bro.html
> >    7. mailto:jlay at slave-tothe-box.net
> >    8. mailto:bro at bro-ids.org
> >    9. http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >   10. http://mailman.icsi.berkeley.edu/pipermail/bro/2014-October/007662.html
> >   11. mailto:bro at bro-ids.org
> >   12. http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 


Thank you so much Aashish....that completely fixed it up.

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150612/04dbb0ba/attachment.html 

From seth at icir.org  Tue Jun 16 06:18:29 2015
From: seth at icir.org (Seth Hall)
Date: Tue, 16 Jun 2015 09:18:29 -0400
Subject: [Bro] DoS in OpenSSL (high priority!)
Message-ID: <9067EC1B-C40D-4A57-A034-849C4C6E6697@icir.org>

I just did a blog post about a DoS in OpenSSL.
	http://blog.bro.org/2015/06/openssl-denial-of-service-impacting-bro.html

The short of it is that everyone, with almost no exceptions, will want to upgrade OpenSSL immediately.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150616/eb27118c/attachment.bin 

From golfnllama at gmail.com  Tue Jun 16 09:55:05 2015
From: golfnllama at gmail.com (Gmail)
Date: Tue, 16 Jun 2015 12:55:05 -0400
Subject: [Bro] Change Log File Name
Message-ID: <7C6ED31B-FB4C-4177-A15E-79ACDFEFF041@gmail.com>

Greetings,
I am feeding various pcaps to Bro and I would like to keep separate DNS log files for each one. Is there a good way to change the name of the dns.log file to include a timestamp? For example, dns201506161254.log. Thanks!

-Eddie Romito-
Sent from my iPhone

From seth at icir.org  Tue Jun 16 12:00:58 2015
From: seth at icir.org (Seth Hall)
Date: Tue, 16 Jun 2015 15:00:58 -0400
Subject: [Bro] DoS in OpenSSL (high priority!)
In-Reply-To: <9067EC1B-C40D-4A57-A034-849C4C6E6697@icir.org>
References: <9067EC1B-C40D-4A57-A034-849C4C6E6697@icir.org>
Message-ID: <561DA4FA-B6DC-4077-8BA4-C433EC003712@icir.org>

I just did some updates to the blog post (RedHat doesn?t exhibit the problem) and I updated the compensation script for people still running 2.3.

  .Seth

> On Jun 16, 2015, at 9:18 AM, Seth Hall <seth at icir.org> wrote:
> 
> I just did a blog post about a DoS in OpenSSL.
> 	http://blog.bro.org/2015/06/openssl-denial-of-service-impacting-bro.html
> 
> The short of it is that everyone, with almost no exceptions, will want to upgrade OpenSSL immediately.
> 
>  .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150616/b2bb1ee9/attachment.bin 

From damonrouse at gmail.com  Tue Jun 16 12:35:28 2015
From: damonrouse at gmail.com (Damon Rouse)
Date: Tue, 16 Jun 2015 12:35:28 -0700
Subject: [Bro] Extract.bro Question
Message-ID: <CAHoWat6FG6dXKSJw6WwaWMDowgfKMzCejJYS20fi7wRYDErZjw@mail.gmail.com>

Hi Everyone

I'm still pretty new to the more complex aspects of Bro, so I'm not sure if
this is possible or not.  I've been testing file extraction and it's
working really well for me.  My question is, can Bro (in extract.bro) get
the file name of the file being extracted?  So the final extracted file
would have a naming convention like Analyzer-FileName.SpecifiedExtension

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150616/22d01d58/attachment.html 

From landy-bible at utulsa.edu  Wed Jun 17 09:17:47 2015
From: landy-bible at utulsa.edu (Bible, Landy)
Date: Wed, 17 Jun 2015 16:17:47 +0000
Subject: [Bro] Bro Cluster Missing Many HTTP Requests
Message-ID: <9065BE5D60E0044DB3EAC4965CFF1C940142040051@POLAR.ad.utulsa.edu>

Hi All,

I'm just getting started with Bro. So far I'm really liking the data I get, even just out of the box. I've got one standalone host running with PF_RING enabled, 8 workers. I am also testing multi host clustering, and have two worker hosts running 6 workers each (again, with PF_RING) with the master and proxy running on a third host. All three worker hosts are being fed tap data from an Arista Networks 7150. The standalone host is getting data from a regular Tool port, and the other two are getting it from a PortChannel. Both tool ports are connected to the same Aggregation Group, so both Bro systems should be getting exactly the same data.

As expected, the standalone box has a much higher CPU load, and it occurred to me today that I should bump the number of workers down so I could free up a core for the manager. I got some stats from yesterday...

Single Bro Host
------------------------
116,853 Packets Dropped (as reported by the notice logs)
56,827,921 Connections Logged (just a wc -l of the connection logs)
17,323,728 HTTP Requests Logged (just a wc -l of the http logs)

Bro Cluster
------------------------
7 Packets Dropped
79,115,195 Connections Logged
7,436,365 HTTP Requests Logged

In addition to the packets being dropped by the host, I see a large number of TX drops on the Arista output for the single Bro host. I suspect that's due to the packet rate exceeding the capacity of the port occasionally, so I'm not too worried about that. And the CPU load on the single box vs the cluster explains why the cluster managed to snag so many more of the connections.  However, what has me very confused is how the cluster missed nearly 10 million HTTP requests relative to the single host, despite logging 25 million more connections. Both Bro systems are configured the same, loading the same scripts. So far I'm just using the out of the box config. The only difference is that I pulled the source from GitHub for the cluster the day after I did the standalone host.

Just looking for HTTP requests generated by my computer yesterday, the cluster snagged 120 of them while the standalone host got 416. Of those, 104 were common between the two logs.

Can someone point me towards where I should look to start trying to figure out why I'm getting such vastly different results from one system over the other? So far I'm not seeing anything obvious in any of the logs I've found.

Thanks!

--
Landy Bible
Information Security Analyst
The University of Tulsa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150617/90d7af65/attachment.html 

From seth at icir.org  Wed Jun 17 10:06:48 2015
From: seth at icir.org (Seth Hall)
Date: Wed, 17 Jun 2015 13:06:48 -0400
Subject: [Bro] Extract.bro Question
In-Reply-To: <CAHoWat6FG6dXKSJw6WwaWMDowgfKMzCejJYS20fi7wRYDErZjw@mail.gmail.com>
References: <CAHoWat6FG6dXKSJw6WwaWMDowgfKMzCejJYS20fi7wRYDErZjw@mail.gmail.com>
Message-ID: <B23F7ACD-39A7-4DBB-B876-AF0FD9D39FAC@icir.org>


> On Jun 16, 2015, at 3:35 PM, Damon Rouse <damonrouse at gmail.com> wrote:
> 
> I'm still pretty new to the more complex aspects of Bro, so I'm not sure if this is possible or not.  I've been testing file extraction and it's working really well for me.  My question is, can Bro (in extract.bro) get the file name of the file being extracted?  So the final extracted file would have a naming convention like Analyzer-FileName.SpecifiedExtension

I started to head in that direction initially but then what bothered me a little bit was that external hosts could affect file names on your system and I started to get concerned about that.  I started imagining scenarios where names are written out that do very unexpected things on your system or break out of the path they?re supposed to be extracted into.  I would match up the filename in the files.log with the fuid on disk.  If you look a files.log it will actually have the filename on disk and a filename (if one was discovered) from the network traffic.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150617/f1556230/attachment.bin 

From seth at icir.org  Wed Jun 17 10:08:41 2015
From: seth at icir.org (Seth Hall)
Date: Wed, 17 Jun 2015 13:08:41 -0400
Subject: [Bro] Fixing deprecated scripts
In-Reply-To: <1433991338.3690.2.camel@JamesiMac>
References: <1433980959.11347.3.camel@JamesiMac>
	<B26E9991-DFDF-4170-AEB4-0B7003129DCC@cwalsh.org>
	<1433991338.3690.2.camel@JamesiMac>
Message-ID: <E4B6B6E2-3438-48C7-9B4C-6852A0AF0166@icir.org>


> On Jun 10, 2015, at 10:55 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> global mail_links = bloomfilter_basic_init(0.00000001, 10000000) ;
> 
> causes this issue?  This worked just fine with 2.3.2.  Thank you.

Jumping back in here for a moment, there were some changes with how and when bloom filters can be initialized in 2.4 and this script took advantage of an accidental ?feature?. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150617/ed803bb3/attachment.bin 

From nb.nospam at gmail.com  Wed Jun 17 10:38:01 2015
From: nb.nospam at gmail.com (N B)
Date: Wed, 17 Jun 2015 10:38:01 -0700
Subject: [Bro] Subclassing from SSL Analyzer
Message-ID: <CA+rfBVxymMgEOdftG7h5qqzTRpozqZNcarST1yC3wsSdt8vMnA@mail.gmail.com>

Hello,

I am trying to subclass from SSL Analyzer such that the derived class can
decrypt the encrypted SSL data (it will have the server's private key).  I
have to #include <SSL.h> in my new plugin's header file to allow the
derived class declaration to work but doing that is causing the following
compiler error:

Scanning dependencies of target plugin-Bro-SSLDecrypt
[ 66%] Building CXX object
src/analyzer/protocol/ssldecrypt/CMakeFiles/plugin-Bro-SSLDecrypt.dir/SSLDecrypt.cc.o
In file included from
/Users/nikunj/git/bro/src/analyzer/protocol/ssldecrypt/SSLDecrypt.cc:9:
In file included from
/Users/nikunj/git/bro/src/analyzer/protocol/ssldecrypt/SSLDecrypt.h:12:
/Users/nikunj/git/bro/src/analyzer/protocol/ssl/SSL.h:4:10: fatal error:
'events.bif.h' file not found
#include "events.bif.h"
         ^
1 error generated.
make[3]: ***
[src/analyzer/protocol/ssldecrypt/CMakeFiles/plugin-Bro-SSLDecrypt.dir/SSLDecrypt.cc.o]
Error 1
make[2]: ***
[src/analyzer/protocol/ssldecrypt/CMakeFiles/plugin-Bro-SSLDecrypt.dir/all]
Error 2
make[1]: *** [all] Error 2
make: *** [all] Error 2


How can I work around this issue?

An option I was thinking of was to directly change the SSL analyzer's code
and not subclass at all. But that would mean I will have to keep patching
it forward as we get newer Bro releases.

Thanks
Nikunj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150617/1c269dce/attachment-0001.html 

From dnthayer at illinois.edu  Wed Jun 17 10:38:08 2015
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Wed, 17 Jun 2015 12:38:08 -0500
Subject: [Bro] Change Log File Name
In-Reply-To: <7C6ED31B-FB4C-4177-A15E-79ACDFEFF041@gmail.com>
References: <7C6ED31B-FB4C-4177-A15E-79ACDFEFF041@gmail.com>
Message-ID: <5581B080.5020000@illinois.edu>



On 06/16/2015 11:55 AM, Gmail wrote:
> Greetings,
> I am feeding various pcaps to Bro and I would like to keep separate DNS log files for each one. Is there a good way to change the name of the dns.log file to include a timestamp? For example, dns201506161254.log. Thanks!
>
> -Eddie Romito-

The Bro logging framework documentation describes how to rename
a log file:  https://www.bro.org/sphinx/frameworks/logging.html


From jlay at slave-tothe-box.net  Wed Jun 17 16:57:42 2015
From: jlay at slave-tothe-box.net (James Lay)
Date: Wed, 17 Jun 2015 17:57:42 -0600
Subject: [Bro] Fixing deprecated scripts
In-Reply-To: <E4B6B6E2-3438-48C7-9B4C-6852A0AF0166@icir.org>
References: <1433980959.11347.3.camel@JamesiMac>
	<B26E9991-DFDF-4170-AEB4-0B7003129DCC@cwalsh.org>
	<1433991338.3690.2.camel@JamesiMac>
	<E4B6B6E2-3438-48C7-9B4C-6852A0AF0166@icir.org>
Message-ID: <1434585462.9467.3.camel@JamesiMac>

On Wed, 2015-06-17 at 13:08 -0400, Seth Hall wrote:

> > On Jun 10, 2015, at 10:55 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> > global mail_links = bloomfilter_basic_init(0.00000001, 10000000) ;
> > 
> > causes this issue?  This worked just fine with 2.3.2.  Thank you.
> 
> Jumping back in here for a moment, there were some changes with how and when bloom filters can be initialized in 2.4 and this script took advantage of an accidental ?feature?. :)
> 
>   .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
> 


Thanks for expanding on this Seth...it helps my understanding.

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150617/24490b1b/attachment.html 

From seth at icir.org  Wed Jun 17 21:12:16 2015
From: seth at icir.org (Seth Hall)
Date: Thu, 18 Jun 2015 00:12:16 -0400
Subject: [Bro] Subclassing from SSL Analyzer
In-Reply-To: <CA+rfBVxymMgEOdftG7h5qqzTRpozqZNcarST1yC3wsSdt8vMnA@mail.gmail.com>
References: <CA+rfBVxymMgEOdftG7h5qqzTRpozqZNcarST1yC3wsSdt8vMnA@mail.gmail.com>
Message-ID: <AE36A573-9B69-4BBB-B385-AD3832025BFD@icir.org>


> On Jun 17, 2015, at 1:38 PM, N B <nb.nospam at gmail.com> wrote:
> 
> An option I was thinking of was to directly change the SSL analyzer's code and not subclass at all. But that would mean I will have to keep patching it forward as we get newer Bro releases.

I don?t believe you want to subclass the analyzer.  The right way is to poke the right decryption into the right place in the analyzer.  It?s remarkably easy if you understand Binpac well.  We wouldn?t intrinsically have any issues with merging SSL decryption into Bro either if it?s done well, there is no reason for you to maintain a patch set moving forward.  If it was brought into Bro we would need tests too so that even for us to maintain it, it shouldn?t be overly onerous.

I guess I?ll go ahead and admit it now... I have some changes to the SSL analyzer that I haven?t pushed out anywhere that poke into the right places in the analyzer to decrypt traffic.  What I?ve gotten stuck on (due to lack of time and inexperience) is doing the actual decryption.  If there is someone out there that has done this before I?d be interested in talking and possibly working together on it.  We can certainly make this happen and get this into a Bro release.  I think that we could even do some really neat stuff that other open source decryption tools aren?t doing due to Bro being so dynamic.

Anyone interested? (I?m still not going to post my code publicly, I don?t want to get the questions that I?d inevitably get if I did)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150618/8a49e55f/attachment.bin 

From seth at icir.org  Wed Jun 17 22:10:17 2015
From: seth at icir.org (Seth Hall)
Date: Thu, 18 Jun 2015 01:10:17 -0400
Subject: [Bro] disable_stream vs remove_filter
In-Reply-To: <1459783901.8638993.1433772113445.JavaMail.yahoo@mail.yahoo.com>
References: <mailman.1.1433703602.9830.bro@bro.org>
	<1459783901.8638993.1433772113445.JavaMail.yahoo@mail.yahoo.com>
Message-ID: <E179277A-ECE9-4BFA-8AE2-1D5546CC7525@icir.org>


> On Jun 8, 2015, at 10:01 AM, Adam Hall <abhall1 at yahoo.com> wrote:
> 
> 1) Will one give me a performance gain over the other?

Probably nothing noticeable.

> 2) Will one cause problems for other calls being made (If I disable_stream and something calls that stream will it break)?

Nope, no problems.

> 3) If I disable a stream and later decide to add a new filter, will that work?

I?m having trouble remembering, but logging settings may not all be changeable at runtime.  I?m actually curious which settings are possible at runtime.  I assume you were talking about changes during runtime?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150618/7e963f05/attachment.bin 

From seth at icir.org  Wed Jun 17 22:12:08 2015
From: seth at icir.org (Seth Hall)
Date: Thu, 18 Jun 2015 01:12:08 -0400
Subject: [Bro] saving raw packet payload to text file
In-Reply-To: <CADsr06y50w-TVGc_LQAGdAZDPnmcrQd138P4LGW_Ba8hfLSa2w@mail.gmail.com>
References: <mailman.15776.1433624430.949.bro@bro.org>
	<CADsr06y50w-TVGc_LQAGdAZDPnmcrQd138P4LGW_Ba8hfLSa2w@mail.gmail.com>
Message-ID: <1EDAE9D2-044C-4649-AA4E-8008BA5A2D4C@icir.org>


> On Jun 6, 2015, at 5:03 PM, Fateme Eskandari <f.eskandary2009 at gmail.com> wrote:
> 
> i have a pcap file that contain data about some protocols. i want to have a text file for every protocol from my pcap file that contains all raws of packet payload in asccii format. just like this:
> which command could i use?

redef Conn::default_extract=T;

That will create a directory named ?contents? in your CWD and fill it with files containing the data you want.  One difference is that Bro writes each flow to a separate file so a connection would write out two files.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150618/50430fae/attachment.bin 

From gc355804 at ohio.edu  Thu Jun 18 04:26:29 2015
From: gc355804 at ohio.edu (Clark, Gilbert)
Date: Thu, 18 Jun 2015 11:26:29 +0000
Subject: [Bro] Instrumentation plugin (WIP)
Message-ID: <BLUPR01MB1380905B47D5D9E445B9A897AAA50@BLUPR01MB1380.prod.exchangelabs.com>


Hi list:

Just a quick note to say that I've been working on a plugin to support profiling bro script execution in my spare time.  The eventual goal is to make it a bit easier to profile / troubleshoot / optimize bro's performance.

I'm soliciting a bit of feedback / thoughts / opinions if folks have time and interest to spare.  Current code is here:

https://github.com/cubic1271/bro-plugin-instrumentation

There's a pretty long README in the repository.  CSV and JSON output formats are supported at present.

There's also a pretty basic web UI included in the 'ui' directory of the project that eats the JSON output and turns it into something a little more human-readable.  An example of what this looks like is available here:

https://cubic1271.github.io/bro-plugin-instrumentation/#/home

The 'What is it?' tab on that page has more information on the UI along with a short explanation of how to deploy the web application to poke through local profiling data.  The short version is that grabbing the 'gh-pages' branch of the instrumentation plugin repository and replacing the JSON files / callgraph.png with your own data, then serving with python -m SimpleHTTPServer or the like and loading in a browser should do the trick.

The example data in the above was pulled from a public example trace I found on the internet somewhere.

Questions / comments / concerns / criticism, please feel free to get in touch.

Cheers,
Gilbert Clark


From earl.eiland at root9b.com  Thu Jun 18 06:28:50 2015
From: earl.eiland at root9b.com (Earl Eiland)
Date: Thu, 18 Jun 2015 13:28:50 +0000
Subject: [Bro] missing fields in conn.log
Message-ID: <BY1PR0401MB13822F098DADEC6C5E5768DAD7A50@BY1PR0401MB1382.namprd04.prod.outlook.com>

I'm running bro 2.4 on an industrial control system pcap file.  According to , https://www.bro.org/sphinx-git/scripts/base/init-bare.bro.html#type-connection,<https://www.bro.org/sphinx-git/scripts/base/init-bare.bro.html#type-connection> there are a number of optional fields in conn.log.  However, conn.log does not seem to include any of the optional fields.


For example, my test data includes MODBUS traffic, and one of the optional conn fields is "modbus".  I've checked loaded-scripts.log: modbus/main.bro is loaded.  Also modbus.log is being output and populated.  conn.log, however, does not include a "modbus" field.


what do I have to do for conn.log to include the optional fields?


Best Regards,

Earl Eiland,
Sr. Cyber Security Engineer,
Emerging Technologies, root9B,
San Antonio, Texas

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity named.  If you are not the named addressee you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.  Please notify the sender immediately by email if you received this email in error and delete this email from your system. Any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of root9B LLC.?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150618/1bf586d6/attachment-0001.html 

From Alec.Waters at dataline.co.uk  Thu Jun 18 06:38:54 2015
From: Alec.Waters at dataline.co.uk (Alec Waters)
Date: Thu, 18 Jun 2015 13:38:54 +0000
Subject: [Bro] Subclassing from SSL Analyzer
In-Reply-To: <AE36A573-9B69-4BBB-B385-AD3832025BFD@icir.org>
References: <CA+rfBVxymMgEOdftG7h5qqzTRpozqZNcarST1yC3wsSdt8vMnA@mail.gmail.com>
	<AE36A573-9B69-4BBB-B385-AD3832025BFD@icir.org>
Message-ID: <8350146BADDCE04480B969B36967473D10852052@ZEUS.olympus.dataline.co.uk>

Even if you've got the key, isn't the analyser going to be stymied in the 
presence of Diffie-Hellman in the cipher suite? SSL decryption (with the 
server's key) works well enough when the client is using the server's public 
key to encrypt the pre-master secret, but the private key is of no use when DH 
is in play.

alec
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6089 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150618/8a77c359/attachment.bin 

From seth at icir.org  Thu Jun 18 07:12:10 2015
From: seth at icir.org (Seth Hall)
Date: Thu, 18 Jun 2015 10:12:10 -0400
Subject: [Bro] Subclassing from SSL Analyzer
In-Reply-To: <8350146BADDCE04480B969B36967473D10852052@ZEUS.olympus.dataline.co.uk>
References: <CA+rfBVxymMgEOdftG7h5qqzTRpozqZNcarST1yC3wsSdt8vMnA@mail.gmail.com>
	<AE36A573-9B69-4BBB-B385-AD3832025BFD@icir.org>
	<8350146BADDCE04480B969B36967473D10852052@ZEUS.olympus.dataline.co.uk>
Message-ID: <A65E2F39-A5A5-4C5A-80CE-D9EC1448EF89@icir.org>


> On Jun 18, 2015, at 9:38 AM, Alec Waters <Alec.Waters at dataline.co.uk> wrote:
> 
> Even if you've got the key, isn't the analyser going to be stymied in the
> presence of Diffie-Hellman in the cipher suite?

Yep, SSL/TLS decryption is far from a panacea.  It is a neat example in the analyzer though and I think we can do it more cleanly than I?ve seen most other projects do it (and make it easier to use!).

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150618/08a0ba3c/attachment.bin 

From seth at icir.org  Thu Jun 18 07:22:51 2015
From: seth at icir.org (Seth Hall)
Date: Thu, 18 Jun 2015 10:22:51 -0400
Subject: [Bro] missing fields in conn.log
In-Reply-To: <BY1PR0401MB13822F098DADEC6C5E5768DAD7A50@BY1PR0401MB1382.namprd04.prod.outlook.com>
References: <BY1PR0401MB13822F098DADEC6C5E5768DAD7A50@BY1PR0401MB1382.namprd04.prod.outlook.com>
Message-ID: <B5C4B216-35B5-4443-9C88-414F69BFA833@icir.org>


> On Jun 18, 2015, at 9:28 AM, Earl Eiland <earl.eiland at root9b.com> wrote:
> 
> For example, my test data includes MODBUS traffic, and one of the optional conn fields is "modbus".  I've checked loaded-scripts.log: modbus/main.bro is loaded.  Also modbus.log is being output and populated.  conn.log, however, does not include a "modbus" field.

Eep!  You just discovered a bug.  The analyzer is never validating the protocol successfully (which is required in order for it to show up in conn.log).  I?m going to do a patch now that fixes it.

?modbus? should be showing up in the ?service? field of conn.log (which represents analyzers that were attached and successfully analyzed a connection.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150618/d6ab444f/attachment.bin 

From seth at icir.org  Thu Jun 18 07:26:14 2015
From: seth at icir.org (Seth Hall)
Date: Thu, 18 Jun 2015 10:26:14 -0400
Subject: [Bro] Instrumentation plugin (WIP)
In-Reply-To: <BLUPR01MB1380905B47D5D9E445B9A897AAA50@BLUPR01MB1380.prod.exchangelabs.com>
References: <BLUPR01MB1380905B47D5D9E445B9A897AAA50@BLUPR01MB1380.prod.exchangelabs.com>
Message-ID: <9E3FB268-8D34-41D7-A02E-29865A9073A4@icir.org>


> On Jun 18, 2015, at 7:26 AM, Clark, Gilbert <gc355804 at ohio.edu> wrote:
> 
> Just a quick note to say that I've been working on a plugin to support profiling bro script execution in my spare time.  The eventual goal is to make it a bit easier to profile / troubleshoot / optimize bro's performance.

That?s really neat!  Thanks for sharing!  You?ve been burning on this in the background for so many year now that it?s really neat to see something like this coming out of it.  I need to play with it some, but I can already imagine that I?ll have some feedback for you soon. : )

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150618/a34c93bf/attachment.bin 

From brunotaf31 at gmail.com  Thu Jun 18 07:27:49 2015
From: brunotaf31 at gmail.com (A Bruno)
Date: Thu, 18 Jun 2015 16:27:49 +0200
Subject: [Bro] binpac++ plugin error
In-Reply-To: <20150420050651.GJ72050@icir.org>
References: <553316CD.6040803@maine.edu>
	<20150420050651.GJ72050@icir.org>
Message-ID: <CANFQkQvdcRZ_6jiKSe9s9RvU8QaX-4HGKZX+n+m8Ev_EES9Kig@mail.gmail.com>

Hello,

I encounter the same problem when I try to reproduce the "BinPAC++ Demo".
I use bro 2.4-beta6 compiled from last revision sources.
Also I compiled Hilti from sources using the

Robin, Troy, have you found a solution on this problem that I could apply
on a non docker context?

Thanks,
Bruno.


2015-04-20 7:06 GMT+02:00 Robin Sommer <robin at icir.org>:

>
> On Sat, Apr 18, 2015 at 22:45 -0400, you wrote:
>
> > /opt/hilti/build/bro//lib/Bro-Hilti.linux-x86_64.so: undefined symbol:
> >
> _ZN6plugin6Plugin11MetaHookPreENS_8HookTypeERKNSt3__14listINS_12HookArgumentENS2_9allocatorIS4_EEEE
>
> Oh, there's actually a problem in the Docker file I believe. I'll see
> that I get that fixed.
>
> Robin
>
> --
> Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150618/50795cd3/attachment.html 

From seth at icir.org  Thu Jun 18 07:51:41 2015
From: seth at icir.org (Seth Hall)
Date: Thu, 18 Jun 2015 10:51:41 -0400
Subject: [Bro] Instrumentation plugin (WIP)
In-Reply-To: <BLUPR01MB1380905B47D5D9E445B9A897AAA50@BLUPR01MB1380.prod.exchangelabs.com>
References: <BLUPR01MB1380905B47D5D9E445B9A897AAA50@BLUPR01MB1380.prod.exchangelabs.com>
Message-ID: <2BE12956-660A-4592-8089-8B08D4C17DFE@icir.org>


> On Jun 18, 2015, at 7:26 AM, Clark, Gilbert <gc355804 at ohio.edu> wrote:
> 
> Questions / comments / concerns / criticism, please feel free to get in touch.

One small thing already, it doesn?t seem to be building for me.  Am I doing something wrong?

/Users/seth/bro/bro-plugin-instrumentation/src/http/civetweb.c:6480:5: error: expected expression
    DEBUG_TRACE("going idle");
    ^
/Users/seth/bro/bro-plugin-instrumentation/src/http/civetweb.c:381:84: note: expanded from macro 'DEBUG_TRACE'
#define DEBUG_TRACE(fmt, ...) DEBUG_TRACE_FUNC(__func__, __LINE__, fmt, __VA_ARGS__)
                                                                                   ^
/Users/seth/bro/bro-plugin-instrumentation/src/http/civetweb.c:6572:5: error: expected expression
    DEBUG_TRACE("exiting");
    ^
/Users/seth/bro/bro-plugin-instrumentation/src/http/civetweb.c:381:84: note: expanded from macro 'DEBUG_TRACE'
#define DEBUG_TRACE(fmt, ...) DEBUG_TRACE_FUNC(__func__, __LINE__, fmt, __VA_ARGS__)
                                                                                   ^
/Users/seth/bro/bro-plugin-instrumentation/src/http/civetweb.c:6724:5: error: expected expression
    DEBUG_TRACE("stopping workers");
    ^
/Users/seth/bro/bro-plugin-instrumentation/src/http/civetweb.c:381:84: note: expanded from macro 'DEBUG_TRACE'
#define DEBUG_TRACE(fmt, ...) DEBUG_TRACE_FUNC(__func__, __LINE__, fmt, __VA_ARGS__)
                                                                                   ^
/Users/seth/bro/bro-plugin-instrumentation/src/http/civetweb.c:6748:5: error: expected expression
    DEBUG_TRACE("exiting");
    ^
/Users/seth/bro/bro-plugin-instrumentation/src/http/civetweb.c:381:84: note: expanded from macro 'DEBUG_TRACE'
#define DEBUG_TRACE(fmt, ...) DEBUG_TRACE_FUNC(__func__, __LINE__, fmt, __VA_ARGS__)
                                                                                   ^
4 errors generated.


  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150618/7bfe0696/attachment-0001.bin 

From earl.eiland at root9b.com  Thu Jun 18 08:16:38 2015
From: earl.eiland at root9b.com (Earl Eiland)
Date: Thu, 18 Jun 2015 15:16:38 +0000
Subject: [Bro] missing fields in conn.log
In-Reply-To: <B5C4B216-35B5-4443-9C88-414F69BFA833@icir.org>
References: <BY1PR0401MB13822F098DADEC6C5E5768DAD7A50@BY1PR0401MB1382.namprd04.prod.outlook.com>,
	<B5C4B216-35B5-4443-9C88-414F69BFA833@icir.org>
Message-ID: <BY1PR0401MB1382F0BE84ED8F30A85BC4F0D7A50@BY1PR0401MB1382.namprd04.prod.outlook.com>

awesome!  Thanks.

Best Regards,

Earl Eiland,


________________________________________
From: Seth Hall <seth at icir.org>
Sent: Thursday, June 18, 2015 9:22 AM
To: Earl Eiland
Cc: bro at bro.org
Subject: Re: [Bro] missing fields in conn.log

> On Jun 18, 2015, at 9:28 AM, Earl Eiland <earl.eiland at root9b.com> wrote:
>
> For example, my test data includes MODBUS traffic, and one of the optional conn fields is "modbus".  I've checked loaded-scripts.log: modbus/main.bro is loaded.  Also modbus.log is being output and populated.  conn.log, however, does not include a "modbus" field.

Eep!  You just discovered a bug.  The analyzer is never validating the protocol successfully (which is required in order for it to show up in conn.log).  I?m going to do a patch now that fixes it.

?modbus? should be showing up in the ?service? field of conn.log (which represents analyzers that were attached and successfully analyzed a connection.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From robin at icir.org  Thu Jun 18 08:41:41 2015
From: robin at icir.org (Robin Sommer)
Date: Thu, 18 Jun 2015 08:41:41 -0700
Subject: [Bro] binpac++ plugin error
In-Reply-To: <CANFQkQvdcRZ_6jiKSe9s9RvU8QaX-4HGKZX+n+m8Ev_EES9Kig@mail.gmail.com>
References: <553316CD.6040803@maine.edu> <20150420050651.GJ72050@icir.org>
	<CANFQkQvdcRZ_6jiKSe9s9RvU8QaX-4HGKZX+n+m8Ev_EES9Kig@mail.gmail.com>
Message-ID: <20150618154141.GH1606@icir.org>



On Thu, Jun 18, 2015 at 16:27 +0200, A Bruno wrote:

> Robin, Troy, have you found a solution on this problem that I could apply
> on a non docker context?

I was hoping it's fixed now, both inside and outside of Docker. Are you
using the most recent HILTI/BinPAC++ version?

The Docker image now seems to be working for folks.

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From gc355804 at ohio.edu  Thu Jun 18 09:36:48 2015
From: gc355804 at ohio.edu (Clark, Gilbert)
Date: Thu, 18 Jun 2015 16:36:48 +0000
Subject: [Bro] Instrumentation plugin (WIP)
In-Reply-To: <2BE12956-660A-4592-8089-8B08D4C17DFE@icir.org>
References: <BLUPR01MB1380905B47D5D9E445B9A897AAA50@BLUPR01MB1380.prod.exchangelabs.com>,
	<2BE12956-660A-4592-8089-8B08D4C17DFE@icir.org>
Message-ID: <BLUPR01MB138075FA8B376322C56CD085AAA50@BLUPR01MB1380.prod.exchangelabs.com>

Hah, nice ... five hours and it's already broken :)  

Also, I've pushed something I believe to be a fix for this issue.  Please give it another try and let me know.

I've also opened an issue for this here: https://github.com/cubic1271/bro-plugin-instrumentation/issues/1

Thanks for taking the time to try this out!

Cheers,
Gilbert

________________________________________
From: Seth Hall <seth at icir.org>
Sent: Thursday, June 18, 2015 10:51 AM
To: Clark, Gilbert
Cc: bro at bro.org
Subject: Re: [Bro] Instrumentation plugin (WIP)

> On Jun 18, 2015, at 7:26 AM, Clark, Gilbert <gc355804 at ohio.edu> wrote:
>
> Questions / comments / concerns / criticism, please feel free to get in touch.

One small thing already, it doesn?t seem to be building for me.  Am I doing something wrong?

/Users/seth/bro/bro-plugin-instrumentation/src/http/civetweb.c:6480:5: error: expected expression
    DEBUG_TRACE("going idle");
    ^
/Users/seth/bro/bro-plugin-instrumentation/src/http/civetweb.c:381:84: note: expanded from macro 'DEBUG_TRACE'
#define DEBUG_TRACE(fmt, ...) DEBUG_TRACE_FUNC(__func__, __LINE__, fmt, __VA_ARGS__)
                                                                                   ^
/Users/seth/bro/bro-plugin-instrumentation/src/http/civetweb.c:6572:5: error: expected expression
    DEBUG_TRACE("exiting");
    ^
/Users/seth/bro/bro-plugin-instrumentation/src/http/civetweb.c:381:84: note: expanded from macro 'DEBUG_TRACE'
#define DEBUG_TRACE(fmt, ...) DEBUG_TRACE_FUNC(__func__, __LINE__, fmt, __VA_ARGS__)
                                                                                   ^
/Users/seth/bro/bro-plugin-instrumentation/src/http/civetweb.c:6724:5: error: expected expression
    DEBUG_TRACE("stopping workers");
    ^
/Users/seth/bro/bro-plugin-instrumentation/src/http/civetweb.c:381:84: note: expanded from macro 'DEBUG_TRACE'
#define DEBUG_TRACE(fmt, ...) DEBUG_TRACE_FUNC(__func__, __LINE__, fmt, __VA_ARGS__)
                                                                                   ^
/Users/seth/bro/bro-plugin-instrumentation/src/http/civetweb.c:6748:5: error: expected expression
    DEBUG_TRACE("exiting");
    ^
/Users/seth/bro/bro-plugin-instrumentation/src/http/civetweb.c:381:84: note: expanded from macro 'DEBUG_TRACE'
#define DEBUG_TRACE(fmt, ...) DEBUG_TRACE_FUNC(__func__, __LINE__, fmt, __VA_ARGS__)
                                                                                   ^
4 errors generated.


  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From seth at icir.org  Thu Jun 18 09:50:04 2015
From: seth at icir.org (Seth Hall)
Date: Thu, 18 Jun 2015 12:50:04 -0400
Subject: [Bro] Instrumentation plugin (WIP)
In-Reply-To: <BLUPR01MB138075FA8B376322C56CD085AAA50@BLUPR01MB1380.prod.exchangelabs.com>
References: <BLUPR01MB1380905B47D5D9E445B9A897AAA50@BLUPR01MB1380.prod.exchangelabs.com>
	<2BE12956-660A-4592-8089-8B08D4C17DFE@icir.org>
	<BLUPR01MB138075FA8B376322C56CD085AAA50@BLUPR01MB1380.prod.exchangelabs.com>
Message-ID: <A9A94327-6559-4E4D-B1D4-C4FA039F3B27@icir.org>


> On Jun 18, 2015, at 12:36 PM, Clark, Gilbert <gc355804 at ohio.edu> wrote:
> 
> Hah, nice ... five hours and it's already broken :)
> 
> Also, I've pushed something I believe to be a fix for this issue.  Please give it another try and let me know.

Yay!  It built.  I?ll try and run it tonight.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150618/040940a5/attachment.bin 

From brunotaf31 at gmail.com  Thu Jun 18 10:39:11 2015
From: brunotaf31 at gmail.com (A Bruno)
Date: Thu, 18 Jun 2015 19:39:11 +0200
Subject: [Bro] binpac++ plugin error
In-Reply-To: <20150618154141.GH1606@icir.org>
References: <553316CD.6040803@maine.edu> <20150420050651.GJ72050@icir.org>
	<CANFQkQvdcRZ_6jiKSe9s9RvU8QaX-4HGKZX+n+m8Ev_EES9Kig@mail.gmail.com>
	<20150618154141.GH1606@icir.org>
Message-ID: <CANFQkQuxfM=o-E86RejJbakCBq_jr4M4q7QMW=tim+Thz2yv4Q@mail.gmail.com>

Hello Robin,

Thanks for your answer.
In fact I have used the http://github.org/rsmmr/install-clang script, so
I'm afraid it is not the last HILTI sources.

I will try with the last HILTI sources if you say it is fixed on them.


2015-06-18 17:41 GMT+02:00 Robin Sommer <robin at icir.org>:

>
>
> On Thu, Jun 18, 2015 at 16:27 +0200, A Bruno wrote:
>
> > Robin, Troy, have you found a solution on this problem that I could apply
> > on a non docker context?
>
> I was hoping it's fixed now, both inside and outside of Docker. Are you
> using the most recent HILTI/BinPAC++ version?
>
> The Docker image now seems to be working for folks.
>
> Robin
>
> --
> Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150618/42e7174d/attachment.html 

From seth at icir.org  Thu Jun 18 11:10:20 2015
From: seth at icir.org (Seth Hall)
Date: Thu, 18 Jun 2015 14:10:20 -0400
Subject: [Bro] Subclassing from SSL Analyzer
In-Reply-To: <CAOfiKk+fXjh8=7FUcDbKvU2giygPbQkVceJ3sCZ+j2dgxwN4VA@mail.gmail.com>
References: <CA+rfBVxymMgEOdftG7h5qqzTRpozqZNcarST1yC3wsSdt8vMnA@mail.gmail.com>
	<AE36A573-9B69-4BBB-B385-AD3832025BFD@icir.org>
	<CAOfiKk+fXjh8=7FUcDbKvU2giygPbQkVceJ3sCZ+j2dgxwN4VA@mail.gmail.com>
Message-ID: <B69E98F0-1CEE-47B2-8E1D-14CB2AA0168E@icir.org>


> On Jun 18, 2015, at 1:41 PM, Nikunj Bansal <nikunj.bansal at gmail.com> wrote:
> 
> We are trying to use openssl to get it going though. Do you see any technical issues with that?

Nope, since OpenSSL is already a required dependency that?s how we?d require it to be implemented anyway. :)

> We could also make it pluggable/decoupled from openssl enough that it could be turned on only when required with the openssl's crypto and ssl DLLs being provided by the specific installation.

No need since we require OpenSSL anyway.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150618/d8f043f3/attachment.bin 

From andrew_duba at wustl.edu  Thu Jun 18 11:13:50 2015
From: andrew_duba at wustl.edu (Duba, Andrew)
Date: Thu, 18 Jun 2015 18:13:50 +0000
Subject: [Bro] logs in bro/spool/manager not consistent with archived logs
Message-ID: <D1A8748C.11B45%Andrew_Duba@wustl.edu>

I'm running bro in my test environment and if I do an ls on the directory where current logs are supposed to be stored I get this

root at spot:/usr/local/bro/logs# ls /usr/local/bro/spool/manager

communication.log  loaded_scripts.log  reporter.log  stderr.log  stdout.log


If I run  an ls in one of the archived directories I get this

app_stats.00:00:00-01:00:00.log.gz      conn.06:00:00-07:00:00.log.gz          dpd.07:00:00-08:00:00.log.gz          known_services.00:00:00-01:00:00.log.gz  reporter.12:49:56-12:58:35.log.gz  ssl.12:00:00-13:00:00.log.gz

app_stats.01:00:00-02:00:00.log.gz      conn.07:00:00-08:00:00.log.gz          dpd.08:00:00-09:00:00.log.gz          known_services.01:00:00-02:00:00.log.gz  reporter.13:02:38-13:06:00.log.gz  tunnel.07:00:00-08:00:00.log.gz

app_stats.02:00:00-03:00:00.log.gz      conn.08:00:00-09:00:00.log.gz          dpd.09:00:00-10:00:00.log.gz          known_services.09:00:00-10:00:00.log.gz  snmp.00:00:00-01:00:00.log.gz      tunnel.08:00:00-09:00:00.log.gz

app_stats.03:00:00-04:00:00.log.gz      conn.09:00:00-10:00:00.log.gz          dpd.10:00:00-11:00:00.log.gz          known_services.12:00:00-13:00:00.log.gz  snmp.01:00:00-02:00:00.log.gz      tunnel.10:00:00-11:00:00.log.gz

app_stats.04:00:00-05:00:00.log.gz      conn.10:00:00-11:00:00.log.gz          dpd.11:00:00-12:00:00.log.gz          loaded_scripts.12:45:56-12:58:35.log.gz  snmp.02:00:00-03:00:00.log.gz      tunnel.11:00:00-12:00:00.log.gz

app_stats.05:00:00-06:00:00.log.gz      conn.11:00:00-12:00:00.log.gz          dpd.12:00:00-13:00:00.log.gz          loaded_scripts.12:58:38-13:00:00.log.gz  snmp.03:00:00-04:00:00.log.gz      tunnel.12:00:00-13:00:00.log.gz

app_stats.06:00:00-07:00:00.log.gz      conn.12:00:00-13:00:00.log.gz          files.00:00:00-01:00:00.log.gz        notice.00:00:00-01:00:00.log.gz          snmp.09:00:00-10:00:00.log.gz      weird.00:00:00-01:00:00.log.gz

app_stats.07:00:00-08:00:00.log.gz      conn-summary.00:00:00-01:00:00.log.gz  files.01:00:00-02:00:00.log.gz        notice.01:00:00-02:00:00.log.gz          snmp.10:00:00-11:00:00.log.gz      weird.01:00:00-02:00:00.log.gz

app_stats.08:00:00-09:00:00.log.gz      conn-summary.01:00:00-02:00:00.log.gz  files.02:00:00-03:00:00.log.gz        notice.02:00:00-03:00:00.log.gz          snmp.11:00:00-12:00:00.log.gz      weird.02:00:00-03:00:00.log.gz

app_stats.09:00:00-10:00:00.log.gz      conn-summary.02:00:00-03:00:00.log.gz  files.03:00:00-04:00:00.log.gz        notice.03:00:00-04:00:00.log.gz          software.00:00:00-01:00:00.log.gz  weird.03:00:00-04:00:00.log.gz

app_stats.10:00:00-11:00:00.log.gz      conn-summary.03:00:00-04:00:00.log.gz  files.04:00:00-05:00:00.log.gz        notice.04:00:00-05:00:00.log.gz          software.01:00:00-02:00:00.log.gz  weird.04:00:00-05:00:00.log.gz

...

Is there a configuration directive that I'm missing?

Thanks in advance for any help.

-Andrew

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150618/b07bef33/attachment-0001.html 

From dnthayer at illinois.edu  Thu Jun 18 11:23:48 2015
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Thu, 18 Jun 2015 13:23:48 -0500
Subject: [Bro] logs in bro/spool/manager not consistent with archived
 logs
In-Reply-To: <D1A8748C.11B45%Andrew_Duba@wustl.edu>
References: <D1A8748C.11B45%Andrew_Duba@wustl.edu>
Message-ID: <55830CB4.7020200@illinois.edu>

The directory "spool/manager" is where the current (i.e., active) logs
are located.  The "logs" directory is where the archived logs are
located.  Logs are archived according to the log rotation interval
specified in your configuration.


On 06/18/2015 01:13 PM, Duba, Andrew wrote:
> I?m running bro in my test environment and if I do an ls on the
> directory where current logs are supposed to be stored I get this
>
> root at spot:/usr/local/bro/logs# ls /usr/local/bro/spool/manager
>
> communication.log  loaded_scripts.log  reporter.log  stderr.log  stdout.log
>
>
>
> If I run  an ls in one of the archived directories I get this
>
> app_stats.00:00:00-01:00:00.log.gzconn.06:00:00-07:00:00.log.gzdpd.07:00:00-08:00:00.log.gzknown_services.00:00:00-01:00:00.log.gzreporter.12:49:56-12:58:35.log.gzssl.12:00:00-13:00:00.log.gz
>
> app_stats.01:00:00-02:00:00.log.gzconn.07:00:00-08:00:00.log.gzdpd.08:00:00-09:00:00.log.gzknown_services.01:00:00-02:00:00.log.gzreporter.13:02:38-13:06:00.log.gztunnel.07:00:00-08:00:00.log.gz
>
> app_stats.02:00:00-03:00:00.log.gzconn.08:00:00-09:00:00.log.gzdpd.09:00:00-10:00:00.log.gzknown_services.09:00:00-10:00:00.log.gzsnmp.00:00:00-01:00:00.log.gztunnel.08:00:00-09:00:00.log.gz
>
> app_stats.03:00:00-04:00:00.log.gzconn.09:00:00-10:00:00.log.gzdpd.10:00:00-11:00:00.log.gzknown_services.12:00:00-13:00:00.log.gzsnmp.01:00:00-02:00:00.log.gztunnel.10:00:00-11:00:00.log.gz
>
> app_stats.04:00:00-05:00:00.log.gzconn.10:00:00-11:00:00.log.gzdpd.11:00:00-12:00:00.log.gzloaded_scripts.12:45:56-12:58:35.log.gzsnmp.02:00:00-03:00:00.log.gztunnel.11:00:00-12:00:00.log.gz
>
> app_stats.05:00:00-06:00:00.log.gzconn.11:00:00-12:00:00.log.gzdpd.12:00:00-13:00:00.log.gzloaded_scripts.12:58:38-13:00:00.log.gzsnmp.03:00:00-04:00:00.log.gztunnel.12:00:00-13:00:00.log.gz
>
> app_stats.06:00:00-07:00:00.log.gzconn.12:00:00-13:00:00.log.gzfiles.00:00:00-01:00:00.log.gznotice.00:00:00-01:00:00.log.gzsnmp.09:00:00-10:00:00.log.gzweird.00:00:00-01:00:00.log.gz
>
> app_stats.07:00:00-08:00:00.log.gzconn-summary.00:00:00-01:00:00.log.gzfiles.01:00:00-02:00:00.log.gznotice.01:00:00-02:00:00.log.gzsnmp.10:00:00-11:00:00.log.gzweird.01:00:00-02:00:00.log.gz
>
> app_stats.08:00:00-09:00:00.log.gzconn-summary.01:00:00-02:00:00.log.gzfiles.02:00:00-03:00:00.log.gznotice.02:00:00-03:00:00.log.gzsnmp.11:00:00-12:00:00.log.gzweird.02:00:00-03:00:00.log.gz
>
> app_stats.09:00:00-10:00:00.log.gzconn-summary.02:00:00-03:00:00.log.gzfiles.03:00:00-04:00:00.log.gznotice.03:00:00-04:00:00.log.gzsoftware.00:00:00-01:00:00.log.gzweird.03:00:00-04:00:00.log.gz
>
> app_stats.10:00:00-11:00:00.log.gzconn-summary.03:00:00-04:00:00.log.gzfiles.04:00:00-05:00:00.log.gznotice.04:00:00-05:00:00.log.gzsoftware.01:00:00-02:00:00.log.gzweird.04:00:00-05:00:00.log.gz
>
> ?
>
>
> Is there a configuration directive that I?m missing?
>
> Thanks in advance for any help.
>
> -Andrew
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>

From andrew_duba at wustl.edu  Thu Jun 18 11:46:11 2015
From: andrew_duba at wustl.edu (Duba, Andrew)
Date: Thu, 18 Jun 2015 18:46:11 +0000
Subject: [Bro] logs in bro/spool/manager not consistent with archived
	logs
In-Reply-To: <55830CB4.7020200@illinois.edu>
References: <D1A8748C.11B45%Andrew_Duba@wustl.edu>
	<55830CB4.7020200@illinois.edu>
Message-ID: <D1A87B58.11B84%Andrew_Duba@wustl.edu>

Right.  The ?logs? directory has compressed versions of the files that are
under ?current? but all I?m seeing under current are the 5 logs which do
not map to the naming scheme in the archived directories.

-Andrew

On 6/18/15, 1:23 PM, "Daniel Thayer" <dnthayer at illinois.edu> wrote:

>The directory "spool/manager" is where the current (i.e., active) logs
>are located.  The "logs" directory is where the archived logs are
>located.  Logs are archived according to the log rotation interval
>specified in your configuration.
>
>
>On 06/18/2015 01:13 PM, Duba, Andrew wrote:
>> I?m running bro in my test environment and if I do an ls on the
>> directory where current logs are supposed to be stored I get this
>>
>> root at spot:/usr/local/bro/logs# ls /usr/local/bro/spool/manager
>>
>> communication.log  loaded_scripts.log  reporter.log  stderr.log
>>stdout.log
>>
>>
>>
>> If I run  an ls in one of the archived directories I get this
>>
>> 
>>app_stats.00:00:00-01:00:00.log.gzconn.06:00:00-07:00:00.log.gzdpd.07:00:
>>00-08:00:00.log.gzknown_services.00:00:00-01:00:00.log.gzreporter.12:49:5
>>6-12:58:35.log.gzssl.12:00:00-13:00:00.log.gz
>>
>> 
>>app_stats.01:00:00-02:00:00.log.gzconn.07:00:00-08:00:00.log.gzdpd.08:00:
>>00-09:00:00.log.gzknown_services.01:00:00-02:00:00.log.gzreporter.13:02:3
>>8-13:06:00.log.gztunnel.07:00:00-08:00:00.log.gz
>>
>> 
>>app_stats.02:00:00-03:00:00.log.gzconn.08:00:00-09:00:00.log.gzdpd.09:00:
>>00-10:00:00.log.gzknown_services.09:00:00-10:00:00.log.gzsnmp.00:00:00-01
>>:00:00.log.gztunnel.08:00:00-09:00:00.log.gz
>>
>> 
>>app_stats.03:00:00-04:00:00.log.gzconn.09:00:00-10:00:00.log.gzdpd.10:00:
>>00-11:00:00.log.gzknown_services.12:00:00-13:00:00.log.gzsnmp.01:00:00-02
>>:00:00.log.gztunnel.10:00:00-11:00:00.log.gz
>>
>> 
>>app_stats.04:00:00-05:00:00.log.gzconn.10:00:00-11:00:00.log.gzdpd.11:00:
>>00-12:00:00.log.gzloaded_scripts.12:45:56-12:58:35.log.gzsnmp.02:00:00-03
>>:00:00.log.gztunnel.11:00:00-12:00:00.log.gz
>>
>> 
>>app_stats.05:00:00-06:00:00.log.gzconn.11:00:00-12:00:00.log.gzdpd.12:00:
>>00-13:00:00.log.gzloaded_scripts.12:58:38-13:00:00.log.gzsnmp.03:00:00-04
>>:00:00.log.gztunnel.12:00:00-13:00:00.log.gz
>>
>> 
>>app_stats.06:00:00-07:00:00.log.gzconn.12:00:00-13:00:00.log.gzfiles.00:0
>>0:00-01:00:00.log.gznotice.00:00:00-01:00:00.log.gzsnmp.09:00:00-10:00:00
>>.log.gzweird.00:00:00-01:00:00.log.gz
>>
>> 
>>app_stats.07:00:00-08:00:00.log.gzconn-summary.00:00:00-01:00:00.log.gzfi
>>les.01:00:00-02:00:00.log.gznotice.01:00:00-02:00:00.log.gzsnmp.10:00:00-
>>11:00:00.log.gzweird.01:00:00-02:00:00.log.gz
>>
>> 
>>app_stats.08:00:00-09:00:00.log.gzconn-summary.01:00:00-02:00:00.log.gzfi
>>les.02:00:00-03:00:00.log.gznotice.02:00:00-03:00:00.log.gzsnmp.11:00:00-
>>12:00:00.log.gzweird.02:00:00-03:00:00.log.gz
>>
>> 
>>app_stats.09:00:00-10:00:00.log.gzconn-summary.02:00:00-03:00:00.log.gzfi
>>les.03:00:00-04:00:00.log.gznotice.03:00:00-04:00:00.log.gzsoftware.00:00
>>:00-01:00:00.log.gzweird.03:00:00-04:00:00.log.gz
>>
>> 
>>app_stats.10:00:00-11:00:00.log.gzconn-summary.03:00:00-04:00:00.log.gzfi
>>les.04:00:00-05:00:00.log.gznotice.04:00:00-05:00:00.log.gzsoftware.01:00
>>:00-02:00:00.log.gzweird.04:00:00-05:00:00.log.gz
>>
>> ?
>>
>>
>> Is there a configuration directive that I?m missing?
>>
>> Thanks in advance for any help.
>>
>> -Andrew
>>
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>



From dnthayer at illinois.edu  Thu Jun 18 12:09:40 2015
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Thu, 18 Jun 2015 14:09:40 -0500
Subject: [Bro] logs in bro/spool/manager not consistent with archived
 logs
In-Reply-To: <D1A87B58.11B84%Andrew_Duba@wustl.edu>
References: <D1A8748C.11B45%Andrew_Duba@wustl.edu>	<55830CB4.7020200@illinois.edu>
	<D1A87B58.11B84%Andrew_Duba@wustl.edu>
Message-ID: <55831774.1050405@illinois.edu>

Correct.  The naming convention used for the archived logs
is to organize them by day (each day gets its own subdirectory under
the "logs" directory), and the filename of each log contains
the time range of that log.  For example, conn.06:00:00-07:00:00.log.gz
is the conn.log for the time period 6:00am to 7:00am.


On 06/18/2015 01:46 PM, Duba, Andrew wrote:
> Right.  The ?logs? directory has compressed versions of the files that are
> under ?current? but all I?m seeing under current are the 5 logs which do
> not map to the naming scheme in the archived directories.
>
> -Andrew
>
> On 6/18/15, 1:23 PM, "Daniel Thayer" <dnthayer at illinois.edu> wrote:
>
>> The directory "spool/manager" is where the current (i.e., active) logs
>> are located.  The "logs" directory is where the archived logs are
>> located.  Logs are archived according to the log rotation interval
>> specified in your configuration.
>>
>>
>> On 06/18/2015 01:13 PM, Duba, Andrew wrote:
>>> I?m running bro in my test environment and if I do an ls on the
>>> directory where current logs are supposed to be stored I get this
>>>
>>> root at spot:/usr/local/bro/logs# ls /usr/local/bro/spool/manager
>>>
>>> communication.log  loaded_scripts.log  reporter.log  stderr.log
>>> stdout.log
>>>
>>>
>>>
>>> If I run  an ls in one of the archived directories I get this
>>>
>>>
>>> app_stats.00:00:00-01:00:00.log.gzconn.06:00:00-07:00:00.log.gzdpd.07:00:
>>> 00-08:00:00.log.gzknown_services.00:00:00-01:00:00.log.gzreporter.12:49:5
>>> 6-12:58:35.log.gzssl.12:00:00-13:00:00.log.gz
>>>
>>>
>>> app_stats.01:00:00-02:00:00.log.gzconn.07:00:00-08:00:00.log.gzdpd.08:00:
>>> 00-09:00:00.log.gzknown_services.01:00:00-02:00:00.log.gzreporter.13:02:3
>>> 8-13:06:00.log.gztunnel.07:00:00-08:00:00.log.gz
>>>
>>>
>>> app_stats.02:00:00-03:00:00.log.gzconn.08:00:00-09:00:00.log.gzdpd.09:00:
>>> 00-10:00:00.log.gzknown_services.09:00:00-10:00:00.log.gzsnmp.00:00:00-01
>>> :00:00.log.gztunnel.08:00:00-09:00:00.log.gz
>>>
>>>
>>> app_stats.03:00:00-04:00:00.log.gzconn.09:00:00-10:00:00.log.gzdpd.10:00:
>>> 00-11:00:00.log.gzknown_services.12:00:00-13:00:00.log.gzsnmp.01:00:00-02
>>> :00:00.log.gztunnel.10:00:00-11:00:00.log.gz
>>>
>>>
>>> app_stats.04:00:00-05:00:00.log.gzconn.10:00:00-11:00:00.log.gzdpd.11:00:
>>> 00-12:00:00.log.gzloaded_scripts.12:45:56-12:58:35.log.gzsnmp.02:00:00-03
>>> :00:00.log.gztunnel.11:00:00-12:00:00.log.gz
>>>
>>>
>>> app_stats.05:00:00-06:00:00.log.gzconn.11:00:00-12:00:00.log.gzdpd.12:00:
>>> 00-13:00:00.log.gzloaded_scripts.12:58:38-13:00:00.log.gzsnmp.03:00:00-04
>>> :00:00.log.gztunnel.12:00:00-13:00:00.log.gz
>>>
>>>
>>> app_stats.06:00:00-07:00:00.log.gzconn.12:00:00-13:00:00.log.gzfiles.00:0
>>> 0:00-01:00:00.log.gznotice.00:00:00-01:00:00.log.gzsnmp.09:00:00-10:00:00
>>> .log.gzweird.00:00:00-01:00:00.log.gz
>>>
>>>
>>> app_stats.07:00:00-08:00:00.log.gzconn-summary.00:00:00-01:00:00.log.gzfi
>>> les.01:00:00-02:00:00.log.gznotice.01:00:00-02:00:00.log.gzsnmp.10:00:00-
>>> 11:00:00.log.gzweird.01:00:00-02:00:00.log.gz
>>>
>>>
>>> app_stats.08:00:00-09:00:00.log.gzconn-summary.01:00:00-02:00:00.log.gzfi
>>> les.02:00:00-03:00:00.log.gznotice.02:00:00-03:00:00.log.gzsnmp.11:00:00-
>>> 12:00:00.log.gzweird.02:00:00-03:00:00.log.gz
>>>
>>>
>>> app_stats.09:00:00-10:00:00.log.gzconn-summary.02:00:00-03:00:00.log.gzfi
>>> les.03:00:00-04:00:00.log.gznotice.03:00:00-04:00:00.log.gzsoftware.00:00
>>> :00-01:00:00.log.gzweird.03:00:00-04:00:00.log.gz
>>>
>>>
>>> app_stats.10:00:00-11:00:00.log.gzconn-summary.03:00:00-04:00:00.log.gzfi
>>> les.04:00:00-05:00:00.log.gznotice.04:00:00-05:00:00.log.gzsoftware.01:00
>>> :00-02:00:00.log.gzweird.04:00:00-05:00:00.log.gz
>>>
>>> ?
>>>
>>>
>>> Is there a configuration directive that I?m missing?
>>>
>>> Thanks in advance for any help.
>>>
>>> -Andrew
>>>
>>>
>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>

From andrew_duba at wustl.edu  Thu Jun 18 12:17:46 2015
From: andrew_duba at wustl.edu (Duba, Andrew)
Date: Thu, 18 Jun 2015 19:17:46 +0000
Subject: [Bro] logs in bro/spool/manager not consistent with archived
	logs
In-Reply-To: <55831774.1050405@illinois.edu>
References: <D1A8748C.11B45%Andrew_Duba@wustl.edu>
	<55830CB4.7020200@illinois.edu> <D1A87B58.11B84%Andrew_Duba@wustl.edu>
	<55831774.1050405@illinois.edu>
Message-ID: <D1A882F9.11BD0%Andrew_Duba@wustl.edu>

So why is it I?m not getting a conn.log in the "current" directory but I?m
getting conn.xx:xx:xx-yy:yy:yy.log.gz in the archive directories?  Is
there some kind of a directive that I need to set that I?m missing?

-Andrew

On 6/18/15, 2:09 PM, "Daniel Thayer" <dnthayer at illinois.edu> wrote:

>Correct.  The naming convention used for the archived logs
>is to organize them by day (each day gets its own subdirectory under
>the "logs" directory), and the filename of each log contains
>the time range of that log.  For example, conn.06:00:00-07:00:00.log.gz
>is the conn.log for the time period 6:00am to 7:00am.
>
>
>On 06/18/2015 01:46 PM, Duba, Andrew wrote:
>> Right.  The ?logs? directory has compressed versions of the files that
>>are
>> under ?current? but all I?m seeing under current are the 5 logs which do
>> not map to the naming scheme in the archived directories.
>>
>> -Andrew
>>
>> On 6/18/15, 1:23 PM, "Daniel Thayer" <dnthayer at illinois.edu> wrote:
>>
>>> The directory "spool/manager" is where the current (i.e., active) logs
>>> are located.  The "logs" directory is where the archived logs are
>>> located.  Logs are archived according to the log rotation interval
>>> specified in your configuration.
>>>
>>>
>>> On 06/18/2015 01:13 PM, Duba, Andrew wrote:
>>>> I?m running bro in my test environment and if I do an ls on the
>>>> directory where current logs are supposed to be stored I get this
>>>>
>>>> root at spot:/usr/local/bro/logs# ls /usr/local/bro/spool/manager
>>>>
>>>> communication.log  loaded_scripts.log  reporter.log  stderr.log
>>>> stdout.log
>>>>
>>>>
>>>>
>>>> If I run  an ls in one of the archived directories I get this
>>>>
>>>>
>>>> 
>>>>app_stats.00:00:00-01:00:00.log.gzconn.06:00:00-07:00:00.log.gzdpd.07:0
>>>>0:
>>>> 
>>>>00-08:00:00.log.gzknown_services.00:00:00-01:00:00.log.gzreporter.12:49
>>>>:5
>>>> 6-12:58:35.log.gzssl.12:00:00-13:00:00.log.gz
>>>>
>>>>
>>>> 
>>>>app_stats.01:00:00-02:00:00.log.gzconn.07:00:00-08:00:00.log.gzdpd.08:0
>>>>0:
>>>> 
>>>>00-09:00:00.log.gzknown_services.01:00:00-02:00:00.log.gzreporter.13:02
>>>>:3
>>>> 8-13:06:00.log.gztunnel.07:00:00-08:00:00.log.gz
>>>>
>>>>
>>>> 
>>>>app_stats.02:00:00-03:00:00.log.gzconn.08:00:00-09:00:00.log.gzdpd.09:0
>>>>0:
>>>> 
>>>>00-10:00:00.log.gzknown_services.09:00:00-10:00:00.log.gzsnmp.00:00:00-
>>>>01
>>>> :00:00.log.gztunnel.08:00:00-09:00:00.log.gz
>>>>
>>>>
>>>> 
>>>>app_stats.03:00:00-04:00:00.log.gzconn.09:00:00-10:00:00.log.gzdpd.10:0
>>>>0:
>>>> 
>>>>00-11:00:00.log.gzknown_services.12:00:00-13:00:00.log.gzsnmp.01:00:00-
>>>>02
>>>> :00:00.log.gztunnel.10:00:00-11:00:00.log.gz
>>>>
>>>>
>>>> 
>>>>app_stats.04:00:00-05:00:00.log.gzconn.10:00:00-11:00:00.log.gzdpd.11:0
>>>>0:
>>>> 
>>>>00-12:00:00.log.gzloaded_scripts.12:45:56-12:58:35.log.gzsnmp.02:00:00-
>>>>03
>>>> :00:00.log.gztunnel.11:00:00-12:00:00.log.gz
>>>>
>>>>
>>>> 
>>>>app_stats.05:00:00-06:00:00.log.gzconn.11:00:00-12:00:00.log.gzdpd.12:0
>>>>0:
>>>> 
>>>>00-13:00:00.log.gzloaded_scripts.12:58:38-13:00:00.log.gzsnmp.03:00:00-
>>>>04
>>>> :00:00.log.gztunnel.12:00:00-13:00:00.log.gz
>>>>
>>>>
>>>> 
>>>>app_stats.06:00:00-07:00:00.log.gzconn.12:00:00-13:00:00.log.gzfiles.00
>>>>:0
>>>> 
>>>>0:00-01:00:00.log.gznotice.00:00:00-01:00:00.log.gzsnmp.09:00:00-10:00:
>>>>00
>>>> .log.gzweird.00:00:00-01:00:00.log.gz
>>>>
>>>>
>>>> 
>>>>app_stats.07:00:00-08:00:00.log.gzconn-summary.00:00:00-01:00:00.log.gz
>>>>fi
>>>> 
>>>>les.01:00:00-02:00:00.log.gznotice.01:00:00-02:00:00.log.gzsnmp.10:00:0
>>>>0-
>>>> 11:00:00.log.gzweird.01:00:00-02:00:00.log.gz
>>>>
>>>>
>>>> 
>>>>app_stats.08:00:00-09:00:00.log.gzconn-summary.01:00:00-02:00:00.log.gz
>>>>fi
>>>> 
>>>>les.02:00:00-03:00:00.log.gznotice.02:00:00-03:00:00.log.gzsnmp.11:00:0
>>>>0-
>>>> 12:00:00.log.gzweird.02:00:00-03:00:00.log.gz
>>>>
>>>>
>>>> 
>>>>app_stats.09:00:00-10:00:00.log.gzconn-summary.02:00:00-03:00:00.log.gz
>>>>fi
>>>> 
>>>>les.03:00:00-04:00:00.log.gznotice.03:00:00-04:00:00.log.gzsoftware.00:
>>>>00
>>>> :00-01:00:00.log.gzweird.03:00:00-04:00:00.log.gz
>>>>
>>>>
>>>> 
>>>>app_stats.10:00:00-11:00:00.log.gzconn-summary.03:00:00-04:00:00.log.gz
>>>>fi
>>>> 
>>>>les.04:00:00-05:00:00.log.gznotice.04:00:00-05:00:00.log.gzsoftware.01:
>>>>00
>>>> :00-02:00:00.log.gzweird.04:00:00-05:00:00.log.gz
>>>>
>>>> ?
>>>>
>>>>
>>>> Is there a configuration directive that I?m missing?
>>>>
>>>> Thanks in advance for any help.
>>>>
>>>> -Andrew
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Bro mailing list
>>>> bro at bro-ids.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>>
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>



From dnthayer at illinois.edu  Thu Jun 18 12:40:08 2015
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Thu, 18 Jun 2015 14:40:08 -0500
Subject: [Bro] logs in bro/spool/manager not consistent with archived
 logs
In-Reply-To: <D1A882F9.11BD0%Andrew_Duba@wustl.edu>
References: <D1A8748C.11B45%Andrew_Duba@wustl.edu>	<55830CB4.7020200@illinois.edu>
	<D1A87B58.11B84%Andrew_Duba@wustl.edu>	<55831774.1050405@illinois.edu>
	<D1A882F9.11BD0%Andrew_Duba@wustl.edu>
Message-ID: <55831E98.8050703@illinois.edu>

There is no special setting needed to get Bro to log
to conn.log.

The "current" conn.log is the log that Bro is writing now,
so if you don't see that file, then that would indicate that
Bro hasn't written anything to that log since the last log
rotation (by default, logs are rotated once per hour).
However, it is quite unusual to not see a conn.log, which
may indicate a problem with your setup.  If your Bro never
writes to conn.log, then you would not see any archived
conn.log either.



On 06/18/2015 02:17 PM, Duba, Andrew wrote:
> So why is it I?m not getting a conn.log in the "current" directory but I?m
> getting conn.xx:xx:xx-yy:yy:yy.log.gz in the archive directories?  Is
> there some kind of a directive that I need to set that I?m missing?
>
> -Andrew
>
> On 6/18/15, 2:09 PM, "Daniel Thayer" <dnthayer at illinois.edu> wrote:
>
>> Correct.  The naming convention used for the archived logs
>> is to organize them by day (each day gets its own subdirectory under
>> the "logs" directory), and the filename of each log contains
>> the time range of that log.  For example, conn.06:00:00-07:00:00.log.gz
>> is the conn.log for the time period 6:00am to 7:00am.
>>
>>
>> On 06/18/2015 01:46 PM, Duba, Andrew wrote:
>>> Right.  The ?logs? directory has compressed versions of the files that
>>> are
>>> under ?current? but all I?m seeing under current are the 5 logs which do
>>> not map to the naming scheme in the archived directories.
>>>
>>> -Andrew
>>>
>>> On 6/18/15, 1:23 PM, "Daniel Thayer" <dnthayer at illinois.edu> wrote:
>>>
>>>> The directory "spool/manager" is where the current (i.e., active) logs
>>>> are located.  The "logs" directory is where the archived logs are
>>>> located.  Logs are archived according to the log rotation interval
>>>> specified in your configuration.
>>>>
>>>>
>>>> On 06/18/2015 01:13 PM, Duba, Andrew wrote:
>>>>> I?m running bro in my test environment and if I do an ls on the
>>>>> directory where current logs are supposed to be stored I get this
>>>>>
>>>>> root at spot:/usr/local/bro/logs# ls /usr/local/bro/spool/manager
>>>>>
>>>>> communication.log  loaded_scripts.log  reporter.log  stderr.log
>>>>> stdout.log
>>>>>
>>>>>
>>>>>
>>>>> If I run  an ls in one of the archived directories I get this
>>>>>
>>>>>
>>>>>
>>>>> app_stats.00:00:00-01:00:00.log.gzconn.06:00:00-07:00:00.log.gzdpd.07:0
>>>>> 0:
>>>>>
>>>>> 00-08:00:00.log.gzknown_services.00:00:00-01:00:00.log.gzreporter.12:49
>>>>> :5
>>>>> 6-12:58:35.log.gzssl.12:00:00-13:00:00.log.gz
>>>>>
>>>>>
>>>>>
>>>>> app_stats.01:00:00-02:00:00.log.gzconn.07:00:00-08:00:00.log.gzdpd.08:0
>>>>> 0:
>>>>>
>>>>> 00-09:00:00.log.gzknown_services.01:00:00-02:00:00.log.gzreporter.13:02
>>>>> :3
>>>>> 8-13:06:00.log.gztunnel.07:00:00-08:00:00.log.gz
>>>>>
>>>>>
>>>>>
>>>>> app_stats.02:00:00-03:00:00.log.gzconn.08:00:00-09:00:00.log.gzdpd.09:0
>>>>> 0:
>>>>>
>>>>> 00-10:00:00.log.gzknown_services.09:00:00-10:00:00.log.gzsnmp.00:00:00-
>>>>> 01
>>>>> :00:00.log.gztunnel.08:00:00-09:00:00.log.gz
>>>>>
>>>>>
>>>>>
>>>>> app_stats.03:00:00-04:00:00.log.gzconn.09:00:00-10:00:00.log.gzdpd.10:0
>>>>> 0:
>>>>>
>>>>> 00-11:00:00.log.gzknown_services.12:00:00-13:00:00.log.gzsnmp.01:00:00-
>>>>> 02
>>>>> :00:00.log.gztunnel.10:00:00-11:00:00.log.gz
>>>>>
>>>>>
>>>>>
>>>>> app_stats.04:00:00-05:00:00.log.gzconn.10:00:00-11:00:00.log.gzdpd.11:0
>>>>> 0:
>>>>>
>>>>> 00-12:00:00.log.gzloaded_scripts.12:45:56-12:58:35.log.gzsnmp.02:00:00-
>>>>> 03
>>>>> :00:00.log.gztunnel.11:00:00-12:00:00.log.gz
>>>>>
>>>>>
>>>>>
>>>>> app_stats.05:00:00-06:00:00.log.gzconn.11:00:00-12:00:00.log.gzdpd.12:0
>>>>> 0:
>>>>>
>>>>> 00-13:00:00.log.gzloaded_scripts.12:58:38-13:00:00.log.gzsnmp.03:00:00-
>>>>> 04
>>>>> :00:00.log.gztunnel.12:00:00-13:00:00.log.gz
>>>>>
>>>>>
>>>>>
>>>>> app_stats.06:00:00-07:00:00.log.gzconn.12:00:00-13:00:00.log.gzfiles.00
>>>>> :0
>>>>>
>>>>> 0:00-01:00:00.log.gznotice.00:00:00-01:00:00.log.gzsnmp.09:00:00-10:00:
>>>>> 00
>>>>> .log.gzweird.00:00:00-01:00:00.log.gz
>>>>>
>>>>>
>>>>>
>>>>> app_stats.07:00:00-08:00:00.log.gzconn-summary.00:00:00-01:00:00.log.gz
>>>>> fi
>>>>>
>>>>> les.01:00:00-02:00:00.log.gznotice.01:00:00-02:00:00.log.gzsnmp.10:00:0
>>>>> 0-
>>>>> 11:00:00.log.gzweird.01:00:00-02:00:00.log.gz
>>>>>
>>>>>
>>>>>
>>>>> app_stats.08:00:00-09:00:00.log.gzconn-summary.01:00:00-02:00:00.log.gz
>>>>> fi
>>>>>
>>>>> les.02:00:00-03:00:00.log.gznotice.02:00:00-03:00:00.log.gzsnmp.11:00:0
>>>>> 0-
>>>>> 12:00:00.log.gzweird.02:00:00-03:00:00.log.gz
>>>>>
>>>>>
>>>>>
>>>>> app_stats.09:00:00-10:00:00.log.gzconn-summary.02:00:00-03:00:00.log.gz
>>>>> fi
>>>>>
>>>>> les.03:00:00-04:00:00.log.gznotice.03:00:00-04:00:00.log.gzsoftware.00:
>>>>> 00
>>>>> :00-01:00:00.log.gzweird.03:00:00-04:00:00.log.gz
>>>>>
>>>>>
>>>>>
>>>>> app_stats.10:00:00-11:00:00.log.gzconn-summary.03:00:00-04:00:00.log.gz
>>>>> fi
>>>>>
>>>>> les.04:00:00-05:00:00.log.gznotice.04:00:00-05:00:00.log.gzsoftware.01:
>>>>> 00
>>>>> :00-02:00:00.log.gzweird.04:00:00-05:00:00.log.gz
>>>>>
>>>>> ?
>>>>>
>>>>>
>>>>> Is there a configuration directive that I?m missing?
>>>>>
>>>>> Thanks in advance for any help.
>>>>>
>>>>> -Andrew
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Bro mailing list
>>>>> bro at bro-ids.org
>>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>>>
>>>
>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>

From andrew_duba at wustl.edu  Thu Jun 18 13:15:33 2015
From: andrew_duba at wustl.edu (Duba, Andrew)
Date: Thu, 18 Jun 2015 20:15:33 +0000
Subject: [Bro] logs in bro/spool/manager not consistent with archived
	logs
In-Reply-To: <55831E98.8050703@illinois.edu>
References: <D1A8748C.11B45%Andrew_Duba@wustl.edu>
	<55830CB4.7020200@illinois.edu> <D1A87B58.11B84%Andrew_Duba@wustl.edu>
	<55831774.1050405@illinois.edu> <D1A882F9.11BD0%Andrew_Duba@wustl.edu>
	<55831E98.8050703@illinois.edu>
Message-ID: <D1A88FF9.11C4E%Andrew_Duba@wustl.edu>

That?s the weird part. I have a complete set of conn logs that have been
archived (see below) and they have real data in them.
-rw-r--r-- 1 root root 1.5M Jun 18 01:00 conn.00:00:00-01:00:00.log.gz
-rw-r--r-- 1 root root 826K Jun 18 02:00 conn.01:00:00-02:00:00.log.gz
-rw-r--r-- 1 root root 443K Jun 18 03:00 conn.02:00:00-03:00:00.log.gz
-rw-r--r-- 1 root root 387K Jun 18 04:00 conn.03:00:00-04:00:00.log.gz
-rw-r--r-- 1 root root 312K Jun 18 05:00 conn.04:00:00-05:00:00.log.gz
-rw-r--r-- 1 root root 366K Jun 18 06:00 conn.05:00:00-06:00:00.log.gz
-rw-r--r-- 1 root root 501K Jun 18 07:00 conn.06:00:00-07:00:00.log.gz
-rw-r--r-- 1 root root 1.3M Jun 18 08:00 conn.07:00:00-08:00:00.log.gz
-rw-r--r-- 1 root root 1.5M Jun 18 09:00 conn.08:00:00-09:00:00.log.gz
-rw-r--r-- 1 root root 3.5M Jun 18 10:00 conn.09:00:00-10:00:00.log.gz
-rw-r--r-- 1 root root 3.6M Jun 18 11:00 conn.10:00:00-11:00:00.log.gz
-rw-r--r-- 1 root root 3.9M Jun 18 12:00 conn.11:00:00-12:00:00.log.gz
-rw-r--r-- 1 root root 6.4M Jun 18 13:00 conn.12:00:00-13:00:00.log.gz
-rw-r--r-- 1 root root 3.7M Jun 18 14:00 conn.13:00:00-14:00:00.log.gz
-rw-r--r-- 1 root root 4.1M Jun 18 15:00 conn.14:00:00-15:00:00.log.gz



But no current/conn.log.  This is a real head scratcher.

-Andrew

On 6/18/15, 2:40 PM, "Daniel Thayer" <dnthayer at illinois.edu> wrote:

>There is no special setting needed to get Bro to log
>to conn.log.
>
>The "current" conn.log is the log that Bro is writing now,
>so if you don't see that file, then that would indicate that
>Bro hasn't written anything to that log since the last log
>rotation (by default, logs are rotated once per hour).
>However, it is quite unusual to not see a conn.log, which
>may indicate a problem with your setup.  If your Bro never
>writes to conn.log, then you would not see any archived
>conn.log either.
>
>
>
>On 06/18/2015 02:17 PM, Duba, Andrew wrote:
>> So why is it I?m not getting a conn.log in the "current" directory but
>>I?m
>> getting conn.xx:xx:xx-yy:yy:yy.log.gz in the archive directories?  Is
>> there some kind of a directive that I need to set that I?m missing?
>>
>> -Andrew
>>
>> On 6/18/15, 2:09 PM, "Daniel Thayer" <dnthayer at illinois.edu> wrote:
>>
>>> Correct.  The naming convention used for the archived logs
>>> is to organize them by day (each day gets its own subdirectory under
>>> the "logs" directory), and the filename of each log contains
>>> the time range of that log.  For example, conn.06:00:00-07:00:00.log.gz
>>> is the conn.log for the time period 6:00am to 7:00am.
>>>
>>>
>>> On 06/18/2015 01:46 PM, Duba, Andrew wrote:
>>>> Right.  The ?logs? directory has compressed versions of the files that
>>>> are
>>>> under ?current? but all I?m seeing under current are the 5 logs which
>>>>do
>>>> not map to the naming scheme in the archived directories.
>>>>
>>>> -Andrew
>>>>
>>>> On 6/18/15, 1:23 PM, "Daniel Thayer" <dnthayer at illinois.edu> wrote:
>>>>
>>>>> The directory "spool/manager" is where the current (i.e., active)
>>>>>logs
>>>>> are located.  The "logs" directory is where the archived logs are
>>>>> located.  Logs are archived according to the log rotation interval
>>>>> specified in your configuration.
>>>>>
>>>>>
>>>>> On 06/18/2015 01:13 PM, Duba, Andrew wrote:
>>>>>> I?m running bro in my test environment and if I do an ls on the
>>>>>> directory where current logs are supposed to be stored I get this
>>>>>>
>>>>>> root at spot:/usr/local/bro/logs# ls /usr/local/bro/spool/manager
>>>>>>
>>>>>> communication.log  loaded_scripts.log  reporter.log  stderr.log
>>>>>> stdout.log
>>>>>>
>>>>>>
>>>>>>
>>>>>> If I run  an ls in one of the archived directories I get this
>>>>>>
>>>>>>
>>>>>>
>>>>>> 
>>>>>>app_stats.00:00:00-01:00:00.log.gzconn.06:00:00-07:00:00.log.gzdpd.07
>>>>>>:0
>>>>>> 0:
>>>>>>
>>>>>> 
>>>>>>00-08:00:00.log.gzknown_services.00:00:00-01:00:00.log.gzreporter.12:
>>>>>>49
>>>>>> :5
>>>>>> 6-12:58:35.log.gzssl.12:00:00-13:00:00.log.gz
>>>>>>
>>>>>>
>>>>>>
>>>>>> 
>>>>>>app_stats.01:00:00-02:00:00.log.gzconn.07:00:00-08:00:00.log.gzdpd.08
>>>>>>:0
>>>>>> 0:
>>>>>>
>>>>>> 
>>>>>>00-09:00:00.log.gzknown_services.01:00:00-02:00:00.log.gzreporter.13:
>>>>>>02
>>>>>> :3
>>>>>> 8-13:06:00.log.gztunnel.07:00:00-08:00:00.log.gz
>>>>>>
>>>>>>
>>>>>>
>>>>>> 
>>>>>>app_stats.02:00:00-03:00:00.log.gzconn.08:00:00-09:00:00.log.gzdpd.09
>>>>>>:0
>>>>>> 0:
>>>>>>
>>>>>> 
>>>>>>00-10:00:00.log.gzknown_services.09:00:00-10:00:00.log.gzsnmp.00:00:0
>>>>>>0-
>>>>>> 01
>>>>>> :00:00.log.gztunnel.08:00:00-09:00:00.log.gz
>>>>>>
>>>>>>
>>>>>>
>>>>>> 
>>>>>>app_stats.03:00:00-04:00:00.log.gzconn.09:00:00-10:00:00.log.gzdpd.10
>>>>>>:0
>>>>>> 0:
>>>>>>
>>>>>> 
>>>>>>00-11:00:00.log.gzknown_services.12:00:00-13:00:00.log.gzsnmp.01:00:0
>>>>>>0-
>>>>>> 02
>>>>>> :00:00.log.gztunnel.10:00:00-11:00:00.log.gz
>>>>>>
>>>>>>
>>>>>>
>>>>>> 
>>>>>>app_stats.04:00:00-05:00:00.log.gzconn.10:00:00-11:00:00.log.gzdpd.11
>>>>>>:0
>>>>>> 0:
>>>>>>
>>>>>> 
>>>>>>00-12:00:00.log.gzloaded_scripts.12:45:56-12:58:35.log.gzsnmp.02:00:0
>>>>>>0-
>>>>>> 03
>>>>>> :00:00.log.gztunnel.11:00:00-12:00:00.log.gz
>>>>>>
>>>>>>
>>>>>>
>>>>>> 
>>>>>>app_stats.05:00:00-06:00:00.log.gzconn.11:00:00-12:00:00.log.gzdpd.12
>>>>>>:0
>>>>>> 0:
>>>>>>
>>>>>> 
>>>>>>00-13:00:00.log.gzloaded_scripts.12:58:38-13:00:00.log.gzsnmp.03:00:0
>>>>>>0-
>>>>>> 04
>>>>>> :00:00.log.gztunnel.12:00:00-13:00:00.log.gz
>>>>>>
>>>>>>
>>>>>>
>>>>>> 
>>>>>>app_stats.06:00:00-07:00:00.log.gzconn.12:00:00-13:00:00.log.gzfiles.
>>>>>>00
>>>>>> :0
>>>>>>
>>>>>> 
>>>>>>0:00-01:00:00.log.gznotice.00:00:00-01:00:00.log.gzsnmp.09:00:00-10:0
>>>>>>0:
>>>>>> 00
>>>>>> .log.gzweird.00:00:00-01:00:00.log.gz
>>>>>>
>>>>>>
>>>>>>
>>>>>> 
>>>>>>app_stats.07:00:00-08:00:00.log.gzconn-summary.00:00:00-01:00:00.log.
>>>>>>gz
>>>>>> fi
>>>>>>
>>>>>> 
>>>>>>les.01:00:00-02:00:00.log.gznotice.01:00:00-02:00:00.log.gzsnmp.10:00
>>>>>>:0
>>>>>> 0-
>>>>>> 11:00:00.log.gzweird.01:00:00-02:00:00.log.gz
>>>>>>
>>>>>>
>>>>>>
>>>>>> 
>>>>>>app_stats.08:00:00-09:00:00.log.gzconn-summary.01:00:00-02:00:00.log.
>>>>>>gz
>>>>>> fi
>>>>>>
>>>>>> 
>>>>>>les.02:00:00-03:00:00.log.gznotice.02:00:00-03:00:00.log.gzsnmp.11:00
>>>>>>:0
>>>>>> 0-
>>>>>> 12:00:00.log.gzweird.02:00:00-03:00:00.log.gz
>>>>>>
>>>>>>
>>>>>>
>>>>>> 
>>>>>>app_stats.09:00:00-10:00:00.log.gzconn-summary.02:00:00-03:00:00.log.
>>>>>>gz
>>>>>> fi
>>>>>>
>>>>>> 
>>>>>>les.03:00:00-04:00:00.log.gznotice.03:00:00-04:00:00.log.gzsoftware.0
>>>>>>0:
>>>>>> 00
>>>>>> :00-01:00:00.log.gzweird.03:00:00-04:00:00.log.gz
>>>>>>
>>>>>>
>>>>>>
>>>>>> 
>>>>>>app_stats.10:00:00-11:00:00.log.gzconn-summary.03:00:00-04:00:00.log.
>>>>>>gz
>>>>>> fi
>>>>>>
>>>>>> 
>>>>>>les.04:00:00-05:00:00.log.gznotice.04:00:00-05:00:00.log.gzsoftware.0
>>>>>>1:
>>>>>> 00
>>>>>> :00-02:00:00.log.gzweird.04:00:00-05:00:00.log.gz
>>>>>>
>>>>>> ?
>>>>>>
>>>>>>
>>>>>> Is there a configuration directive that I?m missing?
>>>>>>
>>>>>> Thanks in advance for any help.
>>>>>>
>>>>>> -Andrew
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Bro mailing list
>>>>>> bro at bro-ids.org
>>>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Bro mailing list
>>>> bro at bro-ids.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>>
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>



From charlie.holiday at gmail.com  Thu Jun 18 13:39:42 2015
From: charlie.holiday at gmail.com (Charlie Holiday)
Date: Thu, 18 Jun 2015 16:39:42 -0400
Subject: [Bro] Nodes are running but no logs associated to network traffic
	going to <prefix>/logs/current/
Message-ID: <CAFOKbx1o4BhA5yLemgU3WG6tuV1i7++p-sVy8LdDn17+0ez+YA@mail.gmail.com>

Just completed installing BRO on a new Dell PowerEdge R420 with a Intel
X520 DP 10Gb DA/SFP+ Server Adapter.

I setup BRO as a cluster on this system in order to use pf_ring to spread
the load across multiple cores.  This setup has worked great for other
systems in my environment with the only difference being this new system is
using 10Gb SFP+ adapters.

Any ideas on what might be causing this issue would be greatly appreciated.

Below is some information on what I am seeing:

bro-2.3.1
PF_RING-6.0.2


*node.cfg:*
[manager]
type=manager
host=localhost
#
[proxy-1]
type=proxy
host=localhost
#
[worker-1]
type=worker
host=localhost
interface=bond0
lb_method=pf_ring
lb_procs=4


*<prefix>/logs/current/$*
communication.log  notice.log  stderr.log  stdout.log  weird.log


[BroControl] > status
Name         Type    Host             Status    Pid    Peers  Started
manager      manager localhost        running   21170  5      18 Jun
14:27:16
proxy-1      proxy   localhost        running   21195  5      18 Jun
14:27:18
worker-1-1   worker  localhost        running   21257  2      18 Jun
14:27:20
worker-1-2   worker  localhost        running   21254  2      18 Jun
14:27:20
worker-1-3   worker  localhost        running   21256  2      18 Jun
14:27:20
worker-1-4   worker  localhost        running   21255  2      18 Jun
14:27:20

[BroControl] > netstats
 worker-1-1: 1434659737.208884 recvd=147334297 dropped=2080 link=147336423
 worker-1-2: 1434659737.408838 recvd=147338710 dropped=405 link=147339135
 worker-1-3: 1434659737.608633 recvd=147342307 dropped=792 link=147343135
 worker-1-4: 1434659737.808998 recvd=147347149 dropped=318 link=147347519

Best Regards,
Charlie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150618/e2480782/attachment.html 

From jlay at slave-tothe-box.net  Sat Jun 20 03:33:52 2015
From: jlay at slave-tothe-box.net (James Lay)
Date: Sat, 20 Jun 2015 04:33:52 -0600
Subject: [Bro] Bro vs Netflow
Message-ID: <1434796432.3632.5.camel@JamesiMac>

So in my internet travels I ran across this:

https://www.rsreese.com/parsing-netflow-using-kibana-via-logstash-to-elasticsearch/

A tad outdated but I thought why not....I have syslogs and Bro's
conn.log going into the ELK stack, so let's add netflow to the mix.
After dinking around with it and getting the data in, I realized that
Bro's conn.log pretty much does everything netflow can...unless I'm
missing something?  For example, if I want to see what a single IP
address is doing I use this as a filter in Kibana:

type:connlog AND conn_state:S* AND src_ip:192.168.1.100 AND proto:TCP
type:connlog AND conn_state:S* AND src_ip:192.168.1.100 AND proto:UDP

What say you all....any reason not to rip out softflowd and just drive
on with Bro's conn.log?  Thank you.

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150620/18625648/attachment.html 

From andrew.ratcliffe at nswcsystems.co.uk  Sat Jun 20 04:05:50 2015
From: andrew.ratcliffe at nswcsystems.co.uk (Andrew Ratcliffe)
Date: Sat, 20 Jun 2015 11:05:50 +0000
Subject: [Bro] Bro vs Netflow
In-Reply-To: <1434796432.3632.5.camel@JamesiMac>
References: <1434796432.3632.5.camel@JamesiMac>
Message-ID: <3373D942-9260-4EC6-80D1-344B0D0722E9@nswcsystems.co.uk>

James,
I think you?re right, but sometimes you can get Netflow from locations where you might not easily be able to put a Bro sensor.

Kind regards,
Andy
Andrew.Ratcliffe at NSWCSystems.co.uk
CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
Blog.InfoSecMatters.net <http://blog.infosecmatters.net/>






> On 20 Jun 2015, at 11:33, James Lay <jlay at slave-tothe-box.net> wrote:
> 
> So in my internet travels I ran across this:
> 
> https://www.rsreese.com/parsing-netflow-using-kibana-via-logstash-to-elasticsearch/ <https://www.rsreese.com/parsing-netflow-using-kibana-via-logstash-to-elasticsearch/>
> 
> A tad outdated but I thought why not....I have syslogs and Bro's conn.log going into the ELK stack, so let's add netflow to the mix.  After dinking around with it and getting the data in, I realized that Bro's conn.log pretty much does everything netflow can...unless I'm missing something?  For example, if I want to see what a single IP address is doing I use this as a filter in Kibana:
> 
> type:connlog AND conn_state:S* AND src_ip:192.168.1.100 AND proto:TCP
> type:connlog AND conn_state:S* AND src_ip:192.168.1.100 AND proto:UDP
> 
> What say you all....any reason not to rip out softflowd and just drive on with Bro's conn.log?  Thank you.
> 
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150620/f7220f22/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150620/f7220f22/attachment.bin 

From lists at g-clef.net  Mon Jun 22 11:22:04 2015
From: lists at g-clef.net (Aaron Gee-Clough)
Date: Mon, 22 Jun 2015 14:22:04 -0400
Subject: [Bro] worker dies out of memory
Message-ID: <5588524C.5080807@g-clef.net>


All,

I've had this problem pop up twice in the past couple weeks, where I
restart bro & learn that one of the workers has crashed. The message
that comes back in the crash report is:

listening on dag0:32, capture length 8192 bytes

==== stderr.log
1434739692.912574 processing suspended
1434739692.912574 processing continued
tcmalloc: large alloc 18446744072965615616 bytes == (nil) @
0x7f265e1ab8f5 0x81550d 0x88b986 0x88ba76 0x8147c9 0x814b54 0x81462b
0x812ea6 0x810265 0x88bb3a 0x5691a1 0x607698 0x608cd5 0x5d5461 0x85bd57
0x5d573b 0x532915 0x7f265d1aa76d 0x53b3bd
out of memory in new.
1434743469.366970 fatal error: out of memory in new.

==== stdout.log
max memory size         (kbytes, -m) unlimited
data seg size           (kbytes, -d) unlimited
virtual memory          (kbytes, -v) unlimited
core file size          (blocks, -c) unlimited

==== .cmdline
-i dag0:32 -U .status -p broctl -p broctl-live -p local -p worker-17
local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto



This is running on Ubuntu 12.04, bro 2.4, linux kernel 3.2.0-70, with
pcap compiled to use the Endace dag card, and bro compiled to use google
perftools. The box has 256GB of RAM.

Has anyone else seen this?

Thanks.

aaron

From mlaterma at ucalgary.ca  Mon Jun 22 11:54:04 2015
From: mlaterma at ucalgary.ca (Michel Laterman)
Date: Mon, 22 Jun 2015 12:54:04 -0600
Subject: [Bro] worker dies out of memory
In-Reply-To: <5588524C.5080807@g-clef.net>
Message-ID: <1942994a-2328-4c0e-9b87-4d134d338e5b@email.android.com>

I ran into the same problem (with the DAG card) a few months ago. 
My issue turned out to be that the firmware on the card needed an update. 
Is your firmware up to date? 

Michel 

On Jun 22, 2015 12:22 PM, Aaron Gee-Clough <lists at g-clef.net> wrote:
>
>
> All, 
>
> I've had this problem pop up twice in the past couple weeks, where I 
> restart bro & learn that one of the workers has crashed. The message 
> that comes back in the crash report is: 
>
> listening on dag0:32, capture length 8192 bytes 
>
> ==== stderr.log 
> 1434739692.912574 processing suspended 
> 1434739692.912574 processing continued 
> tcmalloc: large alloc 18446744072965615616 bytes == (nil) @ 
> 0x7f265e1ab8f5 0x81550d 0x88b986 0x88ba76 0x8147c9 0x814b54 0x81462b 
> 0x812ea6 0x810265 0x88bb3a 0x5691a1 0x607698 0x608cd5 0x5d5461 0x85bd57 
> 0x5d573b 0x532915 0x7f265d1aa76d 0x53b3bd 
> out of memory in new. 
> 1434743469.366970 fatal error: out of memory in new. 
>
> ==== stdout.log 
> max memory size???????? (kbytes, -m) unlimited 
> data seg size?????????? (kbytes, -d) unlimited 
> virtual memory????????? (kbytes, -v) unlimited 
> core file size????????? (blocks, -c) unlimited 
>
> ==== .cmdline 
> -i dag0:32 -U .status -p broctl -p broctl-live -p local -p worker-17 
> local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto 
>
>
>
> This is running on Ubuntu 12.04, bro 2.4, linux kernel 3.2.0-70, with 
> pcap compiled to use the Endace dag card, and bro compiled to use google 
> perftools. The box has 256GB of RAM. 
>
> Has anyone else seen this? 
>
> Thanks. 
>
> aaron 
> _______________________________________________ 
> Bro mailing list 
> bro at bro-ids.org 
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro 


From lists at g-clef.net  Mon Jun 22 12:01:40 2015
From: lists at g-clef.net (Aaron Gee-Clough)
Date: Mon, 22 Jun 2015 15:01:40 -0400
Subject: [Bro] worker dies out of memory
References: <5588524C.5080807@g-clef.net>
	<1942994a-2328-4c0e-9b87-4d134d338e5b@email.android.com>
Message-ID: <55885B94.40404@g-clef.net>


huh. It is not, because I'm running a custom firmware from Endace (card
kept causing the system to kernel panic). Time to bug Endace again, I guess.

Thanks for the pointer.

aaron

On 06/22/2015 02:54 PM, Michel Laterman wrote:
> I ran into the same problem (with the DAG card) a few months ago. 
> My issue turned out to be that the firmware on the card needed an update. 
> Is your firmware up to date? 
> 
> Michel 
> 
> On Jun 22, 2015 12:22 PM, Aaron Gee-Clough <lists at g-clef.net> wrote:
>>
>>
>> All, 
>>
>> I've had this problem pop up twice in the past couple weeks, where I 
>> restart bro & learn that one of the workers has crashed. The message 
>> that comes back in the crash report is: 
>>
>> listening on dag0:32, capture length 8192 bytes 
>>
>> ==== stderr.log 
>> 1434739692.912574 processing suspended 
>> 1434739692.912574 processing continued 
>> tcmalloc: large alloc 18446744072965615616 bytes == (nil) @ 
>> 0x7f265e1ab8f5 0x81550d 0x88b986 0x88ba76 0x8147c9 0x814b54 0x81462b 
>> 0x812ea6 0x810265 0x88bb3a 0x5691a1 0x607698 0x608cd5 0x5d5461 0x85bd57 
>> 0x5d573b 0x532915 0x7f265d1aa76d 0x53b3bd 
>> out of memory in new. 
>> 1434743469.366970 fatal error: out of memory in new. 
>>
>> ==== stdout.log 
>> max memory size         (kbytes, -m) unlimited 
>> data seg size           (kbytes, -d) unlimited 
>> virtual memory          (kbytes, -v) unlimited 
>> core file size          (blocks, -c) unlimited 
>>
>> ==== .cmdline 
>> -i dag0:32 -U .status -p broctl -p broctl-live -p local -p worker-17 
>> local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto 
>>
>>
>>
>> This is running on Ubuntu 12.04, bro 2.4, linux kernel 3.2.0-70, with 
>> pcap compiled to use the Endace dag card, and bro compiled to use google 
>> perftools. The box has 256GB of RAM. 
>>
>> Has anyone else seen this? 
>>
>> Thanks. 
>>
>> aaron 
>> _______________________________________________ 
>> Bro mailing list 
>> bro at bro-ids.org 
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro 

From seth at icir.org  Mon Jun 22 13:00:42 2015
From: seth at icir.org (Seth Hall)
Date: Mon, 22 Jun 2015 16:00:42 -0400
Subject: [Bro] Bro vs Netflow
In-Reply-To: <1434796432.3632.5.camel@JamesiMac>
References: <1434796432.3632.5.camel@JamesiMac>
Message-ID: <3A1A23B2-3B06-4298-AD0D-3DD353E1670B@icir.org>


> On Jun 20, 2015, at 6:33 AM, James Lay <jlay at slave-tothe-box.net> wrote:
> 
> What say you all....any reason not to rip out softflowd and just drive on with Bro's conn.log?  Thank you.

Andrew got the exact reason that you?d still collect netflow.  The Bro conn log is significantly different than netflow though.  It?s bidirectional (IPFIX can be too, but we?ll ignore that for now).  The log doesn?t write out until the connection is complete, whereas netflow breaks and writes out frequently which can be great, but can also be super annoying if you?re trying to pay attention to the full life cycle of a connection forensically.  There are several extra fields in the Bro logs that netflow doesn?t have too (history and service being two that immediately come to mind).

If you?re generating netflow though, there is almost never any benefit these days unless you have a netflow analysis solution in place that you?d like to feed and you can?t collect from routers anymore, usually because your routers can only do sampled netflow.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150622/d5711472/attachment.bin 

From jlay at slave-tothe-box.net  Mon Jun 22 13:37:29 2015
From: jlay at slave-tothe-box.net (James Lay)
Date: Mon, 22 Jun 2015 14:37:29 -0600
Subject: [Bro] Bro vs Netflow
In-Reply-To: <3A1A23B2-3B06-4298-AD0D-3DD353E1670B@icir.org>
References: <1434796432.3632.5.camel@JamesiMac>
	<3A1A23B2-3B06-4298-AD0D-3DD353E1670B@icir.org>
Message-ID: <e2962f7d80560c24be0b9d68b808b854@localhost>

On 2015-06-22 02:00 PM, Seth Hall wrote:
>> On Jun 20, 2015, at 6:33 AM, James Lay <jlay at slave-tothe-box.net> 
>> wrote:
>> 
>> What say you all....any reason not to rip out softflowd and just drive 
>> on with Bro's conn.log?  Thank you.
> 
> Andrew got the exact reason that you?d still collect netflow.  The Bro
> conn log is significantly different than netflow though.  It?s
> bidirectional (IPFIX can be too, but we?ll ignore that for now).  The
> log doesn?t write out until the connection is complete, whereas
> netflow breaks and writes out frequently which can be great, but can
> also be super annoying if you?re trying to pay attention to the full
> life cycle of a connection forensically.  There are several extra
> fields in the Bro logs that netflow doesn?t have too (history and
> service being two that immediately come to mind).
> 
> If you?re generating netflow though, there is almost never any benefit
> these days unless you have a netflow analysis solution in place that
> you?d like to feed and you can?t collect from routers anymore, usually
> because your routers can only do sampled netflow.
> 
>   .Seth

Thanks Seth,

I took out softflowd and reverted to my previous version of 
logstash.conf.  I have to say, it was pretty cool to have my Kibana 
graphs up of Bro's conn.log and softflowd side by side to compare....was 
pretty awesome.

James


From seth at icir.org  Mon Jun 22 13:44:15 2015
From: seth at icir.org (Seth Hall)
Date: Mon, 22 Jun 2015 16:44:15 -0400
Subject: [Bro] Bro vs Netflow
In-Reply-To: <e2962f7d80560c24be0b9d68b808b854@localhost>
References: <1434796432.3632.5.camel@JamesiMac>
	<3A1A23B2-3B06-4298-AD0D-3DD353E1670B@icir.org>
	<e2962f7d80560c24be0b9d68b808b854@localhost>
Message-ID: <D2C98D29-B5D2-42F2-9857-60E7B63D5DA3@icir.org>


> On Jun 22, 2015, at 4:37 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> 
> I took out softflowd and reverted to my previous version of logstash.conf.  I have to say, it was pretty cool to have my Kibana graphs up of Bro's conn.log and softflowd side by side to compare....was pretty awesome.

Nice, did you notice any major discrepancies?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150622/1e56d45a/attachment.bin 

From jlay at slave-tothe-box.net  Mon Jun 22 13:51:22 2015
From: jlay at slave-tothe-box.net (James Lay)
Date: Mon, 22 Jun 2015 14:51:22 -0600
Subject: [Bro] Bro vs Netflow
In-Reply-To: <D2C98D29-B5D2-42F2-9857-60E7B63D5DA3@icir.org>
References: <1434796432.3632.5.camel@JamesiMac>
	<3A1A23B2-3B06-4298-AD0D-3DD353E1670B@icir.org>
	<e2962f7d80560c24be0b9d68b808b854@localhost>
	<D2C98D29-B5D2-42F2-9857-60E7B63D5DA3@icir.org>
Message-ID: <50902a7c2ca9ee710b81358781db0206@localhost>

On 2015-06-22 02:44 PM, Seth Hall wrote:
>> On Jun 22, 2015, at 4:37 PM, James Lay <jlay at slave-tothe-box.net> 
>> wrote:
>> 
>> I took out softflowd and reverted to my previous version of 
>> logstash.conf.  I have to say, it was pretty cool to have my Kibana 
>> graphs up of Bro's conn.log and softflowd side by side to 
>> compare....was pretty awesome.
> 
> Nice, did you notice any major discrepancies?
> 
>   .Seth
> 

I did not besides the minor timing thing you described.  For example 
there's an Android device that fires off to ssl.analytics.google.com at 
exact intervals.  The netflow graph showed these at pretty close to the 
same times (squid logs logged the exact time to syslog), whereas Bro had 
them a little varied, but that was ONLY when you dug in like to a every 
5 minute graph.  If you zoomed out to say showing the last 12 hours you 
couldn't tell a difference at all.  I didn't notice a difference in the 
packet count or size either....a good thing :)

James

From pch66 at cornell.edu  Tue Jun 23 06:48:42 2015
From: pch66 at cornell.edu (Peter Hansen)
Date: Tue, 23 Jun 2015 09:48:42 -0400
Subject: [Bro] Segmentation Fault when using DPD in BinPAC++
Message-ID: <CACPag=03a0gBsd5Gtbm7HCjw6a7Qs8WhgJuFq8CZe+ny0H5qEQ@mail.gmail.com>

Hello,

I am running into a bug implementing DPD with BinPAC++ where it appears to
be causing segmentation fault errors. To reproduce this, all I have to do
is run bro -r on any capture file using bro/pac2/http.evt.

The only output I get is:

Segmentation fault, (core dumped)

Additionally, I cannot seem to find that any of the files in bro/pac2
create events when the kind of traffic they are designed for is produced a
port besides their well known. (i.e., using bro/pac2/ssh.evt does not
create an event when an ssh connection is made on port 2222)

Thanks,
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150623/06bebce2/attachment.html 

From seth at icir.org  Tue Jun 23 07:37:01 2015
From: seth at icir.org (Seth Hall)
Date: Tue, 23 Jun 2015 10:37:01 -0400
Subject: [Bro] Segmentation Fault when using DPD in BinPAC++
In-Reply-To: <CACPag=03a0gBsd5Gtbm7HCjw6a7Qs8WhgJuFq8CZe+ny0H5qEQ@mail.gmail.com>
References: <CACPag=03a0gBsd5Gtbm7HCjw6a7Qs8WhgJuFq8CZe+ny0H5qEQ@mail.gmail.com>
Message-ID: <AB8F546E-C9CA-49F7-86F5-7E01517793B7@icir.org>

Hi Peter.  You might want to move this email over to the HILTI/Spicy mailing list. (Spicy is the new name for BinPAC++)

http://mailman.icsi.berkeley.edu/mailman/listinfo/hilti

Thanks!
  .Seth

> On Jun 23, 2015, at 9:48 AM, Peter Hansen <pch66 at cornell.edu> wrote:
> 
> Hello,
> 
> I am running into a bug implementing DPD with BinPAC++ where it appears to be causing segmentation fault errors. To reproduce this, all I have to do is run bro -r on any capture file using bro/pac2/http.evt.
> 
> The only output I get is:
> 
> Segmentation fault, (core dumped)
> 
> Additionally, I cannot seem to find that any of the files in bro/pac2 create events when the kind of traffic they are designed for is produced a port besides their well known. (i.e., using bro/pac2/ssh.evt does not create an event when an ssh connection is made on port 2222)
> 
> Thanks,
> Peter
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150623/6a2132e8/attachment.bin 

From mdblack98 at yahoo.com  Tue Jun 23 15:59:10 2015
From: mdblack98 at yahoo.com (Michael Black)
Date: Tue, 23 Jun 2015 17:59:10 -0500
Subject: [Bro] Help with simple stuff
Message-ID: <041901d0ae08$37cf5390$a76dfab0$@yahoo.com>

I feel like a complete idiot.

 

I seem to have problems getting this portion of a  simple script to work.

 

event connection_established(c: connection)

{

                local mypair = string_cat(c$id$orig_h,",",c$id$resp_h);

}

 

I get this error:

fatal error in <no location>: Val::CONST_ACCESSOR (addr/string)
(10.207.40.41)


What obvious thing am I missing?  Logically seems like one should be able to
concatenate these together.

 

Mike

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150623/250e337d/attachment.html 

From seth at icir.org  Tue Jun 23 19:13:57 2015
From: seth at icir.org (Seth Hall)
Date: Tue, 23 Jun 2015 22:13:57 -0400
Subject: [Bro] Help with simple stuff
In-Reply-To: <041901d0ae08$37cf5390$a76dfab0$@yahoo.com>
References: <041901d0ae08$37cf5390$a76dfab0$@yahoo.com>
Message-ID: <21E82A22-6398-4079-BCE0-4F65E47855B9@icir.org>


> On Jun 23, 2015, at 6:59 PM, Michael Black <mdblack98 at yahoo.com> wrote:
> 
>                 local mypair = string_cat(c$id$orig_h,",",c$id$resp_h);
> What obvious thing am I missing?  Logically seems like one should be able to concatenate these together.

Easy fix at least, use the ?cat? function instead.  string_cat only accepts strings are parameters, but orig_h and resp_h are addrs.  cat accepts any type for it?s arguments.

Good luck on whatever you?re working on. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150623/c1a9ff48/attachment.bin 

From asharma at lbl.gov  Tue Jun 23 19:23:08 2015
From: asharma at lbl.gov (Aashish Sharma)
Date: Tue, 23 Jun 2015 19:23:08 -0700
Subject: [Bro] Help with simple stuff
In-Reply-To: <21E82A22-6398-4079-BCE0-4F65E47855B9@icir.org>
References: <041901d0ae08$37cf5390$a76dfab0$@yahoo.com>
	<21E82A22-6398-4079-BCE0-4F65E47855B9@icir.org>
Message-ID: <20150624022307.GL17824@yaksha.lbl.gov>

> >                 local mypair = string_cat(c$id$orig_h,",",c$id$resp_h);

I generally endup with:

	local mypair = fmt ("%s, %s", c$id$orig_h, c$id$resp_h) ;

Aashish 

On Tue, Jun 23, 2015 at 10:13:57PM -0400, Seth Hall wrote:
> 
> > On Jun 23, 2015, at 6:59 PM, Michael Black <mdblack98 at yahoo.com> wrote:
> > 
> >                 local mypair = string_cat(c$id$orig_h,",",c$id$resp_h);
> > What obvious thing am I missing?  Logically seems like one should be able to concatenate these together.
> 
> Easy fix at least, use the ?cat? function instead.  string_cat only accepts strings are parameters, but orig_h and resp_h are addrs.  cat accepts any type for it?s arguments.
> 
> Good luck on whatever you?re working on. :)
> 
>   .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
> 



> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


-- 
Aashish Sharma	(asharma at lbl.gov) 				 
Cyber Security, 
Lawrence Berkeley National Laboratory  
http://go.lbl.gov/pgp-aashish 
Office: (510)-495-2680  Cell: (510)-612-7971

From jan.grashofer at cern.ch  Thu Jun 25 04:31:59 2015
From: jan.grashofer at cern.ch (Jan Grashofer)
Date: Thu, 25 Jun 2015 11:31:59 +0000
Subject: [Bro] Threat Intelligence Management
Message-ID: <E3142ED5D6744B4C8E17323B862FEDE5DF670E@CERNXCHG62.cern.ch>

Hi all,

I am having a look at Threat Intelligence Management solutions, which can be used with Bro. What do you use and what are your experiences?

Regards,
Jan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150625/caf72b69/attachment.html 

From lysemose at gmail.com  Thu Jun 25 04:44:07 2015
From: lysemose at gmail.com (Heine Lysemose)
Date: Thu, 25 Jun 2015 13:44:07 +0200
Subject: [Bro] Threat Intelligence Management
In-Reply-To: <E3142ED5D6744B4C8E17323B862FEDE5DF670E@CERNXCHG62.cern.ch>
References: <E3142ED5D6744B4C8E17323B862FEDE5DF670E@CERNXCHG62.cern.ch>
Message-ID: <CAN4C-D=9tS0skytp2AeJDBUNxreQV0_AkrASc8-AA2Ls2bDFVg@mail.gmail.com>

Hi

I encourage you to have a look at, https://intel.criticalstack.com/

Best,
Lysemose

On Thu, Jun 25, 2015 at 1:31 PM, Jan Grashofer <jan.grashofer at cern.ch>
wrote:

>  Hi all,
>
> I am having a look at Threat Intelligence Management solutions, which can
> be used with Bro. What do you use and what are your experiences?
>
> Regards,
> Jan
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150625/3868fc94/attachment.html 

From jlay at slave-tothe-box.net  Thu Jun 25 04:44:46 2015
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 25 Jun 2015 05:44:46 -0600
Subject: [Bro] Threat Intelligence Management
In-Reply-To: <E3142ED5D6744B4C8E17323B862FEDE5DF670E@CERNXCHG62.cern.ch>
References: <E3142ED5D6744B4C8E17323B862FEDE5DF670E@CERNXCHG62.cern.ch>
Message-ID: <1435232686.3610.1.camel@JamesiMac>

On Thu, 2015-06-25 at 11:31 +0000, Jan Grashofer wrote:
> Hi all,
> 
> I am having a look at Threat Intelligence Management solutions, which
> can be used with Bro. What do you use and what are your experiences?
> 
> Regards,
> Jan
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


BOOM:  https://intel.criticalstack.com/

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150625/d06956ec/attachment.html 

From jan.grashofer at cern.ch  Thu Jun 25 04:55:49 2015
From: jan.grashofer at cern.ch (Jan Grashofer)
Date: Thu, 25 Jun 2015 11:55:49 +0000
Subject: [Bro] Threat Intelligence Management
In-Reply-To: <CAN4C-D=9tS0skytp2AeJDBUNxreQV0_AkrASc8-AA2Ls2bDFVg@mail.gmail.com>
References: <E3142ED5D6744B4C8E17323B862FEDE5DF670E@CERNXCHG62.cern.ch>,
	<CAN4C-D=9tS0skytp2AeJDBUNxreQV0_AkrASc8-AA2Ls2bDFVg@mail.gmail.com>
Message-ID: <E3142ED5D6744B4C8E17323B862FEDE5DF6759@CERNXCHG62.cern.ch>

Hi Lysemose,



thanks a lot for your reply! Critical stack is like a marketplace for intel in the cloud, right? What I am looking for is a solution I can deploy at my site to ingest intel of different sources (also putting in manually collected stuff), which can be queried by different parts of our stack (Bro only one of them). CIF seemed promising but whether the idea behind might be great, at least the documentation is horrible.



Jan



________________________________
From: Heine Lysemose [lysemose at gmail.com]
Sent: Thursday, June 25, 2015 13:44
To: Jan Grashofer
Cc: bro at bro.org
Subject: Re: [Bro] Threat Intelligence Management

Hi

I encourage you to have a look at, https://intel.criticalstack.com/

Best,
Lysemose

On Thu, Jun 25, 2015 at 1:31 PM, Jan Grashofer <jan.grashofer at cern.ch<mailto:jan.grashofer at cern.ch>> wrote:
Hi all,

I am having a look at Threat Intelligence Management solutions, which can be used with Bro. What do you use and what are your experiences?

Regards,
Jan

_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150625/71c31d81/attachment.html 

From seth at icir.org  Thu Jun 25 05:06:50 2015
From: seth at icir.org (Seth Hall)
Date: Thu, 25 Jun 2015 08:06:50 -0400
Subject: [Bro] Threat Intelligence Management
In-Reply-To: <E3142ED5D6744B4C8E17323B862FEDE5DF6759@CERNXCHG62.cern.ch>
References: <E3142ED5D6744B4C8E17323B862FEDE5DF670E@CERNXCHG62.cern.ch>
	<CAN4C-D=9tS0skytp2AeJDBUNxreQV0_AkrASc8-AA2Ls2bDFVg@mail.gmail.com>
	<E3142ED5D6744B4C8E17323B862FEDE5DF6759@CERNXCHG62.cern.ch>
Message-ID: <78042C9E-4115-40F2-A922-C0F51C90AAE6@icir.org>


> On Jun 25, 2015, at 7:55 AM, Jan Grashofer <jan.grashofer at cern.ch> wrote:
> 
> CIF seemed promising but whether the idea behind might be great, at least the documentation is horrible.

I know of one organization that has been very happy with MISP and is preparing to grow their deployment.
	https://github.com/misp/misp

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150625/a40d8b29/attachment.bin 

From hhoffman at ip-solutions.net  Thu Jun 25 05:37:10 2015
From: hhoffman at ip-solutions.net (Harry Hoffman)
Date: Thu, 25 Jun 2015 08:37:10 -0400
Subject: [Bro] Threat Intelligence Management
In-Reply-To: <CAN4C-D=9tS0skytp2AeJDBUNxreQV0_AkrASc8-AA2Ls2bDFVg@mail.gmail.com>
Message-ID: <9f25f9f5-be30-41a4-beda-e65b0ec32c49@email.android.com>

Is critical stack based upon CIF (collective intelligence framework)?

It looks very similar.

Cheers,
Harry


On Jun 25, 2015 7:44 AM, Heine Lysemose <lysemose at gmail.com> wrote:
>
> Hi
>
> I encourage you to have a look at,?https://intel.criticalstack.com/
>
> Best,
> Lysemose
>
> On Thu, Jun 25, 2015 at 1:31 PM, Jan Grashofer <jan.grashofer at cern.ch> wrote:
>>
>> Hi all,
>>
>> I am having a look at Threat Intelligence Management solutions, which can be used with Bro. What do you use and what are your experiences?
>>
>> Regards,
>> Jan
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>


From liam.randall at gmail.com  Thu Jun 25 05:51:58 2015
From: liam.randall at gmail.com (Liam Randall)
Date: Thu, 25 Jun 2015 08:51:58 -0400
Subject: [Bro] Threat Intelligence Management
In-Reply-To: <9f25f9f5-be30-41a4-beda-e65b0ec32c49@email.android.com>
References: <CAN4C-D=9tS0skytp2AeJDBUNxreQV0_AkrASc8-AA2Ls2bDFVg@mail.gmail.com>
	<9f25f9f5-be30-41a4-beda-e65b0ec32c49@email.android.com>
Message-ID: <CAC8+5eML6Jsy7U4DkfdTW1cTz1VZwQAZApdzX+vfxeDAekaS+A@mail.gmail.com>

No Critical Stack is entirely custom; we are not building a TIP.  We wanted
to have an easy way to have actionable into stream into bro as it is to
discovered so we built it.  We thought others would want it as well so we
make it freely available to the community.  We are getting ready to launch
a new extension to it called KITTY- Keep Intel Transactions To Yourself
that allow you to privately share and deploy 100's of Millions of
indicators in a fast memory efficient way.  It integrates directly with our
online marketplace- we deployed our first test clients this week.  We'll
announce more shortly @CriticalStack .

For TIPs there are a lot of great solutions you should look at:

Free:
MISP
CRITS

Commercial:
Soltra Edge (has a free version)

ThreatConnect

ThreatStream

ThreatQ (ThreatQuotient)

BrightPoint Security (formerly Vorstack)



V/r,

Liam Randall


On Thu, Jun 25, 2015 at 8:37 AM, Harry Hoffman <hhoffman at ip-solutions.net>
wrote:

> Is critical stack based upon CIF (collective intelligence framework)?
>
> It looks very similar.
>
> Cheers,
> Harry
>
>
> On Jun 25, 2015 7:44 AM, Heine Lysemose <lysemose at gmail.com> wrote:
> >
> > Hi
> >
> > I encourage you to have a look at, https://intel.criticalstack.com/
> >
> > Best,
> > Lysemose
> >
> > On Thu, Jun 25, 2015 at 1:31 PM, Jan Grashofer <jan.grashofer at cern.ch>
> wrote:
> >>
> >> Hi all,
> >>
> >> I am having a look at Threat Intelligence Management solutions, which
> can be used with Bro. What do you use and what are your experiences?
> >>
> >> Regards,
> >> Jan
> >>
> >> _______________________________________________
> >> Bro mailing list
> >> bro at bro-ids.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >
> >
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150625/13e366d8/attachment-0001.html 

From mdblack98 at yahoo.com  Thu Jun 25 11:21:22 2015
From: mdblack98 at yahoo.com (Michael Black)
Date: Thu, 25 Jun 2015 13:21:22 -0500
Subject: [Bro] Record use
Message-ID: <085f01d0af73$bdac2620$39047260$@yahoo.com>

I'm trying to keep a count of total bytes between IP pairs.
Type Bandwidht: record {
	pair: string &log;
	bytesIn: count &log;
	bytesOut: count &log;
};
global bandwidth: set[Bandwidth];

If all I have in the record is just "pair" this works OK.
local mypair = fmt("%s,%s",c$id$orig_h,c$id$resp_h);
local thispair: Bandwidth;
thispair$pair = mypair;
if (thispair in bandwidth)
..do stuff
else {
	add bandwidth[thispair];
}


But.if I add the bytes In/Out to the record and thispair it never finds
thispair in bandwidth as though it's looking for match on bytes too.
I assume there's a simple solution.

Can't seem to find a reference on the syntax.

Thanks
Mike



From abhall1 at yahoo.com  Thu Jun 25 17:14:43 2015
From: abhall1 at yahoo.com (Adam Hall)
Date: Fri, 26 Jun 2015 00:14:43 +0000 (UTC)
Subject: [Bro] Bro Digest, Vol 110, Issue 35
In-Reply-To: <mailman.1.1435258801.15543.bro@bro.org>
References: <mailman.1.1435258801.15543.bro@bro.org>
Message-ID: <135312687.25775.1435277683438.JavaMail.yahoo@mail.yahoo.com>

Hey Michael,
? ?I tried to stick to what you had and assumed this rest. ?I got it to work and here is what I had.
type Bandwidth: record {? ? ? ? pair: string &log;? ? ? ? bytesIn: count &log;? ? ? ? bytesOut: count &log;};
global bandwidth: set[Bandwidth];
event connection_state_remove(c: connection){
local mypair=fmt("%s,%s",c$id$orig_h,c$id$resp_h);
local bin= c$orig$size;
local bout= c$resp$size;
local thispair: Bandwidth;
thispair$pair = mypair;thispair$bytesIn = bin;thispair$bytesOut = bout;
if (thispair in bandwidth)? ? ? ? print fmt("My pair %s",thispair);else{? ? ? ? print fmt("My pair 2 %s",thispair);? ? ? ? add bandwidth[thispair];}
}
Here are results:
My pair 2 [pair=2601:347:c200:xxx:xxx:xxx:xxx:xxx,2607:f8b0:xxx:xxx::xxx, bytesIn=2737, bytesOut=1350]My pair [pair=192.168.x.x,224.0.0.252, bytesIn=54, bytesOut=0]
I don't think any bytes will return NULL, but you may want to consider doing " &log &optional ".Also... this was ran on bro-2.4.
Hope this helps!
RedLight

      From: "bro-request at bro.org" <bro-request at bro.org>
 To: bro at bro.org 
 Sent: Thursday, June 25, 2015 3:00 PM
 Subject: Bro Digest, Vol 110, Issue 35
   
Send Bro mailing list submissions to
??? bro at bro.org

To subscribe or unsubscribe via the World Wide Web, visit
??? http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
or, via email, send a message with subject or body 'help' to
??? bro-request at bro.org

You can reach the person managing the list at
??? bro-owner at bro.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Bro digest..."


Today's Topics:

? 1. Record use (Michael Black)


----------------------------------------------------------------------

Message: 1
Date: Thu, 25 Jun 2015 13:21:22 -0500
From: "Michael Black" <mdblack98 at yahoo.com>
Subject: [Bro] Record use
To: <bro at bro-ids.org>
Message-ID: <085f01d0af73$bdac2620$39047260$@yahoo.com>
Content-Type: text/plain;??? charset="us-ascii"

I'm trying to keep a count of total bytes between IP pairs.
Type Bandwidht: record {
??? pair: string &log;
??? bytesIn: count &log;
??? bytesOut: count &log;
};
global bandwidth: set[Bandwidth];

If all I have in the record is just "pair" this works OK.
local mypair = fmt("%s,%s",c$id$orig_h,c$id$resp_h);
local thispair: Bandwidth;
thispair$pair = mypair;
if (thispair in bandwidth)
..do stuff
else {
??? add bandwidth[thispair];
}


But.if I add the bytes In/Out to the record and thispair it never finds
thispair in bandwidth as though it's looking for match on bytes too.
I assume there's a simple solution.

Can't seem to find a reference on the syntax.

Thanks
Mike




------------------------------

_______________________________________________
Bro mailing list
Bro at bro.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


End of Bro Digest, Vol 110, Issue 35
************************************


  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150626/da306137/attachment.html 

From albert.zaharovits at gmail.com  Fri Jun 26 01:04:38 2015
From: albert.zaharovits at gmail.com (Albert Zaharovits)
Date: Fri, 26 Jun 2015 11:04:38 +0300
Subject: [Bro] PF_PACKET load balancing
Message-ID: <D6B4CD0F-691E-4AED-BA1A-AAEED5CE56D7@gmail.com>

Hello,

I am experimenting with several OpenSource IDS on Linux.
My concern is load balancing across mmap-ed packet rings.
Some of them have AF_PACKET socket load balancing (Suricata) while others don?t, and rely on PF_RING (Bro).
When I say load balancing I mean PACKET_FANOUT sock option.

The following setup looks like a silver bullet for me:
You compile them (the IDS) with the latest version of pcap, and use pcap filters to achieve load balancing.

Am I missing something?

Best,
Albert

From seth at icir.org  Fri Jun 26 08:01:39 2015
From: seth at icir.org (Seth Hall)
Date: Fri, 26 Jun 2015 11:01:39 -0400
Subject: [Bro] PF_PACKET load balancing
In-Reply-To: <D6B4CD0F-691E-4AED-BA1A-AAEED5CE56D7@gmail.com>
References: <D6B4CD0F-691E-4AED-BA1A-AAEED5CE56D7@gmail.com>
Message-ID: <E4092BCB-5D78-4635-A5A9-7BFE0C42C9E4@icir.org>


> On Jun 26, 2015, at 4:04 AM, Albert Zaharovits <albert.zaharovits at gmail.com> wrote:
> 
> You compile them (the IDS) with the latest version of pcap, and use pcap filters to achieve load balancing.

I?ve actually implemented BPF filters for load balancing before and it?s not good.  You end up having to implement the modulus operator in BPF (yes, it?s possible) but then that expensive filter ends up being executed for each separate process.  A user tested it on a large network and the result was bad.

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150626/e2072061/attachment.bin 

From baxter.milliwew at gmail.com  Fri Jun 26 14:03:39 2015
From: baxter.milliwew at gmail.com (Baxter Milliwew)
Date: Fri, 26 Jun 2015 14:03:39 -0700
Subject: [Bro] Bro's limitations with high worker count and memory exhaustion
Message-ID: <CA+J8G6FgNij+Tu9Kk8jQNe=1kAs0z7zf6D_5-ATB8VZE2nmY4A@mail.gmail.com>

There's some sort of association between memory exhaustion and a high
number of workers.  The poor man's fix would be to purchase new servers
with higher CPU speeds as that would reduce the worker count.  Issues with
high worker count and/or memory exhaustion appears to be a well know
problem based on the mailing list archives.

In the current version of bro-2.4 my previous configuration immediately
causes the manager to crash: 15 proxies, 155 workers.  To resolve this I've
lowered the count to 10 proxies and 140 workers.  However even with this
configuration the manager process will exhaust all memory and crash within
about 2 hours.

The manager is threaded; I think this is an issue with the threading
behavior between manager, proxies, and workers.  Debugging threading
problems is complex and I'm a complete novice.. my current tutorial is
using information from a stack overflow thread:

http://stackoverflow.com/questions/981011/c-programming-debugging-with-pthreads

Does anyone else have this problem ?  What have you tried and what do you
suggest ?

Thanks




1435347409.458185       worker-2-18     parent  -       -       -
info    [#10000/10.1.1.1:36994] peer sent class "control"

1435347409.458185       worker-2-18     parent  -       -       -
info    [#10000/10.1.1.1:36994] phase: handshake

1435347409.661085       worker-2-18     parent  -       -       -
info    [#10000/10.1.1.1:36994] request for unknown event save_results

1435347409.661085       worker-2-18     parent  -       -       -
info    [#10000/10.1.1.1:36994] registered for event
Control::peer_status_response

1435347409.694858       worker-2-18     parent  -       -       -
info    [#10000/10.1.1.1:36994] peer does not support 64bit PIDs; using
compatibility mode

1435347409.694858       worker-2-18     parent  -       -       -
info    [#10000/10.1.1.1:36994] peer is a Broccoli

1435347409.694858       worker-2-18     parent  -       -       -
info    [#10000/10.1.1.1:36994] phase: running
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150626/a3845604/attachment.html 

From andrew.ratcliffe at nswcsystems.co.uk  Sat Jun 27 14:37:09 2015
From: andrew.ratcliffe at nswcsystems.co.uk (Andrew Ratcliffe)
Date: Sat, 27 Jun 2015 21:37:09 +0000
Subject: [Bro] Threat Intelligence Management
In-Reply-To: <CAC8+5eML6Jsy7U4DkfdTW1cTz1VZwQAZApdzX+vfxeDAekaS+A@mail.gmail.com>
References: <CAN4C-D=9tS0skytp2AeJDBUNxreQV0_AkrASc8-AA2Ls2bDFVg@mail.gmail.com>
	<9f25f9f5-be30-41a4-beda-e65b0ec32c49@email.android.com>
	<CAC8+5eML6Jsy7U4DkfdTW1cTz1VZwQAZApdzX+vfxeDAekaS+A@mail.gmail.com>
Message-ID: <FA19949E-1476-449F-A1F8-6AB1415E4182@nswcsystems.co.uk>

Hi,
I tried using criticalstack, as it sounds like a really cool idea. I just can?t seem to get any events from it.

Should events go to the notice.log or the intel.log?

I tried a ping from an address present in the feed then looked for output and I have conn.log ICMP entry and a syslog entry but nothing else.
Andys-MacBook-Air:~ andy$ ping 89.106.121.76

[root at bro current]# grep -l '89.106.121.76' *.log
conn.log
syslog.log

1435439487.024865 C6HBUkZ7i07zlYE5a 172.31.254.179 8 89.106.121.76 0 icmp - 9.123324 560 560 OTH T 0 - 1840 10 840 (empty) - BG - - 22.872499 43.990002

I have some Intel loaded from CIF2 and that works OK, I use the test event:
Andys-MacBook-Air:~ andy$ curl http://testmyids.com
uid=0(root) gid=0(root) groups=0(root)
intel.log
1435439895.054961 CaEWz015AEjRJRruN2 172.31.254.179 55025 172.31.254.80 53 - - - testmyids.com<http://testmyids.com> Intel::DOMAIN DNS::IN_REQUEST Tester
1435439895.054965 COdqds1DkdarGlSnY1 172.31.254.179 53210 172.31.254.80 53 - - - testmyids.com<http://testmyids.com> Intel::DOMAIN DNS::IN_REQUEST Tester
1435439895.055305 CLcqwd2xLkH0MUUtf3 172.31.254.80 50910 8.8.4.4 53 - - - testmyids.com<http://testmyids.com> Intel::DOMAIN DNS::IN_REQUEST Tester
1435439895.055309 Cwdyhm1vbT1SnTiSG1 172.31.254.80 50639 8.8.4.4 53 - - - testmyids.com<http://testmyids.com> Intel::DOMAIN DNS::IN_REQUEST Tester
1435439895.253858 CtMoHr3h546C8UmdSi 172.31.254.179 50214 82.165.177.154 80 - - - testmyids.com<http://testmyids.com> Intel::DOMAIN HTTP::IN_HOST_HEADER Tester

Am I doing something wrong?

Kind regards,
Andy
Andrew.Ratcliffe at NSWCSystems.co.uk<mailto:Andrew.Ratcliffe at NSWCSystems.co.uk>
CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
Blog.InfoSecMatters.net<http://blog.infosecmatters.net/>






On 25 Jun 2015, at 13:51, Liam Randall <liam.randall at gmail.com<mailto:liam.randall at gmail.com>> wrote:

No Critical Stack is entirely custom; we are not building a TIP.  We wanted to have an easy way to have actionable into stream into bro as it is to discovered so we built it.  We thought others would want it as well so we make it freely available to the community.  We are getting ready to launch a new extension to it called KITTY- Keep Intel Transactions To Yourself that allow you to privately share and deploy 100's of Millions of indicators in a fast memory efficient way.  It integrates directly with our online marketplace- we deployed our first test clients this week.  We'll announce more shortly @CriticalStack .

For TIPs there are a lot of great solutions you should look at:

Free:
MISP
CRITS

Commercial:
Soltra Edge (has a free version)
ThreatConnect
ThreatStream
ThreatQ (ThreatQuotient)
BrightPoint Security (formerly Vorstack)


V/r,

Liam Randall


On Thu, Jun 25, 2015 at 8:37 AM, Harry Hoffman <hhoffman at ip-solutions.net<mailto:hhoffman at ip-solutions.net>> wrote:
Is critical stack based upon CIF (collective intelligence framework)?

It looks very similar.

Cheers,
Harry


On Jun 25, 2015 7:44 AM, Heine Lysemose <lysemose at gmail.com<mailto:lysemose at gmail.com>> wrote:
>
> Hi
>
> I encourage you to have a look at, https://intel.criticalstack.com/
>
> Best,
> Lysemose
>
> On Thu, Jun 25, 2015 at 1:31 PM, Jan Grashofer <jan.grashofer at cern.ch<mailto:jan.grashofer at cern.ch>> wrote:
>>
>> Hi all,
>>
>> I am having a look at Threat Intelligence Management solutions, which can be used with Bro. What do you use and what are your experiences?
>>
>> Regards,
>> Jan
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org<mailto:bro at bro-ids.org>
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro<http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>
>

_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro<http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>

_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150627/f40c0ba2/attachment-0001.html 

From liburdi.joshua at gmail.com  Sat Jun 27 15:55:17 2015
From: liburdi.joshua at gmail.com (Josh Liburdi)
Date: Sat, 27 Jun 2015 15:55:17 -0700 (PDT)
Subject: [Bro] Threat Intelligence Management
In-Reply-To: <FA19949E-1476-449F-A1F8-6AB1415E4182@nswcsystems.co.uk>
References: <FA19949E-1476-449F-A1F8-6AB1415E4182@nswcsystems.co.uk>
Message-ID: <1435445717505.28d0ca5f@Nodemailer>

Andy,


By default the Intel framework only generates log entries for IP addresses if the connection is a fully established TCP connection. That's probably why pinging an IP did not generate an entry.




Josh







On Saturday, Jun 27, 2015 at 5:39 PM, Andrew Ratcliffe <andrew.ratcliffe at nswcsystems.co.uk>, wrote:

Hi,

I tried using criticalstack, as it sounds like a really cool idea. I just can?t seem to get any events from it.




Should events go to the notice.log or the intel.log?




I tried a ping from an address present in the feed then looked for output and I have conn.log ICMP entry and a syslog entry but nothing else.

Andys-MacBook-Air:~ andy$ ping 89.106.121.76

?

[root at bro current]# grep -l '89.106.121.76' *.log

conn.log

syslog.log





1435439487.024865 C6HBUkZ7i07zlYE5a
172.31.254.179 
8 89.106.121.76
0 icmp
- 9.123324
560 560
OTH T
0 -
1840 10
840 (empty)
- BG
- -
22.872499 43.990002






I have some Intel loaded from CIF2 and that works OK, I use the test event:



Andys-MacBook-Air:~ andy$ curl http://testmyids.com



uid=0(root) gid=0(root) groups=0(root)



intel.log



1435439895.054961 CaEWz015AEjRJRruN2
172.31.254.179 
55025 172.31.254.80
53 -
- -
testmyids.com
Intel::DOMAIN 
DNS::IN_REQUEST Tester


1435439895.054965 COdqds1DkdarGlSnY1
172.31.254.179 
53210 172.31.254.80
53 -
- -
testmyids.com
Intel::DOMAIN 
DNS::IN_REQUEST Tester


1435439895.055305 CLcqwd2xLkH0MUUtf3
172.31.254.80 
50910 8.8.4.4
53 -
- -
testmyids.com
Intel::DOMAIN 
DNS::IN_REQUEST Tester


1435439895.055309 Cwdyhm1vbT1SnTiSG1
172.31.254.80 
50639 8.8.4.4
53 -
- -
testmyids.com
Intel::DOMAIN 
DNS::IN_REQUEST Tester


1435439895.253858 CtMoHr3h546C8UmdSi
172.31.254.179 
50214 82.165.177.154
80 -
- -
testmyids.com
Intel::DOMAIN 
HTTP::IN_HOST_HEADER 
Tester






Am I doing something wrong?




Kind regards,


















Andy


Andrew.Ratcliffe at NSWCSystems.co.uk

CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
Blog.InfoSecMatters.net








































On 25 Jun 2015, at 13:51, Liam Randall <liam.randall at gmail.com> wrote:



No Critical Stack is entirely custom; we are not building a TIP.? We wanted to have an easy way to have actionable into stream into bro as it is to discovered so we built it.? We thought others would want it as well so we make it freely
 available to the community.? We are getting ready to launch a new extension to it called KITTY- Keep Intel Transactions To Yourself that allow you to privately share and deploy 100's of Millions of indicators in a fast memory efficient way.? It integrates
 directly with our online marketplace- we deployed our first test clients this week.? We'll announce more shortly @CriticalStack .



For TIPs there are a lot of great solutions you should look at:




Free:

MISP

CRITS




Commercial:


Soltra Edge (has a free version)



ThreatConnect



ThreatStream



ThreatQ?(ThreatQuotient)



BrightPoint?Security (formerly?Vorstack)



?







V/r,




Liam Randall








On Thu, Jun 25, 2015 at 8:37 AM, Harry Hoffman 
<hhoffman at ip-solutions.net> wrote:

Is critical stack based upon CIF (collective intelligence framework)?


It looks very similar.


Cheers,

Harry





On Jun 25, 2015 7:44 AM, Heine Lysemose <lysemose at gmail.com> wrote:

>

> Hi

>

> I encourage you to have a look at,?https://intel.criticalstack.com/

>

> Best,

> Lysemose

>

> On Thu, Jun 25, 2015 at 1:31 PM, Jan Grashofer <jan.grashofer at cern.ch> wrote:

>>

>> Hi all,

>>

>> I am having a look at Threat Intelligence Management solutions, which can be used with Bro. What do you use and what are your experiences?

>>

>> Regards,

>> Jan

>>

>> _______________________________________________

>> Bro mailing list

>> bro at bro-ids.org

>> 
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

>

>


_______________________________________________

Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro










_______________________________________________

Bro mailing list
bro at bro-ids.org

http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150627/be045942/attachment-0001.html 

From anthony.kasza at gmail.com  Sat Jun 27 22:06:26 2015
From: anthony.kasza at gmail.com (anthony kasza)
Date: Sat, 27 Jun 2015 22:06:26 -0700
Subject: [Bro] HTTP Range/Content-Range PUT and POST Requests
Message-ID: <CAEZw2bwrFBz7A4=BPKBsfzjJnCpUfk5z5j+vg4w+VtvQ22uvug@mail.gmail.com>

Has anyone on list ever seen HTTP client uploads via PUT or POST requests
use Range or Content-Range headers? I'm wondering if any major web servers
support random access bytes based writes of resources similar to how many
support random access byte based reads of resources, see here <
http://stackoverflow.com/questions/716680/difference-between-content-range-and-range-headers>.

I have a PoC for random access writes here <
https://github.com/anthonykasza/frankendropper/blob/master/frankenexfiller.py>
but don't know how practical it is.

I'd like to see how Bro's file framework handles these random access write
requests. It handles random access reads nicely. I've tested it on a server
(example.com) that does not support this request and returns 4xx responses.
The files framework doesn't seem to extract the uploaded file, but that my
be due to the response status code. Thanks all,

-AK
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150627/6f244028/attachment.html 

From jan.grashofer at cern.ch  Sun Jun 28 01:03:49 2015
From: jan.grashofer at cern.ch (Jan Grashofer)
Date: Sun, 28 Jun 2015 08:03:49 +0000
Subject: [Bro] Bro's limitations with high worker count and memory
 exhaustion
In-Reply-To: <CA+J8G6FgNij+Tu9Kk8jQNe=1kAs0z7zf6D_5-ATB8VZE2nmY4A@mail.gmail.com>
References: <CA+J8G6FgNij+Tu9Kk8jQNe=1kAs0z7zf6D_5-ATB8VZE2nmY4A@mail.gmail.com>
Message-ID: <E3142ED5D6744B4C8E17323B862FEDE5DF6A8B@CERNXCHG62.cern.ch>

I experienced similar problems (memory gets eaten up quickly and workers crash with segfault) using tcmalloc. Which malloc do you use?



Regards,

Jan



________________________________
From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Baxter Milliwew [baxter.milliwew at gmail.com]
Sent: Friday, June 26, 2015 23:03
To: bro at bro.org
Subject: [Bro] Bro's limitations with high worker count and memory exhaustion

There's some sort of association between memory exhaustion and a high number of workers.  The poor man's fix would be to purchase new servers with higher CPU speeds as that would reduce the worker count.  Issues with high worker count and/or memory exhaustion appears to be a well know problem based on the mailing list archives.

In the current version of bro-2.4 my previous configuration immediately causes the manager to crash: 15 proxies, 155 workers.  To resolve this I've lowered the count to 10 proxies and 140 workers.  However even with this configuration the manager process will exhaust all memory and crash within about 2 hours.

The manager is threaded; I think this is an issue with the threading behavior between manager, proxies, and workers.  Debugging threading problems is complex and I'm a complete novice.. my current tutorial is using information from a stack overflow thread:

http://stackoverflow.com/questions/981011/c-programming-debugging-with-pthreads

Does anyone else have this problem ?  What have you tried and what do you suggest ?

Thanks





1435347409.458185       worker-2-18     parent  -       -       -       info    [#10000/10.1.1.1:36994<http://10.1.1.1:36994>] peer sent class "control"

1435347409.458185       worker-2-18     parent  -       -       -       info    [#10000/10.1.1.1:36994<http://10.1.1.1:36994>] phase: handshake

1435347409.661085       worker-2-18     parent  -       -       -       info    [#10000/10.1.1.1:36994<http://10.1.1.1:36994>] request for unknown event save_results

1435347409.661085       worker-2-18     parent  -       -       -       info    [#10000/10.1.1.1:36994<http://10.1.1.1:36994>] registered for event Control::peer_status_response

1435347409.694858       worker-2-18     parent  -       -       -       info    [#10000/10.1.1.1:36994<http://10.1.1.1:36994>] peer does not support 64bit PIDs; using compatibility mode

1435347409.694858       worker-2-18     parent  -       -       -       info    [#10000/10.1.1.1:36994<http://10.1.1.1:36994>] peer is a Broccoli

1435347409.694858       worker-2-18     parent  -       -       -       info    [#10000/10.1.1.1:36994<http://10.1.1.1:36994>] phase: running

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150628/80b467f5/attachment.html 

From andrew.ratcliffe at nswcsystems.co.uk  Sun Jun 28 15:02:18 2015
From: andrew.ratcliffe at nswcsystems.co.uk (Andrew Ratcliffe)
Date: Sun, 28 Jun 2015 22:02:18 +0000
Subject: [Bro] Threat Intelligence Management
In-Reply-To: <1435445717505.28d0ca5f@Nodemailer>
References: <FA19949E-1476-449F-A1F8-6AB1415E4182@nswcsystems.co.uk>
	<1435445717505.28d0ca5f@Nodemailer>
Message-ID: <B01CE95A-B0AA-4058-9EA7-30BE1B674E0D@nswcsystems.co.uk>

Hi Josh,
Thanks for pointing that out. However, I still seem to have a problem:
www.etiksecimler.com/appraiser/ipad/	Intel::URL	from http://www.phishtank.com/phish_detail.php?phish_id=3266591 via intel.criticalstack.com	F
Use Curl to get the URL
Andys-MacBook-Air:~ andy$ curl www.etiksecimler.com/appraiser/ipad/
Still no intel.log entry
[root at bro current]# grep -l www.etiksecimler.com *.log
dns.log
http.log

# Critical Stack, Inc - https://intel.criticalstack.com
@load /opt/critical-stack/frameworks/intel
# Uncomment the following line to enable detection of the heartbleed attack. Enabling
# this might impact performance a bit.
# @load policy/protocols/ssl/heartbleed
@load conn-geoip2.bro
@load intel-2.bro
#@load bpf-filter.bro

Kind regards,
Andy
Andrew.Ratcliffe at NSWCSystems.co.uk
CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
Blog.InfoSecMatters.net <http://blog.infosecmatters.net/>






> On 27 Jun 2015, at 23:55, Josh Liburdi <liburdi.joshua at gmail.com> wrote:
> 
> Andy,
> 
> By default the Intel framework only generates log entries for IP addresses if the connection is a fully established TCP connection. That's probably why pinging an IP did not generate an entry.
> 
> Josh
> 
> On Saturday, Jun 27, 2015 at 5:39 PM, Andrew Ratcliffe <andrew.ratcliffe at nswcsystems.co.uk <mailto:andrew.ratcliffe at nswcsystems.co.uk>>, wrote:
> Hi,
> I tried using criticalstack, as it sounds like a really cool idea. I just can?t seem to get any events from it.
> 
> Should events go to the notice.log or the intel.log?
> 
> I tried a ping from an address present in the feed then looked for output and I have conn.log ICMP entry and a syslog entry but nothing else.
> Andys-MacBook-Air:~ andy$ ping 89.106.121.76
> 
> [root at bro current]# grep -l '89.106.121.76' *.log
> conn.log
> syslog.log
> 
> 1435439487.024865 C6HBUkZ7i07zlYE5a
> 172.31.254.179  8 89.106.121.76
> 0 icmp
> - 9.123324
> 560 560
> OTH T
> 0 -
> 1840 10
> 840 (empty)
> - BG
> - -
> 22.872499 43.990002
> 
> I have some Intel loaded from CIF2 and that works OK, I use the test event:
> Andys-MacBook-Air:~ andy$ curl http://testmyids.com <http://testmyids.com/>
> uid=0(root) gid=0(root) groups=0(root)
> intel.log
> 1435439895.054961 CaEWz015AEjRJRruN2
> 172.31.254.179  55025 172.31.254.80
> 53 -
> - -
> testmyids.com <http://testmyids.com/>
> Intel::DOMAIN  DNS::IN_REQUEST Tester
> 1435439895.054965 COdqds1DkdarGlSnY1
> 172.31.254.179  53210 172.31.254.80
> 53 -
> - -
> testmyids.com <http://testmyids.com/>
> Intel::DOMAIN  DNS::IN_REQUEST Tester
> 1435439895.055305 CLcqwd2xLkH0MUUtf3
> 172.31.254.80  50910 8.8.4.4
> 53 -
> - -
> testmyids.com <http://testmyids.com/>
> Intel::DOMAIN  DNS::IN_REQUEST Tester
> 1435439895.055309 Cwdyhm1vbT1SnTiSG1
> 172.31.254.80  50639 8.8.4.4
> 53 -
> - -
> testmyids.com <http://testmyids.com/>
> Intel::DOMAIN  DNS::IN_REQUEST Tester
> 1435439895.253858 CtMoHr3h546C8UmdSi
> 172.31.254.179  50214 82.165.177.154
> 80 -
> - -
> testmyids.com <http://testmyids.com/>
> Intel::DOMAIN  HTTP::IN_HOST_HEADER  Tester
> 
> Am I doing something wrong?
> 
> Kind regards,
> Andy
> Andrew.Ratcliffe at NSWCSystems.co.uk <mailto:Andrew.Ratcliffe at NSWCSystems.co.uk>
> CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
> Blog.InfoSecMatters.net <http://blog.infosecmatters.net/>
> 
> 
> 
> 
> 
> 
>> On 25 Jun 2015, at 13:51, Liam Randall <liam.randall at gmail.com <mailto:liam.randall at gmail.com>> wrote:
>> 
>> No Critical Stack is entirely custom; we are not building a TIP.  We wanted to have an easy way to have actionable into stream into bro as it is to discovered so we built it.  We thought others would want it as well so we make it freely available to the community.  We are getting ready to launch a new extension to it called KITTY- Keep Intel Transactions To Yourself that allow you to privately share and deploy 100's of Millions of indicators in a fast memory efficient way.  It integrates directly with our online marketplace- we deployed our first test clients this week.  We'll announce more shortly @CriticalStack .
>> 
>> For TIPs there are a lot of great solutions you should look at:
>> 
>> Free:
>> MISP
>> CRITS
>> 
>> Commercial:
>> Soltra Edge (has a free version)
>> ThreatConnect
>> ThreatStream
>> ThreatQ (ThreatQuotient)
>> BrightPoint Security (formerly Vorstack)
>> 
>> 
>> V/r,
>> 
>> Liam Randall
>> 
>> 
>> On Thu, Jun 25, 2015 at 8:37 AM, Harry Hoffman <hhoffman at ip-solutions.net <mailto:hhoffman at ip-solutions.net>> wrote:
>> Is critical stack based upon CIF (collective intelligence framework)?
>> 
>> It looks very similar.
>> 
>> Cheers,
>> Harry
>> 
>> 
>> On Jun 25, 2015 7:44 AM, Heine Lysemose <lysemose at gmail.com <mailto:lysemose at gmail.com>> wrote:
>> >
>> > Hi
>> >
>> > I encourage you to have a look at, https://intel.criticalstack.com/ <https://intel.criticalstack.com/>
>> >
>> > Best,
>> > Lysemose
>> >
>> > On Thu, Jun 25, 2015 at 1:31 PM, Jan Grashofer <jan.grashofer at cern.ch <mailto:jan.grashofer at cern.ch>> wrote:
>> >>
>> >> Hi all,
>> >>
>> >> I am having a look at Threat Intelligence Management solutions, which can be used with Bro. What do you use and what are your experiences?
>> >>
>> >> Regards,
>> >> Jan
>> >>
>> >> _______________________________________________
>> >> Bro mailing list
>> >> bro at bro-ids.org <mailto:bro at bro-ids.org>
>> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>> >
>> >
>> 
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org <mailto:bro at bro-ids.org>
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org <mailto:bro at bro-ids.org>
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150628/a861d470/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 235 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150628/a861d470/attachment-0001.bin 

From baxter.milliwew at gmail.com  Sun Jun 28 19:18:27 2015
From: baxter.milliwew at gmail.com (Baxter Milliwew)
Date: Sun, 28 Jun 2015 19:18:27 -0700
Subject: [Bro] Bro's limitations with high worker count and memory
	exhaustion
In-Reply-To: <E3142ED5D6744B4C8E17323B862FEDE5DF6A8B@CERNXCHG62.cern.ch>
References: <CA+J8G6FgNij+Tu9Kk8jQNe=1kAs0z7zf6D_5-ATB8VZE2nmY4A@mail.gmail.com>
	<E3142ED5D6744B4C8E17323B862FEDE5DF6A8B@CERNXCHG62.cern.ch>
Message-ID: <CA+J8G6HRjn4cXb+mjBw-XdJaBHOG-MgdwtKbM0kWKE2B8UPQuQ@mail.gmail.com>

Looks like malloc from glibc, default on Ubuntu.  I will try jemalloc and
others.



On Sun, Jun 28, 2015 at 1:03 AM, Jan Grashofer <jan.grashofer at cern.ch>
wrote:

>  I experienced similar problems (memory gets eaten up quickly and workers
> crash with segfault) using tcmalloc. Which malloc do you use?
>
>
>
> Regards,
>
> Jan
>
>
>  ------------------------------
> *From:* bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Baxter
> Milliwew [baxter.milliwew at gmail.com]
> *Sent:* Friday, June 26, 2015 23:03
> *To:* bro at bro.org
> *Subject:* [Bro] Bro's limitations with high worker count and memory
> exhaustion
>
>   There's some sort of association between memory exhaustion and a high
> number of workers.  The poor man's fix would be to purchase new servers
> with higher CPU speeds as that would reduce the worker count.  Issues with
> high worker count and/or memory exhaustion appears to be a well know
> problem based on the mailing list archives.
>
>  In the current version of bro-2.4 my previous configuration immediately
> causes the manager to crash: 15 proxies, 155 workers.  To resolve this I've
> lowered the count to 10 proxies and 140 workers.  However even with this
> configuration the manager process will exhaust all memory and crash within
> about 2 hours.
>
>  The manager is threaded; I think this is an issue with the threading
> behavior between manager, proxies, and workers.  Debugging threading
> problems is complex and I'm a complete novice.. my current tutorial is
> using information from a stack overflow thread:
>
>
> http://stackoverflow.com/questions/981011/c-programming-debugging-with-pthreads
>
>  Does anyone else have this problem ?  What have you tried and what do
> you suggest ?
>
>  Thanks
>
>
>
>
>   1435347409.458185       worker-2-18     parent  -       -       -
> info    [#10000/10.1.1.1:36994] peer sent class "control"
>
> 1435347409.458185       worker-2-18     parent  -       -       -
> info    [#10000/10.1.1.1:36994] phase: handshake
>
> 1435347409.661085       worker-2-18     parent  -       -       -
> info    [#10000/10.1.1.1:36994] request for unknown event save_results
>
> 1435347409.661085       worker-2-18     parent  -       -       -
> info    [#10000/10.1.1.1:36994] registered for event
> Control::peer_status_response
>
> 1435347409.694858       worker-2-18     parent  -       -       -
> info    [#10000/10.1.1.1:36994] peer does not support 64bit PIDs; using
> compatibility mode
>
> 1435347409.694858       worker-2-18     parent  -       -       -
> info    [#10000/10.1.1.1:36994] peer is a Broccoli
>
> 1435347409.694858       worker-2-18     parent  -       -       -
> info    [#10000/10.1.1.1:36994] phase: running
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150628/d8193c83/attachment.html 

From lists at g-clef.net  Mon Jun 29 09:21:41 2015
From: lists at g-clef.net (Aaron Gee-Clough)
Date: Mon, 29 Jun 2015 12:21:41 -0400
Subject: [Bro] Logging plugin won't configure
Message-ID: <55917095.3060903@g-clef.net>


All,

I'm attempting to build a logging plugin, and hitting a bit of a brick
wall. I have what I think is a reasonable framework, based on the
elasticsearch logging plugin, but when I go to configure it, I get :

g-clef at yog-sothoth:~/workspace/Bro/C++/KafkaLogger$ ./configure
--bro-dist=/home/g-clef/Downloads/bro-2.4/
Build Directory        : build
Bro Source Directory   : /home/g-clef/Downloads/bro-2.4
-- Bro executable      : /home/g-clef/Downloads/bro-2.4/build/src/bro
-- Bro source          : /home/g-clef/Downloads/bro-2.4
-- Bro build           : /home/g-clef/Downloads/bro-2.4/build
-- Bro install prefix  : /usr/local/bro
-- Bro plugin directory: /usr/local/bro/lib/bro/plugins
-- Bro debug mode      : false
RegularExpression::compile(): Nested *?+.
RegularExpression::compile(): Error in compile.
CMake Error at /home/g-clef/Downloads/bro-2.4/cmake/BifCl.cmake:113
(string):
  string sub-command REGEX, mode REPLACE failed to compile regex
  "/home/g-clef/workspace/Bro/C++/KafkaLogger/build/src/".
Call Stack (most recent call first):
  /home/g-clef/Downloads/bro-2.4/cmake/BroPluginDynamic.cmake:112
(bif_target)
  /home/g-clef/Downloads/bro-2.4/cmake/BroPluginCommon.cmake:69
(bro_plugin_bif_dynamic)
  CMakeLists.txt:10 (bro_plugin_bif)


-- Configuring incomplete, errors occurred!
See also
"/home/g-clef/workspace/Bro/C++/KafkaLogger/build/CMakeFiles/CMakeOutput.log".

This looks like a regex error in the bro code, rather than in my code,
but I'm not sure what that code is trying to do. Any ideas?

Thanks.

aaron

From dnthayer at illinois.edu  Mon Jun 29 09:41:19 2015
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Mon, 29 Jun 2015 11:41:19 -0500
Subject: [Bro] Logging plugin won't configure
In-Reply-To: <55917095.3060903@g-clef.net>
References: <55917095.3060903@g-clef.net>
Message-ID: <5591752F.5020904@illinois.edu>

Try renaming the "C++" directory to something else (such as
"Cplusplus").



On 06/29/2015 11:21 AM, Aaron Gee-Clough wrote:
>
> All,
>
> I'm attempting to build a logging plugin, and hitting a bit of a brick
> wall. I have what I think is a reasonable framework, based on the
> elasticsearch logging plugin, but when I go to configure it, I get :
>
> g-clef at yog-sothoth:~/workspace/Bro/C++/KafkaLogger$ ./configure
> --bro-dist=/home/g-clef/Downloads/bro-2.4/
> Build Directory        : build
> Bro Source Directory   : /home/g-clef/Downloads/bro-2.4
> -- Bro executable      : /home/g-clef/Downloads/bro-2.4/build/src/bro
> -- Bro source          : /home/g-clef/Downloads/bro-2.4
> -- Bro build           : /home/g-clef/Downloads/bro-2.4/build
> -- Bro install prefix  : /usr/local/bro
> -- Bro plugin directory: /usr/local/bro/lib/bro/plugins
> -- Bro debug mode      : false
> RegularExpression::compile(): Nested *?+.
> RegularExpression::compile(): Error in compile.
> CMake Error at /home/g-clef/Downloads/bro-2.4/cmake/BifCl.cmake:113
> (string):
>    string sub-command REGEX, mode REPLACE failed to compile regex
>    "/home/g-clef/workspace/Bro/C++/KafkaLogger/build/src/".
> Call Stack (most recent call first):
>    /home/g-clef/Downloads/bro-2.4/cmake/BroPluginDynamic.cmake:112
> (bif_target)
>    /home/g-clef/Downloads/bro-2.4/cmake/BroPluginCommon.cmake:69
> (bro_plugin_bif_dynamic)
>    CMakeLists.txt:10 (bro_plugin_bif)
>
>
> -- Configuring incomplete, errors occurred!
> See also
> "/home/g-clef/workspace/Bro/C++/KafkaLogger/build/CMakeFiles/CMakeOutput.log".
>
> This looks like a regex error in the bro code, rather than in my code,
> but I'm not sure what that code is trying to do. Any ideas?
>
> Thanks.
>
> aaron

From lists at g-clef.net  Mon Jun 29 09:49:47 2015
From: lists at g-clef.net (Aaron Gee-Clough)
Date: Mon, 29 Jun 2015 12:49:47 -0400
Subject: [Bro] Logging plugin won't configure
References: <55917095.3060903@g-clef.net> <5591752F.5020904@illinois.edu>
Message-ID: <5591772B.8070402@g-clef.net>


That was it. Thank you. Now it doesn't make, but that's clearly my problem.

Thanks again.

aaron

On 06/29/2015 12:41 PM, Daniel Thayer wrote:
> 
> Try renaming the "C++" directory to something else (such as
> "Cplusplus").
> 
> 
> 
> On 06/29/2015 11:21 AM, Aaron Gee-Clough wrote:
>>
>> All,
>>
>> I'm attempting to build a logging plugin, and hitting a bit of a brick
>> wall. I have what I think is a reasonable framework, based on the
>> elasticsearch logging plugin, but when I go to configure it, I get :
>>
>> g-clef at yog-sothoth:~/workspace/Bro/C++/KafkaLogger$ ./configure
>> --bro-dist=/home/g-clef/Downloads/bro-2.4/
>> Build Directory        : build
>> Bro Source Directory   : /home/g-clef/Downloads/bro-2.4
>> -- Bro executable      : /home/g-clef/Downloads/bro-2.4/build/src/bro
>> -- Bro source          : /home/g-clef/Downloads/bro-2.4
>> -- Bro build           : /home/g-clef/Downloads/bro-2.4/build
>> -- Bro install prefix  : /usr/local/bro
>> -- Bro plugin directory: /usr/local/bro/lib/bro/plugins
>> -- Bro debug mode      : false
>> RegularExpression::compile(): Nested *?+.
>> RegularExpression::compile(): Error in compile.
>> CMake Error at /home/g-clef/Downloads/bro-2.4/cmake/BifCl.cmake:113
>> (string):
>>    string sub-command REGEX, mode REPLACE failed to compile regex
>>    "/home/g-clef/workspace/Bro/C++/KafkaLogger/build/src/".
>> Call Stack (most recent call first):
>>    /home/g-clef/Downloads/bro-2.4/cmake/BroPluginDynamic.cmake:112
>> (bif_target)
>>    /home/g-clef/Downloads/bro-2.4/cmake/BroPluginCommon.cmake:69
>> (bro_plugin_bif_dynamic)
>>    CMakeLists.txt:10 (bro_plugin_bif)
>>
>>
>> -- Configuring incomplete, errors occurred!
>> See also
>> "/home/g-clef/workspace/Bro/C++/KafkaLogger/build/CMakeFiles/CMakeOutput.log".
>>
>>
>> This looks like a regex error in the bro code, rather than in my code,
>> but I'm not sure what that code is trying to do. Any ideas?
>>
>> Thanks.
>>
>> aaron

From gl89 at cornell.edu  Mon Jun 29 11:48:30 2015
From: gl89 at cornell.edu (Glenn Forbes Fleming Larratt)
Date: Mon, 29 Jun 2015 14:48:30 -0400 (EDT)
Subject: [Bro] Myricom + Bro: are these log entries normal?
Message-ID: <alpine.LRH.2.02.1506291445260.15395@hazel.serverfarm.cornell.edu>

I'm running 11 worker processes on each of my five clustered listeners, 
and each of them, like clockwork, spits out 290 apiece (i.e. every five 
minutes) of these three log entries every day (thank you, logwatch!):

Jun 29 14:25:08 bro01.serverfarm.cornell.edu kernel: myri_snf WARN: SNF rx attach: out of free rx rings. app_id=-1 pid=56578 rings_attached=0 rings_requested=0
Jun 29 14:25:08 bro01.serverfarm.cornell.edu kernel: myri_snf WARN: Failed to attach to ring -1 with err=16
Jun 29 14:25:08 bro01.serverfarm.cornell.edu kernel: myri_snf WARN: eth4: endpt 76, early enable failed

I'm getting oodles of traffic to both my Bro workers and the Snort 
instances running as SNF_APP_ID 2 ; are these logs anything I need to 
worry about, in anyone's experience?

Thanks,
-- 
Glenn Forbes Fleming Larratt
Cornell University IT Security Office

From lists at g-clef.net  Mon Jun 29 12:24:05 2015
From: lists at g-clef.net (Aaron Gee-Clough)
Date: Mon, 29 Jun 2015 15:24:05 -0400
Subject: [Bro] Another logging plugin question
Message-ID: <55919B55.8090400@g-clef.net>


Related to my previous question about a logging plugin, I can get my
plugin to compile (and get bro to recognize it as a plugin), but only if
I comment out a line that appears in the "elasticsearch" and
"dataseries" plugins. The line is:
	AddComponent(new ::logging::Component("KafkaWriter",
::logging::writer::KafkaWriter::Instantiate));


If I leave this line in, I get compile errors:

/home/g-clef/workspace/Bro/Cplusplus/KafkaLogger/src/Plugin.cc: In
member function ?virtual plugin::Configuration
plugin::Kafka_KafkaWriter::Plugin::Configure()?:
/home/g-clef/workspace/Bro/Cplusplus/KafkaLogger/src/Plugin.cc:13:19:
error: expected type-specifier before ?::? token
  AddComponent(new ::logging::Component("KafkaWriter",
::logging::writer::KafkaWriter::Instantiate));


Is this line necessary in logging plugins? Or does it only apply to
bro-built-in plugins?

Thanks.

aaron

From liburdi.joshua at gmail.com  Mon Jun 29 13:35:36 2015
From: liburdi.joshua at gmail.com (Josh Liburdi)
Date: Mon, 29 Jun 2015 16:35:36 -0400
Subject: [Bro] Threat Intelligence Management
In-Reply-To: <B01CE95A-B0AA-4058-9EA7-30BE1B674E0D@nswcsystems.co.uk>
References: <FA19949E-1476-449F-A1F8-6AB1415E4182@nswcsystems.co.uk>
	<1435445717505.28d0ca5f@Nodemailer>
	<B01CE95A-B0AA-4058-9EA7-30BE1B674E0D@nswcsystems.co.uk>
Message-ID: <CANrCRiFpCNCivGOEXVBffdqW=HQ6GsoruNGPmr75PpucjvQL8w@mail.gmail.com>

Andy,

If you still have these log files (or can generate them again), can
you share the line from http.log that contains the URL indicator?

Thanks,
Josh

On Sun, Jun 28, 2015 at 6:02 PM, Andrew Ratcliffe
<andrew.ratcliffe at nswcsystems.co.uk> wrote:
> Hi Josh,
> Thanks for pointing that out. However, I still seem to have a problem:
> www.etiksecimler.com/appraiser/ipad/ Intel::URL from
> http://www.phishtank.com/phish_detail.php?phish_id=3266591 via
> intel.criticalstack.com F
> Use Curl to get the URL
> Andys-MacBook-Air:~ andy$ curl www.etiksecimler.com/appraiser/ipad/
> Still no intel.log entry
> [root at bro current]# grep -l www.etiksecimler.com *.log
> dns.log
> http.log
>
> # Critical Stack, Inc - https://intel.criticalstack.com
> @load /opt/critical-stack/frameworks/intel
> # Uncomment the following line to enable detection of the heartbleed attack.
> Enabling
> # this might impact performance a bit.
> # @load policy/protocols/ssl/heartbleed
> @load conn-geoip2.bro
> @load intel-2.bro
> #@load bpf-filter.bro
>
> Kind regards,
> Andy
> Andrew.Ratcliffe at NSWCSystems.co.uk
> CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
> Blog.InfoSecMatters.net
>
>
>
>
>
>
> On 27 Jun 2015, at 23:55, Josh Liburdi <liburdi.joshua at gmail.com> wrote:
>
> Andy,
>
> By default the Intel framework only generates log entries for IP addresses
> if the connection is a fully established TCP connection. That's probably why
> pinging an IP did not generate an entry.
>
> Josh
>
> On Saturday, Jun 27, 2015 at 5:39 PM, Andrew Ratcliffe
> <andrew.ratcliffe at nswcsystems.co.uk>, wrote:
>>
>> Hi,
>> I tried using criticalstack, as it sounds like a really cool idea. I just
>> can?t seem to get any events from it.
>>
>> Should events go to the notice.log or the intel.log?
>>
>> I tried a ping from an address present in the feed then looked for output
>> and I have conn.log ICMP entry and a syslog entry but nothing else.
>> Andys-MacBook-Air:~ andy$ ping 89.106.121.76
>>
>> [root at bro current]# grep -l '89.106.121.76' *.log
>> conn.log
>> syslog.log
>>
>> 1435439487.024865 C6HBUkZ7i07zlYE5a 172.31.254.179 8 89.106.121.76 0 icmp
>> - 9.123324 560 560 OTH T 0 - 1840 10 840 (empty) - BG - - 22.872499
>> 43.990002
>>
>> I have some Intel loaded from CIF2 and that works OK, I use the test
>> event:
>> Andys-MacBook-Air:~ andy$ curl http://testmyids.com
>> uid=0(root) gid=0(root) groups=0(root)
>> intel.log
>> 1435439895.054961 CaEWz015AEjRJRruN2 172.31.254.179 55025 172.31.254.80 53
>> - - - testmyids.com Intel::DOMAIN DNS::IN_REQUEST Tester
>> 1435439895.054965 COdqds1DkdarGlSnY1 172.31.254.179 53210 172.31.254.80 53
>> - - - testmyids.com Intel::DOMAIN DNS::IN_REQUEST Tester
>> 1435439895.055305 CLcqwd2xLkH0MUUtf3 172.31.254.80 50910 8.8.4.4 53 - - -
>> testmyids.com Intel::DOMAIN DNS::IN_REQUEST Tester
>> 1435439895.055309 Cwdyhm1vbT1SnTiSG1 172.31.254.80 50639 8.8.4.4 53 - - -
>> testmyids.com Intel::DOMAIN DNS::IN_REQUEST Tester
>> 1435439895.253858 CtMoHr3h546C8UmdSi 172.31.254.179 50214 82.165.177.154
>> 80 - - - testmyids.com Intel::DOMAIN HTTP::IN_HOST_HEADER Tester
>>
>> Am I doing something wrong?
>>
>> Kind regards,
>> Andy
>> Andrew.Ratcliffe at NSWCSystems.co.uk
>> CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
>> Blog.InfoSecMatters.net
>>
>>
>>
>>
>>
>>
>> On 25 Jun 2015, at 13:51, Liam Randall <liam.randall at gmail.com> wrote:
>>
>> No Critical Stack is entirely custom; we are not building a TIP.  We
>> wanted to have an easy way to have actionable into stream into bro as it is
>> to discovered so we built it.  We thought others would want it as well so we
>> make it freely available to the community.  We are getting ready to launch a
>> new extension to it called KITTY- Keep Intel Transactions To Yourself that
>> allow you to privately share and deploy 100's of Millions of indicators in a
>> fast memory efficient way.  It integrates directly with our online
>> marketplace- we deployed our first test clients this week.  We'll announce
>> more shortly @CriticalStack .
>>
>> For TIPs there are a lot of great solutions you should look at:
>>
>> Free:
>> MISP
>> CRITS
>>
>> Commercial:
>> Soltra Edge (has a free version)
>> ThreatConnect
>> ThreatStream
>> ThreatQ (ThreatQuotient)
>> BrightPoint Security (formerly Vorstack)
>>
>>
>> V/r,
>>
>> Liam Randall
>>
>>
>> On Thu, Jun 25, 2015 at 8:37 AM, Harry Hoffman <hhoffman at ip-solutions.net>
>> wrote:
>>>
>>> Is critical stack based upon CIF (collective intelligence framework)?
>>>
>>> It looks very similar.
>>>
>>> Cheers,
>>> Harry
>>>
>>>
>>> On Jun 25, 2015 7:44 AM, Heine Lysemose <lysemose at gmail.com> wrote:
>>> >
>>> > Hi
>>> >
>>> > I encourage you to have a look at, https://intel.criticalstack.com/
>>> >
>>> > Best,
>>> > Lysemose
>>> >
>>> > On Thu, Jun 25, 2015 at 1:31 PM, Jan Grashofer <jan.grashofer at cern.ch>
>>> > wrote:
>>> >>
>>> >> Hi all,
>>> >>
>>> >> I am having a look at Threat Intelligence Management solutions, which
>>> >> can be used with Bro. What do you use and what are your experiences?
>>> >>
>>> >> Regards,
>>> >> Jan
>>> >>
>>> >> _______________________________________________
>>> >> Bro mailing list
>>> >> bro at bro-ids.org
>>> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>> >
>>> >
>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>


From andrew.ratcliffe at nswcsystems.co.uk  Mon Jun 29 14:12:49 2015
From: andrew.ratcliffe at nswcsystems.co.uk (Andrew Ratcliffe)
Date: Mon, 29 Jun 2015 21:12:49 +0000
Subject: [Bro] Threat Intelligence Management
In-Reply-To: <CANrCRiFpCNCivGOEXVBffdqW=HQ6GsoruNGPmr75PpucjvQL8w@mail.gmail.com>
References: <FA19949E-1476-449F-A1F8-6AB1415E4182@nswcsystems.co.uk>
	<1435445717505.28d0ca5f@Nodemailer>
	<B01CE95A-B0AA-4058-9EA7-30BE1B674E0D@nswcsystems.co.uk>
	<CANrCRiFpCNCivGOEXVBffdqW=HQ6GsoruNGPmr75PpucjvQL8w@mail.gmail.com>
Message-ID: <26BC6390-DCA5-46D7-972A-D155D041F0B8@nswcsystems.co.uk>

Josh,
I tried a different one just so that it was current in the logs.

cwihosting.com/emsp/data/getproductrequest.htm	Intel::URL	from http://www.phishtank.com/phish_detail.php?phish_id=2479331 via intel.criticalstack.com	F
[root at bro intel]# cd /usr/local/bro/logs/current/
[root at bro current]# grep -l cwihosting.com *.log
dns.log
http.log
[root at bro current]# grep cwihosting.com http.log
1435611906.514899	C31ZazNObk3xTTk86	172.31.254.179	51734	72.52.170.179	80	1	GET	cwihosting.com	/emsp/data/getproductrequest.htm	-	curl/7.37.1	0	18464	200	OK	-	-	-	(empty)	-	-	-	-	-	FdGgt336pWjZZn8MBa	-
[root at bro current]#


Thanks

Kind regards,
Andy
Andrew.Ratcliffe at NSWCSystems.co.uk
CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
Blog.InfoSecMatters.net <http://blog.infosecmatters.net/>






> On 29 Jun 2015, at 21:35, Josh Liburdi <liburdi.joshua at gmail.com> wrote:
> 
> Andy,
> 
> If you still have these log files (or can generate them again), can
> you share the line from http.log that contains the URL indicator?
> 
> Thanks,
> Josh
> 
> On Sun, Jun 28, 2015 at 6:02 PM, Andrew Ratcliffe
> <andrew.ratcliffe at nswcsystems.co.uk> wrote:
>> Hi Josh,
>> Thanks for pointing that out. However, I still seem to have a problem:
>> www.etiksecimler.com/appraiser/ipad/ Intel::URL from
>> http://www.phishtank.com/phish_detail.php?phish_id=3266591 via
>> intel.criticalstack.com F
>> Use Curl to get the URL
>> Andys-MacBook-Air:~ andy$ curl www.etiksecimler.com/appraiser/ipad/
>> Still no intel.log entry
>> [root at bro current]# grep -l www.etiksecimler.com *.log
>> dns.log
>> http.log
>> 
>> # Critical Stack, Inc - https://intel.criticalstack.com
>> @load /opt/critical-stack/frameworks/intel
>> # Uncomment the following line to enable detection of the heartbleed attack.
>> Enabling
>> # this might impact performance a bit.
>> # @load policy/protocols/ssl/heartbleed
>> @load conn-geoip2.bro
>> @load intel-2.bro
>> #@load bpf-filter.bro
>> 
>> Kind regards,
>> Andy
>> Andrew.Ratcliffe at NSWCSystems.co.uk
>> CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
>> Blog.InfoSecMatters.net
>> 
>> 
>> 
>> 
>> 
>> 
>> On 27 Jun 2015, at 23:55, Josh Liburdi <liburdi.joshua at gmail.com> wrote:
>> 
>> Andy,
>> 
>> By default the Intel framework only generates log entries for IP addresses
>> if the connection is a fully established TCP connection. That's probably why
>> pinging an IP did not generate an entry.
>> 
>> Josh
>> 
>> On Saturday, Jun 27, 2015 at 5:39 PM, Andrew Ratcliffe
>> <andrew.ratcliffe at nswcsystems.co.uk>, wrote:
>>> 
>>> Hi,
>>> I tried using criticalstack, as it sounds like a really cool idea. I just
>>> can?t seem to get any events from it.
>>> 
>>> Should events go to the notice.log or the intel.log?
>>> 
>>> I tried a ping from an address present in the feed then looked for output
>>> and I have conn.log ICMP entry and a syslog entry but nothing else.
>>> Andys-MacBook-Air:~ andy$ ping 89.106.121.76
>>> 
>>> [root at bro current]# grep -l '89.106.121.76' *.log
>>> conn.log
>>> syslog.log
>>> 
>>> 1435439487.024865 C6HBUkZ7i07zlYE5a 172.31.254.179 8 89.106.121.76 0 icmp
>>> - 9.123324 560 560 OTH T 0 - 1840 10 840 (empty) - BG - - 22.872499
>>> 43.990002
>>> 
>>> I have some Intel loaded from CIF2 and that works OK, I use the test
>>> event:
>>> Andys-MacBook-Air:~ andy$ curl http://testmyids.com
>>> uid=0(root) gid=0(root) groups=0(root)
>>> intel.log
>>> 1435439895.054961 CaEWz015AEjRJRruN2 172.31.254.179 55025 172.31.254.80 53
>>> - - - testmyids.com Intel::DOMAIN DNS::IN_REQUEST Tester
>>> 1435439895.054965 COdqds1DkdarGlSnY1 172.31.254.179 53210 172.31.254.80 53
>>> - - - testmyids.com Intel::DOMAIN DNS::IN_REQUEST Tester
>>> 1435439895.055305 CLcqwd2xLkH0MUUtf3 172.31.254.80 50910 8.8.4.4 53 - - -
>>> testmyids.com Intel::DOMAIN DNS::IN_REQUEST Tester
>>> 1435439895.055309 Cwdyhm1vbT1SnTiSG1 172.31.254.80 50639 8.8.4.4 53 - - -
>>> testmyids.com Intel::DOMAIN DNS::IN_REQUEST Tester
>>> 1435439895.253858 CtMoHr3h546C8UmdSi 172.31.254.179 50214 82.165.177.154
>>> 80 - - - testmyids.com Intel::DOMAIN HTTP::IN_HOST_HEADER Tester
>>> 
>>> Am I doing something wrong?
>>> 
>>> Kind regards,
>>> Andy
>>> Andrew.Ratcliffe at NSWCSystems.co.uk
>>> CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
>>> Blog.InfoSecMatters.net
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On 25 Jun 2015, at 13:51, Liam Randall <liam.randall at gmail.com> wrote:
>>> 
>>> No Critical Stack is entirely custom; we are not building a TIP.  We
>>> wanted to have an easy way to have actionable into stream into bro as it is
>>> to discovered so we built it.  We thought others would want it as well so we
>>> make it freely available to the community.  We are getting ready to launch a
>>> new extension to it called KITTY- Keep Intel Transactions To Yourself that
>>> allow you to privately share and deploy 100's of Millions of indicators in a
>>> fast memory efficient way.  It integrates directly with our online
>>> marketplace- we deployed our first test clients this week.  We'll announce
>>> more shortly @CriticalStack .
>>> 
>>> For TIPs there are a lot of great solutions you should look at:
>>> 
>>> Free:
>>> MISP
>>> CRITS
>>> 
>>> Commercial:
>>> Soltra Edge (has a free version)
>>> ThreatConnect
>>> ThreatStream
>>> ThreatQ (ThreatQuotient)
>>> BrightPoint Security (formerly Vorstack)
>>> 
>>> 
>>> V/r,
>>> 
>>> Liam Randall
>>> 
>>> 
>>> On Thu, Jun 25, 2015 at 8:37 AM, Harry Hoffman <hhoffman at ip-solutions.net>
>>> wrote:
>>>> 
>>>> Is critical stack based upon CIF (collective intelligence framework)?
>>>> 
>>>> It looks very similar.
>>>> 
>>>> Cheers,
>>>> Harry
>>>> 
>>>> 
>>>> On Jun 25, 2015 7:44 AM, Heine Lysemose <lysemose at gmail.com> wrote:
>>>>> 
>>>>> Hi
>>>>> 
>>>>> I encourage you to have a look at, https://intel.criticalstack.com/
>>>>> 
>>>>> Best,
>>>>> Lysemose
>>>>> 
>>>>> On Thu, Jun 25, 2015 at 1:31 PM, Jan Grashofer <jan.grashofer at cern.ch>
>>>>> wrote:
>>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> I am having a look at Threat Intelligence Management solutions, which
>>>>>> can be used with Bro. What do you use and what are your experiences?
>>>>>> 
>>>>>> Regards,
>>>>>> Jan
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Bro mailing list
>>>>>> bro at bro-ids.org
>>>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>>> 
>>>>> 
>>>> 
>>>> _______________________________________________
>>>> Bro mailing list
>>>> bro at bro-ids.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>> 
>>> 
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>> 
>>> 
>> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150629/121bcdfc/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 235 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150629/121bcdfc/attachment-0001.bin 

From liam.randall at gmail.com  Mon Jun 29 14:21:55 2015
From: liam.randall at gmail.com (Liam Randall)
Date: Mon, 29 Jun 2015 17:21:55 -0400
Subject: [Bro] Threat Intelligence Management
In-Reply-To: <26BC6390-DCA5-46D7-972A-D155D041F0B8@nswcsystems.co.uk>
References: <FA19949E-1476-449F-A1F8-6AB1415E4182@nswcsystems.co.uk>
	<1435445717505.28d0ca5f@Nodemailer>
	<B01CE95A-B0AA-4058-9EA7-30BE1B674E0D@nswcsystems.co.uk>
	<CANrCRiFpCNCivGOEXVBffdqW=HQ6GsoruNGPmr75PpucjvQL8w@mail.gmail.com>
	<26BC6390-DCA5-46D7-972A-D155D041F0B8@nswcsystems.co.uk>
Message-ID: <CAC8+5ePK9W9yb0DGhW_2bXyq1JnWFcdkAAemNTpwypbGaMKHvg@mail.gmail.com>

Hey Andrew,

After installing did you do a

sudo broctl check
sudo broctl install
sudo broctl restart

You only need to perform that once and the future updates will be included
automatically.

If you have included  'load misc/loaded-scripts' in your local.bro you will
generate a loaded_scripts.log that you can use to verify that the scripts
are running:

less loaded_scripts.log | grep critical-stack
  /opt/critical-stack/frameworks/intel/__load__.bro
    /opt/critical-stack/frameworks/intel/feeds.bro

If you'd like please feel free to open a support ticket and we can help you
figure this out offline:
https://criticalstack.zendesk.com/hc/en-us/requests/new

V/r,

Liam Randall







On Mon, Jun 29, 2015 at 5:12 PM, Andrew Ratcliffe <
andrew.ratcliffe at nswcsystems.co.uk> wrote:

> Josh,
> I tried a different one just so that it was current in the logs.
>
> cwihosting.com/emsp/data/getproductrequest.htm Intel::URL from
> http://www.phishtank.com/phish_detail.php?phish_id=2479331 via
> intel.criticalstack.com F
> [root at bro intel]# cd /usr/local/bro/logs/current/
> [root at bro current]# grep -l cwihosting.com *.log
> dns.log
> http.log
> [root at bro current]# grep cwihosting.com http.log
> 1435611906.514899 C31ZazNObk3xTTk86 172.31.254.179 51734 72.52.170.179 80
> 1 GET cwihosting.com /emsp/data/getproductrequest.htm - curl/7.37.1 0
> 18464 200 OK - - - (empty) - - - - - FdGgt336pWjZZn8MBa -
> [root at bro current]#
>
>
> Thanks
>
> Kind regards,
> Andy
> Andrew.Ratcliffe at NSWCSystems.co.uk
> CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
> Blog.InfoSecMatters.net <http://blog.infosecmatters.net/>
>
>
>
>
>
>
> On 29 Jun 2015, at 21:35, Josh Liburdi <liburdi.joshua at gmail.com> wrote:
>
> Andy,
>
> If you still have these log files (or can generate them again), can
> you share the line from http.log that contains the URL indicator?
>
> Thanks,
> Josh
>
> On Sun, Jun 28, 2015 at 6:02 PM, Andrew Ratcliffe
> <andrew.ratcliffe at nswcsystems.co.uk> wrote:
>
> Hi Josh,
> Thanks for pointing that out. However, I still seem to have a problem:
> www.etiksecimler.com/appraiser/ipad/ Intel::URL from
> http://www.phishtank.com/phish_detail.php?phish_id=3266591 via
> intel.criticalstack.com F
> Use Curl to get the URL
> Andys-MacBook-Air:~ andy$ curl www.etiksecimler.com/appraiser/ipad/
> Still no intel.log entry
> [root at bro current]# grep -l www.etiksecimler.com *.log
> dns.log
> http.log
>
> # Critical Stack, Inc - https://intel.criticalstack.com
> @load /opt/critical-stack/frameworks/intel
> # Uncomment the following line to enable detection of the heartbleed
> attack.
> Enabling
> # this might impact performance a bit.
> # @load policy/protocols/ssl/heartbleed
> @load conn-geoip2.bro
> @load intel-2.bro
> #@load bpf-filter.bro
>
> Kind regards,
> Andy
> Andrew.Ratcliffe at NSWCSystems.co.uk
> CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
> Blog.InfoSecMatters.net
>
>
>
>
>
>
> On 27 Jun 2015, at 23:55, Josh Liburdi <liburdi.joshua at gmail.com> wrote:
>
> Andy,
>
> By default the Intel framework only generates log entries for IP addresses
> if the connection is a fully established TCP connection. That's probably
> why
> pinging an IP did not generate an entry.
>
> Josh
>
> On Saturday, Jun 27, 2015 at 5:39 PM, Andrew Ratcliffe
> <andrew.ratcliffe at nswcsystems.co.uk>, wrote:
>
>
> Hi,
> I tried using criticalstack, as it sounds like a really cool idea. I just
> can?t seem to get any events from it.
>
> Should events go to the notice.log or the intel.log?
>
> I tried a ping from an address present in the feed then looked for output
> and I have conn.log ICMP entry and a syslog entry but nothing else.
> Andys-MacBook-Air:~ andy$ ping 89.106.121.76
>
> [root at bro current]# grep -l '89.106.121.76' *.log
> conn.log
> syslog.log
>
> 1435439487.024865 C6HBUkZ7i07zlYE5a 172.31.254.179 8 89.106.121.76 0 icmp
> - 9.123324 560 560 OTH T 0 - 1840 10 840 (empty) - BG - - 22.872499
> 43.990002
>
> I have some Intel loaded from CIF2 and that works OK, I use the test
> event:
> Andys-MacBook-Air:~ andy$ curl http://testmyids.com
> uid=0(root) gid=0(root) groups=0(root)
> intel.log
> 1435439895.054961 CaEWz015AEjRJRruN2 172.31.254.179 55025 172.31.254.80 53
> - - - testmyids.com Intel::DOMAIN DNS::IN_REQUEST Tester
> 1435439895.054965 COdqds1DkdarGlSnY1 172.31.254.179 53210 172.31.254.80 53
> - - - testmyids.com Intel::DOMAIN DNS::IN_REQUEST Tester
> 1435439895.055305 CLcqwd2xLkH0MUUtf3 172.31.254.80 50910 8.8.4.4 53 - - -
> testmyids.com Intel::DOMAIN DNS::IN_REQUEST Tester
> 1435439895.055309 Cwdyhm1vbT1SnTiSG1 172.31.254.80 50639 8.8.4.4 53 - - -
> testmyids.com Intel::DOMAIN DNS::IN_REQUEST Tester
> 1435439895.253858 CtMoHr3h546C8UmdSi 172.31.254.179 50214 82.165.177.154
> 80 - - - testmyids.com Intel::DOMAIN HTTP::IN_HOST_HEADER Tester
>
> Am I doing something wrong?
>
> Kind regards,
> Andy
> Andrew.Ratcliffe at NSWCSystems.co.uk
> CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
> Blog.InfoSecMatters.net
>
>
>
>
>
>
> On 25 Jun 2015, at 13:51, Liam Randall <liam.randall at gmail.com> wrote:
>
> No Critical Stack is entirely custom; we are not building a TIP.  We
> wanted to have an easy way to have actionable into stream into bro as it is
> to discovered so we built it.  We thought others would want it as well so
> we
> make it freely available to the community.  We are getting ready to launch
> a
> new extension to it called KITTY- Keep Intel Transactions To Yourself that
> allow you to privately share and deploy 100's of Millions of indicators in
> a
> fast memory efficient way.  It integrates directly with our online
> marketplace- we deployed our first test clients this week.  We'll announce
> more shortly @CriticalStack .
>
> For TIPs there are a lot of great solutions you should look at:
>
> Free:
> MISP
> CRITS
>
> Commercial:
> Soltra Edge (has a free version)
> ThreatConnect
> ThreatStream
> ThreatQ (ThreatQuotient)
> BrightPoint Security (formerly Vorstack)
>
>
> V/r,
>
> Liam Randall
>
>
> On Thu, Jun 25, 2015 at 8:37 AM, Harry Hoffman <hhoffman at ip-solutions.net>
> wrote:
>
>
> Is critical stack based upon CIF (collective intelligence framework)?
>
> It looks very similar.
>
> Cheers,
> Harry
>
>
> On Jun 25, 2015 7:44 AM, Heine Lysemose <lysemose at gmail.com> wrote:
>
>
> Hi
>
> I encourage you to have a look at, https://intel.criticalstack.com/
>
> Best,
> Lysemose
>
> On Thu, Jun 25, 2015 at 1:31 PM, Jan Grashofer <jan.grashofer at cern.ch>
> wrote:
>
>
> Hi all,
>
> I am having a look at Threat Intelligence Management solutions, which
> can be used with Bro. What do you use and what are your experiences?
>
> Regards,
> Jan
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150629/cc43e0ba/attachment.html 

From andrew.ratcliffe at nswcsystems.co.uk  Mon Jun 29 15:08:04 2015
From: andrew.ratcliffe at nswcsystems.co.uk (Andrew Ratcliffe)
Date: Mon, 29 Jun 2015 22:08:04 +0000
Subject: [Bro] Threat Intelligence Management
In-Reply-To: <CAC8+5ePK9W9yb0DGhW_2bXyq1JnWFcdkAAemNTpwypbGaMKHvg@mail.gmail.com>
References: <FA19949E-1476-449F-A1F8-6AB1415E4182@nswcsystems.co.uk>
	<1435445717505.28d0ca5f@Nodemailer>
	<B01CE95A-B0AA-4058-9EA7-30BE1B674E0D@nswcsystems.co.uk>
	<CANrCRiFpCNCivGOEXVBffdqW=HQ6GsoruNGPmr75PpucjvQL8w@mail.gmail.com>
	<26BC6390-DCA5-46D7-972A-D155D041F0B8@nswcsystems.co.uk>
	<CAC8+5ePK9W9yb0DGhW_2bXyq1JnWFcdkAAemNTpwypbGaMKHvg@mail.gmail.com>
Message-ID: <CFFBADA6-2899-447B-AA56-254014B39FED@nswcsystems.co.uk>

Liam,
Thanks for that. I think it is not loading. I?ll have another look at it.
Kind regards,
Andy
Andrew.Ratcliffe at NSWCSystems.co.uk
CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
Blog.InfoSecMatters.net <http://blog.infosecmatters.net/>






> On 29 Jun 2015, at 22:21, Liam Randall <liam.randall at gmail.com> wrote:
> 
> Hey Andrew,
> 
> After installing did you do a
> 
> sudo broctl check
> sudo broctl install
> sudo broctl restart
> 
> You only need to perform that once and the future updates will be included automatically.
> 
> If you have included  'load misc/loaded-scripts' in your local.bro you will generate a loaded_scripts.log that you can use to verify that the scripts are running:
> 
> less loaded_scripts.log | grep critical-stack
>   /opt/critical-stack/frameworks/intel/__load__.bro
>     /opt/critical-stack/frameworks/intel/feeds.bro
> 
> If you'd like please feel free to open a support ticket and we can help you figure this out offline:
> https://criticalstack.zendesk.com/hc/en-us/requests/new <https://criticalstack.zendesk.com/hc/en-us/requests/new>
> 
> V/r,
> 
> Liam Randall
> 
> 
> 
> 
> 
> 
> 
> On Mon, Jun 29, 2015 at 5:12 PM, Andrew Ratcliffe <andrew.ratcliffe at nswcsystems.co.uk <mailto:andrew.ratcliffe at nswcsystems.co.uk>> wrote:
> Josh,
> I tried a different one just so that it was current in the logs.
> 
> cwihosting.com/emsp/data/getproductrequest.htm <http://cwihosting.com/emsp/data/getproductrequest.htm>	Intel::URL	from http://www.phishtank.com/phish_detail.php?phish_id=2479331 <http://www.phishtank.com/phish_detail.php?phish_id=2479331> via intel.criticalstack.com <http://intel.criticalstack.com/>	F
> [root at bro intel]# cd /usr/local/bro/logs/current/
> [root at bro current]# grep -l cwihosting.com <http://cwihosting.com/> *.log
> dns.log
> http.log
> [root at bro current]# grep cwihosting.com <http://cwihosting.com/> http.log
> 1435611906.514899	C31ZazNObk3xTTk86	172.31.254.179	51734	72.52.170.179	80	1	GET	cwihosting.com <http://cwihosting.com/>	/emsp/data/getproductrequest.htm	-	curl/7.37.1	0	18464	200	OK	-	-	-	(empty)	-	-	-	-	-	FdGgt336pWjZZn8MBa	-
> [root at bro current]#
> 
> 
> Thanks
> 
> Kind regards,
> Andy
> Andrew.Ratcliffe at NSWCSystems.co.uk <mailto:Andrew.Ratcliffe at NSWCSystems.co.uk>
> CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
> Blog.InfoSecMatters.net <http://blog.infosecmatters.net/>
> 
> 
> 
> 
> 
> 
>> On 29 Jun 2015, at 21:35, Josh Liburdi <liburdi.joshua at gmail.com <mailto:liburdi.joshua at gmail.com>> wrote:
>> 
>> Andy,
>> 
>> If you still have these log files (or can generate them again), can
>> you share the line from http.log that contains the URL indicator?
>> 
>> Thanks,
>> Josh
>> 
>> On Sun, Jun 28, 2015 at 6:02 PM, Andrew Ratcliffe
>> <andrew.ratcliffe at nswcsystems.co.uk <mailto:andrew.ratcliffe at nswcsystems.co.uk>> wrote:
>>> Hi Josh,
>>> Thanks for pointing that out. However, I still seem to have a problem:
>>> www.etiksecimler.com/appraiser/ipad/ <http://www.etiksecimler.com/appraiser/ipad/> Intel::URL from
>>> http://www.phishtank.com/phish_detail.php?phish_id=3266591 <http://www.phishtank.com/phish_detail.php?phish_id=3266591> via
>>> intel.criticalstack.com <http://intel.criticalstack.com/> F
>>> Use Curl to get the URL
>>> Andys-MacBook-Air:~ andy$ curl www.etiksecimler.com/appraiser/ipad/ <http://www.etiksecimler.com/appraiser/ipad/>
>>> Still no intel.log entry
>>> [root at bro current]# grep -l www.etiksecimler.com <http://www.etiksecimler.com/> *.log
>>> dns.log
>>> http.log
>>> 
>>> # Critical Stack, Inc - https://intel.criticalstack.com <https://intel.criticalstack.com/>
>>> @load /opt/critical-stack/frameworks/intel
>>> # Uncomment the following line to enable detection of the heartbleed attack.
>>> Enabling
>>> # this might impact performance a bit.
>>> # @load policy/protocols/ssl/heartbleed
>>> @load conn-geoip2.bro
>>> @load intel-2.bro
>>> #@load bpf-filter.bro
>>> 
>>> Kind regards,
>>> Andy
>>> Andrew.Ratcliffe at NSWCSystems.co.uk <mailto:Andrew.Ratcliffe at NSWCSystems.co.uk>
>>> CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
>>> Blog.InfoSecMatters.net <http://blog.infosecmatters.net/>
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On 27 Jun 2015, at 23:55, Josh Liburdi <liburdi.joshua at gmail.com <mailto:liburdi.joshua at gmail.com>> wrote:
>>> 
>>> Andy,
>>> 
>>> By default the Intel framework only generates log entries for IP addresses
>>> if the connection is a fully established TCP connection. That's probably why
>>> pinging an IP did not generate an entry.
>>> 
>>> Josh
>>> 
>>> On Saturday, Jun 27, 2015 at 5:39 PM, Andrew Ratcliffe
>>> <andrew.ratcliffe at nswcsystems.co.uk <mailto:andrew.ratcliffe at nswcsystems.co.uk>>, wrote:
>>>> 
>>>> Hi,
>>>> I tried using criticalstack, as it sounds like a really cool idea. I just
>>>> can?t seem to get any events from it.
>>>> 
>>>> Should events go to the notice.log or the intel.log?
>>>> 
>>>> I tried a ping from an address present in the feed then looked for output
>>>> and I have conn.log ICMP entry and a syslog entry but nothing else.
>>>> Andys-MacBook-Air:~ andy$ ping 89.106.121.76
>>>> 
>>>> [root at bro current]# grep -l '89.106.121.76' *.log
>>>> conn.log
>>>> syslog.log
>>>> 
>>>> 1435439487.024865 C6HBUkZ7i07zlYE5a 172.31.254.179 8 89.106.121.76 0 icmp
>>>> - 9.123324 560 560 OTH T 0 - 1840 10 840 (empty) - BG - - 22.872499
>>>> 43.990002
>>>> 
>>>> I have some Intel loaded from CIF2 and that works OK, I use the test
>>>> event:
>>>> Andys-MacBook-Air:~ andy$ curl http://testmyids.com <http://testmyids.com/>
>>>> uid=0(root) gid=0(root) groups=0(root)
>>>> intel.log
>>>> 1435439895.054961 CaEWz015AEjRJRruN2 172.31.254.179 55025 172.31.254.80 53
>>>> - - - testmyids.com <http://testmyids.com/> Intel::DOMAIN DNS::IN_REQUEST Tester
>>>> 1435439895.054965 COdqds1DkdarGlSnY1 172.31.254.179 53210 172.31.254.80 53
>>>> - - - testmyids.com <http://testmyids.com/> Intel::DOMAIN DNS::IN_REQUEST Tester
>>>> 1435439895.055305 CLcqwd2xLkH0MUUtf3 172.31.254.80 50910 8.8.4.4 53 - - -
>>>> testmyids.com <http://testmyids.com/> Intel::DOMAIN DNS::IN_REQUEST Tester
>>>> 1435439895.055309 Cwdyhm1vbT1SnTiSG1 172.31.254.80 50639 8.8.4.4 53 - - -
>>>> testmyids.com <http://testmyids.com/> Intel::DOMAIN DNS::IN_REQUEST Tester
>>>> 1435439895.253858 CtMoHr3h546C8UmdSi 172.31.254.179 50214 82.165.177.154
>>>> 80 - - - testmyids.com <http://testmyids.com/> Intel::DOMAIN HTTP::IN_HOST_HEADER Tester
>>>> 
>>>> Am I doing something wrong?
>>>> 
>>>> Kind regards,
>>>> Andy
>>>> Andrew.Ratcliffe at NSWCSystems.co.uk <mailto:Andrew.Ratcliffe at NSWCSystems.co.uk>
>>>> CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
>>>> Blog.InfoSecMatters.net <http://blog.infosecmatters.net/>
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On 25 Jun 2015, at 13:51, Liam Randall <liam.randall at gmail.com <mailto:liam.randall at gmail.com>> wrote:
>>>> 
>>>> No Critical Stack is entirely custom; we are not building a TIP.  We
>>>> wanted to have an easy way to have actionable into stream into bro as it is
>>>> to discovered so we built it.  We thought others would want it as well so we
>>>> make it freely available to the community.  We are getting ready to launch a
>>>> new extension to it called KITTY- Keep Intel Transactions To Yourself that
>>>> allow you to privately share and deploy 100's of Millions of indicators in a
>>>> fast memory efficient way.  It integrates directly with our online
>>>> marketplace- we deployed our first test clients this week.  We'll announce
>>>> more shortly @CriticalStack .
>>>> 
>>>> For TIPs there are a lot of great solutions you should look at:
>>>> 
>>>> Free:
>>>> MISP
>>>> CRITS
>>>> 
>>>> Commercial:
>>>> Soltra Edge (has a free version)
>>>> ThreatConnect
>>>> ThreatStream
>>>> ThreatQ (ThreatQuotient)
>>>> BrightPoint Security (formerly Vorstack)
>>>> 
>>>> 
>>>> V/r,
>>>> 
>>>> Liam Randall
>>>> 
>>>> 
>>>> On Thu, Jun 25, 2015 at 8:37 AM, Harry Hoffman <hhoffman at ip-solutions.net <mailto:hhoffman at ip-solutions.net>>
>>>> wrote:
>>>>> 
>>>>> Is critical stack based upon CIF (collective intelligence framework)?
>>>>> 
>>>>> It looks very similar.
>>>>> 
>>>>> Cheers,
>>>>> Harry
>>>>> 
>>>>> 
>>>>> On Jun 25, 2015 7:44 AM, Heine Lysemose <lysemose at gmail.com <mailto:lysemose at gmail.com>> wrote:
>>>>>> 
>>>>>> Hi
>>>>>> 
>>>>>> I encourage you to have a look at, https://intel.criticalstack.com/ <https://intel.criticalstack.com/>
>>>>>> 
>>>>>> Best,
>>>>>> Lysemose
>>>>>> 
>>>>>> On Thu, Jun 25, 2015 at 1:31 PM, Jan Grashofer <jan.grashofer at cern.ch <mailto:jan.grashofer at cern.ch>>
>>>>>> wrote:
>>>>>>> 
>>>>>>> Hi all,
>>>>>>> 
>>>>>>> I am having a look at Threat Intelligence Management solutions, which
>>>>>>> can be used with Bro. What do you use and what are your experiences?
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Jan
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> Bro mailing list
>>>>>>> bro at bro-ids.org <mailto:bro at bro-ids.org>
>>>>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>>>>>> 
>>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Bro mailing list
>>>>> bro at bro-ids.org <mailto:bro at bro-ids.org>
>>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Bro mailing list
>>>> bro at bro-ids.org <mailto:bro at bro-ids.org>
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>>>> 
>>>> 
>>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150629/ca72362a/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 235 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150629/ca72362a/attachment-0001.bin 

From baxter.milliwew at gmail.com  Mon Jun 29 16:03:43 2015
From: baxter.milliwew at gmail.com (Baxter Milliwew)
Date: Mon, 29 Jun 2015 16:03:43 -0700
Subject: [Bro] Bro's limitations with high worker count and memory
	exhaustion
In-Reply-To: <CA+J8G6HRjn4cXb+mjBw-XdJaBHOG-MgdwtKbM0kWKE2B8UPQuQ@mail.gmail.com>
References: <CA+J8G6FgNij+Tu9Kk8jQNe=1kAs0z7zf6D_5-ATB8VZE2nmY4A@mail.gmail.com>
	<E3142ED5D6744B4C8E17323B862FEDE5DF6A8B@CERNXCHG62.cern.ch>
	<CA+J8G6HRjn4cXb+mjBw-XdJaBHOG-MgdwtKbM0kWKE2B8UPQuQ@mail.gmail.com>
Message-ID: <CA+J8G6H+vx8skqR-Q9+qZK7=jQ3--60_6RsqbFtj2ghd9BvxDw@mail.gmail.com>

Switching to jemalloc fixed the stability issue but not the worker count
limitation.

On Sun, Jun 28, 2015 at 7:18 PM, Baxter Milliwew <baxter.milliwew at gmail.com>
wrote:

> Looks like malloc from glibc, default on Ubuntu.  I will try jemalloc and
> others.
>
>
>
> On Sun, Jun 28, 2015 at 1:03 AM, Jan Grashofer <jan.grashofer at cern.ch>
> wrote:
>
>>  I experienced similar problems (memory gets eaten up quickly and
>> workers crash with segfault) using tcmalloc. Which malloc do you use?
>>
>>
>>
>> Regards,
>>
>> Jan
>>
>>
>>  ------------------------------
>> *From:* bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Baxter
>> Milliwew [baxter.milliwew at gmail.com]
>> *Sent:* Friday, June 26, 2015 23:03
>> *To:* bro at bro.org
>> *Subject:* [Bro] Bro's limitations with high worker count and memory
>> exhaustion
>>
>>   There's some sort of association between memory exhaustion and a high
>> number of workers.  The poor man's fix would be to purchase new servers
>> with higher CPU speeds as that would reduce the worker count.  Issues with
>> high worker count and/or memory exhaustion appears to be a well know
>> problem based on the mailing list archives.
>>
>>  In the current version of bro-2.4 my previous configuration immediately
>> causes the manager to crash: 15 proxies, 155 workers.  To resolve this I've
>> lowered the count to 10 proxies and 140 workers.  However even with this
>> configuration the manager process will exhaust all memory and crash within
>> about 2 hours.
>>
>>  The manager is threaded; I think this is an issue with the threading
>> behavior between manager, proxies, and workers.  Debugging threading
>> problems is complex and I'm a complete novice.. my current tutorial is
>> using information from a stack overflow thread:
>>
>>
>> http://stackoverflow.com/questions/981011/c-programming-debugging-with-pthreads
>>
>>  Does anyone else have this problem ?  What have you tried and what do
>> you suggest ?
>>
>>  Thanks
>>
>>
>>
>>
>>   1435347409.458185       worker-2-18     parent  -       -       -
>>   info    [#10000/10.1.1.1:36994] peer sent class "control"
>>
>> 1435347409.458185       worker-2-18     parent  -       -       -
>> info    [#10000/10.1.1.1:36994] phase: handshake
>>
>> 1435347409.661085       worker-2-18     parent  -       -       -
>> info    [#10000/10.1.1.1:36994] request for unknown event save_results
>>
>> 1435347409.661085       worker-2-18     parent  -       -       -
>> info    [#10000/10.1.1.1:36994] registered for event
>> Control::peer_status_response
>>
>> 1435347409.694858       worker-2-18     parent  -       -       -
>> info    [#10000/10.1.1.1:36994] peer does not support 64bit PIDs; using
>> compatibility mode
>>
>> 1435347409.694858       worker-2-18     parent  -       -       -
>> info    [#10000/10.1.1.1:36994] peer is a Broccoli
>>
>> 1435347409.694858       worker-2-18     parent  -       -       -
>> info    [#10000/10.1.1.1:36994] phase: running
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150629/9fdafe60/attachment.html 

From baxter.milliwew at gmail.com  Mon Jun 29 16:09:41 2015
From: baxter.milliwew at gmail.com (Baxter Milliwew)
Date: Mon, 29 Jun 2015 16:09:41 -0700
Subject: [Bro] Bro's limitations with high worker count and memory
	exhaustion
In-Reply-To: <CA+J8G6H+vx8skqR-Q9+qZK7=jQ3--60_6RsqbFtj2ghd9BvxDw@mail.gmail.com>
References: <CA+J8G6FgNij+Tu9Kk8jQNe=1kAs0z7zf6D_5-ATB8VZE2nmY4A@mail.gmail.com>
	<E3142ED5D6744B4C8E17323B862FEDE5DF6A8B@CERNXCHG62.cern.ch>
	<CA+J8G6HRjn4cXb+mjBw-XdJaBHOG-MgdwtKbM0kWKE2B8UPQuQ@mail.gmail.com>
	<CA+J8G6H+vx8skqR-Q9+qZK7=jQ3--60_6RsqbFtj2ghd9BvxDw@mail.gmail.com>
Message-ID: <CA+J8G6ENrX=DoB10XuPnykHZB7Dc2hCuePah5vZcebStYvLbCQ@mail.gmail.com>

Nevermind... new box, default nofile limits.  Thanks for the malloc tip.


On Mon, Jun 29, 2015 at 4:03 PM, Baxter Milliwew <baxter.milliwew at gmail.com>
wrote:

> Switching to jemalloc fixed the stability issue but not the worker count
> limitation.
>
> On Sun, Jun 28, 2015 at 7:18 PM, Baxter Milliwew <
> baxter.milliwew at gmail.com> wrote:
>
>> Looks like malloc from glibc, default on Ubuntu.  I will try jemalloc and
>> others.
>>
>>
>>
>> On Sun, Jun 28, 2015 at 1:03 AM, Jan Grashofer <jan.grashofer at cern.ch>
>> wrote:
>>
>>>  I experienced similar problems (memory gets eaten up quickly and
>>> workers crash with segfault) using tcmalloc. Which malloc do you use?
>>>
>>>
>>>
>>> Regards,
>>>
>>> Jan
>>>
>>>
>>>  ------------------------------
>>> *From:* bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Baxter
>>> Milliwew [baxter.milliwew at gmail.com]
>>> *Sent:* Friday, June 26, 2015 23:03
>>> *To:* bro at bro.org
>>> *Subject:* [Bro] Bro's limitations with high worker count and memory
>>> exhaustion
>>>
>>>   There's some sort of association between memory exhaustion and a high
>>> number of workers.  The poor man's fix would be to purchase new servers
>>> with higher CPU speeds as that would reduce the worker count.  Issues with
>>> high worker count and/or memory exhaustion appears to be a well know
>>> problem based on the mailing list archives.
>>>
>>>  In the current version of bro-2.4 my previous configuration
>>> immediately causes the manager to crash: 15 proxies, 155 workers.  To
>>> resolve this I've lowered the count to 10 proxies and 140 workers.  However
>>> even with this configuration the manager process will exhaust all memory
>>> and crash within about 2 hours.
>>>
>>>  The manager is threaded; I think this is an issue with the threading
>>> behavior between manager, proxies, and workers.  Debugging threading
>>> problems is complex and I'm a complete novice.. my current tutorial is
>>> using information from a stack overflow thread:
>>>
>>>
>>> http://stackoverflow.com/questions/981011/c-programming-debugging-with-pthreads
>>>
>>>  Does anyone else have this problem ?  What have you tried and what do
>>> you suggest ?
>>>
>>>  Thanks
>>>
>>>
>>>
>>>
>>>   1435347409.458185       worker-2-18     parent  -       -       -
>>>   info    [#10000/10.1.1.1:36994] peer sent class "control"
>>>
>>> 1435347409.458185       worker-2-18     parent  -       -       -
>>> info    [#10000/10.1.1.1:36994] phase: handshake
>>>
>>> 1435347409.661085       worker-2-18     parent  -       -       -
>>> info    [#10000/10.1.1.1:36994] request for unknown event save_results
>>>
>>> 1435347409.661085       worker-2-18     parent  -       -       -
>>> info    [#10000/10.1.1.1:36994] registered for event
>>> Control::peer_status_response
>>>
>>> 1435347409.694858       worker-2-18     parent  -       -       -
>>> info    [#10000/10.1.1.1:36994] peer does not support 64bit PIDs; using
>>> compatibility mode
>>>
>>> 1435347409.694858       worker-2-18     parent  -       -       -
>>> info    [#10000/10.1.1.1:36994] peer is a Broccoli
>>>
>>> 1435347409.694858       worker-2-18     parent  -       -       -
>>> info    [#10000/10.1.1.1:36994] phase: running
>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150629/7b7e833e/attachment.html 

From baxter.milliwew at gmail.com  Mon Jun 29 16:22:13 2015
From: baxter.milliwew at gmail.com (Baxter Milliwew)
Date: Mon, 29 Jun 2015 16:22:13 -0700
Subject: [Bro] Bro's limitations with high worker count and memory
	exhaustion
In-Reply-To: <CA+J8G6ENrX=DoB10XuPnykHZB7Dc2hCuePah5vZcebStYvLbCQ@mail.gmail.com>
References: <CA+J8G6FgNij+Tu9Kk8jQNe=1kAs0z7zf6D_5-ATB8VZE2nmY4A@mail.gmail.com>
	<E3142ED5D6744B4C8E17323B862FEDE5DF6A8B@CERNXCHG62.cern.ch>
	<CA+J8G6HRjn4cXb+mjBw-XdJaBHOG-MgdwtKbM0kWKE2B8UPQuQ@mail.gmail.com>
	<CA+J8G6H+vx8skqR-Q9+qZK7=jQ3--60_6RsqbFtj2ghd9BvxDw@mail.gmail.com>
	<CA+J8G6ENrX=DoB10XuPnykHZB7Dc2hCuePah5vZcebStYvLbCQ@mail.gmail.com>
Message-ID: <CA+J8G6Fht9Kb4OBrYxPxrj9xzVzHBy_fV93=JVJpF7vMPpOvEA@mail.gmail.com>

The manager still crashes.  Interesting note about a buffer overflow.


[manager]


Bro 2.4

Linux 3.16.0-38-generic


core

[New LWP 18834]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Core was generated by `/usr/local/3rd-party/bro/bin/bro -U .status -p
broctl -p broctl-live -p local -'.

Program terminated with signal SIGABRT, Aborted.

#0  0x00007f163bb46cc9 in __GI_raise (sig=sig at entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56


Thread 1 (Thread 0x............ (LWP 18834)):

#0  0x00007f163bb46cc9 in __GI_raise (sig=sig at entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56

#1  0x00007f163bb4a0d8 in __GI_abort () at abort.c:89

#2  0x00007f163bb83394 in __libc_message (do_abort=do_abort at entry=2,
fmt=fmt at entry=0x............ "*** %s ***: %s terminated\n") at
../sysdeps/posix/libc_fatal.c:175

#3  0x00007f163bc1ac9c in __GI___fortify_fail (msg=<optimized out>,
msg at entry=0x............ "buffer overflow detected") at fortify_fail.c:37

#4  0x00007f163bc19b60 in __GI___chk_fail () at chk_fail.c:28

#5  0x00007f163bc1abe7 in __fdelt_chk (d=<optimized out>) at fdelt_chk.c:25

#6  0x00000000005e962a in Set (set=0x............, this=0x............) at
/home/bro/Bro-IDS/bro-2.4/src/iosource/FD_Set.h:59

#7  SocketComm::Run (this=0x............) at
/home/bro/Bro-IDS/bro-2.4/src/RemoteSerializer.cc:3406

#8  0x00000000005e9c31 in RemoteSerializer::Fork (this=0x............) at
/home/bro/Bro-IDS/bro-2.4/src/RemoteSerializer.cc:687

#9  0x00000000005e9d4f in RemoteSerializer::Enable (this=0x............) at
/home/bro/Bro-IDS/bro-2.4/src/RemoteSerializer.cc:575

#10 0x00000000005b6943 in BifFunc::bro_enable_communication
(frame=<optimized out>, BiF_ARGS=<optimized out>) at bro.bif:4480

#11 0x00000000005b431d in BuiltinFunc::Call (this=0x............,
args=0x............, parent=0x............) at
/home/bro/Bro-IDS/bro-2.4/src/Func.cc:586

#12 0x0000000000599066 in CallExpr::Eval (this=0x............,
f=0x............) at /home/bro/Bro-IDS/bro-2.4/src/Expr.cc:4544

#13 0x000000000060ceb4 in ExprStmt::Exec (this=0x............,
f=0x............, flow=@0x............: FLOW_NEXT) at
/home/bro/Bro-IDS/bro-2.4/src/Stmt.cc:352

#14 0x000000000060b174 in IfStmt::DoExec (this=0x............,
f=0x............, v=<optimized out>, flow=@0x............: FLOW_NEXT) at
/home/bro/Bro-IDS/bro-2.4/src/Stmt.cc:456

#15 0x000000000060ced1 in ExprStmt::Exec (this=0x............,
f=0x............, flow=@0x............: FLOW_NEXT) at
/home/bro/Bro-IDS/bro-2.4/src/Stmt.cc:356

#16 0x000000000060b211 in StmtList::Exec (this=0x............,
f=0x............, flow=@0x............: FLOW_NEXT) at
/home/bro/Bro-IDS/bro-2.4/src/Stmt.cc:1696

#17 0x000000000060b211 in StmtList::Exec (this=0x............,
f=0x............, flow=@0x............: FLOW_NEXT) at
/home/bro/Bro-IDS/bro-2.4/src/Stmt.cc:1696

#18 0x00000000005c042e in BroFunc::Call (this=0x............,
args=<optimized out>, parent=0x0) at
/home/bro/Bro-IDS/bro-2.4/src/Func.cc:403

#19 0x000000000057ee2a in EventHandler::Call (this=0x............,
vl=0x............, no_remote=no_remote at entry=false) at
/home/bro/Bro-IDS/bro-2.4/src/EventHandler.cc:130

#20 0x000000000057e035 in Dispatch (no_remote=false, this=0x............)
at /home/bro/Bro-IDS/bro-2.4/src/Event.h:50

#21 EventMgr::Dispatch (this=this at entry=0x...... <mgr>) at
/home/bro/Bro-IDS/bro-2.4/src/Event.cc:111

#22 0x000000000057e1d0 in EventMgr::Drain (this=0xbbd720 <mgr>) at
/home/bro/Bro-IDS/bro-2.4/src/Event.cc:128

#23 0x00000000005300ed in main (argc=<optimized out>, argv=<optimized out>)
at /home/bro/Bro-IDS/bro-2.4/src/main.cc:1147




On Mon, Jun 29, 2015 at 4:09 PM, Baxter Milliwew <baxter.milliwew at gmail.com>
wrote:

> Nevermind... new box, default nofile limits.  Thanks for the malloc tip.
>
>
> On Mon, Jun 29, 2015 at 4:03 PM, Baxter Milliwew <
> baxter.milliwew at gmail.com> wrote:
>
>> Switching to jemalloc fixed the stability issue but not the worker count
>> limitation.
>>
>> On Sun, Jun 28, 2015 at 7:18 PM, Baxter Milliwew <
>> baxter.milliwew at gmail.com> wrote:
>>
>>> Looks like malloc from glibc, default on Ubuntu.  I will try jemalloc
>>> and others.
>>>
>>>
>>>
>>> On Sun, Jun 28, 2015 at 1:03 AM, Jan Grashofer <jan.grashofer at cern.ch>
>>> wrote:
>>>
>>>>  I experienced similar problems (memory gets eaten up quickly and
>>>> workers crash with segfault) using tcmalloc. Which malloc do you use?
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Jan
>>>>
>>>>
>>>>  ------------------------------
>>>> *From:* bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Baxter
>>>> Milliwew [baxter.milliwew at gmail.com]
>>>> *Sent:* Friday, June 26, 2015 23:03
>>>> *To:* bro at bro.org
>>>> *Subject:* [Bro] Bro's limitations with high worker count and memory
>>>> exhaustion
>>>>
>>>>   There's some sort of association between memory exhaustion and a
>>>> high number of workers.  The poor man's fix would be to purchase new
>>>> servers with higher CPU speeds as that would reduce the worker count.
>>>> Issues with high worker count and/or memory exhaustion appears to be a well
>>>> know problem based on the mailing list archives.
>>>>
>>>>  In the current version of bro-2.4 my previous configuration
>>>> immediately causes the manager to crash: 15 proxies, 155 workers.  To
>>>> resolve this I've lowered the count to 10 proxies and 140 workers.  However
>>>> even with this configuration the manager process will exhaust all memory
>>>> and crash within about 2 hours.
>>>>
>>>>  The manager is threaded; I think this is an issue with the threading
>>>> behavior between manager, proxies, and workers.  Debugging threading
>>>> problems is complex and I'm a complete novice.. my current tutorial is
>>>> using information from a stack overflow thread:
>>>>
>>>>
>>>> http://stackoverflow.com/questions/981011/c-programming-debugging-with-pthreads
>>>>
>>>>  Does anyone else have this problem ?  What have you tried and what do
>>>> you suggest ?
>>>>
>>>>  Thanks
>>>>
>>>>
>>>>
>>>>
>>>>   1435347409.458185       worker-2-18     parent  -       -       -
>>>>     info    [#10000/10.1.1.1:36994] peer sent class "control"
>>>>
>>>> 1435347409.458185       worker-2-18     parent  -       -       -
>>>> info    [#10000/10.1.1.1:36994] phase: handshake
>>>>
>>>> 1435347409.661085       worker-2-18     parent  -       -       -
>>>> info    [#10000/10.1.1.1:36994] request for unknown event save_results
>>>>
>>>> 1435347409.661085       worker-2-18     parent  -       -       -
>>>> info    [#10000/10.1.1.1:36994] registered for event
>>>> Control::peer_status_response
>>>>
>>>> 1435347409.694858       worker-2-18     parent  -       -       -
>>>> info    [#10000/10.1.1.1:36994] peer does not support 64bit PIDs;
>>>> using compatibility mode
>>>>
>>>> 1435347409.694858       worker-2-18     parent  -       -       -
>>>> info    [#10000/10.1.1.1:36994] peer is a Broccoli
>>>>
>>>> 1435347409.694858       worker-2-18     parent  -       -       -
>>>> info    [#10000/10.1.1.1:36994] phase: running
>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150629/16ebfc31/attachment-0001.html 

From pkelley at hyperionavenue.com  Mon Jun 29 16:52:50 2015
From: pkelley at hyperionavenue.com (Patrick Kelley)
Date: Mon, 29 Jun 2015 16:52:50 -0700
Subject: [Bro] Threat Intelligence Management
In-Reply-To: <CFFBADA6-2899-447B-AA56-254014B39FED@nswcsystems.co.uk>
References: <FA19949E-1476-449F-A1F8-6AB1415E4182@nswcsystems.co.uk>
	<1435445717505.28d0ca5f@Nodemailer>
	<B01CE95A-B0AA-4058-9EA7-30BE1B674E0D@nswcsystems.co.uk>
	<CANrCRiFpCNCivGOEXVBffdqW=HQ6GsoruNGPmr75PpucjvQL8w@mail.gmail.com>
	<26BC6390-DCA5-46D7-972A-D155D041F0B8@nswcsystems.co.uk>
	<CAC8+5ePK9W9yb0DGhW_2bXyq1JnWFcdkAAemNTpwypbGaMKHvg@mail.gmail.com>
	<CFFBADA6-2899-447B-AA56-254014B39FED@nswcsystems.co.uk>
Message-ID: <CAN1NG8_zzgQcYc-1bxTvnwgbbyy0OEK0zwg57TSOsQTxOV6i3g@mail.gmail.com>

Any documentation available on exporting MISP into a BRO-friendly format?

On Mon, Jun 29, 2015 at 3:08 PM, Andrew Ratcliffe <
andrew.ratcliffe at nswcsystems.co.uk> wrote:

> Liam,
> Thanks for that. I think it is not loading. I?ll have another look at it.
> Kind regards,
> Andy
> Andrew.Ratcliffe at NSWCSystems.co.uk
> CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
> Blog.InfoSecMatters.net <http://blog.infosecmatters.net/>
>
>
>
>
>
>
> On 29 Jun 2015, at 22:21, Liam Randall <liam.randall at gmail.com> wrote:
>
> Hey Andrew,
>
> After installing did you do a
>
> sudo broctl check
> sudo broctl install
> sudo broctl restart
>
> You only need to perform that once and the future updates will be included
> automatically.
>
> If you have included  'load misc/loaded-scripts' in your local.bro you
> will generate a loaded_scripts.log that you can use to verify that the
> scripts are running:
>
> less loaded_scripts.log | grep critical-stack
>   /opt/critical-stack/frameworks/intel/__load__.bro
>     /opt/critical-stack/frameworks/intel/feeds.bro
>
> If you'd like please feel free to open a support ticket and we can help
> you figure this out offline:
> https://criticalstack.zendesk.com/hc/en-us/requests/new
>
> V/r,
>
> Liam Randall
>
>
>
>
>
>
>
> On Mon, Jun 29, 2015 at 5:12 PM, Andrew Ratcliffe <
> andrew.ratcliffe at nswcsystems.co.uk> wrote:
>
>> Josh,
>> I tried a different one just so that it was current in the logs.
>>
>> cwihosting.com/emsp/data/getproductrequest.htm Intel::URL from
>> http://www.phishtank.com/phish_detail.php?phish_id=2479331 via
>> intel.criticalstack.com F
>> [root at bro intel]# cd /usr/local/bro/logs/current/
>> [root at bro current]# grep -l cwihosting.com *.log
>> dns.log
>> http.log
>> [root at bro current]# grep cwihosting.com http.log
>> 1435611906.514899 C31ZazNObk3xTTk86 172.31.254.179 51734 72.52.170.179 80
>> 1 GET cwihosting.com /emsp/data/getproductrequest.htm - curl/7.37.1 0
>> 18464 200 OK - - - (empty) - - - - - FdGgt336pWjZZn8MBa -
>> [root at bro current]#
>>
>>
>> Thanks
>>
>> Kind regards,
>> Andy
>> Andrew.Ratcliffe at NSWCSystems.co.uk
>> CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
>> Blog.InfoSecMatters.net <http://blog.infosecmatters.net/>
>>
>>
>>
>>
>>
>>
>> On 29 Jun 2015, at 21:35, Josh Liburdi <liburdi.joshua at gmail.com> wrote:
>>
>> Andy,
>>
>> If you still have these log files (or can generate them again), can
>> you share the line from http.log that contains the URL indicator?
>>
>> Thanks,
>> Josh
>>
>> On Sun, Jun 28, 2015 at 6:02 PM, Andrew Ratcliffe
>> <andrew.ratcliffe at nswcsystems.co.uk> wrote:
>>
>> Hi Josh,
>> Thanks for pointing that out. However, I still seem to have a problem:
>> www.etiksecimler.com/appraiser/ipad/ Intel::URL from
>> http://www.phishtank.com/phish_detail.php?phish_id=3266591 via
>> intel.criticalstack.com F
>> Use Curl to get the URL
>> Andys-MacBook-Air:~ andy$ curl www.etiksecimler.com/appraiser/ipad/
>> Still no intel.log entry
>> [root at bro current]# grep -l www.etiksecimler.com *.log
>> dns.log
>> http.log
>>
>> # Critical Stack, Inc - https://intel.criticalstack.com
>> @load /opt/critical-stack/frameworks/intel
>> # Uncomment the following line to enable detection of the heartbleed
>> attack.
>> Enabling
>> # this might impact performance a bit.
>> # @load policy/protocols/ssl/heartbleed
>> @load conn-geoip2.bro
>> @load intel-2.bro
>> #@load bpf-filter.bro
>>
>> Kind regards,
>> Andy
>> Andrew.Ratcliffe at NSWCSystems.co.uk
>> CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
>> Blog.InfoSecMatters.net <http://blog.infosecmatters.net/>
>>
>>
>>
>>
>>
>>
>> On 27 Jun 2015, at 23:55, Josh Liburdi <liburdi.joshua at gmail.com> wrote:
>>
>> Andy,
>>
>> By default the Intel framework only generates log entries for IP addresses
>> if the connection is a fully established TCP connection. That's probably
>> why
>> pinging an IP did not generate an entry.
>>
>> Josh
>>
>> On Saturday, Jun 27, 2015 at 5:39 PM, Andrew Ratcliffe
>> <andrew.ratcliffe at nswcsystems.co.uk>, wrote:
>>
>>
>> Hi,
>> I tried using criticalstack, as it sounds like a really cool idea. I just
>> can?t seem to get any events from it.
>>
>> Should events go to the notice.log or the intel.log?
>>
>> I tried a ping from an address present in the feed then looked for output
>> and I have conn.log ICMP entry and a syslog entry but nothing else.
>> Andys-MacBook-Air:~ andy$ ping 89.106.121.76
>>
>> [root at bro current]# grep -l '89.106.121.76' *.log
>> conn.log
>> syslog.log
>>
>> 1435439487.024865 C6HBUkZ7i07zlYE5a 172.31.254.179 8 89.106.121.76 0 icmp
>> - 9.123324 560 560 OTH T 0 - 1840 10 840 (empty) - BG - - 22.872499
>> 43.990002
>>
>> I have some Intel loaded from CIF2 and that works OK, I use the test
>> event:
>> Andys-MacBook-Air:~ andy$ curl http://testmyids.com
>> uid=0(root) gid=0(root) groups=0(root)
>> intel.log
>> 1435439895.054961 CaEWz015AEjRJRruN2 172.31.254.179 55025 172.31.254.80 53
>> - - - testmyids.com Intel::DOMAIN DNS::IN_REQUEST Tester
>> 1435439895.054965 COdqds1DkdarGlSnY1 172.31.254.179 53210 172.31.254.80 53
>> - - - testmyids.com Intel::DOMAIN DNS::IN_REQUEST Tester
>> 1435439895.055305 CLcqwd2xLkH0MUUtf3 172.31.254.80 50910 8.8.4.4 53 - - -
>> testmyids.com Intel::DOMAIN DNS::IN_REQUEST Tester
>> 1435439895.055309 Cwdyhm1vbT1SnTiSG1 172.31.254.80 50639 8.8.4.4 53 - - -
>> testmyids.com Intel::DOMAIN DNS::IN_REQUEST Tester
>> 1435439895.253858 CtMoHr3h546C8UmdSi 172.31.254.179 50214 82.165.177.154
>> 80 - - - testmyids.com Intel::DOMAIN HTTP::IN_HOST_HEADER Tester
>>
>> Am I doing something wrong?
>>
>> Kind regards,
>> Andy
>> Andrew.Ratcliffe at NSWCSystems.co.uk
>> CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
>> Blog.InfoSecMatters.net <http://blog.infosecmatters.net/>
>>
>>
>>
>>
>>
>>
>> On 25 Jun 2015, at 13:51, Liam Randall <liam.randall at gmail.com> wrote:
>>
>> No Critical Stack is entirely custom; we are not building a TIP.  We
>> wanted to have an easy way to have actionable into stream into bro as it
>> is
>> to discovered so we built it.  We thought others would want it as well so
>> we
>> make it freely available to the community.  We are getting ready to
>> launch a
>> new extension to it called KITTY- Keep Intel Transactions To Yourself that
>> allow you to privately share and deploy 100's of Millions of indicators
>> in a
>> fast memory efficient way.  It integrates directly with our online
>> marketplace- we deployed our first test clients this week.  We'll announce
>> more shortly @CriticalStack .
>>
>> For TIPs there are a lot of great solutions you should look at:
>>
>> Free:
>> MISP
>> CRITS
>>
>> Commercial:
>> Soltra Edge (has a free version)
>> ThreatConnect
>> ThreatStream
>> ThreatQ (ThreatQuotient)
>> BrightPoint Security (formerly Vorstack)
>>
>>
>> V/r,
>>
>> Liam Randall
>>
>>
>> On Thu, Jun 25, 2015 at 8:37 AM, Harry Hoffman <hhoffman at ip-solutions.net
>> >
>> wrote:
>>
>>
>> Is critical stack based upon CIF (collective intelligence framework)?
>>
>> It looks very similar.
>>
>> Cheers,
>> Harry
>>
>>
>> On Jun 25, 2015 7:44 AM, Heine Lysemose <lysemose at gmail.com> wrote:
>>
>>
>> Hi
>>
>> I encourage you to have a look at, https://intel.criticalstack.com/
>>
>> Best,
>> Lysemose
>>
>> On Thu, Jun 25, 2015 at 1:31 PM, Jan Grashofer <jan.grashofer at cern.ch>
>> wrote:
>>
>>
>> Hi all,
>>
>> I am having a look at Threat Intelligence Management solutions, which
>> can be used with Bro. What do you use and what are your experiences?
>>
>> Regards,
>> Jan
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>>
>>
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>>
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>>
>>
>>
>>
>>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 

Patrick Kelley, CEH
Hyperion Avenue Labs
http://www.hyperionavenue.com
951.291.8310

*The limit to which you have accepted being comfortable is the limit to
which you have grown. Accept new challenges as an opportunity to enrich
yourself and not as a point of potential failure.*

[image: hal_logo]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150629/9b603554/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 12155 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150629/9b603554/attachment-0001.bin 

From albert.zaharovits at gmail.com  Tue Jun 30 00:52:57 2015
From: albert.zaharovits at gmail.com (Albert Zaharovits)
Date: Tue, 30 Jun 2015 10:52:57 +0300
Subject: [Bro] ASCII JSON log stream
Message-ID: <864C1DD0-46CF-4D2F-B520-0A0A3397DC6D@gmail.com>

Hello,

I am writing a bro script which creates a ASCII log stream. I would like JSON output only for this stream. I was able to turn on JSON output globally.

Any idea?

Albert

From jan.grashofer at cern.ch  Tue Jun 30 01:07:48 2015
From: jan.grashofer at cern.ch (Jan Grashofer)
Date: Tue, 30 Jun 2015 08:07:48 +0000
Subject: [Bro] ASCII JSON log stream
In-Reply-To: <864C1DD0-46CF-4D2F-B520-0A0A3397DC6D@gmail.com>
References: <864C1DD0-46CF-4D2F-B520-0A0A3397DC6D@gmail.com>
Message-ID: <E3142ED5D6744B4C8E17323B862FEDE5DF6E18@CERNXCHG62.cern.ch>

Hi Albert,

I have not tried this yet but regarding the documentation a filter may allow you to set JSON logging for a particular stream (see https://www.bro.org/sphinx-git/scripts/base/frameworks/logging/writers/ascii.bro.html).

Regards,
Jan
________________________________________
From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Albert Zaharovits [albert.zaharovits at gmail.com]
Sent: Tuesday, June 30, 2015 09:52
To: bro at bro.org
Subject: [Bro] ASCII JSON log stream

Hello,

I am writing a bro script which creates a ASCII log stream. I would like JSON output only for this stream. I was able to turn on JSON output globally.

Any idea?

Albert
_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

From jsiwek at illinois.edu  Tue Jun 30 07:44:23 2015
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Tue, 30 Jun 2015 14:44:23 +0000
Subject: [Bro] Bro's limitations with high worker count and
	memory	exhaustion
In-Reply-To: <CA+J8G6Fht9Kb4OBrYxPxrj9xzVzHBy_fV93=JVJpF7vMPpOvEA@mail.gmail.com>
References: <CA+J8G6FgNij+Tu9Kk8jQNe=1kAs0z7zf6D_5-ATB8VZE2nmY4A@mail.gmail.com>
	<E3142ED5D6744B4C8E17323B862FEDE5DF6A8B@CERNXCHG62.cern.ch>
	<CA+J8G6HRjn4cXb+mjBw-XdJaBHOG-MgdwtKbM0kWKE2B8UPQuQ@mail.gmail.com>
	<CA+J8G6H+vx8skqR-Q9+qZK7=jQ3--60_6RsqbFtj2ghd9BvxDw@mail.gmail.com>
	<CA+J8G6ENrX=DoB10XuPnykHZB7Dc2hCuePah5vZcebStYvLbCQ@mail.gmail.com>
	<CA+J8G6Fht9Kb4OBrYxPxrj9xzVzHBy_fV93=JVJpF7vMPpOvEA@mail.gmail.com>
Message-ID: <BE09FB89-69C7-42AA-8967-5BF053587FAB@illinois.edu>

A guess is that you?re bumping into an FD_SETSIZE limit ? the way remote I/O is currently structured has at least 5 file descriptors per remote connection from what I can see at a glance (a pair of pipes, 2 fds each, for signaling read/write readiness related to ChunkedIO and one fd for the actual socket).  Typically, FD_SETSIZE is 1024, so with ~150-200 remote connections and 5 fds per connection plus whatever other descriptors Bro may need to have open (e.g. for file I/O), it seems reasonable to guess that?s the problem.  But you could easily verify w/ some code modifications to check whether the FD_SET call is using a fd >= FD_SETSIZE.

Other than making involved code changes to Bro (e.g. to move away from select() for I/O event handling), the only suggestions I have are 1) reducing number of remote connections 2) see if you can increase FD_SETSIZE via preprocessor stuff or CFLAGS/CXXFLAGS upon ./configure?ing (I?ve never done this myself to know if it works, but I?ve googled around before and think the implication was that it may work on Linux).

- Jon

> On Jun 29, 2015, at 6:22 PM, Baxter Milliwew <baxter.milliwew at gmail.com> wrote:
> 
> The manager still crashes.  Interesting note about a buffer overflow.
> 
> 
> [manager]
> 
> Bro 2.4
> Linux 3.16.0-38-generic
> 
> core
> [New LWP 18834]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Core was generated by `/usr/local/3rd-party/bro/bin/bro -U .status -p broctl -p broctl-live -p local -'.
> Program terminated with signal SIGABRT, Aborted.
> #0  0x00007f163bb46cc9 in __GI_raise (sig=sig at entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> 
> Thread 1 (Thread 0x............ (LWP 18834)):
> #0  0x00007f163bb46cc9 in __GI_raise (sig=sig at entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x00007f163bb4a0d8 in __GI_abort () at abort.c:89
> #2  0x00007f163bb83394 in __libc_message (do_abort=do_abort at entry=2, fmt=fmt at entry=0x............ "*** %s ***: %s terminated\n") at ../sysdeps/posix/libc_fatal.c:175
> #3  0x00007f163bc1ac9c in __GI___fortify_fail (msg=<optimized out>, msg at entry=0x............ "buffer overflow detected") at fortify_fail.c:37
> #4  0x00007f163bc19b60 in __GI___chk_fail () at chk_fail.c:28
> #5  0x00007f163bc1abe7 in __fdelt_chk (d=<optimized out>) at fdelt_chk.c:25
> #6  0x00000000005e962a in Set (set=0x............, this=0x............) at /home/bro/Bro-IDS/bro-2.4/src/iosource/FD_Set.h:59
> #7  SocketComm::Run (this=0x............) at /home/bro/Bro-IDS/bro-2.4/src/RemoteSerializer.cc:3406
> #8  0x00000000005e9c31 in RemoteSerializer::Fork (this=0x............) at /home/bro/Bro-IDS/bro-2.4/src/RemoteSerializer.cc:687
> #9  0x00000000005e9d4f in RemoteSerializer::Enable (this=0x............) at /home/bro/Bro-IDS/bro-2.4/src/RemoteSerializer.cc:575
> #10 0x00000000005b6943 in BifFunc::bro_enable_communication (frame=<optimized out>, BiF_ARGS=<optimized out>) at bro.bif:4480
> #11 0x00000000005b431d in BuiltinFunc::Call (this=0x............, args=0x............, parent=0x............) at /home/bro/Bro-IDS/bro-2.4/src/Func.cc:586
> #12 0x0000000000599066 in CallExpr::Eval (this=0x............, f=0x............) at /home/bro/Bro-IDS/bro-2.4/src/Expr.cc:4544
> #13 0x000000000060ceb4 in ExprStmt::Exec (this=0x............, f=0x............, flow=@0x............: FLOW_NEXT) at /home/bro/Bro-IDS/bro-2.4/src/Stmt.cc:352
> #14 0x000000000060b174 in IfStmt::DoExec (this=0x............, f=0x............, v=<optimized out>, flow=@0x............: FLOW_NEXT) at /home/bro/Bro-IDS/bro-2.4/src/Stmt.cc:456
> #15 0x000000000060ced1 in ExprStmt::Exec (this=0x............, f=0x............, flow=@0x............: FLOW_NEXT) at /home/bro/Bro-IDS/bro-2.4/src/Stmt.cc:356
> #16 0x000000000060b211 in StmtList::Exec (this=0x............, f=0x............, flow=@0x............: FLOW_NEXT) at /home/bro/Bro-IDS/bro-2.4/src/Stmt.cc:1696
> #17 0x000000000060b211 in StmtList::Exec (this=0x............, f=0x............, flow=@0x............: FLOW_NEXT) at /home/bro/Bro-IDS/bro-2.4/src/Stmt.cc:1696
> #18 0x00000000005c042e in BroFunc::Call (this=0x............, args=<optimized out>, parent=0x0) at /home/bro/Bro-IDS/bro-2.4/src/Func.cc:403
> #19 0x000000000057ee2a in EventHandler::Call (this=0x............, vl=0x............, no_remote=no_remote at entry=false) at /home/bro/Bro-IDS/bro-2.4/src/EventHandler.cc:130
> #20 0x000000000057e035 in Dispatch (no_remote=false, this=0x............) at /home/bro/Bro-IDS/bro-2.4/src/Event.h:50
> #21 EventMgr::Dispatch (this=this at entry=0x...... <mgr>) at /home/bro/Bro-IDS/bro-2.4/src/Event.cc:111
> #22 0x000000000057e1d0 in EventMgr::Drain (this=0xbbd720 <mgr>) at /home/bro/Bro-IDS/bro-2.4/src/Event.cc:128
> #23 0x00000000005300ed in main (argc=<optimized out>, argv=<optimized out>) at /home/bro/Bro-IDS/bro-2.4/src/main.cc:1147
> 
> 
> 
> On Mon, Jun 29, 2015 at 4:09 PM, Baxter Milliwew <baxter.milliwew at gmail.com> wrote:
> Nevermind... new box, default nofile limits.  Thanks for the malloc tip.
> 
> 
> On Mon, Jun 29, 2015 at 4:03 PM, Baxter Milliwew <baxter.milliwew at gmail.com> wrote:
> Switching to jemalloc fixed the stability issue but not the worker count limitation. 
> 
> On Sun, Jun 28, 2015 at 7:18 PM, Baxter Milliwew <baxter.milliwew at gmail.com> wrote:
> Looks like malloc from glibc, default on Ubuntu.  I will try jemalloc and others.
> 
> 
> 
> On Sun, Jun 28, 2015 at 1:03 AM, Jan Grashofer <jan.grashofer at cern.ch> wrote:
> I experienced similar problems (memory gets eaten up quickly and workers crash with segfault) using tcmalloc. Which malloc do you use?
> 
>  
> Regards,
> 
> Jan 
> 
>  
> From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Baxter Milliwew [baxter.milliwew at gmail.com]
> Sent: Friday, June 26, 2015 23:03
> To: bro at bro.org
> Subject: [Bro] Bro's limitations with high worker count and memory exhaustion
> 
> There's some sort of association between memory exhaustion and a high number of workers.  The poor man's fix would be to purchase new servers with higher CPU speeds as that would reduce the worker count.  Issues with high worker count and/or memory exhaustion appears to be a well know problem based on the mailing list archives.
> 
> In the current version of bro-2.4 my previous configuration immediately causes the manager to crash: 15 proxies, 155 workers.  To resolve this I've lowered the count to 10 proxies and 140 workers.  However even with this configuration the manager process will exhaust all memory and crash within about 2 hours.
> 
> The manager is threaded; I think this is an issue with the threading behavior between manager, proxies, and workers.  Debugging threading problems is complex and I'm a complete novice.. my current tutorial is using information from a stack overflow thread:
> 
> http://stackoverflow.com/questions/981011/c-programming-debugging-with-pthreads
> 
> Does anyone else have this problem ?  What have you tried and what do you suggest ? 
> 
> Thanks
> 
> 
> 
> 
> 1435347409.458185       worker-2-18     parent  -       -       -       info    [#10000/10.1.1.1:36994] peer sent class "control"
> 1435347409.458185       worker-2-18     parent  -       -       -       info    [#10000/10.1.1.1:36994] phase: handshake
> 1435347409.661085       worker-2-18     parent  -       -       -       info    [#10000/10.1.1.1:36994] request for unknown event save_results
> 1435347409.661085       worker-2-18     parent  -       -       -       info    [#10000/10.1.1.1:36994] registered for event Control::peer_status_response
> 1435347409.694858       worker-2-18     parent  -       -       -       info    [#10000/10.1.1.1:36994] peer does not support 64bit PIDs; using compatibility mode
> 1435347409.694858       worker-2-18     parent  -       -       -       info    [#10000/10.1.1.1:36994] peer is a Broccoli
> 1435347409.694858       worker-2-18     parent  -       -       -       info    [#10000/10.1.1.1:36994] phase: running
> 
> 
> 
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From dnthayer at illinois.edu  Tue Jun 30 08:49:01 2015
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Tue, 30 Jun 2015 10:49:01 -0500
Subject: [Bro] ASCII JSON log stream
In-Reply-To: <864C1DD0-46CF-4D2F-B520-0A0A3397DC6D@gmail.com>
References: <864C1DD0-46CF-4D2F-B520-0A0A3397DC6D@gmail.com>
Message-ID: <5592BA6D.7050309@illinois.edu>



On 06/30/2015 02:52 AM, Albert Zaharovits wrote:
> Hello,
>
> I am writing a bro script which creates a ASCII log stream. I would like JSON output only for this stream. I was able to turn on JSON output globally.
>
> Any idea?
>
> Albert

There is an example in the "Logging Framework" documentation that you 
can use:
https://www.bro.org/sphinx/frameworks/logging.html#ascii-writer

Just replace "tsv" in the example with "use_json", and replace
Conn::LOG with your log stream ID.



From baxter.milliwew at gmail.com  Tue Jun 30 11:37:30 2015
From: baxter.milliwew at gmail.com (Baxter Milliwew)
Date: Tue, 30 Jun 2015 11:37:30 -0700
Subject: [Bro] Bro's limitations with high worker count and memory
	exhaustion
In-Reply-To: <BE09FB89-69C7-42AA-8967-5BF053587FAB@illinois.edu>
References: <CA+J8G6FgNij+Tu9Kk8jQNe=1kAs0z7zf6D_5-ATB8VZE2nmY4A@mail.gmail.com>
	<E3142ED5D6744B4C8E17323B862FEDE5DF6A8B@CERNXCHG62.cern.ch>
	<CA+J8G6HRjn4cXb+mjBw-XdJaBHOG-MgdwtKbM0kWKE2B8UPQuQ@mail.gmail.com>
	<CA+J8G6H+vx8skqR-Q9+qZK7=jQ3--60_6RsqbFtj2ghd9BvxDw@mail.gmail.com>
	<CA+J8G6ENrX=DoB10XuPnykHZB7Dc2hCuePah5vZcebStYvLbCQ@mail.gmail.com>
	<CA+J8G6Fht9Kb4OBrYxPxrj9xzVzHBy_fV93=JVJpF7vMPpOvEA@mail.gmail.com>
	<BE09FB89-69C7-42AA-8967-5BF053587FAB@illinois.edu>
Message-ID: <CA+J8G6GsxmJi2s36+mM01YYgrr5qVJO0+gf+UxJxC-7-u6G1hw@mail.gmail.com>

Thanks.  Some limited reading says it's not possible to increase FD_SETSIZE
on linux and it's time to migrate to poll().



On Tue, Jun 30, 2015 at 7:44 AM, Siwek, Jon <jsiwek at illinois.edu> wrote:

> A guess is that you?re bumping into an FD_SETSIZE limit ? the way remote
> I/O is currently structured has at least 5 file descriptors per remote
> connection from what I can see at a glance (a pair of pipes, 2 fds each,
> for signaling read/write readiness related to ChunkedIO and one fd for the
> actual socket).  Typically, FD_SETSIZE is 1024, so with ~150-200 remote
> connections and 5 fds per connection plus whatever other descriptors Bro
> may need to have open (e.g. for file I/O), it seems reasonable to guess
> that?s the problem.  But you could easily verify w/ some code modifications
> to check whether the FD_SET call is using a fd >= FD_SETSIZE.
>
> Other than making involved code changes to Bro (e.g. to move away from
> select() for I/O event handling), the only suggestions I have are 1)
> reducing number of remote connections 2) see if you can increase FD_SETSIZE
> via preprocessor stuff or CFLAGS/CXXFLAGS upon ./configure?ing (I?ve never
> done this myself to know if it works, but I?ve googled around before and
> think the implication was that it may work on Linux).
>
> - Jon
>
> > On Jun 29, 2015, at 6:22 PM, Baxter Milliwew <baxter.milliwew at gmail.com>
> wrote:
> >
> > The manager still crashes.  Interesting note about a buffer overflow.
> >
> >
> > [manager]
> >
> > Bro 2.4
> > Linux 3.16.0-38-generic
> >
> > core
> > [New LWP 18834]
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library
> "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > Core was generated by `/usr/local/3rd-party/bro/bin/bro -U .status -p
> broctl -p broctl-live -p local -'.
> > Program terminated with signal SIGABRT, Aborted.
> > #0  0x00007f163bb46cc9 in __GI_raise (sig=sig at entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> >
> > Thread 1 (Thread 0x............ (LWP 18834)):
> > #0  0x00007f163bb46cc9 in __GI_raise (sig=sig at entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> > #1  0x00007f163bb4a0d8 in __GI_abort () at abort.c:89
> > #2  0x00007f163bb83394 in __libc_message (do_abort=do_abort at entry=2,
> fmt=fmt at entry=0x............ "*** %s ***: %s terminated\n") at
> ../sysdeps/posix/libc_fatal.c:175
> > #3  0x00007f163bc1ac9c in __GI___fortify_fail (msg=<optimized out>,
> msg at entry=0x............ "buffer overflow detected") at fortify_fail.c:37
> > #4  0x00007f163bc19b60 in __GI___chk_fail () at chk_fail.c:28
> > #5  0x00007f163bc1abe7 in __fdelt_chk (d=<optimized out>) at
> fdelt_chk.c:25
> > #6  0x00000000005e962a in Set (set=0x............, this=0x............)
> at /home/bro/Bro-IDS/bro-2.4/src/iosource/FD_Set.h:59
> > #7  SocketComm::Run (this=0x............) at
> /home/bro/Bro-IDS/bro-2.4/src/RemoteSerializer.cc:3406
> > #8  0x00000000005e9c31 in RemoteSerializer::Fork (this=0x............)
> at /home/bro/Bro-IDS/bro-2.4/src/RemoteSerializer.cc:687
> > #9  0x00000000005e9d4f in RemoteSerializer::Enable (this=0x............)
> at /home/bro/Bro-IDS/bro-2.4/src/RemoteSerializer.cc:575
> > #10 0x00000000005b6943 in BifFunc::bro_enable_communication
> (frame=<optimized out>, BiF_ARGS=<optimized out>) at bro.bif:4480
> > #11 0x00000000005b431d in BuiltinFunc::Call (this=0x............,
> args=0x............, parent=0x............) at
> /home/bro/Bro-IDS/bro-2.4/src/Func.cc:586
> > #12 0x0000000000599066 in CallExpr::Eval (this=0x............,
> f=0x............) at /home/bro/Bro-IDS/bro-2.4/src/Expr.cc:4544
> > #13 0x000000000060ceb4 in ExprStmt::Exec (this=0x............,
> f=0x............, flow=@0x............: FLOW_NEXT) at
> /home/bro/Bro-IDS/bro-2.4/src/Stmt.cc:352
> > #14 0x000000000060b174 in IfStmt::DoExec (this=0x............,
> f=0x............, v=<optimized out>, flow=@0x............: FLOW_NEXT) at
> /home/bro/Bro-IDS/bro-2.4/src/Stmt.cc:456
> > #15 0x000000000060ced1 in ExprStmt::Exec (this=0x............,
> f=0x............, flow=@0x............: FLOW_NEXT) at
> /home/bro/Bro-IDS/bro-2.4/src/Stmt.cc:356
> > #16 0x000000000060b211 in StmtList::Exec (this=0x............,
> f=0x............, flow=@0x............: FLOW_NEXT) at
> /home/bro/Bro-IDS/bro-2.4/src/Stmt.cc:1696
> > #17 0x000000000060b211 in StmtList::Exec (this=0x............,
> f=0x............, flow=@0x............: FLOW_NEXT) at
> /home/bro/Bro-IDS/bro-2.4/src/Stmt.cc:1696
> > #18 0x00000000005c042e in BroFunc::Call (this=0x............,
> args=<optimized out>, parent=0x0) at
> /home/bro/Bro-IDS/bro-2.4/src/Func.cc:403
> > #19 0x000000000057ee2a in EventHandler::Call (this=0x............,
> vl=0x............, no_remote=no_remote at entry=false) at
> /home/bro/Bro-IDS/bro-2.4/src/EventHandler.cc:130
> > #20 0x000000000057e035 in Dispatch (no_remote=false,
> this=0x............) at /home/bro/Bro-IDS/bro-2.4/src/Event.h:50
> > #21 EventMgr::Dispatch (this=this at entry=0x...... <mgr>) at
> /home/bro/Bro-IDS/bro-2.4/src/Event.cc:111
> > #22 0x000000000057e1d0 in EventMgr::Drain (this=0xbbd720 <mgr>) at
> /home/bro/Bro-IDS/bro-2.4/src/Event.cc:128
> > #23 0x00000000005300ed in main (argc=<optimized out>, argv=<optimized
> out>) at /home/bro/Bro-IDS/bro-2.4/src/main.cc:1147
> >
> >
> >
> > On Mon, Jun 29, 2015 at 4:09 PM, Baxter Milliwew <
> baxter.milliwew at gmail.com> wrote:
> > Nevermind... new box, default nofile limits.  Thanks for the malloc tip.
> >
> >
> > On Mon, Jun 29, 2015 at 4:03 PM, Baxter Milliwew <
> baxter.milliwew at gmail.com> wrote:
> > Switching to jemalloc fixed the stability issue but not the worker count
> limitation.
> >
> > On Sun, Jun 28, 2015 at 7:18 PM, Baxter Milliwew <
> baxter.milliwew at gmail.com> wrote:
> > Looks like malloc from glibc, default on Ubuntu.  I will try jemalloc
> and others.
> >
> >
> >
> > On Sun, Jun 28, 2015 at 1:03 AM, Jan Grashofer <jan.grashofer at cern.ch>
> wrote:
> > I experienced similar problems (memory gets eaten up quickly and workers
> crash with segfault) using tcmalloc. Which malloc do you use?
> >
> >
> > Regards,
> >
> > Jan
> >
> >
> > From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Baxter
> Milliwew [baxter.milliwew at gmail.com]
> > Sent: Friday, June 26, 2015 23:03
> > To: bro at bro.org
> > Subject: [Bro] Bro's limitations with high worker count and memory
> exhaustion
> >
> > There's some sort of association between memory exhaustion and a high
> number of workers.  The poor man's fix would be to purchase new servers
> with higher CPU speeds as that would reduce the worker count.  Issues with
> high worker count and/or memory exhaustion appears to be a well know
> problem based on the mailing list archives.
> >
> > In the current version of bro-2.4 my previous configuration immediately
> causes the manager to crash: 15 proxies, 155 workers.  To resolve this I've
> lowered the count to 10 proxies and 140 workers.  However even with this
> configuration the manager process will exhaust all memory and crash within
> about 2 hours.
> >
> > The manager is threaded; I think this is an issue with the threading
> behavior between manager, proxies, and workers.  Debugging threading
> problems is complex and I'm a complete novice.. my current tutorial is
> using information from a stack overflow thread:
> >
> >
> http://stackoverflow.com/questions/981011/c-programming-debugging-with-pthreads
> >
> > Does anyone else have this problem ?  What have you tried and what do
> you suggest ?
> >
> > Thanks
> >
> >
> >
> >
> > 1435347409.458185       worker-2-18     parent  -       -       -
>  info    [#10000/10.1.1.1:36994] peer sent class "control"
> > 1435347409.458185       worker-2-18     parent  -       -       -
>  info    [#10000/10.1.1.1:36994] phase: handshake
> > 1435347409.661085       worker-2-18     parent  -       -       -
>  info    [#10000/10.1.1.1:36994] request for unknown event save_results
> > 1435347409.661085       worker-2-18     parent  -       -       -
>  info    [#10000/10.1.1.1:36994] registered for event
> Control::peer_status_response
> > 1435347409.694858       worker-2-18     parent  -       -       -
>  info    [#10000/10.1.1.1:36994] peer does not support 64bit PIDs; using
> compatibility mode
> > 1435347409.694858       worker-2-18     parent  -       -       -
>  info    [#10000/10.1.1.1:36994] peer is a Broccoli
> > 1435347409.694858       worker-2-18     parent  -       -       -
>  info    [#10000/10.1.1.1:36994] phase: running
> >
> >
> >
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150630/b57a07ff/attachment-0001.html