From Thomas.Bryant at parsons.com  Wed Jun  4 06:49:57 2014
From: Thomas.Bryant at parsons.com (Bryant, Thomas)
Date: Wed, 4 Jun 2014 13:49:57 +0000
Subject: [Bro] bro script question.
Message-ID: <466F52083C4AEB459F4271C87D3B890F24145B29@HSV-MB001.huntsville.ads.sparta.com>

All,

   I've run a script against a pcap file:

----------------------------------------------------------------
module TstLog;

export {
  redef enum Log::ID += { LOG };

  type TstRec: record {
    data: string &log;
  };
}

event connection_established(c: connection) {
  local str = fmt("Connection established: %s", c$id);
  print fmt("%s\n", str); #TEST
  # Log::write(TstRec::LOG, $data=str);
  # Log::write(TstLog::LOG, [$data=mkString()]);
}

function mkString(): string {
  return "This is a string.";
}

event bro_init() {
  Log::create_stream(LOG, [$columns=TstRec]);
}

event bro_done() {
  Log::write(TstLog::LOG, [$data=mkString()]);
}
---------------------------------------------------------------

with the following command:

bro tstLog.bro -r myPcapFile.pcap

As written, it works as expected.  If I uncomment either of the two Log::write lines in event connection_established, It produces the following error:

 1216691467.432370 fatal error in <no location>: Val::CONVERTER (string/record) (Connection established: [orig_h=192.168.1.64, orig_p=39153/tcp, resp_h=74.125.19.103, resp_p=443/tcp])

Note that an identical Log::write in bro_done produces no error.

Can someone explain what's going on here?

Thanks in advance!

Tom



From JAzoff at albany.edu  Wed Jun  4 07:08:15 2014
From: JAzoff at albany.edu (Justin Azoff)
Date: Wed, 4 Jun 2014 10:08:15 -0400
Subject: [Bro] bro script question.
In-Reply-To: <466F52083C4AEB459F4271C87D3B890F24145B29@HSV-MB001.huntsville.ads.sparta.com>
References: <466F52083C4AEB459F4271C87D3B890F24145B29@HSV-MB001.huntsville.ads.sparta.com>
Message-ID: <20140604140815.GI3515@datacomm.albany.edu>

On Wed, Jun 04, 2014 at 01:49:57PM +0000, Bryant, Thomas wrote:
>   # Log::write(TstRec::LOG, $data=str);
>   # Log::write(TstLog::LOG, [$data=mkString()]);

You want

  Log::write(LOG, [$data=str]);

-- 
-- Justin Azoff


From Thomas.Bryant at parsons.com  Wed Jun  4 08:21:00 2014
From: Thomas.Bryant at parsons.com (Bryant, Thomas)
Date: Wed, 4 Jun 2014 15:21:00 +0000
Subject: [Bro] bro script question.
In-Reply-To: <20140604140815.GI3515@datacomm.albany.edu>
References: <466F52083C4AEB459F4271C87D3B890F24145B29@HSV-MB001.huntsville.ads.sparta.com>,
	<20140604140815.GI3515@datacomm.albany.edu>
Message-ID: <466F52083C4AEB459F4271C87D3B890F24145B49@HSV-MB001.huntsville.ads.sparta.com>

Justin,

   I added your line to the connection_established event:

event connection_established(c: connection) {
  local str = fmt("Connection established: %s", c$id);
  print fmt("%s\n", str); #TEST
  Log::write(TstRec::LOG, [$data=str]);
  # Log::write(TstLog::LOG, [$data=mkString()]);
}

  and got a different error:

error in ./tstLog.bro, line 14: unknown identifier TstRec::LOG, at or near "TstRec::LOG"

Bro can be very subtle... (subtle = "I don't know how it works!")

Tom
________________________________________
From: Justin Azoff [JAzoff at albany.edu]
Sent: Wednesday, June 04, 2014 10:08 AM
To: Bryant, Thomas
Cc: bro at bro.org
Subject: Re: [Bro] bro script question.

On Wed, Jun 04, 2014 at 01:49:57PM +0000, Bryant, Thomas wrote:
>   # Log::write(TstRec::LOG, $data=str);
>   # Log::write(TstLog::LOG, [$data=mkString()]);

You want

  Log::write(LOG, [$data=str]);

--
-- Justin Azoff



From JAzoff at albany.edu  Wed Jun  4 08:27:08 2014
From: JAzoff at albany.edu (Justin Azoff)
Date: Wed, 4 Jun 2014 11:27:08 -0400
Subject: [Bro] bro script question.
In-Reply-To: <466F52083C4AEB459F4271C87D3B890F24145B49@HSV-MB001.huntsville.ads.sparta.com>
References: <466F52083C4AEB459F4271C87D3B890F24145B29@HSV-MB001.huntsville.ads.sparta.com>
	<20140604140815.GI3515@datacomm.albany.edu>
	<466F52083C4AEB459F4271C87D3B890F24145B49@HSV-MB001.huntsville.ads.sparta.com>
Message-ID: <20140604152708.GJ3515@datacomm.albany.edu>

On Wed, Jun 04, 2014 at 03:21:00PM +0000, Bryant, Thomas wrote:
> Justin,
> 
>    I added your line to the connection_established event:
..
>   Log::write(TstRec::LOG, [$data=str]);

...

> You want
> 
>   Log::write(LOG, [$data=str]);

Spot the difference :-)

-- 
-- Justin Azoff


From mattchess50 at gmail.com  Wed Jun  4 09:13:56 2014
From: mattchess50 at gmail.com (Matt Stucky)
Date: Wed, 4 Jun 2014 11:13:56 -0500
Subject: [Bro] proxy crashing on Bro cluster
Message-ID: <CAMM5JYo86ez8fNn92omQDyMp8n1m4n2g-dnRyRMm+Aah2KEi+A@mail.gmail.com>

I've noticed on our bro cluster that the proxy keeps crashing and
restarting.  The cluster seems to be working, logs are being written, etc.
Has anyone seen this behavior?  What am I missing here?


>From the crash report:

internal error: unknown msg type 115 in Poll()

/opt/bro/share/broctl/scripts/run-bro: line 85: 18479 Aborted
(core dumped) nohup $mybro "$@"


It attempts to restart it, but the status always shows with ??? in the
Peers column:

proxy-1      proxy   1.2.3.4   running   12386  ???    04 Jun 07:30:05



Thanks,
matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140604/b4d593f9/attachment.html 

From mattchess50 at gmail.com  Wed Jun  4 09:49:10 2014
From: mattchess50 at gmail.com (Matt Stucky)
Date: Wed, 4 Jun 2014 11:49:10 -0500
Subject: [Bro] proxy crashing on Bro cluster
In-Reply-To: <6692B762-363A-4640-AA5C-7C7597242FFB@andrew.cmu.edu>
References: <CAMM5JYo86ez8fNn92omQDyMp8n1m4n2g-dnRyRMm+Aah2KEi+A@mail.gmail.com>
	<6692B762-363A-4640-AA5C-7C7597242FFB@andrew.cmu.edu>
Message-ID: <CAMM5JYp=TNpfJdHLNUe-czTe12njRUGPASN+RGhAbnhfLQDT-A@mail.gmail.com>

This cluster has two hosts, one proxy, 12 workers, and sees 900+ mbps of
traffic.

Would it make more sense to have a proxy on each host?



On Wed, Jun 4, 2014 at 11:19 AM, Vlad Grigorescu <vladg at cmu.edu> wrote:

> I've seen this behavior when the proxy is overloaded. Could you share a
> bit about your cluster? How many proxies, how many workers, estimate of
> traffic that's being inspected, hardware being used, etc?
>
>   --Vlad
>
>
> On Jun 4, 2014, at 12:13 PM, Matt Stucky <mattchess50 at gmail.com> wrote:
>
> > I've noticed on our bro cluster that the proxy keeps crashing and
> restarting.  The cluster seems to be working, logs are being written, etc.
>  Has anyone seen this behavior?  What am I missing here?
> >
> >
> > From the crash report:
> > internal error: unknown msg type 115 in Poll()
> >
> > /opt/bro/share/broctl/scripts/run-bro: line 85: 18479 Aborted
>       (core dumped) nohup $mybro "$@"
> >
> >
> >
> > It attempts to restart it, but the status always shows with ??? in the
> Peers column:
> >
> > proxy-1      proxy   1.2.3.4   running   12386  ???    04 Jun 07:30:05
> >
> >
> >
> > Thanks,
> > matt
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140604/e23f6f9d/attachment.html 

From Thomas.Bryant at parsons.com  Wed Jun  4 10:06:54 2014
From: Thomas.Bryant at parsons.com (Bryant, Thomas)
Date: Wed, 4 Jun 2014 17:06:54 +0000
Subject: [Bro] bro script question.
In-Reply-To: <20140604152708.GJ3515@datacomm.albany.edu>
References: <466F52083C4AEB459F4271C87D3B890F24145B29@HSV-MB001.huntsville.ads.sparta.com>
	<20140604140815.GI3515@datacomm.albany.edu>
	<466F52083C4AEB459F4271C87D3B890F24145B49@HSV-MB001.huntsville.ads.sparta.com>,
	<20140604152708.GJ3515@datacomm.albany.edu>
Message-ID: <466F52083C4AEB459F4271C87D3B890F24145B64@HSV-MB001.huntsville.ads.sparta.com>

Yes!! 

That worked.

Thanks Justin.

Tom
________________________________________
From: Justin Azoff [JAzoff at albany.edu]
Sent: Wednesday, June 04, 2014 11:27 AM
To: Bryant, Thomas
Cc: bro at bro.org
Subject: Re: [Bro] bro script question.

On Wed, Jun 04, 2014 at 03:21:00PM +0000, Bryant, Thomas wrote:
> Justin,
>
>    I added your line to the connection_established event:
..
>   Log::write(TstRec::LOG, [$data=str]);

...

> You want
>
>   Log::write(LOG, [$data=str]);

Spot the difference :-)

--
-- Justin Azoff



From seth at icir.org  Wed Jun  4 12:59:24 2014
From: seth at icir.org (Seth Hall)
Date: Wed, 4 Jun 2014 15:59:24 -0400
Subject: [Bro] proxy crashing on Bro cluster
In-Reply-To: <CAMM5JYo86ez8fNn92omQDyMp8n1m4n2g-dnRyRMm+Aah2KEi+A@mail.gmail.com>
References: <CAMM5JYo86ez8fNn92omQDyMp8n1m4n2g-dnRyRMm+Aah2KEi+A@mail.gmail.com>
Message-ID: <5347C42D-721D-4602-8954-15CD0BBEB6A5@icir.org>


On Jun 4, 2014, at 12:13 PM, Matt Stucky <mattchess50 at gmail.com> wrote:

> internal error: unknown msg type 115 in Poll()

This is communication system overload.  One of the many reasons we've been planning to replace the communication infrastructure in Bro.

Try adding another proxy to your cluster.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140604/c9b7a0ed/attachment.bin 

From mattchess50 at gmail.com  Wed Jun  4 14:10:48 2014
From: mattchess50 at gmail.com (Matt Stucky)
Date: Wed, 4 Jun 2014 16:10:48 -0500
Subject: [Bro] proxy crashing on Bro cluster
In-Reply-To: <5347C42D-721D-4602-8954-15CD0BBEB6A5@icir.org>
References: <CAMM5JYo86ez8fNn92omQDyMp8n1m4n2g-dnRyRMm+Aah2KEi+A@mail.gmail.com>
	<5347C42D-721D-4602-8954-15CD0BBEB6A5@icir.org>
Message-ID: <CAMM5JYo0yTsu5EBpu6ngzdXKyPrK95Y8YBSXZq6TPDrR=6k3Cg@mail.gmail.com>

Updated the cluster to run a proxy on each host now, and so far so good, no
crashes.

Thanks,
-matt


On Wed, Jun 4, 2014 at 2:59 PM, Seth Hall <seth at icir.org> wrote:

>
> On Jun 4, 2014, at 12:13 PM, Matt Stucky <mattchess50 at gmail.com> wrote:
>
> > internal error: unknown msg type 115 in Poll()
>
> This is communication system overload.  One of the many reasons we've been
> planning to replace the communication infrastructure in Bro.
>
> Try adding another proxy to your cluster.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140604/cbfdb1e4/attachment.html 

From seth at icir.org  Wed Jun  4 14:20:54 2014
From: seth at icir.org (Seth Hall)
Date: Wed, 4 Jun 2014 17:20:54 -0400
Subject: [Bro] proxy crashing on Bro cluster
In-Reply-To: <CAMM5JYo0yTsu5EBpu6ngzdXKyPrK95Y8YBSXZq6TPDrR=6k3Cg@mail.gmail.com>
References: <CAMM5JYo86ez8fNn92omQDyMp8n1m4n2g-dnRyRMm+Aah2KEi+A@mail.gmail.com>
	<5347C42D-721D-4602-8954-15CD0BBEB6A5@icir.org>
	<CAMM5JYo0yTsu5EBpu6ngzdXKyPrK95Y8YBSXZq6TPDrR=6k3Cg@mail.gmail.com>
Message-ID: <C92B7AF9-41A1-43E1-937E-224F63F0463C@icir.org>


On Jun 4, 2014, at 5:10 PM, Matt Stucky <mattchess50 at gmail.com> wrote:

> Updated the cluster to run a proxy on each host now, and so far so good, no crashes.

Cool, the general rule of thumb I've been giving people lately is somewhere around 14-15 workers per proxy.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140604/430c3ab9/attachment.bin 

From mattchess50 at gmail.com  Wed Jun  4 14:27:32 2014
From: mattchess50 at gmail.com (Matt Stucky)
Date: Wed, 4 Jun 2014 16:27:32 -0500
Subject: [Bro] proxy crashing on Bro cluster
In-Reply-To: <C92B7AF9-41A1-43E1-937E-224F63F0463C@icir.org>
References: <CAMM5JYo86ez8fNn92omQDyMp8n1m4n2g-dnRyRMm+Aah2KEi+A@mail.gmail.com>
	<5347C42D-721D-4602-8954-15CD0BBEB6A5@icir.org>
	<CAMM5JYo0yTsu5EBpu6ngzdXKyPrK95Y8YBSXZq6TPDrR=6k3Cg@mail.gmail.com>
	<C92B7AF9-41A1-43E1-937E-224F63F0463C@icir.org>
Message-ID: <CAMM5JYqaonPGq1RoO6Yswmxrc7dfZfxS3jGh=eheukbU6Qpb+Q@mail.gmail.com>

Is that with a dedicated proxy?  I now have manager, proxy, and six workers
on one host; proxy and six workers on the other.


On Wed, Jun 4, 2014 at 4:20 PM, Seth Hall <seth at icir.org> wrote:

>
> On Jun 4, 2014, at 5:10 PM, Matt Stucky <mattchess50 at gmail.com> wrote:
>
> > Updated the cluster to run a proxy on each host now, and so far so good,
> no crashes.
>
> Cool, the general rule of thumb I've been giving people lately is
> somewhere around 14-15 workers per proxy.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140604/fffc7dd7/attachment.html 

From seth at icir.org  Wed Jun  4 14:35:46 2014
From: seth at icir.org (Seth Hall)
Date: Wed, 4 Jun 2014 17:35:46 -0400
Subject: [Bro] proxy crashing on Bro cluster
In-Reply-To: <CAMM5JYqaonPGq1RoO6Yswmxrc7dfZfxS3jGh=eheukbU6Qpb+Q@mail.gmail.com>
References: <CAMM5JYo86ez8fNn92omQDyMp8n1m4n2g-dnRyRMm+Aah2KEi+A@mail.gmail.com>
	<5347C42D-721D-4602-8954-15CD0BBEB6A5@icir.org>
	<CAMM5JYo0yTsu5EBpu6ngzdXKyPrK95Y8YBSXZq6TPDrR=6k3Cg@mail.gmail.com>
	<C92B7AF9-41A1-43E1-937E-224F63F0463C@icir.org>
	<CAMM5JYqaonPGq1RoO6Yswmxrc7dfZfxS3jGh=eheukbU6Qpb+Q@mail.gmail.com>
Message-ID: <6669CAA0-21A8-4CB9-9980-BD7F7A615ACE@icir.org>


On Jun 4, 2014, at 5:27 PM, Matt Stucky <mattchess50 at gmail.com> wrote:

> Is that with a dedicated proxy?  I now have manager, proxy, and six workers on one host; proxy and six workers on the other.

Ah, it could be a system resource issue causing trouble for you too.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140604/2cf84ddc/attachment.bin 

From michael.wenthold at gmail.com  Thu Jun  5 09:22:15 2014
From: michael.wenthold at gmail.com (Michael Wenthold)
Date: Thu, 5 Jun 2014 16:22:15 +0000
Subject: [Bro] Problems parsing x509 issuer?
Message-ID: <CAMU2yhRD4NxPjP1HsqN6f0tj0ynWduWXK+d710LZ70yL=TiDfQ@mail.gmail.com>

All,

We are experimenting with tracking/whitelisting x509 certificate issuers,
using Bro 2.2.  I'm seeing that certain certificates consistently don't
appear to be getting parsed properly.

For example:

1.311.60.2.1.3=#13025553CN=VeriSign Class 3 Extended Validation SSL SGC
CA,OU=Terms of use at https://www.verisign.com/rpa (c)06,OU=VeriSign Trust
Network,O=VeriSign\, Inc.,C=US

025553CN=VeriSign Class 3 Extended Validation SSL SGC CA,OU=Terms of use at
https://www.verisign.com/rpa (c)06,OU=VeriSign Trust Network,O=VeriSign\,
Inc.,C=US

.1.3=#13025553CN=VeriSign Class 3 Extended Validation SSL SGC CA,OU=Terms
of use at https://www.verisign.com/rpa (c)06,OU=VeriSign Trust
Network,O=VeriSign\, Inc.,C=US


This is just a small sample, but it appears to happen mostly with certain
certificates (like the Verisign extended validation certs).   Is anyone
else seeing this?


Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140605/5dee4939/attachment.html 

From netantho at gmail.com  Thu Jun  5 09:31:09 2014
From: netantho at gmail.com (Anthony VEREZ)
Date: Thu, 05 Jun 2014 09:31:09 -0700
Subject: [Bro] Problems parsing x509 issuer?
In-Reply-To: <CAMU2yhRD4NxPjP1HsqN6f0tj0ynWduWXK+d710LZ70yL=TiDfQ@mail.gmail.com>
References: <CAMU2yhRD4NxPjP1HsqN6f0tj0ynWduWXK+d710LZ70yL=TiDfQ@mail.gmail.com>
Message-ID: <53909B4D.1050008@gmail.com>

Hello Michael,

it may have been fixed in the dev version, see
https://bro-tracker.atlassian.net/browse/BIT-1195

Anthony

On 6/5/14, 9:22 AM, Michael Wenthold wrote:
> All,
> 
> We are experimenting with tracking/whitelisting x509 certificate
> issuers, using Bro 2.2.  I'm seeing that certain certificates
> consistently don't appear to be getting parsed properly.
> 
> For example:
> 
> 1.311.60.2.1.3=#13025553CN=VeriSign Class 3 Extended Validation SSL SGC
> CA,OU=Terms of use at https://www.verisign.com/rpa (c)06,OU=VeriSign
> Trust Network,O=VeriSign\, Inc.,C=US
> 
> 025553CN=VeriSign Class 3 Extended Validation SSL SGC CA,OU=Terms of use
> at https://www.verisign.com/rpa (c)06,OU=VeriSign Trust
> Network,O=VeriSign\, Inc.,C=US
> 
> .1.3=#13025553CN=VeriSign Class 3 Extended Validation SSL SGC
> CA,OU=Terms of use at https://www.verisign.com/rpa (c)06,OU=VeriSign
> Trust Network,O=VeriSign\, Inc.,C=US
> 
> 
> This is just a small sample, but it appears to happen mostly with
> certain certificates (like the Verisign extended validation certs).   Is
> anyone else seeing this?
> 
> 
> Mike
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 



From shane.castle at gmail.com  Thu Jun  5 09:45:20 2014
From: shane.castle at gmail.com (Shane Castle)
Date: Thu, 5 Jun 2014 18:45:20 +0200
Subject: [Bro] Problems parsing x509 issuer?
In-Reply-To: <CAMU2yhRD4NxPjP1HsqN6f0tj0ynWduWXK+d710LZ70yL=TiDfQ@mail.gmail.com>
References: <CAMU2yhRD4NxPjP1HsqN6f0tj0ynWduWXK+d710LZ70yL=TiDfQ@mail.gmail.com>
Message-ID: <CDA72713-8FE1-4CAC-A64C-5F66B0876AB3@gmail.com>

I saw that too, that even if I was careful to get copies in the host's cert storage and run a script like is outlined in (https://www.bro.org/current/solutions/extending/). It seems there is indeed an issue with the cert parsing code.

Sent from my iPad

> On Jun 5, 2014, at 18:22, Michael Wenthold <michael.wenthold at gmail.com> wrote:
> 
> All,
> 
> We are experimenting with tracking/whitelisting x509 certificate issuers, using Bro 2.2.  I'm seeing that certain certificates consistently don't appear to be getting parsed properly.
> 
> For example:
> 
> 1.311.60.2.1.3=#13025553CN=VeriSign Class 3 Extended Validation SSL SGC CA,OU=Terms of use at https://www.verisign.com/rpa (c)06,OU=VeriSign Trust Network,O=VeriSign\, Inc.,C=US
> 
> 025553CN=VeriSign Class 3 Extended Validation SSL SGC CA,OU=Terms of use at https://www.verisign.com/rpa (c)06,OU=VeriSign Trust Network,O=VeriSign\, Inc.,C=US
> 
> .1.3=#13025553CN=VeriSign Class 3 Extended Validation SSL SGC CA,OU=Terms of use at https://www.verisign.com/rpa (c)06,OU=VeriSign Trust Network,O=VeriSign\, Inc.,C=US
> 
> 
> This is just a small sample, but it appears to happen mostly with certain certificates (like the Verisign extended validation certs).   Is anyone else seeing this?
> 
> 
> Mike
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140605/ce84813f/attachment.html 

From bernhard at ICSI.Berkeley.EDU  Thu Jun  5 10:00:13 2014
From: bernhard at ICSI.Berkeley.EDU (bernhard at ICSI.Berkeley.EDU)
Date: Thu, 05 Jun 2014 10:00:13 -0700
Subject: [Bro] Problems parsing x509 issuer?
In-Reply-To: <CAMU2yhRD4NxPjP1HsqN6f0tj0ynWduWXK+d710LZ70yL=TiDfQ@mail.gmail.com>
References: <CAMU2yhRD4NxPjP1HsqN6f0tj0ynWduWXK+d710LZ70yL=TiDfQ@mail.gmail.com>
Message-ID: <518710B3-6B47-43D5-A730-6A2F993B488F@icsi.berkeley.edu>

Hello Michael,

like Anthony said, this bug was probably fixed in the current master 
version. Could you try with that and see if that fixes your problem? I 
think this is the only change since 2.3-beta that made it into master, 
so using it will not break anything else.

Bernhard

On 5 Jun 2014, at 9:22, Michael Wenthold wrote:

> All,
>
> We are experimenting with tracking/whitelisting x509 certificate 
> issuers,
> using Bro 2.2.  I'm seeing that certain certificates consistently 
> don't
> appear to be getting parsed properly.
>
> For example:
>
> 1.311.60.2.1.3=#13025553CN=VeriSign Class 3 Extended Validation SSL 
> SGC
> CA,OU=Terms of use at https://www.verisign.com/rpa (c)06,OU=VeriSign 
> Trust
> Network,O=VeriSign\, Inc.,C=US
>
> 025553CN=VeriSign Class 3 Extended Validation SSL SGC CA,OU=Terms of 
> use at
> https://www.verisign.com/rpa (c)06,OU=VeriSign Trust 
> Network,O=VeriSign\,
> Inc.,C=US
>
> .1.3=#13025553CN=VeriSign Class 3 Extended Validation SSL SGC 
> CA,OU=Terms
> of use at https://www.verisign.com/rpa (c)06,OU=VeriSign Trust
> Network,O=VeriSign\, Inc.,C=US
>
>
> This is just a small sample, but it appears to happen mostly with 
> certain
> certificates (like the Verisign extended validation certs).   Is 
> anyone
> else seeing this?
>
>
> Mike
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From seth at icir.org  Thu Jun  5 10:46:25 2014
From: seth at icir.org (Seth Hall)
Date: Thu, 5 Jun 2014 13:46:25 -0400
Subject: [Bro] Problems parsing x509 issuer?
In-Reply-To: <CAMU2yhRD4NxPjP1HsqN6f0tj0ynWduWXK+d710LZ70yL=TiDfQ@mail.gmail.com>
References: <CAMU2yhRD4NxPjP1HsqN6f0tj0ynWduWXK+d710LZ70yL=TiDfQ@mail.gmail.com>
Message-ID: <B93D99D4-900F-45A1-B08D-94FBA603CE24@icir.org>


On Jun 5, 2014, at 12:22 PM, Michael Wenthold <michael.wenthold at gmail.com> wrote:

> We are experimenting with tracking/whitelisting x509 certificate issuers, using Bro 2.2.  I'm seeing that certain certificates consistently don't appear to be getting parsed properly.

Ignoring the potential certificate parsing issue, it's usually not a good idea to track certs by their subject.  You can collect the hash of the certificate and compare on that too.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140605/c680927c/attachment.bin 

From michael.wenthold at gmail.com  Thu Jun  5 13:26:13 2014
From: michael.wenthold at gmail.com (Michael Wenthold)
Date: Thu, 5 Jun 2014 20:26:13 +0000
Subject: [Bro] Problems parsing x509 issuer?
In-Reply-To: <518710B3-6B47-43D5-A730-6A2F993B488F@icsi.berkeley.edu>
References: <CAMU2yhRD4NxPjP1HsqN6f0tj0ynWduWXK+d710LZ70yL=TiDfQ@mail.gmail.com>
	<518710B3-6B47-43D5-A730-6A2F993B488F@icsi.berkeley.edu>
Message-ID: <CAMU2yhQ_nck+kruGXGTT731W-dh=YWm-0TFwB2Vyp=++Ku_drg@mail.gmail.com>

I'm only using a small sample pcap, but 2.3-beta5 appears to fix the
parsing issue.

thanks!


On Thu, Jun 5, 2014 at 5:00 PM, <bernhard at icsi.berkeley.edu> wrote:

> Hello Michael,
>
> like Anthony said, this bug was probably fixed in the current master
> version. Could you try with that and see if that fixes your problem? I
> think this is the only change since 2.3-beta that made it into master, so
> using it will not break anything else.
>
> Bernhard
>
>
> On 5 Jun 2014, at 9:22, Michael Wenthold wrote:
>
>  All,
>>
>> We are experimenting with tracking/whitelisting x509 certificate issuers,
>> using Bro 2.2.  I'm seeing that certain certificates consistently don't
>> appear to be getting parsed properly.
>>
>> For example:
>>
>> 1.311.60.2.1.3=#13025553CN=VeriSign Class 3 Extended Validation SSL SGC
>> CA,OU=Terms of use at https://www.verisign.com/rpa (c)06,OU=VeriSign
>> Trust
>> Network,O=VeriSign\, Inc.,C=US
>>
>> 025553CN=VeriSign Class 3 Extended Validation SSL SGC CA,OU=Terms of use
>> at
>> https://www.verisign.com/rpa (c)06,OU=VeriSign Trust Network,O=VeriSign\,
>> Inc.,C=US
>>
>> .1.3=#13025553CN=VeriSign Class 3 Extended Validation SSL SGC CA,OU=Terms
>> of use at https://www.verisign.com/rpa (c)06,OU=VeriSign Trust
>> Network,O=VeriSign\, Inc.,C=US
>>
>>
>> This is just a small sample, but it appears to happen mostly with certain
>> certificates (like the Verisign extended validation certs).   Is anyone
>> else seeing this?
>>
>>
>> Mike
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140605/eed5b7fe/attachment.html 

From dn1nj4 at gmail.com  Fri Jun  6 05:34:42 2014
From: dn1nj4 at gmail.com (dn1nj4)
Date: Fri, 6 Jun 2014 08:34:42 -0400
Subject: [Bro] Split PCAPs & Partial Connections
Message-ID: <734AFE64-7A1C-4E9B-BCD2-49E80EE5F5B6@gmail.com>

Hello all,

I am batch processing some periodic PCAP files to extract SSL certificates. I noticed if I do bro -r file1, followed by bro -r file2, I end up with two ssl.log files totalling ~1500 lines. However, if I mergecap file1 & file2 and run bro -r merged, my ssl.log is ~7000 lines. 

After searching the list archive and current bro docs, I thought this might be a partial connection problem. So I tried a redef partial_connection_ok = T; but that did not seem to have any effect.

Any insights into how I might fix the discrepancy would be greatly appreciated.

Jason


From sangdrax8 at gmail.com  Fri Jun  6 05:35:07 2014
From: sangdrax8 at gmail.com (sangdrax8)
Date: Fri, 6 Jun 2014 08:35:07 -0400
Subject: [Bro] Suppress_for issues
Message-ID: <CAL8ieHrr6MHe19oxAZ=pK7V4c3qmCKjVUbj7bMmrTCOLRPAcqQ@mail.gmail.com>

I am having some problems (or maybe misunderstanding) of how the
suppression works. I haven't changed my configuration file and it was
working at one time.  Now after upgrading to the master branch (I was on
the heartbleed) it seems my suppression isn't working as I understand it.

I have activated the SSL certificate checking as follows:
@load policy/protocols/ssl/expiring-certs.bro
redef SSL::notify_certs_expiration = ALL_HOSTS;

now when I watch my notice log, I am seeing what appear to be LOTS of
notice logs for the same certificate.  I thought that perhaps just the
e-mails get suppressed, but after turning on e-mail notifications I get an
e-mail for every notice.  Plus my notice log is filling up rather quickly.

I know this probably won't be very legible, but here is an example of just
2 of the notices I get from a single connection.  They look exactly the
same to me, and they have a time set for the suppression. I  would have
expected to only get one of these, but you can see the time stamp shows
multiple notices happening very quickly.

#fields ts      uid     id.orig_h       id.orig_p       id.resp_h
id.resp_p       fuid    file_mime_type  file_desc       proto   note    msg
    sub     src     dst     p       n       peer_descr      actions
suppress_for    dropped remote_location.country_code
 remote_location.region   remote_location.city    remote_location.latitude
       remote_location.longitude


1402057564.658489       CW6Riz4smTIRpMxWq1      1.1.1.1   51255   2.2.2.2
 5223    F6irMUcwkf1ZcbIok       -       -       tcp
SSL::Certificate_Expired        Certificate emailAddress=,CN=,OU=,O= -
  1.1.1.1   2.2.2.2  5223    -       bro1      Notice::ACTION_LOG
 86400.000000    F       -       -       -       -       -

1402057564.660035       CW6Riz4smTIRpMxWq1      1.1.1.1   51255   2.2.2.2
 5223    F6irMUcwkf1ZcbIok       -       -       tcp
SSL::Certificate_Expired        Certificate emailAddress=,CN=,OU=,O= -
  1.1.1.1   2.2.2.2  5223    -       bro1      Notice::ACTION_LOG
 86400.000000    F       -       -       -       -       -
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140606/7ed99f5e/attachment.html 

From liburdi.joshua at gmail.com  Fri Jun  6 05:54:19 2014
From: liburdi.joshua at gmail.com (Josh Liburdi)
Date: Fri, 6 Jun 2014 08:54:19 -0400
Subject: [Bro] Suppress_for issues
In-Reply-To: <CAL8ieHrr6MHe19oxAZ=pK7V4c3qmCKjVUbj7bMmrTCOLRPAcqQ@mail.gmail.com>
References: <CAL8ieHrr6MHe19oxAZ=pK7V4c3qmCKjVUbj7bMmrTCOLRPAcqQ@mail.gmail.com>
Message-ID: <CANrCRiHu43NGVc6SBvpk4BWZYtVFuNC6u0mJHAeOnJngbmSgoA@mail.gmail.com>

Looks to me like the $identifer field was dropped from those notices
with the move to 2.3 ...

Bro 2.2:

else if ( cert$not_valid_after < network_time() )
NOTICE([$note=Certificate_Expired,
       $conn=c, $suppress_for=1day,
       $msg=fmt("Certificate %s expired at %T", cert$subject,
cert$not_valid_after),
       $identifier=cat(c$id$resp_h, c$id$resp_p, c$ssl$cert_hash)]);


Bro 2.3:

else if ( cert$not_valid_after < network_time() )
NOTICE([$note=Certificate_Expired,
       $conn=c, $suppress_for=1day,
       $msg=fmt("Certificate %s expired at %T", cert$subject,
cert$not_valid_after),
       $fuid=fuid]);


That will break suppression.

-Josh

On Fri, Jun 6, 2014 at 8:35 AM, sangdrax8 <sangdrax8 at gmail.com> wrote:
> I am having some problems (or maybe misunderstanding) of how the suppression
> works. I haven't changed my configuration file and it was working at one
> time.  Now after upgrading to the master branch (I was on the heartbleed) it
> seems my suppression isn't working as I understand it.
>
> I have activated the SSL certificate checking as follows:
> @load policy/protocols/ssl/expiring-certs.bro
> redef SSL::notify_certs_expiration = ALL_HOSTS;
>
> now when I watch my notice log, I am seeing what appear to be LOTS of notice
> logs for the same certificate.  I thought that perhaps just the e-mails get
> suppressed, but after turning on e-mail notifications I get an e-mail for
> every notice.  Plus my notice log is filling up rather quickly.
>
> I know this probably won't be very legible, but here is an example of just 2
> of the notices I get from a single connection.  They look exactly the same
> to me, and they have a time set for the suppression. I  would have expected
> to only get one of these, but you can see the time stamp shows multiple
> notices happening very quickly.
>
> #fields ts      uid     id.orig_h       id.orig_p       id.resp_h
> id.resp_p       fuid    file_mime_type  file_desc       proto   note    msg
> sub     src     dst     p       n       peer_descr      actions suppress_for
> dropped remote_location.country_code    remote_location.region
> remote_location.city    remote_location.latitude
> remote_location.longitude
>
>
> 1402057564.658489       CW6Riz4smTIRpMxWq1      1.1.1.1   51255   2.2.2.2
> 5223    F6irMUcwkf1ZcbIok       -       -       tcp
> SSL::Certificate_Expired        Certificate emailAddress=,CN=,OU=,O= -
> 1.1.1.1   2.2.2.2  5223    -       bro1      Notice::ACTION_LOG
> 86400.000000    F       -       -       -       -       -
>
> 1402057564.660035       CW6Riz4smTIRpMxWq1      1.1.1.1   51255   2.2.2.2
> 5223    F6irMUcwkf1ZcbIok       -       -       tcp
> SSL::Certificate_Expired        Certificate emailAddress=,CN=,OU=,O= -
> 1.1.1.1   2.2.2.2  5223    -       bro1      Notice::ACTION_LOG
> 86400.000000    F       -       -       -       -       -
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From sangdrax8 at gmail.com  Fri Jun  6 06:03:27 2014
From: sangdrax8 at gmail.com (sangdrax8)
Date: Fri, 6 Jun 2014 09:03:27 -0400
Subject: [Bro] Suppress_for issues
In-Reply-To: <CANrCRiHu43NGVc6SBvpk4BWZYtVFuNC6u0mJHAeOnJngbmSgoA@mail.gmail.com>
References: <CAL8ieHrr6MHe19oxAZ=pK7V4c3qmCKjVUbj7bMmrTCOLRPAcqQ@mail.gmail.com>
	<CANrCRiHu43NGVc6SBvpk4BWZYtVFuNC6u0mJHAeOnJngbmSgoA@mail.gmail.com>
Message-ID: <CAL8ieHoNbMyUrzZnincgNqPmT9Db4pXM7e-xtaXU4q27The9sw@mail.gmail.com>

I was just trying to move back from the heartbleed branch in git to the
current "stable."  Should I be checking out something other than master to
make the move back from the heartbleed branch to stable branch?


On Fri, Jun 6, 2014 at 8:54 AM, Josh Liburdi <liburdi.joshua at gmail.com>
wrote:

> Looks to me like the $identifer field was dropped from those notices
> with the move to 2.3 ...
>
> Bro 2.2:
>
> else if ( cert$not_valid_after < network_time() )
> NOTICE([$note=Certificate_Expired,
>        $conn=c, $suppress_for=1day,
>        $msg=fmt("Certificate %s expired at %T", cert$subject,
> cert$not_valid_after),
>        $identifier=cat(c$id$resp_h, c$id$resp_p, c$ssl$cert_hash)]);
>
>
> Bro 2.3:
>
> else if ( cert$not_valid_after < network_time() )
> NOTICE([$note=Certificate_Expired,
>        $conn=c, $suppress_for=1day,
>        $msg=fmt("Certificate %s expired at %T", cert$subject,
> cert$not_valid_after),
>        $fuid=fuid]);
>
>
> That will break suppression.
>
> -Josh
>
> On Fri, Jun 6, 2014 at 8:35 AM, sangdrax8 <sangdrax8 at gmail.com> wrote:
> > I am having some problems (or maybe misunderstanding) of how the
> suppression
> > works. I haven't changed my configuration file and it was working at one
> > time.  Now after upgrading to the master branch (I was on the
> heartbleed) it
> > seems my suppression isn't working as I understand it.
> >
> > I have activated the SSL certificate checking as follows:
> > @load policy/protocols/ssl/expiring-certs.bro
> > redef SSL::notify_certs_expiration = ALL_HOSTS;
> >
> > now when I watch my notice log, I am seeing what appear to be LOTS of
> notice
> > logs for the same certificate.  I thought that perhaps just the e-mails
> get
> > suppressed, but after turning on e-mail notifications I get an e-mail for
> > every notice.  Plus my notice log is filling up rather quickly.
> >
> > I know this probably won't be very legible, but here is an example of
> just 2
> > of the notices I get from a single connection.  They look exactly the
> same
> > to me, and they have a time set for the suppression. I  would have
> expected
> > to only get one of these, but you can see the time stamp shows multiple
> > notices happening very quickly.
> >
> > #fields ts      uid     id.orig_h       id.orig_p       id.resp_h
> > id.resp_p       fuid    file_mime_type  file_desc       proto   note
>  msg
> > sub     src     dst     p       n       peer_descr      actions
> suppress_for
> > dropped remote_location.country_code    remote_location.region
> > remote_location.city    remote_location.latitude
> > remote_location.longitude
> >
> >
> > 1402057564.658489       CW6Riz4smTIRpMxWq1      1.1.1.1   51255   2.2.2.2
> > 5223    F6irMUcwkf1ZcbIok       -       -       tcp
> > SSL::Certificate_Expired        Certificate emailAddress=,CN=,OU=,O= -
> > 1.1.1.1   2.2.2.2  5223    -       bro1      Notice::ACTION_LOG
> > 86400.000000    F       -       -       -       -       -
> >
> > 1402057564.660035       CW6Riz4smTIRpMxWq1      1.1.1.1   51255   2.2.2.2
> > 5223    F6irMUcwkf1ZcbIok       -       -       tcp
> > SSL::Certificate_Expired        Certificate emailAddress=,CN=,OU=,O= -
> > 1.1.1.1   2.2.2.2  5223    -       bro1      Notice::ACTION_LOG
> > 86400.000000    F       -       -       -       -       -
> >
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140606/709fdd55/attachment.html 

From sangdrax8 at gmail.com  Fri Jun  6 07:55:18 2014
From: sangdrax8 at gmail.com (sangdrax8)
Date: Fri, 6 Jun 2014 10:55:18 -0400
Subject: [Bro] Suppress_for issues
In-Reply-To: <CAL8ieHoNbMyUrzZnincgNqPmT9Db4pXM7e-xtaXU4q27The9sw@mail.gmail.com>
References: <CAL8ieHrr6MHe19oxAZ=pK7V4c3qmCKjVUbj7bMmrTCOLRPAcqQ@mail.gmail.com>
	<CANrCRiHu43NGVc6SBvpk4BWZYtVFuNC6u0mJHAeOnJngbmSgoA@mail.gmail.com>
	<CAL8ieHoNbMyUrzZnincgNqPmT9Db4pXM7e-xtaXU4q27The9sw@mail.gmail.com>
Message-ID: <CAL8ieHpY97JwhWi1jWVfH8aDSf9OggckDpRUvY3NnwVk-_=L0w@mail.gmail.com>

For now, I have just added my own identifier back to the ssl check, so I
can stay on master with the Heartbleed code.  Maybe by the time I run
another update from git this will have been fixed and losing my own changes
will be irrelevant.

Thank you!


On Fri, Jun 6, 2014 at 9:03 AM, sangdrax8 <sangdrax8 at gmail.com> wrote:

> I was just trying to move back from the heartbleed branch in git to the
> current "stable."  Should I be checking out something other than master to
> make the move back from the heartbleed branch to stable branch?
>
>
> On Fri, Jun 6, 2014 at 8:54 AM, Josh Liburdi <liburdi.joshua at gmail.com>
> wrote:
>
>> Looks to me like the $identifer field was dropped from those notices
>> with the move to 2.3 ...
>>
>> Bro 2.2:
>>
>> else if ( cert$not_valid_after < network_time() )
>> NOTICE([$note=Certificate_Expired,
>>        $conn=c, $suppress_for=1day,
>>        $msg=fmt("Certificate %s expired at %T", cert$subject,
>> cert$not_valid_after),
>>        $identifier=cat(c$id$resp_h, c$id$resp_p, c$ssl$cert_hash)]);
>>
>>
>> Bro 2.3:
>>
>> else if ( cert$not_valid_after < network_time() )
>> NOTICE([$note=Certificate_Expired,
>>        $conn=c, $suppress_for=1day,
>>        $msg=fmt("Certificate %s expired at %T", cert$subject,
>> cert$not_valid_after),
>>        $fuid=fuid]);
>>
>>
>> That will break suppression.
>>
>> -Josh
>>
>> On Fri, Jun 6, 2014 at 8:35 AM, sangdrax8 <sangdrax8 at gmail.com> wrote:
>> > I am having some problems (or maybe misunderstanding) of how the
>> suppression
>> > works. I haven't changed my configuration file and it was working at one
>> > time.  Now after upgrading to the master branch (I was on the
>> heartbleed) it
>> > seems my suppression isn't working as I understand it.
>> >
>> > I have activated the SSL certificate checking as follows:
>> > @load policy/protocols/ssl/expiring-certs.bro
>> > redef SSL::notify_certs_expiration = ALL_HOSTS;
>> >
>> > now when I watch my notice log, I am seeing what appear to be LOTS of
>> notice
>> > logs for the same certificate.  I thought that perhaps just the e-mails
>> get
>> > suppressed, but after turning on e-mail notifications I get an e-mail
>> for
>> > every notice.  Plus my notice log is filling up rather quickly.
>> >
>> > I know this probably won't be very legible, but here is an example of
>> just 2
>> > of the notices I get from a single connection.  They look exactly the
>> same
>> > to me, and they have a time set for the suppression. I  would have
>> expected
>> > to only get one of these, but you can see the time stamp shows multiple
>> > notices happening very quickly.
>> >
>> > #fields ts      uid     id.orig_h       id.orig_p       id.resp_h
>> > id.resp_p       fuid    file_mime_type  file_desc       proto   note
>>  msg
>> > sub     src     dst     p       n       peer_descr      actions
>> suppress_for
>> > dropped remote_location.country_code    remote_location.region
>> > remote_location.city    remote_location.latitude
>> > remote_location.longitude
>> >
>> >
>> > 1402057564.658489       CW6Riz4smTIRpMxWq1      1.1.1.1   51255
>> 2.2.2.2
>> > 5223    F6irMUcwkf1ZcbIok       -       -       tcp
>> > SSL::Certificate_Expired        Certificate emailAddress=,CN=,OU=,O= -
>> > 1.1.1.1   2.2.2.2  5223    -       bro1      Notice::ACTION_LOG
>> > 86400.000000    F       -       -       -       -       -
>> >
>> > 1402057564.660035       CW6Riz4smTIRpMxWq1      1.1.1.1   51255
>> 2.2.2.2
>> > 5223    F6irMUcwkf1ZcbIok       -       -       tcp
>> > SSL::Certificate_Expired        Certificate emailAddress=,CN=,OU=,O= -
>> > 1.1.1.1   2.2.2.2  5223    -       bro1      Notice::ACTION_LOG
>> > 86400.000000    F       -       -       -       -       -
>> >
>> >
>> >
>> > _______________________________________________
>> > Bro mailing list
>> > bro at bro-ids.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140606/812838df/attachment.html 

From bernhard at ICSI.Berkeley.EDU  Fri Jun  6 09:51:01 2014
From: bernhard at ICSI.Berkeley.EDU (bernhard at ICSI.Berkeley.EDU)
Date: Fri, 06 Jun 2014 09:51:01 -0700
Subject: [Bro] Suppress_for issues
In-Reply-To: <CANrCRiHu43NGVc6SBvpk4BWZYtVFuNC6u0mJHAeOnJngbmSgoA@mail.gmail.com>
References: <CAL8ieHrr6MHe19oxAZ=pK7V4c3qmCKjVUbj7bMmrTCOLRPAcqQ@mail.gmail.com>
	<CANrCRiHu43NGVc6SBvpk4BWZYtVFuNC6u0mJHAeOnJngbmSgoA@mail.gmail.com>
Message-ID: <7E7E6567-BD1E-4911-BF2D-7FAAD29175AB@icsi.berkeley.edu>

Oh, yes, sorry, I probably did that on accident while moving to file 
IDs. I guess we should add the suppression back in, I will try to take a 
look at it later and hopefully it will be back in the 2.3 release...

Bernhard

On 6 Jun 2014, at 5:54, Josh Liburdi wrote:

> Looks to me like the $identifer field was dropped from those notices
> with the move to 2.3 ...
>
> Bro 2.2:
>
> else if ( cert$not_valid_after < network_time() )
> NOTICE([$note=Certificate_Expired,
>     $conn=c, $suppress_for=1day,
>     $msg=fmt("Certificate %s expired at %T", cert$subject,
> cert$not_valid_after),
>     $identifier=cat(c$id$resp_h, c$id$resp_p, c$ssl$cert_hash)]);
>
>
> Bro 2.3:
>
> else if ( cert$not_valid_after < network_time() )
> NOTICE([$note=Certificate_Expired,
>     $conn=c, $suppress_for=1day,
>     $msg=fmt("Certificate %s expired at %T", cert$subject,
> cert$not_valid_after),
>     $fuid=fuid]);
>
>
> That will break suppression.
>
> -Josh
>
> On Fri, Jun 6, 2014 at 8:35 AM, sangdrax8 <sangdrax8 at gmail.com> wrote:
>> I am having some problems (or maybe misunderstanding) of how the 
>> suppression
>> works. I haven't changed my configuration file and it was working at 
>> one
>> time.  Now after upgrading to the master branch (I was on the 
>> heartbleed) it
>> seems my suppression isn't working as I understand it.
>>
>> I have activated the SSL certificate checking as follows:
>> @load policy/protocols/ssl/expiring-certs.bro
>> redef SSL::notify_certs_expiration = ALL_HOSTS;
>>
>> now when I watch my notice log, I am seeing what appear to be LOTS of 
>> notice
>> logs for the same certificate.  I thought that perhaps just the 
>> e-mails get
>> suppressed, but after turning on e-mail notifications I get an e-mail 
>> for
>> every notice.  Plus my notice log is filling up rather quickly.
>>
>> I know this probably won't be very legible, but here is an example of 
>> just 2
>> of the notices I get from a single connection.  They look exactly the 
>> same
>> to me, and they have a time set for the suppression. I  would have 
>> expected
>> to only get one of these, but you can see the time stamp shows 
>> multiple
>> notices happening very quickly.
>>
>> #fields ts      uid     id.orig_h       id.orig_p       id.resp_h
>> id.resp_p       fuid    file_mime_type  file_desc       proto   note  
>>   msg
>> sub     src     dst     p       n       peer_descr      actions 
>> suppress_for
>> dropped remote_location.country_code    remote_location.region
>> remote_location.city    remote_location.latitude
>> remote_location.longitude
>>
>>
>> 1402057564.658489       CW6Riz4smTIRpMxWq1      1.1.1.1   51255   
>> 2.2.2.2
>> 5223    F6irMUcwkf1ZcbIok       -       -       tcp
>> SSL::Certificate_Expired        Certificate emailAddress=,CN=,OU=,O= 
>> -
>> 1.1.1.1   2.2.2.2  5223    -       bro1      Notice::ACTION_LOG
>> 86400.000000    F       -       -       -       -       -
>>
>> 1402057564.660035       CW6Riz4smTIRpMxWq1      1.1.1.1   51255   
>> 2.2.2.2
>> 5223    F6irMUcwkf1ZcbIok       -       -       tcp
>> SSL::Certificate_Expired        Certificate emailAddress=,CN=,OU=,O= 
>> -
>> 1.1.1.1   2.2.2.2  5223    -       bro1      Notice::ACTION_LOG
>> 86400.000000    F       -       -       -       -       -
>>
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From dn1nj4 at gmail.com  Mon Jun  9 00:59:55 2014
From: dn1nj4 at gmail.com (Jason)
Date: Mon, 9 Jun 2014 03:59:55 -0400
Subject: [Bro] Split PCAPs & Partial Connections
In-Reply-To: <734AFE64-7A1C-4E9B-BCD2-49E80EE5F5B6@gmail.com>
References: <734AFE64-7A1C-4E9B-BCD2-49E80EE5F5B6@gmail.com>
Message-ID: <CADBniYg+a7iFcJk9rgxB2vXos+KhLC6g4wW0nEnh21OhWTxNuw@mail.gmail.com>

Has anyone here run into this problem before?  It seems to be the same for
http.log files as well.

Thanks!
Jason


On Fri, Jun 6, 2014 at 8:34 AM, dn1nj4 <dn1nj4 at gmail.com> wrote:

> Hello all,
>
> I am batch processing some periodic PCAP files to extract SSL
> certificates. I noticed if I do bro -r file1, followed by bro -r file2, I
> end up with two ssl.log files totalling ~1500 lines. However, if I mergecap
> file1 & file2 and run bro -r merged, my ssl.log is ~7000 lines.
>
> After searching the list archive and current bro docs, I thought this
> might be a partial connection problem. So I tried a redef
> partial_connection_ok = T; but that did not seem to have any effect.
>
> Any insights into how I might fix the discrepancy would be greatly
> appreciated.
>
> Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140609/5501c4c0/attachment.html 

From shane.castle at gmail.com  Mon Jun  9 02:17:08 2014
From: shane.castle at gmail.com (Shane Castle)
Date: Mon, 09 Jun 2014 11:17:08 +0200
Subject: [Bro] Split PCAPs & Partial Connections
In-Reply-To: <CADBniYg+a7iFcJk9rgxB2vXos+KhLC6g4wW0nEnh21OhWTxNuw@mail.gmail.com>
References: <734AFE64-7A1C-4E9B-BCD2-49E80EE5F5B6@gmail.com>
	<CADBniYg+a7iFcJk9rgxB2vXos+KhLC6g4wW0nEnh21OhWTxNuw@mail.gmail.com>
Message-ID: <53957B94.4030707@gmail.com>

A few questions arise:

How did you run mergecap? Does the behavior change if you run it using
-a? Do the two pcap files immediately follow one another in timestamps,
that is, does file2 start exactly where file1 stops? Is the merged pcap
file about equal to size(file1) plus size(file2), or is it significantly
larger?

-- 
Mit besten Gr??en
Shane Castle

On 09.06.2014 09:59, Jason wrote:
> Has anyone here run into this problem before?  It seems to be the same
> for http.log files as well.
> 
> Thanks!
> Jason
> 
> 
> On Fri, Jun 6, 2014 at 8:34 AM, dn1nj4 <dn1nj4 at gmail.com
> <mailto:dn1nj4 at gmail.com>> wrote:
> 
>     Hello all,
> 
>     I am batch processing some periodic PCAP files to extract SSL
>     certificates. I noticed if I do bro -r file1, followed by bro -r
>     file2, I end up with two ssl.log files totalling ~1500 lines.
>     However, if I mergecap file1 & file2 and run bro -r merged, my
>     ssl.log is ~7000 lines.
> 
>     After searching the list archive and current bro docs, I thought
>     this might be a partial connection problem. So I tried a redef
>     partial_connection_ok = T; but that did not seem to have any effect.
> 
>     Any insights into how I might fix the discrepancy would be greatly
>     appreciated.
> 
>     Jason
> 
> 
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 


From dn1nj4 at gmail.com  Mon Jun  9 03:00:37 2014
From: dn1nj4 at gmail.com (Jason)
Date: Mon, 9 Jun 2014 06:00:37 -0400
Subject: [Bro] Split PCAPs & Partial Connections
In-Reply-To: <53957B94.4030707@gmail.com>
References: <734AFE64-7A1C-4E9B-BCD2-49E80EE5F5B6@gmail.com>
	<CADBniYg+a7iFcJk9rgxB2vXos+KhLC6g4wW0nEnh21OhWTxNuw@mail.gmail.com>
	<53957B94.4030707@gmail.com>
Message-ID: <CADBniYjGMDkjJZqo_metiuaEtuiQpXXvyS7bAs3NP1Pc8cbH+g@mail.gmail.com>

Shane,

I ran mergecap -w merged.cap file1 file2.  When run with -a, merged.cap is
the same file size, but bro results in only 700 log lines in ssl.log.  The
two PCAP files are an overlapping timeframe.  Traffic is split across
both.  Yes, the merged file size is approximately the same as the sum of
the individual files.

Jason


On Mon, Jun 9, 2014 at 5:17 AM, Shane Castle <shane.castle at gmail.com> wrote:

> A few questions arise:
>
> How did you run mergecap? Does the behavior change if you run it using
> -a? Do the two pcap files immediately follow one another in timestamps,
> that is, does file2 start exactly where file1 stops? Is the merged pcap
> file about equal to size(file1) plus size(file2), or is it significantly
> larger?
>
> --
> Mit besten Gr??en
> Shane Castle
>
> On 09.06.2014 09:59, Jason wrote:
> > Has anyone here run into this problem before?  It seems to be the same
> > for http.log files as well.
> >
> > Thanks!
> > Jason
> >
> >
> > On Fri, Jun 6, 2014 at 8:34 AM, dn1nj4 <dn1nj4 at gmail.com
> > <mailto:dn1nj4 at gmail.com>> wrote:
> >
> >     Hello all,
> >
> >     I am batch processing some periodic PCAP files to extract SSL
> >     certificates. I noticed if I do bro -r file1, followed by bro -r
> >     file2, I end up with two ssl.log files totalling ~1500 lines.
> >     However, if I mergecap file1 & file2 and run bro -r merged, my
> >     ssl.log is ~7000 lines.
> >
> >     After searching the list archive and current bro docs, I thought
> >     this might be a partial connection problem. So I tried a redef
> >     partial_connection_ok = T; but that did not seem to have any effect.
> >
> >     Any insights into how I might fix the discrepancy would be greatly
> >     appreciated.
> >
> >     Jason
> >
> >
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140609/30cd5c32/attachment.html 

From darrylm at gmail.com  Mon Jun  9 07:58:17 2014
From: darrylm at gmail.com (Darryl MacLeod)
Date: Mon, 9 Jun 2014 11:58:17 -0300
Subject: [Bro] Network traffic data measurement
Message-ID: <CAMcnPVofMYKO+8okhYyzoH9y-TB5tnkxvagGppc5-AZNRKm-Wg@mail.gmail.com>

Hi All,

I'm relatively new to Bro and have been tasked with determining various
network traffic benchmarks.

For example,

Maximum connections per second
Maximum connections
Maximum throughput (Mbps)
Maximum IMIX throughput
Maximum concurrent TCP sessions
Expected latency when expecting traffic

Is this possible with Bro? If so, can it be done using the default scripts
or will it require custom scripting?

Thanks in advance!


Darryl MacLeod
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140609/4c6ec612/attachment.html 

From seth at icir.org  Mon Jun  9 08:30:25 2014
From: seth at icir.org (Seth Hall)
Date: Mon, 9 Jun 2014 11:30:25 -0400
Subject: [Bro] Network traffic data measurement
In-Reply-To: <CAMcnPVofMYKO+8okhYyzoH9y-TB5tnkxvagGppc5-AZNRKm-Wg@mail.gmail.com>
References: <CAMcnPVofMYKO+8okhYyzoH9y-TB5tnkxvagGppc5-AZNRKm-Wg@mail.gmail.com>
Message-ID: <4C67EDD8-A0A7-46C4-AAC1-FB9D360AA4CD@icir.org>


On Jun 9, 2014, at 10:58 AM, Darryl MacLeod <darrylm at gmail.com> wrote:

> Maximum connections per second
> Maximum connections
> Maximum throughput (Mbps)
> Maximum IMIX throughput
> Maximum concurrent TCP sessions

You could calculate most or all of this with scripts currently.  I don't know of any existing scripts that will do this for you though (and you'll probably want to look into SumStats).  You'll have to write them. :)

> Expected latency when expecting traffic

I think we had this book before our first child...
	http://www.amazon.com/What-Expect-Youre-Expecting-Edition/dp/0761148574

(there isn't a script for that either)

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140609/8b8a3e4f/attachment.bin 

From phucahwa at gmail.com  Mon Jun  9 08:35:37 2014
From: phucahwa at gmail.com (Ohhello Nicetomeetu)
Date: Mon, 9 Jun 2014 11:35:37 -0400
Subject: [Bro] smtp.log field
Message-ID: <CAByaECWKHjt+xr+6P3acdEAf11T+qd9sNW+re7berrQ4CiR4dg@mail.gmail.com>

Hi all,

Im a new BRO user and still have much to learn. Looking at the log files
(http.log and smtp.log), i've noticed some of the fields have the "-" unset
value. Particularly, im looking at the smtp.log file and noticed the
"x-originating-ip" field has unset value. What is the correct way to have
this value correctly logged.

Thanks in advance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140609/da86a2e7/attachment.html 

From darrylm at gmail.com  Mon Jun  9 09:13:26 2014
From: darrylm at gmail.com (Darryl MacLeod)
Date: Mon, 9 Jun 2014 13:13:26 -0300
Subject: [Bro] Network traffic data measurement
In-Reply-To: <4C67EDD8-A0A7-46C4-AAC1-FB9D360AA4CD@icir.org>
References: <CAMcnPVofMYKO+8okhYyzoH9y-TB5tnkxvagGppc5-AZNRKm-Wg@mail.gmail.com>
	<4C67EDD8-A0A7-46C4-AAC1-FB9D360AA4CD@icir.org>
Message-ID: <CAMcnPVrSXuuoLO24WpTbjCuNrSO9nC0HjRX-PrbBa16FO=U=4Q@mail.gmail.com>

Seth,

We had the same book. ;)

I assumed I would have to write the scripts.. just thought I would verify
that.

Thanks.

Darryl


On Mon, Jun 9, 2014 at 12:30 PM, Seth Hall <seth at icir.org> wrote:

>
> On Jun 9, 2014, at 10:58 AM, Darryl MacLeod <darrylm at gmail.com> wrote:
>
> > Maximum connections per second
> > Maximum connections
> > Maximum throughput (Mbps)
> > Maximum IMIX throughput
> > Maximum concurrent TCP sessions
>
> You could calculate most or all of this with scripts currently.  I don't
> know of any existing scripts that will do this for you though (and you'll
> probably want to look into SumStats).  You'll have to write them. :)
>
> > Expected latency when expecting traffic
>
> I think we had this book before our first child...
>
> http://www.amazon.com/What-Expect-Youre-Expecting-Edition/dp/0761148574
>
> (there isn't a script for that either)
>
>  .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140609/98f6a246/attachment.html 

From damonrouse at gmail.com  Mon Jun  9 11:14:24 2014
From: damonrouse at gmail.com (Damon Rouse)
Date: Mon, 9 Jun 2014 11:14:24 -0700
Subject: [Bro] Intel Framework Usage
Message-ID: <CAHoWat7BNwnNs4o6xcB+zoXyfxmWoBx97fJO9oXB6Njgwy3=tA@mail.gmail.com>

Hi Everyone

I'm still pretty new BRO and have a couple questions about the Intel
framework and how to use/leverage it.  I've looked through the docs and
have it loaded in local.bro file.  I don't see an intel.log in my current
or my BRO archive logs (previous dates) directories.  Correct to assume
this means that there have been no hits to the Intel framework?

Secondly, is the Input framework the correct way to scan against intel data
I have internally or obtain from other sources?  If so, do these text files
need to be formatted a certain way like the pre-formatted feeds mentioned
in the docs (mal-dns2bro and CIF)?

Really appreciate the help...Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140609/09989b43/attachment.html 

From kim at blackcatsec.net  Mon Jun  9 11:33:13 2014
From: kim at blackcatsec.net (Kim Halavakoski)
Date: Mon, 9 Jun 2014 21:33:13 +0300
Subject: [Bro] Intel Framework Usage
In-Reply-To: <CAHoWat7BNwnNs4o6xcB+zoXyfxmWoBx97fJO9oXB6Njgwy3=tA@mail.gmail.com>
References: <CAHoWat7BNwnNs4o6xcB+zoXyfxmWoBx97fJO9oXB6Njgwy3=tA@mail.gmail.com>
Message-ID: <8AF62463-6D87-43D4-92F3-BADAFA72753E@blackcatsec.net>

Hi,
I had the same problem when I started using the Intel Framework and in my case it was a simple case of forgetting to install the configurations. Try running the following commands with broctl:

"broctl check && broctl install && broctl restart?

Run those and see if you get any hits when connecting to some known BAD IP-address from the lists you use, the intel.log should be created by Bro if there is match between the seen traffic and your Intel-feeds.


You can easily match against your own data, just remember to format the data with tab-separated values, just as outlined in http://blog.bro.org/2014/01/intelligence-data-and-bro_4980.html :

> Sample file output:
> 
> #fields indicator       indicator_type  meta.source     meta.url        meta.do_notice  meta.if_in
> advanbusiness.com       Intel::DOMAIN   mandiant        -       F       -
> aoldaily.com    Intel::DOMAIN   mandiant        -       F       -
> aolon1ine.com   Intel::DOMAIN   mandiant        -       F       -
> applesoftupdate.com     Intel::DOMAIN   mandiant        -       F       -
> 
> In the example above, mal-dns2bro reads in the mandiant list from stdin and sets the indicator type (``-T'') to DNS because the mandiant list consists of only DNS names. The source (``-s'') field is also set which is a short description of where the intelligence data came from.
> 
> mal-dns2bro will add the necessary tab separated columns for the Intel Framework. It accepts a list of a specific indicator type, but supports all of them, with one entry per line. It can read from stdin or from a file (``-f''). If you don't want to use mal-dnssearch, you can create your own lists with a text editor or other program and have mal-dns2bro format them for Bro.
> 

Kim Halavakoski
kim at blackcatsec.net



On 09 Jun 2014, at 21:14, Damon Rouse <damonrouse at gmail.com> wrote:

> Hi Everyone
> 
> I'm still pretty new BRO and have a couple questions about the Intel framework and how to use/leverage it.  I've looked through the docs and have it loaded in local.bro file.  I don't see an intel.log in my current or my BRO archive logs (previous dates) directories.  Correct to assume this means that there have been no hits to the Intel framework?
> 
> Secondly, is the Input framework the correct way to scan against intel data I have internally or obtain from other sources?  If so, do these text files need to be formatted a certain way like the pre-formatted feeds mentioned in the docs (mal-dns2bro and CIF)?
> 
> Really appreciate the help...Thanks!
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From hckim at narusec.com  Wed Jun 11 01:58:59 2014
From: hckim at narusec.com (=?UTF-8?B?6rmA7Z2s7LKg?=)
Date: Wed, 11 Jun 2014 17:58:59 +0900
Subject: [Bro] http.log mime_type miss match
Message-ID: <CAHs13Agty5sdvBC712RV=gvzra8YHdsuGvw+xdLOhURip30EEQ@mail.gmail.com>

Hi every one

I am testing some of the music streaming service and found one of them miss
matching mime_type compare to Wireshark.

wireshark's type is audio/mpeg  but bro http.log is text/plain

is it bro2.1 is outputting wrong type or do I have to write custom bro
script?



this is the bro 2.1 http.log :

Jun 10 09:20:29 6a7HgNEZlOb 192.X.X.X 53796 8.20.213.33 80 1 POST
stream66c-he.grooveshark.com /stream.php
http://grooveshark.com/static/JSQueue_20140421162423.swf Mozilla/5.0
(Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/35.0.1916.114 Safari/537.36 104 10296924 200 OK - - - (empty) - - -
text/plain - -


this is the Wireshark screenshot link

http://ex.narusec.com/data/public/2b0474.php


and this is the pcap file (it's about 733MB)

http://ex.narusec.com/data/public/23fac8.php
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140611/c4bbce47/attachment.html 

From liburdi.joshua at gmail.com  Wed Jun 11 04:59:37 2014
From: liburdi.joshua at gmail.com (Josh Liburdi)
Date: Wed, 11 Jun 2014 07:59:37 -0400
Subject: [Bro] http.log mime_type miss match
In-Reply-To: <CAHs13Agty5sdvBC712RV=gvzra8YHdsuGvw+xdLOhURip30EEQ@mail.gmail.com>
References: <CAHs13Agty5sdvBC712RV=gvzra8YHdsuGvw+xdLOhURip30EEQ@mail.gmail.com>
Message-ID: <CANrCRiGsNUDJ2LJaME8jVx-6gtoxETkH3fQ2QSL3Q8Xok6UNkA@mail.gmail.com>

I'm not familiar with how Wireshark determines mime type, but that
version of Bro uses the libmagic database and primarily determines
mime type by searching for a magic number in the first 1024 bytes of
the file. It's possible that Bro incorrectly determined the mime type.

Changes were made in Bro 2.3 to move away from the libmagic db.
Hopefully this means that, in the future, mime type determination will
be more accurate / extensible. Check the release notes for more info
on that: http://www.bro.org/sphinx-git/install/release-notes.html

-Josh

On Wed, Jun 11, 2014 at 4:58 AM, ??? <hckim at narusec.com> wrote:
> Hi every one
>
> I am testing some of the music streaming service and found one of them miss
> matching mime_type compare to Wireshark.
>
> wireshark's type is audio/mpeg  but bro http.log is text/plain
>
> is it bro2.1 is outputting wrong type or do I have to write custom bro
> script?
>
>
>
> this is the bro 2.1 http.log :
>
> Jun 10 09:20:29 6a7HgNEZlOb 192.X.X.X 53796 8.20.213.33 80 1 POST
> stream66c-he.grooveshark.com /stream.php
> http://grooveshark.com/static/JSQueue_20140421162423.swf Mozilla/5.0
> (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko)
> Chrome/35.0.1916.114 Safari/537.36 104 10296924 200 OK - - - (empty) - - -
> text/plain - -
>
>
> this is the Wireshark screenshot link
>
> http://ex.narusec.com/data/public/2b0474.php
>
>
> and this is the pcap file (it's about 733MB)
>
> http://ex.narusec.com/data/public/23fac8.php
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From bkellogg at dresser-rand.com  Wed Jun 11 14:25:51 2014
From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN))
Date: Wed, 11 Jun 2014 21:25:51 +0000
Subject: [Bro] Bro hanging on some sensors
Message-ID: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>

I have several SecurityOnion sensors and most are working ok.  There are a couple that I see the below problem on with Bro.

The /nsm/bro/spool/manager/communication.log file shows the below in it on each of the problem sensors:
    1402520922.886012   manager parent  -       -       -       info    warning: cannot increase SO_SNDBUF socket buffer size from 16384K (1024K was requested)
    1402520922.886012   manager parent  -       -       -       info    warning: cannot increase SO_RCVBUF socket buffer size from 16384K (1024K was requested)
    1402520922.886012   manager parent  -       -       -       info    warning: cannot increase SO_SNDBUF socket buffer size from 16384K (1024K was requested)
    1402520922.886012   manager parent  -       -       -       info    warning: cannot increase SO_RCVBUF socket buffer size from 16384K (1024K was requested)
    1402520922.886012   manager parent  -       -       -       info    communication started, parent pid is 3646, child pid is 3660

I only ever see these files created in the Bro log working directory:
    communication.log
    loaded_scripts.log
    eporter.log
    stderr.log
    stdout.log

Most of the sensors are configured exactly the same both software and hardware wise; so I'm not seeing a correlation there as yet.  I've tried rebooting and using broctl commands and so far no resolution.  Many time "broctl check" will hang.  I have all the latest patches on SO installed.  Any help would be appreciated.

The only major change I've made in the last month is to add a few Intel feeds.


Thanks,
Brian




From doug.burks at gmail.com  Thu Jun 12 02:23:41 2014
From: doug.burks at gmail.com (Doug Burks)
Date: Thu, 12 Jun 2014 05:23:41 -0400
Subject: [Bro] Bro hanging on some sensors
In-Reply-To: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
References: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
Message-ID: <CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>

Hi Brian,

How exactly did you add your Intel feeds?

On Wed, Jun 11, 2014 at 5:25 PM, Kellogg, Brian D (OLN)
<bkellogg at dresser-rand.com> wrote:
> I have several SecurityOnion sensors and most are working ok.  There are a couple that I see the below problem on with Bro.
>
> The /nsm/bro/spool/manager/communication.log file shows the below in it on each of the problem sensors:
>     1402520922.886012   manager parent  -       -       -       info    warning: cannot increase SO_SNDBUF socket buffer size from 16384K (1024K was requested)
>     1402520922.886012   manager parent  -       -       -       info    warning: cannot increase SO_RCVBUF socket buffer size from 16384K (1024K was requested)
>     1402520922.886012   manager parent  -       -       -       info    warning: cannot increase SO_SNDBUF socket buffer size from 16384K (1024K was requested)
>     1402520922.886012   manager parent  -       -       -       info    warning: cannot increase SO_RCVBUF socket buffer size from 16384K (1024K was requested)
>     1402520922.886012   manager parent  -       -       -       info    communication started, parent pid is 3646, child pid is 3660
>
> I only ever see these files created in the Bro log working directory:
>     communication.log
>     loaded_scripts.log
>     eporter.log
>     stderr.log
>     stdout.log
>
> Most of the sensors are configured exactly the same both software and hardware wise; so I'm not seeing a correlation there as yet.  I've tried rebooting and using broctl commands and so far no resolution.  Many time "broctl check" will hang.  I have all the latest patches on SO installed.  Any help would be appreciated.
>
> The only major change I've made in the last month is to add a few Intel feeds.
>
>
> Thanks,
> Brian
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
Doug Burks



From bkellogg at dresser-rand.com  Thu Jun 12 05:45:08 2014
From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN))
Date: Thu, 12 Jun 2014 12:45:08 +0000
Subject: [Bro] Bro hanging on some sensors
In-Reply-To: <CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
References: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
Message-ID: <BDF3793B557F624699D4C68A3B8FCF821818D3@DRTEXW238VM.na.d-rco.com>

A non-text attachment was scrubbed...
Name: smime.p7m
Type: application/pkcs7-mime
Size: 7687 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140612/fe1dd096/attachment.p7c 

From bkellogg at dresser-rand.com  Thu Jun 12 06:00:57 2014
From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN))
Date: Thu, 12 Jun 2014 13:00:57 +0000
Subject: [Bro] Bro hanging on some sensors
In-Reply-To: <CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
References: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
Message-ID: <BDF3793B557F624699D4C68A3B8FCF82181922@DRTEXW238VM.na.d-rco.com>

I use a cron job that runs every 30 minutes to download the intel files to:
/opt/bro/share/bro/policy/.  The cron job uses the mal-dnssearch script.

In each sensor's /opt/bro/share/bro/site/local.bro file I have the below:

# load intelligence framework
@load policy/frameworks/intel/seen
@load policy/frameworks/intel/do_notice
#@load policy/integration/collective-intel
redef Intel::read_files += {
        "/opt/bro/share/bro/policy/xxx.intel",
        "/opt/bro/share/bro/policy/xxx.intel",
        "/opt/bro/share/bro/policy/xxx.intel",
        "/opt/bro/share/bro/policy/xxx.intel",
        "/opt/bro/share/bro/policy/xxx.intel",
        "/opt/bro/share/bro/policy/xxx.intel",
};

In the reporter.log file I am now seeing the below warning on the four sensors
having this issue:
    0.000000    Reporter::WARNING       SumStat key request for the
3Mntn3EPhU3 SumStat uid took longer than 1 minute and was automatically
cancelled.  /opt/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line
209
    0.000000    Reporter::WARNING       SumStat key request for the
2HAva5N4Kqf SumStat uid took longer than 1 minute and was automatically
cancelled.  /opt/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line
209

Thanks,
Brian



From brianallen at wustl.edu  Thu Jun 12 07:38:24 2014
From: brianallen at wustl.edu (Allen, Brian)
Date: Thu, 12 Jun 2014 14:38:24 +0000
Subject: [Bro] teamcymru alerts
Message-ID: <CFBF21B6.15E5AF%brianallen@wustl.edu>

Hi,

I'm using the TeamCymru alerts in the notice.log file, but I was wondering
if someone could help interpret the specific field: file_desc which is
listed as "Additional context for file, if available"?

Normally that field will list a website, presumably where the file was
downloaded from and it looks like this <sanitized>:

"http://d pullupdate com/ius/Setup.exe"

Which is extremely useful.  But there are some alerts in that field, which
are going to port 25, where it lists email addresses like this:

"<liuyp1952 at 163.com> -> <french_a at kids.wustl.edu>: Mail Delivery
(failur..."

That's the exact content of the field including the ... at the end.

I assume this is an alert for an email that went from that 163.com account
to the french_a at kids account and the malicious file BRO detected was
attached.  

Do I have this right?  What about that Mail Delivery (failur... at the end?
Thanks,
-Brian

Brian Allen
Network Security Analyst
Washington University





-----Original Message-----
From: <Kellogg>, "Brian D   (OLN)" <bkellogg at dresser-rand.com>
Date: Wednesday, June 11, 2014 4:25 PM
To: bro <bro at bro.org>
Subject: [Bro] Bro hanging on some sensors

I have several SecurityOnion sensors and most are working ok.  There are a
couple that I see the below problem on with Bro.

The /nsm/bro/spool/manager/communication.log file shows the below in it on
each of the problem sensors:
    1402520922.886012   manager parent  -       -       -       info
warning: cannot increase SO_SNDBUF socket buffer size from 16384K (1024K
was requested)
    1402520922.886012   manager parent  -       -       -       info
warning: cannot increase SO_RCVBUF socket buffer size from 16384K (1024K
was requested)
    1402520922.886012   manager parent  -       -       -       info
warning: cannot increase SO_SNDBUF socket buffer size from 16384K (1024K
was requested)
    1402520922.886012   manager parent  -       -       -       info
warning: cannot increase SO_RCVBUF socket buffer size from 16384K (1024K
was requested)
    1402520922.886012   manager parent  -       -       -       info
communication started, parent pid is 3646, child pid is 3660

I only ever see these files created in the Bro log working directory:
    communication.log
    loaded_scripts.log
    eporter.log
    stderr.log
    stdout.log

Most of the sensors are configured exactly the same both software and
hardware wise; so I'm not seeing a correlation there as yet.  I've tried
rebooting and using broctl commands and so far no resolution.  Many time
"broctl check" will hang.  I have all the latest patches on SO installed.
Any help would be appreciated.

The only major change I've made in the last month is to add a few Intel
feeds.


Thanks,
Brian


_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From seth at icir.org  Thu Jun 12 08:02:00 2014
From: seth at icir.org (Seth Hall)
Date: Thu, 12 Jun 2014 11:02:00 -0400
Subject: [Bro] teamcymru alerts
In-Reply-To: <CFBF21B6.15E5AF%brianallen@wustl.edu>
References: <CFBF21B6.15E5AF%brianallen@wustl.edu>
Message-ID: <EDCC3490-B22D-4FCE-AF08-772AD9C0F5EC@icir.org>


On Jun 12, 2014, at 10:38 AM, Allen, Brian <BrianAllen at wustl.edu> wrote:

> I assume this is an alert for an email that went from that 163.com account
> to the french_a at kids account and the malicious file BRO detected was
> attached.  

Yep.

> Do I have this right?  What about that Mail Delivery (failur... at the end?

The "..." indicates that the content was trimmed.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140612/37df38c0/attachment.bin 

From gfaulkner.nsm at gmail.com  Thu Jun 12 12:23:27 2014
From: gfaulkner.nsm at gmail.com (Gary Faulkner)
Date: Thu, 12 Jun 2014 14:23:27 -0500
Subject: [Bro] RMCP+ / IPMI
Message-ID: <5399FE2F.7060907@gmail.com>

Has anyone written any scripts to analyze / detect remote connections to 
out of band management solutions such as IPMI over RMCP+ or similar 
protocols? The aim of this would be to potentially look for unsecured 
services and/or suspicious access.

Regards,
Gary


From doug.burks at gmail.com  Fri Jun 13 03:55:33 2014
From: doug.burks at gmail.com (Doug Burks)
Date: Fri, 13 Jun 2014 06:55:33 -0400
Subject: [Bro] Bro hanging on some sensors
In-Reply-To: <BDF3793B557F624699D4C68A3B8FCF82181922@DRTEXW238VM.na.d-rco.com>
References: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82181922@DRTEXW238VM.na.d-rco.com>
Message-ID: <CAK8kjrDry05X3RBCt7Ph9SgQa13__+rjg28=VZ9KmPP3KgyYqw@mail.gmail.com>

Did these sensors work correctly before you added the Intel files?

Have you tried removing the Intel files from the equation?  If not,
please try commenting out the entries you added to
/opt/bro/share/bro/site/local.bro and then running:
sudo broctl install
sudo broctl restart

Are you running OnionSalt?  Do you have OnionSalt configured to
automatically restart Bro?  There are known issues with doing so,
which is why OnionSalt does not do so by default.  Please see:
https://code.google.com/p/security-onion/wiki/Salt#Features

This may not be an issue strictly related to Bro, so we may need to
move this conversation to the security-onion mailing list:
https://code.google.com/p/security-onion/wiki/MailingLists

On Thu, Jun 12, 2014 at 9:00 AM, Kellogg, Brian D (OLN)
<bkellogg at dresser-rand.com> wrote:
> I use a cron job that runs every 30 minutes to download the intel files to:
> /opt/bro/share/bro/policy/.  The cron job uses the mal-dnssearch script.
>
> In each sensor's /opt/bro/share/bro/site/local.bro file I have the below:
>
> # load intelligence framework
> @load policy/frameworks/intel/seen
> @load policy/frameworks/intel/do_notice
> #@load policy/integration/collective-intel
> redef Intel::read_files += {
>         "/opt/bro/share/bro/policy/xxx.intel",
>         "/opt/bro/share/bro/policy/xxx.intel",
>         "/opt/bro/share/bro/policy/xxx.intel",
>         "/opt/bro/share/bro/policy/xxx.intel",
>         "/opt/bro/share/bro/policy/xxx.intel",
>         "/opt/bro/share/bro/policy/xxx.intel",
> };
>
> In the reporter.log file I am now seeing the below warning on the four sensors
> having this issue:
>     0.000000    Reporter::WARNING       SumStat key request for the
> 3Mntn3EPhU3 SumStat uid took longer than 1 minute and was automatically
> cancelled.  /opt/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line
> 209
>     0.000000    Reporter::WARNING       SumStat key request for the
> 2HAva5N4Kqf SumStat uid took longer than 1 minute and was automatically
> cancelled.  /opt/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line
> 209
>
> Thanks,
> Brian



-- 
Doug Burks


From bkellogg at dresser-rand.com  Fri Jun 13 05:47:06 2014
From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN))
Date: Fri, 13 Jun 2014 12:47:06 +0000
Subject: [Bro] Bro hanging on some sensors
In-Reply-To: <CAK8kjrDry05X3RBCt7Ph9SgQa13__+rjg28=VZ9KmPP3KgyYqw@mail.gmail.com>
References: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82181922@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDry05X3RBCt7Ph9SgQa13__+rjg28=VZ9KmPP3KgyYqw@mail.gmail.com>
Message-ID: <BDF3793B557F624699D4C68A3B8FCF82182A79@DRTEXW238VM.na.d-rco.com>

A non-text attachment was scrubbed...
Name: smime.p7m
Type: application/pkcs7-mime
Size: 9317 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140613/7aaffaed/attachment.p7c 

From bkellogg at dresser-rand.com  Fri Jun 13 05:59:28 2014
From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN))
Date: Fri, 13 Jun 2014 12:59:28 +0000
Subject: [Bro] Bro hanging on some sensors
In-Reply-To: <CAK8kjrDry05X3RBCt7Ph9SgQa13__+rjg28=VZ9KmPP3KgyYqw@mail.gmail.com>
References: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82181922@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDry05X3RBCt7Ph9SgQa13__+rjg28=VZ9KmPP3KgyYqw@mail.gmail.com>
Message-ID: <BDF3793B557F624699D4C68A3B8FCF82182ADD@DRTEXW238VM.na.d-rco.com>

Yeah they were working up to about a week ago.  I commented out the intel
files on one and that fixed it.  Odd, as they should all be the same with the
replication from the SO server policy directory.  The majority of the sensors
are working fine with the same intel files and I'm seeing new results in ELSA
every day.  Hmmm

I do not have Bro restarting via Onion-Salt.  I was under the understanding
that Bro would periodically sense the file changes and re-read the intel
files.

Thanks,
Brian

Thank you,
Brian Kellogg
Security Analyst; IT Governance, Risk, and Compliance
500 Paul Clark Drive, Olean,  NY 14760
T: (716) 375-3186 | F: (716) 375-3557
www.dresser-rand.com     NYSE: DRC

Bringing energy and the environment into harmony?
IMPORTANT NOTICE:
This email may be confidential, may be legally privileged, and is for the intended recipient only. Unauthorized access, disclosure, copying, distribution, or reliance on any of it by anyone else is prohibited and may be a criminal offense. Please delete if obtained in error and email confirmation to the sender.


-----Original Message-----
From: Doug Burks [mailto:doug.burks at gmail.com]
Sent: Friday, June 13, 2014 6:56 AM
To: Kellogg, Brian D (OLN)
Cc: bro
Subject: Re: [Bro] Bro hanging on some sensors

Did these sensors work correctly before you added the Intel files?

Have you tried removing the Intel files from the equation?  If not, please try commenting out the entries you added to /opt/bro/share/bro/site/local.bro and then running:
sudo broctl install
sudo broctl restart

Are you running OnionSalt?  Do you have OnionSalt configured to automatically restart Bro?  There are known issues with doing so, which is why OnionSalt does not do so by default.  Please see:
https://code.google.com/p/security-onion/wiki/Salt#Features

This may not be an issue strictly related to Bro, so we may need to move this conversation to the security-onion mailing list:
https://code.google.com/p/security-onion/wiki/MailingLists

On Thu, Jun 12, 2014 at 9:00 AM, Kellogg, Brian D (OLN) <bkellogg at dresser-rand.com> wrote:
> I use a cron job that runs every 30 minutes to download the intel files to:
> /opt/bro/share/bro/policy/.  The cron job uses the mal-dnssearch script.
>
> In each sensor's /opt/bro/share/bro/site/local.bro file I have the below:
>
> # load intelligence framework
> @load policy/frameworks/intel/seen
> @load policy/frameworks/intel/do_notice #@load
> policy/integration/collective-intel
> redef Intel::read_files += {
>         "/opt/bro/share/bro/policy/xxx.intel",
>         "/opt/bro/share/bro/policy/xxx.intel",
>         "/opt/bro/share/bro/policy/xxx.intel",
>         "/opt/bro/share/bro/policy/xxx.intel",
>         "/opt/bro/share/bro/policy/xxx.intel",
>         "/opt/bro/share/bro/policy/xxx.intel",
> };
>
> In the reporter.log file I am now seeing the below warning on the four
> sensors having this issue:
>     0.000000    Reporter::WARNING       SumStat key request for the
> 3Mntn3EPhU3 SumStat uid took longer than 1 minute and was
> automatically cancelled.
> /opt/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line
> 209
>     0.000000    Reporter::WARNING       SumStat key request for the
> 2HAva5N4Kqf SumStat uid took longer than 1 minute and was
> automatically cancelled.
> /opt/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line
> 209
>
> Thanks,
> Brian



--
Doug Burks



From jxbatchelor at gmail.com  Fri Jun 13 06:01:54 2014
From: jxbatchelor at gmail.com (Jason Batchelor)
Date: Fri, 13 Jun 2014 08:01:54 -0500
Subject: [Bro] Unexplained Performance Differences Between Like Servers
Message-ID: <CANWz5X8M7gbE4iNRYbtLfJrJzC7g-BtNVS1s7CNCHkxtEO_Y7Q@mail.gmail.com>

Hello everyone:

I have Bro installed on two Dell r720s each with the following specs...

Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz x 32
48GB RAM

Running: CentOs 6.5

Both have the following PF_RING configuration:

PF_RING Version          : 6.0.2 ($Revision: 7746$)
Total rings              : 16
Standard (non DNA) Options
Ring slots               : 32768
Slot version             : 15
Capture TX               : No [RX only]
IP Defragment            : No
Socket Mode              : Standard
Transparent mode         : Yes [mode 0]
Total plugins            : 0
Cluster Fragment Queue   : 1917
Cluster Fragment Discard : 26648
The only difference in PF Ring is the other server (Server A) is going off
revision 7601, where B is rev 7746.

I've tuned the NIC to the following settings...

ethtool -K p4p2 tso off
ethtool -K p4p2 gro off
ethtool -K p4p2 lro off
ethtool -K p4p2 gso off
ethtool -K p4p2 rx off
ethtool -K p4p2 tx off
ethtool -K p4p2 sg off
ethtool -K p4p2 rxvlan off
ethtool -K p4p2 txvlan off
ethtool -N p4p2 rx-flow-hash udp4 sdfn
ethtool -N p4p2 rx-flow-hash udp6 sdfn
ethtool -n p4p2 rx-flow-hash udp6
ethtool -n p4p2 rx-flow-hash udp4
ethtool -C p4p2 rx-usecs 1000
ethtool -C p4p2 adaptive-rx off
ethtool -G p4p2 rx 4096
I've got the following sysctl settings on each.

# turn off selective ACK and timestamps
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0
# memory allocation min/pressure/max.
# read buffer, write buffer, and buffer space
net.ipv4.tcp_rmem = 10000000 10000000 10000000
net.ipv4.tcp_wmem = 10000000 10000000 10000000
net.ipv4.tcp_mem = 10000000 10000000 10000000
net.core.rmem_max = 524287
net.core.wmem_max = 524287
net.core.rmem_default = 524287
net.core.wmem_default = 524287
net.core.optmem_max = 524287
net.core.netdev_max_backlog = 300000
Each bro configuration is using the following...

[manager]
type=manager
host=localhost
[proxy-1]
type=proxy
host=localhost
[worker-1]
type=worker
host=localhost
interface=p4p2
lb_method=pf_ring
lb_procs=16
Both have the same NIC driver version (ixgbe):
3.15.1-k

Same services installed (min install).

Slightly different Kernel versions...
Server A (2.6.32-431.11.2.el6.x86_64)
Server B (2.6.32-431.17.1.el6.x86_64)


At the moment Server A is getting about 700MB/s and Server B is getting
about 600Mb/s.

What I don't understand, is Server A is having several orders of magnatude
better performance compared to Server B?

TOP from A (included a few bro workers):

top - 12:48:45 up 1 day, 17:03,  2 users,  load average: 5.30, 3.99, 3.13
Tasks: 706 total,  19 running, 687 sleeping,   0 stopped,   0 zombie
Cpu(s): 33.9%us,  6.6%sy,  1.1%ni, 57.2%id,  0.0%wa,  0.0%hi,  1.2%si,
0.0%st
Mem:  49376004k total, 33605828k used, 15770176k free,    93100k buffers
Swap:  2621432k total,     9760k used,  2611672k free,  9206880k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5768 root      20   0 1808m 1.7g 519m R 100.0  3.6  32:24.92 bro
 5760 root      20   0 1688m 1.6g 519m R 99.7  3.4  34:08.36 bro
 3314 root      20   0 2160m 269m 4764 R 96.1  0.6  30:14.12 bro
 5754 root      20   0 1451m 1.4g 519m R 82.8  2.9  36:40.02 bro
TOP from B (included a few bro workers)

top - 12:49:33 up 14:24,  2 users,  load average: 10.28, 9.31, 8.06
Tasks: 708 total,  25 running, 683 sleeping,   0 stopped,   0 zombie
Cpu(s): 41.6%us,  6.0%sy,  1.0%ni, 50.4%id,  0.0%wa,  0.0%hi,  1.1%si,
0.0%st
Mem:  49376004k total, 31837340k used, 17538664k free,   147212k buffers
Swap:  2621432k total,        0k used,  2621432k free, 13494332k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3178 root      20   0 1073m 1.0g 264m R 100.0  2.1 401:47.31 bro
 3188 root      20   0  881m 832m 264m R 100.0  1.7 377:48.90 bro
 3189 root      20   0 1247m 1.2g 264m R 100.0  2.5 403:22.95 bro
 3193 root      20   0  920m 871m 264m R 100.0  1.8 429:45.98 bro
Both have the same amount of Bro workers. I just do not understand
why Server A is literally half the utilization on top of seeing more
traffic? The only real and consistent difference between the two I see is
that server A seems to have twice the amount of SHR (shared memory)
compared to server B.

Could this be part of the issue, if not the root cause? How might I go
about rectifying the issue?

FWIW, both are not dropping packets and doing well. However, I want to run
other apps on top of this, and the poor performance on Server B is likely
to have effects on it.

Thanks advance for the advice!

-Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140613/66803369/attachment.html 

From john.h.hoyt at gmail.com  Fri Jun 13 06:49:33 2014
From: john.h.hoyt at gmail.com (John Hoyt)
Date: Fri, 13 Jun 2014 09:49:33 -0400
Subject: [Bro] Unexplained Performance Differences Between Like Servers
In-Reply-To: <CANWz5X8M7gbE4iNRYbtLfJrJzC7g-BtNVS1s7CNCHkxtEO_Y7Q@mail.gmail.com>
References: <CANWz5X8M7gbE4iNRYbtLfJrJzC7g-BtNVS1s7CNCHkxtEO_Y7Q@mail.gmail.com>
Message-ID: <CAOxyatSJmqFOqyVUBiWfW7hFQ2trjCR1=27ABA6D7O+QeaXr+Q@mail.gmail.com>

Hey Jason,

What versions of Bro, and it is the same for both?  I had some serious
resource issues from one of the Beta versions recently, and switched back
to the stable version.

-John


On Fri, Jun 13, 2014 at 9:01 AM, Jason Batchelor <jxbatchelor at gmail.com>
wrote:

> Hello everyone:
>
> I have Bro installed on two Dell r720s each with the following specs...
>
> Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz x 32
> 48GB RAM
>
> Running: CentOs 6.5
>
> Both have the following PF_RING configuration:
>
> PF_RING Version          : 6.0.2 ($Revision: 7746$)
> Total rings              : 16
> Standard (non DNA) Options
> Ring slots               : 32768
> Slot version             : 15
> Capture TX               : No [RX only]
> IP Defragment            : No
> Socket Mode              : Standard
> Transparent mode         : Yes [mode 0]
> Total plugins            : 0
> Cluster Fragment Queue   : 1917
> Cluster Fragment Discard : 26648
> The only difference in PF Ring is the other server (Server A) is going off
> revision 7601, where B is rev 7746.
>
> I've tuned the NIC to the following settings...
>
> ethtool -K p4p2 tso off
> ethtool -K p4p2 gro off
> ethtool -K p4p2 lro off
> ethtool -K p4p2 gso off
> ethtool -K p4p2 rx off
> ethtool -K p4p2 tx off
> ethtool -K p4p2 sg off
> ethtool -K p4p2 rxvlan off
> ethtool -K p4p2 txvlan off
> ethtool -N p4p2 rx-flow-hash udp4 sdfn
> ethtool -N p4p2 rx-flow-hash udp6 sdfn
> ethtool -n p4p2 rx-flow-hash udp6
> ethtool -n p4p2 rx-flow-hash udp4
> ethtool -C p4p2 rx-usecs 1000
> ethtool -C p4p2 adaptive-rx off
> ethtool -G p4p2 rx 4096
> I've got the following sysctl settings on each.
>
> # turn off selective ACK and timestamps
> net.ipv4.tcp_sack = 0
> net.ipv4.tcp_timestamps = 0
> # memory allocation min/pressure/max.
> # read buffer, write buffer, and buffer space
> net.ipv4.tcp_rmem = 10000000 10000000 10000000
> net.ipv4.tcp_wmem = 10000000 10000000 10000000
> net.ipv4.tcp_mem = 10000000 10000000 10000000
> net.core.rmem_max = 524287
> net.core.wmem_max = 524287
> net.core.rmem_default = 524287
> net.core.wmem_default = 524287
> net.core.optmem_max = 524287
> net.core.netdev_max_backlog = 300000
> Each bro configuration is using the following...
>
> [manager]
> type=manager
> host=localhost
> [proxy-1]
> type=proxy
> host=localhost
> [worker-1]
> type=worker
> host=localhost
> interface=p4p2
> lb_method=pf_ring
> lb_procs=16
> Both have the same NIC driver version (ixgbe):
> 3.15.1-k
>
> Same services installed (min install).
>
> Slightly different Kernel versions...
> Server A (2.6.32-431.11.2.el6.x86_64)
> Server B (2.6.32-431.17.1.el6.x86_64)
>
>
> At the moment Server A is getting about 700MB/s and Server B is getting
> about 600Mb/s.
>
> What I don't understand, is Server A is having several orders of magnatude
> better performance compared to Server B?
>
> TOP from A (included a few bro workers):
>
> top - 12:48:45 up 1 day, 17:03,  2 users,  load average: 5.30, 3.99, 3.13
> Tasks: 706 total,  19 running, 687 sleeping,   0 stopped,   0 zombie
> Cpu(s): 33.9%us,  6.6%sy,  1.1%ni, 57.2%id,  0.0%wa,  0.0%hi,  1.2%si,
> 0.0%st
> Mem:  49376004k total, 33605828k used, 15770176k free,    93100k buffers
> Swap:  2621432k total,     9760k used,  2611672k free,  9206880k cached
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  5768 root      20   0 1808m 1.7g 519m R 100.0  3.6  32:24.92 bro
>  5760 root      20   0 1688m 1.6g 519m R 99.7  3.4  34:08.36 bro
>  3314 root      20   0 2160m 269m 4764 R 96.1  0.6  30:14.12 bro
>  5754 root      20   0 1451m 1.4g 519m R 82.8  2.9  36:40.02 bro
> TOP from B (included a few bro workers)
>
> top - 12:49:33 up 14:24,  2 users,  load average: 10.28, 9.31, 8.06
> Tasks: 708 total,  25 running, 683 sleeping,   0 stopped,   0 zombie
> Cpu(s): 41.6%us,  6.0%sy,  1.0%ni, 50.4%id,  0.0%wa,  0.0%hi,  1.1%si,
> 0.0%st
> Mem:  49376004k total, 31837340k used, 17538664k free,   147212k buffers
> Swap:  2621432k total,        0k used,  2621432k free, 13494332k cached
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  3178 root      20   0 1073m 1.0g 264m R 100.0  2.1 401:47.31 bro
>  3188 root      20   0  881m 832m 264m R 100.0  1.7 377:48.90 bro
>  3189 root      20   0 1247m 1.2g 264m R 100.0  2.5 403:22.95 bro
>  3193 root      20   0  920m 871m 264m R 100.0  1.8 429:45.98 bro
> Both have the same amount of Bro workers. I just do not understand
> why Server A is literally half the utilization on top of seeing more
> traffic? The only real and consistent difference between the two I see is
> that server A seems to have twice the amount of SHR (shared memory)
> compared to server B.
>
> Could this be part of the issue, if not the root cause? How might I go
> about rectifying the issue?
>
> FWIW, both are not dropping packets and doing well. However, I want to run
> other apps on top of this, and the poor performance on Server B is likely
> to have effects on it.
>
> Thanks advance for the advice!
>
> -Jason
>
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140613/6e40f26b/attachment.html 

From gc355804 at ohio.edu  Fri Jun 13 06:54:51 2014
From: gc355804 at ohio.edu (Gilbert Clark)
Date: Fri, 13 Jun 2014 09:54:51 -0400
Subject: [Bro] Unexplained Performance Differences Between Like Servers
In-Reply-To: <CANWz5X8M7gbE4iNRYbtLfJrJzC7g-BtNVS1s7CNCHkxtEO_Y7Q@mail.gmail.com>
References: <CANWz5X8M7gbE4iNRYbtLfJrJzC7g-BtNVS1s7CNCHkxtEO_Y7Q@mail.gmail.com>
Message-ID: <539B02AB.8060907@ohio.edu>

Hi Jason:

Is the type of traffic in the 600 Mbps stream similar to the type of 
traffic in the 700 Mbps stream?

Cheers,
Gilbert Clark

On 6/13/14, 9:01 AM, Jason Batchelor wrote:
> Hello everyone:
> I have Bro installed on two Dell r720s each with the following specs...
> Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz x 32
> 48GB RAM
> Running: CentOs 6.5
> Both have the following PF_RING configuration:
> PF_RING Version          : 6.0.2 ($Revision: 7746$)
> Total rings              : 16
> Standard (non DNA) Options
> Ring slots               : 32768
> Slot version             : 15
> Capture TX               : No [RX only]
> IP Defragment            : No
> Socket Mode              : Standard
> Transparent mode         : Yes [mode 0]
> Total plugins            : 0
> Cluster Fragment Queue   : 1917
> Cluster Fragment Discard : 26648
> The only difference in PF Ring is the other server (Server A) is going 
> off revision 7601, where B is rev 7746.
> I've tuned the NIC to the following settings...
> ethtool -K p4p2 tso off
> ethtool -K p4p2 gro off
> ethtool -K p4p2 lro off
> ethtool -K p4p2 gso off
> ethtool -K p4p2 rx off
> ethtool -K p4p2 tx off
> ethtool -K p4p2 sg off
> ethtool -K p4p2 rxvlan off
> ethtool -K p4p2 txvlan off
> ethtool -N p4p2 rx-flow-hash udp4 sdfn
> ethtool -N p4p2 rx-flow-hash udp6 sdfn
> ethtool -n p4p2 rx-flow-hash udp6
> ethtool -n p4p2 rx-flow-hash udp4
> ethtool -C p4p2 rx-usecs 1000
> ethtool -C p4p2 adaptive-rx off
> ethtool -G p4p2 rx 4096
> I've got the following sysctl settings on each.
> # turn off selective ACK and timestamps
> net.ipv4.tcp_sack = 0
> net.ipv4.tcp_timestamps = 0
> # memory allocation min/pressure/max.
> # read buffer, write buffer, and buffer space
> net.ipv4.tcp_rmem = 10000000 10000000 10000000
> net.ipv4.tcp_wmem = 10000000 10000000 10000000
> net.ipv4.tcp_mem = 10000000 10000000 10000000
> net.core.rmem_max = 524287
> net.core.wmem_max = 524287
> net.core.rmem_default = 524287
> net.core.wmem_default = 524287
> net.core.optmem_max = 524287
> net.core.netdev_max_backlog = 300000
> Each bro configuration is using the following...
> [manager]
> type=manager
> host=localhost
> [proxy-1]
> type=proxy
> host=localhost
> [worker-1]
> type=worker
> host=localhost
> interface=p4p2
> lb_method=pf_ring
> lb_procs=16
> Both have the same NIC driver version (ixgbe):
> 3.15.1-k
> Same services installed (min install).
> Slightly different Kernel versions...
> Server A (2.6.32-431.11.2.el6.x86_64)
> Server B (2.6.32-431.17.1.el6.x86_64)
> At the moment Server A is getting about 700MB/s and Server B is 
> getting about 600Mb/s.
> What I don't understand, is Server A is having several orders of 
> magnatude better performance compared to Server B?
> TOP from A (included a few bro workers):
> top - 12:48:45 up 1 day, 17:03,  2 users,  load average: 5.30, 3.99, 3.13
> Tasks: 706 total,  19 running, 687 sleeping,   0 stopped,   0 zombie
> Cpu(s): 33.9%us,  6.6%sy,  1.1%ni, 57.2%id,  0.0%wa,  0.0%hi, 1.2%si,  
> 0.0%st
> Mem:  49376004k total, 33605828k used, 15770176k free, 93100k buffers
> Swap:  2621432k total,     9760k used,  2611672k free, 9206880k cached
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM TIME+  COMMAND
>  5768 root      20   0 1808m 1.7g 519m R 100.0  3.6  32:24.92 bro
>  5760 root      20   0 1688m 1.6g 519m R 99.7  3.4  34:08.36 bro
>  3314 root      20   0 2160m 269m 4764 R 96.1  0.6  30:14.12 bro
>  5754 root      20   0 1451m 1.4g 519m R 82.8  2.9  36:40.02 bro
> TOP from B (included a few bro workers)
> top - 12:49:33 up 14:24,  2 users,  load average: 10.28, 9.31, 8.06
> Tasks: 708 total,  25 running, 683 sleeping,   0 stopped,   0 zombie
> Cpu(s): 41.6%us,  6.0%sy,  1.0%ni, 50.4%id,  0.0%wa,  0.0%hi, 1.1%si,  
> 0.0%st
> Mem:  49376004k total, 31837340k used, 17538664k free, 147212k buffers
> Swap:  2621432k total,        0k used,  2621432k free, 13494332k cached
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM TIME+  COMMAND
>  3178 root      20   0 1073m 1.0g 264m R 100.0  2.1 401:47.31 bro
>  3188 root      20   0  881m 832m 264m R 100.0  1.7 377:48.90 bro
>  3189 root      20   0 1247m 1.2g 264m R 100.0  2.5 403:22.95 bro
>  3193 root      20   0  920m 871m 264m R 100.0  1.8 429:45.98 bro
> Both have the same amount of Bro workers. I just do not understand 
> why Server A is literally half the utilization on top of seeing more 
> traffic? The only real and consistent difference between the two I see 
> is that server A seems to have twice the amount of SHR (shared memory) 
> compared to server B.
> Could this be part of the issue, if not the root cause? How might I go 
> about rectifying the issue?
> FWIW, both are not dropping packets and doing well. However, I want to 
> run other apps on top of this, and the poor performance on Server B is 
> likely to have effects on it.
> Thanks advance for the advice!
> -Jason



From luke at geekempire.com  Fri Jun 13 07:05:20 2014
From: luke at geekempire.com (Mike Reeves)
Date: Fri, 13 Jun 2014 10:05:20 -0400
Subject: [Bro] Unexplained Performance Differences Between Like Servers
In-Reply-To: <CAOxyatSJmqFOqyVUBiWfW7hFQ2trjCR1=27ABA6D7O+QeaXr+Q@mail.gmail.com>
References: <CANWz5X8M7gbE4iNRYbtLfJrJzC7g-BtNVS1s7CNCHkxtEO_Y7Q@mail.gmail.com>
	<CAOxyatSJmqFOqyVUBiWfW7hFQ2trjCR1=27ABA6D7O+QeaXr+Q@mail.gmail.com>
Message-ID: <F97AB54F-C192-4EBF-BE3E-B984E310E127@geekempire.com>

The different traffic profiles can can cause different performance. My guess is you are seeing more traffic of a certain type on one of the boxes vs the other. To really know you would need to profile the traffic but if you think about it if more http traffic for instance would be more files processing etc.

Mike

On Jun 13, 2014, at 9:49 AM, John Hoyt <john.h.hoyt at gmail.com> wrote:

> Hey Jason,
> 
> What versions of Bro, and it is the same for both?  I had some serious resource issues from one of the Beta versions recently, and switched back to the stable version. 
> 
> -John
> 
> 
> On Fri, Jun 13, 2014 at 9:01 AM, Jason Batchelor <jxbatchelor at gmail.com> wrote:
> Hello everyone:
>  
> I have Bro installed on two Dell r720s each with the following specs...
>  
> Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz x 32
> 48GB RAM
>  
> Running: CentOs 6.5
>  
> Both have the following PF_RING configuration:
>  
> PF_RING Version          : 6.0.2 ($Revision: 7746$)
> Total rings              : 16
> Standard (non DNA) Options
> Ring slots               : 32768
> Slot version             : 15
> Capture TX               : No [RX only]
> IP Defragment            : No
> Socket Mode              : Standard
> Transparent mode         : Yes [mode 0]
> Total plugins            : 0
> Cluster Fragment Queue   : 1917
> Cluster Fragment Discard : 26648
> The only difference in PF Ring is the other server (Server A) is going off revision 7601, where B is rev 7746.
>  
> I've tuned the NIC to the following settings...
>  
> ethtool -K p4p2 tso off
> ethtool -K p4p2 gro off
> ethtool -K p4p2 lro off
> ethtool -K p4p2 gso off
> ethtool -K p4p2 rx off
> ethtool -K p4p2 tx off
> ethtool -K p4p2 sg off
> ethtool -K p4p2 rxvlan off
> ethtool -K p4p2 txvlan off
> ethtool -N p4p2 rx-flow-hash udp4 sdfn
> ethtool -N p4p2 rx-flow-hash udp6 sdfn
> ethtool -n p4p2 rx-flow-hash udp6
> ethtool -n p4p2 rx-flow-hash udp4
> ethtool -C p4p2 rx-usecs 1000
> ethtool -C p4p2 adaptive-rx off
> ethtool -G p4p2 rx 4096
> I've got the following sysctl settings on each.
>  
> # turn off selective ACK and timestamps
> net.ipv4.tcp_sack = 0
> net.ipv4.tcp_timestamps = 0
> # memory allocation min/pressure/max.
> # read buffer, write buffer, and buffer space
> net.ipv4.tcp_rmem = 10000000 10000000 10000000
> net.ipv4.tcp_wmem = 10000000 10000000 10000000
> net.ipv4.tcp_mem = 10000000 10000000 10000000
> net.core.rmem_max = 524287
> net.core.wmem_max = 524287
> net.core.rmem_default = 524287
> net.core.wmem_default = 524287
> net.core.optmem_max = 524287
> net.core.netdev_max_backlog = 300000
> Each bro configuration is using the following...
>  
> [manager]
> type=manager
> host=localhost
> [proxy-1]
> type=proxy
> host=localhost
> [worker-1]
> type=worker
> host=localhost
> interface=p4p2
> lb_method=pf_ring
> lb_procs=16
> Both have the same NIC driver version (ixgbe):
> 3.15.1-k
>  
> Same services installed (min install).
>  
> Slightly different Kernel versions...
> Server A (2.6.32-431.11.2.el6.x86_64)
> Server B (2.6.32-431.17.1.el6.x86_64)
>  
>  
> At the moment Server A is getting about 700MB/s and Server B is getting about 600Mb/s.
>  
> What I don't understand, is Server A is having several orders of magnatude better performance compared to Server B?
>  
> TOP from A (included a few bro workers):
>  
> top - 12:48:45 up 1 day, 17:03,  2 users,  load average: 5.30, 3.99, 3.13
> Tasks: 706 total,  19 running, 687 sleeping,   0 stopped,   0 zombie
> Cpu(s): 33.9%us,  6.6%sy,  1.1%ni, 57.2%id,  0.0%wa,  0.0%hi,  1.2%si,  0.0%st
> Mem:  49376004k total, 33605828k used, 15770176k free,    93100k buffers
> Swap:  2621432k total,     9760k used,  2611672k free,  9206880k cached
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  5768 root      20   0 1808m 1.7g 519m R 100.0  3.6  32:24.92 bro
>  5760 root      20   0 1688m 1.6g 519m R 99.7  3.4  34:08.36 bro
>  3314 root      20   0 2160m 269m 4764 R 96.1  0.6  30:14.12 bro
>  5754 root      20   0 1451m 1.4g 519m R 82.8  2.9  36:40.02 bro
> TOP from B (included a few bro workers)
>  
> top - 12:49:33 up 14:24,  2 users,  load average: 10.28, 9.31, 8.06
> Tasks: 708 total,  25 running, 683 sleeping,   0 stopped,   0 zombie
> Cpu(s): 41.6%us,  6.0%sy,  1.0%ni, 50.4%id,  0.0%wa,  0.0%hi,  1.1%si,  0.0%st
> Mem:  49376004k total, 31837340k used, 17538664k free,   147212k buffers
> Swap:  2621432k total,        0k used,  2621432k free, 13494332k cached
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  3178 root      20   0 1073m 1.0g 264m R 100.0  2.1 401:47.31 bro
>  3188 root      20   0  881m 832m 264m R 100.0  1.7 377:48.90 bro
>  3189 root      20   0 1247m 1.2g 264m R 100.0  2.5 403:22.95 bro
>  3193 root      20   0  920m 871m 264m R 100.0  1.8 429:45.98 bro
> Both have the same amount of Bro workers. I just do not understand why Server A is literally half the utilization on top of seeing more traffic? The only real and consistent difference between the two I see is that server A seems to have twice the amount of SHR (shared memory) compared to server B.
>  
> Could this be part of the issue, if not the root cause? How might I go about rectifying the issue?
>  
> FWIW, both are not dropping packets and doing well. However, I want to run other apps on top of this, and the poor performance on Server B is likely to have effects on it.
>  
> Thanks advance for the advice!
>  
> -Jason
>  
>  
>  
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140613/2a23b4da/attachment.html 

From JAzoff at albany.edu  Fri Jun 13 07:09:10 2014
From: JAzoff at albany.edu (Justin Azoff)
Date: Fri, 13 Jun 2014 10:09:10 -0400
Subject: [Bro] Unexplained Performance Differences Between Like Servers
In-Reply-To: <CANWz5X8M7gbE4iNRYbtLfJrJzC7g-BtNVS1s7CNCHkxtEO_Y7Q@mail.gmail.com>
References: <CANWz5X8M7gbE4iNRYbtLfJrJzC7g-BtNVS1s7CNCHkxtEO_Y7Q@mail.gmail.com>
Message-ID: <20140613140910.GI3515@datacomm.albany.edu>

On Fri, Jun 13, 2014 at 08:01:54AM -0500, Jason Batchelor wrote:
> At the moment Server A is getting about 700MB/s and Server B is getting about
> 600Mb/s.
>  
> What I don't understand, is Server A is having several orders of magnatude
> better performance compared to Server B?
>  
> TOP from A (included a few bro workers):
>  
> top - 12:48:45 up 1 day, 17:03,  2 users,  load average: 5.30, 3.99, 3.13
> Tasks: 706 total,  19 running, 687 sleeping,   0 stopped,   0 zombie
> Cpu(s): 33.9%us,  6.6%sy,  1.1%ni, 57.2%id,  0.0%wa,  0.0%hi,  1.2%si,  0.0%st
> Mem:  49376004k total, 33605828k used, 15770176k free,    93100k buffers
> Swap:  2621432k total,     9760k used,  2611672k free,  9206880k cached
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  5768 root      20   0 1808m 1.7g 519m R 100.0  3.6  32:24.92 bro
>  5760 root      20   0 1688m 1.6g 519m R 99.7  3.4  34:08.36 bro
>  3314 root      20   0 2160m 269m 4764 R 96.1  0.6  30:14.12 bro
>  5754 root      20   0 1451m 1.4g 519m R 82.8  2.9  36:40.02 bro

Server A Bro cpu utilization = 378.6

> TOP from B (included a few bro workers)
>  
> top - 12:49:33 up 14:24,  2 users,  load average: 10.28, 9.31, 8.06
> Tasks: 708 total,  25 running, 683 sleeping,   0 stopped,   0 zombie
> Cpu(s): 41.6%us,  6.0%sy,  1.0%ni, 50.4%id,  0.0%wa,  0.0%hi,  1.1%si,  0.0%st
> Mem:  49376004k total, 31837340k used, 17538664k free,   147212k buffers
> Swap:  2621432k total,        0k used,  2621432k free, 13494332k cached
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  3178 root      20   0 1073m 1.0g 264m R 100.0  2.1 401:47.31 bro
>  3188 root      20   0  881m 832m 264m R 100.0  1.7 377:48.90 bro
>  3189 root      20   0 1247m 1.2g 264m R 100.0  2.5 403:22.95 bro
>  3193 root      20   0  920m 871m 264m R 100.0  1.8 429:45.98 bro

> Both have the same amount of Bro workers. I just do not understand why Server
> A is literally half the utilization on top of seeing more traffic? The only
> real and consistent difference between the two I see is that server A seems to
> have twice the amount of SHR (shared memory) compared to server B.

Server B Bro cpu utilization = 400%

Are you only running 4 workers or did you truncate the output?  Is that
running at 100% 24/7 or does it vary with the traffic?

Are you doing 4 tuple load balancing or 2 tuple load balancing between
the two servers?  Most likely Server B is seeing more flows.


-- 
-- Justin Azoff


From jxbatchelor at gmail.com  Fri Jun 13 07:19:34 2014
From: jxbatchelor at gmail.com (Jason Batchelor)
Date: Fri, 13 Jun 2014 09:19:34 -0500
Subject: [Bro] Unexplained Performance Differences Between Like Servers
In-Reply-To: <20140613140910.GI3515@datacomm.albany.edu>
References: <CANWz5X8M7gbE4iNRYbtLfJrJzC7g-BtNVS1s7CNCHkxtEO_Y7Q@mail.gmail.com>
	<20140613140910.GI3515@datacomm.albany.edu>
Message-ID: <CANWz5X_3x+Q2Me7OY8HNBz+QG2EYmFcfT+na4QPLr40RDRysnQ@mail.gmail.com>

Wow, thanks for all the quick replies :)

> What versions of Bro, and it is the same for both?

I am using the same version of Bro for each server (1.2).

> Is the type of traffic in the 600 Mbps stream similar to the type of
traffic in the 700 Mbps stream?

I'm not 100% sure but I think that is a really good question to ask. Do you
know of any good tools that might help inform an answer? I know of iptraf
for example, is there one that folks generally prefer the most?

> Are you only running 4 workers or did you truncate the output?
Yes, I truncated the output to show four workers each (I have 16 total).

> Are you doing 4 tuple load balancing or 2 tuple load balancing between
the two servers?

Sorry I am not sure what you mean by this or the implications of one over
the other. Is there an easy way I can find out (I am kinda new to this)? I
agree with the likelihood that B may be recieving more flows.

Thanks!
Jason




On Fri, Jun 13, 2014 at 9:09 AM, Justin Azoff <JAzoff at albany.edu> wrote:

> On Fri, Jun 13, 2014 at 08:01:54AM -0500, Jason Batchelor wrote:
> > At the moment Server A is getting about 700MB/s and Server B is getting
> about
> > 600Mb/s.
> >
> > What I don't understand, is Server A is having several orders of
> magnatude
> > better performance compared to Server B?
> >
> > TOP from A (included a few bro workers):
> >
> > top - 12:48:45 up 1 day, 17:03,  2 users,  load average: 5.30, 3.99, 3.13
> > Tasks: 706 total,  19 running, 687 sleeping,   0 stopped,   0 zombie
> > Cpu(s): 33.9%us,  6.6%sy,  1.1%ni, 57.2%id,  0.0%wa,  0.0%hi,  1.2%si,
>  0.0%st
> > Mem:  49376004k total, 33605828k used, 15770176k free,    93100k buffers
> > Swap:  2621432k total,     9760k used,  2611672k free,  9206880k cached
> >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> >  5768 root      20   0 1808m 1.7g 519m R 100.0  3.6  32:24.92 bro
> >  5760 root      20   0 1688m 1.6g 519m R 99.7  3.4  34:08.36 bro
> >  3314 root      20   0 2160m 269m 4764 R 96.1  0.6  30:14.12 bro
> >  5754 root      20   0 1451m 1.4g 519m R 82.8  2.9  36:40.02 bro
>
> Server A Bro cpu utilization = 378.6
>
> > TOP from B (included a few bro workers)
> >
> > top - 12:49:33 up 14:24,  2 users,  load average: 10.28, 9.31, 8.06
> > Tasks: 708 total,  25 running, 683 sleeping,   0 stopped,   0 zombie
> > Cpu(s): 41.6%us,  6.0%sy,  1.0%ni, 50.4%id,  0.0%wa,  0.0%hi,  1.1%si,
>  0.0%st
> > Mem:  49376004k total, 31837340k used, 17538664k free,   147212k buffers
> > Swap:  2621432k total,        0k used,  2621432k free, 13494332k cached
> >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> >  3178 root      20   0 1073m 1.0g 264m R 100.0  2.1 401:47.31 bro
> >  3188 root      20   0  881m 832m 264m R 100.0  1.7 377:48.90 bro
> >  3189 root      20   0 1247m 1.2g 264m R 100.0  2.5 403:22.95 bro
> >  3193 root      20   0  920m 871m 264m R 100.0  1.8 429:45.98 bro
>
> > Both have the same amount of Bro workers. I just do not understand why
> Server
> > A is literally half the utilization on top of seeing more traffic? The
> only
> > real and consistent difference between the two I see is that server A
> seems to
> > have twice the amount of SHR (shared memory) compared to server B.
>
> Server B Bro cpu utilization = 400%
>
> Are you only running 4 workers or did you truncate the output?  Is that
> running at 100% 24/7 or does it vary with the traffic?
>
> Are you doing 4 tuple load balancing or 2 tuple load balancing between
> the two servers?  Most likely Server B is seeing more flows.
>
>
> --
> -- Justin Azoff
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140613/8b82f05b/attachment.html 

From jxbatchelor at gmail.com  Fri Jun 13 07:32:42 2014
From: jxbatchelor at gmail.com (Jason Batchelor)
Date: Fri, 13 Jun 2014 09:32:42 -0500
Subject: [Bro] Unexplained Performance Differences Between Like Servers
In-Reply-To: <CANWz5X_3x+Q2Me7OY8HNBz+QG2EYmFcfT+na4QPLr40RDRysnQ@mail.gmail.com>
References: <CANWz5X8M7gbE4iNRYbtLfJrJzC7g-BtNVS1s7CNCHkxtEO_Y7Q@mail.gmail.com>
	<20140613140910.GI3515@datacomm.albany.edu>
	<CANWz5X_3x+Q2Me7OY8HNBz+QG2EYmFcfT+na4QPLr40RDRysnQ@mail.gmail.com>
Message-ID: <CANWz5X8w0Anws_A95Pkh2dY3FD4TT4opOwJydSey2jjY-Mv0Vw@mail.gmail.com>

FWIW:

I just ran iptraf for a bit on both and one thing really stuck out to me:

Server A:
Other IP:      5273     633087        5273     633087           0          0

Server B:
 Other IP:    952797    445867K      952797    445867K           0
0

So server A is seeing 633087 bytes of 'other' traffic, while B is seeing
445867 kilobytes of 'other' traffic. Do you think this other traffic could
be the root cause of the issues here? If so, would a bpf filter looking for
only tcp/udp/ipv4 traffic be sufficient? How might I apply that within Bro?

Here is the full view taken some time after the metrics above:

Server A:

x               Total      Total    Incoming   Incoming    Outgoing
Outgoing              x
x             Packets      Bytes     Packets      Bytes     Packets
Bytes              x
x Total:     80187229     51270M    80187229     51270M
0          0              x
x IPv4:      80187193     50026M    80187193     50026M
0          0              x
x IPv6:            36       1296          36       1296
0          0              x
x TCP:       70040618     47342M    70040618     47342M
0          0              x
x UDP:       10052947      2676M    10052947      2676M
0          0              x
x ICMP:         85189    6652550       85189    6652550
0          0              x
x Other IP:      8475    1060993        8475    1060993
0          0

Server B:

x               Total      Total    Incoming   Incoming    Outgoing
Outgoing                   x
x             Packets      Bytes     Packets      Bytes     Packets
Bytes                   x
x Total:     89718860     53317M    89718860     53317M
0          0                   x
x IPv4:      89712988     51882M    89712988     51882M
0          0                   x
x IPv6:          5872      51778        5872      51778
0          0                   x
x TCP:       79615124     49170M    79615124     49170M
0          0                   x
x UDP:        7627607      1682M     7627607      1682M
0          0                   x
x ICMP:         86620    5619078       86620    5619078
0          0                   x
x Other IP:   2389509      1023M     2389509      1023M
0          0                   x
Many thanks in advance for the quick and helpful replies!


On Fri, Jun 13, 2014 at 9:19 AM, Jason Batchelor <jxbatchelor at gmail.com>
wrote:

> Wow, thanks for all the quick replies :)
>
> > What versions of Bro, and it is the same for both?
>
> I am using the same version of Bro for each server (1.2).
>
> > Is the type of traffic in the 600 Mbps stream similar to the type of
> traffic in the 700 Mbps stream?
>
> I'm not 100% sure but I think that is a really good question to ask. Do
> you know of any good tools that might help inform an answer? I know of
> iptraf for example, is there one that folks generally prefer the most?
>
> > Are you only running 4 workers or did you truncate the output?
> Yes, I truncated the output to show four workers each (I have 16 total).
>
> > Are you doing 4 tuple load balancing or 2 tuple load balancing between
> the two servers?
>
> Sorry I am not sure what you mean by this or the implications of one over
> the other. Is there an easy way I can find out (I am kinda new to this)? I
> agree with the likelihood that B may be recieving more flows.
>
> Thanks!
> Jason
>
>
>
>
> On Fri, Jun 13, 2014 at 9:09 AM, Justin Azoff <JAzoff at albany.edu> wrote:
>
>> On Fri, Jun 13, 2014 at 08:01:54AM -0500, Jason Batchelor wrote:
>> > At the moment Server A is getting about 700MB/s and Server B is getting
>> about
>> > 600Mb/s.
>> >
>> > What I don't understand, is Server A is having several orders of
>> magnatude
>> > better performance compared to Server B?
>> >
>> > TOP from A (included a few bro workers):
>> >
>> > top - 12:48:45 up 1 day, 17:03,  2 users,  load average: 5.30, 3.99,
>> 3.13
>> > Tasks: 706 total,  19 running, 687 sleeping,   0 stopped,   0 zombie
>> > Cpu(s): 33.9%us,  6.6%sy,  1.1%ni, 57.2%id,  0.0%wa,  0.0%hi,  1.2%si,
>>  0.0%st
>> > Mem:  49376004k total, 33605828k used, 15770176k free,    93100k buffers
>> > Swap:  2621432k total,     9760k used,  2611672k free,  9206880k cached
>> >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>> >  5768 root      20   0 1808m 1.7g 519m R 100.0  3.6  32:24.92 bro
>> >  5760 root      20   0 1688m 1.6g 519m R 99.7  3.4  34:08.36 bro
>> >  3314 root      20   0 2160m 269m 4764 R 96.1  0.6  30:14.12 bro
>> >  5754 root      20   0 1451m 1.4g 519m R 82.8  2.9  36:40.02 bro
>>
>> Server A Bro cpu utilization = 378.6
>>
>> > TOP from B (included a few bro workers)
>> >
>> > top - 12:49:33 up 14:24,  2 users,  load average: 10.28, 9.31, 8.06
>> > Tasks: 708 total,  25 running, 683 sleeping,   0 stopped,   0 zombie
>> > Cpu(s): 41.6%us,  6.0%sy,  1.0%ni, 50.4%id,  0.0%wa,  0.0%hi,  1.1%si,
>>  0.0%st
>> > Mem:  49376004k total, 31837340k used, 17538664k free,   147212k buffers
>> > Swap:  2621432k total,        0k used,  2621432k free, 13494332k cached
>> >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>> >  3178 root      20   0 1073m 1.0g 264m R 100.0  2.1 401:47.31 bro
>> >  3188 root      20   0  881m 832m 264m R 100.0  1.7 377:48.90 bro
>> >  3189 root      20   0 1247m 1.2g 264m R 100.0  2.5 403:22.95 bro
>> >  3193 root      20   0  920m 871m 264m R 100.0  1.8 429:45.98 bro
>>
>> > Both have the same amount of Bro workers. I just do not understand why
>> Server
>> > A is literally half the utilization on top of seeing more traffic? The
>> only
>> > real and consistent difference between the two I see is that server A
>> seems to
>> > have twice the amount of SHR (shared memory) compared to server B.
>>
>> Server B Bro cpu utilization = 400%
>>
>> Are you only running 4 workers or did you truncate the output?  Is that
>> running at 100% 24/7 or does it vary with the traffic?
>>
>> Are you doing 4 tuple load balancing or 2 tuple load balancing between
>> the two servers?  Most likely Server B is seeing more flows.
>>
>>
>> --
>> -- Justin Azoff
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140613/bcb999b4/attachment.html 

From luke at geekempire.com  Fri Jun 13 07:32:59 2014
From: luke at geekempire.com (Mike Reeves)
Date: Fri, 13 Jun 2014 10:32:59 -0400
Subject: [Bro] Bro hanging on some sensors
In-Reply-To: <BDF3793B557F624699D4C68A3B8FCF82182ADD@DRTEXW238VM.na.d-rco.com>
References: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82181922@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDry05X3RBCt7Ph9SgQa13__+rjg28=VZ9KmPP3KgyYqw@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82182ADD@DRTEXW238VM.na.d-rco.com>
Message-ID: <D422E894-2B8F-4E87-B223-CFFC85C170A2@geekempire.com>

Bro will sense additions but won?t remove old stuff unless it is restarted check out ?Putting it all together? in the blog post below. 

http://blog.bro.org/2014/01/intelligence-data-and-bro_4980.html



On Jun 13, 2014, at 8:59 AM, Kellogg, Brian D (OLN) <bkellogg at dresser-rand.com> wrote:

> Yeah they were working up to about a week ago.  I commented out the intel
> files on one and that fixed it.  Odd, as they should all be the same with the
> replication from the SO server policy directory.  The majority of the sensors
> are working fine with the same intel files and I'm seeing new results in ELSA
> every day.  Hmmm
> 
> I do not have Bro restarting via Onion-Salt.  I was under the understanding
> that Bro would periodically sense the file changes and re-read the intel
> files.
> 
> Thanks,
> Brian
> 
> Thank you,
> Brian Kellogg
> Security Analyst; IT Governance, Risk, and Compliance
> 500 Paul Clark Drive, Olean,  NY 14760
> T: (716) 375-3186 | F: (716) 375-3557
> www.dresser-rand.com     NYSE: DRC
> 
> Bringing energy and the environment into harmony?
> IMPORTANT NOTICE:
> This email may be confidential, may be legally privileged, and is for the intended recipient only. Unauthorized access, disclosure, copying, distribution, or reliance on any of it by anyone else is prohibited and may be a criminal offense. Please delete if obtained in error and email confirmation to the sender.
> 
> 
> -----Original Message-----
> From: Doug Burks [mailto:doug.burks at gmail.com]
> Sent: Friday, June 13, 2014 6:56 AM
> To: Kellogg, Brian D (OLN)
> Cc: bro
> Subject: Re: [Bro] Bro hanging on some sensors
> 
> Did these sensors work correctly before you added the Intel files?
> 
> Have you tried removing the Intel files from the equation?  If not, please try commenting out the entries you added to /opt/bro/share/bro/site/local.bro and then running:
> sudo broctl install
> sudo broctl restart
> 
> Are you running OnionSalt?  Do you have OnionSalt configured to automatically restart Bro?  There are known issues with doing so, which is why OnionSalt does not do so by default.  Please see:
> https://code.google.com/p/security-onion/wiki/Salt#Features
> 
> This may not be an issue strictly related to Bro, so we may need to move this conversation to the security-onion mailing list:
> https://code.google.com/p/security-onion/wiki/MailingLists
> 
> On Thu, Jun 12, 2014 at 9:00 AM, Kellogg, Brian D (OLN) <bkellogg at dresser-rand.com> wrote:
>> I use a cron job that runs every 30 minutes to download the intel files to:
>> /opt/bro/share/bro/policy/.  The cron job uses the mal-dnssearch script.
>> 
>> In each sensor's /opt/bro/share/bro/site/local.bro file I have the below:
>> 
>> # load intelligence framework
>> @load policy/frameworks/intel/seen
>> @load policy/frameworks/intel/do_notice #@load
>> policy/integration/collective-intel
>> redef Intel::read_files += {
>>        "/opt/bro/share/bro/policy/xxx.intel",
>>        "/opt/bro/share/bro/policy/xxx.intel",
>>        "/opt/bro/share/bro/policy/xxx.intel",
>>        "/opt/bro/share/bro/policy/xxx.intel",
>>        "/opt/bro/share/bro/policy/xxx.intel",
>>        "/opt/bro/share/bro/policy/xxx.intel",
>> };
>> 
>> In the reporter.log file I am now seeing the below warning on the four
>> sensors having this issue:
>>    0.000000    Reporter::WARNING       SumStat key request for the
>> 3Mntn3EPhU3 SumStat uid took longer than 1 minute and was
>> automatically cancelled.
>> /opt/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line
>> 209
>>    0.000000    Reporter::WARNING       SumStat key request for the
>> 2HAva5N4Kqf SumStat uid took longer than 1 minute and was
>> automatically cancelled.
>> /opt/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line
>> 209
>> 
>> Thanks,
>> Brian
> 
> 
> 
> --
> Doug Burks
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140613/83a6d5cd/attachment.html 

From gc355804 at ohio.edu  Fri Jun 13 07:45:28 2014
From: gc355804 at ohio.edu (Gilbert Clark)
Date: Fri, 13 Jun 2014 10:45:28 -0400
Subject: [Bro] Unexplained Performance Differences Between Like Servers
In-Reply-To: <CANWz5X_3x+Q2Me7OY8HNBz+QG2EYmFcfT+na4QPLr40RDRysnQ@mail.gmail.com>
References: <CANWz5X8M7gbE4iNRYbtLfJrJzC7g-BtNVS1s7CNCHkxtEO_Y7Q@mail.gmail.com>	<20140613140910.GI3515@datacomm.albany.edu>
	<CANWz5X_3x+Q2Me7OY8HNBz+QG2EYmFcfT+na4QPLr40RDRysnQ@mail.gmail.com>
Message-ID: <539B0E88.1010203@ohio.edu>

Hi Justin:

> > Is the type of traffic in the 600 Mbps stream similar to the type of 
> traffic in the 700 Mbps stream?
> I'm not 100% sure but I think that is a really good question to ask. 
> Do you know of any good tools that might help inform an answer? I know 
> of iptraf for example, is there one that folks generally prefer the most?
bro ships with a utility called 'trace-summary' that will print some 
useful information about the trace.  It is written in Python (I think 
2.6+ should work fine, though someone can feel free to correct me if I'm 
wrong :).  Example run / output available below at [1], though the 
formatting is terrible without a monospace font.

Note that it's possible to run trace-summary against either bro log 
files (-C option) *or* a captured trace, so capturing a trace is not 
necessarily required in order to use the tool.  Additionally, if it's 
desired to run trace-summary against a trace directly, ipsumdump is 
required (http://www.read.seas.harvard.edu/~kohler/ipsumdump/).

Cheers,
Gilbert

[1] clarkg1-osx:trace-summary clarkg1$ python trace-summary 
~/net.2009.12.06.1159.dmp

 >== Total === 2009-12-06-15-00-10 - 2009-12-07-14-59-40
    - Bytes 150.6m - Payload 144.1m - Pkts 169.0k - Frags  88.5% - 
MBit/s      0.0 -
      Ports        | Sources                   | 
Destinations              | Services           | Protocols |
      80     88.0% | 198.189.255.76      22.5% | 192.168.1.103 45.0% 
|             100.0% | 6   90.5% |         |
      1119   23.0% | 192.168.1.105       13.2% | 192.168.1.105 19.6% 
|                    | 17   8.2% |         |
      1115    9.6% | 192.168.1.103       12.9% | 198.189.255.76 4.1% 
|                    | 1    0.0% |         |
      1817    5.9% | 198.189.255.74      10.8% | 192.168.1.255 2.3% 
|                    |           |         |
      49638   3.5% | 151.207.243.129      3.8% | 198.189.255.74 2.3% 
|                    |           |         |
      1117    3.4% | 192.168.1.1          3.4% | 151.207.243.129 2.2% 
|                    |           |         |
      626     3.4% | 74.125.164.32        2.8% | 192.168.1.1 1.8% 
|                    |           |         |
      137     3.2% | 74.125.164.91        2.1% | 224.0.0.1 1.7% 
|                    |           |         |
      53      2.9% | 192.168.1.104        1.9% | 192.168.1.104 1.5% 
|                    |           |         |
      49378   2.5% | 192.168.1.106        1.6% | 0.0.0.0 1.3% 
|                    |           |         |


First: 2009-12-06-15-00-10 (1260129610.426233) Last: 2009-12-07-14-59-40 
1260215980.426237




From bkellogg at dresser-rand.com  Fri Jun 13 08:05:46 2014
From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN))
Date: Fri, 13 Jun 2014 15:05:46 +0000
Subject: [Bro] Bro hanging on some sensors
In-Reply-To: <D422E894-2B8F-4E87-B223-CFFC85C170A2@geekempire.com>
References: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82181922@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDry05X3RBCt7Ph9SgQa13__+rjg28=VZ9KmPP3KgyYqw@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82182ADD@DRTEXW238VM.na.d-rco.com>
	<D422E894-2B8F-4E87-B223-CFFC85C170A2@geekempire.com>
Message-ID: <BDF3793B557F624699D4C68A3B8FCF82182FFE@DRTEXW238VM.na.d-rco.com>

Thanks



I'm not seeing any rhyme or reason to this.  I've tried commenting out one intel feed at a time and the problem will disappear on some sensors and start on others.  I've commented them all out and enabled one at a time with the same results.



The only thing that works is to disable the intel framework entirely so far.



From: Mike Reeves [mailto:reevesmk at gmail.com] On Behalf Of Mike Reeves
Sent: Friday, June 13, 2014 10:33 AM
To: Kellogg, Brian D (OLN)
Cc: Doug Burks; bro
Subject: Re: [Bro] Bro hanging on some sensors



Bro will sense additions but won't remove old stuff unless it is restarted check out "Putting it all together" in the blog post below.



http://blog.bro.org/2014/01/intelligence-data-and-bro_4980.html







On Jun 13, 2014, at 8:59 AM, Kellogg, Brian D (OLN) <bkellogg at dresser-rand.com<mailto:bkellogg at dresser-rand.com>> wrote:





Yeah they were working up to about a week ago.  I commented out the intel
files on one and that fixed it.  Odd, as they should all be the same with the
replication from the SO server policy directory.  The majority of the sensors
are working fine with the same intel files and I'm seeing new results in ELSA
every day.  Hmmm

I do not have Bro restarting via Onion-Salt.  I was under the understanding
that Bro would periodically sense the file changes and re-read the intel
files.

Thanks,
Brian

Thank you,
Brian Kellogg
Security Analyst; IT Governance, Risk, and Compliance
500 Paul Clark Drive, Olean,  NY 14760
T: (716) 375-3186 | F: (716) 375-3557
www.dresser-rand.com<http://www.dresser-rand.com>     NYSE: DRC

Bringing energy and the environment into harmony(r)
IMPORTANT NOTICE:
This email may be confidential, may be legally privileged, and is for the intended recipient only. Unauthorized access, disclosure, copying, distribution, or reliance on any of it by anyone else is prohibited and may be a criminal offense. Please delete if obtained in error and email confirmation to the sender.


-----Original Message-----
From: Doug Burks [mailto:doug.burks at gmail.com]
Sent: Friday, June 13, 2014 6:56 AM
To: Kellogg, Brian D (OLN)
Cc: bro
Subject: Re: [Bro] Bro hanging on some sensors

Did these sensors work correctly before you added the Intel files?

Have you tried removing the Intel files from the equation?  If not, please try commenting out the entries you added to /opt/bro/share/bro/site/local.bro and then running:
sudo broctl install
sudo broctl restart

Are you running OnionSalt?  Do you have OnionSalt configured to automatically restart Bro?  There are known issues with doing so, which is why OnionSalt does not do so by default.  Please see:
https://code.google.com/p/security-onion/wiki/Salt#Features

This may not be an issue strictly related to Bro, so we may need to move this conversation to the security-onion mailing list:
https://code.google.com/p/security-onion/wiki/MailingLists

On Thu, Jun 12, 2014 at 9:00 AM, Kellogg, Brian D (OLN) <bkellogg at dresser-rand.com<mailto:bkellogg at dresser-rand.com>> wrote:



I use a cron job that runs every 30 minutes to download the intel files to:
/opt/bro/share/bro/policy/.  The cron job uses the mal-dnssearch script.

In each sensor's /opt/bro/share/bro/site/local.bro file I have the below:

# load intelligence framework
@load policy/frameworks/intel/seen
@load policy/frameworks/intel/do_notice #@load
policy/integration/collective-intel
redef Intel::read_files += {
       "/opt/bro/share/bro/policy/xxx.intel",
       "/opt/bro/share/bro/policy/xxx.intel",
       "/opt/bro/share/bro/policy/xxx.intel",
       "/opt/bro/share/bro/policy/xxx.intel",
       "/opt/bro/share/bro/policy/xxx.intel",
       "/opt/bro/share/bro/policy/xxx.intel",
};

In the reporter.log file I am now seeing the below warning on the four
sensors having this issue:
   0.000000    Reporter::WARNING       SumStat key request for the
3Mntn3EPhU3 SumStat uid took longer than 1 minute and was
automatically cancelled.
/opt/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line
209
   0.000000    Reporter::WARNING       SumStat key request for the
2HAva5N4Kqf SumStat uid took longer than 1 minute and was
automatically cancelled.
/opt/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line
209

Thanks,
Brian




--
Doug Burks

_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140613/d1b9458d/attachment.html 

From gc355804 at ohio.edu  Sun Jun 15 23:54:07 2014
From: gc355804 at ohio.edu (Gilbert Clark)
Date: Mon, 16 Jun 2014 02:54:07 -0400
Subject: [Bro] Unexplained Performance Differences Between Like Servers
In-Reply-To: <CANWz5X8w0Anws_A95Pkh2dY3FD4TT4opOwJydSey2jjY-Mv0Vw@mail.gmail.com>
References: <CANWz5X8M7gbE4iNRYbtLfJrJzC7g-BtNVS1s7CNCHkxtEO_Y7Q@mail.gmail.com>	<20140613140910.GI3515@datacomm.albany.edu>	<CANWz5X_3x+Q2Me7OY8HNBz+QG2EYmFcfT+na4QPLr40RDRysnQ@mail.gmail.com>
	<CANWz5X8w0Anws_A95Pkh2dY3FD4TT4opOwJydSey2jjY-Mv0Vw@mail.gmail.com>
Message-ID: <539E948F.2000405@ohio.edu>

Hi Jason:

I believe one way to set a BPF filter is to modify site/local.bro to 
include:

redef cmd_line_bpf_filter = "ip or not ip";

I think there's also a packet filter framework 
(http://www.bro.org/sphinx/scripts/base/frameworks/packet-filter/main.html) 
which supports more elaborate filtering schemes, but I don't really know 
much about it offhand :)

Regarding the "other" traffic being the root cause of the issues: I 
think it's pretty difficult to say.  A few ideas:

* check the size of log files for significant differences.  if http.log 
/ reporter.log / weird.log / etc. is much longer on one system than on 
another, maybe that might be a place to start looking
* try setting a filter to only accept a certain type of traffic (e.g. 
HTTP, SSH) to see relative load for that specific traffic type
* try playing with which scripts bro loads (e.g. tweak local.bro and / 
or try running bro in bare mode with a very small set of loaded scripts) 
to see if that has any effect
* bro can be told to dump performance statistics into a human-readable 
ASCII log by including the "misc/profiling.bro" script: some of the 
information included there might be useful to have
* try capturing a trace and playing that trace back to a standalone bro 
process ... using tools like 'time' and 'perf' could help identify how 
performance changes based on the trace and scripts currently being loaded.
     } this has the benefit of not dropping packets while scripts are 
being tweaked...

As some food for thought: in general, bro does a few things every time 
there's a new packet:

* Retrieve the packet from the NIC
* Dissect the packet and generate events
* Spend time in script-land processing events that have been generated
* Spend time handling administrative overhead (e.g. check timers, check 
triggers)

Thus, in general, making bro go faster is probably going to mean making 
one of those things take less time.

Anyway, hope something in there is useful :)

Cheers,
Gilbert

On 6/13/14, 10:32 AM, Jason Batchelor wrote:
> FWIW:
> I just ran iptraf for a bit on both and one thing really stuck out to me:
> Server A:
> Other IP:      5273     633087        5273 633087           0          0
> Server B:
>  Other IP:    952797    445867K      952797 445867K           0          0
> So server A is seeing 633087 bytes of 'other' traffic, while B is 
> seeing  445867 kilobytes of 'other' traffic. Do you think this other 
> traffic could be the root cause of the issues here? If so, would a bpf 
> filter looking for only tcp/udp/ipv4 traffic be sufficient? How might 
> I apply that within Bro?
> Here is the full view taken some time after the metrics above:
> Server A:
> x               Total      Total    Incoming   Incoming Outgoing   
> Outgoing              x
> x             Packets      Bytes     Packets      Bytes Packets      
> Bytes              x
> x Total:     80187229     51270M    80187229 51270M           
> 0          0              x
> x IPv4:      80187193     50026M    80187193 50026M           
> 0          0              x
> x IPv6:            36       1296          36 1296           0          
> 0              x
> x TCP:       70040618     47342M    70040618 47342M           
> 0          0              x
> x UDP:       10052947      2676M    10052947 2676M           
> 0          0              x
> x ICMP:         85189    6652550       85189 6652550           
> 0          0              x
> x Other IP:      8475    1060993        8475 1060993           
> 0          0
> Server B:
> x               Total      Total    Incoming   Incoming Outgoing   
> Outgoing                   x
> x             Packets      Bytes     Packets      Bytes Packets      
> Bytes                   x
> x Total:     89718860     53317M    89718860 53317M           
> 0          0                   x
> x IPv4:      89712988     51882M    89712988 51882M           
> 0          0                   x
> x IPv6:          5872      51778        5872 51778           
> 0          0                   x
> x TCP:       79615124     49170M    79615124 49170M           
> 0          0                   x
> x UDP:        7627607      1682M     7627607 1682M           
> 0          0                   x
> x ICMP:         86620    5619078       86620 5619078           
> 0          0                   x
> x Other IP:   2389509      1023M     2389509 1023M           
> 0          0                   x
> Many thanks in advance for the quick and helpful replies!
>
>
> On Fri, Jun 13, 2014 at 9:19 AM, Jason Batchelor 
> <jxbatchelor at gmail.com <mailto:jxbatchelor at gmail.com>> wrote:
>
>     Wow, thanks for all the quick replies :)
>     > What versions of Bro, and it is the same for both?
>     I am using the same version of Bro for each server (1.2).
>     > Is the type of traffic in the 600 Mbps stream similar to the
>     type of traffic in the 700 Mbps stream?
>     I'm not 100% sure but I think that is a really good question to
>     ask. Do you know of any good tools that might help inform an
>     answer? I know of iptraf for example, is there one that folks
>     generally prefer the most?
>     > Are you only running 4 workers or did you truncate the output?
>     Yes, I truncated the output to show four workers each (I have 16
>     total).
>     > Are you doing 4 tuple load balancing or 2 tuple load balancing
>     between the two servers?
>     Sorry I am not sure what you mean by this or the implications of
>     one over the other. Is there an easy way I can find out (I am
>     kinda new to this)? I agree with the likelihood that B may be
>     recieving more flows.
>     Thanks!
>     Jason
>
>
>     On Fri, Jun 13, 2014 at 9:09 AM, Justin Azoff <JAzoff at albany.edu
>     <mailto:JAzoff at albany.edu>> wrote:
>
>         On Fri, Jun 13, 2014 at 08:01:54AM -0500, Jason Batchelor wrote:
>         > At the moment Server A is getting about 700MB/s and Server B
>         is getting about
>         > 600Mb/s.
>         >
>         > What I don't understand, is Server A is having several
>         orders of magnatude
>         > better performance compared to Server B?
>         >
>         > TOP from A (included a few bro workers):
>         >
>         > top - 12:48:45 up 1 day, 17:03,  2 users,  load average:
>         5.30, 3.99, 3.13
>         > Tasks: 706 total,  19 running, 687 sleeping,   0 stopped,  
>         0 zombie
>         > Cpu(s): 33.9%us,  6.6%sy,  1.1%ni, 57.2%id,  0.0%wa,
>          0.0%hi,  1.2%si,  0.0%st
>         > Mem:  49376004k total, 33605828k used, 15770176k free,  
>          93100k buffers
>         > Swap:  2621432k total,     9760k used,  2611672k free,
>          9206880k cached
>         >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
>          COMMAND
>         >  5768 root      20   0 1808m 1.7g 519m R 100.0  3.6
>          32:24.92 bro
>         >  5760 root      20   0 1688m 1.6g 519m R 99.7  3.4  34:08.36 bro
>         >  3314 root      20   0 2160m 269m 4764 R 96.1  0.6  30:14.12 bro
>         >  5754 root      20   0 1451m 1.4g 519m R 82.8  2.9  36:40.02 bro
>
>         Server A Bro cpu utilization = 378.6
>
>         > TOP from B (included a few bro workers)
>         >
>         > top - 12:49:33 up 14:24,  2 users,  load average: 10.28,
>         9.31, 8.06
>         > Tasks: 708 total,  25 running, 683 sleeping,   0 stopped,  
>         0 zombie
>         > Cpu(s): 41.6%us,  6.0%sy,  1.0%ni, 50.4%id,  0.0%wa,
>          0.0%hi,  1.1%si,  0.0%st
>         > Mem:  49376004k total, 31837340k used, 17538664k free,  
>         147212k buffers
>         > Swap:  2621432k total,        0k used,  2621432k free,
>         13494332k cached
>         >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
>          COMMAND
>         >  3178 root      20   0 1073m 1.0g 264m R 100.0  2.1
>         401:47.31 bro
>         >  3188 root      20   0  881m 832m 264m R 100.0  1.7
>         377:48.90 bro
>         >  3189 root      20   0 1247m 1.2g 264m R 100.0  2.5
>         403:22.95 bro
>         >  3193 root      20   0  920m 871m 264m R 100.0  1.8
>         429:45.98 bro
>
>         > Both have the same amount of Bro workers. I just do not
>         understand why Server
>         > A is literally half the utilization on top of seeing more
>         traffic? The only
>         > real and consistent difference between the two I see is that
>         server A seems to
>         > have twice the amount of SHR (shared memory) compared to
>         server B.
>
>         Server B Bro cpu utilization = 400%
>
>         Are you only running 4 workers or did you truncate the output?
>          Is that
>         running at 100% 24/7 or does it vary with the traffic?
>
>         Are you doing 4 tuple load balancing or 2 tuple load balancing
>         between
>         the two servers?  Most likely Server B is seeing more flows.
>
>
>         --
>         -- Justin Azoff
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140616/7f24e2b7/attachment.html 

From doug.burks at gmail.com  Tue Jun 17 03:59:42 2014
From: doug.burks at gmail.com (Doug Burks)
Date: Tue, 17 Jun 2014 06:59:42 -0400
Subject: [Bro] Bro hanging on some sensors
In-Reply-To: <BDF3793B557F624699D4C68A3B8FCF82182FFE@DRTEXW238VM.na.d-rco.com>
References: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82181922@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDry05X3RBCt7Ph9SgQa13__+rjg28=VZ9KmPP3KgyYqw@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82182ADD@DRTEXW238VM.na.d-rco.com>
	<D422E894-2B8F-4E87-B223-CFFC85C170A2@geekempire.com>
	<BDF3793B557F624699D4C68A3B8FCF82182FFE@DRTEXW238VM.na.d-rco.com>
Message-ID: <CAK8kjrAzcnZ8R+cp0D2hK=iy3x-AvNJP88xwLGAG2GTqUYWSnQ@mail.gmail.com>

When you comment out intel feeds, are you installing the new config
before restarting?
sudo broctl install
sudo broctl restart

Are you sure your intel feeds are formatted properly?
http://www.bro.org/sphinx-git/frameworks/intel.html

Any other errors or warnings in reporter.log, stderr.log, or stdout.log?

On Fri, Jun 13, 2014 at 11:05 AM, Kellogg, Brian D (OLN)
<bkellogg at dresser-rand.com> wrote:
> Thanks
>
>
>
> I?m not seeing any rhyme or reason to this.  I?ve tried commenting out one
> intel feed at a time and the problem will disappear on some sensors and
> start on others.  I?ve commented them all out and enabled one at a time with
> the same results.
>
>
>
> The only thing that works is to disable the intel framework entirely so far.
>
>
>
> From: Mike Reeves [mailto:reevesmk at gmail.com] On Behalf Of Mike Reeves
> Sent: Friday, June 13, 2014 10:33 AM
>
>
> To: Kellogg, Brian D (OLN)
> Cc: Doug Burks; bro
>
> Subject: Re: [Bro] Bro hanging on some sensors
>
>
>
> Bro will sense additions but won?t remove old stuff unless it is restarted
> check out ?Putting it all together? in the blog post below.
>
>
>
> http://blog.bro.org/2014/01/intelligence-data-and-bro_4980.html
>
>
>
>
>
>
>
> On Jun 13, 2014, at 8:59 AM, Kellogg, Brian D (OLN)
> <bkellogg at dresser-rand.com> wrote:
>
>
>
> Yeah they were working up to about a week ago.  I commented out the intel
> files on one and that fixed it.  Odd, as they should all be the same with
> the
> replication from the SO server policy directory.  The majority of the
> sensors
> are working fine with the same intel files and I'm seeing new results in
> ELSA
> every day.  Hmmm
>
> I do not have Bro restarting via Onion-Salt.  I was under the understanding
> that Bro would periodically sense the file changes and re-read the intel
> files.
>
> Thanks,
> Brian
>
> Thank you,
> Brian Kellogg
> Security Analyst; IT Governance, Risk, and Compliance
> 500 Paul Clark Drive, Olean,  NY 14760
> T: (716) 375-3186 | F: (716) 375-3557
> www.dresser-rand.com     NYSE: DRC
>
> Bringing energy and the environment into harmony?
> IMPORTANT NOTICE:
> This email may be confidential, may be legally privileged, and is for the
> intended recipient only. Unauthorized access, disclosure, copying,
> distribution, or reliance on any of it by anyone else is prohibited and may
> be a criminal offense. Please delete if obtained in error and email
> confirmation to the sender.
>
>
> -----Original Message-----
> From: Doug Burks [mailto:doug.burks at gmail.com]
> Sent: Friday, June 13, 2014 6:56 AM
> To: Kellogg, Brian D (OLN)
> Cc: bro
> Subject: Re: [Bro] Bro hanging on some sensors
>
> Did these sensors work correctly before you added the Intel files?
>
> Have you tried removing the Intel files from the equation?  If not, please
> try commenting out the entries you added to
> /opt/bro/share/bro/site/local.bro and then running:
> sudo broctl install
> sudo broctl restart
>
> Are you running OnionSalt?  Do you have OnionSalt configured to
> automatically restart Bro?  There are known issues with doing so, which is
> why OnionSalt does not do so by default.  Please see:
> https://code.google.com/p/security-onion/wiki/Salt#Features
>
> This may not be an issue strictly related to Bro, so we may need to move
> this conversation to the security-onion mailing list:
> https://code.google.com/p/security-onion/wiki/MailingLists
>
> On Thu, Jun 12, 2014 at 9:00 AM, Kellogg, Brian D (OLN)
> <bkellogg at dresser-rand.com> wrote:
>
> I use a cron job that runs every 30 minutes to download the intel files to:
> /opt/bro/share/bro/policy/.  The cron job uses the mal-dnssearch script.
>
> In each sensor's /opt/bro/share/bro/site/local.bro file I have the below:
>
> # load intelligence framework
> @load policy/frameworks/intel/seen
> @load policy/frameworks/intel/do_notice #@load
> policy/integration/collective-intel
> redef Intel::read_files += {
>        "/opt/bro/share/bro/policy/xxx.intel",
>        "/opt/bro/share/bro/policy/xxx.intel",
>        "/opt/bro/share/bro/policy/xxx.intel",
>        "/opt/bro/share/bro/policy/xxx.intel",
>        "/opt/bro/share/bro/policy/xxx.intel",
>        "/opt/bro/share/bro/policy/xxx.intel",
> };
>
> In the reporter.log file I am now seeing the below warning on the four
> sensors having this issue:
>    0.000000    Reporter::WARNING       SumStat key request for the
> 3Mntn3EPhU3 SumStat uid took longer than 1 minute and was
> automatically cancelled.
> /opt/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line
> 209
>    0.000000    Reporter::WARNING       SumStat key request for the
> 2HAva5N4Kqf SumStat uid took longer than 1 minute and was
> automatically cancelled.
> /opt/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line
> 209
>
> Thanks,
> Brian
>
>
>
>
> --
> Doug Burks
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>



-- 
Doug Burks



From marc.giannoni at noaa.gov  Tue Jun 17 06:34:07 2014
From: marc.giannoni at noaa.gov (Marc Giannoni - NOAA Affiliate)
Date: Tue, 17 Jun 2014 09:34:07 -0400
Subject: [Bro] Persistent Connections
Message-ID: <CA+nZdmjMiGUZRq-V4y8Ma3TwZq0_hH4kS4xYBzfejVKNJo5HoA@mail.gmail.com>

Hello!

I am attempting to use BRO to decode back-end SOAP interactions.  I have a
script to assemble the HTTP bodies, and I am able to log SOAP elements from
the HTTP body events.  My problem is that the back-end uses persistent
connections, and I end up missing transactions whenever I re-start BRO.
 The decoder is never triggered because the connection already exists.

I'm uninterested in the TCP handshake, I just need to capture the HTTP
protocol to a specific back-end server-address and port.



##! This script reassembles full HTTP bodies and raises an event with the

##!

##! complete contents.


> module HTTP;


> export {

    ## Flag that indicates whether to hook request bodies.

    const hook_request_bodies = T &redef;


>     ## Flag that indicates whether to hook reply bodies.

    const hook_reply_bodies = T &redef;


>     ## The pattern applies

    const hook_host_pattern = /.*/ &redef;


>     ## Do not buffer more than this amount of bytes per HTTP message.

    ##const max_body_size = 50000000;

    const max_body_size = 500000;


> }


> ## Users write a handler for this event to process the current HTTP body.

global http_body: event(c: connection, is_orig: bool,

                        data: string, size: count);


> type body_info: record {

    data: string;

    size: count;

};


> global bodies: table[string, bool] of body_info;


> function notify_and_remove_body(c: connection, is_orig: bool)

    {

    local info = bodies[c$uid, is_orig];

    event http_body(c, is_orig, info$data, info$size);

    delete bodies[c$uid, is_orig];

    }


> event http_begin_entity(c: connection, is_orig: bool)

    {

    if ( (is_orig && ! hook_request_bodies) ||

         (! is_orig && ! hook_reply_bodies) )

        return;


> #    if ( hook_host_pattern !in c$http$host )

#        return;


>     local info: body_info;

    info$data = "";

    info$size = 0;

    bodies[c$uid, is_orig] = info;


>     # FIXME: Type inference should work here, but it doesn't.

    #bodies[c$uid, is_orig] = ["", 0];

    }


> event http_entity_data(c: connection, is_orig: bool, length: count,

                       data: string)

    {

    if ( [c$uid, is_orig] !in bodies )

        return;


>     local info = bodies[c$uid, is_orig];

    info$data += data;

    info$size += length;


>     if ( info$size < max_body_size )

        return;


>     notify_and_remove_body(c, is_orig);

    }


> event http_end_entity(c: connection, is_orig: bool)

    {

    if ( [c$uid, is_orig] !in bodies )

        return;


>     notify_and_remove_body(c, is_orig);

    }




Cheers!
Marc Giannoni
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140617/0f34ea78/attachment.html 

From bkellogg at dresser-rand.com  Tue Jun 17 07:30:22 2014
From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN))
Date: Tue, 17 Jun 2014 14:30:22 +0000
Subject: [Bro] Bro hanging on some sensors
In-Reply-To: <CAK8kjrAzcnZ8R+cp0D2hK=iy3x-AvNJP88xwLGAG2GTqUYWSnQ@mail.gmail.com>
References: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82181922@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDry05X3RBCt7Ph9SgQa13__+rjg28=VZ9KmPP3KgyYqw@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82182ADD@DRTEXW238VM.na.d-rco.com>
	<D422E894-2B8F-4E87-B223-CFFC85C170A2@geekempire.com>
	<BDF3793B557F624699D4C68A3B8FCF82182FFE@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrAzcnZ8R+cp0D2hK=iy3x-AvNJP88xwLGAG2GTqUYWSnQ@mail.gmail.com>
Message-ID: <BDF3793B557F624699D4C68A3B8FCF82195DC5@DRTEXW238VM.na.d-rco.com>

Yep, I run the install prior to the restart.  The format in the files is set by the mal-dnssearch script; I checked the files to ensure they are tab delimited and they are.  I get no errors about Bro not seeing the intel files in the correct format.  I have seen these before when creating our own custom intel file.  Our custom intel file is maintained via a python script to ensure that tab delimiting is always maintained.



The one consistent thing I see is that when I stop, install, and then start Bro, Bro starts ok and all the appropriate logs are created.  If I stop and restart Bro again then the only logs I see in the "current" directory are: communication, loaded_scripts, reporter, stderr, and stdout.



I have the intel config loaded at the very end of the local.bro file if that makes a difference.



There are not errors that I see in reporter.log, stderr.log, or stdout.log as shown below.

drc at nsm-che:/nsm/bro/logs/current$ cat stderr.log

/opt/bro/share/bro/securityonion/./bpfconf.bro, line 103: BPFConf filename set: /etc/nsm/nsm-che-eth0/bpf-bro.conf

/opt/bro/share/bro/securityonion/./bpfconf.bro, line 103: BPFConf filename set: /etc/nsm/nsm-che-eth0/bpf-bro.conf

drc at nsm-che:/nsm/bro/logs/current$ cat stdout.log

unlimited

unlimited

unlimited

drc at nsm-che:/nsm/bro/logs/current$ cat reporter.log

#separator \x09

#set_separator  ,

#empty_field    (empty)

#unset_field    -

#path   reporter

#open   2014-06-17-14-20-55

#fields ts      level   message location

#types  time    enum    string  string

0.000000        Reporter::WARNING       Template value remaining in BPFConf filename: /etc/nsm/{{hostname}}-{{interface}}/bpf-bro.conf  /opt/bro/share/bro/securityonion/./bpfconf.bro, line 99

0.000000        Reporter::INFO  BPFConf filename set: /etc/nsm/nsm-che-eth0/bpf-bro.conf        /opt/bro/share/bro/securityonion/./bpfconf.bro, line 103

0.000000        Reporter::INFO  BPFConf filename set: /etc/nsm/nsm-che-eth0/bpf-bro.conf        /opt/bro/share/bro/securityonion/./bpfconf.bro, line 103





-----Original Message-----
From: Doug Burks [mailto:doug.burks at gmail.com]
Sent: Tuesday, June 17, 2014 7:00 AM
To: Kellogg, Brian D (OLN)
Cc: Mike Reeves; bro
Subject: Re: [Bro] Bro hanging on some sensors



When you comment out intel feeds, are you installing the new config before restarting?

sudo broctl install

sudo broctl restart



Are you sure your intel feeds are formatted properly?

http://www.bro.org/sphinx-git/frameworks/intel.html



Any other errors or warnings in reporter.log, stderr.log, or stdout.log?



On Fri, Jun 13, 2014 at 11:05 AM, Kellogg, Brian D (OLN) <bkellogg at dresser-rand.com<mailto:bkellogg at dresser-rand.com>> wrote:

> Thanks

>

>

>

> I?m not seeing any rhyme or reason to this.  I?ve tried commenting out

> one intel feed at a time and the problem will disappear on some

> sensors and start on others.  I?ve commented them all out and enabled

> one at a time with the same results.

>

>

>

> The only thing that works is to disable the intel framework entirely so far.

>

>

>

> From: Mike Reeves [mailto:reevesmk at gmail.com] On Behalf Of Mike Reeves

> Sent: Friday, June 13, 2014 10:33 AM

>

>

> To: Kellogg, Brian D (OLN)

> Cc: Doug Burks; bro

>

> Subject: Re: [Bro] Bro hanging on some sensors

>

>

>

> Bro will sense additions but won?t remove old stuff unless it is

> restarted check out ?Putting it all together? in the blog post below.

>

>

>

> http://blog.bro.org/2014/01/intelligence-data-and-bro_4980.html

>

>

>

>

>

>

>

> On Jun 13, 2014, at 8:59 AM, Kellogg, Brian D (OLN)

> <bkellogg at dresser-rand.com<mailto:bkellogg at dresser-rand.com>> wrote:

>

>

>

> Yeah they were working up to about a week ago.  I commented out the

> intel files on one and that fixed it.  Odd, as they should all be the

> same with the replication from the SO server policy directory.  The

> majority of the sensors are working fine with the same intel files and

> I'm seeing new results in ELSA every day.  Hmmm

>

> I do not have Bro restarting via Onion-Salt.  I was under the

> understanding that Bro would periodically sense the file changes and

> re-read the intel files.

>

> Thanks,

> Brian

>

> Thank you,

> Brian Kellogg

> Security Analyst; IT Governance, Risk, and Compliance

> 500 Paul Clark Drive, Olean,  NY 14760

> T: (716) 375-3186 | F: (716) 375-3557

> www.dresser-rand.com<http://www.dresser-rand.com>     NYSE: DRC

>

> Bringing energy and the environment into harmony? IMPORTANT NOTICE:

> This email may be confidential, may be legally privileged, and is for

> the intended recipient only. Unauthorized access, disclosure, copying,

> distribution, or reliance on any of it by anyone else is prohibited

> and may be a criminal offense. Please delete if obtained in error and

> email confirmation to the sender.

>

>

> -----Original Message-----

> From: Doug Burks [mailto:doug.burks at gmail.com]

> Sent: Friday, June 13, 2014 6:56 AM

> To: Kellogg, Brian D (OLN)

> Cc: bro

> Subject: Re: [Bro] Bro hanging on some sensors

>

> Did these sensors work correctly before you added the Intel files?

>

> Have you tried removing the Intel files from the equation?  If not,

> please try commenting out the entries you added to

> /opt/bro/share/bro/site/local.bro and then running:

> sudo broctl install

> sudo broctl restart

>

> Are you running OnionSalt?  Do you have OnionSalt configured to

> automatically restart Bro?  There are known issues with doing so,

> which is why OnionSalt does not do so by default.  Please see:

> https://code.google.com/p/security-onion/wiki/Salt#Features

>

> This may not be an issue strictly related to Bro, so we may need to

> move this conversation to the security-onion mailing list:

> https://code.google.com/p/security-onion/wiki/MailingLists

>

> On Thu, Jun 12, 2014 at 9:00 AM, Kellogg, Brian D (OLN)

> <bkellogg at dresser-rand.com<mailto:bkellogg at dresser-rand.com>> wrote:

>

> I use a cron job that runs every 30 minutes to download the intel files to:

> /opt/bro/share/bro/policy/.  The cron job uses the mal-dnssearch script.

>

> In each sensor's /opt/bro/share/bro/site/local.bro file I have the below:

>

> # load intelligence framework

> @load policy/frameworks/intel/seen

> @load policy/frameworks/intel/do_notice #@load

> policy/integration/collective-intel

> redef Intel::read_files += {

>        "/opt/bro/share/bro/policy/xxx.intel",

>        "/opt/bro/share/bro/policy/xxx.intel",

>        "/opt/bro/share/bro/policy/xxx.intel",

>        "/opt/bro/share/bro/policy/xxx.intel",

>        "/opt/bro/share/bro/policy/xxx.intel",

>        "/opt/bro/share/bro/policy/xxx.intel",

> };

>

> In the reporter.log file I am now seeing the below warning on the four

> sensors having this issue:

>    0.000000    Reporter::WARNING       SumStat key request for the

> 3Mntn3EPhU3 SumStat uid took longer than 1 minute and was

> automatically cancelled.

> /opt/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line

> 209

>    0.000000    Reporter::WARNING       SumStat key request for the

> 2HAva5N4Kqf SumStat uid took longer than 1 minute and was

> automatically cancelled.

> /opt/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line

> 209

>

> Thanks,

> Brian

>

>

>

>

> --

> Doug Burks

>

> _______________________________________________

> Bro mailing list

> bro at bro-ids.org<mailto:bro at bro-ids.org>

> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

>

>







--

Doug Burks

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140617/f5c77aca/attachment.html 

From jdopheid at illinois.edu  Tue Jun 17 08:15:01 2014
From: jdopheid at illinois.edu (Dopheide, Jeannette M)
Date: Tue, 17 Jun 2014 15:15:01 +0000
Subject: [Bro] Bro v2.3 is released
Message-ID: <CFC5C5A5.49DE%jdopheid@illinois.edu>

Bro Community,

Bro v2.3 is released!  To learn more about the new version see our blog
post: 
http://blog.bro.org/2014/06/bro-23-release.html


Download the new version:
http://bro.org/download/index.html


Don't forget to sign up for this year's BroCon:
regonline.com/brocon2014

Thanks to everyone who helped make this release possible.  We extend
special thanks to the community for their feedback and support.
  


The Bro Team

------

Jeannette M. Dopheide
Bro Outreach Coordinator
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign






From sgibbons at clicksecurity.com  Tue Jun 17 08:21:16 2014
From: sgibbons at clicksecurity.com (Scott Gibbons)
Date: Tue, 17 Jun 2014 10:21:16 -0500
Subject: [Bro] Script profiling?
Message-ID: <CAE1e=UL+SKMh1XM86DwMMtpiq=X3bUe64YG8-kxqyf+p-XEe8Q@mail.gmail.com>

I'm rather new to Bro and have been asked to investigate why we're not
seeing very good throughput from Bro.  We have added a significant number
of scripts.

Are there any good script profilers out there that would tell me that I'm
spending a lot of time in some script?  I've considered using vTune or some
other profiler, but that only gives me second-order profiling information.

Thanks,
--Scott
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140617/f53c0199/attachment.html 

From doug.burks at gmail.com  Wed Jun 18 03:55:40 2014
From: doug.burks at gmail.com (Doug Burks)
Date: Wed, 18 Jun 2014 06:55:40 -0400
Subject: [Bro] Bro hanging on some sensors
In-Reply-To: <BDF3793B557F624699D4C68A3B8FCF82195DC5@DRTEXW238VM.na.d-rco.com>
References: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82181922@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDry05X3RBCt7Ph9SgQa13__+rjg28=VZ9KmPP3KgyYqw@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82182ADD@DRTEXW238VM.na.d-rco.com>
	<D422E894-2B8F-4E87-B223-CFFC85C170A2@geekempire.com>
	<BDF3793B557F624699D4C68A3B8FCF82182FFE@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrAzcnZ8R+cp0D2hK=iy3x-AvNJP88xwLGAG2GTqUYWSnQ@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82195DC5@DRTEXW238VM.na.d-rco.com>
Message-ID: <CAK8kjrCt4nMqaomyT+AqyHrQmxxA8kBfqiTEt5gUnsCobdZ7WA@mail.gmail.com>

On Tue, Jun 17, 2014 at 10:30 AM, Kellogg, Brian D (OLN)
<bkellogg at dresser-rand.com> wrote:
> The one consistent thing I see is that when I stop, install, and then start
> Bro, Bro starts ok and all the appropriate logs are created.  If I stop and
> restart Bro again then the only logs I see in the "current" directory are:
> communication, loaded_scripts, reporter, stderr, and stdout.

Yep, I've seen this issue before.  I'm not sure if it's an issue with
the Security-Onion-specific scripts that we load into Bro, or if it
could be an issue with Bro itself.

Has anybody else seen this issue on a vanilla Bro installation (not
using Security Onion)?

-- 
Doug Burks


From jlay at slave-tothe-box.net  Wed Jun 18 07:09:17 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Wed, 18 Jun 2014 08:09:17 -0600
Subject: [Bro] Properly disabling certain rules
Message-ID: <93b9f3149623a7a8dd122be189d7796d@localhost>

Team,

So...after upgrading to Bro 2.3, syslog and ssl have returned, which I 
do not want to see.  I commented them out in init-default.bro, which is 
not the right way to go I know.  How can I disable these in my 
local.bro?  Thank you.

James


From jlay at slave-tothe-box.net  Wed Jun 18 07:53:18 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Wed, 18 Jun 2014 08:53:18 -0600
Subject: [Bro] Properly disabling certain rules
In-Reply-To: <93b9f3149623a7a8dd122be189d7796d@localhost>
References: <93b9f3149623a7a8dd122be189d7796d@localhost>
Message-ID: <2b8481c41da02eae51fba1efb446e5a6@localhost>

On 2014-06-18 08:09, James Lay wrote:
> Team,
>
> So...after upgrading to Bro 2.3, syslog and ssl have returned, which 
> I
> do not want to see.  I commented them out in init-default.bro, which 
> is
> not the right way to go I know.  How can I disable these in my
> local.bro?  Thank you.
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

Heh...got it already:

event bro_init()
        {
        Log::disable_stream(Syslog::LOG);
        }

Thanks all.

James


From vlad at grigorescu.org  Wed Jun 18 07:56:30 2014
From: vlad at grigorescu.org (Vlad Grigorescu)
Date: Wed, 18 Jun 2014 10:56:30 -0400
Subject: [Bro] Properly disabling certain rules
In-Reply-To: <93b9f3149623a7a8dd122be189d7796d@localhost>
References: <93b9f3149623a7a8dd122be189d7796d@localhost>
Message-ID: <465AE4E5-3DDD-4DDE-9073-7496ABAC7003@grigorescu.org>

Hi James,

Just as a matter of terminology, these aren't rules, but analyzers. :-)

Try something like this to your local.bro:

> event bro_init() {
>      Analyzer::disable_analyzer(Analyzer::ANALYZER_SSL);
>      Analyzer::disable_analyzer(Analyzer::ANALYZER_SYSLOG);
> }

 --Vlad


On Jun 18, 2014, at 10:09 AM, James Lay <jlay at slave-tothe-box.net> wrote:

> Team,
> 
> So...after upgrading to Bro 2.3, syslog and ssl have returned, which I 
> do not want to see.  I commented them out in init-default.bro, which is 
> not the right way to go I know.  How can I disable these in my 
> local.bro?  Thank you.
> 
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140618/8cb75203/attachment.bin 

From jlay at slave-tothe-box.net  Wed Jun 18 08:01:19 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Wed, 18 Jun 2014 09:01:19 -0600
Subject: [Bro] Properly disabling certain rules
In-Reply-To: <465AE4E5-3DDD-4DDE-9073-7496ABAC7003@grigorescu.org>
References: <93b9f3149623a7a8dd122be189d7796d@localhost>
	<465AE4E5-3DDD-4DDE-9073-7496ABAC7003@grigorescu.org>
Message-ID: <cff330ad38be272a1e1f37f115d505ae@localhost>

On 2014-06-18 08:56, Vlad Grigorescu wrote:
> Hi James,
>
> Just as a matter of terminology, these aren't rules, but analyzers. 
> :-)
>
> Try something like this to your local.bro:
>
>> event bro_init() {
>>      Analyzer::disable_analyzer(Analyzer::ANALYZER_SSL);
>>      Analyzer::disable_analyzer(Analyzer::ANALYZER_SYSLOG);
>> }
>
>  --Vlad
>
>
> On Jun 18, 2014, at 10:09 AM, James Lay <jlay at slave-tothe-box.net> 
> wrote:
>
>> Team,
>>
>> So...after upgrading to Bro 2.3, syslog and ssl have returned, which 
>> I
>> do not want to see.  I commented them out in init-default.bro, which 
>> is
>> not the right way to go I know.  How can I disable these in my
>> local.bro?  Thank you.
>>
>> James
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

Thanks for the clarification Vlad...helps if I at least SOUND like I 
know what I'm talking about :D

James


From wsladekjr at hotmail.com  Wed Jun 18 08:23:43 2014
From: wsladekjr at hotmail.com (Ward Sladek)
Date: Wed, 18 Jun 2014 10:23:43 -0500
Subject: [Bro] Properly disabling certain rules
In-Reply-To: <93b9f3149623a7a8dd122be189d7796d@localhost>
References: <93b9f3149623a7a8dd122be189d7796d@localhost>
Message-ID: <BLU172-W43A606D69C1A05885580EEB4100@phx.gbl>

I eliminate syslog via the following in my local.bro:

# Disable Syslog 
event bro_init()
    {
    Log::disable_stream(Syslog::LOG);
    }



Not sure if that is the recommended way, but it works.



> To: bro at bro.org
> Date: Wed, 18 Jun 2014 08:09:17 -0600
> From: jlay at slave-tothe-box.net
> Subject: [Bro] Properly disabling certain rules
> 
> Team,
> 
> So...after upgrading to Bro 2.3, syslog and ssl have returned, which I 
> do not want to see.  I commented them out in init-default.bro, which is 
> not the right way to go I know.  How can I disable these in my 
> local.bro?  Thank you.
> 
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140618/d3d8401b/attachment.html 

From dopheide at gmail.com  Wed Jun 18 10:13:26 2014
From: dopheide at gmail.com (Mike Dopheide)
Date: Wed, 18 Jun 2014 12:13:26 -0500
Subject: [Bro] scheduled tasks on existing pcaps
Message-ID: <CAPy2kFY0zyh4B3XXoM+=WHRBPVowha2CE_3x8GrS3LxUScCJ_Q@mail.gmail.com>

Howdy,

We're doing some fairly simple analysis regarding concurrent connections on
existing pcaps.  Bro basically does all of that for us, but I'm hoping to
output the current number of active connections every few seconds.

Do Bro's scheduled tasks run in real time or network time when a pcap is
passed to it?  I'm assuming real time, so my next question would be what's
the best way to output a regular status in original network time?   I could
fake it with tcpreplay, but I'd like to avoid that.

Thanks,
Dop
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140618/b6550ac9/attachment.html 

From robin at icir.org  Wed Jun 18 10:57:38 2014
From: robin at icir.org (Robin Sommer)
Date: Wed, 18 Jun 2014 10:57:38 -0700
Subject: [Bro] scheduled tasks on existing pcaps
In-Reply-To: <CAPy2kFY0zyh4B3XXoM+=WHRBPVowha2CE_3x8GrS3LxUScCJ_Q@mail.gmail.com>
References: <CAPy2kFY0zyh4B3XXoM+=WHRBPVowha2CE_3x8GrS3LxUScCJ_Q@mail.gmail.com>
Message-ID: <20140618175738.GD85923@icir.org>



On Wed, Jun 18, 2014 at 12:13 -0500, Mike Dopheide wrote:

> Do Bro's scheduled tasks run in real time or network time when a pcap is
> passed to it?  I'm assuming real time,

No, it's network time actually, which drives pretty much everything in
Bro so that output doesn't differ when running live vs from a trace.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 *     robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 * www.icir.org/robin


From michael.wenthold at gmail.com  Wed Jun 18 11:20:59 2014
From: michael.wenthold at gmail.com (Michael Wenthold)
Date: Wed, 18 Jun 2014 18:20:59 +0000
Subject: [Bro] port field in file input
Message-ID: <CAMU2yhS_VSkwcnZz6dZTu=Og6R0xLS5H=yPobXdrjq==4Kebkw@mail.gmail.com>

Is there anything that prevents me from loading data from a file with an
idx type "port"?

I'm trying to put the data in a file:

#fields tgt_port        tgt_subnet
#types  port    subnet
5222/tcp        74.127.0.0/24

but when my script tries to load the file I get the following error:
Input::READER_ASCII: Number '5222/tcp' contained non-numeric trailing
characters. Ignored trailing characters '/tcp'

The file reader table idx types are all set to type "port".  I know I can
just put all of the information in the script instead of offloading it to a
file, but I don't see anything that would prevent me from offloading the
information to a file.  I feel like I'm missing something really basic, but
I'm definitely missing something.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140618/0c6c779b/attachment.html 

From jamie.gausemel at gmail.com  Wed Jun 18 12:21:57 2014
From: jamie.gausemel at gmail.com (Jamie Gausemel)
Date: Wed, 18 Jun 2014 15:21:57 -0400
Subject: [Bro] Intel Framework Not Generating Intel Log
Message-ID: <CAHbdUtV_CTtj3iZ6N4B+r4VrR-C7AnTQVskz1djvyWPSND9bZQ@mail.gmail.com>

I am running Bro 2.2 on Security Onion

==========================
sensor1 at sensor1:~/tmp$ sudo tail /opt/bro/share/bro/site/local.bro
#@load apt1

# Bro Intelligence Framework
@load frameworks/intel/seen
@load frameworks/intel/do_notice

# Load Intel Feeds For Bro Intelligence Framework
redef Intel::read_files += {
        "/opt/bro/intel_feeds/test.txt"
};

==========================

sensor1 at sensor1:~/tmp$ sudo cat /opt/bro/intel_feeds/test.txt
#fields indicator       indicator_type  meta.source
216.146.46.11   Intel::ADDR     jamie

==========================

sensor1 at sensor1:~/tmp$ sudo bro -r test.pcap local

generates the following logs:

capture_loss.log  conn.log  loaded_scripts.log  packet_filter.log
 reporter.log

==========================

reporter.log contains:

sensor1 at sensor1:~/tmp$ cat reporter.log
#separator \x09
#set_separator  ,
#empty_field    (empty)
#unset_field    -
#path   reporter
#open   2014-06-18-18-13-17
#fields ts      level   message location
#types  time    enum    string  string
0.000000        Reporter::WARNING       Template value remaining in BPFConf
filename: /etc/nsm/{{hostname}}-{{interface}}/bpf-bro.conf
 /opt/bro/share/bro/securityonion/./bpfconf.bro, line 99
0.000000        Reporter::INFO  BPFConf filename set:
/etc/nsm/sensor1-eth1/bpf-bro.conf
/opt/bro/share/bro/securityonion/./bpfconf.bro, line 103
0.000000        Reporter::INFO  BPFConf filename set:
/etc/nsm/sensor1-eth1/bpf-bro.conf
/opt/bro/share/bro/securityonion/./bpfconf.bro, line 103
1403105006.674182       Reporter::INFO  Failed to open GeoIP database:
/usr/share/GeoIP/GeoIPCity.dat   (empty)
1403105006.674182       Reporter::INFO  Fell back to GeoIP Country database
    (empty)
1403105006.674182       Reporter::INFO  Failed to open GeoIP database:
/usr/share/GeoIP/GeoIPCityv6.dat (empty)
#close  2014-06-18-18-13-17

==========================

test.pcap contains ICMP traffic to 216.146.46.11, and this traffic shows up
in conn.log; however, I am not getting the expected intel.log. The test.txt
is tab delimited, and was created with pico.

Any ideas? Are there other logs I can look at for clues? Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140618/0bc12574/attachment.html 

From asharma at lbl.gov  Wed Jun 18 12:35:15 2014
From: asharma at lbl.gov (Aashish Sharma)
Date: Wed, 18 Jun 2014 12:35:15 -0700
Subject: [Bro] port field in file input
In-Reply-To: <CAMU2yhS_VSkwcnZz6dZTu=Og6R0xLS5H=yPobXdrjq==4Kebkw@mail.gmail.com>
References: <CAMU2yhS_VSkwcnZz6dZTu=Og6R0xLS5H=yPobXdrjq==4Kebkw@mail.gmail.com>
Message-ID: <20140618193514.GC7782@yaksha.lbl.gov>

You have to format the input file in this manner, eg: 

#fields exclude_ip     exclude_port   t       comment
1.2.3.4   25      tcp     mail


and define the input structs as :

type ipportexclude_Idx: record {
    exclude_ip: addr;
    exclude_port: port &type_column="t";
};

type ipportexclude_Val: record {
    exclude_ip: addr;
    exclude_port: port &type_column="t" ;
    comment: string &optional ;
} ;

Hope this helps, 
Aashish 


On Wed, Jun 18, 2014 at 06:20:59PM +0000, Michael Wenthold wrote:
> 
>    Is there anything that prevents me from loading data from a file with an idx
>    type "port"?
>    I'm trying to put the data in a file:
>    #fields tgt_port? ? ? ? ? ? ?  tgt_subnet
>    #types?  port? ? ?  subnet
>    5222/tcp? ? ? ? ? ? ?  [1]74.127.0.0/24
>    but when my script tries to load the file I get the following error:
>    Input::READER_ASCII:  Number '5222/tcp' contained non-numeric trailing
>    characters. Ignored trailing characters '/tcp'
>    The file reader table idx types are all set to type "port".?  I know I can
>    just put all of the information in the script instead of offloading it to a
>    file, but I don't see anything that would prevent me from offloading the
>    information to a file.?  I feel like I'm missing something really basic, but
>    I'm definitely missing something.
> 
> References
> 
>    1. http://74.127.0.0/24

> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


-- 
Aashish Sharma	(asharma at lbl.gov) 				 
Cyber Security, 
Lawrence Berkeley National Laboratory  
http://go.lbl.gov/pgp-aashish 
Office: (510)-495-2680  Cell: (510)-612-7971


From ytl at slac.stanford.edu  Wed Jun 18 16:02:07 2014
From: ytl at slac.stanford.edu (=?utf-8?Q?Li=2C_Yee-Ting?=)
Date: Wed, 18 Jun 2014 16:02:07 -0700
Subject: [Bro] bro cluster with pf ring dna+libzero
Message-ID: <A0EB5C5EAF804AA29B073554D3107C54@slac.stanford.edu>

we're deploying a new bro cluster and am a huge newbie on all of this; so please excuse my ignorance. i have yet to actually start capturing on the cluster (awaiting delivery of a front-end device)

on each worker i have the dna+libzero ixgbe driver installed and insmodded. so i run:

$ sudo insmod pf_ring.ko enable_tx_capture=0 min_num_slots=32768
$ sudo insmod ixgbe.ko RSS=1,1,1,1 num_rx_slots=32768 mtu=9000



$ sudo /usr/sbin/setcap cap_net_raw,cap_net_admin=eip /usr/bin/pfdnacluster_master
$ /usr/bin/pfdnacluster_master -d -P /var/run/pfdnacluster-dna0.pid -D bromaint -c 0 -i dna0 -n 10

i do the setcap as i am running bro as non-root user. looks good?

$ cat /proc/net/pf_ring/13979-dna0.1
Bound Device(s) :
Active : 1
Breed : DNA
Sampling Rate : 1
Capture Direction : RX+TX
Socket Mode : RX only
Appl. Name : dna-cluster-0-socket-0
IP Defragment : No
BPF Filtering : Disabled
# Sw Filt. Rules : 0
# Hw Filt. Rules : 0
Poll Pkt Watermark : 128
Num Poll Calls : 0
Channel Id : 0
Num RX Slots : 32768
Num TX Slots : 8192
Tot Memory : 672399360 bytes
Cluster: Tot Recvd : 11
Cluster: Tot Sent : 0



then on my manager i have the following nodes.cfg:

[manager]
type=manager
host=sec-broman

[proxy-0]
type=proxy
host=sec-broman

[proxy-1]
type=proxy
host=sec-broman

[sec-bro01-0]
type=worker
host=sec-bro01
interface=dnacluster:0
lb_method=pf_ring
lb_procs=10



using bro 2.3; so i believe the lb_pf_ring.py script understands the dnacluster interface spec.

so i do an 'broctl install' (as user bromaint) from the manager, then log onto my worker and run

$ sudo /usr/sbin/setcap cap_net_raw,cap_net_admin=eip /opt/bro/bin/capstats
$ sudo /usr/sbin/setcap cap_net_raw,cap_net_admin=eip /opt/bro/bin/bro



then a 'broctl start' on the manager. everything looks fine so far? then i run 'broctl capstats' and i get:

Interface kpps mbps (10s average)
----------------------------------------
sec-bro01-0-9: capstats failed (error: dnacluster:0: No such device exists (SIOCGIFHWADDR: No such device))



looking at proc for the pid of that bro instance, i get:

$ ps aux | grep sec-bro01-0-9
bromaint 14696 0.0 0.0 108128 1496 ? S 15:28 0:00 bash /opt/bro/share/broctl/scripts/run-bro -1 -i dnacluster:0 at 9 -U .status -p broctl -p broctl-live -p local -p sec-bro01-0-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

bromaint 14778 25.1  0.0 157736 56320 ?        S    15:28   0:02 /opt/bro/bin/bro -i dnacluster:0 at 9 -U .status -p broctl -p broctl-live -p local -p sec-bro01-0-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
bromaint 14846 14.3  0.0 161832 52996 ?        SN   15:28   0:01 /opt/bro/bin/bro -i dnacluster:0 at 9 -U .status -p broctl -p broctl-live -p local -p sec-bro01-0-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

$ cat /proc/net/pf_ring/14778-none.7
Bound Device(s) :
Active : 1
Breed : Non-DNA
Sampling Rate : 1
Capture Direction : RX+TX
Socket Mode : RX+TX
Appl. Name : bro-dnacluster:0 at 9
IP Defragment : No
BPF Filtering : Enabled
# Sw Filt. Rules : 0
# Hw Filt. Rules : 0
Poll Pkt Watermark : 1
Num Poll Calls : 2562490



what gives???

if i manually kill the bro process on the worker and rerun capstats, i get:

Interface kpps mbps (10s average)
----------------------------------------
sec-bro01/dnacluster:0 0.0 0.0
Total 0.0 0.0


also, if i were to change the lb_procs to less than that of the pfdnacluster number of workers (-n), everything (seems to) work fine (bear in mind i'm not capturing any traffic at the moment). but would i loose any data? i'm using pf_ring 6.0.1.


thanks,

Yee.  





From gfaulkner.nsm at gmail.com  Wed Jun 18 17:27:18 2014
From: gfaulkner.nsm at gmail.com (Gary Faulkner)
Date: Wed, 18 Jun 2014 19:27:18 -0500
Subject: [Bro] bro cluster with pf ring dna+libzero
In-Reply-To: <A0EB5C5EAF804AA29B073554D3107C54@slac.stanford.edu>
References: <A0EB5C5EAF804AA29B073554D3107C54@slac.stanford.edu>
Message-ID: <53A22E66.8040507@gmail.com>

Hello,

Capstats is a separate application as far as pfdnacluster_master is 
concerned. You can tell pfdnacluster_master that you want to send the 
same traffic to another application using the -n flag by using a "," and 
then specifying how many instances of the second app you intend to run. 
When you call pfdnacluster_master try "-n 10,1" instead of "-n 10". You 
actually want to run two applications against the same traffic, but the 
second app, capstats, will only run one process that needs to consume 
all of the traffic instead of having slices of traffic load balanced 
between multiple processes.

Regards,
Gary



On 6/18/2014 6:02 PM, Li, Yee-Ting wrote:
> we're deploying a new bro cluster and am a huge newbie on all of this; so please excuse my ignorance. i have yet to actually start capturing on the cluster (awaiting delivery of a front-end device)
>
> on each worker i have the dna+libzero ixgbe driver installed and insmodded. so i run:
>
> $ sudo insmod pf_ring.ko enable_tx_capture=0 min_num_slots=32768
> $ sudo insmod ixgbe.ko RSS=1,1,1,1 num_rx_slots=32768 mtu=9000
>
>
>
> $ sudo /usr/sbin/setcap cap_net_raw,cap_net_admin=eip /usr/bin/pfdnacluster_master
> $ /usr/bin/pfdnacluster_master -d -P /var/run/pfdnacluster-dna0.pid -D bromaint -c 0 -i dna0 -n 10
>
> i do the setcap as i am running bro as non-root user. looks good?
>
> $ cat /proc/net/pf_ring/13979-dna0.1
> Bound Device(s) :
> Active : 1
> Breed : DNA
> Sampling Rate : 1
> Capture Direction : RX+TX
> Socket Mode : RX only
> Appl. Name : dna-cluster-0-socket-0
> IP Defragment : No
> BPF Filtering : Disabled
> # Sw Filt. Rules : 0
> # Hw Filt. Rules : 0
> Poll Pkt Watermark : 128
> Num Poll Calls : 0
> Channel Id : 0
> Num RX Slots : 32768
> Num TX Slots : 8192
> Tot Memory : 672399360 bytes
> Cluster: Tot Recvd : 11
> Cluster: Tot Sent : 0
>
>
>
> then on my manager i have the following nodes.cfg:
>
> [manager]
> type=manager
> host=sec-broman
>
> [proxy-0]
> type=proxy
> host=sec-broman
>
> [proxy-1]
> type=proxy
> host=sec-broman
>
> [sec-bro01-0]
> type=worker
> host=sec-bro01
> interface=dnacluster:0
> lb_method=pf_ring
> lb_procs=10
>
>
>
> using bro 2.3; so i believe the lb_pf_ring.py script understands the dnacluster interface spec.
>
> so i do an 'broctl install' (as user bromaint) from the manager, then log onto my worker and run
>
> $ sudo /usr/sbin/setcap cap_net_raw,cap_net_admin=eip /opt/bro/bin/capstats
> $ sudo /usr/sbin/setcap cap_net_raw,cap_net_admin=eip /opt/bro/bin/bro
>
>
>
> then a 'broctl start' on the manager. everything looks fine so far? then i run 'broctl capstats' and i get:
>
> Interface kpps mbps (10s average)
> ----------------------------------------
> sec-bro01-0-9: capstats failed (error: dnacluster:0: No such device exists (SIOCGIFHWADDR: No such device))
>
>
>
> looking at proc for the pid of that bro instance, i get:
>
> $ ps aux | grep sec-bro01-0-9
> bromaint 14696 0.0 0.0 108128 1496 ? S 15:28 0:00 bash /opt/bro/share/broctl/scripts/run-bro -1 -i dnacluster:0 at 9 -U .status -p broctl -p broctl-live -p local -p sec-bro01-0-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
>
> bromaint 14778 25.1  0.0 157736 56320 ?        S    15:28   0:02 /opt/bro/bin/bro -i dnacluster:0 at 9 -U .status -p broctl -p broctl-live -p local -p sec-bro01-0-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
> bromaint 14846 14.3  0.0 161832 52996 ?        SN   15:28   0:01 /opt/bro/bin/bro -i dnacluster:0 at 9 -U .status -p broctl -p broctl-live -p local -p sec-bro01-0-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
>
> $ cat /proc/net/pf_ring/14778-none.7
> Bound Device(s) :
> Active : 1
> Breed : Non-DNA
> Sampling Rate : 1
> Capture Direction : RX+TX
> Socket Mode : RX+TX
> Appl. Name : bro-dnacluster:0 at 9
> IP Defragment : No
> BPF Filtering : Enabled
> # Sw Filt. Rules : 0
> # Hw Filt. Rules : 0
> Poll Pkt Watermark : 1
> Num Poll Calls : 2562490
>
>
>
> what gives???
>
> if i manually kill the bro process on the worker and rerun capstats, i get:
>
> Interface kpps mbps (10s average)
> ----------------------------------------
> sec-bro01/dnacluster:0 0.0 0.0
> Total 0.0 0.0
>
>
> also, if i were to change the lb_procs to less than that of the pfdnacluster number of workers (-n), everything (seems to) work fine (bear in mind i'm not capturing any traffic at the moment). but would i loose any data? i'm using pf_ring 6.0.1.
>
>
> thanks,
>
> Yee.
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From jxbatchelor at gmail.com  Wed Jun 18 18:10:27 2014
From: jxbatchelor at gmail.com (Jason Batchelor)
Date: Wed, 18 Jun 2014 20:10:27 -0500
Subject: [Bro] Unexplained Performance Differences Between Like Servers
In-Reply-To: <539E948F.2000405@ohio.edu>
References: <CANWz5X8M7gbE4iNRYbtLfJrJzC7g-BtNVS1s7CNCHkxtEO_Y7Q@mail.gmail.com>
	<20140613140910.GI3515@datacomm.albany.edu>
	<CANWz5X_3x+Q2Me7OY8HNBz+QG2EYmFcfT+na4QPLr40RDRysnQ@mail.gmail.com>
	<CANWz5X8w0Anws_A95Pkh2dY3FD4TT4opOwJydSey2jjY-Mv0Vw@mail.gmail.com>
	<539E948F.2000405@ohio.edu>
Message-ID: <CANWz5X8XR_DeAcHi6A2DWVzHm5Rm-QE4NMo6SG=wBTyuzjBYFA@mail.gmail.com>

Thanks for the reply Gilbert! I will take a closer look into these items
tomorrow but off the top of my head, I do not recall there being any great
difference in file size in the log files. On the surface (using iptraf), it
seems like there is a significant amount of non-ip traffic so I modified
local.bro to include the following:

redef cmd_line_bpf_filter = "ip"

In hopes that it has the desired effect.

One other question I had was the effect of implementing TCP sequence
randomization on performance (if it was enabled on an ASA for example)?
What impact would this have on flows (presumably a large increase)? How
might I best quantify the amount of flows being processed compared to the
other server?

Sorry for all the questions, I am very much a novice at this, but very
willing to learn so I appreciate the help so far!



On Mon, Jun 16, 2014 at 1:54 AM, Gilbert Clark <gc355804 at ohio.edu> wrote:

>  Hi Jason:
>
> I believe one way to set a BPF filter is to modify site/local.bro to
> include:
>
> redef cmd_line_bpf_filter = "ip or not ip";
>
> I think there's also a packet filter framework (
> http://www.bro.org/sphinx/scripts/base/frameworks/packet-filter/main.html)
> which supports more elaborate filtering schemes, but I don't really know
> much about it offhand :)
>
> Regarding the "other" traffic being the root cause of the issues: I think
> it's pretty difficult to say.  A few ideas:
>
> * check the size of log files for significant differences.  if http.log /
> reporter.log / weird.log / etc. is much longer on one system than on
> another, maybe that might be a place to start looking
> * try setting a filter to only accept a certain type of traffic (e.g.
> HTTP, SSH) to see relative load for that specific traffic type
> * try playing with which scripts bro loads (e.g. tweak local.bro and / or
> try running bro in bare mode with a very small set of loaded scripts) to
> see if that has any effect
> * bro can be told to dump performance statistics into a human-readable
> ASCII log by including the "misc/profiling.bro" script: some of the
> information included there might be useful to have
> * try capturing a trace and playing that trace back to a standalone bro
> process ... using tools like 'time' and 'perf' could help identify how
> performance changes based on the trace and scripts currently being loaded.
>     } this has the benefit of not dropping packets while scripts are being
> tweaked...
>
> As some food for thought: in general, bro does a few things every time
> there's a new packet:
>
> * Retrieve the packet from the NIC
> * Dissect the packet and generate events
> * Spend time in script-land processing events that have been generated
> * Spend time handling administrative overhead (e.g. check timers, check
> triggers)
>
> Thus, in general, making bro go faster is probably going to mean making
> one of those things take less time.
>
> Anyway, hope something in there is useful :)
>
> Cheers,
> Gilbert
>
>
> On 6/13/14, 10:32 AM, Jason Batchelor wrote:
>
>  FWIW:
>
> I just ran iptraf for a bit on both and one thing really stuck out to me:
>
> Server A:
> Other IP:      5273     633087        5273     633087           0
> 0
>
> Server B:
>  Other IP:    952797    445867K      952797    445867K
> 0          0
>
> So server A is seeing 633087 bytes of 'other' traffic, while B is seeing
> 445867 kilobytes of 'other' traffic. Do you think this other traffic could
> be the root cause of the issues here? If so, would a bpf filter looking for
> only tcp/udp/ipv4 traffic be sufficient? How might I apply that within Bro?
>
> Here is the full view taken some time after the metrics above:
>
> Server A:
>
> x               Total      Total    Incoming   Incoming    Outgoing
> Outgoing              x
> x             Packets      Bytes     Packets      Bytes     Packets
> Bytes              x
> x Total:     80187229     51270M    80187229     51270M
> 0          0              x
> x IPv4:      80187193     50026M    80187193     50026M
> 0          0              x
> x IPv6:            36       1296          36       1296
> 0          0              x
> x TCP:       70040618     47342M    70040618     47342M
> 0          0              x
> x UDP:       10052947      2676M    10052947      2676M
> 0          0              x
> x ICMP:         85189    6652550       85189    6652550
> 0          0              x
> x Other IP:      8475    1060993        8475    1060993
> 0          0
>
> Server B:
>
> x               Total      Total    Incoming   Incoming    Outgoing
> Outgoing                   x
> x             Packets      Bytes     Packets      Bytes     Packets
> Bytes                   x
> x Total:     89718860     53317M    89718860     53317M
> 0          0                   x
> x IPv4:      89712988     51882M    89712988     51882M
> 0          0                   x
> x IPv6:          5872      51778        5872      51778
> 0          0                   x
> x TCP:       79615124     49170M    79615124     49170M
> 0          0                   x
> x UDP:        7627607      1682M     7627607      1682M
> 0          0                   x
> x ICMP:         86620    5619078       86620    5619078
> 0          0                   x
> x Other IP:   2389509      1023M     2389509      1023M
> 0          0                   x
>  Many thanks in advance for the quick and helpful replies!
>
>
> On Fri, Jun 13, 2014 at 9:19 AM, Jason Batchelor <jxbatchelor at gmail.com>
> wrote:
>
>>  Wow, thanks for all the quick replies :)
>>
>> > What versions of Bro, and it is the same for both?
>>
>>  I am using the same version of Bro for each server (1.2).
>>
>> > Is the type of traffic in the 600 Mbps stream similar to the type of
>> traffic in the 700 Mbps stream?
>>
>>  I'm not 100% sure but I think that is a really good question to ask. Do
>> you know of any good tools that might help inform an answer? I know of
>> iptraf for example, is there one that folks generally prefer the most?
>>
>> > Are you only running 4 workers or did you truncate the output?
>>  Yes, I truncated the output to show four workers each (I have 16 total).
>>
>> > Are you doing 4 tuple load balancing or 2 tuple load balancing between
>> the two servers?
>>
>>  Sorry I am not sure what you mean by this or the implications of one
>> over the other. Is there an easy way I can find out (I am kinda new to
>> this)? I agree with the likelihood that B may be recieving more flows.
>>
>> Thanks!
>>  Jason
>>
>>
>>
>>
>>  On Fri, Jun 13, 2014 at 9:09 AM, Justin Azoff <JAzoff at albany.edu> wrote:
>>
>>> On Fri, Jun 13, 2014 at 08:01:54AM -0500, Jason Batchelor wrote:
>>> > At the moment Server A is getting about 700MB/s and Server B is
>>> getting about
>>> > 600Mb/s.
>>> >
>>> > What I don't understand, is Server A is having several orders of
>>> magnatude
>>> > better performance compared to Server B?
>>> >
>>> > TOP from A (included a few bro workers):
>>> >
>>> > top - 12:48:45 up 1 day, 17:03,  2 users,  load average: 5.30, 3.99,
>>> 3.13
>>> > Tasks: 706 total,  19 running, 687 sleeping,   0 stopped,   0 zombie
>>> > Cpu(s): 33.9%us,  6.6%sy,  1.1%ni, 57.2%id,  0.0%wa,  0.0%hi,  1.2%si,
>>>  0.0%st
>>> > Mem:  49376004k total, 33605828k used, 15770176k free,    93100k
>>> buffers
>>> > Swap:  2621432k total,     9760k used,  2611672k free,  9206880k cached
>>> >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>> >  5768 root      20   0 1808m 1.7g 519m R 100.0  3.6  32:24.92 bro
>>> >  5760 root      20   0 1688m 1.6g 519m R 99.7  3.4  34:08.36 bro
>>> >  3314 root      20   0 2160m 269m 4764 R 96.1  0.6  30:14.12 bro
>>> >  5754 root      20   0 1451m 1.4g 519m R 82.8  2.9  36:40.02 bro
>>>
>>>  Server A Bro cpu utilization = 378.6
>>>
>>> > TOP from B (included a few bro workers)
>>> >
>>> > top - 12:49:33 up 14:24,  2 users,  load average: 10.28, 9.31, 8.06
>>> > Tasks: 708 total,  25 running, 683 sleeping,   0 stopped,   0 zombie
>>> > Cpu(s): 41.6%us,  6.0%sy,  1.0%ni, 50.4%id,  0.0%wa,  0.0%hi,  1.1%si,
>>>  0.0%st
>>> > Mem:  49376004k total, 31837340k used, 17538664k free,   147212k
>>> buffers
>>> > Swap:  2621432k total,        0k used,  2621432k free, 13494332k cached
>>> >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>> >  3178 root      20   0 1073m 1.0g 264m R 100.0  2.1 401:47.31 bro
>>> >  3188 root      20   0  881m 832m 264m R 100.0  1.7 377:48.90 bro
>>> >  3189 root      20   0 1247m 1.2g 264m R 100.0  2.5 403:22.95 bro
>>> >  3193 root      20   0  920m 871m 264m R 100.0  1.8 429:45.98 bro
>>>
>>> > Both have the same amount of Bro workers. I just do not understand why
>>> Server
>>> > A is literally half the utilization on top of seeing more traffic? The
>>> only
>>> > real and consistent difference between the two I see is that server A
>>> seems to
>>> > have twice the amount of SHR (shared memory) compared to server B.
>>>
>>>  Server B Bro cpu utilization = 400%
>>>
>>> Are you only running 4 workers or did you truncate the output?  Is that
>>> running at 100% 24/7 or does it vary with the traffic?
>>>
>>> Are you doing 4 tuple load balancing or 2 tuple load balancing between
>>> the two servers?  Most likely Server B is seeing more flows.
>>>
>>>
>>> --
>>> -- Justin Azoff
>>>
>>
>>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140618/9ec0afe1/attachment.html 

From doug.burks at gmail.com  Thu Jun 19 03:47:56 2014
From: doug.burks at gmail.com (Doug Burks)
Date: Thu, 19 Jun 2014 06:47:56 -0400
Subject: [Bro] Intel Framework Not Generating Intel Log
In-Reply-To: <CAHbdUtV_CTtj3iZ6N4B+r4VrR-C7AnTQVskz1djvyWPSND9bZQ@mail.gmail.com>
References: <CAHbdUtV_CTtj3iZ6N4B+r4VrR-C7AnTQVskz1djvyWPSND9bZQ@mail.gmail.com>
Message-ID: <CAK8kjrBt0m15DGgonC8Rv50GEfNfrZFzZzQp2MTQh7ugizNu6Q@mail.gmail.com>

Hi Jamie,

Take a look at loaded_scripts.log and verify that the intel framework
and your intel feed show up there.

On Wed, Jun 18, 2014 at 3:21 PM, Jamie Gausemel
<jamie.gausemel at gmail.com> wrote:
> I am running Bro 2.2 on Security Onion
>
> ==========================
> sensor1 at sensor1:~/tmp$ sudo tail /opt/bro/share/bro/site/local.bro
> #@load apt1
>
> # Bro Intelligence Framework
> @load frameworks/intel/seen
> @load frameworks/intel/do_notice
>
> # Load Intel Feeds For Bro Intelligence Framework
> redef Intel::read_files += {
>         "/opt/bro/intel_feeds/test.txt"
> };
>
> ==========================
>
> sensor1 at sensor1:~/tmp$ sudo cat /opt/bro/intel_feeds/test.txt
> #fields indicator       indicator_type  meta.source
> 216.146.46.11   Intel::ADDR     jamie
>
> ==========================
>
> sensor1 at sensor1:~/tmp$ sudo bro -r test.pcap local
>
> generates the following logs:
>
> capture_loss.log  conn.log  loaded_scripts.log  packet_filter.log
> reporter.log
>
> ==========================
>
> reporter.log contains:
>
> sensor1 at sensor1:~/tmp$ cat reporter.log
> #separator \x09
> #set_separator  ,
> #empty_field    (empty)
> #unset_field    -
> #path   reporter
> #open   2014-06-18-18-13-17
> #fields ts      level   message location
> #types  time    enum    string  string
> 0.000000        Reporter::WARNING       Template value remaining in BPFConf
> filename: /etc/nsm/{{hostname}}-{{interface}}/bpf-bro.conf
> /opt/bro/share/bro/securityonion/./bpfconf.bro, line 99
> 0.000000        Reporter::INFO  BPFConf filename set:
> /etc/nsm/sensor1-eth1/bpf-bro.conf
> /opt/bro/share/bro/securityonion/./bpfconf.bro, line 103
> 0.000000        Reporter::INFO  BPFConf filename set:
> /etc/nsm/sensor1-eth1/bpf-bro.conf
> /opt/bro/share/bro/securityonion/./bpfconf.bro, line 103
> 1403105006.674182       Reporter::INFO  Failed to open GeoIP database:
> /usr/share/GeoIP/GeoIPCity.dat   (empty)
> 1403105006.674182       Reporter::INFO  Fell back to GeoIP Country database
> (empty)
> 1403105006.674182       Reporter::INFO  Failed to open GeoIP database:
> /usr/share/GeoIP/GeoIPCityv6.dat (empty)
> #close  2014-06-18-18-13-17
>
> ==========================
>
> test.pcap contains ICMP traffic to 216.146.46.11, and this traffic shows up
> in conn.log; however, I am not getting the expected intel.log. The test.txt
> is tab delimited, and was created with pico.
>
> Any ideas? Are there other logs I can look at for clues? Thanks.
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
Doug Burks


From jamie.gausemel at gmail.com  Thu Jun 19 07:17:19 2014
From: jamie.gausemel at gmail.com (Jamie Gausemel)
Date: Thu, 19 Jun 2014 10:17:19 -0400
Subject: [Bro] Intel Framework Not Generating Intel Log
In-Reply-To: <CAK8kjrBt0m15DGgonC8Rv50GEfNfrZFzZzQp2MTQh7ugizNu6Q@mail.gmail.com>
References: <CAHbdUtV_CTtj3iZ6N4B+r4VrR-C7AnTQVskz1djvyWPSND9bZQ@mail.gmail.com>
	<CAK8kjrBt0m15DGgonC8Rv50GEfNfrZFzZzQp2MTQh7ugizNu6Q@mail.gmail.com>
Message-ID: <CAHbdUtW3AJ-bz=cY-UyeixjX7QMqU-HidxrLfH119WSKP+1=Cg@mail.gmail.com>

The Bro intel scripts show up in the log, but am I supposed to be seeing my
test.txt feed?

==================

sensor1 at sensor1:~$ cat /home/sensor1/tmp/loaded_scripts.log |grep intel
  /opt/bro/share/bro/base/frameworks/intel/__load__.bro
    /opt/bro/share/bro/base/frameworks/intel/main.bro
    /opt/bro/share/bro/base/frameworks/intel/input.bro
  /opt/bro/share/bro/policy/frameworks/intel/seen/__load__.bro
    /opt/bro/share/bro/policy/frameworks/intel/seen/conn-established.bro
      /opt/bro/share/bro/policy/frameworks/intel/seen/where-locations.bro
    /opt/bro/share/bro/policy/frameworks/intel/seen/dns.bro
    /opt/bro/share/bro/policy/frameworks/intel/seen/file-hashes.bro
    /opt/bro/share/bro/policy/frameworks/intel/seen/file-names.bro
    /opt/bro/share/bro/policy/frameworks/intel/seen/http-headers.bro
    /opt/bro/share/bro/policy/frameworks/intel/seen/http-url.bro
    /opt/bro/share/bro/policy/frameworks/intel/seen/ssl.bro
    /opt/bro/share/bro/policy/frameworks/intel/seen/smtp.bro
    /opt/bro/share/bro/policy/frameworks/intel/seen/smtp-url-extraction.bro
  /opt/bro/share/bro/policy/frameworks/intel/do_notice.bro


On Thu, Jun 19, 2014 at 6:47 AM, Doug Burks <doug.burks at gmail.com> wrote:

> Hi Jamie,
>
> Take a look at loaded_scripts.log and verify that the intel framework
> and your intel feed show up there.
>
> On Wed, Jun 18, 2014 at 3:21 PM, Jamie Gausemel
> <jamie.gausemel at gmail.com> wrote:
> > I am running Bro 2.2 on Security Onion
> >
> > ==========================
> > sensor1 at sensor1:~/tmp$ sudo tail /opt/bro/share/bro/site/local.bro
> > #@load apt1
> >
> > # Bro Intelligence Framework
> > @load frameworks/intel/seen
> > @load frameworks/intel/do_notice
> >
> > # Load Intel Feeds For Bro Intelligence Framework
> > redef Intel::read_files += {
> >         "/opt/bro/intel_feeds/test.txt"
> > };
> >
> > ==========================
> >
> > sensor1 at sensor1:~/tmp$ sudo cat /opt/bro/intel_feeds/test.txt
> > #fields indicator       indicator_type  meta.source
> > 216.146.46.11   Intel::ADDR     jamie
> >
> > ==========================
> >
> > sensor1 at sensor1:~/tmp$ sudo bro -r test.pcap local
> >
> > generates the following logs:
> >
> > capture_loss.log  conn.log  loaded_scripts.log  packet_filter.log
> > reporter.log
> >
> > ==========================
> >
> > reporter.log contains:
> >
> > sensor1 at sensor1:~/tmp$ cat reporter.log
> > #separator \x09
> > #set_separator  ,
> > #empty_field    (empty)
> > #unset_field    -
> > #path   reporter
> > #open   2014-06-18-18-13-17
> > #fields ts      level   message location
> > #types  time    enum    string  string
> > 0.000000        Reporter::WARNING       Template value remaining in
> BPFConf
> > filename: /etc/nsm/{{hostname}}-{{interface}}/bpf-bro.conf
> > /opt/bro/share/bro/securityonion/./bpfconf.bro, line 99
> > 0.000000        Reporter::INFO  BPFConf filename set:
> > /etc/nsm/sensor1-eth1/bpf-bro.conf
> > /opt/bro/share/bro/securityonion/./bpfconf.bro, line 103
> > 0.000000        Reporter::INFO  BPFConf filename set:
> > /etc/nsm/sensor1-eth1/bpf-bro.conf
> > /opt/bro/share/bro/securityonion/./bpfconf.bro, line 103
> > 1403105006.674182       Reporter::INFO  Failed to open GeoIP database:
> > /usr/share/GeoIP/GeoIPCity.dat   (empty)
> > 1403105006.674182       Reporter::INFO  Fell back to GeoIP Country
> database
> > (empty)
> > 1403105006.674182       Reporter::INFO  Failed to open GeoIP database:
> > /usr/share/GeoIP/GeoIPCityv6.dat (empty)
> > #close  2014-06-18-18-13-17
> >
> > ==========================
> >
> > test.pcap contains ICMP traffic to 216.146.46.11, and this traffic shows
> up
> > in conn.log; however, I am not getting the expected intel.log. The
> test.txt
> > is tab delimited, and was created with pico.
> >
> > Any ideas? Are there other logs I can look at for clues? Thanks.
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> --
> Doug Burks
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140619/2f26f85b/attachment.html 

From ytl at slac.stanford.edu  Thu Jun 19 14:14:48 2014
From: ytl at slac.stanford.edu (=?utf-8?Q?Li=2C_Yee-Ting?=)
Date: Thu, 19 Jun 2014 14:14:48 -0700
Subject: [Bro] bro cluster with pf ring dna+libzero
In-Reply-To: <53A22E66.8040507@gmail.com>
References: <A0EB5C5EAF804AA29B073554D3107C54@slac.stanford.edu>
	<53A22E66.8040507@gmail.com>
Message-ID: <13E5DDEF73754B079D97C8C4A7961353@slac.stanford.edu>

Hi Gary,

ah, that makes sense! -n 10,1 works great :) thanks very much.

is anyone using ZC pf_ring for bro?

cheers,

Yee.  


On Wednesday, 18 June 2014 at 17:27, Gary Faulkner wrote:

> Hello,
>  
> Capstats is a separate application as far as pfdnacluster_master is  
> concerned. You can tell pfdnacluster_master that you want to send the  
> same traffic to another application using the -n flag by using a "," and  
> then specifying how many instances of the second app you intend to run.  
> When you call pfdnacluster_master try "-n 10,1" instead of "-n 10". You  
> actually want to run two applications against the same traffic, but the  
> second app, capstats, will only run one process that needs to consume  
> all of the traffic instead of having slices of traffic load balanced  
> between multiple processes.
>  
> Regards,
> Gary
>  
>  
>  
> On 6/18/2014 6:02 PM, Li, Yee-Ting wrote:
> > we're deploying a new bro cluster and am a huge newbie on all of this; so please excuse my ignorance. i have yet to actually start capturing on the cluster (awaiting delivery of a front-end device)
> >  
> > on each worker i have the dna+libzero ixgbe driver installed and insmodded. so i run:
> >  
> > $ sudo insmod pf_ring.ko enable_tx_capture=0 min_num_slots=32768
> > $ sudo insmod ixgbe.ko RSS=1,1,1,1 num_rx_slots=32768 mtu=9000
> >  
> >  
> >  
> > $ sudo /usr/sbin/setcap cap_net_raw,cap_net_admin=eip /usr/bin/pfdnacluster_master
> > $ /usr/bin/pfdnacluster_master -d -P /var/run/pfdnacluster-dna0.pid -D bromaint -c 0 -i dna0 -n 10
> >  
> > i do the setcap as i am running bro as non-root user. looks good?
> >  
> > $ cat /proc/net/pf_ring/13979-dna0.1
> > Bound Device(s) :
> > Active : 1
> > Breed : DNA
> > Sampling Rate : 1
> > Capture Direction : RX+TX
> > Socket Mode : RX only
> > Appl. Name : dna-cluster-0-socket-0
> > IP Defragment : No
> > BPF Filtering : Disabled
> > # Sw Filt. Rules : 0
> > # Hw Filt. Rules : 0
> > Poll Pkt Watermark : 128
> > Num Poll Calls : 0
> > Channel Id : 0
> > Num RX Slots : 32768
> > Num TX Slots : 8192
> > Tot Memory : 672399360 bytes
> > Cluster: Tot Recvd : 11
> > Cluster: Tot Sent : 0
> >  
> >  
> >  
> > then on my manager i have the following nodes.cfg:
> >  
> > [manager]
> > type=manager
> > host=sec-broman
> >  
> > [proxy-0]
> > type=proxy
> > host=sec-broman
> >  
> > [proxy-1]
> > type=proxy
> > host=sec-broman
> >  
> > [sec-bro01-0]
> > type=worker
> > host=sec-bro01
> > interface=dnacluster:0
> > lb_method=pf_ring
> > lb_procs=10
> >  
> >  
> >  
> > using bro 2.3; so i believe the lb_pf_ring.py script understands the dnacluster interface spec.
> >  
> > so i do an 'broctl install' (as user bromaint) from the manager, then log onto my worker and run
> >  
> > $ sudo /usr/sbin/setcap cap_net_raw,cap_net_admin=eip /opt/bro/bin/capstats
> > $ sudo /usr/sbin/setcap cap_net_raw,cap_net_admin=eip /opt/bro/bin/bro
> >  
> >  
> >  
> > then a 'broctl start' on the manager. everything looks fine so far? then i run 'broctl capstats' and i get:
> >  
> > Interface kpps mbps (10s average)
> > ----------------------------------------
> > sec-bro01-0-9: capstats failed (error: dnacluster:0: No such device exists (SIOCGIFHWADDR: No such device))
> >  
> >  
> >  
> > looking at proc for the pid of that bro instance, i get:
> >  
> > $ ps aux | grep sec-bro01-0-9
> > bromaint 14696 0.0 0.0 108128 1496 ? S 15:28 0:00 bash /opt/bro/share/broctl/scripts/run-bro -1 -i dnacluster:0 at 9 -U .status -p broctl -p broctl-live -p local -p sec-bro01-0-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
> >  
> > bromaint 14778 25.1 0.0 157736 56320 ? S 15:28 0:02 /opt/bro/bin/bro -i dnacluster:0 at 9 -U .status -p broctl -p broctl-live -p local -p sec-bro01-0-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
> > bromaint 14846 14.3 0.0 161832 52996 ? SN 15:28 0:01 /opt/bro/bin/bro -i dnacluster:0 at 9 -U .status -p broctl -p broctl-live -p local -p sec-bro01-0-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
> >  
> > $ cat /proc/net/pf_ring/14778-none.7
> > Bound Device(s) :
> > Active : 1
> > Breed : Non-DNA
> > Sampling Rate : 1
> > Capture Direction : RX+TX
> > Socket Mode : RX+TX
> > Appl. Name : bro-dnacluster:0 at 9
> > IP Defragment : No
> > BPF Filtering : Enabled
> > # Sw Filt. Rules : 0
> > # Hw Filt. Rules : 0
> > Poll Pkt Watermark : 1
> > Num Poll Calls : 2562490
> >  
> >  
> >  
> > what gives???
> >  
> > if i manually kill the bro process on the worker and rerun capstats, i get:
> >  
> > Interface kpps mbps (10s average)
> > ----------------------------------------
> > sec-bro01/dnacluster:0 0.0 0.0
> > Total 0.0 0.0
> >  
> >  
> > also, if i were to change the lb_procs to less than that of the pfdnacluster number of workers (-n), everything (seems to) work fine (bear in mind i'm not capturing any traffic at the moment). but would i loose any data? i'm using pf_ring 6.0.1.
> >  
> >  
> > thanks,
> >  
> > Yee.
> >  
> >  
> >  
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org (mailto:bro at bro-ids.org)
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>  
>  
>  
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org (mailto:bro at bro-ids.org)
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro






From bkellogg at dresser-rand.com  Sat Jun 21 07:00:47 2014
From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN))
Date: Sat, 21 Jun 2014 14:00:47 +0000
Subject: [Bro] Bro hanging on some sensors
In-Reply-To: <CAK8kjrCt4nMqaomyT+AqyHrQmxxA8kBfqiTEt5gUnsCobdZ7WA@mail.gmail.com>
References: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82181922@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDry05X3RBCt7Ph9SgQa13__+rjg28=VZ9KmPP3KgyYqw@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82182ADD@DRTEXW238VM.na.d-rco.com>
	<D422E894-2B8F-4E87-B223-CFFC85C170A2@geekempire.com>
	<BDF3793B557F624699D4C68A3B8FCF82182FFE@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrAzcnZ8R+cp0D2hK=iy3x-AvNJP88xwLGAG2GTqUYWSnQ@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82195DC5@DRTEXW238VM.na.d-rco.com>,
	<CAK8kjrCt4nMqaomyT+AqyHrQmxxA8kBfqiTEt5gUnsCobdZ7WA@mail.gmail.com>
Message-ID: <BDF3793B557F624699D4C68A3B8FCF8219A25A@DRTEXW238VM.na.d-rco.com>

The other consistent thing I see is that with the Intel framework disabled I'll have to stop and start Bro usually two times before I start seeing all of the logs generated but usually only once. When I have the Intel framework enable I can stop and start Bro a number of times with only those five log files being generated each time.  And again, on some of the sensors Bro will work with the Intel framework enabled and they all are using the same Intel files replicated via the "policy" directory Security Onion replication.


________________________________________
From: Doug Burks [doug.burks at gmail.com]
Sent: Wednesday, June 18, 2014 6:55 AM
To: Kellogg, Brian D (OLN)
Cc: Mike Reeves; bro
Subject: Re: [Bro] Bro hanging on some sensors

On Tue, Jun 17, 2014 at 10:30 AM, Kellogg, Brian D (OLN)
<bkellogg at dresser-rand.com> wrote:
> The one consistent thing I see is that when I stop, install, and then start
> Bro, Bro starts ok and all the appropriate logs are created.  If I stop and
> restart Bro again then the only logs I see in the "current" directory are:
> communication, loaded_scripts, reporter, stderr, and stdout.

Yep, I've seen this issue before.  I'm not sure if it's an issue with
the Security-Onion-specific scripts that we load into Bro, or if it
could be an issue with Bro itself.

Has anybody else seen this issue on a vanilla Bro installation (not
using Security Onion)?

--
Doug Burks



From doug.burks at gmail.com  Sat Jun 21 13:10:55 2014
From: doug.burks at gmail.com (Doug Burks)
Date: Sat, 21 Jun 2014 16:10:55 -0400
Subject: [Bro] Intel Framework Not Generating Intel Log
In-Reply-To: <CAHbdUtW3AJ-bz=cY-UyeixjX7QMqU-HidxrLfH119WSKP+1=Cg@mail.gmail.com>
References: <CAHbdUtV_CTtj3iZ6N4B+r4VrR-C7AnTQVskz1djvyWPSND9bZQ@mail.gmail.com>
	<CAK8kjrBt0m15DGgonC8Rv50GEfNfrZFzZzQp2MTQh7ugizNu6Q@mail.gmail.com>
	<CAHbdUtW3AJ-bz=cY-UyeixjX7QMqU-HidxrLfH119WSKP+1=Cg@mail.gmail.com>
Message-ID: <CAK8kjrCH2-kpZZbhNSSydEpzPOEG6+pcjo5_4PAfHh94kEr-Yg@mail.gmail.com>

No, it's not supposed to show test.txt in loaded_scripts.log.

I duplicated your scenario and everything worked properly for me.

Is it possible you have a local.bro in your current directory (~/tmp)
that is overriding /opt/bro/share/bro/site/local.bro?

Have you tried something like this?
bro -r test.pcap /opt/bro/share/bro/site/local.bro

On Thu, Jun 19, 2014 at 10:17 AM, Jamie Gausemel
<jamie.gausemel at gmail.com> wrote:
> The Bro intel scripts show up in the log, but am I supposed to be seeing my
> test.txt feed?
>
> ==================
>
> sensor1 at sensor1:~$ cat /home/sensor1/tmp/loaded_scripts.log |grep intel
>   /opt/bro/share/bro/base/frameworks/intel/__load__.bro
>     /opt/bro/share/bro/base/frameworks/intel/main.bro
>     /opt/bro/share/bro/base/frameworks/intel/input.bro
>   /opt/bro/share/bro/policy/frameworks/intel/seen/__load__.bro
>     /opt/bro/share/bro/policy/frameworks/intel/seen/conn-established.bro
>       /opt/bro/share/bro/policy/frameworks/intel/seen/where-locations.bro
>     /opt/bro/share/bro/policy/frameworks/intel/seen/dns.bro
>     /opt/bro/share/bro/policy/frameworks/intel/seen/file-hashes.bro
>     /opt/bro/share/bro/policy/frameworks/intel/seen/file-names.bro
>     /opt/bro/share/bro/policy/frameworks/intel/seen/http-headers.bro
>     /opt/bro/share/bro/policy/frameworks/intel/seen/http-url.bro
>     /opt/bro/share/bro/policy/frameworks/intel/seen/ssl.bro
>     /opt/bro/share/bro/policy/frameworks/intel/seen/smtp.bro
>     /opt/bro/share/bro/policy/frameworks/intel/seen/smtp-url-extraction.bro
>   /opt/bro/share/bro/policy/frameworks/intel/do_notice.bro
>
>
> On Thu, Jun 19, 2014 at 6:47 AM, Doug Burks <doug.burks at gmail.com> wrote:
>>
>> Hi Jamie,
>>
>> Take a look at loaded_scripts.log and verify that the intel framework
>> and your intel feed show up there.
>>
>> On Wed, Jun 18, 2014 at 3:21 PM, Jamie Gausemel
>> <jamie.gausemel at gmail.com> wrote:
>> > I am running Bro 2.2 on Security Onion
>> >
>> > ==========================
>> > sensor1 at sensor1:~/tmp$ sudo tail /opt/bro/share/bro/site/local.bro
>> > #@load apt1
>> >
>> > # Bro Intelligence Framework
>> > @load frameworks/intel/seen
>> > @load frameworks/intel/do_notice
>> >
>> > # Load Intel Feeds For Bro Intelligence Framework
>> > redef Intel::read_files += {
>> >         "/opt/bro/intel_feeds/test.txt"
>> > };
>> >
>> > ==========================
>> >
>> > sensor1 at sensor1:~/tmp$ sudo cat /opt/bro/intel_feeds/test.txt
>> > #fields indicator       indicator_type  meta.source
>> > 216.146.46.11   Intel::ADDR     jamie
>> >
>> > ==========================
>> >
>> > sensor1 at sensor1:~/tmp$ sudo bro -r test.pcap local
>> >
>> > generates the following logs:
>> >
>> > capture_loss.log  conn.log  loaded_scripts.log  packet_filter.log
>> > reporter.log
>> >
>> > ==========================
>> >
>> > reporter.log contains:
>> >
>> > sensor1 at sensor1:~/tmp$ cat reporter.log
>> > #separator \x09
>> > #set_separator  ,
>> > #empty_field    (empty)
>> > #unset_field    -
>> > #path   reporter
>> > #open   2014-06-18-18-13-17
>> > #fields ts      level   message location
>> > #types  time    enum    string  string
>> > 0.000000        Reporter::WARNING       Template value remaining in
>> > BPFConf
>> > filename: /etc/nsm/{{hostname}}-{{interface}}/bpf-bro.conf
>> > /opt/bro/share/bro/securityonion/./bpfconf.bro, line 99
>> > 0.000000        Reporter::INFO  BPFConf filename set:
>> > /etc/nsm/sensor1-eth1/bpf-bro.conf
>> > /opt/bro/share/bro/securityonion/./bpfconf.bro, line 103
>> > 0.000000        Reporter::INFO  BPFConf filename set:
>> > /etc/nsm/sensor1-eth1/bpf-bro.conf
>> > /opt/bro/share/bro/securityonion/./bpfconf.bro, line 103
>> > 1403105006.674182       Reporter::INFO  Failed to open GeoIP database:
>> > /usr/share/GeoIP/GeoIPCity.dat   (empty)
>> > 1403105006.674182       Reporter::INFO  Fell back to GeoIP Country
>> > database
>> > (empty)
>> > 1403105006.674182       Reporter::INFO  Failed to open GeoIP database:
>> > /usr/share/GeoIP/GeoIPCityv6.dat (empty)
>> > #close  2014-06-18-18-13-17
>> >
>> > ==========================
>> >
>> > test.pcap contains ICMP traffic to 216.146.46.11, and this traffic shows
>> > up
>> > in conn.log; however, I am not getting the expected intel.log. The
>> > test.txt
>> > is tab delimited, and was created with pico.
>> >
>> > Any ideas? Are there other logs I can look at for clues? Thanks.
>> >
>> > _______________________________________________
>> > Bro mailing list
>> > bro at bro-ids.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>>
>> --
>> Doug Burks
>
>



-- 
Doug Burks


From doug.burks at gmail.com  Sat Jun 21 13:17:39 2014
From: doug.burks at gmail.com (Doug Burks)
Date: Sat, 21 Jun 2014 16:17:39 -0400
Subject: [Bro] Bro hanging on some sensors
In-Reply-To: <BDF3793B557F624699D4C68A3B8FCF8219A25A@DRTEXW238VM.na.d-rco.com>
References: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82181922@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDry05X3RBCt7Ph9SgQa13__+rjg28=VZ9KmPP3KgyYqw@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82182ADD@DRTEXW238VM.na.d-rco.com>
	<D422E894-2B8F-4E87-B223-CFFC85C170A2@geekempire.com>
	<BDF3793B557F624699D4C68A3B8FCF82182FFE@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrAzcnZ8R+cp0D2hK=iy3x-AvNJP88xwLGAG2GTqUYWSnQ@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82195DC5@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrCt4nMqaomyT+AqyHrQmxxA8kBfqiTEt5gUnsCobdZ7WA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF8219A25A@DRTEXW238VM.na.d-rco.com>
Message-ID: <CAK8kjrBZ5ky2TiM8tV5mc29Y7xLNPrXC+v8qoMpXFmEE=UJo5Q@mail.gmail.com>

As a temporary test (perhaps on a non-production machine), could you
comment out the Security-Onion-specific scripts in
/opt/bro/share/bro/site/local.bro and see if that makes any
difference?

#@load securityonion
#@load file-extraction
#@load apt1

I know there were some issues previously with the hostname/interface
scripts in /opt/bro/share/bro/securityonion/ that resulted in a timing
issue.  Some of the issues were fixed, but perhaps some other issues
remain.

On Sat, Jun 21, 2014 at 10:00 AM, Kellogg, Brian D (OLN)
<bkellogg at dresser-rand.com> wrote:
> The other consistent thing I see is that with the Intel framework disabled I'll have to stop and start Bro usually two times before I start seeing all of the logs generated but usually only once. When I have the Intel framework enable I can stop and start Bro a number of times with only those five log files being generated each time.  And again, on some of the sensors Bro will work with the Intel framework enabled and they all are using the same Intel files replicated via the "policy" directory Security Onion replication.
>
>
> ________________________________________
> From: Doug Burks [doug.burks at gmail.com]
> Sent: Wednesday, June 18, 2014 6:55 AM
> To: Kellogg, Brian D (OLN)
> Cc: Mike Reeves; bro
> Subject: Re: [Bro] Bro hanging on some sensors
>
> On Tue, Jun 17, 2014 at 10:30 AM, Kellogg, Brian D (OLN)
> <bkellogg at dresser-rand.com> wrote:
>> The one consistent thing I see is that when I stop, install, and then start
>> Bro, Bro starts ok and all the appropriate logs are created.  If I stop and
>> restart Bro again then the only logs I see in the "current" directory are:
>> communication, loaded_scripts, reporter, stderr, and stdout.
>
> Yep, I've seen this issue before.  I'm not sure if it's an issue with
> the Security-Onion-specific scripts that we load into Bro, or if it
> could be an issue with Bro itself.
>
> Has anybody else seen this issue on a vanilla Bro installation (not
> using Security Onion)?
>
> --
> Doug Burks



-- 
Doug Burks



From bkellogg at dresser-rand.com  Sat Jun 21 15:50:29 2014
From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN))
Date: Sat, 21 Jun 2014 22:50:29 +0000
Subject: [Bro] Bro hanging on some sensors
In-Reply-To: <CAK8kjrBZ5ky2TiM8tV5mc29Y7xLNPrXC+v8qoMpXFmEE=UJo5Q@mail.gmail.com>
References: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82181922@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDry05X3RBCt7Ph9SgQa13__+rjg28=VZ9KmPP3KgyYqw@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82182ADD@DRTEXW238VM.na.d-rco.com>
	<D422E894-2B8F-4E87-B223-CFFC85C170A2@geekempire.com>
	<BDF3793B557F624699D4C68A3B8FCF82182FFE@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrAzcnZ8R+cp0D2hK=iy3x-AvNJP88xwLGAG2GTqUYWSnQ@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82195DC5@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrCt4nMqaomyT+AqyHrQmxxA8kBfqiTEt5gUnsCobdZ7WA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF8219A25A@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrBZ5ky2TiM8tV5mc29Y7xLNPrXC+v8qoMpXFmEE=UJo5Q@mail.gmail.com>
Message-ID: <BDF3793B557F624699D4C68A3B8FCF8219A5AB@DRTEXW238VM.na.d-rco.com>

A non-text attachment was scrubbed...
Name: smime.p7m
Type: application/pkcs7-mime
Size: 9586 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140621/99674295/attachment.p7c 

From bkellogg at dresser-rand.com  Sat Jun 21 15:54:17 2014
From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN))
Date: Sat, 21 Jun 2014 22:54:17 +0000
Subject: [Bro] Bro hanging on some sensors
In-Reply-To: <CAK8kjrBZ5ky2TiM8tV5mc29Y7xLNPrXC+v8qoMpXFmEE=UJo5Q@mail.gmail.com>
References: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82181922@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDry05X3RBCt7Ph9SgQa13__+rjg28=VZ9KmPP3KgyYqw@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82182ADD@DRTEXW238VM.na.d-rco.com>
	<D422E894-2B8F-4E87-B223-CFFC85C170A2@geekempire.com>
	<BDF3793B557F624699D4C68A3B8FCF82182FFE@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrAzcnZ8R+cp0D2hK=iy3x-AvNJP88xwLGAG2GTqUYWSnQ@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82195DC5@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrCt4nMqaomyT+AqyHrQmxxA8kBfqiTEt5gUnsCobdZ7WA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF8219A25A@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrBZ5ky2TiM8tV5mc29Y7xLNPrXC+v8qoMpXFmEE=UJo5Q@mail.gmail.com>
Message-ID: <BDF3793B557F624699D4C68A3B8FCF8219A5E0@DRTEXW238VM.na.d-rco.com>

Commenting out apt1 seems to make it more reliable.  I can at least load one
intel file now; sometimes two.

The odd thing is that the more I play with this the more unreliable it gets.
I thought I saw a pattern at one point as I commented out securityonion and
that allowed me to load two intel files.  I then uncommented securityonion and
it still worked with two intel files.  The it once worked with four intel
files. I then tried to add a fifth and that broke it.  So I went back to four
and Bro would still not generate more than the five log files.  Then I was
back to two intel files to get Bro working after a couple more restarts.  Then
I uncommented apt1 and it still worked with two intel files loaded.

So far, the most reliable way of getting an intel file loaded is by commenting out apt1 and only enabling one intel file.


-----Original Message-----
From: Doug Burks [mailto:doug.burks at gmail.com]
Sent: Saturday, June 21, 2014 4:18 PM
To: Kellogg, Brian D (OLN)
Cc: bro
Subject: Re: [Bro] Bro hanging on some sensors

As a temporary test (perhaps on a non-production machine), could you comment out the Security-Onion-specific scripts in /opt/bro/share/bro/site/local.bro and see if that makes any difference?

#@load securityonion
#@load file-extraction
#@load apt1

I know there were some issues previously with the hostname/interface scripts in /opt/bro/share/bro/securityonion/ that resulted in a timing issue.  Some of the issues were fixed, but perhaps some other issues remain.

On Sat, Jun 21, 2014 at 10:00 AM, Kellogg, Brian D (OLN) <bkellogg at dresser-rand.com> wrote:
> The other consistent thing I see is that with the Intel framework disabled I'll have to stop and start Bro usually two times before I start seeing all of the logs generated but usually only once. When I have the Intel framework enable I can stop and start Bro a number of times with only those five log files being generated each time.  And again, on some of the sensors Bro will work with the Intel framework enabled and they all are using the same Intel files replicated via the "policy" directory Security Onion replication.
>
>
> ________________________________________
> From: Doug Burks [doug.burks at gmail.com]
> Sent: Wednesday, June 18, 2014 6:55 AM
> To: Kellogg, Brian D (OLN)
> Cc: Mike Reeves; bro
> Subject: Re: [Bro] Bro hanging on some sensors
>
> On Tue, Jun 17, 2014 at 10:30 AM, Kellogg, Brian D (OLN)
> <bkellogg at dresser-rand.com> wrote:
>> The one consistent thing I see is that when I stop, install, and then
>> start Bro, Bro starts ok and all the appropriate logs are created.
>> If I stop and restart Bro again then the only logs I see in the "current" directory are:
>> communication, loaded_scripts, reporter, stderr, and stdout.
>
> Yep, I've seen this issue before.  I'm not sure if it's an issue with
> the Security-Onion-specific scripts that we load into Bro, or if it
> could be an issue with Bro itself.
>
> Has anybody else seen this issue on a vanilla Bro installation (not
> using Security Onion)?
>
> --
> Doug Burks



--
Doug Burks



From mattchess50 at gmail.com  Mon Jun 23 12:49:27 2014
From: mattchess50 at gmail.com (Matt Stucky)
Date: Mon, 23 Jun 2014 14:49:27 -0500
Subject: [Bro] packet broker
Message-ID: <CAMM5JYoSOcfmWWx+8gbquSq=sq816-DVZ2snn29s2au9VL7tVw@mail.gmail.com>

Anyone have experience with some of the packet brokering solutions out
there such as Cpacket, Gigamon, VSS, Arista (with DANZ), etc.?  How well do
they handle bursting, micro bursting when aggregating multiple taps?  How
much packet loss do you see due to buffer overflows on the packet brokering
device?

Thanks,
-matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140623/c2bd9866/attachment.html 

From reevesmk at gmail.com  Mon Jun 23 15:22:31 2014
From: reevesmk at gmail.com (Mike Reeves)
Date: Mon, 23 Jun 2014 18:22:31 -0400
Subject: [Bro] packet broker
In-Reply-To: <CAMM5JYoSOcfmWWx+8gbquSq=sq816-DVZ2snn29s2au9VL7tVw@mail.gmail.com>
References: <CAMM5JYoSOcfmWWx+8gbquSq=sq816-DVZ2snn29s2au9VL7tVw@mail.gmail.com>
Message-ID: <55FC5E78-4203-4052-9E5A-8B02AE39DD3E@gmail.com>

I have used both cPacket and Arista and both have handled great up to 10G with little to no packet loss. 

Sent from my iPad

> On Jun 23, 2014, at 3:49 PM, Matt Stucky <mattchess50 at gmail.com> wrote:
> 
> Anyone have experience with some of the packet brokering solutions out there such as Cpacket, Gigamon, VSS, Arista (with DANZ), etc.?  How well do they handle bursting, micro bursting when aggregating multiple taps?  How much packet loss do you see due to buffer overflows on the packet brokering device?
> 
> Thanks,
> -matt
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From seth at icir.org  Tue Jun 24 09:25:45 2014
From: seth at icir.org (Seth Hall)
Date: Tue, 24 Jun 2014 12:25:45 -0400
Subject: [Bro] bro cluster with pf ring dna+libzero
In-Reply-To: <13E5DDEF73754B079D97C8C4A7961353@slac.stanford.edu>
References: <A0EB5C5EAF804AA29B073554D3107C54@slac.stanford.edu>
	<53A22E66.8040507@gmail.com>
	<13E5DDEF73754B079D97C8C4A7961353@slac.stanford.edu>
Message-ID: <06A47498-0440-4DBE-9371-6E03C711A56E@icir.org>


On Jun 19, 2014, at 5:14 PM, Li, Yee-Ting <ytl at slac.stanford.edu> wrote:

> is anyone using ZC pf_ring for bro?

2.3 should support using the zc load balancing tool.  If you follow the directions here:
	http://bro.org/documentation/load-balancing.html

you should be able use the same config as for pf_ring+DNA but you'll be sniffing an interface named zc:21 (or similar, the "21" is the cluster number that is auto assigned by broctl).  One issue with the zbalance_ipc tool that pf_ring ships with though is that you can't load balance to multiple applications like you could with the older pfdnacluster_master tool.  I couldn't figure out how to do it at least.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140624/28282542/attachment.bin 

From seth at icir.org  Tue Jun 24 13:47:14 2014
From: seth at icir.org (Seth Hall)
Date: Tue, 24 Jun 2014 16:47:14 -0400
Subject: [Bro] packet broker
In-Reply-To: <CAMM5JYoSOcfmWWx+8gbquSq=sq816-DVZ2snn29s2au9VL7tVw@mail.gmail.com>
References: <CAMM5JYoSOcfmWWx+8gbquSq=sq816-DVZ2snn29s2au9VL7tVw@mail.gmail.com>
Message-ID: <D8C02E8D-5A03-4FA8-8565-7DE771BF3F23@icir.org>


On Jun 23, 2014, at 3:49 PM, Matt Stucky <mattchess50 at gmail.com> wrote:

> Anyone have experience with some of the packet brokering solutions out there such as Cpacket, Gigamon, VSS, Arista (with DANZ), etc.?  How well do they handle bursting, micro bursting when aggregating multiple taps?  

No one that I know of has good information about this stuff.  Unfortunately because you're asking about functionality at the very edge of a devices functionality I think you'd have a hard time getting good answers from all of these companies themselves too (although I would love if one of these companies is on the list and they speak up and tell us how they've tested the edge of capability of their devices!).

> How much packet loss do you see due to buffer overflows on the packet brokering device?


I would hope no more than the output interfaces could handle. :)  Unfortunately this fits into the same bucket regarding capability limits.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140624/a232e42d/attachment.bin 

From seth at icir.org  Tue Jun 24 13:51:31 2014
From: seth at icir.org (Seth Hall)
Date: Tue, 24 Jun 2014 16:51:31 -0400
Subject: [Bro] Bro hanging on some sensors
In-Reply-To: <BDF3793B557F624699D4C68A3B8FCF8219A5E0@DRTEXW238VM.na.d-rco.com>
References: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82181922@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDry05X3RBCt7Ph9SgQa13__+rjg28=VZ9KmPP3KgyYqw@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82182ADD@DRTEXW238VM.na.d-rco.com>
	<D422E894-2B8F-4E87-B223-CFFC85C170A2@geekempire.com>
	<BDF3793B557F624699D4C68A3B8FCF82182FFE@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrAzcnZ8R+cp0D2hK=iy3x-AvNJP88xwLGAG2GTqUYWSnQ@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82195DC5@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrCt4nMqaomyT+AqyHrQmxxA8kBfqiTEt5gUnsCobdZ7WA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF8219A25A@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrBZ5ky2TiM8tV5mc29Y7xLNPrXC+v8qoMpXFmEE=UJo5Q@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF8219A5E0@DRTEXW238VM.na.d-rco.com>
Message-ID: <FCE43557-1372-4DA0-AFA3-1B21DD569D64@icir.org>


On Jun 21, 2014, at 6:54 PM, Kellogg, Brian D (OLN) <bkellogg at dresser-rand.com> wrote:

> So far, the most reliable way of getting an intel file loaded is by commenting out apt1 and only enabling one intel file.

I suspect what's going on is some sort of resource overload issue right at Bro startup and that's causing some other "downstream" stuff to fail.  If I get a chance sometime I may write a new script for you to load intelligence with so that it never tries to load more than one intel file at a time. (or someone else could write that!)

Thanks for reporting the issue.  Hopefully we can figure it out together. ;)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140624/a734e5f9/attachment.bin 

From seth at icir.org  Tue Jun 24 14:01:03 2014
From: seth at icir.org (Seth Hall)
Date: Tue, 24 Jun 2014 17:01:03 -0400
Subject: [Bro] Script profiling?
In-Reply-To: <CAE1e=UL+SKMh1XM86DwMMtpiq=X3bUe64YG8-kxqyf+p-XEe8Q@mail.gmail.com>
References: <CAE1e=UL+SKMh1XM86DwMMtpiq=X3bUe64YG8-kxqyf+p-XEe8Q@mail.gmail.com>
Message-ID: <6D7DE7F3-9E51-4E0E-9D5E-2C02F838DDEC@icir.org>


On Jun 17, 2014, at 11:21 AM, Scott Gibbons <sgibbons at clicksecurity.com> wrote:

> We have added a significant number of scripts.

Could you clarify more of what you mean here?  I've seen lots of extremely inefficient scripts (and written quite a few myself!) and it's very possible you're doing something that isn't generally a good idea for operational use.

> Are there any good script profilers out there that would tell me that I'm spending a lot of time in some script?  I've considered using vTune or some other profiler, but that only gives me second-order profiling information.

Gilbert Clark is doing some profiling work but I'm not sure if that's going to address your needs.  I'll let him comment on his work if he's reading this thread.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140624/71dac044/attachment.bin 

From bkellogg at dresser-rand.com  Tue Jun 24 14:08:38 2014
From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN))
Date: Tue, 24 Jun 2014 21:08:38 +0000
Subject: [Bro] Bro hanging on some sensors
In-Reply-To: <FCE43557-1372-4DA0-AFA3-1B21DD569D64@icir.org>
References: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82181922@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDry05X3RBCt7Ph9SgQa13__+rjg28=VZ9KmPP3KgyYqw@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82182ADD@DRTEXW238VM.na.d-rco.com>
	<D422E894-2B8F-4E87-B223-CFFC85C170A2@geekempire.com>
	<BDF3793B557F624699D4C68A3B8FCF82182FFE@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrAzcnZ8R+cp0D2hK=iy3x-AvNJP88xwLGAG2GTqUYWSnQ@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82195DC5@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrCt4nMqaomyT+AqyHrQmxxA8kBfqiTEt5gUnsCobdZ7WA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF8219A25A@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrBZ5ky2TiM8tV5mc29Y7xLNPrXC+v8qoMpXFmEE=UJo5Q@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF8219A5E0@DRTEXW238VM.na.d-rco.com>,
	<FCE43557-1372-4DA0-AFA3-1B21DD569D64@icir.org>
Message-ID: <BDF3793B557F624699D4C68A3B8FCF8219E123@DRTEXW238VM.na.d-rco.com>

Can you point me in the direction to start looking at this myself?  I'll see what I can dig around and find, thanks.

________________________________________
From: Seth Hall [seth at icir.org]
Sent: Tuesday, June 24, 2014 4:51 PM
To: Kellogg, Brian D (OLN)
Cc: Doug Burks; bro
Subject: Re: [Bro] Bro hanging on some sensors

On Jun 21, 2014, at 6:54 PM, Kellogg, Brian D (OLN) <bkellogg at dresser-rand.com> wrote:

> So far, the most reliable way of getting an intel file loaded is by commenting out apt1 and only enabling one intel file.
?
I suspect what's going on is some sort of resource overload issue right at Bro startup and that's causing some other "downstream" stuff to fail.  If I get a chance sometime I may write a new script for you to load intelligence with so that it never tries to load more than one intel file at a time. (or someone else could write that!)

Thanks for reporting the issue.  Hopefully we can figure it out together. ;)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From seth at icir.org  Tue Jun 24 20:59:57 2014
From: seth at icir.org (Seth Hall)
Date: Tue, 24 Jun 2014 23:59:57 -0400
Subject: [Bro] Bro hanging on some sensors
In-Reply-To: <BDF3793B557F624699D4C68A3B8FCF8219E123@DRTEXW238VM.na.d-rco.com>
References: <BDF3793B557F624699D4C68A3B8FCF82180ED3@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDsfbLQk3HKwi9ccxwxaPTY36Je9FUuMNiJJVNOzFVTPA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82181922@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrDry05X3RBCt7Ph9SgQa13__+rjg28=VZ9KmPP3KgyYqw@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82182ADD@DRTEXW238VM.na.d-rco.com>
	<D422E894-2B8F-4E87-B223-CFFC85C170A2@geekempire.com>
	<BDF3793B557F624699D4C68A3B8FCF82182FFE@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrAzcnZ8R+cp0D2hK=iy3x-AvNJP88xwLGAG2GTqUYWSnQ@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF82195DC5@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrCt4nMqaomyT+AqyHrQmxxA8kBfqiTEt5gUnsCobdZ7WA@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF8219A25A@DRTEXW238VM.na.d-rco.com>
	<CAK8kjrBZ5ky2TiM8tV5mc29Y7xLNPrXC+v8qoMpXFmEE=UJo5Q@mail.gmail.com>
	<BDF3793B557F624699D4C68A3B8FCF8219A5E0@DRTEXW238VM.na.d-rco.com>,
	<FCE43557-1372-4DA0-AFA3-1B21DD569D64@icir.org>
	<BDF3793B557F624699D4C68A3B8FCF8219E123@DRTEXW238VM.na.d-rco.com>
Message-ID: <75AA2DCC-8342-4C30-87A7-6614AB8CE507@icir.org>


On Jun 24, 2014, at 5:08 PM, Kellogg, Brian D (OLN) <bkellogg at dresser-rand.com> wrote:

> Can you point me in the direction to start looking at this myself?  I'll see what I can dig around and find, thanks.

You can take a look at the base/frameworks/intel/input.bro script.  Right now that creates all of the input streams at the same time which causes them all to start reading as fast as they can.  It might make more sense (at the moment since we don't have a limiter mechanism on the input framework) to wait until one file is fully read before creating the next input stream.

That input.bro script is an incredibly basic script and it should be possible to create another script and keep it outside of the intel framework that uses the intel framework api to import intelligence data by a different mechanism.

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140624/b089e51c/attachment.bin 

From robin at icir.org  Wed Jun 25 08:40:27 2014
From: robin at icir.org (Robin Sommer)
Date: Wed, 25 Jun 2014 08:40:27 -0700
Subject: [Bro] What OS are you using with Bro?
Message-ID: <20140625154027.GE47118@icir.org>

We are currently reevaluating which platforms we test Bro on, which
binary packages we distribute, and what dependencies we can require
for running future Bro versions.

To help us get a better idea on what platforms people are running Bro
on today, we'd like to ask you to fill out a short survey to tell us
about the OSs you are using:

    https://www.surveymonkey.com/s/Bro_Usage

This shouldn't take longer than a couple minutes and is completely
anonymous (we don't record anything else than your answers. :)

Thanks!

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 *     robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 * www.icir.org/robin


From gc355804 at ohio.edu  Wed Jun 25 09:42:07 2014
From: gc355804 at ohio.edu (Gilbert Clark)
Date: Wed, 25 Jun 2014 12:42:07 -0400
Subject: [Bro] Script profiling?
In-Reply-To: <6D7DE7F3-9E51-4E0E-9D5E-2C02F838DDEC@icir.org>
References: <CAE1e=UL+SKMh1XM86DwMMtpiq=X3bUe64YG8-kxqyf+p-XEe8Q@mail.gmail.com>
	<6D7DE7F3-9E51-4E0E-9D5E-2C02F838DDEC@icir.org>
Message-ID: <53AAFBDF.6050704@ohio.edu>

Hi:

>> Are there any good script profilers out there that would tell me that I'm spending a lot of time in some script?  I've considered using vTune or some other profiler, but that only gives me second-order profiling information.
> Gilbert Clark is doing some profiling work but I'm not sure if that's going to address your needs.  I'll let him comment on his work if he's reading this thread.

Unfortunately, I don't think anything I have would be in a good place to 
release at the moment.  If this isn't a deterrent, feel free to get in 
touch off-list and we can chat: this isn't because the work is any big 
secret, but instead just because I'm a little hesitant to discuss 
anything tentative in a public setting.  In my experience, discussions 
of this nature have a funny way of turning into commitments, and I'd 
hate to be making commitments that I wouldn't be able to later follow 
through on :)

A few other things that might be more practical in the interim, though:

* There is a small script that ships with bro: 
aux/bro-aux/devel-tools/cpu-bench-with-trace.  This script will find all 
the scripts loaded in local.bro and init-default and run bro with each 
of these scripts loaded in sequence, producing time output for each 
one.  Assuming there's a trace available, this can be a simple way to 
help identify (relatively speaking) which scripts are introducing the 
most overhead.

* There are a few configuration settings that can be used to enable 
per-packet(ish) profiling: check out pkt_profile in init-bare.bro 
(http://www.bro.org/sphinx-git/scripts/base/init-bare.bro.html#id-pkt_profile_freq).

* Bro provides an option (-t) that will dump pretty much everything it's 
doing in script land into a log file.  This is *incredibly* verbose and 
*very* expensive but, with a little scripting, can yield some useful 
information about what bro is doing.  The overhead is high enough that I 
really doubt this could be used as an accurate timing mechanism, though.

* There is an option to enable segment profiling 
(http://www.bro.org/sphinx-git/scripts/base/init-bare.bro.html#id-segment_profiling) 
in prof.log.  Segment profiling will show time spent in particular areas 
of bro.  With that said, I have run into a few traces where bro crashes 
when the segment profiler is run.  This could be because I'm doing 
something wrong, though :)

* If systemtap / dtrace is available and you don't mind getting your 
hands dirty (read: feel comfortable building bro and possibly tweaking a 
few things here and there), we can work through finding a few sane probe 
points in the code that might help with this task in a practical way.

Hope something in there is useful.

-Gilbert



From Thomas.Bryant at parsons.com  Thu Jun 26 05:47:59 2014
From: Thomas.Bryant at parsons.com (Bryant, Thomas)
Date: Thu, 26 Jun 2014 12:47:59 +0000
Subject: [Bro] Bro 2.3 command line problem.
Message-ID: <466F52083C4AEB459F4271C87D3B890F2414CF3A@HSV-MB002.huntsville.ads.sparta.com>

When running Bro 2.1 from the command line with this command:

   bro -r myPackets.pcap

all is well.

Trying the same thing from Bro 2.3:

  bro -r myPackets.pcap

results in:

   internal error: Broxygen can't get mtime of bro binary : No such file or directory
   Aborted

If, however, one uses this command:

   /home/myAccount/bro2.3/bin/bro -r myPackets.pcap

all is again OK.

Note that the /home/myAccount/bro2.3/bin directory is in my path, which was changed from the similar Bro 2.1 directory when Bro 2.3 was installed.  We're running  Red Hat Enterprise Linux Server release 5.10 (Tikanga).

FWIW, I set up an account on the JIRA Bro issue tracker page, and could not find the "Create a new ticket" button, so I post this here, instead.




From omer007security at walla.co.il  Thu Jun 26 06:25:56 2014
From: omer007security at walla.co.il (=?UTF-8?Q?=D7=A2=D7=95=D7=9E=D7=A8=20=D7=A2=D7=95=D7=9E=D7=A8?=)
Date: Thu, 26 Jun 2014 16:25:56 +0300
Subject: [Bro] =?utf-8?q?A_way_to_check_if_record_field_is_already_set?=
Message-ID: <1403789156.142000-56440238-31751@walla.co.il>

An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140626/20bc1909/attachment.html 

From liburdi.joshua at gmail.com  Thu Jun 26 06:43:25 2014
From: liburdi.joshua at gmail.com (Josh Liburdi)
Date: Thu, 26 Jun 2014 06:43:25 -0700 (PDT)
Subject: [Bro] A way to check if record field is already set
In-Reply-To: <1403789156.142000-56440238-31751@walla.co.il>
References: <1403789156.142000-56440238-31751@walla.co.il>
Message-ID: <1403790204825.9440f05b@Nodemailer>

"?$" will check if a value exists for any field. For your example, put $http?$host in the script before you try to process the host field.

-Josh

On Thu, Jun 26, 2014 at 09:34 AM, ???? ????, wrote:

_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140626/24942a9a/attachment.html 

From jxbatchelor at gmail.com  Thu Jun 26 08:12:56 2014
From: jxbatchelor at gmail.com (Jason Batchelor)
Date: Thu, 26 Jun 2014 10:12:56 -0500
Subject: [Bro] Memory Consumption
Message-ID: <CANWz5X-zt+_EFD+J5pp-dm2acp7iJ2cgcPUJYdycrr=YwZ2DgA@mail.gmail.com>

Hello Everyone:

I have a question concerning Bro memory utilization. I have two servers,
both with the following specs...

Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz x 32
48GB RAM
Running: CentOs 6.5

w/ PF_RING running on 16 workers with a 65536 ring slots.

I have Suricata and Bro running fine on one server (server A). Then I just
have Bro running on another server by itself (server B).

Both servers see about the same amount of data at peak load (~1gb/s).

On server A, memory utilization operates at capacity. With bro and suri
taking up just about all there is.

On server B, memory utilization operates at capacity as well? With Bro
slowly but surely consuming just about all available memory until it leaves
me with about 200k free for the rest of the system. This same behavior is
seen on server A as well, however, server A is seemingly stable with two
apps running? At certain points about a gig of memory is freed up, but then
it gets consumed all over again to that 200k threshold.

What is going on here? Is it normal for Bro to take up as much memory as it
can? Are others experiancing this too? Is there something I can do to
restrict Bro from being such a memory hog?

I was considering getting more memory for the servers, however based on
these tests I cannot be certain Bro will not just run away with the new
memory I give it in the same fashion :)

FWIW, I have been experiancing this with Bro 2.2 and 2.3.

Many thanks in advance for the help!

Thanks,
Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140626/2e1b7c33/attachment.html 

From jsiwek at illinois.edu  Thu Jun 26 09:13:15 2014
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Thu, 26 Jun 2014 16:13:15 +0000
Subject: [Bro] Bro 2.3 command line problem.
In-Reply-To: <466F52083C4AEB459F4271C87D3B890F2414CF3A@HSV-MB002.huntsville.ads.sparta.com>
References: <466F52083C4AEB459F4271C87D3B890F2414CF3A@HSV-MB002.huntsville.ads.sparta.com>
Message-ID: <13EA3B7A-A987-48E0-8700-BB02777B97D1@illinois.edu>


On Jun 26, 2014, at 7:47 AM, Bryant, Thomas <Thomas.Bryant at parsons.com> wrote:

> Trying the same thing from Bro 2.3:
> 
>  bro -r myPackets.pcap
> 
> results in:
> 
>   internal error: Broxygen can't get mtime of bro binary : No such file or directory
>   Aborted
> 
> If, however, one uses this command:
> 
>   /home/myAccount/bro2.3/bin/bro -r myPackets.pcap
> 
> all is again OK.
> 
> Note that the /home/myAccount/bro2.3/bin directory is in my path, which was changed from the similar Bro 2.1 directory when Bro 2.3 was installed.

What does `which bro` output?  And what does `ls -l $(which bro)` output?

If the bro binary doesn?t have read permissions I think you may get an error like that.  You might also check that the read/search permissions for path components are sane (a blunt way to ensure that would be something like `chmod -R +rX /home/myAccount/bro2.3`).

> FWIW, I set up an account on the JIRA Bro issue tracker page, and could not find the "Create a new ticket" button, so I post this here, instead.

Once logged in, there should be a ?Create? button in the top menu bar.

- Jon


From seth at icir.org  Thu Jun 26 09:28:38 2014
From: seth at icir.org (Seth Hall)
Date: Thu, 26 Jun 2014 12:28:38 -0400
Subject: [Bro] Memory Consumption
In-Reply-To: <CANWz5X-zt+_EFD+J5pp-dm2acp7iJ2cgcPUJYdycrr=YwZ2DgA@mail.gmail.com>
References: <CANWz5X-zt+_EFD+J5pp-dm2acp7iJ2cgcPUJYdycrr=YwZ2DgA@mail.gmail.com>
Message-ID: <AC6C73A5-AE14-48AE-88B2-2110D11E833C@icir.org>


On Jun 26, 2014, at 11:12 AM, Jason Batchelor <jxbatchelor at gmail.com> wrote:

> What is going on here? Is it normal for Bro to take up as much memory as it can? Are others experiancing this too? Is there something I can do to restrict Bro from being such a memory hog?

Bro typically does consume quite a bit of memory and you're a bit tight on memory for the number of workers you're running.  

A common problem is that sometimes there will be state tables that grow too large.  If you load misc/profiling then you should have a prof.log in each of your processes spool directories (they don't log centrally like other logs).  Every so often in there will be an indication of the largest global variables and you can look at that for a start to see if some script is just holding on to more state than you'd like it to.

> I was considering getting more memory for the servers, however based on these tests I cannot be certain Bro will not just run away with the new memory I give it in the same fashion :)

At least memory upgrades are just about the cheapest and easiest upgrade you could possibly do with your system. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140626/46d49005/attachment.bin 

From jxbatchelor at gmail.com  Thu Jun 26 09:43:29 2014
From: jxbatchelor at gmail.com (Jason Batchelor)
Date: Thu, 26 Jun 2014 11:43:29 -0500
Subject: [Bro] Memory Consumption
In-Reply-To: <AC6C73A5-AE14-48AE-88B2-2110D11E833C@icir.org>
References: <CANWz5X-zt+_EFD+J5pp-dm2acp7iJ2cgcPUJYdycrr=YwZ2DgA@mail.gmail.com>
	<AC6C73A5-AE14-48AE-88B2-2110D11E833C@icir.org>
Message-ID: <CANWz5X8MZ5-BCR2YJ50dcqSWUjRs=USfx+voVB6YbZvyvJcsUg@mail.gmail.com>

> At least memory upgrades are just about the cheapest and easiest upgrade
you could possibly do with your system. :)
Very true indeed :)

> Bro typically does consume quite a bit of memory and you're a bit tight
on memory for the number of workers you're running.
Curious what would you recommend for just bro itself? Double, triple this?

> A common problem is that sometimes there will be state tables that grow
too large. If you load misc/profiling then you should have a prof.log in
each of your processes spool directories (they don't log centrally like
other logs). Every so often in there will be an indication of the largest
global variables and you can look at that for a start to see if some script
is just holding on to more state than you'd like it to.

I will definately take a look, thanks for the info!



On Thu, Jun 26, 2014 at 11:28 AM, Seth Hall <seth at icir.org> wrote:

>
> On Jun 26, 2014, at 11:12 AM, Jason Batchelor <jxbatchelor at gmail.com>
> wrote:
>
> > What is going on here? Is it normal for Bro to take up as much memory as
> it can? Are others experiancing this too? Is there something I can do to
> restrict Bro from being such a memory hog?
>
> Bro typically does consume quite a bit of memory and you're a bit tight on
> memory for the number of workers you're running.
>
> A common problem is that sometimes there will be state tables that grow
> too large.  If you load misc/profiling then you should have a prof.log in
> each of your processes spool directories (they don't log centrally like
> other logs).  Every so often in there will be an indication of the largest
> global variables and you can look at that for a start to see if some script
> is just holding on to more state than you'd like it to.
>
> > I was considering getting more memory for the servers, however based on
> these tests I cannot be certain Bro will not just run away with the new
> memory I give it in the same fashion :)
>
> At least memory upgrades are just about the cheapest and easiest upgrade
> you could possibly do with your system. :)
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140626/524ee203/attachment.html 

From seth at icir.org  Thu Jun 26 09:49:30 2014
From: seth at icir.org (Seth Hall)
Date: Thu, 26 Jun 2014 12:49:30 -0400
Subject: [Bro] Memory Consumption
In-Reply-To: <CANWz5X8MZ5-BCR2YJ50dcqSWUjRs=USfx+voVB6YbZvyvJcsUg@mail.gmail.com>
References: <CANWz5X-zt+_EFD+J5pp-dm2acp7iJ2cgcPUJYdycrr=YwZ2DgA@mail.gmail.com>
	<AC6C73A5-AE14-48AE-88B2-2110D11E833C@icir.org>
	<CANWz5X8MZ5-BCR2YJ50dcqSWUjRs=USfx+voVB6YbZvyvJcsUg@mail.gmail.com>
Message-ID: <86D9FB05-B648-4646-A7B7-7F08E663E67F@icir.org>


On Jun 26, 2014, at 12:43 PM, Jason Batchelor <jxbatchelor at gmail.com> wrote:
 
> > Bro typically does consume quite a bit of memory and you're a bit tight on memory for the number of workers you're running.
> Curious what would you recommend for just bro itself? Double, triple this?

It seems like most people just put 128G of memory in Bro boxes now because the cost just isn't really worth going any lower if there's a remote possibility you might use it.

> I will definately take a look, thanks for the info!

Feel free to ask again if you're having trouble.  We really should write up some debugging documentation for this process sometime.  Anyone with experience doing this memory debugging activity up for it?  Doesn't have to be anything fancy, just the steps and various things to look at to figure out what exactly is happening.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140626/3cf52394/attachment.bin 

From jxbatchelor at gmail.com  Thu Jun 26 10:29:37 2014
From: jxbatchelor at gmail.com (Jason Batchelor)
Date: Thu, 26 Jun 2014 12:29:37 -0500
Subject: [Bro] Memory Consumption
In-Reply-To: <86D9FB05-B648-4646-A7B7-7F08E663E67F@icir.org>
References: <CANWz5X-zt+_EFD+J5pp-dm2acp7iJ2cgcPUJYdycrr=YwZ2DgA@mail.gmail.com>
	<AC6C73A5-AE14-48AE-88B2-2110D11E833C@icir.org>
	<CANWz5X8MZ5-BCR2YJ50dcqSWUjRs=USfx+voVB6YbZvyvJcsUg@mail.gmail.com>
	<86D9FB05-B648-4646-A7B7-7F08E663E67F@icir.org>
Message-ID: <CANWz5X-2g9jLKLTOjTLHe4m=kLe14oDrgiMjUHvfk=M9TiZiTg@mail.gmail.com>

Thanks Seth:

I'm not sure I have a license for an experianced bro memory debugger,
however I will document what I've done here for folks in hopes it proves
useful!

I've enabled profiling by adding the following.

Vim /opt/bro/share/bro/site/local.bro
@load misc/profiling

Then enforced the changes...

broctl stop
broctl install
broctl start

At the moment I have 46308184k used  3067820k free memory.

In /var/opt/bro/spool/worker-1-1, prof.log content is captured as you
mentioned (and likewise for all nodes).

Earlier you wrote:

>  Every so often in there will be an indication of the largest global
variables

Is this what you mean (taken from one worker)....?

1403803224.322453 Global_sizes > 100k: 0K
1403803224.322453                Known::known_services = 469K (3130/3130
entries)
1403803224.322453                Cluster::manager2worker_events = 137K
1403803224.322453                Weird::weird_ignore = 31492K
(146569/146569 entries)
1403803224.322453                Known::certs = 58K (310/310 entries)
1403803224.322453                SumStats::threshold_tracker = 668K (4/2916
entries)
1403803224.322453                FTP::ftp_data_expected = 181K (46/46
entries)
1403803224.322453                Notice::suppressing = 595K (2243/2243
entries)
1403803224.322453                Communication::connected_peers = 156K (2/2
entries)
1403803224.322453                SumStats::sending_results = 8028K (3/5545
entries)
1403803224.322453                Software::tracked = 33477K (12424/31111
entries)
1403803224.322453                FTP::cmd_reply_code = 48K (325/325 entries)
1403803224.322453                SumStats::result_store = 27962K (5/19978
entries)
1403803224.322453                SSL::cipher_desc = 97K (356/356 entries)
1403803224.322453                RADIUS::attr_types = 44K (169/169 entries)
1403803224.322453                Weird::actions = 35K (163/163 entries)
1403803224.322453                Known::known_hosts = 3221K (21773/21773
entries)
1403803224.322453                Weird::did_log = 54K (287/287 entries)
1403803224.322453                SSL::recently_validated_certs = 8667K
(24752/24752 entries)
1403803224.322453                Communication::nodes = 188K (4/4 entries)
1403803224.322453                SSL::root_certs = 204K (144/144 entries)
1403803224.322453 Global_sizes total: 116727K
1403803224.322453 Total number of table entries: 213548/260715
1403803239.322685 ------------------------
1403803239.322685 Memory: total=1185296K total_adj=1137108K malloced:
1144576K

Any other pointers on how to interpret this data?

FWIW, here are some additional statistics from the worker prof.log...

grep  "Memory: " prof.log | awk 'NR % 10 == 1'
0.000000 Memory: total=48188K total_adj=0K malloced: 47965K
1403802189.315606 Memory: total=614476K total_adj=566288K malloced: 614022K
1403802339.316381 Memory: total=938380K total_adj=890192K malloced: 938275K
1403802489.317426 Memory: total=1006168K total_adj=957980K malloced:
1003385K
1403802639.318199 Memory: total=1041288K total_adj=993100K malloced:
1035422K
1403802789.319107 Memory: total=1063544K total_adj=1015356K malloced:
1058229K
1403802939.320170 Memory: total=1140652K total_adj=1092464K malloced:
1139608K
1403803089.321327 Memory: total=1184540K total_adj=1136352K malloced:
1179411K
1403803239.322685 Memory: total=1185296K total_adj=1137108K malloced:
1144576K
1403803389.323680 Memory: total=1185296K total_adj=1137108K malloced:
1118961K
1403803539.324677 Memory: total=1185296K total_adj=1137108K malloced:
1092719K
1403803689.325763 Memory: total=1185296K total_adj=1137108K malloced:
1091447K


On Thu, Jun 26, 2014 at 11:49 AM, Seth Hall <seth at icir.org> wrote:

>
> On Jun 26, 2014, at 12:43 PM, Jason Batchelor <jxbatchelor at gmail.com>
> wrote:
>
> > > Bro typically does consume quite a bit of memory and you're a bit
> tight on memory for the number of workers you're running.
> > Curious what would you recommend for just bro itself? Double, triple
> this?
>
> It seems like most people just put 128G of memory in Bro boxes now because
> the cost just isn't really worth going any lower if there's a remote
> possibility you might use it.
>
> > I will definately take a look, thanks for the info!
>
> Feel free to ask again if you're having trouble.  We really should write
> up some debugging documentation for this process sometime.  Anyone with
> experience doing this memory debugging activity up for it?  Doesn't have to
> be anything fancy, just the steps and various things to look at to figure
> out what exactly is happening.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140626/a0b79dd9/attachment.html 

From jxbatchelor at gmail.com  Thu Jun 26 11:26:38 2014
From: jxbatchelor at gmail.com (Jason Batchelor)
Date: Thu, 26 Jun 2014 13:26:38 -0500
Subject: [Bro] Memory Consumption
In-Reply-To: <CANWz5X-2g9jLKLTOjTLHe4m=kLe14oDrgiMjUHvfk=M9TiZiTg@mail.gmail.com>
References: <CANWz5X-zt+_EFD+J5pp-dm2acp7iJ2cgcPUJYdycrr=YwZ2DgA@mail.gmail.com>
	<AC6C73A5-AE14-48AE-88B2-2110D11E833C@icir.org>
	<CANWz5X8MZ5-BCR2YJ50dcqSWUjRs=USfx+voVB6YbZvyvJcsUg@mail.gmail.com>
	<86D9FB05-B648-4646-A7B7-7F08E663E67F@icir.org>
	<CANWz5X-2g9jLKLTOjTLHe4m=kLe14oDrgiMjUHvfk=M9TiZiTg@mail.gmail.com>
Message-ID: <CANWz5X9wwy30aQ16=uvLPOhoGCc6GN2da2+-_0iqXGadJOKxww@mail.gmail.com>

Small follow up to this as well since it may be relevant. I notice the
timers for stale connections seems to increase in paralel with memory...

grep 'ConnectionInactivityTimer' prof.log | awk 'NR % 10 == 1'
1403802069.314888         ConnectionInactivityTimer = 5844
1403802219.315759         ConnectionInactivityTimer = 21747
1403802369.316387         ConnectionInactivityTimer = 32275
1403802519.317613         ConnectionInactivityTimer = 32716
1403802669.318303         ConnectionInactivityTimer = 32597
1403802819.319193         ConnectionInactivityTimer = 34207
1403802969.320204         ConnectionInactivityTimer = 39176
1403803119.321978         ConnectionInactivityTimer = 40394
1403803269.323058         ConnectionInactivityTimer = 38631
1403803419.323688         ConnectionInactivityTimer = 35847
1403803569.324716         ConnectionInactivityTimer = 34432
1403803719.325888         ConnectionInactivityTimer = 34591
1403803869.326713         ConnectionInactivityTimer = 34716
1403804019.327664         ConnectionInactivityTimer = 35361
1403804169.329254         ConnectionInactivityTimer = 35915
1403804319.330507         ConnectionInactivityTimer = 34994
1403804469.331842         ConnectionInactivityTimer = 33212
1403804619.332236         ConnectionInactivityTimer = 32290
1403804769.332993         ConnectionInactivityTimer = 32513
1403804919.333717         ConnectionInactivityTimer = 32592
1403805069.334477         ConnectionInactivityTimer = 32388
1403805219.334875         ConnectionInactivityTimer = 32932
1403805369.335753         ConnectionInactivityTimer = 31771
1403805519.337054         ConnectionInactivityTimer = 28749
1403805669.337563         ConnectionInactivityTimer = 26509
1403805819.339240         ConnectionInactivityTimer = 26654
1403805969.340812         ConnectionInactivityTimer = 26297
1403806119.341841         ConnectionInactivityTimer = 25362
1403806269.344342         ConnectionInactivityTimer = 24435
1403806419.345146         ConnectionInactivityTimer = 24954
1403806569.346057         ConnectionInactivityTimer = 24088
1403806719.347671         ConnectionInactivityTimer = 30207
1403806869.349643         ConnectionInactivityTimer = 34276

Notice the steady increase, then slight decrease, then steady increase
again. Is there a way to control these settings for performance testing
purposes?

I know while I was tuning Suricata, I needed to be mindful of connection
timeouts and due to the volume of flows I am getting I needed to be pretty
aggressive.

Thanks,
Jason



On Thu, Jun 26, 2014 at 12:29 PM, Jason Batchelor <jxbatchelor at gmail.com>
wrote:

> Thanks Seth:
>
> I'm not sure I have a license for an experianced bro memory debugger,
> however I will document what I've done here for folks in hopes it proves
> useful!
>
> I've enabled profiling by adding the following.
>
> Vim /opt/bro/share/bro/site/local.bro
> @load misc/profiling
>
> Then enforced the changes...
>
> broctl stop
> broctl install
> broctl start
>
> At the moment I have 46308184k used  3067820k free memory.
>
> In /var/opt/bro/spool/worker-1-1, prof.log content is captured as you
> mentioned (and likewise for all nodes).
>
> Earlier you wrote:
>
> >  Every so often in there will be an indication of the largest global
> variables
>
> Is this what you mean (taken from one worker)....?
>
> 1403803224.322453 Global_sizes > 100k: 0K
> 1403803224.322453                Known::known_services = 469K (3130/3130
> entries)
> 1403803224.322453                Cluster::manager2worker_events = 137K
> 1403803224.322453                Weird::weird_ignore = 31492K
> (146569/146569 entries)
> 1403803224.322453                Known::certs = 58K (310/310 entries)
> 1403803224.322453                SumStats::threshold_tracker = 668K
> (4/2916 entries)
> 1403803224.322453                FTP::ftp_data_expected = 181K (46/46
> entries)
> 1403803224.322453                Notice::suppressing = 595K (2243/2243
> entries)
> 1403803224.322453                Communication::connected_peers = 156K
> (2/2 entries)
> 1403803224.322453                SumStats::sending_results = 8028K (3/5545
> entries)
> 1403803224.322453                Software::tracked = 33477K (12424/31111
> entries)
> 1403803224.322453                FTP::cmd_reply_code = 48K (325/325
> entries)
> 1403803224.322453                SumStats::result_store = 27962K (5/19978
> entries)
> 1403803224.322453                SSL::cipher_desc = 97K (356/356 entries)
> 1403803224.322453                RADIUS::attr_types = 44K (169/169 entries)
> 1403803224.322453                Weird::actions = 35K (163/163 entries)
> 1403803224.322453                Known::known_hosts = 3221K (21773/21773
> entries)
> 1403803224.322453                Weird::did_log = 54K (287/287 entries)
> 1403803224.322453                SSL::recently_validated_certs = 8667K
> (24752/24752 entries)
> 1403803224.322453                Communication::nodes = 188K (4/4 entries)
> 1403803224.322453                SSL::root_certs = 204K (144/144 entries)
> 1403803224.322453 Global_sizes total: 116727K
> 1403803224.322453 Total number of table entries: 213548/260715
> 1403803239.322685 ------------------------
> 1403803239.322685 Memory: total=1185296K total_adj=1137108K malloced:
> 1144576K
>
> Any other pointers on how to interpret this data?
>
> FWIW, here are some additional statistics from the worker prof.log...
>
> grep  "Memory: " prof.log | awk 'NR % 10 == 1'
> 0.000000 Memory: total=48188K total_adj=0K malloced: 47965K
> 1403802189.315606 Memory: total=614476K total_adj=566288K malloced: 614022K
> 1403802339.316381 Memory: total=938380K total_adj=890192K malloced: 938275K
> 1403802489.317426 Memory: total=1006168K total_adj=957980K malloced:
> 1003385K
> 1403802639.318199 Memory: total=1041288K total_adj=993100K malloced:
> 1035422K
> 1403802789.319107 Memory: total=1063544K total_adj=1015356K malloced:
> 1058229K
> 1403802939.320170 Memory: total=1140652K total_adj=1092464K malloced:
> 1139608K
> 1403803089.321327 Memory: total=1184540K total_adj=1136352K malloced:
> 1179411K
> 1403803239.322685 Memory: total=1185296K total_adj=1137108K malloced:
> 1144576K
> 1403803389.323680 Memory: total=1185296K total_adj=1137108K malloced:
> 1118961K
> 1403803539.324677 Memory: total=1185296K total_adj=1137108K malloced:
> 1092719K
> 1403803689.325763 Memory: total=1185296K total_adj=1137108K malloced:
> 1091447K
>
>
> On Thu, Jun 26, 2014 at 11:49 AM, Seth Hall <seth at icir.org> wrote:
>
>>
>> On Jun 26, 2014, at 12:43 PM, Jason Batchelor <jxbatchelor at gmail.com>
>> wrote:
>>
>> > > Bro typically does consume quite a bit of memory and you're a bit
>> tight on memory for the number of workers you're running.
>> > Curious what would you recommend for just bro itself? Double, triple
>> this?
>>
>> It seems like most people just put 128G of memory in Bro boxes now
>> because the cost just isn't really worth going any lower if there's a
>> remote possibility you might use it.
>>
>> > I will definately take a look, thanks for the info!
>>
>> Feel free to ask again if you're having trouble.  We really should write
>> up some debugging documentation for this process sometime.  Anyone with
>> experience doing this memory debugging activity up for it?  Doesn't have to
>> be anything fancy, just the steps and various things to look at to figure
>> out what exactly is happening.
>>
>>   .Seth
>>
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro.org/
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140626/d5f4136a/attachment.html 

From gc355804 at ohio.edu  Thu Jun 26 15:35:22 2014
From: gc355804 at ohio.edu (Gilbert Clark)
Date: Thu, 26 Jun 2014 18:35:22 -0400
Subject: [Bro] Memory Consumption
In-Reply-To: <CANWz5X9wwy30aQ16=uvLPOhoGCc6GN2da2+-_0iqXGadJOKxww@mail.gmail.com>
References: <CANWz5X-zt+_EFD+J5pp-dm2acp7iJ2cgcPUJYdycrr=YwZ2DgA@mail.gmail.com>	<AC6C73A5-AE14-48AE-88B2-2110D11E833C@icir.org>	<CANWz5X8MZ5-BCR2YJ50dcqSWUjRs=USfx+voVB6YbZvyvJcsUg@mail.gmail.com>	<86D9FB05-B648-4646-A7B7-7F08E663E67F@icir.org>	<CANWz5X-2g9jLKLTOjTLHe4m=kLe14oDrgiMjUHvfk=M9TiZiTg@mail.gmail.com>
	<CANWz5X9wwy30aQ16=uvLPOhoGCc6GN2da2+-_0iqXGadJOKxww@mail.gmail.com>
Message-ID: <53ACA02A.70709@ohio.edu>

Hi:

I believe this particular timer is a general timer used to track 
inactivity for all protocols (but someone can correct me if I'm wrong 
:).  Maybe try tuning the following:

const tcp_inactivity_timeout = 5 min &redef;
const udp_inactivity_timeout = 1 min &redef;
const icmp_inactivity_timeout = 1 min &redef;

Reference: 
http://www.notary.icsi.berkeley.edu/sphinx-git/scripts/base/init-bare.bro.html#id-udp_inactivity_timeout

Also, I believe it's possible to set timeouts per-connection based on 
properties of the established connections.  For an example of doing this 
/ how this might be useful, take a look at:

https://bro.org/sphinx/scripts/base/protocols/conn/inactivity.bro.html

Re: interpreting prof.log output -- a few notes from my experience:

There should be lines that include the number of connections currently 
active for each major protocol type, e.g:

Conns: tcp=1/130 udp=1/70 icmp=0/0

Syntax here is: tcp=<current active connections>/<maximum number of 
active connections ever observed> udp=<current active 
connections>/<maximum number of active connections ever observed> 
icmp=<current active connections>/<maximum number of active connections 
ever observed>

The line following the above includes more detailed connection overhead 
information:

Conns: total=6528 current=2/2 ext=0 mem=9312K avg=4656.0 table=24K 
connvals=6K

A few notes about fields that might be useful there:

* total=total number of connections (aggregate: not just at this 
particular moment)
* current=X/Y:  X and Y are two counts that will usually differ to some 
extent, but both count the number of connections observed
     - X: the number of active connections in total  (not necessarily 
all of which are kept in state tables)
     - Y: the number of connections stored in bro's state tables (tcp + 
udp + icmp) at this moment in time
* avg=average memory use (in bytes) per active connection
* table=total amount of memory used by the state tables (tcp + udp + icmp)

'avg' and 'table' are only recorded occasionally because computing these 
values can be expensive.  When that "Global_sizes ..." output is 
included in a log entry, these numbers will be accurate.  Otherwise, 
they will be 0.

For an idea of the overhead associated with the Timer objects themselves 
(read: the overhead for the timers isn't included in the overhead 
computed for the connection state), take a look at the line that looks 
something like:

Timers: current=19 max=19 mem=1K lag=0.00s

*current=number of timers currently active in total
*max=maximum number of timers ever active at once
*mem=total memory consumed by all of the currently active timers 
(usually pretty small compared to other things, though)

Also, one other note: under 'Threads', there's a bunch of lines that 
look something like:

http/Log::WRITER_ASCII    in=11318 out=10882 pending=0/0 (#queue r/w: 
in=11318/11318 out=10882/10882)
ssl/Log::WRITER_ASCII     in=10931 out=10878 pending=0/0 (#queue r/w: 
in=10931/10931 out=10878/10878)
files/Log::WRITER_ASCII   in=10989 out=10792 pending=0/0 (#queue r/w: 
in=10989/10989 out=10792/10792)
dhcp/Log::WRITER_ASCII    in=1031 out=1029 pending=0/0 (#queue r/w: 
in=1031/1031 out=1029/1029)

Generally, pending X/Y will describe how much memory is currently being 
consumed (relatively speaking) by messages waiting to be written to a 
log file / that have been read from that input source but not yet 
processed by bro.

A pending X/Y that grows steadily over time is an indication that bro 
could eventually run out of room to store outstanding log / input 
framework messages, and that these messages could eventually come to 
consume so much memory that the worker would thrash the machine into 
sweet digital oblivion.

Hope something in there is useful,
Gilbert

On 6/26/14, 2:26 PM, Jason Batchelor wrote:
> Small follow up to this as well since it may be relevant. I notice the 
> timers for stale connections seems to increase in paralel with memory...
> grep 'ConnectionInactivityTimer' prof.log | awk 'NR % 10 == 1'
> 1403802069.314888         ConnectionInactivityTimer = 5844
> 1403802219.315759         ConnectionInactivityTimer = 21747
> 1403802369.316387         ConnectionInactivityTimer = 32275
> 1403802519.317613         ConnectionInactivityTimer = 32716
> 1403802669.318303         ConnectionInactivityTimer = 32597
> 1403802819.319193         ConnectionInactivityTimer = 34207
> 1403802969.320204         ConnectionInactivityTimer = 39176
> 1403803119.321978         ConnectionInactivityTimer = 40394
> 1403803269.323058         ConnectionInactivityTimer = 38631
> 1403803419.323688         ConnectionInactivityTimer = 35847
> 1403803569.324716         ConnectionInactivityTimer = 34432
> 1403803719.325888         ConnectionInactivityTimer = 34591
> 1403803869.326713         ConnectionInactivityTimer = 34716
> 1403804019.327664         ConnectionInactivityTimer = 35361
> 1403804169.329254         ConnectionInactivityTimer = 35915
> 1403804319.330507         ConnectionInactivityTimer = 34994
> 1403804469.331842         ConnectionInactivityTimer = 33212
> 1403804619.332236         ConnectionInactivityTimer = 32290
> 1403804769.332993         ConnectionInactivityTimer = 32513
> 1403804919.333717         ConnectionInactivityTimer = 32592
> 1403805069.334477         ConnectionInactivityTimer = 32388
> 1403805219.334875         ConnectionInactivityTimer = 32932
> 1403805369.335753         ConnectionInactivityTimer = 31771
> 1403805519.337054         ConnectionInactivityTimer = 28749
> 1403805669.337563         ConnectionInactivityTimer = 26509
> 1403805819.339240         ConnectionInactivityTimer = 26654
> 1403805969.340812         ConnectionInactivityTimer = 26297
> 1403806119.341841         ConnectionInactivityTimer = 25362
> 1403806269.344342         ConnectionInactivityTimer = 24435
> 1403806419.345146         ConnectionInactivityTimer = 24954
> 1403806569.346057         ConnectionInactivityTimer = 24088
> 1403806719.347671         ConnectionInactivityTimer = 30207
> 1403806869.349643         ConnectionInactivityTimer = 34276
> Notice the steady increase, then slight decrease, then steady increase 
> again. Is there a way to control these settings for performance 
> testing purposes?
> I know while I was tuning Suricata, I needed to be mindful of 
> connection timeouts and due to the volume of flows I am getting I 
> needed to be pretty aggressive.
> Thanks,
> Jason
>
>
> On Thu, Jun 26, 2014 at 12:29 PM, Jason Batchelor 
> <jxbatchelor at gmail.com <mailto:jxbatchelor at gmail.com>> wrote:
>
>     Thanks Seth:
>
>     I'm not sure I have a license for an experianced bro memory
>     debugger, however I will document what I've done here for folks in
>     hopes it proves useful!
>
>     I've enabled profiling by adding the following.
>
>     Vim /opt/bro/share/bro/site/local.bro
>     @load misc/profiling
>
>     Then enforced the changes...
>
>     broctl stop
>     broctl install
>     broctl start
>
>     At the moment I have 46308184k used  3067820k free memory.
>
>     In /var/opt/bro/spool/worker-1-1, prof.log content is captured as
>     you mentioned (and likewise for all nodes).
>
>     Earlier you wrote:
>
>     >  Every so often in there will be an indication of the largest
>     global variables
>
>     Is this what you mean (taken from one worker)....?
>
>     1403803224.322453 Global_sizes > 100k: 0K
>     1403803224.322453              Known::known_services = 469K
>     (3130/3130 entries)
>     1403803224.322453              Cluster::manager2worker_events = 137K
>     1403803224.322453              Weird::weird_ignore = 31492K
>     (146569/146569 entries)
>     1403803224.322453              Known::certs = 58K (310/310 entries)
>     1403803224.322453              SumStats::threshold_tracker = 668K
>     (4/2916 entries)
>     1403803224.322453              FTP::ftp_data_expected = 181K
>     (46/46 entries)
>     1403803224.322453              Notice::suppressing = 595K
>     (2243/2243 entries)
>     1403803224.322453              Communication::connected_peers =
>     156K (2/2 entries)
>     1403803224.322453              SumStats::sending_results = 8028K
>     (3/5545 entries)
>     1403803224.322453              Software::tracked = 33477K
>     (12424/31111 entries)
>     1403803224.322453              FTP::cmd_reply_code = 48K (325/325
>     entries)
>     1403803224.322453              SumStats::result_store = 27962K
>     (5/19978 entries)
>     1403803224.322453              SSL::cipher_desc = 97K (356/356
>     entries)
>     1403803224.322453              RADIUS::attr_types = 44K (169/169
>     entries)
>     1403803224.322453              Weird::actions = 35K (163/163 entries)
>     1403803224.322453              Known::known_hosts = 3221K
>     (21773/21773 <tel:%2821773%2F21773> entries)
>     1403803224.322453              Weird::did_log = 54K (287/287 entries)
>     1403803224.322453              SSL::recently_validated_certs =
>     8667K (24752/24752 entries)
>     1403803224.322453              Communication::nodes = 188K (4/4
>     entries)
>     1403803224.322453              SSL::root_certs = 204K (144/144
>     entries)
>     1403803224.322453 Global_sizes total: 116727K
>     1403803224.322453 Total number of table entries: 213548/260715
>     1403803239.322685 ------------------------
>     1403803239.322685 Memory: total=1185296K total_adj=1137108K
>     malloced: 1144576K
>
>     Any other pointers on how to interpret this data?
>
>     FWIW, here are some additional statistics from the worker prof.log...
>
>     grep  "Memory: " prof.log | awk 'NR % 10 == 1'
>     0.000000 Memory: total=48188K total_adj=0K malloced: 47965K
>     1403802189.315606 Memory: total=614476K total_adj=566288K
>     malloced: 614022K
>     1403802339.316381 Memory: total=938380K total_adj=890192K
>     malloced: 938275K
>     1403802489.317426 Memory: total=1006168K total_adj=957980K
>     malloced: 1003385K
>     1403802639.318199 Memory: total=1041288K total_adj=993100K
>     malloced: 1035422K
>     1403802789.319107 Memory: total=1063544K total_adj=1015356K
>     malloced: 1058229K
>     1403802939.320170 Memory: total=1140652K total_adj=1092464K
>     malloced: 1139608K
>     1403803089.321327 Memory: total=1184540K total_adj=1136352K
>     malloced: 1179411K
>     1403803239.322685 Memory: total=1185296K total_adj=1137108K
>     malloced: 1144576K
>     1403803389.323680 Memory: total=1185296K total_adj=1137108K
>     malloced: 1118961K
>     1403803539.324677 Memory: total=1185296K total_adj=1137108K
>     malloced: 1092719K
>     1403803689.325763 Memory: total=1185296K total_adj=1137108K
>     malloced: 1091447K
>
>
>     On Thu, Jun 26, 2014 at 11:49 AM, Seth Hall <seth at icir.org
>     <mailto:seth at icir.org>> wrote:
>
>
>         On Jun 26, 2014, at 12:43 PM, Jason Batchelor
>         <jxbatchelor at gmail.com <mailto:jxbatchelor at gmail.com>> wrote:
>
>         > > Bro typically does consume quite a bit of memory and
>         you're a bit tight on memory for the number of workers you're
>         running.
>         > Curious what would you recommend for just bro itself?
>         Double, triple this?
>
>         It seems like most people just put 128G of memory in Bro boxes
>         now because the cost just isn't really worth going any lower
>         if there's a remote possibility you might use it.
>
>         > I will definately take a look, thanks for the info!
>
>         Feel free to ask again if you're having trouble.  We really
>         should write up some debugging documentation for this process
>         sometime.  Anyone with experience doing this memory debugging
>         activity up for it?  Doesn't have to be anything fancy, just
>         the steps and various things to look at to figure out what
>         exactly is happening.
>
>           .Seth
>
>         --
>         Seth Hall
>         International Computer Science Institute
>         (Bro) because everyone has a network
>         http://www.bro.org/
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140626/94bba0f3/attachment.html 

From r.fulton at auckland.ac.nz  Thu Jun 26 21:42:49 2014
From: r.fulton at auckland.ac.nz (Russell Fulton)
Date: Fri, 27 Jun 2014 04:42:49 +0000
Subject: [Bro] can not start bro
Message-ID: <CD4BEE2D-2892-4B88-B5C5-6FFCF1D742D1@auckland.ac.nz>

I had a series of problems on one sensor (disk full and consequent issues) and now when I try 

broctl start

it tries to start the manager and just hangs.

rful011 at secmontst01:/opt/bro/etc$ sudo broctl start
warning: removing stale lock
starting manager ?

after killing it I find:

rful011 at secmontst01:/opt/bro/etc$ cat ~sensors/data/dmzo/bro-spool/manager/stderr.log 
nohup: failed to run command `/run-bro': No such file or directory

timestamp is current.

Bro is installed using securityoinion-bro ? I have tried removing and installing it again but it does not make any difference.

Any ideas on what to look for.  

Russell




From dnthayer at illinois.edu  Thu Jun 26 23:46:26 2014
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Fri, 27 Jun 2014 01:46:26 -0500
Subject: [Bro] can not start bro
In-Reply-To: <CD4BEE2D-2892-4B88-B5C5-6FFCF1D742D1@auckland.ac.nz>
References: <CD4BEE2D-2892-4B88-B5C5-6FFCF1D742D1@auckland.ac.nz>
Message-ID: <53AD1342.5040409@illinois.edu>

You probably forgot to run "broctl install".

-Daniel


On 06/26/2014 11:42 PM, Russell Fulton wrote:
> I had a series of problems on one sensor (disk full and consequent issues) and now when I try
>
> broctl start
>
> it tries to start the manager and just hangs.
>
> rful011 at secmontst01:/opt/bro/etc$ sudo broctl start
> warning: removing stale lock
> starting manager ?
>
> after killing it I find:
>
> rful011 at secmontst01:/opt/bro/etc$ cat ~sensors/data/dmzo/bro-spool/manager/stderr.log
> nohup: failed to run command `/run-bro': No such file or directory
>
> timestamp is current.
>
> Bro is installed using securityoinion-bro ? I have tried removing and installing it again but it does not make any difference.
>
> Any ideas on what to look for.
>
> Russell
>


From r.fulton at auckland.ac.nz  Fri Jun 27 00:20:27 2014
From: r.fulton at auckland.ac.nz (Russell Fulton)
Date: Fri, 27 Jun 2014 19:20:27 +1200
Subject: [Bro] can not start bro
In-Reply-To: <53AD1342.5040409@illinois.edu>
References: <CD4BEE2D-2892-4B88-B5C5-6FFCF1D742D1@auckland.ac.nz>
	<53AD1342.5040409@illinois.edu>
Message-ID: <9805096A-51FF-47F7-8613-B66C8A77E2A0@auckland.ac.nz>


On 27/06/2014, at 6:46 pm, Daniel Thayer <dnthayer at illinois.edu> wrote:

> You probably forgot to run "broctl install?.

I?m an idiot!  but it would be nice if broctl start did some sanity checking!

I manage my sensors with puppet ? I need to add and exec to run broctl install if the bro-spool/installed-scripts? does not exist after the config files are put in place.

> 
> -Daniel
> 
> 
> On 06/26/2014 11:42 PM, Russell Fulton wrote:
>> I had a series of problems on one sensor (disk full and consequent issues) and now when I try
>> 
>> broctl start
>> 
>> it tries to start the manager and just hangs.
>> 
>> rful011 at secmontst01:/opt/bro/etc$ sudo broctl start
>> warning: removing stale lock
>> starting manager ?
>> 
>> after killing it I find:
>> 
>> rful011 at secmontst01:/opt/bro/etc$ cat ~sensors/data/dmzo/bro-spool/manager/stderr.log
>> nohup: failed to run command `/run-bro': No such file or directory
>> 
>> timestamp is current.
>> 
>> Bro is installed using securityoinion-bro ? I have tried removing and installing it again but it does not make any difference.
>> 
>> Any ideas on what to look for.
>> 
>> Russell
>> 




From vlad at grigorescu.org  Fri Jun 27 04:47:00 2014
From: vlad at grigorescu.org (Vlad Grigorescu)
Date: Fri, 27 Jun 2014 07:47:00 -0400
Subject: [Bro] can not start bro
In-Reply-To: <9805096A-51FF-47F7-8613-B66C8A77E2A0@auckland.ac.nz>
References: <CD4BEE2D-2892-4B88-B5C5-6FFCF1D742D1@auckland.ac.nz>
	<53AD1342.5040409@illinois.edu>
	<9805096A-51FF-47F7-8613-B66C8A77E2A0@auckland.ac.nz>
Message-ID: <CANUJ_FnFR3vA6rSAcXaGqssTdW-t3UPn+OTJjZQ5fO+nf5NQTQ@mail.gmail.com>

On Fri, Jun 27, 2014 at 3:20 AM, Russell Fulton <r.fulton at auckland.ac.nz>
wrote:

>
> I manage my sensors with puppet ? I need to add and exec to run broctl
> install if the bro-spool/installed-scripts? does not exist after the config
> files are put in place.


You should try this Puppet module: https://forge.puppetlabs.com/panaman/bro

  --Vlad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140627/c6ed81de/attachment.html 

From jxbatchelor at gmail.com  Fri Jun 27 08:29:12 2014
From: jxbatchelor at gmail.com (Jason Batchelor)
Date: Fri, 27 Jun 2014 10:29:12 -0500
Subject: [Bro] Memory Consumption
In-Reply-To: <53ACA02A.70709@ohio.edu>
References: <CANWz5X-zt+_EFD+J5pp-dm2acp7iJ2cgcPUJYdycrr=YwZ2DgA@mail.gmail.com>
	<AC6C73A5-AE14-48AE-88B2-2110D11E833C@icir.org>
	<CANWz5X8MZ5-BCR2YJ50dcqSWUjRs=USfx+voVB6YbZvyvJcsUg@mail.gmail.com>
	<86D9FB05-B648-4646-A7B7-7F08E663E67F@icir.org>
	<CANWz5X-2g9jLKLTOjTLHe4m=kLe14oDrgiMjUHvfk=M9TiZiTg@mail.gmail.com>
	<CANWz5X9wwy30aQ16=uvLPOhoGCc6GN2da2+-_0iqXGadJOKxww@mail.gmail.com>
	<53ACA02A.70709@ohio.edu>
Message-ID: <CANWz5X_ucqGgLrNbtBVzi-LFWpn5U=0Fm87324=9=z3COWne=w@mail.gmail.com>

Thanks Gilbert!

I think I am getting close to at least isolating the issue.

I redefined some of the inactivity timeout values to something pregger
aggressive...
redef tcp_inactivity_timeout = 15 sec;
redef udp_inactivity_timeout = 5 sec;
redef icmp_inactivity_timeout = 5 sec;

After committing the changes and restarting I am still seeing the same kind
of slow memory consumption behavior.

I checked the IO statistics you gave above and think this is where I am
getting backed up. Below is a brief escalation of just the http logs
themselves.

grep 'http/Log::WRITER_ASCII' prof.log | awk 'NR % 10 == 1'
1403880551.747191   http/Log::WRITER_ASCII    in=40 out=11 pending=0/0
(#queue r/w: in=40/40 out=11/11)
1403880701.759282   http/Log::WRITER_ASCII    in=632 out=160 pending=0/0
(#queue r/w: in=632/632 out=160/160)
1403880851.764553   http/Log::WRITER_ASCII    in=1254 out=310 pending=0/0
(#queue r/w: in=1254/1254 out=310/310)
1403881001.794827   http/Log::WRITER_ASCII    in=1881 out=459 pending=0/0
(#queue r/w: in=1881/1881 out=459/459)
1403881151.907771   http/Log::WRITER_ASCII    in=2496 out=607 pending=0/0
(#queue r/w: in=2496/2496 out=607/607)
1403881302.133110   http/Log::WRITER_ASCII    in=3140 out=754 pending=0/0
(#queue r/w: in=3140/3140 out=754/754)
1403881452.684259   http/Log::WRITER_ASCII    in=3781 out=900 pending=0/0
(#queue r/w: in=3781/3781 out=900/900)
1403881611.446692   http/Log::WRITER_ASCII    in=4321 out=1000 pending=0/0
(#queue r/w: in=4321/4321 out=1000/1000)
1403881783.945970   http/Log::WRITER_ASCII    in=4816 out=1069 pending=0/0
(#queue r/w: in=4816/4816 out=1069/1069)
1403881991.154812   http/Log::WRITER_ASCII    in=5435 out=1105 pending=0/0
(#queue r/w: in=5435/5435 out=1105/1105)
1403882156.814938   http/Log::WRITER_ASCII    in=6066 out=1190 pending=0/0
(#queue r/w: in=6066/6066 out=1190/1190)
1403882336.215055   http/Log::WRITER_ASCII    in=6690 out=1267 pending=0/0
(#queue r/w: in=6690/6690 out=1267/1267)
1403882494.089058   http/Log::WRITER_ASCII    in=7350 out=1377 pending=0/0
(#queue r/w: in=7350/7350 out=1377/1377)

If I am interpreting this correctly, I am far exceeding my ability to write
out logs as time goes on, resulting in a backup of that data in memory
presumably. The same kind of behavior is seen in other log types as well.

Am I interpreting this correctly? If so the solution seems to be I need
faster drives and/or more memory :)



On Thu, Jun 26, 2014 at 5:35 PM, Gilbert Clark <gc355804 at ohio.edu> wrote:

>  Hi:
>
> I believe this particular timer is a general timer used to track
> inactivity for all protocols (but someone can correct me if I'm wrong :).
> Maybe try tuning the following:
>
> const tcp_inactivity_timeout = 5 min &redef;
> const udp_inactivity_timeout = 1 min &redef;
> const icmp_inactivity_timeout = 1 min &redef;
>
> Reference:
> http://www.notary.icsi.berkeley.edu/sphinx-git/scripts/base/init-bare.bro.html#id-udp_inactivity_timeout
>
> Also, I believe it's possible to set timeouts per-connection based on
> properties of the established connections.  For an example of doing this /
> how this might be useful, take a look at:
>
> https://bro.org/sphinx/scripts/base/protocols/conn/inactivity.bro.html
>
> Re: interpreting prof.log output -- a few notes from my experience:
>
> There should be lines that include the number of connections currently
> active for each major protocol type, e.g:
>
> Conns: tcp=1/130 udp=1/70 icmp=0/0
>
> Syntax here is: tcp=<current active connections>/<maximum number of active
> connections ever observed> udp=<current active connections>/<maximum number
> of active connections ever observed> icmp=<current active
> connections>/<maximum number of active connections ever observed>
>
> The line following the above includes more detailed connection overhead
> information:
>
> Conns: total=6528 current=2/2 ext=0 mem=9312K avg=4656.0 table=24K
> connvals=6K
>
> A few notes about fields that might be useful there:
>
> * total=total number of connections (aggregate: not just at this
> particular moment)
> * current=X/Y:  X and Y are two counts that will usually differ to some
> extent, but both count the number of connections observed
>     - X: the number of active connections in total  (not necessarily all
> of which are kept in state tables)
>     - Y: the number of connections stored in bro's state tables (tcp + udp
> + icmp) at this moment in time
> * avg=average memory use (in bytes) per active connection
> * table=total amount of memory used by the state tables (tcp + udp + icmp)
>
> 'avg' and 'table' are only recorded occasionally because computing these
> values can be expensive.  When that "Global_sizes ..." output is included
> in a log entry, these numbers will be accurate.  Otherwise, they will be 0.
>
> For an idea of the overhead associated with the Timer objects themselves
> (read: the overhead for the timers isn't included in the overhead computed
> for the connection state), take a look at the line that looks something
> like:
>
> Timers: current=19 max=19 mem=1K lag=0.00s
>
> *current=number of timers currently active in total
> *max=maximum number of timers ever active at once
> *mem=total memory consumed by all of the currently active timers (usually
> pretty small compared to other things, though)
>
> Also, one other note: under 'Threads', there's a bunch of lines that look
> something like:
>
> http/Log::WRITER_ASCII    in=11318 out=10882 pending=0/0 (#queue r/w:
> in=11318/11318 out=10882/10882)
> ssl/Log::WRITER_ASCII     in=10931 out=10878 pending=0/0 (#queue r/w:
> in=10931/10931 out=10878/10878)
> files/Log::WRITER_ASCII   in=10989 out=10792 pending=0/0 (#queue r/w:
> in=10989/10989 out=10792/10792)
> dhcp/Log::WRITER_ASCII    in=1031 out=1029 pending=0/0 (#queue r/w:
> in=1031/1031 out=1029/1029)
>
> Generally, pending X/Y will describe how much memory is currently being
> consumed (relatively speaking) by messages waiting to be written to a log
> file / that have been read from that input source but not yet processed by
> bro.
>
> A pending X/Y that grows steadily over time is an indication that bro
> could eventually run out of room to store outstanding log / input framework
> messages, and that these messages could eventually come to consume so much
> memory that the worker would thrash the machine into sweet digital oblivion.
>
> Hope something in there is useful,
> Gilbert
>
>
> On 6/26/14, 2:26 PM, Jason Batchelor wrote:
>
>  Small follow up to this as well since it may be relevant. I notice the
> timers for stale connections seems to increase in paralel with memory...
>
> grep 'ConnectionInactivityTimer' prof.log | awk 'NR % 10 == 1'
> 1403802069.314888         ConnectionInactivityTimer = 5844
> 1403802219.315759         ConnectionInactivityTimer = 21747
> 1403802369.316387         ConnectionInactivityTimer = 32275
> 1403802519.317613         ConnectionInactivityTimer = 32716
> 1403802669.318303         ConnectionInactivityTimer = 32597
> 1403802819.319193         ConnectionInactivityTimer = 34207
> 1403802969.320204         ConnectionInactivityTimer = 39176
> 1403803119.321978         ConnectionInactivityTimer = 40394
> 1403803269.323058         ConnectionInactivityTimer = 38631
> 1403803419.323688         ConnectionInactivityTimer = 35847
> 1403803569.324716         ConnectionInactivityTimer = 34432
> 1403803719.325888         ConnectionInactivityTimer = 34591
> 1403803869.326713         ConnectionInactivityTimer = 34716
> 1403804019.327664         ConnectionInactivityTimer = 35361
> 1403804169.329254         ConnectionInactivityTimer = 35915
> 1403804319.330507         ConnectionInactivityTimer = 34994
> 1403804469.331842         ConnectionInactivityTimer = 33212
> 1403804619.332236         ConnectionInactivityTimer = 32290
> 1403804769.332993         ConnectionInactivityTimer = 32513
> 1403804919.333717         ConnectionInactivityTimer = 32592
> 1403805069.334477         ConnectionInactivityTimer = 32388
> 1403805219.334875         ConnectionInactivityTimer = 32932
> 1403805369.335753         ConnectionInactivityTimer = 31771
> 1403805519.337054         ConnectionInactivityTimer = 28749
> 1403805669.337563         ConnectionInactivityTimer = 26509
> 1403805819.339240         ConnectionInactivityTimer = 26654
> 1403805969.340812         ConnectionInactivityTimer = 26297
> 1403806119.341841         ConnectionInactivityTimer = 25362
> 1403806269.344342         ConnectionInactivityTimer = 24435
> 1403806419.345146         ConnectionInactivityTimer = 24954
> 1403806569.346057         ConnectionInactivityTimer = 24088
> 1403806719.347671         ConnectionInactivityTimer = 30207
> 1403806869.349643         ConnectionInactivityTimer = 34276
>
> Notice the steady increase, then slight decrease, then steady increase
> again. Is there a way to control these settings for performance testing
> purposes?
>
> I know while I was tuning Suricata, I needed to be mindful of connection
> timeouts and due to the volume of flows I am getting I needed to be pretty
> aggressive.
>
> Thanks,
> Jason
>
>
>
> On Thu, Jun 26, 2014 at 12:29 PM, Jason Batchelor <jxbatchelor at gmail.com>
> wrote:
>
>> Thanks Seth:
>>
>>  I'm not sure I have a license for an experianced bro memory debugger,
>> however I will document what I've done here for folks in hopes it proves
>> useful!
>>
>>  I've enabled profiling by adding the following.
>>
>>  Vim /opt/bro/share/bro/site/local.bro
>> @load misc/profiling
>>
>>  Then enforced the changes...
>>
>>  broctl stop
>> broctl install
>> broctl start
>>
>>  At the moment I have 46308184k used  3067820k free memory.
>>
>>  In /var/opt/bro/spool/worker-1-1, prof.log content is captured as you
>> mentioned (and likewise for all nodes).
>>
>>  Earlier you wrote:
>>
>>  >  Every so often in there will be an indication of the largest global
>> variables
>>
>>  Is this what you mean (taken from one worker)....?
>>
>>  1403803224.322453 Global_sizes > 100k: 0K
>> 1403803224.322453                Known::known_services = 469K (3130/3130
>> entries)
>> 1403803224.322453                Cluster::manager2worker_events = 137K
>> 1403803224.322453                Weird::weird_ignore = 31492K
>> (146569/146569 entries)
>> 1403803224.322453                Known::certs = 58K (310/310 entries)
>> 1403803224.322453                SumStats::threshold_tracker = 668K
>> (4/2916 entries)
>> 1403803224.322453                FTP::ftp_data_expected = 181K (46/46
>> entries)
>> 1403803224.322453                Notice::suppressing = 595K (2243/2243
>> entries)
>> 1403803224.322453                Communication::connected_peers = 156K
>> (2/2 entries)
>> 1403803224.322453                SumStats::sending_results = 8028K
>> (3/5545 entries)
>> 1403803224.322453                Software::tracked = 33477K (12424/31111
>> entries)
>> 1403803224.322453                FTP::cmd_reply_code = 48K (325/325
>> entries)
>> 1403803224.322453                SumStats::result_store = 27962K (5/19978
>> entries)
>> 1403803224.322453                SSL::cipher_desc = 97K (356/356 entries)
>> 1403803224.322453                RADIUS::attr_types = 44K (169/169
>> entries)
>> 1403803224.322453                Weird::actions = 35K (163/163 entries)
>> 1403803224.322453                Known::known_hosts = 3221K (21773/21773
>> entries)
>> 1403803224.322453                Weird::did_log = 54K (287/287 entries)
>> 1403803224.322453                SSL::recently_validated_certs = 8667K
>> (24752/24752 entries)
>> 1403803224.322453                Communication::nodes = 188K (4/4 entries)
>> 1403803224.322453                SSL::root_certs = 204K (144/144 entries)
>> 1403803224.322453 Global_sizes total: 116727K
>>  1403803224.322453 Total number of table entries: 213548/260715
>> 1403803239.322685 ------------------------
>> 1403803239.322685 Memory: total=1185296K total_adj=1137108K malloced:
>> 1144576K
>>
>>  Any other pointers on how to interpret this data?
>>
>>  FWIW, here are some additional statistics from the worker prof.log...
>>
>>  grep  "Memory: " prof.log | awk 'NR % 10 == 1'
>> 0.000000 Memory: total=48188K total_adj=0K malloced: 47965K
>> 1403802189.315606 Memory: total=614476K total_adj=566288K malloced:
>> 614022K
>> 1403802339.316381 Memory: total=938380K total_adj=890192K malloced:
>> 938275K
>> 1403802489.317426 Memory: total=1006168K total_adj=957980K malloced:
>> 1003385K
>> 1403802639.318199 Memory: total=1041288K total_adj=993100K malloced:
>> 1035422K
>> 1403802789.319107 Memory: total=1063544K total_adj=1015356K malloced:
>> 1058229K
>> 1403802939.320170 Memory: total=1140652K total_adj=1092464K malloced:
>> 1139608K
>> 1403803089.321327 Memory: total=1184540K total_adj=1136352K malloced:
>> 1179411K
>> 1403803239.322685 Memory: total=1185296K total_adj=1137108K malloced:
>> 1144576K
>> 1403803389.323680 Memory: total=1185296K total_adj=1137108K malloced:
>> 1118961K
>> 1403803539.324677 Memory: total=1185296K total_adj=1137108K malloced:
>> 1092719K
>> 1403803689.325763 Memory: total=1185296K total_adj=1137108K malloced:
>> 1091447K
>>
>>
>> On Thu, Jun 26, 2014 at 11:49 AM, Seth Hall <seth at icir.org> wrote:
>>
>>>
>>> On Jun 26, 2014, at 12:43 PM, Jason Batchelor <jxbatchelor at gmail.com>
>>> wrote:
>>>
>>> > > Bro typically does consume quite a bit of memory and you're a bit
>>> tight on memory for the number of workers you're running.
>>> > Curious what would you recommend for just bro itself? Double, triple
>>> this?
>>>
>>>  It seems like most people just put 128G of memory in Bro boxes now
>>> because the cost just isn't really worth going any lower if there's a
>>> remote possibility you might use it.
>>>
>>> > I will definately take a look, thanks for the info!
>>>
>>>  Feel free to ask again if you're having trouble.  We really should
>>> write up some debugging documentation for this process sometime.  Anyone
>>> with experience doing this memory debugging activity up for it?  Doesn't
>>> have to be anything fancy, just the steps and various things to look at to
>>> figure out what exactly is happening.
>>>
>>>   .Seth
>>>
>>> --
>>> Seth Hall
>>> International Computer Science Institute
>>> (Bro) because everyone has a network
>>> http://www.bro.org/
>>>
>>>
>>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140627/5e9eba0d/attachment.html 

From gc355804 at ohio.edu  Fri Jun 27 13:39:12 2014
From: gc355804 at ohio.edu (Gilbert Clark)
Date: Fri, 27 Jun 2014 16:39:12 -0400
Subject: [Bro] Memory Consumption
In-Reply-To: <CANWz5X_ucqGgLrNbtBVzi-LFWpn5U=0Fm87324=9=z3COWne=w@mail.gmail.com>
References: <CANWz5X-zt+_EFD+J5pp-dm2acp7iJ2cgcPUJYdycrr=YwZ2DgA@mail.gmail.com>	<AC6C73A5-AE14-48AE-88B2-2110D11E833C@icir.org>	<CANWz5X8MZ5-BCR2YJ50dcqSWUjRs=USfx+voVB6YbZvyvJcsUg@mail.gmail.com>	<86D9FB05-B648-4646-A7B7-7F08E663E67F@icir.org>	<CANWz5X-2g9jLKLTOjTLHe4m=kLe14oDrgiMjUHvfk=M9TiZiTg@mail.gmail.com>	<CANWz5X9wwy30aQ16=uvLPOhoGCc6GN2da2+-_0iqXGadJOKxww@mail.gmail.com>	<53ACA02A.70709@ohio.edu>
	<CANWz5X_ucqGgLrNbtBVzi-LFWpn5U=0Fm87324=9=z3COWne=w@mail.gmail.com>
Message-ID: <53ADD670.3000401@ohio.edu>

Hi Jason:

It's only when *pending* begins to grow that there's a large queue of 
messages waiting to be written (as far as I know).  In this case, 
pending stays at 0/0 for each of those log updates, so I don't think 
that's an issue (for this log, at least :)

One possibility here is going to be that bro is actually leaking memory 
somewhere, and you're lucky enough to have found a bug :) Usually this 
is going to be identified by a steady rise in memory usage over time 
with a relatively constant traffic volume / types. One thing to try (if 
it's something you feel comfortable with) might be to pull down the 
current copy of bro in master, building that, and pushing this out to 
the nodes to see what kind of an effect that has (if anything).

Another thing to try might be to start tweaking which scripts are 
actually loaded in local.bro: comment out all the @load statements in 
local.bro (add a '#' to the beginning of each line) and see what kind of 
an effect that has on memory utilization.  Assuming memory usage drops, 
then you can start slowly start removing the # characters / restarting 
bro to re-load scripts one at a time.  This is going to be pretty 
tedious, though.

A third thing to try might be doing a bit of sampling so that bro only 
sees some % of incoming packets.  From there, slowly start to bring 
traffic back to see how bro's memory utilization rises as traffic is 
restored.  Might want to define a few different levels (drop 90% of 
connections, drop 75% of connections, drop 50% of connections, drop 25% 
of connections) and just leave the drop rate at each step for a while 
(read: a few hours would probably be a good start) to see what happens 
to the memory utilization.  If memory utilization stays pretty constant 
at each level, and the overall memory pattern ends up looking like a 
staircase, it might be time to consider that RAM upgrade :)

*** Note that it's important that traffic be connection-sampled when 
trying the above: packet-sampling will lead to dropped packets in the 
middle of connections, which might not work quite as expected. Seth: how 
does connection sampling work for the packet filter framework?  I 
haven't ever really used it, but I think it can do that, right?  Also, 
any other ideas / things I might be missing here?

One other question: did adjusting those timeouts change the number of 
inactivity timers reported?  Probably not relevant to this issue, but 
just wondering if the change had any measurable effect.

-Gilbert

On 6/27/14, 11:29 AM, Jason Batchelor wrote:
> Thanks Gilbert!
> I think I am getting close to at least isolating the issue.
> I redefined some of the inactivity timeout values to something pregger 
> aggressive...
> redef tcp_inactivity_timeout = 15 sec;
> redef udp_inactivity_timeout = 5 sec;
> redef icmp_inactivity_timeout = 5 sec;
> After committing the changes and restarting I am still seeing the same 
> kind of slow memory consumption behavior.
> I checked the IO statistics you gave above and think this is where I 
> am getting backed up. Below is a brief escalation of just the http 
> logs themselves.
> grep 'http/Log::WRITER_ASCII' prof.log | awk 'NR % 10 == 1'
> 1403880551.747191   http/Log::WRITER_ASCII    in=40 out=11 pending=0/0 
> (#queue r/w: in=40/40 out=11/11)
> 1403880701.759282   http/Log::WRITER_ASCII    in=632 out=160 
> pending=0/0 (#queue r/w: in=632/632 out=160/160)
> 1403880851.764553   http/Log::WRITER_ASCII    in=1254 out=310 
> pending=0/0 (#queue r/w: in=1254/1254 out=310/310)
> 1403881001.794827   http/Log::WRITER_ASCII    in=1881 out=459 
> pending=0/0 (#queue r/w: in=1881/1881 out=459/459)
> 1403881151.907771   http/Log::WRITER_ASCII    in=2496 out=607 
> pending=0/0 (#queue r/w: in=2496/2496 out=607/607)
> 1403881302.133110   http/Log::WRITER_ASCII    in=3140 out=754 
> pending=0/0 (#queue r/w: in=3140/3140 out=754/754)
> 1403881452.684259   http/Log::WRITER_ASCII    in=3781 out=900 
> pending=0/0 (#queue r/w: in=3781/3781 out=900/900)
> 1403881611.446692   http/Log::WRITER_ASCII    in=4321 out=1000 
> pending=0/0 (#queue r/w: in=4321/4321 out=1000/1000)
> 1403881783.945970   http/Log::WRITER_ASCII    in=4816 out=1069 
> pending=0/0 (#queue r/w: in=4816/4816 out=1069/1069)
> 1403881991.154812   http/Log::WRITER_ASCII    in=5435 out=1105 
> pending=0/0 (#queue r/w: in=5435/5435 out=1105/1105)
> 1403882156.814938   http/Log::WRITER_ASCII    in=6066 out=1190 
> pending=0/0 (#queue r/w: in=6066/6066 out=1190/1190)
> 1403882336.215055   http/Log::WRITER_ASCII    in=6690 out=1267 
> pending=0/0 (#queue r/w: in=6690/6690 out=1267/1267)
> 1403882494.089058   http/Log::WRITER_ASCII    in=7350 out=1377 
> pending=0/0 (#queue r/w: in=7350/7350 out=1377/1377)
> If I am interpreting this correctly, I am far exceeding my ability to 
> write out logs as time goes on, resulting in a backup of that data in 
> memory presumably. The same kind of behavior is seen in other log 
> types as well.
> Am I interpreting this correctly? If so the solution seems to be I 
> need faster drives and/or more memory :)
>
>
> On Thu, Jun 26, 2014 at 5:35 PM, Gilbert Clark <gc355804 at ohio.edu 
> <mailto:gc355804 at ohio.edu>> wrote:
>
>     Hi:
>
>     I believe this particular timer is a general timer used to track
>     inactivity for all protocols (but someone can correct me if I'm
>     wrong :).  Maybe try tuning the following:
>
>     const tcp_inactivity_timeout = 5 min &redef;
>     const udp_inactivity_timeout = 1 min &redef;
>     const icmp_inactivity_timeout = 1 min &redef;
>
>     Reference:
>     http://www.notary.icsi.berkeley.edu/sphinx-git/scripts/base/init-bare.bro.html#id-udp_inactivity_timeout
>
>     Also, I believe it's possible to set timeouts per-connection based
>     on properties of the established connections.  For an example of
>     doing this / how this might be useful, take a look at:
>
>     https://bro.org/sphinx/scripts/base/protocols/conn/inactivity.bro.html
>
>     Re: interpreting prof.log output -- a few notes from my experience:
>
>     There should be lines that include the number of connections
>     currently active for each major protocol type, e.g:
>
>     Conns: tcp=1/130 udp=1/70 icmp=0/0
>
>     Syntax here is: tcp=<current active connections>/<maximum number
>     of active connections ever observed> udp=<current active
>     connections>/<maximum number of active connections ever observed>
>     icmp=<current active connections>/<maximum number of active
>     connections ever observed>
>
>     The line following the above includes more detailed connection
>     overhead information:
>
>     Conns: total=6528 current=2/2 ext=0 mem=9312K avg=4656.0 table=24K
>     connvals=6K
>
>     A few notes about fields that might be useful there:
>
>     * total=total number of connections (aggregate: not just at this
>     particular moment)
>     * current=X/Y:  X and Y are two counts that will usually differ to
>     some extent, but both count the number of connections observed
>         - X: the number of active connections in total  (not
>     necessarily all of which are kept in state tables)
>         - Y: the number of connections stored in bro's state tables
>     (tcp + udp + icmp) at this moment in time
>     * avg=average memory use (in bytes) per active connection
>     * table=total amount of memory used by the state tables (tcp + udp
>     + icmp)
>
>     'avg' and 'table' are only recorded occasionally because computing
>     these values can be expensive.  When that "Global_sizes ..."
>     output is included in a log entry, these numbers will be
>     accurate.  Otherwise, they will be 0.
>
>     For an idea of the overhead associated with the Timer objects
>     themselves (read: the overhead for the timers isn't included in
>     the overhead computed for the connection state), take a look at
>     the line that looks something like:
>
>     Timers: current=19 max=19 mem=1K lag=0.00s
>
>     *current=number of timers currently active in total
>     *max=maximum number of timers ever active at once
>     *mem=total memory consumed by all of the currently active timers
>     (usually pretty small compared to other things, though)
>
>     Also, one other note: under 'Threads', there's a bunch of lines
>     that look something like:
>
>     http/Log::WRITER_ASCII    in=11318 out=10882 pending=0/0 (#queue
>     r/w: in=11318/11318 out=10882/10882)
>     ssl/Log::WRITER_ASCII     in=10931 out=10878 pending=0/0 (#queue
>     r/w: in=10931/10931 out=10878/10878)
>     files/Log::WRITER_ASCII   in=10989 out=10792 pending=0/0 (#queue
>     r/w: in=10989/10989 out=10792/10792)
>     dhcp/Log::WRITER_ASCII    in=1031 out=1029 pending=0/0 (#queue
>     r/w: in=1031/1031 out=1029/1029)
>
>     Generally, pending X/Y will describe how much memory is currently
>     being consumed (relatively speaking) by messages waiting to be
>     written to a log file / that have been read from that input source
>     but not yet processed by bro.
>
>     A pending X/Y that grows steadily over time is an indication that
>     bro could eventually run out of room to store outstanding log /
>     input framework messages, and that these messages could eventually
>     come to consume so much memory that the worker would thrash the
>     machine into sweet digital oblivion.
>
>     Hope something in there is useful,
>     Gilbert
>
>
>     On 6/26/14, 2:26 PM, Jason Batchelor wrote:
>>     Small follow up to this as well since it may be relevant. I
>>     notice the timers for stale connections seems to increase in
>>     paralel with memory...
>>     grep 'ConnectionInactivityTimer' prof.log | awk 'NR % 10 == 1'
>>     1403802069.314888 ConnectionInactivityTimer = 5844
>>     1403802219.315759 ConnectionInactivityTimer = 21747
>>     1403802369.316387 ConnectionInactivityTimer = 32275
>>     1403802519.317613 ConnectionInactivityTimer = 32716
>>     1403802669.318303 ConnectionInactivityTimer = 32597
>>     1403802819.319193 ConnectionInactivityTimer = 34207
>>     1403802969.320204 ConnectionInactivityTimer = 39176
>>     1403803119.321978 ConnectionInactivityTimer = 40394
>>     1403803269.323058 ConnectionInactivityTimer = 38631
>>     1403803419.323688 ConnectionInactivityTimer = 35847
>>     1403803569.324716 ConnectionInactivityTimer = 34432
>>     1403803719.325888 ConnectionInactivityTimer = 34591
>>     1403803869.326713 ConnectionInactivityTimer = 34716
>>     1403804019.327664 ConnectionInactivityTimer = 35361
>>     1403804169.329254 ConnectionInactivityTimer = 35915
>>     1403804319.330507 ConnectionInactivityTimer = 34994
>>     1403804469.331842 ConnectionInactivityTimer = 33212
>>     1403804619.332236 ConnectionInactivityTimer = 32290
>>     1403804769.332993 ConnectionInactivityTimer = 32513
>>     1403804919.333717 ConnectionInactivityTimer = 32592
>>     1403805069.334477 ConnectionInactivityTimer = 32388
>>     1403805219.334875 ConnectionInactivityTimer = 32932
>>     1403805369.335753 ConnectionInactivityTimer = 31771
>>     1403805519.337054 ConnectionInactivityTimer = 28749
>>     1403805669.337563 ConnectionInactivityTimer = 26509
>>     1403805819.339240 ConnectionInactivityTimer = 26654
>>     1403805969.340812 ConnectionInactivityTimer = 26297
>>     1403806119.341841 ConnectionInactivityTimer = 25362
>>     1403806269.344342 ConnectionInactivityTimer = 24435
>>     1403806419.345146 ConnectionInactivityTimer = 24954
>>     1403806569.346057 ConnectionInactivityTimer = 24088
>>     1403806719.347671 ConnectionInactivityTimer = 30207
>>     1403806869.349643 ConnectionInactivityTimer = 34276
>>     Notice the steady increase, then slight decrease, then steady
>>     increase again. Is there a way to control these settings for
>>     performance testing purposes?
>>     I know while I was tuning Suricata, I needed to be mindful of
>>     connection timeouts and due to the volume of flows I am getting I
>>     needed to be pretty aggressive.
>>     Thanks,
>>     Jason
>>
>>
>>     On Thu, Jun 26, 2014 at 12:29 PM, Jason Batchelor
>>     <jxbatchelor at gmail.com <mailto:jxbatchelor at gmail.com>> wrote:
>>
>>         Thanks Seth:
>>
>>         I'm not sure I have a license for an experianced bro memory
>>         debugger, however I will document what I've done here for
>>         folks in hopes it proves useful!
>>
>>         I've enabled profiling by adding the following.
>>
>>         Vim /opt/bro/share/bro/site/local.bro
>>         @load misc/profiling
>>
>>         Then enforced the changes...
>>
>>         broctl stop
>>         broctl install
>>         broctl start
>>
>>         At the moment I have 46308184k used  3067820k free memory.
>>
>>         In /var/opt/bro/spool/worker-1-1, prof.log content is
>>         captured as you mentioned (and likewise for all nodes).
>>
>>         Earlier you wrote:
>>
>>         >  Every so often in there will be an indication of the
>>         largest global variables
>>
>>         Is this what you mean (taken from one worker)....?
>>
>>         1403803224.322453 Global_sizes > 100k: 0K
>>         1403803224.322453                Known::known_services = 469K
>>         (3130/3130 entries)
>>         1403803224.322453  Cluster::manager2worker_events = 137K
>>         1403803224.322453                Weird::weird_ignore = 31492K
>>         (146569/146569 entries)
>>         1403803224.322453                Known::certs = 58K (310/310
>>         entries)
>>         1403803224.322453  SumStats::threshold_tracker = 668K (4/2916
>>         entries)
>>         1403803224.322453                FTP::ftp_data_expected =
>>         181K (46/46 entries)
>>         1403803224.322453                Notice::suppressing = 595K
>>         (2243/2243 entries)
>>         1403803224.322453  Communication::connected_peers = 156K (2/2
>>         entries)
>>         1403803224.322453  SumStats::sending_results = 8028K (3/5545
>>         entries)
>>         1403803224.322453                Software::tracked = 33477K
>>         (12424/31111 entries)
>>         1403803224.322453                FTP::cmd_reply_code = 48K
>>         (325/325 entries)
>>         1403803224.322453                SumStats::result_store =
>>         27962K (5/19978 entries)
>>         1403803224.322453                SSL::cipher_desc = 97K
>>         (356/356 entries)
>>         1403803224.322453                RADIUS::attr_types = 44K
>>         (169/169 entries)
>>         1403803224.322453                Weird::actions = 35K
>>         (163/163 entries)
>>         1403803224.322453                Known::known_hosts = 3221K
>>         (21773/21773 <tel:%2821773%2F21773> entries)
>>         1403803224.322453                Weird::did_log = 54K
>>         (287/287 entries)
>>         1403803224.322453  SSL::recently_validated_certs = 8667K
>>         (24752/24752 entries)
>>         1403803224.322453                Communication::nodes = 188K
>>         (4/4 entries)
>>         1403803224.322453                SSL::root_certs = 204K
>>         (144/144 entries)
>>         1403803224.322453 Global_sizes total: 116727K
>>         1403803224.322453 Total number of table entries: 213548/260715
>>         1403803239.322685 ------------------------
>>         1403803239.322685 Memory: total=1185296K total_adj=1137108K
>>         malloced: 1144576K
>>
>>         Any other pointers on how to interpret this data?
>>
>>         FWIW, here are some additional statistics from the worker
>>         prof.log...
>>
>>         grep  "Memory: " prof.log | awk 'NR % 10 == 1'
>>         0.000000 Memory: total=48188K total_adj=0K malloced: 47965K
>>         1403802189.315606 Memory: total=614476K total_adj=566288K
>>         malloced: 614022K
>>         1403802339.316381 Memory: total=938380K total_adj=890192K
>>         malloced: 938275K
>>         1403802489.317426 Memory: total=1006168K total_adj=957980K
>>         malloced: 1003385K
>>         1403802639.318199 Memory: total=1041288K total_adj=993100K
>>         malloced: 1035422K
>>         1403802789.319107 Memory: total=1063544K total_adj=1015356K
>>         malloced: 1058229K
>>         1403802939.320170 Memory: total=1140652K total_adj=1092464K
>>         malloced: 1139608K
>>         1403803089.321327 Memory: total=1184540K total_adj=1136352K
>>         malloced: 1179411K
>>         1403803239.322685 Memory: total=1185296K total_adj=1137108K
>>         malloced: 1144576K
>>         1403803389.323680 Memory: total=1185296K total_adj=1137108K
>>         malloced: 1118961K
>>         1403803539.324677 Memory: total=1185296K total_adj=1137108K
>>         malloced: 1092719K
>>         1403803689.325763 Memory: total=1185296K total_adj=1137108K
>>         malloced: 1091447K
>>
>>
>>         On Thu, Jun 26, 2014 at 11:49 AM, Seth Hall <seth at icir.org
>>         <mailto:seth at icir.org>> wrote:
>>
>>
>>             On Jun 26, 2014, at 12:43 PM, Jason Batchelor
>>             <jxbatchelor at gmail.com <mailto:jxbatchelor at gmail.com>> wrote:
>>
>>             > > Bro typically does consume quite a bit of memory and
>>             you're a bit tight on memory for the number of workers
>>             you're running.
>>             > Curious what would you recommend for just bro itself?
>>             Double, triple this?
>>
>>             It seems like most people just put 128G of memory in Bro
>>             boxes now because the cost just isn't really worth going
>>             any lower if there's a remote possibility you might use it.
>>
>>             > I will definately take a look, thanks for the info!
>>
>>             Feel free to ask again if you're having trouble.  We
>>             really should write up some debugging documentation for
>>             this process sometime.  Anyone with experience doing this
>>             memory debugging activity up for it?  Doesn't have to be
>>             anything fancy, just the steps and various things to look
>>             at to figure out what exactly is happening.
>>
>>               .Seth
>>
>>             --
>>             Seth Hall
>>             International Computer Science Institute
>>             (Bro) because everyone has a network
>>             http://www.bro.org/
>>
>>
>>
>
>
>     _______________________________________________
>     Bro mailing list
>     bro at bro-ids.org <mailto:bro at bro-ids.org>
>     http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140627/0375b4be/attachment.html 

From donaldson8 at llnl.gov  Fri Jun 27 14:49:31 2014
From: donaldson8 at llnl.gov (Donaldson, John)
Date: Fri, 27 Jun 2014 21:49:31 +0000
Subject: [Bro] Logging to Accumolo?
Message-ID: <C623401B7419BD4E8A983814FEFA02F64256C900@PRDEXMBX-07.the-lab.llnl.gov>

I know that there is support for logging into ElasticSearch, but has anyone looked into sending logs into Accumulo?

John Donaldson



From grutz at jingojango.net  Fri Jun 27 15:11:04 2014
From: grutz at jingojango.net (Kurt Grutzmacher)
Date: Fri, 27 Jun 2014 15:11:04 -0700
Subject: [Bro] Logging to Accumolo?
In-Reply-To: <C623401B7419BD4E8A983814FEFA02F64256C900@PRDEXMBX-07.the-lab.llnl.gov>
References: <C623401B7419BD4E8A983814FEFA02F64256C900@PRDEXMBX-07.the-lab.llnl.gov>
Message-ID: <CAMDnkSm=bWOUygdAvVw7VzepB8ryOAFNLfzz5A_Zj9hMbMEoig@mail.gmail.com>

Logging is going through an update so we'll have to see what comes out at
the end.

I have code working for Kafka that was fairly easy to integrate in 2.2
(haven't tried 2.3 yet). Should be fairly simple to integrate others.
 On Jun 27, 2014 6:04 PM, "Donaldson, John" <donaldson8 at llnl.gov> wrote:

> I know that there is support for logging into ElasticSearch, but has
> anyone looked into sending logs into Accumulo?
>
> John Donaldson
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140627/8e95dd86/attachment.html 

From ben.bt.wood at gmail.com  Mon Jun 30 06:40:44 2014
From: ben.bt.wood at gmail.com (ben.bt.wood at gmail.com)
Date: Mon, 30 Jun 2014 09:40:44 -0400
Subject: [Bro] Logging to Accumolo?
Message-ID: <53b168f7.529ce00a.139f.7e6c@mx.google.com>

You may try experimenting with Flume. It's part of that larger group of Hadoop related Apache projects, and included in Cloudera's distro.

It's quite a capable log management tool, and can provide a lot of flexibility.

http://flume.apache.org/



-----Original Message-----
From: "Kurt Grutzmacher" <grutz at jingojango.net>
Sent: ?6/?27/?2014 6:29 PM
To: "Donaldson, John" <donaldson8 at llnl.gov>
Cc: "bro at bro.org List" <bro at bro.org>
Subject: Re: [Bro] Logging to Accumolo?

Logging is going through an update so we'll have to see what comes out at the end. 
I have code working for Kafka that was fairly easy to integrate in 2.2 (haven't tried 2.3 yet). Should be fairly simple to integrate others.

On Jun 27, 2014 6:04 PM, "Donaldson, John" <donaldson8 at llnl.gov> wrote:

I know that there is support for logging into ElasticSearch, but has anyone looked into sending logs into Accumulo?

John Donaldson

_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140630/e7e81c6a/attachment.html 

From grutz at jingojango.net  Mon Jun 30 08:24:09 2014
From: grutz at jingojango.net (Kurt Grutzmacher)
Date: Mon, 30 Jun 2014 08:24:09 -0700
Subject: [Bro] Logging to Accumolo?
In-Reply-To: <53b168f7.529ce00a.139f.7e6c@mx.google.com>
References: <53b168f7.529ce00a.139f.7e6c@mx.google.com>
Message-ID: <CAMDnkSndwNjw+2ZrV4TkwD=JXTwGq7osS0zpUs2efq6S3MtZOA@mail.gmail.com>

Hi Ben,

We do use Flume for applications we can't extend with Kafka. There are
pluses and minuses for either route but with Bro we wanted to queue
directly from the output process code to avoid the additional IO of disk
read/writes. Adding another processor like Flume also increases the
complexity and potential failure points.

You do give up some reliability of message delivery but it's an acceptable
risk for us at the moment. A good Kafka/Zookeeper environment is
over-engineered anyway and Bro already is a single point of failure.




--
 Kurt Grutzmacher -=- grutz at jingojango.net


On Mon, Jun 30, 2014 at 6:40 AM, <ben.bt.wood at gmail.com> wrote:

>  You may try experimenting with Flume. It's part of that larger group of
> Hadoop related Apache projects, and included in Cloudera's distro.
>
> It's quite a capable log management tool, and can provide a lot of
> flexibility.
>
> http://flume.apache.org/
>
>  ------------------------------
> From: Kurt Grutzmacher <grutz at jingojango.net>
> Sent: ?6/?27/?2014 6:29 PM
> To: Donaldson, John <donaldson8 at llnl.gov>
> Cc: bro at bro.org List <bro at bro.org>
> Subject: Re: [Bro] Logging to Accumolo?
>
> Logging is going through an update so we'll have to see what comes out at
> the end.
>
> I have code working for Kafka that was fairly easy to integrate in 2.2
> (haven't tried 2.3 yet). Should be fairly simple to integrate others.
> On Jun 27, 2014 6:04 PM, "Donaldson, John" <donaldson8 at llnl.gov> wrote:
>
>> I know that there is support for logging into ElasticSearch, but has
>> anyone looked into sending logs into Accumulo?
>>
>> John Donaldson
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140630/ce5259b0/attachment.html 

From jxbatchelor at gmail.com  Mon Jun 30 13:32:32 2014
From: jxbatchelor at gmail.com (Jason Batchelor)
Date: Mon, 30 Jun 2014 15:32:32 -0500
Subject: [Bro] Memory Consumption
In-Reply-To: <53ADD670.3000401@ohio.edu>
References: <CANWz5X-zt+_EFD+J5pp-dm2acp7iJ2cgcPUJYdycrr=YwZ2DgA@mail.gmail.com>
	<AC6C73A5-AE14-48AE-88B2-2110D11E833C@icir.org>
	<CANWz5X8MZ5-BCR2YJ50dcqSWUjRs=USfx+voVB6YbZvyvJcsUg@mail.gmail.com>
	<86D9FB05-B648-4646-A7B7-7F08E663E67F@icir.org>
	<CANWz5X-2g9jLKLTOjTLHe4m=kLe14oDrgiMjUHvfk=M9TiZiTg@mail.gmail.com>
	<CANWz5X9wwy30aQ16=uvLPOhoGCc6GN2da2+-_0iqXGadJOKxww@mail.gmail.com>
	<53ACA02A.70709@ohio.edu>
	<CANWz5X_ucqGgLrNbtBVzi-LFWpn5U=0Fm87324=9=z3COWne=w@mail.gmail.com>
	<53ADD670.3000401@ohio.edu>
Message-ID: <CANWz5X-bSnj7id=PUQJ4jQdwHHW8MM4PXz8XzHiVGqPNy=EgZg@mail.gmail.com>

Thanks again Gilbert.

I put in an aggressive bpf filter that eliminates 90% of traffic and memory
flat lined so I do not believe there to be a memory leak.

I then removed the old restrictive filter and replaced it with a more broad
one that does nix out mulicast and broadcast traffic. I did notice the
ConnectionInactivityTimer to be much more stable actually.

grep 'ConnectionInactivityTimer' prof.log | awk 'NR % 10 == 1'
1404155766.423613         ConnectionInactivityTimer = 4412
1404155916.424489         ConnectionInactivityTimer = 4776
1404156066.425171         ConnectionInactivityTimer = 4627
1404156216.426120         ConnectionInactivityTimer = 4974
1404156366.426889         ConnectionInactivityTimer = 4784
1404156516.428065         ConnectionInactivityTimer = 4750
1404156666.429125         ConnectionInactivityTimer = 4687
1404156816.431119         ConnectionInactivityTimer = 5006
1404156966.431837         ConnectionInactivityTimer = 4830

So far with multicast and broadcast traffic being filtered out, I have
noticed the memory to gradually increase but at a much slower rate. It has
not come close to using all resources yet but I will check again tomorrow.
It would seem that the perscription for more RAM needs to be written based
on the observables collected so far. Many thanks for the helpful tips.



On Fri, Jun 27, 2014 at 3:39 PM, Gilbert Clark <gc355804 at ohio.edu> wrote:

>  Hi Jason:
>
> It's only when *pending* begins to grow that there's a large queue of
> messages waiting to be written (as far as I know).  In this case, pending
> stays at 0/0 for each of those log updates, so I don't think that's an
> issue (for this log, at least :)
>
> One possibility here is going to be that bro is actually leaking memory
> somewhere, and you're lucky enough to have found a bug :)  Usually this is
> going to be identified by a steady rise in memory usage over time with a
> relatively constant traffic volume / types.  One thing to try (if it's
> something you feel comfortable with) might be to pull down the current copy
> of bro in master, building that, and pushing this out to the nodes to see
> what kind of an effect that has (if anything).
>
> Another thing to try might be to start tweaking which scripts are actually
> loaded in local.bro: comment out all the @load statements in local.bro (add
> a '#' to the beginning of each line) and see what kind of an effect that
> has on memory utilization.  Assuming memory usage drops, then you can start
> slowly start removing the # characters / restarting bro to re-load scripts
> one at a time.  This is going to be pretty tedious, though.
>
> A third thing to try might be doing a bit of sampling so that bro only
> sees some % of incoming packets.  From there, slowly start to bring traffic
> back to see how bro's memory utilization rises as traffic is restored.
> Might want to define a few different levels (drop 90% of connections, drop
> 75% of connections, drop 50% of connections, drop 25% of connections) and
> just leave the drop rate at each step for a while (read: a few hours would
> probably be a good start) to see what happens to the memory utilization.
> If memory utilization stays pretty constant at each level, and the overall
> memory pattern ends up looking like a staircase, it might be time to
> consider that RAM upgrade :)
>
> *** Note that it's important that traffic be connection-sampled when
> trying the above: packet-sampling will lead to dropped packets in the
> middle of connections, which might not work quite as expected.  Seth: how
> does connection sampling work for the packet filter framework?  I haven't
> ever really used it, but I think it can do that, right?  Also, any other
> ideas / things I might be missing here?
>
> One other question: did adjusting those timeouts change the number of
> inactivity timers reported?  Probably not relevant to this issue, but just
> wondering if the change had any measurable effect.
>
> -Gilbert
>
>
> On 6/27/14, 11:29 AM, Jason Batchelor wrote:
>
>  Thanks Gilbert!
>
> I think I am getting close to at least isolating the issue.
>
> I redefined some of the inactivity timeout values to something pregger
> aggressive...
> redef tcp_inactivity_timeout = 15 sec;
> redef udp_inactivity_timeout = 5 sec;
> redef icmp_inactivity_timeout = 5 sec;
>
> After committing the changes and restarting I am still seeing the same
> kind of slow memory consumption behavior.
>
> I checked the IO statistics you gave above and think this is where I am
> getting backed up. Below is a brief escalation of just the http logs
> themselves.
>
> grep 'http/Log::WRITER_ASCII' prof.log | awk 'NR % 10 == 1'
> 1403880551.747191   http/Log::WRITER_ASCII    in=40 out=11 pending=0/0
> (#queue r/w: in=40/40 out=11/11)
> 1403880701.759282   http/Log::WRITER_ASCII    in=632 out=160 pending=0/0
> (#queue r/w: in=632/632 out=160/160)
> 1403880851.764553   http/Log::WRITER_ASCII    in=1254 out=310 pending=0/0
> (#queue r/w: in=1254/1254 out=310/310)
> 1403881001.794827   http/Log::WRITER_ASCII    in=1881 out=459 pending=0/0
> (#queue r/w: in=1881/1881 out=459/459)
> 1403881151.907771   http/Log::WRITER_ASCII    in=2496 out=607 pending=0/0
> (#queue r/w: in=2496/2496 out=607/607)
> 1403881302.133110   http/Log::WRITER_ASCII    in=3140 out=754 pending=0/0
> (#queue r/w: in=3140/3140 out=754/754)
> 1403881452.684259   http/Log::WRITER_ASCII    in=3781 out=900 pending=0/0
> (#queue r/w: in=3781/3781 out=900/900)
> 1403881611.446692   http/Log::WRITER_ASCII    in=4321 out=1000 pending=0/0
> (#queue r/w: in=4321/4321 out=1000/1000)
> 1403881783.945970   http/Log::WRITER_ASCII    in=4816 out=1069 pending=0/0
> (#queue r/w: in=4816/4816 out=1069/1069)
> 1403881991.154812   http/Log::WRITER_ASCII    in=5435 out=1105 pending=0/0
> (#queue r/w: in=5435/5435 out=1105/1105)
> 1403882156.814938   http/Log::WRITER_ASCII    in=6066 out=1190 pending=0/0
> (#queue r/w: in=6066/6066 out=1190/1190)
> 1403882336.215055   http/Log::WRITER_ASCII    in=6690 out=1267 pending=0/0
> (#queue r/w: in=6690/6690 out=1267/1267)
> 1403882494.089058   http/Log::WRITER_ASCII    in=7350 out=1377 pending=0/0
> (#queue r/w: in=7350/7350 out=1377/1377)
>
> If I am interpreting this correctly, I am far exceeding my ability to
> write out logs as time goes on, resulting in a backup of that data in
> memory presumably. The same kind of behavior is seen in other log types as
> well.
>
> Am I interpreting this correctly? If so the solution seems to be I need
> faster drives and/or more memory :)
>
>
>
> On Thu, Jun 26, 2014 at 5:35 PM, Gilbert Clark <gc355804 at ohio.edu> wrote:
>
>>  Hi:
>>
>> I believe this particular timer is a general timer used to track
>> inactivity for all protocols (but someone can correct me if I'm wrong :).
>> Maybe try tuning the following:
>>
>> const tcp_inactivity_timeout = 5 min &redef;
>> const udp_inactivity_timeout = 1 min &redef;
>> const icmp_inactivity_timeout = 1 min &redef;
>>
>> Reference:
>> http://www.notary.icsi.berkeley.edu/sphinx-git/scripts/base/init-bare.bro.html#id-udp_inactivity_timeout
>>
>> Also, I believe it's possible to set timeouts per-connection based on
>> properties of the established connections.  For an example of doing this /
>> how this might be useful, take a look at:
>>
>> https://bro.org/sphinx/scripts/base/protocols/conn/inactivity.bro.html
>>
>> Re: interpreting prof.log output -- a few notes from my experience:
>>
>> There should be lines that include the number of connections currently
>> active for each major protocol type, e.g:
>>
>> Conns: tcp=1/130 udp=1/70 icmp=0/0
>>
>> Syntax here is: tcp=<current active connections>/<maximum number of
>> active connections ever observed> udp=<current active connections>/<maximum
>> number of active connections ever observed> icmp=<current active
>> connections>/<maximum number of active connections ever observed>
>>
>> The line following the above includes more detailed connection overhead
>> information:
>>
>> Conns: total=6528 current=2/2 ext=0 mem=9312K avg=4656.0 table=24K
>> connvals=6K
>>
>> A few notes about fields that might be useful there:
>>
>> * total=total number of connections (aggregate: not just at this
>> particular moment)
>> * current=X/Y:  X and Y are two counts that will usually differ to some
>> extent, but both count the number of connections observed
>>     - X: the number of active connections in total  (not necessarily all
>> of which are kept in state tables)
>>     - Y: the number of connections stored in bro's state tables (tcp +
>> udp + icmp) at this moment in time
>> * avg=average memory use (in bytes) per active connection
>> * table=total amount of memory used by the state tables (tcp + udp + icmp)
>>
>> 'avg' and 'table' are only recorded occasionally because computing these
>> values can be expensive.  When that "Global_sizes ..." output is
>> included in a log entry, these numbers will be accurate.  Otherwise, they
>> will be 0.
>>
>> For an idea of the overhead associated with the Timer objects themselves
>> (read: the overhead for the timers isn't included in the overhead computed
>> for the connection state), take a look at the line that looks something
>> like:
>>
>> Timers: current=19 max=19 mem=1K lag=0.00s
>>
>> *current=number of timers currently active in total
>> *max=maximum number of timers ever active at once
>> *mem=total memory consumed by all of the currently active timers (usually
>> pretty small compared to other things, though)
>>
>> Also, one other note: under 'Threads', there's a bunch of lines that look
>> something like:
>>
>> http/Log::WRITER_ASCII    in=11318 out=10882 pending=0/0 (#queue r/w:
>> in=11318/11318 out=10882/10882)
>> ssl/Log::WRITER_ASCII     in=10931 out=10878 pending=0/0 (#queue r/w:
>> in=10931/10931 out=10878/10878)
>> files/Log::WRITER_ASCII   in=10989 out=10792 pending=0/0 (#queue r/w:
>> in=10989/10989 out=10792/10792)
>> dhcp/Log::WRITER_ASCII    in=1031 out=1029 pending=0/0 (#queue r/w:
>> in=1031/1031 out=1029/1029)
>>
>> Generally, pending X/Y will describe how much memory is currently being
>> consumed (relatively speaking) by messages waiting to be written to a log
>> file / that have been read from that input source but not yet processed by
>> bro.
>>
>> A pending X/Y that grows steadily over time is an indication that bro
>> could eventually run out of room to store outstanding log / input framework
>> messages, and that these messages could eventually come to consume so much
>> memory that the worker would thrash the machine into sweet digital oblivion.
>>
>> Hope something in there is useful,
>> Gilbert
>>
>>
>> On 6/26/14, 2:26 PM, Jason Batchelor wrote:
>>
>>  Small follow up to this as well since it may be relevant. I notice the
>> timers for stale connections seems to increase in paralel with memory...
>>
>> grep 'ConnectionInactivityTimer' prof.log | awk 'NR % 10 == 1'
>> 1403802069.314888         ConnectionInactivityTimer = 5844
>> 1403802219.315759         ConnectionInactivityTimer = 21747
>> 1403802369.316387         ConnectionInactivityTimer = 32275
>> 1403802519.317613         ConnectionInactivityTimer = 32716
>> 1403802669.318303         ConnectionInactivityTimer = 32597
>> 1403802819.319193         ConnectionInactivityTimer = 34207
>> 1403802969.320204         ConnectionInactivityTimer = 39176
>> 1403803119.321978         ConnectionInactivityTimer = 40394
>> 1403803269.323058         ConnectionInactivityTimer = 38631
>> 1403803419.323688         ConnectionInactivityTimer = 35847
>> 1403803569.324716         ConnectionInactivityTimer = 34432
>> 1403803719.325888         ConnectionInactivityTimer = 34591
>> 1403803869.326713         ConnectionInactivityTimer = 34716
>> 1403804019.327664         ConnectionInactivityTimer = 35361
>> 1403804169.329254         ConnectionInactivityTimer = 35915
>> 1403804319.330507         ConnectionInactivityTimer = 34994
>> 1403804469.331842         ConnectionInactivityTimer = 33212
>> 1403804619.332236         ConnectionInactivityTimer = 32290
>> 1403804769.332993         ConnectionInactivityTimer = 32513
>> 1403804919.333717         ConnectionInactivityTimer = 32592
>> 1403805069.334477         ConnectionInactivityTimer = 32388
>> 1403805219.334875         ConnectionInactivityTimer = 32932
>> 1403805369.335753         ConnectionInactivityTimer = 31771
>> 1403805519.337054         ConnectionInactivityTimer = 28749
>> 1403805669.337563         ConnectionInactivityTimer = 26509
>> 1403805819.339240         ConnectionInactivityTimer = 26654
>> 1403805969.340812         ConnectionInactivityTimer = 26297
>> 1403806119.341841         ConnectionInactivityTimer = 25362
>> 1403806269.344342         ConnectionInactivityTimer = 24435
>> 1403806419.345146         ConnectionInactivityTimer = 24954
>> 1403806569.346057         ConnectionInactivityTimer = 24088
>> 1403806719.347671         ConnectionInactivityTimer = 30207
>> 1403806869.349643         ConnectionInactivityTimer = 34276
>>
>> Notice the steady increase, then slight decrease, then steady increase
>> again. Is there a way to control these settings for performance testing
>> purposes?
>>
>> I know while I was tuning Suricata, I needed to be mindful of connection
>> timeouts and due to the volume of flows I am getting I needed to be pretty
>> aggressive.
>>
>> Thanks,
>> Jason
>>
>>
>>
>> On Thu, Jun 26, 2014 at 12:29 PM, Jason Batchelor <jxbatchelor at gmail.com>
>> wrote:
>>
>>> Thanks Seth:
>>>
>>>  I'm not sure I have a license for an experianced bro memory debugger,
>>> however I will document what I've done here for folks in hopes it proves
>>> useful!
>>>
>>>  I've enabled profiling by adding the following.
>>>
>>>  Vim /opt/bro/share/bro/site/local.bro
>>> @load misc/profiling
>>>
>>>  Then enforced the changes...
>>>
>>>  broctl stop
>>> broctl install
>>> broctl start
>>>
>>>  At the moment I have 46308184k used  3067820k free memory.
>>>
>>>  In /var/opt/bro/spool/worker-1-1, prof.log content is captured as you
>>> mentioned (and likewise for all nodes).
>>>
>>>  Earlier you wrote:
>>>
>>>  >  Every so often in there will be an indication of the largest global
>>> variables
>>>
>>>  Is this what you mean (taken from one worker)....?
>>>
>>>  1403803224.322453 Global_sizes > 100k: 0K
>>> 1403803224.322453                Known::known_services = 469K (3130/3130
>>> entries)
>>> 1403803224.322453                Cluster::manager2worker_events = 137K
>>> 1403803224.322453                Weird::weird_ignore = 31492K
>>> (146569/146569 entries)
>>> 1403803224.322453                Known::certs = 58K (310/310 entries)
>>> 1403803224.322453                SumStats::threshold_tracker = 668K
>>> (4/2916 entries)
>>> 1403803224.322453                FTP::ftp_data_expected = 181K (46/46
>>> entries)
>>> 1403803224.322453                Notice::suppressing = 595K (2243/2243
>>> entries)
>>> 1403803224.322453                Communication::connected_peers = 156K
>>> (2/2 entries)
>>> 1403803224.322453                SumStats::sending_results = 8028K
>>> (3/5545 entries)
>>> 1403803224.322453                Software::tracked = 33477K (12424/31111
>>> entries)
>>> 1403803224.322453                FTP::cmd_reply_code = 48K (325/325
>>> entries)
>>> 1403803224.322453                SumStats::result_store = 27962K
>>> (5/19978 entries)
>>> 1403803224.322453                SSL::cipher_desc = 97K (356/356 entries)
>>> 1403803224.322453                RADIUS::attr_types = 44K (169/169
>>> entries)
>>> 1403803224.322453                Weird::actions = 35K (163/163 entries)
>>> 1403803224.322453                Known::known_hosts = 3221K (21773/21773
>>> entries)
>>> 1403803224.322453                Weird::did_log = 54K (287/287 entries)
>>> 1403803224.322453                SSL::recently_validated_certs = 8667K
>>> (24752/24752 entries)
>>> 1403803224.322453                Communication::nodes = 188K (4/4
>>> entries)
>>> 1403803224.322453                SSL::root_certs = 204K (144/144 entries)
>>> 1403803224.322453 Global_sizes total: 116727K
>>>  1403803224.322453 Total number of table entries: 213548/260715
>>> 1403803239.322685 ------------------------
>>> 1403803239.322685 Memory: total=1185296K total_adj=1137108K malloced:
>>> 1144576K
>>>
>>>  Any other pointers on how to interpret this data?
>>>
>>>  FWIW, here are some additional statistics from the worker prof.log...
>>>
>>>  grep  "Memory: " prof.log | awk 'NR % 10 == 1'
>>> 0.000000 Memory: total=48188K total_adj=0K malloced: 47965K
>>> 1403802189.315606 Memory: total=614476K total_adj=566288K malloced:
>>> 614022K
>>> 1403802339.316381 Memory: total=938380K total_adj=890192K malloced:
>>> 938275K
>>> 1403802489.317426 Memory: total=1006168K total_adj=957980K malloced:
>>> 1003385K
>>> 1403802639.318199 Memory: total=1041288K total_adj=993100K malloced:
>>> 1035422K
>>> 1403802789.319107 Memory: total=1063544K total_adj=1015356K malloced:
>>> 1058229K
>>> 1403802939.320170 Memory: total=1140652K total_adj=1092464K malloced:
>>> 1139608K
>>> 1403803089.321327 Memory: total=1184540K total_adj=1136352K malloced:
>>> 1179411K
>>> 1403803239.322685 Memory: total=1185296K total_adj=1137108K malloced:
>>> 1144576K
>>> 1403803389.323680 Memory: total=1185296K total_adj=1137108K malloced:
>>> 1118961K
>>> 1403803539.324677 Memory: total=1185296K total_adj=1137108K malloced:
>>> 1092719K
>>> 1403803689.325763 Memory: total=1185296K total_adj=1137108K malloced:
>>> 1091447K
>>>
>>>
>>> On Thu, Jun 26, 2014 at 11:49 AM, Seth Hall <seth at icir.org> wrote:
>>>
>>>>
>>>> On Jun 26, 2014, at 12:43 PM, Jason Batchelor <jxbatchelor at gmail.com>
>>>> wrote:
>>>>
>>>> > > Bro typically does consume quite a bit of memory and you're a bit
>>>> tight on memory for the number of workers you're running.
>>>> > Curious what would you recommend for just bro itself? Double, triple
>>>> this?
>>>>
>>>>  It seems like most people just put 128G of memory in Bro boxes now
>>>> because the cost just isn't really worth going any lower if there's a
>>>> remote possibility you might use it.
>>>>
>>>> > I will definately take a look, thanks for the info!
>>>>
>>>>  Feel free to ask again if you're having trouble.  We really should
>>>> write up some debugging documentation for this process sometime.  Anyone
>>>> with experience doing this memory debugging activity up for it?  Doesn't
>>>> have to be anything fancy, just the steps and various things to look at to
>>>> figure out what exactly is happening.
>>>>
>>>>   .Seth
>>>>
>>>> --
>>>> Seth Hall
>>>> International Computer Science Institute
>>>> (Bro) because everyone has a network
>>>> http://www.bro.org/
>>>>
>>>>
>>>
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140630/7fbdc151/attachment.html