From vallentin at icir.org  Thu Oct  1 02:36:29 2015
From: vallentin at icir.org (Matthias Vallentin)
Date: Thu, 1 Oct 2015 11:36:29 +0200
Subject: [Bro] Evaluating compression algorithms for Bro logs and PCAPs
Message-ID: <20151001093629.GO87241@ninja.local>

Recently I've heard about Google's new Brotli algorithm outperforming so
many others. I wanted to figure out whether it holds up to his words and
wrote a small benchmark [1]. This benchmark compares various algorithms
along space (compresion ratio) and time (throughput in bytes/sec). As
input I've taken both PCAPs and Bro logs.

Except for the tradeoff plot, I hope the figures speak for themselves.
The tradeoff figure visualizes the space-time spectrum of the
compression algorithm. It plots space savings [2] against throughput
during compression. The further a point sits in the top-right corner,
the more of a best-of-both-worlds algorithm it is. ZSTD appears to have
that property for my tested input.

Brotli compresses at a higher ratio, but is also much slower. So for
high-throughput applications that pump out logs, it may not be
well-suited if a minimal CPU footprint has priority. Even thought Brotli
would have been a nice name for the best algorithm for Bro logs, it only
ranks second after ZPAQ in terms of space savings. However, Brotli is
3 orders of magnitude faster than ZPAQ during decompression, rendering
it a solid choice when you have enough cycles available during
compression.

    Matthias

[1] https://github.com/mavam/compbench
[2] https://en.wikipedia.org/wiki/Data_compression_ratio

From blackhole.em at gmail.com  Fri Oct  2 08:17:43 2015
From: blackhole.em at gmail.com (Joe Blow)
Date: Fri, 2 Oct 2015 11:17:43 -0400
Subject: [Bro] Memory Issue with Bro
In-Reply-To: <CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
References: <e9c88f86c0b144d0a33afafa324f9555@ocex03.SolarFlarecom.com>
	<7E1F7811-8496-4F0F-A2BD-7D2FB419C6D2@illinois.edu>
	<69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com>
	<454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu>
	<7689d3f6ddb6454d855e6fd3a3b8ea7b@ocex03.SolarFlarecom.com>
	<3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org>
	<CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
Message-ID: <CAMCSCoV8RUFm3DEnCO3vHwbCs9wE5L8YbeVJWdWJ4RrbHAFcMw@mail.gmail.com>

Any update on this?  If not I think i'm going to submit a support case to
Solarflare.

Cheers,

JB

On Wed, Sep 30, 2015 at 12:55 PM, Joe Blow <blackhole.em at gmail.com> wrote:

> I'm super interested in this thread, as I believe i'm experiencing the
> same memory leak, using the solarflare cards.
> i'm running a similar setup, with 20 workers and lots of traffic, but i'm
> having to bounce the entire NIC once Bro goes haywire.  Bro doesn't take
> too long before it's wiped the whole box out of memory (all 192GB).
>
> Please let me know how to troubleshooting goes.  I'm happy to provide logs.
>
> Cheers,
>
> JB
>
> On Wed, Sep 30, 2015 at 9:48 AM, Seth Hall <seth at icir.org> wrote:
>
>>
>> > On Sep 29, 2015, at 7:15 PM, Sampada Kathare <skathare at solarflare.com>
>> wrote:
>> >
>> > What is the meaning of this error? ?
>> >
>> > SolarCapture session=4382/1 log=/var/tmp/solar_capture_root_4382/1
>> > ERROR: errno=-114 from core/sc_ef_vi.c:1323 in sc_ef_vi_alloc_ts():
>> > ERROR: sc_ef_vi_alloc_ts: ef_vi_alloc_from_pd failed (-114)
>>
>> That appears to be a problem with your own NIC (I?m assuming...
>> solarflare error messages, solarflare.com email address).
>>
>>   .Seth
>>
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro.org/
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151002/2d38815b/attachment.html 

From jazoff at illinois.edu  Fri Oct  2 09:21:59 2015
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Fri, 2 Oct 2015 16:21:59 +0000
Subject: [Bro] Memory Issue with Bro
In-Reply-To: <CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
References: <e9c88f86c0b144d0a33afafa324f9555@ocex03.SolarFlarecom.com>
	<7E1F7811-8496-4F0F-A2BD-7D2FB419C6D2@illinois.edu>
	<69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com>
	<454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu>
	<7689d3f6ddb6454d855e6fd3a3b8ea7b@ocex03.SolarFlarecom.com>
	<3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org>
	<CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
Message-ID: <DE5BED9E-D710-4763-A978-423D917509A5@illinois.edu>


> On Sep 30, 2015, at 11:55 AM, Joe Blow <blackhole.em at gmail.com> wrote:
> 
> I'm super interested in this thread, as I believe i'm experiencing the same memory leak, using the solarflare cards.
> i'm running a similar setup, with 20 workers and lots of traffic, but i'm having to bounce the entire NIC once Bro goes haywire.  Bro doesn't take too long before it's wiped the whole box out of memory (all 192GB).
> 
> Please let me know how to troubleshooting goes.  I'm happy to provide logs.
> 
> Cheers,
> 
> JB

Memory leaks are tricky.  It is important to make a distinction about what component is using a lot of memory:

1) the workers - analyzer issues and leaks in general would show up here.
2) the proxies - communication related
3) the manager - child - if the manager is overloaded the child will buffer log data
4) the manager - parent - if a logging destination is overloaded the parent will buffer log writes

If your manager processes are using a lot of ram, that doesn't have anything to do with the capture library in use.

-- 
- Justin Azoff



From theflakes at gmail.com  Fri Oct  2 10:52:35 2015
From: theflakes at gmail.com (Brian Kellogg)
Date: Fri, 2 Oct 2015 13:52:35 -0400
Subject: [Bro] DNS behavior alerting
Message-ID: <CAOAzwcdMT-w32e9T4pui3JZV81Uz4q8GDvxvqxeKzJm5VhjOWQ@mail.gmail.com>

I started a Bro script a while ago that I haven't had time to develop much
beyond the starter framework.  The script is meant to do the below.  I
started working on it again but I'd welcome any help/feedback anyone would
be willing to offer.  It does try to do some basic DNS tunneling detection
but it needs more intelligence built into it.  For DNS tunneling the script
looks at the query size and the return message size and then uses sumstats
to alert on any host that crosses a specified threshold of supicious DNS
reqs/msgs seen.

I have seen that there are a lot of services out there conducting large
hostname queries which creates some FPs.

# Raises notices for odd or suspicious DNS traffic
#   - Detects DNS on non-standard ports
#   - Attempts to detect DNS tunneling
#       - intelligence for different query types --- TO DO
#       - statistical analysis' --- TO DO
#   - Detect DNS responses with interesting IPs --- TO DO

*Script on GitHub:*
https://github.com/theflakes/bro-scripts/blob/master/2.4-scripts/dns-bad_behavior.bro



-Brian Kellogg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151002/400ce842/attachment.html 

From blackhole.em at gmail.com  Fri Oct  2 10:55:29 2015
From: blackhole.em at gmail.com (Joe Blow)
Date: Fri, 2 Oct 2015 13:55:29 -0400
Subject: [Bro] Memory Issue with Bro
In-Reply-To: <DE5BED9E-D710-4763-A978-423D917509A5@illinois.edu>
References: <e9c88f86c0b144d0a33afafa324f9555@ocex03.SolarFlarecom.com>
	<7E1F7811-8496-4F0F-A2BD-7D2FB419C6D2@illinois.edu>
	<69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com>
	<454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu>
	<7689d3f6ddb6454d855e6fd3a3b8ea7b@ocex03.SolarFlarecom.com>
	<3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org>
	<CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
	<DE5BED9E-D710-4763-A978-423D917509A5@illinois.edu>
Message-ID: <CAMCSCoW6AisyP479iHN5pms+AcHPmFUAEdUvsJvhUPoRTNxD5w@mail.gmail.com>

It's my manager processes using tons of memory...

How would you suggest debugging the manager processes?

Cheers,

JB

On Fri, Oct 2, 2015 at 12:21 PM, Azoff, Justin S <jazoff at illinois.edu>
wrote:

>
> > On Sep 30, 2015, at 11:55 AM, Joe Blow <blackhole.em at gmail.com> wrote:
> >
> > I'm super interested in this thread, as I believe i'm experiencing the
> same memory leak, using the solarflare cards.
> > i'm running a similar setup, with 20 workers and lots of traffic, but
> i'm having to bounce the entire NIC once Bro goes haywire.  Bro doesn't
> take too long before it's wiped the whole box out of memory (all 192GB).
> >
> > Please let me know how to troubleshooting goes.  I'm happy to provide
> logs.
> >
> > Cheers,
> >
> > JB
>
> Memory leaks are tricky.  It is important to make a distinction about what
> component is using a lot of memory:
>
> 1) the workers - analyzer issues and leaks in general would show up here.
> 2) the proxies - communication related
> 3) the manager - child - if the manager is overloaded the child will
> buffer log data
> 4) the manager - parent - if a logging destination is overloaded the
> parent will buffer log writes
>
> If your manager processes are using a lot of ram, that doesn't have
> anything to do with the capture library in use.
>
> --
> - Justin Azoff
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151002/3d30ffed/attachment.html 

From anthony.kasza at gmail.com  Fri Oct  2 12:04:00 2015
From: anthony.kasza at gmail.com (anthony kasza)
Date: Fri, 2 Oct 2015 12:04:00 -0700
Subject: [Bro] DNS behavior alerting
In-Reply-To: <CAOAzwcdMT-w32e9T4pui3JZV81Uz4q8GDvxvqxeKzJm5VhjOWQ@mail.gmail.com>
References: <CAOAzwcdMT-w32e9T4pui3JZV81Uz4q8GDvxvqxeKzJm5VhjOWQ@mail.gmail.com>
Message-ID: <CAEZw2bz9sw738DnrTDV5cuUH40xrbGOVAop3f8LN4_GZzzK3QQ@mail.gmail.com>

For DNS tunneling detection, look for long qnames with few labels, very low
TTLs, odd qclasses and types (like null), and response rr's with high
ratios of ASCII to non ASCII bytes.

You can also look for bursty queries from a single orig_h for a single
qname (likely with many different subdomains).

Remember that both the client software issuing queries and the name server
software issuing responses need to know how to decode tunneling queries.
Check out the Iodine protocol specification for more info.

-AK
On Oct 2, 2015 10:54 AM, "Brian Kellogg" <theflakes at gmail.com> wrote:

> I started a Bro script a while ago that I haven't had time to develop much
> beyond the starter framework.  The script is meant to do the below.  I
> started working on it again but I'd welcome any help/feedback anyone would
> be willing to offer.  It does try to do some basic DNS tunneling detection
> but it needs more intelligence built into it.  For DNS tunneling the script
> looks at the query size and the return message size and then uses sumstats
> to alert on any host that crosses a specified threshold of supicious DNS
> reqs/msgs seen.
>
> I have seen that there are a lot of services out there conducting large
> hostname queries which creates some FPs.
>
> # Raises notices for odd or suspicious DNS traffic
> #   - Detects DNS on non-standard ports
> #   - Attempts to detect DNS tunneling
> #       - intelligence for different query types --- TO DO
> #       - statistical analysis' --- TO DO
> #   - Detect DNS responses with interesting IPs --- TO DO
>
> *Script on GitHub:*
>
> https://github.com/theflakes/bro-scripts/blob/master/2.4-scripts/dns-bad_behavior.bro
>
>
>
> -Brian Kellogg
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151002/1c5c7bec/attachment.html 

From jazoff at illinois.edu  Fri Oct  2 12:41:41 2015
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Fri, 2 Oct 2015 19:41:41 +0000
Subject: [Bro] Memory Issue with Bro
In-Reply-To: <CAMCSCoW6AisyP479iHN5pms+AcHPmFUAEdUvsJvhUPoRTNxD5w@mail.gmail.com>
References: <e9c88f86c0b144d0a33afafa324f9555@ocex03.SolarFlarecom.com>
	<7E1F7811-8496-4F0F-A2BD-7D2FB419C6D2@illinois.edu>
	<69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com>
	<454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu>
	<7689d3f6ddb6454d855e6fd3a3b8ea7b@ocex03.SolarFlarecom.com>
	<3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org>
	<CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
	<DE5BED9E-D710-4763-A978-423D917509A5@illinois.edu>
	<CAMCSCoW6AisyP479iHN5pms+AcHPmFUAEdUvsJvhUPoRTNxD5w@mail.gmail.com>
Message-ID: <AE9E9388-EC16-4040-A8CD-3752E1E3E3A8@illinois.edu>

Is it the manager parent or the child process?

-- 
- Justin Azoff

> On Oct 2, 2015, at 12:55 PM, Joe Blow <blackhole.em at gmail.com> wrote:
> 
> It's my manager processes using tons of memory...
> 
> How would you suggest debugging the manager processes?
> 
> Cheers,
> 
> JB
> 
> On Fri, Oct 2, 2015 at 12:21 PM, Azoff, Justin S <jazoff at illinois.edu> wrote:
> 
> > On Sep 30, 2015, at 11:55 AM, Joe Blow <blackhole.em at gmail.com> wrote:
> >
> > I'm super interested in this thread, as I believe i'm experiencing the same memory leak, using the solarflare cards.
> > i'm running a similar setup, with 20 workers and lots of traffic, but i'm having to bounce the entire NIC once Bro goes haywire.  Bro doesn't take too long before it's wiped the whole box out of memory (all 192GB).
> >
> > Please let me know how to troubleshooting goes.  I'm happy to provide logs.
> >
> > Cheers,
> >
> > JB
> 
> Memory leaks are tricky.  It is important to make a distinction about what component is using a lot of memory:
> 
> 1) the workers - analyzer issues and leaks in general would show up here.
> 2) the proxies - communication related
> 3) the manager - child - if the manager is overloaded the child will buffer log data
> 4) the manager - parent - if a logging destination is overloaded the parent will buffer log writes
> 
> If your manager processes are using a lot of ram, that doesn't have anything to do with the capture library in use.
> 
> --
> - Justin Azoff
> 
> 



From liburdi.joshua at gmail.com  Fri Oct  2 13:45:50 2015
From: liburdi.joshua at gmail.com (Josh Liburdi)
Date: Fri, 2 Oct 2015 16:45:50 -0400
Subject: [Bro] DNS behavior alerting
In-Reply-To: <CAEZw2bz9sw738DnrTDV5cuUH40xrbGOVAop3f8LN4_GZzzK3QQ@mail.gmail.com>
References: <CAOAzwcdMT-w32e9T4pui3JZV81Uz4q8GDvxvqxeKzJm5VhjOWQ@mail.gmail.com>
	<CAEZw2bz9sw738DnrTDV5cuUH40xrbGOVAop3f8LN4_GZzzK3QQ@mail.gmail.com>
Message-ID: <CANrCRiE+7iA8t8rBn4VjR9e3WkDF_cguAUcqqT1Uu5rW5kz+1A@mail.gmail.com>

In my experience, detecting DNS tunneling with Anthony's first
suggestion may be the easiest and most effective way. I have a script
that does that and it's very high quality (no false positives except
for anti-virus DNS activity, which is easily whitelisted).

On Fri, Oct 2, 2015 at 3:04 PM, anthony kasza <anthony.kasza at gmail.com> wrote:
> For DNS tunneling detection, look for long qnames with few labels, very low
> TTLs, odd qclasses and types (like null), and response rr's with high ratios
> of ASCII to non ASCII bytes.
>
> You can also look for bursty queries from a single orig_h for a single qname
> (likely with many different subdomains).
>
> Remember that both the client software issuing queries and the name server
> software issuing responses need to know how to decode tunneling queries.
> Check out the Iodine protocol specification for more info.
>
> -AK
>
> On Oct 2, 2015 10:54 AM, "Brian Kellogg" <theflakes at gmail.com> wrote:
>>
>> I started a Bro script a while ago that I haven't had time to develop much
>> beyond the starter framework.  The script is meant to do the below.  I
>> started working on it again but I'd welcome any help/feedback anyone would
>> be willing to offer.  It does try to do some basic DNS tunneling detection
>> but it needs more intelligence built into it.  For DNS tunneling the script
>> looks at the query size and the return message size and then uses sumstats
>> to alert on any host that crosses a specified threshold of supicious DNS
>> reqs/msgs seen.
>>
>> I have seen that there are a lot of services out there conducting large
>> hostname queries which creates some FPs.
>>
>> # Raises notices for odd or suspicious DNS traffic
>> #   - Detects DNS on non-standard ports
>> #   - Attempts to detect DNS tunneling
>> #       - intelligence for different query types --- TO DO
>> #       - statistical analysis' --- TO DO
>> #   - Detect DNS responses with interesting IPs --- TO DO
>>
>> Script on GitHub:
>>
>> https://github.com/theflakes/bro-scripts/blob/master/2.4-scripts/dns-bad_behavior.bro
>>
>>
>> -Brian Kellogg
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

From kyilmaz80 at gmail.com  Sat Oct  3 07:41:27 2015
From: kyilmaz80 at gmail.com (Koray YILMAZ)
Date: Sat, 3 Oct 2015 17:41:27 +0300
Subject: [Bro] Capturing SMTP MIME message content
Message-ID: <CAFPRsBuNFCVKFCXzofirADYbaZFqKL0_ONvTN26zN_oGRRc1Aw@mail.gmail.com>

Hello,

I am doing a master thesis about e-mail data loss prevention. I want to
capture mime content online from a postfix mail server. Is it possible to
capture the mime content of messages with bro? Also I want to extract the
file attachments from message.

Thanks.

-Koray


-- 

*Koray YILMAZ,Msc*
System Administrator
Physics Engineer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151003/474ec5bc/attachment.html 

From vern at berkeley.edu  Sun Oct  4 17:32:34 2015
From: vern at berkeley.edu (Vern Paxson)
Date: Sun, 04 Oct 2015 17:32:34 -0700
Subject: [Bro] DNS behavior alerting
In-Reply-To: <CANrCRiE+7iA8t8rBn4VjR9e3WkDF_cguAUcqqT1Uu5rW5kz+1A@mail.gmail.com>
	(Fri, 02 Oct 2015 16:45:50 EDT).
Message-ID: <20151005003214.61B442C4046@rock.ICSI.Berkeley.EDU>

> In my experience, detecting DNS tunneling with Anthony's first
> suggestion may be the easiest and most effective way. I have a script
> that does that and it's very high quality (no false positives except
> for anti-virus DNS activity, which is easily whitelisted).

For those interested in this, we developed a general framework for detecting
surreptitious communication over DNS:

	http://www.icir.org/vern/papers/covert-dns-usec13.pdf

We mainly explored it for off-line use, but also showed that in principle
it could run in real-time.  We didn't do a Bro implementation, though.

		Vern

From vitologrillo at gmail.com  Mon Oct  5 09:34:21 2015
From: vitologrillo at gmail.com (Vito Logrillo)
Date: Mon, 5 Oct 2015 18:34:21 +0200
Subject: [Bro] About signatures
Message-ID: <CANdYiFHx-aVMmi=xh3pNZbH_pbQhnSC+chNEDNQNn_Epbpqy=w@mail.gmail.com>

Hi All,
i'm studying your signature framework
https://www.bro.org/sphinx/frameworks/signatures.html
and i've found this explanation

" However, in our experience this didn?t turn out to be a very useful
thing to do because by simply using Snort signatures, one can?t
benefit from the additional capabilities that Bro provides; the
approaches of the two systems are just too different"

I understand that Bro and Snort have different approaches, but if i
need a detailed research on a specific string (for example) should i
write a script?And for several strings?
Which is the best approach to avoid signatures?
Thanks


From robin at icir.org  Mon Oct  5 09:54:34 2015
From: robin at icir.org (Robin Sommer)
Date: Mon, 5 Oct 2015 09:54:34 -0700
Subject: [Bro] About signatures
In-Reply-To: <CANdYiFHx-aVMmi=xh3pNZbH_pbQhnSC+chNEDNQNn_Epbpqy=w@mail.gmail.com>
References: <CANdYiFHx-aVMmi=xh3pNZbH_pbQhnSC+chNEDNQNn_Epbpqy=w@mail.gmail.com>
Message-ID: <20151005165434.GF94902@icir.org>

You might want to read this paper for more context about Bro's
signature framework: http://www.icir.org/robin/papers/ccs03.ps.

The comment you cite below is not saying signatures that aren't useful
at all in Bro; it's just saying that blindly converting Snort
signatures to Bro signatures hasn't proven to be a very useful thing
to do in practice.

Robin

On Mon, Oct 05, 2015 at 18:34 +0200, Vito Logrillo wrote:

> Hi All,
> i'm studying your signature framework
> https://www.bro.org/sphinx/frameworks/signatures.html
> and i've found this explanation
> 
> " However, in our experience this didn?t turn out to be a very useful
> thing to do because by simply using Snort signatures, one can?t
> benefit from the additional capabilities that Bro provides; the
> approaches of the two systems are just too different"
> 
> I understand that Bro and Snort have different approaches, but if i
> need a detailed research on a specific string (for example) should i
> write a script?And for several strings?
> Which is the best approach to avoid signatures?
> Thanks
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 



-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From blackhole.em at gmail.com  Mon Oct  5 11:31:22 2015
From: blackhole.em at gmail.com (Joe Blow)
Date: Mon, 5 Oct 2015 14:31:22 -0400
Subject: [Bro] Memory Issue with Bro
In-Reply-To: <AE9E9388-EC16-4040-A8CD-3752E1E3E3A8@illinois.edu>
References: <e9c88f86c0b144d0a33afafa324f9555@ocex03.SolarFlarecom.com>
	<7E1F7811-8496-4F0F-A2BD-7D2FB419C6D2@illinois.edu>
	<69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com>
	<454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu>
	<7689d3f6ddb6454d855e6fd3a3b8ea7b@ocex03.SolarFlarecom.com>
	<3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org>
	<CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
	<DE5BED9E-D710-4763-A978-423D917509A5@illinois.edu>
	<CAMCSCoW6AisyP479iHN5pms+AcHPmFUAEdUvsJvhUPoRTNxD5w@mail.gmail.com>
	<AE9E9388-EC16-4040-A8CD-3752E1E3E3A8@illinois.edu>
Message-ID: <CAMCSCoVYWJROF1V_JMwYer1HkHr8Nn6ViUgsN=nohmkR_4-03A@mail.gmail.com>

Ok so it looks like it's the parent manager process.  I'm pointing bro at
an haproxy pool for ES, and ES isn't terribly bogged down.  Is there any
way to specify multiple ES nodes?  Or should we be using a pool like i'm
using?

Cheers,

JB

On Fri, Oct 2, 2015 at 3:41 PM, Azoff, Justin S <jazoff at illinois.edu> wrote:

> Is it the manager parent or the child process?
>
> --
> - Justin Azoff
>
> > On Oct 2, 2015, at 12:55 PM, Joe Blow <blackhole.em at gmail.com> wrote:
> >
> > It's my manager processes using tons of memory...
> >
> > How would you suggest debugging the manager processes?
> >
> > Cheers,
> >
> > JB
> >
> > On Fri, Oct 2, 2015 at 12:21 PM, Azoff, Justin S <jazoff at illinois.edu>
> wrote:
> >
> > > On Sep 30, 2015, at 11:55 AM, Joe Blow <blackhole.em at gmail.com> wrote:
> > >
> > > I'm super interested in this thread, as I believe i'm experiencing the
> same memory leak, using the solarflare cards.
> > > i'm running a similar setup, with 20 workers and lots of traffic, but
> i'm having to bounce the entire NIC once Bro goes haywire.  Bro doesn't
> take too long before it's wiped the whole box out of memory (all 192GB).
> > >
> > > Please let me know how to troubleshooting goes.  I'm happy to provide
> logs.
> > >
> > > Cheers,
> > >
> > > JB
> >
> > Memory leaks are tricky.  It is important to make a distinction about
> what component is using a lot of memory:
> >
> > 1) the workers - analyzer issues and leaks in general would show up here.
> > 2) the proxies - communication related
> > 3) the manager - child - if the manager is overloaded the child will
> buffer log data
> > 4) the manager - parent - if a logging destination is overloaded the
> parent will buffer log writes
> >
> > If your manager processes are using a lot of ram, that doesn't have
> anything to do with the capture library in use.
> >
> > --
> > - Justin Azoff
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151005/bf529110/attachment.html 

From vitologrillo at gmail.com  Mon Oct  5 12:11:40 2015
From: vitologrillo at gmail.com (Vito Logrillo)
Date: Mon, 5 Oct 2015 21:11:40 +0200
Subject: [Bro] About signatures
In-Reply-To: <20151005165434.GF94902@icir.org>
References: <CANdYiFHx-aVMmi=xh3pNZbH_pbQhnSC+chNEDNQNn_Epbpqy=w@mail.gmail.com>
	<20151005165434.GF94902@icir.org>
Message-ID: <CANdYiFEiuU74H57toHjca5JXn05=6Bbs6eG6V_zXEhsV-4F63Q@mail.gmail.com>

Thanks Robin for your reply.
I've read your paper and i think i've understood why a blindy
convertion is not so useful: one reason is the possible generation of
many false positives(correct me if i'm wrong).
Can you suggest me a repository or a link where i can find signatures
specifically written for Bro?
Thanks
Vito



2015-10-05 18:54 GMT+02:00 Robin Sommer <robin at icir.org>:
> You might want to read this paper for more context about Bro's
> signature framework: http://www.icir.org/robin/papers/ccs03.ps.
>
> The comment you cite below is not saying signatures that aren't useful
> at all in Bro; it's just saying that blindly converting Snort
> signatures to Bro signatures hasn't proven to be a very useful thing
> to do in practice.
>
> Robin
>
> On Mon, Oct 05, 2015 at 18:34 +0200, Vito Logrillo wrote:
>
>> Hi All,
>> i'm studying your signature framework
>> https://www.bro.org/sphinx/frameworks/signatures.html
>> and i've found this explanation
>>
>> " However, in our experience this didn?t turn out to be a very useful
>> thing to do because by simply using Snort signatures, one can?t
>> benefit from the additional capabilities that Bro provides; the
>> approaches of the two systems are just too different"
>>
>> I understand that Bro and Snort have different approaches, but if i
>> need a detailed research on a specific string (for example) should i
>> write a script?And for several strings?
>> Which is the best approach to avoid signatures?
>> Thanks
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>
>
> --
> Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin


From anthony.kasza at gmail.com  Mon Oct  5 12:38:21 2015
From: anthony.kasza at gmail.com (anthony kasza)
Date: Mon, 5 Oct 2015 12:38:21 -0700
Subject: [Bro] About signatures
In-Reply-To: <CANdYiFEiuU74H57toHjca5JXn05=6Bbs6eG6V_zXEhsV-4F63Q@mail.gmail.com>
References: <CANdYiFHx-aVMmi=xh3pNZbH_pbQhnSC+chNEDNQNn_Epbpqy=w@mail.gmail.com>
	<20151005165434.GF94902@icir.org>
	<CANdYiFEiuU74H57toHjca5JXn05=6Bbs6eG6V_zXEhsV-4F63Q@mail.gmail.com>
Message-ID: <CAEZw2byO+jxBefNoW4c=b1GRpf6uZoVRKRpgbn_4qDSAZNJMrw@mail.gmail.com>

Bro's use of signatures is focussed more on protocol identification than on
alerting an operator to malicious/benign packets.

-AK
On Oct 5, 2015 12:13 PM, "Vito Logrillo" <vitologrillo at gmail.com> wrote:

> Thanks Robin for your reply.
> I've read your paper and i think i've understood why a blindy
> convertion is not so useful: one reason is the possible generation of
> many false positives(correct me if i'm wrong).
> Can you suggest me a repository or a link where i can find signatures
> specifically written for Bro?
> Thanks
> Vito
>
>
>
> 2015-10-05 18:54 GMT+02:00 Robin Sommer <robin at icir.org>:
> > You might want to read this paper for more context about Bro's
> > signature framework: http://www.icir.org/robin/papers/ccs03.ps.
> >
> > The comment you cite below is not saying signatures that aren't useful
> > at all in Bro; it's just saying that blindly converting Snort
> > signatures to Bro signatures hasn't proven to be a very useful thing
> > to do in practice.
> >
> > Robin
> >
> > On Mon, Oct 05, 2015 at 18:34 +0200, Vito Logrillo wrote:
> >
> >> Hi All,
> >> i'm studying your signature framework
> >> https://www.bro.org/sphinx/frameworks/signatures.html
> >> and i've found this explanation
> >>
> >> " However, in our experience this didn?t turn out to be a very useful
> >> thing to do because by simply using Snort signatures, one can?t
> >> benefit from the additional capabilities that Bro provides; the
> >> approaches of the two systems are just too different"
> >>
> >> I understand that Bro and Snort have different approaches, but if i
> >> need a detailed research on a specific string (for example) should i
> >> write a script?And for several strings?
> >> Which is the best approach to avoid signatures?
> >> Thanks
> >>
> >> _______________________________________________
> >> Bro mailing list
> >> bro at bro-ids.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >>
> >
> >
> >
> > --
> > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151005/e9527f86/attachment.html 

From sudo.darkstar at gmail.com  Mon Oct  5 14:59:55 2015
From: sudo.darkstar at gmail.com (John B. Althouse III)
Date: Mon, 5 Oct 2015 17:59:55 -0400
Subject: [Bro] Capturing the SSL cert via HTTP Connect Method
Message-ID: <CALW-b1z80Lszcmuu9gJUH=Wd1mMH_pY7SxR4MnDhMJSxAuXCXw@mail.gmail.com>

Has anyone come up with a way to get Bro to capture the SSL cert details
when it's over a HTTP Connect tunnel? Attached is a sample PCAP.

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151005/41b3de41/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 151005-93OevfuI6-RCPI4P1roPumvw.pcap
Type: application/octet-stream
Size: 6776 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151005/41b3de41/attachment.obj 

From liburdi.joshua at gmail.com  Mon Oct  5 15:25:08 2015
From: liburdi.joshua at gmail.com (Josh Liburdi)
Date: Mon, 5 Oct 2015 18:25:08 -0400
Subject: [Bro] Capturing the SSL cert via HTTP Connect Method
In-Reply-To: <CALW-b1z80Lszcmuu9gJUH=Wd1mMH_pY7SxR4MnDhMJSxAuXCXw@mail.gmail.com>
References: <CALW-b1z80Lszcmuu9gJUH=Wd1mMH_pY7SxR4MnDhMJSxAuXCXw@mail.gmail.com>
Message-ID: <CANrCRiEHVhb2vt337oboAZDbAX=PjoPRvcjSRBjKDAmn7S5_UA@mail.gmail.com>

I don't think this would be too difficult to add-- there's already
code that hands off the HTTP stream to other child analyzers if it's a
CONNECT tunnel (see lines 998 through 1019 in
src/analyzer/protocol/http/HTTP.cc). It's a bit beyond me how to get
this working, I needed help from Seth to get it working with RDP, but
maybe someone with more experience can add this to their todo list.

On Mon, Oct 5, 2015 at 5:59 PM, John B. Althouse III
<sudo.darkstar at gmail.com> wrote:
> Has anyone come up with a way to get Bro to capture the SSL cert details
> when it's over a HTTP Connect tunnel? Attached is a sample PCAP.
>
> Thanks!
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

From johanna at icir.org  Mon Oct  5 15:37:37 2015
From: johanna at icir.org (Johanna Amann)
Date: Mon, 5 Oct 2015 15:37:37 -0700
Subject: [Bro] Capturing the SSL cert via HTTP Connect Method
In-Reply-To: <CALW-b1z80Lszcmuu9gJUH=Wd1mMH_pY7SxR4MnDhMJSxAuXCXw@mail.gmail.com>
References: <CALW-b1z80Lszcmuu9gJUH=Wd1mMH_pY7SxR4MnDhMJSxAuXCXw@mail.gmail.com>
Message-ID: <20151005223732.GA45205@wifi79.sys.ICSI.Berkeley.EDU>

This actually is usually already supported in Bro. If I am not mistaken,
the reason why this does not work in this case is the proxy-agent header
in the response from the HTTP server.

https://bro-tracker.atlassian.net/browse/BIT-1487 has the details and a
patch that might fix your problem.

I hope this helps,
 Johanna

On Mon, Oct 05, 2015 at 05:59:55PM -0400, John B. Althouse III wrote:
> Has anyone come up with a way to get Bro to capture the SSL cert details
> when it's over a HTTP Connect tunnel? Attached is a sample PCAP.
> 
> Thanks!


> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From jdopheid at illinois.edu  Tue Oct  6 08:16:03 2015
From: jdopheid at illinois.edu (Dopheide, Jeannette M)
Date: Tue, 6 Oct 2015 15:16:03 +0000
Subject: [Bro] Bro has joined Software Freedom Conservancy
Message-ID: <D2394FE3.1A38B%jdopheid@illinois.edu>

The Bro Project is excited to announce it has joined Software Freedom
Conservancy (SFC), a non-profit organization that promotes and protects
open source projects. To learn more about this decision, see our blog
post: 


http://blog.bro.org/2015/10/bro-software-freedom-conservancy.html



- The Bro Development Team




From sudo.darkstar at gmail.com  Tue Oct  6 08:21:23 2015
From: sudo.darkstar at gmail.com (John B. Althouse III)
Date: Tue, 6 Oct 2015 11:21:23 -0400
Subject: [Bro] Capturing the SSL cert via HTTP Connect Method
In-Reply-To: <20151005223732.GA45205@wifi79.sys.ICSI.Berkeley.EDU>
References: <CALW-b1z80Lszcmuu9gJUH=Wd1mMH_pY7SxR4MnDhMJSxAuXCXw@mail.gmail.com>
	<20151005223732.GA45205@wifi79.sys.ICSI.Berkeley.EDU>
Message-ID: <CALW-b1xD9K42K1SBtOY3NYK6Q=xaCHVUrLMmgaCBP0ZtoSX6vw@mail.gmail.com>

Thanks Johanna! That's exactly what I was looking for. Any idea when this
will make it into the master repo?

On Mon, Oct 5, 2015 at 6:37 PM, Johanna Amann <johanna at icir.org> wrote:

> This actually is usually already supported in Bro. If I am not mistaken,
> the reason why this does not work in this case is the proxy-agent header
> in the response from the HTTP server.
>
> https://bro-tracker.atlassian.net/browse/BIT-1487 has the details and a
> patch that might fix your problem.
>
> I hope this helps,
>  Johanna
>
> On Mon, Oct 05, 2015 at 05:59:55PM -0400, John B. Althouse III wrote:
> > Has anyone come up with a way to get Bro to capture the SSL cert details
> > when it's over a HTTP Connect tunnel? Attached is a sample PCAP.
> >
> > Thanks!
>
>
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151006/fc05b74a/attachment.html 

From franky.meier.1 at gmx.de  Wed Oct  7 01:27:00 2015
From: franky.meier.1 at gmx.de (Frank Meier)
Date: Wed, 7 Oct 2015 10:27:00 +0200
Subject: [Bro] feature proposal: bro batch processing
Message-ID: <20151007102700.69695324@NB181106>

Hi,

as a follow-up to my question about "delayed bro operation" I would
like to propose a new feature for bro. I call it batch mode and it
helps to run bro over a large amount of pcap files. 

Until now I used a modified version of tcpslice to send multiple pcaps
to bro through a pipe. This setup works but is a bit complicated. It
also has the downside, that bro blocks reading from the pipe which
breaks the usual event loop.

What I hacked together is a simple change to PcapSource: When passed a
directory instead of a file it looks for files with the suffix pcap and
processes them. When EOF is reached, it closes the file and renames it. 
But instead of also closing the IOSource it looks for the next file.
When there are no files to process, it behaves just like live capture
mode when there are no packets available. 

My patch is quite basic right now: just grab the first pcap you can
find and work with it, but one could think of extended features:

1) Read files to work on from text file: This would also come handy
when the source files are distributed in the file system, e.g. sorted by
date or just to avoid to many files in one directory. Compared to
passing multiple file names to bro via command line, this also works
around the problem of a "to long argument list".

2) sort mode: check timestamps of all available files in directory and
process them in the right order. This mode would have to be smart
enough not to open all files at the same time running out of file
descriptors (like mergecap). So check timestamps first and open only the
files needed. This could be more than one when there are separate pcaps
for rx/tx.

3) For each flow save the the name/path of the first and last file it
was read from. So when detailed analysis is necessary, your exactly know
which files to open.

I would like to know if others would be interested in this kind of
feature. Also better ideas how to solve this "the bro way" are welcome. 

Franky

From thomastan81 at gmail.com  Wed Oct  7 06:38:42 2015
From: thomastan81 at gmail.com (Thomas Tan)
Date: Wed, 7 Oct 2015 15:38:42 +0200
Subject: [Bro] How can I invoke event tcp_option()?
Message-ID: <CANxb75BKC4EK84428NbN5XLtKVHU5BAGko445FZd1dXnbPZU3g@mail.gmail.com>

Dear All,

I am new to Bro. I am testing the ?*event tcp_option ()*?.  However, the
event is not invoked by the event engine.  Can anyone kindly advise me what
I have done wrong? My code is as follows.



------------------------------------------

@load base/protocols/conn



event bro_init() {

    print "Hello, World!";

}



event tcp_option (c:connection, is_orig:bool, opt:count, optlen:count){

                print "inside the tcp_option";

}



event bro_done() {

    print "Goodbye, World!";

}

-----------------------------------------



Best regards,


Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151007/8262815b/attachment.html 

From jazoff at illinois.edu  Wed Oct  7 06:58:44 2015
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Wed, 7 Oct 2015 13:58:44 +0000
Subject: [Bro] How can I invoke event tcp_option()?
In-Reply-To: <CANxb75BKC4EK84428NbN5XLtKVHU5BAGko445FZd1dXnbPZU3g@mail.gmail.com>
References: <CANxb75BKC4EK84428NbN5XLtKVHU5BAGko445FZd1dXnbPZU3g@mail.gmail.com>
Message-ID: <0C75B3BB-C63A-441E-A860-CEAFF9C98A1C@illinois.edu>


> On Oct 7, 2015, at 9:38 AM, Thomas Tan <thomastan81 at gmail.com> wrote:
> 
> Dear All,
> 
> I am new to Bro. I am testing the ?event tcp_option ()?.  However, the event is not invoked by the event engine.  Can anyone kindly advise me what I have done wrong? My code is as follows.

Are you running this against a pcap?  Are you maybe not running bro with the -C option to ignore invalid checksums from checksum offloading?

This works on when I try it on try.bro.org against the example caps:

event tcp_option (c:connection, is_orig:bool, opt:count, optlen:count){
	print c$id$orig_h, is_orig, opt, optlen;
}


http://try.bro.org/#/trybro/saved/21633



-- 
- Justin Azoff



From seth at icir.org  Wed Oct  7 07:09:38 2015
From: seth at icir.org (Seth Hall)
Date: Wed, 7 Oct 2015 10:09:38 -0400
Subject: [Bro] feature proposal: bro batch processing
In-Reply-To: <20151007102700.69695324@NB181106>
References: <20151007102700.69695324@NB181106>
Message-ID: <E746A7B6-4135-490E-AA5C-B86DD89A6DF7@icir.org>


> On Oct 7, 2015, at 4:27 AM, Frank Meier <franky.meier.1 at gmx.de> wrote:
> 
> as a follow-up to my question about "delayed bro operation" I would
> like to propose a new feature for bro. I call it batch mode and it
> helps to run bro over a large amount of pcap files. 

This is a feature that we actually expect packet-bricks to be able to solve shortly.  We?ve been thinking about this for a while and with packet-bricks I actually expect we?ll be able to take this even a bit further to process large sets of traces as a cluster as well.  You could have packet-bricks essentially play the role of coordinator to pass packets to the processes and continue passing packets as more traces show up (I know of some people that are doing ?real-time? sniffing by copying traces from a remote location to their Bro installation).  This allows the Bro processes to remain up and keep their state without actually making any changes to Bro.  I can see making a small change to Bro to support collecting the timestamps from the trace by having packet-bricks annotating the packets with timestamps.

It?s nice to see that other people are concretely thinking about solving this same problem.  Definitely keep an eye on packet-bricks over the next couple of months. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From masoom.alam at gmail.com  Sat Oct 10 02:54:15 2015
From: masoom.alam at gmail.com (masoom alam)
Date: Sat, 10 Oct 2015 02:54:15 -0700
Subject: [Bro] BRO sniffing traffic on a VLAN
Message-ID: <CAPihp9m4fxvvhg08rgfFbyvYEV2dwfNi-_jGJFZiPvJvWgOsCg@mail.gmail.com>

Dear ALL,

I have plugged BRO in my lab in the mirrored port of a physical switch.
Thus BRO is able to sniff all the traffic.

My question is that if we install BRO on a simple linux machine and try
sniffing a LAN traffic for analysis for example, do we need some special
measure? I mean is it necessary that BRO should be plugged in the mirrored
port...

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151010/fc79ad95/attachment.html 

From hhoffman at ip-solutions.net  Sat Oct 10 05:13:27 2015
From: hhoffman at ip-solutions.net (Harry Hoffman)
Date: Sat, 10 Oct 2015 08:13:27 -0400
Subject: [Bro] BRO sniffing traffic on a VLAN
In-Reply-To: <CAPihp9m4fxvvhg08rgfFbyvYEV2dwfNi-_jGJFZiPvJvWgOsCg@mail.gmail.com>
Message-ID: <a8ec1829-6b7b-438d-bc75-df4537197905@email.android.com>

Hi Masoom,

Bro will still see some traffic (traffic destined to the bro box, ARP, broadcast, potential port floods) but not other traffic.

Cheers,
Harry

On Oct 10, 2015 5:54 AM, masoom alam <masoom.alam at gmail.com> wrote:
>
> Dear ALL,?
>
> I have plugged BRO in my lab in the mirrored port of a physical switch. Thus BRO is able to sniff all the traffic.?
>
> My question is that if we install BRO on a simple linux machine and try sniffing a LAN traffic for analysis for example, do we need some special measure? I mean is it necessary that BRO should be plugged in the mirrored port...
>
> Thanks


From doris at bro.org  Wed Oct 14 11:09:01 2015
From: doris at bro.org (Doris Schioberg)
Date: Wed, 14 Oct 2015 11:09:01 -0700
Subject: [Bro] Bro News #6
Message-ID: <561E9A3D.5070608@bro.org>

The 6th issue of the Bro Newsletter is ready. Read about the exciting
things we did this year:
http://blog.bro.org/2015/10/bro-news-6.html

- The Bro Team

From daniel.guerra69 at gmail.com  Wed Oct 14 16:39:04 2015
From: daniel.guerra69 at gmail.com (Daniel Guerra)
Date: Thu, 15 Oct 2015 01:39:04 +0200
Subject: [Bro]  broker compile problem
Message-ID: <1E3CEB14-9885-469C-85F5-4A8943272753@gmail.com>

Hi,

I have been trying over and over to solve this but failed.
I keep getting this error when compiling bro
[  9%] Building CXX object aux/broker/tests/CMakeFiles/test_data.dir/test_data.cc.o
Linking CXX executable test_data
../libbroker.so.0.4-9.0: undefined reference to `inflateInit2_'
../libbroker.so.0.4-9.0: undefined reference to `inflate'
../libbroker.so.0.4-9.0: undefined reference to `deflateInit2_'
../libbroker.so.0.4-9.0: undefined reference to `deflate'
../libbroker.so.0.4-9.0: undefined reference to `deflateEnd'
../libbroker.so.0.4-9.0: undefined reference to `inflateEnd'
collect2: error: ld returned 1 exit status
I have attached the Dockerfile I used


Regards,

Daniel Guerra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151015/18675082/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Dockerfile
Type: application/octet-stream
Size: 1825 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151015/18675082/attachment.obj 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151015/18675082/attachment-0001.html 

From BLMILLER at comerica.com  Thu Oct 15 11:29:22 2015
From: BLMILLER at comerica.com (MILLER, BRAD L)
Date: Thu, 15 Oct 2015 18:29:22 +0000
Subject: [Bro] archive-log process apparently failing
Message-ID: <D7A10B8F46BF1F47A4D30051C9CA31CC11367232@MIMESMSW03AHCP.US.comerica.net>

We are on Bro 2.3.x and have run into a very occasional process that appears to indicate the archive-log process fails.  The symptom we see is a logjam (the word kind of fit here) of logs staying in the current directory and getting larger and larger, with no rotation into gz files outside of this directory.  Broctl restart sets it straight again, but this issue came up twice now in recent memory.  We tend to lose logs in the logjam when this is corrected via broctl restart.

Anything we can do?  Cause?

Brad Miller | Comerica Bank
Information Security Architecture
IT Security
Office: 248.371.4249  | Mobile: 920.378.8138



Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151015/9f32f604/attachment.html 

From dnthayer at illinois.edu  Thu Oct 15 13:19:41 2015
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Thu, 15 Oct 2015 15:19:41 -0500
Subject: [Bro] archive-log process apparently failing
In-Reply-To: <D7A10B8F46BF1F47A4D30051C9CA31CC11367232@MIMESMSW03AHCP.US.comerica.net>
References: <D7A10B8F46BF1F47A4D30051C9CA31CC11367232@MIMESMSW03AHCP.US.comerica.net>
Message-ID: <56200A5D.7020900@illinois.edu>

On 10/15/2015 01:29 PM, MILLER, BRAD L wrote:
> We are on Bro 2.3.x and have run into a very occasional process that
> appears to indicate the archive-log process fails.  The symptom we see
> is a logjam (the word kind of fit here) of logs staying in the current
> directory and getting larger and larger, with no rotation into gz files
> outside of this directory.  Broctl restart sets it straight again, but
> this issue came up twice now in recent memory.  We tend to lose logs in
> the logjam when this is corrected via broctl restart.
>
> Anything we can do?  Cause?
>
> Brad Miller | Comerica Bank
>

Are you seeing rotated logs? (rotated logs have a timestamp in the 
filename, such as "conn.2015-10-15-14-42-00.log")
Or, are you just seeing the current logs getting larger and larger? 
(such as "conn.log")

If you don't see any rotated logs (and your logs aren't getting 
archived), then you should check if your
log rotation interval is set to a reasonable value (and you must
do "broctl install" and restart Bro if you change your config).

When logs are archived, they are compressed and moved into a
subdirectory named like this:
<PREFIX>/logs/XXXX-XX-XX

If you don't see the logs being archived, then (after doing a
broctl restart) you can check if there are any directories with
names like this:
<PREFIX>/spool/tmp/post-terminate-XXXX-XX-XX-XX-XX-XX-XXXXX
Those directories are where you can find your "lost" log files
(however, if you do "broctl cleanup --all", then broctl will remove
all of those directories without warning).

From albert.zaharovits at gmail.com  Fri Oct 16 00:44:20 2015
From: albert.zaharovits at gmail.com (Albert Zaharovits)
Date: Fri, 16 Oct 2015 10:44:20 +0300
Subject: [Bro] HTTP Content Disposition
Message-ID: <20151016074420.GA32367@BATTLESTAR>

Hi there,

I'm having an issue with the HTTP Content Dispositon header: the 
filename field of the HTTP::Info record doesn't get filed.

Event **event http_header(c: connection, is_orig: bool, name: string, 
value: string)** gets the header, but filename is empty in **event 
HTTP::log_http(rec: HTTP::Info)**.

I'm testing with:
```bash
curl -i 
'http://httpbin.org/response-headers?Content-Type=application/octet-stream;%20charset=UTF-8&Content-Disposition=attachment;%20filename%3d%22test.json%22'
```

What say you?

Thanks,
Albert

From seth at icir.org  Fri Oct 16 06:15:59 2015
From: seth at icir.org (Seth Hall)
Date: Fri, 16 Oct 2015 09:15:59 -0400
Subject: [Bro] HTTP Content Disposition
In-Reply-To: <20151016074420.GA32367@BATTLESTAR>
References: <20151016074420.GA32367@BATTLESTAR>
Message-ID: <4C3BB5D7-A5BB-4804-8159-609E1679FF2B@icir.org>


> On Oct 16, 2015, at 3:44 AM, Albert Zaharovits <albert.zaharovits at gmail.com> wrote:
> 
> I'm having an issue with the HTTP Content Dispositon header: the 
> filename field of the HTTP::Info record doesn't get filed.

I just tested this and you?re right, the filename doesn?t show up in the http.log, that?s definitely a bug.  It does at least show up in the files.log correctly.

I?ll file a ticket about that field missing in the http log.  It looks like some of our http entity tracking might be a bit off.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From albert.zaharovits at gmail.com  Fri Oct 16 06:22:53 2015
From: albert.zaharovits at gmail.com (Albert Zaharovits)
Date: Fri, 16 Oct 2015 16:22:53 +0300
Subject: [Bro] HTTP Content Disposition
In-Reply-To: <4C3BB5D7-A5BB-4804-8159-609E1679FF2B@icir.org>
References: <20151016074420.GA32367@BATTLESTAR>
	<4C3BB5D7-A5BB-4804-8159-609E1679FF2B@icir.org>
Message-ID: <CFF317D3-DA68-4C75-97BF-5736E6BB95A7@gmail.com>

I thought it might be a bug too?
I already started browsing the source/debugging, but no success so far?
Any leads? Where should I focus my attention?
I would like to submit a pull request!

Albert

> On 16 Oct 2015, at 16:15, Seth Hall <seth at icir.org> wrote:
> 
> 
>> On Oct 16, 2015, at 3:44 AM, Albert Zaharovits <albert.zaharovits at gmail.com> wrote:
>> 
>> I'm having an issue with the HTTP Content Dispositon header: the 
>> filename field of the HTTP::Info record doesn't get filed.
> 
> I just tested this and you?re right, the filename doesn?t show up in the http.log, that?s definitely a bug.  It does at least show up in the files.log correctly.
> 
> I?ll file a ticket about that field missing in the http log.  It looks like some of our http entity tracking might be a bit off.
> 
>  .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
> 



From seth at icir.org  Fri Oct 16 06:42:43 2015
From: seth at icir.org (Seth Hall)
Date: Fri, 16 Oct 2015 09:42:43 -0400
Subject: [Bro] HTTP Content Disposition
In-Reply-To: <CFF317D3-DA68-4C75-97BF-5736E6BB95A7@gmail.com>
References: <20151016074420.GA32367@BATTLESTAR>
	<4C3BB5D7-A5BB-4804-8159-609E1679FF2B@icir.org>
	<CFF317D3-DA68-4C75-97BF-5736E6BB95A7@gmail.com>
Message-ID: <9224879D-5410-4953-AD1B-4F5440DD1FA2@icir.org>


> On Oct 16, 2015, at 9:22 AM, Albert Zaharovits <albert.zaharovits at gmail.com> wrote:
> 
> I thought it might be a bug too?
> I already started browsing the source/debugging, but no success so far?
> Any leads? Where should I focus my attention?
> I would like to submit a pull request!

I would focus on the interplay where the c$http$current_entity field is set.  I?m not exactly sure which script it will be in, but you should be able to find all of the places where that?s set pretty easily.

Looking forward to a fix! :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From seth at icir.org  Fri Oct 16 06:43:43 2015
From: seth at icir.org (Seth Hall)
Date: Fri, 16 Oct 2015 09:43:43 -0400
Subject: [Bro] HTTP Content Disposition
In-Reply-To: <CFF317D3-DA68-4C75-97BF-5736E6BB95A7@gmail.com>
References: <20151016074420.GA32367@BATTLESTAR>
	<4C3BB5D7-A5BB-4804-8159-609E1679FF2B@icir.org>
	<CFF317D3-DA68-4C75-97BF-5736E6BB95A7@gmail.com>
Message-ID: <E372E787-4E13-4F8B-813C-7F639A17E0F7@icir.org>


> On Oct 16, 2015, at 9:22 AM, Albert Zaharovits <albert.zaharovits at gmail.com> wrote:
> 
> I thought it might be a bug too?
> I already started browsing the source/debugging, but no success so far?
> Any leads? Where should I focus my attention?
> I would like to submit a pull request!

Also, here the ticket: https://bro-tracker.atlassian.net/projects/BIT/issues/BIT-1493

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From jsiwek at illinois.edu  Fri Oct 16 07:17:06 2015
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Fri, 16 Oct 2015 14:17:06 +0000
Subject: [Bro] broker compile problem
In-Reply-To: <1E3CEB14-9885-469C-85F5-4A8943272753@gmail.com>
References: <1E3CEB14-9885-469C-85F5-4A8943272753@gmail.com>
Message-ID: <369532ED-A52A-4BE8-85BB-AA237E92C029@illinois.edu>


> On Oct 14, 2015, at 6:39 PM, Daniel Guerra <daniel.guerra69 at gmail.com> wrote:
> 
> I have been trying over and over to solve this but failed.
> I keep getting this error when compiling bro
> 
> ../libbroker.so.0.4-9.0: undefined reference to `inflate?

Is zlib installed and in a path the linker searches?

- Jon


From vitologrillo at gmail.com  Fri Oct 16 08:31:05 2015
From: vitologrillo at gmail.com (Vito Logrillo)
Date: Fri, 16 Oct 2015 17:31:05 +0200
Subject: [Bro] Bro and Snort together
Message-ID: <CANdYiFH=JkijtYShJO5Te4D+vjFz9jXYeCenZ0dvVdv9iPx4vQ@mail.gmail.com>

Hi all,
Anyone have used Bro and Snort together to the same live traffic?
If yes, any suggestion?
For example, is it possible to send the same traffic to snort and bro
without packet loss?
Thanks

From lysemose at gmail.com  Fri Oct 16 08:46:28 2015
From: lysemose at gmail.com (Heine Lysemose)
Date: Fri, 16 Oct 2015 17:46:28 +0200
Subject: [Bro] Bro and Snort together
In-Reply-To: <CANdYiFH=JkijtYShJO5Te4D+vjFz9jXYeCenZ0dvVdv9iPx4vQ@mail.gmail.com>
References: <CANdYiFH=JkijtYShJO5Te4D+vjFz9jXYeCenZ0dvVdv9iPx4vQ@mail.gmail.com>
Message-ID: <CAN4C-D=iXMwYXGq5mDPNc2Cn4+FAUzpGkocNFmrkAUzv1OfK1w@mail.gmail.com>

Hi

Have a look at SecurityOnion, www.securityonion.net

Regards,
Lysemose
On Oct 16, 2015 17:32, "Vito Logrillo" <vitologrillo at gmail.com> wrote:

> Hi all,
> Anyone have used Bro and Snort together to the same live traffic?
> If yes, any suggestion?
> For example, is it possible to send the same traffic to snort and bro
> without packet loss?
> Thanks
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151016/8ab6f698/attachment.html 

From donaldson8 at llnl.gov  Fri Oct 16 09:13:09 2015
From: donaldson8 at llnl.gov (Donaldson, John)
Date: Fri, 16 Oct 2015 16:13:09 +0000
Subject: [Bro] Bro and Snort together
In-Reply-To: <CANdYiFH=JkijtYShJO5Te4D+vjFz9jXYeCenZ0dvVdv9iPx4vQ@mail.gmail.com>
References: <CANdYiFH=JkijtYShJO5Te4D+vjFz9jXYeCenZ0dvVdv9iPx4vQ@mail.gmail.com>
Message-ID: <D2468518.ABD8%donaldson8@llnl.gov>

Vito,

We're running Bro and Snort in parallel, but we're using DAG cards to
duplicate streams to Bro and Snort processes, so our performance
characteristics are a bit different. In general, though, it really depends
on how you manage the traffic that you're throwing at both, and how many
rules you have enabled in Snort. It *is* possible to keep packet loss
manageable, running them in parallel, but you'll have to trim down what
you have Snort running.


John Donaldson



On 10/16/15, 10:31 AM, "bro-bounces at bro.org on behalf of Vito Logrillo"
<bro-bounces at bro.org on behalf of vitologrillo at gmail.com> wrote:

>Hi all,
>Anyone have used Bro and Snort together to the same live traffic?
>If yes, any suggestion?
>For example, is it possible to send the same traffic to snort and bro
>without packet loss?
>Thanks
>_______________________________________________
>Bro mailing list
>bro at bro-ids.org
>http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From cw13 at umbc.edu  Fri Oct 16 09:16:12 2015
From: cw13 at umbc.edu (Chris Williams)
Date: Fri, 16 Oct 2015 12:16:12 -0400
Subject: [Bro] Bro and Snort together
In-Reply-To: <D2468518.ABD8%donaldson8@llnl.gov>
References: <CANdYiFH=JkijtYShJO5Te4D+vjFz9jXYeCenZ0dvVdv9iPx4vQ@mail.gmail.com>
	<D2468518.ABD8%donaldson8@llnl.gov>
Message-ID: <CAGEyGovaeAX1PFQOHnO=CMz-uDxpVC3RMTKzxSxyKFQ5NpbOZg@mail.gmail.com>

Is it possible to do this with multiple instances of pf_ring?
On Oct 16, 2015 12:13 PM, "Donaldson, John" <donaldson8 at llnl.gov> wrote:

> Vito,
>
> We're running Bro and Snort in parallel, but we're using DAG cards to
> duplicate streams to Bro and Snort processes, so our performance
> characteristics are a bit different. In general, though, it really depends
> on how you manage the traffic that you're throwing at both, and how many
> rules you have enabled in Snort. It *is* possible to keep packet loss
> manageable, running them in parallel, but you'll have to trim down what
> you have Snort running.
>
>
> John Donaldson
>
>
>
> On 10/16/15, 10:31 AM, "bro-bounces at bro.org on behalf of Vito Logrillo"
> <bro-bounces at bro.org on behalf of vitologrillo at gmail.com> wrote:
>
> >Hi all,
> >Anyone have used Bro and Snort together to the same live traffic?
> >If yes, any suggestion?
> >For example, is it possible to send the same traffic to snort and bro
> >without packet loss?
> >Thanks
> >_______________________________________________
> >Bro mailing list
> >bro at bro-ids.org
> >http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151016/471229a7/attachment.html 

From shirkdog.bsd at gmail.com  Fri Oct 16 10:42:00 2015
From: shirkdog.bsd at gmail.com (Michael Shirk)
Date: Fri, 16 Oct 2015 13:42:00 -0400
Subject: [Bro] Bro and Snort together
In-Reply-To: <D2468518.ABD8%donaldson8@llnl.gov>
References: <CANdYiFH=JkijtYShJO5Te4D+vjFz9jXYeCenZ0dvVdv9iPx4vQ@mail.gmail.com>
	<D2468518.ABD8%donaldson8@llnl.gov>
Message-ID: <CAL8PkUWc5XTL0JU4R_aOq2xi28hb9C9FXq56cSiuc-syEPz9sA@mail.gmail.com>

On FreeBSD, I have created a script that sets up Bro+Snort with pulledpork
so you can test:

https://github.com/shirkdog/hunter-nsm

The key thing will be your specific use case for Bro+Snort as others have
mentioned, but with this install, you can tune down the Snort rules.

--
Michael Shirk
Daemon Security, Inc.
http://www.daemon-security.com
Vito,

We're running Bro and Snort in parallel, but we're using DAG cards to
duplicate streams to Bro and Snort processes, so our performance
characteristics are a bit different. In general, though, it really depends
on how you manage the traffic that you're throwing at both, and how many
rules you have enabled in Snort. It *is* possible to keep packet loss
manageable, running them in parallel, but you'll have to trim down what
you have Snort running.


John Donaldson



On 10/16/15, 10:31 AM, "bro-bounces at bro.org on behalf of Vito Logrillo"
<bro-bounces at bro.org on behalf of vitologrillo at gmail.com> wrote:

>Hi all,
>Anyone have used Bro and Snort together to the same live traffic?
>If yes, any suggestion?
>For example, is it possible to send the same traffic to snort and bro
>without packet loss?
>Thanks
>_______________________________________________
>Bro mailing list
>bro at bro-ids.org
>http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151016/cebf840a/attachment.html 

From jwilliams at emergingthreats.net  Fri Oct 16 10:44:41 2015
From: jwilliams at emergingthreats.net (Jason Williams)
Date: Fri, 16 Oct 2015 12:44:41 -0500
Subject: [Bro] Bro and Snort together
In-Reply-To: <CAGEyGovaeAX1PFQOHnO=CMz-uDxpVC3RMTKzxSxyKFQ5NpbOZg@mail.gmail.com>
References: <CANdYiFH=JkijtYShJO5Te4D+vjFz9jXYeCenZ0dvVdv9iPx4vQ@mail.gmail.com>
	<D2468518.ABD8%donaldson8@llnl.gov>
	<CAGEyGovaeAX1PFQOHnO=CMz-uDxpVC3RMTKzxSxyKFQ5NpbOZg@mail.gmail.com>
Message-ID: <CAPpdu9Et+9=M+s=x98qm845ZT9pMWRCMUMNXFjEBzFw8gPL17A@mail.gmail.com>

I've had instances where i have used zbalance_ipc to help load balance
larger links for moloch. You can also use zbalance_ipc to create duplicate
zc streams that you can attach different processes to. I run bro / suri on
the same interface and haven't seen issues in operation.

On Fri, Oct 16, 2015 at 11:16 AM, Chris Williams <cw13 at umbc.edu> wrote:

> Is it possible to do this with multiple instances of pf_ring?
> On Oct 16, 2015 12:13 PM, "Donaldson, John" <donaldson8 at llnl.gov> wrote:
>
>> Vito,
>>
>> We're running Bro and Snort in parallel, but we're using DAG cards to
>> duplicate streams to Bro and Snort processes, so our performance
>> characteristics are a bit different. In general, though, it really depends
>> on how you manage the traffic that you're throwing at both, and how many
>> rules you have enabled in Snort. It *is* possible to keep packet loss
>> manageable, running them in parallel, but you'll have to trim down what
>> you have Snort running.
>>
>>
>> John Donaldson
>>
>>
>>
>> On 10/16/15, 10:31 AM, "bro-bounces at bro.org on behalf of Vito Logrillo"
>> <bro-bounces at bro.org on behalf of vitologrillo at gmail.com> wrote:
>>
>> >Hi all,
>> >Anyone have used Bro and Snort together to the same live traffic?
>> >If yes, any suggestion?
>> >For example, is it possible to send the same traffic to snort and bro
>> >without packet loss?
>> >Thanks
>> >_______________________________________________
>> >Bro mailing list
>> >bro at bro-ids.org
>> >http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151016/2a6da13c/attachment.html 

From longjohngolf at gmail.com  Fri Oct 16 11:15:03 2015
From: longjohngolf at gmail.com (John Daly)
Date: Fri, 16 Oct 2015 11:15:03 -0700
Subject: [Bro] archive-log process apparently failing
Message-ID: <CALRZ962KZJDv4_MeLWLbpxnh992bHj0LOFwimy0rswpw0fOz_g@mail.gmail.com>

Brad,

At the time of log rotation, Bro copies all logs from the "current"
dir (more accurately spool/manager) to the archive directory
(logs/YYYY-MM-DD) and gzips the logs. Be sure that you have CPU and IO
cycles to do both of those tasks. If you want to optimize this, tweak
the following settings in the broctl.cfg:

* Set CompressLogs = 0. This will prevent broctl from compressing the
logs, freeing up CPU cycles at log rotation time.

* Set TraceSummary = "". This will prevent the connection summary
script from being run, freeing up CPU cycles at log rotation time.

-jd

From daniel.guerra69 at gmail.com  Fri Oct 16 14:17:17 2015
From: daniel.guerra69 at gmail.com (Daniel Guerra)
Date: Fri, 16 Oct 2015 23:17:17 +0200
Subject: [Bro] broker compile problem
In-Reply-To: <369532ED-A52A-4BE8-85BB-AA237E92C029@illinois.edu>
References: <1E3CEB14-9885-469C-85F5-4A8943272753@gmail.com>
	<369532ED-A52A-4BE8-85BB-AA237E92C029@illinois.edu>
Message-ID: <A19A2803-2BD3-41ED-952A-EBAAF722CB1F@gmail.com>

I use zlib1g-dev (in Dockerfile).
I was thinking of building zlib from source.

> On 16 Oct 2015, at 16:17, Siwek, Jon <jsiwek at illinois.edu> wrote:
> 
> 
>> On Oct 14, 2015, at 6:39 PM, Daniel Guerra <daniel.guerra69 at gmail.com> wrote:
>> 
>> I have been trying over and over to solve this but failed.
>> I keep getting this error when compiling bro
>> 
>> ../libbroker.so.0.4-9.0: undefined reference to `inflate?
> 
> Is zlib installed and in a path the linker searches?
> 
> - Jon



From michalpurzynski1 at gmail.com  Fri Oct 16 14:33:11 2015
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Fri, 16 Oct 2015 21:33:11 +0000
Subject: [Bro] Bro and Snort together
In-Reply-To: <CAPpdu9Et+9=M+s=x98qm845ZT9pMWRCMUMNXFjEBzFw8gPL17A@mail.gmail.com>
References: <CANdYiFH=JkijtYShJO5Te4D+vjFz9jXYeCenZ0dvVdv9iPx4vQ@mail.gmail.com>
	<D2468518.ABD8%donaldson8@llnl.gov>
	<CAGEyGovaeAX1PFQOHnO=CMz-uDxpVC3RMTKzxSxyKFQ5NpbOZg@mail.gmail.com>
	<CAPpdu9Et+9=M+s=x98qm845ZT9pMWRCMUMNXFjEBzFw8gPL17A@mail.gmail.com>
Message-ID: <CAJ6bFK0nPoS6P9oq+vwMRTXG7wo-L-A2od_uaBKmW1XxaOWk1g@mail.gmail.com>

Sure. Take a look at securityonion or do it yourself. Works with pfring and
(soon)  afpacket.

On Fri, Oct 16, 2015, 20:03 Jason Williams <jwilliams at emergingthreats.net>
wrote:

> I've had instances where i have used zbalance_ipc to help load balance
> larger links for moloch. You can also use zbalance_ipc to create duplicate
> zc streams that you can attach different processes to. I run bro / suri on
> the same interface and haven't seen issues in operation.
>
> On Fri, Oct 16, 2015 at 11:16 AM, Chris Williams <cw13 at umbc.edu> wrote:
>
>> Is it possible to do this with multiple instances of pf_ring?
>> On Oct 16, 2015 12:13 PM, "Donaldson, John" <donaldson8 at llnl.gov> wrote:
>>
>>> Vito,
>>>
>>> We're running Bro and Snort in parallel, but we're using DAG cards to
>>> duplicate streams to Bro and Snort processes, so our performance
>>> characteristics are a bit different. In general, though, it really
>>> depends
>>> on how you manage the traffic that you're throwing at both, and how many
>>> rules you have enabled in Snort. It *is* possible to keep packet loss
>>> manageable, running them in parallel, but you'll have to trim down what
>>> you have Snort running.
>>>
>>>
>>> John Donaldson
>>>
>>>
>>>
>>> On 10/16/15, 10:31 AM, "bro-bounces at bro.org on behalf of Vito Logrillo"
>>> <bro-bounces at bro.org on behalf of vitologrillo at gmail.com> wrote:
>>>
>>> >Hi all,
>>> >Anyone have used Bro and Snort together to the same live traffic?
>>> >If yes, any suggestion?
>>> >For example, is it possible to send the same traffic to snort and bro
>>> >without packet loss?
>>> >Thanks
>>> >_______________________________________________
>>> >Bro mailing list
>>> >bro at bro-ids.org
>>> >http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>
>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151016/6445e053/attachment.html 

From daniel.guerra69 at gmail.com  Sat Oct 17 11:19:41 2015
From: daniel.guerra69 at gmail.com (Daniel Guerra)
Date: Sat, 17 Oct 2015 20:19:41 +0200
Subject: [Bro] broker compile problem
In-Reply-To: <369532ED-A52A-4BE8-85BB-AA237E92C029@illinois.edu>
References: <1E3CEB14-9885-469C-85F5-4A8943272753@gmail.com>
	<369532ED-A52A-4BE8-85BB-AA237E92C029@illinois.edu>
Message-ID: <2352C02E-A4A2-41D8-B6C0-2AF5AD703B2B@gmail.com>

It is a rocksdb thing. It needs all kind of libs related to zip.

> On 16 Oct 2015, at 16:17, Siwek, Jon <jsiwek at illinois.edu> wrote:
> 
> 
>> On Oct 14, 2015, at 6:39 PM, Daniel Guerra <daniel.guerra69 at gmail.com> wrote:
>> 
>> I have been trying over and over to solve this but failed.
>> I keep getting this error when compiling bro
>> 
>> ../libbroker.so.0.4-9.0: undefined reference to `inflate?
> 
> Is zlib installed and in a path the linker searches?
> 
> - Jon



From daniel.guerra69 at gmail.com  Sat Oct 17 16:08:12 2015
From: daniel.guerra69 at gmail.com (Daniel Guerra)
Date: Sun, 18 Oct 2015 01:08:12 +0200
Subject: [Bro] broker compile problem
In-Reply-To: <369532ED-A52A-4BE8-85BB-AA237E92C029@illinois.edu>
References: <1E3CEB14-9885-469C-85F5-4A8943272753@gmail.com>
	<369532ED-A52A-4BE8-85BB-AA237E92C029@illinois.edu>
Message-ID: <10BD8872-032D-409F-BD30-3E6B0DC13C85@gmail.com>

Without installing rocksdb everything compiles perfect.
Sorry Mark .. :D

> On 16 Oct 2015, at 16:17, Siwek, Jon <jsiwek at illinois.edu> wrote:
> 
> 
>> On Oct 14, 2015, at 6:39 PM, Daniel Guerra <daniel.guerra69 at gmail.com> wrote:
>> 
>> I have been trying over and over to solve this but failed.
>> I keep getting this error when compiling bro
>> 
>> ../libbroker.so.0.4-9.0: undefined reference to `inflate?
> 
> Is zlib installed and in a path the linker searches?
> 
> - Jon



From vallentin at icir.org  Sun Oct 18 09:20:40 2015
From: vallentin at icir.org (Matthias Vallentin)
Date: Sun, 18 Oct 2015 18:20:40 +0200
Subject: [Bro] Bro and Snort together
In-Reply-To: <CANdYiFH=JkijtYShJO5Te4D+vjFz9jXYeCenZ0dvVdv9iPx4vQ@mail.gmail.com>
References: <CANdYiFH=JkijtYShJO5Te4D+vjFz9jXYeCenZ0dvVdv9iPx4vQ@mail.gmail.com>
Message-ID: <20151018162040.GH16537@ninja.local>

> Anyone have used Bro and Snort together to the same live traffic?

You could give packet-bricks a shot:

    https://github.com/bro/packet-bricks

It requires netmap, however. You'd use a Duplicator brick to split up
the traffic over two pipes.

    Matthias

From joshi.pradyumna at gmail.com  Sun Oct 18 22:31:44 2015
From: joshi.pradyumna at gmail.com (Pradyumna Joshi)
Date: Mon, 19 Oct 2015 11:01:44 +0530
Subject: [Bro] Monitoring of intra virtual machines network traffic on same
	physical host
Message-ID: <CA+wJ6zbDYefaq-D=KJVTb=6gAtWFfgGt8CAZ5SUuhrD8bmCgpA@mail.gmail.com>

Is it possible to monitor network traffic between different Virtual
machines on the same physical machine using Bro?

Thanks.
Joshi Pradyumna
Computer Center,
Homi Bhabha National Institute,
Mumbai.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151019/d2e5c0ee/attachment.html 

From daniel.guerra69 at gmail.com  Mon Oct 19 03:08:33 2015
From: daniel.guerra69 at gmail.com (Daniel Guerra)
Date: Mon, 19 Oct 2015 12:08:33 +0200
Subject: [Bro] broker compile problem
In-Reply-To: <369532ED-A52A-4BE8-85BB-AA237E92C029@illinois.edu>
References: <1E3CEB14-9885-469C-85F5-4A8943272753@gmail.com>
	<369532ED-A52A-4BE8-85BB-AA237E92C029@illinois.edu>
Message-ID: <25253BAF-B683-4BA7-BEC3-AA7887847742@gmail.com>

Problem solved, rocksdb works too?
In debian I needed zlib1g-dev libsnappy-dev & libbz2-dev.

https://github.com/facebook/rocksdb/blob/master/INSTALL.md <https://github.com/facebook/rocksdb/blob/master/INSTALL.md>

> On 16 Oct 2015, at 16:17, Siwek, Jon <jsiwek at illinois.edu> wrote:
> 
> 
>> On Oct 14, 2015, at 6:39 PM, Daniel Guerra <daniel.guerra69 at gmail.com> wrote:
>> 
>> I have been trying over and over to solve this but failed.
>> I keep getting this error when compiling bro
>> 
>> ../libbroker.so.0.4-9.0: undefined reference to `inflate?
> 
> Is zlib installed and in a path the linker searches?
> 
> - Jon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151019/9ceef258/attachment.html 

From shane.castle at gmail.com  Mon Oct 19 03:27:41 2015
From: shane.castle at gmail.com (Shane Castle)
Date: Mon, 19 Oct 2015 12:27:41 +0200
Subject: [Bro] Monitoring of intra virtual machines network traffic on
 same physical host
In-Reply-To: <CA+wJ6zbDYefaq-D=KJVTb=6gAtWFfgGt8CAZ5SUuhrD8bmCgpA@mail.gmail.com>
References: <CA+wJ6zbDYefaq-D=KJVTb=6gAtWFfgGt8CAZ5SUuhrD8bmCgpA@mail.gmail.com>
Message-ID: <5624C59D.2010308@gmail.com>

This is really a specific version of the more general question, is it possible
to sniff traffic between VMs on the same physical host? The answer is, it
depends: what virtualization tool is being used? If it's VMware vSphere, then
yes:
https://blogs.vmware.com/vsphere/2011/08/vsphere-5-new-networking-features-port-mirroring.html

I don't know of this capability existing on any other virtualization platform.

On 19.10.2015 07:31, Pradyumna Joshi wrote:
> Is it possible to monitor network traffic between different Virtual machines on
> the same physical machine using Bro?


-- 
Mit besten Gr??en
Shane Castle

From joshi.pradyumna at gmail.com  Mon Oct 19 09:43:26 2015
From: joshi.pradyumna at gmail.com (Pradyumna Joshi)
Date: Mon, 19 Oct 2015 22:13:26 +0530
Subject: [Bro] Monitoring of intra virtual machines network traffic on
 same physical host
In-Reply-To: <A8E19DCE-002E-4739-9105-003B90779B36@lbl.gov>
References: <CA+wJ6zbDYefaq-D=KJVTb=6gAtWFfgGt8CAZ5SUuhrD8bmCgpA@mail.gmail.com>
	<A8E19DCE-002E-4739-9105-003B90779B36@lbl.gov>
Message-ID: <CA+wJ6zb-Uma6sBU0Bkr92KbeUS5EKd8-EAC+wBMHm+JD7KHs8g@mail.gmail.com>

Thanks Aashish for the quick response.

Your response has provided one more option for me - to run workers on VM
instances and run manager on Host.

I was thinking of using multiple options and was not sure which one to go
for:

1) Using Daemonlogger <http://sourceforge.net/projects/daemonlogger/> for
capturing traffic from bridged interfaces and feeding this traffic to Bro.
2) Using OpenvSwitch <http://openvswitch.org/> to achieve bridge
functionality and feed it to Bro. From the docs, it is seen that OVSDB
<https://tools.ietf.org/html/rfc7047> supports full virtual switch
management functionality.

I wanted to know if anybody in Bro Community had implemented similar
solutions and wanted to know their experiences/feedback.

regards,
- Pradyumna Joshi



On Mon, Oct 19, 2015 at 12:53 PM, Aashish Sharma <asharma at lbl.gov> wrote:

> Hello
>
> (Let me think some more on this)
>
> Meanwhile a quick solution is to run bro instances as worker nodes on each
> of the VM's and then run manager on the host OS.
>
> I don't anticipate that you'd have such high volumes that bro workers will
> demand more CPU then your applications on the VM.
>
> However, this is a quick and somewhat in optimal solution.  Would
> certainly work but may be cheaper (in CPU) to do it a different way.
>
> Basically bro needs to see traffic to and from each of the interfaces in
> the VM.
>
> Let me see if you can tap out of bridged interfaces or if our network/tap
> experts have some other ideas or workaround for this.
>
> Aashish
>
> > On Oct 18, 2015, at 10:31 PM, Pradyumna Joshi <joshi.pradyumna at gmail.com>
> wrote:
> >
> > Is it possible to monitor network traffic between different Virtual
> machines on the same physical machine using Bro?
> >
> > Thanks.
> > Joshi Pradyumna
> > Computer Center,
> > Homi Bhabha National Institute,
> > Mumbai.
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 
Pradyumna Joshi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151019/dbc7b0c3/attachment.html 

From dopheide at gmail.com  Mon Oct 19 10:34:16 2015
From: dopheide at gmail.com (Mike Dopheide)
Date: Mon, 19 Oct 2015 12:34:16 -0500
Subject: [Bro] Monitoring of intra virtual machines network traffic on
 same physical host
In-Reply-To: <CA+wJ6zb-Uma6sBU0Bkr92KbeUS5EKd8-EAC+wBMHm+JD7KHs8g@mail.gmail.com>
References: <CA+wJ6zbDYefaq-D=KJVTb=6gAtWFfgGt8CAZ5SUuhrD8bmCgpA@mail.gmail.com>
	<A8E19DCE-002E-4739-9105-003B90779B36@lbl.gov>
	<CA+wJ6zb-Uma6sBU0Bkr92KbeUS5EKd8-EAC+wBMHm+JD7KHs8g@mail.gmail.com>
Message-ID: <CAPy2kFYG3LsDACbqLnG6Gj83xYQrGJxgFV5MYp11LbDgiRH+Og@mail.gmail.com>

Adding a little to this, we just started playing with running Bro on a VM
to monitor VM-to-VM traffic on a HP bladesystem running VMWare using the
port mirroring that Shane mentioned.   It's going well enough that I'm
considering deploying it on all the other bladesystems as well.

Actually, we're using it as a monitoring point for VM to non-VM traffic as
well since it sees everything coming in-out of the chassis as well.

-Dop

On Mon, Oct 19, 2015 at 11:43 AM, Pradyumna Joshi <joshi.pradyumna at gmail.com
> wrote:

> Thanks Aashish for the quick response.
>
> Your response has provided one more option for me - to run workers on VM
> instances and run manager on Host.
>
> I was thinking of using multiple options and was not sure which one to go
> for:
>
> 1) Using Daemonlogger <http://sourceforge.net/projects/daemonlogger/> for
> capturing traffic from bridged interfaces and feeding this traffic to Bro.
> 2) Using OpenvSwitch <http://openvswitch.org/> to achieve bridge
> functionality and feed it to Bro. From the docs, it is seen that OVSDB
> <https://tools.ietf.org/html/rfc7047> supports full virtual switch
> management functionality.
>
> I wanted to know if anybody in Bro Community had implemented similar
> solutions and wanted to know their experiences/feedback.
>
> regards,
> - Pradyumna Joshi
>
>
>
> On Mon, Oct 19, 2015 at 12:53 PM, Aashish Sharma <asharma at lbl.gov> wrote:
>
>> Hello
>>
>> (Let me think some more on this)
>>
>> Meanwhile a quick solution is to run bro instances as worker nodes on
>> each of the VM's and then run manager on the host OS.
>>
>> I don't anticipate that you'd have such high volumes that bro workers
>> will demand more CPU then your applications on the VM.
>>
>> However, this is a quick and somewhat in optimal solution.  Would
>> certainly work but may be cheaper (in CPU) to do it a different way.
>>
>> Basically bro needs to see traffic to and from each of the interfaces in
>> the VM.
>>
>> Let me see if you can tap out of bridged interfaces or if our network/tap
>> experts have some other ideas or workaround for this.
>>
>> Aashish
>>
>>
>> > On Oct 18, 2015, at 10:31 PM, Pradyumna Joshi <
>> joshi.pradyumna at gmail.com> wrote:
>> >
>> > Is it possible to monitor network traffic between different Virtual
>> machines on the same physical machine using Bro?
>> >
>> > Thanks.
>> > Joshi Pradyumna
>> > Computer Center,
>> > Homi Bhabha National Institute,
>> > Mumbai.
>> > _______________________________________________
>> > Bro mailing list
>> > bro at bro-ids.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>
>
> --
> Pradyumna Joshi
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151019/2e00addd/attachment.html 

From asharma at lbl.gov  Mon Oct 19 10:43:11 2015
From: asharma at lbl.gov (Aashish Sharma)
Date: Mon, 19 Oct 2015 10:43:11 -0700
Subject: [Bro] Monitoring of intra virtual machines network traffic on
 same physical host
In-Reply-To: <CA+wJ6zb-Uma6sBU0Bkr92KbeUS5EKd8-EAC+wBMHm+JD7KHs8g@mail.gmail.com>
References: <CA+wJ6zbDYefaq-D=KJVTb=6gAtWFfgGt8CAZ5SUuhrD8bmCgpA@mail.gmail.com>
	<A8E19DCE-002E-4739-9105-003B90779B36@lbl.gov>
	<CA+wJ6zb-Uma6sBU0Bkr92KbeUS5EKd8-EAC+wBMHm+JD7KHs8g@mail.gmail.com>
Message-ID: <20151019174308.GD28723@yaksha.lbl.gov>

I think openswitch and port mirroring that Shane mentioned look like very promising options. Much better than clusterify the virtual machines. 

While, I haven't run bro on VM systems, I would be very interested in the performance numbers, if any of you have those in future, please do share. 

Thanks, 
Aashish 

On Mon, Oct 19, 2015 at 10:13:26PM +0530, Pradyumna Joshi wrote:
> 
>    Thanks Aashish for the quick response.
>    Your response has provided one more option for me - to run workers on VM
>    instances and run manager on Host.
>    I was thinking of using multiple options and was not sure which one to go
>    for:
>    1) Using [1]Daemonlogger for capturing traffic from bridged interfaces and
>    feeding this traffic to Bro.
>    2) Using [2]OpenvSwitch to achieve bridge functionality and feed it to Bro.
>    From  the  docs, it is seen that [3]OVSDB supports full virtual switch
>    management functionality.
>    I  wanted  to know if anybody in Bro Community had implemented similar
>    solutions and wanted to know their experiences/feedback.
>    regards,
>    - Pradyumna Joshi
>    ?
> 
>    On Mon, Oct 19, 2015 at 12:53 PM, Aashish Sharma <[4]asharma at lbl.gov> wrote:
> 
>      Hello
>      (Let me think some more on this)
>      Meanwhile a quick solution is to run bro instances as worker nodes on each
>      of the VM's and then run manager on the host OS.
>      I don't anticipate that you'd have such high volumes that bro workers will
>      demand more CPU then your applications on the VM.
>      However,  this is a quick and somewhat in optimal solution.?  Would
>      certainly work but may be cheaper (in CPU) to do it a different way.
>      Basically bro needs to see traffic to and from each of the interfaces in
>      the VM.
>      Let me see if you can tap out of bridged interfaces or if our network/tap
>      experts have some other ideas or workaround for this.
>      Aashish
> 
>    >    On    Oct    18,    2015,    at   10:31   PM,   Pradyumna   Joshi
>    <[5]joshi.pradyumna at gmail.com> wrote:
>    >
>    >  Is it possible to monitor network traffic between different Virtual
>    machines on the same physical machine using Bro?
>    >
>    > Thanks.
>    > Joshi Pradyumna
>    > Computer Center,
>    > Homi Bhabha National Institute,
>    > Mumbai.
> 
>      > _______________________________________________
>      > Bro mailing list
>      > [6]bro at bro-ids.org
>      > [7]http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
>    --
>    Pradyumna Joshi
> 
> References
> 
>    1. http://sourceforge.net/projects/daemonlogger/
>    2. http://openvswitch.org/
>    3. https://tools.ietf.org/html/rfc7047
>    4. mailto:asharma at lbl.gov
>    5. mailto:joshi.pradyumna at gmail.com
>    6. mailto:bro at bro-ids.org
>    7. http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From ali_cancerian786 at hotmail.com  Tue Oct 20 00:16:30 2015
From: ali_cancerian786 at hotmail.com (ali abbas)
Date: Tue, 20 Oct 2015 12:16:30 +0500
Subject: [Bro] Bro IDS anomaly detection
Message-ID: <SNT147-W2969EAB1370534BD074004B3390@phx.gbl>

Dear All.I am working in the area of Anomaly detection. I am interested in understanding the existing mechanism implemented in BRO. Please refer me some useful material and/or research papers, especialy how it is different than SNORT. Thanks
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151020/105eab1a/attachment.html 

From ali.abbas.hussain90 at gmail.com  Tue Oct 20 00:52:25 2015
From: ali.abbas.hussain90 at gmail.com (Syed Muhammad Ali Abbas)
Date: Tue, 20 Oct 2015 12:52:25 +0500
Subject: [Bro] mailing list for bro
Message-ID: <CA+T3tkJy5FxrwA0civ_khmxrLzQ2N9qJ-sQpsrYNiYQnqcX0gg@mail.gmail.com>

ali_cancerian786 at hotmail.com
ali.abbas.hussain90 at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151020/9a970684/attachment.html 

From ali.abbas.hussain90 at gmail.com  Tue Oct 20 00:56:25 2015
From: ali.abbas.hussain90 at gmail.com (Syed Muhammad Ali Abbas)
Date: Tue, 20 Oct 2015 12:56:25 +0500
Subject: [Bro] BRO IDS Anomaly detection
Message-ID: <CA+T3tkLywfCi=X=MgKiNH+1jrp1kMNN2-+mXOHi-9PuKBTYDwQ@mail.gmail.com>

dear all.
I am working in the area of Anomaly detection. I am interested in
understanding the existing mechanism implemented in BRO.

Please refer me some useful material and/or research papers, especialy how
it is different than SNORT.

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151020/f047a645/attachment.html 

From ali.abbas.hussain90 at gmail.com  Tue Oct 20 00:58:56 2015
From: ali.abbas.hussain90 at gmail.com (Syed Muhammad Ali Abbas)
Date: Tue, 20 Oct 2015 12:58:56 +0500
Subject: [Bro] Mailing list
Message-ID: <CA+T3tkKaJwFbKwNWRTgBg7erNuObsxmGJV55DWLjEv3M0qbWmA@mail.gmail.com>

bro at bro-ids.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151020/63a894a8/attachment.html 

From ali.abbas.hussain90 at gmail.com  Tue Oct 20 01:02:40 2015
From: ali.abbas.hussain90 at gmail.com (Syed Muhammad Ali Abbas)
Date: Tue, 20 Oct 2015 13:02:40 +0500
Subject: [Bro] BRO anomaly detection
Message-ID: <CA+T3tkLuS9OeiwduMtnrC0yimfQq+n0-NzxyY_J-YqMgUg4+RQ@mail.gmail.com>

Dear All
I am working in the area of Anomaly detection. I am interested in
understanding the existing mechanism implemented in BRO.

Please refer me some useful material and/or research papers, especialy how
it is different than SNORT.

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151020/20b6a780/attachment.html 

From ali.abbas.hussain90 at gmail.com  Tue Oct 20 02:57:07 2015
From: ali.abbas.hussain90 at gmail.com (Syed Muhammad Ali Abbas)
Date: Tue, 20 Oct 2015 14:57:07 +0500
Subject: [Bro] BRO anomaly detection
In-Reply-To: <CA+T3tkLuS9OeiwduMtnrC0yimfQq+n0-NzxyY_J-YqMgUg4+RQ@mail.gmail.com>
References: <CA+T3tkLuS9OeiwduMtnrC0yimfQq+n0-NzxyY_J-YqMgUg4+RQ@mail.gmail.com>
Message-ID: <CA+T3tkKkX78NwPBna-fTsyafrnbT7QO-eg_tgHznKcLdwzRG3w@mail.gmail.com>

any updates?

On Tue, Oct 20, 2015 at 1:02 PM, Syed Muhammad Ali Abbas <
ali.abbas.hussain90 at gmail.com> wrote:

> Dear All
> I am working in the area of Anomaly detection. I am interested in
> understanding the existing mechanism implemented in BRO.
>
> Please refer me some useful material and/or research papers, especialy how
> it is different than SNORT.
>
> Thanks
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151020/605764a3/attachment.html 

From thomastan81 at gmail.com  Tue Oct 20 04:00:48 2015
From: thomastan81 at gmail.com (Thomas Tan)
Date: Tue, 20 Oct 2015 13:00:48 +0200
Subject: [Bro] OS Fingerprinting
Message-ID: <CANxb75DopfK6wU8fPrAYOdrkTO3B=Dgg4GeTS4+3m5FQPaRbMg@mail.gmail.com>

Hi All,

I am writing an extension of Operating System Fingerprinting for Bro, and
want to use the OS signatures generated (NOT the p0f fingerprint file) by
Bro in my component for classification.

According to my best knowledge, the following two events can help collect
almost all the fields of an OS signature.

1) event tcp_option (c:connection, is_orig:bool, opt:count, optlen:count)
2) event connection_SYN_packet(c:connection, pkt:SYN_packet)

However, they are two separate events, and extra efforts is required to
construct an accurate OS signature.

Just wondering if there is an event that can do it at one go?

Thank you in advance.

Best regards,

Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151020/4ce8e0d5/attachment.html 

From jdopheid at illinois.edu  Tue Oct 20 06:26:51 2015
From: jdopheid at illinois.edu (Dopheide, Jeannette M)
Date: Tue, 20 Oct 2015 13:26:51 +0000
Subject: [Bro] BRO anomaly detection
In-Reply-To: <CA+T3tkKkX78NwPBna-fTsyafrnbT7QO-eg_tgHznKcLdwzRG3w@mail.gmail.com>
References: <CA+T3tkLuS9OeiwduMtnrC0yimfQq+n0-NzxyY_J-YqMgUg4+RQ@mail.gmail.com>
	<CA+T3tkKkX78NwPBna-fTsyafrnbT7QO-eg_tgHznKcLdwzRG3w@mail.gmail.com>
Message-ID: <D24BAACC.1A685%jdopheid@illinois.edu>

Hello Syed,

We maintain a list of research papers involving Bro. You can find them here:
https://www.bro.org/research/index.html

Thanks,
Jeannette

------
Jeannette Dopheide
Bro Outreach Coordinator
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign


From: <bro-bounces at bro.org<mailto:bro-bounces at bro.org>> on behalf of Syed Muhammad Ali Abbas <ali.abbas.hussain90 at gmail.com<mailto:ali.abbas.hussain90 at gmail.com>>
Date: Tuesday, October 20, 2015 at 4:57 AM
To: "bro at bro-ids.org<mailto:bro at bro-ids.org>" <bro at bro-ids.org<mailto:bro at bro-ids.org>>
Subject: Re: [Bro] BRO anomaly detection

any updates?

On Tue, Oct 20, 2015 at 1:02 PM, Syed Muhammad Ali Abbas <ali.abbas.hussain90 at gmail.com<mailto:ali.abbas.hussain90 at gmail.com>> wrote:
Dear All
I am working in the area of Anomaly detection. I am interested in understanding the existing mechanism implemented in BRO.

Please refer me some useful material and/or research papers, especialy how it is different than SNORT.

Thanks

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151020/2bd63a49/attachment-0001.html 

From albert.zaharovits at gmail.com  Tue Oct 20 07:49:24 2015
From: albert.zaharovits at gmail.com (Albert Zaharovits)
Date: Tue, 20 Oct 2015 17:49:24 +0300
Subject: [Bro] snabbswitch
Message-ID: <C7EAD656-671C-4FFA-A772-BFC617BB6201@gmail.com>

Hi guys,

Can you please highlight the differences between [1] and [2]?
[1]: https://github.com/bro/packet-bricks <https://github.com/bro/packet-bricks>
[2]: https://github.com/SnabbCo/snabbswitch <https://github.com/SnabbCo/snabbswitch>

Any insights would be greatly appreciated!

Thank you,
Albert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151020/0bcc2ed8/attachment.html 

From masoom.alam at gmail.com  Tue Oct 20 10:02:48 2015
From: masoom.alam at gmail.com (masoom alam)
Date: Tue, 20 Oct 2015 10:02:48 -0700
Subject: [Bro] Bro IDS anomaly detection
In-Reply-To: <SNT147-W2969EAB1370534BD074004B3390@phx.gbl>
References: <SNT147-W2969EAB1370534BD074004B3390@phx.gbl>
Message-ID: <CAPihp9=v_F3gwz+O-HrYVcDbx1ZBOJvKuAJFTKMffT3UrA-3Ag@mail.gmail.com>

Actually there is a lot of work in the literature on anomaly detection
using snort but in bro there is no or less work.....whether the existing
bro scripts are enough for network intrusion detection.....where u think
enhancements can be made? A sans document on finding web application attack
using bro scripting.....what do u think if bro logs are used for anomaly
detection.... Any work already done in this direction?
On Oct 20, 2015 12:19 PM, "ali abbas" <ali_cancerian786 at hotmail.com> wrote:

> Dear All.
>
> I am working in the area of Anomaly detection. I am interested in
> understanding the existing mechanism implemented in BRO.
>
> Please refer me some useful material and/or research papers, especialy how
> it is different than SNORT.
>
> Thanks
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151020/26eaf4f8/attachment.html 

From cw13 at umbc.edu  Wed Oct 21 09:16:34 2015
From: cw13 at umbc.edu (Chris Williams)
Date: Wed, 21 Oct 2015 12:16:34 -0400
Subject: [Bro] Help with Bro & ES
Message-ID: <CAGEyGospsVvuMzp22ASG35ZNrgfba6KXkYkjaBHAmTAOz-YWEA@mail.gmail.com>

I recently installed Bro, and I am trying to get it to work with elastic
search (with Kibana as a front end.) I have alerts getting to ES and it
shows up in Kibana, but it is a mix of unintelligible json messages. For
example, some don't have timestamps:











*{  "_index": "bro-201510191500",  "_type": "loaded_scripts",  "_id":
"AVCBw07WiyISA4W_6X0j",  "_score": 1,  "_source": {    "name": "
/usr/local/bro/share/bro/base/bif/bro.bif.bro"  }}*






Can anyone provide guidance, or suggest resources on organizing and sorting
alerts/messages such that I can use it with ES/Kibana? I am not sure if I
missed something in configuration and I am having a tough time finding
resources online for further suggestions.

Thanks!

Chris Williams
---
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151021/f790f692/attachment.html 

From jazoff at illinois.edu  Wed Oct 21 09:28:25 2015
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Wed, 21 Oct 2015 16:28:25 +0000
Subject: [Bro] Help with Bro & ES
In-Reply-To: <CAGEyGospsVvuMzp22ASG35ZNrgfba6KXkYkjaBHAmTAOz-YWEA@mail.gmail.com>
References: <CAGEyGospsVvuMzp22ASG35ZNrgfba6KXkYkjaBHAmTAOz-YWEA@mail.gmail.com>
Message-ID: <FEDF3156-964A-470D-9FEF-256F182D17EF@illinois.edu>


> On Oct 21, 2015, at 12:16 PM, Chris Williams <cw13 at umbc.edu> wrote:
> 
> I recently installed Bro, and I am trying to get it to work with elastic search (with Kibana as a front end.) I have alerts getting to ES and it shows up in Kibana, but it is a mix of unintelligible json messages. For example, some don't have timestamps:

...

>   "_type": "loaded_scripts",
>   

The loaded_scripts.log is 'special' and does not have timestamps.  How do entries from things like the conn.log or http.log look?


-- 
- Justin Azoff



From cw13 at umbc.edu  Wed Oct 21 09:32:27 2015
From: cw13 at umbc.edu (Chris Williams)
Date: Wed, 21 Oct 2015 12:32:27 -0400
Subject: [Bro] Help with Bro & ES
In-Reply-To: <FEDF3156-964A-470D-9FEF-256F182D17EF@illinois.edu>
References: <CAGEyGospsVvuMzp22ASG35ZNrgfba6KXkYkjaBHAmTAOz-YWEA@mail.gmail.com>
	<FEDF3156-964A-470D-9FEF-256F182D17EF@illinois.edu>
Message-ID: <CAGEyGotRZ5-tMybrRw-Jt_f6Kb5b3YZJ+r6tvYJ15Djb+Vj08g@mail.gmail.com>

I mean... I think they look ok. Again, I understand that I have to learn
how to organize the information in such a way that will make sense. This is
an example of a conn message:































*{  "_index": "bro-201510191500",  "_type": "conn",  "_id":
"AVCBxqIWiyISA4W_6X6I",  "_score": 1,  "_source": {    "ts":
1445286221580,    "uid": "CG7qWz2Xgs7J8LcO5d",    "id.orig_h":
"*.*.9.119",    "id.orig_p": 123,    "id.resp_h": "*.*.1.3",
"id.resp_p": 123,    "proto": "udp",    "duration": 0.0002,
"orig_bytes": 0,    "resp_bytes": 48,    "conn_state": "SHR",
"local_orig": false,    "local_resp": false,    "missed_bytes": 0,
"history": "Cd",    "orig_pkts": 0,    "orig_ip_bytes": 0,    "resp_pkts":
1,    "resp_ip_bytes": 76,    "tunnel_parents": []  }}*


Chris Williams
---
BS Information Systems - '16
CWIT Y2 Cyber Scholar <http://cybersecurity.umbc.edu/cyberscholars/>
Work: 5-0933
Cell: (202) 596-5406

"If you think technology can solve your security problems, then you don't
understand the problems and you don't understand the technology." Bruce
Schneier


On Wed, Oct 21, 2015 at 12:28 PM, Azoff, Justin S <jazoff at illinois.edu>
wrote:

>
> > On Oct 21, 2015, at 12:16 PM, Chris Williams <cw13 at umbc.edu> wrote:
> >
> > I recently installed Bro, and I am trying to get it to work with elastic
> search (with Kibana as a front end.) I have alerts getting to ES and it
> shows up in Kibana, but it is a mix of unintelligible json messages. For
> example, some don't have timestamps:
>
> ...
>
> >   "_type": "loaded_scripts",
> >
>
> The loaded_scripts.log is 'special' and does not have timestamps.  How do
> entries from things like the conn.log or http.log look?
>
>
> --
> - Justin Azoff
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151021/533dac07/attachment.html 

From daniel.guerra69 at gmail.com  Wed Oct 21 11:20:27 2015
From: daniel.guerra69 at gmail.com (Daniel Guerra)
Date: Wed, 21 Oct 2015 20:20:27 +0200
Subject: [Bro] Help with Bro & ES
In-Reply-To: <CAGEyGotRZ5-tMybrRw-Jt_f6Kb5b3YZJ+r6tvYJ15Djb+Vj08g@mail.gmail.com>
References: <CAGEyGospsVvuMzp22ASG35ZNrgfba6KXkYkjaBHAmTAOz-YWEA@mail.gmail.com>
	<FEDF3156-964A-470D-9FEF-256F182D17EF@illinois.edu>
	<CAGEyGotRZ5-tMybrRw-Jt_f6Kb5b3YZJ+r6tvYJ15Djb+Vj08g@mail.gmail.com>
Message-ID: <B15DC30F-1FA6-4848-BC2A-A20281C0B051@gmail.com>

I?m working on elastic for a while
I changed a bit on the elasticsearch source,
recompile after doing this.
sed -i "s/JSON::TS_MILLIS/JSON::TS_ISO8601/g" bro/aux/plugins/elasticsearch/src/ElasticSearch.cc <http://elasticsearch.cc/>

my conn.log looks like this in kibana check ts difference

{
  "_index": "bro-201509160700",
  "_type": "conn",
  "_id": "AVCLfROKixyabuRJCOlt",
  "_score": null,
  "_source": {
    "_timestamp": 1442388234879,
    "ts": "2015-09-16T07:16:54.185442Z",
    "uid": "Cv7R6a19zHzfu1H6U4",
    "id.orig_h": "192.168.1.122",
    "id.orig_p": 49428,
    "id.resp_h": "192.168.102.97",
    "id.resp_p": 514,
    "proto": "udp",
    "duration": 360.241984,
    "orig_bytes": 32096,
    "resp_bytes": 0,
    "conn_state": "S0",
    "missed_bytes": 0,
    "history": "D",
    "orig_pkts": 191,
    "orig_ip_bytes": 37444,
    "resp_pkts": 0,
    "resp_ip_bytes": 0,
    "tunnel_parents": []
  }

The next chalenge are coordinates from geoip ..

> On 21 Oct 2015, at 18:32, Chris Williams <cw13 at umbc.edu> wrote:
> 
> I mean... I think they look ok. Again, I understand that I have to learn how to organize the information in such a way that will make sense. This is an example of a conn message:
> 
> {
>   "_index": "bro-201510191500",
>   "_type": "conn",
>   "_id": "AVCBxqIWiyISA4W_6X6I",
>   "_score": 1,
>   "_source": {
>     "ts": 1445286221580,
>     "uid": "CG7qWz2Xgs7J8LcO5d",
>     "id.orig_h": "*.*.9.119",
>     "id.orig_p": 123,
>     "id.resp_h": "*.*.1.3",
>     "id.resp_p": 123,
>     "proto": "udp",
>     "duration": 0.0002,
>     "orig_bytes": 0,
>     "resp_bytes": 48,
>     "conn_state": "SHR",
>     "local_orig": false,
>     "local_resp": false,
>     "missed_bytes": 0,
>     "history": "Cd",
>     "orig_pkts": 0,
>     "orig_ip_bytes": 0,
>     "resp_pkts": 1,
>     "resp_ip_bytes": 76,
>     "tunnel_parents": []
>   }
> }
> 
> 
> 
> 
> Chris Williams 
> ---
> BS Information Systems - '16
> CWIT Y2 Cyber Scholar <http://cybersecurity.umbc.edu/cyberscholars/>
> Work: 5-0933
> Cell: (202) 596-5406
> 
> "If you think technology can solve your security problems, then you don't understand the problems and you don't understand the technology." Bruce Schneier
> 
> 
> On Wed, Oct 21, 2015 at 12:28 PM, Azoff, Justin S <jazoff at illinois.edu <mailto:jazoff at illinois.edu>> wrote:
> 
> > On Oct 21, 2015, at 12:16 PM, Chris Williams <cw13 at umbc.edu <mailto:cw13 at umbc.edu>> wrote:
> >
> > I recently installed Bro, and I am trying to get it to work with elastic search (with Kibana as a front end.) I have alerts getting to ES and it shows up in Kibana, but it is a mix of unintelligible json messages. For example, some don't have timestamps:
> 
> ...
> 
> >   "_type": "loaded_scripts",
> >
> 
> The loaded_scripts.log is 'special' and does not have timestamps.  How do entries from things like the conn.log or http.log look?
> 
> 
> --
> - Justin Azoff
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151021/c746e65f/attachment-0001.html 

From mfw113 at psu.edu  Thu Oct 22 05:12:54 2015
From: mfw113 at psu.edu (Mike Waite)
Date: Thu, 22 Oct 2015 08:12:54 -0400
Subject: [Bro] Memory Issue with Bro
In-Reply-To: <AE9E9388-EC16-4040-A8CD-3752E1E3E3A8@illinois.edu>
References: <e9c88f86c0b144d0a33afafa324f9555@ocex03.SolarFlarecom.com>
	<7E1F7811-8496-4F0F-A2BD-7D2FB419C6D2@illinois.edu>
	<69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com>
	<454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu>
	<7689d3f6ddb6454d855e6fd3a3b8ea7b@ocex03.SolarFlarecom.com>
	<3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org>
	<CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
	<DE5BED9E-D710-4763-A978-423D917509A5@illinois.edu>
	<CAMCSCoW6AisyP479iHN5pms+AcHPmFUAEdUvsJvhUPoRTNxD5w@mail.gmail.com>
	<AE9E9388-EC16-4040-A8CD-3752E1E3E3A8@illinois.edu>
Message-ID: <20151022121254.GA26216@redquill.sos.its.psu.edu>

I know we are still seeing issues with the manager child proccess.  The process will consume over 200GB of RAM in 8 hours.

--
Mike Waite
CyberSecurity Intrusion Analyst
Office of Information Security
The Pennsylvania State University
? 15-10-02 15:41:41, Azoff, Justin S <jazoff at illinois.edu>:
>Is it the manager parent or the child process?
>
>-- 
>- Justin Azoff
>
>> On Oct 2, 2015, at 12:55 PM, Joe Blow <blackhole.em at gmail.com> wrote:
>>
>> It's my manager processes using tons of memory...
>>
>> How would you suggest debugging the manager processes?
>>
>> Cheers,
>>
>> JB
>>
>> On Fri, Oct 2, 2015 at 12:21 PM, Azoff, Justin S <jazoff at illinois.edu> wrote:
>>
>> > On Sep 30, 2015, at 11:55 AM, Joe Blow <blackhole.em at gmail.com> wrote:
>> >
>> > I'm super interested in this thread, as I believe i'm experiencing the same memory leak, using the solarflare cards.
>> > i'm running a similar setup, with 20 workers and lots of traffic, but i'm having to bounce the entire NIC once Bro goes haywire.  Bro doesn't take too long before it's wiped the whole box out of memory (all 192GB).
>> >
>> > Please let me know how to troubleshooting goes.  I'm happy to provide logs.
>> >
>> > Cheers,
>> >
>> > JB
>>
>> Memory leaks are tricky.  It is important to make a distinction about what component is using a lot of memory:
>>
>> 1) the workers - analyzer issues and leaks in general would show up here.
>> 2) the proxies - communication related
>> 3) the manager - child - if the manager is overloaded the child will buffer log data
>> 4) the manager - parent - if a logging destination is overloaded the parent will buffer log writes
>>
>> If your manager processes are using a lot of ram, that doesn't have anything to do with the capture library in use.
>>
>> --
>> - Justin Azoff
>>
>>
>
>
>_______________________________________________
>Bro mailing list
>bro at bro-ids.org
>http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151022/afb0d442/attachment.bin 

From cw13 at umbc.edu  Thu Oct 22 06:39:25 2015
From: cw13 at umbc.edu (Chris Williams)
Date: Thu, 22 Oct 2015 09:39:25 -0400
Subject: [Bro] Help with Bro & ES
In-Reply-To: <B15DC30F-1FA6-4848-BC2A-A20281C0B051@gmail.com>
References: <CAGEyGospsVvuMzp22ASG35ZNrgfba6KXkYkjaBHAmTAOz-YWEA@mail.gmail.com>
	<FEDF3156-964A-470D-9FEF-256F182D17EF@illinois.edu>
	<CAGEyGotRZ5-tMybrRw-Jt_f6Kb5b3YZJ+r6tvYJ15Djb+Vj08g@mail.gmail.com>
	<B15DC30F-1FA6-4848-BC2A-A20281C0B051@gmail.com>
Message-ID: <CAGEyGos+ALZ+-jnb9Qs69UnExkS5+g9QmWo+aX-YaOEGP9um6w@mail.gmail.com>

Justin,

I understand that "loaded scripts" is special, but it is creating issues
with the index because there are no timestamps. Is there a recommended
method for handling these messages?
--
Daniel- my connlog looks the same with the exception of "ts" which I assume
you changed?
--

Thanks


On Wed, Oct 21, 2015 at 2:20 PM, Daniel Guerra <daniel.guerra69 at gmail.com>
wrote:

> I?m working on elastic for a while
> I changed a bit on the elasticsearch source,
> recompile after doing this.
> sed -i "s/JSON::TS_MILLIS/JSON::TS_ISO8601/g"
> bro/aux/plugins/elasticsearch/src/ElasticSearch.cc
> <http://elasticsearch.cc>
>
> my conn.log looks like this in kibana check ts difference
>
> {
>   "_index": "bro-201509160700",
>   "_type": "conn",
>   "_id": "AVCLfROKixyabuRJCOlt",
>   "_score": null,
>   "_source": {
>     "_timestamp": 1442388234879,
>     "ts": "2015-09-16T07:16:54.185442Z",
>     "uid": "Cv7R6a19zHzfu1H6U4",
>     "id.orig_h": "192.168.1.122",
>     "id.orig_p": 49428,
>     "id.resp_h": "192.168.102.97",
>     "id.resp_p": 514,
>     "proto": "udp",
>     "duration": 360.241984,
>     "orig_bytes": 32096,
>     "resp_bytes": 0,
>     "conn_state": "S0",
>     "missed_bytes": 0,
>     "history": "D",
>     "orig_pkts": 191,
>     "orig_ip_bytes": 37444,
>     "resp_pkts": 0,
>     "resp_ip_bytes": 0,
>     "tunnel_parents": []
>   }
>
> The next chalenge are coordinates from geoip ..
>
> On 21 Oct 2015, at 18:32, Chris Williams <cw13 at umbc.edu> wrote:
>
> I mean... I think they look ok. Again, I understand that I have to learn
> how to organize the information in such a way that will make sense. This is
> an example of a conn message:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *{  "_index": "bro-201510191500",  "_type": "conn",  "_id":
> "AVCBxqIWiyISA4W_6X6I",  "_score": 1,  "_source": {    "ts":
> 1445286221580,    "uid": "CG7qWz2Xgs7J8LcO5d",    "id.orig_h":
> "*.*.9.119",    "id.orig_p": 123,    "id.resp_h": "*.*.1.3",
> "id.resp_p": 123,    "proto": "udp",    "duration": 0.0002,
> "orig_bytes": 0,    "resp_bytes": 48,    "conn_state": "SHR",
> "local_orig": false,    "local_resp": false,    "missed_bytes": 0,
> "history": "Cd",    "orig_pkts": 0,    "orig_ip_bytes": 0,    "resp_pkts":
> 1,    "resp_ip_bytes": 76,    "tunnel_parents": []  }}*
>
>
> Chris Williams
> ---
> BS Information Systems - '16
> CWIT Y2 Cyber Scholar <http://cybersecurity.umbc.edu/cyberscholars/>
> Work: 5-0933
> Cell: (202) 596-5406
>
> "If you think technology can solve your security problems, then you don't
> understand the problems and you don't understand the technology." Bruce
> Schneier
>
>
> On Wed, Oct 21, 2015 at 12:28 PM, Azoff, Justin S <jazoff at illinois.edu>
> wrote:
>
>>
>> > On Oct 21, 2015, at 12:16 PM, Chris Williams <cw13 at umbc.edu> wrote:
>> >
>> > I recently installed Bro, and I am trying to get it to work with
>> elastic search (with Kibana as a front end.) I have alerts getting to ES
>> and it shows up in Kibana, but it is a mix of unintelligible json messages.
>> For example, some don't have timestamps:
>>
>> ...
>>
>> >   "_type": "loaded_scripts",
>> >
>>
>> The loaded_scripts.log is 'special' and does not have timestamps.  How do
>> entries from things like the conn.log or http.log look?
>>
>>
>> --
>> - Justin Azoff
>>
>>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151022/a8c88605/attachment.html 

From jazoff at illinois.edu  Thu Oct 22 07:22:18 2015
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 22 Oct 2015 14:22:18 +0000
Subject: [Bro] Memory Issue with Bro
In-Reply-To: <20151022121254.GA26216@redquill.sos.its.psu.edu>
References: <e9c88f86c0b144d0a33afafa324f9555@ocex03.SolarFlarecom.com>
	<7E1F7811-8496-4F0F-A2BD-7D2FB419C6D2@illinois.edu>
	<69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com>
	<454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu>
	<7689d3f6ddb6454d855e6fd3a3b8ea7b@ocex03.SolarFlarecom.com>
	<3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org>
	<CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
	<DE5BED9E-D710-4763-A978-423D917509A5@illinois.edu>
	<CAMCSCoW6AisyP479iHN5pms+AcHPmFUAEdUvsJvhUPoRTNxD5w@mail.gmail.com>
	<AE9E9388-EC16-4040-A8CD-3752E1E3E3A8@illinois.edu>
	<20151022121254.GA26216@redquill.sos.its.psu.edu>
Message-ID: <E7981188-57E2-4E65-8064-8E1A0FD24172@illinois.edu>

> On Oct 22, 2015, at 8:12 AM, Mike Waite <mfw113 at psu.edu> wrote:
>
> I know we are still seeing issues with the manager child proccess.  The process will consume over 200GB of RAM in 8 hours.
>

Give the attached patch a try.



Monitor by using

    cat logs/current/communication.log |egrep 'manager.child'

And check to see if pending=0 or at least not growing.


--
- Justin Azoff


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151022/4797d8e9/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: flare_fix.patch
Type: application/octet-stream
Size: 839 bytes
Desc: flare_fix.patch
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151022/4797d8e9/attachment.obj 

From vitologrillo at gmail.com  Thu Oct 22 07:44:42 2015
From: vitologrillo at gmail.com (Vito Logrillo)
Date: Thu, 22 Oct 2015 16:44:42 +0200
Subject: [Bro] Bro and Yara together
Message-ID: <CANdYiFGwMocc5q1oRQFXn_9E5SKPbqRRNq3JyDzzNiOQ3_xL0g@mail.gmail.com>

Hi all,
the git code below can be used to integrate Bro with Yara:
https://github.com/hempnall/broyara

I've tried to use it without any result. Should i download the files
in the same bro directory and then change the CMake files as
described?
Thanks

From lloyd_brown at byu.edu  Thu Oct 22 08:05:18 2015
From: lloyd_brown at byu.edu (Lloyd Brown)
Date: Thu, 22 Oct 2015 09:05:18 -0600
Subject: [Bro] Patterns and Word Boundaries
Message-ID: <5628FB2E.9050604@byu.edu>

Hopefully this isn't too simplistic of a question, but I'm just getting
started with Bro.

In the text pattern syntax for Bro [1], is there an easy way to define
word boundaries, similar to how some of the RegEx dialects use '\b',
'\<', '\>', etc.? [2]

I'm trying to match for specific strings in a data stream.  For example,
the word "nmap".  I'm trying several approaches, based on past RegEx
knowledge, and I'm having trouble coming up with a single pattern that
would handle it all.  Example bro test script attached; hopefully it's
clear.

Fundamentally, is there a syntax reference for pattern matching, or does
it conform to a commonly known dialect (eg. POSIX-style RegEx, or PCRE
RegEx)?


[1] https://www.bro.org/sphinx/scripting/index.html#pattern
[2] http://www.regular-expressions.info/wordboundaries.html

-- 
Lloyd Brown
Systems Administrator
Fulton Supercomputing Lab
Brigham Young University
http://marylou.byu.edu
-------------- next part --------------
event bro_init() {
	local testcases = set( 
		"nmap", 		#Should match something
		"test nmap", 		#Should match something
		"nmap test", 		#Should match something
		"test nmap test",	#should match something
		"unmapped_entries",	#Should NOT match any of the patterns
		"test\tnmap",		#Should match something
		"nmap\ttest",		#Should match something
		"test\tnmap\ttest"	#Should match something
		);
	local nmap_patterns = vector( 
		/ nmap /, 		#Works, but what if it's non-space whitespace, eg '\t'?
		/^nmap /, 		
		/ nmap$/, 
		/^nmap$/, 
		/\bnmap\b/, 		#doesn't seem to match word boundaries as expected
		/\<nmap\>/, 		#doesn't seem to match word boundaries as expected
		/[ \t]nmap$/,		#this works, but I have to anticipate which whitespace chars will be used
		/^nmap[ \t]/,		#this works, but I have to anticipate which whitespace chars will be used
		/[ \t]nmap[ \t]/	#this works, but I have to anticipate which whitespace chars will be used
		
		#I wanted to try this one involving negative lookahead and negative lookbehind, but it won't even compile
		#/(?<!\s)nmap(?>!\s)/	#probably won't work; not sure if \s means what I think, and negative lookarounds are hard to get right...
		);

	for (testcase in testcases) {
		print fmt("Testcase: \"%s\"", testcase);
		for (pi in nmap_patterns) {
			if ( nmap_patterns[pi] in testcase ) {
				print fmt("     Pattern: %s - Matched", nmap_patterns[pi]);
			} else {
				print fmt("     Pattern: %s - Did NOT match", nmap_patterns[pi]);
			}
		}
	}

}

From soehlert at es.net  Thu Oct 22 09:03:45 2015
From: soehlert at es.net (Samuel Oehlert)
Date: Thu, 22 Oct 2015 09:03:45 -0700
Subject: [Bro] Patterns and Word Boundaries
In-Reply-To: <5628FB2E.9050604@byu.edu>
References: <5628FB2E.9050604@byu.edu>
Message-ID: <CAMLD0uJaZx+1ReQ3vpSZdp2JtmBCR3NR3rj+XDL4+wJHTkDkPw@mail.gmail.com>

I know Bro's regex syntax is almost exactly the same as Flex (only
differing in some very edge cases). I am not positive, but from a cursory
google it seems Flex doesn't understand word boundaries.

-Sam

On Thu, Oct 22, 2015 at 8:05 AM, Lloyd Brown <lloyd_brown at byu.edu> wrote:

> Hopefully this isn't too simplistic of a question, but I'm just getting
> started with Bro.
>
> In the text pattern syntax for Bro [1], is there an easy way to define
> word boundaries, similar to how some of the RegEx dialects use '\b',
> '\<', '\>', etc.? [2]
>
> I'm trying to match for specific strings in a data stream.  For example,
> the word "nmap".  I'm trying several approaches, based on past RegEx
> knowledge, and I'm having trouble coming up with a single pattern that
> would handle it all.  Example bro test script attached; hopefully it's
> clear.
>
> Fundamentally, is there a syntax reference for pattern matching, or does
> it conform to a commonly known dialect (eg. POSIX-style RegEx, or PCRE
> RegEx)?
>
>
> [1] https://www.bro.org/sphinx/scripting/index.html#pattern
> [2] http://www.regular-expressions.info/wordboundaries.html
>
> --
> Lloyd Brown
> Systems Administrator
> Fulton Supercomputing Lab
> Brigham Young University
> http://marylou.byu.edu
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151022/872c38c4/attachment.html 

From michael.wenthold at gmail.com  Thu Oct 22 09:09:05 2015
From: michael.wenthold at gmail.com (Michael Wenthold)
Date: Thu, 22 Oct 2015 16:09:05 +0000
Subject: [Bro] Help with Bro & ES
In-Reply-To: <CAGEyGos+ALZ+-jnb9Qs69UnExkS5+g9QmWo+aX-YaOEGP9um6w@mail.gmail.com>
References: <CAGEyGospsVvuMzp22ASG35ZNrgfba6KXkYkjaBHAmTAOz-YWEA@mail.gmail.com>
	<FEDF3156-964A-470D-9FEF-256F182D17EF@illinois.edu>
	<CAGEyGotRZ5-tMybrRw-Jt_f6Kb5b3YZJ+r6tvYJ15Djb+Vj08g@mail.gmail.com>
	<B15DC30F-1FA6-4848-BC2A-A20281C0B051@gmail.com>
	<CAGEyGos+ALZ+-jnb9Qs69UnExkS5+g9QmWo+aX-YaOEGP9um6w@mail.gmail.com>
Message-ID: <CAMU2yhTPZXKar56_qknetKFM0EjwzUV2XgW15Onow9MYTnW80Q@mail.gmail.com>

The easiest way might be to just disable the loaded scripts log by adding
this to local.bro:

event bro_init()
    {
    Log::disable_stream(LoadedScripts::LOG);
    }



On Thu, Oct 22, 2015 at 1:39 PM, Chris Williams <cw13 at umbc.edu> wrote:

> Justin,
>
> I understand that "loaded scripts" is special, but it is creating issues
> with the index because there are no timestamps. Is there a recommended
> method for handling these messages?
> --
> Daniel- my connlog looks the same with the exception of "ts" which I
> assume you changed?
> --
>
> Thanks
>
>
> On Wed, Oct 21, 2015 at 2:20 PM, Daniel Guerra <daniel.guerra69 at gmail.com>
> wrote:
>
>> I?m working on elastic for a while
>> I changed a bit on the elasticsearch source,
>> recompile after doing this.
>> sed -i "s/JSON::TS_MILLIS/JSON::TS_ISO8601/g"
>> bro/aux/plugins/elasticsearch/src/ElasticSearch.cc
>> <http://elasticsearch.cc>
>>
>> my conn.log looks like this in kibana check ts difference
>>
>> {
>>   "_index": "bro-201509160700",
>>   "_type": "conn",
>>   "_id": "AVCLfROKixyabuRJCOlt",
>>   "_score": null,
>>   "_source": {
>>     "_timestamp": 1442388234879,
>>     "ts": "2015-09-16T07:16:54.185442Z",
>>     "uid": "Cv7R6a19zHzfu1H6U4",
>>     "id.orig_h": "192.168.1.122",
>>     "id.orig_p": 49428,
>>     "id.resp_h": "192.168.102.97",
>>     "id.resp_p": 514,
>>     "proto": "udp",
>>     "duration": 360.241984,
>>     "orig_bytes": 32096,
>>     "resp_bytes": 0,
>>     "conn_state": "S0",
>>     "missed_bytes": 0,
>>     "history": "D",
>>     "orig_pkts": 191,
>>     "orig_ip_bytes": 37444,
>>     "resp_pkts": 0,
>>     "resp_ip_bytes": 0,
>>     "tunnel_parents": []
>>   }
>>
>> The next chalenge are coordinates from geoip ..
>>
>> On 21 Oct 2015, at 18:32, Chris Williams <cw13 at umbc.edu> wrote:
>>
>> I mean... I think they look ok. Again, I understand that I have to learn
>> how to organize the information in such a way that will make sense. This is
>> an example of a conn message:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *{  "_index": "bro-201510191500",  "_type": "conn",  "_id":
>> "AVCBxqIWiyISA4W_6X6I",  "_score": 1,  "_source": {    "ts":
>> 1445286221580,    "uid": "CG7qWz2Xgs7J8LcO5d",    "id.orig_h":
>> "*.*.9.119",    "id.orig_p": 123,    "id.resp_h": "*.*.1.3",
>> "id.resp_p": 123,    "proto": "udp",    "duration": 0.0002,
>> "orig_bytes": 0,    "resp_bytes": 48,    "conn_state": "SHR",
>> "local_orig": false,    "local_resp": false,    "missed_bytes": 0,
>> "history": "Cd",    "orig_pkts": 0,    "orig_ip_bytes": 0,    "resp_pkts":
>> 1,    "resp_ip_bytes": 76,    "tunnel_parents": []  }}*
>>
>>
>> Chris Williams
>> ---
>> BS Information Systems - '16
>> CWIT Y2 Cyber Scholar <http://cybersecurity.umbc.edu/cyberscholars/>
>> Work: 5-0933
>> Cell: (202) 596-5406
>>
>> "If you think technology can solve your security problems, then you don't
>> understand the problems and you don't understand the technology." Bruce
>> Schneier
>>
>>
>> On Wed, Oct 21, 2015 at 12:28 PM, Azoff, Justin S <jazoff at illinois.edu>
>> wrote:
>>
>>>
>>> > On Oct 21, 2015, at 12:16 PM, Chris Williams <cw13 at umbc.edu> wrote:
>>> >
>>> > I recently installed Bro, and I am trying to get it to work with
>>> elastic search (with Kibana as a front end.) I have alerts getting to ES
>>> and it shows up in Kibana, but it is a mix of unintelligible json messages.
>>> For example, some don't have timestamps:
>>>
>>> ...
>>>
>>> >   "_type": "loaded_scripts",
>>> >
>>>
>>> The loaded_scripts.log is 'special' and does not have timestamps.  How
>>> do entries from things like the conn.log or http.log look?
>>>
>>>
>>> --
>>> - Justin Azoff
>>>
>>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151022/377ec056/attachment.html 

From jazoff at illinois.edu  Thu Oct 22 09:40:16 2015
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 22 Oct 2015 16:40:16 +0000
Subject: [Bro] Help with Bro & ES
In-Reply-To: <CAMU2yhTPZXKar56_qknetKFM0EjwzUV2XgW15Onow9MYTnW80Q@mail.gmail.com>
References: <CAGEyGospsVvuMzp22ASG35ZNrgfba6KXkYkjaBHAmTAOz-YWEA@mail.gmail.com>
	<FEDF3156-964A-470D-9FEF-256F182D17EF@illinois.edu>
	<CAGEyGotRZ5-tMybrRw-Jt_f6Kb5b3YZJ+r6tvYJ15Djb+Vj08g@mail.gmail.com>
	<B15DC30F-1FA6-4848-BC2A-A20281C0B051@gmail.com>
	<CAGEyGos+ALZ+-jnb9Qs69UnExkS5+g9QmWo+aX-YaOEGP9um6w@mail.gmail.com>
	<CAMU2yhTPZXKar56_qknetKFM0EjwzUV2XgW15Onow9MYTnW80Q@mail.gmail.com>
Message-ID: <37B29E6B-56B1-4FBF-9891-06A59D4C6291@illinois.edu>

One doesn't need to disable it entirely, you can just not send it to ES by using

redef LogElasticSearch::excluded_log_ids += {LoadedScripts::LOG};
	
-- 
- Justin Azoff

> On Oct 22, 2015, at 12:09 PM, Michael Wenthold <michael.wenthold at gmail.com> wrote:
> 
> The easiest way might be to just disable the loaded scripts log by adding this to local.bro:
> 
> event bro_init()
>     {
>     Log::disable_stream(LoadedScripts::LOG);
>     }
> 
> 
> 
> On Thu, Oct 22, 2015 at 1:39 PM, Chris Williams <cw13 at umbc.edu> wrote:
> Justin, 
> 
> I understand that "loaded scripts" is special, but it is creating issues with the index because there are no timestamps. Is there a recommended method for handling these messages?
> --
> Daniel- my connlog looks the same with the exception of "ts" which I assume you changed?
> --
> 
> Thanks
> 
> 
> On Wed, Oct 21, 2015 at 2:20 PM, Daniel Guerra <daniel.guerra69 at gmail.com> wrote:
> I?m working on elastic for a while
> I changed a bit on the elasticsearch source,
> recompile after doing this.
> sed -i "s/JSON::TS_MILLIS/JSON::TS_ISO8601/g" bro/aux/plugins/elasticsearch/src/ElasticSearch.cc
> 
> my conn.log looks like this in kibana check ts difference
> 
> {
>   "_index": "bro-201509160700",
>   "_type": "conn",
>   "_id": "AVCLfROKixyabuRJCOlt",
>   "_score": null,
>   "_source": {
>     "_timestamp": 1442388234879,
>     "ts": "2015-09-16T07:16:54.185442Z",
>     "uid": "Cv7R6a19zHzfu1H6U4",
>     "id.orig_h": "192.168.1.122",
>     "id.orig_p": 49428,
>     "id.resp_h": "192.168.102.97",
>     "id.resp_p": 514,
>     "proto": "udp",
>     "duration": 360.241984,
>     "orig_bytes": 32096,
>     "resp_bytes": 0,
>     "conn_state": "S0",
>     "missed_bytes": 0,
>     "history": "D",
>     "orig_pkts": 191,
>     "orig_ip_bytes": 37444,
>     "resp_pkts": 0,
>     "resp_ip_bytes": 0,
>     "tunnel_parents": []
>   }
> 
> The next chalenge are coordinates from geoip ..
> 
>> On 21 Oct 2015, at 18:32, Chris Williams <cw13 at umbc.edu> wrote:
>> 
>> I mean... I think they look ok. Again, I understand that I have to learn how to organize the information in such a way that will make sense. This is an example of a conn message:
>> 
>> {
>>   "_index": "bro-201510191500",
>>   "_type": "conn",
>>   "_id": "AVCBxqIWiyISA4W_6X6I",
>>   "_score": 1,
>>   "_source": {
>>     "ts": 1445286221580,
>>     "uid": "CG7qWz2Xgs7J8LcO5d",
>>     "id.orig_h": "*.*.9.119",
>>     "id.orig_p": 123,
>>     "id.resp_h": "*.*.1.3",
>>     "id.resp_p": 123,
>>     "proto": "udp",
>>     "duration": 0.0002,
>>     "orig_bytes": 0,
>>     "resp_bytes": 48,
>>     "conn_state": "SHR",
>>     "local_orig": false,
>>     "local_resp": false,
>>     "missed_bytes": 0,
>>     "history": "Cd",
>>     "orig_pkts": 0,
>>     "orig_ip_bytes": 0,
>>     "resp_pkts": 1,
>>     "resp_ip_bytes": 76,
>>     "tunnel_parents": []
>>   }
>> }
>> 
>> 
>> 
>> 
>> Chris Williams 
>> ---
>> BS Information Systems - '16
>> CWIT Y2 Cyber Scholar
>> Work: 5-0933
>> Cell: (202) 596-5406
>> 
>> "If you think technology can solve your security problems, then you don't understand the problems and you don't understand the technology." Bruce Schneier
>> 
>> 
>> On Wed, Oct 21, 2015 at 12:28 PM, Azoff, Justin S <jazoff at illinois.edu> wrote:
>> 
>> > On Oct 21, 2015, at 12:16 PM, Chris Williams <cw13 at umbc.edu> wrote:
>> >
>> > I recently installed Bro, and I am trying to get it to work with elastic search (with Kibana as a front end.) I have alerts getting to ES and it shows up in Kibana, but it is a mix of unintelligible json messages. For example, some don't have timestamps:
>> 
>> ...
>> 
>> >   "_type": "loaded_scripts",
>> >
>> 
>> The loaded_scripts.log is 'special' and does not have timestamps.  How do entries from things like the conn.log or http.log look?
>> 
>> 
>> --
>> - Justin Azoff
>> 
>> 
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From lloyd_brown at byu.edu  Thu Oct 22 10:08:50 2015
From: lloyd_brown at byu.edu (Lloyd Brown)
Date: Thu, 22 Oct 2015 11:08:50 -0600
Subject: [Bro] Patterns and Word Boundaries
In-Reply-To: <CAMLD0uJaZx+1ReQ3vpSZdp2JtmBCR3NR3rj+XDL4+wJHTkDkPw@mail.gmail.com>
References: <5628FB2E.9050604@byu.edu>
	<CAMLD0uJaZx+1ReQ3vpSZdp2JtmBCR3NR3rj+XDL4+wJHTkDkPw@mail.gmail.com>
Message-ID: <56291822.2000205@byu.edu>

Well, okay.  From what I can tell experimentally, it doesn't have
working shortcuts like "\s" or "[:space:]" either, so I guess I'm left
to do it more like *this* attachment.

Unless I'm missing something obvious.  I'd be happy to be wrong on this one.

Lloyd Brown
Systems Administrator
Fulton Supercomputing Lab
Brigham Young University
http://marylou.byu.edu

On 10/22/2015 10:03 AM, Samuel Oehlert wrote:
> I know Bro's regex syntax is almost exactly the same as Flex (only
> differing in some very edge cases). I am not positive, but from a
> cursory google it seems Flex doesn't understand word boundaries.
> 
> -Sam
> 
> On Thu, Oct 22, 2015 at 8:05 AM, Lloyd Brown <lloyd_brown at byu.edu
> <mailto:lloyd_brown at byu.edu>> wrote:
> 
>     Hopefully this isn't too simplistic of a question, but I'm just getting
>     started with Bro.
> 
>     In the text pattern syntax for Bro [1], is there an easy way to define
>     word boundaries, similar to how some of the RegEx dialects use '\b',
>     '\<', '\>', etc.? [2]
> 
>     I'm trying to match for specific strings in a data stream.  For example,
>     the word "nmap".  I'm trying several approaches, based on past RegEx
>     knowledge, and I'm having trouble coming up with a single pattern that
>     would handle it all.  Example bro test script attached; hopefully it's
>     clear.
> 
>     Fundamentally, is there a syntax reference for pattern matching, or does
>     it conform to a commonly known dialect (eg. POSIX-style RegEx, or PCRE
>     RegEx)?
> 
> 
>     [1] https://www.bro.org/sphinx/scripting/index.html#pattern
>     [2] http://www.regular-expressions.info/wordboundaries.html
> 
>     --
>     Lloyd Brown
>     Systems Administrator
>     Fulton Supercomputing Lab
>     Brigham Young University
>     http://marylou.byu.edu
> 
>     _______________________________________________
>     Bro mailing list
>     bro at bro-ids.org <mailto:bro at bro-ids.org>
>     http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 
-------------- next part --------------
event bro_init() {
	local testcases = set( 
		"nmap", 		#Should match something
		"test nmap", 		#Should match something
		"nmap test", 		#Should match something
		"test nmap test",	#should match something
		"unmapped_entries",	#Should NOT match any of the patterns
		"test\tnmap",		#Should match something
		"nmap\ttest",		#Should match something
		"test\tnmap\ttest"	#Should match something
		);
	local nmap_patterns = vector( 
		/^nmap$/, 
		/^nmap[ \f\n\r\t\v]/,	#This should be the definition of \s or [:space:]
		/[ \f\n\r\t\v]nmap$/,
		/[ \f\n\r\t\v]nmap[ \f\n\r\t\v]/
		);

	for (testcase in testcases) {
		print fmt("Testcase: \"%s\"", testcase);
		for (pi in nmap_patterns) {
			if ( nmap_patterns[pi] in testcase ) {
				print fmt("     Pattern: %s - Matched", nmap_patterns[pi]);
			} else {
				print fmt("     Pattern: %s - Did NOT match", nmap_patterns[pi]);
			}
		}
	}

}

From cw13 at umbc.edu  Thu Oct 22 10:24:35 2015
From: cw13 at umbc.edu (Chris Williams)
Date: Thu, 22 Oct 2015 13:24:35 -0400
Subject: [Bro] Help with Bro & ES
In-Reply-To: <37B29E6B-56B1-4FBF-9891-06A59D4C6291@illinois.edu>
References: <CAGEyGospsVvuMzp22ASG35ZNrgfba6KXkYkjaBHAmTAOz-YWEA@mail.gmail.com>
	<FEDF3156-964A-470D-9FEF-256F182D17EF@illinois.edu>
	<CAGEyGotRZ5-tMybrRw-Jt_f6Kb5b3YZJ+r6tvYJ15Djb+Vj08g@mail.gmail.com>
	<B15DC30F-1FA6-4848-BC2A-A20281C0B051@gmail.com>
	<CAGEyGos+ALZ+-jnb9Qs69UnExkS5+g9QmWo+aX-YaOEGP9um6w@mail.gmail.com>
	<CAMU2yhTPZXKar56_qknetKFM0EjwzUV2XgW15Onow9MYTnW80Q@mail.gmail.com>
	<37B29E6B-56B1-4FBF-9891-06A59D4C6291@illinois.edu>
Message-ID: <CAGEyGouMy-38Qu5LmfE_rDqNKP8NkQN=-dBMUyFwyDxnOf9xaA@mail.gmail.com>

Thanks Mike and Justin,

Please excuse my ignorance... but what are the implications of doing this?
I would prefer Justin's method just in case I need to dive in.... but could
someone explain the significance of loaded_scripts?



On Thu, Oct 22, 2015 at 12:40 PM, Azoff, Justin S <jazoff at illinois.edu>
wrote:

> One doesn't need to disable it entirely, you can just not send it to ES by
> using
>
> redef LogElasticSearch::excluded_log_ids += {LoadedScripts::LOG};
>
> --
> - Justin Azoff
>
> > On Oct 22, 2015, at 12:09 PM, Michael Wenthold <
> michael.wenthold at gmail.com> wrote:
> >
> > The easiest way might be to just disable the loaded scripts log by
> adding this to local.bro:
> >
> > event bro_init()
> >     {
> >     Log::disable_stream(LoadedScripts::LOG);
> >     }
> >
> >
> >
> > On Thu, Oct 22, 2015 at 1:39 PM, Chris Williams <cw13 at umbc.edu> wrote:
> > Justin,
> >
> > I understand that "loaded scripts" is special, but it is creating issues
> with the index because there are no timestamps. Is there a recommended
> method for handling these messages?
> > --
> > Daniel- my connlog looks the same with the exception of "ts" which I
> assume you changed?
> > --
> >
> > Thanks
> >
> >
> > On Wed, Oct 21, 2015 at 2:20 PM, Daniel Guerra <
> daniel.guerra69 at gmail.com> wrote:
> > I?m working on elastic for a while
> > I changed a bit on the elasticsearch source,
> > recompile after doing this.
> > sed -i "s/JSON::TS_MILLIS/JSON::TS_ISO8601/g"
> bro/aux/plugins/elasticsearch/src/ElasticSearch.cc
> >
> > my conn.log looks like this in kibana check ts difference
> >
> > {
> >   "_index": "bro-201509160700",
> >   "_type": "conn",
> >   "_id": "AVCLfROKixyabuRJCOlt",
> >   "_score": null,
> >   "_source": {
> >     "_timestamp": 1442388234879,
> >     "ts": "2015-09-16T07:16:54.185442Z",
> >     "uid": "Cv7R6a19zHzfu1H6U4",
> >     "id.orig_h": "192.168.1.122",
> >     "id.orig_p": 49428,
> >     "id.resp_h": "192.168.102.97",
> >     "id.resp_p": 514,
> >     "proto": "udp",
> >     "duration": 360.241984,
> >     "orig_bytes": 32096,
> >     "resp_bytes": 0,
> >     "conn_state": "S0",
> >     "missed_bytes": 0,
> >     "history": "D",
> >     "orig_pkts": 191,
> >     "orig_ip_bytes": 37444,
> >     "resp_pkts": 0,
> >     "resp_ip_bytes": 0,
> >     "tunnel_parents": []
> >   }
> >
> > The next chalenge are coordinates from geoip ..
> >
> >> On 21 Oct 2015, at 18:32, Chris Williams <cw13 at umbc.edu> wrote:
> >>
> >> I mean... I think they look ok. Again, I understand that I have to
> learn how to organize the information in such a way that will make sense.
> This is an example of a conn message:
> >>
> >> {
> >>   "_index": "bro-201510191500",
> >>   "_type": "conn",
> >>   "_id": "AVCBxqIWiyISA4W_6X6I",
> >>   "_score": 1,
> >>   "_source": {
> >>     "ts": 1445286221580,
> >>     "uid": "CG7qWz2Xgs7J8LcO5d",
> >>     "id.orig_h": "*.*.9.119",
> >>     "id.orig_p": 123,
> >>     "id.resp_h": "*.*.1.3",
> >>     "id.resp_p": 123,
> >>     "proto": "udp",
> >>     "duration": 0.0002,
> >>     "orig_bytes": 0,
> >>     "resp_bytes": 48,
> >>     "conn_state": "SHR",
> >>     "local_orig": false,
> >>     "local_resp": false,
> >>     "missed_bytes": 0,
> >>     "history": "Cd",
> >>     "orig_pkts": 0,
> >>     "orig_ip_bytes": 0,
> >>     "resp_pkts": 1,
> >>     "resp_ip_bytes": 76,
> >>     "tunnel_parents": []
> >>   }
> >> }
> >>
> >>
> >>
> >>
> >> Chris Williams
> >> ---
> >> BS Information Systems - '16
> >> CWIT Y2 Cyber Scholar
> >> Work: 5-0933
> >> Cell: (202) 596-5406
> >>
> >> "If you think technology can solve your security problems, then you
> don't understand the problems and you don't understand the technology."
> Bruce Schneier
> >>
> >>
> >> On Wed, Oct 21, 2015 at 12:28 PM, Azoff, Justin S <jazoff at illinois.edu>
> wrote:
> >>
> >> > On Oct 21, 2015, at 12:16 PM, Chris Williams <cw13 at umbc.edu> wrote:
> >> >
> >> > I recently installed Bro, and I am trying to get it to work with
> elastic search (with Kibana as a front end.) I have alerts getting to ES
> and it shows up in Kibana, but it is a mix of unintelligible json messages.
> For example, some don't have timestamps:
> >>
> >> ...
> >>
> >> >   "_type": "loaded_scripts",
> >> >
> >>
> >> The loaded_scripts.log is 'special' and does not have timestamps.  How
> do entries from things like the conn.log or http.log look?
> >>
> >>
> >> --
> >> - Justin Azoff
> >>
> >>
> >> _______________________________________________
> >> Bro mailing list
> >> bro at bro-ids.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151022/3b4dd93a/attachment.html 

From jazoff at illinois.edu  Thu Oct 22 10:30:27 2015
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 22 Oct 2015 17:30:27 +0000
Subject: [Bro] Help with Bro & ES
In-Reply-To: <CAGEyGouMy-38Qu5LmfE_rDqNKP8NkQN=-dBMUyFwyDxnOf9xaA@mail.gmail.com>
References: <CAGEyGospsVvuMzp22ASG35ZNrgfba6KXkYkjaBHAmTAOz-YWEA@mail.gmail.com>
	<FEDF3156-964A-470D-9FEF-256F182D17EF@illinois.edu>
	<CAGEyGotRZ5-tMybrRw-Jt_f6Kb5b3YZJ+r6tvYJ15Djb+Vj08g@mail.gmail.com>
	<B15DC30F-1FA6-4848-BC2A-A20281C0B051@gmail.com>
	<CAGEyGos+ALZ+-jnb9Qs69UnExkS5+g9QmWo+aX-YaOEGP9um6w@mail.gmail.com>
	<CAMU2yhTPZXKar56_qknetKFM0EjwzUV2XgW15Onow9MYTnW80Q@mail.gmail.com>
	<37B29E6B-56B1-4FBF-9891-06A59D4C6291@illinois.edu>
	<CAGEyGouMy-38Qu5LmfE_rDqNKP8NkQN=-dBMUyFwyDxnOf9xaA@mail.gmail.com>
Message-ID: <F58E8B08-3C22-4538-8855-20CC6BFAC1F7@illinois.edu>


> On Oct 22, 2015, at 1:24 PM, Chris Williams <cw13 at umbc.edu> wrote:
> 
> Thanks Mike and Justin, 
> 
> Please excuse my ignorance... but what are the implications of doing this? I would prefer Justin's method just in case I need to dive in.... but could someone explain the significance of loaded_scripts?
> 

Open it up and look at it.. It's literally just a log file created when bro starts that lists out all the scripts you have loaded.  Other than for occasional troubleshooting it has no value whatsoever.



-- 
- Justin Azoff



From seth at icir.org  Thu Oct 22 11:14:07 2015
From: seth at icir.org (Seth Hall)
Date: Thu, 22 Oct 2015 14:14:07 -0400
Subject: [Bro] Bro and Yara together
In-Reply-To: <CANdYiFGwMocc5q1oRQFXn_9E5SKPbqRRNq3JyDzzNiOQ3_xL0g@mail.gmail.com>
References: <CANdYiFGwMocc5q1oRQFXn_9E5SKPbqRRNq3JyDzzNiOQ3_xL0g@mail.gmail.com>
Message-ID: <797E1EC3-0393-4FD1-A146-3D128DAEA9E4@icir.org>


> On Oct 22, 2015, at 10:44 AM, Vito Logrillo <vitologrillo at gmail.com> wrote:
> 
> the git code below can be used to integrate Bro with Yara:
> https://github.com/hempnall/broyara
> 
> I've tried to use it without any result. Should i download the files
> in the same bro directory and then change the CMake files as
> described?

I?m not sure this is going to be a great solution for many people.  You are pulling the entire file into system memory and then having Yara analyze it once the file is completed.  I?ve worked with Yara before and even added a streaming API (which hasn?t been accept back into Yara proper yet).

I guess I?m just trying to get across that I?d be a little concerned about running this on a real network. :)

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From robin at icir.org  Thu Oct 22 11:40:49 2015
From: robin at icir.org (Robin Sommer)
Date: Thu, 22 Oct 2015 11:40:49 -0700
Subject: [Bro] Patterns and Word Boundaries
In-Reply-To: <56291822.2000205@byu.edu>
References: <5628FB2E.9050604@byu.edu>
	<CAMLD0uJaZx+1ReQ3vpSZdp2JtmBCR3NR3rj+XDL4+wJHTkDkPw@mail.gmail.com>
	<56291822.2000205@byu.edu>
Message-ID: <20151022184049.GE17436@icir.org>



On Thu, Oct 22, 2015 at 11:08 -0600, Lloyd Brown wrote:

> working shortcuts like "\s" or "[:space:]" either

It does actually support the standard "[:...:]" cases.

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From lloyd_brown at byu.edu  Thu Oct 22 14:32:09 2015
From: lloyd_brown at byu.edu (Lloyd Brown)
Date: Thu, 22 Oct 2015 15:32:09 -0600
Subject: [Bro] Patterns and Word Boundaries
In-Reply-To: <56291822.2000205@byu.edu>
References: <5628FB2E.9050604@byu.edu>	<CAMLD0uJaZx+1ReQ3vpSZdp2JtmBCR3NR3rj+XDL4+wJHTkDkPw@mail.gmail.com>
	<56291822.2000205@byu.edu>
Message-ID: <562955D9.6040608@byu.edu>

For future list-viewers, yes, I was missing something obvious.  The word
boundaries are genuinely missing, but I was using the shortcuts like
'[:space:]' incorrectly.

In short, '[:space:]' and others like it, are not character classes
themselves, but they can exist in a character class.  The '[:space:]' is
not the equivalent of '[ \f\n\r\t\v]', but '[[:space:]]' is.

Thanks for the feedback on this, Robin.  Sorry for the unnecessary list
noise.

Lloyd Brown
Systems Administrator
Fulton Supercomputing Lab
Brigham Young University
http://marylou.byu.edu

On 10/22/2015 11:08 AM, Lloyd Brown wrote:
> Well, okay.  From what I can tell experimentally, it doesn't have
> working shortcuts like "\s" or "[:space:]" either, so I guess I'm left
> to do it more like *this* attachment.
> 
> Unless I'm missing something obvious.  I'd be happy to be wrong on this one.
> 
> Lloyd Brown
> Systems Administrator
> Fulton Supercomputing Lab
> Brigham Young University
> http://marylou.byu.edu
> 
> On 10/22/2015 10:03 AM, Samuel Oehlert wrote:
>> I know Bro's regex syntax is almost exactly the same as Flex (only
>> differing in some very edge cases). I am not positive, but from a
>> cursory google it seems Flex doesn't understand word boundaries.
>>
>> -Sam
>>
>> On Thu, Oct 22, 2015 at 8:05 AM, Lloyd Brown <lloyd_brown at byu.edu
>> <mailto:lloyd_brown at byu.edu>> wrote:
>>
>>     Hopefully this isn't too simplistic of a question, but I'm just getting
>>     started with Bro.
>>
>>     In the text pattern syntax for Bro [1], is there an easy way to define
>>     word boundaries, similar to how some of the RegEx dialects use '\b',
>>     '\<', '\>', etc.? [2]
>>
>>     I'm trying to match for specific strings in a data stream.  For example,
>>     the word "nmap".  I'm trying several approaches, based on past RegEx
>>     knowledge, and I'm having trouble coming up with a single pattern that
>>     would handle it all.  Example bro test script attached; hopefully it's
>>     clear.
>>
>>     Fundamentally, is there a syntax reference for pattern matching, or does
>>     it conform to a commonly known dialect (eg. POSIX-style RegEx, or PCRE
>>     RegEx)?
>>
>>
>>     [1] https://www.bro.org/sphinx/scripting/index.html#pattern
>>     [2] http://www.regular-expressions.info/wordboundaries.html
>>
>>     --
>>     Lloyd Brown
>>     Systems Administrator
>>     Fulton Supercomputing Lab
>>     Brigham Young University
>>     http://marylou.byu.edu
>>
>>     _______________________________________________
>>     Bro mailing list
>>     bro at bro-ids.org <mailto:bro at bro-ids.org>
>>     http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

From daniel.guerra69 at gmail.com  Thu Oct 22 15:56:28 2015
From: daniel.guerra69 at gmail.com (Daniel Guerra)
Date: Fri, 23 Oct 2015 00:56:28 +0200
Subject: [Bro] Patterns and Word Boundaries
In-Reply-To: <562955D9.6040608@byu.edu>
References: <5628FB2E.9050604@byu.edu>
	<CAMLD0uJaZx+1ReQ3vpSZdp2JtmBCR3NR3rj+XDL4+wJHTkDkPw@mail.gmail.com>
	<56291822.2000205@byu.edu> <562955D9.6040608@byu.edu>
Message-ID: <35C65E3C-369D-469E-9591-E47175462E70@gmail.com>

Have you read this ??

http://flex.sourceforge.net/manual/Patterns.html

Regex != Flex 

> On 22 Oct 2015, at 23:32, Lloyd Brown <lloyd_brown at byu.edu> wrote:
> 
> For future list-viewers, yes, I was missing something obvious.  The word
> boundaries are genuinely missing, but I was using the shortcuts like
> '[:space:]' incorrectly.
> 
> In short, '[:space:]' and others like it, are not character classes
> themselves, but they can exist in a character class.  The '[:space:]' is
> not the equivalent of '[ \f\n\r\t\v]', but '[[:space:]]' is.
> 
> Thanks for the feedback on this, Robin.  Sorry for the unnecessary list
> noise.
> 
> Lloyd Brown
> Systems Administrator
> Fulton Supercomputing Lab
> Brigham Young University
> http://marylou.byu.edu
> 
> On 10/22/2015 11:08 AM, Lloyd Brown wrote:
>> Well, okay.  From what I can tell experimentally, it doesn't have
>> working shortcuts like "\s" or "[:space:]" either, so I guess I'm left
>> to do it more like *this* attachment.
>> 
>> Unless I'm missing something obvious.  I'd be happy to be wrong on this one.
>> 
>> Lloyd Brown
>> Systems Administrator
>> Fulton Supercomputing Lab
>> Brigham Young University
>> http://marylou.byu.edu
>> 
>> On 10/22/2015 10:03 AM, Samuel Oehlert wrote:
>>> I know Bro's regex syntax is almost exactly the same as Flex (only
>>> differing in some very edge cases). I am not positive, but from a
>>> cursory google it seems Flex doesn't understand word boundaries.
>>> 
>>> -Sam
>>> 
>>> On Thu, Oct 22, 2015 at 8:05 AM, Lloyd Brown <lloyd_brown at byu.edu
>>> <mailto:lloyd_brown at byu.edu>> wrote:
>>> 
>>>    Hopefully this isn't too simplistic of a question, but I'm just getting
>>>    started with Bro.
>>> 
>>>    In the text pattern syntax for Bro [1], is there an easy way to define
>>>    word boundaries, similar to how some of the RegEx dialects use '\b',
>>>    '\<', '\>', etc.? [2]
>>> 
>>>    I'm trying to match for specific strings in a data stream.  For example,
>>>    the word "nmap".  I'm trying several approaches, based on past RegEx
>>>    knowledge, and I'm having trouble coming up with a single pattern that
>>>    would handle it all.  Example bro test script attached; hopefully it's
>>>    clear.
>>> 
>>>    Fundamentally, is there a syntax reference for pattern matching, or does
>>>    it conform to a commonly known dialect (eg. POSIX-style RegEx, or PCRE
>>>    RegEx)?
>>> 
>>> 
>>>    [1] https://www.bro.org/sphinx/scripting/index.html#pattern
>>>    [2] http://www.regular-expressions.info/wordboundaries.html
>>> 
>>>    --
>>>    Lloyd Brown
>>>    Systems Administrator
>>>    Fulton Supercomputing Lab
>>>    Brigham Young University
>>>    http://marylou.byu.edu
>>> 
>>>    _______________________________________________
>>>    Bro mailing list
>>>    bro at bro-ids.org <mailto:bro at bro-ids.org>
>>>    http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From lloyd_brown at byu.edu  Thu Oct 22 16:03:38 2015
From: lloyd_brown at byu.edu (Lloyd Brown)
Date: Thu, 22 Oct 2015 17:03:38 -0600
Subject: [Bro] Patterns and Word Boundaries
In-Reply-To: <35C65E3C-369D-469E-9591-E47175462E70@gmail.com>
References: <5628FB2E.9050604@byu.edu>
	<CAMLD0uJaZx+1ReQ3vpSZdp2JtmBCR3NR3rj+XDL4+wJHTkDkPw@mail.gmail.com>
	<56291822.2000205@byu.edu> <562955D9.6040608@byu.edu>
	<35C65E3C-369D-469E-9591-E47175462E70@gmail.com>
Message-ID: <56296B4A.4070304@byu.edu>

Yes.  I had seen that.  And I just missed the double-bracket detail.

Having said that, this looks to me like as much of a RegEx dialect as
any other.  Those extended shortcuts like I've been referring to are
reasonably common, but not required.  And the first sentence on that
page even says the following:

> The patterns in the input ... are written using an extended set of regular expressions.

So Flex and RegEx at least share a lot of features and syntax.  Whether
or not it's truly RegEx, seems like a purely semantic discussion.


Lloyd Brown
Systems Administrator
Fulton Supercomputing Lab
Brigham Young University
http://marylou.byu.edu

On 10/22/2015 04:56 PM, Daniel Guerra wrote:
> Have you read this ??
> 
> http://flex.sourceforge.net/manual/Patterns.html
> 
> Regex != Flex 
> 
>> On 22 Oct 2015, at 23:32, Lloyd Brown <lloyd_brown at byu.edu> wrote:
>>
>> For future list-viewers, yes, I was missing something obvious.  The word
>> boundaries are genuinely missing, but I was using the shortcuts like
>> '[:space:]' incorrectly.
>>
>> In short, '[:space:]' and others like it, are not character classes
>> themselves, but they can exist in a character class.  The '[:space:]' is
>> not the equivalent of '[ \f\n\r\t\v]', but '[[:space:]]' is.
>>
>> Thanks for the feedback on this, Robin.  Sorry for the unnecessary list
>> noise.
>>
>> Lloyd Brown
>> Systems Administrator
>> Fulton Supercomputing Lab
>> Brigham Young University
>> http://marylou.byu.edu
>>
>> On 10/22/2015 11:08 AM, Lloyd Brown wrote:
>>> Well, okay.  From what I can tell experimentally, it doesn't have
>>> working shortcuts like "\s" or "[:space:]" either, so I guess I'm left
>>> to do it more like *this* attachment.
>>>
>>> Unless I'm missing something obvious.  I'd be happy to be wrong on this one.
>>>
>>> Lloyd Brown
>>> Systems Administrator
>>> Fulton Supercomputing Lab
>>> Brigham Young University
>>> http://marylou.byu.edu
>>>
>>> On 10/22/2015 10:03 AM, Samuel Oehlert wrote:
>>>> I know Bro's regex syntax is almost exactly the same as Flex (only
>>>> differing in some very edge cases). I am not positive, but from a
>>>> cursory google it seems Flex doesn't understand word boundaries.
>>>>
>>>> -Sam
>>>>
>>>> On Thu, Oct 22, 2015 at 8:05 AM, Lloyd Brown <lloyd_brown at byu.edu
>>>> <mailto:lloyd_brown at byu.edu>> wrote:
>>>>
>>>>    Hopefully this isn't too simplistic of a question, but I'm just getting
>>>>    started with Bro.
>>>>
>>>>    In the text pattern syntax for Bro [1], is there an easy way to define
>>>>    word boundaries, similar to how some of the RegEx dialects use '\b',
>>>>    '\<', '\>', etc.? [2]
>>>>
>>>>    I'm trying to match for specific strings in a data stream.  For example,
>>>>    the word "nmap".  I'm trying several approaches, based on past RegEx
>>>>    knowledge, and I'm having trouble coming up with a single pattern that
>>>>    would handle it all.  Example bro test script attached; hopefully it's
>>>>    clear.
>>>>
>>>>    Fundamentally, is there a syntax reference for pattern matching, or does
>>>>    it conform to a commonly known dialect (eg. POSIX-style RegEx, or PCRE
>>>>    RegEx)?
>>>>
>>>>
>>>>    [1] https://www.bro.org/sphinx/scripting/index.html#pattern
>>>>    [2] http://www.regular-expressions.info/wordboundaries.html
>>>>
>>>>    --
>>>>    Lloyd Brown
>>>>    Systems Administrator
>>>>    Fulton Supercomputing Lab
>>>>    Brigham Young University
>>>>    http://marylou.byu.edu
>>>>
>>>>    _______________________________________________
>>>>    Bro mailing list
>>>>    bro at bro-ids.org <mailto:bro at bro-ids.org>
>>>>    http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Bro mailing list
>>>> bro at bro-ids.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 

From mfw113 at psu.edu  Fri Oct 23 06:21:46 2015
From: mfw113 at psu.edu (Mike Waite)
Date: Fri, 23 Oct 2015 09:21:46 -0400
Subject: [Bro] Memory Issue with Bro
In-Reply-To: <E7981188-57E2-4E65-8064-8E1A0FD24172@illinois.edu>
References: <69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com>
	<454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu>
	<7689d3f6ddb6454d855e6fd3a3b8ea7b@ocex03.SolarFlarecom.com>
	<3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org>
	<CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
	<DE5BED9E-D710-4763-A978-423D917509A5@illinois.edu>
	<CAMCSCoW6AisyP479iHN5pms+AcHPmFUAEdUvsJvhUPoRTNxD5w@mail.gmail.com>
	<AE9E9388-EC16-4040-A8CD-3752E1E3E3A8@illinois.edu>
	<20151022121254.GA26216@redquill.sos.its.psu.edu>
	<E7981188-57E2-4E65-8064-8E1A0FD24172@illinois.edu>
Message-ID: <20151023132146.GA24138@redquill.sos.its.psu.edu>

Patch applied, after 15 minutes I am seeing

Oct 23 09:00:43	manager	child	-	-	-	info	selects=300000 canwrites=216206 pending=0
Oct 23 09:01:29	manager	child	-	-	-	info	selects=400000 canwrites=216206 pending=0
Oct 23 09:02:08	manager	child	-	-	-	info	selects=500000 canwrites=216552 pending=0
Oct 23 09:02:49	manager	child	-	-	-	info	selects=600000 canwrites=216557 pending=0
Oct 23 09:03:34	manager	child	-	-	-	info	selects=700000 canwrites=216557 pending=0
Oct 23 09:04:29	manager	child	-	-	-	info	selects=800000 canwrites=255305 pending=4007
Oct 23 09:05:21	manager	child	-	-	-	info	selects=900000 canwrites=355305 pending=6593
Oct 23 09:06:13	manager	child	-	-	-	info	selects=1000000 canwrites=455305 pending=6003
Oct 23 09:07:04	manager	child	-	-	-	info	selects=1100000 canwrites=555305 pending=3077
Oct 23 09:07:55	manager	child	-	-	-	info	selects=1200000 canwrites=640438 pending=3399
Oct 23 09:08:45	manager	child	-	-	-	info	selects=1300000 canwrites=740438 pending=3163
Oct 23 09:09:36	manager	child	-	-	-	info	selects=1400000 canwrites=840438 pending=5245
Oct 23 09:10:25	manager	child	-	-	-	info	selects=1500000 canwrites=940438 pending=6027
Oct 23 09:11:15	manager	child	-	-	-	info	selects=1600000 canwrites=1040438 pending=6713
Oct 23 09:12:01	manager	child	-	-	-	info	selects=1700000 canwrites=1140438 pending=5713
Oct 23 09:12:50	manager	child	-	-	-	info	selects=1800000 canwrites=1240438 pending=6747
Oct 23 09:13:39	manager	child	-	-	-	info	selects=1900000 canwrites=1340438 pending=7417
Oct 23 09:14:32	manager	child	-	-	-	info	selects=2000000 canwrites=1440438 pending=13117
Oct 23 09:15:10	manager	child	-	-	-	info	selects=2100000 canwrites=1540438 pending=20825
Oct 23 09:15:59	manager	child	-	-	-	info	selects=2200000 canwrites=1640438 pending=18539
Oct 23 09:16:47	manager	child	-	-	-	info	selects=2300000 canwrites=1740438 pending=15881
Oct 23 09:17:35	manager	child	-	-	-	info	selects=2400000 canwrites=1840438 pending=15389
Oct 23 09:18:28	manager	child	-	-	-	info	selects=2500000 canwrites=1940438 pending=16685
Oct 23 09:19:18	manager	child	-	-	-	info	selects=2600000 canwrites=2040438 pending=17031


I will let you know about the mem usage after a bit

--
Mike Waite
CyberSecurity Intrusion Analyst
Office of Information Security
The Pennsylvania State University
? 15-10-22 10:22:18, Azoff, Justin S <jazoff at illinois.edu>:
>> On Oct 22, 2015, at 8:12 AM, Mike Waite <mfw113 at psu.edu> wrote:
>>
>> I know we are still seeing issues with the manager child proccess.  The process will consume over 200GB of RAM in 8 hours.
>>
>
>Give the attached patch a try.
>
>
>
>Monitor by using
>
>    cat logs/current/communication.log |egrep 'manager.child'
>
>And check to see if pending=0 or at least not growing.
>
>
>--
>- Justin Azoff
>
>


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151023/505a24f1/attachment.bin 

From jazoff at illinois.edu  Fri Oct 23 07:19:26 2015
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Fri, 23 Oct 2015 14:19:26 +0000
Subject: [Bro] Memory Issue with Bro
In-Reply-To: <20151023132146.GA24138@redquill.sos.its.psu.edu>
References: <69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com>
	<454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu>
	<7689d3f6ddb6454d855e6fd3a3b8ea7b@ocex03.SolarFlarecom.com>
	<3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org>
	<CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
	<DE5BED9E-D710-4763-A978-423D917509A5@illinois.edu>
	<CAMCSCoW6AisyP479iHN5pms+AcHPmFUAEdUvsJvhUPoRTNxD5w@mail.gmail.com>
	<AE9E9388-EC16-4040-A8CD-3752E1E3E3A8@illinois.edu>
	<20151022121254.GA26216@redquill.sos.its.psu.edu>
	<E7981188-57E2-4E65-8064-8E1A0FD24172@illinois.edu>
	<20151023132146.GA24138@redquill.sos.its.psu.edu>
Message-ID: <97BEC855-2FD0-4F53-9F6F-DE6C1AA56FF7@illinois.edu>

Well that doesn't look great, but could be a lot worse.  Hard to say without knowing what it looked like before the patch.

The fact that pending ever goes down at all is a good sign, but pending=0 is really the optimal state.

-- 
- Justin Azoff

> On Oct 23, 2015, at 9:21 AM, Mike Waite <mfw113 at psu.edu> wrote:
> 
> Patch applied, after 15 minutes I am seeing
> 
> Oct 23 09:00:43	manager	child	-	-	-	info	selects=300000 canwrites=216206 pending=0
> Oct 23 09:01:29	manager	child	-	-	-	info	selects=400000 canwrites=216206 pending=0
> Oct 23 09:02:08	manager	child	-	-	-	info	selects=500000 canwrites=216552 pending=0
> Oct 23 09:02:49	manager	child	-	-	-	info	selects=600000 canwrites=216557 pending=0
> Oct 23 09:03:34	manager	child	-	-	-	info	selects=700000 canwrites=216557 pending=0
> Oct 23 09:04:29	manager	child	-	-	-	info	selects=800000 canwrites=255305 pending=4007
> Oct 23 09:05:21	manager	child	-	-	-	info	selects=900000 canwrites=355305 pending=6593
> Oct 23 09:06:13	manager	child	-	-	-	info	selects=1000000 canwrites=455305 pending=6003
> Oct 23 09:07:04	manager	child	-	-	-	info	selects=1100000 canwrites=555305 pending=3077
> Oct 23 09:07:55	manager	child	-	-	-	info	selects=1200000 canwrites=640438 pending=3399
> Oct 23 09:08:45	manager	child	-	-	-	info	selects=1300000 canwrites=740438 pending=3163
> Oct 23 09:09:36	manager	child	-	-	-	info	selects=1400000 canwrites=840438 pending=5245
> Oct 23 09:10:25	manager	child	-	-	-	info	selects=1500000 canwrites=940438 pending=6027
> Oct 23 09:11:15	manager	child	-	-	-	info	selects=1600000 canwrites=1040438 pending=6713
> Oct 23 09:12:01	manager	child	-	-	-	info	selects=1700000 canwrites=1140438 pending=5713
> Oct 23 09:12:50	manager	child	-	-	-	info	selects=1800000 canwrites=1240438 pending=6747
> Oct 23 09:13:39	manager	child	-	-	-	info	selects=1900000 canwrites=1340438 pending=7417
> Oct 23 09:14:32	manager	child	-	-	-	info	selects=2000000 canwrites=1440438 pending=13117
> Oct 23 09:15:10	manager	child	-	-	-	info	selects=2100000 canwrites=1540438 pending=20825
> Oct 23 09:15:59	manager	child	-	-	-	info	selects=2200000 canwrites=1640438 pending=18539
> Oct 23 09:16:47	manager	child	-	-	-	info	selects=2300000 canwrites=1740438 pending=15881
> Oct 23 09:17:35	manager	child	-	-	-	info	selects=2400000 canwrites=1840438 pending=15389
> Oct 23 09:18:28	manager	child	-	-	-	info	selects=2500000 canwrites=1940438 pending=16685
> Oct 23 09:19:18	manager	child	-	-	-	info	selects=2600000 canwrites=2040438 pending=17031
> 
> 
> I will let you know about the mem usage after a bit
> 
> --
> Mike Waite
> CyberSecurity Intrusion Analyst
> Office of Information Security
> The Pennsylvania State University
> ? 15-10-22 10:22:18, Azoff, Justin S <jazoff at illinois.edu>:
>>> On Oct 22, 2015, at 8:12 AM, Mike Waite <mfw113 at psu.edu> wrote:
>>> 
>>> I know we are still seeing issues with the manager child proccess.  The process will consume over 200GB of RAM in 8 hours.
>>> 
>> 
>> Give the attached patch a try.
>> 
>> 
>> 
>> Monitor by using
>> 
>>   cat logs/current/communication.log |egrep 'manager.child'
>> 
>> And check to see if pending=0 or at least not growing.
>> 
>> 
>> --
>> - Justin Azoff
>> 
>> 
> 
> 



From seth at icir.org  Fri Oct 23 08:09:58 2015
From: seth at icir.org (Seth Hall)
Date: Fri, 23 Oct 2015 11:09:58 -0400
Subject: [Bro] Memory Issue with Bro
In-Reply-To: <97BEC855-2FD0-4F53-9F6F-DE6C1AA56FF7@illinois.edu>
References: <69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com>
	<454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu>
	<7689d3f6ddb6454d855e6fd3a3b8ea7b@ocex03.SolarFlarecom.com>
	<3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org>
	<CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
	<DE5BED9E-D710-4763-A978-423D917509A5@illinois.edu>
	<CAMCSCoW6AisyP479iHN5pms+AcHPmFUAEdUvsJvhUPoRTNxD5w@mail.gmail.com>
	<AE9E9388-EC16-4040-A8CD-3752E1E3E3A8@illinois.edu>
	<20151022121254.GA26216@redquill.sos.its.psu.edu>
	<E7981188-57E2-4E65-8064-8E1A0FD24172@illinois.edu>
	<20151023132146.GA24138@redquill.sos.its.psu.edu>
	<97BEC855-2FD0-4F53-9F6F-DE6C1AA56FF7@illinois.edu>
Message-ID: <5A7DD598-CE76-4839-8156-67187148AA95@icir.org>

Mike, could you back out that patch and try my branch, topic/seth/remove-flare ?

  .Seth


> On Oct 23, 2015, at 10:19 AM, Azoff, Justin S <jazoff at illinois.edu> wrote:
> 
> Well that doesn't look great, but could be a lot worse.  Hard to say without knowing what it looked like before the patch.
> 
> The fact that pending ever goes down at all is a good sign, but pending=0 is really the optimal state.
> 
> -- 
> - Justin Azoff
> 
>> On Oct 23, 2015, at 9:21 AM, Mike Waite <mfw113 at psu.edu> wrote:
>> 
>> Patch applied, after 15 minutes I am seeing
>> 
>> Oct 23 09:00:43	manager	child	-	-	-	info	selects=300000 canwrites=216206 pending=0
>> Oct 23 09:01:29	manager	child	-	-	-	info	selects=400000 canwrites=216206 pending=0
>> Oct 23 09:02:08	manager	child	-	-	-	info	selects=500000 canwrites=216552 pending=0
>> Oct 23 09:02:49	manager	child	-	-	-	info	selects=600000 canwrites=216557 pending=0
>> Oct 23 09:03:34	manager	child	-	-	-	info	selects=700000 canwrites=216557 pending=0
>> Oct 23 09:04:29	manager	child	-	-	-	info	selects=800000 canwrites=255305 pending=4007
>> Oct 23 09:05:21	manager	child	-	-	-	info	selects=900000 canwrites=355305 pending=6593
>> Oct 23 09:06:13	manager	child	-	-	-	info	selects=1000000 canwrites=455305 pending=6003
>> Oct 23 09:07:04	manager	child	-	-	-	info	selects=1100000 canwrites=555305 pending=3077
>> Oct 23 09:07:55	manager	child	-	-	-	info	selects=1200000 canwrites=640438 pending=3399
>> Oct 23 09:08:45	manager	child	-	-	-	info	selects=1300000 canwrites=740438 pending=3163
>> Oct 23 09:09:36	manager	child	-	-	-	info	selects=1400000 canwrites=840438 pending=5245
>> Oct 23 09:10:25	manager	child	-	-	-	info	selects=1500000 canwrites=940438 pending=6027
>> Oct 23 09:11:15	manager	child	-	-	-	info	selects=1600000 canwrites=1040438 pending=6713
>> Oct 23 09:12:01	manager	child	-	-	-	info	selects=1700000 canwrites=1140438 pending=5713
>> Oct 23 09:12:50	manager	child	-	-	-	info	selects=1800000 canwrites=1240438 pending=6747
>> Oct 23 09:13:39	manager	child	-	-	-	info	selects=1900000 canwrites=1340438 pending=7417
>> Oct 23 09:14:32	manager	child	-	-	-	info	selects=2000000 canwrites=1440438 pending=13117
>> Oct 23 09:15:10	manager	child	-	-	-	info	selects=2100000 canwrites=1540438 pending=20825
>> Oct 23 09:15:59	manager	child	-	-	-	info	selects=2200000 canwrites=1640438 pending=18539
>> Oct 23 09:16:47	manager	child	-	-	-	info	selects=2300000 canwrites=1740438 pending=15881
>> Oct 23 09:17:35	manager	child	-	-	-	info	selects=2400000 canwrites=1840438 pending=15389
>> Oct 23 09:18:28	manager	child	-	-	-	info	selects=2500000 canwrites=1940438 pending=16685
>> Oct 23 09:19:18	manager	child	-	-	-	info	selects=2600000 canwrites=2040438 pending=17031
>> 
>> 
>> I will let you know about the mem usage after a bit
>> 
>> --
>> Mike Waite
>> CyberSecurity Intrusion Analyst
>> Office of Information Security
>> The Pennsylvania State University
>> ? 15-10-22 10:22:18, Azoff, Justin S <jazoff at illinois.edu>:
>>>> On Oct 22, 2015, at 8:12 AM, Mike Waite <mfw113 at psu.edu> wrote:
>>>> 
>>>> I know we are still seeing issues with the manager child proccess.  The process will consume over 200GB of RAM in 8 hours.
>>>> 
>>> 
>>> Give the attached patch a try.
>>> 
>>> 
>>> 
>>> Monitor by using
>>> 
>>>  cat logs/current/communication.log |egrep 'manager.child'
>>> 
>>> And check to see if pending=0 or at least not growing.
>>> 
>>> 
>>> --
>>> - Justin Azoff
>>> 
>>> 
>> 
>> 
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From the.warl0ck.1989 at gmail.com  Fri Oct 23 08:29:38 2015
From: the.warl0ck.1989 at gmail.com (Aaron Lewis)
Date: Fri, 23 Oct 2015 23:29:38 +0800
Subject: [Bro] Memory Issue with Bro
In-Reply-To: <20151023132146.GA24138@redquill.sos.its.psu.edu>
References: <69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com>
	<454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu>
	<7689d3f6ddb6454d855e6fd3a3b8ea7b@ocex03.SolarFlarecom.com>
	<3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org>
	<CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
	<DE5BED9E-D710-4763-A978-423D917509A5@illinois.edu>
	<CAMCSCoW6AisyP479iHN5pms+AcHPmFUAEdUvsJvhUPoRTNxD5w@mail.gmail.com>
	<AE9E9388-EC16-4040-A8CD-3752E1E3E3A8@illinois.edu>
	<20151022121254.GA26216@redquill.sos.its.psu.edu>
	<E7981188-57E2-4E65-8064-8E1A0FD24172@illinois.edu>
	<20151023132146.GA24138@redquill.sos.its.psu.edu>
Message-ID: <CAJZVxRnJ_YCqLXa_YGRRA2g+LZjvL_j-J6u400-4v4Mv+_JnUA@mail.gmail.com>

Hey guys

Fresh installation of bro 2.4.1, didn't modify scripts either

I'm experiencing similar issues, now I wanted to know, can we limit
the queue size (pending size)?

'Cause I don't care about packet losses

I already checked the help messages in bro, there's simply no such
option to configure, am I wrong?






On Fri, Oct 23, 2015 at 9:21 PM, Mike Waite <mfw113 at psu.edu> wrote:
> Patch applied, after 15 minutes I am seeing
>
> Oct 23 09:00:43 manager child   -       -       -       info
> selects=300000 canwrites=216206 pending=0
> Oct 23 09:01:29 manager child   -       -       -       info
> selects=400000 canwrites=216206 pending=0
> Oct 23 09:02:08 manager child   -       -       -       info
> selects=500000 canwrites=216552 pending=0
> Oct 23 09:02:49 manager child   -       -       -       info
> selects=600000 canwrites=216557 pending=0
> Oct 23 09:03:34 manager child   -       -       -       info
> selects=700000 canwrites=216557 pending=0
> Oct 23 09:04:29 manager child   -       -       -       info
> selects=800000 canwrites=255305 pending=4007
> Oct 23 09:05:21 manager child   -       -       -       info
> selects=900000 canwrites=355305 pending=6593
> Oct 23 09:06:13 manager child   -       -       -       info
> selects=1000000 canwrites=455305 pending=6003
> Oct 23 09:07:04 manager child   -       -       -       info
> selects=1100000 canwrites=555305 pending=3077
> Oct 23 09:07:55 manager child   -       -       -       info
> selects=1200000 canwrites=640438 pending=3399
> Oct 23 09:08:45 manager child   -       -       -       info
> selects=1300000 canwrites=740438 pending=3163
> Oct 23 09:09:36 manager child   -       -       -       info
> selects=1400000 canwrites=840438 pending=5245
> Oct 23 09:10:25 manager child   -       -       -       info
> selects=1500000 canwrites=940438 pending=6027
> Oct 23 09:11:15 manager child   -       -       -       info
> selects=1600000 canwrites=1040438 pending=6713
> Oct 23 09:12:01 manager child   -       -       -       info
> selects=1700000 canwrites=1140438 pending=5713
> Oct 23 09:12:50 manager child   -       -       -       info
> selects=1800000 canwrites=1240438 pending=6747
> Oct 23 09:13:39 manager child   -       -       -       info
> selects=1900000 canwrites=1340438 pending=7417
> Oct 23 09:14:32 manager child   -       -       -       info
> selects=2000000 canwrites=1440438 pending=13117
> Oct 23 09:15:10 manager child   -       -       -       info
> selects=2100000 canwrites=1540438 pending=20825
> Oct 23 09:15:59 manager child   -       -       -       info
> selects=2200000 canwrites=1640438 pending=18539
> Oct 23 09:16:47 manager child   -       -       -       info
> selects=2300000 canwrites=1740438 pending=15881
> Oct 23 09:17:35 manager child   -       -       -       info
> selects=2400000 canwrites=1840438 pending=15389
> Oct 23 09:18:28 manager child   -       -       -       info
> selects=2500000 canwrites=1940438 pending=16685
> Oct 23 09:19:18 manager child   -       -       -       info
> selects=2600000 canwrites=2040438 pending=17031
>
>
> I will let you know about the mem usage after a bit
>
> --
> Mike Waite
> CyberSecurity Intrusion Analyst
> Office of Information Security
> The Pennsylvania State University
> ? 15-10-22 10:22:18, Azoff, Justin S <jazoff at illinois.edu>:
>>>
>>> On Oct 22, 2015, at 8:12 AM, Mike Waite <mfw113 at psu.edu> wrote:
>>>
>>> I know we are still seeing issues with the manager child proccess.  The
>>> process will consume over 200GB of RAM in 8 hours.
>>>
>>
>> Give the attached patch a try.
>>
>>
>>
>> Monitor by using
>>
>>    cat logs/current/communication.log |egrep 'manager.child'
>>
>> And check to see if pending=0 or at least not growing.
>>
>>
>> --
>> - Justin Azoff
>>
>>
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
Best Regards,
Aaron Lewis - PGP: 0x13714D33 - http://pgp.mit.edu/
Finger Print:   9F67 391B B770 8FF6 99DC  D92D 87F6 2602 1371 4D33


From seth at icir.org  Fri Oct 23 09:14:12 2015
From: seth at icir.org (Seth Hall)
Date: Fri, 23 Oct 2015 12:14:12 -0400
Subject: [Bro] Memory Issue with Bro
In-Reply-To: <CAJZVxRnJ_YCqLXa_YGRRA2g+LZjvL_j-J6u400-4v4Mv+_JnUA@mail.gmail.com>
References: <69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com>
	<454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu>
	<7689d3f6ddb6454d855e6fd3a3b8ea7b@ocex03.SolarFlarecom.com>
	<3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org>
	<CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
	<DE5BED9E-D710-4763-A978-423D917509A5@illinois.edu>
	<CAMCSCoW6AisyP479iHN5pms+AcHPmFUAEdUvsJvhUPoRTNxD5w@mail.gmail.com>
	<AE9E9388-EC16-4040-A8CD-3752E1E3E3A8@illinois.edu>
	<20151022121254.GA26216@redquill.sos.its.psu.edu>
	<E7981188-57E2-4E65-8064-8E1A0FD24172@illinois.edu>
	<20151023132146.GA24138@redquill.sos.its.psu.edu>
	<CAJZVxRnJ_YCqLXa_YGRRA2g+LZjvL_j-J6u400-4v4Mv+_JnUA@mail.gmail.com>
Message-ID: <701534B9-425F-44AB-857D-0D7D17C32C91@icir.org>


> On Oct 23, 2015, at 11:29 AM, Aaron Lewis <the.warl0ck.1989 at gmail.com> wrote:
> 
> I'm experiencing similar issues, now I wanted to know, can we limit
> the queue size (pending size)?

This isn?t an issue with packet queueing.  It?s unfortunately quite a bit deeper than that and is related to Bro existing communication mechanism and a problem with it that we fairly recently became aware of.  We?re still digging into the solution for the problem.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From mfw113 at psu.edu  Fri Oct 23 09:38:52 2015
From: mfw113 at psu.edu (Mike Waite)
Date: Fri, 23 Oct 2015 12:38:52 -0400
Subject: [Bro] Memory Issue with Bro
In-Reply-To: <5A7DD598-CE76-4839-8156-67187148AA95@icir.org>
References: <3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org>
	<CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
	<DE5BED9E-D710-4763-A978-423D917509A5@illinois.edu>
	<CAMCSCoW6AisyP479iHN5pms+AcHPmFUAEdUvsJvhUPoRTNxD5w@mail.gmail.com>
	<AE9E9388-EC16-4040-A8CD-3752E1E3E3A8@illinois.edu>
	<20151022121254.GA26216@redquill.sos.its.psu.edu>
	<E7981188-57E2-4E65-8064-8E1A0FD24172@illinois.edu>
	<20151023132146.GA24138@redquill.sos.its.psu.edu>
	<97BEC855-2FD0-4F53-9F6F-DE6C1AA56FF7@illinois.edu>
	<5A7DD598-CE76-4839-8156-67187148AA95@icir.org>
Message-ID: <20151023163852.GB24138@redquill.sos.its.psu.edu>

After 30 min things look better, I will let you know how the rest of it makes out after a bit.

Oct 23 12:09:52	manager	child	-	-	-	info	selects=100000 canwrites=97046 pending=0
Oct 23 12:11:54	manager	child	-	-	-	info	selects=200000 canwrites=97046 pending=0
Oct 23 12:14:04	manager	child	-	-	-	info	selects=300000 canwrites=97046 pending=0
Oct 23 12:14:43	manager	child	-	-	-	info	selects=400000 canwrites=97046 pending=0
Oct 23 12:15:20	manager	child	-	-	-	info	selects=500000 canwrites=97046 pending=0
Oct 23 12:15:54	manager	child	-	-	-	info	selects=600000 canwrites=97046 pending=0
Oct 23 12:16:38	manager	child	-	-	-	info	selects=700000 canwrites=97046 pending=0
Oct 23 12:17:41	manager	child	-	-	-	info	selects=800000 canwrites=97046 pending=0
Oct 23 12:19:03	manager	child	-	-	-	info	selects=900000 canwrites=97046 pending=0
Oct 23 12:20:46	manager	child	-	-	-	info	selects=1000000 canwrites=97046 pending=0
Oct 23 12:23:04	manager	child	-	-	-	info	selects=1100000 canwrites=97046 pending=0
Oct 23 12:25:10	manager	child	-	-	-	info	selects=1200000 canwrites=104987 pending=0
Oct 23 12:26:40	manager	child	-	-	-	info	selects=1300000 canwrites=104987 pending=0
Oct 23 12:28:13	manager	child	-	-	-	info	selects=1400000 canwrites=104987 pending=0
Oct 23 12:31:12	manager	child	-	-	-	info	selects=1600000 canwrites=110134 pending=0
Oct 23 12:32:24	manager	child	-	-	-	info	selects=1700000 canwrites=110134 pending=0
Oct 23 12:34:03	manager	child	-	-	-	info	selects=1800000 canwrites=110134 pending=0
Oct 23 12:35:12	manager	child	-	-	-	info	selects=1900000 canwrites=110134 pending=0
Oct 23 12:36:15	manager	child	-	-	-	info	selects=2000000 canwrites=110134 pending=0
Oct 23 12:37:31	manager	child	-	-	-	info	selects=2100000 canwrites=110134 pending=0


--
Mike Waite
CyberSecurity Intrusion Analyst
Office of Information Security
The Pennsylvania State University
? 15-10-23 11:09:58, Seth Hall <seth at icir.org>:
>Mike, could you back out that patch and try my branch, topic/seth/remove-flare ?
>
>  .Seth
>
>
>> On Oct 23, 2015, at 10:19 AM, Azoff, Justin S <jazoff at illinois.edu> wrote:
>>
>> Well that doesn't look great, but could be a lot worse.  Hard to say without knowing what it looked like before the patch.
>>
>> The fact that pending ever goes down at all is a good sign, but pending=0 is really the optimal state.
>>
>> --
>> - Justin Azoff
>>
>>> On Oct 23, 2015, at 9:21 AM, Mike Waite <mfw113 at psu.edu> wrote:
>>>
>>> Patch applied, after 15 minutes I am seeing
>>>
>>> Oct 23 09:00:43	manager	child	-	-	-	info	selects=300000 canwrites=216206 pending=0
>>> Oct 23 09:01:29	manager	child	-	-	-	info	selects=400000 canwrites=216206 pending=0
>>> Oct 23 09:02:08	manager	child	-	-	-	info	selects=500000 canwrites=216552 pending=0
>>> Oct 23 09:02:49	manager	child	-	-	-	info	selects=600000 canwrites=216557 pending=0
>>> Oct 23 09:03:34	manager	child	-	-	-	info	selects=700000 canwrites=216557 pending=0
>>> Oct 23 09:04:29	manager	child	-	-	-	info	selects=800000 canwrites=255305 pending=4007
>>> Oct 23 09:05:21	manager	child	-	-	-	info	selects=900000 canwrites=355305 pending=6593
>>> Oct 23 09:06:13	manager	child	-	-	-	info	selects=1000000 canwrites=455305 pending=6003
>>> Oct 23 09:07:04	manager	child	-	-	-	info	selects=1100000 canwrites=555305 pending=3077
>>> Oct 23 09:07:55	manager	child	-	-	-	info	selects=1200000 canwrites=640438 pending=3399
>>> Oct 23 09:08:45	manager	child	-	-	-	info	selects=1300000 canwrites=740438 pending=3163
>>> Oct 23 09:09:36	manager	child	-	-	-	info	selects=1400000 canwrites=840438 pending=5245
>>> Oct 23 09:10:25	manager	child	-	-	-	info	selects=1500000 canwrites=940438 pending=6027
>>> Oct 23 09:11:15	manager	child	-	-	-	info	selects=1600000 canwrites=1040438 pending=6713
>>> Oct 23 09:12:01	manager	child	-	-	-	info	selects=1700000 canwrites=1140438 pending=5713
>>> Oct 23 09:12:50	manager	child	-	-	-	info	selects=1800000 canwrites=1240438 pending=6747
>>> Oct 23 09:13:39	manager	child	-	-	-	info	selects=1900000 canwrites=1340438 pending=7417
>>> Oct 23 09:14:32	manager	child	-	-	-	info	selects=2000000 canwrites=1440438 pending=13117
>>> Oct 23 09:15:10	manager	child	-	-	-	info	selects=2100000 canwrites=1540438 pending=20825
>>> Oct 23 09:15:59	manager	child	-	-	-	info	selects=2200000 canwrites=1640438 pending=18539
>>> Oct 23 09:16:47	manager	child	-	-	-	info	selects=2300000 canwrites=1740438 pending=15881
>>> Oct 23 09:17:35	manager	child	-	-	-	info	selects=2400000 canwrites=1840438 pending=15389
>>> Oct 23 09:18:28	manager	child	-	-	-	info	selects=2500000 canwrites=1940438 pending=16685
>>> Oct 23 09:19:18	manager	child	-	-	-	info	selects=2600000 canwrites=2040438 pending=17031
>>>
>>>
>>> I will let you know about the mem usage after a bit
>>>
>>> --
>>> Mike Waite
>>> CyberSecurity Intrusion Analyst
>>> Office of Information Security
>>> The Pennsylvania State University
>>> ? 15-10-22 10:22:18, Azoff, Justin S <jazoff at illinois.edu>:
>>>>> On Oct 22, 2015, at 8:12 AM, Mike Waite <mfw113 at psu.edu> wrote:
>>>>>
>>>>> I know we are still seeing issues with the manager child proccess.  The process will consume over 200GB of RAM in 8 hours.
>>>>>
>>>>
>>>> Give the attached patch a try.
>>>>
>>>>
>>>>
>>>> Monitor by using
>>>>
>>>>  cat logs/current/communication.log |egrep 'manager.child'
>>>>
>>>> And check to see if pending=0 or at least not growing.
>>>>
>>>>
>>>> --
>>>> - Justin Azoff
>>>>
>>>>
>>>
>>>
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>
>--
>Seth Hall
>International Computer Science Institute
>(Bro) because everyone has a network
>http://www.bro.org/
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151023/9a761c5e/attachment.bin 

From martin at gta.ufrj.br  Fri Oct 23 09:56:59 2015
From: martin at gta.ufrj.br (Martin Andreoni)
Date: Fri, 23 Oct 2015 14:56:59 -0200
Subject: [Bro] Origin-Destination (OD) Flows
Message-ID: <c8f047570be2e14a9653c11fdc2de335@gta.ufrj.br>

 
Hi Community! 
I would like to make a Origen-Destination (OD) flow analysis with Bro. I
need to take flow parameters such as a netflow output. 
 I have been reading and with the conn_stats event I need to wait until
the TCP connection is closed. 
Is there any other event or way to make an on-line analysis of OD flows?

-- 

-------------------------

MARTIN ANDREONI 

_PhD. Student at __Grupo de Teleinform?tica e Automa??o (GTA) _

_Universidade Federal do Rio de Janeiro (UFRJ)_

_www.gta.ufrj.br/~martin [1]_
 

Links:
------
[1] http://www.gta.ufrj.br/~martin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151023/6d650c78/attachment-0001.html 

From ericooi at gmail.com  Sat Oct 24 07:39:43 2015
From: ericooi at gmail.com (Eric Ooi)
Date: Sat, 24 Oct 2015 09:39:43 -0500
Subject: [Bro] Memory Issue with Bro
In-Reply-To: <20151023163852.GB24138@redquill.sos.its.psu.edu>
References: <3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org>
	<CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
	<DE5BED9E-D710-4763-A978-423D917509A5@illinois.edu>
	<CAMCSCoW6AisyP479iHN5pms+AcHPmFUAEdUvsJvhUPoRTNxD5w@mail.gmail.com>
	<AE9E9388-EC16-4040-A8CD-3752E1E3E3A8@illinois.edu>
	<20151022121254.GA26216@redquill.sos.its.psu.edu>
	<E7981188-57E2-4E65-8064-8E1A0FD24172@illinois.edu>
	<20151023132146.GA24138@redquill.sos.its.psu.edu>
	<97BEC855-2FD0-4F53-9F6F-DE6C1AA56FF7@illinois.edu>
	<5A7DD598-CE76-4839-8156-67187148AA95@icir.org>
	<20151023163852.GB24138@redquill.sos.its.psu.edu>
Message-ID: <C63AA2D4-8DAC-406E-8FF1-70242A31FF0B@gmail.com>

I have two Security Onion sensors running 2.4, one monitors a combination of 100M general office internet traffic + 10G network, the other monitors four 1G networks which includes a publicly available website and lots of syslog and SMTP traffic.  I ran the default Security Onion configuration on both and noticed the sensor monitoring four 1G networks would run out of memory over the course of a few hours.  I spent a day turning off various analyzers until I isolated it to the intel analyzer.  Ever since I turned intel.log off for that sensor, it?s run great for weeks.  I tried adding intel feeds (via CriticalStack) and using a blank intel file, with no luck.  Simply having the intel analyzer on always resulted in memory loss over time.  I?m guessing it has something to do with the type of traffic that particular sensor sees (more HTTP, syslog, and SMTP), but I?m not entirely sure.

Don?t know if it?s related, but just thought I?d share my experience with Bro memory issues.

Eric

> On Oct 23, 2015, at 11:38 AM, Mike Waite <mfw113 at psu.edu> wrote:
> 
> After 30 min things look better, I will let you know how the rest of it makes out after a bit.
> 
> Oct 23 12:09:52	manager	child	-	-	-	info	selects=100000 canwrites=97046 pending=0
> Oct 23 12:11:54	manager	child	-	-	-	info	selects=200000 canwrites=97046 pending=0
> Oct 23 12:14:04	manager	child	-	-	-	info	selects=300000 canwrites=97046 pending=0
> Oct 23 12:14:43	manager	child	-	-	-	info	selects=400000 canwrites=97046 pending=0
> Oct 23 12:15:20	manager	child	-	-	-	info	selects=500000 canwrites=97046 pending=0
> Oct 23 12:15:54	manager	child	-	-	-	info	selects=600000 canwrites=97046 pending=0
> Oct 23 12:16:38	manager	child	-	-	-	info	selects=700000 canwrites=97046 pending=0
> Oct 23 12:17:41	manager	child	-	-	-	info	selects=800000 canwrites=97046 pending=0
> Oct 23 12:19:03	manager	child	-	-	-	info	selects=900000 canwrites=97046 pending=0
> Oct 23 12:20:46	manager	child	-	-	-	info	selects=1000000 canwrites=97046 pending=0
> Oct 23 12:23:04	manager	child	-	-	-	info	selects=1100000 canwrites=97046 pending=0
> Oct 23 12:25:10	manager	child	-	-	-	info	selects=1200000 canwrites=104987 pending=0
> Oct 23 12:26:40	manager	child	-	-	-	info	selects=1300000 canwrites=104987 pending=0
> Oct 23 12:28:13	manager	child	-	-	-	info	selects=1400000 canwrites=104987 pending=0
> Oct 23 12:31:12	manager	child	-	-	-	info	selects=1600000 canwrites=110134 pending=0
> Oct 23 12:32:24	manager	child	-	-	-	info	selects=1700000 canwrites=110134 pending=0
> Oct 23 12:34:03	manager	child	-	-	-	info	selects=1800000 canwrites=110134 pending=0
> Oct 23 12:35:12	manager	child	-	-	-	info	selects=1900000 canwrites=110134 pending=0
> Oct 23 12:36:15	manager	child	-	-	-	info	selects=2000000 canwrites=110134 pending=0
> Oct 23 12:37:31	manager	child	-	-	-	info	selects=2100000 canwrites=110134 pending=0
> 
> 
> --
> Mike Waite
> CyberSecurity Intrusion Analyst
> Office of Information Security
> The Pennsylvania State University
> ? 15-10-23 11:09:58, Seth Hall <seth at icir.org>:
>> Mike, could you back out that patch and try my branch, topic/seth/remove-flare ?
>> 
>> .Seth
>> 
>> 
>>> On Oct 23, 2015, at 10:19 AM, Azoff, Justin S <jazoff at illinois.edu> wrote:
>>> 
>>> Well that doesn't look great, but could be a lot worse.  Hard to say without knowing what it looked like before the patch.
>>> 
>>> The fact that pending ever goes down at all is a good sign, but pending=0 is really the optimal state.
>>> 
>>> --
>>> - Justin Azoff
>>> 
>>>> On Oct 23, 2015, at 9:21 AM, Mike Waite <mfw113 at psu.edu> wrote:
>>>> 
>>>> Patch applied, after 15 minutes I am seeing
>>>> 
>>>> Oct 23 09:00:43	manager	child	-	-	-	info	selects=300000 canwrites=216206 pending=0
>>>> Oct 23 09:01:29	manager	child	-	-	-	info	selects=400000 canwrites=216206 pending=0
>>>> Oct 23 09:02:08	manager	child	-	-	-	info	selects=500000 canwrites=216552 pending=0
>>>> Oct 23 09:02:49	manager	child	-	-	-	info	selects=600000 canwrites=216557 pending=0
>>>> Oct 23 09:03:34	manager	child	-	-	-	info	selects=700000 canwrites=216557 pending=0
>>>> Oct 23 09:04:29	manager	child	-	-	-	info	selects=800000 canwrites=255305 pending=4007
>>>> Oct 23 09:05:21	manager	child	-	-	-	info	selects=900000 canwrites=355305 pending=6593
>>>> Oct 23 09:06:13	manager	child	-	-	-	info	selects=1000000 canwrites=455305 pending=6003
>>>> Oct 23 09:07:04	manager	child	-	-	-	info	selects=1100000 canwrites=555305 pending=3077
>>>> Oct 23 09:07:55	manager	child	-	-	-	info	selects=1200000 canwrites=640438 pending=3399
>>>> Oct 23 09:08:45	manager	child	-	-	-	info	selects=1300000 canwrites=740438 pending=3163
>>>> Oct 23 09:09:36	manager	child	-	-	-	info	selects=1400000 canwrites=840438 pending=5245
>>>> Oct 23 09:10:25	manager	child	-	-	-	info	selects=1500000 canwrites=940438 pending=6027
>>>> Oct 23 09:11:15	manager	child	-	-	-	info	selects=1600000 canwrites=1040438 pending=6713
>>>> Oct 23 09:12:01	manager	child	-	-	-	info	selects=1700000 canwrites=1140438 pending=5713
>>>> Oct 23 09:12:50	manager	child	-	-	-	info	selects=1800000 canwrites=1240438 pending=6747
>>>> Oct 23 09:13:39	manager	child	-	-	-	info	selects=1900000 canwrites=1340438 pending=7417
>>>> Oct 23 09:14:32	manager	child	-	-	-	info	selects=2000000 canwrites=1440438 pending=13117
>>>> Oct 23 09:15:10	manager	child	-	-	-	info	selects=2100000 canwrites=1540438 pending=20825
>>>> Oct 23 09:15:59	manager	child	-	-	-	info	selects=2200000 canwrites=1640438 pending=18539
>>>> Oct 23 09:16:47	manager	child	-	-	-	info	selects=2300000 canwrites=1740438 pending=15881
>>>> Oct 23 09:17:35	manager	child	-	-	-	info	selects=2400000 canwrites=1840438 pending=15389
>>>> Oct 23 09:18:28	manager	child	-	-	-	info	selects=2500000 canwrites=1940438 pending=16685
>>>> Oct 23 09:19:18	manager	child	-	-	-	info	selects=2600000 canwrites=2040438 pending=17031
>>>> 
>>>> 
>>>> I will let you know about the mem usage after a bit
>>>> 
>>>> --
>>>> Mike Waite
>>>> CyberSecurity Intrusion Analyst
>>>> Office of Information Security
>>>> The Pennsylvania State University
>>>> ? 15-10-22 10:22:18, Azoff, Justin S <jazoff at illinois.edu>:
>>>>>> On Oct 22, 2015, at 8:12 AM, Mike Waite <mfw113 at psu.edu> wrote:
>>>>>> 
>>>>>> I know we are still seeing issues with the manager child proccess.  The process will consume over 200GB of RAM in 8 hours.
>>>>>> 
>>>>> 
>>>>> Give the attached patch a try.
>>>>> 
>>>>> 
>>>>> 
>>>>> Monitor by using
>>>>> 
>>>>> cat logs/current/communication.log |egrep 'manager.child'
>>>>> 
>>>>> And check to see if pending=0 or at least not growing.
>>>>> 
>>>>> 
>>>>> --
>>>>> - Justin Azoff
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>> 
>>> 
>> 
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro.org/
>> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From the.warl0ck.1989 at gmail.com  Sat Oct 24 08:51:40 2015
From: the.warl0ck.1989 at gmail.com (Aaron Lewis)
Date: Sat, 24 Oct 2015 23:51:40 +0800
Subject: [Bro] Memory Issue with Bro
In-Reply-To: <C63AA2D4-8DAC-406E-8FF1-70242A31FF0B@gmail.com>
References: <3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org>
	<CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
	<DE5BED9E-D710-4763-A978-423D917509A5@illinois.edu>
	<CAMCSCoW6AisyP479iHN5pms+AcHPmFUAEdUvsJvhUPoRTNxD5w@mail.gmail.com>
	<AE9E9388-EC16-4040-A8CD-3752E1E3E3A8@illinois.edu>
	<20151022121254.GA26216@redquill.sos.its.psu.edu>
	<E7981188-57E2-4E65-8064-8E1A0FD24172@illinois.edu>
	<20151023132146.GA24138@redquill.sos.its.psu.edu>
	<97BEC855-2FD0-4F53-9F6F-DE6C1AA56FF7@illinois.edu>
	<5A7DD598-CE76-4839-8156-67187148AA95@icir.org>
	<20151023163852.GB24138@redquill.sos.its.psu.edu>
	<C63AA2D4-8DAC-406E-8FF1-70242A31FF0B@gmail.com>
Message-ID: <CAJZVxR=KY3AnvFYraT598AY=S+aTkWYYgM=cNhPO3M1pJ8aSSg@mail.gmail.com>

Hey Eric,

How exactly did you turn off intel log? I tried to comment out this
line, but nothing changed on my sensor,

@load policy/frameworks/intel/seen

Am I wrong? I was following this link:
https://www.bro.org/sphinx/frameworks/intel.html



On Sat, Oct 24, 2015 at 10:39 PM, Eric Ooi <ericooi at gmail.com> wrote:
> I have two Security Onion sensors running 2.4, one monitors a combination of 100M general office internet traffic + 10G network, the other monitors four 1G networks which includes a publicly available website and lots of syslog and SMTP traffic.  I ran the default Security Onion configuration on both and noticed the sensor monitoring four 1G networks would run out of memory over the course of a few hours.  I spent a day turning off various analyzers until I isolated it to the intel analyzer.  Ever since I turned intel.log off for that sensor, it?s run great for weeks.  I tried adding intel feeds (via CriticalStack) and using a blank intel file, with no luck.  Simply having the intel analyzer on always resulted in memory loss over time.  I?m guessing it has something to do with the type of traffic that particular sensor sees (more HTTP, syslog, and SMTP), but I?m not entirely sure.
>
> Don?t know if it?s related, but just thought I?d share my experience with Bro memory issues.
>
> Eric
>
>> On Oct 23, 2015, at 11:38 AM, Mike Waite <mfw113 at psu.edu> wrote:
>>
>> After 30 min things look better, I will let you know how the rest of it makes out after a bit.
>>
>> Oct 23 12:09:52       manager child   -       -       -       info    selects=100000 canwrites=97046 pending=0
>> Oct 23 12:11:54       manager child   -       -       -       info    selects=200000 canwrites=97046 pending=0
>> Oct 23 12:14:04       manager child   -       -       -       info    selects=300000 canwrites=97046 pending=0
>> Oct 23 12:14:43       manager child   -       -       -       info    selects=400000 canwrites=97046 pending=0
>> Oct 23 12:15:20       manager child   -       -       -       info    selects=500000 canwrites=97046 pending=0
>> Oct 23 12:15:54       manager child   -       -       -       info    selects=600000 canwrites=97046 pending=0
>> Oct 23 12:16:38       manager child   -       -       -       info    selects=700000 canwrites=97046 pending=0
>> Oct 23 12:17:41       manager child   -       -       -       info    selects=800000 canwrites=97046 pending=0
>> Oct 23 12:19:03       manager child   -       -       -       info    selects=900000 canwrites=97046 pending=0
>> Oct 23 12:20:46       manager child   -       -       -       info    selects=1000000 canwrites=97046 pending=0
>> Oct 23 12:23:04       manager child   -       -       -       info    selects=1100000 canwrites=97046 pending=0
>> Oct 23 12:25:10       manager child   -       -       -       info    selects=1200000 canwrites=104987 pending=0
>> Oct 23 12:26:40       manager child   -       -       -       info    selects=1300000 canwrites=104987 pending=0
>> Oct 23 12:28:13       manager child   -       -       -       info    selects=1400000 canwrites=104987 pending=0
>> Oct 23 12:31:12       manager child   -       -       -       info    selects=1600000 canwrites=110134 pending=0
>> Oct 23 12:32:24       manager child   -       -       -       info    selects=1700000 canwrites=110134 pending=0
>> Oct 23 12:34:03       manager child   -       -       -       info    selects=1800000 canwrites=110134 pending=0
>> Oct 23 12:35:12       manager child   -       -       -       info    selects=1900000 canwrites=110134 pending=0
>> Oct 23 12:36:15       manager child   -       -       -       info    selects=2000000 canwrites=110134 pending=0
>> Oct 23 12:37:31       manager child   -       -       -       info    selects=2100000 canwrites=110134 pending=0
>>
>>
>> --
>> Mike Waite
>> CyberSecurity Intrusion Analyst
>> Office of Information Security
>> The Pennsylvania State University
>> ? 15-10-23 11:09:58, Seth Hall <seth at icir.org>:
>>> Mike, could you back out that patch and try my branch, topic/seth/remove-flare ?
>>>
>>> .Seth
>>>
>>>
>>>> On Oct 23, 2015, at 10:19 AM, Azoff, Justin S <jazoff at illinois.edu> wrote:
>>>>
>>>> Well that doesn't look great, but could be a lot worse.  Hard to say without knowing what it looked like before the patch.
>>>>
>>>> The fact that pending ever goes down at all is a good sign, but pending=0 is really the optimal state.
>>>>
>>>> --
>>>> - Justin Azoff
>>>>
>>>>> On Oct 23, 2015, at 9:21 AM, Mike Waite <mfw113 at psu.edu> wrote:
>>>>>
>>>>> Patch applied, after 15 minutes I am seeing
>>>>>
>>>>> Oct 23 09:00:43    manager child   -       -       -       info    selects=300000 canwrites=216206 pending=0
>>>>> Oct 23 09:01:29    manager child   -       -       -       info    selects=400000 canwrites=216206 pending=0
>>>>> Oct 23 09:02:08    manager child   -       -       -       info    selects=500000 canwrites=216552 pending=0
>>>>> Oct 23 09:02:49    manager child   -       -       -       info    selects=600000 canwrites=216557 pending=0
>>>>> Oct 23 09:03:34    manager child   -       -       -       info    selects=700000 canwrites=216557 pending=0
>>>>> Oct 23 09:04:29    manager child   -       -       -       info    selects=800000 canwrites=255305 pending=4007
>>>>> Oct 23 09:05:21    manager child   -       -       -       info    selects=900000 canwrites=355305 pending=6593
>>>>> Oct 23 09:06:13    manager child   -       -       -       info    selects=1000000 canwrites=455305 pending=6003
>>>>> Oct 23 09:07:04    manager child   -       -       -       info    selects=1100000 canwrites=555305 pending=3077
>>>>> Oct 23 09:07:55    manager child   -       -       -       info    selects=1200000 canwrites=640438 pending=3399
>>>>> Oct 23 09:08:45    manager child   -       -       -       info    selects=1300000 canwrites=740438 pending=3163
>>>>> Oct 23 09:09:36    manager child   -       -       -       info    selects=1400000 canwrites=840438 pending=5245
>>>>> Oct 23 09:10:25    manager child   -       -       -       info    selects=1500000 canwrites=940438 pending=6027
>>>>> Oct 23 09:11:15    manager child   -       -       -       info    selects=1600000 canwrites=1040438 pending=6713
>>>>> Oct 23 09:12:01    manager child   -       -       -       info    selects=1700000 canwrites=1140438 pending=5713
>>>>> Oct 23 09:12:50    manager child   -       -       -       info    selects=1800000 canwrites=1240438 pending=6747
>>>>> Oct 23 09:13:39    manager child   -       -       -       info    selects=1900000 canwrites=1340438 pending=7417
>>>>> Oct 23 09:14:32    manager child   -       -       -       info    selects=2000000 canwrites=1440438 pending=13117
>>>>> Oct 23 09:15:10    manager child   -       -       -       info    selects=2100000 canwrites=1540438 pending=20825
>>>>> Oct 23 09:15:59    manager child   -       -       -       info    selects=2200000 canwrites=1640438 pending=18539
>>>>> Oct 23 09:16:47    manager child   -       -       -       info    selects=2300000 canwrites=1740438 pending=15881
>>>>> Oct 23 09:17:35    manager child   -       -       -       info    selects=2400000 canwrites=1840438 pending=15389
>>>>> Oct 23 09:18:28    manager child   -       -       -       info    selects=2500000 canwrites=1940438 pending=16685
>>>>> Oct 23 09:19:18    manager child   -       -       -       info    selects=2600000 canwrites=2040438 pending=17031
>>>>>
>>>>>
>>>>> I will let you know about the mem usage after a bit
>>>>>
>>>>> --
>>>>> Mike Waite
>>>>> CyberSecurity Intrusion Analyst
>>>>> Office of Information Security
>>>>> The Pennsylvania State University
>>>>> ? 15-10-22 10:22:18, Azoff, Justin S <jazoff at illinois.edu>:
>>>>>>> On Oct 22, 2015, at 8:12 AM, Mike Waite <mfw113 at psu.edu> wrote:
>>>>>>>
>>>>>>> I know we are still seeing issues with the manager child proccess.  The process will consume over 200GB of RAM in 8 hours.
>>>>>>>
>>>>>>
>>>>>> Give the attached patch a try.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Monitor by using
>>>>>>
>>>>>> cat logs/current/communication.log |egrep 'manager.child'
>>>>>>
>>>>>> And check to see if pending=0 or at least not growing.
>>>>>>
>>>>>>
>>>>>> --
>>>>>> - Justin Azoff
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Bro mailing list
>>>> bro at bro-ids.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>>
>>>>
>>>
>>> --
>>> Seth Hall
>>> International Computer Science Institute
>>> (Bro) because everyone has a network
>>> http://www.bro.org/
>>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
Best Regards,
Aaron Lewis - PGP: 0x13714D33 - http://pgp.mit.edu/
Finger Print:   9F67 391B B770 8FF6 99DC  D92D 87F6 2602 1371 4D33


From ericooi at gmail.com  Sat Oct 24 09:36:01 2015
From: ericooi at gmail.com (Eric Ooi)
Date: Sat, 24 Oct 2015 11:36:01 -0500
Subject: [Bro] Memory Issue with Bro
In-Reply-To: <CAJZVxR=KY3AnvFYraT598AY=S+aTkWYYgM=cNhPO3M1pJ8aSSg@mail.gmail.com>
References: <3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org>
	<CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
	<DE5BED9E-D710-4763-A978-423D917509A5@illinois.edu>
	<CAMCSCoW6AisyP479iHN5pms+AcHPmFUAEdUvsJvhUPoRTNxD5w@mail.gmail.com>
	<AE9E9388-EC16-4040-A8CD-3752E1E3E3A8@illinois.edu>
	<20151022121254.GA26216@redquill.sos.its.psu.edu>
	<E7981188-57E2-4E65-8064-8E1A0FD24172@illinois.edu>
	<20151023132146.GA24138@redquill.sos.its.psu.edu>
	<97BEC855-2FD0-4F53-9F6F-DE6C1AA56FF7@illinois.edu>
	<5A7DD598-CE76-4839-8156-67187148AA95@icir.org>
	<20151023163852.GB24138@redquill.sos.its.psu.edu>
	<C63AA2D4-8DAC-406E-8FF1-70242A31FF0B@gmail.com>
	<CAJZVxR=KY3AnvFYraT598AY=S+aTkWYYgM=cNhPO3M1pJ8aSSg@mail.gmail.com>
Message-ID: <CAEuvLSaZwSEOaCGGi_U4UtUK8tTSBET8rXQwExL96YbZXBgMug@mail.gmail.com>

Hey Aaron,

I did it by commenting out "@load intel"
in /opt/bro/share/bro/site/local.bro.  Give that a try and see if it works
for you.

Eric

On Sat, Oct 24, 2015 at 10:51 AM, Aaron Lewis <the.warl0ck.1989 at gmail.com>
wrote:

> Hey Eric,
>
> How exactly did you turn off intel log? I tried to comment out this
> line, but nothing changed on my sensor,
>
> @load policy/frameworks/intel/seen
>
> Am I wrong? I was following this link:
> https://www.bro.org/sphinx/frameworks/intel.html
>
>
>
> On Sat, Oct 24, 2015 at 10:39 PM, Eric Ooi <ericooi at gmail.com> wrote:
> > I have two Security Onion sensors running 2.4, one monitors a
> combination of 100M general office internet traffic + 10G network, the
> other monitors four 1G networks which includes a publicly available website
> and lots of syslog and SMTP traffic.  I ran the default Security Onion
> configuration on both and noticed the sensor monitoring four 1G networks
> would run out of memory over the course of a few hours.  I spent a day
> turning off various analyzers until I isolated it to the intel analyzer.
> Ever since I turned intel.log off for that sensor, it?s run great for
> weeks.  I tried adding intel feeds (via CriticalStack) and using a blank
> intel file, with no luck.  Simply having the intel analyzer on always
> resulted in memory loss over time.  I?m guessing it has something to do
> with the type of traffic that particular sensor sees (more HTTP, syslog,
> and SMTP), but I?m not entirely sure.
> >
> > Don?t know if it?s related, but just thought I?d share my experience
> with Bro memory issues.
> >
> > Eric
> >
> >> On Oct 23, 2015, at 11:38 AM, Mike Waite <mfw113 at psu.edu> wrote:
> >>
> >> After 30 min things look better, I will let you know how the rest of it
> makes out after a bit.
> >>
> >> Oct 23 12:09:52       manager child   -       -       -       info
> selects=100000 canwrites=97046 pending=0
> >> Oct 23 12:11:54       manager child   -       -       -       info
> selects=200000 canwrites=97046 pending=0
> >> Oct 23 12:14:04       manager child   -       -       -       info
> selects=300000 canwrites=97046 pending=0
> >> Oct 23 12:14:43       manager child   -       -       -       info
> selects=400000 canwrites=97046 pending=0
> >> Oct 23 12:15:20       manager child   -       -       -       info
> selects=500000 canwrites=97046 pending=0
> >> Oct 23 12:15:54       manager child   -       -       -       info
> selects=600000 canwrites=97046 pending=0
> >> Oct 23 12:16:38       manager child   -       -       -       info
> selects=700000 canwrites=97046 pending=0
> >> Oct 23 12:17:41       manager child   -       -       -       info
> selects=800000 canwrites=97046 pending=0
> >> Oct 23 12:19:03       manager child   -       -       -       info
> selects=900000 canwrites=97046 pending=0
> >> Oct 23 12:20:46       manager child   -       -       -       info
> selects=1000000 canwrites=97046 pending=0
> >> Oct 23 12:23:04       manager child   -       -       -       info
> selects=1100000 canwrites=97046 pending=0
> >> Oct 23 12:25:10       manager child   -       -       -       info
> selects=1200000 canwrites=104987 pending=0
> >> Oct 23 12:26:40       manager child   -       -       -       info
> selects=1300000 canwrites=104987 pending=0
> >> Oct 23 12:28:13       manager child   -       -       -       info
> selects=1400000 canwrites=104987 pending=0
> >> Oct 23 12:31:12       manager child   -       -       -       info
> selects=1600000 canwrites=110134 pending=0
> >> Oct 23 12:32:24       manager child   -       -       -       info
> selects=1700000 canwrites=110134 pending=0
> >> Oct 23 12:34:03       manager child   -       -       -       info
> selects=1800000 canwrites=110134 pending=0
> >> Oct 23 12:35:12       manager child   -       -       -       info
> selects=1900000 canwrites=110134 pending=0
> >> Oct 23 12:36:15       manager child   -       -       -       info
> selects=2000000 canwrites=110134 pending=0
> >> Oct 23 12:37:31       manager child   -       -       -       info
> selects=2100000 canwrites=110134 pending=0
> >>
> >>
> >> --
> >> Mike Waite
> >> CyberSecurity Intrusion Analyst
> >> Office of Information Security
> >> The Pennsylvania State University
> >> ? 15-10-23 11:09:58, Seth Hall <seth at icir.org>:
> >>> Mike, could you back out that patch and try my branch,
> topic/seth/remove-flare ?
> >>>
> >>> .Seth
> >>>
> >>>
> >>>> On Oct 23, 2015, at 10:19 AM, Azoff, Justin S <jazoff at illinois.edu>
> wrote:
> >>>>
> >>>> Well that doesn't look great, but could be a lot worse.  Hard to say
> without knowing what it looked like before the patch.
> >>>>
> >>>> The fact that pending ever goes down at all is a good sign, but
> pending=0 is really the optimal state.
> >>>>
> >>>> --
> >>>> - Justin Azoff
> >>>>
> >>>>> On Oct 23, 2015, at 9:21 AM, Mike Waite <mfw113 at psu.edu> wrote:
> >>>>>
> >>>>> Patch applied, after 15 minutes I am seeing
> >>>>>
> >>>>> Oct 23 09:00:43    manager child   -       -       -       info
> selects=300000 canwrites=216206 pending=0
> >>>>> Oct 23 09:01:29    manager child   -       -       -       info
> selects=400000 canwrites=216206 pending=0
> >>>>> Oct 23 09:02:08    manager child   -       -       -       info
> selects=500000 canwrites=216552 pending=0
> >>>>> Oct 23 09:02:49    manager child   -       -       -       info
> selects=600000 canwrites=216557 pending=0
> >>>>> Oct 23 09:03:34    manager child   -       -       -       info
> selects=700000 canwrites=216557 pending=0
> >>>>> Oct 23 09:04:29    manager child   -       -       -       info
> selects=800000 canwrites=255305 pending=4007
> >>>>> Oct 23 09:05:21    manager child   -       -       -       info
> selects=900000 canwrites=355305 pending=6593
> >>>>> Oct 23 09:06:13    manager child   -       -       -       info
> selects=1000000 canwrites=455305 pending=6003
> >>>>> Oct 23 09:07:04    manager child   -       -       -       info
> selects=1100000 canwrites=555305 pending=3077
> >>>>> Oct 23 09:07:55    manager child   -       -       -       info
> selects=1200000 canwrites=640438 pending=3399
> >>>>> Oct 23 09:08:45    manager child   -       -       -       info
> selects=1300000 canwrites=740438 pending=3163
> >>>>> Oct 23 09:09:36    manager child   -       -       -       info
> selects=1400000 canwrites=840438 pending=5245
> >>>>> Oct 23 09:10:25    manager child   -       -       -       info
> selects=1500000 canwrites=940438 pending=6027
> >>>>> Oct 23 09:11:15    manager child   -       -       -       info
> selects=1600000 canwrites=1040438 pending=6713
> >>>>> Oct 23 09:12:01    manager child   -       -       -       info
> selects=1700000 canwrites=1140438 pending=5713
> >>>>> Oct 23 09:12:50    manager child   -       -       -       info
> selects=1800000 canwrites=1240438 pending=6747
> >>>>> Oct 23 09:13:39    manager child   -       -       -       info
> selects=1900000 canwrites=1340438 pending=7417
> >>>>> Oct 23 09:14:32    manager child   -       -       -       info
> selects=2000000 canwrites=1440438 pending=13117
> >>>>> Oct 23 09:15:10    manager child   -       -       -       info
> selects=2100000 canwrites=1540438 pending=20825
> >>>>> Oct 23 09:15:59    manager child   -       -       -       info
> selects=2200000 canwrites=1640438 pending=18539
> >>>>> Oct 23 09:16:47    manager child   -       -       -       info
> selects=2300000 canwrites=1740438 pending=15881
> >>>>> Oct 23 09:17:35    manager child   -       -       -       info
> selects=2400000 canwrites=1840438 pending=15389
> >>>>> Oct 23 09:18:28    manager child   -       -       -       info
> selects=2500000 canwrites=1940438 pending=16685
> >>>>> Oct 23 09:19:18    manager child   -       -       -       info
> selects=2600000 canwrites=2040438 pending=17031
> >>>>>
> >>>>>
> >>>>> I will let you know about the mem usage after a bit
> >>>>>
> >>>>> --
> >>>>> Mike Waite
> >>>>> CyberSecurity Intrusion Analyst
> >>>>> Office of Information Security
> >>>>> The Pennsylvania State University
> >>>>> ? 15-10-22 10:22:18, Azoff, Justin S <jazoff at illinois.edu>:
> >>>>>>> On Oct 22, 2015, at 8:12 AM, Mike Waite <mfw113 at psu.edu> wrote:
> >>>>>>>
> >>>>>>> I know we are still seeing issues with the manager child
> proccess.  The process will consume over 200GB of RAM in 8 hours.
> >>>>>>>
> >>>>>>
> >>>>>> Give the attached patch a try.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Monitor by using
> >>>>>>
> >>>>>> cat logs/current/communication.log |egrep 'manager.child'
> >>>>>>
> >>>>>> And check to see if pending=0 or at least not growing.
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> - Justin Azoff
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Bro mailing list
> >>>> bro at bro-ids.org
> >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >>>>
> >>>>
> >>>
> >>> --
> >>> Seth Hall
> >>> International Computer Science Institute
> >>> (Bro) because everyone has a network
> >>> http://www.bro.org/
> >>>
> >> _______________________________________________
> >> Bro mailing list
> >> bro at bro-ids.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> --
> Best Regards,
> Aaron Lewis - PGP: 0x13714D33 - http://pgp.mit.edu/
> Finger Print:   9F67 391B B770 8FF6 99DC  D92D 87F6 2602 1371 4D33
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151024/f79d028c/attachment-0001.html 

From the.warl0ck.1989 at gmail.com  Sun Oct 25 22:21:03 2015
From: the.warl0ck.1989 at gmail.com (Aaron Lewis)
Date: Mon, 26 Oct 2015 13:21:03 +0800
Subject: [Bro] Memory Issue with Bro
In-Reply-To: <CAEuvLSaZwSEOaCGGi_U4UtUK8tTSBET8rXQwExL96YbZXBgMug@mail.gmail.com>
References: <3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org>
	<CAMCSCoVjBu6ErqEd50W_-FnNFhOQEJ16jHziZCqrfkR9bSZjEQ@mail.gmail.com>
	<DE5BED9E-D710-4763-A978-423D917509A5@illinois.edu>
	<CAMCSCoW6AisyP479iHN5pms+AcHPmFUAEdUvsJvhUPoRTNxD5w@mail.gmail.com>
	<AE9E9388-EC16-4040-A8CD-3752E1E3E3A8@illinois.edu>
	<20151022121254.GA26216@redquill.sos.its.psu.edu>
	<E7981188-57E2-4E65-8064-8E1A0FD24172@illinois.edu>
	<20151023132146.GA24138@redquill.sos.its.psu.edu>
	<97BEC855-2FD0-4F53-9F6F-DE6C1AA56FF7@illinois.edu>
	<5A7DD598-CE76-4839-8156-67187148AA95@icir.org>
	<20151023163852.GB24138@redquill.sos.its.psu.edu>
	<C63AA2D4-8DAC-406E-8FF1-70242A31FF0B@gmail.com>
	<CAJZVxR=KY3AnvFYraT598AY=S+aTkWYYgM=cNhPO3M1pJ8aSSg@mail.gmail.com>
	<CAEuvLSaZwSEOaCGGi_U4UtUK8tTSBET8rXQwExL96YbZXBgMug@mail.gmail.com>
Message-ID: <CAJZVxR=_5Yktur2q7gJTYpSkCQAcZT50VaBSUOL+10qj-xy2kQ@mail.gmail.com>

Thanks Eric,

I tried that and it's not working for me ... I even commented out
every protocol analyser. memory usage still goes up, just slower in
time

Perhaps the amount of traffic is too much for bro ids. I'm gonna send
another email




On Sun, Oct 25, 2015 at 12:36 AM, Eric Ooi <ericooi at gmail.com> wrote:
> Hey Aaron,
>
> I did it by commenting out "@load intel" in
> /opt/bro/share/bro/site/local.bro.  Give that a try and see if it works for
> you.
>
> Eric
>
> On Sat, Oct 24, 2015 at 10:51 AM, Aaron Lewis <the.warl0ck.1989 at gmail.com>
> wrote:
>>
>> Hey Eric,
>>
>> How exactly did you turn off intel log? I tried to comment out this
>> line, but nothing changed on my sensor,
>>
>> @load policy/frameworks/intel/seen
>>
>> Am I wrong? I was following this link:
>> https://www.bro.org/sphinx/frameworks/intel.html
>>
>>
>>
>> On Sat, Oct 24, 2015 at 10:39 PM, Eric Ooi <ericooi at gmail.com> wrote:
>> > I have two Security Onion sensors running 2.4, one monitors a
>> > combination of 100M general office internet traffic + 10G network, the other
>> > monitors four 1G networks which includes a publicly available website and
>> > lots of syslog and SMTP traffic.  I ran the default Security Onion
>> > configuration on both and noticed the sensor monitoring four 1G networks
>> > would run out of memory over the course of a few hours.  I spent a day
>> > turning off various analyzers until I isolated it to the intel analyzer.
>> > Ever since I turned intel.log off for that sensor, it?s run great for weeks.
>> > I tried adding intel feeds (via CriticalStack) and using a blank intel file,
>> > with no luck.  Simply having the intel analyzer on always resulted in memory
>> > loss over time.  I?m guessing it has something to do with the type of
>> > traffic that particular sensor sees (more HTTP, syslog, and SMTP), but I?m
>> > not entirely sure.
>> >
>> > Don?t know if it?s related, but just thought I?d share my experience
>> > with Bro memory issues.
>> >
>> > Eric
>> >
>> >> On Oct 23, 2015, at 11:38 AM, Mike Waite <mfw113 at psu.edu> wrote:
>> >>
>> >> After 30 min things look better, I will let you know how the rest of it
>> >> makes out after a bit.
>> >>
>> >> Oct 23 12:09:52       manager child   -       -       -       info
>> >> selects=100000 canwrites=97046 pending=0
>> >> Oct 23 12:11:54       manager child   -       -       -       info
>> >> selects=200000 canwrites=97046 pending=0
>> >> Oct 23 12:14:04       manager child   -       -       -       info
>> >> selects=300000 canwrites=97046 pending=0
>> >> Oct 23 12:14:43       manager child   -       -       -       info
>> >> selects=400000 canwrites=97046 pending=0
>> >> Oct 23 12:15:20       manager child   -       -       -       info
>> >> selects=500000 canwrites=97046 pending=0
>> >> Oct 23 12:15:54       manager child   -       -       -       info
>> >> selects=600000 canwrites=97046 pending=0
>> >> Oct 23 12:16:38       manager child   -       -       -       info
>> >> selects=700000 canwrites=97046 pending=0
>> >> Oct 23 12:17:41       manager child   -       -       -       info
>> >> selects=800000 canwrites=97046 pending=0
>> >> Oct 23 12:19:03       manager child   -       -       -       info
>> >> selects=900000 canwrites=97046 pending=0
>> >> Oct 23 12:20:46       manager child   -       -       -       info
>> >> selects=1000000 canwrites=97046 pending=0
>> >> Oct 23 12:23:04       manager child   -       -       -       info
>> >> selects=1100000 canwrites=97046 pending=0
>> >> Oct 23 12:25:10       manager child   -       -       -       info
>> >> selects=1200000 canwrites=104987 pending=0
>> >> Oct 23 12:26:40       manager child   -       -       -       info
>> >> selects=1300000 canwrites=104987 pending=0
>> >> Oct 23 12:28:13       manager child   -       -       -       info
>> >> selects=1400000 canwrites=104987 pending=0
>> >> Oct 23 12:31:12       manager child   -       -       -       info
>> >> selects=1600000 canwrites=110134 pending=0
>> >> Oct 23 12:32:24       manager child   -       -       -       info
>> >> selects=1700000 canwrites=110134 pending=0
>> >> Oct 23 12:34:03       manager child   -       -       -       info
>> >> selects=1800000 canwrites=110134 pending=0
>> >> Oct 23 12:35:12       manager child   -       -       -       info
>> >> selects=1900000 canwrites=110134 pending=0
>> >> Oct 23 12:36:15       manager child   -       -       -       info
>> >> selects=2000000 canwrites=110134 pending=0
>> >> Oct 23 12:37:31       manager child   -       -       -       info
>> >> selects=2100000 canwrites=110134 pending=0
>> >>
>> >>
>> >> --
>> >> Mike Waite
>> >> CyberSecurity Intrusion Analyst
>> >> Office of Information Security
>> >> The Pennsylvania State University
>> >> ? 15-10-23 11:09:58, Seth Hall <seth at icir.org>:
>> >>> Mike, could you back out that patch and try my branch,
>> >>> topic/seth/remove-flare ?
>> >>>
>> >>> .Seth
>> >>>
>> >>>
>> >>>> On Oct 23, 2015, at 10:19 AM, Azoff, Justin S <jazoff at illinois.edu>
>> >>>> wrote:
>> >>>>
>> >>>> Well that doesn't look great, but could be a lot worse.  Hard to say
>> >>>> without knowing what it looked like before the patch.
>> >>>>
>> >>>> The fact that pending ever goes down at all is a good sign, but
>> >>>> pending=0 is really the optimal state.
>> >>>>
>> >>>> --
>> >>>> - Justin Azoff
>> >>>>
>> >>>>> On Oct 23, 2015, at 9:21 AM, Mike Waite <mfw113 at psu.edu> wrote:
>> >>>>>
>> >>>>> Patch applied, after 15 minutes I am seeing
>> >>>>>
>> >>>>> Oct 23 09:00:43    manager child   -       -       -       info
>> >>>>> selects=300000 canwrites=216206 pending=0
>> >>>>> Oct 23 09:01:29    manager child   -       -       -       info
>> >>>>> selects=400000 canwrites=216206 pending=0
>> >>>>> Oct 23 09:02:08    manager child   -       -       -       info
>> >>>>> selects=500000 canwrites=216552 pending=0
>> >>>>> Oct 23 09:02:49    manager child   -       -       -       info
>> >>>>> selects=600000 canwrites=216557 pending=0
>> >>>>> Oct 23 09:03:34    manager child   -       -       -       info
>> >>>>> selects=700000 canwrites=216557 pending=0
>> >>>>> Oct 23 09:04:29    manager child   -       -       -       info
>> >>>>> selects=800000 canwrites=255305 pending=4007
>> >>>>> Oct 23 09:05:21    manager child   -       -       -       info
>> >>>>> selects=900000 canwrites=355305 pending=6593
>> >>>>> Oct 23 09:06:13    manager child   -       -       -       info
>> >>>>> selects=1000000 canwrites=455305 pending=6003
>> >>>>> Oct 23 09:07:04    manager child   -       -       -       info
>> >>>>> selects=1100000 canwrites=555305 pending=3077
>> >>>>> Oct 23 09:07:55    manager child   -       -       -       info
>> >>>>> selects=1200000 canwrites=640438 pending=3399
>> >>>>> Oct 23 09:08:45    manager child   -       -       -       info
>> >>>>> selects=1300000 canwrites=740438 pending=3163
>> >>>>> Oct 23 09:09:36    manager child   -       -       -       info
>> >>>>> selects=1400000 canwrites=840438 pending=5245
>> >>>>> Oct 23 09:10:25    manager child   -       -       -       info
>> >>>>> selects=1500000 canwrites=940438 pending=6027
>> >>>>> Oct 23 09:11:15    manager child   -       -       -       info
>> >>>>> selects=1600000 canwrites=1040438 pending=6713
>> >>>>> Oct 23 09:12:01    manager child   -       -       -       info
>> >>>>> selects=1700000 canwrites=1140438 pending=5713
>> >>>>> Oct 23 09:12:50    manager child   -       -       -       info
>> >>>>> selects=1800000 canwrites=1240438 pending=6747
>> >>>>> Oct 23 09:13:39    manager child   -       -       -       info
>> >>>>> selects=1900000 canwrites=1340438 pending=7417
>> >>>>> Oct 23 09:14:32    manager child   -       -       -       info
>> >>>>> selects=2000000 canwrites=1440438 pending=13117
>> >>>>> Oct 23 09:15:10    manager child   -       -       -       info
>> >>>>> selects=2100000 canwrites=1540438 pending=20825
>> >>>>> Oct 23 09:15:59    manager child   -       -       -       info
>> >>>>> selects=2200000 canwrites=1640438 pending=18539
>> >>>>> Oct 23 09:16:47    manager child   -       -       -       info
>> >>>>> selects=2300000 canwrites=1740438 pending=15881
>> >>>>> Oct 23 09:17:35    manager child   -       -       -       info
>> >>>>> selects=2400000 canwrites=1840438 pending=15389
>> >>>>> Oct 23 09:18:28    manager child   -       -       -       info
>> >>>>> selects=2500000 canwrites=1940438 pending=16685
>> >>>>> Oct 23 09:19:18    manager child   -       -       -       info
>> >>>>> selects=2600000 canwrites=2040438 pending=17031
>> >>>>>
>> >>>>>
>> >>>>> I will let you know about the mem usage after a bit
>> >>>>>
>> >>>>> --
>> >>>>> Mike Waite
>> >>>>> CyberSecurity Intrusion Analyst
>> >>>>> Office of Information Security
>> >>>>> The Pennsylvania State University
>> >>>>> ? 15-10-22 10:22:18, Azoff, Justin S <jazoff at illinois.edu>:
>> >>>>>>> On Oct 22, 2015, at 8:12 AM, Mike Waite <mfw113 at psu.edu> wrote:
>> >>>>>>>
>> >>>>>>> I know we are still seeing issues with the manager child proccess.
>> >>>>>>> The process will consume over 200GB of RAM in 8 hours.
>> >>>>>>>
>> >>>>>>
>> >>>>>> Give the attached patch a try.
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> Monitor by using
>> >>>>>>
>> >>>>>> cat logs/current/communication.log |egrep 'manager.child'
>> >>>>>>
>> >>>>>> And check to see if pending=0 or at least not growing.
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> - Justin Azoff
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> Bro mailing list
>> >>>> bro at bro-ids.org
>> >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> >>>>
>> >>>>
>> >>>
>> >>> --
>> >>> Seth Hall
>> >>> International Computer Science Institute
>> >>> (Bro) because everyone has a network
>> >>> http://www.bro.org/
>> >>>
>> >> _______________________________________________
>> >> Bro mailing list
>> >> bro at bro-ids.org
>> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> >
>> >
>> > _______________________________________________
>> > Bro mailing list
>> > bro at bro-ids.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>>
>> --
>> Best Regards,
>> Aaron Lewis - PGP: 0x13714D33 - http://pgp.mit.edu/
>> Finger Print:   9F67 391B B770 8FF6 99DC  D92D 87F6 2602 1371 4D33
>
>



-- 
Best Regards,
Aaron Lewis - PGP: 0x13714D33 - http://pgp.mit.edu/
Finger Print:   9F67 391B B770 8FF6 99DC  D92D 87F6 2602 1371 4D33


From the.warl0ck.1989 at gmail.com  Sun Oct 25 22:36:30 2015
From: the.warl0ck.1989 at gmail.com (Aaron Lewis)
Date: Mon, 26 Oct 2015 13:36:30 +0800
Subject: [Bro] Suggestions on handling 1Gb/s HTTP traffic?
Message-ID: <CAJZVxRkRHPYXBu5QQBJ9Eatijnt42Yc9vFa9B6RR2yCzdQP34A@mail.gmail.com>

Hi,

I recently tested bro 2.4.1 with ~1Gb/s HTTP traffic, it works but the
processes die out of OOM within a few hours.

(The box has 16 cores and 64 GB memory, it should be enough right?)

Now I'm trying to resolve this matter, perhaps one of the following,

1. Limit the volume of traffic that bro will process
2. Tune bro

Can someone please help?

And .. what's the maximum amount of traffic you guys ever tested?

-- 
Best Regards,
Aaron Lewis - PGP: 0x13714D33 - http://pgp.mit.edu/
Finger Print:   9F67 391B B770 8FF6 99DC  D92D 87F6 2602 1371 4D33

From init.conf at gmail.com  Sun Oct 25 23:20:30 2015
From: init.conf at gmail.com (Aashish Sharma)
Date: Sun, 25 Oct 2015 23:20:30 -0700
Subject: [Bro] Suggestions on handling 1Gb/s HTTP traffic?
In-Reply-To: <CAJZVxRkRHPYXBu5QQBJ9Eatijnt42Yc9vFa9B6RR2yCzdQP34A@mail.gmail.com>
References: <CAJZVxRkRHPYXBu5QQBJ9Eatijnt42Yc9vFa9B6RR2yCzdQP34A@mail.gmail.com>
Message-ID: <24753B25-CE38-49CD-B50C-6A0260B90A9B@gmail.com>

Aaron, 

What OS are you  running Bro on ? 

Aashish 

> On Oct 25, 2015, at 10:36 PM, Aaron Lewis <the.warl0ck.1989 at gmail.com> wrote:
> 
> Hi,
> 
> I recently tested bro 2.4.1 with ~1Gb/s HTTP traffic, it works but the
> processes die out of OOM within a few hours.
> 
> (The box has 16 cores and 64 GB memory, it should be enough right?)
> 
> Now I'm trying to resolve this matter, perhaps one of the following,
> 
> 1. Limit the volume of traffic that bro will process
> 2. Tune bro
> 
> Can someone please help?
> 
> And .. what's the maximum amount of traffic you guys ever tested?
> 
> -- 
> Best Regards,
> Aaron Lewis - PGP: 0x13714D33 - http://pgp.mit.edu/
> Finger Print:   9F67 391B B770 8FF6 99DC  D92D 87F6 2602 1371 4D33
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From the.warl0ck.1989 at gmail.com  Sun Oct 25 23:25:34 2015
From: the.warl0ck.1989 at gmail.com (Aaron Lewis)
Date: Mon, 26 Oct 2015 14:25:34 +0800
Subject: [Bro] Suggestions on handling 1Gb/s HTTP traffic?
In-Reply-To: <24753B25-CE38-49CD-B50C-6A0260B90A9B@gmail.com>
References: <CAJZVxRkRHPYXBu5QQBJ9Eatijnt42Yc9vFa9B6RR2yCzdQP34A@mail.gmail.com>
	<24753B25-CE38-49CD-B50C-6A0260B90A9B@gmail.com>
Message-ID: <CAJZVxR=Vj-RavdaQiETk_G6NRxAh6k4znkhuMoe2_dQB8-VzkQ@mail.gmail.com>

Linux, CentOS 6.3

On Mon, Oct 26, 2015 at 2:20 PM, Aashish Sharma <init.conf at gmail.com> wrote:
> Aaron,
>
> What OS are you  running Bro on ?
>
> Aashish
>
>> On Oct 25, 2015, at 10:36 PM, Aaron Lewis <the.warl0ck.1989 at gmail.com> wrote:
>>
>> Hi,
>>
>> I recently tested bro 2.4.1 with ~1Gb/s HTTP traffic, it works but the
>> processes die out of OOM within a few hours.
>>
>> (The box has 16 cores and 64 GB memory, it should be enough right?)
>>
>> Now I'm trying to resolve this matter, perhaps one of the following,
>>
>> 1. Limit the volume of traffic that bro will process
>> 2. Tune bro
>>
>> Can someone please help?
>>
>> And .. what's the maximum amount of traffic you guys ever tested?
>>
>> --
>> Best Regards,
>> Aaron Lewis - PGP: 0x13714D33 - http://pgp.mit.edu/
>> Finger Print:   9F67 391B B770 8FF6 99DC  D92D 87F6 2602 1371 4D33
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 
Best Regards,
Aaron Lewis - PGP: 0x13714D33 - http://pgp.mit.edu/
Finger Print:   9F67 391B B770 8FF6 99DC  D92D 87F6 2602 1371 4D33

From hhoffman at ip-solutions.net  Mon Oct 26 04:21:13 2015
From: hhoffman at ip-solutions.net (Harry Hoffman)
Date: Mon, 26 Oct 2015 07:21:13 -0400
Subject: [Bro] Suggestions on handling 1Gb/s HTTP traffic?
In-Reply-To: <CAJZVxR=Vj-RavdaQiETk_G6NRxAh6k4znkhuMoe2_dQB8-VzkQ@mail.gmail.com>
Message-ID: <c74ab1cf-5198-486b-9dc3-4d6f0295e9ca@email.android.com>

Hi Aaron,

I run a similarly sized box (although with myricom network cards) and RedHat 6.5 that is inspecting about 3x as much total traffic.

Can you share more of your configuration? What network cards? What does /etc/sysctl.conf look like? Are selinux or auditd  running? What does your bro configuration look like?

Cheers,
Harry

On Oct 26, 2015 2:25 AM, Aaron Lewis <the.warl0ck.1989 at gmail.com> wrote:
>
> Linux, CentOS 6.3 
>
> On Mon, Oct 26, 2015 at 2:20 PM, Aashish Sharma <init.conf at gmail.com> wrote: 
> > Aaron, 
> > 
> > What OS are you? running Bro on ? 
> > 
> > Aashish 
> > 
> >> On Oct 25, 2015, at 10:36 PM, Aaron Lewis <the.warl0ck.1989 at gmail.com> wrote: 
> >> 
> >> Hi, 
> >> 
> >> I recently tested bro 2.4.1 with ~1Gb/s HTTP traffic, it works but the 
> >> processes die out of OOM within a few hours. 
> >> 
> >> (The box has 16 cores and 64 GB memory, it should be enough right?) 
> >> 
> >> Now I'm trying to resolve this matter, perhaps one of the following, 
> >> 
> >> 1. Limit the volume of traffic that bro will process 
> >> 2. Tune bro 
> >> 
> >> Can someone please help? 
> >> 
> >> And .. what's the maximum amount of traffic you guys ever tested? 
> >> 
> >> -- 
> >> Best Regards, 
> >> Aaron Lewis - PGP: 0x13714D33 - http://pgp.mit.edu/ 
> >> Finger Print:?? 9F67 391B B770 8FF6 99DC? D92D 87F6 2602 1371 4D33 
> >> _______________________________________________ 
> >> Bro mailing list 
> >> bro at bro-ids.org 
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro 
> > 
>
>
>
> -- 
> Best Regards, 
> Aaron Lewis - PGP: 0x13714D33 - http://pgp.mit.edu/ 
> Finger Print:?? 9F67 391B B770 8FF6 99DC? D92D 87F6 2602 1371 4D33 
> _______________________________________________ 
> Bro mailing list 
> bro at bro-ids.org 
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro 


From jazoff at illinois.edu  Mon Oct 26 05:29:50 2015
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Mon, 26 Oct 2015 12:29:50 +0000
Subject: [Bro] Suggestions on handling 1Gb/s HTTP traffic?
In-Reply-To: <CAJZVxRkRHPYXBu5QQBJ9Eatijnt42Yc9vFa9B6RR2yCzdQP34A@mail.gmail.com>
References: <CAJZVxRkRHPYXBu5QQBJ9Eatijnt42Yc9vFa9B6RR2yCzdQP34A@mail.gmail.com>
Message-ID: <24F744B1-F8E9-4A1E-9D75-1F4024DFE8A8@illinois.edu>


> On Oct 26, 2015, at 1:36 AM, Aaron Lewis <the.warl0ck.1989 at gmail.com> wrote:
> 
> Hi,
> 
> I recently tested bro 2.4.1 with ~1Gb/s HTTP traffic, it works but the
> processes die out of OOM within a few hours.

You need to elaborate on which processes are using memory and getting killed.

Posting this again:

Memory leaks are tricky.  It is important to make a distinction about what component is using a lot of memory:

1) the workers - analyzer issues and leaks in general would show up here.
2) the proxies - communication related
3) the manager - child - if the manager is overloaded the child will buffer log data
4) the manager - parent - if a logging destination is overloaded the parent will buffer log writes


-- 
- Justin Azoff



From longjohngolf at gmail.com  Mon Oct 26 08:49:50 2015
From: longjohngolf at gmail.com (John Daly)
Date: Mon, 26 Oct 2015 08:49:50 -0700
Subject: [Bro] Suggestions on handling 1Gb/s HTTP traffic?
In-Reply-To: <24F744B1-F8E9-4A1E-9D75-1F4024DFE8A8@illinois.edu>
References: <CAJZVxRkRHPYXBu5QQBJ9Eatijnt42Yc9vFa9B6RR2yCzdQP34A@mail.gmail.com>
	<24F744B1-F8E9-4A1E-9D75-1F4024DFE8A8@illinois.edu>
Message-ID: <CALRZ963OcdJNNa76A1q4MhbwNSHyOPv3Nj1VnTXrXsd_VO+qhg@mail.gmail.com>

Aaron,

Have you confirmed that you're getting all of the traffic you expect?
Is the traffic simulated or real HTTP? How are you doing on-box load
balancing? PF_RING vanilla?

On Mon, Oct 26, 2015 at 5:29 AM, Azoff, Justin S <jazoff at illinois.edu> wrote:
>
>> On Oct 26, 2015, at 1:36 AM, Aaron Lewis <the.warl0ck.1989 at gmail.com> wrote:
>>
>> Hi,
>>
>> I recently tested bro 2.4.1 with ~1Gb/s HTTP traffic, it works but the
>> processes die out of OOM within a few hours.
>
> You need to elaborate on which processes are using memory and getting killed.
>
> Posting this again:
>
> Memory leaks are tricky.  It is important to make a distinction about what component is using a lot of memory:
>
> 1) the workers - analyzer issues and leaks in general would show up here.
> 2) the proxies - communication related
> 3) the manager - child - if the manager is overloaded the child will buffer log data
> 4) the manager - parent - if a logging destination is overloaded the parent will buffer log writes
>
>
> --
> - Justin Azoff
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

From af7 at umbc.edu  Mon Oct 26 11:50:03 2015
From: af7 at umbc.edu (Arash Fallah)
Date: Mon, 26 Oct 2015 14:50:03 -0400
Subject: [Bro] Bro -> Elastic Search -> Kibana Issues | Default Analyzed
	Fields
Message-ID: <CAKDt6fSavmXLe_4pZRCQ6+h5r3XhzMp9QPRCh7ZKTA0c3AMVfA@mail.gmail.com>

I am having an issue when trying to process Bro data through Elastic Search
and Kibana.

Specifically, I am doing basic quantitative statistics such as pulling the
Top 5 Originating IP's addresses by id_orig_h. However, Elastic Search and
Kibana breaks apart this field as shown below in the screenshot linked
below:

http://i.imgur.com/m3BH6LP.png

Basically, for some strings, the default Elastic Search analyzer will
segment them into different pieces based on the "." character. For example,
130.85.12.20 will be broken apart into 130, 85, 12, and 80. This is because
in the Core Type mappings, the not index attribute is not explicitly set to
no (it defaults to yes). There is no way to adjust this for existing fields.

Here is the current mapping created by Bro:

"id.resp_h" : {
       "type" : "string"
}

It should be:

"id.resp_h" : {
       "type" : "string"
        "index": "not_analyzed"
}

Suggestions?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151026/00ffb7e2/attachment.html 

From af7 at umbc.edu  Mon Oct 26 12:16:56 2015
From: af7 at umbc.edu (Arash Fallah)
Date: Mon, 26 Oct 2015 15:16:56 -0400
Subject: [Bro] Bro -> Elastic Search -> Kibana Issues | Default Analyzed
	Fields
In-Reply-To: <CAKDt6fSavmXLe_4pZRCQ6+h5r3XhzMp9QPRCh7ZKTA0c3AMVfA@mail.gmail.com>
References: <CAKDt6fSavmXLe_4pZRCQ6+h5r3XhzMp9QPRCh7ZKTA0c3AMVfA@mail.gmail.com>
Message-ID: <CAKDt6fRGWpjHSyhi1cGtb0Rg1Y2oTzArZXj4D8AFqqC2a-zx2g@mail.gmail.com>

Real Quick Update:

My initial analysis was incorrect. IPv4 fields are processed correctly. The
issue is with IPv6 fields. Same concept, different trigger. Here is a
picture illustrating the problem:

http://i.imgur.com/pdxRbmX.png

On Mon, Oct 26, 2015 at 2:50 PM, Arash Fallah <af7 at umbc.edu> wrote:

> I am having an issue when trying to process Bro data through Elastic
> Search and Kibana.
>
> Specifically, I am doing basic quantitative statistics such as pulling the
> Top 5 Originating IP's addresses by id_orig_h. However, Elastic Search and
> Kibana breaks apart this field as shown below in the screenshot linked
> below:
>
> http://i.imgur.com/m3BH6LP.png
>
> Basically, for some strings, the default Elastic Search analyzer will
> segment them into different pieces based on the "." character. For example,
> 130.85.12.20 will be broken apart into 130, 85, 12, and 80. This is because
> in the Core Type mappings, the not index attribute is not explicitly set to
> no (it defaults to yes). There is no way to adjust this for existing fields.
>
> Here is the current mapping created by Bro:
>
> "id.resp_h" : {
>        "type" : "string"
> }
>
> It should be:
>
> "id.resp_h" : {
>        "type" : "string"
>         "index": "not_analyzed"
> }
>
> Suggestions?
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151026/8971a5e4/attachment.html 

From mark.parsons at gmail.com  Mon Oct 26 17:00:19 2015
From: mark.parsons at gmail.com (Mark Parsons)
Date: Mon, 26 Oct 2015 20:00:19 -0400
Subject: [Bro] Bro -> Elastic Search -> Kibana Issues | Default Analyzed
	Fields
In-Reply-To: <CAKDt6fRGWpjHSyhi1cGtb0Rg1Y2oTzArZXj4D8AFqqC2a-zx2g@mail.gmail.com>
References: <CAKDt6fSavmXLe_4pZRCQ6+h5r3XhzMp9QPRCh7ZKTA0c3AMVfA@mail.gmail.com>
	<CAKDt6fRGWpjHSyhi1cGtb0Rg1Y2oTzArZXj4D8AFqqC2a-zx2g@mail.gmail.com>
Message-ID: <9065249B-8ED3-477F-97C8-12A88E7D03A9@gmail.com>

Arash,
I imagine that is because elasticsearch currently does not support IPv6 fields. 

See this elasticsearch github issue for more background

https://github.com/elastic/elasticsearch/issues/3714

Thanks,
Mark

Sent from my iPhone

> On Oct 26, 2015, at 3:16 PM, Arash Fallah <af7 at umbc.edu> wrote:
> 
> Real Quick Update:
> 
> My initial analysis was incorrect. IPv4 fields are processed correctly. The issue is with IPv6 fields. Same concept, different trigger. Here is a picture illustrating the problem:
> 
> http://i.imgur.com/pdxRbmX.png
> 
>> On Mon, Oct 26, 2015 at 2:50 PM, Arash Fallah <af7 at umbc.edu> wrote:
>> I am having an issue when trying to process Bro data through Elastic Search and Kibana.
>> 
>> Specifically, I am doing basic quantitative statistics such as pulling the Top 5 Originating IP's addresses by id_orig_h. However, Elastic Search and Kibana breaks apart this field as shown below in the screenshot linked below:
>> 
>> http://i.imgur.com/m3BH6LP.png
>> 
>> Basically, for some strings, the default Elastic Search analyzer will segment them into different pieces based on the "." character. For example, 130.85.12.20 will be broken apart into 130, 85, 12, and 80. This is because in the Core Type mappings, the not index attribute is not explicitly set to no (it defaults to yes). There is no way to adjust this for existing fields.
>> 
>> Here is the current mapping created by Bro:
>> 
>> "id.resp_h" : {
>>        "type" : "string"
>> }
>> 
>> It should be:
>> 
>> "id.resp_h" : {
>>        "type" : "string"
>>         "index": "not_analyzed"
>> }
>> 
>> Suggestions?
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151026/b6b03ed8/attachment.html 

From af7 at umbc.edu  Mon Oct 26 17:47:00 2015
From: af7 at umbc.edu (Arash Fallah)
Date: Mon, 26 Oct 2015 20:47:00 -0400
Subject: [Bro] Bro -> Elastic Search -> Kibana Issues | Default Analyzed
	Fields
In-Reply-To: <9065249B-8ED3-477F-97C8-12A88E7D03A9@gmail.com>
References: <CAKDt6fSavmXLe_4pZRCQ6+h5r3XhzMp9QPRCh7ZKTA0c3AMVfA@mail.gmail.com>
	<CAKDt6fRGWpjHSyhi1cGtb0Rg1Y2oTzArZXj4D8AFqqC2a-zx2g@mail.gmail.com>
	<9065249B-8ED3-477F-97C8-12A88E7D03A9@gmail.com>
Message-ID: <707CB89B-FCD1-42E3-9AA1-ECE282D4704E@umbc.edu>

Exactly, which is why I suggest disabling string analyzation in Elastic Search for this field (and all IP fields). Otherwise, the IPv6 entries are ruining the results.

> On Oct 26, 2015, at 8:00 PM, Mark Parsons <mark.parsons at gmail.com> wrote:
> 
> Arash,
> I imagine that is because elasticsearch currently does not support IPv6 fields. 
> 
> See this elasticsearch github issue for more background
> 
> https://github.com/elastic/elasticsearch/issues/3714 <https://github.com/elastic/elasticsearch/issues/3714>
> 
> Thanks,
> Mark
> 
> Sent from my iPhone
> 
> On Oct 26, 2015, at 3:16 PM, Arash Fallah <af7 at umbc.edu <mailto:af7 at umbc.edu>> wrote:
> 
>> Real Quick Update:
>> 
>> My initial analysis was incorrect. IPv4 fields are processed correctly. The issue is with IPv6 fields. Same concept, different trigger. Here is a picture illustrating the problem:
>> 
>> http://i.imgur.com/pdxRbmX.png <http://i.imgur.com/pdxRbmX.png>
>> 
>> On Mon, Oct 26, 2015 at 2:50 PM, Arash Fallah <af7 at umbc.edu <mailto:af7 at umbc.edu>> wrote:
>> I am having an issue when trying to process Bro data through Elastic Search and Kibana.
>> 
>> Specifically, I am doing basic quantitative statistics such as pulling the Top 5 Originating IP's addresses by id_orig_h. However, Elastic Search and Kibana breaks apart this field as shown below in the screenshot linked below:
>> 
>> http://i.imgur.com/m3BH6LP.png <http://i.imgur.com/m3BH6LP.png>
>> 
>> Basically, for some strings, the default Elastic Search analyzer will segment them into different pieces based on the "." character. For example, 130.85.12.20 will be broken apart into 130, 85, 12, and 80. This is because in the Core Type mappings, the not index attribute is not explicitly set to no (it defaults to yes). There is no way to adjust this for existing fields.
>> 
>> Here is the current mapping created by Bro:
>> 
>> "id.resp_h" : {
>>        "type" : "string"
>> }
>> 
>> It should be:
>> 
>> "id.resp_h" : {
>>        "type" : "string"
>>         "index": "not_analyzed"
>> }
>> 
>> Suggestions?
>> 
>> 
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org <mailto:bro at bro-ids.org>
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151026/2c41aa57/attachment.html 

From daniel.guerra69 at gmail.com  Wed Oct 28 02:35:12 2015
From: daniel.guerra69 at gmail.com (Daniel Guerra)
Date: Wed, 28 Oct 2015 10:35:12 +0100
Subject: [Bro]  Bro -> Elasticsearch -> Kibana4beta -> GeoLocation
Message-ID: <67EEF60C-7D84-432B-9787-7F8F27521214@gmail.com>

Hi All,

I want to use locations in kibana. The problem is control
over the json output. I don?t know if bro can produce complex
json structures.
Elasticsearch needs a structure like this
coordinate: {lat:53.00,long:4.00} 
but bro uses an output like
coordinate.lat:53.00,coordinate.long:4.00
Does anyone know how to get control over
the output without rewriting the ascii/json writer

Regards,

Daniel Guerra

From daniel.guerra69 at gmail.com  Wed Oct 28 06:31:05 2015
From: daniel.guerra69 at gmail.com (Daniel Guerra)
Date: Wed, 28 Oct 2015 14:31:05 +0100
Subject: [Bro]  Bro -> Elasticsearch -> Kibana4beta -> GeoLocation
Message-ID: <C8E556C1-9066-4AED-8BA5-8CD77B236498@gmail.com>

To be more clear
I use this bro script for geo location

##! Add geo_location for the originator and responder of a connection
##! to the connection logs.

module Conn;

export
 {
	redef record Conn::Info += 
	{
		orig_loc: geo_location &optional &log;
    		resp_loc: geo_location &optional &log;
	};
}

event connection_state_remove(c: connection) 
{
	local orig_loc = lookup_location(c$id$orig_h);
  	if (orig_loc?$longitude && orig_loc?$latitude)
    		c$conn$orig_loc= orig_loc;
	local resp_loc = lookup_location(c$id$resp_h);
  	if (resp_loc?$longitude && resp_loc?$latitude)
    		c$conn$resp_loc= resp_loc;
}

Produces this output in json example
{
    "ts": "2013-04-26T12:12:02.341149Z",
    "uid": "C0GaiXWHKY4Uj0qke",
    "id.orig_h": "83.161.249.149",
    "id.orig_p": 49318,
    "id.resp_h": "68.232.35.139",
    "id.resp_p": 443,
    "proto": "tcp",
    "conn_state": "SHR",
    "missed_bytes": 0,
    "history": "f",
    "orig_pkts": 0,
    "orig_ip_bytes": 0,
    "resp_pkts": 1,
    "resp_ip_bytes": 67,
    "tunnel_parents": [],
    "orig_loc.country_code": "NL",
    "orig_loc.latitude": 52.366699,
    "orig_loc.longitude": 4.9,
    "resp_loc.country_code": "US",
    "resp_loc.region": "CA",
    "resp_loc.city": "Santa Monica",
    "resp_loc.latitude": 34.011902,
    "resp_loc.longitude": -118.468201
  }

According to the elasticsearch documentation i need an output like

?resp_loc?: {
	?lat?: 52.366699,
	?long?:4.9
}

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-geo-point-type.html <https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-geo-point-type.html>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151028/29aed11b/attachment.html 

From af7 at umbc.edu  Wed Oct 28 09:45:55 2015
From: af7 at umbc.edu (Arash Fallah)
Date: Wed, 28 Oct 2015 12:45:55 -0400
Subject: [Bro] Bro -> Elastic Search -> Kibana Issues | Default Analyzed
	Fields
In-Reply-To: <707CB89B-FCD1-42E3-9AA1-ECE282D4704E@umbc.edu>
References: <CAKDt6fSavmXLe_4pZRCQ6+h5r3XhzMp9QPRCh7ZKTA0c3AMVfA@mail.gmail.com>
	<CAKDt6fRGWpjHSyhi1cGtb0Rg1Y2oTzArZXj4D8AFqqC2a-zx2g@mail.gmail.com>
	<9065249B-8ED3-477F-97C8-12A88E7D03A9@gmail.com>
	<707CB89B-FCD1-42E3-9AA1-ECE282D4704E@umbc.edu>
Message-ID: <CAKDt6fRrAXv3pTGpvrThkwJx-0D-bNC7=ySk9_3LZSOx9ncXdg@mail.gmail.com>

Does anyone have any suggestions on working around this? I was thinking of
using LogStash but I'm worried about scalability with that solution.

I understand Bro wasn't explicitly made to interface with ElasticSearch and
that this might be a process.

On Mon, Oct 26, 2015 at 8:47 PM, Arash Fallah <af7 at umbc.edu> wrote:

> Exactly, which is why I suggest disabling string analyzation in Elastic
> Search for this field (and all IP fields). Otherwise, the IPv6 entries are
> ruining the results.
>
> On Oct 26, 2015, at 8:00 PM, Mark Parsons <mark.parsons at gmail.com> wrote:
>
> Arash,
> I imagine that is because elasticsearch currently does not support IPv6
> fields.
>
> See this elasticsearch github issue for more background
>
> https://github.com/elastic/elasticsearch/issues/3714
>
> Thanks,
> Mark
>
> Sent from my iPhone
>
> On Oct 26, 2015, at 3:16 PM, Arash Fallah <af7 at umbc.edu> wrote:
>
> Real Quick Update:
>
> My initial analysis was incorrect. IPv4 fields are processed correctly.
> The issue is with IPv6 fields. Same concept, different trigger. Here is a
> picture illustrating the problem:
>
> http://i.imgur.com/pdxRbmX.png
>
> On Mon, Oct 26, 2015 at 2:50 PM, Arash Fallah <af7 at umbc.edu> wrote:
>
>> I am having an issue when trying to process Bro data through Elastic
>> Search and Kibana.
>>
>> Specifically, I am doing basic quantitative statistics such as pulling
>> the Top 5 Originating IP's addresses by id_orig_h. However, Elastic Search
>> and Kibana breaks apart this field as shown below in the screenshot linked
>> below:
>>
>> http://i.imgur.com/m3BH6LP.png
>>
>> Basically, for some strings, the default Elastic Search analyzer will
>> segment them into different pieces based on the "." character. For example,
>> 130.85.12.20 will be broken apart into 130, 85, 12, and 80. This is because
>> in the Core Type mappings, the not index attribute is not explicitly set to
>> no (it defaults to yes). There is no way to adjust this for existing fields.
>>
>> Here is the current mapping created by Bro:
>>
>> "id.resp_h" : {
>>        "type" : "string"
>> }
>>
>> It should be:
>>
>> "id.resp_h" : {
>>        "type" : "string"
>>         "index": "not_analyzed"
>> }
>>
>> Suggestions?
>>
>>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151028/7610bb56/attachment.html 

From masoom.alam at gmail.com  Wed Oct 28 16:04:32 2015
From: masoom.alam at gmail.com (masoom alam)
Date: Wed, 28 Oct 2015 16:04:32 -0700
Subject: [Bro] BRO logs after http attacks
Message-ID: <CAPihp9m02MevgDMB=OADUDEA-B0Be4-XPa2Q_bQGq1NWehqWzQ@mail.gmail.com>

Hi Everyone,

We are trying to monitor the BRO logs after self generated HTTP attacks. In
our lab we are trying to attack a web server through metasploit for HTTP
SQL injection attacks. The goal is to monitor the attacks
parameters/indicators via BRO logs. Are we on the right track. In
particular what is the ALERT/ALARM mechanism for BRO when it detect an
attack....is it indicated in the logs.....or there are some places to look
for it and not just logs. Till now, while surfing the BRO logs, we have not
found any attack information....

Please guide.

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151028/af02de40/attachment.html 

From jazoff at illinois.edu  Wed Oct 28 16:20:44 2015
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Wed, 28 Oct 2015 23:20:44 +0000
Subject: [Bro] BRO logs after http attacks
In-Reply-To: <CAPihp9m02MevgDMB=OADUDEA-B0Be4-XPa2Q_bQGq1NWehqWzQ@mail.gmail.com>
References: <CAPihp9m02MevgDMB=OADUDEA-B0Be4-XPa2Q_bQGq1NWehqWzQ@mail.gmail.com>
Message-ID: <5ECC4C0B-8642-481E-A5B5-888567CC34FB@illinois.edu>


> On Oct 28, 2015, at 7:04 PM, masoom alam <masoom.alam at gmail.com> wrote:
> 
> Hi Everyone, 
> 
> We are trying to monitor the BRO logs after self generated HTTP attacks. In our lab we are trying to attack a web server through metasploit for HTTP SQL injection attacks. The goal is to monitor the attacks parameters/indicators via BRO logs. Are we on the right track. In particular what is the ALERT/ALARM mechanism for BRO when it detect an attack....is it indicated in the logs.....or there are some places to look for it and not just logs. Till now, while surfing the BRO logs, we have not found any attack information....
> 
> Please guide. 
> 
> Thanks

Bro calls ALERT/ALARM things notices.  Logs for those events go to the notice.log, so that should have something about your sql injection attempts.

The protocols/http/detect-sqli handles that sort of thing.  It will raise notices for scans and add entries to the 'tags' column of the http log for matching connections.



-- 
- Justin Azoff



From masoom.alam at gmail.com  Wed Oct 28 18:24:23 2015
From: masoom.alam at gmail.com (masoom alam)
Date: Wed, 28 Oct 2015 18:24:23 -0700
Subject: [Bro] BRO logs after http attacks
In-Reply-To: <5ECC4C0B-8642-481E-A5B5-888567CC34FB@illinois.edu>
References: <CAPihp9m02MevgDMB=OADUDEA-B0Be4-XPa2Q_bQGq1NWehqWzQ@mail.gmail.com>
	<5ECC4C0B-8642-481E-A5B5-888567CC34FB@illinois.edu>
Message-ID: <CAPihp9n0B9zssKmcfsYns6xXQs6bmqKFyXFXjN-2FQj4Bo3foQ@mail.gmail.com>

Thanks for the response.

This means within http logs there is an indication of a possible attack
through tags....which is detailed in notices.log......?
On Oct 29, 2015 4:20 AM, "Azoff, Justin S" <jazoff at illinois.edu> wrote:

>
> > On Oct 28, 2015, at 7:04 PM, masoom alam <masoom.alam at gmail.com> wrote:
> >
> > Hi Everyone,
> >
> > We are trying to monitor the BRO logs after self generated HTTP attacks.
> In our lab we are trying to attack a web server through metasploit for HTTP
> SQL injection attacks. The goal is to monitor the attacks
> parameters/indicators via BRO logs. Are we on the right track. In
> particular what is the ALERT/ALARM mechanism for BRO when it detect an
> attack....is it indicated in the logs.....or there are some places to look
> for it and not just logs. Till now, while surfing the BRO logs, we have not
> found any attack information....
> >
> > Please guide.
> >
> > Thanks
>
> Bro calls ALERT/ALARM things notices.  Logs for those events go to the
> notice.log, so that should have something about your sql injection attempts.
>
> The protocols/http/detect-sqli handles that sort of thing.  It will raise
> notices for scans and add entries to the 'tags' column of the http log for
> matching connections.
>
>
>
> --
> - Justin Azoff
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151028/94bc78dc/attachment.html 

From vlad at grigorescu.org  Thu Oct 29 13:15:10 2015
From: vlad at grigorescu.org (Vlad Grigorescu)
Date: Thu, 29 Oct 2015 15:15:10 -0500
Subject: [Bro] Bro -> Elasticsearch -> Kibana4beta -> GeoLocation
In-Reply-To: <C8E556C1-9066-4AED-8BA5-8CD77B236498@gmail.com>
References: <C8E556C1-9066-4AED-8BA5-8CD77B236498@gmail.com>
Message-ID: <CANUJ_FkNeeRzEqT0niWXh+Vy_2d3JwZU--zeCA79inkEDAqNPg@mail.gmail.com>

You should be able to customize how ElasticSearch stores the data via an
explicit mapping:
https://www.elastic.co/guide/en/elasticsearch/reference/current/object.html
(i.e., you would add lat and long as objects under the resp_loc object).

Alternatively, if all you want is to present a geo-point, there are some
other ways to represent that:
https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html

  --Vlad

On Wed, Oct 28, 2015 at 8:31 AM, Daniel Guerra <daniel.guerra69 at gmail.com>
wrote:

> To be more clear
> I use this bro script for geo location
>
> ##! Add geo_location for the originator and responder of a connection
> ##! to the connection logs.
>
> module Conn;
>
> export
>  {
> redef record Conn::Info +=
> {
> orig_loc: geo_location &optional &log;
>     resp_loc: geo_location &optional &log;
> };
> }
>
> event connection_state_remove(c: connection)
> {
> local orig_loc = lookup_location(c$id$orig_h);
>   if (orig_loc?$longitude && orig_loc?$latitude)
>     c$conn$orig_loc= orig_loc;
> local resp_loc = lookup_location(c$id$resp_h);
>   if (resp_loc?$longitude && resp_loc?$latitude)
>     c$conn$resp_loc= resp_loc;
> }
>
> Produces this output in json example
> {
>     "ts": "2013-04-26T12:12:02.341149Z",
>     "uid": "C0GaiXWHKY4Uj0qke",
>     "id.orig_h": "83.161.249.149",
>     "id.orig_p": 49318,
>     "id.resp_h": "68.232.35.139",
>     "id.resp_p": 443,
>     "proto": "tcp",
>     "conn_state": "SHR",
>     "missed_bytes": 0,
>     "history": "f",
>     "orig_pkts": 0,
>     "orig_ip_bytes": 0,
>     "resp_pkts": 1,
>     "resp_ip_bytes": 67,
>     "tunnel_parents": [],
>     "orig_loc.country_code": "NL",
>     "orig_loc.latitude": 52.366699,
>     "orig_loc.longitude": 4.9,
>     "resp_loc.country_code": "US",
>     "resp_loc.region": "CA",
>     "resp_loc.city": "Santa Monica",
>     "resp_loc.latitude": 34.011902,
>     "resp_loc.longitude": -118.468201
>   }
>
> According to the elasticsearch documentation i need an output like
>
> ?resp_loc?: {
> ?lat?: 52.366699,
> ?long?:4.9
> }
>
>
> https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-geo-point-type.html
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151029/b70ca052/attachment.html 

From daniel.guerra69 at gmail.com  Thu Oct 29 14:28:47 2015
From: daniel.guerra69 at gmail.com (Daniel Guerra)
Date: Thu, 29 Oct 2015 22:28:47 +0100
Subject: [Bro] Bro -> Elasticsearch -> Kibana4beta -> GeoLocation
In-Reply-To: <CANUJ_FkNeeRzEqT0niWXh+Vy_2d3JwZU--zeCA79inkEDAqNPg@mail.gmail.com>
References: <C8E556C1-9066-4AED-8BA5-8CD77B236498@gmail.com>
	<CANUJ_FkNeeRzEqT0niWXh+Vy_2d3JwZU--zeCA79inkEDAqNPg@mail.gmail.com>
Message-ID: <F18E445D-45A5-41A0-AF26-BFD37F31400A@gmail.com>

Hi Vlad,

I?m interested in how I can present the json the way elastic wants it.

Regards,

Daniel
> On 29 Oct 2015, at 21:15, Vlad Grigorescu <vlad at grigorescu.org> wrote:
> 
> You should be able to customize how ElasticSearch stores the data via an explicit mapping: https://www.elastic.co/guide/en/elasticsearch/reference/current/object.html <https://www.elastic.co/guide/en/elasticsearch/reference/current/object.html> (i.e., you would add lat and long as objects under the resp_loc object).
> 
> Alternatively, if all you want is to present a geo-point, there are some other ways to represent that: https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html <https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html>
> 
>   --Vlad
> 
> On Wed, Oct 28, 2015 at 8:31 AM, Daniel Guerra <daniel.guerra69 at gmail.com <mailto:daniel.guerra69 at gmail.com>> wrote:
> To be more clear
> I use this bro script for geo location
> 
> ##! Add geo_location for the originator and responder of a connection
> ##! to the connection logs.
> 
> module Conn;
> 
> export
>  {
> 	redef record Conn::Info += 
> 	{
> 		orig_loc: geo_location &optional &log;
>     		resp_loc: geo_location &optional &log;
> 	};
> }
> 
> event connection_state_remove(c: connection) 
> {
> 	local orig_loc = lookup_location(c$id$orig_h);
>   	if (orig_loc?$longitude && orig_loc?$latitude)
>     		c$conn$orig_loc= orig_loc;
> 	local resp_loc = lookup_location(c$id$resp_h);
>   	if (resp_loc?$longitude && resp_loc?$latitude)
>     		c$conn$resp_loc= resp_loc;
> }
> 
> Produces this output in json example
> {
>     "ts": "2013-04-26T12:12:02.341149Z",
>     "uid": "C0GaiXWHKY4Uj0qke",
>     "id.orig_h": "83.161.249.149",
>     "id.orig_p": 49318,
>     "id.resp_h": "68.232.35.139",
>     "id.resp_p": 443,
>     "proto": "tcp",
>     "conn_state": "SHR",
>     "missed_bytes": 0,
>     "history": "f",
>     "orig_pkts": 0,
>     "orig_ip_bytes": 0,
>     "resp_pkts": 1,
>     "resp_ip_bytes": 67,
>     "tunnel_parents": [],
>     "orig_loc.country_code": "NL",
>     "orig_loc.latitude": 52.366699,
>     "orig_loc.longitude": 4.9,
>     "resp_loc.country_code": "US",
>     "resp_loc.region": "CA",
>     "resp_loc.city": "Santa Monica",
>     "resp_loc.latitude": 34.011902,
>     "resp_loc.longitude": -118.468201
>   }
> 
> According to the elasticsearch documentation i need an output like
> 
> ?resp_loc?: {
> 	?lat?: 52.366699,
> 	?long?:4.9
> }
> 
> https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-geo-point-type.html <https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-geo-point-type.html>
> 
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org <mailto:bro at bro-ids.org>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151029/3c5da3c6/attachment.html 

From andrew.ratcliffe at nswcsystems.co.uk  Thu Oct 29 15:55:37 2015
From: andrew.ratcliffe at nswcsystems.co.uk (Andrew Ratcliffe)
Date: Thu, 29 Oct 2015 22:55:37 +0000
Subject: [Bro] Bro -> Elasticsearch -> Kibana4beta -> GeoLocation
In-Reply-To: <C8E556C1-9066-4AED-8BA5-8CD77B236498@gmail.com>
References: <C8E556C1-9066-4AED-8BA5-8CD77B236498@gmail.com>
Message-ID: <D21C7E43-9F11-4DED-98C9-C1CC6C7F45DB@nswcsystems.co.uk>

I?m not sure if this is what you are looking for but just in case it?s useful.

# BRO_connlog ######################
  if [type] == "BRO_connlog" {
    grok {
      match => [ "message", "(?<ts>(.*?))\t(?<uid>(.*?))\t(?<id.orig_h>(.*?))\t(?<id.orig_p>(.*?))\t(?<id.resp_h>(.*?))\t(?<id.resp_p>(.*?))\t(?<proto>(.*?))\t(?<service>(.*?))\t(?<duration>(.*?))\t(?<orig_bytes>(.*?))\t(?<resp_bytes>(.*?))\t(?<conn_state>(.*?))\t(?<local_orig>(.*?))\t(?<missed_bytes>(.*?))\t(?<history>(.*?))\t(?<orig_pkts>(.*?))\t(?<orig_ip_bytes>(.*?))\t(?<resp_pkts>(.*?))\t(?<resp_ip_bytes>(.*?))\t(?<tunnel_parents>(.*))\t(?<orig_cc>(.*))\t(?<resp_cc>(.*))\t(?<orig_longitude>(.*))\t(?<orig_latitude>(.*))\t(?<resp_longitude>(.*))\t(?<resp_latitude>(.*))" ]
    }
     mutate {
           # 'coords' will be kept, 'tmplat' is temporary.
           # Both of these new fields are strings.
           add_field => [ "resp_coords", "%{resp_longitude}",
                     "tmplat", "%{resp_latitude}" ]
                }
     mutate {
           # Merge 'tmplat' into 'coords'
           merge => [ "resp_coords", "tmplat" ]
                }
     mutate {
           # Convert our new array of strings back to float
           convert => [ "resp_coords", "float" ]
           # Delete our temporary latitude field
           remove => [ "tmplat" ]
                }
  }

I just used the resp but could duplicate for orig.

Kind regards,
Andy
Andrew.Ratcliffe at NSWCSystems.co.uk
CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
Blog.InfoSecMatters.net <http://blog.infosecmatters.net/>






> On 28 Oct 2015, at 13:31, Daniel Guerra <daniel.guerra69 at gmail.com> wrote:
> 
> To be more clear
> I use this bro script for geo location
> 
> ##! Add geo_location for the originator and responder of a connection
> ##! to the connection logs.
> 
> module Conn;
> 
> export
>  {
> 	redef record Conn::Info +=
> 	{
> 		orig_loc: geo_location &optional &log;
>     		resp_loc: geo_location &optional &log;
> 	};
> }
> 
> event connection_state_remove(c: connection)
> {
> 	local orig_loc = lookup_location(c$id$orig_h);
>   	if (orig_loc?$longitude && orig_loc?$latitude)
>     		c$conn$orig_loc= orig_loc;
> 	local resp_loc = lookup_location(c$id$resp_h);
>   	if (resp_loc?$longitude && resp_loc?$latitude)
>     		c$conn$resp_loc= resp_loc;
> }
> 
> Produces this output in json example
> {
>     "ts": "2013-04-26T12:12:02.341149Z",
>     "uid": "C0GaiXWHKY4Uj0qke",
>     "id.orig_h": "83.161.249.149",
>     "id.orig_p": 49318,
>     "id.resp_h": "68.232.35.139",
>     "id.resp_p": 443,
>     "proto": "tcp",
>     "conn_state": "SHR",
>     "missed_bytes": 0,
>     "history": "f",
>     "orig_pkts": 0,
>     "orig_ip_bytes": 0,
>     "resp_pkts": 1,
>     "resp_ip_bytes": 67,
>     "tunnel_parents": [],
>     "orig_loc.country_code": "NL",
>     "orig_loc.latitude": 52.366699,
>     "orig_loc.longitude": 4.9,
>     "resp_loc.country_code": "US",
>     "resp_loc.region": "CA",
>     "resp_loc.city": "Santa Monica",
>     "resp_loc.latitude": 34.011902,
>     "resp_loc.longitude": -118.468201
>   }
> 
> According to the elasticsearch documentation i need an output like
> 
> ?resp_loc?: {
> 	?lat?: 52.366699,
> 	?long?:4.9
> }
> 
> https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-geo-point-type.html <https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-geo-point-type.html>
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151029/86d7073e/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 235 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151029/86d7073e/attachment-0001.bin 

From daniel.guerra69 at gmail.com  Thu Oct 29 18:33:27 2015
From: daniel.guerra69 at gmail.com (Daniel Guerra)
Date: Fri, 30 Oct 2015 02:33:27 +0100
Subject: [Bro] Bro -> Elasticsearch -> Kibana4beta -> GeoLocation
In-Reply-To: <D21C7E43-9F11-4DED-98C9-C1CC6C7F45DB@nswcsystems.co.uk>
References: <C8E556C1-9066-4AED-8BA5-8CD77B236498@gmail.com>
	<D21C7E43-9F11-4DED-98C9-C1CC6C7F45DB@nswcsystems.co.uk>
Message-ID: <1DA686BF-8B1F-4E85-B6CF-826E4F7A3E4D@gmail.com>

I use the elasticsearch plugin in bro. I know logstash works fine but its
very cpu intensive. Thanx anyway. 
> On 29 Oct 2015, at 23:55, Andrew Ratcliffe <andrew.ratcliffe at nswcsystems.co.uk> wrote:
> 
> I?m not sure if this is what you are looking for but just in case it?s useful. 
> 
> # BRO_connlog ######################
>   if [type] == "BRO_connlog" {
>     grok {
>       match => [ "message", "(?<ts>(.*?))\t(?<uid>(.*?))\t(?<id.orig_h>(.*?))\t(?<id.orig_p>(.*?))\t(?<id.resp_h>(.*?))\t(?<id.resp_p>(.*?))\t(?<proto>(.*?))\t(?<service>(.*?))\t(?<duration>(.*?))\t(?<orig_bytes>(.*?))\t(?<resp_bytes>(.*?))\t(?<conn_state>(.*?))\t(?<local_orig>(.*?))\t(?<missed_bytes>(.*?))\t(?<history>(.*?))\t(?<orig_pkts>(.*?))\t(?<orig_ip_bytes>(.*?))\t(?<resp_pkts>(.*?))\t(?<resp_ip_bytes>(.*?))\t(?<tunnel_parents>(.*))\t(?<orig_cc>(.*))\t(?<resp_cc>(.*))\t(?<orig_longitude>(.*))\t(?<orig_latitude>(.*))\t(?<resp_longitude>(.*))\t(?<resp_latitude>(.*))" ]
>     }
>      mutate {
>            # 'coords' will be kept, 'tmplat' is temporary.
>            # Both of these new fields are strings.
>            add_field => [ "resp_coords", "%{resp_longitude}",
>                      "tmplat", "%{resp_latitude}" ]
>                 }
>      mutate {
>            # Merge 'tmplat' into 'coords'
>            merge => [ "resp_coords", "tmplat" ]
>                 }
>      mutate {
>            # Convert our new array of strings back to float
>            convert => [ "resp_coords", "float" ]
>            # Delete our temporary latitude field
>            remove => [ "tmplat" ]
>                 }
>   }
> 
> I just used the resp but could duplicate for orig. 
> 
> Kind regards,
> Andy
> Andrew.Ratcliffe at NSWCSystems.co.uk <mailto:Andrew.Ratcliffe at nswcsystems.co.uk>
> CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
> Blog.InfoSecMatters.net <http://blog.infosecmatters.net/>
> 
> 
> 
> 
> 
> 
>> On 28 Oct 2015, at 13:31, Daniel Guerra <daniel.guerra69 at gmail.com <mailto:daniel.guerra69 at gmail.com>> wrote:
>> 
>> To be more clear
>> I use this bro script for geo location
>> 
>> ##! Add geo_location for the originator and responder of a connection
>> ##! to the connection logs.
>> 
>> module Conn;
>> 
>> export
>>  {
>> 	redef record Conn::Info += 
>> 	{
>> 		orig_loc: geo_location &optional &log;
>>     		resp_loc: geo_location &optional &log;
>> 	};
>> }
>> 
>> event connection_state_remove(c: connection) 
>> {
>> 	local orig_loc = lookup_location(c$id$orig_h);
>>   	if (orig_loc?$longitude && orig_loc?$latitude)
>>     		c$conn$orig_loc= orig_loc;
>> 	local resp_loc = lookup_location(c$id$resp_h);
>>   	if (resp_loc?$longitude && resp_loc?$latitude)
>>     		c$conn$resp_loc= resp_loc;
>> }
>> 
>> Produces this output in json example
>> {
>>     "ts": "2013-04-26T12:12:02.341149Z",
>>     "uid": "C0GaiXWHKY4Uj0qke",
>>     "id.orig_h": "83.161.249.149",
>>     "id.orig_p": 49318,
>>     "id.resp_h": "68.232.35.139",
>>     "id.resp_p": 443,
>>     "proto": "tcp",
>>     "conn_state": "SHR",
>>     "missed_bytes": 0,
>>     "history": "f",
>>     "orig_pkts": 0,
>>     "orig_ip_bytes": 0,
>>     "resp_pkts": 1,
>>     "resp_ip_bytes": 67,
>>     "tunnel_parents": [],
>>     "orig_loc.country_code": "NL",
>>     "orig_loc.latitude": 52.366699,
>>     "orig_loc.longitude": 4.9,
>>     "resp_loc.country_code": "US",
>>     "resp_loc.region": "CA",
>>     "resp_loc.city": "Santa Monica",
>>     "resp_loc.latitude": 34.011902,
>>     "resp_loc.longitude": -118.468201
>>   }
>> 
>> According to the elasticsearch documentation i need an output like
>> 
>> ?resp_loc?: {
>> 	?lat?: 52.366699,
>> 	?long?:4.9
>> }
>> 
>> https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-geo-point-type.html <https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-geo-point-type.html>
>> 
>> 
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org <mailto:bro at bro-ids.org>
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151030/0f93b987/attachment.html 

From seth at icir.org  Fri Oct 30 06:46:32 2015
From: seth at icir.org (Seth Hall)
Date: Fri, 30 Oct 2015 09:46:32 -0400
Subject: [Bro] Bro -> Elasticsearch -> Kibana4beta -> GeoLocation
In-Reply-To: <1DA686BF-8B1F-4E85-B6CF-826E4F7A3E4D@gmail.com>
References: <C8E556C1-9066-4AED-8BA5-8CD77B236498@gmail.com>
	<D21C7E43-9F11-4DED-98C9-C1CC6C7F45DB@nswcsystems.co.uk>
	<1DA686BF-8B1F-4E85-B6CF-826E4F7A3E4D@gmail.com>
Message-ID: <8129DD67-D968-415D-A564-1F4AF96C70C9@icir.org>


> On Oct 29, 2015, at 9:33 PM, Daniel Guerra <daniel.guerra69 at gmail.com> wrote:
> 
> I use the elasticsearch plugin in bro. I know logstash works fine but its
> very cpu intensive. Thanx anyway. 

Technically it can be done, but it would require changes to the JSON formatter (in the core).  This is actually a pretty reasonable request (and I like the idea a lot!).  It might not be too much work to implement it, it just needs to be done.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From af7 at umbc.edu  Fri Oct 30 11:20:13 2015
From: af7 at umbc.edu (Arash Fallah)
Date: Fri, 30 Oct 2015 14:20:13 -0400
Subject: [Bro] Bro -> Elastic Search -> Kibana Issues | Default Analyzed
	Fields
In-Reply-To: <CAO79gBhFmz4M6THX1JfPDGRue9=hsH106BMcfbD7aEQG=5bOgw@mail.gmail.com>
References: <CAKDt6fSavmXLe_4pZRCQ6+h5r3XhzMp9QPRCh7ZKTA0c3AMVfA@mail.gmail.com>
	<CAKDt6fRGWpjHSyhi1cGtb0Rg1Y2oTzArZXj4D8AFqqC2a-zx2g@mail.gmail.com>
	<9065249B-8ED3-477F-97C8-12A88E7D03A9@gmail.com>
	<707CB89B-FCD1-42E3-9AA1-ECE282D4704E@umbc.edu>
	<CAKDt6fRrAXv3pTGpvrThkwJx-0D-bNC7=ySk9_3LZSOx9ncXdg@mail.gmail.com>
	<CAO79gBhFmz4M6THX1JfPDGRue9=hsH106BMcfbD7aEQG=5bOgw@mail.gmail.com>
Message-ID: <CAKDt6fSzzRh5dViOd+ajiYxZm2-otEj+R4ksCTFvNmSBykLWwA@mail.gmail.com>

Thanks Jay, it worked.

I used index templates as suggested by Jay,
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html,
to override the mapping for all string fields in Bro. I do not know the
complete ramifications of this but it works as a bandaid fix for now.

I got the entire mapping by doing the following: curl -XGET '
http://localhost:9200/bro-*/_mapping/?pretty=true'

Then I modified it via the index template linked here:
http://pastebin.com/qRFCzaMT

Warning, Bro might change its ElasticSearch mapping / field names at any
time and this index template should only be used as a reference. I fully
expect the current ElasticSearch writer to change so to take full advantage
of ElasticSearch and Kibana, it is very much a work in progress.

On Thu, Oct 29, 2015 at 9:36 AM, Jay Swan <sanjuanswan at gmail.com> wrote:

> I think you could use dynamic templates in Elasticsearch to have it set
> desired string fields to "not_analyzed" at index creation time, but I've
> never tried it myself:
>
>
> https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html
>
> Jay
>
> On Wed, Oct 28, 2015 at 10:45 AM, Arash Fallah <af7 at umbc.edu> wrote:
>
>> Does anyone have any suggestions on working around this? I was thinking
>> of using LogStash but I'm worried about scalability with that solution.
>>
>> I understand Bro wasn't explicitly made to interface with ElasticSearch
>> and that this might be a process.
>>
>> On Mon, Oct 26, 2015 at 8:47 PM, Arash Fallah <af7 at umbc.edu> wrote:
>>
>>> Exactly, which is why I suggest disabling string analyzation in Elastic
>>> Search for this field (and all IP fields). Otherwise, the IPv6 entries are
>>> ruining the results.
>>>
>>> On Oct 26, 2015, at 8:00 PM, Mark Parsons <mark.parsons at gmail.com>
>>> wrote:
>>>
>>> Arash,
>>> I imagine that is because elasticsearch currently does not support IPv6
>>> fields.
>>>
>>> See this elasticsearch github issue for more background
>>>
>>> https://github.com/elastic/elasticsearch/issues/3714
>>>
>>> Thanks,
>>> Mark
>>>
>>> Sent from my iPhone
>>>
>>> On Oct 26, 2015, at 3:16 PM, Arash Fallah <af7 at umbc.edu> wrote:
>>>
>>> Real Quick Update:
>>>
>>> My initial analysis was incorrect. IPv4 fields are processed correctly.
>>> The issue is with IPv6 fields. Same concept, different trigger. Here is a
>>> picture illustrating the problem:
>>>
>>> http://i.imgur.com/pdxRbmX.png
>>>
>>> On Mon, Oct 26, 2015 at 2:50 PM, Arash Fallah <af7 at umbc.edu> wrote:
>>>
>>>> I am having an issue when trying to process Bro data through Elastic
>>>> Search and Kibana.
>>>>
>>>> Specifically, I am doing basic quantitative statistics such as pulling
>>>> the Top 5 Originating IP's addresses by id_orig_h. However, Elastic Search
>>>> and Kibana breaks apart this field as shown below in the screenshot linked
>>>> below:
>>>>
>>>> http://i.imgur.com/m3BH6LP.png
>>>>
>>>> Basically, for some strings, the default Elastic Search analyzer will
>>>> segment them into different pieces based on the "." character. For example,
>>>> 130.85.12.20 will be broken apart into 130, 85, 12, and 80. This is because
>>>> in the Core Type mappings, the not index attribute is not explicitly set to
>>>> no (it defaults to yes). There is no way to adjust this for existing fields.
>>>>
>>>> Here is the current mapping created by Bro:
>>>>
>>>> "id.resp_h" : {
>>>>        "type" : "string"
>>>> }
>>>>
>>>> It should be:
>>>>
>>>> "id.resp_h" : {
>>>>        "type" : "string"
>>>>         "index": "not_analyzed"
>>>> }
>>>>
>>>> Suggestions?
>>>>
>>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>> <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>>>
>>>
>>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151030/0c230339/attachment-0001.html 

From daniel.guerra69 at gmail.com  Fri Oct 30 11:25:37 2015
From: daniel.guerra69 at gmail.com (Daniel Guerra)
Date: Fri, 30 Oct 2015 19:25:37 +0100
Subject: [Bro] Bro -> Elasticsearch -> Kibana4beta -> GeoLocation
In-Reply-To: <8129DD67-D968-415D-A564-1F4AF96C70C9@icir.org>
References: <C8E556C1-9066-4AED-8BA5-8CD77B236498@gmail.com>
	<D21C7E43-9F11-4DED-98C9-C1CC6C7F45DB@nswcsystems.co.uk>
	<1DA686BF-8B1F-4E85-B6CF-826E4F7A3E4D@gmail.com>
	<8129DD67-D968-415D-A564-1F4AF96C70C9@icir.org>
Message-ID: <D4CF5077-BD04-4F61-ABA0-0177F0A2A91F@gmail.com>

The funny thing is that elasticsearch stores the data internal
like the bro output is.

quote from the object document
Internally, this document is indexed as a simple, flat list of key-value pairs, something like this:

{
  "region":             "US",
  "manager.age":        30,
  "manager.name.first": "John",
  "manager.name.last":  "Smith"
}
Maybe this is an elasticsearch problem ?
To make it all work ElasticSearch.cc <http://elasticsearch.cc/> has to change to do
the geopoint mapping. And maybe stop analyse strings like
user_agent to avoid chopping of the result in the first word.
This could be solved by using url formatted strings you want
to show in graphs etc (no spaces).
The last thing is some naming collisions elasticsearch is 
confused about, like version in ssh & socks, but thats easy
to change in their main scripts.

Daniel
> On 30 Oct 2015, at 14:46, Seth Hall <seth at icir.org> wrote:
> 
> 
>> On Oct 29, 2015, at 9:33 PM, Daniel Guerra <daniel.guerra69 at gmail.com> wrote:
>> 
>> I use the elasticsearch plugin in bro. I know logstash works fine but its
>> very cpu intensive. Thanx anyway. 
> 
> Technically it can be done, but it would require changes to the JSON formatter (in the core).  This is actually a pretty reasonable request (and I like the idea a lot!).  It might not be too much work to implement it, it just needs to be done.
> 
>  .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151030/8a71a262/attachment.html 

From close at ou.edu  Fri Oct 30 12:19:24 2015
From: close at ou.edu (Close, Jason M.)
Date: Fri, 30 Oct 2015 19:19:24 +0000
Subject: [Bro] Help with getting PLUGIN_DBG_LOG to work
Message-ID: <D51DF55F-C45A-417B-AAEE-4C7CF0C6FD53@ou.edu>

I?ve looked at all of the plugins that come standard with Bro, and I haven?t really found PLUGIN_DBG_LOG being used.  So it is hard to find a good working example of how to print out values for debugging purposes.

As an example program, I?ve used the init program to create a plugin TestSpace::testplugin.  I?ve set ?plugin::TestSpace_testplugin::Plugin TestPlugin;? inside the ?TestSpace_testplugin? namespace in the Plugin.h.

I then call PLUGIN_DBG_LOG(TestPlugin, ???); inside my constructor of my TestPlugin.cc file, and I?m not seeing anything in the log directory.

Anyone every try to use this, and have something I can look at?

Jason Close
Information Security Analyst
OU Information Technology
Office: 405.325.8661  Cell: 405.421.1096

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151030/39845f16/attachment.html 

From robin at icir.org  Sat Oct 31 09:01:30 2015
From: robin at icir.org (Robin Sommer)
Date: Sat, 31 Oct 2015 09:01:30 -0700
Subject: [Bro] Help with getting PLUGIN_DBG_LOG to work
In-Reply-To: <D51DF55F-C45A-417B-AAEE-4C7CF0C6FD53@ou.edu>
References: <D51DF55F-C45A-417B-AAEE-4C7CF0C6FD53@ou.edu>
Message-ID: <20151031160130.GN47820@icir.org>

I suppose you have seen this?

    https://www.bro.org/sphinx-git/devel/plugins.html#debugging-plugins

A plugin that's using it is this one:
https://github.com/rsmmr/hilti/tree/master/bro/src (that's a bit more
of a complex one though :-)

Robin

On Fri, Oct 30, 2015 at 19:19 +0000, you wrote:

> I?ve looked at all of the plugins that come standard with Bro, and I haven?t really found PLUGIN_DBG_LOG being used.  So it is hard to find a good working example of how to print out values for debugging purposes.
> 
> As an example program, I?ve used the init program to create a plugin TestSpace::testplugin.  I?ve set ?plugin::TestSpace_testplugin::Plugin TestPlugin;? inside the ?TestSpace_testplugin? namespace in the Plugin.h.
> 
> I then call PLUGIN_DBG_LOG(TestPlugin, ???); inside my constructor of my TestPlugin.cc file, and I?m not seeing anything in the log directory.
> 
> Anyone every try to use this, and have something I can look at?
> 
> Jason Close
> Information Security Analyst
> OU Information Technology
> Office: 405.325.8661  Cell: 405.421.1096
> 

> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin