From seth at icir.org  Thu Aug  1 07:41:16 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 1 Aug 2013 10:41:16 -0400
Subject: [Bro] Bro working with a Cisco Ironport WSA
In-Reply-To: <20130729085407056.00000001536@ISIS-WIN7-5>
References: <20130729085407056.00000001536@ISIS-WIN7-5>
Message-ID: <891655E9-D1E3-4DF8-9A8F-2D72BF83A953@icir.org>

On Jul 29, 2013, at 8:54 AM, Chris Bennett <bennetc at lcc.edu> wrote:

> I am thinking that I will have to better engineer where I am looking at traffic, but I thought I would ask first.

I don't know anything about what's going on with the Cisco box you have, but it seems likely that you're right and you're going to want to changing how you're monitoring traffic.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130801/ae68153a/attachment.bin 

From jamesfhook at gmail.com  Thu Aug  1 08:23:22 2013
From: jamesfhook at gmail.com (James Hook)
Date: Thu, 1 Aug 2013 16:23:22 +0100
Subject: [Bro] Extracting files from HTTP
Message-ID: <CAC+_1SaiQ-xJgyNpuRp17KkvE7kV4Z_tijDWt7HUFGJ97UtCdQ@mail.gmail.com>

Hi all,
I'm new to Bro and I'm trying to follow Liam Randall's video tutorial on
extracting files from HTTP streams.
I've just cloned the git repository

git clone --recursive git://git.bro-ids.org/bro

and have built Bro according to the quick start guide on the Bro
documentation web page.

When Liam looks at "$PREFIX/share/base/protocols/http" I do not see
(on my computer) some of the files that Liam has in his installation
of Bro in his video.(e.g. file-extract.bro and file-ident.bro are not
there)

Can anyone point me in the right direction?

Thanks in advance

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130801/d3b7a82e/attachment.html 

From seth at icir.org  Thu Aug  1 08:57:13 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 1 Aug 2013 11:57:13 -0400
Subject: [Bro] Extracting files from HTTP
In-Reply-To: <CAC+_1SaiQ-xJgyNpuRp17KkvE7kV4Z_tijDWt7HUFGJ97UtCdQ@mail.gmail.com>
References: <CAC+_1SaiQ-xJgyNpuRp17KkvE7kV4Z_tijDWt7HUFGJ97UtCdQ@mail.gmail.com>
Message-ID: <4101635E-EDFD-42DF-A752-750F150AA44E@icir.org>

On Aug 1, 2013, at 11:23 AM, James Hook <jamesfhook at gmail.com> wrote:

> Can anyone point me in the right direction?

If you want to follow Liam's video you will need to use 2.1.  Git master has huge changes to how file handling works.

  .Seth


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130801/268f73bc/attachment.bin 

From lists at g-clef.net  Fri Aug  2 11:33:03 2013
From: lists at g-clef.net (aaron gee-clough)
Date: Fri, 02 Aug 2013 14:33:03 -0400
Subject: [Bro] troubleshooting bro memory usage?
Message-ID: <51FBFB5F.9090507@g-clef.net>


Hello,

I've just put in two sensors running bro (with security onion), and am 
having trouble with the bro processes progressively growing in RAM 
usage, until they crash or become unresponsive. For example, I have one 
bro worker process right now that's reached 2.8 GB in 2 hours while 
watching a < 100MB link. None of the other processes 
(manager/proxy/other workers) are anywhere near that...it's just this 
one worker.

Are there any config options I can enable to attempt to find the cause 
of the memory leak? Also, since I'm confident the link I'm watching is 
missing some traffic (the span it's on is slightly mis-configured at the 
moment), where can I configure protocol timeouts?

Thanks.

aaron


From aj27744 at gmail.com  Mon Aug  5 01:18:47 2013
From: aj27744 at gmail.com (Anil Joshi)
Date: Mon, 5 Aug 2013 13:48:47 +0530
Subject: [Bro] Bro as an Anomaly Detector.
Message-ID: <CALF0v5eCtP-+3bWwHMSBxXjSLABXUWBQ74s_VsmQhTe47tJ-5w@mail.gmail.com>

Hi All,

I am all new to this bro software.
Before starting my work on it,let me explain what's my goal is :
My goal is to develop an ANOMALY DETECTION software using an open source
software.

Now my query is has anyone used bro as an anomaly detector?
If yes can you please tell which anomalies have you detected through bro?


Thanks
Anil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130805/074c77c0/attachment.html 

From sheharbano.k at gmail.com  Mon Aug  5 12:43:24 2013
From: sheharbano.k at gmail.com (Sheharbano Khattak)
Date: Tue, 6 Aug 2013 00:43:24 +0500
Subject: [Bro] Bro as an Anomaly Detector.
Message-ID: <CAA6v-1bf2gHabCA73NM4csey_E57xEMxQcKGEKU3oe8qS8ejZg@mail.gmail.com>

Dear Anil,

Bro is more a network monitor than an anomaly detector. If you wish
to write an anomaly detector, Bro's domain scripting language will greatly
simplify network analysis for you. I believe Bro doesn't have the more
involved
machine learning style anomaly detection* at the moment. However, there are
some scripts for detection of SSH brute forcing, SQL injection attacks and
malicious network scan that rely on deviation from a threshold. You will
find these scripts in the directory /usr/local/bro/share/bro/scripts/policy
(you might
need to adjust the path depending on where you installed Bro on your
machine).
There is a new framework SumStats**  (Bro frameworks are similar to what we
call
libraries in most other languages--they facilitate tasks which would be
otherwise
rather tedious to perform) that simplifies the overall task of performing
measurements
over network data. Hope this helps.

* You might be interested in looking at the paper [www.icir.org/*robin*
/papers/oakland10-ml.pdf] to know why.

**http://trac.bro-ids.org/sphinx-git/_downloads/main16.bro

Regards.
-- 
Sheharbano Khattak

http://etheryell.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130806/52443d6f/attachment.html 

From aj27744 at gmail.com  Tue Aug  6 06:07:40 2013
From: aj27744 at gmail.com (Anil Joshi)
Date: Tue, 6 Aug 2013 18:37:40 +0530
Subject: [Bro] Reference book on Anomaly Detection
Message-ID: <CALF0v5eFMuG3A9euPPoLbx+WyWr50eKkCQqVZHFogd3Jk+Vn0w@mail.gmail.com>

Hi ALL,

Can you all just tell some good reference book on Anomaly Detection.
Where i can get some knowledge about Anomaly Detection from basics.
I don't know anything about anomaly detection so need an book from basics.

Thanks
Anil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130806/190d0e62/attachment.html 

From bernhard at ICSI.Berkeley.EDU  Tue Aug  6 07:11:39 2013
From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann)
Date: Tue, 6 Aug 2013 09:11:39 -0500
Subject: [Bro] Reference book on Anomaly Detection
In-Reply-To: <CALF0v5eFMuG3A9euPPoLbx+WyWr50eKkCQqVZHFogd3Jk+Vn0w@mail.gmail.com>
References: <CALF0v5eFMuG3A9euPPoLbx+WyWr50eKkCQqVZHFogd3Jk+Vn0w@mail.gmail.com>
Message-ID: <2C04ED56-D393-4CD0-B3B9-FC6805CBA210@icsi.berkeley.edu>

Hi Anil,

if I am not very much mistaken, we talked on IRC a few days ago and you
are writing your thesis on anomaly detection system.

I think you should look up the resources that were already pointed out to you 
by me and others - you also can take a look at the paper section on bro.org, 
some of them talk about anomaly detection.

Other than that, I think it would be a good idea to ask your advisor for
guidance on how to approach the work on your thesis.

Bernhard

On Aug 6, 2013, at 8:07 AM, Anil Joshi <aj27744 at gmail.com> wrote:

> Hi ALL,
> 
> Can you all just tell some good reference book on Anomaly Detection.
> Where i can get some knowledge about Anomaly Detection from basics.
> I don't know anything about anomaly detection so need an book from basics.
> 
> Thanks
> Anil
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From chris.doman at cantab.net  Wed Aug  7 03:08:47 2013
From: chris.doman at cantab.net (Chris Doman)
Date: Wed, 7 Aug 2013 11:08:47 +0100
Subject: [Bro]  Reference book on Anomaly Detection
Message-ID: <CAOjvSKCitcKq3Wwraeiea7_gEoNNXy=+7eDsnw0BLqxGJkWmeQ@mail.gmail.com>

Hi Anil,

 Just as a quick note I've used bro data imported into mongodb to perform
anomaly detection with some success. I used a different approach but
http://www.cert.org/flocon/2008/presentations/balland_flocon2008.pdf may
give you some ideas.

Thanks,
Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130807/bb30f013/attachment.html 

From chris.doman at cantab.net  Wed Aug  7 03:16:04 2013
From: chris.doman at cantab.net (Chris Doman)
Date: Wed, 7 Aug 2013 11:16:04 +0100
Subject: [Bro] What goes into http_log?
Message-ID: <CAOjvSKCtcPL0Y7JXASv+Ax5WLn-Jch8A0wmJgPNF=iYVwP7Tdw@mail.gmail.com>

Hi all,

 Does anyone know if http_log records everything from port 80, or anything
detected as the HTTP protocol etc?

I'm asking as I would like to detect software that communicates over port
80 or 8080 but that isn't infact using HTTP (some beaconing malware for
example communicates over port 80).

 And similarly it would be very useful to be able to detect non SSL over
port 443. I'm thinking that checking for ssl.log where cipher="-" might be
a good idea, if ssl.log records everything over port 443.

 Apologies if this has been answered before, I couldn't find the answer
from a quick google and code check.

Thanks,
Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130807/87a24dae/attachment.html 

From punchpernickle at gmail.com  Wed Aug  7 09:07:03 2013
From: punchpernickle at gmail.com (Dani Witherspoon)
Date: Wed, 7 Aug 2013 12:07:03 -0400
Subject: [Bro] Adding a human-readable timestamp field.
Message-ID: <CAO266SHF4_vW2G5PQfLtV=uaurVH9oN3AQvLsed=XdV4OHDH6A@mail.gmail.com>

Hi all!

Full disclosure: I'm a bit of a bro-ginner, only been working with bro for
about a month now.

I'm working on a bro script to add a human-readable timestamp field to my
(http) logs, but I've run into a bit of a pickle.

Though my script checks out ("bro is ok!"), installs just fine, and even
adds the appropriate field...every entry in the field column is unset! I'm
not sure where I've gone astray , and I would appreciate any pointers.

I've included the text of my script below.

Thank you!
-Dani

@load base/protocols/http

module HTTP;

export {
        redef record Info += {
                ## A human-readable timestamp
                human_time: string &log &optional;
        };
}

event time_translate(c: connection, rec: HTTP::Info)
        {
    local format: string = "%F-%H-%M";
        c$http$human_time = strftime(format, rec$ts);
        }
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130807/4a4cb159/attachment.html 

From ashwin.shirvanthe at gmail.com  Wed Aug  7 09:44:19 2013
From: ashwin.shirvanthe at gmail.com (Ashwin Rao)
Date: Wed, 7 Aug 2013 18:44:19 +0200
Subject: [Bro] Encrypting bro logs before storing to disk
Message-ID: <CAJNbMrzEXRRCvTrb=_erR-hV0KL_ZeF-7PmD87BddaPs9+o=zQ@mail.gmail.com>

Hi,

I am setting up bro to monitor traffic passing through my proxy that I
shall use for some experiments and measurements.

For IRB compliance, I need to encrypt the logs using a RSA public key
before the logs are stored on the disk. I would like to know if anyone has
run into a similar requirement while using bro.

In any case, the only way I can currently think of encrypting the logs
before a write is by wrapping the safe_write and safe_close functions (in
"src/util.cc" file in the source tree). The wrapper function shall keep the
file specific encryption state in the Ascii class present in
src/logging/writers/Ascii.cc. This wrapper function shall first encrypt the
data and then call either safe_write or safe_close respectively.

I would like get feedback on whether this seems right and if I missing
something that has already been done and can be used without this hack.

Regards,
Ashwin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130807/ee2c9d28/attachment.html 

From jlay at slave-tothe-box.net  Wed Aug  7 10:02:20 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Wed, 07 Aug 2013 11:02:20 -0600
Subject: [Bro] Adding a human-readable timestamp field.
In-Reply-To: <CAO266SHF4_vW2G5PQfLtV=uaurVH9oN3AQvLsed=XdV4OHDH6A@mail.gmail.com>
References: <CAO266SHF4_vW2G5PQfLtV=uaurVH9oN3AQvLsed=XdV4OHDH6A@mail.gmail.com>
Message-ID: <ece6b7a9759086cf4caa4f43b4af721d@localhost>

On 2013-08-07 10:07, Dani Witherspoon wrote:
> Hi all!
>
> Full disclosure: Im a bit of a bro-ginner, only been working with bro
> for about a month now.
>
> Im working on a bro script to add a human-readable timestamp field to
> my (http) logs, but Ive run into a bit of a pickle.
>
> Though my script checks out ("bro is ok!"), installs just fine, and
> even adds the appropriate field...every entry in the field column is
> unset! Im not sure where Ive gone astray , and I would appreciate any
> pointers.
>
> Ive included the text of my script below.
>
> Thank you!
> -Dani
>
> @load base/protocols/http
>
> module HTTP;
>
> export {
> ??????? redef record Info += {
> ??????????????? ## A human-readable timestamp
>  ??????????????? human_time: string &log &optional;
> ??????? };
> }
>
> event time_translate(c: connection, rec: HTTP::Info)
> ??????? {
> ??? local format: string = "%F-%H-%M";
> ??????? c$http$human_time = strftime(format, rec$ts);
>  ??????? }

Seth,

I see a lot of these...any chance we could get a config feature request 
that would default to human readable.

James


From jswan at sugf.com  Wed Aug  7 10:24:25 2013
From: jswan at sugf.com (Swan, Jay)
Date: Wed, 7 Aug 2013 17:24:25 +0000
Subject: [Bro] What goes into http_log?
In-Reply-To: <CAOjvSKCtcPL0Y7JXASv+Ax5WLn-Jch8A0wmJgPNF=iYVwP7Tdw@mail.gmail.com>
References: <CAOjvSKCtcPL0Y7JXASv+Ax5WLn-Jch8A0wmJgPNF=iYVwP7Tdw@mail.gmail.com>
Message-ID: <20130807172426.BEA262C4005@rock.ICSI.Berkeley.EDU>

It records everything detected as HTTP. Here's a sample showing a bunch of ports detected as HTTP:

me at so1204:/nsm/bro/logs/current$ bro-cut id.resp_p < http_eth1.log | sort -u
2350
3690
4004
80
8014
8080
8888
9090

For the second part I think the right way would be to search conn.log for tcp/443, then "grep -v ssl" on the results. But I'm not sure.



From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Chris Doman
Sent: Wednesday, August 07, 2013 4:16 AM
To: bro at bro.org
Subject: [Bro] What goes into http_log?

Hi all,

 Does anyone know if http_log records everything from port 80, or anything detected as the HTTP protocol etc?

I'm asking as I would like to detect software that communicates over port 80 or 8080 but that isn't infact using HTTP (some beaconing malware for example communicates over port 80).

 And similarly it would be very useful to be able to detect non SSL over port 443. I'm thinking that checking for ssl.log where cipher="-" might be a good idea, if ssl.log records everything over port 443.

 Apologies if this has been answered before, I couldn't find the answer from a quick google and code check.

Thanks,
Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130807/a9aef6c3/attachment.html 

From JAzoff at albany.edu  Wed Aug  7 10:48:34 2013
From: JAzoff at albany.edu (Justin Azoff)
Date: Wed, 7 Aug 2013 13:48:34 -0400
Subject: [Bro] Adding a human-readable timestamp field.
In-Reply-To: <CAO266SHF4_vW2G5PQfLtV=uaurVH9oN3AQvLsed=XdV4OHDH6A@mail.gmail.com>
References: <CAO266SHF4_vW2G5PQfLtV=uaurVH9oN3AQvLsed=XdV4OHDH6A@mail.gmail.com>
Message-ID: <20130807174834.GJ5291@datacomm.albany.edu>

On Wed, Aug 07, 2013 at 12:07:03PM -0400, Dani Witherspoon wrote:
> event time_translate(c: connection, rec: HTTP::Info)
>         {
>     local format: string = "%F-%H-%M";
>         c$http$human_time = strftime(format, rec$ts);
>         }
> 

You're right up to here.. the problem is nothing will trigger the
time_translate event.  You need to use one of the existing events that
will fire for http connections.

I would try:

event HTTP::log_http(rec: HTTP::Info)
{
    ..
}

I believe that fires just before the entry is logged, if that doesn't
work an event like connection_established or http_request would
definitely work.

-- 
-- Justin Azoff
-- Network Security & Performance Analyst


From la_arshadi at yahoo.com  Wed Aug  7 11:13:55 2013
From: la_arshadi at yahoo.com (Laleh Arshadi)
Date: Wed, 7 Aug 2013 11:13:55 -0700 (PDT)
Subject: [Bro] truncated packets
Message-ID: <1375899235.79828.YahooMailNeo@web140604.mail.bf1.yahoo.com>

Dear All,
?
I know that Bro can analyze offline traffic with its -r option but I wonder if it can analyze the traffic contains?truncated packets? I remember a few years ago when I run old versions of Bro on the MAWI traffic, it didn't work properly since the packets were all truncated at 54 bytes. Maybe this has changed in the newer versions? 
?
Regards
Laleh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130807/b81d8adb/attachment.html 

From slagell at illinois.edu  Wed Aug  7 11:29:25 2013
From: slagell at illinois.edu (Slagell, Adam J)
Date: Wed, 7 Aug 2013 18:29:25 +0000
Subject: [Bro] truncated packets
In-Reply-To: <1375899235.79828.YahooMailNeo@web140604.mail.bf1.yahoo.com>
References: <1375899235.79828.YahooMailNeo@web140604.mail.bf1.yahoo.com>
Message-ID: <558D23D33781EF45A69229CDAC6BF151111B2369@CITESMBX6.ad.uillinois.edu>

You may try turning off the checksum verification.

On Aug 7, 2013, at 1:13 PM, Laleh Arshadi <la_arshadi at yahoo.com<mailto:la_arshadi at yahoo.com>>
 wrote:

Dear All,

I know that Bro can analyze offline traffic with its -r option but I wonder if it can analyze the traffic contains truncated packets? I remember a few years ago when I run old versions of Bro on the MAWI traffic, it didn't work properly since the packets were all truncated at 54 bytes. Maybe this has changed in the newer versions?

Regards
Laleh
_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

------

Adam J. Slagell
Chief Information Security Officer
Sr. Research Scientist
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
www.slagell.info<http://www.slagell.info>

"Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure."









-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130807/5ccc892e/attachment.html 

From slagell at illinois.edu  Wed Aug  7 11:30:38 2013
From: slagell at illinois.edu (Slagell, Adam J)
Date: Wed, 7 Aug 2013 18:30:38 +0000
Subject: [Bro] truncated packets
In-Reply-To: <5F71BFD3-F812-4F72-89F8-D4A3C967FDCD@illinois.edu>
References: <1375899235.79828.YahooMailNeo@web140604.mail.bf1.yahoo.com>
	<5F71BFD3-F812-4F72-89F8-D4A3C967FDCD@illinois.edu>
Message-ID: <558D23D33781EF45A69229CDAC6BF151111B2398@CITESMBX6.ad.uillinois.edu>

See http://comments.gmane.org/gmane.comp.security.detection.bro/3168

On Aug 7, 2013, at 1:29 PM, Adam J. Slagell <slagell at illinois.edu<mailto:slagell at illinois.edu>> wrote:

You may try turning off the checksum verification.

On Aug 7, 2013, at 1:13 PM, Laleh Arshadi <la_arshadi at yahoo.com<mailto:la_arshadi at yahoo.com>>
 wrote:

Dear All,

I know that Bro can analyze offline traffic with its -r option but I wonder if it can analyze the traffic contains truncated packets? I remember a few years ago when I run old versions of Bro on the MAWI traffic, it didn't work properly since the packets were all truncated at 54 bytes. Maybe this has changed in the newer versions?

Regards
Laleh
_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

------

Adam J. Slagell
Chief Information Security Officer
Sr. Research Scientist
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
www.slagell.info<http://www.slagell.info/>

"Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure."










------

Adam J. Slagell
Chief Information Security Officer
Sr. Research Scientist
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
www.slagell.info<http://www.slagell.info>

"Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure."









-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130807/ba004eee/attachment.html 

From vladg at cmu.edu  Wed Aug  7 11:37:32 2013
From: vladg at cmu.edu (Vlad Grigorescu)
Date: Wed, 7 Aug 2013 18:37:32 +0000
Subject: [Bro] truncated packets
In-Reply-To: <9537_1375900327_r77IW5fS015057_558D23D33781EF45A69229CDAC6BF151111B2398@CITESMBX6.ad.uillinois.edu>
References: <1375899235.79828.YahooMailNeo@web140604.mail.bf1.yahoo.com>
	<5F71BFD3-F812-4F72-89F8-D4A3C967FDCD@illinois.edu>
	<9537_1375900327_r77IW5fS015057_558D23D33781EF45A69229CDAC6BF151111B2398@CITESMBX6.ad.uillinois.edu>
Message-ID: <1202BE242E080642B0CD0AD0A03E8552D7A72C@PGH-MSGMB-03.andrew.ad.cmu.edu>

Disabling checksum verification won't help much. You'll end up getting protocol violations because the protocol truncates so quickly. 54 bytes really doesn't give you much to work with. I assume you're just interested in getting connection logs?

  --Vlad

"The Bro list is public record anyway."


On Aug 7, 2013, at 1:30 PM, Slagell, Adam J <slagell at illinois.edu> wrote:

> See http://comments.gmane.org/gmane.comp.security.detection.bro/3168
> 
> On Aug 7, 2013, at 1:29 PM, Adam J. Slagell <slagell at illinois.edu> wrote:
> 
>> You may try turning off the checksum verification.
>> 
>> On Aug 7, 2013, at 1:13 PM, Laleh Arshadi <la_arshadi at yahoo.com>
>>  wrote:
>> 
>>> Dear All,
>>>  
>>> I know that Bro can analyze offline traffic with its -r option but I wonder if it can analyze the traffic contains truncated packets? I remember a few years ago when I run old versions of Bro on the MAWI traffic, it didn't work properly since the packets were all truncated at 54 bytes. Maybe this has changed in the newer versions?
>>>  
>>> Regards
>>> Laleh
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> 
>> ------
>> 
>> Adam J. Slagell
>> Chief Information Security Officer
>> Sr. Research Scientist
>> National Center for Supercomputing Applications
>> University of Illinois at Urbana-Champaign
>> www.slagell.info
>> 
>> "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." 
> 
> ------
> 
> Adam J. Slagell
> Chief Information Security Officer
> Sr. Research Scientist
> National Center for Supercomputing Applications
> University of Illinois at Urbana-Champaign
> www.slagell.info
> 
> "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From la_arshadi at yahoo.com  Wed Aug  7 11:41:32 2013
From: la_arshadi at yahoo.com (Laleh Arshadi)
Date: Wed, 7 Aug 2013 11:41:32 -0700 (PDT)
Subject: [Bro] truncated packets
In-Reply-To: <1202BE242E080642B0CD0AD0A03E8552D7A72C@PGH-MSGMB-03.andrew.ad.cmu.edu>
References: <1375899235.79828.YahooMailNeo@web140604.mail.bf1.yahoo.com>
	<5F71BFD3-F812-4F72-89F8-D4A3C967FDCD@illinois.edu>
	<9537_1375900327_r77IW5fS015057_558D23D33781EF45A69229CDAC6BF151111B2398@CITESMBX6.ad.uillinois.edu>
	<1202BE242E080642B0CD0AD0A03E8552D7A72C@PGH-MSGMB-03.andrew.ad.cmu.edu>
Message-ID: <1375900892.24717.YahooMailNeo@web140602.mail.bf1.yahoo.com>




Disabling checksum verification won't help much. You'll end up getting protocol violations because the protocol truncates so quickly. 54 bytes really doesn't give you much to work with. I assume you're just interested in getting connection logs?

? --Vlad

Yes... exactly. Is it possible to do so?
?
Laleh



On Aug 7, 2013, at 1:30 PM, Slagell, Adam J <slagell at illinois.edu> wrote:

> See http://comments.gmane.org/gmane.comp.security.detection.bro/3168
> 
> On Aug 7, 2013, at 1:29 PM, Adam J. Slagell <slagell at illinois.edu> wrote:
> 
>> You may try turning off the checksum verification.
>> 
>> On Aug 7, 2013, at 1:13 PM, Laleh Arshadi <la_arshadi at yahoo.com>
>>? wrote:
>> 
>>> Dear All,
>>>? 
>>> I know that Bro can analyze offline traffic with its -r option but I wonder if it can analyze the traffic contains truncated packets? I remember a few years ago when I run old versions of Bro on the MAWI traffic, it didn't work properly since the packets were all truncated at 54 bytes. Maybe this has changed in the newer versions?
>>>? 
>>> Regards
>>> Laleh
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> 
>> ------
>> 
>> Adam J. Slagell
>> Chief Information Security Officer
>> Sr. Research Scientist
>> National Center for Supercomputing Applications
>> University of Illinois at Urbana-Champaign
>> www.slagell.info
>> 
>> "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." 
> 
> ------
> 
> Adam J. Slagell
> Chief Information Security Officer
> Sr. Research Scientist
> National Center for Supercomputing Applications
> University of Illinois at Urbana-Champaign
> www.slagell.info
> 
> "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130807/5f6e58b1/attachment.html 

From punchpernickle at gmail.com  Thu Aug  8 06:10:43 2013
From: punchpernickle at gmail.com (Dani Witherspoon)
Date: Thu, 8 Aug 2013 09:10:43 -0400
Subject: [Bro] Adding a human-readable timestamp field.
In-Reply-To: <20130807174834.GJ5291@datacomm.albany.edu>
References: <CAO266SHF4_vW2G5PQfLtV=uaurVH9oN3AQvLsed=XdV4OHDH6A@mail.gmail.com>
	<20130807174834.GJ5291@datacomm.albany.edu>
Message-ID: <CAO266SG2F3KwTs_uuNOVVT+_YUtbjYPa0_CB3RJaH42yc4Z1Wg@mail.gmail.com>

Thank you so much, Justin! This did the trick --  I really appreciate the
guidance!

If anybody's interested, here's the working bro-code:

@load base/protocols/http

module HTTP;

export {
        redef record Info += {
                ## A human-readable timestamp
                human_time: string &log &optional;
        };
}

event http_request(c: connection, method: string, original_URI: string,
unescaped_URI: string, version: string)
        {
        local format: string = "%F, %H:%M";
        c$http$human_time = strftime(format, c$http$ts);
        }







On Wed, Aug 7, 2013 at 1:48 PM, Justin Azoff <JAzoff at albany.edu> wrote:

> On Wed, Aug 07, 2013 at 12:07:03PM -0400, Dani Witherspoon wrote:
> > event time_translate(c: connection, rec: HTTP::Info)
> >         {
> >     local format: string = "%F-%H-%M";
> >         c$http$human_time = strftime(format, rec$ts);
> >         }
> >
>
> You're right up to here.. the problem is nothing will trigger the
> time_translate event.  You need to use one of the existing events that
> will fire for http connections.
>
> I would try:
>
> event HTTP::log_http(rec: HTTP::Info)
> {
>     ..
> }
>
> I believe that fires just before the entry is logged, if that doesn't
> work an event like connection_established or http_request would
> definitely work.
>
> --
> -- Justin Azoff
> -- Network Security & Performance Analyst
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130808/eb64e369/attachment.html 

From harrison.wood at gmail.com  Thu Aug  8 09:09:56 2013
From: harrison.wood at gmail.com (Harrison Wood)
Date: Thu, 8 Aug 2013 11:09:56 -0500
Subject: [Bro] Adding a human-readable timestamp field.
In-Reply-To: <CAO266SG2F3KwTs_uuNOVVT+_YUtbjYPa0_CB3RJaH42yc4Z1Wg@mail.gmail.com>
References: <CAO266SHF4_vW2G5PQfLtV=uaurVH9oN3AQvLsed=XdV4OHDH6A@mail.gmail.com>
	<20130807174834.GJ5291@datacomm.albany.edu>
	<CAO266SG2F3KwTs_uuNOVVT+_YUtbjYPa0_CB3RJaH42yc4Z1Wg@mail.gmail.com>
Message-ID: <CALH+DwXLbKoT7atST7P0ZVpu_KF7xbthMRgNXpLrUwj=YrC5qw@mail.gmail.com>

Thanks for posting your script! I just added it to my install so I can stop
doing date -d@ all the time.


On Thu, Aug 8, 2013 at 8:10 AM, Dani Witherspoon
<punchpernickle at gmail.com>wrote:

> Thank you so much, Justin! This did the trick --  I really appreciate the
> guidance!
>
> If anybody's interested, here's the working bro-code:
>
>
> @load base/protocols/http
>
> module HTTP;
>
> export {
>         redef record Info += {
>                 ## A human-readable timestamp
>                 human_time: string &log &optional;
>         };
> }
>
> event http_request(c: connection, method: string, original_URI: string,
> unescaped_URI: string, version: string)
>         {
>         local format: string = "%F, %H:%M";
>         c$http$human_time = strftime(format, c$http$ts);
>         }
>
>
>
>
>
>
>
> On Wed, Aug 7, 2013 at 1:48 PM, Justin Azoff <JAzoff at albany.edu> wrote:
>
>> On Wed, Aug 07, 2013 at 12:07:03PM -0400, Dani Witherspoon wrote:
>> > event time_translate(c: connection, rec: HTTP::Info)
>> >         {
>> >     local format: string = "%F-%H-%M";
>> >         c$http$human_time = strftime(format, rec$ts);
>> >         }
>> >
>>
>> You're right up to here.. the problem is nothing will trigger the
>> time_translate event.  You need to use one of the existing events that
>> will fire for http connections.
>>
>> I would try:
>>
>> event HTTP::log_http(rec: HTTP::Info)
>> {
>>     ..
>> }
>>
>> I believe that fires just before the entry is logged, if that doesn't
>> work an event like connection_established or http_request would
>> definitely work.
>>
>> --
>> -- Justin Azoff
>> -- Network Security & Performance Analyst
>>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130808/a3bf38af/attachment.html 

From punchpernickle at gmail.com  Thu Aug  8 09:30:40 2013
From: punchpernickle at gmail.com (Dani Witherspoon)
Date: Thu, 8 Aug 2013 12:30:40 -0400
Subject: [Bro] Adding a human-readable timestamp field.
In-Reply-To: <CALH+DwXLbKoT7atST7P0ZVpu_KF7xbthMRgNXpLrUwj=YrC5qw@mail.gmail.com>
References: <CAO266SHF4_vW2G5PQfLtV=uaurVH9oN3AQvLsed=XdV4OHDH6A@mail.gmail.com>
	<20130807174834.GJ5291@datacomm.albany.edu>
	<CAO266SG2F3KwTs_uuNOVVT+_YUtbjYPa0_CB3RJaH42yc4Z1Wg@mail.gmail.com>
	<CALH+DwXLbKoT7atST7P0ZVpu_KF7xbthMRgNXpLrUwj=YrC5qw@mail.gmail.com>
Message-ID: <CAO266SE9j86oQDfEBMLG=dr51_-reCeW=++tDEGm=u-zudGnrw@mail.gmail.com>

No worries! This only works for HTTP logs --  you'd have to edit it for
other protocols, which I've done for SSL, FTP, and SSH. I've included those
scripts below, in case anybody else would like to use them. Let me know if
any issues crop up, or if the coding isn't in the bro-spirit. :)

# File:
human_time_ftp.bro

@load base/protocols/ftp

module FTP;

export {
        redef record Info += {
                ## A human-readable timestamp
                human_time: string &log &optional;
        };
}

event ftp_request(c: connection , command: string , arg: string)
        {
    local format: string = "%F, %H:%M:%S";
        c$ftp$human_time = strftime(format, c$ftp$ts);
        }

--------------------------

# File: human_time_ssl

@load base/protocols/ssl

module SSL;

export {
        redef record Info += {
                ## A human-readable timestamp
                human_time: string &log &optional;
        };
}

event ssl_established(c: connection)
        {
        local format: string = "%F, %H:%M:%S";
        c$ssl$human_time = strftime(format, c$ssl$ts);
        }

-----------------------------

# File: human_time_ssh

@load base/protocols/ssh

module SSH;

export {
        redef record Info += {
                ## A human-readable timestamp
                human_time: string &log &optional;
        };
}

event ssh_client_version(c: connection , version: string)
        {
        local format: string = "%F, %H:%M:%S";
        c$ssh$human_time = strftime(format, c$ssh$ts);
        }

-----------------------

etc, etc, etc --  I'm sure you see how you could continue extending it to
other protocols! Best of luck. :)



On Thu, Aug 8, 2013 at 12:09 PM, Harrison Wood <harrison.wood at gmail.com>wrote:

> Thanks for posting your script! I just added it to my install so I can
> stop doing date -d@ all the time.
>
>
> On Thu, Aug 8, 2013 at 8:10 AM, Dani Witherspoon <punchpernickle at gmail.com
> > wrote:
>
>> Thank you so much, Justin! This did the trick --  I really appreciate the
>> guidance!
>>
>> If anybody's interested, here's the working bro-code:
>>
>>
>> @load base/protocols/http
>>
>> module HTTP;
>>
>> export {
>>         redef record Info += {
>>                 ## A human-readable timestamp
>>                 human_time: string &log &optional;
>>         };
>> }
>>
>> event http_request(c: connection, method: string, original_URI: string,
>> unescaped_URI: string, version: string)
>>         {
>>         local format: string = "%F, %H:%M";
>>         c$http$human_time = strftime(format, c$http$ts);
>>         }
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Aug 7, 2013 at 1:48 PM, Justin Azoff <JAzoff at albany.edu> wrote:
>>
>>> On Wed, Aug 07, 2013 at 12:07:03PM -0400, Dani Witherspoon wrote:
>>> > event time_translate(c: connection, rec: HTTP::Info)
>>> >         {
>>> >     local format: string = "%F-%H-%M";
>>> >         c$http$human_time = strftime(format, rec$ts);
>>> >         }
>>> >
>>>
>>> You're right up to here.. the problem is nothing will trigger the
>>> time_translate event.  You need to use one of the existing events that
>>> will fire for http connections.
>>>
>>> I would try:
>>>
>>> event HTTP::log_http(rec: HTTP::Info)
>>> {
>>>     ..
>>> }
>>>
>>> I believe that fires just before the entry is logged, if that doesn't
>>> work an event like connection_established or http_request would
>>> definitely work.
>>>
>>> --
>>> -- Justin Azoff
>>> -- Network Security & Performance Analyst
>>>
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130808/6cb3152a/attachment.html 

From jsiwek at illinois.edu  Thu Aug  8 13:34:30 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Thu, 8 Aug 2013 20:34:30 +0000
Subject: [Bro] Encrypting bro logs before storing to disk
In-Reply-To: <CAJNbMrzEXRRCvTrb=_erR-hV0KL_ZeF-7PmD87BddaPs9+o=zQ@mail.gmail.com>
References: <CAJNbMrzEXRRCvTrb=_erR-hV0KL_ZeF-7PmD87BddaPs9+o=zQ@mail.gmail.com>
Message-ID: <B3E27C90B7D2DD4CAEF09215A29EB0D117086306@CITESMBX2.ad.uillinois.edu>

> I would like get feedback on whether this seems right and if I missing something that has already been done and can be used without this hack. 

That approach (changing every place the Ascii writer does a write in the C++ land) sounds like it would work.  And if the encryption behavior were made toggle-able (possibly via some script-land variables that could be set/redef'd), that would make a patch to do such a thing more acceptable.

- Jon


From lists at g-clef.net  Fri Aug  9 12:30:48 2013
From: lists at g-clef.net (aaron gee-clough)
Date: Fri, 09 Aug 2013 15:30:48 -0400
Subject: [Bro] troubleshooting bro memory usage?
References: <51FBFB5F.9090507@g-clef.net>
Message-ID: <52054368.7030001@g-clef.net>


Hello,

I've just come across something that implies Bro is caching all DNS 
resolutions that go past it 
(https://bro-tracker.atlassian.net/browse/BIT-964). The bro systems I 
recently put in are in front of our main internal DNS resolvers, so 
almost all of the traffic they see is DNS resolution requests/answers. 
If Bro is caching all DNS, that would go a long way to explaining why 
bro's memory usage is continually increasing for my two sensors.

Is there a way to disable this caching? (or have I mis-understood what 
bro's doing with DNS?)

Thanks.

aaron

On 08/02/2013 02:33 PM, aaron gee-clough wrote:
>
> Hello,
>
> I've just put in two sensors running bro (with security onion), and am
> having trouble with the bro processes progressively growing in RAM
> usage, until they crash or become unresponsive. For example, I have one
> bro worker process right now that's reached 2.8 GB in 2 hours while
> watching a < 100MB link. None of the other processes
> (manager/proxy/other workers) are anywhere near that...it's just this
> one worker.
>
> Are there any config options I can enable to attempt to find the cause
> of the memory leak? Also, since I'm confident the link I'm watching is
> missing some traffic (the span it's on is slightly mis-configured at the
> moment), where can I configure protocol timeouts?
>
> Thanks.
>
> aaron
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From seth at icir.org  Sat Aug 10 08:19:40 2013
From: seth at icir.org (Seth Hall)
Date: Sat, 10 Aug 2013 11:19:40 -0400
Subject: [Bro] troubleshooting bro memory usage?
In-Reply-To: <52054368.7030001@g-clef.net>
References: <51FBFB5F.9090507@g-clef.net> <52054368.7030001@g-clef.net>
Message-ID: <298E63A0-6C1C-4580-A6FA-87822422E165@icir.org>

On Aug 9, 2013, at 3:30 PM, aaron gee-clough <lists at g-clef.net> wrote:

> Is there a way to disable this caching? (or have I mis-understood what 
> bro's doing with DNS?)


That's unrelated.  It's referring to DNS lookup requests happening at script land.  We ran into a case once where someone had written a script that did two reverse hostname lookups for every connection that was established (don't do this, it's *really* not a good idea).  Although I should point out that their Bro cluster was running quite well even in the face of that, but I don't think their DNS resolver was very happy about it. :)

In general, monitoring in front of a DNS resolver should be just fine.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130810/1fabeadc/attachment.bin 

From lists at g-clef.net  Sun Aug 11 05:39:03 2013
From: lists at g-clef.net (Aaron Gee-Clough)
Date: Sun, 11 Aug 2013 08:39:03 -0400
Subject: [Bro] troubleshooting bro memory usage?
References: <51FBFB5F.9090507@g-clef.net> <52054368.7030001@g-clef.net>
	<298E63A0-6C1C-4580-A6FA-87822422E165@icir.org>
Message-ID: <520785E7.9000902@g-clef.net>


On 8/10/2013 11:19 AM, Seth Hall wrote:
> On Aug 9, 2013, at 3:30 PM, aaron gee-clough <lists at g-clef.net> wrote:
>
>> Is there a way to disable this caching? (or have I mis-understood what
>> bro's doing with DNS?)
>
> That's unrelated.  It's referring to DNS lookup requests happening at script land.  We ran into a case once where someone had written a script that did two reverse hostname lookups for every connection that was established (don't do this, it's *really* not a good idea).  Although I should point out that their Bro cluster was running quite well even in the face of that, but I don't think their DNS resolver was very happy about it. :)

Heh. I'll keep that in mind.

> In general, monitoring in front of a DNS resolver should be just fine.
>

Hmm...that leaves me with my original problem, then: I have two vanilla 
securityonion installs (no custom .bro scripts added, just the ones that 
came with securityonion), watching just traffic to two different DNS 
resolvers...right now one of the worker parent processes (according to 
"broctl top") on each securityonion box grows monotonically in RAM usage 
until it gets killed by Linux (and is then restarted by broctl's cron job).

Any ideas on where I should start looking to identify what's causing the 
worker to grow in RAM like that?

Thanks.

aaron


From vladg at cmu.edu  Sun Aug 11 18:02:31 2013
From: vladg at cmu.edu (Vlad Grigorescu)
Date: Mon, 12 Aug 2013 01:02:31 +0000
Subject: [Bro] troubleshooting bro memory usage?
In-Reply-To: <8718_1376224792_r7BCdpfm009844_520785E7.9000902@g-clef.net>
References: <51FBFB5F.9090507@g-clef.net> <52054368.7030001@g-clef.net>
	<298E63A0-6C1C-4580-A6FA-87822422E165@icir.org>
	<8718_1376224792_r7BCdpfm009844_520785E7.9000902@g-clef.net>
Message-ID: <1202BE242E080642B0CD0AD0A03E8552D80F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>


On Aug 11, 2013, at 8:39 AM, Aaron Gee-Clough <lists at g-clef.net> wrote:

> I have two vanilla 
> securityonion installs (no custom .bro scripts added, just the ones that 
> came with securityonion), watching just traffic to two different DNS 
> resolvers

What traffic rate do you see?

> right now one of the worker parent processes (according to 
> "broctl top") on each securityonion box grows monotonically in RAM usage 
> until it gets killed by Linux (and is then restarted by broctl's cron job).

How much RAM is in the box?

  --Vlad



From punchpernickle at gmail.com  Mon Aug 12 05:54:54 2013
From: punchpernickle at gmail.com (Dani Witherspoon)
Date: Mon, 12 Aug 2013 08:54:54 -0400
Subject: [Bro] Implementing Brownian
Message-ID: <CAO266SG1qh9zp0TLTUpxqtK50DJCwN7SktjncDrWzJ_xj4McEA@mail.gmail.com>

Hi all,

I'm trying to implement Brownian (
https://github.com/grigorescu/Brownian), but I've run into an issue: I
followed the directions just as written,
but when I get to the point of configuration and try to navigate to the
settings file at path
"Brownian/lib/python2.X/site-packages/Brownian/settings.py", I can only get
as far as the "site-packages" folder, which is empty.

Anybody else had this issue or know a fix?

Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130812/ba49df09/attachment.html 

From lists at g-clef.net  Mon Aug 12 05:55:17 2013
From: lists at g-clef.net (aaron gee-clough)
Date: Mon, 12 Aug 2013 08:55:17 -0400
Subject: [Bro] troubleshooting bro memory usage?
References: <51FBFB5F.9090507@g-clef.net> <52054368.7030001@g-clef.net>
	<298E63A0-6C1C-4580-A6FA-87822422E165@icir.org>
	<8718_1376224792_r7BCdpfm009844_520785E7.9000902@g-clef.net>
	<1202BE242E080642B0CD0AD0A03E8552D80F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
Message-ID: <5208DB35.5010000@g-clef.net>


On 08/11/2013 09:02 PM, Vlad Grigorescu wrote:
> On Aug 11, 2013, at 8:39 AM, Aaron Gee-Clough <lists at g-clef.net> wrote:
>
>> I have two vanilla
>> securityonion installs (no custom .bro scripts added, just the ones that
>> came with securityonion), watching just traffic to two different DNS
>> resolvers
> What traffic rate do you see?

95th percentile over a week (according to MRTG): Box 1: 34.6 Mbps. Box 
2: 28Mbps

>
>> right now one of the worker parent processes (according to
>> "broctl top") on each securityonion box grows monotonically in RAM usage
>> until it gets killed by Linux (and is then restarted by broctl's cron job).
> How much RAM is in the box?
>
16 GB. Both have 6-core 2.2GHz CPUs, also.

Thanks.

aaron


From carlopmart at gmail.com  Tue Aug 13 02:55:51 2013
From: carlopmart at gmail.com (C. L. Martinez)
Date: Tue, 13 Aug 2013 09:55:51 +0000
Subject: [Bro] Problems adding http ports to bro (git version)
Message-ID: <CAEjQA5JeFakU2dYS85Ho7K-JC+VuBDKb8iW1Vq_awgUAWjXViQ@mail.gmail.com>

HI all,

 I have installed Bro from git to try new features (release 2.1-1052).
I need to detect http conns in non standard ports like 80. To
accomplish this I have created the following policy:

# New DPD configuration.
const ports = {
        80/tcp, 81/tcp, 82/tcp, 631/tcp, 1080/tcp, 1090/tcp, 3128/tcp,
3200/tcp, 3210/tcp, 3300/tcp, 3310/tcp, 3333/tcp, 3600/tcp, 3610/tcp,
        8000/tcp, 8080/tcp, 8100/tcp, 8888/tcp, 50000/tcp, 50001/tcp,
50002/tcp, 50003/tcp, 50004/tcp, 50005/tcp, 50006/tcp, 50007/tcp,
50008/tcp,
        50009/tcp, 50010/tcp, 51000/tcp, 51001/tcp, 51002/tcp,
51003/tcp, 51004/tcp, 51005/tcp, 51006/tcp, 51007/tcp, 51008/tcp,
51009/tcp, 51010/tcp,
};

redef dpd_config += {
        [[ANALYZER_HTTP, ANALYZER_HTTP_BINPAC]] = [$ports = ports],
};

redef capture_filters +=  {
        ["http"] = "tcp and port (80 or 81 or 82 or 631 or 1080 or
1090 or 3128 or 3200 or 3210 or 3300 or 3310 or 3333 or 3600 or 3610
or 8000 or 8080 or 8100 or 8888 or 50000 or 50001 or 50002 or 50003
 or 50004 or 50005 or 50006 or 50007 or 50008 or 50009 or 50010 or
51001 or 51002 or 51003 or 51004 or 51005 or 51006 or 51007 or 51008
or 51009 or 51010)"
};

redef likely_server_ports += { 82/tcp };
redef likely_server_ports += { 1090/tcp };
redef likely_server_ports += { 3200/tcp };
redef likely_server_ports += { 3210/tcp };
redef likely_server_ports += { 3300/tcp };
redef likely_server_ports += { 3310/tcp };
redef likely_server_ports += { 3333/tcp };
redef likely_server_ports += { 3600/tcp };
redef likely_server_ports += { 3610/tcp };
redef likely_server_ports += { 8100/tcp };
redef likely_server_ports += { 50001/tcp };
redef likely_server_ports += { 50002/tcp };
redef likely_server_ports += { 50003/tcp };
redef likely_server_ports += { 50004/tcp };
redef likely_server_ports += { 50005/tcp };
redef likely_server_ports += { 50006/tcp };
redef likely_server_ports += { 50007/tcp };
redef likely_server_ports += { 50008/tcp };
redef likely_server_ports += { 50009/tcp };
redef likely_server_ports += { 50010/tcp };
redef likely_server_ports += { 51000/tcp };
redef likely_server_ports += { 51001/tcp };
redef likely_server_ports += { 51002/tcp };
redef likely_server_ports += { 51003/tcp };
redef likely_server_ports += { 51004/tcp };
redef likely_server_ports += { 51005/tcp };
redef likely_server_ports += { 51006/tcp };
redef likely_server_ports += { 51007/tcp };
redef likely_server_ports += { 51008/tcp };
redef likely_server_ports += { 51009/tcp };
redef likely_server_ports += { 51010/tcp };

But it doesn't works. Error is:

bro failed.
   error in /opt/bro/share/bro/site/more-http-ports.bro, line 13:
unknown identifier ANALYZER_HTTP, at or near "ANALYZER_HTTP"

Same policy works for release 2.1.

Any idea??

Thanks.


From srunnels at gmail.com  Tue Aug 13 03:37:53 2013
From: srunnels at gmail.com (Scott Runnels)
Date: Tue, 13 Aug 2013 06:37:53 -0400
Subject: [Bro] Problems adding http ports to bro (git version)
In-Reply-To: <CAEjQA5JeFakU2dYS85Ho7K-JC+VuBDKb8iW1Vq_awgUAWjXViQ@mail.gmail.com>
References: <CAEjQA5JeFakU2dYS85Ho7K-JC+VuBDKb8iW1Vq_awgUAWjXViQ@mail.gmail.com>
Message-ID: <CANYzqZ6uxL_JkxfRMLkKV41qWc0VbUOXJiQuTzcpRuE6BiXqHA@mail.gmail.com>

Do you get a different result if you remove the trailing comma from "
51010/tcp,};" in the ports constant?


On Tue, Aug 13, 2013 at 5:55 AM, C. L. Martinez <carlopmart at gmail.com>wrote:

> redef dpd_config += {
>         [[ANALYZER_HTTP, ANALYZER_HTTP_BINPAC]] = [$ports = ports],
> };
>



Scott Runnels
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130813/a4d58f2b/attachment.html 

From carlopmart at gmail.com  Tue Aug 13 03:47:12 2013
From: carlopmart at gmail.com (C. L. Martinez)
Date: Tue, 13 Aug 2013 10:47:12 +0000
Subject: [Bro] Problems adding http ports to bro (git version)
In-Reply-To: <CANYzqZ6uxL_JkxfRMLkKV41qWc0VbUOXJiQuTzcpRuE6BiXqHA@mail.gmail.com>
References: <CAEjQA5JeFakU2dYS85Ho7K-JC+VuBDKb8iW1Vq_awgUAWjXViQ@mail.gmail.com>
	<CANYzqZ6uxL_JkxfRMLkKV41qWc0VbUOXJiQuTzcpRuE6BiXqHA@mail.gmail.com>
Message-ID: <CAEjQA5J4bcBAadWeC0Ks5BUXJ+ke67pQq9p9FLVZR4PfCG05Hw@mail.gmail.com>

Nop, same result.

On Tue, Aug 13, 2013 at 10:37 AM, Scott Runnels <srunnels at gmail.com> wrote:
> Do you get a different result if you remove the trailing comma from
> "51010/tcp,};" in the ports constant?
>
>
> On Tue, Aug 13, 2013 at 5:55 AM, C. L. Martinez <carlopmart at gmail.com>
> wrote:
>>
>> redef dpd_config += {
>>         [[ANALYZER_HTTP, ANALYZER_HTTP_BINPAC]] = [$ports = ports],
>> };
>
>
>
>
> Scott Runnels
>


From vladg at cmu.edu  Tue Aug 13 04:18:47 2013
From: vladg at cmu.edu (Vlad Grigorescu)
Date: Tue, 13 Aug 2013 11:18:47 +0000
Subject: [Bro] Problems adding http ports to bro (git version)
In-Reply-To: <20083_1376391494_r7DAwCeI002718_CAEjQA5J4bcBAadWeC0Ks5BUXJ+ke67pQq9p9FLVZR4PfCG05Hw@mail.gmail.com>
References: <CAEjQA5JeFakU2dYS85Ho7K-JC+VuBDKb8iW1Vq_awgUAWjXViQ@mail.gmail.com>
	<CANYzqZ6uxL_JkxfRMLkKV41qWc0VbUOXJiQuTzcpRuE6BiXqHA@mail.gmail.com>
	<20083_1376391494_r7DAwCeI002718_CAEjQA5J4bcBAadWeC0Ks5BUXJ+ke67pQq9p9FLVZR4PfCG05Hw@mail.gmail.com>
Message-ID: <1202BE242E080642B0CD0AD0A03E8552D85AC7@PGH-MSGMB-03.andrew.ad.cmu.edu>

Let's back up a bit. Bro uses signatures to detect protocols on non-standard ports, and it should also be able to identify the server and the client.

Out of the box, Bro should be able to automatically detect HTTP on all ports for you. If that's not working, that means that there's a problem with either how you're running Bro, or that there's a bug in Bro.

How are you running Bro? What does the conn.log line look like for an HTTP connection on a non-standard port that Bro failed to detect? Do you have a PCAP of such traffic that you could share (anonymized is fine)?

Having said all that, to answer your original question: The way you specify these ports for DPD changed in 2.2. If you take a look at base/protocols/http/main.bro:

 126 const ports = {
 127         80/tcp, 81/tcp, 631/tcp, 1080/tcp, 3128/tcp,
 128         8000/tcp, 8080/tcp, 8888/tcp,
 129 };
 130 redef likely_server_ports += { ports };
 131 
 132 # Initialize the HTTP logging stream and ports.
 133 event bro_init() &priority=5
 134         {
 135         Log::create_stream(HTTP::LOG, [$columns=Info, $ev=log_http]);
 136         Analyzer::register_for_ports(Analyzer::ANALYZER_HTTP, ports);
 137         }

   --Vlad

On Aug 13, 2013, at 6:47 AM, C. L. Martinez <carlopmart at gmail.com> wrote:

> Nop, same result.
> 
> On Tue, Aug 13, 2013 at 10:37 AM, Scott Runnels <srunnels at gmail.com> wrote:
>> Do you get a different result if you remove the trailing comma from
>> "51010/tcp,};" in the ports constant?
>> 
>> 
>> On Tue, Aug 13, 2013 at 5:55 AM, C. L. Martinez <carlopmart at gmail.com>
>> wrote:
>>> 
>>> redef dpd_config += {
>>>        [[ANALYZER_HTTP, ANALYZER_HTTP_BINPAC]] = [$ports = ports],
>>> };
>> 
>> 
>> 
>> 
>> Scott Runnels
>> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From carlopmart at gmail.com  Tue Aug 13 04:24:58 2013
From: carlopmart at gmail.com (C. L. Martinez)
Date: Tue, 13 Aug 2013 11:24:58 +0000
Subject: [Bro] Problems adding http ports to bro (git version)
In-Reply-To: <1202BE242E080642B0CD0AD0A03E8552D85AC7@PGH-MSGMB-03.andrew.ad.cmu.edu>
References: <CAEjQA5JeFakU2dYS85Ho7K-JC+VuBDKb8iW1Vq_awgUAWjXViQ@mail.gmail.com>
	<CANYzqZ6uxL_JkxfRMLkKV41qWc0VbUOXJiQuTzcpRuE6BiXqHA@mail.gmail.com>
	<20083_1376391494_r7DAwCeI002718_CAEjQA5J4bcBAadWeC0Ks5BUXJ+ke67pQq9p9FLVZR4PfCG05Hw@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E8552D85AC7@PGH-MSGMB-03.andrew.ad.cmu.edu>
Message-ID: <CAEjQA5Kb8=3uazj73di-wddPbLgGNC6e_gieDrF5X+MV8sJKzA@mail.gmail.com>

Uhmm ... well, I don't know if bro can detect http requests on
non-standard ports. I have not yet been able to start it :)).

I will try it and if these http ports are not detected and I will open
a new thread ....

Many thanks Vlad.


On Tue, Aug 13, 2013 at 11:18 AM, Vlad Grigorescu <vladg at cmu.edu> wrote:
> Let's back up a bit. Bro uses signatures to detect protocols on non-standard ports, and it should also be able to identify the server and the client.
>
> Out of the box, Bro should be able to automatically detect HTTP on all ports for you. If that's not working, that means that there's a problem with either how you're running Bro, or that there's a bug in Bro.
>
> How are you running Bro? What does the conn.log line look like for an HTTP connection on a non-standard port that Bro failed to detect? Do you have a PCAP of such traffic that you could share (anonymized is fine)?
>
> Having said all that, to answer your original question: The way you specify these ports for DPD changed in 2.2. If you take a look at base/protocols/http/main.bro:
>
>  126 const ports = {
>  127         80/tcp, 81/tcp, 631/tcp, 1080/tcp, 3128/tcp,
>  128         8000/tcp, 8080/tcp, 8888/tcp,
>  129 };
>  130 redef likely_server_ports += { ports };
>  131
>  132 # Initialize the HTTP logging stream and ports.
>  133 event bro_init() &priority=5
>  134         {
>  135         Log::create_stream(HTTP::LOG, [$columns=Info, $ev=log_http]);
>  136         Analyzer::register_for_ports(Analyzer::ANALYZER_HTTP, ports);
>  137         }
>
>    --Vlad
>
> On Aug 13, 2013, at 6:47 AM, C. L. Martinez <carlopmart at gmail.com> wrote:
>
>> Nop, same result.
>>
>> On Tue, Aug 13, 2013 at 10:37 AM, Scott Runnels <srunnels at gmail.com> wrote:
>>> Do you get a different result if you remove the trailing comma from
>>> "51010/tcp,};" in the ports constant?
>>>
>>>
>>> On Tue, Aug 13, 2013 at 5:55 AM, C. L. Martinez <carlopmart at gmail.com>
>>> wrote:
>>>>
>>>> redef dpd_config += {
>>>>        [[ANALYZER_HTTP, ANALYZER_HTTP_BINPAC]] = [$ports = ports],
>>>> };
>>>
>>>
>>>
>>>
>>> Scott Runnels
>>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>


From lists at g-clef.net  Tue Aug 13 07:27:05 2013
From: lists at g-clef.net (aaron gee-clough)
Date: Tue, 13 Aug 2013 10:27:05 -0400
Subject: [Bro] troubleshooting bro memory usage?
References: <51FBFB5F.9090507@g-clef.net> <52054368.7030001@g-clef.net>
	<298E63A0-6C1C-4580-A6FA-87822422E165@icir.org>
	<8718_1376224792_r7BCdpfm009844_520785E7.9000902@g-clef.net>
	<1202BE242E080642B0CD0AD0A03E8552D80F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<5208DB35.5010000@g-clef.net>
Message-ID: <520A4239.1050105@g-clef.net>

All,

I think I know what's causing this on the surface, but I'm unsure of the 
deeper cause. When I commented out the SecurityOnion bro scripts, bro's 
memory usage was stable and reasonable. So the problem was clearly 
coming from securityonion's scripts. I then started adding the 
SecurityOnion rules back in one by one, adding a ton of Reporter::warn 
statements, and watching the reporter.log. What I noticed was the 
securityonion hostname.bro script never completed *if* the device's 
hostname had a dash in it ("location-onion", for example). When I 
changed the server's hostname to not have a dash, the hostname script 
completed without issue.

I suspect this means that the "hostname" and "interface" variables from 
the securityonion scripts weren't being initialized properly while 
trying to start up with a dashed hostname, doing who-knows-what when bro 
was told to add those variables to every logged event.

Given that, I have an easy fix in the short term, which is to rename the 
box running securityonion to not have a dash in its hostname. What I'm 
confused by is why this would happen in the first place. (So I'm not 
clear yet on what patch to suggest to the securityonion folks to prevent 
this from coming up again.)

The securityonion hostname.bro file does the following:

    module SecurityOnion;

    @load base/frameworks/input

    export {
             ## Event to capture when the hostname is discovered.
             global SecurityOnion::found_hostname: event(hostname: string);

             ## Hostname for this box.
             global hostname = "";

    type HostnameCmdLine: record { s: string; };

    event SecurityOnion::hostname_line(description:
    Input::EventDescription, tpe: Input::Event, s: string)
             {
             hostname = s;
             system(fmt("rm %s", description$source));
             event SecurityOnion::found_hostname(hostname);
             }



    event add_hostname_reader(name: string)
             {
             Input::add_event([$source=name,
                               $name=name,
                               $reader=Input::READER_RAW,
                               $want_record=F,
                               $fields=HostnameCmdLine,
                               $ev=SecurityOnion::hostname_line]);
             }

    event bro_init() &priority=5
             {
             local tmpfile = "/tmp/bro-hostname-" + unique_id("");
             system(fmt("hostname > %s", tmpfile));
             event add_hostname_reader(tmpfile);
             }


The SecurityOnion::hostname_line event never fires if the hostname has a 
dash in it (for example, if the contents of the tmpfile are 
"location-onion"). I see the add_hostname_reader event fire, but not the 
hostname_line event. Do you all have any idea why that would fail if 
there's a string with a dash in the file? Is bro thinking it's an 
expression rather than a string? Two strings?

Thanks for all the help so far. This has been hard to nail down.

aaron


On 08/12/2013 08:55 AM, aaron gee-clough wrote:
>
> On 08/11/2013 09:02 PM, Vlad Grigorescu wrote:
>> On Aug 11, 2013, at 8:39 AM, Aaron Gee-Clough <lists at g-clef.net> wrote:
>>
>>> I have two vanilla
>>> securityonion installs (no custom .bro scripts added, just the ones that
>>> came with securityonion), watching just traffic to two different DNS
>>> resolvers
>> What traffic rate do you see?
> 95th percentile over a week (according to MRTG): Box 1: 34.6 Mbps. Box
> 2: 28Mbps
>
>>> right now one of the worker parent processes (according to
>>> "broctl top") on each securityonion box grows monotonically in RAM usage
>>> until it gets killed by Linux (and is then restarted by broctl's cron job).
>> How much RAM is in the box?
>>
> 16 GB. Both have 6-core 2.2GHz CPUs, also.
>
> Thanks.
>
> aaron
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130813/f6cfd3b9/attachment.html 

From doug.burks at gmail.com  Tue Aug 13 07:53:38 2013
From: doug.burks at gmail.com (Doug Burks)
Date: Tue, 13 Aug 2013 10:53:38 -0400
Subject: [Bro] troubleshooting bro memory usage?
In-Reply-To: <520A4239.1050105@g-clef.net>
References: <51FBFB5F.9090507@g-clef.net> <52054368.7030001@g-clef.net>
	<298E63A0-6C1C-4580-A6FA-87822422E165@icir.org>
	<8718_1376224792_r7BCdpfm009844_520785E7.9000902@g-clef.net>
	<1202BE242E080642B0CD0AD0A03E8552D80F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<5208DB35.5010000@g-clef.net> <520A4239.1050105@g-clef.net>
Message-ID: <CAK8kjrAFQjAg2BL3pbEGC2i5jY3t6Gt5yQbPF_tpHcUp=8dJNw@mail.gmail.com>

Hi Aaron,

There are definitely some issues with the hostname and interface scripts.

My demo at Bro Exchange last week failed due to the hostname script,
even though I put precautions in place which had always worked in the
past.  My hostname did include a hyphen, but I recorded a video later
with the same VM (and same hostname) and everything worked fine:
http://youtu.be/0a2WDyBsxzk?t=2m36s

I'll also mention that all of my production servers have a hyphen in
the hostname and they work fine.

Another thing I noticed in testing a few weeks ago in a VM was that if
the VM had only a single CPU core the scripts were much more likely to
fail.  Increasing to 2 or more CPU cores resulted in much higher
levels of success.  Perhaps resource contention on Bro startup?

Seth, I know you're going to rewrite these scripts for Bro 2.2, but do
you have any ideas for troubleshooting in the meantime?

Thanks!

Doug



On Tue, Aug 13, 2013 at 10:27 AM, aaron gee-clough <lists at g-clef.net> wrote:
> All,
>
> I think I know what's causing this on the surface, but I'm unsure of the
> deeper cause. When I commented out the SecurityOnion bro scripts, bro's
> memory usage was stable and reasonable. So the problem was clearly coming
> from securityonion's scripts. I then started adding the SecurityOnion rules
> back in one by one, adding a ton of Reporter::warn statements, and watching
> the reporter.log. What I noticed was the securityonion hostname.bro script
> never completed *if* the device's hostname had a dash in it
> ("location-onion", for example). When I changed the server's hostname to not
> have a dash, the hostname script completed without issue.
>
> I suspect this means that the "hostname" and "interface" variables from the
> securityonion scripts weren't being initialized properly while trying to
> start up with a dashed hostname, doing who-knows-what when bro was told to
> add those variables to every logged event.
>
> Given that, I have an easy fix in the short term, which is to rename the box
> running securityonion to not have a dash in its hostname. What I'm confused
> by is why this would happen in the first place. (So I'm not clear yet on
> what patch to suggest to the securityonion folks to prevent this from coming
> up again.)
>
> The securityonion hostname.bro file does the following:
>
> module SecurityOnion;
>
> @load base/frameworks/input
>
> export {
>         ## Event to capture when the hostname is discovered.
>         global SecurityOnion::found_hostname: event(hostname: string);
>
>         ## Hostname for this box.
>         global hostname = "";
>
> type HostnameCmdLine: record { s: string; };
>
> event SecurityOnion::hostname_line(description: Input::EventDescription,
> tpe: Input::Event, s: string)
>         {
>         hostname = s;
>         system(fmt("rm %s", description$source));
>         event SecurityOnion::found_hostname(hostname);
>         }
>
>
>
> event add_hostname_reader(name: string)
>         {
>         Input::add_event([$source=name,
>                           $name=name,
>                           $reader=Input::READER_RAW,
>                           $want_record=F,
>                           $fields=HostnameCmdLine,
>                           $ev=SecurityOnion::hostname_line]);
>         }
>
> event bro_init() &priority=5
>         {
>         local tmpfile = "/tmp/bro-hostname-" + unique_id("");
>         system(fmt("hostname > %s", tmpfile));
>         event add_hostname_reader(tmpfile);
>         }
>
>
> The SecurityOnion::hostname_line event never fires if the hostname has a
> dash in it (for example, if the contents of the tmpfile are
> "location-onion"). I see the add_hostname_reader event fire, but not the
> hostname_line event. Do you all have any idea why that would fail if there's
> a string with a dash in the file? Is bro thinking it's an expression rather
> than a string? Two strings?
>
> Thanks for all the help so far. This has been hard to nail down.
>
> aaron
>
>
>
> On 08/12/2013 08:55 AM, aaron gee-clough wrote:
>
> On 08/11/2013 09:02 PM, Vlad Grigorescu wrote:
>
> On Aug 11, 2013, at 8:39 AM, Aaron Gee-Clough <lists at g-clef.net> wrote:
>
> I have two vanilla
> securityonion installs (no custom .bro scripts added, just the ones that
> came with securityonion), watching just traffic to two different DNS
> resolvers
>
> What traffic rate do you see?
>
> 95th percentile over a week (according to MRTG): Box 1: 34.6 Mbps. Box
> 2: 28Mbps
>
> right now one of the worker parent processes (according to
> "broctl top") on each securityonion box grows monotonically in RAM usage
> until it gets killed by Linux (and is then restarted by broctl's cron job).
>
> How much RAM is in the box?
>
> 16 GB. Both have 6-core 2.2GHz CPUs, also.
>
> Thanks.
>
> aaron
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
Doug Burks
http://securityonion.blogspot.com


From robin at icir.org  Tue Aug 13 08:15:57 2013
From: robin at icir.org (Robin Sommer)
Date: Tue, 13 Aug 2013 08:15:57 -0700
Subject: [Bro] troubleshooting bro memory usage?
In-Reply-To: <520A4239.1050105@g-clef.net>
References: <51FBFB5F.9090507@g-clef.net> <52054368.7030001@g-clef.net>
	<298E63A0-6C1C-4580-A6FA-87822422E165@icir.org>
	<8718_1376224792_r7BCdpfm009844_520785E7.9000902@g-clef.net>
	<1202BE242E080642B0CD0AD0A03E8552D80F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<5208DB35.5010000@g-clef.net> <520A4239.1050105@g-clef.net>
Message-ID: <20130813151557.GD84947@icir.org>



On Tue, Aug 13, 2013 at 10:27 -0400, aaron gee-clough wrote:

> coming from securityonion's scripts. I then started adding the  
> SecurityOnion rules back in one by one, adding a ton of Reporter::warn  
> statements, and watching the reporter.log.

Can you send a sample of those message? How much is a ton? :)

There's a known memory leak in Bro when the script interpreter reports
certain errors in script code. If this happens very often, it could
explain what you're seeing (unfortunately the leak is hard to fix, but
the messages usually indicate a problem in the corresponding script in
the first place).

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 *     robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 * www.icir.org/robin


From robin at icir.org  Tue Aug 13 08:22:37 2013
From: robin at icir.org (Robin Sommer)
Date: Tue, 13 Aug 2013 08:22:37 -0700
Subject: [Bro] Encrypting bro logs before storing to disk
In-Reply-To: <B3E27C90B7D2DD4CAEF09215A29EB0D117086306@CITESMBX2.ad.uillinois.edu>
References: <CAJNbMrzEXRRCvTrb=_erR-hV0KL_ZeF-7PmD87BddaPs9+o=zQ@mail.gmail.com>
	<B3E27C90B7D2DD4CAEF09215A29EB0D117086306@CITESMBX2.ad.uillinois.edu>
Message-ID: <20130813152237.GE84947@icir.org>



On Thu, Aug 08, 2013 at 20:34 +0000, Jonathan Siwek wrote:

> the C++ land) sounds like it would work.  And if the encryption
> behavior were made toggle-able (possibly via some script-land
> variables that could be set/redef'd), that would make a patch to do
> such a thing more acceptable.

Still wanted to chime in here: having that as an option would indeed
be quite nice.

Actually Bro used to have that functionality, and we still have
left-overs from that in the code, e.g., in scripts/base/init-bare.bro

    ## Deprecated.
    const log_encryption_key = "<undefined>" &redef;

I believe even the encryption code itself is still in there, but
afaict it hasn't been exercised in a while and it's kind of useless
now that we have the new logging system which does things differently
internally. 

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 *     robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 * www.icir.org/robin


From jsiwek at illinois.edu  Tue Aug 13 08:26:28 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Tue, 13 Aug 2013 15:26:28 +0000
Subject: [Bro] troubleshooting bro memory usage?
In-Reply-To: <520A4239.1050105@g-clef.net>
References: <51FBFB5F.9090507@g-clef.net> <52054368.7030001@g-clef.net>
	<298E63A0-6C1C-4580-A6FA-87822422E165@icir.org>
	<8718_1376224792_r7BCdpfm009844_520785E7.9000902@g-clef.net>
	<1202BE242E080642B0CD0AD0A03E8552D80F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<5208DB35.5010000@g-clef.net> <520A4239.1050105@g-clef.net>
Message-ID: <B3E27C90B7D2DD4CAEF09215A29EB0D117095A35@CITESMBX2.ad.uillinois.edu>


On Aug 13, 2013, at 9:27 AM, aaron gee-clough <lists at g-clef.net> wrote:

> The SecurityOnion::hostname_line event never fires if the hostname has a dash in it (for example, if the contents of the tmpfile are "location-onion"). I see the add_hostname_reader event fire, but not the hostname_line event. Do you all have any idea why that would fail if there's a string with a dash in the file? Is bro thinking it's an expression rather than a string? Two strings?

The hyphen-in-hostname might be a red herring when at least part of the issue is there's a bit of a race condition in the script -- the system() call to invoke `hostname` and put the output in a temporary file happens in a different background process, subject to the OS scheduler.  So if that process gets scheduled after the input reader has already tried and failed to open the temporary file, the input reader won't automatically recover from that.

I put a revision to the script you showed at [1] that *should* be a way to perform the same function without a race condition (though at the moment I'm not confident that the internals of the raw input reader are race-free in all cases, I'm looking in to some things).

Still, I don't really know if this was actually the cause of your memory issues.

- Jon

[1] https://gist.github.com/jsiwek/6222106


From lists at g-clef.net  Tue Aug 13 08:48:50 2013
From: lists at g-clef.net (aaron gee-clough)
Date: Tue, 13 Aug 2013 11:48:50 -0400
Subject: [Bro] troubleshooting bro memory usage?
References: <51FBFB5F.9090507@g-clef.net> <52054368.7030001@g-clef.net>
	<298E63A0-6C1C-4580-A6FA-87822422E165@icir.org>
	<8718_1376224792_r7BCdpfm009844_520785E7.9000902@g-clef.net>
	<1202BE242E080642B0CD0AD0A03E8552D80F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<5208DB35.5010000@g-clef.net> <520A4239.1050105@g-clef.net>
	<20130813151557.GD84947@icir.org>
Message-ID: <520A5562.7040204@g-clef.net>

I *added* a ton of Reporter::warn messages. Before this, bro was issuing 
one interesting error (see below), but I was basically adding lines like 
"script <x> started with variables <y>", "script <x> finished", etc to 
the reporter.log.

So, the log messages looked like:

    0.000000    Reporter::WARNING    making tempfile:
    /tmp/bro-hostname-ndOXgWQ3v52
    /opt/bro/share/bro/securityonion/./hostname.bro, line 40
    0.000000    Reporter::WARNING    wrote hostname to tempfile
    /opt/bro/share/bro/securityonion/./hostname.bro, line 42
    0.000000    Reporter::WARNING    called event to add hostname
    reader    /opt/bro/share/bro/securityonion/./hostname.bro, line 44
    0.000000    Reporter::WARNING    hostname reader starting on file:
    /tmp/bro-hostname-ndOXgWQ3v52
    /opt/bro/share/bro/securityonion/./hostname.bro, line 28
    1376401730.326379    Reporter::INFO    processing suspended (empty)
    1376401730.326379    Reporter::INFO    processing continued (empty)
    1376401730.370328    Reporter::INFO    processing continued (empty)



What got me going this way was an error earlier that was:

    0.000000    Reporter::WARNING    Template value remaining in BPFConf
    filename: /etc/nsm/{{hostname}}-{{interface}}/bpf-bro.conf
    /opt/bro/share/bro/securityonion/./bpfconf.bro, line 99


which said to me that either the "hostname" or "interface" variable 
hadn't been initialized in the bro setup.

aaron

On 08/13/2013 11:15 AM, Robin Sommer wrote:
>
>
> On Tue, Aug 13, 2013 at 10:27 -0400, aaron gee-clough wrote:
>
>> coming from securityonion's scripts. I then started adding the
>> SecurityOnion rules back in one by one, adding a ton of Reporter::warn
>> statements, and watching the reporter.log.
> Can you send a sample of those message? How much is a ton? :)
>
> There's a known memory leak in Bro when the script interpreter reports
> certain errors in script code. If this happens very often, it could
> explain what you're seeing (unfortunately the leak is hard to fix, but
> the messages usually indicate a problem in the corresponding script in
> the first place).
>
> Robin
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130813/f29f0e85/attachment.html 

From seth at icir.org  Tue Aug 13 09:06:33 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 13 Aug 2013 12:06:33 -0400
Subject: [Bro] Encrypting bro logs before storing to disk
In-Reply-To: <20130813152237.GE84947@icir.org>
References: <CAJNbMrzEXRRCvTrb=_erR-hV0KL_ZeF-7PmD87BddaPs9+o=zQ@mail.gmail.com>
	<B3E27C90B7D2DD4CAEF09215A29EB0D117086306@CITESMBX2.ad.uillinois.edu>
	<20130813152237.GE84947@icir.org>
Message-ID: <3D40C90E-F683-4A76-899A-6C21AAA3A632@icir.org>

On Aug 13, 2013, at 11:22 AM, Robin Sommer <robin at icir.org> wrote:

> I believe even the encryption code itself is still in there, but
> afaict it hasn't been exercised in a while and it's kind of useless
> now that we have the new logging system which does things differently
> internally. 


I've been waiting for *just* the right moment to either implement encryption in the logging framework or get someone else to do it.  I'd like to approach it in a way where you could either encrypt entire logs, specific lines, or even individual fields.  We'd then just have to have the tooling on the log processing side that can understand this encryption and decrypt it.

Generally though, I think it's fine to remove the log encryption stuff from files and move it all over to the logging framework.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130813/5d1a0ecb/attachment.bin 

From seth at icir.org  Tue Aug 13 09:14:23 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 13 Aug 2013 12:14:23 -0400
Subject: [Bro] troubleshooting bro memory usage?
In-Reply-To: <CAK8kjrAFQjAg2BL3pbEGC2i5jY3t6Gt5yQbPF_tpHcUp=8dJNw@mail.gmail.com>
References: <51FBFB5F.9090507@g-clef.net> <52054368.7030001@g-clef.net>
	<298E63A0-6C1C-4580-A6FA-87822422E165@icir.org>
	<8718_1376224792_r7BCdpfm009844_520785E7.9000902@g-clef.net>
	<1202BE242E080642B0CD0AD0A03E8552D80F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<5208DB35.5010000@g-clef.net> <520A4239.1050105@g-clef.net>
	<CAK8kjrAFQjAg2BL3pbEGC2i5jY3t6Gt5yQbPF_tpHcUp=8dJNw@mail.gmail.com>
Message-ID: <31E7B9F7-2396-4E42-8DDF-36CF10D629AD@icir.org>

On Aug 13, 2013, at 10:53 AM, Doug Burks <doug.burks at gmail.com> wrote:

> Seth, I know you're going to rewrite these scripts for Bro 2.2, but do
> you have any ideas for troubleshooting in the meantime?


I just looked through the scripts and I really don't know why that would happen.  If anything, my guess is that Jon's probably right and there is a race condition that is causing it to fail in unpredictable ways.  I'll start updating the scripts in that repository for 2.2 soon which might help a little.  If I just update those in the master branch, could that cause any problems for SO?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130813/5434bd1c/attachment.bin 

From doug.burks at gmail.com  Tue Aug 13 10:38:13 2013
From: doug.burks at gmail.com (Doug Burks)
Date: Tue, 13 Aug 2013 13:38:13 -0400
Subject: [Bro] troubleshooting bro memory usage?
In-Reply-To: <B3E27C90B7D2DD4CAEF09215A29EB0D117095A35@CITESMBX2.ad.uillinois.edu>
References: <51FBFB5F.9090507@g-clef.net> <52054368.7030001@g-clef.net>
	<298E63A0-6C1C-4580-A6FA-87822422E165@icir.org>
	<8718_1376224792_r7BCdpfm009844_520785E7.9000902@g-clef.net>
	<1202BE242E080642B0CD0AD0A03E8552D80F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<5208DB35.5010000@g-clef.net> <520A4239.1050105@g-clef.net>
	<B3E27C90B7D2DD4CAEF09215A29EB0D117095A35@CITESMBX2.ad.uillinois.edu>
Message-ID: <CAK8kjrBDXd4MsGQQJjj1oseevCkSGgpqiNLwwCXt9QGK6nFASQ@mail.gmail.com>

Hi Jon,

Thanks for the revised script!  I'll try it out this week and see if
it's more consistent.

Thanks,
Doug

On Tue, Aug 13, 2013 at 11:26 AM, Siwek, Jonathan Luke
<jsiwek at illinois.edu> wrote:
>
> On Aug 13, 2013, at 9:27 AM, aaron gee-clough <lists at g-clef.net> wrote:
>
>> The SecurityOnion::hostname_line event never fires if the hostname has a dash in it (for example, if the contents of the tmpfile are "location-onion"). I see the add_hostname_reader event fire, but not the hostname_line event. Do you all have any idea why that would fail if there's a string with a dash in the file? Is bro thinking it's an expression rather than a string? Two strings?
>
> The hyphen-in-hostname might be a red herring when at least part of the issue is there's a bit of a race condition in the script -- the system() call to invoke `hostname` and put the output in a temporary file happens in a different background process, subject to the OS scheduler.  So if that process gets scheduled after the input reader has already tried and failed to open the temporary file, the input reader won't automatically recover from that.
>
> I put a revision to the script you showed at [1] that *should* be a way to perform the same function without a race condition (though at the moment I'm not confident that the internals of the raw input reader are race-free in all cases, I'm looking in to some things).
>
> Still, I don't really know if this was actually the cause of your memory issues.
>
> - Jon
>
> [1] https://gist.github.com/jsiwek/6222106
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
Doug Burks
http://securityonion.blogspot.com



From doug.burks at gmail.com  Tue Aug 13 10:38:23 2013
From: doug.burks at gmail.com (Doug Burks)
Date: Tue, 13 Aug 2013 13:38:23 -0400
Subject: [Bro] troubleshooting bro memory usage?
In-Reply-To: <31E7B9F7-2396-4E42-8DDF-36CF10D629AD@icir.org>
References: <51FBFB5F.9090507@g-clef.net> <52054368.7030001@g-clef.net>
	<298E63A0-6C1C-4580-A6FA-87822422E165@icir.org>
	<8718_1376224792_r7BCdpfm009844_520785E7.9000902@g-clef.net>
	<1202BE242E080642B0CD0AD0A03E8552D80F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<5208DB35.5010000@g-clef.net> <520A4239.1050105@g-clef.net>
	<CAK8kjrAFQjAg2BL3pbEGC2i5jY3t6Gt5yQbPF_tpHcUp=8dJNw@mail.gmail.com>
	<31E7B9F7-2396-4E42-8DDF-36CF10D629AD@icir.org>
Message-ID: <CAK8kjrC_26HEgYswZdsQP1GVkYmXrvGeXwk4muScsNg3WwPouA@mail.gmail.com>

Updating in the master branch shouldn't cause any problems for SO
since I packaged a static copy of the files and we're not actively
pulling anything from the master branch.

Thanks,
Doug

On Tue, Aug 13, 2013 at 12:14 PM, Seth Hall <seth at icir.org> wrote:
> On Aug 13, 2013, at 10:53 AM, Doug Burks <doug.burks at gmail.com> wrote:
>
>> Seth, I know you're going to rewrite these scripts for Bro 2.2, but do
>> you have any ideas for troubleshooting in the meantime?
>
>
> I just looked through the scripts and I really don't know why that would happen.  If anything, my guess is that Jon's probably right and there is a race condition that is causing it to fail in unpredictable ways.  I'll start updating the scripts in that repository for 2.2 soon which might help a little.  If I just update those in the master branch, could that cause any problems for SO?
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>



-- 
Doug Burks
http://securityonion.blogspot.com



From jbabio at po-box.esu.edu  Tue Aug 13 16:28:11 2013
From: jbabio at po-box.esu.edu (John Babio)
Date: Tue, 13 Aug 2013 23:28:11 +0000
Subject: [Bro] creating bro scripts
Message-ID: <DD2F7A87231C3C449C1F125012B6221E62CAF333@msxmb2.admin.esu.edu>

I wanted to start working on something to get aquainted with the bro programming. I figured DNS might be a good start. It seems to be the way I learn the best and I learned python this way. My goals are maybe create something simple that displays a notice for a particular query type, PTR, NS, MX etc.

Where is there a good example of how I go about this? Inside of policy/protocols/dns ?
Once I create this I can call it from local.bro correct?




From jbabio at po-box.esu.edu  Wed Aug 14 08:07:07 2013
From: jbabio at po-box.esu.edu (John Babio)
Date: Wed, 14 Aug 2013 15:07:07 +0000
Subject: [Bro] creating bro scripts
In-Reply-To: <CAEZw2bxPHrTzT_dZ2zbeavFSS7ZTvbO-o7m6DqJwgS9K4J7MVw@mail.gmail.com>
Message-ID: <CE311555.3FEB%jbabio@po-box.esu.edu>

Thanks Anthony,
Here is what I have so far. How do I create a notice out of it?

event dns_request(c: connection, msg: dns_msg, query: string, qtype: count, qclass: count) &priority=5
        {
        if ( c$dns$qtype == PTR )
                return;
}

From: anthony kasza <anthony.kasza at gmail.com<mailto:anthony.kasza at gmail.com>>
Date: Tuesday, August 13, 2013 7:42 PM
To: John Babio <jbabio at po-box.esu.edu<mailto:jbabio at po-box.esu.edu>>
Subject: Re: [Bro] creating bro scripts


Determine the event you want to act on (sounds like you want dns_request) and write a code block for it. Put that into a file and call it when you run Bro or load the file in the local.bro script.
Check out Liam Randall's fire scripts on github. They print to screen or count when an event occurs.

On Aug 13, 2013 4:32 PM, "John Babio" <jbabio at po-box.esu.edu<mailto:jbabio at po-box.esu.edu>> wrote:
I wanted to start working on something to get aquainted with the bro programming. I figured DNS might be a good start. It seems to be the way I learn the best and I learned python this way. My goals are maybe create something simple that displays a notice for a particular query type, PTR, NS, MX etc.

Where is there a good example of how I go about this? Inside of policy/protocols/dns ?
Once I create this I can call it from local.bro correct?


_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From anthony.kasza at gmail.com  Wed Aug 14 08:20:13 2013
From: anthony.kasza at gmail.com (anthony kasza)
Date: Wed, 14 Aug 2013 08:20:13 -0700
Subject: [Bro] creating bro scripts
In-Reply-To: <CE311555.3FEB%jbabio@po-box.esu.edu>
References: <CAEZw2bxPHrTzT_dZ2zbeavFSS7ZTvbO-o7m6DqJwgS9K4J7MVw@mail.gmail.com>
	<CE311555.3FEB%jbabio@po-box.esu.edu>
Message-ID: <CAEZw2byYssd8aNnNv9Aa1VCXcEM8g7V-9z0Zqj9i3kD=0NMmiw@mail.gmail.com>

See the 'raising notices' section here
http://bro.org/sphinx/notice.html
On Aug 14, 2013 8:07 AM, "John Babio" <jbabio at po-box.esu.edu> wrote:

> Thanks Anthony,
> Here is what I have so far. How do I create a notice out of it?
>
> event dns_request(c: connection, msg: dns_msg, query: string, qtype:
> count, qclass: count) &priority=5
>         {
>         if ( c$dns$qtype == PTR )
>                 return;
> }
>
> From: anthony kasza <anthony.kasza at gmail.com<mailto:
> anthony.kasza at gmail.com>>
> Date: Tuesday, August 13, 2013 7:42 PM
> To: John Babio <jbabio at po-box.esu.edu<mailto:jbabio at po-box.esu.edu>>
> Subject: Re: [Bro] creating bro scripts
>
>
> Determine the event you want to act on (sounds like you want dns_request)
> and write a code block for it. Put that into a file and call it when you
> run Bro or load the file in the local.bro script.
> Check out Liam Randall's fire scripts on github. They print to screen or
> count when an event occurs.
>
> On Aug 13, 2013 4:32 PM, "John Babio" <jbabio at po-box.esu.edu<mailto:
> jbabio at po-box.esu.edu>> wrote:
> I wanted to start working on something to get aquainted with the bro
> programming. I figured DNS might be a good start. It seems to be the way I
> learn the best and I learned python this way. My goals are maybe create
> something simple that displays a notice for a particular query type, PTR,
> NS, MX etc.
>
> Where is there a good example of how I go about this? Inside of
> policy/protocols/dns ?
> Once I create this I can call it from local.bro correct?
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org<mailto:bro at bro-ids.org>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130814/fd00a61b/attachment.html 

From tritium.cat at gmail.com  Wed Aug 14 09:28:18 2013
From: tritium.cat at gmail.com (Tritium Cat)
Date: Wed, 14 Aug 2013 09:28:18 -0700
Subject: [Bro] troubleshooting bro memory usage?
In-Reply-To: <51FBFB5F.9090507@g-clef.net>
References: <51FBFB5F.9090507@g-clef.net>
Message-ID: <CAMPgRd7KZ4g3EhACgZ4PAhTm_W-wAt5tnTOxijDPbyjyjvNMcg@mail.gmail.com>

I've had this problem for too long.  Wish I knew too.  Seems each time it's
brought up on a mailing list the discussion gets hijacked and turns into
feature requests or debates on new concepts and looses sight of the
original problem.

Keep hammering away.  Good luck.


On Fri, Aug 2, 2013 at 11:33 AM, aaron gee-clough <lists at g-clef.net> wrote:

>
> Hello,
>
> I've just put in two sensors running bro (with security onion), and am
> having trouble with the bro processes progressively growing in RAM
> usage, until they crash or become unresponsive. For example, I have one
> bro worker process right now that's reached 2.8 GB in 2 hours while
> watching a < 100MB link. None of the other processes
> (manager/proxy/other workers) are anywhere near that...it's just this
> one worker.
>
> Are there any config options I can enable to attempt to find the cause
> of the memory leak? Also, since I'm confident the link I'm watching is
> missing some traffic (the span it's on is slightly mis-configured at the
> moment), where can I configure protocol timeouts?
>
> Thanks.
>
> aaron
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130814/e736ff81/attachment.html 

From tritium.cat at gmail.com  Wed Aug 14 11:08:22 2013
From: tritium.cat at gmail.com (Tritium Cat)
Date: Wed, 14 Aug 2013 11:08:22 -0700
Subject: [Bro] troubleshooting bro memory usage?
In-Reply-To: <CAMPgRd7KZ4g3EhACgZ4PAhTm_W-wAt5tnTOxijDPbyjyjvNMcg@mail.gmail.com>
References: <51FBFB5F.9090507@g-clef.net>
	<CAMPgRd7KZ4g3EhACgZ4PAhTm_W-wAt5tnTOxijDPbyjyjvNMcg@mail.gmail.com>
Message-ID: <CAMPgRd7QXn7rgwffWBhWgfwm1UAKnr8z9M+Jj1Z7xkktAQwvCA@mail.gmail.com>

Here's a suggestion that has helped me in the past, disable all scripts
except the SSH and SSH brute force detection.  Basically you're using
process of elimination to find what aspect of Bro is not performing well in
your environment.  Turn on features of Bro one by one until you find which
one is the culprit.  It's tricky to debug Bro from site to site because of
different traffic profiles.

--TC



On Wed, Aug 14, 2013 at 9:28 AM, Tritium Cat <tritium.cat at gmail.com> wrote:

> I've had this problem for too long.  Wish I knew too.  Seems each time
> it's brought up on a mailing list the discussion gets hijacked and turns
> into feature requests or debates on new concepts and looses sight of the
> original problem.
>
> Keep hammering away.  Good luck.
>
>
> On Fri, Aug 2, 2013 at 11:33 AM, aaron gee-clough <lists at g-clef.net>wrote:
>
>>
>> Hello,
>>
>> I've just put in two sensors running bro (with security onion), and am
>> having trouble with the bro processes progressively growing in RAM
>> usage, until they crash or become unresponsive. For example, I have one
>> bro worker process right now that's reached 2.8 GB in 2 hours while
>> watching a < 100MB link. None of the other processes
>> (manager/proxy/other workers) are anywhere near that...it's just this
>> one worker.
>>
>> Are there any config options I can enable to attempt to find the cause
>> of the memory leak? Also, since I'm confident the link I'm watching is
>> missing some traffic (the span it's on is slightly mis-configured at the
>> moment), where can I configure protocol timeouts?
>>
>> Thanks.
>>
>> aaron
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130814/a4f30a66/attachment.html 

From punchpernickle at gmail.com  Wed Aug 14 11:09:26 2013
From: punchpernickle at gmail.com (Dani Witherspoon)
Date: Wed, 14 Aug 2013 14:09:26 -0400
Subject: [Bro] Implementing Brownian
In-Reply-To: <CAO266SG1qh9zp0TLTUpxqtK50DJCwN7SktjncDrWzJ_xj4McEA@mail.gmail.com>
References: <CAO266SG1qh9zp0TLTUpxqtK50DJCwN7SktjncDrWzJ_xj4McEA@mail.gmail.com>
Message-ID: <CAO266SHciCRti5+LkGxFYHoLG=sTk0WmyL4SGVgKLmHoKmiB2Q@mail.gmail.com>

Problem was between keyboard and chair. Was looking in the wrong directory.


On Mon, Aug 12, 2013 at 8:54 AM, Dani Witherspoon
<punchpernickle at gmail.com>wrote:

> Hi all,
>
> I'm trying to implement Brownian ( https://github.com/grigorescu/Brownian), but I've run into an issue: I followed the directions just as written,
> but when I get to the point of configuration and try to navigate to the
> settings file at path
> "Brownian/lib/python2.X/site-packages/Brownian/settings.py", I can only get
> as far as the "site-packages" folder, which is empty.
>
> Anybody else had this issue or know a fix?
>
> Thanks.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130814/6daf7e10/attachment.html 

From lists at g-clef.net  Wed Aug 14 13:12:43 2013
From: lists at g-clef.net (aaron gee-clough)
Date: Wed, 14 Aug 2013 16:12:43 -0400
Subject: [Bro] troubleshooting bro memory usage?
References: <51FBFB5F.9090507@g-clef.net>
	<CAMPgRd7KZ4g3EhACgZ4PAhTm_W-wAt5tnTOxijDPbyjyjvNMcg@mail.gmail.com>
	<CAMPgRd7QXn7rgwffWBhWgfwm1UAKnr8z9M+Jj1Z7xkktAQwvCA@mail.gmail.com>
Message-ID: <520BE4BB.1080200@g-clef.net>

Thanks. Of the two boxes I have, one got better when I changed the 
hostname (have no idea why that helped, but it's been stable across 
reboots and restarts since then...perhaps luch). The other one I'm still 
working on.

aaron

On 08/14/2013 02:08 PM, Tritium Cat wrote:
> Here's a suggestion that has helped me in the past, disable all 
> scripts except the SSH and SSH brute force detection.  Basically 
> you're using process of elimination to find what aspect of Bro is not 
> performing well in your environment.  Turn on features of Bro one by 
> one until you find which one is the culprit.  It's tricky to debug Bro 
> from site to site because of different traffic profiles.
>
> --TC
>
>
>
> On Wed, Aug 14, 2013 at 9:28 AM, Tritium Cat <tritium.cat at gmail.com 
> <mailto:tritium.cat at gmail.com>> wrote:
>
>     I've had this problem for too long.  Wish I knew too.  Seems each
>     time it's brought up on a mailing list the discussion gets
>     hijacked and turns into feature requests or debates on new
>     concepts and looses sight of the original problem.
>
>     Keep hammering away.  Good luck.
>
>
>     On Fri, Aug 2, 2013 at 11:33 AM, aaron gee-clough
>     <lists at g-clef.net <mailto:lists at g-clef.net>> wrote:
>
>
>         Hello,
>
>         I've just put in two sensors running bro (with security
>         onion), and am
>         having trouble with the bro processes progressively growing in RAM
>         usage, until they crash or become unresponsive. For example, I
>         have one
>         bro worker process right now that's reached 2.8 GB in 2 hours
>         while
>         watching a < 100MB link. None of the other processes
>         (manager/proxy/other workers) are anywhere near that...it's
>         just this
>         one worker.
>
>         Are there any config options I can enable to attempt to find
>         the cause
>         of the memory leak? Also, since I'm confident the link I'm
>         watching is
>         missing some traffic (the span it's on is slightly
>         mis-configured at the
>         moment), where can I configure protocol timeouts?
>
>         Thanks.
>
>         aaron
>         _______________________________________________
>         Bro mailing list
>         bro at bro-ids.org <mailto:bro at bro-ids.org>
>         http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130814/7a44cb94/attachment.html 

From dkovar at gmail.com  Wed Aug 14 13:27:29 2013
From: dkovar at gmail.com (David Kovar)
Date: Wed, 14 Aug 2013 15:27:29 -0500
Subject: [Bro] troubleshooting bro memory usage?
In-Reply-To: <520BE4BB.1080200@g-clef.net>
References: <51FBFB5F.9090507@g-clef.net>
	<CAMPgRd7KZ4g3EhACgZ4PAhTm_W-wAt5tnTOxijDPbyjyjvNMcg@mail.gmail.com>
	<CAMPgRd7QXn7rgwffWBhWgfwm1UAKnr8z9M+Jj1Z7xkktAQwvCA@mail.gmail.com>
	<520BE4BB.1080200@g-clef.net>
Message-ID: <4F78AECE-DAC6-4873-AEFD-6837D5D5EA49@gmail.com>

Greetings,

Are you running Bro as part of Security Onion? I saw a discussion about SO issues with hostnames containing hyphens.

-David

On Aug 14, 2013, at 3:12 PM, "aaron gee-clough" <lists at g-clef.net> wrote:

> Thanks. Of the two boxes I have, one got better when I changed the hostname (have no idea why that helped, but it's been stable across reboots and restarts since then...perhaps luch). The other one I'm still working on.
> 
> aaron
> 
> On 08/14/2013 02:08 PM, Tritium Cat wrote:
>> Here's a suggestion that has helped me in the past, disable all scripts except the SSH and SSH brute force detection.  Basically you're using process of elimination to find what aspect of Bro is not performing well in your environment.  Turn on features of Bro one by one until you find which one is the culprit.  It's tricky to debug Bro from site to site because of different traffic profiles.
>> 
>> --TC
>> 
>> 
>> 
>> On Wed, Aug 14, 2013 at 9:28 AM, Tritium Cat <tritium.cat at gmail.com> wrote:
>> I've had this problem for too long.  Wish I knew too.  Seems each time it's brought up on a mailing list the discussion gets hijacked and turns into feature requests or debates on new concepts and looses sight of the original problem.
>> 
>> Keep hammering away.  Good luck.
>> 
>> 
>> On Fri, Aug 2, 2013 at 11:33 AM, aaron gee-clough <lists at g-clef.net> wrote:
>> 
>> Hello,
>> 
>> I've just put in two sensors running bro (with security onion), and am
>> having trouble with the bro processes progressively growing in RAM
>> usage, until they crash or become unresponsive. For example, I have one
>> bro worker process right now that's reached 2.8 GB in 2 hours while
>> watching a < 100MB link. None of the other processes
>> (manager/proxy/other workers) are anywhere near that...it's just this
>> one worker.
>> 
>> Are there any config options I can enable to attempt to find the cause
>> of the memory leak? Also, since I'm confident the link I'm watching is
>> missing some traffic (the span it's on is slightly mis-configured at the
>> moment), where can I configure protocol timeouts?
>> 
>> Thanks.
>> 
>> aaron
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> 
>> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130814/92bc5a94/attachment.html 

From doug.burks at gmail.com  Wed Aug 14 13:42:43 2013
From: doug.burks at gmail.com (Doug Burks)
Date: Wed, 14 Aug 2013 16:42:43 -0400
Subject: [Bro] troubleshooting bro memory usage?
In-Reply-To: <4F78AECE-DAC6-4873-AEFD-6837D5D5EA49@gmail.com>
References: <51FBFB5F.9090507@g-clef.net>
	<CAMPgRd7KZ4g3EhACgZ4PAhTm_W-wAt5tnTOxijDPbyjyjvNMcg@mail.gmail.com>
	<CAMPgRd7QXn7rgwffWBhWgfwm1UAKnr8z9M+Jj1Z7xkktAQwvCA@mail.gmail.com>
	<520BE4BB.1080200@g-clef.net>
	<4F78AECE-DAC6-4873-AEFD-6837D5D5EA49@gmail.com>
Message-ID: <CAK8kjrBBKi13=5CiZUBo+QKonEvbJq0C+izecM3f10=LwLuHFA@mail.gmail.com>

On Wed, Aug 14, 2013 at 4:27 PM, David Kovar <dkovar at gmail.com> wrote:
> Greetings,
>
> Are you running Bro as part of Security Onion? I saw a discussion about SO
> issues with hostnames containing hyphens.
>
> -David

Hi David,

I think the hyphenated hostname was circumstantial evidence as the
hostname/interface scripts were inconsistent even with non-hyphenated
hostnames.

Jon provided a workaround earlier in the thread that appears to be
more consistent so far.  I've packaged the updated scripts and
uploaded to our "test" repo.  Here's the email I sent to our testers
last night:
https://groups.google.com/d/topic/security-onion-testing/KR_Q-e-SjPQ/discussion

Thanks,
Doug


From hckim at narusec.com  Fri Aug 16 02:33:38 2013
From: hckim at narusec.com (=?UTF-8?B?6rmA7Z2s7LKg?=)
Date: Fri, 16 Aug 2013 18:33:38 +0900
Subject: [Bro] Adding a human-readable timestamp field.
Message-ID: <CAHs13Ahk-f16=Mgk+VsYt4WgAVF1hZdOBKy8jC1zcWbd2vhuAg@mail.gmail.com>

Hello
I try to add human-readable timestamp field to capture_loss.log and
stats.log

for stats.log
event Stats::log_http(rec: Stats::Info)
{     ..


 }

this did not work

and for the capture_loss there was no rec event to trigger.

is there another event I can use?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130816/1cdbc6f3/attachment.html 

From kebutler at gmail.com  Fri Aug 16 04:45:39 2013
From: kebutler at gmail.com (KB)
Date: Fri, 16 Aug 2013 07:45:39 -0400
Subject: [Bro] Adding a human-readable timestamp field.
In-Reply-To: <CAHs13Ahk-f16=Mgk+VsYt4WgAVF1hZdOBKy8jC1zcWbd2vhuAg@mail.gmail.com>
References: <CAHs13Ahk-f16=Mgk+VsYt4WgAVF1hZdOBKy8jC1zcWbd2vhuAg@mail.gmail.com>
Message-ID: <E1769CAD-FF8A-450F-8E81-F2CD8542D821@gmail.com>

My guess is the suggestion would be to derive the human readable time from epoch time in the log already.  Unless you have your reasons of course.
See "cf" in the list on this page:
http://bro.org/community/software.html
ftp://ee.lbl.gov/cf.tar.gz

# Original epoch time in log
$ grep -v "^#" capture_loss.log  | head -n 4
1376652216.898400	900.001180	worker-4	0	328165	0.000%
1376652223.161080	900.095410	worker-5	0	340367	0.000%
1376652224.511310	900.052610	worker-7	0	372860	0.000%
1376652224.377070	900.109850	worker-9	0	294452	0.000%

# Epoch time converted to human readable time using CF
$ grep -v "^#" capture_loss.log  | head -n 4 | /usr/local/bin/cf 
Aug 16 11:23:36	900.001180	worker-4	0	328165	0.000%
Aug 16 11:23:43	900.095410	worker-5	0	340367	0.000%
Aug 16 11:23:44	900.052610	worker-7	0	372860	0.000%
Aug 16 11:23:44	900.109850	worker-9	0	294452	0.000%

-kb







On Aug 16, 2013, at 5:33 AM, ??? <hckim at narusec.com> wrote:

> Hello
> I try to add human-readable timestamp field to capture_loss.log and stats.log
> 
> for stats.log
> event Stats::log_http(rec: Stats::Info)
> {     ..
> 
> 
>  }
> 
> this did not work
> 
> and for the capture_loss there was no rec event to trigger.
> 
> is there another event I can use?
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130816/2bb085a2/attachment.html 

From jbabio at po-box.esu.edu  Mon Aug 19 09:27:46 2013
From: jbabio at po-box.esu.edu (John Babio)
Date: Mon, 19 Aug 2013 16:27:46 +0000
Subject: [Bro] auto start script freebsd
Message-ID: <CE37BFC3.4173%jbabio@po-box.esu.edu>

Anyone have one of these for bro. The script I have just hangs and never starts bro.



From kebutler at gmail.com  Mon Aug 19 09:54:22 2013
From: kebutler at gmail.com (KB)
Date: Mon, 19 Aug 2013 12:54:22 -0400
Subject: [Bro] auto start script freebsd
In-Reply-To: <CE37BFC3.4173%jbabio@po-box.esu.edu>
References: <CE37BFC3.4173%jbabio@po-box.esu.edu>
Message-ID: <E42B9161-B98B-4F82-9723-C59FDDDFA3B3@gmail.com>


On Aug 19, 2013, at 12:27 PM, John Babio <jbabio at po-box.esu.edu> wrote:

> Anyone have one of these for bro. The script I have just hangs and never starts bro.
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

Are you running from ports or did you compile manually?

Did you see this link (Author provides a startup script):
http://www.sysadminwiki.net/site/doku.php/services/bro/bro_security_monitor_freebsd_installation_guide

Also seems like maybe there was a problem with ports and broctl.
http://lists.freebsd.org/pipermail/freebsd-ports-bugs/2013-January/247421.html

-kb


From r.fulton at auckland.ac.nz  Wed Aug 21 16:06:42 2013
From: r.fulton at auckland.ac.nz (Russell Fulton)
Date: Thu, 22 Aug 2013 11:06:42 +1200
Subject: [Bro] newbie questions...
Message-ID: <3A3B6C34-9436-4C4C-ADD4-A7158EF1A247@auckland.ac.nz>

Hi

First a minor nit:

I am setting up a new sensor with argus, suricata and bro.  I thought I had everything right and then broctl start would just hang with "starting manager?"

I eventually worked out that in reorganising directories after running out of disk I had managed to move the bro install files.  Re running the broctl install fixed things.  If it is straight forward for the script to check for the install files before trying to start the manager and give an informative error message that would be nice ;)

For the record I am running on a 16 core box running Ubuntu SPC and using the binary from SO (but not the SO config or scripts).

I have suricata set up to use cores 10-15 ? is there a straight forward way to assign bro to particular cores or should I just use open slather for everything?

I have assumed that the SO version of bro will use pf_ring by default? or do I need to do something to get bro to use pf_ring?

Russell ( confession ? it has only taken 4 years for implementing bro to get to the top of my todo list :(  )


From seth at icir.org  Wed Aug 21 16:36:25 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 21 Aug 2013 19:36:25 -0400
Subject: [Bro] newbie questions...
In-Reply-To: <3A3B6C34-9436-4C4C-ADD4-A7158EF1A247@auckland.ac.nz>
References: <3A3B6C34-9436-4C4C-ADD4-A7158EF1A247@auckland.ac.nz>
Message-ID: <860F9196-6809-4C48-93A0-1F3BCD61750F@icir.org>

Hi Russell!

On Aug 21, 2013, at 7:06 PM, Russell Fulton <r.fulton at auckland.ac.nz> wrote:

> For the record I am running on a 16 core box running Ubuntu SPC and using the binary from SO (but not the SO config or scripts).

Why's that?

> I have suricata set up to use cores 10-15 ? is there a straight forward way to assign bro to particular cores or should I just use open slather for everything?

In the 2.2 release that is coming soon there is a new config option for node.cfg where you can pin processes.  It will make your worker configs look like this?

[worker-1]
type=worker
host=1.2.3.4
interface=eth2
lb_method=pf_ring
lb_procs=10
pin_cpus=2,3,4,5,6,7,8,9,10,11

I think that's a pretty straight forward configuration, but let me know if there isn't anything clear in it or if you have questions.  You will only need to configure a single worker like that to load balance traffic on that host with the configured interface.  broctl will create all of the worker processes it needs.

> I have assumed that the SO version of bro will use pf_ring by default? or do I need to do something to get bro to use pf_ring?

I put it in the config above, you just need to make sure you have all of the pf_ring bits installed.  I'm a little unsure how different what you're running is from securityonion so I'm not sure I can authoritatively answer your question.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130821/1bc87558/attachment.bin 

From r.fulton at auckland.ac.nz  Wed Aug 21 17:48:59 2013
From: r.fulton at auckland.ac.nz (Russell Fulton)
Date: Thu, 22 Aug 2013 12:48:59 +1200
Subject: [Bro] newbie questions...
In-Reply-To: <860F9196-6809-4C48-93A0-1F3BCD61750F@icir.org>
References: <3A3B6C34-9436-4C4C-ADD4-A7158EF1A247@auckland.ac.nz>
	<860F9196-6809-4C48-93A0-1F3BCD61750F@icir.org>
Message-ID: <6B0E92D7-3947-4C19-A6B8-1606170AD40F@auckland.ac.nz>

Hi Seth,  nice to chat with you again!  I did get there in the end :)

Actually the biggest constraint was getting hardware capable of doing everything I wanted.

On 22/08/2013, at 11:36 AM, Seth Hall <seth at icir.org> wrote:

> Hi Russell!
> 
> On Aug 21, 2013, at 7:06 PM, Russell Fulton <r.fulton at auckland.ac.nz> wrote:
> 
>> For the record I am running on a 16 core box running Ubuntu SPC and using the binary from SO (but not the SO config or scripts).
> 
> Why's that?

I am managing the sensors using puppet ? all the config data, rule files, etc are all managed from a manager box which runs puppet. Using the SO .deb package frees me from having to fiddle around managing binary distros of the software I needed.  I figure that SO keep close enough to the bleeding edge for me ;)  The OS stuff is also managed by puppet ? my puppet server mirrors the config from our central puppet server. 

I have been using this arrangement for several years and the most painful part was always upgrading the various sensor binaries.  Puppet really assumes you have native package for all your software.  I see there are .debs on the download server ? is there an apt repository I can get them from too?

That still leaves argus for which there are no up to date official binary packages.  But SO does have them.

> 
>> I have suricata set up to use cores 10-15 ? is there a straight forward way to assign bro to particular cores or should I just use open slather for everything?
> 
> In the 2.2 release that is coming soon there is a new config option for node.cfg where you can pin processes.  It will make your worker configs look like this?
> 
> [worker-1]
> type=worker
> host=1.2.3.4
> interface=eth2
> lb_method=pf_ring
> lb_procs=10
> pin_cpus=2,3,4,5,6,7,8,9,10,11
> 
> I think that's a pretty straight forward configuration, but let me know if there isn't anything clear in it or if you have questions.  You will only need to configure a single worker like that to load balance traffic on that host with the configured interface.  broctl will create all of the worker processes it needs.
> 

Even I can deal with that :)  AT the moment I have entries for worker 1-4.

I've added the pf_ring spec to each of them.

>> I have assumed that the SO version of bro will use pf_ring by default? or do I need to do something to get bro to use pf_ring?
> 
> I put it in the config above, you just need to make sure you have all of the pf_ring bits installed.  I'm a little unsure how different what you're running is from securityonion so I'm not sure I can authoritatively answer your question.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4637 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130822/f2c8f830/attachment.bin 

From jp.bourget at gmail.com  Thu Aug 22 12:06:20 2013
From: jp.bourget at gmail.com (JP Gmail)
Date: Thu, 22 Aug 2013 15:06:20 -0400
Subject: [Bro] Bro script spacing
Message-ID: <AA2B615F-EF40-406C-A496-A6C35177DFCF@gmail.com>

Hello,

I hear there is a "style/format" for spacing syntax in bro scripts. Where can I become familiar with this?

JP


From vallentin at icir.org  Thu Aug 22 12:24:24 2013
From: vallentin at icir.org (Matthias Vallentin)
Date: Thu, 22 Aug 2013 12:24:24 -0700
Subject: [Bro] Bro script spacing
In-Reply-To: <AA2B615F-EF40-406C-A496-A6C35177DFCF@gmail.com>
References: <AA2B615F-EF40-406C-A496-A6C35177DFCF@gmail.com>
Message-ID: <CADTpMNadFkTaMPvSQFmf+54kRk+oSrm-j5oVRQb_J+dvYQ9ywQ@mail.gmail.com>

> I hear there is a "style/format" for spacing syntax in bro scripts. Where can I become familiar with this?

The indentation is called Whitesmiths style [1]. We currently don't
have an exact guideline, for example like Google's style guide [2].
Maybe some editors have the ability to explicitly set a style. At
least in Vim, you can use this:

     autocmd FileType bro set noexpandtab cino='>1s,f1s,{1s'

This gives you reasonably close support for brace placement. (It also
assumes you have a Bro syntax plugin.)

I'm curious how others have tweaked their editors, not for syntax
highlighting, but to get an efficient and style-compatible working
environment.

     Matthias

[1] http://en.wikipedia.org/wiki/Indent_style#Whitesmiths_style
[2] https://code.google.com/p/google-styleguide/


From robin at icir.org  Thu Aug 22 13:39:54 2013
From: robin at icir.org (Robin Sommer)
Date: Thu, 22 Aug 2013 13:39:54 -0700
Subject: [Bro] Bro script spacing
In-Reply-To: <CADTpMNadFkTaMPvSQFmf+54kRk+oSrm-j5oVRQb_J+dvYQ9ywQ@mail.gmail.com>
References: <AA2B615F-EF40-406C-A496-A6C35177DFCF@gmail.com>
	<CADTpMNadFkTaMPvSQFmf+54kRk+oSrm-j5oVRQb_J+dvYQ9ywQ@mail.gmail.com>
Message-ID: <20130822203954.GN25209@icir.org>

On Thu, Aug 22, 2013 at 12:24 -0700, Matthias Vallentin wrote:

> This gives you reasonably close support for brace placement.

Btw, I'd very much like to have a "formatting mode" in Bro that
reformats a given script into a standard style. We could then run that
consistently over all the standard scripts, and users could use it
adapt their own ones accordingly. Should actually not be too hard to
implement for somebody who's a bit familiar with C++.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 *     robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 * www.icir.org/robin


From slagell at illinois.edu  Thu Aug 22 13:45:12 2013
From: slagell at illinois.edu (Slagell, Adam J)
Date: Thu, 22 Aug 2013 20:45:12 +0000
Subject: [Bro] Bro script spacing
In-Reply-To: <20130822203954.GN25209@icir.org>
References: <AA2B615F-EF40-406C-A496-A6C35177DFCF@gmail.com>
	<CADTpMNadFkTaMPvSQFmf+54kRk+oSrm-j5oVRQb_J+dvYQ9ywQ@mail.gmail.com>
	<20130822203954.GN25209@icir.org>
Message-ID: <558D23D33781EF45A69229CDAC6BF151111E285D@CITESMBX6.ad.uillinois.edu>


On Aug 22, 2013, at 3:39 PM, Robin Sommer <robin at icir.org>
 wrote:

> On Thu, Aug 22, 2013 at 12:24 -0700, Matthias Vallentin wrote:
> 
>> This gives you reasonably close support for brace placement.
> 
> Btw, I'd very much like to have a "formatting mode" in Bro that
> reformats a given script into a standard style. We could then run that
> consistently over all the standard scripts, and users could use it
> adapt their own ones accordingly. Should actually not be too hard to
> implement for somebody who's a bit familiar with C++.

I could have sworn someone has made an emacs mode for Bro scripts and posted that to the list once.




From slagell at illinois.edu  Thu Aug 22 13:52:24 2013
From: slagell at illinois.edu (Slagell, Adam J)
Date: Thu, 22 Aug 2013 20:52:24 +0000
Subject: [Bro] Bro script spacing
In-Reply-To: <558D23D33781EF45A69229CDAC6BF151111E285D@CITESMBX6.ad.uillinois.edu>
References: <AA2B615F-EF40-406C-A496-A6C35177DFCF@gmail.com>
	<CADTpMNadFkTaMPvSQFmf+54kRk+oSrm-j5oVRQb_J+dvYQ9ywQ@mail.gmail.com>
	<20130822203954.GN25209@icir.org>
	<558D23D33781EF45A69229CDAC6BF151111E285D@CITESMBX6.ad.uillinois.edu>
Message-ID: <558D23D33781EF45A69229CDAC6BF151111E2926@CITESMBX6.ad.uillinois.edu>


On Aug 22, 2013, at 3:45 PM, "Slagell, Adam J" <slagell at illinois.edu>
 wrote:

> 
> On Aug 22, 2013, at 3:39 PM, Robin Sommer <robin at icir.org>
> wrote:
> 
>> On Thu, Aug 22, 2013 at 12:24 -0700, Matthias Vallentin wrote:
>> 
>>> This gives you reasonably close support for brace placement.
>> 
>> Btw, I'd very much like to have a "formatting mode" in Bro that
>> reformats a given script into a standard style. We could then run that
>> consistently over all the standard scripts, and users could use it
>> adapt their own ones accordingly. Should actually not be too hard to
>> implement for somebody who's a bit familiar with C++.
> 
> I could have sworn someone has made an emacs mode for Bro scripts and posted that to the list once.

Yes, Scott did it.

https://github.com/srunnels/bro-mode


From srunnels at gmail.com  Thu Aug 22 13:59:41 2013
From: srunnels at gmail.com (Scott Runnels)
Date: Thu, 22 Aug 2013 16:59:41 -0400
Subject: [Bro] Bro script spacing
In-Reply-To: <558D23D33781EF45A69229CDAC6BF151111E285D@CITESMBX6.ad.uillinois.edu>
References: <AA2B615F-EF40-406C-A496-A6C35177DFCF@gmail.com>
	<CADTpMNadFkTaMPvSQFmf+54kRk+oSrm-j5oVRQb_J+dvYQ9ywQ@mail.gmail.com>
	<20130822203954.GN25209@icir.org>
	<558D23D33781EF45A69229CDAC6BF151111E285D@CITESMBX6.ad.uillinois.edu>
Message-ID: <CANYzqZ6SxWBPagLfgOSpHffNwLoeUykY4K1EraEEAFxDa986dw@mail.gmail.com>

That was me.  http://github.com/srunnels/bro-mode

It needs some work to catch up to some of the new changes though.

If Bro wants to make it's official stance "just use emacs", hey I'm totally
in! :)



Scott Runnels



On Thu, Aug 22, 2013 at 4:45 PM, Slagell, Adam J <slagell at illinois.edu>wrote:

>
> On Aug 22, 2013, at 3:39 PM, Robin Sommer <robin at icir.org>
>  wrote:
>
> > On Thu, Aug 22, 2013 at 12:24 -0700, Matthias Vallentin wrote:
> >
> >> This gives you reasonably close support for brace placement.
> >
> > Btw, I'd very much like to have a "formatting mode" in Bro that
> > reformats a given script into a standard style. We could then run that
> > consistently over all the standard scripts, and users could use it
> > adapt their own ones accordingly. Should actually not be too hard to
> > implement for somebody who's a bit familiar with C++.
>
> I could have sworn someone has made an emacs mode for Bro scripts and
> posted that to the list once.
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130822/9157b502/attachment.html 

From jp.bourget at gmail.com  Thu Aug 22 14:18:39 2013
From: jp.bourget at gmail.com (JP Bourget)
Date: Thu, 22 Aug 2013 17:18:39 -0400
Subject: [Bro] Bro script spacing
In-Reply-To: <CADTpMNadFkTaMPvSQFmf+54kRk+oSrm-j5oVRQb_J+dvYQ9ywQ@mail.gmail.com>
References: <AA2B615F-EF40-406C-A496-A6C35177DFCF@gmail.com>
	<CADTpMNadFkTaMPvSQFmf+54kRk+oSrm-j5oVRQb_J+dvYQ9ywQ@mail.gmail.com>
Message-ID: <CAGb0YioLfmYjQieR82V5p9Gs=hDnvr_U44r8F3mUD4=6F3E2bA@mail.gmail.com>

Mathias,

Thanks man! This makes sense - I wonder is Seth or someone has a trick to
get this work in Sublime Text - I'll try and find one myself and post it if
I can.

JP


On Thu, Aug 22, 2013 at 3:24 PM, Matthias Vallentin <vallentin at icir.org>wrote:

> > I hear there is a "style/format" for spacing syntax in bro scripts.
> Where can I become familiar with this?
>
> The indentation is called Whitesmiths style [1]. We currently don't
> have an exact guideline, for example like Google's style guide [2].
> Maybe some editors have the ability to explicitly set a style. At
> least in Vim, you can use this:
>
>      autocmd FileType bro set noexpandtab cino='>1s,f1s,{1s'
>
> This gives you reasonably close support for brace placement. (It also
> assumes you have a Bro syntax plugin.)
>
> I'm curious how others have tweaked their editors, not for syntax
> highlighting, but to get an efficient and style-compatible working
> environment.
>
>      Matthias
>
> [1] http://en.wikipedia.org/wiki/Indent_style#Whitesmiths_style
> [2] https://code.google.com/p/google-styleguide/
>



-- 
JP
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130822/6c9032e5/attachment.html 

From r.fulton at auckland.ac.nz  Thu Aug 22 15:03:07 2013
From: r.fulton at auckland.ac.nz (Russell Fulton)
Date: Fri, 23 Aug 2013 10:03:07 +1200
Subject: [Bro] installing time machine.
Message-ID: <4996658F-578A-4C49-AE3C-C0494F75B773@auckland.ac.nz>

I have decided to give time machine a try so I cloned the git repository but when I tried ./configure I found that it could not find the broccoli library.  I had installed bro from the security onion client library and assumed that it must not have included the library.

I then tried to install broccoli from http://www.bro.org/downloads/release/broccoli-1.92.tar.gz

Now I am getting:

rful011 at secmontst01:~/broccoli-1.92$ ./configure
Build Directory : build
Source Directory: /home/rful011/broccoli-1.92
-- The C compiler identification is GNU
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libssl.so;/usr/lib/x86_64-linux-gnu/libcrypto.so 
-- FLEX_EXECUTABLE (missing:  FLEX_VERSION) 
-- Could NOT find BISON (missing:  BISON_EXECUTABLE) 
-- Found PCAP: /usr/lib/x86_64-linux-gnu/libpcap.so 
-- Performing Test PCAP_LINKS_SOLO
-- Performing Test PCAP_LINKS_SOLO - Success
-- Looking for pcap_get_pfring_id
-- Looking for pcap_get_pfring_id - not found

do I need to point configure to a different pcap library to get it to use pf_ring?

Which brings up the wider question of whether or not time machine will use pf_ring?

apart from that the install just worked and I tweaked the cfg file and it is now logging data!

Thinks:  "It can't be that easy"  ;)

Russell





From charles.fair at mac.com  Thu Aug 22 15:55:08 2013
From: charles.fair at mac.com (Charles A. Fair)
Date: Thu, 22 Aug 2013 17:55:08 -0500
Subject: [Bro] Bro script spacing
In-Reply-To: <CAGb0YioLfmYjQieR82V5p9Gs=hDnvr_U44r8F3mUD4=6F3E2bA@mail.gmail.com>
References: <AA2B615F-EF40-406C-A496-A6C35177DFCF@gmail.com>
	<CADTpMNadFkTaMPvSQFmf+54kRk+oSrm-j5oVRQb_J+dvYQ9ywQ@mail.gmail.com>
	<CAGb0YioLfmYjQieR82V5p9Gs=hDnvr_U44r8F3mUD4=6F3E2bA@mail.gmail.com>
Message-ID: <D2333847-3A15-4E15-95C5-048A26CC4267@mac.com>

Courtesy of being pointed out by Chris Crawford: 

http://liamrandall.com/syntax-highlighting-for-bro-network-programming-language/
http://www.appliednsm.com/syntax-highlighting-for-bro-for-nano/

Also:  
https://dl.dropboxusercontent.com/u/4303535/Screencasts/bro-mode.mpg 

Charles "Chuck" A. Fair 
charles.fair at mac.com




On Aug 22, 2013, at 4:18 PM, JP Bourget <jp.bourget at gmail.com> wrote:

> Mathias, 
> 
> Thanks man! This makes sense - I wonder is Seth or someone has a trick to get this work in Sublime Text - I'll try and find one myself and post it if I can. 
> 
> JP
> 
> 
> On Thu, Aug 22, 2013 at 3:24 PM, Matthias Vallentin <vallentin at icir.org> wrote:
> > I hear there is a "style/format" for spacing syntax in bro scripts. Where can I become familiar with this?
> 
> The indentation is called Whitesmiths style [1]. We currently don't
> have an exact guideline, for example like Google's style guide [2].
> Maybe some editors have the ability to explicitly set a style. At
> least in Vim, you can use this:
> 
>      autocmd FileType bro set noexpandtab cino='>1s,f1s,{1s'
> 
> This gives you reasonably close support for brace placement. (It also
> assumes you have a Bro syntax plugin.)
> 
> I'm curious how others have tweaked their editors, not for syntax
> highlighting, but to get an efficient and style-compatible working
> environment.
> 
>      Matthias
> 
> [1] http://en.wikipedia.org/wiki/Indent_style#Whitesmiths_style
> [2] https://code.google.com/p/google-styleguide/
> 
> 
> 
> -- 
> JP
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130822/2768fdc6/attachment.html 

From seth at icir.org  Thu Aug 22 16:12:43 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 22 Aug 2013 19:12:43 -0400
Subject: [Bro] installing time machine.
In-Reply-To: <4996658F-578A-4C49-AE3C-C0494F75B773@auckland.ac.nz>
References: <4996658F-578A-4C49-AE3C-C0494F75B773@auckland.ac.nz>
Message-ID: <91A5BBB5-792E-4111-AA0F-3AAF16A6CA6D@icir.org>

On Aug 22, 2013, at 6:03 PM, Russell Fulton <r.fulton at auckland.ac.nz> wrote:
> I have decided to give time machine a try so I cloned the git repository but when I tried ./configure I found that it could not find the broccoli library.

You probably could have just pointed to the Bro install directory for Broccoli when you ran configure?

--with-broccoli=/where/ever/so/puts/bro

> Which brings up the wider question of whether or not time machine will use pf_ring?

Nope, no pf_ring support.

> Thinks:  "It can't be that easy"  ;)


I suspect that's about it.  I did a lot of clean up of time machine a while back to make it like that. :P

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130822/14d36819/attachment.bin 

From seth at icir.org  Thu Aug 22 16:14:35 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 22 Aug 2013 19:14:35 -0400
Subject: [Bro] Bro script spacing
In-Reply-To: <CAGb0YioLfmYjQieR82V5p9Gs=hDnvr_U44r8F3mUD4=6F3E2bA@mail.gmail.com>
References: <AA2B615F-EF40-406C-A496-A6C35177DFCF@gmail.com>
	<CADTpMNadFkTaMPvSQFmf+54kRk+oSrm-j5oVRQb_J+dvYQ9ywQ@mail.gmail.com>
	<CAGb0YioLfmYjQieR82V5p9Gs=hDnvr_U44r8F3mUD4=6F3E2bA@mail.gmail.com>
Message-ID: <C0EC492D-14C8-44EE-BDAE-89D43ED7C438@icir.org>

On Aug 22, 2013, at 5:18 PM, JP Bourget <jp.bourget at gmail.com> wrote:

> Thanks man! This makes sense - I wonder is Seth or someone has a trick to get this work in Sublime Text - I'll try and find one myself and post it if I can. 

I just deal with it manually in sublime text.  I've been meaning to fix my Bro support for textmate and sublimetext *forever* to make it work correctly.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130822/8b68bf58/attachment.bin 

From r.fulton at auckland.ac.nz  Fri Aug 23 03:10:44 2013
From: r.fulton at auckland.ac.nz (Russell Fulton)
Date: Fri, 23 Aug 2013 22:10:44 +1200
Subject: [Bro] telling broctl where to get broctl.cfg
Message-ID: <42077246-1783-4260-A7FE-DEB4B2585F63@auckland.ac.nz>

Hi

It would be convenient if I could have the broctl.cfg somewhere other than the ?./bro/etc.  I have all the other local configs in another directory referred to in broctl.cfg but I would like to move it there too.

I am using using puppet to maintain all my config files and it make it a bit tidier to have them all in one place.

Russell


From kebutler at gmail.com  Fri Aug 23 04:53:13 2013
From: kebutler at gmail.com (KB)
Date: Fri, 23 Aug 2013 07:53:13 -0400
Subject: [Bro] installing time machine.
In-Reply-To: <4996658F-578A-4C49-AE3C-C0494F75B773@auckland.ac.nz>
References: <4996658F-578A-4C49-AE3C-C0494F75B773@auckland.ac.nz>
Message-ID: <3EDF3AF4-BE6E-4C5C-AF41-09DB07FA7924@gmail.com>


On Aug 22, 2013, at 6:03 PM, Russell Fulton <r.fulton at auckland.ac.nz> wrote:

> I have decided to give time machine a try so I cloned the git repository but when I tried ./configure I found that it could not find the broccoli library.  I had installed bro from the security onion client library and assumed that it must not have included the library.

It has the library.  It's in /opt/bro/{include,lib}.  So adding "--with-broccoli=/opt/bro" should take care of it for you.
Here is the configure command that I got working with included pcap and bro.
$ CXXFLAGS="-I/opt/pfring/include" CFLAGS="-I/opt/pfring/include" ./configure --with-broccoli=/opt/bro --with-pcap=/opt/pfring

From your output, it looks like you could install bison and flex, too.





> 
> I then tried to install broccoli from http://www.bro.org/downloads/release/broccoli-1.92.tar.gz
> 
> Now I am getting:
> 
> rful011 at secmontst01:~/broccoli-1.92$ ./configure
> Build Directory : build
> Source Directory: /home/rful011/broccoli-1.92
> -- The C compiler identification is GNU
> -- Check for working C compiler: /usr/bin/gcc
> -- Check for working C compiler: /usr/bin/gcc -- works
> -- Detecting C compiler ABI info
> -- Detecting C compiler ABI info - done
> -- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libssl.so;/usr/lib/x86_64-linux-gnu/libcrypto.so 
> -- FLEX_EXECUTABLE (missing:  FLEX_VERSION) 
> -- Could NOT find BISON (missing:  BISON_EXECUTABLE) 
> -- Found PCAP: /usr/lib/x86_64-linux-gnu/libpcap.so 
> -- Performing Test PCAP_LINKS_SOLO
> -- Performing Test PCAP_LINKS_SOLO - Success
> -- Looking for pcap_get_pfring_id
> -- Looking for pcap_get_pfring_id - not found
> 
> do I need to point configure to a different pcap library to get it to use pf_ring?
> 
> Which brings up the wider question of whether or not time machine will use pf_ring?
> 
> apart from that the install just worked and I tweaked the cfg file and it is now logging data!
> 
> Thinks:  "It can't be that easy"  ;)
> 
> Russell
> 
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 881 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130823/7e4ca8b4/attachment.bin 

From seth at icir.org  Fri Aug 23 06:35:55 2013
From: seth at icir.org (Seth Hall)
Date: Fri, 23 Aug 2013 09:35:55 -0400
Subject: [Bro] telling broctl where to get broctl.cfg
In-Reply-To: <42077246-1783-4260-A7FE-DEB4B2585F63@auckland.ac.nz>
References: <42077246-1783-4260-A7FE-DEB4B2585F63@auckland.ac.nz>
Message-ID: <4609D6FA-EFBA-4A41-920F-B05C48E509D7@icir.org>

On Aug 23, 2013, at 6:10 AM, Russell Fulton <r.fulton at auckland.ac.nz> wrote:

> I am using using puppet to maintain all my config files and it make it a bit tidier to have them all in one place.


Since you are using a pre-build package of Bro you might just have to play with symlinks.  If you build Bro you can use this option?

    --conf-files-dir=PATH  config files installation directory [PREFIX/etc]

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130823/1dae500f/attachment.bin 

From phatbuckett at gmail.com  Sun Aug 25 01:01:30 2013
From: phatbuckett at gmail.com (Darren Spruell)
Date: Sun, 25 Aug 2013 01:01:30 -0700
Subject: [Bro] Redux - source build on OpenBSD (5.3) and BIND libs
Message-ID: <CAKVSOJVa60eX7P34NDW-Oh_TLzgAJ285XtPVRWn8jtt5NT53JQ@mail.gmail.com>

Greetings,

CMake 2.8.10.2
Perl 5.12.2
libmagic 5.11
SWIG 1.3.36
Bison 2.3
Flex 2.5.4
Bash 4.2.42

Got stumped trying to build Bro on OpenBSD 5.3 i386 related to finding
BIND8 headers/libs.

I see a few past similar issues, notably:
http://marc.info/?l=bro&m=132806089033571

Installing Bro 2.1.

$ ./configure
Build Directory : build
Source Directory: /home/dspruell/downloads/bro-2.1
-- The C compiler identification is GNU 4.2.1
-- The CXX compiler identification is GNU 4.2.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found sed: /usr/bin/sed
-- Found Perl: /usr/bin/perl (found version "5.12.2")
-- Found FLEX: /usr/bin/flex version 2.5.4
-- Found BISON: /usr/local/bin/bison
-- Found PCAP: /usr/lib/libpcap.so.7.0
-- Performing Test PCAP_LINKS_SOLO
-- Performing Test PCAP_LINKS_SOLO - Success
-- Looking for pcap_get_pfring_id
-- Looking for pcap_get_pfring_id - not found
-- Found OpenSSL: /usr/lib/libssl.so.19.0;/usr/lib/libcrypto.so.22.0
-- Performing Test ns_initparse_works_none
-- Performing Test ns_initparse_works_none - Failed
-- Performing Test res_mkquery_works_none
-- Performing Test res_mkquery_works_none - Success
-- Performing Test ns_initparse_works_resolv
-- Performing Test ns_initparse_works_resolv - Failed
-- Performing Test res_mkquery_works_resolv
-- Performing Test res_mkquery_works_resolv - Success
-- Performing Test ns_initparse_works_bind
-- Performing Test ns_initparse_works_bind - Failed
-- Performing Test res_mkquery_works_bind
-- Performing Test res_mkquery_works_bind - Success
-- Could NOT find BIND (missing:  BIND_LIBRARY)
-- Found LibMagic: /usr/local/lib/libmagic.so.3.0
-- Found ZLIB: /usr/lib/libz.so.4.1 (found version "1.2.3")
CMake Error at aux/binpac/CMakeLists.txt:17 (message):
   Could not find prerequisite package 'BIND'


CMake Error at aux/binpac/CMakeLists.txt:19 (message):
  Configuration aborted due to missing prerequisites


-- Configuring incomplete, errors occurred!


I'm hung up trying to figure out where the necessary
routines/libraries would be. OpenBSD ships with BIND 9 by default, and
has res_* functions in libc (there is no libresolv.a, etc.; libresolv
was removed ~2005). The previously referenced thread mentions libbind
package; this doesn't seem to exist any more, although there is an an
upstream ISC BIND 9 package (isc-bind 9.9.2-P2). When this package is
installed, I can see the following library:

$ ldconfig -r |egrep 'bind'
        398:-lbind9.0.0 => /usr/local/lib/libbind9.so.0.0

...but the library does not provide ns_* symbols and the error at
configure is still the same (maybe because of BIND 9 vs. BIND 8?):

-- Performing Test ns_initparse_works_none
-- Performing Test ns_initparse_works_none - Failed
-- Performing Test res_mkquery_works_none
-- Performing Test res_mkquery_works_none - Success
-- Performing Test ns_initparse_works_resolv
-- Performing Test ns_initparse_works_resolv - Failed
-- Performing Test res_mkquery_works_resolv
-- Performing Test res_mkquery_works_resolv - Success
-- Performing Test ns_initparse_works_bind
-- Performing Test ns_initparse_works_bind - Failed
-- Performing Test res_mkquery_works_bind
-- Performing Test res_mkquery_works_bind - Success
-- Could NOT find BIND (missing:  BIND_LIBRARY)

CMake Error at aux/binpac/CMakeLists.txt:17 (message):
   Could not find prerequisite package 'BIND'



It looks to me that OpenBSD doesn't include the ns_* routines; this
discussion might support that:

http://bugs.bitlbee.org/bitlbee/ticket/421

Wondering if I'm at a dead end on this. Any ideas?

-- 
Darren Spruell
phatbuckett at gmail.com


From ppowell at 21ct.com  Sun Aug 25 10:56:43 2013
From: ppowell at 21ct.com (Patrick Powell)
Date: Sun, 25 Aug 2013 12:56:43 -0500
Subject: [Bro] Bro 2.1 http cookies extraction
Message-ID: <CE3FAF8B.1DE7%ppowell@21ct.com>

Hello all.  I am trying to extract cookie key/value pairs with bro.  Bro 2.1
out of the box comes with
/usr/local/bro/share/bro/policy/protocols/http/var-extraction-cookies.bro
that extracts only the keys.  Below is the baked in way of keys and I'm
looking for something similar for the values.  I would rather use something
that is already there if it exists but cannot find it.

If it does not exist, what would be the recommendation for creating it?   My
understanding is that modifying anything outside of
/usr/local/bro/share/bro/site can get overwritten with updates.  Should I
create a whole new protocols/http directory structure under
/usr/local/bro/share/bro/site and keep the configuration of cookies
separate?

###################################################################
/usr/local/bro/share/bro/policy/protocols/http/var-extraction-cookies.bro

##! Extracts and logs variables names from cookies sent by clients.

@load base/protocols/http/main
@load base/protocols/http/utils

module HTTP;

redef record Info += {
        ## Variable names extracted from all cookies.
        cookie_vars: vector of string &optional &log;
};

event http_header(c: connection, is_orig: bool, name: string, value: string)
&priority=2
        {
        if ( is_orig && name == "COOKIE" )
                c$http$cookie_vars = extract_keys(value, /;[[:blank:]]*/);
        }
###################################################################
function extract_keys from
/usr/local/bro/share/bro/base/protocols/http/utils.bro
###################################################################
function extract_keys(data: string, kv_splitter: pattern): string_vec
        {
        local key_vec: vector of string = vector();

        local parts = split(data, kv_splitter);
        for ( part_index in parts )
                {
                local key_val = split1(parts[part_index], /=/);
                if ( 1 in key_val )
                        key_vec[|key_vec|] = key_val[1];
                }
        return key_vec;
        }

Thanks,
Patrick Powell



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130825/d103c327/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3799 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130825/d103c327/attachment.bin 

From jsiwek at illinois.edu  Mon Aug 26 07:59:52 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Mon, 26 Aug 2013 14:59:52 +0000
Subject: [Bro] Redux - source build on OpenBSD (5.3) and BIND libs
In-Reply-To: <CAKVSOJVa60eX7P34NDW-Oh_TLzgAJ285XtPVRWn8jtt5NT53JQ@mail.gmail.com>
References: <CAKVSOJVa60eX7P34NDW-Oh_TLzgAJ285XtPVRWn8jtt5NT53JQ@mail.gmail.com>
Message-ID: <B3E27C90B7D2DD4CAEF09215A29EB0D1170A2A3E@CITESMBX2.ad.uillinois.edu>

> I'm hung up trying to figure out where the necessary
> routines/libraries would be. OpenBSD ships with BIND 9 by default, and
> has res_* functions in libc (there is no libresolv.a, etc.; libresolv
> was removed ~2005). The previously referenced thread mentions libbind
> package; this doesn't seem to exist any more, although there is an an
> upstream ISC BIND 9 package (isc-bind 9.9.2-P2).

There's a libbind 6.0 package at [1] (third from bottom of list) that might have what's needed.  From what I read [2], the resolver library and headers it provides used to be part of BIND 8/9, but later removed as its own separate package.  Not sure if OpenBSD has it's own package corresponding to that.

- Jon

[1] https://www.isc.org/downloads/
[2] https://lists.isc.org/pipermail/bind-users/2009-May/076322.html


From himself at louruppert.com  Mon Aug 26 08:14:49 2013
From: himself at louruppert.com (Lou RUPPERT)
Date: Mon, 26 Aug 2013 11:14:49 -0400
Subject: [Bro] Redux - source build on OpenBSD (5.3) and BIND libs
In-Reply-To: <B3E27C90B7D2DD4CAEF09215A29EB0D1170A2A3E@CITESMBX2.ad.uillinois.edu>
References: <CAKVSOJVa60eX7P34NDW-Oh_TLzgAJ285XtPVRWn8jtt5NT53JQ@mail.gmail.com>
	<B3E27C90B7D2DD4CAEF09215A29EB0D1170A2A3E@CITESMBX2.ad.uillinois.edu>
Message-ID: <521B70E9.1040206@louruppert.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 08/26/2013 10:59 AM, Siwek, Jonathan Luke wrote:
>> I'm hung up trying to figure out where the necessary 
>> routines/libraries would be. OpenBSD ships with BIND 9 by
>> default, and has res_* functions in libc (there is no
>> libresolv.a, etc.; libresolv was removed ~2005). The previously
>> referenced thread mentions libbind package; this doesn't seem to
>> exist any more, although there is an an upstream ISC BIND 9
>> package (isc-bind 9.9.2-P2).
> 
> There's a libbind 6.0 package at [1] (third from bottom of list)
> that might have what's needed.  From what I read [2], the resolver
> library and headers it provides used to be part of BIND 8/9, but
> later removed as its own separate package.  Not sure if OpenBSD has
> it's own package corresponding to that.

5.3 doesn't. I had to grab libbind 6.0 from isc.org in order to get it
to compile. IIRC that's the only thing I had to do.


- -- 
I prefer encrypted email.  Get my key here:
http://www.louruppert.com/keys/115DCF62.asc
PGP Fingerprint: 3261 B9F9 9363 D512 56F8  12DD 127F 4D6A 115D CF62
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Icedove - http://www.enigmail.net/

iEYEARECAAYFAlIbcOMACgkQEn9NahFdz2JN/gCgsSTTJQcCFlIS/nRxYn4hsWsC
hR0AoOXrr0A1Ilx6JVlkXC6Jejy960+T
=U9Ry
-----END PGP SIGNATURE-----


From phatbuckett at gmail.com  Mon Aug 26 23:00:22 2013
From: phatbuckett at gmail.com (Darren Spruell)
Date: Mon, 26 Aug 2013 23:00:22 -0700
Subject: [Bro] Redux - source build on OpenBSD (5.3) and BIND libs
In-Reply-To: <521B70E9.1040206@louruppert.com>
References: <CAKVSOJVa60eX7P34NDW-Oh_TLzgAJ285XtPVRWn8jtt5NT53JQ@mail.gmail.com>
	<B3E27C90B7D2DD4CAEF09215A29EB0D1170A2A3E@CITESMBX2.ad.uillinois.edu>
	<521B70E9.1040206@louruppert.com>
Message-ID: <CAKVSOJUj+C6fB0U7G82BOqD6X9fVhjCZLRq6UrP57QA8gUX79w@mail.gmail.com>

Indeed, appears that the libbind port disappeared for 5.2 and 5.3 but
has since been reinstated.

http://www.openbsd.org/cgi-bin/cvsweb/ports/net/libbind/
http://marc.info/?l=openbsd-ports&m=136465613319561&w=2

I'll give it a go with the source package, thanks for the help.

-- 
Darren Spruell
phatbuckett at gmail.com


On Mon, Aug 26, 2013 at 8:14 AM, Lou RUPPERT <himself at louruppert.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 08/26/2013 10:59 AM, Siwek, Jonathan Luke wrote:
>>> I'm hung up trying to figure out where the necessary
>>> routines/libraries would be. OpenBSD ships with BIND 9 by
>>> default, and has res_* functions in libc (there is no
>>> libresolv.a, etc.; libresolv was removed ~2005). The previously
>>> referenced thread mentions libbind package; this doesn't seem to
>>> exist any more, although there is an an upstream ISC BIND 9
>>> package (isc-bind 9.9.2-P2).
>>
>> There's a libbind 6.0 package at [1] (third from bottom of list)
>> that might have what's needed.  From what I read [2], the resolver
>> library and headers it provides used to be part of BIND 8/9, but
>> later removed as its own separate package.  Not sure if OpenBSD has
>> it's own package corresponding to that.
>
> 5.3 doesn't. I had to grab libbind 6.0 from isc.org in order to get it
> to compile. IIRC that's the only thing I had to do.
>
>
> - --
> I prefer encrypted email.  Get my key here:
> http://www.louruppert.com/keys/115DCF62.asc
> PGP Fingerprint: 3261 B9F9 9363 D512 56F8  12DD 127F 4D6A 115D CF62
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
> Comment: Using GnuPG with Icedove - http://www.enigmail.net/
>
> iEYEARECAAYFAlIbcOMACgkQEn9NahFdz2JN/gCgsSTTJQcCFlIS/nRxYn4hsWsC
> hR0AoOXrr0A1Ilx6JVlkXC6Jejy960+T
> =U9Ry
> -----END PGP SIGNATURE-----
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From chrisroose at fastmail.fm  Tue Aug 27 08:00:26 2013
From: chrisroose at fastmail.fm (Chris Roose)
Date: Tue, 27 Aug 2013 11:00:26 -0400
Subject: [Bro] Connection summary email inaccuracies
Message-ID: <521CBF0A.8030008@fastmail.fm>

Hello,

I've used Bro on and off for a couple years and love its unix-ness and
application-layer smarts. I use it to augment my NetFlow and SNMP data,
and it gives me just enough information to complement those logs. I
haven't dug into the scripting and IDS aspects yet, but I hope to soon.

I have an issue with the connection summary email. Aside from the fact
that I could do without it altogether, because it doesn't really tell me
anything that NetFlow can't, I'm confused by how inaccurate the
information in the email seems to be.

To take the example that always jumps out at me, here are the incoming
port statistics from this morning's email.

>== Incoming === 2013-08-25-23-50-18 - 2013-08-26-23-19-23
   - Connections  306.0 - Payload 137.0m -
     Ports        |
     9997   78.1% |
     3       9.2% |
     514     5.2% |
     50664   3.6% |
     22      1.3% |
     52145   0.7% |
     51222   0.7% |
     52140   0.3% |
     51735   0.3% |
     51724   0.3% |

The reason I know something is strange about this is that I get NetFlow
data around the clock from three different sites on ports 9997, 9998,
and 9999. How could it be that one site accounts for 78.1% of all of my
incoming traffic and the other two are nowhere to be seen? Also, the
number of connections and payload information is way off. Here is the
same information queried from NetFlow:

Port     Flows(%)
0        4831(13.0)
9999     1985( 5.3)
9997     1797( 4.8)
9998     1510( 4.1)
22       559( 1.5)
123      398( 1.1)
64115    349( 0.9)
65138    162( 0.4)
40767    135( 0.4)
13496    120( 0.3)

Summary: total flows: 37254, total bytes: 2.1 G, total packets: 1.7 M,
avg bps: 194612, avg pps: 19, avg bpp: 1237
Time window: 2013-08-25 23:49:42 - 2013-08-26 23:24:49
Total flows processed: 102134, Blocks skipped: 0, Bytes read: 5318892

Netstat doesn't indicate any dropped packets, and conn.log doesn't
indicate any missed_bytes. Can anyone shed some light on why bro could
be so wrong about these statistics? Would it matter that I am using a
single instance of bro to monitor two interfaces (bro -i em0 -i em1)?

Thanks for any help you can provide...

Best,
Chris


From david at mandelberg.org  Tue Aug 27 12:38:29 2013
From: david at mandelberg.org (David Mandelberg)
Date: Tue, 27 Aug 2013 15:38:29 -0400
Subject: [Bro] syntax error in local-networks.bro
Message-ID: <b19bc68a0dd4a2b4623347b81504191a@mail.mandelberg.org>

Hi,

I'm trying to set up another Bro cluster, and I'm getting this error on 
the worker nodes:

error in 
/usr/local/bro/spool/installed-scripts-do-not-touch/auto/local-networks.bro, 
line 3: syntax error, at or near "redef"

This is the file:

# Automatically generated. Do not edit.

redef Site::local_nets = {
	10.0.0.0/8, 	# Private IP space
	192.168.0.0/16, 	# Private IP space
};


I don't see a syntax error. Has anybody seen anything like this before? 
I'm running from git master, commit 927f534.

-- 
David Eric Mandelberg / dseomn
http://david.mandelberg.org/


From seth at icir.org  Tue Aug 27 20:20:31 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 27 Aug 2013 23:20:31 -0400
Subject: [Bro] syntax error in local-networks.bro
In-Reply-To: <b19bc68a0dd4a2b4623347b81504191a@mail.mandelberg.org>
References: <b19bc68a0dd4a2b4623347b81504191a@mail.mandelberg.org>
Message-ID: <9C2F5926-DCF1-40C0-93DB-20C1F6DF1666@icir.org>

On Aug 27, 2013, at 3:38 PM, David Mandelberg <david at mandelberg.org> wrote:

> error in 
> /usr/local/bro/spool/installed-scripts-do-not-touch/auto/local-networks.bro, 
> line 3: syntax error, at or near "redef"


You probably have an error at the end of your local.bro

  .Seth


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130827/e144f3de/attachment.bin 

From nicolas.retrain at cea.fr  Wed Aug 28 05:20:24 2013
From: nicolas.retrain at cea.fr (nicolas.retrain at cea.fr)
Date: Wed, 28 Aug 2013 14:20:24 +0200
Subject: [Bro] @load missing in the init-bare.bro
Message-ID: <521DEB08.3020508@cea.fr>

I clone the master this morning,

I install it with:
./configure --disable-auxtools --disable-broccoli --disable-broctl
make
make install

and then I had an error at the execution  :

"internal error in [mypath]/bro/base/init-bar.bro, line 3098:internal 
type Files::AnalyzerArgs missing
Abandon"

Is there a @load missing in the init-bare.bro file?



From omer007security at walla.co.il  Wed Aug 28 06:36:08 2013
From: omer007security at walla.co.il (=?UTF-8?Q?=D7=A2=D7=95=D7=9E=D7=A8=20=D7=A2=D7=95=D7=9E=D7=A8?=)
Date: Wed, 28 Aug 2013 16:36:08 +0300
Subject: [Bro] =?utf-8?q?Bro_problem_-_no_software=2Elog_written?=
Message-ID: <1377696968.732000-70822245-31691@walla.co.il>

An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130828/a411f0ec/attachment.html 

From jsiwek at illinois.edu  Wed Aug 28 09:05:57 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Wed, 28 Aug 2013 16:05:57 +0000
Subject: [Bro] @load missing in the init-bare.bro
In-Reply-To: <521DEB08.3020508@cea.fr>
References: <521DEB08.3020508@cea.fr>
Message-ID: <B3E27C90B7D2DD4CAEF09215A29EB0D1170A48E9@CITESMBX2.ad.uillinois.edu>

> "internal error in [mypath]/bro/base/init-bar.bro, line 3098:internal 
> type Files::AnalyzerArgs missing
> Abandon"
> 
> Is there a @load missing in the init-bare.bro file?

I don't think so, or at least I'm not seeing the same thing.  Can you give more details on how you're running bro and any local changes/scripts or altering of BROPATH environment variable?

- Jon


From seth at icir.org  Wed Aug 28 09:17:53 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 28 Aug 2013 12:17:53 -0400
Subject: [Bro] @load missing in the init-bare.bro
In-Reply-To: <521DEB08.3020508@cea.fr>
References: <521DEB08.3020508@cea.fr>
Message-ID: <CF6678E6-91E5-4A7A-839D-5C1CD45F618B@icir.org>

On Aug 28, 2013, at 8:20 AM, nicolas.retrain at cea.fr wrote:

> "internal error in [mypath]/bro/base/init-bar.bro, line 3098:internal 
> type Files::AnalyzerArgs missing
> Abandon"


You're missing the 'e' at the end of init-bare.bro

  .Seth


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130828/3487f0eb/attachment.bin 

From kebutler at gmail.com  Wed Aug 28 10:09:27 2013
From: kebutler at gmail.com (Keith Butler)
Date: Wed, 28 Aug 2013 13:09:27 -0400
Subject: [Bro] Bro problem - no software.log written
In-Reply-To: <1377696968.732000-70822245-31691@walla.co.il>
References: <1377696968.732000-70822245-31691@walla.co.il>
Message-ID: <66B82052-33E9-488D-A1CE-57117831367F@gmail.com>

Are you running against a pcap or sniffing an interface?

If sniffing an interface, as a first step check that the software scripts are being loaded:

$ pwd
/path/to/bro/logs/2013-08-28

$ zgrep software loaded_scripts.16\:59\:36-17\:00\:00.log.gz 
  /usr/local/bro/share/bro/base/frameworks/software/__load__.bro
  /usr/local/bro/share/bro/base/frameworks/software/./main.bro
  /usr/local/bro/share/bro/policy/frameworks/software/vulnerable.bro
  /usr/local/bro/share/bro/policy/frameworks/software/version-changes.bro
  /usr/local/bro/share/bro/policy/protocols/ftp/software.bro
  /usr/local/bro/share/bro/policy/protocols/smtp/software.bro
  /usr/local/bro/share/bro/policy/protocols/ssh/software.bro
  /usr/local/bro/share/bro/policy/protocols/http/software.bro

If running against a pcap, add local to the end of your command:
$ bro -r my.pcap local

-kb


On Aug 28, 2013, at 9:36 AM, ???? ???? <omer007security at walla.co.il> wrote:

> Hi,
> 
> 
> 
> How can I debug why no software log  is written..?
> 
> 
> 
> I use Bro 2.1 compiled from source.
> 
> 
> 
> Thanks,
> 
> 
> 
> Omer
> 
> Walla! Mail - Get your free unlimited mail today
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130828/e106fe39/attachment.html 

From david at mandelberg.org  Wed Aug 28 10:13:29 2013
From: david at mandelberg.org (David Mandelberg)
Date: Wed, 28 Aug 2013 13:13:29 -0400
Subject: [Bro] syntax error in local-networks.bro
In-Reply-To: <9C2F5926-DCF1-40C0-93DB-20C1F6DF1666@icir.org>
References: <b19bc68a0dd4a2b4623347b81504191a@mail.mandelberg.org>
	<9C2F5926-DCF1-40C0-93DB-20C1F6DF1666@icir.org>
Message-ID: <0f322234bb446fa4b76200e0c2db54b7@mail.mandelberg.org>

On 2013-08-27 23:20, Seth Hall wrote:
> On Aug 27, 2013, at 3:38 PM, David Mandelberg <david at mandelberg.org> 
> wrote:
>
>> error in
>> 
>> /usr/local/bro/spool/installed-scripts-do-not-touch/auto/local-networks.bro,
>> line 3: syntax error, at or near "redef"
>
>
> You probably have an error at the end of your local.bro

Thanks. My local-worker.bro was missing a '}'.

-- 
David Eric Mandelberg / dseomn
http://david.mandelberg.org/


From jsiwek at illinois.edu  Wed Aug 28 10:30:48 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Wed, 28 Aug 2013 17:30:48 +0000
Subject: [Bro] Connection summary email inaccuracies
In-Reply-To: <521CBF0A.8030008@fastmail.fm>
References: <521CBF0A.8030008@fastmail.fm>
Message-ID: <B3E27C90B7D2DD4CAEF09215A29EB0D1170A4BFE@CITESMBX2.ad.uillinois.edu>

> Netstat doesn't indicate any dropped packets, and conn.log doesn't
> indicate any missed_bytes. Can anyone shed some light on why bro could
> be so wrong about these statistics? Would it matter that I am using a
> single instance of bro to monitor two interfaces (bro -i em0 -i em1)?

The interface thing shouldn't matter.  What version of Bro are you using?  I think there was some race in how log rotation postprocessing occurred that was fixed in git [1] that may be a cause for what you're seeing.  So you might try testing from git sources as a first step to see if suddenly the summary starts looking correct.

Else the approach to finding where it's going wrong would be:  Does conn.log look correct?  If no, then it's a Bro problem.  If yes, then it's a problem with how conn.log is parsed by $prefix/bin/trace-summary or how that python script is invoked by BroControl.  You should be able to run that trace-summary script manually on one of your conn.log's to see if it actually gives sane output.  That looks like:

    PYTHONPATH=/usr/local/bro/lib/broctl/ /usr/local/bro/bin/trace-summary -c -r -l /usr/local/bro/etc/networks.cfg conn.log

Doing a quick test myself I don't think I see anything overtly wrong, though there's some warnings from it that make me think payload may be under-reported.  Another weird thing is that if a connection is between two "local" hosts specified in networks.cfg, that's categorized as "outgoing", not "incoming".

- Jon

[1] https://bro-tracker.atlassian.net/browse/BIT-970


From seth at icir.org  Wed Aug 28 16:23:10 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 28 Aug 2013 19:23:10 -0400
Subject: [Bro] Connection summary email inaccuracies
In-Reply-To: <B3E27C90B7D2DD4CAEF09215A29EB0D1170A4BFE@CITESMBX2.ad.uillinois.edu>
References: <521CBF0A.8030008@fastmail.fm>
	<B3E27C90B7D2DD4CAEF09215A29EB0D1170A4BFE@CITESMBX2.ad.uillinois.edu>
Message-ID: <19FEC727-4598-47A2-A4CF-596CAE731835@icir.org>

On Aug 28, 2013, at 1:30 PM, "Siwek, Jonathan Luke" <jsiwek at illinois.edu> wrote:
>> single instance of bro to monitor two interfaces (bro -i em0 -i em1)?
> 
> Else the approach to finding where it's going wrong would be:  Does conn.log look correct?

More specifically, could you paste a few lines from your conn.log? (feel free to obfuscate ip addresses).

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130828/3148b582/attachment.bin 

From chrisroose at fastmail.fm  Wed Aug 28 17:04:13 2013
From: chrisroose at fastmail.fm (Chris Roose)
Date: Wed, 28 Aug 2013 20:04:13 -0400
Subject: [Bro] Connection summary email inaccuracies
In-Reply-To: <19FEC727-4598-47A2-A4CF-596CAE731835@icir.org>
References: <521CBF0A.8030008@fastmail.fm>
	<B3E27C90B7D2DD4CAEF09215A29EB0D1170A4BFE@CITESMBX2.ad.uillinois.edu>
	<19FEC727-4598-47A2-A4CF-596CAE731835@icir.org>
Message-ID: <521E8FFD.4050006@fastmail.fm>

On 8/28/13 7:23 PM, Seth Hall wrote:
>> Does conn.log look correct?
> 
> More specifically, could you paste a few lines from your conn.log? 
> (feel free to obfuscate ip addresses).

Sure... any particular details you'd like to see?

-- 
Chris



From seth at icir.org  Wed Aug 28 18:24:50 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 28 Aug 2013 21:24:50 -0400
Subject: [Bro] Connection summary email inaccuracies
In-Reply-To: <521E8FFD.4050006@fastmail.fm>
References: <521CBF0A.8030008@fastmail.fm>
	<B3E27C90B7D2DD4CAEF09215A29EB0D1170A4BFE@CITESMBX2.ad.uillinois.edu>
	<19FEC727-4598-47A2-A4CF-596CAE731835@icir.org>
	<521E8FFD.4050006@fastmail.fm>
Message-ID: <4A15DBA1-BD51-4768-8112-678A56ADA32F@icir.org>

On Aug 28, 2013, at 8:04 PM, Chris Roose <chrisroose at fastmail.fm> wrote:

> Sure... any particular details you'd like to see?


Just a few full connection records for tcp traffic.  There is typically a lot of detail you can tease out if you know what you're looking for. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130828/7632018c/attachment.bin 

From sconzo at visiblerisk.com  Wed Aug 28 19:54:03 2013
From: sconzo at visiblerisk.com (Mike Sconzo)
Date: Wed, 28 Aug 2013 21:54:03 -0500
Subject: [Bro] Connection summary email inaccuracies
In-Reply-To: <4A15DBA1-BD51-4768-8112-678A56ADA32F@icir.org>
References: <521CBF0A.8030008@fastmail.fm>
	<B3E27C90B7D2DD4CAEF09215A29EB0D1170A4BFE@CITESMBX2.ad.uillinois.edu>
	<19FEC727-4598-47A2-A4CF-596CAE731835@icir.org>
	<521E8FFD.4050006@fastmail.fm>
	<4A15DBA1-BD51-4768-8112-678A56ADA32F@icir.org>
Message-ID: <CAGZoo6GWVu1fQK5BTkeRTXw2RwPSFK3iRhe=JCNNsi6XywbdJQ@mail.gmail.com>

I'm curious about this statement. Can you share some examples?


On Wed, Aug 28, 2013 at 8:24 PM, Seth Hall <seth at icir.org> wrote:

> On Aug 28, 2013, at 8:04 PM, Chris Roose <chrisroose at fastmail.fm> wrote:
>
> > Sure... any particular details you'd like to see?
>
>
> Just a few full connection records for tcp traffic.  There is typically a
> lot of detail you can tease out if you know what you're looking for. :)
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 
cat ~/.bash_history > documentation.txt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130828/9f9b6a92/attachment.html 

From nicolas.retrain at cea.fr  Thu Aug 29 00:39:51 2013
From: nicolas.retrain at cea.fr (nicolas.retrain at cea.fr)
Date: Thu, 29 Aug 2013 09:39:51 +0200
Subject: [Bro] @load missing in the init-bare.bro
In-Reply-To: <B3E27C90B7D2DD4CAEF09215A29EB0D1170A48E9@CITESMBX2.ad.uillinois.edu>
References: <521DEB08.3020508@cea.fr>
	<B3E27C90B7D2DD4CAEF09215A29EB0D1170A48E9@CITESMBX2.ad.uillinois.edu>
Message-ID: <521EFAC7.5090009@cea.fr>

Le 28/08/2013 18:05, Siwek, Jonathan Luke a ?crit :
>> "internal error in [mypath]/bro/base/init-bar.bro, line 3098:internal
>> type Files::AnalyzerArgs missing
>> Abandon"
>>
>> Is there a @load missing in the init-bare.bro file?
> I don't think so, or at least I'm not seeing the same thing.  Can you give more details on how you're running bro and any local changes/scripts or altering of BROPATH environment variable?
>
> - Jon
After testing it again, it works ! :)
However, I still have issues and weird errors when I try to run my own 
previous module.. debug time !

Thanks


From jessebort at hushmail.com  Thu Aug 29 18:09:00 2013
From: jessebort at hushmail.com (jessebort at hushmail.com)
Date: Thu, 29 Aug 2013 21:09:00 -0400
Subject: [Bro] broctl worker-1 cluster problem
In-Reply-To: <mailman.0.1377824773.8069.bro@bro.org>
Message-ID: <20130830010900.DC6CDC0562@smtp.hushmail.com>

I'm a new bro user and  have tried to find the answer to this, but
have had no luck.  I've got  version 2.1 installed.  I can run bro in
standalone mode with no  problem, but I've tried to install a bro
cluster with worker-1 on a  remote host/VM with the same problem. 
Here is what I've tried to do:

created user jesse on both manager/proxy - 192.168.43.1
   o configured node.cfg for manager and proxy to be 192.168.43.1
   o configured node.cfg for worker-1 to be 192.168.43.130
   o performed ssh-keygen as user jesse
   o copied .ssh/rsa_id.pub to 192.168.43.130
/home/jesse/.ssh/authorized_keys
   o able to ssh as jesse from 192.168.43.1 to 192.168.43.130 with no
required password/passphrase
   o added jesse to /etc/sudoers to do everything root can

created user jesse on worker-1 192.168.43.130 (VM)
   o changed owner of /usr/local/bro to jesse
   o added jesse to /etc/sudoers to do everything root can

as user jesse on manager/proxy > sudo broctl
[BroControl] > install
removing old policies in
/usr/local/bro/spool/installed-scripts-do-not-touch/site ... done.
removing old policies in
/usr/local/bro/spool/installed-scripts-do-not-touch/auto ... done.
creating policy directories ... done.
installing site policies ... done.
generating cluster-layout.bro ... done.
generating local-networks.bro ... done.
generating broctl-config.bro ... done.
updating nodes ... warning: host 192.168.43.130 is not alive   install
waiting for lock ..... ok
removing old policies in
/usr/local/bro/spool/installed-scripts-do-not-touch/site ... done.
removing old policies in
/usr/local/bro/spool/installed-scripts-do-not-touch/auto ... done.
creating policy directories ... done.
installing site policies ... done.
generating cluster-layout.bro ... done.
generating local-networks.bro ... done.
generating broctl-config.bro ... done.
updating nodes ... done.                                              
    diag worker-1
[worker-1]
No work dir found
[BroControl] > start
starting manager ...
starting proxy-1 ...
starting worker-1 ...
cannot create working directory for worker-1                         
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130829/2882fb72/attachment.html 

From init.conf at gmail.com  Thu Aug 29 18:31:35 2013
From: init.conf at gmail.com (Aashish Sharma)
Date: Thu, 29 Aug 2013 18:31:35 -0700
Subject: [Bro] broctl worker-1 cluster problem
In-Reply-To: <20130830010900.DC6CDC0562@smtp.hushmail.com>
References: <20130830010900.DC6CDC0562@smtp.hushmail.com>
Message-ID: <2B8520B2-C6BC-4EC0-AB0B-15D22E3B0624@gmail.com>

I think when manager is installing policies/scripts etc on worker nodes, bro ssh'es to the worker nodes as user "bro" and not  as user "jesse"

That's where you are seeing problem. 

Aashish 

On Aug 29, 2013, at 6:09 PM, jessebort at hushmail.com wrote:

> I'm a new bro user and have tried to find the answer to this, but have had no luck. I've got version 2.1 installed. I can run bro in standalone mode with no problem, but I've tried to install a bro cluster with worker-1 on a remote host/VM with the same problem. Here is what I've tried to do:
> 
> created user jesse on both manager/proxy - 192.168.43.1
>    o configured node.cfg for manager and proxy to be 192.168.43.1
>    o configured node.cfg for worker-1 to be 192.168.43.130
>    o performed ssh-keygen as user jesse
>    o copied .ssh/rsa_id.pub to 192.168.43.130 /home/jesse/.ssh/authorized_keys
>    o able to ssh as jesse from 192.168.43.1 to 192.168.43.130 with no required password/passphrase
>    o added jesse to /etc/sudoers to do everything root can
> 
> created user jesse on worker-1 192.168.43.130 (VM)
>    o changed owner of /usr/local/bro to jesse
>    o added jesse to /etc/sudoers to do everything root can
> 
> as user jesse on manager/proxy > sudo broctl
>                                 
> [BroControl] > install
> removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/site ... done.
> removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/auto ... done.
> creating policy directories ... done.
> installing site policies ... done.
> generating cluster-layout.bro ... done.
> generating local-networks.bro ... done.
> generating broctl-config.bro ... done.
> updating nodes ... warning: host 192.168.43.130 is not alive <== Not sure why I got this
> done.
> [BroControl] > install
> waiting for lock ..... ok
> removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/site ... done.
> removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/auto ... done.
> creating policy directories ... done.
> installing site policies ... done.
> generating cluster-layout.bro ... done.
> generating local-networks.bro ... done.
> generating broctl-config.bro ... done.
> updating nodes ... done.                                                  <== Able to find 192.168.43.130 next time
> [BroControl] > diag worker-1
> [worker-1]
> No work dir found
> [BroControl] > start
> starting manager ...
> starting proxy-1 ...
> starting worker-1 ...
> cannot create working directory for worker-1                         <== Issue
> cannot create working directory for [(<BroControl.node.Node instance at 0x18c14d0>, '/usr/local/bro/spool/worker-1')] <== tried to put a debug statement in control.py to see the actual directory it was having issues with 
> 
> I see nothing installed in /usr/local/bro/spool on worker-1
> 
> Thanks for any help you can give me
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130829/47b7ec8b/attachment.html 

From phatbuckett at gmail.com  Sat Aug 31 13:42:22 2013
From: phatbuckett at gmail.com (Darren Spruell)
Date: Sat, 31 Aug 2013 13:42:22 -0700
Subject: [Bro] 404 in INSTALL.html -> quickstart
Message-ID: <CAKVSOJWPjfE1s_SvVjWY8HyG-pszfGsdpfLEKRMTxnfXBroWtw@mail.gmail.com>

Noticed http://www.bro.org/sphinx/INSTALL.html has a broken link to
Quick Start Guide.

X http://www.bro.org/documentation/quickstart.html

O http://www.bro.org/sphinx/quickstart.html

-- 
Darren Spruell
phatbuckett at gmail.com