From seth at corelight.com  Thu Nov  1 06:46:22 2018
From: seth at corelight.com (Seth Hall)
Date: Thu, 01 Nov 2018 09:46:22 -0400
Subject: [Bro] Bro/Zeek project leadership update - welcome,
	Keith Lehigh!
In-Reply-To: <201810292010.w9TKAOZu025710@fruitcake.ICSI.Berkeley.EDU>
References: <201810292010.w9TKAOZu025710@fruitcake.ICSI.Berkeley.EDU>
Message-ID: <AFBD0A66-EF7A-484C-B6CC-E4FB5BB18911@corelight.com>

I just wanted to say thanks to Keith for taking on this role.  I have 
high hopes for the upcoming time!

Also, just echoing Vern, thanks Adam for all of the time you've spent 
the past few years in the role!

   .Seth

On 29 Oct 2018, at 16:10, Vern Paxson wrote:

> Adam Slagell decided earlier this year that it was time to step down 
> from
> his service as the chair of the Bro/Zeek project's Leadership Team[*].
> (He'll be continuing as a member of the LT.)  The LT has 
> enthusiastically
> selected Keith Lehigh of Indiana University to serve as its new chair.
> Many of you will know Keith from his service as this year's program 
> committee
> chair of BroCon 2018.
>
> Our thanks to Adam for his industrious and invaluable service!
>
> 		Vern
>
>
> [*] https://www.bro.org/team.html
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

--
Seth Hall * Corelight, Inc * www.corelight.com

From matt.thoreson at summitinfosec.com  Thu Nov  1 07:45:47 2018
From: matt.thoreson at summitinfosec.com (Matt Thoreson)
Date: Thu, 1 Nov 2018 07:45:47 -0700
Subject: [Bro] Bro decapsulating ERSPAN (GRE)
In-Reply-To: <CAMzgZ0++PDRDfR5aYVxqwB0BcaCo=Wun6=b2dweo0Mo-pTSYqA@mail.gmail.com>
References: <CAKM8Hq21sGefmL+guVVaO7Dm65ME89bzSZwD2Bn1e+8nL4Yz+g@mail.gmail.com>
	<CAMzgZ0Kb0LhoD1w85m4qkU3L2YqzcTNnu5vCtDZJ0W11aB+YPQ@mail.gmail.com>
	<CAMzgZ0++PDRDfR5aYVxqwB0BcaCo=Wun6=b2dweo0Mo-pTSYqA@mail.gmail.com>
Message-ID: <CAKM8Hq2mWrBXAozUjWUsAEhycnJ+O4PtweE+_eEHnDXtnMc-=w@mail.gmail.com>

For what it's worth I tried to stripping 50 bytes off the header in the
init-bar.bro file in the encap_hdr_size=50 line.  That seems to be the
magic number with this unusual erspan gre header size.  After doing that,
bro is recognizing and splitting all the logs out properly.

On Wed, Oct 31, 2018 at 2:32 PM Jon Siwek <jsiwek at corelight.com> wrote:

> On Wed, Oct 31, 2018 at 1:07 PM Jon Siwek <jsiwek at corelight.com> wrote:
> >
> > On Wed, Oct 31, 2018 at 12:40 PM Matt Thoreson
> > <matt.thoreson at summitinfosec.com> wrote:
> >
> > > I thought Bro could by default recognize and decapsulate the real
> traffic from the GRE tunnel (according to the bro notes it should be able
> to do this) but so far when bro runs it just sees the gre traffic in it's
> weird.log.
> >
> > It currently only handles a few GRE protocol types, and doesn't seem
> > the ERSPAN ones are among them.
>
> To clarify that further: I totally missed that the changelog does say
> ERSPAN support was implemented, but I was just looking at the actual
> code, which does not seem to handle ERSPAN Type II or III (protocol
> types 0x88BE, 0x22EB).  The associated commit seems to instead handle
> Transparent Ethernet Bridging (protocol type 0x6558).  Not sure if I'm
> missing something.  Or if you can give a pcap to test against, that
> could help to verify what's going and also serve as test case for
> fixing anything that's broken/unimplemented in Bro.
>
> - Jon
>


-- 

Matt Thoreson, CISSP, OSCP
*Security Engineer | Summit Security Group, LLC*
m: 360.787.8998
w: http://summitinfosec.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181101/8aa10115/attachment.html 

From turbidtarantula at gmail.com  Fri Nov  2 09:51:35 2018
From: turbidtarantula at gmail.com (Mike M)
Date: Fri, 2 Nov 2018 12:51:35 -0400
Subject: [Bro] Getting a Broctl Stack Trace
Message-ID: <CAK051agH+hWgr-bZrfqVBQEKHZrU9AkCsb=CwFx4ohUp+YfkuA@mail.gmail.com>

I'm having an issue with broctl crashing when I try to run it on Alpine
Linux. I mentioned it previously [1] but I'm circling back around to try to
get it resolved. I've built it with the appropriate patches [2] but broctl
is still reporting "crashed" state when I checks the status after starting
it. The bro binary itself runs fine.

What do I need to do to collect a stack trace from broctl to determine the
root cause?

Bro is built in debug mode and I set "ulimit -c unlimited" per the
instructions on reporting problems. I see a
/usr/local/bro/spool/tmp/post-terminate-standalone-2018-11-02-02-56-06-13765-crash
directory but there's no core dump anywhere obvious. The .crash-diag.out
file says "No core file found" and doesn't provide any useful information
about the cause of the crash.

Thanks,
Mike

[1]
http://mailman.icsi.berkeley.edu/pipermail/bro/2018-September/013580.html
[2]
http://mailman.icsi.berkeley.edu/pipermail/bro/2018-September/013581.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181102/2d495654/attachment.html 

From ambros.novak.89 at gmail.com  Mon Nov  5 02:07:18 2018
From: ambros.novak.89 at gmail.com (Ambros Novak)
Date: Mon, 5 Nov 2018 05:07:18 -0500
Subject: [Bro] large notice.log
In-Reply-To: <25399D6E-AF3A-4D7F-B7E7-2AFBA00E2020@corelight.com>
References: <CAHifxyCGfhp1Sne9=HbLVNVU0CiVvrGZ9SFs7ukLGtvyC17ewg@mail.gmail.com>
	<25399D6E-AF3A-4D7F-B7E7-2AFBA00E2020@corelight.com>
Message-ID: <CAHifxyCVxvU0Zg1gpO3KhunRa=mafQ0AjaCj+VNZthOEx8FW+Q@mail.gmail.com>

Thank you Stephen and Seth!

I am working still troubleshooting this issue, but I will update once it is
resolved.

Cheers!

On Tue, Oct 30, 2018 at 12:14 PM Seth Hall <seth at corelight.com> wrote:

> On 29 Oct 2018, at 19:12, Ambros Novak wrote:
>
> > notice.log is extremely large before it rotates, sometimes 140G+.
>
> Wow! What notices do you have? It sounds like you may have a notice that
> is getting out of control and it might make more sense figuring out
> what's going on rather than just trying to muffle the creation of these
> notices.
>
>    .Seth
>
> --
> Seth Hall * Corelight, Inc * www.corelight.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181105/a0334cc5/attachment.html 

From ambros.novak.89 at gmail.com  Mon Nov  5 02:12:06 2018
From: ambros.novak.89 at gmail.com (Ambros Novak)
Date: Mon, 5 Nov 2018 05:12:06 -0500
Subject: [Bro] proper install directory - /opt/bro or /usr/local/bro?
Message-ID: <CAHifxyDGnSwXHWmQkM4Jfm4LAVz9BugDijY=ay_NCUKDggXdTg@mail.gmail.com>

Heya listserv,

The default install directory for Bro is /usr/local/bro but I know there is
a lot of documentation using /opt/bro. Which is the proper directory to
install bro/zeek into?

And if the proper install directory is /opt/bro, then why is the default
/usr/local/bro?

Thank you in advance!

Cheers!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181105/91367823/attachment.html 

From jsiwek at corelight.com  Mon Nov  5 07:55:04 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Mon, 5 Nov 2018 09:55:04 -0600
Subject: [Bro] proper install directory - /opt/bro or /usr/local/bro?
In-Reply-To: <CAHifxyDGnSwXHWmQkM4Jfm4LAVz9BugDijY=ay_NCUKDggXdTg@mail.gmail.com>
References: <CAHifxyDGnSwXHWmQkM4Jfm4LAVz9BugDijY=ay_NCUKDggXdTg@mail.gmail.com>
Message-ID: <CAMzgZ0L3FJ265OFwcwG=o7b2DtO5=w1ak5PqKKGohNhLS-Grcg@mail.gmail.com>

On Mon, Nov 5, 2018 at 4:20 AM Ambros Novak <ambros.novak.89 at gmail.com> wrote:

> The default install directory for Bro is /usr/local/bro but I know there is a lot of documentation using /opt/bro. Which is the proper directory to install bro/zeek into?
>
> And if the proper install directory is /opt/bro, then why is the default /usr/local/bro?

Default when building/installing from source code: /usr/local/bro
Default when installing from pre-built binary package (from bro.org): /opt/bro

The difference is conventional and possibly also suggested by the
Filesystem Hierarchy Standard (FHS).  That is, /usr/local is usually
for software a sysadmin explicitly builds/installs/manages themselves,
while /opt is for pre-packaged software that is installed from a
source that's external to the operating system's default package
repositories.

Ultimately, there's no single install path to call proper, just
different defaults that try to conform to expected conventions and
standards.  Bro still functions the same for any arbitrary install
path, so feel free to change to your preference.

- Jon

From robin at corelight.com  Mon Nov  5 11:41:52 2018
From: robin at corelight.com (Robin Sommer)
Date: Mon, 5 Nov 2018 11:41:52 -0800
Subject: [Bro] Job posting: Director of Community
Message-ID: <20181105194152.GH80620@corelight.com>

Corelight is looking for a Director of Community to lead, coordinate,
and manage our global open-source community program. Please see
https://www.corelight.com/company/careers/1425439 for more
information.

Robin

-- 
Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com

From seth at corelight.com  Mon Nov  5 14:01:19 2018
From: seth at corelight.com (Seth Hall)
Date: Mon, 05 Nov 2018 17:01:19 -0500
Subject: [Bro] Getting a Broctl Stack Trace
In-Reply-To: <CAK051agH+hWgr-bZrfqVBQEKHZrU9AkCsb=CwFx4ohUp+YfkuA@mail.gmail.com>
References: <CAK051agH+hWgr-bZrfqVBQEKHZrU9AkCsb=CwFx4ohUp+YfkuA@mail.gmail.com>
Message-ID: <AB6568DA-7AD5-4D84-B1AB-AD00877F7114@corelight.com>

Make sure you are setting the core pattern on your system so that the 
core dump will be written into the CWD.

sudo sysctl -w kernel.core_pattern="core.%e-%t-%p"

   .Seth

On 2 Nov 2018, at 12:51, Mike M wrote:

> I'm having an issue with broctl crashing when I try to run it on 
> Alpine
> Linux. I mentioned it previously [1] but I'm circling back around to 
> try to
> get it resolved. I've built it with the appropriate patches [2] but 
> broctl
> is still reporting "crashed" state when I checks the status after 
> starting
> it. The bro binary itself runs fine.
>
> What do I need to do to collect a stack trace from broctl to determine 
> the
> root cause?
>
> Bro is built in debug mode and I set "ulimit -c unlimited" per the
> instructions on reporting problems. I see a
> /usr/local/bro/spool/tmp/post-terminate-standalone-2018-11-02-02-56-06-13765-crash
> directory but there's no core dump anywhere obvious. The 
> .crash-diag.out
> file says "No core file found" and doesn't provide any useful 
> information
> about the cause of the crash.
>
> Thanks,
> Mike
>
> [1]
> http://mailman.icsi.berkeley.edu/pipermail/bro/2018-September/013580.html
> [2]
> http://mailman.icsi.berkeley.edu/pipermail/bro/2018-September/013581.html
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

--
Seth Hall * Corelight, Inc * www.corelight.com


From johanna at icir.org  Tue Nov  6 07:36:46 2018
From: johanna at icir.org (Johanna Amann)
Date: Tue, 6 Nov 2018 22:36:46 +0700
Subject: [Bro] Zeek (Bro) Workshop Europe 2019 - Registration open
Message-ID: <20181106153645.zvqilb4qwktci673@dhcp-9e80.meeting.ietf.org>

Hi,

the registration for the Zeek Workshop Europe 2019 (April 9?11 @CERN,
Geneva, Switzerland) is now open.

To register, please visit https://indico.cern.ch/event/762505/ (this is also
linked from https://bro.org).

The workshop is a two-day event, split over three days to allow easy
traveling for participants in Europe: the program will start with lunch on
Tuesday and will end after lunch on Thursday.

The program will consist of talks by the Bro development team and external
contributors. As in our last event, a large part of the development team
will be attending the workshop. We will send followup messages to
bro at bro.org once a more detailed program is available.

If you are interested in giving a talk at the Workshop, please send an
email to info at bro.org.

Johanna

From nothinrandom at gmail.com  Tue Nov  6 16:01:17 2018
From: nothinrandom at gmail.com (TQ)
Date: Tue, 6 Nov 2018 16:01:17 -0800
Subject: [Bro] Listening on both UDP/TCP
Message-ID: <CAM2MKSpnSx+602o=3AGEBhNNPbaVqmvt_1jS9n3N+6GDL=ewtA@mail.gmail.com>

Hello There,

I see many of the existing protocols focus on either TCP or UDP, but
nothing for both.  I did notice that SIP has both TCP and UDP, however, the
TCP portion is "not activated" (
https://github.com/bro/bro/tree/master/src/analyzer/protocol/sip).  Is
there a good example of how to handle both?  Is this something where I
would need register listener in main.bro? For example:

const ports = { 5060/udp }; # existing
const ports_tcp = { 1234/tcp }; # new
redef likely_server_ports += { ports, ports_tcp };

event bro_init() &priority=5 {

Log::create_stream(SIP::LOG, [$columns=Info, $ev=log_sip, $path="sip"]);
Analyzer::register_for_ports(Analyzer::ANALYZER_SIP, ports); # existing
Analyzer_TCP::register_for_ports(Analyzer_TCP::ANALYZER_SIP_TCP,
ports_tcp); # new

}

Thanks,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181106/b82300c0/attachment.html 

From johanna at icir.org  Tue Nov  6 17:11:10 2018
From: johanna at icir.org (Johanna Amann)
Date: Wed, 7 Nov 2018 08:11:10 +0700
Subject: [Bro] Listening on both UDP/TCP
In-Reply-To: <CAM2MKSpnSx+602o=3AGEBhNNPbaVqmvt_1jS9n3N+6GDL=ewtA@mail.gmail.com>
References: <CAM2MKSpnSx+602o=3AGEBhNNPbaVqmvt_1jS9n3N+6GDL=ewtA@mail.gmail.com>
Message-ID: <20181107011110.k74knswhbs3zifuw@dhcp-9e80.meeting.ietf.org>

Hi,

> I see many of the existing protocols focus on either TCP or UDP, but
> nothing for both.  I did notice that SIP has both TCP and UDP, however, the
> TCP portion is "not activated" (
> https://github.com/bro/bro/tree/master/src/analyzer/protocol/sip).  Is
> there a good example of how to handle both?  Is this something where I
> would need register listener in main.bro? For example:

[...]

the closest to this is probably the TLS/DTLS analyzer. Similarly to SIP,
it actually is 2 analyzers (one for TLS over TCP and one for DTLS over
UDP) that share a lot of the code.

scripts/base/protocols/ssl/main.bro shows that both of them are just
initialized separately from each other. From a very cursory glance over
SIP, I think that one could just do the same there.

I hope this helps,
 Johanna

From wangdj at ffcs.cn  Tue Nov  6 18:20:02 2018
From: wangdj at ffcs.cn (wangdj at ffcs.cn)
Date: Wed, 7 Nov 2018 10:20:02 +0800
Subject: [Bro] Is there any document that can helps with read code
Message-ID: <2018110710200199783881@ffcs.cn>

Hello?

I want to read Bro code,  is there any guidance document which can give me some help.

Best Regards
DeJin Wang


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181107/65953e02/attachment.html 

From neslog at gmail.com  Wed Nov  7 13:24:23 2018
From: neslog at gmail.com (Neslog)
Date: Wed, 7 Nov 2018 16:24:23 -0500
Subject: [Bro] broker python receive event
Message-ID: <CALgfRGLSrnnWi3DVHukuAX3OnHRvGbwEgb+03GfdrEBb951gLg@mail.gmail.com>

Does anyone have an example of Bro sending events to a python script using
python bindings?  I'd like to generate an event in Bro and have the python
script listen and kick off a script.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181107/c1945cba/attachment.html 

From vallentin at icir.org  Wed Nov  7 16:47:01 2018
From: vallentin at icir.org (Matthias Vallentin)
Date: Wed, 7 Nov 2018 16:47:01 -0800
Subject: [Bro] broker python receive event
In-Reply-To: <CALgfRGLSrnnWi3DVHukuAX3OnHRvGbwEgb+03GfdrEBb951gLg@mail.gmail.com>
References: <CALgfRGLSrnnWi3DVHukuAX3OnHRvGbwEgb+03GfdrEBb951gLg@mail.gmail.com>
Message-ID: <CADTpMNaga3uJOQB3hE7Oyt56zeqqYq9fP9+Mr8Z69dN5-SqOWA@mail.gmail.com>

You can have a look at our BroCon 2018 material, which contain Broker
examples for C++, Python, and Bro:
https://github.com/tenzir/events/tree/master/brocon18.

    Matthias


On Wed, Nov 7, 2018 at 1:33 PM Neslog <neslog at gmail.com> wrote:
>
> Does anyone have an example of Bro sending events to a python script using python bindings?  I'd like to generate an event in Bro and have the python script listen and kick off a script.
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

From tbarbatsalou at gmail.com  Fri Nov  9 01:04:22 2018
From: tbarbatsalou at gmail.com (Tina Barbatsalou)
Date: Fri, 9 Nov 2018 11:04:22 +0200
Subject: [Bro] Script conversion to 2.6.2
Message-ID: <CAFxUBkDHOzd0+HpE06FCa5kh_vp-FdaC9qMpFCapEY7uXBp0fw@mail.gmail.com>

Hello everyone,

I am trying to convert a chunk of bro scripting code to the new version,
but, despite reading the documentation, I don't know what to precisely
replace.

event dhcp_ack(c: connection, msg: dhcp_msg, mask: addr, router:
dhcp_router_list, lease: interval, serv_addr: addr)
      {
          # Store info from the DHCP acknowledgment, to create a mapping
between SHA and assigned IP
          DHCP_state[dhcp_msg$h_addr] = dhcp_msg$yiaddr;
      }

Apparently, the dhcp_ack event has to be replaced by the dhcp_message
equivalent, with a syntax similar to (?) the following:
event dhcp_message(c: connection, is_orig: bool, msg: DHCP::Msg, options:
DHCP::Options).

I am not sure if it is correct and what I should include in the DHCP::Msg
and DHCP::Options parts in order to construct an ack.
Moreover, by what should the dhcp_msg be replaced in the following
function? (DHCP_state[dhcp_msg$h_addr] = dhcp_msg$yiaddr;)

Excuse my ignorance; These are my first bro tryouts.
Best regards,
TB
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181109/b2234407/attachment.html 

From michalpurzynski1 at gmail.com  Fri Nov  9 01:56:53 2018
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Fri, 9 Nov 2018 01:56:53 -0800
Subject: [Bro] Script conversion to 2.6.2
In-Reply-To: <CAFxUBkDHOzd0+HpE06FCa5kh_vp-FdaC9qMpFCapEY7uXBp0fw@mail.gmail.com>
References: <CAFxUBkDHOzd0+HpE06FCa5kh_vp-FdaC9qMpFCapEY7uXBp0fw@mail.gmail.com>
Message-ID: <CAJ6bFK0e=9uf4oTowEdOW9dMeQNG99aOYbEkDh0=1ZHauKHr2A@mail.gmail.com>

While not quite providing the answer to your question, this might help a bit.

https://github.com/bro/bro/blob/master/NEWS

It tells me that there is a script
"policy/protocols/dhcp/deprecated_events.bro" that can bring back your
old events back from the new dhcp_message() only.

You might take a look at what it does and how it constructs the
dhcp_ack from the dhcp_message()

It takes the dhcp_message(c: connection, is_orig: bool, msg:
DHCP::Msg, options: DHCP::Options)

checks for the type of the DHCP message

switch ( DHCP::message_types[msg$m_type] )
case "ACK":

calls an artificially built event - event dhcp_ack(c, old_msg, sm,
routers, le, sa, hn)

This should get you started. Welcome to the community, please come
back and ask more questions.


On Fri, Nov 9, 2018 at 1:06 AM Tina Barbatsalou <tbarbatsalou at gmail.com> wrote:
>
> Hello everyone,
>
> I am trying to convert a chunk of bro scripting code to the new version, but, despite reading the documentation, I don't know what to precisely replace.
>
> event dhcp_ack(c: connection, msg: dhcp_msg, mask: addr, router: dhcp_router_list, lease: interval, serv_addr: addr)
>       {
>           # Store info from the DHCP acknowledgment, to create a mapping between SHA and assigned IP
>           DHCP_state[dhcp_msg$h_addr] = dhcp_msg$yiaddr;
>       }
>
> Apparently, the dhcp_ack event has to be replaced by the dhcp_message equivalent, with a syntax similar to (?) the following:
> event dhcp_message(c: connection, is_orig: bool, msg: DHCP::Msg, options: DHCP::Options).
>
> I am not sure if it is correct and what I should include in the DHCP::Msg and DHCP::Options parts in order to construct an ack.
> Moreover, by what should the dhcp_msg be replaced in the following function? (DHCP_state[dhcp_msg$h_addr] = dhcp_msg$yiaddr;)
>
> Excuse my ignorance; These are my first bro tryouts.
> Best regards,
> TB
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

From tbarbatsalou at gmail.com  Fri Nov  9 06:50:49 2018
From: tbarbatsalou at gmail.com (Tina Barbatsalou)
Date: Fri, 9 Nov 2018 16:50:49 +0200
Subject: [Bro] Script conversion to 2.6.2
In-Reply-To: <CAJ6bFK0e=9uf4oTowEdOW9dMeQNG99aOYbEkDh0=1ZHauKHr2A@mail.gmail.com>
References: <CAFxUBkDHOzd0+HpE06FCa5kh_vp-FdaC9qMpFCapEY7uXBp0fw@mail.gmail.com>
	<CAJ6bFK0e=9uf4oTowEdOW9dMeQNG99aOYbEkDh0=1ZHauKHr2A@mail.gmail.com>
Message-ID: <CAFxUBkCq4PNJBQLP=1ecFvpkv2A4U-BVsFE0aGn4iEgeYsuRjw@mail.gmail.com>

I finally managed to address the issue, by replacing the equivalent values
for the following:

event dhcp_message(c: connection, is_orig: bool, msg: DHCP::Msg, options:
DHCP::Options)
       {# Store info from the DHCP acknowledgment, to create a mapping
between SHA and assigned IP
           DHCP_state[msg$chaddr] = msg$yiaddr;
      }

and I was able to replicate an arp poisoning attack and get it detected in
the bro -C -i <interface> <path_to_script> mode.

However, when i integrated the script in the broctl infrastructure, it
didn't detect the attack, by producing the equivalent log file.

I have configured the local.bro and respective configuration files
correctly but the attack is not getting detected and no arp spoofing log
file is generated.

On Fri, Nov 9, 2018 at 11:57 AM Micha? Purzy?ski <michalpurzynski1 at gmail.com>
wrote:

> While not quite providing the answer to your question, this might help a
> bit.
>
> https://github.com/bro/bro/blob/master/NEWS
>
> It tells me that there is a script
> "policy/protocols/dhcp/deprecated_events.bro" that can bring back your
> old events back from the new dhcp_message() only.
>
> You might take a look at what it does and how it constructs the
> dhcp_ack from the dhcp_message()
>
> It takes the dhcp_message(c: connection, is_orig: bool, msg:
> DHCP::Msg, options: DHCP::Options)
>
> checks for the type of the DHCP message
>
> switch ( DHCP::message_types[msg$m_type] )
> case "ACK":
>
> calls an artificially built event - event dhcp_ack(c, old_msg, sm,
> routers, le, sa, hn)
>
> This should get you started. Welcome to the community, please come
> back and ask more questions.
>
>
> On Fri, Nov 9, 2018 at 1:06 AM Tina Barbatsalou <tbarbatsalou at gmail.com>
> wrote:
> >
> > Hello everyone,
> >
> > I am trying to convert a chunk of bro scripting code to the new version,
> but, despite reading the documentation, I don't know what to precisely
> replace.
> >
> > event dhcp_ack(c: connection, msg: dhcp_msg, mask: addr, router:
> dhcp_router_list, lease: interval, serv_addr: addr)
> >       {
> >           # Store info from the DHCP acknowledgment, to create a mapping
> between SHA and assigned IP
> >           DHCP_state[dhcp_msg$h_addr] = dhcp_msg$yiaddr;
> >       }
> >
> > Apparently, the dhcp_ack event has to be replaced by the dhcp_message
> equivalent, with a syntax similar to (?) the following:
> > event dhcp_message(c: connection, is_orig: bool, msg: DHCP::Msg,
> options: DHCP::Options).
> >
> > I am not sure if it is correct and what I should include in the
> DHCP::Msg and DHCP::Options parts in order to construct an ack.
> > Moreover, by what should the dhcp_msg be replaced in the following
> function? (DHCP_state[dhcp_msg$h_addr] = dhcp_msg$yiaddr;)
> >
> > Excuse my ignorance; These are my first bro tryouts.
> > Best regards,
> > TB
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181109/18a23188/attachment.html 

From jlay at slave-tothe-box.net  Mon Nov 12 11:07:30 2018
From: jlay at slave-tothe-box.net (James Lay)
Date: Mon, 12 Nov 2018 12:07:30 -0700
Subject: [Bro] Issues since 2.5.5
Message-ID: <a073ce2a8d4e70dd5ac015ec189ab51d@slave-tothe-box.net>

Well..I think I'll also put my name in the "something is funky with 
2.5.5" group.  I have seen far more crashes and OOM's with 2.5.5 than 
with 2.5.4.  Case in point just now:

[425271.774232] bro invoked oom-killer: 
gfp_mask=0x14280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=0, 
order=0, oom_score_adj=0
[425271.774234] bro cpuset=/ mems_allowed=0
[425271.774239] CPU: 0 PID: 2482 Comm: bro Tainted: G          I     
4.10.0-35-generic #39~16.04.1-Ubuntu
[425271.774240] Hardware name: Dell Inc. PowerEdge
[425271.774241] Call Trace:
[425271.774249]  dump_stack+0x63/0x90
[425271.774253]  dump_header+0x7b/0x1fd
[425271.774257]  ? security_capable_noaudit+0x45/0x60
[425271.774261]  oom_kill_process+0x219/0x3e0
[425271.774264]  out_of_memory+0x120/0x4b0
[425271.774267]  __alloc_pages_slowpath+0x9ea/0xb30
[425271.774270]  __alloc_pages_nodemask+0x21a/0x2a0
[425271.774272]  alloc_pages_vma+0xa2/0x270
[425271.774277]  handle_mm_fault+0xdbc/0x1270
[425271.774282]  __do_page_fault+0x240/0x4e0
[425271.774285]  do_page_fault+0x22/0x30
[425271.774289]  page_fault+0x28/0x30
[425271.774291] RIP: 0033:0x7fb47bef3786
[425271.774292] RSP: 002b:00007fff751e5330 EFLAGS: 00010206
[425271.774294] RAX: 000000000000ffe1 RBX: 00007fb46c000020 RCX: 
0000000000000065
[425271.774295] RDX: 00007fb46d670fc0 RSI: 00007fb46d671020 RDI: 
0000000000000000
[425271.774296] RBP: 0000000000000061 R08: 0000000000000000 R09: 
0000000000000000
[425271.774297] R10: 0000000000000001 R11: 00007fb46d6521b0 R12: 
00007fb46c000078
[425271.774298] R13: 00007fb46c000078 R14: 0000000000002710 R15: 
00007fb46c0000c8
[425271.774300] Mem-Info:
[425271.774305] active_anon:5547944 inactive_anon:396201 isolated_anon:0
                  active_file:1269 inactive_file:726 isolated_file:32
                  unevictable:0 dirty:0 writeback:83 unstable:0
                  slab_reclaimable:5205 slab_unreclaimable:6731
                  mapped:133175 shmem:1921 pagetables:17900 bounce:0
                  free:41669 free_pcp:583 free_cma:0
[425271.774310] Node 0 active_anon:22191776kB inactive_anon:1584804kB 
active_file:5076kB inactive_file:2904kB unevictable:0kB 
isolated(anon):0kB isolated(file):128kB mapped:532700kB dirty:0kB 
writeback:332kB shmem:7684kB shmem_thp: 0kB shmem_pmdmapped: 0kB 
anon_thp: 8192kB writeback_tmp:0kB unstable:0kB pages_scanned:17295 
all_unreclaimable? yes
[425271.774311] Node 0 DMA free:15896kB min:40kB low:52kB high:64kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB writepending:0kB present:15996kB managed:15904kB 
mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB 
pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[425271.774316] lowmem_reserve[]: 0 3178 24041 24041 24041
[425271.774320] Node 0 DMA32 free:92204kB min:8928kB low:12180kB 
high:15432kB active_anon:2645908kB inactive_anon:543392kB 
active_file:580kB inactive_file:1504kB unevictable:0kB writepending:0kB 
present:3378660kB managed:3296088kB mlocked:0kB slab_reclaimable:596kB 
slab_unreclaimable:500kB kernel_stack:16kB pagetables:7936kB bounce:0kB 
free_pcp:1188kB local_pcp:272kB free_cma:0kB
[425271.774325] lowmem_reserve[]: 0 0 20862 20862 20862
[425271.774328] Node 0 Normal free:58576kB min:58608kB low:79968kB 
high:101328kB active_anon:19546072kB inactive_anon:1041212kB 
active_file:4508kB inactive_file:1316kB unevictable:0kB writepending:4kB 
present:21757952kB managed:21363532kB mlocked:0kB 
slab_reclaimable:20224kB slab_unreclaimable:26416kB kernel_stack:2736kB 
pagetables:63664kB bounce:0kB free_pcp:1144kB local_pcp:240kB 
free_cma:0kB
[425271.774334] lowmem_reserve[]: 0 0 0 0 0
[425271.774337] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 
1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 
15896kB
[425271.774349] Node 0 DMA32: 129*4kB (UE) 97*8kB (UME) 60*16kB (UE) 
94*32kB (UME) 34*64kB (UME) 24*128kB (UME) 31*256kB (UME) 18*512kB (UE) 
11*1024kB (UE) 8*2048kB (UME) 9*4096kB (UME) = 92172kB
[425271.774363] Node 0 Normal: 602*4kB (UMEH) 761*8kB (UMEH) 480*16kB 
(UMEH) 976*32kB (UMEH) 125*64kB (UMEH) 20*128kB (UMH) 2*256kB (UH) 
0*512kB 0*1024kB 0*2048kB 0*4096kB = 58480kB
[425271.774375] Node 0 hugepages_total=0 hugepages_free=0 
hugepages_surp=0 hugepages_size=1048576kB
[425271.774377] Node 0 hugepages_total=0 hugepages_free=0 
hugepages_surp=0 hugepages_size=2048kB
[425271.774378] 6037 total pagecache pages
[425271.774379] 2026 pages in swap cache
[425271.774380] Swap cache stats: add 19065590, delete 19063564, find 
8257735/16409127
[425271.774381] Free swap  = 0kB
[425271.774381] Total swap = 9782268kB
[425271.774382] 6288152 pages RAM
[425271.774383] 0 pages HighMem/MovableOnly
[425271.774383] 119271 pages reserved
[425271.774384] 0 pages cma reserved
[425271.774384] 0 pages hwpoisoned
[425271.774385] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds 
swapents oom_score_adj name
[425271.774390] [  769]     0   769    10865      714      25       3    
    50             0 systemd-journal
[425271.774393] [  780]     0   780    11465       12      24       3    
   518         -1000 systemd-udevd
[425271.774395] [ 1119]     0  1119     7155       25      19       3    
    52             0 systemd-logind
[425271.774397] [ 1125]     0  1125     6801       36      19       3    
   216             0 smartd
[425271.774399] [ 1187]     0  1187     1098        0       7       3    
    33             0 acpid
[425271.774401] [ 1190]     0  1190     7468        0      19       3    
    61             0 cgmanager
[425271.774403] [ 1191]     0  1191     6931       35      18       3    
    35             0 cron
[425271.774405] [ 1197]   101  1197    64097      475      28       3    
   610             0 rsyslogd
[425271.774407] [ 1201]   102  1201    10726       65      28       3    
    58          -900 dbus-daemon
[425271.774409] [ 1216]     0  1216     6510        8      18       3    
    44             0 atd
[425271.774411] [ 1217]     0  1217    68647       28      36       3    
   150             0 accounts-daemon
[425271.774413] [ 1405]     0  1405     4900       37      14       3    
    39             0 irqbalance
[425271.774415] [ 1406]     0  1406    69271       67      39       3    
   109             0 polkitd
[425271.774417] [ 1419]   103  1419    86258       26      69       4    
   836             0 whoopsie
[425271.774419] [ 1437]     0  1437    16376        0      37       3    
   177         -1000 sshd
[425271.774421] [ 1438]   106  1438    22001     1564      39       3    
12831             0 redis-server
[425271.774423] [ 1633]     0  1633     4934        2      14       3    
    71             0 run-bro
[425271.774425] [ 1639]     0  1639   611863     6111     129       6    
14689             0 bro
[425271.774427] [ 1640]     0  1640    27527      420      56       3    
14833             0 bro
[425271.774430] [ 1823]     0  1823     4934        2      13       3    
    71             0 run-bro
[425271.774432] [ 1829]     0  1829    32819     8914      68       3    
13033             0 bro
[425271.774433] [ 1830]     0  1830    27365       94      55       3    
13927             0 bro
[425271.774435] [ 2456]     0  2456     4934        2      14       3    
    72             0 run-bro
[425271.774437] [ 2461]     0  2461     4934        2      14       3    
    72             0 run-bro
[425271.774439] [ 2467]     0  2467     4934        2      14       3    
    71             0 run-bro
[425271.774441] [ 2470]     0  2470     4934        2      15       3    
    71             0 run-bro
[425271.774443] [ 2482]     0  2482   734289   702195    1419       5    
11537             0 bro
[425271.774445] [ 2488]     0  2488   600424   569376    1176       5    
20589             0 bro
[425271.774447] [ 2492]     0  2492  2711715  1785937    5298      13   
913479             0 bro
[425271.774449] [ 2493]     0  2493  4439547  3000280    8675      19  
1428890             0 bro
[425271.774451] [ 2495]     0  2495    58638    33238     116       3    
13250             0 bro
[425271.774453] [ 2494]     0  2494    58630    33605     116       3    
13627             0 bro
[425271.774455] [ 2496]     0  2496    58616    33114     117       3    
13422             0 bro
[425271.774457] [ 2497]     0  2497    58619    33112     117       3    
14094             0 bro
[425271.774459] [ 2564]     0  2564     3663        0      11       3    
    37             0 agetty
[425271.774461] Out of memory: Kill process 2493 (bro) score 499 or 
sacrifice child
[425271.774515] Killed process 2497 (bro) total-vm:234476kB, 
anon-rss:1340kB, file-rss:131108kB, shmem-rss:0kB


I think I'll try the 2.6 beta and see if that helps.  If there's other 
info I can provide just let me know..thank you.

James

From mkrenz at iu.edu  Mon Nov 12 11:47:53 2018
From: mkrenz at iu.edu (Mark Krenz)
Date: Mon, 12 Nov 2018 14:47:53 -0500
Subject: [Bro] Where is my conn.log?
Message-ID: <ea66aa32-7cec-303e-2580-04bcaeaca38f@iu.edu>

I've inherited a Bro 2.5.5 setup from someone else and am coming to it 
after it's been running for a while without producing any conn or other 
protocol logs.  I've tried restarting Bro and redeploying, but the only 
logs that get started are

communication.log
loaded_scripts.log
packet_filter.log
reporter.log
stats.log
stderr.log
stdout.log
weird.log

None of these logs are filling up with anything useful or indicating 
what the problem may be. The only useful message is 
"non_ip_packet_in_ethernet" in the weird.log. That seems to point to a 
network issue rather than a Bro issue, but I'd like to rule out a Bro 
issue first if possible. At one point this setup did produce useful logs 
but apparently it just stopped at some point and I'm not sure why. The 
only thing somewhat unique about this setup is that at one point it 
required me to use the setting 'redef encap_hdr_size=10;' to handle an 
incompatibility between Bro and a vlan technology this network uses. 
I've also verified that the taps that Bro is listening on are seeing 
actual traffic by using tshark, which is able to decode the protocols.

Any suggestions as to where to start and how to diagnose this?

Thanks,

Mark

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181112/c41d3077/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4146 bytes
Desc: S/MIME Cryptographic Signature
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181112/c41d3077/attachment.bin 

From jlay at slave-tothe-box.net  Mon Nov 12 11:52:19 2018
From: jlay at slave-tothe-box.net (James Lay)
Date: Mon, 12 Nov 2018 12:52:19 -0700
Subject: [Bro] Bro beta install
Message-ID: <90aa25bd1caf38e8d019878e970602d2@slave-tothe-box.net>

Wow what a complete disaster this was.  Errors:

fatal error in /usr/local/bro/share/bro/base/init-bare.bro, line 1: 
cannot load plugin library 
/usr/local/bro/lib/bro/plugins/packages/Bro_AF_Packet//lib/Bro-AF_Packet.linux-x86_64.so: 
/usr/local/bro/lib/bro/plugins/packages/Bro_AF_Packet//lib/Bro-AF_Packet.linux-x86_64.so: 
undefined symbol: 
_ZN6plugin6Plugin12HookLoadFileERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_

warning in /usr/local/bro/share/bro/policy/protocols/smb/__load__.bro, 
line 1: deprecated script loaded from 
/usr/local/bro/share/bro/site/local.bro:98 "Use '@load 
base/protocols/smb' instead"

error in 
/usr/local/bro/share/bro/base/bif/plugins/./Bro_SSL.events.bif.bro, line 
41 and /usr/local/bro/share/bro/site/packages/./ja3/./ja3.bro, line 118: 
incompatible types (event(c:connection; version:count; 
record_version:count; possible_ts:tme; client_random:string; 
session_id:string; ciphers:vector of count; comp_methods:vector of 
count;) and event(c:connection; version:count; possible_ts:time; 
client_random:string; session_id:string; ciphers:vector of count;))

and my redef line below:
redef Communication::listen_interface = 127.0.0.1;

gets me:
error in /usr/local/bro/share/bro/site/local.bro, line 102: "redef" used 
but not previously defined (Communication::listen_interface)

reverted back to 2.5.5 now....yeesh.

James

From hosom at battelle.org  Mon Nov 12 12:25:17 2018
From: hosom at battelle.org (Hosom, Stephen M)
Date: Mon, 12 Nov 2018 20:25:17 +0000
Subject: [Bro] Where is my conn.log?
In-Reply-To: <ea66aa32-7cec-303e-2580-04bcaeaca38f@iu.edu>
References: <ea66aa32-7cec-303e-2580-04bcaeaca38f@iu.edu>
Message-ID: <4ea4d035e53e4fcb85a2bc8d63059978@battelle.org>

Check the reporter.log. I highly suspect that it will have an error related to checksum offloading.


You'll want to try running bro with the -C option to see if that produces logs. If it does, then you'll need to modify your interface configuration. You can do this by installing the interface setup package from NCSA: https://github.com/ncsa/bro-interface-setup or manually configuring your interface along the lines of the guide located here: https://blog.securityonion.net/2011/10/when-is-full-packet-capture-not-full.html



Security Onion: When is full packet capture NOT full packet capture?<https://blog.securityonion.net/2011/10/when-is-full-packet-capture-not-full.html>
blog.securityonion.net
I was looking at some packets recently and noticed the Wireshark message "Packet size limited during capture". &nbsp;This was strange since the p...


________________________________
From: bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Mark Krenz <mkrenz at iu.edu>
Sent: Monday, November 12, 2018 2:47:53 PM
To: bro at bro-ids.org
Subject: [Bro] Where is my conn.log?

Message received from outside the Battelle network. Carefully examine it before you open any links or attachments.


I've inherited a Bro 2.5.5 setup from someone else and am coming to it after it's been running for a while without producing any conn or other protocol logs.  I've tried restarting Bro and redeploying, but the only logs that get started are

communication.log
loaded_scripts.log
packet_filter.log
reporter.log
stats.log
stderr.log
stdout.log
weird.log

None of these logs are filling up with anything useful or indicating what the problem may be. The only useful message is "non_ip_packet_in_ethernet" in the weird.log. That seems to point to a network issue rather than a Bro issue, but I'd like to rule out a Bro issue first if possible. At one point this setup did produce useful logs but apparently it just stopped at some point and I'm not sure why. The only thing somewhat unique about this setup is that at one point it required me to use the setting 'redef encap_hdr_size=10;' to handle an incompatibility between Bro and a vlan technology this network uses. I've also verified that the taps that Bro is listening on are seeing actual traffic by using tshark, which is able to decode the protocols.

Any suggestions as to where to start and how to diagnose this?

Thanks,

Mark


From jsiwek at corelight.com  Mon Nov 12 12:51:45 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Mon, 12 Nov 2018 14:51:45 -0600
Subject: [Bro] Issues since 2.5.5
In-Reply-To: <a073ce2a8d4e70dd5ac015ec189ab51d@slave-tothe-box.net>
References: <a073ce2a8d4e70dd5ac015ec189ab51d@slave-tothe-box.net>
Message-ID: <CAMzgZ0+Y94sBQPY_N4HGGRi+JfHb4OG7UjheDptnrv8Qy5v+YA@mail.gmail.com>

On Mon, Nov 12, 2018 at 1:16 PM James Lay <jlay at slave-tothe-box.net> wrote:
>
> Well..I think I'll also put my name in the "something is funky with
> 2.5.5" group.  I have seen far more crashes and OOM's with 2.5.5 than
> with 2.5.4.

Various thoughts:

* This is the first I've heard of trouble directly related to 2.5.5 in
contrast to 2.5.4. If you have reference to others reporting similar,
please point me at it as it may help with correlating/diagnosing.

* For any crashes, forwarding stack traces to reports at bro.org would help.

* For OOM, a first sanity check is to make sure reporter.log isn't
showing any scripting errors.  E.g. unitialized record field access is
known to leak memory, but it's also an underlying scripting mistake
that needs to fixed.

* Similarly, remember that memory utilization is effected by scripting
logic.  If you use any custom or external scripts/packages that are
not conservative with how they track state over time, that's always a
possible source of OOM problems that's independent of Bro version.  So
a question would be whether you are comparing the same configuration
between 2.5.4 and 2.5.5 or were some scripts/packages different?

- Jon

From jsiwek at corelight.com  Mon Nov 12 13:34:51 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Mon, 12 Nov 2018 15:34:51 -0600
Subject: [Bro] Bro beta install
In-Reply-To: <90aa25bd1caf38e8d019878e970602d2@slave-tothe-box.net>
References: <90aa25bd1caf38e8d019878e970602d2@slave-tothe-box.net>
Message-ID: <CAMzgZ0JRhKo6boQpmqM9AG4Vsz9h70AAG6cEeOogjhjn96gJ7g@mail.gmail.com>

On Mon, Nov 12, 2018 at 2:06 PM James Lay <jlay at slave-tothe-box.net> wrote:
>
> Wow what a complete disaster this was.  Errors:
>
> fatal error in /usr/local/bro/share/bro/base/init-bare.bro, line 1:
> cannot load plugin library
> /usr/local/bro/lib/bro/plugins/packages/Bro_AF_Packet//lib/Bro-AF_Packet.linux-x86_64.so:
> /usr/local/bro/lib/bro/plugins/packages/Bro_AF_Packet//lib/Bro-AF_Packet.linux-x86_64.so:
> undefined symbol:
> _ZN6plugin6Plugin12HookLoadFileERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_

May need to recompile the plugin against new Bro version?

> warning in /usr/local/bro/share/bro/policy/protocols/smb/__load__.bro,
> line 1: deprecated script loaded from
> /usr/local/bro/share/bro/site/local.bro:98 "Use '@load
> base/protocols/smb' instead"

It's only warning, but you can just update local.bro as indicated.

> error in
> /usr/local/bro/share/bro/base/bif/plugins/./Bro_SSL.events.bif.bro, line
> 41 and /usr/local/bro/share/bro/site/packages/./ja3/./ja3.bro, line 118:
> incompatible types (event(c:connection; version:count;
> record_version:count; possible_ts:tme; client_random:string;
> session_id:string; ciphers:vector of count; comp_methods:vector of
> count;) and event(c:connection; version:count; possible_ts:time;
> client_random:string; session_id:string; ciphers:vector of count;))

This will need to be fixed in the ja3 package.  I have a PR open now:

https://github.com/salesforce/ja3/pull/27

So either wait for that to get merged and then update the local
package or use my fork/patch directly if eager to try Bro 2.6-beta.

> and my redef line below:
> redef Communication::listen_interface = 127.0.0.1;
>
> gets me:
> error in /usr/local/bro/share/bro/site/local.bro, line 102: "redef" used
> but not previously defined (Communication::listen_interface)

The equivalent functionality is now:

    redef Broker::default_listen_address = "127.0.0.1";

(The underlying communication systems in Bro have been completely
replaced with a new library called "Broker").

> reverted back to 2.5.5 now....yeesh.

There's quite a few potential incompatibilities with upcoming Bro 2.6,
so worth seeing release notes:

https://www.bro.org/sphinx-git/install/release-notes.html

Generally a hope is that future releases minimize breakages for users,
but there were some big fundamental changes that made it hard to avoid
for 2.6.

Let me know if you give it another shot and have further trouble.

- Jon

From alshaboti.it at gmail.com  Mon Nov 12 19:40:09 2018
From: alshaboti.it at gmail.com (Mohammed Alshaboti)
Date: Tue, 13 Nov 2018 16:40:09 +1300
Subject: [Bro] Integrate Bro with network controller
Message-ID: <CAKbXeiCus+tXGdWouJDNStoqeqkp0aUb4_OvWRbHB1JHvEtv3w@mail.gmail.com>

Hi,
I am working on integrating Bro with Faucet (an SDN controller that use ACL
file to push OpenFlow rules to switches).

My question is what is the ideal way to do that?
Do I have to create netcontrol plugin for that, and can the plugin read and
modify the ACL file  (faucet ACL file)?
or
Should I just use the netcontrol broker to communicate with an external
python client like (https://github.com/bro/bro-netcontrol/tree/master/test
).

Another general question, what is the main purpose of the
netcontrol plugin?
Is it to add extra functionality within Bro, or to connect provide
integration with external systems.

Thank you,
Mohammed
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181113/8eae46ac/attachment.html 

From turbidtarantula at gmail.com  Tue Nov 13 05:54:17 2018
From: turbidtarantula at gmail.com (Mike M)
Date: Tue, 13 Nov 2018 08:54:17 -0500
Subject: [Bro] Dumping Sumstats on Bro Termination
Message-ID: <CAK051agj1FpxnWqPKTXjOudZGo+mHCYOTEfgf2NPaO5GfkTf=w@mail.gmail.com>

I'm using Sumstats and I've got some pcaps I want to run through Bro.
Depending on the pcap duration I end up with interim results in Sumstats
when Bro exits, because the epoch threshold hasn't been crossed. That makes
sense, but I'd like to log any partial results that haven't crossed the
threshold when reading a pcap.

Is there a way to force the epoch_result function to run when Bro finishes
the pcap, or otherwise dump the partial results when Bro exits?

Thanks,
Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181113/35d5a575/attachment.html 

From turbidtarantula at gmail.com  Tue Nov 13 08:22:10 2018
From: turbidtarantula at gmail.com (Mike M)
Date: Tue, 13 Nov 2018 11:22:10 -0500
Subject: [Bro] Getting a Broctl Stack Trace
In-Reply-To: <AB6568DA-7AD5-4D84-B1AB-AD00877F7114@corelight.com>
References: <CAK051agH+hWgr-bZrfqVBQEKHZrU9AkCsb=CwFx4ohUp+YfkuA@mail.gmail.com>
	<AB6568DA-7AD5-4D84-B1AB-AD00877F7114@corelight.com>
Message-ID: <CAK051ai0=gFXbd96BOZG9RddE6qA-vB4XY+i+em=PotNv2vJkA@mail.gmail.com>

I gave this a shot but I'm still not seeing a core file. I tried both the
setting you recommended and setting an absolute path to /tmp. When I force
a core dump on another process the core file shows up as expected, but
broctl isn't producing one.

I'm open to suggestions on this one... not sure how to determine the root
cause.

thanks,
Mike

On Mon, Nov 5, 2018 at 5:01 PM Seth Hall <seth at corelight.com> wrote:

> Make sure you are setting the core pattern on your system so that the
> core dump will be written into the CWD.
>
> sudo sysctl -w kernel.core_pattern="core.%e-%t-%p"
>
>    .Seth
>
> On 2 Nov 2018, at 12:51, Mike M wrote:
>
> > I'm having an issue with broctl crashing when I try to run it on
> > Alpine
> > Linux. I mentioned it previously [1] but I'm circling back around to
> > try to
> > get it resolved. I've built it with the appropriate patches [2] but
> > broctl
> > is still reporting "crashed" state when I checks the status after
> > starting
> > it. The bro binary itself runs fine.
> >
> > What do I need to do to collect a stack trace from broctl to determine
> > the
> > root cause?
> >
> > Bro is built in debug mode and I set "ulimit -c unlimited" per the
> > instructions on reporting problems. I see a
> >
> /usr/local/bro/spool/tmp/post-terminate-standalone-2018-11-02-02-56-06-13765-crash
> > directory but there's no core dump anywhere obvious. The
> > .crash-diag.out
> > file says "No core file found" and doesn't provide any useful
> > information
> > about the cause of the crash.
> >
> > Thanks,
> > Mike
> >
> > [1]
> >
> http://mailman.icsi.berkeley.edu/pipermail/bro/2018-September/013580.html
> > [2]
> >
> http://mailman.icsi.berkeley.edu/pipermail/bro/2018-September/013581.html
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> --
> Seth Hall * Corelight, Inc * www.corelight.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181113/6999b022/attachment-0001.html 

From nothinrandom at gmail.com  Tue Nov 13 09:25:11 2018
From: nothinrandom at gmail.com (TQ)
Date: Tue, 13 Nov 2018 09:25:11 -0800
Subject: [Bro] Listening on both UDP/TCP
In-Reply-To: <20181107011110.k74knswhbs3zifuw@dhcp-9e80.meeting.ietf.org>
References: <CAM2MKSpnSx+602o=3AGEBhNNPbaVqmvt_1jS9n3N+6GDL=ewtA@mail.gmail.com>
	<20181107011110.k74knswhbs3zifuw@dhcp-9e80.meeting.ietf.org>
Message-ID: <CAM2MKSp=GGHXB+PGsK2CGayKaYgV19EVhKHA6cy1pwcA2N=3dA@mail.gmail.com>

Hey Johanna,

I followed your suggestion and looked at SSL, works great!

Thanks,

On Tue, Nov 6, 2018 at 5:11 PM Johanna Amann <johanna at icir.org> wrote:

> Hi,
>
> > I see many of the existing protocols focus on either TCP or UDP, but
> > nothing for both.  I did notice that SIP has both TCP and UDP, however,
> the
> > TCP portion is "not activated" (
> > https://github.com/bro/bro/tree/master/src/analyzer/protocol/sip).  Is
> > there a good example of how to handle both?  Is this something where I
> > would need register listener in main.bro? For example:
>
> [...]
>
> the closest to this is probably the TLS/DTLS analyzer. Similarly to SIP,
> it actually is 2 analyzers (one for TLS over TCP and one for DTLS over
> UDP) that share a lot of the code.
>
> scripts/base/protocols/ssl/main.bro shows that both of them are just
> initialized separately from each other. From a very cursory glance over
> SIP, I think that one could just do the same there.
>
> I hope this helps,
>  Johanna
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181113/de602c1a/attachment.html 

From alshaboti.it at gmail.com  Wed Nov 14 21:29:20 2018
From: alshaboti.it at gmail.com (Mohammed Alshaboti)
Date: Thu, 15 Nov 2018 18:29:20 +1300
Subject: [Bro] React based on Bro event (block/unblock connection)
Message-ID: <CAKbXeiAaa1gsAhfiV+h_cZracMjWJHe+9nTssKo7kDPWM=RVLA@mail.gmail.com>

Hi,
I would like to send Bro data (e.g. connection) to a backend python
program on some events.
I tried to use the netcontrol broker to communicate with an external
python client like (https://github.com/bro/bro-netcontrol/tree/master/test
).
But when I added event it crashed.


-- 
Best regards,
Mohammed Al-Shaboti
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181115/aa2184ca/attachment.html 

From bucweat at rushpost.com  Thu Nov 15 07:30:07 2018
From: bucweat at rushpost.com (Charlie)
Date: Thu, 15 Nov 2018 07:30:07 -0800
Subject: [Bro] VNC payload
Message-ID: <6BE7D8B7-E9EF-4CEA-988E-079759E52BF7@rushpost.com>

Hi,

I am new to bro so please forgive what I hope is a simple question with simple answer....

Running bro 2.5.5 on MacOS 10.12. I have a pcap file with traffic on port 5900. I'm trying to use contents.bro to write the payload of this conversation to a file so that I can process it later. I'm using something like 

 bro -r ../capture.pcap -f 'tcp port 5900' /usr/local/share/bro/base/protocols/conn/contents.bro

This results in the following connection log (sorry for the wrap) which tells me bro sees the traffic, but does not generate data files for port 5900:

#separator \x09
#set_separator	,
#empty_field	(empty)
#unset_field	-
#path	conn
#open	2018-11-14-17-19-18
#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	service	duration	orig_bytes	resp_bytes	conn_state	local_orig	local_resp	missed_bytes	history	orig_pkts	orig_ip_bytes	resp_pkts	resp_ip_bytes	tunnel_parents
#types	time	string	addr	port	addr	port	enum	string	interval	count	count	string	bool	bool	count	string	count	count	count	count	set[string]
1496000046.839399	C9iQbO1Y4veE4M2MDe	192.168.1.19	5900	192.168.1.14	50663	tcp	-	184.468970	9195324	68250	OTH	-	-	0	DadAT	12266	9838046	8647	518684	(empty)
#close	2018-11-14-17-19-18

If I run without the BPF filter:

bro -r ../capture.pcap /usr/local/share/bro/base/protocols/conn/contents.bro

I get data files for other conversations with filenames that look like

contents_192.168.1.19:50560-192.168.1.8:62078_orig.dat

which contain content that I expect based on looking at pcap in wireshark, but nothing for the conversation on port 5900. 

I know the data is there (shows up in wireshark) and I can run tcpflow and get data files for port 5900. So yes I realize there are other ways to do this, however, I'm trying to learn how to bro and so want to understand in a more generic sense why bro would not produce files for a given conversation in general and hope to learn by figuring out why it specifically does not generate anything for port 5900. 

My actual use case has nothing to do with data on port 5900. I used this pcap file as an example to try out contents.bro, and was surprised/stumped as to why it would not generate a data file for port 5900 data. It might turn out that bro works just fine for my actual use case...but would still like to understand why it doesn't work in this case.

The .bro scripts in my installation have not been modified. I made a copy of local.bro (called it something else) and have been playing with changes there, but in the above example I'm not using that...just the bro default settings and contents.bro. 

Thanks in advance, Charlie






-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181115/627b080d/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2352 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181115/627b080d/attachment.bin 

From jsiwek at corelight.com  Thu Nov 15 08:03:08 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 15 Nov 2018 10:03:08 -0600
Subject: [Bro] VNC payload
In-Reply-To: <6BE7D8B7-E9EF-4CEA-988E-079759E52BF7@rushpost.com>
References: <6BE7D8B7-E9EF-4CEA-988E-079759E52BF7@rushpost.com>
Message-ID: <CAMzgZ0LNu5HiCu181kPDtUbL2Vb5kba8zcQ8Vjmjn_mhDFF=uQ@mail.gmail.com>

On Thu, Nov 15, 2018 at 9:41 AM Charlie <bucweat at rushpost.com> wrote:

> 1496000046.839399 C9iQbO1Y4veE4M2MDe 192.168.1.19 5900 192.168.1.14 50663 tcp - 184.468970 9195324 68250 OTH - - 0 DadAT 12266 9838046 8647 518684 (empty)
>
> If I run without the BPF filter:
>
> bro -r ../capture.pcap /usr/local/share/bro/base/protocols/conn/contents.bro
>
> I get data files for other conversations with filenames that look like
>
> contents_192.168.1.19:50560-192.168.1.8:62078_orig.dat
>
> which contain content that I expect based on looking at pcap in wireshark, but nothing for the conversation on port 5900.

The conn.log history field of "DadAT" indicates the TCP handshake is
not present for that connection in the pcap, however, contents.bro is
handling a "connection_established" event in order to trigger the
content-dumping, and that is only generated "when seeing a SYN-ACK
packet from the responder in a TCP handshake".

Another caveat from the "set_contents_file" documentation that may be
useful to know and relevant to your use-case: "If any data is missing,
the recording stops at the missing data".

- Jon

From jsiwek at corelight.com  Thu Nov 15 09:02:35 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 15 Nov 2018 11:02:35 -0600
Subject: [Bro] React based on Bro event (block/unblock connection)
In-Reply-To: <CAKbXeiAaa1gsAhfiV+h_cZracMjWJHe+9nTssKo7kDPWM=RVLA@mail.gmail.com>
References: <CAKbXeiAaa1gsAhfiV+h_cZracMjWJHe+9nTssKo7kDPWM=RVLA@mail.gmail.com>
Message-ID: <CAMzgZ0Je=ty_He+=aBfY63gRBfRTscqFZnpjXwg5EtCR03--VA@mail.gmail.com>

On Wed, Nov 14, 2018 at 11:38 PM Mohammed Alshaboti
<alshaboti.it at gmail.com> wrote:

> I would like to send Bro data (e.g. connection) to a backend python program on some events.
> I tried to use the netcontrol broker to communicate with an external
> python client like (https://github.com/bro/bro-netcontrol/tree/master/test
> ).
> But when I added event it crashed.

Can you provide more info?  e.g. exact code that you're trying.  Was
it bro or the python program that crashed?  Any other relevant output
or error messages?

- Jon

From jsiwek at corelight.com  Thu Nov 15 09:12:06 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 15 Nov 2018 11:12:06 -0600
Subject: [Bro] Bro 2.6-beta3 available
Message-ID: <CAMzgZ0Liw82y9+E27d4Kw8w1HgWy7hb70tEy01jbGbsaGCJf-Q@mail.gmail.com>

A new beta/candidate for Bro 2.6, tagged -beta3, is now available at:

https://www.bro.org/download/index.html

The NEWS file contains the significant changes in relation to 2.5.x:

https://www.bro.org/documentation/beta/NEWS.bro.html

However, there's no major differences between -beta2 and -beta3, just
many bug fixes and internal changes related to the new Broker
communication system that we'd like to see tested more thoroughly.  So
please try it out and report any issues as it's hoped to finalize to
2.6 soon.

- Jon

From lee at shiry.org  Fri Nov 16 09:23:05 2018
From: lee at shiry.org (Lee Shiry)
Date: Fri, 16 Nov 2018 12:23:05 -0500
Subject: [Bro] Help with intel framework
Message-ID: <ae422f90-f5af-1982-018c-d671f6945f19@shiry.org>

Hi,

I am trying to use Bro's intel framework and can't seem to get it to
generate anything in the intel or notice logs.? I'm on version 2.5.5 in
cluster mode.? Everything else seems to work fine.? I see all the logs,
and notices are working for other event types. I have checked to make
sure the dat file has only tabs in it to separate fields.? I don't see
anything coming up in the stderr or reporter log files.? I must be
missing something.? Any help is appreciated.

Here is what I have added to local.bro:

##################
@load frameworks/intel/seen
@load frameworks/intel/do_notice

redef Intel::read_files += {
??????? "/usr/local/intel-bad-user-agents.dat",
};
##################


Here is the dat file:

##################
# cat /usr/local/intel-bad-user-agents.dat
#fields??? indicator??? indicator_type??? meta.do_notice??? meta.if_in
360Spider??? Intel::SOFTWARE??? T??? HTTP::IN_USER_AGENT_HEADER
Mozilla??? Intel::SOFTWARE??? T??? HTTP::IN_USER_AGENT_HEADER
##################

(I temporarily put Mozilla in there to generate lots of events for
testing purposes)


Thanks,
lms
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181116/01ae40af/attachment.html 

From shirkdog.bsd at gmail.com  Fri Nov 16 09:44:55 2018
From: shirkdog.bsd at gmail.com (Michael Shirk)
Date: Fri, 16 Nov 2018 12:44:55 -0500
Subject: [Bro] Help with intel framework
In-Reply-To: <ae422f90-f5af-1982-018c-d671f6945f19@shiry.org>
References: <ae422f90-f5af-1982-018c-d671f6945f19@shiry.org>
Message-ID: <CAL8PkUX-y7ntxOipspD=7MuvWnZK77Jx2pNXLW=C0B1xRhN=oA@mail.gmail.com>

The most important thing is the format of that ".dat" file. If you do
not have tabs entered correctly, the files may not be loaded. Check
your "reporter.log" to see if there are any errors with the input of
your intel file.

Example error:
0.000000        Reporter::WARNING
/nsm/intel/custom.intel/Input::READER_ASCII: Field: meta.do_notice
Invalid value for boolean: meta.do_notice    (empty)
On Fri, Nov 16, 2018 at 12:24 PM Lee Shiry <lee at shiry.org> wrote:
>
> Hi,
>
> I am trying to use Bro's intel framework and can't seem to get it to generate anything in the intel or notice logs.  I'm on version 2.5.5 in cluster mode.  Everything else seems to work fine.  I see all the logs, and notices are working for other event types. I have checked to make sure the dat file has only tabs in it to separate fields.  I don't see anything coming up in the stderr or reporter log files.  I must be missing something.  Any help is appreciated.
>
> Here is what I have added to local.bro:
>
> ##################
> @load frameworks/intel/seen
> @load frameworks/intel/do_notice
>
> redef Intel::read_files += {
>         "/usr/local/intel-bad-user-agents.dat",
> };
> ##################
>
>
> Here is the dat file:
>
> ##################
> # cat /usr/local/intel-bad-user-agents.dat
> #fields    indicator    indicator_type    meta.do_notice    meta.if_in
> 360Spider    Intel::SOFTWARE    T    HTTP::IN_USER_AGENT_HEADER
> Mozilla    Intel::SOFTWARE    T    HTTP::IN_USER_AGENT_HEADER
> ##################
>
> (I temporarily put Mozilla in there to generate lots of events for testing purposes)
>
>
> Thanks,
> lms
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
Michael Shirk
Daemon Security, Inc.
https://www.daemon-security.com


From shirkdog.bsd at gmail.com  Fri Nov 16 09:47:32 2018
From: shirkdog.bsd at gmail.com (Michael Shirk)
Date: Fri, 16 Nov 2018 12:47:32 -0500
Subject: [Bro] Help with intel framework
In-Reply-To: <CAL8PkUX-y7ntxOipspD=7MuvWnZK77Jx2pNXLW=C0B1xRhN=oA@mail.gmail.com>
References: <ae422f90-f5af-1982-018c-d671f6945f19@shiry.org>
	<CAL8PkUX-y7ntxOipspD=7MuvWnZK77Jx2pNXLW=C0B1xRhN=oA@mail.gmail.com>
Message-ID: <CAL8PkUXt1WOaetfPpAy2pRzx=qN2g=H+y1zCQYdyvk8PPJPxEg@mail.gmail.com>

And I misread you already did check reporter. Sorry for the noise. I
would get a pcap and test this offline with the intel framework to
make sure everything is working as it should.
On Fri, Nov 16, 2018 at 12:44 PM Michael Shirk <shirkdog.bsd at gmail.com> wrote:
>
> The most important thing is the format of that ".dat" file. If you do
> not have tabs entered correctly, the files may not be loaded. Check
> your "reporter.log" to see if there are any errors with the input of
> your intel file.
>
> Example error:
> 0.000000        Reporter::WARNING
> /nsm/intel/custom.intel/Input::READER_ASCII: Field: meta.do_notice
> Invalid value for boolean: meta.do_notice    (empty)
> On Fri, Nov 16, 2018 at 12:24 PM Lee Shiry <lee at shiry.org> wrote:
> >
> > Hi,
> >
> > I am trying to use Bro's intel framework and can't seem to get it to generate anything in the intel or notice logs.  I'm on version 2.5.5 in cluster mode.  Everything else seems to work fine.  I see all the logs, and notices are working for other event types. I have checked to make sure the dat file has only tabs in it to separate fields.  I don't see anything coming up in the stderr or reporter log files.  I must be missing something.  Any help is appreciated.
> >
> > Here is what I have added to local.bro:
> >
> > ##################
> > @load frameworks/intel/seen
> > @load frameworks/intel/do_notice
> >
> > redef Intel::read_files += {
> >         "/usr/local/intel-bad-user-agents.dat",
> > };
> > ##################
> >
> >
> > Here is the dat file:
> >
> > ##################
> > # cat /usr/local/intel-bad-user-agents.dat
> > #fields    indicator    indicator_type    meta.do_notice    meta.if_in
> > 360Spider    Intel::SOFTWARE    T    HTTP::IN_USER_AGENT_HEADER
> > Mozilla    Intel::SOFTWARE    T    HTTP::IN_USER_AGENT_HEADER
> > ##################
> >
> > (I temporarily put Mozilla in there to generate lots of events for testing purposes)
> >
> >
> > Thanks,
> > lms
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> --
> Michael Shirk
> Daemon Security, Inc.
> https://www.daemon-security.com



-- 
Michael Shirk
Daemon Security, Inc.
https://www.daemon-security.com


From lee at shiry.org  Fri Nov 16 09:51:10 2018
From: lee at shiry.org (Lee Shiry)
Date: Fri, 16 Nov 2018 12:51:10 -0500
Subject: [Bro] Help with intel framework
In-Reply-To: <CAL8PkUX-y7ntxOipspD=7MuvWnZK77Jx2pNXLW=C0B1xRhN=oA@mail.gmail.com>
References: <ae422f90-f5af-1982-018c-d671f6945f19@shiry.org>
	<CAL8PkUX-y7ntxOipspD=7MuvWnZK77Jx2pNXLW=C0B1xRhN=oA@mail.gmail.com>
Message-ID: <65923035-58db-e0d7-534e-0167a73bff41@shiry.org>

Thanks for the reply.

I am not seeing any errors show up in the reporter.log or the
stderr.log.? I verified the tabs are there.? I was wondering if I could
get it to generate an error, so I intentionally misspelled the name of
the dat file in the script, but it did not generate an error.? Now I
wonder if something else is preventing it from getting that far.

On 11/16/18 12:44 PM, Michael Shirk wrote:
> The most important thing is the format of that ".dat" file. If you do
> not have tabs entered correctly, the files may not be loaded. Check
> your "reporter.log" to see if there are any errors with the input of
> your intel file.
>
> Example error:
> 0.000000        Reporter::WARNING
> /nsm/intel/custom.intel/Input::READER_ASCII: Field: meta.do_notice
> Invalid value for boolean: meta.do_notice    (empty)
> On Fri, Nov 16, 2018 at 12:24 PM Lee Shiry <lee at shiry.org> wrote:
>> Hi,
>>
>> I am trying to use Bro's intel framework and can't seem to get it to generate anything in the intel or notice logs.  I'm on version 2.5.5 in cluster mode.  Everything else seems to work fine.  I see all the logs, and notices are working for other event types. I have checked to make sure the dat file has only tabs in it to separate fields.  I don't see anything coming up in the stderr or reporter log files.  I must be missing something.  Any help is appreciated.
>>
>> Here is what I have added to local.bro:
>>
>> ##################
>> @load frameworks/intel/seen
>> @load frameworks/intel/do_notice
>>
>> redef Intel::read_files += {
>>         "/usr/local/intel-bad-user-agents.dat",
>> };
>> ##################
>>
>>
>> Here is the dat file:
>>
>> ##################
>> # cat /usr/local/intel-bad-user-agents.dat
>> #fields    indicator    indicator_type    meta.do_notice    meta.if_in
>> 360Spider    Intel::SOFTWARE    T    HTTP::IN_USER_AGENT_HEADER
>> Mozilla    Intel::SOFTWARE    T    HTTP::IN_USER_AGENT_HEADER
>> ##################
>>
>> (I temporarily put Mozilla in there to generate lots of events for testing purposes)
>>
>>
>> Thanks,
>> lms
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181116/bf07758e/attachment.html 

From lee at shiry.org  Fri Nov 16 10:28:31 2018
From: lee at shiry.org (Lee Shiry)
Date: Fri, 16 Nov 2018 13:28:31 -0500
Subject: [Bro] Help with intel framework
In-Reply-To: <CAL8PkUXt1WOaetfPpAy2pRzx=qN2g=H+y1zCQYdyvk8PPJPxEg@mail.gmail.com>
References: <ae422f90-f5af-1982-018c-d671f6945f19@shiry.org>
	<CAL8PkUX-y7ntxOipspD=7MuvWnZK77Jx2pNXLW=C0B1xRhN=oA@mail.gmail.com>
	<CAL8PkUXt1WOaetfPpAy2pRzx=qN2g=H+y1zCQYdyvk8PPJPxEg@mail.gmail.com>
Message-ID: <cc28fc06-d1c7-63af-6deb-b3582eaf33eb@shiry.org>

I took a pcap file and ran it standalone against the file.? This time it
complained that the requested field meta.source was missing.? I don't
know why that was not showing up in the reporter or stderr logs.? I
added the field, and now there are no errors, but still no intel.log.

On 11/16/18 12:47 PM, Michael Shirk wrote:
> And I misread you already did check reporter. Sorry for the noise. I
> would get a pcap and test this offline with the intel framework to
> make sure everything is working as it should.
> On Fri, Nov 16, 2018 at 12:44 PM Michael Shirk <shirkdog.bsd at gmail.com> wrote:
>> The most important thing is the format of that ".dat" file. If you do
>> not have tabs entered correctly, the files may not be loaded. Check
>> your "reporter.log" to see if there are any errors with the input of
>> your intel file.
>>
>> Example error:
>> 0.000000        Reporter::WARNING
>> /nsm/intel/custom.intel/Input::READER_ASCII: Field: meta.do_notice
>> Invalid value for boolean: meta.do_notice    (empty)
>> On Fri, Nov 16, 2018 at 12:24 PM Lee Shiry <lee at shiry.org> wrote:
>>> Hi,
>>>
>>> I am trying to use Bro's intel framework and can't seem to get it to generate anything in the intel or notice logs.  I'm on version 2.5.5 in cluster mode.  Everything else seems to work fine.  I see all the logs, and notices are working for other event types. I have checked to make sure the dat file has only tabs in it to separate fields.  I don't see anything coming up in the stderr or reporter log files.  I must be missing something.  Any help is appreciated.
>>>
>>> Here is what I have added to local.bro:
>>>
>>> ##################
>>> @load frameworks/intel/seen
>>> @load frameworks/intel/do_notice
>>>
>>> redef Intel::read_files += {
>>>         "/usr/local/intel-bad-user-agents.dat",
>>> };
>>> ##################
>>>
>>>
>>> Here is the dat file:
>>>
>>> ##################
>>> # cat /usr/local/intel-bad-user-agents.dat
>>> #fields    indicator    indicator_type    meta.do_notice    meta.if_in
>>> 360Spider    Intel::SOFTWARE    T    HTTP::IN_USER_AGENT_HEADER
>>> Mozilla    Intel::SOFTWARE    T    HTTP::IN_USER_AGENT_HEADER
>>> ##################
>>>
>>> (I temporarily put Mozilla in there to generate lots of events for testing purposes)
>>>
>>>
>>> Thanks,
>>> lms
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>> --
>> Michael Shirk
>> Daemon Security, Inc.
>> https://www.daemon-security.com
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181116/734f11b8/attachment-0001.html 

From lee at shiry.org  Fri Nov 16 11:44:33 2018
From: lee at shiry.org (Lee Shiry)
Date: Fri, 16 Nov 2018 14:44:33 -0500
Subject: [Bro] Help with intel framework
In-Reply-To: <cc28fc06-d1c7-63af-6deb-b3582eaf33eb@shiry.org>
References: <ae422f90-f5af-1982-018c-d671f6945f19@shiry.org>
	<CAL8PkUX-y7ntxOipspD=7MuvWnZK77Jx2pNXLW=C0B1xRhN=oA@mail.gmail.com>
	<CAL8PkUXt1WOaetfPpAy2pRzx=qN2g=H+y1zCQYdyvk8PPJPxEg@mail.gmail.com>
	<cc28fc06-d1c7-63af-6deb-b3582eaf33eb@shiry.org>
Message-ID: <cbd62d07-93f4-51f4-9d5f-4184daa8e171@shiry.org>

I have tried several things in offline mode.? I am searching for
"Firefox" in "Intel::IN_ANYWHERE", and still no intel hits, even though
"firefox" clealry shows up in the http.log in the user agent field.?

On 11/16/18 1:28 PM, Lee Shiry wrote:
> I took a pcap file and ran it standalone against the file.? This time
> it complained that the requested field meta.source was missing.? I
> don't know why that was not showing up in the reporter or stderr
> logs.? I added the field, and now there are no errors, but still no
> intel.log.
>
> On 11/16/18 12:47 PM, Michael Shirk wrote:
>> And I misread you already did check reporter. Sorry for the noise. I
>> would get a pcap and test this offline with the intel framework to
>> make sure everything is working as it should.
>> On Fri, Nov 16, 2018 at 12:44 PM Michael Shirk <shirkdog.bsd at gmail.com> wrote:
>>> The most important thing is the format of that ".dat" file. If you do
>>> not have tabs entered correctly, the files may not be loaded. Check
>>> your "reporter.log" to see if there are any errors with the input of
>>> your intel file.
>>>
>>> Example error:
>>> 0.000000        Reporter::WARNING
>>> /nsm/intel/custom.intel/Input::READER_ASCII: Field: meta.do_notice
>>> Invalid value for boolean: meta.do_notice    (empty)
>>> On Fri, Nov 16, 2018 at 12:24 PM Lee Shiry <lee at shiry.org> wrote:
>>>> Hi,
>>>>
>>>> I am trying to use Bro's intel framework and can't seem to get it to generate anything in the intel or notice logs.  I'm on version 2.5.5 in cluster mode.  Everything else seems to work fine.  I see all the logs, and notices are working for other event types. I have checked to make sure the dat file has only tabs in it to separate fields.  I don't see anything coming up in the stderr or reporter log files.  I must be missing something.  Any help is appreciated.
>>>>
>>>> Here is what I have added to local.bro:
>>>>
>>>> ##################
>>>> @load frameworks/intel/seen
>>>> @load frameworks/intel/do_notice
>>>>
>>>> redef Intel::read_files += {
>>>>         "/usr/local/intel-bad-user-agents.dat",
>>>> };
>>>> ##################
>>>>
>>>>
>>>> Here is the dat file:
>>>>
>>>> ##################
>>>> # cat /usr/local/intel-bad-user-agents.dat
>>>> #fields    indicator    indicator_type    meta.do_notice    meta.if_in
>>>> 360Spider    Intel::SOFTWARE    T    HTTP::IN_USER_AGENT_HEADER
>>>> Mozilla    Intel::SOFTWARE    T    HTTP::IN_USER_AGENT_HEADER
>>>> ##################
>>>>
>>>> (I temporarily put Mozilla in there to generate lots of events for testing purposes)
>>>>
>>>>
>>>> Thanks,
>>>> lms
>>>> _______________________________________________
>>>> Bro mailing list
>>>> bro at bro-ids.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>> --
>>> Michael Shirk
>>> Daemon Security, Inc.
>>> https://www.daemon-security.com
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181116/2c0b74e3/attachment.html 

From fatema.bannatwala at gmail.com  Fri Nov 16 13:03:25 2018
From: fatema.bannatwala at gmail.com (fatema bannatwala)
Date: Fri, 16 Nov 2018 16:03:25 -0500
Subject: [Bro] Help with intel framework
Message-ID: <CACX0rUStDaYi0F6rSB7-2PV+dm-tgucyXow13d8zRSkbDYPURw@mail.gmail.com>

Hey,

Just a quick check, Bro won't generate the intel.log if it's unable to load
the intel input file to read from.
was looking at your intel file re-definition:

redef Intel::read_files += {
          "/usr/local/intel-bad-user-agents.dat",
};

Can you remove the trailing "," after
"/usr/local/intel-bad-user-agents.dat" line and see if it works.
I am not sure if that line should be ended with a comma.

Also,can you try with an "Intel::ADDR" type just to check if it's getting
triggered?
You can add any IP that you can test with Intel::ADDR and see if that works.

Fatema
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181116/f569b8f0/attachment.html 

From lee at shiry.org  Fri Nov 16 13:09:45 2018
From: lee at shiry.org (Lee Shiry)
Date: Fri, 16 Nov 2018 16:09:45 -0500
Subject: [Bro] Help with intel framework
In-Reply-To: <CACX0rUStDaYi0F6rSB7-2PV+dm-tgucyXow13d8zRSkbDYPURw@mail.gmail.com>
References: <CACX0rUStDaYi0F6rSB7-2PV+dm-tgucyXow13d8zRSkbDYPURw@mail.gmail.com>
Message-ID: <421dc3ea-e0b6-cfe4-c3f6-8c88dbd062f3@shiry.org>

I removed the comma, and added a line in the dat file using Intel::ADDR,
still no intel.log.

On 11/16/18 4:03 PM, fatema bannatwala wrote:
> Hey,
>
> Just a quick check, Bro won't generate the intel.log if it's unable to
> load the intel input file to read from.
> was looking at your intel file re-definition:
>
> redef Intel::read_files += {
> ? ? ? ? ? "/usr/local/intel-bad-user-agents.dat",
> };
>
> Can you remove the trailing "," after
> "/usr/local/intel-bad-user-agents.dat" line and see if it works.
> I am not sure if that line should be ended with a comma.
>
> Also,can you try with an "Intel::ADDR" type just to check if it's
> getting triggered?
> You can add any IP that you can test with?Intel::ADDR and see if that
> works.
>
> Fatema
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181116/df41ca64/attachment.html 

From lee at shiry.org  Fri Nov 16 13:26:57 2018
From: lee at shiry.org (Lee Shiry)
Date: Fri, 16 Nov 2018 16:26:57 -0500
Subject: [Bro] Help with intel framework
In-Reply-To: <421dc3ea-e0b6-cfe4-c3f6-8c88dbd062f3@shiry.org>
References: <CACX0rUStDaYi0F6rSB7-2PV+dm-tgucyXow13d8zRSkbDYPURw@mail.gmail.com>
	<421dc3ea-e0b6-cfe4-c3f6-8c88dbd062f3@shiry.org>
Message-ID: <32cc294e-3808-df59-0cde-e42d5da81833@shiry.org>

I made a tracefile, and can see in there that it reads the dat file:

# cat tracefile.log |grep Input
0.000000
/usr/local/bro/share/bro/base/frameworks/input/./main.bro:263??? ???
function called: Input::add_event(description =
'[source=/usr/local/intel-bad-user-agents.dat,
reader=Input::READER_ASCII, mode=Input::REREAD,
name=intel-/usr/local/intel-bad-user-agents.dat, fields=<no value
description>, want_record=T, ev=Intel::read_entry
0.000000
/usr/local/bro/share/bro/base/frameworks/input/./main.bro:263??? ??? ???
Builtin Function called: Input::__create_event_stream(description =
'[source=/usr/local/intel-bad-user-agents.dat,
reader=Input::READER_ASCII, mode=Input::REREAD,
name=intel-/usr/local/intel-bad-user-agents.dat, fields=<no value
description>, want_record=T, ev=Intel::read_entry
0.000000
/usr/local/bro/share/bro/base/frameworks/intel/./input.bro:17??? event
called: Intel::read_entry(desc =
'[source=/usr/local/intel-bad-user-agents.dat,
reader=Input::READER_ASCII, mode=Input::REREAD,
name=intel-/usr/local/intel-bad-user-agents.dat, fields=<no value
description>, want_record=T, ev=Intel::read_entry
}]', tpe = 'Input::EVENT_NEW', item = '[indicator=360Spider,
indicator_type=Intel::SOFTWARE, meta=[source=mysource,
desc=<uninitialized>, url=<uninitialized>, do_notice=F,
if_in=<uninitialized>]]')
0.000000
/usr/local/bro/share/bro/base/frameworks/intel/./input.bro:17??? event
called: Intel::read_entry(desc =
'[source=/usr/local/intel-bad-user-agents.dat,
reader=Input::READER_ASCII, mode=Input::REREAD,
name=intel-/usr/local/intel-bad-user-agents.dat, fields=<no value
description>, want_record=T, ev=Intel::read_entry
}]', tpe = 'Input::EVENT_NEW', item = '[indicator=Firefox,
indicator_type=Intel::SOFTWARE, meta=[source=mysource,
desc=<uninitialized>, url=<uninitialized>, do_notice=F,
if_in=<uninitialized>]]')
0.000000
/usr/local/bro/share/bro/base/frameworks/intel/./input.bro:17??? event
called: Intel::read_entry(desc =
'[source=/usr/local/intel-bad-user-agents.dat,
reader=Input::READER_ASCII, mode=Input::REREAD,
name=intel-/usr/local/intel-bad-user-agents.dat, fields=<no value
description>, want_record=T, ev=Intel::read_entry
}]', tpe = 'Input::EVENT_NEW', item = '[indicator=192.168.89.130,
indicator_type=Intel::ADDR, meta=[source=mysource, desc=<uninitialized>,
url=<uninitialized>, do_notice=F, if_in=<uninitialized>]]')
0.000000
/usr/local/bro/share/bro/base/frameworks/input/./main.bro:248??? event
called: Input::end_of_data(name =
'intel-/usr/local/intel-bad-user-agents.dat', source =
'/usr/local/intel-bad-user-agents.dat')


On 11/16/18 4:09 PM, Lee Shiry wrote:
> I removed the comma, and added a line in the dat file using
> Intel::ADDR, still no intel.log.
>
> On 11/16/18 4:03 PM, fatema bannatwala wrote:
>> Hey,
>>
>> Just a quick check, Bro won't generate the intel.log if it's unable
>> to load the intel input file to read from.
>> was looking at your intel file re-definition:
>>
>> redef Intel::read_files += {
>> ? ? ? ? ? "/usr/local/intel-bad-user-agents.dat",
>> };
>>
>> Can you remove the trailing "," after
>> "/usr/local/intel-bad-user-agents.dat" line and see if it works.
>> I am not sure if that line should be ended with a comma.
>>
>> Also,can you try with an "Intel::ADDR" type just to check if it's
>> getting triggered?
>> You can add any IP that you can test with?Intel::ADDR and see if that
>> works.
>>
>> Fatema
>>
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181116/3c235028/attachment.html 

From jazoff at illinois.edu  Fri Nov 16 13:43:57 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Fri, 16 Nov 2018 21:43:57 +0000
Subject: [Bro] Help with intel framework
In-Reply-To: <cbd62d07-93f4-51f4-9d5f-4184daa8e171@shiry.org>
References: <ae422f90-f5af-1982-018c-d671f6945f19@shiry.org>
	<CAL8PkUX-y7ntxOipspD=7MuvWnZK77Jx2pNXLW=C0B1xRhN=oA@mail.gmail.com>
	<CAL8PkUXt1WOaetfPpAy2pRzx=qN2g=H+y1zCQYdyvk8PPJPxEg@mail.gmail.com>
	<cc28fc06-d1c7-63af-6deb-b3582eaf33eb@shiry.org>,
	<cbd62d07-93f4-51f4-9d5f-4184daa8e171@shiry.org>
Message-ID: <SN6PR11MB3216CC1CC7A0F8F4BD3DB1E0A5DD0@SN6PR11MB3216.namprd11.prod.outlook.com>

The HTTP::IN_USER_AGENT_HEADER needs to be an exact match, so unless you used a user agent of just "Mozilla" you would never get a hit.
________________________________
From: bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Lee Shiry <lee at shiry.org>
Sent: Friday, November 16, 2018 2:44:33 PM
To: Michael Shirk
Cc: bro
Subject: Re: [Bro] Help with intel framework

I have tried several things in offline mode.  I am searching for "Firefox" in "Intel::IN_ANYWHERE", and still no intel hits, even though "firefox" clealry shows up in the http.log in the user agent field.

On 11/16/18 1:28 PM, Lee Shiry wrote:
I took a pcap file and ran it standalone against the file.  This time it complained that the requested field meta.source was missing.  I don't know why that was not showing up in the reporter or stderr logs.  I added the field, and now there are no errors, but still no intel.log.

On 11/16/18 12:47 PM, Michael Shirk wrote:

And I misread you already did check reporter. Sorry for the noise. I
would get a pcap and test this offline with the intel framework to
make sure everything is working as it should.
On Fri, Nov 16, 2018 at 12:44 PM Michael Shirk <shirkdog.bsd at gmail.com><mailto:shirkdog.bsd at gmail.com> wrote:


The most important thing is the format of that ".dat" file. If you do
not have tabs entered correctly, the files may not be loaded. Check
your "reporter.log" to see if there are any errors with the input of
your intel file.

Example error:
0.000000        Reporter::WARNING
/nsm/intel/custom.intel/Input::READER_ASCII: Field: meta.do_notice
Invalid value for boolean: meta.do_notice    (empty)
On Fri, Nov 16, 2018 at 12:24 PM Lee Shiry <lee at shiry.org><mailto:lee at shiry.org> wrote:


Hi,

I am trying to use Bro's intel framework and can't seem to get it to generate anything in the intel or notice logs.  I'm on version 2.5.5 in cluster mode.  Everything else seems to work fine.  I see all the logs, and notices are working for other event types. I have checked to make sure the dat file has only tabs in it to separate fields.  I don't see anything coming up in the stderr or reporter log files.  I must be missing something.  Any help is appreciated.

Here is what I have added to local.bro:

##################
@load frameworks/intel/seen
@load frameworks/intel/do_notice

redef Intel::read_files += {
        "/usr/local/intel-bad-user-agents.dat",
};
##################


Here is the dat file:

##################
# cat /usr/local/intel-bad-user-agents.dat
#fields    indicator    indicator_type    meta.do_notice    meta.if_in
360Spider    Intel::SOFTWARE    T    HTTP::IN_USER_AGENT_HEADER
Mozilla    Intel::SOFTWARE    T    HTTP::IN_USER_AGENT_HEADER
##################

(I temporarily put Mozilla in there to generate lots of events for testing purposes)


Thanks,
lms
_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro<https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.ICSI.Berkeley.EDU_mailman_listinfo_bro&d=DwMDaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=QsGNffSpnvD7fMueBVWQjvpanS3NbAcDdaRSUvjTSvA&s=n7mxkUBQ8GtWKLM0DmUkWCvuwo_fX_Xu-RRdTLD10MU&e=>



--
Michael Shirk
Daemon Security, Inc.
https://www.daemon-security.com<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.daemon-2Dsecurity.com&d=DwMDaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=QsGNffSpnvD7fMueBVWQjvpanS3NbAcDdaRSUvjTSvA&s=0nyVXTNBSY8avOrgUQhsSV_lBptzcSfLMPkMWQ1X4io&e=>








_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro<https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.ICSI.Berkeley.EDU_mailman_listinfo_bro&d=DwMDaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=QsGNffSpnvD7fMueBVWQjvpanS3NbAcDdaRSUvjTSvA&s=n7mxkUBQ8GtWKLM0DmUkWCvuwo_fX_Xu-RRdTLD10MU&e=>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181116/6242f998/attachment-0001.html 

From alshaboti.it at gmail.com  Fri Nov 16 16:47:47 2018
From: alshaboti.it at gmail.com (Mohammed Alshaboti)
Date: Sat, 17 Nov 2018 13:47:47 +1300
Subject: [Bro] React based on Bro event (block/unblock connection)
In-Reply-To: <CAMzgZ0Je=ty_He+=aBfY63gRBfRTscqFZnpjXwg5EtCR03--VA@mail.gmail.com>
References: <CAKbXeiAaa1gsAhfiV+h_cZracMjWJHe+9nTssKo7kDPWM=RVLA@mail.gmail.com>
	<CAMzgZ0Je=ty_He+=aBfY63gRBfRTscqFZnpjXwg5EtCR03--VA@mail.gmail.com>
Message-ID: <CAKbXeiA0SzR0F72_dUO=1umyru7gNnUR3Ne7uSuph6zEjnDfDg@mail.gmail.com>

Hi Jone,
I modified the code many times, and couldn't reproduce the error.
Simply I modified this
https://github.com/bro/bro-netcontrol/blob/master/test/simple-test.bro to
add event
https://github.com/bro/bro-netcontrol/blob/master/test/simple-client.py

I would like to add rules inside  'connection_established()' event rather
than in netControl::init(), so python script can react based on established
connection_established event.
""
@load base/frameworks/netcontrol
redef exit_only_after_terminate = T;
event NetControl::init()
        {
        local netcontrol_broker =
NetControl::create_broker(NetControl::BrokerConfig($host=127.0.0.1,
$bport=9977/tcp, $topic="bro/event/ne$
        NetControl::activate(netcontrol_broker, 0);
        }
event NetControl::init_done() &priority=-5
        {
        print "Init done";
        # drop rule goes through to simple-client.py
        NetControl::drop_address(1.1.2.2, 15sec, "Hi there");
        }
event connection_established(c: connection)
    {
   # can't receive this drop in simple-client.py, only it gets
connectionestablished not the drop rule!!
        NetControl::drop_address(1.1.2.2, 15sec, "Hi there");
     }
""
however, I only on python client I get connection_established but not the
drop rule of  NetControl::drop_address

I run it like this:
bro -C -r ../traces/tls/ecdhe.pcap simple-test.bro

python simple-client.py
 ?netcontrol-3-ssh-guesser.bro      todo.txt
DEBUG:netcontrol.api:Set up listener for 127.0.0.1:9977
(bro/event/net?netcontrol-9-skeleton.bro         weird.log
control-example)
?netcontrol-9-use-skeleton.bro     x509.log
DEBUG:netcontrol.api:Waiting for broker message...
?netcontrol.log
DEBUG:netcontrol.api:Handling broker status message...
?pi at raspberrypi:~/test_bro $
INFO:netcontrol.api:Incoming connection established                   ?rm
*.log
<ResponseType.ConnectionEstablished: 1>

Thank you,
Mohammed



The python program crash or give me communication established

On Fri, Nov 16, 2018 at 6:02 AM Jon Siwek <jsiwek at corelight.com> wrote:

> On Wed, Nov 14, 2018 at 11:38 PM Mohammed Alshaboti
> <alshaboti.it at gmail.com> wrote:
>
> > I would like to send Bro data (e.g. connection) to a backend python
> program on some events.
> > I tried to use the netcontrol broker to communicate with an external
> > python client like (
> https://github.com/bro/bro-netcontrol/tree/master/test
> > ).
> > But when I added event it crashed.
>
> Can you provide more info?  e.g. exact code that you're trying.  Was
> it bro or the python program that crashed?  Any other relevant output
> or error messages?
>
> - Jon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181117/e0d16b35/attachment.html 

From jazoff at illinois.edu  Fri Nov 16 18:39:49 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Sat, 17 Nov 2018 02:39:49 +0000
Subject: [Bro] React based on Bro event (block/unblock connection)
In-Reply-To: <CAKbXeiA0SzR0F72_dUO=1umyru7gNnUR3Ne7uSuph6zEjnDfDg@mail.gmail.com>
References: <CAKbXeiAaa1gsAhfiV+h_cZracMjWJHe+9nTssKo7kDPWM=RVLA@mail.gmail.com>
	<CAMzgZ0Je=ty_He+=aBfY63gRBfRTscqFZnpjXwg5EtCR03--VA@mail.gmail.com>,
	<CAKbXeiA0SzR0F72_dUO=1umyru7gNnUR3Ne7uSuph6zEjnDfDg@mail.gmail.com>
Message-ID: <SN6PR11MB3216AA4D0CF0624EC064E937A5DE0@SN6PR11MB3216.namprd11.prod.outlook.com>

You may be running into a common race condition where the pcap file is read before the netcontrol broker connection is initialized.  There are 2 ways of going about testing this differently.  The first way would be to run bro on live traffic by using -i eth0 instead of reading a pcap file.  I would also change


NetControl::drop_address(1.1.2.2, 15sec, "Hi there");

to

NetControl::drop_address(c$id$resp_h, 15sec, "Hi there");

so that for each connection bro sees it will try to drop a different address and not just 1.1.2.2 each time.  I believe netcontrol tracks drops internally so by dropping the same 1.1.2.2 each time you would only see one broker message every 15 seconds instead of each time.


If you need to test using a pcap file you should be able to use the method that is used in the test suite: https://github.com/bro/bro/blob/master/testing/btest/scripts/base/frameworks/netcontrol/broker.bro

Essentially you would add a

event bro_init()
{
suspend_processing();
}

so that bro pauses processing of the pcap traffic as soon as it starts.  Then, inside NetControl::init_done you would call continue_processing().  This way the pcap is only analyzed after netcontrol is fully initialized.



________________________________
From: bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Mohammed Alshaboti <alshaboti.it at gmail.com>
Sent: Friday, November 16, 2018 7:47:47 PM
To: jsiwek at corelight.com
Cc: Bro at bro.org
Subject: Re: [Bro] React based on Bro event (block/unblock connection)

Hi Jone,
I modified the code many times, and couldn't reproduce the error.
Simply I modified this https://github.com/bro/bro-netcontrol/blob/master/test/simple-test.bro<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_bro_bro-2Dnetcontrol_blob_master_test_simple-2Dtest.bro&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=KEa-1-nZ6w_QYP6Aa8AwgNIGKGoLVgQvWco4qasKkOU&s=RuJ6dVV1LvAnLJF7Gr8UIV295b5v5ZG5q0BlWhyyROM&e=> to add event
https://github.com/bro/bro-netcontrol/blob/master/test/simple-client.py<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_bro_bro-2Dnetcontrol_blob_master_test_simple-2Dclient.py&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=KEa-1-nZ6w_QYP6Aa8AwgNIGKGoLVgQvWco4qasKkOU&s=-G5dZDne3LT8FSfKkRWuWYrkhfhGhZFOkul383f6vzA&e=>

I would like to add rules inside  'connection_established()' event rather than in netControl::init(), so python script can react based on established connection_established event.
""
@load base/frameworks/netcontrol
redef exit_only_after_terminate = T;
event NetControl::init()
        {
        local netcontrol_broker = NetControl::create_broker(NetControl::BrokerConfig($host=127.0.0.1, $bport=9977/tcp, $topic="bro/event/ne$
        NetControl::activate(netcontrol_broker, 0);
        }
event NetControl::init_done() &priority=-5
        {
        print "Init done";
        # drop rule goes through to simple-client.py
        NetControl::drop_address(1.1.2.2, 15sec, "Hi there");
        }
event connection_established(c: connection)
    {
   # can't receive this drop in simple-client.py, only it gets connectionestablished not the drop rule!!
        NetControl::drop_address(1.1.2.2, 15sec, "Hi there");
     }
""
however, I only on python client I get connection_established but not the drop rule of  NetControl::drop_address

I run it like this:
bro -C -r ../traces/tls/ecdhe.pcap simple-test.bro

python simple-client.py                                               ?netcontrol-3-ssh-guesser.bro      todo.txt
DEBUG:netcontrol.api:Set up listener for 127.0.0.1:9977<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A9977&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=KEa-1-nZ6w_QYP6Aa8AwgNIGKGoLVgQvWco4qasKkOU&s=kRhaqblI7XVAaKhbJO8R3yisk_ipC8x0w85INov1PiY&e=> (bro/event/net?netcontrol-9-skeleton.bro         weird.log
control-example)                                                      ?netcontrol-9-use-skeleton.bro     x509.log
DEBUG:netcontrol.api:Waiting for broker message...                    ?netcontrol.log
DEBUG:netcontrol.api:Handling broker status message...                ?pi at raspberrypi:~/test_bro $
INFO:netcontrol.api:Incoming connection established                   ?rm *.log
<ResponseType.ConnectionEstablished: 1>

Thank you,
Mohammed



The python program crash or give me communication established

On Fri, Nov 16, 2018 at 6:02 AM Jon Siwek <jsiwek at corelight.com<mailto:jsiwek at corelight.com>> wrote:
On Wed, Nov 14, 2018 at 11:38 PM Mohammed Alshaboti
<alshaboti.it at gmail.com<mailto:alshaboti.it at gmail.com>> wrote:

> I would like to send Bro data (e.g. connection) to a backend python program on some events.
> I tried to use the netcontrol broker to communicate with an external
> python client like (https://github.com/bro/bro-netcontrol/tree/master/test<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_bro_bro-2Dnetcontrol_tree_master_test&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=KEa-1-nZ6w_QYP6Aa8AwgNIGKGoLVgQvWco4qasKkOU&s=9uwugavHogTI8cWBQ6wifv6D9d9yC0D5bN2vuZWYg8E&e=>
> ).
> But when I added event it crashed.

Can you provide more info?  e.g. exact code that you're trying.  Was
it bro or the python program that crashed?  Any other relevant output
or error messages?

- Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181117/0e71412e/attachment-0001.html 

From jazoff at illinois.edu  Fri Nov 16 18:59:39 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Sat, 17 Nov 2018 02:59:39 +0000
Subject: [Bro] Getting a Broctl Stack Trace
In-Reply-To: <CAK051ai0=gFXbd96BOZG9RddE6qA-vB4XY+i+em=PotNv2vJkA@mail.gmail.com>
References: <CAK051agH+hWgr-bZrfqVBQEKHZrU9AkCsb=CwFx4ohUp+YfkuA@mail.gmail.com>
	<AB6568DA-7AD5-4D84-B1AB-AD00877F7114@corelight.com>,
	<CAK051ai0=gFXbd96BOZG9RddE6qA-vB4XY+i+em=PotNv2vJkA@mail.gmail.com>
Message-ID: <SN6PR11MB32164A06FA71E0AB3A053157A5DE0@SN6PR11MB3216.namprd11.prod.outlook.com>

I'm not 100% sure about the root cause, but I know one thing that may help.. there's a code path for 'broctl start' that will say something has "crashed" when it is "not running".. but "not running" doesn't have to be a segfault... just that it didn't fully initialize in the way that broctl was expecting it to.


Hosom was also looking into this today and said he was seeing:


warning in /usr/local/bro/share/bro/base/init-bare.bro, line 1: problem initializing NB-DNS: no valid nameservers in resolver config


It may be a red herring, but that's easy to rule out:


add


env_vars=BRO_DNS_FAKE=1


to broctl.cfg


and deploy and see if things start properly with real DNS disabled.


________________________________
From: bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Mike M <turbidtarantula at gmail.com>
Sent: Tuesday, November 13, 2018 11:22:10 AM
To: seth at corelight.com
Cc: bro at bro.org
Subject: Re: [Bro] Getting a Broctl Stack Trace

I gave this a shot but I'm still not seeing a core file. I tried both the setting you recommended and setting an absolute path to /tmp. When I force a core dump on another process the core file shows up as expected, but broctl isn't producing one.

I'm open to suggestions on this one... not sure how to determine the root cause.

thanks,
Mike

On Mon, Nov 5, 2018 at 5:01 PM Seth Hall <seth at corelight.com<mailto:seth at corelight.com>> wrote:
Make sure you are setting the core pattern on your system so that the
core dump will be written into the CWD.

sudo sysctl -w kernel.core_pattern="core.%e-%t-%p"

   .Seth

On 2 Nov 2018, at 12:51, Mike M wrote:

> I'm having an issue with broctl crashing when I try to run it on
> Alpine
> Linux. I mentioned it previously [1] but I'm circling back around to
> try to
> get it resolved. I've built it with the appropriate patches [2] but
> broctl
> is still reporting "crashed" state when I checks the status after
> starting
> it. The bro binary itself runs fine.
>
> What do I need to do to collect a stack trace from broctl to determine
> the
> root cause?
>
> Bro is built in debug mode and I set "ulimit -c unlimited" per the
> instructions on reporting problems. I see a
> /usr/local/bro/spool/tmp/post-terminate-standalone-2018-11-02-02-56-06-13765-crash
> directory but there's no core dump anywhere obvious. The
> .crash-diag.out
> file says "No core file found" and doesn't provide any useful
> information
> about the cause of the crash.
>
> Thanks,
> Mike
>
> [1]
> http://mailman.icsi.berkeley.edu/pipermail/bro/2018-September/013580.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.icsi.berkeley.edu_pipermail_bro_2018-2DSeptember_013580.html&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=XNJ9sWJIJe1PnbIhx0wdY5wWHuR6-JXLc9LyKMMaaeY&s=6-Snv23fdhhesPXq1ctSBZKMk7OC-nGgFfdKgfxFczA&e=>
> [2]
> http://mailman.icsi.berkeley.edu/pipermail/bro/2018-September/013581.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.icsi.berkeley.edu_pipermail_bro_2018-2DSeptember_013581.html&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=XNJ9sWJIJe1PnbIhx0wdY5wWHuR6-JXLc9LyKMMaaeY&s=isM59BgpWPxaWENNIit-XpqFwCdcUtlDM2P3prrsMk8&e=>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org<mailto:bro at bro-ids.org>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro<https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.ICSI.Berkeley.EDU_mailman_listinfo_bro&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=XNJ9sWJIJe1PnbIhx0wdY5wWHuR6-JXLc9LyKMMaaeY&s=deY5uW7Ow0QD4FZmgjaC7hwBcdB5GPcZ52CcOiq2m8Q&e=>

--
Seth Hall * Corelight, Inc * www.corelight.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.corelight.com&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=XNJ9sWJIJe1PnbIhx0wdY5wWHuR6-JXLc9LyKMMaaeY&s=bXYMmbM6QwdeBb_Dnnc91CEJSnxe-T7MPXIPjs_b2us&e=>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181117/f8e47c37/attachment.html 

From nothinrandom at gmail.com  Sat Nov 17 10:50:23 2018
From: nothinrandom at gmail.com (TQ)
Date: Sat, 17 Nov 2018 10:50:23 -0800
Subject: [Bro] Accessing index of an uint8 array inside record
Message-ID: <CAM2MKSpfOgWHi1Mt5zUPB9vwq0Fk7tRzVnCHTNDT8LhjgrGbeQ@mail.gmail.com>

What is the correct way of accessing the index of an array of uint8 inside
a record?

In protocol.pac, I have:

type Record_A = record {
    record_b : Record_B;
} &byteorder=littleendian;

type Record_B = record {
    data: uint8[4];
} &byteorder=littleendian;


In analyzer.pac, I tried using ${Record_A.record_b.data[0]}, but the log
becomes empty.  However, if I change data from uint8[4] to uint32 and used
${Record_A.record_b.data}, then the log gets generated correctly.  I also
tried: test->Assign(i, new Val((*Record_A.record_b.data)[0], TYPE_COUNT));
but no luck.  I'm probably missing something silly here, but can't seem to
figure it out.  Welcome all pointers!

Thanks,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181117/8caf0080/attachment.html 

From jsiwek at corelight.com  Sun Nov 18 07:05:18 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Sun, 18 Nov 2018 09:05:18 -0600
Subject: [Bro] Accessing index of an uint8 array inside record
In-Reply-To: <CAM2MKSpfOgWHi1Mt5zUPB9vwq0Fk7tRzVnCHTNDT8LhjgrGbeQ@mail.gmail.com>
References: <CAM2MKSpfOgWHi1Mt5zUPB9vwq0Fk7tRzVnCHTNDT8LhjgrGbeQ@mail.gmail.com>
Message-ID: <CAMzgZ0J5uV8+wZrG87j75Hn_OTJgXqP005AZFpj_vsMO6AadxQ@mail.gmail.com>

On Sat, Nov 17, 2018 at 1:06 PM TQ <nothinrandom at gmail.com> wrote:
>
> What is the correct way of accessing the index of an array of uint8 inside a record?
>
> In protocol.pac, I have:
>
> type Record_A = record {
>     record_b : Record_B;
> } &byteorder=littleendian;
>
> type Record_B = record {
>     data: uint8[4];
> } &byteorder=littleendian;
>
>
> In analyzer.pac, I tried using ${Record_A.record_b.data[0]}, but the log becomes empty.  However, if I change data from uint8[4] to uint32 and used ${Record_A.record_b.data}, then the log gets generated correctly.

How about this:

${Record_A.record_b.data}[0]

I didn't explicitly try it out, but just assuming that the syntax is
only sophisticated enough to recognize fields, not arrays, so
${Record_A.record_b.data} transforms from uint8[] (binpac) to a uint8*
(c++) and then you index into that.

- Jon

From nothinrandom at gmail.com  Sun Nov 18 08:44:58 2018
From: nothinrandom at gmail.com (TQ)
Date: Sun, 18 Nov 2018 08:44:58 -0800
Subject: [Bro] Accessing index of an uint8 array inside record
In-Reply-To: <CAMzgZ0J5uV8+wZrG87j75Hn_OTJgXqP005AZFpj_vsMO6AadxQ@mail.gmail.com>
References: <CAM2MKSpfOgWHi1Mt5zUPB9vwq0Fk7tRzVnCHTNDT8LhjgrGbeQ@mail.gmail.com>
	<CAMzgZ0J5uV8+wZrG87j75Hn_OTJgXqP005AZFpj_vsMO6AadxQ@mail.gmail.com>
Message-ID: <CAM2MKSrmK+XVVy4t6X=gSbnWLX=0r5tv6c-KEhqZ1wRe4BcgdA@mail.gmail.com>

Hey Jon,

I tried that, but the log is empty for some reason, with only the headers
being populated.  Dpd.log wasn't generated, so I couldn't figure out
error.  I ended up using bytestring &length=4, and this allows me to access
using ${Record_A.record_b.data[0]}.  Bizarre as these two are equivalent,
right?

Thanks,

On Sun, Nov 18, 2018 at 7:05 AM Jon Siwek <jsiwek at corelight.com> wrote:

> On Sat, Nov 17, 2018 at 1:06 PM TQ <nothinrandom at gmail.com> wrote:
> >
> > What is the correct way of accessing the index of an array of uint8
> inside a record?
> >
> > In protocol.pac, I have:
> >
> > type Record_A = record {
> >     record_b : Record_B;
> > } &byteorder=littleendian;
> >
> > type Record_B = record {
> >     data: uint8[4];
> > } &byteorder=littleendian;
> >
> >
> > In analyzer.pac, I tried using ${Record_A.record_b.data[0]}, but the log
> becomes empty.  However, if I change data from uint8[4] to uint32 and used
> ${Record_A.record_b.data}, then the log gets generated correctly.
>
> How about this:
>
> ${Record_A.record_b.data}[0]
>
> I didn't explicitly try it out, but just assuming that the syntax is
> only sophisticated enough to recognize fields, not arrays, so
> ${Record_A.record_b.data} transforms from uint8[] (binpac) to a uint8*
> (c++) and then you index into that.
>
> - Jon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181118/0b1ab767/attachment.html 

From jsiwek at corelight.com  Sun Nov 18 09:28:45 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Sun, 18 Nov 2018 11:28:45 -0600
Subject: [Bro] Accessing index of an uint8 array inside record
In-Reply-To: <CAM2MKSrmK+XVVy4t6X=gSbnWLX=0r5tv6c-KEhqZ1wRe4BcgdA@mail.gmail.com>
References: <CAM2MKSpfOgWHi1Mt5zUPB9vwq0Fk7tRzVnCHTNDT8LhjgrGbeQ@mail.gmail.com>
	<CAMzgZ0J5uV8+wZrG87j75Hn_OTJgXqP005AZFpj_vsMO6AadxQ@mail.gmail.com>
	<CAM2MKSrmK+XVVy4t6X=gSbnWLX=0r5tv6c-KEhqZ1wRe4BcgdA@mail.gmail.com>
Message-ID: <CAMzgZ0L2n=aeDY5UhmSXgOMqwzzbjXmBQVRoiuUM6PYUozA3Xw@mail.gmail.com>

On Sun, Nov 18, 2018 at 10:45 AM TQ <nothinrandom at gmail.com> wrote:

> I ended up using bytestring &length=4, and this allows me to access using ${Record_A.record_b.data[0]}.  Bizarre as these two are equivalent, right?

Logically, "bytestring &length=4" and uint8[4] are equivalent but seem
to be implemented differently.

A bytestring field ends up being a "datastring<uint8>"
object/reference from binpac_bytestring.h

A uint8[4] ends up being a pointer to a "std::vector<uint8>".

So that last bit about it being a pointer is likely important, means
you were probably missing a pointer dereference e.g. you could try
${Record_A.record_b.data}->at(0) for accessing a uint8[] at index 0.

All this info should be available for you to find in the generated
*_pac.cc files and checking those is the best way to confirm what you
need to be doing.

- Jon

From nothinrandom at gmail.com  Sun Nov 18 09:30:23 2018
From: nothinrandom at gmail.com (TQ)
Date: Sun, 18 Nov 2018 09:30:23 -0800
Subject: [Bro] Accessing index of an uint8 array inside record
In-Reply-To: <CAMzgZ0L2n=aeDY5UhmSXgOMqwzzbjXmBQVRoiuUM6PYUozA3Xw@mail.gmail.com>
References: <CAM2MKSpfOgWHi1Mt5zUPB9vwq0Fk7tRzVnCHTNDT8LhjgrGbeQ@mail.gmail.com>
	<CAMzgZ0J5uV8+wZrG87j75Hn_OTJgXqP005AZFpj_vsMO6AadxQ@mail.gmail.com>
	<CAM2MKSrmK+XVVy4t6X=gSbnWLX=0r5tv6c-KEhqZ1wRe4BcgdA@mail.gmail.com>
	<CAMzgZ0L2n=aeDY5UhmSXgOMqwzzbjXmBQVRoiuUM6PYUozA3Xw@mail.gmail.com>
Message-ID: <CAM2MKSrxXi=R_qjzx6gQqkKRa4Pbz8_RfoAPneTuKcKyfi8FcA@mail.gmail.com>

Ah, ok. Thanks for this tip!

On Sun, Nov 18, 2018 at 9:28 AM Jon Siwek <jsiwek at corelight.com> wrote:

> On Sun, Nov 18, 2018 at 10:45 AM TQ <nothinrandom at gmail.com> wrote:
>
> > I ended up using bytestring &length=4, and this allows me to access
> using ${Record_A.record_b.data[0]}.  Bizarre as these two are equivalent,
> right?
>
> Logically, "bytestring &length=4" and uint8[4] are equivalent but seem
> to be implemented differently.
>
> A bytestring field ends up being a "datastring<uint8>"
> object/reference from binpac_bytestring.h
>
> A uint8[4] ends up being a pointer to a "std::vector<uint8>".
>
> So that last bit about it being a pointer is likely important, means
> you were probably missing a pointer dereference e.g. you could try
> ${Record_A.record_b.data}->at(0) for accessing a uint8[] at index 0.
>
> All this info should be available for you to find in the generated
> *_pac.cc files and checking those is the best way to confirm what you
> need to be doing.
>
> - Jon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181118/3e37688c/attachment.html 

From alshaboti.it at gmail.com  Sun Nov 18 12:59:23 2018
From: alshaboti.it at gmail.com (Mohammed Alshaboti)
Date: Mon, 19 Nov 2018 09:59:23 +1300
Subject: [Bro] Bro Digest, Vol 151, Issue 16
In-Reply-To: <mailman.72230.1542422395.837.bro@bro.org>
References: <mailman.72230.1542422395.837.bro@bro.org>
Message-ID: <CAKbXeiAKTpzb1wk8jXxzhwequtJPOp8F+jNTf_Gopbu2CTY41Q@mail.gmail.com>

Thanks Jon
It works when I use it with an online data ( -i eth0), as you said it seems
to be race condition.

Thanks

On Sat, Nov 17, 2018 at 3:40 PM <bro-request at bro.org> wrote:

> Send Bro mailing list submissions to
>         bro at bro.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> or, via email, send a message with subject or body 'help' to
>         bro-request at bro.org
>
> You can reach the person managing the list at
>         bro-owner at bro.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Bro digest..."
>
>
> Today's Topics:
>
>    1. Re: React based on Bro event (block/unblock connection)
>       (Mohammed Alshaboti)
>    2. Re: React based on Bro event (block/unblock connection)
>       (Azoff, Justin S)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 17 Nov 2018 13:47:47 +1300
> From: Mohammed Alshaboti <alshaboti.it at gmail.com>
> Subject: Re: [Bro] React based on Bro event (block/unblock connection)
> To: jsiwek at corelight.com
> Cc: Bro at bro.org
> Message-ID:
>         <CAKbXeiA0SzR0F72_dUO=
> 1umyru7gNnUR3Ne7uSuph6zEjnDfDg at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Jone,
> I modified the code many times, and couldn't reproduce the error.
> Simply I modified this
> https://github.com/bro/bro-netcontrol/blob/master/test/simple-test.bro to
> add event
> https://github.com/bro/bro-netcontrol/blob/master/test/simple-client.py
>
> I would like to add rules inside  'connection_established()' event rather
> than in netControl::init(), so python script can react based on established
> connection_established event.
> ""
> @load base/frameworks/netcontrol
> redef exit_only_after_terminate = T;
> event NetControl::init()
>         {
>         local netcontrol_broker =
> NetControl::create_broker(NetControl::BrokerConfig($host=127.0.0.1,
> $bport=9977/tcp, $topic="bro/event/ne$
>         NetControl::activate(netcontrol_broker, 0);
>         }
> event NetControl::init_done() &priority=-5
>         {
>         print "Init done";
>         # drop rule goes through to simple-client.py
>         NetControl::drop_address(1.1.2.2, 15sec, "Hi there");
>         }
> event connection_established(c: connection)
>     {
>    # can't receive this drop in simple-client.py, only it gets
> connectionestablished not the drop rule!!
>         NetControl::drop_address(1.1.2.2, 15sec, "Hi there");
>      }
> ""
> however, I only on python client I get connection_established but not the
> drop rule of  NetControl::drop_address
>
> I run it like this:
> bro -C -r ../traces/tls/ecdhe.pcap simple-test.bro
>
> python simple-client.py
>  ?netcontrol-3-ssh-guesser.bro      todo.txt
> DEBUG:netcontrol.api:Set up listener for 127.0.0.1:9977
> (bro/event/net?netcontrol-9-skeleton.bro         weird.log
> control-example)
> ?netcontrol-9-use-skeleton.bro     x509.log
> DEBUG:netcontrol.api:Waiting for broker message...
> ?netcontrol.log
> DEBUG:netcontrol.api:Handling broker status message...
> ?pi at raspberrypi:~/test_bro $
> INFO:netcontrol.api:Incoming connection established                   ?rm
> *.log
> <ResponseType.ConnectionEstablished: 1>
>
> Thank you,
> Mohammed
>
>
>
> The python program crash or give me communication established
>
> On Fri, Nov 16, 2018 at 6:02 AM Jon Siwek <jsiwek at corelight.com> wrote:
>
> > On Wed, Nov 14, 2018 at 11:38 PM Mohammed Alshaboti
> > <alshaboti.it at gmail.com> wrote:
> >
> > > I would like to send Bro data (e.g. connection) to a backend python
> > program on some events.
> > > I tried to use the netcontrol broker to communicate with an external
> > > python client like (
> > https://github.com/bro/bro-netcontrol/tree/master/test
> > > ).
> > > But when I added event it crashed.
> >
> > Can you provide more info?  e.g. exact code that you're trying.  Was
> > it bro or the python program that crashed?  Any other relevant output
> > or error messages?
> >
> > - Jon
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181117/e0d16b35/attachment-0001.html
>
> ------------------------------
>
> Message: 2
> Date: Sat, 17 Nov 2018 02:39:49 +0000
> From: "Azoff, Justin S" <jazoff at illinois.edu>
> Subject: Re: [Bro] React based on Bro event (block/unblock connection)
> To: Mohammed Alshaboti <alshaboti.it at gmail.com>,
>         "jsiwek at corelight.com"  <jsiwek at corelight.com>
> Cc: "Bro at bro.org" <Bro at bro.org>
> Message-ID:
>         <
> SN6PR11MB3216AA4D0CF0624EC064E937A5DE0 at SN6PR11MB3216.namprd11.prod.outlook.com
> >
>
> Content-Type: text/plain; charset="koi8-r"
>
> You may be running into a common race condition where the pcap file is
> read before the netcontrol broker connection is initialized.  There are 2
> ways of going about testing this differently.  The first way would be to
> run bro on live traffic by using -i eth0 instead of reading a pcap file.  I
> would also change
>
>
> NetControl::drop_address(1.1.2.2, 15sec, "Hi there");
>
> to
>
> NetControl::drop_address(c$id$resp_h, 15sec, "Hi there");
>
> so that for each connection bro sees it will try to drop a different
> address and not just 1.1.2.2 each time.  I believe netcontrol tracks drops
> internally so by dropping the same 1.1.2.2 each time you would only see one
> broker message every 15 seconds instead of each time.
>
>
> If you need to test using a pcap file you should be able to use the method
> that is used in the test suite:
> https://github.com/bro/bro/blob/master/testing/btest/scripts/base/frameworks/netcontrol/broker.bro
>
> Essentially you would add a
>
> event bro_init()
> {
> suspend_processing();
> }
>
> so that bro pauses processing of the pcap traffic as soon as it starts.
> Then, inside NetControl::init_done you would call continue_processing().
> This way the pcap is only analyzed after netcontrol is fully initialized.
>
>
>
> ________________________________
> From: bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Mohammed
> Alshaboti <alshaboti.it at gmail.com>
> Sent: Friday, November 16, 2018 7:47:47 PM
> To: jsiwek at corelight.com
> Cc: Bro at bro.org
> Subject: Re: [Bro] React based on Bro event (block/unblock connection)
>
> Hi Jone,
> I modified the code many times, and couldn't reproduce the error.
> Simply I modified this
> https://github.com/bro/bro-netcontrol/blob/master/test/simple-test.bro<
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_bro_bro-2Dnetcontrol_blob_master_test_simple-2Dtest.bro&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=KEa-1-nZ6w_QYP6Aa8AwgNIGKGoLVgQvWco4qasKkOU&s=RuJ6dVV1LvAnLJF7Gr8UIV295b5v5ZG5q0BlWhyyROM&e=>
> to add event
> https://github.com/bro/bro-netcontrol/blob/master/test/simple-client.py<
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_bro_bro-2Dnetcontrol_blob_master_test_simple-2Dclient.py&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=KEa-1-nZ6w_QYP6Aa8AwgNIGKGoLVgQvWco4qasKkOU&s=-G5dZDne3LT8FSfKkRWuWYrkhfhGhZFOkul383f6vzA&e=
> >
>
> I would like to add rules inside  'connection_established()' event rather
> than in netControl::init(), so python script can react based on established
> connection_established event.
> ""
> @load base/frameworks/netcontrol
> redef exit_only_after_terminate = T;
> event NetControl::init()
>         {
>         local netcontrol_broker =
> NetControl::create_broker(NetControl::BrokerConfig($host=127.0.0.1,
> $bport=9977/tcp, $topic="bro/event/ne$
>         NetControl::activate(netcontrol_broker, 0);
>         }
> event NetControl::init_done() &priority=-5
>         {
>         print "Init done";
>         # drop rule goes through to simple-client.py
>         NetControl::drop_address(1.1.2.2, 15sec, "Hi there");
>         }
> event connection_established(c: connection)
>     {
>    # can't receive this drop in simple-client.py, only it gets
> connectionestablished not the drop rule!!
>         NetControl::drop_address(1.1.2.2, 15sec, "Hi there");
>      }
> ""
> however, I only on python client I get connection_established but not the
> drop rule of  NetControl::drop_address
>
> I run it like this:
> bro -C -r ../traces/tls/ecdhe.pcap simple-test.bro
>
> python simple-client.py
>  ?netcontrol-3-ssh-guesser.bro      todo.txt
> DEBUG:netcontrol.api:Set up listener for 127.0.0.1:9977<
> https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1-3A9977&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=KEa-1-nZ6w_QYP6Aa8AwgNIGKGoLVgQvWco4qasKkOU&s=kRhaqblI7XVAaKhbJO8R3yisk_ipC8x0w85INov1PiY&e=>
> (bro/event/net?netcontrol-9-skeleton.bro         weird.log
> control-example)
> ?netcontrol-9-use-skeleton.bro     x509.log
> DEBUG:netcontrol.api:Waiting for broker message...
> ?netcontrol.log
> DEBUG:netcontrol.api:Handling broker status message...
> ?pi at raspberrypi:~/test_bro $
> INFO:netcontrol.api:Incoming connection established                   ?rm
> *.log
> <ResponseType.ConnectionEstablished: 1>
>
> Thank you,
> Mohammed
>
>
>
> The python program crash or give me communication established
>
> On Fri, Nov 16, 2018 at 6:02 AM Jon Siwek <jsiwek at corelight.com<mailto:
> jsiwek at corelight.com>> wrote:
> On Wed, Nov 14, 2018 at 11:38 PM Mohammed Alshaboti
> <alshaboti.it at gmail.com<mailto:alshaboti.it at gmail.com>> wrote:
>
> > I would like to send Bro data (e.g. connection) to a backend python
> program on some events.
> > I tried to use the netcontrol broker to communicate with an external
> > python client like (
> https://github.com/bro/bro-netcontrol/tree/master/test<
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_bro_bro-2Dnetcontrol_tree_master_test&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=KEa-1-nZ6w_QYP6Aa8AwgNIGKGoLVgQvWco4qasKkOU&s=9uwugavHogTI8cWBQ6wifv6D9d9yC0D5bN2vuZWYg8E&e=
> >
> > ).
> > But when I added event it crashed.
>
> Can you provide more info?  e.g. exact code that you're trying.  Was
> it bro or the python program that crashed?  Any other relevant output
> or error messages?
>
> - Jon
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181117/0e71412e/attachment.html
>
> ------------------------------
>
> _______________________________________________
> Bro mailing list
> Bro at bro.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> End of Bro Digest, Vol 151, Issue 16
> ************************************
>


-- 
Best regards,
Mohammed Al-Shaboti
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181119/208c7364/attachment-0001.html 

From lee at shiry.org  Mon Nov 19 10:18:56 2018
From: lee at shiry.org (Lee Shiry)
Date: Mon, 19 Nov 2018 13:18:56 -0500
Subject: [Bro]
 =?utf-8?b?5Zue5aSN77yaIEhlbHAgd2l0aCBpbnRlbCBmcmFtZXdvcms=?=
In-Reply-To: <tencent_4ADB2DA8480D9FE924047BAD9D8E25C97805@qq.com>
References: <ae422f90-f5af-1982-018c-d671f6945f19@shiry.org>
	<tencent_4ADB2DA8480D9FE924047BAD9D8E25C97805@qq.com>
Message-ID: <2af53e49-aab5-f4ab-d55a-2f9e7a2fc871@shiry.org>

Update:

I copied the script and dat file to another box and they seem to work
fine.? I'm not sure why it is not working on the first box.? It is not
logging any errors, and other things seem to work.? I will try
recompiling and reinstalling.

The other problem is that I was not aware that the intel match had to be
an exact match.? Does anyone know if it is possible to use a wildcard or
do a substring search with the intel match?? I tried "*" as a wildcard,
that does not work.

Thanks for all the help!


On 11/19/18 1:09 AM, Zer0d0y wrote:
> Hi,
> ? ??This configuration works for me.
>
> # bro -v
> /opt/bro/bin/bro version 2.5-1001-debug
>
> 1.intel.bro
> @load frameworks/intel/seen
> @load frameworks/intel/do_notice
>
> redef Intel::read_files += {
> ? ? ? ? "/path/to/intel-bad-user-agents.dat"
> };
>
> 2.intel-bad-user-agents.dat
> #fieldsindicatorindicator_typemeta.do_noticemeta.source
> MozillaIntel::SOFTWARETHTTP::IN_USER_AGENT_HEADER
>
> 3.tcpdump -nnvv -i eth0 host "IP Address of google.com" and port 80 -w
> intel.pcap
>
> 4.curl -I -A "Mozilla" http://google.com
>
> 5.bro -C -r intel.pcap intel.bro
>
> ########################
> intel.log
>
> 1542606537.171248CkvxUu1MujtxrdxMgl192.168.8.258248192.168.8.180MozillaIntel::SOFTWAREHTTP::IN_USER_AGENT_HEADERbroIntel::SOFTWAREHTTP::IN_USER_AGENT_HEADER---
> #close2018-11-19-14-06-44
>
> ------------------
> --?
> Zer0d0y
> Threat Detection & Hunting
>
> Zer0d0y at tianyulab.com
>
> ????? <https://github.com/tianyulab>
> ?
>
>
> ------------------??????------------------
> *???:*?"Lee Shiry"<lee at shiry.org>;
> *????:*?2018?11?17?(???) ??1:23
> *???:*?"bro"<bro at bro.org>;
> *??:*?[Bro] Help with intel framework
>
> Hi,
>
> I am trying to use Bro's intel framework and can't seem to get it to
> generate anything in the intel or notice logs.? I'm on version 2.5.5
> in cluster mode.? Everything else seems to work fine.? I see all the
> logs, and notices are working for other event types. I have checked to
> make sure the dat file has only tabs in it to separate fields.? I
> don't see anything coming up in the stderr or reporter log files.? I
> must be missing something.? Any help is appreciated.
>
> Here is what I have added to local.bro:
>
> ##################
> @load frameworks/intel/seen
> @load frameworks/intel/do_notice
>
> redef Intel::read_files += {
> ??????? "/usr/local/intel-bad-user-agents.dat",
> };
> ##################
>
>
> Here is the dat file:
>
> ##################
> # cat /usr/local/intel-bad-user-agents.dat
> #fields??? indicator??? indicator_type??? meta.do_notice??? meta.if_in
> 360Spider??? Intel::SOFTWARE??? T??? HTTP::IN_USER_AGENT_HEADER
> Mozilla??? Intel::SOFTWARE??? T??? HTTP::IN_USER_AGENT_HEADER
> ##################
>
> (I temporarily put Mozilla in there to generate lots of events for
> testing purposes)
>
>
> Thanks,
> lms

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181119/f0e96cf6/attachment.html 

From jlay at slave-tothe-box.net  Tue Nov 20 13:55:36 2018
From: jlay at slave-tothe-box.net (James Lay)
Date: Tue, 20 Nov 2018 14:55:36 -0700
Subject: [Bro] Bro beta install
In-Reply-To: <CAMzgZ0JRhKo6boQpmqM9AG4Vsz9h70AAG6cEeOogjhjn96gJ7g@mail.gmail.com>
References: <90aa25bd1caf38e8d019878e970602d2@slave-tothe-box.net>
	<CAMzgZ0JRhKo6boQpmqM9AG4Vsz9h70AAG6cEeOogjhjn96gJ7g@mail.gmail.com>
Message-ID: <9ddfc661a9e4b306428097ca466b502a@slave-tothe-box.net>

Wow this took me forever to get back to.  Thank you...will wait a bit 
and see how the beta progresses.

James

On 2018-11-12 14:34, Jon Siwek wrote:
> On Mon, Nov 12, 2018 at 2:06 PM James Lay <jlay at slave-tothe-box.net> 
> wrote:
>> 
>> Wow what a complete disaster this was.  Errors:
>> 
>> fatal error in /usr/local/bro/share/bro/base/init-bare.bro, line 1:
>> cannot load plugin library
>> /usr/local/bro/lib/bro/plugins/packages/Bro_AF_Packet//lib/Bro-AF_Packet.linux-x86_64.so:
>> /usr/local/bro/lib/bro/plugins/packages/Bro_AF_Packet//lib/Bro-AF_Packet.linux-x86_64.so:
>> undefined symbol:
>> _ZN6plugin6Plugin12HookLoadFileERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_
> 
> May need to recompile the plugin against new Bro version?
> 
>> warning in /usr/local/bro/share/bro/policy/protocols/smb/__load__.bro,
>> line 1: deprecated script loaded from
>> /usr/local/bro/share/bro/site/local.bro:98 "Use '@load
>> base/protocols/smb' instead"
> 
> It's only warning, but you can just update local.bro as indicated.
> 
>> error in
>> /usr/local/bro/share/bro/base/bif/plugins/./Bro_SSL.events.bif.bro, 
>> line
>> 41 and /usr/local/bro/share/bro/site/packages/./ja3/./ja3.bro, line 
>> 118:
>> incompatible types (event(c:connection; version:count;
>> record_version:count; possible_ts:tme; client_random:string;
>> session_id:string; ciphers:vector of count; comp_methods:vector of
>> count;) and event(c:connection; version:count; possible_ts:time;
>> client_random:string; session_id:string; ciphers:vector of count;))
> 
> This will need to be fixed in the ja3 package.  I have a PR open now:
> 
> https://github.com/salesforce/ja3/pull/27
> 
> So either wait for that to get merged and then update the local
> package or use my fork/patch directly if eager to try Bro 2.6-beta.
> 
>> and my redef line below:
>> redef Communication::listen_interface = 127.0.0.1;
>> 
>> gets me:
>> error in /usr/local/bro/share/bro/site/local.bro, line 102: "redef" 
>> used
>> but not previously defined (Communication::listen_interface)
> 
> The equivalent functionality is now:
> 
>     redef Broker::default_listen_address = "127.0.0.1";
> 
> (The underlying communication systems in Bro have been completely
> replaced with a new library called "Broker").
> 
>> reverted back to 2.5.5 now....yeesh.
> 
> There's quite a few potential incompatibilities with upcoming Bro 2.6,
> so worth seeing release notes:
> 
> https://www.bro.org/sphinx-git/install/release-notes.html
> 
> Generally a hope is that future releases minimize breakages for users,
> but there were some big fundamental changes that made it hard to avoid
> for 2.6.
> 
> Let me know if you give it another shot and have further trouble.
> 
> - Jon

From jlay at slave-tothe-box.net  Tue Nov 20 13:56:14 2018
From: jlay at slave-tothe-box.net (James Lay)
Date: Tue, 20 Nov 2018 14:56:14 -0700
Subject: [Bro] Issues since 2.5.5
In-Reply-To: <CAMzgZ0+Y94sBQPY_N4HGGRi+JfHb4OG7UjheDptnrv8Qy5v+YA@mail.gmail.com>
References: <a073ce2a8d4e70dd5ac015ec189ab51d@slave-tothe-box.net>
	<CAMzgZ0+Y94sBQPY_N4HGGRi+JfHb4OG7UjheDptnrv8Qy5v+YA@mail.gmail.com>
Message-ID: <e7eb05b6753785d093cbc5a6dd7d0405@slave-tothe-box.net>

Thanks Jon...I'll do my research and report my findings.

James

On 2018-11-12 13:51, Jon Siwek wrote:
> On Mon, Nov 12, 2018 at 1:16 PM James Lay <jlay at slave-tothe-box.net> 
> wrote:
>> 
>> Well..I think I'll also put my name in the "something is funky with
>> 2.5.5" group.  I have seen far more crashes and OOM's with 2.5.5 than
>> with 2.5.4.
> 
> Various thoughts:
> 
> * This is the first I've heard of trouble directly related to 2.5.5 in
> contrast to 2.5.4. If you have reference to others reporting similar,
> please point me at it as it may help with correlating/diagnosing.
> 
> * For any crashes, forwarding stack traces to reports at bro.org would 
> help.
> 
> * For OOM, a first sanity check is to make sure reporter.log isn't
> showing any scripting errors.  E.g. unitialized record field access is
> known to leak memory, but it's also an underlying scripting mistake
> that needs to fixed.
> 
> * Similarly, remember that memory utilization is effected by scripting
> logic.  If you use any custom or external scripts/packages that are
> not conservative with how they track state over time, that's always a
> possible source of OOM problems that's independent of Bro version.  So
> a question would be whether you are comparing the same configuration
> between 2.5.4 and 2.5.5 or were some scripts/packages different?
> 
> - Jon

From turbidtarantula at gmail.com  Tue Nov 20 14:43:28 2018
From: turbidtarantula at gmail.com (Mike M)
Date: Tue, 20 Nov 2018 17:43:28 -0500
Subject: [Bro] Getting a Broctl Stack Trace
In-Reply-To: <SN6PR11MB32164A06FA71E0AB3A053157A5DE0@SN6PR11MB3216.namprd11.prod.outlook.com>
References: <CAK051agH+hWgr-bZrfqVBQEKHZrU9AkCsb=CwFx4ohUp+YfkuA@mail.gmail.com>
	<AB6568DA-7AD5-4D84-B1AB-AD00877F7114@corelight.com>
	<CAK051ai0=gFXbd96BOZG9RddE6qA-vB4XY+i+em=PotNv2vJkA@mail.gmail.com>
	<SN6PR11MB32164A06FA71E0AB3A053157A5DE0@SN6PR11MB3216.namprd11.prod.outlook.com>
Message-ID: <CAK051aimotFjk=jNG6hsVCqvucbqsh-NKwO+UnVgJM23c_tASg@mail.gmail.com>

Thank you for the suggestion. Adding that setting to broctl.cfg didn't make
a difference, but I did notice that when I run broctl deploy I get "(bro
still initializing)" at the end. Since I don't normally see that it makes
me think something isn't coming up correctly, rather than an actual
segfault.

Are there other options I should try setting in broctl.cfg, or anything
else I can do to diagnose what's not working as broctl expects?

thanks,
Mike

On Fri, Nov 16, 2018 at 9:59 PM Azoff, Justin S <jazoff at illinois.edu> wrote:

> I'm not 100% sure about the root cause, but I know one thing that may
> help.. there's a code path for 'broctl start' that will say something has
> "crashed" when it is "not running".. but "not running" doesn't have to be a
> segfault... just that it didn't fully initialize in the way that broctl was
> expecting it to.
>
>
> Hosom was also looking into this today and said he was seeing:
>
>
> warning in /usr/local/bro/share/bro/base/init-bare.bro, line 1: problem
> initializing NB-DNS: no valid nameservers in resolver config
>
>
> It may be a red herring, but that's easy to rule out:
>
>
> add
>
>
> env_vars=BRO_DNS_FAKE=1
>
>
> to broctl.cfg
>
>
> and deploy and see if things start properly with real DNS disabled.
>
>
> ------------------------------
> *From:* bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Mike M <
> turbidtarantula at gmail.com>
> *Sent:* Tuesday, November 13, 2018 11:22:10 AM
> *To:* seth at corelight.com
> *Cc:* bro at bro.org
> *Subject:* Re: [Bro] Getting a Broctl Stack Trace
>
> I gave this a shot but I'm still not seeing a core file. I tried both the
> setting you recommended and setting an absolute path to /tmp. When I force
> a core dump on another process the core file shows up as expected, but
> broctl isn't producing one.
>
> I'm open to suggestions on this one... not sure how to determine the root
> cause.
>
> thanks,
> Mike
>
> On Mon, Nov 5, 2018 at 5:01 PM Seth Hall <seth at corelight.com> wrote:
>
> Make sure you are setting the core pattern on your system so that the
> core dump will be written into the CWD.
>
> sudo sysctl -w kernel.core_pattern="core.%e-%t-%p"
>
>    .Seth
>
> On 2 Nov 2018, at 12:51, Mike M wrote:
>
> > I'm having an issue with broctl crashing when I try to run it on
> > Alpine
> > Linux. I mentioned it previously [1] but I'm circling back around to
> > try to
> > get it resolved. I've built it with the appropriate patches [2] but
> > broctl
> > is still reporting "crashed" state when I checks the status after
> > starting
> > it. The bro binary itself runs fine.
> >
> > What do I need to do to collect a stack trace from broctl to determine
> > the
> > root cause?
> >
> > Bro is built in debug mode and I set "ulimit -c unlimited" per the
> > instructions on reporting problems. I see a
> >
> /usr/local/bro/spool/tmp/post-terminate-standalone-2018-11-02-02-56-06-13765-crash
> > directory but there's no core dump anywhere obvious. The
> > .crash-diag.out
> > file says "No core file found" and doesn't provide any useful
> > information
> > about the cause of the crash.
> >
> > Thanks,
> > Mike
> >
> > [1]
> >
> http://mailman.icsi.berkeley.edu/pipermail/bro/2018-September/013580.html
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.icsi.berkeley.edu_pipermail_bro_2018-2DSeptember_013580.html&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=XNJ9sWJIJe1PnbIhx0wdY5wWHuR6-JXLc9LyKMMaaeY&s=6-Snv23fdhhesPXq1ctSBZKMk7OC-nGgFfdKgfxFczA&e=>
> > [2]
> >
> http://mailman.icsi.berkeley.edu/pipermail/bro/2018-September/013581.html
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.icsi.berkeley.edu_pipermail_bro_2018-2DSeptember_013581.html&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=XNJ9sWJIJe1PnbIhx0wdY5wWHuR6-JXLc9LyKMMaaeY&s=isM59BgpWPxaWENNIit-XpqFwCdcUtlDM2P3prrsMk8&e=>
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.ICSI.Berkeley.EDU_mailman_listinfo_bro&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=XNJ9sWJIJe1PnbIhx0wdY5wWHuR6-JXLc9LyKMMaaeY&s=deY5uW7Ow0QD4FZmgjaC7hwBcdB5GPcZ52CcOiq2m8Q&e=>
>
> --
> Seth Hall * Corelight, Inc * www.corelight.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.corelight.com&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=XNJ9sWJIJe1PnbIhx0wdY5wWHuR6-JXLc9LyKMMaaeY&s=bXYMmbM6QwdeBb_Dnnc91CEJSnxe-T7MPXIPjs_b2us&e=>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181120/2fece635/attachment.html 

From zhaojiahui555 at gmail.com  Tue Nov 20 17:56:38 2018
From: zhaojiahui555 at gmail.com (jiahui zhao)
Date: Wed, 21 Nov 2018 09:56:38 +0800
Subject: [Bro] Dropped Packets too much
Message-ID: <CAEX=Gtm99-v0c9WOhwf4E0dxDJmpZrKi1mqVBoN1FkQgFhUimg@mail.gmail.com>

I use netstats command on broctl ,find that the dropped packets is about
80% of the total packets
[BroControl]  > netstats
worker - 1 - 1: 1542761937.244889 recvd=117848801 dropped=509235620
link=117848801
worker - 1 - 2: 1542761937.600276 recvd=145398921 dropped=481697589
link=145398921

I had installed pf_ring, the node configuration is as follows:
bro/etc/node.cfg:
[worker-1]
type=worker
host=''
interface=eno2
lb_method=pf_ring
lb_procs=2

Does anybody have the same problem as me?How is it solved
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181121/f3a924cb/attachment-0001.html 

From Robert.Cotter at endace.com  Tue Nov 20 18:17:06 2018
From: Robert.Cotter at endace.com (Robert Cotter)
Date: Wed, 21 Nov 2018 02:17:06 +0000
Subject: [Bro] Dropped Packets too much (jiahui zhao)
Message-ID: <08eb235e469a4d0c8872fc0d56c921c0@endace.com>

I would suggest doing some reading on Bro clustering going a little deeper on your 'lb' configuration.

Not knowing what the data/packet rates you are attempting to process but in my experience asking a single process thread to do more than 300 Mb is going to ensure you get packet drops.

Below is part of my node.cfg for a 500Mb complex network data test lab setup I am currently running hosted in Centos KVM so I can learn/test some of the DNS/SSL scripting features.

[worker-1]
type=worker
host=localhost
#Interface=dag0
lb_procs=4
lb_method=interfaces
lb_interfaces=dag0,dag1,dag2,dag3
pin_cpus=4,5,6,7


Hope this helps you.

Regards

Robert Cotter

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181121/d49c33d5/attachment.html 

From al.kefallonitis at gmail.com  Wed Nov 21 01:28:32 2018
From: al.kefallonitis at gmail.com (Alex Kefallonitis)
Date: Wed, 21 Nov 2018 11:28:32 +0200
Subject: [Bro] Disable Log Stream but not the analyzers
Message-ID: <CAHv=Muojid4SQsv0YvDKPpSRGr4+Q0vSX+m8dzESF_trB_b8ZA@mail.gmail.com>

I have disabled the Log Stream for HTTP :

event bro_init()
  {
    Log::disable_stream(HTTP::LOG);
 }

But i want scripts using HTTP protocol to work e.g
https://raw.githubusercontent.com/sethhall/bro-scripts/master/top-websites.bro

Is there any other way to do it ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181121/923f0989/attachment.html 

From jazoff at illinois.edu  Wed Nov 21 13:03:23 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Wed, 21 Nov 2018 21:03:23 +0000
Subject: [Bro] Disable Log Stream but not the analyzers
In-Reply-To: <CAHv=Muojid4SQsv0YvDKPpSRGr4+Q0vSX+m8dzESF_trB_b8ZA@mail.gmail.com>
References: <CAHv=Muojid4SQsv0YvDKPpSRGr4+Q0vSX+m8dzESF_trB_b8ZA@mail.gmail.com>
Message-ID: <SN6PR11MB32169DCDFAF521730958C661A5DA0@SN6PR11MB3216.namprd11.prod.outlook.com>

Hi,


Using


    Log::remove_default_filter(HTTP::LOG);

instead of disable_stream should do what you want.

________________________________
From: bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Alex Kefallonitis <al.kefallonitis at gmail.com>
Sent: Wednesday, November 21, 2018 4:28:32 AM
To: Bro at bro.org
Subject: [Bro] Disable Log Stream but not the analyzers

I have disabled the Log Stream for HTTP :

event bro_init()
  {
    Log::disable_stream(HTTP::LOG);
 }

But i want scripts using HTTP protocol to work e.g https://raw.githubusercontent.com/sethhall/bro-scripts/master/top-websites.bro<https://urldefense.proofpoint.com/v2/url?u=https-3A__raw.githubusercontent.com_sethhall_bro-2Dscripts_master_top-2Dwebsites.bro&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=Upw7RnEppKcwibJKc4KDIUBeI-V2RkeYtIWL5FNnXH0&s=CC6T1M5j865G11CwDqWWObidSeRZpMkhelQhnJtSHXw&e=>

Is there any other way to do it ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181121/6ff6c75e/attachment.html 

From zhaojiahui555 at gmail.com  Wed Nov 21 22:50:34 2018
From: zhaojiahui555 at gmail.com (jiahui zhao)
Date: Thu, 22 Nov 2018 14:50:34 +0800
Subject: [Bro] Dropped Packets too much (jiahui zhao) (Robert Cotter)
In-Reply-To: <mailman.1.1542830402.17610.bro@bro.org>
References: <mailman.1.1542830402.17610.bro@bro.org>
Message-ID: <CAEX=GtkzgN7USBefVmSR15beCsHOgsWjt2cAWGLsqRYgx2mYKQ@mail.gmail.com>

@Robert Cotter  Thank you for your reply ?
 I try the solution you given , but i didn't work.

Maybe it's the pf_ring that causes the problem.
When i used tcpdump, i finded  the same problem of Dropped Packets.
Runtime environment:
 NIC is Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe
 pf_ring version is 7.1.0
 bro 2.5.5
 linux:centos


<bro-request at bro.org> ?

> Send Bro mailing list submissions to
>         bro at bro.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> or, via email, send a message with subject or body 'help' to
>         bro-request at bro.org
>
> You can reach the person managing the list at
>         bro-owner at bro.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Bro digest..."
>
>
> Today's Topics:
>
>    1. Re: Dropped Packets too much (jiahui zhao) (Robert Cotter)
>    2. Disable Log Stream but not the analyzers (Alex Kefallonitis)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 21 Nov 2018 02:17:06 +0000
> From: Robert Cotter <Robert.Cotter at endace.com>
> Subject: Re: [Bro] Dropped Packets too much (jiahui zhao)
> To: "bro at bro.org" <bro at bro.org>
> Message-ID: <08eb235e469a4d0c8872fc0d56c921c0 at endace.com>
> Content-Type: text/plain; charset="us-ascii"
>
> I would suggest doing some reading on Bro clustering going a little deeper
> on your 'lb' configuration.
>
> Not knowing what the data/packet rates you are attempting to process but
> in my experience asking a single process thread to do more than 300 Mb is
> going to ensure you get packet drops.
>
> Below is part of my node.cfg for a 500Mb complex network data test lab
> setup I am currently running hosted in Centos KVM so I can learn/test some
> of the DNS/SSL scripting features.
>
> [worker-1]
> type=worker
> host=localhost
> #Interface=dag0
> lb_procs=4
> lb_method=interfaces
> lb_interfaces=dag0,dag1,dag2,dag3
> pin_cpus=4,5,6,7
>
>
> Hope this helps you.
>
> Regards
>
> Robert Cotter
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181121/d49c33d5/attachment-0001.html
>
> ------------------------------
>
> Message: 2
> Date: Wed, 21 Nov 2018 11:28:32 +0200
> From: Alex Kefallonitis <al.kefallonitis at gmail.com>
> Subject: [Bro] Disable Log Stream but not the analyzers
> To: Bro at bro.org
> Message-ID:
>         <CAHv=
> Muojid4SQsv0YvDKPpSRGr4+Q0vSX+m8dzESF_trB_b8ZA at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> I have disabled the Log Stream for HTTP :
>
> event bro_init()
>   {
>     Log::disable_stream(HTTP::LOG);
>  }
>
> But i want scripts using HTTP protocol to work e.g
>
> https://raw.githubusercontent.com/sethhall/bro-scripts/master/top-websites.bro
>
> Is there any other way to do it ?
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181121/923f0989/attachment-0001.html
>
> ------------------------------
>
> _______________________________________________
> Bro mailing list
> Bro at bro.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> End of Bro Digest, Vol 151, Issue 22
> ************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181122/19f648a9/attachment.html 

From al.kefallonitis at gmail.com  Thu Nov 22 00:39:37 2018
From: al.kefallonitis at gmail.com (Alex Kefallonitis)
Date: Thu, 22 Nov 2018 10:39:37 +0200
Subject: [Bro] Disable Log Stream but not the analyzers
In-Reply-To: <SN6PR11MB32169DCDFAF521730958C661A5DA0@SN6PR11MB3216.namprd11.prod.outlook.com>
References: <CAHv=Muojid4SQsv0YvDKPpSRGr4+Q0vSX+m8dzESF_trB_b8ZA@mail.gmail.com>
	<SN6PR11MB32169DCDFAF521730958C661A5DA0@SN6PR11MB3216.namprd11.prod.outlook.com>
Message-ID: <CAHv=MuqDh4LB4zRNY+wo5Jm4AydpEfcYNKDmgeByd2Mt2QHS2A@mail.gmail.com>

Hi i did change it but no logs regarding http are produced like
https://raw.githubusercontent.com/sethhall/bro-scripts/master/top-websites.bro
or
https://github.com/BrashEndeavours/bro-scripts/blob/master/http_entropy.bro
.


[image: image.png]

[image: image.png]

???? ???, 21 ??? 2018 ???? 11:03 ?.?., ?/? Azoff, Justin S <
jazoff at illinois.edu> ??????:

> Hi,
>
>
> Using
>
>
>     Log::remove_default_filter(HTTP::LOG);
>
> instead of disable_stream should do what you want.
>
> ------------------------------
> *From:* bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Alex
> Kefallonitis <al.kefallonitis at gmail.com>
> *Sent:* Wednesday, November 21, 2018 4:28:32 AM
> *To:* Bro at bro.org
> *Subject:* [Bro] Disable Log Stream but not the analyzers
>
> I have disabled the Log Stream for HTTP :
>
> event bro_init()
>   {
>     Log::disable_stream(HTTP::LOG);
>  }
>
> But i want scripts using HTTP protocol to work e.g
> https://raw.githubusercontent.com/sethhall/bro-scripts/master/top-websites.bro
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__raw.githubusercontent.com_sethhall_bro-2Dscripts_master_top-2Dwebsites.bro&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=Upw7RnEppKcwibJKc4KDIUBeI-V2RkeYtIWL5FNnXH0&s=CC6T1M5j865G11CwDqWWObidSeRZpMkhelQhnJtSHXw&e=>
>
> Is there any other way to do it ?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181122/1cabda5a/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 5544 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181122/1cabda5a/attachment-0002.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 6415 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181122/1cabda5a/attachment-0003.bin 

From michalpurzynski1 at gmail.com  Thu Nov 22 00:58:06 2018
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Thu, 22 Nov 2018 00:58:06 -0800
Subject: [Bro] Disable Log Stream but not the analyzers
In-Reply-To: <CAHv=MuqDh4LB4zRNY+wo5Jm4AydpEfcYNKDmgeByd2Mt2QHS2A@mail.gmail.com>
References: <CAHv=Muojid4SQsv0YvDKPpSRGr4+Q0vSX+m8dzESF_trB_b8ZA@mail.gmail.com>
	<SN6PR11MB32169DCDFAF521730958C661A5DA0@SN6PR11MB3216.namprd11.prod.outlook.com>
	<CAHv=MuqDh4LB4zRNY+wo5Jm4AydpEfcYNKDmgeByd2Mt2QHS2A@mail.gmail.com>
Message-ID: <73FDE352-F198-4B90-877C-AA03A7A6CF1F@gmail.com>

Indeed, scripts you?re showing depend on the log streams you just disabled.

> On Nov 22, 2018, at 12:39 AM, Alex Kefallonitis <al.kefallonitis at gmail.com> wrote:
> 
> 
> Hi i did change it but no logs regarding http are produced like https://raw.githubusercontent.com/sethhall/bro-scripts/master/top-websites.bro or https://github.com/BrashEndeavours/bro-scripts/blob/master/http_entropy.bro .
> 
> 
> <image.png>
> 
> <image.png>
> 
> ???? ???, 21 ??? 2018 ???? 11:03 ?.?., ?/? Azoff, Justin S <jazoff at illinois.edu> ??????:
>> Hi,
>> 
>> Using 
>> 
>>     Log::remove_default_filter(HTTP::LOG);
>> 
>> instead of disable_stream should do what you want.
>> From: bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Alex Kefallonitis <al.kefallonitis at gmail.com>
>> Sent: Wednesday, November 21, 2018 4:28:32 AM
>> To: Bro at bro.org
>> Subject: [Bro] Disable Log Stream but not the analyzers
>>  
>> I have disabled the Log Stream for HTTP :
>> 
>> event bro_init()
>>   {
>>     Log::disable_stream(HTTP::LOG);
>>  }
>> 
>> But i want scripts using HTTP protocol to work e.g https://raw.githubusercontent.com/sethhall/bro-scripts/master/top-websites.bro
>> 
>> Is there any other way to do it ?
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181122/39fbaf6b/attachment.html 

From al.kefallonitis at gmail.com  Thu Nov 22 02:05:48 2018
From: al.kefallonitis at gmail.com (Alex Kefallonitis)
Date: Thu, 22 Nov 2018 12:05:48 +0200
Subject: [Bro] Disable Log Stream but not the analyzers
In-Reply-To: <73FDE352-F198-4B90-877C-AA03A7A6CF1F@gmail.com>
References: <CAHv=Muojid4SQsv0YvDKPpSRGr4+Q0vSX+m8dzESF_trB_b8ZA@mail.gmail.com>
	<SN6PR11MB32169DCDFAF521730958C661A5DA0@SN6PR11MB3216.namprd11.prod.outlook.com>
	<CAHv=MuqDh4LB4zRNY+wo5Jm4AydpEfcYNKDmgeByd2Mt2QHS2A@mail.gmail.com>
	<73FDE352-F198-4B90-877C-AA03A7A6CF1F@gmail.com>
Message-ID: <CAHv=Murebw2Gk_ayRnStB+dQ6GVzVRqu3gVnJAE9fi2jH_cfQA@mail.gmail.com>

So there is no way to disable specific logs but still use the analyzers in
the script ? The scripts are reading the actual logs and needed from them
to work ?

???? ???, 22 ??? 2018 ???? 10:58 ?.?., ?/? Micha? Purzy?ski <
michalpurzynski1 at gmail.com> ??????:

> Indeed, scripts you?re showing depend on the log streams you just disabled.
>
> On Nov 22, 2018, at 12:39 AM, Alex Kefallonitis <al.kefallonitis at gmail.com>
> wrote:
>
>
> Hi i did change it but no logs regarding http are produced like
> https://raw.githubusercontent.com/sethhall/bro-scripts/master/top-websites.bro
> or
> https://github.com/BrashEndeavours/bro-scripts/blob/master/http_entropy.bro
> .
>
>
> <image.png>
>
> <image.png>
>
> ???? ???, 21 ??? 2018 ???? 11:03 ?.?., ?/? Azoff, Justin S <
> jazoff at illinois.edu> ??????:
>
>> Hi,
>>
>>
>> Using
>>
>>
>>     Log::remove_default_filter(HTTP::LOG);
>>
>> instead of disable_stream should do what you want.
>>
>> ------------------------------
>> *From:* bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Alex
>> Kefallonitis <al.kefallonitis at gmail.com>
>> *Sent:* Wednesday, November 21, 2018 4:28:32 AM
>> *To:* Bro at bro.org
>> *Subject:* [Bro] Disable Log Stream but not the analyzers
>>
>> I have disabled the Log Stream for HTTP :
>>
>> event bro_init()
>>   {
>>     Log::disable_stream(HTTP::LOG);
>>  }
>>
>> But i want scripts using HTTP protocol to work e.g
>> https://raw.githubusercontent.com/sethhall/bro-scripts/master/top-websites.bro
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__raw.githubusercontent.com_sethhall_bro-2Dscripts_master_top-2Dwebsites.bro&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=Upw7RnEppKcwibJKc4KDIUBeI-V2RkeYtIWL5FNnXH0&s=CC6T1M5j865G11CwDqWWObidSeRZpMkhelQhnJtSHXw&e=>
>>
>> Is there any other way to do it ?
>>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181122/9c61edf3/attachment.html 

From jazoff at illinois.edu  Fri Nov 23 08:47:03 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Fri, 23 Nov 2018 16:47:03 +0000
Subject: [Bro] Disable Log Stream but not the analyzers
In-Reply-To: <CAHv=Murebw2Gk_ayRnStB+dQ6GVzVRqu3gVnJAE9fi2jH_cfQA@mail.gmail.com>
References: <CAHv=Muojid4SQsv0YvDKPpSRGr4+Q0vSX+m8dzESF_trB_b8ZA@mail.gmail.com>
	<SN6PR11MB32169DCDFAF521730958C661A5DA0@SN6PR11MB3216.namprd11.prod.outlook.com>
	<CAHv=MuqDh4LB4zRNY+wo5Jm4AydpEfcYNKDmgeByd2Mt2QHS2A@mail.gmail.com>
	<73FDE352-F198-4B90-877C-AA03A7A6CF1F@gmail.com>,
	<CAHv=Murebw2Gk_ayRnStB+dQ6GVzVRqu3gVnJAE9fi2jH_cfQA@mail.gmail.com>
Message-ID: <SN6PR11MB32160064BD851E5F83F973F6A5D40@SN6PR11MB3216.namprd11.prod.outlook.com>

Read my response again...

Using Log::remove_default_filter does what you want.  You used remove_stream which is something different.

________________________________
From: Alex Kefallonitis <al.kefallonitis at gmail.com>
Sent: Thursday, November 22, 2018 5:05:48 AM
To: michalpurzynski1 at gmail.com
Cc: Azoff, Justin S; Bro at bro.org
Subject: Re: [Bro] Disable Log Stream but not the analyzers

So there is no way to disable specific logs but still use the analyzers in the script ? The scripts are reading the actual logs and needed from them to work ?

???? ???, 22 ??? 2018 ???? 10:58 ?.?., ?/? Micha? Purzy?ski <michalpurzynski1 at gmail.com<mailto:michalpurzynski1 at gmail.com>> ??????:
Indeed, scripts you?re showing depend on the log streams you just disabled.

On Nov 22, 2018, at 12:39 AM, Alex Kefallonitis <al.kefallonitis at gmail.com<mailto:al.kefallonitis at gmail.com>> wrote:


Hi i did change it but no logs regarding http are produced like https://raw.githubusercontent.com/sethhall/bro-scripts/master/top-websites.bro or https://github.com/BrashEndeavours/bro-scripts/blob/master/http_entropy.bro .


<image.png>

<image.png>

???? ???, 21 ??? 2018 ???? 11:03 ?.?., ?/? Azoff, Justin S <jazoff at illinois.edu<mailto:jazoff at illinois.edu>> ??????:

Hi,


Using


    Log::remove_default_filter(HTTP::LOG);

instead of disable_stream should do what you want.

________________________________
From: bro-bounces at bro.org<mailto:bro-bounces at bro.org> <bro-bounces at bro.org<mailto:bro-bounces at bro.org>> on behalf of Alex Kefallonitis <al.kefallonitis at gmail.com<mailto:al.kefallonitis at gmail.com>>
Sent: Wednesday, November 21, 2018 4:28:32 AM
To: Bro at bro.org<mailto:Bro at bro.org>
Subject: [Bro] Disable Log Stream but not the analyzers

I have disabled the Log Stream for HTTP :

event bro_init()
  {
    Log::disable_stream(HTTP::LOG);
 }

But i want scripts using HTTP protocol to work e.g https://raw.githubusercontent.com/sethhall/bro-scripts/master/top-websites.bro<https://urldefense.proofpoint.com/v2/url?u=https-3A__raw.githubusercontent.com_sethhall_bro-2Dscripts_master_top-2Dwebsites.bro&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=Upw7RnEppKcwibJKc4KDIUBeI-V2RkeYtIWL5FNnXH0&s=CC6T1M5j865G11CwDqWWObidSeRZpMkhelQhnJtSHXw&e=>

Is there any other way to do it ?
_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181123/d597f015/attachment.html 

From al.kefallonitis at gmail.com  Sat Nov 24 00:19:52 2018
From: al.kefallonitis at gmail.com (Alex Kefallonitis)
Date: Sat, 24 Nov 2018 10:19:52 +0200
Subject: [Bro] Disable Log Stream but not the analyzers
In-Reply-To: <SN6PR11MB32160064BD851E5F83F973F6A5D40@SN6PR11MB3216.namprd11.prod.outlook.com>
References: <CAHv=Muojid4SQsv0YvDKPpSRGr4+Q0vSX+m8dzESF_trB_b8ZA@mail.gmail.com>
	<SN6PR11MB32169DCDFAF521730958C661A5DA0@SN6PR11MB3216.namprd11.prod.outlook.com>
	<CAHv=MuqDh4LB4zRNY+wo5Jm4AydpEfcYNKDmgeByd2Mt2QHS2A@mail.gmail.com>
	<73FDE352-F198-4B90-877C-AA03A7A6CF1F@gmail.com>
	<CAHv=Murebw2Gk_ayRnStB+dQ6GVzVRqu3gVnJAE9fi2jH_cfQA@mail.gmail.com>
	<SN6PR11MB32160064BD851E5F83F973F6A5D40@SN6PR11MB3216.namprd11.prod.outlook.com>
Message-ID: <CAHv=MuqZsB8OzK5TSebsom_OW08d+GreXi+b8BvAHt+LHnCF4Q@mail.gmail.com>

Yes you are correct it works ! Thanks a lot



On Fri, 23 Nov 2018, 18:47 Azoff, Justin S <jazoff at illinois.edu wrote:

> Read my response again...
>
> Using Log::remove_default_filter does what you want.  You used
> remove_stream which is something different.
> ------------------------------
> *From:* Alex Kefallonitis <al.kefallonitis at gmail.com>
> *Sent:* Thursday, November 22, 2018 5:05:48 AM
> *To:* michalpurzynski1 at gmail.com
> *Cc:* Azoff, Justin S; Bro at bro.org
> *Subject:* Re: [Bro] Disable Log Stream but not the analyzers
>
> So there is no way to disable specific logs but still use the analyzers in
> the script ? The scripts are reading the actual logs and needed from them
> to work ?
>
> ???? ???, 22 ??? 2018 ???? 10:58 ?.?., ?/? Micha? Purzy?ski <
> michalpurzynski1 at gmail.com> ??????:
>
> Indeed, scripts you?re showing depend on the log streams you just disabled.
>
> On Nov 22, 2018, at 12:39 AM, Alex Kefallonitis <al.kefallonitis at gmail.com>
> wrote:
>
>
> Hi i did change it but no logs regarding http are produced like
> https://raw.githubusercontent.com/sethhall/bro-scripts/master/top-websites.bro
> or
> https://github.com/BrashEndeavours/bro-scripts/blob/master/http_entropy.bro
> .
>
>
> <image.png>
>
> <image.png>
>
> ???? ???, 21 ??? 2018 ???? 11:03 ?.?., ?/? Azoff, Justin S <
> jazoff at illinois.edu> ??????:
>
> Hi,
>
>
> Using
>
>
>     Log::remove_default_filter(HTTP::LOG);
>
> instead of disable_stream should do what you want.
>
> ------------------------------
> *From:* bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Alex
> Kefallonitis <al.kefallonitis at gmail.com>
> *Sent:* Wednesday, November 21, 2018 4:28:32 AM
> *To:* Bro at bro.org
> *Subject:* [Bro] Disable Log Stream but not the analyzers
>
> I have disabled the Log Stream for HTTP :
>
> event bro_init()
>   {
>     Log::disable_stream(HTTP::LOG);
>  }
>
> But i want scripts using HTTP protocol to work e.g
> https://raw.githubusercontent.com/sethhall/bro-scripts/master/top-websites.bro
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__raw.githubusercontent.com_sethhall_bro-2Dscripts_master_top-2Dwebsites.bro&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=JB1gr8Q2U3j_GvRbWa2WDpXSSrvReahkLBFDmdXlCh0&m=Upw7RnEppKcwibJKc4KDIUBeI-V2RkeYtIWL5FNnXH0&s=CC6T1M5j865G11CwDqWWObidSeRZpMkhelQhnJtSHXw&e=>
>
> Is there any other way to do it ?
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181124/40d8b2c1/attachment.html 

From yisun at heliosdata.com  Sun Nov 25 15:22:49 2018
From: yisun at heliosdata.com (Yi Sun)
Date: Sun, 25 Nov 2018 23:22:49 +0000
Subject: [Bro] Why Broccoli is marked as (DEPRECATED)
Message-ID: <9E99D9A6-D10E-4403-A237-8178F0550082@heliosdata.com>

Hi,
I?m seeing Broccoli is marked as (DEPRECATED) form github page https://github.com/bro/broccoli. What does it mean? Will broccoli being deprecated or removed from bro soon?
Yi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181125/dc03f3b3/attachment.html 

From jsiwek at corelight.com  Mon Nov 26 07:38:03 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Mon, 26 Nov 2018 09:38:03 -0600
Subject: [Bro] Why Broccoli is marked as (DEPRECATED)
In-Reply-To: <9E99D9A6-D10E-4403-A237-8178F0550082@heliosdata.com>
References: <9E99D9A6-D10E-4403-A237-8178F0550082@heliosdata.com>
Message-ID: <CAMzgZ0+mE_c3uNte3udo=PH73sLWbUGqA_thTirD1PMA8yOxZw@mail.gmail.com>

On Sun, Nov 25, 2018 at 5:31 PM Yi Sun <yisun at heliosdata.com> wrote:

> I?m seeing Broccoli is marked as (DEPRECATED) form github page https://github.com/bro/broccoli. What does it mean? Will broccoli being deprecated or removed from bro soon?

Broccoli, used by default in Bro 2.5 and previous versions, is
deprecated because it's being replaced with a new component called
Broker:

https://github.com/bro/broker

Currently, Bro's master branch in git and the upcoming 2.6 release use
Broker for remote communication by default and re-enabling Broccoli
communication requires some additional effort from the user.  The 2.6
release is likely on the order of weeks away.  Broccoli will be
completely removed at some point, likely for 2.7, and historically
these minor-version releases have been paced on the order of 1-2 years
apart (not strictly scheduled).

- Jon


From nothinrandom at gmail.com  Mon Nov 26 14:34:20 2018
From: nothinrandom at gmail.com (TQ)
Date: Mon, 26 Nov 2018 14:34:20 -0800
Subject: [Bro] Allocating Dynamic Bytestring Length
Message-ID: <CAM2MKSoxFXxC_-W4er4R7LkvRW70BT_QdVu7kFutrFL+aG=e1w@mail.gmail.com>

Hello There,

I have some data that is delimited by a comma, but the data length could
vary in between. For example:

1) A,B,C,D,E
2) AA,BBB,C,DDDDD,EE
3) AAA,BB,CCCCCCCC,DD,EEE

I was thinking of using bytestring inside the record and have it read until
each delimiter... something like...

type My_Data = record {

field1 : bytestring & length=readuntil 0x2c;
field2 : bytestring & length=readuntil 0x2c;
field3 : bytestring & length=readuntil 0x2c;
field4 : bytestring & length=readuntil 0x2c;
field1 : bytestring &restofdata;

};

Does this feature exist in Bro?  I vaguely remember seeing a readuntil
feature for Bro, but can't pull up the exact info.

Thanks,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181126/60b952ad/attachment.html 

From al.kefallonitis at gmail.com  Wed Nov 28 01:54:22 2018
From: al.kefallonitis at gmail.com (Alex Kefallonitis)
Date: Wed, 28 Nov 2018 11:54:22 +0200
Subject: [Bro] General Whitelisting IP's or Domains
Message-ID: <CAHv=Muqh4i=xYqL9QvO-4qFzHyQxqdcG2A7yMTsz+i135ZLaFQ@mail.gmail.com>

Hi,

 Is there a generic way to whitelist certain IP's/Subets or Domains in
local.bro for the whole Bro configuration as not to produce logs and or
notices.

 For e.g whitelist 8.8.8.8 or google.com ?

Thanks in advanced,
Alex Kefallonitis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181128/ce0419a9/attachment.html 

From dnj0496 at gmail.com  Wed Nov 28 17:31:46 2018
From: dnj0496 at gmail.com (Dk Jack)
Date: Wed, 28 Nov 2018 17:31:46 -0800
Subject: [Bro] sum stats q.
Message-ID: <CAGATen6=jzbiWEfAy2KEdtoCEUcxtZqV9Xwa-4+mVBXnjffq1w@mail.gmail.com>

Hi,
I am trying to use Bro sumstats framework. Based on the examples, I came up
with the script shown at the end of the email. In the script, I am counting
the number of http requests for each method+uri combination.

As dictated by the framework, I am calling observe for each request. At the
end, I expected the total sumstats equal to the number of requests in my
pcap. However, this doesn't seem to be the case. I am trying understand if
I made a mistake in how I am using the framework of if something else is
going on.

For example, I ran the script on try.bro.org website using the http.pcap
available there. Per my analysis, there should be 197 requests in the pcap.
However, when I dump each of my stat into a log file, I expected the hits
column from the log to add up to 197. However, that's not the case. Running
the script against my own pcap is giving different numbers from what I
would expect.

Any help understanding the issue is appreciated... Thanks

Dk.

PS: you can copy paste this script in to try.bro.org website and run it
against the http.pcap.

@load base/utils/site
@load base/frameworks/sumstats

module HttpStats;

export {
  redef enum Log::ID += { LOG };

  type Info: record {
    ts:         time   &log;
    method:     string &log;
    uri:        string &log;
    hits:       count  &log;
  };

  global update_http_stats: function(method: string, uri: string);
}

global scount: count = 0;

event bro_init() &priority=5
{
  print "Creating HttpStats log stream and HTTP sumstats";
  flush_all();

  # Create the stream.
  Log::create_stream(HttpStats::LOG, [$columns=Info, $path="http-stats"]);

  local r1 = SumStats::Reducer($stream="http-stats",
$apply=set(SumStats::SUM));

  SumStats::create([$name="http-stats",
                    $epoch=5sec,
                    $reducers=set(r1),
                    $epoch_result(ts: time, key: SumStats::Key, result:
SumStats::Result) =
                    {
                      local r = result["http-stats"];
                      local host_uri_vec = split_string(key$str, /,/);
                      local method = host_uri_vec[0];
                      local uri = host_uri_vec[1];
                      #local hits = double_to_count(floor(r$sum));
                      local hits = double_to_count(floor(r$num));

                      # prep the record
                      local log_rec: Info = [$ts=ts, $method=method,
$uri=uri, $hits=hits];
                      Log::write(HttpStats::LOG, log_rec);
                    }
                    ]);
}

event bro_done()
{
    Reporter::info(fmt("scount=%d", scount));
}

function update_http_stats(method: string, uri: string)
{
  local key = cat_sep(",", "-", method, uri);

  scount += 1;

  # count URI hits.
  SumStats::observe("http-stats", SumStats::Key($str=key),
SumStats::Observation($num=1));
}

event http_request(c: connection, method: string, original_URI: string,
unescaped_URI: string, version: string)
{
    update_http_stats(method, unescaped_URI);
}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181128/09929bd6/attachment.html 

From luks at crimsoncore.be  Thu Nov 29 01:00:29 2018
From: luks at crimsoncore.be (Luk Schoonaert)
Date: Thu, 29 Nov 2018 09:00:29 +0000
Subject: [Bro] SMB files log
Message-ID: <1A838220-919A-4DE2-8F5A-8ACDB943DF97@crimsoncore.be>

Hey guys,

I?m new to this mailing list - and I have a question about enabling the SMB analyser, I?m sure I?m missing something simple.

I enabled /opt/bro/share/bro/site/local.bro -> @load policy/protocols/smb

Running BRO 2.5.1 - I never get the smb_file.log, I do get these:

smb_cmd.log
smb_mapping.log

When I copy a file over SMB I;d expect ths smb_files.log to be populated - I?m sure I?m missing something very simple, anyone have an idea?

Many Thanks,
Luk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181129/83e90038/attachment.html 

From jazoff at illinois.edu  Thu Nov 29 09:24:04 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 29 Nov 2018 17:24:04 +0000
Subject: [Bro] sum stats q.
In-Reply-To: <CAGATen6=jzbiWEfAy2KEdtoCEUcxtZqV9Xwa-4+mVBXnjffq1w@mail.gmail.com>
References: <CAGATen6=jzbiWEfAy2KEdtoCEUcxtZqV9Xwa-4+mVBXnjffq1w@mail.gmail.com>
Message-ID: <SN6PR11MB3216DB8C8E6A41D6FB5A238DA5D20@SN6PR11MB3216.namprd11.prod.outlook.com>

Hi!

This is all my fault ??.? Currently trybro limits log output to 200 lines for each file.  It shows the first 100 and the last 100.? I had always intended on making that more obvious and allowing that '200' parameter to be changed, but forgot all about it.  It was mostly done as a performance optimization - the log output can be quite large and the result would either take too long to transfer to the client or the browser would freeze trying to render a table with 20k rows.  The good news is that it is already a parameter on the backend, it just needs to be exposed to the api.

If you increase the interval on your script to 500secs that outputs all the records since the total number of rows is just under 200.

If you run it with a local bro binary you should get the output you are expecting as well.

That said.. the script you posted would likely have issues if ran on a cluster.  The short time interval combined with the potential for a large number of unique 'keys' in sumstats would cause a large amount of load on the manager.  If you're not running it on a cluster on live traffic it should work fine though.  If you do want to run that exact analysis on a cluster I can write you a version that uses events directly and would perform a bit better under load.

-- 
- Justin


From jazoff at illinois.edu  Thu Nov 29 09:29:46 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 29 Nov 2018 17:29:46 +0000
Subject: [Bro] General Whitelisting IP's or Domains
In-Reply-To: <CAHv=Muqh4i=xYqL9QvO-4qFzHyQxqdcG2A7yMTsz+i135ZLaFQ@mail.gmail.com>
References: <CAHv=Muqh4i=xYqL9QvO-4qFzHyQxqdcG2A7yMTsz+i135ZLaFQ@mail.gmail.com>
Message-ID: <SN6PR11MB3216AACEA7381A435F3C4AADA5D20@SN6PR11MB3216.namprd11.prod.outlook.com>

> Is there a generic way to whitelist certain IP's/Subets or Domains in local.bro for the whole Bro configuration as not to produce logs and or notices.
> 
> For e.g whitelist 8.8.8.8 or  google.com ?

It depends.. if you wanted to ignore ALL traffic to 8.8.8.8 you could add this:

    redef restrict_filters += [ ["not-google-dns"] = "not (host 8.8.8.8)" ];

Ignoring a 'google.com' is possible as well, but a little more involved since it
could appear in dns, ssl, or http logs.  Is there a particular kind of log that
you are seeing domains in that you want to ignore, or all of the above?

-- 
- Justin

From al.kefallonitis at gmail.com  Thu Nov 29 09:34:45 2018
From: al.kefallonitis at gmail.com (Alex Kefallonitis)
Date: Thu, 29 Nov 2018 19:34:45 +0200
Subject: [Bro] General Whitelisting IP's or Domains
In-Reply-To: <SN6PR11MB3216AACEA7381A435F3C4AADA5D20@SN6PR11MB3216.namprd11.prod.outlook.com>
References: <CAHv=Muqh4i=xYqL9QvO-4qFzHyQxqdcG2A7yMTsz+i135ZLaFQ@mail.gmail.com>
	<SN6PR11MB3216AACEA7381A435F3C4AADA5D20@SN6PR11MB3216.namprd11.prod.outlook.com>
Message-ID: <CAHv=MuoK6mjkCKb5NZ3-n5koKNRE_vOZDhyXn3upW+_vic0nGQ@mail.gmail.com>

Hi and thanks for the response

 I want to be able to apply the whitelist in all of the above as generic
solution when something is spamming or hits as false positive. So is there
any generic solution ?

Thanks in advanced,
Alex Kefallonitis

???? ???, 29 ??? 2018 ???? 7:30 ?.?., ?/? Azoff, Justin S <
jazoff at illinois.edu> ??????:

> > Is there a generic way to whitelist certain IP's/Subets or Domains in
> local.bro for the whole Bro configuration as not to produce logs and or
> notices.
> >
> > For e.g whitelist 8.8.8.8 or  google.com ?
>
> It depends.. if you wanted to ignore ALL traffic to 8.8.8.8 you could add
> this:
>
>     redef restrict_filters += [ ["not-google-dns"] = "not (host 8.8.8.8)"
> ];
>
> Ignoring a 'google.com' is possible as well, but a little more involved
> since it
> could appear in dns, ssl, or http logs.  Is there a particular kind of log
> that
> you are seeing domains in that you want to ignore, or all of the above?
>
> --
> - Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181129/36d94e9c/attachment.html 

From jsiwek at corelight.com  Thu Nov 29 10:24:50 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 29 Nov 2018 12:24:50 -0600
Subject: [Bro] Bro 2.6 release
Message-ID: <CAMzgZ0JGUxgkXF96Y4dJYwsEQ3cMHQ5BL4+ktZppCvB5M-nDZw@mail.gmail.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Bro v2.6 is now released and available for download:

    https://www.bro.org/download/index.html
    https://www.bro.org/downloads/bro-2.6.tar.gz

The most significant change to be aware of is that Bro has
switched to using the new Broker communication library.  As a
result, user-written scripts related to cluster operation or
remote communication that worked in previous versions may
require porting to new APIs.

Please read the release notes carefully for helpful porting tips
or other changes relevant to the upgrade process:

    https://www.bro.org/sphinx/install/release-notes.html

Also note that the Bro project is in the process of being
renamed to Zeek, however, the software distribution for this
release is still named Bro.  There's not yet been any related
naming changes that alter usage for any provided tools or APIs.
-----BEGIN PGP SIGNATURE-----

iQIcBAEBAgAGBQJcAC5uAAoJEKfUHOR63zbzp4sQAJgD8Nqr3V89LbIQdytLkwkT
3z9JJw0BRN1/0cx2NW0t8JudbpiwMMzKwbvoYLrc5sM7I3OSMKhJ6uoWZ5Sgennd
W9kZDJZnmEMT1cfgtkuJHTiSthooDaFsTUNQOmWaebVvQCZdrNyaNKRu00IAbMFG
Ut1OyBur980wAWLDwEq+XjygS0gRxJOvX/MZyBuambGbtTGt2/qvtHrLONQoMXTz
jJSHLf6DICjvpVbfgBijezQj7Zi1adgrWa1pl7FOBGJIeDq5bMuf03Wo8pm/16sA
0AkHV2kLa2QABxRl+aQLKChpvO/28SbHy9glg+a1gBr6QAeKgEudvI/k0Sfoc01u
eTZfm2lYxCxLm+nD6hXEyzGL2ZTEWJoB9F1AedE4AhYzClgo/7MjTO42LwG4igWL
5U0th7U5EsIaCtnRv0TtxXgr8c5zGLvkC8rwLqJC3+zL08SlK8NxLu8ivZ5k1IF4
OZfD/8sZF9EQs64+tDdgHy2iW5vsKOcWz6HCaxOJwt6veYZMglVXJDGtbcRymp2l
2eriwQpes3cZArYxGdTvGw3DZyyqjqWxjIib+832vblXHtoik9FXyhFJqMfX3Iyk
4w3GLkkI9QNjLTlVAWjdFxJPg/63Xb28ymcsWkK46bFdrO7Bs8NVn2lO6Xlq3oiW
EEM4plKvcp8mh4KSWJiF
=F3YY
-----END PGP SIGNATURE-----

From johanna at icir.org  Thu Nov 29 16:52:36 2018
From: johanna at icir.org (Johanna Amann)
Date: Thu, 29 Nov 2018 16:52:36 -0800
Subject: [Bro] SMB files log
In-Reply-To: <1A838220-919A-4DE2-8F5A-8ACDB943DF97@crimsoncore.be>
References: <1A838220-919A-4DE2-8F5A-8ACDB943DF97@crimsoncore.be>
Message-ID: <20181130005236.qxinther2ilc37mt@Tranquility.local>

Hi Luk,

On Thu, Nov 29, 2018 at 09:00:29AM +0000, Luk Schoonaert wrote:
> I enabled /opt/bro/share/bro/site/local.bro -> @load policy/protocols/smb
> 
> Running BRO 2.5.1 - I never get the smb_file.log, I do get these:

First the thing I have to say - please update to 2.5.5. There are only
minor changes to 2.5.1 and a lot of fixed security issues.

Or - consider upgrading to 2.6 (which admittedly has a bunch of changes). 

> smb_cmd.log
> smb_mapping.log
> 
> When I copy a file over SMB I;d expect ths smb_files.log to be populated
> - I?m sure I?m missing something very simple, anyone have an idea?

I think you are right and that it should typically be logged.

There are 2 ways that I would start debugging this. First - if possible,
make a pcap of an operation that you would expect to create the
smb_files.log.

Run that through bro, and see if it is there now; if not, take a look at
smb_cmd.log and look if you can find activity that corresponds to the file
copying in there.

Johanna

From yisun at heliosdata.com  Thu Nov 29 18:32:43 2018
From: yisun at heliosdata.com (Yi Sun)
Date: Fri, 30 Nov 2018 02:32:43 +0000
Subject: [Bro] SMB files log
In-Reply-To: <20181130005236.qxinther2ilc37mt@Tranquility.local>
References: <1A838220-919A-4DE2-8F5A-8ACDB943DF97@crimsoncore.be>
	<20181130005236.qxinther2ilc37mt@Tranquility.local>
Message-ID: <AE7221E6-FD27-420F-9375-670BFA9C6CBE@heliosdata.com>

Here is what happened in my env, I can see the smb_file.log if I use smbclient from Linux. But when I do mount, I don't see the log. I'm not expert on this, and it is only what I see.
Yi

?On 11/29/18, 5:05 PM, "bro-bounces at bro.org on behalf of Johanna Amann" <bro-bounces at bro.org on behalf of johanna at icir.org> wrote:

    Hi Luk,
    
    On Thu, Nov 29, 2018 at 09:00:29AM +0000, Luk Schoonaert wrote:
    > I enabled /opt/bro/share/bro/site/local.bro -> @load policy/protocols/smb
    > 
    > Running BRO 2.5.1 - I never get the smb_file.log, I do get these:
    
    First the thing I have to say - please update to 2.5.5. There are only
    minor changes to 2.5.1 and a lot of fixed security issues.
    
    Or - consider upgrading to 2.6 (which admittedly has a bunch of changes). 
    
    > smb_cmd.log
    > smb_mapping.log
    > 
    > When I copy a file over SMB I;d expect ths smb_files.log to be populated
    > - I?m sure I?m missing something very simple, anyone have an idea?
    
    I think you are right and that it should typically be logged.
    
    There are 2 ways that I would start debugging this. First - if possible,
    make a pcap of an operation that you would expect to create the
    smb_files.log.
    
    Run that through bro, and see if it is there now; if not, take a look at
    smb_cmd.log and look if you can find activity that corresponds to the file
    copying in there.
    
    Johanna
    _______________________________________________
    Bro mailing list
    bro at bro-ids.org
    http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From seth at corelight.com  Fri Nov 30 06:54:12 2018
From: seth at corelight.com (Seth Hall)
Date: Fri, 30 Nov 2018 09:54:12 -0500
Subject: [Bro] SMB files log
In-Reply-To: <1A838220-919A-4DE2-8F5A-8ACDB943DF97@crimsoncore.be>
References: <1A838220-919A-4DE2-8F5A-8ACDB943DF97@crimsoncore.be>
Message-ID: <26469FBF-875F-47BB-92AF-44CBA43FC3BF@corelight.com>

Are you sure that you have activity occurring that would result in the 
smb_files.log being created?

   .Seth

On 29 Nov 2018, at 4:00, Luk Schoonaert wrote:

> Hey guys,
>
> I?m new to this mailing list - and I have a question about enabling 
> the SMB analyser, I?m sure I?m missing something simple.
>
> I enabled /opt/bro/share/bro/site/local.bro -> @load 
> policy/protocols/smb
>
> Running BRO 2.5.1 - I never get the smb_file.log, I do get these:
>
> smb_cmd.log
> smb_mapping.log
>
> When I copy a file over SMB I;d expect ths smb_files.log to be 
> populated - I?m sure I?m missing something very simple, anyone 
> have an idea?
>
> Many Thanks,
> Luk
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

--
Seth Hall * Corelight, Inc * www.corelight.com

From seth at corelight.com  Fri Nov 30 06:54:28 2018
From: seth at corelight.com (Seth Hall)
Date: Fri, 30 Nov 2018 09:54:28 -0500
Subject: [Bro] Bro 2.6 release
In-Reply-To: <CAMzgZ0JGUxgkXF96Y4dJYwsEQ3cMHQ5BL4+ktZppCvB5M-nDZw@mail.gmail.com>
References: <CAMzgZ0JGUxgkXF96Y4dJYwsEQ3cMHQ5BL4+ktZppCvB5M-nDZw@mail.gmail.com>
Message-ID: <72BD17BE-D813-47C9-8C14-FB92B03E1B5F@corelight.com>

Woo!  Always an exciting day.

  .Seth

On 29 Nov 2018, at 13:24, Jon Siwek wrote:

> Bro v2.6 is now released and available for download:
>
>     https://www.bro.org/download/index.html
>     https://www.bro.org/downloads/bro-2.6.tar.gz
>
> The most significant change to be aware of is that Bro has
> switched to using the new Broker communication library.  As a
> result, user-written scripts related to cluster operation or
> remote communication that worked in previous versions may
> require porting to new APIs.
>
> Please read the release notes carefully for helpful porting tips
> or other changes relevant to the upgrade process:
>
>     https://www.bro.org/sphinx/install/release-notes.html
>
> Also note that the Bro project is in the process of being
> renamed to Zeek, however, the software distribution for this
> release is still named Bro.  There's not yet been any related
> naming changes that alter usage for any provided tools or APIs.
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

--
Seth Hall * Corelight, Inc * www.corelight.com

From jazoff at illinois.edu  Fri Nov 30 11:14:41 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Fri, 30 Nov 2018 19:14:41 +0000
Subject: [Bro] sum stats q.
In-Reply-To: <CAGATen5C6xCKOZWBpB+i0tKTBiRiEDBdboSrSD-UZHpmGxC32g@mail.gmail.com>
References: <CAGATen6=jzbiWEfAy2KEdtoCEUcxtZqV9Xwa-4+mVBXnjffq1w@mail.gmail.com>
	<SN6PR11MB3216DB8C8E6A41D6FB5A238DA5D20@SN6PR11MB3216.namprd11.prod.outlook.com>,
	<CAGATen5C6xCKOZWBpB+i0tKTBiRiEDBdboSrSD-UZHpmGxC32g@mail.gmail.com>
Message-ID: <SN6PR11MB3216C5D5FD148036B605F8E2A5D30@SN6PR11MB3216.namprd11.prod.outlook.com>

> 
> Hi Justin,
> Thanks for responding. My problem is not with try.bro.org but with how sumstats seem to work. I was just using try.bro.org to demonstrate the issue in case someone wanted to try my test.
> 

Hi,

While trying to reproduce your problem I found that this was fixed a few months ago:

https://github.com/bro/bro/commit/3495b2fa9d84e8105a79e24e4e9a2f9181318f1a#diff-3248d64d10c61bb0656f5c167feca5f0

I ended up tracking down the root cause only to realize this is already fixed
in 2.6 :-)  Never hurts to practice bro script debugging though.  Turns out the old script was deleting entries from a table while iterating over it, which is undefined behavior in bro (and in many other languages).

I have a directory with http.pcap and your script (s.bro)

I run a bro 2.5.5 container and count the results, getting 128 instead of 197.

    justin at mbp:~/b$ docker run -t -i --rm -v `pwd`:/b broplatform/bro:2.5.5
    root at cbd05c9035c3:/# cd /b
    root at cbd05c9035c3:/b# bro -r http.pcap s.bro 
    Creating HttpStats log stream and HTTP sumstats
    1320279683.449294 ./s.bro, line 55: scount=197
    root at cbd05c9035c3:/b# 
    root at cbd05c9035c3:/b# cat http-stats.log |bro-cut hits | awk '{s+=$1} END {printf "%.0f\n", s}'
    128

Now I do the same test again but using bro 2.6 released yesterday and get the correct result of 197:

    justin at mbp:~/b$ docker run -t -i --rm -v `pwd`:/b broplatform/bro:2.6
    root at 869655245d1d:/# cd /b
    root at 869655245d1d:/b# bro -r http.pcap s.bro 
    Creating HttpStats log stream and HTTP sumstats
    1320279683.449294 ./s.bro, line 55: scount=197
    root at 869655245d1d:/b# 
    root at 869655245d1d:/b# cat http-stats.log |bro-cut hits | awk '{s+=$1} END {printf "%.0f\n", s}'
    197


-- 
Justin

From dnj0496 at gmail.com  Fri Nov 30 12:03:59 2018
From: dnj0496 at gmail.com (Dk Jack)
Date: Fri, 30 Nov 2018 12:03:59 -0800
Subject: [Bro] sum stats q.
In-Reply-To: <SN6PR11MB3216C5D5FD148036B605F8E2A5D30@SN6PR11MB3216.namprd11.prod.outlook.com>
References: <CAGATen6=jzbiWEfAy2KEdtoCEUcxtZqV9Xwa-4+mVBXnjffq1w@mail.gmail.com>
	<SN6PR11MB3216DB8C8E6A41D6FB5A238DA5D20@SN6PR11MB3216.namprd11.prod.outlook.com>
	<CAGATen5C6xCKOZWBpB+i0tKTBiRiEDBdboSrSD-UZHpmGxC32g@mail.gmail.com>
	<SN6PR11MB3216C5D5FD148036B605F8E2A5D30@SN6PR11MB3216.namprd11.prod.outlook.com>
Message-ID: <565D162F-4992-44F8-92D2-0A85BDB343B9@gmail.com>

Thanks for investigating this Justin. I was scratching my head for two days :)

Btw, I am using 2.4.1. Since my requirements were very simple, I ended up creating my own table and writing the accumulated counts to the log periodically using the ?schedule? primitive. That?s working correctly. Hopefully, I can get rid of that and move to the sumstats version when I upgrade my bro to 2.6. 

Thanks again. 

Dk. 

On Nov 30, 2018, at 11:14 AM, Azoff, Justin S <jazoff at illinois.edu> wrote:

>> 
>> Hi Justin,
>> Thanks for responding. My problem is not with try.bro.org but with how sumstats seem to work. I was just using try.bro.org to demonstrate the issue in case someone wanted to try my test.
>> 
> 
> Hi,
> 
> While trying to reproduce your problem I found that this was fixed a few months ago:
> 
> https://github.com/bro/bro/commit/3495b2fa9d84e8105a79e24e4e9a2f9181318f1a#diff-3248d64d10c61bb0656f5c167feca5f0
> 
> I ended up tracking down the root cause only to realize this is already fixed
> in 2.6 :-)  Never hurts to practice bro script debugging though.  Turns out the old script was deleting entries from a table while iterating over it, which is undefined behavior in bro (and in many other languages).
> 
> I have a directory with http.pcap and your script (s.bro)
> 
> I run a bro 2.5.5 container and count the results, getting 128 instead of 197.
> 
>    justin at mbp:~/b$ docker run -t -i --rm -v `pwd`:/b broplatform/bro:2.5.5
>    root at cbd05c9035c3:/# cd /b
>    root at cbd05c9035c3:/b# bro -r http.pcap s.bro 
>    Creating HttpStats log stream and HTTP sumstats
>    1320279683.449294 ./s.bro, line 55: scount=197
>    root at cbd05c9035c3:/b# 
>    root at cbd05c9035c3:/b# cat http-stats.log |bro-cut hits | awk '{s+=$1} END {printf "%.0f\n", s}'
>    128
> 
> Now I do the same test again but using bro 2.6 released yesterday and get the correct result of 197:
> 
>    justin at mbp:~/b$ docker run -t -i --rm -v `pwd`:/b broplatform/bro:2.6
>    root at 869655245d1d:/# cd /b
>    root at 869655245d1d:/b# bro -r http.pcap s.bro 
>    Creating HttpStats log stream and HTTP sumstats
>    1320279683.449294 ./s.bro, line 55: scount=197
>    root at 869655245d1d:/b# 
>    root at 869655245d1d:/b# cat http-stats.log |bro-cut hits | awk '{s+=$1} END {printf "%.0f\n", s}'
>    197
> 
> 
> -- 
> Justin