From mirugy at gmail.com  Mon Jan  4 02:50:34 2016
From: mirugy at gmail.com (=?UTF-8?B?R3nDtnJneSBNaXJ1?=)
Date: Mon, 4 Jan 2016 11:50:34 +0100
Subject: [Bro] log writer issue
In-Reply-To: <D9C3A112-D79C-426D-8A49-BC04E0B8398C@illinois.edu>
References: <CABm_LfsXEYXGOuxzmMr4-f+9no9kKHmbO07zn-3cz+RYqZS5XQ@mail.gmail.com>
	<CABm_LftWmpirU=4iTFV+=PnRcb5_36G+gKB_tmiOV9w36OtLZA@mail.gmail.com>
	<D9C3A112-D79C-426D-8A49-BC04E0B8398C@illinois.edu>
Message-ID: <CABm_Lfu0KSLFQ_km008vqiZgunzWe73u=pZVWJOAijUdJ-GeWA@mail.gmail.com>

There was no stderr.log, however -B logging helped. It was in fact a type
mismatch.

Thanks for the help,

Gy M

On Tue, Dec 29, 2015 at 4:15 PM, Azoff, Justin S <jazoff at illinois.edu>
wrote:

>
> > On Dec 29, 2015, at 9:34 AM, Gy?rgy Miru <mirugy at gmail.com> wrote:
> >
> > This happens before the first event is logged, however the headers are
> already written into the logfile
>
> Was there a stderr.log ?
>
> Does it happen before the event would have been logged at all, or in the
> process of logging the event?
>
> If you add a
>
>     print "This is siemenss7_write_data_unsigned"; #or
> siemenss7_read_data_unsigned
>     print c$s7data;
>
> before the calls to
>
>     Log::write(S7comm::LOG3, c$s7data);
>
> what gets output to stdout (or the stdout.log if you are using broctl)?
>
> I think this may be caused by one of the fields in one of your events
> being invalid somehow...
>
> > debug_s7data.log: relevant part of the debug.log file, when bro was run
> with -B threading switch
>
> You really want -B logging
>
> I have a feeling you'll see a "Field type doesn't match in
> WriterBackend::Write" message
>
> --
> - Justin Azoff
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160104/35bafeed/attachment.html 

From franky.meier.1 at gmx.de  Tue Jan  5 00:57:18 2016
From: franky.meier.1 at gmx.de (Frank Meier)
Date: Tue, 5 Jan 2016 09:57:18 +0100
Subject: [Bro] =?utf-8?b?QnJvQ29uIOKAmDE2?=
Message-ID: <20160105095718.3cde59ef@NB181106>

Hi and Happy New Year!

Will there be a BroCon in 2016? When will that be?

Thanks

Franky

From jdopheid at illinois.edu  Tue Jan  5 06:14:49 2016
From: jdopheid at illinois.edu (Dopheide, Jeannette M)
Date: Tue, 5 Jan 2016 14:14:49 +0000
Subject: [Bro] =?utf-8?b?QnJvQ29uIOKAmDE2?=
In-Reply-To: <20160105095718.3cde59ef@NB181106>
References: <20160105095718.3cde59ef@NB181106>
Message-ID: <597C289E-1A81-449C-BBEA-2CD2204FB92F@illinois.edu>

Happy New Year Franky!

Yes. We're still negotiating the location and dates with the event venue. We're very close to announcing, though. 

------

Jeannette Dopheide
Bro Outreach Coordinator
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign








On 1/5/16, 2:57 AM, "bro-bounces at bro.org on behalf of Frank Meier" <bro-bounces at bro.org on behalf of franky.meier.1 at gmx.de> wrote:

Hi and Happy New Year!

Will there be a BroCon in 2016? When will that be?

Thanks

Franky
_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From tgdesrochers at gmail.com  Wed Jan  6 07:33:41 2016
From: tgdesrochers at gmail.com (Tim Desrochers)
Date: Wed, 6 Jan 2016 10:33:41 -0500
Subject: [Bro] ACTION_ALARM and ACTION_EMAIL
Message-ID: <CANrcqc3kzts=5Y8z+Xq6PTw-N_wO=koFEwDbFAB3V9nnoCtjGQ@mail.gmail.com>

I have my sensor set up to email me notices with:

hook Notice::policy(n: Notice::Info)
            {
            add n$actions[Notice::ACTION_EMAIL];
            }

If I understand correct this will email me upon any entry in the
notice.log.  Is there a way to:
1. only get specific items emailed upon entry
2. get the rest of notice.log entries emailed with ACTON_ALARM in the
alarm-mail.txt and have that ignore anything that was previously emailed.
3. Only get one notice email per alert?

What I am doing is in the /opt/bro/share/bro/intel folder creating
different folders with IOS's I want the intel framework to look over and I
am using meta.do_notice to send the items of importance to the notice log.

Excuse my ignorance with this subject I am just now trying to get things
emailed out efficiently to reduce some noise and redundancy my analysts are
seeing.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160106/4d0dd394/attachment.html 

From jazoff at illinois.edu  Wed Jan  6 07:50:47 2016
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Wed, 6 Jan 2016 15:50:47 +0000
Subject: [Bro] ACTION_ALARM and ACTION_EMAIL
In-Reply-To: <CANrcqc3kzts=5Y8z+Xq6PTw-N_wO=koFEwDbFAB3V9nnoCtjGQ@mail.gmail.com>
References: <CANrcqc3kzts=5Y8z+Xq6PTw-N_wO=koFEwDbFAB3V9nnoCtjGQ@mail.gmail.com>
Message-ID: <CBC518B3-3BC8-4EA8-92DF-1AE453C3F373@illinois.edu>

I'm not sure about #2, but for 1 and 3 there is an easy way to do this with the default configuration.  The notice framework has this as the notice policy:

hook Notice::policy(n: Notice::Info) &priority=10
    {
    if ( n$note in Notice::ignored_types )
        break;

    if ( n$note in Notice::not_suppressed_types )
        n$suppress_for=0secs;
    if ( n$note in Notice::alarmed_types )
        add n$actions[ACTION_ALARM];
    if ( n$note in Notice::emailed_types )
        add n$actions[ACTION_EMAIL];

    if ( n$note in Notice::type_suppression_intervals )
        n$suppress_for=Notice::type_suppression_intervals[n$note];

    # Logging is a default action.  It can be removed in a later hook if desired.
    add n$actions[ACTION_LOG];
    }

Those tables are all setup to make it easy to toggle actions:

    ## Ignored notice types.
    const ignored_types: set[Notice::Type] = {} &redef;
    ## Emailed notice types.
    const emailed_types: set[Notice::Type] = {} &redef;
    ## Alarmed notice types.
    const alarmed_types: set[Notice::Type] = {} &redef;
    ## Types that should be suppressed for the default suppression interval.
    const not_suppressed_types: set[Notice::Type] = {} &redef;

So you simply need something like this in your local.bro:

redef Notice::emailed_types += {
    HTTP::SQL_Injection_Attacker,
    HTTP::SQL_Injection_Victim,
}

If you do need to do anything more complicated, you can use your own Notice::policy and add whatever logic you want.

To not get multiple emails for the same notice you need to ensure that the notice has the $identifier set that uniquely identifies the notice.  This is minimally something like cat(id$orig_h).  If you look at any of the scripts in policy/ you can see how they do this.



-- 
- Justin Azoff

> On Jan 6, 2016, at 10:33 AM, Tim Desrochers <tgdesrochers at gmail.com> wrote:
> 
> I have my sensor set up to email me notices with:
> 
> hook Notice::policy(n: Notice::Info)
>             {
>             add n$actions[Notice::ACTION_EMAIL];
>             }
> 
> If I understand correct this will email me upon any entry in the notice.log.  Is there a way to:
> 1. only get specific items emailed upon entry
> 2. get the rest of notice.log entries emailed with ACTON_ALARM in the alarm-mail.txt and have that ignore anything that was previously emailed.
> 3. Only get one notice email per alert?
> 
> What I am doing is in the /opt/bro/share/bro/intel folder creating different folders with IOS's I want the intel framework to look over and I am using meta.do_notice to send the items of importance to the notice log.
> 
> Excuse my ignorance with this subject I am just now trying to get things emailed out efficiently to reduce some noise and redundancy my analysts are seeing.
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From tgdesrochers at gmail.com  Wed Jan  6 08:00:51 2016
From: tgdesrochers at gmail.com (Tim Desrochers)
Date: Wed, 6 Jan 2016 11:00:51 -0500
Subject: [Bro] ACTION_ALARM and ACTION_EMAIL
In-Reply-To: <CBC518B3-3BC8-4EA8-92DF-1AE453C3F373@illinois.edu>
References: <CANrcqc3kzts=5Y8z+Xq6PTw-N_wO=koFEwDbFAB3V9nnoCtjGQ@mail.gmail.com>
	<CBC518B3-3BC8-4EA8-92DF-1AE453C3F373@illinois.edu>
Message-ID: <CANrcqc0q8KLEuYdeMW5b179j4paz_dU-+B3fHBuOqe300dSShw@mail.gmail.com>

Thanks I'll give it a shot
On Jan 6, 2016 10:50 AM, "Azoff, Justin S" <jazoff at illinois.edu> wrote:

> I'm not sure about #2, but for 1 and 3 there is an easy way to do this
> with the default configuration.  The notice framework has this as the
> notice policy:
>
> hook Notice::policy(n: Notice::Info) &priority=10
>     {
>     if ( n$note in Notice::ignored_types )
>         break;
>
>     if ( n$note in Notice::not_suppressed_types )
>         n$suppress_for=0secs;
>     if ( n$note in Notice::alarmed_types )
>         add n$actions[ACTION_ALARM];
>     if ( n$note in Notice::emailed_types )
>         add n$actions[ACTION_EMAIL];
>
>     if ( n$note in Notice::type_suppression_intervals )
>         n$suppress_for=Notice::type_suppression_intervals[n$note];
>
>     # Logging is a default action.  It can be removed in a later hook if
> desired.
>     add n$actions[ACTION_LOG];
>     }
>
> Those tables are all setup to make it easy to toggle actions:
>
>     ## Ignored notice types.
>     const ignored_types: set[Notice::Type] = {} &redef;
>     ## Emailed notice types.
>     const emailed_types: set[Notice::Type] = {} &redef;
>     ## Alarmed notice types.
>     const alarmed_types: set[Notice::Type] = {} &redef;
>     ## Types that should be suppressed for the default suppression
> interval.
>     const not_suppressed_types: set[Notice::Type] = {} &redef;
>
> So you simply need something like this in your local.bro:
>
> redef Notice::emailed_types += {
>     HTTP::SQL_Injection_Attacker,
>     HTTP::SQL_Injection_Victim,
> }
>
> If you do need to do anything more complicated, you can use your own
> Notice::policy and add whatever logic you want.
>
> To not get multiple emails for the same notice you need to ensure that the
> notice has the $identifier set that uniquely identifies the notice.  This
> is minimally something like cat(id$orig_h).  If you look at any of the
> scripts in policy/ you can see how they do this.
>
>
>
> --
> - Justin Azoff
>
> > On Jan 6, 2016, at 10:33 AM, Tim Desrochers <tgdesrochers at gmail.com>
> wrote:
> >
> > I have my sensor set up to email me notices with:
> >
> > hook Notice::policy(n: Notice::Info)
> >             {
> >             add n$actions[Notice::ACTION_EMAIL];
> >             }
> >
> > If I understand correct this will email me upon any entry in the
> notice.log.  Is there a way to:
> > 1. only get specific items emailed upon entry
> > 2. get the rest of notice.log entries emailed with ACTON_ALARM in the
> alarm-mail.txt and have that ignore anything that was previously emailed.
> > 3. Only get one notice email per alert?
> >
> > What I am doing is in the /opt/bro/share/bro/intel folder creating
> different folders with IOS's I want the intel framework to look over and I
> am using meta.do_notice to send the items of importance to the notice log.
> >
> > Excuse my ignorance with this subject I am just now trying to get things
> emailed out efficiently to reduce some noise and redundancy my analysts are
> seeing.
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160106/a43eece2/attachment.html 

From cw13 at umbc.edu  Wed Jan  6 09:54:50 2016
From: cw13 at umbc.edu (Chris Williams)
Date: Wed, 6 Jan 2016 12:54:50 -0500
Subject: [Bro] CIDR and limitation on indicators
Message-ID: <CAGEyGov88xT9A8svGGJUxyr55LA0Kg9s=uRN1-hXxzuahRDjuw@mail.gmail.com>

I have been happily utilizing the criticalstack plugin which allows for
integrating a number of feeds as indicators. I am ultimately trying to
establish a system of creating, maintaining, and pushing a blacklist of my
own, but I have come across an issue/question regarding the limitations of
BRO.

I am currently using the intel framework with 5 feeds and 68k+ indicators.
Their documentation suggested that it can allow the integration of your own
list, and I wanted to try Spamhaus' list located here: [
https://www.spamhaus.org/drop/drop.txt]

I wrote Critical Stack to ask about the integration of CIDR blocks as
indicators and I was told:

"The sheer volume of addresses was actually one of the reasons that we did
not include lists like Spamhaus. With the current limitations in Bro of
only being able to handle 100-200K indicators most of the CIDR lists pushed
us way over that limit very quickly.

We did start work on CIDR expansion but it just wasn't going to fit into
the current state of Bro's Intel Framework."

I am curious because the documentation [
https://www.bro.org/sphinx/script-reference/types.html#type-addr] suggests
that BRO can read CIDR notation, so is there an upper limitation with
respect to BRO's indicators? If so, what is it? Is it hardware bound, or
can it be improved somewhow?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160106/db8e8742/attachment.html 

From jazoff at illinois.edu  Wed Jan  6 10:11:34 2016
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Wed, 6 Jan 2016 18:11:34 +0000
Subject: [Bro] CIDR and limitation on indicators
In-Reply-To: <CAGEyGov88xT9A8svGGJUxyr55LA0Kg9s=uRN1-hXxzuahRDjuw@mail.gmail.com>
References: <CAGEyGov88xT9A8svGGJUxyr55LA0Kg9s=uRN1-hXxzuahRDjuw@mail.gmail.com>
Message-ID: <1D56BE40-E044-4168-8924-792A75B9278D@illinois.edu>


> On Jan 6, 2016, at 12:54 PM, Chris Williams <cw13 at umbc.edu> wrote:
> 
> I am curious because the documentation [https://www.bro.org/sphinx/script-reference/types.html#type-addr] suggests that BRO can read CIDR notation, so is there an upper limitation with respect to BRO's indicators? If so, what is it? Is it hardware bound, or can it be improved somewhow? 

This has come up a few times, there isn't a huge technical reason why the intel framework could not be modified to handle CIDR blocks as well as individual hosts.

The one potential problem is overlap.  If one indicator exists for 192.168.1.0/24 and another has 192.168.0.0/16, a match for 192.168.1.1 will only pull up the record for the larger prefix.  This can be a problem if you are really need the information present in both entries.

The most recent ticket I can find about this is https://bro-tracker.atlassian.net/browse/BIT-1167

That has a patch, but it needs to be modified as the comments mention.






-- 
- Justin Azoff



From rfjl12345 at gmail.com  Wed Jan  6 17:17:40 2016
From: rfjl12345 at gmail.com (Robert Young)
Date: Wed, 6 Jan 2016 19:17:40 -0600
Subject: [Bro] NTP Analyzer not working as expected
Message-ID: <CACw8FnZysQBj=+H6-4RP_Hi96yy2JWt_P7EBiQG4g=xagZ47aQ@mail.gmail.com>

Hi Guys, I am trying to detect hosts that are ntp clients to verify they
are not also acting as a server.  I have setup the basic script as seen
below using event ntp_msg().  When I run the code I see the msg code for
client(3) and server(4) as expected.  But what does not look correct is the
orig_h is the same for both the request from the client and the response
from the server.  In this test the client is 172.16.1.7 and they server is
172.16.1. 41  Anyone have any ideas of what I may have missed ? or have I
hit a bug ?

Regards,
Robert


Debug output:
>>>>>>>>>>>>>>>>>>>>
ID:=, 1229867348
orig_h=, 172.16.1.7
resp_host=, 172.16.1.41
msg code=, 3
excess=,
 <<<<<<<<<<<<<<<<<<
>>>>>>>>>>>>>>>>>>>>
ID:=, 2733850379
orig_h=, 172.16.1.7
resp_host=, 172.16.1.41
msg code=, 4
excess=,
 <<<<<<<<<<<<<<<<<<

[image: Inline image 1]


Code:

module NTP;

export {
        redef enum Log::ID += { LOG };

        redef enum Notice::Type += {
                NTP_ALARM,
                NTP_Monlist_Queries,
                };

        type ntp_record: record {
                ts: time &log;
                uid: string &log;
                orig: addr &log;
                resp: addr &log;
                refid: count &default=0 &log;
                code: count &default=0 &log;
                stratum: count &default=0 &log;
                poll: count &default=0 &log;
                precision: int &default=to_int("0") &log;
                #distance: interval;
                #dispersion: interval;
                reftime: time &log;
                #orig: time;
                #rec: time;
                #xmt: time;
                excess: string &default="NULL" &log;
                };

        # The code value maps to the NTP mode type - for now I am mostly
        #  interested in control messages.
        #
        # Mode  Description
        # 0     reserved.
        # 1     Symmetric active.
        # 2     Symmetric passive.
        # 3     Client.
        # 4     Server.
        # 5     Broadcast.
        # 6     NTP control message.
        # 7     private use.
        const NTP_RESERVED = 0;
        const NTP_SYM_ACTIVE = 1;
        const NTP_SYM_PASSIVE = 2;
        const NTP_CLIENT = 3;
        const NTP_SERVER = 4;
        const NTP_BROADCAST = 5;
        const NTP_CONTROL = 6;
        const NTP_PRIVATE = 7;
        const ports = { 123/udp,};
        redef likely_server_ports += { ports };

        const log_only_control: bool = F &redef;

        # So we don't warn more than one time
        global ntp_host: table[addr] of count;
 hook Notice::policy(n: Notice::Info)
        {
        if ( n$note == NTP::NTP_ALARM )
         add n$actions[Notice::ACTION_EMAIL];
        }

redef capture_filters += {
    ["watched network"] = "net 172.16.1.41/32"};
        } # end export


redef Site::local_nets += {
        172.16.1.41/32
};

event bro_init() &priority=5
        {
        Analyzer::register_for_ports(Analyzer::ANALYZER_NTP, ports);
        }

event ntp_message(c: connection, msg: ntp_msg, excess: string)

 {
        local manage_ip:addr = 0.0.0.0;
        local int_desc:string ="none";
        print">>>>>>>>>>>>>>>>>>>>";
        print"ID:=",msg$id;
        print "orig_h=",c$id$orig_h;
        print "resp_host=",c$id$resp_h;
        print"msg code=",msg$code;
        print "excess=",excess;
        print " <<<<<<<<<<<<<<<<<<";

}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160106/c505dad4/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 26911 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160106/c505dad4/attachment-0002.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: wireshark.PNG
Type: image/png
Size: 21980 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160106/c505dad4/attachment-0003.bin 

From giedrius.ramas at gmail.com  Thu Jan  7 07:10:13 2016
From: giedrius.ramas at gmail.com (Giedrius Ramas)
Date: Thu, 7 Jan 2016 17:10:13 +0200
Subject: [Bro] increase log rotation interval
Message-ID: <CAHr6V5rjto-6RKQ884BN-1qzXw=_68PrmuwWj_5RYqUDCWO66w@mail.gmail.com>

Hi all ,
I was just wondering if any issues could cause if I increase
LogRotationInterval from 3600 to higher . Could I run into any problems If
so what kind of ? I would like to have day rotation instead of one hour.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160107/3179f44b/attachment.html 

From jlay at slave-tothe-box.net  Thu Jan  7 07:34:40 2016
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 07 Jan 2016 08:34:40 -0700
Subject: [Bro] increase log rotation interval
In-Reply-To: <CAHr6V5rjto-6RKQ884BN-1qzXw=_68PrmuwWj_5RYqUDCWO66w@mail.gmail.com>
References: <CAHr6V5rjto-6RKQ884BN-1qzXw=_68PrmuwWj_5RYqUDCWO66w@mail.gmail.com>
Message-ID: <c08a47c7edb80ac6ca82c4af470aad05@localhost>

 

Set: 

LogRotationInterval = 86400 

in your broctl.cfg 

James 

On 2016-01-07 08:10, Giedrius Ramas wrote: 

> Hi all ,  
> I was just wondering if any issues could cause if I increase LogRotationInterval from 3600 to higher . Could I run into any problems If so what kind of ? I would like to have day rotation instead of one hour.  
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160107/25fe4155/attachment.html 

From vitologrillo at gmail.com  Thu Jan  7 07:49:54 2016
From: vitologrillo at gmail.com (Vito Logrillo)
Date: Thu, 7 Jan 2016 16:49:54 +0100
Subject: [Bro] Bro and pf_ring
Message-ID: <CANdYiFHt5Y8S1EVnNTAxtvMCvus5g-U=nfjPmgogNuXvm0F8zQ@mail.gmail.com>

Hi all,
i'm using bro with the pf_ring driver in a cluster architecture as
written in the link below

https://www.bro.org/documentation/load-balancing.html

Now i've seen a plugin for bro able to provide native pf_ring support

https://github.com/bro/bro-plugins/tree/master/pf_ring

Sorry for the lazy question, but which are the benefits of this plugin?
Thanks

From seth at icir.org  Thu Jan  7 08:04:32 2016
From: seth at icir.org (Seth Hall)
Date: Thu, 7 Jan 2016 11:04:32 -0500
Subject: [Bro] NTP Analyzer not working as expected
In-Reply-To: <CACw8FnZysQBj=+H6-4RP_Hi96yy2JWt_P7EBiQG4g=xagZ47aQ@mail.gmail.com>
References: <CACw8FnZysQBj=+H6-4RP_Hi96yy2JWt_P7EBiQG4g=xagZ47aQ@mail.gmail.com>
Message-ID: <67DE3C88-A0FD-4761-99FB-9D9CB125462F@icir.org>


> On Jan 6, 2016, at 8:17 PM, Robert Young <rfjl12345 at gmail.com> wrote:
> 
> Hi Guys, I am trying to detect hosts that are ntp clients to verify they are not also acting as a server.  I have setup the basic script as seen below using event ntp_msg().  When I run the code I see the msg code for client(3) and server(4) as expected.  But what does not look correct is the orig_h is the same for both the request from the client and the response from the server.  In this test the client is 172.16.1.7 and they server is 172.16.1. 41  Anyone have any ideas of what I may have missed ? or have I hit a bug ?

Bro ?sessionizes? UDP traffic.  What you are seeing is the result of that.  The assumption is the first to speak is the originator of the ?connection?.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From lucovsky at reservoir.com  Thu Jan  7 12:30:52 2016
From: lucovsky at reservoir.com (Jeff Lucovsky)
Date: Thu, 7 Jan 2016 15:30:52 -0500
Subject: [Bro] Bro and pf_ring
In-Reply-To: <CANdYiFHt5Y8S1EVnNTAxtvMCvus5g-U=nfjPmgogNuXvm0F8zQ@mail.gmail.com>
References: <CANdYiFHt5Y8S1EVnNTAxtvMCvus5g-U=nfjPmgogNuXvm0F8zQ@mail.gmail.com>
Message-ID: <CAHqrUQ7zz+4FBhyE7pzsDd1CjpDPL2Ag=t3wJjF-PJLXU1GbQQ@mail.gmail.com>

Hi Vito,

If you're using Bro 2.4 or later, the pf_ring plugin is included in the
stock Bro distribution. If that's the case, then you do not need the plugin
from github.

The plugin provides packet access when your setup includes PF_RING *and*
you're using Bro 2.4 or above. If you're using Bro 2.3 or earlier, plugins
don't apply as the architecture didn't support them.

When Bro introduced I/O sources with 2.4, the plugin architecture was
created; Bro 2.3 didn't use plugins for packet access. Check out the other
Bro 2.4 plugins in <bro-source-tree>/aux/bro-plugins.


-- 
Jeff Lucovsky
*Reservoir *Labs
212 780 0527 x173 <212%20780%200527%20x100>

On Thu, Jan 7, 2016 at 10:49 AM, Vito Logrillo <vitologrillo at gmail.com>
wrote:

> Hi all,
> i'm using bro with the pf_ring driver in a cluster architecture as
> written in the link below
>
> https://www.bro.org/documentation/load-balancing.html
>
> Now i've seen a plugin for bro able to provide native pf_ring support
>
> https://github.com/bro/bro-plugins/tree/master/pf_ring
>
> Sorry for the lazy question, but which are the benefits of this plugin?
> Thanks
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160107/4ac53248/attachment.html 

From ryoung16 at harris.com  Thu Jan  7 12:55:02 2016
From: ryoung16 at harris.com (Young, Robert (ryoung16))
Date: Thu, 7 Jan 2016 20:55:02 +0000
Subject: [Bro] NTP Analyzer not working as expected
In-Reply-To: <67DE3C88-A0FD-4761-99FB-9D9CB125462F@icir.org>
References: <CACw8FnZysQBj=+H6-4RP_Hi96yy2JWt_P7EBiQG4g=xagZ47aQ@mail.gmail.com>
	<67DE3C88-A0FD-4761-99FB-9D9CB125462F@icir.org>
Message-ID: <45bccc0a33214b12a25128be421f7743@MLBXCH21.cs.myharris.net>

This was the response I received

Robert Young
Senior Network Engineer/Team Lead, Terrestrial Network Engineering, Shared Services
HARRIS CAPROCK
Office: +1-832-668-2635 / Mobile: +1-281-701-9684


-----Original Message-----
From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Seth Hall
Sent: Thursday, January 07, 2016 10:05 AM
To: Robert Young <rfjl12345 at gmail.com>
Cc: bro at bro.org
Subject: Re: [Bro] NTP Analyzer not working as expected


> On Jan 6, 2016, at 8:17 PM, Robert Young <rfjl12345 at gmail.com> wrote:
> 
> Hi Guys, I am trying to detect hosts that are ntp clients to verify they are not also acting as a server.  I have setup the basic script as seen below using event ntp_msg().  When I run the code I see the msg code for client(3) and server(4) as expected.  But what does not look correct is the orig_h is the same for both the request from the client and the response from the server.  In this test the client is 172.16.1.7 and they server is 172.16.1. 41  Anyone have any ideas of what I may have missed ? or have I hit a bug ?

Bro ?sessionizes? UDP traffic.  What you are seeing is the result of that.  The assumption is the first to speak is the originator of the ?connection?.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/


_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From seth at icir.org  Thu Jan  7 13:15:39 2016
From: seth at icir.org (Seth Hall)
Date: Thu, 7 Jan 2016 16:15:39 -0500
Subject: [Bro] Bro and pf_ring
In-Reply-To: <CANdYiFHt5Y8S1EVnNTAxtvMCvus5g-U=nfjPmgogNuXvm0F8zQ@mail.gmail.com>
References: <CANdYiFHt5Y8S1EVnNTAxtvMCvus5g-U=nfjPmgogNuXvm0F8zQ@mail.gmail.com>
Message-ID: <C74D1129-4198-45EB-A9A9-39E352BE7011@icir.org>


> On Jan 7, 2016, at 10:49 AM, Vito Logrillo <vitologrillo at gmail.com> wrote:
> 
> Now i've seen a plugin for bro able to provide native pf_ring support
> 
> https://github.com/bro/bro-plugins/tree/master/pf_ring
> 
> Sorry for the lazy question, but which are the benefits of this plugin?

You likely won?t see much of a benefit to using the plugin over using the libpcap wrapper honestly.  Doing the single layer of indirection that is caused by the wrapper doesn?t add up to much overhead.  Bro actually *doing* things causes most of the overhead.

The other small thing to keep in mind is that I haven?t heard many experiences of people using the plugin so the ?not widely used code? caveat applies. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From Paul.Nash at tufts.edu  Thu Jan  7 13:37:18 2016
From: Paul.Nash at tufts.edu (Nash, Paul)
Date: Thu, 7 Jan 2016 21:37:18 +0000
Subject: [Bro] Bro Packet Loss / 10gb ixgbe / pf_ring
Message-ID: <723D2963-E446-4251-ADF9-BEB7BE321202@tufts.edu>


I?m trying to debug some packet drops that I?m experiencing and am turning to the list for help.   The recorded packet loss is ~50 ? 70% at times.   The packet loss is recorded in broctl?s netstats as well as in the notice.log file.

Running netstats at startup ? I?m dropping more than I?m receiving from the very start.


[BroControl] > netstats

 worker-1-1: 1452200459.635155 recvd=734100 dropped=1689718 link=2424079

worker-1-10: 1452200451.830143 recvd=718461 dropped=1414234 link=718461

worker-1-11: 1452200460.036766 recvd=481010 dropped=2019289 link=2500560

worker-1-12: 1452200460.239585 recvd=720895 dropped=1805574 link=2526730

worker-1-13: 1452200460.440611 recvd=753365 dropped=1800827 link=2554453

worker-1-14: 1452200460.647368 recvd=784145 dropped=1800831 link=2585237

worker-1-15: 1452200460.844842 recvd=750921 dropped=1868186 link=2619368

worker-1-16: 1452200461.049237 recvd=742718 dropped=1908528 link=2651507

?

System information:
- 64 AMD Opteron System
- 128gb of RAM
- Intel 10gb IXGBE interface (dual 10gb interfaces, eth3 is the sniffer)
- Licensed copy of PF_Ring ZC

I?m running Bro 2.4.1, PF_Ring 6.2.0 on Centos  / 2.6.32-411 kernel

I have the proxy, manager & 16 workers running on the same system.  16 CPUs are pinned (0-15)

Startup scripts to load the various kernel modules (from PF_RING 6.2.0 src)


insmod /lib/modules/2.6.32-431.11.2.el6.x86_64/kernel/net/pf_ring/pf_ring.ko enable_tx_capture=0 min_num_slots=32768 quick_mode=1

insmod  /lib/modules/2.6.32-431.11.2.el6.x86_64/kernel/drivers/net/ixgbe/ixgbe.ko numa_cpu_affinity=0,0 MQ=0,1 RSS=0,0

I checked /proc/sys/pci/devices to confirm that the interface is running on numa_node 0.  ?lscpu? shows that cpus 0-7 are one node 0, socket 0, and cpus 8-15 are on node 1, socket 0.  I figured having the 16 RSS queues on the same socket is probably better than having them bounce around.

I?ve disabled a bunch of the ixgbe offloading stuff:


ethtool -K eth3 rx off

ethtool -K eth3 tx off

ethtool -K eth3 sg off

ethtool -K eth3 tso off

ethtool -K eth3 gso off

ethtool -K eth3 gro off

ethtool -K eth3 lro off

ethtool -K eth3 rxvlan off

ethtool -K eth3 txvlan off

ethtool -K eth3 ntuple off

ethtool -K eth3 rxhash off

ethtool -K eth3 rx 32768

I?ve also tuned the stack, per recommendations from SANS:


net.ipv4.tcp_timestamps = 0

net.ipv4.tcp_sack = 0

net.ipv4.tcp_rmem = 10000000 10000000 10000000

net.ipv4.tcp_wmem = 10000000 10000000 10000000

net.ipv4.tcp_mem = 10000000 10000000 10000000

net.core.rmem_max = 134217728

net.core.wmem_max = 134217728

net.core.netdev_max_backlog = 250000


The node.cfg looks like this:


[manager]

type=manager

host=10.99.99.15


#

[proxy-1]

type=proxy

host=10.99.99.15


#

[worker-1]

type=worker

host=10.99.99.15

interface=eth3

lb_method=pf_ring

lb_procs=16

pin_cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15


I have a license for ZC, and if I change the interface from eth3 to zc:eth3, it will spawn up 16 workers, but only one of them is receiving any traffic.  I?m assuming that it is looking at zc:eth3 at 0 only.   Netstats proves that out.   If I run pfcount ?I zc at eth3, it will show me that I?m receiving ~1gbp/s of traffic on the interface and not dropping anything.

Am I missing something obvious?  I saw many threads about disabling hyper threading, but that seems specific to intel processors ? I?m running AMD operterons with their own hyper transport stuff which doesn?t create virtual cpus.

Thanks,
 -Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160107/d241bd89/attachment-0001.html 

From luke at geekempire.com  Thu Jan  7 14:15:47 2016
From: luke at geekempire.com (Mike Reeves)
Date: Thu, 7 Jan 2016 17:15:47 -0500
Subject: [Bro] Bro Packet Loss / 10gb ixgbe / pf_ring
In-Reply-To: <723D2963-E446-4251-ADF9-BEB7BE321202@tufts.edu>
References: <723D2963-E446-4251-ADF9-BEB7BE321202@tufts.edu>
Message-ID: <CADC84wRQX-UmQvZAc3Rz43Esw5cQp5J4X7wnjOfZcASLk6YqeQ@mail.gmail.com>

Change your min_num_slots to be 65535. I would add an additional proxy as
well as an additional 8 workers.



On Thu, Jan 7, 2016 at 4:37 PM, Nash, Paul <Paul.Nash at tufts.edu> wrote:

>
> I?m trying to debug some packet drops that I?m experiencing and am turning
> to the list for help.   The recorded packet loss is ~50 ? 70% at times.
> The packet loss is recorded in broctl?s netstats as well as in the
> notice.log file.
>
> Running netstats at startup ? I?m dropping more than I?m receiving from
> the very start.
>
> [BroControl] > netstats
>
>  worker-1-1: 1452200459.635155 recvd=734100 dropped=1689718 link=2424079
>
> worker-1-10: 1452200451.830143 recvd=718461 dropped=1414234 link=718461
>
> worker-1-11: 1452200460.036766 recvd=481010 dropped=2019289 link=2500560
>
> worker-1-12: 1452200460.239585 recvd=720895 dropped=1805574 link=2526730
>
> worker-1-13: 1452200460.440611 recvd=753365 dropped=1800827 link=2554453
>
> worker-1-14: 1452200460.647368 recvd=784145 dropped=1800831 link=2585237
>
> worker-1-15: 1452200460.844842 recvd=750921 dropped=1868186 link=2619368
>
> worker-1-16: 1452200461.049237 recvd=742718 dropped=1908528 link=2651507
> ?
>
> System information:
>
> - 64 AMD Opteron System
> - 128gb of RAM
> - Intel 10gb IXGBE interface (dual 10gb interfaces, eth3 is the sniffer)
> - Licensed copy of PF_Ring ZC
>
> I?m running Bro 2.4.1, PF_Ring 6.2.0 on Centos  / 2.6.32-411 kernel
>
> I have the proxy, manager & 16 workers running on the same system.  16
> CPUs are pinned (0-15)
>
> Startup scripts to load the various kernel modules (from PF_RING 6.2.0 src)
>
> *insmod
> /lib/modules/2.6.32-431.11.2.el6.x86_64/kernel/net/pf_ring/pf_ring.ko
> enable_tx_capture=0 min_num_slots=32768 quick_mode=1*
>
> *insmod
> /lib/modules/2.6.32-431.11.2.el6.x86_64/kernel/drivers/net/ixgbe/ixgbe.ko
> numa_cpu_affinity=0,0 MQ=0,1 RSS=0,0*
>
>
> I checked /proc/sys/pci/devices to confirm that the interface is running
> on numa_node 0.  ?lscpu? shows that cpus 0-7 are one node 0, socket 0, and
> cpus 8-15 are on node 1, socket 0.  I figured having the 16 RSS queues on
> the same socket is probably better than having them bounce around.
>
> I?ve disabled a bunch of the ixgbe offloading stuff:
>
> *ethtool -K eth3 rx off*
>
> *ethtool -K eth3 tx off*
>
> *ethtool -K eth3 sg off*
>
> *ethtool -K eth3 tso off*
>
> *ethtool -K eth3 gso off*
>
> *ethtool -K eth3 gro off*
>
> *ethtool -K eth3 lro off*
>
> *ethtool -K eth3 rxvlan off*
>
> *ethtool -K eth3 txvlan off*
>
> *ethtool -K eth3 ntuple off*
>
> *ethtool -K eth3 rxhash off*
>
> *ethtool -K eth3 rx 32768*
>
>
> I?ve also tuned the stack, per recommendations from SANS:
>
> *net.ipv4.tcp_timestamps = 0*
>
> *net.ipv4.tcp_sack = 0*
>
> *net.ipv4.tcp_rmem = 10000000 10000000 10000000*
>
> *net.ipv4.tcp_wmem = 10000000 10000000 10000000*
>
> *net.ipv4.tcp_mem = 10000000 10000000 10000000*
>
> *net.core.rmem_max = 134217728*
>
> *net.core.wmem_max = 134217728*
>
> *net.core.netdev_max_backlog = 250000*
>
>
> The node.cfg looks like this:
>
> *[manager]*
>
> *type=manager*
>
> *host=10.99.99.15*
>
> *#*
>
> *[proxy-1]*
>
> *type=proxy*
>
> *host=10.99.99.15*
>
> *#*
>
> *[worker-1]*
>
> *type=worker*
>
> *host=10.99.99.15*
>
> *interface=eth3*
>
> *lb_method=pf_ring*
>
> *lb_procs=16*
>
> *pin_cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15*
>
>
>
> I have a license for ZC, and if I change the interface from eth3 to
> zc:eth3, it will spawn up 16 workers, but only one of them is receiving any
> traffic.  I?m assuming that it is looking at zc:eth3 at 0 only.   Netstats
> proves that out.   If I run pfcount ?I zc at eth3, it will show me that I?m
> receiving ~1gbp/s of traffic on the interface and not dropping anything.
>
> Am I missing something obvious?  I saw many threads about disabling hyper
> threading, but that seems specific to intel processors ? I?m running AMD
> operterons with their own hyper transport stuff which doesn?t create
> virtual cpus.
>
> Thanks,
>  -Paul
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160107/4d31e65f/attachment.html 

From gfaulkner.nsm at gmail.com  Thu Jan  7 16:04:05 2016
From: gfaulkner.nsm at gmail.com (Gary Faulkner)
Date: Thu, 7 Jan 2016 18:04:05 -0600
Subject: [Bro] Bro Packet Loss / 10gb ixgbe / pf_ring
In-Reply-To: <723D2963-E446-4251-ADF9-BEB7BE321202@tufts.edu>
References: <723D2963-E446-4251-ADF9-BEB7BE321202@tufts.edu>
Message-ID: <568EFCF5.1060008@gmail.com>

Some thoughts inline...

On 1/7/16 3:37 PM, Nash, Paul wrote:
> I?m trying to debug some packet drops that I?m experiencing and am turning to the list for help.   The recorded packet loss is ~50 ? 70% at times.   The packet loss is recorded in broctl?s netstats as well as in the notice.log file.
>
> Running netstats at startup ? I?m dropping more than I?m receiving from the very start.
Have you tried enabling the bro capture_loss script in your local.bro as 
a way to double check your loss numbers? It will give you per worker 
loss on 15 minute intervals in a separate log file.

In local.bro:
@load policy/misc/capture-loss

> insmod /lib/modules/2.6.32-431.11.2.el6.x86_64/kernel/net/pf_ring/pf_ring.ko enable_tx_capture=0 min_num_slots=32768 quick_mode=1
>
> insmod  /lib/modules/2.6.32-431.11.2.el6.x86_64/kernel/drivers/net/ixgbe/ixgbe.ko numa_cpu_affinity=0,0 MQ=0,1 RSS=0,0
>
> I checked /proc/sys/pci/devices to confirm that the interface is running on numa_node 0.  ?lscpu? shows that cpus 0-7 are one node 0, socket 0, and cpus 8-15 are on node 1, socket 0.  I figured having the 16 RSS queues on the same socket is probably better than having them bounce around.
>
>
> The node.cfg looks like this:
>
>
> [manager]
>
> type=manager
>
> host=10.99.99.15
>
>
> #
>
> [proxy-1]
>
> type=proxy
>
> host=10.99.99.15
>
>
> #
>
> [worker-1]
>
> type=worker
>
> host=10.99.99.15
>
> interface=eth3
>
> lb_method=pf_ring
>
> lb_procs=16
>
> pin_cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
>
>
> I have a license for ZC, and if I change the interface from eth3 to zc:eth3, it will spawn up 16 workers, but only one of them is receiving any traffic.  I?m assuming that it is looking at zc:eth3 at 0 only.   Netstats proves that out.   If I run pfcount ?I zc at eth3, it will show me that I?m receiving ~1gbp/s of traffic on the interface and not dropping anything.
As far as ZC usage, when using in ZC mode did you specify which adapters 
to enable at the end of your ixgbe insmod statement like this --> 
adapters_to_enable=<insert comma separated list of licensed mac 
addresses you want to use>? Also did you try setting RSS to match the 
number of workers instead of leaving it up to the NIC? Example RSS=16 
instead of 0 (comma separated per NIC if more than 1 NIC). Did you try 
pfcount ?I zc at eth3@0 (thru 15) etc to test each RSS queue? Did you put 
the necessary license files in /etc/pf_ring? Also, just to be certain, 
are you using the IXGBE drivers that come with PF_RING and have you 
compiled Bro against the PF_RING libpcap?

> Am I missing something obvious?  I saw many threads about disabling hyper threading, but that seems specific to intel processors ? I?m running AMD operterons with their own hyper transport stuff which doesn?t create virtual cpus.

I'm not sure I understand AMD architecture well enough to know how cores 
map to nodes, so I can't comment on your pinning configuration in terms 
of workers per core, but assuming each worker is pinned to a physical 
core and you truly have 16 physical cores on that socket, have you left 
any cores unpinned somewhere else (maybe a processor in another socket), 
for the system, bro manager, proxy etc to use? If not you could have 
other processes stomping on your workers. If any workers are sharing 
physical cores that could be problematic as well. Do you have htop or 
something similar installed where you can easily watch whether processes 
seem to be competing for the same physical core?

Have you tried running capstats (broctl capstats if using broctl) to see 
what sort of traffic bro thinks it is seeing across all workers when you 
are seeing loss? Depending on the clock speed and efficiency of each 
core you may be able to process anywhere from 100-300+Mbps per core, but 
if that 1Gbps of traffic was only representative of a single RSS queue 
on your 10G NIC you could be oversubscribed. If you have free cores on 
another socket it might be worth taking whatever small performance hit 
there is over the bus to have more workers running on those other cores. 
Also, I tend to leave the 1st couple logical cores open for the system 
as Linux at least seems to prefer them for system use. I do tend to find 
pinning workers to specific cores helps overall in the loss department 
vs letting workers bounce between cores, so I think you are on the right 
track.

~Gary

From Paul.Nash at tufts.edu  Thu Jan  7 16:22:08 2016
From: Paul.Nash at tufts.edu (Nash, Paul)
Date: Fri, 8 Jan 2016 00:22:08 +0000
Subject: [Bro] Bro Packet Loss / 10gb ixgbe / pf_ring
In-Reply-To: <CADC84wRQX-UmQvZAc3Rz43Esw5cQp5J4X7wnjOfZcASLk6YqeQ@mail.gmail.com>
References: <723D2963-E446-4251-ADF9-BEB7BE321202@tufts.edu>,
	<CADC84wRQX-UmQvZAc3Rz43Esw5cQp5J4X7wnjOfZcASLk6YqeQ@mail.gmail.com>
Message-ID: <2241134E2734344E8D8863FEEDB3F65F9312D9F1@SSVMEXDAG01MB01.tufts.ad.tufts.edu>

Thanks Mike -
  I'm using 16 workers because the ixgbe 10gb nic support hardware receive side scaling.  16 is the max number of queues that it supports.  While trying to monitor traffic this afternoon, I was seeing ~700 - 800mb/s based on pfcount stats.

If I disabled the hardware RSS I'd have to switch over to using pf_ring standard or DNA/ZC.  I have a license for ZC, but I've been unable to figure out how to get bro to monitor all of the zc:eth3 queues. The current Bro load-balancing documentation only covers pf_ring+DNA, but not the newer/supported zero-copy functionality.  I can't find the right "interface=" configuration for node.cfg.

"interface=zc:eth3" only monitors one of the queues.
interface="zc:eth3 at 0,zc:eth3 at 1,etc.." causes the workers to crash
interface="zc:eth3 at 0 -i zc:eth3 at 1 -i .." didn't work either.

The pf_ringZC documentation implies the use of zbalance_ipc to start up a set of queues and a cluster ID, with a call to zc:## where ## is the clusterID.  I also ran into issues with that.

For tonight, I'll disable the hardware RSS and switch over to running straight pf_ring with 24 workers.  I'll pin the first 8 so that they are on the same numa node as the NIC. Not sure what to do with the other 16 workers - does anyone have any insight if it is better to pin them to the same socket? I'm on AMD, which isn't as well documented as the intel world.


Thanks,
 -Paul

________________________________
From: reevesmk at gmail.com [reevesmk at gmail.com] on behalf of Mike Reeves [luke at geekempire.com]
Sent: Thursday, January 07, 2016 5:15 PM
To: Nash, Paul
Cc: bro at bro.org
Subject: Re: [Bro] Bro Packet Loss / 10gb ixgbe / pf_ring

Change your min_num_slots to be 65535. I would add an additional proxy as well as an additional 8 workers.



On Thu, Jan 7, 2016 at 4:37 PM, Nash, Paul <Paul.Nash at tufts.edu<mailto:Paul.Nash at tufts.edu>> wrote:

I?m trying to debug some packet drops that I?m experiencing and am turning to the list for help.   The recorded packet loss is ~50 ? 70% at times.   The packet loss is recorded in broctl?s netstats as well as in the notice.log file.

Running netstats at startup ? I?m dropping more than I?m receiving from the very start.


[BroControl] > netstats

 worker-1-1: 1452200459.635155 recvd=734100 dropped=1689718 link=2424079

worker-1-10: 1452200451.830143 recvd=718461 dropped=1414234 link=718461

worker-1-11: 1452200460.036766 recvd=481010 dropped=2019289 link=2500560

worker-1-12: 1452200460.239585 recvd=720895 dropped=1805574 link=2526730

worker-1-13: 1452200460.440611 recvd=753365 dropped=1800827 link=2554453

worker-1-14: 1452200460.647368 recvd=784145 dropped=1800831 link=2585237

worker-1-15: 1452200460.844842 recvd=750921 dropped=1868186 link=2619368

worker-1-16: 1452200461.049237 recvd=742718 dropped=1908528 link=2651507

?

System information:
- 64 AMD Opteron System
- 128gb of RAM
- Intel 10gb IXGBE interface (dual 10gb interfaces, eth3 is the sniffer)
- Licensed copy of PF_Ring ZC

I?m running Bro 2.4.1, PF_Ring 6.2.0 on Centos  / 2.6.32-411 kernel

I have the proxy, manager & 16 workers running on the same system.  16 CPUs are pinned (0-15)

Startup scripts to load the various kernel modules (from PF_RING 6.2.0 src)


insmod /lib/modules/2.6.32-431.11.2.el6.x86_64/kernel/net/pf_ring/pf_ring.ko enable_tx_capture=0 min_num_slots=32768 quick_mode=1

insmod  /lib/modules/2.6.32-431.11.2.el6.x86_64/kernel/drivers/net/ixgbe/ixgbe.ko numa_cpu_affinity=0,0 MQ=0,1 RSS=0,0

I checked /proc/sys/pci/devices to confirm that the interface is running on numa_node 0.  ?lscpu? shows that cpus 0-7 are one node 0, socket 0, and cpus 8-15 are on node 1, socket 0.  I figured having the 16 RSS queues on the same socket is probably better than having them bounce around.

I?ve disabled a bunch of the ixgbe offloading stuff:


ethtool -K eth3 rx off

ethtool -K eth3 tx off

ethtool -K eth3 sg off

ethtool -K eth3 tso off

ethtool -K eth3 gso off

ethtool -K eth3 gro off

ethtool -K eth3 lro off

ethtool -K eth3 rxvlan off

ethtool -K eth3 txvlan off

ethtool -K eth3 ntuple off

ethtool -K eth3 rxhash off

ethtool -K eth3 rx 32768

I?ve also tuned the stack, per recommendations from SANS:


net.ipv4.tcp_timestamps = 0

net.ipv4.tcp_sack = 0

net.ipv4.tcp_rmem = 10000000 10000000 10000000

net.ipv4.tcp_wmem = 10000000 10000000 10000000

net.ipv4.tcp_mem = 10000000 10000000 10000000

net.core.rmem_max = 134217728

net.core.wmem_max = 134217728

net.core.netdev_max_backlog = 250000


The node.cfg looks like this:


[manager]

type=manager

host=10.99.99.15


#

[proxy-1]

type=proxy

host=10.99.99.15


#

[worker-1]

type=worker

host=10.99.99.15

interface=eth3

lb_method=pf_ring

lb_procs=16

pin_cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15


I have a license for ZC, and if I change the interface from eth3 to zc:eth3, it will spawn up 16 workers, but only one of them is receiving any traffic.  I?m assuming that it is looking at zc:eth3 at 0 only.   Netstats proves that out.   If I run pfcount ?I zc at eth3, it will show me that I?m receiving ~1gbp/s of traffic on the interface and not dropping anything.

Am I missing something obvious?  I saw many threads about disabling hyper threading, but that seems specific to intel processors ? I?m running AMD operterons with their own hyper transport stuff which doesn?t create virtual cpus.

Thanks,
 -Paul

_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160108/66081c16/attachment-0001.html 

From Paul.Nash at tufts.edu  Thu Jan  7 17:03:22 2016
From: Paul.Nash at tufts.edu (Nash, Paul)
Date: Fri, 8 Jan 2016 01:03:22 +0000
Subject: [Bro] Bro Packet Loss / 10gb ixgbe / pf_ring
In-Reply-To: <568EFCF5.1060008@gmail.com>
References: <723D2963-E446-4251-ADF9-BEB7BE321202@tufts.edu>,
	<568EFCF5.1060008@gmail.com>
Message-ID: <2241134E2734344E8D8863FEEDB3F65F9312DB18@SSVMEXDAG01MB01.tufts.ad.tufts.edu>

Thanks Gary. 
  Sorry to top post, I'm stuck on OWA at the moment.  Thanks for your suggestions - here are some quick replies:

- capture_loss.bro - running it, every 15min it reports ~70% packet loss (or greater) across all of the workers
- 'adapters_to_enable' ixgbe.ko argument doesn't exist in the latest driver bundled w/pf_ring 6.2.0
- I've enabled the multi-queue stuff (MQ=1) on the 2nd interface (MQ=0,2) as well as enabled the 16 hw RSS queues = (RSS=1,16)
- I have a license in /etc/pf_ring
- bro is linked against the pf_ring enabled libpcap 
- I've confirmed that the .ko's I'm loading are the latest from pf_ring 6.2.0


Right now, pfcount says that eth3 is receiving 462Mbit/sec - I left it running for 5 minutes or so and there are zero dropped packets.  As soon as I start up bro, I'm already dropping 50%+ packets per worker. 

The only other thing I can think of could be packet duplication from some new taps that we deployed and potentially protocols that bro isn't parsing?

 -Paul

________________________________________
From: Gary Faulkner [gfaulkner.nsm at gmail.com]
Sent: Thursday, January 07, 2016 7:04 PM
To: Nash, Paul
Cc: bro at bro.org
Subject: Re: [Bro] Bro Packet Loss / 10gb ixgbe / pf_ring

Some thoughts inline...

On 1/7/16 3:37 PM, Nash, Paul wrote:
> I?m trying to debug some packet drops that I?m experiencing and am turning to the list for help.   The recorded packet loss is ~50 ? 70% at times.   The packet loss is recorded in broctl?s netstats as well as in the notice.log file.
>
> Running netstats at startup ? I?m dropping more than I?m receiving from the very start.

Have you tried enabling the bro capture_loss script in your local.bro as
a way to double check your loss numbers? It will give you per worker
loss on 15 minute intervals in a separate log file.

In local.bro:
@load policy/misc/capture-loss

> insmod /lib/modules/2.6.32-431.11.2.el6.x86_64/kernel/net/pf_ring/pf_ring.ko enable_tx_capture=0 min_num_slots=32768 quick_mode=1
>
> insmod  /lib/modules/2.6.32-431.11.2.el6.x86_64/kernel/drivers/net/ixgbe/ixgbe.ko numa_cpu_affinity=0,0 MQ=0,1 RSS=0,0
>
> I checked /proc/sys/pci/devices to confirm that the interface is running on numa_node 0.  ?lscpu? shows that cpus 0-7 are one node 0, socket 0, and cpus 8-15 are on node 1, socket 0.  I figured having the 16 RSS queues on the same socket is probably better than having them bounce around.
>
>
> The node.cfg looks like this:
>
>
> [manager]
>
> type=manager
>
> host=10.99.99.15
>
>
> #
>
> [proxy-1]
>
> type=proxy
>
> host=10.99.99.15
>
>
> #
>
> [worker-1]
>
> type=worker
>
> host=10.99.99.15
>
> interface=eth3
>
> lb_method=pf_ring
>
> lb_procs=16
>
> pin_cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
>
>
> I have a license for ZC, and if I change the interface from eth3 to zc:eth3, it will spawn up 16 workers, but only one of them is receiving any traffic.  I?m assuming that it is looking at zc:eth3 at 0 only.   Netstats proves that out.   If I run pfcount ?I zc at eth3, it will show me that I?m receiving ~1gbp/s of traffic on the interface and not dropping anything.
As far as ZC usage, when using in ZC mode did you specify which adapters
to enable at the end of your ixgbe insmod statement like this -->
adapters_to_enable=<insert comma separated list of licensed mac
addresses you want to use>? Also did you try setting RSS to match the
number of workers instead of leaving it up to the NIC? Example RSS=16
instead of 0 (comma separated per NIC if more than 1 NIC). Did you try
pfcount ?I zc at eth3@0 (thru 15) etc to test each RSS queue? Did you put
the necessary license files in /etc/pf_ring? Also, just to be certain,
are you using the IXGBE drivers that come with PF_RING and have you
compiled Bro against the PF_RING libpcap?

> Am I missing something obvious?  I saw many threads about disabling hyper threading, but that seems specific to intel processors ? I?m running AMD operterons with their own hyper transport stuff which doesn?t create virtual cpus.

I'm not sure I understand AMD architecture well enough to know how cores
map to nodes, so I can't comment on your pinning configuration in terms
of workers per core, but assuming each worker is pinned to a physical
core and you truly have 16 physical cores on that socket, have you left
any cores unpinned somewhere else (maybe a processor in another socket),
for the system, bro manager, proxy etc to use? If not you could have
other processes stomping on your workers. If any workers are sharing
physical cores that could be problematic as well. Do you have htop or
something similar installed where you can easily watch whether processes
seem to be competing for the same physical core?

Have you tried running capstats (broctl capstats if using broctl) to see
what sort of traffic bro thinks it is seeing across all workers when you
are seeing loss? Depending on the clock speed and efficiency of each
core you may be able to process anywhere from 100-300+Mbps per core, but
if that 1Gbps of traffic was only representative of a single RSS queue
on your 10G NIC you could be oversubscribed. If you have free cores on
another socket it might be worth taking whatever small performance hit
there is over the bus to have more workers running on those other cores.
Also, I tend to leave the 1st couple logical cores open for the system
as Linux at least seems to prefer them for system use. I do tend to find
pinning workers to specific cores helps overall in the loss department
vs letting workers bounce between cores, so I think you are on the right
track.

~Gary


From gfaulkner.nsm at gmail.com  Thu Jan  7 18:14:38 2016
From: gfaulkner.nsm at gmail.com (Gary Faulkner)
Date: Thu, 7 Jan 2016 20:14:38 -0600
Subject: [Bro] Bro Packet Loss / 10gb ixgbe / pf_ring
In-Reply-To: <2241134E2734344E8D8863FEEDB3F65F9312DB18@SSVMEXDAG01MB01.tufts.ad.tufts.edu>
References: <723D2963-E446-4251-ADF9-BEB7BE321202@tufts.edu>
	<568EFCF5.1060008@gmail.com>
	<2241134E2734344E8D8863FEEDB3F65F9312DB18@SSVMEXDAG01MB01.tufts.ad.tufts.edu>
Message-ID: <568F1B8E.2090807@gmail.com>

Ah, I'm still on 6.0.3 with DNA/Libzero, so didn't realize that
adapters_to_enable was changed/removed. I'm about to start testing 6.2
with ZC soon, but probably using zbalance_ipc instead of relying on RSS.
If you are running into RSS limits, but have more cores on another
socket using zbalance_ipc should allow you to do things like aggregate
NICs and do 2,4,5-tuple hashing to as many worker queues/ring buffers as
you can handle, as well as duplicate traffic to a secondary app such as
capstats, since it takes over acting as the on host load-balancer and
doesn't rely on RSS. In that case you set RSS=1 for each interface going
into zbalance_ipc.  Also looking at the 2.4.1 broctl related code in
<path-to-bro>/lib/broctl/plugins/lb_pf_ring.py seems to imply that
possibly the ZC interface naming is only supported when using
zbalance_ipc, but perhaps I'm wrong, relevant snippet is below:

            if nn.interface.startswith("zc"):
                # For the case where a user is running zbalance_ipc
                nn.interface = "%s@%d" % (nn.interface, app_instance)

For DNA there are specific entries for DNA using RSS and DNA using
pfdnacluster_master in the code. So possibly try ZC with zbalance_ipc
with an interface name in node.cfg of zc:whatever clusterid you assigned.

There is a thread discussing using zbalance_ipc, including syntax in the
bro archives that might be more helpful than myself starting with this
post
<http://mailman.icsi.berkeley.edu/pipermail/bro/2015-February/008154.html>.
It might be worth reading through that whole thread as it involves
troubleshooting.   

Bro probably isn't going to like duplicate packets such as if you are
tapping both the inside and outside interfaces of a firewall. Have you
checked weird.log to see if it is complaining about that? Are the taps
you refer to plugged directly into your Bro sensor, or coming off some
sort of tap aggregation load-balancer, or are you really using span port
(the latter can sometimes see performance hits due to sampling or
router/switch cpu load)? If using an optical tap is there any chance the
fiber plant isn't installed such that you see both send and receive? Do
you do any sort of packet slicing that might throw off loss numbers?
Looking at weird.log might give you an indication if you are seeing one
sided conversations as well or have other upstream network issues.

Another thought is that if you have jumbo frames enabled on your network
you may want to check MTU sizes. I currently have mine set to 9216 to
match the max packet size on our upstream router. If you are collecting
flows somewhere it might also be worth looking to see if you have any
sources of large flows that might be impacting overall sensor performance.

~Gary

On 1/7/2016 7:03 PM, Nash, Paul wrote:
> Thanks Gary. 
>   Sorry to top post, I'm stuck on OWA at the moment.  Thanks for your suggestions - here are some quick replies:
>
> - capture_loss.bro - running it, every 15min it reports ~70% packet loss (or greater) across all of the workers
> - 'adapters_to_enable' ixgbe.ko argument doesn't exist in the latest driver bundled w/pf_ring 6.2.0
> - I've enabled the multi-queue stuff (MQ=1) on the 2nd interface (MQ=0,2) as well as enabled the 16 hw RSS queues = (RSS=1,16)
> - I have a license in /etc/pf_ring
> - bro is linked against the pf_ring enabled libpcap 
> - I've confirmed that the .ko's I'm loading are the latest from pf_ring 6.2.0
>
>
> Right now, pfcount says that eth3 is receiving 462Mbit/sec - I left it running for 5 minutes or so and there are zero dropped packets.  As soon as I start up bro, I'm already dropping 50%+ packets per worker. 
>
> The only other thing I can think of could be packet duplication from some new taps that we deployed and potentially protocols that bro isn't parsing?
>
>  -Paul
>
> ________________________________________
> From: Gary Faulkner [gfaulkner.nsm at gmail.com]
> Sent: Thursday, January 07, 2016 7:04 PM
> To: Nash, Paul
> Cc: bro at bro.org
> Subject: Re: [Bro] Bro Packet Loss / 10gb ixgbe / pf_ring
>
> Some thoughts inline...
>
> On 1/7/16 3:37 PM, Nash, Paul wrote:
>> I?m trying to debug some packet drops that I?m experiencing and am turning to the list for help.   The recorded packet loss is ~50 ? 70% at times.   The packet loss is recorded in broctl?s netstats as well as in the notice.log file.
>>
>> Running netstats at startup ? I?m dropping more than I?m receiving from the very start.
> Have you tried enabling the bro capture_loss script in your local.bro as
> a way to double check your loss numbers? It will give you per worker
> loss on 15 minute intervals in a separate log file.
>
> In local.bro:
> @load policy/misc/capture-loss
>
>> insmod /lib/modules/2.6.32-431.11.2.el6.x86_64/kernel/net/pf_ring/pf_ring.ko enable_tx_capture=0 min_num_slots=32768 quick_mode=1
>>
>> insmod  /lib/modules/2.6.32-431.11.2.el6.x86_64/kernel/drivers/net/ixgbe/ixgbe.ko numa_cpu_affinity=0,0 MQ=0,1 RSS=0,0
>>
>> I checked /proc/sys/pci/devices to confirm that the interface is running on numa_node 0.  ?lscpu? shows that cpus 0-7 are one node 0, socket 0, and cpus 8-15 are on node 1, socket 0.  I figured having the 16 RSS queues on the same socket is probably better than having them bounce around.
>>
>>
>> The node.cfg looks like this:
>>
>>
>> [manager]
>>
>> type=manager
>>
>> host=10.99.99.15
>>
>>
>> #
>>
>> [proxy-1]
>>
>> type=proxy
>>
>> host=10.99.99.15
>>
>>
>> #
>>
>> [worker-1]
>>
>> type=worker
>>
>> host=10.99.99.15
>>
>> interface=eth3
>>
>> lb_method=pf_ring
>>
>> lb_procs=16
>>
>> pin_cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
>>
>>
>> I have a license for ZC, and if I change the interface from eth3 to zc:eth3, it will spawn up 16 workers, but only one of them is receiving any traffic.  I?m assuming that it is looking at zc:eth3 at 0 only.   Netstats proves that out.   If I run pfcount ?I zc at eth3, it will show me that I?m receiving ~1gbp/s of traffic on the interface and not dropping anything.
> As far as ZC usage, when using in ZC mode did you specify which adapters
> to enable at the end of your ixgbe insmod statement like this -->
> adapters_to_enable=<insert comma separated list of licensed mac
> addresses you want to use>? Also did you try setting RSS to match the
> number of workers instead of leaving it up to the NIC? Example RSS=16
> instead of 0 (comma separated per NIC if more than 1 NIC). Did you try
> pfcount ?I zc at eth3@0 (thru 15) etc to test each RSS queue? Did you put
> the necessary license files in /etc/pf_ring? Also, just to be certain,
> are you using the IXGBE drivers that come with PF_RING and have you
> compiled Bro against the PF_RING libpcap?
>
>> Am I missing something obvious?  I saw many threads about disabling hyper threading, but that seems specific to intel processors ? I?m running AMD operterons with their own hyper transport stuff which doesn?t create virtual cpus.
> I'm not sure I understand AMD architecture well enough to know how cores
> map to nodes, so I can't comment on your pinning configuration in terms
> of workers per core, but assuming each worker is pinned to a physical
> core and you truly have 16 physical cores on that socket, have you left
> any cores unpinned somewhere else (maybe a processor in another socket),
> for the system, bro manager, proxy etc to use? If not you could have
> other processes stomping on your workers. If any workers are sharing
> physical cores that could be problematic as well. Do you have htop or
> something similar installed where you can easily watch whether processes
> seem to be competing for the same physical core?
>
> Have you tried running capstats (broctl capstats if using broctl) to see
> what sort of traffic bro thinks it is seeing across all workers when you
> are seeing loss? Depending on the clock speed and efficiency of each
> core you may be able to process anywhere from 100-300+Mbps per core, but
> if that 1Gbps of traffic was only representative of a single RSS queue
> on your 10G NIC you could be oversubscribed. If you have free cores on
> another socket it might be worth taking whatever small performance hit
> there is over the bus to have more workers running on those other cores.
> Also, I tend to leave the 1st couple logical cores open for the system
> as Linux at least seems to prefer them for system use. I do tend to find
> pinning workers to specific cores helps overall in the loss department
> vs letting workers bounce between cores, so I think you are on the right
> track.
>
> ~Gary

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160107/d1d2d614/attachment.html 

From giedrius.ramas at gmail.com  Fri Jan  8 05:19:11 2016
From: giedrius.ramas at gmail.com (Giedrius Ramas)
Date: Fri, 8 Jan 2016 15:19:11 +0200
Subject: [Bro] increase log rotation interval
In-Reply-To: <c08a47c7edb80ac6ca82c4af470aad05@localhost>
References: <CAHr6V5rjto-6RKQ884BN-1qzXw=_68PrmuwWj_5RYqUDCWO66w@mail.gmail.com>
	<c08a47c7edb80ac6ca82c4af470aad05@localhost>
Message-ID: <CAHr6V5pcZBOxgc4mgEBnccRfDhfsS4qDRfMH7w=P53U8eyFrKg@mail.gmail.com>

Thanks for reply,

However I wonder if there are any issues I could run into once Log rotation
interval is changed to higher value.

On Thu, Jan 7, 2016 at 5:34 PM, James Lay <jlay at slave-tothe-box.net> wrote:

> Set:
>
> LogRotationInterval = 86400
>
> in your broctl.cfg
>
> James
>
>
>
> On 2016-01-07 08:10, Giedrius Ramas wrote:
>
> Hi all ,
> I was just wondering if any issues could cause if I increase
> LogRotationInterval from 3600 to higher . Could I run into any problems If
> so what kind of ? I would like to have day rotation instead of one hour.
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160108/bc1714c5/attachment-0001.html 

From jlay at slave-tothe-box.net  Fri Jan  8 07:13:05 2016
From: jlay at slave-tothe-box.net (James Lay)
Date: Fri, 08 Jan 2016 08:13:05 -0700
Subject: [Bro] increase log rotation interval
In-Reply-To: <CAHr6V5pcZBOxgc4mgEBnccRfDhfsS4qDRfMH7w=P53U8eyFrKg@mail.gmail.com>
References: <CAHr6V5rjto-6RKQ884BN-1qzXw=_68PrmuwWj_5RYqUDCWO66w@mail.gmail.com>
	<c08a47c7edb80ac6ca82c4af470aad05@localhost>
	<CAHr6V5pcZBOxgc4mgEBnccRfDhfsS4qDRfMH7w=P53U8eyFrKg@mail.gmail.com>
Message-ID: <c6f7920505fca2d47a94ed9ef3390e15@localhost>

 

I've not had a single issue...it's wonderful and allows me to search by
an entire day. 

James 

On 2016-01-08 06:19, Giedrius Ramas wrote: 

> Thanks for reply, 
> 
> However I wonder if there are any issues I could run into once Log rotation interval is changed to higher value.  
> 
> On Thu, Jan 7, 2016 at 5:34 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> 
> Set: 
> 
> LogRotationInterval = 86400 
> 
> in your broctl.cfg 
> 
> James 
> 
> On 2016-01-07 08:10, Giedrius Ramas wrote: 
> Hi all ,  
> I was just wondering if any issues could cause if I increase LogRotationInterval from 3600 to higher . Could I run into any problems If so what kind of ? I would like to have day rotation instead of one hour.  
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160108/f21e0877/attachment.html 

From seth at icir.org  Fri Jan  8 07:28:42 2016
From: seth at icir.org (Seth Hall)
Date: Fri, 8 Jan 2016 10:28:42 -0500
Subject: [Bro] Bro Packet Loss / 10gb ixgbe / pf_ring
In-Reply-To: <723D2963-E446-4251-ADF9-BEB7BE321202@tufts.edu>
References: <723D2963-E446-4251-ADF9-BEB7BE321202@tufts.edu>
Message-ID: <41AA3B68-CF28-40D7-A38D-C7D1B8C1591D@icir.org>


> On Jan 7, 2016, at 4:37 PM, Nash, Paul <Paul.Nash at tufts.edu> wrote:
> 
> I have a license for ZC, and if I change the interface from eth3 to zc:eth3, it will spawn up 16 workers, but only one of them is receiving any traffic.  I?m assuming that it is looking at zc:eth3 at 0 only.   Netstats proves that out.   If I run pfcount ?I zc at eth3, it will show me that I?m receiving ~1gbp/s of traffic on the interface and not dropping anything.  

If you make the line ?interface=zc:eth3?, the pf_ring plugin for broctl should automatically change the interface that each Bro process is sniffing to the correct name as you?ve indicated (zc:eth3@[0-15]).  Configure it that way and the check with ps what interface is being sniffed (you will see it as part of the command line that broctl is executing).

I added support for ZC to that plugin for the 2.4 release and I got it working and validated.  There are some issues with this path though because if a Bro process crashes or is shut down you will need to restart zbalance_ipc as well in order for that output ring to be reconnected.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From seth at icir.org  Fri Jan  8 07:30:11 2016
From: seth at icir.org (Seth Hall)
Date: Fri, 8 Jan 2016 10:30:11 -0500
Subject: [Bro] increase log rotation interval
In-Reply-To: <CAHr6V5pcZBOxgc4mgEBnccRfDhfsS4qDRfMH7w=P53U8eyFrKg@mail.gmail.com>
References: <CAHr6V5rjto-6RKQ884BN-1qzXw=_68PrmuwWj_5RYqUDCWO66w@mail.gmail.com>
	<c08a47c7edb80ac6ca82c4af470aad05@localhost>
	<CAHr6V5pcZBOxgc4mgEBnccRfDhfsS4qDRfMH7w=P53U8eyFrKg@mail.gmail.com>
Message-ID: <6FCACDC1-19D7-443B-B668-C115CCB91B84@icir.org>


> On Jan 8, 2016, at 8:19 AM, Giedrius Ramas <giedrius.ramas at gmail.com> wrote:
> 
> However I wonder if there are any issues I could run into once Log rotation interval is changed to higher value. 

You could run into issue when the file rotation happens because of disk contention and increased cpu utilization (a days worth of logs will obviously take longer to compress that an hour?s worth).

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From Paul.Nash at tufts.edu  Fri Jan  8 08:24:22 2016
From: Paul.Nash at tufts.edu (Nash, Paul)
Date: Fri, 8 Jan 2016 16:24:22 +0000
Subject: [Bro] Bro Packet Loss / 10gb ixgbe / pf_ring
In-Reply-To: <41AA3B68-CF28-40D7-A38D-C7D1B8C1591D@icir.org>
References: <723D2963-E446-4251-ADF9-BEB7BE321202@tufts.edu>
	<41AA3B68-CF28-40D7-A38D-C7D1B8C1591D@icir.org>
Message-ID: <FF4F5A27-8001-4393-87B7-E7C50F338FBF@tufts.edu>


Thanks Seth - I have my node.cfg to point to zc:eth3

interface=zc:eth3

Upon running broctl cleanup/deploy, I?m seeing that bro is called with only "-i zc:eth3?.  I tried calling it with ?zc:2? (cluster ID) and zbalance_ipc handed out 8k packets before the bro workers crashed. 

 -Paul




On 1/8/16, 10:28 AM, "Seth Hall" <seth at icir.org> wrote:

>
>> On Jan 7, 2016, at 4:37 PM, Nash, Paul <Paul.Nash at tufts.edu> wrote:
>> 
>> I have a license for ZC, and if I change the interface from eth3 to zc:eth3, it will spawn up 16 workers, but only one of them is receiving any traffic.  I?m assuming that it is looking at zc:eth3 at 0 only.   Netstats proves that out.   If I run pfcount ?I zc at eth3, it will show me that I?m receiving ~1gbp/s of traffic on the interface and not dropping anything.  
>?
>If you make the line ?interface=zc:eth3?, the pf_ring plugin for broctl should automatically change the interface that each Bro process is sniffing to the correct name as you?ve indicated (zc:eth3@[0-15]).  Configure it that way and the check with ps what interface is being sniffed (you will see it as part of the command line that broctl is executing).
>
>I added support for ZC to that plugin for the 2.4 release and I got it working and validated.  There are some issues with this path though because if a Bro process crashes or is shut down you will need to restart zbalance_ipc as well in order for that output ring to be reconnected.
>
>  .Seth
>
>--
>Seth Hall
>International Computer Science Institute
>(Bro) because everyone has a network
>http://www.bro.org/
>


From seth at icir.org  Fri Jan  8 09:22:56 2016
From: seth at icir.org (Seth Hall)
Date: Fri, 8 Jan 2016 12:22:56 -0500
Subject: [Bro] Bro Packet Loss / 10gb ixgbe / pf_ring
In-Reply-To: <FF4F5A27-8001-4393-87B7-E7C50F338FBF@tufts.edu>
References: <723D2963-E446-4251-ADF9-BEB7BE321202@tufts.edu>
	<41AA3B68-CF28-40D7-A38D-C7D1B8C1591D@icir.org>
	<FF4F5A27-8001-4393-87B7-E7C50F338FBF@tufts.edu>
Message-ID: <FAA7F03F-88FE-48F9-A1B9-0C0884D31494@icir.org>


> On Jan 8, 2016, at 11:24 AM, Nash, Paul <Paul.Nash at tufts.edu> wrote:
> 
> 
> Thanks Seth - I have my node.cfg to point to zc:eth3
> 
> interface=zc:eth3
> 
> Upon running broctl cleanup/deploy, I?m seeing that bro is called with only "-i zc:eth3?.  I tried calling it with ?zc:2? (cluster ID) and zbalance_ipc handed out 8k packets before the bro workers crashed. 

Oh!  I forgot, that?s the right way to use it.  I forgot about the cluster ID thing.  Forget what I said before. :)

Could you send me a log from Bro crashing (hopefully you got a stack trace in your crash dump from broctl)?  That could be some unrelated problem.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From gfaulkner.nsm at gmail.com  Fri Jan  8 09:25:40 2016
From: gfaulkner.nsm at gmail.com (Gary Faulkner)
Date: Fri, 8 Jan 2016 11:25:40 -0600
Subject: [Bro] Bro Packet Loss / 10gb ixgbe / pf_ring
In-Reply-To: <FF4F5A27-8001-4393-87B7-E7C50F338FBF@tufts.edu>
References: <723D2963-E446-4251-ADF9-BEB7BE321202@tufts.edu>
	<41AA3B68-CF28-40D7-A38D-C7D1B8C1591D@icir.org>
	<FF4F5A27-8001-4393-87B7-E7C50F338FBF@tufts.edu>
Message-ID: <568FF114.7080208@gmail.com>

That last bit about only processing 8k packets sounds almost like 
PF_RING might be stuck in demo mode for ZC (runs for 10 secs or so) and 
then stops. This could mean that PF_RING isn't seeing your license file. 
Any chance it can't read the files in /etc/pf_ring/ or that there is a 
PATH problem somewhere?

On 1/8/16 10:24 AM, Nash, Paul wrote:
> Thanks Seth - I have my node.cfg to point to zc:eth3
>
> interface=zc:eth3
>
> Upon running broctl cleanup/deploy, I?m seeing that bro is called with only "-i zc:eth3?.  I tried calling it with ?zc:2? (cluster ID) and zbalance_ipc handed out 8k packets before the bro workers crashed.
>
>   -Paul
>
>
>
>
> On 1/8/16, 10:28 AM, "Seth Hall" <seth at icir.org> wrote:
>
>>> On Jan 7, 2016, at 4:37 PM, Nash, Paul <Paul.Nash at tufts.edu> wrote:
>>>
>>> I have a license for ZC, and if I change the interface from eth3 to zc:eth3, it will spawn up 16 workers, but only one of them is receiving any traffic.  I?m assuming that it is looking at zc:eth3 at 0 only.   Netstats proves that out.   If I run pfcount ?I zc at eth3, it will show me that I?m receiving ~1gbp/s of traffic on the interface and not dropping anything.
>> ?
>> If you make the line ?interface=zc:eth3?, the pf_ring plugin for broctl should automatically change the interface that each Bro process is sniffing to the correct name as you?ve indicated (zc:eth3@[0-15]).  Configure it that way and the check with ps what interface is being sniffed (you will see it as part of the command line that broctl is executing).
>>
>> I added support for ZC to that plugin for the 2.4 release and I got it working and validated.  There are some issues with this path though because if a Bro process crashes or is shut down you will need to restart zbalance_ipc as well in order for that output ring to be reconnected.
>>
>>   .Seth
>>
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro.org/
>>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From dnthayer at illinois.edu  Fri Jan  8 09:26:34 2016
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Fri, 8 Jan 2016 11:26:34 -0600
Subject: [Bro] Bro Packet Loss / 10gb ixgbe / pf_ring
In-Reply-To: <FF4F5A27-8001-4393-87B7-E7C50F338FBF@tufts.edu>
References: <723D2963-E446-4251-ADF9-BEB7BE321202@tufts.edu>	<41AA3B68-CF28-40D7-A38D-C7D1B8C1591D@icir.org>
	<FF4F5A27-8001-4393-87B7-E7C50F338FBF@tufts.edu>
Message-ID: <568FF14A.5030503@illinois.edu>

Did you check that the broctl config option "pfringclusterid"
has a non-zero value?

broctl config | grep pfring

You can also check that broctl is using the correct interface
name by looking at the "interface=" field in the output of
the following command:

broctl nodes


On 01/08/2016 10:24 AM, Nash, Paul wrote:
>
> Thanks Seth - I have my node.cfg to point to zc:eth3
>
> interface=zc:eth3
>
> Upon running broctl cleanup/deploy, I?m seeing that bro is called with only "-i zc:eth3?.  I tried calling it with ?zc:2? (cluster ID) and zbalance_ipc handed out 8k packets before the bro workers crashed.
>
>   -Paul
>
>
>
>
> On 1/8/16, 10:28 AM, "Seth Hall" <seth at icir.org> wrote:
>
>>
>>> On Jan 7, 2016, at 4:37 PM, Nash, Paul <Paul.Nash at tufts.edu> wrote:
>>>
>>> I have a license for ZC, and if I change the interface from eth3 to zc:eth3, it will spawn up 16 workers, but only one of them is receiving any traffic.  I?m assuming that it is looking at zc:eth3 at 0 only.   Netstats proves that out.   If I run pfcount ?I zc at eth3, it will show me that I?m receiving ~1gbp/s of traffic on the interface and not dropping anything.
>> ?
>> If you make the line ?interface=zc:eth3?, the pf_ring plugin for broctl should automatically change the interface that each Bro process is sniffing to the correct name as you?ve indicated (zc:eth3@[0-15]).  Configure it that way and the check with ps what interface is being sniffed (you will see it as part of the command line that broctl is executing).
>>
>> I added support for ZC to that plugin for the 2.4 release and I got it working and validated.  There are some issues with this path though because if a Bro process crashes or is shut down you will need to restart zbalance_ipc as well in order for that output ring to be reconnected.
>>
>>   .Seth
>>
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro.org/
>>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>

From jlay at slave-tothe-box.net  Tue Jan 12 11:32:26 2016
From: jlay at slave-tothe-box.net (James Lay)
Date: Tue, 12 Jan 2016 12:32:26 -0700
Subject: [Bro] Bro's software log
Message-ID: <5e969c4eeddc70ff40aa4f8479232235@localhost>

 
I LOVE the software log.  Legit.  It's awesome.  I'm trying to create a
report of sorts, with sed and awk, and for the life of me I'm having a
tough time.  Here's what I got so far:

zcat software.log.gz | bro-cut -d | sed -e 's/<tab character here, ie
ctrl-v, tab>/-/g' -e 's/\-\-\-[A-Z]\{3,5\}::/ /' -e 's/^.*0000-//'

This get me kinda close, but not close enough...here's the raw entry: 

2016-01-01T14:57:02+0000        x.x.x.x  -       HTTP::BROWSER  
Windows-Update-Agent    7       9       9600    18145   Client 
Windows-Update-Agent/7.9.9600.18145 Client-Protocol/1.21 

What' I'm really hoping for is this: 
x.x.x.x Windows-Update-Agent/7.9.9600.18145 Client-Protocol/1.21 

Just the IP address, and the last bit...the entire unparsed_version
field.  Anyone got a clever script to do something like this?  Thank
you. 

James 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160112/134a433c/attachment.html 

From dopheide at gmail.com  Tue Jan 12 11:55:45 2016
From: dopheide at gmail.com (Mike Dopheide)
Date: Tue, 12 Jan 2016 13:55:45 -0600
Subject: [Bro] Bro's software log
In-Reply-To: <5e969c4eeddc70ff40aa4f8479232235@localhost>
References: <5e969c4eeddc70ff40aa4f8479232235@localhost>
Message-ID: <CAPy2kFbqWaSpZrsr6_9JyAJNYzZJzzSFaewvMP9ApyUCJg4DHw@mail.gmail.com>

Unless I'm missing what you're trying to do, bro-cut already can do this
for you:

cat software.log |bro-cut host unparsed_version

-Dop

On Tue, Jan 12, 2016 at 1:32 PM, James Lay <jlay at slave-tothe-box.net> wrote:

> I LOVE the software log.  Legit.  It's awesome.  I'm trying to create a
> report of sorts, with sed and awk, and for the life of me I'm having a
> tough time.  Here's what I got so far:
>
> zcat software.log.gz | bro-cut -d | sed -e 's/<tab character here, ie
> ctrl-v, tab>/-/g' -e 's/\-\-\-[A-Z]\{3,5\}::/ /' -e 's/^.*0000-//'
>
> This get me kinda close, but not close enough...here's the raw entry:
>
> 2016-01-01T14:57:02+0000        x.x.x.x  -       HTTP::BROWSER
> Windows-Update-Agent    7       9       9600    18145   Client
> Windows-Update-Agent/7.9.9600.18145 Client-Protocol/1.21
>
> What' I'm really hoping for is this:
> x.x.x.x Windows-Update-Agent/7.9.9600.18145 Client-Protocol/1.21
>
> Just the IP address, and the last bit...the entire unparsed_version
> field.  Anyone got a clever script to do something like this?  Thank you.
>
> James
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160112/a4ea4e42/attachment.html 

From jazoff at illinois.edu  Tue Jan 12 11:58:21 2016
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Tue, 12 Jan 2016 19:58:21 +0000
Subject: [Bro] Bro's software log
In-Reply-To: <5e969c4eeddc70ff40aa4f8479232235@localhost>
References: <5e969c4eeddc70ff40aa4f8479232235@localhost>
Message-ID: <B8C3115C-B188-4094-8657-3DC32B8E665A@illinois.edu>

You're going to laugh... That's what bro-cut is for :-)


    # zcat software.log.gz | bro-cut host unparsed_version


Regular cut kind of works too, but bro-cut is faster and easier to use:

    # zcat software.log.gz | egrep -v "^#" | cut -f 2,11

-- 
- Justin Azoff

> On Jan 12, 2016, at 2:32 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> 
> I LOVE the software log.  Legit.  It's awesome.  I'm trying to create a report of sorts, with sed and awk, and for the life of me I'm having a tough time.  Here's what I got so far:
> 
> zcat software.log.gz | bro-cut -d | sed -e 's/<tab character here, ie ctrl-v, tab>/-/g' -e 's/\-\-\-[A-Z]\{3,5\}::/ /' -e 's/^.*0000-//'
> 
> This get me kinda close, but not close enough...here's the raw entry:
>  
> 2016-01-01T14:57:02+0000        x.x.x.x  -       HTTP::BROWSER   Windows-Update-Agent    7       9       9600    18145   Client  Windows-Update-Agent/7.9.9600.18145 Client-Protocol/1.21
> 
> What' I'm really hoping for is this:
> x.x.x.x Windows-Update-Agent/7.9.9600.18145 Client-Protocol/1.21
>  
> Just the IP address, and the last bit...the entire unparsed_version field.  Anyone got a clever script to do something like this?  Thank you.
>  
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From paul.halliday at gmail.com  Tue Jan 12 12:01:52 2016
From: paul.halliday at gmail.com (Paul Halliday)
Date: Tue, 12 Jan 2016 16:01:52 -0400
Subject: [Bro] Bro's software log
In-Reply-To: <5e969c4eeddc70ff40aa4f8479232235@localhost>
References: <5e969c4eeddc70ff40aa4f8479232235@localhost>
Message-ID: <CAJfn-RHhR-Fz+uKUOqGgzw-HZ6GkBYONyQVm+P81ViTC_5mZZw@mail.gmail.com>

~% awk -F "\t" '{print $2,$11}' software.log

On Tue, Jan 12, 2016 at 3:32 PM, James Lay <jlay at slave-tothe-box.net> wrote:

> I LOVE the software log.  Legit.  It's awesome.  I'm trying to create a
> report of sorts, with sed and awk, and for the life of me I'm having a
> tough time.  Here's what I got so far:
>
> zcat software.log.gz | bro-cut -d | sed -e 's/<tab character here, ie
> ctrl-v, tab>/-/g' -e 's/\-\-\-[A-Z]\{3,5\}::/ /' -e 's/^.*0000-//'
>
> This get me kinda close, but not close enough...here's the raw entry:
>
> 2016-01-01T14:57:02+0000        x.x.x.x  -       HTTP::BROWSER
> Windows-Update-Agent    7       9       9600    18145   Client
> Windows-Update-Agent/7.9.9600.18145 Client-Protocol/1.21
>
> What' I'm really hoping for is this:
> x.x.x.x Windows-Update-Agent/7.9.9600.18145 Client-Protocol/1.21
>
> Just the IP address, and the last bit...the entire unparsed_version
> field.  Anyone got a clever script to do something like this?  Thank you.
>
> James
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 
Paul Halliday
http://www.pintumbler.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160112/b011b5d9/attachment.html 

From dnthayer at illinois.edu  Tue Jan 12 12:03:51 2016
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Tue, 12 Jan 2016 14:03:51 -0600
Subject: [Bro] Bro's software log
In-Reply-To: <5e969c4eeddc70ff40aa4f8479232235@localhost>
References: <5e969c4eeddc70ff40aa4f8479232235@localhost>
Message-ID: <56955C27.7090508@illinois.edu>

Have you tried using bro-cut to specify the field names that you want?

zcat software.log.gz | bro-cut host name unparsed_version



On 01/12/2016 01:32 PM, James Lay wrote:
> I LOVE the software log.  Legit.  It's awesome.  I'm trying to create a
> report of sorts, with sed and awk, and for the life of me I'm having a
> tough time.  Here's what I got so far:
>
> zcat software.log.gz | bro-cut -d | sed -e 's/<tab character here, ie
> ctrl-v, tab>/-/g' -e 's/\-\-\-[A-Z]\{3,5\}::/ /' -e 's/^.*0000-//'
>
> This get me kinda close, but not close enough...here's the raw entry:
> 2016-01-01T14:57:02+0000        x.x.x.x  -       HTTP::BROWSER
> Windows-Update-Agent    7       9       9600    18145   Client
> Windows-Update-Agent/7.9.9600.18145 Client-Protocol/1.21
>
> What' I'm really hoping for is this:
> x.x.x.x Windows-Update-Agent/7.9.9600.18145 Client-Protocol/1.21
> Just the IP address, and the last bit...the entire unparsed_version
> field.  Anyone got a clever script to do something like this?  Thank you.
> James
>

From johanna at icir.org  Tue Jan 12 12:04:34 2016
From: johanna at icir.org (Johanna Amann)
Date: Tue, 12 Jan 2016 12:04:34 -0800
Subject: [Bro] Bro's software log
In-Reply-To: <5e969c4eeddc70ff40aa4f8479232235@localhost>
References: <5e969c4eeddc70ff40aa4f8479232235@localhost>
Message-ID: <20160112200428.GA39992@wifi168.sys.ICSI.Berkeley.EDU>

On Tue, Jan 12, 2016 at 12:32:26PM -0700, James Lay wrote:
> Just the IP address, and the last bit...the entire unparsed_version
> field.  Anyone got a clever script to do something like this?  Thank
> you. 

# bro-cut host unparsed_version < software.log

should do it.

Johanna

From jlay at slave-tothe-box.net  Tue Jan 12 12:45:38 2016
From: jlay at slave-tothe-box.net (James Lay)
Date: Tue, 12 Jan 2016 13:45:38 -0700
Subject: [Bro] Bro's software log
In-Reply-To: <B8C3115C-B188-4094-8657-3DC32B8E665A@illinois.edu>
References: <5e969c4eeddc70ff40aa4f8479232235@localhost>
	<B8C3115C-B188-4094-8657-3DC32B8E665A@illinois.edu>
Message-ID: <07321d5fcd319f11acefd383f37e9a9d@localhost>

On 2016-01-12 12:58, Azoff, Justin S wrote:
> You're going to laugh... That's what bro-cut is for :-)
> 
> 
>     # zcat software.log.gz | bro-cut host unparsed_version
> 
> 
> Regular cut kind of works too, but bro-cut is faster and easier to use:
> 
>     # zcat software.log.gz | egrep -v "^#" | cut -f 2,11
> 
> --
> - Justin Azoff
> 
>> On Jan 12, 2016, at 2:32 PM, James Lay <jlay at slave-tothe-box.net> 
>> wrote:
>> 
>> I LOVE the software log.  Legit.  It's awesome.  I'm trying to create 
>> a report of sorts, with sed and awk, and for the life of me I'm having 
>> a tough time.  Here's what I got so far:
>> 
>> zcat software.log.gz | bro-cut -d | sed -e 's/<tab character here, ie 
>> ctrl-v, tab>/-/g' -e 's/\-\-\-[A-Z]\{3,5\}::/ /' -e 's/^.*0000-//'
>> 
>> This get me kinda close, but not close enough...here's the raw entry:
>> 
>> 2016-01-01T14:57:02+0000        x.x.x.x  -       HTTP::BROWSER   
>> Windows-Update-Agent    7       9       9600    18145   Client  
>> Windows-Update-Agent/7.9.9600.18145 Client-Protocol/1.21
>> 
>> What' I'm really hoping for is this:
>> x.x.x.x Windows-Update-Agent/7.9.9600.18145 Client-Protocol/1.21
>> 
>> Just the IP address, and the last bit...the entire unparsed_version 
>> field.  Anyone got a clever script to do something like this?  Thank 
>> you.
>> 
>> James
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

Oh for......yugh 8-|  Sigh....some days even the simplest of tasks are 
MIGHTY chores for me.  OH LOOKIE HERE, HERE'S bro-cut --help!  
Gagh....thanks all...I'm going to go back to pretending I have a clue.

James

From paul.halliday at gmail.com  Tue Jan 12 13:05:56 2016
From: paul.halliday at gmail.com (Paul Halliday)
Date: Tue, 12 Jan 2016 17:05:56 -0400
Subject: [Bro] Bro's software log
In-Reply-To: <07321d5fcd319f11acefd383f37e9a9d@localhost>
References: <5e969c4eeddc70ff40aa4f8479232235@localhost>
	<B8C3115C-B188-4094-8657-3DC32B8E665A@illinois.edu>
	<07321d5fcd319f11acefd383f37e9a9d@localhost>
Message-ID: <CAJfn-RG=LueshFNaemc2_HTwsrkK+mSHC+SdE-oFmxCAz=Kuwg@mail.gmail.com>

I actually thought you were trolling for a sec with that sed line. Is he
trying to turn this into an animated gif?

:)

On Tue, Jan 12, 2016 at 4:45 PM, James Lay <jlay at slave-tothe-box.net> wrote:

> On 2016-01-12 12:58, Azoff, Justin S wrote:
> > You're going to laugh... That's what bro-cut is for :-)
> >
> >
> >     # zcat software.log.gz | bro-cut host unparsed_version
> >
> >
> > Regular cut kind of works too, but bro-cut is faster and easier to use:
> >
> >     # zcat software.log.gz | egrep -v "^#" | cut -f 2,11
> >
> > --
> > - Justin Azoff
> >
> >> On Jan 12, 2016, at 2:32 PM, James Lay <jlay at slave-tothe-box.net>
> >> wrote:
> >>
> >> I LOVE the software log.  Legit.  It's awesome.  I'm trying to create
> >> a report of sorts, with sed and awk, and for the life of me I'm having
> >> a tough time.  Here's what I got so far:
> >>
> >> zcat software.log.gz | bro-cut -d | sed -e 's/<tab character here, ie
> >> ctrl-v, tab>/-/g' -e 's/\-\-\-[A-Z]\{3,5\}::/ /' -e 's/^.*0000-//'
> >>
> >> This get me kinda close, but not close enough...here's the raw entry:
> >>
> >> 2016-01-01T14:57:02+0000        x.x.x.x  -       HTTP::BROWSER
> >> Windows-Update-Agent    7       9       9600    18145   Client
> >> Windows-Update-Agent/7.9.9600.18145 Client-Protocol/1.21
> >>
> >> What' I'm really hoping for is this:
> >> x.x.x.x Windows-Update-Agent/7.9.9600.18145 Client-Protocol/1.21
> >>
> >> Just the IP address, and the last bit...the entire unparsed_version
> >> field.  Anyone got a clever script to do something like this?  Thank
> >> you.
> >>
> >> James
> >> _______________________________________________
> >> Bro mailing list
> >> bro at bro-ids.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> Oh for......yugh 8-|  Sigh....some days even the simplest of tasks are
> MIGHTY chores for me.  OH LOOKIE HERE, HERE'S bro-cut --help!
> Gagh....thanks all...I'm going to go back to pretending I have a clue.
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 
Paul Halliday
http://www.pintumbler.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160112/5b110a41/attachment.html 

From hongdal at g.clemson.edu  Tue Jan 12 18:58:09 2016
From: hongdal at g.clemson.edu (Hongda Li)
Date: Tue, 12 Jan 2016 21:58:09 -0500
Subject: [Bro] stdout.log does not work?
Message-ID: <CAK5-Cn7iB4gPyiNACtOZMTscJjpae1+LVbCO7Z2omweQMiS51w@mail.gmail.com>

Hello, Community.

I've a problem with stdout.log.

I add some "print" statements in bro scripts (e.g.,
detect-bruteforcing.bro).
When I start Bro from the command line:
    *bro -i eth2 local *
it prints outputs of the print statements on to the stdout.

However, I start Bro from the broctl:
    *[BroControl] > start*
*    [BroControl] > exit*
then I go to "logs/current/" and try to see the outputs of the print
statements:
    *cat stdout.log*
the only results that I can see are:
    *max memory size         (kbytes, -m) unlimited*
*    data seg size           (kbytes, -d) unlimited*
*    virtual memory          (kbytes, -v) unlimited*
*    core file size          (blocks, -c) unlimited*

I expected that I can read the outputs of the print statements in the
*stdout.log*.

The content of /logs/stderr.log looks like (only one line):
    *listening on eth2, capture length 8192 bytes*
This error message also appears as I start Bro by command line.
I don't think this makes any difference.

I have no idea what is going wrong.

Thanks in advance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160112/9768e56e/attachment-0001.html 

From jan.grashoefer at gmail.com  Wed Jan 13 04:03:30 2016
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Wed, 13 Jan 2016 13:03:30 +0100
Subject: [Bro] stdout.log does not work?
In-Reply-To: <CAK5-Cn7iB4gPyiNACtOZMTscJjpae1+LVbCO7Z2omweQMiS51w@mail.gmail.com>
References: <CAK5-Cn7iB4gPyiNACtOZMTscJjpae1+LVbCO7Z2omweQMiS51w@mail.gmail.com>
Message-ID: <56963D12.6000304@gmail.com>

Hi,

if I remember correctly stdout.log and stderr.log are separately
generated per worker. You should be able to find them in spool/ inside
the worker directories.

Jan

From dirk.leinenbach at consistec.de  Wed Jan 13 08:23:05 2016
From: dirk.leinenbach at consistec.de (Dirk Leinenbach)
Date: Wed, 13 Jan 2016 17:23:05 +0100
Subject: [Bro] loading scripts only if file exists
Message-ID: <569679E9.1090705@consistec.de>

Hi all,

is there a way to @load a script in bro only if the file exists?

My scenario is that I'd like to be able to include configuration data 
from, e.g., /etc/bro-config.bro. But this script might not exist in all 
scenarios. Loading it via @load gives an error if the file doesn't exist.

I've tried to use @prefixes to conditionally load scripts, but couldn't 
figure out exactly where bro would be searching for those scripts.

If there an equivalent for something like "if [-f /etc/bro-config.bro] " ?

Best regards,

Dirk

From grant at grantstavely.com  Wed Jan 13 08:44:22 2016
From: grant at grantstavely.com (Grant Stavely)
Date: Wed, 13 Jan 2016 08:44:22 -0800
Subject: [Bro] loading scripts only if file exists
In-Reply-To: <569679E9.1090705@consistec.de>
References: <569679E9.1090705@consistec.de>
Message-ID: <etPan.56967ee6.2ec14c3b.1cf@Grants-MacBook-Pro.local>

Hi Dirk,

This just came up last month :)

http://mailman.icsi.berkeley.edu/pipermail/bro/2015-November/009312.html

And clarified:?

http://mailman.icsi.berkeley.edu/pipermail/bro/2015-December/009324.html?

Grant

On January 13, 2016 at 08:39:19, Dirk Leinenbach (dirk.leinenbach at consistec.de) wrote:

Hi all,  

is there a way to @load a script in bro only if the file exists?  

My scenario is that I'd like to be able to include configuration data  
from, e.g., /etc/bro-config.bro. But this script might not exist in all  
scenarios. Loading it via @load gives an error if the file doesn't exist.  

I've tried to use @prefixes to conditionally load scripts, but couldn't  
figure out exactly where bro would be searching for those scripts.  

If there an equivalent for something like "if [-f /etc/bro-config.bro] " ?  

Best regards,  

Dirk  
_______________________________________________  
Bro mailing list  
bro at bro-ids.org  
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160113/fbd30cec/attachment.html 

From seth at icir.org  Wed Jan 13 09:04:50 2016
From: seth at icir.org (Seth Hall)
Date: Wed, 13 Jan 2016 12:04:50 -0500
Subject: [Bro] loading scripts only if file exists
In-Reply-To: <569679E9.1090705@consistec.de>
References: <569679E9.1090705@consistec.de>
Message-ID: <254B9ACD-F48A-47D4-8751-43176656F9AD@icir.org>


> On Jan 13, 2016, at 11:23 AM, Dirk Leinenbach <dirk.leinenbach at consistec.de> wrote:
> 
> I've tried to use @prefixes to conditionally load scripts, but couldn't 
> figure out exactly where bro would be searching for those scripts.

I think that prefixes are probably exactly what you?re looking for.  If you add a prefix and then create a file by the right name it will only load it when it finds the file.  This is a little tricky with the current setup unfortunately, I didn?t get this setup quite the way I wanted to a number of years ago.

If you write a file named ?local.local.bro? to your site/ directory, that script will automatically load if it?s found.

> If there an equivalent for something like "if [-f /etc/bro-config.bro] " ?

Not really.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From jlay at slave-tothe-box.net  Wed Jan 13 09:37:51 2016
From: jlay at slave-tothe-box.net (James Lay)
Date: Wed, 13 Jan 2016 10:37:51 -0700
Subject: [Bro] Broctl and ssmtp
Message-ID: <b3aa64b03abf491fde6a9fbbb010a77b@localhost>

Hey all,

I checked gmane for this topic...and I did find one but I'm not sure how 
I need to do this.  I have a couple devices that don't have a full blown 
sendmail install, and instead have an install of ssmtp 
(http://www.havetheknowhow.com/Configure-the-server/Install-ssmtp.html):

[10:35:15 jlay at analysis:~$] ls -l /usr/sbin/sendmail
lrwxrwxrwx 1 root root 5 Jun 30  2012 /usr/sbin/sendmail -> ssmtp

this works well for other things (OpenVAS, etc...) but I can't seem to 
get bro to send me the connection summary at the end of rotating.  I'm I 
supposed to redef something, or do I need to do something else?  Thanks 
all.

James

From josh.guild at morphick.com  Wed Jan 13 10:25:58 2016
From: josh.guild at morphick.com (Josh Guild)
Date: Wed, 13 Jan 2016 13:25:58 -0500
Subject: [Bro] Bro signature for detecting iframes
Message-ID: <CALiPqw4M4vDkf1mwgAidMQNo8aeReWE7BGg77s6iqPg=gsW_EQ@mail.gmail.com>

Howdy all,
Is there any way for Bro to detect/view/parse/whatevs the HTML and/or JS
from a visited webpage? If I need to script something out, I don't mind. My
Google-fu has failed to direct me towards any resource that would be a good
starting point.

Any help would be much appreciated, thanks!

-- 
Josh Guild
Network Intelligence Analyst
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160113/4aa13ae9/attachment.html 

From anthony.kasza at gmail.com  Wed Jan 13 10:50:47 2016
From: anthony.kasza at gmail.com (anthony kasza)
Date: Wed, 13 Jan 2016 10:50:47 -0800
Subject: [Bro] Bro signature for detecting iframes
In-Reply-To: <CALiPqw4M4vDkf1mwgAidMQNo8aeReWE7BGg77s6iqPg=gsW_EQ@mail.gmail.com>
References: <CALiPqw4M4vDkf1mwgAidMQNo8aeReWE7BGg77s6iqPg=gsW_EQ@mail.gmail.com>
Message-ID: <CAEZw2bzY6UKuREfr0G5N__vYO0S218ZF7j0=PThCjKyygMq5rQ@mail.gmail.com>

Bro can reassemble and extract the HTML or Javascript file but does not
have the ability to parse, render, or emulate them.

-AK
On Jan 13, 2016 1:34 PM, "Josh Guild" <josh.guild at morphick.com> wrote:

> Howdy all,
> Is there any way for Bro to detect/view/parse/whatevs the HTML and/or JS
> from a visited webpage? If I need to script something out, I don't mind. My
> Google-fu has failed to direct me towards any resource that would be a good
> starting point.
>
> Any help would be much appreciated, thanks!
>
> --
> Josh Guild
> Network Intelligence Analyst
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160113/9144e3aa/attachment-0001.html 

From josh.guild at morphick.com  Wed Jan 13 10:53:27 2016
From: josh.guild at morphick.com (Josh Guild)
Date: Wed, 13 Jan 2016 18:53:27 +0000
Subject: [Bro] Bro signature for detecting iframes
In-Reply-To: <CAEZw2bzY6UKuREfr0G5N__vYO0S218ZF7j0=PThCjKyygMq5rQ@mail.gmail.com>
References: <CALiPqw4M4vDkf1mwgAidMQNo8aeReWE7BGg77s6iqPg=gsW_EQ@mail.gmail.com>
	<CAEZw2bzY6UKuREfr0G5N__vYO0S218ZF7j0=PThCjKyygMq5rQ@mail.gmail.com>
Message-ID: <CALiPqw5un8fftdnORGin=QcJZzoTqc-DhKzz2pVBSLUD9yKuqQ@mail.gmail.com>

Cool. Doing that for every site just to run a script against sounds like it
would melt the box. Ah well, back to the drawing board.

Thanks!

On Wed, Jan 13, 2016, 13:50 anthony kasza <anthony.kasza at gmail.com> wrote:

> Bro can reassemble and extract the HTML or Javascript file but does not
> have the ability to parse, render, or emulate them.
>
> -AK
> On Jan 13, 2016 1:34 PM, "Josh Guild" <josh.guild at morphick.com> wrote:
>
>> Howdy all,
>> Is there any way for Bro to detect/view/parse/whatevs the HTML and/or JS
>> from a visited webpage? If I need to script something out, I don't mind. My
>> Google-fu has failed to direct me towards any resource that would be a good
>> starting point.
>>
>> Any help would be much appreciated, thanks!
>>
>> --
>> Josh Guild
>> Network Intelligence Analyst
>>
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160113/da8c4cdb/attachment.html 

From hongdal at g.clemson.edu  Wed Jan 13 13:39:07 2016
From: hongdal at g.clemson.edu (Hongda Li)
Date: Wed, 13 Jan 2016 16:39:07 -0500
Subject: [Bro] stdout.log does not work?
In-Reply-To: <56963D12.6000304@gmail.com>
References: <CAK5-Cn7iB4gPyiNACtOZMTscJjpae1+LVbCO7Z2omweQMiS51w@mail.gmail.com>
	<56963D12.6000304@gmail.com>
Message-ID: <CAK5-Cn6oJanhjyo-iJeKw4yffVAYqPF8nPF4LHjyn81bQL2kBA@mail.gmail.com>

Hi,

I was not running Bro in a cluster.
I checked the spool/bro/ direcotry and found stdout.log and stderr.log
there.
However, contents in these two files are the same with the contents in
stdout.log and stderr.log under logs/current/.

I am a newbie to the Bro system.
As a result, I can't get outputs of 'print' statements from
spool/bro/stdout.log or logs/current/stdout.log.
But I am sure I can get the outputs when I run Bro from command line.
Anything gets wrong?

Many thanks!



On Wed, Jan 13, 2016 at 7:03 AM, Jan Grash?fer <jan.grashoefer at gmail.com>
wrote:

> Hi,
>
> if I remember correctly stdout.log and stderr.log are separately
> generated per worker. You should be able to find them in spool/ inside
> the worker directories.
>
> Jan
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>

> I've a problem with stdout.log.
>> I add some "print" statements in bro scripts (e.g.,
>> detect-bruteforcing.bro).
>> When I start Bro from the command line:
>>
>> *bro -i eth2 local *it prints outputs of the print statements on to the
>> stdout.
>> However, I start Bro from the broctl:
>>
>> *[BroControl] > start*
>> *    [BroControl] > exit*then I go to "logs/current/" and try to see the
>> outputs of the print statements:
>>
>> *cat stdout.log*the only results that I can see are:
>>
>> *max memory size         (kbytes, -m) unlimited*
>> *    data seg size           (kbytes, -d) unlimited*
>> *    virtual memory          (kbytes, -v) unlimited**    core file size
>>          (blocks, -c) unlimited*
>> I expected that I can read the outputs of the print statements in the
>> *stdout.log*.
>> The content of /logs/stderr.log looks like (only one line):
>>
>> *listening on eth2, capture length 8192 bytes*This error message also
>> appears as I start Bro by command line.
>> I don't think this makes any difference.
>> I have no idea what is going wrong.
>> Thanks in advance.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160113/10785917/attachment.html 

From doris at bro.org  Wed Jan 13 14:51:53 2016
From: doris at bro.org (Doris Schioberg)
Date: Wed, 13 Jan 2016 14:51:53 -0800
Subject: [Bro] Bro4Pros 2016: The Bro workshop for advanced users
Message-ID: <5696D509.4070008@bro.org>

We are happy to announce that we will continue Bro4Pros, our workshop
for advanced Bro users. Going beyond the introductory level of our
regular BroCons, this workshop aims at users who are already using Bro
on a daily basis, feel comfortable customizing its configuration, and
have written a few scripts of their own already. The event will consist
mainly of interactive live demos and practical talks.

Bro4Pros will take place on March 15, 2016 in San Francisco, CA.
We are grateful to Mozilla for hosting us at their San Francisco office
on 2 Harrison St. Thank you Mozilla!

There is no registration fee but we are limiting attendance on a
first-come first-served basis. Please RSVP here:
https://www.eventbrite.com/e/bro4pros-2016-tickets-20748120186

We will have a few slots for external presentations. We are looking for
practical talks and demos on advanced topics that help experienced Bro
users make even more of their setup.

For more details please refer to
https://bro.org/community/bro4pros2016.html.

We?re still looking for an organization interested in sponsoring food &
beverages for the event. Contact us at info at bro.org for more details.

Also, keep following our news. We will soon announce details for BroCon'16.

Looking forward to seeing you,

The Bro Team


-- 
Doris Schioberg
Bro Outreach, Training, and Education Coordinator
International Computer Science Institute (ICSI Berkeley)
Phone: +1 (510) 289-8406 * doris at bro.org

From seth at icir.org  Thu Jan 14 06:11:04 2016
From: seth at icir.org (Seth Hall)
Date: Thu, 14 Jan 2016 09:11:04 -0500
Subject: [Bro] Bro signature for detecting iframes
In-Reply-To: <CALiPqw5un8fftdnORGin=QcJZzoTqc-DhKzz2pVBSLUD9yKuqQ@mail.gmail.com>
References: <CALiPqw4M4vDkf1mwgAidMQNo8aeReWE7BGg77s6iqPg=gsW_EQ@mail.gmail.com>
	<CAEZw2bzY6UKuREfr0G5N__vYO0S218ZF7j0=PThCjKyygMq5rQ@mail.gmail.com>
	<CALiPqw5un8fftdnORGin=QcJZzoTqc-DhKzz2pVBSLUD9yKuqQ@mail.gmail.com>
Message-ID: <638A75B4-3AF2-4C76-8459-912C6DD3E828@icir.org>


> On Jan 13, 2016, at 1:53 PM, Josh Guild <josh.guild at morphick.com> wrote:
> 
> Cool. Doing that for every site just to run a script against sounds like it would melt the box. Ah well, back to the drawing board.

Yeah, it?s one of the general problems with doing parsing of HTML, etc. is that it?s way too easy to melt the box. :)   Some of it can be done but it needs to be done in very particular ways.  You generally don?t get the opportunity to just blindly load an entire HTML document into memory and parse it into a DOM (unfortunately, because that would be awesome if it was reasonable!).

There is some very early that has been done that lets you parse out various bits from files.  For example, I have a script that uses that functionality to parse titles out of webpages and add it to the http log.  It does need some more work though, there are some functionality and behavior issues that need corrected.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From dirk.leinenbach at consistec.de  Fri Jan 15 07:01:02 2016
From: dirk.leinenbach at consistec.de (Dirk Leinenbach)
Date: Fri, 15 Jan 2016 16:01:02 +0100
Subject: [Bro] loading scripts only if file exists
In-Reply-To: <254B9ACD-F48A-47D4-8751-43176656F9AD@icir.org>
References: <569679E9.1090705@consistec.de>
	<254B9ACD-F48A-47D4-8751-43176656F9AD@icir.org>
Message-ID: <569909AE.304@consistec.de>

Hi Seth,

 1. Is ?local.local.bro?a hard-coded name that will always work, even
    without extending @prefixes?
 2. If I extend @prefixes inside a plugin script, which directory would
    be searched? E.g., if I extend prefixes in the following file:

            /opt/bro/lib/bro/plugins/Demo_demo/scripts/Demo/demo/bla.bro
            @prefixes = myprefix

            while my bro plugin search path is: /opt/bro/lib/bro/plugins


        If I understand the documentation correctly, this would search
        for files named

            myprefix.Demo_demo.scripts.Demo.demo.bla.bro


        Is that correct?


Thanks a lot!

Dirk

On 13.01.2016 18:04, Seth Hall wrote:
>> On Jan 13, 2016, at 11:23 AM, Dirk Leinenbach <dirk.leinenbach at consistec.de> wrote:
>>
>> I've tried to use @prefixes to conditionally load scripts, but couldn't
>> figure out exactly where bro would be searching for those scripts.
> I think that prefixes are probably exactly what you?re looking for.  If you add a prefix and then create a file by the right name it will only load it when it finds the file.  This is a little tricky with the current setup unfortunately, I didn?t get this setup quite the way I wanted to a number of years ago.
>
> If you write a file named ?local.local.bro? to your site/ directory, that script will automatically load if it?s found.
>
>> If there an equivalent for something like "if [-f /etc/bro-config.bro] " ?
> 
> Not really.
>
>    .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>

-- 

Dr.-Ing. Dirk Leinenbach - Leitung Softwareentwicklung
consistec Engineering & Consulting GmbH
------------------------------------------------------------------

Europaallee 5                      Fon:   +49 (0)681 / 959044-0
D-66113 Saarbr?cken                Fax:   +49 (0)681 / 959044-11
http://www.consistec.de            e-mail: dirk.leinenbach at consistec.de

Registergericht: Amtsgericht Saarbr?cken
Registerblatt:   HRB12003
Gesch?ftsf?hrer: Dr. Thomas Sinnwell, Volker Leiendecker, Stefan Sinnwell

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160115/ee06442f/attachment.html 

From edthoma at sandia.gov  Fri Jan 15 11:54:16 2016
From: edthoma at sandia.gov (Thomas, Eric D)
Date: Fri, 15 Jan 2016 19:54:16 +0000
Subject: [Bro] Does Broccoli work in cluster config?
Message-ID: <D2BE8E67.26F36%edthoma@sandia.gov>

I have a python Broccoli script that works when bro is running in standalone mode and listening on a port I specify. Now I want to get it working when Bro is running in a localhost cluster config. I can see my system listening on the port, and I know my broccoli script is connecting to the port. But the broccoli script never receives any events.

Is it the manager, proxy, or worker process that listens on the port? If it's the manager or proxy I suppose it could broker the events. But if multiple workers try to bind to the specified port, of course only one could. If I don't specify the listening port, would the workers try to open multiple ports? How would my broccoli script know which ports to connect to? Would I have to run multiple instances of my broccoli script, one for each port (or heavily alter my script to connect to multiple ports)?

In short, how is this supposed to be done in a cluster setup?

I saw a question from July 2015 on this very subject, but I didn't find any responses. http://mailman.icsi.berkeley.edu/pipermail/bro/2015-July/008726.html

--
Eric Thomas
edthoma at sandia.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160115/a60f4032/attachment.html 

From donaldson8 at llnl.gov  Fri Jan 15 12:14:36 2016
From: donaldson8 at llnl.gov (Donaldson, John)
Date: Fri, 15 Jan 2016 20:14:36 +0000
Subject: [Bro] High packet drop rates
Message-ID: <C623401B7419BD4E8A983814FEFA02F6E47CCC4E@PRDEXMBX-07.the-lab.llnl.gov>

I've recently run into a problem, with Bro 2.4.1, where I have extremely high packet loss.

I'm running on a server with dual quad-core Xeon processors, with no hyperthreading, and 64GB of RAM, monitoring a few small links, averaging, in aggregate, about 100Mbps of traffic. There isn't too much else running on this system, but we're seeing drop rates that average in the 70-99% range (via the capture-loss.bro policy), even though Bro's CPU utilization sits at around 20-30%. 

Most of the traffic comes in from a bond interface, running on top of some Intel NICs, but we're seeing similarly high drop rates when directly capturing from another, non-bonded interface. Using other tools, we're not seeing any dropped packets (even with a heavily-loaded Snort instance). We've tried PF_RING and load-balancing across several workers, pinned to several CPUs, but all that we end up with, then, are multiple processes with 2-30% CPU utilization and 70-99% drop rates. PF_RING isn't showing any drops on its side, and hans't had issues with insufficient memory. We're pretty sure that we're not just seeing TCP-related chaff that's throwing off our numbers, because records of known connections are showing up malformed.

Any insights?

John Donaldson




From jazoff at illinois.edu  Fri Jan 15 12:31:45 2016
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Fri, 15 Jan 2016 20:31:45 +0000
Subject: [Bro] High packet drop rates
In-Reply-To: <C623401B7419BD4E8A983814FEFA02F6E47CCC4E@PRDEXMBX-07.the-lab.llnl.gov>
References: <C623401B7419BD4E8A983814FEFA02F6E47CCC4E@PRDEXMBX-07.the-lab.llnl.gov>
Message-ID: <844360EA-ACCB-4C1D-A7AA-91F625060F00@illinois.edu>


> On Jan 15, 2016, at 3:14 PM, Donaldson, John <donaldson8 at llnl.gov> wrote:
> 
> I've recently run into a problem, with Bro 2.4.1, where I have extremely high packet loss.
> 
> I'm running on a server with dual quad-core Xeon processors, with no hyperthreading, and 64GB of RAM, monitoring a few small links, averaging, in aggregate, about 100Mbps of traffic. There isn't too much else running on this system, but we're seeing drop rates that average in the 70-99% range (via the capture-loss.bro policy), even though Bro's CPU utilization sits at around 20-30%. 

A likely reason for this is the various NIC offloading features being enabled causing bro to not properly capture entire packets.  Is your reporter.log complaining about this at all?

Can you try running

    for i in rx tx sg tso ufo gso gro lro; do ethtool -K eth0 $i off; done  

to disable all of those optional features (replace eth0 with appropriate interfaces)

Also, if you are using jumbo frames you may need to add to local.bro:

    redef snaplen = 9000; #potentially as high as 9216

and ensure that the mtu on the nic is set appropriately as well.

Once you know you have capture loss it can be a good idea to look at the lower level records to see which connections are missing bytes:

    cat conn.log | bro-cut id.orig_h id.resp_h id.resp_p orig_bytes resp_bytes missed_bytes

It's possible that all of your missed_bytes are coming from a small subset of hosts.  We ran into that issue a while ago due to our MTU being just too small to properly capture traffic between 2 hosts.  Since it was a large backup transfer our capture loss would shoot up to 75%+ even though it was only one flow being missed.



Since your traffic rate is relatively low one potential troubleshooting option is to dump the full traffic to a file using something like 'tcpdump -s 0 -w dump.cap -i eth0' and then run bro against that pcap file and see what that reports.


-- 
- Justin Azoff



From barak.work.email at gmail.com  Sun Jan 17 01:02:39 2016
From: barak.work.email at gmail.com (barak gilboa)
Date: Sun, 17 Jan 2016 11:02:39 +0200
Subject: [Bro] bro manager stops writing logs - EINTR issue ?
Message-ID: <CAG=sQiXSWPjy45m67uyomOSNKDvGRkPWa12EW3YPNDZWi3phcg@mail.gmail.com>

Hello,
I would appreciate anyone's help on the following issue :

setup: 24 workers,1 proxy, 1 manager. each worker has a bloomfilter of its
own so eventually very few events are passed on to the manager for writing.
there is only 1 log file being written (dns.log) which fills at a rate of
about 10k lines per sec.

problem: after a few hours, manager stops writing the log file though
everything is still running. no errors on debug.log or stderr.log.
I ran strace and found that the manager's child process has *EINTR* issue:

ERESTARTNOHAND to be restarted if no handler
SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL}
rt_sigreturn()=-1 EINTR (interrupted system call)

I read that bro should handle EINTR errors internally.
any suggestions on what can be done ?

thanks !
Barak
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160117/55b3629c/attachment.html 

From hckim at narusec.com  Mon Jan 18 23:48:46 2016
From: hckim at narusec.com (=?UTF-8?B?6rmA7Z2s7LKg?=)
Date: Tue, 19 Jan 2016 16:48:46 +0900
Subject: [Bro] trying to read space separate file to bro
Message-ID: <CAHs13AiwkoFuujcQ4CTk5xjRAdOTfvEzrUqXMxyYx55-0szGLg@mail.gmail.com>

Hi
I am trying to read file which has space separate

I add     redef separator = " " ; but it's gave me some errors

error: ./aaa.txt/Input::READER_ASCII: Did not find requested field sip in
input data file ./aaa.txt.

error: ./aaa.txt/Input::READER_ASCII: Init: cannot open ./aaa.txt; headers
are incorrect

error: ./aaa.txt/Input::READER_ASCII: Init failed


if aaa.txt is tsv file and with out redef separator, it works fine

Is there a way to read a file which is not tsv


here is my sample aaa.txt and bro script

aaa.txt

#fields sip sport dip dport

192.168.1.116 61711 172.16.100.132 22



bro script

export {

type Val: record {

sip: addr;

sport: port;

dip: addr;

dport: port;

};

redef Input::separator = " ";

}


event TEST(description: Input::EventDescription, tpe: Input::Event, sip:
addr, sport: port, dip: addr, dport: port){

print fmt("%s %d %s %d",sip,sport,dip,dport);

}


event bro_init()

{

print fmt("test");

Input::add_event([$source="./aaa.txt", $name="test", $fields=Val, $ev=TEST
,$want_record=F]);

}

-- 
------------------------------------------------------
Hichul Kim ??? ?? ???

Naru Security  (?)??????
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160119/d15ade34/attachment.html 

From dnthayer at illinois.edu  Tue Jan 19 08:53:20 2016
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Tue, 19 Jan 2016 10:53:20 -0600
Subject: [Bro] trying to read space separate file to bro
In-Reply-To: <CAHs13AiwkoFuujcQ4CTk5xjRAdOTfvEzrUqXMxyYx55-0szGLg@mail.gmail.com>
References: <CAHs13AiwkoFuujcQ4CTk5xjRAdOTfvEzrUqXMxyYx55-0szGLg@mail.gmail.com>
Message-ID: <569E6A00.1000706@illinois.edu>

In your script, you need to change one line to use this:

redef InputAscii::separator = " ";



On 01/19/2016 01:48 AM, ??? wrote:
> Hi
> I am trying to read file which has space separate
>
> I add redef separator = " " ; but it's gave me some errors
>
> error: ./aaa.txt/Input::READER_ASCII: Did not find requested field sip
> in input data file ./aaa.txt.
>
> error: ./aaa.txt/Input::READER_ASCII: Init: cannot open ./aaa.txt;
> headers are incorrect
>
> error: ./aaa.txt/Input::READER_ASCII: Init failed
>
>
> if aaa.txt is tsv file and with out redef separator, it works fine
>
> Is there a way to read a file which is not tsv
>
>
> here is my sample aaa.txt and bro script
>
> aaa.txt
>
> #fields sip sport dip dport
>
> 192.168.1.116 61711 172.16.100.132 22
>
>
>
> bro script
>
> export {
>
>     type Val: record {
>
>         sip: addr;
>
>         sport: port;
>
>         dip: addr;
>
>         dport: port;
>
>     };
>
>     redef Input::separator = " ";
>
> }
>
>
> event TEST(description: Input::EventDescription, tpe: Input::Event, sip:
> addr, sport: port, dip: addr, dport: port){
>
>     print fmt("%s %d %s %d",sip,sport,dip,dport);
>
> }
>
>
> event bro_init()
>
> {
>
>     print fmt("test");
>
>     Input::add_event([$source="./aaa.txt", $name="test", $fields=Val,
>     $ev=TEST ,$want_record=F]);
>
> }
>
>
> --
> ------------------------------------------------------
> Hichul Kim ??? ?? ???
>
> Naru Security  (?)??????
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>

From jdopheid at illinois.edu  Tue Jan 19 09:24:43 2016
From: jdopheid at illinois.edu (Dopheide, Jeannette M)
Date: Tue, 19 Jan 2016 17:24:43 +0000
Subject: [Bro] Bro training at Educause Security Professionals Conference,
 April 18th
Message-ID: <8A994F26-26FC-47B5-9481-B494CA01A10B@illinois.edu>

The Bro Team is presenting a training at the 2016 Educause Security Professionals Conference on Monday April 18th in Seattle, Washington. Our topic is "Monitoring Your Science DMZ with Bro." 

The Science DMZ architecture is spreading throughout higher ed, with both the National Science Foundation and ESNet promoting this more open model to campus cyberinfrastructure providers. NSF has invested in building 10G and 100G research networks to meet the growing demands of the nation's scientists and engineers, but this brings challenges to many traditional ways of securing campus infrastructure. Bro has a successful track record protecting some of the most demanding networks in the country, and it comes well prepared to address the security challenges of bringing data and compute resources outside the firewall for performance. The NSF Bro Center of Expertise is hosting a one-day Bro training workshop aimed at higher ed operators of cyberinfrastructure. This hands-on training will demonstrate Bro's capabilities, teach you how to set up and configure Bro, discuss strategies for deployment on high-speed networks, and present opportunities for further individual assistance from the center.

OUTCOMES: 
* Understand the purpose of Bro and its features 
* Learn how to install Bro with suggested configuration modifications (e.g., load balancing, pinning workers) 
* Learn how to seek assistance from the Bro Center of Expertise


Event link:
http://www.educause.edu/events/security-professionals-conference


Bro Abstract: 
http://www.educause.edu/events/security-professionals-conference/2016/monitoring-your-science-dmz-bro

------

Jeannette Dopheide
Bro Outreach Coordinator
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign






From JStallard at enquizit.com  Tue Jan 19 09:37:27 2016
From: JStallard at enquizit.com (James Stallard)
Date: Tue, 19 Jan 2016 17:37:27 +0000
Subject: [Bro] Info on configuring bro inline in AWS as IDS
Message-ID: <BLUPR0501MB2066D1993A598629E8F7DC10CFC10@BLUPR0501MB2066.namprd05.prod.outlook.com>

Hello Bros:


I'm just now installing bro for the government website at Small Business Admin.

The plan is to have bro behind our public ELBs as an in-line IDS, then route traffic to internal ELBs in front of our application / web servers.


As this is AWS, no tap is possible and the EC2s can be run in promiscuous mode either.


After a quick review of the documentation, I don't see where I can configure the routing once bro has done its work.


I.E. if I configure:


bro -i en0 <list of scripts to load>

do I need to then configure a script that will export all traffic to another agent such as an ELB or nginx ?


Any help would be appreceated.


JMS
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160119/ce1b6530/attachment.html 

From jlay at slave-tothe-box.net  Tue Jan 19 10:12:55 2016
From: jlay at slave-tothe-box.net (James Lay)
Date: Tue, 19 Jan 2016 11:12:55 -0700
Subject: [Bro] Smtp.log missing x-originating-ip
Message-ID: <c72210887bd5deb66c149b28631dc49e@localhost>

Topic says it...after a digging into this it appears my smtp.log is 
missing all x-originating-ip:

[18:11:06 ids:~/current$] head -n 40 smtp.log | bro-cut -d ts 
x_originating_ip
2016-01-18T23:58:31+0000        -
2016-01-18T23:58:34+0000        -
2016-01-18T23:58:32+0000        -
2016-01-18T23:58:35+0000        -
2016-01-18T23:58:39+0000        -
2016-01-18T23:58:46+0000        -
2016-01-18T23:58:52+0000        -
2016-01-18T23:59:02+0000        -
2016-01-18T23:59:04+0000        -

I can see the field in full packet captures.  Any hints on what I'm 
missing?  Thank you.

James

From dopheide at gmail.com  Tue Jan 19 12:32:17 2016
From: dopheide at gmail.com (Mike Dopheide)
Date: Tue, 19 Jan 2016 14:32:17 -0600
Subject: [Bro] Info on configuring bro inline in AWS as IDS
In-Reply-To: <BLUPR0501MB2066D1993A598629E8F7DC10CFC10@BLUPR0501MB2066.namprd05.prod.outlook.com>
References: <BLUPR0501MB2066D1993A598629E8F7DC10CFC10@BLUPR0501MB2066.namprd05.prod.outlook.com>
Message-ID: <CAPy2kFZaCdfq-d5==q+GH5A=wbSnzFf0HezYHa0DNm9STm_6qg@mail.gmail.com>

I'm not very familiar with Amazon ELBs, but this is an interesting model so
I have a couple clarifying questions to make sure we understand what you're
trying to do

1)  So the model is ext_ELB -> Bro/router -> int_ELB, using Bro as an IPS
rather than IDS?   Are you planning multiple Bro instances to handle the
load and provide failover?

2)  Bro, by itself, is not a routing engine.  It doesn't pass traffic out
to another interface once it's done examining it.

If I understand what you're trying to do, you'd need to setup a software
router (pfSense, Clickrouter, PacketBricks?, Microtik's RouterOS) have it
mirror traffic to Bro, and then write Bro policies to inject rules into the
router as needed.  I'm not sure if someone has already done it, but it
wouldn't be an insignificant effort.

(I believe Amazon supports a few virtual IPS appliances, like Palo Alto or
Juniper as well.)

-Dop

On Tue, Jan 19, 2016 at 11:37 AM, James Stallard <JStallard at enquizit.com>
wrote:

> Hello Bros:
>
>
> I'm just now installing bro for the government website at Small Business
> Admin.
>
> The plan is to have bro behind our public ELBs as an in-line IDS, then
> route traffic to internal ELBs in front of our application / web servers.
>
>
> As this is AWS, no tap is possible and the EC2s can be run in promiscuous
> mode either.
>
>
> After a quick review of the documentation, I don't see where I can
> configure the routing once bro has done its work.
>
>
> I.E. if I configure:
>
>
> bro -i en0 <list of scripts to load>
>
> do I need to then configure a script that will export all traffic to
> another agent such as an ELB or nginx ?
>
>
> Any help would be appreceated.
>
>
> JMS
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160119/e6d45f2b/attachment.html 

From JStallard at enquizit.com  Tue Jan 19 12:43:32 2016
From: JStallard at enquizit.com (James Stallard)
Date: Tue, 19 Jan 2016 20:43:32 +0000
Subject: [Bro] Info on configuring bro inline in AWS as IDS
In-Reply-To: <CAPy2kFZaCdfq-d5==q+GH5A=wbSnzFf0HezYHa0DNm9STm_6qg@mail.gmail.com>
References: <BLUPR0501MB2066D1993A598629E8F7DC10CFC10@BLUPR0501MB2066.namprd05.prod.outlook.com>,
	<CAPy2kFZaCdfq-d5==q+GH5A=wbSnzFf0HezYHa0DNm9STm_6qg@mail.gmail.com>
Message-ID: <BLUPR0501MB20660103210628DFDDB3E23ECFC10@BLUPR0501MB2066.namprd05.prod.outlook.com>

Oh, thanks, Mike...

Comments/responses posted below.


________________________________
From: Mike Dopheide <dopheide at gmail.com>
Sent: Tuesday, January 19, 2016 3:32 PM
To: James Stallard
Cc: bro at bro.org
Subject: Re: [Bro] Info on configuring bro inline in AWS as IDS

I'm not very familiar with Amazon ELBs, but this is an interesting model so I have a couple clarifying questions to make sure we understand what you're trying to do

1)  So the model is ext_ELB -> Bro/router -> int_ELB,
>>YES using Bro as an IPS rather than IDS?
>>No, initially just an IDS
  Are you planning multiple Bro instances to handle the load and provide failover?
>>Yes, initially 2 loadbalanced behind ext_ELB.

2)  Bro, by itself, is not a routing engine.  It doesn't pass traffic out to another interface once it's done examining it.
>>Ikes!

If I understand what you're trying to do, you'd need to setup a software router (pfSense, Clickrouter, PacketBricks?, Microtik's RouterOS) have it mirror traffic to Bro, and then write Bro policies to inject rules into the router as needed.  I'm not sure if someone has already done it, but it wouldn't be an insignificant effort.

(I believe Amazon supports a few virtual IPS appliances, like Palo Alto or Juniper as well.)
>> OK, I'll touch base with our AWS contact  for this.

>>Thanks for the tips.

JMS
-Dop

On Tue, Jan 19, 2016 at 11:37 AM, James Stallard <JStallard at enquizit.com<mailto:JStallard at enquizit.com>> wrote:

Hello Bros:


I'm just now installing bro for the government website at Small Business Admin.

The plan is to have bro behind our public ELBs as an in-line IDS, then route traffic to internal ELBs in front of our application / web servers.


As this is AWS, no tap is possible and the EC2s can be run in promiscuous mode either.


After a quick review of the documentation, I don't see where I can configure the routing once bro has done its work.


I.E. if I configure:


bro -i en0 <list of scripts to load>

do I need to then configure a script that will export all traffic to another agent such as an ELB or nginx ?


Any help would be appreceated.


JMS

_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160119/032200de/attachment.html 

From seth at icir.org  Tue Jan 19 12:53:36 2016
From: seth at icir.org (Seth Hall)
Date: Tue, 19 Jan 2016 15:53:36 -0500
Subject: [Bro] Smtp.log missing x-originating-ip
In-Reply-To: <c72210887bd5deb66c149b28631dc49e@localhost>
References: <c72210887bd5deb66c149b28631dc49e@localhost>
Message-ID: <FD286C7B-ACCF-48EF-AC01-0B239C4B0F51@icir.org>


> On Jan 19, 2016, at 1:12 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> 
> I can see the field in full packet captures.  Any hints on what I'm 
> missing?  Thank you.

Could you privately send along a couple of headers that it's messing up?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/


From seth at icir.org  Tue Jan 19 12:54:50 2016
From: seth at icir.org (Seth Hall)
Date: Tue, 19 Jan 2016 15:54:50 -0500
Subject: [Bro] trying to read space separate file to bro
In-Reply-To: <569E6A00.1000706@illinois.edu>
References: <CAHs13AiwkoFuujcQ4CTk5xjRAdOTfvEzrUqXMxyYx55-0szGLg@mail.gmail.com>
	<569E6A00.1000706@illinois.edu>
Message-ID: <C1990C68-FE99-4F36-9369-E46A32F398DA@icir.org>

You need to be careful with this setting too.  It's easy to mess up other activities (like intelligence import) if you do a setting like this globally.  There is a $config field in the input description where you should be able to specify that field too.

 .Seth



> On Jan 19, 2016, at 11:53 AM, Daniel Thayer <dnthayer at illinois.edu> wrote:
> 
> In your script, you need to change one line to use this:
> 
> redef InputAscii::separator = " ";
> 
> 
> 
> On 01/19/2016 01:48 AM, ??? wrote:
>> Hi
>> I am trying to read file which has space separate
>> 
>> I add redef separator = " " ; but it's gave me some errors
>> 
>> error: ./aaa.txt/Input::READER_ASCII: Did not find requested field sip
>> in input data file ./aaa.txt.
>> 
>> error: ./aaa.txt/Input::READER_ASCII: Init: cannot open ./aaa.txt;
>> headers are incorrect
>> 
>> error: ./aaa.txt/Input::READER_ASCII: Init failed
>> 
>> 
>> if aaa.txt is tsv file and with out redef separator, it works fine
>> 
>> Is there a way to read a file which is not tsv
>> 
>> 
>> here is my sample aaa.txt and bro script
>> 
>> aaa.txt
>> 
>> #fields sip sport dip dport
>> 
>> 192.168.1.116 61711 172.16.100.132 22
>> 
>> 
>> 
>> bro script
>> 
>> export {
>> 
>>    type Val: record {
>> 
>>        sip: addr;
>> 
>>        sport: port;
>> 
>>        dip: addr;
>> 
>>        dport: port;
>> 
>>    };
>> 
>>    redef Input::separator = " ";
>> 
>> }
>> 
>> 
>> event TEST(description: Input::EventDescription, tpe: Input::Event, sip:
>> addr, sport: port, dip: addr, dport: port){
>> 
>>    print fmt("%s %d %s %d",sip,sport,dip,dport);
>> 
>> }
>> 
>> 
>> event bro_init()
>> 
>> {
>> 
>>    print fmt("test");
>> 
>>    Input::add_event([$source="./aaa.txt", $name="test", $fields=Val,
>>    $ev=TEST ,$want_record=F]);
>> 
>> }
>> 
>> 
>> --
>> ------------------------------------------------------
>> Hichul Kim ??? ?? ???
>> 
>> Naru Security  (?)??????
>> 
>> 
>> 
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From dnthayer at illinois.edu  Tue Jan 19 13:27:35 2016
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Tue, 19 Jan 2016 15:27:35 -0600
Subject: [Bro] trying to read space separate file to bro
In-Reply-To: <C1990C68-FE99-4F36-9369-E46A32F398DA@icir.org>
References: <CAHs13AiwkoFuujcQ4CTk5xjRAdOTfvEzrUqXMxyYx55-0szGLg@mail.gmail.com>
	<569E6A00.1000706@illinois.edu>
	<C1990C68-FE99-4F36-9369-E46A32F398DA@icir.org>
Message-ID: <569EAA47.1060000@illinois.edu>

Good point, Seth.  Here is an example of how to use the $config field:

Input::add_event([$source="./aaa.txt", $name="test", $fields=Val,
                   $ev=TEST, $want_record=F,
                   $config=table(["separator"]=" ")]);



On 01/19/2016 02:54 PM, Seth Hall wrote:
> You need to be careful with this setting too.  It's easy to mess up other activities (like intelligence import) if you do a setting like this globally.  There is a $config field in the input description where you should be able to specify that field too.
>
>   .Seth
>
>
>
>> On Jan 19, 2016, at 11:53 AM, Daniel Thayer <dnthayer at illinois.edu> wrote:
>>
>> In your script, you need to change one line to use this:
>>
>> redef InputAscii::separator = " ";
>>
>>
>>
>> On 01/19/2016 01:48 AM, ??? wrote:
>>> Hi
>>> I am trying to read file which has space separate
>>>
>>> I add redef separator = " " ; but it's gave me some errors
>>>
>>> error: ./aaa.txt/Input::READER_ASCII: Did not find requested field sip
>>> in input data file ./aaa.txt.
>>>
>>> error: ./aaa.txt/Input::READER_ASCII: Init: cannot open ./aaa.txt;
>>> headers are incorrect
>>>
>>> error: ./aaa.txt/Input::READER_ASCII: Init failed
>>>
>>>
>>> if aaa.txt is tsv file and with out redef separator, it works fine
>>>
>>> Is there a way to read a file which is not tsv
>>>
>>>
>>> here is my sample aaa.txt and bro script
>>>
>>> aaa.txt
>>>
>>> #fields sip sport dip dport
>>>
>>> 192.168.1.116 61711 172.16.100.132 22
>>>
>>>
>>>
>>> bro script
>>>
>>> export {
>>>
>>>     type Val: record {
>>>
>>>         sip: addr;
>>>
>>>         sport: port;
>>>
>>>         dip: addr;
>>>
>>>         dport: port;
>>>
>>>     };
>>>
>>>     redef Input::separator = " ";
>>>
>>> }
>>>
>>>
>>> event TEST(description: Input::EventDescription, tpe: Input::Event, sip:
>>> addr, sport: port, dip: addr, dport: port){
>>>
>>>     print fmt("%s %d %s %d",sip,sport,dip,dport);
>>>
>>> }
>>>
>>>
>>> event bro_init()
>>>
>>> {
>>>
>>>     print fmt("test");
>>>
>>>     Input::add_event([$source="./aaa.txt", $name="test", $fields=Val,
>>>     $ev=TEST ,$want_record=F]);
>>>
>>> }
>>>
>>>
>>> --
>>> ------------------------------------------------------
>>> Hichul Kim ??? ?? ???
>>>
>>> Naru Security  (?)??????
>>>
>>>
>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>

From hckim at narusec.com  Tue Jan 19 16:40:05 2016
From: hckim at narusec.com (=?UTF-8?B?6rmA7Z2s7LKg?=)
Date: Wed, 20 Jan 2016 09:40:05 +0900
Subject: [Bro] trying to read space separate file to bro
In-Reply-To: <569EAA47.1060000@illinois.edu>
References: <CAHs13AiwkoFuujcQ4CTk5xjRAdOTfvEzrUqXMxyYx55-0szGLg@mail.gmail.com>
	<569E6A00.1000706@illinois.edu>
	<C1990C68-FE99-4F36-9369-E46A32F398DA@icir.org>
	<569EAA47.1060000@illinois.edu>
Message-ID: <CAHs13AhqpHEHXr7z4DP+wA9y9FRhM=h8i7To-P2=E_=UNKTP7Q@mail.gmail.com>

It works perfectly

thank you very much

On Wed, Jan 20, 2016 at 6:27 AM, Daniel Thayer <dnthayer at illinois.edu>
wrote:

> Good point, Seth.  Here is an example of how to use the $config field:
>
> Input::add_event([$source="./aaa.txt", $name="test", $fields=Val,
>                   $ev=TEST, $want_record=F,
>                   $config=table(["separator"]=" ")]);
>
>
>
>
> On 01/19/2016 02:54 PM, Seth Hall wrote:
>
>> You need to be careful with this setting too.  It's easy to mess up other
>> activities (like intelligence import) if you do a setting like this
>> globally.  There is a $config field in the input description where you
>> should be able to specify that field too.
>>
>>   .Seth
>>
>>
>>
>> On Jan 19, 2016, at 11:53 AM, Daniel Thayer <dnthayer at illinois.edu>
>>> wrote:
>>>
>>> In your script, you need to change one line to use this:
>>>
>>> redef InputAscii::separator = " ";
>>>
>>>
>>>
>>> On 01/19/2016 01:48 AM, ??? wrote:
>>>
>>>> Hi
>>>> I am trying to read file which has space separate
>>>>
>>>> I add redef separator = " " ; but it's gave me some errors
>>>>
>>>> error: ./aaa.txt/Input::READER_ASCII: Did not find requested field sip
>>>> in input data file ./aaa.txt.
>>>>
>>>> error: ./aaa.txt/Input::READER_ASCII: Init: cannot open ./aaa.txt;
>>>> headers are incorrect
>>>>
>>>> error: ./aaa.txt/Input::READER_ASCII: Init failed
>>>>
>>>>
>>>> if aaa.txt is tsv file and with out redef separator, it works fine
>>>>
>>>> Is there a way to read a file which is not tsv
>>>>
>>>>
>>>> here is my sample aaa.txt and bro script
>>>>
>>>> aaa.txt
>>>>
>>>> #fields sip sport dip dport
>>>>
>>>> 192.168.1.116 61711 172.16.100.132 22
>>>>
>>>>
>>>>
>>>> bro script
>>>>
>>>> export {
>>>>
>>>>     type Val: record {
>>>>
>>>>         sip: addr;
>>>>
>>>>         sport: port;
>>>>
>>>>         dip: addr;
>>>>
>>>>         dport: port;
>>>>
>>>>     };
>>>>
>>>>     redef Input::separator = " ";
>>>>
>>>> }
>>>>
>>>>
>>>> event TEST(description: Input::EventDescription, tpe: Input::Event, sip:
>>>> addr, sport: port, dip: addr, dport: port){
>>>>
>>>>     print fmt("%s %d %s %d",sip,sport,dip,dport);
>>>>
>>>> }
>>>>
>>>>
>>>> event bro_init()
>>>>
>>>> {
>>>>
>>>>     print fmt("test");
>>>>
>>>>     Input::add_event([$source="./aaa.txt", $name="test", $fields=Val,
>>>>     $ev=TEST ,$want_record=F]);
>>>>
>>>> }
>>>>
>>>>
>>>> --
>>>> ------------------------------------------------------
>>>> Hichul Kim ??? ?? ???
>>>>
>>>> Naru Security  (?)??????
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Bro mailing list
>>>> bro at bro-ids.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>>
>>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>
>>>
>>>
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro.org/
>>
>>


-- 
------------------------------------------------------
Hichul Kim ??? ?? ???

Naru Security  (?)??????
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160120/b8770094/attachment.html 

From tgdesrochers at gmail.com  Tue Jan 19 16:58:57 2016
From: tgdesrochers at gmail.com (Tim Desrochers)
Date: Tue, 19 Jan 2016 19:58:57 -0500
Subject: [Bro] [bro]Intel Card Question
Message-ID: <CANrcqc3ONBhWCbjyCJT7xw+Qd_kbu+M5xN=Pd6vBMt88ZqkzVQ@mail.gmail.com>

I thought I'd push this out to the group.

I'm having issues with a bunch of Intel x520 10gig cards. I install the
newest ixgbe driver from the Intel page and run "modprobe -r ixgbe;
modprobe allow_unsupported_sfp=1,1" but I still cannot get the cards to
recognize any sfp's. I've used Cisco, Intel, and others with no luck.  I've
tried a number of cables as well.

The strangest part is, after a while the server seems to stop recognizing
the nic all together. I run IP link and all I get are eth0-eth3 not 4 or 5
(the 10gig card).

Anyone using these cards that would be able to assist me in getting them to
recognize an sfp and show link.

I've tried all I can think and using copper I'm dropping packets because am
mirroring 4, 1 gig ports to a single 10gig port out to my sensor.

Any thoughts/suggestions would be greatly appreciated.

Thanks
Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160119/701455fd/attachment.html 

From inetjunkmail at gmail.com  Wed Jan 20 05:35:20 2016
From: inetjunkmail at gmail.com (inetjunkmail)
Date: Wed, 20 Jan 2016 08:35:20 -0500
Subject: [Bro] [bro]Intel Card Question
In-Reply-To: <CANrcqc3ONBhWCbjyCJT7xw+Qd_kbu+M5xN=Pd6vBMt88ZqkzVQ@mail.gmail.com>
References: <CANrcqc3ONBhWCbjyCJT7xw+Qd_kbu+M5xN=Pd6vBMt88ZqkzVQ@mail.gmail.com>
Message-ID: <CAAJ38WXY2NpcF54G3njKM_W4UMswhoaP813R0j7d5zzRwmV9jg@mail.gmail.com>

We use CENTOS 7 for what that's worth and use:

# cat /etc/modprobe.d/ixgbe.conf

options ixgbe allow_unsupported_sfp=0,1 LRO=0

Using modprobe.d allows us to pass the parameters automatically on load.
Alternatively, it may have been a typo but you don't have ixgbe in your
second modprobe command.  I think it should be "modprobe -r ixgbe; modprobe
ixgbe allow_unsupported_sfp=1,1"

On Tue, Jan 19, 2016 at 7:58 PM, Tim Desrochers <tgdesrochers at gmail.com>
wrote:

> I thought I'd push this out to the group.
>
> I'm having issues with a bunch of Intel x520 10gig cards. I install the
> newest ixgbe driver from the Intel page and run "modprobe -r ixgbe;
> modprobe allow_unsupported_sfp=1,1" but I still cannot get the cards to
> recognize any sfp's. I've used Cisco, Intel, and others with no luck.  I've
> tried a number of cables as well.
>
> The strangest part is, after a while the server seems to stop recognizing
> the nic all together. I run IP link and all I get are eth0-eth3 not 4 or 5
> (the 10gig card).
>
> Anyone using these cards that would be able to assist me in getting them
> to recognize an sfp and show link.
>
> I've tried all I can think and using copper I'm dropping packets because
> am mirroring 4, 1 gig ports to a single 10gig port out to my sensor.
>
> Any thoughts/suggestions would be greatly appreciated.
>
> Thanks
> Tim
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160120/fb89289c/attachment.html 

From damian.gerow at shopify.com  Thu Jan 21 08:25:45 2016
From: damian.gerow at shopify.com (Damian Gerow)
Date: Thu, 21 Jan 2016 11:25:45 -0500
Subject: [Bro] Info on configuring bro inline in AWS as IDS
In-Reply-To: <BLUPR0501MB20660103210628DFDDB3E23ECFC10@BLUPR0501MB2066.namprd05.prod.outlook.com>
References: <BLUPR0501MB2066D1993A598629E8F7DC10CFC10@BLUPR0501MB2066.namprd05.prod.outlook.com>
	<CAPy2kFZaCdfq-d5==q+GH5A=wbSnzFf0HezYHa0DNm9STm_6qg@mail.gmail.com>
	<BLUPR0501MB20660103210628DFDDB3E23ECFC10@BLUPR0501MB2066.namprd05.prod.outlook.com>
Message-ID: <CAJ0FatTuZW9a_iAkUvachbKBjEQq1NSF8SAaosD55m21mG2zFA@mail.gmail.com>

Because AWS doesn't provide any concept of a tap, there are really only two
ways to run an IDS in AWS:

1. Have a central Bro worker node, and on each system where you want to
examine the traffic, set up a VPN tunnel with OpenVPN to your Bro node, and
use a tool called daemonlogger to sniff traffic off the ethernet interface
and send it out over the OpenVPN interface. On the Bro worker node, you run
one Bro instance per OpenVPN interface.
2. Directly on the nodes whose traffic you want to monitor, run a Bro
worker, configured to monitor its network interface.

(I recall reading a document from Amazon about setting up the former, but
I'm unable to find it right now.)


On Tue, Jan 19, 2016 at 3:43 PM, James Stallard <JStallard at enquizit.com>
wrote:

> Oh, thanks, Mike...
>
> Comments/responses posted below.
>
>
> ------------------------------
> *From:* Mike Dopheide <dopheide at gmail.com>
> *Sent:* Tuesday, January 19, 2016 3:32 PM
> *To:* James Stallard
> *Cc:* bro at bro.org
> *Subject:* Re: [Bro] Info on configuring bro inline in AWS as IDS
>
> I'm not very familiar with Amazon ELBs, but this is an interesting model
> so I have a couple clarifying questions to make sure we understand what
> you're trying to do
>
> 1)  So the model is ext_ELB -> Bro/router -> int_ELB,
> >>YES using Bro as an IPS rather than IDS?
> >>No, initially just an IDS
>   Are you planning multiple Bro instances to handle the load and provide
> failover?
> >>Yes, initially 2 loadbalanced behind ext_ELB.
>
> 2)  Bro, by itself, is not a routing engine.  It doesn't pass traffic out
> to another interface once it's done examining it.
> >>Ikes!
>
> If I understand what you're trying to do, you'd need to setup a software
> router (pfSense, Clickrouter, PacketBricks?, Microtik's RouterOS) have it
> mirror traffic to Bro, and then write Bro policies to inject rules into the
> router as needed.  I'm not sure if someone has already done it, but it
> wouldn't be an insignificant effort.
>
> (I believe Amazon supports a few virtual IPS appliances, like Palo Alto or
> Juniper as well.)
> >> OK, I'll touch base with our AWS contact  for this.
>
> >>Thanks for the tips.
>
> JMS
> -Dop
>
> On Tue, Jan 19, 2016 at 11:37 AM, James Stallard <JStallard at enquizit.com>
> wrote:
>
>> Hello Bros:
>>
>>
>> I'm just now installing bro for the government website at Small Business
>> Admin.
>>
>> The plan is to have bro behind our public ELBs as an in-line IDS, then
>> route traffic to internal ELBs in front of our application / web servers.
>>
>>
>> As this is AWS, no tap is possible and the EC2s can be run in promiscuous
>> mode either.
>>
>>
>> After a quick review of the documentation, I don't see where I can
>> configure the routing once bro has done its work.
>>
>>
>> I.E. if I configure:
>>
>>
>> bro -i en0 <list of scripts to load>
>>
>> do I need to then configure a script that will export all traffic to
>> another agent such as an ELB or nginx ?
>>
>>
>> Any help would be appreceated.
>>
>>
>> JMS
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160121/a3059223/attachment.html 

From bglaze at gmail.com  Thu Jan 21 08:39:15 2016
From: bglaze at gmail.com (Brandon Glaze)
Date: Thu, 21 Jan 2016 08:39:15 -0800
Subject: [Bro] Mellanox ConnectX-3 cards and Bro
Message-ID: <CAM5AJtZNERStU7rUuwa0wR+rPMpkhPm=Mzg=qHHk1rhUzkaK0w@mail.gmail.com>

Good morning all,

Has anyone worked with the advanced features on Mellanox ConnectX-3 cards?
Our servers are coming with dual 40G (QSFP) cards and I want to figure out
how best to use them. I am currently using the Netronome NFP cards with
great success, but want to experiment with these.



Appreciate any help or insight. I will send out what I learn during my
testing if helpful, but wanted to reuse any community lessons learned.
=====================
Brandon Glaze
bglaze at gmail.com

"Lead me, follow me, or get the hell out of my way."
- General George Patton Jr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160121/b37eff98/attachment.html 

From monahbaki at gmail.com  Thu Jan 21 09:19:38 2016
From: monahbaki at gmail.com (Monah Baki)
Date: Thu, 21 Jan 2016 12:19:38 -0500
Subject: [Bro] Critical Stack requirements
Message-ID: <CALP3=x-F=nDC9rbj8qU0qRH75KZ5TNbttJzsA6onWyroc2CJSg@mail.gmail.com>

Hi all,


Running SecurityOnion and trying to implement Criticial Stack with
Bro, server running 24GB RAM the system becomes unresponsive in 30
seconds. All memory and swap is utilized by then. Any documentation
that show sizing of Bro and Critical Stack?

If I remove criticalstack from local.bro, it's back to normal.

Thanks
Monah

From jlay at slave-tothe-box.net  Thu Jan 21 09:25:29 2016
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 21 Jan 2016 10:25:29 -0700
Subject: [Bro] Critical Stack requirements
In-Reply-To: <CALP3=x-F=nDC9rbj8qU0qRH75KZ5TNbttJzsA6onWyroc2CJSg@mail.gmail.com>
References: <CALP3=x-F=nDC9rbj8qU0qRH75KZ5TNbttJzsA6onWyroc2CJSg@mail.gmail.com>
Message-ID: <f06cf9bfb65f21ddf307790ece02bff7@localhost>

Sounds like you might be subscribed to a lot of lists...how big is your 
/opt/critical-stack/frameworks/intel/master-public.bro.dat file?  Mine 
is 9 megs.

James

On 2016-01-21 10:19, Monah Baki wrote:
> Hi all,
> 
> 
> Running SecurityOnion and trying to implement Criticial Stack with
> Bro, server running 24GB RAM the system becomes unresponsive in 30
> seconds. All memory and swap is utilized by then. Any documentation
> that show sizing of Bro and Critical Stack?
> 
> If I remove criticalstack from local.bro, it's back to normal.
> 
> Thanks
> Monah
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

From dopheide at gmail.com  Thu Jan 21 09:27:22 2016
From: dopheide at gmail.com (Mike Dopheide)
Date: Thu, 21 Jan 2016 11:27:22 -0600
Subject: [Bro] Critical Stack requirements
In-Reply-To: <CALP3=x-F=nDC9rbj8qU0qRH75KZ5TNbttJzsA6onWyroc2CJSg@mail.gmail.com>
References: <CALP3=x-F=nDC9rbj8qU0qRH75KZ5TNbttJzsA6onWyroc2CJSg@mail.gmail.com>
Message-ID: <CAPy2kFbrLanTDxgXGkDz2OAO7uyKTuZnS6_hWFZpjZ3Ht+w7zA@mail.gmail.com>

How many CriticalStack feeds are you subscribing to and against how much
bandwidth are you monitoring?

I've heard a rough recommendation that anything more than 100k indicators
can be pretty rough.  We run with 90k against an average 1G traffic without
any problems (14 workers).

-Dop

On Thu, Jan 21, 2016 at 11:19 AM, Monah Baki <monahbaki at gmail.com> wrote:

> Hi all,
>
>
> Running SecurityOnion and trying to implement Criticial Stack with
> Bro, server running 24GB RAM the system becomes unresponsive in 30
> seconds. All memory and swap is utilized by then. Any documentation
> that show sizing of Bro and Critical Stack?
>
> If I remove criticalstack from local.bro, it's back to normal.
>
> Thanks
> Monah
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160121/6fd9431b/attachment.html 

From monahbaki at gmail.com  Thu Jan 21 09:40:59 2016
From: monahbaki at gmail.com (Monah Baki)
Date: Thu, 21 Jan 2016 12:40:59 -0500
Subject: [Bro] Critical Stack requirements
In-Reply-To: <CAPy2kFbrLanTDxgXGkDz2OAO7uyKTuZnS6_hWFZpjZ3Ht+w7zA@mail.gmail.com>
References: <CALP3=x-F=nDC9rbj8qU0qRH75KZ5TNbttJzsA6onWyroc2CJSg@mail.gmail.com>
	<CAPy2kFbrLanTDxgXGkDz2OAO7uyKTuZnS6_hWFZpjZ3Ht+w7zA@mail.gmail.com>
Message-ID: <CALP3=x_6L-XZBO6roZF4v4=hwAtxJCx=furyj6HegN+JeYhQwg@mail.gmail.com>

I subscribed to bambenekconsulting.com-DGA-Domains and the
master-public.bro.dat is 132MB in size.

I went with the most popular feed, I am open to suggestions as to what
feed to subscribe. I am interested in CNC alerts and malicious sites.

We have a 150MB pipe to the internet and around 70 users in the office.

I am running 1 worker though.

Thanks


On Thu, Jan 21, 2016 at 12:27 PM, Mike Dopheide <dopheide at gmail.com> wrote:
> How many CriticalStack feeds are you subscribing to and against how much
> bandwidth are you monitoring?
>
> I've heard a rough recommendation that anything more than 100k indicators
> can be pretty rough.  We run with 90k against an average 1G traffic without
> any problems (14 workers).
>
> -Dop
>
> On Thu, Jan 21, 2016 at 11:19 AM, Monah Baki <monahbaki at gmail.com> wrote:
>>
>> Hi all,
>>
>>
>> Running SecurityOnion and trying to implement Criticial Stack with
>> Bro, server running 24GB RAM the system becomes unresponsive in 30
>> seconds. All memory and swap is utilized by then. Any documentation
>> that show sizing of Bro and Critical Stack?
>>
>> If I remove criticalstack from local.bro, it's back to normal.
>>
>> Thanks
>> Monah
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>

From paul.halliday at gmail.com  Thu Jan 21 10:06:47 2016
From: paul.halliday at gmail.com (Paul Halliday)
Date: Thu, 21 Jan 2016 14:06:47 -0400
Subject: [Bro] Critical Stack requirements
In-Reply-To: <CALP3=x_6L-XZBO6roZF4v4=hwAtxJCx=furyj6HegN+JeYhQwg@mail.gmail.com>
References: <CALP3=x-F=nDC9rbj8qU0qRH75KZ5TNbttJzsA6onWyroc2CJSg@mail.gmail.com>
	<CAPy2kFbrLanTDxgXGkDz2OAO7uyKTuZnS6_hWFZpjZ3Ht+w7zA@mail.gmail.com>
	<CALP3=x_6L-XZBO6roZF4v4=hwAtxJCx=furyj6HegN+JeYhQwg@mail.gmail.com>
Message-ID: <CAJfn-RFicYKt_wzVzV7Yy97TA9pmN7sd2_L6RbfLW3j+tYZrgw@mail.gmail.com>

Mike is correct.

When you create a collection the status indicator will actually warn your
if the collection has too many indicators.

Try sorting the feeds by "Most Subscribers" and cherry pick from there. You
can also try searching for terms like C&C, Botnet, Malware, Malicious etc
via the search box at the top of the feeds page.

As for why have a feed with 700K+ indicators? There are quite a few folks
that use the feeds outside of tools like Bro that consume and use all
available indicators.

HTH.


On Thu, Jan 21, 2016 at 1:40 PM, Monah Baki <monahbaki at gmail.com> wrote:

> I subscribed to bambenekconsulting.com-DGA-Domains and the
> master-public.bro.dat is 132MB in size.
>
> I went with the most popular feed, I am open to suggestions as to what
> feed to subscribe. I am interested in CNC alerts and malicious sites.
>
> We have a 150MB pipe to the internet and around 70 users in the office.
>
> I am running 1 worker though.
>
> Thanks
>
>
> On Thu, Jan 21, 2016 at 12:27 PM, Mike Dopheide <dopheide at gmail.com>
> wrote:
> > How many CriticalStack feeds are you subscribing to and against how much
> > bandwidth are you monitoring?
> >
> > I've heard a rough recommendation that anything more than 100k indicators
> > can be pretty rough.  We run with 90k against an average 1G traffic
> without
> > any problems (14 workers).
> >
> > -Dop
> >
> > On Thu, Jan 21, 2016 at 11:19 AM, Monah Baki <monahbaki at gmail.com>
> wrote:
> >>
> >> Hi all,
> >>
> >>
> >> Running SecurityOnion and trying to implement Criticial Stack with
> >> Bro, server running 24GB RAM the system becomes unresponsive in 30
> >> seconds. All memory and swap is utilized by then. Any documentation
> >> that show sizing of Bro and Critical Stack?
> >>
> >> If I remove criticalstack from local.bro, it's back to normal.
> >>
> >> Thanks
> >> Monah
> >> _______________________________________________
> >> Bro mailing list
> >> bro at bro-ids.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >
> >
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 
Paul Halliday
http://www.pintumbler.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160121/6fe30439/attachment.html 

From liam.randall at gmail.com  Thu Jan 21 10:13:01 2016
From: liam.randall at gmail.com (Liam Randall)
Date: Thu, 21 Jan 2016 13:13:01 -0500
Subject: [Bro] Critical Stack requirements
In-Reply-To: <CALP3=x_6L-XZBO6roZF4v4=hwAtxJCx=furyj6HegN+JeYhQwg@mail.gmail.com>
References: <CALP3=x-F=nDC9rbj8qU0qRH75KZ5TNbttJzsA6onWyroc2CJSg@mail.gmail.com>
	<CAPy2kFbrLanTDxgXGkDz2OAO7uyKTuZnS6_hWFZpjZ3Ht+w7zA@mail.gmail.com>
	<CALP3=x_6L-XZBO6roZF4v4=hwAtxJCx=furyj6HegN+JeYhQwg@mail.gmail.com>
Message-ID: <CAC8+5ePU-H3dyXJKGEvkSqy1mhpT_48OOe27X0v6irEq+ucfxg@mail.gmail.com>

Hey Baki,

Using the "Metrics" tab you can analyze the size in "count" of indicators
by collection over time.

You may want to limit your deployments to between 100-200k indicators
depending on cluster size, traffic, traffic types, etc.

There are three bambenek feeds available:
-- precomputed dga feed (900k + elements)
-- C&C IPs (260+)
-- C&C Domains (330+)

Try building a collection with fewer items on it and then issuing an update.

If you look under your "collections" tab the "status" column will give you
some feedback about the size of your collection.


Please feel free to open a ticket with us directly if you have any further
problems.

V/r,

Liam Randall








On Thu, Jan 21, 2016 at 12:40 PM, Monah Baki <monahbaki at gmail.com> wrote:

> I subscribed to bambenekconsulting.com-DGA-Domains and the
> master-public.bro.dat is 132MB in size.
>
> I went with the most popular feed, I am open to suggestions as to what
> feed to subscribe. I am interested in CNC alerts and malicious sites.
>
> We have a 150MB pipe to the internet and around 70 users in the office.
>
> I am running 1 worker though.
>
> Thanks
>
>
> On Thu, Jan 21, 2016 at 12:27 PM, Mike Dopheide <dopheide at gmail.com>
> wrote:
> > How many CriticalStack feeds are you subscribing to and against how much
> > bandwidth are you monitoring?
> >
> > I've heard a rough recommendation that anything more than 100k indicators
> > can be pretty rough.  We run with 90k against an average 1G traffic
> without
> > any problems (14 workers).
> >
> > -Dop
> >
> > On Thu, Jan 21, 2016 at 11:19 AM, Monah Baki <monahbaki at gmail.com>
> wrote:
> >>
> >> Hi all,
> >>
> >>
> >> Running SecurityOnion and trying to implement Criticial Stack with
> >> Bro, server running 24GB RAM the system becomes unresponsive in 30
> >> seconds. All memory and swap is utilized by then. Any documentation
> >> that show sizing of Bro and Critical Stack?
> >>
> >> If I remove criticalstack from local.bro, it's back to normal.
> >>
> >> Thanks
> >> Monah
> >> _______________________________________________
> >> Bro mailing list
> >> bro at bro-ids.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >
> >
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160121/025bdc14/attachment.html 

From hosom at battelle.org  Fri Jan 22 08:57:35 2016
From: hosom at battelle.org (Hosom, Stephen M)
Date: Fri, 22 Jan 2016 16:57:35 +0000
Subject: [Bro] Critical Stack requirements
In-Reply-To: <CAC8+5ePU-H3dyXJKGEvkSqy1mhpT_48OOe27X0v6irEq+ucfxg@mail.gmail.com>
References: <CALP3=x-F=nDC9rbj8qU0qRH75KZ5TNbttJzsA6onWyroc2CJSg@mail.gmail.com>
	<CAPy2kFbrLanTDxgXGkDz2OAO7uyKTuZnS6_hWFZpjZ3Ht+w7zA@mail.gmail.com>
	<CALP3=x_6L-XZBO6roZF4v4=hwAtxJCx=furyj6HegN+JeYhQwg@mail.gmail.com>
	<CAC8+5ePU-H3dyXJKGEvkSqy1mhpT_48OOe27X0v6irEq+ucfxg@mail.gmail.com>
Message-ID: <E5A2071C6FFFF640B714C224139C9B3327C4FFA5@WP-MBX2B.milky-way.battelle.org>

Monah,

I don?t think that your subscriptions to intel feeds are what is causing this issue. I wouldn?t expect intel feeds to expand to fill that much memory space unless it were a truly massive amount of intel. You can certainly try reducing your subscriptions?I?ve definitely been wrong before. However, you should also check out a few other things:

How are you measuring memory utilization? The output of the linux ?free? command can be confusing to new users.
How big of a link are you trying to monitor?
Have you loaded any custom scripts into your Bro instance? It can be easy to fill a large amount of memory with Bro scripts.
What else is running on your Security Onion instance? A few of the tools distributed in Security Onion can be quite memory hungry.

Best of luck,

Stephen

From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Liam Randall
Sent: Thursday, January 21, 2016 1:13 PM
To: Monah Baki <monahbaki at gmail.com>
Cc: Bro-IDS <bro at bro.org>
Subject: Re: [Bro] Critical Stack requirements

Hey Baki,

Using the "Metrics" tab you can analyze the size in "count" of indicators by collection over time.

You may want to limit your deployments to between 100-200k indicators depending on cluster size, traffic, traffic types, etc.

There are three bambenek feeds available:
-- precomputed dga feed (900k + elements)
-- C&C IPs (260+)
-- C&C Domains (330+)

Try building a collection with fewer items on it and then issuing an update.

If you look under your "collections" tab the "status" column will give you some feedback about the size of your collection.


Please feel free to open a ticket with us directly if you have any further problems.

V/r,

Liam Randall








On Thu, Jan 21, 2016 at 12:40 PM, Monah Baki <monahbaki at gmail.com<mailto:monahbaki at gmail.com>> wrote:
I subscribed to bambenekconsulting.com-DGA-Domains and the
master-public.bro.dat is 132MB in size.

I went with the most popular feed, I am open to suggestions as to what
feed to subscribe. I am interested in CNC alerts and malicious sites.

We have a 150MB pipe to the internet and around 70 users in the office.

I am running 1 worker though.

Thanks


On Thu, Jan 21, 2016 at 12:27 PM, Mike Dopheide <dopheide at gmail.com<mailto:dopheide at gmail.com>> wrote:
> How many CriticalStack feeds are you subscribing to and against how much
> bandwidth are you monitoring?
>
> I've heard a rough recommendation that anything more than 100k indicators
> can be pretty rough.  We run with 90k against an average 1G traffic without
> any problems (14 workers).
>
> -Dop
>
> On Thu, Jan 21, 2016 at 11:19 AM, Monah Baki <monahbaki at gmail.com<mailto:monahbaki at gmail.com>> wrote:
>>
>> Hi all,
>>
>>
>> Running SecurityOnion and trying to implement Criticial Stack with
>> Bro, server running 24GB RAM the system becomes unresponsive in 30
>> seconds. All memory and swap is utilized by then. Any documentation
>> that show sizing of Bro and Critical Stack?
>>
>> If I remove criticalstack from local.bro, it's back to normal.
>>
>> Thanks
>> Monah
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org<mailto:bro at bro-ids.org>
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160122/dd67cdb7/attachment.html 

From baxter.milliwew at gmail.com  Mon Jan 25 08:27:43 2016
From: baxter.milliwew at gmail.com (Baxter Milliwew)
Date: Mon, 25 Jan 2016 08:27:43 -0800
Subject: [Bro] Bro Packet Loss / 10gb ixgbe / pf_ring
In-Reply-To: <568FF114.7080208@gmail.com>
References: <723D2963-E446-4251-ADF9-BEB7BE321202@tufts.edu>
	<41AA3B68-CF28-40D7-A38D-C7D1B8C1591D@icir.org>
	<FF4F5A27-8001-4393-87B7-E7C50F338FBF@tufts.edu>
	<568FF114.7080208@gmail.com>
Message-ID: <CA+J8G6HC1SogrEniMkDA1MRQW1hd2LW7a6pbNC_cQhEMiqR3hQ@mail.gmail.com>

PF_RING ZC works, you have to set the ClusterID as the single interface in
node.cfg (zc:2).  What you are missing is the PF_RING ancillary application
zbalance_ipc to PCAP and fanout to virtual interfaces.  You need to read
the pf_ring docs for that as you'll also need hugepages support.

Also you do not need or want to use RSS with ZC so disable that by
configuring a single queue.

For example: Start zbalance_ipc PCAP from ZC interface eth4 with a
clusterID of 2 and 16 virtual interfaces, bind the app to the NIC attached
NUMA core #3, use IP hashing, set a queue size that won't exhaust local
NUMA memory.

example:  zbalance_ipc -i zc:eth4 -c 2 -n 16 -g 3 -m 1 -q 4096  then start
bro

As someone else mentioned the current pf_ring compatibility requires an
entire cluster restart when a worker crashes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160125/ea83066e/attachment.html 

From jlay at slave-tothe-box.net  Mon Jan 25 13:39:56 2016
From: jlay at slave-tothe-box.net (James Lay)
Date: Mon, 25 Jan 2016 14:39:56 -0700
Subject: [Bro] NIC recommends
Message-ID: <f5a61135330be92066c8d0335678dc30@localhost>

Hey All,

Topic says it....sizing up for capturing LAN traffic...thinking a 40 or 
even 100 gig nic.  This would be a single machine doing nothing else but 
bro.  Thanks for any assistance.

James

From bglaze at gmail.com  Mon Jan 25 14:21:14 2016
From: bglaze at gmail.com (Brandon Glaze)
Date: Mon, 25 Jan 2016 14:21:14 -0800
Subject: [Bro] NIC recommends
In-Reply-To: <f5a61135330be92066c8d0335678dc30@localhost>
References: <f5a61135330be92066c8d0335678dc30@localhost>
Message-ID: <CAM5AJtZ=ZNyA4WUqGqpPRGS-ZsEeOVwwhzRUcipetxNjQwsxNA@mail.gmail.com>

James,
I have had really good luck with the Netronome NFE-3240 NIC's. I have only
run them to 20Gbps each. Their NFP-6xxx series have multiple 100Gb or 40Gb
interfaces, and their support is awesome (they helped me build my Bro
servers initially).

The Berkeley Lab 100Gb bro sensors (search for 100GIntrusionDetection) used
Myricom 10GPCIE28C22+ cards, and their PDF has all the configs (plus its a
great practical research and proof of concept paper). I am sure Myricom has
40Gb cards, but you would have to research it.

=====================
Brandon Glaze
bglaze at gmail.com

"Lead me, follow me, or get the hell out of my way."
- General George Patton Jr


On Mon, Jan 25, 2016 at 1:39 PM, James Lay <jlay at slave-tothe-box.net> wrote:

> Hey All,
>
> Topic says it....sizing up for capturing LAN traffic...thinking a 40 or
> even 100 gig nic.  This would be a single machine doing nothing else but
> bro.  Thanks for any assistance.
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160125/a8f64428/attachment.html 

From jlay at slave-tothe-box.net  Mon Jan 25 14:30:08 2016
From: jlay at slave-tothe-box.net (James Lay)
Date: Mon, 25 Jan 2016 15:30:08 -0700
Subject: [Bro] NIC recommends
In-Reply-To: <CAM5AJtZ=ZNyA4WUqGqpPRGS-ZsEeOVwwhzRUcipetxNjQwsxNA@mail.gmail.com>
References: <f5a61135330be92066c8d0335678dc30@localhost>
	<CAM5AJtZ=ZNyA4WUqGqpPRGS-ZsEeOVwwhzRUcipetxNjQwsxNA@mail.gmail.com>
Message-ID: <829c3fa8d88f0b247bce382ee8359ce3@localhost>

 

On 2016-01-25 15:21, Brandon Glaze wrote: 

> James, I have had really good luck with the Netronome NFE-3240 NIC's. I have only run them to 20Gbps each. Their NFP-6xxx series have multiple 100Gb or 40Gb interfaces, and their support is awesome (they helped me build my Bro servers initially).
> 
> The Berkeley Lab 100Gb bro sensors (search for 100GIntrusionDetection) used Myricom 10GPCIE28C22+ cards, and their PDF has all the configs (plus its a great practical research and proof of concept paper). I am sure Myricom has 40Gb cards, but you would have to research it. 
> 
> =====================
> Brandon Glaze
> bglaze at gmail.com
> 
> "Lead me, follow me, or get the hell out of my way." 
> - General George Patton Jr 
> 
> On Mon, Jan 25, 2016 at 1:39 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> 
>> Hey All,
>> 
>> Topic says it....sizing up for capturing LAN traffic...thinking a 40 or
>> even 100 gig nic.  This would be a single machine doing nothing else but
>> bro.  Thanks for any assistance.
>> 
>> James
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

Wonderful information...thanks so much. 

James 
  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160125/9bbc6138/attachment.html 

From michalpurzynski1 at gmail.com  Mon Jan 25 14:50:07 2016
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Mon, 25 Jan 2016 23:50:07 +0100
Subject: [Bro] NIC recommends
In-Reply-To: <CAM5AJtZ=ZNyA4WUqGqpPRGS-ZsEeOVwwhzRUcipetxNjQwsxNA@mail.gmail.com>
References: <f5a61135330be92066c8d0335678dc30@localhost>
	<CAM5AJtZ=ZNyA4WUqGqpPRGS-ZsEeOVwwhzRUcipetxNjQwsxNA@mail.gmail.com>
Message-ID: <5B0A66A4-7FFA-4D10-8EFE-0FDCF3C80FBC@gmail.com>

We have been running Bro sensors with multiple Myricom cards here, 10Gbit per sensor (it's the interface speed, not the actual traffic) with the Sniffer v3 software. Rock solid, with a few minor problems from time to time. Traffic is load balanced with the Arista switch in tap aggregator mode.





Sent from my iPhone
> On 25 Jan 2016, at 23:21, Brandon Glaze <bglaze at gmail.com> wrote:
> 
> James,
> I have had really good luck with the Netronome NFE-3240 NIC's. I have only run them to 20Gbps each. Their NFP-6xxx series have multiple 100Gb or 40Gb interfaces, and their support is awesome (they helped me build my Bro servers initially).
> 
> The Berkeley Lab 100Gb bro sensors (search for 100GIntrusionDetection) used Myricom 10GPCIE28C22+ cards, and their PDF has all the configs (plus its a great practical research and proof of concept paper). I am sure Myricom has 40Gb cards, but you would have to research it.
> 
> =====================
> Brandon Glaze
> bglaze at gmail.com
> 
> "Lead me, follow me, or get the hell out of my way." 
> - General George Patton Jr
> 
> 
> 
>> On Mon, Jan 25, 2016 at 1:39 PM, James Lay <jlay at slave-tothe-box.net> wrote:
>> Hey All,
>> 
>> Topic says it....sizing up for capturing LAN traffic...thinking a 40 or
>> even 100 gig nic.  This would be a single machine doing nothing else but
>> bro.  Thanks for any assistance.
>> 
>> James
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160125/1a568040/attachment.html 

From jlay at slave-tothe-box.net  Tue Jan 26 04:05:04 2016
From: jlay at slave-tothe-box.net (James Lay)
Date: Tue, 26 Jan 2016 05:05:04 -0700
Subject: [Bro] NIC recommends
In-Reply-To: <5B0A66A4-7FFA-4D10-8EFE-0FDCF3C80FBC@gmail.com>
References: <f5a61135330be92066c8d0335678dc30@localhost>
	<CAM5AJtZ=ZNyA4WUqGqpPRGS-ZsEeOVwwhzRUcipetxNjQwsxNA@mail.gmail.com>
	<5B0A66A4-7FFA-4D10-8EFE-0FDCF3C80FBC@gmail.com>
Message-ID: <1453809904.3400.2.camel@gamebox>

On Mon, 2016-01-25 at 23:50 +0100, Micha? Purzy?ski wrote:
> We have been running Bro sensors with multiple Myricom cards here,
> 10Gbit per sensor (it's the interface speed, not the actual traffic)
> with the Sniffer v3 software. Rock solid, with a few minor problems
> from time to time. Traffic is load balanced with the Arista switch in
> tap aggregator mode.
> 
> 
> 
> 
> 
> 
> 
> 
> Sent from my iPhone
> 
> On 25 Jan 2016, at 23:21, Brandon Glaze <bglaze at gmail.com> wrote:
> 
> 
> > James,
> > 
> > 
> > I have had really good luck with the Netronome NFE-3240 NIC's. I
> > have only run them to 20Gbps each. Their NFP-6xxx series have
> > multiple 100Gb or 40Gb interfaces, and their support is awesome
> > (they helped me build my Bro servers initially).
> > 
> > 
> > 
> > The Berkeley Lab 100Gb bro sensors (search for
> > 100GIntrusionDetection) used Myricom 10GPCIE28C22+ cards, and their
> > PDF has all the configs (plus its a great practical research and
> > proof of concept paper). I am sure Myricom has 40Gb cards, but you
> > would have to research it.
> > 
> > 
> > 
> > =====================
> > Brandon Glaze
> > bglaze at gmail.com
> > 
> > 
> > 
> > "Lead me, follow me, or get the hell out of my way." 
> > - General George Patton Jr
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > On Mon, Jan 25, 2016 at 1:39 PM, James Lay
> > <jlay at slave-tothe-box.net> wrote:
> > 
> >         Hey All,
> >         
> >         Topic says it....sizing up for capturing LAN
> >         traffic...thinking a 40 or
> >         even 100 gig nic.  This would be a single machine doing
> >         nothing else but
> >         bro.  Thanks for any assistance.
> >         
> >         James
> >         _______________________________________________
> >         Bro mailing list
> >         bro at bro-ids.org
> >         http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> > 
> > 
> > 
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

Thank you Michat...I will also take a look at those cards.

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160126/5c348495/attachment-0001.html 

From jlay at slave-tothe-box.net  Tue Jan 26 09:44:47 2016
From: jlay at slave-tothe-box.net (James Lay)
Date: Tue, 26 Jan 2016 10:44:47 -0700
Subject: [Bro] Hardware recommends
Message-ID: <8c48a723fee2d6b3c3db92472e26d096@localhost>

And on the heels of the NIC question, how about hardware experiences?  
I'm looking at the PCIE2 NIC's at both Myricom and Netronome....any 
recommends for the server hardware to wrap around these cards?  The plan 
is to have this machine monitor a corporate LAN...lot's of traffic.  
Guessing the team will want to go Dell if that helps.  Thanks for the 
advice all.

James

From latt0050 at umn.edu  Tue Jan 26 12:01:07 2016
From: latt0050 at umn.edu (Brandon Lattin)
Date: Tue, 26 Jan 2016 14:01:07 -0600
Subject: [Bro] Hardware recommends
In-Reply-To: <8c48a723fee2d6b3c3db92472e26d096@localhost>
References: <8c48a723fee2d6b3c3db92472e26d096@localhost>
Message-ID: <CAJWRZ8TsY8X6AJOdTW9-ONRy11vkBdD2yQDFQGB_9jJ7R+mU3A@mail.gmail.com>

We've been doing the following:

Dell R630
2x Intel? Xeon? E5?2687W v3
3.1GHz,25MCache,9.60GT/sQPI,Turbo,HT,10C/20T(160W)
128GB RAM
With whatever disk fits your needs. Our worker boxes are a mirrored pair of
120GB SSD. The manager node has slightly larger disk to handle 12h of
storage. A Splunk forwarder ingests from the manager box for
retention/analysis.

Most of this is in 'dev' right now, but we'll be run around 7x 100GB sets
by the end of the year following the Berkley model. Post-shunting we'll be
running Suricata on the traffic as well.

As a general rule, faster proc > more procs (Seth correct me here if this
has changed!)

On Tue, Jan 26, 2016 at 11:44 AM, James Lay <jlay at slave-tothe-box.net>
wrote:

> And on the heels of the NIC question, how about hardware experiences?
> I'm looking at the PCIE2 NIC's at both Myricom and Netronome....any
> recommends for the server hardware to wrap around these cards?  The plan
> is to have this machine monitor a corporate LAN...lot's of traffic.
> Guessing the team will want to go Dell if that helps.  Thanks for the
> advice all.
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 
Brandon Lattin
Security Analyst
University of Minnesota - University Information Security
Office: 612-626-6672
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160126/ca048f63/attachment.html 

From Paul.Nash at tufts.edu  Tue Jan 26 12:12:32 2016
From: Paul.Nash at tufts.edu (Nash, Paul)
Date: Tue, 26 Jan 2016 20:12:32 +0000
Subject: [Bro] Hardware recommends
In-Reply-To: <8c48a723fee2d6b3c3db92472e26d096@localhost>
References: <8c48a723fee2d6b3c3db92472e26d096@localhost>
Message-ID: <0EB912B4-7090-48DD-A1A6-7AD984241971@tufts.edu>


Of note - we ran into some problems with an AMD setup.  Others can chime in with more detail, but the synopsis was that the AMD cores run at a slower clock speed and just can?t keep up with the amount of data coming in.  For reference, it was a 64core AMD Opteron system w/128gb of ram.  I?m in the process of horse trading the AMD box for a 32core Intel Xeon system that should be able to better keep up with the data coming in.  We were pushing ~1.2gbs and the manager couldn?t keep up and would slowly consume all memory.

 -Paul




On 1/26/16, 12:44 PM, "bro-bounces at bro.org on behalf of James Lay" <bro-bounces at bro.org on behalf of jlay at slave-tothe-box.net> wrote:

>And on the heels of the NIC question, how about hardware experiences?  
>I'm looking at the PCIE2 NIC's at both Myricom and Netronome....any 
>recommends for the server hardware to wrap around these cards?  The plan 
>is to have this machine monitor a corporate LAN...lot's of traffic.  
>Guessing the team will want to go Dell if that helps.  Thanks for the 
>advice all.
>
>James
>_______________________________________________
>Bro mailing list
>bro at bro-ids.org
>http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From jlay at slave-tothe-box.net  Tue Jan 26 12:34:27 2016
From: jlay at slave-tothe-box.net (James Lay)
Date: Tue, 26 Jan 2016 13:34:27 -0700
Subject: [Bro] Hardware recommends
In-Reply-To: <8c48a723fee2d6b3c3db92472e26d096@localhost>
References: <8c48a723fee2d6b3c3db92472e26d096@localhost>
Message-ID: <3a83b02c33da6d78f350e84d4a012a33@localhost>

On 2016-01-26 10:44, James Lay wrote:
> And on the heels of the NIC question, how about hardware experiences?
> I'm looking at the PCIE2 NIC's at both Myricom and Netronome....any
> recommends for the server hardware to wrap around these cards?  The 
> plan
> is to have this machine monitor a corporate LAN...lot's of traffic.
> Guessing the team will want to go Dell if that helps.  Thanks for the
> advice all.
> 
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

Thanks for the great information all..it really does help.

James

From gfaulkner.nsm at gmail.com  Tue Jan 26 19:16:35 2016
From: gfaulkner.nsm at gmail.com (Gary Faulkner)
Date: Tue, 26 Jan 2016 21:16:35 -0600
Subject: [Bro] Hardware recommends
In-Reply-To: <3a83b02c33da6d78f350e84d4a012a33@localhost>
References: <8c48a723fee2d6b3c3db92472e26d096@localhost>
	<3a83b02c33da6d78f350e84d4a012a33@localhost>
Message-ID: <56A83693.10908@gmail.com>

The Bro architecture documents still seem to suggest you can only
process 80Mb/s or so of traffic per core, but even at 2.6 - 2.7 GHz you
end up getting closer to 250-300Mb/s+. 3.1 will boost this a bit and
allow you to handle slightly larger flows per core, but you may be able
to get many more cores on a single host at 2.6 for similar or less
money. I'd just be ware of optimizing too heavily on core count and
ending up with 1.8GHz clocks. If you are doing good flow shunting up
front I think you are likely to stray into the territory of more smaller
flows which probably lends itself better to having more moderately
clocked cores than fewer slightly higher clocked cores.

Also, unless you are considering ludicrous core counts per machine you
might find you are oversubscribing your server long before you can take
advantage of 40Gbps or 100Gbps NICs over a 10Gbps NIC. I've been fairly
happy with Intel 10Gbps NICs and PF_RING DNA/ZC, but some prefer to use
Myricom to avoid dealing with third party Intel NIC drivers. Be wary of
artificial worker limits using RSS or vendor provided host based
load-balancing (Myricom comes to mind). There are cases where folks have
been stuck with not being able to take full advantage of their server
hardware without having to run additional NICs, or do other work-arounds
due to having more cores than queues/rings on the cards.

On a side note: I found out a lot of interesting things about how my
sensors were performing, as well as my upstream load-balancer by using
Justin's statsd plugin (assuming your upstream shunting doesn't throw
off the output) to send the capture-loss script output to a time series
DB and graphing it. For example I discovered a port going to an
unrelated tool was becoming oversubscribed over the lunch hour causing
back-pressure on the load-balancer that translated to every worker on my
Bro cluster reporting 25-50% loss even though Bro should have been
seeing relatively little traffic and was itself not over-subscribed. In
that case I found it is sometimes desirable to have an extra 10G NIC in
each server so that the tool, not the load-balancer gets over-subscribed
until I can add more capacity to the tool and better spread the load.

On 1/26/2016 2:34 PM, James Lay wrote:
> On 2016-01-26 10:44, James Lay wrote:
>> And on the heels of the NIC question, how about hardware experiences?
>> I'm looking at the PCIE2 NIC's at both Myricom and Netronome....any
>> recommends for the server hardware to wrap around these cards?  The 
>> plan
>> is to have this machine monitor a corporate LAN...lot's of traffic.
>> Guessing the team will want to go Dell if that helps.  Thanks for the
>> advice all.
>>
>> James
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> Thanks for the great information all..it really does help.
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From seth at icir.org  Tue Jan 26 21:17:41 2016
From: seth at icir.org (Seth Hall)
Date: Wed, 27 Jan 2016 00:17:41 -0500
Subject: [Bro] Hardware recommends
In-Reply-To: <CAJWRZ8TsY8X6AJOdTW9-ONRy11vkBdD2yQDFQGB_9jJ7R+mU3A@mail.gmail.com>
References: <8c48a723fee2d6b3c3db92472e26d096@localhost>
	<CAJWRZ8TsY8X6AJOdTW9-ONRy11vkBdD2yQDFQGB_9jJ7R+mU3A@mail.gmail.com>
Message-ID: <4B7EB6C4-7371-4CCA-BC90-E484D8192352@icir.org>


> On Jan 26, 2016, at 3:01 PM, Brandon Lattin <latt0050 at umn.edu> wrote:
> 
> As a general rule, faster proc > more procs (Seth correct me here if this has changed!)

Hah!  Not really, it's pretty much still that as long as you still have a lot of cores.  Like most things in life, you're searching for balance here. :)

If I could order systems with 400 Xeon cores though, I'd be all over that!

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From seth at icir.org  Tue Jan 26 21:19:58 2016
From: seth at icir.org (Seth Hall)
Date: Wed, 27 Jan 2016 00:19:58 -0500
Subject: [Bro] Hardware recommends
In-Reply-To: <56A83693.10908@gmail.com>
References: <8c48a723fee2d6b3c3db92472e26d096@localhost>
	<3a83b02c33da6d78f350e84d4a012a33@localhost>
	<56A83693.10908@gmail.com>
Message-ID: <F16C82B7-2450-4731-B685-A0C96295693F@icir.org>


> On Jan 26, 2016, at 10:16 PM, Gary Faulkner <gfaulkner.nsm at gmail.com> wrote:
> 
> On a side note: I found out a lot of interesting things about how my
> sensors were performing, as well as my upstream load-balancer by using
> Justin's statsd plugin (assuming your upstream shunting doesn't throw
> off the output) to send the capture-loss script output to a time series
> DB and graphing it. 

I'm working on improving the stats output of Bro now too so the 2.5 version will have lots of internal details in the stats.log that should provide a much better picture for people to see what's going on in their clusters (a couple of people have already sent me data and graphs which has been super exciting!).  Capture loss is cool, but there is so much more data available that can really help you get a deeper understanding of what Bro is doing while it's running.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From mohamed.mostafa at consistec.de  Wed Jan 27 04:08:37 2016
From: mohamed.mostafa at consistec.de (Mohamed Mostafa)
Date: Wed, 27 Jan 2016 12:08:37 +0000
Subject: [Bro] [BinPac] Splitting generated code files
Message-ID: <1453896517.13474.6.camel@consistec.de>

Hallo,
I would like to know if it is possible to split the generated header/cpp
files (header files more important) into multiple files. For example, I
would like to have a separate generated header file for the record types
so that I can include it in a separate project.

Regards,
Mostafa.


From slagell at illinois.edu  Wed Jan 27 04:57:58 2016
From: slagell at illinois.edu (Slagell, Adam J)
Date: Wed, 27 Jan 2016 12:57:58 +0000
Subject: [Bro] Hardware recommends
In-Reply-To: <56A83693.10908@gmail.com>
References: <8c48a723fee2d6b3c3db92472e26d096@localhost>
	<3a83b02c33da6d78f350e84d4a012a33@localhost>, <56A83693.10908@gmail.com>
Message-ID: <B789C38C-9309-49BA-8D6B-D83A201F04DB@illinois.edu>

Which document? We should update that. 

> On Jan 26, 2016, at 9:18 PM, Gary Faulkner <gfaulkner.nsm at gmail.com> wrote:
> 
> The Bro architecture documents still seem to suggest you can only
> process 80Mb/s or so of traffic per core, but even at 2.6 - 2.7 GHz you
> end up getting closer to 250-300Mb/s+. 3.1 will boost this a bit and
> allow you to handle slightly larger flows per core, but you may be able
> to get many more cores on a single host at 2.6 for similar or less
> money. I'd just be ware of optimizing too heavily on core count and
> ending up with 1.8GHz clocks. If you are doing good flow shunting up
> front I think you are likely to stray into the territory of more smaller
> flows which probably lends itself better to having more moderately
> clocked cores than fewer slightly higher clocked cores.
> 
> Also, unless you are considering ludicrous core counts per machine you
> might find you are oversubscribing your server long before you can take
> advantage of 40Gbps or 100Gbps NICs over a 10Gbps NIC. I've been fairly
> happy with Intel 10Gbps NICs and PF_RING DNA/ZC, but some prefer to use
> Myricom to avoid dealing with third party Intel NIC drivers. Be wary of
> artificial worker limits using RSS or vendor provided host based
> load-balancing (Myricom comes to mind). There are cases where folks have
> been stuck with not being able to take full advantage of their server
> hardware without having to run additional NICs, or do other work-arounds
> due to having more cores than queues/rings on the cards.
> 
> On a side note: I found out a lot of interesting things about how my
> sensors were performing, as well as my upstream load-balancer by using
> Justin's statsd plugin (assuming your upstream shunting doesn't throw
> off the output) to send the capture-loss script output to a time series
> DB and graphing it. For example I discovered a port going to an
> unrelated tool was becoming oversubscribed over the lunch hour causing
> back-pressure on the load-balancer that translated to every worker on my
> Bro cluster reporting 25-50% loss even though Bro should have been
> seeing relatively little traffic and was itself not over-subscribed. In
> that case I found it is sometimes desirable to have an extra 10G NIC in
> each server so that the tool, not the load-balancer gets over-subscribed
> until I can add more capacity to the tool and better spread the load.
> 
>> On 1/26/2016 2:34 PM, James Lay wrote:
>>> On 2016-01-26 10:44, James Lay wrote:
>>> And on the heels of the NIC question, how about hardware experiences?
>>> I'm looking at the PCIE2 NIC's at both Myricom and Netronome....any
>>> recommends for the server hardware to wrap around these cards?  The 
>>> plan
>>> is to have this machine monitor a corporate LAN...lot's of traffic.
>>> Guessing the team will want to go Dell if that helps.  Thanks for the
>>> advice all.
>>> 
>>> James
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> Thanks for the great information all..it really does help.
>> 
>> James
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From robin at icir.org  Wed Jan 27 07:56:05 2016
From: robin at icir.org (Robin Sommer)
Date: Wed, 27 Jan 2016 07:56:05 -0800
Subject: [Bro] [BinPac] Splitting generated code files
In-Reply-To: <1453896517.13474.6.camel@consistec.de>
References: <1453896517.13474.6.camel@consistec.de>
Message-ID: <20160127155605.GC61143@icir.org>



On Wed, Jan 27, 2016 at 12:08 +0000, Mohamed Mostafa wrote:

> files (header files more important) into multiple files. For example, I
> would like to have a separate generated header file for the record types
> so that I can include it in a separate project.

No, not really, the only control you have is splitting up the input
file and run binpac multiple times. But I imagine that's not what
you're looking for. I guess you could write a script to postprocess
the output and pick only the pieces you want.

Mind elaborating why you can't just include the whole header into the
other project? Just curious what the problem is that you encounter
there.

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From michalpurzynski1 at gmail.com  Wed Jan 27 09:29:15 2016
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Wed, 27 Jan 2016 18:29:15 +0100
Subject: [Bro] Hardware recommends
In-Reply-To: <56A83693.10908@gmail.com>
References: <8c48a723fee2d6b3c3db92472e26d096@localhost>
	<3a83b02c33da6d78f350e84d4a012a33@localhost>
	<56A83693.10908@gmail.com>
Message-ID: <728FBB44-7542-4420-AECB-431BC12418E7@gmail.com>

You should look carefully at the Intel manuals and spec sheets. Here I had a great success with the E5-2697 v3 (and older) Xeons. They seem slower, at 2.6Ghz but it's the only Intel CPU that can overclock itself so much on all cores for arbitrary time.

When you read about the turbo mode, the number you get (for example 3.6Ghz) assumes a single core workload only - while all remaining cores stay idle.

What matters how high can the CPU go, on all cores at the same time.

For example sensors here are happy with 3Ghz!

You check it yourself in this Intel document

http://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e5-v3-spec-update.pdf

Page 9

As for the capture cards Myricom does the job nicely. People had success with Intel cards, with or without the pfring ZC. Note, that Bro now supports Afpacket natively, without going through libpcap, in the most recent Tpacketv3 version.

And you need RAM. Lots. I have 128GB per sensor and 64 on the manager. Fortunately the next version of Bro will have much lower memory requirements for the manager. It's not just Bro that uses a lot of memory here, my sensors with with a giant Myricom buffers, that's why.

Turbostat results attached


Package    Core     CPU Avg_MHz   %Busy Bzy_MHz TSC_MHz     SMI  CPU%c1  CPU%c3  CPU%c6  CPU%c7 CoreTmp  PkgTmp Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 PkgWatt RAMWatt   PKG_%   RAM_%
       -       -       -     683   22.06    3096    2597       0   77.94    0.00    0.00    0.00      76      77    0.00    0.00    0.00    0.00  199.22   69.00  199.78    0.00
       0       0       0    1232   39.78    3097    2597    3229   60.22    0.00    0.00    0.00      51      54    0.00    0.00    0.00    0.00   97.59   35.13   99.90    0.00
       0       0      28     273    8.81    3097    2597    3229   91.19
       0       1       1     515   16.64    3097    2597    3229   83.36    0.00    0.00    0.00      51
       0       1      29     274    8.86    3097    2597    3229   91.14
       0       2       2    1315   42.47    3097    2597    3229   57.53    0.00    0.00    0.00      51
       0       2      30     280    9.04    3097    2597    3229   90.96
       0       3       3     536   17.30    3097    2597    3229   82.70    0.00    0.00    0.00      49
       0       3      31     275    8.86    3097    2597    3229   91.14
       0       4       4     842   27.18    3097    2597    3229   72.82    0.00    0.00    0.00      52
       0       4      32     274    8.85    3097    2597    3229   91.15
       0       5       5     924   29.84    3097    2597    3229   70.16    0.00    0.00    0.00      51
       0       5      33     274    8.85    3097    2597    3229   91.15
       0       6       6     821   26.51    3097    2597    3229   73.49    0.00    0.00    0.00      50
       0       6      34     275    8.87    3097    2597    3229   91.13
       0       8       7     868   28.04    3097    2597    3229   71.96    0.00    0.00    0.00      51
       0       8      35     274    8.85    3097    2597    3229   91.15
       0       9       8     948   30.61    3097    2597    3229   69.39    0.00    0.00    0.00      50
       0       9      36     274    8.86    3097    2597    3229   91.14
       0      10       9    1005   32.46    3097    2597    3226   67.54    0.00    0.00    0.00      53
       0      10      37     274    8.84    3096    2597    3223   91.16
       0      11      10    1683   54.34    3096    2597    3223   45.66    0.00    0.00    0.00      51
       0      11      38     274    8.85    3096    2597    3223   91.15
       0      12      11     865   27.94    3096    2597    3223   72.06    0.00    0.00    0.00      51
       0      12      39     274    8.84    3096    2597    3223   91.16
       0      13      12     860   27.76    3096    2597    3223   72.24    0.00    0.00    0.00      48
       0      13      40     278    8.99    3096    2597    3223   91.01
       0      14      13    1083   34.99    3096    2597    3223   65.01    0.00    0.00    0.00      49
       0      14      41     274    8.84    3096    2597    3223   91.16
       1       0      14     929   30.00    3096    2597    3223   70.00    0.00    0.00    0.00      74      77    0.00    0.00    0.00    0.00  101.63   33.87   99.88    0.00
       1       0      42     288    9.29    3096    2597    3223   90.71
       1       1      15     978   31.59    3096    2597    3224   68.41    0.00    0.00    0.00      70
       1       1      43     281    9.08    3096    2597    3226   90.92
       1       2      16    2583   83.42    3096    2597    3226   16.58    0.00    0.00    0.00      72
       1       2      44     279    9.01    3096    2597    3226   90.99
       1       3      17    2073   66.93    3096    2597    3226   33.07    0.00    0.00    0.00      76
       1       3      45     284    9.16    3096    2597    3226   90.84
       1       4      18     928   29.96    3096    2597    3226   70.04    0.00    0.00    0.00      74
       1       4      46     285    9.19    3096    2597    3226   90.81
       1       5      19    1303   42.08    3096    2597    3226   57.92    0.00    0.00    0.00      75
       1       5      47     286    9.23    3096    2597    3226   90.77
       1       6      20     883   28.52    3096    2597    3226   71.48    0.00    0.00    0.00      74
       1       6      48     281    9.07    3096    2597    3226   90.93
       1       8      21    1779   57.46    3096    2597    3226   42.54    0.00    0.00    0.00      75
       1       8      49     284    9.18    3096    2597    3226   90.82
       1       9      22     958   30.93    3096    2597    3226   69.07    0.00    0.00    0.00      76
       1       9      50     282    9.12    3096    2597    3226   90.88
       1      10      23     892   28.82    3096    2597    3226   71.18    0.00    0.00    0.00      73
       1      10      51     283    9.12    3096    2597    3226   90.88
       1      11      24     954   30.82    3096    2597    3226   69.18    0.00    0.00    0.00      72
       1      11      52     280    9.05    3096    2597    3226   90.95
       1      12      25     954   30.82    3096    2597    3226   69.18    0.00    0.00    0.00      71
       1      12      53     280    9.05    3096    2597    3226   90.95
       1      13      26     888   28.67    3096    2597    3226   71.33    0.00    0.00    0.00      74
       1      13      54     280    9.05    3096    2597    3226   90.95
       1      14      27     853   27.56    3096    2597    3226   72.44    0.00    0.00    0.00      71
       1      14      55     281    9.08    3096    2597    3226   90.92




Sent from my iPad
> On 27 Jan 2016, at 04:16, Gary Faulkner <gfaulkner.nsm at gmail.com> wrote:
> 
> The Bro architecture documents still seem to suggest you can only
> process 80Mb/s or so of traffic per core, but even at 2.6 - 2.7 GHz you
> end up getting closer to 250-300Mb/s+. 3.1 will boost this a bit and
> allow you to handle slightly larger flows per core, but you may be able
> to get many more cores on a single host at 2.6 for similar or less
> money. I'd just be ware of optimizing too heavily on core count and
> ending up with 1.8GHz clocks. If you are doing good flow shunting up
> front I think you are likely to stray into the territory of more smaller
> flows which probably lends itself better to having more moderately
> clocked cores than fewer slightly higher clocked cores.
> 
> Also, unless you are considering ludicrous core counts per machine you
> might find you are oversubscribing your server long before you can take
> advantage of 40Gbps or 100Gbps NICs over a 10Gbps NIC. I've been fairly
> happy with Intel 10Gbps NICs and PF_RING DNA/ZC, but some prefer to use
> Myricom to avoid dealing with third party Intel NIC drivers. Be wary of
> artificial worker limits using RSS or vendor provided host based
> load-balancing (Myricom comes to mind). There are cases where folks have
> been stuck with not being able to take full advantage of their server
> hardware without having to run additional NICs, or do other work-arounds
> due to having more cores than queues/rings on the cards.
> 
> On a side note: I found out a lot of interesting things about how my
> sensors were performing, as well as my upstream load-balancer by using
> Justin's statsd plugin (assuming your upstream shunting doesn't throw
> off the output) to send the capture-loss script output to a time series
> DB and graphing it. For example I discovered a port going to an
> unrelated tool was becoming oversubscribed over the lunch hour causing
> back-pressure on the load-balancer that translated to every worker on my
> Bro cluster reporting 25-50% loss even though Bro should have been
> seeing relatively little traffic and was itself not over-subscribed. In
> that case I found it is sometimes desirable to have an extra 10G NIC in
> each server so that the tool, not the load-balancer gets over-subscribed
> until I can add more capacity to the tool and better spread the load.
> 
>> On 1/26/2016 2:34 PM, James Lay wrote:
>>> On 2016-01-26 10:44, James Lay wrote:
>>> And on the heels of the NIC question, how about hardware experiences?
>>> I'm looking at the PCIE2 NIC's at both Myricom and Netronome....any
>>> recommends for the server hardware to wrap around these cards?  The 
>>> plan
>>> is to have this machine monitor a corporate LAN...lot's of traffic.
>>> Guessing the team will want to go Dell if that helps.  Thanks for the
>>> advice all.
>>> 
>>> James
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> Thanks for the great information all..it really does help.
>> 
>> James
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160127/0afbdc78/attachment-0001.html 

From jlay at slave-tothe-box.net  Wed Jan 27 09:36:25 2016
From: jlay at slave-tothe-box.net (James Lay)
Date: Wed, 27 Jan 2016 10:36:25 -0700
Subject: [Bro] Hardware recommends
In-Reply-To: <8c48a723fee2d6b3c3db92472e26d096@localhost>
References: <8c48a723fee2d6b3c3db92472e26d096@localhost>
Message-ID: <f8282e71a3b82113a2369b571243f695@localhost>

On 2016-01-26 10:44, James Lay wrote:
> And on the heels of the NIC question, how about hardware experiences?
> I'm looking at the PCIE2 NIC's at both Myricom and Netronome....any
> recommends for the server hardware to wrap around these cards?  The 
> plan
> is to have this machine monitor a corporate LAN...lot's of traffic.
> Guessing the team will want to go Dell if that helps.  Thanks for the
> advice all.
> 
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

Thanks all SO much for the great information...I'll be able to make a 
good decision on the hardware going forward.  Thanks again.

James

From gfaulkner.nsm at gmail.com  Wed Jan 27 12:35:53 2016
From: gfaulkner.nsm at gmail.com (Gary Faulkner)
Date: Wed, 27 Jan 2016 14:35:53 -0600
Subject: [Bro] Hardware recommends
In-Reply-To: <B789C38C-9309-49BA-8D6B-D83A201F04DB@illinois.edu>
References: <8c48a723fee2d6b3c3db92472e26d096@localhost>
	<3a83b02c33da6d78f350e84d4a012a33@localhost> <56A83693.10908@gmail.com>
	<B789C38C-9309-49BA-8D6B-D83A201F04DB@illinois.edu>
Message-ID: <56A92A29.2030008@gmail.com>

I reached this page via Google, but the breadcrumbs show it as being 
part of the 2.4.1 documentation, which I believe is considered current.

https://www.bro.org/sphinx/cluster/index.html

Under Workers section:

"The rule of thumb we have followed recently is to allocate 
approximately 1 core for every 80Mbps of traffic that is being analyzed. 
However, this estimate could be extremely traffic mix-specific. It has 
generally worked for mixed traffic with many users and servers. For 
example, if your traffic peaks around 2Gbps (combined) and you want to 
handle traffic at peak load, you may want to have 26 cores available 
(2048 / 80 == 25.6). If the 80Mbps estimate works for your traffic, this 
could be handled by 3 physical hosts dedicated to being workers with 
each one containing dual 6-core processors."

On 1/27/16 6:57 AM, Slagell, Adam J wrote:
> Which document? We should update that.
>
>> On Jan 26, 2016, at 9:18 PM, Gary Faulkner <gfaulkner.nsm at gmail.com> wrote:
>>
>> The Bro architecture documents still seem to suggest you can only
>> process 80Mb/s or so of traffic per core, but even at 2.6 - 2.7 GHz you
>> end up getting closer to 250-300Mb/s+. 3.1 will boost this a bit and
>> allow you to handle slightly larger flows per core, but you may be able
>> to get many more cores on a single host at 2.6 for similar or less
>> money. I'd just be ware of optimizing too heavily on core count and
>> ending up with 1.8GHz clocks.
>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160127/7e129ee7/attachment.html 

From seth at icir.org  Wed Jan 27 13:59:41 2016
From: seth at icir.org (Seth Hall)
Date: Wed, 27 Jan 2016 16:59:41 -0500
Subject: [Bro] Hardware recommends
In-Reply-To: <56A92A29.2030008@gmail.com>
References: <8c48a723fee2d6b3c3db92472e26d096@localhost>
	<3a83b02c33da6d78f350e84d4a012a33@localhost>
	<56A83693.10908@gmail.com>
	<B789C38C-9309-49BA-8D6B-D83A201F04DB@illinois.edu>
	<56A92A29.2030008@gmail.com>
Message-ID: <31300CDE-F86A-45D3-839B-5DFDA5082977@icir.org>


> On Jan 27, 2016, at 3:35 PM, Gary Faulkner <gfaulkner.nsm at gmail.com> wrote:
> 
> I reached this page via Google, but the breadcrumbs show it as being part of the 2.4.1 documentation, which I believe is considered current.

Thanks for the link reminder.  Unfortunately that was current as of about 6 or 7 years ago, it's just been a very long time since it was updated. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From tgdesrochers at gmail.com  Thu Jan 28 04:20:50 2016
From: tgdesrochers at gmail.com (Tim Desrochers)
Date: Thu, 28 Jan 2016 07:20:50 -0500
Subject: [Bro] ACTION_ALARM and ACTION_EMAIL
In-Reply-To: <CBC518B3-3BC8-4EA8-92DF-1AE453C3F373@illinois.edu>
References: <CANrcqc3kzts=5Y8z+Xq6PTw-N_wO=koFEwDbFAB3V9nnoCtjGQ@mail.gmail.com>
	<CBC518B3-3BC8-4EA8-92DF-1AE453C3F373@illinois.edu>
Message-ID: <CANrcqc3bNBhLi2ihVeT4q2VCOA09UKs=K_vrSHthoRt9Ft3+QQ@mail.gmail.com>

When trying what you suggested I get the following output from broctl check:

error in /opt/bro/share/bro/policy/frameworks/control/controllee.bro, line
15: syntax error, at or near "module"

At the bottom of my local.bro I added:
redef Notice::emailed_types += {
  FTP::Bruteforcing,
  FTP::Site_Exec_Success,
  HTTP::SQL_Injection_Attacker,
  HTTP::SQL_Injection_Victim,
  SMTP::Blocklist_Error_Message,
  SMTP::Blocklist_Blocked_Host,
  SMTP::Suspicious_Origination,
  SSH::Password_Guessing,
  SSH::Login_By_Password_Guesser,
  }

Any reason why broctl is finding an error in the controllee.bro script.

On Wed, Jan 6, 2016 at 10:50 AM, Azoff, Justin S <jazoff at illinois.edu>
wrote:

> I'm not sure about #2, but for 1 and 3 there is an easy way to do this
> with the default configuration.  The notice framework has this as the
> notice policy:
>
> hook Notice::policy(n: Notice::Info) &priority=10
>     {
>     if ( n$note in Notice::ignored_types )
>         break;
>
>     if ( n$note in Notice::not_suppressed_types )
>         n$suppress_for=0secs;
>     if ( n$note in Notice::alarmed_types )
>         add n$actions[ACTION_ALARM];
>     if ( n$note in Notice::emailed_types )
>         add n$actions[ACTION_EMAIL];
>
>     if ( n$note in Notice::type_suppression_intervals )
>         n$suppress_for=Notice::type_suppression_intervals[n$note];
>
>     # Logging is a default action.  It can be removed in a later hook if
> desired.
>     add n$actions[ACTION_LOG];
>     }
>
> Those tables are all setup to make it easy to toggle actions:
>
>     ## Ignored notice types.
>     const ignored_types: set[Notice::Type] = {} &redef;
>     ## Emailed notice types.
>     const emailed_types: set[Notice::Type] = {} &redef;
>     ## Alarmed notice types.
>     const alarmed_types: set[Notice::Type] = {} &redef;
>     ## Types that should be suppressed for the default suppression
> interval.
>     const not_suppressed_types: set[Notice::Type] = {} &redef;
>
> So you simply need something like this in your local.bro:
>
> redef Notice::emailed_types += {
>     HTTP::SQL_Injection_Attacker,
>     HTTP::SQL_Injection_Victim,
> }
>
> If you do need to do anything more complicated, you can use your own
> Notice::policy and add whatever logic you want.
>
> To not get multiple emails for the same notice you need to ensure that the
> notice has the $identifier set that uniquely identifies the notice.  This
> is minimally something like cat(id$orig_h).  If you look at any of the
> scripts in policy/ you can see how they do this.
>
>
>
> --
> - Justin Azoff
>
> > On Jan 6, 2016, at 10:33 AM, Tim Desrochers <tgdesrochers at gmail.com>
> wrote:
> >
> > I have my sensor set up to email me notices with:
> >
> > hook Notice::policy(n: Notice::Info)
> >             {
> >             add n$actions[Notice::ACTION_EMAIL];
> >             }
> >
> > If I understand correct this will email me upon any entry in the
> notice.log.  Is there a way to:
> > 1. only get specific items emailed upon entry
> > 2. get the rest of notice.log entries emailed with ACTON_ALARM in the
> alarm-mail.txt and have that ignore anything that was previously emailed.
> > 3. Only get one notice email per alert?
> >
> > What I am doing is in the /opt/bro/share/bro/intel folder creating
> different folders with IOS's I want the intel framework to look over and I
> am using meta.do_notice to send the items of importance to the notice log.
> >
> > Excuse my ignorance with this subject I am just now trying to get things
> emailed out efficiently to reduce some noise and redundancy my analysts are
> seeing.
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160128/a066d682/attachment.html 

From jazoff at illinois.edu  Thu Jan 28 05:46:44 2016
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 28 Jan 2016 13:46:44 +0000
Subject: [Bro] ACTION_ALARM and ACTION_EMAIL
In-Reply-To: <CANrcqc3bNBhLi2ihVeT4q2VCOA09UKs=K_vrSHthoRt9Ft3+QQ@mail.gmail.com>
References: <CANrcqc3kzts=5Y8z+Xq6PTw-N_wO=koFEwDbFAB3V9nnoCtjGQ@mail.gmail.com>
	<CBC518B3-3BC8-4EA8-92DF-1AE453C3F373@illinois.edu>
	<CANrcqc3bNBhLi2ihVeT4q2VCOA09UKs=K_vrSHthoRt9Ft3+QQ@mail.gmail.com>
Message-ID: <E67363C9-7180-4B13-A4AF-9EB4E3340D61@illinois.edu>

Ah, I made a common error. The redef line is a regular statement(compared to something like an 'if' or 'export' block) and it needs a ; at the end. The final line needs to be

};

Bro complains about controllee.bro because it is still expecting a ; and it is the next file that is parsed.

-- 
- Justin Azoff

> On Jan 28, 2016, at 6:20 AM, Tim Desrochers <tgdesrochers at gmail.com> wrote:
> 
> When trying what you suggested I get the following output from broctl check:
> 
> error in /opt/bro/share/bro/policy/frameworks/control/controllee.bro, line 15: syntax error, at or near "module"
> 
> At the bottom of my local.bro I added:
> redef Notice::emailed_types += {
>   FTP::Bruteforcing,
>   FTP::Site_Exec_Success,
>   HTTP::SQL_Injection_Attacker,
>   HTTP::SQL_Injection_Victim,
>   SMTP::Blocklist_Error_Message,
>   SMTP::Blocklist_Blocked_Host,
>   SMTP::Suspicious_Origination,
>   SSH::Password_Guessing,
>   SSH::Login_By_Password_Guesser,
>   }
> 
> Any reason why broctl is finding an error in the controllee.bro script. 
> 
> On Wed, Jan 6, 2016 at 10:50 AM, Azoff, Justin S <jazoff at illinois.edu> wrote:
> I'm not sure about #2, but for 1 and 3 there is an easy way to do this with the default configuration.  The notice framework has this as the notice policy:
> 
> hook Notice::policy(n: Notice::Info) &priority=10
>     {
>     if ( n$note in Notice::ignored_types )
>         break;
> 
>     if ( n$note in Notice::not_suppressed_types )
>         n$suppress_for=0secs;
>     if ( n$note in Notice::alarmed_types )
>         add n$actions[ACTION_ALARM];
>     if ( n$note in Notice::emailed_types )
>         add n$actions[ACTION_EMAIL];
> 
>     if ( n$note in Notice::type_suppression_intervals )
>         n$suppress_for=Notice::type_suppression_intervals[n$note];
> 
>     # Logging is a default action.  It can be removed in a later hook if desired.
>     add n$actions[ACTION_LOG];
>     }
> 
> Those tables are all setup to make it easy to toggle actions:
> 
>     ## Ignored notice types.
>     const ignored_types: set[Notice::Type] = {} &redef;
>     ## Emailed notice types.
>     const emailed_types: set[Notice::Type] = {} &redef;
>     ## Alarmed notice types.
>     const alarmed_types: set[Notice::Type] = {} &redef;
>     ## Types that should be suppressed for the default suppression interval.
>     const not_suppressed_types: set[Notice::Type] = {} &redef;
> 
> So you simply need something like this in your local.bro:
> 
> redef Notice::emailed_types += {
>     HTTP::SQL_Injection_Attacker,
>     HTTP::SQL_Injection_Victim,
> }
> 
> If you do need to do anything more complicated, you can use your own Notice::policy and add whatever logic you want.
> 
> To not get multiple emails for the same notice you need to ensure that the notice has the $identifier set that uniquely identifies the notice.  This is minimally something like cat(id$orig_h).  If you look at any of the scripts in policy/ you can see how they do this.
> 
> 
> 
> --
> - Justin Azoff
> 
> > On Jan 6, 2016, at 10:33 AM, Tim Desrochers <tgdesrochers at gmail.com> wrote:
> >
> > I have my sensor set up to email me notices with:
> >
> > hook Notice::policy(n: Notice::Info)
> >             {
> >             add n$actions[Notice::ACTION_EMAIL];
> >             }
> >
> > If I understand correct this will email me upon any entry in the notice.log.  Is there a way to:
> > 1. only get specific items emailed upon entry
> > 2. get the rest of notice.log entries emailed with ACTON_ALARM in the alarm-mail.txt and have that ignore anything that was previously emailed.
> > 3. Only get one notice email per alert?
> >
> > What I am doing is in the /opt/bro/share/bro/intel folder creating different folders with IOS's I want the intel framework to look over and I am using meta.do_notice to send the items of importance to the notice log.
> >
> > Excuse my ignorance with this subject I am just now trying to get things emailed out efficiently to reduce some noise and redundancy my analysts are seeing.
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 



From mahboobehdaftari at gmail.com  Thu Jan 28 07:45:27 2016
From: mahboobehdaftari at gmail.com (mahboobe daftari)
Date: Thu, 28 Jan 2016 16:45:27 +0100
Subject: [Bro] Monitor the traffic to qemu
Message-ID: <CAKEFYq5jGt4jOSHGWZ52z5c-0EoOU+hG_M+CSs1K395U1Afwhw@mail.gmail.com>

Hello,

I have Ubuntu installed as a host system with Debian running as the guest
system on qemu. I need to send created traffic from the host to an Apache
web server installed on the guest and monitor this traffic using bro which
is installed on the host.
What is the best way to make this connection? I would really appreciate if
you could provide me with a link or something on how to make the bridge. I
have tried several ways but haven't been successful yet!

Thank you so much in advance,
Mahboobeh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160128/bcc9c14e/attachment.html 

From struck at ICSI.Berkeley.EDU  Thu Jan 28 08:05:25 2016
From: struck at ICSI.Berkeley.EDU (Christian Struck)
Date: Thu, 28 Jan 2016 17:05:25 +0100
Subject: [Bro] Monitor the traffic to qemu
In-Reply-To: <CAKEFYq5jGt4jOSHGWZ52z5c-0EoOU+hG_M+CSs1K395U1Afwhw@mail.gmail.com>
References: <CAKEFYq5jGt4jOSHGWZ52z5c-0EoOU+hG_M+CSs1K395U1Afwhw@mail.gmail.com>
Message-ID: <56AA3C45.1040701@icsi.berkeley.edu>

Hi,

On 28.01.2016 16:45, mahboobe daftari wrote:
> Hello,
> 
> I have Ubuntu installed as a host system with Debian running as the
> guest system on qemu. I need to send created traffic from the host to an
> Apache web server installed on the guest and monitor this traffic using
> bro which is installed on the host.
> What is the best way to make this connection? I would really appreciate
> if you could provide me with a link or something on how to make
> the bridge. I have tried several ways but haven't been successful yet!

I think you could just tap bro to the virtual ethernet interface of your
VM, I'm not very familiar with using only QEMU but I think the
networking section in their documentation should help you. There is also
a monitoring subsection.

http://wiki.qemu.org/Documentation/Networking

Best regards

Christian

From jmlynch at gmail.com  Thu Jan 28 11:53:48 2016
From: jmlynch at gmail.com (Jason Lynch)
Date: Thu, 28 Jan 2016 14:53:48 -0500
Subject: [Bro] Monitor the traffic to qemu
In-Reply-To: <CAKEFYq5jGt4jOSHGWZ52z5c-0EoOU+hG_M+CSs1K395U1Afwhw@mail.gmail.com>
References: <CAKEFYq5jGt4jOSHGWZ52z5c-0EoOU+hG_M+CSs1K395U1Afwhw@mail.gmail.com>
Message-ID: <CALHnNNVAWo_FdqXVjVqZvMzJ25oGGJh1Wx25=E5osSk_z62kJw@mail.gmail.com>

Openvswitch has port mirroring capability if you would like to go that
route on your VM host:

http://git.openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plain;f=FAQ;hb=HEAD

On Thu, Jan 28, 2016 at 10:45 AM, mahboobe daftari <
mahboobehdaftari at gmail.com> wrote:

> Hello,
>
> I have Ubuntu installed as a host system with Debian running as the guest
> system on qemu. I need to send created traffic from the host to an Apache
> web server installed on the guest and monitor this traffic using bro which
> is installed on the host.
> What is the best way to make this connection? I would really appreciate if
> you could provide me with a link or something on how to make the bridge. I
> have tried several ways but haven't been successful yet!
>
> Thank you so much in advance,
> Mahboobeh
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160128/4b12cf89/attachment.html 

From jdopheid at illinois.edu  Fri Jan 29 11:58:25 2016
From: jdopheid at illinois.edu (Dopheide, Jeannette M)
Date: Fri, 29 Jan 2016 19:58:25 +0000
Subject: [Bro] =?utf-8?q?BroCon_=E2=80=9916=3A_Sept=2E_13th_-_15th_in_Aust?=
	=?utf-8?q?in=2C_TX?=
Message-ID: <B4D82D01-4346-45FF-8622-3DB487AD0A00@illinois.edu>

We are happy to announce that BroCon ?16 will occur on Tuesday, September 13th - Thursday, September 15th at the Texas Advanced Computing Center in Austin, Texas.

See our event page:
https://www.bro.org/community/brocon2016.html

Early bird registration is open! CFP is open!

Interested in sponsoring BroCon? Contact us at info at bro.org for more information.

Thank you for your continued support, and see you in September!

Regards,
The Bro Project


------

Jeannette Dopheide
Bro Outreach Coordinator
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign






From matiasdavaro at gmail.com  Sun Jan 31 20:22:57 2016
From: matiasdavaro at gmail.com (Matias Davaro)
Date: Sun, 31 Jan 2016 23:22:57 -0500
Subject: [Bro] Binpac analyzer error
Message-ID: <CAAHbCpFEcPjuKwvoji+9VztENu5QgNk2V8huzqaR=XzzAOhRzw@mail.gmail.com>

Hello,

I am trying to learn the bro programming language and following along with
Jon Schipp's youtube video bro - writing an analyzer. When I attempt to
generate the files for the protocol I receive the following error:

/binpac_quickstart$ ./start.py RIP "Routing Internet Protocol" ../bro --udp
Traceback (most recent call last):
  File "./start.py", line 177, in <module>
    main(arguments)
  File "./start.py", line 59, in main
    if do_plugin:
UnboundLocalError: local variable 'do_plugin' referenced before assignment
elcabezon at elcabezon:~/binpac_quickstart$


However when I append plugin at the end of the command, the files are
generated in the src and scripts directory. I just want to know what I am
doing wrong. Thank you very much.

Very respectfully,

Matias
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160131/809486b5/attachment.html