From wren3 at illinois.edu  Wed Nov  1 14:18:08 2017
From: wren3 at illinois.edu (Ren, Wenyu)
Date: Wed, 1 Nov 2017 21:18:08 +0000
Subject: [Bro] Potential bug of network_time() when used together with
 suspend_processing() and continue_processing()
Message-ID: <D1DD2D198D059443B177F461727B015977A5A53C@CHIMBX6.ad.uillinois.edu>

Dear all,

I find network_time() sometimes returns the wall time instead of the timestamp of the packet when used together with suspend_processing() and continue_processing(). I have submitted a ticket for this (link as follows). Could someone take a look? I really appreciate it.

https://bro-tracker.atlassian.net/browse/BIT-1861

Best,
Wenyu



Wenyu Ren
Ph.D. Candidate
Department of Computer Science
University of Illinois at Urbana-Champaign


From pssunu6 at gmail.com  Thu Nov  2 02:25:37 2017
From: pssunu6 at gmail.com (ps sunu)
Date: Thu, 2 Nov 2017 14:55:37 +0530
Subject: [Bro] smb log add new field
Message-ID: <CALGAZaBGJX=o443NBsbt9C=tXjOHAf1MrrqEF9xwM1bEZj3hUg@mail.gmail.com>

Hi,
                    I need to add new filed in smb_mapping.log and and
smb_files.log, i tried below code
but its giving error


@load base/protocols/smb

module SMB;

#redef LogAscii::use_json = T;
export
  {

 }
redef record SMB::TreeInfo +=   {
    orig_country_code: string &log &optional;


  };

nternal warning in ././trybro.bro, line 10: Can't document redef of
SMB::TreeInfo, identifier lookup failed
internal warning in ././trybro.bro, line 11: Can't document record field
orig_country_code, unknown record: SMB::TreeInfo
error in ././trybro.bro, line 10: unknown identifier (SMB::TreeInfo)



Regards,
Sunu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171102/db81c608/attachment.html 

From bill.de.ping at gmail.com  Thu Nov  2 05:15:58 2017
From: bill.de.ping at gmail.com (william de ping)
Date: Thu, 2 Nov 2017 14:15:58 +0200
Subject: [Bro] - large double are not printed/written correctly
In-Reply-To: <D328482B-3F51-4880-BCDC-BB3C596D0F30@corelight.com>
References: <CAKZA8xiehdE3tcYbwQ51PX2mTJZsOQBWm+b7qsyqoN4Jcp34JQ@mail.gmail.com>
	<D328482B-3F51-4880-BCDC-BB3C596D0F30@corelight.com>
Message-ID: <CAKZA8xhhuttoSzCOfGM6kO1Sup3EO3wNh62wYxiubCJwJ3Auuw@mail.gmail.com>

Hi,

Yes, it seems that bro logging and writing large\small numbers is rounded
up\down.

I cannot find any workaround besides printing it as a string (which is not
a good approach).

Any suggestions ?

Thanks
B

On Mon, Oct 30, 2017 at 4:47 PM, Seth Hall <seth at corelight.com> wrote:

> If you are talking about logging through the logging framework, then
> that's a different code path for value serialization (I think we try and
> avoid scientific notation there).  Generally just allowing Bro to serialize
> values how ever it feels like it as you are doing with the bare unformatted
> print statement can occasionally result in some undesirable behavior.
>
> Are you talking about having this trouble with the logging framework?
>
>   .Seth
>
>
>
> On 29 Oct 2017, at 10:20, william de ping wrote:
>
> Hi,
>>
>> I have this simple script :
>>
>> event bro_init()
>> {
>> local a=-3.019159e-8;
>> print "a",a;
>> local s=fmt("%e",a);
>> print "s",s;
>> }
>>
>> results :
>> a, -0
>> s, -3.019159e-08
>>
>> the printing of variable a is important because its -0 once written in a
>> log file.
>>
>> is there anyway of printing\writing large doubles ?
>>
>> Thanks
>> B
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
> --
> Seth Hall * Corelight, Inc * www.corelight.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171102/0285830a/attachment.html 

From seth at corelight.com  Thu Nov  2 06:15:00 2017
From: seth at corelight.com (Seth Hall)
Date: Thu, 02 Nov 2017 09:15:00 -0400
Subject: [Bro] smb log add new field
In-Reply-To: <CALGAZaBGJX=o443NBsbt9C=tXjOHAf1MrrqEF9xwM1bEZj3hUg@mail.gmail.com>
References: <CALGAZaBGJX=o443NBsbt9C=tXjOHAf1MrrqEF9xwM1bEZj3hUg@mail.gmail.com>
Message-ID: <4C3B0900-E548-43B0-B406-F97537AD0FF6@corelight.com>



On 2 Nov 2017, at 5:25, ps sunu wrote:

> @load base/protocols/smb

This is the problem.  Because we chose to leave the SMB scripts out of 
the default base load in 2.5 (they'll be there for 2.6!) you need to 
load them first.  Replace the line you used above with this...

```bro
@load policy/protocols/smb
```

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171102/e2595972/attachment.html 

From dnthayer at illinois.edu  Thu Nov  2 10:19:26 2017
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Thu, 2 Nov 2017 12:19:26 -0500
Subject: [Bro] - large double are not printed/written correctly
In-Reply-To: <CAKZA8xhhuttoSzCOfGM6kO1Sup3EO3wNh62wYxiubCJwJ3Auuw@mail.gmail.com>
References: <CAKZA8xiehdE3tcYbwQ51PX2mTJZsOQBWm+b7qsyqoN4Jcp34JQ@mail.gmail.com>
	<D328482B-3F51-4880-BCDC-BB3C596D0F30@corelight.com>
	<CAKZA8xhhuttoSzCOfGM6kO1Sup3EO3wNh62wYxiubCJwJ3Auuw@mail.gmail.com>
Message-ID: <b5962bdb-c228-7f1f-3313-3ad09e02b84c@illinois.edu>

When you want to print "double" values with the "print" statement,
you can always choose your preferred format with the "fmt" built-in
function.

However, there is no way (that I'm aware of) to specify a format for
"double" values written to a log file.  The current behavior appears to
be broken (because very large or small values cannot be represented in
the logs), so the only workaround would probably be to log as a string.


On 11/2/17 7:15 AM, william de ping wrote:
> Hi,
> 
> Yes, it seems that bro logging and writing large\small numbers is 
> rounded up\down.
> 
> I cannot find any workaround besides printing it as a string (which is 
> not a good approach).
> 
> Any suggestions ?
> 
> Thanks
> B
> 
> On Mon, Oct 30, 2017 at 4:47 PM, Seth Hall <seth at corelight.com 
> <mailto:seth at corelight.com>> wrote:
> 
>     If you are talking about logging through the logging framework, then
>     that's a different code path for value serialization (I think we try
>     and avoid scientific notation there).? Generally just allowing Bro
>     to serialize values how ever it feels like it as you are doing with
>     the bare unformatted print statement can occasionally result in some
>     undesirable behavior.
> 
>     Are you talking about having this trouble with the logging framework?
> 
>      ? .Seth
> 
> 
> 
>     On 29 Oct 2017, at 10:20, william de ping wrote:
> 
>         Hi,
> 
>         I have this simple script :
> 
>         event bro_init()
>         {
>         local a=-3.019159e-8;
>         print "a",a;
>         local s=fmt("%e",a);
>         print "s",s;
>         }
> 
>         results :
>         a, -0
>         s, -3.019159e-08
> 
>         the printing of variable a is important because its -0 once
>         written in a
>         log file.
> 
>         is there anyway of printing\writing large doubles ?
> 
>         Thanks
>         B
>         _______________________________________________
>         Bro mailing list
>         bro at bro-ids.org <mailto:bro at bro-ids.org>
>         http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.ICSI.Berkeley.EDU_mailman_listinfo_bro&d=DwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Bi5qPBnY0NmYPqnRTPj_AfXQKpfQTZUpCzpfFBcawv0&m=t1WmmPaQy0vN_UfB578UX2RnXTIKO78_t4pI6zG6i1s&s=WyplhHwnKERpszF6HU-MBPsMzVmScwpIuqHa8-7Bxng&e=>
> 
> 
>     --
>     Seth Hall * Corelight, Inc * www.corelight.com
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.corelight.com&d=DwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Bi5qPBnY0NmYPqnRTPj_AfXQKpfQTZUpCzpfFBcawv0&m=t1WmmPaQy0vN_UfB578UX2RnXTIKO78_t4pI6zG6i1s&s=Z_dO3CiDvaO3DH5fSGqgXqG1UYq5JCclJ8taMigqAaA&e=>
> 
> 
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 


From pssunu6 at gmail.com  Fri Nov  3 02:57:08 2017
From: pssunu6 at gmail.com (ps sunu)
Date: Fri, 3 Nov 2017 15:27:08 +0530
Subject: [Bro] smb_mapping log add new field from conn.log
Message-ID: <CALGAZaCCZtp+KY5qUHb4+3Fd3iJf6V1Us1ybD+Wooj=vekN_9w@mail.gmail.com>

Hi,

              i need to copy a field from conn.log and add it in
smb_mapping.log,  in my script copy part is working but its not writing in
to smb_mapping log , i am using correct smb events ? or anything wrong
there?

my code



@load policy/protocols/smb
module TrackSMB;

redef LogAscii::use_json = T;



export {

   global conn_resp_ip_bytes: table[addr] of count &synchronized
&write_expire=7day;
   global conn_name_proto: table[addr] of transport_proto &synchronized
&write_expire=7day;


   redef record SMB::TreeInfo +=      {
    proto: transport_proto &optional &log;
     resp_ip_bytes: count &optional &log;



 };
}

if(rec?$resp_ip_bytes) {

conn_resp_ip_bytes[rec$id$orig_h] = rec$resp_ip_bytes;


}




if(rec?$proto) {
conn_name_proto[rec$id$orig_h] = rec$proto;
}

}


event file_over_new_connection(f:fa_file; c: connection, is_orig: bool)
&priority=10
{

 if ( c$id$orig_h in TrackSMB::conn_resp_ip_bytes )
     f$info$resp_ip_bytes = TrackSMB::conn_resp_ip_bytes[c$id$orig_h];

if ( c$id$resp_h in TrackSMB::conn_resp_ip_bytes )
     f$info$resp_ip_bytes = TrackSMB::conn_resp_ip_bytes[c$id$resp_h];


if ( c$id$orig_h in TrackSMB::conn_name_proto )
    c$smb$proto = TrackSMB::conn_name_proto[c$id$orig_h];
    if ( c$id$resp_h in TrackSMB::conn_name_proto )
    f$info$proto = TrackSMB::conn_name_proto[c$id$resp_h];
}
                                                          anything wrong in
above script or need to change any events ?

Regards,
Sunu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171103/7907dd4d/attachment.html 

From pssunu6 at gmail.com  Fri Nov  3 02:59:12 2017
From: pssunu6 at gmail.com (ps sunu)
Date: Fri, 3 Nov 2017 15:29:12 +0530
Subject: [Bro] Fwd: smb_mapping log add new field from conn.log
In-Reply-To: <CALGAZaCCZtp+KY5qUHb4+3Fd3iJf6V1Us1ybD+Wooj=vekN_9w@mail.gmail.com>
References: <CALGAZaCCZtp+KY5qUHb4+3Fd3iJf6V1Us1ybD+Wooj=vekN_9w@mail.gmail.com>
Message-ID: <CALGAZaAb0Q6L=jEDKEEB_hbxTELxsdEJrhr594AxeHnY_z2wVw@mail.gmail.com>

Hi,

              i need to copy a field from conn.log and add it in
smb_mapping.log,  in my script copy part is working but its not writing in
to smb_mapping log , i am using correct smb events ? or anything wrong
there?

my code



@load policy/protocols/smb
module TrackSMB;

redef LogAscii::use_json = T;



export {

   global conn_resp_ip_bytes: table[addr] of count &synchronized
&write_expire=7day;
   global conn_name_proto: table[addr] of transport_proto &synchronized
&write_expire=7day;


   redef record SMB::TreeInfo +=      {
    proto: transport_proto &optional &log;
     resp_ip_bytes: count &optional &log;



 };
}

event Conn::log_conn (rec: Conn::Info)

{

if(rec?$resp_ip_bytes) {

conn_resp_ip_bytes[rec$id$orig_h] = rec$resp_ip_bytes;


}




if(rec?$proto) {
conn_name_proto[rec$id$orig_h] = rec$proto;
}

}


event file_over_new_connection(f:fa_file; c: connection, is_orig: bool)
&priority=10
{

 if ( c$id$orig_h in TrackSMB::conn_resp_ip_bytes )
     f$info$resp_ip_bytes = TrackSMB::conn_resp_ip_bytes[c$id$orig_h];

if ( c$id$resp_h in TrackSMB::conn_resp_ip_bytes )
     f$info$resp_ip_bytes = TrackSMB::conn_resp_ip_bytes[c$id$resp_h];


if ( c$id$orig_h in TrackSMB::conn_name_proto )
    c$smb$proto = TrackSMB::conn_name_proto[c$id$orig_h];
    if ( c$id$resp_h in TrackSMB::conn_name_proto )
    f$info$proto = TrackSMB::conn_name_proto[c$id$resp_h];
}
                                                          anything wrong in
above script or need to change any events ?

Regards,
Sunu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171103/e2619cfe/attachment.html 

From seth at corelight.com  Fri Nov  3 05:09:04 2017
From: seth at corelight.com (Seth Hall)
Date: Fri, 03 Nov 2017 08:09:04 -0400
Subject: [Bro] smb_mapping log add new field from conn.log
In-Reply-To: <CALGAZaAb0Q6L=jEDKEEB_hbxTELxsdEJrhr594AxeHnY_z2wVw@mail.gmail.com>
References: <CALGAZaCCZtp+KY5qUHb4+3Fd3iJf6V1Us1ybD+Wooj=vekN_9w@mail.gmail.com>
	<CALGAZaAb0Q6L=jEDKEEB_hbxTELxsdEJrhr594AxeHnY_z2wVw@mail.gmail.com>
Message-ID: <F13A0FB0-F125-42D0-8EF3-5F94DB140EBA@corelight.com>

You have several problems that I'm seeing but I haven't actually your 
script to see if there are more.

On 3 Nov 2017, at 5:59, ps sunu wrote:

>    redef record SMB::TreeInfo +=      {
>     proto: transport_proto &optional &log;
>      resp_ip_bytes: count &optional &log;
>  };

You added these fields to the SMB::TreeInfo record (just keep that in 
mind)...

> event Conn::log_conn (rec: Conn::Info)
> {
> if(rec?$resp_ip_bytes) {
> conn_resp_ip_bytes[rec$id$orig_h] = rec$resp_ip_bytes;
> }
>
> if(rec?$proto) {
> conn_name_proto[rec$id$orig_h] = rec$proto;
> }
> }

This whole section is only run when a conn log entry is being written so 
if everything only happens over a single connection you won't see your 
tables have any data since nothing will be written until the connection 
ends.

> event file_over_new_connection(f:fa_file; c: connection, is_orig: 
> bool)
> &priority=10
> {
>  if ( c$id$orig_h in TrackSMB::conn_resp_ip_bytes )
>      f$info$resp_ip_bytes = TrackSMB::conn_resp_ip_bytes[c$id$orig_h];

You put those fields into the SMB::TreeInfo record, but f$info is the 
Files::Info record.


>     c$smb$proto = TrackSMB::conn_name_proto[c$id$orig_h];
>     if ( c$id$resp_h in TrackSMB::conn_name_proto )
>     f$info$proto = TrackSMB::conn_name_proto[c$id$resp_h];

You are jumping around a bit here, you added the proto field to 
SMB::TreeInfo, but c$smb doesn't even exist.  To access the 
SMB::TreeInfo record, it should be c$smb_state$current_tree

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com

From roberixion at gmail.com  Fri Nov  3 05:19:28 2017
From: roberixion at gmail.com (=?UTF-8?Q?Rober_Fern=C3=A1ndez?=)
Date: Fri, 3 Nov 2017 13:19:28 +0100
Subject: [Bro] size file
Message-ID: <CAL71vfWqVdtN7a9FULw0AGAHzGpDkO25qSAMuiP+tV7GbDrEwA@mail.gmail.com>

hi,

I have this code:

event connection_established(c: connection)  {

    local orig_file = generate_extraction_filename(extrac_prefix, c,
"orig.dat");
    local orig_f = open(orig_file);
    set_contents_file(c$id, CONTENTS_ORIG, orig_f);

    local resp_file = generate_extraction_filename(extrac_prefix, c,
"resp.dat");
    local resp_f = open(resp_file);
    set_contents_file(c$id, CONTENTS_RESP, resp_f);

}

and I would like set a maximum size, I think that I have two options,
1. set a maximum size file
2. control the data so that it does not exceed the size

How can I do this?

thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171103/0fb16ee9/attachment.html 

From seth at corelight.com  Fri Nov  3 06:07:42 2017
From: seth at corelight.com (Seth Hall)
Date: Fri, 03 Nov 2017 09:07:42 -0400
Subject: [Bro] smb_mapping log add new field from conn.log
In-Reply-To: <CALGAZaDMb1+CBO--YbY-5JTx34mzKsfoi5tDUWYBYKwZtJJXCg@mail.gmail.com>
References: <CALGAZaCCZtp+KY5qUHb4+3Fd3iJf6V1Us1ybD+Wooj=vekN_9w@mail.gmail.com>
	<CALGAZaAb0Q6L=jEDKEEB_hbxTELxsdEJrhr594AxeHnY_z2wVw@mail.gmail.com>
	<F13A0FB0-F125-42D0-8EF3-5F94DB140EBA@corelight.com>
	<CALGAZaDMb1+CBO--YbY-5JTx34mzKsfoi5tDUWYBYKwZtJJXCg@mail.gmail.com>
Message-ID: <F562A4CC-2E82-492C-8C5F-B0C0274DF0DF@corelight.com>

Keep in mind that if you are running a cluster, your global values are 
stored locally on each worker so connections happening on different 
workers won't know the values for the other workers.

   .Seth


On 3 Nov 2017, at 9:06, ps sunu wrote:

> Sir,
>                         now i changed as you mentioned , and its 
> running
> without error but its not hitting into log
>
> my changed code
>
> @load policy/protocols/smb
> module TrackSMB;
>
> redef LogAscii::use_json = T;
>
>
>
> export {
>
>    global conn_resp_ip_bytes: table[addr] of count &synchronized
> &write_expire=7day;
>    global conn_name_proto: table[addr] of transport_proto 
> &synchronized
> &write_expire=7day;
>
>
>    redef record SMB::TreeInfo +=      {
>     proto: transport_proto &optional &log;
>      resp_ip_bytes: count &optional &log;
>
>
>
>  };
> }
>
> event Conn::log_conn (rec: Conn::Info)
>
> {
>
>
> if(rec?$resp_ip_bytes) {
>
> conn_resp_ip_bytes[rec$id$orig_h] = rec$resp_ip_bytes;
>
>
> }
>
>
>
>
> if(rec?$proto) {
> conn_name_proto[rec$id$orig_h] = rec$proto;
> }
>
> }
>
> event smb_pipe_connect_heuristic (c: connection ) &priority=10
> {
>  if ( c$id$orig_h in TrackSMB::conn_resp_ip_bytes )
>      c$smb_state$current_tree$resp_ip_bytes =
> TrackSMB::conn_resp_ip_bytes[c$id$orig_h];
>
> if ( c$id$resp_h in TrackSMB::conn_resp_ip_bytes )
>       c$smb_state$current_tree$resp_ip_bytes =
> TrackSMB::conn_resp_ip_bytes[c$id$resp_h];
>
>
> if ( c$id$orig_h in TrackSMB::conn_name_proto )
>    c$smb_state$current_tree$proto = 
> TrackSMB::conn_name_proto[c$id$orig_h];
>      if ( c$id$resp_h in TrackSMB::conn_name_proto )
>      c$smb_state$current_tree$proto =
> TrackSMB::conn_name_proto[c$id$resp_h];
>
>  #    print c$smb_state$current_tree$proto;
> }
>
>
>
>
> Regards,
> Sunu
>
>
> On Fri, Nov 3, 2017 at 5:39 PM, Seth Hall <seth at corelight.com> wrote:
>
>> You have several problems that I'm seeing but I haven't actually your
>> script to see if there are more.
>>
>> On 3 Nov 2017, at 5:59, ps sunu wrote:
>>
>>    redef record SMB::TreeInfo +=      {
>>>     proto: transport_proto &optional &log;
>>>      resp_ip_bytes: count &optional &log;
>>>  };
>>>
>>
>> You added these fields to the SMB::TreeInfo record (just keep that in
>> mind)...
>>
>> event Conn::log_conn (rec: Conn::Info)
>>> {
>>> if(rec?$resp_ip_bytes) {
>>> conn_resp_ip_bytes[rec$id$orig_h] = rec$resp_ip_bytes;
>>> }
>>>
>>> if(rec?$proto) {
>>> conn_name_proto[rec$id$orig_h] = rec$proto;
>>> }
>>> }
>>>
>>
>> This whole section is only run when a conn log entry is being written 
>> so
>> if everything only happens over a single connection you won't see 
>> your
>> tables have any data since nothing will be written until the 
>> connection
>> ends.
>>
>> event file_over_new_connection(f:fa_file; c: connection, is_orig: 
>> bool)
>>> &priority=10
>>> {
>>>  if ( c$id$orig_h in TrackSMB::conn_resp_ip_bytes )
>>>      f$info$resp_ip_bytes = 
>>> TrackSMB::conn_resp_ip_bytes[c$id$orig_h];
>>>
>>
>> You put those fields into the SMB::TreeInfo record, but f$info is the
>> Files::Info record.
>>
>>
>>     c$smb$proto = TrackSMB::conn_name_proto[c$id$orig_h];
>>>     if ( c$id$resp_h in TrackSMB::conn_name_proto )
>>>     f$info$proto = TrackSMB::conn_name_proto[c$id$resp_h];
>>>
>>
>> You are jumping around a bit here, you added the proto field to
>> SMB::TreeInfo, but c$smb doesn't even exist.  To access the 
>> SMB::TreeInfo
>> record, it should be c$smb_state$current_tree
>>
>>   .Seth
>>
>> --
>> Seth Hall * Corelight, Inc * www.corelight.com
>>



--
Seth Hall * Corelight, Inc * www.corelight.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171103/bcd06bde/attachment-0001.html 

From fatema.bannatwala at gmail.com  Fri Nov  3 06:11:21 2017
From: fatema.bannatwala at gmail.com (fatema bannatwala)
Date: Fri, 3 Nov 2017 09:11:21 -0400
Subject: [Bro] smb_mapping log add new field from conn.log
In-Reply-To: <F13A0FB0-F125-42D0-8EF3-5F94DB140EBA@corelight.com>
References: <CALGAZaCCZtp+KY5qUHb4+3Fd3iJf6V1Us1ybD+Wooj=vekN_9w@mail.gmail.com>
	<CALGAZaAb0Q6L=jEDKEEB_hbxTELxsdEJrhr594AxeHnY_z2wVw@mail.gmail.com>
	<F13A0FB0-F125-42D0-8EF3-5F94DB140EBA@corelight.com>
Message-ID: <CACX0rURpmLgr0NFGgQ6t1WckAQNjdsLCLU426aKiPwcUvtC9qA@mail.gmail.com>

Hi Sunu,

Seth have pretty much addressed all the problems the script has.
Just wanted to add, that be careful in the types of events you choose
to log entries for.

event "file_over_new_connection" will be triggered for "every" file Bro sees
over a connection.
And event "log_conn" will be triggered for every connection Bro logs record
for.

what your code is trying to do currently, is recording protocol and
resp_ip_bytes for every connection,
and if the connection happens to have a file transfer, then, logging those
two fields in "file" record,
which doesn't seem to be your use case.

Also, I do not see any checks for "SMB" protocol specifically. Hence, your
all the if conditions will always
be true, if the connection happen to have resp_ip_bytes and proto, which
majority of Bro connections would have.

Thanks,
Fatema.

On Fri, Nov 3, 2017 at 8:09 AM, Seth Hall <seth at corelight.com> wrote:

> You have several problems that I'm seeing but I haven't actually your
> script to see if there are more.
>
> On 3 Nov 2017, at 5:59, ps sunu wrote:
>
>    redef record SMB::TreeInfo +=      {
>>     proto: transport_proto &optional &log;
>>      resp_ip_bytes: count &optional &log;
>>  };
>>
>
> You added these fields to the SMB::TreeInfo record (just keep that in
> mind)...
>
> event Conn::log_conn (rec: Conn::Info)
>> {
>> if(rec?$resp_ip_bytes) {
>> conn_resp_ip_bytes[rec$id$orig_h] = rec$resp_ip_bytes;
>> }
>>
>> if(rec?$proto) {
>> conn_name_proto[rec$id$orig_h] = rec$proto;
>> }
>> }
>>
>
> This whole section is only run when a conn log entry is being written so
> if everything only happens over a single connection you won't see your
> tables have any data since nothing will be written until the connection
> ends.
>
> event file_over_new_connection(f:fa_file; c: connection, is_orig: bool)
>> &priority=10
>> {
>>  if ( c$id$orig_h in TrackSMB::conn_resp_ip_bytes )
>>      f$info$resp_ip_bytes = TrackSMB::conn_resp_ip_bytes[c$id$orig_h];
>>
>
> You put those fields into the SMB::TreeInfo record, but f$info is the
> Files::Info record.
>
>
>     c$smb$proto = TrackSMB::conn_name_proto[c$id$orig_h];
>>     if ( c$id$resp_h in TrackSMB::conn_name_proto )
>>     f$info$proto = TrackSMB::conn_name_proto[c$id$resp_h];
>>
>
> You are jumping around a bit here, you added the proto field to
> SMB::TreeInfo, but c$smb doesn't even exist.  To access the SMB::TreeInfo
> record, it should be c$smb_state$current_tree
>
>   .Seth
>
> --
> Seth Hall * Corelight, Inc * www.corelight.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171103/136d2f72/attachment.html 

From haoscs at gmail.com  Sun Nov  5 14:37:08 2017
From: haoscs at gmail.com (Shuai Hao)
Date: Sun, 5 Nov 2017 17:37:08 -0500
Subject: [Bro] Bro cluster's CPU usage
Message-ID: <CAAAW5fmLWoOtt_s1ZHgcLmoR149+S5=fjWhrb36i9L=R1R1xSg@mail.gmail.com>

Hi All,

We are performing a benchmark to quantify the cpu usages of our bro
deployment. We setup a testbed, two hosts are connected by a switch, and
third host is running the bro and inspecting the traffic by port mirroring.

Bro is running with the cluster mode, and the pf_ring is enabled to
utilized the dual cores. We use iperf to send traffic with controlled
target bandwidth to investigate the cpu usages of bro's processes.

We see two workers' processes ($~bro/bin/bro -i eth -U .status -p broctl
...) typically run at cpu usage of 20% - 25% at each core, which is
consistent with some previous discussion. When we run the benchmark, with
the increase of transmission rates, we see the cpu usages of workers'
processes are increasing, which is reasonable.

The question is that sometimes we see a significant increase (60%~70%) from
cpu usages from two loggers' processes ($~bro/bin/bro -U .status -p broctl
... logger ...). How we should understand the resources consumed by the
loggers? For our case, what is the reasonable approach to evaluate Bro's
cpu usage?

Thanks for your comments!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171105/eb894c0c/attachment.html 

From seth at corelight.com  Mon Nov  6 05:21:25 2017
From: seth at corelight.com (Seth Hall)
Date: Mon, 06 Nov 2017 08:21:25 -0500
Subject: [Bro] Bro cluster's CPU usage
In-Reply-To: <CAAAW5fmLWoOtt_s1ZHgcLmoR149+S5=fjWhrb36i9L=R1R1xSg@mail.gmail.com>
References: <CAAAW5fmLWoOtt_s1ZHgcLmoR149+S5=fjWhrb36i9L=R1R1xSg@mail.gmail.com>
Message-ID: <766795FF-ABF3-410E-A345-5A07BE9BA249@corelight.com>

On 5 Nov 2017, at 17:37, Shuai Hao wrote:

> Bro is running with the cluster mode, and the pf_ring is enabled to
> utilized the dual cores. We use iperf to send traffic with controlled
> target bandwidth to investigate the cpu usages of bro's processes.

That's not a good way to test Bro performance unless iperf traffic is 
what will be monitored when you move to operational use.  Since Bro is 
so heavily centered around packet parsing and logging you are just 
overwhelming certain areas of Bro and not even executing other areas of 
it.

> The question is that sometimes we see a significant increase (60%~70%) 
> from
> cpu usages from two loggers' processes ($~bro/bin/bro -U .status -p 
> broctl
> ... logger ...). How we should understand the resources consumed by 
> the
> loggers? For our case, what is the reasonable approach to evaluate 
> Bro's
> cpu usage?

Those two logger processes you are seeing are actually a parent and 
child.  The child does the socket communication with the other Bro 
processes and the parent is the Bro process.  You will see increased cpu 
utilization from the logger as more logs are written.  The next question 
that is brought up is what logs are being written.  It's not always easy 
to guess.

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com

From pssunu6 at gmail.com  Tue Nov  7 11:13:11 2017
From: pssunu6 at gmail.com (ps sunu)
Date: Wed, 8 Nov 2017 00:43:11 +0530
Subject: [Bro] smb_files.log generating duplicate entry
Message-ID: <CALGAZaCxapt_wbAdLJh9Lwid8PJxhSRs+wpmQ4-KP=m0WnxbLg@mail.gmail.com>

Hi ,

                i need to copy  2 fields from conn.log and  add it in
smb_files.log it is adding but its entering as a duplicate entry
my code


@load policy/protocols/smb
module TrackSMB;
redef LogAscii::use_json = T;

export {

   global conn_resp_ip_bytes: table[addr] of count &synchronized
&write_expire=7day;
   global conn_name_proto: table[addr] of transport_proto &synchronized
&write_expire=7day;
#   global conn_service: table[addr] of string &synchronized
&write_expire=7day;

  redef record SMB::FileInfo += {
    proto: transport_proto &optional &log;

    resp_ip_bytes: count &optional &log;



 };
}



event Conn::log_conn (rec: Conn::Info)

{


if(rec?$resp_ip_bytes) {

conn_resp_ip_bytes[rec$id$orig_h] = rec$resp_ip_bytes;


}




if(rec?$proto) {
conn_name_proto[rec$id$orig_h] = rec$proto;
}

z


}







event file_over_new_connection(f: fa_file, c: connection, is_orig: bool)
&priority=5
{

 #if (f$source == "SMB") {
  if ( c$id$orig_h in TrackSMB::conn_resp_ip_bytes )
if (c?$smb_state && c$smb_state?$current_cmd) {
  c$smb_state$current_cmd$referenced_file$resp_ip_bytes =
TrackSMB::conn_resp_ip_bytes[c$id$orig_h];
}
    if ( c$id$resp_h in TrackSMB::conn_resp_ip_bytes )
if (c?$smb_state && c$smb_state?$current_cmd) {
  c$smb_state$current_cmd$referenced_file$resp_ip_bytes =
TrackSMB::conn_resp_ip_bytes[c$id$resp_h];
}

if ( c$id$orig_h in TrackSMB::conn_name_proto )
if (c?$smb_state && c$smb_state?$current_cmd) {
   c$smb_state$current_cmd$referenced_file$proto =
TrackSMB::conn_name_proto[c$id$orig_h];
}
  if ( c$id$resp_h in TrackSMB::conn_name_proto )
if (c?$smb_state && c$smb_state?$current_cmd) {
   c$smb_state$current_cmd$referenced_file$proto =
TrackSMB::conn_name_proto[c$id$resp_h];
}

Log::write(SMB::FILES_LOG, c$smb_state$current_cmd$referenced_file);
#  print c$smb_state$current_cmd$referenced_file$proto;



  #  }

}


http://try.bro.org/#/trybro/saved/184583



Regards,
Sunu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171108/4833a396/attachment.html 

From dnj0496 at gmail.com  Wed Nov  8 10:59:13 2017
From: dnj0496 at gmail.com (Dk Jack)
Date: Wed, 8 Nov 2017 10:59:13 -0800
Subject: [Bro] print
Message-ID: <CAGATen6bvr8w=eUDakMP-Mfw-hHTxLuxz2koq6vBMMZ6q+tSfQ@mail.gmail.com>

Hi,
Where does the 'print' output from scripts end up in a cluster scenario? I
am not seeing in the stdout.log. Thanks.

Dk.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171108/996532b7/attachment.html 

From jazoff at illinois.edu  Wed Nov  8 11:35:40 2017
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Wed, 8 Nov 2017 19:35:40 +0000
Subject: [Bro] print
In-Reply-To: <CAGATen6bvr8w=eUDakMP-Mfw-hHTxLuxz2koq6vBMMZ6q+tSfQ@mail.gmail.com>
References: <CAGATen6bvr8w=eUDakMP-Mfw-hHTxLuxz2koq6vBMMZ6q+tSfQ@mail.gmail.com>
Message-ID: <24052074-3765-46FE-BAC4-E4BDA1C593C1@illinois.edu>


> On Nov 8, 2017, at 1:59 PM, Dk Jack <dnj0496 at gmail.com> wrote:
> 
> Hi,
> Where does the 'print' output from scripts end up in a cluster scenario? I am not seeing in the stdout.log. Thanks.
> 
> Dk.


It ends up in stdout.log on the specific worker.  Unless you are doing something that involves debugging cluster communication you probably just want to replace print with Reporter::info()

? 
Justin Azoff



From stefano.rinaldi at aizoongroup.com  Thu Nov  9 02:11:26 2017
From: stefano.rinaldi at aizoongroup.com (Rinaldi Stefano)
Date: Thu, 9 Nov 2017 10:11:26 +0000
Subject: [Bro] R:  SMB copied files not showing in files.log
In-Reply-To: <A767C8E6-4690-4813-A670-D1ED822D8B39@corelight.com>
References: <59f7196b.4955650a.ebea6.ec93@mx.google.com>
	<A767C8E6-4690-4813-A670-D1ED822D8B39@corelight.com>
Message-ID: <96759744dff54d4889e70a39e5490be2@SRVEX03.aizoon.local>

Hi all, I and my colleagues have identified a scenario where one or more files are read without any real intention from the client (user or process). We have found that the presence of an Antivirus (AV) product (in our case: Sophos Endpoint Security and Control) have a considerable influence on SMB traffic and Bro log entries. The AV implies that all the files present in the selected remote directory are partially read and this leads to spurious entries in Bro SMB log files.
Regards,
Stefano

Da: bro-bounces at bro.org [mailto:bro-bounces at bro.org] Per conto di Seth Hall
Inviato: luned? 30 ottobre 2017 15:52
A: Vikram Basu <vikrambasu059 at gmail.com>
Cc: bro at bro.org
Oggetto: Re: [Bro] SMB copied files not showing in files.log


SMB is a complicated protocol. Windows systems will frequently call open on remote files but not actually transfer any of the bytes of the file. I think there may be several scenarios where they do that and I may not understand them all completely yet unfortunately.

Generally if some bytes of a file are transferred over SMB, that file will show up in files.log since files.log is meant to represent the actual transfer of files. The confusion arising from the smb_cmds.log file (where you saw the SMB::FILE_OPEN command) is one of the many reasons that that log is disabled by default too.

Are you experiencing a case where you know that a file was actually transferred over SMB but you didn't see a corresponding entry in files.log? If that's true, then I would really appreciate a pcap of the problem! I would really like to know about any cases where that isn't working correctly.

Thanks,
.Seth

On 30 Oct 2017, at 8:22, Vikram Basu wrote:

Hi,



So I am using the SMB plugin for Bro by loading in local.bro but it seems to be very inconsistent.

Often times when I am copying files between two windows machines over the domain there is no corresponding file in the files.log.

The smb_files.log itself seems to filled up with a lot of .ini files as well and they all seem to have the ?SMB::FILE_OPEN? action even when I haven?t opened any of them.

I thought I would use files showing source as SMB in files.log to differentiate when files are actually copied over the network but often times Bro does not detect the same.

Is there any particular way I need to share the files in windows to get the copied files to show up consistently in bro?



Regards



Vikram Basu

_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

--
Seth Hall * Corelight, Inc * www.corelight.com<http://www.corelight.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171109/79141cde/attachment.html 

From seth at corelight.com  Thu Nov  9 08:54:28 2017
From: seth at corelight.com (Seth Hall)
Date: Thu, 09 Nov 2017 11:54:28 -0500
Subject: [Bro] SMB copied files not showing in files.log
In-Reply-To: <96759744dff54d4889e70a39e5490be2@SRVEX03.aizoon.local>
References: <59f7196b.4955650a.ebea6.ec93@mx.google.com>
	<A767C8E6-4690-4813-A670-D1ED822D8B39@corelight.com>
	<96759744dff54d4889e70a39e5490be2@SRVEX03.aizoon.local>
Message-ID: <4CC21285-B4A3-417F-B789-06DCB67BDAE2@corelight.com>



On 9 Nov 2017, at 5:11, Rinaldi Stefano wrote:

> The AV implies that all the files present in the selected remote 
> directory are partially read and this leads to spurious entries in Bro 
> SMB log files.

If a portion of a file is actually transferred then you should 
definitely expect to see the file represented in smb_files.log and 
files.log.

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com

From lagoon7 at gmail.com  Thu Nov  9 17:02:30 2017
From: lagoon7 at gmail.com (Ludwig Goon)
Date: Thu, 9 Nov 2017 20:02:30 -0500
Subject: [Bro] HELP? bro-pkg hosom/file-extraction to write files when
	reading pcaps
Message-ID: <CAOHA7O=3tsh024S-XaxQvKjoHA5jFkBHvO4m0ipiBKAqWxR_cQ@mail.gmail.com>

Wanted to post this to the bro community.

I am trying to use the hosom/file-extraction plugin however I can't get it
to create a directory like ./extracted-files to put them in when I am
reading a pcap file. Does anyone in the bro community have any insight on
how to do this?

I am using bro 2.5.1 and the current github version of bro.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171109/1d4e8686/attachment.html 

From pssunu6 at gmail.com  Fri Nov 10 04:30:38 2017
From: pssunu6 at gmail.com (ps sunu)
Date: Fri, 10 Nov 2017 18:00:38 +0530
Subject: [Bro] http.log need conn.log filed
Message-ID: <CALGAZaASBfeU=DjLnJ=2YN4GCMZkXfFvG_Z89DT6PeW5xTZ-Pw@mail.gmail.com>

Hi,
             I need to copy conn record proto field into  http.log script
running but not htting the case my script


module TrackHTTP;
redef LogAscii::use_json=T;

  redef record HTTP::Info += {
    proto: transport_proto &optional &log;




 };


event http_reply(c: connection, version: string, code: count, reason:
string) &priority= 10
{

{
 print c$conn$proto;
 c$http$proto = c$conn$proto;
}
 # for ( id in c$conns ) {

      # print f$conns;
  #      local c = c$conns[id];

   #c$http$proto = c$conn$proto;


}




event http_message_done(c: connection, is_orig: bool, stat:
http_message_stat) &priority = 10
       {

if (c?$http && c?$conn )
{
 print c$conn$proto;
 c$http$proto = c$conn$proto;
}
       }








event http_request(c: connection, method: string, original_URI: string,
                  unescaped_URI: string, version: string) &priority=10
       {

if (c?$conn)
{
 print c$conn$proto;
 c$http$proto = c$conn$proto;
}


               }

Regards,
sunu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171110/ec2c7ffa/attachment.html 

From vedran.ivanec at gmail.com  Fri Nov 10 08:49:11 2017
From: vedran.ivanec at gmail.com (Vedran Ivanec)
Date: Fri, 10 Nov 2017 17:49:11 +0100
Subject: [Bro] Problem with FTP filename in Bro 2.5.2
Message-ID: <CAKZDWXVFRxOSpTv5AdOGE7UCPNiwr0KsaaGkchKuB_HQRcoUng@mail.gmail.com>

Hi,

I'm trying to upgrade Bro from 2.4.1 to 2.5.2.
I've managed to port all custom script to new Bro but one thing doesn't
work.

In Bro 2.4.2 file_cap.log for FTP traffic looks like this:
1510317723.341233       FTP_DATA        application/x-zip       Starting
file extraction, file name=Test-01.zip, total_bytes=0

In Bro 2.5.2 file_cap.log looks like:
1510319840.650513    FTP_DATA    application/x-zip    Starting file
extraction, file name=FTP_DATA-FF9XlZGkkngVBK1Z1.zip, total_bytes=0

Can someone point me in the right direction which variable changed in the
meantime and how to fix this issue?


Thanks in advance

--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171110/243123e1/attachment.html 

From hosom at battelle.org  Fri Nov 10 10:24:44 2017
From: hosom at battelle.org (Hosom, Stephen M)
Date: Fri, 10 Nov 2017 18:24:44 +0000
Subject: [Bro] HELP? bro-pkg hosom/file-extraction to write files
	when	reading pcaps
In-Reply-To: <CAOHA7O=3tsh024S-XaxQvKjoHA5jFkBHvO4m0ipiBKAqWxR_cQ@mail.gmail.com>
References: <CAOHA7O=3tsh024S-XaxQvKjoHA5jFkBHvO4m0ipiBKAqWxR_cQ@mail.gmail.com>
Message-ID: <26f40bba12f64e2f94767b78ca42513f@battelle.org>

When you run Bro against a pcap, the easiest thing to do is use the default plugin(s) for file extraction...


The command I use for ad-hoc extraction from pcaps that I am working is:


bro -r foo.pcap ./file-extraction/plugins/extract-all-files.bro


Be careful when using that to read packet captures that are large.

________________________________
From: bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Ludwig Goon <lagoon7 at gmail.com>
Sent: Thursday, November 9, 2017 8:02:30 PM
To: bro at bro.org
Subject: [Bro] HELP? bro-pkg hosom/file-extraction to write files when reading pcaps

Message received from outside the Battelle network. Carefully examine it before you open any links or attachments.

Wanted to post this to the bro community.


I am trying to use the hosom/file-extraction plugin however I can't get it to create a directory like ./extracted-files to put them in when I am reading a pcap file. Does anyone in the bro community have any insight on how to do this?

I am using bro 2.5.1 and the current github version of bro.


From wren3 at illinois.edu  Fri Nov 10 16:00:33 2017
From: wren3 at illinois.edu (Ren, Wenyu)
Date: Sat, 11 Nov 2017 00:00:33 +0000
Subject: [Bro] Convert count (storing float) to double
Message-ID: <D1DD2D198D059443B177F461727B015977A5DFA5@CHIMBX6.ad.uillinois.edu>

Hi Everyone,

This might be a dumb question, but does anyone know how to convert a float value stored in count type to double type in Bro? For example, 1092616192 should be converted to 10.0. I want to use DNP3 events such as "dnp3_analog_input_SPwFlag". However, the value is given in count but is actually a floating point value.

Thanks a lot.

Best,
Wenyu



Wenyu Ren
Ph.D. Candidate
Department of Computer Science
University of Illinois at Urbana-Champaign


From hlin33 at illinois.edu  Fri Nov 10 16:31:44 2017
From: hlin33 at illinois.edu (Hui Lin (Hugo) )
Date: Fri, 10 Nov 2017 16:31:44 -0800
Subject: [Bro] Convert count (storing float) to double
In-Reply-To: <faa774496abf454ba0cb3d9560caca6f@CITESHT3.ad.uillinois.edu>
References: <faa774496abf454ba0cb3d9560caca6f@CITESHT3.ad.uillinois.edu>
Message-ID: <CAKq214kpVe=LWyfepkZmttk8bvnKNLVq_ps_O_bi5tGBDGLjNQ@mail.gmail.com>

Hi Wenyu,

There are multiple guys before you ask me about this question. Sorry that I
have not included this conversion function in the original DNP3 analyzer.
The main reason at that time is that this conversion is kind of ad-hoc to
DNP3 not very general. My suggestion is to take a look at the DNP3 protocol
to see how floating point is stored and then use basic
arithmetic calculation to obtain the result of that floating point.

Also, may I ask you why want to use DNP3 to deliver floating point? In many
situations, I see that people use integer to store floating point for
simplicity. For example, if one wants to deliver floating point 10.012, he
first multiplies it by 1000 and delivers the value of 10012. On the
receiving end, one just divides the value by 1000 to obtain the floating
point.

Hope it helps,

Best,

Hui Lin

On Fri, Nov 10, 2017 at 4:00 PM, Ren, Wenyu <wren3 at illinois.edu> wrote:

> Hi Everyone,
>
> This might be a dumb question, but does anyone know how to convert a float
> value stored in count type to double type in Bro? For example, 1092616192
> should be converted to 10.0. I want to use DNP3 events such as
> "dnp3_analog_input_SPwFlag". However, the value is given in count but is
> actually a floating point value.
>
> Thanks a lot.
>
> Best,
> Wenyu
>
>
>
> Wenyu Ren
> Ph.D. Candidate
> Department of Computer Science
> University of Illinois at Urbana-Champaign
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 
Hui Lin
Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/)
DEPEND (http://depend.csl.illinois.edu/)
ECE, Uni. of Illinois at Urbana-Champaign
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171110/0a844658/attachment.html 

From hlin33 at illinois.edu  Fri Nov 10 18:59:11 2017
From: hlin33 at illinois.edu (Hui Lin (Hugo) )
Date: Fri, 10 Nov 2017 18:59:11 -0800
Subject: [Bro] Convert count (storing float) to double
In-Reply-To: <e412ea22991a4d56ace4d76d69c07ded@CHIHT4.ad.uillinois.edu>
References: <faa774496abf454ba0cb3d9560caca6f@CITESHT3.ad.uillinois.edu>
	<CAKq214kpVe=LWyfepkZmttk8bvnKNLVq_ps_O_bi5tGBDGLjNQ@mail.gmail.com>
	<e412ea22991a4d56ace4d76d69c07ded@CHIHT4.ad.uillinois.edu>
Message-ID: <CAKq214mkhCNHMewvXzpFpRD+F_CiJkky8Qy6hQXRifq1Rfnpeg@mail.gmail.com>

As far as I know, there is no built-in function. I don't think that DNP3
follow a common rule to encode float point number; so I am afraid that you
probably need to write a parsing function.

Best,

Hui

On Fri, Nov 10, 2017 at 4:39 PM, Ren, Wenyu <wren3 at illinois.edu> wrote:

> Hi Hui,
>
> So there is no built-in function in Bro to do such things? Well, I need to
> write my parser so that it can parse floating point even if it might be the
> rare case I guess. I will take a look at how floating point is stored and
> write my own conversion function then. Thanks a lot.
>
> Best,
> Wenyu
>
>
> *Wenyu Ren*
> Ph.D. Candidate
> Department of Computer Science
> University of Illinois at Urbana-Champaign
> ------------------------------
> *From:* hugolin615 at gmail.com [hugolin615 at gmail.com] on behalf of Hui Lin
> (Hugo) [hlin33 at illinois.edu]
> *Sent:* Friday, November 10, 2017 6:31 PM
> *To:* Ren, Wenyu
> *Cc:* bro at bro.org
> *Subject:* Re: [Bro] Convert count (storing float) to double
>
> Hi Wenyu,
>
> There are multiple guys before you ask me about this question. Sorry that
> I have not included this conversion function in the original DNP3 analyzer.
> The main reason at that time is that this conversion is kind of ad-hoc to
> DNP3 not very general. My suggestion is to take a look at the DNP3 protocol
> to see how floating point is stored and then use basic
> arithmetic calculation to obtain the result of that floating point.
>
> Also, may I ask you why want to use DNP3 to deliver floating point? In
> many situations, I see that people use integer to store floating point
> for simplicity. For example, if one wants to deliver floating point 10.012,
> he first multiplies it by 1000 and delivers the value of 10012. On the
> receiving end, one just divides the value by 1000 to obtain the floating
> point.
>
> Hope it helps,
>
> Best,
>
> Hui Lin
>
> On Fri, Nov 10, 2017 at 4:00 PM, Ren, Wenyu <wren3 at illinois.edu> wrote:
>
>> Hi Everyone,
>>
>> This might be a dumb question, but does anyone know how to convert a
>> float value stored in count type to double type in Bro? For example,
>> 1092616192 should be converted to 10.0. I want to use DNP3 events such as
>> "dnp3_analog_input_SPwFlag". However, the value is given in count but is
>> actually a floating point value.
>>
>> Thanks a lot.
>>
>> Best,
>> Wenyu
>>
>>
>>
>> Wenyu Ren
>> Ph.D. Candidate
>> Department of Computer Science
>> University of Illinois at Urbana-Champaign
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>
>
> --
> Hui Lin
> Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/)
> DEPEND (http://depend.csl.illinois.edu/)
> ECE, Uni. of Illinois at Urbana-Champaign
>
>


-- 
Hui Lin
Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/)
DEPEND (http://depend.csl.illinois.edu/)
ECE, Uni. of Illinois at Urbana-Champaign
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171110/9f9b79b8/attachment.html 

From bro-ml at razaborg.fr  Sun Nov 12 05:05:19 2017
From: bro-ml at razaborg.fr (bro-ml at razaborg.fr)
Date: Sun, 12 Nov 2017 14:05:19 +0100
Subject: [Bro] Multi tap architecture
Message-ID: <b144424b-588f-bc71-8b46-a9f6f2b3e815@razaborg.fr>

Hi everyone,

I'm looking to build a Bro architecture with several Tap components (I
mean the tcpdump stuff), all separated from the core.
I've seen the "cluster" architecture
(https://www.bro.org/sphinx/cluster/index.html), but as I said I want to
split out the capture work, not the protocol analysis stuff.

My situation is the following : I have several "boxes" (with not enough
power to do the protocol analysis work, that's the point) in different
networks, all connected to one single "core" component. I would like to
deploy network capture (Tap) instances on all those boxes, and let the
core component do all the hard stuff (I can potentially install a
front-end on this core component to set up many "workers" behind it).

Is there any way to do this ? Any documentation ? Does anyone have any
clue about how to set it up that way ?

Thanks a lot,
Pierre


From haoscs at gmail.com  Mon Nov 13 14:30:46 2017
From: haoscs at gmail.com (Shuai Hao)
Date: Mon, 13 Nov 2017 17:30:46 -0500
Subject: [Bro] TCP normalization and reassembly decision
Message-ID: <CAAAW5fnhnO_b+sEH4WiqeJmGafGho6Vx_9_ZPJ3Q4YFG75Ns+w@mail.gmail.com>

Hi All,

I have some questions regarding to TCP normalization and reassembly.

In /src/analyzer/protocol/tcp/tcp.cc, I find a comment "we could be fooled
by an inconsistent SYN retransmission. Where is a normalizer". So I assume
Bro doesn't come with a TCP normalizer. What is the consideration for such
decision? It will be not necessary, or it will be implemented in future?

On the other hand, I wonder that does bro implement the Rules against
adversaries mentioned in Vern's paper "Robust TCP Stream Reassembly In the
Presence of Adversaries"?

Thanks very much in advance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171113/7070986f/attachment.html 

From christian at corelight.com  Mon Nov 13 17:44:40 2017
From: christian at corelight.com (Christian Kreibich)
Date: Mon, 13 Nov 2017 17:44:40 -0800
Subject: [Bro] TCP normalization and reassembly decision
In-Reply-To: <CAAAW5fnhnO_b+sEH4WiqeJmGafGho6Vx_9_ZPJ3Q4YFG75Ns+w@mail.gmail.com>
References: <CAAAW5fnhnO_b+sEH4WiqeJmGafGho6Vx_9_ZPJ3Q4YFG75Ns+w@mail.gmail.com>
Message-ID: <2203e9df-fef7-19c2-be04-b23ad163e8b0@corelight.com>

Hi Shuai,

On 11/13/2017 02:30 PM, Shuai Hao wrote:
> In /src/analyzer/protocol/tcp/tcp.cc, I find a comment "we could be fooled
> by an inconsistent SYN retransmission. Where is a normalizer". So I assume
> Bro doesn't come with a TCP normalizer. What is the consideration for such
> decision? It will be not necessary, or it will be implemented in future?

A TCP normalizer, in the sense referred to here, is a middlebox that 
removes ambiguities in the traffic by actually modifying the packet flow 
and payloads in-path, to simplify the job of subsequent network 
monitors. So in order to implement this Bro would need to support 
in-path deployment, which isn't a priority for us.

There's old (entirely unsupported) code for such a normalizer available 
here, if you'd like to experiment:

http://icir.org/christian/downloads/norm-0.2.0.tar.gz

There are also commercial products in this space that support varying 
extents of traffic normalization.

Best,
-C.

From haoscs at gmail.com  Mon Nov 13 21:10:35 2017
From: haoscs at gmail.com (Shuai Hao)
Date: Tue, 14 Nov 2017 00:10:35 -0500
Subject: [Bro] TCP normalization and reassembly decision
In-Reply-To: <2203e9df-fef7-19c2-be04-b23ad163e8b0@corelight.com>
References: <CAAAW5fnhnO_b+sEH4WiqeJmGafGho6Vx_9_ZPJ3Q4YFG75Ns+w@mail.gmail.com>
	<2203e9df-fef7-19c2-be04-b23ad163e8b0@corelight.com>
Message-ID: <CAAAW5fmki+9G-dzu-qE0a0S0jSsBb_o_gq5icvxUG4mtEKWzoA@mail.gmail.com>

Thanks for your sharing the example code, Christian!

It also explains that at least the rule 3 in Vern's paper cannot be
implemented since it has to be operated in in-line mode. But how the first
two rules?

For rule 1 (limit the buffer of per-connection), is the rule implemented in
current Bro and does the 100KB buffer of per-connection hold?

For rule 2 (randomly evict connections), given we typically have
capture_loss and dropped_packets which reflect Bro's behavior, is there any
rule on the connection-level when Bro evicts connections?

Thanks,

On Mon, Nov 13, 2017 at 8:44 PM, Christian Kreibich <christian at corelight.com
> wrote:

> Hi Shuai,
>
> On 11/13/2017 02:30 PM, Shuai Hao wrote:
>
>> In /src/analyzer/protocol/tcp/tcp.cc, I find a comment "we could be
>> fooled
>> by an inconsistent SYN retransmission. Where is a normalizer". So I assume
>> Bro doesn't come with a TCP normalizer. What is the consideration for such
>> decision? It will be not necessary, or it will be implemented in future?
>>
>
> A TCP normalizer, in the sense referred to here, is a middlebox that
> removes ambiguities in the traffic by actually modifying the packet flow
> and payloads in-path, to simplify the job of subsequent network monitors.
> So in order to implement this Bro would need to support in-path deployment,
> which isn't a priority for us.
>
> There's old (entirely unsupported) code for such a normalizer available
> here, if you'd like to experiment:
>
> http://icir.org/christian/downloads/norm-0.2.0.tar.gz
>
> There are also commercial products in this space that support varying
> extents of traffic normalization.
>
> Best,
> -C.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171114/b4950deb/attachment.html 

From bill.de.ping at gmail.com  Tue Nov 14 07:34:29 2017
From: bill.de.ping at gmail.com (william de ping)
Date: Tue, 14 Nov 2017 17:34:29 +0200
Subject: [Bro]  integrate FPGA\PF_RING supported NIC with Bro - offload
Message-ID: <CAKZA8xjKdYVysYiZ3_u0DhPuODvT6NA0bmeBE9mR1W0RsTrqow@mail.gmail.com>

Hi all,

I wonder if anyone had a chance of somehow incorporate a FPGA NIC \ Smart
NIC \ Napatech \ PF_RING with Bro ?

I would like to scale my Bro cluster and process higher volumes of traffic
(5-10Gbps). Currently I use PF_RING supported nic (intel X710) to LB the
traffic to multiple workers - this environment cannot handle the above
traffic load.
I could always beef up my server but that does not seem like an elegant
solution.

Could there be any benefits for moving to Napatech NIC or FPGA supported
NIC ?
In what scenario an FPGA card can be integrated with Bro to offload some of
its more tedious tasks to the smart nic ?

Any ideas would be appreciated
Thanks
B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171114/76a02acb/attachment.html 

From michalpurzynski1 at gmail.com  Tue Nov 14 07:57:43 2017
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Tue, 14 Nov 2017 16:57:43 +0100
Subject: [Bro] integrate FPGA\PF_RING supported NIC with Bro - offload
In-Reply-To: <CAKZA8xjKdYVysYiZ3_u0DhPuODvT6NA0bmeBE9mR1W0RsTrqow@mail.gmail.com>
References: <CAKZA8xjKdYVysYiZ3_u0DhPuODvT6NA0bmeBE9mR1W0RsTrqow@mail.gmail.com>
Message-ID: <E447FB83-DF94-4E57-9A31-B021FF4CD20B@gmail.com>

X520 and X710 can handle that easily, with afpacket.  No need for pfring, it would not even come close to 10Gbit anyway.

Which distribution do you have to use?

> On Nov 14, 2017, at 4:34 PM, william de ping <bill.de.ping at gmail.com> wrote:
> 
> Hi all,
> 
> I wonder if anyone had a chance of somehow incorporate a FPGA NIC \ Smart NIC \ Napatech \ PF_RING with Bro ?
> 
> I would like to scale my Bro cluster and process higher volumes of traffic (5-10Gbps). Currently I use PF_RING supported nic (intel X710) to LB the traffic to multiple workers - this environment cannot handle the above traffic load.
> I could always beef up my server but that does not seem like an elegant solution.
> 
> Could there be any benefits for moving to Napatech NIC or FPGA supported NIC ?
> In what scenario an FPGA card can be integrated with Bro to offload some of its more tedious tasks to the smart nic ?
> 
> Any ideas would be appreciated
> Thanks
> B
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From bill.de.ping at gmail.com  Tue Nov 14 08:20:35 2017
From: bill.de.ping at gmail.com (william de ping)
Date: Tue, 14 Nov 2017 18:20:35 +0200
Subject: [Bro] integrate FPGA\PF_RING supported NIC with Bro - offload
In-Reply-To: <E447FB83-DF94-4E57-9A31-B021FF4CD20B@gmail.com>
References: <CAKZA8xjKdYVysYiZ3_u0DhPuODvT6NA0bmeBE9mR1W0RsTrqow@mail.gmail.com>
	<E447FB83-DF94-4E57-9A31-B021FF4CD20B@gmail.com>
Message-ID: <CAKZA8xgUjm2MfLGiWaheO2FJ2BXa4RF99dNwYo2pOGXaraLs2Q@mail.gmail.com>

I use Bro 2.5, ubuntu server 16.04, PF_RING 6.6

So af_packet will speed up the packet source \ IO loop part of Bro more
than PF_RING ?
After reading about it they seem to provide roughly the same output in
terms of speed.

It still seems like speeding up the reading of network traffic to Bro can
get you so far, no other ways of taking some of Bro's processing and
offload them to a network card\ FPGA card ?

Thanks

On Tue, Nov 14, 2017 at 5:57 PM, Micha? Purzy?ski <
michalpurzynski1 at gmail.com> wrote:

> X520 and X710 can handle that easily, with afpacket.  No need for pfring,
> it would not even come close to 10Gbit anyway.
>
> Which distribution do you have to use?
>
> > On Nov 14, 2017, at 4:34 PM, william de ping <bill.de.ping at gmail.com>
> wrote:
> >
> > Hi all,
> >
> > I wonder if anyone had a chance of somehow incorporate a FPGA NIC \
> Smart NIC \ Napatech \ PF_RING with Bro ?
> >
> > I would like to scale my Bro cluster and process higher volumes of
> traffic (5-10Gbps). Currently I use PF_RING supported nic (intel X710) to
> LB the traffic to multiple workers - this environment cannot handle the
> above traffic load.
> > I could always beef up my server but that does not seem like an elegant
> solution.
> >
> > Could there be any benefits for moving to Napatech NIC or FPGA supported
> NIC ?
> > In what scenario an FPGA card can be integrated with Bro to offload some
> of its more tedious tasks to the smart nic ?
> >
> > Any ideas would be appreciated
> > Thanks
> > B
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171114/f65ebfa7/attachment-0001.html 

From seth at corelight.com  Tue Nov 14 08:42:45 2017
From: seth at corelight.com (Seth Hall)
Date: Tue, 14 Nov 2017 11:42:45 -0500
Subject: [Bro] integrate FPGA\PF_RING supported NIC with Bro - offload
In-Reply-To: <CAKZA8xgUjm2MfLGiWaheO2FJ2BXa4RF99dNwYo2pOGXaraLs2Q@mail.gmail.com>
References: <CAKZA8xjKdYVysYiZ3_u0DhPuODvT6NA0bmeBE9mR1W0RsTrqow@mail.gmail.com>
	<E447FB83-DF94-4E57-9A31-B021FF4CD20B@gmail.com>
	<CAKZA8xgUjm2MfLGiWaheO2FJ2BXa4RF99dNwYo2pOGXaraLs2Q@mail.gmail.com>
Message-ID: <A14F7D82-9F62-4EBA-9638-83F0B3A173A2@corelight.com>


On 14 Nov 2017, at 11:20, william de ping wrote:

> It still seems like speeding up the reading of network traffic to Bro 
> can get you so far, no other ways of taking some of Bro's processing 
> and offload them to a network card\ FPGA card ?

There aren't any code paths in Bro that offload work into any 
specialized NICs.  It's fairly hard to find the exact right abstraction 
that would provide some benefit to Bro and still be technically 
achievable.

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com

From r.bortolameotti at utwente.nl  Tue Nov 14 10:09:47 2017
From: r.bortolameotti at utwente.nl (BortolameottiR)
Date: Tue, 14 Nov 2017 19:09:47 +0100
Subject: [Bro] HTTP responses details are missing
Message-ID: <4fbbd2d1-4445-32d3-df74-ad93cf920889@utwente.nl>

Dear all,

I have a simple question. When I run bro against a .pcap file, it
happens that some log lines do not show any detail regarding the
response e.g., response_body_len, status_msg, status_code, resp_fuids
etc. Is it a problem of the HTTP analyzer?

I am currently trying to extract all the text/files of all responses,
however it seems that some connections responses are not parsed by the
HTTP analyzer.

I tried to extract the files (following the scripts below), however also
in these settings some "files" where missing. In my case I am talking
about .css / .html / .js in the response content.

-
https://www.bro.org/sphinx/scripts/policy/frameworks/files/extract-all-files.bro.html

- https://www.bro.org/sphinx-git/httpmonitor/index.html (at the bottom)

When you look in conn.log, the same connection (according to the id)
shows the amount of bytes of the response. If you inspect the file using
Wireshark you can also see that there was a response.?

Any idea on what could be the issue?

I can even share the .pcap if needed.

Best,

R.




From seth at corelight.com  Tue Nov 14 10:49:30 2017
From: seth at corelight.com (Seth Hall)
Date: Tue, 14 Nov 2017 13:49:30 -0500
Subject: [Bro] HTTP responses details are missing
In-Reply-To: <4fbbd2d1-4445-32d3-df74-ad93cf920889@utwente.nl>
References: <4fbbd2d1-4445-32d3-df74-ad93cf920889@utwente.nl>
Message-ID: <F2E38331-C281-4CD0-8CE5-D163CA6A72B5@corelight.com>

It's most likely that you have had offloaded checksums when you captures 
the PCAP.  More information here:
	https://www.bro.org/documentation/faq.html#why-isn-t-bro-producing-the-logs-i-expect-a-note-about-checksums

   .Seth

On 14 Nov 2017, at 13:09, BortolameottiR wrote:

> Dear all,
>
> I have a simple question. When I run bro against a .pcap file, it
> happens that some log lines do not show any detail regarding the
> response e.g., response_body_len, status_msg, status_code, resp_fuids
> etc. Is it a problem of the HTTP analyzer?
>
> I am currently trying to extract all the text/files of all responses,
> however it seems that some connections responses are not parsed by the
> HTTP analyzer.
>
> I tried to extract the files (following the scripts below), however 
> also
> in these settings some "files" where missing. In my case I am talking
> about .css / .html / .js in the response content.
>
> -
> https://www.bro.org/sphinx/scripts/policy/frameworks/files/extract-all-files.bro.html
>
> - https://www.bro.org/sphinx-git/httpmonitor/index.html (at the 
> bottom)
>
> When you look in conn.log, the same connection (according to the id)
> shows the amount of bytes of the response. If you inspect the file 
> using
> Wireshark you can also see that there was a response.?
>
> Any idea on what could be the issue?
>
> I can even share the .pcap if needed.
>
> Best,
>
> R.
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

--
Seth Hall * Corelight, Inc * www.corelight.com


From vern at berkeley.edu  Tue Nov 14 10:53:14 2017
From: vern at berkeley.edu (Vern Paxson)
Date: Tue, 14 Nov 2017 10:53:14 -0800
Subject: [Bro] TCP normalization and reassembly decision
In-Reply-To: <CAAAW5fmki+9G-dzu-qE0a0S0jSsBb_o_gq5icvxUG4mtEKWzoA@mail.gmail.com>
	(Tue, 14 Nov 2017 00:10:35 EST).
Message-ID: <20171114185314.561852C40B9@rock.ICSI.Berkeley.EDU>

> It also explains that at least the rule 3 in Vern's paper cannot be
> implemented since it has to be operated in in-line mode. But how the first
> two rules?

That paper focuses on in-line network processing: none of the rules in
it help thwart evasion for passive monitoring.

		Vern

From r.bortolameotti at utwente.nl  Wed Nov 15 02:41:29 2017
From: r.bortolameotti at utwente.nl (BortolameottiR)
Date: Wed, 15 Nov 2017 11:41:29 +0100
Subject: [Bro] HTTP responses details are missing
In-Reply-To: <F2E38331-C281-4CD0-8CE5-D163CA6A72B5@corelight.com>
References: <4fbbd2d1-4445-32d3-df74-ad93cf920889@utwente.nl>
	<F2E38331-C281-4CD0-8CE5-D163CA6A72B5@corelight.com>
Message-ID: <067592de-99c5-b678-4aee-0ea0f9042a84@utwente.nl>

Hi Seth,

Thanks for the suggestion. That's was not the case. While I was
debugging I saw that most connections without files where missing bytes
(in the conn.log) and where present in the weird.log due to truncated TCP.

The .pcap in question was generated by replaying a capture (with tcp
replay), and we have injected some traffic in it. Bro apparently did not
like it. With the original .pcap we did not encounter this issue.

So the problem was not in Bro. :)

Thanks again,

Best regards,

Riccardo

On 11/14/2017 07:49 PM, Seth Hall wrote:
> It's most likely that you have had offloaded checksums when you
> captures the PCAP.? More information here:
> ????https://www.bro.org/documentation/faq.html#why-isn-t-bro-producing-the-logs-i-expect-a-note-about-checksums
>
>
> ? .Seth
>
> On 14 Nov 2017, at 13:09, BortolameottiR wrote:
>
>> Dear all,
>>
>> I have a simple question. When I run bro against a .pcap file, it
>> happens that some log lines do not show any detail regarding the
>> response e.g., response_body_len, status_msg, status_code, resp_fuids
>> etc. Is it a problem of the HTTP analyzer?
>>
>> I am currently trying to extract all the text/files of all responses,
>> however it seems that some connections responses are not parsed by the
>> HTTP analyzer.
>>
>> I tried to extract the files (following the scripts below), however also
>> in these settings some "files" where missing. In my case I am talking
>> about .css / .html / .js in the response content.
>>
>> -
>> https://www.bro.org/sphinx/scripts/policy/frameworks/files/extract-all-files.bro.html
>>
>>
>> - https://www.bro.org/sphinx-git/httpmonitor/index.html (at the bottom)
>>
>> When you look in conn.log, the same connection (according to the id)
>> shows the amount of bytes of the response. If you inspect the file using
>> Wireshark you can also see that there was a response.?
>>
>> Any idea on what could be the issue?
>>
>> I can even share the .pcap if needed.
>>
>> Best,
>>
>> R.
>>
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> -- 
> Seth Hall * Corelight, Inc * www.corelight.com




From Hendra.Gunadi at murdoch.edu.au  Fri Nov 17 03:28:15 2017
From: Hendra.Gunadi at murdoch.edu.au (Hendra Gunadi)
Date: Fri, 17 Nov 2017 19:28:15 +0800
Subject: [Bro] Covert Channel Detection Framework in Bro (BroCCaDe)
Message-ID: <36dd4ff7-d28a-e42c-a581-00aa78513375@murdoch.edu.au>

Hi All,


We are from Murdoch University in Perth, exploring the opportunity to 
integrate covert channel detection

into an open source IDS. After looking/comparing around some IDS, we 
decided to work with Bro.

Our framework is implemented as a collection of Plugins:

1. Plugin to do a feature extraction such as packets' inter-arrival time

2. Analysis plugin which implements some analysis methods, such as KS 
test, Entropy, CCE, Multi Modality,

 ?? Autocorrelation, and Regularity analysis.

3. Classifier plugin to classify whether a flow contains covert 
communication or not. Currently the only

 ?? classifier we implemented is C4.5 decision tree classifier.

4. Training plugin to train model for the C4.5 decision tree classifier.


If you are interested, please have a look into our project's website and 
let us know what you think

http://www.it.murdoch.edu.au/nsrg/cc_detection_ids/introduction.html


Regards,

Hendra


From dopheide at gmail.com  Fri Nov 17 11:01:51 2017
From: dopheide at gmail.com (Mike Dopheide)
Date: Fri, 17 Nov 2017 13:01:51 -0600
Subject: [Bro] Covert Channel Detection Framework in Bro (BroCCaDe)
In-Reply-To: <36dd4ff7-d28a-e42c-a581-00aa78513375@murdoch.edu.au>
References: <36dd4ff7-d28a-e42c-a581-00aa78513375@murdoch.edu.au>
Message-ID: <CAPy2kFbiWmyvHm-dVeeqzWKd8c3uZ4HpcvDMK9FDPt8KNDFeTQ@mail.gmail.com>

There is a little prior work along these lines, see the second half of this
talk:

https://www.youtube.com/watch?v=OycQ1aiNqEM

It's pretty resource intense.  I don't speak for the development team, but
it kinda felt like the majority of the Bro community didn't think it was
that high of a priority.  At least not for the University and un-classified
lab communities that I talk to.  :)  For Enterprise though, I could see
them potentially wanting to fund some additional work.

-Dop

On Fri, Nov 17, 2017 at 5:28 AM, Hendra Gunadi <Hendra.Gunadi at murdoch.edu.au
> wrote:

> Hi All,
>
>
> We are from Murdoch University in Perth, exploring the opportunity to
> integrate covert channel detection
>
> into an open source IDS. After looking/comparing around some IDS, we
> decided to work with Bro.
>
> Our framework is implemented as a collection of Plugins:
>
> 1. Plugin to do a feature extraction such as packets' inter-arrival time
>
> 2. Analysis plugin which implements some analysis methods, such as KS
> test, Entropy, CCE, Multi Modality,
>
>     Autocorrelation, and Regularity analysis.
>
> 3. Classifier plugin to classify whether a flow contains covert
> communication or not. Currently the only
>
>     classifier we implemented is C4.5 decision tree classifier.
>
> 4. Training plugin to train model for the C4.5 decision tree classifier.
>
>
> If you are interested, please have a look into our project's website and
> let us know what you think
>
> http://www.it.murdoch.edu.au/nsrg/cc_detection_ids/introduction.html
>
>
> Regards,
>
> Hendra
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171117/e8c282e4/attachment.html 

From shirkdog.bsd at gmail.com  Fri Nov 17 11:30:24 2017
From: shirkdog.bsd at gmail.com (Michael Shirk)
Date: Fri, 17 Nov 2017 14:30:24 -0500
Subject: [Bro] Covert Channel Detection Framework in Bro (BroCCaDe)
In-Reply-To: <CAPy2kFbiWmyvHm-dVeeqzWKd8c3uZ4HpcvDMK9FDPt8KNDFeTQ@mail.gmail.com>
References: <36dd4ff7-d28a-e42c-a581-00aa78513375@murdoch.edu.au>
	<CAPy2kFbiWmyvHm-dVeeqzWKd8c3uZ4HpcvDMK9FDPt8KNDFeTQ@mail.gmail.com>
Message-ID: <CAL8PkUXx-gfbwgXMZqqEsD78SqXug-1MeGnVWZfQkGsZ4=2ywQ@mail.gmail.com>

This looks like a good candidate for a bro-pkg so users can test and
submit feedback :)

On Fri, Nov 17, 2017 at 2:01 PM, Mike Dopheide <dopheide at gmail.com> wrote:
> There is a little prior work along these lines, see the second half of this
> talk:
>
> https://www.youtube.com/watch?v=OycQ1aiNqEM
>
> It's pretty resource intense.  I don't speak for the development team, but
> it kinda felt like the majority of the Bro community didn't think it was
> that high of a priority.  At least not for the University and un-classified
> lab communities that I talk to.  :)  For Enterprise though, I could see them
> potentially wanting to fund some additional work.
>
> -Dop
>
> On Fri, Nov 17, 2017 at 5:28 AM, Hendra Gunadi
> <Hendra.Gunadi at murdoch.edu.au> wrote:
>>
>> Hi All,
>>
>>
>> We are from Murdoch University in Perth, exploring the opportunity to
>> integrate covert channel detection
>>
>> into an open source IDS. After looking/comparing around some IDS, we
>> decided to work with Bro.
>>
>> Our framework is implemented as a collection of Plugins:
>>
>> 1. Plugin to do a feature extraction such as packets' inter-arrival time
>>
>> 2. Analysis plugin which implements some analysis methods, such as KS
>> test, Entropy, CCE, Multi Modality,
>>
>>     Autocorrelation, and Regularity analysis.
>>
>> 3. Classifier plugin to classify whether a flow contains covert
>> communication or not. Currently the only
>>
>>     classifier we implemented is C4.5 decision tree classifier.
>>
>> 4. Training plugin to train model for the C4.5 decision tree classifier.
>>
>>
>> If you are interested, please have a look into our project's website and
>> let us know what you think
>>
>> http://www.it.murdoch.edu.au/nsrg/cc_detection_ids/introduction.html
>>
>>
>> Regards,
>>
>> Hendra
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
Michael Shirk
Daemon Security, Inc.
http://www.daemon-security.com

From Hendra.Gunadi at murdoch.edu.au  Sun Nov 19 18:41:24 2017
From: Hendra.Gunadi at murdoch.edu.au (Hendra Gunadi)
Date: Mon, 20 Nov 2017 10:41:24 +0800
Subject: [Bro] Covert Channel Detection Framework in Bro (BroCCaDe)
In-Reply-To: <CAL8PkUXx-gfbwgXMZqqEsD78SqXug-1MeGnVWZfQkGsZ4=2ywQ@mail.gmail.com>
References: <36dd4ff7-d28a-e42c-a581-00aa78513375@murdoch.edu.au>
	<CAPy2kFbiWmyvHm-dVeeqzWKd8c3uZ4HpcvDMK9FDPt8KNDFeTQ@mail.gmail.com>
	<CAL8PkUXx-gfbwgXMZqqEsD78SqXug-1MeGnVWZfQkGsZ4=2ywQ@mail.gmail.com>
Message-ID: <d5e3bbc9-9f8a-e418-2e29-236936710d4d@murdoch.edu.au>

Hi,


@Michael Shirk: That sounds like a good idea, thanks for letting us 
know. We'll work on it :)

@Dop: Thanks for the feedback. Yeah we saw this video and contacted Ross 
for some pointers

 ? on how he did it. But we were not really aware of the research outcome


- Hendra


On 18/11/17 03:30, Michael Shirk wrote:
> This looks like a good candidate for a bro-pkg so users can test and
> submit feedback :)
>
> On Fri, Nov 17, 2017 at 2:01 PM, Mike Dopheide <dopheide at gmail.com> wrote:
>> There is a little prior work along these lines, see the second half of this
>> talk:
>>
>> https://clicktime.symantec.com/a/1/OiOr-JQuH-DXlzUjeFllAFwjf_vTQOHO-IJcheQecjA=?d=Ias3MDtt5fNP1Vh2OVzBed_cZi9R6Je3GPjRkwhZwimEiYQuqGAUjs9vP4uazbdBN_U_0ftctOXEzfF2rRMA-7cNSfvCOTPHGn5UuZbqWX3EbQH9d-qW1JdNf6wu5o7xDrOa_ykihtOm2nIAW1_KIPu0CI_cPxDqppYcydHX2Hm0N-4IncQbwPOF0TPzF8fijKk7TkPvFxLPBk62_aTrwU_BicRK6geepgyCD8A8aqc59qSSzOPmbwasRNgdoOEeIKL22UKZ7qPyEwdwQgak9KeQPADIIN8CFm2SBYWd7ZPYuEka13qv9wD058GX_8VNo1u7qSwkU7v0vqMPDT2r4E5t8KaJn7UzQ57ZyPZOSs5nMtMUm1YF2RwcElilmp6XN49oMeLpQAOkgLRFUKJq1uNdWqWwaHlhNBa6uC5mfMSZTf6a8SvagxOijfE%3D&u=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DOycQ1aiNqEM
>>
>> It's pretty resource intense.  I don't speak for the development team, but
>> it kinda felt like the majority of the Bro community didn't think it was
>> that high of a priority.  At least not for the University and un-classified
>> lab communities that I talk to.  :)  For Enterprise though, I could see them
>> potentially wanting to fund some additional work.
>>
>> -Dop
>>
>> On Fri, Nov 17, 2017 at 5:28 AM, Hendra Gunadi
>> <Hendra.Gunadi at murdoch.edu.au> wrote:
>>> Hi All,
>>>
>>>
>>> We are from Murdoch University in Perth, exploring the opportunity to
>>> integrate covert channel detection
>>>
>>> into an open source IDS. After looking/comparing around some IDS, we
>>> decided to work with Bro.
>>>
>>> Our framework is implemented as a collection of Plugins:
>>>
>>> 1. Plugin to do a feature extraction such as packets' inter-arrival time
>>>
>>> 2. Analysis plugin which implements some analysis methods, such as KS
>>> test, Entropy, CCE, Multi Modality,
>>>
>>>      Autocorrelation, and Regularity analysis.
>>>
>>> 3. Classifier plugin to classify whether a flow contains covert
>>> communication or not. Currently the only
>>>
>>>      classifier we implemented is C4.5 decision tree classifier.
>>>
>>> 4. Training plugin to train model for the C4.5 decision tree classifier.
>>>
>>>
>>> If you are interested, please have a look into our project's website and
>>> let us know what you think
>>>
>>> http://www.it.murdoch.edu.au/nsrg/cc_detection_ids/introduction.html
>>>
>>>
>>> Regards,
>>>
>>> Hendra
>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> https://clicktime.symantec.com/a/1/h8dJ04K2E7G-5RbN44Im-uA9D05p9DVutvnoYOFcMdA=?d=Ias3MDtt5fNP1Vh2OVzBed_cZi9R6Je3GPjRkwhZwimEiYQuqGAUjs9vP4uazbdBN_U_0ftctOXEzfF2rRMA-7cNSfvCOTPHGn5UuZbqWX3EbQH9d-qW1JdNf6wu5o7xDrOa_ykihtOm2nIAW1_KIPu0CI_cPxDqppYcydHX2Hm0N-4IncQbwPOF0TPzF8fijKk7TkPvFxLPBk62_aTrwU_BicRK6geepgyCD8A8aqc59qSSzOPmbwasRNgdoOEeIKL22UKZ7qPyEwdwQgak9KeQPADIIN8CFm2SBYWd7ZPYuEka13qv9wD058GX_8VNo1u7qSwkU7v0vqMPDT2r4E5t8KaJn7UzQ57ZyPZOSs5nMtMUm1YF2RwcElilmp6XN49oMeLpQAOkgLRFUKJq1uNdWqWwaHlhNBa6uC5mfMSZTf6a8SvagxOijfE%3D&u=http%3A%2F%2Fmailman.ICSI.Berkeley.EDU%2Fmailman%2Flistinfo%2Fbro
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> https://clicktime.symantec.com/a/1/h8dJ04K2E7G-5RbN44Im-uA9D05p9DVutvnoYOFcMdA=?d=Ias3MDtt5fNP1Vh2OVzBed_cZi9R6Je3GPjRkwhZwimEiYQuqGAUjs9vP4uazbdBN_U_0ftctOXEzfF2rRMA-7cNSfvCOTPHGn5UuZbqWX3EbQH9d-qW1JdNf6wu5o7xDrOa_ykihtOm2nIAW1_KIPu0CI_cPxDqppYcydHX2Hm0N-4IncQbwPOF0TPzF8fijKk7TkPvFxLPBk62_aTrwU_BicRK6geepgyCD8A8aqc59qSSzOPmbwasRNgdoOEeIKL22UKZ7qPyEwdwQgak9KeQPADIIN8CFm2SBYWd7ZPYuEka13qv9wD058GX_8VNo1u7qSwkU7v0vqMPDT2r4E5t8KaJn7UzQ57ZyPZOSs5nMtMUm1YF2RwcElilmp6XN49oMeLpQAOkgLRFUKJq1uNdWqWwaHlhNBa6uC5mfMSZTf6a8SvagxOijfE%3D&u=http%3A%2F%2Fmailman.ICSI.Berkeley.EDU%2Fmailman%2Flistinfo%2Fbro
>
>


From reswob10 at gmail.com  Mon Nov 20 04:26:56 2017
From: reswob10 at gmail.com (craig bowser)
Date: Mon, 20 Nov 2017 07:26:56 -0500
Subject: [Bro] Rotate logs individually?
In-Reply-To: <CADq7+KBFZtW9rChNOT6rGwGz88=NqaXW2Rz67Mq1it+0BXjCUg@mail.gmail.com>
References: <CADq7+KBFZtW9rChNOT6rGwGz88=NqaXW2Rz67Mq1it+0BXjCUg@mail.gmail.com>
Message-ID: <CADq7+KBKbGcDwdZ2C35vv0wTm_AFtO-f5EYxMZQYU+yBw9xT2Q@mail.gmail.com>

We have one particular bro log that fills up much faster than all the
others.   Is there a way to rotate that one log one a different timetable
than the others?



I found this in the documentation which seems to indicate that it is
possible (the example given is for the conn.log):



https://www.bro.org/sphinx-git/frameworks/logging.html#rotation





event bro_init()

    {

    local f = Log::get_filter(Conn::LOG, "default");

    f$interv = 30 min;

    Log::add_filter(Conn::LOG, f);

    }





Can you put this script into /usr/local/bro/share/bro/site/local.bro to
force only that log to rotate on a different schedule?





Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171120/91387963/attachment.html 

From dnthayer at illinois.edu  Mon Nov 20 05:52:23 2017
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Mon, 20 Nov 2017 07:52:23 -0600
Subject: [Bro] Rotate logs individually?
In-Reply-To: <CADq7+KBKbGcDwdZ2C35vv0wTm_AFtO-f5EYxMZQYU+yBw9xT2Q@mail.gmail.com>
References: <CADq7+KBFZtW9rChNOT6rGwGz88=NqaXW2Rz67Mq1it+0BXjCUg@mail.gmail.com>
	<CADq7+KBKbGcDwdZ2C35vv0wTm_AFtO-f5EYxMZQYU+yBw9xT2Q@mail.gmail.com>
Message-ID: <16c04d2e-fe50-564f-9eec-fe110abc6009@illinois.edu>

It works for me.


On 11/20/17 6:26 AM, craig bowser wrote:
> We have one particular bro log that fills up much faster than all the 
> others.? ?Is there a way to rotate that one log one a different 
> timetable than the others?____
> 
> __ __
> 
> I found this in the documentation which seems to indicate that it is 
> possible (the example given is for the conn.log):____
> 
> __ __
> 
> https://www.bro.org/sphinx-git/frameworks/logging.html#rotation 
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.bro.org_sphinx-2Dgit_frameworks_logging.html-23rotation&d=DwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Bi5qPBnY0NmYPqnRTPj_AfXQKpfQTZUpCzpfFBcawv0&m=L8hn8vIcjCTKcv6EA6is69KSOcGUeSnlGtsLQBtf5RU&s=ul_hpaORoyS57rQ4txkikkfl3TYQBACXViS7THruN30&e=>____
> 
> __ __
> 
> __ __
> 
> event bro_init()____
> 
>  ??? {____
> 
>  ??? local f = Log::get_filter(Conn::LOG, "default");____
> 
>  ??? f$interv = 30 min;____
> 
>  ??? Log::add_filter(Conn::LOG, f);____
> 
>  ??? }____
> 
> __ __
> 
> __ __
> 
> Can you put this script into /usr/local/bro/share/bro/site/local.bro to 
> force only that log to rotate on a different schedule? ____
> 
> __ __
> 
> __ __
> 
> Thanks.

From samjohnson5678 at outlook.com  Fri Nov 24 07:22:27 2017
From: samjohnson5678 at outlook.com (Sam Johnson)
Date: Fri, 24 Nov 2017 15:22:27 +0000
Subject: [Bro] Bro - Machine Learning Project
Message-ID: <AM0PR0202MB3505898AFA0AAF2603B9A7AEDE260@AM0PR0202MB3505.eurprd02.prod.outlook.com>

Hi,

I am an intern student at Lancaster University in the UK. I am undertaking a project to determine the feasibility of developing an intrusion detection system with a machine learning engine rather than a signature based system. Similar to the Darktrace product that sells for many thousands of pounds but a scaled down version. I thought about implementing it on Bro and developing it as an add-in? Do you think this would be possible, I have some experience of programming but not lots although my coding is tidy and well commented. What is Bro written in? Would I have to use that language? Do you think it is possible?

Thank you, Sam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171124/6d4c61e1/attachment.html 

From anthony.kasza at gmail.com  Fri Nov 24 09:04:19 2017
From: anthony.kasza at gmail.com (anthony kasza)
Date: Fri, 24 Nov 2017 10:04:19 -0700
Subject: [Bro] Bro - Machine Learning Project
In-Reply-To: <AM0PR0202MB3505898AFA0AAF2603B9A7AEDE260@AM0PR0202MB3505.eurprd02.prod.outlook.com>
References: <AM0PR0202MB3505898AFA0AAF2603B9A7AEDE260@AM0PR0202MB3505.eurprd02.prod.outlook.com>
Message-ID: <CAEZw2bwVcgNCshNj42h+NmbF4mk9trVMUZS3iT4jxdZpBW1iAg@mail.gmail.com>

Hi Sam,

I would read this paper first.
https://www.icir.org/robin/papers/oakland10-ml.pdf

-AK

On Nov 24, 2017 08:37, "Sam Johnson" <samjohnson5678 at outlook.com> wrote:

> Hi,
>
>
>
> I am an intern student at Lancaster University in the UK. I am undertaking
> a project to determine the feasibility of developing an intrusion detection
> system with a machine learning engine rather than a signature based system.
> Similar to the Darktrace product that sells for many thousands of pounds
> but a scaled down version. I thought about implementing it on Bro and
> developing it as an add-in? Do you think this would be possible, I have
> some experience of programming but not lots although my coding is tidy and
> well commented. What is Bro written in? Would I have to use that language?
> Do you think it is possible?
>
>
>
> Thank you, Sam
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171124/6245cbf7/attachment.html 

From briford.wylie at gmail.com  Fri Nov 24 10:38:31 2017
From: briford.wylie at gmail.com (Brian Wylie)
Date: Fri, 24 Nov 2017 11:38:31 -0700
Subject: [Bro] Bro - Machine Learning Project
In-Reply-To: <AM0PR0202MB3505898AFA0AAF2603B9A7AEDE260@AM0PR0202MB3505.eurprd02.prod.outlook.com>
References: <AM0PR0202MB3505898AFA0AAF2603B9A7AEDE260@AM0PR0202MB3505.eurprd02.prod.outlook.com>
Message-ID: <CAOp_npCteqVSAypQyet9g73DDOueQU9MPmfv3_8a7=vE=Nft6g@mail.gmail.com>

Given the popularity and depth of the machine learning in Python
(scikit-learn, statsmodels, Keras, pyspark) you might consider 'offloading'
the data analysis and machine learning into a Python processes (set of
processes). The BAT Bro Analysis Tools might be a good place to start
https://github.com/Kitware/bat. Obviously I'm biased but the intention of
the repository is to 'bridge' from Bro to these machine learning libraries
so at a minimum it's worth looking at before diving in.

Cheers,
-Brian

On Fri, Nov 24, 2017 at 8:22 AM, Sam Johnson <samjohnson5678 at outlook.com>
wrote:

> Hi,
>
>
>
> I am an intern student at Lancaster University in the UK. I am undertaking
> a project to determine the feasibility of developing an intrusion detection
> system with a machine learning engine rather than a signature based system.
> Similar to the Darktrace product that sells for many thousands of pounds
> but a scaled down version. I thought about implementing it on Bro and
> developing it as an add-in? Do you think this would be possible, I have
> some experience of programming but not lots although my coding is tidy and
> well commented. What is Bro written in? Would I have to use that language?
> Do you think it is possible?
>
>
>
> Thank you, Sam
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171124/e0ba1511/attachment.html 

From Lewis_David at bah.com  Fri Nov 24 13:39:40 2017
From: Lewis_David at bah.com (Lewis, David [USA])
Date: Fri, 24 Nov 2017 21:39:40 +0000
Subject: [Bro] Bro Logs Sending to Kafka
Message-ID: <0FF5BACE-7FE1-41EE-B208-F9893AF1443D@contoso.com>

Hi all,

Apologies if this is not the place to post this question. If it is not, please re-direct me to the correct location.
Question:
I?ve added the following code (below ? host:port says an actual host and port number) to local.bro in order to send 2 different logs (DNS and CONN) to 3 different Kafka topics.
However, the conn logs being sent to one of the kafka topics is sending messages very slowly (~single digit messages per batch) while the other conn logs are sending to the other kafka topic very quickly (~2 thousand message per batch). Why is this happening?

To follow up, is there an optimal batch size to max.ms in order to send thousands of batched message to different Kafka topics. I?ve been able to max out at around 2-3K per batch even if I increase the max ms to 30000.

To run: sudo bro ?i lo /path/to/local.bro

Code:

@load Bro/Kafka/logs-to-kafka.bro
#redef Kafka::logs_to_send = set(Conn::LOG);
#redef LogAscii::include_meta T;
#redef LogAscii::separator ",";
redef Kafka::topic_name = "";
#redef Kafka::tag_json = T;
#redef Kafka::max_wait_on_shutdown = 3000;
redef Kafka::kafka_conf = table(
    ["metadata.broker.list"] = "host:port", #host:port
    ["debug"] = "generic, broker, topic, msg", #debugging lines
    ["queue.buffering.max.messages"] = "5000", #send in batches of this number
    ["queue.buffering.max.ms"] = "3500" #wait this many ms until sending to topic
);


#init function to name kafka topic (path), set KAFKAWRITER, and receive only LOG files
event bro_init()
{
        #Conn logs sent straight to
        Log::add_filter(Conn::LOG, [
                $name = "kafka-conn",
                $writer = Log::WRITER_KAFKAWRITER,
                $path = "conn-logs-batches"
        ]);
        #Conn logs sent to
        Log::add_filter(Conn::LOG, [
                $name = "kafka-conn-kate",
                $writer = Log::WRITER_KAFKAWRITER,
                $path = "conn-logs-kate"
        ]);
        #DNS logs to DGA
        Log::add_filter(DNS::LOG, [
                $name = "kafka-dns",
                $writer = Log::WRITER_KAFKAWRITER,
                $path = "dns-logs-dga"
        ]);

}


Thank you,
David


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171124/bb89b456/attachment.html 

From mekici at netas.com.tr  Mon Nov 27 04:24:29 2017
From: mekici at netas.com.tr (Mehmet EKICI)
Date: Mon, 27 Nov 2017 12:24:29 +0000
Subject: [Bro] bro http message verbosity
Message-ID: <DB5PR03MB173464762AC1CA1E1E2C209CF0250@DB5PR03MB1734.eurprd03.prod.outlook.com>

Hi All,
We are trying to use bro to monitor http messages on a wire. We are getting very coarse logs and wonder how can we increase verbosity to see all the parsed message details in the log.

Bro version is 2.4.1

Here are some example  messages we get;

{"ts":"2017-11-27T12:14:29.850476Z","uid":"CvtiHbXu0dt9pdFMa","id.orig_h":"10.2.150.237","id.orig_p":42798,"id.resp_h":"10.2.150.226","id.resp_p":9441,"name":"inappropriate_FIN","notice":false,"peer":"bro"}
{"ts":"2017-11-27T12:14:33.578491Z","uid":"CuGJzp3JYtJLxu3NN1","id.orig_h":"10.2.150.226","id.orig_p":54376,"id.resp_h":"10.2.150.228","id.resp_p":6188,"name":"active_connection_reuse","notice":false,"peer":"bro"}
{"ts":"2017-11-27T12:14:33.578491Z","uid":"COJykR3r39KwcvIPae","id.orig_h":"10.2.150.228","id.orig_p":6188,"id.resp_h":"10.2.150.226","id.resp_p":54376,"name":"data_before_established","notice":false,"peer":"bro"}
{"ts":"2017-11-27T12:14:41.454466Z","uid":"CuKy3C1TkabD0KvC26","id.orig_h":"10.2.150.227","id.orig_p":38672,"id.resp_h":"10.2.150.226","id.resp_p":8020,"name":"data_before_established","notice":false,"peer":"bro"}
{"ts":"2017-11-27T12:14:43.578437Z","uid":"CFLCdn2bwXadx8g0al","id.orig_h":"10.2.150.226","id.orig_p":54378,"id.resp_h":"10.2.150.228","id.resp_p":6188,"name":"active_connection_reuse","notice":false,"peer":"bro"}
{"ts":"2017-11-27T12:14:43.578437Z","uid":"CKZBqUlmUlkdtvMDd","id.orig_h":"10.2.150.228","id.orig_p":6188,"id.resp_h":"10.2.150.226","id.resp_p":54378,"name":"data_before_established","notice":false,"peer":"bro"}

Regards,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171127/0006ae07/attachment.html 

From avz at dvo.ru  Mon Nov 27 04:58:38 2017
From: avz at dvo.ru (Alexander Zatserkovniy)
Date: Mon, 27 Nov 2017 22:58:38 +1000
Subject: [Bro] 10Gbps bro + netmap + ixgbe + lb?
Message-ID: <5cf950bb-0f34-ee59-b582-abb92e34a0d9@dvo.ru>

Hi!

I have a Bro cluster on a 2xE5-2660 with 64GB RAM. The traffic comes via
a 10Gbps port of an Intel 82599ES (it's not cleared from FIN_storms and
so on). I use Netmap with the Netmap ixgbe drivers and lb. lb and Bro
workers pinned to cores. The interface offloads are turned off (ethtool
-L eth0 combined 1; ... ). The basic installation (1xlb, 20 cores for
Bro) begins drops packets hard near 7Gbps (~900Kpps). It looks like the
bottleneck was the lb (100% the core usage and it shows drops). Lite lb
optimisation ( I unify the loops in sym_hash_fn) doesn't help too much.

I try the following lb pyramid:
lb -i eth0 -B10000 -p bro:2 -o 60
lb -i netmap:bro}0 -B10000 -p broa:13 -o 60
lb -i netmap:bro}1 -B10000 -p brob:13 -o 60
and :
...
[worker-1]
type=worker
host=localhost
interface=netmap::broa
lb_method=custom
lb_procs=13
pin_cpus=7,8,9,10,11,12,13,14,15,16,17,18,19

[worker-2]
type=worker
host=localhost
interface=netmap::brob
lb_method=custom
lb_procs=13
pin_cpus=22,23,24,25,26,27,28,29,30,31,32,33,34
in node.cfg .

Now I can get up to 8Gbps (~1000Kpps) without a lot of losses.
But is it possible to get full 10Gbps on a port with netmap and lb on a
host like mine? How to offload lb?

Thanks,
Alexander Zatserkovnyy

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171127/05abdc8c/attachment.bin 

From michalpurzynski1 at gmail.com  Mon Nov 27 05:21:08 2017
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Mon, 27 Nov 2017 14:21:08 +0100
Subject: [Bro] 10Gbps bro + netmap + ixgbe + lb?
In-Reply-To: <5cf950bb-0f34-ee59-b582-abb92e34a0d9@dvo.ru>
References: <5cf950bb-0f34-ee59-b582-abb92e34a0d9@dvo.ru>
Message-ID: <1C1D3234-3489-4588-A01F-FC0D40BAB4E8@gmail.com>

Do you have to use net map? 

> On Nov 27, 2017, at 1:58 PM, Alexander Zatserkovniy <avz at dvo.ru> wrote:
> 
> Hi!
> 
> I have a Bro cluster on a 2xE5-2660 with 64GB RAM. The traffic comes via
> a 10Gbps port of an Intel 82599ES (it's not cleared from FIN_storms and
> so on). I use Netmap with the Netmap ixgbe drivers and lb. lb and Bro
> workers pinned to cores. The interface offloads are turned off (ethtool
> -L eth0 combined 1; ... ). The basic installation (1xlb, 20 cores for
> Bro) begins drops packets hard near 7Gbps (~900Kpps). It looks like the
> bottleneck was the lb (100% the core usage and it shows drops). Lite lb
> optimisation ( I unify the loops in sym_hash_fn) doesn't help too much.
> 
> I try the following lb pyramid:
> lb -i eth0 -B10000 -p bro:2 -o 60
> lb -i netmap:bro}0 -B10000 -p broa:13 -o 60
> lb -i netmap:bro}1 -B10000 -p brob:13 -o 60
> and :
> ...
> [worker-1]
> type=worker
> host=localhost
> interface=netmap::broa
> lb_method=custom
> lb_procs=13
> pin_cpus=7,8,9,10,11,12,13,14,15,16,17,18,19
> 
> [worker-2]
> type=worker
> host=localhost
> interface=netmap::brob
> lb_method=custom
> lb_procs=13
> pin_cpus=22,23,24,25,26,27,28,29,30,31,32,33,34
> in node.cfg .
> 
> Now I can get up to 8Gbps (~1000Kpps) without a lot of losses.
> But is it possible to get full 10Gbps on a port with netmap and lb on a
> host like mine? How to offload lb?
> 
> Thanks,
> Alexander Zatserkovnyy
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

From jazoff at illinois.edu  Mon Nov 27 09:16:30 2017
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Mon, 27 Nov 2017 17:16:30 +0000
Subject: [Bro] bro http message verbosity
In-Reply-To: <DB5PR03MB173464762AC1CA1E1E2C209CF0250@DB5PR03MB1734.eurprd03.prod.outlook.com>
References: <DB5PR03MB173464762AC1CA1E1E2C209CF0250@DB5PR03MB1734.eurprd03.prod.outlook.com>
Message-ID: <B77DDD84-7E16-4AEB-B956-2B98C45081FC@illinois.edu>


> On Nov 27, 2017, at 7:24 AM, Mehmet EKICI <mekici at netas.com.tr> wrote:
> 
> Hi All,
> We are trying to use bro to monitor http messages on a wire. We are getting very coarse logs and wonder how can we increase verbosity to see all the parsed message details in the log.
>  
> Bro version is 2.4.1

2.4.1 is over 2 years old at this point, You should be on 2.5.x, or minimally, 2.4.2

>  
> Here are some example  messages we get;
>  
> {"ts":"2017-11-27T12:14:29.850476Z","uid":"CvtiHbXu0dt9pdFMa","id.orig_h":"10.2.150.237","id.orig_p":42798,"id.resp_h":"10.2.150.226","id.resp_p":9441,"name":"inappropriate_FIN","notice":false,"peer":"bro"}
> {"ts":"2017-11-27T12:14:33.578491Z","uid":"CuGJzp3JYtJLxu3NN1","id.orig_h":"10.2.150.226","id.orig_p":54376,"id.resp_h":"10.2.150.228","id.resp_p":6188,"name":"active_connection_reuse","notice":false,"peer":"bro"}
> {"ts":"2017-11-27T12:14:33.578491Z","uid":"COJykR3r39KwcvIPae","id.orig_h":"10.2.150.228","id.orig_p":6188,"id.resp_h":"10.2.150.226","id.resp_p":54376,"name":"data_before_established","notice":false,"peer":"bro"}
> {"ts":"2017-11-27T12:14:41.454466Z","uid":"CuKy3C1TkabD0KvC26","id.orig_h":"10.2.150.227","id.orig_p":38672,"id.resp_h":"10.2.150.226","id.resp_p":8020,"name":"data_before_established","notice":false,"peer":"bro"}
> {"ts":"2017-11-27T12:14:43.578437Z","uid":"CFLCdn2bwXadx8g0al","id.orig_h":"10.2.150.226","id.orig_p":54378,"id.resp_h":"10.2.150.228","id.resp_p":6188,"name":"active_connection_reuse","notice":false,"peer":"bro"}
> {"ts":"2017-11-27T12:14:43.578437Z","uid":"CKZBqUlmUlkdtvMDd","id.orig_h":"10.2.150.228","id.orig_p":6188,"id.resp_h":"10.2.150.226","id.resp_p":54378,"name":"data_before_established","notice":false,"peer":"bro"}
> 

Well, that's the weird.log, not the http.log.  The http.log will have http related entries.  If you're still not seeing what you expect there, it's probably because of

https://www.bro.org/documentation/faq.html#why-isn-t-bro-producing-the-logs-i-expect-a-note-about-checksums





? 
Justin Azoff




From tobias.brunnwieser at ppro.com  Mon Nov 27 10:24:23 2017
From: tobias.brunnwieser at ppro.com (Tobias Brunnwieser)
Date: Mon, 27 Nov 2017 19:24:23 +0100
Subject: [Bro] No log when bro script is run with broctl
Message-ID: <aeeae8af-9695-06f0-101a-3a19bb44dfc3@ppro.com>

Hi all,

I have a simple Bro script that records SSL-related info to a own log.
The log is created with Log::create_stream, as described on this page:
https://www.bro.org/sphinx/frameworks/logging.html.
When running Bro directly from command line with:

bro -C -b -i eth0 script.bro

The expected log file is written in the directory in which Bro is run
(together with some logs that are produced automatically, like ssl.log).
But when I try to do the same through a broctl setup, I dont get a log
any more. Here's my configuration:
	- modified sites/local.bro to include only my script
	- broctl config files are otherwise only slightly modified, using the
standard setup of a single, local node
	- used broctl for deploying and starting node
It seems that the script is properly deployed to the node, at least the
diag command from broctl shows that the script got loaded (from the
spooling location). But I do not get the log that the script is supposed
to produce. All other logs (like ssl.log) are created ordinarily and are
found in the expected location.
My question is: do I miss something, is there a difference between the
scripts shipped with Bro and my deployed script that prevents it from
writing logs? Or is this most likely a bug?

Another question: what are the prerequisites of broctl? Apparently you
can build it without Broker and Broccoli, but then some commands from
broctl do not work.

Any help is appreciated,

Tobi

PS. I work with a slightly patched version of Bro, you'll find the patch
here: https://bro-tracker.atlassian.net/browse/BIT-1855. But it
shouldn't make a difference, it only adds an additional parameter to the
ssl_client_hello event. As explained previously, when running Bro from
command line, I do not have any issues, even with the patched version.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171127/b9c6d8dc/attachment-0001.bin 

From johanna at icir.org  Mon Nov 27 18:57:21 2017
From: johanna at icir.org (Johanna Amann)
Date: Mon, 27 Nov 2017 18:57:21 -0800
Subject: [Bro] No log when bro script is run with broctl
In-Reply-To: <aeeae8af-9695-06f0-101a-3a19bb44dfc3@ppro.com>
References: <aeeae8af-9695-06f0-101a-3a19bb44dfc3@ppro.com>
Message-ID: <20171128025721.f6mblhd73je3zly3@Trafalgar.local>

Hi,

On Mon, Nov 27, 2017 at 07:24:23PM +0100, Tobias Brunnwieser wrote:
> Hi all,
> 
> I have a simple Bro script that records SSL-related info to a own log.
> The log is created with Log::create_stream, as described on this page:
> https://www.bro.org/sphinx/frameworks/logging.html.
> When running Bro directly from command line with:
> 
> bro -C -b -i eth0 script.bro
> 
> The expected log file is written in the directory in which Bro is run
> (together with some logs that are produced automatically, like ssl.log).
> But when I try to do the same through a broctl setup, I dont get a log
> any more. Here's my configuration:
> 	- modified sites/local.bro to include only my script
> 	- broctl config files are otherwise only slightly modified, using the
> standard setup of a single, local node
> 	- used broctl for deploying and starting node
> It seems that the script is properly deployed to the node, at least the
> diag command from broctl shows that the script got loaded (from the
> spooling location). But I do not get the log that the script is supposed
> to produce. All other logs (like ssl.log) are created ordinarily and are
> found in the expected location.
> My question is: do I miss something, is there a difference between the
> scripts shipped with Bro and my deployed script that prevents it from
> writing logs? Or is this most likely a bug?

This sounds a bit odd and I am not really sure what you are doing wrong.
My first intuition would be to do a full stop of all nodes that broctl is
running, and then do a deploy again just to make sure that the logger,
manager, and worker nodes are really all running the same scripts.

Apart from that, to answer your question - the scripts that are shipped
with Bro are not handled differently from scripts that you write. They use
the exact same mechanics to create log files - so if it works for them it
should work for you.

It also is always worthwhile to check reporter.log for script error
messages that creep up.

> Another question: what are the prerequisites of broctl? Apparently you
> can build it without Broker and Broccoli, but then some commands from
> broctl do not work.

Other people are probably more qualified to answer this question - but
currently Bro is typically built without broker support (one has to
specifically enable it) and with broccoli support. Without broccoli you
miss some features like being able to directly access the value of
variables; however the base functionality should still work.

I hope this helps,
 Johanna

From mekici at netas.com.tr  Mon Nov 27 21:45:50 2017
From: mekici at netas.com.tr (Mehmet EKICI)
Date: Tue, 28 Nov 2017 05:45:50 +0000
Subject: [Bro] bro http message verbosity
In-Reply-To: <DB5PR03MB173464762AC1CA1E1E2C209CF0250@DB5PR03MB1734.eurprd03.prod.outlook.com>
References: <DB5PR03MB173464762AC1CA1E1E2C209CF0250@DB5PR03MB1734.eurprd03.prod.outlook.com>
Message-ID: <DB5PR03MB1734AC83929FDFC7AE475CCCF03A0@DB5PR03MB1734.eurprd03.prod.outlook.com>

Hi All,
Ping ?

From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Mehmet EKICI
Sent: Monday, November 27, 2017 3:24 PM
To: bro at bro.org
Subject: [Bro] bro http message verbosity

Hi All,
We are trying to use bro to monitor http messages on a wire. We are getting very coarse logs and wonder how can we increase verbosity to see all the parsed message details in the log.

Bro version is 2.4.1

Here are some example  messages we get;

{"ts":"2017-11-27T12:14:29.850476Z","uid":"CvtiHbXu0dt9pdFMa","id.orig_h":"10.2.150.237","id.orig_p":42798,"id.resp_h":"10.2.150.226","id.resp_p":9441,"name":"inappropriate_FIN","notice":false,"peer":"bro"}
{"ts":"2017-11-27T12:14:33.578491Z","uid":"CuGJzp3JYtJLxu3NN1","id.orig_h":"10.2.150.226","id.orig_p":54376,"id.resp_h":"10.2.150.228","id.resp_p":6188,"name":"active_connection_reuse","notice":false,"peer":"bro"}
{"ts":"2017-11-27T12:14:33.578491Z","uid":"COJykR3r39KwcvIPae","id.orig_h":"10.2.150.228","id.orig_p":6188,"id.resp_h":"10.2.150.226","id.resp_p":54376,"name":"data_before_established","notice":false,"peer":"bro"}
{"ts":"2017-11-27T12:14:41.454466Z","uid":"CuKy3C1TkabD0KvC26","id.orig_h":"10.2.150.227","id.orig_p":38672,"id.resp_h":"10.2.150.226","id.resp_p":8020,"name":"data_before_established","notice":false,"peer":"bro"}
{"ts":"2017-11-27T12:14:43.578437Z","uid":"CFLCdn2bwXadx8g0al","id.orig_h":"10.2.150.226","id.orig_p":54378,"id.resp_h":"10.2.150.228","id.resp_p":6188,"name":"active_connection_reuse","notice":false,"peer":"bro"}
{"ts":"2017-11-27T12:14:43.578437Z","uid":"CKZBqUlmUlkdtvMDd","id.orig_h":"10.2.150.228","id.orig_p":6188,"id.resp_h":"10.2.150.226","id.resp_p":54378,"name":"data_before_established","notice":false,"peer":"bro"}

Regards,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171128/b9b97ada/attachment.html 

From tobias.brunnwieser at ppro.com  Tue Nov 28 00:47:15 2017
From: tobias.brunnwieser at ppro.com (Tobias Brunnwieser)
Date: Tue, 28 Nov 2017 09:47:15 +0100
Subject: [Bro] No log when bro script is run with broctl
In-Reply-To: <20171128025721.f6mblhd73je3zly3@Trafalgar.local>
References: <aeeae8af-9695-06f0-101a-3a19bb44dfc3@ppro.com>
	<20171128025721.f6mblhd73je3zly3@Trafalgar.local>
Message-ID: <b8cc9e64-148f-e6ef-c26c-0f05d183e763@ppro.com>

Hi,

On 28.11.2017 03:57, Johanna Amann wrote:

> This sounds a bit odd and I am not really sure what you are doing wrong.
> My first intuition would be to do a full stop of all nodes that broctl is
> running, and then do a deploy again just to make sure that the logger,
> manager, and worker nodes are really all running the same scripts.

I did that several times, that was not the issue.

> It also is always worthwhile to check reporter.log for script error
> messages that creep up.

This was a good hint: it reported checksum errors. Since I explicitly
disabled checksums on the command line with -C, I added ignore_checksums
in bro.local and it worked. Probably I missed just the ssl_established
event that triggers the log write. This is weird, since the default SSL
analyzer makes use of it, too, and its log got written...

Thanks for your help!

Tobias

From jdopheid at illinois.edu  Tue Nov 28 06:26:17 2017
From: jdopheid at illinois.edu (Dopheide, Jeannette M)
Date: Tue, 28 Nov 2017 14:26:17 +0000
Subject: [Bro] bro http message verbosity
Message-ID: <C735B606-FC75-4DB8-906C-3670FFDC645E@illinois.edu>

Hello, Justin replied to this message yesterday:

On 11/27/17, 11:16 AM, "bro-bounces at bro.org on behalf of Azoff, Justin S" <bro-bounces at bro.org on behalf of jazoff at illinois.edu> wrote:

> On Nov 27, 2017, at 7:24 AM, Mehmet EKICI <mekici at netas.com.tr> wrote:
>
> Hi All,
> We are trying to use bro to monitor http messages on a wire. We are getting very coarse logs and wonder how can we increase verbosity to see all the parsed message details in the log.
>
> Bro version is 2.4.1

2.4.1 is over 2 years old at this point, You should be on 2.5.x, or minimally, 2.4.2

>
> Here are some example  messages we get;
>
> {"ts":"2017-11-27T12:14:29.850476Z","uid":"CvtiHbXu0dt9pdFMa","id.orig_h":"10.2.150.237","id.orig_p":42798,"id.resp_h":"10.2.150.226","id.resp_p":9441,"name":"inappropriate_FIN","notice":false,"peer":"bro"}
> {"ts":"2017-11-27T12:14:33.578491Z","uid":"CuGJzp3JYtJLxu3NN1","id.orig_h":"10.2.150.226","id.orig_p":54376,"id.resp_h":"10.2.150.228","id.resp_p":6188,"name":"active_connection_reuse","notice":false,"peer":"bro"}
> {"ts":"2017-11-27T12:14:33.578491Z","uid":"COJykR3r39KwcvIPae","id.orig_h":"10.2.150.228","id.orig_p":6188,"id.resp_h":"10.2.150.226","id.resp_p":54376,"name":"data_before_established","notice":false,"peer":"bro"}
> {"ts":"2017-11-27T12:14:41.454466Z","uid":"CuKy3C1TkabD0KvC26","id.orig_h":"10.2.150.227","id.orig_p":38672,"id.resp_h":"10.2.150.226","id.resp_p":8020,"name":"data_before_established","notice":false,"peer":"bro"}
> {"ts":"2017-11-27T12:14:43.578437Z","uid":"CFLCdn2bwXadx8g0al","id.orig_h":"10.2.150.226","id.orig_p":54378,"id.resp_h":"10.2.150.228","id.resp_p":6188,"name":"active_connection_reuse","notice":false,"peer":"bro"}
> {"ts":"2017-11-27T12:14:43.578437Z","uid":"CKZBqUlmUlkdtvMDd","id.orig_h":"10.2.150.228","id.orig_p":6188,"id.resp_h":"10.2.150.226","id.resp_p":54378,"name":"data_before_established","notice":false,"peer":"bro"}
>

Well, that's the weird.log, not the http.log.  The http.log will have http related entries.  If you're still not seeing what you expect there, it's probably because of

https://www.bro.org/documentation/faq.html#why-isn-t-bro-producing-the-logs-i-expect-a-note-about-checksums



------
Jeannette Dopheide
Sr. Education, Outreach, and Training Coordinator
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign


From: <bro-bounces at bro.org> on behalf of Mehmet EKICI <mekici at netas.com.tr>
Date: Monday, November 27, 2017 at 11:45 PM
To: "bro at bro.org" <bro at bro.org>
Subject: Re: [Bro] bro http message verbosity

Hi All,
Ping ?

From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Mehmet EKICI
Sent: Monday, November 27, 2017 3:24 PM
To: bro at bro.org
Subject: [Bro] bro http message verbosity

Hi All,
We are trying to use bro to monitor http messages on a wire. We are getting very coarse logs and wonder how can we increase verbosity to see all the parsed message details in the log.

Bro version is 2.4.1

Here are some example  messages we get;

{"ts":"2017-11-27T12:14:29.850476Z","uid":"CvtiHbXu0dt9pdFMa","id.orig_h":"10.2.150.237","id.orig_p":42798,"id.resp_h":"10.2.150.226","id.resp_p":9441,"name":"inappropriate_FIN","notice":false,"peer":"bro"}
{"ts":"2017-11-27T12:14:33.578491Z","uid":"CuGJzp3JYtJLxu3NN1","id.orig_h":"10.2.150.226","id.orig_p":54376,"id.resp_h":"10.2.150.228","id.resp_p":6188,"name":"active_connection_reuse","notice":false,"peer":"bro"}
{"ts":"2017-11-27T12:14:33.578491Z","uid":"COJykR3r39KwcvIPae","id.orig_h":"10.2.150.228","id.orig_p":6188,"id.resp_h":"10.2.150.226","id.resp_p":54376,"name":"data_before_established","notice":false,"peer":"bro"}
{"ts":"2017-11-27T12:14:41.454466Z","uid":"CuKy3C1TkabD0KvC26","id.orig_h":"10.2.150.227","id.orig_p":38672,"id.resp_h":"10.2.150.226","id.resp_p":8020,"name":"data_before_established","notice":false,"peer":"bro"}
{"ts":"2017-11-27T12:14:43.578437Z","uid":"CFLCdn2bwXadx8g0al","id.orig_h":"10.2.150.226","id.orig_p":54378,"id.resp_h":"10.2.150.228","id.resp_p":6188,"name":"active_connection_reuse","notice":false,"peer":"bro"}
{"ts":"2017-11-27T12:14:43.578437Z","uid":"CKZBqUlmUlkdtvMDd","id.orig_h":"10.2.150.228","id.orig_p":6188,"id.resp_h":"10.2.150.226","id.resp_p":54378,"name":"data_before_established","notice":false,"peer":"bro"}

Regards,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171128/c62d46ff/attachment-0001.html 

From roberixion at gmail.com  Wed Nov 29 04:12:58 2017
From: roberixion at gmail.com (=?UTF-8?Q?Rober_Fern=C3=A1ndez?=)
Date: Wed, 29 Nov 2017 13:12:58 +0100
Subject: [Bro] Broccoli-Python
Message-ID: <CAL71vfW45E532q+1qOi+g87aQ9iYAVuRogN44rt9UnyOd7cq2g@mail.gmail.com>

When execute  python -c "import broccoli" , python give me this error

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.5/dist-packages/broccoli.py", line 6, in
<module>
    from broccoli_intern import *
ImportError: No module named 'broccoli_intern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171129/077cb27c/attachment.html 

From dopheide at gmail.com  Wed Nov 29 05:57:08 2017
From: dopheide at gmail.com (Mike Dopheide)
Date: Wed, 29 Nov 2017 13:57:08 +0000
Subject: [Bro] Broccoli-Python
In-Reply-To: <CAL71vfW45E532q+1qOi+g87aQ9iYAVuRogN44rt9UnyOd7cq2g@mail.gmail.com>
References: <CAL71vfW45E532q+1qOi+g87aQ9iYAVuRogN44rt9UnyOd7cq2g@mail.gmail.com>
Message-ID: <CAPy2kFZG5pmCZZebahZ3Xb94UnkjzE2yn315Daq_huZviyn=yw@mail.gmail.com>

Follow up question for the list, I noticed Rober is using Python 3, is
broccoli going to work with that?

On Wed, Nov 29, 2017 at 6:25 AM Rober Fern?ndez <roberixion at gmail.com>
wrote:

> When execute  python -c "import broccoli" , python give me this error
>
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "/usr/local/lib/python3.5/dist-packages/broccoli.py", line 6, in
> <module>
>     from broccoli_intern import *
> ImportError: No module named 'broccoli_intern
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171129/ec78e150/attachment.html 

From dnthayer at illinois.edu  Wed Nov 29 09:11:23 2017
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Wed, 29 Nov 2017 11:11:23 -0600
Subject: [Bro] Broccoli-Python
In-Reply-To: <CAL71vfW45E532q+1qOi+g87aQ9iYAVuRogN44rt9UnyOd7cq2g@mail.gmail.com>
References: <CAL71vfW45E532q+1qOi+g87aQ9iYAVuRogN44rt9UnyOd7cq2g@mail.gmail.com>
Message-ID: <85dd3dcc-65c6-98e5-6bde-61d9fd9762e5@illinois.edu>

On 11/29/17 6:12 AM, Rober Fern?ndez wrote:
> When execute? python -c "import broccoli" , python give me this error
> 
> Traceback (most recent call last):
>  ? File "<string>", line 1, in <module>
>  ? File "/usr/local/lib/python3.5/dist-packages/broccoli.py", line 6, in 
> <module>
>  ??? from broccoli_intern import *
> ImportError: No module named 'broccoli_intern
> 
> 

Does your system have both Python 2.x and Python 3.x installed?

When using any of the Bro python components, you must use the
same version of Python that was used when building Bro (for example,
if you build Bro with Python 3.5, then you must use Python 3.5 when
running any of the Bro python components).

You could try something like this:
   python3 -c "import broccoli"

If that doesn't work, you could try exporting
the PYTHONPATH environment variable like this:
export PYTHONPATH=PREFIX/lib/broctl

Replace "PREFIX" with your Bro install prefix
directory.

Also check that there is a file "broccoli_intern.py"
installed somewhere (it is normally in PREFIX/lib/broctl/).


From dnthayer at illinois.edu  Wed Nov 29 09:18:45 2017
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Wed, 29 Nov 2017 11:18:45 -0600
Subject: [Bro] Broccoli-Python
In-Reply-To: <CAPy2kFZG5pmCZZebahZ3Xb94UnkjzE2yn315Daq_huZviyn=yw@mail.gmail.com>
References: <CAL71vfW45E532q+1qOi+g87aQ9iYAVuRogN44rt9UnyOd7cq2g@mail.gmail.com>
	<CAPy2kFZG5pmCZZebahZ3Xb94UnkjzE2yn315Daq_huZviyn=yw@mail.gmail.com>
Message-ID: <b6cce8d9-0c54-a2b3-3dca-b6e8d5e78cf0@illinois.edu>

On 11/29/17 7:57 AM, Mike Dopheide wrote:
> Follow up question for the list, I noticed Rober is using Python 3, is 
> broccoli going to work with that?

The most recent release (i.e., Bro 2.5.x) should work with Python3, but
older releases most likely won't work with Python3.

From sunari1031 at gmail.com  Wed Nov 29 22:55:42 2017
From: sunari1031 at gmail.com (=?UTF-8?B?6rmA7IiY66Co?=)
Date: Thu, 30 Nov 2017 15:55:42 +0900
Subject: [Bro] How to convert name field in smb_files.log to "readable"
	string?
Message-ID: <CACV4cAOz4nMhDYBbuD1Tvm68Tw+wsh+n1JVgU=5b2ejMfd04rA@mail.gmail.com>

Hi, all

Is there some way that convert name field of smb_files.log to "readable"?

I got name value like "\u00ec\u0099\u0084"

It seems like unicode and I read weird string(e.g. ????? ???) when I send
to ELK(characterset: utf-8).

I might need to convert it.

Any comments would be appreciated!

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171130/5ca87384/attachment.html 

From seth at corelight.com  Thu Nov 30 09:18:20 2017
From: seth at corelight.com (Seth Hall)
Date: Thu, 30 Nov 2017 12:18:20 -0500
Subject: [Bro] How to convert name field in smb_files.log to "readable"
 string?
In-Reply-To: <CACV4cAOz4nMhDYBbuD1Tvm68Tw+wsh+n1JVgU=5b2ejMfd04rA@mail.gmail.com>
References: <CACV4cAOz4nMhDYBbuD1Tvm68Tw+wsh+n1JVgU=5b2ejMfd04rA@mail.gmail.com>
Message-ID: <996D872D-0A7B-49D8-BD5C-A8328A4FBD4C@corelight.com>

I've been thinking about how to handle this for a while.  The data that 
is being written into the log is technically already UTF-8, it's just 
that non-ascii bytes are escaped.

I think we can deal with this by making a switch for the logs to make 
them "UTF-8".  It would incur a bit of overhead because each string 
would have to be scanned for valid UTF-8 characters before being written 
and then only non-valid bytes would be escaped.

   .Seth

On 30 Nov 2017, at 1:55, ??? wrote:

> Hi, all
>
> Is there some way that convert name field of smb_files.log to 
> "readable"?
>
> I got name value like "\u00ec\u0099\u0084"
>
> It seems like unicode and I read weird string(e.g. ????? ???) 
> when I send
> to ELK(characterset: utf-8).
>
> I might need to convert it.
>
> Any comments would be appreciated!
>
> Thanks!
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

--
Seth Hall * Corelight, Inc * www.corelight.com

From jazoff at illinois.edu  Thu Nov 30 10:47:34 2017
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 30 Nov 2017 18:47:34 +0000
Subject: [Bro] How to convert name field in smb_files.log to "readable"
 string?
In-Reply-To: <996D872D-0A7B-49D8-BD5C-A8328A4FBD4C@corelight.com>
References: <CACV4cAOz4nMhDYBbuD1Tvm68Tw+wsh+n1JVgU=5b2ejMfd04rA@mail.gmail.com>
	<996D872D-0A7B-49D8-BD5C-A8328A4FBD4C@corelight.com>
Message-ID: <8A3473AB-7301-4F39-8A13-CFE4DBE918CF@illinois.edu>


> On Nov 30, 2017, at 12:18 PM, Seth Hall <seth at corelight.com> wrote:
> 
> I've been thinking about how to handle this for a while.  The data that 
> is being written into the log is technically already UTF-8, it's just 
> that non-ascii bytes are escaped.
> 
> I think we can deal with this by making a switch for the logs to make 
> them "UTF-8".  It would incur a bit of overhead because each string 
> would have to be scanned for valid UTF-8 characters before being written 
> and then only non-valid bytes would be escaped.
> 
>   .Seth

Does the json log writer make this simpler for users?  I think bro writes out valid json for this,
so any json parser should give you proper UTF-8 strings.

? 
Justin Azoff