From sherine.davis at flipkart.com  Wed Jun  1 02:55:29 2016
From: sherine.davis at flipkart.com (Sherine Davis (Security Engineering))
Date: Wed, 1 Jun 2016 15:25:29 +0530
Subject: [Bro] Broccoli !!! Where to place the .bro files ?
Message-ID: <CAL2jvWGmW8_Y28LB7F_pYmWLvMLgzDUMTk63g=yE3RzQFtioCQ@mail.gmail.com>

Hi, could someone give me a neat explanation on where to place the broccoli
files.
A basic broccoli based program requires 2 files, i.e the .bro file and then
the .c file, now where do i place these files and how do i execute them ?

Please help

Regards,
Sherine Davis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160601/02858219/attachment.html 

From sherine.davis at flipkart.com  Wed Jun  1 03:29:21 2016
From: sherine.davis at flipkart.com (Sherine Davis (Security Engineering))
Date: Wed, 1 Jun 2016 15:59:21 +0530
Subject: [Bro] Please provide info on how to compile c code for broccoli !
Message-ID: <CAL2jvWEqubt80cH6Cu+5qhuvoMrTwGcQ0yo99U27ZmO5mc-C1g@mail.gmail.com>

Hello, Please provide info on how to compile c code for broccoli !
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160601/e0260906/attachment-0001.html 

From vladg at illinois.edu  Wed Jun  1 06:50:37 2016
From: vladg at illinois.edu (Vlad Grigorescu)
Date: Wed, 01 Jun 2016 08:50:37 -0500
Subject: [Bro] Brownian and Ubuntu 16.04
In-Reply-To: <CAPOahmPkPGU7XdCgSvXWTOHLAUWui2yak+2WCrAtk4BRBmtXgQ@mail.gmail.com>
References: <CAPOahmPkPGU7XdCgSvXWTOHLAUWui2yak+2WCrAtk4BRBmtXgQ@mail.gmail.com>
Message-ID: <m04m9dauhe.fsf@vladg.net>

Hi Hamid,

> My configurations are:
> Elasticsearch 2.3.3
> Bro 2.4.1
> Ubuntu 16.04 LTS

Brownian doesn't support Elasticsearch 2.x. I'd like to get that support
added, but it requires a rather massive overhaul of some of the underpinnings.

  --Vlad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 800 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160601/4f82b6dc/attachment.bin 

From robin at icir.org  Wed Jun  1 07:55:45 2016
From: robin at icir.org (Robin Sommer)
Date: Wed, 1 Jun 2016 07:55:45 -0700
Subject: [Bro] plugin help
In-Reply-To: <CAGATen7d2-042p3VrHZrs9x_UPiZP-oDuaZikBk4Wbw83PF=kg@mail.gmail.com>
References: <CAGATen7d2-042p3VrHZrs9x_UPiZP-oDuaZikBk4Wbw83PF=kg@mail.gmail.com>
Message-ID: <20160601145545.GB80373@icir.org>

Should generally be the same. Does it work if you point
BRO_PLUGIN_PATH to the plugins "build/" directory?

Robin

On Tue, May 31, 2016 at 22:32 -0700, Dk Jack wrote:

> Hi,
> I have a written a small bro plugin. I followed the instructions on the bro
> plugin page.
> If I put my plugin code in <bro-base>/lib/bro/plugins directory everything
> works fine.
> For testing purposes, I'd like to keep plugins directory different from the
> final directory.
> If I have my plugin in /abc/def and set my BRO_PLUGIN_PATH=/abc/def
> directory, it fails to load. I run bro using the following command:
> 
>     > <bro-base>/bin/bro -N
> 
> The program coredumps...The failure happens in the InitBifs code... which
> calls my
> plugin bif init function. The plugin directory layout is exactly the same
> as when it works
> if I put the plugins directory in <bro-base> directory. Is there something
> I need to setup
> if I have the plugins directory in a different location? thanks.
> 
> Dk

> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From doris at bro.org  Wed Jun  1 10:48:05 2016
From: doris at bro.org (Doris Schioberg)
Date: Wed, 1 Jun 2016 10:48:05 -0700
Subject: [Bro] Possibility storing results into system registers ?
In-Reply-To: <CAL2jvWEwbeBkmWKzCz-T5wBdvWaSwJLT9eYBTVB20ejvsouXvQ@mail.gmail.com>
References: <CAL2jvWEwbeBkmWKzCz-T5wBdvWaSwJLT9eYBTVB20ejvsouXvQ@mail.gmail.com>
Message-ID: <5366f10c-e27c-44c8-84c5-503da64be008@bro.org>

Hello Sherine,

I can't answer your questions, but it sounds like you are working on Bro
right now and have a lot of questions on the go. Maybe you want to give
it a try and join either our #Bro channel on IRC (freenode) or our
Gitter channel (gitter.im/bro/bro). That's a more direct and faste way
to get your questions answered.

I hope you get answers soon, either here or in the chats.

Best
Doris Schioberg

On 5/31/16 10:45 PM, Sherine Davis (Security Engineering) wrote:
> Hi,
> I am trying to build a cpp app, that shows information about the traffic.
> So i would like to know if it is possible to store results obtained using
> bro scripts into system registers or somewhere that another cpp file can
> access those scripts
> 
> If any other suggestions feel free to comment
> 
> Regards,
> Sherine Davis
> 
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 


From jekrous at lbl.gov  Wed Jun  1 16:15:56 2016
From: jekrous at lbl.gov (Jay Krous)
Date: Wed, 1 Jun 2016 16:15:56 -0700
Subject: [Bro] Job Posting: Cyber Security Engineer at Berkeley Lab
Message-ID: <CAARtKGkaPczLU_K4pzaYQ7dnR7S_5Cec0RvBvQknDfBVnKGdfw@mail.gmail.com>

Bro enthusiast:

Lawrence Berkeley National Lab, a University of California managed
lab, has an immediate opening for a cyber security engineer:

http://go.lbl.gov/cyber-position

If you like Bro, it was born here and is the cornerstone of our
operations. You will use Bro everyday  to detect and prevent
incidents. We have multiple Bro cluster installs, including one
monitoring a 100G link and one that spans multiple internal network
links.

A few other reasons why it's awesome to work at Berkeley Lab.

 - Mission - http://go.lbl.gov/mission (pdf)
 - Smart colleagues - you will teach and learn
 - Location - famous California Bay Area weather, activities, and food
 - Work Environment - science driven environment, less politics than usual
 - Fun and challenging - we enjoy the challenge of hard problems
 - Benefits - excellent benefits and retirement

-Jay


-- 
Jack (Jay) E. Krous III
Cyber Security, Information Technology Division
Lawrence Berkeley National Laboratory
(510) 495-2522

From beikejinmiao at gmail.com  Wed Jun  1 19:16:21 2016
From: beikejinmiao at gmail.com (=?UTF-8?B?5p2O6YeR6IuX?=)
Date: Thu, 2 Jun 2016 10:16:21 +0800
Subject: [Bro] Intelligence framework not work in bro cluster
Message-ID: <CADFdfdg2s0OAxsJ2OgEyuLoUSwegJs-HFfmTrBa=gogRcayUMg@mail.gmail.com>

I want to use the framework of intelligence to detect malicious IP and
Domain.
There is the bro's script:
```
@load frameworks/intel/seen
@load frameworks/intel/do_notice
export {
    redef Intel::read_files += {
        fmt("%s/../data/block_list_domain.intel", @DIR),
        fmt("%s/../data/block_list_ip.intel", @DIR),
    };
}
```
And there are some intelligence data:
#fields indicator   indicator_type  meta.source
113.23.72.15    Intel::ADDR testip
189.174.159.120 Intel::ADDR testip
27.159.231.181  Intel::ADDR testip
119.254.102.90  Intel::ADDR testip

#fields indicator   indicator_type  meta.source
nudmmflaurbthpw.www.w88top.com  Intel::DOMAIN   testdomain
a.ns.igcdn.com  Intel::DOMAIN   testdomain
bttracker.crunchbanglinux.org   Intel::DOMAIN   testdomain
mail.yinpiao.com    Intel::DOMAIN   testdomain

And I set `do_notice` to `T` in `do_notice.bro`.
It work fine in standalone type.
But there are not any data in notice.log or intel.log if I use the bro's
cluster.
And there is my node.cfg:
[manager]
type=manager
host=localhost

[proxy]
type=proxy
host=localhost

[worker]
type=worker
host=localhost
interface=em4
lb_method=pf_ring
lb_procs=8
pin_cpus=0,2,4,6,8,10,12,14

As you can see, all of the manager and the proxy and the workers are in one
computer.
I have read the document
<https://www.bro.org/sphinx/frameworks/intel.html>about
intelligence framework and the document said:"*Remember, the files only
need to be present on the file system of the manager node on cluster
deployments*."
So I modify my bro script as follow:
@load frameworks/intel/seen
@load frameworks/intel/do_notice
export {

@if ( Cluster::is_enabled() && Cluster::local_node_type() ==
Cluster::MANAGER )
    redef Intel::read_files += {
        fmt("%s/../data/block_list_domain.intel", @DIR),
        fmt("%s/../data/block_list_ip.intel", @DIR),
    };
@endif

}

But it also can not work and have not notice.log or intel.log.

Could any one help me. Thanks very much.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160602/68511055/attachment.html 

From dnj0496 at gmail.com  Wed Jun  1 19:30:59 2016
From: dnj0496 at gmail.com (Dk Jack)
Date: Wed, 1 Jun 2016 19:30:59 -0700
Subject: [Bro] plugin help
In-Reply-To: <20160601145545.GB80373@icir.org>
References: <CAGATen7d2-042p3VrHZrs9x_UPiZP-oDuaZikBk4Wbw83PF=kg@mail.gmail.com>
	<20160601145545.GB80373@icir.org>
Message-ID: <CAGATen6+vQG=C3+=V8jOr9TNZh8rC1ugwtB53nMXHKehMoX-bw@mail.gmail.com>

Hi Robin,
Thanks for responding to my query...

Yes, pointing it to the build directory works.
If I have the plugin under base bro (brobas/lib/bro/plugins) directory, it
works.
However if I move the plugins directory to another path it doesn't work.

Any help is appreciated...


On Wed, Jun 1, 2016 at 7:55 AM, Robin Sommer <robin at icir.org> wrote:

> Should generally be the same. Does it work if you point
> BRO_PLUGIN_PATH to the plugins "build/" directory?
>
> Robin
>
> On Tue, May 31, 2016 at 22:32 -0700, Dk Jack wrote:
>
> > Hi,
> > I have a written a small bro plugin. I followed the instructions on the
> bro
> > plugin page.
> > If I put my plugin code in <bro-base>/lib/bro/plugins directory
> everything
> > works fine.
> > For testing purposes, I'd like to keep plugins directory different from
> the
> > final directory.
> > If I have my plugin in /abc/def and set my BRO_PLUGIN_PATH=/abc/def
> > directory, it fails to load. I run bro using the following command:
> >
> >     > <bro-base>/bin/bro -N
> >
> > The program coredumps...The failure happens in the InitBifs code... which
> > calls my
> > plugin bif init function. The plugin directory layout is exactly the same
> > as when it works
> > if I put the plugins directory in <bro-base> directory. Is there
> something
> > I need to setup
> > if I have the plugins directory in a different location? thanks.
> >
> > Dk
>
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> --
> Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160601/65be8823/attachment.html 

From robin at icir.org  Wed Jun  1 21:50:58 2016
From: robin at icir.org (Robin Sommer)
Date: Wed, 1 Jun 2016 21:50:58 -0700
Subject: [Bro] plugin help
In-Reply-To: <CAGATen6+vQG=C3+=V8jOr9TNZh8rC1ugwtB53nMXHKehMoX-bw@mail.gmail.com>
References: <CAGATen7d2-042p3VrHZrs9x_UPiZP-oDuaZikBk4Wbw83PF=kg@mail.gmail.com>
	<20160601145545.GB80373@icir.org>
	<CAGATen6+vQG=C3+=V8jOr9TNZh8rC1ugwtB53nMXHKehMoX-bw@mail.gmail.com>
Message-ID: <20160602045058.GH80373@icir.org>


On Wed, Jun 01, 2016 at 19:30 -0700, Dk Jack wrote:

> Yes, pointing it to the build directory works.

That sounds like it could be a bug in Bro then, I'll try it with
another plugin when I get a chance. In the meantime, please file a
ticket describing the problem so that we can track it. Please attach a
full stack backtrace from the core dump if possible.

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From jazoff at illinois.edu  Thu Jun  2 06:26:51 2016
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 2 Jun 2016 13:26:51 +0000
Subject: [Bro] Intelligence framework not work in bro cluster
In-Reply-To: <CADFdfdg2s0OAxsJ2OgEyuLoUSwegJs-HFfmTrBa=gogRcayUMg@mail.gmail.com>
References: <CADFdfdg2s0OAxsJ2OgEyuLoUSwegJs-HFfmTrBa=gogRcayUMg@mail.gmail.com>
Message-ID: <566AFB0B-96D5-48F1-B098-BC10863AE703@illinois.edu>


> On Jun 1, 2016, at 10:16 PM, ??? <beikejinmiao at gmail.com> wrote:
> 
> I want to use the framework of intelligence to detect malicious IP and Domain. 
> There is the bro's script:
> ```
> @load frameworks/intel/seen
> @load frameworks/intel/do_notice
> export {
>     redef Intel::read_files += {
>         fmt("%s/../data/block_list_domain.intel", @DIR),
>         fmt("%s/../data/block_list_ip.intel", @DIR),
>     };
> }
> ```
> And there are some intelligence data:
> #fields indicator   indicator_type  meta.source
> 113.23.72.15    Intel::ADDR testip
> 189.174.159.120 Intel::ADDR testip
> 27.159.231.181  Intel::ADDR testip
> 119.254.102.90  Intel::ADDR testip
> 
> #fields indicator   indicator_type  meta.source
> nudmmflaurbthpw.www.w88top.com  Intel::DOMAIN   testdomain
> a.ns.igcdn.com  Intel::DOMAIN   testdomain
> bttracker.crunchbanglinux.org   Intel::DOMAIN   testdomain
> mail.yinpiao.com    Intel::DOMAIN   testdomain
> 
> And I set `do_notice` to `T` in `do_notice.bro`.
> It work fine in standalone type. 

Great, that all looks good... though you shouldn't modify do_notice.bro directly.

> But there are not any data in notice.log or intel.log if I use the bro's cluster. 
> And there is my node.cfg:
> [manager]
> type=manager
> host=localhost
> 
> [proxy]
> type=proxy
> host=localhost
> 
> [worker]
> type=worker
> host=localhost
> interface=em4
> lb_method=pf_ring
> lb_procs=8
> pin_cpus=0,2,4,6,8,10,12,14

Unrelated to your problem, but you may want to double check those cpu ids.  On most systems the 'real' cores are the first ones, followed by the hyperthreading ones, so, 

pin_cpus=0,1,2,3,4,5,6,7

is the optimal setting

> As you can see, all of the manager and the proxy and the workers are in one computer.
> I have read the document about intelligence framework and the document said:"Remember, the files only need to be present on the file system of the manager node on cluster deployments."
> So I modify my bro script as follow:
> @load frameworks/intel/seen
> @load frameworks/intel/do_notice
> export {
> 
> @if ( Cluster::is_enabled() && Cluster::local_node_type() == Cluster::MANAGER )
>     redef Intel::read_files += {
>         fmt("%s/../data/block_list_domain.intel", @DIR),
>         fmt("%s/../data/block_list_ip.intel", @DIR),
>     };
> @endif
> 
> }

This isn't required, what you had should have worked in cluster mode too.

> But it also can not work and have not notice.log or intel.log.
> 
> Could any one help me. Thanks very much.
> 

Ok, the first thing to do would be to see if cluster mode was working at all.  In cluster mode, did you have a conn.log and a dns.log ?  If you have no logs when running in cluster mode then you have a more general problem.

If everything is working the way it should be, check the loaded_scripts.log to ensure that your custom script is being loaded (it probably is since things were working in standalone mode)

Then, check stderr.log and reporter.log - especially a minute or so after startup.  If there are any problems with your intel configuration, errors will be logged there.

-- 
- Justin Azoff




From sherine.davis at flipkart.com  Thu Jun  2 22:42:40 2016
From: sherine.davis at flipkart.com (Sherine Davis (Security Engineering))
Date: Fri, 3 Jun 2016 11:12:40 +0530
Subject: [Bro] Broccoli Code Not Working : Not receiving any events
Message-ID: <CAL2jvWGjz=EuJ6b604vwWZR9M==ie+APD3v0bsxWkZxU_g+UxQ@mail.gmail.com>

Broccoli code :
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/wait.h>
#include <sys/time.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <errno.h>
#include <string.h>
#include <inttypes.h>
#include <broccoli.h>
#include <unistd.h>
#include <stdbool.h>
#include <string.h>

#ifdef HAVE_CONFIG_H
#include <config.h>
#endif

char *host_default = "127.0.0.1";
char *port_default = "64646";
char *host_str;
char *port_str;

// The variables that monitor the rate
float tcp_packet_out = 0;
float tcp_packet_in = 0;
float udp_packet_in = 0;
float udp_packet_out = 0;

uint64 seq;

static void
usage(void)
{
  printf("cero_traffic\n");
  exit(0);
}

// For every TCP_PACKET event
static void
bro_tcp_packet(BroConn *bc, void *data, BroRecord *conn, int *is_orig,
BroString *flags,int *seq, int *ack, int *len, BroString *payload)
{
  printf("1\n\n");
  if(is_orig)
  {
    tcp_packet_in++;
  }
  else
  {
    tcp_packet_out++;
  }

  conn = NULL;
  data = NULL;
}

// For every UDP_REQUEST event
static void
bro_udp_request(BroConn *conn, void *data)
{
  printf("2\n\n");
  udp_packet_in++;

  conn = NULL;
  data = NULL;
}

// For every UDP_REPLY event
static void
bro_udp_reply(BroConn *conn, void *data)
{
  udp_packet_out++;

  conn = NULL;
  data = NULL;
}


// Main driver function
// Mainly deals with creating and establishing the connection with Bro
int
main(int argc, char **argv)
{

  printf("Starting program");
  BroConn *bc;
  char hostname[512];
  int fd = -1;

  bro_init(NULL);

  host_str = host_default;
  port_str = port_default;
  printf("Marker1-success");
  snprintf(hostname, 512, "%s:%s", host_str, port_str);

  if (! (bc = bro_conn_new_str(hostname, BRO_CFLAG_RECONNECT)))
  {
    printf("Could not get Bro connection handle.\n");
    exit(-1);
  }

  bro_event_registry_add(bc, "tcp_packet", (BroEventFunc) bro_tcp_packet,
NULL);
 /* bro_event_registry_add(bc, "udp_request", (BroEventFunc)
bro_udp_request, NULL);
bro_event_registry_add(bc, "udp_reply", (BroEventFunc) bro_udp_reply, NULL);
    */

  printf("Marker2-success");
  if (! bro_conn_connect(bc))
  {
    printf("Could not connect to Bro at %s:%s.\n", host_str, port_str);
    exit(-1);
  }

  printf("Marker3-success");
  for(;;)
  {
    sleep(1);
    printf("in\n");
    bro_event_registry_request(bc);
    bro_conn_process_input(bc);

   // printf("tcp_packet_out : %f tcp_packet_in : %f udp_packet_in : %f
udp_packet_out : %f
\n",tcp_packet_out,tcp_packet_in,udp_packet_in,udp_packet_out);
  }

  /* Disconnect from Bro and release state. */
  bro_conn_delete(bc);

  return 0;
}

Bro Code :

@load policy/frameworks/communication/listen

# Let's make sure we use the same port no matter whether we use encryption
or not:
redef Communication::listen_port = 64646/tcp;

# Redef this to T if you want to use SSL.
redef Communication::listen_ssl = F;

# Set the SSL certificates being used to something real if you are using
encryption.
#redef ssl_ca_certificate   = "<path>/ca_cert.pem";
#redef ssl_private_key      = "<path>/bro.pem";

redef Communication::nodes += {
["cero_traffic"] = [$host = 127.0.0.1, $connect=F, $ssl=F]
};

global ct_log = open_log_file("cero_traffic");

event tcp_packet(c: connection, is_orig: bool, flags: string, seq: count,
ack: count, len: count, payload: string)
{
if(is_orig)
print fmt("TCP PACKET | CONN: %s:%s > %s:%s |FLAG:  %s |LEN:
%s",c$id$orig_h,c$id$orig_p,c$id$resp_h,c$id$resp_p,flags,len);
else
print fmt("TCP PACKET | CONN: %s:%s > %s:%s |FLAG:  %s |LEN:
%s",c$id$resp_h,c$id$resp_p,c$id$orig_h,c$id$orig_p,flags,len);

}

#event udp_request(u: connection)
#{
# print fmt("UDP PACKET | CONN: %s:%s >
%s:%s",u$id$orig_h,u$id$orig_p,u$id$resp_h,u$id$resp_p);
#}

#event udp_reply(u: connection)
#{
# print fmt("UDP PACKET | CONN: %s:%s >
%s:%s",u$id$resp_h,u$id$resp_p,u$id$orig_h,u$id$orig_p);
#}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160603/a3e61e15/attachment.html 

From johanna at icir.org  Fri Jun  3 09:50:57 2016
From: johanna at icir.org (Johanna Amann)
Date: Fri, 3 Jun 2016 09:50:57 -0700
Subject: [Bro] Broccoli Code Not Working : Not receiving any events
In-Reply-To: <CAL2jvWGjz=EuJ6b604vwWZR9M==ie+APD3v0bsxWkZxU_g+UxQ@mail.gmail.com>
References: <CAL2jvWGjz=EuJ6b604vwWZR9M==ie+APD3v0bsxWkZxU_g+UxQ@mail.gmail.com>
Message-ID: <20160603165053.GA28414@wifi241.sys.ICSI.Berkeley.EDU>

Hello Sherine,

I am answering to basically most of your mails and a bit of your activity
on IRC. Note that on IRC, you will nearly exclusively find people online
weekdays during US daytime hours. During other times, you are unlikely to
get an answer there - and it is a good idea to just keep your client
running since you will probably get an answer, even if it might be hours
later.

That being said - as I mentioned on IRC before, you should look at the
broping examples of Broccoli on how to sent an event from Bro to broccoli.
You will have to re-throw the event in Bro (so e.g. catch the tcp_packet
event and then send it off again under a different name, to which broccoli
listens as given in the event argument for the connection).

However, I think generally you are trying to do the wrong thing here for a
number of reasons.

First - broccoli is actually deprecated and will no longer be supported in
future versions of Bro (it will be supported in 2.5, it might be in 2.6,
it will probably not be in 2.7). Newer code might start looking into the
new communication library called Broker.

Second - sending on events like tcp_packet, udp_packet, etc via the
communication libraries seems like a bad idea. On even a quite low amount
of network traffic, you will be generating enormous numbers of events that
will have to be sent out via broccoli/broker. This will not scale to any
significant link speed. In cluster mode, you get problems with event
distribution on top of that (either you have to send events to the manager
before or each cluster worker has to have its own broccoli/broker
connection).

For your inter-arrival-time analysis, the best way probably would be to
implement that directly in C++-code as a Bro module. However, implementing
that will require quite a bit of understanding of the internal workings of
Bro, which are not really documented too much; so you will have to do
quite a bit of reading code and doing research yourself. So - that will
probably take at least weeks of your time.

An example project that takes this approach is
https://github.com/bro/bro-plugins/tree/master/tcprs.

On a sidenote - it would be nice if you could put all your emails with
questions on the same topic in one email thread - and also, if you put a
bit more work into the way that you phrase your questions to make them
easier to understand. Posting a stream of different mail messages is
actually less likely to get a response (at least from me) - please
remember that this is community support.

Johanna

On Fri, Jun 03, 2016 at 11:12:40AM +0530, Sherine Davis (Security Engineering) wrote:
> Broccoli code :
> #include <sys/types.h>
> #include <sys/socket.h>
> #include <sys/wait.h>
> #include <sys/time.h>
> #include <stdio.h>
> #include <stdlib.h>
> #include <time.h>
> #include <errno.h>
> #include <string.h>
> #include <inttypes.h>
> #include <broccoli.h>
> #include <unistd.h>
> #include <stdbool.h>
> #include <string.h>
> 
> #ifdef HAVE_CONFIG_H
> #include <config.h>
> #endif
> 
> char *host_default = "127.0.0.1";
> char *port_default = "64646";
> char *host_str;
> char *port_str;
> 
> // The variables that monitor the rate
> float tcp_packet_out = 0;
> float tcp_packet_in = 0;
> float udp_packet_in = 0;
> float udp_packet_out = 0;
> 
> uint64 seq;
> 
> static void
> usage(void)
> {
>   printf("cero_traffic\n");
>   exit(0);
> }
> 
> // For every TCP_PACKET event
> static void
> bro_tcp_packet(BroConn *bc, void *data, BroRecord *conn, int *is_orig,
> BroString *flags,int *seq, int *ack, int *len, BroString *payload)
> {
>   printf("1\n\n");
>   if(is_orig)
>   {
>     tcp_packet_in++;
>   }
>   else
>   {
>     tcp_packet_out++;
>   }
> 
>   conn = NULL;
>   data = NULL;
> }
> 
> // For every UDP_REQUEST event
> static void
> bro_udp_request(BroConn *conn, void *data)
> {
>   printf("2\n\n");
>   udp_packet_in++;
> 
>   conn = NULL;
>   data = NULL;
> }
> 
> // For every UDP_REPLY event
> static void
> bro_udp_reply(BroConn *conn, void *data)
> {
>   udp_packet_out++;
> 
>   conn = NULL;
>   data = NULL;
> }
> 
> 
> // Main driver function
> // Mainly deals with creating and establishing the connection with Bro
> int
> main(int argc, char **argv)
> {
> 
>   printf("Starting program");
>   BroConn *bc;
>   char hostname[512];
>   int fd = -1;
> 
>   bro_init(NULL);
> 
>   host_str = host_default;
>   port_str = port_default;
>   printf("Marker1-success");
>   snprintf(hostname, 512, "%s:%s", host_str, port_str);
> 
>   if (! (bc = bro_conn_new_str(hostname, BRO_CFLAG_RECONNECT)))
>   {
>     printf("Could not get Bro connection handle.\n");
>     exit(-1);
>   }
> 
>   bro_event_registry_add(bc, "tcp_packet", (BroEventFunc) bro_tcp_packet,
> NULL);
>  /* bro_event_registry_add(bc, "udp_request", (BroEventFunc)
> bro_udp_request, NULL);
> bro_event_registry_add(bc, "udp_reply", (BroEventFunc) bro_udp_reply, NULL);
>     */
> 
>   printf("Marker2-success");
>   if (! bro_conn_connect(bc))
>   {
>     printf("Could not connect to Bro at %s:%s.\n", host_str, port_str);
>     exit(-1);
>   }
> 
>   printf("Marker3-success");
>   for(;;)
>   {
>     sleep(1);
>     printf("in\n");
>     bro_event_registry_request(bc);
>     bro_conn_process_input(bc);
> 
>    // printf("tcp_packet_out : %f tcp_packet_in : %f udp_packet_in : %f
> udp_packet_out : %f
> \n",tcp_packet_out,tcp_packet_in,udp_packet_in,udp_packet_out);
>   }
> 
>   /* Disconnect from Bro and release state. */
>   bro_conn_delete(bc);
> 
>   return 0;
> }
> 
> Bro Code :
> 
> @load policy/frameworks/communication/listen
> 
> # Let's make sure we use the same port no matter whether we use encryption
> or not:
> redef Communication::listen_port = 64646/tcp;
> 
> # Redef this to T if you want to use SSL.
> redef Communication::listen_ssl = F;
> 
> # Set the SSL certificates being used to something real if you are using
> encryption.
> #redef ssl_ca_certificate   = "<path>/ca_cert.pem";
> #redef ssl_private_key      = "<path>/bro.pem";
> 
> redef Communication::nodes += {
> ["cero_traffic"] = [$host = 127.0.0.1, $connect=F, $ssl=F]
> };
> 
> global ct_log = open_log_file("cero_traffic");
> 
> event tcp_packet(c: connection, is_orig: bool, flags: string, seq: count,
> ack: count, len: count, payload: string)
> {
> if(is_orig)
> print fmt("TCP PACKET | CONN: %s:%s > %s:%s |FLAG:  %s |LEN:
> %s",c$id$orig_h,c$id$orig_p,c$id$resp_h,c$id$resp_p,flags,len);
> else
> print fmt("TCP PACKET | CONN: %s:%s > %s:%s |FLAG:  %s |LEN:
> %s",c$id$resp_h,c$id$resp_p,c$id$orig_h,c$id$orig_p,flags,len);
> 
> }
> 
> #event udp_request(u: connection)
> #{
> # print fmt("UDP PACKET | CONN: %s:%s >
> %s:%s",u$id$orig_h,u$id$orig_p,u$id$resp_h,u$id$resp_p);
> #}
> 
> #event udp_reply(u: connection)
> #{
> # print fmt("UDP PACKET | CONN: %s:%s >
> %s:%s",u$id$resp_h,u$id$resp_p,u$id$orig_h,u$id$orig_p);
> #}

> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From af7 at umbc.edu  Fri Jun  3 10:20:27 2016
From: af7 at umbc.edu (Arash Fallah)
Date: Fri, 3 Jun 2016 13:20:27 -0400
Subject: [Bro] Question About Namespaces
Message-ID: <CAKDt6fTCam1OgLOUumji__AX2p3kVgX5i7LDPEmP5-H-C6gKbQ@mail.gmail.com>

If I put the following code inside the export section, would I be changing
the separator for all subsequently loaded scripts as well?

ex.

export{

redef InputAscii::separator = ",";

}


Whereas if I put it outside the export block, would it only be changed for
my script and not apply to another?

export{

#foo here

}

redef InputAscii::separator = ",";

Additionally, when you are redefining variables, why is it that redef enum
is necessary when redefining an enumerable type whereas no other variable
requires the type to be explicitly stated for the redefinition (i.e.
strings as above).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160603/f4507029/attachment.html 

From johanna at icir.org  Fri Jun  3 10:56:11 2016
From: johanna at icir.org (Johanna Amann)
Date: Fri, 03 Jun 2016 10:56:11 -0700
Subject: [Bro] Question About Namespaces
In-Reply-To: <CAKDt6fTCam1OgLOUumji__AX2p3kVgX5i7LDPEmP5-H-C6gKbQ@mail.gmail.com>
References: <CAKDt6fTCam1OgLOUumji__AX2p3kVgX5i7LDPEmP5-H-C6gKbQ@mail.gmail.com>
Message-ID: <5A91767F-50AC-48D8-8094-4A1A0DA1CD78@icir.org>

Hi Arash,

it will always be changed for all scripts.

As for redef enum - it actually is also necessary for record types (you 
will find a lot of "redef record" in Bro scripts. However, it would 
probably be possible to get rid of those as the parser should, in 
theory, be able to determine the type by itself. I did not write that 
code, but I think it is just a Bro syntax quirk that might also make 
writing parsing a bit easier.

Johanna

On 3 Jun 2016, at 10:20, Arash Fallah wrote:

> If I put the following code inside the export section, would I be 
> changing
> the separator for all subsequently loaded scripts as well?
>
> ex.
>
> export{
>
> redef InputAscii::separator = ",";
>
> }
>
>
> Whereas if I put it outside the export block, would it only be changed 
> for
> my script and not apply to another?
>
> export{
>
> #foo here
>
> }
>
> redef InputAscii::separator = ",";
>
> Additionally, when you are redefining variables, why is it that redef 
> enum
> is necessary when redefining an enumerable type whereas no other 
> variable
> requires the type to be explicitly stated for the redefinition (i.e.
> strings as above).
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

From af7 at umbc.edu  Fri Jun  3 13:43:56 2016
From: af7 at umbc.edu (Arash Fallah)
Date: Fri, 3 Jun 2016 16:43:56 -0400
Subject: [Bro] Question About Namespaces
In-Reply-To: <5A91767F-50AC-48D8-8094-4A1A0DA1CD78@icir.org>
References: <CAKDt6fTCam1OgLOUumji__AX2p3kVgX5i7LDPEmP5-H-C6gKbQ@mail.gmail.com>
	<5A91767F-50AC-48D8-8094-4A1A0DA1CD78@icir.org>
Message-ID: <CAKDt6fRuAq=WtV5oo1=4SFx4OTcegFv7sHB+d94Ch+sX4o91Qg@mail.gmail.com>

Thanks Johanna,

Unfortunately, that would break functionality in a myriad of other
scripts.  Is there a more elegant solution to this problem than redefining
the separator after the end of the Input::end_of_data event is triggered?
Since input is performed through an asynchronous call, it is not guaranteed
that the change to the separator would be reverted before another file is
read by Bro.

On Fri, Jun 3, 2016 at 1:56 PM, Johanna Amann <johanna at icir.org> wrote:

> Hi Arash,
>
> it will always be changed for all scripts.
>
> As for redef enum - it actually is also necessary for record types (you
> will find a lot of "redef record" in Bro scripts. However, it would
> probably be possible to get rid of those as the parser should, in theory,
> be able to determine the type by itself. I did not write that code, but I
> think it is just a Bro syntax quirk that might also make writing parsing a
> bit easier.
>
> Johanna
>
>
> On 3 Jun 2016, at 10:20, Arash Fallah wrote:
>
> If I put the following code inside the export section, would I be changing
>> the separator for all subsequently loaded scripts as well?
>>
>> ex.
>>
>> export{
>>
>> redef InputAscii::separator = ",";
>>
>> }
>>
>>
>> Whereas if I put it outside the export block, would it only be changed for
>> my script and not apply to another?
>>
>> export{
>>
>> #foo here
>>
>> }
>>
>> redef InputAscii::separator = ",";
>>
>> Additionally, when you are redefining variables, why is it that redef enum
>> is necessary when redefining an enumerable type whereas no other variable
>> requires the type to be explicitly stated for the redefinition (i.e.
>> strings as above).
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160603/9e2fcddf/attachment.html 

From jdopheid at illinois.edu  Sat Jun  4 11:07:11 2016
From: jdopheid at illinois.edu (Dopheide, Jeannette M)
Date: Sat, 4 Jun 2016 18:07:11 +0000
Subject: [Bro] =?windows-1252?q?BroCon_=9216_CFP_deadline_extended_to_June?=
	=?windows-1252?q?_10th?=
Message-ID: <7EFD7D614A2BB84ABEA19B2CEDD246580C01ACF6@CITESMBX5.ad.uillinois.edu>

Bro Community,

We are extending the BroCon ?16 call for presentations deadline to Friday, June 10th. For more information about the CFP, see our blog post [1]. And don't forget to register!

[1] http://blog.bro.org/2016/06/brocon-16-cfp-deadline-extended-to-june.html

See you in September,

The Bro Project
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160604/14865b11/attachment.html 

From giorgio.apuzzo at gmail.com  Mon Jun  6 08:29:48 2016
From: giorgio.apuzzo at gmail.com (Giorgio Apuzzo)
Date: Mon, 6 Jun 2016 17:29:48 +0200
Subject: [Bro] Question: How to block a malicious file
Message-ID: <1E582584-C84F-4E47-A032-BA640C922927@gmail.com>

Hi,
I?m trying to write a script that after checking on virus total the hash of a file will block it if malicious.
I run a ruby script that checks the hash against virus total and return 0 if not malicious and more if not.
I have looked into the documentation but I can?t figure out how to block a file once I know it?s malicious..

Do I need an external tool?

Thanks

Giorgio Apuzzo
giorgio.apuzzo at gmail.com



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160606/66a81e58/attachment.html 

From andrew at atomicmole.com  Mon Jun  6 12:10:22 2016
From: andrew at atomicmole.com (Andrew Beard)
Date: Mon, 6 Jun 2016 19:10:22 +0000
Subject: [Bro] Bro Digest, Vol 122, Issue 6
In-Reply-To: <mailman.1.1465239601.859.bro@bro.org>
References: <mailman.1.1465239601.859.bro@bro.org>
Message-ID: <93EA0E76-29C1-4BC5-B200-54262ABB6037@atomicmole.com>

Maybe I?m misunderstanding what you?re trying to do, but if the entire file has already been transferred (which you need to do to calculate the hash) there?s not a lot of hope of being able to block the file.  It?s already made it?s way across the wire.  I don?t think Bro has built-in blocking capabilities, but by waiting for the file hash it sounds like it?s already too late without some sort of proxy in the mix.  

> Date: Mon, 6 Jun 2016 17:29:48 +0200
> From: Giorgio Apuzzo <giorgio.apuzzo at gmail.com>
> Subject: [Bro] Question: How to block a malicious file
> To: bro at bro.org
> Message-ID: <1E582584-C84F-4E47-A032-BA640C922927 at gmail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> Hi,
> I?m trying to write a script that after checking on virus total the hash of a file will block it if malicious.
> I run a ruby script that checks the hash against virus total and return 0 if not malicious and more if not.
> I have looked into the documentation but I can?t figure out how to block a file once I know it?s malicious..
> 
> Do I need an external tool?
> 
> Thanks
> 
> Giorgio Apuzzo
> giorgio.apuzzo at gmail.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4205 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160606/4cfb99cd/attachment.bin 

From johanna at icir.org  Mon Jun  6 12:25:49 2016
From: johanna at icir.org (Johanna Amann)
Date: Mon, 6 Jun 2016 12:25:49 -0700
Subject: [Bro] Question: How to block a malicious file
In-Reply-To: <1E582584-C84F-4E47-A032-BA640C922927@gmail.com>
References: <1E582584-C84F-4E47-A032-BA640C922927@gmail.com>
Message-ID: <20160606192549.GA92909@wifi241.sys.ICSI.Berkeley.EDU>

Hello Giorgio,

Since Bro works completely passively, and is not an in-line component, Bro
itself cannot block a file. By the time that Bro can calculate the file
hash, the whole file already will have been transferred to the client who
was downloading it. You can just do an after-the-fact reporting.

You could potentially use the NetControl framework (in master, will be
part of 2.5) to block future network connections of the hosts.

I hope this helps,
 Johanna

On Mon, Jun 06, 2016 at 05:29:48PM +0200, Giorgio Apuzzo wrote:
> Hi,
> I?m trying to write a script that after checking on virus total the hash of a file will block it if malicious.
> I run a ruby script that checks the hash against virus total and return 0 if not malicious and more if not.
> I have looked into the documentation but I can?t figure out how to block a file once I know it?s malicious..
> 
> Do I need an external tool?
> 
> Thanks
> 
> Giorgio Apuzzo
> giorgio.apuzzo at gmail.com
> 
> 
> 

> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From fatema.bannatwala at gmail.com  Mon Jun  6 12:29:11 2016
From: fatema.bannatwala at gmail.com (fatema bannatwala)
Date: Mon, 6 Jun 2016 15:29:11 -0400
Subject: [Bro] Bro Digest, Vol 122, Issue 6
In-Reply-To: <mailman.1.1465239601.859.bro@bro.org>
References: <mailman.1.1465239601.859.bro@bro.org>
Message-ID: <CACX0rURxBom=Sz=YvOmE7OJdbQ1_dugv_sUhTLw8X_iqAHRyBA@mail.gmail.com>

Hi Giorgio,

As I recall, BRO only provides tap mode so far, haven't heard of using BRO
in inline mode, or I might be wrong.
So BRO really can't block anything in your traffic, you need to use
external scripts to perform the trick for you.
One of the possible solutions, as far as I can think on top of my head, is
to block the source IP from which file is being transferred,
because I think once BRO logs the file details in log file, the transfer
have already happened, so I think you can't block the file transfer in the
transit. Or there might be ways which I might not be familiar with.
Can you share your script?

Thanks,
Fatema.


On Mon, Jun 6, 2016 at 3:00 PM, <bro-request at bro.org> wrote:

> Send Bro mailing list submissions to
>         bro at bro.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> or, via email, send a message with subject or body 'help' to
>         bro-request at bro.org
>
> You can reach the person managing the list at
>         bro-owner at bro.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Bro digest..."
>
>
> Today's Topics:
>
>    1. Question: How to block a malicious file (Giorgio Apuzzo)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 6 Jun 2016 17:29:48 +0200
> From: Giorgio Apuzzo <giorgio.apuzzo at gmail.com>
> Subject: [Bro] Question: How to block a malicious file
> To: bro at bro.org
> Message-ID: <1E582584-C84F-4E47-A032-BA640C922927 at gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi,
> I?m trying to write a script that after checking on virus total the hash
> of a file will block it if malicious.
> I run a ruby script that checks the hash against virus total and return 0
> if not malicious and more if not.
> I have looked into the documentation but I can?t figure out how to block a
> file once I know it?s malicious..
>
> Do I need an external tool?
>
> Thanks
>
> Giorgio Apuzzo
> giorgio.apuzzo at gmail.com
>
>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160606/66a81e58/attachment-0001.html
>
> ------------------------------
>
> _______________________________________________
> Bro mailing list
> Bro at bro.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> End of Bro Digest, Vol 122, Issue 6
> ***********************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160606/6889fa7a/attachment.html 

From johanna at icir.org  Mon Jun  6 12:35:04 2016
From: johanna at icir.org (Johanna Amann)
Date: Mon, 6 Jun 2016 12:35:04 -0700
Subject: [Bro] Question About Namespaces
In-Reply-To: <CAKDt6fRuAq=WtV5oo1=4SFx4OTcegFv7sHB+d94Ch+sX4o91Qg@mail.gmail.com>
References: <CAKDt6fTCam1OgLOUumji__AX2p3kVgX5i7LDPEmP5-H-C6gKbQ@mail.gmail.com>
	<5A91767F-50AC-48D8-8094-4A1A0DA1CD78@icir.org>
	<CAKDt6fRuAq=WtV5oo1=4SFx4OTcegFv7sHB+d94Ch+sX4o91Qg@mail.gmail.com>
Message-ID: <20160606193504.GB92909@wifi241.sys.ICSI.Berkeley.EDU>

Hi Arash,

Yup, you actually can specify the separator per by passing it
as a configuration option (named separator).

You could, e.g. do something like this:

local config_strings: table[string] of string = {
		 ["separator"] = ";",
	};
Input::add_table([$source="../input.log", $name="ssh", $idx=Idx, $val=Val, $destination=servers, $config=config_strings]);

I hope this helps,
 Johanna


On Fri, Jun 03, 2016 at 04:43:56PM -0400, Arash Fallah wrote:
> Thanks Johanna,
> 
> Unfortunately, that would break functionality in a myriad of other
> scripts.  Is there a more elegant solution to this problem than redefining
> the separator after the end of the Input::end_of_data event is triggered?
> Since input is performed through an asynchronous call, it is not guaranteed
> that the change to the separator would be reverted before another file is
> read by Bro.
> 
> On Fri, Jun 3, 2016 at 1:56 PM, Johanna Amann <johanna at icir.org> wrote:
> 
> > Hi Arash,
> >
> > it will always be changed for all scripts.
> >
> > As for redef enum - it actually is also necessary for record types (you
> > will find a lot of "redef record" in Bro scripts. However, it would
> > probably be possible to get rid of those as the parser should, in theory,
> > be able to determine the type by itself. I did not write that code, but I
> > think it is just a Bro syntax quirk that might also make writing parsing a
> > bit easier.
> >
> > Johanna
> >
> >
> > On 3 Jun 2016, at 10:20, Arash Fallah wrote:
> >
> > If I put the following code inside the export section, would I be changing
> >> the separator for all subsequently loaded scripts as well?
> >>
> >> ex.
> >>
> >> export{
> >>
> >> redef InputAscii::separator = ",";
> >>
> >> }
> >>
> >>
> >> Whereas if I put it outside the export block, would it only be changed for
> >> my script and not apply to another?
> >>
> >> export{
> >>
> >> #foo here
> >>
> >> }
> >>
> >> redef InputAscii::separator = ",";
> >>
> >> Additionally, when you are redefining variables, why is it that redef enum
> >> is necessary when redefining an enumerable type whereas no other variable
> >> requires the type to be explicitly stated for the redefinition (i.e.
> >> strings as above).
> >> _______________________________________________
> >> Bro mailing list
> >> bro at bro-ids.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >>
> >

From dwdixon at umich.edu  Mon Jun  6 15:07:33 2016
From: dwdixon at umich.edu (Drew Dixon)
Date: Mon, 6 Jun 2016 18:07:33 -0400
Subject: [Bro] Question about network cards
In-Reply-To: <ED679517-9DF4-49C6-AD4F-96D4937D3F9A@gmail.com>
References: <40481EA2-57C7-4D7C-97C5-A27D80AEBC96@oregonstate.edu>
	<D7A10B8F46BF1F47A4D30051C9CA31CC114634B4@MIMESMSW03AHCP.US.comerica.net>
	<96727EE0-7D0C-400D-A37D-77E97E1B0BED@gmail.com>
	<E5A2071C6FFFF640B714C224139C9B3327CCA2C1@WP-MBX2B.milky-way.battelle.org>
	<570E5A36.5050301@gmail.com>
	<ED679517-9DF4-49C6-AD4F-96D4937D3F9A@gmail.com>
Message-ID: <CA+pNqTPFM68ZF-JNnwNFJk1-zYO8UHCGbzpdLEQRe3=Y9sJO6g@mail.gmail.com>

Hi All,

Wondering what model Myricom card is most commonly purchased for 10G Bro
monitoring connections?  I see Myricom has many options but I'm wondering
which exact model is purchased most commonly and/or recommended amongst
those who have Bro deployed in a production environment?

Assuming most people probably go with the SPF+ model (if not please let me
know), do most people go with the 1-port card (10G-PCIE-8B-S) or did anyone
purchase the 2-port card (10G-PCIE2-8C2-2S) and does anyone see any real
value or purpose in going with the 2-port card for IDS/monitoring interface
purposes?  I'm assuming the answer is no to finding value/purpose in the
2-port card but I wanted to get some valuable input on all of this before
making any purchases of the Myricom cards.

Of those who run the Myricom cards currently, did most go with the
10G-PCIE-8B-S model?

https://www.myricom.com/products/network-adapters/product-selector.html

Thank you


On Wed, Apr 13, 2016 at 11:27 AM, Micha? Purzy?ski <
michalpurzynski1 at gmail.com> wrote:

> Of only I had enough patience for ZC. When it worked, I saw some packed
> loss that wasn't there when I used Myricom on the same sensor.
>
> A nice alternative would be an Intel plus NetMap.
>
> > On 13 Apr 2016, at 16:39, Gary Faulkner <gfaulkner.nsm at gmail.com> wrote:
> >
> > What are folks thoughts on Intel Cards with the fully licensed PF_RING
> DNA+Libzero or ZC drivers and libraries, which NTOP typically offers to
> EDUs at no cost. Shouldn't these perform much more closely to the Myricoms
> with Sniffer v3 than standard PF_RING drivers and libraries?
> >
> >> On 4/13/16 6:46 AM, Hosom, Stephen M wrote:
> >> Intel x520s work fine with both af_packet and pf_ring.
> >>
> >>
> >> On 04/12/2016 06:03 PM, Micha? Purzy?ski wrote:
> >> Another voice for myricoms. Single port with the sniffer v3 license was
> nowhere close to 1000, but much cheaper.
> >>
> >> Maintaining that, comparing to pfring, is day and night.
> >>
> >> Netmap with Intel should be the future, I don't have much experience
> with that yet.
> >>
> >> Another option is afpacket and intels, works well.
> >>
> >> On 12 Apr 2016, at 22:41, Miller, Brad L <<mailto:BLMILLER at comerica.com
> >BLMILLER at comerica.com<mailto:BLMILLER at comerica.com>> wrote:
> >>
> >> We are using Endace cards which are quite a bit more pricey, but we are
> actively looking at the Myricom cards now.
> >>
> >> My advice ? get the Myricom cards.  While you can do pfring using
> standard cards, nothing beats the low to no capture loss hardware.  The
> ability to do onboard load distribution with multiple sub interfaces is a
> killer feature and your Bro config is greatly simplified.  We use a patched
> version of libpacap for Endace.. but I hear that 2.5 may incorporate native
> Myricom support.
> >>
> >> Without cards like these it is like getting a new mustang but skimping
> on the powertrain options.
> >>
> >>
> >>
> >>
> >> From: bro-bounces at bro.org<mailto:bro-bounces at bro.org> [mailto:
> bro-bounces at bro.org] On Behalf Of Giesige, Rich
> >> Sent: Tuesday, April 12, 2016 4:24 PM
> >> To: bro at bro.org<mailto:bro at bro.org>
> >> Subject: [Bro] Question about network cards
> >>
> >> Hello,
> >>
> >> I?m wondering what people are using for network cards in their bro
> clusters that are not using the Myricom Network Cards. We don?t have a
> $1,000 dollars per a card + license to spend on the cards. Is anyone using
> Intel or other brands that aren?t as expensive to capture their traffic? We
> are looking at doing all 10 Gig connections into the Bro Cluster.
> >>
> >> Thanks for all your answers.
> >>
> >> --
> >> Richard Giesige
> >> IT Security Analyst
> >> Office of Information Security
> >> Oregon State University
> >>
> >> "OSU staff will NEVER ask for you password.
> >> Never email or share your password with anyone."
> >>
> >>
> >> Please be aware that if you reply directly to this particular message,
> your reply may not be secure. Do not use email to send us communications
> that contain unencrypted confidential information such as passwords,
> account numbers or Social Security numbers. If you must provide this type
> of information, please visit comerica.com<http://comerica.com> to submit
> a secure form using any of the ?Contact Us? forms. In addition, you should
> not send via email any inquiry or request that may be time sensitive. The
> information in this e-mail is confidential. It is intended for the
> individual or entity to whom it is addressed. If you have received this
> email in error, please destroy or delete the message and advise the sender
> of the error by return email.
> >> _______________________________________________
> >> Bro mailing list
> >> bro at bro-ids.org<mailto:bro at bro-ids.org>
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro<
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
> >>
> >>
> >> _______________________________________________
> >> Bro mailing list
> >> bro at bro-ids.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160606/c4fcc702/attachment.html 

From lattin at umn.edu  Mon Jun  6 15:42:02 2016
From: lattin at umn.edu (Brandon Lattin)
Date: Mon, 6 Jun 2016 17:42:02 -0500
Subject: [Bro] Question about network cards
In-Reply-To: <CA+pNqTPFM68ZF-JNnwNFJk1-zYO8UHCGbzpdLEQRe3=Y9sJO6g@mail.gmail.com>
References: <40481EA2-57C7-4D7C-97C5-A27D80AEBC96@oregonstate.edu>
	<D7A10B8F46BF1F47A4D30051C9CA31CC114634B4@MIMESMSW03AHCP.US.comerica.net>
	<96727EE0-7D0C-400D-A37D-77E97E1B0BED@gmail.com>
	<E5A2071C6FFFF640B714C224139C9B3327CCA2C1@WP-MBX2B.milky-way.battelle.org>
	<570E5A36.5050301@gmail.com>
	<ED679517-9DF4-49C6-AD4F-96D4937D3F9A@gmail.com>
	<CA+pNqTPFM68ZF-JNnwNFJk1-zYO8UHCGbzpdLEQRe3=Y9sJO6g@mail.gmail.com>
Message-ID: <CAJWRZ8QANxH_xf+ympyK_DYf56JL6LT8g5wbSg5oxnBDHsBqpQ@mail.gmail.com>

We use all 10G-PCIE2-8C2-2S with Sniffer10G v3.

We rarely use the second port, but it's handy to have.

On Mon, Jun 6, 2016 at 5:07 PM, Drew Dixon <dwdixon at umich.edu> wrote:

> Hi All,
>
> Wondering what model Myricom card is most commonly purchased for 10G Bro
> monitoring connections?  I see Myricom has many options but I'm wondering
> which exact model is purchased most commonly and/or recommended amongst
> those who have Bro deployed in a production environment?
>
> Assuming most people probably go with the SPF+ model (if not please let me
> know), do most people go with the 1-port card (10G-PCIE-8B-S) or did anyone
> purchase the 2-port card (10G-PCIE2-8C2-2S) and does anyone see any real
> value or purpose in going with the 2-port card for IDS/monitoring interface
> purposes?  I'm assuming the answer is no to finding value/purpose in the
> 2-port card but I wanted to get some valuable input on all of this before
> making any purchases of the Myricom cards.
>
> Of those who run the Myricom cards currently, did most go with the
> 10G-PCIE-8B-S model?
>
> https://www.myricom.com/products/network-adapters/product-selector.html
>
> Thank you
>
>
> On Wed, Apr 13, 2016 at 11:27 AM, Micha? Purzy?ski <
> michalpurzynski1 at gmail.com> wrote:
>
>> Of only I had enough patience for ZC. When it worked, I saw some packed
>> loss that wasn't there when I used Myricom on the same sensor.
>>
>> A nice alternative would be an Intel plus NetMap.
>>
>> > On 13 Apr 2016, at 16:39, Gary Faulkner <gfaulkner.nsm at gmail.com>
>> wrote:
>> >
>> > What are folks thoughts on Intel Cards with the fully licensed PF_RING
>> DNA+Libzero or ZC drivers and libraries, which NTOP typically offers to
>> EDUs at no cost. Shouldn't these perform much more closely to the Myricoms
>> with Sniffer v3 than standard PF_RING drivers and libraries?
>> >
>> >> On 4/13/16 6:46 AM, Hosom, Stephen M wrote:
>> >> Intel x520s work fine with both af_packet and pf_ring.
>> >>
>> >>
>> >> On 04/12/2016 06:03 PM, Micha? Purzy?ski wrote:
>> >> Another voice for myricoms. Single port with the sniffer v3 license
>> was nowhere close to 1000, but much cheaper.
>> >>
>> >> Maintaining that, comparing to pfring, is day and night.
>> >>
>> >> Netmap with Intel should be the future, I don't have much experience
>> with that yet.
>> >>
>> >> Another option is afpacket and intels, works well.
>> >>
>> >> On 12 Apr 2016, at 22:41, Miller, Brad L <<mailto:
>> BLMILLER at comerica.com>BLMILLER at comerica.com<mailto:BLMILLER at comerica.com>>
>> wrote:
>> >>
>> >> We are using Endace cards which are quite a bit more pricey, but we
>> are actively looking at the Myricom cards now.
>> >>
>> >> My advice ? get the Myricom cards.  While you can do pfring using
>> standard cards, nothing beats the low to no capture loss hardware.  The
>> ability to do onboard load distribution with multiple sub interfaces is a
>> killer feature and your Bro config is greatly simplified.  We use a patched
>> version of libpacap for Endace.. but I hear that 2.5 may incorporate native
>> Myricom support.
>> >>
>> >> Without cards like these it is like getting a new mustang but skimping
>> on the powertrain options.
>> >>
>> >>
>> >>
>> >>
>> >> From: bro-bounces at bro.org<mailto:bro-bounces at bro.org> [mailto:
>> bro-bounces at bro.org] On Behalf Of Giesige, Rich
>> >> Sent: Tuesday, April 12, 2016 4:24 PM
>> >> To: bro at bro.org<mailto:bro at bro.org>
>> >> Subject: [Bro] Question about network cards
>> >>
>> >> Hello,
>> >>
>> >> I?m wondering what people are using for network cards in their bro
>> clusters that are not using the Myricom Network Cards. We don?t have a
>> $1,000 dollars per a card + license to spend on the cards. Is anyone using
>> Intel or other brands that aren?t as expensive to capture their traffic? We
>> are looking at doing all 10 Gig connections into the Bro Cluster.
>> >>
>> >> Thanks for all your answers.
>> >>
>> >> --
>> >> Richard Giesige
>> >> IT Security Analyst
>> >> Office of Information Security
>> >> Oregon State University
>> >>
>> >> "OSU staff will NEVER ask for you password.
>> >> Never email or share your password with anyone."
>> >>
>> >>
>> >> Please be aware that if you reply directly to this particular message,
>> your reply may not be secure. Do not use email to send us communications
>> that contain unencrypted confidential information such as passwords,
>> account numbers or Social Security numbers. If you must provide this type
>> of information, please visit comerica.com<http://comerica.com> to submit
>> a secure form using any of the ?Contact Us? forms. In addition, you should
>> not send via email any inquiry or request that may be time sensitive. The
>> information in this e-mail is confidential. It is intended for the
>> individual or entity to whom it is addressed. If you have received this
>> email in error, please destroy or delete the message and advise the sender
>> of the error by return email.
>> >> _______________________________________________
>> >> Bro mailing list
>> >> bro at bro-ids.org<mailto:bro at bro-ids.org>
>> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro<
>> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>> >>
>> >>
>> >> _______________________________________________
>> >> Bro mailing list
>> >> bro at bro-ids.org
>> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> >
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 
Brandon Lattin
Security Analyst
University of Minnesota - University Information Security
Office: 612-626-6672
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160606/702e296e/attachment-0001.html 

From blackhole.em at gmail.com  Mon Jun  6 18:20:51 2016
From: blackhole.em at gmail.com (Joe Blow)
Date: Mon, 06 Jun 2016 21:20:51 -0400
Subject: [Bro] Question about network cards
In-Reply-To: <CAJWRZ8QANxH_xf+ympyK_DYf56JL6LT8g5wbSg5oxnBDHsBqpQ@mail.gmail.com>
Message-ID: <57562176.5574810a.8fe41.ffff9fd3@mx.google.com>

An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160606/75c2de3f/attachment.html 

From michalpurzynski1 at gmail.com  Mon Jun  6 20:37:14 2016
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Tue, 7 Jun 2016 05:37:14 +0200
Subject: [Bro] Question about network cards
In-Reply-To: <57562176.5574810a.8fe41.ffff9fd3@mx.google.com>
References: <57562176.5574810a.8fe41.ffff9fd3@mx.google.com>
Message-ID: <45083CCC-7508-43E9-9D34-74B28CFAF190@gmail.com>

https://www.myricom.com/products/network-adapters/10g-pcie-8b-s.html

15 of them with sniffer v3. Boring. Just works.
Myricom support is non-existent though.

> On 07 Jun 2016, at 03:20, Joe Blow <blackhole.em at gmail.com> wrote:
> 
> I'm still a huge fan of the solarflare cards. Great support and you just compile against libpcap and slipstream the driver. Pick how many cores you want to give any program and bingo-bango. It certainly makes compiling any program for 10gb a cinch. 
> 
> Cheers,
> 
> JB
> 
> Sent from my BlackBerry Smartphone on the Verizon 4G LTE Network
> From:lattin at umn.edu
> Sent:June 6, 2016 6:50 PM
> To:dwdixon at umich.edu
> Cc:BLMILLER at comerica.com; bro at bro.org
> Subject:Re: [Bro] Question about network cards
> 
> We use all 10G-PCIE2-8C2-2S with Sniffer10G v3.
> 
> We rarely use the second port, but it's handy to have.
> 
>> On Mon, Jun 6, 2016 at 5:07 PM, Drew Dixon <dwdixon at umich.edu> wrote:
>> Hi All,
>> 
>> Wondering what model Myricom card is most commonly purchased for 10G Bro monitoring connections?  I see Myricom has many options but I'm wondering which exact model is purchased most commonly and/or recommended amongst those who have Bro deployed in a production environment?
>> 
>> Assuming most people probably go with the SPF+ model (if not please let me know), do most people go with the 1-port card (10G-PCIE-8B-S) or did anyone purchase the 2-port card (10G-PCIE2-8C2-2S) and does anyone see any real value or purpose in going with the 2-port card for IDS/monitoring interface purposes?  I'm assuming the answer is no to finding value/purpose in the 2-port card but I wanted to get some valuable input on all of this before making any purchases of the Myricom cards.
>> 
>> Of those who run the Myricom cards currently, did most go with the 10G-PCIE-8B-S model?
>> 
>> https://www.myricom.com/products/network-adapters/product-selector.html 
>> 
>> Thank you
>> 
>> 
>>> On Wed, Apr 13, 2016 at 11:27 AM, Micha? Purzy?ski <michalpurzynski1 at gmail.com> wrote:
>>> Of only I had enough patience for ZC. When it worked, I saw some packed loss that wasn't there when I used Myricom on the same sensor.
>>> 
>>> A nice alternative would be an Intel plus NetMap.
>>> 
>>> > On 13 Apr 2016, at 16:39, Gary Faulkner <gfaulkner.nsm at gmail.com> wrote:
>>> >
>>> > What are folks thoughts on Intel Cards with the fully licensed PF_RING DNA+Libzero or ZC drivers and libraries, which NTOP typically offers to EDUs at no cost. Shouldn't these perform much more closely to the Myricoms with Sniffer v3 than standard PF_RING drivers and libraries?
>>> >
>>> >> On 4/13/16 6:46 AM, Hosom, Stephen M wrote:
>>> >> Intel x520s work fine with both af_packet and pf_ring.
>>> >>
>>> >>
>>> >> On 04/12/2016 06:03 PM, Micha? Purzy?ski wrote:
>>> >> Another voice for myricoms. Single port with the sniffer v3 license was nowhere close to 1000, but much cheaper.
>>> >>
>>> >> Maintaining that, comparing to pfring, is day and night.
>>> >>
>>> >> Netmap with Intel should be the future, I don't have much experience with that yet.
>>> >>
>>> >> Another option is afpacket and intels, works well.
>>> >>
>>> >> On 12 Apr 2016, at 22:41, Miller, Brad L <<mailto:BLMILLER at comerica.com>BLMILLER at comerica.com<mailto:BLMILLER at comerica.com>> wrote:
>>> >>
>>> >> We are using Endace cards which are quite a bit more pricey, but we are actively looking at the Myricom cards now.
>>> >>
>>> >> My advice ? get the Myricom cards.  While you can do pfring using standard cards, nothing beats the low to no capture loss hardware.  The ability to do onboard load distribution with multiple sub interfaces is a killer feature and your Bro config is greatly simplified.  We use a patched version of libpacap for Endace.. but I hear that 2.5 may incorporate native Myricom support.
>>> >>
>>> >> Without cards like these it is like getting a new mustang but skimping on the powertrain options.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> From: bro-bounces at bro.org<mailto:bro-bounces at bro.org> [mailto:bro-bounces at bro.org] On Behalf Of Giesige, Rich
>>> >> Sent: Tuesday, April 12, 2016 4:24 PM
>>> >> To: bro at bro.org<mailto:bro at bro.org>
>>> >> Subject: [Bro] Question about network cards
>>> >>
>>> >> Hello,
>>> >>
>>> >> I?m wondering what people are using for network cards in their bro clusters that are not using the Myricom Network Cards. We don?t have a $1,000 dollars per a card + license to spend on the cards. Is anyone using Intel or other brands that aren?t as expensive to capture their traffic? We are looking at doing all 10 Gig connections into the Bro Cluster.
>>> >>
>>> >> Thanks for all your answers.
>>> >>
>>> >> --
>>> >> Richard Giesige
>>> >> IT Security Analyst
>>> >> Office of Information Security
>>> >> Oregon State University
>>> >>
>>> >> "OSU staff will NEVER ask for you password.
>>> >> Never email or share your password with anyone."
>>> >>
>>> >>
>>> >> Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com<http://comerica.com> to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email.
>>> >> _______________________________________________
>>> >> Bro mailing list
>>> >> bro at bro-ids.org<mailto:bro at bro-ids.org>
>>> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro<http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> Bro mailing list
>>> >> bro at bro-ids.org
>>> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>> >
>>> 
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> 
>> 
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 
> 
> -- 
> Brandon Lattin
> Security Analyst
> University of Minnesota - University Information Security
> Office: 612-626-6672
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160607/eb2ae4b1/attachment-0001.html 

From af7 at umbc.edu  Tue Jun  7 08:56:48 2016
From: af7 at umbc.edu (Arash Fallah)
Date: Tue, 7 Jun 2016 11:56:48 -0400
Subject: [Bro] Question About Namespaces
In-Reply-To: <20160606193504.GB92909@wifi241.sys.ICSI.Berkeley.EDU>
References: <CAKDt6fTCam1OgLOUumji__AX2p3kVgX5i7LDPEmP5-H-C6gKbQ@mail.gmail.com>
	<5A91767F-50AC-48D8-8094-4A1A0DA1CD78@icir.org>
	<CAKDt6fRuAq=WtV5oo1=4SFx4OTcegFv7sHB+d94Ch+sX4o91Qg@mail.gmail.com>
	<20160606193504.GB92909@wifi241.sys.ICSI.Berkeley.EDU>
Message-ID: <CAKDt6fSm=SkFjbjetKXYiTzYCLP0ULLzm0Q_xS6oMyJtfMLqAw@mail.gmail.com>

Thank you very much.

I'm still going over the documentation as I'm very much new to Bro
scripting. Once again, thanks for your help.

On Mon, Jun 6, 2016 at 3:35 PM, Johanna Amann <johanna at icir.org> wrote:

> Hi Arash,
>
> Yup, you actually can specify the separator per by passing it
> as a configuration option (named separator).
>
> You could, e.g. do something like this:
>
> local config_strings: table[string] of string = {
>                  ["separator"] = ";",
>         };
> Input::add_table([$source="../input.log", $name="ssh", $idx=Idx, $val=Val,
> $destination=servers, $config=config_strings]);
>
> I hope this helps,
>  Johanna
>
>
> On Fri, Jun 03, 2016 at 04:43:56PM -0400, Arash Fallah wrote:
> > Thanks Johanna,
> >
> > Unfortunately, that would break functionality in a myriad of other
> > scripts.  Is there a more elegant solution to this problem than
> redefining
> > the separator after the end of the Input::end_of_data event is triggered?
> > Since input is performed through an asynchronous call, it is not
> guaranteed
> > that the change to the separator would be reverted before another file is
> > read by Bro.
> >
> > On Fri, Jun 3, 2016 at 1:56 PM, Johanna Amann <johanna at icir.org> wrote:
> >
> > > Hi Arash,
> > >
> > > it will always be changed for all scripts.
> > >
> > > As for redef enum - it actually is also necessary for record types (you
> > > will find a lot of "redef record" in Bro scripts. However, it would
> > > probably be possible to get rid of those as the parser should, in
> theory,
> > > be able to determine the type by itself. I did not write that code,
> but I
> > > think it is just a Bro syntax quirk that might also make writing
> parsing a
> > > bit easier.
> > >
> > > Johanna
> > >
> > >
> > > On 3 Jun 2016, at 10:20, Arash Fallah wrote:
> > >
> > > If I put the following code inside the export section, would I be
> changing
> > >> the separator for all subsequently loaded scripts as well?
> > >>
> > >> ex.
> > >>
> > >> export{
> > >>
> > >> redef InputAscii::separator = ",";
> > >>
> > >> }
> > >>
> > >>
> > >> Whereas if I put it outside the export block, would it only be
> changed for
> > >> my script and not apply to another?
> > >>
> > >> export{
> > >>
> > >> #foo here
> > >>
> > >> }
> > >>
> > >> redef InputAscii::separator = ",";
> > >>
> > >> Additionally, when you are redefining variables, why is it that redef
> enum
> > >> is necessary when redefining an enumerable type whereas no other
> variable
> > >> requires the type to be explicitly stated for the redefinition (i.e.
> > >> strings as above).
> > >> _______________________________________________
> > >> Bro mailing list
> > >> bro at bro-ids.org
> > >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> > >>
> > >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160607/b0032993/attachment.html 

From bglaze at gmail.com  Tue Jun  7 09:19:15 2016
From: bglaze at gmail.com (Brandon Glaze)
Date: Tue, 7 Jun 2016 09:19:15 -0700
Subject: [Bro] Question about network cards
In-Reply-To: <40481EA2-57C7-4D7C-97C5-A27D80AEBC96@oregonstate.edu>
References: <40481EA2-57C7-4D7C-97C5-A27D80AEBC96@oregonstate.edu>
Message-ID: <CAM5AJtY5b4cF8dDSK5LpfRPZjYKrKoh8=ob2auJd1x6dNb-7cA@mail.gmail.com>

I have had a great deal of success using Netronome cards. I built a couple
of clusters using older Netronome NFE-3240's, but am getting ready to test
their new NFP-4000 based cards (AgilIO 40Gx1 cards). The netronome NFP
(Network Flow Processor) uses a packet coalescing driver or network flow
capture driver to load balance traffic to seperate "rings". For Bro, and
the load balancers we use, I use both 10G ports on each card (1 card per
server), then have the packet coalescing driver load balance the traffic
from both ports to all available rings (at 100Mb per ring), then tie a CPU
core to each ring. It takes some tuning, and depends on your traffic, but I
have successfully hit 80G using one cluster with off the shelf servers and
the older netronome cards, which were far cheaper than the Myricoms.

There is more support from the community with the Myricom cards, and Bro
has native support, so that should be factored in...

Just a note, SourceFire and Cisco use the Netronome cards in their network
security products (or used to before Cisco assimilated SourceFire), so they
are high end and work very well. Their API is well documented as well.

=====================
Brandon Glaze
bglaze at gmail.com

"Lead me, follow me, or get the hell out of my way."
- General George Patton Jr


On Tue, Apr 12, 2016 at 1:23 PM, Giesige, Rich <Rich.Giesige at oregonstate.edu
> wrote:

> Hello,
>
>
>
> I?m wondering what people are using for network cards in their bro
> clusters that are not using the Myricom Network Cards. We don?t have a
> $1,000 dollars per a card + license to spend on the cards. Is anyone using
> Intel or other brands that aren?t as expensive to capture their traffic? We
> are looking at doing all 10 Gig connections into the Bro Cluster.
>
>
>
> Thanks for all your answers.
>
>
>
> --
>
> Richard Giesige
>
> IT Security Analyst
>
> Office of Information Security
>
> Oregon State University
>
>
>
> "OSU staff will NEVER ask for you password.
>
> Never email or share your password with anyone."
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160607/8306609a/attachment.html 

From troyj at maine.edu  Tue Jun  7 19:21:07 2016
From: troyj at maine.edu (Troy Jordan)
Date: Tue, 7 Jun 2016 22:21:07 -0400
Subject: [Bro] spicy performance question
Message-ID: <878a7986-acbb-db1f-33f1-46ddb672aae4@maine.edu>

Spicy devs,

Since Spicy is still in development, is it to be expected that a
Spicy-based Bro parser would perform significantly slower than an
existing .pac parser of the same protocol?

In my particular testing environment, the pac-baseed modbus parser
processes 99% of a given modbus trace file when replayed at a specific
speed with tcpreplay (logging enabled).

The attached parser is a minimal modbus parser in Spicy which processes
< 50% of the same file (no logging enabled). Is this to be expected?

- Troy


-- 


                     	  Troy Jordan 
                   t r o y j @ m a i n e . e d u
			   GIAC GCIH,GCIA
------------------------------------------------------------
                Network Systems Security Analyst
             Information Technology Security Office
                    University of Maine System
------------------------------------------------------------
233 Science Building           |     voice: 207.561.3590
Portland, ME 04103             |     fax:   509.351.3650



"As you all know, Security Is Mortals chiefest Enemy"
 William Shakespeare, Macbeth

-------------- next part --------------


module MODBUS;
import Bro;


# MessageStream: a list of Messages 

export type MessageStream = unit {
    Messages : list<Message> ;  
};

#
# Header:
#

export type ModbusHeader = unit {

    trans  : uint<16> ;
    proto  : uint<16> ;
    len    : uint<16> ;
    unitId : uint<8> ;
};

#
# Message:  
#
export type Message = unit {

    header : ModbusHeader;
    fcode  : uint8;
    payload: bytes &length=self.header.len - 2;

};
-------------- next part --------------

grammar modbus.pac2;

protocol analyzer pac2::MODBUS over TCP:
    parse with MODBUS::MessageStream,
    port 502/tcp,
    replaces MODBUS;

From martin.liras at gmail.com  Wed Jun  8 01:43:04 2016
From: martin.liras at gmail.com (Luis Martin Liras)
Date: Wed, 8 Jun 2016 10:43:04 +0200
Subject: [Bro] log streams in a bro cluster
Message-ID: <5757DA98.5010606@gmail.com>

Hi all,

I need some help with the logs generated by a Bro Cluster:


I have 5 bro scripts that run in all workers of my cluster 
infrastructure. All of them work OK, sending notices to the manager and 
all the staff, but one of them should create a LOG stream (warnings.log) 
that I can't find anywhere:

Log::create_stream(umas::WARN, [$columns=warn_info,$path="warnings"]);

If I run my script in a single bro installation, all logs and notices 
seem to work, but I need it working in a cluster infrastructure.


I expected this Log stream to be sent to the 'logs' directory in the 
manager, but that log file is not there. Only standard log files 
(dns.log, http.log, stdout.log, etc) are copied to the 'logs' directory.

This warnings.log file do not appear either anywhere in the worker, and 
not error log file is shown, so... I'm lost.

I anyone can shed some light into this, I would appreciate it.


The other problem I have is the following: My script should open a 
config file. In a single machine infrastructure this config file is in 
the same directory of the scripts, and everything work fine. The file is 
opened and read. However in a cluster infrastructure the file is not 
opened in the workers. I find that the file is copied by broctl to the 
worker BUT it is not read when the bro script is running. Anyone can 
tell me what I'm doing wrong or where should I locate that file in the 
workers?

Thank you for any help!!



From grahambridgeland at yahoo.co.uk  Wed Jun  8 03:31:14 2016
From: grahambridgeland at yahoo.co.uk (Graham Bridgeland)
Date: Wed, 8 Jun 2016 10:31:14 +0000 (UTC)
Subject: [Bro] Script examples usign the x509 Functions
References: <1958790734.549582.1465381874850.JavaMail.yahoo.ref@mail.yahoo.com>
Message-ID: <1958790734.549582.1465381874850.JavaMail.yahoo@mail.yahoo.com>

Hello
Can anyone point me to any simple examples of using the various x509 functions, e.g. x509_verify and x509_ocsp_verify?
I've trawled through the site and the SSL exercise has a huge amount of great information and have implemented the various events to extract good information. However,?I can't find a simple starting point of how to implement these functions?to extract additional information out of the pcap files I've collected.
Any assistance would be appreciated.
RegardsGraham
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160608/b17b12af/attachment.html 

From jazoff at illinois.edu  Wed Jun  8 05:36:14 2016
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Wed, 8 Jun 2016 12:36:14 +0000
Subject: [Bro] log streams in a bro cluster
In-Reply-To: <5757DA98.5010606@gmail.com>
References: <5757DA98.5010606@gmail.com>
Message-ID: <E944C4EE-735F-464A-9753-B4CE8C19418F@illinois.edu>


> On Jun 8, 2016, at 4:43 AM, Luis Martin Liras <martin.liras at gmail.com> wrote:
> 
> Hi all,
> 
> I need some help with the logs generated by a Bro Cluster:
> 
> 
> I have 5 bro scripts that run in all workers of my cluster 
> infrastructure. All of them work OK, sending notices to the manager and 
> all the staff, but one of them should create a LOG stream (warnings.log) 
> that I can't find anywhere:
> 
> Log::create_stream(umas::WARN, [$columns=warn_info,$path="warnings"]);
> 
> If I run my script in a single bro installation, all logs and notices 
> seem to work, but I need it working in a cluster infrastructure.
> 
> 
> I expected this Log stream to be sent to the 'logs' directory in the 
> manager, but that log file is not there. Only standard log files 
> (dns.log, http.log, stdout.log, etc) are copied to the 'logs' directory.
> 
> This warnings.log file do not appear either anywhere in the worker, and 
> not error log file is shown, so... I'm lost.
> 
> I anyone can shed some light into this, I would appreciate it.
> 

When are you writing to that log?  Just creating the log stream doesn't create the file until you do a 

    Log::write(umas::WARN, record);


> The other problem I have is the following: My script should open a 
> config file. In a single machine infrastructure this config file is in 
> the same directory of the scripts, and everything work fine. The file is 
> opened and read. However in a cluster infrastructure the file is not 
> opened in the workers. I find that the file is copied by broctl to the 
> worker BUT it is not read when the bro script is running. Anyone can 
> tell me what I'm doing wrong or where should I locate that file in the 
> workers?
> 
> Thank you for any help!!

How are you loading the configuration file?

You should be using something like

    local config_path = fmt("%s/my-config.something", @DIR);

otherwise a relative or absolute path may not be what you expect.

-- 
- Justin Azoff



From jazoff at illinois.edu  Wed Jun  8 05:39:32 2016
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Wed, 8 Jun 2016 12:39:32 +0000
Subject: [Bro] Script examples usign the x509 Functions
In-Reply-To: <1958790734.549582.1465381874850.JavaMail.yahoo@mail.yahoo.com>
References: <1958790734.549582.1465381874850.JavaMail.yahoo.ref@mail.yahoo.com>
	<1958790734.549582.1465381874850.JavaMail.yahoo@mail.yahoo.com>
Message-ID: <0ECA4054-9ED2-4D39-9ECF-820B35A46D8A@illinois.edu>


> On Jun 8, 2016, at 6:31 AM, Graham Bridgeland <grahambridgeland at yahoo.co.uk> wrote:
> 
> Hello
> 
> Can anyone point me to any simple examples of using the various x509 functions, e.g. x509_verify and x509_ocsp_verify?
> 
> I've trawled through the site and the SSL exercise has a huge amount of great information and have implemented the various events to extract good information. However, I can't find a simple starting point of how to implement these functions to extract additional information out of the pcap files I've collected.
> 
> Any assistance would be appreciated.
> 
> Regards
> Graham

The script source code and test suite are often the best place to find how certain functions are used:

~/src/bro  $ git grep x509_verify|egrep 'scripts|testing'
scripts/policy/protocols/ssl/validate-certs.bro:	local result = x509_verify(chain, root_certs);
testing/btest/bifs/x509_verify.bro:	local result = x509_verify(chain, SSL::root_certs);
testing/btest/core/leaks/x509_verify.bro:	local result = x509_verify(chain, SSL::root_certs);


scripts/policy/protocols/ssl/validate-certs.bro "Perform full certificate chain validation for SSL certificates." and the two test cases show similar usage.

-- 
- Justin Azoff



From martin.liras at gmail.com  Wed Jun  8 06:19:39 2016
From: martin.liras at gmail.com (Luis Martin Liras)
Date: Wed, 8 Jun 2016 15:19:39 +0200
Subject: [Bro] log streams in a bro cluster
In-Reply-To: <E944C4EE-735F-464A-9753-B4CE8C19418F@illinois.edu>
References: <5757DA98.5010606@gmail.com>
	<E944C4EE-735F-464A-9753-B4CE8C19418F@illinois.edu>
Message-ID: <57581B6B.5040704@gmail.com>

Thank you for your reply Justin.

You are right, probably I didn't explain myself. There's data to be 
logged but the log file is not created.

Actually, if I set:

redef Log::enable_local_logging = T;

in /usr/local/bro/share/bro/base/frameworks/cluster/nodes/worker.bro

...and deploy again, I can see the mentioned log file in the worker node 
(stored in /home/bro/bro/spool/worker-1/warnings.log), but this log file 
is NOT created in the manager.

I expected all the logs from the worker nodes to be copied somehow to 
manager, but it does not seem to work like that.

Rgds



On 08/06/16 14:36, Azoff, Justin S wrote:
>> On Jun 8, 2016, at 4:43 AM, Luis Martin Liras <martin.liras at gmail.com> wrote:
>>
>> Hi all,
>>
>> I need some help with the logs generated by a Bro Cluster:
>>
>>
>> I have 5 bro scripts that run in all workers of my cluster
>> infrastructure. All of them work OK, sending notices to the manager and
>> all the staff, but one of them should create a LOG stream (warnings.log)
>> that I can't find anywhere:
>>
>> Log::create_stream(umas::WARN, [$columns=warn_info,$path="warnings"]);
>>
>> If I run my script in a single bro installation, all logs and notices
>> seem to work, but I need it working in a cluster infrastructure.
>>
>>
>> I expected this Log stream to be sent to the 'logs' directory in the
>> manager, but that log file is not there. Only standard log files
>> (dns.log, http.log, stdout.log, etc) are copied to the 'logs' directory.
>>
>> This warnings.log file do not appear either anywhere in the worker, and
>> not error log file is shown, so... I'm lost.
>>
>> I anyone can shed some light into this, I would appreciate it.
>>
> When are you writing to that log?  Just creating the log stream doesn't create the file until you do a
>
>      Log::write(umas::WARN, record);
>
>
>> The other problem I have is the following: My script should open a
>> config file. In a single machine infrastructure this config file is in
>> the same directory of the scripts, and everything work fine. The file is
>> opened and read. However in a cluster infrastructure the file is not
>> opened in the workers. I find that the file is copied by broctl to the
>> worker BUT it is not read when the bro script is running. Anyone can
>> tell me what I'm doing wrong or where should I locate that file in the
>> workers?
>>
>> Thank you for any help!!
> How are you loading the configuration file?
>
> You should be using something like
>
>      local config_path = fmt("%s/my-config.something", @DIR);
>
> otherwise a relative or absolute path may not be what you expect.
>


From robin at icir.org  Wed Jun  8 13:41:26 2016
From: robin at icir.org (Robin Sommer)
Date: Wed, 8 Jun 2016 13:41:26 -0700
Subject: [Bro] spicy performance question
In-Reply-To: <878a7986-acbb-db1f-33f1-46ddb672aae4@maine.edu>
References: <878a7986-acbb-db1f-33f1-46ddb672aae4@maine.edu>
Message-ID: <20160608204126.GB4809@icir.org>



On Tue, Jun 07, 2016 at 22:21 -0400, Troy Jordan wrote:

> Since Spicy is still in development, is it to be expected that a
> Spicy-based Bro parser would perform significantly slower than an
> existing .pac parser of the same protocol?

In general, no, it's not expected; see the measurement results in the
HILTI paper, which show slightly worse performance in comparison with
existing analyzers (binpac or not), but not dramatically. That said, I
can't rule out that there are still major bottlenecks somewhere in
Spicy that trigger in specific situations.

> In my particular testing environment, the pac-baseed modbus parser
> processes 99% of a given modbus trace file when replayed at a specific
> speed with tcpreplay (logging enabled).

Measuring performance with live traffic is tricky, I usually do such
profiling offline from a trace: just measure execution time Bro needs
to get through the trace (with Spicy you need to exclude the startup
time, though, as that can still be substantial still right now).

The one thing that running offline doesn't get you are potential brief
processing spikes that cause trouble down the read: say there's one
packet causing Spicy go to into a looong loop for some reason; that
could then lead Bro to drop packets because it can't keep up buffering
incoming traffic long enough. In fact, that's one large reason why
Spicy remains not ready for production: as far as I know, it hasn't
been used for live analysis much at all so far, i.e., such effects are
not understood yet. Bro used to have such problems in early versions
as well, which got rooted out only over time.

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From daniel.guerra69 at gmail.com  Wed Jun  8 16:27:03 2016
From: daniel.guerra69 at gmail.com (Daniel Guerra)
Date: Thu, 9 Jun 2016 01:27:03 +0200
Subject: [Bro]  docker-compose bro elasticsearch
Message-ID: <4BE5F7F5-EADD-4A4A-89E3-F0AC21A2723D@gmail.com>

Hi All,

I have made a docker-compose yml file that starts and configures a bro ELK combination.
The connection pcap is available from kibana and the extracted files too. It might need some
improvement but the basic concept works. 

wget https://raw.githubusercontent.com/danielguerra69/bro-debian-elasticsearch/master/docker-compose.yml
export DOCKERHOST=<dockerhost-ip>
export COMPOSE_API_VERSION=1.18
docker-compose pull
docker-compose up

It listens on tcp port 1969 for pcap files. 
nc <dockerhost-ip> 1969 < my.pcap
tcpdump -i eth0 -s 0 -w - not host <dockerhost-ip> | nc <dockerhost-ip> 1969

And kibana listens to 5601

http://<dockerhost-ip>:5601/

Regards,

Daniel

From bmixonb1 at cs.unm.edu  Wed Jun  8 18:53:54 2016
From: bmixonb1 at cs.unm.edu (Ben Mixon-Baca)
Date: Wed, 8 Jun 2016 18:53:54 -0700
Subject: [Bro] SSL Question
Message-ID: <5758CC32.5030104@cs.unm.edu>

Does Bro make the server's prime it sent to a client in the diffie
hellman key exchange visible?

For example, if a client on my network is talking to an apache server,
would I be able to print the prime the server sends to the client?
-- 
Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: OpenPGP digital signature
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160608/05d124f7/attachment.bin 

From slagell at illinois.edu  Wed Jun  8 19:01:26 2016
From: slagell at illinois.edu (Slagell, Adam J)
Date: Thu, 9 Jun 2016 02:01:26 +0000
Subject: [Bro] SSL Question
In-Reply-To: <5758CC32.5030104@cs.unm.edu>
References: <5758CC32.5030104@cs.unm.edu>
Message-ID: <E5565200-5B2B-4E49-A522-59DC79259D98@illinois.edu>

I don?t think you mean to ask what you are asking. In regular DH over a finite field, the prime that determines the group is not even secret or terribly interesting.

Stepping back a bit, what are you trying to accomplish?

:Adam
> On Jun 8, 2016, at 8:53 PM, Ben Mixon-Baca <bmixonb1 at cs.unm.edu> wrote:
> 
> Does Bro make the server's prime it sent to a client in the diffie
> hellman key exchange visible?
> 
> For example, if a client on my network is talking to an apache server,
> would I be able to print the prime the server sends to the client?
> -- 
> Ben
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

------

Adam J. Slagell
Chief Information Security Officer
Director, Cybersecurity Division
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
www.slagell.info

"Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." 










From bmixonb1 at cs.unm.edu  Wed Jun  8 19:30:23 2016
From: bmixonb1 at cs.unm.edu (Ben Mixon-Baca)
Date: Wed, 8 Jun 2016 19:30:23 -0700
Subject: [Bro] SSL Question
In-Reply-To: <E5565200-5B2B-4E49-A522-59DC79259D98@illinois.edu>
References: <5758CC32.5030104@cs.unm.edu>
	<E5565200-5B2B-4E49-A522-59DC79259D98@illinois.edu>
Message-ID: <5758D4BF.8080706@cs.unm.edu>

Right, but the server and client do have to agree on one of the primes
the will end up using right? I was under the impression that there were
different primes the client and server could decide to use to make a
shared secret.

On 06/08/2016 07:01 PM, Slagell, Adam J wrote:
> I don?t think you mean to ask what you are asking. In regular DH over a finite field, the prime that determines the group is not even secret or terribly interesting.
> 
> Stepping back a bit, what are you trying to accomplish?
> 
> :Adam
>> On Jun 8, 2016, at 8:53 PM, Ben Mixon-Baca <bmixonb1 at cs.unm.edu> wrote:
>>
>> Does Bro make the server's prime it sent to a client in the diffie
>> hellman key exchange visible?
>>
>> For example, if a client on my network is talking to an apache server,
>> would I be able to print the prime the server sends to the client?
>> -- 
>> Ben
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> ------
> 
> Adam J. Slagell
> Chief Information Security Officer
> Director, Cybersecurity Division
> National Center for Supercomputing Applications
> University of Illinois at Urbana-Champaign
> www.slagell.info
> 
> "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." 
> 
> 
> 
> 
> 
> 
> 
> 

-- 
Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: OpenPGP digital signature
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160608/c9f95e5a/attachment.bin 

From bmixonb1 at cs.unm.edu  Wed Jun  8 19:33:53 2016
From: bmixonb1 at cs.unm.edu (Ben Mixon-Baca)
Date: Wed, 8 Jun 2016 19:33:53 -0700
Subject: [Bro] SSL Question
In-Reply-To: <E5565200-5B2B-4E49-A522-59DC79259D98@illinois.edu>
References: <5758CC32.5030104@cs.unm.edu>
	<E5565200-5B2B-4E49-A522-59DC79259D98@illinois.edu>
Message-ID: <5758D591.8000409@cs.unm.edu>

I am trying to determine if the prime being used is from apache's
mod_ssl. I didn't know if it was possible to use some field available in
the Cert record or another record to determine the prime implicitly
since they are public.

On 06/08/2016 07:01 PM, Slagell, Adam J wrote:
> I don?t think you mean to ask what you are asking. In regular DH over a finite field, the prime that determines the group is not even secret or terribly interesting.
> 
> Stepping back a bit, what are you trying to accomplish?
> 
> :Adam
>> On Jun 8, 2016, at 8:53 PM, Ben Mixon-Baca <bmixonb1 at cs.unm.edu> wrote:
>>
>> Does Bro make the server's prime it sent to a client in the diffie
>> hellman key exchange visible?
>>
>> For example, if a client on my network is talking to an apache server,
>> would I be able to print the prime the server sends to the client?
>> -- 
>> Ben
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> ------
> 
> Adam J. Slagell
> Chief Information Security Officer
> Director, Cybersecurity Division
> National Center for Supercomputing Applications
> University of Illinois at Urbana-Champaign
> www.slagell.info
> 
> "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." 
> 
> 
> 
> 
> 
> 
> 
> 

-- 
Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: OpenPGP digital signature
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160608/94f31ca0/attachment.bin 

From johanna at icir.org  Wed Jun  8 19:45:30 2016
From: johanna at icir.org (Johanna Amann)
Date: Wed, 08 Jun 2016 19:45:30 -0700
Subject: [Bro] SSL Question
In-Reply-To: <5758D591.8000409@cs.unm.edu>
References: <5758CC32.5030104@cs.unm.edu>
	<E5565200-5B2B-4E49-A522-59DC79259D98@illinois.edu>
	<5758D591.8000409@cs.unm.edu>
Message-ID: <829A772B-5A51-476F-BD85-F3CFB0C74DB5@icir.org>

The server message sent to the client, including p, g, and Ys is 
available in the event ssl_dh_server_params: 
https://www.bro.org/sphinx/scripts/base/bif/plugins/Bro_SSL.events.bif.bro.html#id-ssl_dh_server_params

I hope this helps,
  Johanna

On 8 Jun 2016, at 19:33, Ben Mixon-Baca wrote:

> I am trying to determine if the prime being used is from apache's
> mod_ssl. I didn't know if it was possible to use some field available 
> in
> the Cert record or another record to determine the prime implicitly
> since they are public.
>
> On 06/08/2016 07:01 PM, Slagell, Adam J wrote:
>> I don?t think you mean to ask what you are asking. In regular DH 
>> over a finite field, the prime that determines the group is not even 
>> secret or terribly interesting.
>>
>> Stepping back a bit, what are you trying to accomplish?
>>
>> :Adam
>>> On Jun 8, 2016, at 8:53 PM, Ben Mixon-Baca <bmixonb1 at cs.unm.edu> 
>>> wrote:
>>>
>>> Does Bro make the server's prime it sent to a client in the diffie
>>> hellman key exchange visible?
>>>
>>> For example, if a client on my network is talking to an apache 
>>> server,
>>> would I be able to print the prime the server sends to the client?
>>> -- 
>>> Ben
>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>> ------
>>
>> Adam J. Slagell
>> Chief Information Security Officer
>> Director, Cybersecurity Division
>> National Center for Supercomputing Applications
>> University of Illinois at Urbana-Champaign
>> www.slagell.info
>>
>> "Under the Illinois Freedom of Information Act (FOIA), any written 
>> communication to or from University employees regarding University 
>> business is a public record and may be subject to public disclosure."
>>
>>
>>
>>
>>
>>
>>
>>
>
> -- 
> Ben
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From bmixonb1 at cs.unm.edu  Wed Jun  8 19:49:10 2016
From: bmixonb1 at cs.unm.edu (Ben Mixon-Baca)
Date: Wed, 8 Jun 2016 19:49:10 -0700
Subject: [Bro] SSL Question
In-Reply-To: <5758D591.8000409@cs.unm.edu>
References: <5758CC32.5030104@cs.unm.edu>
	<E5565200-5B2B-4E49-A522-59DC79259D98@illinois.edu>
	<5758D591.8000409@cs.unm.edu>
Message-ID: <5758D926.9070900@cs.unm.edu>

Maybe a better question is, are the parameters negotiated in the client
and server hello available directly in Bro through the SSL::Info record,
X509::Info record, or some other record? Or are the not directly available?

I am trying to determine if a specific prime is being used.

On 06/08/2016 07:33 PM, Ben Mixon-Baca wrote:
> I am trying to determine if the prime being used is from apache's
> mod_ssl. I didn't know if it was possible to use some field available in
> the Cert record or another record to determine the prime implicitly
> since they are public.
> 
> On 06/08/2016 07:01 PM, Slagell, Adam J wrote:
>> I don?t think you mean to ask what you are asking. In regular DH over a finite field, the prime that determines the group is not even secret or terribly interesting.
>>
>> Stepping back a bit, what are you trying to accomplish?
>>
>> :Adam
>>> On Jun 8, 2016, at 8:53 PM, Ben Mixon-Baca <bmixonb1 at cs.unm.edu> wrote:
>>>
>>> Does Bro make the server's prime it sent to a client in the diffie
>>> hellman key exchange visible?
>>>
>>> For example, if a client on my network is talking to an apache server,
>>> would I be able to print the prime the server sends to the client?
>>> -- 
>>> Ben
>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>> ------
>>
>> Adam J. Slagell
>> Chief Information Security Officer
>> Director, Cybersecurity Division
>> National Center for Supercomputing Applications
>> University of Illinois at Urbana-Champaign
>> www.slagell.info
>>
>> "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." 
>>
>>
>>
>>
>>
>>
>>
>>
> 
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 

-- 
Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: OpenPGP digital signature
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160608/289abeeb/attachment.bin 

From bmixonb1 at cs.unm.edu  Wed Jun  8 19:50:43 2016
From: bmixonb1 at cs.unm.edu (Ben Mixon-Baca)
Date: Wed, 8 Jun 2016 19:50:43 -0700
Subject: [Bro] SSL Question
In-Reply-To: <829A772B-5A51-476F-BD85-F3CFB0C74DB5@icir.org>
References: <5758CC32.5030104@cs.unm.edu>
	<E5565200-5B2B-4E49-A522-59DC79259D98@illinois.edu>
	<5758D591.8000409@cs.unm.edu>
	<829A772B-5A51-476F-BD85-F3CFB0C74DB5@icir.org>
Message-ID: <5758D983.4000600@cs.unm.edu>

Awesome that is exactly what I was looking for. Sorry if my question
wasn't clear, I only understand DH at a high level so my terminology
might not have been consistent with convention.

On 06/08/2016 07:45 PM, Johanna Amann wrote:
> The server message sent to the client, including p, g, and Ys is
> available in the event ssl_dh_server_params:
> https://www.bro.org/sphinx/scripts/base/bif/plugins/Bro_SSL.events.bif.bro.html#id-ssl_dh_server_params
> 
> 
> I hope this helps,
>  Johanna
> 
> On 8 Jun 2016, at 19:33, Ben Mixon-Baca wrote:
> 
>> I am trying to determine if the prime being used is from apache's
>> mod_ssl. I didn't know if it was possible to use some field available in
>> the Cert record or another record to determine the prime implicitly
>> since they are public.
>>
>> On 06/08/2016 07:01 PM, Slagell, Adam J wrote:
>>> I don?t think you mean to ask what you are asking. In regular DH over
>>> a finite field, the prime that determines the group is not even
>>> secret or terribly interesting.
>>>
>>> Stepping back a bit, what are you trying to accomplish?
>>>
>>> :Adam
>>>> On Jun 8, 2016, at 8:53 PM, Ben Mixon-Baca <bmixonb1 at cs.unm.edu> wrote:
>>>>
>>>> Does Bro make the server's prime it sent to a client in the diffie
>>>> hellman key exchange visible?
>>>>
>>>> For example, if a client on my network is talking to an apache server,
>>>> would I be able to print the prime the server sends to the client?
>>>> -- 
>>>> Ben
>>>>
>>>> _______________________________________________
>>>> Bro mailing list
>>>> bro at bro-ids.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>
>>> ------
>>>
>>> Adam J. Slagell
>>> Chief Information Security Officer
>>> Director, Cybersecurity Division
>>> National Center for Supercomputing Applications
>>> University of Illinois at Urbana-Champaign
>>> www.slagell.info
>>>
>>> "Under the Illinois Freedom of Information Act (FOIA), any written
>>> communication to or from University employees regarding University
>>> business is a public record and may be subject to public disclosure."
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> -- 
>> Ben
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-- 
Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: OpenPGP digital signature
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160608/a3f48228/attachment.bin 

From dnj0496 at gmail.com  Wed Jun  8 22:23:25 2016
From: dnj0496 at gmail.com (Dk Jack)
Date: Wed, 8 Jun 2016 22:23:25 -0700
Subject: [Bro] timestamp
Message-ID: <CAGATen6cgrx2TgCcjph3XmA9xtaF9bZJxejQowaOYx9_r4PbVA@mail.gmail.com>

Hi,
Seems like the timestamp in the bro log file come from the system/wall
clock. Is there for bro to force it to use the timestamp in the pcap file?
Thanks.

dk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160608/ae6431c6/attachment.html 

From sharon_sachin at yahoo.co.in  Thu Jun  9 02:02:21 2016
From: sharon_sachin at yahoo.co.in (sachin sharma)
Date: Thu, 9 Jun 2016 09:02:21 +0000 (UTC)
Subject: [Bro] Information regarding worker thread in the bro architecture
References: <1916074450.130961.1465462941904.JavaMail.yahoo.ref@mail.yahoo.com>
Message-ID: <1916074450.130961.1465462941904.JavaMail.yahoo@mail.yahoo.com>

Dear all,
I am currently studying the bro architecture and bit confused how the worker thread implemented in the cluster architecture. My question is: does each worker thread perform the same set of activities as of others worker threads or each one ?can have different set of capabilities? If so, could you please let me know the set of capabilities a work thread can have? In addition, does each worker thread runs an event engine separately? Could you please elaborate a bit on the worker thread?

Kind Regards,Sachin.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160609/8c15c8a0/attachment.html 

From martin.liras at gmail.com  Thu Jun  9 05:16:01 2016
From: martin.liras at gmail.com (Luis Martin Liras)
Date: Thu, 9 Jun 2016 14:16:01 +0200
Subject: [Bro] log streams in a bro cluster
In-Reply-To: <E944C4EE-735F-464A-9753-B4CE8C19418F@illinois.edu>
References: <5757DA98.5010606@gmail.com>
	<E944C4EE-735F-464A-9753-B4CE8C19418F@illinois.edu>
Message-ID: <57595E01.6050603@gmail.com>

Hi all,

A little bit of investigation and I found (with a tcpdump) that the logs 
arrive to the manager process BUT they are not stored to disk. Then I 
found the following entry at the beginning of the communication.log file :

1465472892.006482    manager    parent    -    -    -    error 
[#10002/192.168.1.10:57322] unserializing event/function 
Notice::cluster_notice: write error for creating writer


followed by a lot of errors like:

1465473767.549373    manager    parent    -    -    -    error 
[#10001/192.168.1.10:57322] unserializing event/function 
Notice::cluster_notice: write error for log entry




It seems that for any reason, the deserializer writer is not able to 
open a writer ad then it's not possible to write the log files.

I found this error in github in RemoteSerializer.cc, in its function 
'ProcessLogCreateWriter':

https://github.com/bro/bro/blob/f5ce4785ea96b56643c092331a16308f071c8092/src/RemoteSerializer.cc



But I still cannot figure out why is this happening.


Tried to change permissions to all log directories to 777, but didn't work.

Any idea why could be happening this error?


Thank you!!


On 08/06/16 14:36, Azoff, Justin S wrote:
>> On Jun 8, 2016, at 4:43 AM, Luis Martin Liras <martin.liras at gmail.com> wrote:
>>
>> Hi all,
>>
>> I need some help with the logs generated by a Bro Cluster:
>>
>>
>> I have 5 bro scripts that run in all workers of my cluster
>> infrastructure. All of them work OK, sending notices to the manager and
>> all the staff, but one of them should create a LOG stream (warnings.log)
>> that I can't find anywhere:
>>
>> Log::create_stream(umas::WARN, [$columns=warn_info,$path="warnings"]);
>>
>> If I run my script in a single bro installation, all logs and notices
>> seem to work, but I need it working in a cluster infrastructure.
>>
>>
>> I expected this Log stream to be sent to the 'logs' directory in the
>> manager, but that log file is not there. Only standard log files
>> (dns.log, http.log, stdout.log, etc) are copied to the 'logs' directory.
>>
>> This warnings.log file do not appear either anywhere in the worker, and
>> not error log file is shown, so... I'm lost.
>>
>> I anyone can shed some light into this, I would appreciate it.
>>
> When are you writing to that log?  Just creating the log stream doesn't create the file until you do a
>
>      Log::write(umas::WARN, record);
>
>
>> The other problem I have is the following: My script should open a
>> config file. In a single machine infrastructure this config file is in
>> the same directory of the scripts, and everything work fine. The file is
>> opened and read. However in a cluster infrastructure the file is not
>> opened in the workers. I find that the file is copied by broctl to the
>> worker BUT it is not read when the bro script is running. Anyone can
>> tell me what I'm doing wrong or where should I locate that file in the
>> workers?
>>
>> Thank you for any help!!
> How are you loading the configuration file?
>
> You should be using something like
>
>      local config_path = fmt("%s/my-config.something", @DIR);
>
> otherwise a relative or absolute path may not be what you expect.
>


From jazoff at illinois.edu  Thu Jun  9 06:01:32 2016
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 9 Jun 2016 13:01:32 +0000
Subject: [Bro] log streams in a bro cluster
In-Reply-To: <57595E01.6050603@gmail.com>
References: <5757DA98.5010606@gmail.com>
	<E944C4EE-735F-464A-9753-B4CE8C19418F@illinois.edu>
	<57595E01.6050603@gmail.com>
Message-ID: <C8D59376-7A90-4ADD-AC7D-4AA0A897B21C@illinois.edu>

> On Jun 9, 2016, at 7:16 AM, Luis Martin Liras <martin.liras at gmail.com> wrote:
> 
> Hi all,
> 
> A little bit of investigation and I found (with a tcpdump) that the logs arrive to the manager process BUT they are not stored to disk. Then I found the following entry at the beginning of the communication.log file :
> 
> 1465472892.006482    manager    parent    -    -    -    error [#10002/192.168.1.10:57322] unserializing event/function Notice::cluster_notice: write error for creating writer
> 
> 
> followed by a lot of errors like:
> 
> 1465473767.549373    manager    parent    -    -    -    error [#10001/192.168.1.10:57322] unserializing event/function Notice::cluster_notice: write error for log entry
> 

Ah.. the "write error for creating writer" message is a bit misleading, it outputs that for any error in the process.

Those messages also point to an issue with notices, not with your log file.  Are you also calling NOTICE somewhere?

Your problem could be that there is a discrepancy between how you defined warn_info and what you are passing Log::write.  Non clustered bro doesn't need to serialize/deserialize the messages so you can get away with certain mistakes that break once you use a cluster.

The standard log files all use the same mechanism, so if you are getting an http.log then your remote communication is working and there should be nothing preventing your log file from being written.

It would help if you could post your scripts somewhere or try to come up with a minimal example that shows the problem.

This is the simplest example for writing a custom log file:

http://try.bro.org/#/trybro?example=log

If you modify it like this and deploy it to a cluster you should get a foo.log containing things like

1465477100.871640	hello from manager
1465477105.884494	hello from manager
1465477104.537564	hello from proxy-1
1465477108.648193	hello from worker-1-2
1465477108.527117	hello from worker-1-1
1465477110.887240	hello from manager
1465477113.652352	hello from worker-1-2
1465477109.552765	hello from proxy-1



module FOO;

export {
    redef enum Log::ID += { LOG };

    type Info: record {
        ts: time &log;
        msg: string &log;
    };
}

event do_log()
    {
    local l = [$ts = network_time(), $msg=fmt("hello from %s", peer_description)];
    Log::write(LOG, l);
    schedule 5sec {do_log() };
    }

event bro_init()
{
    Log::create_stream(LOG, [$columns=Info]);

    schedule 5sec {do_log() };
}



-- 
- Justin Azoff




From rpostal at bricata.com  Thu Jun  9 07:18:33 2016
From: rpostal at bricata.com (Rikki Postal)
Date: Thu, 9 Jun 2016 14:18:33 +0000
Subject: [Bro] Consulting Opportunity
Message-ID: <DM2PR0601MB09704A55DB6485156A10AD7EBC5F0@DM2PR0601MB0970.namprd06.prod.outlook.com>

[X]



Consulting Opportunity



Location: Columbia MD / Flexible



Reports to: Chief Technology Officer



Main Purpose: Consult on (or lead) the design and development of IDS/IPS architecture that integrates the event-driven analytic capabilities of Bro with the high speed Suricata engine



Job Level: Senior Technical Leader



Skills and Experience:





* 15+ years of work experience in software development of network security products
* 5+ years of relevant work experience designing and developing event-driven distributed software architectures for enterprise applications, preferably around network security
* Prior experience with IDS/IPS systems such as Suricata or Snort
* Significant Bro scripting experience
* Skills in installing, configuring and managing Bro in various high and low speed environments.
* Solid programming skills.
* Ability to work as part of a cross-functional team; strong written and oral communication skills.
* C/C++ Appreciated, not required
* Working knowledge of Debian-HIGHLY DESIRED
* BS in computer science





Role Requirements:  Consult/Lead design and development of NextGen architecture that integrates Bro's event-driven analytic capability with Suricata IDS/IPS.  Design and develop overall system architecture within hardware and software constraints that is aligned with current/future product roadmap and enterprise customer needs to deliver on Bricata vision of 5th generation IDS/IPS.  Articulate clearly technical trades and options for CTO and senior leadership team in terms of costs/benefits to product lifecycle.  Define time-phased quantifiable goals and objectives that align with overall product roadmap and vision.  Develop time-phased plan and resource needs to achieve vision.  Ensure product is well-aligned with current and future enterprise architecture needs.



Lucrative compensation with equity.



To apply contact Rikki Postal; rpostal at bricata.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160609/da44ca4c/attachment.html 

From martin.liras at gmail.com  Thu Jun  9 07:48:16 2016
From: martin.liras at gmail.com (Luis Martin Liras)
Date: Thu, 9 Jun 2016 16:48:16 +0200
Subject: [Bro] log streams in a bro cluster
In-Reply-To: <C8D59376-7A90-4ADD-AC7D-4AA0A897B21C@illinois.edu>
References: <5757DA98.5010606@gmail.com>
	<E944C4EE-735F-464A-9753-B4CE8C19418F@illinois.edu>
	<57595E01.6050603@gmail.com>
	<C8D59376-7A90-4ADD-AC7D-4AA0A897B21C@illinois.edu>
Message-ID: <575981B0.7020707@gmail.com>

Hi Justin, thank you for your help.

I reply inline in your email. Sorry for the capital letters, they are 
just to make a difference between your comments and mine:


On 09/06/16 15:01, Azoff, Justin S wrote:
>> On Jun 9, 2016, at 7:16 AM, Luis Martin Liras <martin.liras at gmail.com> wrote:
>>
>> Hi all,
>>
>> A little bit of investigation and I found (with a tcpdump) that the logs arrive to the manager process BUT they are not stored to disk. Then I found the following entry at the beginning of the communication.log file :
>>
>> 1465472892.006482    manager    parent    -    -    -    error [#10002/192.168.1.10:57322] unserializing event/function Notice::cluster_notice: write error for creating writer
>>
>>
>> followed by a lot of errors like:
>>
>> 1465473767.549373    manager    parent    -    -    -    error [#10001/192.168.1.10:57322] unserializing event/function Notice::cluster_notice: write error for log entry
>>
> Ah.. the "write error for creating writer" message is a bit misleading, it outputs that for any error in the process.
>
> Those messages also point to an issue with notices, not with your log file.  Are you also calling NOTICE somewhere?

YES I AM. PROBABLY YOU ARE RIGHT AND THAT ERROR IS RELATED TO THE NOTICE 
FRAMEWORK. BUT NOTICES AS BEING LOGGED FINE.


> Your problem could be that there is a discrepancy between how you defined warn_info and what you are passing Log::write.
I DON'T THINK SO BECAUSE WHEN RUNNING STANDALONE, THE LOGS ARE WRITTEN FINE.

> Non clustered bro doesn't need to serialize/deserialize the messages so you can get away with certain mistakes that break once you use a cluster.
...BUT I'M WORKING IN A CLUSTER CONFIGURATION. HENCE THE DESERIALIZE 
ERROR. WHEN I MENTION STANDALONE I BECAUSE I CAN CHECK ALSO IN A SINGLE 
NODE BRO CONFIGURATION, BUT MY OBJECTIVE IS TO MAKE IT WORK IN A CLUSTER.

> The standard log files all use the same mechanism, so if you are getting an http.log then your remote communication is working and there should be nothing preventing your log file from being written.
YEAP, I'M GETTING DNS or CONN LOGS, SO THE LOGGING FRAMEWORK SEEMS TO BE 
LOGGING FINE BUT STILL MY OWN LOG FILES ARE NOT WRITTEN TO DISK IN THE 
MANAGER NODE.

HAVE A LOOK TO THIS.  THIS IS THE LOG DIRECTORY IN ONE OF THE WORKERS:

root at Worker1:/home/bro/bro/spool/worker-1# ls -l
total 2340
-rw-r--r-- 1 root root    6005 Jun  9 16:35 communication.log
-rw-r--r-- 1 root root    8709 Jun  9 16:35 conn.log
-rw-r--r-- 1 root root   20596 Jun  9 16:35 dns.log
-rw-r--r-- 1 root root     292 Jun  9 16:32 known_services.log
-rw-r--r-- 1 root root   23596 Jun  9 16:32 loaded_scripts.log
-rw-r--r-- 1 root root 1425948 Jun  9 16:35 my-prot.log
-rw-r--r-- 1 root root     231 Jun  9 16:32 packet_filter.log
-rw-r--r-- 1 root root     290 Jun  9 16:32 reporter.log
-rw-r--r-- 1 root root      79 Jun  9 16:32 stderr.log
-rw-r--r-- 1 root root     188 Jun  9 16:32 stdout.log
-rw-r--r-- 1 root root     898 Jun  9 16:34 weird.log


...AND THIS IS THE LOG DIRECTORY IN THE MANAGER:

root at Manager:/home/bro/logs# ls -l
total 160
-rw-r--r-- 1 root root 75440 Jun  9 16:36 communication.log
-rw-r--r-- 1 root root   485 Jun  9 16:33 conn.log
-rw-r--r-- 1 root root   429 Jun  9 16:33 dns.log
-rw-r--r-- 1 root root   199 Jun  9 16:32 known_services.log
-rw-r--r-- 1 root root 23113 Jun  9 16:32 loaded_scripts.log
-rw-r--r-- 1 root root   187 Jun  9 16:32 packet_filter.log
-rw-r--r-- 1 root root   237 Jun  9 16:32 reporter.log
-rw-r--r-- 1 root root     0 Jun  9 16:32 stderr.log
-rw-r--r-- 1 root root   188 Jun  9 16:32 stdout.log
-rw-r--r-- 1 root root   249 Jun  9 16:32 weird.log



MY_PROT.LOG IS NOT THERE BUT THE OTHERS LOGS ARE. THERE'S SOMETHING I'M 
DOING WRONG AND DON'T KNOW WHAT IT IS..



IT'S STRANGE BECAUSE I CAN SEE WITH TCPDUMP NETWORK TRAFFIC WITH THE 
PROPIETARY LOGS SENT TO THE MANAGER:

     0x0190:  0008 0100 0000 1243 517a 7a36 5733 5567 .......CQzz6W3Ug
     0x01a0:  6d66 4e34 4c59 3372 6600 0000 0d01 040a mfN4LY3rf.......
     0x01b0:  0100 0c00 0000 0d01 040a 0100 6500 0000 ............e...
     0x01c0:  0201 ffff ffff ffff ffff 0000 0008 0100 ................
     0x01d0:  0000 1f30 7835 3020 2d20 5641 5249 4142 ...0x50.-.VARIAB
     0x01e0:  4c45 5f4d 4f4e 4954 4f52 5f52 4551 5545 LE_MONITOR_REQUE
     0x01f0:  5354 0000 0008 0100 0000 0000 0000 1c00 ST..............
     0x0200:  0000 0100 0000 066d 6f64 6275 7300 0000 ................
     0x0210:  0800 0000 0601 41d5 d65d ab02 e200 0000 ......A..]......
     0x0220:  0008 0100 0000 1243 517a 7a36 5733 5567 .......CQzz6W3Ug
     0x0230:  6d66 4e34 4c59 3372 6600 0000 0d01 040a mfN4LY3rf.......
     0x0240:  0100 0c00 0000 0c01 0000 0000 0000 d472 ...............r
     0x0250:  0000 0001 0000 000d 0104 0a01 0065 0000 .............e..
     0x0260:  000c 0100 0000 0000 0001 f600 0000 0100 ................


THIS IS AN EXTRACT FROM A COMMUNICATION BETWEEN WORKER AND MANAGER AND 
THAT'S WHAT MY-PROT.LOG SHOULD LOOK LIKE...


> It would help if you could post your scripts somewhere or try to come up with a minimal example that shows the problem.

OK, I will  try to summarize what my scripts do:

Workers
=====

1.- I have developed an analyzer for a proprietary protocol. It seems to 
work fine.

2.- local-worker.bro:

event bro_init() &priority=5
{

         #############
         # ANALYZERS #
         #############
         Analyzer::enable_analyzer(Analyzer::ANALYZER_MY-PROT);
         Analyzer::register_for_ports(Analyzer::ANALYZER_MY-PROT, My-ports);

         ###########
         # LOGGING #
         ###########
         Log::create_stream(myprot::MY-PROT_LOG, [$columns=myprot_info, 
$path="myprot"]);
         Log::create_stream(myprot::MY-PROT_WARN, 
[$columns=myprot_info,$path="warnings"]);

         ...

}


Both proprietary protocol messages and warnings are written to 
MY-PROT_LOG or MY-PROT_WARN streams with a command like:

    Log::write(log, [$ts=network_time(), $uid=c$uid,  $origin=host, 
$destination=dest, $warn_level=warning_level, $message=texto, 
$options=options]);


At the same time there are other situations (FTP traffic warnings, etc, 
that are generated as notices) with a command like:

    NOTICE([$note=tipo, $msg=texto, $conn=c, $sub=options, 
$n=int_to_count(warning_level)]);


So Yes, I think it's what you are saying, I'm mixing logs and notices... 
but, Shouldn't do I?.


If I run my scripts locally (standalone), they all seem to work fine, 
both logs and notice files are generated , but of course, with this 
configuration, there's only one Bro process and just a single node. The 
problem comes with the manager node.


3.- local-manager.bro

The manager policy script simply manages the Notices through a call to:

hook Notice::policy(not:Notice::Info)
{
    ...
}

...and depending if the notices are related to some allowed or forbidden 
IPs, and depending on the warning_level of the notice, it generates or 
not, an alarm or an email:

         ...
         if (warning_level>email_warning_threshold)
         {
                 add not$actions[Notice::ACTION_EMAIL];
         }
         else
         {
                 add not$actions[Notice::ACTION_ALARM];
         }
         ...


The notices are logged fine in the manager, this is notice.log:

(...)
1465481767.026893    CqZikc1SK1fLN2cvb2    192.168.0.12    54386 
192.168.0.101    502    -    -    -    tcp    (empty) NOT_SEEN_CONN   
  SLAVE=0    192.168.0.12    192.168.0.101    502  2    worker-2   
  Notice::ACTION_LOG    3600.000000    F    -    -  -    -    -
(...)

  but I have just realized that they are all logged as 
NOTICE::ACTION_LOG and not as ACTION_EMAIL or ACTION_ALARM.

It's important to note that I'm not trying to manage my-prot logs in the 
manager policy script. Maybe there's something I need to add in the 
manager policy script to get those logs?


Thank you for your help!.




From jdopheid at illinois.edu  Thu Jun  9 07:54:15 2016
From: jdopheid at illinois.edu (Dopheide, Jeannette M)
Date: Thu, 9 Jun 2016 14:54:15 +0000
Subject: [Bro] =?utf-8?q?BroCon_=E2=80=9916_CFP_expires_tomorrow!?=
Message-ID: <D9AF3BFA-5935-45EF-A7AF-ACA1442946CC@illinois.edu>

Don?t forget to submit your proposal to speak at BroCon. It?s due tomorrow!

------
Jeannette Dopheide
Training and Outreach Coordinator
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign


From: <bro-bounces at bro.org> on behalf of Jeannette Dopheide <jdopheid at illinois.edu>
Date: Saturday, June 4, 2016 at 1:07 PM
To: "bro at bro.org" <bro at bro.org>
Subject: [Bro] BroCon ?16 CFP deadline extended to June 10th

Bro Community,

We are extending the BroCon ?16 call for presentations deadline to Friday, June 10th. For more information about the CFP, see our blog post [1]. And don't forget to register!

[1] http://blog.bro.org/2016/06/brocon-16-cfp-deadline-extended-to-june.html

See you in September,

The Bro Project
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160609/6682f4d6/attachment-0001.html 

From luis.e.jimenez01 at gmail.com  Thu Jun  9 08:22:26 2016
From: luis.e.jimenez01 at gmail.com (Luis Jimenez)
Date: Thu, 9 Jun 2016 11:22:26 -0400
Subject: [Bro] UID missing
Message-ID: <CAOrRJ9TB4bW_CFztRantxZOGj8hcJKw1DSUO62zXQeM+GHCCKA@mail.gmail.com>

Hello,
I'm fairly new to Bro so please excuse my ignorance.

I'm looking through logs from the Tunnel::LOG analyzer and am seeing that
many records do not have UIDs. I would have thought that every session
would get a UID and am wondering why there would be records without them.

Thank you for your help,
Luis Jimenez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160609/661035d0/attachment.html 

From johanna at icir.org  Thu Jun  9 09:57:41 2016
From: johanna at icir.org (Johanna Amann)
Date: Thu, 9 Jun 2016 09:57:41 -0700
Subject: [Bro] timestamp
In-Reply-To: <CAGATen6cgrx2TgCcjph3XmA9xtaF9bZJxejQowaOYx9_r4PbVA@mail.gmail.com>
References: <CAGATen6cgrx2TgCcjph3XmA9xtaF9bZJxejQowaOYx9_r4PbVA@mail.gmail.com>
Message-ID: <20160609165741.GA15941@Beezling.local>

Hello dk,

if you run Bro on a pcap, the timestamp in the logfile actually are driven
by the timestamps in the pcap file.

If you just do, e.g. bro -r [bro source path]/testing/btest/Traces/irc-dcc-send.trace

you will get timestamps from 2011, when that pcap file was generated.

Johanna

On Wed, Jun 08, 2016 at 10:23:25PM -0700, Dk Jack wrote:
> Hi,
> Seems like the timestamp in the bro log file come from the system/wall
> clock. Is there for bro to force it to use the timestamp in the pcap file?
> Thanks.
> 
> dk

> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From jazoff at illinois.edu  Thu Jun  9 09:58:39 2016
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 9 Jun 2016 16:58:39 +0000
Subject: [Bro] log streams in a bro cluster
In-Reply-To: <575981B0.7020707@gmail.com>
References: <5757DA98.5010606@gmail.com>
	<E944C4EE-735F-464A-9753-B4CE8C19418F@illinois.edu>
	<57595E01.6050603@gmail.com>
	<C8D59376-7A90-4ADD-AC7D-4AA0A897B21C@illinois.edu>
	<575981B0.7020707@gmail.com>
Message-ID: <A425758B-B0BB-4C96-AD90-626EB806DC11@illinois.edu>

> On Jun 9, 2016, at 9:48 AM, Luis Martin Liras <martin.liras at gmail.com> wrote:
> 2.- local-worker.bro:
> 
> event bro_init() &priority=5
> {
> ...

>       Log::create_stream(myprot::MY-PROT_LOG, [$columns=myprot_info, $path="myprot"]);
>        Log::create_stream(myprot::MY-PROT_WARN, [$columns=myprot_info,$path="warnings"]);
> 
>        ...
> 
> }
> 
> 
> 3.- local-manager.bro
> 
...

Ah!  the files you are putting things in is your problem. Take everything you put in local-worker.bro and local-manager.bro and put it into a script and load that script from local.bro

When you use local-worker directly and call create_stream there then the manager knows nothing about about those log streams.

Additionally, if that is where you are defining the notice types, that will break notices as well because when the notice reaches the manager it also has no idea what to do with it.

There aren't many things that people do that actually belong in the local-worker or local-manger scripts, everything should almost always  just go in local.bro.  things like create_stream and NOTICE automatically do the right thing depending on what node it is ran on.

-- 
- Justin Azoff

From johanna at icir.org  Thu Jun  9 10:00:37 2016
From: johanna at icir.org (Johanna Amann)
Date: Thu, 9 Jun 2016 10:00:37 -0700
Subject: [Bro] Information regarding worker thread in the bro
	architecture
In-Reply-To: <1916074450.130961.1465462941904.JavaMail.yahoo@mail.yahoo.com>
References: <1916074450.130961.1465462941904.JavaMail.yahoo.ref@mail.yahoo.com>
	<1916074450.130961.1465462941904.JavaMail.yahoo@mail.yahoo.com>
Message-ID: <20160609170037.GB15941@Beezling.local>

Hello Sachin,

> I am currently studying the bro architecture and bit confused how the
> worker thread implemented in the cluster architecture. My question is:
> does each worker thread perform the same set of activities as of others
> worker threads or each one ?can have different set of capabilities? If
> so, could you please let me know the set of capabilities a work thread
> can have? In addition, does each worker thread runs an event engine
> separately? Could you please elaborate a bit on the worker thread?

Actually, Bro uses multiprocessing instead of multithreading. Which means
that in a cluster, several Bro processes are started using the same
binary. All workers do exactly the same work, just processing packages
from a different network card queue (or, if running on a different machine
from a different cards).

The manager of the cluster is another Bro process which does not handle
network traffic and is mostly responsible to write log files.

All workers and the manager run completely separate event engines and can
also exchange networks between each other (this has to be specified
explicitly for each event).

Scripts can determine if they run on a manager or a worker node and adjust
their behavior depending on that.

I hope this helps,
 Johanna

From johanna at icir.org  Thu Jun  9 10:08:53 2016
From: johanna at icir.org (Johanna Amann)
Date: Thu, 9 Jun 2016 10:08:53 -0700
Subject: [Bro] UID missing
In-Reply-To: <CAOrRJ9TB4bW_CFztRantxZOGj8hcJKw1DSUO62zXQeM+GHCCKA@mail.gmail.com>
References: <CAOrRJ9TB4bW_CFztRantxZOGj8hcJKw1DSUO62zXQeM+GHCCKA@mail.gmail.com>
Message-ID: <20160609170853.GC15941@Beezling.local>

Hello Luis,

> I'm looking through logs from the Tunnel::LOG analyzer and am seeing that
> many records do not have UIDs. I would have thought that every session
> would get a UID and am wondering why there would be records without them.

As far as I am aware, this is currently the case with Socks and HTTP
tunnels. The reasoning there is, that in these case, the tunnel (from the
source machine to the http or socks proxy) will use many different
connections, which together form the tunnel. In this case (many
connections forming a tunnel), no singular connection ID, over which the
traffic is sent, can be logged. Instead, the log-file will contain the
source IP address, a source port of 0, the destination IP address and the
destination port to show the tunnel source (with unspecified port) and the
server destination IP and port.

I hope this helps,
 Johanna

From martin.liras at gmail.com  Fri Jun 10 01:41:29 2016
From: martin.liras at gmail.com (Luis Martin Liras)
Date: Fri, 10 Jun 2016 10:41:29 +0200
Subject: [Bro] log streams in a bro cluster
In-Reply-To: <A425758B-B0BB-4C96-AD90-626EB806DC11@illinois.edu>
References: <5757DA98.5010606@gmail.com>
	<E944C4EE-735F-464A-9753-B4CE8C19418F@illinois.edu>
	<57595E01.6050603@gmail.com>
	<C8D59376-7A90-4ADD-AC7D-4AA0A897B21C@illinois.edu>
	<575981B0.7020707@gmail.com>
	<A425758B-B0BB-4C96-AD90-626EB806DC11@illinois.edu>
Message-ID: <575A7D39.8040205@gmail.com>

THAT WORKED!!!

Thank you Justin!

but... in this case... what is local-worker.bro for?. I understood that 
local-worker is what you are running in the workers... Actually the 
linux process still uses local-worker.bro (which is empty...):

  /usr/local/bro/bin/bro -i eth0 -U .status -p broctl -p broctl-live -p 
local -p worker-1 local.bro broctl base/frameworks/cluster 
*local-worker.bro* broctl/auto


Thank you Justin, I would never have figured out this was the problem...


On 09/06/16 18:58, Azoff, Justin S wrote:
>> On Jun 9, 2016, at 9:48 AM, Luis Martin Liras <martin.liras at gmail.com> wrote:
>> 2.- local-worker.bro:
>>
>> event bro_init() &priority=5
>> {
>> ...
>>        Log::create_stream(myprot::MY-PROT_LOG, [$columns=myprot_info, $path="myprot"]);
>>         Log::create_stream(myprot::MY-PROT_WARN, [$columns=myprot_info,$path="warnings"]);
>>
>>         ...
>>
>> }
>>
>>
>> 3.- local-manager.bro
>>
> ...
>
> Ah!  the files you are putting things in is your problem. Take everything you put in local-worker.bro and local-manager.bro and put it into a script and load that script from local.bro
>
> When you use local-worker directly and call create_stream there then the manager knows nothing about about those log streams.
>
> Additionally, if that is where you are defining the notice types, that will break notices as well because when the notice reaches the manager it also has no idea what to do with it.
>
> There aren't many things that people do that actually belong in the local-worker or local-manger scripts, everything should almost always  just go in local.bro.  things like create_stream and NOTICE automatically do the right thing depending on what node it is ran on.
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160610/dd974b21/attachment.html 

From elhijo at 0lim.net  Mon Jun 13 05:48:18 2016
From: elhijo at 0lim.net (David)
Date: Mon, 13 Jun 2016 14:48:18 +0200
Subject: [Bro] Bro clustered limit load
Message-ID: <575EAB92.2040101@0lim.net>

Hi,

I'd like to know if there is a way to select which script a worker is 
loading.
The goal is to limit the packets that needs to be analyzed.
On a dedicated interface I've mirrored traffic going to one of our 
server which has, along other protocols, tones of dns and nfs traffic, 
I'm only interested in dns traffic.
Nfs can be bandwidth consuming (up to 600mbps with capstats) so I'd like 
Bro to only analyses dns packets.
Can we tell Bro to only load dns inspector for a given interface/worker ?

I've also think of firewalling everything except udp/53 but i would have 
to give network interface an ip address....


Thanks,

David

From hosom at battelle.org  Mon Jun 13 07:08:55 2016
From: hosom at battelle.org (Hosom, Stephen M)
Date: Mon, 13 Jun 2016 14:08:55 +0000
Subject: [Bro] Bro clustered limit load
In-Reply-To: <575EAB92.2040101@0lim.net>
References: <575EAB92.2040101@0lim.net>
Message-ID: <E5A2071C6FFFF640B714C224139C9B3327D2E0A9@WP-MBX2B.milky-way.battelle.org>

David, 

You could apply a BPF in Bro.

https://www.bro.org/sphinx/scripts/base/frameworks/packet-filter/main.bro.html

The packets will still hit the interface, but Bro will only monitor the packets based on the BPF.

________________________________________
From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of David [elhijo at 0lim.net]
Sent: Monday, June 13, 2016 8:48 AM
To: bro at bro.org
Subject: [Bro] Bro clustered limit load

Hi,

I'd like to know if there is a way to select which script a worker is
loading.
The goal is to limit the packets that needs to be analyzed.
On a dedicated interface I've mirrored traffic going to one of our
server which has, along other protocols, tones of dns and nfs traffic,
I'm only interested in dns traffic.
Nfs can be bandwidth consuming (up to 600mbps with capstats) so I'd like
Bro to only analyses dns packets.
Can we tell Bro to only load dns inspector for a given interface/worker ?

I've also think of firewalling everything except udp/53 but i would have
to give network interface an ip address....


Thanks,

David
_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From jazoff at illinois.edu  Mon Jun 13 07:24:17 2016
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Mon, 13 Jun 2016 14:24:17 +0000
Subject: [Bro] Bro clustered limit load
In-Reply-To: <575EAB92.2040101@0lim.net>
References: <575EAB92.2040101@0lim.net>
Message-ID: <7930A1D9-855A-4A98-A31B-649CEA534E5D@illinois.edu>


> On Jun 13, 2016, at 7:48 AM, David <elhijo at 0lim.net> wrote:
> 
> Hi,
> 
> I'd like to know if there is a way to select which script a worker is 
> loading.

broctl starts each worker with an option that tells each worker to use its own name as a script prefix.  If you look at the running bro command you should see something like 

/usr/local/bro/bin/bro -i p1p1 -U .status -p broctl -p broctl-live -p local -p worker-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

where 'worker-1' is the name of the worker.

This means that if you create a script called worker-1.local.bro, it will be loaded for that worker only

-- 
- Justin Azoff




From elhijo at 0lim.net  Mon Jun 13 07:57:13 2016
From: elhijo at 0lim.net (David)
Date: Mon, 13 Jun 2016 16:57:13 +0200
Subject: [Bro] Bro clustered limit load
In-Reply-To: <7930A1D9-855A-4A98-A31B-649CEA534E5D@illinois.edu>
References: <575EAB92.2040101@0lim.net>
	<7930A1D9-855A-4A98-A31B-649CEA534E5D@illinois.edu>
Message-ID: <575EC9C9.4050600@0lim.net>

Thanks Justin, looks like exactly what I want.

David

On 06/13/2016 04:24 PM, Azoff, Justin S wrote:
>> On Jun 13, 2016, at 7:48 AM, David <elhijo at 0lim.net> wrote:
>>
>> Hi,
>>
>> I'd like to know if there is a way to select which script a worker is
>> loading.
> broctl starts each worker with an option that tells each worker to use its own name as a script prefix.  If you look at the running bro command you should see something like
>
> /usr/local/bro/bin/bro -i p1p1 -U .status -p broctl -p broctl-live -p local -p worker-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
>
> where 'worker-1' is the name of the worker.
>
> This means that if you create a script called worker-1.local.bro, it will be loaded for that worker only
>


From seth at icir.org  Mon Jun 13 07:57:44 2016
From: seth at icir.org (Seth Hall)
Date: Mon, 13 Jun 2016 10:57:44 -0400
Subject: [Bro] log streams in a bro cluster
In-Reply-To: <575A7D39.8040205@gmail.com>
References: <5757DA98.5010606@gmail.com>
	<E944C4EE-735F-464A-9753-B4CE8C19418F@illinois.edu>
	<57595E01.6050603@gmail.com>
	<C8D59376-7A90-4ADD-AC7D-4AA0A897B21C@illinois.edu>
	<575981B0.7020707@gmail.com>
	<A425758B-B0BB-4C96-AD90-626EB806DC11@illinois.edu>
	<575A7D39.8040205@gmail.com>
Message-ID: <46E2EB68-7224-47FB-8D02-9D8A2C7D4C56@icir.org>


> On Jun 10, 2016, at 4:41 AM, Luis Martin Liras <martin.liras at gmail.com> wrote:
> 
> but... in this case... what is local-worker.bro for?. I understood that local-worker is what you are running in the workers... Actually the linux process still uses local-worker.bro (which is empty...):]

We've considered getting rid of those extra files.  You have to understand the entire architecture too much to understand why they are there.  We ended up going in a direction where the distinctions between the types of processes are hidden from users because it can be too difficult to know when to use those different files.

The rule of thumb is to just put everything into local.bro or your own scripts that you load in local.bro.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From seth at icir.org  Mon Jun 13 08:44:45 2016
From: seth at icir.org (Seth Hall)
Date: Mon, 13 Jun 2016 11:44:45 -0400
Subject: [Bro] ElasticSearch plugin
Message-ID: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>

Is there anyone here relying on the elasticsearch writer plugin in the bro-plugins repository?  It doesn't appear to work with current versions of elasticsearch anymore and it has always had trouble at sites with high rates of logging.

If we don't get much of a response on this we will be deprecating and/or removing the elasticsearch writer.  There should be more reliable mechanisms available soon anyway by either writing to a Kafka server and then forwarding to ElasticSearch or writing files as JSON and the forwarding to ElasticSearch.

Thanks,
  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From d7om.ph at hotmail.com  Mon Jun 13 09:15:36 2016
From: d7om.ph at hotmail.com (=?utf-8?B?2ZBBQkRVTCBBTEVBTkFaSQ==?=)
Date: Mon, 13 Jun 2016 09:15:36 -0700
Subject: [Bro] ElasticSearch plugin
In-Reply-To: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
References: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
Message-ID: <DUB405-EAS158B52A1E328C8FA104E492A8530@phx.gbl>

I never used it with Bro, However, I am really interested too.


Sent from my iPhone

> On Jun 13, 2016, at 8:47 AM, Seth Hall <seth at icir.org> wrote:
> 
> Is there anyone here relying on the elasticsearch writer plugin in the bro-plugins repository?  It doesn't appear to work with current versions of elasticsearch anymore and it has always had trouble at sites with high rates of logging.
> 
> If we don't get much of a response on this we will be deprecating and/or removing the elasticsearch writer.  There should be more reliable mechanisms available soon anyway by either writing to a Kafka server and then forwarding to ElasticSearch or writing files as JSON and the forwarding to ElasticSearch.
> 
> Thanks,
>  .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From jlay at slave-tothe-box.net  Mon Jun 13 09:29:11 2016
From: jlay at slave-tothe-box.net (James Lay)
Date: Mon, 13 Jun 2016 10:29:11 -0600
Subject: [Bro] ElasticSearch plugin
In-Reply-To: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
References: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
Message-ID: <492a938924677a02cae00d07ea6d0587@localhost>

On 2016-06-13 09:44, Seth Hall wrote:
> Is there anyone here relying on the elasticsearch writer plugin in the
> bro-plugins repository?  It doesn't appear to work with current
> versions of elasticsearch anymore and it has always had trouble at
> sites with high rates of logging.
> 
> If we don't get much of a response on this we will be deprecating
> and/or removing the elasticsearch writer.  There should be more
> reliable mechanisms available soon anyway by either writing to a Kafka
> server and then forwarding to ElasticSearch or writing files as JSON
> and the forwarding to ElasticSearch.
> 
> Thanks,
>   .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

Not I...straight up using rsyslog to pipe to Logstash.

James

From shirkdog.bsd at gmail.com  Mon Jun 13 11:22:57 2016
From: shirkdog.bsd at gmail.com (Michael Shirk)
Date: Mon, 13 Jun 2016 14:22:57 -0400
Subject: [Bro] ElasticSearch plugin
In-Reply-To: <492a938924677a02cae00d07ea6d0587@localhost>
References: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
	<492a938924677a02cae00d07ea6d0587@localhost>
Message-ID: <CAL8PkUU=eDOC-6_EC+nbXao_oQpG_rhbt+W1TuUw0mA8_HVX+g@mail.gmail.com>

I would be interested in this working, as it does not work with later
versions of Elastic.

--
Michael Shirk
Daemon Security, Inc.
http://www.daemon-security.com
On Jun 13, 2016 12:43 PM, "James Lay" <jlay at slave-tothe-box.net> wrote:

> On 2016-06-13 09:44, Seth Hall wrote:
> > Is there anyone here relying on the elasticsearch writer plugin in the
> > bro-plugins repository?  It doesn't appear to work with current
> > versions of elasticsearch anymore and it has always had trouble at
> > sites with high rates of logging.
> >
> > If we don't get much of a response on this we will be deprecating
> > and/or removing the elasticsearch writer.  There should be more
> > reliable mechanisms available soon anyway by either writing to a Kafka
> > server and then forwarding to ElasticSearch or writing files as JSON
> > and the forwarding to ElasticSearch.
> >
> > Thanks,
> >   .Seth
> >
> > --
> > Seth Hall
> > International Computer Science Institute
> > (Bro) because everyone has a network
> > http://www.bro.org/
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> Not I...straight up using rsyslog to pipe to Logstash.
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160613/4b0937cd/attachment-0001.html 

From blackhole.em at gmail.com  Mon Jun 13 11:28:49 2016
From: blackhole.em at gmail.com (Joe Blow)
Date: Mon, 13 Jun 2016 14:28:49 -0400
Subject: [Bro] ElasticSearch plugin
In-Reply-To: <492a938924677a02cae00d07ea6d0587@localhost>
References: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
	<492a938924677a02cae00d07ea6d0587@localhost>
Message-ID: <CAMCSCoW0Ydu0BB8=C0zA1LBugA+sQDO+pLmaxxPQQ_OYf-wCgw@mail.gmail.com>

I use it a whole bunch, but it is quite clunky...

Part of me wishes bro would just write JSON to syslog, so that we could use
the native rsyslog queuing and output modules (much more widely supported).

Any chance that could be easily implemented?

Cheers,

JB

On Mon, Jun 13, 2016 at 12:29 PM, James Lay <jlay at slave-tothe-box.net>
wrote:

> On 2016-06-13 09:44, Seth Hall wrote:
> > Is there anyone here relying on the elasticsearch writer plugin in the
> > bro-plugins repository?  It doesn't appear to work with current
> > versions of elasticsearch anymore and it has always had trouble at
> > sites with high rates of logging.
> >
> > If we don't get much of a response on this we will be deprecating
> > and/or removing the elasticsearch writer.  There should be more
> > reliable mechanisms available soon anyway by either writing to a Kafka
> > server and then forwarding to ElasticSearch or writing files as JSON
> > and the forwarding to ElasticSearch.
> >
> > Thanks,
> >   .Seth
> >
> > --
> > Seth Hall
> > International Computer Science Institute
> > (Bro) because everyone has a network
> > http://www.bro.org/
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> Not I...straight up using rsyslog to pipe to Logstash.
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160613/1a1d5344/attachment.html 

From jazoff at illinois.edu  Mon Jun 13 11:31:47 2016
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Mon, 13 Jun 2016 18:31:47 +0000
Subject: [Bro] ElasticSearch plugin
In-Reply-To: <CAMCSCoW0Ydu0BB8=C0zA1LBugA+sQDO+pLmaxxPQQ_OYf-wCgw@mail.gmail.com>
References: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
	<492a938924677a02cae00d07ea6d0587@localhost>
	<CAMCSCoW0Ydu0BB8=C0zA1LBugA+sQDO+pLmaxxPQQ_OYf-wCgw@mail.gmail.com>
Message-ID: <2941C766-940C-4D72-95A4-866D81E490D4@illinois.edu>


> On Jun 13, 2016, at 1:28 PM, Joe Blow <blackhole.em at gmail.com> wrote:
> 
> I use it a whole bunch, but it is quite clunky...
> 
> Part of me wishes bro would just write JSON to syslog, so that we could use the native rsyslog queuing and output modules (much more widely supported).
> 
> Any chance that could be easily implemented?
> 
> Cheers,
> 
> JB

You can tell bro to write to the json logs as usual, and then use rsyslog with the imfile module.

-- 
- Justin Azoff



From blackhole.em at gmail.com  Mon Jun 13 11:35:04 2016
From: blackhole.em at gmail.com (Joe Blow)
Date: Mon, 13 Jun 2016 14:35:04 -0400
Subject: [Bro] ElasticSearch plugin
In-Reply-To: <2941C766-940C-4D72-95A4-866D81E490D4@illinois.edu>
References: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
	<492a938924677a02cae00d07ea6d0587@localhost>
	<CAMCSCoW0Ydu0BB8=C0zA1LBugA+sQDO+pLmaxxPQQ_OYf-wCgw@mail.gmail.com>
	<2941C766-940C-4D72-95A4-866D81E490D4@illinois.edu>
Message-ID: <CAMCSCoXFJgwJcBJHC4qHx-2GqPgMWWFwuafPhqV9MebPUaUa0g@mail.gmail.com>

I hate sucking IOPs out of my boxes if i can help it...  Is there no clean
way to write directly to rsyslog?  I can crank the allowable message size
up fairly large, and then either write directly to a local file, or simply
ship off box.

Writing to a file, only to immediately tail that file seems a bit clunky if
you ask me, but what do I know :).

Thoughts?

Cheers,

JB

On Mon, Jun 13, 2016 at 2:31 PM, Azoff, Justin S <jazoff at illinois.edu>
wrote:

>
> > On Jun 13, 2016, at 1:28 PM, Joe Blow <blackhole.em at gmail.com> wrote:
> >
> > I use it a whole bunch, but it is quite clunky...
> >
> > Part of me wishes bro would just write JSON to syslog, so that we could
> use the native rsyslog queuing and output modules (much more widely
> supported).
> >
> > Any chance that could be easily implemented?
> >
> > Cheers,
> >
> > JB
>
> You can tell bro to write to the json logs as usual, and then use rsyslog
> with the imfile module.
>
> --
> - Justin Azoff
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160613/61ab5564/attachment.html 

From gfaulkner.nsm at gmail.com  Mon Jun 13 11:45:08 2016
From: gfaulkner.nsm at gmail.com (Gary Faulkner)
Date: Mon, 13 Jun 2016 13:45:08 -0500
Subject: [Bro] ElasticSearch plugin
In-Reply-To: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
References: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
Message-ID: <902a97b6-fd4f-48e1-4b27-2aa5cebc6724@gmail.com>

I believe supporting dots in field names again is on the fix list for 
Elastic Stack v5 which is currently in development, so that part might 
at least get fixed on the Elastic end. Technically I believe that fix is 
a plus even for folks using another plugin such as Kafka as those folks 
still potentially had to do something to rewrite the field names. I 
can't speak for anything else that might be broken in the plugin.

Here is the reference to that bit in the Elastic Blog: 
https://www.elastic.co/blog/elasticsearch-5-0-0-alpha3-released

I could still see cases where someone could have a low volume Bro + 
local ES where it might not be desirable or necessary to run a bunch of 
other stuff in between Bro and ES, but maybe it just isn't too big a 
deal then to just let Logstash do the work of reading local log files 
assuming that one is OK with still writing normal Bro log output in 
addition to ES. An example might be if I wanted to do something akin to 
a stand-alone Security Onion node, but with Bro and Elastic instead of 
Bro and ELSA.

I'm not using that functionality currently and will probably look at 
something like Kafka, but I already have a log volume where the overhead 
of running something like Kafka probably makes sense.

~Gary


On 6/13/16 10:44 AM, Seth Hall wrote:
> Is there anyone here relying on the elasticsearch writer plugin in the bro-plugins repository?  It doesn't appear to work with current versions of elasticsearch anymore and it has always had trouble at sites with high rates of logging.
>
> If we don't get much of a response on this we will be deprecating and/or removing the elasticsearch writer.  There should be more reliable mechanisms available soon anyway by either writing to a Kafka server and then forwarding to ElasticSearch or writing files as JSON and the forwarding to ElasticSearch.
>
> Thanks,
>    .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From landy-bible at utulsa.edu  Mon Jun 13 11:46:49 2016
From: landy-bible at utulsa.edu (Landy Bible)
Date: Mon, 13 Jun 2016 13:46:49 -0500
Subject: [Bro] ElasticSearch plugin
In-Reply-To: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
References: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
Message-ID: <CADtxXLnsSRwCKOFCfYeWNVyuTVEO1GEyvkm2-M-nW+OpaebcdQ@mail.gmail.com>

I have bro output json, then use logstash to ship to redis where another
set of logstash servers pull it out to process and insert into
elasticsearch. One of the filters is to remove the dots so I can upgrade to
elasticsearch 2. I plan to replace the first logstash with filebeats.

-Landy
On Jun 13, 2016 10:46 AM, "Seth Hall" <seth at icir.org> wrote:

> Is there anyone here relying on the elasticsearch writer plugin in the
> bro-plugins repository?  It doesn't appear to work with current versions of
> elasticsearch anymore and it has always had trouble at sites with high
> rates of logging.
>
> If we don't get much of a response on this we will be deprecating and/or
> removing the elasticsearch writer.  There should be more reliable
> mechanisms available soon anyway by either writing to a Kafka server and
> then forwarding to ElasticSearch or writing files as JSON and the
> forwarding to ElasticSearch.
>
> Thanks,
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160613/14cdd97e/attachment.html 

From blackhole.em at gmail.com  Mon Jun 13 12:21:38 2016
From: blackhole.em at gmail.com (Joe Blow)
Date: Mon, 13 Jun 2016 15:21:38 -0400
Subject: [Bro] ElasticSearch plugin
In-Reply-To: <902a97b6-fd4f-48e1-4b27-2aa5cebc6724@gmail.com>
References: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
	<902a97b6-fd4f-48e1-4b27-2aa5cebc6724@gmail.com>
Message-ID: <CAMCSCoXv4SZQaNpct-bMMFxgB+NdxCCtTB3FGG5F+j9XHSMX7w@mail.gmail.com>

What if you *hate* using logstash because liblognormalize is much, much
faster than a regex engine, like logstash?  I'd much prefer to simply get
the data into rsyslog (which should be trivial), then use rsyslog queueing
and batching which is much more flexible, IMO.  Use RELP to reliably
forward, send to kafka, do whatever, but once you've got it into rsyslog
you've got pretty solid queueing which you can use, along with a whole host
of output modules.

I'd just love to see a better pure syslog integration, without having to
write to disks, then read from the disks.  It starts potentially cause
problems when you're capturing 10Gb/s+.

Cheers,

JB

On Mon, Jun 13, 2016 at 2:45 PM, Gary Faulkner <gfaulkner.nsm at gmail.com>
wrote:

> I believe supporting dots in field names again is on the fix list for
> Elastic Stack v5 which is currently in development, so that part might
> at least get fixed on the Elastic end. Technically I believe that fix is
> a plus even for folks using another plugin such as Kafka as those folks
> still potentially had to do something to rewrite the field names. I
> can't speak for anything else that might be broken in the plugin.
>
> Here is the reference to that bit in the Elastic Blog:
> https://www.elastic.co/blog/elasticsearch-5-0-0-alpha3-released
>
> I could still see cases where someone could have a low volume Bro +
> local ES where it might not be desirable or necessary to run a bunch of
> other stuff in between Bro and ES, but maybe it just isn't too big a
> deal then to just let Logstash do the work of reading local log files
> assuming that one is OK with still writing normal Bro log output in
> addition to ES. An example might be if I wanted to do something akin to
> a stand-alone Security Onion node, but with Bro and Elastic instead of
> Bro and ELSA.
>
> I'm not using that functionality currently and will probably look at
> something like Kafka, but I already have a log volume where the overhead
> of running something like Kafka probably makes sense.
>
> ~Gary
>
>
> On 6/13/16 10:44 AM, Seth Hall wrote:
> > Is there anyone here relying on the elasticsearch writer plugin in the
> bro-plugins repository?  It doesn't appear to work with current versions of
> elasticsearch anymore and it has always had trouble at sites with high
> rates of logging.
> >
> > If we don't get much of a response on this we will be deprecating and/or
> removing the elasticsearch writer.  There should be more reliable
> mechanisms available soon anyway by either writing to a Kafka server and
> then forwarding to ElasticSearch or writing files as JSON and the
> forwarding to ElasticSearch.
> >
> > Thanks,
> >    .Seth
> >
> > --
> > Seth Hall
> > International Computer Science Institute
> > (Bro) because everyone has a network
> > http://www.bro.org/
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160613/e0948677/attachment.html 

From gfaulkner.nsm at gmail.com  Mon Jun 13 13:44:07 2016
From: gfaulkner.nsm at gmail.com (Gary Faulkner)
Date: Mon, 13 Jun 2016 15:44:07 -0500
Subject: [Bro] ElasticSearch plugin
In-Reply-To: <CAMCSCoXv4SZQaNpct-bMMFxgB+NdxCCtTB3FGG5F+j9XHSMX7w@mail.gmail.com>
References: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
	<902a97b6-fd4f-48e1-4b27-2aa5cebc6724@gmail.com>
	<CAMCSCoXv4SZQaNpct-bMMFxgB+NdxCCtTB3FGG5F+j9XHSMX7w@mail.gmail.com>
Message-ID: <d98a0fb0-e800-ea07-c976-6aea882e8984@gmail.com>

I've actually had instances where it "appeared" that Bro couldn't 
process conn events fast enough and Bro "appeared" to drop events 
further down the pipleline or outright crash under certain high stress 
situations. I suspected it was related to not being able to write the 
conn.log events to disk fast enough as I'd see a huge spike in calls to 
write the conn.log file and subsequent dips in the counters for all of 
the protocol based logs. I was writing the counters from the workers to 
an external stats server whenever a script called its specific write log 
event, so I could see the spikes and correlate them to drops in other 
areas, but I could never definitively prove that disk-IO for log writes 
were the bottle-neck as opposed to some other processing task as some of 
the system monitors on the master side would drop stats when it was too 
busy. Ultimately I largely addressed the causes of the processing spikes 
(DDOS attacks etc) upstream from Bro, but I could see the potential for 
wanting to directly forward events to an external location and not write 
them locally at all instead of trying to scale a single Bro master to 
handle writing hundreds of thousands or more events per second to disk.

~Gary


On 6/13/16 2:21 PM, Joe Blow wrote:
> What if you *hate* using logstash because liblognormalize is much, much
> faster than a regex engine, like logstash?  I'd much prefer to simply get
> the data into rsyslog (which should be trivial), then use rsyslog queueing
> and batching which is much more flexible, IMO.  Use RELP to reliably
> forward, send to kafka, do whatever, but once you've got it into rsyslog
> you've got pretty solid queueing which you can use, along with a whole host
> of output modules.
>
> I'd just love to see a better pure syslog integration, without having to
> write to disks, then read from the disks.  It starts potentially cause
> problems when you're capturing 10Gb/s+.
>
> Cheers,
>
> JB
>
> On Mon, Jun 13, 2016 at 2:45 PM, Gary Faulkner <gfaulkner.nsm at gmail.com>
> wrote:
>
>> I believe supporting dots in field names again is on the fix list for
>> Elastic Stack v5 which is currently in development, so that part might
>> at least get fixed on the Elastic end. Technically I believe that fix is
>> a plus even for folks using another plugin such as Kafka as those folks
>> still potentially had to do something to rewrite the field names. I
>> can't speak for anything else that might be broken in the plugin.
>>
>> Here is the reference to that bit in the Elastic Blog:
>> https://www.elastic.co/blog/elasticsearch-5-0-0-alpha3-released
>>
>> I could still see cases where someone could have a low volume Bro +
>> local ES where it might not be desirable or necessary to run a bunch of
>> other stuff in between Bro and ES, but maybe it just isn't too big a
>> deal then to just let Logstash do the work of reading local log files
>> assuming that one is OK with still writing normal Bro log output in
>> addition to ES. An example might be if I wanted to do something akin to
>> a stand-alone Security Onion node, but with Bro and Elastic instead of
>> Bro and ELSA.
>>
>> I'm not using that functionality currently and will probably look at
>> something like Kafka, but I already have a log volume where the overhead
>> of running something like Kafka probably makes sense.
>>
>> ~Gary
>>
>>
>> On 6/13/16 10:44 AM, Seth Hall wrote:
>>> Is there anyone here relying on the elasticsearch writer plugin in the
>> bro-plugins repository?  It doesn't appear to work with current versions of
>> elasticsearch anymore and it has always had trouble at sites with high
>> rates of logging.
>>> If we don't get much of a response on this we will be deprecating and/or
>> removing the elasticsearch writer.  There should be more reliable
>> mechanisms available soon anyway by either writing to a Kafka server and
>> then forwarding to ElasticSearch or writing files as JSON and the
>> forwarding to ElasticSearch.
>>> Thanks,
>>>     .Seth
>>>
>>> --
>>> Seth Hall
>>> International Computer Science Institute
>>> (Bro) because everyone has a network
>>> http://www.bro.org/
>>>
>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>


From vladg at illinois.edu  Tue Jun 14 07:04:39 2016
From: vladg at illinois.edu (Vlad Grigorescu)
Date: Tue, 14 Jun 2016 07:04:39 -0700
Subject: [Bro] ElasticSearch plugin
In-Reply-To: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
References: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
Message-ID: <m0d1njsw7c.fsf@vladg.net>

Seth Hall <seth at icir.org> writes:

> Is there anyone here relying on the elasticsearch writer plugin in the
> bro-plugins repository?  It doesn't appear to work with current
> versions of elasticsearch anymore and it has always had trouble at
> sites with high rates of logging.

I think we should be a bit cautious here. Let's not forget that this is
really an ElasticSearch and NSQ writer. I've had very good success with
NSQ at high rate, so I don't really see much value to the second
argument.

> If we don't get much of a response on this we will be deprecating
> and/or removing the elasticsearch writer. There should be more
> reliable mechanisms available soon anyway by either writing to a Kafka
> server and then forwarding to ElasticSearch or writing files as JSON
> and the forwarding to ElasticSearch.

Do we know what the specific problems are with new versions of
ElasticSearch? Since the writer is just writing out JSON, either it's
doing something that's not compatible (which I'd think would be an easy
fix), or there's an issue with the JSON writer, which would affect
people regardless of how they get their logs to ElasticSearch.

The only concrete issue I've heard of is 'no periods in field names',
which I believe there are fixes for here:

https://github.com/danielguerra69/bro-debian-elasticsearch/tree/master/bro-patch

I think the better solution would simply be to make the record separator
redef-able in the formatter. I can *maybe* see the argument for using
'.' instead of '$' in the ASCII logs, but since the other separators are
user-definable, I think this one should be as well.

As far as which is more reliable, I think that should be up to the users
to decide. Personally, I'd rather use NSQ for a number of reasons
(easier to setup and manage, latency is over an order of magnitude less
compared with Kafka, etc.), and there are issues with JSON output to the
disk as well (unnecessary IOPs as someone mentioned).

This is already out of the Bro source code; I see more benefits than
downsides to leaving it in the bro-plugins repo.

I do agree that a RELP writer would be a great addition, and then we
could just use their great collection of output modules:

http://www.rsyslog.com/doc/v8-stable/configuration/modules/idx_output.html

  --Vlad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 800 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160614/de5d50e0/attachment.bin 

From blackhole.em at gmail.com  Tue Jun 14 07:18:36 2016
From: blackhole.em at gmail.com (Joe Blow)
Date: Tue, 14 Jun 2016 10:18:36 -0400
Subject: [Bro] ElasticSearch plugin
In-Reply-To: <m0d1njsw7c.fsf@vladg.net>
References: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
	<m0d1njsw7c.fsf@vladg.net>
Message-ID: <CAMCSCoX1H1kHNTh8+YZhyHJ1dphiHf84O2J-KFjsghR6CYfqPw@mail.gmail.com>

So it's settled then!!  When will the RELP writer be done?!? :)

Cheers,

JB

On Tue, Jun 14, 2016 at 10:04 AM, Vlad Grigorescu <vladg at illinois.edu>
wrote:

> Seth Hall <seth at icir.org> writes:
>
> > Is there anyone here relying on the elasticsearch writer plugin in the
> > bro-plugins repository?  It doesn't appear to work with current
> > versions of elasticsearch anymore and it has always had trouble at
> > sites with high rates of logging.
>
> I think we should be a bit cautious here. Let's not forget that this is
> really an ElasticSearch and NSQ writer. I've had very good success with
> NSQ at high rate, so I don't really see much value to the second
> argument.
>
> > If we don't get much of a response on this we will be deprecating
> > and/or removing the elasticsearch writer. There should be more
> > reliable mechanisms available soon anyway by either writing to a Kafka
> > server and then forwarding to ElasticSearch or writing files as JSON
> > and the forwarding to ElasticSearch.
>
> Do we know what the specific problems are with new versions of
> ElasticSearch? Since the writer is just writing out JSON, either it's
> doing something that's not compatible (which I'd think would be an easy
> fix), or there's an issue with the JSON writer, which would affect
> people regardless of how they get their logs to ElasticSearch.
>
> The only concrete issue I've heard of is 'no periods in field names',
> which I believe there are fixes for here:
>
>
> https://github.com/danielguerra69/bro-debian-elasticsearch/tree/master/bro-patch
>
> I think the better solution would simply be to make the record separator
> redef-able in the formatter. I can *maybe* see the argument for using
> '.' instead of '$' in the ASCII logs, but since the other separators are
> user-definable, I think this one should be as well.
>
> As far as which is more reliable, I think that should be up to the users
> to decide. Personally, I'd rather use NSQ for a number of reasons
> (easier to setup and manage, latency is over an order of magnitude less
> compared with Kafka, etc.), and there are issues with JSON output to the
> disk as well (unnecessary IOPs as someone mentioned).
>
> This is already out of the Bro source code; I see more benefits than
> downsides to leaving it in the bro-plugins repo.
>
> I do agree that a RELP writer would be a great addition, and then we
> could just use their great collection of output modules:
>
> http://www.rsyslog.com/doc/v8-stable/configuration/modules/idx_output.html
>
>   --Vlad
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160614/edc689bb/attachment-0001.html 

From jazoff at illinois.edu  Tue Jun 14 07:21:20 2016
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Tue, 14 Jun 2016 14:21:20 +0000
Subject: [Bro] ElasticSearch plugin
In-Reply-To: <m0d1njsw7c.fsf@vladg.net>
References: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
	<m0d1njsw7c.fsf@vladg.net>
Message-ID: <B53FA4AF-67AB-4715-9B08-ED5D220DA8A0@illinois.edu>


> On Jun 14, 2016, at 9:04 AM, Vlad Grigorescu <vladg at illinois.edu> wrote:
> 
> I think the better solution would simply be to make the record separator
> redef-able in the formatter. I can *maybe* see the argument for using
> '.' instead of '$' in the ASCII logs, but since the other separators are
> user-definable, I think this one should be as well.

I know we talked about this at one point, I think the real fix is to log nested records natively in json.

The ascii writer needs to expand nested fields, but the json writer doesn't, so it can natively log a conn record as

{id: {orig_h: "1.2.3.4", orig_h: 123, resp_h: "5.6.7.8", resp_p: 456}, ... }


-- 
- Justin Azoff



From landy-bible at utulsa.edu  Tue Jun 14 09:03:36 2016
From: landy-bible at utulsa.edu (Landy Bible)
Date: Tue, 14 Jun 2016 11:03:36 -0500
Subject: [Bro] ElasticSearch plugin
In-Reply-To: <B53FA4AF-67AB-4715-9B08-ED5D220DA8A0@illinois.edu>
References: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
	<m0d1njsw7c.fsf@vladg.net>
	<B53FA4AF-67AB-4715-9B08-ED5D220DA8A0@illinois.edu>
Message-ID: <CADtxXLnKCLb2cBMYe3ArJ+ZJN7+CaWnKM=07fSXJxA3t=t_h7Q@mail.gmail.com>

For what it's worth, using the de_dot filter in logstash with the following
config converts the fields to be nested, and didn't even require any
changes to any of my kibana queries or dashboards. Everything just worked.
ElasticSearch is happy and I can upgrade to v2 now and nothing changed from
the user's point of view. All I did was tack this on the end of my filter
config file on my logstash servers.

filter {
  de_dot {
    nested => true
  }
}

Of course, I wouldn't complain about bro just nesting correctly in JSON. :)

-Landy

Landy Bible
Information Security Analyst
The University of Tulsa

On Tue, Jun 14, 2016 at 9:21 AM, Azoff, Justin S <jazoff at illinois.edu>
wrote:

>
> > On Jun 14, 2016, at 9:04 AM, Vlad Grigorescu <vladg at illinois.edu> wrote:
> >
> > I think the better solution would simply be to make the record separator
> > redef-able in the formatter. I can *maybe* see the argument for using
> > '.' instead of '$' in the ASCII logs, but since the other separators are
> > user-definable, I think this one should be as well.
>
> I know we talked about this at one point, I think the real fix is to log
> nested records natively in json.
>
> The ascii writer needs to expand nested fields, but the json writer
> doesn't, so it can natively log a conn record as
>
> {id: {orig_h: "1.2.3.4", orig_h: 123, resp_h: "5.6.7.8", resp_p: 456}, ...
> }
>
>
> --
> - Justin Azoff
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160614/a5ad10fd/attachment.html 

From seth at icir.org  Tue Jun 14 09:30:26 2016
From: seth at icir.org (Seth Hall)
Date: Tue, 14 Jun 2016 12:30:26 -0400
Subject: [Bro] ElasticSearch plugin
In-Reply-To: <m0d1njsw7c.fsf@vladg.net>
References: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
	<m0d1njsw7c.fsf@vladg.net>
Message-ID: <4B17CDD3-CEDD-4D03-A126-990057779497@icir.org>


> On Jun 14, 2016, at 10:04 AM, Vlad Grigorescu <vladg at illinois.edu> wrote:
> 
> I think we should be a bit cautious here. Let's not forget that this is
> really an ElasticSearch and NSQ writer. I've had very good success with
> NSQ at high rate, so I don't really see much value to the second
> argument.

Are you proposing that you'll take over responsibility for the module?

I think it would make sense to have a separate NSQ module too if you find value in that.  That way if/when ES or NSQ specific tweaks (or other HTTP-based outputs) come into play we aren't creating a mess of various configuration options in a single module.

> I think the better solution would simply be to make the record separator
> redef-able in the formatter. I can *maybe* see the argument for using
> '.' instead of '$' in the ASCII logs, but since the other separators are
> user-definable, I think this one should be as well.

This already exists in topic/seth/log-framework-ext and hopefully will be getting merged soon along with some other logging framework changes I did recently.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160614/af1a9775/attachment.bin 

From seth at icir.org  Tue Jun 14 19:14:59 2016
From: seth at icir.org (Seth Hall)
Date: Tue, 14 Jun 2016 22:14:59 -0400
Subject: [Bro] ElasticSearch plugin
In-Reply-To: <CADtxXLnKCLb2cBMYe3ArJ+ZJN7+CaWnKM=07fSXJxA3t=t_h7Q@mail.gmail.com>
References: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
	<m0d1njsw7c.fsf@vladg.net>
	<B53FA4AF-67AB-4715-9B08-ED5D220DA8A0@illinois.edu>
	<CADtxXLnKCLb2cBMYe3ArJ+ZJN7+CaWnKM=07fSXJxA3t=t_h7Q@mail.gmail.com>
Message-ID: <260CE837-39DF-4B62-9B09-92D80E453DF2@icir.org>


> On Jun 14, 2016, at 12:03 PM, Landy Bible <landy-bible at utulsa.edu> wrote:
> 
> For what it's worth, using the de_dot filter in logstash

Hah!  Interesting.

I wanted to briefly thank everyone that has participated in this thread so far.  It's really worthwhile to hear where people are struggling and see how everyone has addressed things for their own situation.  We are still working on making it easier to do the sort of integration that everyone is working toward and should hopefully be addressing some of the pain points in the 2.5 release.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From daniel.guerra69 at gmail.com  Wed Jun 15 16:24:43 2016
From: daniel.guerra69 at gmail.com (Daniel Guerra)
Date: Thu, 16 Jun 2016 01:24:43 +0200
Subject: [Bro] ElasticSearch plugin
In-Reply-To: <260CE837-39DF-4B62-9B09-92D80E453DF2@icir.org>
References: <1DC2414E-171A-4F77-9ACE-ED8EA0D0EE9C@icir.org>
	<m0d1njsw7c.fsf@vladg.net>
	<B53FA4AF-67AB-4715-9B08-ED5D220DA8A0@illinois.edu>
	<CADtxXLnKCLb2cBMYe3ArJ+ZJN7+CaWnKM=07fSXJxA3t=t_h7Q@mail.gmail.com>
	<260CE837-39DF-4B62-9B09-92D80E453DF2@icir.org>
Message-ID: <5932252D-15B2-41F9-BA28-E6D5DCB520D7@gmail.com>

Hi All,

I have been playing with elastic for a while. It works well and
besides the dot there are a few script changes needed to
avoid name/type confusion. A few have been solved but
I use these changes in my docker image on this subject.
Mapping is also very important to make things work. After
this you are ready to dump. For the kibana config I used
 elasticsearchdump (a alpine elasticdump). I preconfigured
kibana with searches, visualisations and dashboards.
In the ideal world, I would write to kafka combined with
an elastic-river for kafka. Graylog is implented like this.
But compiling the kafka plugin ends in complains, it needs
more time and reading installing etc. TODO ... 
Currently I?m quite happy with my elastic combination,
it is way faster when there are no errors, and elastic does
a lot with the current git. Elastic is memory hungry and prefers
to run on 3 nodes.

Regards,

Daniel

For the details on docker check this (I had to split them because
of dockerhub compile time) .
#docker-compose
https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/docker-compose.yml

#docker image (check develop for your source experiments)
https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/

#preperations
https://github.com/danielguerra69/debian-bro-develop

#compiling bro
https://github.com/danielguerra69/bro-debian-elasticsearch


>>>>>>>>bro script changes<<<<<

RUN sed -i "s/version:     count           \&log/socks_version:     count           \&log/g" /usr/local/bro/share/bro/base/protocols/socks/main.bro
RUN sed -i "s/\$version=/\$socks_version=/g" /usr/local/bro/share/bro/base/protocols/socks/main.bro
RUN sed -i "s/version:          string \&log/ssl_version:     string \&log/g" /usr/local/bro/share/bro/base/protocols/ssl/main.bro
RUN sed -i "s/\$version=/\$ssl_version=/g" /usr/local/bro/share/bro/base/protocols/ssl/main.bro
RUN sed -i "s/version:         count        \&log/ssh_version:         count        \&log/g" /usr/local/bro/share/bro/base/protocols/ssh/main.bro
RUN sed -i "s/\$version =/\$ssh_version =/g" /usr/local/bro/share/bro/base/protocols/ssh/main.bro
RUN sed -i "s/version: string \&log/snmp_version: string \&log/g" /usr/local/bro/share/bro/base/protocols/snmp/main.bro
RUN sed -i "s/\$version=/\$snmp_version=/g" /usr/local/bro/share/bro/base/protocols/snmp/main.bro


>>>>>>> mapping script <<<<<<< 

#!/bin/bash
until curl -XGET elasticsearch:9200/; do
  >&2 echo "Elasticsearch is unavailable - sleeping"
  sleep 5
done

>&2 echo "Elasticsearch is up - executing command"
curl -XPUT elasticsearch:9200/_template/fixstrings_bro -d '{
  "template": "bro-*",
    "index": {
      "number_of_shards": 7,
      "number_of_replicas": 1
    },
    "mappings" : {
      "http" : {
        "properties" : {
          "status_msg" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "user_agent" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "uri" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      },
        "conn" : {
          "properties" : {
            "orig_location" : {
              "type" : "geo_point"
            },
            "resp_location" : {
              "type" : "geo_point"
            }
          }
      },
      "files" : {
        "properties" : {
          "mime_type" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      },
      "location": {
        "properties" : {
          "ext_location" : {
            "type" : "geo_point"
          }
        }
      },
      "notice" : {
        "properties" : {
          "note" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      },
      "ssl" : {
        "properties" : {
          "validation_status" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "server_name" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      },
      "dns" : {
        "properties" : {
          "answers" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "query" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      },
      "intel" : {
        "properties" : {
          "sources" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "seen_indicator_type" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "seen_where" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      },
      "weird" : {
        "properties" : {
          "name" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "query" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      }
    }
  }'



> On 15 Jun 2016, at 04:14, Seth Hall <seth at icir.org> wrote:
> 
> 
>> On Jun 14, 2016, at 12:03 PM, Landy Bible <landy-bible at utulsa.edu> wrote:
>> 
>> For what it's worth, using the de_dot filter in logstash
> 
> Hah!  Interesting.
> 
> I wanted to briefly thank everyone that has participated in this thread so far.  It's really worthwhile to hear where people are struggling and see how everyone has addressed things for their own situation.  We are still working on making it easier to do the sort of integration that everyone is working toward and should hopefully be addressing some of the pain points in the 2.5 release.
> 
>  .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From dnj0496 at gmail.com  Fri Jun 17 19:12:05 2016
From: dnj0496 at gmail.com (Dk Jack)
Date: Fri, 17 Jun 2016 19:12:05 -0700
Subject: [Bro] memory q.
Message-ID: <CAGATen6Xp3pa_GDFrf=ML2XK1Gy5Jw5Fe6vuqu+QxDBOtSn_TQ@mail.gmail.com>

Hi,
While performing a longevity test on bro, I see bro memory consumption
going up. I ran valgrind on a large pcap. After analyzing the valgrind
logs, I came across some instance where I think bro is leaking. I've
enclosed my analysis below. I'd appreciate if someone more familiar in
these areas comments on analysis i.e. if my suspicions are correct or not.
Thanks.

Dk.

Bro Version: 2.4.1

1) PersistenceSerializer::CheckTimestamp (PersistenceSerializer.cc:102)

   Valgrind Stack:
   ---------------
   8 bytes in 1 blocks are definitely lost in loss record 32 of 6,072
   at 0x4C2AB80: malloc
   by 0x5DA094: PersistenceSerializer::CheckTimestamp(char const*)
(PersistenceSerializer.cc:102)
   by 0x5DA10D: PersistenceSerializer::CheckForFile(UnserialInfo*, char
const*, bool) (PersistenceSerializer.cc:121)
   by 0x5DA2AA: PersistenceSerializer::ReadAll(bool, bool)
(PersistenceSerializer.cc:157)
   by 0x52F857: main (main.cc:1068)

   Comments:
   ---------
    - CheckTimestamp function allocats memory for 'time_t'. This is then
inserted into
      files dictionary. This memory never seems to be freed.
    - The files dictionary is of type PDict i.e Dictionary. The Dictionary
destructor seems
      to be deleting/freeing the memory for key and values. However, this
requires the
      Dictionary user to set the delete_function which will be called to
free the value pointer.
    - Since the delete_function is never set (using SetDeleteFunc), the
time_t structures are
      never freed.

2) NFA_State::EpsilonClosure() (NFA.cc:82)

   Valgrind Stack:
   ---------------
   24 bytes in 1 blocks are possibly lost in loss record 582 of 6,072
   at 0x4C2B0E0: operator new(unsigned long)
   by 0x5CEEC4: NFA_State::EpsilonClosure() (NFA.cc:82)
   by 0x5CF955: epsilon_closure(NFA_StatePList*) (NFA.cc:324)
   by 0x56B939: DFA_State::ComputeXtion(int, DFA_Machine*) (DFA.cc:93)
   by 0x5DDB59: Xtion (DFA.h:160)
   by 0x5DDB59: RE_Match_State::Match(unsigned char const*, int, bool,
bool, bool) (RE.cc:318)

   Comments:
   ---------
    - In function EpsilonClosure, NFA_state_list object is instantiated.
This object never
      seems to be freed.

3) NFA_State::DeepCopy() (NFA.cc:58)

   Valgrind Stack:
   ---------------
   96 bytes in 1 blocks are possibly lost in loss record 3,374 of 6,072
   at 0x4C2B0E0: operator new(unsigned long)
   by 0x5CEDB0: NFA_State::DeepCopy() (NFA.cc:58)
   by 0x5CF305: NFA_Machine::DuplicateMachine() (NFA.cc:210)
   by 0x5CF63B: NFA_Machine::MakeRepl(int, int) (NFA.cc:252)
   by 0x53B648: RE_parse() (re-parse.y:76)
   by 0x5DD449: Specific_RE_Matcher::CompileSet(charPList const&,
ptr_compat_intList const&) (RE.cc:142)

   Comments:
   ---------
    - New NFA_State allocate in DeepCopy function is not freed.
    - The pointer is saved in 'mark' member variable. However, in
ClearMarks function
      the member is clear is simply cleared without freeing the pointer.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160617/959fbfb2/attachment.html 

From robin at icir.org  Sat Jun 18 13:16:41 2016
From: robin at icir.org (Robin Sommer)
Date: Sat, 18 Jun 2016 13:16:41 -0700
Subject: [Bro] Spicy: Looking for contributors
Message-ID: <20160618201641.GE3268@icir.org>

You may have heard about Spicy, a next-generation parser generator for
Bro and beyond; see http://www.icir.org/hilti for more. Spicy is
currently still in prototype state, and you might have noticed that
progress has stalled recently. The reason is simple: the research
project that made the work possible so far has concluded, which
unfortunately leaves only limited resources right now to transition
the code to production state.

Recently, however, a few people have indicated to me that they would
be willing to contribute time to help move the project forward. So I
thought I'd see if we can get a group of volunteers together to work
on the code towards an initial production Spicy release. If we get
some folks together, I can offer guidance for understanding codebase
and work that needs to be done, and also perform code review and
upstream merging.

As a starting point, I've compiled an initial roadmap of items that
need work: https://github.com/rsmmr/hilti/wiki/Roadmap. Not all of
that is crucial; I've marked the most important pieces that I would
deem necessary for an initial release (assuming we'd put the focus on
usability and stability at first, rather than performance).

So, if you are interested in contributing to Spicy, raise you hand.
Please send any replies to the HILTI list or, if you prefer, to me
personally.

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From obdnanr at gmail.com  Mon Jun 20 08:13:10 2016
From: obdnanr at gmail.com (Obndnar smith)
Date: Mon, 20 Jun 2016 15:13:10 +0000
Subject: [Bro] non-void function returns without a value
Message-ID: <CALCRDTQt0s0_O+ooq0767mrceMK7-VxG+nk+0q9uDJ771emh0w@mail.gmail.com>

We're using the following script to white list DNS requests.  In the
reporter.log we are getting the following error multiple times every
millisecond.  Anyone know what we're doing wrong?  Can bad scripts cause
the logs to fail to rotate correctly?

Jun 20 10:58:05 Reporter::ERROR field value missing [FILTER::rec$query]
/usr/local/bro/spool/installed-scripts-do-not-touch/site/mysite/dns-filter.bro,
line 13
Jun 20 10:58:05 Reporter::WARNING       non-void function returns without a
value: FILTER::filter_pred  (empty)

module FILTER;

function filter_pred (rec: DNS::Info) : bool
{
   if("microsoft.com" in rec$query)
      return F;
   return T;
}

event bro_init()
{
   Log::remove_default_filter(DNS::LOG);
   Log::add_filter(DNS::LOG, [$name="dns-filter",
                              $path="dns",
                              $exclude=set("trans_id", "qclass",
"qclass_name", "qtype", "rcode", "rcode_name", "QR", "AA", "TC", "RD",
"RA", "Z", "TLLs", "rejected"),
                              $pred=filter_pred]);
}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160620/d5ddbab3/attachment.html 

From johanna at icir.org  Mon Jun 20 09:01:48 2016
From: johanna at icir.org (Johanna Amann)
Date: Mon, 20 Jun 2016 09:01:48 -0700
Subject: [Bro] non-void function returns without a value
In-Reply-To: <CALCRDTQt0s0_O+ooq0767mrceMK7-VxG+nk+0q9uDJ771emh0w@mail.gmail.com>
References: <CALCRDTQt0s0_O+ooq0767mrceMK7-VxG+nk+0q9uDJ771emh0w@mail.gmail.com>
Message-ID: <20160620160148.GA27727@wifi241.sys.ICSI.Berkeley.EDU>

You need to check if rec$query is defined before accessing it; otherwhise
the function will just abort if query is not set without returning
anything.

So

if (rec?$query && "microsoft.com" in rec$query)
  return F;

return T;

as a function body instead of what you are using should probably work.

Johanna


On Mon, Jun 20, 2016 at 03:13:10PM +0000, Obndnar smith wrote:
> We're using the following script to white list DNS requests.  In the
> reporter.log we are getting the following error multiple times every
> millisecond.  Anyone know what we're doing wrong?  Can bad scripts cause
> the logs to fail to rotate correctly?
> 
> Jun 20 10:58:05 Reporter::ERROR field value missing [FILTER::rec$query]
> /usr/local/bro/spool/installed-scripts-do-not-touch/site/mysite/dns-filter.bro,
> line 13
> Jun 20 10:58:05 Reporter::WARNING       non-void function returns without a
> value: FILTER::filter_pred  (empty)
> 
> module FILTER;
> 
> function filter_pred (rec: DNS::Info) : bool
> {
>    if("microsoft.com" in rec$query)
>       return F;
>    return T;
> }
> 
> event bro_init()
> {
>    Log::remove_default_filter(DNS::LOG);
>    Log::add_filter(DNS::LOG, [$name="dns-filter",
>                               $path="dns",
>                               $exclude=set("trans_id", "qclass",
> "qclass_name", "qtype", "rcode", "rcode_name", "QR", "AA", "TC", "RD",
> "RA", "Z", "TLLs", "rejected"),
>                               $pred=filter_pred]);
> }

> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From johanna at icir.org  Mon Jun 20 11:36:26 2016
From: johanna at icir.org (Johanna Amann)
Date: Mon, 20 Jun 2016 11:36:26 -0700
Subject: [Bro] memory q.
In-Reply-To: <CAGATen6Xp3pa_GDFrf=ML2XK1Gy5Jw5Fe6vuqu+QxDBOtSn_TQ@mail.gmail.com>
References: <CAGATen6Xp3pa_GDFrf=ML2XK1Gy5Jw5Fe6vuqu+QxDBOtSn_TQ@mail.gmail.com>
Message-ID: <20160620183622.GA32905@wifi241.sys.ICSI.Berkeley.EDU>

Hello D,

thank you for your analysis. I took a look at the leaks you found and
think the first two are indeed memory leaks; this is fixed in
a156f2b4e63d64b3cbe888e8cb88f1c05d3e7b86 (see
https://bro-tracker.atlassian.net/browse/BIT-1633 to track the merge
process).

I think the last one is not a memory leak:

> 3) NFA_State::DeepCopy() (NFA.cc:58)
> 
>    Valgrind Stack:
>    ---------------
>    96 bytes in 1 blocks are possibly lost in loss record 3,374 of 6,072
>    at 0x4C2B0E0: operator new(unsigned long)
>    by 0x5CEDB0: NFA_State::DeepCopy() (NFA.cc:58)
>    by 0x5CF305: NFA_Machine::DuplicateMachine() (NFA.cc:210)
>    by 0x5CF63B: NFA_Machine::MakeRepl(int, int) (NFA.cc:252)
>    by 0x53B648: RE_parse() (re-parse.y:76)
>    by 0x5DD449: Specific_RE_Matcher::CompileSet(charPList const&,
> ptr_compat_intList const&) (RE.cc:142)
> 
>    Comments:
>    ---------
>     - New NFA_State allocate in DeepCopy function is not freed.
>     - The pointer is saved in 'mark' member variable. However, in
> ClearMarks function
>       the member is clear is simply cleared without freeing the pointer.

DeepCopy is only called when creating a new NFA_Machine in
DuplicateMachine. The pointer of the DeepCopy is stored in the first_state
member of NFA_Machine, which is Unref'd in NFA.cc:160. So I think this
seems fine.

Thanks again,
 Johanna

From slagell at illinois.edu  Mon Jun 20 14:27:43 2016
From: slagell at illinois.edu (Slagell, Adam J)
Date: Mon, 20 Jun 2016 21:27:43 +0000
Subject: [Bro] Security Officer opening at LIGO
Message-ID: <2171AC87-69BB-4C1B-90AF-583532E1481A@illinois.edu>

The NSF-funded LIGO project, responsible for the recent breakthrough discovery of gravitational waves that validate Einstein's theory, has posted an opening for a Cybersecurity Officer. This represents an opportunity to undertake cybersecurity in the support of scientific research with one of NSF's largest projects.

CTSC and NCSA are working with LIGO to help advertise the position. Please see the LIGO posting for more information and details on how to apply: https://jobs.caltech.edu/postings/4919

------

Adam J. Slagell
Chief Information Security Officer
Director, Cybersecurity Division
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
www.slagell.info

"Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." 










From jlay at slave-tothe-box.net  Wed Jun 22 16:19:05 2016
From: jlay at slave-tothe-box.net (James Lay)
Date: Wed, 22 Jun 2016 17:19:05 -0600
Subject: [Bro] Chacha Poly ciphers
Message-ID: <162e5d99853ec4d76c5ebbbb48774784@localhost>

FYI:

1466635836.174656       C42BzN1MQAC2spvAZe      192.168.1.101   39389   
31.13.76.84     443     TLSv12  unknown-52392   -       
graph.instagram.com     F       -       h2      T       
FcMVXF29wZnV4HnQqk,Fc87jcRtH8QGurEX5    (empty) 
CN=*.instagram.com,O=Instagram LLC,L=Menlo Park,ST=CA,C=US      
CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert 
Inc,C=US     -       -       ok


"unknown-52392" is chacha-poly1305

Interestingly, this is correctly detected using ssh:

1466634297.341693       Ca8K4v48feChdL1pQg      192.168.1.253   41500   
192.168.1.5     22      2       T       -       SSH-2.0-OpenSSH_6.6.1p1 
Ubuntu-2ubuntu2.7       SSH-2.0-OpenSSH_7.2p2 Debian-5  
chacha20-poly1305 at openssh.com   umac-64-etm at openssh.com none    
curve25519-sha256 at libssh.org    ssh-rsa

James

From johanna at icir.org  Wed Jun 22 17:09:41 2016
From: johanna at icir.org (Johanna Amann)
Date: Wed, 22 Jun 2016 17:09:41 -0700
Subject: [Bro] Chacha Poly ciphers
In-Reply-To: <162e5d99853ec4d76c5ebbbb48774784@localhost>
References: <162e5d99853ec4d76c5ebbbb48774784@localhost>
Message-ID: <E526BA7A-5FBD-4427-8568-31FD2775F562@icir.org>

Thank you - we already had polu1305 in there, but only the ones that 
were assigned in an earlier draft. I will update that.

Johanna

On 22 Jun 2016, at 16:19, James Lay wrote:

> FYI:
>
> 1466635836.174656       C42BzN1MQAC2spvAZe      192.168.1.101   39389
> 31.13.76.84     443     TLSv12  unknown-52392   -
> graph.instagram.com     F       -       h2      T
> FcMVXF29wZnV4HnQqk,Fc87jcRtH8QGurEX5    (empty)
> CN=*.instagram.com,O=Instagram LLC,L=Menlo Park,ST=CA,C=US
> CN=DigiCert SHA2 High Assurance Server 
> CA,OU=www.digicert.com,O=DigiCert
> Inc,C=US     -       -       ok
>
>
> "unknown-52392" is chacha-poly1305
>
> Interestingly, this is correctly detected using ssh:
>
> 1466634297.341693       Ca8K4v48feChdL1pQg      192.168.1.253   41500
> 192.168.1.5     22      2       T       -       
> SSH-2.0-OpenSSH_6.6.1p1
> Ubuntu-2ubuntu2.7       SSH-2.0-OpenSSH_7.2p2 Debian-5
> chacha20-poly1305 at openssh.com   umac-64-etm at openssh.com none
> curve25519-sha256 at libssh.org    ssh-rsa
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

From philosnef at yahoo.com  Thu Jun 23 07:08:05 2016
From: philosnef at yahoo.com (philosnef)
Date: Thu, 23 Jun 2016 14:08:05 +0000 (UTC)
Subject: [Bro] problem with traffic/statistics/caploss
References: <931767725.266855.1466690885918.JavaMail.yahoo.ref@mail.yahoo.com>
Message-ID: <931767725.266855.1466690885918.JavaMail.yahoo@mail.yahoo.com>

So, we are running pf_ring zc with bro. I have 128 gigs of ram on a 32 core system. According to pfcount, I am receiving ~3.25Gb/s. According to cap-stats, Bro is saying I am only getting 2500Mb/s. Both traffic analysis tools say I get ~440kpps. I trust pf_ring more than cap-stats when looking at throughput, but they both accurately identify the pps associated with this box. The capture-loss.log is indicating I am losing anywhere from 10-25% of my traffic. Pfcount says I am dropping 0 packets. I have tried doing ethtool -L 20 $iface (running 20 workers), but that caused my capture loss to skyrocket. I am running pfrings smp affinity, and have the standard set of ethtool flags set according to Bro documentation.?
Eventually, Bro eats all the ram in the box, but does not dip into swap. I have seen simaliar behavior on another box with 386 gigs of ram and 1mpps, but only 2.25Gb/s. On that box, Bro eats up all 386 gigs of ram...?
Does anyone have a clue exactly what is going on?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160623/59eed781/attachment.html 

From bro at pingtrip.com  Fri Jun 24 05:28:59 2016
From: bro at pingtrip.com (Dave Crawford)
Date: Fri, 24 Jun 2016 08:28:59 -0400
Subject: [Bro] PF_RING ZC Config
Message-ID: <CD0D7475-481B-45BC-85B6-AE5196423EC3@pingtrip.com>

Would anyone happen to have documentation for configuring ZC and Bro? I have NTop's PF_RING and ixgbe driver packages installed, the proper license in /etc/pf_ring, and have compiled Bro with the NTop libraries but I'm seeing the kernel error below along with a ton of ?split routing? messages in weird.conf, so I suspect the flows aren?t being load balanced correctly.
 
Jun 22 15:10:03 win-csignsm-01 kernel: [11060.244524] [PF_RING] Unable to activate two or more ZC sockets on the same interface eth6/link direction

The monitored NIC is an Intel X520-LR1.

Contents of /etc/pf_ring/zc/ixgbe/ixgbe.conf: 
RSS=10 allow_unsupported_sfp=0

Contents of /etc/pf_ring/hugepages.conf
node=1 hugepages=1024


And Bro is configured as:
[MID_INT]
type=worker
host=10.20.30.123
interface=zc:eth6
lb_method=pf_ring
lb_procs=10
pin_cpus=10,11,12,13,14,15,16,17,18,19

Thanks!
-Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160624/8631c1fc/attachment.html 

From aidaros.dev at gmail.com  Fri Jun 24 16:45:13 2016
From: aidaros.dev at gmail.com (Hashem Alaidaros)
Date: Sat, 25 Jun 2016 07:45:13 +0800
Subject: [Bro] Bro drop packets while not using CPU at full capacity
Message-ID: <CAFmcO-E39ukLwYpe55dTsB3=k51pHPuFADRF5aJWJWi1UxfTnA@mail.gmail.com>

Hi All
I use Bro for my PhD research, I add scripts in Bro and then see the CPU
and packet drop rate, using @load stats.bro. I'm using Bro 2.3 with
standard libcap.
I use tcpreplay from Machine A to replay the pre-captured traffic into Bro
multi-core machine B through port mirror switch. I replay the traffic from
100 to 1000 Mbps , When reach 200 Mbps and onward, packet start drop and
increases. Surprisingly, the CPU is not fully utilized, CPU still 40%
usage. What we know is that drop packet resulted from CPU full load, but in
our case CPU still less than 50%, so  My question, what is the cause of
this packet drop? Is it normal?

Best regards
Aidaros
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160625/76f79a94/attachment.html 

From bro at pingtrip.com  Fri Jun 24 19:50:22 2016
From: bro at pingtrip.com (Dave Crawford)
Date: Fri, 24 Jun 2016 22:50:22 -0400
Subject: [Bro] Bro drop packets while not using CPU at full capacity
In-Reply-To: <CAFmcO-E39ukLwYpe55dTsB3=k51pHPuFADRF5aJWJWi1UxfTnA@mail.gmail.com>
References: <CAFmcO-E39ukLwYpe55dTsB3=k51pHPuFADRF5aJWJWi1UxfTnA@mail.gmail.com>
Message-ID: <9BF3DE8E-C6A9-4F12-A642-E63A6020C855@pingtrip.com>

Is it possible that the CPU has two cores and Bro is consuming 100% of one core? Some tools average the core utilization to report "CPU usage".

> On Jun 24, 2016, at 7:45 PM, Hashem Alaidaros <aidaros.dev at gmail.com> wrote:
> 
> Hi All
> I use Bro for my PhD research, I add scripts in Bro and then see the CPU and packet drop rate, using @load stats.bro. I'm using Bro 2.3 with standard libcap.
> I use tcpreplay from Machine A to replay the pre-captured traffic into Bro multi-core machine B through port mirror switch. I replay the traffic from 100 to 1000 Mbps , When reach 200 Mbps and onward, packet start drop and increases. Surprisingly, the CPU is not fully utilized, CPU still 40% usage. What we know is that drop packet resulted from CPU full load, but in our case CPU still less than 50%, so  My question, what is the cause of this packet drop? Is it normal?
> 
> Best regards
> Aidaros
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From aidaros.dev at gmail.com  Sat Jun 25 04:05:59 2016
From: aidaros.dev at gmail.com (Hashem Alaidaros)
Date: Sat, 25 Jun 2016 19:05:59 +0800
Subject: [Bro] Bro drop packets while not using CPU at full capacity
In-Reply-To: <9BF3DE8E-C6A9-4F12-A642-E63A6020C855@pingtrip.com>
References: <CAFmcO-E39ukLwYpe55dTsB3=k51pHPuFADRF5aJWJWi1UxfTnA@mail.gmail.com>
	<9BF3DE8E-C6A9-4F12-A642-E63A6020C855@pingtrip.com>
Message-ID: <CAFmcO-HA8Td+QQi_SDRSdXDs-H7zLKduFin9R4p-D0WDW+-GJg@mail.gmail.com>

Thanks Dave,
I couldn't get what you mean. How stats.bro calculate CPU usage, is it per
core utilization? My bro machine is quad-core with hypertheading enabled,
means 8 logical cores. So, if one core is fully utilized then stats should
report 12.5% (100/8), not 40% or 60% as in my case. How my Bro report 60%
CPU with 20% drop packet rate reported? Is there any reason that make
packet drop?
Anyone could clarify please.
Thanks in advance

On Sat, Jun 25, 2016 at 10:50 AM, Dave Crawford <bro at pingtrip.com> wrote:

> Is it possible that the CPU has two cores and Bro is consuming 100% of one
> core? Some tools average the core utilization to report "CPU usage".
>
> > On Jun 24, 2016, at 7:45 PM, Hashem Alaidaros <aidaros.dev at gmail.com>
> wrote:
> >
> > Hi All
> > I use Bro for my PhD research, I add scripts in Bro and then see the CPU
> and packet drop rate, using @load stats.bro. I'm using Bro 2.3 with
> standard libcap.
> > I use tcpreplay from Machine A to replay the pre-captured traffic into
> Bro multi-core machine B through port mirror switch. I replay the traffic
> from 100 to 1000 Mbps , When reach 200 Mbps and onward, packet start drop
> and increases. Surprisingly, the CPU is not fully utilized, CPU still 40%
> usage. What we know is that drop packet resulted from CPU full load, but in
> our case CPU still less than 50%, so  My question, what is the cause of
> this packet drop? Is it normal?
> >
> > Best regards
> > Aidaros
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>


-- 
A friend in need Is a friend indeed
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160625/d3594a0a/attachment.html 

From blackhole.em at gmail.com  Sat Jun 25 10:50:24 2016
From: blackhole.em at gmail.com (Joe Blow)
Date: Sat, 25 Jun 2016 13:50:24 -0400
Subject: [Bro] Bro drop packets while not using CPU at full capacity
In-Reply-To: <CAFmcO-HA8Td+QQi_SDRSdXDs-H7zLKduFin9R4p-D0WDW+-GJg@mail.gmail.com>
Message-ID: <576ec463.cf98810a.e0d40.ffff9481@mx.google.com>

An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160625/d9a947ef/attachment.html 

From aidaros.dev at gmail.com  Sat Jun 25 16:32:08 2016
From: aidaros.dev at gmail.com (Hashem Alaidaros)
Date: Sun, 26 Jun 2016 07:32:08 +0800
Subject: [Bro] Bro drop packets while not using CPU at full capacity
In-Reply-To: <576ec463.cf98810a.e0d40.ffff9481@mx.google.com>
References: <CAFmcO-HA8Td+QQi_SDRSdXDs-H7zLKduFin9R4p-D0WDW+-GJg@mail.gmail.com>
	<576ec463.cf98810a.e0d40.ffff9481@mx.google.com>
Message-ID: <CAFmcO-F0Ym5TQUVkG5MhKoe9DvLLK=RrvDDPY9_PxXJ_veo0JA@mail.gmail.com>

I started my experiments when Bro 2.3 was the latest stable version. All my
results are based on 2.3, I can not shift to newer version now.
Anyone can clarify why Packet are dropping while no fully CPU utilization.?

Best regards

On Sun, Jun 26, 2016 at 1:50 AM, Joe Blow <blackhole.em at gmail.com> wrote:

> Is there any reason you aren't using 2.4.x?  Step one would be to use that
> I would think. 2.4.x fixed a great many bugs I believe.
>
> Cheers,
>
> JB
>
> Sent from my BlackBerry Smartphone on the Verizon 4G LTE Network
> *From:*aidaros.dev at gmail.com
> *Sent:*June 25, 2016 7:15 AM
> *To:*bro at pingtrip.com
> *Cc:*bro at bro.org
> *Subject:*Re: [Bro] Bro drop packets while not using CPU at full capacity
>
> Thanks Dave,
> I couldn't get what you mean. How stats.bro calculate CPU usage, is it per
> core utilization? My bro machine is quad-core with hypertheading enabled,
> means 8 logical cores. So, if one core is fully utilized then stats should
> report 12.5% (100/8), not 40% or 60% as in my case. How my Bro report 60%
> CPU with 20% drop packet rate reported? Is there any reason that make
> packet drop?
> Anyone could clarify please.
> Thanks in advance
>
> On Sat, Jun 25, 2016 at 10:50 AM, Dave Crawford <bro at pingtrip.com> wrote:
>
>> Is it possible that the CPU has two cores and Bro is consuming 100% of
>> one core? Some tools average the core utilization to report "CPU usage".
>>
>> > On Jun 24, 2016, at 7:45 PM, Hashem Alaidaros <aidaros.dev at gmail.com>
>> wrote:
>> >
>> > Hi All
>> > I use Bro for my PhD research, I add scripts in Bro and then see the
>> CPU and packet drop rate, using @load stats.bro. I'm using Bro 2.3 with
>> standard libcap.
>> > I use tcpreplay from Machine A to replay the pre-captured traffic into
>> Bro multi-core machine B through port mirror switch. I replay the traffic
>> from 100 to 1000 Mbps , When reach 200 Mbps and onward, packet start drop
>> and increases. Surprisingly, the CPU is not fully utilized, CPU still 40%
>> usage. What we know is that drop packet resulted from CPU full load, but in
>> our case CPU still less than 50%, so  My question, what is the cause of
>> this packet drop? Is it normal?
>> >
>> > Best regards
>> > Aidaros
>> >
>> > _______________________________________________
>> > Bro mailing list
>> > bro at bro-ids.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>
>
> --
> A friend in need Is a friend indeed
>



-- 
A friend in need Is a friend indeed
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160626/52337fb6/attachment.html 

From neslog at gmail.com  Mon Jun 27 05:42:25 2016
From: neslog at gmail.com (Neslog)
Date: Mon, 27 Jun 2016 08:42:25 -0400
Subject: [Bro] Bro drop packets while not using CPU at full capacity
In-Reply-To: <CAFmcO-F0Ym5TQUVkG5MhKoe9DvLLK=RrvDDPY9_PxXJ_veo0JA@mail.gmail.com>
References: <CAFmcO-HA8Td+QQi_SDRSdXDs-H7zLKduFin9R4p-D0WDW+-GJg@mail.gmail.com>
	<576ec463.cf98810a.e0d40.ffff9481@mx.google.com>
	<CAFmcO-F0Ym5TQUVkG5MhKoe9DvLLK=RrvDDPY9_PxXJ_veo0JA@mail.gmail.com>
Message-ID: <CALgfRGJdLepBGu8LswsPY5dgo_P0GO52d_8vVf8p3OzYUJa-PA@mail.gmail.com>

I've been troubleshooting my clusters recently. I'm seeing some drops in
the kernel using drop watch.  Previously I've seen loss from spans when
approaching link saturation
On Jun 25, 2016 7:34 PM, "Hashem Alaidaros" <aidaros.dev at gmail.com> wrote:

> I started my experiments when Bro 2.3 was the latest stable version. All
> my results are based on 2.3, I can not shift to newer version now.
> Anyone can clarify why Packet are dropping while no fully CPU utilization.?
>
> Best regards
>
> On Sun, Jun 26, 2016 at 1:50 AM, Joe Blow <blackhole.em at gmail.com> wrote:
>
>> Is there any reason you aren't using 2.4.x?  Step one would be to use
>> that I would think. 2.4.x fixed a great many bugs I believe.
>>
>> Cheers,
>>
>> JB
>>
>> Sent from my BlackBerry Smartphone on the Verizon 4G LTE Network
>> *From:*aidaros.dev at gmail.com
>> *Sent:*June 25, 2016 7:15 AM
>> *To:*bro at pingtrip.com
>> *Cc:*bro at bro.org
>> *Subject:*Re: [Bro] Bro drop packets while not using CPU at full capacity
>>
>> Thanks Dave,
>> I couldn't get what you mean. How stats.bro calculate CPU usage, is it
>> per core utilization? My bro machine is quad-core with hypertheading
>> enabled, means 8 logical cores. So, if one core is fully utilized then
>> stats should report 12.5% (100/8), not 40% or 60% as in my case. How my Bro
>> report 60% CPU with 20% drop packet rate reported? Is there any reason that
>> make packet drop?
>> Anyone could clarify please.
>> Thanks in advance
>>
>> On Sat, Jun 25, 2016 at 10:50 AM, Dave Crawford <bro at pingtrip.com> wrote:
>>
>>> Is it possible that the CPU has two cores and Bro is consuming 100% of
>>> one core? Some tools average the core utilization to report "CPU usage".
>>>
>>> > On Jun 24, 2016, at 7:45 PM, Hashem Alaidaros <aidaros.dev at gmail.com>
>>> wrote:
>>> >
>>> > Hi All
>>> > I use Bro for my PhD research, I add scripts in Bro and then see the
>>> CPU and packet drop rate, using @load stats.bro. I'm using Bro 2.3 with
>>> standard libcap.
>>> > I use tcpreplay from Machine A to replay the pre-captured traffic into
>>> Bro multi-core machine B through port mirror switch. I replay the traffic
>>> from 100 to 1000 Mbps , When reach 200 Mbps and onward, packet start drop
>>> and increases. Surprisingly, the CPU is not fully utilized, CPU still 40%
>>> usage. What we know is that drop packet resulted from CPU full load, but in
>>> our case CPU still less than 50%, so  My question, what is the cause of
>>> this packet drop? Is it normal?
>>> >
>>> > Best regards
>>> > Aidaros
>>> >
>>> > _______________________________________________
>>> > Bro mailing list
>>> > bro at bro-ids.org
>>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>
>>>
>>
>>
>> --
>> A friend in need Is a friend indeed
>>
>
>
>
> --
> A friend in need Is a friend indeed
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160627/e732df6d/attachment.html 

From aidaros.dev at gmail.com  Mon Jun 27 08:26:31 2016
From: aidaros.dev at gmail.com (Hashem Alaidaros)
Date: Mon, 27 Jun 2016 23:26:31 +0800
Subject: [Bro] Bro drop packets while not using CPU at full capacity
In-Reply-To: <CALgfRGJdLepBGu8LswsPY5dgo_P0GO52d_8vVf8p3OzYUJa-PA@mail.gmail.com>
References: <CAFmcO-HA8Td+QQi_SDRSdXDs-H7zLKduFin9R4p-D0WDW+-GJg@mail.gmail.com>
	<576ec463.cf98810a.e0d40.ffff9481@mx.google.com>
	<CAFmcO-F0Ym5TQUVkG5MhKoe9DvLLK=RrvDDPY9_PxXJ_veo0JA@mail.gmail.com>
	<CALgfRGJdLepBGu8LswsPY5dgo_P0GO52d_8vVf8p3OzYUJa-PA@mail.gmail.com>
Message-ID: <CAFmcO-F2JJR3ggGD_6nFg9pS3SbWQFvCgzPa57EzviL-FjnukA@mail.gmail.com>

Thanks Neslog and Slawek, for reply.

In my experiments, I do not use cluster. My main question now, what is the
cause of drop packets reported from the following stats.bro

info$pkts_dropped = ns$pkts_dropped  - last_ns$pkts_dropped;

Is it packet drop by: 1) Bro, 2)libpcap 3)kernel OS ?

Regards

On Mon, Jun 27, 2016 at 8:42 PM, Neslog <neslog at gmail.com> wrote:

> I've been troubleshooting my clusters recently. I'm seeing some drops in
> the kernel using drop watch.  Previously I've seen loss from spans when
> approaching link saturation
> On Jun 25, 2016 7:34 PM, "Hashem Alaidaros" <aidaros.dev at gmail.com> wrote:
>
>> I started my experiments when Bro 2.3 was the latest stable version. All
>> my results are based on 2.3, I can not shift to newer version now.
>> Anyone can clarify why Packet are dropping while no fully CPU
>> utilization.?
>>
>> Best regards
>>
>> On Sun, Jun 26, 2016 at 1:50 AM, Joe Blow <blackhole.em at gmail.com> wrote:
>>
>>> Is there any reason you aren't using 2.4.x?  Step one would be to use
>>> that I would think. 2.4.x fixed a great many bugs I believe.
>>>
>>> Cheers,
>>>
>>> JB
>>>
>>> Sent from my BlackBerry Smartphone on the Verizon 4G LTE Network
>>> *From:*aidaros.dev at gmail.com
>>> *Sent:*June 25, 2016 7:15 AM
>>> *To:*bro at pingtrip.com
>>> *Cc:*bro at bro.org
>>> *Subject:*Re: [Bro] Bro drop packets while not using CPU at full
>>> capacity
>>>
>>> Thanks Dave,
>>> I couldn't get what you mean. How stats.bro calculate CPU usage, is it
>>> per core utilization? My bro machine is quad-core with hypertheading
>>> enabled, means 8 logical cores. So, if one core is fully utilized then
>>> stats should report 12.5% (100/8), not 40% or 60% as in my case. How my Bro
>>> report 60% CPU with 20% drop packet rate reported? Is there any reason that
>>> make packet drop?
>>> Anyone could clarify please.
>>> Thanks in advance
>>>
>>> On Sat, Jun 25, 2016 at 10:50 AM, Dave Crawford <bro at pingtrip.com>
>>> wrote:
>>>
>>>> Is it possible that the CPU has two cores and Bro is consuming 100% of
>>>> one core? Some tools average the core utilization to report "CPU usage".
>>>>
>>>> > On Jun 24, 2016, at 7:45 PM, Hashem Alaidaros <aidaros.dev at gmail.com>
>>>> wrote:
>>>> >
>>>> > Hi All
>>>> > I use Bro for my PhD research, I add scripts in Bro and then see the
>>>> CPU and packet drop rate, using @load stats.bro. I'm using Bro 2.3 with
>>>> standard libcap.
>>>> > I use tcpreplay from Machine A to replay the pre-captured traffic
>>>> into Bro multi-core machine B through port mirror switch. I replay the
>>>> traffic from 100 to 1000 Mbps , When reach 200 Mbps and onward, packet
>>>> start drop and increases. Surprisingly, the CPU is not fully utilized, CPU
>>>> still 40% usage. What we know is that drop packet resulted from CPU full
>>>> load, but in our case CPU still less than 50%, so  My question, what is the
>>>> cause of this packet drop? Is it normal?
>>>> >
>>>> > Best regards
>>>> > Aidaros
>>>> >
>>>> > _______________________________________________
>>>> > Bro mailing list
>>>> > bro at bro-ids.org
>>>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>>
>>>>
>>>
>>>
>>> --
>>> A friend in need Is a friend indeed
>>>
>>
>>
>>
>> --
>> A friend in need Is a friend indeed
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>


-- 
A friend in need Is a friend indeed
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160627/4879b7a0/attachment-0001.html 

From mehmetleb at gmail.com  Mon Jun 27 08:31:02 2016
From: mehmetleb at gmail.com (=?UTF-8?B?TWVobWV0IExFQkxFQsSwQ8Sw?=)
Date: Mon, 27 Jun 2016 18:31:02 +0300
Subject: [Bro] Bro drop packets while not using CPU at full capacity
In-Reply-To: <CALgfRGJdLepBGu8LswsPY5dgo_P0GO52d_8vVf8p3OzYUJa-PA@mail.gmail.com>
References: <CAFmcO-HA8Td+QQi_SDRSdXDs-H7zLKduFin9R4p-D0WDW+-GJg@mail.gmail.com>
	<576ec463.cf98810a.e0d40.ffff9481@mx.google.com>
	<CAFmcO-F0Ym5TQUVkG5MhKoe9DvLLK=RrvDDPY9_PxXJ_veo0JA@mail.gmail.com>
	<CALgfRGJdLepBGu8LswsPY5dgo_P0GO52d_8vVf8p3OzYUJa-PA@mail.gmail.com>
Message-ID: <CANnYDkUauUFzV3=p4H3aOMU2Pk3h9WMDKPQq2ZENSNui++HjQA@mail.gmail.com>

Hi,

I had such a problem before, there was no cpu or other problem but bro was
dropping the packets. It was saying packets are larger than expected MTU,
so it was dropping then. It turned out that some NICs offload the
reassembly of traffic into bigger packets so that fewer packets are passed
up to stack. Actually the intention is to reduce burden on CPU, but it also
causes bro to drop packets. So it may be a similar problem in your case. I
solved the problem changing interface settings, you can try the following
settings. I hope this will solve your problem.

ethtool -K eth0 sg off
ethtool -K eth0 tso off
ethtool -K eth0 ufo off
ethtool -K eth0 gso off
ethtool -K eth0 gro off
ethtool -K eth0 lro off

Best regards,

Mehmet Leblebici


27 Haziran 2016 Pazartesi tarihinde, Neslog <neslog at gmail.com> yazd?:

> I've been troubleshooting my clusters recently. I'm seeing some drops in
> the kernel using drop watch.  Previously I've seen loss from spans when
> approaching link saturation
> On Jun 25, 2016 7:34 PM, "Hashem Alaidaros" <aidaros.dev at gmail.com
> <javascript:_e(%7B%7D,'cvml','aidaros.dev at gmail.com');>> wrote:
>
>> I started my experiments when Bro 2.3 was the latest stable version. All
>> my results are based on 2.3, I can not shift to newer version now.
>> Anyone can clarify why Packet are dropping while no fully CPU
>> utilization.?
>>
>> Best regards
>>
>> On Sun, Jun 26, 2016 at 1:50 AM, Joe Blow <blackhole.em at gmail.com
>> <javascript:_e(%7B%7D,'cvml','blackhole.em at gmail.com');>> wrote:
>>
>>> Is there any reason you aren't using 2.4.x?  Step one would be to use
>>> that I would think. 2.4.x fixed a great many bugs I believe.
>>>
>>> Cheers,
>>>
>>> JB
>>>
>>> Sent from my BlackBerry Smartphone on the Verizon 4G LTE Network
>>> *From:*aidaros.dev at gmail.com
>>> <javascript:_e(%7B%7D,'cvml','aidaros.dev at gmail.com');>
>>> *Sent:*June 25, 2016 7:15 AM
>>> *To:*bro at pingtrip.com <javascript:_e(%7B%7D,'cvml','bro at pingtrip.com');>
>>> *Cc:*bro at bro.org <javascript:_e(%7B%7D,'cvml','bro at bro.org');>
>>> *Subject:*Re: [Bro] Bro drop packets while not using CPU at full
>>> capacity
>>>
>>> Thanks Dave,
>>> I couldn't get what you mean. How stats.bro calculate CPU usage, is it
>>> per core utilization? My bro machine is quad-core with hypertheading
>>> enabled, means 8 logical cores. So, if one core is fully utilized then
>>> stats should report 12.5% (100/8), not 40% or 60% as in my case. How my Bro
>>> report 60% CPU with 20% drop packet rate reported? Is there any reason that
>>> make packet drop?
>>> Anyone could clarify please.
>>> Thanks in advance
>>>
>>> On Sat, Jun 25, 2016 at 10:50 AM, Dave Crawford <bro at pingtrip.com
>>> <javascript:_e(%7B%7D,'cvml','bro at pingtrip.com');>> wrote:
>>>
>>>> Is it possible that the CPU has two cores and Bro is consuming 100% of
>>>> one core? Some tools average the core utilization to report "CPU usage".
>>>>
>>>> > On Jun 24, 2016, at 7:45 PM, Hashem Alaidaros <aidaros.dev at gmail.com
>>>> <javascript:_e(%7B%7D,'cvml','aidaros.dev at gmail.com');>> wrote:
>>>> >
>>>> > Hi All
>>>> > I use Bro for my PhD research, I add scripts in Bro and then see the
>>>> CPU and packet drop rate, using @load stats.bro. I'm using Bro 2.3 with
>>>> standard libcap.
>>>> > I use tcpreplay from Machine A to replay the pre-captured traffic
>>>> into Bro multi-core machine B through port mirror switch. I replay the
>>>> traffic from 100 to 1000 Mbps , When reach 200 Mbps and onward, packet
>>>> start drop and increases. Surprisingly, the CPU is not fully utilized, CPU
>>>> still 40% usage. What we know is that drop packet resulted from CPU full
>>>> load, but in our case CPU still less than 50%, so  My question, what is the
>>>> cause of this packet drop? Is it normal?
>>>> >
>>>> > Best regards
>>>> > Aidaros
>>>> >
>>>> > _______________________________________________
>>>> > Bro mailing list
>>>> > bro at bro-ids.org <javascript:_e(%7B%7D,'cvml','bro at bro-ids.org');>
>>>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>>
>>>>
>>>
>>>
>>> --
>>> A friend in need Is a friend indeed
>>>
>>
>>
>>
>> --
>> A friend in need Is a friend indeed
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org <javascript:_e(%7B%7D,'cvml','bro at bro-ids.org');>
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160627/48789e7e/attachment.html 

From dnj0496 at gmail.com  Tue Jun 28 16:25:45 2016
From: dnj0496 at gmail.com (Dk Jack)
Date: Tue, 28 Jun 2016 16:25:45 -0700
Subject: [Bro] delay pcap processing
Message-ID: <CAGATen45vqvu4UYn6zZG0gOfuch8P9KfDCm5eA4Lymsp0zkCAw@mail.gmail.com>

Hi,
I am trying to run in batch mode i.e. using '-r' option. In my script, I am
trying read some data into
bro from text file. As per the input framework documentation, reading data
from file is an
asynchronous event, my packet processing is completing before I receive the
Input::end_of_data
event. Is there a way to delay packet processing till file read is complete?

Thanks,
Dk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160628/44c17d8c/attachment.html 

From anthony.kasza at gmail.com  Tue Jun 28 16:49:36 2016
From: anthony.kasza at gmail.com (anthony kasza)
Date: Tue, 28 Jun 2016 16:49:36 -0700
Subject: [Bro] delay pcap processing
In-Reply-To: <CAEZw2bz0OtS=K81f2v8jSJMZE+gvqx-OD6Ez8sKYST-ihRUpKQ@mail.gmail.com>
References: <CAGATen45vqvu4UYn6zZG0gOfuch8P9KfDCm5eA4Lymsp0zkCAw@mail.gmail.com>
	<CAEZw2bz0OtS=K81f2v8jSJMZE+gvqx-OD6Ez8sKYST-ihRUpKQ@mail.gmail.com>
Message-ID: <CAEZw2bzDZYHhXXeMHXvmh7BZAEhidnap-C01gftjVEh+rFN+KQ@mail.gmail.com>

Call suspend_processing() from scriptland until your table is ready. Here's
an example:

https://github.com/anthonykasza/scratch_pad/tree/master/input_for_pcaps

-AK
On Jun 28, 2016 7:40 PM, "Dk Jack" <dnj0496 at gmail.com> wrote:

Hi,
I am trying to run in batch mode i.e. using '-r' option. In my script, I am
trying read some data into
bro from text file. As per the input framework documentation, reading data
from file is an
asynchronous event, my packet processing is completing before I receive the
Input::end_of_data
event. Is there a way to delay packet processing till file read is complete?

Thanks,
Dk

_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160628/92f57a4c/attachment.html 

From dnj0496 at gmail.com  Tue Jun 28 16:51:11 2016
From: dnj0496 at gmail.com (Dk Jack)
Date: Tue, 28 Jun 2016 16:51:11 -0700
Subject: [Bro] delay pcap processing
In-Reply-To: <CAEZw2bzDZYHhXXeMHXvmh7BZAEhidnap-C01gftjVEh+rFN+KQ@mail.gmail.com>
References: <CAGATen45vqvu4UYn6zZG0gOfuch8P9KfDCm5eA4Lymsp0zkCAw@mail.gmail.com>
	<CAEZw2bz0OtS=K81f2v8jSJMZE+gvqx-OD6Ez8sKYST-ihRUpKQ@mail.gmail.com>
	<CAEZw2bzDZYHhXXeMHXvmh7BZAEhidnap-C01gftjVEh+rFN+KQ@mail.gmail.com>
Message-ID: <CAGATen4P57CG1Ke+VwRkRyTNG1fVKp3WXAhSSCRNn4mCfO1sfA@mail.gmail.com>

Awesome! Thank you.

On Tue, Jun 28, 2016 at 4:49 PM, anthony kasza <anthony.kasza at gmail.com>
wrote:

> Call suspend_processing() from scriptland until your table is ready.
> Here's an example:
>
> https://github.com/anthonykasza/scratch_pad/tree/master/input_for_pcaps
>
> -AK
> On Jun 28, 2016 7:40 PM, "Dk Jack" <dnj0496 at gmail.com> wrote:
>
> Hi,
> I am trying to run in batch mode i.e. using '-r' option. In my script, I
> am trying read some data into
> bro from text file. As per the input framework documentation, reading data
> from file is an
> asynchronous event, my packet processing is completing before I receive
> the Input::end_of_data
> event. Is there a way to delay packet processing till file read is
> complete?
>
> Thanks,
> Dk
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160628/81fc2734/attachment.html 

From greg at broala.com  Tue Jun 28 18:41:31 2016
From: greg at broala.com (Gregory Bell)
Date: Tue, 28 Jun 2016 18:41:31 -0700
Subject: [Bro] CENIC seeking senior security person
Message-ID: <CAJHFxAoPx3XJg3gA=bgY9otFpvLB8PSKA8qOc7dr30gzcOEZGg@mail.gmail.com>

CENIC (one of the largest and most important public networks in the work)
is recruiting an experienced security engineer - with Bro skills - to lead
a new security group:

http://cenic.org/about/career-detail/SeniorInformationSecurityAnalyst_Engineer

Please forward to anyone you think might be interested.  The initial job
location is LA, but moving to the CENIC office in Berkeley might be
possible in time.

I'm on the CENIC board, and it's a great organization. Louis Fox (the CEO)
is a very gifted leader.  This network serves 20 million people, and
interconnects most universities, colleges, K-12s, and libraries in
California.

- Greg
-- 
Gregory Bell, PhD
CEO - Broala
www.broala.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160628/8f665671/attachment.html 

From eyrich at illinois.edu  Wed Jun 29 07:23:30 2016
From: eyrich at illinois.edu (James Eyrich)
Date: Wed, 29 Jun 2016 09:23:30 -0500
Subject: [Bro] Security Engineer position open at NCSA
In-Reply-To: <2171AC87-69BB-4C1B-90AF-583532E1481A@illinois.edu>
References: <2171AC87-69BB-4C1B-90AF-583532E1481A@illinois.edu>
Message-ID: <27244645-dbbc-1afe-cf61-823d97d6841c@illinois.edu>

NCSA is looking for at least one Security Engineer

https://jobs.illinois.edu/search-jobs/job-details?jobID=65782&job=security-engineer-senior-security-engineer-national-center-for-supercomputing-applications-a1600299

Interviews and offers may be made before the closing date, so don't wait
to apply.

Initial job duties will include:

1) Nagios. We have a fairly extensive system, but we'd like to start
generating the config programmatically instead of by hand and start
fine-tuning our checks, alerts, etc.
2) Automating system provisioning and updates using Katello.
3) Vetting. Important systems undergo a security vetting process before
they go in production. We're working on refining this and automating it.
4) CentOS 7 upgrade. We're upgrading our systems from CentOS 6 to 7.
5) Some VMware care and feeding.




From michael.fry at morphick.com  Wed Jun 29 15:58:02 2016
From: michael.fry at morphick.com (Michael Fry)
Date: Wed, 29 Jun 2016 22:58:02 +0000
Subject: [Bro] Assertion failure, Tag.cc
In-Reply-To: <CA+6Kvzoc44iWzOf4ZpDi86PBdogFRbZ0wyn=YJuo47V1km9h3w@mail.gmail.com>
References: <CA+6KvzoT30u0N85XUZb0SxbsLM0HZdnb2UVDBjNLCw=oRSrdig@mail.gmail.com>
	<F8A6A23F-862D-482A-B248-4A6825994E75@illinois.edu>
	<CA+6Kvzoc44iWzOf4ZpDi86PBdogFRbZ0wyn=YJuo47V1km9h3w@mail.gmail.com>
Message-ID: <CA+6KvzrQk0mmjObi37nBVuT1UuKABoYORddUxcxO9YdAdLQJYQ@mail.gmail.com>

I've been able to narrow down the circumstances where we see this core
dump. In the dozens of times that I've seen it, the file being extracted is
always delivered over HTTP via the BITS client.

- I have set an extract limit of 20MB.
- The last messages that I see logged from a bro worker before it crashes
indicates that my file_extraction_limit() event handler is called twice
with the same FUID. I would expect this event to be fired only once.
- I have confirmed by examining a backtrace of a core file that the crashed
Bro worker was analyzing the very same FUID mentioned above at the time of
the crash.
- Below is log output from a recent occurrence. The file FBzliCXoEipv0oEM8
appears in the log under 4 different connections with the same IPs within a
couple of seconds. Note that seen_bytes is just a fraction of the
total_bytes. I have confirmed that file_state_remove() is fired 4 times for
the same FUID.
- Since this activity originated from 4 different connections, I would
expect 4 different FUIDs.
- In some cases, Bro handles extraction of similar files just fine, but
only if the activity occurs on a single connection (its FUID is logged
exactly once in files.log). In these cases, seen_bytes=total_bytes, and the
file_extraction_limit() event is called only once, and thus no core dump.
- My script has a file_state_remove() handler that forwards the extracted
file for additional external analysis.

Here is an example that led to a crash. Note the same FUID is logged
multiple times with different connection UIDs.

root at redacted:/opt/cbts/log/bro/2016-06-28# zgrep FBzliCXoEipv0oEM8 *
files.2016-06-28-18-00-00.log.dz:1467138209.682287      FBzliCXoEipv0oEM8
    13.107.4.50     10.100.162.133  CapHn31QDgLbeGCqIh      HTTP    0
EXTRACT,SHA1,MD5        application/zip
Microsoft.Getstarted_3.11.1.0_neutral_~_8wekyb3d8bbwe.AppxBundle
 0.010122        F       F       22610   8811168 0       0       F       -
      02af20e0f97904b4c9da2c5a0071c324
 b09760ed7e7f64e389eb709abf97cbbc395cd22a        -
HTTP-FBzliCXoEipv0oEM8-Microsoft.Getstarted_3.11.1.0_neutral_~_8wekyb3d8bbwe.AppxBundle.file
files.2016-06-28-18-00-00.log.dz:1467138211.315952      FBzliCXoEipv0oEM8
    13.107.4.50     10.100.162.133  CbGwtH2ajMS3nq8oR9      HTTP    0
MD5,SHA1,EXTRACT        application/zip
Microsoft.Getstarted_3.11.1.0_neutral_~_8wekyb3d8bbwe.AppxBundle
 0.007328        F       F       13850   8811168 0       0       F       -
      7d8f20dc6941d42cb735bfbb1f5dee1d
 dccb1e3e513ab377ee909823c538ba196140a6df        -
HTTP-FBzliCXoEipv0oEM8-Microsoft.Getstarted_3.11.1.0_neutral_~_8wekyb3d8bbwe.AppxBundle.file
files.2016-06-28-18-00-00.log.dz:1467138211.234745      FBzliCXoEipv0oEM8
    13.107.4.50     10.100.162.133  ClZAlx4VvTPCGXWnGf      HTTP    0
SHA1,EXTRACT,MD5        application/zip
Microsoft.Getstarted_3.11.1.0_neutral_~_8wekyb3d8bbwe.AppxBundle
 0.010127        F       F       22610   8811168 0       0       F       -
      02af20e0f97904b4c9da2c5a0071c324
 b09760ed7e7f64e389eb709abf97cbbc395cd22a        -
HTTP-FBzliCXoEipv0oEM8-Microsoft.Getstarted_3.11.1.0_neutral_~_8wekyb3d8bbwe.AppxBundle.file
files.2016-06-28-18-00-00.log.dz:1467138211.395275      FBzliCXoEipv0oEM8
    13.107.4.50     10.100.162.133  C1PiIFx6hRC32DES9       HTTP    0
SHA1,MD5,EXTRACT        application/zip
Microsoft.Getstarted_3.11.1.0_neutral_~_8wekyb3d8bbwe.AppxBundle
 0.008257        F       F       16770   8811168 0       0       F       -
      c6c39cc6b8a6773a8b4b416a919cfca6
 0ae53083b1fc02d7710e9c2b12ad930990e73c8e        -
HTTP-FBzliCXoEipv0oEM8-Microsoft.Getstarted_3.11.1.0_neutral_~_8wekyb3d8bbwe.AppxBundle.file
http.2016-06-28-18-00-00.log.dz:1467138209.662449       CapHn31QDgLbeGCqIh
     10.100.162.133  61074   13.107.4.50     80      2       GET
tlu.dl.delivery.mp.microsoft.com
 /filestreamingservice/files/9f5dbc38-f074-45be-8346-ca45b940a576?P1=1467138990&P2=301&P3=2&P4=M61a7Qj+Xwl55JKMXfAzkJPsnDstij/KQXct1zwrcGI=
     -       Microsoft BITS/7.8      0       22610   200     OK      -
  --      (empty) -       -       -       -       -       FBzliCXoEipv0oEM8
      application/zip
http.2016-06-28-18-00-00.log.dz:1467138209.748598       CalgEr4pL03r1nZKql
     10.100.162.133  61075   13.107.4.50     80      1       GET
tlu.dl.delivery.mp.microsoft.com
 /filestreamingservice/files/9f5dbc38-f074-45be-8346-ca45b940a576?P1=1467138990&P2=301&P3=2&P4=M61a7Qj+Xwl55JKMXfAzkJPsnDstij/KQXct1zwrcGI=
     -       Microsoft BITS/7.8      0       94880   206     Partial
Content -       -       -       (empty) -       -       -       -       -
    FBzliCXoEipv0oEM8       -
http.2016-06-28-18-00-00.log.dz:1467138211.295292       CbGwtH2ajMS3nq8oR9
     10.100.162.133  61077   13.107.4.50     80      1       GET
tlu.dl.delivery.mp.microsoft.com
 /filestreamingservice/files/9f5dbc38-f074-45be-8346-ca45b940a576?P1=1467138990&P2=301&P3=2&P4=M61a7Qj+Xwl55JKMXfAzkJPsnDstij/KQXct1zwrcGI=
     -       Microsoft BITS/7.8      0       13850   200     OK      -
  --      (empty) -       -       -       -       -       FBzliCXoEipv0oEM8
      application/zip
http.2016-06-28-18-00-00.log.dz:1467138211.214133       ClZAlx4VvTPCGXWnGf
     10.100.162.133  61076   13.107.4.50     80      1       GET
tlu.dl.delivery.mp.microsoft.com
 /filestreamingservice/files/9f5dbc38-f074-45be-8346-ca45b940a576?P1=1467138990&P2=301&P3=2&P4=M61a7Qj+Xwl55JKMXfAzkJPsnDstij/KQXct1zwrcGI=
     -       Microsoft BITS/7.8      0       22610   200     OK      -
  --      (empty) -       -       -       -       -       FBzliCXoEipv0oEM8
      application/zip
http.2016-06-28-18-00-00.log.dz:1467138211.373513       C1PiIFx6hRC32DES9
    10.100.162.133  61078   13.107.4.50     80      1       GET
tlu.dl.delivery.mp.microsoft.com
 /filestreamingservice/files/9f5dbc38-f074-45be-8346-ca45b940a576?P1=1467138990&P2=301&P3=2&P4=M61a7Qj+Xwl55JKMXfAzkJPsnDstij/KQXct1zwrcGI=
     -       Microsoft BITS/7.8      0       16770   200     OK      -
  --      (empty) -       -       -       -       -       FBzliCXoEipv0oEM8
      application/zip

Removing my file_extraction_limit() handler seems to have eliminated the
crash, but since the file_state_remove() event is fired multiple times for
the same FUID, and the extracted filename is the same, the content of the
extracted file is suspect.

>From my limited understanding of BITS, these are zipped file segments being
delivered in the background, but shouldn't the content delivered over each
connection be assigned a different FUID?

Is this a known issue? We're using the Bro 2.4.1 Debian package built for
Ubuntu 12.04.

Regards,
Mike



On Tue, Apr 5, 2016 at 11:01 AM Michael Fry <michael.fry at morphick.com>
wrote:

> Thanks for the response. We do have a custom script running that chooses
> files to extract based on mime type. Here is the stack trace that I'm
> seeing. It's always the same.
>
> Regards
>
> #0  0x00007ffb28a680d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>
> #1  0x00007ffb28a6b83b in abort () from /lib/x86_64-linux-gnu/libc.so.6
>
> #2  0x00007ffb28a60d9e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>
> #3  0x00007ffb28a60e42 in __assert_fail () from
> /lib/x86_64-linux-gnu/libc.so.6
>
> #4  0x0000000000852850 in Tag::AsEnumVal (this=0x7fff27c7a8e0,
> etype=0x246a700) at /home/mfry/dev/bro24/bro-2.4.1/src/Tag.cc:72
>
> #5  0x0000000000b51b7b in file_analysis::Tag::AsEnumVal
> (this=0x7fff27c7a8e0) at
> /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/Tag.cc:23
>
> #6  0x0000000000b4d95b in file_analysis::AnalyzerSet::GetKey
> (this=0xb74e850, t=..., args=0xff59ca0) at
> /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/AnalyzerSet.cc:155
>
> #7  0x0000000000b4d7c7 in file_analysis::AnalyzerSet::QueueRemove
> (this=0xb74e850, tag=..., args=0xff59ca0) at
> /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/AnalyzerSet.cc:140
>
> #8  0x0000000000b4b3c9 in file_analysis::File::DeliverStream
> (this=0xb74e820,
>
>     data=0xc02f4f0
> "\217\020\272\214\355\361\301\272p\200\371\355\f\301\305\066\247Bj\216\027v\217^\303\t\221.6Fgrv\250\347Clcc\264\212\224\210\230!\336;\271Y\336!\355?\210\332$]\b\217l\372[\240\362O\261K'\234b\v\376",
> len=1460)
>
>     at /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/File.cc:405
>
> #9  0x0000000000b4c995 in file_analysis::FileReassembler::BlockInserted
> (this=0x10a0e400, start_block=0x11637850) at
> /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/FileReassembler.cc:66
>
> #10 0x0000000000b4cabd in file_analysis::FileReassembler::Undelivered
> (this=0x10a0e400, up_to_seq=70713344) at
> /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/FileReassembler.cc:96
>
> #11 0x0000000000806342 in Reassembler::TrimToSeq (this=0x10a0e400,
> seq=70713344) at /home/mfry/dev/bro24/bro-2.4.1/src/Reassem.cc:101
>
> #12 0x0000000000b4c879 in file_analysis::FileReassembler::Flush
> (this=0x10a0e400) at
> /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/FileReassembler.cc:33
>
> #13 0x0000000000b4b533 in file_analysis::File::DeliverChunk
> (this=0xb74e820, data=0xf0b2870
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
> len=1460, offset=1147283)
>
>     at /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/File.cc:421
>
> #14 0x0000000000b4b9dc in file_analysis::File::DataIn (this=0xb74e820,
> data=0xf0b2870
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
> len=1460, offset=1147283)
>
>     at /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/File.cc:481
>
> #15 0x0000000000b45722 in file_analysis::Manager::DataIn (this=0x246a440,
> data=0xf0b2870
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
> len=1460, offset=1147283, tag=...,
>
>     conn=0x10fb36f0, is_orig=false, precomputed_id=...) at
> /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/Manager.cc:121
>
> #16 0x00000000009bea07 in analyzer::http::HTTP_Entity::SubmitData
> (this=0xa3eb330, len=1460, buf=0xf0b2870
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y")
>
>     at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/http/HTTP.cc:322
>
> #17 0x0000000000a2140b in analyzer::mime::MIME_Entity::FlushData
> (this=0xa3eb330) at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/mime/MIME.cc:1217
>
> #18 0x0000000000a20a97 in analyzer::mime::MIME_Entity::DecodeDataLine
> (this=0xa3eb330, len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
>
>     trailing_CRLF=0) at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/mime/MIME.cc:1013
>
> #19 0x0000000000a1fe3c in analyzer::mime::MIME_Entity::NewDataLine
> (this=0xa3eb330, len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
> trailing_CRLF=0)
>
>     at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/mime/MIME.cc:698
>
> #20 0x0000000000a1fbd3 in analyzer::mime::MIME_Entity::Deliver
> (this=0xa3eb330, len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
> trailing_CRLF=0)
>
>     at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/mime/MIME.cc:617
>
> #21 0x00000000009be2cb in analyzer::http::HTTP_Entity::DeliverBodyClear
> (this=0xa3eb330, len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
>
>     trailing_CRLF=0) at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/http/HTTP.cc:217
>
> #22 0x00000000009be218 in analyzer::http::HTTP_Entity::DeliverBody
> (this=0xa3eb330, len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
> trailing_CRLF=0)
>
>     at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/http/HTTP.cc:205
>
> #23 0x00000000009be086 in analyzer::http::HTTP_Entity::Deliver
> (this=0xa3eb330, len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
> trailing_CRLF=0)
>
>     at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/http/HTTP.cc:159
>
> #24 0x0000000000a1fe1a in analyzer::mime::MIME_Entity::NewDataLine
> (this=0x12b37890, len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
> trailing_CRLF=0)
>
>     at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/mime/MIME.cc:693
>
> #25 0x0000000000a1fbd3 in analyzer::mime::MIME_Entity::Deliver
> (this=0x12b37890, len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
> trailing_CRLF=0)
>
>     at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/mime/MIME.cc:617
>
> #26 0x00000000009be2cb in analyzer::http::HTTP_Entity::DeliverBodyClear
> (this=0x12b37890, len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
>
>     trailing_CRLF=0) at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/http/HTTP.cc:217
>
> #27 0x00000000009be218 in analyzer::http::HTTP_Entity::DeliverBody
> (this=0x12b37890, len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
> trailing_CRLF=0)
>
>     at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/http/HTTP.cc:205
>
> #28 0x00000000009bddfe in analyzer::http::HTTP_Entity::Deliver
> (this=0x12b37890, len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
> trailing_CRLF=0)
>
>     at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/http/HTTP.cc:107
>
> #29 0x00000000009c43ea in analyzer::mime::MIME_Message::Deliver
> (this=0xbbf60b0, len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
> trailing_CRLF=0)
>
>     at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/mime/MIME.h:207
>
> #30 0x00000000009c10e0 in analyzer::http::HTTP_Analyzer::DeliverStream
> (this=0xaba9ed0, len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
>
>     is_orig=false) at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/http/HTTP.cc:908
>
> #31 0x0000000000b74c0b in analyzer::SupportAnalyzer::ForwardStream
> (this=0x96e9790, len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
> is_orig=false)
>
>     at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/Analyzer.cc:835
>
> #32 0x0000000000adcd94 in analyzer::tcp::ContentLine_Analyzer::DoDeliver
> (this=0x96e9790, len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y")
>
>     at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/tcp/ContentLine.cc:168
>
> #33 0x0000000000adcb44 in
> analyzer::tcp::ContentLine_Analyzer::DeliverStream (this=0x96e9790,
> len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
>
>     is_orig=false) at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/tcp/ContentLine.cc:108
>
> #34 0x0000000000b72383 in analyzer::Analyzer::NextStream (this=0x96e9790,
> len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
> is_orig=false)
>
>     at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/Analyzer.cc:245
>
> #35 0x0000000000b7235e in analyzer::Analyzer::NextStream (this=0xaba9ed0,
> len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
> is_orig=false)
>
>     at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/Analyzer.cc:239
>
> #36 0x0000000000b72784 in analyzer::Analyzer::ForwardStream
> (this=0x77aa210, len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y",
> is_orig=false)
>
>     at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/Analyzer.cc:331
>
> #37 0x0000000000adb93f in analyzer::tcp::TCP_Reassembler::Deliver
> (this=0xd8f75b0, seq=6422408, len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y")
>
>     at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/tcp/TCP_Reassembler.cc:457
>
> #38 0x0000000000adc279 in analyzer::tcp::TCP_Reassembler::DeliverBlock
> (this=0xd8f75b0, seq=6422408, len=1460, data=0xc15aab0
> "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y")
>
>     at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/tcp/TCP_Reassembler.cc:647
>
> #39 0x0000000000adb599 in analyzer::tcp::TCP_Reassembler::BlockInserted
> (this=0xd8f75b0, start_block=0x117aba30) at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/tcp/TCP_Reassembler.cc:393
>
> #40 0x0000000000806256 in Reassembler::NewBlock (this=0xd8f75b0,
> t=1459362263.205771, seq=6422408, len=1460, data=0x7ffb2bd72536 <Address
> 0x7ffb2bd72536 out of bounds>) at
> /home/mfry/dev/bro24/bro-2.4.1/src/Reassem.cc:73
>
> #41 0x0000000000adba48 in analyzer::tcp::TCP_Reassembler::DataSent
> (this=0xd8f75b0, t=1459362263.205771, seq=6422408, len=1460,
> data=0x7ffb2bd72536 <Address 0x7ffb2bd72536 out of bounds>, replaying=true)
>
>     at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/tcp/TCP_Reassembler.cc:492
>
> #42 0x0000000000ad9eb1 in analyzer::tcp::TCP_Endpoint::DataSent
> (this=0xbd1a7f0, t=1459362263.205771, seq=6422408, len=1460, caplen=1460,
> data=0x7ffb2bd72536 <Address 0x7ffb2bd72536 out of bounds>,
> ip=0x7fff27c7bd60, tp=0x7ffb2bd72522)
>
>     at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/tcp/TCP_Endpoint.cc:205
>
> #43 0x0000000000ad4eac in analyzer::tcp::TCP_Analyzer::DeliverData
> (this=0x77aa210, t=1459362263.205771, data=0x7ffb2bd72536 <Address
> 0x7ffb2bd72536 out of bounds>, len=1460, caplen=1460, ip=0x7fff27c7bd60,
> tp=0x7ffb2bd72522, endpoint=0xbd1a7f0, rel_data_seq=6422408,
>
>     is_orig=0, flags=...) at
> /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/tcp/TCP.cc:982
>
> #44 0x0000000000ad5f82 in analyzer::tcp::TCP_Analyzer::DeliverPacket
> (this=0x77aa210, len=1460, data=0x7ffb2bd72536 <Address 0x7ffb2bd72536 out
> of bounds>, is_orig=false, seq=18446744073709551615, ip=0x7fff27c7bd60,
> caplen=1460)
>
>     at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/tcp/TCP.cc:1382
>
> #45 0x0000000000b7229e in analyzer::Analyzer::NextPacket (this=0x77aa210,
> len=1480, data=0x7ffb2bd72522 <Address 0x7ffb2bd72522 out of bounds>,
> is_orig=false, seq=18446744073709551615, ip=0x7fff27c7bd60, caplen=1480)
>
>     at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/Analyzer.cc:222
>
> #46 0x000000000076c0c9 in Connection::NextPacket (this=0x10fb36f0,
> t=1459362263.205771, is_orig=0, ip=0x7fff27c7bd60, len=1480, caplen=1480,
> data=@0x7fff27c7b9a0: 0x7ffb2bd72522 <Address 0x7ffb2bd72522 out of
> bounds>, record_packet=@0x7fff27c7ba10: 1,
>
>     record_content=@0x7fff27c7ba14: 1, hdr=0x48d3bb8, pkt=0x7ffb2bd72500
> <Address 0x7ffb2bd72500 out of bounds>, hdr_size=14) at
> /home/mfry/dev/bro24/bro-2.4.1/src/Conn.cc:260
>
> #47 0x0000000000837be5 in NetSessions::DoNextPacket (this=0x5f2db20,
> t=1459362263.205771, hdr=0x48d3bb8, ip_hdr=0x7fff27c7bd60,
> pkt=0x7ffb2bd72500 <Address 0x7ffb2bd72500 out of bounds>, hdr_size=14,
> encapsulation=0x0)
>
>     at /home/mfry/dev/bro24/bro-2.4.1/src/Sessions.cc:760
>
> #48 0x00000000008361e9 in NetSessions::NextPacket (this=0x5f2db20,
> t=1459362263.205771, hdr=0x48d3bb8, pkt=0x7ffb2bd72500 <Address
> 0x7ffb2bd72500 out of bounds>, hdr_size=14) at
> /home/mfry/dev/bro24/bro-2.4.1/src/Sessions.cc:231
>
> #49 0x0000000000835ff7 in NetSessions::DispatchPacket (this=0x5f2db20,
> t=1459362263.205771, hdr=0x48d3bb8, pkt=0x7ffb2bd72500 <Address
> 0x7ffb2bd72500 out of bounds>, hdr_size=14, src_ps=0x48d3a70) at
> /home/mfry/dev/bro24/bro-2.4.1/src/Sessions.cc:187
>
> #50 0x00000000007f2082 in net_packet_dispatch (t=1459362263.205771,
> hdr=0x48d3bb8, pkt=0x7ffb2bd72500 <Address 0x7ffb2bd72500 out of bounds>,
> hdr_size=14, src_ps=0x48d3a70) at
> /home/mfry/dev/bro24/bro-2.4.1/src/Net.cc:281
>
> #51 0x0000000000b33fc2 in iosource::PktSrc::Process (this=0x48d3a70) at
> /home/mfry/dev/bro24/bro-2.4.1/src/iosource/PktSrc.cc:456
>
> #52 0x00000000007f2290 in net_run () at
> /home/mfry/dev/bro24/bro-2.4.1/src/Net.cc:329
>
> #53 0x000000000074815e in main (argc=18, argv=0x7fff27c7c628) at
> /home/mfry/dev/bro24/bro-2.4.1/src/main.cc:1190
>
> On Tue, Apr 5, 2016 at 10:37 AM Azoff, Justin S <jazoff at illinois.edu>
> wrote:
>
>> I've not seen that before.  Are you running any custom local scripts?
>>
>> Do you still have the core file that it dumped?  There's a good chance
>> gdb can be used to figure out what script triggered that assertion.
>>
>> https://www.bro.org/support/reporting-problems.html outlines the process
>> on how to use gdb against a bro core file.
>> --
>> - Justin Azoff
>>
>> > On Apr 5, 2016, at 10:04 AM, Michael Fry <michael.fry at morphick.com>
>> wrote:
>> >
>> > Hi All,
>> >
>> > Has anyone encountered the assertion failure below with Bro 2.4.1? This
>> is happening with live traffic at least a couple times a day since
>> upgrading from Bro 2.3.2. I didn't see anything the bug tracker, so thought
>> I'd float it here first.
>> >
>> > listening on zc:99 at 1, capture length 8192 bytes
>> >
>> > 1459786307.312525 processing suspended
>> > 1459786307.312525 processing continued
>> > 1459802619.911190 Failed to open GeoIP City database:
>> /usr/share/GeoIP/GeoIPCity.dat
>> > 1459802619.911190 Fell back to GeoIP Country database
>> > 1459802619.911190 Failed to open GeoIP Cityv6 database:
>> /usr/share/GeoIP/GeoIPCityv6.dat
>> > bro: /home/mfry/dev/bro24/bro-2.4.1/src/Tag.cc:72: EnumVal*
>> Tag::AsEnumVal(EnumType*) const: Assertion `type == 0 && subtype == 0'
>> failed.
>> > /opt/bro/share/broctl/scripts/run-bro: line 100: 11312 Aborted
>>        (core dumped) nohup ${pin_command} $pin_cpu "$mybro" "$@"
>> >
>> > Regards,
>> > Michael
>> >
>> > _______________________________________________
>> > Bro mailing list
>> > bro at bro-ids.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160629/1e78053c/attachment-0001.html 

From seth at icir.org  Wed Jun 29 19:57:43 2016
From: seth at icir.org (Seth Hall)
Date: Wed, 29 Jun 2016 22:57:43 -0400
Subject: [Bro] Assertion failure, Tag.cc
In-Reply-To: <CA+6KvzrQk0mmjObi37nBVuT1UuKABoYORddUxcxO9YdAdLQJYQ@mail.gmail.com>
References: <CA+6KvzoT30u0N85XUZb0SxbsLM0HZdnb2UVDBjNLCw=oRSrdig@mail.gmail.com>
	<F8A6A23F-862D-482A-B248-4A6825994E75@illinois.edu>
	<CA+6Kvzoc44iWzOf4ZpDi86PBdogFRbZ0wyn=YJuo47V1km9h3w@mail.gmail.com>
	<CA+6KvzrQk0mmjObi37nBVuT1UuKABoYORddUxcxO9YdAdLQJYQ@mail.gmail.com>
Message-ID: <36D97979-14C9-4BA6-988B-715A4B79E205@icir.org>


> On Jun 29, 2016, at 6:58 PM, Michael Fry <michael.fry at morphick.com> wrote:
> 
> I've been able to narrow down the circumstances where we see this core dump. In the dozens of times that I've seen it, the file being extracted is always delivered over HTTP via the BITS client. 

Could you send the script that you are using which causes this problem?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From maxime.lambert at insa-cvl.fr  Thu Jun 30 01:19:34 2016
From: maxime.lambert at insa-cvl.fr (Maxime Lambert)
Date: Thu, 30 Jun 2016 10:19:34 +0200 (CEST)
Subject: [Bro] IP <-> MAC Address
In-Reply-To: <1511952208.35505.1467273712665.JavaMail.zimbra@insa-cvl.fr>
Message-ID: <1938974012.42837.1467274774480.JavaMail.zimbra@insa-cvl.fr>

Hi everyone, 

Is there a way to obtain the source and/or destination mac address from a connection record ? 

I've been looking through the scripts roam.bro, known-devices.bro and known-devices-and-hostnames.bro, but I'am not sure how it works. I'm wondering it I missed something. I've this 2 files : 

cat /opt/bro/logs/current/ known_devices.log 
#separator \x09 
#set_separator? , 
#empty_field??? (empty) 
#unset_field??? - 
#path?? known_devices 
#open?? 2016-06-30-09-08-33 
#fields ts????? mac???? dhcp_host_name 
#types? time??? string? string 
1467260412.707446?????? 00:11:22:33:44:55?????? android-684541321657432 
1467260416.339490?????? 00:11:22:33:44:66?????? android-213857946354179 
1467260447.207524?????? 00:11:22:33:44:77????? iPhone-XXXX 
1467261341.099450?????? 00:11:22:33:44:88????? iPhone -YYYY 
1467271833.863474?????? 00:11:22:33:44:99????? iPhone -ZZZZ 
1467272311.523445?????? 00:11:22:33:44:00?????  bitcoin-computer 
1467272443.463545?????? 00:11:22:33:44:11????? iPhone-UUUU 
1467272517.623516?????? 00:11:22:33:44:22????? iPhone-TTTT 
1467272692.387523?????? 00:11:22:33:44:33????? iPhone-VVVV 
1467273783.775451?????? 00:11:22:33:44:44????? SDM-00239 
1467273899.667460?????? 00:11:22:33:33:55????  iPhone-AAAA 

AND 

cat /opt/bro/logs/current/ dhcp.log 
#separator \x09 
#set_separator? , 
#empty_field??? (empty) 
#unset_field??? - 
#path?? dhcp 
#open?? 2016-06-30-09-51-23 
#fields ts????? uid???? id.orig_h?????? id.orig_p?????? id.resp_h?????? id.resp_p?????? mac???? assigned_ip???? lease_time????? trans_id 
#types? time??? string? addr??? port??? addr??? port??? string? addr??? interval??????? count 
1467264083.815462?????? C4jfqvVzuapDS69dz ???? 255.255.255.255 68????? 192.X.X.X???? 67 XX:XX:XX:XX:XX:XX 192.X.X.X 86400.000000??? 146703799 
1467264083.815462 C4jfqvVzuapDS69dz 255.255.255.255 68????? 192.X.X.X???? 67 XX:XX:XX:XX:XX:XX 192.X.X.X 86400.000000??? 146703799 
1467264083.815462 C4jfqvVzuapDS69dz 255.255.255.255 68????? 192.X.X.X???? 67 XX:XX:XX:XX:XX:XX 192.X.X.X 86400.000000??? 146703799 
1467264083.815462 C4jfqvVzuapDS69dz 255.255.255.255 68????? 192.X.X.X???? 67 XX:XX:XX:XX:XX:XX 192.X.X.X 86400.000000??? 146703799 


So what I want is a fusion between dhcp.log and know-devices.log to know to each devices their IP and MAC address. 


I think bro must monitor ARP Request/Response to an output log file like this : 

ts???????????????????????????????????? string (Mac Addr)????????????? string (hostname)??????????????????????? string (IP Addr) 
1467260401.707446?????? XX:XX:XX:XX:XX:XX?????? ?Android-XXXXXXXXXXX??????????? X.X.X.X 
Or maybe anyone has another solution ? 

Thank you for your help. 


Maxime Lambert 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160630/91f1b343/attachment.html 

From seth at broala.com  Thu Jun 30 07:28:11 2016
From: seth at broala.com (Seth Hall)
Date: Thu, 30 Jun 2016 10:28:11 -0400
Subject: [Bro] IP <-> MAC Address
In-Reply-To: <1938974012.42837.1467274774480.JavaMail.zimbra@insa-cvl.fr>
References: <1938974012.42837.1467274774480.JavaMail.zimbra@insa-cvl.fr>
Message-ID: <5161DDCB-60E7-4A6F-8F3C-2E0B797BB099@broala.com>


> On Jun 30, 2016, at 4:19 AM, Maxime Lambert <maxime.lambert at insa-cvl.fr> wrote:
> 
> Is there a way to obtain the source and/or destination mac address from a connection record ?

This a feature in the next release of Bro.  If you use git master, you can load the "policy/protocols/conn/mac-logging.bro" script and it will add MAC address fields to your conn log.

  .Seth

--
Seth Hall * Broala * seth at broala.com * www.broala.com



From tara.salman at wustl.edu  Thu Jun 30 10:30:46 2016
From: tara.salman at wustl.edu (Salman, Tara)
Date: Thu, 30 Jun 2016 17:30:46 +0000
Subject: [Bro] number of connections to the same port in 100 connections
Message-ID: <45FE0E1E-9A6A-429D-B051-417BB0213C4E@wustl.edu>

Hi everyone, 
I am trying to find the number of connections having the same source ip and destination port in the last 100 connection using bro commands 
I managed to get the number in all connections using: 
bro-cut id.orig_h id.orgi_p < conn.log | sort| uniq -c| sort -rn  

which is working fine but i need to modify this to include only the last 100 connections in the log file. is there a way to do that ? 

thanks in advance 

From jazoff at illinois.edu  Thu Jun 30 11:03:21 2016
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 30 Jun 2016 18:03:21 +0000
Subject: [Bro] number of connections to the same port in 100 connections
In-Reply-To: <45FE0E1E-9A6A-429D-B051-417BB0213C4E@wustl.edu>
References: <45FE0E1E-9A6A-429D-B051-417BB0213C4E@wustl.edu>
Message-ID: <C80D8AD1-7C2E-4ED7-A553-891EB89885D7@illinois.edu>


> On Jun 30, 2016, at 1:30 PM, Salman, Tara <tara.salman at wustl.edu> wrote:
> 
> Hi everyone, 
> I am trying to find the number of connections having the same source ip and destination port in the last 100 connection using bro commands 
> I managed to get the number in all connections using: 
> bro-cut id.orig_h id.orgi_p < conn.log | sort| uniq -c| sort -rn  
> 
> which is working fine but i need to modify this to include only the last 100 connections in the log file. is there a way to do that ? 
> 
> thanks in advance 

Give this a try:

(head -n 8 conn.log ;tail -n 100 conn.log ) | bro-cut id.orig_h id.orig_p  | sort| uniq -c| sort -rn


you need the first 8 lines for the header so bro-cut works.
-- 
- Justin Azoff