From seth at icir.org  Wed Mar  1 08:38:23 2017
From: seth at icir.org (Seth Hall)
Date: Wed, 1 Mar 2017 11:38:23 -0500
Subject: [Bro-Dev] Bro packages with plugins
Message-ID: <B96DA1FB-729B-4881-82A1-0DDE87B2ED13@icir.org>

I was emailing with Justin this morning and he reminded me that we still have the issue with people that install Bro from binary packages and their inability to install packages with binary plugins.  I don't know if I'll be able to kick start a brief discussion on this right now or not, but I'll throw a couple of ideas out there to see if any stick.

 - Include header files with the binary package installs. 
    PROS
      - Solves the problem.  Package users could install plugins.
    CONS
      - "Feels" nasty since we don't have a fully formalized API and we'd just have to include all Bro header files.
      - Requires users to have a compiler.


 - Create mechanism to release binary plugins in the bro package manager.
   PROS
    - Super easy installation for users.
   CONS
    - Here be (library linking) dragons. I don't know how many we could avoid with static linking.
    - We would probably have to create more project infrastructure to build packages.

Other thoughts?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From jdopheid at illinois.edu  Thu Mar  2 09:10:19 2017
From: jdopheid at illinois.edu (Dopheide, Jeannette M)
Date: Thu, 2 Mar 2017 17:10:19 +0000
Subject: [Bro-Dev] =?utf-8?q?BroCon_=E2=80=9917=3A_Registration_is_open!?=
Message-ID: <876DB55E-1A3A-4CEC-BB48-C75F1BEB02BD@illinois.edu>

Bro Community,

BroCon ?17 will occur on Tuesday, September 12th - Thursday, September 14th at the National Center for Supercomputing Applications in Urbana, IL.

See our event page:
https://www.bro.org/community/brocon2017.html

Early bird registration is open! CFP is open! Don't forget to book your hotel.

Interested in sponsoring BroCon? Contact us at info at bro.org for more information.

Thank you for your continued support, and see you in September!

Regards,
The Bro Project

------
Jeannette Dopheide
Training and Outreach Coordinator
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
 



From jsiwek at illinois.edu  Thu Mar  2 09:19:24 2017
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Thu, 2 Mar 2017 17:19:24 +0000
Subject: [Bro-Dev] Bro packages with plugins
In-Reply-To: <B96DA1FB-729B-4881-82A1-0DDE87B2ED13@icir.org>
References: <B96DA1FB-729B-4881-82A1-0DDE87B2ED13@icir.org>
Message-ID: <DBEB5D73-A640-4972-8B48-6EA8942BFB35@illinois.edu>


> On Mar 1, 2017, at 10:38 AM, Seth Hall <seth at icir.org> wrote:
> 
> I was emailing with Justin this morning and he reminded me that we still have the issue with people that install Bro from binary packages and their inability to install packages with binary plugins.

Is there something that makes this currently impossible or just that the user would additionally need to download Bro source code and tell bro-pkg where to find it before installing a package where the build_command will build a plugin?

> - Create mechanism to release binary plugins in the bro package manager.

I wonder if pre-built plugins can?t already be distributed in packages?  E.g. put the pre-built plugin in the package, specify its location via plugin_dir, and leave the build_command blank.  Then after the package is installed, I think Bro should automatically try to load the plugin, provided the architecture matches.

In other words, I think bro-pkg is already set up to handle distributing pre-built binary plugins, but the more difficult problems to address will relate to how to actually produce plugins that will be binary-compatible w/ a user?s Bro installation.  E.g. lack of stable/standard ABI in Bro and C++ itself means there?s myriad ways to produce unportable binaries.

The only other idea I have, is if bro-pkg is not able to locate Bro source code (i.e. the user installed Bro from a binary package), then it could automatically download the appropriate version (e.g. clone Bro?s git repo and checkout a specific version tag).

- Jon


From commike at reservoir.com  Thu Mar  2 09:46:17 2017
From: commike at reservoir.com (Alan Commike)
Date: Thu, 02 Mar 2017 17:46:17 +0000
Subject: [Bro-Dev] Bro packages with plugins
In-Reply-To: <DBEB5D73-A640-4972-8B48-6EA8942BFB35@illinois.edu>
References: <B96DA1FB-729B-4881-82A1-0DDE87B2ED13@icir.org>
	<DBEB5D73-A640-4972-8B48-6EA8942BFB35@illinois.edu>
Message-ID: <CAMnzEaiB_qgBhWGGZJBqwc+itQFDCJqdoGs4-ArjebwDp51ibQ@mail.gmail.com>

On Thu, Mar 2, 2017 at 9:19 AM Siwek, Jon <jsiwek at illinois.edu> wrote:

> In other words, I think bro-pkg is already set up to handle distributing
> pre-built binary plugins, but the more difficult problems to address will
> relate to how to actually produce plugins that will be binary-compatible w/
> a user?s Bro installation.  E.g. lack of stable/standard ABI in Bro and C++
> itself means there?s myriad ways to produce unportable binaries.
>

This, I think, is the big gotcha. Different linux distros, different
compilers, different libraries. It even goes into which version of boost is
used. There's also matching to the version of Bro whether that's master or
release N.

I'm trying to come up with parallels with the Linux kernel. Loadable
modules have vermagic and there's DKMS. Neither fully covers the breadth of
issues that can occur.


>
> The only other idea I have, is if bro-pkg is not able to locate Bro source
> code (i.e. the user installed Bro from a binary package), then it could
> automatically download the appropriate version (e.g. clone Bro?s git repo
> and checkout a specific version tag).
>

>From a deployment perspective, there really shouldn't be a compiler on the
system running a deployed Bro. Even further, the system might be limited on
where on the internet it can reach. At least with many of our customers,
that's the case.

   ...alan




>
> - Jon
>
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20170302/5e1b8254/attachment.html 

From jsiwek at illinois.edu  Thu Mar  2 16:11:35 2017
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Fri, 3 Mar 2017 00:11:35 +0000
Subject: [Bro-Dev] Bro packages with plugins
In-Reply-To: <CAMnzEaiB_qgBhWGGZJBqwc+itQFDCJqdoGs4-ArjebwDp51ibQ@mail.gmail.com>
References: <B96DA1FB-729B-4881-82A1-0DDE87B2ED13@icir.org>
	<DBEB5D73-A640-4972-8B48-6EA8942BFB35@illinois.edu>
	<CAMnzEaiB_qgBhWGGZJBqwc+itQFDCJqdoGs4-ArjebwDp51ibQ@mail.gmail.com>
Message-ID: <E016A69D-D8DF-413E-9029-655B631CAFD2@illinois.edu>


> On Mar 2, 2017, at 11:46 AM, Alan Commike <commike at reservoir.com> wrote: 
> 
> The only other idea I have, is if bro-pkg is not able to locate Bro source code (i.e. the user installed Bro from a binary package), then it could automatically download the appropriate version (e.g. clone Bro?s git repo and checkout a specific version tag).
> 
> From a deployment perspective, there really shouldn't be a compiler on the system running a deployed Bro. Even further, the system might be limited on where on the internet it can reach. At least with many of our customers, that's the case. 

I imagine bro-pkg could still use its bundling system for when the Bro nodes have limited network/compiler functionality.  E.g package management and building occurs on a dedicated host that does have network/compiler access and then a bundle file can be copied over to the Bro nodes to reproduce the desired package ecosystem.

- Jon


From asharma at lbl.gov  Fri Mar  3 13:36:06 2017
From: asharma at lbl.gov (Aashish Sharma)
Date: Fri, 3 Mar 2017 13:36:06 -0800
Subject: [Bro-Dev] [desired broker api as oppose to whats in known-hosts.bro]
Message-ID: <20170303213605.GE14819@mac-822.local>

SO I came across a sample of Broker-API usage:

when (local res = Broker::exists(Cluster::cluster_store, Broker::data("known_hosts")))
	{
       		local res_bool = Broker::refine_to_bool(res$result);
		if(res_bool)
		{
		when ( local res2 = Broker::lookup(Cluster::cluster_store, Broker::data("known_hosts")) )
			{
			local res2_bool = Broker::set_contains(res2$result, Broker::data(host));

			if(!res2_bool)
				{
				Broker::add_to_set(Cluster::cluster_store, Broker::data("known_hosts"), Broker::data(host));
				Log::write(Known::HOSTS_LOG, [$ts=network_time(), $host=host]);
				}
			}
		timeout 10sec
                	{ print "timeout"; }
                 }
             }
             timeout 20sec
             { print "timeout"; }


Now this isn't too complicated but I find it cumbersome and one needs to understand execution flow since "when" is involved etc etc etc. 

====

Here is how I'd envision broker usage (and I know easier said than done...)

define: 

global known_hosts: table[addr] of blah &store ; 

now 

when I query the table :
	if (addr in known_hosts)
	{  
	....
	} 

	
1a) Given &store directive above, Bro should go and check store if value isn't already in the table and update as needed in background - if value isn't in the store, if condition would fail anyways. 

may be 	
1b) You can probably maintain a bloomfilter which builds itself from whats out there in table and works as an index or a membership check 

and 

1c) We also need another directive something akin to "&expire_store = 1 hrs", which expires entires from the table in memory and puts into the store. 

on implementation side I am pretty sure there are complexities since broker model is different and I don't grasp it yet. 

But from current bro scripting prespective: 

(1a) will be If value isn't in the bloomfilter, call a Input::Event (which reads the data from the store) and fire end-of-data like event or similiar to input-framework give me a capability to fire events when data read from store is complete. So this would eliminate the "when" construct and give a much clearer event based code execution path (from scripters prespective) 

for (1c) Likewise, for &expire_store - I am merely using &(read|write|create)_expire functions to write to the database. 

So in-summary we need a directive &store which works with sets, tables and bro data types. 

If value is in member OK else broker in background goes and checks store and in background updates sets, tables, datastructures and then fires events when done. 

Save me from all the above "when" constructs and Broker::lookup routines. Let those be in background. 

I'd be happy to talk in person or video conference if more clearification is needed. 

Thanks, 
Aashish 



From jazoff at illinois.edu  Fri Mar  3 16:21:16 2017
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Sat, 4 Mar 2017 00:21:16 +0000
Subject: [Bro-Dev] [desired broker api as oppose to whats in
 known-hosts.bro]
In-Reply-To: <20170303213605.GE14819@mac-822.local>
References: <20170303213605.GE14819@mac-822.local>
Message-ID: <A92CBE6E-9474-4C4D-B7C3-6EFFC63C08A6@illinois.edu>


> On Mar 3, 2017, at 4:36 PM, Aashish Sharma <asharma at lbl.gov> wrote:
> 
> SO I came across a sample of Broker-API usage:

Yeah.. there's a lot of things wrong with how that is being done.  There are a few things going on here.

One is that &synchronized is no longer functions.  I think we should bring this back, it may not be in the form of &synchronized, but at least some way to create a simple data structure that is automatically kept in sync between nodes.

The other is that the api that known hosts is currently using is too high level:

Broker::exists(Cluster::cluster_store, Broker::data("known_hosts"))
Broker::lookup(Cluster::cluster_store, Broker::data("known_hosts"))
Broker::set_contains(res2$result, Broker::data(host))
Broker::add_to_set(Cluster::cluster_store, Broker::data("known_hosts"), Broker::data(host));

which in english is:

1. see if the known_hosts table exists (why would it not exist?)
2. transfer the entire known_hosts table over from the data node
3. see if it contains host
4. add host if not present

And (due to probably an oversight), it does this twice resulting in the known_hosts table being transferred twice.

This would work a lot better if it keep a persistent copy of the known_hosts set between calls and only updated it from the data node if the host wasn't found.  The only downside there is that the entire table is still being copied between nodes instead of just updates.


To accomplish what known hosts really needs, which is just "Have I seen this host before", we could just do something like:

local added = Broker::add_to_set(Cluster::cluster_store, Broker::data("known_hosts"), Broker::data(host));
if(added) {
    # host did not previously exist in the set



The only problem in this case is there is no local cache to prevent the same host from being checked multiple times.  That would require a local copy of the set, or like you said, a bloomfilter of sorts (probably one of those reverse bloomfilters that has false negatives but not false positives).

So, for the case of tracking things using a set across the cluster all one needs is a simple function that:

Checks to see if the item is in the local cache or bloom filter
Sends it over to the data node and inspects the response (new or duplicate)

Things get a little more complicated in that I want the ability to scale out the data nodes.  So that means the slight variation:

Checks to see if the item is in the local cache or bloom filter
Sends it over to the data node that corresponds to the hash of the item and inspects the response (new or duplicate)

so from a users point of view, the Broker part of the function could just be

if(Broker::check_or_add_to_set("known_hosts", host)) {
    Log::write(Known::HOSTS_LOG, [$ts=network_time(), $host=host]);
}

Another way of writing this, which corresponds to your 'event based' approach is to just have the function instead do:

Check to see if the item is in the local cache or bloom filter
Send an event over to the data node that says a new host was potentially found.

For known hosts purposes, the data node doesn't even need to send anything back to the worker, it can just log it(or not).

It would help me think about this if you could outline some of your use cases for broker stores.  I have a good idea of what needs to be done to fix known hosts/services/certs and sumstats/scan detection.  But I don't know what things you have in mind :-)





-- 
- Justin Azoff



From robin at icir.org  Mon Mar  6 08:43:34 2017
From: robin at icir.org (Robin Sommer)
Date: Mon, 6 Mar 2017 08:43:34 -0800
Subject: [Bro-Dev] [desired broker api as oppose to whats in
 known-hosts.bro]
In-Reply-To: <A92CBE6E-9474-4C4D-B7C3-6EFFC63C08A6@illinois.edu>
References: <20170303213605.GE14819@mac-822.local>
	<A92CBE6E-9474-4C4D-B7C3-6EFFC63C08A6@illinois.edu>
Message-ID: <20170306164334.GF87782@icir.org>



On Sat, Mar 04, 2017 at 00:21 +0000, you wrote:

> Broker::exists(Cluster::cluster_store, Broker::data("known_hosts"))
> Broker::lookup(Cluster::cluster_store, Broker::data("known_hosts"))
> Broker::set_contains(res2$result, Broker::data(host))
> Broker::add_to_set(Cluster::cluster_store, Broker::data("known_hosts"), Broker::data(host));

The first step will be getting this down to (some version of) this
form instead:

    Broker::exists(Cluster::cluster_store, "known_hosts")
    result = Broker::lookup(Cluster::cluster_store, "known_hosts")
    Broker::set_contains(result, host)
    Broker::add_to_set(Cluster::cluster_store, "known_hosts"), host);

This should be relatively straight-forward to achieve once we have the
changes in that Matthias is currently working on.

And hopefully we can then also get rid of the "when" statement for
using these.

As reminder, this is the current proposal for next Broker steps:

    https://www.bro.org/development/projects/broker-lang-ext.html

Whether we want to go further, like with Aashish's &store proposal, we
can see then, I'm still a bit undecided there. Either way, a better
Bro-side UI for Broker is coming, we just need to get the various
pieces in place first that people are working on currently before we
can move forward.

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From seth at icir.org  Mon Mar  6 10:24:49 2017
From: seth at icir.org (Seth Hall)
Date: Mon, 6 Mar 2017 13:24:49 -0500
Subject: [Bro-Dev] [desired broker api as oppose to whats in
 known-hosts.bro]
In-Reply-To: <20170306164334.GF87782@icir.org>
References: <20170303213605.GE14819@mac-822.local>
	<A92CBE6E-9474-4C4D-B7C3-6EFFC63C08A6@illinois.edu>
	<20170306164334.GF87782@icir.org>
Message-ID: <CFAF8A84-8BF4-4E0B-869D-F7B64493A050@icir.org>


> On Mar 6, 2017, at 11:43 AM, Robin Sommer <robin at icir.org> wrote:
> 
> Whether we want to go further, like with Aashish's &store proposal, we
> can see then, I'm still a bit undecided there. Either way, a better
> Bro-side UI for Broker is coming, we just need to get the various
> pieces in place first that people are working on currently before we
> can move forward.

I'm playing with rewriting some of the scripts for 2.6 right now with events because I think that the broker api is too complicated and could potentially have too many side effects. (like overuse of "when" for instance).

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From robin at icir.org  Mon Mar  6 10:31:57 2017
From: robin at icir.org (Robin Sommer)
Date: Mon, 6 Mar 2017 10:31:57 -0800
Subject: [Bro-Dev] [desired broker api as oppose to whats in
 known-hosts.bro]
In-Reply-To: <CFAF8A84-8BF4-4E0B-869D-F7B64493A050@icir.org>
References: <20170303213605.GE14819@mac-822.local>
	<A92CBE6E-9474-4C4D-B7C3-6EFFC63C08A6@illinois.edu>
	<20170306164334.GF87782@icir.org>
	<CFAF8A84-8BF4-4E0B-869D-F7B64493A050@icir.org>
Message-ID: <20170306183157.GH87782@icir.org>



On Mon, Mar 06, 2017 at 13:24 -0500, you wrote:

> I'm playing with rewriting some of the scripts for 2.6 right now with
> events because I think that the broker api is too complicated and
> could potentially have too many side effects. (like overuse of "when"
> for instance).

Yeah, that makes sense. We can switch back once we have something
better.

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From jazoff at illinois.edu  Wed Mar  8 16:26:39 2017
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 9 Mar 2017 00:26:39 +0000
Subject: [Bro-Dev] Scaling out bro cluster communication
In-Reply-To: <79E2D8B3-AA8D-4018-A172-4CB91DDEB13C@illinois.edu>
References: <1FFA7E0D-A383-4E96-8C6A-5406B73414B4@illinois.edu>
	<20170210164956.GB18262@samurai.icir.org>
	<79E2D8B3-AA8D-4018-A172-4CB91DDEB13C@illinois.edu>
Message-ID: <E3BE9F72-9BD7-410D-A7F6-C1650919B968@illinois.edu>

Hi,

I was able to put together of a prototype of the functionality I had in mind.

I learned a bit more about broker including the Broker::send_event function vs. the auto_event method.

send_event doesn't let you pick the node you want to send data to, but it does let you pick the queue.  By creating multiple nodes and subscribing each node to it's own queue I was able to achieve the end result I wanted.  It boiled down to this:


function send_event_hashed(key: any, args: Broker::EventArgs)
{
    local destination_count = node_count; #FIXME: how to figure out dynamically
    local dest = 1+ md5_hash_count(key) % destination_count;
    local queue = fmt("bro/data/%s", dest);
    print fmt("Send hash(%s)=%s: %s", key, queue, args);
    Broker::send_event(queue, args);
}



I have the full example here https://github.com/JustinAzoff/broker_distributed_events

It implements a fake known hosts and scan detection policy.

the main things to figure out is:

* How to work out the proper node_count at runtime.  I think on a real bro cluster the Cluster namespace has the data I need for this, including which nodes are reachable.

* How to handle one node becoming unreachable or a new node showing up.  Ideally bro would use a form of consistent ring hashing.

If this were worked out, and implemented for logging as well, you could run a bro cluster with 2 'manager' nodes and have a fully functioning cluster even if one of them died.

As is, I can probably use this on our test cluster to run 4 data nodes and distribute scan detection to 4 cpu cores.


The example doesn't show it, but for things like the known hosts tracking it would be useful if the data could be replicated to the other data nodes.  Because the sender-side hash based distribution also acts to de-duplicate the data, the replication would not be latency sensitive. It would not have the problem that the current known hosts policy has where 2 nodes can detect and log a new host before the data synchronizes.  As long as the data replicated before a node outage occurred, you would get consistent logs.






-- 
- Justin Azoff



From jazoff at illinois.edu  Fri Mar 10 13:57:35 2017
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Fri, 10 Mar 2017 21:57:35 +0000
Subject: [Bro-Dev] Scaling out bro cluster communication
In-Reply-To: <E3BE9F72-9BD7-410D-A7F6-C1650919B968@illinois.edu>
References: <1FFA7E0D-A383-4E96-8C6A-5406B73414B4@illinois.edu>
	<20170210164956.GB18262@samurai.icir.org>
	<79E2D8B3-AA8D-4018-A172-4CB91DDEB13C@illinois.edu>
	<E3BE9F72-9BD7-410D-A7F6-C1650919B968@illinois.edu>
Message-ID: <26A05F21-3125-4E54-8F26-50C0B066E80C@illinois.edu>

> 
> It implements a fake known hosts and scan detection policy.
> 
> the main things to figure out is:
> 
> * How to work out the proper node_count at runtime.  I think on a real bro cluster the Cluster namespace has the data I need for this, including which nodes are reachable.
> 
> * How to handle one node becoming unreachable or a new node showing up.  Ideally bro would use a form of consistent ring hashing.
> 
> If this were worked out, and implemented for logging as well, you could run a bro cluster with 2 'manager' nodes and have a fully functioning cluster even if one of them died.
> 
> As is, I can probably use this on our test cluster to run 4 data nodes and distribute scan detection to 4 cpu cores.

I got this to work on a real cluster running the broker integration branch!

https://github.com/JustinAzoff/broker_distributed_events/blob/master/distributed_broker.bro

Took me a while to figure out that a change somewhere made Broker::publish_topic required to send messages to other nodes, and that to raise notices on a datanode bro needs:

redef datanode2manager_events += {"Notice::cluster_notice"};

The main screwy thing is that I couldn't figure out a consistent way to enumerate the data nodes[1], so I just hardcoded the index in the name:

[datanode-0]
type=datanode
host=bro-test

[datanode-1]
type=datanode
host=bro-test

etc.

Once I had the base functions in place I changed my scan.bro from

    #this is a worker2manager event
    event Scan::scan_attempt(scanner, attempt);

to

    local args = Broker::event_args(scan_attempt, scanner, attempt);
    Cluster::send_event_hashed(scanner, args);

And.. it just worked.  I have 8 data nodes running that are each handling 1/8th of the scan detection aggregation.


-- 
- Justin Azoff


[1] They are stored in a set, and the 'first' data node on different workers was not always the same... would need to convert the set of data nodes to a list and sort it.

From robin at icir.org  Fri Mar 24 11:41:48 2017
From: robin at icir.org (Robin Sommer)
Date: Fri, 24 Mar 2017 11:41:48 -0700
Subject: [Bro-Dev] Broctld design
Message-ID: <20170324184148.GC29819@icir.org>

Summarizing some recent discussions, I've put together a high-level
design proposal for "broctld", the future BroControl daemon:

    https://www.bro.org/development/projects/broctld.html

Feedback welcome.

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From jdopheid at illinois.edu  Fri Mar 31 09:38:16 2017
From: jdopheid at illinois.edu (Dopheide, Jeannette M)
Date: Fri, 31 Mar 2017 16:38:16 +0000
Subject: [Bro-Dev] login change to bro-tracker JIRA account
Message-ID: <71B1F593-9938-45D9-973A-C6B3BD2A1B95@illinois.edu>

Hello Bro-Dev,

Atlassian hosts our JIRA site (Bro Issue Tracker) and will be making some changes to their account management system in April. The changes will allow users to log into any Atlassian Cloud product with the same email address. 

What does this mean for you? Once the changes are active you will be asked to verify your email address the next time you log into Bro-Tracker and will continue logging in with that email address. More information about these changes and how they impact users can be found here:

https://confluence.atlassian.com/cloud/atlassian-account-for-administrators-873871203.html

Please let us know if you have any questions. 

Thanks,

Jeannette


------
Jeannette Dopheide
Training and Outreach Coordinator
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign