From vern at corelight.com  Wed Sep  5 03:33:21 2018
From: vern at corelight.com (Vern Paxson)
Date: Wed, 05 Sep 2018 03:33:21 -0700
Subject: [Bro-Dev] attributes & named types
Message-ID: <201809051033.w85AXLN6019525@fruitcake.ICSI.Berkeley.EDU>

For some scripting I'm doing, I'm running into the problem that
attributes associated with named types don't get propagated.  For
example,

	type a: table[count] of count &default = 5;
	global c: a;
	print c[9];

complains that c[9] isn't set, instead of returning the value 5.

Having dived[*] into this and examined some potential fixes, I've identified
that the problem is that (1) the attribute "&default = 5" in the above
example gets associated with the identifier 'a' rather than with a's type,
and (2) when the parser processes the second line, early in the process 'a'
gets converted to its underlying type, with the attributes lost at that
point since, internally, BroType's do not have attributes.

This is a pain to deal with.  If we simply add attributes to BroType's and
for statements like the first line associate the attributes with the type,
then a sequence like:

	type a: table[count] of count &default = 5;
	type b: a &read_expire = 1 min;

will wind up changing the attributes associated with 'a' even though it
really shouldn't.

I think the right solution for this is to introduce a new BroType called
a NamedType.  A NamedType has an identifier, an underlying BroType, and a
set of attributes.  When a NamedType is constructed, for its attributes
it combines both the explicit attributes in its declaration (like the
&read_expire for 'b' above) and the implicit (i.e., it inherits/copies the
&default from 'a').

I plan to implement this soon, so please speak up if you have thoughts.

		Vern


[*] The dive also exposed some other bugs in attribute processing, which
    I'll enter into the tracker shortly.

From jsiwek at corelight.com  Wed Sep  5 15:35:57 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Wed, 5 Sep 2018 17:35:57 -0500
Subject: [Bro-Dev] Bro 2.6-beta plans
Message-ID: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>

There's no significant code changes/features planned to get added to
the master branch from now until the 2.6-beta gets released (maybe in
about a week).  Until that happens, please help test the latest master
branch and provide any feedback about how it's working if you can.

- Jon

From dopheide at es.net  Wed Sep  5 15:43:17 2018
From: dopheide at es.net (Michael Dopheide)
Date: Wed, 5 Sep 2018 17:43:17 -0500
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
Message-ID: <CAFfiZAWO10YobDiPgtK9r6u=QP9es5c7JqXDww7_NQbJt3A65A@mail.gmail.com>

To be clear, Cluster::relay_rr() is gone forever?  I?ll need to rewrite
some policies, but also update the blog to be correct.

Dop

On Wed, Sep 5, 2018 at 5:37 PM Jon Siwek <jsiwek at corelight.com> wrote:

> There's no significant code changes/features planned to get added to
> the master branch from now until the 2.6-beta gets released (maybe in
> about a week).  Until that happens, please help test the latest master
> branch and provide any feedback about how it's working if you can.
>
> - Jon
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180905/c6a5f19c/attachment.html 

From vern at corelight.com  Wed Sep  5 16:40:18 2018
From: vern at corelight.com (Vern Paxson)
Date: Wed, 05 Sep 2018 16:40:18 -0700
Subject: [Bro-Dev] attributes & named types
In-Reply-To: <201809051033.w85AXLN6019525@fruitcake.ICSI.Berkeley.EDU> (Wed,
	05 Sep 2018 03:33:21 PDT).
Message-ID: <201809052340.w85NeIh9012559@fruitcake.ICSI.Berkeley.EDU>

> I think the right solution for this is to introduce a new BroType called
> a NamedType.  A NamedType has an identifier, an underlying BroType, and a
> set of attributes.  When a NamedType is constructed ...

Huh, turns out there's already a "TypeType", which is the equivalent.
All I need to do, I believe, is allow these to have attributes.

		Vern

From jsiwek at corelight.com  Thu Sep  6 07:52:49 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 6 Sep 2018 09:52:49 -0500
Subject: [Bro-Dev] attributes & named types
In-Reply-To: <201809052340.w85NeIh9012559@fruitcake.ICSI.Berkeley.EDU>
References: <201809051033.w85AXLN6019525@fruitcake.ICSI.Berkeley.EDU>
	<201809052340.w85NeIh9012559@fruitcake.ICSI.Berkeley.EDU>
Message-ID: <CAMzgZ0KUKsx0kyKGvtpnpCrH+mO8S+jnDJXjPnT+RHedNdFP-A@mail.gmail.com>

On Wed, Sep 5, 2018 at 6:47 PM Vern Paxson <vern at corelight.com> wrote:
>
> > I think the right solution for this is to introduce a new BroType called
> > a NamedType.  A NamedType has an identifier, an underlying BroType, and a
> > set of attributes.  When a NamedType is constructed ...
>
> Huh, turns out there's already a "TypeType", which is the equivalent.
> All I need to do, I believe, is allow these to have attributes.

My recollection is TypeType is the type that a value gets when the
stored-value is actually just a type and doesn't quite fit what's
needed here.

Take the original example:

        type a: table[count] of count &default = 5;
        type b: a &read_expire = 1 min;

Declaring a variable of type 'a' or type 'b' doesn't mean that the
values stored in that variable are types themselves, they're still
just storing values of the table-type (but with different attributes
depending on 'a' or 'b').

Other ideas I'm having for solving the overall problem are:

(1) 'a' and 'b' need to actually be distinct BroType's.  Instead of
'b' being a reference/alias to 'a' with extended attributes, just
create it as it's own BroType by first cloning 'a', then adding any
additional attributes.

(2) BroType could somehow store a mapping of identifier -> attributes
so that on declaring a variable of either 'a' or 'b', we can choose
which attributes apply.  But this is relying on the idea that you have
to push the desired attributes into each new Val because you can't
mutate the underlying the BroType since multiple Vals of either type
'a' or 'b' are going to be referencing it.

- Jon

From jsiwek at corelight.com  Thu Sep  6 08:28:09 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 6 Sep 2018 10:28:09 -0500
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <CAFfiZAWO10YobDiPgtK9r6u=QP9es5c7JqXDww7_NQbJt3A65A@mail.gmail.com>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
	<CAFfiZAWO10YobDiPgtK9r6u=QP9es5c7JqXDww7_NQbJt3A65A@mail.gmail.com>
Message-ID: <CAMzgZ0KSWPC6YkZEUvskr4s7oe-j=BJuT+3huf87JQNTkT0GrA@mail.gmail.com>

On Wed, Sep 5, 2018 at 5:43 PM Michael Dopheide <dopheide at es.net> wrote:
>
> To be clear, Cluster::relay_rr() is gone forever?  I?ll need to rewrite some policies, but also update the blog to be correct.

Yes.

- Jon


From jazoff at illinois.edu  Thu Sep  6 12:41:48 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 6 Sep 2018 19:41:48 +0000
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
Message-ID: <C5300240-689A-4BC2-8C5A-F3796B69EEE9@illinois.edu>


> On Sep 5, 2018, at 6:35 PM, Jon Siwek <jsiwek at corelight.com> wrote:
> 
> There's no significant code changes/features planned to get added to
> the master branch from now until the 2.6-beta gets released (maybe in
> about a week).  Until that happens, please help test the latest master
> branch and provide any feedback about how it's working if you can.
> 
> - Jon
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

I just got 2 clusters upgraded from 

fa7fa5aa to
452eb0cb

And now everything is broken..

cpu and memory are through the roof across the board, as well as network traffic, but it's not logging much.

I may have created a message loop replacing the relay_rr stuff, but it's kind of hard to tell.

I'll do some more testing but so far this is the first issue I've ran into in months.

I guess one observation is that it is really hard to tell what bro/broker are doing.   Before you could minimally
tcpdump the communication and see what events were being sent back and forth, but now that is encrypted.



? 
Justin Azoff



From jazoff at illinois.edu  Thu Sep  6 13:11:41 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 6 Sep 2018 20:11:41 +0000
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <C5300240-689A-4BC2-8C5A-F3796B69EEE9@illinois.edu>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
	<C5300240-689A-4BC2-8C5A-F3796B69EEE9@illinois.edu>
Message-ID: <0A252CF9-0CE2-4938-BA02-449BF8352945@illinois.edu>


> On Sep 6, 2018, at 3:41 PM, Azoff, Justin S <jazoff at illinois.edu> wrote:
> 
> I just got 2 clusters upgraded from 
> 
> fa7fa5aa to
> 452eb0cb
> 
> 
> I may have created a message loop replacing the relay_rr stuff, but it's kind of hard to tell.
> 
> I'll do some more testing but so far this is the first issue I've ran into in months.
> 

I tested an almost stock local.bro (a few additional things disabled) and saw the same thing.

fa7fa5aa is fine, but with 452eb0cb everything is working really hard to do something.

The most noticeable thing is the network traffic on the manager changing from being almost idle at

300KiB TX and 3MiB RX and jumping to
20MiB TX and 12MiB RX

? 
Justin Azoff



From jsiwek at corelight.com  Thu Sep  6 13:14:51 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 6 Sep 2018 15:14:51 -0500
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <C5300240-689A-4BC2-8C5A-F3796B69EEE9@illinois.edu>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
	<C5300240-689A-4BC2-8C5A-F3796B69EEE9@illinois.edu>
Message-ID: <CAMzgZ0+U4f78SJ-FR2sP3KHD4d-RJqq=K3ZU3fJETLLaEjVP8g@mail.gmail.com>

On Thu, Sep 6, 2018 at 2:47 PM Azoff, Justin S <jazoff at illinois.edu> wrote:

> I just got 2 clusters upgraded from
>
> fa7fa5aa to
> 452eb0cb
>
> And now everything is broken..
>
> cpu and memory are through the roof across the board, as well as network traffic, but it's not logging much.
>
> I may have created a message loop replacing the relay_rr stuff, but it's kind of hard to tell.

The recent forwarding changes would be my main suspicion and, at least
in the default scripts, there's no communication patterns that
actually make use of the automatic forwarding, so can you check if
adding "redef Broker::forward_messages = F;" to site/local.bro makes a
difference?

If it does fix things, then yeah, either I missed a forwarding loop in
the default scripts or potentially you introduced one when replacing
relay_rr (feel free to point me at stuff to look over).

(Generally may want to just leave message forwarding turned off due to
these types of dangers if that's what it turns out to be...).

> I guess one observation is that it is really hard to tell what bro/broker are doing.   Before you could minimally
> tcpdump the communication and see what events were being sent back and forth, but now that is encrypted.

You can redef Broker::disable_ssl=T.  I don't recall how readable the
non-encrypted communications are, but I think I did it at least once
or twice and still was able to spot event names.

- Jon

From jsiwek at corelight.com  Thu Sep  6 13:19:16 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 6 Sep 2018 15:19:16 -0500
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <0A252CF9-0CE2-4938-BA02-449BF8352945@illinois.edu>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
	<C5300240-689A-4BC2-8C5A-F3796B69EEE9@illinois.edu>
	<0A252CF9-0CE2-4938-BA02-449BF8352945@illinois.edu>
Message-ID: <CAMzgZ0LCBDJVpR0LA9aw5z3WO49HgnVmRwZmHnFq0MQL8x-qcQ@mail.gmail.com>

On Thu, Sep 6, 2018 at 3:14 PM Azoff, Justin S <jazoff at illinois.edu> wrote:


> I tested an almost stock local.bro (a few additional things disabled) and saw the same thing.
>
> fa7fa5aa is fine, but with 452eb0cb everything is working really hard to do something.

Thanks for that, I'll start looking into it, but still would be
helpful if you could try disabling message forwarding (or disable ssl
+ look at some captured traffic to see if you can understand what
might be happening).  Thanks.

- Jon

From jazoff at illinois.edu  Thu Sep  6 13:34:52 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Thu, 6 Sep 2018 20:34:52 +0000
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <CAMzgZ0LCBDJVpR0LA9aw5z3WO49HgnVmRwZmHnFq0MQL8x-qcQ@mail.gmail.com>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
	<C5300240-689A-4BC2-8C5A-F3796B69EEE9@illinois.edu>
	<0A252CF9-0CE2-4938-BA02-449BF8352945@illinois.edu>
	<CAMzgZ0LCBDJVpR0LA9aw5z3WO49HgnVmRwZmHnFq0MQL8x-qcQ@mail.gmail.com>
Message-ID: <4C66160C-4E96-453B-B540-A8A122C57232@illinois.edu>


> On Sep 6, 2018, at 4:19 PM, Jon Siwek <jsiwek at corelight.com> wrote:
> 
> On Thu, Sep 6, 2018 at 3:14 PM Azoff, Justin S <jazoff at illinois.edu> wrote:
> 
> 
>> I tested an almost stock local.bro (a few additional things disabled) and saw the same thing.
>> 
>> fa7fa5aa is fine, but with 452eb0cb everything is working really hard to do something.
> 
> Thanks for that, I'll start looking into it, but still would be
> helpful if you could try disabling message forwarding (or disable ssl
> + look at some captured traffic to see if you can understand what
> might be happening).  Thanks.
> 
> - Jon

Yeah, that fixed it!

I re-enabled that and then disabled ssl and I am looking at the comm stuff going to the logger, which should just be logs


This seems to work for basic quick analysis:

[root at bro40-dev ~]# tcpdump  -n -i em1 port 47761 -A|sed "s/\.\.\.\.\./\n/g"|egrep -io  broker.* |head -n 10000|sort|uniq  -c|sort -nr
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 262144 bytes
tcpdump: Unable to write output: Broken pipe
   8842 broker::topic+broker::internal_command+ at u32.bro/known/certs/<$>/data/clone
   1124 broker::topic+broker::internal_command+ at u32.bro/known/hosts/<$>/data/clone
      8 broker::internal_command+ at u32.bro/known/certs/<$>/data/clone
      5 broker::topic+broker::internal_command+@




? 
Justin Azoff



From jsiwek at corelight.com  Thu Sep  6 16:40:00 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 6 Sep 2018 18:40:00 -0500
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <4C66160C-4E96-453B-B540-A8A122C57232@illinois.edu>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
	<C5300240-689A-4BC2-8C5A-F3796B69EEE9@illinois.edu>
	<0A252CF9-0CE2-4938-BA02-449BF8352945@illinois.edu>
	<CAMzgZ0LCBDJVpR0LA9aw5z3WO49HgnVmRwZmHnFq0MQL8x-qcQ@mail.gmail.com>
	<4C66160C-4E96-453B-B540-A8A122C57232@illinois.edu>
Message-ID: <CAMzgZ0Ln=jz2cVfu-=bNXYJMT=ZFLb6dmq=Pq9MtA4aqWSy7wQ@mail.gmail.com>

On Thu, Sep 6, 2018 at 3:40 PM Azoff, Justin S <jazoff at illinois.edu> wrote:

>    8842 broker::topic+broker::internal_command+ at u32.bro/known/certs/<$>/data/clone

Thanks, there was an unintended forwarding loop in data store
communication.  It's fixed in master now, but I've also just reverted
to generally disabling the forwarding mechanisms by default (any
specific/advanced use-cases can always choose to selectively enable
it, but the default cluster config doesn't need it for anything
currently).

For anyone doing testing, please pull the latest change from master
and give feedback on that version.

- Jon

From jazoff at illinois.edu  Fri Sep  7 13:41:16 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Fri, 7 Sep 2018 20:41:16 +0000
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <CAMzgZ0Ln=jz2cVfu-=bNXYJMT=ZFLb6dmq=Pq9MtA4aqWSy7wQ@mail.gmail.com>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
	<C5300240-689A-4BC2-8C5A-F3796B69EEE9@illinois.edu>
	<0A252CF9-0CE2-4938-BA02-449BF8352945@illinois.edu>
	<CAMzgZ0LCBDJVpR0LA9aw5z3WO49HgnVmRwZmHnFq0MQL8x-qcQ@mail.gmail.com>
	<4C66160C-4E96-453B-B540-A8A122C57232@illinois.edu>
	<CAMzgZ0Ln=jz2cVfu-=bNXYJMT=ZFLb6dmq=Pq9MtA4aqWSy7wQ@mail.gmail.com>
Message-ID: <4B2986E8-7A32-48AD-92B5-7CE4E2365D81@illinois.edu>


> On Sep 6, 2018, at 7:40 PM, Jon Siwek <jsiwek at corelight.com> wrote:
> 
> On Thu, Sep 6, 2018 at 3:40 PM Azoff, Justin S <jazoff at illinois.edu> wrote:
> 
>>   8842 broker::topic+broker::internal_command+ at u32.bro/known/certs/<$>/data/clone
> 
> Thanks, there was an unintended forwarding loop in data store
> communication.  It's fixed in master now, but I've also just reverted
> to generally disabling the forwarding mechanisms by default (any
> specific/advanced use-cases can always choose to selectively enable
> it, but the default cluster config doesn't need it for anything
> currently).
> 
> For anyone doing testing, please pull the latest change from master
> and give feedback on that version.
> 
> - Jon

Latest master works much better.

One thing I'm still seeing when I switch from an old version to latest master is that huge spike
in Content switches/interrupts and cpu spent in the kernel.

On one test worker box (that is receiving a full 10G and a bit overloaded)

before: context switches 200k/sec, interrupts 150k/sec
after: context switches 1mil/sec, interrupts 700k/sec

Before, cpu maxed out but spending 60% in user and 30% in system
After, cpu maxed out but spending 12% in user and 80% in system

On the manager side, cpu went up a little, but not by much.

I tried redef Broker::max_threads=2; but that didn't make much of a difference.

? 
Justin Azoff



From jsiwek at corelight.com  Fri Sep  7 15:58:50 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Fri, 7 Sep 2018 17:58:50 -0500
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <4B2986E8-7A32-48AD-92B5-7CE4E2365D81@illinois.edu>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
	<C5300240-689A-4BC2-8C5A-F3796B69EEE9@illinois.edu>
	<0A252CF9-0CE2-4938-BA02-449BF8352945@illinois.edu>
	<CAMzgZ0LCBDJVpR0LA9aw5z3WO49HgnVmRwZmHnFq0MQL8x-qcQ@mail.gmail.com>
	<4C66160C-4E96-453B-B540-A8A122C57232@illinois.edu>
	<CAMzgZ0Ln=jz2cVfu-=bNXYJMT=ZFLb6dmq=Pq9MtA4aqWSy7wQ@mail.gmail.com>
	<4B2986E8-7A32-48AD-92B5-7CE4E2365D81@illinois.edu>
Message-ID: <CAMzgZ0Latuta0coY5VL93JXgbdVvXeZVoKR-T4+jLPR8kwbyeA@mail.gmail.com>

On Fri, Sep 7, 2018 at 3:41 PM Azoff, Justin S <jazoff at illinois.edu> wrote:

> One thing I'm still seeing when I switch from an old version to latest master is that huge spike
> in Content switches/interrupts and cpu spent in the kernel.

I just updated the default tuning parameters for CAF's scheduling
policy and exposed them all in Bro [1] and get within 10% of the
number of context switches that 2.5.5 had in a very simple test case.

Let me know how that goes for everyone.

[1] https://github.com/bro/bro/blob/4bd6da71864b4ab68b372954c6268a023d69e52b/scripts/base/frameworks/broker/main.bro#L64-L94

From jazoff at illinois.edu  Sat Sep  8 08:20:05 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Sat, 8 Sep 2018 15:20:05 +0000
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <CAMzgZ0Latuta0coY5VL93JXgbdVvXeZVoKR-T4+jLPR8kwbyeA@mail.gmail.com>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
	<C5300240-689A-4BC2-8C5A-F3796B69EEE9@illinois.edu>
	<0A252CF9-0CE2-4938-BA02-449BF8352945@illinois.edu>
	<CAMzgZ0LCBDJVpR0LA9aw5z3WO49HgnVmRwZmHnFq0MQL8x-qcQ@mail.gmail.com>
	<4C66160C-4E96-453B-B540-A8A122C57232@illinois.edu>
	<CAMzgZ0Ln=jz2cVfu-=bNXYJMT=ZFLb6dmq=Pq9MtA4aqWSy7wQ@mail.gmail.com>
	<4B2986E8-7A32-48AD-92B5-7CE4E2365D81@illinois.edu>
	<CAMzgZ0Latuta0coY5VL93JXgbdVvXeZVoKR-T4+jLPR8kwbyeA@mail.gmail.com>
Message-ID: <604BF7D1-F959-496F-A798-39FAA53A62AD@illinois.edu>


> On Sep 7, 2018, at 6:58 PM, Jon Siwek <jsiwek at corelight.com> wrote:
> 
> On Fri, Sep 7, 2018 at 3:41 PM Azoff, Justin S <jazoff at illinois.edu> wrote:
> 
>> One thing I'm still seeing when I switch from an old version to latest master is that huge spike
>> in Content switches/interrupts and cpu spent in the kernel.
> 
> I just updated the default tuning parameters for CAF's scheduling
> policy and exposed them all in Bro [1] and get within 10% of the
> number of context switches that 2.5.5 had in a very simple test case.
> 
> Let me know how that goes for everyone.
> 
> [1] https://github.com/bro/bro/blob/4bd6da71864b4ab68b372954c6268a023d69e52b/scripts/base/frameworks/broker/main.bro#L64-L94

That helped a little with the context switches, but not much with the cpu load.

Do many of those options do anything?  I tried looking in the CAF source to figure out how they are used, and it looks like they are all defined in 
libcaf_core/caf/actor_system_config.hpp as

  // -- work-stealing parameters -----------------------------------------------

  size_t work_stealing_aggressive_poll_attempts CAF_DEPRECATED;
  size_t work_stealing_aggressive_steal_interval CAF_DEPRECATED;
  size_t work_stealing_moderate_poll_attempts CAF_DEPRECATED;
  size_t work_stealing_moderate_steal_interval CAF_DEPRECATED;
  size_t work_stealing_moderate_sleep_duration_us CAF_DEPRECATED;
  size_t work_stealing_relaxed_steal_interval CAF_DEPRECATED;
  size_t work_stealing_relaxed_sleep_duration_us CAF_DEPRECATED;



? 
Justin Azoff




From jazoff at illinois.edu  Sat Sep  8 09:10:35 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Sat, 8 Sep 2018 16:10:35 +0000
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <604BF7D1-F959-496F-A798-39FAA53A62AD@illinois.edu>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
	<C5300240-689A-4BC2-8C5A-F3796B69EEE9@illinois.edu>
	<0A252CF9-0CE2-4938-BA02-449BF8352945@illinois.edu>
	<CAMzgZ0LCBDJVpR0LA9aw5z3WO49HgnVmRwZmHnFq0MQL8x-qcQ@mail.gmail.com>
	<4C66160C-4E96-453B-B540-A8A122C57232@illinois.edu>
	<CAMzgZ0Ln=jz2cVfu-=bNXYJMT=ZFLb6dmq=Pq9MtA4aqWSy7wQ@mail.gmail.com>
	<4B2986E8-7A32-48AD-92B5-7CE4E2365D81@illinois.edu>
	<CAMzgZ0Latuta0coY5VL93JXgbdVvXeZVoKR-T4+jLPR8kwbyeA@mail.gmail.com>
	<604BF7D1-F959-496F-A798-39FAA53A62AD@illinois.edu>
Message-ID: <47E2FE33-3EA1-4B05-B312-45A41B97C5C0@illinois.edu>


> On Sep 8, 2018, at 11:20 AM, Azoff, Justin S <jazoff at illinois.edu> wrote:
> 
> Do many of those options do anything?  I tried looking in the CAF source to figure out how they are used, and it looks like they are all defined in 
> libcaf_core/caf/actor_system_config.hpp as

Scratch that.. the deprecation refers to something else, they are still used here:

work_stealing::worker_data::worker_data(scheduler::abstract_coordinator* p)
    : rengine(std::random_device{}()),
      // no need to worry about wrap-around; if `p->num_workers() < 2`,
      // `uniform` will not be used anyway
      uniform(0, p->num_workers() - 2),
      strategies{{
        {CONFIG("aggressive-poll-attempts", aggressive_poll_attempts), 1,
         CONFIG("aggressive-steal-interval", aggressive_steal_interval),
         timespan{0}},
        {CONFIG("moderate-poll-attempts", moderate_poll_attempts), 1,
         CONFIG("moderate-steal-interval", moderate_steal_interval),
         CONFIG("moderate-sleep-duration", moderate_sleep_duration)},
        {1, 0, CONFIG("relaxed-steal-interval", relaxed_steal_interval),
         CONFIG("relaxed-sleep-duration", relaxed_sleep_duration)}}} {
  // nop
}

So that strategies is an array of structs of [attempts, step size, steel interval, sleep interval]

so as I understand it, the current default is

5 polls, 1 step, interval 4, 0 sleep
5 polls, 1 step, interval 2, 16msec sleep
1 poll, 0 step, interval 1, 64msec sleep

Which means if a thread has no work to do it will loop for 5 times with no sleep, then 5 more times with 16msec sleep, then once with 64ms sleep.

This doesn't seem that bad.. worse case is 11 attempts every 149ms or ~70/second.

I went as far as trying 

redef Broker::aggressive_polls = 1;
redef Broker::moderate_polls = 1;
redef Broker::moderate_sleep = 100msec;
redef Broker::relaxed_sleep = 200msec;

but that didn't make a noticeable change, so the scheduler may not be the problem.  Will have to do some more testing.



? 
Justin Azoff



From vallentin at icir.org  Mon Sep 10 06:00:21 2018
From: vallentin at icir.org (Matthias Vallentin)
Date: Mon, 10 Sep 2018 15:00:21 +0200
Subject: [Bro-Dev] How to use Broker::Data in an event handler?
Message-ID: <20180910130021.GN73386@tenzir.com>

I'm trying to figure out if/how it is possible to use Broker::Data in an 
event handler as follows:

    event foo(x: Broker::Data)
      {
      print x;
      }

I'm trying to send an event via the Python bindings:

    event = broker.bro.Event("foo", broker.Data(42))
    endpoint.publish("/test", event)

However, Bro complains:

    warning: failed to convert remote event 'foo' arg #0, got integer, expected record

I tried both 

    event = broker.bro.Event("foo", 42)

and a wrapped version

    event = broker.bro.Event("foo", broker.Data(42))

and even

    event = broker.bro.Event("foo", broker.Data(broker.Data(42)))

but it seems that nesting is not possible.

The use case for having a Broker::Data in the Bro event handler is that 
the structure of the data is varying at runtime (similar to JSON).

    Matthias

(The code is a slightly adapted version from 
https://github.com/bro/broker/issues/11.)


From jazoff at illinois.edu  Mon Sep 10 09:09:42 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Mon, 10 Sep 2018 16:09:42 +0000
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <4B2986E8-7A32-48AD-92B5-7CE4E2365D81@illinois.edu>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
	<C5300240-689A-4BC2-8C5A-F3796B69EEE9@illinois.edu>
	<0A252CF9-0CE2-4938-BA02-449BF8352945@illinois.edu>
	<CAMzgZ0LCBDJVpR0LA9aw5z3WO49HgnVmRwZmHnFq0MQL8x-qcQ@mail.gmail.com>
	<4C66160C-4E96-453B-B540-A8A122C57232@illinois.edu>
	<CAMzgZ0Ln=jz2cVfu-=bNXYJMT=ZFLb6dmq=Pq9MtA4aqWSy7wQ@mail.gmail.com>
	<4B2986E8-7A32-48AD-92B5-7CE4E2365D81@illinois.edu>
Message-ID: <D05B0124-D375-471A-BD85-7563728A6C1A@illinois.edu>


> On Sep 7, 2018, at 4:41 PM, Azoff, Justin S <jazoff at illinois.edu> wrote:
> 
> Before, cpu maxed out but spending 60% in user and 30% in system
> After, cpu maxed out but spending 12% in user and 80% in system
> 

I did some more testing and profiling and figured out what is going on..

The new version is much more efficient, so it's spending a lot less time in user space
and a lot more time in the kernel fetching packets.

On the managers, overall CPU using is a bit lower, but I think some of that is from removing all uses
of &synchronized, especially for large input tables.

If anything there's just a slight overhead in the bro-myricom plugin in how it uses snf_ring_recv to
receive a single packet instead of snf_ring_recv_many to grab multiple packets like how af_packet works.

I'd look into fixing it but we are moving to intel 40g cards anyway.



? 
Justin Azoff



From jsiwek at corelight.com  Mon Sep 10 13:20:24 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Mon, 10 Sep 2018 15:20:24 -0500
Subject: [Bro-Dev] How to use Broker::Data in an event handler?
In-Reply-To: <20180910130021.GN73386@tenzir.com>
References: <20180910130021.GN73386@tenzir.com>
Message-ID: <CAMzgZ0JxSRmPAZyfJgqwf2v3FTYaR+Y87wP58C9RA-vBaJEZJA@mail.gmail.com>

On Mon, Sep 10, 2018 at 8:01 AM Matthias Vallentin <vallentin at icir.org> wrote:
>
> I'm trying to figure out if/how it is possible to use Broker::Data in an
> event handler as follows:
>
>     event foo(x: Broker::Data)
>       {
>       print x;
>       }

No, but you can try to use 'any' instead of 'Broker::Data'.  Examples/ideas:

# Bro code

type myvec: vector of any;

type myrec: record {
    a: string &optional;
    b: count &optional;
    c: int &optional;
};

event bar(x: any)
    {
    switch ( x ) {
    case type myrec as r:
        print "record", r;
        break;
    case type string as s:
        print "string", s;
        break;
    case type int as i:
        print "int", i;
        break;
    case type count as c:
        print "count", c;
        break;
    case type myvec as v:
        {
        print "vector", v;

        for ( i in v )
            event bar(v[i]);
        }
        break;
    default:
        print "got unknown type", x;
        break;
    }
}

# Python code

endpoint.publish("/test", broker.bro.Event("bar", "one"))
endpoint.publish("/test", broker.bro.Event("bar", 2))
endpoint.publish("/test", broker.bro.Event("bar", broker.Count(3)))
endpoint.publish("/test", broker.bro.Event("bar",
        ["one", "two", broker.Count(3)]))
endpoint.publish("/test", broker.bro.Event("bar",
        ["one", None, None]))

> The use case for having a Broker::Data in the Bro event handler is that
> the structure of the data is varying at runtime (similar to JSON).

Should be the same idea if you use the 'any' type along with
appropriate type checking/casting.

> (The code is a slightly adapted version from
> https://github.com/bro/broker/issues/11.)

This, plus a couple other bugs should now be fixed in bro + broker, so
make sure to update both if trying the above examples.

- Jon
On Mon, Sep 10, 2018 at 8:01 AM Matthias Vallentin <vallentin at icir.org> wrote:
>
> I'm trying to figure out if/how it is possible to use Broker::Data in an
> event handler as follows:
>
>     event foo(x: Broker::Data)
>       {
>       print x;
>       }
>
> I'm trying to send an event via the Python bindings:
>
>     event = broker.bro.Event("foo", broker.Data(42))
>     endpoint.publish("/test", event)
>
> However, Bro complains:
>
>     warning: failed to convert remote event 'foo' arg #0, got integer, expected record
>
> I tried both
>
>     event = broker.bro.Event("foo", 42)
>
> and a wrapped version
>
>     event = broker.bro.Event("foo", broker.Data(42))
>
> and even
>
>     event = broker.bro.Event("foo", broker.Data(broker.Data(42)))
>
> but it seems that nesting is not possible.
>
> The use case for having a Broker::Data in the Bro event handler is that
> the structure of the data is varying at runtime (similar to JSON).
>
>     Matthias
>
> (The code is a slightly adapted version from
> https://github.com/bro/broker/issues/11.)
>
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

From vern at corelight.com  Mon Sep 10 14:16:16 2018
From: vern at corelight.com (Vern Paxson)
Date: Mon, 10 Sep 2018 14:16:16 -0700
Subject: [Bro-Dev] attributes & named types
In-Reply-To: <CAMzgZ0KUKsx0kyKGvtpnpCrH+mO8S+jnDJXjPnT+RHedNdFP-A@mail.gmail.com>
	(Thu, 06 Sep 2018 09:52:49 CDT).
Message-ID: <201809102116.w8ALGGCu015942@fruitcake.ICSI.Berkeley.EDU>

> Other ideas I'm having for solving the overall problem are:
> 
> (1) 'a' and 'b' need to actually be distinct BroType's.  Instead of
> 'b' being a reference/alias to 'a' with extended attributes, just
> create it as it's own BroType by first cloning 'a', then adding any
> additional attributes.

I originally went down this route, but this involves adding attributes to
all types, since currently types don't have attributes.  Seems like a poor
fit given there's only one class of type (named ones) that need this.

> (2) BroType could somehow store a mapping of identifier -> attributes
> so that on declaring a variable of either 'a' or 'b', we can choose
> which attributes apply.

That seems more hacky than (1), as it involves adding still more to
types-in-general.

Seems simplest to me to have TypeType's (and only those) include attributes.
The rest of it is easy to do from there.  We could also do this with a
separate NameType, but I don't see what that gains, since TypeType's already
come into existence because of binding names to types.

		Vern

From jsiwek at corelight.com  Mon Sep 10 15:01:00 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Mon, 10 Sep 2018 17:01:00 -0500
Subject: [Bro-Dev] attributes & named types
In-Reply-To: <201809102116.w8ALGGCu015942@fruitcake.ICSI.Berkeley.EDU>
References: <CAMzgZ0KUKsx0kyKGvtpnpCrH+mO8S+jnDJXjPnT+RHedNdFP-A@mail.gmail.com>
	<201809102116.w8ALGGCu015942@fruitcake.ICSI.Berkeley.EDU>
Message-ID: <CAMzgZ0JnV+3FuFxP1FE2SVwNdTWfPiyVd=N_CCptLR-B9gUTdg@mail.gmail.com>

On Mon, Sep 10, 2018 at 4:16 PM Vern Paxson <vern at corelight.com> wrote:

> Seems simplest to me to have TypeType's (and only those) include attributes.
> The rest of it is easy to do from there.  We could also do this with a
> separate NameType, but I don't see what that gains, since TypeType's already
> come into existence because of binding names to types.

My understanding was just that TypeType's currently are *not* used
when creating type aliases.  Example:

# "mytype" is an ID w/ type "count" (i.e. it's a type alias).
# It's not using "TypeType" at this point.
type mytype: count;

function foo(x: any)
    { print x; }

# Passing the type name/alias as a value.
# The local variable/argument 'x' becomes of type
# "TypeType".  It's not of type "count".
foo(mytype);

# Here, 'y' and 'x' are now actually type "count".
# They aren't a "TypeType".
local y: mytype = 3;
foo(y);

At least that's how I think it's currently working, so are you going
start using TypeType as a means of type aliasing in addition to adding
attributes to them?

Just trying to clarify/understand how things currently work vs. what is planned.

- Jon

From vern at corelight.com  Mon Sep 10 15:08:05 2018
From: vern at corelight.com (Vern Paxson)
Date: Mon, 10 Sep 2018 15:08:05 -0700
Subject: [Bro-Dev] attributes & named types
In-Reply-To: <CAMzgZ0JnV+3FuFxP1FE2SVwNdTWfPiyVd=N_CCptLR-B9gUTdg@mail.gmail.com>
	(Mon, 10 Sep 2018 17:01:00 CDT).
Message-ID: <201809102208.w8AM85cJ024295@fruitcake.ICSI.Berkeley.EDU>

> My understanding was just that TypeType's currently are *not* used
> when creating type aliases.

Correct.

> # Passing the type name/alias as a value.
> # The local variable/argument 'x' becomes of type
> # "TypeType".  It's not of type "count".
> foo(mytype);

(Note that here any attributes are lost, since there's currently no
 place for TypeType's to hold them.)

> At least that's how I think it's currently working, so are you going
> start using TypeType as a means of type aliasing in addition to adding
> attributes to them?

Not quite.  Rather, the type of "mytype" would be a TypeType, which would
have attributes.  The TypeType instance however would not know that it
belongs to "mytype" (just as is currently the case).

We could continue to support a call like "foo(mytype)" above by hoisting
the base type (what the TypeType points to) when evaluating the expression
"mytype", just as currently the identifier gets turned into a TypeType at
that point.  That said, just what is the use for calls like "foo(mytype)"
anyway?  Seems a bit peculiar, but maybe I'm missing something.

		Vern

From jsiwek at corelight.com  Mon Sep 10 15:34:21 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Mon, 10 Sep 2018 17:34:21 -0500
Subject: [Bro-Dev] attributes & named types
In-Reply-To: <201809102208.w8AM85cJ024295@fruitcake.ICSI.Berkeley.EDU>
References: <CAMzgZ0JnV+3FuFxP1FE2SVwNdTWfPiyVd=N_CCptLR-B9gUTdg@mail.gmail.com>
	<201809102208.w8AM85cJ024295@fruitcake.ICSI.Berkeley.EDU>
Message-ID: <CAMzgZ0+=nTm+7EGNWUfuUYky5ayC_BZmkd2Nv_18Mu3xnZFfOg@mail.gmail.com>

On Mon, Sep 10, 2018 at 5:08 PM Vern Paxson <vern at corelight.com> wrote:

> > At least that's how I think it's currently working, so are you going
> > start using TypeType as a means of type aliasing in addition to adding
> > attributes to them?
>
> Not quite.  Rather, the type of "mytype" would be a TypeType, which would
> have attributes.  The TypeType instance however would not know that it
> belongs to "mytype" (just as is currently the case).

Ok, think I got it now (and agree it seems like a possible way forward
regarding attributes).

> We could continue to support a call like "foo(mytype)" above by hoisting
> the base type (what the TypeType points to) when evaluating the expression
> "mytype", just as currently the identifier gets turned into a TypeType at
> that point.  That said, just what is the use for calls like "foo(mytype)"
> anyway?  Seems a bit peculiar, but maybe I'm missing something.

The actual usage I know is from Log::create_stream(), like this one:

  Log::create_stream(DPD::LOG, [$columns=Info, $path="dpd"]);

There, the Info is a type that's being stored into the $columns record
field of type 'any'.  It's not exactly the same as "passed as function
argument" example I gave, but same idea.

I think TypeType was added particularly for this logging framework usage.

- Jon

From vallentin at icir.org  Tue Sep 11 03:35:51 2018
From: vallentin at icir.org (Matthias Vallentin)
Date: Tue, 11 Sep 2018 12:35:51 +0200
Subject: [Bro-Dev] switch/case type recognition
Message-ID: <20180911103551.GQ73386@tenzir.com>

Consider this snippet:

    function f(x: any)
      {
      switch (x)
        {
        case type any:
          print "any";
          break;
        case type vector of any:
          print "vector of any";
          break;
        }
      }

    event bro_init()
      {
      f(42);
      }

I would have imagined that the two cases are considered different types, 
but Bro doesn't think so:

    error in ./test.bro, lines 9-10: duplicate case label (case type vector of any:{ print anybreak })

Adding a type alias for either type doesn't change the error message.

I am aware that this is a somewhat pathological case, because 'case type 
any' is probably equivalent to the 'default' case.

    Matthias

From vallentin at icir.org  Tue Sep 11 03:52:28 2018
From: vallentin at icir.org (Matthias Vallentin)
Date: Tue, 11 Sep 2018 12:52:28 +0200
Subject: [Bro-Dev] How to use Broker::Data in an event handler?
In-Reply-To: <CAMzgZ0JxSRmPAZyfJgqwf2v3FTYaR+Y87wP58C9RA-vBaJEZJA@mail.gmail.com>
References: <20180910130021.GN73386@tenzir.com>
	<CAMzgZ0JxSRmPAZyfJgqwf2v3FTYaR+Y87wP58C9RA-vBaJEZJA@mail.gmail.com>
Message-ID: <CADTpMNYamJrbvJXk-whbE-gFPapSBT5-sUOTy6zUUM_PWOo3yQ@mail.gmail.com>

> This, plus a couple other bugs should now be fixed in bro + broker, so
make sure to update both if trying the above examples.

Great, it works now!

One more question: how would I capture a default-constructed
broker::Data() in a case statement? This would happen when I publish
just "None" on the Python side. In Bro, it shows up on the command
line as "broker::data{nil}".

    Matthias

From jsiwek at corelight.com  Tue Sep 11 07:34:13 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Tue, 11 Sep 2018 09:34:13 -0500
Subject: [Bro-Dev] switch/case type recognition
In-Reply-To: <20180911103551.GQ73386@tenzir.com>
References: <20180911103551.GQ73386@tenzir.com>
Message-ID: <CAMzgZ0KiGef+6vyy+qZNDgqHhd=YkdTfMmybjOW6UL5XHsaVrg@mail.gmail.com>

On Tue, Sep 11, 2018 at 5:42 AM Matthias Vallentin <vallentin at icir.org> wrote:

> I am aware that this is a somewhat pathological case, because 'case type
> any' is probably equivalent to the 'default' case.

A 'vector of any' also qualifies as an 'any', so while the error
message of "duplicate case label" could possibly be improved, I think
it's still correct in that it is meant to prevent an ambiguous case
match.  In your example, if you did pass in a vector to that function,
it could choose either the 'any' case or the 'vector of any' case.
One might have the expectation that it chooses the first case that
matches in top-down order, however that might conflict with other
expectations that switch statements can be implemented via hash table
lookup and require unambiguous cases.  In Bro, regular 'case' is
currently a hash table lookup, while actually 'case type' is a linear
search, but that seems to be just implementation detail and not a
conscious decision that can be relied on to allow ordered-case-type
matching.

- Jon

From jsiwek at corelight.com  Tue Sep 11 08:09:58 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Tue, 11 Sep 2018 10:09:58 -0500
Subject: [Bro-Dev] How to use Broker::Data in an event handler?
In-Reply-To: <CADTpMNYamJrbvJXk-whbE-gFPapSBT5-sUOTy6zUUM_PWOo3yQ@mail.gmail.com>
References: <20180910130021.GN73386@tenzir.com>
	<CAMzgZ0JxSRmPAZyfJgqwf2v3FTYaR+Y87wP58C9RA-vBaJEZJA@mail.gmail.com>
	<CADTpMNYamJrbvJXk-whbE-gFPapSBT5-sUOTy6zUUM_PWOo3yQ@mail.gmail.com>
Message-ID: <CAMzgZ0K_45uqRh3bQ6MKQ=atCx3LMndvqvVd4sYvtJKYJxkFbg@mail.gmail.com>

On Tue, Sep 11, 2018 at 5:52 AM Matthias Vallentin <vallentin at icir.org> wrote:

> One more question: how would I capture a default-constructed
> broker::Data() in a case statement? This would happen when I publish
> just "None" on the Python side. In Bro, it shows up on the command
> line as "broker::data{nil}".

There's no nil/null/none type in Bro, so only way I can think to do it
at the moment is:

function is_nil(x: any): bool
    {
    if ( ! (x is Broker::Data) )
        return F;

    local d = x as Broker::Data;

    if ( ! d?$data )
        return T;

    if ( cat(d$data) != "broker::data{nil}" )
        return F;

    return T;
    }

Or in switch case, it's like:

    case type Broker::Data as d:
        print "Broker::Data, expected to be nil", d?$data, d?$data ?
cat(d$data) : "nil";
        # or use the same logic from the is_nil() function above

- Jon

From vallentin at icir.org  Wed Sep 12 03:20:16 2018
From: vallentin at icir.org (Matthias Vallentin)
Date: Wed, 12 Sep 2018 12:20:16 +0200
Subject: [Bro-Dev] switch/case type recognition
In-Reply-To: <CAMzgZ0KiGef+6vyy+qZNDgqHhd=YkdTfMmybjOW6UL5XHsaVrg@mail.gmail.com>
References: <20180911103551.GQ73386@tenzir.com>
	<CAMzgZ0KiGef+6vyy+qZNDgqHhd=YkdTfMmybjOW6UL5XHsaVrg@mail.gmail.com>
Message-ID: <CADTpMNbYSoiSe27-as9wj4iNxTdoyspp9MsenVAunhFiSWYGUA@mail.gmail.com>

> A 'vector of any' also qualifies as an 'any', so while the error
> message of "duplicate case label" could possibly be improved, I think
> it's still correct in that it is meant to prevent an ambiguous case
> match.

Yeah, with that reasoning anything qualifies as "any", so adding "any"
should never show up in a "case type" statement. Probably a better way
to handle "any" is by just using the "default" case.

    Matthias

From vallentin at icir.org  Wed Sep 12 03:24:09 2018
From: vallentin at icir.org (Matthias Vallentin)
Date: Wed, 12 Sep 2018 12:24:09 +0200
Subject: [Bro-Dev] How to use Broker::Data in an event handler?
In-Reply-To: <CAMzgZ0K_45uqRh3bQ6MKQ=atCx3LMndvqvVd4sYvtJKYJxkFbg@mail.gmail.com>
References: <20180910130021.GN73386@tenzir.com>
	<CAMzgZ0JxSRmPAZyfJgqwf2v3FTYaR+Y87wP58C9RA-vBaJEZJA@mail.gmail.com>
	<CADTpMNYamJrbvJXk-whbE-gFPapSBT5-sUOTy6zUUM_PWOo3yQ@mail.gmail.com>
	<CAMzgZ0K_45uqRh3bQ6MKQ=atCx3LMndvqvVd4sYvtJKYJxkFbg@mail.gmail.com>
Message-ID: <CADTpMNaThSHfPTXjrzk-hxp8MeMCu89AexmuvExsmkEHtH2mFg@mail.gmail.com>

>     case type Broker::Data as d:
>         print "Broker::Data, expected to be nil", d?$data, d?$data ?
> cat(d$data) : "nil";
>         # or use the same logic from the is_nil() function above

This is the logic I was looking for, thanks! The generic is_nil
function might come in handy some other point.

    Matthias

From jazoff at illinois.edu  Wed Sep 12 07:17:18 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Wed, 12 Sep 2018 14:17:18 +0000
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
Message-ID: <DDFEFC57-B541-4BFF-B86E-DA3BE1E04CA7@illinois.edu>


On Sep 5, 2018, at 6:35 PM, Jon Siwek <jsiwek at corelight.com<mailto:jsiwek at corelight.com>> wrote:

There's no significant code changes/features planned to get added to
the master branch from now until the 2.6-beta gets released (maybe in
about a week).  Until that happens, please help test the latest master
branch and provide any feedback about how it's working if you can.

- Jon

Hi again!

Just finished the migration to master across the board, and it's looking REALLY good.

No crashes, memory is stable, cpu is pretty good.  The only thing I noticed is the manager userspace CPU utilization on one cluster
jumped up a bit after the switch.

This graph shows the system and userspace utilization across an 8 physical node cluster.  All the lower lines are the system usage
and the higher lines are userspace.  The Y axis is cpu seconds per second, so 4 is full 4 cores worth of usage.

After the switchover:

* the worker userspace cpu utilization was unchanged
* the worker system cpu utilization increased a bit, but I believe that is due to more time spent in the myricom driver waiting for packets
* the manager system cpu utilization dropped a bunch
* the manager userspace cpu increased 1-3x

The manager box in this cluster only runs the manager and logger processes, no proxies.  It also has something like 20 idle cores,
so this isn't a problem at all, but could affect people who run a cluster-in-a-box.

[cid:3BF8BC03-721F-44E6-A052-6E1ADA454C02 at home]

I do seem to be seeing a bunch of reporter errors like

Reporter::ERROR string with embedded NUL: "\\x00\\x00\\x00\\x00OPTIONS"

but I'm not sure if that is a new thing.

?
Justin Azoff

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180912/b09f9202/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cluster 2.6 cpu.jpeg
Type: image/jpeg
Size: 23588 bytes
Desc: cluster 2.6 cpu.jpeg
Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180912/b09f9202/attachment-0001.jpeg 

From vallentin at icir.org  Wed Sep 12 08:08:57 2018
From: vallentin at icir.org (Matthias Vallentin)
Date: Wed, 12 Sep 2018 17:08:57 +0200
Subject: [Bro-Dev] Module prefix in sending and receiving Broker events
Message-ID: <20180912150857.GE47519@tenzir.com>

When I receive events from Bro via Broker, they have the prefix of the 
enclosing module:

    module Foo;

    event foo() { ... }
    event bar() { ... }

When I publish "foo" via Broker, it arrives as "Foo::foo". However, when 
I publish an event "Foo::bar" from Broker, Bro doesn't recognize it. I
must published it as "bar" - without the module prefix. Is this 
intentional?

    Matthias

From jsiwek at corelight.com  Wed Sep 12 08:14:34 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Wed, 12 Sep 2018 10:14:34 -0500
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <DDFEFC57-B541-4BFF-B86E-DA3BE1E04CA7@illinois.edu>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
	<DDFEFC57-B541-4BFF-B86E-DA3BE1E04CA7@illinois.edu>
Message-ID: <CAMzgZ0J8JfU_SD7e9D8Z9SYU-OhQcOPQpasE_cjKhFV-qzTRRA@mail.gmail.com>

On Wed, Sep 12, 2018 at 9:18 AM Azoff, Justin S <jazoff at illinois.edu> wrote:
>
> Just finished the migration to master across the board, and it's looking REALLY good.

Great, thanks for helping test and provide performance data.

> The manager box in this cluster only runs the manager and logger processes, no proxies.  It also has something like 20 idle cores,
> so this isn't a problem at all, but could affect people who run a cluster-in-a-box.

An idea in this type of situation could be to tune Broker::max_threads
per node type.  E.g. leave at 1 for workers and bump to ~4 for
manager/logger since there's idle cores on their host and they're
inherently in a less-scalable/centralized location.  That may not
lower overall cpu usage, but may help prevent some bottlenecks in the
processing of remote messages.  Particularly the work of processing
data store communication should distribute among threads, potentially
each data store could be processing messages independently on separate
threads.  (The default scripts have 3 stores, one for each known-*
script).

> I do seem to be seeing a bunch of reporter errors like
>
> Reporter::ERROR string with embedded NUL: "\\x00\\x00\\x00\\x00OPTIONS"

Ok, not necessarily that bad, but would be nice to find where that's
coming from to handle it more properly

- Jon

From jsiwek at corelight.com  Wed Sep 12 08:30:57 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Wed, 12 Sep 2018 10:30:57 -0500
Subject: [Bro-Dev] Module prefix in sending and receiving Broker events
In-Reply-To: <20180912150857.GE47519@tenzir.com>
References: <20180912150857.GE47519@tenzir.com>
Message-ID: <CAMzgZ0LQqhRKfSAvo=T184DU8iA5q8nfi1umVra4R-9e2Z4M6Q@mail.gmail.com>

On Wed, Sep 12, 2018 at 10:09 AM Matthias Vallentin <vallentin at icir.org> wrote:
>
> When I receive events from Bro via Broker, they have the prefix of the
> enclosing module:
>
>     module Foo;
>
>     event foo() { ... }
>     event bar() { ... }
>
> When I publish "foo" via Broker, it arrives as "Foo::foo". However, when
> I publish an event "Foo::bar" from Broker, Bro doesn't recognize it. I
> must published it as "bar" - without the module prefix. Is this
> intentional?

Maybe not so much intentional, but expected at this point.  Does the
suggestion [1] to always explicitly scope events by their
namespace/module address your problem?  There's some longstanding
oddities [2] with the way events interact with module namespacing.

- Jon

[1] https://www.bro.org/sphinx-git/frameworks/broker.html#a-reminder-about-events-and-module-namespaces
[2] https://bro-tracker.atlassian.net/browse/BIT-71

From commike at reservoir.com  Wed Sep 12 09:57:52 2018
From: commike at reservoir.com (Alan Commike)
Date: Wed, 12 Sep 2018 09:57:52 -0700
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <CAMzgZ0J8JfU_SD7e9D8Z9SYU-OhQcOPQpasE_cjKhFV-qzTRRA@mail.gmail.com>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
	<DDFEFC57-B541-4BFF-B86E-DA3BE1E04CA7@illinois.edu>
	<CAMzgZ0J8JfU_SD7e9D8Z9SYU-OhQcOPQpasE_cjKhFV-qzTRRA@mail.gmail.com>
Message-ID: <CAMnzEagEXORKM7W62H=GTGqGrWpg5p+Lybh-4DmfY=H3s-mkVw@mail.gmail.com>

On Wed, Sep 12, 2018 at 8:21 AM Jon Siwek <jsiwek at corelight.com> wrote:

>
> An idea in this type of situation could be to tune Broker::max_threads
> per node type.  E.g. leave at 1 for workers and bump to ~4 for
> manager/logger since there's idle cores on their host and they're
> inherently in a less-scalable/centralized location.


For those that haven't yet dug into all that is Broker, this brings up a
question on the general architecture. Are there still parent/child
processes handling comms/work?  I'd assume the fork is gone, so the new
code is dropping a process but gaining a thread, in essence a wash.

Is there a mechanism today for per node type tuneables?

   ...alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180912/c0d5d61c/attachment.html 

From jsiwek at corelight.com  Wed Sep 12 12:26:29 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Wed, 12 Sep 2018 14:26:29 -0500
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <CAMnzEagEXORKM7W62H=GTGqGrWpg5p+Lybh-4DmfY=H3s-mkVw@mail.gmail.com>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
	<DDFEFC57-B541-4BFF-B86E-DA3BE1E04CA7@illinois.edu>
	<CAMzgZ0J8JfU_SD7e9D8Z9SYU-OhQcOPQpasE_cjKhFV-qzTRRA@mail.gmail.com>
	<CAMnzEagEXORKM7W62H=GTGqGrWpg5p+Lybh-4DmfY=H3s-mkVw@mail.gmail.com>
Message-ID: <CAMzgZ0L9f7NnMUycXjZxMOPq5b3bvHGLDQAUUw1bAx6PbBCGzA@mail.gmail.com>

On Wed, Sep 12, 2018 at 11:58 AM Alan Commike <commike at reservoir.com> wrote:

> Are there still parent/child processes handling comms/work?

No.  Single process, configurable number of threads (default 1).

> Is there a mechanism today for per node type tuneables?

One should be able to use an @if directive [1] to tune differently
per-node if they want.

- Jon

[1] https://www.bro.org/sphinx-git/script-reference/directives.html

From vallentin at icir.org  Thu Sep 13 02:28:18 2018
From: vallentin at icir.org (Matthias Vallentin)
Date: Thu, 13 Sep 2018 11:28:18 +0200
Subject: [Bro-Dev] Module prefix in sending and receiving Broker events
In-Reply-To: <CAMzgZ0LQqhRKfSAvo=T184DU8iA5q8nfi1umVra4R-9e2Z4M6Q@mail.gmail.com>
References: <20180912150857.GE47519@tenzir.com>
	<CAMzgZ0LQqhRKfSAvo=T184DU8iA5q8nfi1umVra4R-9e2Z4M6Q@mail.gmail.com>
Message-ID: <CADTpMNagdawF6g3RPKaQAUiKaAJTMVDGL2JhYOSo7Kptu-OPYQ@mail.gmail.com>

>  Does the
> suggestion [1] to always explicitly scope events by their
> namespace/module address your problem?

That's what I thought would work, but it's the opposite: when I add
the module name as prefix, Bro silently ignores the event. I found a
surprising solution though: if I declare the event first, then it
works. Here's an example:

    module Foo;
    global foo: event();
    event foo() { ... }

This works as expected: when I create an event with name Foo::foo, the
handler gets dispatched. Without the declaration, it doesn't work.

As mentioned in your reference [1], explicit module qualification works as well:

    event Foo::foo() { ... }

I'm essentially running into the inverse of BIT-71, the ticket you referenced.

    Matthias

From jsiwek at corelight.com  Thu Sep 13 10:00:54 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 13 Sep 2018 12:00:54 -0500
Subject: [Bro-Dev] Module prefix in sending and receiving Broker events
In-Reply-To: <CADTpMNagdawF6g3RPKaQAUiKaAJTMVDGL2JhYOSo7Kptu-OPYQ@mail.gmail.com>
References: <20180912150857.GE47519@tenzir.com>
	<CAMzgZ0LQqhRKfSAvo=T184DU8iA5q8nfi1umVra4R-9e2Z4M6Q@mail.gmail.com>
	<CADTpMNagdawF6g3RPKaQAUiKaAJTMVDGL2JhYOSo7Kptu-OPYQ@mail.gmail.com>
Message-ID: <CAMzgZ0LP_wF4Rc=Yf4j_9620xvbFDkn9vMkDikqu+92GmvCp0A@mail.gmail.com>

On Thu, Sep 13, 2018 at 4:28 AM Matthias Vallentin <vallentin at icir.org> wrote:
>
> >  Does the
> > suggestion [1] to always explicitly scope events by their
> > namespace/module address your problem?
>
> That's what I thought would work, but it's the opposite: when I add
> the module name as prefix, Bro silently ignores the event.

So [1] didn't work...

> As mentioned in your reference [1], explicit module qualification works as well:
>
>     event Foo::foo() { ... }

But [1] also worked? :)

> I'm essentially running into the inverse of BIT-71, the ticket you referenced.

Yeah, I think I see how it's the reverse of the original example, but
it's likely the same underlying module/namespacing ambiguities with
events, so I'd still suggest explicitly scoping events always and
everywhere (which was the [1] suggestion).

It might also help if you send actual examples that can be run if that
still doesn't work because it's hard to interpret what you mean by
"publish via Broker".  That could mean Bro's Broker::publish() API or
the standalone Broker API itself and they are potentially different.

> I found a
> surprising solution though: if I declare the event first, then it
> works. Here's an example:
>
>     module Foo;
>     global foo: event();
>     event foo() { ... }
>
> This works as expected: when I create an event with name Foo::foo, the
> handler gets dispatched. Without the declaration, it doesn't work.

Sure, but that's also not the [1] suggestion either.  Adding the
namespace scoping always and everywhere means to the event
declaration, the handlers, event/schedule dispatching, any strings
that contain the event name, etc.

- Jon

From john at alt.house  Thu Sep 13 14:57:54 2018
From: john at alt.house (John Althouse)
Date: Thu, 13 Sep 2018 17:57:54 -0400
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <CAMzgZ0L9f7NnMUycXjZxMOPq5b3bvHGLDQAUUw1bAx6PbBCGzA@mail.gmail.com>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
	<DDFEFC57-B541-4BFF-B86E-DA3BE1E04CA7@illinois.edu>
	<CAMzgZ0J8JfU_SD7e9D8Z9SYU-OhQcOPQpasE_cjKhFV-qzTRRA@mail.gmail.com>
	<CAMnzEagEXORKM7W62H=GTGqGrWpg5p+Lybh-4DmfY=H3s-mkVw@mail.gmail.com>
	<CAMzgZ0L9f7NnMUycXjZxMOPq5b3bvHGLDQAUUw1bAx6PbBCGzA@mail.gmail.com>
Message-ID: <CAHT3Yk8krmKh298c7f-oXK2i2=tGGb7npJmmmdYMrVwdcDGV+g@mail.gmail.com>

Could anyone update Bro on docker to include 2.5.5, 2.6-beta and master?
https://hub.docker.com/r/broplatform/bro/tags/

We use this with our internal trybro instance which is fantastic for
quickly collaborating and testing scripts. :)

On Wed, Sep 12, 2018 at 3:26 PM, Jon Siwek <jsiwek at corelight.com> wrote:

> On Wed, Sep 12, 2018 at 11:58 AM Alan Commike <commike at reservoir.com>
> wrote:
>
> > Are there still parent/child processes handling comms/work?
>
> No.  Single process, configurable number of threads (default 1).
>
> > Is there a mechanism today for per node type tuneables?
>
> One should be able to use an @if directive [1] to tune differently
> per-node if they want.
>
> - Jon
>
> [1] https://www.bro.org/sphinx-git/script-reference/directives.html
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180913/e196854d/attachment.html 

From jsiwek at corelight.com  Thu Sep 13 17:39:35 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 13 Sep 2018 19:39:35 -0500
Subject: [Bro-Dev] JIRA to GitHub ticket migration plan
Message-ID: <CAMzgZ0JDfVHg015M7y=1cwJhEVOWEPqijYiX_y8UYHFyUhJuWw@mail.gmail.com>

A preview of what migrated issues will look like along with new labeling scheme:

    https://github.com/jsiwek/test/issues

The selection strategy of which tickets to migrate is something like
"anything that is a reproducible bug or a simple/uncontroversial
enhancement".  Any tickets not in that preview list will be left open
in JIRA, but frozen as read-only.  Going forward, JIRA will serve as a
historical archive or Good Idea Backlog that we can occasionally pull
from, but that can be done manually whenever the situation comes up
(just create a GH issue and link to old JIRA ticket).

Remaining tasks:

* Perform the actual migration
* Update website/docs with new directions/process for issues
* Email bro + bro-dev mailing lists once tickets are migration
* Adapt workflow permissions for JIRA "BIT" project so it's read-only

I expect I'll do those next Monday unless there's objections or
feedback to address.

- Jon

From jazoff at illinois.edu  Fri Sep 14 06:54:02 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Fri, 14 Sep 2018 13:54:02 +0000
Subject: [Bro-Dev] Bro 2.6-beta plans
In-Reply-To: <CAHT3Yk8krmKh298c7f-oXK2i2=tGGb7npJmmmdYMrVwdcDGV+g@mail.gmail.com>
References: <CAMzgZ0K0=x5d63j1xSFyn3qUAmqqraC_1TkeC+0_f=UMNf_a7A@mail.gmail.com>
	<DDFEFC57-B541-4BFF-B86E-DA3BE1E04CA7@illinois.edu>
	<CAMzgZ0J8JfU_SD7e9D8Z9SYU-OhQcOPQpasE_cjKhFV-qzTRRA@mail.gmail.com>
	<CAMnzEagEXORKM7W62H=GTGqGrWpg5p+Lybh-4DmfY=H3s-mkVw@mail.gmail.com>
	<CAMzgZ0L9f7NnMUycXjZxMOPq5b3bvHGLDQAUUw1bAx6PbBCGzA@mail.gmail.com>
	<CAHT3Yk8krmKh298c7f-oXK2i2=tGGb7npJmmmdYMrVwdcDGV+g@mail.gmail.com>
Message-ID: <F99B9C9B-95EA-49E7-BA32-37C72F74A35E@illinois.edu>


> On Sep 13, 2018, at 5:57 PM, John Althouse <john at alt.house> wrote:
> 
> Could anyone update Bro on docker to include 2.5.5, 2.6-beta and master? https://hub.docker.com/r/broplatform/bro/tags/
> 
> We use this with our internal trybro instance which is fantastic for quickly collaborating and testing scripts. :)

Working on it now :-)

There isn't an official 2.6 beta, but I will build and upload 2.5.5 and current master.


? 
Justin Azoff



From robin at corelight.com  Fri Sep 14 07:54:24 2018
From: robin at corelight.com (Robin Sommer)
Date: Fri, 14 Sep 2018 07:54:24 -0700
Subject: [Bro-Dev] JIRA to GitHub ticket migration plan
In-Reply-To: <CAMzgZ0JDfVHg015M7y=1cwJhEVOWEPqijYiX_y8UYHFyUhJuWw@mail.gmail.com>
References: <CAMzgZ0JDfVHg015M7y=1cwJhEVOWEPqijYiX_y8UYHFyUhJuWw@mail.gmail.com>
Message-ID: <20180914145424.GF56640@corelight.com>

On Thu, Sep 13, 2018 at 19:39 -0500, Jon Siwek wrote:

> A preview of what migrated issues will look like along with new labeling scheme:

Looks great, nice job. The only thing I noticed is that the labels are
quite long, making the list of tickets appear somewhat crowded. Could
we skip the prefixes ("Type:", "Component:") and instead use colors to
encode them? So, say, all types would be green, all components yellow
(which they already are), etc.

> Remaining tasks:

We are leaving switching to github as authoritative source for the
repositories to later, right? Doing it all at the same time could
avoid confusion ("everything is on github now" is an easier
statement), but would also make the process more complex. Maybe the
real question here is if we want to switch repositories before or
after 2.6?

Robin

-- 
Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com

From jsiwek at corelight.com  Fri Sep 14 09:36:09 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Fri, 14 Sep 2018 11:36:09 -0500
Subject: [Bro-Dev] JIRA to GitHub ticket migration plan
In-Reply-To: <20180914145424.GF56640@corelight.com>
References: <CAMzgZ0JDfVHg015M7y=1cwJhEVOWEPqijYiX_y8UYHFyUhJuWw@mail.gmail.com>
	<20180914145424.GF56640@corelight.com>
Message-ID: <CAMzgZ0LPV14LWP9PshK6EVa4OowTbCxvzjj7s-e_iw_2AbLHYw@mail.gmail.com>

On Fri, Sep 14, 2018 at 9:54 AM Robin Sommer <robin at corelight.com> wrote:
>
> On Thu, Sep 13, 2018 at 19:39 -0500, Jon Siwek wrote:
>
> > A preview of what migrated issues will look like along with new labeling scheme:
>
> The only thing I noticed is that the labels are
> quite long, making the list of tickets appear somewhat crowded. Could
> we skip the prefixes ("Type:", "Component:") and instead use colors to
> encode them?

I did some label tweaking and reduced some prefix names: "Component"
-> "Area" and "Difficulty" -> "Pain".

It's possible the color design could be done more carefully, but the
basic idea is the colors are mostly encoding importance/difficulty.
Green = go, yellow = caution, red = stop.  That may help people scan
the list to find potential things to work on related to their
abilities or available time.

A problem with removing the prefixes totally is that labels are sorted
by name.  So when one is viewing issues, they can get inconsistent
label orders between issues: the priority label could be at the start,
end, or middle of the label set.  A more annoying thing is that when
someone is trying to add labels to a new issue, they have to jump
around a list to find which labels to add and there's not a clear end
point.  With prefixes, I can scan the list from top to bottom: pick an
area, pick a pain, pick a priority, pick a type, done.  Without
prefixes, it's also less intuitive whether one should be mixing
various labels together and accidentally pick two types that aren't
meant to go together.  Especially for those that are color blind,
having a secondary way (the prefix) to distinguish label categories is
a simple solution without having to do more color design and
verification effort.   I don't know enough to say how much a problem
the current colors actually would be for what number of people, but
trying to be thoughtful just in case.

> We are leaving switching to github as authoritative source for the
> repositories to later, right? Doing it all at the same time could
> avoid confusion ("everything is on github now" is an easier
> statement), but would also make the process more complex. Maybe the
> real question here is if we want to switch repositories before or
> after 2.6?

I'd like to do the steps separately.  Moving tickets shouldn't make
the location of the git repo any more confusing than it is already.
Some users are already using github instead of bro.org and devs are
likely less-confusable anyway (or if not, it's still not a big deal if
someone accidentally pushes commits to a non-master branch of github
instead of bro.org).

- Jon

From jsiwek at corelight.com  Fri Sep 14 11:45:21 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Fri, 14 Sep 2018 13:45:21 -0500
Subject: [Bro-Dev] JIRA to GitHub ticket migration plan
In-Reply-To: <20180914145424.GF56640@corelight.com>
References: <CAMzgZ0JDfVHg015M7y=1cwJhEVOWEPqijYiX_y8UYHFyUhJuWw@mail.gmail.com>
	<20180914145424.GF56640@corelight.com>
Message-ID: <CAMzgZ0+-529xSJWeS6YtzBcgktxeuQq2pAx4mcnxT5t0JTuYfA@mail.gmail.com>

On Fri, Sep 14, 2018 at 9:54 AM Robin Sommer <robin at corelight.com> wrote:
>
> real question here is if we want to switch repositories before or
> after 2.6?

Forgot to say that my sentiment is conditional -- if I find that it's
not disruptive/risky to the release process (e.g. whether I need to
make commits within bro itself), then changing whenever seems fine,
else wait until after 2.6.

It's maybe a week or two away before I have more complete
investigation/answer, but it's possibly that risky.  Here's the
general things I'll be looking into and fleshing out:

* GitHub "protected branches" for permissions and access control
* Which private repos can actually be moved to GitHub (we'll need to
update to Team Account)
* Where all in the docs uses the old bro.org git URL.
* How to handle commit email notifications.
* How decouple-able is the script-reference building process.  I'm
guessing I can just point to the github url and it will work, but
maybe there's some more we can offload from bro.org to Travis or AWS
at this point as well.

Anything else to worry about?

- Jon

From robin at corelight.com  Fri Sep 14 18:24:26 2018
From: robin at corelight.com (Robin Sommer)
Date: Fri, 14 Sep 2018 18:24:26 -0700
Subject: [Bro-Dev] JIRA to GitHub ticket migration plan
In-Reply-To: <CAMzgZ0LPV14LWP9PshK6EVa4OowTbCxvzjj7s-e_iw_2AbLHYw@mail.gmail.com>
References: <CAMzgZ0JDfVHg015M7y=1cwJhEVOWEPqijYiX_y8UYHFyUhJuWw@mail.gmail.com>
	<20180914145424.GF56640@corelight.com>
	<CAMzgZ0LPV14LWP9PshK6EVa4OowTbCxvzjj7s-e_iw_2AbLHYw@mail.gmail.com>
Message-ID: <20180915012426.GD18614@corelight.com>



On Fri, Sep 14, 2018 at 11:36 -0500, Jonathan Siwek wrote:

> I did some label tweaking and reduced some prefix names: "Component"
> -> "Area" and "Difficulty" -> "Pain".

Ok, thanks, makes sense.

Robin

-- 
Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com

From robin at corelight.com  Fri Sep 14 18:28:24 2018
From: robin at corelight.com (Robin Sommer)
Date: Fri, 14 Sep 2018 18:28:24 -0700
Subject: [Bro-Dev] JIRA to GitHub ticket migration plan
In-Reply-To: <CAMzgZ0+-529xSJWeS6YtzBcgktxeuQq2pAx4mcnxT5t0JTuYfA@mail.gmail.com>
References: <CAMzgZ0JDfVHg015M7y=1cwJhEVOWEPqijYiX_y8UYHFyUhJuWw@mail.gmail.com>
	<20180914145424.GF56640@corelight.com>
	<CAMzgZ0+-529xSJWeS6YtzBcgktxeuQq2pAx4mcnxT5t0JTuYfA@mail.gmail.com>
Message-ID: <20180915012824.GE18614@corelight.com>



On Fri, Sep 14, 2018 at 13:45 -0500, Jonathan Siwek wrote:

> Anything else to worry about?

Are Jenkins and Coverity already pulling from GitHub?

I don't know if there's anything we can do on the old server to make
existing clones deal with the relocation more gracefully. I don't
wthink there's a way just redirect a git client, but maybe we could get
some error message into the output or something? Not sure.

Robin

-- 
Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com

From vlad at es.net  Tue Sep 18 05:34:38 2018
From: vlad at es.net (Vlad Grigorescu)
Date: Tue, 18 Sep 2018 12:34:38 +0000
Subject: [Bro-Dev] JIRA to GitHub ticket migration plan
In-Reply-To: <20180915012824.GE18614@corelight.com>
References: <CAMzgZ0JDfVHg015M7y=1cwJhEVOWEPqijYiX_y8UYHFyUhJuWw@mail.gmail.com>
	<20180914145424.GF56640@corelight.com>
	<CAMzgZ0+-529xSJWeS6YtzBcgktxeuQq2pAx4mcnxT5t0JTuYfA@mail.gmail.com>
	<20180915012824.GE18614@corelight.com>
Message-ID: <CAPqbkwsz=+2Hwmf3HOL+cOuW_KyNEbBUpvsk309MBRv73_sx5g@mail.gmail.com>

On Sat, Sep 15, 2018 at 1:28 AM Robin Sommer <robin at corelight.com> wrote:

> Are Jenkins and Coverity already pulling from GitHub?
>

No, I thought Jenkins was pushing to Coverity.

Is the plan to have GitHub issues within each repo? That is, bro, binpac,
etc. I think we'd lose the easy way to see all issues, but if I recall,
there was a way in GitHub to see issues across a few repos. Maybe by
organization?

  --Vlad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180918/f66e710a/attachment.html 

From jsiwek at corelight.com  Tue Sep 18 08:37:54 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Tue, 18 Sep 2018 10:37:54 -0500
Subject: [Bro-Dev] JIRA to GitHub ticket migration plan
In-Reply-To: <CAPqbkwsz=+2Hwmf3HOL+cOuW_KyNEbBUpvsk309MBRv73_sx5g@mail.gmail.com>
References: <CAMzgZ0JDfVHg015M7y=1cwJhEVOWEPqijYiX_y8UYHFyUhJuWw@mail.gmail.com>
	<20180914145424.GF56640@corelight.com>
	<CAMzgZ0+-529xSJWeS6YtzBcgktxeuQq2pAx4mcnxT5t0JTuYfA@mail.gmail.com>
	<20180915012824.GE18614@corelight.com>
	<CAPqbkwsz=+2Hwmf3HOL+cOuW_KyNEbBUpvsk309MBRv73_sx5g@mail.gmail.com>
Message-ID: <CAMzgZ0KdN_Z6d2t1-tsmZZPKgw4Z1tji7g8xHzSHabPcsiEO1A@mail.gmail.com>

On Tue, Sep 18, 2018 at 7:35 AM Vlad Grigorescu <vlad at es.net> wrote:
>
> On Sat, Sep 15, 2018 at 1:28 AM Robin Sommer <robin at corelight.com> wrote:
>>
>> Are Jenkins and Coverity already pulling from GitHub?
>
> No, I thought Jenkins was pushing to Coverity.

Travis is triggered by commits on GitHub and looks like it has a
monthly cron to upload to Coverity.

However, some submodule + testing repos that Travis clones are
actually using the bro.org repo instead of github.com URL, but that's
maybe not something to try to change now, but rather wait until
performing the actual move.

> Is the plan to have GitHub issues within each repo? That is, bro, binpac, etc. I think we'd lose the easy way to see all issues,

Yeah, issues are better kept within the associated GH repo.  We rarely
saw JIRA tickets made for non-bro subcomponents and a good portion of
people were already using GitHub issues for those anyway.

> but if I recall, there was a way in GitHub to see issues across a few repos. Maybe by organization?

Probably thinking of "project boards".  You can create an
organization-wide board that contains issues from separate repos.  I
haven't looked extensively at project boards, but I don't think it's
something we'll generally use because you have to manually add issues
to boards rather than treat them like an automated filter based on
issue labels/status/etc.  That could make it more useful when
organizing specific long-term projects, but still likely decide to use
it on a case-by-case basis rather than generally.

- Jon

From jsiwek at corelight.com  Wed Sep 19 13:00:36 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Wed, 19 Sep 2018 15:00:36 -0500
Subject: [Bro-Dev] 2.6-beta and 2.7 parallel development
Message-ID: <CAMzgZ0JB=EqdzDZh3La1mrhAR1omg29_u+wQSAiqYGtwz4A+sg@mail.gmail.com>

Due to how our version tagging works and to prevent things destined
for 2.7 getting incorrectly labelled as part of 2.6-beta in CHANGES,
I'm going to start a dev/2.7 branch.

* Any new commits that should be included for final 2.6: merge to master
* Any new commits that are meant for 2.7: merge to dev/2.7

After a final 2.6 release is tagged, dev/2.7 can then be merged into master.

- Jon

From dopheide at es.net  Thu Sep 20 12:13:36 2018
From: dopheide at es.net (Michael Dopheide)
Date: Thu, 20 Sep 2018 14:13:36 -0500
Subject: [Bro-Dev] 2.6-beta build error
Message-ID: <CAFfiZAWcZs8Rg1AeCLw_kgrkU-oNE7Erz8=ia32Q_2Q47fjBPA@mail.gmail.com>

I'm hitting the same error when trying to build release/2.6 or master:

[ 99%] In file included from /usr/local/src/bro/src/Func.cc:702:0:
bro.bif: In constructor ?MMDB::MMDB(const char*, stat)?:
bro.bif:3630:47: error: cannot convert ?stat? to ?__dev_t {aka long
unsigned int}? in initialization

CentOS Linux release 7.5.1804

Let me know if you need any other information.  As a point of comparison,
bro-2.5-897 builds fine.

-Dop
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180920/cae48ebd/attachment.html 

From jazoff at illinois.edu  Fri Sep 21 07:22:18 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Fri, 21 Sep 2018 14:22:18 +0000
Subject: [Bro-Dev] 2.6-beta build error
In-Reply-To: <CAFfiZAWcZs8Rg1AeCLw_kgrkU-oNE7Erz8=ia32Q_2Q47fjBPA@mail.gmail.com>
References: <CAFfiZAWcZs8Rg1AeCLw_kgrkU-oNE7Erz8=ia32Q_2Q47fjBPA@mail.gmail.com>
Message-ID: <A1640E7B-A185-4BD7-AFCF-255990356EA8@illinois.edu>

> On Sep 20, 2018, at 3:13 PM, Michael Dopheide <dopheide at es.net> wrote:
> 
> 
> I'm hitting the same error when trying to build release/2.6 or master:
> 
> [ 99%] In file included from /usr/local/src/bro/src/Func.cc:702:0:
> bro.bif: In constructor ?MMDB::MMDB(const char*, stat)?:
> bro.bif:3630:47: error: cannot convert ?stat? to ?__dev_t {aka long unsigned int}? in initialization
> 
> CentOS Linux release 7.5.1804
> 
> Let me know if you need any other information.  As a point of comparison, bro-2.5-897 builds fine.

Looks like travis-ci didn't catch this because the travis-job script does not install libmaxminddb-devel/libmaxminddb-dev
so bro is not built with mmdb support.

? 
Justin Azoff




From jsiwek at corelight.com  Fri Sep 21 08:24:23 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Fri, 21 Sep 2018 10:24:23 -0500
Subject: [Bro-Dev] 2.6-beta build error
In-Reply-To: <CAFfiZAWcZs8Rg1AeCLw_kgrkU-oNE7Erz8=ia32Q_2Q47fjBPA@mail.gmail.com>
References: <CAFfiZAWcZs8Rg1AeCLw_kgrkU-oNE7Erz8=ia32Q_2Q47fjBPA@mail.gmail.com>
Message-ID: <CAMzgZ0JTvMJBMYyBi2ksd8_XpUyHGKgAjCZ2oYS7CiATh60fzQ@mail.gmail.com>

On Thu, Sep 20, 2018 at 2:14 PM Michael Dopheide <dopheide at es.net> wrote:
>
> bro.bif:3630:47: error: cannot convert ?stat? to ?__dev_t {aka long unsigned int}? in initialization

Thanks, it's fixed by a simple patch [1] now in master.

- Jon

[1] https://github.com/bro/bro/commit/d7097635f4b1ec9adca548b81ee42968629987a5


From jazoff at illinois.edu  Fri Sep 21 09:58:14 2018
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Fri, 21 Sep 2018 16:58:14 +0000
Subject: [Bro-Dev] 2.6-beta build error
In-Reply-To: <CAMzgZ0JTvMJBMYyBi2ksd8_XpUyHGKgAjCZ2oYS7CiATh60fzQ@mail.gmail.com>
References: <CAFfiZAWcZs8Rg1AeCLw_kgrkU-oNE7Erz8=ia32Q_2Q47fjBPA@mail.gmail.com>
	<CAMzgZ0JTvMJBMYyBi2ksd8_XpUyHGKgAjCZ2oYS7CiATh60fzQ@mail.gmail.com>
Message-ID: <B107F5A7-8616-45FB-AA8C-EDB88D7C2F50@illinois.edu>


> On Sep 21, 2018, at 11:24 AM, Jon Siwek <jsiwek at corelight.com> wrote:
> 
> On Thu, Sep 20, 2018 at 2:14 PM Michael Dopheide <dopheide at es.net> wrote:
>> 
>> bro.bif:3630:47: error: cannot convert ?stat? to ?__dev_t {aka long unsigned int}? in initialization
> 
> Thanks, it's fixed by a simple patch [1] now in master.
> 
> - Jon
> 
> [1] https://github.com/bro/bro/commit/d7097635f4b1ec9adca548b81ee42968629987a5

That fixed the build, but I think I found a new issue.

If bro is built with maxminddb support but the db files are not present yet, bro logs this twice for each connection:

1537548892.975875       Reporter::ERROR Failed to open GeoIP location database (lookup_location(Conn::c$id$resp_h))     /srv/bro/spool/installed-scripts-do-not-touch/site/ncsa/./bro-scripts/./conn-add-geodata.bro, line 23
1537548892.978846       Reporter::ERROR Failed to open GeoIP location database (lookup_location(Conn::c$id$orig_h))     /srv/bro/spool/installed-scripts-do-not-touch/site/ncsa/./bro-scripts/./conn-add-geodata.bro, line 19

I think I remember the older geoip bifs logging this as well, but only once at startup, not once per call to lookup_location.

? 
Justin Azoff



From jsiwek at corelight.com  Fri Sep 21 11:32:01 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Fri, 21 Sep 2018 13:32:01 -0500
Subject: [Bro-Dev] 2.6-beta build error
In-Reply-To: <B107F5A7-8616-45FB-AA8C-EDB88D7C2F50@illinois.edu>
References: <CAFfiZAWcZs8Rg1AeCLw_kgrkU-oNE7Erz8=ia32Q_2Q47fjBPA@mail.gmail.com>
	<CAMzgZ0JTvMJBMYyBi2ksd8_XpUyHGKgAjCZ2oYS7CiATh60fzQ@mail.gmail.com>
	<B107F5A7-8616-45FB-AA8C-EDB88D7C2F50@illinois.edu>
Message-ID: <CAMzgZ0KwFjmjVqKcyfv08FtCvY-Z4B2Vz48Gr+zP9RWWLn1arg@mail.gmail.com>

On Fri, Sep 21, 2018 at 12:00 PM Azoff, Justin S <jazoff at illinois.edu> wrote:

> I think I remember the older geoip bifs logging this as well, but only once at startup, not once per call to lookup_location.

Yes, looks like it used to be that way.  Changed via [1] now in master.

- Jon

[1] https://github.com/bro/bro/commit/2ede95422bab0c4853b8db7cc15fe5c6e77aa75a

From brot212 at googlemail.com  Sun Sep 23 07:58:05 2018
From: brot212 at googlemail.com (DW)
Date: Sun, 23 Sep 2018 16:58:05 +0200
Subject: [Bro-Dev] S7Comm/S7CommPlus Analyzer
Message-ID: <7bf27d0f-2612-e3a1-82c0-eee5b0063951@googlemail.com>

Hi there,

as a part of my master's thesis I wrote a protocol analyzer for the 
Siemens protocols S7Comm and S7CommPlus.

I posted the code on my GitHub repository, which can be seen here:

https://github.com/dw2102/S7Comm-Analyzer

The S7Comm nearly covers all protocol functions and was tested on .pcap 
files I generated with a Siemens S7-1204 and other files which I found 
on other GitHub repositores.
The S7CommPlus analyzer isn't finished yet. It covers all base 
functions, but without handeling the data of the packets. There is a lot 
to do, like fragmentation, parsing of data, testing etc. which I 
couldn't do, because it would have exceeded my time limit.

Feel free to use, modify or share it.

Dane


From vlad at es.net  Mon Sep 24 08:13:37 2018
From: vlad at es.net (Vlad Grigorescu)
Date: Mon, 24 Sep 2018 15:13:37 +0000
Subject: [Bro-Dev] S7Comm/S7CommPlus Analyzer
In-Reply-To: <7bf27d0f-2612-e3a1-82c0-eee5b0063951@googlemail.com>
References: <7bf27d0f-2612-e3a1-82c0-eee5b0063951@googlemail.com>
Message-ID: <CAPqbkwsCxQ=ZJysUc=bosqtBpmcDYPoe0tk-m+GFQ8oNLfjRcA@mail.gmail.com>

Hi Dane,

Thanks for sending this along. I'll have to check it out.

One thing I noticed -- do you mind filling out the license in
COPYING.edit-me? Without a valid copyright, it's hard to figure out what
all we can do this. Thanks,

  --Vlad

On Sun, Sep 23, 2018 at 3:04 PM DW <brot212 at googlemail.com> wrote:

> Hi there,
>
> as a part of my master's thesis I wrote a protocol analyzer for the
> Siemens protocols S7Comm and S7CommPlus.
>
> I posted the code on my GitHub repository, which can be seen here:
>
> https://github.com/dw2102/S7Comm-Analyzer
>
> The S7Comm nearly covers all protocol functions and was tested on .pcap
> files I generated with a Siemens S7-1204 and other files which I found
> on other GitHub repositores.
> The S7CommPlus analyzer isn't finished yet. It covers all base
> functions, but without handeling the data of the packets. There is a lot
> to do, like fragmentation, parsing of data, testing etc. which I
> couldn't do, because it would have exceeded my time limit.
>
> Feel free to use, modify or share it.
>
> Dane
>
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180924/d3dbd3b8/attachment.html 

From brot212 at googlemail.com  Mon Sep 24 09:37:12 2018
From: brot212 at googlemail.com (Dane Wullen)
Date: Mon, 24 Sep 2018 18:37:12 +0200
Subject: [Bro-Dev] S7Comm/S7CommPlus Analyzer
In-Reply-To: <CAPqbkwsCxQ=ZJysUc=bosqtBpmcDYPoe0tk-m+GFQ8oNLfjRcA@mail.gmail.com>
References: <7bf27d0f-2612-e3a1-82c0-eee5b0063951@googlemail.com>
	<CAPqbkwsCxQ=ZJysUc=bosqtBpmcDYPoe0tk-m+GFQ8oNLfjRcA@mail.gmail.com>
Message-ID: <CAOUkO_QVx2japx1kR-meyzsLQM5PbUJAbmiWRsN4jmLnG01XhQ@mail.gmail.com>

Hi Vlad,

oh yeah, I forgot to do so. I will update the COPYING file as soon as I get
home.

Dane

Vlad Grigorescu <vlad at es.net> schrieb am Mo., 24. Sep. 2018, 17:13:

> Hi Dane,
>
> Thanks for sending this along. I'll have to check it out.
>
> One thing I noticed -- do you mind filling out the license in
> COPYING.edit-me? Without a valid copyright, it's hard to figure out what
> all we can do this. Thanks,
>
>   --Vlad
>
> On Sun, Sep 23, 2018 at 3:04 PM DW <brot212 at googlemail.com> wrote:
>
>> Hi there,
>>
>> as a part of my master's thesis I wrote a protocol analyzer for the
>> Siemens protocols S7Comm and S7CommPlus.
>>
>> I posted the code on my GitHub repository, which can be seen here:
>>
>> https://github.com/dw2102/S7Comm-Analyzer
>>
>> The S7Comm nearly covers all protocol functions and was tested on .pcap
>> files I generated with a Siemens S7-1204 and other files which I found
>> on other GitHub repositores.
>> The S7CommPlus analyzer isn't finished yet. It covers all base
>> functions, but without handeling the data of the packets. There is a lot
>> to do, like fragmentation, parsing of data, testing etc. which I
>> couldn't do, because it would have exceeded my time limit.
>>
>> Feel free to use, modify or share it.
>>
>> Dane
>>
>> _______________________________________________
>> bro-dev mailing list
>> bro-dev at bro.org
>> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180924/e8ce7a32/attachment.html 

From vallentin at icir.org  Thu Sep 27 01:47:51 2018
From: vallentin at icir.org (Matthias Vallentin)
Date: Thu, 27 Sep 2018 10:47:51 +0200
Subject: [Bro-Dev] Module prefix in sending and receiving Broker events
In-Reply-To: <CAMzgZ0LP_wF4Rc=Yf4j_9620xvbFDkn9vMkDikqu+92GmvCp0A@mail.gmail.com>
References: <20180912150857.GE47519@tenzir.com>
	<CAMzgZ0LQqhRKfSAvo=T184DU8iA5q8nfi1umVra4R-9e2Z4M6Q@mail.gmail.com>
	<CADTpMNagdawF6g3RPKaQAUiKaAJTMVDGL2JhYOSo7Kptu-OPYQ@mail.gmail.com>
	<CAMzgZ0LP_wF4Rc=Yf4j_9620xvbFDkn9vMkDikqu+92GmvCp0A@mail.gmail.com>
Message-ID: <20180927084751.GV47519@tenzir.com>

>It might also help if you send actual examples that can be run if that
>still doesn't work because it's hard to interpret what you mean by
>"publish via Broker".  

Okay, so first a pure Bro example. This one works:

    module Foo;

    global foo: event(); // declaration

    event foo()
      {
      print "foo";
      }

    event bro_init()
      {
      event Foo::foo();
      }

If I remove the event declaration, I get:

    error in ./foo.bro, line 10: not an event (Foo::foo())

If I remove the Foo:: qualification in bro_init *and* the declaration, I 
get the expected output again. If I don't remove the declaration, 
nothing gets printed.

Now with Broker, a working example:

    redef exit_only_after_terminate = T;

    module Foo;

    global foo: event(); // declaration

    event foo()
      {
      print "foo";
      }

    event Broker::peer_added(endpoint: Broker::EndpointInfo, msg: string)
      {
      Broker::publish("foo", Broker::make_event(foo));
      }

    event Broker::peer_lost(endpoint: Broker::EndpointInfo, msg: string)
      {
      terminate();
      }

    event bro_init()
      {
      Broker::subscribe("foo");
      Broker::peer("localhost", 55555/tcp);
      }

The Broker Python script:

    #!/usr/bin/env python

    import broker
    import broker.bro

    # Setup endpoint and subscribers.
    endpoint = broker.Endpoint()
    endpoint.listen("localhost", 55555)
    subscriber = endpoint.make_subscriber("foo")
    while True:
        topic, data = subscriber.get()
        print(data)
        endpoint.publish("foo", broker.bro.Event("Foo::foo"))

The Python script prints

    [<broker._broker.Count object at 0x10ac9f5e0>, <broker._broker.Count object at 0x10ace2880>, ['Foo::foo', []]]

and the Bro script "foo". So far so good. If I remove the event 
declaration, the Python script prints

    [<broker._broker.Count object at 0x10ac7c810>, <broker._broker.Count object at 0x10a04e8b8>, ['foo', []]]

and the Bro script nothing.

I hope this illustrates the issue a bit better.

    Matthias

-- 
Matthias Vallentin
CEO
 
Tenzir GmbH
c/o Regus
Valentinskamp 24
20354 Hamburg
 
info at tenzir.com
+49 40 209337260
 
Commercial Register Hamburg
HRB 148081
VAT Number DE314897407
 
CEOs:
Dr. Matthias Vallentin
Dominik Charousset

From jan.grashoefer at gmail.com  Thu Sep 27 04:49:17 2018
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Thu, 27 Sep 2018 13:49:17 +0200
Subject: [Bro-Dev] Module prefix in sending and receiving Broker events
In-Reply-To: <20180927084751.GV47519@tenzir.com>
References: <20180912150857.GE47519@tenzir.com>
	<CAMzgZ0LQqhRKfSAvo=T184DU8iA5q8nfi1umVra4R-9e2Z4M6Q@mail.gmail.com>
	<CADTpMNagdawF6g3RPKaQAUiKaAJTMVDGL2JhYOSo7Kptu-OPYQ@mail.gmail.com>
	<CAMzgZ0LP_wF4Rc=Yf4j_9620xvbFDkn9vMkDikqu+92GmvCp0A@mail.gmail.com>
	<20180927084751.GV47519@tenzir.com>
Message-ID: <22feb22b-f9b8-3cce-2125-45dbc74d4ecc@gmail.com>

On 27/09/2018 10:47, Matthias Vallentin wrote:
> If I remove the event declaration, I get:
> 
>      error in ./foo.bro, line 10: not an event (Foo::foo())

The following works for me:

     module Foo;

     event Foo::foo()
       {
       print "foo";
       }

     event bro_init()
       {
       event Foo::foo();
       }

Just as Jon suggested:
> Adding the
> namespace scoping always and everywhere means to the event
> declaration, the handlers, event/schedule dispatching, any strings
> that contain the event name, etc.

Event handlers just don't seem to inherit the surrounding module 
namespace, which kind of makes sense if you want to handle events 
generated in the global namespace.

Jan

From jsiwek at corelight.com  Thu Sep 27 08:12:16 2018
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 27 Sep 2018 10:12:16 -0500
Subject: [Bro-Dev] Module prefix in sending and receiving Broker events
In-Reply-To: <20180927084751.GV47519@tenzir.com>
References: <20180912150857.GE47519@tenzir.com>
	<CAMzgZ0LQqhRKfSAvo=T184DU8iA5q8nfi1umVra4R-9e2Z4M6Q@mail.gmail.com>
	<CADTpMNagdawF6g3RPKaQAUiKaAJTMVDGL2JhYOSo7Kptu-OPYQ@mail.gmail.com>
	<CAMzgZ0LP_wF4Rc=Yf4j_9620xvbFDkn9vMkDikqu+92GmvCp0A@mail.gmail.com>
	<20180927084751.GV47519@tenzir.com>
Message-ID: <CAMzgZ0Kq5Rj95db-TGP6ABQ1K5E27TCRqLt881vpq62WBDYdqg@mail.gmail.com>

On Thu, Sep 27, 2018 at 3:47 AM Matthias Vallentin <vallentin at icir.org> wrote:

> I hope this illustrates the issue a bit better.

Thanks, that helps.  Underlying issue/pitfall still seems related to
module interactions with event handler definition/declaration, so
added a link back from [#163].  Namely, seems you can try to reason
about events + modules by trying to remember that event declarations
will implicitly get prefixed by module names while event handlers
don't (i.e. a handler for something that hasn't been declared yet
isn't considered an implicit declaration... or at least not the same
kind of declaration).  It's not in the scope of Bro 2.6 to try
improving anything here, so I still suggest the simplest rule/approach
to remember is to always/everywhere prefix event names with an
intended module name.

- Jon

[#163] https://github.com/bro/bro/issues/163

From vallentin at icir.org  Thu Sep 27 08:27:02 2018
From: vallentin at icir.org (Matthias Vallentin)
Date: Thu, 27 Sep 2018 17:27:02 +0200
Subject: [Bro-Dev] Module prefix in sending and receiving Broker events
In-Reply-To: <22feb22b-f9b8-3cce-2125-45dbc74d4ecc@gmail.com>
References: <20180912150857.GE47519@tenzir.com>
	<CAMzgZ0LQqhRKfSAvo=T184DU8iA5q8nfi1umVra4R-9e2Z4M6Q@mail.gmail.com>
	<CADTpMNagdawF6g3RPKaQAUiKaAJTMVDGL2JhYOSo7Kptu-OPYQ@mail.gmail.com>
	<CAMzgZ0LP_wF4Rc=Yf4j_9620xvbFDkn9vMkDikqu+92GmvCp0A@mail.gmail.com>
	<20180927084751.GV47519@tenzir.com>
	<22feb22b-f9b8-3cce-2125-45dbc74d4ecc@gmail.com>
Message-ID: <CADTpMNa2zv8sFzhF+dU9gKreqcyJsXNPqb5E0uQmu+wbSPKc_g@mail.gmail.com>

Jan:
> Event handlers just don't seem to inherit the surrounding module
> namespace, which kind of makes sense if you want to handle events
> generated in the global namespace.

I agree that it does make sense in that regard. It's certainly prudent
- as mentioned - to always qualify your events. The problematic part
that I see is that absence/presence of a forward declaration
interferes with module scoping. This subtlety took me a while to
figure out.

Jon:
>  [..] so I still suggest the simplest rule/approach
> to remember is to always/everywhere prefix event names with an
> intended module name.

Yeah, learned that the hard way. ;-)

    Matthias