From rgentz at asu.edu  Tue Aug  8 11:04:23 2017
From: rgentz at asu.edu (Reinhard Gentz)
Date: Tue, 8 Aug 2017 11:04:23 -0700
Subject: [Bro-Dev] Check if table element exists
Message-ID: <CALtrmTcTDrMZM=4OHQFKkZw5fcTjn_tmjWj-9aA_OnBwP4O_Qg@mail.gmail.com>

Hi,

I would like to check if a certain table element exists and then take
corresponding action like the following:

if (exists(mytable["my_dynamic_name"]))
    do something
else
    do something else


Can someone give me a hint?
Reinhard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20170808/bde6d8ed/attachment.html 

From johanna at icir.org  Tue Aug  8 11:24:09 2017
From: johanna at icir.org (Johanna Amann)
Date: Tue, 08 Aug 2017 11:24:09 -0700
Subject: [Bro-Dev] Check if table element exists
In-Reply-To: <CALtrmTcTDrMZM=4OHQFKkZw5fcTjn_tmjWj-9aA_OnBwP4O_Qg@mail.gmail.com>
References: <CALtrmTcTDrMZM=4OHQFKkZw5fcTjn_tmjWj-9aA_OnBwP4O_Qg@mail.gmail.com>
Message-ID: <17EF111B-B669-4F7D-8875-AF3935204B68@icir.org>

Use "in":

if ("my_dynamic_name" in mytable)

Johanna

On 8 Aug 2017, at 11:04, Reinhard Gentz wrote:

> Hi,
>
> I would like to check if a certain table element exists and then take
> corresponding action like the following:
>
> if (exists(mytable["my_dynamic_name"]))
>     do something
> else
>     do something else
>
>
> Can someone give me a hint?
> Reinhard
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

From jazoff at illinois.edu  Tue Aug  8 11:24:25 2017
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Tue, 8 Aug 2017 18:24:25 +0000
Subject: [Bro-Dev] Check if table element exists
In-Reply-To: <CALtrmTcTDrMZM=4OHQFKkZw5fcTjn_tmjWj-9aA_OnBwP4O_Qg@mail.gmail.com>
References: <CALtrmTcTDrMZM=4OHQFKkZw5fcTjn_tmjWj-9aA_OnBwP4O_Qg@mail.gmail.com>
Message-ID: <02104C7B-7911-45DB-A939-F44FCEE46867@illinois.edu>

You're looking for 'in'

http://try.bro.org/#/?example=basics-composite-types-table


-- 
- Justin Azoff

> On Aug 8, 2017, at 2:04 PM, Reinhard Gentz <rgentz at asu.edu> wrote:
> 
> Hi,
> 
> I would like to check if a certain table element exists and then take corresponding action like the following:
> 
> if (exists(mytable["my_dynamic_name"]))
>     do something
> else
>     do something else
> 
> 
> Can someone give me a hint?
> Reinhard
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev



From asharma at lbl.gov  Tue Aug  8 11:27:38 2017
From: asharma at lbl.gov (Aashish Sharma)
Date: Tue, 8 Aug 2017 11:27:38 -0700
Subject: [Bro-Dev] Check if table element exists
In-Reply-To: <CALtrmTcTDrMZM=4OHQFKkZw5fcTjn_tmjWj-9aA_OnBwP4O_Qg@mail.gmail.com>
References: <CALtrmTcTDrMZM=4OHQFKkZw5fcTjn_tmjWj-9aA_OnBwP4O_Qg@mail.gmail.com>
Message-ID: <20170808182736.GA11759@MacPro.local>

(Not sure if I am interpreting your question right but here is how I read it) 

basically use "in" operator


local my_ip_table : table[addr] of bool ; 

local ip: addr = 127.0.0.1 

if ( ip in my_ip_table)
	found 
else 
	not found 


btw, you can also use "!in" operator too which is rather more handy 


if (something !in table)
	initialize it here
	table[something] = Blah; 


or 

most of scripts just flat return in membership doesn't exist that helps
eliminate a lot of unneeded run through scripts. 

if (ip !in Site::local_nets)
	return 

<do rest of the heuristics here for local IPs> 


Here is a (rather complicated but) useful example which stretches
above problem with an extended use-case:

/usr/local/bro/share/bro/policy/frameworks/software/vulnerable.bro

Hope this helps. Let me know if you need more clearifications. 

Aashish


On Tue, Aug 08, 2017 at 11:04:23AM -0700, Reinhard Gentz wrote:
> Hi,
> 
> I would like to check if a certain table element exists and then take
> corresponding action like the following:
> 
> if (exists(mytable["my_dynamic_name"]))
>     do something
> else
>     do something else
> 
> 
> Can someone give me a hint?
> Reinhard

> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev


From rgentz at asu.edu  Tue Aug  8 11:38:51 2017
From: rgentz at asu.edu (Reinhard Gentz)
Date: Tue, 8 Aug 2017 11:38:51 -0700
Subject: [Bro-Dev] Check if table element exists
In-Reply-To: <20170808182736.GA11759@MacPro.local>
References: <CALtrmTcTDrMZM=4OHQFKkZw5fcTjn_tmjWj-9aA_OnBwP4O_Qg@mail.gmail.com>
	<20170808182736.GA11759@MacPro.local>
Message-ID: <CALtrmTdta6XKRRvj787H8zA=0vkVxC3sBeof_Y3iZDNarLbBjw@mail.gmail.com>

Thank you! Exactly what I was looking for

On Tue, Aug 8, 2017 at 11:27 AM, Aashish Sharma <asharma at lbl.gov> wrote:

> (Not sure if I am interpreting your question right but here is how I read
> it)
>
> basically use "in" operator
>
>
> local my_ip_table : table[addr] of bool ;
>
> local ip: addr = 127.0.0.1
>
> if ( ip in my_ip_table)
>         found
> else
>         not found
>
>
> btw, you can also use "!in" operator too which is rather more handy
>
>
> if (something !in table)
>         initialize it here
>         table[something] = Blah;
>
>
> or
>
> most of scripts just flat return in membership doesn't exist that helps
> eliminate a lot of unneeded run through scripts.
>
> if (ip !in Site::local_nets)
>         return
>
> <do rest of the heuristics here for local IPs>
>
>
> Here is a (rather complicated but) useful example which stretches
> above problem with an extended use-case:
>
> /usr/local/bro/share/bro/policy/frameworks/software/vulnerable.bro
>
> Hope this helps. Let me know if you need more clearifications.
>
> Aashish
>
>
> On Tue, Aug 08, 2017 at 11:04:23AM -0700, Reinhard Gentz wrote:
> > Hi,
> >
> > I would like to check if a certain table element exists and then take
> > corresponding action like the following:
> >
> > if (exists(mytable["my_dynamic_name"]))
> >     do something
> > else
> >     do something else
> >
> >
> > Can someone give me a hint?
> > Reinhard
>
> > _______________________________________________
> > bro-dev mailing list
> > bro-dev at bro.org
> > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20170808/0e9647bf/attachment.html 

From asharma at lbl.gov  Fri Aug 25 13:56:13 2017
From: asharma at lbl.gov (Aashish Sharma)
Date: Fri, 25 Aug 2017 13:56:13 -0700
Subject: [Bro-Dev] input-framework file locations
In-Reply-To: <20160709004104.GA49737@icir.org>
References: <20160708235945.GJ36120@mac-4.local>
	<20160709004104.GA49737@icir.org>
Message-ID: <CAAzQiAYjaQ-1LoeKNtuqUxtTj4pfjoiw0jB+8UwhBWP7j6Hoaw@mail.gmail.com>

[ re-igniting an OLD thread ]

OK so @DIR sort of works.

I've used this as

 global smtp_indicator_feed= fmt
("%s/feeds/smtp_malicious_indicators.out", at DIR) &redef ;

Problem is: @DIR gives the path of the directory where script is residing.

So when I do broctl install - all the scripts go into :
../spool/installed-scripts-do-not-touch/

so while file is referenced correctly and input-data is read just
fine. Humans or no other process can now append to the input file
anymore.

Does the problem make sense ?

I think I am looking for something which can point back to a 'static' path.

On Fri, Jul 8, 2016 at 5:41 PM, Robin Sommer <robin at icir.org> wrote:
>
>
> On Fri, Jul 08, 2016 at 16:59 -0700, you wrote:
>
>> Something similar to __load__.bro model
>
> @DIR gives you the path to the directory the current script is located
> in. Does that help?
>
> Robin
>

===

Original thread:

I have been thinking and trying different things but for now, it
appears that if we are to share policies around, there is no easy way
to be able to distribute input-files along with policy files.

Basically, right now I use

redef Scan::whitelist_ip_file = "/usr/local/bro/feeds/ip-whitelist.scan" ;

and then expect everyone to edit path as their setup demands it and
place accompanying sample file in the directory or create one for
themselves  - this all introduces errors as well as slows down
deployment.

Is there a way I can use relative paths instead of absolute  paths for
input-framework digestion.  At present a new-heuristics dir can have
__load__.bro with all policies but input-framework won't read files
relative to that directory or where it is placed.

redef Scan::whitelist_ip_file = "../feeds/ip-whitelist.scan" ;

Something similar to __load__.bro model

Also, one question I have is should all input-files go to a 'standard'
feeds/input dir in bro or be scattered around along with their
accompanied bro policies (ie in individual directories )

Something to think about as with more and more reliance on
input-framework i think there is a need for 'standardization' on where
to put input-files and how to easily find and read them.

From robin at icir.org  Sat Aug 26 08:12:07 2017
From: robin at icir.org (Robin Sommer)
Date: Sat, 26 Aug 2017 08:12:07 -0700
Subject: [Bro-Dev] send_id (Re: [Bro-Commits] [git/bro]
 topic/jsiwek/actor-system: Finish port of control framework to use broker.
 (8dddae1))
In-Reply-To: <201708251827.v7PIRCbp021912@bro-ids.icir.org>
References: <201708251827.v7PIRCbp021912@bro-ids.icir.org>
Message-ID: <20170826151207.GA18388@icir.org>

Jon, replacing send_id() may indeed work better with an extension at
the C++/Broker level. I'd like to avoid introducing new dependencies
on Bro's serialization code, as I'm very much hoping that once the old
communication code code goes we won't need that serialization layer
anymore either (I know we're using it for opaque values over Broker
too, but that's quite contained and should be easy to replace).

Some thoughts here:

     - I'm thinking the best approach may be a new Bro-specific
       message for Broker, similar to the log-create/write messages.
       We can add that to the Bro shim in Broker. It'd send the name
       of the ID and the new value as a broker::data instance, and the
       receiving Bro updates the value as the message comes in.

     - There's one larger problem with replacing send_id() though: the
       old communication system has logic to send large values
       incrementally, so that send_id() won't block stuff. As Seth
       reminded me the SumStats framework is relying on that quite
       extensively for sending tables around. Incremental operation is
       something we don't have with Broker. I think that's ok, we can
       replace the few existing use cases of sending large values with
       something else. For SumStats that should probably be data
       stores. I don't remember if there even are further instances of
       this, my guess is no (I don't think we need to consider
       broctl's configuration updating here; those values are small
       and a non-incrememtnal send_id() is fine).

     - Another approach for broctl's updating could be switching to
       the upcoming configuration framework, which takes a different
       approach to dynamic reconfiguration. However, it's a bit out
       still until we can that switch completely, so for now providing
       a substitute for send_id() that can cover the simple uses cases
       looks like the best way to go.

Robin

On Fri, Aug 25, 2017 at 13:15 -0500, Jonathan Siwek wrote:

> commit 8dddae17db9340f5261d11382aa9b67e965d5fef
> Author: Jon Siwek <jsiwek at illinois.edu>
> Date:   Fri Aug 25 13:15:00 2017 -0500
> 
>     Finish port of control framework to use broker.
>     
>     To replace usages of the send_id BIF, I had to make 3 new BIFs
>     
>     - serialize
>     - deserialize
>     - update_ID
>     
>     Using those, any Bro value can be explicitly converted to a string
>     of bytes, sent to a peer via a Broker event, unserialized to a Bro
>     val on the other side and then installed into a global identifier
>     via its name.
>     
>     I think this may be the most straightforward way to get this to
>     work for now, at least without changing significantly the Broker
>     internals or messaging format.  It mostly boils down to not being
>     able to deserialize into a Bro value with the 'any' type, at least
>     not without also carrying Bro's actual type information somewhere
>     inside Broker's serialized message.
>     
>     And I think deserializing into 'any' would be needed because it's
>     not possible to e.g. explicitly enumerate all possible types in a Bro
>     script and have a particular event signature to use for any given one.
>     That's not possible because there's infinite ways you can create
>     composite types (tables, sets, vectors, etc).

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From seth at corelight.com  Sat Aug 26 19:03:51 2017
From: seth at corelight.com (Seth Hall)
Date: Sun, 27 Aug 2017 02:03:51 +0000
Subject: [Bro-Dev] send_id (Re: [Bro-Commits] [git/bro]
 topic/jsiwek/actor-system: Finish port of control framework to use broker.
 (8dddae1))
In-Reply-To: <20170826151207.GA18388@icir.org>
References: <201708251827.v7PIRCbp021912@bro-ids.icir.org>
	<20170826151207.GA18388@icir.org>
Message-ID: <CAAAo3g9YfcbJJdF5QHiE0FhuA+otzmg0FDY8houm7nonXmAe1Q@mail.gmail.com>

I believe that Robin meant the intel framework instead of sumstats.
(Hopefully this avoids some confusion)

  .Seth

On Sat, Aug 26, 2017 at 11:12 AM Robin Sommer <robin at icir.org> wrote:

> Jon, replacing send_id() may indeed work better with an extension at
> the C++/Broker level. I'd like to avoid introducing new dependencies
> on Bro's serialization code, as I'm very much hoping that once the old
> communication code code goes we won't need that serialization layer
> anymore either (I know we're using it for opaque values over Broker
> too, but that's quite contained and should be easy to replace).
>
> Some thoughts here:
>
>      - I'm thinking the best approach may be a new Bro-specific
>        message for Broker, similar to the log-create/write messages.
>        We can add that to the Bro shim in Broker. It'd send the name
>        of the ID and the new value as a broker::data instance, and the
>        receiving Bro updates the value as the message comes in.
>
>      - There's one larger problem with replacing send_id() though: the
>        old communication system has logic to send large values
>        incrementally, so that send_id() won't block stuff. As Seth
>        reminded me the SumStats framework is relying on that quite
>        extensively for sending tables around. Incremental operation is
>        something we don't have with Broker. I think that's ok, we can
>        replace the few existing use cases of sending large values with
>        something else. For SumStats that should probably be data
>        stores. I don't remember if there even are further instances of
>        this, my guess is no (I don't think we need to consider
>        broctl's configuration updating here; those values are small
>        and a non-incrememtnal send_id() is fine).
>
>      - Another approach for broctl's updating could be switching to
>        the upcoming configuration framework, which takes a different
>        approach to dynamic reconfiguration. However, it's a bit out
>        still until we can that switch completely, so for now providing
>        a substitute for send_id() that can cover the simple uses cases
>        looks like the best way to go.
>
> Robin
>
> On Fri, Aug 25, 2017 at 13:15 -0500, Jonathan Siwek wrote:
>
> > commit 8dddae17db9340f5261d11382aa9b67e965d5fef
> > Author: Jon Siwek <jsiwek at illinois.edu>
> > Date:   Fri Aug 25 13:15:00 2017 -0500
> >
> >     Finish port of control framework to use broker.
> >
> >     To replace usages of the send_id BIF, I had to make 3 new BIFs
> >
> >     - serialize
> >     - deserialize
> >     - update_ID
> >
> >     Using those, any Bro value can be explicitly converted to a string
> >     of bytes, sent to a peer via a Broker event, unserialized to a Bro
> >     val on the other side and then installed into a global identifier
> >     via its name.
> >
> >     I think this may be the most straightforward way to get this to
> >     work for now, at least without changing significantly the Broker
> >     internals or messaging format.  It mostly boils down to not being
> >     able to deserialize into a Bro value with the 'any' type, at least
> >     not without also carrying Bro's actual type information somewhere
> >     inside Broker's serialized message.
> >
> >     And I think deserializing into 'any' would be needed because it's
> >     not possible to e.g. explicitly enumerate all possible types in a Bro
> >     script and have a particular event signature to use for any given
> one.
> >     That's not possible because there's infinite ways you can create
> >     composite types (tables, sets, vectors, etc).
>
> --
> Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>
-- 
Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20170827/6c7a4abd/attachment.html 

From jan.grashoefer at gmail.com  Mon Aug 28 02:09:50 2017
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Mon, 28 Aug 2017 11:09:50 +0200
Subject: [Bro-Dev] send_id (Re: [Bro-Commits] [git/bro]
 topic/jsiwek/actor-system: Finish port of control framework to use broker.
 (8dddae1))
In-Reply-To: <CAAAo3g9YfcbJJdF5QHiE0FhuA+otzmg0FDY8houm7nonXmAe1Q@mail.gmail.com>
References: <201708251827.v7PIRCbp021912@bro-ids.icir.org>
	<20170826151207.GA18388@icir.org>
	<CAAAo3g9YfcbJJdF5QHiE0FhuA+otzmg0FDY8houm7nonXmAe1Q@mail.gmail.com>
Message-ID: <f775639b-cc6b-6671-09ef-adf7f91091fb@gmail.com>

On 27/08/17 04:03, Seth Hall wrote:
> I believe that Robin meant the intel framework instead of sumstats.
> (Hopefully this avoids some confusion)

Thanks for the clarification! I was thinking about send_id() in context 
of the intel framework as well. As you might noticed, I enjoyed playing 
around with the intel framework :) Thus, some questions to make sure I 
got everything correctly:

> On Sat, Aug 26, 2017 at 11:12 AM Robin Sommer <robin at icir.org> wrote:
> 
>> Jon, replacing send_id() may indeed work better with an extension at
>> the C++/Broker level. I'd like to avoid introducing new dependencies
>> on Bro's serialization code, as I'm very much hoping that once the old
>> communication code code goes we won't need that serialization layer
>> anymore either (I know we're using it for opaque values over Broker
>> too, but that's quite contained and should be easy to replace).

So sending opaque values will still be possible using broker, right?

>>       - There's one larger problem with replacing send_id() though: the
>>         old communication system has logic to send large values
>>         incrementally, so that send_id() won't block stuff. As Seth
>>         reminded me the SumStats framework is relying on that quite
>>         extensively for sending tables around. Incremental operation is
>>         something we don't have with Broker. I think that's ok, we can
>>         replace the few existing use cases of sending large values with
>>         something else. For SumStats that should probably be data
>>         stores.

As far as I understand the broker data stores (straight forward 
key-value stores), a data store does not fit for the intel framework, as 
it uses e.g. the patricia-trie implementation in tables to efficiently 
match subnets. Additionally I was thinking about using cuckoo-filters, 
implemented as opaque type, to further improve matching on workers.

However, the intel framework uses send_id() only initially to transfer 
the current min_data_store to newly connected workers. Every further 
update is handled "manually". Thus I guess there would be two options:
1. Sending all data at once. Maybe ok for that use case.
2. Sending stuff incrementally using some script-layer logic.
Am I right here?

Thanks,
Jan

From robin at icir.org  Mon Aug 28 07:59:02 2017
From: robin at icir.org (Robin Sommer)
Date: Mon, 28 Aug 2017 07:59:02 -0700
Subject: [Bro-Dev] send_id (Re: [Bro-Commits] [git/bro]
 topic/jsiwek/actor-system: Finish port of control framework to use broker.
 (8dddae1))
In-Reply-To: <f775639b-cc6b-6671-09ef-adf7f91091fb@gmail.com>
References: <201708251827.v7PIRCbp021912@bro-ids.icir.org>
	<20170826151207.GA18388@icir.org>
	<CAAAo3g9YfcbJJdF5QHiE0FhuA+otzmg0FDY8houm7nonXmAe1Q@mail.gmail.com>
	<f775639b-cc6b-6671-09ef-adf7f91091fb@gmail.com>
Message-ID: <20170828145902.GE17704@icir.org>



On Mon, Aug 28, 2017 at 11:09 +0200, you wrote:

> Thanks for the clarification! I was thinking about send_id() in context 
> of the intel framework as well.

Yep, I meant Intel framework of course. :)

> So sending opaque values will still be possible using broker, right?

Yes, correct (one downside of opaque values is that only Bro itself
can send & receive them, for external Broker clients they will remain,
um, opaque :)

> As far as I understand the broker data stores (straight forward 
> key-value stores), a data store does not fit for the intel framework, as 
> it uses e.g. the patricia-trie implementation in tables to efficiently 
> match subnets.

Good point. Support for efficient subnet lookups is something we
should probably add to Broker stores at some point, but that's for the
future.

> 1. Sending all data at once. Maybe ok for that use case.

That would be ok for the new Intel client I think, but sending the
whole thing will put load on the sending Bro as well, which could be a
problem. It depends on the volume of the data of course, it's hard to
say where the limit is.

> 2. Sending stuff incrementally using some script-layer logic.

This might be the best way to go then. In the future I'd like to have
a script-level framework that offers some higher-level communication
patterns on top of Broker, like this one: "send large table safely".
For now, the Intel framework could implement that itself and then
maybe we can even reuse that implementation later by making it more
generic.

Robin


-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From jsiwek at illinois.edu  Mon Aug 28 12:53:32 2017
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Mon, 28 Aug 2017 19:53:32 +0000
Subject: [Bro-Dev] send_id (Re: [Bro-Commits] [git/bro]
 topic/jsiwek/actor-system: Finish port of control framework to use broker.
 (8dddae1))
In-Reply-To: <20170826151207.GA18388@icir.org>
References: <201708251827.v7PIRCbp021912@bro-ids.icir.org>
	<20170826151207.GA18388@icir.org>
Message-ID: <D04715FA-C815-4151-A4F7-4B433850DB1E@illinois.edu>


> On Aug 26, 2017, at 10:12 AM, Robin Sommer <robin at icir.org> wrote:
> 
> Jon, replacing send_id() may indeed work better with an extension at
> the C++/Broker level.

>     - I'm thinking the best approach may be a new Bro-specific
>       message for Broker, similar to the log-create/write messages.
>       We can add that to the Bro shim in Broker. It'd send the name
>       of the ID and the new value as a broker::data instance, and the
>       receiving Bro updates the value as the message comes in.

Sounds ok.  Were you going to work on adding such a message or want me to try?

>     - There's one larger problem with replacing send_id() though: the
>       old communication system has logic to send large values
>       incrementally, so that send_id() won't block stuff.

Thanks, I had missed that fact.

Generally for porting scripts to use Broker, I was probably going to end up doing the most straightforward thing I can think of to just get things to work and hope people follow commit messages well enough to call out anything questionable.  Do you think there?s a better process for capturing high-level requirements earlier?

- Jon


From robin at icir.org  Mon Aug 28 16:34:30 2017
From: robin at icir.org (Robin Sommer)
Date: Mon, 28 Aug 2017 16:34:30 -0700
Subject: [Bro-Dev] send_id (Re: [Bro-Commits] [git/bro]
 topic/jsiwek/actor-system: Finish port of control framework to use broker.
 (8dddae1))
In-Reply-To: <D04715FA-C815-4151-A4F7-4B433850DB1E@illinois.edu>
References: <201708251827.v7PIRCbp021912@bro-ids.icir.org>
	<20170826151207.GA18388@icir.org>
	<D04715FA-C815-4151-A4F7-4B433850DB1E@illinois.edu>
Message-ID: <20170828233430.GF43180@icir.org>



On Mon, Aug 28, 2017 at 19:53 +0000, you wrote:

> Sounds ok.  Were you going to work on adding such a message or want me to try?

I can work on it, but it'll probably take me a few days to get to it.
If can make progress with other stuff in the meantime, I'll do that;
otherwise give it a try if you want.

> Generally for porting scripts to use Broker, I was probably going to
> end up doing the most straightforward thing I can think of to just get
> things to work and hope people follow commit messages well enough to
> call out anything questionable.

That works for me. The downside for you might be a bit of wasted time
if discussion afterwards leads to a different approach. So feel free
to also ask for feedback upfront, whatever seems best. Either way,
getting it to work by applying the most straightforward approach is
definitely the right rule of thumb.

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From seth at corelight.com  Mon Aug 28 18:48:42 2017
From: seth at corelight.com (Seth Hall)
Date: Mon, 28 Aug 2017 21:48:42 -0400
Subject: [Bro-Dev] input-framework file locations
In-Reply-To: <CAAzQiAYjaQ-1LoeKNtuqUxtTj4pfjoiw0jB+8UwhBWP7j6Hoaw@mail.gmail.com>
References: <20160708235945.GJ36120@mac-4.local>
	<20160709004104.GA49737@icir.org>
	<CAAzQiAYjaQ-1LoeKNtuqUxtTj4pfjoiw0jB+8UwhBWP7j6Hoaw@mail.gmail.com>
Message-ID: <A89BA4FB-C0DB-4065-9860-E891E296F8E9@corelight.com>



On 25 Aug 2017, at 16:56, Aashish Sharma wrote:

>  global smtp_indicator_feed= fmt
> ("%s/feeds/smtp_malicious_indicators.out", at DIR) &redef ;
>
> Problem is: @DIR gives the path of the directory where script is 
> residing.
>
> So when I do broctl install - all the scripts go into :
> ../spool/installed-scripts-do-not-touch/

Huh, that's definitely a problem that I can see limiting people.  What 
you might want to do is reference a particular directory and having 
instructions for people that they need to make it writable by the user 
running the Bro process (and the directory could be redef-able).

Alternately, it looks like you're only using that to persist state 
across executions.  Is that right?  If you're doing that, then you could 
possibly get away with storing in $TMP.

Once Broker is in Bro, you can use Broker data stores to store and 
retrieve your data.

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com

From jan.grashoefer at gmail.com  Tue Aug 29 00:59:50 2017
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Tue, 29 Aug 2017 09:59:50 +0200
Subject: [Bro-Dev] send_id (Re: [Bro-Commits] [git/bro]
 topic/jsiwek/actor-system: Finish port of control framework to use broker.
 (8dddae1))
In-Reply-To: <20170828145902.GE17704@icir.org>
References: <201708251827.v7PIRCbp021912@bro-ids.icir.org>
	<20170826151207.GA18388@icir.org>
	<CAAAo3g9YfcbJJdF5QHiE0FhuA+otzmg0FDY8houm7nonXmAe1Q@mail.gmail.com>
	<f775639b-cc6b-6671-09ef-adf7f91091fb@gmail.com>
	<20170828145902.GE17704@icir.org>
Message-ID: <25905f3a-3ed4-ee0e-deb8-949110cc7087@gmail.com>

On 28/08/17 16:59, Robin Sommer wrote:
>> 2. Sending stuff incrementally using some script-layer logic.
> 
> This might be the best way to go then. In the future I'd like to have
> a script-level framework that offers some higher-level communication
> patterns on top of Broker, like this one: "send large table safely".
> For now, the Intel framework could implement that itself and then
> maybe we can even reuse that implementation later by making it more
> generic.

I agree, that seems to be the preferable solution in anticipation of a 
higher-level communication framework. I will keep following the broker 
development and give it a try.

Thanks,
Jan

From jazoff at illinois.edu  Wed Aug 30 09:32:18 2017
From: jazoff at illinois.edu (Azoff, Justin S)
Date: Wed, 30 Aug 2017 16:32:18 +0000
Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/jsiwek/actor-system:
 Add Broker::publish_id function to replace send_id (096ec69)
In-Reply-To: <201708292006.v7TK65NZ000935@bro-ids.icir.org>
References: <201708292006.v7TK65NZ000935@bro-ids.icir.org>
Message-ID: <A44F3766-BBFF-42DE-A01A-0CA5643D9485@illinois.edu>


> On Aug 29, 2017, at 3:58 PM, Jonathan Siwek <jsiwek at ncsa.illinois.edu> wrote:
> 
>    A minor/unrelated detail of this change is that it also allows
>    'pattern' values to be exchanged over broker messages.  It currently
>    does this via Bro's serialization, but can likely be done via the
>    pattern's textual representation instead (need to look into it more).

That's interesting.. Does that somehow get around the bro limitation that patterns can not be constructed at runtime?

-- 
- Justin Azoff



From rajeev.k.agrawal at gmail.com  Wed Aug 30 11:05:23 2017
From: rajeev.k.agrawal at gmail.com (Rajeev Agrawal)
Date: Wed, 30 Aug 2017 13:05:23 -0500
Subject: [Bro-Dev] Bro events hierarchy
Message-ID: <CAPTmXgR+C7Xmko-LU4gJORzufX-7WF4QOv5e4F+Ba3j9RS1iGQ@mail.gmail.com>

Hi

Little background of my question: I have access to Bro logs (conn,ftp,http,
files etc.) and want to use them for machine learning. It will be really
helpful to know the Bro logs relationship.

I am looking for a list of all Bro events hierarchy. For example, HTTP
session will generate a conn, http and files events. I just want to
understand how these logs are created based on a particular event. TLS/SSL
will generate conn, ssl, files and x509 events,

Is there any such document available or is there any other way to figure
this out?

Thanks for your help

Rajeev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20170830/936e957d/attachment.html 

From jsiwek at illinois.edu  Thu Aug 31 07:59:05 2017
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Thu, 31 Aug 2017 14:59:05 +0000
Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/jsiwek/actor-system:
 Add Broker::publish_id function to replace send_id (096ec69)
In-Reply-To: <A44F3766-BBFF-42DE-A01A-0CA5643D9485@illinois.edu>
References: <201708292006.v7TK65NZ000935@bro-ids.icir.org>
	<A44F3766-BBFF-42DE-A01A-0CA5643D9485@illinois.edu>
Message-ID: <3B1B40AC-F403-4AFB-9EAD-199AC49B3F8E@illinois.edu>


> On Aug 30, 2017, at 11:32 AM, Azoff, Justin S <jazoff at illinois.edu> wrote:
> 
>> On Aug 29, 2017, at 3:58 PM, Jonathan Siwek <jsiwek at ncsa.illinois.edu> wrote:
>> 
>>   A minor/unrelated detail of this change is that it also allows
>>   'pattern' values to be exchanged over broker messages.  It currently
>>   does this via Bro's serialization, but can likely be done via the
>>   pattern's textual representation instead (need to look into it more).
> 
> That's interesting.. Does that somehow get around the bro limitation that patterns can not be constructed at runtime?

In the sense that a Bro instance can receive a new pattern value from another Bro instance, yes, but each individual Bro instance is still limited to creating pattern values at init time (AFAIK).  So each time you wanted to change a given pattern, you?d need to restart the Bro instance that is doing the sending and have it init the pattern with the desired value.  I?m not sure if there?s other ways to e.g. dynamically create new patterns from arbitrary strings, I think Seth had showed me some hacky way to do it before, but I forgot what it was.

Also note that this doesn?t differ from what the ?send_id? BIF can currently do.  Maybe the commit description made it seem like it was completely new capability, but I think it was just something that got overlooked when initially creating the Broker framework.

- Jon