From asharma at lbl.gov Fri Sep 8 10:29:15 2017 From: asharma at lbl.gov (Aashish Sharma) Date: Fri, 8 Sep 2017 10:29:15 -0700 Subject: [Bro-Dev] bro-pkg dependencies ? Message-ID: <20170908172913.GA658@MacPro.local> Can we specify dependent packages in bro-pkg and would bro-pkg go and resolve (install) those dependencies by itself ? Also, can we make the bro-pkg dump some output (notes) before? or after? pkg installation - something like see this file for details etc ? Aashish From slagell at illinois.edu Fri Sep 8 11:05:16 2017 From: slagell at illinois.edu (Slagell, Adam J) Date: Fri, 8 Sep 2017 18:05:16 +0000 Subject: [Bro-Dev] bro-pkg dependencies ? In-Reply-To: <20170908172913.GA658@MacPro.local> References: <20170908172913.GA658@MacPro.local> Message-ID: <43D7F0FE-FF1E-4CD7-A1B3-1870998C0213@illinois.edu> At list recursively finding all of those dependencies is on the roadmap. Terry has that on his queue. > On Sep 8, 2017, at 12:29 PM, Aashish Sharma wrote: > > Can we specify dependent packages in bro-pkg and would bro-pkg go and resolve > (install) those dependencies by itself ? > > Also, can we make the bro-pkg dump some output (notes) before? or after? pkg > installation - something like see this file for details etc ? > > Aashish > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From asharma at lbl.gov Fri Sep 8 11:36:03 2017 From: asharma at lbl.gov (Aashish Sharma) Date: Fri, 8 Sep 2017 11:36:03 -0700 Subject: [Bro-Dev] bro-pkg dependencies ? In-Reply-To: <43D7F0FE-FF1E-4CD7-A1B3-1870998C0213@illinois.edu> References: <20170908172913.GA658@MacPro.local> <43D7F0FE-FF1E-4CD7-A1B3-1870998C0213@illinois.edu> Message-ID: <20170908183602.GB658@MacPro.local> Thanks Adam! I think we might also need to introduce the concept of pkg-conflicts too: cannot install B if A is installed. Aashish On Fri, Sep 08, 2017 at 06:05:16PM +0000, Slagell, Adam J wrote: > At list recursively finding all of those dependencies is on the roadmap. Terry has that on his queue. > > > On Sep 8, 2017, at 12:29 PM, Aashish Sharma wrote: > > > > Can we specify dependent packages in bro-pkg and would bro-pkg go and resolve > > (install) those dependencies by itself ? > > > > Also, can we make the bro-pkg dump some output (notes) before? or after? pkg > > installation - something like see this file for details etc ? > > > > Aashish > > _______________________________________________ > > bro-dev mailing list > > bro-dev at bro.org > > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > From seth at corelight.com Fri Sep 8 11:45:21 2017 From: seth at corelight.com (Seth Hall) Date: Fri, 08 Sep 2017 14:45:21 -0400 Subject: [Bro-Dev] bro-pkg dependencies ? In-Reply-To: <20170908172913.GA658@MacPro.local> References: <20170908172913.GA658@MacPro.local> Message-ID: On 8 Sep 2017, at 13:29, Aashish Sharma wrote: > Can we specify dependent packages in bro-pkg and would bro-pkg go and > resolve > (install) those dependencies by itself ? Yep. Here's one that does.. https://github.com/corelight/top-dns/blob/master/bro-pkg.meta > Also, can we make the bro-pkg dump some output (notes) before? or > after? pkg > installation - something like see this file for details etc ? bro-pkg tells you that it's going to install dependencies and asks if you want to proceed. If you want to see how it works, install the corelight/top-dns package. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From asharma at lbl.gov Fri Sep 8 11:56:19 2017 From: asharma at lbl.gov (Aashish Sharma) Date: Fri, 8 Sep 2017 11:56:19 -0700 Subject: [Bro-Dev] bro-pkg dependencies ? In-Reply-To: References: <20170908172913.GA658@MacPro.local> Message-ID: <20170908185618.GC658@MacPro.local> Ah! Nice. Yes, this is what I was looking for. Thanks for the pointer Seth! On Fri, Sep 08, 2017 at 02:45:21PM -0400, Seth Hall wrote: > > > On 8 Sep 2017, at 13:29, Aashish Sharma wrote: > > > Can we specify dependent packages in bro-pkg and would bro-pkg go and > > resolve > > (install) those dependencies by itself ? > > Yep. Here's one that does.. > https://github.com/corelight/top-dns/blob/master/bro-pkg.meta > > > Also, can we make the bro-pkg dump some output (notes) before? or after? > > pkg > > installation - something like see this file for details etc ? > > bro-pkg tells you that it's going to install dependencies and asks if you > want to proceed. If you want to see how it works, install the > corelight/top-dns package. > > .Seth > > -- > Seth Hall * Corelight, Inc * www.corelight.com From mkrenz at iu.edu Wed Sep 13 09:54:23 2017 From: mkrenz at iu.edu (Krenz, Mark Steven) Date: Wed, 13 Sep 2017 16:54:23 +0000 Subject: [Bro-Dev] Adding a non-Bro script packages to bro-pkg Message-ID: <1505321663236.73828@iu.edu> Hello Bro devs. I was talking with Seth yesterday at Bro Con about how an independent command line script for working with Bro logs could fit into bro-pkg. The bawk shell script I wrote (https://github.com/deltaray/bawk) is a command meant to work with Bro logs from the command line. The installation involves putting an executable script into the path of people analyzing Bro logs and making some associated libraries available to that script somewhere. Right now I'm just putting the script in /opt/bro/bin/bawk and the libraries in /opt/bro/lib/bawk. Maybe there is a better place. Seth said that this is not something that is covered by bro-pkg at the moment, but maybe is a category of community contribution that should be considered. On a related note, someone at BroCon also brought up an interesting question about whether bro-pkg could be used to share Intel data with the community. Sounds like bro-pkg could benefit from having additional categories for package types. Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20170913/a24887c4/attachment.html From seth at corelight.com Fri Sep 15 07:42:32 2017 From: seth at corelight.com (Seth Hall) Date: Fri, 15 Sep 2017 10:42:32 -0400 Subject: [Bro-Dev] Adding a non-Bro script packages to bro-pkg In-Reply-To: <1505321663236.73828@iu.edu> References: <1505321663236.73828@iu.edu> Message-ID: <2BC64FA5-1EB2-4A3E-92B2-61750109DECC@corelight.com> On 13 Sep 2017, at 12:54, Krenz, Mark Steven wrote: > Hello Bro devs. I was talking with Seth yesterday at Bro Con about how > an independent command line script for working with Bro logs could fit > into bro-pkg. Yeah, I think that including command line tools as a component of Bro Packages makes sense. I always forget how many little tools are laying around that various people have written to process logs. Having those in the central repository would be really nice. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From klehigh at iu.edu Mon Sep 18 13:43:00 2017 From: klehigh at iu.edu (Keith Lehigh) Date: Mon, 18 Sep 2017 16:43:00 -0400 Subject: [Bro-Dev] ASCII response filetype Message-ID: Hi Folks, I?ve been mulling over an addition to the file mime type signature that consists of ?1 to 16 ASCII readable characters?. 16 is an arbitrary length cutoff. The purpose of this signature would be to log instances where a short status code is returned by a web service. I see lots of responses like ?[]? or ?OK? or ?Success? and currently these are logged in files.log as unknown file types. I think Bro would be improved by logging a filetype for these responses. Using the entire set of readable ASCII characters would make this flexible enough to handle various responses w/o trying to enumerate all possibilities. A downside would be differentiating a short text file. I don?t have much of a solution for that problem at this point, but I?d be open to suggestions. I?m sure there are other downsides I?m not seeing. Thoughts? - Keith -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20170918/efa02d56/attachment.html From seth at corelight.com Mon Sep 18 13:59:31 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 18 Sep 2017 16:59:31 -0400 Subject: [Bro-Dev] ASCII response filetype In-Reply-To: References: Message-ID: On 18 Sep 2017, at 16:43, Keith Lehigh wrote: > Hi Folks, > I?ve been mulling over an addition to the file mime type > signature that consists of ?1 to 16 ASCII readable characters?. > 16 is an arbitrary length cutoff. The purpose of this signature would > be to log instances where a short status code is returned by a web > service. I see lots of responses like ?[]? or ?OK? or > ?Success? and currently these are logged in files.log as unknown > file types. I think Bro would be improved by logging a filetype for > these responses. What about creating a mime type for an enumerated list of all of the ones you find? With a pattern like /^(OK|Success|0|1)$/ That was you could avoid other short responses from getting caught up in the net. I also suspect that [] should be something different because if you see that over HTTP, it's probably in most cases just an empty JSON array. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From robin at icir.org Tue Sep 19 09:39:41 2017 From: robin at icir.org (Robin Sommer) Date: Tue, 19 Sep 2017 09:39:41 -0700 Subject: [Bro-Dev] 'async' keyword Message-ID: <20170919163941.GE76922@icir.org> At BroCon a few folks asked me about the proposed "async" keyword for Bro's scripting language. "async" is coroutine-style language construct that puts blocking operations on hold until they conclude, working on other stuff first. It could replace most uses of "when" and is arguably much nicer to use. "async" is implemented as a proof-of-concept in the topic/robin/async branch. Note that that Bro branch is not fully functional at the moment, nor are performance implications clear. However, before going any further with it I'd like to reach consensus if the current implementation is acceptable for the Bro code base, as it's very low-level and not easy to follow. I'd be interested in opinions here. The commit to look at is: https://github.com/bro/bro/commit/8653b333431648e5a33d61c3f61c6d093cff5b72 The exercise here is: Can you understand how "async" works? (If you can honestly answer "yes" in under 15 minutes, I buy you a beer. ;-) Robin PS: See the TODOs in that commit for the current state of the code.) -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at illinois.edu Tue Sep 19 10:32:52 2017 From: jsiwek at illinois.edu (Siwek, Jon) Date: Tue, 19 Sep 2017 17:32:52 +0000 Subject: [Bro-Dev] 'async' keyword In-Reply-To: <20170919163941.GE76922@icir.org> References: <20170919163941.GE76922@icir.org> Message-ID: <452BDD34-4AB8-483F-8D0F-FE3F69B39CB4@illinois.edu> > On Sep 19, 2017, at 11:39 AM, Robin Sommer wrote: > > The exercise here is: Can you understand how "async" works? (If you > can honestly answer "yes" in under 15 minutes, I buy you a beer. ;-) A feat like that deserves a larger reward. Understanding the new code also requires understanding the context in which it is implemented and I wonder if the later is more of a hurdle here. Maybe it?s understandable for the exercise to take on the order of days/weeks for a person that?s not done much hacking/exploring of the bro internals? Just trying to give a sane reference point so anyone taking the challenge isn't discouraged after only 15 mins :) - Jon From robin at icir.org Tue Sep 19 11:55:04 2017 From: robin at icir.org (Robin Sommer) Date: Tue, 19 Sep 2017 11:55:04 -0700 Subject: [Bro-Dev] 'async' keyword In-Reply-To: <452BDD34-4AB8-483F-8D0F-FE3F69B39CB4@illinois.edu> References: <20170919163941.GE76922@icir.org> <452BDD34-4AB8-483F-8D0F-FE3F69B39CB4@illinois.edu> Message-ID: <20170919185503.GA87766@icir.org> On Tue, Sep 19, 2017 at 17:32 +0000, you wrote: > Understanding the new code also requires understanding the context in > which it is implemented and I wonder if the later is more of a hurdle > here. Hey, this is bro-dev, are you saying not everybody here is intimately familiar with the Bro source code? ;-) So yes, I'll allow for more time to understand the context. :) The point though is that conceptually it's both simple and difficult at the same time. Even without all the context, one might get the basic idea pretty quickly if looking at the right parts---or maybe one doesn't, I don't know, that's part of why I keep looking for feedback. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From johanna at corelight.com Wed Sep 20 15:24:44 2017 From: johanna at corelight.com (Johanna Amann) Date: Wed, 20 Sep 2017 15:24:44 -0700 Subject: [Bro-Dev] Configuration framework syntax proposal Message-ID: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> Hello bro-dev, in this email I want to get feedback on a possible syntax for the configuration framework. The aim of the configuration framework is to provide an easy method for Bro users and script writers to change configuration options during the runtime of Bro (as opposed to only on startup as already possible using redef). Let me start with an example what a script using the configuration framework could look like: ======================================================================== module PcapFilter; export { ## Documentation goes here: ## Set the PCAP filter that will be applied. const filter = "ip" &config="input.pcap.filter"; } redef enum PcapFilterId += { NewFilter }; function install_filter() { precompile_pcap_filter(NewFilter, filter); install_pcap_filter(NewFilter); } # hook that is called when the configuration value changes hook config_change(name: string, old_value: string) { install_filter(); } event bro_init() { # Register and cause the hook to be called on configuration changes Config::register_for_change("input.pcap.filter", config_change); install_filter(); } ======================================================================== The two parts here are the definition of the configuration value and an (optional) hook that can be defined to execute when a configuration option changes. The line 'const filter = "ip" &config="input.pcap.filter"' binds the constant filter to the configuration option input.pcap.filter. When a user updates the configuration option, the const will automatically change. For simple configuration options, this line is all that is required to have a configurable options. For cases in which additional actions have to be performed when a configuration option changes (e.g. calling BIFs), a hook can be registered to listen to the change. The hook will be executed right after the value has been updated, gets access to the previous value, and can perform any necessary actions. Note that this proposal preserves typing: register_for_change will enforce that the hook that is passed in (in this case config_change) will have a signature that fits to the configuration option. If a user desires to change a configuration option from script-land (e.g. in local.bro), this is possible using 'Config::set'. For example: Config::set("input.pcap.filter", "not ip"); Config::set also enforces typing and only accepts variables with the same type as the const that was defined. There obviously are a few more parts to this (storage backends, etc); however these should be very flexible as we aim to implement as much as possible in the script-land. There will be a few more BIFs that are necessary to write the script-part of the config-framework; these are mainly needed for introspection. In this email I am mostly interested in seeing if people think that this syntax is a good idea, or if there are any better way to do this. I already talked to a few people where a few other ideas were brought up, all having different advantages or disadvantages. One idea is to have a different syntax to define the configuration option. Instead of const filter = "ip" &config="input.pcap.filter"; the definition could look like configopt filter = "ip"; In this case, the name of the option would have to be automatically deduced and look like "NameSpace::filter". The syntax is slightly nicer in this case, but the configuration option naming is not as easy to structure. So - I would be happy for any feedback or ideas on this. Johanna From jazoff at illinois.edu Wed Sep 20 16:04:05 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 20 Sep 2017 23:04:05 +0000 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> Message-ID: > On Sep 20, 2017, at 6:24 PM, Johanna Amann wrote: > > > const filter = "ip" &config="input.pcap.filter"; > > the definition could look like > > configopt filter = "ip"; Could the definition be const filter = ?ip? &config; if you just wanted to use NameSpace::filter ? That kinda seems like the best of both worlds? Especially if anything marked &redef was automatically registered as a configuration variable. Thinking of all my scripts that could use this feature I think I would always want NameSpace::option. ? Justin Azoff From commike at reservoir.com Wed Sep 20 16:12:34 2017 From: commike at reservoir.com (Alan Commike) Date: Wed, 20 Sep 2017 16:12:34 -0700 Subject: [Bro-Dev] 'async' keyword In-Reply-To: <20170919163941.GE76922@icir.org> References: <20170919163941.GE76922@icir.org> Message-ID: What are your thoughts on error handling? Exec::run() returns an Exec::Result, which is nice in that we can recover if something goes wrong. I would think one would want most calls of Exec::run() in an async context, but we lose the return value. ...alan On Tue, Sep 19, 2017 at 9:39 AM, Robin Sommer wrote: > At BroCon a few folks asked me about the proposed "async" keyword for > Bro's scripting language. "async" is coroutine-style language > construct that puts blocking operations on hold until they conclude, > working on other stuff first. It could replace most uses of "when" and > is arguably much nicer to use. > > "async" is implemented as a proof-of-concept in the topic/robin/async > branch. Note that that Bro branch is not fully functional at the > moment, nor are performance implications clear. However, before going > any further with it I'd like to reach consensus if the current > implementation is acceptable for the Bro code base, as it's very > low-level and not easy to follow. I'd be interested in opinions here. > > The commit to look at is: > https://github.com/bro/bro/commit/8653b333431648e5a33d61c3f61c6d093cff5b72 > > The exercise here is: Can you understand how "async" works? (If you > can honestly answer "yes" in under 15 minutes, I buy you a beer. ;-) > > Robin > > PS: See the TODOs in that commit for the current state of the code.) > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20170920/76527c1d/attachment.html From jazoff at illinois.edu Wed Sep 20 16:24:47 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 20 Sep 2017 23:24:47 +0000 Subject: [Bro-Dev] 'async' keyword In-Reply-To: References: <20170919163941.GE76922@icir.org> Message-ID: > On Sep 20, 2017, at 7:12 PM, Alan Commike wrote: > > What are your thoughts on error handling? > > Exec::run() returns an Exec::Result, which is nice in that we can recover if something goes wrong. I would think one would want most calls of Exec::run() in an async context, but we lose the return value. > > ...alan It looks like this wouldn?t be lost at all, you would just go from when (local res = Exec::run([$cmd="?"])){ do_something_with(res); } to local res = async Exec::run([$cmd="?"]); do_something_with(res); ? Justin Azoff From johanna at corelight.com Wed Sep 20 17:19:59 2017 From: johanna at corelight.com (Johanna Amann) Date: Wed, 20 Sep 2017 17:19:59 -0700 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> Message-ID: <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> > Could the definition be > > const filter = ?ip? &config; > > if you just wanted to use NameSpace::filter ? That kinda seems like the best of both worlds? Especially if anything marked &redef was automatically registered as a configuration variable. technically - yes. Though I am not quite sure that I like it :). On the redef side - this specifically does not touch the functionality of redef and also does not aim to automatically integrate with redef. The background is that we do not know if a variable that is currently redef-able will work as a configuration variable, or needs additional commands to be run (or just works if set at startup as it is actually the case with a lot of the current consts). I don't think going the route of intermingling that would be a good idea - if someone wants something to be a config variable, I think it should be an explicit opt-in. > Thinking of all my scripts that could use this feature I think I would always want NameSpace::option. Ok. That would actually more pull me to using the other syntax again (configopt varname) and not doing &plugin at all. Thanks a lot :) Johanna From seth at corelight.com Wed Sep 20 18:30:32 2017 From: seth at corelight.com (Seth Hall) Date: Wed, 20 Sep 2017 21:30:32 -0400 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> Message-ID: <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> On 20 Sep 2017, at 20:19, Johanna Amann wrote: > technically - yes. Though I am not quite sure that I like it :). I also don't like it. I think with this proposal there is some recognition that our community has separate and distinct parts (and some overlap between them). The people that program Bro scripts and the people that use Bro scripts. I feel like there are benefits to getting a chance to separate the notion of configuration away from the notion of programming. Invariably, there will be variable names chosen within software that will be short and convenient or long and explanatory but may not end up being just right if someone simply wants to configure a behavior. There's also the problem of single level namespaces which will limit the expressiveness and depth that you could possibly give through configuration keys. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From dnthayer at illinois.edu Wed Sep 20 20:00:33 2017 From: dnthayer at illinois.edu (Daniel Thayer) Date: Wed, 20 Sep 2017 22:00:33 -0500 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> Message-ID: <32abf6e8-2ecd-8349-3c19-20998a9525c6@illinois.edu> On 9/20/17 5:24 PM, Johanna Amann wrote: > advantages or disadvantages. One idea is to have a different syntax to define > the configuration option. Instead of > > const filter = "ip" &config="input.pcap.filter"; > > the definition could look like > > configopt filter = "ip"; > If we decide to use the "const" syntax, then is the plan to allow a const to have both the &config and &redef attributes? (presumably, we wouldn't allow "&redef" with "configopt") From johanna at corelight.com Wed Sep 20 20:51:13 2017 From: johanna at corelight.com (Johanna Amann) Date: Wed, 20 Sep 2017 20:51:13 -0700 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: <32abf6e8-2ecd-8349-3c19-20998a9525c6@illinois.edu> References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> <32abf6e8-2ecd-8349-3c19-20998a9525c6@illinois.edu> Message-ID: <20170921035113.i2vc4pravql5h7j7@Trafalgar.local> On Wed, Sep 20, 2017 at 10:00:33PM -0500, Daniel Thayer wrote: > On 9/20/17 5:24 PM, Johanna Amann wrote: > > advantages or disadvantages. One idea is to have a different syntax to define > > the configuration option. Instead of > > > > const filter = "ip" &config="input.pcap.filter"; > > > > the definition could look like > > > > configopt filter = "ip"; > > > If we decide to use the "const" syntax, then is the plan > to allow a const to have both the &config and &redef attributes? > (presumably, we wouldn't allow "&redef" with "configopt") Actually, yes, that was the thought so far; since they do not interact, they are combineable if someone would desire this. Johanna From jan.grashoefer at gmail.com Thu Sep 21 01:17:53 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Thu, 21 Sep 2017 10:17:53 +0200 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> Message-ID: <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com> On 21/09/17 03:30, Seth Hall wrote: > I also don't like it. I think with this proposal there is some > recognition that our community has separate and distinct parts (and some > overlap between them). The people that program Bro scripts and the > people that use Bro scripts. I feel like there are benefits to getting > a chance to separate the notion of configuration away from the notion of > programming. While I agree that there are two (more or less) distinct groups and that the notion of configuration should be separated from the notion of brogramming, I don't think that anyone would profit from introducing something like &config="just.another.name". Because of the additional name mapping, this will make scripts harder to understand for brogrammers (especially for beginners), whereas the benefit for users would be minimal. > Invariably, there will be variable names chosen within software that > will be short and convenient or long and explanatory but may not end up > being just right if someone simply wants to configure a behavior. I think that would just be bad coding style. > There's also the problem of single level namespaces which will limit the > expressiveness and depth that you could possibly give through > configuration keys. So that is definitively a valid point! But instead of coming up with a "new language element" I would prefer to add support for multi-level namespaces into Bro. As the Bro language is already quite extensive, I think that every new syntax, which does not follow the already existing concepts, reduces usability. That's also why I would choose &config instead of configopt. That all said, what's about the poor users? Instead of throwing a huge config file of key-value-pairs at someone who does not know about the internals of the corresponding scripts, I would provide a UI to them (maybe someone comes up with a bro-package...). The UI could display all options in a structured way and further support configuration by also showing the corresponding documentation for each option. This would allow a clean cut between users and brogrammers without intermingling both worlds. Jan From seth at corelight.com Thu Sep 21 06:18:36 2017 From: seth at corelight.com (Seth Hall) Date: Thu, 21 Sep 2017 09:18:36 -0400 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com> References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com> Message-ID: On 21 Sep 2017, at 4:17, Jan Grash?fer wrote: > While I agree that there are two (more or less) distinct groups and > that > the notion of configuration should be separated from the notion of > brogramming, I don't think that anyone would profit from introducing > something like &config="just.another.name". Because of the additional > name mapping, this will make scripts harder to understand for > brogrammers (especially for beginners), whereas the benefit for users > would be minimal. I don't think that this proposal makes scripts any harder to understand for programmers than things are currently. From a programmers perspective, they would still use variables the same way they currently do. The only difference is that a user might not be using redef to change values, but the programmer doesn't see that happening anyway. It also doesn't change anything about what configuration values the programmer makes available since the programmer is already forced to expose their configuration through the export section. >> Invariably, there will be variable names chosen within software that >> will be short and convenient or long and explanatory but may not end >> up >> being just right if someone simply wants to configure a behavior. > > I think that would just be bad coding style. That's a fair statement. In an ideal world, all programmers would name their variables perfectly. I know that the opportunity for poor naming is still present with the &config attribute value, but for me at least, it puts me in a different frame of mind because this is the name that I'm exposing to users as a configuration option. It almost forces people to step out of the programming mentality. > So that is definitively a valid point! But instead of coming up with a > "new language element" I would prefer to add support for multi-level > namespaces into Bro. Yes, please do this! :) > That all said, what's about the poor users? Instead of throwing a huge > config file of key-value-pairs at someone who does not know about the > internals of the corresponding scripts, I would provide a UI to them > (maybe someone comes up with a bro-package...). The UI could display > all > options in a structured way and further support configuration by also > showing the corresponding documentation for each option. This would > allow a clean cut between users and brogrammers without intermingling > both worlds. Yep, this notion of making things abstract-able into easy configuration interfaces and/or good documentation (using the inline broxygen comments) was always in the proposal, Johanna pointed it out in the original code sample. There is just something about the idea of exposing variable names to users (even if it's wrapped in a gui) that is intensely unpalatable to me. It's pretty much unheard of among other types of software. It would be like exposing internal variable names to command line programs instead of abstracting it into easy flags (i.e. -a or --help) or, if in a gui a text entry box had a label next to it like "GUI::My_Program::user_name" instead of showing "Username". Sometimes abstraction like this isn't warranted, but I think it has to be done here. Bro needs to turn into a platform that treats users as first class citizens in the community and we need to acknowledge that there will be a day that they won't be reading script source code and they won't want to be exposed to programmer-isms. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From jazoff at illinois.edu Thu Sep 21 07:30:10 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 21 Sep 2017 14:30:10 +0000 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com> Message-ID: > On Sep 21, 2017, at 9:18 AM, Seth Hall wrote: > > There is just something about the idea of exposing variable names to > users (even if it's wrapped in a gui) that is intensely unpalatable to > me. It's pretty much unheard of among other types of software. It > would be like exposing internal variable names to command line programs > instead of abstracting it into easy flags (i.e. -a or --help) I guess I was just thinking from my perspective, every script I would write would just have module Foo; export { ## Set the threshold in bytes. const threshold = 1234 &config="foo.threshold"; } And I would just be repeating the namespace + variable name for each option with no added value. It would just become unnecessary repetition and a source of errors: const one = 1234 &config="foo.one; const two = 1234 &config="foo.tow"; #oops const three = 1234 &config="foo.tow"; #oops! I say this as someone that will absolutely screw this up :-) Maybe the design should support renaming variables for the configuration, but programmers should be strongly discouraged from renaming things unless they have a good reason from deviating from the automatic namespace + variable > > or, if in > a gui a text entry box had a label next to it like > "GUI::My_Program::user_name" instead of showing "Username". I'm not sure how exposing something like "input.pcap.filter" is any different from exposing something like "Pcap::filter" from that standpoint. Maybe there's a larger discussion here around what the user experience should look like? I feel like two different things are being talked about now. Directly using variable names in UI elements is not unheard of though, a lot of UI frameworks will do things like present a variable like user_name as "User Name" in the UI. This is usually a simple text transform like >>> s='My_Program::user_name' >>> s.replace("::", " - ").replace("_", " ").title() 'My Program - User Name' This way you don't end up with code that looks something like Args { user_name .. display as "User Name" age .. display as "Age" favorite_color .. display as "Favorite Color" favorite_food .. display as "Favorite Food" pin .. display as "PIN" } Instead you only need to override the display when you have a good reason to deviate from the standard underscore to space and Title Case transform: Args { user_name age favorite_color favorite_food pin .. display as "PIN" } Having a bro configuration tool display something like the current SSH::password_guesses_limit as SSH Password Guesses Limit The number of failed SSH connections before a host is designated as guessing passwords. Type: count Current Value: 30 Or Site::darknet_mode as Site Darknet Mode I just realized I didn't document the variable name itself :-) Type: DarknetMode enum Current Value: DARKNET Choices: DARKNET: Only hosts defined in darknet_address_space are dark NOT_ALLOCATED: Only hosts NOT listed in used_address_space are dark DARKNET_OR_NOT_ALLOCATED: Only hosts defined in darknet_address_space OR NOT listed in used_address_space are dar... DARKNET_AND_NOT_ALLOCATED: Only hosts both defined in darknet_address_space AND NOT listed in used_address... wouldn't be crazy, and such a tool seems like it would be pretty user friendly to me. ? Justin Azoff From jsiwek at illinois.edu Thu Sep 21 07:51:32 2017 From: jsiwek at illinois.edu (Siwek, Jon) Date: Thu, 21 Sep 2017 14:51:32 +0000 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com> Message-ID: <1DCAABCB-8F6E-414B-8C4B-D44AD737E297@illinois.edu> > On Sep 21, 2017, at 8:18 AM, Seth Hall wrote: > > Yep, this notion of making things abstract-able into easy configuration > interfaces and/or good documentation (using the inline broxygen > comments) was always in the proposal, Johanna pointed it out in the > original code sample. Yeah, I was wondering what a UI would currently look like if you tried to use existing functionality, e.g. just identifier names and broxygen comments. Like Jan, I had a hard time understanding the benefit having two names for the same value: the identifier and config string. It seems to push more burden than needed onto script authors, like maybe they don?t really care about a UI, but want the improved configuration capabilities. i.e. maybe the requirements of a UI can be separate from the requirements of the new ?configuration variables? concept. Maybe one thing to do is try to actually build/design your ideal UI and/or configuration tool starting with just the existing Bro functionality. You?ll definitely get an understanding of the low-level requirements that way. i.e. first design/build the most basic user experience that functionally works and then, from that state, add whatever you think will be an improvement. > There is just something about the idea of exposing variable names to > users (even if it's wrapped in a gui) that is intensely unpalatable to > me. It's pretty much unheard of among other types of software. It > would be like exposing internal variable names to command line programs > instead of abstracting it into easy flags (i.e. -a or --help) or, if in > a gui a text entry box had a label next to it like > "GUI::My_Program::user_name" instead of showing "Username". I?m half facetious in bringing it up, but have you seen CMake? https://cmake.org/runningcmake/ - Jon From jsiwek at illinois.edu Thu Sep 21 07:57:31 2017 From: jsiwek at illinois.edu (Siwek, Jon) Date: Thu, 21 Sep 2017 14:57:31 +0000 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com>

Message-ID: > I'm not sure how exposing something like "input.pcap.filter" is any different from exposing something like "Pcap::filter" from that standpoint. Maybe there's a larger discussion here around what the user experience should look like? I feel like two different things are being talked about now. I?m thinking on the same lines. > Directly using variable names in UI elements is not unheard of though, a lot of UI frameworks will do things like present a variable like user_name as "User Name" in the UI. This is usually a simple text transform like > >>>> s='My_Program::user_name' >>>> s.replace("::", " - ").replace("_", " ").title() > 'My Program - User Name? Yeah, I?ve noticed this approach in other UIs as well. - Jon From robin at icir.org Thu Sep 21 08:22:23 2017 From: robin at icir.org (Robin Sommer) Date: Thu, 21 Sep 2017 08:22:23 -0700 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: <1DCAABCB-8F6E-414B-8C4B-D44AD737E297@illinois.edu> References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com> <1DCAABCB-8F6E-414B-8C4B-D44AD737E297@illinois.edu> Message-ID: <20170921152223.GC92700@icir.org> On Thu, Sep 21, 2017 at 14:51 +0000, you wrote: > comments. Like Jan, I had a hard time understanding the benefit having > two names for the same value: the identifier and config string. Yeah, that's been my original concern as well. What if we focused that new attribute just on displaying something to the user: const user_name: string &redef &display_name="User name" A UI would show it as "User name", but everything else (incl. internally the configuration framework) would use My_Program::user_name. This would even work more generically, anything could have a &display_name and we'd have Broxygen pick up on it too. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From johanna at corelight.com Thu Sep 21 08:33:26 2017 From: johanna at corelight.com (Johanna Amann) Date: Thu, 21 Sep 2017 08:33:26 -0700 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: <20170921152223.GC92700@icir.org> References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com> <1DCAABCB-8F6E-414B-8C4B-D44AD737E297@illinois.edu> <20170921152223.GC92700@icir.org> Message-ID: <20170921153326.b3h6ujrn3l6gz6vz@Trafalgar.local> On Thu, Sep 21, 2017 at 08:22:23AM -0700, Robin Sommer wrote: > > comments. Like Jan, I had a hard time understanding the benefit having > > two names for the same value: the identifier and config string. > > Yeah, that's been my original concern as well. What if we focused that > new attribute just on displaying something to the user: > > const user_name: string &redef &display_name="User name" I am not sure that we do need a new language element for that at all. If we want a new attribute for just displaying information in a different way, that (at least to me) feels more like something broxygen would do (i.e. something that a script writer could put into one of the ## comments if they so desire for the respective variable). That being said, I still think it would be nice to have something in the Bro language to denote that a value is a configuration option, mostly for the reasons stated in the very first email. The biggest reason from my point of view is strong typing - we tried to implement this just as Bro scripts and it ends up not so nice. So - how about something like this: ## The username for our new feature ## ## Display: Feature User Name const user_name: string &config; or configopt user_name: string; The comment block identifies the display name which can be picked up by the UI (and documentation generation). The &config attribute (or the configopt specifier) specifies this as a configuration option. The name in the comment blocks also can be used for other language elements and will be used in the documentation. Johanna From johanna at corelight.com Thu Sep 21 08:37:16 2017 From: johanna at corelight.com (Johanna Amann) Date: Thu, 21 Sep 2017 08:37:16 -0700 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com>

Message-ID: <20170921153716.khkli2hpmtvtit55@Trafalgar.local> On Thu, Sep 21, 2017 at 02:57:31PM +0000, Siwek, Jon wrote: > > > I'm not sure how exposing something like "input.pcap.filter" is any different from exposing something like "Pcap::filter" from that standpoint. Maybe there's a larger discussion here around what the user experience should look like? I feel like two different things are being talked about now. > > I?m thinking on the same lines. Yes, I can see that argument. > > Directly using variable names in UI elements is not unheard of though, a lot of UI frameworks will do things like present a variable like user_name as "User Name" in the UI. This is usually a simple text transform like > > > >>>> s='My_Program::user_name' > >>>> s.replace("::", " - ").replace("_", " ").title() > > 'My Program - User Name? > > Yeah, I?ve noticed this approach in other UIs as well. I admittedly did not think of this - that could work as a neat default. The only thing that I would like to avoid (which is obviously separate from this) is internally remapping variable names to configuration names in a non-reversible manner; then one suddenly has to think about what to do when names conflict (several variable names being able to automatically map to the same configuration name). But - that seem to be separate concern :) Johanna From robin at icir.org Thu Sep 21 09:06:01 2017 From: robin at icir.org (Robin Sommer) Date: Thu, 21 Sep 2017 09:06:01 -0700 Subject: [Bro-Dev] 'async' keyword In-Reply-To: References: <20170919163941.GE76922@icir.org> Message-ID: <20170921160601.GF92700@icir.org> On Wed, Sep 20, 2017 at 16:12 -0700, you wrote: > What are your thoughts on error handling? Good question, two parts to that. First, along the lines of Justin's response, the function is free to return whatever it considers the appropiate result for an error case. The only constraint is that it *has* to return something because there's no way to get back without passing on a result. Second, one thing that "async" doesn't have right now is an equivalent of "when"'s "timeout", i.e., alternative code to execute if the asynchronous code takes too long. I'm not sure if we could support that with "async" because we'd need an alternative way to leave the calling function and could run into an issue with memory leaks (not completely sure here, but it's smells similar to the problem we have with propagating interpreter exceptions up the stack). That said, I'm also not sure how a timeout would even fit in syntactically, so maybe the nest answer here is just sticking to "when" if one needs a timeout. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at illinois.edu Thu Sep 21 09:10:49 2017 From: jsiwek at illinois.edu (Siwek, Jon) Date: Thu, 21 Sep 2017 16:10:49 +0000 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: <20170921153716.khkli2hpmtvtit55@Trafalgar.local> References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com>

<20170921153716.khkli2hpmtvtit55@Trafalgar.local> Message-ID: <416FDF3C-4B02-48C5-B480-271F66A7D6F6@illinois.edu> > On Sep 21, 2017, at 10:37 AM, Johanna Amann wrote: > > The only thing that I would like to avoid (which is obviously separate > from this) is internally remapping variable names to configuration names > in a non-reversible manner; then one suddenly has to think about what to > do when names conflict (several variable names being able to automatically > map to the same configuration name). But - that seem to be separate > concern :) Still not sure how much of an issue that is, provided the display names are only for display and not used to actually locate/update identifier values. E.g. if a user sees 2 ?User Name? fields in a UI, I think we?re still able to fall back on the broxygen documentation comments to provide more context to the user. Or if theres standardized/automatic conventions for these display names that are based on modules/namespacing, I?m not sure how often you?d even see such conflicts, or ?d expect they?d get patched out pretty rapidly by the community when they pop up. - Jon From johanna at corelight.com Thu Sep 21 09:13:36 2017 From: johanna at corelight.com (Johanna Amann) Date: Thu, 21 Sep 2017 09:13:36 -0700 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: <416FDF3C-4B02-48C5-B480-271F66A7D6F6@illinois.edu> References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com>

<20170921153716.khkli2hpmtvtit55@Trafalgar.local> <416FDF3C-4B02-48C5-B480-271F66A7D6F6@illinois.edu> Message-ID: <4F9007A8-547A-4606-B940-C712107C7AED@corelight.com> On 21 Sep 2017, at 9:10, Siwek, Jon wrote: >> On Sep 21, 2017, at 10:37 AM, Johanna Amann >> wrote: >> >> The only thing that I would like to avoid (which is obviously >> separate >> from this) is internally remapping variable names to configuration >> names >> in a non-reversible manner; then one suddenly has to think about what >> to >> do when names conflict (several variable names being able to >> automatically >> map to the same configuration name). But - that seem to be separate >> concern :) > > Still not sure how much of an issue that is, provided the display > names are only for display and not used to actually locate/update > identifier values. E.g. if a user sees 2 ?User Name? fields in a > UI, I think we?re still able to fall back on the broxygen > documentation comments to provide more context to the user. Or if > theres standardized/automatic conventions for these display names that > are based on modules/namespacing, I?m not sure how often you?d > even see such conflicts, or ?d expect they?d get patched out > pretty rapidly by the community when they pop up. This actually was my point - which I apparently did not make clear. As long as it is only for display it is not a problem - I just don't want it to be used for identification :) Johanna From vlad at grigorescu.org Thu Sep 21 09:34:12 2017 From: vlad at grigorescu.org (Vlad Grigorescu) Date: Thu, 21 Sep 2017 11:34:12 -0500 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: <20170921152223.GC92700@icir.org> References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com> <1DCAABCB-8F6E-414B-8C4B-D44AD737E297@illinois.edu> <20170921152223.GC92700@icir.org> Message-ID: First of all, thanks to Johanna for getting this discussion going, and thanks to everyone who's weighed in so far. I'm really excited to see this feature in Bro, and I'm also happy to see how much interest this has already garnered. To extend what Seth said about our two user groups -- I think that this feature is where those two groups intersect. While a lot of thought has gone into what this looks like from an end-user perspective, I want to make sure that we also make this easy and elegant from a developer's perspective. Bro scripting already has a high barrier of entry, and I think that we need to be careful not to raise that barrier further. As was discussed during BroCon, I think that the Bro community is increasingly relying on developers outside of the core team to contribute scripts -- and that's a great thing! I think that it's important to have this behavior come with a reasonable default. I think that whatever we choose should just work out of the box. For example, I think the existing construct should continue to work: > const user_name: string &redef At the end of the day, what we're discussing is how a developer should document and expose a feature to an end-user. If, as a developer, I choose a bad variable name, then I'm not providing a good experience for the end-user, but that's my decision. I don't think that forcing developers to essentially add documentation via syntactic sugar is the right approach. If their variable names are confusing, people are less likely to use their script. I think that a lot of what users might want to re-configure is too complex to be explained through a variable name anyway. We already have a system in place to document variables, and I think that we need to rely on that instead of focusing so much on which name is exposed to the user. As we look at moving Bro scripts to packages, I think we need to look at how other package repositories have handled similar configuration options. Puppet Forge, for instance, has a types tab which documents the names of the parameters, and what they do: https://forge.puppet.com/puppetlabs/mysql/types This would be pretty easy to do with the Broxygen documentation, and a UI could also expose this. tl;dr version: I want to find something that makes life easy for both developers and end-users, and I think we already have the documentation mechanism in place to be more expressive about variables. --Vlad On Thu, Sep 21, 2017 at 10:22 AM, Robin Sommer wrote: > > > On Thu, Sep 21, 2017 at 14:51 +0000, you wrote: > > > comments. Like Jan, I had a hard time understanding the benefit having > > two names for the same value: the identifier and config string. > > Yeah, that's been my original concern as well. What if we focused that > new attribute just on displaying something to the user: > > const user_name: string &redef &display_name="User name" > > A UI would show it as "User name", but everything else (incl. > internally the configuration framework) would use > My_Program::user_name. This would even work more generically, anything > could have a &display_name and we'd have Broxygen pick up on it too. > > Robin > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20170921/28d14300/attachment.html From johanna at corelight.com Thu Sep 21 09:50:07 2017 From: johanna at corelight.com (Johanna Amann) Date: Thu, 21 Sep 2017 09:50:07 -0700 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com> <1DCAABCB-8F6E-414B-8C4B-D44AD737E297@illinois.edu> <20170921152223.GC92700@icir.org> Message-ID: <20170921165007.lmemaqqacn7kyka7@Trafalgar.local> > I think that it's important to have this behavior come with a reasonable > default. I think that whatever we choose should just work out of the box. > For example, I think the existing construct should continue to work: > > > const user_name: string &redef I agree; note that what I proposed preserves this compatibility (it does not change anything at all about redefs). The feature that the configuration feature wants to bring is the ability to change options during runtime - which cannot be accomplished with redefs. redef-able consts will still have their place afterwards (for everything that still cannot be changed during runtime). > At the end of the day, what we're discussing is how a developer should > document and expose a feature to an end-user. If, as a developer, I choose > a bad variable name, then I'm not providing a good experience for the > end-user, but that's my decision. I don't think that forcing developers to > essentially add documentation via syntactic sugar is the right approach. If > their variable names are confusing, people are less likely to use their > script. > > I think that a lot of what users might want to re-configure is too complex > to be explained through a variable name anyway. We already have a system in > place to document variables, and I think that we need to rely on that > instead of focusing so much on which name is exposed to the user. I agree with this. > As we look at moving Bro scripts to packages, I think we need to look at > how other package repositories have handled similar configuration options. > Puppet Forge, for instance, has a types tab which documents the names of > the parameters, and what they do: > https://forge.puppet.com/puppetlabs/mysql/types This would be pretty easy > to do with the Broxygen documentation, and a UI could also expose this. Yup, I also agree with this. Johanna From seth at corelight.com Fri Sep 22 07:02:46 2017 From: seth at corelight.com (Seth Hall) Date: Fri, 22 Sep 2017 10:02:46 -0400 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: <20170921165007.lmemaqqacn7kyka7@Trafalgar.local> References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com> <1DCAABCB-8F6E-414B-8C4B-D44AD737E297@illinois.edu> <20170921152223.GC92700@icir.org> <20170921165007.lmemaqqacn7kyka7@Trafalgar.local> Message-ID: I just wanted to note that after keeping up on this thread that I agree with those same points. :) .Seth On 21 Sep 2017, at 12:50, Johanna Amann wrote: >> I think that it's important to have this behavior come with a >> reasonable >> default. I think that whatever we choose should just work out of the >> box. >> For example, I think the existing construct should continue to work: >> >>> const user_name: string &redef > > I agree; note that what I proposed preserves this compatibility (it > does > not change anything at all about redefs). The feature that the > configuration feature wants to bring is the ability to change options > during runtime - which cannot be accomplished with redefs. redef-able > consts will still have their place afterwards (for everything that > still > cannot be changed during runtime). > >> At the end of the day, what we're discussing is how a developer >> should >> document and expose a feature to an end-user. If, as a developer, I >> choose >> a bad variable name, then I'm not providing a good experience for the >> end-user, but that's my decision. I don't think that forcing >> developers to >> essentially add documentation via syntactic sugar is the right >> approach. If >> their variable names are confusing, people are less likely to use >> their >> script. >> >> I think that a lot of what users might want to re-configure is too >> complex >> to be explained through a variable name anyway. We already have a >> system in >> place to document variables, and I think that we need to rely on that >> instead of focusing so much on which name is exposed to the user. > > I agree with this. > >> As we look at moving Bro scripts to packages, I think we need to look >> at >> how other package repositories have handled similar configuration >> options. >> Puppet Forge, for instance, has a types tab which documents the names >> of >> the parameters, and what they do: >> https://forge.puppet.com/puppetlabs/mysql/types This would be pretty >> easy >> to do with the Broxygen documentation, and a UI could also expose >> this. > > Yup, I also agree with this. > > Johanna > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -- Seth Hall * Corelight, Inc * www.corelight.com From jsiwek at illinois.edu Fri Sep 22 08:59:08 2017 From: jsiwek at illinois.edu (Siwek, Jon) Date: Fri, 22 Sep 2017 15:59:08 +0000 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: <20170921165007.lmemaqqacn7kyka7@Trafalgar.local> References: <20170920222444.4ea3o6eugd2k4zcs@Trafalgar.local> <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com> <1DCAABCB-8F6E-414B-8C4B-D44AD737E297@illinois.edu> <20170921152223.GC92700@icir.org> <20170921165007.lmemaqqacn7kyka7@Trafalgar.local> Message-ID: <6F7BC0F1-50BB-430C-A794-2DB876B890F6@illinois.edu> > On Sep 21, 2017, at 11:50 AM, Johanna Amann wrote: > > The feature that the > configuration feature wants to bring is the ability to change options > during runtime - which cannot be accomplished with redefs. redef-able > consts will still have their place afterwards (for everything that still > cannot be changed during runtime). Just had a misc. thought related to the use of ?const?: I remember first being confused/unfamiliar with Bro?s use of ?const? and thought it meant ?never changes? only to learn it?s further qualified as ?never changes at run-time? so that the ?const? + ?&redef? combo ultimately means ?never changes at run-time, but initial value may be changed at parse-time?. Though, technically, ?const? can still be modified at run-time, if you know how? e.g. send_id... And that?s maybe all ok -- it?s easy to explain unfamiliar context as I did above and the means of subverting runtime modification restrictions isn?t actively advertised as such. Though, with a new config framework, there?s opportunities: 1) could remove need for the backdoor method of modifying ?const? values at runtime, (e.g. via send_id) as that?s done through new identifiers explicitly tagged for config purposes 2) using a new ?configopt? access modifier may be warranted over re-using ?const? for configuration values as the semantics are now blatantly different than what ?const? is expected to mean. i.e. config values are meant to change at run-time, but only via a restricted API and ?const? is still intended to never change at run-time - Jon From robin at icir.org Fri Sep 22 09:31:07 2017 From: robin at icir.org (Robin Sommer) Date: Fri, 22 Sep 2017 09:31:07 -0700 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: <6F7BC0F1-50BB-430C-A794-2DB876B890F6@illinois.edu> References: <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com> <1DCAABCB-8F6E-414B-8C4B-D44AD737E297@illinois.edu> <20170921152223.GC92700@icir.org> <20170921165007.lmemaqqacn7kyka7@Trafalgar.local> <6F7BC0F1-50BB-430C-A794-2DB876B890F6@illinois.edu> Message-ID: <20170922163107.GA14842@icir.org> I was thinking the same when discussing an earlier proposal with Johanna. The "configopt" idea came out of that with the observation that "const &redef" isn't quite fitting here (and, as you say, it's already blurred anyways). At that time, however, the thinking was still to have a 2nd namespace, and writing 'configopt X: string &config="a.b.c"' seemed a bit too much. But if we just go with a more generic display name via Broxygen, then I'm back to liking it -- except maybe for the name, how about "option" instead of "configopt"? So we'd arrive at something like this (similar to what has been said already): module Foo; export { ## The username for our new feature. ## ## Display: User Name option user_name: string; } And we could even start deprecating "const ... &redef" if we wanted. Robin On Fri, Sep 22, 2017 at 15:59 +0000, you wrote: > > > On Sep 21, 2017, at 11:50 AM, Johanna Amann wrote: > > > > The feature that the > > configuration feature wants to bring is the ability to change options > > during runtime - which cannot be accomplished with redefs. redef-able > > consts will still have their place afterwards (for everything that still > > cannot be changed during runtime). > > Just had a misc. thought related to the use of ?const?: > > I remember first being confused/unfamiliar with Bro?s use of ?const? and thought it meant ?never changes? only to learn it?s further qualified as ?never changes at run-time? so that the ?const? + ?&redef? combo ultimately means ?never changes at run-time, but initial value may be changed at parse-time?. > > Though, technically, ?const? can still be modified at run-time, if you know how? e.g. send_id... > > And that?s maybe all ok -- it?s easy to explain unfamiliar context as I did above and the means of subverting runtime modification restrictions isn?t actively advertised as such. Though, with a new config framework, there?s opportunities: > > 1) could remove need for the backdoor method of modifying ?const? values at runtime, (e.g. via send_id) as that?s done through new identifiers explicitly tagged for config purposes > > 2) using a new ?configopt? access modifier may be warranted over re-using ?const? for configuration values as the semantics are now blatantly different than what ?const? is expected to mean. i.e. config values are meant to change at run-time, but only via a restricted API and ?const? is still intended to never change at run-time > > - Jon > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From johanna at corelight.com Fri Sep 22 09:48:40 2017 From: johanna at corelight.com (Johanna Amann) Date: Fri, 22 Sep 2017 09:48:40 -0700 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: <20170922163107.GA14842@icir.org> References: <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com> <1DCAABCB-8F6E-414B-8C4B-D44AD737E297@illinois.edu> <20170921152223.GC92700@icir.org> <20170921165007.lmemaqqacn7kyka7@Trafalgar.local> <6F7BC0F1-50BB-430C-A794-2DB876B890F6@illinois.edu> <20170922163107.GA14842@icir.org> Message-ID: <20170922164840.l3krv6v2rs2ygcln@user203.sys.ICSI.Berkeley.EDU> I like that - I honestly did not really like configopt from the beginning, the only reason for choosing it is that "config" conflicts with the input/logging framework. I am not completely sure we should deprecate const &redefs - there might actually be a need for constants that can really only be changed on startup. I like the idea of Jon to also remove the current ways to manipulate &costs at runtime though :) Johanna On Fri, Sep 22, 2017 at 04:31:07PM +0000, Robin Sommer wrote: > I was thinking the same when discussing an earlier proposal with > Johanna. The "configopt" idea came out of that with the observation > that "const &redef" isn't quite fitting here (and, as you say, it's > already blurred anyways). At that time, however, the thinking was > still to have a 2nd namespace, and writing 'configopt X: string > &config="a.b.c"' seemed a bit too much. But if we just go with a more > generic display name via Broxygen, then I'm back to liking it -- > except maybe for the name, how about "option" instead of "configopt"? > So we'd arrive at something like this (similar to what has been said > already): > > module Foo; > > export { > > ## The username for our new feature. > ## > ## Display: User Name > option user_name: string; > > } > > And we could even start deprecating "const ... &redef" if we wanted. > > Robin > > On Fri, Sep 22, 2017 at 15:59 +0000, you wrote: > > > > > > On Sep 21, 2017, at 11:50 AM, Johanna Amann wrote: > > > > > > The feature that the > > > configuration feature wants to bring is the ability to change options > > > during runtime - which cannot be accomplished with redefs. redef-able > > > consts will still have their place afterwards (for everything that still > > > cannot be changed during runtime). > > > > Just had a misc. thought related to the use of ?const?: > > > > I remember first being confused/unfamiliar with Bro?s use of ?const? and thought it meant ?never changes? only to learn it?s further qualified as ?never changes at run-time? so that the ?const? + ?&redef? combo ultimately means ?never changes at run-time, but initial value may be changed at parse-time?. > > > > Though, technically, ?const? can still be modified at run-time, if you know how? e.g. send_id... > > > > And that?s maybe all ok -- it?s easy to explain unfamiliar context as I did above and the means of subverting runtime modification restrictions isn?t actively advertised as such. Though, with a new config framework, there?s opportunities: > > > > 1) could remove need for the backdoor method of modifying ?const? values at runtime, (e.g. via send_id) as that?s done through new identifiers explicitly tagged for config purposes > > > > 2) using a new ?configopt? access modifier may be warranted over re-using ?const? for configuration values as the semantics are now blatantly different than what ?const? is expected to mean. i.e. config values are meant to change at run-time, but only via a restricted API and ?const? is still intended to never change at run-time > > > > - Jon > > > > _______________________________________________ > > bro-dev mailing list > > bro-dev at bro.org > > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > > > > > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jan.grashoefer at gmail.com Fri Sep 22 13:26:54 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Fri, 22 Sep 2017 22:26:54 +0200 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: <20170922163107.GA14842@icir.org> References: <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com> <1DCAABCB-8F6E-414B-8C4B-D44AD737E297@illinois.edu> <20170921152223.GC92700@icir.org> <20170921165007.lmemaqqacn7kyka7@Trafalgar.local> <6F7BC0F1-50BB-430C-A794-2DB876B890F6@illinois.edu> <20170922163107.GA14842@icir.org> Message-ID: <2c4264ec-de4a-1835-7a4c-922d6fcf672f@gmail.com> > module Foo; > > export { > > ## The username for our new feature. > ## > ## Display: User Name > option user_name: string; > > } I really like that syntax! I think Jon is right regarding the semantics that the "const" keyword implies (I missed that in my first mail). Using "option" instead looks like a nice and clean solution. Following the discussion, I think Vlad managed to explain what I had on my mind as well: The language should not be polluted with elements just to force developers to add documentation. Especially because there is already a convenient mechanism for documentation. Adding "Display:" as a broxygen keyword fades in nicely while separate concerns (i.e. code and documentation) keep separate. Jan From seth at corelight.com Mon Sep 25 08:45:08 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 25 Sep 2017 11:45:08 -0400 Subject: [Bro-Dev] Configuration framework syntax proposal In-Reply-To: <2c4264ec-de4a-1835-7a4c-922d6fcf672f@gmail.com> References: <20170921001959.twjk3lqd6ekp76y5@Trafalgar.local> <332CAD59-7004-4321-8385-BB4A6710D60F@corelight.com> <03e3589e-717d-d82b-5246-3b90c93c4a34@gmail.com> <1DCAABCB-8F6E-414B-8C4B-D44AD737E297@illinois.edu> <20170921152223.GC92700@icir.org> <20170921165007.lmemaqqacn7kyka7@Trafalgar.local> <6F7BC0F1-50BB-430C-A794-2DB876B890F6@illinois.edu> <20170922163107.GA14842@icir.org> <2c4264ec-de4a-1835-7a4c-922d6fcf672f@gmail.com> Message-ID: On 22 Sep 2017, at 16:26, Jan Grash?fer wrote: >> module Foo; >> >> export { >> >> ## The username for our new feature. >> ## >> ## Display: User Name >> option user_name: string; >> >> } > > I really like that syntax! I agree. That looks really nice! .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From seth at corelight.com Fri Sep 29 07:41:39 2017 From: seth at corelight.com (Seth Hall) Date: Fri, 29 Sep 2017 10:41:39 -0400 Subject: [Bro-Dev] NETMAP plugin Message-ID: <3277C92A-D518-40CD-9289-4276CC67FF5D@corelight.com> I *finally* pushed out the Bro NETMAP plugin into the Bro package manager. This is the same thing that was in the old plugins repository that has now been deprecated. I would appreciate if people would try it if you use NETMAP. It can be installed like this: bro-pkg refresh bro-pkg install bro/bro-netmap You can read some more directions about how to use it in the repository here: https://github.com/bro/bro-netmap .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From robin at icir.org Fri Sep 29 09:28:44 2017 From: robin at icir.org (Robin Sommer) Date: Fri, 29 Sep 2017 09:28:44 -0700 Subject: [Bro-Dev] New CAF release for new Broker Message-ID: <20170929162844.GI74728@icir.org> For those tracking the new Broker version in the topic/actor-system branch: A new CAF release 0.15.4 is out now that's incorporating everything that code requires, so no need to use the CAF "develop" branch anymore. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From seth at corelight.com Fri Sep 29 13:53:47 2017 From: seth at corelight.com (Seth Hall) Date: Fri, 29 Sep 2017 16:53:47 -0400 Subject: [Bro-Dev] New CAF release for new Broker In-Reply-To: <20170929162844.GI74728@icir.org> References: <20170929162844.GI74728@icir.org> Message-ID: <588E2531-3844-4B39-ADD9-F8DC5EE17B52@corelight.com> On 29 Sep 2017, at 12:28, Robin Sommer wrote: > For those tracking the new Broker version in the topic/actor-system > branch: A new CAF release 0.15.4 is out now that's incorporating > everything that code requires, so no need to use the CAF "develop" > branch anymore. Woohoo! It's getting closer. :) .Seth -- Seth Hall * Corelight, Inc * www.corelight.com