From jan.grashoefer at gmail.com Thu Dec 1 02:31:52 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Thu, 1 Dec 2016 11:31:52 +0100 Subject: [Bro-Dev] Updating NEWS for 2.5 In-Reply-To: References: <20160808190459.GK52034@icir.org> <20160809152120.GQ52034@icir.org> <60728D5A-75E2-43B9-A76B-FAFB49CA7EC2@gmail.com> <70d9858d-4f6e-6a70-6663-732275e145e5@gmail.com> <73947624-41EA-44DF-836F-5ED43CEF7A70@gmail.com> <5fd75102-9d4f-fbb8-e255-d0e520317b37@gmail.com> <48dff4b0-b50f-bfd1-3ee4-c43bcea3adae@gmail.com> Message-ID: > Thanks. Let?s space out our blogs a little bit (We posted one yesterday on bro-pkg) and save this one for after the 2.5 release. As 2.5 has been released and there have already been some questions on the mailing list, it would be good to publish the blog post. Jan From slagell at illinois.edu Thu Dec 1 04:54:42 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Thu, 1 Dec 2016 12:54:42 +0000 Subject: [Bro-Dev] Updating NEWS for 2.5 In-Reply-To: References: <20160808190459.GK52034@icir.org> <20160809152120.GQ52034@icir.org> <60728D5A-75E2-43B9-A76B-FAFB49CA7EC2@gmail.com> <70d9858d-4f6e-6a70-6663-732275e145e5@gmail.com> <73947624-41EA-44DF-836F-5ED43CEF7A70@gmail.com> <5fd75102-9d4f-fbb8-e255-d0e520317b37@gmail.com> <48dff4b0-b50f-bfd1-3ee4-c43bcea3adae@gmail.com> , Message-ID: <3C58A9C4-09A5-4DE1-A065-ADC63C3BA1E5@illinois.edu> On Dec 1, 2016, at 4:32 AM, Jan Grash?fer wrote: >> Thanks. Let?s space out our blogs a little bit (We posted one yesterday on bro-pkg) and save this one for after the 2.5 release. > > As 2.5 has been released and there have already been some questions on > the mailing list, it would be good to publish the blog post. > > Jan Agreed. Please work with Jeannette to post it. Thanks, Adam From asharma at lbl.gov Thu Dec 1 12:49:24 2016 From: asharma at lbl.gov (Aashish Sharma) Date: Thu, 1 Dec 2016 12:49:24 -0800 Subject: [Bro-Dev] missing worker2manager and manager2worker events Message-ID: <20161201204922.GG40549@mac-799.local> I have noticed that sometimes (more often than not), not all workers see a manager2worker event or likewise not all workers report a worker2manager event on manager - missing as high as 10% of the events and as little as 1% of such events are 'missing' ie don't show up. This is puzzling since I am not sure why/where bottleneck is or what might cause the disappreance. Is there a way to assure that these events are movig data as expected ? Thanks, Aashish From robin at icir.org Thu Dec 1 19:39:29 2016 From: robin at icir.org (Robin Sommer) Date: Thu, 1 Dec 2016 19:39:29 -0800 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support Message-ID: <20161202033929.GB36289@icir.org> Bro's current Broker framework has a few pretty inelegant API parts because Bro's scripting language doesn't support some of its operations well currently. I've put some thoughts together on potential language extensions to improve the situation and come to a nicer Broker framework API: https://www.bro.org/development/projects/broker-lang-ext.html Feedback welcome, this is just a first draft. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jazoff at illinois.edu Fri Dec 2 06:19:12 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 2 Dec 2016 14:19:12 +0000 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161202033929.GB36289@icir.org> References: <20161202033929.GB36289@icir.org> Message-ID: > On Dec 1, 2016, at 9:39 PM, Robin Sommer wrote: > > Bro's current Broker framework has a few pretty inelegant API parts > because Bro's scripting language doesn't support some of its > operations well currently. I've put some thoughts together on > potential language extensions to improve the situation and come to a > nicer Broker framework API: > > https://www.bro.org/development/projects/broker-lang-ext.html > > Feedback welcome, this is just a first draft. > > Robin Asynchronous executions without when: yes! Was just talking to Vlad about this yesterday. The examples get even worse as soon as you need to do more than one broker operation in sequence. Something like this with when statements would be unmaintainable: #Check to see if any of these keys exists local v1 = Broker::lookup(h, 41); local v2 = Broker::lookup(h, 42); local v3 = Broker::lookup(h, 43); if (v1 || v2 || v3) { Broker::set(h, ...) } Or I could see a trivial example like this for counting things per day: event connection_established(c: connection) { # not hardcoded ideally.. Broker::inc(h, fmt("connections:2016-12-02:addr:%s, c$id$orig_h), 1); Broker::inc(h, fmt("connections:2016-12-02:addr:%s, c$id$resp_h), 1); Broker::inc(h, fmt("connections:2016-12-02:port:%s, c$id$orig_p), 1); Broker::inc(h, fmt("connections:2016-12-02:port:%s, c$id$resp_p), 1); } Having a way to send a batch of operations would be nice, but that's a separate issue :-) -- - Justin Azoff From robin at icir.org Fri Dec 2 07:52:27 2016 From: robin at icir.org (Robin Sommer) Date: Fri, 2 Dec 2016 07:52:27 -0800 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: References: <20161202033929.GB36289@icir.org> Message-ID: <20161202155227.GC74500@icir.org> On Fri, Dec 02, 2016 at 14:19 +0000, you wrote: > Something like this with when statements would be > unmaintainable: Yeah, exactly. > local v1 = Broker::lookup(h, 41); > local v2 = Broker::lookup(h, 42); > local v3 = Broker::lookup(h, 43); > Having a way to send a batch of operations would be nice, but that's a separate issue :-) Good thought, those three lookups above could all proceed in parallel, and then one would wait for them all to finish before continuing. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at illinois.edu Fri Dec 2 09:00:00 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Fri, 2 Dec 2016 17:00:00 +0000 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161202033929.GB36289@icir.org> References: <20161202033929.GB36289@icir.org> Message-ID: > On Dec 1, 2016, at 9:39 PM, Robin Sommer wrote: > > https://www.bro.org/development/projects/broker-lang-ext.html > > Feedback welcome, this is just a first draft. Looks like a big improvement, some ideas: Not that syntax is super important to nail down right away, but to me, ?v as T? syntax is simpler than "as(v)?. Is there a reason "type(v)? can?t be stored? I?d probably find it more intuitive if it could, else I can see myself forgetting or making mistakes related to that. Alternative to even providing ?type(v)?, you could have a ?v is T? operation and to use an explicit/new ?typeswitch (v)? statement instead of re-using ?switch (type(v))?. Between the two switch syntaxes you gave, maybe provide both and allow mixing of syntax between cases. E.g.: switch ( type(v) ) { case bool -> b: print ?it?s a bool, and I need to inspect the value", b; break; case count: print ?it?s a count, but I don?t care what the value is, I just wanted to know if it?s a count?; break; } "Asynchronous executions without when?: I?d go with an explicit keyword to denote when a handler will yield execution ? it makes it easier to understand the behavior of a function at a glance without requiring a perfect memory for what functions cause execution to yield. Full co-routine support is a nice step to take and this is already similar enough that adding a keyword like ?yield? at this point may make it easier to evolve/plan the language into having generalized co-routine support. It might even be useful to try and spec out the co-routine support now and see how this specific use-case fits in with that. - Jon From vallentin at icir.org Fri Dec 2 09:17:22 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Fri, 2 Dec 2016 09:17:22 -0800 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161202033929.GB36289@icir.org> References: <20161202033929.GB36289@icir.org> Message-ID: <20161202171722.GK19512@shogun.local> > Feedback welcome, this is just a first draft. I like this. Some initial feedback: - In the switch statement, I would require that a user either provide a default case or fully enumerates all cases. Otherwise it's too easy to cause harder-to-debug run-time errors. - For the abbreviated switch statement, what do you think about this variation? switch ( type(v) ) { case b: bool print "bool", b; break; case s: set[int] print "set", s; break; case r: some_record print "record", r; break; Since we don't use the arrow operator anywhere and already declare variable type with colons as in Pascal, this could feel more natural to Bro. - Why do we need a string comparison here? if ( type(v) == string ) Wouldn't it suffice to have if ( type(v) == X ) where X is a type instance (e.g., addr, count, set[int]) or a user-defined type (e.g., connection)? In other words, I don't understand why equality comparison need to be between a type and a string. - Uplevelling, why do we need type() in the first place? Can't we just overload the switch statement to work on types as well? Or do you have other use cases in mind? - Regarding asynchronous execution: local h = async lookup_hostname(?www.icir.org?); I like the explicit keyword here to signal asynchrony. Ignoring the when statement for now, users rely on non-preempted execution both within a function and an event handler, and I would argue that many scripts are built around this assumption. When these semantics change, it could become harder to reason about the execution model. However, if we make timeouts mandatory, I woudn't mind dropping the async keyword. Matthias From jan.grashoefer at gmail.com Fri Dec 2 12:18:49 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Fri, 2 Dec 2016 21:18:49 +0100 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161202033929.GB36289@icir.org> References: <20161202033929.GB36289@icir.org> Message-ID: I really like the suggested improvements and I agree to Jon and Matthias regarding the use of "async" to make corresponding calls explicit. The first thing I thought of was the get_current_packet() or the get_current_packet_header() function. If there is some asynchronous execution in context of a "new" packet, a transparent syntax might obfuscate what is going on there. I also like the switch syntax Matthias suggested. Jan From asharma at lbl.gov Sun Dec 4 11:16:06 2016 From: asharma at lbl.gov (Aashish Sharma) Date: Sun, 4 Dec 2016 11:16:06 -0800 Subject: [Bro-Dev] proxies and tree data structures Message-ID: <20161204191605.GB42943@mac-799.local> I have noticed that at times my proxies are spending way too much CPU (100% for extended duration) in tree operations which include inserts and tree_balance_after_insert. Anyone has any pointers to what might be going on proxies ? Aashish From robin at icir.org Sun Dec 4 12:40:06 2016 From: robin at icir.org (Robin Sommer) Date: Sun, 4 Dec 2016 12:40:06 -0800 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161202171722.GK19512@shogun.local> Message-ID: <20161204204006.GJ58055@icir.org> Thanks for the feedback. I'm replying below to all three of you, as it's all related. I'm hearing strong support for making asynchronous calls explicit, which is fine with me. On Fri, Dec 02, 2016 at 17:00 +0000, Jon wrote: > ?v as T? syntax is simpler than "as(v)?. I see one disadvantage with the "as" syntax: the whitespace will lead to additional parentheses being required in some contexts. For example, with T being a record: "(v as T)$foobar" vs "as(v)$foobar". Not necessarily a showstopper though. The "as" syntax does feel more in line with other Bro syntax. Are there other ideas to express casts? > Is there a reason "type(v)? can?t be stored? I?d probably find it > more intuitive if it could, else I can see myself forgetting or making > mistakes related to that. Mind elaborating how you would expect to use that? My reasoning for foregoing storage was that I couldn't really see much use for having a type stored in variable in the first place because there aren't really any operations that you could then use on that variable (other than generic ones, like printing it). > Alternative to even providing ?type(v)?, you could have a ?v is T? I like that. That avoids the question of storing types altogether. > operation and to use an explicit/new ?typeswitch (v)? statement > instead of re-using ?switch (type(v))?. Hmm ... I'd prefer not introduce a new switch statement. However, seems the existing switch could just infer that it's maching types by looking at case values: if they are types, it'd use "is" for comparision. > case bool -> b: > case count: Good point, I like that. > It might even be useful to try and spec out the co-routine support now > and see how this specific use-case fits in with that. Need to think about that, would clearly be nice if we could pave the way for that already. On Fri, Dec 02, 2016 at 09:17 -0800, Matthias wrote: > - In the switch statement, I would require that a user either provide > a default case or fully enumerates all cases. Otherwise it's too > easy to cause harder-to-debug run-time errors. Full enumeration doesn't seem possible, that would mean all possible types, no? Are we requiring "default" for the standard switch statement? If so, I agree it makes sense to do the same here (otherwise, not so sure, because of consistency). > case b: bool > case s: set[int] > case r: some_record Interesting thought. I had this at first: case b: bool: ... case s: set[int]: ... case r: some_record: ... That mimics the existing syntax more closely, but ends up being ugly with the two colons. I kind of like your idea, even though it makes the syntax a bit non-standard. However, I'm not sure the parser could deal with that easily (because there's no clear separation between the type and the following code). Also, this might not fit so well with Jon's idea of making the identifier optional. > - Why do we need a string comparison here? > > if ( type(v) == string ) I believe you misread this: "string" *is* a type. > - Uplevelling, why do we need type() in the first place? Can't we just > overload the switch statement to work on types as well? Or do you > have other use cases in mind? My main other use case was offering if-style comparision as well for types (see previous point). But if we do "is" for that, we indeed wouldn't need the type() anymore. > However, if we make timeouts mandatory, I woudn't mind dropping the > async keyword. I'm going back and forth on whether to make timeouts mandatory. I think we need to have a default timeout in any case, we cannot have calls linger around forever. But explicitly specifying one each time takes away some of the simplicity. On the other hand, with a default timeout we'd probably need to throw runtime errors instead of returning something, and Bro isn't good with tons of runtime errors (which could happen here). On Fri, Dec 02, 2016 at 21:18 +0100, Jan wrote: > The first thing I thought of was the get_current_packet() or the > get_current_packet_header() function. Can you elaborate? These functions aren't asynchronous currently. Would you change them to being so; and if so, what would that do? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jan.grashoefer at gmail.com Sun Dec 4 15:50:05 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Mon, 5 Dec 2016 00:50:05 +0100 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161204204006.GJ58055@icir.org> References: <20161204204006.GJ58055@icir.org> Message-ID: >> The first thing I thought of was the get_current_packet() or the >> get_current_packet_header() function. > Can you elaborate? These functions aren't asynchronous currently. > Would you change them to being so; and if so, what would that do? That was just another example for a situation in which "hidden" asynchronous execution could easily lead to unintended behavior. Calls of get_current_packet() in context of the same event handler will return different results in case the execution gets interrupted due to an async call. But that's most likely an esoteric edge case. Sorry for the noise :) Jan From robin at icir.org Mon Dec 5 07:51:18 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 5 Dec 2016 07:51:18 -0800 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: References: <20161204204006.GJ58055@icir.org> Message-ID: <20161205155118.GC70389@icir.org> On Mon, Dec 05, 2016 at 00:50 +0100, you wrote: > That was just another example for a situation in which "hidden" > asynchronous execution could easily lead to unintended behavior. Ah, got it, I had misread what you meant. Thanks, Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From robin at icir.org Mon Dec 5 11:46:12 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 5 Dec 2016 11:46:12 -0800 Subject: [Bro-Dev] Marking commits for 2.5.1 Message-ID: <20161205194612.GL70389@icir.org> In BIT-1763, Johanna asks for a way to mark commits for a point release (assuming we do one). Let's do this: set the "fix version" of the ticket to include 2.5.1 and add the commit hash for the fix to the notes. Then we can later search for these and cherry-pick. (We'll still close the ticket once fixed in master to avoid confusion.) Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From vallentin at icir.org Tue Dec 6 03:17:05 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Tue, 6 Dec 2016 03:17:05 -0800 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161204204006.GJ58055@icir.org> References: <20161202171722.GK19512@shogun.local> <20161204204006.GJ58055@icir.org> Message-ID: <20161206111705.GE90069@ninja.local> > > Alternative to even providing ?type(v)?, you could have a ?v is T? > > I like that. That avoids the question of storing types altogether. I think we can cover all type-based dispatching with "is" and "switch." In fact, I see "x is T" as syntactic sugar for: function is(x: any) { switch(x) { default: return false; case T: return true; } } > > It might even be useful to try and spec out the co-routine support now > > and see how this specific use-case fits in with that. > > Need to think about that, would clearly be nice if we could pave the > way for that already. +1 > Full enumeration doesn't seem possible, that would mean all possible > types, no? Yeah, not in a distributed systems, at least. I think I was coming from C++ here where dispatching on type-safe unions requires complete enumeration of all overloads. In Bro, we don't need that because we cannot know the complete list of types a priori. > Are we requiring "default" for the standard switch statement? If so, I > agree it makes sense to do the same here (otherwise, not so sure, > because of consistency). Sounds good to me. > > if ( type(v) == string ) > > I believe you misread this: "string" *is* a type. Ah, now that makes sense! > > - Uplevelling, why do we need type() in the first place? Can't we just > > overload the switch statement to work on types as well? Or do you > > have other use cases in mind? > > My main other use case was offering if-style comparision as well for > types (see previous point). But if we do "is" for that, we indeed > wouldn't need the type() anymore. Good, I like "is" better than a type function because of readability. > But explicitly specifying one each time takes away some of the > simplicity. > > On the other hand, with a default timeout we'd probably need to throw > runtime errors instead of returning something, and Bro isn't good with > tons of runtime errors (which could happen here). How is a timeout different from a failed computation due to a different reason (e.g., connection failure, store backend error)? I think we need to consider errors as possible outcomes of *any* asynchronous operation. More generally, we need well-defined semantics for composing asynchronous computations. For example, what do we do here? # f :: T -> U # g :: T local x = async f(async g())? One solution: if g fails, x should contain g's error---while f will never execute. But in the common case of success, the user just wants to work with x as an instance of T, e.g., when T = count: count y = 42; print x + y; # use x as type count If x holds an error, then Bro would raise a runtime error in the addition operator. Would that make sense? Matthias From robin at icir.org Tue Dec 6 12:24:07 2016 From: robin at icir.org (Robin Sommer) Date: Tue, 6 Dec 2016 12:24:07 -0800 Subject: [Bro-Dev] Package manager meta data In-Reply-To: <8A9CBA4C-4527-49A1-8F38-C57FC79D6F74@illinois.edu> References: <12c859cf-b998-e726-c852-1caa32333e16@gmail.com> <536BD67A-7A29-4B3F-9ABB-38DA079C53F3@illinois.edu> <1760cb97-de75-1cf7-b42e-899e4688418f@gmail.com> <1303E27E-FC8D-4FA5-AD68-DA64FC401F84@illinois.edu> <63694BF3-3E9D-4419-B31D-B96C9E980364@illinois.edu> <8A9CBA4C-4527-49A1-8F38-C57FC79D6F74@illinois.edu> Message-ID: <20161206202407.GA36960@icir.org> On Tue, Nov 29, 2016 at 17:42 +0000, you wrote: > The metadata changes discussed in this thread are now in bro-pkg 0.8 I've tried this out now, it's all working great, thanks! I just noticed one tiny little thing, for which I filed a ticket: https://bro-tracker.atlassian.net/browse/BIT-1766 Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at illinois.edu Tue Dec 6 12:43:06 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Tue, 6 Dec 2016 20:43:06 +0000 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161204204006.GJ58055@icir.org> References: <20161204204006.GJ58055@icir.org> Message-ID: <694E3B3D-5216-4EF9-8B4F-7FD00864FBC8@illinois.edu> > On Dec 4, 2016, at 2:40 PM, Robin Sommer wrote: > > "(v as T)$foobar? vs "as(v)$foobar?. Could just do some time trials to see which one people can type faster. There?s online typing speed tests that let you enter the test?s text. I consistently typed "(v as T)$foobar? faster. >> Is there a reason "type(v)? can?t be stored? > > Mind elaborating how you would expect to use that? Was thinking I?d end up doing something like (more a personal habit/style thing): local t = type(v); if ( t == count ) ? else if ( t == int ) ... But this example came to mind only because the draft didn?t have ?v is T?. Point seems moot now. > seems the existing switch could just infer that it's maching types by > looking at case values: if they are types, it'd use "is" for > comparision. +1 >> - In the switch statement, I would require that a user either provide >> a default case or fully enumerates all cases. Otherwise it's too >> easy to cause harder-to-debug run-time errors. > > Are we requiring "default" for the standard switch statement? If so, I > agree it makes sense to do the same here (otherwise, not so sure, > because of consistency). The ?default" case is optional. If it were required, I wouldn?t feel safer ? maybe I don?t frequently make/see mistakes related to that and/or don?t find them hard to debug. More often I?ve seen forgotten ?break? at the end of a cases causing unintentional fallthroughs, but Bro does require explicit ?break? or ?fallthrough? statement to end cases. - Jon From robin at icir.org Thu Dec 8 08:49:19 2016 From: robin at icir.org (Robin Sommer) Date: Thu, 8 Dec 2016 08:49:19 -0800 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <694E3B3D-5216-4EF9-8B4F-7FD00864FBC8@illinois.edu> <20161206111705.GE90069@ninja.local> Message-ID: <20161208164919.GX60813@icir.org> Sounds like agreement on most suggestions, I'll update the web page shortly with the conclusions. Couple further comments: On Tue, Dec 06, 2016 at 20:43 +0000, Jon wrote: > I consistently typed "(v as T)$foobar? faster. Personally, I see this more as a question of readability (as opposed to typeability :). But it's a matter of taste, and I'd be fine with using "as" instead of "cast<>". On Tue, Dec 06, 2016 at 03:17 -0800, Matthias wrote: > How is a timeout different from a failed computation due to a different > reason (e.g., connection failure, store backend error)? I'm thinking such errors need to be addressed by the function itself, using whatever mechanism it deems appropriate (e.g., signaling trouble through its return value). The problem is that we don't have any further error handling mechanisms (e.g., exceptions) in the language right now (that's a project for some other time). I see timeouts as different as that's something the function itself may have a hard time catching internally, if at all; we couldn't rely on that. It's kind of last line of defense against trouble: if for whatever reason the function ends up never returning, we'll catch it and clean up at least. > asynchronous operation. More generally, we need well-defined > semantics for composing asynchronous computations. I think this would make sense only if we already had some model for propagating errors as part of the language. In the absence of that, I don't really see much to do here. For the normal (non-error) case, return values are just passed along as expected. And in the abnormal error case, there's not much else to do than abort the whole event handler. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From francois.pennaneach at free.fr Fri Dec 9 05:40:55 2016 From: francois.pennaneach at free.fr (francois.pennaneach at free.fr) Date: Fri, 9 Dec 2016 14:40:55 +0100 (CET) Subject: [Bro-Dev] discussion about "violation-confirmation" In-Reply-To: <535100323.102093469.1481290545269.JavaMail.root@zimbra8-e1.priv.proxad.net> Message-ID: <2086465964.102110368.1481290855191.JavaMail.root@zimbra8-e1.priv.proxad.net> Hi, In BIT-313 "DPD: ProtocolConfirmation, ProtocolViolation revamp", there is a link to a discussion about "violation-confirmation". Unfortunately, I could not access this paper (broken link ?) As the ticket also says "ask bro-dev at ... if you want access to the above website", I'm asking :-) Could I get a link to or a copy of the "violation-confirmation" discussion ? Thank you. Fran?ois From jsiwek at illinois.edu Fri Dec 9 09:51:09 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Fri, 9 Dec 2016 17:51:09 +0000 Subject: [Bro-Dev] bro-pkg upgrade and over-writing of files In-Reply-To: <20161129195853.GB3305@mac-799.local> References: <20161129170609.GO67232@mac-799.local> <5C96343F-76A6-4A1F-B365-AE494F6C8E82@illinois.edu> <20161129195853.GB3305@mac-799.local> Message-ID: <2CEF803B-0CD5-4CEA-8920-5265DC96DE8E@illinois.edu> In bro-pkg 0.9, there?s now a config_files field [1] you can put in bro-pkg.meta to prevent bro-pkg operations from clobbering local user modifications to certain files. - Jon [1] http://bro-package-manager.readthedocs.io/en/stable/package.html#config-files From vallentin at icir.org Sun Dec 11 03:49:06 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Sun, 11 Dec 2016 12:49:06 +0100 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161208164919.GX60813@icir.org> References: <694E3B3D-5216-4EF9-8B4F-7FD00864FBC8@illinois.edu> <20161206111705.GE90069@ninja.local> <20161208164919.GX60813@icir.org> Message-ID: <20161211114906.GN26945@ninja.local> > Personally, I see this more as a question of readability (as opposed > to typeability :). But it's a matter of taste, and I'd be fine with > using "as" instead of "cast<>". Probably aligned with that thought is consistency and intuition: we don't have C++-style angle brackets in Bro, so "as" feels more in line with, e.g., type declarations of the form table[T] of U. Angle brackets would feel like a one-off, whereas space-separated short keywords fit in more naturally. > I'm thinking such errors need to be addressed by the function itself, > using whatever mechanism it deems appropriate (e.g., signaling trouble > through its return value). The problem is that we don't have any > further error handling mechanisms (e.g., exceptions) in the language > right now (that's a project for some other time). I see timeouts as > different as that's something the function itself may have a hard time > catching internally, if at all; we couldn't rely on that. It's kind of > last line of defense against trouble: if for whatever reason the > function ends up never returning, we'll catch it and clean up at > least. Indeed, timeouts are a last line of defense and generated by the runtime, as opposed to the function itself. You point out that these are two separate kinds of errors. However, from a user perspective, I think errors should be treated uniformly. For example, when performing a Broker store lookup of a key, but the returned value has a type different than expected, script execution (in particular: a chain of asynchronous functions) should not continue. The same thing happens for a timeout, even though the runtime---as opposed to the actual function---generated the error. The alternative strikes me as complicated, because it is application dependent. If a response to an asynchronous request doesn't arrive within a time bound, *the effect* might be equivalent to an error within the function itself. An invalid cast might generated a runtime error to the console (as Bro does now for use of uninitialized variables) and abort, whereas a timeout would simply abort. In Bro, one way to represent represent errors would be as a record with an optional error field. If the field is set, an error took place and one can examine it. Either the function returns one of these explicitly, or the runtime (i.e., the interpreter) populates one. Here's a strawman of what I mean: type result = record { error: failure &optional; value: any &optional; } function f(x: T) : result { if (store lookup with bad cast) return [$error=..]; if (nothing to do) return []; if (computed something) return [$data=42]; } local r = f(x); if (r$?error) { // handle error, check if timeout or some other form of error // occurred by inspecting the failure instance. } Then, the runtime would allow for automatic chaining of asynchronous functions returning instances of type result: function f(x: T) : result; function g(y: U) : result; // Calls g iff f did not fail and return value unpacking succeeded. local r = g(f(x)); Here, the runtime could do the unpacking of f's result, checking for errors, and feeding the result data into g---assuming types work out. In essence, this is a monad for a fixed set of types. The runtime performs the unboxing of return values automatically. Matthias From jsiwek at illinois.edu Sun Dec 11 06:15:38 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Sun, 11 Dec 2016 14:15:38 +0000 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161211114906.GN26945@ninja.local> References: <694E3B3D-5216-4EF9-8B4F-7FD00864FBC8@illinois.edu> <20161206111705.GE90069@ninja.local> <20161208164919.GX60813@icir.org> <20161211114906.GN26945@ninja.local> Message-ID: <70454DF2-AF93-434D-B78D-0E7080B4B40D@illinois.edu> > On Dec 11, 2016, at 5:49 AM, Matthias Vallentin wrote: > > In Bro, one way to represent represent errors would be as a record with > an optional error field. If the field is set, an error took place and > one can examine it. Either the function returns one of these explicitly, > or the runtime (i.e., the interpreter) populates one. > > Here's a strawman of what I mean: > > type result = record { > error: failure &optional; > value: any &optional; > } That type of structure for error handling/propagation seems fine to me. (Don?t see the advantage of using exceptions instead). > Then, the runtime would allow for automatic chaining of asynchronous > functions returning instances of type result: > > function f(x: T) : result; > function g(y: U) : result; > > // Calls g iff f did not fail and return value unpacking succeeded. > local r = g(f(x)); I might prefer just doing the unpacking myself. Having separate/explicit checks for each individual return value would naturally occur over time anyway (e.g. when debugging, adding extra logging, improving error handling). How common do you expect async function chains to occur? Any specific examples in mind where auto-chaining is very helpful? Maybe it?s more of a nice-to-have feature than a requirement? - Jon From robin at icir.org Mon Dec 12 08:03:53 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 12 Dec 2016 08:03:53 -0800 Subject: [Bro-Dev] discussion about "violation-confirmation" In-Reply-To: <2086465964.102110368.1481290855191.JavaMail.root@zimbra8-e1.priv.proxad.net> References: <535100323.102093469.1481290545269.JavaMail.root@zimbra8-e1.priv.proxad.net> <2086465964.102110368.1481290855191.JavaMail.root@zimbra8-e1.priv.proxad.net> Message-ID: <20161212160353.GD44060@icir.org> On Fri, Dec 09, 2016 at 14:40 +0100, you wrote: > In BIT-313 "DPD: ProtocolConfirmation, ProtocolViolation revamp", there is a link to a discussion about "violation-confirmation". > Unfortunately, I could not access this paper (broken link ?) I just updated the link in the ticket, it's now at https://www.bro.org/development/projects/violation-confirmation.html Given the long time that has passed, I'm sure this would need another round of discussion to see if it still sounds like the right thing to do. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From vallentin at icir.org Tue Dec 13 04:02:23 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Tue, 13 Dec 2016 13:02:23 +0100 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <70454DF2-AF93-434D-B78D-0E7080B4B40D@illinois.edu> References: <694E3B3D-5216-4EF9-8B4F-7FD00864FBC8@illinois.edu> <20161206111705.GE90069@ninja.local> <20161208164919.GX60813@icir.org> <20161211114906.GN26945@ninja.local> <70454DF2-AF93-434D-B78D-0E7080B4B40D@illinois.edu> Message-ID: <20161213120223.GZ26945@ninja.local> > I might prefer just doing the unpacking myself. Having > separate/explicit checks for each individual return value would > naturally occur over time anyway (e.g. when debugging, adding extra > logging, improving error handling). > How common do you expect async function chains to occur? Any specific > examples in mind where auto-chaining is very helpful? Maybe it?s more > of a nice-to-have feature than a requirement? This is a bit of a chicken-and-egg problem: we don't have much use of when at the moment, because it's difficult to nest. If we provide a convenient means for composition from the get-go, I'd imagine we see a quicker adoption of the new features. The standard workflow I anticipate is a lookup-compute-update cycle. Here's a half-baked example: function test(attempts: count): result { if (attempts > threshold) NOTICE(...); return [$value = attempts + 1]; // no error } function check(..) : result { local key = fmt("/outbound/%s", host); local r = async lookup(store, key); if (r?$error) return local?$error; return async put(store, key, failed_attempts + 1); } With the proposed monad, you could factor the implementation of the check function check into a single line: local r = put(store, key, test(lookup(store, key))); Conceptually, this is equivalent to "binding" your function calls: local r = lookup(store, key)) >>= test >>= function(x: count) { return put(store, key, x); }; // curry ...where >>= would be the bind operator performing error/value unpacking. Less functional clutter with the "do" notation: do x1 <- lookup store key x2 <- test x1 x3 <- put store key x2 The last two snippets derail the conversation a bit---just to show where these ideas come from. Note that the function "test" is synchronous in my example and that this doesn't matter. The proposed error handling makes it straight-forward to mix the two invocation styles. Matthias From jsiwek at illinois.edu Tue Dec 13 08:53:41 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Tue, 13 Dec 2016 16:53:41 +0000 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161213120223.GZ26945@ninja.local> References: <694E3B3D-5216-4EF9-8B4F-7FD00864FBC8@illinois.edu> <20161206111705.GE90069@ninja.local> <20161208164919.GX60813@icir.org> <20161211114906.GN26945@ninja.local> <70454DF2-AF93-434D-B78D-0E7080B4B40D@illinois.edu> <20161213120223.GZ26945@ninja.local> Message-ID: <4C73FBE9-1143-42D6-A17B-9FC483248A83@illinois.edu> > On Dec 13, 2016, at 6:02 AM, Matthias Vallentin wrote: > > local r = put(store, key, test(lookup(store, key))); For prototyping purposes, I see the convenience in that, but wonder if the runtime will do something that?s useful and widely applicable enough for that to translate directly into production code. What exactly does the runtime do if ?lookup? fails here besides turn the outer function calls into no-ops? Just guessing, but in many cases one would additionally want to log a warning and others where they even want to schedule the operation to retry at a later time. i.e. the treatment of the failure varies with the context. Is this type of composition flexible enough for that? - Jon From vallentin at icir.org Tue Dec 13 09:42:10 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Tue, 13 Dec 2016 18:42:10 +0100 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <4C73FBE9-1143-42D6-A17B-9FC483248A83@illinois.edu> References: <694E3B3D-5216-4EF9-8B4F-7FD00864FBC8@illinois.edu> <20161206111705.GE90069@ninja.local> <20161208164919.GX60813@icir.org> <20161211114906.GN26945@ninja.local> <70454DF2-AF93-434D-B78D-0E7080B4B40D@illinois.edu> <20161213120223.GZ26945@ninja.local> <4C73FBE9-1143-42D6-A17B-9FC483248A83@illinois.edu> Message-ID: <20161213174210.GD26945@ninja.local> > > local r = put(store, key, test(lookup(store, key))); > > For prototyping purposes, I see the convenience in that, but wonder if > the runtime will do something that?s useful and widely applicable > enough for that to translate directly into production code. What > exactly does the runtime do if ?lookup? fails here besides turn the > outer function calls into no-ops? The runtime doesn't do anything but error propagation. That's a plus in my opinion, because it's simple to understand and efficient to implement. > Just guessing, but in many cases one would additionally want to log a > warning and others where they even want to schedule the operation to > retry at a later time. i.e. the treatment of the failure varies with > the context. Is this type of composition flexible enough for that? It's up to the user to check the result variable (here: r) and decide what to do: abort, retry, continue, or report an error. Based on what constitutes a self-contained unit in an algorithm, there are natural points where one would try again. In the above example, perhaps at the granularity of that put-test-lookup chain. To try again after a timeout, I would imagine it as follows: local algorithm = function() : result { return put(store, key, test(lookup(store, key))) &timeout = 3 sec; }; local r = result(); do { r = algorithm(); } while (r?$error && r$error == timeout); Based on how flexible we design the type in r$error, it can represent all sorts of errors. The runtime could abort for critical errors, but give control back to the user when it's a recoverable one. Matthias From robin at icir.org Tue Dec 13 10:52:55 2016 From: robin at icir.org (Robin Sommer) Date: Tue, 13 Dec 2016 10:52:55 -0800 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161213174210.GD26945@ninja.local> <20161213120223.GZ26945@ninja.local> <20161213102811.GY26945@ninja.local> <70454DF2-AF93-434D-B78D-0E7080B4B40D@illinois.edu> Message-ID: <20161213185255.GJ940@icir.org> > > type result = record { > > error: failure &optional; > > value: any &optional; > > } I don't really like using a record like that, as that would associate specific semantics with what's really a user-definable type. I.e., we'd hardcode into the language that the record used here needs to have exactly these fields. On top of that, it also feels rather clumsy to me as well. Indeed it's one of the things I don't like about the current Broker framework API (which currently has to do it that way just because there's no better mechanism). My draft proposal extends opaque types to support conversion to boolean to allow for more elegant error checks. We could generalize that to support other data types as well, although I don't really see a need for going beyond opaque right now (assuming we also add the cast operator, per the proposal, so that we aren't passing anys around). > With the proposed monad, you could factor the implementation of the > check function check into a single line: > > local r = put(store, key, test(lookup(store, key))); Honestly, I think this operates on a level that's quite beyond any other concepts that the language currently offers, and therefore doesn't really fit in. I'm pretty certain this wouldn't really enter the "active working set" of pretty much any Bro user out there. For the time being at least, explicit error checking seems good/convinient enough to me. > r$error == timeout > Based on how flexible we design the type in r$error, it can represent > all sorts of errors. That's a piece that I like: Being able to flag different error types. Maybe the conversion to bool that I proposed originally should really be a conversion to a dedicated error type, so that one can differentiate what happened. I need to think about it, but that could eliminate the need for simply aborting the event handler on timeout (which in turn would help with the problem that Bro isn't good at aborting code) Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From vallentin at icir.org Tue Dec 13 11:51:39 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Tue, 13 Dec 2016 20:51:39 +0100 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161213185255.GJ940@icir.org> References: <20161213174210.GD26945@ninja.local> <20161213120223.GZ26945@ninja.local> <20161213102811.GY26945@ninja.local> <70454DF2-AF93-434D-B78D-0E7080B4B40D@illinois.edu> <20161213185255.GJ940@icir.org> Message-ID: <20161213195139.GG26945@ninja.local> > I don't really like using a record like that, as that would associate > specific semantics with what's really a user-definable type. It was only meant to illustrate the idea of error handling and function composition. These ideas still hold up when substituting the user-defined result type with the proper language-level construct. > We could generalize that to support other data types as well, although > I don't really see a need for going beyond opaque right now Going beyond opaque would have the advantage of applying a uniform error handling strategy to both synchronous and asynchronous code. That's not needed right now, because "when" doesn't make it easy to handle errors, but it could be an opportunity to provide a unified mechanism for a currently neglected aspect of the Bro language. > Maybe the conversion to bool that I proposed originally should really > be a conversion to a dedicated error type, so that one can > differentiate what happened. I like that. Matthias From asharma at lbl.gov Tue Dec 13 12:12:25 2016 From: asharma at lbl.gov (Aashish Sharma) Date: Tue, 13 Dec 2016 12:12:25 -0800 Subject: [Bro-Dev] broctl archive copy vs move Message-ID: <20161213201225.GH18801@mac-822.local> So if we have compresscmd unset then archive-log script does a copy: archive-log: nice cp $file_name "$dest" Any reason why it doesn't do move instead ? I propose changing cp to mv Aashish From jazoff at illinois.edu Tue Dec 13 12:22:58 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 13 Dec 2016 20:22:58 +0000 Subject: [Bro-Dev] broctl archive copy vs move In-Reply-To: <20161213201225.GH18801@mac-822.local> References: <20161213201225.GH18801@mac-822.local> Message-ID: > On Dec 13, 2016, at 3:12 PM, Aashish Sharma wrote: > > So if we have compresscmd unset then archive-log script does a copy: > > archive-log: nice cp $file_name "$dest" > > Any reason why it doesn't do move instead ? > > I propose changing cp to mv No.. I don't think there's any good reason. The way it was originally written must have assumed the logs were going to be archived to a different filesystem. -- - Justin Azoff From seth at icir.org Wed Dec 14 06:39:16 2016 From: seth at icir.org (Seth Hall) Date: Wed, 14 Dec 2016 09:39:16 -0500 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161213195139.GG26945@ninja.local> References: <20161213174210.GD26945@ninja.local> <20161213120223.GZ26945@ninja.local> <20161213102811.GY26945@ninja.local> <70454DF2-AF93-434D-B78D-0E7080B4B40D@illinois.edu> <20161213185255.GJ940@icir.org> <20161213195139.GG26945@ninja.local> Message-ID: > On Dec 13, 2016, at 2:51 PM, Matthias Vallentin wrote: > >> Maybe the conversion to bool that I proposed originally should really >> be a conversion to a dedicated error type, so that one can >> differentiate what happened. > > I like that. I like that too. Having nicely generalized error handling in Bro would be such a huge benefit for script authors. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jazoff at illinois.edu Wed Dec 14 06:44:24 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 14 Dec 2016 14:44:24 +0000 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161213185255.GJ940@icir.org> References: <20161213185255.GJ940@icir.org> Message-ID: <9F5E676F-5737-45F7-BDCF-535B8E12A242@illinois.edu> > On Dec 13, 2016, at 1:52 PM, Robin Sommer wrote: > > My draft proposal extends > opaque types to support conversion to boolean to allow for more > elegant error checks. Would that mean you wouldn't be able to return a bool and check for errors at the same time? -- - Justin Azoff From robin at icir.org Wed Dec 14 07:11:07 2016 From: robin at icir.org (Robin Sommer) Date: Wed, 14 Dec 2016 07:11:07 -0800 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <9F5E676F-5737-45F7-BDCF-535B8E12A242@illinois.edu> References: <20161213185255.GJ940@icir.org> <9F5E676F-5737-45F7-BDCF-535B8E12A242@illinois.edu> Message-ID: <20161214151107.GA44048@icir.org> On Wed, Dec 14, 2016 at 14:44 +0000, you wrote: > Would that mean you wouldn't be able to return a bool and check for > errors at the same time? No, it would always return an opaque ("opaque of Broker::Data" in this case). If you used that in a boolean context, it would evaluate to the error flag. If you wanted to get the actual value out of it, you'd cast it accordingly with the new cast operator; and that includes casting to "bool". Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From robin at icir.org Wed Dec 14 07:27:28 2016 From: robin at icir.org (Robin Sommer) Date: Wed, 14 Dec 2016 07:27:28 -0800 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: References: <20161213174210.GD26945@ninja.local> <20161213120223.GZ26945@ninja.local> <20161213102811.GY26945@ninja.local> <70454DF2-AF93-434D-B78D-0E7080B4B40D@illinois.edu> <20161213185255.GJ940@icir.org> <20161213195139.GG26945@ninja.local> Message-ID: <20161214152728.GB44048@icir.org> On Wed, Dec 14, 2016 at 09:39 -0500, you wrote: > I like that too. Having nicely generalized error handling in Bro > would be such a huge benefit for script authors. So let's think that through. Let's say we did an Error enum that can reflect various error conditions, something like this: local v = Broker::lookup(h, 42) # Continues to return 'opaque of Broker::Data' if ( v == Error::Success ) print (v as string); else if ( v == Error::NotFound ) print("not found"); else if ( v == Error::Timeout ) print("timeout"); else print("unknown error"); } (That could also be a switch statement, naturally). I think that looks pretty good. But where's that Error type defined? Is it a new global type that Bro predefines for everybody to use? Then we'd either need to limit the type of errors to a small predefined set, or allow scripts to redef the enum and add their own types; but the latter risks naming clashes. Alternatively, we could leave it to frameworks to define their own error types. So for Broker, we'd have Broker::NotFound, Broker::Timeout, etc. And the opaque types would define internally what they convert to, and how. Seth, a question: do you have more in mind by "nicely generalized error handling" than this? Do you see a way to generalize this to purely script-level logic (i.e., no opaques, just normal Bro types being passed around)? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From robin at icir.org Wed Dec 14 07:31:38 2016 From: robin at icir.org (Robin Sommer) Date: Wed, 14 Dec 2016 07:31:38 -0800 Subject: [Bro-Dev] broctl archive copy vs move In-Reply-To: References: <20161213201225.GH18801@mac-822.local> Message-ID: <20161214153138.GC44048@icir.org> On Tue, Dec 13, 2016 at 20:22 +0000, you wrote: > The way it was originally written must have assumed the logs were > going to be archived to a different filesystem. Yes, indeed, pretty sure that was the reason. If we just replaced "cp" with "mv" that would still be an issue, right? We'd need something a bit more smart that does the right thing in either case, like Python's shutil.move does, https://docs.python.org/2/library/shutil.html#shutil.move Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at illinois.edu Wed Dec 14 08:17:26 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Wed, 14 Dec 2016 16:17:26 +0000 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161213174210.GD26945@ninja.local> References: <694E3B3D-5216-4EF9-8B4F-7FD00864FBC8@illinois.edu> <20161206111705.GE90069@ninja.local> <20161208164919.GX60813@icir.org> <20161211114906.GN26945@ninja.local> <70454DF2-AF93-434D-B78D-0E7080B4B40D@illinois.edu> <20161213120223.GZ26945@ninja.local> <4C73FBE9-1143-42D6-A17B-9FC483248A83@illinois.edu> <20161213174210.GD26945@ninja.local> Message-ID: > On Dec 13, 2016, at 11:42 AM, Matthias Vallentin wrote: > >>> local r = put(store, key, test(lookup(store, key))); > > It's up to the user to check the result variable (here: r) and decide > what to do: abort, retry, continue, or report an error. The thing that got me about that for this particular example was that I can?t distinguish whether the "lookup" or the ?put? failed, which might be important since the ?test? operation is between them and I may or may not want ?test" to happen in the retry attempt depending on what exactly failed. Not really making an argument/vote against this feature, just thinking out loud whether it would be something I?d personally use on the chance it helps you better plan/design. Still think I?d instead always manually unpack/test individual async return values. - Jon From jsiwek at illinois.edu Wed Dec 14 09:56:10 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Wed, 14 Dec 2016 17:56:10 +0000 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161214152728.GB44048@icir.org> References: <20161213174210.GD26945@ninja.local> <20161213120223.GZ26945@ninja.local> <20161213102811.GY26945@ninja.local> <70454DF2-AF93-434D-B78D-0E7080B4B40D@illinois.edu> <20161213185255.GJ940@icir.org> <20161213195139.GG26945@ninja.local> <20161214152728.GB44048@icir.org> Message-ID: > On Dec 14, 2016, at 9:27 AM, Robin Sommer wrote: > > I think that looks pretty good. But where's that Error type defined? > Is it a new global type that Bro predefines for everybody to use? Then > we'd either need to limit the type of errors to a small predefined > set, or allow scripts to redef the enum and add their own types; but > the latter risks naming clashes. Is there risk of clashes here? Don?t recall how intentional it is, but don?t enum redefinitions export the identifiers into the current module/namespace? Example: module GLOBAL; type Error: enum { Success, InternalTimeout, }; module Broker; export { redef enum Error += { NotFound, }; } module SomeOtherFramework; export { redef enum Error += { WeirdError, }; } module GLOBAL; print Success; print Broker::NotFound; print SomeOtherFramework::WeirdError; > Alternatively, we could leave it to frameworks to define their own > error types. So for Broker, we'd have Broker::NotFound, > Broker::Timeout, etc. And the opaque types would define internally > what they convert to, and how. So far, there's 3 pieces of information that can come out of an async function: 1) An error code that is specific to that function (e.g. Error::NotFound) 2) An error code that is not specific to that function, but comes predefined in Bro internals (e.g. Error::InternalTimeout) 3) The actual data you?d want to use in the case the function succeeds Seems appropriate to let each framework define (1) and (3), but somehow have (2) internally predefined. I guess the above example allows that, depending on how much you want to rely on the ?redef enum? namespacing behavior? - Jon From robin at icir.org Thu Dec 15 08:36:44 2016 From: robin at icir.org (Robin Sommer) Date: Thu, 15 Dec 2016 08:36:44 -0800 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: References: <20161213174210.GD26945@ninja.local> <20161213120223.GZ26945@ninja.local> <20161213102811.GY26945@ninja.local> <70454DF2-AF93-434D-B78D-0E7080B4B40D@illinois.edu> <20161213185255.GJ940@icir.org> <20161213195139.GG26945@ninja.local> <20161214152728.GB44048@icir.org> Message-ID: <20161215163644.GN74331@icir.org> On Wed, Dec 14, 2016 at 17:56 +0000, you wrote: > Is there risk of clashes here? Don?t recall how intentional it is, > but don?t enum redefinitions export the identifiers into the current > module/namespace? Good point! I've always found this a bit confusing, though. In fact I can never even remember how exactly it works. So not sure if that's a good thing to rely on here? On the other hand, given that it's there and can solve the problem, we might as well use it ... I'm torn. :-) Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From seth at icir.org Thu Dec 15 09:04:40 2016 From: seth at icir.org (Seth Hall) Date: Thu, 15 Dec 2016 12:04:40 -0500 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161214152728.GB44048@icir.org> References: <20161213174210.GD26945@ninja.local> <20161213120223.GZ26945@ninja.local> <20161213102811.GY26945@ninja.local> <70454DF2-AF93-434D-B78D-0E7080B4B40D@illinois.edu> <20161213185255.GJ940@icir.org> <20161213195139.GG26945@ninja.local> <20161214152728.GB44048@icir.org> Message-ID: > On Dec 14, 2016, at 10:27 AM, Robin Sommer wrote: > > local v = Broker::lookup(h, 42) # Continues to return 'opaque of Broker::Data' > > if ( v == Error::Success ) > print (v as string); This is a neat trick and would be nice if all Bro types could be compared against error enums. > Seth, a question: do you have more in mind by "nicely generalized > error handling" than this? Do you see a way to generalize this to > purely script-level logic (i.e., no opaques, just normal Bro types > being passed around)? I don't have any concrete suggestions beyond or alternate to the one you suggested above. I actually really like the look but I think I need to try writing some code like that to see if I find issues with it. One thing I see immediately is that it seems like we're want to force people to cast the value to an error type, maybe like this... if ( v as error == Error::Success ) I find "Error::Success" really unintuitive and kind of funny too. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From dnthayer at illinois.edu Thu Dec 15 10:44:30 2016 From: dnthayer at illinois.edu (Daniel Thayer) Date: Thu, 15 Dec 2016 12:44:30 -0600 Subject: [Bro-Dev] broctl archive copy vs move In-Reply-To: <20161214153138.GC44048@icir.org> References: <20161213201225.GH18801@mac-822.local> <20161214153138.GC44048@icir.org> Message-ID: <0a488b2b-b49f-981d-bdfa-bc28d39d12fa@illinois.edu> On 12/14/16 9:31 AM, Robin Sommer wrote: > > > > On Tue, Dec 13, 2016 at 20:22 +0000, you wrote: > >> The way it was originally written must have assumed the logs were >> going to be archived to a different filesystem. > > Yes, indeed, pretty sure that was the reason. If we just replaced "cp" > with "mv" that would still be an issue, right? We'd need something a > bit more smart that does the right thing in either case, like Python's > shutil.move does, > https://docs.python.org/2/library/shutil.html#shutil.move > > Robin > I'm not aware of any variety of Linux or BSD where the "mv" command can't handle moving a file to a different filesystem. I think the real issue is that with "mv", a copy of the log would no longer be in the working directory, which is only an issue if Bro had crashed (that's when post-terminate runs archive-log with the "-c" option, which indicates that we shouldn't remove the log after archiving it). -Daniel From robin at icir.org Thu Dec 15 17:01:37 2016 From: robin at icir.org (Robin Sommer) Date: Thu, 15 Dec 2016 17:01:37 -0800 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: References: <20161213174210.GD26945@ninja.local> <20161213120223.GZ26945@ninja.local> <20161213102811.GY26945@ninja.local> <70454DF2-AF93-434D-B78D-0E7080B4B40D@illinois.edu> <20161213185255.GJ940@icir.org> <20161213195139.GG26945@ninja.local> <20161214152728.GB44048@icir.org> Message-ID: <20161216010137.GE14434@icir.org> On Thu, Dec 15, 2016 at 12:04 -0500, you wrote: > force people to cast the value to an error type, maybe like this... > > if ( v as error == Error::Success ) Mind elaboratong why you think we need to force the cast? That seems to take away some of the appeal of the solution. Btw, the above is an example of why I'm still a bit reluctant on the "as" syntax. The line above would really benefit from parentheses: if ( (v as error) == Error::Success ) > I find "Error::Success" really unintuitive and kind of funny too. :) Yeah, agree, that's part of the question what namespace to use. "Broker::Success" would certainly be nicer. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From seth at icir.org Thu Dec 15 22:36:35 2016 From: seth at icir.org (Seth Hall) Date: Fri, 16 Dec 2016 01:36:35 -0500 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161216010137.GE14434@icir.org> References: <20161213174210.GD26945@ninja.local> <20161213120223.GZ26945@ninja.local> <20161213102811.GY26945@ninja.local> <70454DF2-AF93-434D-B78D-0E7080B4B40D@illinois.edu> <20161213185255.GJ940@icir.org> <20161213195139.GG26945@ninja.local> <20161214152728.GB44048@icir.org> <20161216010137.GE14434@icir.org> Message-ID: > On Dec 15, 2016, at 8:01 PM, Robin Sommer wrote: > > On Thu, Dec 15, 2016 at 12:04 -0500, you wrote: > >> force people to cast the value to an error type, maybe like this... >> >> if ( v as error == Error::Success ) > > Mind elaboratong why you think we need to force the cast? That seems > to take away some of the appeal of the solution. I don't know. It just feels too magic otherwise, but I guess it's no different than what the system is already doing in it's normal automatic type casting situations so I don't think I can really complain here. >> I find "Error::Success" really unintuitive and kind of funny too. :) > > Yeah, agree, that's part of the question what namespace to use. > "Broker::Success" would certainly be nicer. Yep. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From robin at icir.org Fri Dec 16 07:46:54 2016 From: robin at icir.org (Robin Sommer) Date: Fri, 16 Dec 2016 07:46:54 -0800 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: References: <20161213120223.GZ26945@ninja.local> <20161213102811.GY26945@ninja.local> <70454DF2-AF93-434D-B78D-0E7080B4B40D@illinois.edu> <20161213185255.GJ940@icir.org> <20161213195139.GG26945@ninja.local> <20161214152728.GB44048@icir.org> <20161216010137.GE14434@icir.org> Message-ID: <20161216154654.GE31505@icir.org> On Fri, Dec 16, 2016 at 01:36 -0500, you wrote: > I don't know. It just feels too magic otherwise, To me, "v == Error::XYZ" would feel ok still, yet "v == Broker::XYZ" is getting borderline (because the latter means there's no clear single concept here (error checking), it could mean whatever the value type wants). Maybe we need something else altogether. We could add a BIF that takes the role of the cast, but gives more context what happens: local v = Broker::lookup(h, 42) if ( status(v) == Broker::SUCCESS ) print (v as string); That could be a Broker-specific BIF (then probably Broker::status()) or a generic oben working with opaques in general. We could also make the two different return values explicit: [result, value] = Broker::lookup(h, 42) # Returns [Broker::Result, opaque of Broker::Data] if ( result == Broker::SUCCESS ) ... This would be new syntax, as we don't have tuples yet to implement this directly. But for-loops already support something not too different for iterating over multi-value indices. We could also bite the bullet and add full tuples to the language (but that's quite a bit of work). Another downside is that "result", and "value" would need to be declared first, making it less concise. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From robin at icir.org Fri Dec 16 09:53:05 2016 From: robin at icir.org (Robin Sommer) Date: Fri, 16 Dec 2016 09:53:05 -0800 Subject: [Bro-Dev] [Proposal] Language extensions for better Broker support In-Reply-To: <20161216154654.GE31505@icir.org> References: <20161213102811.GY26945@ninja.local> <70454DF2-AF93-434D-B78D-0E7080B4B40D@illinois.edu> <20161213185255.GJ940@icir.org> <20161213195139.GG26945@ninja.local> <20161214152728.GB44048@icir.org> <20161216010137.GE14434@icir.org> <20161216154654.GE31505@icir.org> Message-ID: <20161216175305.GA63829@icir.org> I've updated the web page according to what I believe we have converged on so far. I've also added some "Notes" blocks where I have diverged from the discussion, let me know what you think. See here: http://bro.org/development/projects/broker-lang-ext.html On Fri, Dec 16, 2016 at 07:46 -0800, I wrote: > if ( status(v) == Broker::SUCCESS ) Thinking more about this, I kind of like this version actually, and have for now included that into the proposal. Curious to hear what others think about this. It would be an easy solution actually. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin