From robin at icir.org Fri Dec 1 06:58:25 2017 From: robin at icir.org (Robin Sommer) Date: Fri, 1 Dec 2017 06:58:25 -0800 Subject: [Bro-Dev] Feedback on configuration framework implementation In-Reply-To: <9F9FE5A5-044F-4AEA-B4AE-6F39F66AA18E@corelight.com> References: <20171129220224.ciiekzsc36of57r6@Trafalgar.local> <28946f51-7733-73e5-9c13-839e5dcf8ac4@gmail.com> <20171130180119.ssqmgzy4hgosrlht@Beezling.local> <9F9FE5A5-044F-4AEA-B4AE-6F39F66AA18E@corelight.com> Message-ID: <20171201145825.GD8269@icir.org> On Thu, Nov 30, 2017 at 10:28 -0800, you wrote: > think of that. I honestly also never liked modifying the values that are > passed in arguments; this is for example also theoretically possible for > events, but something that we have avoided to use in practice so far. Yeah, and it also won't work for atomic values, at least not since https://github.com/bro/bro/commit/5b889360705120c9061390214881ea376819c669 went in. :) Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jazoff at illinois.edu Fri Dec 1 07:22:39 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 1 Dec 2017 15:22:39 +0000 Subject: [Bro-Dev] Feedback on configuration framework implementation In-Reply-To: <20171201145825.GD8269@icir.org> References: <20171129220224.ciiekzsc36of57r6@Trafalgar.local> <28946f51-7733-73e5-9c13-839e5dcf8ac4@gmail.com> <20171130180119.ssqmgzy4hgosrlht@Beezling.local> <9F9FE5A5-044F-4AEA-B4AE-6F39F66AA18E@corelight.com> <20171201145825.GD8269@icir.org> Message-ID: > On Dec 1, 2017, at 9:58 AM, Robin Sommer wrote: > > > > On Thu, Nov 30, 2017 at 10:28 -0800, you wrote: > >> think of that. I honestly also never liked modifying the values that are >> passed in arguments; this is for example also theoretically possible for >> events, but something that we have avoided to use in practice so far. > > Yeah, and it also won't work for atomic values, at least not since > https://github.com/bro/bro/commit/5b889360705120c9061390214881ea376819c669 > went in. :) > > Robin I almost used this feature (bug) when I was trying to figure out how to implement user configurable dynamic scan thresholds! Now I'm glad I never got it to work right :-) ? Justin Azoff From robin at icir.org Fri Dec 1 07:42:51 2017 From: robin at icir.org (Robin Sommer) Date: Fri, 1 Dec 2017 07:42:51 -0800 Subject: [Bro-Dev] Feedback on configuration framework implementation In-Reply-To: References: <20171129220224.ciiekzsc36of57r6@Trafalgar.local> <28946f51-7733-73e5-9c13-839e5dcf8ac4@gmail.com> <20171130180119.ssqmgzy4hgosrlht@Beezling.local> <9F9FE5A5-044F-4AEA-B4AE-6F39F66AA18E@corelight.com> <20171201145825.GD8269@icir.org> Message-ID: <20171201154251.GG8269@icir.org> On Fri, Dec 01, 2017 at 15:22 +0000, you wrote: > Now I'm glad I never got it to work right :-) Me too. :-) Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From johanna at icir.org Fri Dec 1 14:59:58 2017 From: johanna at icir.org (Johanna Amann) Date: Fri, 1 Dec 2017 14:59:58 -0800 Subject: [Bro-Dev] Feedback on configuration framework implementation In-Reply-To: <20171201145825.GD8269@icir.org> References: <20171129220224.ciiekzsc36of57r6@Trafalgar.local> <28946f51-7733-73e5-9c13-839e5dcf8ac4@gmail.com> <20171130180119.ssqmgzy4hgosrlht@Beezling.local> <9F9FE5A5-044F-4AEA-B4AE-6F39F66AA18E@corelight.com> <20171201145825.GD8269@icir.org> Message-ID: <20171201225958.ywd7hstg5myaumll@user140.sys.ICSI.Berkeley.EDU> > > think of that. I honestly also never liked modifying the values that are > > passed in arguments; this is for example also theoretically possible for > > events, but something that we have avoided to use in practice so far. > > Yeah, and it also won't work for atomic values, at least not since > https://github.com/bro/bro/commit/5b889360705120c9061390214881ea376819c669 > went in. :) And as far as I can tell that applies to hooks too, true? This is actually a but sneaky - it should not be a problem for Intel::extend_match that Jan mentioned earlier because it is unlikely that someone will just assign a new value to info. But if someone does it will fail. Which, after thinking about it for a few moments seems like the right choice in any case. :) Johanna From robin at icir.org Sun Dec 3 15:08:47 2017 From: robin at icir.org (Robin Sommer) Date: Sun, 3 Dec 2017 15:08:47 -0800 Subject: [Bro-Dev] Feedback on configuration framework implementation In-Reply-To: <20171201225958.ywd7hstg5myaumll@user140.sys.ICSI.Berkeley.EDU> References: <20171129220224.ciiekzsc36of57r6@Trafalgar.local> <28946f51-7733-73e5-9c13-839e5dcf8ac4@gmail.com> <20171130180119.ssqmgzy4hgosrlht@Beezling.local> <9F9FE5A5-044F-4AEA-B4AE-6F39F66AA18E@corelight.com> <20171201145825.GD8269@icir.org> <20171201225958.ywd7hstg5myaumll@user140.sys.ICSI.Berkeley.EDU> Message-ID: <20171203230847.GA3627@icir.org> On Fri, Dec 01, 2017 at 14:59 -0800, you wrote: > And as far as I can tell that applies to hooks too, true? Yeah, correct. > But if someone does it will fail. > > Which, after thinking about it for a few moments seems like the right > choice in any case. :) Yeah, that seems like behaviour that's really not good to rely on. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jan.grashoefer at gmail.com Mon Dec 4 01:29:59 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Mon, 4 Dec 2017 10:29:59 +0100 Subject: [Bro-Dev] Feedback on configuration framework implementation In-Reply-To: <20171201225958.ywd7hstg5myaumll@user140.sys.ICSI.Berkeley.EDU> References: <20171129220224.ciiekzsc36of57r6@Trafalgar.local> <28946f51-7733-73e5-9c13-839e5dcf8ac4@gmail.com> <20171130180119.ssqmgzy4hgosrlht@Beezling.local> <9F9FE5A5-044F-4AEA-B4AE-6F39F66AA18E@corelight.com> <20171201145825.GD8269@icir.org> <20171201225958.ywd7hstg5myaumll@user140.sys.ICSI.Berkeley.EDU> Message-ID: <96897129-972b-9fc6-d422-e42b10d417dd@gmail.com> On 01/12/17 23:59, Johanna Amann wrote: > Which, after thinking about it for a few moments seems like the right > choice in any case.:) I think in case of hooks it might be a nice feature making hooks even more useful. For events I agree that allowing to reassign atomic values is undesirable. Jan From jazoff at illinois.edu Thu Dec 7 14:11:58 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 7 Dec 2017 22:11:58 +0000 Subject: [Bro-Dev] Feedback on configuration framework implementation In-Reply-To: <20171129220224.ciiekzsc36of57r6@Trafalgar.local> References: <20171129220224.ciiekzsc36of57r6@Trafalgar.local> Message-ID: > On Nov 29, 2017, at 5:02 PM, Johanna Amann wrote: > > The config reader provides a way to read configuration files back into > Bro. Most importantly it automatically converts values to the correct > types. This is important because it is at least inconvenient (and > sometimes near impossible) to perform the necessary type conversions in > Bro scripts themselves. This is especially true for sets/vectors. > > Configuration generally look like this: > > [option name][tab/spaces][new variable value] > > so, for example: > > testaddr 2607:f8b0:4005:801::200e > testinterval 60 > testtime 1507321987 > test_set a b c d erdbeerschnitzel > > The reader uses the option name to look up the type that variable has in > the Bro core and automatically converts the value to the correct type. What are the limits of this automatic conversion? There's currently a few use cases that are difficult to do using the input framework when then involve loading things into a nested data structure... like a table[subnet] of set[port] It can be done, but requires using the input events and doing bookkeeping yourself. Bro can serialize stuff to json, but I don't think we have the inverse implemented anywhere Could be nice to be able to lay out values using something like port_whitelist {192.168.0.0/24: [22/tcp,80/tcp], 192.168.1.0/24: [443/tcp]} Maybe this is more of a job for broker? I know broker can easily serialize and transfer such a data structure over the network, is there a plain text serialization implementation too? ? Justin Azoff From jazoff at illinois.edu Thu Dec 7 14:56:56 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 7 Dec 2017 22:56:56 +0000 Subject: [Bro-Dev] Feedback on configuration framework implementation In-Reply-To: References: <20171129220224.ciiekzsc36of57r6@Trafalgar.local>

Message-ID: <76BC2699-029D-4027-B8BF-8C0BE6EDC44E@illinois.edu> > On Dec 7, 2017, at 5:22 PM, Johanna Amann wrote: > Indeed, that is my thought. This seems like a job for broker, instead of trying to somehow force this into a complex ascii-representation. > > Note that this is just a limitation of the config reader - the rest of the config framework (that does not deal with file reading) does not care what you throw at it and will happily accept tables, etc. So if you get Broker to give you a table you should be able to just use the calls for setting options with that table afterwards. > > Johanna Cool.. so you figure something like a python script to load/organize your data from whatever upstream source you have, then just call Option::set using broker? I think the ascii representation of data structures would still help in a few places.. bro is in a weird place right now where we have json and 'print' that can output an ascii representation of almost any data structure, but what it outputs is not always valid bro code that can be parsed in the other direction. Like, const foo: table[subnet] of set[port] = { [192.168.0.0/24] = set(22/tcp) }; Gets turned into { [192.168.0.0/24] = { 22/tcp } } But the bro parser doesn't parse {...} as a set. and const foo: table[subnet] of vector of port = { [192.168.0.0/24] = vector(22/tcp) }; Gets turned into { [192.168.0.0/24] = [22/tcp] } But trying to parse that crashes: internal error in ././trybro.bro, line 2: missing aggregate in ListExpr::InitVal (22/tcp) ? Justin Azoff From johanna at corelight.com Thu Dec 7 15:01:40 2017 From: johanna at corelight.com (Johanna Amann) Date: Thu, 07 Dec 2017 15:01:40 -0800 Subject: [Bro-Dev] Feedback on configuration framework implementation In-Reply-To: <76BC2699-029D-4027-B8BF-8C0BE6EDC44E@illinois.edu> References: <20171129220224.ciiekzsc36of57r6@Trafalgar.local>

<76BC2699-029D-4027-B8BF-8C0BE6EDC44E@illinois.edu> Message-ID: <1DED12C9-74FD-4D9B-8FEA-0CD10935DDA1@corelight.com> On 7 Dec 2017, at 14:56, Azoff, Justin S wrote: >> On Dec 7, 2017, at 5:22 PM, Johanna Amann >> wrote: > >> Indeed, that is my thought. This seems like a job for broker, instead >> of trying to somehow force this into a complex ascii-representation. >> >> Note that this is just a limitation of the config reader - the rest >> of the config framework (that does not deal with file reading) does >> not care what you throw at it and will happily accept tables, etc. So >> if you get Broker to give you a table you should be able to just use >> the calls for setting options with that table afterwards. >> >> Johanna > > Cool.. so you figure something like a python script to load/organize > your data from whatever upstream source you have, then just call > Option::set using broker? Yup. > I think the ascii representation of data structures would still help > in a few places.. bro is in a weird place right now where we have json > and 'print' that can output an > ascii representation of almost any data structure, but what it outputs > is not always valid bro code that can be parsed in the other > direction. You might have a point - however that kind of is a problem that I think is out of scope for the config framework. Plus - if we ever introduce something it will work without changes with the config framework as it is now (and the config reader should be able to read anything new that the input framework supports because it just re-uses functionality) Johanna From robin at icir.org Thu Dec 14 12:07:28 2017 From: robin at icir.org (Robin Sommer) Date: Thu, 14 Dec 2017 12:07:28 -0800 Subject: [Bro-Dev] [Bro-Commits] [git/broker] topic/actor-system: Add broker::bro::RelayEvent message type. (ce86016) In-Reply-To: <201712141906.vBEJ6tPw006767@bro-ids.icir.org> References: <201712141906.vBEJ6tPw006767@bro-ids.icir.org> Message-ID: <20171214200728.GE13971@icir.org> Hi Jon, I'm curious what's the use case for this? Generally I think it's best to use a combination of Broker-internal forwarding and topics to control where messages get propagated too, as that will better generalize to larger topologies later. The less of the routing we control manually at the Bro level, the better I'd say. But that's not an absolute rule of course, and I may just be missing what this is aiming at. Robin On Thu, Dec 14, 2017 at 12:55 -0600, Jonathan Siwek wrote: > Repository : ssh://git at bro-ids.icir.org/broker > On branch : topic/actor-system > > >--------------------------------------------------------------- > > commit ce860168961285c6d961973aa9e1ef6b7de87887 > Author: Jon Siwek > Date: Thu Dec 14 12:55:52 2017 -0600 > > Add broker::bro::RelayEvent message type. > > This is meant to be a more convenient/controlled/explicit way of doing > simple one-hop message forwarding. > > > >--------------------------------------------------------------- > > ce860168961285c6d961973aa9e1ef6b7de87887 > broker/bro.hh | 36 ++++++++++++++++++++++++++++++++---- > 1 file changed, 32 insertions(+), 4 deletions(-) > > diff --git a/broker/bro.hh b/broker/bro.hh > index 0d7f0e7..0ec93f2 100644 > --- a/broker/bro.hh > +++ b/broker/bro.hh > @@ -18,6 +18,7 @@ public: > LogWrite = 3, > IdentifierUpdate = 4, > Batch = 5, > + RelayEvent = 6, > }; > > Type type() const { > @@ -50,7 +51,7 @@ protected: > class Event : public Message { > public: > Event(std::string name, vector args) > - : Message(Message::Type::Event, {name, std::move(args)}) {} > + : Message(Message::Type::Event, {std::move(name), std::move(args)}) {} > Event(data msg) : Message(std::move(msg)) {} > > const std::string& name() const { > @@ -62,6 +63,30 @@ class Event : public Message { > } > }; > > +/// A Bro relayed event (automatically republished after a single hop). > +class RelayEvent : public Message { > + public: > + RelayEvent(set relay_topics, std::string name, vector args) > + : Message(Message::Type::RelayEvent, {std::move(relay_topics), > + std::move(name), > + std::move(args)}) > + {} > + RelayEvent(data msg) : Message(std::move(msg)) {} > + > + const set& topics() const { > + return get(get(msg_[2])[0]); > + } > + > + const std::string& name() const { > + return get(get(msg_[2])[1]); > + } > + > + const vector& args() const { > + return get(get(msg_[2])[2]); > + } > +}; > + > + > /// A batch of other messages. > class Batch : public Message { > public: > @@ -81,7 +106,8 @@ public: > LogCreate(enum_value stream_id, enum_value writer_id, data writer_info, > data fields_data) > : Message(Message::Type::LogCreate, > - {stream_id, writer_id, writer_info, fields_data}) { > + {std::move(stream_id), std::move(writer_id), > + std::move(writer_info), std::move(fields_data)}) { > } > > LogCreate(data msg) : Message(std::move(msg)) { > @@ -108,7 +134,8 @@ public: > LogWrite(enum_value stream_id, enum_value writer_id, data path, > data vals_data) > : Message(Message::Type::LogWrite, > - {stream_id, writer_id, path, vals_data}) { > + {std::move(stream_id), std::move(writer_id), > + std::move(path), std::move(vals_data)}) { > } > > LogWrite(data msg) : Message(std::move(msg)) { > @@ -131,7 +158,8 @@ public: > class IdentifierUpdate : public Message { > public: > IdentifierUpdate(std::string id_name, data id_value) > - : Message(Message::Type::IdentifierUpdate, {id_name, id_value}) { > + : Message(Message::Type::IdentifierUpdate, {std::move(id_name), > + std::move(id_value)}) { > } > > IdentifierUpdate(data msg) : Message(std::move(msg)) { > > > > _______________________________________________ > bro-commits mailing list > bro-commits at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-commits > -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at corelight.com Thu Dec 14 13:27:16 2017 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 14 Dec 2017 15:27:16 -0600 Subject: [Bro-Dev] [Bro-Commits] [git/broker] topic/actor-system: Add broker::bro::RelayEvent message type. (ce86016) In-Reply-To: <20171214200728.GE13971@icir.org> References: <201712141906.vBEJ6tPw006767@bro-ids.icir.org> <20171214200728.GE13971@icir.org> Message-ID: On Thu, Dec 14, 2017 at 2:07 PM, Robin Sommer wrote: > I'm curious what's the use case for this? >From last month's thread regarding cluster API/topology changes, the message pattern of sending an event from a worker to all other workers or proxy/data to all other proxy/data nodes (within these classes, nodes aren't connected with each other) was frequent enough to warrant putting in a single, simple function call like this. e.g. the next thing to come will be adding a function call to do this: Cluster::relay_round_robin(Cluster::proxy_pool, key, Cluster::worker_topic, event, event_args); Which you could use to send an event from a worker to all other workers via a rotating proxy/data node. I think at least Justin has been pushing for something like this. > Generally I think it's best > to use a combination of Broker-internal forwarding and topics to > control where messages get propagated too, as that will better > generalize to larger topologies later. At least currently, Bro has the forwarding capability of Broker turned off. IIRC, it was very easy to unintentionally create routing loops when it was turned on and when I asked about it, no one gave any justification or use-case for it, thus it got turned off. > The less of the routing we > control manually at the Bro level, the better I'd say. But that's not > an absolute rule of course, and I may just be missing what this is > aiming at. I'm probably definitely missing some info on long-term plans for larger topologies and deep-clustering, so let me know if it's something important to be considering, though it seems like in the near-term goals and use-cases for the cluster topology, the implicit routing actually may be more confusing/difficult/error-prone to use than a simple function call like this which clearly convey's the sender's intent and expectations. - Jon From robin at icir.org Fri Dec 15 08:29:12 2017 From: robin at icir.org (Robin Sommer) Date: Fri, 15 Dec 2017 08:29:12 -0800 Subject: [Bro-Dev] [Bro-Commits] [git/broker] topic/actor-system: Add broker::bro::RelayEvent message type. (ce86016) In-Reply-To: References: <201712141906.vBEJ6tPw006767@bro-ids.icir.org> <20171214200728.GE13971@icir.org> Message-ID: <20171215162912.GC75940@icir.org> On Thu, Dec 14, 2017 at 15:27 -0600, you wrote: > or proxy/data to all other proxy/data nodes (within these classes, > nodes aren't connected with each other) was frequent enough to warrant > putting in a single, simple function call like this. Yeah, though then I'd actually say it's worth keeping multi-hop topologies in mind at least. We don't need to fully solve this right now; nobody really knows yet how topologies may end up looking in the future. But a good rule of thumb I think is considering that generally nodes may not be reachable directly, but only through other Broker hops, and that Broker takes care of the routing to get messages there as long as topic subscriptions are set up appropiately. Seems that's actually what we have here already for some nodes. Could we use Broker-level routing here to get connectivity between the nodes that aren't directly connected? > At least currently, Bro has the forwarding capability of Broker turned > off. IIRC, it was very easy to unintentionally create routing loops > when it was turned on and when I asked about it, no one gave any > justification or use-case for it, thus it got turned off. It's off because we indeed don't really have understood yet how to use it. :) But Broker has message TTLs already for detecting loops, and generally I don't think there's anything preventing us from turning it on; it should work. That all said, the priority right now is on getting a basic cluster to work, so no problem doing the relaying as you have it now if that gets us there the quickest. Let's just keep thinking about how such mechanisms will look in the future in other topologies, so that we don't back us into a corner (and I hear your of course: to really do that, we need to understand better where we want to get to). Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at corelight.com Fri Dec 15 10:35:36 2017 From: jsiwek at corelight.com (Jon Siwek) Date: Fri, 15 Dec 2017 12:35:36 -0600 Subject: [Bro-Dev] [Bro-Commits] [git/broker] topic/actor-system: Add broker::bro::RelayEvent message type. (ce86016) In-Reply-To: <20171215162912.GC75940@icir.org> References: <201712141906.vBEJ6tPw006767@bro-ids.icir.org> <20171214200728.GE13971@icir.org> <20171215162912.GC75940@icir.org> Message-ID: On Fri, Dec 15, 2017 at 10:29 AM, Robin Sommer wrote: > Yeah, though then I'd actually say it's worth keeping multi-hop > topologies in mind at least. We don't need to fully solve this right > now; nobody really knows yet how topologies may end up looking in the > future. But a good rule of thumb I think is considering that generally > nodes may not be reachable directly, but only through other Broker > hops, and that Broker takes care of the routing to get messages there > as long as topic subscriptions are set up appropiately. Seems that's > actually what we have here already for some nodes. Could we use > Broker-level routing here to get connectivity between the nodes that > aren't directly connected? For the case of worker to all other workers via a single proxy, I'm not sure it's simple to do with implicit routing. You would need a topic subscription in common between the proxy and workers and you also want that topic to be unique among all proxies, otherwise you'd generate many duplicate messages as you send to all proxies and then all proxies send to all workers. I think you'd end up with a long subscription lists that look like: proxy-1: relay_to_workers_via_proxy_1 proxy-2: relay_to_workers_via_proxy_2 ... worker-1: relay_to_workers_via_proxy_1, relay_to_workers_via_proxy_2, ... worker-2: same as worker-1 And that works and could be automatically generated, though I think it's conflicting w/ how I intuitively want to try and categorize topics/subscriptions: what I end up looking for is to have topics identify either a single node or a whole class of nodes, so instead of encoding the intent of the message pattern in the topic names, I want function calls that define the semantics of the messaging pattern and supply to it the identities/classes of which nodes should be involved in that pattern. Another thing I like more about an explicit relay function is that it's clear that I don't want the relaying node(s) to actually do any any event handling, I just want them to forward it on, but if I did want them to handle it, then I still have the option of sending to them directly using the publish() function. With the internal routing mechanism, the sender doesn't have as good control/flexibility of which nodes on the route will end up handling events, at least not without doing more planning/configuration in advance to set up those separate avenues. >> At least currently, Bro has the forwarding capability of Broker turned >> off. IIRC, it was very easy to unintentionally create routing loops >> when it was turned on and when I asked about it, no one gave any >> justification or use-case for it, thus it got turned off. > > It's off because we indeed don't really have understood yet how to use > it. :) But Broker has message TTLs already for detecting loops, and > generally I don't think there's anything preventing us from turning it > on; it should work. I meant that I was the one who turned it off when I started porting scripts because the first thing I tried to do ended up creating a loop. Yeah, the 20-hop TTL eventually killed things eventually, though it wasn't a good first impression :) And I think that particular loop would still be there if it got turned on right now... also not certain, but that particular one might not be so harmless to leave routing around, it might actually have side effects with how cluster nodes track each other's identities when they join/leave the cluster and completely break it. > That all said, the priority right now is on getting a basic cluster to > work, so no problem doing the relaying as you have it now if that gets > us there the quickest. Let's just keep thinking about how such > mechanisms will look in the future in other topologies, so that we > don't back us into a corner (and I hear your of course: to really do > that, we need to understand better where we want to get to). Yeah, agree that it should become easier to think about how the extended cluster/routing stuff should come into play after the first step of porting to broker is done and is available to iterate upon instead of trying to do both steps in one go. - Jon From jan.grashoefer at gmail.com Tue Dec 19 07:54:37 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Tue, 19 Dec 2017 16:54:37 +0100 Subject: [Bro-Dev] Closures in Bro Message-ID: <90195bf8-caa8-e151-dcdd-1ebccd2bb503@gmail.com> Hi everyone, playing around with the add-json package, I realized that I will need to "extend" functions like the path_func at some point. That is, I need to calculate a value using the original version of the function and "add" my calculations on top. My approach is to wrap the function, which works as long as I don't need "multiple versions" of that function. In the latter case closures would help, e.g.: function do_add(i: int): function(j: int): int { return function(j: int): int {return i+j; }; } However, referencing outer function IDs is not supported (see http://try.bro.org/#/trybro/saved/196147). In the light of the flexibility that comes with packages, I think supporting closures would be a nice feature. Was there any fundamental design decision against supporting closures? Jan From jsiwek at corelight.com Tue Dec 19 09:31:27 2017 From: jsiwek at corelight.com (Jon Siwek) Date: Tue, 19 Dec 2017 11:31:27 -0600 Subject: [Bro-Dev] Closures in Bro In-Reply-To: <90195bf8-caa8-e151-dcdd-1ebccd2bb503@gmail.com> References: <90195bf8-caa8-e151-dcdd-1ebccd2bb503@gmail.com> Message-ID: On Tue, Dec 19, 2017 at 9:54 AM, Jan Grash?fer wrote: > In the light of the flexibility that comes with packages, I think > supporting closures would be a nice feature. Was there any fundamental > design decision against supporting closures? I think it's mostly unsupported due to trickiness of implementation (or cowardice) rather than by design decision. Or at least something scared me off when I looked into it a few years ago [1]. Don't remember the details, though I likely didn't give it more than a day or two of thought/effort, so maybe don't let that deter you if you had plans to look into it. - Jon [1] https://bro-tracker.atlassian.net/browse/BIT-1233 From scampbell at lbl.gov Tue Dec 19 16:36:18 2017 From: scampbell at lbl.gov (Scott Campbell) Date: Tue, 19 Dec 2017 16:36:18 -0800 Subject: [Bro-Dev] netmap lb issues - not forwarding packets after a few min Message-ID: <1e1f8985-8630-21a8-ae46-97880cf7410f@lbl.gov> Currently running bro current (2.5-372) on a Scientific Linux release 6.9 kernel 2.6.32-696.13.2.el6.x86_64 . The bro netmap module was added per directions: bro at xdev-m ~/bin> ./bro -N Bro::Netmap Bro::Netmap - Packet acquisition via Netmap (dynamic, version 1.0) I downloaded and installed the current git netmap ( MODULE_INFO(srcversion, "70F039B58865AAE47076678") ) without issue, and there are no messages when the modules load besides what you would expect to see. What I am seeing is when I run the lb application it runs as expected for a few minutes, then stops forwarding packets and continues logging the same line regardless of how long you wait. Sample logs follow. lb starts: > [root at xdev-w1 lb]# ./lb -i eth5 -o 10 -p8 -B 1024 > 933.751533 main [600] interface is eth5 > 933.751614 main [621] requested 1024 extra buffers > 934.080683 main [714] successfully opened netmap:eth5 (tx rings: 512) > 934.080699 main [725] obtained 1024 extra buffers > 934.081586 main [784] opening pipe named netmap:eth5{0/xT at 1 > 934.081627 nm_mmap [987] do not mmap, inherit from parent > 934.081639 main [799] successfully opened pipe #1 netmap:eth5{0/xT at 1 (tx slots: 512) > 934.081646 main [803] zerocopy enabled > 934.081671 main [784] opening pipe named netmap:eth5{1/xT at 1 > 934.081692 nm_mmap [987] do not mmap, inherit from parent > 934.081700 main [799] successfully opened pipe #2 netmap:eth5{1/xT at 1 (tx slots: 512) > 934.081706 main [803] zerocopy enabled > 934.081729 main [784] opening pipe named netmap:eth5{2/xT at 1 > 934.081746 nm_mmap [987] do not mmap, inherit from parent > 934.081754 main [799] successfully opened pipe #3 netmap:eth5{2/xT at 1 (tx slots: 512) > 934.081760 main [803] zerocopy enabled > 934.081786 main [784] opening pipe named netmap:eth5{3/xT at 1 > 934.081803 nm_mmap [987] do not mmap, inherit from parent > 934.081813 main [799] successfully opened pipe #4 netmap:eth5{3/xT at 1 (tx slots: 512) > 934.081819 main [803] zerocopy enabled > 934.081842 main [784] opening pipe named netmap:eth5{4/xT at 1 > 934.081862 nm_mmap [987] do not mmap, inherit from parent > 934.081870 main [799] successfully opened pipe #5 netmap:eth5{4/xT at 1 (tx slots: 512) > 934.081876 main [803] zerocopy enabled > 934.081899 main [784] opening pipe named netmap:eth5{5/xT at 1 > 934.081916 nm_mmap [987] do not mmap, inherit from parent > 934.081923 main [799] successfully opened pipe #6 netmap:eth5{5/xT at 1 (tx slots: 512) > 934.081929 main [803] zerocopy enabled > 934.081954 main [784] opening pipe named netmap:eth5{6/xT at 1 > 934.081972 nm_mmap [987] do not mmap, inherit from parent > 934.081980 main [799] successfully opened pipe #7 netmap:eth5{6/xT at 1 (tx slots: 512) > 934.081986 main [803] zerocopy enabled > 934.082013 main [784] opening pipe named netmap:eth5{7/xT at 1 > 934.082031 nm_mmap [987] do not mmap, inherit from parent > 934.082041 main [799] successfully opened pipe #8 netmap:eth5{7/xT at 1 (tx slots: 512) > 934.082046 main [803] zerocopy enabled > {"ts":1513728945.082923,"interface":"netmap:eth5{0/xT at 1","output_ring":0,"packets_forwarded":85397,"packets_dropped":0,"data_forward_rate_Mbps":160.3504,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":19.4110,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728945.082923,"interface":"netmap:eth5{1/xT at 1","output_ring":1,"packets_forwarded":65319,"packets_dropped":0,"data_forward_rate_Mbps":29.0976,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":8.5940,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728945.082923,"interface":"netmap:eth5{2/xT at 1","output_ring":2,"packets_forwarded":317351,"packets_dropped":1300,"data_forward_rate_Mbps":395.4482,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":35.4900,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728945.082923,"interface":"netmap:eth5{3/xT at 1","output_ring":3,"packets_forwarded":100570,"packets_dropped":0,"data_forward_rate_Mbps":148.8784,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":16.3190,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728945.082923,"interface":"netmap:eth5{4/xT at 1","output_ring":4,"packets_forwarded":75111,"packets_dropped":0,"data_forward_rate_Mbps":91.5148,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":11.4440,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728945.082923,"interface":"netmap:eth5{5/xT at 1","output_ring":5,"packets_forwarded":66920,"packets_dropped":0,"data_forward_rate_Mbps":66.1700,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":8.0000,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728945.082923,"interface":"netmap:eth5{6/xT at 1","output_ring":6,"packets_forwarded":143992,"packets_dropped":0,"data_forward_rate_Mbps":170.3500,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":17.5980,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728945.082923,"interface":"netmap:eth5{7/xT at 1","output_ring":7,"packets_forwarded":67032,"packets_dropped":0,"data_forward_rate_Mbps":29.2870,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":5.2020,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728945.082923,"interface":"netmap:eth5","output_ring":null,"packets_received":1035728,"packets_forwarded":921692,"packets_dropped":1300,"non_ip_packets":18,"data_forward_rate_Mbps":1091.0964,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":122.0570,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":1024} > {"ts":1513728955.083739,"interface":"netmap:eth5{0/xT at 1","output_ring":0,"packets_forwarded":517056,"packets_dropped":31870,"data_forward_rate_Mbps":583.9244,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":46.9350,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728955.083739,"interface":"netmap:eth5{1/xT at 1","output_ring":1,"packets_forwarded":415852,"packets_dropped":0,"data_forward_rate_Mbps":583.7359,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":48.9310,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728955.083739,"interface":"netmap:eth5{2/xT at 1","output_ring":2,"packets_forwarded":998058,"packets_dropped":1300,"data_forward_rate_Mbps":925.2677,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":106.8910,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728955.083739,"interface":"netmap:eth5{3/xT at 1","output_ring":3,"packets_forwarded":836948,"packets_dropped":14154,"data_forward_rate_Mbps":1321.3131,"data_drop_rate_Mbps":15.6000,"packet_forward_rate_kpps":91.2690,"packet_drop_rate_kpps":1.0790,"overflow_queue_size":0} > {"ts":1513728955.083739,"interface":"netmap:eth5{4/xT at 1","output_ring":4,"packets_forwarded":409670,"packets_dropped":0,"data_forward_rate_Mbps":519.5414,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":40.8310,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728955.083739,"interface":"netmap:eth5{5/xT at 1","output_ring":5,"packets_forwarded":395510,"packets_dropped":0,"data_forward_rate_Mbps":545.0672,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":42.6580,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728955.083739,"interface":"netmap:eth5{6/xT at 1","output_ring":6,"packets_forwarded":555556,"packets_dropped":0,"data_forward_rate_Mbps":592.1166,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":47.5850,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728955.083739,"interface":"netmap:eth5{7/xT at 1","output_ring":7,"packets_forwarded":393490,"packets_dropped":0,"data_forward_rate_Mbps":515.1810,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":41.6850,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728955.083739,"interface":"netmap:eth5","output_ring":null,"packets_received":5323612,"packets_forwarded":4522140,"packets_dropped":47324,"non_ip_packets":18,"data_forward_rate_Mbps":5586.1473,"data_drop_rate_Mbps":15.6000,"packet_forward_rate_kpps":466.7870,"packet_drop_rate_kpps":1.0790,"free_buffer_slots":1024} this continues for some time then : > {"ts":1513729195.106307,"interface":"netmap:eth5{0/xT at 1","output_ring":0,"packets_forwarded":4920687,"packets_dropped":33825,"data_forward_rate_Mbps":10.7695,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":2.0180,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513729195.106307,"interface":"netmap:eth5{1/xT at 1","output_ring":1,"packets_forwarded":5930891,"packets_dropped":3760,"data_forward_rate_Mbps":47.3625,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":9.5640,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513729195.106307,"interface":"netmap:eth5{2/xT at 1","output_ring":2,"packets_forwarded":16009392,"packets_dropped":130519,"data_forward_rate_Mbps":849.3118,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":77.2920,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513729195.106307,"interface":"netmap:eth5{3/xT at 1","output_ring":3,"packets_forwarded":7001918,"packets_dropped":1337878,"data_forward_rate_Mbps":36.4586,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":5.6990,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513729195.106307,"interface":"netmap:eth5{4/xT at 1","output_ring":4,"packets_forwarded":5250042,"packets_dropped":1403,"data_forward_rate_Mbps":161.3338,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":17.7030,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513729195.106307,"interface":"netmap:eth5{5/xT at 1","output_ring":5,"packets_forwarded":5426001,"packets_dropped":0,"data_forward_rate_Mbps":67.0789,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":7.9150,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513729195.106307,"interface":"netmap:eth5{6/xT at 1","output_ring":6,"packets_forwarded":5991695,"packets_dropped":0,"data_forward_rate_Mbps":58.6742,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":7.5420,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513729195.106307,"interface":"netmap:eth5{7/xT at 1","output_ring":7,"packets_forwarded":5094833,"packets_dropped":0,"data_forward_rate_Mbps":152.4217,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":15.5730,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513729195.106307,"interface":"netmap:eth5","output_ring":null,"packets_received":62232503,"packets_forwarded":55625459,"packets_dropped":1507385,"non_ip_packets":413,"data_forward_rate_Mbps":1383.4110,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":143.3050,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":1024} > {"ts":1513729195.106307,"interface":"netmap:eth5","output_ring":null,"packets_received":62232503,"packets_forwarded":55625459,"packets_dropped":1507385,"non_ip_packets":413,"data_forward_rate_Mbps":1383.4110,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":143.3050,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":1024} > {"ts":1513729195.106307,"interface":"netmap:eth5","output_ring":null,"packets_received":62232503,"packets_forwarded":55625459,"packets_dropped":1507385,"non_ip_packets":413,"data_forward_rate_Mbps":1383.4110,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":143.3050,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":1024} > {"ts":1513729195.106307,"interface":"netmap:eth5","output_ring":null,"packets_received":62232503,"packets_forwarded":55625459,"packets_dropped":1507385,"non_ip_packets":413,"data_forward_rate_Mbps":1383.4110,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":143.3050,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":1024} > {"ts":1513729195.106307,"interface":"netmap:eth5","output_ring":null,"packets_received":62232503,"packets_forwarded":55625459,"packets_dropped":1507385,"non_ip_packets":413,"data_forward_rate_Mbps":1383.4110,"data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":143.3050,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":1024} The individual sub interfaces created by lb no longer report, but every 10 seconds (as configured) 9 identical lines (8 from -p 8 , and one from eth5) print out with the numbers no longer changing. Interface counters on eth5 continue to move so the interface is still seeing data. As well the time it takes for this to happen seems to vary. I have reloaded the modules, run lb as a user or root, changed the configuration parameters, rebooted the system all to no avail. Any thoughts? scott From balint.martina at gmail.com Tue Dec 19 22:27:48 2017 From: balint.martina at gmail.com (Martina Balintova) Date: Wed, 20 Dec 2017 06:27:48 +0000 Subject: [Bro-Dev] netmap lb issues - not forwarding packets after a few min In-Reply-To: <1e1f8985-8630-21a8-ae46-97880cf7410f@lbl.gov> References: <1e1f8985-8630-21a8-ae46-97880cf7410f@lbl.gov> Message-ID: Try to run bro or tcpdump feeding from the pipes eth}0,.. I think I encountered the same problem, if there is no application feeding of the lb's output, it can get stuck. Martina On Dec 20, 2017 00:40, "Scott Campbell" wrote: Currently running bro current (2.5-372) on a Scientific Linux release 6.9 kernel 2.6.32-696.13.2.el6.x86_64 . The bro netmap module was added per directions: bro at xdev-m ~/bin> ./bro -N Bro::Netmap Bro::Netmap - Packet acquisition via Netmap (dynamic, version 1.0) I downloaded and installed the current git netmap ( MODULE_INFO(srcversion, "70F039B58865AAE47076678") ) without issue, and there are no messages when the modules load besides what you would expect to see. What I am seeing is when I run the lb application it runs as expected for a few minutes, then stops forwarding packets and continues logging the same line regardless of how long you wait. Sample logs follow. lb starts: > [root at xdev-w1 lb]# ./lb -i eth5 -o 10 -p8 -B 1024 > 933.751533 main [600] interface is eth5 > 933.751614 main [621] requested 1024 extra buffers > 934.080683 main [714] successfully opened netmap:eth5 (tx rings: 512) > 934.080699 main [725] obtained 1024 extra buffers > 934.081586 main [784] opening pipe named netmap:eth5{0/xT at 1 > 934.081627 nm_mmap [987] do not mmap, inherit from parent > 934.081639 main [799] successfully opened pipe #1 netmap:eth5{0/xT at 1 (tx slots: 512) > 934.081646 main [803] zerocopy enabled > 934.081671 main [784] opening pipe named netmap:eth5{1/xT at 1 > 934.081692 nm_mmap [987] do not mmap, inherit from parent > 934.081700 main [799] successfully opened pipe #2 netmap:eth5{1/xT at 1 (tx slots: 512) > 934.081706 main [803] zerocopy enabled > 934.081729 main [784] opening pipe named netmap:eth5{2/xT at 1 > 934.081746 nm_mmap [987] do not mmap, inherit from parent > 934.081754 main [799] successfully opened pipe #3 netmap:eth5{2/xT at 1 (tx slots: 512) > 934.081760 main [803] zerocopy enabled > 934.081786 main [784] opening pipe named netmap:eth5{3/xT at 1 > 934.081803 nm_mmap [987] do not mmap, inherit from parent > 934.081813 main [799] successfully opened pipe #4 netmap:eth5{3/xT at 1 (tx slots: 512) > 934.081819 main [803] zerocopy enabled > 934.081842 main [784] opening pipe named netmap:eth5{4/xT at 1 > 934.081862 nm_mmap [987] do not mmap, inherit from parent > 934.081870 main [799] successfully opened pipe #5 netmap:eth5{4/xT at 1 (tx slots: 512) > 934.081876 main [803] zerocopy enabled > 934.081899 main [784] opening pipe named netmap:eth5{5/xT at 1 > 934.081916 nm_mmap [987] do not mmap, inherit from parent > 934.081923 main [799] successfully opened pipe #6 netmap:eth5{5/xT at 1 (tx slots: 512) > 934.081929 main [803] zerocopy enabled > 934.081954 main [784] opening pipe named netmap:eth5{6/xT at 1 > 934.081972 nm_mmap [987] do not mmap, inherit from parent > 934.081980 main [799] successfully opened pipe #7 netmap:eth5{6/xT at 1 (tx slots: 512) > 934.081986 main [803] zerocopy enabled > 934.082013 main [784] opening pipe named netmap:eth5{7/xT at 1 > 934.082031 nm_mmap [987] do not mmap, inherit from parent > 934.082041 main [799] successfully opened pipe #8 netmap:eth5{7/xT at 1 (tx slots: 512) > 934.082046 main [803] zerocopy enabled > {"ts":1513728945.082923,"interface":"netmap:eth5{0/xT at 1 ","output_ring":0,"packets_forwarded":85397,"packets_ dropped":0,"data_forward_rate_Mbps":160.3504,"data_drop_ rate_Mbps":0.0000,"packet_forward_rate_kpps":19.4110," packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728945.082923,"interface":"netmap:eth5{1/xT at 1 ","output_ring":1,"packets_forwarded":65319,"packets_ dropped":0,"data_forward_rate_Mbps":29.0976,"data_drop_rate_ Mbps":0.0000,"packet_forward_rate_kpps":8.5940,"packet_ drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728945.082923,"interface":"netmap:eth5{2/xT at 1 ","output_ring":2,"packets_forwarded":317351,"packets_ dropped":1300,"data_forward_rate_Mbps":395.4482,"data_ drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":35. 4900,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728945.082923,"interface":"netmap:eth5{3/xT at 1 ","output_ring":3,"packets_forwarded":100570,"packets_ dropped":0,"data_forward_rate_Mbps":148.8784,"data_drop_ rate_Mbps":0.0000,"packet_forward_rate_kpps":16.3190," packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728945.082923,"interface":"netmap:eth5{4/xT at 1 ","output_ring":4,"packets_forwarded":75111,"packets_ dropped":0,"data_forward_rate_Mbps":91.5148,"data_drop_rate_ Mbps":0.0000,"packet_forward_rate_kpps":11.4440,"packet_ drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728945.082923,"interface":"netmap:eth5{5/xT at 1 ","output_ring":5,"packets_forwarded":66920,"packets_ dropped":0,"data_forward_rate_Mbps":66.1700,"data_drop_rate_ Mbps":0.0000,"packet_forward_rate_kpps":8.0000,"packet_ drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728945.082923,"interface":"netmap:eth5{6/xT at 1 ","output_ring":6,"packets_forwarded":143992,"packets_ dropped":0,"data_forward_rate_Mbps":170.3500,"data_drop_ rate_Mbps":0.0000,"packet_forward_rate_kpps":17.5980," packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728945.082923,"interface":"netmap:eth5{7/xT at 1 ","output_ring":7,"packets_forwarded":67032,"packets_ dropped":0,"data_forward_rate_Mbps":29.2870,"data_drop_rate_ Mbps":0.0000,"packet_forward_rate_kpps":5.2020,"packet_ drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728945.082923,"interface":"netmap:eth5"," output_ring":null,"packets_received":1035728,"packets_ forwarded":921692,"packets_dropped":1300,"non_ip_packets" :18,"data_forward_rate_Mbps":1091.0964,"data_drop_rate_ Mbps":0.0000,"packet_forward_rate_kpps":122.0570,"packet_ drop_rate_kpps":0.0000,"free_buffer_slots":1024} > {"ts":1513728955.083739,"interface":"netmap:eth5{0/xT at 1 ","output_ring":0,"packets_forwarded":517056,"packets_ dropped":31870,"data_forward_rate_Mbps":583.9244,"data_ drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":46. 9350,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728955.083739,"interface":"netmap:eth5{1/xT at 1 ","output_ring":1,"packets_forwarded":415852,"packets_ dropped":0,"data_forward_rate_Mbps":583.7359,"data_drop_ rate_Mbps":0.0000,"packet_forward_rate_kpps":48.9310," packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728955.083739,"interface":"netmap:eth5{2/xT at 1 ","output_ring":2,"packets_forwarded":998058,"packets_ dropped":1300,"data_forward_rate_Mbps":925.2677,"data_ drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":106. 8910,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728955.083739,"interface":"netmap:eth5{3/xT at 1 ","output_ring":3,"packets_forwarded":836948,"packets_ dropped":14154,"data_forward_rate_Mbps":1321.3131,"data_ drop_rate_Mbps":15.6000,"packet_forward_rate_kpps":91. 2690,"packet_drop_rate_kpps":1.0790,"overflow_queue_size":0} > {"ts":1513728955.083739,"interface":"netmap:eth5{4/xT at 1 ","output_ring":4,"packets_forwarded":409670,"packets_ dropped":0,"data_forward_rate_Mbps":519.5414,"data_drop_ rate_Mbps":0.0000,"packet_forward_rate_kpps":40.8310," packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728955.083739,"interface":"netmap:eth5{5/xT at 1 ","output_ring":5,"packets_forwarded":395510,"packets_ dropped":0,"data_forward_rate_Mbps":545.0672,"data_drop_ rate_Mbps":0.0000,"packet_forward_rate_kpps":42.6580," packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728955.083739,"interface":"netmap:eth5{6/xT at 1 ","output_ring":6,"packets_forwarded":555556,"packets_ dropped":0,"data_forward_rate_Mbps":592.1166,"data_drop_ rate_Mbps":0.0000,"packet_forward_rate_kpps":47.5850," packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728955.083739,"interface":"netmap:eth5{7/xT at 1 ","output_ring":7,"packets_forwarded":393490,"packets_ dropped":0,"data_forward_rate_Mbps":515.1810,"data_drop_ rate_Mbps":0.0000,"packet_forward_rate_kpps":41.6850," packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513728955.083739,"interface":"netmap:eth5"," output_ring":null,"packets_received":5323612,"packets_ forwarded":4522140,"packets_dropped":47324,"non_ip_ packets":18,"data_forward_rate_Mbps":5586.1473,"data_ drop_rate_Mbps":15.6000,"packet_forward_rate_kpps":466. 7870,"packet_drop_rate_kpps":1.0790,"free_buffer_slots":1024} this continues for some time then : > {"ts":1513729195.106307,"interface":"netmap:eth5{0/xT at 1 ","output_ring":0,"packets_forwarded":4920687,"packets_ dropped":33825,"data_forward_rate_Mbps":10.7695,"data_drop_ rate_Mbps":0.0000,"packet_forward_rate_kpps":2.0180," packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513729195.106307,"interface":"netmap:eth5{1/xT at 1 ","output_ring":1,"packets_forwarded":5930891,"packets_ dropped":3760,"data_forward_rate_Mbps":47.3625,"data_drop_ rate_Mbps":0.0000,"packet_forward_rate_kpps":9.5640," packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513729195.106307,"interface":"netmap:eth5{2/xT at 1 ","output_ring":2,"packets_forwarded":16009392,"packets_ dropped":130519,"data_forward_rate_Mbps":849.3118,"data_ drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":77. 2920,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513729195.106307,"interface":"netmap:eth5{3/xT at 1 ","output_ring":3,"packets_forwarded":7001918,"packets_ dropped":1337878,"data_forward_rate_Mbps":36.4586," data_drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":5. 6990,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513729195.106307,"interface":"netmap:eth5{4/xT at 1 ","output_ring":4,"packets_forwarded":5250042,"packets_ dropped":1403,"data_forward_rate_Mbps":161.3338,"data_ drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":17. 7030,"packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513729195.106307,"interface":"netmap:eth5{5/xT at 1 ","output_ring":5,"packets_forwarded":5426001,"packets_ dropped":0,"data_forward_rate_Mbps":67.0789,"data_drop_rate_ Mbps":0.0000,"packet_forward_rate_kpps":7.9150,"packet_ drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513729195.106307,"interface":"netmap:eth5{6/xT at 1 ","output_ring":6,"packets_forwarded":5991695,"packets_ dropped":0,"data_forward_rate_Mbps":58.6742,"data_drop_rate_ Mbps":0.0000,"packet_forward_rate_kpps":7.5420,"packet_ drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513729195.106307,"interface":"netmap:eth5{7/xT at 1 ","output_ring":7,"packets_forwarded":5094833,"packets_ dropped":0,"data_forward_rate_Mbps":152.4217,"data_drop_ rate_Mbps":0.0000,"packet_forward_rate_kpps":15.5730," packet_drop_rate_kpps":0.0000,"overflow_queue_size":0} > {"ts":1513729195.106307,"interface":"netmap:eth5"," output_ring":null,"packets_received":62232503,"packets_ forwarded":55625459,"packets_dropped":1507385,"non_ip_ packets":413,"data_forward_rate_Mbps":1383.4110,"data_ drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":143. 3050,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":1024} > {"ts":1513729195.106307,"interface":"netmap:eth5"," output_ring":null,"packets_received":62232503,"packets_ forwarded":55625459,"packets_dropped":1507385,"non_ip_ packets":413,"data_forward_rate_Mbps":1383.4110,"data_ drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":143. 3050,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":1024} > {"ts":1513729195.106307,"interface":"netmap:eth5"," output_ring":null,"packets_received":62232503,"packets_ forwarded":55625459,"packets_dropped":1507385,"non_ip_ packets":413,"data_forward_rate_Mbps":1383.4110,"data_ drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":143. 3050,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":1024} > {"ts":1513729195.106307,"interface":"netmap:eth5"," output_ring":null,"packets_received":62232503,"packets_ forwarded":55625459,"packets_dropped":1507385,"non_ip_ packets":413,"data_forward_rate_Mbps":1383.4110,"data_ drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":143. 3050,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":1024} > {"ts":1513729195.106307,"interface":"netmap:eth5"," output_ring":null,"packets_received":62232503,"packets_ forwarded":55625459,"packets_dropped":1507385,"non_ip_ packets":413,"data_forward_rate_Mbps":1383.4110,"data_ drop_rate_Mbps":0.0000,"packet_forward_rate_kpps":143. 3050,"packet_drop_rate_kpps":0.0000,"free_buffer_slots":1024} The individual sub interfaces created by lb no longer report, but every 10 seconds (as configured) 9 identical lines (8 from -p 8 , and one from eth5) print out with the numbers no longer changing. Interface counters on eth5 continue to move so the interface is still seeing data. As well the time it takes for this to happen seems to vary. I have reloaded the modules, run lb as a user or root, changed the configuration parameters, rebooted the system all to no avail. Any thoughts? scott _______________________________________________ bro-dev mailing list bro-dev at bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20171220/656dd76d/attachment-0001.html From jan.grashoefer at gmail.com Wed Dec 20 02:07:13 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Wed, 20 Dec 2017 11:07:13 +0100 Subject: [Bro-Dev] Closures in Bro In-Reply-To: References: <90195bf8-caa8-e151-dcdd-1ebccd2bb503@gmail.com> Message-ID: On 19/12/17 18:31, Jon Siwek wrote:> I think it's mostly unsupported due to trickiness of implementation > (or cowardice) rather than by design decision. Or at least something > scared me off when I looked into it a few years ago [1]. Don't > remember the details, though I likely didn't give it more than a day > or two of thought/effort, so maybe don't let that deter you if you had > plans to look into it. Indeed, it seems tricky to get that working! I am not sure I will be able to find the time to look deeper into this. At a first glance, I would try to wrap function IDs into a new Val type that is able to carry the context. The particular context would be determined, when evaluating the expression, which returns the anonymous function. In the end, when the function is called, the context values could be treated as additional arguments. Jan From jeffrey.bencteux at ssi.gouv.fr Fri Dec 29 00:19:51 2017 From: jeffrey.bencteux at ssi.gouv.fr (Bencteux Jeffrey) Date: Fri, 29 Dec 2017 09:19:51 +0100 Subject: [Bro-Dev] SMB transaction messages pull request Message-ID: Hi all, I made a pull request a while ago to add/update messages for the SMB analyzer and I did not get no feedback. Is there something wrong with it? I'd be happy to modify it to fit your requirements if necessary. You can find it here : https://github.com/bro/bro/pull/119. Regards,