From johanna at icir.org Wed May 2 07:59:08 2018 From: johanna at icir.org (Johanna Amann) Date: Wed, 02 May 2018 07:59:08 -0700 Subject: [Bro-Dev] Final Broker branch testing In-Reply-To: <20180427155551.GI24938@icir.org> References: <4A4BEC38-8163-454C-80C7-0192B59144F7@icir.org> <20180427155551.GI24938@icir.org> Message-ID: On 27 Apr 2018, at 8:55, Robin Sommer wrote: > On Thu, Apr 26, 2018 at 16:54 -0500, you wrote: > >> (1) Users whose OS has insufficient CMake will need to compile/obtain >> a newer one. > >> (2) We go back to CMake 2.8.12 and have people compile CAF >> themselves. >> (Or maybe we could conditionally require only 2.8.12 users to compile >> CAF and others get the embedded CAF). > >> (3) I need to try to hack our CMake system more to try to get back >> down >> to 2.8.12 while still being able to embed CAF. > > If there's something quick that ends up making (3) work, that'd be > ideal of course, but I don't think it's worth spending much time on, > given that there are reasonable ways to get a more recent cmake. > > I wouldn't want to go back to not shipping CAF at all, but if we can > tell cmake that 2.8.12 is fine if users build CAF themselves, that > would be the 2nd best option I think. (1) ist worst case, which still > isn't too bad IMO, unless it does actually prevent us from building > binary packages for RH, that would be a problem. > Yup, 2 would be ok I guess. One should still be able to just compile the CAF in the Bro subdirectory in that case, right? 1 I would rather avoid if possible. Johanna From dane.wullen at alumni.fh-aachen.de Wed May 2 13:22:22 2018 From: dane.wullen at alumni.fh-aachen.de (Dane Wullen) Date: Wed, 2 May 2018 22:22:22 +0200 Subject: [Bro-Dev] Writing analyzer for Siemens PLC Message-ID: <3b0ef48d-a7fe-ac50-08cd-f8f942140518@alumni.fh-aachen.de> Hi there, as a part of my master's thesis I'm going to write two analyzer which will be able detect S7Comm and S7CommPlus traffic used by Siemens PLCs. Both protocols are proprietary, so I have no official documentation. The goal is to monitor this kind of traffic and/or detect threats against those PLCs. So far, I am able to read S7Comm traffic and generate various events, covering all standard and most of the additional (UserData) functions. This part is based on the code of Wireshark S7Comm dissector. The S7CommPlus part is a little bit trickier. Unfortunately, the only code base I can use is an 4 year old git-repo, probably forked from the first S7CommPlus dissector written by Thomas Wiens (https://github.com/moki-ics/s7commwireshark/blob/master/src/s7comm_plus/packet-s7comm_plus.c). After a quick glance I have discovered 2 "problems", where I hope you could help me. 1) Reassembling packets: Some S7CommPlus packets which payload is over a certain amount of bytes will be split and need to be reassembled. I don't know how to do that in Bro nor how I can keep track of which fragments belongs to the which packet. (LoC: 3917 to 4135) 2) Various payload: Some packets contain different payload, e.g. different addressing types, attributes and/or structures, even though the packet-type is the same (for example: a request packet can contain one or more objects. In each object, there can be various items of type A, B, C or D, each of them have a different structure). If I want to generate a Bro events which contains the payload as a parameter, how do I do that? Any help will be appreciated! Thanks. Dane From jsiwek at corelight.com Thu May 3 14:20:15 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 3 May 2018 16:20:15 -0500 Subject: [Bro-Dev] Final Broker branch testing In-Reply-To: References: <4A4BEC38-8163-454C-80C7-0192B59144F7@icir.org> <20180427155551.GI24938@icir.org> Message-ID: <1d030866-ae72-ddfe-e228-caeb5827f683@corelight.com> On 5/2/18 9:59 AM, Johanna Amann wrote: >>> (3) I need to try to hack our CMake system more to try to get back down >>> to 2.8.12 while still being able to embed CAF. I think (hope!) I was mistaken and everything already works with 2.8.12 (structure of CMake docs previously led me to think it wouldn't) and just needed the version check moved back down, sorry for the noise. Otherwise, I've stabilized some unit tests and made a merge request [1] for the broker branch. - Jon [1] https://bro-tracker.atlassian.net/browse/BIT-1653 From johanna at icir.org Thu May 3 17:14:15 2018 From: johanna at icir.org (Johanna Amann) Date: Thu, 03 May 2018 17:14:15 -0700 Subject: [Bro-Dev] Final Broker branch testing In-Reply-To: <1d030866-ae72-ddfe-e228-caeb5827f683@corelight.com> References: <4A4BEC38-8163-454C-80C7-0192B59144F7@icir.org> <20180427155551.GI24938@icir.org> <1d030866-ae72-ddfe-e228-caeb5827f683@corelight.com> Message-ID: <35F0E926-8E4F-4316-852A-7063952199CA@icir.org> On 3 May 2018, at 14:20, Jon Siwek wrote: > On 5/2/18 9:59 AM, Johanna Amann wrote: > >>>> (3) I need to try to hack our CMake system more to try to get back >>>> down >>>> to 2.8.12 while still being able to embed CAF. > > I think (hope!) I was mistaken and everything already works with > 2.8.12 (structure of CMake docs previously led me to think it > wouldn't) and just needed the version check moved back down, sorry for > the noise. > Yay, that is really good news, thanks :) Johanna From robin at icir.org Thu May 3 18:16:28 2018 From: robin at icir.org (Robin Sommer) Date: Thu, 3 May 2018 18:16:28 -0700 Subject: [Bro-Dev] Writing analyzer for Siemens PLC In-Reply-To: <3b0ef48d-a7fe-ac50-08cd-f8f942140518@alumni.fh-aachen.de> References: <3b0ef48d-a7fe-ac50-08cd-f8f942140518@alumni.fh-aachen.de> Message-ID: <20180504011628.GA22216@icir.org> On Wed, May 02, 2018 at 22:22 +0200, you wrote: > 1) Reassembling packets: Some S7CommPlus packets which payload is over a > certain amount of bytes will be split and need to be reassembled. As a couple quick pointers, the DNP3 and DTLS analyzers face a similar task, you might find some ideas there. > If I want to generate a Bro events which contains the payload as a > parameter, how do I do that? If with "payload" you mean the raw bytes, you would pass that as a string into the event. But it's hard to do much with raw data that in script-land. The common way would be instead creating one event per type of payload and then raising the corresponding event as you parse packets and find out what's in there. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From dane.wullen at alumni.fh-aachen.de Fri May 4 02:33:47 2018 From: dane.wullen at alumni.fh-aachen.de (Dane Wullen) Date: Fri, 4 May 2018 11:33:47 +0200 Subject: [Bro-Dev] Writing analyzer for Siemens PLC In-Reply-To: <20180504011628.GA22216@icir.org> References: <3b0ef48d-a7fe-ac50-08cd-f8f942140518@alumni.fh-aachen.de> <20180504011628.GA22216@icir.org> Message-ID: Hey Robin, thanks for you answer. I will look through this files and see if I can use this kind of reassembling. > If with "payload" you mean the raw bytes, you would pass that as a > string into the event. But it's hard to do much with raw data that in > script-land. The common way would be instead creating one event per > type of payload and then raising the corresponding event as you parse > packets and find out what's in there. No, I don't just want to put the whole data as a string into the event. Well, seems like I have to define a lot of different events and/or bro types (I don't know how many data types there are in total). Thanks alot. Dane Am 04.05.2018 um 03:16 schrieb Robin Sommer: > > On Wed, May 02, 2018 at 22:22 +0200, you wrote: > >> 1) Reassembling packets: Some S7CommPlus packets which payload is over a >> certain amount of bytes will be split and need to be reassembled. > As a couple quick pointers, the DNP3 and DTLS analyzers face a similar > task, you might find some ideas there. > >> If I want to generate a Bro events which contains the payload as a >> parameter, how do I do that? > If with "payload" you mean the raw bytes, you would pass that as a > string into the event. But it's hard to do much with raw data that in > script-land. The common way would be instead creating one event per > type of payload and then raising the corresponding event as you parse > packets and find out what's in there. > > Robin > From seth at corelight.com Thu May 10 07:51:24 2018 From: seth at corelight.com (Seth Hall) Date: Thu, 10 May 2018 10:51:24 -0400 Subject: [Bro-Dev] Final Broker branch testing In-Reply-To: <1d030866-ae72-ddfe-e228-caeb5827f683@corelight.com> References: <4A4BEC38-8163-454C-80C7-0192B59144F7@icir.org> <20180427155551.GI24938@icir.org> <1d030866-ae72-ddfe-e228-caeb5827f683@corelight.com> Message-ID: <0996C9E4-1B97-43AB-92AF-6B7DC02BD0E2@corelight.com> On 3 May 2018, at 17:20, Jon Siwek wrote: > Otherwise, I've stabilized some unit tests and made a merge request > [1] > for the broker branch. I'm really late on this, but congratulations Jon! That's super exciting and it's been a long slog to getting to this point but now I can't wait for you to be able to turn your attention to other stuff soon. :) Thanks! .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From dopheide at es.net Thu May 10 13:53:32 2018 From: dopheide at es.net (Michael Dopheide) Date: Thu, 10 May 2018 15:53:32 -0500 Subject: [Bro-Dev] Broker data store use case and questions Message-ID: Maybe I'm jumping the gun a little bit, but I want to start wrapping my head around the upcoming changes. Let's start by stating my use case... I wanted to stop the repetitive reverse DNS queries caused by ssh/interesting-hostnames.bro by rebuilding known-hosts.bro to include the names, allowing a simple lookup*. I started re-writing the old one and Justin pointed me towards the 'new' version of known-hosts in the topic/actor-system branch. Looking at the new known-hosts.bro.. 1) My initial gut feeling was that all of the when() calls for insertion could get really expensive on a brand new cluster before the store is populated. 2) Correct me if I'm wrong, but it seems like the check for a host already being in known_hosts (now host_store) no longer exists. As a result, we try to re-insert the host, calling when(), every time we see an established connection with a local host. Which leads me to... 3) How do I retrieve values from the store to test for existence? 4) Assuming that requires another Broker call inside a when(), does it make sense to pull the data store into memory at bro_init() and do a Cluster::publish_hrw? Thanks, Dop * - Yes, on the edges this breaks DNS TTLs, but saves thousands of when() calls to lookup_addr() and our names don't change very frequently. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180510/448b3706/attachment.html From jsiwek at corelight.com Fri May 11 07:13:27 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Fri, 11 May 2018 09:13:27 -0500 Subject: [Bro-Dev] Broker data store use case and questions In-Reply-To: References: Message-ID: <8125ff1a-9131-38e2-44df-2adfbf72bc54@corelight.com> On 5/10/18 3:53 PM, Michael Dopheide wrote: > 1) My initial gut feeling was that all of the when() calls for insertion > could get really expensive on a brand new cluster before the store is > populated. I've not tried to explicitly measure differences yet, though my hunch is that the overhead of needing to use when() to drive data store communication patterns could be slightly more expensive than just using remote events (or &synchronized as the previous implementation used). I'm thinking of overhead more in terms of memory here, as it needs to save the state of the current frame making the call so it can resume later. Another difference is the data store implementation of known-hosts is that it does always require remote communication just to check for whether a given key is in the data store yet, which may be a bottleneck for some use-cases. You can also compare/contrast with another implementation of known-hosts.bro if you toggle the `Known::use_host_store = F` code path. There, instead of using a data store, it sends remote events via Cluster::publish_hrw to uniformly partition the data set across proxies in a scalable manner but without persistence. Yet another idea for an implementation, if you need persistence + scalability, would be combining the HRW stuff with data stores. e.g. partitioning the total data set across proxies while using a data store on each one for local storage instead of a table/set. I don't know if there's a general answer to which way is best. Likely varies per use-case / network. > 2) Correct me if I'm wrong, but it seems like the check for a host > already being in known_hosts (now host_store) no longer exists.? As a > result, we try to re-insert the host, calling when(), every time we see > an established connection with a local host. Sounds right. Specifically, it's Broker::put_unique() that hides the following: (1) tell master data store "insert this key if it does not exist" (2) wait for master data store to tell us if the key was inserted, and thus did not exist before There's no check against the local cache to first see if the key exists as going down that path leads to race conditions. > 3) How do I retrieve values from the store to test for existence? Broker::exists() to just check existence of a key or Broker::get() to retrieve value at a key. You can also infer existence from the result of Broker::get(). Either requires calling inside 'when()'. Generally, any function in the API you see return a Broker::QueryResult needs to use 'when()'. > 4) Assuming that requires another Broker call inside a when(), does it > make sense to pull the data store into memory at bro_init() and do > a?Cluster::publish_hrw? Not sure I follow since, in the current implementation of known-hosts, the data store and Cluster::publish_hrw code paths don't interact (they're alternate implementations of the same thing as mentioned before). If the question is just whether it makes sense to go the Cluster::publish_hrw route instead of using a data store: yes, just depends on what you prefer. IMO, the data store approach has downsides that make it less preferable to me. - Jon From dopheide at es.net Fri May 11 08:15:53 2018 From: dopheide at es.net (Michael Dopheide) Date: Fri, 11 May 2018 10:15:53 -0500 Subject: [Bro-Dev] Broker data store use case and questions In-Reply-To: <8125ff1a-9131-38e2-44df-2adfbf72bc54@corelight.com> References: <8125ff1a-9131-38e2-44df-2adfbf72bc54@corelight.com> Message-ID: Let me clarify point 4, my goal is just to keep the knownhosts data persistent across restarts. (Or any data set in the general case.) So if HRW is the best way to keep data in memory I need a way to write it out to disk on Bro exit so I can read it back in later. On Fri, May 11, 2018 at 9:13 AM Jon Siwek wrote: > > > On 5/10/18 3:53 PM, Michael Dopheide wrote: > > > 1) My initial gut feeling was that all of the when() calls for insertion > > could get really expensive on a brand new cluster before the store is > > populated. > > I've not tried to explicitly measure differences yet, though my hunch is > that the overhead of needing to use when() to drive data store > communication patterns could be slightly more expensive than just using > remote events (or &synchronized as the previous implementation used). > I'm thinking of overhead more in terms of memory here, as it needs to > save the state of the current frame making the call so it can resume > later. > > Another difference is the data store implementation of known-hosts is > that it does always require remote communication just to check for > whether a given key is in the data store yet, which may be a bottleneck > for some use-cases. > > You can also compare/contrast with another implementation of > known-hosts.bro if you toggle the `Known::use_host_store = F` code path. > There, instead of using a data store, it sends remote events via > Cluster::publish_hrw to uniformly partition the data set across proxies > in a scalable manner but without persistence. > > Yet another idea for an implementation, if you need persistence + > scalability, would be combining the HRW stuff with data stores. e.g. > partitioning the total data set across proxies while using a data store > on each one for local storage instead of a table/set. > > I don't know if there's a general answer to which way is best. Likely > varies per use-case / network. > > > 2) Correct me if I'm wrong, but it seems like the check for a host > > already being in known_hosts (now host_store) no longer exists. As a > > result, we try to re-insert the host, calling when(), every time we see > > an established connection with a local host. > > Sounds right. > > Specifically, it's Broker::put_unique() that hides the following: > > (1) tell master data store "insert this key if it does not exist" > (2) wait for master data store to tell us if the key was inserted, and > thus did not exist before > > There's no check against the local cache to first see if the key exists > as going down that path leads to race conditions. > > > 3) How do I retrieve values from the store to test for existence? > > Broker::exists() to just check existence of a key or Broker::get() to > retrieve value at a key. You can also infer existence from the result > of Broker::get(). > > Either requires calling inside 'when()'. Generally, any function in the > API you see return a Broker::QueryResult needs to use 'when()'. > > > 4) Assuming that requires another Broker call inside a when(), does it > > make sense to pull the data store into memory at bro_init() and do > > a Cluster::publish_hrw? > > Not sure I follow since, in the current implementation of known-hosts, > the data store and Cluster::publish_hrw code paths don't interact > (they're alternate implementations of the same thing as mentioned > before). If the question is just whether it makes sense to go the > Cluster::publish_hrw route instead of using a data store: yes, just > depends on what you prefer. IMO, the data store approach has downsides > that make it less preferable to me. > > - Jon > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180511/14031c03/attachment.html From jsiwek at corelight.com Fri May 11 10:34:05 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Fri, 11 May 2018 12:34:05 -0500 Subject: [Bro-Dev] Broker data store use case and questions In-Reply-To: References: <8125ff1a-9131-38e2-44df-2adfbf72bc54@corelight.com> Message-ID: On 5/11/18 10:15 AM, Michael Dopheide wrote: > Let me clarify point 4, my goal is just to keep the knownhosts data > persistent across restarts. ?(Or any data set in the general case.) ?So > if HRW is the best way to keep data in memory I need a way to write it > out to disk on Bro exit so I can read it back in later. Ok, maybe first try a single data store for persistence (what known-hosts will do by default now). If you then find you overload the centralized manager/master node, my next suggestion is to partition the data set across proxies via HRW events and combine that with a data store configured for persistence on each proxy. - Jon From jazoff at illinois.edu Fri May 11 11:38:32 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 11 May 2018 18:38:32 +0000 Subject: [Bro-Dev] Broker data store use case and questions In-Reply-To: <8125ff1a-9131-38e2-44df-2adfbf72bc54@corelight.com> References: <8125ff1a-9131-38e2-44df-2adfbf72bc54@corelight.com> Message-ID: <2942C8C9-D2B0-442E-8733-15F5FFE5E622@illinois.edu> > On May 11, 2018, at 10:13 AM, Jon Siwek wrote: > > > There's no check against the local cache to first see if the key exists > as going down that path leads to race conditions. What sort of race conditions? Right now I see a lot of events going around so it seems like there may be a bit of overhead in this area. For example, in about a minute one node in a two node cluster sent 2104 Software::register events (so likely 4k total) In that time, only 7 new entries were logged to software.log. It's always good to reduce memory usage, but I think especially for things like known hosts which are generally kept at LOCAL_HOSTS the small amount of memory used for caching already seen hosts saves more resources than would be spent sending redundant events around. Especially if those events end up queued and buffered in ram anyway. Things are a bit better off now in that we can use a short lived cache, since the cache doesn't need to be the actual data store anymore like the old known hosts set was. ? Justin Azoff From dopheide at es.net Fri May 11 16:33:09 2018 From: dopheide at es.net (Michael Dopheide) Date: Fri, 11 May 2018 18:33:09 -0500 Subject: [Bro-Dev] Broker data store use case and questions In-Reply-To: <2942C8C9-D2B0-442E-8733-15F5FFE5E622@illinois.edu> References: <8125ff1a-9131-38e2-44df-2adfbf72bc54@corelight.com> <2942C8C9-D2B0-442E-8733-15F5FFE5E622@illinois.edu> Message-ID: Couple questions. (Let me know if this isn't appropriate bro-dev content, but I don't want to cause confusion on the normal bro list.) First, can Cluster::default_master_node be changed to default to the name of the current manager node rather than specifying the name as 'manager'? Easy to redef to the manager's name, but less easy when you use the same code base on multiple clusters with different names. ==== stderr.log fatal error in /usr/local/bro/share/bro/base/frameworks/cluster/./main.bro, lines 399-400: master node 'manager' for cluster store 'bro/known/certs' does not exist Second, when during startup should Bro know that it's persistent stores exist via Cluster::stores() ? It appears bro_init may be too soon, but I'm still playing. Also, it'd be nice if the persistence of built-in stores (like known/hosts, known/certs, etc) were redef-able. Thanks, -Dop On Fri, May 11, 2018 at 1:38 PM, Azoff, Justin S wrote: > > > On May 11, 2018, at 10:13 AM, Jon Siwek wrote: > > > > > > There's no check against the local cache to first see if the key exists > > as going down that path leads to race conditions. > > What sort of race conditions? > > Right now I see a lot of events going around so it seems like there may be > a bit of overhead in this area. > > For example, in about a minute one node in a two node cluster sent 2104 > Software::register events (so likely 4k total) > In that time, only 7 new entries were logged to software.log. > > It's always good to reduce memory usage, but I think especially for things > like known hosts which are generally kept at LOCAL_HOSTS > the small amount of memory used for caching already seen hosts saves more > resources than would be spent sending redundant events around. > Especially if those events end up queued and buffered in ram anyway. > > Things are a bit better off now in that we can use a short lived cache, > since the cache doesn't need to be the actual data store anymore like the > old known hosts set was. > > > > ? > Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180511/700d6767/attachment.html From jsiwek at corelight.com Mon May 14 07:12:43 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Mon, 14 May 2018 09:12:43 -0500 Subject: [Bro-Dev] Broker data store use case and questions In-Reply-To: <2942C8C9-D2B0-442E-8733-15F5FFE5E622@illinois.edu> References: <8125ff1a-9131-38e2-44df-2adfbf72bc54@corelight.com> <2942C8C9-D2B0-442E-8733-15F5FFE5E622@illinois.edu> Message-ID: <89dcabf4-56fd-78cc-ca67-d9fef6a4a7c3@corelight.com> On 5/11/18 1:38 PM, Azoff, Justin S wrote: > >> On May 11, 2018, at 10:13 AM, Jon Siwek wrote: >> >> >> There's no check against the local cache to first see if the key exists >> as going down that path leads to race conditions. > > What sort of race conditions? By "local cache", I mean the data store "clone" here. And one race with checking for existence in the local clone could look like: (1) master: delete an expired key, "foo", send notification to clones (2) clone: check for existence of key, "foo" and find it exists locally, then suppress further logic based on that (3) clone: receive expiry notification for key "foo" In that way, you can miss an (re)insertion that should have taken place if the query/insertion were together in sequence directly on the master data set. > Things are a bit better off now in that we can use a short lived cache, since the cache doesn't need to be the actual data store anymore like the old known hosts set was. A short-lived cache, separate from the data store, still has problems like the above: there can be times where the local cache contains the key and the master store does not and so you may miss some (re)insertions. The main goal I had when re-writing these was correctness: I can't know what network they will run on, and so don't want to assume it will be ok to miss an event here or there because "typically those should be seen frequently enough that it will get picked up soon after the miss". If we can optimize the scripts that ship w/ Bro while still maintaining correctness, that would be great, else I'd rather sites decide for themselves what trade-offs are acceptable and write their own scripts to optimize for those. - Jon From jsiwek at corelight.com Mon May 14 07:43:15 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Mon, 14 May 2018 09:43:15 -0500 Subject: [Bro-Dev] Broker data store use case and questions In-Reply-To: References: <8125ff1a-9131-38e2-44df-2adfbf72bc54@corelight.com> <2942C8C9-D2B0-442E-8733-15F5FFE5E622@illinois.edu> Message-ID: <77110edc-fe36-a345-f33c-c8456a59ee7d@corelight.com> On 5/11/18 6:33 PM, Michael Dopheide wrote: > First, can Cluster::default_master_node be changed to default to the > name of the current manager node rather than specifying the name as > 'manager'? Maybe. I'll try having broctl communicate that to Bro via a new environment variable. > Easy to redef to the manager's name, but less easy when you > use the same code base on multiple clusters with different names. If you don't want to wait for me to try the above fix, you could also try redef'ing it yourself with a call to getenv(), using an environment variable whose value you can set differently for each cluster. > Second, when during startup should Bro know that it's persistent stores > exist via Cluster::stores() ?? It appears bro_init may be too soon, but > I'm still playing. The comments for the Cluster::stores table may help in case you missed it -- Cluster::create_store() is intended to be called in bro_init() and will end up populating Cluster::stores. Though, you can pre-populate and customize the Cluster::stores table via a redef and those will all automatically get picked up when during the Cluster::create_store() process. > Also, it'd be nice if the persistence of built-in > stores (like known/hosts, known/certs, etc) were redef-able. It should be possible like putting this in local.bro: redef Cluster::stores += { [Known::host_store_name] = Cluster::StoreInfo($backend = Broker::SQLITE) }; - Jon From jsiwek at corelight.com Mon May 14 08:36:38 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Mon, 14 May 2018 10:36:38 -0500 Subject: [Bro-Dev] Broker data store use case and questions In-Reply-To: <77110edc-fe36-a345-f33c-c8456a59ee7d@corelight.com> References: <8125ff1a-9131-38e2-44df-2adfbf72bc54@corelight.com> <2942C8C9-D2B0-442E-8733-15F5FFE5E622@illinois.edu> <77110edc-fe36-a345-f33c-c8456a59ee7d@corelight.com> Message-ID: <11620c43-a7b7-8042-66de-f320d9cea893@corelight.com> On 5/14/18 9:43 AM, Jon Siwek wrote: >> First, can Cluster::default_master_node be changed to default to the >> name of the current manager node rather than specifying the name as >> 'manager'? > > Maybe.? I'll try having broctl communicate that to Bro via a new > environment variable. Nevermind, I did it differently, but the topic/actor-system branch now automatically tries to determine the name of the manager node. - Jon From jazoff at illinois.edu Mon May 14 11:34:36 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 14 May 2018 18:34:36 +0000 Subject: [Bro-Dev] Broker data store use case and questions In-Reply-To: <89dcabf4-56fd-78cc-ca67-d9fef6a4a7c3@corelight.com> References: <8125ff1a-9131-38e2-44df-2adfbf72bc54@corelight.com> <2942C8C9-D2B0-442E-8733-15F5FFE5E622@illinois.edu> <89dcabf4-56fd-78cc-ca67-d9fef6a4a7c3@corelight.com> Message-ID: > On May 14, 2018, at 10:12 AM, Jon Siwek wrote: > > A short-lived cache, separate from the data store, still has problems like the above: there can be times where the local cache contains the key and the master store does not and so you may miss some (re)insertions. I see what you mean.. I can almost see a solution involving using create_expire and expire_func to trigger a re-submit when the local cache expires, but that may cause the opposite problem. This would mean that a record would be sent the first time it was seen and then at most once again N minutes after that. If N minutes after that is 00:03 the entry would be logged on the following day even if it was not seen yet. I suppose if the value in the cache table was the network_time of the last time seen that could used to fill in the HostInfo record. ? Justin Azoff From robin at icir.org Tue May 15 17:19:30 2018 From: robin at icir.org (Robin Sommer) Date: Tue, 15 May 2018 17:19:30 -0700 Subject: [Bro-Dev] Moving to GitHub? Message-ID: <20180516001930.GH43145@icir.org> This has been coming up in various contexts & subgroups of people, and I wanted to send it out as a proposal to gather some broader feedback: Do we want to move Bro's git repositories and tickets to GitHub? Currently we're hosting our Git repositories on our own server at git.bro.org; from there, we mirror them to GitHub. For issue tracking, we use JIRA at tracker.bro.org. Much of all this is a legacy setup in some sense. I believe it would simplify life for both users & developers if we moved to GitHub as the authoratative place for both code and tickets. More specifically, I propose that we: 1. Turn the current git repository mirrors on github.com/bro into authoritative source for all Bro code. Then we discontinue git.bro.org. We can set up up some notification system there instead that points people still using the old URLs to GitHub. 2. Switch to using GitHub as our primary issue tracker. Giving the large amount of old tickets in JIRA, I think we should just start from scratch there, porting over just the most recent pending tickets. We'd keep the JIRA running in read-only mode so that we don't loose the history and can always refer back to old tickets. 3. Switch to a mostly PR-based development model (i.e., no more merge requests tracked separately through tickets), and also use GitHub for code review & feedback. 4. Make sure it all integrates nicely with Travis CI (which has already been set up recently). About the only downside I see is that emailing out our standard commit notifications won't be quite as smooth anymore: we'd still get them, but with a bit of delay as some system would need to be polling for changes, rather than being triggered immediately. Not much of a problem I think (and with some additional work, we could make them push-triggered, too; but probably not worth it). What do people think? Any support, or concerns? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From johanna at icir.org Tue May 15 17:34:37 2018 From: johanna at icir.org (Johanna Amann) Date: Tue, 15 May 2018 17:34:37 -0700 Subject: [Bro-Dev] Moving to GitHub? In-Reply-To: <20180516001930.GH43145@icir.org> References: <20180516001930.GH43145@icir.org> Message-ID: > What do people think? Any support, or concerns? I am in favor. The only thing I would miss are the immediate change notifications by email - I really like those... Johanna From slagell at illinois.edu Tue May 15 17:43:14 2018 From: slagell at illinois.edu (Slagell, Adam J) Date: Wed, 16 May 2018 00:43:14 +0000 Subject: [Bro-Dev] Moving to GitHub? In-Reply-To: <20180516001930.GH43145@icir.org> References: <20180516001930.GH43145@icir.org> Message-ID: <6C60BF6B-763B-499A-93B4-E9A91FD105BB@illinois.edu> I am in favor. I would like to still maintain the the Jira and wiki for a couple months until we finish up some work. Really just the BPM tickets. > On May 15, 2018, at 7:19 PM, Robin Sommer wrote: > > This has been coming up in various contexts & subgroups of people, and > I wanted to send it out as a proposal to gather some broader feedback: > Do we want to move Bro's git repositories and tickets to GitHub? > > Currently we're hosting our Git repositories on our own server at > git.bro.org; from there, we mirror them to GitHub. For issue tracking, > we use JIRA at tracker.bro.org. Much of all this is a legacy setup in > some sense. I believe it would simplify life for both users & > developers if we moved to GitHub as the authoratative place for both > code and tickets. > > More specifically, I propose that we: > > 1. Turn the current git repository mirrors on github.com/bro into > authoritative source for all Bro code. Then we discontinue > git.bro.org. We can set up up some notification system there > instead that points people still using the old URLs to GitHub. > > 2. Switch to using GitHub as our primary issue tracker. Giving > the large amount of old tickets in JIRA, I think we should > just start from scratch there, porting over just the most > recent pending tickets. We'd keep the JIRA running in > read-only mode so that we don't loose the history and can > always refer back to old tickets. > > 3. Switch to a mostly PR-based development model (i.e., no more > merge requests tracked separately through tickets), and also > use GitHub for code review & feedback. > > 4. Make sure it all integrates nicely with Travis CI (which has > already been set up recently). > > About the only downside I see is that emailing out our standard commit > notifications won't be quite as smooth anymore: we'd still get them, > but with a bit of delay as some system would need to be polling for > changes, rather than being triggered immediately. Not much of a > problem I think (and with some additional work, we could make them > push-triggered, too; but probably not worth it). > > What do people think? Any support, or concerns? > > Robin > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From michalpurzynski1 at gmail.com Tue May 15 19:40:57 2018 From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=) Date: Tue, 15 May 2018 19:40:57 -0700 Subject: [Bro-Dev] Moving to GitHub? In-Reply-To: References: <20180516001930.GH43145@icir.org> Message-ID: <83D7EC9C-4625-4443-936C-4EA5B4705560@gmail.com> Big thumbs up. On May 15, 2018, at 5:34 PM, Johanna Amann wrote: >> What do people think? Any support, or concerns? > > I am in favor. The only thing I would miss are the immediate change > notifications by email - I really like those... > > Johanna > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From hosom at battelle.org Wed May 16 08:23:57 2018 From: hosom at battelle.org (Hosom, Stephen M) Date: Wed, 16 May 2018 15:23:57 +0000 Subject: [Bro-Dev] Moving to GitHub? In-Reply-To: References: <20180516001930.GH43145@icir.org> Message-ID: <0EBEAC7E-A4EE-49DE-B8ED-546DBA030E05@battelle.org> I too would miss the commit / change notifications, however, I think that this can be set up in GitHub in some way. We use Slack / Email internally for commit notifications. For Slack, we use webhooks to send notifications. Email commit notifications are actually a builtin integration in GitHub... You can set them up under Settings -> Integrations & Services -> Add service -> Email. ?On 5/15/18, 8:41 PM, "bro-dev-bounces at bro.org on behalf of Johanna Amann" wrote: Message received from outside the Battelle network. Carefully examine it before you open any links or attachments. > What do people think? Any support, or concerns? I am in favor. The only thing I would miss are the immediate change notifications by email - I really like those... Johanna _______________________________________________ bro-dev mailing list bro-dev at bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From hosom at battelle.org Wed May 16 08:31:22 2018 From: hosom at battelle.org (Hosom, Stephen M) Date: Wed, 16 May 2018 15:31:22 +0000 Subject: [Bro-Dev] Moving to GitHub? In-Reply-To: <0EBEAC7E-A4EE-49DE-B8ED-546DBA030E05@battelle.org> References: <20180516001930.GH43145@icir.org> <0EBEAC7E-A4EE-49DE-B8ED-546DBA030E05@battelle.org> Message-ID: <35444E70-07ED-48C9-8127-2D3AAEDA4994@battelle.org> Well... in looking at how I set this up internally, I now realize that this feature is being EOL and will no longer work as of January 31, 2019. I guess it would have to be webhooks. ?On 5/16/18, 11:30 AM, "bro-dev-bounces at bro.org on behalf of Hosom, Stephen M" wrote: I too would miss the commit / change notifications, however, I think that this can be set up in GitHub in some way. We use Slack / Email internally for commit notifications. For Slack, we use webhooks to send notifications. Email commit notifications are actually a builtin integration in GitHub... You can set them up under Settings -> Integrations & Services -> Add service -> Email. On 5/15/18, 8:41 PM, "bro-dev-bounces at bro.org on behalf of Johanna Amann" wrote: Message received from outside the Battelle network. Carefully examine it before you open any links or attachments. > What do people think? Any support, or concerns? I am in favor. The only thing I would miss are the immediate change notifications by email - I really like those... Johanna _______________________________________________ bro-dev mailing list bro-dev at bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev _______________________________________________ bro-dev mailing list bro-dev at bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From robin at icir.org Wed May 16 13:02:19 2018 From: robin at icir.org (Robin Sommer) Date: Wed, 16 May 2018 13:02:19 -0700 Subject: [Bro-Dev] Moving to GitHub? In-Reply-To: <0EBEAC7E-A4EE-49DE-B8ED-546DBA030E05@battelle.org> References: <20180516001930.GH43145@icir.org> <0EBEAC7E-A4EE-49DE-B8ED-546DBA030E05@battelle.org> Message-ID: <20180516200219.GC48952@icir.org> On Wed, May 16, 2018 at 15:23 +0000, you wrote: > I too would miss the commit / change notifications, however, I think > that this can be set up in GitHub in some way. We can still get the same email notifications as today (which have a bit more information that GitHub's standard ones), they will just come with a little bit of a delay (within 10-15 minutes should be reasonable). And if we want we can trigger that through webbhooks, too, for immediate notification, would just need a bit of work to get it set up. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at corelight.com Thu May 17 09:21:43 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 17 May 2018 11:21:43 -0500 Subject: [Bro-Dev] Moving to GitHub? In-Reply-To: <20180516001930.GH43145@icir.org> References: <20180516001930.GH43145@icir.org> Message-ID: <48511fe5-d534-e682-a886-7e3b4ad586e4@corelight.com> On 5/15/18 7:19 PM, Robin Sommer wrote: > What do people think? Any support, or concerns? Yeah, generally in favor with some comments: * For porting over JIRA tickets to GitHub, "most recent" doesn't seem like a good metric to use. e.g. BIT-1829 (pcap triggering assertion in binpac) seems kind of important and not something I'd want to have lost in the move, although it had no activity in almost a year. So, I think it's worth it to be more comprehensive here, and as long as someone is going through to review all the tickets, they may as well just port all the older ones that are still valid over to GitHub. (Yeah, I guess I'm volunteering). * One thing I did like about using JIRA for merge requests is that I could make a single ticket and just say I have a given branch name in a bunch of repos that are ready for review/merge. I find myself in that situation quite often, actually, so transitioning to GitHub PRs, I wonder if we'd want a PR to be created against each individual repo? Seems a bit much in terms of overhead. Alternatively, could still create a GH issue and just say "please review branch foo in repos X, Y, Z and merge them". Or else create a single PR in the "root" repo and mention in the PR that the same branch in child submodules also exists and needs merging. That may play better with Travis CI integration even, although maybe not in the case where you need to change things in the "external testing" repos, which are not connected to bro via a submodule. - Jon From seth at corelight.com Thu May 17 12:22:08 2018 From: seth at corelight.com (Seth Hall) Date: Thu, 17 May 2018 15:22:08 -0400 Subject: [Bro-Dev] Moving to GitHub? In-Reply-To: <20180516001930.GH43145@icir.org> References: <20180516001930.GH43145@icir.org> Message-ID: <34F2E8D5-6956-41C5-AF92-0DA87F90E52A@corelight.com> On 15 May 2018, at 20:19, Robin Sommer wrote: > This has been coming up in various contexts & subgroups of people, and > I wanted to send it out as a proposal to gather some broader feedback: > Do we want to move Bro's git repositories and tickets to GitHub? I like the idea. It'll be nice to have one process for handling everything instead of the multiple avenues for tickets and merge requests that we have today. It seems that we should even be able to solve the issue with the immediate diff emails by using webhooks with an AWS lambda function. I'm not sure if we'd be able to do all of the git work from that though. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From robin at icir.org Thu May 17 16:27:55 2018 From: robin at icir.org (Robin Sommer) Date: Thu, 17 May 2018 16:27:55 -0700 Subject: [Bro-Dev] Moving to GitHub? In-Reply-To: <48511fe5-d534-e682-a886-7e3b4ad586e4@corelight.com> References: <20180516001930.GH43145@icir.org> <48511fe5-d534-e682-a886-7e3b4ad586e4@corelight.com> Message-ID: <20180517232755.GC6659@icir.org> On Thu, May 17, 2018 at 11:21 -0500, you wrote: > * For porting over JIRA tickets to GitHub, "most recent" doesn't seem like a > good metric to use. Agree. :) > they may as well just port all the older ones that are still valid > over to GitHub. That may be a bit too broad though. How about "still valid and either (1) quite important or (2) something we expect will be addresses reasonably soon"? We have many old tickets that are technically still valid but unlikely to see any work anytime soon (otherwise they would have been addressed already), and I'm worried that they would just add noise without value. The old tickets won't go away, the JIRA will remain. If something becomes relevant/active, we can always bring it over at that time. > I find myself in that situation quite often, actually, so > transitioning to GitHub PRs, I wonder if we'd want a PR to be created > against each individual repo? Good point. Creating just one root PR that mentions the others sounds good to me for such cases. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at corelight.com Fri May 18 06:27:13 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Fri, 18 May 2018 08:27:13 -0500 Subject: [Bro-Dev] Moving to GitHub? In-Reply-To: <20180517232755.GC6659@icir.org> References: <20180516001930.GH43145@icir.org> <48511fe5-d534-e682-a886-7e3b4ad586e4@corelight.com> <20180517232755.GC6659@icir.org> Message-ID: <04ee0627-3f66-8e77-6ab9-610de32def68@corelight.com> On 5/17/18 6:27 PM, Robin Sommer wrote: > That may be a bit too broad though. How about "still valid and either > (1) quite important or (2) something we expect will be addresses > reasonably soon"? We have many old tickets that are technically still > valid but unlikely to see any work anytime soon (otherwise they would > have been addressed already), and I'm worried that they would just add > noise without value. That sounds like a general concern about project/ticket organization and management. What if valid-but-old tickets were moved into GitHub with a simple "backlog" tag that you can filter out? Or we could take the opportunity to be better about sorting on other categories as well (bug/feature/etc). There's also the possibility to start a project board page [1] to aid in visual organization of tasks/issues and further reduce perceived noise. See [2] for an example of what one of those pages could look like. > The old tickets won't go away, the JIRA will > remain. If something becomes relevant/active, we can always bring it > over at that time. Keeping the JIRA around as a backlog like that decreases focus/visibility -- if we want to be optimistic about community contributions and bug fixes, we'd have to keep calling attention to search in two locations, GH and JIRA, instead of just one. We'd also need to stay on top of maintaining JIRA to be relatively in sync with anything that gets resolved in GH else keeping JIRA around may become more liability than useful as it gets harder and harder to tell if something there has actually been addressed. Doing a half-hearted effort to migrate tickets from JIRA undermines the goal of having an authoritative/central location for all code + tickets. Can we instead try to deal with it once and for all? - Jon [1] https://github.com/orgs/bro/projects [2] https://github.com/git-lfs/git-lfs/projects/3 From robin at icir.org Fri May 18 09:11:29 2018 From: robin at icir.org (Robin Sommer) Date: Fri, 18 May 2018 09:11:29 -0700 Subject: [Bro-Dev] Moving to GitHub? In-Reply-To: <04ee0627-3f66-8e77-6ab9-610de32def68@corelight.com> References: <20180516001930.GH43145@icir.org> <48511fe5-d534-e682-a886-7e3b4ad586e4@corelight.com> <20180517232755.GC6659@icir.org> <04ee0627-3f66-8e77-6ab9-610de32def68@corelight.com> Message-ID: <20180518161129.GC59866@icir.org> On Fri, May 18, 2018 at 08:27 -0500, you wrote: > Doing a half-hearted effort to migrate tickets from JIRA undermines the goal > of having an authoritative/central location for all code + tickets. Can we > instead try to deal with it once and for all? What I was envisioning is more or less a clean slate: we'd migrate over a few tickets, but essentially we'd start with an empty list. I realize that sounds pretty harsh. However, I hardly ever see any activity on older tickets in JIRA, and I generally believe that the less open tickets a tracker has, the easier it is for people to understand what's actually relevant and worth spending cycles on. Tagging tickets may help, but in the end if everybody just filters 95% out all the time anyways, I'm not sure what the value is. That said, I'm open to a real porting effort if people do believe it's helpful to get all the JIRA tickets into GitHub. What do others think? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From slagell at illinois.edu Fri May 18 09:17:33 2018 From: slagell at illinois.edu (Slagell, Adam J) Date: Fri, 18 May 2018 16:17:33 +0000 Subject: [Bro-Dev] Moving to GitHub? In-Reply-To: <20180518161129.GC59866@icir.org> References: <20180516001930.GH43145@icir.org> <48511fe5-d534-e682-a886-7e3b4ad586e4@corelight.com> <20180517232755.GC6659@icir.org> <04ee0627-3f66-8e77-6ab9-610de32def68@corelight.com> <20180518161129.GC59866@icir.org> Message-ID: <9120B70D-D811-40B3-86BA-006DD60D0118@illinois.edu> On May 18, 2018, at 11:11 AM, Robin Sommer > wrote: What I was envisioning is more or less a clean slate: we'd migrate over a few tickets, but essentially we'd start with an empty list. I realize that sounds pretty harsh. However, I hardly ever see any activity on older tickets in JIRA, and I generally believe that the less open tickets a tracker has, the easier it is for people to understand what's actually relevant and worth spending cycles on. Tagging tickets may help, but in the end if everybody just filters 95% out all the time anyways, I'm not sure what the value is. That said, I'm open to a real porting effort if people do believe it's helpful to get all the JIRA tickets into GitHub. What do others think? Start clean for BIT. Unless it is marked critical, I don?t think it needs to go over. If people have tickets of their own they want to move over, it should not be too hard to manually recreate a couple. And after BroCon I think we kill the Jira and wiki completely. We should be done with the project management and event tickets by then. ------ Adam J. Slagell Director, Cybersecurity & Networking Division Chief Information Security Officer National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180518/4843e11f/attachment-0001.html From johanna at icir.org Fri May 18 09:37:51 2018 From: johanna at icir.org (Johanna Amann) Date: Fri, 18 May 2018 09:37:51 -0700 Subject: [Bro-Dev] Moving to GitHub? In-Reply-To: <9120B70D-D811-40B3-86BA-006DD60D0118@illinois.edu> References: <20180516001930.GH43145@icir.org> <48511fe5-d534-e682-a886-7e3b4ad586e4@corelight.com> <20180517232755.GC6659@icir.org> <04ee0627-3f66-8e77-6ab9-610de32def68@corelight.com> <20180518161129.GC59866@icir.org> <9120B70D-D811-40B3-86BA-006DD60D0118@illinois.edu> Message-ID: <581A51E3-B69B-4304-8377-B4F5E6793993@icir.org> I am also more in favor of starting clean and manually letting people move tickets that they think are important over. But - currently there is a lot in the tracker that are nice to have or potential problems that I do not ever see getting addressed. Johanna On 18 May 2018, at 9:17, Slagell, Adam J wrote: > On May 18, 2018, at 11:11 AM, Robin Sommer > > wrote: > > What I was envisioning is more or less a clean slate: we'd migrate > over a few tickets, but essentially we'd start with an empty list. I > realize that sounds pretty harsh. However, I hardly ever see any > activity on older tickets in JIRA, and I generally believe that the > less open tickets a tracker has, the easier it is for people to > understand what's actually relevant and worth spending cycles on. > Tagging tickets may help, but in the end if everybody just filters 95% > out all the time anyways, I'm not sure what the value is. > > That said, I'm open to a real porting effort if people do believe it's > helpful to get all the JIRA tickets into GitHub. What do others think? > > Start clean for BIT. Unless it is marked critical, I don?t think it > needs to go over. If people have tickets of their own they want to > move over, it should not be too hard to manually recreate a couple. > > And after BroCon I think we kill the Jira and wiki completely. We > should be done with the project management and event tickets by then. > > ------ > > Adam J. Slagell > Director, Cybersecurity & Networking Division > Chief Information Security Officer > National Center for Supercomputing Applications > University of Illinois at Urbana-Champaign > www.slagell.info > > "Under the Illinois Freedom of Information Act (FOIA), any written > communication to or from University employees regarding University > business is a public record and may be subject to public disclosure." > > > > > > > > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From commike at reservoir.com Fri May 18 10:24:21 2018 From: commike at reservoir.com (Alan Commike) Date: Fri, 18 May 2018 10:24:21 -0700 Subject: [Bro-Dev] Moving to GitHub? In-Reply-To: <20180518161129.GC59866@icir.org> References: <20180516001930.GH43145@icir.org> <48511fe5-d534-e682-a886-7e3b4ad586e4@corelight.com> <20180517232755.GC6659@icir.org> <04ee0627-3f66-8e77-6ab9-610de32def68@corelight.com> <20180518161129.GC59866@icir.org> Message-ID: On Fri, May 18, 2018 at 9:12 AM Robin Sommer wrote: > > > That said, I'm open to a real porting effort if people do believe it's > helpful to get all the JIRA tickets into GitHub. What do others think? > Having the historical tickets available are useful for searching to see if someone else had a similar issue and if there's a possible work-around. How about a solution somewhere in the middle - push all the tickets over but mark the older/non-critical as 'won't fix'. They'll come up in searches and can more easily be brought back alive if needed. ...alan > > Robin > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180518/f4093933/attachment.html From jsiwek at corelight.com Fri May 18 10:52:02 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Fri, 18 May 2018 12:52:02 -0500 Subject: [Bro-Dev] Moving to GitHub? In-Reply-To: <20180518161129.GC59866@icir.org> References: <20180516001930.GH43145@icir.org> <48511fe5-d534-e682-a886-7e3b4ad586e4@corelight.com> <20180517232755.GC6659@icir.org> <04ee0627-3f66-8e77-6ab9-610de32def68@corelight.com> <20180518161129.GC59866@icir.org> Message-ID: <5d48d977-21bb-0cbc-aabd-12e716ed1189@corelight.com> On 5/18/18 11:11 AM, Robin Sommer wrote: > What I was envisioning is more or less a clean slate: we'd migrate > over a few tickets, but essentially we'd start with an empty list. I > realize that sounds pretty harsh. However, I hardly ever see any > activity on older tickets in JIRA, and I generally believe that the > less open tickets a tracker has, the easier it is for people to > understand what's actually relevant and worth spending cycles on. I see those as independent issues: (1) There's many open tickets: you solve that by actually addressing the tickets (2) People don't know what tickets are relevant: you solve that by organizing (and maintaining) them according to relevance My take is that there's been a lack of effort in (2) and if that's not solved, starting with a clean slate on GitHub now only means it's likely to eventually end up in the same situation as JIRA later. What then? Move to another tracker again? > Tagging tickets may help, but in the end if everybody just filters 95% > out all the time anyways, I'm not sure what the value is. The value is actually having a central place for all tickets and knowing there won't be ongoing hassle with keeping JIRA updated. Or in reverse, I'm not sure what the value of keeping JIRA open at all is -- we either have to acknowledge its potential to go out of sync with GH in terms of duplicate reports, which makes it more of a hindrance IMO, or we still have to put effort to maintain it in addition to GH. > That said, I'm open to a real porting effort if people do believe it's > helpful to get all the JIRA tickets into GitHub. What do others think? Pedantically, I'm not saying "port all JIRA tickets", I still want just the "valid" ones whatever we decide that to mean. So more clarifications on potential porting criteria: * Someone is likely to report the same problem again * There's clear directions on how to reproduce an undesired behavior * There been a proposed plan of action recently And many tickets can be ruled out: * Vague feature requests * Not enough details / difficult to reproduce * Exceptionally old proposals / plans My plea is: extend the porting criteria beyond "important" and "recent", perform a review of all existing tickets, and, if JIRA is going to stick around as a read-only archive, leave it in a coherent final state. - Jon From dane.wullen at alumni.fh-aachen.de Fri May 18 10:59:41 2018 From: dane.wullen at alumni.fh-aachen.de (Dane Wullen) Date: Fri, 18 May 2018 19:59:41 +0200 Subject: [Bro-Dev] Include C++ header file in plugin function Message-ID: Hi there, I want to write a function as a plugin to convert a 4 byte hex string like "405f612f" to float/double (3.49031 in this case). An easy way to do so in C++ is this: #include union ulf { unsigned long ul; float f; }; int main() { ulf u; string str ="405f612f"; stringstream ss(str); ss >> hex >> u.ul; float f = u.f; cout << f << endl; } Unfortunatly, to use "stringstream" I will have to include the header file. Is this possible to do in plugin functions? Thanks. Dane -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180518/ac61ba82/attachment.html From johanna at icir.org Fri May 18 11:33:55 2018 From: johanna at icir.org (Johanna Amann) Date: Fri, 18 May 2018 11:33:55 -0700 Subject: [Bro-Dev] Include C++ header file in plugin function In-Reply-To: References: Message-ID: <217819CD-026E-4561-A47A-BDCE4C288976@icir.org> > > Unfortunatly, to use "stringstream" I will have to include the > header file. Is this possible to do in plugin functions? > Sure, plugins include libraries/other headers/etc. all the time. Johanna From dane.wullen at alumni.fh-aachen.de Fri May 18 12:09:44 2018 From: dane.wullen at alumni.fh-aachen.de (Dane Wullen) Date: Fri, 18 May 2018 21:09:44 +0200 Subject: [Bro-Dev] Include C++ header file in plugin function In-Reply-To: <217819CD-026E-4561-A47A-BDCE4C288976@icir.org> References: <217819CD-026E-4561-A47A-BDCE4C288976@icir.org> Message-ID: <24a45974-bd08-2a82-0d91-3f9b305b7884@alumni.fh-aachen.de> Hi Johanna, I should've asked the question differently - If it is possible AND how I can do it. Sorry! :-) But after a little bit of looking through the bro source code I found it. %%{ ??? #include %%} Greetings Dane Am 18.05.2018 um 20:33 schrieb Johanna Amann: >> >> Unfortunatly, to use "stringstream" I will have to include the >> header file. Is this possible to do in plugin functions? >> > > Sure, plugins include libraries/other headers/etc. all the time. > > Johanna From robin at icir.org Tue May 22 07:26:23 2018 From: robin at icir.org (Robin Sommer) Date: Tue, 22 May 2018 07:26:23 -0700 Subject: [Bro-Dev] Broker has landed in master, please test Message-ID: <20180522142623.GE34893@icir.org> We merged the new Broker version into Bro master yesterday. As this a major change to one of Bro's core components, I wanted to send a quick heads-up here, along with a couple of notes. With this merge we are completely replacing usage of Bro's classic communication system with Broker. All the standard scripts have been ported over, and BroControl has been adapted as well. The old communication system remains available for the time being, but is now deprecated and scheduled to be removed in Bro 2.7 (not 2.6). Broccoli is now turned off by default. With such a large change, I'm sure there'll be some more kinks to iron out still; that's where we need everybody's help. If you have an environment where you can test drive new Bro versions, please give this a try. We're interested in any feedback you have, both specific issues you encounter (best to file tickets) and general experiences with the new version, including in particular any observations about performance (best to send to this list). >From a user's perspective, not much should even be changing, most of the new stuff is under the hood. The exception are custom scripts doing communication themselves, they need to be ported over to Broker. Documentation for that is here: https://www.bro.org/sphinx-git/frameworks/broker.html, including a porting guide for existing scripts. Let us know if there's anything missing there that would be helpful. The Broker library itself comes with a new user manual as well, we'll get that online shortly. One specific note on upgrading existing Bro clusters: the meaning of "proxy" has changed. They still exist, but play a quite different role now. If you're currently using more than one proxy, we recommend going back to one; that'll most likely be fine with the standard scripts (and if not, please let us know!) Many thanks to Jon Siwek for the recent integration work tying up all the loose ends and getting Broker mergeable. Also thanks to those who have tested it already from the actor-system branch. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From robin at icir.org Tue May 22 08:56:01 2018 From: robin at icir.org (Robin Sommer) Date: Tue, 22 May 2018 08:56:01 -0700 Subject: [Bro-Dev] Moving to GitHub? In-Reply-To: <5d48d977-21bb-0cbc-aabd-12e716ed1189@corelight.com> References: <20180516001930.GH43145@icir.org> <48511fe5-d534-e682-a886-7e3b4ad586e4@corelight.com> <20180517232755.GC6659@icir.org> <04ee0627-3f66-8e77-6ab9-610de32def68@corelight.com> <20180518161129.GC59866@icir.org> <5d48d977-21bb-0cbc-aabd-12e716ed1189@corelight.com> Message-ID: <20180522155601.GA72456@icir.org> > * Someone is likely to report the same problem again > * There's clear directions on how to reproduce an undesired behavior > * There been a proposed plan of action recently > > And many tickets can be ruled out: > > * Vague feature requests > * Not enough details / difficult to reproduce > * Exceptionally old proposals / plans Yeah, I'm on board with these. I'd probably interpret them more conservatively than you towards closing more tickets, but that's fine. As you have volunteered to take this one, I'd say you get to make the call: just go through and port over what you think makes sense along those lines. As one additional piece, let's also think about some tags to use for classifying tickets, including one for what's good tasks for newcomers who want to get into the code. (In principle I also like Alan's suggestion of moving everything over and then just close them out so that the history remains. But I'm afraid that couldn't be automated easily and would then just be too much work.) > starting with a clean slate on GitHub now only means it's likely to > eventually end up in the same situation as JIRA later. What then? > Move to another tracker again? Doesn't need to be as drastic: as some people here can confirm, I have no problem doing extensive sweeps if things get too overwhelming. :-) But yes, point taken, my hope is that we can stay on top of things on the new tracker and make an effort to get stuff addressed and resolved. We'll see. :) Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From dopheide at es.net Wed May 23 13:12:57 2018 From: dopheide at es.net (Michael Dopheide) Date: Wed, 23 May 2018 15:12:57 -0500 Subject: [Bro-Dev] Broker has landed in master, please test In-Reply-To: <20180522142623.GE34893@icir.org> References: <20180522142623.GE34893@icir.org> Message-ID: Do you want dumb programming questions asked here or on the main Bro list? While most people might not need it yet, discussion there might help get more people interested or help avoid issues with custom policy conversion. For here though, can you elaborate on the going down to one proxy? My understanding still isn't strong, but that seems to be opposed to the idea of using Cluster::publish_hrw to spread memory across proxies. Thanks, -Dop On Tue, May 22, 2018 at 9:26 AM, Robin Sommer wrote: > We merged the new Broker version into Bro master yesterday. As this a > major change to one of Bro's core components, I wanted to send a quick > heads-up here, along with a couple of notes. > > With this merge we are completely replacing usage of Bro's classic > communication system with Broker. All the standard scripts have been > ported over, and BroControl has been adapted as well. The old > communication system remains available for the time being, but is now > deprecated and scheduled to be removed in Bro 2.7 (not 2.6). Broccoli > is now turned off by default. > > With such a large change, I'm sure there'll be some more kinks to iron > out still; that's where we need everybody's help. If you have an > environment where you can test drive new Bro versions, please give > this a try. We're interested in any feedback you have, both specific > issues you encounter (best to file tickets) and general experiences > with the new version, including in particular any observations about > performance (best to send to this list). > > >From a user's perspective, not much should even be changing, most of > the new stuff is under the hood. The exception are custom scripts > doing communication themselves, they need to be ported over to Broker. > Documentation for that is here: > https://www.bro.org/sphinx-git/frameworks/broker.html, including a > porting guide for existing scripts. Let us know if there's anything > missing there that would be helpful. The Broker library itself comes > with a new user manual as well, we'll get that online shortly. > > One specific note on upgrading existing Bro clusters: the meaning of > "proxy" has changed. They still exist, but play a quite different role > now. If you're currently using more than one proxy, we recommend going > back to one; that'll most likely be fine with the standard scripts > (and if not, please let us know!) > > Many thanks to Jon Siwek for the recent integration work tying up all > the loose ends and getting Broker mergeable. Also thanks to those who > have tested it already from the actor-system branch. > > Robin > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180523/79873e5e/attachment.html From slagell at illinois.edu Wed May 23 13:16:41 2018 From: slagell at illinois.edu (Slagell, Adam J) Date: Wed, 23 May 2018 20:16:41 +0000 Subject: [Bro-Dev] Broker has landed in master, please test In-Reply-To: References: <20180522142623.GE34893@icir.org> Message-ID: On May 23, 2018, at 3:12 PM, Michael Dopheide > wrote: Do you want dumb programming questions asked here or on the main Bro list? While most people might not need it yet, discussion there might help get more people interested or help avoid issues with custom policy conversion. I think those question belong on the main list which is for using Bro and its language. This list is really more about development of Bro itself. ------ Adam J. Slagell Director, Cybersecurity & Networking Division Chief Information Security Officer National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180523/1fb9c80d/attachment.html From jsiwek at corelight.com Wed May 23 15:29:09 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Wed, 23 May 2018 17:29:09 -0500 Subject: [Bro-Dev] Broker has landed in master, please test In-Reply-To: References: <20180522142623.GE34893@icir.org> Message-ID: On 5/23/18 3:12 PM, Michael Dopheide wrote: > For here though, can you elaborate on the going down to one proxy?? My > understanding still isn't strong, but that seems to be opposed to the > idea of using?Cluster::publish_hrw to spread memory across proxies. The idea is to try starting with a single proxy and then scale your deployment based on what you actually need, and there may not be that great of a need at the moment as the default scripts that ship with Bro do not widely use the HRW/pool/partitioning APIs yet. By default, it's currently just the Software framework that will use Cluster::publish_hrw. I also plan to soon change the Intel framework to make use of Cluster::relay_rr. There's also an option in the various Known::* scripts for users to opt-in to an alternate implementation that uses HRW + tables instead of the default approach of data stores. Different sites could also have different requirements/usage of those default scripts and it's all too new to give better suggestions other than "try one proxy, add more as needed". - Jon From robin at icir.org Thu May 24 08:08:04 2018 From: robin at icir.org (Robin Sommer) Date: Thu, 24 May 2018 08:08:04 -0700 Subject: [Bro-Dev] Broker has landed in master, please test In-Reply-To: References: <20180522142623.GE34893@icir.org> Message-ID: <20180524150804.GF71036@icir.org> On Wed, May 23, 2018 at 20:16 +0000, Adam wrote: > I think those question belong on the main list which is for using Bro > and its language. This list is really more about development of Bro > itself. Just to give context here, the reason I sent the original mail about Broker here, including the request for feedback, was to limit the initial round of testing to folks quite familiar with Bro and its internals. That gives us a chance to spot any obvious issues quickly before annoying everybody else. :-) But discussing it at either place is fine of course, whatever works best for folks. If things seem to work, we should definitely also announce the merge more broadly. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From vlad at es.net Thu May 24 12:50:32 2018 From: vlad at es.net (Vlad Grigorescu) Date: Thu, 24 May 2018 19:50:32 +0000 Subject: [Bro-Dev] bro-devel package? Message-ID: There are a couple of cases where I think it'd be useful to have a bro-devel package -- a package that I can install on a system, and then be able to build plugins against Bro. (This is the same model as other *-devel packages, such as openssl, libpcap, etc.) Right now, if I compile Bro from source, run make install, I can't build a package with what's in /usr/local -- Bro needs the entire source tree, since the required headers are not present in /usr/local. I'm curious how people are dealing with this issue, and if anyone has thoughts on whether this would be useful, and if so, what it would take to build such a package. Thanks, --Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180524/c41b5c09/attachment.html From michalpurzynski1 at gmail.com Thu May 24 13:04:53 2018 From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=) Date: Thu, 24 May 2018 22:04:53 +0200 Subject: [Bro-Dev] bro-devel package? In-Reply-To: References: Message-ID: Well that?s s great question! Right now my spec for building plugins downloads entire bro tree and does some cmake plugin to avoid waiting for the tree to build. > On May 24, 2018, at 9:50 PM, Vlad Grigorescu wrote: > > There are a couple of cases where I think it'd be useful to have a bro-devel package -- a package that I can install on a system, and then be able to build plugins against Bro. (This is the same model as other *-devel packages, such as openssl, libpcap, etc.) > > Right now, if I compile Bro from source, run make install, I can't build a package with what's in /usr/local -- Bro needs the entire source tree, since the required headers are not present in /usr/local. > > I'm curious how people are dealing with this issue, and if anyone has thoughts on whether this would be useful, and if so, what it would take to build such a package. > > Thanks, > > --Vlad > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From jazoff at illinois.edu Thu May 24 13:21:47 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 24 May 2018 20:21:47 +0000 Subject: [Bro-Dev] bro-devel package? In-Reply-To: References: Message-ID: > On May 24, 2018, at 3:50 PM, Vlad Grigorescu wrote: > > There are a couple of cases where I think it'd be useful to have a bro-devel package -- a package that I can install on a system, and then be able to build plugins against Bro. (This is the same model as other *-devel packages, such as openssl, libpcap, etc.) > > Right now, if I compile Bro from source, run make install, I can't build a package with what's in /usr/local -- Bro needs the entire source tree, since the required headers are not present in /usr/local. > > I'm curious how people are dealing with this issue, and if anyone has thoughts on whether this would be useful, and if so, what it would take to build such a package. > I ran into this issue in two places: on our infrastructure and working out how to support bro as a language on travis-ci. For our installations I worked around it by just building packages that contain bro plus all the plugins I need and dump the entire thing into a single RPM. This works ok for me, but would never work for something like debian that is probably trying to figure out how to package some of the bro plugins. This also doesn't work for adding bro support for travis.. the OBS RPMs come close.. we can install those plus bro-pkg, but any packages that need to be built against bro won't compile. ? Justin Azoff From jsiwek at corelight.com Thu May 24 13:28:30 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 24 May 2018 15:28:30 -0500 Subject: [Bro-Dev] bro-devel package? In-Reply-To: References: Message-ID: On Thu, May 24, 2018 at 2:50 PM, Vlad Grigorescu wrote: > There are a couple of cases where I think it'd be useful to have a bro-devel > package -- a package that I can install on a system, and then be able to > build plugins against Bro. (This is the same model as other *-devel > packages, such as openssl, libpcap, etc.) Yes, I think it's useful and something that should be done. It was fairly low on my list of things to try to do before 2.6. Related: https://bro-tracker.atlassian.net/browse/BIT-1922 > I'm curious how people are dealing with this issue, and if anyone has > thoughts on whether this would be useful, and if so, what it would take to > build such a package. My guess as to what needs to be done: * separate bifcl into its own submodule: I think this should be easy * install Bro's headers: at least I think that's all that we need to do on our end, it's also a bit of an open question as to whether we just install them all for now until we get a more organized API or can get away with a smaller subset * update the plugin CMake/configure skeletons and/or bro-config to be able to make use of the above two points - Jon From johanna at icir.org Thu May 31 08:13:04 2018 From: johanna at icir.org (Johanna Amann) Date: Thu, 31 May 2018 08:13:04 -0700 Subject: [Bro-Dev] System upgrades Message-ID: <20180531151304.s2vgs6dlxdyhnlev@user157.sys.ICSI.Berkeley.EDU> Hi, the main bro server will go down for system upgrades a bit. This should mostly affect the git server, the webpage should stay accessible. I will write another email once I am done. Johanna From johanna at icir.org Thu May 31 09:47:09 2018 From: johanna at icir.org (Johanna Amann) Date: Thu, 31 May 2018 09:47:09 -0700 Subject: [Bro-Dev] System upgrades In-Reply-To: <20180531151304.s2vgs6dlxdyhnlev@user157.sys.ICSI.Berkeley.EDU> References: <20180531151304.s2vgs6dlxdyhnlev@user157.sys.ICSI.Berkeley.EDU> Message-ID: <20180531164709.up26m2sv2oukl2g4@user157.sys.ICSI.Berkeley.EDU> This is done, please let me know if you encounter any problems. Johanna On Thu, May 31, 2018 at 08:13:04AM -0700, Johanna Amann wrote: > Hi, > > the main bro server will go down for system upgrades a bit. This should > mostly affect the git server, the webpage should stay accessible. > > I will write another email once I am done. > > Johanna > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev >