From vlad at grigorescu.org Thu Mar 1 18:52:56 2018 From: vlad at grigorescu.org (Vlad Grigorescu) Date: Thu, 1 Mar 2018 20:52:56 -0600 Subject: [Bro-Dev] UDP connection_established event? Message-ID: I would like to propose a new event in Bro, one that would fire when a UDP connection is established (i.e. a response is observed within some time frame after a request is seen). Basically, the UDP equivalent of connection_established. Currently, I think the only way to do this would be either with new_connection or with udp_reply. Neither of these seem like great solutions, as they'd require keeping state in script-land, and most of the events wouldn't be useful. Does anyone have thoughts about this? Thanks, --Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180301/5fd10c47/attachment.html From jan.grashoefer at gmail.com Mon Mar 5 02:55:56 2018 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Mon, 5 Mar 2018 11:55:56 +0100 Subject: [Bro-Dev] UDP connection_established event? In-Reply-To: References: Message-ID: On 02/03/18 03:52, Vlad Grigorescu wrote: > I would like to propose a new event in Bro, one that would fire when a UDP > connection is established (i.e. a response is observed within some time > frame after a request is seen). Basically, the UDP equivalent of > connection_established. > > [...] > > Does anyone have thoughts about this? I definitely see the need to correlate request-response-pairs for UDP protocols but as UDP is *connectionless*, the term UDP connection sounds very strange to me. Maybe a general notion of request-response protocols could be established. Corresponding protocols could trigger general events. For some protocols there might be even a session concept. Jan From vlad at grigorescu.org Mon Mar 5 07:46:16 2018 From: vlad at grigorescu.org (Vlad Grigorescu) Date: Mon, 5 Mar 2018 09:46:16 -0600 Subject: [Bro-Dev] UDP connection_established event? In-Reply-To: References: Message-ID: True, I'm just basing it off of Bro's mechanism to turn some UDP traffic into "connections" that fit into its model. I guess what I'm looking for is a connection_state_add to go with the existing connection_state_remove. It wouldn't be UDP-specific, but it might fit the current event model a bit better. On Mon, Mar 5, 2018 at 4:55 AM, Jan Grash?fer wrote: > On 02/03/18 03:52, Vlad Grigorescu wrote: > > I would like to propose a new event in Bro, one that would fire when a > UDP > > connection is established (i.e. a response is observed within some time > > frame after a request is seen). Basically, the UDP equivalent of > > connection_established. > > > > [...] > > > > Does anyone have thoughts about this? > > I definitely see the need to correlate request-response-pairs for UDP > protocols but as UDP is *connectionless*, the term UDP connection sounds > very strange to me. Maybe a general notion of request-response protocols > could be established. Corresponding protocols could trigger general > events. For some protocols there might be even a session concept. > > Jan > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180305/920097de/attachment.html From robin at icir.org Thu Mar 8 09:15:34 2018 From: robin at icir.org (Robin Sommer) Date: Thu, 8 Mar 2018 09:15:34 -0800 Subject: [Bro-Dev] Offline Broker usage (Re: [Bro-Commits] [git/bro] topic/actor-system: Fix Known scripts to be able to use alternate implemenation (50e1498)) In-Reply-To: <201803080338.w283ci8q031101@bro-ids.icir.org> References: <201803080338.w283ci8q031101@bro-ids.icir.org> Message-ID: <20180308171534.GA4416@corelight.com> Jon, I noticed your commit message on data store expiration: > commit 50e1498d2b39d6af1f70dbc042ab544506a67e43 > Author: Jon Siwek > Date: Wed Mar 7 21:24:46 2018 -0600 > > Fix Known scripts to be able to use alternate implemenation > > And run the external test suite using the alternate implementation > due to data stores behaving differently when running on offline pcaps. > E.g. expirations are based on wall time, not packet time, and timeouts > (which *are* based on packet time) may occur when the store is still > initializing due to a large interval of packet time passing. That brings up an interesting question on data store semantics in offline vs online mode. Ideally, there wouldn't be any difference between the two operation modes, so that running on a trace gives exactly the same results as online. That would match how Bro generally operates. Could we make data store expiration driven by network time? That'd need an API for Bro to drive Broker time forward. And for the initialization, maybe Bro could wait for the initialization to finish? Although I'm not quite sure here which initialization that refers too, may not be feasible. Are there other differences with stores between online and offline operation? Robin -- Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com From jsiwek at corelight.com Thu Mar 8 09:50:18 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 8 Mar 2018 11:50:18 -0600 Subject: [Bro-Dev] Offline Broker usage (Re: [Bro-Commits] [git/bro] topic/actor-system: Fix Known scripts to be able to use alternate implemenation (50e1498)) In-Reply-To: <20180308171200.GR69087@corelight.com> References: <201803080338.w283ci8q031101@bro-ids.icir.org> <20180308171200.GR69087@corelight.com> Message-ID: On Thu, Mar 8, 2018 at 11:12 AM, Robin Sommer wrote: > That brings up an interesting question on data store semantics in > offline vs online mode. Ideally, there wouldn't be any difference > between the two operation modes, so that running on a trace gives > exactly the same results as online. That would match how Bro generally > operates. Yeah, that's ideal. I was mostly eager to get into a stable "all tests pass" state with this possibly temporary commit. > Could we make data store expiration driven by network time? > That'd need an API for Bro to drive Broker time forward. And for the > initialization, maybe Bro could wait for the initialization to finish? Those were also my basic thoughts, though needs investigation to try things out (it's on my todo list). > Are there other differences with stores between online and offline > operation? Not that I've found yet. - Jon From jsiwek at corelight.com Thu Mar 8 11:28:26 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 8 Mar 2018 13:28:26 -0600 Subject: [Bro-Dev] Broker port status Message-ID: Hi all, this is just a status update on porting Bro to use the new Broker communication system. I'd say the topic/actor-system branch is now functionally complete with still a few weeks left of internal cleanup/improvements. Open questions: * Rename "proxy" nodes? The previous idea was to call them "data nodes", though I don't see the benefit. It's awkward to talk about because there's no shorthand for that node type: you can say "logger", "manager", or "worker", though you'd have to say the whole "data node" phrase to make any sense. That also shows that maybe it's helpful to name nodes based upon a personifiable role that they play: "data" isn't a role/action performed by a node/person. IMO, "proxy" is still accurate to the role that these nodes perform: they are intermediaries for offloading analysis and/or storage. Are there other ideas or is everyone wanting to go ahead with "data node" ? * How much of the old comm. system to deprecate vs. remove? (1) &synchronized, &mergeable, &persistent. Seems fine to deprecate these now. (2) Communication framework scripts. It's awkward to deprecate stuff here since they internally will be using what is deprecated. I think it makes sense to just remove it and let users manually make a copy if they still need it. Thoughts? (3) Old C++ comm. code, e.g. RemoteSerializer. I think we'll leave this untouched for the 2.6 release? (4) BIFs associated with old comm. system. Depends on (3) (and also actually (2)), though if the internal core code remains, I think it makes sense to add &deprecated to these. Anything else? And just a quick summary of planned internal improvements: There's still several usability issues with Broker data stores. * Online vs. offline operation discrepancies. Expiration times in Broker are driven by wall time, though for a more familiar/expected offline usage experience, Bro would need to be able to drive expiration via network time. There's also some wall-time-based data store initialization delays that make offline Bro usage difficult as the Bro-based query timeouts are network-time-based. * The way clone snapshots are distributed needs to be optimized. Currently, for any clone that connects, the master floods *all* clones with a snapshot. * Certain store operations like "increment" fail if the key they are to operate on does not exist. It makes more sense to me to assume the user wants to insert the key with a sane default value (e.g. zero/empty) in those cases, otherwise, it's awkward/race-prone to require they do it themselves. Let me know if there's thoughts on anything else still left to add/improve. - Jon From jazoff at illinois.edu Thu Mar 8 12:18:08 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 8 Mar 2018 20:18:08 +0000 Subject: [Bro-Dev] Broker port status In-Reply-To: References: Message-ID: <4C9D8853-A1A9-4E36-B27C-78267F5229D6@illinois.edu> > On Mar 8, 2018, at 1:28 PM, Jon Siwek wrote: > > Hi all, this is just a status update on porting Bro to use the new > Broker communication system. I'd say the topic/actor-system branch is > now functionally complete with still a few weeks left of internal > cleanup/improvements. Awesome, I'll get it deployed here on a test cluster. > Open questions: > > * Rename "proxy" nodes? > > The previous idea was to call them "data nodes", though I don't see > the benefit. It's awkward to talk about because there's no shorthand > for that node type: you can say "logger", "manager", or "worker", > though you'd have to say the whole "data node" phrase to make any > sense. That also shows that maybe it's helpful to name nodes based > upon a personifiable role that they play: "data" isn't a role/action > performed by a node/person. IMO, "proxy" is still accurate to the > role that these nodes perform: they are intermediaries for offloading > analysis and/or storage. Are there other ideas or is everyone wanting > to go ahead with "data node" ? I think I was the one calling them data nodes, but I only did that because that's what they were called in the original broker integration branch that Mathias Fischer started. I don't care about the name, as long as it's documented as a proxy node is for "offloading analysis and storage" that works for me. > * How much of the old comm. system to deprecate vs. remove? > > (1) &synchronized, &mergeable, &persistent. Seems fine to deprecate these now. I'm fine with it going away, but there needs to be some sort of replacement for &synchronized, minimally a how-to for porting existing scripts to something else. I don't think anyone currently uses mergeable or persistent. ? Justin Azoff From jsiwek at corelight.com Thu Mar 8 12:51:40 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 8 Mar 2018 14:51:40 -0600 Subject: [Bro-Dev] Broker port status In-Reply-To: <4C9D8853-A1A9-4E36-B27C-78267F5229D6@illinois.edu> References: <4C9D8853-A1A9-4E36-B27C-78267F5229D6@illinois.edu> Message-ID: On Thu, Mar 8, 2018 at 2:18 PM, Azoff, Justin S wrote: >> (1) &synchronized, &mergeable, &persistent. Seems fine to deprecate these now. > > I'm fine with it going away, but there needs to be some sort of replacement for &synchronized, > minimally a how-to for porting existing scripts to something else. The answer is generally "use the new store API" to replace &synchronized or &persistent. It's difficult to give a canonical porting strategy as it depends on how the data is being accessed/modified, though you can find a preview of the how-to port document at [1]. Links and stuff will get rendered more nicely when it's live on bro.org. - Jon [1] https://github.com/bro/bro/blob/topic/actor-system/doc/frameworks/broker.rst From jsiwek at corelight.com Thu Mar 8 15:46:00 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 8 Mar 2018 17:46:00 -0600 Subject: [Bro-Dev] Broker port status In-Reply-To: References: <4C9D8853-A1A9-4E36-B27C-78267F5229D6@illinois.edu> Message-ID: On Thu, Mar 8, 2018 at 5:07 PM, Azoff, Justin S wrote: > One thing I notice is that the traffic to/from the manager box and cpu has increased quite a bit. > > Ignoring the large CPU spikes from building the new branch just before the switch at 15:30, the overall cpu load on the manager box is about 3x higher. > The bandwidth isn't terribly excessive, but it increased from about 16mbit to 40mbit (stupid graph is in mebibytes). > > Based on some capstats runs, it's all going to the logger port 47761. I have another graph that shows the total log volume being written to disk and that hasn't changed. > So it looks like this branch uses 2x the cpu and bandwidth to write the same volume of logs. Interesting, I would have thought maybe the manager could be utilized more, though not the logger. Will have to think about that. An interesting experiment you could try is switching to an alternate implementation of the Known scripts. E.g. stick this in local.bro: redef Known::use_host_store = F; redef Known::use_cert_store = F; redef Known::use_device_store = F; redef Known::use_service_store = F; I'd expect that could reduce bandwidth a bit. Using data stores the Known scripts do: worker -> manager: put this key into your store if it doesn't exist manager -> worker: the key was accepted (or rejected) worker -> logger: log the key (sent only if manager accepted key as unique) Else the alternate version of Known scripts do: worker -> proxy: here's a key proxy -> logger: log the key (if proxy hasn't seen it recently) Maybe cuts out 1/3 the message volume contributed by Known scripts that way. - Jon From jazoff at illinois.edu Thu Mar 8 16:11:20 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 9 Mar 2018 00:11:20 +0000 Subject: [Bro-Dev] Broker port status In-Reply-To: References: <4C9D8853-A1A9-4E36-B27C-78267F5229D6@illinois.edu> Message-ID: <2E5D9D40-FE69-4442-9FE6-9417BEB120FC@illinois.edu> > On Mar 8, 2018, at 5:46 PM, Jon Siwek wrote: > > On Thu, Mar 8, 2018 at 5:07 PM, Azoff, Justin S wrote: > >> One thing I notice is that the traffic to/from the manager box and cpu has increased quite a bit. >> >> Ignoring the large CPU spikes from building the new branch just before the switch at 15:30, the overall cpu load on the manager box is about 3x higher. >> The bandwidth isn't terribly excessive, but it increased from about 16mbit to 40mbit (stupid graph is in mebibytes). >> >> Based on some capstats runs, it's all going to the logger port 47761. I have another graph that shows the total log volume being written to disk and that hasn't changed. >> So it looks like this branch uses 2x the cpu and bandwidth to write the same volume of logs. > > Interesting, I would have thought maybe the manager could be utilized > more, though not the logger. Will have to think about that. Yeah.. it's definitely worker -> logger: logger=47761, manager=47762, proxy=47763 [root at bro-test ~]# capstats -i enp3s0f0 -I 1 -f 'port 47761' -n 5 1520553009.488934 pkts=1173 kpps=1.1 kbytes=2837 mbps=22.6 nic_pkts=1175 nic_drops=0 u=0 t=1173 i=0 o=0 nonip=0 1520553010.489053 pkts=1280 kpps=1.3 kbytes=3218 mbps=26.4 nic_pkts=2455 nic_drops=0 u=0 t=1280 i=0 o=0 nonip=0 1520553011.489177 pkts=1086 kpps=1.1 kbytes=2912 mbps=23.9 nic_pkts=3541 nic_drops=0 u=0 t=1086 i=0 o=0 nonip=0 1520553012.489310 pkts=1382 kpps=1.4 kbytes=3757 mbps=30.8 nic_pkts=4923 nic_drops=0 u=0 t=1382 i=0 o=0 nonip=0 1520553013.489471 pkts=1563 kpps=1.6 kbytes=3816 mbps=31.3 nic_pkts=6486 nic_drops=0 u=0 t=1563 i=0 o=0 nonip=0 [root at bro-test ~]# capstats -i enp3s0f0 -I 1 -f 'port 47762' -n 5 1520553018.874154 pkts=346 kpps=0.3 kbytes=92 mbps=0.7 nic_pkts=348 nic_drops=0 u=0 t=346 i=0 o=0 nonip=0 1520553019.875111 pkts=1044 kpps=1.0 kbytes=283 mbps=2.3 nic_pkts=1391 nic_drops=0 u=0 t=1044 i=0 o=0 nonip=0 1520553020.875448 pkts=759 kpps=0.8 kbytes=199 mbps=1.6 nic_pkts=2150 nic_drops=0 u=0 t=759 i=0 o=0 nonip=0 1520553021.875575 pkts=654 kpps=0.7 kbytes=196 mbps=1.6 nic_pkts=2804 nic_drops=0 u=0 t=654 i=0 o=0 nonip=0 1520553022.875920 pkts=470 kpps=0.5 kbytes=123 mbps=1.0 nic_pkts=3274 nic_drops=0 u=0 t=470 i=0 o=0 nonip=0 [root at bro-test ~]# capstats -i enp3s0f0 -I 1 -f 'port 47763' -n 5 1520553025.700919 pkts=26 kpps=0.0 kbytes=8 mbps=0.1 nic_pkts=31 nic_drops=0 u=0 t=26 i=0 o=0 nonip=0 1520553026.701047 pkts=27 kpps=0.0 kbytes=9 mbps=0.1 nic_pkts=58 nic_drops=0 u=0 t=27 i=0 o=0 nonip=0 1520553027.701185 pkts=24 kpps=0.0 kbytes=8 mbps=0.1 nic_pkts=82 nic_drops=0 u=0 t=24 i=0 o=0 nonip=0 1520553028.701306 pkts=55 kpps=0.1 kbytes=18 mbps=0.2 nic_pkts=137 nic_drops=0 u=0 t=55 i=0 o=0 nonip=0 1520553029.701434 pkts=9 kpps=0.0 kbytes=3 mbps=0.0 nic_pkts=146 nic_drops=0 u=0 t=9 i=0 o=0 nonip=0 [root at bro-test ~]# On another cluster seeing the same traffic with 56 workers running 2.5.3ish shows this for the logger port at the same time: 1520553725.875230 pkts=681 kpps=0.7 kbytes=1946 mbps=15.5 nic_pkts=768 nic_drops=0 u=0 t=681 i=0 o=0 nonip=0 1520553726.875374 pkts=508 kpps=0.5 kbytes=1605 mbps=13.1 nic_pkts=1392 nic_drops=0 u=0 t=508 i=0 o=0 nonip=0 1520553727.875520 pkts=554 kpps=0.6 kbytes=1681 mbps=13.8 nic_pkts=2247 nic_drops=0 u=0 t=554 i=0 o=0 nonip=0 1520553728.875713 pkts=536 kpps=0.5 kbytes=1708 mbps=14.0 nic_pkts=2902 nic_drops=0 u=0 t=536 i=0 o=0 nonip=0 1520553729.875845 pkts=518 kpps=0.5 kbytes=1669 mbps=13.7 nic_pkts=3641 nic_drops=0 u=0 t=518 i=0 o=0 nonip=0 [root at bro-test ~]# broctl top manager logger Name Type Host Pid Proc VSize Rss Cpu Cmd logger logger bro-test 2016 parent 938M 283M 81% bro manager manager bro-test 2107 parent 619M 244M 25% bro [root at bro-test ~]# [root at bro-dev ~]# broctl top manager logger Name Type Host Pid Proc VSize Rss Cpu Cmd logger logger bro-dev 10704 child 492M 150M 6% bro logger logger bro-dev 10571 parent 1G 221M 0% bro manager manager bro-dev 10734 parent 909M 497M 0% bro manager manager bro-dev 10782 child 482M 110M 0% bro cpus are about the same, test has X5650 @ 2.67GHz and dev has E5-2420 @ 1.90GHz according to passmark, the older X5650 has a slightly faster single core performance. > An interesting experiment you could try is switching to an alternate > implementation of the Known scripts. E.g. stick this in local.bro: > > redef Known::use_host_store = F; > redef Known::use_cert_store = F; > redef Known::use_device_store = F; > redef Known::use_service_store = F; I tried doing that for an unrelated reason.. but I'm not sure if it works right. You can't redef Known::use_service_store before the script is loaded, because otherwise you get Can't document redef of Known::use_service_store, identifier lookup failed but if you load the script first, the @if ( Known::use_service_store ) block is already evaluated before you change the value. [jazoff at bro-test ~]$ cat foo.bro @load protocols/conn/known-services redef Known::use_service_store=F; event bro_init() { print Known::services; } [jazoff at bro-test ~]$ bro foo.bro error in ./foo.bro, line 6: unknown identifier Known::services, at or near "Known::services" [jazoff at bro-test ~]$ cat foo.bro redef Known::use_service_store=F; @load protocols/conn/known-services event bro_init() { print Known::services; } [jazoff at bro-test ~]$ bro foo.bro error in ./foo.bro, line 1: "redef" used but not previously defined (Known::use_service_store) internal warning in ./foo.bro, line 1: Can't document redef of Known::use_service_store, identifier lookup failed error in ./foo.bro, line 1 and /srv/bro/share/bro/policy/protocols/conn/known-services.bro, line 34: already defined (Known::use_service_store) error in /srv/bro/share/bro/policy/protocols/conn/known-services.bro, line 40: value used but not set (Known::use_service_store) error in /srv/bro/share/bro/policy/protocols/conn/known-services.bro, line 40: invalid expression in @if (Known::use_service_store) error in /srv/bro/share/bro/policy/protocols/conn/known-services.bro, line 93: value used but not set (Known::use_service_store) error in /srv/bro/share/bro/policy/protocols/conn/known-services.bro, line 93: invalid expression in @if (Known::use_service_store) ? Justin Azoff From jsiwek at corelight.com Thu Mar 8 16:32:30 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 8 Mar 2018 18:32:30 -0600 Subject: [Bro-Dev] Broker port status In-Reply-To: <2E5D9D40-FE69-4442-9FE6-9417BEB120FC@illinois.edu> References: <4C9D8853-A1A9-4E36-B27C-78267F5229D6@illinois.edu> <2E5D9D40-FE69-4442-9FE6-9417BEB120FC@illinois.edu> Message-ID: On Thu, Mar 8, 2018 at 6:11 PM, Azoff, Justin S wrote: >> redef Known::use_host_store = F; >> redef Known::use_cert_store = F; >> redef Known::use_device_store = F; >> redef Known::use_service_store = F; > but if you load the script first, the > > @if ( Known::use_service_store ) > > block is already evaluated before you change the value. Yeah, I realized that and fixed it yesterday. Have you pulled the latest commits from the topic/actor-system branch? - Jon From jazoff at illinois.edu Thu Mar 8 15:07:32 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 8 Mar 2018 23:07:32 +0000 Subject: [Bro-Dev] Broker port status In-Reply-To: References: <4C9D8853-A1A9-4E36-B27C-78267F5229D6@illinois.edu> Message-ID: On Mar 8, 2018, at 2:51 PM, Jon Siwek > wrote: On Thu, Mar 8, 2018 at 2:18 PM, Azoff, Justin S > wrote: (1) &synchronized, &mergeable, &persistent. Seems fine to deprecate these now. I'm fine with it going away, but there needs to be some sort of replacement for &synchronized, minimally a how-to for porting existing scripts to something else. The answer is generally "use the new store API" to replace &synchronized or &persistent. It's difficult to give a canonical porting strategy as it depends on how the data is being accessed/modified, though you can find a preview of the how-to port document at [1]. Links and stuff will get rendered more nicely when it's live on bro.org. - Jon [1] https://github.com/bro/bro/blob/topic/actor-system/doc/frameworks/broker.rst The most common use of &synchronized here is for loading files into tables using input+reread. a cron job updates the table on the manager which updates the in memory table and syncs it over to all the workers. These are read only tables, so I think the best replacement for those may be to switch to the new configuration framework, which would probably mean the deletion of a ton of verbose Input::add_table statements. I got the actor-system branch running on a small test cluster.. manager+logger+proxy on one box and 32 workers on 2 physical boxes. It seems to be working for the most part. known hosts and friends are working great now with no duplicate entries at startup. One thing I notice is that the traffic to/from the manager box and cpu has increased quite a bit. Ignoring the large CPU spikes from building the new branch just before the switch at 15:30, the overall cpu load on the manager box is about 3x higher. The bandwidth isn't terribly excessive, but it increased from about 16mbit to 40mbit (stupid graph is in mebibytes). Based on some capstats runs, it's all going to the logger port 47761. I have another graph that shows the total log volume being written to disk and that hasn't changed. So it looks like this branch uses 2x the cpu and bandwidth to write the same volume of logs. [cid:498F993F-1122-4661-B343-AB3683018F09] [cid:D6C31831-533C-40AB-995B-B025A0237E2C] ? Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180308/43c9175a/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2018-03-08 at 4.58.39 PM.png Type: image/png Size: 66285 bytes Desc: Screen Shot 2018-03-08 at 4.58.39 PM.png Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180308/43c9175a/attachment-0002.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2018-03-08 at 4.33.19 PM.png Type: image/png Size: 336516 bytes Desc: Screen Shot 2018-03-08 at 4.33.19 PM.png Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180308/43c9175a/attachment-0003.bin From robin at icir.org Thu Mar 8 17:29:14 2018 From: robin at icir.org (Robin Sommer) Date: Thu, 8 Mar 2018 17:29:14 -0800 Subject: [Bro-Dev] Broker port status In-Reply-To: References: Message-ID: <20180309012914.GE75717@icir.org> On Thu, Mar 08, 2018 at 13:28 -0600, you wrote: > * Rename "proxy" nodes? "compute" maybe? > (1) &synchronized, &mergeable, &persistent. Seems fine to deprecate these now. Agree. > (2) Communication framework scripts. It's awkward to deprecate stuff > here since they internally will be using what is deprecated. I think > it makes sense to just remove it and let users manually make a copy if > they still need it. Likewise agree. > (3) Old C++ comm. code, e.g. RemoteSerializer. I think we'll leave > this untouched for the 2.6 release? Yep, I'm looking forward to ripping that out for 2.7. :) > (4) BIFs associated with old comm. system. Depends on (3) (and also > actually (2)), though if the internal core code remains, I think it > makes sense to add &deprecated to these. Deprecating them sounds good. > It makes more sense to me to assume the user wants to insert the key > with a sane default value (e.g. zero/empty) in those cases, otherwise, > it's awkward/race-prone to require they do it themselves. Agree with this as well, has always felt a bit awkward to me too. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From vitaly.r at seed4.me Thu Mar 8 21:55:54 2018 From: vitaly.r at seed4.me (Vitaly Repin) Date: Fri, 9 Mar 2018 05:55:54 +0000 Subject: [Bro-Dev] "bro" name is non-inclusive? (Contribution is rejected based ) Message-ID: Hello, I would like to bring your attention to one strange and unexpected problem I faced while trying to include user agent parser for bro I have developed (plugin: https://github.com/vitalyrepin/uap-bro ) into the list of uap implementations https://github.com/ua-parser The req -- WBR & WBW, Vitaly -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180309/dad6e421/attachment.html From vitaly.r at seed4.me Thu Mar 8 22:03:34 2018 From: vitaly.r at seed4.me (Vitaly Repin) Date: Fri, 9 Mar 2018 06:03:34 +0000 Subject: [Bro-Dev] "bro" name is non-inclusive? (Contribution is rejected based ) Message-ID: Hello, I would like to bring your attention to one strange and extremely unexpected difficulty I faced while trying to include user agent parser for bro I have developed (plugin: https://github.com/vitalyrepin/uap-bro ) into the list of uap implementations https://github.com/ua-parser The request I have made has been rejected: https://groups.google.com/forum/#!topic/ua-parser/nJEc6SZIxyU The reason: /quote begin/ While I understand that you're not responsible for the unfortunate name of the Bro language, and while it might be that the name references an Orwellian "Big Brother" more than it refers to bro culture, I still can't get behind including or linking to a project in a language that has such a non-inclusive name. That obviously sucks for the effort you've put into this project, and sucks for Bro users that won't be able to find easily. But overall, it sucks even more for women who feel excluded from programming because of such poor naming choices. Of course, I?ll happily include your project if the programming language changes name. /quote end/ It sounded very weird for me originally. But I have googled for "bro" and "non inclusive" and found this discussion: https://news.ycombinator.com/item?id=7121268 Have anybody faced this problem before? Has it been discussed in any way in the bro community? I think it might be a good idea to have at least an article with explanation of the origins of the "bro" name and the stand towards inclusiveness. (Not sure it will help with ua-parser project, however). Thanks! P.S. Sorry for the partial mail sent a couple of minutes ago. -- WBR & WBW, Vitaly -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180309/8d7dcf92/attachment.html From jazoff at illinois.edu Fri Mar 9 06:24:17 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 9 Mar 2018 14:24:17 +0000 Subject: [Bro-Dev] Broker port status In-Reply-To: References: <4C9D8853-A1A9-4E36-B27C-78267F5229D6@illinois.edu> <2E5D9D40-FE69-4442-9FE6-9417BEB120FC@illinois.edu> Message-ID: > On Mar 8, 2018, at 6:32 PM, Jon Siwek wrote: > > Yeah, I realized that and fixed it yesterday. Have you pulled the > latest commits from the topic/actor-system branch? > > - Jon Ah.. I grabbed the latest commit from the wrong checkout and was using a commit from a few weeks ago. ? Justin Azoff From slagell at illinois.edu Fri Mar 9 08:57:28 2018 From: slagell at illinois.edu (Slagell, Adam J) Date: Fri, 9 Mar 2018 16:57:28 +0000 Subject: [Bro-Dev] "bro" name is non-inclusive? (Contribution is rejected based ) In-Reply-To: References: Message-ID: <3C2109A3-BFD1-4BBB-BD53-A2B2326A747C@illinois.edu> The project announced that it is changing its name this fall. We are still in the process of doing this. See http://blog.bro.org/2017/10/a-new-name-for-bro-project.html There was also a note in early January noting that the project was seeking outside help from a branding firm, and so there are delays to the original schedule. That was on the main Bro list. On Mar 9, 2018, at 12:03 AM, Vitaly Repin > wrote: Hello, I would like to bring your attention to one strange and extremely unexpected difficulty I faced while trying to include user agent parser for bro I have developed (plugin: https://github.com/vitalyrepin/uap-bro ) into the list of uap implementations https://github.com/ua-parser The request I have made has been rejected: https://groups.google.com/forum/#!topic/ua-parser/nJEc6SZIxyU The reason: /quote begin/ While I understand that you're not responsible for the unfortunate name of the Bro language, and while it might be that the name references an Orwellian "Big Brother" more than it refers to bro culture, I still can't get behind including or linking to a project in a language that has such a non-inclusive name. That obviously sucks for the effort you've put into this project, and sucks for Bro users that won't be able to find easily. But overall, it sucks even more for women who feel excluded from programming because of such poor naming choices. Of course, I?ll happily include your project if the programming language changes name. /quote end/ It sounded very weird for me originally. But I have googled for "bro" and "non inclusive" and found this discussion: https://news.ycombinator.com/item?id=7121268 Have anybody faced this problem before? Has it been discussed in any way in the bro community? I think it might be a good idea to have at least an article with explanation of the origins of the "bro" name and the stand towards inclusiveness. (Not sure it will help with ua-parser project, however). Thanks! P.S. Sorry for the partial mail sent a couple of minutes ago. -- WBR & WBW, Vitaly _______________________________________________ bro-dev mailing list bro-dev at bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev ------ Adam J. Slagell Director, Cybersecurity & Networking Division Chief Information Security Officer National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180309/a6896664/attachment-0001.html From jazoff at illinois.edu Fri Mar 9 09:10:15 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 9 Mar 2018 17:10:15 +0000 Subject: [Bro-Dev] Broker port status In-Reply-To: References: <4C9D8853-A1A9-4E36-B27C-78267F5229D6@illinois.edu> <2E5D9D40-FE69-4442-9FE6-9417BEB120FC@illinois.edu> Message-ID: <2FEAA79E-02E8-4F8A-9C3D-88502667690D@illinois.edu> > On Mar 9, 2018, at 8:24 AM, Azoff, Justin S wrote: > > Ah.. I grabbed the latest commit from the wrong checkout and was using a commit from a few weeks ago. > Ok.. running the latest commit now (105fc386ef0f65a91839706641abae664c7f3e49) Have noticed a problem where at the top of an hour, the logger runs into some issue while rotating the logs and all logging stops but the logger buffers everything in memory until the box OOMs: [root at bro-test ~]# broctl top manager logger proxy-{1,2,3} Name Type Host Pid Proc VSize Rss Cpu Cmd logger logger bro-test 14267 parent 5G 5G 93% bro manager manager bro-test 15478 parent 649M 277M 25% bro proxy-1 proxy bro-test 32061 parent 643M 309M 0% bro proxy-2 proxy bro-test 32063 parent 642M 351M 12% bro proxy-3 proxy bro-test 15732 parent 619M 322M 18% bro [root at bro-test ~]# cat /bro/logs/current/*.log|wc -l 2627 [root at bro-test ~]# broctl top manager logger proxy-{1,2,3} Name Type Host Pid Proc VSize Rss Cpu Cmd logger logger bro-test 14267 parent 6G 5G 66% bro manager manager bro-test 15478 parent 649M 277M 33% bro proxy-1 proxy bro-test 32061 parent 643M 309M 26% bro proxy-2 proxy bro-test 32063 parent 642M 351M 20% bro proxy-3 proxy bro-test 15732 parent 619M 322M 26% bro [root at bro-test ~]# cat /bro/logs/current/*.log|wc -l 2627 [root at bro-test ~]# broctl top manager logger proxy-{1,2,3} Name Type Host Pid Proc VSize Rss Cpu Cmd logger logger bro-test 14267 parent 7G 6G 75% bro manager manager bro-test 15478 parent 649M 277M 25% bro proxy-1 proxy bro-test 32061 parent 643M 309M 12% bro proxy-2 proxy bro-test 32063 parent 642M 353M 25% bro proxy-3 proxy bro-test 15732 parent 619M 322M 12% bro [root at bro-test ~]# cat /bro/logs/current/*.log|wc -l 2627 [root at bro-test ~]# broctl top manager logger proxy-{1,2,3} Name Type Host Pid Proc VSize Rss Cpu Cmd logger logger bro-test 14267 parent 12G 11G 68% bro manager manager bro-test 15478 parent 649M 277M 31% bro proxy-1 proxy bro-test 32061 parent 643M 313M 18% bro proxy-2 proxy bro-test 32063 parent 642M 359M 12% bro proxy-3 proxy bro-test 15732 parent 619M 322M 6% bro [root at bro-test ~]# ls -tl /bro/logs/current/|head total 412 -rw-r--r--. 1 root root 682 Mar 9 11:00 stats.log -rw-r--r--. 1 root root 14470 Mar 9 11:00 conn.log [root at bro-test ~]# date Fri Mar 9 11:06:59 CST 2018 up until log rotation is supposed to occur the log rate is normal. ? Justin Azoff From vallentin at icir.org Tue Mar 13 12:26:04 2018 From: vallentin at icir.org (Matthias Vallentin) Date: Tue, 13 Mar 2018 20:26:04 +0100 Subject: [Bro-Dev] 'async' update and proposal In-Reply-To: <20180126194003.GA1786@icir.org> References: <20180126194003.GA1786@icir.org> Message-ID: <20180313192604.GT2723@tenzir.com> >I think we should instead just skip the "async" keyword altogether. >Requiring it at some places, but not others, hurts more than it helps >in my opinion. This sounds fine to me. Given that Bro is inherently an asynchronous language, it makes sense to for those semantics to trickle down to the function call level. In my opinion, it's in line with user expectations: processing is network-event driven. A function finishes when it has all data it needs for its processing - whether being synchronous or asynchronous. On a separate note: for asynchronous operations to be truly useful, they need to propagate to the lowest level. That concerns particularly file I/O in addition to network I/O. Do you have any plans to go there as well (perhaps later down the line)? I am asking because there's a natural fit to do asynchronous file I/O with CAF. It just hasn't been tackled yet. Matthias From asharma at lbl.gov Mon Mar 19 16:11:07 2018 From: asharma at lbl.gov (Aashish Sharma) Date: Mon, 19 Mar 2018 16:11:07 -0700 Subject: [Bro-Dev] [Cron /usr/local/bin/randsleep 59 && broctl cron] Message-ID: <20180319231107.GD73235@MacPro-2331.local> So I just moved one of my boxes to bro-2.5.3 and see this report. Any ideas - ? permission issues or something else going on with broctl cron ? Aashish ----- Forwarded message from Cron Daemon ----- Date: Mon, 19 Mar 2018 16:05:38 -0700 (PDT) From: Cron Daemon To: bro Subject: Cron /usr/local/bin/randsleep 59 && broctl cron Traceback (most recent call last): File "/usr/local/bro/bin/broctl", line 833, in sys.exit(main()) File "/usr/local/bro/bin/broctl", line 809, in main cmdsuccess = loop.onecmd(cmdline) File "/usr/local/lib/python2.7/cmd.py", line 221, in onecmd return func(arg) File "/usr/local/bro/bin/broctl", line 388, in do_cron self.broctl.cron(watch) File "/usr/local/bro-2.5.3/lib/broctl/BroControl/broctl.py", line 44, in wrapper return func(self, *args, **kwargs) File "/usr/local/bro-2.5.3/lib/broctl/BroControl/broctl.py", line 353, in cron self.controller.cron(watch) File "/usr/local/bro-2.5.3/lib/broctl/BroControl/control.py", line 1451, in cron tasks.log_stats(5) File "/usr/local/bro-2.5.3/lib/broctl/BroControl/cron.py", line 81, in log_stats if self.config.mailreceivingpackets: File "/usr/local/bro-2.5.3/lib/broctl/BroControl/config.py", line 236, in __getattr__ raise AttributeError("unknown config attribute %s" % attr) AttributeError: unknown config attribute mailreceivingpackets ----- End forwarded message ----- From asharma at lbl.gov Mon Mar 19 16:33:32 2018 From: asharma at lbl.gov (Aashish Sharma) Date: Mon, 19 Mar 2018 16:33:32 -0700 Subject: [Bro-Dev] [Cron /usr/local/bin/randsleep 59 && broctl cron] In-Reply-To: <20180319231107.GD73235@MacPro-2331.local> References: <20180319231107.GD73235@MacPro-2331.local> Message-ID: Nevermind! Please ignore! Issue resolved - it was a mistake on my end! Aashish On Mon, Mar 19, 2018 at 4:11 PM, Aashish Sharma wrote: > So I just moved one of my boxes to bro-2.5.3 and see this report. > > Any ideas - ? permission issues or something else going on with broctl cron ? > > Aashish > > ----- Forwarded message from Cron Daemon ----- > > Date: Mon, 19 Mar 2018 16:05:38 -0700 (PDT) > From: Cron Daemon > To: bro > Subject: Cron /usr/local/bin/randsleep 59 && broctl cron > > Traceback (most recent call last): > File "/usr/local/bro/bin/broctl", line 833, in > sys.exit(main()) > File "/usr/local/bro/bin/broctl", line 809, in main > cmdsuccess = loop.onecmd(cmdline) > File "/usr/local/lib/python2.7/cmd.py", line 221, in onecmd > return func(arg) > File "/usr/local/bro/bin/broctl", line 388, in do_cron > self.broctl.cron(watch) > File "/usr/local/bro-2.5.3/lib/broctl/BroControl/broctl.py", line 44, in wrapper > return func(self, *args, **kwargs) > File "/usr/local/bro-2.5.3/lib/broctl/BroControl/broctl.py", line 353, in cron > self.controller.cron(watch) > File "/usr/local/bro-2.5.3/lib/broctl/BroControl/control.py", line 1451, in cron > tasks.log_stats(5) > File "/usr/local/bro-2.5.3/lib/broctl/BroControl/cron.py", line 81, in log_stats > if self.config.mailreceivingpackets: > File "/usr/local/bro-2.5.3/lib/broctl/BroControl/config.py", line 236, in __getattr__ > raise AttributeError("unknown config attribute %s" % attr) > AttributeError: unknown config attribute mailreceivingpackets > > ----- End forwarded message ----- From p.antoine at catenacyber.fr Wed Mar 21 07:52:00 2018 From: p.antoine at catenacyber.fr (Philippe Antoine) Date: Wed, 21 Mar 2018 15:52:00 +0100 Subject: [Bro-Dev] string without NUL terminator In-Reply-To: References: <20180319231107.GD73235@MacPro-2331.local> Message-ID: Hi everyone, I have been fuzzing some HTTP requests while running Bro. Is it "normal" that I get the following line in reporter.log : 1521641846.381768 Reporter::ERROR string without NUL terminator: "Analyzer::ANALYZER_HTTPF127.0.0.1127.0.0.1:8002/tcp/htdocs/wp-admin/css/install.css?ver=../../../../../../../../..B94967296F../boot.ini\\x00pix.gix" (empty) Or is it a sign that something is not correct ? Best regards, Philippe -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180321/e5de6c51/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180321/e5de6c51/attachment.bin From jazoff at illinois.edu Mon Mar 26 08:50:47 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 26 Mar 2018 15:50:47 +0000 Subject: [Bro-Dev] Configuration framework on a cluster? Message-ID: How is the Configuration framework intended to be used on a cluster? It has code that does if ( Cluster::is_enabled() && Cluster::local_node_type() != Cluster::MANAGER ) return; to set the input files to be only read on the manager, and appears to not allow you to use &synchronized on options. How do you update an 'option' on the workers? ? Justin Azoff From johanna at corelight.com Mon Mar 26 10:59:58 2018 From: johanna at corelight.com (Johanna Amann) Date: Mon, 26 Mar 2018 10:59:58 -0700 Subject: [Bro-Dev] Configuration framework on a cluster? In-Reply-To: References: Message-ID: <2B157028-74A5-468A-94FA-665B10F7C94A@corelight.com> Hi Justin, > How is the Configuration framework intended to be used on a cluster? It is intended to read the configuration on the manager node; events then should be used internally to distribute the data to all other nodes. And you are right - that is something that is completely missing at the moment. Which is an oversight - I completely forgot to write this last part of the config framework. I am going to update the scripts and submit a merge request for them in the next few days. Johanna