From brot212 at googlemail.com Thu Apr 12 12:23:21 2018 From: brot212 at googlemail.com (DW) Date: Thu, 12 Apr 2018 21:23:21 +0200 Subject: [Bro-Dev] Overload Bro Events Message-ID: Hello, it is possible to overload events in Bro based on the event-parameter and trigger the "right" event based on the given parameter? E.g. I would define events like this event overload%(c: connection%); event overload%(c: connection, h: header%); event overload%(c: connection, h: header, d: data%); ... In the protocol parser source: ... EventHandlerPtr ev = overload; val_list* vl = new val_list(); switch(type) { ??? case 1: ??? ??? vl->append(BuildConnVal()); ??? case 2: ??? ??? vl->append(BuildConnVal()); ??? ??? vl->append(header); ??? case 3: ??? ??? vl->append(BuildConnVal()); ??? ??? vl->append(header); ??? ??? vl->append(data); } ConnectionEvent(ev, vl); Dane From jsiwek at corelight.com Thu Apr 12 12:44:07 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 12 Apr 2018 14:44:07 -0500 Subject: [Bro-Dev] Overload Bro Events In-Reply-To: References: Message-ID: <719ca548-1396-7e38-96b1-f945feb20777@corelight.com> On 4/12/18 2:23 PM, DW wrote: > Hello, > > it is possible to overload events in Bro based on the event-parameter > and trigger the "right" event based on the given parameter? > > E.g. I would define events like this > > event overload%(c: connection%); > event overload%(c: connection, h: header%); > event overload%(c: connection, h: header, d: data%); Overloading is not supported for functions in general (function/event/hook). - Jon From robin at icir.org Thu Apr 12 14:33:05 2018 From: robin at icir.org (Robin Sommer) Date: Thu, 12 Apr 2018 14:33:05 -0700 Subject: [Bro-Dev] Overload Bro Events In-Reply-To: <719ca548-1396-7e38-96b1-f945feb20777@corelight.com> References: <719ca548-1396-7e38-96b1-f945feb20777@corelight.com> Message-ID: <20180412213305.GM24967@icir.org> On Thu, Apr 12, 2018 at 14:44 -0500, you wrote: > > event overload%(c: connection%); > > event overload%(c: connection, h: header%); > > event overload%(c: connection, h: header, d: data%); > > Overloading is not supported for functions in general (function/event/hook). This has interesting implication for BIT-1431: if overloading worked work, that could take the place of the &default attribute suggested there. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From asharma at lbl.gov Thu Apr 12 21:30:12 2018 From: asharma at lbl.gov (Aashish Sharma) Date: Thu, 12 Apr 2018 21:30:12 -0700 Subject: [Bro-Dev] timer delays between different events for same connection Message-ID: <20180413043011.GV27389@MacPro-2331.local> So I am seeing some weird stuff in my sample pcap of scanners. May be too obvious and I am just not seeing why/how of it. Here is the issue : ( I have time in human format for easier read): SO I just pick one session from conn.log and this is the connection in question: (there are many more like this): $ fgrep CspAa42NoEGEaXK4ci conn.log | cf Apr 12 05:37:42 CspAa42NoEGEaXK4ci 191.254.157.138 45107 128.3.97.204 23 tcp - - - - S0 F T 0 S 1 40 0 0 - Now as part of debugging I have dumped network_time for various events which process this connection: Apr 12 05:37:42 new_connection CspAa42NoEGEaXK4ci Apr 12 06:13:48 connection_attempt CspAa42NoEGEaXK4ci Apr 12 06:13:48 connection_state_remove CspAa42NoEGEaXK4ci Now my understanding is there are various timers involved upon whose expirations bro infers events such as connection_attempt, connection_reset etc etc. Timers such as tcp_attempt_delay, tcp_SYN_timeout, tcp_close_delay amongst others. But all these timers are generally 5 seconds. Q. Why would connection_attempt event kick in after 36 minutes and 6 seconds ? ( 06:13:48 - 05:37:42 ) ? I have a pcap to share if anyone is interested and replicate on their end. Aashish From seth at corelight.com Fri Apr 13 04:46:33 2018 From: seth at corelight.com (Seth Hall) Date: Fri, 13 Apr 2018 07:46:33 -0400 Subject: [Bro-Dev] timer delays between different events for same connection In-Reply-To: <20180413043011.GV27389@MacPro-2331.local> References: <20180413043011.GV27389@MacPro-2331.local> Message-ID: On 13 Apr 2018, at 0:30, Aashish Sharma wrote: > So I am seeing some weird stuff in my sample pcap of scanners. May be > too > obvious and I am just not seeing why/how of it. It's a straight forward answer but not completely obvious. :) > Q. Why would connection_attempt event kick in after 36 minutes and 6 > seconds ? ( > 06:13:48 - 05:37:42 ) ? I bet that you have a jump in timestamps in your pcap. Since Bro's internal clock is driven forward by seeing timestamps associated with packets it's possible that your pcap has a 36 minute jump in timestamps so Bro couldn't have possibly expired anything in the time before that because from its perspective there was an immediate jump in time. You don't normally experience the effects of this behavior in traffic you're sniffing live because you will tend to have many packets every second so you see Bro's clock driven forward in very tiny increments as you would expect. If you go a long time without receiving a packet is when stuff gets tricky. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From asharma at lbl.gov Fri Apr 13 07:07:01 2018 From: asharma at lbl.gov (Aashish Sharma) Date: Fri, 13 Apr 2018 07:07:01 -0700 Subject: [Bro-Dev] timer delays between different events for same connection In-Reply-To: References: <20180413043011.GV27389@MacPro-2331.local> Message-ID: <20180413140700.GW27389@MacPro-2331.local> Ah! it is obvious now! Yep that was it. Its a relatively slow scan and I only extracted all activity for this specific source IP out of timemachine. I didn't see this behavior in other test cases because there I pull scanners based on ports so somewhat of more 'fluid' traffic. Thanks, Aashish On Fri, Apr 13, 2018 at 07:46:33AM -0400, Seth Hall wrote: > > > On 13 Apr 2018, at 0:30, Aashish Sharma wrote: > > > So I am seeing some weird stuff in my sample pcap of scanners. May be > > too > > obvious and I am just not seeing why/how of it. > > It's a straight forward answer but not completely obvious. :) > > > Q. Why would connection_attempt event kick in after 36 minutes and 6 > > seconds ? ( > > 06:13:48 - 05:37:42 ) ? > > I bet that you have a jump in timestamps in your pcap. Since Bro's internal > clock is driven forward by seeing timestamps associated with packets it's > possible that your pcap has a 36 minute jump in timestamps so Bro couldn't > have possibly expired anything in the time before that because from its > perspective there was an immediate jump in time. You don't normally > experience the effects of this behavior in traffic you're sniffing live > because you will tend to have many packets every second so you see Bro's > clock driven forward in very tiny increments as you would expect. If you go > a long time without receiving a packet is when stuff gets tricky. > > .Seth > > -- > Seth Hall * Corelight, Inc * www.corelight.com From asharma at lbl.gov Fri Apr 13 16:14:52 2018 From: asharma at lbl.gov (Aashish Sharma) Date: Fri, 13 Apr 2018 16:14:52 -0700 Subject: [Bro-Dev] scheduling events vs using &expire_func ? Message-ID: <20180413231451.GG35928@MacPro-2331.local> I have a aggregation policy where I am trying to keep counts of number of connections an IP made in a cluster setup. For now, I am using table on workers and manager and using expire_func to trigger worker2manager and manager2worker events. All works great until tables grow to > 1 million after which expire_functions start clogging on manager and slowing down. Example of Timer from prof.log on manager: 1523636760.591416 Timers: current=57509 max=68053 mem=4942K lag=0.44s 1523636943.983521 Timers: current=54653 max=68053 mem=4696K lag=168.39s 1523638289.808519 Timers: current=49623 max=68053 mem=4264K lag=1330.82s 1523638364.873338 Timers: current=48441 max=68053 mem=4162K lag=60.06s 1523638380.344700 Timers: current=50841 max=68053 mem=4369K lag=0.47s So Instead of using &expire_func, I can probably try schedule {} ; but I am not sure how scheduling events are any different internally then scheduling expire_funcs ? I'd like to think/guess that scheduling events is probably less taxing. but wanted to check with the greater group on thoughts - esp insights into their internal processing queues. Thanks, Aashish From jsiwek at corelight.com Mon Apr 16 08:32:45 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Mon, 16 Apr 2018 10:32:45 -0500 Subject: [Bro-Dev] scheduling events vs using &expire_func ? In-Reply-To: <20180413231451.GG35928@MacPro-2331.local> References: <20180413231451.GG35928@MacPro-2331.local> Message-ID: <4a1568e4-a48c-8159-3435-79a0d8d58f75@corelight.com> On 4/13/18 6:14 PM, Aashish Sharma wrote: > I have a aggregation policy where I am trying to keep counts of number of > connections an IP made in a cluster setup. > > For now, I am using table on workers and manager and using expire_func to > trigger worker2manager and manager2worker events. > > All works great until tables grow to > 1 million after which expire_functions > start clogging on manager and slowing down. > > Example of Timer from prof.log on manager: > > 1523636760.591416 Timers: current=57509 max=68053 mem=4942K lag=0.44s > 1523636943.983521 Timers: current=54653 max=68053 mem=4696K lag=168.39s > 1523638289.808519 Timers: current=49623 max=68053 mem=4264K lag=1330.82s > 1523638364.873338 Timers: current=48441 max=68053 mem=4162K lag=60.06s > 1523638380.344700 Timers: current=50841 max=68053 mem=4369K lag=0.47s > > So Instead of using &expire_func, I can probably try schedule {} ; but I am not > sure how scheduling events are any different internally then scheduling > expire_funcs ? There's a single timer per table that continuously triggers incremental iteration over fixed-size chunks of the table, looking for entries to expire. The relevant options that you can tune here: * `table_expire_interval` * `table_incremental_step` * `table_expire_delay` > I'd like to think/guess that scheduling events is probably less taxing. but > wanted to check with the greater group on thoughts - esp insights into their > internal processing queues. I'm not clear on exactly how your code would be restructured around scheduled events, though guessing if you just did one event per entry that needs to be expired, it's not going to be better. You would then have one timer per table entry (up from a single timer), or possibly more depending on expiration scheme (e.g. if it's expiring on something other than create times, you're going to need a way to invalidate previously scheduled events). Ultimately, you'd likely still have the same amount of equivalent function calls (whatever work you're doing in &expire_func, would still need to happen). With the way table expiration is implemented, my guess is that the actual work required to call and evaluate the &expire_func code becomes too great at some point, so maybe first try decreasing `table_incremental_step` or reducing the work that you need to do in the &expire_func. With new features in the upcoming broker-enabled cluster framework (soon to be merged into git/master), I'd suggest a different way to think about structuring the problem: you could Rendezvous Hash the IP addresses across proxies, with each one managing expiration in just their own table. In that way, the storage/computation can be uniformly distributed and you should be able to simply adjust number of proxies to fit the required scale. - Jon From asharma at lbl.gov Tue Apr 17 16:04:51 2018 From: asharma at lbl.gov (Aashish Sharma) Date: Tue, 17 Apr 2018 16:04:51 -0700 Subject: [Bro-Dev] scheduling events vs using &expire_func ? In-Reply-To: <4a1568e4-a48c-8159-3435-79a0d8d58f75@corelight.com> References: <20180413231451.GG35928@MacPro-2331.local> <4a1568e4-a48c-8159-3435-79a0d8d58f75@corelight.com> Message-ID: <20180417230450.GC61317@MacPro-2331.local> For now, I am resorting to &expire_func route only. I think by using some more heuristics in worker's expire functions for more aggregated stats, I am able to shed load on manager where manager doesn't need to track ALL potential scanners. Lets see, I am running to see if new code works without exhausting memory for few days. Yes certainly, the following changed did address the manager network_time() stall issues: redef table_expire_interval = 0.1 secs ; redef table_incremental_step=25 ; Useful observation: if you want to expire a lot of entires from a table/set, expire few but expire often. I Still need to determine limits of both table_incremental_step, table_expire_interval and this works for million or million(s) of entires. > on expiration scheme (e.g. if it's expiring on something other than create > times, you're going to need a way to invalidate previously scheduled > events). Actually, in this case - I was more thinking in terms of let scheduled event kick in and in that event I decide to further schedule a next one or delete the entry from the table. > actual work required to call and evaluate the &expire_func code becomes too > great at some point, so maybe first try decreasing `table_incremental_step` Yes, It seems like that. I still don't know at what point. In previous runs it appears after table had 1.7-2.3 Million entires. But then I don't think its function of counts, but how much RAM i've got on the system. Somewhere in the range is when manager ran out of memory. HOwever (as stated above), I was able to come up with a little heuristics which still allows me to keep track of really slow scanners, while not burdening manager but rather let load be on workers. Simple observation that really slow scanners aren't going to have a lot of connections allows to keep those in (few) worker table. This would potentially be a problem if there really a LOT of very slow scanners. but, still, those all get divided by number of workers we run. > I'd suggest a different way to think about > structuring the problem: you could Rendezvous Hash the IP addresses across > proxies, with each one managing expiration in just their own table. In that > way, the storage/computation can be uniformly distributed and you should be > able to simply adjust number of proxies to fit the required scale. I think above might work reasonable. So previously I was making manager keep count of potential scanners but now moving that work instead to workers. New model would let us just move all this to proxy(ies) and proxies can decide if delete or send to manager for aggregation. I suppose, given proxies don't process packets, it will be cheaper there to do all this work. Only thing bothers me is scan-detection is a complicated problem only because of distribution of data in cluster. Its a lot simple problem if we could just do a tail -f conn.log | ./python-script So yes we can shed load from manger -> workers -> proxies. I'll try this approach. But I think I am also going to try (with new broker-enabled cluster) approach of sending all connections to one proxy/data-store and just do aggregation there and see if that works out (the tail -f conn.log | 'python-script' approach). Admittedly, this needs more thinking to get the right architecture in the new cluster era! Thanks, Aashish On Mon, Apr 16, 2018 at 10:32:45AM -0500, Jon Siwek wrote: > > > On 4/13/18 6:14 PM, Aashish Sharma wrote: > > I have a aggregation policy where I am trying to keep counts of number of > > connections an IP made in a cluster setup. > > > > For now, I am using table on workers and manager and using expire_func to > > trigger worker2manager and manager2worker events. > > > > All works great until tables grow to > 1 million after which expire_functions > > start clogging on manager and slowing down. > > > > Example of Timer from prof.log on manager: > > > > 1523636760.591416 Timers: current=57509 max=68053 mem=4942K lag=0.44s > > 1523636943.983521 Timers: current=54653 max=68053 mem=4696K lag=168.39s > > 1523638289.808519 Timers: current=49623 max=68053 mem=4264K lag=1330.82s > > 1523638364.873338 Timers: current=48441 max=68053 mem=4162K lag=60.06s > > 1523638380.344700 Timers: current=50841 max=68053 mem=4369K lag=0.47s > > > > So Instead of using &expire_func, I can probably try schedule {} ; but I am not > > sure how scheduling events are any different internally then scheduling > > expire_funcs ? > > There's a single timer per table that continuously triggers incremental > iteration over fixed-size chunks of the table, looking for entries to > expire. The relevant options that you can tune here: > > * `table_expire_interval` > * `table_incremental_step` > * `table_expire_delay` > > > I'd like to think/guess that scheduling events is probably less taxing. but > > wanted to check with the greater group on thoughts - esp insights into their > > internal processing queues. > > I'm not clear on exactly how your code would be restructured around > scheduled events, though guessing if you just did one event per entry that > needs to be expired, it's not going to be better. You would then have one > timer per table entry (up from a single timer), or possibly more depending > on expiration scheme (e.g. if it's expiring on something other than create > times, you're going to need a way to invalidate previously scheduled > events). > > Ultimately, you'd likely still have the same amount of equivalent function > calls (whatever work you're doing in &expire_func, would still need to > happen). With the way table expiration is implemented, my guess is that the > actual work required to call and evaluate the &expire_func code becomes too > great at some point, so maybe first try decreasing `table_incremental_step` > or reducing the work that you need to do in the &expire_func. > > With new features in the upcoming broker-enabled cluster framework (soon to > be merged into git/master), I'd suggest a different way to think about > structuring the problem: you could Rendezvous Hash the IP addresses across > proxies, with each one managing expiration in just their own table. In that > way, the storage/computation can be uniformly distributed and you should be > able to simply adjust number of proxies to fit the required scale. > > - Jon From jazoff at illinois.edu Wed Apr 18 06:46:08 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 18 Apr 2018 13:46:08 +0000 Subject: [Bro-Dev] scheduling events vs using &expire_func ? In-Reply-To: <20180417230450.GC61317@MacPro-2331.local> References: <20180413231451.GG35928@MacPro-2331.local> <4a1568e4-a48c-8159-3435-79a0d8d58f75@corelight.com> <20180417230450.GC61317@MacPro-2331.local> Message-ID: > On Apr 17, 2018, at 4:04 PM, Aashish Sharma wrote: > > For now, I am resorting to &expire_func route only. I think by using some more > heuristics in worker's expire functions for more aggregated stats, I am able to > shed load on manager where manager doesn't need to track ALL potential scanners. > > Lets see, I am running to see if new code works without exhausting memory for few days. > > Yes certainly, the following changed did address the manager network_time() > stall issues: > > redef table_expire_interval = 0.1 secs ; > redef table_incremental_step=25 ; > > Useful observation: if you want to expire a lot of entires from a table/set, > expire few but expire often. > > I Still need to determine limits of both table_incremental_step, > table_expire_interval and this works for million or million(s) of entires. That should probably work unless you are adding new table entries at a rate higher than 250/second. You may need to tweak that so interval x step is at least the rate of new entries. > > Yes, It seems like that. I still don't know at what point. In previous runs it > appears after table had 1.7-2.3 Million entires. But then I don't think its > function of counts, but how much RAM i've got on the system. Somewhere in the > range is when manager ran out of memory. HOwever (as stated above), I was able > to come up with a little heuristics which still allows me to keep track of > really slow scanners, while not burdening manager but rather let load be on > workers. Simple observation that really slow scanners aren't going to have a lot > of connections allows to keep those in (few) worker table. This would > potentially be a problem if there really a LOT of very slow scanners. but, > still, those all get divided by number of workers we run. How are you tracking slow scanners on the workers? If you have 50 workers and you are not distributing the data between them, there's only a 1 in 50 chance that you'll see the same scanner twice on the same worker, and a one in 2500 that you'd see 3 packets in a row on the same worker... and 1:125,000 for 4 in a row. >> I'd suggest a different way to think about >> structuring the problem: you could Rendezvous Hash the IP addresses across >> proxies, with each one managing expiration in just their own table. In that >> way, the storage/computation can be uniformly distributed and you should be >> able to simply adjust number of proxies to fit the required scale. > > I think above might work reasonable. It does, it works great. > So previously I was making manager keep count of potential scanners but now > moving that work instead to workers. New model would let us just move all this > to proxy(ies) and proxies can decide if delete or send to manager for > aggregation. There's no need, you just aggregate it on the proxies, the manager never sees anything. > I suppose, given proxies don't process packets, it will be cheaper there to do > all this work. > > Only thing bothers me is scan-detection is a complicated problem only because of > distribution of data in cluster. Its a lot simple problem if we could just do a > tail -f conn.log | ./python-script That doesn't simplify anything, that just moves the problem. You can only tail the single conn.log because the logger aggregated the records from all the workers. If the manager is running out of memory tracking all of the scanners, then the single instance of the python script is going to run into the same issue at some point. > So yes we can shed load from manger -> workers -> proxies. I'll try this > approach. But I think I am also going to try (with new broker-enabled cluster) > approach of sending all connections to one proxy/data-store and just do > aggregation there and see if that works out (the tail -f conn.log | > 'python-script' approach). Admittedly, this needs more thinking to get the right > architecture in the new cluster era! No.. this is just moving the problem again. If your manager is running out of memory and you move everything to one proxy, that's just going to have the same problem. The fix is to use the distributing message routing features that I've been talking about for a while (and that Jon implemented in the actor-system branch!) The entire change to switch simple-scan from aggregating all scanners on a single manager to aggregating scanners across all proxies (which can be on multiple machines) is swapping event Scan::scan_attempt(scanner, attempt); with Cluster::publish_hrw(Cluster::proxy_pool, scanner, Scan::scan_attempt, scanner, attempt); (with some @ifdefs to make it work on both versions of bro) ? Justin Azoff From asharma at lbl.gov Thu Apr 19 07:03:52 2018 From: asharma at lbl.gov (Aashish Sharma) Date: Thu, 19 Apr 2018 07:03:52 -0700 Subject: [Bro-Dev] scheduling events vs using &expire_func ? In-Reply-To: References: <20180413231451.GG35928@MacPro-2331.local> <4a1568e4-a48c-8159-3435-79a0d8d58f75@corelight.com> <20180417230450.GC61317@MacPro-2331.local> Message-ID: <20180419140342.GX67233@MacPro-2331.local> Justin, On Wed, Apr 18, 2018 at 01:46:08PM +0000, Azoff, Justin S wrote: > How are you tracking slow scanners on the workers? If you have 50 workers and you > are not distributing the data between them, there's only a 1 in 50 chance that you'll > see the same scanner twice on the same worker, and a one in 2500 that you'd see > 3 packets in a row on the same worker... and 1:125,000 for 4 in a row. Yes, that was the observation and idea. If real slow scanners, (won't be way too many way too soon (obviously)) let each one of their start time be tracked on workers. ODDS of hitting same worker are too low, so burden of tracking start time for 1000's of slow scanner is distributed fairly even across 10/50 workers *instead* of manager having to store all this. So 600K slow scanners means |manager_table| +=600K vs |worker_table| = 600K/50 so burden of memory is more distributed. I checked some numbers on my end = since midnight we flagged 172K scanners while tracking 630K potential scanners which will eventually be flagged. Issue isn't flagging these. Issue was being accurate on when was the very first time we saw a IP connect to us and keep that in memory - this is not needed but good to have stat. > >> I'd suggest a different way to think about > >> structuring the problem: you could Rendezvous Hash the IP addresses across > >> proxies, with each one managing expiration in just their own table. In that > >> way, the storage/computation can be uniformly distributed and you should be > >> able to simply adjust number of proxies to fit the required scale. > > > That doesn't simplify anything, that just moves the problem. You can only tail the single > conn.log because the logger aggregated the records from all the workers. If the manager > is running out of memory tracking all of the scanners, then the single instance of the python > script is going to run into the same issue at some point. I agree, but we digress on the actual issue here. see below. > > So yes we can shed load from manger -> workers -> proxies. I'll try this > > approach. But I think I am also going to try (with new broker-enabled cluster) > > approach of sending all connections to one proxy/data-store and just do > > aggregation there and see if that works out (the tail -f conn.log | > > 'python-script' approach). Admittedly, this needs more thinking to get the right > > architecture in the new cluster era! > > No.. this is just moving the problem again. If your manager is running out of memory and you > move everything to one proxy, that's just going to have the same problem. I think we've talked about this roughly over 2 years. I am probably mis-understanding or may be unclear. the issue is complexity of aggregation due to clusterization in scan-detection. now you can use many proxies, many data nodes etc but as long as distributed nature of data is there, aggregation in realtime is problem. Data needs to be concentrated at one place. A tail -f conn.log is data concentrated at one place. Now its a different issue that conn.log entry is 5+ seconds late which can already miss a significant scan etc. > The fix is to use the distributing message routing features that I've been talking about for a while > (and that Jon implemented in the actor-system branch!) > > The entire change to switch simple-scan from aggregating all scanners on a single manager to > aggregating scanners across all proxies (which can be on multiple machines) is swapping aggregating across all proxies is still distributing data around. So the way I see is you are moving the problem around :) But as I said, I don't know more how this works since I haven't tried new broker stuff just yet. > event Scan::scan_attempt(scanner, attempt); > with > Cluster::publish_hrw(Cluster::proxy_pool, scanner, Scan::scan_attempt, scanner, attempt); > (with some @ifdefs to make it work on both versions of bro) I am *really* looking forward to trying this stuff out in new broker model. Aashish From asharma at lbl.gov Thu Apr 19 07:08:56 2018 From: asharma at lbl.gov (Aashish Sharma) Date: Thu, 19 Apr 2018 07:08:56 -0700 Subject: [Bro-Dev] scheduling events vs using &expire_func ? In-Reply-To: References: <20180413231451.GG35928@MacPro-2331.local> <4a1568e4-a48c-8159-3435-79a0d8d58f75@corelight.com> <20180417230450.GC61317@MacPro-2331.local> Message-ID: <20180419140855.GY67233@MacPro-2331.local> > conn.log because the logger aggregated the records from all the workers. If the manager > is running out of memory tracking all of the scanners, then the single instance of the python > script is going to run into the same issue at some point. Oh totally forgot to add an important point. Issue isn't memory to begin with. Issue was 2Million entires in a table can result in expire_function to be slow clogging the event queue resulting in stall network_time and manager lagging significantly behind. Holding potential scanners over workers helps me break manager table down to much smaller size across workers. > How are you tracking slow scanners on the workers? If you have 50 workers and you > are not distributing the data between them, there's only a 1 in 50 chance that you'll Exactly, thats what keeps worker table sizes small. until there are enough connections to flag something as scanner. Note: worker_tables only keep IP->start_time They report to manager a potential scanner but manager doesn't need to keep that in table. I think I use combination of bloomfilter and hyperloglog there to scale to Millions easily. Note2: this is to generate scan_summary and not a scanner. this thing: #fields ts scanner state detection start_ts end_ts detect_ts detect_latency total_conn total_hosts_scanned duration scan_rate country_code region city distance event_peer #types time addr enum string time time time interval count count interval double string string string double string 1522090899.384438 122.224.112.162 Scan::DETECT LandMine 1522090219.816163 1522090744.317532 1522090744.317532 524.501369 19 19 524.501369 27.605335 CN 02 Hangzhou 6243.450744 bro 1522094550.487969 122.224.112.162 Scan::UPDATE LandMine 1522090219.816163 1522094128.634672 1522090744.317532 524.501369 110 109 3908.818509 35.86072 CN 02 Hangzhou 6243.450744 bro 1522098156.871486 122.224.112.162 Scan::UPDATE LandMine 1522090219.816163 1522097984.861365 1522090744.317532 524.501369 225 227 7765.045202 34.207248 CN 02 Hangzhou 6243.450744 bro 1522101784.996068 122.224.112.162 Scan::UPDATE LandMine 1522090219.816163 1522101081.946002 1522090744.317532 524.501369 359 365 10862.129839 29.75926 CN 02 Hangzhou 6243.450744 bro 1522354414.224635 122.224.112.162 Scan::FINISH LandMine 1522090219.816163 1522103520.055414 1522090744.317532 524.501369 488 507 13300.239251 26.233214 CN 02 Hangzhou 6243.450744 bro Aashish On Wed, Apr 18, 2018 at 01:46:08PM +0000, Azoff, Justin S wrote: > > On Apr 17, 2018, at 4:04 PM, Aashish Sharma wrote: > > > > For now, I am resorting to &expire_func route only. I think by using some more > > heuristics in worker's expire functions for more aggregated stats, I am able to > > shed load on manager where manager doesn't need to track ALL potential scanners. > > > > Lets see, I am running to see if new code works without exhausting memory for few days. > > > > Yes certainly, the following changed did address the manager network_time() > > stall issues: > > > > redef table_expire_interval = 0.1 secs ; > > redef table_incremental_step=25 ; > > > > Useful observation: if you want to expire a lot of entires from a table/set, > > expire few but expire often. > > > > I Still need to determine limits of both table_incremental_step, > > table_expire_interval and this works for million or million(s) of entires. > > That should probably work unless you are adding new table entries at a rate higher than 250/second. > You may need to tweak that so interval x step is at least the rate of new entries. > > > > > Yes, It seems like that. I still don't know at what point. In previous runs it > > appears after table had 1.7-2.3 Million entires. But then I don't think its > > function of counts, but how much RAM i've got on the system. Somewhere in the > > range is when manager ran out of memory. HOwever (as stated above), I was able > > to come up with a little heuristics which still allows me to keep track of > > really slow scanners, while not burdening manager but rather let load be on > > workers. Simple observation that really slow scanners aren't going to have a lot > > of connections allows to keep those in (few) worker table. This would > > potentially be a problem if there really a LOT of very slow scanners. but, > > still, those all get divided by number of workers we run. > > How are you tracking slow scanners on the workers? If you have 50 workers and you > are not distributing the data between them, there's only a 1 in 50 chance that you'll > see the same scanner twice on the same worker, and a one in 2500 that you'd see > 3 packets in a row on the same worker... and 1:125,000 for 4 in a row. > > >> I'd suggest a different way to think about > >> structuring the problem: you could Rendezvous Hash the IP addresses across > >> proxies, with each one managing expiration in just their own table. In that > >> way, the storage/computation can be uniformly distributed and you should be > >> able to simply adjust number of proxies to fit the required scale. > > > > I think above might work reasonable. > > It does, it works great. > > > So previously I was making manager keep count of potential scanners but now > > moving that work instead to workers. New model would let us just move all this > > to proxy(ies) and proxies can decide if delete or send to manager for > > aggregation. > > There's no need, you just aggregate it on the proxies, the manager never sees anything. > > > I suppose, given proxies don't process packets, it will be cheaper there to do > > all this work. > > > > Only thing bothers me is scan-detection is a complicated problem only because of > > distribution of data in cluster. Its a lot simple problem if we could just do a > > tail -f conn.log | ./python-script > > That doesn't simplify anything, that just moves the problem. You can only tail the single > conn.log because the logger aggregated the records from all the workers. If the manager > is running out of memory tracking all of the scanners, then the single instance of the python > script is going to run into the same issue at some point. > > > So yes we can shed load from manger -> workers -> proxies. I'll try this > > approach. But I think I am also going to try (with new broker-enabled cluster) > > approach of sending all connections to one proxy/data-store and just do > > aggregation there and see if that works out (the tail -f conn.log | > > 'python-script' approach). Admittedly, this needs more thinking to get the right > > architecture in the new cluster era! > > No.. this is just moving the problem again. If your manager is running out of memory and you > move everything to one proxy, that's just going to have the same problem. > > The fix is to use the distributing message routing features that I've been talking about for a while > (and that Jon implemented in the actor-system branch!) > > The entire change to switch simple-scan from aggregating all scanners on a single manager to > aggregating scanners across all proxies (which can be on multiple machines) is swapping > > event Scan::scan_attempt(scanner, attempt); > > with > > Cluster::publish_hrw(Cluster::proxy_pool, scanner, Scan::scan_attempt, scanner, attempt); > > (with some @ifdefs to make it work on both versions of bro) > > > ? > Justin Azoff > > From jazoff at illinois.edu Thu Apr 19 07:29:31 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 19 Apr 2018 14:29:31 +0000 Subject: [Bro-Dev] scheduling events vs using &expire_func ? In-Reply-To: <20180419140342.GX67233@MacPro-2331.local> References: <20180413231451.GG35928@MacPro-2331.local> <4a1568e4-a48c-8159-3435-79a0d8d58f75@corelight.com> <20180417230450.GC61317@MacPro-2331.local> <20180419140342.GX67233@MacPro-2331.local> Message-ID: <74CAA9FE-D644-4D1B-80FC-0657D31F37FF@illinois.edu> > On Apr 19, 2018, at 7:03 AM, Aashish Sharma wrote: > > aggregating across all proxies is still distributing data around. So the way I > see is you are moving the problem around :) But as I said, I don't know more how > this works since I haven't tried new broker stuff just yet. I think the part that you are missing is that the data is PARTITIONED across the proxies. So say you saw a few scan attempts: attacker.1 -> local.1:22 attacker.2 -> local.1:22 attacker.1 -> local.2:22 attacker.3 -> local.1:80 attacker.4 -> local.1:23 attacker.2 -> local.3:80 attacker.1 -> local.3:22 To do scan detection they need to be grouped into a table like this: attacker.1 -> [local.1:22, local.2:22, local.3:22] attacker.2 -> [local.1:22, local.3:80] attacker.3 -> [local.1:80] attacker.4 -> [local1:23] This is how scan-ng, simple-scan work now. Currently this needs to be aggregated in a single place (the manager) but with publish_hrw, bro will do something like this: hash(attacker.1) == proxy-1 hash(attacker.2) == proxy-2 hash(attacker.3) == proxy-2 hash(attacker.4) == proxy-1 Which will distribute the data across the proxies and you will end up with: Data stored on proxy-1: attacker.1 -> [local.1:22, local.2:22, local.3:22] attacker.4 -> [local1:23] Data stored on proxy-2: attacker.2 -> [local.1:22, local.3:80] attacker.3 -> [local.1:80] Each proxy will have a consistent view of a single scanner but not need to store all data for all scanners. ? Justin Azoff From johanna at icir.org Thu Apr 19 11:41:33 2018 From: johanna at icir.org (Johanna Amann) Date: Thu, 19 Apr 2018 11:41:33 -0700 Subject: [Bro-Dev] Bro git/webserver updates Message-ID: <20180419184133.jr7hjh7rasdphujz@user158.sys.ICSI.Berkeley.EDU> Hi, I will be updating the Bro git/webserver. There might be intermittent outages in the next few hours (mostly of git - the webserver should automatically fail over). Johanna From johanna at icir.org Thu Apr 19 13:42:09 2018 From: johanna at icir.org (Johanna Amann) Date: Thu, 19 Apr 2018 13:42:09 -0700 Subject: [Bro-Dev] Bro git/webserver updates In-Reply-To: <20180419184133.jr7hjh7rasdphujz@user158.sys.ICSI.Berkeley.EDU> References: <20180419184133.jr7hjh7rasdphujz@user158.sys.ICSI.Berkeley.EDU> Message-ID: <20180419204204.fear2toxi2xfezpo@user158.sys.ICSI.Berkeley.EDU> After a brief fight with httpd (the old config file did not work without changes), everything seems to be up and working again. Please let me know if you notice/encounter any problems. Johanna On Thu, Apr 19, 2018 at 11:41:33AM -0700, Johanna Amann wrote: > Hi, > > I will be updating the Bro git/webserver. There might be intermittent > outages in the next few hours (mostly of git - the webserver should > automatically fail over). > > Johanna > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > From asharma at lbl.gov Fri Apr 20 14:09:13 2018 From: asharma at lbl.gov (Aashish Sharma) Date: Fri, 20 Apr 2018 14:09:13 -0700 Subject: [Bro-Dev] input-framework reporter_error vs reporter_warning events ? Message-ID: <20180420210912.GI77874@MacPro-2331.local> While testing other stuff, I realized that if input-framework cannot find a file its now generating reporter_warning event instead of reporter_error ? Did "error" changed to "warning" for some reason ? Wasn't previously this a error condition ? 0.000000 Reporter::WARNING /DATA/feeds/BRO-feeds/WIRED.blocknet.2/Input::READER_ASCII: Init: cannot open /DATA/feeds/BRO-feeds/WIRED.blocknet.2 (empty) 1) Also in what situations input-framework would generate WARNINGS vs ERRORS ? 2) Does warnings means READER_ASCII will try to read file again after some time or does it just gives up and waits for me to tap into reporter_warning event to handle the situation ? Thanks, Aashish From johanna at icir.org Fri Apr 20 14:34:13 2018 From: johanna at icir.org (Johanna Amann) Date: Fri, 20 Apr 2018 14:34:13 -0700 Subject: [Bro-Dev] input-framework reporter_error vs reporter_warning events ? In-Reply-To: <20180420210912.GI77874@MacPro-2331.local> References: <20180420210912.GI77874@MacPro-2331.local> Message-ID: Hi Aashish, This changed with Bro 2.5.1. To quote NEWS: - The input framework's Ascii reader is now more resilient. If an input is marked to reread a file when it changes and the file didn't exist during a check Bro would stop watching the file in previous versions. The same could happen with bad data in a line of a file. These situations do not cause Bro to stop watching input files anymore. The old behavior is available through settings in the Ascii reader. Johanna On 20 Apr 2018, at 14:09, Aashish Sharma wrote: > While testing other stuff, I realized that if input-framework cannot > find a file > its now generating reporter_warning event instead of reporter_error ? > > Did "error" changed to "warning" for some reason ? Wasn't previously > this a > error condition ? > > > 0.000000 Reporter::WARNING > /DATA/feeds/BRO-feeds/WIRED.blocknet.2/Input::READER_ASCII: Init: > cannot open /DATA/feeds/BRO-feeds/WIRED.blocknet.2 (empty) > > > 1) Also in what situations input-framework would generate WARNINGS vs > ERRORS ? > > 2) Does warnings means READER_ASCII will try to read file again after > some time > or does it just gives up and waits for me to tap into reporter_warning > event to > handle the situation ? > > Thanks, > Aashish > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From asharma at lbl.gov Fri Apr 20 14:39:32 2018 From: asharma at lbl.gov (Aashish Sharma) Date: Fri, 20 Apr 2018 14:39:32 -0700 Subject: [Bro-Dev] input-framework reporter_error vs reporter_warning events ? In-Reply-To: References: <20180420210912.GI77874@MacPro-2331.local> Message-ID: <20180420213931.GK77874@MacPro-2331.local> Hah! Should have checked change notes more carefully! No wonder a bunch of pre 2.5.1 scripts never complained :) they all were tapping into event reporter_error. I'll forward port those now. Thanks Johanna! Aashish On Fri, Apr 20, 2018 at 02:34:13PM -0700, Johanna Amann wrote: > Hi Aashish, > > This changed with Bro 2.5.1. To quote NEWS: > > - The input framework's Ascii reader is now more resilient. If an input > is marked to reread a file when it changes and the file didn't exist > during a check Bro would stop watching the file in previous versions. > The same could happen with bad data in a line of a file. These > situations do not cause Bro to stop watching input files anymore. The > old behavior is available through settings in the Ascii reader. > > Johanna > > On 20 Apr 2018, at 14:09, Aashish Sharma wrote: > > > While testing other stuff, I realized that if input-framework cannot > > find a file > > its now generating reporter_warning event instead of reporter_error ? > > > > Did "error" changed to "warning" for some reason ? Wasn't previously > > this a > > error condition ? > > > > > > 0.000000 Reporter::WARNING > > /DATA/feeds/BRO-feeds/WIRED.blocknet.2/Input::READER_ASCII: Init: cannot > > open /DATA/feeds/BRO-feeds/WIRED.blocknet.2 (empty) > > > > > > 1) Also in what situations input-framework would generate WARNINGS vs > > ERRORS ? > > > > 2) Does warnings means READER_ASCII will try to read file again after > > some time > > or does it just gives up and waits for me to tap into reporter_warning > > event to > > handle the situation ? > > > > Thanks, > > Aashish > > _______________________________________________ > > bro-dev mailing list > > bro-dev at bro.org > > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From john at alt.house Tue Apr 24 14:57:19 2018 From: john at alt.house (John Althouse) Date: Tue, 24 Apr 2018 17:57:19 -0400 Subject: [Bro-Dev] Weirdness with event ssh_capabilities Message-ID: Take a look http://try.bro.org/#/trybro/saved/228251 In this instance, the client sent the Key Exchange Init packet first and the server sent it's Key Exchange Init packet second. The client packet actually contained the list of encryption algorithms seen here, but it's being printed out when I specify is_server == T, it should be printed when is_server == F, right? It also looks like ssh_capabilities is only capturing details within the first Key Exchange Init packet, whether that be the one from the server or the client, and ignoring the second one. So sometimes the server will send the KEI first, Bro captures that, then the client sends it's KEI and it looks like Bro ignores it. Same thing happens when it's the other way around. I want to be able to look at the details in both KEI's. Does this make sense? Does anyone know how this can be fixed or maybe I'm doing something wrong here? Thanks! John -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180424/1a2e7ae3/attachment.html From vlad at grigorescu.org Tue Apr 24 15:38:10 2018 From: vlad at grigorescu.org (Vlad Grigorescu) Date: Tue, 24 Apr 2018 22:38:10 +0000 Subject: [Bro-Dev] Weirdness with event ssh_capabilities In-Reply-To: References: Message-ID: Hi John, First, here's code that works: http://try.bro.org/#/trybro/saved/228261 (This is longer, because technically, clients and servers can specify different algs for each direction). Here's the relevant bit of Bro code: https://github.com/bro/bro/blob/master/scripts/base/protocols/ssh/main.bro#L225 c$ssh$capabilities is only designed for internal use. The comment for it[1] says "Store capabilities from the first host for comparison with the second (internal use)." So, yeah. It's basically exactly as you said. The data's only stored there in order to find common algorithms once we see the other KEX packet. If you want to access the raw capabilities coming from the client or the server, you need to use the capabilities parameter in that event. I feel like there's a better way to document that that parameter is for internal use only, but I'm not sure what it is. Does that help? --Vlad [1] - < https://github.com/bro/bro/blob/master/scripts/base/protocols/ssh/main.bro#L125 > On Tue, Apr 24, 2018 at 9:57 PM, John Althouse wrote: > Take a look > > http://try.bro.org/#/trybro/saved/228251 > > In this instance, the client sent the Key Exchange Init packet first and > the server sent it's Key Exchange Init packet second. The client packet > actually contained the list of encryption algorithms seen here, but it's > being printed out when I specify is_server == T, it should be printed when > is_server == F, right? > > It also looks like ssh_capabilities is only capturing details within the > first Key Exchange Init packet, whether that be the one from the server or > the client, and ignoring the second one. > > So sometimes the server will send the KEI first, Bro captures that, then > the client sends it's KEI and it looks like Bro ignores it. Same thing > happens when it's the other way around. I want to be able to look at the > details in both KEI's. > > Does this make sense? Does anyone know how this can be fixed or maybe I'm > doing something wrong here? > > Thanks! > > John > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180424/54c9bf7b/attachment.html From vern at icir.org Wed Apr 25 10:40:16 2018 From: vern at icir.org (Vern Paxson) Date: Wed, 25 Apr 2018 10:40:16 -0700 Subject: [Bro-Dev] set and vector operators Message-ID: <20180425174016.364942C416C@rock.ICSI.Berkeley.EDU> I'm working on some scripts that use sets and vectors, sometimes together, and am finding it clunky that Bro doesn't offer much in the way of operators for this. To that end, I'm thinking of implementing some along the following lines, where values starting with 's' are sets, 'v' are vectors, and 'e' are type-compatible elements: s1 + s2 Set union (for sets of the same type, of course) s1 || s2 Set union s1 && s2 Set intersection s1 - s2 Set difference s += e Add the element 'e' to set 's' (same as the current "add s[e]") s -= e Remove the 'e' element from 's', if present (same as the current "delete s[e]") s1 += s2 Same as "s1 = s1 + s2" s1 -= s2 Same as "s1 = s1 - s2" v += e Append the element 'e' to the vector 'v' v += s Append the elements of 's' to the vector 'v', with the order not being defined s += v Add the elements of 'v' to 's' s -= v Remove the elements of 'v' from 's', if present These strike me as pretty straightforward, but please chime in if you have comments! Vern From jazoff at illinois.edu Wed Apr 25 10:54:31 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 25 Apr 2018 17:54:31 +0000 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180425174016.364942C416C@rock.ICSI.Berkeley.EDU> References: <20180425174016.364942C416C@rock.ICSI.Berkeley.EDU> Message-ID: > On Apr 25, 2018, at 1:40 PM, Vern Paxson wrote: > > I'm working on some scripts that use sets and vectors, sometimes together, > and am finding it clunky that Bro doesn't offer much in the way of operators > for this. To that end, I'm thinking of implementing some along the following > lines, where values starting with 's' are sets, 'v' are vectors, and 'e' > are type-compatible elements: > > s1 + s2 Set union (for sets of the same type, of course) > s1 || s2 Set union > > s1 && s2 Set intersection > s1 - s2 Set difference > > s += e Add the element 'e' to set 's' > (same as the current "add s[e]") > s -= e Remove the 'e' element from 's', if present > (same as the current "delete s[e]") > > s1 += s2 Same as "s1 = s1 + s2" > s1 -= s2 Same as "s1 = s1 - s2" > > v += e Append the element 'e' to the vector 'v' > v += s Append the elements of 's' to the vector 'v', > with the order not being defined > > s += v Add the elements of 'v' to 's' > s -= v Remove the elements of 'v' from 's', if present > > These strike me as pretty straightforward, but please chime in if you > have comments! That's very similar to what python does, except they use & and | instead of && and ||. I think they do that because 'set or' is closer to 'bitwise or' than 'logical or' They also use ^ for symmetric difference. >>> a=set([1,2,3]) >>> b=set([2,3,4]) >>> a & b {2, 3} >>> a | b {1, 2, 3, 4} >>> a - b {1} >>> b - a {4} >>> a ^ b {1, 4} ? Justin Azoff From vern at icir.org Wed Apr 25 10:58:15 2018 From: vern at icir.org (Vern Paxson) Date: Wed, 25 Apr 2018 10:58:15 -0700 Subject: [Bro-Dev] set and vector operators In-Reply-To: (Wed, 25 Apr 2018 17:54:31 -0000). Message-ID: <20180425175815.74C162C4083@rock.ICSI.Berkeley.EDU> > That's very similar to what python does, except they use & and | instead of && and ||. > I think they do that because 'set or' is closer to 'bitwise or' than 'logical or' Yeah, I thought of that, but Bro currently doesn't have any '&' or '|' operators, which makes me reluctant to add them just for this. '&' is particularly problematic, as it would introduce ambiguity as to whether "&redef" means the redef attribute, or "use the 'and' operator on the value of the variable 'redef'". We'd have to add a bunch of reserved words to accommodate this. > They also use ^ for symmetric difference. (same here re being a new operator) Vern From robin at icir.org Wed Apr 25 13:56:29 2018 From: robin at icir.org (Robin Sommer) Date: Wed, 25 Apr 2018 13:56:29 -0700 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180425174016.364942C416C@rock.ICSI.Berkeley.EDU> References: <20180425174016.364942C416C@rock.ICSI.Berkeley.EDU> Message-ID: <20180425205629.GG14203@icir.org> On Wed, Apr 25, 2018 at 10:40 -0700, you wrote: > s1 + s2 Set union (for sets of the same type, of course) > s1 || s2 Set union (What's the difference between the two? Or do you mean either one or the other?) Like Justin, I was also thinking "|" and "&" might be more intuitive. "||"/"&&" is really typically associated with boolean contexts, and other languages mgiht also coerce set operands into booleans in such a context, so that, e.g., "s1 || s2" evaluates to true if either is non-empty. I see the problem with the parser but maybe adding keywords is the way to go. > s += e Add the element 'e' to set 's' > (same as the current "add s[e]") > s -= e Remove the 'e' element from 's', if present > (same as the current "delete s[e]") I'd skip these. I don't think we should add an additional set of operators for things that Bro already supports, that's seems confusing to me (like Perl :) > s1 += s2 Same as "s1 = s1 + s2" (Or s1 |= s2 if we pick "|" for union.) > v += e Append the element 'e' to the vector 'v' That's probably the most requested Bro operator ever! :) > v += s Append the elements of 's' to the vector 'v', > with the order not being defined This one I'm unsure about. The point about the order being undefined seems odd. If I don't care about order, wouldn't I stay with a set? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From vern at icir.org Wed Apr 25 14:13:20 2018 From: vern at icir.org (Vern Paxson) Date: Wed, 25 Apr 2018 14:13:20 -0700 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180425205629.GG14203@icir.org> (Wed, 25 Apr 2018 13:56:29 PDT). Message-ID: <20180425211320.F006D2C4190@rock.ICSI.Berkeley.EDU> > On Wed, Apr 25, 2018 at 10:40 -0700, you wrote: > > > s1 + s2 Set union (for sets of the same type, of course) > > s1 || s2 Set union > > (What's the difference between the two? Or do you mean either one or > the other?) No difference. It just seems to me that we need something for intersection, and using existing operators, the natural for that is "&&". Once we have that, might as well support "||" for union. But given symmetry with other operators, "+" should work too. > Like Justin, I was also thinking "|" and "&" might be more intuitive. If we didn't have the keyword issue with &attributes, then I could see that. But that strikes me as a significant drawback. (Also, if we do add these, then a user might reasonably expect them to work bitwise for count's. We could then consider implementing that too I guess.) > other languages mgiht also coerce set operands into booleans in such a > context, so that, e.g., "s1 || s2" evaluates to true if either is > non-empty. Hey I don't care about other seriously busted languages! ;-) > I see the problem with the parser but maybe adding keywords is the way > to go. Yuck. > > s += e Add the element 'e' to set 's' > > (same as the current "add s[e]") > > s -= e Remove the 'e' element from 's', if present > > (same as the current "delete s[e]") > > I'd skip these. I don't think we should add an additional set of > operators for things that Bro already supports I actually feel the opposite, that "add" is clunky ("delete" a bit less so) and thus these are more natural. But in particular it seems we ought to support these due to needing to support "v += e" (which is the one that I most want!). > > s1 += s2 Same as "s1 = s1 + s2" > > (Or s1 |= s2 if we pick "|" for union.) Yeah, if we bite off the '&'-keyword ugliness. Ugh. > > v += e Append the element 'e' to the vector 'v' > > That's probably the most requested Bro operator ever! :) Yee-up, per my note above! > > v += s Append the elements of 's' to the vector 'v', > > with the order not being defined > > This one I'm unsure about. The point about the order being undefined > seems odd. If I don't care about order, wouldn't I stay with a set? I do have a use case, but I agree it's odd; let me revisit it to see if I really do need it. I might instead settle for "vector of set[xxx]". Vern From vern at icir.org Wed Apr 25 22:19:36 2018 From: vern at icir.org (Vern Paxson) Date: Wed, 25 Apr 2018 22:19:36 -0700 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180425205629.GG14203@icir.org> (Wed, 25 Apr 2018 13:56:29 PDT). Message-ID: <20180426051936.E96A92C419C@rock.ICSI.Berkeley.EDU> Hmmm thinking about it, we can get away with '&' with minimal keyword conflict because there's such an easy (and natural-to-presume) fix - namely, rather than "x&attrkeyword" you use "x & attrkeyword". Now there's no problem, since the lexer only recognizes "&attrkeyword" as a unit, with no whitespace allowed. Given that, here's my updated proposal, with 'c' standing for values of type count: s1 + s2 Set union s1 - s2 Set difference s1 | s2 Set union s1 & s2 Set intersection s1 ^ s2 Set symmetric difference s + e The set resulting from adding the element 'e' to the set 's' s - e The set resulting from removing the element 'e' from the set 's', if present s1 {+=, -=, |=, &=, ^=} s2 Perform the corresponding set operation between s1 and s2 and put the result in s1. s {+=, -=} e Add or remove the element e from the set s c1 | c2 c1 & c2 Bitwise or/and/xor of two count values c1 ^ c2 c1 {|=, &=, ^=} c2 Perform the corresponding bitwise operation between c1 and c2 and put the result in c1. v += e Append the element 'e' to the vector 'v' s += v Add the elements of 'v' to 's' s -= v Remove the elements of 'v' from 's', if present How does that sound? Vern From jsiwek at corelight.com Thu Apr 26 07:48:09 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 26 Apr 2018 09:48:09 -0500 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180426051936.E96A92C419C@rock.ICSI.Berkeley.EDU> References: <20180426051936.E96A92C419C@rock.ICSI.Berkeley.EDU> Message-ID: <26638efb-7321-eb58-87f7-3447515b8e1c@corelight.com> On 4/26/18 12:19 AM, Vern Paxson wrote: > Hmmm thinking about it, we can get away with '&' with minimal keyword > conflict because there's such an easy (and natural-to-presume) fix - > namely, rather than "x&attrkeyword" you use "x & attrkeyword". Now > there's no problem, since the lexer only recognizes "&attrkeyword" > as a unit, with no whitespace allowed. Yeah, I think it could turn out ok. Using '&' and '|' in the set operations seems more natural/consistent to me and so maybe worthwhile to try that approach first. It's also a nice time to introduce the bitwise operations between count values. I thought about adding that on one occasion and also know others have wanted it. > How does that sound? Sounds good to me. I'd find most of them more convenient and wish I'd had them in the past. - Jon From robin at icir.org Thu Apr 26 08:12:39 2018 From: robin at icir.org (Robin Sommer) Date: Thu, 26 Apr 2018 08:12:39 -0700 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180426051936.E96A92C419C@rock.ICSI.Berkeley.EDU> References: <20180425205629.GG14203@icir.org> <20180426051936.E96A92C419C@rock.ICSI.Berkeley.EDU> Message-ID: <20180426151239.GF9692@icir.org> On Wed, Apr 25, 2018 at 22:19 -0700, you wrote: > Now there's no problem, since the lexer only recognizes "&attrkeyword" > as a unit, with no whitespace allowed. Good idea, sounds right. And in case it did turn out to be problematic, we could still go the way of adding all as keywords later. > How does that sound? Sounds good to me, the bitwise operations will be great to have, too. Just one more thing still: I'm actually feeling pretty strongly against having multiple different operators for the same operation (set union, set addition/removal). I just see that as leading to confusion: scripts will inconsistently use on or the other, people will have different preferences which version to prefer; they may not even remember the other one. And we'd end up having to explain why there are two versions, without having much of a great explanation ("one is ugly" doesn't sound great to me :-). Is it just me feeling that way? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at corelight.com Thu Apr 26 08:16:41 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 26 Apr 2018 10:16:41 -0500 Subject: [Bro-Dev] Final Broker branch testing Message-ID: The latest version of the new Broker-ized cluster/communication system for Bro in 'topic/actor-system' branch is wrapping up and, in my opinion, ready to be merged into Bro's 'master' branch. However, since it's such a big change, I'd like a last round of feedback before merging. If you want to test, the build process should now be as simple as: $ git clone --recursive --branch=topic/actor-system git://git.bro.org/bro $ cd bro && ./configure && make Configuring BroControl is not any different from before. If you had custom scripts, they may require porting. There's a guide and examples for that at [1] and [2] (hyperlinks in those docs will render more nicely when it's up on bro.org). Though, for this round of testing, I'd be most interested just in any general stability issues or major feature breakages from a vanilla Bro installation. Mild performance issues, minor bugs, or other issues w/ porting custom scripts are things I think we can iron out even after merging into 'master'. - Jon [1] https://github.com/bro/bro/blob/topic/actor-system/doc/frameworks/broker.rst [2] https://github.com/bro/bro/tree/topic/actor-system/doc/frameworks/broker From vlad at es.net Thu Apr 26 09:06:02 2018 From: vlad at es.net (Vlad Grigorescu) Date: Thu, 26 Apr 2018 16:06:02 +0000 Subject: [Bro-Dev] How to deal with stale branches? Message-ID: I have a couple of branches that, for whatever reason, aren't headed for a merge request into master. The branches were left around for reference, in case someone wanted to pick up and continue the work. However, this too now looks very unlikely, as some of these are several years old, and of questionable value at this point. I'm torn between deleting the branches, in an effort to not clog up git with unneeded branches, and leaving them around or perhaps archiving them somewhere, in order to not completely lose the work in case it's of value to someone down the road. I'm curious if anyone has thoughts on the best way to proceed. --Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180426/0ec7b5bb/attachment.html From vern at icir.org Thu Apr 26 09:29:24 2018 From: vern at icir.org (Vern Paxson) Date: Thu, 26 Apr 2018 09:29:24 -0700 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180426151239.GF9692@icir.org> (Thu, 26 Apr 2018 08:12:39 PDT). Message-ID: <20180426162924.3D83E2C408B@rock.ICSI.Berkeley.EDU> > Just one more thing still: I'm actually feeling pretty strongly > against having multiple different operators for the same operation > (set union, set addition/removal). I'm fine with removing "add" and "delete" for sets! (But seems we gotta keep them for a good while for backward compatibility. Plus, what would be the remove operator for tables? "t -= index" seems pretty weird to me.) But I don't think we should forego '+' and '-' for sets. It would be too weird that "v += e" works but "s += e" does not (and "add v[e]" blows, so let's not consider going down that path :-P). Once we have s += e, we certainly should have s -= e. And once we have "s + e", "s1 + s2" seems very natural to me too; I don't relish having to explain "oh *that* doesn't work, you have to use s1 | s2" :-P. > will have different preferences which version to prefer; they may not > even remember the other one. Seems to me they're both sufficiently mnemonic that this isn't a big worry. Vern From johanna at icir.org Thu Apr 26 12:04:23 2018 From: johanna at icir.org (Johanna Amann) Date: Thu, 26 Apr 2018 12:04:23 -0700 Subject: [Bro-Dev] Final Broker branch testing In-Reply-To: References: Message-ID: <4A4BEC38-8163-454C-80C7-0192B59144F7@icir.org> Trying this I noticed a few things (ordered by urgency from my point of view). With this change, we Bro cannot be compiled out of the Box on RedHat/Centos 7 anymore. Since that is the latest release of RedHat and probably used in production by quite a few people a potentially significant amount of people might not be able to (easily) compile Bro with this merge. It aborts in configure, with: -- Performing Test cxx11_header_works - Success CMake Error at aux/broker/CMakeLists.txt:4 (cmake_minimum_required): CMake 3.0.2 or higher is required. You are running version 2.8.12.2 --snip Compiling on Debian 8 gives some CAF warnings that are a tad ugly: In file included from /root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/serializer.hpp:32:0, from /root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/detail/tuple_vals.hpp:25, from /root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/make_message.hpp:28, from /root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/mailbox_element.hpp:27, from /root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/abstract_actor.hpp:37, from /root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/actor.hpp:32, from /root/bro/aux/broker/broker/data.hh:11, from /root/bro/aux/broker/broker/broker.hh:8, from /root/bro/src/broker/Data.h:4, from /root/bro/src/broker/Data.cc:1: /root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/data_processor.hpp: In function ?typename std::enable_if().caf::data_processor::apply(declval()))>::value>::type caf::operator&(caf::deserializer&, T&) [with T = std::chrono::time_point > >; typename std::enable_if().caf::data_processor::apply(declval()))>::value>::type = void]?: /root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/data_processor.hpp:478:7: warning: ?dur? may be used uninitialized in this function [-Wmaybe-uninitialized] t = std::chrono::time_point{dur}; ^ /root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/data_processor.hpp:476:16: note: ?dur? was declared here Duration dur; ^ --snip /root/bro/aux/broker/3rdparty/caf/libcaf_core/src/scheduled_actor.cpp:892:55: warning: unused parameter ?sender? [-Wunused-parameter] actor_addr& sender, --snip I noticed one small thing while building with make -j4; in this case you get several different % numbers simultaneously (one for car and one for broker). Example: [ 25%] Built target plugin-Bro-BackDoor [ 25%] Building CXX object src/analyzer/protocol/bittorrent/CMakeFiles/plugin-Bro-BitTorrent.dir/bittorrent_pac.cc.o [ 85%] Building CXX object libcaf_io/CMakeFiles/libcaf_io_shared.dir/src/interfaces.cpp.o [ 25%] Building CXX object src/analyzer/protocol/bittorrent/CMakeFiles/plugin-Bro-BitTorrent.dir/events.bif.cc.o While this is obviously cosmetic, it still looks weird to me :). Apart from that it compiled and ran all tests on all systems I tried it on. There were a few test failures on the first run (that succeeded on a rerun) though. These were (from different systems): MacOs: [ 76%] scripts.base.frameworks.logging.field-extension-cluster ... failed [ 21%] broker.disconnect ... failed [ 56%] broker.ssl_auth_failure ... failed [ 89%] scripts.base.frameworks.control.shutdown ... failed [ 99%] scripts.base.frameworks.openflow.log-cluster ... failed There were also a couple that did not succeed after several reruns for me. This was on a digital ocean 4cpu optimized debian8 instance for me; the reruns were not parallel: root at debian-c-4-8gib-sfo2-01:~/bro/testing/btest# ../../aux/btest/btest -r -d [ 0%] scripts.base.frameworks.control.configuration_update ... failed % 'btest-bg-wait 10' failed unexpectedly (exit code 1) % cat .stderr The following processes did not terminate: BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:.. bro /root/bro/testing/btest/.tmp/scripts.base.frameworks.control.configuration_update/configuration_update.bro frameworks/control/controller Control::host=127.0.0.1 Control::host_port=65531/tcp Control::cmd=shutdown ----------- <<< [15700] BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:.. bro /root/bro/testing/btest/.tmp/scripts.base.frameworks.control.configuration_update/configuration_update.bro frameworks/control/controllee Broker::default_port=65531/tcp , line 1: received termination signal >>> <<< [15738] BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:.. bro /root/bro/testing/btest/.tmp/scripts.base.frameworks.control.configuration_update/configuration_update.bro test-redef frameworks/control/controller Control::host=127.0.0.1 Control::host_port=65531/tcp Control::cmd=configuration_update /root/bro/scripts/policy/frameworks/control/controller.bro, line 136: Control framework sent 330 IDs , line 1: received termination signal >>> <<< [15779] BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:.. bro /root/bro/testing/btest/.tmp/scripts.base.frameworks.control.configuration_update/configuration_update.bro frameworks/control/controller Control::host=127.0.0.1 Control::host_port=65531/tcp Control::cmd=shutdown , line 1: received termination signal >>> [ 20%] scripts.base.frameworks.control.id_value ... failed % 'btest-diff controller/.stdout' failed unexpectedly (exit code 1) % cat .diag == File =============================== == Diff =============================== --- /tmp/test-diff.15967.controller..stdout.baseline.tmp 2018-04-26 19:02:46.156000000 +0000 +++ /tmp/test-diff.15967.controller..stdout.tmp 2018-04-26 19:02:46.156000000 +0000 @@ -1 +0,0 @@ -Got an id_value_response(test_var, This is the value from the controllee) event ======================================= % cat .stderr <<< [15865] BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:.. bro /root/bro/testing/btest/.tmp/scripts.base.frameworks.control.id_value/id_value.bro only-for-controllee frameworks/control/controllee Broker::default_port=65532/tcp , line 1: received termination signal >>> <<< [15893] BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:.. bro /root/bro/testing/btest/.tmp/scripts.base.frameworks.control.id_value/id_value.bro frameworks/control/controller Control::host=127.0.0.1 Control::host_port=65532/tcp Control::cmd=id_value Control::arg=test_var , line 1: received termination signal >>> [ 60%] scripts.base.frameworks.intel.remove-item-cluster ... failed % 'TEST_DIFF_CANONIFIER=$SCRIPTS/diff-sort btest-diff worker-1/.stdout' failed unexpectedly (exit code 1) % cat .diag == File =============================== Purging 192.168.0.1. Purging 192.168.0.2. Removing 192.168.1.2 (source: source1). Removing 192.168.1.2 (source: source2). == Diff =============================== --- /tmp/test-diff.16234.worker-1..stdout.baseline.tmp 2018-04-26 19:02:50.396000000 +0000 +++ /tmp/test-diff.16234.worker-1..stdout.tmp 2018-04-26 19:02:50.404000000 +0000 @@ -1,6 +1,5 @@ ### NOTE: This file has been sorted with diff-sort. Purging 192.168.0.1. Purging 192.168.0.2. -Purging 192.168.1.2. Removing 192.168.1.2 (source: source1). Removing 192.168.1.2 (source: source2). ======================================= % cat .stderr <<< [16113] BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:.. CLUSTER_NODE=manager-1 bro /root/bro/testing/btest/.tmp/scripts.base.frameworks.intel.remove-item-cluster/remove-item-cluster.bro received termination signal >>> <<< [16142] BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:.. CLUSTER_NODE=worker-1 bro /root/bro/testing/btest/.tmp/scripts.base.frameworks.intel.remove-item-cluster/remove-item-cluster.bro received termination signal >>> [ 80%] scripts.base.frameworks.logging.field-extension-cluster ... failed % 'btest-diff manager-1/http.log' failed unexpectedly (exit code 1) % cat .diag == File =============================== #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path http #open 2018-04-26-19-02-54 #fields _write_ts _stream _system_name ts uid id_orig_h id_orig_p id_resp_h id_resp_p trans_depth method host uri referrer version user_agent request_body_len response_body_len status_code status_msg info_code info_msg tags username password proxied orig_fuids orig_filenames orig_mime_types resp_fuids resp_filenames resp_mime_types #types time string string time string addr port addr port count string string string string string string count count count string count string set[enum] string string set[string] vector[string] vector[string] vector[string] vector[string] vector[string] vector[string] 1524769374.222098 http worker-1 1524769374.162224 CUM0KZ3MLUfNB0cl11 141.142.220.118 48649 208.80.152.118 80 1 GET bits.wikimedia.org /skins-1.5/monobook/main.css http://www.wikipedia.org/ 1.1 Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - (empty) - - - - - - - - - #close 2018-04-26-19-03-00 == Diff =============================== --- /tmp/test-diff.16396.manager-1.http.log.baseline.tmp 2018-04-26 19:03:00.504000000 +0000 +++ /tmp/test-diff.16396.manager-1.http.log.tmp 2018-04-26 19:03:00.508000000 +0000 @@ -7,17 +7,4 @@ #fields _write_ts _stream _system_name ts uid id_orig_h id_orig_p id_resp_h id_resp_p trans_depth method host uri referrer version user_agent request_body_len response_body_len status_code status_msg info_code info_msg tags username password proxied orig_fuids orig_filenames orig_mime_types resp_fuids resp_filenames resp_mime_types #types time string string time string addr port addr port count string string string string string string count count count string count string set[enum] string string set[string] vector[string] vector[string] vector[string] vector[string] vector[string] vector[string] XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX CUM0KZ3MLUfNB0cl11 141.142.220.118 48649 208.80.152.118 80 1 GET bits.wikimedia.org /skins-1.5/monobook/main.css http://www.wikipedia.org/ 1.1 Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - (empty) - - - - - - - - - -XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX CwjjYJ2WqgTbAqiHl6 141.142.220.118 49997 208.80.152.3 80 1 GET upload.wikimedia.org /wikipedia/commons/6/63/Wikipedia-logo.png http://www.wikipedia.org/ 1.0 Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - (empty) - - - - - - - - - -XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX C3eiCBGOLw3VtHfOj 141.142.220.118 49996 208.80.152.3 80 1 GET upload.wikimedia.org /wikipedia/commons/thumb/b/bb/Wikipedia_wordmark.svg/174px-Wikipedia_wordmark.svg.png http://www.wikipedia.org/ 1.0 Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - (empty) - - - - - - - - - -XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX Ck51lg1bScffFj34Ri 141.142.220.118 49998 208.80.152.3 80 1 GET upload.wikimedia.org /wikipedia/commons/b/bd/Bookshelf-40x201_6.png http://www.wikipedia.org/ 1.0 Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - (empty) - - - - - - - - - -XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX CtxTCR2Yer0FR1tIBg 141.142.220.118 50000 208.80.152.3 80 1 GET upload.wikimedia.org /wikipedia/commons/thumb/8/8a/Wikinews-logo.png/35px-Wikinews-logo.png http://www.wikipedia.org/ 1.0 Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - (empty) - - - - - - - - - -XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX CykQaM33ztNt0csB9a 141.142.220.118 49999 208.80.152.3 80 1 GET upload.wikimedia.org /wikipedia/commons/4/4a/Wiktionary-logo-en-35px.png http://www.wikipedia.org/ 1.0 Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - (empty) - - - - - - - - - -XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX CLNN1k2QMum1aexUK7 141.142.220.118 50001 208.80.152.3 80 1 GET upload.wikimedia.org /wikipedia/commons/thumb/f/fa/Wikiquote-logo.svg/35px-Wikiquote-logo.svg.png http://www.wikipedia.org/ 1.0 Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - (empty) - - - - - - - - - -XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX CiyBAq1bBLNaTiTAc 141.142.220.118 35642 208.80.152.2 80 1 GET meta.wikimedia.org /images/wikimedia-button.png http://www.wikipedia.org/ 1.0 Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - (empty) - - - - - - - - - -XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX CwjjYJ2WqgTbAqiHl6 141.142.220.118 49997 208.80.152.3 80 2 GET upload.wikimedia.org /wikipedia/commons/thumb/f/fa/Wikibooks-logo.svg/35px-Wikibooks-logo.svg.png http://www.wikipedia.org/ 1.0 Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - (empty) - - - - - - - - - -XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX C3eiCBGOLw3VtHfOj 141.142.220.118 49996 208.80.152.3 80 2 GET upload.wikimedia.org /wikipedia/commons/thumb/d/df/Wikispecies-logo.svg/35px-Wikispecies-logo.svg.png http://www.wikipedia.org/ 1.0 Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - (empty) - - - - - - - - - -XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX Ck51lg1bScffFj34Ri 141.142.220.118 49998 208.80.152.3 80 2 GET upload.wikimedia.org /wikipedia/commons/thumb/4/4c/Wikisource-logo.svg/35px-Wikisource-logo.svg.png http://www.wikipedia.org/ 1.0 Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - (empty) - - - - - - - - - -XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX CtxTCR2Yer0FR1tIBg 141.142.220.118 50000 208.80.152.3 80 2 GET upload.wikimedia.org /wikipedia/commons/thumb/4/4a/Commons-logo.svg/35px-Commons-logo.svg.png http://www.wikipedia.org/ 1.0 Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - (empty) - - - - - - - - - -XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX CykQaM33ztNt0csB9a 141.142.220.118 49999 208.80.152.3 80 2 GET upload.wikimedia.org /wikipedia/commons/thumb/9/91/Wikiversity-logo.svg/35px-Wikiversity-logo.svg.png http://www.wikipedia.org/ 1.0 Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - (empty) - - - - - - - - - -XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX CLNN1k2QMum1aexUK7 141.142.220.118 50001 208.80.152.3 80 2 GET upload.wikimedia.org /wikipedia/commons/thumb/7/75/Wikimedia_Community_Logo.svg/35px-Wikimedia_Community_Logo.svg.png http://www.wikipedia.org/ 1.0 Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - (empty) - - - - - - - - - #close XXXX-XX-XX-XX-XX-XX ======================================= % cat .stderr <<< [16284] cp ../cluster-layout.bro . && CLUSTER_NODE=manager-1 bro /root/bro/testing/btest/.tmp/scripts.base.frameworks.logging.field-extension-cluster/field-extension-cluster.bro received termination signal >>> <<< [16325] cp ../cluster-layout.bro . && CLUSTER_NODE=worker-1 bro --pseudo-realtime -C -r /root/bro/testing/btest/Traces/wikipedia.trace /root/bro/testing/btest/.tmp/scripts.base.frameworks.logging.field-extension-cluster/field-extension-cluster.bro >>> Johanna On 26 Apr 2018, at 8:16, Jon Siwek wrote: > The latest version of the new Broker-ized cluster/communication system > for Bro in 'topic/actor-system' branch is wrapping up and, in my > opinion, ready to be merged into Bro's 'master' branch. > > However, since it's such a big change, I'd like a last round of > feedback > before merging. If you want to test, the build process should now be > as > simple as: > > $ git clone --recursive --branch=topic/actor-system > git://git.bro.org/bro > $ cd bro && ./configure && make > > Configuring BroControl is not any different from before. > > If you had custom scripts, they may require porting. There's a guide > and examples for that at [1] and [2] (hyperlinks in those docs will > render more nicely when it's up on bro.org). > > Though, for this round of testing, I'd be most interested just in any > general stability issues or major feature breakages from a vanilla Bro > installation. Mild performance issues, minor bugs, or other issues w/ > porting custom scripts are things I think we can iron out even after > merging into 'master'. > > - Jon > > [1] > https://github.com/bro/bro/blob/topic/actor-system/doc/frameworks/broker.rst > [2] > https://github.com/bro/bro/tree/topic/actor-system/doc/frameworks/broker > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From dnthayer at illinois.edu Thu Apr 26 12:34:15 2018 From: dnthayer at illinois.edu (Daniel Thayer) Date: Thu, 26 Apr 2018 14:34:15 -0500 Subject: [Bro-Dev] Final Broker branch testing In-Reply-To: References: Message-ID: I noticed that Bro no longer builds on any version of RHEL/CentOS: CMake Error at aux/broker/CMakeLists.txt:4 (cmake_minimum_required): CMake 3.0.2 or higher is required. You are running version 2.8.12.2 -------------------------------------------------------------------- Also, I observed the following test failures on debian 9 (there were a few other failures, but they went away after running "btest -r" once): $ cat diag.log scripts.base.frameworks.control.id_value ... failed % 'btest-diff controller/.stdout' failed unexpectedly (exit code 1) % cat .diag == File =============================== == Diff =============================== --- /tmp/test-diff.13239.controller..stdout.baseline.tmp 2018-04-26 19:16:51.327612039 +0000 +++ /tmp/test-diff.13239.controller..stdout.tmp 2018-04-26 19:16:51.327612039 +0000 @@ -1 +0,0 @@ -Got an id_value_response(test_var, This is the value from the controllee) event ======================================= % cat .stderr <<< [13137] BROPATH=.:/home/repo/bro/scripts:/home/repo/bro/scripts/policy:/home/repo/bro/scripts/site:/home/repo/bro/build/scripts:.. bro /home/repo/bro/testing/btest/.tmp/scripts.base.frameworks.control.id_value/id_value.bro only-for-controllee frameworks/control/controllee Broker::default_port=65532/tcp , line 1: received termination signal >>> <<< [13165] BROPATH=.:/home/repo/bro/scripts:/home/repo/bro/scripts/policy:/home/repo/bro/scripts/site:/home/repo/bro/build/scripts:.. bro /home/repo/bro/testing/btest/.tmp/scripts.base.frameworks.control.id_value/id_value.bro frameworks/control/controller Control::host=127.0.0.1 Control::host_port=65532/tcp Control::cmd=id_value Control::arg=test_var , line 1: received termination signal >>> scripts.base.frameworks.control.shutdown ... failed % 'btest-bg-wait 10' failed unexpectedly (exit code 1) % cat .stderr The following processes did not terminate: BROPATH=.:/home/repo/bro/scripts:/home/repo/bro/scripts/policy:/home/repo/bro/scripts/site:/home/repo/bro/build/scripts:.. bro /home/repo/bro/testing/btest/.tmp/scripts.base.frameworks.control.shutdown/shutdown.bro frameworks/control/controller Control::host=127.0.0.1 Control::host_port=65530/tcp Control::cmd=shutdown ----------- <<< [13281] BROPATH=.:/home/repo/bro/scripts:/home/repo/bro/scripts/policy:/home/repo/bro/scripts/site:/home/repo/bro/build/scripts:.. bro /home/repo/bro/testing/btest/.tmp/scripts.base.frameworks.control.shutdown/shutdown.bro frameworks/control/controllee Broker::default_port=65530/tcp , line 1: received termination signal >>> <<< [13309] BROPATH=.:/home/repo/bro/scripts:/home/repo/bro/scripts/policy:/home/repo/bro/scripts/site:/home/repo/bro/build/scripts:.. bro /home/repo/bro/testing/btest/.tmp/scripts.base.frameworks.control.shutdown/shutdown.bro frameworks/control/controller Control::host=127.0.0.1 Control::host_port=65530/tcp Control::cmd=shutdown , line 1: received termination signal >>> ----------------------------------------------------------------------- I also observed the following test failures on ubuntu 16.04LTS (there were a few other test failures, but they went away after running "btest -r" once): $ cat diag.log scripts.base.frameworks.control.id_value ... failed % 'btest-diff controller/.stdout' failed unexpectedly (exit code 1) % cat .diag == File =============================== == Diff =============================== --- /tmp/test-diff.14643.controller..stdout.baseline.tmp 2018-04-26 19:18:05.246312235 +0000 +++ /tmp/test-diff.14643.controller..stdout.tmp 2018-04-26 19:18:05.246312235 +0000 @@ -1 +0,0 @@ -Got an id_value_response(test_var, This is the value from the controllee) event ======================================= % cat .stderr <<< [14541] BROPATH=.:/home/repo/bro/scripts:/home/repo/bro/scripts/policy:/home/repo/bro/scripts/site:/home/repo/bro/build/scripts:.. bro /home/repo/bro/testing/btest/.tmp/scripts.base.frameworks.control.id_value/id_value.bro only-for-controllee frameworks/control/controllee Broker::default_port=65532/tcp , line 1: received termination signal >>> <<< [14569] BROPATH=.:/home/repo/bro/scripts:/home/repo/bro/scripts/policy:/home/repo/bro/scripts/site:/home/repo/bro/build/scripts:.. bro /home/repo/bro/testing/btest/.tmp/scripts.base.frameworks.control.id_value/id_value.bro frameworks/control/controller Control::host=127.0.0.1 Control::host_port=65532/tcp Control::cmd=id_value Control::arg=test_var , line 1: received termination signal >>> ----------------------------------------------------------------------- From jsiwek at corelight.com Thu Apr 26 13:10:34 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 26 Apr 2018 15:10:34 -0500 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180426162924.3D83E2C408B@rock.ICSI.Berkeley.EDU> References: <20180426162924.3D83E2C408B@rock.ICSI.Berkeley.EDU> Message-ID: On 4/26/18 11:29 AM, Vern Paxson wrote: >> Just one more thing still: I'm actually feeling pretty strongly >> against having multiple different operators for the same operation >> (set union, set addition/removal). I'm maybe convincing myself that it's at least not that useful or there's alternative ways forward that don't introduce redundancies. > I'm fine with removing "add" and "delete" for sets! (But seems we gotta > keep them for a good while for backward compatibility. Plus, what would > be the remove operator for tables? "t -= index" seems pretty weird to me.) A nice thing about "add" and "delete" for sets is that you can infer the data type that you're operating on just looking at the local code/line. E.g. say you come back to some code after a few months and see "foo += 1". Not obvious what 'foo' is anymore. Could be vector, set, or count, etc. > But I don't think we should forego '+' and '-' for sets. It would be too > weird that "v += e" works but "s += e" does not (and "add v[e]" blows, so > let's not consider going down that path :-P). Yeah, I agree with that sentiment (on the condition that we did add +/- for sets). I do also notice that you had "s + e" in the proposal and not "v + e". Isn't that weird by the same logic or is it just an accidental omission? > Once we have s += e, we > certainly should have s -= e. And once we have "s + e", "s1 + s2" seems > very natural to me too; I don't relish having to explain "oh *that* doesn't > work, you have to use s1 | s2" :-P. That also makes sense, though it's worth seeing what a minimal proposal looks like without the contentious '+' operations: s1 - s2, s1 -= s2 Set difference s1 | s2, s1 |= s2 Set union s1 & s2, s1 &= s2 Set intersection s1 ^ s2, s1 ^= s2 Set symmetric difference c1 | c2, c1 |= c2 Bitwise-or 'count' values c1 & c2, c1 &= c2 Bitwise-and 'count' values c1 ^ c2, c1 ^= c2 Bitwise-xor 'count' values The only one now missing that I'd probably find myself using is: v += e Append element to vector And for that, a BiF or generic script-layer function call (if that were possible) would even make me happy: push(v, e) That also could go back to what I was saying before about readability: it would then be more obvious than "v += e" regarding what data types are involved. Same idea would apply to "s += v" and "s -= v" (if we were inclined): add_to_set(s, v) delete_from_set(s, v) Could that all be an alternative way forward? Or is it missing other important aspects? - Jon From jsiwek at corelight.com Thu Apr 26 13:15:49 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 26 Apr 2018 15:15:49 -0500 Subject: [Bro-Dev] How to deal with stale branches? In-Reply-To: References: Message-ID: <5c5c49d2-60fc-17bd-8938-db37b9523c0c@corelight.com> On 4/26/18 11:06 AM, Vlad Grigorescu wrote: > I'm torn between deleting the branches, in an effort to not clog up git > with unneeded branches, and leaving them around or perhaps archiving > them somewhere, in order to not completely lose the work in case it's of > value to someone down the road. > > I'm curious if anyone has thoughts on the best way to proceed. Maybe delete the branch from the official git repo and push it to your own github fork. - Jon From jazoff at illinois.edu Thu Apr 26 13:25:15 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 26 Apr 2018 20:25:15 +0000 Subject: [Bro-Dev] Final Broker branch testing In-Reply-To: References: Message-ID: > On Apr 26, 2018, at 11:16 AM, Jon Siwek wrote: > > The latest version of the new Broker-ized cluster/communication system > for Bro in 'topic/actor-system' branch is wrapping up and, in my > opinion, ready to be merged into Bro's 'master' branch. > [..] > > Though, for this round of testing, I'd be most interested just in any > general stability issues or major feature breakages from a vanilla Bro > installation. Mild performance issues, minor bugs, or other issues w/ > porting custom scripts are things I think we can iron out even after > merging into 'master'. > > - Jon I threw this on our test cluster, and whatever that issue was with rotation breaking causing the logger to buffer and the OOM is fixed now.. logs have rotated twice now without issue. cpu usage is still higher, but I think it is just busy waiting like you suggested.. perf top on a proxy shows: 5.32% [kernel] [k] system_call_after_swapgs 3.48% libcaf_core.so.0.15.7 [.] caf::scheduler::worker::run 3.12% libc-2.17.so [.] __GI___libc_nanosleep 3.10% [kernel] [k] sysret_check 3.05% libcaf_core.so.0.15.7 [.] caf::detail::double_ended_queue::take_head 2.61% [kernel] [k] __schedule 2.20% libc-2.17.so [.] __sleep 2.19% [kernel] [k] timerqueue_add 2.06% [kernel] [k] __audit_syscall_exit 1.89% [kernel] [k] native_write_msr_safe 1.85% [kernel] [k] cpuacct_charge 1.84% [kernel] [k] __audit_syscall_entry 1.74% [kernel] [k] hrtimer_start_range_ns 1.50% libstdc++.so.6.0.19 [.] std::this_thread::__sleep_for 1.40% libc-2.17.so [.] __libc_disable_asynccancel 1.37% [kernel] [k] _raw_spin_unlock_irqrestore 1.37% [kernel] [k] do_nanosleep 1.25% libc-2.17.so [.] usleep 1.22% [kernel] [k] rb_insert_color 1.20% [kernel] [k] update_curr 1.18% [kernel] [k] idle_cpu 1.14% [kernel] [k] copy_user_generic_string 1.09% [kernel] [k] finish_task_switch 1.07% [kernel] [k] __x86_indirect_thunk_rax 1.06% [kernel] [k] ktime_get 0.93% [kernel] [k] native_sched_clock 0.92% [kernel] [k] sys_nanosleep which seems almost entirely related to timers and sleeping. Other than that things are working great. Cluster::publish_hrw is distributing data cross proxies perfectly: # for x in 1 2 3; do broctl print Scan::attacks proxy-$x|grep attempts= -c;done 3304 3405 3397 # cat /bro/logs/current/notice.log |bro-cut note peer_descr|grep Scan::|cut -f 2|sort|uniq -c 454 proxy-1 463 proxy-2 417 proxy-3 Once this is stable for a bit i'll start trying things like killing a proxy and verifying that things failover. ? Justin Azoff From vern at icir.org Thu Apr 26 13:43:53 2018 From: vern at icir.org (Vern Paxson) Date: Thu, 26 Apr 2018 13:43:53 -0700 Subject: [Bro-Dev] set and vector operators In-Reply-To: (Thu, 26 Apr 2018 15:10:34 CDT). Message-ID: <20180426204353.3010C2C4009@rock.ICSI.Berkeley.EDU> > A nice thing about "add" and "delete" for sets is that you can infer the > data type that you're operating on just looking at the local code/line. Only sort of. For delete, you don't know whether it's a table or a set, and for neither do you know what type of table/set if you can't immediately apprehend the type of the index (e.g., "delete foo[foo2]" doesn't tell me what type of index the set/table foo uses). > E.g. say you come back to some code after a few months and see "foo += > 1". Not obvious what 'foo' is anymore. I don't think it's reasonable to have the bar be "can you tell what's going on in isolation". It should include consideration of associated context, variable names, and comments. In fact, even now you don't know whether for "foo += 1" foo is an integer, a count, or a double. > I do also notice that you had "s + e" in the proposal and not "v + e". > Isn't that weird by the same logic or is it just an accidental omission? This is because "v + e" already has a meaning: apply "+ e" to each element of v. (Note though that "v += e" is not currently allowed.) > v += e Append element to vector > > And for that, a BiF or generic script-layer function call (if that were > possible) would even make me happy: > > push(v, e) > ... > add_to_set(s, v) > delete_from_set(s, v) Yuck. I would hate this. Might as well use Lisp! A basic tenet of Bro's language design has been to minimize verbosity. (For example, its use of type inference, and its support of C-style operators like "++".) Let's please not move in that direction. Vern From vern at icir.org Thu Apr 26 14:04:23 2018 From: vern at icir.org (Vern Paxson) Date: Thu, 26 Apr 2018 14:04:23 -0700 Subject: [Bro-Dev] pattern values and "||"/"&&" operators Message-ID: <20180426210423.7C6372C41AF@rock.ICSI.Berkeley.EDU> In implementing bitwise operations for counts (now pretty much done!), I found that Bro's internals actually support "||" and "&&" operators for patterns: p1 || p2 returns a pattern that matches either p1 or p2 p1 && p2 returns a pattern that matches p1 followed by p2 (Note that this second is not obvious. One might instead expect it to return a pattern that matches only if both p1 and p2 match, where the two could overlap.) This doesn't appear to be documented anywhere. I discussed it with Robin and we both agree that this seems weird and likely not used anywhere. Therefore we're thinking we should deprecate it and eventually remove it. Anyone think otherwise? Vern From jan.grashoefer at gmail.com Thu Apr 26 14:04:21 2018 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=C3=B6fer?=) Date: Thu, 26 Apr 2018 23:04:21 +0200 Subject: [Bro-Dev] Final Broker branch testing In-Reply-To: <4A4BEC38-8163-454C-80C7-0192B59144F7@icir.org> References: <4A4BEC38-8163-454C-80C7-0192B59144F7@icir.org> Message-ID: > > Trying this I noticed a few things (ordered by urgency from my point of > view). > > With this change, we Bro cannot be compiled out of the Box on > RedHat/Centos 7 anymore. Since that is the latest release of RedHat and > probably used in production by quite a few people a potentially > significant amount of people might not be able to (easily) compile Bro > with this merge. > > It aborts in configure, with: > > -- Performing Test cxx11_header_works - Success > CMake Error at aux/broker/CMakeLists.txt:4 (cmake_minimum_required): > CMake 3.0.2 or higher is required. You are running version 2.8.12.2 > Using EPEL (which should be quite easy), a cmake3 package is available. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180426/58ac1498/attachment-0001.html From jsiwek at corelight.com Thu Apr 26 14:05:23 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 26 Apr 2018 16:05:23 -0500 Subject: [Bro-Dev] Final Broker branch testing In-Reply-To: <4A4BEC38-8163-454C-80C7-0192B59144F7@icir.org> References: <4A4BEC38-8163-454C-80C7-0192B59144F7@icir.org> Message-ID: On 4/26/18 2:04 PM, Johanna Amann wrote: > With this change, we Bro cannot be compiled out of the Box on > RedHat/Centos 7 anymore. Since that is the latest release of RedHat and > probably used in production by quite a few people a potentially > significant amount of people might not be able to (easily) compile Bro > with this merge. > > It aborts in configure, with: > > -- Performing Test cxx11_header_works - Success > CMake Error at aux/broker/CMakeLists.txt:4 (cmake_minimum_required): > ? CMake 3.0.2 or higher is required.? You are running version 2.8.12.2 Is "use cmake3 from EPEL" an acceptable answer? The main reason for it (IIRC) is for embedding CAF as a CMake ExternalProject, which I was struggling to hack around with lack of features in CMake 2.8. > Compiling on Debian 8 gives some CAF warnings that are a tad ugly: It's CAF's master branch at the moment, so I don't feel much pressure to report/patch these unless they're still there when we're close to moving to beta/release version. > I noticed one small thing while building with make -j4; in this case you > get several different % numbers simultaneously (one for car and one for > broker). Not sure I can help that. The thing that comes to mind would be enforcing that CAF does not build in parallel and thus wasting your time. Or else try to patch CMake to do a better job when using external projects. > There were a few test failures on the first run (that succeeded on a > rerun) though. Thanks, I also still see occasional failures that pass when re-running -- it's lower priority on my list to stabilize these. And I think doesn't prevent merging to master since they are most often problems with the unit test itself. > There were also a couple that did not succeed after several reruns for > me. This was on a digital ocean 4cpu optimized debian8 instance for me; > the reruns were not parallel: Ok, will take a closer look at those. I should also mention that I've run unit tests myself on the following: * MacOS 10.13.4 * FreeBSD 11 * CentOS 7 * Debian 8 I find the tests are usually stable with a few needing some reruns. I haven't had any single test persistently fail on any of those systems. I was also testing --build-type=debug everywhere and only a few places without, so that could be a difference. - Jon From johanna at icir.org Thu Apr 26 14:30:42 2018 From: johanna at icir.org (Johanna Amann) Date: Thu, 26 Apr 2018 14:30:42 -0700 Subject: [Bro-Dev] Final Broker branch testing In-Reply-To: References: <4A4BEC38-8163-454C-80C7-0192B59144F7@icir.org> Message-ID: On 26 Apr 2018, at 14:05, Jon Siwek wrote: > On 4/26/18 2:04 PM, Johanna Amann wrote: > >> With this change, we Bro cannot be compiled out of the Box on >> RedHat/Centos 7 anymore. Since that is the latest release of RedHat >> and probably used in production by quite a few people a potentially >> significant amount of people might not be able to (easily) compile >> Bro with this merge. >> >> It aborts in configure, with: >> >> -- Performing Test cxx11_header_works - Success >> CMake Error at aux/broker/CMakeLists.txt:4 (cmake_minimum_required): >> ? CMake 3.0.2 or higher is required.? You are running version >> 2.8.12.2 > > Is "use cmake3 from EPEL" an acceptable answer? > > The main reason for it (IIRC) is for embedding CAF as a CMake > ExternalProject, which I was struggling to hack around with lack of > features in CMake 2.8. It might be. I am honestly not sure - I suspect that this still will mean that some places might not be able to easily use Bro anymore--adding external package sources does not seem to be a viable option everywhere. As a side-note, it also looks like that means that we cannot provide binary packages for RedHat/CentOS anymore. Johanna From johanna at icir.org Thu Apr 26 14:32:11 2018 From: johanna at icir.org (Johanna Amann) Date: Thu, 26 Apr 2018 14:32:11 -0700 Subject: [Bro-Dev] pattern values and "||"/"&&" operators In-Reply-To: <20180426210423.7C6372C41AF@rock.ICSI.Berkeley.EDU> References: <20180426210423.7C6372C41AF@rock.ICSI.Berkeley.EDU> Message-ID: <5E4FAE3B-6F86-4C87-861C-D2FBFFD1ABA1@icir.org> On 26 Apr 2018, at 14:04, Vern Paxson wrote: > In implementing bitwise operations for counts (now pretty much done!), > I found that Bro's internals actually support "||" and "&&" operators > for > patterns: > > p1 || p2 returns a pattern that matches either p1 or p2 > p1 && p2 returns a pattern that matches p1 followed by p2 > > (Note that this second is not obvious. One might instead expect it to > return a pattern that matches only if both p1 and p2 match, where the > two > could overlap.) > > This doesn't appear to be documented anywhere. I discussed it with > Robin > and we both agree that this seems weird and likely not used anywhere. > Therefore we're thinking we should deprecate it and eventually remove > it. Sounds fine to me - I totally was not aware of this. Though having the functionality of the former available in some form would be kind of nice-I have a few scripts where I would have used that. Johanna From jsiwek at corelight.com Thu Apr 26 14:33:44 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 26 Apr 2018 16:33:44 -0500 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180426204353.3010C2C4009@rock.ICSI.Berkeley.EDU> References: <20180426204353.3010C2C4009@rock.ICSI.Berkeley.EDU> Message-ID: On 4/26/18 3:43 PM, Vern Paxson wrote: >> E.g. say you come back to some code after a few months and see "foo += >> 1". Not obvious what 'foo' is anymore. > > I don't think it's reasonable to have the bar be "can you tell what's going > on in isolation". It should include consideration of associated context, > variable names, and comments. In fact, even now you don't know whether > for "foo += 1" foo is an integer, a count, or a double. Yeah, it was maybe a bit of a stretch -- more just an observation I was trying to see if we could run with. Also in relation to my recent experiences with trying to read/debug some C++ code with a lot of operator usage I found myself wishing some were just a named function call so I could more easily navigate the code and even just find where certain operators were defined/declared. So the point I'm at now is that it would probably be nice not to have multiple operators for the same thing, though don't have a strong feeling about it. - Jon From vern at icir.org Thu Apr 26 14:38:39 2018 From: vern at icir.org (Vern Paxson) Date: Thu, 26 Apr 2018 14:38:39 -0700 Subject: [Bro-Dev] pattern values and "||"/"&&" operators In-Reply-To: <5E4FAE3B-6F86-4C87-861C-D2FBFFD1ABA1@icir.org> (Thu, 26 Apr 2018 14:32:11 PDT). Message-ID: <20180426213839.E7A762C41B2@rock.ICSI.Berkeley.EDU> > > p1 || p2 returns a pattern that matches either p1 or p2 > > ... Though having the > functionality of the former available in some form would be kind of > nice-I have a few scripts where I would have used that. Oh I left out that that's in fact supported: print /foo/ | /bar/ in "hello Barbara"; prints "T" :-). As does print (/foo/ || /bar/) in "hello Barbara"; (parens now required), which is the one we'd be getting rid of. Vern From johanna at icir.org Thu Apr 26 14:39:24 2018 From: johanna at icir.org (Johanna Amann) Date: Thu, 26 Apr 2018 14:39:24 -0700 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180426204353.3010C2C4009@rock.ICSI.Berkeley.EDU> References: <20180426204353.3010C2C4009@rock.ICSI.Berkeley.EDU> Message-ID: <20180426213924.c4hyoo2d3oc5kc3p@user158.sys.ICSI.Berkeley.EDU> On Thu, Apr 26, 2018 at 01:43:53PM -0700, Vern Paxson wrote: > > A nice thing about "add" and "delete" for sets is that you can infer the > > I do also notice that you had "s + e" in the proposal and not "v + e". > > Isn't that weird by the same logic or is it just an accidental omission? > > This is because "v + e" already has a meaning: apply "+ e" to each element > of v. (Note though that "v += e" is not currently allowed.) I was actually not aware of this - and if we keep this behavior I am a bit opposed to add a few of the others. It especially feels weird to me if v + e and v += e are operations that perform something completely different. I also think it is a bad idea to have s + e and v + e perform completely different operations. Johanna From johanna at icir.org Thu Apr 26 14:53:18 2018 From: johanna at icir.org (Johanna Amann) Date: Thu, 26 Apr 2018 14:53:18 -0700 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180426162924.3D83E2C408B@rock.ICSI.Berkeley.EDU> References: <20180426151239.GF9692@icir.org> <20180426162924.3D83E2C408B@rock.ICSI.Berkeley.EDU> Message-ID: <20180426215318.y2flrutl2ziu6ubq@user158.sys.ICSI.Berkeley.EDU> On Thu, Apr 26, 2018 at 09:29:24AM -0700, Vern Paxson wrote: > > Just one more thing still: I'm actually feeling pretty strongly > > against having multiple different operators for the same operation > > (set union, set addition/removal). > > I'm fine with removing "add" and "delete" for sets! (But seems we gotta > keep them for a good while for backward compatibility. Plus, what would > be the remove operator for tables? "t -= index" seems pretty weird to me.) Perhaps my thinking is too driven by how this is handled internally - but thinking that way I am kind of opposed to get rid of add and delete for sets. For me sets were always a special case of tables - and it made complete sense that they operate in the same way. I think I actually would prefer just keeping add/delete, at least for sets, and not introduce the plus-syntax. While it is a bit shorter, add/delete is also not that ugly. And while you did not like the argument of Jon that that lets you more easily determine what is going on in the code I kind of think it has a bit of merit. Johanna From robin at icir.org Thu Apr 26 14:53:49 2018 From: robin at icir.org (Robin Sommer) Date: Thu, 26 Apr 2018 14:53:49 -0700 Subject: [Bro-Dev] Final Broker branch testing In-Reply-To: References: <4A4BEC38-8163-454C-80C7-0192B59144F7@icir.org> Message-ID: <20180426215349.GI14203@icir.org> On Thu, Apr 26, 2018 at 14:30 -0700, you wrote: > It might be. I am honestly not sure - I suspect that this still will > mean that some places might not be able to easily use Bro > anymore--adding external package sources does not seem to be a viable > option everywhere. Is it a feasible compromise to allow cmake 2.8 if we don't need to build CAF? So either people have cmake 3.0 or they need to build CAF themselves and say --with-caf=...? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at corelight.com Thu Apr 26 14:54:55 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 26 Apr 2018 16:54:55 -0500 Subject: [Bro-Dev] Final Broker branch testing In-Reply-To: References: <4A4BEC38-8163-454C-80C7-0192B59144F7@icir.org> Message-ID: On 4/26/18 4:30 PM, Johanna Amann wrote: > It might be. I am honestly not sure - I suspect that this still will > mean that some places might not be able to easily use Bro > anymore--adding external package sources does not seem to be a viable > option everywhere. They could still build CMake themselves? (CMake itself is easy to build) The options to go forward are: (1) Users whose OS has insufficient CMake will need to compile/obtain a newer one. This would mostly be Ubuntu 14.04 (LTS until April 2019) and RHEL/CentOS 6+7 (LTS for these is in the 2020-2024 range). (2) We go back to CMake 2.8.12 and have people compile CAF themselves. (Or maybe we could conditionally require only 2.8.12 users to compile CAF and others get the embedded CAF). (3) I need to try to hack our CMake system more to try to get back down to 2.8.12 while still being able to embed CAF. I can give (3) a bit more time to see if I didn't miss something (the line I was drawing before was having to manually do platform-dependent RPATH manipulation), though would be nice to hear more feedback on the approaches. > As a side-note, it also looks like that means that we cannot provide > binary packages for RedHat/CentOS anymore. We should be able to use whatever we want to create the binary package, shouldn't we? Or do you mean it wouldn't be accepted as part of the official repos even if it's just a build-time dependency? - Jon From vlad at es.net Thu Apr 26 20:01:05 2018 From: vlad at es.net (Vlad Grigorescu) Date: Fri, 27 Apr 2018 03:01:05 +0000 Subject: [Bro-Dev] How to deal with stale branches? In-Reply-To: <5c5c49d2-60fc-17bd-8938-db37b9523c0c@corelight.com> References: <5c5c49d2-60fc-17bd-8938-db37b9523c0c@corelight.com> Message-ID: Yeah, that's certainly one option, but I think it'd be hard for people to find. On Thu, Apr 26, 2018 at 8:15 PM, Jon Siwek wrote: > > > On 4/26/18 11:06 AM, Vlad Grigorescu wrote: > > I'm torn between deleting the branches, in an effort to not clog up git >> with unneeded branches, and leaving them around or perhaps archiving them >> somewhere, in order to not completely lose the work in case it's of value >> to someone down the road. >> >> I'm curious if anyone has thoughts on the best way to proceed. >> > > Maybe delete the branch from the official git repo and push it to your own > github fork. > > - Jon > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180427/c940a526/attachment.html From jazoff at illinois.edu Fri Apr 27 08:35:56 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 27 Apr 2018 15:35:56 +0000 Subject: [Bro-Dev] Final Broker branch testing In-Reply-To: References: Message-ID: <971C97B2-4AAF-4267-8FE5-BC46437EEE8C@illinois.edu> > On Apr 26, 2018, at 4:25 PM, Azoff, Justin S wrote: > > Other than that things are working great. Cluster::publish_hrw is distributing data cross proxies perfectly: > > # for x in 1 2 3; do broctl print Scan::attacks proxy-$x|grep attempts= -c;done > 3304 > 3405 > 3397 > > # cat /bro/logs/current/notice.log |bro-cut note peer_descr|grep Scan::|cut -f 2|sort|uniq -c > 454 proxy-1 > 463 proxy-2 > 417 proxy-3 > > Once this is stable for a bit i'll start trying things like killing a proxy and verifying that things failover. > I tested this and it works great! I killed proxy-3, and cluster.log immediately logged it as 'node down' The publish_hrw sent the new data to proxy 1 and 2 and when proxy 3 was restarted it rejoined and started receiving data again. The next step is 2+ managers and 2+ loggers and we can finally have a bro cluster with no SPOF :) ? Justin Azoff From robin at icir.org Fri Apr 27 08:40:07 2018 From: robin at icir.org (Robin Sommer) Date: Fri, 27 Apr 2018 08:40:07 -0700 Subject: [Bro-Dev] How to deal with stale branches? In-Reply-To: References: <5c5c49d2-60fc-17bd-8938-db37b9523c0c@corelight.com> Message-ID: <20180427154007.GF24938@icir.org> Personally I don't really mind such branches sticking around for reference purposes. We have plenty stale branches anyways all over, it would probably be more to clean up those (looking at myself there, too :) Robin On Fri, Apr 27, 2018 at 03:01 +0000, you wrote: > Yeah, that's certainly one option, but I think it'd be hard for people to > find. > > On Thu, Apr 26, 2018 at 8:15 PM, Jon Siwek wrote: > > > > > > > On 4/26/18 11:06 AM, Vlad Grigorescu wrote: > > > > I'm torn between deleting the branches, in an effort to not clog up git > >> with unneeded branches, and leaving them around or perhaps archiving them > >> somewhere, in order to not completely lose the work in case it's of value > >> to someone down the road. > >> > >> I'm curious if anyone has thoughts on the best way to proceed. > >> > > > > Maybe delete the branch from the official git repo and push it to your own > > github fork. > > > > - Jon > > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From robin at icir.org Fri Apr 27 08:55:51 2018 From: robin at icir.org (Robin Sommer) Date: Fri, 27 Apr 2018 08:55:51 -0700 Subject: [Bro-Dev] Final Broker branch testing In-Reply-To: References: <4A4BEC38-8163-454C-80C7-0192B59144F7@icir.org> Message-ID: <20180427155551.GI24938@icir.org> On Thu, Apr 26, 2018 at 16:54 -0500, you wrote: > (1) Users whose OS has insufficient CMake will need to compile/obtain a newer one. > (2) We go back to CMake 2.8.12 and have people compile CAF themselves. > (Or maybe we could conditionally require only 2.8.12 users to compile > CAF and others get the embedded CAF). > (3) I need to try to hack our CMake system more to try to get back down > to 2.8.12 while still being able to embed CAF. If there's something quick that ends up making (3) work, that'd be ideal of course, but I don't think it's worth spending much time on, given that there are reasonable ways to get a more recent cmake. I wouldn't want to go back to not shipping CAF at all, but if we can tell cmake that 2.8.12 is fine if users build CAF themselves, that would be the 2nd best option I think. (1) ist worst case, which still isn't too bad IMO, unless it does actually prevent us from building binary packages for RH, that would be a problem. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From vern at icir.org Mon Apr 30 07:10:36 2018 From: vern at icir.org (Vern Paxson) Date: Mon, 30 Apr 2018 07:10:36 -0700 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180426213924.c4hyoo2d3oc5kc3p@user158.sys.ICSI.Berkeley.EDU> (Thu, 26 Apr 2018 14:39:24 PDT). Message-ID: <20180430141036.692782C413F@rock.ICSI.Berkeley.EDU> > It especially feels weird to me if v + e and > v += e are operations that perform something completely different. Yeah, I hear you. OTOH, I *really* would like a succinct way to say "add this to the end of this vector", it's such a common idiom. Robin and I discussed this a bit. Our ultimate thinking was along the lines of: (1) likely there's no significant use right now of "v op e" semantics, given how none of us initially remembered it (2) down the line such semantics could be quite handy if we start pushing on vector operations for doing statistical or ML computations (that's why I added "v op e" in the first place, inspired by how easy R makes these), (3) "v += e" really is a nice append-to-vector idiom (4) so how about we change "v op e" into something else to avoid the conceptual clash w/ v += e, while still having it available for the possible uses in (2) above? The question then was what would be the new "v op e" syntax. The best we could come up with (which we both found not-too-awful) is "vector(v op e)". Wrapped in "vector(...)", the operation becomes the current semantics (apply "op e" separately to each element of v). "v op e" by itself would now be an error (which could point the user at the "vector(...)" syntax as possibly providing what they're looking for). "v += e" would be "append e to v". Do you buy that? Vern From vern at icir.org Mon Apr 30 07:10:40 2018 From: vern at icir.org (Vern Paxson) Date: Mon, 30 Apr 2018 07:10:40 -0700 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180426215318.y2flrutl2ziu6ubq@user158.sys.ICSI.Berkeley.EDU> (Thu, 26 Apr 2018 14:53:18 PDT). Message-ID: <20180430141040.600A62C413F@rock.ICSI.Berkeley.EDU> > I think I actually would prefer just keeping add/delete, at least for > sets, and not introduce the plus-syntax. Okay, I can live with this as long as '|' and '-' support add-to-set and remove-from-set. But I think those have to work, given we'll enable them for operations on two sets. Vern From robin at icir.org Mon Apr 30 07:54:53 2018 From: robin at icir.org (Robin Sommer) Date: Mon, 30 Apr 2018 07:54:53 -0700 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180430141036.692782C413F@rock.ICSI.Berkeley.EDU> References: <20180426213924.c4hyoo2d3oc5kc3p@user158.sys.ICSI.Berkeley.EDU> <20180430141036.692782C413F@rock.ICSI.Berkeley.EDU> Message-ID: <20180430145453.GB60428@icir.org> On Mon, Apr 30, 2018 at 07:10 -0700, you wrote: > "vector(v op e)". Wrapped in "vector(...)", the operation becomes the > current semantics (apply "op e" separately to each element of v). One additional piece of context here: That vector(...) syntax could then be used more broadly in the sense of creating a different semantic context for the operations inside. That kind of opens up a whole new set of of type-specific operator meanings, without affecting current/standard ones (it's like introducing R inside parentheses :-). It's not the super-great, but at least it's explicit and we couldn't come up with anything better if we want to have such operations as operators. Might work for some other types as well. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at corelight.com Mon Apr 30 08:10:03 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Mon, 30 Apr 2018 10:10:03 -0500 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180430141036.692782C413F@rock.ICSI.Berkeley.EDU> References: <20180430141036.692782C413F@rock.ICSI.Berkeley.EDU> Message-ID: <71c9e23e-dc85-015f-6217-46986caa39a1@corelight.com> On 4/30/18 9:10 AM, Vern Paxson wrote: > The question then was what would be the new "v op e" syntax. > The best we could come up with (which we both found not-too-awful) is > "vector(v op e)". Wrapped in "vector(...)", the operation becomes the > current semantics (apply "op e" separately to each element of v). Maybe "vectorize(v op e)" ? Implies implementing via SIMD instructions. > "v op e" by itself would now be an error (which could point the user > at the "vector(...)" syntax as possibly providing what they're looking > for). "v += e" would be "append e to v". That still seems odd to me. If "v += e" means "append", then I might expect "v + e" to do the same, except producing a new value w/ original vector not modified. Maybe that's a less common use-case, though, and so "v op e" being an error would be less weird than suddenly changing the meaning of that operation. - Jon From robin at icir.org Mon Apr 30 08:13:22 2018 From: robin at icir.org (Robin Sommer) Date: Mon, 30 Apr 2018 08:13:22 -0700 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180430141040.600A62C413F@rock.ICSI.Berkeley.EDU> References: <20180426215318.y2flrutl2ziu6ubq@user158.sys.ICSI.Berkeley.EDU> <20180430141040.600A62C413F@rock.ICSI.Berkeley.EDU> Message-ID: <20180430151322.GC60428@icir.org> On Mon, Apr 30, 2018 at 07:10 -0700, you wrote: > Okay, I can live with this as long as '|' and '-' support add-to-set and > remove-from-set. But I think those have to work, given we'll enable them > for operations on two sets. Well, my vote then remains not adding new set operators for add/delete, so that we don't have multiple ways to do the same thing. Just looked at Python again, as a data point: That's what they do, too. There are '|'/'&'/'-' for set/set operations, but no versions of those for individual elements (they do that through methods instead; add/delete are kind of our version of methods). Same for Ruby. I looked around for a few more minutes for other languages, but didn't immediately find any that even have any set operators at all (only methods/functions for union/intersection/etc.). Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From johanna at icir.org Mon Apr 30 08:15:46 2018 From: johanna at icir.org (Johanna Amann) Date: Mon, 30 Apr 2018 11:15:46 -0400 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180430151322.GC60428@icir.org> References: <20180426215318.y2flrutl2ziu6ubq@user158.sys.ICSI.Berkeley.EDU> <20180430141040.600A62C413F@rock.ICSI.Berkeley.EDU> <20180430151322.GC60428@icir.org> Message-ID: <55889C50-C360-4C82-B78D-2E5F4F5B43FD@icir.org> On 30 Apr 2018, at 11:13, Robin Sommer wrote: > On Mon, Apr 30, 2018 at 07:10 -0700, you wrote: > >> Okay, I can live with this as long as '|' and '-' support add-to-set and >> remove-from-set. But I think those have to work, given we'll enable them >> for operations on two sets. > > Well, my vote then remains not adding new set operators for > add/delete, so that we don't have multiple ways to do the same thing. > Just looked at Python again, as a data point: That's what they do, > too. There are '|'/'&'/'-' for set/set operations, but no versions of > those for individual elements (they do that through methods instead; > add/delete are kind of our version of methods). Same for Ruby. I > looked around for a few more minutes for other languages, but didn't > immediately find any that even have any set operators at all (only > methods/functions for union/intersection/etc.). Yup, I think I concur. Johanna From jsiwek at corelight.com Mon Apr 30 08:21:43 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Mon, 30 Apr 2018 10:21:43 -0500 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180430145453.GB60428@icir.org> References: <20180426213924.c4hyoo2d3oc5kc3p@user158.sys.ICSI.Berkeley.EDU> <20180430141036.692782C413F@rock.ICSI.Berkeley.EDU> <20180430145453.GB60428@icir.org> Message-ID: <08866f78-8030-3ae3-43eb-bc290afd9b5e@corelight.com> On 4/30/18 9:54 AM, Robin Sommer wrote: > One additional piece of context here: That vector(...) syntax could > then be used more broadly in the sense of creating a different > semantic context for the operations inside. That kind of opens up a > whole new set of of type-specific operator meanings, without affecting > current/standard ones (it's like introducing R inside parentheses :-). Yeah, framing it as type-specific operators makes sense. > It's not the super-great, but at least it's explicit and we couldn't > come up with anything better if we want to have such operations as > operators. Might work for some other types as well. Isn't the existing "v + e" an alternative that one could say is "better" ? Other languages also take that to mean "vectorized/array operation". I think for Bro, it's just that vectorized/array operations will be a less common use-case than appending and so there's incentive to make the later more succinct? Just to put it out there again: I wouldn't mind something like "append(v, e)" and leaving the existing vector operations like "v + e" alone. It may make the common case more verbose, though there's also less ambiguities. - Jon From robin at icir.org Mon Apr 30 08:34:20 2018 From: robin at icir.org (Robin Sommer) Date: Mon, 30 Apr 2018 08:34:20 -0700 Subject: [Bro-Dev] set and vector operators In-Reply-To: <71c9e23e-dc85-015f-6217-46986caa39a1@corelight.com> <08866f78-8030-3ae3-43eb-bc290afd9b5e@corelight.com> Message-ID: <20180430153420.GA70745@icir.org> On Mon, Apr 30, 2018 at 10:10 -0500, you wrote: > That still seems odd to me. If "v += e" means "append", then I might > expect "v + e" to do the same, except producing a new value w/ original > vector not modified. Yeah, I agree with that actually. Getting rid of the current "v + e" semantics (which nobody knows/uses anyways) means we can turn both '+' und '+=' into add-element-to-vector operations. Question is if we want to do that right away, or wait for a version for deprecating/flagging current usage of "v + e" (although I doubt there's any!) On Mon, Apr 30, 2018 at 10:21 -0500, you wrote: > Isn't the existing "v + e" an alternative that one could say is "better" ? Only if we don't want "+" to mean "append element to vector". If we switch '+' to meaning that, then there's nothing good left for the operation "add e to all vector elements". And the "append" operation indeed seems more common in Bro than the element-wise operation, so I'd prefer switching. > Just to put it out there again: I wouldn't mind something like "append(v, e)" I don't really like the free function style very much, what I could see is using methods: "v.append(e)". I actually like that style quite a bit. But as I said to Vern last week, it'd be a bit odd to introduce that pattern to Bro now, we don't use that style anywhere else. May be just too late for starting that. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From klehigh at iu.edu Mon Apr 30 08:50:21 2018 From: klehigh at iu.edu (Keith Lehigh) Date: Mon, 30 Apr 2018 11:50:21 -0400 Subject: [Bro-Dev] Final Broker branch testing In-Reply-To: References: Message-ID: <22D27EAE-F35F-4205-9389-EBF391C009DA@iu.edu> I?m able to get this built and running on FreeBSD 11.1 . - Keith On 26 Apr 2018, at 11:16, Jon Siwek wrote: > The latest version of the new Broker-ized cluster/communication system > for Bro in 'topic/actor-system' branch is wrapping up and, in my > opinion, ready to be merged into Bro's 'master' branch. > > However, since it's such a big change, I'd like a last round of feedback > before merging. If you want to test, the build process should now be as > simple as: > > $ git clone --recursive --branch=topic/actor-system git://git.bro.org/bro > $ cd bro && ./configure && make > > Configuring BroControl is not any different from before. > > If you had custom scripts, they may require porting. There's a guide > and examples for that at [1] and [2] (hyperlinks in those docs will > render more nicely when it's up on bro.org). > > Though, for this round of testing, I'd be most interested just in any > general stability issues or major feature breakages from a vanilla Bro > installation. Mild performance issues, minor bugs, or other issues w/ > porting custom scripts are things I think we can iron out even after > merging into 'master'. > > - Jon > > [1] > https://github.com/bro/bro/blob/topic/actor-system/doc/frameworks/broker.rst > [2] https://github.com/bro/bro/tree/topic/actor-system/doc/frameworks/broker > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3740 bytes Desc: S/MIME digital signature Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180430/7f577cf4/attachment-0001.bin