From haas at informatik.uni-hamburg.de Thu Oct 11 00:46:22 2018 From: haas at informatik.uni-hamburg.de (Steffen Haas) Date: Thu, 11 Oct 2018 09:46:22 +0200 Subject: [Bro-Dev] Event handling within script modules Message-ID: Not sure if this is a bug or I am using it wrong. Consider the following snippet. I start it using `sudo ./bin/bro -i eth0 module_test`. *File: share/bro/site/module_test/raise.bro* ``` module module_test; # Remove the module to make the example work # OR define the event handler within another module export { global my_event: event(a: int, s: string); } event raise_my_event() { local a: int = 42; local s: string = "Hello World"; print(fmt("Raising event with a=%d and s=%s", a, s)); event my_event(a, s); } #module module2; event my_event(a: int, s: string) { print(fmt("Received event with a=%d and s=%s", a, s)); } event bro_init() { local w: interval = 10sec; print(fmt("Starting example, waiting for %s", w)); #"raise_my_event()" is executed immediately schedule w {raise_my_event()}; } ``` # Problem 1: Event Handling within module The handler for "my_event" is not invoked as long as it is defined within the same module "module_test". - Workaround a): Do not use modules at all - Workaround b): Define the handler in a different module But why should not it be allowed to raise and handle events within the same module? # Problem 2: Direct execution of scheduled events At least within this example, I expected the "raise_my_event" to be invoked after 10 seconds. However, it is done immediately. Am I using the schedule wrong? I am running the current master (0f550806252b46d3e13be24cd2ab4bb8a63bf49b). Any help appreciated. Best, Steffen. -- M.Sc. Steffen Haas Research Assistant Phone: +49 40 42883 2353 IT-Sicherheit und Sicherheitsmanagement (ISS) Universit?t Hamburg Fachbereich Informatik Vogt-K?lln-Stra?e 30 22527 Hamburg Deutschland From jsiwek at corelight.com Thu Oct 11 07:40:41 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 11 Oct 2018 10:40:41 -0400 Subject: [Bro-Dev] Event handling within script modules In-Reply-To: References: Message-ID: On Thu, Oct 11, 2018 at 3:53 AM Steffen Haas wrote: > # Problem 1: Event Handling within module > The handler for "my_event" is not invoked as long as it is defined > within the same module "module_test". > - Workaround a): Do not use modules at all > - Workaround b): Define the handler in a different module > But why should not it be allowed to raise and handle events within the > same module? This is related to issues discussed at [1]. You can get the behavior you want, but the short explanation/recommendation is to always use the module prefix when raising/handling the event. So, in your example, when you raise and handle the event, explicitly add "module_test::" even when inside the module. > # Problem 2: Direct execution of scheduled events > At least within this example, I expected the "raise_my_event" to be > invoked after 10 seconds. However, it is done immediately. Am I using > the schedule wrong? When there's no input sources (not listening on an interface) and you're just running a script, timers may get processed as fast as possible, but you can add this and you'll then see timers behave based on wall clock: redef exit_only_after_terminate = T; - Jon [1] https://github.com/bro/bro/issues/163 From seth at corelight.com Fri Oct 12 15:09:20 2018 From: seth at corelight.com (Seth Hall) Date: Fri, 12 Oct 2018 18:09:20 -0400 Subject: [Bro-Dev] "bro" name is non-inclusive? (Contribution is rejected based ) In-Reply-To: References: Message-ID: <16116DB3-7CEA-4BC2-A0E2-18951E95DD24@corelight.com> Hi Vitaly! We renamed Bro to Zeek a couple of days ago. You should be good to reach out to the person that runs the ua-parser project and let them know so that your parser can be included on their webpage. If you need to give evidence of the name change, it's provided here: http://blog.bro.org/2018/10/renaming-bro-project_11.html Thanks for writing the uap plugin! .Seth -- Seth Hall * Corelight, Inc * www.corelight.com > On Mar 9, 2018, at 12:55 AM, Vitaly Repin wrote: > > Hello, > > I would like to bring your attention to one strange and unexpected problem I faced > while trying to include user agent parser for bro I have developed (plugin: https://github.com/vitalyrepin/uap-bro ) into the list of uap implementations https://github.com/ua-parser > > The req > -- > WBR & WBW, Vitaly > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20181012/7ea49f4c/attachment.html From leres at ee.lbl.gov Fri Oct 12 16:26:37 2018 From: leres at ee.lbl.gov (Craig Leres) Date: Fri, 12 Oct 2018 16:26:37 -0700 Subject: [Bro-Dev] Building Bro with openssl 1.1.1 Message-ID: <8fe01eea-323c-906c-4e95-0a79fabf6ed9@ee.lbl.gov> The FreeBSD security/bro port does not build under the latest FreeBSD 12.0 alpha due to compile errors in the X509 module. I verified that this is due to openssl 1.1.1 by building on 11.2-RELEASE with the openssl111 port. I've attached an example failing build log. Is openssl 1.1.1 support on the roadmap? Craig -------------- next part -------------- A non-text attachment was scrubbed... Name: bro-2.5.5.log.gz Type: application/gzip Size: 51147 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20181012/d4afc306/attachment-0001.bin From jsiwek at corelight.com Fri Oct 12 18:52:57 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Fri, 12 Oct 2018 20:52:57 -0500 Subject: [Bro-Dev] Building Bro with openssl 1.1.1 In-Reply-To: <8fe01eea-323c-906c-4e95-0a79fabf6ed9@ee.lbl.gov> References: <8fe01eea-323c-906c-4e95-0a79fabf6ed9@ee.lbl.gov> Message-ID: On Fri, Oct 12, 2018 at 6:33 PM Craig Leres wrote: > Is openssl 1.1.1 support on the roadmap? Bro 2.6 will support OpenSSL 1.1. Possibly only a couple more weeks of beta remain before that release. - Jon From vlad at es.net Mon Oct 15 13:26:15 2018 From: vlad at es.net (Vlad Grigorescu) Date: Mon, 15 Oct 2018 20:26:15 +0000 Subject: [Bro-Dev] SSH Capabilities Bug: Fix for 2.6? Message-ID: During BroCon, someone brought a bug in the SSH analyzer to my attention. The SSH Capabilities record has the following field, which is being set incorrectly: ## Are these the capabilities of the server? > is_server: bool; > > result->Assign(6, new Val(${msg.is_orig}, TYPE_BOOL)); > Obviously, I'd like to fix this. I'm curious to hear thoughts about getting this into 2.6. I know that the hassh package currently works around the broken logic. --Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20181015/8a704da8/attachment.html From jsiwek at corelight.com Mon Oct 15 14:19:23 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Mon, 15 Oct 2018 16:19:23 -0500 Subject: [Bro-Dev] SSH Capabilities Bug: Fix for 2.6? In-Reply-To: References: Message-ID: On Mon, Oct 15, 2018 at 3:33 PM Vlad Grigorescu wrote: > The SSH Capabilities record has the following field, which is being set incorrectly: > >> ## Are these the capabilities of the server? >> is_server: bool; > >> result->Assign(6, new Val(${msg.is_orig}, TYPE_BOOL)); > > Obviously, I'd like to fix this. I'm curious to hear thoughts about getting this into 2.6. Yes, that seems like a bug fix that can be included in 2.6. Do you want to make a PR for that and verify/update any unit test baselines that change? - Jon From vlad at es.net Mon Oct 15 15:21:34 2018 From: vlad at es.net (Vlad Grigorescu) Date: Mon, 15 Oct 2018 17:21:34 -0500 Subject: [Bro-Dev] SSH Capabilities Bug: Fix for 2.6? In-Reply-To: References: Message-ID: Sure, I'll do that. On Mon, Oct 15, 2018 at 16:19 Jon Siwek wrote: > On Mon, Oct 15, 2018 at 3:33 PM Vlad Grigorescu wrote: > > > The SSH Capabilities record has the following field, which is being set > incorrectly: > > > >> ## Are these the capabilities of the server? > >> is_server: bool; > > > >> result->Assign(6, new Val(${msg.is_orig}, TYPE_BOOL)); > > > > Obviously, I'd like to fix this. I'm curious to hear thoughts about > getting this into 2.6. > > Yes, that seems like a bug fix that can be included in 2.6. Do you > want to make a PR for that and verify/update any unit test baselines > that change? > > - Jon > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20181015/03cdd03a/attachment.html From Tim.McMullan at sig.com Tue Oct 16 07:30:52 2018 From: Tim.McMullan at sig.com (McMullan, Tim) Date: Tue, 16 Oct 2018 14:30:52 +0000 Subject: [Bro-Dev] Intermittent bro crashes Message-ID: <74f45f735303454199053f2ba9e4996d@msx.bala.susq.com> Good morning! We have been seeing occasional core dumps from bro, currently running on 2.5-870. We've tried a few things to reproduce it on-demand but haven't been successful. We were wondering if you might have some insight into the crash. This is the backtrace we get: #0 TableEntryVal::ExpireAccessTime (this=0x9b349bd32ebb5614) at /hostname/bro-devel-src/src/Val.h:741 #1 TableVal::DoExpire (this=0x3955a40, t=1539447574.1403711) at /hostname/bro-devel-src/src/Val.cc:2353 #2 0x00000000006a34cb in PQ_TimerMgr::DoAdvance (this=0x2e520c0, new_t=1539447574.1403711, max_expire=300) at /hostname/bro-devel-src/src/Timer.cc:166 #3 0x0000000000664d38 in expire_timers (src_ps=src_ps at entry=0x2eab8c0) at /hostname/bro-devel-src/src/Net.cc:232 #4 0x0000000000664de1 in net_packet_dispatch (t=1539447574.1403711, pkt=pkt at entry=0x2eab8f0, src_ps=src_ps at entry=0x2eab8c0) at /hostname/bro-devel-src/src/Net.cc:249 #5 0x0000000000925a89 in iosource::PktSrc::Process (this=0x2eab8c0) at /hostname/bro-devel-src/src/iosource/PktSrc.cc:263 #6 0x00000000006652c5 in net_run () at /hostname/bro-devel-src/src/Net.cc:315 #7 0x00000000005acab1 in main (argc=, argv=) at /hostname/bro-devel-src/src/main.cc:1227 Thank you for your help! --Tim ________________________________ IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20181016/2d5526e0/attachment.html From vlad at es.net Tue Oct 16 08:37:49 2018 From: vlad at es.net (Vlad Grigorescu) Date: Tue, 16 Oct 2018 15:37:49 +0000 Subject: [Bro-Dev] SSH Capabilities Bug: Fix for 2.6? In-Reply-To: References: Message-ID: Just for anyone who wanted some closure on this, I've submitted the PR: https://github.com/bro/bro/pull/191 On Mon, Oct 15, 2018 at 10:21 PM Vlad Grigorescu wrote: > Sure, I'll do that. > On Mon, Oct 15, 2018 at 16:19 Jon Siwek wrote: > >> On Mon, Oct 15, 2018 at 3:33 PM Vlad Grigorescu wrote: >> >> > The SSH Capabilities record has the following field, which is being set >> incorrectly: >> > >> >> ## Are these the capabilities of the server? >> >> is_server: bool; >> > >> >> result->Assign(6, new Val(${msg.is_orig}, TYPE_BOOL)); >> > >> > Obviously, I'd like to fix this. I'm curious to hear thoughts about >> getting this into 2.6. >> >> Yes, that seems like a bug fix that can be included in 2.6. Do you >> want to make a PR for that and verify/update any unit test baselines >> that change? >> >> - Jon >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20181016/c49ba198/attachment-0001.html From jsiwek at corelight.com Tue Oct 16 09:01:47 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Tue, 16 Oct 2018 11:01:47 -0500 Subject: [Bro-Dev] Intermittent bro crashes In-Reply-To: <74f45f735303454199053f2ba9e4996d@msx.bala.susq.com> References: <74f45f735303454199053f2ba9e4996d@msx.bala.susq.com> Message-ID: On Tue, Oct 16, 2018 at 9:38 AM McMullan, Tim wrote: > We have been seeing occasional core dumps from bro, currently running on 2.5-870. May be nice to try most recent master version to see if it still pops up. > We?ve tried a few things to reproduce it on-demand but haven?t been successful. We were wondering if you might have some insight into the crash. This is the backtrace we get: > > #0 TableEntryVal::ExpireAccessTime (this=0x9b349bd32ebb5614) at /hostname/bro-devel-src/src/Val.h:741 > > #1 TableVal::DoExpire (this=0x3955a40, t=1539447574.1403711) at /hostname/bro-devel-src/src/Val.cc:2353 I don't see an immediate culprit there -- it's the table entry expiration algorithm that I guess is well-traveled by most people running Bro, so maybe need to work on narrowing down a way to reproduce or what specific script is triggering it. Do you have any custom scripts that make use of &expire_func or &{read, write, create}_expire table attributes? That could be a first place to inspect. I also fixed a bug [1] in related code just last week, but I'd expect that to give a different stack trace if it was the same problem here (still doesn't hurt to try to rule that out as a contributing factor by testing w/ latest git/master). - Jon [1] https://github.com/bro/bro/commit/8792f5545cd5b7de433d0eee510fde94371fdee3 From vlad at es.net Tue Oct 16 12:22:22 2018 From: vlad at es.net (Vlad Grigorescu) Date: Tue, 16 Oct 2018 19:22:22 +0000 Subject: [Bro-Dev] bro-pkg Bro version requirements Message-ID: It strikes me that as Bro development marches on, package maintainers don't have great choices in terms of maintaining compatibility with multiple Bro versions. For JA3, to maintain compatibility, you have to do something like this, due to the SSL event change: @if ( Version::at_least("2.6") ) > event ssl_client_hello(c: connection, version: count, record_version:count, > possible_ts: time, client_random: string, session_id: string, ciphers: > index_vec, comp_methods: vector of count) &priority=1 > @else > event ssl_client_hello(c: connection, version: count, possible_ts: time, > client_random: string, session_id: string, ciphers: index_vec) &priority=1 > @endif > That works, but I worry that the overhead of trying to maintain that will grow out of hand. I'm wondering if there's a better mechanism for this. A naive approach might be to include an option in the package metadata, which specifies minimum/maximum Bro versions that it requires. The installer, then, would simply install the latest version that supports your Bro version. I don't want to overcomplicate things, but it does feel like there's a mechanism that's currently missing. Any other thoughts? --Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20181016/950f8bb2/attachment.html From jsiwek at corelight.com Tue Oct 16 15:00:03 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Tue, 16 Oct 2018 17:00:03 -0500 Subject: [Bro-Dev] bro-pkg Bro version requirements In-Reply-To: References: Message-ID: On Tue, Oct 16, 2018 at 2:23 PM Vlad Grigorescu wrote: >> @if ( Version::at_least("2.6") ) >> event ssl_client_hello(c: connection, version: count, record_version:count, possible_ts: time, client_random: string, session_id: string, ciphers: index_vec, comp_methods: vector of count) &priority=1 >> @else >> event ssl_client_hello(c: connection, version: count, possible_ts: time, client_random: string, session_id: string, ciphers: index_vec) &priority=1 >> @endif > > > That works, but I worry that the overhead of trying to maintain that will grow out of hand. I don't have the same worry at least at the moment. The maintenance overhead doesn't seem that great: those are fairly trivial lines of code to understand. The growth potential also seems low considering we really don't have cycles to maintain past the last version of Bro ourselves, so I wouldn't expect packagers to maintain their code to work against all possible versions. e.g. in the already-unlikely chance the same event introduces a breaking change in 2.7, the branch complexity would not grow if they just throw out the 2.5 branch. > I'm wondering if there's a better mechanism for this. A naive approach might be to include an option in the package metadata, which specifies minimum/maximum Bro versions that it requires. The installer, then, would simply install the latest version that supports your Bro version. Interesting thought, but my hunch is it just makes the complexity implicit instead of explicit. e.g. if metadata claims it works for 2.6, it's tricky the answer why that is, but with the code above you can easily search for those specific areas that are version-sensitive. I'm thinking it may be best to wait and see if there is actually a problem (lots of people complaining) before trying to decide how to solve anything. - Jon From Tim.McMullan at sig.com Wed Oct 17 05:44:00 2018 From: Tim.McMullan at sig.com (McMullan, Tim) Date: Wed, 17 Oct 2018 12:44:00 +0000 Subject: [Bro-Dev] Intermittent bro crashes In-Reply-To: References: <74f45f735303454199053f2ba9e4996d@msx.bala.susq.com> Message-ID: Hi Jon, Our scripts do make extensive use of &read_expire and &create_expire, but we don't have expire_func anywhere. I'll check against the latest master and see how it goes! Thanks! --Tim -----Original Message----- From: Jon Siwek [mailto:jsiwek at corelight.com] Sent: Tuesday, October 16, 2018 12:02 PM To: McMullan, Tim Cc: ; Wallior, Julien Subject: Re: [Bro-Dev] Intermittent bro crashes On Tue, Oct 16, 2018 at 9:38 AM McMullan, Tim wrote: > We have been seeing occasional core dumps from bro, currently running on 2.5-870. May be nice to try most recent master version to see if it still pops up. > We?ve tried a few things to reproduce it on-demand but haven?t been successful. We were wondering if you might have some insight into the crash. This is the backtrace we get: > > #0 TableEntryVal::ExpireAccessTime (this=0x9b349bd32ebb5614) at /hostname/bro-devel-src/src/Val.h:741 > > #1 TableVal::DoExpire (this=0x3955a40, t=1539447574.1403711) at /hostname/bro-devel-src/src/Val.cc:2353 I don't see an immediate culprit there -- it's the table entry expiration algorithm that I guess is well-traveled by most people running Bro, so maybe need to work on narrowing down a way to reproduce or what specific script is triggering it. Do you have any custom scripts that make use of &expire_func or &{read, write, create}_expire table attributes? That could be a first place to inspect. I also fixed a bug [1] in related code just last week, but I'd expect that to give a different stack trace if it was the same problem here (still doesn't hurt to try to rule that out as a contributing factor by testing w/ latest git/master). - Jon [1] https://github.com/bro/bro/commit/8792f5545cd5b7de433d0eee510fde94371fdee3 ________________________________ IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses. From vern at corelight.com Mon Oct 29 11:49:53 2018 From: vern at corelight.com (Vern Paxson) Date: Mon, 29 Oct 2018 11:49:53 -0700 Subject: [Bro-Dev] consistency checking for attributes Message-ID: <201810291849.w9TInrJK012017@fruitcake.ICSI.Berkeley.EDU> Attributes currently receive essentially no consistency checking. For example, executing this script: global a: count &default = 10 &default = 9 &optional &log &add_func = function(d: double, t: time): count { return 3; }; print a; simply results in: error in /Users/vern/tmp/attr-type-check.bro, line 7: value used but not set (a) I'm planning to add basic consistency checking, which will look for (1) attributes that are repeated (which doesn't appear to be meaningful for any of them) and (2) attributes that don't make sense in a given context, like the ones listed above. I'm thinking of implementing this as an internal table of meta-attributes, i.e., each attribute type, like ATTR_OPTIONAL, will have its own attributes like whether it requires a record context, only makes sense for aggregates, etc. Here are the ones that come to mind, based on looking at the attributes at https://www.bro.org/sphinx/script-reference/attributes.html with examples in parens: applies to global variable (&redef) to global type (&redef) to event handler (&priority) to record field (&log) to indexed type (&default) to global indexed type (&add_func) to type with expirable entries (&expire_func) to a file (&rotate_interval) Any feedback? Vern From jsiwek at corelight.com Mon Oct 29 12:15:21 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Mon, 29 Oct 2018 14:15:21 -0500 Subject: [Bro-Dev] Any 2.6 release blockers? Message-ID: Anyone have any last minute issues that would block the 2.6 release? Or any remaining changes (bug fixes) they want to still try to get into it? - Jon From vlad at es.net Mon Oct 29 12:24:32 2018 From: vlad at es.net (Vlad Grigorescu) Date: Mon, 29 Oct 2018 19:24:32 +0000 Subject: [Bro-Dev] Any 2.6 release blockers? In-Reply-To: References: Message-ID: I'd really like to fix this: https://github.com/bro/bro/issues/195 I've gotten reports from a few people that that fills up the disk in environments that encrypt MySQL. I'll take one more crack at it now. --Vlad On Mon, Oct 29, 2018 at 7:22 PM Jon Siwek wrote: > Anyone have any last minute issues that would block the 2.6 release? > > Or any remaining changes (bug fixes) they want to still try to get into it? > > - Jon > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20181029/5293779c/attachment.html From vern at corelight.com Mon Oct 29 13:42:07 2018 From: vern at corelight.com (Vern Paxson) Date: Mon, 29 Oct 2018 13:42:07 -0700 Subject: [Bro-Dev] consistency checking for attributes In-Reply-To: <201810291849.w9TInrJK012017@fruitcake.ICSI.Berkeley.EDU> (Mon, 29 Oct 2018 11:49:53 PDT). Message-ID: <201810292042.w9TKg7Ds001064@fruitcake.ICSI.Berkeley.EDU> > I'm thinking of implementing this as an internal table of meta-attributes, > i.e., each attribute type, like ATTR_OPTIONAL, will have its own attributes > ... Looking into this further, it appears that the basic issue is that while Attributes::CheckAttr already tries to make sure that a given attribute makes sense, its checking is deficient. Some of this might be because it lacks information about whether the attribute is appearing in the context of a type or a variable. Also, Attributes::AddAttr already checks for repeated attributes, but it just silently replaces them. This works okay for a situation where a type is being refined by introducing updated attributes, but doesn't make sense when all of the attributes are presented together (like in my example); again, it needs more context to figure that out. Vern From vlad at es.net Mon Oct 29 14:06:33 2018 From: vlad at es.net (Vlad Grigorescu) Date: Mon, 29 Oct 2018 21:06:33 +0000 Subject: [Bro-Dev] Any 2.6 release blockers? In-Reply-To: References: Message-ID: Ok, just submitted: https://github.com/bro/bro/pull/198 On Mon, Oct 29, 2018 at 7:24 PM Vlad Grigorescu wrote: > I'd really like to fix this: https://github.com/bro/bro/issues/195 > > I've gotten reports from a few people that that fills up the disk in > environments that encrypt MySQL. I'll take one more crack at it now. > > --Vlad > > On Mon, Oct 29, 2018 at 7:22 PM Jon Siwek wrote: > >> Anyone have any last minute issues that would block the 2.6 release? >> >> Or any remaining changes (bug fixes) they want to still try to get into >> it? >> >> - Jon >> _______________________________________________ >> bro-dev mailing list >> bro-dev at bro.org >> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20181029/45d539fc/attachment.html From hosom at battelle.org Tue Oct 30 11:09:21 2018 From: hosom at battelle.org (Hosom, Stephen M) Date: Tue, 30 Oct 2018 18:09:21 +0000 Subject: [Bro-Dev] Config Framework Feedback Message-ID: I bumped into a limitation that was a little frustrating to work around with the config framework. It is inconvenient that values stored in files read by adding to Config::config_files are not available within the bro_init event. After reviewing how the Config framework works, I understand why this is the case, but it means that if you want to use configuration values on startup, you're not guaranteed to be working with the anticipated value. I think the introduction of an event that ensures that configuration files have been read by the time that the event fires might be worthwhile. I was wondering if anyone had any thoughts on how to resolve/work-around this issue. known.dat: Known::KnownServers 10.230.21.220,10.230.21.221 try-config.bro: module Known; redef Config::config_files += {"Known.dat"}; export { option KnownServers: set[addr] = set(); } event bro_init() { print KnownServers; } event bro_done() { print KnownServers; } ======= output ======= bro -r stream-1.pcap ./try-config.bro { } { 10.230.21.220, 10.230.21.221 } Thanks, Stephen From jazoff at illinois.edu Tue Oct 30 12:41:00 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 30 Oct 2018 19:41:00 +0000 Subject: [Bro-Dev] Config Framework Feedback In-Reply-To: References: Message-ID: <4F4FE1FA-8790-4701-8190-4F02D8EAAC19@illinois.edu> > On Oct 30, 2018, at 2:09 PM, Hosom, Stephen M wrote: > > I bumped into a limitation that was a little frustrating to work around with the config framework. > > > It is inconvenient that values stored in files read by adding to Config::config_files are not available within the bro_init event. After reviewing how the Config framework works, I understand why this is the case, but it means that if you want to use configuration values on startup, you're not guaranteed to be working with the anticipated value. > > > I think the introduction of an event that ensures that configuration files have been read by the time that the event fires might be worthwhile. I was wondering if anyone had any thoughts on how to resolve/work-around this issue. This is an issue, but probably not the one you are thinking of. even if configuration files were fully read by the time bro_init was ran, you'd still need to handle the value changing at runtime. If you need to run code when an option changes, you can use the Option::set_change_handler to make bro raise an event. Using this instead of bro_init would fix the startup problem, and handle it changing at any time. There's a test that uses this in testing/btest/core/option-priorities.bro and a few other files. ? Justin Azoff From hosom at battelle.org Tue Oct 30 12:58:27 2018 From: hosom at battelle.org (Hosom, Stephen M) Date: Tue, 30 Oct 2018 19:58:27 +0000 Subject: [Bro-Dev] Config Framework Feedback In-Reply-To: <4F4FE1FA-8790-4701-8190-4F02D8EAAC19@illinois.edu> References: , <4F4FE1FA-8790-4701-8190-4F02D8EAAC19@illinois.edu> Message-ID: I agree that more complex code on my side would definitely make this a non-issue. In my case, I was setting up something in bro_init that would be used repeatedly throughout execution, and you're right--my code never handled option updates. In this case, I opted to just use old redefable consts, as they will perform exactly as expected for any downstream users (changes require a bro restart). Honestly, I only brought this up at all because I can definitely see a person new to scripting wondering why their options do not have the expected values at bro_init time and troubleshooting for hours. Maybe it would be worthwhile to just add a note in the documentation for the config framework about it and leave it at that? ________________________________ From: Azoff, Justin S Sent: Tuesday, October 30, 2018 3:41:00 PM To: Hosom, Stephen M Cc: bro-dev at bro.org Subject: Re: [Bro-Dev] Config Framework Feedback Message received from outside the Battelle network. Carefully examine it before you open any links or attachments. > On Oct 30, 2018, at 2:09 PM, Hosom, Stephen M wrote: > > I bumped into a limitation that was a little frustrating to work around with the config framework. > > > It is inconvenient that values stored in files read by adding to Config::config_files are not available within the bro_init event. After reviewing how the Config framework works, I understand why this is the case, but it means that if you want to use configuration values on startup, you're not guaranteed to be working with the anticipated value. > > > I think the introduction of an event that ensures that configuration files have been read by the time that the event fires might be worthwhile. I was wondering if anyone had any thoughts on how to resolve/work-around this issue. This is an issue, but probably not the one you are thinking of. even if configuration files were fully read by the time bro_init was ran, you'd still need to handle the value changing at runtime. If you need to run code when an option changes, you can use the Option::set_change_handler to make bro raise an event. Using this instead of bro_init would fix the startup problem, and handle it changing at any time. There's a test that uses this in testing/btest/core/option-priorities.bro and a few other files. ? Justin Azoff From christian at corelight.com Tue Oct 30 13:17:57 2018 From: christian at corelight.com (Christian Kreibich) Date: Tue, 30 Oct 2018 13:17:57 -0700 Subject: [Bro-Dev] Config Framework Feedback In-Reply-To: References: Message-ID: <69986ffb-6abc-33fe-70a0-a523eeba7958@corelight.com> Hi folks, I would agree that it takes a bit of experimentation to figure out exactly when a change handler fires and how to reliably initialize or update things based on an option's value. Consider this: module Foo; export { option foo = F; } function foo_handler(ID: string, foo_new: bool): bool { print fmt("New foo: %s", foo_new); # Update stuff here based on foo's value # ... return foo_new; } event bro_init() { Option::set_change_handler("Foo::foo", foo_handler); } ... foo_handler doesn't get called when you simply run the script without redefing Config::config_files. When you do redef it, the handler fires both when the config file sets foo to T, and when it sets it to F. So you have to make sure that your initialization happens even when the handler doesn't get called, and you cannot write your handler assuming that the new value is actually different from the old one. These arguably aren't bugs, but imo they do take getting used to. Best, -C. From robin at corelight.com Wed Oct 31 08:05:55 2018 From: robin at corelight.com (Robin Sommer) Date: Wed, 31 Oct 2018 08:05:55 -0700 Subject: [Bro-Dev] consistency checking for attributes In-Reply-To: <201810291849.w9TInrJK012017@fruitcake.ICSI.Berkeley.EDU> References: <201810291849.w9TInrJK012017@fruitcake.ICSI.Berkeley.EDU> Message-ID: <20181031150555.GE19344@corelight.com> On Mon, Oct 29, 2018 at 11:49 -0700, Vern Paxson wrote: > I'm planning to add basic consistency checking, which will look for > (1) attributes that are repeated (which doesn't appear to be meaningful for > any of them) and (2) attributes that don't make sense in a given context, > like the ones listed above. Sounds good to me. Robin -- Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com From michalpurzynski1 at gmail.com Wed Oct 31 11:14:13 2018 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Wed, 31 Oct 2018 11:14:13 -0700 Subject: [Bro-Dev] consistency checking for attributes In-Reply-To: <201810291849.w9TInrJK012017@fruitcake.ICSI.Berkeley.EDU> References: <201810291849.w9TInrJK012017@fruitcake.ICSI.Berkeley.EDU> Message-ID: Sounds good and I like it. To be honest I'm not even sure if the behavior is defined right now, i.e. if the later value will overwrite the first one. Do you want to error out when two &defaults are found or overwrite the first with the second one? On Mon, Oct 29, 2018 at 11:56 AM Vern Paxson wrote: > Attributes currently receive essentially no consistency checking. > For example, executing this script: > > global a: count > &default = 10 > &default = 9 > &optional > &log > &add_func = function(d: double, t: time): count { return > 3; }; > print a; > > simply results in: > > error in /Users/vern/tmp/attr-type-check.bro, line 7: value used but not > set (a) > > I'm planning to add basic consistency checking, which will look for > (1) attributes that are repeated (which doesn't appear to be meaningful for > any of them) and (2) attributes that don't make sense in a given context, > like the ones listed above. > > I'm thinking of implementing this as an internal table of meta-attributes, > i.e., each attribute type, like ATTR_OPTIONAL, will have its own attributes > like whether it requires a record context, only makes sense for aggregates, > etc. Here are the ones that come to mind, based on looking at the > attributes > at https://www.bro.org/sphinx/script-reference/attributes.html with > examples > in parens: > > applies to global variable (&redef) > to global type (&redef) > to event handler (&priority) > to record field (&log) > to indexed type (&default) > to global indexed type (&add_func) > to type with expirable entries (&expire_func) > to a file (&rotate_interval) > > Any feedback? > > Vern > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20181031/469a2595/attachment.html From vern at corelight.com Wed Oct 31 11:17:26 2018 From: vern at corelight.com (Vern Paxson) Date: Wed, 31 Oct 2018 11:17:26 -0700 Subject: [Bro-Dev] consistency checking for attributes In-Reply-To: (Wed, 31 Oct 2018 11:14:13 PDT). Message-ID: <201810311817.w9VIHQdH014976@fruitcake.ICSI.Berkeley.EDU> > To be honest I'm not even sure if the behavior > is defined right now, i.e. if the later value will overwrite the first one. I don't think it's defined. Looking at the code, the later one will indeed overwrite the first one. > Do you want to error out when two &defaults are found or overwrite the > first with the second one? My thinking is that listing them together should be an error. I don't see why it would be legitimate for this to occur, and it seems like a mistake that could be made if there's a lengthy list of attributes. Vern From michalpurzynski1 at gmail.com Wed Oct 31 11:19:42 2018 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Wed, 31 Oct 2018 11:19:42 -0700 Subject: [Bro-Dev] consistency checking for attributes In-Reply-To: <201810311817.w9VIHQdH014976@fruitcake.ICSI.Berkeley.EDU> References: <201810311817.w9VIHQdH014976@fruitcake.ICSI.Berkeley.EDU> Message-ID: Yeah, error out and let there be no surprises. On Wed, Oct 31, 2018 at 11:17 AM Vern Paxson wrote: > > To be honest I'm not even sure if the behavior > > is defined right now, i.e. if the later value will overwrite the first > one. > > I don't think it's defined. Looking at the code, the later one will > indeed overwrite the first one. > > > Do you want to error out when two &defaults are found or overwrite the > > first with the second one? > > My thinking is that listing them together should be an error. I don't > see why it would be legitimate for this to occur, and it seems like a > mistake that could be made if there's a lengthy list of attributes. > > Vern > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20181031/091ea27e/attachment.html