From jazoff at illinois.edu Mon Aug 1 11:13:23 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 1 Aug 2016 18:13:23 +0000 Subject: [Bro-Dev] Making scan.bro great again. In-Reply-To: References: Message-ID: <00E7DC07-5F68-4FE4-ACDC-D7FE9D8496B7@illinois.edu> I know.. I send too many emails :-) I let the rewritten script run over the weekend, cpu and memory was stable. I added one additional table to store known scanners so it can completely purge a scanners state, this cut down on the total amount of data stored by 1/2, as measured by while true;do echo $(date) $(broctl print Scan::recent_scan_attempts |sort -u| wc -l);sleep 30m;done | tee -a keys.log currently this is around 155,000 for us. That is 155,000 addr, port records. approx 16 bytes for each ip and 2 bytes for each port, gives ~3 MB of raw data, times whatever the overhead in bro is. I also fixed the duration and port formatting issues, so now it properly shows things like ... scanned at least 100 unique hosts on port 3306/tcp in 13m18s ... scanned at least 70 unique hosts on ports 23/tcp, 2222/tcp, 22/tcp in 102m27s ... scanned at least 100 unique hosts on port 23/tcp in 0m1s ... scanned at least 99 hosts on 80 ports in 0m52s I also even further simplified the connection filtering that feeds into scan detecting, I think it now finally has the bare minimum needed to detect scans and does not flag connections with capture loss as scans. The last graph I included was a bit of a mess, this one is a little more clear [cid:fd51c96b-a455-49b6-93ef-13f31ce0325c at mx.uillinois.edu] It shows 3 experiments, from left to right: * Stock scan.bro * Unified scan.bro that still uses sumstats * Unified scan.bro rewritten to not use sumstats and to work like the 1.5 version did (attached) Also interesting is a graph of the network traffic during the same timeframe: [cid:d97f3aab-1be2-466d-965c-d8a1e57a127e at mx.uillinois.edu] The positive line is manager -> worker traffic, and the negative line is worker -> manager traffic. The negative line includes log writes, so the floor there won't be zero. -- - Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160801/e4495fef/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: bro-test-manager-cpu.png Type: image/png Size: 55644 bytes Desc: bro-test-manager-cpu.png Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160801/e4495fef/attachment-0002.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: bro-test-network.png Type: image/png Size: 53849 bytes Desc: bro-test-network.png Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160801/e4495fef/attachment-0003.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: scan.bro Type: application/octet-stream Size: 6695 bytes Desc: scan.bro Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160801/e4495fef/attachment-0001.obj From jazoff at illinois.edu Tue Aug 2 16:36:37 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 2 Aug 2016 23:36:37 +0000 Subject: [Bro-Dev] Making scan.bro great again. In-Reply-To: <66854414-BC88-4850-81E0-D64C0560D710@illinois.edu> References: <00E7DC07-5F68-4FE4-ACDC-D7FE9D8496B7@illinois.edu> <66854414-BC88-4850-81E0-D64C0560D710@illinois.edu> Message-ID: <5D55E13A-B923-4A6D-BA6B-D7A3AFE5A400@illinois.edu> > On Aug 2, 2016, at 6:03 PM, Slagell, Adam J wrote: > > Wow. Big difference Indeed :-) I realized one of the bigger issues in the sumstats based code is not really the detection of scans, but what happens AFTER the detection. After detection it keeps accumulating data, or possibly only slightly better, keeps trying to accumulate data. Connection events are used to generate sumstats observations sumstats observations feed into the sumstats framework which may cross a threshold which generates notices that are fed into the notice framework which are often suppressed for at least 1hr by default. However, sumstats has no idea that the only reason it is collecting observations is to raise a notice that could be currently suppressed for an entire day. The observations don't stop once attacker has already triggered a notice. The whole machine keeps running even though nothing visible will ever come out of it. I managed to fix this in the sumstats based unified scan.bro, but it is only a partial fix. The code that does this is in this version: https://github.com/bro/bro/blob/b12153653dddef0c5b2f9fa190cd3ab62c0652a4/scripts/policy/misc/scan.bro#L50 Basically this part: global known_scanners: table[addr] of interval &create_expire=10secs &expire_func=adjust_known_scanner_expiration; # There's no way to set a key to expire at a specific time, so we # First set the keys value to the duration we want, and then # use expire_func to return the desired time. event Notice::begin_suppression(n: Notice::Info) { if (n$note == Port_Scan || n$note == Address_Scan || n$note == Random_Scan) known_scanners[n$src] = n$suppress_for; } function adjust_known_scanner_expiration(s: table[addr] of interval, idx: addr): interval { local duration = s[idx]; s[idx] = 0secs; return duration; } and then later, the checks are aborted early with: if ( scanner in known_scanners ) return; This works, but the reason it is incomplete is that if the notice was triggered by an intermediate update, sumstats still contains data for this attacker. This data will be sent over to the manager at the end of the epoch, even though by that time it isn't needed anymore since there was only one threshold and it was already crossed. The fix for that would be some more changes inside of sumstats.. if (intermediate update crossed a threshold && the number of thresholds is 1) instruct all worker nodes to purge any data associated with this key my non-sumstats based scan.bro and AAshish's scan-NG both handle 'known scanner' suppression directly with a cluster event. Because there's no middleman (sumstats) it ends up being a bit simpler. This Notice::begin_suppression trick itself is nice though, since it lets you configure suppression intervals in one place, instead of potentially having to configure policy specific suppression and keep them in sync. -- - Justin Azoff From slagell at illinois.edu Thu Aug 4 19:47:36 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Fri, 5 Aug 2016 02:47:36 +0000 Subject: [Bro-Dev] package manager progress In-Reply-To: <20160727175710.GD4480@icir.org> References:

<20160727160519.GB49032@ninja.local> <039A75A9-720C-42FC-BC2C-93B603A1D9DE@icir.org> <20160727175710.GD4480@icir.org> Message-ID: <74E9CB0B-1A7C-4F2A-8984-11C43091B327@illinois.edu> Hhhm, is it naming conventions that people have a problem with or the implication of policing? These can be separated. I don?t see a downside to promoting conventions. It also seems that some of the reason (e.g., that we have metadata is based on an assumption that we will have good metadata). But I recall a lot of resistance to requiring basic metadata. I believe this merits a little more discussion and would like to nudge behavior if possible, though not compel it. We could do this by simply providing a skeleton taxonomy into which people could always just through things in ?misc? or some equivalent. > On Jul 27, 2016, at 12:57 PM, Robin Sommer wrote: > > Make it four. :) I'm with Seth, too, better not to enforce any naming > scheme because the boundaries are unclear. Also, note that a single > binary Bro plugin can provide multiple quite different things (say, a > reader and an analyzer and a packet source all at the same time, if > one so desires :). > > Also agree with Johanna: the username is part of the package name if I > follow correctly, so there's disambiguation there. > > I have some more feeback on the package manager and Jon's questions > starting this thread, will send soon. > > Robin > > On Wed, Jul 27, 2016 at 09:15 -0700, you wrote: > >> And to add a me three to this - I am also with him on this one. On top >> of things - I might misremember this, but didn't we plan package names >> to include the github user name at one point of time? So a package name >> would be user/redis, for example, and there also could be user2/redis? >> >> Johanna >> >> On 27 Jul 2016, at 9:05, Matthias Vallentin wrote: >> >>>> I actually don't like this that much because some of these can cross >>>> boundaries and do all sorts of different things in a single plugin. >>>> It makes more sense to me to leave the naming open. >>> >>> I'm with Seth on this one. The reason why I think we should keep the >>> naming open is that it's the job of the meta data tags to take care of >>> the grouping. If someone writes a redis package, then they should >>> apply >>> the redis package. Encoding this meta data into the package name is >>> quite limited, however. >>> >>> Matthias >>> _______________________________________________ >>> bro-dev mailing list >>> bro-dev at bro.org >>> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev >> _______________________________________________ >> bro-dev mailing list >> bro-dev at bro.org >> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev >> > > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From robin at icir.org Fri Aug 5 10:52:09 2016 From: robin at icir.org (Robin Sommer) Date: Fri, 5 Aug 2016 10:52:09 -0700 Subject: [Bro-Dev] package manager progress In-Reply-To: <74E9CB0B-1A7C-4F2A-8984-11C43091B327@illinois.edu> References:

<20160727160519.GB49032@ninja.local> <039A75A9-720C-42FC-BC2C-93B603A1D9DE@icir.org> <20160727175710.GD4480@icir.org> <74E9CB0B-1A7C-4F2A-8984-11C43091B327@illinois.edu> Message-ID: <20160805175209.GD51491@icir.org> On Fri, Aug 05, 2016 at 02:47 +0000, you wrote: > Hhhm, is it naming conventions that people have a problem with or the implication of policing? The problem is that the suggested naming convention wouldn't work: it's not clear how somebody would name their plugin if it provided more than one specific piece of functionality. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From slagell at illinois.edu Fri Aug 5 11:29:48 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Fri, 5 Aug 2016 18:29:48 +0000 Subject: [Bro-Dev] package manager progress In-Reply-To: <20160805175209.GD51491@icir.org> References:

<20160727160519.GB49032@ninja.local> <039A75A9-720C-42FC-BC2C-93B603A1D9DE@icir.org> <20160727175710.GD4480@icir.org> <74E9CB0B-1A7C-4F2A-8984-11C43091B327@illinois.edu> <20160805175209.GD51491@icir.org> <6AB08DBA-A6FE-4A46-83BF-4E67944E0783@illinois.edu>, Message-ID: <642A5B08-BAA6-44FA-BF75-F643D891D54D@illinois.edu> > On Aug 5, 2016, at 6:37 PM, Siwek, Jon wrote: > > >> On Aug 5, 2016, at 1:29 PM, Slagell, Adam J wrote: >> >> If we are going to rely on metadata, which I agree can be better as you can tag a package with multiple categories, we should probably have some basic requirements for this type of metadata. > > At a minimum, it?s useful to provide a list of ?suggested keywords? that people can choose from when tagging their packages so there?s at least a common terminology to search within. > > Do you mean to require something more than that? E.g. ?packages submitted to the ?bro? package source MUST NOT use keywords outside the pre-approved list? ? Or ?packages submitted to the ?bro? package source MUST contain at least one keyword tag? ? > I was thinking of requiring at least one keyword. > - Jon From robin at icir.org Mon Aug 8 08:20:10 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 8 Aug 2016 08:20:10 -0700 Subject: [Bro-Dev] [Bro] broker make error In-Reply-To: <073B9213-B640-404F-9B1F-7C8BA94D5208@icir.org> References: <073B9213-B640-404F-9B1F-7C8BA94D5208@icir.org> Message-ID: <20160808152010.GA78417@icir.org> (Moving to bro-dev). On Sun, Aug 07, 2016 at 09:59 -0700, Johanna Amann wrote: > Yup, that is exactly it. There currently is a rewrite of Broker > underway, which will use the newer library versions, but it is not > quite done yet. I'm wondering if we should add a version check to Broker that clearly says what's is needed if it cannot find the right CAF version? People have run into this a few times now both ways (Broker wants newer/older CAF version). That seems something worth adding for 2.5 still. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From johanna at icir.org Mon Aug 8 09:11:23 2016 From: johanna at icir.org (Johanna Amann) Date: Mon, 08 Aug 2016 09:11:23 -0700 Subject: [Bro-Dev] [Bro] broker make error In-Reply-To: <20160808152010.GA78417@icir.org> References: <073B9213-B640-404F-9B1F-7C8BA94D5208@icir.org> <20160808152010.GA78417@icir.org> Message-ID: <88C8EED7-E4EB-41BB-95B2-3FF318E938DE@icir.org> On 8 Aug 2016, at 8:20, Robin Sommer wrote: > (Moving to bro-dev). > > On Sun, Aug 07, 2016 at 09:59 -0700, Johanna Amann wrote: > >> Yup, that is exactly it. There currently is a rewrite of Broker >> underway, which will use the newer library versions, but it is not >> quite done yet. > > I'm wondering if we should add a version check to Broker that clearly > says what's is needed if it cannot find the right CAF version? People > have run into this a few times now both ways (Broker wants newer/older > CAF version). That seems something worth adding for 2.5 still. Agreed; that also should not be very hard, I can try to take a look at it. Johanna From robin at icir.org Mon Aug 8 12:04:59 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 8 Aug 2016 12:04:59 -0700 Subject: [Bro-Dev] Updating NEWS for 2.5 Message-ID: <20160808190459.GK52034@icir.org> Could folks take a look at NEWS and see what's missing? Couple of things I think we should add at least: - Document the cluster framework's new logger node, with an example how to use it. - Document the recent intel framework updates. - Add BroControl news/changes. Any takers for these? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From dnthayer at illinois.edu Mon Aug 8 12:38:08 2016 From: dnthayer at illinois.edu (Daniel Thayer) Date: Mon, 8 Aug 2016 14:38:08 -0500 Subject: [Bro-Dev] Updating NEWS for 2.5 In-Reply-To: <20160808190459.GK52034@icir.org> References: <20160808190459.GK52034@icir.org> Message-ID: I will create a list of BroControl changes. On 8/8/16 2:04 PM, Robin Sommer wrote: > Could folks take a look at NEWS and see what's missing? > > Couple of things I think we should add at least: > > - Document the cluster framework's new logger node, with an > example how to use it. > > - Document the recent intel framework updates. > > - Add BroControl news/changes. > > Any takers for these? > > Robin > From seth at icir.org Mon Aug 8 13:58:32 2016 From: seth at icir.org (Seth Hall) Date: Mon, 8 Aug 2016 16:58:32 -0400 Subject: [Bro-Dev] SMB in master Message-ID: <179BE5D9-A702-477F-A4EF-F484E2B0E536@icir.org> After having a bit of trouble last week with merging SMB into master and having Robin revert my commit, I finally got it merged in again today. If anyone has any trouble with master now, please say something. There is always the possibility that we messed up something. BTW... this does mean that SMB is now in master. :) If you want to run it, make sure you load policy/protocols/smb because it isn't loaded by default. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From robin at icir.org Mon Aug 8 16:11:02 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 8 Aug 2016 16:11:02 -0700 Subject: [Bro-Dev] SMB in master In-Reply-To: <179BE5D9-A702-477F-A4EF-F484E2B0E536@icir.org> References: <179BE5D9-A702-477F-A4EF-F484E2B0E536@icir.org> Message-ID: <20160808231102.GA50271@icir.org> > BTW... this does mean that SMB is now in master. :) > Cool. :) How much of a performance impact would you expect? I'm seeing these runtime increases (with a debug version): [ 50%] tests.short ... failed (+12.0%) [ 75%] tests.medium ... failed (+1.7%) [ 28%] tests.m57-short ... failed (+2.7%) [ 85%] tests.m57-long ... failed (+1.6%) That may be reasonable, but I wanted check what you'd expect. (The tests.short number doesn't bother me too much, that one can go up and down pretty easily I have noticed). I'm also seeing one test failure, see below. Robin [ 0%] coverage.bare-mode-errors ... failed % 'btest-diff errors' failed unexpectedly (exit code 1) % cat .diag == File =============================== error in /home/robin/bro/master/testing/btest/../../scripts/base/protocols/ntlm/main.bro, line 49: "redef" used but not previously defined (DPD::ignore_violations) internal warning in /home/robin/bro/master/testing/btest/../../scripts/base/protocols/ntlm/main.bro, line 49: Can't document redef of DPD::ignore_violations, identifier lookup failed error in /home/robin/bro/master/testing/btest/../../scripts/base/protocols/ntlm/./main.bro, line 49: "redef" used but not previously defined (DPD::ignore_violations) internal warning in /home/robin/bro/master/testing/btest/../../scripts/base/protocols/ntlm/./main.bro, line 49: Can't document redef of DPD::ignore_violations, identifier lookup failed == Diff =============================== --- /tmp/test-diff.63086.errors.baseline.tmp 2016-08-08 23:07:43.089330553 +0000 +++ /tmp/test-diff.63086.errors.tmp 2016-08-08 23:07:43.103330433 +0000 @@ -0,0 +1,4 @@ +error in /home/robin/bro/master/testing/btest/../../scripts/base/protocols/ntlm/main.bro, line 49: "redef" used but not previously defined (DPD::ignore_violations) +internal warning in /home/robin/bro/master/testing/btest/../../scripts/base/protocols/ntlm/main.bro, line 49: Can't document redef of DPD::ignore_violations, identifier lookup failed +error in /home/robin/bro/master/testing/btest/../../scripts/base/protocols/ntlm/./main.bro, line 49: "redef" used but not previously defined (DPD::ignore_violations) +internal warning in /home/robin/bro/master/testing/btest/../../scripts/base/protocols/ntlm/./main.bro, line 49: Can't document redef of DPD::ignore_violations, identifier lookup failed ======================================= Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jan.grashoefer at gmail.com Tue Aug 9 07:23:01 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Tue, 9 Aug 2016 16:23:01 +0200 Subject: [Bro-Dev] Updating NEWS for 2.5 In-Reply-To: <20160808190459.GK52034@icir.org> References: <20160808190459.GK52034@icir.org> Message-ID: > Could folks take a look at NEWS and see what's missing? > ... > - Document the recent intel framework updates. For the NEWS (all changes, feel free to cut down): +++ - Bro's Intelligence Framework was refactored and new functionality has been added: - The intel framework now supports the new indicator type Intel::SUBNET. As subnets are matched against seen addresses, the field 'matched' was introduced to indicate which indicator type(s) caused the hit. - The new function remove() allows to delete intelligence items. - The intel framework now supports expiration of intelligence items. Expiration can be configured by using Intel::item_expiration and can be handled by using the item_expired() hook. The new script do_expire.bro removes expired items. - The new hook extend_match() allows extending the framework. The new policy script whitelist.bro uses the hook to implement whitelisting. - Intel notices are now suppressible and mails for intel notices now list the identified services as well as the intel source. +++ Additionally I talked to Seth about documentation of the new features. He suggested to write a blog post. I've already started but as I am quite busy at the moment it will take some more time. Best regards, Jan From seth at icir.org Tue Aug 9 07:29:57 2016 From: seth at icir.org (Seth Hall) Date: Tue, 9 Aug 2016 10:29:57 -0400 Subject: [Bro-Dev] Kerberos changes Message-ID: <500108E1-DBF9-4087-8284-B20B15B7D134@icir.org> I just pushed some changes to the GSSAPI and Kerberos analyzers in the master branch. Bro is now handling kerberos carried over GSSAPI although I don't think there is any script support for how it's used yet so it's likely you won't see any change in your krb.log. If anyone sees any trouble, let me know. Thanks, .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From robin at icir.org Tue Aug 9 08:21:20 2016 From: robin at icir.org (Robin Sommer) Date: Tue, 9 Aug 2016 08:21:20 -0700 Subject: [Bro-Dev] Updating NEWS for 2.5 In-Reply-To: References: <20160808190459.GK52034@icir.org> Message-ID: <20160809152120.GQ52034@icir.org> Thanks, will add. Robin On Tue, Aug 09, 2016 at 16:23 +0200, you wrote: > > Could folks take a look at NEWS and see what's missing? > > ... > > - Document the recent intel framework updates. > > For the NEWS (all changes, feel free to cut down): > > +++ > - Bro's Intelligence Framework was refactored and new functionality > has been added: > > - The intel framework now supports the new indicator type > Intel::SUBNET. As subnets are matched against seen addresses, > the field 'matched' was introduced to indicate which indicator > type(s) caused the hit. > > - The new function remove() allows to delete intelligence items. > > - The intel framework now supports expiration of intelligence items. > Expiration can be configured by using Intel::item_expiration and > can be handled by using the item_expired() hook. The new script > do_expire.bro removes expired items. > > - The new hook extend_match() allows extending the framework. The new > policy script whitelist.bro uses the hook to implement whitelisting. > > - Intel notices are now suppressible and mails for intel notices now > list the identified services as well as the intel source. > +++ > > Additionally I talked to Seth about documentation of the new features. > He suggested to write a blog post. I've already started but as I am > quite busy at the moment it will take some more time. > > Best regards, > Jan > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From michalpurzynski1 at gmail.com Tue Aug 9 08:38:26 2016 From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=) Date: Tue, 9 Aug 2016 17:38:26 +0200 Subject: [Bro-Dev] Updating NEWS for 2.5 In-Reply-To: <20160809152120.GQ52034@icir.org> References: <20160808190459.GK52034@icir.org> <20160809152120.GQ52034@icir.org> Message-ID: <60728D5A-75E2-43B9-A76B-FAFB49CA7EC2@gmail.com> Since the changes are impressive, a blog post would be a great idea indeed. Bro is first (?) on the market (again!) to come up with something like this :) Such post should describe all new Intel framework feature and give examples how to use them. > On 09 Aug 2016, at 17:21, Robin Sommer wrote: > > Thanks, will add. > > Robin > > On Tue, Aug 09, 2016 at 16:23 +0200, you wrote: > >>> Could folks take a look at NEWS and see what's missing? >>> ... >>> - Document the recent intel framework updates. >> >> For the NEWS (all changes, feel free to cut down): >> >> +++ >> - Bro's Intelligence Framework was refactored and new functionality >> has been added: >> >> - The intel framework now supports the new indicator type >> Intel::SUBNET. As subnets are matched against seen addresses, >> the field 'matched' was introduced to indicate which indicator >> type(s) caused the hit. >> >> - The new function remove() allows to delete intelligence items. >> >> - The intel framework now supports expiration of intelligence items. >> Expiration can be configured by using Intel::item_expiration and >> can be handled by using the item_expired() hook. The new script >> do_expire.bro removes expired items. >> >> - The new hook extend_match() allows extending the framework. The new >> policy script whitelist.bro uses the hook to implement whitelisting. >> >> - Intel notices are now suppressible and mails for intel notices now >> list the identified services as well as the intel source. >> +++ >> >> Additionally I talked to Seth about documentation of the new features. >> He suggested to write a blog post. I've already started but as I am >> quite busy at the moment it will take some more time. >> >> Best regards, >> Jan >> _______________________________________________ >> bro-dev mailing list >> bro-dev at bro.org >> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From seth at icir.org Tue Aug 9 12:43:57 2016 From: seth at icir.org (Seth Hall) Date: Tue, 9 Aug 2016 15:43:57 -0400 Subject: [Bro-Dev] SMB in master In-Reply-To: <20160808231102.GA50271@icir.org> References: <179BE5D9-A702-477F-A4EF-F484E2B0E536@icir.org> <20160808231102.GA50271@icir.org> Message-ID: <090B5360-0BC0-4165-91C9-4B951841421C@icir.org> > On Aug 8, 2016, at 7:11 PM, Robin Sommer wrote: > > How much of a performance impact would you expect? I'm seeing these > runtime increases (with a debug version): I don't really know what to expect from this in terms of performance yet. I've heard anecdotally that things are working ok, but I don't have any clue what would feel right or wrong. > [ 0%] coverage.bare-mode-errors ... failed Whoops. Fixed. Thanks, .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From dnthayer at illinois.edu Tue Aug 9 14:53:07 2016 From: dnthayer at illinois.edu (Daniel Thayer) Date: Tue, 9 Aug 2016 16:53:07 -0500 Subject: [Bro-Dev] Updating NEWS for 2.5 In-Reply-To: <20160808190459.GK52034@icir.org> References: <20160808190459.GK52034@icir.org> Message-ID: I've attached the list of significant BroControl changes. On 8/8/16 2:04 PM, Robin Sommer wrote: > Could folks take a look at NEWS and see what's missing? > > Couple of things I think we should add at least: > > - Document the cluster framework's new logger node, with an > example how to use it. > > - Document the recent intel framework updates. > > - Add BroControl news/changes. > > Any takers for these? > > Robin > -------------- next part -------------- New Functionality ----------------- - There is a new node type "logger" that can be specified in node.cfg (that file has a commented-out example). The purpose of this new node type is to receive logs from all nodes in a cluster in order to reduce the load on the manager node. However, if there is no "logger" node, then the manager node will handle logging as usual. - The post-terminate script will send email if it fails to archive any log files. These mails can be turned off by changing the value of the new BroControl option MailArchiveLogFail. - Added the ability for "broctl deploy" to reload the BroControl configuration (both broctl.cfg and node.cfg). This happens automatically if broctl detects any changes to those config files since the last time the config was loaded. Note that this feature is relevant only when using the BroControl shell interactively. - The BroControl plugin API has a new function "broctl_config". This gives plugin authors the ability to add their own script code to the autogenerated broctl-config.bro script. - There is a new BroControl plugin for custom load balancing. This plugin can be used by setting "lb_method=custom" for your worker nodes in node.cfg. To support packet source plugins, it allows configuration of a prefix and suffix for the interface name. Changed Functionality --------------------- - The networks.cfg file now contains private IP space 172.16.0.0/12 by default. - Upon startup, if broctl can't get IP addresses from the "ifconfig" command for any reason, then broctl will now also try to use the "ip" command. - BroControl will now automatically search the Bro plugin directory for BroControl plugins (in addition to all the other places where BroControl searches). This enables automatic loading of BroControl plugins that are provided by a Bro plugin. - Changed the default value of the StatusCmdShowAll option so that the "broctl status" command runs faster. This also means that there is no longer a "Peers" column in the status output by default. - Users can now specify a more granular log expiration interval. The BroControl option LogExpireInterval can be set to an arbitrary time interval instead of just an integer number of days. The time interval is specified as an integer followed by a time unit: "day", "hr", or "min". For backward compatibility, an integer value without a time unit is still interpreted as a number of days. - Changed the text of crash report emails. Now crash reports tell the user to forward the mail to the Bro team only when a backtrace is included in the crash report. If there is no backtrace, then the crash report includes instructions on how to get backtraces included in future crash reports. From robin at icir.org Wed Aug 10 08:48:04 2016 From: robin at icir.org (Robin Sommer) Date: Wed, 10 Aug 2016 08:48:04 -0700 Subject: [Bro-Dev] Updating NEWS for 2.5 In-Reply-To: References: <20160808190459.GK52034@icir.org> Message-ID: <20160810154804.GA30193@icir.org> Thanks, integrated! Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jan.grashoefer at gmail.com Thu Aug 11 14:53:31 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Thu, 11 Aug 2016 23:53:31 +0200 Subject: [Bro-Dev] Warnings building current master Message-ID: <4c502bfb-892a-09e7-fb6f-9b51de4747bb@gmail.com> Building current master I see some warnings that seem new to me (the only one I remember is the one in Hasher.cc): [ 1%] Building CXX object aux/binpac/src/CMakeFiles/binpac.dir/pac_scan.cc.o /home/jgras/devel/bro/build/aux/binpac/src/pac_scan.cc: In function ?yy_buffer_state* yy_scan_bytes(const char*, yy_size_t)?: /home/jgras/devel/bro/build/aux/binpac/src/pac_scan.cc:2290:19: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] for ( i = 0; i < _yybytes_len; ++i ) ^ -- [ 5%] Building CXX object src/CMakeFiles/bifcl.dir/bif_lex.cc.o /home/jgras/devel/bro/build/src/bif_lex.cc: In function ?yy_buffer_state* yy_scan_bytes(const char*, yy_size_t)?: /home/jgras/devel/bro/build/src/bif_lex.cc:1867:19: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] for ( i = 0; i < _yybytes_len; ++i ) ^ -- [ 77%] Building CXX object src/input/CMakeFiles/bro_input.dir/Manager.cc.o /home/jgras/devel/bro/src/input/Manager.cc: In member function ?void input::Manager::ErrorHandler(const input::Manager::Stream*, input::Manager::ErrorType, bool, const char*, __va_list_tag*)?: /home/jgras/devel/bro/src/input/Manager.cc:2537:67: warning: ?ev? may be used uninitialized in this function [-Wmaybe-uninitialized] SendEvent(i->error_event, 3, i->description->Ref(), message, ev); ^ -- [ 80%] Building CXX object src/probabilistic/CMakeFiles/bro_probabilistic.dir/Hasher.cc.o /home/jgras/devel/bro/src/probabilistic/Hasher.cc: In member function ?probabilistic::Hasher::digest probabilistic::UHF::hash(const void*, size_t) const?: /home/jgras/devel/bro/src/probabilistic/Hasher.cc:130:51: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing] return *reinterpret_cast(d); ^ -- [ 82%] Building CXX object src/CMakeFiles/bro.dir/rule-scan.cc.o /home/jgras/devel/bro/build/src/rule-scan.cc: In function ?yy_buffer_state* rules__scan_bytes(const char*, yy_size_t)?: /home/jgras/devel/bro/build/src/rule-scan.cc:2303:19: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] for ( i = 0; i < _yybytes_len; ++i ) ^ [ 82%] Building CXX object src/CMakeFiles/bro.dir/re-scan.cc.o /home/jgras/devel/bro/build/src/re-scan.cc: In function ?yy_buffer_state* re__scan_bytes(const char*, yy_size_t)?: /home/jgras/devel/bro/build/src/re-scan.cc:2083:19: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] for ( i = 0; i < _yybytes_len; ++i ) ^ [ 82%] Building CXX object src/CMakeFiles/bro.dir/scan.cc.o /home/jgras/devel/bro/build/src/scan.cc: In function ?yy_buffer_state* bro_scan_bytes(const char*, yy_size_t)?: /home/jgras/devel/bro/build/src/scan.cc:3286:19: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] for ( i = 0; i < _yybytes_len; ++i ) From seth at icir.org Thu Aug 11 20:29:45 2016 From: seth at icir.org (Seth Hall) Date: Thu, 11 Aug 2016 23:29:45 -0400 Subject: [Bro-Dev] Warnings building current master In-Reply-To: <4c502bfb-892a-09e7-fb6f-9b51de4747bb@gmail.com> References: <4c502bfb-892a-09e7-fb6f-9b51de4747bb@gmail.com> Message-ID: What compiler is giving these errors? I haven't been seeing these. .Seth > On Aug 11, 2016, at 5:53 PM, Jan Grash?fer wrote: > > Building current master I see some warnings that seem new to me (the > only one I remember is the one in Hasher.cc): > > [ 1%] Building CXX object > aux/binpac/src/CMakeFiles/binpac.dir/pac_scan.cc.o > /home/jgras/devel/bro/build/aux/binpac/src/pac_scan.cc: In function > ?yy_buffer_state* yy_scan_bytes(const char*, yy_size_t)?: > /home/jgras/devel/bro/build/aux/binpac/src/pac_scan.cc:2290:19: warning: > comparison between signed and unsigned integer expressions [-Wsign-compare] > for ( i = 0; i < _yybytes_len; ++i ) > ^ > -- > [ 5%] Building CXX object src/CMakeFiles/bifcl.dir/bif_lex.cc.o > /home/jgras/devel/bro/build/src/bif_lex.cc: In function > ?yy_buffer_state* yy_scan_bytes(const char*, yy_size_t)?: > /home/jgras/devel/bro/build/src/bif_lex.cc:1867:19: warning: comparison > between signed and unsigned integer expressions [-Wsign-compare] > for ( i = 0; i < _yybytes_len; ++i ) > ^ > -- > [ 77%] Building CXX object src/input/CMakeFiles/bro_input.dir/Manager.cc.o > /home/jgras/devel/bro/src/input/Manager.cc: In member function ?void > input::Manager::ErrorHandler(const input::Manager::Stream*, > input::Manager::ErrorType, bool, const char*, __va_list_tag*)?: > /home/jgras/devel/bro/src/input/Manager.cc:2537:67: warning: ?ev? may be > used uninitialized in this function [-Wmaybe-uninitialized] > SendEvent(i->error_event, 3, i->description->Ref(), message, ev); > ^ > -- > [ 80%] Building CXX object > src/probabilistic/CMakeFiles/bro_probabilistic.dir/Hasher.cc.o > /home/jgras/devel/bro/src/probabilistic/Hasher.cc: In member function > ?probabilistic::Hasher::digest probabilistic::UHF::hash(const void*, > size_t) const?: > /home/jgras/devel/bro/src/probabilistic/Hasher.cc:130:51: warning: > dereferencing type-punned pointer will break strict-aliasing rules > [-Wstrict-aliasing] > return *reinterpret_cast(d); > ^ > -- > [ 82%] Building CXX object src/CMakeFiles/bro.dir/rule-scan.cc.o > /home/jgras/devel/bro/build/src/rule-scan.cc: In function > ?yy_buffer_state* rules__scan_bytes(const char*, yy_size_t)?: > /home/jgras/devel/bro/build/src/rule-scan.cc:2303:19: warning: > comparison between signed and unsigned integer expressions [-Wsign-compare] > for ( i = 0; i < _yybytes_len; ++i ) > ^ > [ 82%] Building CXX object src/CMakeFiles/bro.dir/re-scan.cc.o > /home/jgras/devel/bro/build/src/re-scan.cc: In function > ?yy_buffer_state* re__scan_bytes(const char*, yy_size_t)?: > /home/jgras/devel/bro/build/src/re-scan.cc:2083:19: warning: comparison > between signed and unsigned integer expressions [-Wsign-compare] > for ( i = 0; i < _yybytes_len; ++i ) > ^ > [ 82%] Building CXX object src/CMakeFiles/bro.dir/scan.cc.o > /home/jgras/devel/bro/build/src/scan.cc: In function ?yy_buffer_state* > bro_scan_bytes(const char*, yy_size_t)?: > /home/jgras/devel/bro/build/src/scan.cc:3286:19: warning: comparison > between signed and unsigned integer expressions [-Wsign-compare] > for ( i = 0; i < _yybytes_len; ++i ) > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > > -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jan.grashoefer at gmail.com Fri Aug 12 01:37:31 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Fri, 12 Aug 2016 10:37:31 +0200 Subject: [Bro-Dev] Warnings building current master In-Reply-To: References: <4c502bfb-892a-09e7-fb6f-9b51de4747bb@gmail.com> Message-ID: <101b58d4-3285-8a8b-488b-b8244388bf62@gmail.com> > What compiler is giving these errors? I haven't been seeing these. I was using gcc 4.8.5 on CentOS 7. From dave.a.florek at gmail.com Fri Aug 12 11:03:48 2016 From: dave.a.florek at gmail.com (Dave Florek) Date: Fri, 12 Aug 2016 14:03:48 -0400 Subject: [Bro-Dev] Bro IDS request Message-ID: Hello, Because I lose so much processing power when manually converting Bro output logs from Epoch to EST using bro-cut, can I have a feature that automatically outputs the Bro logs to EST automatically instead of Epoch while Bro is timestamping the logs as it sees the traffic? I'm not sure if using the Epoch format makes Bro much faster while it's processing, but I would like a more integrated solution aside from using the bro-cut utility. Thank you for your time, -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160812/3b5183b6/attachment.html From asharma at lbl.gov Fri Aug 12 11:14:26 2016 From: asharma at lbl.gov (Aashish Sharma) Date: Fri, 12 Aug 2016 11:14:26 -0700 Subject: [Bro-Dev] Bro IDS request In-Reply-To: References: Message-ID: <20160812181425.GH53181@mac-4.local> May be try: ftp://ftp.ee.lbl.gov/cf-1.2.5.tar.gz eg: cf conn.log | less On Fri, Aug 12, 2016 at 02:03:48PM -0400, Dave Florek wrote: > Hello, > > Because I lose so much processing power when manually converting Bro output > logs from Epoch to EST using bro-cut, can I have a feature that > automatically outputs the Bro logs to EST automatically instead of Epoch > while Bro is timestamping the logs as it sees the traffic? > > I'm not sure if using the Epoch format makes Bro much faster while it's > processing, but I would like a more integrated solution aside from using > the bro-cut utility. > > Thank you for your time, > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From jazoff at illinois.edu Fri Aug 12 11:40:26 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 12 Aug 2016 18:40:26 +0000 Subject: [Bro-Dev] Bro IDS request In-Reply-To: <20160812181425.GH53181@mac-4.local> References: <20160812181425.GH53181@mac-4.local> Message-ID: > On Aug 12, 2016, at 2:14 PM, Aashish Sharma wrote: > > May be try: ftp://ftp.ee.lbl.gov/cf-1.2.5.tar.gz > > eg: cf conn.log | less > Yeah.. cf should be a few times faster than bro-cut for busy log files, especially if the only thing you are doing is converting the timestamp. It has an optimization that bro-cut doesn't have yet for avoiding converting timestamps if the current one is the same second as the previous one. If you are using both tools though and only extracting a few fields, piping bro-cut to cf should be faster than piping cf to bro-cut. I'm not sure why converting the timestamp is so important though. What are you doing with the data once you convert the timestamps? -- - Justin Azoff From dave.a.florek at gmail.com Fri Aug 12 12:10:17 2016 From: dave.a.florek at gmail.com (Dave Florek) Date: Fri, 12 Aug 2016 15:10:17 -0400 Subject: [Bro-Dev] Bro IDS request In-Reply-To: References: <20160812181425.GH53181@mac-4.local> Message-ID: Manual searching to establish a timeline of events that I can understand when my intel.log chirps. On Fri, Aug 12, 2016 at 2:40 PM, Azoff, Justin S wrote: > > > On Aug 12, 2016, at 2:14 PM, Aashish Sharma wrote: > > > > May be try: ftp://ftp.ee.lbl.gov/cf-1.2.5.tar.gz > > > > eg: cf conn.log | less > > > > Yeah.. cf should be a few times faster than bro-cut for busy log files, > especially if the only thing you are doing is converting the timestamp. > It has an optimization that bro-cut doesn't have yet for avoiding > converting timestamps if the current one is the same second as the previous > one. > > If you are using both tools though and only extracting a few fields, > piping bro-cut to cf should be faster than piping cf to bro-cut. > > I'm not sure why converting the timestamp is so important though. What > are you doing with the data once you convert the timestamps? > > > -- > - Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160812/70738ccb/attachment.html From jsiwek at illinois.edu Fri Aug 12 16:54:47 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Fri, 12 Aug 2016 23:54:47 +0000 Subject: [Bro-Dev] bro-pkg v0.3 Message-ID: Feedback from the previous bro-dev thread is now addressed and I?d consider this version of the package manager ready for people to start trying out. Documentation: http://bro-package-manager.readthedocs.io/ Notes/questions: * Docs are now hosted at Read the Docs instead of GitHub I found it more straightforward to setup/use/maintain docs on RtD. It automatically builds docs upon committing to master or making a release tag. See [1] for directions on how the docs could be made reachable at a bro.org domain, but still hosted at RtD. * A package submission process is found in the README of https://github.com/bro/packages Which is just my idea of what the process will look like, so please comment if it differs from anyone?s expectations. * I?ve put it on PyPi [2] and it?s installable via `pip` What are thoughts on making this the standard installation process for bro-pkg instead of bundling it into the Bro distribution? Benefits: 1) Decoupling from Bro?s release cycle. Seems like the kind of tool where it?s important to have quick/lightweight releases/patches that are easy for users to upgrade to. Would bundling in with Bro make for a more cumbersome upgrade path? 2) Installing via pip will automatically install dependencies (gitpython, semantic-version). Installing those automatically via Bro?s CMake, while probably possible, seems unusual/hacky. A downside would be that the first-time configuration is currently easier if done via Bro?s CMake since it can fill in all the paths of bro-pkg.config automatically. But I also see potential for addressing that in a way that doesn?t involve putting bro-pkg into Bro?s CMake. E.g. if there were `bro ?config site_dir` and `bro ?config plugin_dir` commands that could be executed from the command-line, bro-pkg could just use those to autogenerate an appropriate config. - Jon [1] http://read-the-docs.readthedocs.io/en/latest/alternate_domains.html [2] https://pypi.python.org/pypi/bro-pkg From robin at icir.org Mon Aug 15 08:20:21 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 15 Aug 2016 08:20:21 -0700 Subject: [Bro-Dev] bro-pkg v0.3 In-Reply-To: References: Message-ID: <20160815152021.GE95947@icir.org> On Fri, Aug 12, 2016 at 23:54 +0000, you wrote: > http://bro-package-manager.readthedocs.io/ Nice! I like the Quickstart guide, that works now. The How-Tos are great, too. One suggestion for "Create a Package": I believe the text doesn't really say anywhere yet explicitly that a plugin can contain these three different things. Maybe add paragraph summarizing what "package" actually means at the beginning? > * A package submission process is found in the README of https://github.com/bro/packages That sounds all good to me. One nit: In (2): the "this" before repository is ambiguous because (1) talks about a different one. Looks like this the location where we can also require a couple more pieces of meta data. I still vote for keeping this minimal, but I suggest we require at least an email contact and a license statement before accepting a pull request. I'm torn on requiring a version number as well: I would like too, but we can't enforce that people update it as they make changes; and it would just be useless if people don't. So probably better to skip. Once this is finalized, do you plan to add a pointer back here from the BPM docs? > What are thoughts on making this the standard installation process for > bro-pkg instead of bundling it into the Bro distribution? Benefits: Yeah, I agree, that makes sense. pip is the standard way to install Python stuff and really straight-forward. > E.g. if there were `bro ?config site_dir` and `bro ?config plugin_dir` Good thought, though why not keep it out of Bro: we can add a script "bro-config" to Bro that cmake populates with the right data. That would actually be helpful for other stuff as well. That's something we could still add for 2.5, too. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at illinois.edu Mon Aug 15 11:36:37 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Mon, 15 Aug 2016 18:36:37 +0000 Subject: [Bro-Dev] bro-pkg v0.3 In-Reply-To: <20160815152021.GE95947@icir.org> References: <20160815152021.GE95947@icir.org> Message-ID: > On Aug 15, 2016, at 10:20 AM, Robin Sommer wrote: > > One suggestion for "Create a Package": I believe the text > doesn't really say anywhere yet explicitly that a plugin can contain > these three different things. Maybe add paragraph summarizing what > "package" actually means at the beginning? Done. >> * A package submission process is found in the README of https://github.com/bro/packages > > That sounds all good to me. One nit: In (2): the "this" before > repository is ambiguous because (1) talks about a different one. Changed. > Looks like this the location where we can also require a couple more > pieces of meta data. I still vote for keeping this minimal, but I > suggest we require at least an email contact and a license statement > before accepting a pull request. Yeah, makes sense to put those bits in the index data, if you decide to require them. > I'm torn on requiring a version > number as well: I would like too, but we can't enforce that people > update it as they make changes; and it would just be useless if people > don't. So probably better to skip. I agree. We depend on git being the authority for versioning anyway, so requiring people to update a metadata file in addition to creating a git tag still borders on useless even as an enforced rule. > Once this is finalized, do you plan to add a pointer back here from > the BPM docs? There was one in the main README/landing-page. I added another link in the ?Registering to a Package Source? walkthrough. Let me know if there was another place you were expecting to see it, but didn?t. >> E.g. if there were `bro ?config site_dir` and `bro ?config plugin_dir` > > Good thought, though why not keep it out of Bro: we can add a script > "bro-config" to Bro that cmake populates with the right data. That > would actually be helpful for other stuff as well. That's something we > could still add for 2.5, too. Good idea. I?ll get a branch for that today/tomorrow. - Jon From robin at icir.org Tue Aug 16 09:57:07 2016 From: robin at icir.org (Robin Sommer) Date: Tue, 16 Aug 2016 09:57:07 -0700 Subject: [Bro-Dev] bro-pkg v0.3 In-Reply-To: References: <20160815152021.GE95947@icir.org> Message-ID: <20160816165707.GH35439@icir.org> On Mon, Aug 15, 2016 at 18:36 +0000, you wrote: > There was one in the main README/landing-page. Ah, missed that. I think we need to state that more explicitly and give some more context about this repo being the central hub for the community to share scripts. Although not sure if the BPM docs are actually the right place for that then, should that go into the community section on bro.org? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at illinois.edu Tue Aug 16 11:51:09 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Tue, 16 Aug 2016 18:51:09 +0000 Subject: [Bro-Dev] bro-pkg v0.3 In-Reply-To: <20160816165707.GH35439@icir.org> References: <20160815152021.GE95947@icir.org> <20160816165707.GH35439@icir.org> Message-ID: > On Aug 16, 2016, at 11:57 AM, Robin Sommer wrote: > > I think we need to state that more explicitly and > give some more context about this repo being the central hub for the > community to share scripts. Although not sure if the BPM docs are > actually the right place for that then, should that go into the > community section on bro.org? The ?contributing? section [1] feels like one place to put it. Even looks like there?s an old reference to the project there that needs to be updated. I can do that now, but should I add a temporary disclaimer that it?s ?not yet open for submissions? or would it be alright at this point if a person came across it and tried to submit a package? - Jon [1] https://www.bro.org/community/index.html#contributing-tobro From robin at icir.org Tue Aug 16 12:35:57 2016 From: robin at icir.org (Robin Sommer) Date: Tue, 16 Aug 2016 12:35:57 -0700 Subject: [Bro-Dev] bro-pkg v0.3 In-Reply-To: References: <20160815152021.GE95947@icir.org> <20160816165707.GH35439@icir.org> Message-ID: <20160816193557.GB9607@icir.org> On Tue, Aug 16, 2016 at 18:51 +0000, you wrote: > be updated. I can do that now, but should I add a temporary > disclaimer that it?s ?not yet open for submissions? or would it be > alright at this point if a person came across it and tried to submit a > package? Let's wait until we're ready to announce it. A blog posting would be good once we believe we have the process down. Before we do that, maybe we can get a couple people here on this list to give publishing some of their scripts a try? If that goes well, we start over once more and announce. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From dave.a.florek at gmail.com Tue Aug 16 14:12:58 2016 From: dave.a.florek at gmail.com (Dave Florek) Date: Tue, 16 Aug 2016 17:12:58 -0400 Subject: [Bro-Dev] Bro IDS request In-Reply-To: References: <20160812181425.GH53181@mac-4.local> Message-ID: I'll check it out. Glad to know there are alternatives to bro-cut. Thanks for your time guys, On Fri, Aug 12, 2016 at 3:10 PM, Dave Florek wrote: > Manual searching to establish a timeline of events that I can understand > when my intel.log chirps. > > On Fri, Aug 12, 2016 at 2:40 PM, Azoff, Justin S > wrote: > >> >> > On Aug 12, 2016, at 2:14 PM, Aashish Sharma wrote: >> > >> > May be try: ftp://ftp.ee.lbl.gov/cf-1.2.5.tar.gz >> > >> > eg: cf conn.log | less >> > >> >> Yeah.. cf should be a few times faster than bro-cut for busy log files, >> especially if the only thing you are doing is converting the timestamp. >> It has an optimization that bro-cut doesn't have yet for avoiding >> converting timestamps if the current one is the same second as the previous >> one. >> >> If you are using both tools though and only extracting a few fields, >> piping bro-cut to cf should be faster than piping cf to bro-cut. >> >> I'm not sure why converting the timestamp is so important though. What >> are you doing with the data once you convert the timestamps? >> >> >> -- >> - Justin Azoff >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160816/8e697de3/attachment.html From jan.grashoefer at gmail.com Wed Aug 24 16:22:57 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Thu, 25 Aug 2016 01:22:57 +0200 Subject: [Bro-Dev] Updating NEWS for 2.5 In-Reply-To: <60728D5A-75E2-43B9-A76B-FAFB49CA7EC2@gmail.com> References: <20160808190459.GK52034@icir.org> <20160809152120.GQ52034@icir.org> <60728D5A-75E2-43B9-A76B-FAFB49CA7EC2@gmail.com> Message-ID: <70d9858d-4f6e-6a70-6663-732275e145e5@gmail.com> I have finished my draft for a blog post explaining the new intel framework features including some examples: https://gist.github.com/J-Gras/3ff4d5308a69e91fb61c65c12ecb818c Feedback welcome! Jan From michalpurzynski1 at gmail.com Tue Aug 30 17:10:23 2016 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Wed, 31 Aug 2016 02:10:23 +0200 Subject: [Bro-Dev] Updating NEWS for 2.5 In-Reply-To: <70d9858d-4f6e-6a70-6663-732275e145e5@gmail.com> References: <20160808190459.GK52034@icir.org> <20160809152120.GQ52034@icir.org> <60728D5A-75E2-43B9-A76B-FAFB49CA7EC2@gmail.com> <70d9858d-4f6e-6a70-6663-732275e145e5@gmail.com> Message-ID: Feedback shared privately. Come on, let's read it and polish it a bit. It's already very good! On Thu, Aug 25, 2016 at 1:22 AM, Jan Grash?fer wrote: > I have finished my draft for a blog post explaining the new intel > framework features including some examples: > > https://gist.github.com/J-Gras/3ff4d5308a69e91fb61c65c12ecb818c > > Feedback welcome! > > Jan > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160831/fbeb7e11/attachment.html From seth at icir.org Wed Aug 31 06:23:29 2016 From: seth at icir.org (Seth Hall) Date: Wed, 31 Aug 2016 09:23:29 -0400 Subject: [Bro-Dev] Updating NEWS for 2.5 In-Reply-To: References: <20160808190459.GK52034@icir.org> <20160809152120.GQ52034@icir.org> <60728D5A-75E2-43B9-A76B-FAFB49CA7EC2@gmail.com> <70d9858d-4f6e-6a70-6663-732275e145e5@gmail.com> Message-ID: > On Aug 30, 2016, at 8:10 PM, Micha? Purzy?ski wrote: > > Feedback shared privately. > > Come on, let's read it and polish it a bit. It's already very good! Let's wait until the 2.5 release. Seems like it might be a bit premature to post it now since the 2.5 release hasn't happened yet. :) Jan, would you mind if we posted this directly to the Bro blog? (obviously with all credit given to you!) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From michalpurzynski1 at gmail.com Wed Aug 31 08:08:01 2016 From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=) Date: Wed, 31 Aug 2016 17:08:01 +0200 Subject: [Bro-Dev] Updating NEWS for 2.5 In-Reply-To: References: <20160808190459.GK52034@icir.org> <20160809152120.GQ52034@icir.org> <60728D5A-75E2-43B9-A76B-FAFB49CA7EC2@gmail.com> <70d9858d-4f6e-6a70-6663-732275e145e5@gmail.com>

Message-ID: <73947624-41EA-44DF-836F-5ED43CEF7A70@gmail.com> That's why I said polish not publish ;-) Send feedback, have it included or rejected, all can be done before 2.5 > On 31 Aug 2016, at 15:23, Seth Hall wrote: > > >> On Aug 30, 2016, at 8:10 PM, Micha? Purzy?ski wrote: >> >> Feedback shared privately. >> >> Come on, let's read it and polish it a bit. It's already very good! > > Let's wait until the 2.5 release. Seems like it might be a bit premature to post it now since the 2.5 release hasn't happened yet. :) > > Jan, would you mind if we posted this directly to the Bro blog? (obviously with all credit given to you!) > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > From jan.grashoefer at gmail.com Wed Aug 31 08:42:21 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Wed, 31 Aug 2016 17:42:21 +0200 Subject: [Bro-Dev] Updating NEWS for 2.5 In-Reply-To: <73947624-41EA-44DF-836F-5ED43CEF7A70@gmail.com> References: <20160808190459.GK52034@icir.org> <20160809152120.GQ52034@icir.org> <60728D5A-75E2-43B9-A76B-FAFB49CA7EC2@gmail.com> <70d9858d-4f6e-6a70-6663-732275e145e5@gmail.com>

<73947624-41EA-44DF-836F-5ED43CEF7A70@gmail.com> Message-ID: <5fd75102-9d4f-fbb8-e255-d0e520317b37@gmail.com> > Feedback shared privately. Thanks a lot for your detailed feedback Michal! I will definitely add the changes you suggested. > That's why I said polish not publish ;-) Send feedback, have it included or rejected, all can be done before 2.5 As already mentioned, I am kind of busy at the moment. So making the changes might take some time again. However, in general I agree. Depending on when the release will be, we might even get some feedback that can be considered for the final release. > Jan, would you mind if we posted this directly to the Bro blog? I thought that was the original idea. I would be glad if the post is shared on the Bro blog once we have a final version. Jan From seth at icir.org Wed Aug 31 08:59:57 2016 From: seth at icir.org (Seth Hall) Date: Wed, 31 Aug 2016 11:59:57 -0400 Subject: [Bro-Dev] Updating NEWS for 2.5 In-Reply-To: <5fd75102-9d4f-fbb8-e255-d0e520317b37@gmail.com> References: <20160808190459.GK52034@icir.org> <20160809152120.GQ52034@icir.org> <60728D5A-75E2-43B9-A76B-FAFB49CA7EC2@gmail.com> <70d9858d-4f6e-6a70-6663-732275e145e5@gmail.com>

<73947624-41EA-44DF-836F-5ED43CEF7A70@gmail.com> <5fd75102-9d4f-fbb8-e255-d0e520317b37@gmail.com> Message-ID: > On Aug 31, 2016, at 11:42 AM, Jan Grash?fer wrote: > >> Jan, would you mind if we posted this directly to the Bro blog? > > I thought that was the original idea. I would be glad if the post is > shared on the Bro blog once we have a final version. Hah, ignore me. I was reading Michal's email poorly. Everyone seems to be on the same page. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/