From noreply at bro.org Sat Mar 1 00:00:16 2014 From: noreply at bro.org (Merge Tracker) Date: Sat, 1 Mar 2014 00:00:16 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403010800.s2180G8R029588@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ -------------- ---------- ------------- ---------- ------------------------------- BIT-1144 [1] Bro Brian Little Bernhard Amann 2014-02-25 - Low topk_get_top returned data type [1] BIT-1144 https://bro-tracker.atlassian.net/browse/BIT-1144 From noreply at bro.org Sun Mar 2 00:00:16 2014 From: noreply at bro.org (Merge Tracker) Date: Sun, 2 Mar 2014 00:00:16 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403020800.s2280GHK001682@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ -------------- ---------- ------------- ---------- ------------------------------- BIT-1144 [1] Bro Brian Little Bernhard Amann 2014-02-25 - Low topk_get_top returned data type [1] BIT-1144 https://bro-tracker.atlassian.net/browse/BIT-1144 From noreply at bro.org Mon Mar 3 00:00:14 2014 From: noreply at bro.org (Merge Tracker) Date: Mon, 3 Mar 2014 00:00:14 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403030800.s2380Emo006486@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ -------------- ---------- ------------- ---------- ------------------------------- BIT-1144 [1] Bro Brian Little Bernhard Amann 2014-02-25 - Low topk_get_top returned data type [1] BIT-1144 https://bro-tracker.atlassian.net/browse/BIT-1144 From jira at bro-tracker.atlassian.net Mon Mar 3 07:14:20 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 3 Mar 2014 09:14:20 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1132) topic/seth/http-connect In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15700#comment-15700 ] Robin Sommer commented on BIT-1132: ----------------------------------- New version in topic/seth/http-connect. I'll merge it later. Seth, if you could give it quick try that would be good. > topic/seth/http-connect > ----------------------- > > Key: BIT-1132 > URL: https://bro-tracker.atlassian.net/browse/BIT-1132 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > Assignee: Robin Sommer > Fix For: 2.3 > > > New support for the HTTP analyzer to correct support and decapsulate HTTP CONNECT proxying. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 3 07:16:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 3 Mar 2014 09:16:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1132) topic/seth/http-connect In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1132: ------------------------------ Status: Merge Request (was: Open) > topic/seth/http-connect > ----------------------- > > Key: BIT-1132 > URL: https://bro-tracker.atlassian.net/browse/BIT-1132 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > Assignee: Robin Sommer > Fix For: 2.3 > > > New support for the HTTP analyzer to correct support and decapsulate HTTP CONNECT proxying. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 3 07:30:18 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Mon, 3 Mar 2014 09:30:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1132) topic/seth/http-connect In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15701#comment-15701 ] Seth Hall commented on BIT-1132: -------------------------------- Looks good to me. Thanks! > topic/seth/http-connect > ----------------------- > > Key: BIT-1132 > URL: https://bro-tracker.atlassian.net/browse/BIT-1132 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > Assignee: Robin Sommer > Fix For: 2.3 > > > New support for the HTTP analyzer to correct support and decapsulate HTTP CONNECT proxying. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 3 17:09:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 3 Mar 2014 19:09:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1132) topic/seth/http-connect In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1132: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > topic/seth/http-connect > ----------------------- > > Key: BIT-1132 > URL: https://bro-tracker.atlassian.net/browse/BIT-1132 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > Assignee: Robin Sommer > Fix For: 2.3 > > > New support for the HTTP analyzer to correct support and decapsulate HTTP CONNECT proxying. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 3 17:09:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 3 Mar 2014 19:09:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1144) topk_get_top returned data type In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1144: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > topk_get_top returned data type > ------------------------------- > > Key: BIT-1144 > URL: https://bro-tracker.atlassian.net/browse/BIT-1144 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Environment: Ubuntu, both 2.2 release and 2.2.117 from git. > Reporter: Brian Little > Assignee: Bernhard Amann > Priority: Low > Labels: topk > Attachments: topk.bro > > > I'm trying to get the top few results in a topk data type, and then loop over them to print. > Running for over the results brings up an error: > target to iterate over must be a table, set, vector, or string > The docs say it is of type vector, and running |topk_result| brings back the correct count. > example demonstration script attached -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 4 05:47:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Tue, 4 Mar 2014 07:47:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-953) SSL Analyzer: return the root CA used to validate a cert In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15702#comment-15702 ] Bernhard Amann commented on BIT-953: ------------------------------------ Ok, the split of x509 handling into the file-analysis framework is basically ready in the topic/bernhard/file-analysis-x509 branch. I have a few small loose ends to tie up (mostly: update the test baselines), which I already started to do. But - before investing too much work in this - could someone take a look if the new Interface looks ok? The big changes basically are: * the certificate handling completely moved into a file analysis framework plugin * there is a new x509.log, which contains information about any certificate encountered on the wire. This contains more information than the old ssl.log, including a few certificate extensions like the subject alternative name, used ec curve names, etc. * the ssl.log has slightly less information about the certificates than before. It includes the certificate file IDs as well as the subject and the issuer of the host (and client) certificates. Validity, etc. was stripped (and not used by any base scripts) * the certificate der values are not passed around scriptland anymore. Instead, a opaque of x509 is included into the x509_certificate event, which can be used to access the string form of a certificate using the x509_get_certificate_string function * the certificate validation function was changed quite a lot. It now returns the full validated certificate chain and takes arguments in a more convenient manner (sorted list of opaque of x509). This also should reduce overhead by quite a bit. >From a users point of view, the biggest changes probably are the new logfiles. Do these look ok? diff-link for the lazy: https://github.com/bro/bro/compare/topic;bernhard;file-analysis-x509 > SSL Analyzer: return the root CA used to validate a cert > -------------------------------------------------------- > > Key: BIT-953 > URL: https://bro-tracker.atlassian.net/browse/BIT-953 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: liamrandall > Assignee: Bernhard Amann > Priority: Low > Labels: Analyzer,, CA, Root,, SSL > Fix For: 2.4 > > > Since Bro will validate certs can we add a variable that says who the root CA was; would be useful for CA pinning, white listing or black listing. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From kmcmahon at mitre.org Tue Mar 4 14:39:37 2014 From: kmcmahon at mitre.org (McMahon, Kevin J) Date: Tue, 4 Mar 2014 22:39:37 +0000 Subject: [Bro-Dev] Bug in Connection::FlipRoles Message-ID: <00D3CD29F7C24A44B4D23450BB8E55B30AB7CE4C@IMCMBX03.MITRE.ORG> To whom it may concern, Sorry if I'm not following the proper procedure; this is my first post on this list (please be gentle and point me in the right direction). There is a bug in Conn.cc in the Connection::FlipRoles routine: 725,726c725,726 < resp_addr = orig_addr; < orig_addr = tmp_addr; --- > orig_addr = resp_addr; > resp_addr = tmp_addr; This causes the process of correcting the assignment of client/server roles when the SYN and SYN/ACK packets are out of order. Making the above change (be careful with that as I typed it in by hand) allowed my system to process quite a few more connections than I was able to otherwise. However, this change does not address the issue when it occurs in an a connection that is to be captured via expect_connection (e.g., ftp_data). I did some digging into this aspect of out-of-order handshakes but it is a bit more involved than the main line connection processing. If anyone has advice on that aspect of this issue I'm all ears. From jira at bro-tracker.atlassian.net Tue Mar 4 15:11:18 2014 From: jira at bro-tracker.atlassian.net (aashish (JIRA)) Date: Tue, 4 Mar 2014 17:11:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1145) Individual set_seperator for different feeds In-Reply-To: References: Message-ID: aashish created BIT-1145: ---------------------------- Summary: Individual set_seperator for different feeds Key: BIT-1145 URL: https://bro-tracker.atlassian.net/browse/BIT-1145 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: 2.3 Reporter: aashish Can we assign an individual set_separator per feed ? Why ?: Various data feeds from different sources have their own fields separators. We need to post process these feeds in order to digest the data into bro using input-framework, this creates a need to have two tiered storage for each of the data feeds (original data + re-formatted data for input framework). At present the workaround is to basically format all data feeds to use intel-framework and this works very well. There is still useful needs to have data feeds outside intel-framework for example - digesting list of subnets+building allocations in the network or digesting auth data... and so on. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 4 16:03:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Tue, 4 Mar 2014 18:03:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1145) Individual set_seperator for different feeds In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703#comment-15703 ] Bernhard Amann commented on BIT-1145: ------------------------------------- I think we talked about this a while back and the conclusion was that it would be nice to have the ability to override the global choices via configuration options in the config map. We probably should do this for the logging framework as well. If I am not mistaken this should be a rather easy change. > Individual set_seperator for different feeds > -------------------------------------------- > > Key: BIT-1145 > URL: https://bro-tracker.atlassian.net/browse/BIT-1145 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: aashish > Labels: feeds, framework, input, logging > Fix For: 2.4 > > > Can we assign an individual set_separator per feed ? > Why ?: > Various data feeds from different sources have their own fields separators. > We need to post process these feeds in order to digest the data into bro using input-framework, this creates a need to have two tiered storage for each of the data feeds (original data + re-formatted data for input framework). > At present the workaround is to basically format all data feeds to use intel-framework and this works very well. There is still useful needs to have data feeds outside intel-framework for example - digesting list of subnets+building allocations in the network or digesting auth data... and so on. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 4 16:03:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Tue, 4 Mar 2014 18:03:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1145) Individual set_seperator for different feeds In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1145: -------------------------------- Affects Version/s: (was: 2.3) git/master > Individual set_seperator for different feeds > -------------------------------------------- > > Key: BIT-1145 > URL: https://bro-tracker.atlassian.net/browse/BIT-1145 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: aashish > Labels: feeds, framework, input, logging > Fix For: 2.4 > > > Can we assign an individual set_separator per feed ? > Why ?: > Various data feeds from different sources have their own fields separators. > We need to post process these feeds in order to digest the data into bro using input-framework, this creates a need to have two tiered storage for each of the data feeds (original data + re-formatted data for input framework). > At present the workaround is to basically format all data feeds to use intel-framework and this works very well. There is still useful needs to have data feeds outside intel-framework for example - digesting list of subnets+building allocations in the network or digesting auth data... and so on. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 4 16:03:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Tue, 4 Mar 2014 18:03:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1145) Individual set_seperator for different feeds In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1145: -------------------------------- Labels: feeds framework input logging (was: feeds framework input) > Individual set_seperator for different feeds > -------------------------------------------- > > Key: BIT-1145 > URL: https://bro-tracker.atlassian.net/browse/BIT-1145 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: aashish > Labels: feeds, framework, input, logging > Fix For: 2.4 > > > Can we assign an individual set_separator per feed ? > Why ?: > Various data feeds from different sources have their own fields separators. > We need to post process these feeds in order to digest the data into bro using input-framework, this creates a need to have two tiered storage for each of the data feeds (original data + re-formatted data for input framework). > At present the workaround is to basically format all data feeds to use intel-framework and this works very well. There is still useful needs to have data feeds outside intel-framework for example - digesting list of subnets+building allocations in the network or digesting auth data... and so on. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 4 16:03:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Tue, 4 Mar 2014 18:03:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1145) Individual set_seperator for different feeds In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1145: -------------------------------- Fix Version/s: 2.4 > Individual set_seperator for different feeds > -------------------------------------------- > > Key: BIT-1145 > URL: https://bro-tracker.atlassian.net/browse/BIT-1145 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: aashish > Labels: feeds, framework, input, logging > Fix For: 2.4 > > > Can we assign an individual set_separator per feed ? > Why ?: > Various data feeds from different sources have their own fields separators. > We need to post process these feeds in order to digest the data into bro using input-framework, this creates a need to have two tiered storage for each of the data feeds (original data + re-formatted data for input framework). > At present the workaround is to basically format all data feeds to use intel-framework and this works very well. There is still useful needs to have data feeds outside intel-framework for example - digesting list of subnets+building allocations in the network or digesting auth data... and so on. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 4 16:14:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Tue, 4 Mar 2014 18:14:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-556) Extended CA certificate information In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15704#comment-15704 ] Bernhard Amann commented on BIT-556: ------------------------------------ Hm. I am not entirely sure if the additional data in there is especially interesting. The OCSP URLs don't really help a lot (they are included in the certificates themselves). The EV oids might be interesting; but actually using them in validation is not trivial... (old ticket, I know, but still) > Extended CA certificate information > ----------------------------------- > > Key: BIT-556 > URL: https://bro-tracker.atlassian.net/browse/BIT-556 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: bro-aux > Reporter: Seth Hall > > At some point I'd like to include more information in the auto-generated script that currently only has Mozilla's CA certs. > At the very least I'd like to include extended validation OIDs for the various approved EV certificate vendors and OCSP URLs. > The extra data can be found in the XML file located here: > https://www.mozilla.org/projects/security/certs/included/ -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 4 16:24:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Tue, 4 Mar 2014 18:24:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1146) Merge topic/bernhard/ssl-failure In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1146: -------------------------------- Status: Merge Request (was: Open) > Merge topic/bernhard/ssl-failure > -------------------------------- > > Key: BIT-1146 > URL: https://bro-tracker.atlassian.net/browse/BIT-1146 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Fix For: 2.3 > > > topic/bernhard/ssl-failure in the bro and bro-testing repositories slightly changes the way SSL/TLS connections are handled. > So far, if the ssl_established event did not fire for a connection, it was never logged. This means that connections that fail somewhere in the handshake will never appear in the ssl.log file. I think so far there also was no possibility for a handshake error to actually be logged (even though the fields already were present in the logfile). > The patch also adds an additional field to ssl.log which shows if TLS connection was successfully established. > I think this should be ready for merging. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 4 16:24:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Tue, 4 Mar 2014 18:24:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1146) Merge topic/bernhard/ssl-failure In-Reply-To: References: Message-ID: Bernhard Amann created BIT-1146: ----------------------------------- Summary: Merge topic/bernhard/ssl-failure Key: BIT-1146 URL: https://bro-tracker.atlassian.net/browse/BIT-1146 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.3 topic/bernhard/ssl-failure in the bro and bro-testing repositories slightly changes the way SSL/TLS connections are handled. So far, if the ssl_established event did not fire for a connection, it was never logged. This means that connections that fail somewhere in the handshake will never appear in the ssl.log file. I think so far there also was no possibility for a handshake error to actually be logged (even though the fields already were present in the logfile). The patch also adds an additional field to ssl.log which shows if TLS connection was successfully established. I think this should be ready for merging. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Wed Mar 5 00:00:12 2014 From: noreply at bro.org (Merge Tracker) Date: Wed, 5 Mar 2014 00:00:12 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403050800.s2580CUD019797@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- -------------------------------- BIT-1146 [1] Bro Bernhard Amann - 2014-03-04 2.3 Normal Merge topic/bernhard/ssl-failure Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- --------- ---------- -------------------------- 4fd1098 [2] bro Jon Siwek 2014-03-04 Misc. documentation fixes. a2c23b4 [3] btest Jon Siwek 2014-03-04 Fix a link in the README. [1] BIT-1146 https://bro-tracker.atlassian.net/browse/BIT-1146 [2] 4fd1098 https://github.com/bro/bro/commit/4fd1098949183a4c0e0f4aa7aa724220a1929d19 [3] a2c23b4 https://github.com/bro/btest/commit/a2c23b4ffe4b58f0b5c186f156f9111f15ac4a3b From jira at bro-tracker.atlassian.net Wed Mar 5 07:15:18 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Wed, 5 Mar 2014 09:15:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1145) Individual set_seperator for different feeds In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705#comment-15705 ] Seth Hall commented on BIT-1145: -------------------------------- Yep, definitely something we need. It's pretty painful (for input and logging) right now. > Individual set_seperator for different feeds > -------------------------------------------- > > Key: BIT-1145 > URL: https://bro-tracker.atlassian.net/browse/BIT-1145 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: aashish > Labels: feeds, framework, input, logging > Fix For: 2.4 > > > Can we assign an individual set_separator per feed ? > Why ?: > Various data feeds from different sources have their own fields separators. > We need to post process these feeds in order to digest the data into bro using input-framework, this creates a need to have two tiered storage for each of the data feeds (original data + re-formatted data for input framework). > At present the workaround is to basically format all data feeds to use intel-framework and this works very well. There is still useful needs to have data feeds outside intel-framework for example - digesting list of subnets+building allocations in the network or digesting auth data... and so on. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Wed Mar 5 07:37:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Wed, 5 Mar 2014 09:37:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1146) Merge topic/bernhard/ssl-failure In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1146: --------------------------------- Assignee: Seth Hall > Merge topic/bernhard/ssl-failure > -------------------------------- > > Key: BIT-1146 > URL: https://bro-tracker.atlassian.net/browse/BIT-1146 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Seth Hall > Fix For: 2.3 > > > topic/bernhard/ssl-failure in the bro and bro-testing repositories slightly changes the way SSL/TLS connections are handled. > So far, if the ssl_established event did not fire for a connection, it was never logged. This means that connections that fail somewhere in the handshake will never appear in the ssl.log file. I think so far there also was no possibility for a handshake error to actually be logged (even though the fields already were present in the logfile). > The patch also adds an additional field to ssl.log which shows if TLS connection was successfully established. > I think this should be ready for merging. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Thu Mar 6 00:00:12 2014 From: noreply at bro.org (Merge Tracker) Date: Thu, 6 Mar 2014 00:00:12 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403060800.s2680Cdc006648@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- -------------------------------- BIT-1146 [1] Bro Bernhard Amann Seth Hall 2014-03-05 2.3 Normal Merge topic/bernhard/ssl-failure Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- --------- ---------- -------------------------- 4fd1098 [2] bro Jon Siwek 2014-03-04 Misc. documentation fixes. a2c23b4 [3] btest Jon Siwek 2014-03-04 Fix a link in the README. [1] BIT-1146 https://bro-tracker.atlassian.net/browse/BIT-1146 [2] 4fd1098 https://github.com/bro/bro/commit/4fd1098949183a4c0e0f4aa7aa724220a1929d19 [3] a2c23b4 https://github.com/bro/btest/commit/a2c23b4ffe4b58f0b5c186f156f9111f15ac4a3b From jira at bro-tracker.atlassian.net Thu Mar 6 07:26:18 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 6 Mar 2014 09:26:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1147) topic/seth/dns-srv-fix - Fixing some problems with DNS In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1147: --------------------------- Assignee: Robin Sommer > topic/seth/dns-srv-fix - Fixing some problems with DNS > ------------------------------------------------------ > > Key: BIT-1147 > URL: https://bro-tracker.atlassian.net/browse/BIT-1147 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > Assignee: Robin Sommer > > This branch and equivalently named branches are ready for merging in the public and private test suites. > We generate the event for SRV responses in DNS now. > Fixed several annoying issues with NetBios name service requests and responses. Fewer incorrect weirds and more correct dns logs now. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 6 07:26:18 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 6 Mar 2014 09:26:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1147) topic/seth/dns-srv-fix - Fixing some problems with DNS In-Reply-To: References: Message-ID: Seth Hall created BIT-1147: ------------------------------ Summary: topic/seth/dns-srv-fix - Fixing some problems with DNS Key: BIT-1147 URL: https://bro-tracker.atlassian.net/browse/BIT-1147 Project: Bro Issue Tracker Issue Type: Patch Components: Bro Affects Versions: 2.3 Reporter: Seth Hall This branch and equivalently named branches are ready for merging in the public and private test suites. We generate the event for SRV responses in DNS now. Fixed several annoying issues with NetBios name service requests and responses. Fewer incorrect weirds and more correct dns logs now. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 6 07:26:18 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 6 Mar 2014 09:26:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1147) topic/seth/dns-srv-fix - Fixing some problems with DNS In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1147: --------------------------- Status: Merge Request (was: Open) > topic/seth/dns-srv-fix - Fixing some problems with DNS > ------------------------------------------------------ > > Key: BIT-1147 > URL: https://bro-tracker.atlassian.net/browse/BIT-1147 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > > This branch and equivalently named branches are ready for merging in the public and private test suites. > We generate the event for SRV responses in DNS now. > Fixed several annoying issues with NetBios name service requests and responses. Fewer incorrect weirds and more correct dns logs now. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 6 12:39:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 6 Mar 2014 14:39:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1143) Investigate replacing libmagic w/ signatures for file identificaiton In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706#comment-15706 ] Jon Siwek commented on BIT-1143: -------------------------------- I've got topic/jsiwek/file-signatures in bro, 3rdparty, bro-testing, and bro-testing-private repos to a point where they might be ready to merge or at least I'm unsure what more to do w/ it at the moment. Seth do you want this assigned to you to first look over the new file magic signatures (maybe look for important mime types that are somehow missing, or try improving some regexes) ? Also open to others to take a look and make suggestions. New file magic signatures: these are derived from the default libmagic magic database in a semi-automatic/assisted way. I instrumented a version of the {{file}} command, see https://github.com/jsiwek/file/tree/bro-signatures, to get at the internal representation of the magic rules and had it emit Bro signatures for any set of rules associated with a MIME type. The conversion logic is not currently perfect for all combinations of magic rules and the effort to make it perfect didn't seem worth it, so warnings are emitted upon encountering tricky scenarios. Afterward, I did a pass over everything and manually fixed (or just removed, depending on circumstances) the cases where it indicated an automatic conversion might not be correct. Signature maintenance: Going forward, Bro's file signatures can be considered on their own and improved independently of libmagic's rules (i.e. there's no required/extra/continual maintenance task in updating signatures, though the libmagic database would probably still be useful for reference when someone is trying to improve/add signatures). Signature accuracy: Surprisingly, Bro's test suites don't detect file types much differently using the new signatures over libmagic. The variance is actually less than I've seen in switching between versions of libmagic. And the differences in detected MIME types are at least somewhat reasonable -- the most questionable differences are the text/plain detections because libmagic has builtin logic for various text encodings/charsets, but the signature I ended up writing to fill that gap just does ASCII for now. Signature performance: Didn't do very robust profiling/benchmarking, but I found slight improvements in various configurations in terms of instructions and time running against the long m57 pcap. That at least matches expectations of it not theoretically being able to be worse than libmagic's approach, so didn't dig any deeper. And it also should scale better as the number of signatures increases. Signature unit tests: there's no new regression tests in place for the new file magic signatures. That could take a while to make, is it required to have immediately or can wait? And any opinion on the structure of such a test suite? I imagine just having the test suite in the bro repo, but a corpus of file types to test against is probably going to need some other canonical place to live. > Investigate replacing libmagic w/ signatures for file identificaiton > -------------------------------------------------------------------- > > Key: BIT-1143 > URL: https://bro-tracker.atlassian.net/browse/BIT-1143 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Jon Siwek > Fix For: 2.3 > > > I think it makes sense to try to make the switch from libmagic to using Bro's own signature engine for file identification before the next release. Don't want people getting used to magic file format for their own custom file identification rules. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 6 12:51:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 6 Mar 2014 14:51:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1143) Investigate replacing libmagic w/ signatures for file identificaiton In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707#comment-15707 ] Jon Siwek commented on BIT-1143: -------------------------------- I also forgot to mention another improvement of the signature approach over libmagic is that a file is no longer limited to matching a single MIME type. One can now programmatically get at the full list of signature matches along with a value indicating the "strength" of the match. > Investigate replacing libmagic w/ signatures for file identificaiton > -------------------------------------------------------------------- > > Key: BIT-1143 > URL: https://bro-tracker.atlassian.net/browse/BIT-1143 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Jon Siwek > Fix For: 2.3 > > > I think it makes sense to try to make the switch from libmagic to using Bro's own signature engine for file identification before the next release. Don't want people getting used to magic file format for their own custom file identification rules. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 6 12:53:18 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 6 Mar 2014 14:53:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1143) Investigate replacing libmagic w/ signatures for file identificaiton In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708#comment-15708 ] Seth Hall commented on BIT-1143: -------------------------------- I was already working on this branch, so I'll go ahead and claim it for a day or so while I play around. It looks really awesome though. > Investigate replacing libmagic w/ signatures for file identificaiton > -------------------------------------------------------------------- > > Key: BIT-1143 > URL: https://bro-tracker.atlassian.net/browse/BIT-1143 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Seth Hall > Fix For: 2.3 > > > I think it makes sense to try to make the switch from libmagic to using Bro's own signature engine for file identification before the next release. Don't want people getting used to magic file format for their own custom file identification rules. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 6 12:53:18 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 6 Mar 2014 14:53:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1143) Investigate replacing libmagic w/ signatures for file identificaiton In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1143: --------------------------- Assignee: Seth Hall (was: Jon Siwek) > Investigate replacing libmagic w/ signatures for file identificaiton > -------------------------------------------------------------------- > > Key: BIT-1143 > URL: https://bro-tracker.atlassian.net/browse/BIT-1143 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Seth Hall > Fix For: 2.3 > > > I think it makes sense to try to make the switch from libmagic to using Bro's own signature engine for file identification before the next release. Don't want people getting used to magic file format for their own custom file identification rules. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 6 15:04:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 6 Mar 2014 17:04:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1148) Bug in Connection::FlipRoles In-Reply-To: References: Message-ID: Jon Siwek created BIT-1148: ------------------------------ Summary: Bug in Connection::FlipRoles Key: BIT-1148 URL: https://bro-tracker.atlassian.net/browse/BIT-1148 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.2 Reporter: Jon Siwek Fix For: 2.3 This method doesn't correctly swap address values. Also, since scheduled analyzers for a connection are looked up based on the endpoint addresses, it's possible they will miss being attached to connections that end up taking the Connection::FlipRoles code path. An idea to fix would be just to have FlipRoles do a check for scheduled analyzers on the new connection tuple and attach any that turn up. (These were reported by Kevin McMahon on the bro-dev list). -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 6 15:11:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 6 Mar 2014 17:11:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1148) Bug in Connection::FlipRoles In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709#comment-15709 ] Jon Siwek commented on BIT-1148: -------------------------------- I have a patch for both issues in "topic/jsiwek/flip-roles", but it didn't trigger any differences in test suites, so I may try to come up with a test case. > Bug in Connection::FlipRoles > ---------------------------- > > Key: BIT-1148 > URL: https://bro-tracker.atlassian.net/browse/BIT-1148 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Jon Siwek > Fix For: 2.3 > > > This method doesn't correctly swap address values. > Also, since scheduled analyzers for a connection are looked up based on the endpoint addresses, it's possible they will miss being attached to connections that end up taking the Connection::FlipRoles code path. An idea to fix would be just to have FlipRoles do a check for scheduled analyzers on the new connection tuple and attach any that turn up. > (These were reported by Kevin McMahon on the bro-dev list). -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jsiwek at illinois.edu Thu Mar 6 15:22:47 2014 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Thu, 6 Mar 2014 23:22:47 +0000 Subject: [Bro-Dev] Bug in Connection::FlipRoles In-Reply-To: <00D3CD29F7C24A44B4D23450BB8E55B30AB7CE4C@IMCMBX03.MITRE.ORG> References: <00D3CD29F7C24A44B4D23450BB8E55B30AB7CE4C@IMCMBX03.MITRE.ORG> Message-ID: <4554D256-1FCB-4878-BC04-2267ADE7BF5C@illinois.edu> On Mar 4, 2014, at 4:39 PM, McMahon, Kevin J wrote: > Sorry if I'm not following the proper procedure; this is my first post on this list (please be gentle and point me in the right direction). There?s some suggestions on how to contribute at [1]. For straight-forward/complete/small patches it?s probably easiest to fork on github and submit a pull request. For anything else, creating a ticket at tracker.bro.org w/ a proposed patch attached is helpful so things don?t get lost. I created a ticket for this at [2] for now if you want to create an account and ?watch? it. [1] http://bro.org/development/contribute.html [2] https://bro-tracker.atlassian.net/browse/BIT-1148 > There is a bug in Conn.cc in the Connection::FlipRoles routine: > > 725,726c725,726 > < resp_addr = orig_addr; > < orig_addr = tmp_addr; > --- >> orig_addr = resp_addr; >> resp_addr = tmp_addr; That does indeed look wrong, thanks. > However, this change does not address the issue when it occurs in an a connection that is to be captured via expect_connection (e.g., ftp_data). I did some digging into this aspect of out-of-order handshakes but it is a bit more involved than the main line connection processing. If anyone has advice on that aspect of this issue I'm all ears. If I understand right, this is a separate issue from the bad address swapping. If you?re getting at the scheduled/expected analyzers mechanism doesn?t take in to account this Connection::FlipRoles code path, I think you?re right. - Jon From jira at bro-tracker.atlassian.net Thu Mar 6 19:59:18 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 6 Mar 2014 21:59:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1146) Merge topic/bernhard/ssl-failure In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1146: --------------------------- Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Merge topic/bernhard/ssl-failure > -------------------------------- > > Key: BIT-1146 > URL: https://bro-tracker.atlassian.net/browse/BIT-1146 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Seth Hall > Fix For: 2.3 > > > topic/bernhard/ssl-failure in the bro and bro-testing repositories slightly changes the way SSL/TLS connections are handled. > So far, if the ssl_established event did not fire for a connection, it was never logged. This means that connections that fail somewhere in the handshake will never appear in the ssl.log file. I think so far there also was no possibility for a handshake error to actually be logged (even though the fields already were present in the logfile). > The patch also adds an additional field to ssl.log which shows if TLS connection was successfully established. > I think this should be ready for merging. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Fri Mar 7 00:00:14 2014 From: noreply at bro.org (Merge Tracker) Date: Fri, 7 Mar 2014 00:00:14 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403070800.s2780ETH030098@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ---------- ------------ ---------- ------------- ---------- ---------------------------------------------------------- BIT-1147 [1] Bro Seth Hall Robin Sommer 2014-03-06 - Normal topic/seth/dns-srv-fix - Fixing some problems with DNS [2] Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- --------- ---------- -------------------------- 4fd1098 [3] bro Jon Siwek 2014-03-04 Misc. documentation fixes. a2c23b4 [4] btest Jon Siwek 2014-03-04 Fix a link in the README. [1] BIT-1147 https://bro-tracker.atlassian.net/browse/BIT-1147 [2] dns-srv-fix https://github.com/bro/bro/tree/topic/seth/dns-srv-fix [3] 4fd1098 https://github.com/bro/bro/commit/4fd1098949183a4c0e0f4aa7aa724220a1929d19 [4] a2c23b4 https://github.com/bro/btest/commit/a2c23b4ffe4b58f0b5c186f156f9111f15ac4a3b From jira at bro-tracker.atlassian.net Fri Mar 7 11:08:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 7 Mar 2014 13:08:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1128) Add configure options for linking against jemalloc In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1128: --------------------------------- Assignee: Jon Siwek > Add configure options for linking against jemalloc > -------------------------------------------------- > > Key: BIT-1128 > URL: https://bro-tracker.atlassian.net/browse/BIT-1128 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > To gather experiences with using jemalloc, add a configure options --with-jemalloc= that links Bro against it if found. Default should be off. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 7 11:11:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 7 Mar 2014 13:11:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1134) DNS_Mgr::LookupAddr does not respect DNS_FAKE In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1134: --------------------------------- Assignee: Jon Siwek > DNS_Mgr::LookupAddr does not respect DNS_FAKE > --------------------------------------------- > > Key: BIT-1134 > URL: https://bro-tracker.atlassian.net/browse/BIT-1134 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 7 11:14:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 7 Mar 2014 13:14:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1137) Investigate sumstats / scan detector performance In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1137: --------------------------------- Assignee: Seth Hall (was: Gilbert Clark) > Investigate sumstats / scan detector performance > ------------------------------------------------- > > Key: BIT-1137 > URL: https://bro-tracker.atlassian.net/browse/BIT-1137 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > > It's not clear if sumstats is causing more CPU and/or memory load than expected. There's also some indication that it may perform less well in standalone mode than cluster mode. Need to understand and potential improve. > A part of this is also understanding how the new scan detector performs in terms of CPU/memory when compared against the 1.x version. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 7 11:17:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 7 Mar 2014 13:17:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1138) UDP scan detection generates a large number of triggers In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1138: ------------------------------ Resolution: Invalid Status: Closed (was: Open) Not in distribution yet. > UDP scan detection generates a large number of triggers > ------------------------------------------------------- > > Key: BIT-1138 > URL: https://bro-tracker.atlassian.net/browse/BIT-1138 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Fix For: 2.3 > > Attachments: CPU-all-scan-policies.png, Memory-All-Scan-Policies.png > > > These triggers then cause high CPU load. We had a fix already but I'm not sure if it has been confirmed that it solved the problem? -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 7 11:19:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 7 Mar 2014 13:19:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1139) MHR lookups can cause significant CPU overhead in tests In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1139: --------------------------------- Assignee: Jon Siwek (was: Gilbert Clark) > MHR lookups can cause significant CPU overhead in tests > ------------------------------------------------------- > > Key: BIT-1139 > URL: https://bro-tracker.atlassian.net/browse/BIT-1139 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > Live operation seems fine, need to understand what's going on. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 7 11:30:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 7 Mar 2014 13:30:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1143) Investigate replacing libmagic w/ signatures for file identificaiton In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1143: ------------------------------ Reporter: Seth Hall (was: Jon Siwek) > Investigate replacing libmagic w/ signatures for file identificaiton > -------------------------------------------------------------------- > > Key: BIT-1143 > URL: https://bro-tracker.atlassian.net/browse/BIT-1143 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Seth Hall > Assignee: Seth Hall > Fix For: 2.3 > > > I think it makes sense to try to make the switch from libmagic to using Bro's own signature engine for file identification before the next release. Don't want people getting used to magic file format for their own custom file identification rules. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 7 11:30:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 7 Mar 2014 13:30:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1143) Investigate replacing libmagic w/ signatures for file identificaiton In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1143: ------------------------------ Reporter: Jon Siwek (was: Seth Hall) > Investigate replacing libmagic w/ signatures for file identificaiton > -------------------------------------------------------------------- > > Key: BIT-1143 > URL: https://bro-tracker.atlassian.net/browse/BIT-1143 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Seth Hall > Fix For: 2.3 > > > I think it makes sense to try to make the switch from libmagic to using Bro's own signature engine for file identification before the next release. Don't want people getting used to magic file format for their own custom file identification rules. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 7 11:42:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 7 Mar 2014 13:42:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-348) Reassembler integer overflow issues. Data not delivered after 2GB In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-348: -------------------------------- Assignee: Bernhard Amann > Reassembler integer overflow issues. Data not delivered after 2GB > ----------------------------------------------------------------- > > Key: BIT-348 > URL: https://bro-tracker.atlassian.net/browse/BIT-348 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: gregor > Assignee: Bernhard Amann > Priority: High > Labels: inttypes > Fix For: 2.3 > > > {noformat} > #!rst > The TCP Reassembler does not deliver any data to analyzers after the first 2GB due to signed integer overflow (Actually it will deliver again between 4--6GB, etc.) This happens silently, i.e., without content_gap events or Undelivered calls. > This report superseded BIT-315, BIT-137 > The TCP Reassembler (and Reassem) base class use ``int`` to keep track of sequence numbers and ``seq_delta`` to check for differences. If a connection exceeds 2GB, the relative sequence numbers (int) used by the Reassembler become negative. While many parts of the Reassembler still work (because seq_delta still reports the correct difference) some parts do not. In particular ``seq_to_skip`` is broken (and fails silently). There might well be other parts of the Reassembler that fail > silently as well, that I haven't found yet. > See Comments in TCP_Reassembler.cc for more details. > The Reassembler should use int64. However this will require deep changes to the Reassembler and the TCP Analyzer and TCP_Endpoint classes (since we also store sequence numbers there). Also, the analyzer framework will need tweaks as well (e.g., Undelivered uses ``int`` for sequence numbers, also has to go to 64 bit) > As a hotfix that seems to work I disabled the ``seq_to_skip`` features. It wasn't used by any analyzer or policy script (Note, that seq_to_skip is different from skip_deliveries). Hotfix is in > topic/gregor/reassembler-hotfix > {noformat} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 7 11:42:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 7 Mar 2014 13:42:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-348) Reassembler integer overflow issues. Data not delivered after 2GB In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-348: ----------------------------- Fix Version/s: (was: 2.4) 2.3 > Reassembler integer overflow issues. Data not delivered after 2GB > ----------------------------------------------------------------- > > Key: BIT-348 > URL: https://bro-tracker.atlassian.net/browse/BIT-348 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: gregor > Assignee: Bernhard Amann > Priority: High > Labels: inttypes > Fix For: 2.3 > > > {noformat} > #!rst > The TCP Reassembler does not deliver any data to analyzers after the first 2GB due to signed integer overflow (Actually it will deliver again between 4--6GB, etc.) This happens silently, i.e., without content_gap events or Undelivered calls. > This report superseded BIT-315, BIT-137 > The TCP Reassembler (and Reassem) base class use ``int`` to keep track of sequence numbers and ``seq_delta`` to check for differences. If a connection exceeds 2GB, the relative sequence numbers (int) used by the Reassembler become negative. While many parts of the Reassembler still work (because seq_delta still reports the correct difference) some parts do not. In particular ``seq_to_skip`` is broken (and fails silently). There might well be other parts of the Reassembler that fail > silently as well, that I haven't found yet. > See Comments in TCP_Reassembler.cc for more details. > The Reassembler should use int64. However this will require deep changes to the Reassembler and the TCP Analyzer and TCP_Endpoint classes (since we also store sequence numbers there). Also, the analyzer framework will need tweaks as well (e.g., Undelivered uses ``int`` for sequence numbers, also has to go to 64 bit) > As a hotfix that seems to work I disabled the ``seq_to_skip`` features. It wasn't used by any analyzer or policy script (Note, that seq_to_skip is different from skip_deliveries). Hotfix is in > topic/gregor/reassembler-hotfix > {noformat} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 7 11:46:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 7 Mar 2014 13:46:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1149) Check Coverity PIA message In-Reply-To: References: Message-ID: Robin Sommer created BIT-1149: --------------------------------- Summary: Check Coverity PIA message Key: BIT-1149 URL: https://bro-tracker.atlassian.net/browse/BIT-1149 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Reporter: Robin Sommer Assignee: Robin Sommer Fix For: 2.3 -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 7 11:48:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 7 Mar 2014 13:48:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1151) JSON output In-Reply-To: References: Message-ID: Robin Sommer created BIT-1151: --------------------------------- Summary: JSON output Key: BIT-1151 URL: https://bro-tracker.atlassian.net/browse/BIT-1151 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Reporter: Robin Sommer Assignee: Seth Hall Fix For: 2.3 -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 7 11:48:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 7 Mar 2014 13:48:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: Robin Sommer created BIT-1150: --------------------------------- Summary: X509 updates Key: BIT-1150 URL: https://bro-tracker.atlassian.net/browse/BIT-1150 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Reporter: Robin Sommer Assignee: Bernhard Amann Fix For: 2.3 -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 7 11:50:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 7 Mar 2014 13:50:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1152) BroControl version check In-Reply-To: References: Message-ID: Robin Sommer created BIT-1152: --------------------------------- Summary: BroControl version check Key: BIT-1152 URL: https://bro-tracker.atlassian.net/browse/BIT-1152 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Reporter: Robin Sommer Assignee: Daniel Thayer Fix For: 2.3 Show warning if version has been upgraded. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 7 12:10:18 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Fri, 7 Mar 2014 14:10:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-845) PF_RING+DNA In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-845: -------------------------- Assignee: Seth Hall (was: Daniel Thayer) > PF_RING+DNA > ----------- > > Key: BIT-845 > URL: https://bro-tracker.atlassian.net/browse/BIT-845 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: BroControl > Affects Versions: git/master > Reporter: Daniel Thayer > Assignee: Seth Hall > Fix For: 2.3 > > Attachments: lb_pf_ring_dna.py > > > This is a feature that didn't make it into 2.1-beta. > The idea is to have a broctl plugin that has a pre-start > hook to automatically run this on each worker host: > pfdnacluster_master \-i dna0 \-c 21 \-n > A worker entry in node.cfg would look something like this: > [worker-1] > type=worker > host=host1 > interface=dna0 > lb_procs=4 > lb_method=pf_ring_dna -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 7 12:10:18 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Fri, 7 Mar 2014 14:10:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-845) PF_RING+DNA In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-845: -------------------------- Fix Version/s: (was: 2.4) 2.3 > PF_RING+DNA > ----------- > > Key: BIT-845 > URL: https://bro-tracker.atlassian.net/browse/BIT-845 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: BroControl > Affects Versions: git/master > Reporter: Daniel Thayer > Assignee: Daniel Thayer > Fix For: 2.3 > > Attachments: lb_pf_ring_dna.py > > > This is a feature that didn't make it into 2.1-beta. > The idea is to have a broctl plugin that has a pre-start > hook to automatically run this on each worker host: > pfdnacluster_master \-i dna0 \-c 21 \-n > A worker entry in node.cfg would look something like this: > [worker-1] > type=worker > host=host1 > interface=dna0 > lb_procs=4 > lb_method=pf_ring_dna -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 7 12:12:18 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Fri, 7 Mar 2014 14:12:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-845) PF_RING+DNA In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15711#comment-15711 ] Seth Hall commented on BIT-845: ------------------------------- I committed a new branch a while ago that improves this. I'm planning on getting some documentation written and merged in for 2.3. > PF_RING+DNA > ----------- > > Key: BIT-845 > URL: https://bro-tracker.atlassian.net/browse/BIT-845 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: BroControl > Affects Versions: git/master > Reporter: Daniel Thayer > Assignee: Seth Hall > Fix For: 2.3 > > Attachments: lb_pf_ring_dna.py > > > This is a feature that didn't make it into 2.1-beta. > The idea is to have a broctl plugin that has a pre-start > hook to automatically run this on each worker host: > pfdnacluster_master \-i dna0 \-c 21 \-n > A worker entry in node.cfg would look something like this: > [worker-1] > type=worker > host=host1 > interface=dna0 > lb_procs=4 > lb_method=pf_ring_dna -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 7 12:15:18 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Fri, 7 Mar 2014 14:15:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-845) PF_RING+DNA In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15712#comment-15712 ] Daniel Thayer commented on BIT-845: ----------------------------------- Branch topic/dnthayer/ticket845 implements this functionality (but it does not manage the pfdnacluster_master process). > PF_RING+DNA > ----------- > > Key: BIT-845 > URL: https://bro-tracker.atlassian.net/browse/BIT-845 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: BroControl > Affects Versions: git/master > Reporter: Daniel Thayer > Assignee: Seth Hall > Fix For: 2.3 > > Attachments: lb_pf_ring_dna.py > > > This is a feature that didn't make it into 2.1-beta. > The idea is to have a broctl plugin that has a pre-start > hook to automatically run this on each worker host: > pfdnacluster_master \-i dna0 \-c 21 \-n > A worker entry in node.cfg would look something like this: > [worker-1] > type=worker > host=host1 > interface=dna0 > lb_procs=4 > lb_method=pf_ring_dna -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 7 12:17:18 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Fri, 7 Mar 2014 14:17:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-78) Binpac DNS Analyzer does not use dns_skip_* settings In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-78?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-78: ------------------------- Resolution: Invalid Status: Closed (was: Open) The binpac dns analyzer is gone. > Binpac DNS Analyzer does not use dns_skip_* settings > ---------------------------------------------------- > > Key: BIT-78 > URL: https://bro-tracker.atlassian.net/browse/BIT-78 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 1.5.2 > Reporter: gregor > Priority: Low > Labels: analyzer, binpac, dns, dns_skip > > The binpac based DNS Analyzer ignores the dns_skip_* settings, that are defined in bro.init -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Sat Mar 8 00:00:13 2014 From: noreply at bro.org (Merge Tracker) Date: Sat, 8 Mar 2014 00:00:13 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403080800.s2880DPh031422@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ---------- ------------ ---------- ------------- ---------- ---------------------------------------------------------- BIT-1147 [1] Bro Seth Hall Robin Sommer 2014-03-06 - Normal topic/seth/dns-srv-fix - Fixing some problems with DNS [2] Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- --------- ---------- -------------------------- 4fd1098 [3] bro Jon Siwek 2014-03-04 Misc. documentation fixes. a2c23b4 [4] btest Jon Siwek 2014-03-04 Fix a link in the README. [1] BIT-1147 https://bro-tracker.atlassian.net/browse/BIT-1147 [2] dns-srv-fix https://github.com/bro/bro/tree/topic/seth/dns-srv-fix [3] 4fd1098 https://github.com/bro/bro/commit/4fd1098949183a4c0e0f4aa7aa724220a1929d19 [4] a2c23b4 https://github.com/bro/btest/commit/a2c23b4ffe4b58f0b5c186f156f9111f15ac4a3b From jira at bro-tracker.atlassian.net Sat Mar 8 05:37:18 2014 From: jira at bro-tracker.atlassian.net (gclark (JIRA)) Date: Sat, 8 Mar 2014 07:37:18 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1139) MHR lookups can cause significant CPU overhead in tests In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714#comment-15714 ] gclark commented on BIT-1139: ----------------------------- Hi Robin: Was going to comment on these tickets, but seem to lack permissions to do so. If you don't mind taking a look, I'll update these tickets with current status. Cheers, Gilbert > MHR lookups can cause significant CPU overhead in tests > ------------------------------------------------------- > > Key: BIT-1139 > URL: https://bro-tracker.atlassian.net/browse/BIT-1139 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > Live operation seems fine, need to understand what's going on. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Sun Mar 9 00:00:13 2014 From: noreply at bro.org (Merge Tracker) Date: Sun, 9 Mar 2014 00:00:13 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403090800.s2980DpW008873@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ---------- ------------ ---------- ------------- ---------- ---------------------------------------------------------- BIT-1147 [1] Bro Seth Hall Robin Sommer 2014-03-06 - Normal topic/seth/dns-srv-fix - Fixing some problems with DNS [2] Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- --------- ---------- -------------------------- 4fd1098 [3] bro Jon Siwek 2014-03-04 Misc. documentation fixes. a2c23b4 [4] btest Jon Siwek 2014-03-04 Fix a link in the README. [1] BIT-1147 https://bro-tracker.atlassian.net/browse/BIT-1147 [2] dns-srv-fix https://github.com/bro/bro/tree/topic/seth/dns-srv-fix [3] 4fd1098 https://github.com/bro/bro/commit/4fd1098949183a4c0e0f4aa7aa724220a1929d19 [4] a2c23b4 https://github.com/bro/btest/commit/a2c23b4ffe4b58f0b5c186f156f9111f15ac4a3b From jira at bro-tracker.atlassian.net Sun Mar 9 13:18:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Sun, 9 Mar 2014 15:18:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1147) topic/seth/dns-srv-fix - Fixing some problems with DNS In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15715#comment-15715 ] Robin Sommer commented on BIT-1147: ----------------------------------- Let me ask the expected question: any better way to recognize NetBios than hard-coding the port? > topic/seth/dns-srv-fix - Fixing some problems with DNS > ------------------------------------------------------ > > Key: BIT-1147 > URL: https://bro-tracker.atlassian.net/browse/BIT-1147 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > Assignee: Robin Sommer > > This branch and equivalently named branches are ready for merging in the public and private test suites. > We generate the event for SRV responses in DNS now. > Fixed several annoying issues with NetBios name service requests and responses. Fewer incorrect weirds and more correct dns logs now. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Mon Mar 10 00:00:13 2014 From: noreply at bro.org (Merge Tracker) Date: Mon, 10 Mar 2014 00:00:13 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403100700.s2A70Dvf004304@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ---------- ------------ ---------- ------------- ---------- ---------------------------------------------------------- BIT-1147 [1] Bro Seth Hall Robin Sommer 2014-03-09 - Normal topic/seth/dns-srv-fix - Fixing some problems with DNS [2] Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- --------- ---------- -------------------------- 4fd1098 [3] bro Jon Siwek 2014-03-04 Misc. documentation fixes. a2c23b4 [4] btest Jon Siwek 2014-03-04 Fix a link in the README. [1] BIT-1147 https://bro-tracker.atlassian.net/browse/BIT-1147 [2] dns-srv-fix https://github.com/bro/bro/tree/topic/seth/dns-srv-fix [3] 4fd1098 https://github.com/bro/bro/commit/4fd1098949183a4c0e0f4aa7aa724220a1929d19 [4] a2c23b4 https://github.com/bro/btest/commit/a2c23b4ffe4b58f0b5c186f156f9111f15ac4a3b From jira at bro-tracker.atlassian.net Mon Mar 10 05:51:18 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Mon, 10 Mar 2014 07:51:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1147) topic/seth/dns-srv-fix - Fixing some problems with DNS In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15716#comment-15716 ] Seth Hall commented on BIT-1147: -------------------------------- Hm.. not really. It's probably the most reliable technique to identify it. They literally use the exact same DNS structure, we just encountered reuse of a RR identifier between NBNS and one of the DNS RFCs. We're actually using the port mechanism to identify NBNS queries in script-land anyway (to decide when to decode the encoded MS host names). > topic/seth/dns-srv-fix - Fixing some problems with DNS > ------------------------------------------------------ > > Key: BIT-1147 > URL: https://bro-tracker.atlassian.net/browse/BIT-1147 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > Assignee: Robin Sommer > > This branch and equivalently named branches are ready for merging in the public and private test suites. > We generate the event for SRV responses in DNS now. > Fixed several annoying issues with NetBios name service requests and responses. Fewer incorrect weirds and more correct dns logs now. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 10 06:01:18 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Mon, 10 Mar 2014 08:01:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-845) PF_RING+DNA In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15717#comment-15717 ] Seth Hall commented on BIT-845: ------------------------------- I rewrote the code for the plugin and merged it into the pf_ring plugin so we only have one plugin for pf_ring with DNA and without DNA. We'll remove the ticket845 branch when we merge my new branch. Thanks. > PF_RING+DNA > ----------- > > Key: BIT-845 > URL: https://bro-tracker.atlassian.net/browse/BIT-845 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: BroControl > Affects Versions: git/master > Reporter: Daniel Thayer > Assignee: Seth Hall > Fix For: 2.3 > > Attachments: lb_pf_ring_dna.py > > > This is a feature that didn't make it into 2.1-beta. > The idea is to have a broctl plugin that has a pre-start > hook to automatically run this on each worker host: > pfdnacluster_master \-i dna0 \-c 21 \-n > A worker entry in node.cfg would look something like this: > [worker-1] > type=worker > host=host1 > interface=dna0 > lb_procs=4 > lb_method=pf_ring_dna -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 10 07:57:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 10 Mar 2014 09:57:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1147) topic/seth/dns-srv-fix - Fixing some problems with DNS In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1147: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > topic/seth/dns-srv-fix - Fixing some problems with DNS > ------------------------------------------------------ > > Key: BIT-1147 > URL: https://bro-tracker.atlassian.net/browse/BIT-1147 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > Assignee: Robin Sommer > > This branch and equivalently named branches are ready for merging in the public and private test suites. > We generate the event for SRV responses in DNS now. > Fixed several annoying issues with NetBios name service requests and responses. Fewer incorrect weirds and more correct dns logs now. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 10 07:57:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 10 Mar 2014 09:57:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1147) topic/seth/dns-srv-fix - Fixing some problems with DNS In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15718#comment-15718 ] Robin Sommer commented on BIT-1147: ----------------------------------- Yeah, I saw that, but it's uglier to hardcode a port inside the event engine. But I was expecting the "no" answer. :) (Wondering if anybody has ever seen NB on a non-standard port?) Anyways, going to push the merge. > topic/seth/dns-srv-fix - Fixing some problems with DNS > ------------------------------------------------------ > > Key: BIT-1147 > URL: https://bro-tracker.atlassian.net/browse/BIT-1147 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > Assignee: Robin Sommer > > This branch and equivalently named branches are ready for merging in the public and private test suites. > We generate the event for SRV responses in DNS now. > Fixed several annoying issues with NetBios name service requests and responses. Fewer incorrect weirds and more correct dns logs now. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 10 08:01:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 10 Mar 2014 10:01:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1153) DNS inconsistency In-Reply-To: References: Message-ID: Robin Sommer created BIT-1153: --------------------------------- Summary: DNS inconsistency Key: BIT-1153 URL: https://bro-tracker.atlassian.net/browse/BIT-1153 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Reporter: Robin Sommer Fix For: 2.3 Something's not deterministic in the DNS analyzer, this is with a small trace of just 6 empty DNS replies with different transaction IDs:: {code} # ( bro -b -r dns2-anon.trace base/protocols/dns && cat dns.log ) >>log # ( bro -b -r dns2-anon.trace base/protocols/dns && cat dns.log ) >>log # ( bro -b -r dns2-anon.trace base/protocols/dns && cat dns.log ) >>log # cat log #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path dns #open 2014-03-09-21-36-40 #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected #types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool 1359400918.103013 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 50261 - - - - - 3 NXDOMAIN F F F F 0 - - F 1359400918.102517 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 14740 - - - - - 3 NXDOMAIN F F F F 0 - - F 1359400918.103641 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 22908 - - - - - 3 NXDOMAIN F F F F 0 - - F 1359400918.102812 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 58133 - - - - - 3 NXDOMAIN F F F F 0 - - F #close 2014-03-09-21-36-40 #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path dns #open 2014-03-09-21-36-42 #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected #types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool 1359400918.102812 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 58133 - - - - - 3 NXDOMAIN F F F F 0 - - F 1359400918.104054 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 45557 - - - - - 3 NXDOMAIN F F F F 0 - - F 1359400918.103013 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 50261 - - - - - 3 NXDOMAIN F F F F 0 - - F 1359400918.102517 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 14740 - - - - - 3 NXDOMAIN F F F F 0 - - F 1359400918.103390 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 31341 - - - - - 3 NXDOMAIN F F F F 0 - - F #close 2014-03-09-21-36-42 #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path dns #open 2014-03-09-21-36-43 #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected #types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool 1359400918.103641 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 22908 - - - - - 3 NXDOMAIN F F F F 0 - - F 1359400918.103390 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 31341 - - - - - 3 NXDOMAIN F F F F 0 - - F 1359400918.103013 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 50261 - - - - - 3 NXDOMAIN F F F F 0 - - F 1359400918.102517 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 14740 - - - - - 3 NXDOMAIN F F F F 0 - - F 1359400918.102812 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 58133 - - - - - 3 NXDOMAIN F F F F 0 - - F 1359400918.104054 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 45557 - - - - - 3 NXDOMAIN F F F F 0 - - F #close 2014-03-09-21-36-43 {code} I'll provide the trace on request, don't want to attach it here. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 10 09:40:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 10 Mar 2014 11:40:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1153) DNS inconsistency In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1153: --------------------------- Status: Merge Request (was: Open) > DNS inconsistency > ----------------- > > Key: BIT-1153 > URL: https://bro-tracker.atlassian.net/browse/BIT-1153 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Fix For: 2.3 > > > Something's not deterministic in the DNS analyzer, this is with a small trace of just 6 empty DNS replies with different transaction IDs:: > {code} > # ( bro -b -r dns2-anon.trace base/protocols/dns && cat dns.log ) >>log > # ( bro -b -r dns2-anon.trace base/protocols/dns && cat dns.log ) >>log > # ( bro -b -r dns2-anon.trace base/protocols/dns && cat dns.log ) >>log > # cat log > #separator \x09 > #set_separator , > #empty_field (empty) > #unset_field - > #path dns > #open 2014-03-09-21-36-40 > #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected > #types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool > 1359400918.103013 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 50261 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102517 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 14740 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103641 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 22908 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102812 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 58133 - - - - - 3 NXDOMAIN F F F F 0 - - F > #close 2014-03-09-21-36-40 > #separator \x09 > #set_separator , > #empty_field (empty) > #unset_field - > #path dns > #open 2014-03-09-21-36-42 > #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected > #types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool > 1359400918.102812 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 58133 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.104054 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 45557 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103013 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 50261 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102517 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 14740 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103390 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 31341 - - - - - 3 NXDOMAIN F F F F 0 - - F > #close 2014-03-09-21-36-42 > #separator \x09 > #set_separator , > #empty_field (empty) > #unset_field - > #path dns > #open 2014-03-09-21-36-43 > #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected > #types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool > 1359400918.103641 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 22908 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103390 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 31341 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103013 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 50261 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102517 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 14740 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102812 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 58133 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.104054 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 45557 - - - - - 3 NXDOMAIN F F F F 0 - - F > #close 2014-03-09-21-36-43 > {code} > I'll provide the trace on request, don't want to attach it here. > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 10 09:40:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 10 Mar 2014 11:40:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1153) DNS inconsistency In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15719#comment-15719 ] Jon Siwek commented on BIT-1153: -------------------------------- topic/jsiwek/bit-1153 in bro, bro-testing, bro-testing-private > DNS inconsistency > ----------------- > > Key: BIT-1153 > URL: https://bro-tracker.atlassian.net/browse/BIT-1153 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Fix For: 2.3 > > > Something's not deterministic in the DNS analyzer, this is with a small trace of just 6 empty DNS replies with different transaction IDs:: > {code} > # ( bro -b -r dns2-anon.trace base/protocols/dns && cat dns.log ) >>log > # ( bro -b -r dns2-anon.trace base/protocols/dns && cat dns.log ) >>log > # ( bro -b -r dns2-anon.trace base/protocols/dns && cat dns.log ) >>log > # cat log > #separator \x09 > #set_separator , > #empty_field (empty) > #unset_field - > #path dns > #open 2014-03-09-21-36-40 > #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected > #types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool > 1359400918.103013 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 50261 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102517 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 14740 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103641 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 22908 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102812 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 58133 - - - - - 3 NXDOMAIN F F F F 0 - - F > #close 2014-03-09-21-36-40 > #separator \x09 > #set_separator , > #empty_field (empty) > #unset_field - > #path dns > #open 2014-03-09-21-36-42 > #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected > #types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool > 1359400918.102812 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 58133 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.104054 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 45557 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103013 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 50261 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102517 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 14740 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103390 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 31341 - - - - - 3 NXDOMAIN F F F F 0 - - F > #close 2014-03-09-21-36-42 > #separator \x09 > #set_separator , > #empty_field (empty) > #unset_field - > #path dns > #open 2014-03-09-21-36-43 > #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected > #types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool > 1359400918.103641 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 22908 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103390 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 31341 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103013 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 50261 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102517 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 14740 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102812 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 58133 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.104054 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 45557 - - - - - 3 NXDOMAIN F F F F 0 - - F > #close 2014-03-09-21-36-43 > {code} > I'll provide the trace on request, don't want to attach it here. > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 10 09:46:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 10 Mar 2014 11:46:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1153) DNS inconsistency In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15720#comment-15720 ] Jon Siwek commented on BIT-1153: -------------------------------- (The order of log entries will still be non-deterministic unless using a seed.) > DNS inconsistency > ----------------- > > Key: BIT-1153 > URL: https://bro-tracker.atlassian.net/browse/BIT-1153 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Fix For: 2.3 > > > Something's not deterministic in the DNS analyzer, this is with a small trace of just 6 empty DNS replies with different transaction IDs:: > {code} > # ( bro -b -r dns2-anon.trace base/protocols/dns && cat dns.log ) >>log > # ( bro -b -r dns2-anon.trace base/protocols/dns && cat dns.log ) >>log > # ( bro -b -r dns2-anon.trace base/protocols/dns && cat dns.log ) >>log > # cat log > #separator \x09 > #set_separator , > #empty_field (empty) > #unset_field - > #path dns > #open 2014-03-09-21-36-40 > #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected > #types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool > 1359400918.103013 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 50261 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102517 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 14740 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103641 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 22908 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102812 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 58133 - - - - - 3 NXDOMAIN F F F F 0 - - F > #close 2014-03-09-21-36-40 > #separator \x09 > #set_separator , > #empty_field (empty) > #unset_field - > #path dns > #open 2014-03-09-21-36-42 > #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected > #types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool > 1359400918.102812 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 58133 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.104054 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 45557 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103013 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 50261 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102517 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 14740 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103390 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 31341 - - - - - 3 NXDOMAIN F F F F 0 - - F > #close 2014-03-09-21-36-42 > #separator \x09 > #set_separator , > #empty_field (empty) > #unset_field - > #path dns > #open 2014-03-09-21-36-43 > #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected > #types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool > 1359400918.103641 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 22908 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103390 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 31341 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103013 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 50261 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102517 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 14740 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102812 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 58133 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.104054 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 45557 - - - - - 3 NXDOMAIN F F F F 0 - - F > #close 2014-03-09-21-36-43 > {code} > I'll provide the trace on request, don't want to attach it here. > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 10 10:55:19 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Mon, 10 Mar 2014 12:55:19 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-760) Lift Server Alternative Name (SAN) field to scripting layer In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-760: ------------------------------- Resolution: Fixed Status: Closed (was: Open) > Lift Server Alternative Name (SAN) field to scripting layer > ----------------------------------------------------------- > > Key: BIT-760 > URL: https://bro-tracker.atlassian.net/browse/BIT-760 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Matthias Vallentin > Assignee: Bernhard Amann > Labels: analyzer > Fix For: 2.4 > > > It would be nice to have the *Subject Alternative Name (SAN)* field of an X.509 certificate available at the scripting layer. It contains a list of domains that should be used in addition to the CN field of the subject to verify that a domain matches the certificate. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 10 11:24:19 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Mon, 10 Mar 2014 13:24:19 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-760) Lift Server Alternative Name (SAN) field to scripting layer In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-760: ------------------------------- Status: Reopened (was: Closed) > Lift Server Alternative Name (SAN) field to scripting layer > ----------------------------------------------------------- > > Key: BIT-760 > URL: https://bro-tracker.atlassian.net/browse/BIT-760 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Matthias Vallentin > Assignee: Bernhard Amann > Labels: analyzer > Fix For: 2.4 > > > It would be nice to have the *Subject Alternative Name (SAN)* field of an X.509 certificate available at the scripting layer. It contains a list of domains that should be used in addition to the CN field of the subject to verify that a domain matches the certificate. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 10 11:26:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 10 Mar 2014 13:26:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1153) DNS inconsistency In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1153: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > DNS inconsistency > ----------------- > > Key: BIT-1153 > URL: https://bro-tracker.atlassian.net/browse/BIT-1153 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Fix For: 2.3 > > > Something's not deterministic in the DNS analyzer, this is with a small trace of just 6 empty DNS replies with different transaction IDs:: > {code} > # ( bro -b -r dns2-anon.trace base/protocols/dns && cat dns.log ) >>log > # ( bro -b -r dns2-anon.trace base/protocols/dns && cat dns.log ) >>log > # ( bro -b -r dns2-anon.trace base/protocols/dns && cat dns.log ) >>log > # cat log > #separator \x09 > #set_separator , > #empty_field (empty) > #unset_field - > #path dns > #open 2014-03-09-21-36-40 > #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected > #types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool > 1359400918.103013 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 50261 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102517 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 14740 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103641 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 22908 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102812 C3UnB71Lb5jHQuxYi9 10.69.49.58 41664 10.32.136.13 53 udp 58133 - - - - - 3 NXDOMAIN F F F F 0 - - F > #close 2014-03-09-21-36-40 > #separator \x09 > #set_separator , > #empty_field (empty) > #unset_field - > #path dns > #open 2014-03-09-21-36-42 > #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected > #types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool > 1359400918.102812 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 58133 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.104054 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 45557 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103013 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 50261 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102517 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 14740 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103390 CF4yYh4S0wIWnHYKka 10.69.49.58 41664 10.32.136.13 53 udp 31341 - - - - - 3 NXDOMAIN F F F F 0 - - F > #close 2014-03-09-21-36-42 > #separator \x09 > #set_separator , > #empty_field (empty) > #unset_field - > #path dns > #open 2014-03-09-21-36-43 > #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected > #types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool > 1359400918.103641 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 22908 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103390 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 31341 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.103013 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 50261 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102517 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 14740 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.102812 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 58133 - - - - - 3 NXDOMAIN F F F F 0 - - F > 1359400918.104054 CrJZTqkaJJe3L4VUk 10.69.49.58 41664 10.32.136.13 53 udp 45557 - - - - - 3 NXDOMAIN F F F F 0 - - F > #close 2014-03-09-21-36-43 > {code} > I'll provide the trace on request, don't want to attach it here. > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 10 11:50:18 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Mon, 10 Mar 2014 13:50:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1154) Formatters restructed in: topic/seth/json-formatter In-Reply-To: References: Message-ID: Seth Hall created BIT-1154: ------------------------------ Summary: Formatters restructed in: topic/seth/json-formatter Key: BIT-1154 URL: https://bro-tracker.atlassian.net/browse/BIT-1154 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: 2.3 Reporter: Seth Hall Assignee: Robin Sommer topic/seth/json-formatter has an abstraction for Formatters and I created a formatters directory under threading. There is also a new JSON formatter and support in the Ascii and ElasticSearch writers for the JSON formatter. I went ahead and threw in per-filter configuration options for the Ascii writer for all of the options that were exposed globally too. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 10 11:50:18 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Mon, 10 Mar 2014 13:50:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1154) Formatters restructed in: topic/seth/json-formatter In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1154: --------------------------- Status: Merge Request (was: Open) > Formatters restructed in: topic/seth/json-formatter > --------------------------------------------------- > > Key: BIT-1154 > URL: https://bro-tracker.atlassian.net/browse/BIT-1154 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > Assignee: Robin Sommer > > topic/seth/json-formatter has an abstraction for Formatters and I created a formatters directory under threading. There is also a new JSON formatter and support in the Ascii and ElasticSearch writers for the JSON formatter. > I went ahead and threw in per-filter configuration options for the Ascii writer for all of the options that were exposed globally too. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 10 12:49:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 10 Mar 2014 14:49:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1128) Add configure options for linking against jemalloc In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721#comment-15721 ] Jon Siwek commented on BIT-1128: -------------------------------- topic/jsiwek/jemalloc in bro & cmake repos > Add configure options for linking against jemalloc > -------------------------------------------------- > > Key: BIT-1128 > URL: https://bro-tracker.atlassian.net/browse/BIT-1128 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > To gather experiences with using jemalloc, add a configure options --with-jemalloc= that links Bro against it if found. Default should be off. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 10 12:49:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 10 Mar 2014 14:49:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1128) Add configure options for linking against jemalloc In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1128: --------------------------- Status: Merge Request (was: Open) > Add configure options for linking against jemalloc > -------------------------------------------------- > > Key: BIT-1128 > URL: https://bro-tracker.atlassian.net/browse/BIT-1128 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > To gather experiences with using jemalloc, add a configure options --with-jemalloc= that links Bro against it if found. Default should be off. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 10 14:57:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 10 Mar 2014 16:57:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1134) DNS_Mgr::LookupAddr does not respect DNS_FAKE In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1134: --------------------------- Status: Merge Request (was: Open) > DNS_Mgr::LookupAddr does not respect DNS_FAKE > --------------------------------------------- > > Key: BIT-1134 > URL: https://bro-tracker.atlassian.net/browse/BIT-1134 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 10 14:57:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 10 Mar 2014 16:57:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1134) DNS_Mgr::LookupAddr does not respect DNS_FAKE In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15722#comment-15722 ] Jon Siwek commented on BIT-1134: -------------------------------- topic/jsiwek/dns_fake in bro and bro-testing-private repos > DNS_Mgr::LookupAddr does not respect DNS_FAKE > --------------------------------------------- > > Key: BIT-1134 > URL: https://bro-tracker.atlassian.net/browse/BIT-1134 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Tue Mar 11 00:00:11 2014 From: noreply at bro.org (Merge Tracker) Date: Tue, 11 Mar 2014 00:00:11 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403110700.s2B70B8t023982@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ------------ ---------- ------------- ---------- --------------------------------------------------- BIT-1154 [1] Bro Seth Hall Robin Sommer 2014-03-10 - Normal Formatters restructed in: topic/seth/json-formatter BIT-1134 [2] Bro Justin Azoff Jon Siwek 2014-03-10 2.3 Low DNS_Mgr::LookupAddr does not respect DNS_FAKE BIT-1128 [3] Bro Robin Sommer Jon Siwek 2014-03-10 2.3 Normal Add configure options for linking against jemalloc Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- --------- ---------- -------------------------- 4fd1098 [4] bro Jon Siwek 2014-03-04 Misc. documentation fixes. a2c23b4 [5] btest Jon Siwek 2014-03-04 Fix a link in the README. [1] BIT-1154 https://bro-tracker.atlassian.net/browse/BIT-1154 [2] BIT-1134 https://bro-tracker.atlassian.net/browse/BIT-1134 [3] BIT-1128 https://bro-tracker.atlassian.net/browse/BIT-1128 [4] 4fd1098 https://github.com/bro/bro/commit/4fd1098949183a4c0e0f4aa7aa724220a1929d19 [5] a2c23b4 https://github.com/bro/btest/commit/a2c23b4ffe4b58f0b5c186f156f9111f15ac4a3b From jira at bro-tracker.atlassian.net Tue Mar 11 11:43:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 11 Mar 2014 13:43:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1139) MHR lookups can cause significant CPU overhead in tests In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15723#comment-15723 ] Jon Siwek commented on BIT-1139: -------------------------------- topic/jsiwek/faster-mhr in just the bro repo. It's purely a change in Bro scripts, so assigning to Seth to review, but general feedback also nice. The problem is mostly w/ the fact that the "when" statement involved in the MHR lookup ends up cloning a fa_file record, which is expensive. The change in the branch sidesteps this by unrolling the needed fields from the fa_file record before the scope of the "when" statement to avoid cloning the full data structure. I can see benefit in following up w/ a more robust answer to the potential cost of "when" statements, but I'd rather not have to touch the serialization or trigger code (at least for this release). Also I don't get the comment in the ticket description about live operation exhibiting different behavior. I'd expect it to be the same deal provided that the live traffic includes enough files in {{TeamCymruMalwareHashRegistry::match_file_types}} for the "when" stmt to actually get hit. > MHR lookups can cause significant CPU overhead in tests > ------------------------------------------------------- > > Key: BIT-1139 > URL: https://bro-tracker.atlassian.net/browse/BIT-1139 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > Live operation seems fine, need to understand what's going on. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 11 11:43:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 11 Mar 2014 13:43:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1139) MHR lookups can cause significant CPU overhead in tests In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1139: --------------------------- Assignee: Seth Hall (was: Jon Siwek) > MHR lookups can cause significant CPU overhead in tests > ------------------------------------------------------- > > Key: BIT-1139 > URL: https://bro-tracker.atlassian.net/browse/BIT-1139 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > > Live operation seems fine, need to understand what's going on. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 11 11:43:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 11 Mar 2014 13:43:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1139) MHR lookups can cause significant CPU overhead in tests In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1139: --------------------------- Status: Merge Request (was: Open) > MHR lookups can cause significant CPU overhead in tests > ------------------------------------------------------- > > Key: BIT-1139 > URL: https://bro-tracker.atlassian.net/browse/BIT-1139 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > Live operation seems fine, need to understand what's going on. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 11 11:58:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 11 Mar 2014 13:58:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1155) bro.org sidebar breadcrumbs links often broken In-Reply-To: References: Message-ID: Jon Siwek created BIT-1155: ------------------------------ Summary: bro.org sidebar breadcrumbs links often broken Key: BIT-1155 URL: https://bro-tracker.atlassian.net/browse/BIT-1155 Project: Bro Issue Tracker Issue Type: Problem Components: Website Affects Versions: git/master Reporter: Jon Siwek Assignee: Jon Siwek On pages like http://www.bro.org/bro-exchange-2013/exercises/faf.html the sidebar will have links to http://www.bro.org/bro-exchange-2013/exercises and http://www.bro.org/bro-exchange-2013, which don't have any index and so generate an access error. Need to see if those links can be auto-generated more intelligently. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 11 15:11:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 11 Mar 2014 17:11:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1148) Bug in Connection::FlipRoles In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1148: --------------------------- Status: Merge Request (was: Open) > Bug in Connection::FlipRoles > ---------------------------- > > Key: BIT-1148 > URL: https://bro-tracker.atlassian.net/browse/BIT-1148 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Jon Siwek > Fix For: 2.3 > > > This method doesn't correctly swap address values. > Also, since scheduled analyzers for a connection are looked up based on the endpoint addresses, it's possible they will miss being attached to connections that end up taking the Connection::FlipRoles code path. An idea to fix would be just to have FlipRoles do a check for scheduled analyzers on the new connection tuple and attach any that turn up. > (These were reported by Kevin McMahon on the bro-dev list). -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Wed Mar 12 00:00:15 2014 From: noreply at bro.org (Merge Tracker) Date: Wed, 12 Mar 2014 00:00:15 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403120700.s2C70Fhq010671@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ------------ ---------- ------------- ---------- ------------------------------------------------------- BIT-1154 [1] Bro Seth Hall Robin Sommer 2014-03-10 - Normal Formatters restructed in: topic/seth/json-formatter BIT-1148 [2] Bro Jon Siwek - 2014-03-11 2.3 Normal Bug in Connection::FlipRoles BIT-1139 [3] Bro Robin Sommer Seth Hall 2014-03-11 2.3 Normal MHR lookups can cause significant CPU overhead in tests BIT-1134 [4] Bro Justin Azoff Jon Siwek 2014-03-10 2.3 Low DNS_Mgr::LookupAddr does not respect DNS_FAKE BIT-1128 [5] Bro Robin Sommer Jon Siwek 2014-03-10 2.3 Normal Add configure options for linking against jemalloc Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- --------- ---------- -------------------------- 4fd1098 [6] bro Jon Siwek 2014-03-04 Misc. documentation fixes. a2c23b4 [7] btest Jon Siwek 2014-03-04 Fix a link in the README. [1] BIT-1154 https://bro-tracker.atlassian.net/browse/BIT-1154 [2] BIT-1148 https://bro-tracker.atlassian.net/browse/BIT-1148 [3] BIT-1139 https://bro-tracker.atlassian.net/browse/BIT-1139 [4] BIT-1134 https://bro-tracker.atlassian.net/browse/BIT-1134 [5] BIT-1128 https://bro-tracker.atlassian.net/browse/BIT-1128 [6] 4fd1098 https://github.com/bro/bro/commit/4fd1098949183a4c0e0f4aa7aa724220a1929d19 [7] a2c23b4 https://github.com/bro/btest/commit/a2c23b4ffe4b58f0b5c186f156f9111f15ac4a3b From jira at bro-tracker.atlassian.net Wed Mar 12 17:33:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Wed, 12 Mar 2014 19:33:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1128) Add configure options for linking against jemalloc In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1128: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Add configure options for linking against jemalloc > -------------------------------------------------- > > Key: BIT-1128 > URL: https://bro-tracker.atlassian.net/browse/BIT-1128 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > To gather experiences with using jemalloc, add a configure options --with-jemalloc= that links Bro against it if found. Default should be off. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Wed Mar 12 17:51:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Wed, 12 Mar 2014 19:51:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1156) DNS analyzer parses TXT records imcompletely In-Reply-To: References: Message-ID: Robin Sommer created BIT-1156: --------------------------------- Summary: DNS analyzer parses TXT records imcompletely Key: BIT-1156 URL: https://bro-tracker.atlassian.net/browse/BIT-1156 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Reporter: Robin Sommer Fix For: 2.3 The payload of DNS TXT records can consist of multiple character strings but the DNS analyzer parses out only the first. We should parse them out all and then probably concatenate into a single string to pass to the event, separated with semicolons or something. I have a trace with an example but it would need anonymization before inclusion into the test suite. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Wed Mar 12 18:02:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Wed, 12 Mar 2014 20:02:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1156) DNS analyzer parses TXT records imcompletely In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15724#comment-15724 ] Robin Sommer commented on BIT-1156: ----------------------------------- ,,, or better: pass a set[string] to the event. > DNS analyzer parses TXT records imcompletely > -------------------------------------------- > > Key: BIT-1156 > URL: https://bro-tracker.atlassian.net/browse/BIT-1156 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Fix For: 2.3 > > > The payload of DNS TXT records can consist of multiple character strings but the DNS analyzer parses out only the first. We should parse them out all and then probably concatenate into a single string to pass to the event, separated with semicolons or something. > I have a trace with an example but it would need anonymization before inclusion into the test suite. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Thu Mar 13 00:00:21 2014 From: noreply at bro.org (Merge Tracker) Date: Thu, 13 Mar 2014 00:00:21 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403130700.s2D70Lk2011973@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ------------ ---------- ------------- ---------- ------------------------------------------------------- BIT-1154 [1] Bro Seth Hall Robin Sommer 2014-03-10 - Normal Formatters restructed in: topic/seth/json-formatter BIT-1148 [2] Bro Jon Siwek - 2014-03-11 2.3 Normal Bug in Connection::FlipRoles BIT-1139 [3] Bro Robin Sommer Seth Hall 2014-03-11 2.3 Normal MHR lookups can cause significant CPU overhead in tests BIT-1134 [4] Bro Justin Azoff Jon Siwek 2014-03-10 2.3 Low DNS_Mgr::LookupAddr does not respect DNS_FAKE Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- --------- ---------- ------------------------- a2c23b4 [5] btest Jon Siwek 2014-03-04 Fix a link in the README. [1] BIT-1154 https://bro-tracker.atlassian.net/browse/BIT-1154 [2] BIT-1148 https://bro-tracker.atlassian.net/browse/BIT-1148 [3] BIT-1139 https://bro-tracker.atlassian.net/browse/BIT-1139 [4] BIT-1134 https://bro-tracker.atlassian.net/browse/BIT-1134 [5] a2c23b4 https://github.com/bro/btest/commit/a2c23b4ffe4b58f0b5c186f156f9111f15ac4a3b From robin at icir.org Thu Mar 13 07:30:37 2014 From: robin at icir.org (Robin Sommer) Date: Thu, 13 Mar 2014 07:30:37 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/bernhard/file-analysis-x509: Change x509 log - now certificates are only logged once per hour. (0d50b8b) In-Reply-To: <201403130717.s2D7HusM012132@bro-ids.icir.org> References: <201403130717.s2D7HusM012132@bro-ids.icir.org> Message-ID: <20140313143037.GA56207@icir.org> On Thu, Mar 13, 2014 at 00:17 -0700, Bernhard Amann wrote: > You apparently have to be very careful which EndOfFile function of > the file analysis framework you call... otherwhise it might try > to close another file id. This took me quite a while to find. Can you elaborate? I sense an opportuntity to improve our API. :-) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From seth at icir.org Thu Mar 13 07:37:05 2014 From: seth at icir.org (Seth Hall) Date: Thu, 13 Mar 2014 10:37:05 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/bernhard/file-analysis-x509: Change x509 log - now certificates are only logged once per hour. (0d50b8b) In-Reply-To: <20140313143037.GA56207@icir.org> References: <201403130717.s2D7HusM012132@bro-ids.icir.org> <20140313143037.GA56207@icir.org> Message-ID: <80CC8A1F-9392-4CEC-878F-79BFEA877829@icir.org> On Mar 13, 2014, at 10:30 AM, Robin Sommer wrote: > On Thu, Mar 13, 2014 at 00:17 -0700, Bernhard Amann wrote: > >> You apparently have to be very careful which EndOfFile function of >> the file analysis framework you call... otherwhise it might try >> to close another file id. This took me quite a while to find. > > Can you elaborate? I sense an opportuntity to improve our API. :-) I think he was running into the interplay between script land and the core. Actually, I think that for the SSL/TLS analyzer, this is one of the times we don't need a file id generated in script land. That's probably a better choice in this case. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140313/c66d3ae1/attachment.bin From jsiwek at illinois.edu Thu Mar 13 07:38:24 2014 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Thu, 13 Mar 2014 14:38:24 +0000 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/bernhard/file-analysis-x509: Change x509 log - now certificates are only logged once per hour. (0d50b8b) In-Reply-To: <201403130717.s2D7HusM012132@bro-ids.icir.org> References: <201403130717.s2D7HusM012132@bro-ids.icir.org> Message-ID: <489D2417-9E14-44CD-87B8-EB47F3C481FE@illinois.edu> On Mar 13, 2014, at 2:17 AM, Bernhard Amann wrote: > You apparently have to be very careful which EndOfFile function of > the file analysis framework you call... otherwhise it might try > to close another file id. This took me quite a while to find. I think that should be the case for any methods of the file analysis interface that don?t use a pre-computed file id, but only if the file handles returned from the script layer ?get_file_handle? function end up differing between calls to the file analysis interface. So the question I?d try to answer would be ?are the file handles returned from my get_file_handle function differing unintentionally?? Ultimately, moving to the pre-computed file id interface like you did makes sense for this code since you know the two calls to the file analysis interface right next to each other are associated w/ the same file. - Jon From bernhard at ICSI.Berkeley.EDU Thu Mar 13 07:45:44 2014 From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann) Date: Thu, 13 Mar 2014 07:45:44 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/bernhard/file-analysis-x509: Change x509 log - now certificates are only logged once per hour. (0d50b8b) In-Reply-To: <80CC8A1F-9392-4CEC-878F-79BFEA877829@icir.org> References: <201403130717.s2D7HusM012132@bro-ids.icir.org> <20140313143037.GA56207@icir.org> <80CC8A1F-9392-4CEC-878F-79BFEA877829@icir.org> Message-ID: On Mar 13, 2014, at 7:37 AM, Seth Hall wrote: > > On Mar 13, 2014, at 10:30 AM, Robin Sommer wrote: > >> On Thu, Mar 13, 2014 at 00:17 -0700, Bernhard Amann wrote: >> >>> You apparently have to be very careful which EndOfFile function of >>> the file analysis framework you call... otherwhise it might try >>> to close another file id. This took me quite a while to find. >> >> Can you elaborate? I sense an opportuntity to improve our API. :-) > > > I think he was running into the interplay between script land and the core. Actually, I think that for the SSL/TLS analyzer, this is one of the times we don't need a file id generated in script land. That's probably a better choice in this case. That might be true, I am not entirely sure. What I did was to call? file_mgr->DataIn(reinterpret_cast(cert.data()), cert.length(), bro_analyzer()->GetAnalyzerTag(), bro_analyzer()->Conn(), ${rec.is_orig}); file_mgr->EndOfFile(bro_analyzer()->GetAnalyzerTag(), bro_analyzer()->Conn(), ${rec.is_orig}); in exactly this order (so - directly following each other). Which does not work. Changing it to... string fid = file_mgr->DataIn(reinterpret_cast(cert.data()), cert.length(), bro_analyzer()->GetAnalyzerTag(), bro_analyzer()->Conn(), ${rec.is_orig}); file_mgr->EndOfFile(fid); makes it work perfectly. Thinking about it, it makes sense that this might be due to the interplay of the script land file ID generation and the core file ID. However? just looking at the code, it still seems a bit non-intuitive that the first version does not work, and the second version does. I also do not really think this is sufficiently documented in the comments of Manager.h. This basically is not mentioned at all there? I have no idea if that is just me that got a wrong impression that this should work when looking at the Interface. But - I could see other people perhaps making the same mistake. Bernhard From jsiwek at illinois.edu Thu Mar 13 08:50:26 2014 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Thu, 13 Mar 2014 15:50:26 +0000 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/bernhard/file-analysis-x509: Change x509 log - now certificates are only logged once per hour. (0d50b8b) In-Reply-To: References: <201403130717.s2D7HusM012132@bro-ids.icir.org> <20140313143037.GA56207@icir.org> <80CC8A1F-9392-4CEC-878F-79BFEA877829@icir.org> Message-ID: <7D52BB42-31F4-441C-9775-F5F9BF36EE35@illinois.edu> On Mar 13, 2014, at 9:45 AM, Bernhard Amann wrote: > What I did was to call? > > file_mgr->DataIn(reinterpret_cast(cert.data()), cert.length(), > bro_analyzer()->GetAnalyzerTag(), bro_analyzer()->Conn(), ${rec.is_orig}); > file_mgr->EndOfFile(bro_analyzer()->GetAnalyzerTag(), bro_analyzer()->Conn(), ${rec.is_orig}); > > in exactly this order (so - directly following each other). Which does not work. It think it should work provided that matching file handles are generated at the script-layer for this type of file. (not sure whether they are in this case, didn?t check) > I also do not really think this is sufficiently documented in the comments of > Manager.h. This basically is not mentioned at all there? Yeah, it should probably at least link to [1] at least once. Do you think it would help to link to that in each method where it matters? [1] http://www.bro.org/development/howtos/file-analysis-file-id.html - Jon From jira at bro-tracker.atlassian.net Thu Mar 13 10:27:18 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 13 Mar 2014 12:27:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1152) BroControl version check In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1152: ------------------------------- Component/s: (was: Bro) BroControl > BroControl version check > ------------------------ > > Key: BIT-1152 > URL: https://bro-tracker.atlassian.net/browse/BIT-1152 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Reporter: Robin Sommer > Assignee: Daniel Thayer > Fix For: 2.3 > > > Show warning if version has been upgraded. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From bernhard at ICSI.Berkeley.EDU Thu Mar 13 10:54:38 2014 From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann) Date: Thu, 13 Mar 2014 10:54:38 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/bernhard/file-analysis-x509: Change x509 log - now certificates are only logged once per hour. (0d50b8b) In-Reply-To: <7D52BB42-31F4-441C-9775-F5F9BF36EE35@illinois.edu> References: <201403130717.s2D7HusM012132@bro-ids.icir.org> <20140313143037.GA56207@icir.org> <80CC8A1F-9392-4CEC-878F-79BFEA877829@icir.org> <7D52BB42-31F4-441C-9775-F5F9BF36EE35@illinois.edu> Message-ID: On Mar 13, 2014, at 8:50 AM, Siwek, Jonathan Luke wrote: >> file_mgr->DataIn(reinterpret_cast(cert.data()), cert.length(), >> bro_analyzer()->GetAnalyzerTag(), bro_analyzer()->Conn(), ${rec.is_orig}); >> file_mgr->EndOfFile(bro_analyzer()->GetAnalyzerTag(), bro_analyzer()->Conn(), ${rec.is_orig}); >> >> in exactly this order (so - directly following each other). Which does not work. > > It think it should work provided that matching file handles are generated at the script-layer for this type of file. (not sure whether they are in this case, didn?t check) Well, file handles are generated at the script layer. The current code is? function get_file_handle(c: connection, is_orig: bool): string { set_session(c); local depth: count; if ( is_orig ) { depth = c$ssl$client_depth; ++c$ssl$client_depth; } else { depth = c$ssl$server_depth; ++c$ssl$server_depth; } return cat(Analyzer::ANALYZER_SSL, c$start_time, is_orig, id_string(c$id), depth); } I have no idea if that is ?matching? or not - in any case, the above C code does not work in combination with that file handle generation. Thinking about it, it makes sense - the EndOfFile function probably calls get_file_handle again for that connection, gets a different file handle back (because the counters are increased), and hence removal will not work. But?. >> I also do not really think this is sufficiently documented in the comments of >> Manager.h. This basically is not mentioned at all there? > > Yeah, it should probably at least link to [1] at least once. Do you think it would help to link to that in each method where it matters? > > [1] http://www.bro.org/development/howtos/file-analysis-file-id.html ?even after reading through the how to, I was not quite clear on the fact, that get_file_handle has to always return the same value for the same file. Which is impossible to accomplish in cases like this, because, several certificates are sent over a connection, and you do not get any further information with the get_file_handle call. So - you more or less have to return differing IDs for everything. Perhaps the EndOfFile functions should get some warning that details in which circumstances you can use them (probably - static per-connection ID). Also - it might be nice to throw some kind of reporter warning when EndOfFile is called for a file ID that does not exists. If I understood everything right, that should never happen during normal operations, should it? Bernhard From jsiwek at illinois.edu Thu Mar 13 12:36:40 2014 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Thu, 13 Mar 2014 19:36:40 +0000 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/bernhard/file-analysis-x509: Change x509 log - now certificates are only logged once per hour. (0d50b8b) In-Reply-To: References: <201403130717.s2D7HusM012132@bro-ids.icir.org> <20140313143037.GA56207@icir.org> <80CC8A1F-9392-4CEC-878F-79BFEA877829@icir.org> <7D52BB42-31F4-441C-9775-F5F9BF36EE35@illinois.edu> Message-ID: <571EA7CB-D413-4E9F-88D6-A810E8304DC7@illinois.edu> On Mar 13, 2014, at 12:54 PM, Bernhard Amann wrote: > function get_file_handle(c: connection, is_orig: bool): string > { > set_session(c); > > local depth: count; > > if ( is_orig ) > { > depth = c$ssl$client_depth; > ++c$ssl$client_depth; > } > else > { > depth = c$ssl$server_depth; > ++c$ssl$server_depth; > } > > return cat(Analyzer::ANALYZER_SSL, c$start_time, is_orig, id_string(c$id), depth); > } > > I have no idea if that is ?matching? or not - in any case, the above C code does not work in combination with that > file handle generation. Thinking about it, it makes sense - the EndOfFile function probably calls get_file_handle again for > that connection, gets a different file handle back (because the counters are increased), and hence removal will not work. Yes, that looks like what would happen. > ?even after reading through the how to, I was not quite clear on the fact, that get_file_handle > has to always return the same value for the same file. Which is impossible to accomplish in cases > like this, because, several certificates are sent over a connection, and you do not get any further information > with the get_file_handle call. So - you more or less have to return differing IDs for everything. Instead of incrementing the depth in get_file_handle, it could increment it in a x509_certificate handler? That way the handle remains the same between the DataIn/EndOfFile pairs, but the next DataIn/EndOfFile will get a new file handle due to the incremented depth. Not suggesting you change it from using the pre-computed file id method you currently have, just challenging the impossibility claim :). And I do agree the whole script-layer generation of file handle string is difficult to understand/use, but I don?t have any ideas at the moment for helping that... > Perhaps the EndOfFile functions should get some warning that details in which circumstances you can use > them (probably - static per-connection ID). I don?t think there?s any limitation/circumstances unique to EndOfFile that differs from the other FAF APIs. > Also - it might be nice to throw some kind of reporter warning when EndOfFile is called for a file ID that > does not exists. > > If I understood everything right, that should never happen during normal operations, should it? It could be that the file timed out and was already cleaned up before a protocol analyzer called EndOfFile. I think it can also be the case that a protocol analyzer only ever calls EndOfFile because a connection got closed/reset before any file data was seen. - Jon From bernhard at ICSI.Berkeley.EDU Thu Mar 13 12:55:06 2014 From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann) Date: Thu, 13 Mar 2014 12:55:06 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/bernhard/file-analysis-x509: Change x509 log - now certificates are only logged once per hour. (0d50b8b) In-Reply-To: <571EA7CB-D413-4E9F-88D6-A810E8304DC7@illinois.edu> References: <201403130717.s2D7HusM012132@bro-ids.icir.org> <20140313143037.GA56207@icir.org> <80CC8A1F-9392-4CEC-878F-79BFEA877829@icir.org> <7D52BB42-31F4-441C-9775-F5F9BF36EE35@illinois.edu> <571EA7CB-D413-4E9F-88D6-A810E8304DC7@illinois.edu> Message-ID: <3FEB5418-9D33-4C05-ACDE-9FB1D1DB26A3@icsi.berkeley.edu> > >> ?even after reading through the how to, I was not quite clear on the fact, that get_file_handle >> has to always return the same value for the same file. Which is impossible to accomplish in cases >> like this, because, several certificates are sent over a connection, and you do not get any further information >> with the get_file_handle call. So - you more or less have to return differing IDs for everything. > > Instead of incrementing the depth in get_file_handle, it could increment it in a x509_certificate handler? That way the handle remains the same between the DataIn/EndOfFile pairs, but the next DataIn/EndOfFile will get a new file handle due to the incremented depth. > > Not suggesting you change it from using the pre-computed file id method you currently have, just challenging the impossibility claim :). And I do agree the whole script-layer generation of file handle string is difficult to understand/use, but I don?t have any ideas at the moment for helping that... That might work, yes? but.. If I do several DataIn -> EndOfFile -> DataIn -> EndOfFile calls in rapid succession (without anything in-between), as I am doing at the moment, is is guaranteed that the events that are raised by the FaF in an EndOfFile call are processed by scriptland before the (directly following) next DataIn call? As I understood things so far, the event queue cannot / will not be deleted in-between these calls. But I might be wrong. Bernhard From robin at icir.org Thu Mar 13 13:15:50 2014 From: robin at icir.org (Robin Sommer) Date: Thu, 13 Mar 2014 13:15:50 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/broctl] topic/dnthayer/broctl-fixes: Do not ping when checking if a host is alive (b71fc1d) In-Reply-To: <201403131827.s2DIR5bq022587@bro-ids.icir.org> References: <201403131827.s2DIR5bq022587@bro-ids.icir.org> Message-ID: <20140313201550.GE56207@icir.org> The ping was to quickly notice if a host is down, which iirc ssh wasn't always able to di (though I don't remember exactly what ssh did in those cases where it was a problem; too long ago). I'm wondering if it's worth keeping the ping check even it indeed means the fw needs to be configured accordingly. Any opinions? Robin On Thu, Mar 13, 2014 at 11:27 -0700, Daniel Thayer wrote: > Repository : ssh://git at bro-ids.icir.org/broctl > > On branch : topic/dnthayer/broctl-fixes > > >--------------------------------------------------------------- > > commit b71fc1d973ab9bba53b1d2fac6b36bf3aee042c4 > Author: Daniel Thayer > Date: Thu Mar 13 12:20:02 2014 -0500 > > Do not ping when checking if a host is alive > > Removed the ping from the host alive check because the ping > might be blocked by a firewall, and neither bro nor broctl needs > the ability to ping hosts. > > > >--------------------------------------------------------------- > > b71fc1d973ab9bba53b1d2fac6b36bf3aee042c4 > BroControl/execute.py | 4 ++-- > BroControl/plugin.py | 4 ++-- > CMakeLists.txt | 1 - > bin/is-alive | 24 ------------------------ > 4 files changed, 4 insertions(+), 29 deletions(-) > > diff --git a/BroControl/execute.py b/BroControl/execute.py > index 3b34d44..f667184 100644 > --- a/BroControl/execute.py > +++ b/BroControl/execute.py > @@ -189,14 +189,14 @@ def sync(nodes, paths): > # Keep track of hosts that are not alive. > _deadHosts = {} > > -# Return true if the given host is alive (i.e., we can ping it and establish > +# Return true if the given host is alive (i.e., we can establish > # an ssh session), and false otherwise. > def isAlive(host): > > if host in _deadHosts: > return False > > - (success, output) = runLocalCmd(os.path.join(config.Config.scriptsdir, "is-alive") + " " + util.scopeAddr(host)) > + (success, output) = runLocalCmd("ssh -o ConnectTimeout=30 %s true" % util.scopeAddr(host)) > > if not success: > _deadHosts[host] = True > diff --git a/BroControl/plugin.py b/BroControl/plugin.py > index cbfa135..ea06014 100644 > --- a/BroControl/plugin.py > +++ b/BroControl/plugin.py > @@ -341,8 +341,8 @@ class Plugin(object): > """Called when BroControl's ``cron`` command finds the availability of > a cluster system to have changed. Initially, all systems are assumed > to be up and running. Once BroControl notices that a system isn't > - responding (defined as either it doesn't ping at all, or does not > - accept SSH sessions), it calls this method, passing in a string with > + responding (defined as not accepting SSH sessions), it calls > + this method, passing in a string with > the name of the *host* and a boolean *status* set to False. Once the > host becomes available again, the method will be called again for the > same host with *status* now set to True. > diff --git a/CMakeLists.txt b/CMakeLists.txt > index 8a2ddf4..9a48847 100644 > --- a/CMakeLists.txt > +++ b/CMakeLists.txt > @@ -86,7 +86,6 @@ InstallShellScript(share/broctl/scripts bin/create-link-for-log) > InstallShellScript(share/broctl/scripts bin/delete-log) > InstallShellScript(share/broctl/scripts bin/expire-logs) > InstallShellScript(share/broctl/scripts bin/get-prof-log) > -InstallShellScript(share/broctl/scripts bin/is-alive) > InstallShellScript(share/broctl/scripts bin/local-interfaces) > InstallShellScript(share/broctl/scripts bin/make-archive-name) > InstallShellScript(share/broctl/scripts bin/post-terminate) > diff --git a/bin/is-alive b/bin/is-alive > deleted file mode 100755 > index 4137fc5..0000000 > --- a/bin/is-alive > +++ /dev/null > @@ -1,24 +0,0 @@ > -#! /usr/bin/env bash > -# > -# is-alive > - > -. `dirname $0`/broctl-config.sh > - > -if [ "${os}" == "linux" ]; then > - cmd='ping -q -c 1 -W 1' > - cmd6='ping6 -q -c 1 -W 1' > -elif [ "${os}" == "openbsd" -o "${os}" == "netbsd" ]; then > - cmd='ping -q -c 1 -w 1' > - cmd6='ping6 -q -c 1' > -else > - cmd='ping -q -t 1 -o' > - cmd6='ping6 -q -o' > -fi > - > -if [[ "$1" == *:* ]]; then > - cmd=$cmd6 > -fi > - > -$cmd $1 >/dev/null 2>&1 || exit 1 > - > -ssh -o ConnectTimeout=30 $1 true >/dev/null 2>&1 > > _______________________________________________ > bro-commits mailing list > bro-commits at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-commits > -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From jsiwek at illinois.edu Thu Mar 13 14:18:20 2014 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Thu, 13 Mar 2014 21:18:20 +0000 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/bernhard/file-analysis-x509: Change x509 log - now certificates are only logged once per hour. (0d50b8b) In-Reply-To: <3FEB5418-9D33-4C05-ACDE-9FB1D1DB26A3@icsi.berkeley.edu> References: <201403130717.s2D7HusM012132@bro-ids.icir.org> <20140313143037.GA56207@icir.org> <80CC8A1F-9392-4CEC-878F-79BFEA877829@icir.org> <7D52BB42-31F4-441C-9775-F5F9BF36EE35@illinois.edu> <571EA7CB-D413-4E9F-88D6-A810E8304DC7@illinois.edu> <3FEB5418-9D33-4C05-ACDE-9FB1D1DB26A3@icsi.berkeley.edu> Message-ID: <3F781EC8-EB60-4917-8A9A-C16198FB9523@illinois.edu> On Mar 13, 2014, at 2:55 PM, Bernhard Amann wrote: >> >>> ?even after reading through the how to, I was not quite clear on the fact, that get_file_handle >>> has to always return the same value for the same file. Which is impossible to accomplish in cases >>> like this, because, several certificates are sent over a connection, and you do not get any further information >>> with the get_file_handle call. So - you more or less have to return differing IDs for everything. >> >> Instead of incrementing the depth in get_file_handle, it could increment it in a x509_certificate handler? That way the handle remains the same between the DataIn/EndOfFile pairs, but the next DataIn/EndOfFile will get a new file handle due to the incremented depth. >> >> Not suggesting you change it from using the pre-computed file id method you currently have, just challenging the impossibility claim :). And I do agree the whole script-layer generation of file handle string is difficult to understand/use, but I don?t have any ideas at the moment for helping that... > > That might work, yes? but.. > > If I do several DataIn -> EndOfFile -> DataIn -> EndOfFile calls in rapid succession (without anything in-between), as I am doing at the moment, is is guaranteed that the events that are raised by the FaF in an EndOfFile call are processed by scriptland before the (directly following) next DataIn call? If you are using a pre-computed file-id, then no. If you aren?t, then yes, it flushes the event queue every time it needs to call out to the script-layer in order to generate a file-id. - Jon From jira at bro-tracker.atlassian.net Thu Mar 13 14:34:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 13 Mar 2014 16:34:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1134) DNS_Mgr::LookupAddr does not respect DNS_FAKE In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15725#comment-15725 ] Robin Sommer commented on BIT-1134: ----------------------------------- Merging. I'm suggesting a further change though: what do you think about making the fake values deterministic based on the queries? E.g., BuildFakeNameResult(ip) could return "fake_" and BuildFakeAddrResult(name) an address derived from hash(name). Advantage: now that the test-suite runs in DNS_FAKE mode, changes that affect DNS resolution wouldn't trigger diffs. > DNS_Mgr::LookupAddr does not respect DNS_FAKE > --------------------------------------------- > > Key: BIT-1134 > URL: https://bro-tracker.atlassian.net/browse/BIT-1134 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 13 14:59:18 2014 From: jira at bro-tracker.atlassian.net (Vern Paxson (JIRA)) Date: Thu, 13 Mar 2014 16:59:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1156) DNS analyzer parses TXT records imcompletely In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726#comment-15726 ] Vern Paxson commented on BIT-1156: ---------------------------------- Does "payload of DNS TXT records" mean that an individual TXT record can consist of multiple character strings? If so, and if the order is significant/preserved, then set[string] wouldn't be the right type. If instead this is referring to multiple TXT RRs, then likely set[string] is okay (but worth double-checking the RFC regarding the semantics for that case). > DNS analyzer parses TXT records imcompletely > -------------------------------------------- > > Key: BIT-1156 > URL: https://bro-tracker.atlassian.net/browse/BIT-1156 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Fix For: 2.3 > > > The payload of DNS TXT records can consist of multiple character strings but the DNS analyzer parses out only the first. We should parse them out all and then probably concatenate into a single string to pass to the event, separated with semicolons or something. > I have a trace with an example but it would need anonymization before inclusion into the test suite. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 13 15:05:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 13 Mar 2014 17:05:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1156) DNS analyzer parses TXT records imcompletely In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727#comment-15727 ] Robin Sommer commented on BIT-1156: ----------------------------------- Yes, this is what I meant; and right: it should be a vector, not a set. > DNS analyzer parses TXT records imcompletely > -------------------------------------------- > > Key: BIT-1156 > URL: https://bro-tracker.atlassian.net/browse/BIT-1156 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Fix For: 2.3 > > > The payload of DNS TXT records can consist of multiple character strings but the DNS analyzer parses out only the first. We should parse them out all and then probably concatenate into a single string to pass to the event, separated with semicolons or something. > I have a trace with an example but it would need anonymization before inclusion into the test suite. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From dnthayer at illinois.edu Thu Mar 13 14:58:04 2014 From: dnthayer at illinois.edu (Daniel Thayer) Date: Thu, 13 Mar 2014 16:58:04 -0500 Subject: [Bro-Dev] [Bro-Commits] [git/broctl] topic/dnthayer/broctl-fixes: Do not ping when checking if a host is alive (b71fc1d) In-Reply-To: <20140313201550.GE56207@icir.org> References: <201403131827.s2DIR5bq022587@bro-ids.icir.org> <20140313201550.GE56207@icir.org> Message-ID: <532229EC.5000607@illinois.edu> I believe Seth requested that we remove the ping. In my testing, I haven't noticed any problems without the ping. On 03/13/2014 03:15 PM, Robin Sommer wrote: > The ping was to quickly notice if a host is down, which iirc ssh > wasn't always able to di (though I don't remember exactly what ssh did > in those cases where it was a problem; too long ago). I'm wondering if > it's worth keeping the ping check even it indeed means the fw needs to > be configured accordingly. > > Any opinions? > > Robin > > On Thu, Mar 13, 2014 at 11:27 -0700, Daniel Thayer wrote: > >> Repository : ssh://git at bro-ids.icir.org/broctl >> >> On branch : topic/dnthayer/broctl-fixes >> >>> --------------------------------------------------------------- >> >> commit b71fc1d973ab9bba53b1d2fac6b36bf3aee042c4 >> Author: Daniel Thayer >> Date: Thu Mar 13 12:20:02 2014 -0500 >> >> Do not ping when checking if a host is alive >> >> Removed the ping from the host alive check because the ping >> might be blocked by a firewall, and neither bro nor broctl needs >> the ability to ping hosts. >> >> >>> --------------------------------------------------------------- >> >> b71fc1d973ab9bba53b1d2fac6b36bf3aee042c4 >> BroControl/execute.py | 4 ++-- >> BroControl/plugin.py | 4 ++-- >> CMakeLists.txt | 1 - >> bin/is-alive | 24 ------------------------ >> 4 files changed, 4 insertions(+), 29 deletions(-) >> >> diff --git a/BroControl/execute.py b/BroControl/execute.py >> index 3b34d44..f667184 100644 >> --- a/BroControl/execute.py >> +++ b/BroControl/execute.py >> @@ -189,14 +189,14 @@ def sync(nodes, paths): >> # Keep track of hosts that are not alive. >> _deadHosts = {} >> >> -# Return true if the given host is alive (i.e., we can ping it and establish >> +# Return true if the given host is alive (i.e., we can establish >> # an ssh session), and false otherwise. >> def isAlive(host): >> >> if host in _deadHosts: >> return False >> >> - (success, output) = runLocalCmd(os.path.join(config.Config.scriptsdir, "is-alive") + " " + util.scopeAddr(host)) >> + (success, output) = runLocalCmd("ssh -o ConnectTimeout=30 %s true" % util.scopeAddr(host)) >> >> if not success: >> _deadHosts[host] = True >> diff --git a/BroControl/plugin.py b/BroControl/plugin.py >> index cbfa135..ea06014 100644 >> --- a/BroControl/plugin.py >> +++ b/BroControl/plugin.py >> @@ -341,8 +341,8 @@ class Plugin(object): >> """Called when BroControl's ``cron`` command finds the availability of >> a cluster system to have changed. Initially, all systems are assumed >> to be up and running. Once BroControl notices that a system isn't >> - responding (defined as either it doesn't ping at all, or does not >> - accept SSH sessions), it calls this method, passing in a string with >> + responding (defined as not accepting SSH sessions), it calls >> + this method, passing in a string with >> the name of the *host* and a boolean *status* set to False. Once the >> host becomes available again, the method will be called again for the >> same host with *status* now set to True. >> diff --git a/CMakeLists.txt b/CMakeLists.txt >> index 8a2ddf4..9a48847 100644 >> --- a/CMakeLists.txt >> +++ b/CMakeLists.txt >> @@ -86,7 +86,6 @@ InstallShellScript(share/broctl/scripts bin/create-link-for-log) >> InstallShellScript(share/broctl/scripts bin/delete-log) >> InstallShellScript(share/broctl/scripts bin/expire-logs) >> InstallShellScript(share/broctl/scripts bin/get-prof-log) >> -InstallShellScript(share/broctl/scripts bin/is-alive) >> InstallShellScript(share/broctl/scripts bin/local-interfaces) >> InstallShellScript(share/broctl/scripts bin/make-archive-name) >> InstallShellScript(share/broctl/scripts bin/post-terminate) >> diff --git a/bin/is-alive b/bin/is-alive >> deleted file mode 100755 >> index 4137fc5..0000000 >> --- a/bin/is-alive >> +++ /dev/null >> @@ -1,24 +0,0 @@ >> -#! /usr/bin/env bash >> -# >> -# is-alive >> - >> -. `dirname $0`/broctl-config.sh >> - >> -if [ "${os}" == "linux" ]; then >> - cmd='ping -q -c 1 -W 1' >> - cmd6='ping6 -q -c 1 -W 1' >> -elif [ "${os}" == "openbsd" -o "${os}" == "netbsd" ]; then >> - cmd='ping -q -c 1 -w 1' >> - cmd6='ping6 -q -c 1' >> -else >> - cmd='ping -q -t 1 -o' >> - cmd6='ping6 -q -o' >> -fi >> - >> -if [[ "$1" == *:* ]]; then >> - cmd=$cmd6 >> -fi >> - >> -$cmd $1 >/dev/null 2>&1 || exit 1 >> - >> -ssh -o ConnectTimeout=30 $1 true >/dev/null 2>&1 >> >> _______________________________________________ >> bro-commits mailing list >> bro-commits at bro.org >> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-commits >> > > > From jira at bro-tracker.atlassian.net Thu Mar 13 17:03:19 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 13 Mar 2014 19:03:19 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1148) Bug in Connection::FlipRoles In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1148: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Bug in Connection::FlipRoles > ---------------------------- > > Key: BIT-1148 > URL: https://bro-tracker.atlassian.net/browse/BIT-1148 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Jon Siwek > Fix For: 2.3 > > > This method doesn't correctly swap address values. > Also, since scheduled analyzers for a connection are looked up based on the endpoint addresses, it's possible they will miss being attached to connections that end up taking the Connection::FlipRoles code path. An idea to fix would be just to have FlipRoles do a check for scheduled analyzers on the new connection tuple and attach any that turn up. > (These were reported by Kevin McMahon on the bro-dev list). -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 13 17:03:19 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 13 Mar 2014 19:03:19 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1154) Formatters restructed in: topic/seth/json-formatter In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15728#comment-15728 ] Robin Sommer commented on BIT-1154: ----------------------------------- Merged. The tests are a bit on the weak side though ... I'm leaving the ticket open hoping that we can add a few more: - More extensive tests of the JSON output (though I realize that's not quite trivial as the JSON output requires some normalization to diff larger log volumes) - I don't think there's any test of the new per-filter config options? Also, it might be nice to format the JSON a bit prettier, like with some more white space? > Formatters restructed in: topic/seth/json-formatter > --------------------------------------------------- > > Key: BIT-1154 > URL: https://bro-tracker.atlassian.net/browse/BIT-1154 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > Assignee: Robin Sommer > > topic/seth/json-formatter has an abstraction for Formatters and I created a formatters directory under threading. There is also a new JSON formatter and support in the Ascii and ElasticSearch writers for the JSON formatter. > I went ahead and threw in per-filter configuration options for the Ascii writer for all of the options that were exposed globally too. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 13 17:07:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 13 Mar 2014 19:07:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1154) Formatters restructed in: topic/seth/json-formatter In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729#comment-15729 ] Robin Sommer commented on BIT-1154: ----------------------------------- Also, regarding renaming: agree that the Ascii formatter is misnamed now. No great ideas for alternatives though, maybe one of "Log" or "Bro" or "Standard" or "Default"? > Formatters restructed in: topic/seth/json-formatter > --------------------------------------------------- > > Key: BIT-1154 > URL: https://bro-tracker.atlassian.net/browse/BIT-1154 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > Assignee: Robin Sommer > > topic/seth/json-formatter has an abstraction for Formatters and I created a formatters directory under threading. There is also a new JSON formatter and support in the Ascii and ElasticSearch writers for the JSON formatter. > I went ahead and threw in per-filter configuration options for the Ascii writer for all of the options that were exposed globally too. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 13 17:07:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 13 Mar 2014 19:07:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1154) Formatters restructed in: topic/seth/json-formatter In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1154: --------------------------------- Assignee: (was: Robin Sommer) > Formatters restructed in: topic/seth/json-formatter > --------------------------------------------------- > > Key: BIT-1154 > URL: https://bro-tracker.atlassian.net/browse/BIT-1154 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > > topic/seth/json-formatter has an abstraction for Formatters and I created a formatters directory under threading. There is also a new JSON formatter and support in the Ascii and ElasticSearch writers for the JSON formatter. > I went ahead and threw in per-filter configuration options for the Ascii writer for all of the options that were exposed globally too. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 13 17:07:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 13 Mar 2014 19:07:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1154) Formatters restructed in: topic/seth/json-formatter In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1154: ------------------------------ Status: Open (was: Merge Request) > Formatters restructed in: topic/seth/json-formatter > --------------------------------------------------- > > Key: BIT-1154 > URL: https://bro-tracker.atlassian.net/browse/BIT-1154 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > Assignee: Robin Sommer > > topic/seth/json-formatter has an abstraction for Formatters and I created a formatters directory under threading. There is also a new JSON formatter and support in the Ascii and ElasticSearch writers for the JSON formatter. > I went ahead and threw in per-filter configuration options for the Ascii writer for all of the options that were exposed globally too. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 13 17:11:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 13 Mar 2014 19:11:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1134) DNS_Mgr::LookupAddr does not respect DNS_FAKE In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1134: ------------------------------ Status: Open (was: Merge Request) > DNS_Mgr::LookupAddr does not respect DNS_FAKE > --------------------------------------------- > > Key: BIT-1134 > URL: https://bro-tracker.atlassian.net/browse/BIT-1134 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 13 17:36:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Thu, 13 Mar 2014 19:36:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1150: -------------------------------- Status: Merge Request (was: Open) > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Bernhard Amann > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 13 17:36:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Thu, 13 Mar 2014 19:36:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730#comment-15730 ] Bernhard Amann commented on BIT-1150: ------------------------------------- X509 file analyzer is in the topic/bernhard/file-analysis-x509 branch of bro and bro-testing. github diff link: https://github.com/bro/bro/compare/topic;bernhard;file-analysis-x509 > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Bernhard Amann > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 13 17:36:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Thu, 13 Mar 2014 19:36:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-953) SSL Analyzer: return the root CA used to validate a cert In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-953: ------------------------------- Status: Merge Request (was: Open) > SSL Analyzer: return the root CA used to validate a cert > -------------------------------------------------------- > > Key: BIT-953 > URL: https://bro-tracker.atlassian.net/browse/BIT-953 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: liamrandall > Assignee: Bernhard Amann > Priority: Low > Labels: Analyzer,, CA, Root,, SSL > Fix For: 2.4 > > > Since Bro will validate certs can we add a variable that says who the root CA was; would be useful for CA pinning, white listing or black listing. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 13 17:38:19 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Thu, 13 Mar 2014 19:38:19 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-760) Lift Server Alternative Name (SAN) field to scripting layer In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-760: ------------------------------- Status: Merge Request (was: Reopened) > Lift Server Alternative Name (SAN) field to scripting layer > ----------------------------------------------------------- > > Key: BIT-760 > URL: https://bro-tracker.atlassian.net/browse/BIT-760 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Matthias Vallentin > Assignee: Bernhard Amann > Labels: analyzer > Fix For: 2.4 > > > It would be nice to have the *Subject Alternative Name (SAN)* field of an X.509 certificate available at the scripting layer. It contains a list of domains that should be used in addition to the CN field of the subject to verify that a domain matches the certificate. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Fri Mar 14 00:00:16 2014 From: noreply at bro.org (Merge Tracker) Date: Fri, 14 Mar 2014 00:00:16 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403140700.s2E70GSB026217@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------------ -------------- ---------- ------------- ---------- ----------------------------------------------------------- BIT-1150 [1] Bro Robin Sommer Bernhard Amann 2014-03-13 2.3 Normal X509 updates BIT-1139 [2] Bro Robin Sommer Seth Hall 2014-03-11 2.3 Normal MHR lookups can cause significant CPU overhead in tests BIT-953 [3] Bro liamrandall Bernhard Amann 2014-03-13 2.4 Low SSL Analyzer: return the root CA used to validate a cert BIT-760 [4] Bro Matthias Vallentin Bernhard Amann 2014-03-13 2.4 Normal Lift Server Alternative Name (SAN) field to scripting layer [1] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [2] BIT-1139 https://bro-tracker.atlassian.net/browse/BIT-1139 [3] BIT-953 https://bro-tracker.atlassian.net/browse/BIT-953 [4] BIT-760 https://bro-tracker.atlassian.net/browse/BIT-760 From jira at bro-tracker.atlassian.net Fri Mar 14 07:25:18 2014 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Fri, 14 Mar 2014 09:25:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1157) optional fields are missing from JSON logs In-Reply-To: References: Message-ID: Justin Azoff created BIT-1157: --------------------------------- Summary: optional fields are missing from JSON logs Key: BIT-1157 URL: https://bro-tracker.atlassian.net/browse/BIT-1157 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: git/master Reporter: Justin Azoff Assignee: Seth Hall -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 07:27:18 2014 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Fri, 14 Mar 2014 09:27:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1157) optional fields are missing from JSON logs In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731#comment-15731 ] Justin Azoff commented on BIT-1157: ----------------------------------- For example, a DNS log entry that does not have an answer does not contain the 'answers' or 'TTLs' fields: {code} { "rejected": false, "Z": 1, "RA": false, "RD": false, "TC": false, "trans_id": 14902, "proto": "udp", "id.resp_p": 137, "id.resp_h": "192.168.2.8", "id.orig_p": 54887, "id.orig_h": "192.168.2.1", "uid": "CQwqq34KjPClu3aD38", "ts": 1394806566.399907, "query": "*\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000", "qclass": 1, "qclass_name": "C_INTERNET", "qtype": 33, "qtype_name": "NBSTAT", "rcode": 0, "rcode_name": "NOERROR", "AA": false } {code} I'd expect it to have {code} "answers": [], "TTLs": [], {code} but I suppose the above is correct two, just different from the .csv format which has to show something for that column. > optional fields are missing from JSON logs > ------------------------------------------ > > Key: BIT-1157 > URL: https://bro-tracker.atlassian.net/browse/BIT-1157 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Justin Azoff > Assignee: Seth Hall > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 08:34:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 14 Mar 2014 10:34:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1134) DNS_Mgr::LookupAddr does not respect DNS_FAKE In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732#comment-15732 ] Jon Siwek commented on BIT-1134: -------------------------------- Technically, I'm not sure it's actually an advantage in terms of less test suite diffs getting triggered because I think output will always be deterministic and based on the order in which lookups are executed (fake results are available immediately, so callbacks are always executed in order and given the same fake result value across runs). If fake result values are derived from the input, test baselines still change under the same circumstances -- when Bro scripts change the order in which lookup_* builtins are invoked. Or is there something else I'm not getting about it? Practically, having the fake results derived from the input does probably make output easier to follow for a human inspector. > DNS_Mgr::LookupAddr does not respect DNS_FAKE > --------------------------------------------- > > Key: BIT-1134 > URL: https://bro-tracker.atlassian.net/browse/BIT-1134 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 08:58:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 14 Mar 2014 10:58:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1155) bro.org sidebar breadcrumbs links often broken In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1155: --------------------------- Resolution: Won't Fix Status: Closed (was: Open) This just ended up being removed from the sidebar. > bro.org sidebar breadcrumbs links often broken > ---------------------------------------------- > > Key: BIT-1155 > URL: https://bro-tracker.atlassian.net/browse/BIT-1155 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Website > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Jon Siwek > > On pages like http://www.bro.org/bro-exchange-2013/exercises/faf.html > the sidebar will have links to http://www.bro.org/bro-exchange-2013/exercises and http://www.bro.org/bro-exchange-2013, which don't have any index and so generate an access error. > Need to see if those links can be auto-generated more intelligently. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 09:00:18 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Fri, 14 Mar 2014 11:00:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1157) optional fields are missing from JSON logs In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734#comment-15734 ] Seth Hall commented on BIT-1157: -------------------------------- Are those fields null or are they empty? I think if they aren't represented in the output then they're actually null. I could change the behavior to be if a field is null, have it actually write out "answers":null but I feel like that is just needlessly filling the output since not including the field has essentially the same effect with less junk being output. > optional fields are missing from JSON logs > ------------------------------------------ > > Key: BIT-1157 > URL: https://bro-tracker.atlassian.net/browse/BIT-1157 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Justin Azoff > Assignee: Seth Hall > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 09:02:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 14 Mar 2014 11:02:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1134) DNS_Mgr::LookupAddr does not respect DNS_FAKE In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735#comment-15735 ] Robin Sommer commented on BIT-1134: ----------------------------------- If just order changes, or say new lookups get introduced, the output wouldn't change if we derive the fake results directly from the queries. If the input changes to trigger different queries, we'd still get diffs but maybe they would now affect only a subset of the test run's output, not all subsequent lookups. In other words, I think this would reduce the likelihood and volume of diffs, but it can't fiully avoid them of course. {quote} Practically, having the fake results derived from the input does probably make output easier to follow for a human inspector. {quote} Ack. > DNS_Mgr::LookupAddr does not respect DNS_FAKE > --------------------------------------------- > > Key: BIT-1134 > URL: https://bro-tracker.atlassian.net/browse/BIT-1134 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 09:13:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 14 Mar 2014 11:13:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1134) DNS_Mgr::LookupAddr does not respect DNS_FAKE In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15736#comment-15736 ] Jon Siwek commented on BIT-1134: -------------------------------- {quote} we'd still get diffs but maybe they would now affect only a subset of the test run's output, not all subsequent lookups. In other words, I think this would reduce the likelihood and volume of diffs, but it can't fiully avoid them of course. {quote} Ah, I get it now and agree. > DNS_Mgr::LookupAddr does not respect DNS_FAKE > --------------------------------------------- > > Key: BIT-1134 > URL: https://bro-tracker.atlassian.net/browse/BIT-1134 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 10:05:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 14 Mar 2014 12:05:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1150: ------------------------------ Assignee: Seth Hall (was: Bernhard Amann) > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 10:05:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 14 Mar 2014 12:05:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15737#comment-15737 ] Robin Sommer commented on BIT-1150: ----------------------------------- I made a pass over the changes but focussed on src/. I've pushed my changes back into the same branch. Assigning to Seth for reviewing the script-level code. > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Bernhard Amann > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 10:15:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 14 Mar 2014 12:15:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15738#comment-15738 ] Robin Sommer commented on BIT-1150: ----------------------------------- One more for Bernhard: the code in {{file_analysis::X509::GetTimeFromAsn1()}} makes me nervous. :) I know we had that before but I can't tell if it's safe how it works on the char buffers. Any chance to convert that over to using std::string? > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 10:20:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 14 Mar 2014 12:20:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15739#comment-15739 ] Robin Sommer commented on BIT-1150: ----------------------------------- Now, here's what looks like a major problem: something in this branch must be quite expensive: compared to master, I'm getting these timing results: {code} [ 71%] tests.m57-short ... failed (+28.2%) [ 85%] tests.m57-long ... failed (+48.1%) 2 of 7 tests failed, 4 skipped make[1]: *** [btest-verbose] Error 1 [ 50%] tests.short ... failed (+5.4%) 1 of 4 tests failed, 2 skipped {code} (compiled in debug mode, but still). > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 10:26:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 14 Mar 2014 12:26:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1149) Check Coverity PIA message In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15740#comment-15740 ] Robin Sommer commented on BIT-1149: ----------------------------------- Fixed in 00755f1e40c7e > Check Coverity PIA message > -------------------------- > > Key: BIT-1149 > URL: https://bro-tracker.atlassian.net/browse/BIT-1149 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Robin Sommer > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 10:26:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 14 Mar 2014 12:26:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1149) Check Coverity PIA message In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1149: ------------------------------ Resolution: Fixed Status: Closed (was: Open) > Check Coverity PIA message > -------------------------- > > Key: BIT-1149 > URL: https://bro-tracker.atlassian.net/browse/BIT-1149 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Robin Sommer > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 11:04:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 14 Mar 2014 13:04:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-953) SSL Analyzer: return the root CA used to validate a cert In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-953: ----------------------------- Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) The merge is tracked by BIT-1150. Closing this one. > SSL Analyzer: return the root CA used to validate a cert > -------------------------------------------------------- > > Key: BIT-953 > URL: https://bro-tracker.atlassian.net/browse/BIT-953 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: liamrandall > Assignee: Bernhard Amann > Priority: Low > Labels: Analyzer,, CA, Root,, SSL > Fix For: 2.4 > > > Since Bro will validate certs can we add a variable that says who the root CA was; would be useful for CA pinning, white listing or black listing. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 11:06:20 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 14 Mar 2014 13:06:20 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15744#comment-15744 ] Robin Sommer commented on BIT-1150: ----------------------------------- Note, BIT-953 and BIT-760 have some additional information on functionality going into this change. Check for CHANGES and NEWS. > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 11:06:20 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 14 Mar 2014 13:06:20 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-760) Lift Server Alternative Name (SAN) field to scripting layer In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-760: ----------------------------- Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) The merge is tracked by BIT-1150. Closing this one. > Lift Server Alternative Name (SAN) field to scripting layer > ----------------------------------------------------------- > > Key: BIT-760 > URL: https://bro-tracker.atlassian.net/browse/BIT-760 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Matthias Vallentin > Assignee: Bernhard Amann > Labels: analyzer > Fix For: 2.4 > > > It would be nice to have the *Subject Alternative Name (SAN)* field of an X.509 certificate available at the scripting layer. It contains a list of domains that should be used in addition to the CN field of the subject to verify that a domain matches the certificate. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 11:06:20 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 14 Mar 2014 13:06:20 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743#comment-15743 ] Bernhard Amann commented on BIT-1150: ------------------------------------- Ok, I am really surprised by the increase in the timing results. The branch does not really do _that_ much more than the previous implementation. Basically, things were shifted to the file analysis framework, plus a few additional extensions are parsed (and events raised). Extension parsing should not really incur any significant CPU load (we basically already did that before, we just have a few more special cases). ...so... could it be possible that just handling stuff in the file analysis framework, instead of handling it inline in the ssl protocol parser incurs that overhead? I would look into it, but I am a bit at a loss on where to start with this... > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 11:12:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 14 Mar 2014 13:12:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1145) Individual set_seperator for different feeds In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745#comment-15745 ] Bernhard Amann commented on BIT-1145: ------------------------------------- This was merged with BIT-1154 > Individual set_seperator for different feeds > -------------------------------------------- > > Key: BIT-1145 > URL: https://bro-tracker.atlassian.net/browse/BIT-1145 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: aashish > Labels: feeds, framework, input, logging > Fix For: 2.4 > > > Can we assign an individual set_separator per feed ? > Why ?: > Various data feeds from different sources have their own fields separators. > We need to post process these feeds in order to digest the data into bro using input-framework, this creates a need to have two tiered storage for each of the data feeds (original data + re-formatted data for input framework). > At present the workaround is to basically format all data feeds to use intel-framework and this works very well. There is still useful needs to have data feeds outside intel-framework for example - digesting list of subnets+building allocations in the network or digesting auth data... and so on. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 11:12:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 14 Mar 2014 13:12:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1145) Individual set_seperator for different feeds In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1145: -------------------------------- Resolution: Fixed Status: Closed (was: Open) > Individual set_seperator for different feeds > -------------------------------------------- > > Key: BIT-1145 > URL: https://bro-tracker.atlassian.net/browse/BIT-1145 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: aashish > Labels: feeds, framework, input, logging > Fix For: 2.4 > > > Can we assign an individual set_separator per feed ? > Why ?: > Various data feeds from different sources have their own fields separators. > We need to post process these feeds in order to digest the data into bro using input-framework, this creates a need to have two tiered storage for each of the data feeds (original data + re-formatted data for input framework). > At present the workaround is to basically format all data feeds to use intel-framework and this works very well. There is still useful needs to have data feeds outside intel-framework for example - digesting list of subnets+building allocations in the network or digesting auth data... and so on. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 11:21:18 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Fri, 14 Mar 2014 13:21:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1158) topic/dnthayer/broctl-fixes In-Reply-To: References: Message-ID: Daniel Thayer created BIT-1158: ---------------------------------- Summary: topic/dnthayer/broctl-fixes Key: BIT-1158 URL: https://bro-tracker.atlassian.net/browse/BIT-1158 Project: Bro Issue Tracker Issue Type: Problem Components: BroControl Reporter: Daniel Thayer Fix For: 2.3 Branch topic/dnthayer/broctl-fixes contains a collection of small fixes and improvements to broctl. This fixes the memlimit broctl option and improves output of the "status", "check", and "start" commands. The rest of the changes in this branch improve the way broctl handles various error conditions. Here is a list of all changes: fix the memlimit broctl option, fix start command helper to report errors, prevent bad plugin from crashing broctl, show error message if a user tries to use lb_method without lb_procs, prevent infinite loop in start command, simplify some code that executes commands to avoid a potential broctl hang, add ssh options to avoid broctl hanging if a host is disconnected, flush list of dead hosts after each broctl command finishes (so that a user doesn't need to exit broctl), improved error handling of various broctl commands when a user forgets to first do a broctl install, change the way broctl writes dynamic state variables to disk to reduce the risk of losing important information, added a check if the bro version has changed since the last install (if so, warn the user to do another broctl install), removed ping from the host alive check (these are sometimes blocked by firewalls), improve column formatting for broctl status command, improve start/stop command output for crashed nodes, changed broctl check command output to be more specific. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 11:23:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 14 Mar 2014 13:23:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15746#comment-15746 ] Jon Siwek commented on BIT-1150: -------------------------------- I can help track the perf. difference. Is this merged in master now, or do I have to look at the branch? > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 11:23:18 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Fri, 14 Mar 2014 13:23:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1158) topic/dnthayer/broctl-fixes In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1158: ------------------------------- Status: Merge Request (was: Open) > topic/dnthayer/broctl-fixes > --------------------------- > > Key: BIT-1158 > URL: https://bro-tracker.atlassian.net/browse/BIT-1158 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Reporter: Daniel Thayer > Fix For: 2.3 > > > Branch topic/dnthayer/broctl-fixes contains a collection of small > fixes and improvements to broctl. This fixes the memlimit broctl > option and improves output of the "status", "check", and "start" > commands. The rest of the changes in this branch improve > the way broctl handles various error conditions. > Here is a list of all changes: > fix the memlimit broctl option, fix start command helper to report errors, > prevent bad plugin from crashing broctl, show error message if a > user tries to use lb_method without lb_procs, prevent infinite loop > in start command, simplify some code that executes commands to avoid > a potential broctl hang, add ssh options to avoid broctl hanging if > a host is disconnected, flush list of dead hosts after each broctl > command finishes (so that a user doesn't need to exit broctl), > improved error handling of various broctl commands when a user > forgets to first do a broctl install, change the way broctl > writes dynamic state variables to disk to reduce the risk of > losing important information, added a check if the bro version > has changed since the last install (if so, warn the user to do > another broctl install), removed ping from the host alive check > (these are sometimes blocked by firewalls), improve column formatting > for broctl status command, improve start/stop command output for > crashed nodes, changed broctl check command output to be more specific. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 11:25:18 2014 From: jira at bro-tracker.atlassian.net (Adam Slagell (JIRA)) Date: Fri, 14 Mar 2014 13:25:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1152) BroControl version check In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15747#comment-15747 ] Adam Slagell commented on BIT-1152: ----------------------------------- Isn't this fixed by the branch you have in for a merge request now? > BroControl version check > ------------------------ > > Key: BIT-1152 > URL: https://bro-tracker.atlassian.net/browse/BIT-1152 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Reporter: Robin Sommer > Assignee: Daniel Thayer > Fix For: 2.3 > > > Show warning if version has been upgraded. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 11:45:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 14 Mar 2014 13:45:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15748#comment-15748 ] Robin Sommer commented on BIT-1150: ----------------------------------- Not merged yet. > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 12:04:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 14 Mar 2014 14:04:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15749#comment-15749 ] Bernhard Amann commented on BIT-1150: ------------------------------------- Ok, after a bit of poking around... the increase in time is nearly completely caused by the frameworks/files/detect-MHR.bro script. When not loading it, the times for master and x509 are virtually equivalent for me. > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 12:24:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 14 Mar 2014 14:24:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15750#comment-15750 ] Bernhard Amann commented on BIT-1150: ------------------------------------- I don't quite get why that script is or should be causing this - but there seems to be some issue there. On my system, not loading it in m57-long reduces the total test time by about 75%... > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 12:29:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 14 Mar 2014 14:29:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15751#comment-15751 ] Jon Siwek commented on BIT-1150: -------------------------------- I see similar: {noformat} jsiwek at brotestbed:build (master) $ perf stat -e instructions bro -r ~/2009-M57-day11-18.trace Performance counter stats for 'bro -r /home/jsiwek/2009-M57-day11-18.trace': 61,627,649,626 instructions # 0.00 insns per cycle 21.226079244 seconds time elapsed jsiwek at brotestbed:build (topic/bernhard/file-analysis-x509) $ perf stat -e instructions bro -r ~/2009-M57-day11-18.trace Performance counter stats for 'bro -r /home/jsiwek/2009-M57-day11-18.trace': 61,971,043,004 instructions # 0.00 insns per cycle 20.798635594 seconds time elapsed $ perf stat -e instructions bro -r ~/2009-M57-day11-18.trace local WARNING: No Site::local_nets have been defined. It's usually a good idea to define your local networks. Performance counter stats for 'bro -r /home/jsiwek/2009-M57-day11-18.trace local': 360,108,745,169 instructions # 0.00 insns per cycle 90.165166702 seconds time elapsed jsiwek at brotestbed:build (topic/bernhard/file-analysis-x509) $ perf stat -e instructions bro -r ~/2009-M57-day11-18.trace local WARNING: No Site::local_nets have been defined. It's usually a good idea to define your local networks. Performance counter stats for 'bro -r /home/jsiwek/2009-M57-day11-18.trace local': 598,916,494,754 instructions # 0.00 insns per cycle 150.838914725 seconds time elapsed {noformat} If fields were added to either the connection or fa_file record, then it makes sense that the MHR script spends more time cloning values. Though the degree to which it increased is surprising. The faster-mhr branch should solve the immediate problem, but maybe there actually is something grossly inefficient about how certain Vals are cloned that deserves investigation. > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 12:56:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 14 Mar 2014 14:56:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752#comment-15752 ] Bernhard Amann commented on BIT-1150: ------------------------------------- Yes, merging your faster-mhr branch seems to fix the performance issue. :) Robin also told me to see how often the Trigger::EvaluatePending function is called in master vs. my branch. There is no significant difference (1907723 vs 1908617 calls). > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 13:16:18 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 14 Mar 2014 15:16:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753#comment-15753 ] Jon Siwek commented on BIT-1150: -------------------------------- {quote} Yes, merging your faster-mhr branch seems to fix the performance issue. {quote} Nice. I can follow up separately to check on why Val::Clone() is so expensive, but probably the perf. diff in this branch shouldn't block the merge. > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From robin at icir.org Fri Mar 14 13:24:20 2014 From: robin at icir.org (Robin Sommer) Date: Fri, 14 Mar 2014 13:24:20 -0700 Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: <20140314202420.GO6070@icir.org> On Fri, Mar 14, 2014 at 15:16 -0500, you wrote: > Nice. I can follow up separately to check on why Val::Clone() is so > expensive, but probably the perf. diff in this branch shouldn't block > the merge. Agreed. Seth, when you have reviewed the scripts, either go ahead with the merge, or assign the ticket back to me. From jira at bro-tracker.atlassian.net Fri Mar 14 13:26:18 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 14 Mar 2014 15:26:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754#comment-15754 ] Robin Sommer commented on BIT-1150: ----------------------------------- Agreed. Seth, when you have reviewed the scripts, either go ahead with the merge, or assign the ticket back to me. > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From seth at icir.org Fri Mar 14 13:30:24 2014 From: seth at icir.org (Seth Hall) Date: Fri, 14 Mar 2014 16:30:24 -0400 Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: <20140314202420.GO6070@icir.org> References: <20140314202420.GO6070@icir.org> Message-ID: <10EE8781-6647-4D7C-A638-03DE42250E24@icir.org> On Mar 14, 2014, at 4:24 PM, Robin Sommer wrote: > Agreed. Seth, when you have reviewed the scripts, either go ahead with > the merge, or assign the ticket back to me.  Ok. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140314/12c3c185/attachment.bin From jira at bro-tracker.atlassian.net Fri Mar 14 13:32:18 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Fri, 14 Mar 2014 15:32:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1150: --------------------------- Attachment: signature.asc  Ok. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 14:52:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 14 Mar 2014 16:52:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-911) SRV replies don't get processed by DNS analyzer In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-911: ------------------------------- Resolution: Fixed Status: Closed (was: Open) This was fixed in BIT-1147 > SRV replies don't get processed by DNS analyzer > ----------------------------------------------- > > Key: BIT-911 > URL: https://bro-tracker.atlassian.net/browse/BIT-911 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Vern Paxson > Fix For: 2.4 > > Attachments: tdns-srv.bug.trace > > > The event engine doesn't appear to generate {{dns_SRV_reply}} in some cases, as indicated by running on the attached trace. I've tried this with both the default DNS analysis and my own custom analysis (that uses \-b to not run other stuff) and have confirmed that the reply event isn't getting generated, even though there aren't any checksum issues or such. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 16:08:18 2014 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Fri, 14 Mar 2014 18:08:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1159) count/port comparisons silently fail when part of a record In-Reply-To: References: Message-ID: Justin Azoff created BIT-1159: --------------------------------- Summary: count/port comparisons silently fail when part of a record Key: BIT-1159 URL: https://bro-tracker.atlassian.net/browse/BIT-1159 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.2, git/master Reporter: Justin Azoff Priority: Low If you try to compare a count to a port directly, you get the following: {code} operands must be of the same type (1500/tcp < 2000) {code} but if you have a record, and mixup the types like so, it silently fails: {code} type PortRange: record { min: port &default=1/tcp; max: port &default=65535/tcp; }; global pr = PortRange($min=1000,$max=2000); #CORRECT: global pr = PortRange($min=1000/tcp,$max=2000/tcp); event bro_init() { print (pr$min <= 1500/tcp && 1500/tcp < pr$max) ? "OK" : "NOTOK"; } {code} {code} $ bro a.bro NOTOK {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 19:05:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 14 Mar 2014 21:05:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1159) count/port comparisons silently fail when part of a record In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757#comment-15757 ] Bernhard Amann commented on BIT-1159: ------------------------------------- To expand a bit on that - this way of constructing a record simply does not seem to do any kind of type checking, so the problem is not with count/port comparisons. {{{ type PortRange: record { min: port &default=1/tcp; max: port &default=65535/tcp; t: count; }; global pr = PortRange($min="a",$max=2000, $t=127.0.0.1); print pr; }}} works perfectly (and it should definitely not) > count/port comparisons silently fail when part of a record > ---------------------------------------------------------- > > Key: BIT-1159 > URL: https://bro-tracker.atlassian.net/browse/BIT-1159 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.2 > Reporter: Justin Azoff > Priority: Low > Labels: language > > If you try to compare a count to a port directly, you get the following: > {code} > operands must be of the same type (1500/tcp < 2000) > {code} > but if you have a record, and mixup the types like so, it silently fails: > {code} > type PortRange: record { > min: port &default=1/tcp; > max: port &default=65535/tcp; > }; > global pr = PortRange($min=1000,$max=2000); > #CORRECT: global pr = PortRange($min=1000/tcp,$max=2000/tcp); > event bro_init() > { > print (pr$min <= 1500/tcp && 1500/tcp < pr$max) ? "OK" : "NOTOK"; > } > {code} > {code} > $ bro a.bro > NOTOK > {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 14 19:05:18 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 14 Mar 2014 21:05:18 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1159) count/port comparisons silently fail when part of a record In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757#comment-15757 ] Bernhard Amann edited comment on BIT-1159 at 3/14/14 9:04 PM: -------------------------------------------------------------- To expand a bit on that - this way of constructing a record simply does not seem to do any kind of type checking, so the problem is not with count/port comparisons. {code} type PortRange: record { min: port &default=1/tcp; max: port &default=65535/tcp; t: count; }; global pr = PortRange($min="a",$max=2000, $t=127.0.0.1); print pr; {code} works perfectly (and it should definitely not) was (Author: amannb): To expand a bit on that - this way of constructing a record simply does not seem to do any kind of type checking, so the problem is not with count/port comparisons. {{{ type PortRange: record { min: port &default=1/tcp; max: port &default=65535/tcp; t: count; }; global pr = PortRange($min="a",$max=2000, $t=127.0.0.1); print pr; }}} works perfectly (and it should definitely not) > count/port comparisons silently fail when part of a record > ---------------------------------------------------------- > > Key: BIT-1159 > URL: https://bro-tracker.atlassian.net/browse/BIT-1159 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.2 > Reporter: Justin Azoff > Priority: Low > Labels: language > > If you try to compare a count to a port directly, you get the following: > {code} > operands must be of the same type (1500/tcp < 2000) > {code} > but if you have a record, and mixup the types like so, it silently fails: > {code} > type PortRange: record { > min: port &default=1/tcp; > max: port &default=65535/tcp; > }; > global pr = PortRange($min=1000,$max=2000); > #CORRECT: global pr = PortRange($min=1000/tcp,$max=2000/tcp); > event bro_init() > { > print (pr$min <= 1500/tcp && 1500/tcp < pr$max) ? "OK" : "NOTOK"; > } > {code} > {code} > $ bro a.bro > NOTOK > {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Sat Mar 15 00:00:19 2014 From: noreply at bro.org (Merge Tracker) Date: Sat, 15 Mar 2014 00:00:19 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403150700.s2F70JmX009585@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ---------- ---------- ------------- ---------- ------------------------------------------------------- BIT-1158 [1] BroControl Daniel Thayer - 2014-03-14 2.3 Normal topic/dnthayer/broctl-fixes [2] BIT-1150 [3] Bro Robin Sommer Seth Hall 2014-03-14 2.3 Normal X509 updates BIT-1139 [4] Bro Robin Sommer Seth Hall 2014-03-11 2.3 Normal MHR lookups can cause significant CPU overhead in tests [1] BIT-1158 https://bro-tracker.atlassian.net/browse/BIT-1158 [2] broctl-fixes https://github.com/bro/brocontrol/tree/topic/dnthayer/broctl-fixes [3] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [4] BIT-1139 https://bro-tracker.atlassian.net/browse/BIT-1139 From noreply at bro.org Sun Mar 16 00:00:15 2014 From: noreply at bro.org (Merge Tracker) Date: Sun, 16 Mar 2014 00:00:15 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403160700.s2G70FrI016412@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ---------- ---------- ------------- ---------- ------------------------------------------------------- BIT-1158 [1] BroControl Daniel Thayer - 2014-03-14 2.3 Normal topic/dnthayer/broctl-fixes [2] BIT-1150 [3] Bro Robin Sommer Seth Hall 2014-03-14 2.3 Normal X509 updates BIT-1139 [4] Bro Robin Sommer Seth Hall 2014-03-11 2.3 Normal MHR lookups can cause significant CPU overhead in tests [1] BIT-1158 https://bro-tracker.atlassian.net/browse/BIT-1158 [2] broctl-fixes https://github.com/bro/brocontrol/tree/topic/dnthayer/broctl-fixes [3] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [4] BIT-1139 https://bro-tracker.atlassian.net/browse/BIT-1139 From noreply at bro.org Mon Mar 17 00:00:15 2014 From: noreply at bro.org (Merge Tracker) Date: Mon, 17 Mar 2014 00:00:15 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403170700.s2H70FdT024428@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ---------- ---------- ------------- ---------- ------------------------------------------------------- BIT-1158 [1] BroControl Daniel Thayer - 2014-03-14 2.3 Normal topic/dnthayer/broctl-fixes [2] BIT-1150 [3] Bro Robin Sommer Seth Hall 2014-03-14 2.3 Normal X509 updates BIT-1139 [4] Bro Robin Sommer Seth Hall 2014-03-11 2.3 Normal MHR lookups can cause significant CPU overhead in tests Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- -------------- ---------- ----------------------------------------------------------- 636d25e [5] bro Bernhard Amann 2014-03-16 Fix compile errror on freebsd - defines have to be moved up [1] BIT-1158 https://bro-tracker.atlassian.net/browse/BIT-1158 [2] broctl-fixes https://github.com/bro/brocontrol/tree/topic/dnthayer/broctl-fixes [3] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [4] BIT-1139 https://bro-tracker.atlassian.net/browse/BIT-1139 [5] 636d25e https://github.com/bro/bro/commit/636d25e526d899bdea072a8831d57dfeaddac175 From jira at bro-tracker.atlassian.net Mon Mar 17 07:30:44 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Mon, 17 Mar 2014 09:30:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall reassigned BIT-1150: ------------------------------ Assignee: Bernhard Amann (was: Seth Hall) As we discussed, there is really still something I don't like about how the X509 analyzer is enabled. I agree that having all of those logs is pretty badly redundant in many cases but I think it's better to log them all by default in the base/ scripts. Unfortunately I don't have any suggestions on how to do what you're doing differently right now so I think we can go ahead with how it is. I'm assigning it back to you so you can commit a script in policy/ that suppresses the cert chains from being logged. > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Bernhard Amann > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 17 07:40:44 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Mon, 17 Mar 2014 09:40:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1150: -------------------------------- Status: Open (was: Merge Request) > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Bernhard Amann > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 17 08:59:44 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 17 Mar 2014 10:59:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1158) topic/dnthayer/broctl-fixes In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1158: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > topic/dnthayer/broctl-fixes > --------------------------- > > Key: BIT-1158 > URL: https://bro-tracker.atlassian.net/browse/BIT-1158 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Reporter: Daniel Thayer > Fix For: 2.3 > > > Branch topic/dnthayer/broctl-fixes contains a collection of small > fixes and improvements to broctl. This fixes the memlimit broctl > option and improves output of the "status", "check", and "start" > commands. The rest of the changes in this branch improve > the way broctl handles various error conditions. > Here is a list of all changes: > fix the memlimit broctl option, fix start command helper to report errors, > prevent bad plugin from crashing broctl, show error message if a > user tries to use lb_method without lb_procs, prevent infinite loop > in start command, simplify some code that executes commands to avoid > a potential broctl hang, add ssh options to avoid broctl hanging if > a host is disconnected, flush list of dead hosts after each broctl > command finishes (so that a user doesn't need to exit broctl), > improved error handling of various broctl commands when a user > forgets to first do a broctl install, change the way broctl > writes dynamic state variables to disk to reduce the risk of > losing important information, added a check if the bro version > has changed since the last install (if so, warn the user to do > another broctl install), removed ping from the host alive check > (these are sometimes blocked by firewalls), improve column formatting > for broctl status command, improve start/stop command output for > crashed nodes, changed broctl check command output to be more specific. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 17 09:23:46 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Mon, 17 Mar 2014 11:23:46 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1152) BroControl version check In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15801#comment-15801 ] Daniel Thayer commented on BIT-1152: ------------------------------------ This issue was fixed in branch topic/dnthayer/broctl-fixes (BIT-1158). > BroControl version check > ------------------------ > > Key: BIT-1152 > URL: https://bro-tracker.atlassian.net/browse/BIT-1152 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Reporter: Robin Sommer > Assignee: Daniel Thayer > Fix For: 2.3 > > > Show warning if version has been upgraded. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Tue Mar 18 00:00:13 2014 From: noreply at bro.org (Merge Tracker) Date: Tue, 18 Mar 2014 00:00:13 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403180700.s2I70De9022998@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ------------------------------------------------------- BIT-1139 [1] Bro Robin Sommer Seth Hall 2014-03-11 2.3 Normal MHR lookups can cause significant CPU overhead in tests [1] BIT-1139 https://bro-tracker.atlassian.net/browse/BIT-1139 From jira at bro-tracker.atlassian.net Tue Mar 18 10:56:44 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Tue, 18 Mar 2014 12:56:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1160) Update cluster documentation In-Reply-To: References: Message-ID: Bernhard Amann created BIT-1160: ----------------------------------- Summary: Update cluster documentation Key: BIT-1160 URL: https://bro-tracker.atlassian.net/browse/BIT-1160 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.3 We should update the Cluster documentation, if possible before releasing 2.3. I set up a Bro cluster for the first time yesterday - and when you look at the current state at the documentation it is not very useful... ...for example it contains things like (link to an example for the config) in the text. Furthermore it does not really mention how to actually configure Bro for a cluster, there is no mention of node.cfg, etc. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 18 10:58:44 2014 From: jira at bro-tracker.atlassian.net (Adam Slagell (JIRA)) Date: Tue, 18 Mar 2014 12:58:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1160) Update cluster documentation In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Slagell reassigned BIT-1160: --------------------------------- Assignee: Daniel Thayer > Update cluster documentation > ---------------------------- > > Key: BIT-1160 > URL: https://bro-tracker.atlassian.net/browse/BIT-1160 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Daniel Thayer > Labels: documentation > Fix For: 2.3 > > > We should update the Cluster documentation, if possible before releasing 2.3. > I set up a Bro cluster for the first time yesterday - and when you look at the current state at the documentation it is not very useful... > ...for example it contains things like (link to an example for the config) in the text. Furthermore it does not really mention how to actually configure Bro for a cluster, there is no mention of node.cfg, etc. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 18 10:58:44 2014 From: jira at bro-tracker.atlassian.net (Adam Slagell (JIRA)) Date: Tue, 18 Mar 2014 12:58:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1160) Update cluster documentation In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15802#comment-15802 ] Adam Slagell commented on BIT-1160: ----------------------------------- Daniel, can you update this? > Update cluster documentation > ---------------------------- > > Key: BIT-1160 > URL: https://bro-tracker.atlassian.net/browse/BIT-1160 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Labels: documentation > Fix For: 2.3 > > > We should update the Cluster documentation, if possible before releasing 2.3. > I set up a Bro cluster for the first time yesterday - and when you look at the current state at the documentation it is not very useful... > ...for example it contains things like (link to an example for the config) in the text. Furthermore it does not really mention how to actually configure Bro for a cluster, there is no mention of node.cfg, etc. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 18 11:32:45 2014 From: jira at bro-tracker.atlassian.net (Jeannette Dopheide (JIRA)) Date: Tue, 18 Mar 2014 13:32:45 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1160) Update cluster documentation In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15803#comment-15803 ] Jeannette Dopheide commented on BIT-1160: ----------------------------------------- Bernhard, When I was working with John Schipp to pick our "The More You Bro" topics we chose setting up a Bro cluster for this very reason. He wrote a blog post about setting up a cluster: http://sickbits.net/create-a-minimal-bro-cluster/ and we are planning on using this as the basis of our video. This blog might be useful to you until more documentation is written. > Update cluster documentation > ---------------------------- > > Key: BIT-1160 > URL: https://bro-tracker.atlassian.net/browse/BIT-1160 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Daniel Thayer > Labels: documentation > Fix For: 2.3 > > > We should update the Cluster documentation, if possible before releasing 2.3. > I set up a Bro cluster for the first time yesterday - and when you look at the current state at the documentation it is not very useful... > ...for example it contains things like (link to an example for the config) in the text. Furthermore it does not really mention how to actually configure Bro for a cluster, there is no mention of node.cfg, etc. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 18 11:50:44 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Tue, 18 Mar 2014 13:50:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1160) Update cluster documentation In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15804#comment-15804 ] Bernhard Amann commented on BIT-1160: ------------------------------------- Thanks :). I was able to set it up without too many problems - but I just think the current state of documentation might make it a tad difficult for people that are not as familiar with Bro. Having a step by step walkthough (like in the one in the blog post you liked) in the official documentation would probably be a good idea. > Update cluster documentation > ---------------------------- > > Key: BIT-1160 > URL: https://bro-tracker.atlassian.net/browse/BIT-1160 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Daniel Thayer > Labels: documentation > Fix For: 2.3 > > > We should update the Cluster documentation, if possible before releasing 2.3. > I set up a Bro cluster for the first time yesterday - and when you look at the current state at the documentation it is not very useful... > ...for example it contains things like (link to an example for the config) in the text. Furthermore it does not really mention how to actually configure Bro for a cluster, there is no mention of node.cfg, etc. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 18 12:42:44 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Tue, 18 Mar 2014 14:42:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1160) Update cluster documentation In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1160: --------------------------- Attachment: signature.asc Agreed, this is definitely something that we need to have official documentation on. Supplementing with videos and other documents couldn't hurt but we need the official docs first. > Update cluster documentation > ---------------------------- > > Key: BIT-1160 > URL: https://bro-tracker.atlassian.net/browse/BIT-1160 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Daniel Thayer > Labels: documentation > Fix For: 2.3 > > Attachments: signature.asc > > > We should update the Cluster documentation, if possible before releasing 2.3. > I set up a Bro cluster for the first time yesterday - and when you look at the current state at the documentation it is not very useful... > ...for example it contains things like (link to an example for the config) in the text. Furthermore it does not really mention how to actually configure Bro for a cluster, there is no mention of node.cfg, etc. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 18 14:14:44 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Tue, 18 Mar 2014 16:14:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1160) Update cluster documentation In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15806#comment-15806 ] Daniel Thayer commented on BIT-1160: ------------------------------------ I believe that the cluster doc that exists currently was not intended to provide details like which config files to edit, but rather to explain the components that make up the bro cluster architecture and provide a high-level discussion of related topics such as load balancing. It's the type of document someone should read before installing Bro, because it provides background information that could help to guess what hardware might be needed to deploy a Bro cluster, and which type of load-balancing to use. I think the title of the page is misleading (rather than "setting up a bro cluster", it should probably be something like "bro cluster architecture"). Similarly, it would make more sense if the document were moved up, between the introduction section and the installing bro section. Adding more links to other documents (for example, Justin wrote some useful documentation on configuring Bro with PF_RING) will help avoid duplicating information. The existing instructions on configuring a Bro cluster could be improved, as well. > Update cluster documentation > ---------------------------- > > Key: BIT-1160 > URL: https://bro-tracker.atlassian.net/browse/BIT-1160 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Daniel Thayer > Labels: documentation > Fix For: 2.3 > > Attachments: signature.asc > > > We should update the Cluster documentation, if possible before releasing 2.3. > I set up a Bro cluster for the first time yesterday - and when you look at the current state at the documentation it is not very useful... > ...for example it contains things like (link to an example for the config) in the text. Furthermore it does not really mention how to actually configure Bro for a cluster, there is no mention of node.cfg, etc. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 18 14:41:44 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Tue, 18 Mar 2014 16:41:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1160) Update cluster documentation In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15807#comment-15807 ] Bernhard Amann commented on BIT-1160: ------------------------------------- Do we actually have any documentation on how to configure a Bro cluster? I just might be really blind, but I drew a blank yesterday... > Update cluster documentation > ---------------------------- > > Key: BIT-1160 > URL: https://bro-tracker.atlassian.net/browse/BIT-1160 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Daniel Thayer > Labels: documentation > Fix For: 2.3 > > Attachments: signature.asc > > > We should update the Cluster documentation, if possible before releasing 2.3. > I set up a Bro cluster for the first time yesterday - and when you look at the current state at the documentation it is not very useful... > ...for example it contains things like (link to an example for the config) in the text. Furthermore it does not really mention how to actually configure Bro for a cluster, there is no mention of node.cfg, etc. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 18 14:53:44 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 18 Mar 2014 16:53:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1161) topic/jsiwek/faster-val-clone In-Reply-To: References: Message-ID: Jon Siwek created BIT-1161: ------------------------------ Summary: topic/jsiwek/faster-val-clone Key: BIT-1161 URL: https://bro-tracker.atlassian.net/browse/BIT-1161 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: Jon Siwek Fix For: 2.3 This branch makes it less expensive to serialize large/complex values (e.g. connection and/or fa_file records). The obvious overhead that could be reduced was from the fixed growth incrementation of the buffer used to contain serialized data. With records that expand out to ~1.6M (master) or ~3M (topic/bernhard/file-analysis-x509) in serialized form, it takes a bit too many allocations when trying to get there in growth increments of 64K. It may also help some to use realloc instead of new/memcpy/delete each time it needs to grow. I didn't find it helped much to increase the initial buffer size from 64K (and 90% of the things needing serialization fit in that size buffer anyway). It could possibly help to preallocate a buffer that gets re-used across serializations instead of repeatedly allocating small buffers that will need to be resized. I don't have a complete breakdown/view of the bytes that make up the serialized version of the large/complex records, but taking a quick look I note that the filenames from Location information of each BroObj/Val make up a third of ~1.6M (master). And that's the full path of each file, so this all will depend on where the Bro scripts reside on the file system (i.e. put them as close to the root dir as possible and you might increase performance!). Any other quick ideas of what can be done here? If not, improving the serialization seems to deserve its own project (which also might be part of the new comm. library project) for later. In the meantime, it's at least shown that avoiding situations where large/complex records are serialized can help (BIT-1139). And that might always be a useful optimization strategy if the serialized representation of Vals is going to scale not just as a function of their value, but also w/ their type/attribute/location information. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 18 14:53:44 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 18 Mar 2014 16:53:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1161) topic/jsiwek/faster-val-clone In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1161: --------------------------- Status: Merge Request (was: Open) > topic/jsiwek/faster-val-clone > ----------------------------- > > Key: BIT-1161 > URL: https://bro-tracker.atlassian.net/browse/BIT-1161 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch makes it less expensive to serialize large/complex values (e.g. connection and/or fa_file records). > The obvious overhead that could be reduced was from the fixed growth incrementation of the buffer used to contain serialized data. With records that expand out to ~1.6M (master) or ~3M (topic/bernhard/file-analysis-x509) in serialized form, it takes a bit too many allocations when trying to get there in growth increments of 64K. It may also help some to use realloc instead of new/memcpy/delete each time it needs to grow. > I didn't find it helped much to increase the initial buffer size from 64K (and 90% of the things needing serialization fit in that size buffer anyway). > It could possibly help to preallocate a buffer that gets re-used across serializations instead of repeatedly allocating small buffers that will need to be resized. > I don't have a complete breakdown/view of the bytes that make up the serialized version of the large/complex records, but taking a quick look I note that the filenames from Location information of each BroObj/Val make up a third of ~1.6M (master). And that's the full path of each file, so this all will depend on where the Bro scripts reside on the file system (i.e. put them as close to the root dir as possible and you might increase performance!). > Any other quick ideas of what can be done here? If not, improving the serialization seems to deserve its own project (which also might be part of the new comm. library project) for later. > In the meantime, it's at least shown that avoiding situations where large/complex records are serialized can help (BIT-1139). And that might always be a useful optimization strategy if the serialized representation of Vals is going to scale not just as a function of their value, but also w/ their type/attribute/location information. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 18 15:05:44 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 18 Mar 2014 17:05:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1161) topic/jsiwek/faster-val-clone In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15808#comment-15808 ] Jon Siwek commented on BIT-1161: -------------------------------- {quote} The obvious overhead that could be reduced was from the fixed growth incrementation of the buffer used to contain serialized data. With records that expand out to ~1.6M (master) or ~3M (topic/bernhard/file-analysis-x509) in serialized form, it takes a bit too many allocations when trying to get there in growth increments of 64K. It may also help some to use realloc instead of new/memcpy/delete each time it needs to grow. {quote} Note that the benefit of this optimization is more pronounced on Bernhard's branch. And I don't think doubling the size of the serialized data there is necessarily something wrong or needs to be fixed/changed. But it might be something to double-check whether some of the redefs of SSL::Info or Files::Info can be streamlined. > topic/jsiwek/faster-val-clone > ----------------------------- > > Key: BIT-1161 > URL: https://bro-tracker.atlassian.net/browse/BIT-1161 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch makes it less expensive to serialize large/complex values (e.g. connection and/or fa_file records). > The obvious overhead that could be reduced was from the fixed growth incrementation of the buffer used to contain serialized data. With records that expand out to ~1.6M (master) or ~3M (topic/bernhard/file-analysis-x509) in serialized form, it takes a bit too many allocations when trying to get there in growth increments of 64K. It may also help some to use realloc instead of new/memcpy/delete each time it needs to grow. > I didn't find it helped much to increase the initial buffer size from 64K (and 90% of the things needing serialization fit in that size buffer anyway). > It could possibly help to preallocate a buffer that gets re-used across serializations instead of repeatedly allocating small buffers that will need to be resized. > I don't have a complete breakdown/view of the bytes that make up the serialized version of the large/complex records, but taking a quick look I note that the filenames from Location information of each BroObj/Val make up a third of ~1.6M (master). And that's the full path of each file, so this all will depend on where the Bro scripts reside on the file system (i.e. put them as close to the root dir as possible and you might increase performance!). > Any other quick ideas of what can be done here? If not, improving the serialization seems to deserve its own project (which also might be part of the new comm. library project) for later. > In the meantime, it's at least shown that avoiding situations where large/complex records are serialized can help (BIT-1139). And that might always be a useful optimization strategy if the serialized representation of Vals is going to scale not just as a function of their value, but also w/ their type/attribute/location information. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Wed Mar 19 00:00:14 2014 From: noreply at bro.org (Merge Tracker) Date: Wed, 19 Mar 2014 00:00:14 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403190700.s2J70EFt010968@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ------------------------------------------------------- BIT-1161 [1] Bro Jon Siwek - 2014-03-18 2.3 Normal topic/jsiwek/faster-val-clone [2] BIT-1139 [3] Bro Robin Sommer Seth Hall 2014-03-11 2.3 Normal MHR lookups can cause significant CPU overhead in tests Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- --------- ---------- --------------------------------------- #4 [4] bro mareq [5] 2014-03-18 Protocol identification heuristics. [6] [1] BIT-1161 https://bro-tracker.atlassian.net/browse/BIT-1161 [2] faster-val-clone https://github.com/bro/bro/tree/topic/jsiwek/faster-val-clone [3] BIT-1139 https://bro-tracker.atlassian.net/browse/BIT-1139 [4] Pull Request #4 https://api.github.com/repos/bro/bro/issues/4 [5] mareq https://github.com/mareq [6] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request From jira at bro-tracker.atlassian.net Wed Mar 19 07:32:44 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 19 Mar 2014 09:32:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1159) count/port comparisons silently fail when part of a record In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1159: --------------------------- Assignee: Jon Siwek > count/port comparisons silently fail when part of a record > ---------------------------------------------------------- > > Key: BIT-1159 > URL: https://bro-tracker.atlassian.net/browse/BIT-1159 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Labels: language > > If you try to compare a count to a port directly, you get the following: > {code} > operands must be of the same type (1500/tcp < 2000) > {code} > but if you have a record, and mixup the types like so, it silently fails: > {code} > type PortRange: record { > min: port &default=1/tcp; > max: port &default=65535/tcp; > }; > global pr = PortRange($min=1000,$max=2000); > #CORRECT: global pr = PortRange($min=1000/tcp,$max=2000/tcp); > event bro_init() > { > print (pr$min <= 1500/tcp && 1500/tcp < pr$max) ? "OK" : "NOTOK"; > } > {code} > {code} > $ bro a.bro > NOTOK > {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Wed Mar 19 10:27:44 2014 From: jira at bro-tracker.atlassian.net (Marek Balint (JIRA)) Date: Wed, 19 Mar 2014 12:27:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (TM-16) Index not working when traffic encapsulated in 802.1q trunk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/TM-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15809#comment-15809 ] Marek Balint commented on TM-16: -------------------------------- Hi Tyler, honestly I do not see any direct connection between these two problems, but I do not know time-machine very well - just had the same problem as you did, found it, fixed it and the fix works for me. I have created pull request addressing your issue: https://github.com/bro/time-machine/pull/1 .mq. > Index not working when traffic encapsulated in 802.1q trunk > ----------------------------------------------------------- > > Key: TM-16 > URL: https://bro-tracker.atlassian.net/browse/TM-16 > Project: Time Machine > Issue Type: Problem > Affects Versions: git/master > Environment: Ubuntu 10.04 , pf_ring > Reporter: tyler.schoenke > Labels: 802.1Q, indexes > Attachments: tm-16.patch > > > Hi All, > When I query the time machine index, I am not receiving any results. > I just restarted time machine, and checked one of the recent class files to see there is traffic for a particular IP address. > tcpdump -e -v -n -r class_all_1385406639.023206 "vlan and host 128.138.44.198" > It shows some traffic, example: > 128.138.44.198.54014 > 74.125.225.209.443: Flags [.], cksum 0x8d2c (correct), seq 1283940799:1283940800, ack 615539104, win 16311, length 1 > 19:11:00.571731632 10:8c:cf:57:46:00 > 00:1d:09:6a:d9:a9, ethertype 802.1Q (0x8100), length 70: vlan 987, p 0, ethertype IPv4, (tos 0x0, ttl 56, id 17482, offset 0, flags [none], proto TCP (6), length 52) > When I telnet localhost 42042 and run the following command, I don't receive any results. > query to_file "128.138.44.198.pcap" index ip "128.138.44.198" > In the above tcpdump, you can see my traffic is 802.1Q trunked. I have to use the "vlan" BPF to extract it with tcpdump, and am wondering if the trunking is causing problems with indexing? > I tested the same version of time machine on non-trunked traffic, and the index works fine. > Let me know if you need any other configuration info. > Tyler -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Wed Mar 19 14:24:44 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Wed, 19 Mar 2014 16:24:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1139) MHR lookups can cause significant CPU overhead in tests In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1139: --------------------------- Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > MHR lookups can cause significant CPU overhead in tests > ------------------------------------------------------- > > Key: BIT-1139 > URL: https://bro-tracker.atlassian.net/browse/BIT-1139 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > > Live operation seems fine, need to understand what's going on. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Wed Mar 19 14:52:44 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Wed, 19 Mar 2014 16:52:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1162) Sumstat measurements stop working on clusters with single slow nodes In-Reply-To: References: Message-ID: Bernhard Amann created BIT-1162: ----------------------------------- Summary: Sumstat measurements stop working on clusters with single slow nodes Key: BIT-1162 URL: https://bro-tracker.atlassian.net/browse/BIT-1162 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.3 If you use sumstats on a Bro cluster and have one (or more) overloaded nodes, sumstats is near unusable at the moment. Sumstats asks for all keys in order. The speed of getting them seems to depend on the speed in which the individual cluster worker nodes answer. If there is a very slow node in the network, for sumstats with a higher number of keys, processing will just stop after just a few keys. No warning message or similar is output. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Wed Mar 19 15:07:44 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Wed, 19 Mar 2014 17:07:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1160) Update cluster documentation In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1160: ------------------------------- Component/s: BroControl > Update cluster documentation > ---------------------------- > > Key: BIT-1160 > URL: https://bro-tracker.atlassian.net/browse/BIT-1160 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro, BroControl > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Daniel Thayer > Labels: documentation > Fix For: 2.3 > > Attachments: signature.asc > > > We should update the Cluster documentation, if possible before releasing 2.3. > I set up a Bro cluster for the first time yesterday - and when you look at the current state at the documentation it is not very useful... > ...for example it contains things like (link to an example for the config) in the text. Furthermore it does not really mention how to actually configure Bro for a cluster, there is no mention of node.cfg, etc. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Wed Mar 19 15:07:44 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Wed, 19 Mar 2014 17:07:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1160) Update cluster documentation In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1160: ------------------------------- Status: Merge Request (was: Open) > Update cluster documentation > ---------------------------- > > Key: BIT-1160 > URL: https://bro-tracker.atlassian.net/browse/BIT-1160 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro, BroControl > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Daniel Thayer > Labels: documentation > Fix For: 2.3 > > Attachments: signature.asc > > > We should update the Cluster documentation, if possible before releasing 2.3. > I set up a Bro cluster for the first time yesterday - and when you look at the current state at the documentation it is not very useful... > ...for example it contains things like (link to an example for the config) in the text. Furthermore it does not really mention how to actually configure Bro for a cluster, there is no mention of node.cfg, etc. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Wed Mar 19 15:07:44 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Wed, 19 Mar 2014 17:07:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1160) Update cluster documentation In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15810#comment-15810 ] Daniel Thayer commented on BIT-1160: ------------------------------------ In branch topic/dnthayer/ticket1160 (bro and broctl repos), I've improved the documentation (and added links so that the content is easier to find). > Update cluster documentation > ---------------------------- > > Key: BIT-1160 > URL: https://bro-tracker.atlassian.net/browse/BIT-1160 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro, BroControl > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Daniel Thayer > Labels: documentation > Fix For: 2.3 > > Attachments: signature.asc > > > We should update the Cluster documentation, if possible before releasing 2.3. > I set up a Bro cluster for the first time yesterday - and when you look at the current state at the documentation it is not very useful... > ...for example it contains things like (link to an example for the config) in the text. Furthermore it does not really mention how to actually configure Bro for a cluster, there is no mention of node.cfg, etc. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Wed Mar 19 15:12:44 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 19 Mar 2014 17:12:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1159) count/port comparisons silently fail when part of a record In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15811#comment-15811 ] Jon Siwek commented on BIT-1159: -------------------------------- I'm playing around w/ enabling more consistent type checking and found something interesting: {code} diff --git a/scripts/base/protocols/conn/inactivity.bro b/scripts/base/protocols/conn/inactivity.bro index b383f1a..99233d3 100644 --- a/scripts/base/protocols/conn/inactivity.bro +++ b/scripts/base/protocols/conn/inactivity.bro @@ -8,12 +8,16 @@ export { ## the connection. const analyzer_inactivity_timeouts: table[Analyzer::Tag] of interval = { # For interactive services, allow longer periods of inactivity. - [[Analyzer::ANALYZER_SSH, Analyzer::ANALYZER_FTP]] = 1 hrs, + [Analyzer::ANALYZER_SSH] = 1 hrs, + [Analyzer::ANALYZER_FTP] = 1 hrs, } &redef; ## Define inactivity timeouts based on common protocol ports. const port_inactivity_timeouts: table[port] of interval = { - [[21/tcp, 22/tcp, 23/tcp, 513/tcp]] = 1 hrs, + [21/tcp] = 1 hrs, + [22/tcp] = 1 hrs, + [23/tcp] = 1 hrs, + [513/tcp] = 1 hrs, } &redef; } {code} Is the original code actually supposed to work? I kind of hope not... the container ctor/init code is complicated enough without a shorthand way of unrolling table/set elements based on a list of indices that are all supposed to yield the same value. {{FTP::cmd_reply_code}} does something similar. Can I assume my new type-checking code is catching incorrect initializations in these cases? > count/port comparisons silently fail when part of a record > ---------------------------------------------------------- > > Key: BIT-1159 > URL: https://bro-tracker.atlassian.net/browse/BIT-1159 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Labels: language > > If you try to compare a count to a port directly, you get the following: > {code} > operands must be of the same type (1500/tcp < 2000) > {code} > but if you have a record, and mixup the types like so, it silently fails: > {code} > type PortRange: record { > min: port &default=1/tcp; > max: port &default=65535/tcp; > }; > global pr = PortRange($min=1000,$max=2000); > #CORRECT: global pr = PortRange($min=1000/tcp,$max=2000/tcp); > event bro_init() > { > print (pr$min <= 1500/tcp && 1500/tcp < pr$max) ? "OK" : "NOTOK"; > } > {code} > {code} > $ bro a.bro > NOTOK > {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Wed Mar 19 21:59:44 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Wed, 19 Mar 2014 23:59:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1150: -------------------------------- Status: Merge Request (was: Open) > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Bernhard Amann > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Wed Mar 19 22:01:44 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Thu, 20 Mar 2014 00:01:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812#comment-15812 ] Bernhard Amann commented on BIT-1150: ------------------------------------- The repository now contains a policy script that prevents logging of all non-host certificates. scripts/policy/protocols/ssl/log-hostcerts-only.bro Seth, could you take a look if it is ok, and if yes merge the branch (or tell Robin to merge it...) > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Bernhard Amann > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Wed Mar 19 22:01:44 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Thu, 20 Mar 2014 00:01:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812#comment-15812 ] Bernhard Amann edited comment on BIT-1150 at 3/20/14 12:01 AM: --------------------------------------------------------------- The repository now contains a policy script that prevents logging of all non-host certificates. scripts/policy/protocols/ssl/log-hostcerts-only.bro The script is loaded by default in local.bro. Seth, could you take a look if it is ok, and if yes merge the branch (or tell Robin to merge it...) was (Author: amannb): The repository now contains a policy script that prevents logging of all non-host certificates. scripts/policy/protocols/ssl/log-hostcerts-only.bro Seth, could you take a look if it is ok, and if yes merge the branch (or tell Robin to merge it...) > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Bernhard Amann > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Wed Mar 19 22:01:44 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Thu, 20 Mar 2014 00:01:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1150: -------------------------------- Assignee: Seth Hall (was: Bernhard Amann) > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Thu Mar 20 00:00:11 2014 From: noreply at bro.org (Merge Tracker) Date: Thu, 20 Mar 2014 00:00:11 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403200700.s2K70B4a026182@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ -------------- -------------- ------------- ---------- ------------- ---------- --------------------------------- BIT-1161 [1] Bro Jon Siwek - 2014-03-18 2.3 Normal topic/jsiwek/faster-val-clone [2] BIT-1160 [3] Bro,BroControl Bernhard Amann Daniel Thayer 2014-03-19 2.3 Normal Update cluster documentation BIT-1150 [4] Bro Robin Sommer Seth Hall 2014-03-20 2.3 Normal X509 updates Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ --------- ---------- ------------------------------------------------- #4 [5] bro mareq [6] 2014-03-18 Protocol identification heuristics. [7] #1 [8] time-machine mareq [9] 2014-03-19 TM-16: Really skip VLAN header for indexing. [10] [1] BIT-1161 https://bro-tracker.atlassian.net/browse/BIT-1161 [2] faster-val-clone https://github.com/bro/bro/tree/topic/jsiwek/faster-val-clone [3] BIT-1160 https://bro-tracker.atlassian.net/browse/BIT-1160 [4] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [5] Pull Request #4 https://api.github.com/repos/bro/bro/issues/4 [6] mareq https://github.com/mareq [7] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [8] Pull Request #1 https://api.github.com/repos/bro/time-machine/issues/1 [9] mareq https://github.com/mareq [10] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From robin at icir.org Thu Mar 20 04:22:09 2014 From: robin at icir.org (Robin Sommer) Date: Thu, 20 Mar 2014 04:22:09 -0700 Subject: [Bro-Dev] [JIRA] (BIT-1159) count/port comparisons silently fail when part of a record In-Reply-To: References: Message-ID: <20140320112208.GC33300@icir.org> > Is the original code actually supposed to work? Yeah, I'm afraid it is ... That has indeed been a legitimate shortcut since Bro's early versions. However, I wouldn't veto removing it; it does indeed make some code parts quite a bit more complex, and I don't think it's a crucial feature to have and rarely used anyways. From jira at bro-tracker.atlassian.net Thu Mar 20 04:26:44 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 20 Mar 2014 06:26:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1159) count/port comparisons silently fail when part of a record In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813#comment-15813 ] Robin Sommer commented on BIT-1159: ----------------------------------- Yeah, I'm afraid it is ... That has indeed been a legitimate shortcut since Bro's early versions. However, I wouldn't veto removing it; it does indeed make some code parts quite a bit more complex, and I don't think it's a crucial feature to have and rarely used anyways. > count/port comparisons silently fail when part of a record > ---------------------------------------------------------- > > Key: BIT-1159 > URL: https://bro-tracker.atlassian.net/browse/BIT-1159 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Labels: language > > If you try to compare a count to a port directly, you get the following: > {code} > operands must be of the same type (1500/tcp < 2000) > {code} > but if you have a record, and mixup the types like so, it silently fails: > {code} > type PortRange: record { > min: port &default=1/tcp; > max: port &default=65535/tcp; > }; > global pr = PortRange($min=1000,$max=2000); > #CORRECT: global pr = PortRange($min=1000/tcp,$max=2000/tcp); > event bro_init() > { > print (pr$min <= 1500/tcp && 1500/tcp < pr$max) ? "OK" : "NOTOK"; > } > {code} > {code} > $ bro a.bro > NOTOK > {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From seth at icir.org Thu Mar 20 05:55:24 2014 From: seth at icir.org (Seth Hall) Date: Thu, 20 Mar 2014 08:55:24 -0400 Subject: [Bro-Dev] [JIRA] (BIT-1159) count/port comparisons silently fail when part of a record In-Reply-To: <20140320112208.GC33300@icir.org> References: <20140320112208.GC33300@icir.org> Message-ID: <99BBD9E6-1701-49B5-8E84-8E5C056CD713@icir.org> On Mar 20, 2014, at 7:22 AM, Robin Sommer wrote: >> Is the original code actually supposed to work? > > Yeah, I'm afraid it is ... That has indeed been a legitimate shortcut > since Bro's early versions. However, I wouldn't veto removing it; it > does indeed make some code parts quite a bit more complex, and I don't > think it's a crucial feature to have and rarely used anyways. I don?t actually like shorthand either and would be fine removing it. I always forget that it?s even possible to do that. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140320/f567ad89/attachment.bin From jira at bro-tracker.atlassian.net Thu Mar 20 05:57:44 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 20 Mar 2014 07:57:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1159) count/port comparisons silently fail when part of a record In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1159: --------------------------- Attachment: signature.asc I don?t actually like shorthand either and would be fine removing it. I always forget that it?s even possible to do that. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ > count/port comparisons silently fail when part of a record > ---------------------------------------------------------- > > Key: BIT-1159 > URL: https://bro-tracker.atlassian.net/browse/BIT-1159 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Labels: language > Attachments: signature.asc > > > If you try to compare a count to a port directly, you get the following: > {code} > operands must be of the same type (1500/tcp < 2000) > {code} > but if you have a record, and mixup the types like so, it silently fails: > {code} > type PortRange: record { > min: port &default=1/tcp; > max: port &default=65535/tcp; > }; > global pr = PortRange($min=1000,$max=2000); > #CORRECT: global pr = PortRange($min=1000/tcp,$max=2000/tcp); > event bro_init() > { > print (pr$min <= 1500/tcp && 1500/tcp < pr$max) ? "OK" : "NOTOK"; > } > {code} > {code} > $ bro a.bro > NOTOK > {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 20 06:26:44 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 20 Mar 2014 08:26:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1163) Logging framework text (ascii) writer writes sets as table[...] In-Reply-To: References: Message-ID: Seth Hall created BIT-1163: ------------------------------ Summary: Logging framework text (ascii) writer writes sets as table[...] Key: BIT-1163 URL: https://bro-tracker.atlassian.net/browse/BIT-1163 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.3 Reporter: Seth Hall Assignee: Bernhard Amann This is a minor point, but in our logs we should be writing set types as "set" and not table (even though internally they are the same thing). Here's an example header from files.log: {quote} #types time string table[addr] table[addr] table[string] string count table[string] string string interval bool bool count count count count bool string string string stringstring {quote} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 20 07:28:44 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 20 Mar 2014 09:28:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1159) type checking inconsistencies In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1159: --------------------------- Summary: type checking inconsistencies (was: count/port comparisons silently fail when part of a record) > type checking inconsistencies > ----------------------------- > > Key: BIT-1159 > URL: https://bro-tracker.atlassian.net/browse/BIT-1159 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Labels: language > Attachments: signature.asc > > > If you try to compare a count to a port directly, you get the following: > {code} > operands must be of the same type (1500/tcp < 2000) > {code} > but if you have a record, and mixup the types like so, it silently fails: > {code} > type PortRange: record { > min: port &default=1/tcp; > max: port &default=65535/tcp; > }; > global pr = PortRange($min=1000,$max=2000); > #CORRECT: global pr = PortRange($min=1000/tcp,$max=2000/tcp); > event bro_init() > { > print (pr$min <= 1500/tcp && 1500/tcp < pr$max) ? "OK" : "NOTOK"; > } > {code} > {code} > $ bro a.bro > NOTOK > {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 20 08:01:44 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 20 Mar 2014 10:01:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1159) type checking inconsistencies In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15815#comment-15815 ] Jon Siwek commented on BIT-1159: -------------------------------- Ok, I may end up removing support for the table/set ctor initialization shorthands. It's already inconsistent because it only works with "global" variable inits, not with "local" ones due to a difference in they way they go through type checks. Making globals do the same type checking as locals is what made me aware of the shorthand because then it doesn't pass the type check anymore. > type checking inconsistencies > ----------------------------- > > Key: BIT-1159 > URL: https://bro-tracker.atlassian.net/browse/BIT-1159 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Labels: language > Attachments: signature.asc > > > If you try to compare a count to a port directly, you get the following: > {code} > operands must be of the same type (1500/tcp < 2000) > {code} > but if you have a record, and mixup the types like so, it silently fails: > {code} > type PortRange: record { > min: port &default=1/tcp; > max: port &default=65535/tcp; > }; > global pr = PortRange($min=1000,$max=2000); > #CORRECT: global pr = PortRange($min=1000/tcp,$max=2000/tcp); > event bro_init() > { > print (pr$min <= 1500/tcp && 1500/tcp < pr$max) ? "OK" : "NOTOK"; > } > {code} > {code} > $ bro a.bro > NOTOK > {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 20 08:03:44 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 20 Mar 2014 10:03:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1159) type checking inconsistencies In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15815#comment-15815 ] Jon Siwek edited comment on BIT-1159 at 3/20/14 10:01 AM: ---------------------------------------------------------- Ok, I may end up removing support for the table/set ctor initialization shorthands. It's already inconsistent because it only works with "global" variable inits, not with "local" ones due to a difference in they way they go through type checks. Making globals do the same type checking as locals is what made me aware of the shorthand because then current usages don't pass the type check anymore. was (Author: jsiwek): Ok, I may end up removing support for the table/set ctor initialization shorthands. It's already inconsistent because it only works with "global" variable inits, not with "local" ones due to a difference in they way they go through type checks. Making globals do the same type checking as locals is what made me aware of the shorthand because then it doesn't pass the type check anymore. > type checking inconsistencies > ----------------------------- > > Key: BIT-1159 > URL: https://bro-tracker.atlassian.net/browse/BIT-1159 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Labels: language > Attachments: signature.asc > > > If you try to compare a count to a port directly, you get the following: > {code} > operands must be of the same type (1500/tcp < 2000) > {code} > but if you have a record, and mixup the types like so, it silently fails: > {code} > type PortRange: record { > min: port &default=1/tcp; > max: port &default=65535/tcp; > }; > global pr = PortRange($min=1000,$max=2000); > #CORRECT: global pr = PortRange($min=1000/tcp,$max=2000/tcp); > event bro_init() > { > print (pr$min <= 1500/tcp && 1500/tcp < pr$max) ? "OK" : "NOTOK"; > } > {code} > {code} > $ bro a.bro > NOTOK > {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 20 11:25:44 2014 From: jira at bro-tracker.atlassian.net (Vern Paxson (JIRA)) Date: Thu, 20 Mar 2014 13:25:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1159) type checking inconsistencies In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15816#comment-15816 ] Vern Paxson commented on BIT-1159: ---------------------------------- I'm okay with removing it, too. FYI, I don't regret having it there initially, though. Back when Bro's analysis was heavier on firewall-type rules, this was a good way to keep rules concise and thus less likely to be buggy. (And I can imagine introducing a cleaner language construct to still provide this sort of compactness in the future ?. if we find plausible use cases for modern scripts.) Also, where are things today with supporting constructs like {code} global x: table[addr] of string = { [badguy.foo.com] = "uh-oh" }; {code} where {{badguy.foo.com}} might resolve to multiple addresses? That was another reason why Bro has support for list-based expansion in initializers. > type checking inconsistencies > ----------------------------- > > Key: BIT-1159 > URL: https://bro-tracker.atlassian.net/browse/BIT-1159 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Labels: language > Attachments: signature.asc > > > If you try to compare a count to a port directly, you get the following: > {code} > operands must be of the same type (1500/tcp < 2000) > {code} > but if you have a record, and mixup the types like so, it silently fails: > {code} > type PortRange: record { > min: port &default=1/tcp; > max: port &default=65535/tcp; > }; > global pr = PortRange($min=1000,$max=2000); > #CORRECT: global pr = PortRange($min=1000/tcp,$max=2000/tcp); > event bro_init() > { > print (pr$min <= 1500/tcp && 1500/tcp < pr$max) ? "OK" : "NOTOK"; > } > {code} > {code} > $ bro a.bro > NOTOK > {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 20 12:05:44 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 20 Mar 2014 14:05:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1159) type checking inconsistencies In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1159: --------------------------- Attachment: signature.asc I actually avoid the parse time name resolution and tend toward runtime lookups with the dns bifs going on the assumption that Bro is an extremely long lived process and I?d rather not rely on a name lookup that happened a (potentially) long time ago. > type checking inconsistencies > ----------------------------- > > Key: BIT-1159 > URL: https://bro-tracker.atlassian.net/browse/BIT-1159 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Labels: language > Attachments: signature.asc, signature.asc > > > If you try to compare a count to a port directly, you get the following: > {code} > operands must be of the same type (1500/tcp < 2000) > {code} > but if you have a record, and mixup the types like so, it silently fails: > {code} > type PortRange: record { > min: port &default=1/tcp; > max: port &default=65535/tcp; > }; > global pr = PortRange($min=1000,$max=2000); > #CORRECT: global pr = PortRange($min=1000/tcp,$max=2000/tcp); > event bro_init() > { > print (pr$min <= 1500/tcp && 1500/tcp < pr$max) ? "OK" : "NOTOK"; > } > {code} > {code} > $ bro a.bro > NOTOK > {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 20 12:18:44 2014 From: jira at bro-tracker.atlassian.net (Vern Paxson (JIRA)) Date: Thu, 20 Mar 2014 14:18:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1159) type checking inconsistencies In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15818#comment-15818 ] Vern Paxson commented on BIT-1159: ---------------------------------- Runtime as a general style for this sounds fine, but I'd still like to know whether it works at compile-time like it used to. It's also not clear to me that at run-time one will always know when it's necessary to look up a name (for which there may be zillions) on the off chance that it has changed. > type checking inconsistencies > ----------------------------- > > Key: BIT-1159 > URL: https://bro-tracker.atlassian.net/browse/BIT-1159 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Labels: language > Attachments: signature.asc, signature.asc > > > If you try to compare a count to a port directly, you get the following: > {code} > operands must be of the same type (1500/tcp < 2000) > {code} > but if you have a record, and mixup the types like so, it silently fails: > {code} > type PortRange: record { > min: port &default=1/tcp; > max: port &default=65535/tcp; > }; > global pr = PortRange($min=1000,$max=2000); > #CORRECT: global pr = PortRange($min=1000/tcp,$max=2000/tcp); > event bro_init() > { > print (pr$min <= 1500/tcp && 1500/tcp < pr$max) ? "OK" : "NOTOK"; > } > {code} > {code} > $ bro a.bro > NOTOK > {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 20 12:31:49 2014 From: jira at bro-tracker.atlassian.net (Adam Slagell (JIRA)) Date: Thu, 20 Mar 2014 14:31:49 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1164) Memory Allocation bug in cq.c In-Reply-To: References: Message-ID: Adam Slagell created BIT-1164: --------------------------------- Summary: Memory Allocation bug in cq.c Key: BIT-1164 URL: https://bro-tracker.atlassian.net/browse/BIT-1164 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.3 Reporter: Adam Slagell Assignee: Jon Siwek Priority: High Line 107 in cq.c Seems problematic to use a pointer after it is freed. Extra credit if you know how I found this. if (cq_resize(hp, 0) < 0) { free(hp); memory_allocation -= sizeof(*hp); return (NULL); } -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 20 12:33:44 2014 From: jira at bro-tracker.atlassian.net (Adam Slagell (JIRA)) Date: Thu, 20 Mar 2014 14:33:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1164) Memory Allocation bug in cq.c In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Slagell updated BIT-1164: ------------------------------ Affects Version/s: git/master 2.1 2.2 > Memory Allocation bug in cq.c > ----------------------------- > > Key: BIT-1164 > URL: https://bro-tracker.atlassian.net/browse/BIT-1164 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.1, 2.2 > Reporter: Adam Slagell > Assignee: Jon Siwek > Priority: High > Fix For: 2.3 > > > Line 107 in cq.c > Seems problematic to use a pointer after it is freed. Extra credit if you know how I found this. > if (cq_resize(hp, 0) < 0) { > free(hp); > memory_allocation -= sizeof(*hp); > return (NULL); > } -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 20 12:33:44 2014 From: jira at bro-tracker.atlassian.net (Adam Slagell (JIRA)) Date: Thu, 20 Mar 2014 14:33:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1164) Memory Allocation bug in cq.c In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Slagell updated BIT-1164: ------------------------------ Affects Version/s: (was: 2.3) > Memory Allocation bug in cq.c > ----------------------------- > > Key: BIT-1164 > URL: https://bro-tracker.atlassian.net/browse/BIT-1164 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.1, 2.2 > Reporter: Adam Slagell > Assignee: Jon Siwek > Priority: High > Fix For: 2.3 > > > Line 107 in cq.c > Seems problematic to use a pointer after it is freed. Extra credit if you know how I found this. > if (cq_resize(hp, 0) < 0) { > free(hp); > memory_allocation -= sizeof(*hp); > return (NULL); > } -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 20 12:33:44 2014 From: jira at bro-tracker.atlassian.net (Adam Slagell (JIRA)) Date: Thu, 20 Mar 2014 14:33:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1164) Memory Allocation bug in cq.c In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Slagell updated BIT-1164: ------------------------------ Fix Version/s: 2.3 > Memory Allocation bug in cq.c > ----------------------------- > > Key: BIT-1164 > URL: https://bro-tracker.atlassian.net/browse/BIT-1164 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.1, 2.2 > Reporter: Adam Slagell > Assignee: Jon Siwek > Priority: High > Fix For: 2.3 > > > Line 107 in cq.c > Seems problematic to use a pointer after it is freed. Extra credit if you know how I found this. > if (cq_resize(hp, 0) < 0) { > free(hp); > memory_allocation -= sizeof(*hp); > return (NULL); > } -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 20 13:08:44 2014 From: jira at bro-tracker.atlassian.net (Adam Slagell (JIRA)) Date: Thu, 20 Mar 2014 15:08:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1164) Memory Allocation bug in cq.c In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Slagell updated BIT-1164: ------------------------------ Resolution: Invalid Status: Closed (was: Open) > Memory Allocation bug in cq.c > ----------------------------- > > Key: BIT-1164 > URL: https://bro-tracker.atlassian.net/browse/BIT-1164 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.1, 2.2 > Reporter: Adam Slagell > Assignee: Jon Siwek > Priority: High > Fix For: 2.3 > > > Line 107 in cq.c > Seems problematic to use a pointer after it is freed. Extra credit if you know how I found this. > if (cq_resize(hp, 0) < 0) { > free(hp); > memory_allocation -= sizeof(*hp); > return (NULL); > } -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 20 14:01:44 2014 From: jira at bro-tracker.atlassian.net (Adam Slagell (JIRA)) Date: Thu, 20 Mar 2014 16:01:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1152) BroControl version check In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Slagell updated BIT-1152: ------------------------------ Resolution: Fixed Status: Closed (was: Open) > BroControl version check > ------------------------ > > Key: BIT-1152 > URL: https://bro-tracker.atlassian.net/browse/BIT-1152 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Reporter: Robin Sommer > Assignee: Daniel Thayer > Fix For: 2.3 > > > Show warning if version has been upgraded. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 20 14:27:44 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 20 Mar 2014 16:27:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1159) type checking inconsistencies In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15819#comment-15819 ] Jon Siwek commented on BIT-1159: -------------------------------- {quote} Also, where are things today with supporting constructs like {code} global x: table[addr] of string = { [badguy.foo.com] = "uh-oh" }; {code} where {{badguy.foo.com}} might resolve to multiple addresses? That was another reason why Bro has support for list-based expansion in initializers. {quote} It should still work -- the lookup happens while parsing. And yeah, it still relies on the list expansion in the initializer to work, but I'm probably going to try and pass on removing the list-based expansion stuff at least as part of this ticket. > type checking inconsistencies > ----------------------------- > > Key: BIT-1159 > URL: https://bro-tracker.atlassian.net/browse/BIT-1159 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Labels: language > Attachments: signature.asc, signature.asc > > > If you try to compare a count to a port directly, you get the following: > {code} > operands must be of the same type (1500/tcp < 2000) > {code} > but if you have a record, and mixup the types like so, it silently fails: > {code} > type PortRange: record { > min: port &default=1/tcp; > max: port &default=65535/tcp; > }; > global pr = PortRange($min=1000,$max=2000); > #CORRECT: global pr = PortRange($min=1000/tcp,$max=2000/tcp); > event bro_init() > { > print (pr$min <= 1500/tcp && 1500/tcp < pr$max) ? "OK" : "NOTOK"; > } > {code} > {code} > $ bro a.bro > NOTOK > {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 20 17:41:44 2014 From: jira at bro-tracker.atlassian.net (Paolo Galtieri (JIRA)) Date: Thu, 20 Mar 2014 19:41:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1165) [Bro] cron: error running update-stats In-Reply-To: References: Message-ID: Paolo Galtieri created BIT-1165: ----------------------------------- Summary: [Bro] cron: error running update-stats Key: BIT-1165 URL: https://bro-tracker.atlassian.net/browse/BIT-1165 Project: Bro Issue Tracker Issue Type: Problem Components: BroControl Affects Versions: 2.2 Environment: Linux Fedora 19 on x86_64 Reporter: Paolo Galtieri Attachments: broctl.cfg I have configured bro so that all log files and spool files are saved to an external hard drive. I have modified the broctl.cfg file to point to the new locations. However, I continue to get lots of email about the following problem: error running update-stats ['cat: /usr/local/bro/spool/stats.log: No such file or directory'] How do I stop this email? The stats.log file is located in /media/NSM/NSM-SENSOR-2/logs/bro/logs/stats ls -l /media/NSM/NSM-SENSOR-2/logs/bro/logs/stats total 13456 -rw-r--r--. 1 root root 238 Mar 20 17:35 meta.dat -rw-r--r--. 1 root root 13762847 Mar 18 22:50 stats.log drwxr-xr-x. 2 root root 4096 Dec 5 13:25 www Paolo -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Thu Mar 20 20:37:44 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 20 Mar 2014 22:37:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1165) [Bro] cron: error running update-stats In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15820#comment-15820 ] Daniel Thayer commented on BIT-1165: ------------------------------------ Whenever you change anything in the broctl configuration, you need to run "broctl install". > [Bro] cron: error running update-stats > -------------------------------------- > > Key: BIT-1165 > URL: https://bro-tracker.atlassian.net/browse/BIT-1165 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: 2.2 > Environment: Linux Fedora 19 on x86_64 > Reporter: Paolo Galtieri > Labels: logging > Attachments: broctl.cfg > > > I have configured bro so that all log files and spool files are saved to an external hard drive. I have modified the broctl.cfg file to point to the new locations. However, I continue to get lots of email about the following problem: > error running update-stats > ['cat: /usr/local/bro/spool/stats.log: No such file or directory'] > How do I stop this email? > The stats.log file is located in > /media/NSM/NSM-SENSOR-2/logs/bro/logs/stats > ls -l /media/NSM/NSM-SENSOR-2/logs/bro/logs/stats > total 13456 > -rw-r--r--. 1 root root 238 Mar 20 17:35 meta.dat > -rw-r--r--. 1 root root 13762847 Mar 18 22:50 stats.log > drwxr-xr-x. 2 root root 4096 Dec 5 13:25 www > Paolo -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Fri Mar 21 00:00:13 2014 From: noreply at bro.org (Merge Tracker) Date: Fri, 21 Mar 2014 00:00:13 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403210700.s2L70Dw2009354@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ -------------- -------------- ------------- ---------- ------------- ---------- --------------------------------- BIT-1161 [1] Bro Jon Siwek - 2014-03-18 2.3 Normal topic/jsiwek/faster-val-clone [2] BIT-1160 [3] Bro,BroControl Bernhard Amann Daniel Thayer 2014-03-19 2.3 Normal Update cluster documentation BIT-1150 [4] Bro Robin Sommer Seth Hall 2014-03-20 2.3 Normal X509 updates Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ --------- ---------- ------------------------------------------------- #4 [5] bro mareq [6] 2014-03-18 Protocol identification heuristics. [7] #1 [8] time-machine mareq [9] 2014-03-19 TM-16: Really skip VLAN header for indexing. [10] [1] BIT-1161 https://bro-tracker.atlassian.net/browse/BIT-1161 [2] faster-val-clone https://github.com/bro/bro/tree/topic/jsiwek/faster-val-clone [3] BIT-1160 https://bro-tracker.atlassian.net/browse/BIT-1160 [4] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [5] Pull Request #4 https://api.github.com/repos/bro/bro/issues/4 [6] mareq https://github.com/mareq [7] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [8] Pull Request #1 https://api.github.com/repos/bro/time-machine/issues/1 [9] mareq https://github.com/mareq [10] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Fri Mar 21 08:04:44 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 21 Mar 2014 10:04:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1159) type checking inconsistencies In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15821#comment-15821 ] Jon Siwek commented on BIT-1159: -------------------------------- topic/jsiwek/improve-type-checks improves the type checking for records and fixes a problem with named table constructors in local scope. I didn't end up removing list expansion of table/set indices. > type checking inconsistencies > ----------------------------- > > Key: BIT-1159 > URL: https://bro-tracker.atlassian.net/browse/BIT-1159 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Labels: language > Attachments: signature.asc, signature.asc > > > If you try to compare a count to a port directly, you get the following: > {code} > operands must be of the same type (1500/tcp < 2000) > {code} > but if you have a record, and mixup the types like so, it silently fails: > {code} > type PortRange: record { > min: port &default=1/tcp; > max: port &default=65535/tcp; > }; > global pr = PortRange($min=1000,$max=2000); > #CORRECT: global pr = PortRange($min=1000/tcp,$max=2000/tcp); > event bro_init() > { > print (pr$min <= 1500/tcp && 1500/tcp < pr$max) ? "OK" : "NOTOK"; > } > {code} > {code} > $ bro a.bro > NOTOK > {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 21 08:04:44 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 21 Mar 2014 10:04:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1159) type checking inconsistencies In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1159: --------------------------- Status: Merge Request (was: Open) > type checking inconsistencies > ----------------------------- > > Key: BIT-1159 > URL: https://bro-tracker.atlassian.net/browse/BIT-1159 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Labels: language > Attachments: signature.asc, signature.asc > > > If you try to compare a count to a port directly, you get the following: > {code} > operands must be of the same type (1500/tcp < 2000) > {code} > but if you have a record, and mixup the types like so, it silently fails: > {code} > type PortRange: record { > min: port &default=1/tcp; > max: port &default=65535/tcp; > }; > global pr = PortRange($min=1000,$max=2000); > #CORRECT: global pr = PortRange($min=1000/tcp,$max=2000/tcp); > event bro_init() > { > print (pr$min <= 1500/tcp && 1500/tcp < pr$max) ? "OK" : "NOTOK"; > } > {code} > {code} > $ bro a.bro > NOTOK > {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Sat Mar 22 00:00:13 2014 From: noreply at bro.org (Merge Tracker) Date: Sat, 22 Mar 2014 00:00:13 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403220700.s2M70DRr019464@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ -------------- -------------- ------------- ---------- ------------- ---------- --------------------------------- BIT-1161 [1] Bro Jon Siwek - 2014-03-18 2.3 Normal topic/jsiwek/faster-val-clone [2] BIT-1160 [3] Bro,BroControl Bernhard Amann Daniel Thayer 2014-03-19 2.3 Normal Update cluster documentation BIT-1159 [4] Bro Justin Azoff Jon Siwek 2014-03-21 - Low type checking inconsistencies BIT-1150 [5] Bro Robin Sommer Seth Hall 2014-03-20 2.3 Normal X509 updates Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ---------- ---------- ------------------------------------------------- #4 [6] bro mareq [7] 2014-03-18 Protocol identification heuristics. [8] #1 [9] time-machine mareq [10] 2014-03-19 TM-16: Really skip VLAN header for indexing. [11] [1] BIT-1161 https://bro-tracker.atlassian.net/browse/BIT-1161 [2] faster-val-clone https://github.com/bro/bro/tree/topic/jsiwek/faster-val-clone [3] BIT-1160 https://bro-tracker.atlassian.net/browse/BIT-1160 [4] BIT-1159 https://bro-tracker.atlassian.net/browse/BIT-1159 [5] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [6] Pull Request #4 https://api.github.com/repos/bro/bro/issues/4 [7] mareq https://github.com/mareq [8] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [9] Pull Request #1 https://api.github.com/repos/bro/time-machine/issues/1 [10] mareq https://github.com/mareq [11] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From noreply at bro.org Sun Mar 23 00:00:14 2014 From: noreply at bro.org (Merge Tracker) Date: Sun, 23 Mar 2014 00:00:14 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403230700.s2N70EHK031515@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ -------------- -------------- ------------- ---------- ------------- ---------- --------------------------------- BIT-1161 [1] Bro Jon Siwek - 2014-03-18 2.3 Normal topic/jsiwek/faster-val-clone [2] BIT-1160 [3] Bro,BroControl Bernhard Amann Daniel Thayer 2014-03-19 2.3 Normal Update cluster documentation BIT-1159 [4] Bro Justin Azoff Jon Siwek 2014-03-21 - Low type checking inconsistencies BIT-1150 [5] Bro Robin Sommer Seth Hall 2014-03-20 2.3 Normal X509 updates Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ---------- ---------- ------------------------------------------------- #4 [6] bro mareq [7] 2014-03-18 Protocol identification heuristics. [8] #1 [9] time-machine mareq [10] 2014-03-19 TM-16: Really skip VLAN header for indexing. [11] [1] BIT-1161 https://bro-tracker.atlassian.net/browse/BIT-1161 [2] faster-val-clone https://github.com/bro/bro/tree/topic/jsiwek/faster-val-clone [3] BIT-1160 https://bro-tracker.atlassian.net/browse/BIT-1160 [4] BIT-1159 https://bro-tracker.atlassian.net/browse/BIT-1159 [5] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [6] Pull Request #4 https://api.github.com/repos/bro/bro/issues/4 [7] mareq https://github.com/mareq [8] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [9] Pull Request #1 https://api.github.com/repos/bro/time-machine/issues/1 [10] mareq https://github.com/mareq [11] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Sun Mar 23 12:03:44 2014 From: jira at bro-tracker.atlassian.net (Matthias Vallentin (JIRA)) Date: Sun, 23 Mar 2014 14:03:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1166) installation does not take place in given prefix entirely In-Reply-To: References: Message-ID: Matthias Vallentin created BIT-1166: --------------------------------------- Summary: installation does not take place in given prefix entirely Key: BIT-1166 URL: https://bro-tracker.atlassian.net/browse/BIT-1166 Project: Bro Issue Tracker Issue Type: Problem Components: Bro, BroControl Affects Versions: git/master Reporter: Matthias Vallentin When configuring Bro to remain in a given prefix, say {{/opt/bro}}, the installation of BroControl still attempts to create a spool directory outside of the prefix: {code} ./configure --prefix=/opt/bro make make install [...] CMake Error at aux/broctl/cmake_install.cmake:200 (FILE): file cannot create directory: /var/opt/bro/spool. Maybe need administrative privileges. {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Sun Mar 23 12:10:44 2014 From: jira at bro-tracker.atlassian.net (Matthias Vallentin (JIRA)) Date: Sun, 23 Mar 2014 14:10:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1140) Bloomfilter hashing problem In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15822#comment-15822 ] Matthias Vallentin commented on BIT-1140: ----------------------------------------- I have not yet had the chance to investigate the issue. [~aashish], since you reported the problem, maybe you could put together a small example, including data, that reproduces it? I sent along a patch which switches to double-hashing, but have lost track how it affected the problem. > Bloomfilter hashing problem > --------------------------- > > Key: BIT-1140 > URL: https://bro-tracker.atlassian.net/browse/BIT-1140 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Matthias Vallentin > Fix For: 2.3 > > > It seems bloomfilter hashing isn't working correctly. Has that been confirmed? Is there a fix? -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Sun Mar 23 13:33:44 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Sun, 23 Mar 2014 15:33:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1166) installation does not take place in given prefix entirely In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15823#comment-15823 ] Bernhard Amann commented on BIT-1166: ------------------------------------- That is odd, that does definitely not happen for me (I install Bro as a user all the time). I just re-checked, it works fine for me on os-x and fedora.... > installation does not take place in given prefix entirely > --------------------------------------------------------- > > Key: BIT-1166 > URL: https://bro-tracker.atlassian.net/browse/BIT-1166 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro, BroControl > Affects Versions: git/master > Reporter: Matthias Vallentin > Labels: build > > When configuring Bro to remain in a given prefix, say {{/opt/bro}}, the installation of BroControl still attempts to create a spool directory outside of the prefix: > {code} > ./configure --prefix=/opt/bro > make > make install > [...] > CMake Error at aux/broctl/cmake_install.cmake:200 (FILE): > file cannot create directory: /var/opt/bro/spool. Maybe need > administrative privileges. > {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Sun Mar 23 13:51:44 2014 From: jira at bro-tracker.atlassian.net (Matthias Vallentin (JIRA)) Date: Sun, 23 Mar 2014 15:51:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1166) installation does not take place in given prefix entirely In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Vallentin updated BIT-1166: ------------------------------------ Resolution: Cannot Reproduce Status: Closed (was: Open) > installation does not take place in given prefix entirely > --------------------------------------------------------- > > Key: BIT-1166 > URL: https://bro-tracker.atlassian.net/browse/BIT-1166 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro, BroControl > Affects Versions: git/master > Reporter: Matthias Vallentin > Labels: build > > When configuring Bro to remain in a given prefix, say {{/opt/bro}}, the installation of BroControl still attempts to create a spool directory outside of the prefix: > {code} > ./configure --prefix=/opt/bro > make > make install > [...] > CMake Error at aux/broctl/cmake_install.cmake:200 (FILE): > file cannot create directory: /var/opt/bro/spool. Maybe need > administrative privileges. > {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Sun Mar 23 13:51:44 2014 From: jira at bro-tracker.atlassian.net (Matthias Vallentin (JIRA)) Date: Sun, 23 Mar 2014 15:51:44 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1166) installation does not take place in given prefix entirely In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824#comment-15824 ] Matthias Vallentin commented on BIT-1166: ----------------------------------------- A student attempting to build Bro sent me this bug report. I asked for details on how to reproduce, but the problem apparently no longer occurs. Presumably this had to do with some submodule initialization problem. Closing it for now. > installation does not take place in given prefix entirely > --------------------------------------------------------- > > Key: BIT-1166 > URL: https://bro-tracker.atlassian.net/browse/BIT-1166 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro, BroControl > Affects Versions: git/master > Reporter: Matthias Vallentin > Labels: build > > When configuring Bro to remain in a given prefix, say {{/opt/bro}}, the installation of BroControl still attempts to create a spool directory outside of the prefix: > {code} > ./configure --prefix=/opt/bro > make > make install > [...] > CMake Error at aux/broctl/cmake_install.cmake:200 (FILE): > file cannot create directory: /var/opt/bro/spool. Maybe need > administrative privileges. > {code} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Mon Mar 24 00:00:19 2014 From: noreply at bro.org (Merge Tracker) Date: Mon, 24 Mar 2014 00:00:19 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403240700.s2O70Jb3018307@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ -------------- -------------- ------------- ---------- ------------- ---------- --------------------------------- BIT-1161 [1] Bro Jon Siwek - 2014-03-18 2.3 Normal topic/jsiwek/faster-val-clone [2] BIT-1160 [3] Bro,BroControl Bernhard Amann Daniel Thayer 2014-03-19 2.3 Normal Update cluster documentation BIT-1159 [4] Bro Justin Azoff Jon Siwek 2014-03-21 - Low type checking inconsistencies BIT-1150 [5] Bro Robin Sommer Seth Hall 2014-03-20 2.3 Normal X509 updates Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ---------- ---------- ------------------------------------------------- #4 [6] bro mareq [7] 2014-03-18 Protocol identification heuristics. [8] #1 [9] time-machine mareq [10] 2014-03-19 TM-16: Really skip VLAN header for indexing. [11] [1] BIT-1161 https://bro-tracker.atlassian.net/browse/BIT-1161 [2] faster-val-clone https://github.com/bro/bro/tree/topic/jsiwek/faster-val-clone [3] BIT-1160 https://bro-tracker.atlassian.net/browse/BIT-1160 [4] BIT-1159 https://bro-tracker.atlassian.net/browse/BIT-1159 [5] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [6] Pull Request #4 https://api.github.com/repos/bro/bro/issues/4 [7] mareq https://github.com/mareq [8] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [9] Pull Request #1 https://api.github.com/repos/bro/time-machine/issues/1 [10] mareq https://github.com/mareq [11] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Mon Mar 24 08:32:39 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 24 Mar 2014 10:32:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1134) DNS_Mgr::LookupAddr does not respect DNS_FAKE In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1134: --------------------------- Resolution: Fixed Status: Closed (was: Open) > DNS_Mgr::LookupAddr does not respect DNS_FAKE > --------------------------------------------- > > Key: BIT-1134 > URL: https://bro-tracker.atlassian.net/browse/BIT-1134 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 24 08:47:39 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 24 Mar 2014 10:47:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1143) Investigate replacing libmagic w/ signatures for file identificaiton In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15900#comment-15900 ] Jon Siwek commented on BIT-1143: -------------------------------- Seth do you have any feedback in these areas: * Notice anything missing from script-layer support of file-type detection? The only difference should be all matches are available instead of just one, so I don't expect any issue, but asking just in case. * Notice any problems with the file-magic signature grammar? * Are the default set of file-magic rules adequate or is there something that definitely needs work before merging (opposed to making iterative improvements later on) ? If no problems, I'll set this to a merge request. > Investigate replacing libmagic w/ signatures for file identificaiton > -------------------------------------------------------------------- > > Key: BIT-1143 > URL: https://bro-tracker.atlassian.net/browse/BIT-1143 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Seth Hall > Fix For: 2.3 > > > I think it makes sense to try to make the switch from libmagic to using Bro's own signature engine for file identification before the next release. Don't want people getting used to magic file format for their own custom file identification rules. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 24 09:28:39 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Mon, 24 Mar 2014 11:28:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1143) Investigate replacing libmagic w/ signatures for file identificaiton In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1143: --------------------------- Assignee: Robin Sommer (was: Seth Hall) > Investigate replacing libmagic w/ signatures for file identificaiton > -------------------------------------------------------------------- > > Key: BIT-1143 > URL: https://bro-tracker.atlassian.net/browse/BIT-1143 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Robin Sommer > Fix For: 2.3 > > > I think it makes sense to try to make the switch from libmagic to using Bro's own signature engine for file identification before the next release. Don't want people getting used to magic file format for their own custom file identification rules. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 24 09:28:39 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Mon, 24 Mar 2014 11:28:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1143) Investigate replacing libmagic w/ signatures for file identificaiton In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901#comment-15901 ] Seth Hall commented on BIT-1143: -------------------------------- Everything looked ok to me when I was playing with it. I think it's probably ready to be merged. > Investigate replacing libmagic w/ signatures for file identificaiton > -------------------------------------------------------------------- > > Key: BIT-1143 > URL: https://bro-tracker.atlassian.net/browse/BIT-1143 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Seth Hall > Fix For: 2.3 > > > I think it makes sense to try to make the switch from libmagic to using Bro's own signature engine for file identification before the next release. Don't want people getting used to magic file format for their own custom file identification rules. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Mon Mar 24 13:02:39 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 24 Mar 2014 15:02:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-348) Reassembler integer overflow issues. Data not delivered after 2GB In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-348: -------------------------- Assignee: Jon Siwek (was: Bernhard Amann) > Reassembler integer overflow issues. Data not delivered after 2GB > ----------------------------------------------------------------- > > Key: BIT-348 > URL: https://bro-tracker.atlassian.net/browse/BIT-348 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: gregor > Assignee: Jon Siwek > Priority: High > Labels: inttypes > Fix For: 2.3 > > > {noformat} > #!rst > The TCP Reassembler does not deliver any data to analyzers after the first 2GB due to signed integer overflow (Actually it will deliver again between 4--6GB, etc.) This happens silently, i.e., without content_gap events or Undelivered calls. > This report superseded BIT-315, BIT-137 > The TCP Reassembler (and Reassem) base class use ``int`` to keep track of sequence numbers and ``seq_delta`` to check for differences. If a connection exceeds 2GB, the relative sequence numbers (int) used by the Reassembler become negative. While many parts of the Reassembler still work (because seq_delta still reports the correct difference) some parts do not. In particular ``seq_to_skip`` is broken (and fails silently). There might well be other parts of the Reassembler that fail > silently as well, that I haven't found yet. > See Comments in TCP_Reassembler.cc for more details. > The Reassembler should use int64. However this will require deep changes to the Reassembler and the TCP Analyzer and TCP_Endpoint classes (since we also store sequence numbers there). Also, the analyzer framework will need tweaks as well (e.g., Undelivered uses ``int`` for sequence numbers, also has to go to 64 bit) > As a hotfix that seems to work I disabled the ``seq_to_skip`` features. It wasn't used by any analyzer or policy script (Note, that seq_to_skip is different from skip_deliveries). Hotfix is in > topic/gregor/reassembler-hotfix > {noformat} -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Tue Mar 25 00:00:19 2014 From: noreply at bro.org (Merge Tracker) Date: Tue, 25 Mar 2014 00:00:19 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403250700.s2P70Jao001251@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ -------------- -------------- ------------- ---------- ------------- ---------- --------------------------------- BIT-1161 [1] Bro Jon Siwek - 2014-03-18 2.3 Normal topic/jsiwek/faster-val-clone [2] BIT-1160 [3] Bro,BroControl Bernhard Amann Daniel Thayer 2014-03-19 2.3 Normal Update cluster documentation BIT-1159 [4] Bro Justin Azoff Jon Siwek 2014-03-21 - Low type checking inconsistencies BIT-1150 [5] Bro Robin Sommer Seth Hall 2014-03-20 2.3 Normal X509 updates Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ---------- ---------- ------------------------------------------------- #4 [6] bro mareq [7] 2014-03-18 Protocol identification heuristics. [8] #1 [9] time-machine mareq [10] 2014-03-19 TM-16: Really skip VLAN header for indexing. [11] [1] BIT-1161 https://bro-tracker.atlassian.net/browse/BIT-1161 [2] faster-val-clone https://github.com/bro/bro/tree/topic/jsiwek/faster-val-clone [3] BIT-1160 https://bro-tracker.atlassian.net/browse/BIT-1160 [4] BIT-1159 https://bro-tracker.atlassian.net/browse/BIT-1159 [5] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [6] Pull Request #4 https://api.github.com/repos/bro/bro/issues/4 [7] mareq https://github.com/mareq [8] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [9] Pull Request #1 https://api.github.com/repos/bro/time-machine/issues/1 [10] mareq https://github.com/mareq [11] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Tue Mar 25 08:25:39 2014 From: jira at bro-tracker.atlassian.net (Brian Little (JIRA)) Date: Tue, 25 Mar 2014 10:25:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1167) Add subnet support to intel framework In-Reply-To: References: Message-ID: Brian Little created BIT-1167: --------------------------------- Summary: Add subnet support to intel framework Key: BIT-1167 URL: https://bro-tracker.atlassian.net/browse/BIT-1167 Project: Bro Issue Tracker Issue Type: Patch Components: Bro Affects Versions: 2.2 Reporter: Brian Little Priority: Low Attachments: bro-intel-subnet.patch Here is a patch to add Intel::NET data as a type to search on. This allows adding whole subnets to the intel data rather than just individual addresses. I have also updated the btest. I'm not sure if the lookup is the best way of doing it - currently if loops through each subnet and then checks if the host is part of each. Is it possible to do it in a more efficient way? -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 25 08:28:39 2014 From: jira at bro-tracker.atlassian.net (Brian Little (JIRA)) Date: Tue, 25 Mar 2014 10:28:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1168) Add Java version to software framework In-Reply-To: References: Message-ID: Brian Little created BIT-1168: --------------------------------- Summary: Add Java version to software framework Key: BIT-1168 URL: https://bro-tracker.atlassian.net/browse/BIT-1168 Project: Bro Issue Tracker Issue Type: Patch Components: Bro Affects Versions: 2.2 Reporter: Brian Little Priority: Low Attachments: bro-java-software.patch A small patch to add Java into the list of Mozilla user agents searched for (parse_mozilla function). This is useful for the vulnerable software check. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 25 08:45:39 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 25 Mar 2014 10:45:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1167) Add subnet support to intel framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902#comment-15902 ] Jon Siwek commented on BIT-1167: -------------------------------- {quote} I'm not sure if the lookup is the best way of doing it - currently if loops through each subnet and then checks if the host is part of each. Is it possible to do it in a more efficient way? {quote} I believe sets/tables with an index of type subnet are already internally organized in to an efficient data structure, so no need to loop. Try: {code} const my_nets: set[subnet] = { 192.168.0.0/16, 10.0.0.0/8, } &redef; print 192.168.0.1 in my_nets; print 10.0.0.1 in my_nets; print 1.2.3.4 in my_nets; {code} > Add subnet support to intel framework > ------------------------------------- > > Key: BIT-1167 > URL: https://bro-tracker.atlassian.net/browse/BIT-1167 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.2 > Reporter: Brian Little > Priority: Low > Labels: intel, subnet > Attachments: bro-intel-subnet.patch > > > Here is a patch to add Intel::NET data as a type to search on. This allows adding whole subnets to the intel data rather than just individual addresses. > I have also updated the btest. > I'm not sure if the lookup is the best way of doing it - currently if loops through each subnet and then checks if the host is part of each. Is it possible to do it in a more efficient way? -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 25 09:31:39 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Tue, 25 Mar 2014 11:31:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1167) Add subnet support to intel framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15903#comment-15903 ] Seth Hall commented on BIT-1167: -------------------------------- Robin, what do you think? I had removed subnet support from the intel framework because you said there was some problem with using subnets at table indexes or something (maybe you remember the problem better than I do?). > Add subnet support to intel framework > ------------------------------------- > > Key: BIT-1167 > URL: https://bro-tracker.atlassian.net/browse/BIT-1167 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.2 > Reporter: Brian Little > Priority: Low > Labels: intel, subnet > Attachments: bro-intel-subnet.patch > > > Here is a patch to add Intel::NET data as a type to search on. This allows adding whole subnets to the intel data rather than just individual addresses. > I have also updated the btest. > I'm not sure if the lookup is the best way of doing it - currently if loops through each subnet and then checks if the host is part of each. Is it possible to do it in a more efficient way? -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 25 10:56:39 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 25 Mar 2014 12:56:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1143) Investigate replacing libmagic w/ signatures for file identificaiton In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1143: --------------------------- Status: Merge Request (was: Open) > Investigate replacing libmagic w/ signatures for file identificaiton > -------------------------------------------------------------------- > > Key: BIT-1143 > URL: https://bro-tracker.atlassian.net/browse/BIT-1143 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Robin Sommer > Fix For: 2.3 > > > I think it makes sense to try to make the switch from libmagic to using Bro's own signature engine for file identification before the next release. Don't want people getting used to magic file format for their own custom file identification rules. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Tue Mar 25 10:58:39 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 25 Mar 2014 12:58:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1143) Investigate replacing libmagic w/ signatures for file identificaiton In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15904#comment-15904 ] Jon Siwek commented on BIT-1143: -------------------------------- merge-ready version is still topic/jsiwek/file-signatures in bro, 3rdparty, bro-testing, and bro-testing-private > Investigate replacing libmagic w/ signatures for file identificaiton > -------------------------------------------------------------------- > > Key: BIT-1143 > URL: https://bro-tracker.atlassian.net/browse/BIT-1143 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Robin Sommer > Fix For: 2.3 > > > I think it makes sense to try to make the switch from libmagic to using Bro's own signature engine for file identification before the next release. Don't want people getting used to magic file format for their own custom file identification rules. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Wed Mar 26 00:00:16 2014 From: noreply at bro.org (Merge Tracker) Date: Wed, 26 Mar 2014 00:00:16 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403260700.s2Q70GgO021029@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ -------------- -------------- ------------- ---------- ------------- ---------- -------------------------------------------------------------------- BIT-1161 [1] Bro Jon Siwek - 2014-03-18 2.3 Normal topic/jsiwek/faster-val-clone [2] BIT-1160 [3] Bro,BroControl Bernhard Amann Daniel Thayer 2014-03-19 2.3 Normal Update cluster documentation BIT-1159 [4] Bro Justin Azoff Jon Siwek 2014-03-21 - Low type checking inconsistencies BIT-1150 [5] Bro Robin Sommer Seth Hall 2014-03-20 2.3 Normal X509 updates BIT-1143 [6] Bro Jon Siwek Robin Sommer 2014-03-25 2.3 Normal Investigate replacing libmagic w/ signatures for file identificaiton Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- -------------------------------------------- 11d3685 [7] bro Daniel Thayer 2014-03-25 Update instructions on how to build Bro docs Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ----------- ---------- ------------------------------------------------- #4 [8] bro mareq [9] 2014-03-18 Protocol identification heuristics. [10] #2 [11] pysubnettree hstern [12] 2014-03-25 Upload pysubnettree to pypi [13] #1 [14] time-machine mareq [15] 2014-03-19 TM-16: Really skip VLAN header for indexing. [16] [1] BIT-1161 https://bro-tracker.atlassian.net/browse/BIT-1161 [2] faster-val-clone https://github.com/bro/bro/tree/topic/jsiwek/faster-val-clone [3] BIT-1160 https://bro-tracker.atlassian.net/browse/BIT-1160 [4] BIT-1159 https://bro-tracker.atlassian.net/browse/BIT-1159 [5] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [6] BIT-1143 https://bro-tracker.atlassian.net/browse/BIT-1143 [7] 11d3685 https://github.com/bro/bro/commit/11d3685f88467bf8d3d5566ed3ea1113555ccd4d [8] Pull Request #4 https://api.github.com/repos/bro/bro/issues/4 [9] mareq https://github.com/mareq [10] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [11] Pull Request #2 https://api.github.com/repos/bro/pysubnettree/issues/2 [12] hstern https://github.com/hstern [13] Merge Pull Request #2 with git pull https://github.com/hstern/pysubnettree.git pypi [14] Pull Request #1 https://api.github.com/repos/bro/time-machine/issues/1 [15] mareq https://github.com/mareq [16] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Wed Mar 26 03:11:39 2014 From: jira at bro-tracker.atlassian.net (Marek Balint (JIRA)) Date: Wed, 26 Mar 2014 05:11:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (TM-16) Index not working when traffic encapsulated in 802.1q trunk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/TM-16?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marek Balint updated TM-16: --------------------------- Status: Merge Request (was: In Progress) > Index not working when traffic encapsulated in 802.1q trunk > ----------------------------------------------------------- > > Key: TM-16 > URL: https://bro-tracker.atlassian.net/browse/TM-16 > Project: Time Machine > Issue Type: Problem > Affects Versions: git/master > Environment: Ubuntu 10.04 , pf_ring > Reporter: tyler.schoenke > Labels: 802.1Q, indexes > Attachments: tm-16.patch > > > Hi All, > When I query the time machine index, I am not receiving any results. > I just restarted time machine, and checked one of the recent class files to see there is traffic for a particular IP address. > tcpdump -e -v -n -r class_all_1385406639.023206 "vlan and host 128.138.44.198" > It shows some traffic, example: > 128.138.44.198.54014 > 74.125.225.209.443: Flags [.], cksum 0x8d2c (correct), seq 1283940799:1283940800, ack 615539104, win 16311, length 1 > 19:11:00.571731632 10:8c:cf:57:46:00 > 00:1d:09:6a:d9:a9, ethertype 802.1Q (0x8100), length 70: vlan 987, p 0, ethertype IPv4, (tos 0x0, ttl 56, id 17482, offset 0, flags [none], proto TCP (6), length 52) > When I telnet localhost 42042 and run the following command, I don't receive any results. > query to_file "128.138.44.198.pcap" index ip "128.138.44.198" > In the above tcpdump, you can see my traffic is 802.1Q trunked. I have to use the "vlan" BPF to extract it with tcpdump, and am wondering if the trunking is causing problems with indexing? > I tested the same version of time machine on non-trunked traffic, and the index works fine. > Let me know if you need any other configuration info. > Tyler -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Thu Mar 27 00:00:14 2014 From: noreply at bro.org (Merge Tracker) Date: Thu, 27 Mar 2014 00:00:14 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403270700.s2R70EBw031783@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ -------------- -------------- ------------- ---------- ------------- ---------- -------------------------------------------------------------------- BIT-1161 [1] Bro Jon Siwek - 2014-03-18 2.3 Normal topic/jsiwek/faster-val-clone [2] BIT-1160 [3] Bro,BroControl Bernhard Amann Daniel Thayer 2014-03-19 2.3 Normal Update cluster documentation BIT-1159 [4] Bro Justin Azoff Jon Siwek 2014-03-21 - Low type checking inconsistencies BIT-1150 [5] Bro Robin Sommer Seth Hall 2014-03-20 2.3 Normal X509 updates BIT-1143 [6] Bro Jon Siwek Robin Sommer 2014-03-25 2.3 Normal Investigate replacing libmagic w/ signatures for file identificaiton Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- -------------------------------------------- 11d3685 [7] bro Daniel Thayer 2014-03-25 Update instructions on how to build Bro docs Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ----------- ---------- ------------------------------------------------- #4 [8] bro mareq [9] 2014-03-18 Protocol identification heuristics. [10] #2 [11] pysubnettree hstern [12] 2014-03-25 Upload pysubnettree to pypi [13] #1 [14] time-machine mareq [15] 2014-03-19 TM-16: Really skip VLAN header for indexing. [16] [1] BIT-1161 https://bro-tracker.atlassian.net/browse/BIT-1161 [2] faster-val-clone https://github.com/bro/bro/tree/topic/jsiwek/faster-val-clone [3] BIT-1160 https://bro-tracker.atlassian.net/browse/BIT-1160 [4] BIT-1159 https://bro-tracker.atlassian.net/browse/BIT-1159 [5] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [6] BIT-1143 https://bro-tracker.atlassian.net/browse/BIT-1143 [7] 11d3685 https://github.com/bro/bro/commit/11d3685f88467bf8d3d5566ed3ea1113555ccd4d [8] Pull Request #4 https://api.github.com/repos/bro/bro/issues/4 [9] mareq https://github.com/mareq [10] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [11] Pull Request #2 https://api.github.com/repos/bro/pysubnettree/issues/2 [12] hstern https://github.com/hstern [13] Merge Pull Request #2 with git pull https://github.com/hstern/pysubnettree.git pypi [14] Pull Request #1 https://api.github.com/repos/bro/time-machine/issues/1 [15] mareq https://github.com/mareq [16] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Thu Mar 27 20:17:39 2014 From: jira at bro-tracker.atlassian.net (Vern Paxson (JIRA)) Date: Thu, 27 Mar 2014 22:17:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1167) Add subnet support to intel framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905#comment-15905 ] Vern Paxson commented on BIT-1167: ---------------------------------- I don't know if this is the issue Robin had in mind, but one thing about subnets as table indexes is that they can overlap (two indices, one of which is a superset of the other), introducing ambiguity. > Add subnet support to intel framework > ------------------------------------- > > Key: BIT-1167 > URL: https://bro-tracker.atlassian.net/browse/BIT-1167 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.2 > Reporter: Brian Little > Priority: Low > Labels: intel, subnet > Attachments: bro-intel-subnet.patch > > > Here is a patch to add Intel::NET data as a type to search on. This allows adding whole subnets to the intel data rather than just individual addresses. > I have also updated the btest. > I'm not sure if the lookup is the best way of doing it - currently if loops through each subnet and then checks if the host is part of each. Is it possible to do it in a more efficient way? -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Fri Mar 28 00:00:18 2014 From: noreply at bro.org (Merge Tracker) Date: Fri, 28 Mar 2014 00:00:18 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403280700.s2S70IS2019016@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ -------------- -------------- ------------- ---------- ------------- ---------- -------------------------------------------------------------------- BIT-1161 [1] Bro Jon Siwek - 2014-03-18 2.3 Normal topic/jsiwek/faster-val-clone [2] BIT-1160 [3] Bro,BroControl Bernhard Amann Daniel Thayer 2014-03-19 2.3 Normal Update cluster documentation BIT-1159 [4] Bro Justin Azoff Jon Siwek 2014-03-21 - Low type checking inconsistencies BIT-1150 [5] Bro Robin Sommer Seth Hall 2014-03-20 2.3 Normal X509 updates BIT-1143 [6] Bro Jon Siwek Robin Sommer 2014-03-25 2.3 Normal Investigate replacing libmagic w/ signatures for file identificaiton Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- -------------------------------------------- 11d3685 [7] bro Daniel Thayer 2014-03-25 Update instructions on how to build Bro docs Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ----------- ---------- ------------------------------------------------- #4 [8] bro mareq [9] 2014-03-18 Protocol identification heuristics. [10] #2 [11] pysubnettree hstern [12] 2014-03-25 Upload pysubnettree to pypi [13] #1 [14] time-machine mareq [15] 2014-03-19 TM-16: Really skip VLAN header for indexing. [16] [1] BIT-1161 https://bro-tracker.atlassian.net/browse/BIT-1161 [2] faster-val-clone https://github.com/bro/bro/tree/topic/jsiwek/faster-val-clone [3] BIT-1160 https://bro-tracker.atlassian.net/browse/BIT-1160 [4] BIT-1159 https://bro-tracker.atlassian.net/browse/BIT-1159 [5] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [6] BIT-1143 https://bro-tracker.atlassian.net/browse/BIT-1143 [7] 11d3685 https://github.com/bro/bro/commit/11d3685f88467bf8d3d5566ed3ea1113555ccd4d [8] Pull Request #4 https://api.github.com/repos/bro/bro/issues/4 [9] mareq https://github.com/mareq [10] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [11] Pull Request #2 https://api.github.com/repos/bro/pysubnettree/issues/2 [12] hstern https://github.com/hstern [13] Merge Pull Request #2 with git pull https://github.com/hstern/pysubnettree.git pypi [14] Pull Request #1 https://api.github.com/repos/bro/time-machine/issues/1 [15] mareq https://github.com/mareq [16] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Fri Mar 28 07:02:39 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 28 Mar 2014 09:02:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1169) topic/jsiwek/parse-only In-Reply-To: References: Message-ID: Jon Siwek created BIT-1169: ------------------------------ Summary: topic/jsiwek/parse-only Key: BIT-1169 URL: https://bro-tracker.atlassian.net/browse/BIT-1169 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: git/master Reporter: Jon Siwek Fix For: 2.3 Adds a {{--parse-only}} option to Bro to exit right after parsing scripts w/ an appropriate exit status. Justin had a vim plugin for checking syntax by running the edited script through bro. This option helps tools like that not have to do their own cleanups of log files or other things bro may create if it actually gets a chance to run. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 28 07:02:39 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 28 Mar 2014 09:02:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1169) topic/jsiwek/parse-only In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1169: --------------------------- Status: Merge Request (was: Open) > topic/jsiwek/parse-only > ----------------------- > > Key: BIT-1169 > URL: https://bro-tracker.atlassian.net/browse/BIT-1169 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > Adds a {{--parse-only}} option to Bro to exit right after parsing scripts w/ an appropriate exit status. > Justin had a vim plugin for checking syntax by running the edited script through bro. This option helps tools like that not have to do their own cleanups of log files or other things bro may create if it actually gets a chance to run. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 28 11:11:39 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 28 Mar 2014 13:11:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1170) merge topic/bernhard/sumstats-read-expire In-Reply-To: References: Message-ID: Bernhard Amann created BIT-1170: ----------------------------------- Summary: merge topic/bernhard/sumstats-read-expire Key: BIT-1170 URL: https://bro-tracker.atlassian.net/browse/BIT-1170 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.3 topic/bernhard/sumstats-read-expire changes the &create_expire attributes in the sumstats framework to &read_expire. I talked to Seth about it and Justin tested the modification for about a day - it seems to get rid of some error messages for him. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 28 11:11:39 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 28 Mar 2014 13:11:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1170) merge topic/bernhard/sumstats-read-expire In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1170: -------------------------------- Status: Merge Request (was: Open) > merge topic/bernhard/sumstats-read-expire > ----------------------------------------- > > Key: BIT-1170 > URL: https://bro-tracker.atlassian.net/browse/BIT-1170 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Fix For: 2.3 > > > topic/bernhard/sumstats-read-expire changes the &create_expire attributes in the sumstats framework to &read_expire. > I talked to Seth about it and Justin tested the modification for about a day - it seems to get rid of some error messages for him. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 28 12:13:39 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 28 Mar 2014 14:13:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906#comment-15906 ] Bernhard Amann commented on BIT-1150: ------------------------------------- Seth says the script looks, good, reassigning to Robin. > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 28 12:15:39 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 28 Mar 2014 14:15:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1150: -------------------------------- Assignee: Robin Sommer (was: Seth Hall) > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Robin Sommer > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 28 12:24:39 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 28 Mar 2014 14:24:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (TM-16) Index not working when traffic encapsulated in 802.1q trunk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/TM-16?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated TM-16: ----------------------------- Assignee: Seth Hall > Index not working when traffic encapsulated in 802.1q trunk > ----------------------------------------------------------- > > Key: TM-16 > URL: https://bro-tracker.atlassian.net/browse/TM-16 > Project: Time Machine > Issue Type: Problem > Affects Versions: git/master > Environment: Ubuntu 10.04 , pf_ring > Reporter: tyler.schoenke > Assignee: Seth Hall > Labels: 802.1Q, indexes > Attachments: tm-16.patch > > > Hi All, > When I query the time machine index, I am not receiving any results. > I just restarted time machine, and checked one of the recent class files to see there is traffic for a particular IP address. > tcpdump -e -v -n -r class_all_1385406639.023206 "vlan and host 128.138.44.198" > It shows some traffic, example: > 128.138.44.198.54014 > 74.125.225.209.443: Flags [.], cksum 0x8d2c (correct), seq 1283940799:1283940800, ack 615539104, win 16311, length 1 > 19:11:00.571731632 10:8c:cf:57:46:00 > 00:1d:09:6a:d9:a9, ethertype 802.1Q (0x8100), length 70: vlan 987, p 0, ethertype IPv4, (tos 0x0, ttl 56, id 17482, offset 0, flags [none], proto TCP (6), length 52) > When I telnet localhost 42042 and run the following command, I don't receive any results. > query to_file "128.138.44.198.pcap" index ip "128.138.44.198" > In the above tcpdump, you can see my traffic is 802.1Q trunked. I have to use the "vlan" BPF to extract it with tcpdump, and am wondering if the trunking is causing problems with indexing? > I tested the same version of time machine on non-trunked traffic, and the index works fine. > Let me know if you need any other configuration info. > Tyler -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 28 12:36:39 2014 From: jira at bro-tracker.atlassian.net (aashish (JIRA)) Date: Fri, 28 Mar 2014 14:36:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (TM-16) Index not working when traffic encapsulated in 802.1q trunk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/TM-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907#comment-15907 ] aashish commented on TM-16: --------------------------- I have an intern coming in this summer dedicated to work on time-machine. I can have him look at this issue too. > Index not working when traffic encapsulated in 802.1q trunk > ----------------------------------------------------------- > > Key: TM-16 > URL: https://bro-tracker.atlassian.net/browse/TM-16 > Project: Time Machine > Issue Type: Problem > Affects Versions: git/master > Environment: Ubuntu 10.04 , pf_ring > Reporter: tyler.schoenke > Assignee: Seth Hall > Labels: 802.1Q, indexes > Attachments: tm-16.patch > > > Hi All, > When I query the time machine index, I am not receiving any results. > I just restarted time machine, and checked one of the recent class files to see there is traffic for a particular IP address. > tcpdump -e -v -n -r class_all_1385406639.023206 "vlan and host 128.138.44.198" > It shows some traffic, example: > 128.138.44.198.54014 > 74.125.225.209.443: Flags [.], cksum 0x8d2c (correct), seq 1283940799:1283940800, ack 615539104, win 16311, length 1 > 19:11:00.571731632 10:8c:cf:57:46:00 > 00:1d:09:6a:d9:a9, ethertype 802.1Q (0x8100), length 70: vlan 987, p 0, ethertype IPv4, (tos 0x0, ttl 56, id 17482, offset 0, flags [none], proto TCP (6), length 52) > When I telnet localhost 42042 and run the following command, I don't receive any results. > query to_file "128.138.44.198.pcap" index ip "128.138.44.198" > In the above tcpdump, you can see my traffic is 802.1Q trunked. I have to use the "vlan" BPF to extract it with tcpdump, and am wondering if the trunking is causing problems with indexing? > I tested the same version of time machine on non-trunked traffic, and the index works fine. > Let me know if you need any other configuration info. > Tyler -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 28 13:13:39 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 28 Mar 2014 15:13:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1162) Sumstat measurements stop working on clusters with single slow nodes In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15908#comment-15908 ] Bernhard Amann commented on BIT-1162: ------------------------------------- It looks like BIT-1170 might completely (or at least nearly completely) solve this issue. However, we probably still should look for a way to make data aggregation quicker in this cases - it can take quite a while until all nodes have reported back... > Sumstat measurements stop working on clusters with single slow nodes > -------------------------------------------------------------------- > > Key: BIT-1162 > URL: https://bro-tracker.atlassian.net/browse/BIT-1162 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Fix For: 2.3 > > > If you use sumstats on a Bro cluster and have one (or more) overloaded nodes, sumstats is near unusable at the moment. > Sumstats asks for all keys in order. The speed of getting them seems to depend on the speed in which the individual cluster worker nodes answer. > If there is a very slow node in the network, for sumstats with a higher number of keys, processing will just stop after just a few keys. No warning message or similar is output. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 28 13:28:39 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 28 Mar 2014 15:28:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1171) misc/app-stats/main.bro broken for a few sites In-Reply-To: References: Message-ID: Bernhard Amann created BIT-1171: ----------------------------------- Summary: misc/app-stats/main.bro broken for a few sites Key: BIT-1171 URL: https://bro-tracker.atlassian.net/browse/BIT-1171 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.3 Currently the reporting of misc/app-stats/main.bro seems to be quite wrong for some of the sites it monitors. At the very least the numbers for youtube and netflix are completely off, gmail also seems slightly unbelievable. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From jira at bro-tracker.atlassian.net Fri Mar 28 16:19:39 2014 From: jira at bro-tracker.atlassian.net (tyler.schoenke (JIRA)) Date: Fri, 28 Mar 2014 18:19:39 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (TM-16) Index not working when traffic encapsulated in 802.1q trunk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/TM-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15909#comment-15909 ] tyler.schoenke commented on TM-16: ---------------------------------- Cool, thanks Aashish. Tyler "aashish (JIRA)" wrote: [ https://bro-tracker.atlassian.net/browse/TM-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907#comment-15907 ] aashish commented on TM-16: --------------------------- I have an intern coming in this summer dedicated to work on time-machine. I can have him look at this issue too. -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) > Index not working when traffic encapsulated in 802.1q trunk > ----------------------------------------------------------- > > Key: TM-16 > URL: https://bro-tracker.atlassian.net/browse/TM-16 > Project: Time Machine > Issue Type: Problem > Affects Versions: git/master > Environment: Ubuntu 10.04 , pf_ring > Reporter: tyler.schoenke > Assignee: Seth Hall > Labels: 802.1Q, indexes > Attachments: tm-16.patch > > > Hi All, > When I query the time machine index, I am not receiving any results. > I just restarted time machine, and checked one of the recent class files to see there is traffic for a particular IP address. > tcpdump -e -v -n -r class_all_1385406639.023206 "vlan and host 128.138.44.198" > It shows some traffic, example: > 128.138.44.198.54014 > 74.125.225.209.443: Flags [.], cksum 0x8d2c (correct), seq 1283940799:1283940800, ack 615539104, win 16311, length 1 > 19:11:00.571731632 10:8c:cf:57:46:00 > 00:1d:09:6a:d9:a9, ethertype 802.1Q (0x8100), length 70: vlan 987, p 0, ethertype IPv4, (tos 0x0, ttl 56, id 17482, offset 0, flags [none], proto TCP (6), length 52) > When I telnet localhost 42042 and run the following command, I don't receive any results. > query to_file "128.138.44.198.pcap" index ip "128.138.44.198" > In the above tcpdump, you can see my traffic is 802.1Q trunked. I have to use the "vlan" BPF to extract it with tcpdump, and am wondering if the trunking is causing problems with indexing? > I tested the same version of time machine on non-trunked traffic, and the index works fine. > Let me know if you need any other configuration info. > Tyler -- This message was sent by Atlassian JIRA (v6.2-OD-10-004-WN#6253) From noreply at bro.org Sat Mar 29 00:00:14 2014 From: noreply at bro.org (Merge Tracker) Date: Sat, 29 Mar 2014 00:00:14 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403290700.s2T70E8g006194@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ -------------- -------------- ------------- ---------- ------------- ---------- -------------------------------------------------------------------- BIT-1170 [1] Bro Bernhard Amann - 2014-03-28 2.3 Normal merge topic/bernhard/sumstats-read-expire BIT-1169 [2] Bro Jon Siwek - 2014-03-28 2.3 Normal topic/jsiwek/parse-only [3] BIT-1161 [4] Bro Jon Siwek - 2014-03-18 2.3 Normal topic/jsiwek/faster-val-clone [5] BIT-1160 [6] Bro,BroControl Bernhard Amann Daniel Thayer 2014-03-19 2.3 Normal Update cluster documentation BIT-1159 [7] Bro Justin Azoff Jon Siwek 2014-03-21 - Low type checking inconsistencies BIT-1150 [8] Bro Robin Sommer Robin Sommer 2014-03-28 2.3 Normal X509 updates BIT-1143 [9] Bro Jon Siwek Robin Sommer 2014-03-25 2.3 Normal Investigate replacing libmagic w/ signatures for file identificaiton Open Fastpath Commits ====================== Commit Component Author Date Summary ------------ ----------- ------------- ---------- -------------------------------------------- 11d3685 [10] bro Daniel Thayer 2014-03-25 Update instructions on how to build Bro docs Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ----------- ---------- ------------------------------------------------- #4 [11] bro mareq [12] 2014-03-18 Protocol identification heuristics. [13] #2 [14] pysubnettree hstern [15] 2014-03-25 Upload pysubnettree to pypi [16] #1 [17] time-machine mareq [18] 2014-03-19 TM-16: Really skip VLAN header for indexing. [19] [1] BIT-1170 https://bro-tracker.atlassian.net/browse/BIT-1170 [2] BIT-1169 https://bro-tracker.atlassian.net/browse/BIT-1169 [3] parse-only https://github.com/bro/bro/tree/topic/jsiwek/parse-only [4] BIT-1161 https://bro-tracker.atlassian.net/browse/BIT-1161 [5] faster-val-clone https://github.com/bro/bro/tree/topic/jsiwek/faster-val-clone [6] BIT-1160 https://bro-tracker.atlassian.net/browse/BIT-1160 [7] BIT-1159 https://bro-tracker.atlassian.net/browse/BIT-1159 [8] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [9] BIT-1143 https://bro-tracker.atlassian.net/browse/BIT-1143 [10] 11d3685 https://github.com/bro/bro/commit/11d3685f88467bf8d3d5566ed3ea1113555ccd4d [11] Pull Request #4 https://api.github.com/repos/bro/bro/issues/4 [12] mareq https://github.com/mareq [13] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [14] Pull Request #2 https://api.github.com/repos/bro/pysubnettree/issues/2 [15] hstern https://github.com/hstern [16] Merge Pull Request #2 with git pull https://github.com/hstern/pysubnettree.git pypi [17] Pull Request #1 https://api.github.com/repos/bro/time-machine/issues/1 [18] mareq https://github.com/mareq [19] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From noreply at bro.org Sun Mar 30 00:00:14 2014 From: noreply at bro.org (Merge Tracker) Date: Sun, 30 Mar 2014 00:00:14 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403300700.s2U70EKf024695@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ -------------- -------------- ------------- ---------- ------------- ---------- -------------------------------------------------------------------- BIT-1170 [1] Bro Bernhard Amann - 2014-03-28 2.3 Normal merge topic/bernhard/sumstats-read-expire BIT-1169 [2] Bro Jon Siwek - 2014-03-28 2.3 Normal topic/jsiwek/parse-only [3] BIT-1161 [4] Bro Jon Siwek - 2014-03-18 2.3 Normal topic/jsiwek/faster-val-clone [5] BIT-1160 [6] Bro,BroControl Bernhard Amann Daniel Thayer 2014-03-19 2.3 Normal Update cluster documentation BIT-1159 [7] Bro Justin Azoff Jon Siwek 2014-03-21 - Low type checking inconsistencies BIT-1150 [8] Bro Robin Sommer Robin Sommer 2014-03-28 2.3 Normal X509 updates BIT-1143 [9] Bro Jon Siwek Robin Sommer 2014-03-25 2.3 Normal Investigate replacing libmagic w/ signatures for file identificaiton Open Fastpath Commits ====================== Commit Component Author Date Summary ------------ ----------- ------------- ---------- -------------------------------------------- 11d3685 [10] bro Daniel Thayer 2014-03-25 Update instructions on how to build Bro docs Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ----------- ---------- ------------------------------------------------- #4 [11] bro mareq [12] 2014-03-18 Protocol identification heuristics. [13] #2 [14] pysubnettree hstern [15] 2014-03-25 Upload pysubnettree to pypi [16] #1 [17] time-machine mareq [18] 2014-03-19 TM-16: Really skip VLAN header for indexing. [19] [1] BIT-1170 https://bro-tracker.atlassian.net/browse/BIT-1170 [2] BIT-1169 https://bro-tracker.atlassian.net/browse/BIT-1169 [3] parse-only https://github.com/bro/bro/tree/topic/jsiwek/parse-only [4] BIT-1161 https://bro-tracker.atlassian.net/browse/BIT-1161 [5] faster-val-clone https://github.com/bro/bro/tree/topic/jsiwek/faster-val-clone [6] BIT-1160 https://bro-tracker.atlassian.net/browse/BIT-1160 [7] BIT-1159 https://bro-tracker.atlassian.net/browse/BIT-1159 [8] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [9] BIT-1143 https://bro-tracker.atlassian.net/browse/BIT-1143 [10] 11d3685 https://github.com/bro/bro/commit/11d3685f88467bf8d3d5566ed3ea1113555ccd4d [11] Pull Request #4 https://api.github.com/repos/bro/bro/issues/4 [12] mareq https://github.com/mareq [13] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [14] Pull Request #2 https://api.github.com/repos/bro/pysubnettree/issues/2 [15] hstern https://github.com/hstern [16] Merge Pull Request #2 with git pull https://github.com/hstern/pysubnettree.git pypi [17] Pull Request #1 https://api.github.com/repos/bro/time-machine/issues/1 [18] mareq https://github.com/mareq [19] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From noreply at bro.org Mon Mar 31 00:00:15 2014 From: noreply at bro.org (Merge Tracker) Date: Mon, 31 Mar 2014 00:00:15 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201403310700.s2V70FOc000310@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ -------------- -------------- ------------- ---------- ------------- ---------- -------------------------------------------------------------------- BIT-1170 [1] Bro Bernhard Amann - 2014-03-28 2.3 Normal merge topic/bernhard/sumstats-read-expire BIT-1169 [2] Bro Jon Siwek - 2014-03-28 2.3 Normal topic/jsiwek/parse-only [3] BIT-1161 [4] Bro Jon Siwek - 2014-03-18 2.3 Normal topic/jsiwek/faster-val-clone [5] BIT-1160 [6] Bro,BroControl Bernhard Amann Daniel Thayer 2014-03-19 2.3 Normal Update cluster documentation BIT-1159 [7] Bro Justin Azoff Jon Siwek 2014-03-21 - Low type checking inconsistencies BIT-1150 [8] Bro Robin Sommer Robin Sommer 2014-03-30 2.3 Normal X509 updates BIT-1143 [9] Bro Jon Siwek Robin Sommer 2014-03-25 2.3 Normal Investigate replacing libmagic w/ signatures for file identificaiton Open Fastpath Commits ====================== Commit Component Author Date Summary ------------ ----------- ------------- ---------- -------------------------------------------- 11d3685 [10] bro Daniel Thayer 2014-03-25 Update instructions on how to build Bro docs Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ----------- ---------- ------------------------------------------------- #4 [11] bro mareq [12] 2014-03-18 Protocol identification heuristics. [13] #2 [14] pysubnettree hstern [15] 2014-03-25 Upload pysubnettree to pypi [16] #1 [17] time-machine mareq [18] 2014-03-19 TM-16: Really skip VLAN header for indexing. [19] [1] BIT-1170 https://bro-tracker.atlassian.net/browse/BIT-1170 [2] BIT-1169 https://bro-tracker.atlassian.net/browse/BIT-1169 [3] parse-only https://github.com/bro/bro/tree/topic/jsiwek/parse-only [4] BIT-1161 https://bro-tracker.atlassian.net/browse/BIT-1161 [5] faster-val-clone https://github.com/bro/bro/tree/topic/jsiwek/faster-val-clone [6] BIT-1160 https://bro-tracker.atlassian.net/browse/BIT-1160 [7] BIT-1159 https://bro-tracker.atlassian.net/browse/BIT-1159 [8] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [9] BIT-1143 https://bro-tracker.atlassian.net/browse/BIT-1143 [10] 11d3685 https://github.com/bro/bro/commit/11d3685f88467bf8d3d5566ed3ea1113555ccd4d [11] Pull Request #4 https://api.github.com/repos/bro/bro/issues/4 [12] mareq https://github.com/mareq [13] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [14] Pull Request #2 https://api.github.com/repos/bro/pysubnettree/issues/2 [15] hstern https://github.com/hstern [16] Merge Pull Request #2 with git pull https://github.com/hstern/pysubnettree.git pypi [17] Pull Request #1 https://api.github.com/repos/bro/time-machine/issues/1 [18] mareq https://github.com/mareq [19] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Mon Mar 31 14:41:52 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 31 Mar 2014 16:41:52 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000#comment-16000 ] Robin Sommer commented on BIT-1150: ----------------------------------- Please give me some text for CHANGES and NEWS that summarize the changes, it's tricky for me to pull that out of the commits. > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Robin Sommer > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 14:41:52 2014 From: jira at bro-tracker.atlassian.net (Anthony Verez (JIRA)) Date: Mon, 31 Mar 2014 16:41:52 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1172) Add uid field to the signatures log stream In-Reply-To: References: Message-ID: Anthony Verez created BIT-1172: ---------------------------------- Summary: Add uid field to the signatures log stream Key: BIT-1172 URL: https://bro-tracker.atlassian.net/browse/BIT-1172 Project: Bro Issue Tracker Issue Type: Patch Components: Bro Affects Versions: git/master Environment: Tested on Debian wheezy and Security Onion Reporter: Anthony Verez Attachments: 0001-add-uid-field-to-the-signatures-log-stream.patch This patchs adds a uid field (conn) to the signatures log stream. I wanted to have that to analyze connections that triggered a signature match. Thanks, Anthony Verez -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 14:41:52 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 31 Mar 2014 16:41:52 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1143) Investigate replacing libmagic w/ signatures for file identificaiton In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1143: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Investigate replacing libmagic w/ signatures for file identificaiton > -------------------------------------------------------------------- > > Key: BIT-1143 > URL: https://bro-tracker.atlassian.net/browse/BIT-1143 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Robin Sommer > Fix For: 2.3 > > > I think it makes sense to try to make the switch from libmagic to using Bro's own signature engine for file identification before the next release. Don't want people getting used to magic file format for their own custom file identification rules. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 14:41:52 2014 From: jira at bro-tracker.atlassian.net (Anthony Verez (JIRA)) Date: Mon, 31 Mar 2014 16:41:52 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1172) Add uid field to the signatures log stream In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001#comment-16001 ] Anthony Verez commented on BIT-1172: ------------------------------------ Let me know if you prefer a pull request on github. > Add uid field to the signatures log stream > ------------------------------------------ > > Key: BIT-1172 > URL: https://bro-tracker.atlassian.net/browse/BIT-1172 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: git/master > Environment: Tested on Debian wheezy and Security Onion > Reporter: Anthony Verez > Attachments: 0001-add-uid-field-to-the-signatures-log-stream.patch > > > This patchs adds a uid field (conn) to the signatures log stream. > I wanted to have that to analyze connections that triggered a signature match. > Thanks, > Anthony Verez -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 14:41:53 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 31 Mar 2014 16:41:53 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1169) topic/jsiwek/parse-only In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1169: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > topic/jsiwek/parse-only > ----------------------- > > Key: BIT-1169 > URL: https://bro-tracker.atlassian.net/browse/BIT-1169 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > Adds a {{--parse-only}} option to Bro to exit right after parsing scripts w/ an appropriate exit status. > Justin had a vim plugin for checking syntax by running the edited script through bro. This option helps tools like that not have to do their own cleanups of log files or other things bro may create if it actually gets a chance to run. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 14:41:52 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 31 Mar 2014 16:41:52 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1150: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Robin Sommer > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 14:41:53 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 31 Mar 2014 16:41:53 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1159) type checking inconsistencies In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1159: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > type checking inconsistencies > ----------------------------- > > Key: BIT-1159 > URL: https://bro-tracker.atlassian.net/browse/BIT-1159 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.2 > Reporter: Justin Azoff > Assignee: Jon Siwek > Priority: Low > Labels: language > Attachments: signature.asc, signature.asc > > > If you try to compare a count to a port directly, you get the following: > {code} > operands must be of the same type (1500/tcp < 2000) > {code} > but if you have a record, and mixup the types like so, it silently fails: > {code} > type PortRange: record { > min: port &default=1/tcp; > max: port &default=65535/tcp; > }; > global pr = PortRange($min=1000,$max=2000); > #CORRECT: global pr = PortRange($min=1000/tcp,$max=2000/tcp); > event bro_init() > { > print (pr$min <= 1500/tcp && 1500/tcp < pr$max) ? "OK" : "NOTOK"; > } > {code} > {code} > $ bro a.bro > NOTOK > {code} -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 14:41:53 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 31 Mar 2014 16:41:53 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1160) Update cluster documentation In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1160: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Update cluster documentation > ---------------------------- > > Key: BIT-1160 > URL: https://bro-tracker.atlassian.net/browse/BIT-1160 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro, BroControl > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Daniel Thayer > Labels: documentation > Fix For: 2.3 > > Attachments: signature.asc > > > We should update the Cluster documentation, if possible before releasing 2.3. > I set up a Bro cluster for the first time yesterday - and when you look at the current state at the documentation it is not very useful... > ...for example it contains things like (link to an example for the config) in the text. Furthermore it does not really mention how to actually configure Bro for a cluster, there is no mention of node.cfg, etc. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 14:41:53 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 31 Mar 2014 16:41:53 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002#comment-16002 ] Robin Sommer commented on BIT-1150: ----------------------------------- Merged, but leaving open as CHANGES still needs an update. > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Robin Sommer > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 14:41:53 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 31 Mar 2014 16:41:53 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1170) merge topic/bernhard/sumstats-read-expire In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1170: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > merge topic/bernhard/sumstats-read-expire > ----------------------------------------- > > Key: BIT-1170 > URL: https://bro-tracker.atlassian.net/browse/BIT-1170 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Fix For: 2.3 > > > topic/bernhard/sumstats-read-expire changes the &create_expire attributes in the sumstats framework to &read_expire. > I talked to Seth about it and Justin tested the modification for about a day - it seems to get rid of some error messages for him. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 14:41:53 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 31 Mar 2014 16:41:53 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1172) Add uid field to the signatures log stream In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1172: ------------------------------ Status: Merge Request (was: Open) > Add uid field to the signatures log stream > ------------------------------------------ > > Key: BIT-1172 > URL: https://bro-tracker.atlassian.net/browse/BIT-1172 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: git/master > Environment: Tested on Debian wheezy and Security Onion > Reporter: Anthony Verez > Attachments: 0001-add-uid-field-to-the-signatures-log-stream.patch > > > This patchs adds a uid field (conn) to the signatures log stream. > I wanted to have that to analyze connections that triggered a signature match. > Thanks, > Anthony Verez -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 14:41:53 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 31 Mar 2014 16:41:53 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1150: ------------------------------ Status: Reopened (was: Closed) > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Robin Sommer > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 14:41:53 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 31 Mar 2014 16:41:53 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1168) Add Java version to software framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1168: ------------------------------ Status: Merge Request (was: Open) > Add Java version to software framework > -------------------------------------- > > Key: BIT-1168 > URL: https://bro-tracker.atlassian.net/browse/BIT-1168 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.2 > Reporter: Brian Little > Priority: Low > Labels: framework, java, software > Attachments: bro-java-software.patch > > > A small patch to add Java into the list of Mozilla user agents searched for (parse_mozilla function). This is useful for the vulnerable software check. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 14:41:53 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 31 Mar 2014 16:41:53 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1142) SNMP Analysis In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003#comment-16003 ] Robin Sommer commented on BIT-1142: ----------------------------------- I'm fine merging this but I would then still like to have an snmp.log for 2.3 ... > SNMP Analysis > ------------- > > Key: BIT-1142 > URL: https://bro-tracker.atlassian.net/browse/BIT-1142 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: BinPAC, Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Seth Hall > Fix For: 2.3 > > > /topic/jsiwek/snmp in bro, binpac, and bro-testing-private adds support for parsing SNMP datagrams. It's only absent a snmp.log. > Seth, do you mind taking a look at what might make sense for a default snmp.log? I'm guessing it might look similar in concept to dns.log. A difference is I'm not sure how meaningful raw OID to value mappings will be. > The code is in a merge-able state as it is in the branch/repos I mentioned, and IMO, has value even without a default snmp.log. So if you just want to flip to a merge request and postpone thinking up an snmp.log for later, I think that's fine, too. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 14:41:53 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Mon, 31 Mar 2014 16:41:53 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005#comment-16005 ] Bernhard Amann commented on BIT-1150: ------------------------------------- Does this work as a short summary? I think it should be the gist of it... Rework and move X509 certificate processing from the SSL protocol analyzer to a dedicated file analyzer. This will allow us to examine X509 certificates from sources other than SSL in the future. Furthermore, we now parse more fields and extensions from the certificates (e.g. elliptic curve information, subject alternative names, basic constraints). Certificate validation also was improved, should be easier to use and exposes information like the full verified certificate chain. Note - this update changes the output of ssl.log, adds a new x509.log with certificate information. Furthermore all x509 events and handling functions changed. > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Robin Sommer > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 14:41:53 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 31 Mar 2014 16:41:53 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1129) RADIUS Protocol Analyzer In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004#comment-16004 ] Robin Sommer commented on BIT-1129: ----------------------------------- Vlad, any trace? What about the two questions above? > RADIUS Protocol Analyzer > ------------------------ > > Key: BIT-1129 > URL: https://bro-tracker.atlassian.net/browse/BIT-1129 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: grigorescu > Assignee: Vlad Grigorescu > Fix For: 2.3 > > > topic/vladg/radius is ready to be merged. It's been running at CMU for a few months with no issues. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 14:41:53 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 31 Mar 2014 16:41:53 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1161) topic/jsiwek/faster-val-clone In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1161: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > topic/jsiwek/faster-val-clone > ----------------------------- > > Key: BIT-1161 > URL: https://bro-tracker.atlassian.net/browse/BIT-1161 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch makes it less expensive to serialize large/complex values (e.g. connection and/or fa_file records). > The obvious overhead that could be reduced was from the fixed growth incrementation of the buffer used to contain serialized data. With records that expand out to ~1.6M (master) or ~3M (topic/bernhard/file-analysis-x509) in serialized form, it takes a bit too many allocations when trying to get there in growth increments of 64K. It may also help some to use realloc instead of new/memcpy/delete each time it needs to grow. > I didn't find it helped much to increase the initial buffer size from 64K (and 90% of the things needing serialization fit in that size buffer anyway). > It could possibly help to preallocate a buffer that gets re-used across serializations instead of repeatedly allocating small buffers that will need to be resized. > I don't have a complete breakdown/view of the bytes that make up the serialized version of the large/complex records, but taking a quick look I note that the filenames from Location information of each BroObj/Val make up a third of ~1.6M (master). And that's the full path of each file, so this all will depend on where the Bro scripts reside on the file system (i.e. put them as close to the root dir as possible and you might increase performance!). > Any other quick ideas of what can be done here? If not, improving the serialization seems to deserve its own project (which also might be part of the new comm. library project) for later. > In the meantime, it's at least shown that avoiding situations where large/complex records are serialized can help (BIT-1139). And that might always be a useful optimization strategy if the serialized representation of Vals is going to scale not just as a function of their value, but also w/ their type/attribute/location information. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 21:09:10 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 31 Mar 2014 23:09:10 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1172) Add uid field to the signatures log stream In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1172: --------------------------------- Assignee: Seth Hall > Add uid field to the signatures log stream > ------------------------------------------ > > Key: BIT-1172 > URL: https://bro-tracker.atlassian.net/browse/BIT-1172 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: git/master > Environment: Tested on Debian wheezy and Security Onion > Reporter: Anthony Verez > Assignee: Seth Hall > Attachments: 0001-add-uid-field-to-the-signatures-log-stream.patch > > > This patchs adds a uid field (conn) to the signatures log stream. > I wanted to have that to analyze connections that triggered a signature match. > Thanks, > Anthony Verez -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 21:09:09 2014 From: jira at bro-tracker.atlassian.net (grigorescu (JIRA)) Date: Mon, 31 Mar 2014 23:09:09 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1129) RADIUS Protocol Analyzer In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16006#comment-16006 ] grigorescu commented on BIT-1129: --------------------------------- Found my trace. I need to sanitize it and will create a test for it. * For the expiration logic - you're right, expire should log the entry. I'll try to get that a test for that as well. * For the attribute list - it's a vector because technically you could have multiple entries of the same attribute type. The only place this seems to happen in the real world is for the vendor-specific type. I have some code that would deal with those types, but it requires some further work (and that won't be a base script). I'm not sure what to do in the case that other attribute types (e.g. username, calling station id, etc.) are present multiple times. It's not a violation of the RFC, so perhaps just a weird? > RADIUS Protocol Analyzer > ------------------------ > > Key: BIT-1129 > URL: https://bro-tracker.atlassian.net/browse/BIT-1129 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: grigorescu > Assignee: Vlad Grigorescu > Fix For: 2.3 > > > topic/vladg/radius is ready to be merged. It's been running at CMU for a few months with no issues. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 21:09:09 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 31 Mar 2014 23:09:09 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1168) Add Java version to software framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1168: --------------------------------- Assignee: Seth Hall > Add Java version to software framework > -------------------------------------- > > Key: BIT-1168 > URL: https://bro-tracker.atlassian.net/browse/BIT-1168 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.2 > Reporter: Brian Little > Assignee: Seth Hall > Priority: Low > Labels: framework, java, software > Attachments: bro-java-software.patch > > > A small patch to add Java into the list of Mozilla user agents searched for (parse_mozilla function). This is useful for the vulnerable software check. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 21:09:10 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 31 Mar 2014 23:09:10 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1173) Upload pysubnettree to pypi In-Reply-To: References: Message-ID: Robin Sommer created BIT-1173: --------------------------------- Summary: Upload pysubnettree to pypi Key: BIT-1173 URL: https://bro-tracker.atlassian.net/browse/BIT-1173 Project: Bro Issue Tracker Issue Type: Task Components: pysubnettree Reporter: Robin Sommer See https://github.com/bro/pysubnettree/pull/2 -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Mon Mar 31 21:09:10 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 31 Mar 2014 23:09:10 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1173) Upload pysubnettree to pypi In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1173: ------------------------------ Resolution: Invalid Status: Closed (was: Open) > Upload pysubnettree to pypi > --------------------------- > > Key: BIT-1173 > URL: https://bro-tracker.atlassian.net/browse/BIT-1173 > Project: Bro Issue Tracker > Issue Type: Task > Components: pysubnettree > Reporter: Robin Sommer > > See https://github.com/bro/pysubnettree/pull/2 -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307)