From noreply at bro.org Tue Apr 1 00:00:16 2014 From: noreply at bro.org (Merge Tracker) Date: Tue, 1 Apr 2014 00:00:16 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404010700.s3170G46022904@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ---------- ---------- ------------- ---------- ------------------------------------------ BIT-1172 [1] Bro Anthony Verez Seth Hall 2014-03-31 - Normal Add uid field to the signatures log stream BIT-1168 [2] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ --------- ---------- ------------------------------------------------ #4 [3] bro mareq [4] 2014-04-01 Protocol identification heuristics. [5] #1 [6] time-machine mareq [7] 2014-03-19 TM-16: Really skip VLAN header for indexing. [8] [1] BIT-1172 https://bro-tracker.atlassian.net/browse/BIT-1172 [2] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [3] Pull Request #4 https://github.com/bro/bro/pull/4 [4] mareq https://github.com/mareq [5] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [6] Pull Request #1 https://github.com/bro/time-machine/pull/1 [7] mareq https://github.com/mareq [8] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Tue Apr 1 08:38:07 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Tue, 1 Apr 2014 10:38:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1047) Delete old scripts before installing new ones In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16007#comment-16007 ] Bernhard Amann commented on BIT-1047: ------------------------------------- Do we still want to add this? It would be kind of nice for 2.3. I just noticed again, that if you install the current master over an old 2.2 release you get errors. One of them is my fault - function definitions moved from the ssl bif file (which no longer exists) to the x509 bif file. If the ssl one continues hanging around, it leads to a "already defined" error when trying to start Bro. > Delete old scripts before installing new ones > --------------------------------------------- > > Key: BIT-1047 > URL: https://bro-tracker.atlassian.net/browse/BIT-1047 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Reporter: Robin Sommer > Fix For: 2.4 > > > People keep having problems when they install a new Bro version > over the installation of an old one because scripts that have disappeared in the new version will keep sticking around from the previous installation. > We should simply remove the old scripts/base and scripts/policy before installing anything new. People aren't supposed to edit in there so that should be safe. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Tue Apr 1 09:17:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 1 Apr 2014 11:17:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1174) topic/jsiwek/coverity In-Reply-To: References: Message-ID: Jon Siwek created BIT-1174: ------------------------------ Summary: topic/jsiwek/coverity Key: BIT-1174 URL: https://bro-tracker.atlassian.net/browse/BIT-1174 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: git/master Reporter: Jon Siwek Fix For: 2.3 I went through the list of "untriaged" Coverity issues to see if there's anything major that slipped by. Didn't notice anything, but some were easy/obvious to fix. There's still ~45 untriaged issues (not new) that I'm not sure how to categorize, but my own impression was that they weren't severe enough to require immediate attention. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Tue Apr 1 09:18:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 1 Apr 2014 11:18:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1174) topic/jsiwek/coverity In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1174: --------------------------- Status: Merge Request (was: Open) > topic/jsiwek/coverity > --------------------- > > Key: BIT-1174 > URL: https://bro-tracker.atlassian.net/browse/BIT-1174 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > I went through the list of "untriaged" Coverity issues to see if there's anything major that slipped by. Didn't notice anything, but some were easy/obvious to fix. > There's still ~45 untriaged issues (not new) that I'm not sure how to categorize, but my own impression was that they weren't severe enough to require immediate attention. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Tue Apr 1 09:47:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 1 Apr 2014 11:47:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1047) Delete old scripts before installing new ones In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008#comment-16008 ] Jon Siwek commented on BIT-1047: -------------------------------- {quote} Do we still want to add this? It would be kind of nice for 2.3. I just noticed again, that if you install the current master over an old 2.2 release you get errors. One of them is my fault - function definitions moved from the ssl bif file (which no longer exists) to the x509 bif file. If the ssl one continues hanging around, it leads to a "already defined" error when trying to start Bro. {quote} I don't think deleting old scripts at install-time completely fixes this problem because {{base/bif/plugins/__load__.bro}} actually still has {{@load ./Bro_SSL.functions.bif.bro}} when that file is no longer supposed to be loaded (because technically it doesn't exist in the newer Bro version). That is, we can remove stale scripts on install, but then it's still going to {{@load}} a file that doesn't exist and error out in a different way. This is also not just an install-time problem, but you also get in this bad state if you try to run from the build dir, but have not done a {{make clean}} between the last time you build Bro and the time you built Bro w/ the SSL/X509 BIF changes in place. So I think the actual bug is that {{base/bif/plugins/__load__.bro}} does not remove references to things that should no longer exist. Whether or not to delete old scripts at install-time as suggested by this ticket should be unrelated. > Delete old scripts before installing new ones > --------------------------------------------- > > Key: BIT-1047 > URL: https://bro-tracker.atlassian.net/browse/BIT-1047 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Reporter: Robin Sommer > Fix For: 2.4 > > > People keep having problems when they install a new Bro version > over the installation of an old one because scripts that have disappeared in the new version will keep sticking around from the previous installation. > We should simply remove the old scripts/base and scripts/policy before installing anything new. People aren't supposed to edit in there so that should be safe. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Tue Apr 1 12:34:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 1 Apr 2014 14:34:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1175) topic/jsiwek/bif-loader-scripts In-Reply-To: References: Message-ID: Jon Siwek created BIT-1175: ------------------------------ Summary: topic/jsiwek/bif-loader-scripts Key: BIT-1175 URL: https://bro-tracker.atlassian.net/browse/BIT-1175 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: git/master Reporter: Jon Siwek Fix For: 2.3 This branch is in the cmake and bro repos. And fixes a problem with __load__.bro file generation for .bif.bro stubs that have had their .bif file removed since a previous build of Bro occurred. Bernhard, can you please double-check this fixes the issue you mentioned in BIT-1047 ? -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Tue Apr 1 13:13:08 2014 From: jira at bro-tracker.atlassian.net (Aashish Sharma (JIRA)) Date: Tue, 1 Apr 2014 15:13:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1140) Bloomfilter hashing problem In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aashish Sharma updated BIT-1140: -------------------------------- Attachment: bloom-test2.bro bloom-test-short.bro Test files to reproduce the problem. > Bloomfilter hashing problem > --------------------------- > > Key: BIT-1140 > URL: https://bro-tracker.atlassian.net/browse/BIT-1140 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Matthias Vallentin > Fix For: 2.3 > > Attachments: bloom-test2.bro, bloom-test-short.bro > > > It seems bloomfilter hashing isn't working correctly. Has that been confirmed? Is there a fix? -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Tue Apr 1 13:13:08 2014 From: jira at bro-tracker.atlassian.net (Aashish Sharma (JIRA)) Date: Tue, 1 Apr 2014 15:13:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1140) Bloomfilter hashing problem In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16010#comment-16010 ] Aashish Sharma commented on BIT-1140: ------------------------------------- Matthias, I have created two simple test files. Both of these files add a bunch of URL's to a bloomfilter. Then, scripts do a bloomfilter_lookup on a *different* set of URLs. You should notice two problems 1) URLs which aren't even added to the filter show up as in the filter ( bloomfilter_lookup returns 1) 2) Return 1 is inconsistent on multiple runs (sometimes it shows 0, sometimes 1) The URLs' added are from in smtp extracted URLs while URLs looked up are in http stream. Basically, I am making a bloomfilter for all the URLs extracted from emails and then testing against HTTP to see if any of smtp URLs "has been clicked". (Currently I use a table which gives me correct results but with a much bigger memory footprint) With boomfilter, we see quite a bit of false positives. Here are two examples: 1) bloom-test-short.bro - only does lookup for 4 URLs. on repeated run (bro ./bloom-test-short.bro ) you should see different outputs on hits (0 - miss, 1 hit) and the URLs we are looking up aren't added to the filter. 2) bloom-test2.bro - Has much more extensive Lookup set. On a run you should see the lookup results as 0 or 1 and it varies. Again all the lookup URLs are different from the ones added. Please let me know if you have problems reproducing this. I can send you the actual smtp-embedded-url.bro scripts as well. > Bloomfilter hashing problem > --------------------------- > > Key: BIT-1140 > URL: https://bro-tracker.atlassian.net/browse/BIT-1140 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Matthias Vallentin > Fix For: 2.3 > > Attachments: bloom-test2.bro, bloom-test-short.bro > > > It seems bloomfilter hashing isn't working correctly. Has that been confirmed? Is there a fix? -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Tue Apr 1 16:22:08 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Tue, 1 Apr 2014 18:22:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1175) topic/jsiwek/bif-loader-scripts In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011#comment-16011 ] Bernhard Amann commented on BIT-1175: ------------------------------------- Yes, it does. Thank you. > topic/jsiwek/bif-loader-scripts > ------------------------------- > > Key: BIT-1175 > URL: https://bro-tracker.atlassian.net/browse/BIT-1175 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch is in the cmake and bro repos. And fixes a problem with __load__.bro file generation for .bif.bro stubs that have had their .bif file removed since a previous build of Bro occurred. > Bernhard, can you please double-check this fixes the issue you mentioned in BIT-1047 ? -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Tue Apr 1 16:43:07 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Tue, 1 Apr 2014 18:43:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1163) Logging framework text (ascii) writer writes sets as table[...] In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1163: -------------------------------- Fix Version/s: 2.3 > Logging framework text (ascii) writer writes sets as table[...] > --------------------------------------------------------------- > > Key: BIT-1163 > URL: https://bro-tracker.atlassian.net/browse/BIT-1163 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > Assignee: Bernhard Amann > Fix For: 2.3 > > > This is a minor point, but in our logs we should be writing set types as "set" and not table (even though internally they are the same thing). > Here's an example header from files.log: > {quote} > #types time string table[addr] table[addr] table[string] string count table[string] string string interval bool bool count count count count bool string string string stringstring > {quote} -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Tue Apr 1 16:45:07 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Tue, 1 Apr 2014 18:45:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1163) Logging framework text (ascii) writer writes sets as table[...] In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1163: -------------------------------- Status: Merge Request (was: Open) > Logging framework text (ascii) writer writes sets as table[...] > --------------------------------------------------------------- > > Key: BIT-1163 > URL: https://bro-tracker.atlassian.net/browse/BIT-1163 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > Assignee: Bernhard Amann > Fix For: 2.3 > > > This is a minor point, but in our logs we should be writing set types as "set" and not table (even though internally they are the same thing). > Here's an example header from files.log: > {quote} > #types time string table[addr] table[addr] table[string] string count table[string] string string interval bool bool count count count count bool string string string stringstring > {quote} -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Tue Apr 1 16:45:07 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Tue, 1 Apr 2014 18:45:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1163) Logging framework text (ascii) writer writes sets as table[...] In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16012#comment-16012 ] Bernhard Amann commented on BIT-1163: ------------------------------------- fix is in branch topic/bernhard/ticket-1163 of bro and testing > Logging framework text (ascii) writer writes sets as table[...] > --------------------------------------------------------------- > > Key: BIT-1163 > URL: https://bro-tracker.atlassian.net/browse/BIT-1163 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > Assignee: Bernhard Amann > Fix For: 2.3 > > > This is a minor point, but in our logs we should be writing set types as "set" and not table (even though internally they are the same thing). > Here's an example header from files.log: > {quote} > #types time string table[addr] table[addr] table[string] string count table[string] string string interval bool bool count count count count bool string string string stringstring > {quote} -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Tue Apr 1 16:58:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 1 Apr 2014 18:58:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1163) Logging framework text (ascii) writer writes sets as table[...] In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1163: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Logging framework text (ascii) writer writes sets as table[...] > --------------------------------------------------------------- > > Key: BIT-1163 > URL: https://bro-tracker.atlassian.net/browse/BIT-1163 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > Assignee: Bernhard Amann > Fix For: 2.3 > > > This is a minor point, but in our logs we should be writing set types as "set" and not table (even though internally they are the same thing). > Here's an example header from files.log: > {quote} > #types time string table[addr] table[addr] table[string] string count table[string] string string interval bool bool count count count count bool string string string stringstring > {quote} -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Tue Apr 1 16:58:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 1 Apr 2014 18:58:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1174) topic/jsiwek/coverity In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1174: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > topic/jsiwek/coverity > --------------------- > > Key: BIT-1174 > URL: https://bro-tracker.atlassian.net/browse/BIT-1174 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > I went through the list of "untriaged" Coverity issues to see if there's anything major that slipped by. Didn't notice anything, but some were easy/obvious to fix. > There's still ~45 untriaged issues (not new) that I'm not sure how to categorize, but my own impression was that they weren't severe enough to require immediate attention. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From noreply at bro.org Wed Apr 2 00:00:14 2014 From: noreply at bro.org (Merge Tracker) Date: Wed, 2 Apr 2014 00:00:14 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404020700.s3270Eww024285@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ---------- ---------- ------------- ---------- ------------------------------------------ BIT-1172 [1] Bro Anthony Verez Seth Hall 2014-03-31 - Normal Add uid field to the signatures log stream BIT-1168 [2] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ --------- ---------- ------------------------------------------------ #4 [3] bro mareq [4] 2014-04-01 Protocol identification heuristics. [5] #1 [6] time-machine mareq [7] 2014-03-19 TM-16: Really skip VLAN header for indexing. [8] [1] BIT-1172 https://bro-tracker.atlassian.net/browse/BIT-1172 [2] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [3] Pull Request #4 https://github.com/bro/bro/pull/4 [4] mareq https://github.com/mareq [5] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [6] Pull Request #1 https://github.com/bro/time-machine/pull/1 [7] mareq https://github.com/mareq [8] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Wed Apr 2 07:26:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 2 Apr 2014 09:26:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1175) topic/jsiwek/bif-loader-scripts In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1175: --------------------------- Status: Merge Request (was: Open) > topic/jsiwek/bif-loader-scripts > ------------------------------- > > Key: BIT-1175 > URL: https://bro-tracker.atlassian.net/browse/BIT-1175 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch is in the cmake and bro repos. And fixes a problem with __load__.bro file generation for .bif.bro stubs that have had their .bif file removed since a previous build of Bro occurred. > Bernhard, can you please double-check this fixes the issue you mentioned in BIT-1047 ? -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From noreply at bro.org Thu Apr 3 00:00:19 2014 From: noreply at bro.org (Merge Tracker) Date: Thu, 3 Apr 2014 00:00:19 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404030700.s3370JvY016415@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ---------- ---------- ------------- ---------- ------------------------------------------ BIT-1175 [1] Bro Jon Siwek - 2014-04-02 2.3 Normal topic/jsiwek/bif-loader-scripts [2] BIT-1172 [3] Bro Anthony Verez Seth Hall 2014-03-31 - Normal Add uid field to the signatures log stream BIT-1168 [4] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ --------- ---------- ------------------------------------------------- #4 [5] bro mareq [6] 2014-04-01 Protocol identification heuristics. [7] #1 [8] time-machine mareq [9] 2014-03-19 TM-16: Really skip VLAN header for indexing. [10] [1] BIT-1175 https://bro-tracker.atlassian.net/browse/BIT-1175 [2] bif-loader-scripts https://github.com/bro/bro/tree/topic/jsiwek/bif-loader-scripts [3] BIT-1172 https://bro-tracker.atlassian.net/browse/BIT-1172 [4] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [5] Pull Request #4 https://github.com/bro/bro/pull/4 [6] mareq https://github.com/mareq [7] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [8] Pull Request #1 https://github.com/bro/time-machine/pull/1 [9] mareq https://github.com/mareq [10] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Thu Apr 3 06:58:07 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 3 Apr 2014 08:58:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1142) SNMP Analysis In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16013#comment-16013 ] Seth Hall commented on BIT-1142: -------------------------------- I'm almost done with it and going to try and get it committed today (i know i've said this before...). > SNMP Analysis > ------------- > > Key: BIT-1142 > URL: https://bro-tracker.atlassian.net/browse/BIT-1142 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: BinPAC, Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Seth Hall > Fix For: 2.3 > > > /topic/jsiwek/snmp in bro, binpac, and bro-testing-private adds support for parsing SNMP datagrams. It's only absent a snmp.log. > Seth, do you mind taking a look at what might make sense for a default snmp.log? I'm guessing it might look similar in concept to dns.log. A difference is I'm not sure how meaningful raw OID to value mappings will be. > The code is in a merge-able state as it is in the branch/repos I mentioned, and IMO, has value even without a default snmp.log. So if you just want to flip to a merge request and postpone thinking up an snmp.log for later, I think that's fine, too. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Thu Apr 3 10:36:07 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 3 Apr 2014 12:36:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1172) Add uid field to the signatures log stream In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1172: --------------------------- Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Add uid field to the signatures log stream > ------------------------------------------ > > Key: BIT-1172 > URL: https://bro-tracker.atlassian.net/browse/BIT-1172 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: git/master > Environment: Tested on Debian wheezy and Security Onion > Reporter: Anthony Verez > Assignee: Seth Hall > Attachments: 0001-add-uid-field-to-the-signatures-log-stream.patch > > > This patchs adds a uid field (conn) to the signatures log stream. > I wanted to have that to analyze connections that triggered a signature match. > Thanks, > Anthony Verez -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Thu Apr 3 16:09:09 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 3 Apr 2014 18:09:09 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1175) topic/jsiwek/bif-loader-scripts In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1175: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > topic/jsiwek/bif-loader-scripts > ------------------------------- > > Key: BIT-1175 > URL: https://bro-tracker.atlassian.net/browse/BIT-1175 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch is in the cmake and bro repos. And fixes a problem with __load__.bro file generation for .bif.bro stubs that have had their .bif file removed since a previous build of Bro occurred. > Bernhard, can you please double-check this fixes the issue you mentioned in BIT-1047 ? -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From noreply at bro.org Fri Apr 4 00:00:16 2014 From: noreply at bro.org (Merge Tracker) Date: Fri, 4 Apr 2014 00:00:16 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404040700.s3470GcM005382@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- -------------------------------------- BIT-1168 [1] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ --------- ---------- ------------------------------------------------ #4 [2] bro mareq [3] 2014-04-01 Protocol identification heuristics. [4] #1 [5] time-machine mareq [6] 2014-03-19 TM-16: Really skip VLAN header for indexing. [7] [1] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [2] Pull Request #4 https://github.com/bro/bro/pull/4 [3] mareq https://github.com/mareq [4] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [5] Pull Request #1 https://github.com/bro/time-machine/pull/1 [6] mareq https://github.com/mareq [7] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Fri Apr 4 06:26:07 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 4 Apr 2014 08:26:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1176) Using an undefined function in a when statement causes a segfault In-Reply-To: References: Message-ID: Bernhard Amann created BIT-1176: ----------------------------------- Summary: Using an undefined function in a when statement causes a segfault Key: BIT-1176 URL: https://bro-tracker.atlassian.net/browse/BIT-1176 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.3 Attachments: crashme.bro Running the following script crashes bro with a null-pointer exception: {code:title=crashMe.bro} global crashMe: function():string; when( local result = crashMe() ) { print result; } {code} Backtrace: {code} * thread #1: tid = 0x226111, 0x000000010022bddf bro`Val::IsZero(this=0x0000000000000000) const + 15 at Val.cc:323, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x30) frame #0: 0x000000010022bddf bro`Val::IsZero(this=0x0000000000000000) const + 15 at Val.cc:323 320 321 int Val::IsZero() const 322 { -> 323 switch ( type->InternalType() ) { 324 case TYPE_INTERNAL_INT: return val.int_val == 0; 325 case TYPE_INTERNAL_UNSIGNED: return val.uint_val == 0; 326 case TYPE_INTERNAL_DOUBLE: return val.double_val == 0.0; (lldb) bt * thread #1: tid = 0x226111, 0x000000010022bddf bro`Val::IsZero(this=0x0000000000000000) const + 15 at Val.cc:323, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x30) * frame #0: 0x000000010022bddf bro`Val::IsZero(this=0x0000000000000000) const + 15 at Val.cc:323 frame #1: 0x000000010020b452 bro`Trigger::Eval(this=0x0000000105d45d60) + 578 at Trigger.cc:209 frame #2: 0x000000010020ae95 bro`Trigger(this=0x0000000105d45d60, arg_cond=0x0000000104a00390, arg_body=0x0000000104a00500, arg_timeout_stmts=0x0000000000000000, arg_timeout=0x0000000000000000, arg_frame=0x00007fff5fbfec80, arg_is_return=false, arg_location=0x00000001049fb7a0) + 1285 at Trigger.cc:140 frame #3: 0x000000010020a98a bro`Trigger(this=0x0000000105d45d60, arg_cond=0x0000000104a00390, arg_body=0x0000000104a00500, arg_timeout_stmts=0x0000000000000000, arg_timeout=0x0000000000000000, arg_frame=0x00007fff5fbfec80, arg_is_return=false, arg_location=0x00000001049fb7a0) + 106 at Trigger.cc:147 frame #4: 0x000000010020566f bro`WhenStmt::Exec(this=0x0000000104a00900, f=0x00007fff5fbfec80, flow=0x00007fff5fbfece8) const + 239 at Stmt.cc:2041 frame #5: 0x0000000100203204 bro`StmtList::Exec(this=0x00000001049fbe80, f=0x00007fff5fbfec80, flow=0x00007fff5fbfece8) const + 228 at Stmt.cc:1639 frame #6: 0x000000010003d244 bro`main(argc=2, argv=0x00007fff5fbffa40) + 15476 at main.cc:1116 {code} -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 06:29:07 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 4 Apr 2014 08:29:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1177) SumStats dynamic updates do not work in cluster mode In-Reply-To: References: Message-ID: Bernhard Amann created BIT-1177: ----------------------------------- Summary: SumStats dynamic updates do not work in cluster mode Key: BIT-1177 URL: https://bro-tracker.atlassian.net/browse/BIT-1177 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.3 In cluster mode, dynamic updates via the request_key function do not work. The reason is, that, at the moment, in cluster mode the function is defined only on the manager. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 08:49:07 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 4 Apr 2014 10:49:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1178) SSL/TLS analyzer does not abort early enough on non-ssl connections In-Reply-To: References: Message-ID: Bernhard Amann created BIT-1178: ----------------------------------- Summary: SSL/TLS analyzer does not abort early enough on non-ssl connections Key: BIT-1178 URL: https://bro-tracker.atlassian.net/browse/BIT-1178 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.3 Some sites see quite a bit of non-ssl traffic on port 443. At the moment, the SSL analyzer manages to parse quite a lot of this non-ssl traffic and generates ssl-events for it (including sending things to the certificate analyzer which generates warnings). We should probably try to abort much earlier. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 11:29:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Apr 2014 13:29:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1178) SSL/TLS analyzer does not abort early enough on non-ssl connections In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1178: ------------------------------ Assignee: Bernhard Amann > SSL/TLS analyzer does not abort early enough on non-ssl connections > ------------------------------------------------------------------- > > Key: BIT-1178 > URL: https://bro-tracker.atlassian.net/browse/BIT-1178 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Bernhard Amann > Fix For: 2.3 > > > Some sites see quite a bit of non-ssl traffic on port 443. At the moment, the SSL analyzer manages to parse quite a lot of this non-ssl traffic and generates ssl-events for it (including sending things to the certificate analyzer which generates warnings). > We should probably try to abort much earlier. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 11:30:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Apr 2014 13:30:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1176) Using an undefined function in a when statement causes a segfault In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1176: ------------------------------ Fix Version/s: (was: 2.3) 2.4 > Using an undefined function in a when statement causes a segfault > ----------------------------------------------------------------- > > Key: BIT-1176 > URL: https://bro-tracker.atlassian.net/browse/BIT-1176 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Fix For: 2.4 > > Attachments: crashme.bro > > > Running the following script crashes bro with a null-pointer exception: > {code:title=crashMe.bro} > global crashMe: function():string; > when( local result = crashMe() ) { > print result; > } > {code} > Backtrace: > {code} > * thread #1: tid = 0x226111, 0x000000010022bddf bro`Val::IsZero(this=0x0000000000000000) const + 15 at Val.cc:323, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x30) > frame #0: 0x000000010022bddf bro`Val::IsZero(this=0x0000000000000000) const + 15 at Val.cc:323 > 320 > 321 int Val::IsZero() const > 322 { > -> 323 switch ( type->InternalType() ) { > 324 case TYPE_INTERNAL_INT: return val.int_val == 0; > 325 case TYPE_INTERNAL_UNSIGNED: return val.uint_val == 0; > 326 case TYPE_INTERNAL_DOUBLE: return val.double_val == 0.0; > (lldb) bt > * thread #1: tid = 0x226111, 0x000000010022bddf bro`Val::IsZero(this=0x0000000000000000) const + 15 at Val.cc:323, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x30) > * frame #0: 0x000000010022bddf bro`Val::IsZero(this=0x0000000000000000) const + 15 at Val.cc:323 > frame #1: 0x000000010020b452 bro`Trigger::Eval(this=0x0000000105d45d60) + 578 at Trigger.cc:209 > frame #2: 0x000000010020ae95 bro`Trigger(this=0x0000000105d45d60, arg_cond=0x0000000104a00390, arg_body=0x0000000104a00500, arg_timeout_stmts=0x0000000000000000, arg_timeout=0x0000000000000000, arg_frame=0x00007fff5fbfec80, arg_is_return=false, arg_location=0x00000001049fb7a0) + 1285 at Trigger.cc:140 > frame #3: 0x000000010020a98a bro`Trigger(this=0x0000000105d45d60, arg_cond=0x0000000104a00390, arg_body=0x0000000104a00500, arg_timeout_stmts=0x0000000000000000, arg_timeout=0x0000000000000000, arg_frame=0x00007fff5fbfec80, arg_is_return=false, arg_location=0x00000001049fb7a0) + 106 at Trigger.cc:147 > frame #4: 0x000000010020566f bro`WhenStmt::Exec(this=0x0000000104a00900, f=0x00007fff5fbfec80, flow=0x00007fff5fbfece8) const + 239 at Stmt.cc:2041 > frame #5: 0x0000000100203204 bro`StmtList::Exec(this=0x00000001049fbe80, f=0x00007fff5fbfec80, flow=0x00007fff5fbfece8) const + 228 at Stmt.cc:1639 > frame #6: 0x000000010003d244 bro`main(argc=2, argv=0x00007fff5fbffa40) + 15476 at main.cc:1116 > {code} -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 11:30:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Apr 2014 13:30:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1177) SumStats dynamic updates do not work in cluster mode In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1177: ------------------------------ Assignee: Bernhard Amann > SumStats dynamic updates do not work in cluster mode > ---------------------------------------------------- > > Key: BIT-1177 > URL: https://bro-tracker.atlassian.net/browse/BIT-1177 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Bernhard Amann > Fix For: 2.3 > > > In cluster mode, dynamic updates via the request_key function do not work. > The reason is, that, at the moment, in cluster mode the function is defined only on the manager. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 11:35:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Apr 2014 13:35:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1171) misc/app-stats/main.bro broken for a few sites In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1171: ------------------------------ Assignee: Seth Hall > misc/app-stats/main.bro broken for a few sites > ---------------------------------------------- > > Key: BIT-1171 > URL: https://bro-tracker.atlassian.net/browse/BIT-1171 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Seth Hall > Fix For: 2.3 > > > Currently the reporting of misc/app-stats/main.bro seems to be quite wrong for some of the sites it monitors. > At the very least the numbers for youtube and netflix are completely off, gmail also seems slightly unbelievable. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 11:36:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Apr 2014 13:36:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1171) misc/app-stats/main.bro broken for a few sites In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16014#comment-16014 ] Robin Sommer commented on BIT-1171: ----------------------------------- We'll remove (comment out) the ones which don't work, which will likely just leave Facebook in there. > misc/app-stats/main.bro broken for a few sites > ---------------------------------------------- > > Key: BIT-1171 > URL: https://bro-tracker.atlassian.net/browse/BIT-1171 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Seth Hall > Fix For: 2.3 > > > Currently the reporting of misc/app-stats/main.bro seems to be quite wrong for some of the sites it monitors. > At the very least the numbers for youtube and netflix are completely off, gmail also seems slightly unbelievable. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 11:39:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Apr 2014 13:39:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1162) Sumstat measurements stop working on clusters with single slow nodes In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1162: ------------------------------ Resolution: Fixed Status: Closed (was: Open) BIT-1170 fixed the immediate problem, although it's still slowish. Leave as it is for now. > Sumstat measurements stop working on clusters with single slow nodes > -------------------------------------------------------------------- > > Key: BIT-1162 > URL: https://bro-tracker.atlassian.net/browse/BIT-1162 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Fix For: 2.3 > > > If you use sumstats on a Bro cluster and have one (or more) overloaded nodes, sumstats is near unusable at the moment. > Sumstats asks for all keys in order. The speed of getting them seems to depend on the speed in which the individual cluster worker nodes answer. > If there is a very slow node in the network, for sumstats with a higher number of keys, processing will just stop after just a few keys. No warning message or similar is output. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 11:41:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Apr 2014 13:41:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1156) DNS analyzer parses TXT records imcompletely In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1156: ------------------------------ Assignee: Robin Sommer > DNS analyzer parses TXT records imcompletely > -------------------------------------------- > > Key: BIT-1156 > URL: https://bro-tracker.atlassian.net/browse/BIT-1156 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Robin Sommer > Fix For: 2.3 > > > The payload of DNS TXT records can consist of multiple character strings but the DNS analyzer parses out only the first. We should parse them out all and then probably concatenate into a single string to pass to the event, separated with semicolons or something. > I have a trace with an example but it would need anonymization before inclusion into the test suite. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 11:41:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Apr 2014 13:41:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1151) JSON output In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1151: ------------------------------ Resolution: Merged Status: Closed (was: Open) > JSON output > ----------- > > Key: BIT-1151 > URL: https://bro-tracker.atlassian.net/browse/BIT-1151 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 11:42:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Apr 2014 13:42:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1150: ------------------------------ Assignee: Bernhard Amann (was: Robin Sommer) > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Bernhard Amann > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 11:43:09 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Apr 2014 13:43:09 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1142) SNMP Analysis In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1142: ------------------------------ Status: Merge Request (was: Open) > SNMP Analysis > ------------- > > Key: BIT-1142 > URL: https://bro-tracker.atlassian.net/browse/BIT-1142 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: BinPAC, Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Seth Hall > Fix For: 2.3 > > > /topic/jsiwek/snmp in bro, binpac, and bro-testing-private adds support for parsing SNMP datagrams. It's only absent a snmp.log. > Seth, do you mind taking a look at what might make sense for a default snmp.log? I'm guessing it might look similar in concept to dns.log. A difference is I'm not sure how meaningful raw OID to value mappings will be. > The code is in a merge-able state as it is in the branch/repos I mentioned, and IMO, has value even without a default snmp.log. So if you just want to flip to a merge request and postpone thinking up an snmp.log for later, I think that's fine, too. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 11:43:09 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Apr 2014 13:43:09 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1142) SNMP Analysis In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1142: ------------------------------ Assignee: Robin Sommer (was: Seth Hall) > SNMP Analysis > ------------- > > Key: BIT-1142 > URL: https://bro-tracker.atlassian.net/browse/BIT-1142 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: BinPAC, Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Robin Sommer > Fix For: 2.3 > > > /topic/jsiwek/snmp in bro, binpac, and bro-testing-private adds support for parsing SNMP datagrams. It's only absent a snmp.log. > Seth, do you mind taking a look at what might make sense for a default snmp.log? I'm guessing it might look similar in concept to dns.log. A difference is I'm not sure how meaningful raw OID to value mappings will be. > The code is in a merge-able state as it is in the branch/repos I mentioned, and IMO, has value even without a default snmp.log. So if you just want to flip to a merge request and postpone thinking up an snmp.log for later, I think that's fine, too. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 11:46:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Apr 2014 13:46:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1137) Investigate sumstats / scan detector performance In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1137: ------------------------------ Resolution: Fixed Status: Closed (was: Open) As there no clear task here., let's close it for now. We did a number of changes to sumstats in the meantime. > Investigate sumstats / scan detector performance > ------------------------------------------------- > > Key: BIT-1137 > URL: https://bro-tracker.atlassian.net/browse/BIT-1137 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Seth Hall > Fix For: 2.3 > > > It's not clear if sumstats is causing more CPU and/or memory load than expected. There's also some indication that it may perform less well in standalone mode than cluster mode. Need to understand and potential improve. > A part of this is also understanding how the new scan detector performs in terms of CPU/memory when compared against the 1.x version. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 11:51:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Apr 2014 13:51:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-845) PF_RING+DNA In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017#comment-16017 ] Robin Sommer commented on BIT-845: ---------------------------------- Still needs documentation. > PF_RING+DNA > ----------- > > Key: BIT-845 > URL: https://bro-tracker.atlassian.net/browse/BIT-845 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: BroControl > Affects Versions: git/master > Reporter: Daniel Thayer > Assignee: Seth Hall > Fix For: 2.3 > > Attachments: lb_pf_ring_dna.py > > > This is a feature that didn't make it into 2.1-beta. > The idea is to have a broctl plugin that has a pre-start > hook to automatically run this on each worker host: > pfdnacluster_master \-i dna0 \-c 21 \-n > A worker entry in node.cfg would look something like this: > [worker-1] > type=worker > host=host1 > interface=dna0 > lb_procs=4 > lb_method=pf_ring_dna -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 11:53:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Apr 2014 13:53:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-845) PF_RING+DNA In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017#comment-16017 ] Robin Sommer edited comment on BIT-845 at 4/4/14 1:52 PM: ---------------------------------------------------------- Still needs documentation (Daniel) was (Author: robin): Still needs documentation. > PF_RING+DNA > ----------- > > Key: BIT-845 > URL: https://bro-tracker.atlassian.net/browse/BIT-845 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: BroControl > Affects Versions: git/master > Reporter: Daniel Thayer > Assignee: Seth Hall > Fix For: 2.3 > > Attachments: lb_pf_ring_dna.py > > > This is a feature that didn't make it into 2.1-beta. > The idea is to have a broctl plugin that has a pre-start > hook to automatically run this on each worker host: > pfdnacluster_master \-i dna0 \-c 21 \-n > A worker entry in node.cfg would look something like this: > [worker-1] > type=worker > host=host1 > interface=dna0 > lb_procs=4 > lb_method=pf_ring_dna -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 16:09:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Apr 2014 18:09:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1179) HTTP messages missing in files.log In-Reply-To: References: Message-ID: Robin Sommer created BIT-1179: --------------------------------- Summary: HTTP messages missing in files.log Key: BIT-1179 URL: https://bro-tracker.atlassian.net/browse/BIT-1179 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Reporter: Robin Sommer Assignee: Jon Siwek Fix For: 2.3 I have a trace with multiple HTTP requests inside a persistent HTTP session. for which only the first two appear in files.log, the remaining ones are missing. Looks like a bug. -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 22:04:07 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Sat, 5 Apr 2014 00:04:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-845) PF_RING+DNA In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-845: -------------------------- Assignee: Daniel Thayer (was: Seth Hall) > PF_RING+DNA > ----------- > > Key: BIT-845 > URL: https://bro-tracker.atlassian.net/browse/BIT-845 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: BroControl > Affects Versions: git/master > Reporter: Daniel Thayer > Assignee: Daniel Thayer > Fix For: 2.3 > > Attachments: lb_pf_ring_dna.py > > > This is a feature that didn't make it into 2.1-beta. > The idea is to have a broctl plugin that has a pre-start > hook to automatically run this on each worker host: > pfdnacluster_master \-i dna0 \-c 21 \-n > A worker entry in node.cfg would look something like this: > [worker-1] > type=worker > host=host1 > interface=dna0 > lb_procs=4 > lb_method=pf_ring_dna -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From jira at bro-tracker.atlassian.net Fri Apr 4 22:05:07 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Sat, 5 Apr 2014 00:05:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-845) PF_RING+DNA In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018#comment-16018 ] Seth Hall commented on BIT-845: ------------------------------- I just merged my latest changes into master and I'm sending an email with my configuration notes that need to be turned into formal documentation. > PF_RING+DNA > ----------- > > Key: BIT-845 > URL: https://bro-tracker.atlassian.net/browse/BIT-845 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: BroControl > Affects Versions: git/master > Reporter: Daniel Thayer > Assignee: Daniel Thayer > Fix For: 2.3 > > Attachments: lb_pf_ring_dna.py > > > This is a feature that didn't make it into 2.1-beta. > The idea is to have a broctl plugin that has a pre-start > hook to automatically run this on each worker host: > pfdnacluster_master \-i dna0 \-c 21 \-n > A worker entry in node.cfg would look something like this: > [worker-1] > type=worker > host=host1 > interface=dna0 > lb_procs=4 > lb_method=pf_ring_dna -- This message was sent by Atlassian JIRA (v6.3-OD-01-067#6307) From noreply at bro.org Sat Apr 5 00:00:15 2014 From: noreply at bro.org (Merge Tracker) Date: Sat, 5 Apr 2014 00:00:15 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404050700.s3570FjF015557@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ------------ ---------- ------------- ---------- -------------------------------------- BIT-1168 [1] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-1142 [2] BinPAC,Bro Jon Siwek Robin Sommer 2014-04-04 2.3 Normal SNMP Analysis Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ --------- ---------- ------------------------------------------------ #4 [3] bro mareq [4] 2014-04-01 Protocol identification heuristics. [5] #1 [6] time-machine mareq [7] 2014-03-19 TM-16: Really skip VLAN header for indexing. [8] [1] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [2] BIT-1142 https://bro-tracker.atlassian.net/browse/BIT-1142 [3] Pull Request #4 https://github.com/bro/bro/pull/4 [4] mareq https://github.com/mareq [5] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [6] Pull Request #1 https://github.com/bro/time-machine/pull/1 [7] mareq https://github.com/mareq [8] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From noreply at bro.org Sun Apr 6 00:00:16 2014 From: noreply at bro.org (Merge Tracker) Date: Sun, 6 Apr 2014 00:00:16 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404060700.s3670GbC020828@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ------------ ---------- ------------- ---------- -------------------------------------- BIT-1168 [1] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-1142 [2] BinPAC,Bro Jon Siwek Robin Sommer 2014-04-04 2.3 Normal SNMP Analysis Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ --------- ---------- ------------------------------------------------ #4 [3] bro mareq [4] 2014-04-01 Protocol identification heuristics. [5] #1 [6] time-machine mareq [7] 2014-03-19 TM-16: Really skip VLAN header for indexing. [8] [1] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [2] BIT-1142 https://bro-tracker.atlassian.net/browse/BIT-1142 [3] Pull Request #4 https://github.com/bro/bro/pull/4 [4] mareq https://github.com/mareq [5] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [6] Pull Request #1 https://github.com/bro/time-machine/pull/1 [7] mareq https://github.com/mareq [8] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From noreply at bro.org Mon Apr 7 00:00:18 2014 From: noreply at bro.org (Merge Tracker) Date: Mon, 7 Apr 2014 00:00:18 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404070700.s3770Iju027449@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ------------ ---------- ------------- ---------- -------------------------------------- BIT-1168 [1] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-1142 [2] BinPAC,Bro Jon Siwek Robin Sommer 2014-04-04 2.3 Normal SNMP Analysis Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ --------- ---------- ------------------------------------------------ #4 [3] bro mareq [4] 2014-04-01 Protocol identification heuristics. [5] #1 [6] time-machine mareq [7] 2014-03-19 TM-16: Really skip VLAN header for indexing. [8] [1] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [2] BIT-1142 https://bro-tracker.atlassian.net/browse/BIT-1142 [3] Pull Request #4 https://github.com/bro/bro/pull/4 [4] mareq https://github.com/mareq [5] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [6] Pull Request #1 https://github.com/bro/time-machine/pull/1 [7] mareq https://github.com/mareq [8] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Mon Apr 7 06:11:08 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Mon, 7 Apr 2014 08:11:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1167) Add subnet support to intel framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100#comment-16100 ] Seth Hall commented on BIT-1167: -------------------------------- Yeah, that'd definitely one of the problems but I think there was something else even more insidious. :) > Add subnet support to intel framework > ------------------------------------- > > Key: BIT-1167 > URL: https://bro-tracker.atlassian.net/browse/BIT-1167 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.2 > Reporter: Brian Little > Priority: Low > Labels: intel, subnet > Attachments: bro-intel-subnet.patch > > > Here is a patch to add Intel::NET data as a type to search on. This allows adding whole subnets to the intel data rather than just individual addresses. > I have also updated the btest. > I'm not sure if the lookup is the best way of doing it - currently if loops through each subnet and then checks if the host is part of each. Is it possible to do it in a more efficient way? -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Mon Apr 7 07:51:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 7 Apr 2014 09:51:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1167) Add subnet support to intel framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101#comment-16101 ] Robin Sommer commented on BIT-1167: ----------------------------------- I can't recall right now what it was either ... > Add subnet support to intel framework > ------------------------------------- > > Key: BIT-1167 > URL: https://bro-tracker.atlassian.net/browse/BIT-1167 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.2 > Reporter: Brian Little > Priority: Low > Labels: intel, subnet > Attachments: bro-intel-subnet.patch > > > Here is a patch to add Intel::NET data as a type to search on. This allows adding whole subnets to the intel data rather than just individual addresses. > I have also updated the btest. > I'm not sure if the lookup is the best way of doing it - currently if loops through each subnet and then checks if the host is part of each. Is it possible to do it in a more efficient way? -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Mon Apr 7 13:44:07 2014 From: jira at bro-tracker.atlassian.net (Aashish Sharma (JIRA)) Date: Mon, 7 Apr 2014 15:44:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1180) Input framework subsiquient REREAD fails after file update In-Reply-To: References: Message-ID: Aashish Sharma created BIT-1180: ----------------------------------- Summary: Input framework subsiquient REREAD fails after file update Key: BIT-1180 URL: https://bro-tracker.atlassian.net/browse/BIT-1180 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.2 Reporter: Aashish Sharma Priority: High I have a file that gets updated every hour and I am using it as a feed into bro using input framework. Every hour I write a list of IP addresses into this file. For many updates everything works fine but Occasionally, I see the following error: Apr 6 05:00:09 Reporter::ERROR /feeds/Blacklist/CURRENT.24hrs_BRO/Input::READER_ASCII: could not read first line (empty) After this failure/message, any subsequent updates on the file are ignored by the input framework. >From visual inspection the file looks just fine and header/data (1 column of IP addresses) is there as expected but somehow input framework doesn't like it. It seems that every hour when update the file using a cron script, on a rare occasion the file is empty for a minuscule duration after which this error starts. for further REREADS data won't get updated into the tables anymore once the above Reporter::ERROR kicks in. Please let me know if you need ways to reproduce this error condition or have more questions for me. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Mon Apr 7 13:49:07 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Mon, 7 Apr 2014 15:49:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1180) Input framework subsiquient REREAD fails after file update In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann reassigned BIT-1180: ----------------------------------- Assignee: Bernhard Amann > Input framework subsiquient REREAD fails after file update > ----------------------------------------------------------- > > Key: BIT-1180 > URL: https://bro-tracker.atlassian.net/browse/BIT-1180 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Aashish Sharma > Assignee: Bernhard Amann > Priority: High > Labels: input-framework > > I have a file that gets updated every hour and I am using it as a feed into bro using input framework. Every hour I write a list of IP addresses into this file. For many updates everything works fine but Occasionally, I see the following error: > Apr 6 05:00:09 Reporter::ERROR /feeds/Blacklist/CURRENT.24hrs_BRO/Input::READER_ASCII: could not read first line (empty) > After this failure/message, any subsequent updates on the file are ignored by the input framework. > From visual inspection the file looks just fine and header/data (1 column of IP addresses) is there as expected but somehow input framework doesn't like it. It seems that every hour when update the file using a cron script, on a rare occasion the file is empty for a minuscule duration after which this error starts. > for further REREADS data won't get updated into the tables anymore once the above Reporter::ERROR kicks in. > Please let me know if you need ways to reproduce this error condition or have more questions for me. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Mon Apr 7 13:50:07 2014 From: jira at bro-tracker.atlassian.net (Aashish Sharma (JIRA)) Date: Mon, 7 Apr 2014 15:50:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1181) Input-framework errors should be fatal (or Notice_Alarm) instead of silent reporter::error failures In-Reply-To: References: Message-ID: Aashish Sharma created BIT-1181: ----------------------------------- Summary: Input-framework errors should be fatal (or Notice_Alarm) instead of silent reporter::error failures Key: BIT-1181 URL: https://bro-tracker.atlassian.net/browse/BIT-1181 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.2 Reporter: Aashish Sharma I noticed many times that if there is a problem in a feed file (syntax, or some other issue) and input-framework is unable to read the file, it generates a Reporter::Error. This is a silent failure condition ie bro continues to operate as normal and the error is logged into reporter log. Ideally above is the right thing to do. However, This failure results in no data in the tables getting updated any more while I continue to operate under-impression that Bro is working fine (unless I have explicitly been looking at reporter log for this issue , which now I do). If input-framework is unable to read/digest data from a feed, I believe that should be a (configurable) fatal error or something which at least triggers an alarm/alert/email. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Mon Apr 7 13:51:07 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Mon, 7 Apr 2014 15:51:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1180) Input framework subsiquient REREAD fails after file update In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102#comment-16102 ] Bernhard Amann commented on BIT-1180: ------------------------------------- I think the way to fix this is to ignore files on re-reads when they are empty (perhaps logging a warning message). Should not be a too difficult change. > Input framework subsiquient REREAD fails after file update > ----------------------------------------------------------- > > Key: BIT-1180 > URL: https://bro-tracker.atlassian.net/browse/BIT-1180 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Aashish Sharma > Assignee: Bernhard Amann > Priority: High > Labels: input-framework > > I have a file that gets updated every hour and I am using it as a feed into bro using input framework. Every hour I write a list of IP addresses into this file. For many updates everything works fine but Occasionally, I see the following error: > Apr 6 05:00:09 Reporter::ERROR /feeds/Blacklist/CURRENT.24hrs_BRO/Input::READER_ASCII: could not read first line (empty) > After this failure/message, any subsequent updates on the file are ignored by the input framework. > From visual inspection the file looks just fine and header/data (1 column of IP addresses) is there as expected but somehow input framework doesn't like it. It seems that every hour when update the file using a cron script, on a rare occasion the file is empty for a minuscule duration after which this error starts. > for further REREADS data won't get updated into the tables anymore once the above Reporter::ERROR kicks in. > Please let me know if you need ways to reproduce this error condition or have more questions for me. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Mon Apr 7 13:52:07 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Mon, 7 Apr 2014 15:52:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1181) Input-framework errors should be fatal (or Notice_Alarm) instead of silent reporter::error failures In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann reassigned BIT-1181: ----------------------------------- Assignee: Bernhard Amann > Input-framework errors should be fatal (or Notice_Alarm) instead of silent reporter::error failures > --------------------------------------------------------------------------------------------------- > > Key: BIT-1181 > URL: https://bro-tracker.atlassian.net/browse/BIT-1181 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Aashish Sharma > Assignee: Bernhard Amann > Labels: input-framework > > I noticed many times that if there is a problem in a feed file (syntax, or some other issue) and input-framework is unable to read the file, it generates a Reporter::Error. This is a silent failure condition ie bro continues to operate as normal and the error is logged into reporter log. > Ideally above is the right thing to do. However, This failure results in no data in the tables getting updated any more while I continue to operate under-impression that Bro is working fine (unless I have explicitly been looking at reporter log for this issue , which now I do). > If input-framework is unable to read/digest data from a feed, I believe that should be a (configurable) fatal error or something which at least triggers an alarm/alert/email. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Mon Apr 7 13:54:07 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Mon, 7 Apr 2014 15:54:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1181) Input-framework errors should be fatal (or Notice_Alarm) instead of silent reporter::error failures In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16103#comment-16103 ] Bernhard Amann commented on BIT-1181: ------------------------------------- Adding an option to make input errors fatal would not be too hard - we also could add an event that is thrown in the case of an error to allow a user to handle stuff like that themselves more gracefully. But I think we should leave the default as non-fatal. Though the errors are easy to miss at the moment. > Input-framework errors should be fatal (or Notice_Alarm) instead of silent reporter::error failures > --------------------------------------------------------------------------------------------------- > > Key: BIT-1181 > URL: https://bro-tracker.atlassian.net/browse/BIT-1181 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Aashish Sharma > Assignee: Bernhard Amann > Labels: input-framework > > I noticed many times that if there is a problem in a feed file (syntax, or some other issue) and input-framework is unable to read the file, it generates a Reporter::Error. This is a silent failure condition ie bro continues to operate as normal and the error is logged into reporter log. > Ideally above is the right thing to do. However, This failure results in no data in the tables getting updated any more while I continue to operate under-impression that Bro is working fine (unless I have explicitly been looking at reporter log for this issue , which now I do). > If input-framework is unable to read/digest data from a feed, I believe that should be a (configurable) fatal error or something which at least triggers an alarm/alert/email. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Mon Apr 7 14:02:07 2014 From: jira at bro-tracker.atlassian.net (Aashish Sharma (JIRA)) Date: Mon, 7 Apr 2014 16:02:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1182) Input-framework thread spwan In-Reply-To: References: Message-ID: Aashish Sharma created BIT-1182: ----------------------------------- Summary: Input-framework thread spwan Key: BIT-1182 URL: https://bro-tracker.atlassian.net/browse/BIT-1182 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.2 Reporter: Aashish Sharma Using the mode REREAD, I noticed that input-framework spawns a thread for every add/change/delete for the elements in the feed file. this is a VERY desired feature and powerful capability and works quite well in general settings. Since, all the changes in a file spawns a thread to process for: EVENT_NEW, EVENT_CHANGED, EVENT_REMOVED, If there are lets say 5000 Changes in the file, there would be 5000 threads spawned at the same time. this is still alright and system can handle load and processing is done in a few seconds. However, if I include a when statement along with exec framework usage to execute an action in Input::EVENT_NEW, Input::EVENT_CHANGED or Input::EVENT_REMOVED - all threads spawned together freezes bro from processing any packets at all. It would be nice if we can serialize this thread creation and spawn only a few at a time. This way we can spread out the increased load over next N mins instead of freezing bro to a standstill. (As always, please let me know if you want code to be able to re-produce this issue). -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Mon Apr 7 14:43:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 7 Apr 2014 16:43:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1181) Input-framework errors should be fatal (or Notice_Alarm) instead of silent reporter::error failures In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16104#comment-16104 ] Robin Sommer commented on BIT-1181: ----------------------------------- I don't think we should turn this into a fatal error, not even optionally. Fatal errors should actually never occur during normal operation. A different suggestion: we already have a reporter_error() event. Seems one could just look out for these errors and turn them into a corresponding notice as part of the site policy. > Input-framework errors should be fatal (or Notice_Alarm) instead of silent reporter::error failures > --------------------------------------------------------------------------------------------------- > > Key: BIT-1181 > URL: https://bro-tracker.atlassian.net/browse/BIT-1181 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Aashish Sharma > Assignee: Bernhard Amann > Labels: input-framework > > I noticed many times that if there is a problem in a feed file (syntax, or some other issue) and input-framework is unable to read the file, it generates a Reporter::Error. This is a silent failure condition ie bro continues to operate as normal and the error is logged into reporter log. > Ideally above is the right thing to do. However, This failure results in no data in the tables getting updated any more while I continue to operate under-impression that Bro is working fine (unless I have explicitly been looking at reporter log for this issue , which now I do). > If input-framework is unable to read/digest data from a feed, I believe that should be a (configurable) fatal error or something which at least triggers an alarm/alert/email. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From noreply at bro.org Tue Apr 8 00:00:19 2014 From: noreply at bro.org (Merge Tracker) Date: Tue, 8 Apr 2014 00:00:19 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404080700.s3870JMF004053@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ------------ ---------- ------------- ---------- -------------------------------------- BIT-1168 [1] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-1142 [2] BinPAC,Bro Jon Siwek Robin Sommer 2014-04-04 2.3 Normal SNMP Analysis Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ --------- ---------- ------------------------------------------------ #4 [3] bro mareq [4] 2014-04-01 Protocol identification heuristics. [5] #1 [6] time-machine mareq [7] 2014-03-19 TM-16: Really skip VLAN header for indexing. [8] [1] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [2] BIT-1142 https://bro-tracker.atlassian.net/browse/BIT-1142 [3] Pull Request #4 https://github.com/bro/bro/pull/4 [4] mareq https://github.com/mareq [5] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [6] Pull Request #1 https://github.com/bro/time-machine/pull/1 [7] mareq https://github.com/mareq [8] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Tue Apr 8 15:48:11 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 8 Apr 2014 17:48:11 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1142) SNMP Analysis In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1142: --------------------------------- Assignee: Seth Hall (was: Robin Sommer) Merged the analyzer. Assigning to Seth for script-level part. > SNMP Analysis > ------------- > > Key: BIT-1142 > URL: https://bro-tracker.atlassian.net/browse/BIT-1142 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: BinPAC, Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Seth Hall > Fix For: 2.3 > > > /topic/jsiwek/snmp in bro, binpac, and bro-testing-private adds support for parsing SNMP datagrams. It's only absent a snmp.log. > Seth, do you mind taking a look at what might make sense for a default snmp.log? I'm guessing it might look similar in concept to dns.log. A difference is I'm not sure how meaningful raw OID to value mappings will be. > The code is in a merge-able state as it is in the branch/repos I mentioned, and IMO, has value even without a default snmp.log. So if you just want to flip to a merge request and postpone thinking up an snmp.log for later, I think that's fine, too. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Tue Apr 8 15:48:11 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 8 Apr 2014 17:48:11 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1142) SNMP Analysis In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1142: ------------------------------ Status: Open (was: Merge Request) > SNMP Analysis > ------------- > > Key: BIT-1142 > URL: https://bro-tracker.atlassian.net/browse/BIT-1142 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: BinPAC, Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Seth Hall > Fix For: 2.3 > > > /topic/jsiwek/snmp in bro, binpac, and bro-testing-private adds support for parsing SNMP datagrams. It's only absent a snmp.log. > Seth, do you mind taking a look at what might make sense for a default snmp.log? I'm guessing it might look similar in concept to dns.log. A difference is I'm not sure how meaningful raw OID to value mappings will be. > The code is in a merge-able state as it is in the branch/repos I mentioned, and IMO, has value even without a default snmp.log. So if you just want to flip to a merge request and postpone thinking up an snmp.log for later, I think that's fine, too. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Tue Apr 8 19:01:07 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Tue, 8 Apr 2014 21:01:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1181) Input-framework errors should be fatal (or Notice_Alarm) instead of silent reporter::error failures In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16106#comment-16106 ] Bernhard Amann commented on BIT-1181: ------------------------------------- Ok. So - basically since, if you want to, you already can catch this case via reporter_error we won't change anything? One problem I see at the moment is, that it might be really easy to miss errors of the input framework. This can mean that feeds do not get updated and no one notices it for an extended period of time. Not sure where I am going with this - I don't really know any better solution. > Input-framework errors should be fatal (or Notice_Alarm) instead of silent reporter::error failures > --------------------------------------------------------------------------------------------------- > > Key: BIT-1181 > URL: https://bro-tracker.atlassian.net/browse/BIT-1181 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Aashish Sharma > Assignee: Bernhard Amann > Labels: input-framework > > I noticed many times that if there is a problem in a feed file (syntax, or some other issue) and input-framework is unable to read the file, it generates a Reporter::Error. This is a silent failure condition ie bro continues to operate as normal and the error is logged into reporter log. > Ideally above is the right thing to do. However, This failure results in no data in the tables getting updated any more while I continue to operate under-impression that Bro is working fine (unless I have explicitly been looking at reporter log for this issue , which now I do). > If input-framework is unable to read/digest data from a feed, I believe that should be a (configurable) fatal error or something which at least triggers an alarm/alert/email. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From noreply at bro.org Wed Apr 9 00:00:15 2014 From: noreply at bro.org (Merge Tracker) Date: Wed, 9 Apr 2014 00:00:15 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404090700.s3970FSt018010@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- -------------------------------------- BIT-1168 [1] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ --------- ---------- ------------------------------------------------ #4 [2] bro mareq [3] 2014-04-01 Protocol identification heuristics. [4] #1 [5] time-machine mareq [6] 2014-03-19 TM-16: Really skip VLAN header for indexing. [7] [1] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [2] Pull Request #4 https://github.com/bro/bro/pull/4 [3] mareq https://github.com/mareq [4] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [5] Pull Request #1 https://github.com/bro/time-machine/pull/1 [6] mareq https://github.com/mareq [7] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Wed Apr 9 07:52:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Wed, 9 Apr 2014 09:52:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1181) Input-framework errors should be fatal (or Notice_Alarm) instead of silent reporter::error failures In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107#comment-16107 ] Robin Sommer commented on BIT-1181: ----------------------------------- We could in principle add an input_framework_error() event or so, and then handle that accordingly. But I'm reluctant to do that because I don't really want want to start given subsystems their own error reporting--the reporter is the attempt to unify such reporting. Maybe we need to extend the reporter interface with more context though. What if it had a "component" parameter telling what part of Bro generated the message. That could be a framework or a specific analyzer. Then one could easily, e.g., escalate all input framework errors. And if one really wanted to turn them into fatal errors, one can call exit() manually at that point. How does that sound? It's a bit work though because the reporter calls need to be adapted throughout Bro ... > Input-framework errors should be fatal (or Notice_Alarm) instead of silent reporter::error failures > --------------------------------------------------------------------------------------------------- > > Key: BIT-1181 > URL: https://bro-tracker.atlassian.net/browse/BIT-1181 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Aashish Sharma > Assignee: Bernhard Amann > Labels: input-framework > > I noticed many times that if there is a problem in a feed file (syntax, or some other issue) and input-framework is unable to read the file, it generates a Reporter::Error. This is a silent failure condition ie bro continues to operate as normal and the error is logged into reporter log. > Ideally above is the right thing to do. However, This failure results in no data in the tables getting updated any more while I continue to operate under-impression that Bro is working fine (unless I have explicitly been looking at reporter log for this issue , which now I do). > If input-framework is unable to read/digest data from a feed, I believe that should be a (configurable) fatal error or something which at least triggers an alarm/alert/email. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Wed Apr 9 10:12:07 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Wed, 9 Apr 2014 12:12:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1178) SSL/TLS analyzer does not abort early enough on non-ssl connections In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108#comment-16108 ] Bernhard Amann commented on BIT-1178: ------------------------------------- I have looked at this a bit yesterday and... the TLS analyzer has a few other small problems that I would like to fix before the next release. To make a short list, so I don't forget anything: * DPD might not work reliably with TLSv1.2 * The TLS state machine does not work reliably with anything that is not plain HTTPS. Even in HTTPS cases it will not always manage the full transition to encrypted state. This also means that the ssl-established event will not fire in a quite bit chunk of cases where the TLS connection already is established * Due to state machine issues a few other events misfire, we can e.g. get alert events with encrypted alert values that make no sense * The TLS protocol version detection in the analyzer seems to be broken-ish. I have quite a few traces that are not categorized correctly as TLS. This is a problem in master at the moment, because the tls connection will more or less be ignored. * Not aborting early enough for some connections also is a Problem. > SSL/TLS analyzer does not abort early enough on non-ssl connections > ------------------------------------------------------------------- > > Key: BIT-1178 > URL: https://bro-tracker.atlassian.net/browse/BIT-1178 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Bernhard Amann > Fix For: 2.3 > > > Some sites see quite a bit of non-ssl traffic on port 443. At the moment, the SSL analyzer manages to parse quite a lot of this non-ssl traffic and generates ssl-events for it (including sending things to the certificate analyzer which generates warnings). > We should probably try to abort much earlier. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Wed Apr 9 12:36:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 9 Apr 2014 14:36:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-348) Reassembler integer overflow issues. Data not delivered after 2GB In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109#comment-16109 ] Jon Siwek commented on BIT-348: ------------------------------- Addressed in topic/jsiwek/bit-348 branch in bro and bro-testing-private repos. The test baselines change little so that's a good sign, however, I did end up touching a decent amount of code which may make it better to postpone for a later release. IMO, keeping it scheduled for 2.3 is probably the best way to get the new coded tested, but I'm fine if others judge differently. Also I think I kind of botched the commit message -- this change doesn't really "fix" reassembly to deliver data after 2GB (which mostly worked before despite the int overflows), it more just makes the reassembly not depend on "int" to be 32-bit and for "seq_delta" to provide correct sequence space arithmetic despite overflows of that storage unit. That is, the reassembler code now deals in a 64-bit relative sequence space and it's up to the TCP analyzer to track the 32-bit sequence space and translate in to the 64-bit relative sequence space when passing data on for reassembly. Robin: good luck w/ the code review! I ended up refactoring more than I probably needed to in TCP.cc, but I found it helpful to untangle some of what was going on there. > Reassembler integer overflow issues. Data not delivered after 2GB > ----------------------------------------------------------------- > > Key: BIT-348 > URL: https://bro-tracker.atlassian.net/browse/BIT-348 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: gregor > Assignee: Jon Siwek > Priority: High > Labels: inttypes > Fix For: 2.3 > > > {noformat} > #!rst > The TCP Reassembler does not deliver any data to analyzers after the first 2GB due to signed integer overflow (Actually it will deliver again between 4--6GB, etc.) This happens silently, i.e., without content_gap events or Undelivered calls. > This report superseded BIT-315, BIT-137 > The TCP Reassembler (and Reassem) base class use ``int`` to keep track of sequence numbers and ``seq_delta`` to check for differences. If a connection exceeds 2GB, the relative sequence numbers (int) used by the Reassembler become negative. While many parts of the Reassembler still work (because seq_delta still reports the correct difference) some parts do not. In particular ``seq_to_skip`` is broken (and fails silently). There might well be other parts of the Reassembler that fail > silently as well, that I haven't found yet. > See Comments in TCP_Reassembler.cc for more details. > The Reassembler should use int64. However this will require deep changes to the Reassembler and the TCP Analyzer and TCP_Endpoint classes (since we also store sequence numbers there). Also, the analyzer framework will need tweaks as well (e.g., Undelivered uses ``int`` for sequence numbers, also has to go to 64 bit) > As a hotfix that seems to work I disabled the ``seq_to_skip`` features. It wasn't used by any analyzer or policy script (Note, that seq_to_skip is different from skip_deliveries). Hotfix is in > topic/gregor/reassembler-hotfix > {noformat} -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Wed Apr 9 12:36:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 9 Apr 2014 14:36:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-348) Reassembler integer overflow issues. Data not delivered after 2GB In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-348: -------------------------- Status: Merge Request (was: Open) > Reassembler integer overflow issues. Data not delivered after 2GB > ----------------------------------------------------------------- > > Key: BIT-348 > URL: https://bro-tracker.atlassian.net/browse/BIT-348 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: gregor > Assignee: Jon Siwek > Priority: High > Labels: inttypes > Fix For: 2.3 > > > {noformat} > #!rst > The TCP Reassembler does not deliver any data to analyzers after the first 2GB due to signed integer overflow (Actually it will deliver again between 4--6GB, etc.) This happens silently, i.e., without content_gap events or Undelivered calls. > This report superseded BIT-315, BIT-137 > The TCP Reassembler (and Reassem) base class use ``int`` to keep track of sequence numbers and ``seq_delta`` to check for differences. If a connection exceeds 2GB, the relative sequence numbers (int) used by the Reassembler become negative. While many parts of the Reassembler still work (because seq_delta still reports the correct difference) some parts do not. In particular ``seq_to_skip`` is broken (and fails silently). There might well be other parts of the Reassembler that fail > silently as well, that I haven't found yet. > See Comments in TCP_Reassembler.cc for more details. > The Reassembler should use int64. However this will require deep changes to the Reassembler and the TCP Analyzer and TCP_Endpoint classes (since we also store sequence numbers there). Also, the analyzer framework will need tweaks as well (e.g., Undelivered uses ``int`` for sequence numbers, also has to go to 64 bit) > As a hotfix that seems to work I disabled the ``seq_to_skip`` features. It wasn't used by any analyzer or policy script (Note, that seq_to_skip is different from skip_deliveries). Hotfix is in > topic/gregor/reassembler-hotfix > {noformat} -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Wed Apr 9 13:32:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 9 Apr 2014 15:32:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1179) HTTP messages missing in files.log In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110#comment-16110 ] Jon Siwek commented on BIT-1179: -------------------------------- There's a missing TCP segment in the middle of that pcap that looks like it would have contained an HTTP reply. And the thing about the HTTP analyzer seems to be that it stops parsing the rest of the connection if there's a gap that's not isolated to an HTTP message body. So two files end up being pushed from the HTTP analyzer over to the file analysis stuff, then the HTTP analyzer stops parsing anything else due to the missing TCP segment. Since that seems intentional and it's an HTTP analysis limitation not a file analysis bug, think there's anything to do here right now? > HTTP messages missing in files.log > ---------------------------------- > > Key: BIT-1179 > URL: https://bro-tracker.atlassian.net/browse/BIT-1179 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > I have a trace with multiple HTTP requests inside a persistent HTTP session. for which only the first two appear in files.log, the remaining ones are missing. Looks like a bug. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From noreply at bro.org Thu Apr 10 00:00:14 2014 From: noreply at bro.org (Merge Tracker) Date: Thu, 10 Apr 2014 00:00:14 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404100700.s3A70Ebv011852@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1168 [1] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-348 [2] Bro gregor Jon Siwek 2014-04-09 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ----------------------------------------------- 1b75267 [3] broctl Daniel Thayer 2014-04-09 Update test baselines, and minor code cleanup c617be6 [4] bro Jon Siwek 2014-04-09 Remove unused data member of SMTP_Analyzer. d4ef9f3 [5] bro Jon Siwek 2014-04-09 Fix missing @load dependencies in some scripts. Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ---------- ---------- -------------------------------------------------------- #4 [6] bro mareq [7] 2014-04-01 Protocol identification heuristics. [8] #2 [9] time-machine mareq [10] 2014-04-09 Query interval start/end is now taken into account. [11] #1 [12] time-machine mareq [13] 2014-03-19 TM-16: Really skip VLAN header for indexing. [14] [1] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [2] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [3] 1b75267 https://github.com/bro/broctl/commit/1b7526729c9f72448bb17c44959c86549398b755 [4] c617be6 https://github.com/bro/bro/commit/c617be6f50d88f8771a238799e60c37179f9c982 [5] d4ef9f3 https://github.com/bro/bro/commit/d4ef9f36930168574b78d548916c6f8eb7a16ce3 [6] Pull Request #4 https://github.com/bro/bro/pull/4 [7] mareq https://github.com/mareq [8] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [9] Pull Request #2 https://github.com/bro/time-machine/pull/2 [10] mareq https://github.com/mareq [11] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [12] Pull Request #1 https://github.com/bro/time-machine/pull/1 [13] mareq https://github.com/mareq [14] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Thu Apr 10 03:17:07 2014 From: jira at bro-tracker.atlassian.net (Marek Balint (JIRA)) Date: Thu, 10 Apr 2014 05:17:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (TM-16) Index not working when traffic encapsulated in 802.1q trunk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/TM-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111#comment-16111 ] Marek Balint commented on TM-16: -------------------------------- Hi Tyler, I have found one more problem with VLAN headers ( solution is here: https://github.com/bro/time-machine/pull/3 ). It might or might not solve your problem, but it might be worth to give it a try... .mq. > Index not working when traffic encapsulated in 802.1q trunk > ----------------------------------------------------------- > > Key: TM-16 > URL: https://bro-tracker.atlassian.net/browse/TM-16 > Project: Time Machine > Issue Type: Problem > Affects Versions: git/master > Environment: Ubuntu 10.04 , pf_ring > Reporter: tyler.schoenke > Assignee: Seth Hall > Labels: 802.1Q, indexes > Attachments: tm-16.patch > > > Hi All, > When I query the time machine index, I am not receiving any results. > I just restarted time machine, and checked one of the recent class files to see there is traffic for a particular IP address. > tcpdump -e -v -n -r class_all_1385406639.023206 "vlan and host 128.138.44.198" > It shows some traffic, example: > 128.138.44.198.54014 > 74.125.225.209.443: Flags [.], cksum 0x8d2c (correct), seq 1283940799:1283940800, ack 615539104, win 16311, length 1 > 19:11:00.571731632 10:8c:cf:57:46:00 > 00:1d:09:6a:d9:a9, ethertype 802.1Q (0x8100), length 70: vlan 987, p 0, ethertype IPv4, (tos 0x0, ttl 56, id 17482, offset 0, flags [none], proto TCP (6), length 52) > When I telnet localhost 42042 and run the following command, I don't receive any results. > query to_file "128.138.44.198.pcap" index ip "128.138.44.198" > In the above tcpdump, you can see my traffic is 802.1Q trunked. I have to use the "vlan" BPF to extract it with tcpdump, and am wondering if the trunking is causing problems with indexing? > I tested the same version of time machine on non-trunked traffic, and the index works fine. > Let me know if you need any other configuration info. > Tyler -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 10 08:30:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 10 Apr 2014 10:30:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1179) HTTP messages missing in files.log In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112#comment-16112 ] Robin Sommer commented on BIT-1179: ----------------------------------- Agree, if that's indeed the reason, it's nothing to fix. Though the client side of those requests still appears in http.log, are they parsed before the gap comes on the server side? Also, I noticed this when comparing against output of when using the BinPAC++ HTTP analyzer; with that one, they all get reported in files.log. However, that one doesn't deal with gaps either so not sure how that comes. I'll take another look later. Robin > HTTP messages missing in files.log > ---------------------------------- > > Key: BIT-1179 > URL: https://bro-tracker.atlassian.net/browse/BIT-1179 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > I have a trace with multiple HTTP requests inside a persistent HTTP session. for which only the first two appear in files.log, the remaining ones are missing. Looks like a bug. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 10 08:50:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 10 Apr 2014 10:50:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1179) HTTP messages missing in files.log In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113#comment-16113 ] Jon Siwek commented on BIT-1179: -------------------------------- {quote} Though the client side of those requests still appears in http.log, are they parsed before the gap comes on the server side? {quote} A gap from the server's side just disables further parsing of responses, but any more requests will still be parsed. HTTP.cc:1158 is where this happens. And though it doesn't apply to this case, the code for dealing with a gap in a request seems fishy as the comment implies an intention that doesn't match what the code actually does and the code has an extra {{content_line->SetSkipDeliveries(1)}} statement which was probably meant to be invoked on the other side's {{ContentLine_Analyzer}}. > HTTP messages missing in files.log > ---------------------------------- > > Key: BIT-1179 > URL: https://bro-tracker.atlassian.net/browse/BIT-1179 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > I have a trace with multiple HTTP requests inside a persistent HTTP session. for which only the first two appear in files.log, the remaining ones are missing. Looks like a bug. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 10 09:08:07 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 10 Apr 2014 11:08:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1179) HTTP messages missing in files.log In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114#comment-16114 ] Seth Hall commented on BIT-1179: -------------------------------- This has been a long standing issue that we've always kind of deferred on so we just left the HTTP analyzer as one that can't tolerate packet loss. We've discussed some ways of making this "re-synchronization" support a general concept in binpac++ which probably makes the most sense. I'm really not sure there is much point in dumping lots of time in dealing with the current http analyzer since the analyzer has had this behavior for a long time. That said, if it's an easy change and unlikely to break stuff too badly it would then make sense to do it. :P If we changed this, it's possible that we'd have to revisit the base http scripts too to make sure they can cope with re-synchronization appropriately. > HTTP messages missing in files.log > ---------------------------------- > > Key: BIT-1179 > URL: https://bro-tracker.atlassian.net/browse/BIT-1179 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > I have a trace with multiple HTTP requests inside a persistent HTTP session. for which only the first two appear in files.log, the remaining ones are missing. Looks like a bug. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From noreply at bro.org Fri Apr 11 00:00:14 2014 From: noreply at bro.org (Merge Tracker) Date: Fri, 11 Apr 2014 00:00:14 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404110700.s3B70ELf000770@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1168 [1] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-348 [2] Bro gregor Jon Siwek 2014-04-09 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ---------- ---------- --------------------------------------------------------------------------- #4 [3] bro mareq [4] 2014-04-01 Protocol identification heuristics. [5] #4 [6] time-machine mareq [7] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [8] #3 [9] time-machine mareq [10] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [11] #2 [12] time-machine mareq [13] 2014-04-09 Query interval start/end is now taken into account. [14] #1 [15] time-machine mareq [16] 2014-03-19 TM-16: Really skip VLAN header for indexing. [17] [1] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [2] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [3] Pull Request #4 https://github.com/bro/bro/pull/4 [4] mareq https://github.com/mareq [5] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [6] Pull Request #4 https://github.com/bro/time-machine/pull/4 [7] mareq https://github.com/mareq [8] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [9] Pull Request #3 https://github.com/bro/time-machine/pull/3 [10] mareq https://github.com/mareq [11] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [12] Pull Request #2 https://github.com/bro/time-machine/pull/2 [13] mareq https://github.com/mareq [14] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [15] Pull Request #1 https://github.com/bro/time-machine/pull/1 [16] mareq https://github.com/mareq [17] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From noreply at bro.org Sat Apr 12 00:00:16 2014 From: noreply at bro.org (Merge Tracker) Date: Sat, 12 Apr 2014 00:00:16 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404120700.s3C70G7E012943@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1168 [1] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-348 [2] Bro gregor Jon Siwek 2014-04-09 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ------------- ---------- ---------------------------------------------------------------------------- #5 [3] bro colonelxc [4] 2014-04-12 BloomFilter.cc fix to Clear() [5] #4 [6] bro mareq [7] 2014-04-01 Protocol identification heuristics. [8] #4 [9] time-machine mareq [10] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [11] #3 [12] time-machine mareq [13] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [14] #2 [15] time-machine mareq [16] 2014-04-09 Query interval start/end is now taken into account. [17] #1 [18] time-machine mareq [19] 2014-03-19 TM-16: Really skip VLAN header for indexing. [20] [1] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [2] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [3] Pull Request #5 https://github.com/bro/bro/pull/5 [4] colonelxc https://github.com/colonelxc [5] Merge Pull Request #5 with git pull https://github.com/colonelxc/bro.git master [6] Pull Request #4 https://github.com/bro/bro/pull/4 [7] mareq https://github.com/mareq [8] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [9] Pull Request #4 https://github.com/bro/time-machine/pull/4 [10] mareq https://github.com/mareq [11] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [12] Pull Request #3 https://github.com/bro/time-machine/pull/3 [13] mareq https://github.com/mareq [14] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [15] Pull Request #2 https://github.com/bro/time-machine/pull/2 [16] mareq https://github.com/mareq [17] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [18] Pull Request #1 https://github.com/bro/time-machine/pull/1 [19] mareq https://github.com/mareq [20] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From noreply at bro.org Sun Apr 13 00:00:22 2014 From: noreply at bro.org (Merge Tracker) Date: Sun, 13 Apr 2014 00:00:22 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404130700.s3D70MLD000956@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1168 [1] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-348 [2] Bro gregor Jon Siwek 2014-04-09 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ------------- ---------- ---------------------------------------------------------------------------- #5 [3] bro colonelxc [4] 2014-04-12 BloomFilter.cc fix to Clear() [5] #4 [6] bro mareq [7] 2014-04-01 Protocol identification heuristics. [8] #4 [9] time-machine mareq [10] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [11] #3 [12] time-machine mareq [13] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [14] #2 [15] time-machine mareq [16] 2014-04-09 Query interval start/end is now taken into account. [17] #1 [18] time-machine mareq [19] 2014-03-19 TM-16: Really skip VLAN header for indexing. [20] [1] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [2] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [3] Pull Request #5 https://github.com/bro/bro/pull/5 [4] colonelxc https://github.com/colonelxc [5] Merge Pull Request #5 with git pull https://github.com/colonelxc/bro.git master [6] Pull Request #4 https://github.com/bro/bro/pull/4 [7] mareq https://github.com/mareq [8] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [9] Pull Request #4 https://github.com/bro/time-machine/pull/4 [10] mareq https://github.com/mareq [11] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [12] Pull Request #3 https://github.com/bro/time-machine/pull/3 [13] mareq https://github.com/mareq [14] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [15] Pull Request #2 https://github.com/bro/time-machine/pull/2 [16] mareq https://github.com/mareq [17] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [18] Pull Request #1 https://github.com/bro/time-machine/pull/1 [19] mareq https://github.com/mareq [20] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From noreply at bro.org Mon Apr 14 00:00:22 2014 From: noreply at bro.org (Merge Tracker) Date: Mon, 14 Apr 2014 00:00:22 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404140700.s3E70ME1013375@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1168 [1] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-348 [2] Bro gregor Jon Siwek 2014-04-09 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ------------- ---------- ---------------------------------------------------------------------------- #5 [3] bro colonelxc [4] 2014-04-12 BloomFilter.cc fix to Clear() [5] #4 [6] bro mareq [7] 2014-04-01 Protocol identification heuristics. [8] #4 [9] time-machine mareq [10] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [11] #3 [12] time-machine mareq [13] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [14] #2 [15] time-machine mareq [16] 2014-04-09 Query interval start/end is now taken into account. [17] #1 [18] time-machine mareq [19] 2014-03-19 TM-16: Really skip VLAN header for indexing. [20] [1] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [2] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [3] Pull Request #5 https://github.com/bro/bro/pull/5 [4] colonelxc https://github.com/colonelxc [5] Merge Pull Request #5 with git pull https://github.com/colonelxc/bro.git master [6] Pull Request #4 https://github.com/bro/bro/pull/4 [7] mareq https://github.com/mareq [8] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [9] Pull Request #4 https://github.com/bro/time-machine/pull/4 [10] mareq https://github.com/mareq [11] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [12] Pull Request #3 https://github.com/bro/time-machine/pull/3 [13] mareq https://github.com/mareq [14] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [15] Pull Request #2 https://github.com/bro/time-machine/pull/2 [16] mareq https://github.com/mareq [17] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [18] Pull Request #1 https://github.com/bro/time-machine/pull/1 [19] mareq https://github.com/mareq [20] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From noreply at bro.org Tue Apr 15 00:00:18 2014 From: noreply at bro.org (Merge Tracker) Date: Tue, 15 Apr 2014 00:00:18 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404150700.s3F70IRt001160@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1168 [1] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-348 [2] Bro gregor Jon Siwek 2014-04-09 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ------------- ---------- ---------------------------------------------------------------------------- #5 [3] bro colonelxc [4] 2014-04-14 BloomFilter.cc fix to Clear() [5] #4 [6] bro mareq [7] 2014-04-01 Protocol identification heuristics. [8] #4 [9] time-machine mareq [10] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [11] #3 [12] time-machine mareq [13] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [14] #2 [15] time-machine mareq [16] 2014-04-09 Query interval start/end is now taken into account. [17] #1 [18] time-machine mareq [19] 2014-03-19 TM-16: Really skip VLAN header for indexing. [20] [1] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [2] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [3] Pull Request #5 https://github.com/bro/bro/pull/5 [4] colonelxc https://github.com/colonelxc [5] Merge Pull Request #5 with git pull https://github.com/colonelxc/bro.git master [6] Pull Request #4 https://github.com/bro/bro/pull/4 [7] mareq https://github.com/mareq [8] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [9] Pull Request #4 https://github.com/bro/time-machine/pull/4 [10] mareq https://github.com/mareq [11] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [12] Pull Request #3 https://github.com/bro/time-machine/pull/3 [13] mareq https://github.com/mareq [14] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [15] Pull Request #2 https://github.com/bro/time-machine/pull/2 [16] mareq https://github.com/mareq [17] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [18] Pull Request #1 https://github.com/bro/time-machine/pull/1 [19] mareq https://github.com/mareq [20] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Tue Apr 15 11:11:08 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 15 Apr 2014 13:11:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1183) topic/jsiwek/ascii-log-memleak-fix In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1183: --------------------------- Status: Merge Request (was: Open) > topic/jsiwek/ascii-log-memleak-fix > ---------------------------------- > > Key: BIT-1183 > URL: https://bro-tracker.atlassian.net/browse/BIT-1183 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch fixes a memory leak in the ASCII log writer that occurs after each rotation. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Tue Apr 15 11:11:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 15 Apr 2014 13:11:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1183) topic/jsiwek/ascii-log-memleak-fix In-Reply-To: References: Message-ID: Jon Siwek created BIT-1183: ------------------------------ Summary: topic/jsiwek/ascii-log-memleak-fix Key: BIT-1183 URL: https://bro-tracker.atlassian.net/browse/BIT-1183 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: git/master Reporter: Jon Siwek Fix For: 2.3 This branch fixes a memory leak in the ASCII log writer that occurs after each rotation. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From noreply at bro.org Wed Apr 16 00:00:16 2014 From: noreply at bro.org (Merge Tracker) Date: Wed, 16 Apr 2014 00:00:16 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404160700.s3G70GuG021453@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1183 [1] Bro Jon Siwek - 2014-04-15 2.3 Normal topic/jsiwek/ascii-log-memleak-fix [2] BIT-1168 [3] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-348 [4] Bro gregor Jon Siwek 2014-04-09 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------------ ---------- ------------------------------------------------ c9b40f1 [5] bro Jon Siwek 2014-04-15 Change how input/logging threads set their name. cb4eaf7 [6] bro Matthias Vallentin 2014-04-15 Fix bug when clearing Bloom filter contents. Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ----------- ---------- ---------------------------------------------------------------------------- #6 [7] bro jshlbrd [8] 2014-04-15 Intel::ADDR indicators in http host field [9] #4 [10] bro mareq [11] 2014-04-01 Protocol identification heuristics. [12] #4 [13] time-machine mareq [14] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [15] #3 [16] time-machine mareq [17] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [18] #2 [19] time-machine mareq [20] 2014-04-09 Query interval start/end is now taken into account. [21] #1 [22] time-machine mareq [23] 2014-03-19 TM-16: Really skip VLAN header for indexing. [24] [1] BIT-1183 https://bro-tracker.atlassian.net/browse/BIT-1183 [2] ascii-log-memleak-fix https://github.com/bro/bro/tree/topic/jsiwek/ascii-log-memleak-fix [3] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [4] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [5] c9b40f1 https://github.com/bro/bro/commit/c9b40f1ca7bdbf9c0cb799477e695b93fcf3ba04 [6] cb4eaf7 https://github.com/bro/bro/commit/cb4eaf762c967aaede75a5da8ace06eca508f82e [7] Pull Request #6 https://github.com/bro/bro/pull/6 [8] jshlbrd https://github.com/jshlbrd [9] Merge Pull Request #6 with git pull https://github.com/jshlbrd/bro.git master [10] Pull Request #4 https://github.com/bro/bro/pull/4 [11] mareq https://github.com/mareq [12] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [13] Pull Request #4 https://github.com/bro/time-machine/pull/4 [14] mareq https://github.com/mareq [15] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [16] Pull Request #3 https://github.com/bro/time-machine/pull/3 [17] mareq https://github.com/mareq [18] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [19] Pull Request #2 https://github.com/bro/time-machine/pull/2 [20] mareq https://github.com/mareq [21] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [22] Pull Request #1 https://github.com/bro/time-machine/pull/1 [23] mareq https://github.com/mareq [24] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From noreply at bro.org Thu Apr 17 00:00:15 2014 From: noreply at bro.org (Merge Tracker) Date: Thu, 17 Apr 2014 00:00:15 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404170700.s3H70FeP015482@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1183 [1] Bro Jon Siwek - 2014-04-15 2.3 Normal topic/jsiwek/ascii-log-memleak-fix [2] BIT-1168 [3] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-348 [4] Bro gregor Jon Siwek 2014-04-09 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------------ ---------- ------------------------------------------------ c9b40f1 [5] bro Jon Siwek 2014-04-15 Change how input/logging threads set their name. cb4eaf7 [6] bro Matthias Vallentin 2014-04-15 Fix bug when clearing Bloom filter contents. Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ----------- ---------- ---------------------------------------------------------------------------- #6 [7] bro jshlbrd [8] 2014-04-15 Intel::ADDR indicators in http host field [9] #4 [10] bro mareq [11] 2014-04-01 Protocol identification heuristics. [12] #4 [13] time-machine mareq [14] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [15] #3 [16] time-machine mareq [17] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [18] #2 [19] time-machine mareq [20] 2014-04-09 Query interval start/end is now taken into account. [21] #1 [22] time-machine mareq [23] 2014-03-19 TM-16: Really skip VLAN header for indexing. [24] [1] BIT-1183 https://bro-tracker.atlassian.net/browse/BIT-1183 [2] ascii-log-memleak-fix https://github.com/bro/bro/tree/topic/jsiwek/ascii-log-memleak-fix [3] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [4] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [5] c9b40f1 https://github.com/bro/bro/commit/c9b40f1ca7bdbf9c0cb799477e695b93fcf3ba04 [6] cb4eaf7 https://github.com/bro/bro/commit/cb4eaf762c967aaede75a5da8ace06eca508f82e [7] Pull Request #6 https://github.com/bro/bro/pull/6 [8] jshlbrd https://github.com/jshlbrd [9] Merge Pull Request #6 with git pull https://github.com/jshlbrd/bro.git master [10] Pull Request #4 https://github.com/bro/bro/pull/4 [11] mareq https://github.com/mareq [12] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [13] Pull Request #4 https://github.com/bro/time-machine/pull/4 [14] mareq https://github.com/mareq [15] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [16] Pull Request #3 https://github.com/bro/time-machine/pull/3 [17] mareq https://github.com/mareq [18] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [19] Pull Request #2 https://github.com/bro/time-machine/pull/2 [20] mareq https://github.com/mareq [21] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [22] Pull Request #1 https://github.com/bro/time-machine/pull/1 [23] mareq https://github.com/mareq [24] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Thu Apr 17 09:21:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 17 Apr 2014 11:21:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1183) topic/jsiwek/ascii-log-memleak-fix In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200#comment-16200 ] Jon Siwek commented on BIT-1183: -------------------------------- The second commit on this branch I think should resolve problems people have w/ increased memory usage over time on the manager. The problem was this line: https://github.com/bro/bro/blob/a56c3437151985a1d0e4c881c047796109fbd81d/src/logging/writers/Ascii.cc#L158 At each rotation, that will add a new string that the Desc object has to check for when escaping strings. Over time, the cost of escaping things in the ASCII logs will increase until the rate at which logs can be formatted/written is lower than the rate at which logs are produced. Once that happens, Bro is unlikely to be able to catch up on the pending logs and so runs out of memory. > topic/jsiwek/ascii-log-memleak-fix > ---------------------------------- > > Key: BIT-1183 > URL: https://bro-tracker.atlassian.net/browse/BIT-1183 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch fixes a memory leak in the ASCII log writer that occurs after each rotation. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 17 21:25:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 17 Apr 2014 23:25:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-348) Reassembler integer overflow issues. Data not delivered after 2GB In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201#comment-16201 ] Robin Sommer commented on BIT-348: ---------------------------------- I took a quick a look. This looks like the right approach and I would like to get it into 2.3. It will take me a bit more to go through in more detail. In the meantime, could you do some additional testing comparing output before and after on a trace of live traffic, including in particular traffic with some large flows that exercise the TCP wrap-around? > Reassembler integer overflow issues. Data not delivered after 2GB > ----------------------------------------------------------------- > > Key: BIT-348 > URL: https://bro-tracker.atlassian.net/browse/BIT-348 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: gregor > Assignee: Jon Siwek > Priority: High > Labels: inttypes > Fix For: 2.3 > > > {noformat} > #!rst > The TCP Reassembler does not deliver any data to analyzers after the first 2GB due to signed integer overflow (Actually it will deliver again between 4--6GB, etc.) This happens silently, i.e., without content_gap events or Undelivered calls. > This report superseded BIT-315, BIT-137 > The TCP Reassembler (and Reassem) base class use ``int`` to keep track of sequence numbers and ``seq_delta`` to check for differences. If a connection exceeds 2GB, the relative sequence numbers (int) used by the Reassembler become negative. While many parts of the Reassembler still work (because seq_delta still reports the correct difference) some parts do not. In particular ``seq_to_skip`` is broken (and fails silently). There might well be other parts of the Reassembler that fail > silently as well, that I haven't found yet. > See Comments in TCP_Reassembler.cc for more details. > The Reassembler should use int64. However this will require deep changes to the Reassembler and the TCP Analyzer and TCP_Endpoint classes (since we also store sequence numbers there). Also, the analyzer framework will need tweaks as well (e.g., Undelivered uses ``int`` for sequence numbers, also has to go to 64 bit) > As a hotfix that seems to work I disabled the ``seq_to_skip`` features. It wasn't used by any analyzer or policy script (Note, that seq_to_skip is different from skip_deliveries). Hotfix is in > topic/gregor/reassembler-hotfix > {noformat} -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 17 21:46:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 17 Apr 2014 23:46:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1183) topic/jsiwek/ascii-log-memleak-fix In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1183: ------------------------------ Status: Closed (was: Merge Request) > topic/jsiwek/ascii-log-memleak-fix > ---------------------------------- > > Key: BIT-1183 > URL: https://bro-tracker.atlassian.net/browse/BIT-1183 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch fixes a memory leak in the ASCII log writer that occurs after each rotation. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From noreply at bro.org Fri Apr 18 00:00:13 2014 From: noreply at bro.org (Merge Tracker) Date: Fri, 18 Apr 2014 00:00:13 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404180700.s3I70DHR017010@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1168 [1] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-348 [2] Bro gregor Jon Siwek 2014-04-17 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ----------- ---------- ---------------------------------------------------------------------------- #6 [3] bro jshlbrd [4] 2014-04-15 Intel::ADDR indicators in http host field [5] #4 [6] bro mareq [7] 2014-04-01 Protocol identification heuristics. [8] #4 [9] time-machine mareq [10] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [11] #3 [12] time-machine mareq [13] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [14] #2 [15] time-machine mareq [16] 2014-04-09 Query interval start/end is now taken into account. [17] #1 [18] time-machine mareq [19] 2014-03-19 TM-16: Really skip VLAN header for indexing. [20] [1] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [2] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [3] Pull Request #6 https://github.com/bro/bro/pull/6 [4] jshlbrd https://github.com/jshlbrd [5] Merge Pull Request #6 with git pull https://github.com/jshlbrd/bro.git master [6] Pull Request #4 https://github.com/bro/bro/pull/4 [7] mareq https://github.com/mareq [8] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [9] Pull Request #4 https://github.com/bro/time-machine/pull/4 [10] mareq https://github.com/mareq [11] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [12] Pull Request #3 https://github.com/bro/time-machine/pull/3 [13] mareq https://github.com/mareq [14] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [15] Pull Request #2 https://github.com/bro/time-machine/pull/2 [16] mareq https://github.com/mareq [17] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [18] Pull Request #1 https://github.com/bro/time-machine/pull/1 [19] mareq https://github.com/mareq [20] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Fri Apr 18 07:34:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 18 Apr 2014 09:34:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-348) Reassembler integer overflow issues. Data not delivered after 2GB In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16202#comment-16202 ] Jon Siwek commented on BIT-348: ------------------------------- For testing the TCP wrap-around, I had a trace of a ~5GB file downloaded via FTP and that seemed to be handled alright AFAICT. > Reassembler integer overflow issues. Data not delivered after 2GB > ----------------------------------------------------------------- > > Key: BIT-348 > URL: https://bro-tracker.atlassian.net/browse/BIT-348 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: gregor > Assignee: Jon Siwek > Priority: High > Labels: inttypes > Fix For: 2.3 > > > {noformat} > #!rst > The TCP Reassembler does not deliver any data to analyzers after the first 2GB due to signed integer overflow (Actually it will deliver again between 4--6GB, etc.) This happens silently, i.e., without content_gap events or Undelivered calls. > This report superseded BIT-315, BIT-137 > The TCP Reassembler (and Reassem) base class use ``int`` to keep track of sequence numbers and ``seq_delta`` to check for differences. If a connection exceeds 2GB, the relative sequence numbers (int) used by the Reassembler become negative. While many parts of the Reassembler still work (because seq_delta still reports the correct difference) some parts do not. In particular ``seq_to_skip`` is broken (and fails silently). There might well be other parts of the Reassembler that fail > silently as well, that I haven't found yet. > See Comments in TCP_Reassembler.cc for more details. > The Reassembler should use int64. However this will require deep changes to the Reassembler and the TCP Analyzer and TCP_Endpoint classes (since we also store sequence numbers there). Also, the analyzer framework will need tweaks as well (e.g., Undelivered uses ``int`` for sequence numbers, also has to go to 64 bit) > As a hotfix that seems to work I disabled the ``seq_to_skip`` features. It wasn't used by any analyzer or policy script (Note, that seq_to_skip is different from skip_deliveries). Hotfix is in > topic/gregor/reassembler-hotfix > {noformat} -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Fri Apr 18 07:35:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 18 Apr 2014 09:35:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1183) topic/jsiwek/ascii-log-memleak-fix In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203#comment-16203 ] Robin Sommer commented on BIT-1183: ----------------------------------- Excellent catch! > topic/jsiwek/ascii-log-memleak-fix > ---------------------------------- > > Key: BIT-1183 > URL: https://bro-tracker.atlassian.net/browse/BIT-1183 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > This branch fixes a memory leak in the ASCII log writer that occurs after each rotation. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Fri Apr 18 11:50:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 18 Apr 2014 13:50:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1184) topic/jsiwek/odesc-escaping In-Reply-To: References: Message-ID: Jon Siwek created BIT-1184: ------------------------------ Summary: topic/jsiwek/odesc-escaping Key: BIT-1184 URL: https://bro-tracker.atlassian.net/browse/BIT-1184 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: Jon Siwek Fix For: 2.3 Minor refactor of how ODesc hex escapes stuff. Most significant: it now uses a std::set instead of std::list internally to store what strings need escaping which would prevent the recent bug of that growing out of control. Otherwise, just changed some things to re-use code and be more readable (IMO). -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Fri Apr 18 11:51:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 18 Apr 2014 13:51:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1184) topic/jsiwek/odesc-escaping In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1184: --------------------------- Status: Merge Request (was: Open) > topic/jsiwek/odesc-escaping > --------------------------- > > Key: BIT-1184 > URL: https://bro-tracker.atlassian.net/browse/BIT-1184 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > Minor refactor of how ODesc hex escapes stuff. Most significant: it now uses a std::set instead of std::list internally to store what strings need escaping which would prevent the recent bug of that growing out of control. Otherwise, just changed some things to re-use code and be more readable (IMO). -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Fri Apr 18 16:32:09 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 18 Apr 2014 18:32:09 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1177) SumStats dynamic updates do not work in cluster mode In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1177: -------------------------------- Status: Merge Request (was: Open) > SumStats dynamic updates do not work in cluster mode > ---------------------------------------------------- > > Key: BIT-1177 > URL: https://bro-tracker.atlassian.net/browse/BIT-1177 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Bernhard Amann > Fix For: 2.3 > > > In cluster mode, dynamic updates via the request_key function do not work. > The reason is, that, at the moment, in cluster mode the function is defined only on the manager. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Fri Apr 18 16:32:08 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Fri, 18 Apr 2014 18:32:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1177) SumStats dynamic updates do not work in cluster mode In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204#comment-16204 ] Bernhard Amann commented on BIT-1177: ------------------------------------- I addressed this in the branch topic/bernhard/ticket1177. Basically, the function request_key is now ignored on worker nodes. This fixes the problem (if you have a non-cluster-aware script file, that calls the function on both, the master and the worker nodes it will now work flawlessly). However, it is not entirely pretty - there might be use cases where we actually want to allow just worker nodes to trigger data collection. That case just silently fails at the moment. Not entirely pretty, but it should work for now. If anyone has a better solution, I am open for suggestions > SumStats dynamic updates do not work in cluster mode > ---------------------------------------------------- > > Key: BIT-1177 > URL: https://bro-tracker.atlassian.net/browse/BIT-1177 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Bernhard Amann > Fix For: 2.3 > > > In cluster mode, dynamic updates via the request_key function do not work. > The reason is, that, at the moment, in cluster mode the function is defined only on the manager. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From noreply at bro.org Sat Apr 19 00:00:17 2014 From: noreply at bro.org (Merge Tracker) Date: Sat, 19 Apr 2014 00:00:17 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404190700.s3J70HeP016836@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- -------------- ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1184 [1] Bro Jon Siwek - 2014-04-18 2.3 Normal topic/jsiwek/odesc-escaping [2] BIT-1177 [3] Bro Bernhard Amann Bernhard Amann 2014-04-18 2.3 Normal SumStats dynamic updates do not work in cluster mode BIT-1168 [4] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-348 [5] Bro gregor Jon Siwek 2014-04-18 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ----------- ---------- ---------------------------------------------------------------------------- #6 [6] bro jshlbrd [7] 2014-04-15 Intel::ADDR indicators in http host field [8] #4 [9] bro mareq [10] 2014-04-01 Protocol identification heuristics. [11] #4 [12] time-machine mareq [13] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [14] #3 [15] time-machine mareq [16] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [17] #2 [18] time-machine mareq [19] 2014-04-09 Query interval start/end is now taken into account. [20] #1 [21] time-machine mareq [22] 2014-03-19 TM-16: Really skip VLAN header for indexing. [23] [1] BIT-1184 https://bro-tracker.atlassian.net/browse/BIT-1184 [2] odesc-escaping https://github.com/bro/bro/tree/topic/jsiwek/odesc-escaping [3] BIT-1177 https://bro-tracker.atlassian.net/browse/BIT-1177 [4] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [5] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [6] Pull Request #6 https://github.com/bro/bro/pull/6 [7] jshlbrd https://github.com/jshlbrd [8] Merge Pull Request #6 with git pull https://github.com/jshlbrd/bro.git master [9] Pull Request #4 https://github.com/bro/bro/pull/4 [10] mareq https://github.com/mareq [11] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [12] Pull Request #4 https://github.com/bro/time-machine/pull/4 [13] mareq https://github.com/mareq [14] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [15] Pull Request #3 https://github.com/bro/time-machine/pull/3 [16] mareq https://github.com/mareq [17] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [18] Pull Request #2 https://github.com/bro/time-machine/pull/2 [19] mareq https://github.com/mareq [20] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [21] Pull Request #1 https://github.com/bro/time-machine/pull/1 [22] mareq https://github.com/mareq [23] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Sat Apr 19 09:03:07 2014 From: jira at bro-tracker.atlassian.net (Paolo Galtieri (JIRA)) Date: Sat, 19 Apr 2014 11:03:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1165) [Bro] cron: error running update-stats In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16205#comment-16205 ] Paolo Galtieri commented on BIT-1165: ------------------------------------- I tried that several times and I still got the errors. I updated to the latest version and did the broctl install and I'll see what happens. > [Bro] cron: error running update-stats > -------------------------------------- > > Key: BIT-1165 > URL: https://bro-tracker.atlassian.net/browse/BIT-1165 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: 2.2 > Environment: Linux Fedora 19 on x86_64 > Reporter: Paolo Galtieri > Labels: logging > Attachments: broctl.cfg > > > I have configured bro so that all log files and spool files are saved to an external hard drive. I have modified the broctl.cfg file to point to the new locations. However, I continue to get lots of email about the following problem: > error running update-stats > ['cat: /usr/local/bro/spool/stats.log: No such file or directory'] > How do I stop this email? > The stats.log file is located in > /media/NSM/NSM-SENSOR-2/logs/bro/logs/stats > ls -l /media/NSM/NSM-SENSOR-2/logs/bro/logs/stats > total 13456 > -rw-r--r--. 1 root root 238 Mar 20 17:35 meta.dat > -rw-r--r--. 1 root root 13762847 Mar 18 22:50 stats.log > drwxr-xr-x. 2 root root 4096 Dec 5 13:25 www > Paolo -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From noreply at bro.org Sun Apr 20 00:00:13 2014 From: noreply at bro.org (Merge Tracker) Date: Sun, 20 Apr 2014 00:00:13 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404200700.s3K70DAc024681@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- -------------- ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1184 [1] Bro Jon Siwek - 2014-04-18 2.3 Normal topic/jsiwek/odesc-escaping [2] BIT-1177 [3] Bro Bernhard Amann Bernhard Amann 2014-04-18 2.3 Normal SumStats dynamic updates do not work in cluster mode BIT-1168 [4] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-348 [5] Bro gregor Jon Siwek 2014-04-18 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ----------- ---------- ---------------------------------------------------------------------------- #6 [6] bro jshlbrd [7] 2014-04-15 Intel::ADDR indicators in http host field [8] #4 [9] bro mareq [10] 2014-04-01 Protocol identification heuristics. [11] #4 [12] time-machine mareq [13] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [14] #3 [15] time-machine mareq [16] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [17] #2 [18] time-machine mareq [19] 2014-04-09 Query interval start/end is now taken into account. [20] #1 [21] time-machine mareq [22] 2014-03-19 TM-16: Really skip VLAN header for indexing. [23] [1] BIT-1184 https://bro-tracker.atlassian.net/browse/BIT-1184 [2] odesc-escaping https://github.com/bro/bro/tree/topic/jsiwek/odesc-escaping [3] BIT-1177 https://bro-tracker.atlassian.net/browse/BIT-1177 [4] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [5] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [6] Pull Request #6 https://github.com/bro/bro/pull/6 [7] jshlbrd https://github.com/jshlbrd [8] Merge Pull Request #6 with git pull https://github.com/jshlbrd/bro.git master [9] Pull Request #4 https://github.com/bro/bro/pull/4 [10] mareq https://github.com/mareq [11] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [12] Pull Request #4 https://github.com/bro/time-machine/pull/4 [13] mareq https://github.com/mareq [14] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [15] Pull Request #3 https://github.com/bro/time-machine/pull/3 [16] mareq https://github.com/mareq [17] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [18] Pull Request #2 https://github.com/bro/time-machine/pull/2 [19] mareq https://github.com/mareq [20] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [21] Pull Request #1 https://github.com/bro/time-machine/pull/1 [22] mareq https://github.com/mareq [23] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From noreply at bro.org Mon Apr 21 00:00:17 2014 From: noreply at bro.org (Merge Tracker) Date: Mon, 21 Apr 2014 00:00:17 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404210700.s3L70HIl019825@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- -------------- ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1184 [1] Bro Jon Siwek - 2014-04-18 2.3 Normal topic/jsiwek/odesc-escaping [2] BIT-1177 [3] Bro Bernhard Amann Bernhard Amann 2014-04-18 2.3 Normal SumStats dynamic updates do not work in cluster mode BIT-1168 [4] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-348 [5] Bro gregor Jon Siwek 2014-04-18 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ----------- ---------- ---------------------------------------------------------------------------- #6 [6] bro jshlbrd [7] 2014-04-15 Intel::ADDR indicators in http host field [8] #4 [9] bro mareq [10] 2014-04-01 Protocol identification heuristics. [11] #4 [12] time-machine mareq [13] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [14] #3 [15] time-machine mareq [16] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [17] #2 [18] time-machine mareq [19] 2014-04-09 Query interval start/end is now taken into account. [20] #1 [21] time-machine mareq [22] 2014-03-19 TM-16: Really skip VLAN header for indexing. [23] [1] BIT-1184 https://bro-tracker.atlassian.net/browse/BIT-1184 [2] odesc-escaping https://github.com/bro/bro/tree/topic/jsiwek/odesc-escaping [3] BIT-1177 https://bro-tracker.atlassian.net/browse/BIT-1177 [4] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [5] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [6] Pull Request #6 https://github.com/bro/bro/pull/6 [7] jshlbrd https://github.com/jshlbrd [8] Merge Pull Request #6 with git pull https://github.com/jshlbrd/bro.git master [9] Pull Request #4 https://github.com/bro/bro/pull/4 [10] mareq https://github.com/mareq [11] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [12] Pull Request #4 https://github.com/bro/time-machine/pull/4 [13] mareq https://github.com/mareq [14] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [15] Pull Request #3 https://github.com/bro/time-machine/pull/3 [16] mareq https://github.com/mareq [17] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [18] Pull Request #2 https://github.com/bro/time-machine/pull/2 [19] mareq https://github.com/mareq [20] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [21] Pull Request #1 https://github.com/bro/time-machine/pull/1 [22] mareq https://github.com/mareq [23] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Mon Apr 21 10:02:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 21 Apr 2014 12:02:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1177) SumStats dynamic updates do not work in cluster mode In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1177: ------------------------------ Assignee: Seth Hall (was: Bernhard Amann) > SumStats dynamic updates do not work in cluster mode > ---------------------------------------------------- > > Key: BIT-1177 > URL: https://bro-tracker.atlassian.net/browse/BIT-1177 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Seth Hall > Fix For: 2.3 > > > In cluster mode, dynamic updates via the request_key function do not work. > The reason is, that, at the moment, in cluster mode the function is defined only on the manager. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Mon Apr 21 14:31:07 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Mon, 21 Apr 2014 16:31:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1185) topic/dnthayer/broctld-work In-Reply-To: References: Message-ID: Daniel Thayer created BIT-1185: ---------------------------------- Summary: topic/dnthayer/broctld-work Key: BIT-1185 URL: https://bro-tracker.atlassian.net/browse/BIT-1185 Project: Bro Issue Tracker Issue Type: Problem Components: BroControl Reporter: Daniel Thayer Fix For: 2.3 This branch contains some code cleanup and also fixes or improves the following issues: The df, exec, and top commands now run only once per host. Avoid reporting same disk check error msg multiple times for same host. Improve output column formatting. Added warning to do a "broctl install" if broctl or node config changes. Don't email about "$total" pseudo-node not receiving any packets. Remove unused "home" broctl option. Changed plugin API hosts() function to be more useful. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Mon Apr 21 14:32:07 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Mon, 21 Apr 2014 16:32:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1185) topic/dnthayer/broctld-work In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1185: ------------------------------- Status: Merge Request (was: Open) > topic/dnthayer/broctld-work > --------------------------- > > Key: BIT-1185 > URL: https://bro-tracker.atlassian.net/browse/BIT-1185 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Reporter: Daniel Thayer > Fix For: 2.3 > > > This branch contains some code cleanup and also fixes or improves the > following issues: > The df, exec, and top commands now run only once per host. > Avoid reporting same disk check error msg multiple times for same host. > Improve output column formatting. > Added warning to do a "broctl install" if broctl or node config changes. > Don't email about "$total" pseudo-node not receiving any packets. > Remove unused "home" broctl option. > Changed plugin API hosts() function to be more useful. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Mon Apr 21 14:42:07 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Mon, 21 Apr 2014 16:42:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1186) Improve cluster configuration documentation In-Reply-To: References: Message-ID: Daniel Thayer created BIT-1186: ---------------------------------- Summary: Improve cluster configuration documentation Key: BIT-1186 URL: https://bro-tracker.atlassian.net/browse/BIT-1186 Project: Bro Issue Tracker Issue Type: Problem Components: Bro, BroControl Reporter: Daniel Thayer Fix For: 2.3 To make it easier to find "how-to" documentation for configuring a cluster, the plan is to create a new section in the Bro manual after the Quick Start Guide which will contain a step-by-step example of how to configure a Bro cluster (most of this content will be moved from the existing broctl manual) using broctl and optionally a load balancing method such as PF_RING. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From noreply at bro.org Tue Apr 22 00:00:14 2014 From: noreply at bro.org (Merge Tracker) Date: Tue, 22 Apr 2014 00:00:14 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404220700.s3M70Er2011136@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1185 [1] BroControl Daniel Thayer - 2014-04-21 2.3 Normal topic/dnthayer/broctld-work [2] BIT-1184 [3] Bro Jon Siwek - 2014-04-18 2.3 Normal topic/jsiwek/odesc-escaping [4] BIT-1177 [5] Bro Bernhard Amann Seth Hall 2014-04-21 2.3 Normal SumStats dynamic updates do not work in cluster mode BIT-1168 [6] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-348 [7] Bro gregor Jon Siwek 2014-04-18 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ----------- ---------- ---------------------------------------------------------------------------- #6 [8] bro jshlbrd [9] 2014-04-15 Intel::ADDR indicators in http host field [10] #4 [11] bro mareq [12] 2014-04-01 Protocol identification heuristics. [13] #4 [14] time-machine mareq [15] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [16] #3 [17] time-machine mareq [18] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [19] #2 [20] time-machine mareq [21] 2014-04-09 Query interval start/end is now taken into account. [22] #1 [23] time-machine mareq [24] 2014-03-19 TM-16: Really skip VLAN header for indexing. [25] [1] BIT-1185 https://bro-tracker.atlassian.net/browse/BIT-1185 [2] broctld-work https://github.com/bro/brocontrol/tree/topic/dnthayer/broctld-work [3] BIT-1184 https://bro-tracker.atlassian.net/browse/BIT-1184 [4] odesc-escaping https://github.com/bro/bro/tree/topic/jsiwek/odesc-escaping [5] BIT-1177 https://bro-tracker.atlassian.net/browse/BIT-1177 [6] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [7] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [8] Pull Request #6 https://github.com/bro/bro/pull/6 [9] jshlbrd https://github.com/jshlbrd [10] Merge Pull Request #6 with git pull https://github.com/jshlbrd/bro.git master [11] Pull Request #4 https://github.com/bro/bro/pull/4 [12] mareq https://github.com/mareq [13] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [14] Pull Request #4 https://github.com/bro/time-machine/pull/4 [15] mareq https://github.com/mareq [16] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [17] Pull Request #3 https://github.com/bro/time-machine/pull/3 [18] mareq https://github.com/mareq [19] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [20] Pull Request #2 https://github.com/bro/time-machine/pull/2 [21] mareq https://github.com/mareq [22] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [23] Pull Request #1 https://github.com/bro/time-machine/pull/1 [24] mareq https://github.com/mareq [25] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Tue Apr 22 07:28:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 22 Apr 2014 09:28:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-348) Reassembler integer overflow issues. Data not delivered after 2GB In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-348: -------------------------------- Assignee: Robin Sommer (was: Jon Siwek) > Reassembler integer overflow issues. Data not delivered after 2GB > ----------------------------------------------------------------- > > Key: BIT-348 > URL: https://bro-tracker.atlassian.net/browse/BIT-348 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: gregor > Assignee: Robin Sommer > Priority: High > Labels: inttypes > Fix For: 2.3 > > > {noformat} > #!rst > The TCP Reassembler does not deliver any data to analyzers after the first 2GB due to signed integer overflow (Actually it will deliver again between 4--6GB, etc.) This happens silently, i.e., without content_gap events or Undelivered calls. > This report superseded BIT-315, BIT-137 > The TCP Reassembler (and Reassem) base class use ``int`` to keep track of sequence numbers and ``seq_delta`` to check for differences. If a connection exceeds 2GB, the relative sequence numbers (int) used by the Reassembler become negative. While many parts of the Reassembler still work (because seq_delta still reports the correct difference) some parts do not. In particular ``seq_to_skip`` is broken (and fails silently). There might well be other parts of the Reassembler that fail > silently as well, that I haven't found yet. > See Comments in TCP_Reassembler.cc for more details. > The Reassembler should use int64. However this will require deep changes to the Reassembler and the TCP Analyzer and TCP_Endpoint classes (since we also store sequence numbers there). Also, the analyzer framework will need tweaks as well (e.g., Undelivered uses ``int`` for sequence numbers, also has to go to 64 bit) > As a hotfix that seems to work I disabled the ``seq_to_skip`` features. It wasn't used by any analyzer or policy script (Note, that seq_to_skip is different from skip_deliveries). Hotfix is in > topic/gregor/reassembler-hotfix > {noformat} -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Tue Apr 22 20:33:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 22 Apr 2014 22:33:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1185) topic/dnthayer/broctld-work In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206#comment-16206 ] Robin Sommer commented on BIT-1185: ----------------------------------- > Added warning to do a "broctl install" if broctl or node config changes. The code here looks a bit fragile to me as it depends on the right format being written out so that it can then later be reparsed. I'm wondering if we could just compute a hash of the two files instead and suggest a reinstall once that changes? That would trigger on insignificant changes, too, like comments, but I don't think that would hurt much. Also, the current code can miss some cases: like when a node gets removed, I don't think it will catch that (because it's not in the list of nodes iterated through in warnBroctlInstall(). > topic/dnthayer/broctld-work > --------------------------- > > Key: BIT-1185 > URL: https://bro-tracker.atlassian.net/browse/BIT-1185 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Reporter: Daniel Thayer > Fix For: 2.3 > > > This branch contains some code cleanup and also fixes or improves the > following issues: > The df, exec, and top commands now run only once per host. > Avoid reporting same disk check error msg multiple times for same host. > Improve output column formatting. > Added warning to do a "broctl install" if broctl or node config changes. > Don't email about "$total" pseudo-node not receiving any packets. > Remove unused "home" broctl option. > Changed plugin API hosts() function to be more useful. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Tue Apr 22 20:36:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 22 Apr 2014 22:36:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1185) topic/dnthayer/broctld-work In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16207#comment-16207 ] Robin Sommer commented on BIT-1185: ----------------------------------- A note for the future: I would prefer separate merge requests for individual features. Collecting multiple into one is fine if it's all just very small changes that each just touches a few lines, but this one has enough changes overall that it's getting tricky for me to follow which code change is associated with which semantic change (I'm usually looking at the whole diff between master and branch, not at individual commits inside the branch). > topic/dnthayer/broctld-work > --------------------------- > > Key: BIT-1185 > URL: https://bro-tracker.atlassian.net/browse/BIT-1185 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Reporter: Daniel Thayer > Fix For: 2.3 > > > This branch contains some code cleanup and also fixes or improves the > following issues: > The df, exec, and top commands now run only once per host. > Avoid reporting same disk check error msg multiple times for same host. > Improve output column formatting. > Added warning to do a "broctl install" if broctl or node config changes. > Don't email about "$total" pseudo-node not receiving any packets. > Remove unused "home" broctl option. > Changed plugin API hosts() function to be more useful. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Tue Apr 22 21:34:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 22 Apr 2014 23:34:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1184) topic/jsiwek/odesc-escaping In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1184: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > topic/jsiwek/odesc-escaping > --------------------------- > > Key: BIT-1184 > URL: https://bro-tracker.atlassian.net/browse/BIT-1184 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > Minor refactor of how ODesc hex escapes stuff. Most significant: it now uses a std::set instead of std::list internally to store what strings need escaping which would prevent the recent bug of that growing out of control. Otherwise, just changed some things to re-use code and be more readable (IMO). -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From robin at icir.org Tue Apr 22 21:41:05 2014 From: robin at icir.org (Robin Sommer) Date: Tue, 22 Apr 2014 21:41:05 -0700 Subject: [Bro-Dev] DNS fix (Re: [Bro-Commits] [git/bro] master: Updating CHANGES and VERSION.) (e24f3f5) In-Reply-To: <201404230443.s3N4hbYH007574@bro-ids.icir.org> References: <201404230443.s3N4hbYH007574@bro-ids.icir.org> Message-ID: <20140423044105.GA17424@icir.org> I messed up the commit message here. This was supposed to be this: Fix duplicate DNS log entries. An earlier change to clear state didn't have the intended effect; fixed by using a bif instead. Without this fix it could still happen that DNS entries got logged multiple times, I saw a case where dns.log grew by a factor of more than 10. I guess this might also have caused some of the trouble with logging on the manager, just by nature of the additional volume. In fact, I'm wondering if this could explain memory trouble as well. Does this help with worker problems by any chance? Robin On Tue, Apr 22, 2014 at 21:43 -0700, I wrote: > Repository : ssh://git at bro-ids.icir.org/bro > > On branch : master > Link : https://github.com/bro/bro/commit/e24f3f5fd514832ecb26c9b606b9dbfed56792f2 > > >--------------------------------------------------------------- > > commit e24f3f5fd514832ecb26c9b606b9dbfed56792f2 > Author: Robin Sommer > Date: Tue Apr 22 16:32:22 2014 -0700 > > Updating CHANGES and VERSION. > > > >--------------------------------------------------------------- > > e24f3f5fd514832ecb26c9b606b9dbfed56792f2 > CHANGES | 4 ++++ > aux/broctl | 2 +- > scripts/base/protocols/dns/main.bro | 2 +- > 3 files changed, 6 insertions(+), 2 deletions(-) > > diff --git a/CHANGES b/CHANGES > index d91a2e4..aad4732 100644 > --- a/CHANGES > +++ b/CHANGES > @@ -1,4 +1,8 @@ > > +2.2-341 | 2014-04-17 18:01:41 -0500 > + > + * Fix duplicate DNS log entries. (Robin Sommer) > + > 2.2-341 | 2014-04-17 18:01:01 -0500 > > * Refactor initialization of ASCII log writer options. (Jon Siwek) > diff --git a/aux/broctl b/aux/broctl > index d991508..f249570 160000 > --- a/aux/broctl > +++ b/aux/broctl > @@ -1 +1 @@ > -Subproject commit d99150801b7844e082b5421d1efe4050702d350e > +Subproject commit f249570e3fb4c83e532cc0813786f0ff60c4dea9 > diff --git a/scripts/base/protocols/dns/main.bro b/scripts/base/protocols/dns/main.bro > index fe371de..d728060 100644 > --- a/scripts/base/protocols/dns/main.bro > +++ b/scripts/base/protocols/dns/main.bro > @@ -183,7 +183,7 @@ function log_unmatched_msgs(msgs: PendingMessages) > for ( trans_id in msgs ) > log_unmatched_msgs_queue(msgs[trans_id]); > > - msgs = PendingMessages(); > + clear_table(msgs); > } > > function enqueue_new_msg(msgs: PendingMessages, id: count, msg: Info) > > _______________________________________________ > bro-commits mailing list > bro-commits at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-commits > -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From noreply at bro.org Wed Apr 23 00:00:12 2014 From: noreply at bro.org (Merge Tracker) Date: Wed, 23 Apr 2014 00:00:12 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404230700.s3N70CqD015394@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1185 [1] BroControl Daniel Thayer - 2014-04-22 2.3 Normal topic/dnthayer/broctld-work [2] BIT-1177 [3] Bro Bernhard Amann Seth Hall 2014-04-21 2.3 Normal SumStats dynamic updates do not work in cluster mode BIT-1168 [4] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-348 [5] Bro gregor Robin Sommer 2014-04-22 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ----------- ---------- ---------------------------------------------------------------------------- #6 [6] bro jshlbrd [7] 2014-04-15 Intel::ADDR indicators in http host field [8] #4 [9] bro mareq [10] 2014-04-23 Protocol identification heuristics. [11] #4 [12] time-machine mareq [13] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [14] #3 [15] time-machine mareq [16] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [17] #2 [18] time-machine mareq [19] 2014-04-09 Query interval start/end is now taken into account. [20] #1 [21] time-machine mareq [22] 2014-03-19 TM-16: Really skip VLAN header for indexing. [23] [1] BIT-1185 https://bro-tracker.atlassian.net/browse/BIT-1185 [2] broctld-work https://github.com/bro/brocontrol/tree/topic/dnthayer/broctld-work [3] BIT-1177 https://bro-tracker.atlassian.net/browse/BIT-1177 [4] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [5] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [6] Pull Request #6 https://github.com/bro/bro/pull/6 [7] jshlbrd https://github.com/jshlbrd [8] Merge Pull Request #6 with git pull https://github.com/jshlbrd/bro.git master [9] Pull Request #4 https://github.com/bro/bro/pull/4 [10] mareq https://github.com/mareq [11] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [12] Pull Request #4 https://github.com/bro/time-machine/pull/4 [13] mareq https://github.com/mareq [14] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [15] Pull Request #3 https://github.com/bro/time-machine/pull/3 [16] mareq https://github.com/mareq [17] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [18] Pull Request #2 https://github.com/bro/time-machine/pull/2 [19] mareq https://github.com/mareq [20] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [21] Pull Request #1 https://github.com/bro/time-machine/pull/1 [22] mareq https://github.com/mareq [23] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Wed Apr 23 08:51:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 23 Apr 2014 10:51:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1187) topic/jsiwek/remove-val-attribs In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1187: --------------------------- Status: Merge Request (was: Open) > topic/jsiwek/remove-val-attribs > ------------------------------- > > Key: BIT-1187 > URL: https://bro-tracker.atlassian.net/browse/BIT-1187 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, Broccoli > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > Dead code removal in bro and broccoli repos. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Wed Apr 23 08:51:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 23 Apr 2014 10:51:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1187) topic/jsiwek/remove-val-attribs In-Reply-To: References: Message-ID: Jon Siwek created BIT-1187: ------------------------------ Summary: topic/jsiwek/remove-val-attribs Key: BIT-1187 URL: https://bro-tracker.atlassian.net/browse/BIT-1187 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro, Broccoli Affects Versions: git/master Reporter: Jon Siwek Fix For: 2.3 Dead code removal in bro and broccoli repos. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Wed Apr 23 08:58:07 2014 From: jira at bro-tracker.atlassian.net (Adam Slagell (JIRA)) Date: Wed, 23 Apr 2014 10:58:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1185) topic/dnthayer/broctld-work In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16208#comment-16208 ] Adam Slagell commented on BIT-1185: ----------------------------------- Yeah, I think that probably makes more sense. ------ Adam J. Slagell Chief Information Security Officer Assistant Director, Cybersecurity National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.ncsa.illinois.edu/~slagell/ "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." > topic/dnthayer/broctld-work > --------------------------- > > Key: BIT-1185 > URL: https://bro-tracker.atlassian.net/browse/BIT-1185 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Reporter: Daniel Thayer > Fix For: 2.3 > > > This branch contains some code cleanup and also fixes or improves the > following issues: > The df, exec, and top commands now run only once per host. > Avoid reporting same disk check error msg multiple times for same host. > Improve output column formatting. > Added warning to do a "broctl install" if broctl or node config changes. > Don't email about "$total" pseudo-node not receiving any packets. > Remove unused "home" broctl option. > Changed plugin API hosts() function to be more useful. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Wed Apr 23 12:33:07 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Wed, 23 Apr 2014 14:33:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1185) topic/dnthayer/broctld-work In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209#comment-16209 ] Daniel Thayer commented on BIT-1185: ------------------------------------ {quote} > Added warning to do a "broctl install" if broctl or node config changes. The code here looks a bit fragile to me as it depends on the right format being written out so that it can then later be reparsed. I'm wondering if we could just compute a hash of the two files instead and suggest a reinstall once that changes? That would trigger on insignificant changes, too, like comments, but I don't think that would hurt much. {quote} Just doing a hash of broctl.cfg itself wouldn't work very well, because there are various ways that the broctl config can change: 1) someone edits broctl.cfg (although some changes here wouldn't actually cause anything in the config to change, such as comments, whitespace, option ordering, or when someone adds an option that wasn't previously in the file but uses the default value), 2) someone changes a default option value in the broctl source code, 3) a dynamically-generated broctl option value changes (such as "standalone", or "time"), 4) someone changes a custom plugin, which defines its own broctl options Admittedly, #2 and #3 are probably quite unlikely to happen in real usage. So, we'd need to do the hash over the entire config (not just broctl.cfg) in order to make sure that we notice all possible config changes. But, this would sometimes warn users to "install" even when they don't need to because there are a significant number of broctl options that do not require an "install" when their value changes (but not sure how often users would encounter this situation). As a future enhancement, we could add a new attribute to each broctl option which would be a boolean that determines whether or not the option's value change should trigger an "install" or not. However, this wouldn't work if we just do a hash of the configuration. {quote} Also, the current code can miss some cases: like when a node gets removed, I don't think it will catch that (because it's not in the list of nodes iterated through in warnBroctlInstall(). {quote} Yes, you're right. I've fixed that now. > topic/dnthayer/broctld-work > --------------------------- > > Key: BIT-1185 > URL: https://bro-tracker.atlassian.net/browse/BIT-1185 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Reporter: Daniel Thayer > Fix For: 2.3 > > > This branch contains some code cleanup and also fixes or improves the > following issues: > The df, exec, and top commands now run only once per host. > Avoid reporting same disk check error msg multiple times for same host. > Improve output column formatting. > Added warning to do a "broctl install" if broctl or node config changes. > Don't email about "$total" pseudo-node not receiving any packets. > Remove unused "home" broctl option. > Changed plugin API hosts() function to be more useful. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From robin at icir.org Wed Apr 23 20:52:04 2014 From: robin at icir.org (Robin Sommer) Date: Wed, 23 Apr 2014 20:52:04 -0700 Subject: [Bro-Dev] [JIRA] (BIT-1185) topic/dnthayer/broctld-work In-Reply-To: References: Message-ID: <20140424035204.GI83390@icir.org> > Just doing a hash of broctl.cfg itself wouldn't work very well, because there are various ways that the > broctl config can change: I don't think I'm too concerned about these, as generally they all seem rare enough that an additional warning wouldn't hurt much. I have another idea though: could we pickle the relevant Python data structures (i.e., config and nodes) and hash the binary representation coming out of that? From jira at bro-tracker.atlassian.net Wed Apr 23 20:54:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Wed, 23 Apr 2014 22:54:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1185) topic/dnthayer/broctld-work In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16210#comment-16210 ] Robin Sommer commented on BIT-1185: ----------------------------------- I don't think I'm too concerned about these, as generally they all seem rare enough that an additional warning wouldn't hurt much. I have another idea though: could we pickle the relevant Python data structures (i.e., config and nodes) and hash the binary representation coming out of that? > topic/dnthayer/broctld-work > --------------------------- > > Key: BIT-1185 > URL: https://bro-tracker.atlassian.net/browse/BIT-1185 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Reporter: Daniel Thayer > Fix For: 2.3 > > > This branch contains some code cleanup and also fixes or improves the > following issues: > The df, exec, and top commands now run only once per host. > Avoid reporting same disk check error msg multiple times for same host. > Improve output column formatting. > Added warning to do a "broctl install" if broctl or node config changes. > Don't email about "$total" pseudo-node not receiving any packets. > Remove unused "home" broctl option. > Changed plugin API hosts() function to be more useful. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From noreply at bro.org Thu Apr 24 00:00:16 2014 From: noreply at bro.org (Merge Tracker) Date: Thu, 24 Apr 2014 00:00:16 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404240700.s3O70GsQ020504@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ------------ -------------- ------------ ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1187 [1] Bro,Broccoli Jon Siwek - 2014-04-23 2.3 Normal topic/jsiwek/remove-val-attribs [2] BIT-1185 [3] BroControl Daniel Thayer - 2014-04-23 2.3 Normal topic/dnthayer/broctld-work [4] BIT-1177 [5] Bro Bernhard Amann Seth Hall 2014-04-21 2.3 Normal SumStats dynamic updates do not work in cluster mode BIT-1168 [6] Bro Brian Little Seth Hall 2014-03-31 - Low Add Java version to software framework BIT-348 [7] Bro gregor Robin Sommer 2014-04-22 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ----------- ---------- ---------------------------------------------------------------------------- #6 [8] bro jshlbrd [9] 2014-04-15 Intel::ADDR indicators in http host field [10] #4 [11] bro mareq [12] 2014-04-23 Protocol identification heuristics. [13] #4 [14] time-machine mareq [15] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [16] #3 [17] time-machine mareq [18] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [19] #2 [20] time-machine mareq [21] 2014-04-09 Query interval start/end is now taken into account. [22] #1 [23] time-machine mareq [24] 2014-03-19 TM-16: Really skip VLAN header for indexing. [25] [1] BIT-1187 https://bro-tracker.atlassian.net/browse/BIT-1187 [2] remove-val-attribs https://github.com/bro/bro,broccoli/tree/topic/jsiwek/remove-val-attribs [3] BIT-1185 https://bro-tracker.atlassian.net/browse/BIT-1185 [4] broctld-work https://github.com/bro/brocontrol/tree/topic/dnthayer/broctld-work [5] BIT-1177 https://bro-tracker.atlassian.net/browse/BIT-1177 [6] BIT-1168 https://bro-tracker.atlassian.net/browse/BIT-1168 [7] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [8] Pull Request #6 https://github.com/bro/bro/pull/6 [9] jshlbrd https://github.com/jshlbrd [10] Merge Pull Request #6 with git pull https://github.com/jshlbrd/bro.git master [11] Pull Request #4 https://github.com/bro/bro/pull/4 [12] mareq https://github.com/mareq [13] Merge Pull Request #4 with git pull https://github.com/mareq/bro.git topic/mareq/analyzer-for-missing-request [14] Pull Request #4 https://github.com/bro/time-machine/pull/4 [15] mareq https://github.com/mareq [16] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [17] Pull Request #3 https://github.com/bro/time-machine/pull/3 [18] mareq https://github.com/mareq [19] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [20] Pull Request #2 https://github.com/bro/time-machine/pull/2 [21] mareq https://github.com/mareq [22] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [23] Pull Request #1 https://github.com/bro/time-machine/pull/1 [24] mareq https://github.com/mareq [25] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Thu Apr 24 07:46:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 24 Apr 2014 09:46:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1065) packet_filter.log is broken In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1065: --------------------------- Fix Version/s: (was: git/master) 2.2 > packet_filter.log is broken > --------------------------- > > Key: BIT-1065 > URL: https://bro-tracker.atlassian.net/browse/BIT-1065 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Seth Hall > Fix For: 2.2 > > > Apparently I broke the packet_filter.log when I overhauled the packet filter framework. I need to make sure and fix that prior to the 2.2 release (I think it's an ok thing to be broken during the beta). -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 07:53:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 24 Apr 2014 09:53:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1145) Individual set_seperator for different feeds In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1145: --------------------------- Fix Version/s: (was: 2.4) 2.3 > Individual set_seperator for different feeds > -------------------------------------------- > > Key: BIT-1145 > URL: https://bro-tracker.atlassian.net/browse/BIT-1145 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Aashish Sharma > Labels: feeds, framework, input, logging > Fix For: 2.3 > > > Can we assign an individual set_separator per feed ? > Why ?: > Various data feeds from different sources have their own fields separators. > We need to post process these feeds in order to digest the data into bro using input-framework, this creates a need to have two tiered storage for each of the data feeds (original data + re-formatted data for input framework). > At present the workaround is to basically format all data feeds to use intel-framework and this works very well. There is still useful needs to have data feeds outside intel-framework for example - digesting list of subnets+building allocations in the network or digesting auth data... and so on. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 07:54:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 24 Apr 2014 09:54:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1145) Individual set_seperator for different feeds In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211#comment-16211 ] Jon Siwek commented on BIT-1145: -------------------------------- I think this is all done and fixed in 2.3, but if there was some reason this was set to 2.4, please correct. > Individual set_seperator for different feeds > -------------------------------------------- > > Key: BIT-1145 > URL: https://bro-tracker.atlassian.net/browse/BIT-1145 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Aashish Sharma > Labels: feeds, framework, input, logging > Fix For: 2.3 > > > Can we assign an individual set_separator per feed ? > Why ?: > Various data feeds from different sources have their own fields separators. > We need to post process these feeds in order to digest the data into bro using input-framework, this creates a need to have two tiered storage for each of the data feeds (original data + re-formatted data for input framework). > At present the workaround is to basically format all data feeds to use intel-framework and this works very well. There is still useful needs to have data feeds outside intel-framework for example - digesting list of subnets+building allocations in the network or digesting auth data... and so on. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 07:56:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 24 Apr 2014 09:56:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1154) Formatters restructed in: topic/seth/json-formatter In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212#comment-16212 ] Jon Siwek commented on BIT-1154: -------------------------------- Is there still more to do here? If so, should a "fix version" be set to put it on the roadmap? > Formatters restructed in: topic/seth/json-formatter > --------------------------------------------------- > > Key: BIT-1154 > URL: https://bro-tracker.atlassian.net/browse/BIT-1154 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: 2.3 > Reporter: Seth Hall > > topic/seth/json-formatter has an abstraction for Formatters and I created a formatters directory under threading. There is also a new JSON formatter and support in the Ascii and ElasticSearch writers for the JSON formatter. > I went ahead and threw in per-filter configuration options for the Ascii writer for all of the options that were exposed globally too. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 07:58:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 24 Apr 2014 09:58:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-953) SSL Analyzer: return the root CA used to validate a cert In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-953: -------------------------- Fix Version/s: (was: 2.4) 2.3 > SSL Analyzer: return the root CA used to validate a cert > -------------------------------------------------------- > > Key: BIT-953 > URL: https://bro-tracker.atlassian.net/browse/BIT-953 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: liamrandall > Assignee: Bernhard Amann > Priority: Low > Labels: Analyzer,, CA, Root,, SSL > Fix For: 2.3 > > > Since Bro will validate certs can we add a variable that says who the root CA was; would be useful for CA pinning, white listing or black listing. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 07:59:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 24 Apr 2014 09:59:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-911) SRV replies don't get processed by DNS analyzer In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-911: -------------------------- Fix Version/s: (was: 2.4) 2.3 > SRV replies don't get processed by DNS analyzer > ----------------------------------------------- > > Key: BIT-911 > URL: https://bro-tracker.atlassian.net/browse/BIT-911 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Vern Paxson > Fix For: 2.3 > > Attachments: tdns-srv.bug.trace > > > The event engine doesn't appear to generate {{dns_SRV_reply}} in some cases, as indicated by running on the attached trace. I've tried this with both the default DNS analysis and my own custom analysis (that uses \-b to not run other stuff) and have confirmed that the reply event isn't getting generated, even though there aren't any checksum issues or such. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 08:00:08 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 24 Apr 2014 10:00:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-760) Lift Server Alternative Name (SAN) field to scripting layer In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-760: -------------------------- Fix Version/s: (was: 2.4) 2.3 > Lift Server Alternative Name (SAN) field to scripting layer > ----------------------------------------------------------- > > Key: BIT-760 > URL: https://bro-tracker.atlassian.net/browse/BIT-760 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Matthias Vallentin > Assignee: Bernhard Amann > Labels: analyzer > Fix For: 2.3 > > > It would be nice to have the *Subject Alternative Name (SAN)* field of an X.509 certificate available at the scripting layer. It contains a list of domains that should be used in addition to the CN field of the subject to verify that a domain matches the certificate. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 08:01:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 24 Apr 2014 10:01:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-700) PacketSorter In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-700: -------------------------- Fix Version/s: (was: 2.4) 2.3 > PacketSorter > ------------ > > Key: BIT-700 > URL: https://bro-tracker.atlassian.net/browse/BIT-700 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: gregor > Assignee: Robin Sommer > Labels: BroV6,, IPv6 > Fix For: 2.3 > > > (from an e-mail I sent a while ago) > Might relevant for IPv6 so setting milestone to 2.1 > Hi, > I was wondering about Bro's packet sorter. From a quick glance it > appears that it's only enabled if packet_sort_window is set to a non > zero value. When enabled it will sort packets > a) based on timestamps and > b) for TCP packets based on SEQ/ACK numbers (I presume to ensure that > ACKs are delivered after the data packet) > Note, this is independent from Bro's ability to process multiple trace > files (or multiple interfaces) in order. So I was wondering about the > use cases for PacketSorter, especially (a) > If the packet sorter is enabled Bro's behavior will slightly change: It > won't pass ARP packets to the ARP analyzer, and it won't create a weird > if it's not an IP packet. > I was just wondering whether anybody has recently used the packet > sorter. If not I'm wondering whether we should test this code path to > see whether it works correctly esp wrt IPv6. > Or, actually, whether the packet sorter is worth keeping or whether we > should remove the code. > And another question would be if the TCP sorting would better be handled > by the TCP analyzer? > Opinions? -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 08:05:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 24 Apr 2014 10:05:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1179) HTTP analyzer too sensitive to content gaps, was: HTTP messages missing in files.log In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1179: --------------------------- Summary: HTTP analyzer too sensitive to content gaps, was: HTTP messages missing in files.log (was: HTTP messages missing in files.log) > HTTP analyzer too sensitive to content gaps, was: HTTP messages missing in files.log > ------------------------------------------------------------------------------------ > > Key: BIT-1179 > URL: https://bro-tracker.atlassian.net/browse/BIT-1179 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.4 > > > I have a trace with multiple HTTP requests inside a persistent HTTP session. for which only the first two appear in files.log, the remaining ones are missing. Looks like a bug. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 08:05:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 24 Apr 2014 10:05:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1179) HTTP analyzer too sensitive to content gaps, was: HTTP messages missing in files.log In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1179: --------------------------- Fix Version/s: (was: 2.3) 2.4 > HTTP analyzer too sensitive to content gaps, was: HTTP messages missing in files.log > ------------------------------------------------------------------------------------ > > Key: BIT-1179 > URL: https://bro-tracker.atlassian.net/browse/BIT-1179 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.4 > > > I have a trace with multiple HTTP requests inside a persistent HTTP session. for which only the first two appear in files.log, the remaining ones are missing. Looks like a bug. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 08:06:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 24 Apr 2014 10:06:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1179) HTTP analyzer too sensitive to content gaps, was: HTTP messages missing in files.log In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1179: --------------------------- Assignee: (was: Jon Siwek) > HTTP analyzer too sensitive to content gaps, was: HTTP messages missing in files.log > ------------------------------------------------------------------------------------ > > Key: BIT-1179 > URL: https://bro-tracker.atlassian.net/browse/BIT-1179 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Fix For: 2.4 > > > I have a trace with multiple HTTP requests inside a persistent HTTP session. for which only the first two appear in files.log, the remaining ones are missing. Looks like a bug. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 08:28:07 2014 From: jira at bro-tracker.atlassian.net (scampbell (JIRA)) Date: Thu, 24 Apr 2014 10:28:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1188) subinterface defns for dnacluster:21@x interfaces in lb_pf_ring.py not reset In-Reply-To: References: Message-ID: scampbell created BIT-1188: ------------------------------ Summary: subinterface defns for dnacluster:21 at x interfaces in lb_pf_ring.py not reset Key: BIT-1188 URL: https://bro-tracker.atlassian.net/browse/BIT-1188 Project: Bro Issue Tracker Issue Type: Patch Components: BroControl Affects Versions: git/master Environment: scientific linux Reporter: scampbell Priority: High When more than one system in node.cgf is defined with dnacluster interfaces, the subinterface (ie x value in dnacluster:21 at x) is not reset when the parser goes to a new physical system. This causes a "no interface" error. One line fix to lb_pf_ring.py *** /home/bro/SOFTWARE/bro/aux/broctl/BroControl/plugins/lb_pf_ring.py 2014-04-22 08:47:24.884212793 -0700 --- /home/bro/lib/broctl/plugins/lb_pf_ring.py 2014-04-24 08:19:07.128480987 -0700 *************** *** 48,53 **** --- 48,54 ---- app_instance = first_app_instance dd[nn.host][nn.interface] = cluster_id + len(dd[nn.host]) else: + app_instance = 0 dd[nn.host] = { nn.interface : cluster_id } # Apply environment variables, but do not override values from -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 09:50:07 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 24 Apr 2014 11:50:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1145) Individual set_seperator for different feeds In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1145: --------------------------- Attachment: signature.asc Nope, no reason. I just fixed this when I did the formatter refactoring and forgot to close this ticket. > Individual set_seperator for different feeds > -------------------------------------------- > > Key: BIT-1145 > URL: https://bro-tracker.atlassian.net/browse/BIT-1145 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Aashish Sharma > Labels: feeds, framework, input, logging > Fix For: 2.3 > > Attachments: signature.asc > > > Can we assign an individual set_separator per feed ? > Why ?: > Various data feeds from different sources have their own fields separators. > We need to post process these feeds in order to digest the data into bro using input-framework, this creates a need to have two tiered storage for each of the data feeds (original data + re-formatted data for input framework). > At present the workaround is to basically format all data feeds to use intel-framework and this works very well. There is still useful needs to have data feeds outside intel-framework for example - digesting list of subnets+building allocations in the network or digesting auth data... and so on. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 10:11:07 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 24 Apr 2014 12:11:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1188) subinterface defns for dnacluster:21@x interfaces in lb_pf_ring.py not reset In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16214#comment-16214 ] Daniel Thayer commented on BIT-1188: ------------------------------------ {quote} app_instance = first_app_instance dd[nn.host][nn.interface] = cluster_id + len(dd[nn.host]) else: + app_instance = 0 dd[nn.host] = { nn.interface : cluster_id }app_instance = first_app_instance {quote} Shouldn't that new line be this instead: app_instance = first_app_instance > subinterface defns for dnacluster:21 at x interfaces in lb_pf_ring.py not reset > ----------------------------------------------------------------------------- > > Key: BIT-1188 > URL: https://bro-tracker.atlassian.net/browse/BIT-1188 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Affects Versions: git/master > Environment: scientific linux > Reporter: scampbell > Priority: High > Labels: broctl > > When more than one system in node.cgf is defined with dnacluster interfaces, the subinterface (ie x value in dnacluster:21 at x) is not reset when the parser goes to a new physical system. This causes a "no interface" error. > One line fix to lb_pf_ring.py > *** /home/bro/SOFTWARE/bro/aux/broctl/BroControl/plugins/lb_pf_ring.py 2014-04-22 08:47:24.884212793 -0700 > --- /home/bro/lib/broctl/plugins/lb_pf_ring.py 2014-04-24 08:19:07.128480987 -0700 > *************** > *** 48,53 **** > --- 48,54 ---- > app_instance = first_app_instance > dd[nn.host][nn.interface] = cluster_id + len(dd[nn.host]) > else: > + app_instance = 0 > dd[nn.host] = { nn.interface : cluster_id } > # Apply environment variables, but do not override values from -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 10:18:07 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 24 Apr 2014 12:18:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1188) subinterface defns for dnacluster:21@x interfaces in lb_pf_ring.py not reset In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1188: --------------------------- Attachment: signature.asc It should. This is strange because I thought I fixed this in a commit. I?ll go ahead and commit this directly into broctl master. > subinterface defns for dnacluster:21 at x interfaces in lb_pf_ring.py not reset > ----------------------------------------------------------------------------- > > Key: BIT-1188 > URL: https://bro-tracker.atlassian.net/browse/BIT-1188 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Affects Versions: git/master > Environment: scientific linux > Reporter: scampbell > Priority: High > Labels: broctl > Attachments: signature.asc > > > When more than one system in node.cgf is defined with dnacluster interfaces, the subinterface (ie x value in dnacluster:21 at x) is not reset when the parser goes to a new physical system. This causes a "no interface" error. > One line fix to lb_pf_ring.py > *** /home/bro/SOFTWARE/bro/aux/broctl/BroControl/plugins/lb_pf_ring.py 2014-04-22 08:47:24.884212793 -0700 > --- /home/bro/lib/broctl/plugins/lb_pf_ring.py 2014-04-24 08:19:07.128480987 -0700 > *************** > *** 48,53 **** > --- 48,54 ---- > app_instance = first_app_instance > dd[nn.host][nn.interface] = cluster_id + len(dd[nn.host]) > else: > + app_instance = 0 > dd[nn.host] = { nn.interface : cluster_id } > # Apply environment variables, but do not override values from -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 10:26:07 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 24 Apr 2014 12:26:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1188) subinterface defns for dnacluster:21@x interfaces in lb_pf_ring.py not reset In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216#comment-16216 ] Seth Hall commented on BIT-1188: -------------------------------- Done. I also went ahead and added support for the new zerocopy (zc) naming convention for the new PF_Ring zerocopy interfaces. > subinterface defns for dnacluster:21 at x interfaces in lb_pf_ring.py not reset > ----------------------------------------------------------------------------- > > Key: BIT-1188 > URL: https://bro-tracker.atlassian.net/browse/BIT-1188 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Affects Versions: git/master > Environment: scientific linux > Reporter: scampbell > Priority: High > Labels: broctl > Attachments: signature.asc > > > When more than one system in node.cgf is defined with dnacluster interfaces, the subinterface (ie x value in dnacluster:21 at x) is not reset when the parser goes to a new physical system. This causes a "no interface" error. > One line fix to lb_pf_ring.py > *** /home/bro/SOFTWARE/bro/aux/broctl/BroControl/plugins/lb_pf_ring.py 2014-04-22 08:47:24.884212793 -0700 > --- /home/bro/lib/broctl/plugins/lb_pf_ring.py 2014-04-24 08:19:07.128480987 -0700 > *************** > *** 48,53 **** > --- 48,54 ---- > app_instance = first_app_instance > dd[nn.host][nn.interface] = cluster_id + len(dd[nn.host]) > else: > + app_instance = 0 > dd[nn.host] = { nn.interface : cluster_id } > # Apply environment variables, but do not override values from -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 10:27:07 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 24 Apr 2014 12:27:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1188) subinterface defns for dnacluster:21@x interfaces in lb_pf_ring.py not reset In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1188: --------------------------- Resolution: Fixed Status: Closed (was: Open) Thanks Scott! > subinterface defns for dnacluster:21 at x interfaces in lb_pf_ring.py not reset > ----------------------------------------------------------------------------- > > Key: BIT-1188 > URL: https://bro-tracker.atlassian.net/browse/BIT-1188 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Affects Versions: git/master > Environment: scientific linux > Reporter: scampbell > Priority: High > Labels: broctl > Attachments: signature.asc > > > When more than one system in node.cgf is defined with dnacluster interfaces, the subinterface (ie x value in dnacluster:21 at x) is not reset when the parser goes to a new physical system. This causes a "no interface" error. > One line fix to lb_pf_ring.py > *** /home/bro/SOFTWARE/bro/aux/broctl/BroControl/plugins/lb_pf_ring.py 2014-04-22 08:47:24.884212793 -0700 > --- /home/bro/lib/broctl/plugins/lb_pf_ring.py 2014-04-24 08:19:07.128480987 -0700 > *************** > *** 48,53 **** > --- 48,54 ---- > app_instance = first_app_instance > dd[nn.host][nn.interface] = cluster_id + len(dd[nn.host]) > else: > + app_instance = 0 > dd[nn.host] = { nn.interface : cluster_id } > # Apply environment variables, but do not override values from -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 10:51:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 24 Apr 2014 12:51:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1141) Investigate further improvements to file analysis performance In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218#comment-16218 ] Jon Siwek commented on BIT-1141: -------------------------------- I have some performance improvement changes in topic/jsiwek/faf-perf in the bro repo w/ minor updates to test baselines in bro-testing and bro-testing-private repos. The degree to which the changes improve performance are, of course, sensitive to the sampled traffic. I think traffic that contains a lot of files, largish files that Bro only sees in small chunks at a time, and/or many HTTP partial content messages would see the most improvement from the changes. Summary of "hot spot" findings (using 2009-M57-day11-18.trace from bro-testing as "representative" sample): ~8% of total time spent on MIME magic signature matching. This was also the slowest part of file analysis when it relied on libmagic. Using Bro's signatures engine instead is, theoretically, an improvement because the amount of time spent on matching shouldn't increase w/ number of signatures. However, I didn't actually measure much of a performance increase after the switch. No easy optimizations for the RE/signature matching stood out to me, code-wise, that would significantly help -- most time spend appears to be in just having to iterate over each byte of input and the number of bytes being so large that the smallish amount of work each iteration adds up. Possibly, we could be strict about only allowing file-magic signatures that are anchored close to the beginning of the file so that the matching code can bail out earlier when it detects no further matches are possible. ~4.4% of total time spent on requesting file handles from the script-layer. This includes the cost of creating new Vals to pass in as event arguments, the execution of the event handler, and the MD5 hashing of the file handle in to a file ID. The most costly aspect seems to be the first, creating new Vals, while the later two, hashing and execution of the event handler, weren't significant. For the moment, the best way I can think to reduce the cost of file handle requests is just to not make them as often (or at all if it's possible). Being more aggressive about caching file IDs or just generating file handle+ID internally, I was able to reduce this from 4.4% to 1.7%. ~2% of total time spent on updating fa_file record field values (e.g. seen bytes, missed bytes, last active time) The fa_file Val is always kept up-to-date as new info is available, but this results in a lot of new Val creations and Val destructions. Similar to connection records, it may be possible to only update the fa_file record "on-demand" when it's needed for an event and thus reduce this cost. I don't like that much as I've always found that behavior confusing and it makes it more difficult to "poll" those values at the script-layer (ConnPolling currently uses a hack to force updates, and there's no equivalent hack at the moment for fa_file records). How ugly would it be (or would it even work) to have an interface to change Val's internal values (e.g. Val::val.uint_val) so a new Val doesn't have to be created to update fields in a fa_file record? Or is there another option? ~0.5% of total time spent on internal file ID string to File* lookups. This seems like a reasonable cost. Don't think it would be that beneficial for file_analysis::Manager to have an interface that exposes File* to callers in order to avoid the lookup. > Investigate further improvements to file analysis performance > ------------------------------------------------------------- > > Key: BIT-1141 > URL: https://bro-tracker.atlassian.net/browse/BIT-1141 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > Some further ideas for measuring and improving the performance of maintaining the handles were floating around. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 10:51:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 24 Apr 2014 12:51:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1141) Investigate further improvements to file analysis performance In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1141: --------------------------- Status: Merge Request (was: Open) > Investigate further improvements to file analysis performance > ------------------------------------------------------------- > > Key: BIT-1141 > URL: https://bro-tracker.atlassian.net/browse/BIT-1141 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > Some further ideas for measuring and improving the performance of maintaining the handles were floating around. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 11:48:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 24 Apr 2014 13:48:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1156) DNS analyzer parses TXT records imcompletely In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1156: --------------------------- Assignee: Jon Siwek (was: Robin Sommer) > DNS analyzer parses TXT records imcompletely > -------------------------------------------- > > Key: BIT-1156 > URL: https://bro-tracker.atlassian.net/browse/BIT-1156 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > The payload of DNS TXT records can consist of multiple character strings but the DNS analyzer parses out only the first. We should parse them out all and then probably concatenate into a single string to pass to the event, separated with semicolons or something. > I have a trace with an example but it would need anonymization before inclusion into the test suite. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 14:34:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 24 Apr 2014 16:34:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1156) DNS analyzer parses TXT records imcompletely In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1156: --------------------------- Status: Merge Request (was: Open) > DNS analyzer parses TXT records imcompletely > -------------------------------------------- > > Key: BIT-1156 > URL: https://bro-tracker.atlassian.net/browse/BIT-1156 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > The payload of DNS TXT records can consist of multiple character strings but the DNS analyzer parses out only the first. We should parse them out all and then probably concatenate into a single string to pass to the event, separated with semicolons or something. > I have a trace with an example but it would need anonymization before inclusion into the test suite. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 14:34:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 24 Apr 2014 16:34:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1156) DNS analyzer parses TXT records imcompletely In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16219#comment-16219 ] Jon Siwek commented on BIT-1156: -------------------------------- topic/jsiwek/bit-1156 in bro, bro-testing, bro-testing-private > DNS analyzer parses TXT records imcompletely > -------------------------------------------- > > Key: BIT-1156 > URL: https://bro-tracker.atlassian.net/browse/BIT-1156 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > The payload of DNS TXT records can consist of multiple character strings but the DNS analyzer parses out only the first. We should parse them out all and then probably concatenate into a single string to pass to the event, separated with semicolons or something. > I have a trace with an example but it would need anonymization before inclusion into the test suite. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 14:57:07 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Thu, 24 Apr 2014 16:57:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1178) SSL/TLS analyzer does not abort early enough on non-ssl connections In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1178: -------------------------------- Status: Merge Request (was: Open) > SSL/TLS analyzer does not abort early enough on non-ssl connections > ------------------------------------------------------------------- > > Key: BIT-1178 > URL: https://bro-tracker.atlassian.net/browse/BIT-1178 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Bernhard Amann > Fix For: 2.3 > > > Some sites see quite a bit of non-ssl traffic on port 443. At the moment, the SSL analyzer manages to parse quite a lot of this non-ssl traffic and generates ssl-events for it (including sending things to the certificate analyzer which generates warnings). > We should probably try to abort much earlier. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 14:57:07 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Thu, 24 Apr 2014 16:57:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1178) SSL/TLS analyzer does not abort early enough on non-ssl connections In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220#comment-16220 ] Bernhard Amann commented on BIT-1178: ------------------------------------- topic/bernhard/ssl-analyzer contains fixes to all the aforementioned issues. The branch... * removes the TLS state machine * fixes TLS protocol version detection. It also should bail-out correctly on non-tls-connections now * adds support for a few TLS extensions, including server_name (which might work slightly better now), alpn and ec-curves. * also adds support for the heartbeat events and the heartbleed detector script * adds very basic support for OCSP stapling (one can get access to the data). I think it should be ready for merging as it is. I hope I did not break anything... github diff link: https://github.com/bro/bro/compare/topic;bernhard;ssl-analyzer There still are a few issues that I might fix in the future - e.g. I might add real ocsp stapling support. However, those seem much less urgent and probably will not be interesting to a lot of users at the moment. > SSL/TLS analyzer does not abort early enough on non-ssl connections > ------------------------------------------------------------------- > > Key: BIT-1178 > URL: https://bro-tracker.atlassian.net/browse/BIT-1178 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Bernhard Amann > Fix For: 2.3 > > > Some sites see quite a bit of non-ssl traffic on port 443. At the moment, the SSL analyzer manages to parse quite a lot of this non-ssl traffic and generates ssl-events for it (including sending things to the certificate analyzer which generates warnings). > We should probably try to abort much earlier. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 16:00:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 24 Apr 2014 18:00:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1141) Investigate further improvements to file analysis performance In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221#comment-16221 ] Robin Sommer commented on BIT-1141: ----------------------------------- Two questions: (1) {code} typedef std::set AcceptingSet; typedef std::map AcceptingMatchSet; {code} Did you change these to improve performance, or to simplify the code? I'm actually wondering about performance here as set/map can potentially be expensive in particular for small sizes (compared to using a vector for example), and these will be instantiated and manipulated quite often. (2) Baseline/tests.m57-long/http.log: some MIME types change from text/html to text/plain, is that expected? > Investigate further improvements to file analysis performance > ------------------------------------------------------------- > > Key: BIT-1141 > URL: https://bro-tracker.atlassian.net/browse/BIT-1141 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > Some further ideas for measuring and improving the performance of maintaining the handles were floating around. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 16:09:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 24 Apr 2014 18:09:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1141) Investigate further improvements to file analysis performance In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222#comment-16222 ] Robin Sommer commented on BIT-1141: ----------------------------------- {quote} How ugly would it be (or would it even work) to have an interface to change Val's internal values (e.g. Val::val.uint_val) so a new Val doesn't have to be created to update fields in a fa_file record? {quote} Not sure that would be safe. I believe the Val can end up being shared with other locations which wouldn't expect it to change. Say if somebody assigned the current value to somewhere else, then when it would later be changed under the hood it would show up there too. > Investigate further improvements to file analysis performance > ------------------------------------------------------------- > > Key: BIT-1141 > URL: https://bro-tracker.atlassian.net/browse/BIT-1141 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > Some further ideas for measuring and improving the performance of maintaining the handles were floating around. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 16:12:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 24 Apr 2014 18:12:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1141) Investigate further improvements to file analysis performance In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16223#comment-16223 ] Robin Sommer commented on BIT-1141: ----------------------------------- Here are the performance improvements I'm seeing: {code} [ 28%] tests.m57-short ... failed (-15.8%) [ 71%] tests.ipv6 ... failed (-4.9%) [ 85%] tests.m57-long ... failed (-2.6%) [ 50%] tests.short ... failed (-1.4%) [ 75%] tests.medium ... failed (-1.1%) {code} Pretty neat (though as you say, also clearly traffic dependent). > Investigate further improvements to file analysis performance > ------------------------------------------------------------- > > Key: BIT-1141 > URL: https://bro-tracker.atlassian.net/browse/BIT-1141 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > Some further ideas for measuring and improving the performance of maintaining the handles were floating around. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 16:16:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 24 Apr 2014 18:16:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1141) Investigate further improvements to file analysis performance In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221#comment-16221 ] Robin Sommer edited comment on BIT-1141 at 4/24/14 6:15 PM: ------------------------------------------------------------ Two questions: (1) {code} typedef std::set AcceptingSet; typedef std::map AcceptingMatchSet; {code} I'm actually wondering about performance here as set/map can potentially be expensive in particular for small sizes (compared to using a vector for example), and these will be instantiated and manipulated quite often. Put differently: I wouldn't be sure that using a set here is necessarily faster overall than a list as long as there's just a few elements in there. Were you able to confirm that? (2) Baseline/tests.m57-long/http.log: some MIME types change from text/html to text/plain, is that expected? was (Author: robin): Two questions: (1) {code} typedef std::set AcceptingSet; typedef std::map AcceptingMatchSet; {code} Did you change these to improve performance, or to simplify the code? I'm actually wondering about performance here as set/map can potentially be expensive in particular for small sizes (compared to using a vector for example), and these will be instantiated and manipulated quite often. (2) Baseline/tests.m57-long/http.log: some MIME types change from text/html to text/plain, is that expected? > Investigate further improvements to file analysis performance > ------------------------------------------------------------- > > Key: BIT-1141 > URL: https://bro-tracker.atlassian.net/browse/BIT-1141 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > Some further ideas for measuring and improving the performance of maintaining the handles were floating around. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 16:16:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 24 Apr 2014 18:16:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1141) Investigate further improvements to file analysis performance In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221#comment-16221 ] Robin Sommer edited comment on BIT-1141 at 4/24/14 6:15 PM: ------------------------------------------------------------ Two questions: (1) {code} typedef std::set AcceptingSet; typedef std::map AcceptingMatchSet; {code} I'm actually wondering about performance here as set/map can potentially be expensive in particular for small sizes (compared to using a vector for example), and these will be instantiated and manipulated quite often. Put differently: I wouldn't be sure that using a set here is necessarily faster overall than a list as long as there's just a few elements in there. Were you able to confirm that? (2) Baseline/tests.m57-long/http.log: some MIME types change from text/html to text/plain, is that expected? (Update: Ah, is that the bof_buffer_size change?) was (Author: robin): Two questions: (1) {code} typedef std::set AcceptingSet; typedef std::map AcceptingMatchSet; {code} I'm actually wondering about performance here as set/map can potentially be expensive in particular for small sizes (compared to using a vector for example), and these will be instantiated and manipulated quite often. Put differently: I wouldn't be sure that using a set here is necessarily faster overall than a list as long as there's just a few elements in there. Were you able to confirm that? (2) Baseline/tests.m57-long/http.log: some MIME types change from text/html to text/plain, is that expected? > Investigate further improvements to file analysis performance > ------------------------------------------------------------- > > Key: BIT-1141 > URL: https://bro-tracker.atlassian.net/browse/BIT-1141 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > Some further ideas for measuring and improving the performance of maintaining the handles were floating around. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 16:42:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 24 Apr 2014 18:42:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1156) DNS analyzer parses TXT records imcompletely In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16224#comment-16224 ] Robin Sommer commented on BIT-1156: ----------------------------------- I don't really like the "TXT ddd xxxx" logging but don't have much of a better idea either right now. > DNS analyzer parses TXT records imcompletely > -------------------------------------------- > > Key: BIT-1156 > URL: https://bro-tracker.atlassian.net/browse/BIT-1156 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > The payload of DNS TXT records can consist of multiple character strings but the DNS analyzer parses out only the first. We should parse them out all and then probably concatenate into a single string to pass to the event, separated with semicolons or something. > I have a trace with an example but it would need anonymization before inclusion into the test suite. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 17:06:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 24 Apr 2014 19:06:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1185) topic/dnthayer/broctld-work In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1185: ------------------------------ Status: Open (was: Merge Request) > topic/dnthayer/broctld-work > --------------------------- > > Key: BIT-1185 > URL: https://bro-tracker.atlassian.net/browse/BIT-1185 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Reporter: Daniel Thayer > Fix For: 2.3 > > > This branch contains some code cleanup and also fixes or improves the > following issues: > The df, exec, and top commands now run only once per host. > Avoid reporting same disk check error msg multiple times for same host. > Improve output column formatting. > Added warning to do a "broctl install" if broctl or node config changes. > Don't email about "$total" pseudo-node not receiving any packets. > Remove unused "home" broctl option. > Changed plugin API hosts() function to be more useful. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 18:17:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 24 Apr 2014 20:17:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-348) Reassembler integer overflow issues. Data not delivered after 2GB In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16225#comment-16225 ] Robin Sommer commented on BIT-348: ---------------------------------- Is there a reason for keeping the seq_delta() function at all now? If we just replaced all calls with "a-b", that would help readability I think. > Reassembler integer overflow issues. Data not delivered after 2GB > ----------------------------------------------------------------- > > Key: BIT-348 > URL: https://bro-tracker.atlassian.net/browse/BIT-348 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: gregor > Assignee: Robin Sommer > Priority: High > Labels: inttypes > Fix For: 2.3 > > > {noformat} > #!rst > The TCP Reassembler does not deliver any data to analyzers after the first 2GB due to signed integer overflow (Actually it will deliver again between 4--6GB, etc.) This happens silently, i.e., without content_gap events or Undelivered calls. > This report superseded BIT-315, BIT-137 > The TCP Reassembler (and Reassem) base class use ``int`` to keep track of sequence numbers and ``seq_delta`` to check for differences. If a connection exceeds 2GB, the relative sequence numbers (int) used by the Reassembler become negative. While many parts of the Reassembler still work (because seq_delta still reports the correct difference) some parts do not. In particular ``seq_to_skip`` is broken (and fails silently). There might well be other parts of the Reassembler that fail > silently as well, that I haven't found yet. > See Comments in TCP_Reassembler.cc for more details. > The Reassembler should use int64. However this will require deep changes to the Reassembler and the TCP Analyzer and TCP_Endpoint classes (since we also store sequence numbers there). Also, the analyzer framework will need tweaks as well (e.g., Undelivered uses ``int`` for sequence numbers, also has to go to 64 bit) > As a hotfix that seems to work I disabled the ``seq_to_skip`` features. It wasn't used by any analyzer or policy script (Note, that seq_to_skip is different from skip_deliveries). Hotfix is in > topic/gregor/reassembler-hotfix > {noformat} -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 18:52:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 24 Apr 2014 20:52:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1156) DNS analyzer parses TXT records imcompletely In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1156: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > DNS analyzer parses TXT records imcompletely > -------------------------------------------- > > Key: BIT-1156 > URL: https://bro-tracker.atlassian.net/browse/BIT-1156 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > The payload of DNS TXT records can consist of multiple character strings but the DNS analyzer parses out only the first. We should parse them out all and then probably concatenate into a single string to pass to the event, separated with semicolons or something. > I have a trace with an example but it would need anonymization before inclusion into the test suite. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 18:52:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 24 Apr 2014 20:52:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1168) Add Java version to software framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1168: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Add Java version to software framework > -------------------------------------- > > Key: BIT-1168 > URL: https://bro-tracker.atlassian.net/browse/BIT-1168 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Affects Versions: 2.2 > Reporter: Brian Little > Assignee: Seth Hall > Priority: Low > Labels: framework, java, software > Attachments: bro-java-software.patch > > > A small patch to add Java into the list of Mozilla user agents searched for (parse_mozilla function). This is useful for the vulnerable software check. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 18:52:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 24 Apr 2014 20:52:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1187) topic/jsiwek/remove-val-attribs In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1187: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > topic/jsiwek/remove-val-attribs > ------------------------------- > > Key: BIT-1187 > URL: https://bro-tracker.atlassian.net/browse/BIT-1187 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, Broccoli > Affects Versions: git/master > Reporter: Jon Siwek > Fix For: 2.3 > > > Dead code removal in bro and broccoli repos. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 18:52:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 24 Apr 2014 20:52:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1178) SSL/TLS analyzer does not abort early enough on non-ssl connections In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1178: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > SSL/TLS analyzer does not abort early enough on non-ssl connections > ------------------------------------------------------------------- > > Key: BIT-1178 > URL: https://bro-tracker.atlassian.net/browse/BIT-1178 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Assignee: Bernhard Amann > Fix For: 2.3 > > > Some sites see quite a bit of non-ssl traffic on port 443. At the moment, the SSL analyzer manages to parse quite a lot of this non-ssl traffic and generates ssl-events for it (including sending things to the certificate analyzer which generates warnings). > We should probably try to abort much earlier. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 20:22:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 24 Apr 2014 22:22:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-348) Reassembler integer overflow issues. Data not delivered after 2GB In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226#comment-16226 ] Robin Sommer commented on BIT-348: ---------------------------------- Alright, I'm giving up on reviewing the changes in TCP.cc, with the refactoring it's really hard to track what's part of the 66-bit changes vs. what's just moved around. I would certainly have preferred having the refactoring separately, though I understand that it's all related. > Reassembler integer overflow issues. Data not delivered after 2GB > ----------------------------------------------------------------- > > Key: BIT-348 > URL: https://bro-tracker.atlassian.net/browse/BIT-348 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: gregor > Assignee: Robin Sommer > Priority: High > Labels: inttypes > Fix For: 2.3 > > > {noformat} > #!rst > The TCP Reassembler does not deliver any data to analyzers after the first 2GB due to signed integer overflow (Actually it will deliver again between 4--6GB, etc.) This happens silently, i.e., without content_gap events or Undelivered calls. > This report superseded BIT-315, BIT-137 > The TCP Reassembler (and Reassem) base class use ``int`` to keep track of sequence numbers and ``seq_delta`` to check for differences. If a connection exceeds 2GB, the relative sequence numbers (int) used by the Reassembler become negative. While many parts of the Reassembler still work (because seq_delta still reports the correct difference) some parts do not. In particular ``seq_to_skip`` is broken (and fails silently). There might well be other parts of the Reassembler that fail > silently as well, that I haven't found yet. > See Comments in TCP_Reassembler.cc for more details. > The Reassembler should use int64. However this will require deep changes to the Reassembler and the TCP Analyzer and TCP_Endpoint classes (since we also store sequence numbers there). Also, the analyzer framework will need tweaks as well (e.g., Undelivered uses ``int`` for sequence numbers, also has to go to 64 bit) > As a hotfix that seems to work I disabled the ``seq_to_skip`` features. It wasn't used by any analyzer or policy script (Note, that seq_to_skip is different from skip_deliveries). Hotfix is in > topic/gregor/reassembler-hotfix > {noformat} -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 20:26:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 24 Apr 2014 22:26:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-348) Reassembler integer overflow issues. Data not delivered after 2GB In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227#comment-16227 ] Robin Sommer commented on BIT-348: ---------------------------------- I did some more testing on a large trace, and I am seeing differences in the duration of a few connections. Have you seen that as well / do you have an idea where that's coming from? Here's an example: {code} before: 1359400833.646398 CHrUQS2wXshH59NEb6 XXXX 35752 YYYY 80 tcp http 3.497443 380 428 SF 429 ShADaFfRR 8 712 4 172 (empty) after: 1359400833.646398 C9yL7F1JnxhPtfoLMi XXXX 35752 YYYY 80 tcp http 0.240429 380 428 SF 429 ShADaFfRR 8 712 4 172 (empty) {code} (I didn't sync the rnd seed for the uids). I'll provide the trace if you want to look at it. > Reassembler integer overflow issues. Data not delivered after 2GB > ----------------------------------------------------------------- > > Key: BIT-348 > URL: https://bro-tracker.atlassian.net/browse/BIT-348 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: gregor > Assignee: Robin Sommer > Priority: High > Labels: inttypes > Fix For: 2.3 > > > {noformat} > #!rst > The TCP Reassembler does not deliver any data to analyzers after the first 2GB due to signed integer overflow (Actually it will deliver again between 4--6GB, etc.) This happens silently, i.e., without content_gap events or Undelivered calls. > This report superseded BIT-315, BIT-137 > The TCP Reassembler (and Reassem) base class use ``int`` to keep track of sequence numbers and ``seq_delta`` to check for differences. If a connection exceeds 2GB, the relative sequence numbers (int) used by the Reassembler become negative. While many parts of the Reassembler still work (because seq_delta still reports the correct difference) some parts do not. In particular ``seq_to_skip`` is broken (and fails silently). There might well be other parts of the Reassembler that fail > silently as well, that I haven't found yet. > See Comments in TCP_Reassembler.cc for more details. > The Reassembler should use int64. However this will require deep changes to the Reassembler and the TCP Analyzer and TCP_Endpoint classes (since we also store sequence numbers there). Also, the analyzer framework will need tweaks as well (e.g., Undelivered uses ``int`` for sequence numbers, also has to go to 64 bit) > As a hotfix that seems to work I disabled the ``seq_to_skip`` features. It wasn't used by any analyzer or policy script (Note, that seq_to_skip is different from skip_deliveries). Hotfix is in > topic/gregor/reassembler-hotfix > {noformat} -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Thu Apr 24 20:26:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 24 Apr 2014 22:26:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-348) Reassembler integer overflow issues. Data not delivered after 2GB In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226#comment-16226 ] Robin Sommer edited comment on BIT-348 at 4/24/14 10:25 PM: ------------------------------------------------------------ Alright, I'm giving up on reviewing the changes in TCP.cc, with the refactoring it's really hard to track what's part of the 64-bit changes vs. what's just moved around. I would certainly have preferred having the refactoring separately, though I understand that it's all related. was (Author: robin): Alright, I'm giving up on reviewing the changes in TCP.cc, with the refactoring it's really hard to track what's part of the 66-bit changes vs. what's just moved around. I would certainly have preferred having the refactoring separately, though I understand that it's all related. > Reassembler integer overflow issues. Data not delivered after 2GB > ----------------------------------------------------------------- > > Key: BIT-348 > URL: https://bro-tracker.atlassian.net/browse/BIT-348 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: gregor > Assignee: Robin Sommer > Priority: High > Labels: inttypes > Fix For: 2.3 > > > {noformat} > #!rst > The TCP Reassembler does not deliver any data to analyzers after the first 2GB due to signed integer overflow (Actually it will deliver again between 4--6GB, etc.) This happens silently, i.e., without content_gap events or Undelivered calls. > This report superseded BIT-315, BIT-137 > The TCP Reassembler (and Reassem) base class use ``int`` to keep track of sequence numbers and ``seq_delta`` to check for differences. If a connection exceeds 2GB, the relative sequence numbers (int) used by the Reassembler become negative. While many parts of the Reassembler still work (because seq_delta still reports the correct difference) some parts do not. In particular ``seq_to_skip`` is broken (and fails silently). There might well be other parts of the Reassembler that fail > silently as well, that I haven't found yet. > See Comments in TCP_Reassembler.cc for more details. > The Reassembler should use int64. However this will require deep changes to the Reassembler and the TCP Analyzer and TCP_Endpoint classes (since we also store sequence numbers there). Also, the analyzer framework will need tweaks as well (e.g., Undelivered uses ``int`` for sequence numbers, also has to go to 64 bit) > As a hotfix that seems to work I disabled the ``seq_to_skip`` features. It wasn't used by any analyzer or policy script (Note, that seq_to_skip is different from skip_deliveries). Hotfix is in > topic/gregor/reassembler-hotfix > {noformat} -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From noreply at bro.org Fri Apr 25 00:00:12 2014 From: noreply at bro.org (Merge Tracker) Date: Fri, 25 Apr 2014 00:00:12 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404250700.s3P70Clx017925@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1177 [1] Bro Bernhard Amann Seth Hall 2014-04-21 2.3 Normal SumStats dynamic updates do not work in cluster mode BIT-1141 [2] Bro Robin Sommer Jon Siwek 2014-04-24 2.3 Normal Investigate further improvements to file analysis performance BIT-348 [3] Bro gregor Robin Sommer 2014-04-24 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ---------- ---------- --------------------------------------------------------------------------- #4 [4] time-machine mareq [5] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [6] #3 [7] time-machine mareq [8] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [9] #2 [10] time-machine mareq [11] 2014-04-09 Query interval start/end is now taken into account. [12] #1 [13] time-machine mareq [14] 2014-03-19 TM-16: Really skip VLAN header for indexing. [15] [1] BIT-1177 https://bro-tracker.atlassian.net/browse/BIT-1177 [2] BIT-1141 https://bro-tracker.atlassian.net/browse/BIT-1141 [3] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [4] Pull Request #4 https://github.com/bro/time-machine/pull/4 [5] mareq https://github.com/mareq [6] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [7] Pull Request #3 https://github.com/bro/time-machine/pull/3 [8] mareq https://github.com/mareq [9] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [10] Pull Request #2 https://github.com/bro/time-machine/pull/2 [11] mareq https://github.com/mareq [12] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [13] Pull Request #1 https://github.com/bro/time-machine/pull/1 [14] mareq https://github.com/mareq [15] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Fri Apr 25 08:05:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 25 Apr 2014 10:05:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1150) X509 updates In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1150: ------------------------------ Status: Merge Request (was: Reopened) > X509 updates > ------------ > > Key: BIT-1150 > URL: https://bro-tracker.atlassian.net/browse/BIT-1150 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Bernhard Amann > Fix For: 2.3 > > Attachments: signature.asc > > -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Fri Apr 25 14:34:07 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Fri, 25 Apr 2014 16:34:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1185) topic/dnthayer/broctld-work In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1185: ------------------------------- Status: Merge Request (was: Open) > topic/dnthayer/broctld-work > --------------------------- > > Key: BIT-1185 > URL: https://bro-tracker.atlassian.net/browse/BIT-1185 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Reporter: Daniel Thayer > Fix For: 2.3 > > > This branch contains some code cleanup and also fixes or improves the > following issues: > The df, exec, and top commands now run only once per host. > Avoid reporting same disk check error msg multiple times for same host. > Improve output column formatting. > Added warning to do a "broctl install" if broctl or node config changes. > Don't email about "$total" pseudo-node not receiving any packets. > Remove unused "home" broctl option. > Changed plugin API hosts() function to be more useful. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Fri Apr 25 14:34:07 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Fri, 25 Apr 2014 16:34:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1185) topic/dnthayer/broctld-work In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16228#comment-16228 ] Daniel Thayer commented on BIT-1185: ------------------------------------ I've changed the code so that it now just compares a hash of the configuration, instead of comparing each (key,value) pair. > topic/dnthayer/broctld-work > --------------------------- > > Key: BIT-1185 > URL: https://bro-tracker.atlassian.net/browse/BIT-1185 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Reporter: Daniel Thayer > Fix For: 2.3 > > > This branch contains some code cleanup and also fixes or improves the > following issues: > The df, exec, and top commands now run only once per host. > Avoid reporting same disk check error msg multiple times for same host. > Improve output column formatting. > Added warning to do a "broctl install" if broctl or node config changes. > Don't email about "$total" pseudo-node not receiving any packets. > Remove unused "home" broctl option. > Changed plugin API hosts() function to be more useful. -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From noreply at bro.org Sat Apr 26 00:00:19 2014 From: noreply at bro.org (Merge Tracker) Date: Sat, 26 Apr 2014 00:00:19 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404260700.s3Q70JrB024000@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- -------------- ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1185 [1] BroControl Daniel Thayer - 2014-04-25 2.3 Normal topic/dnthayer/broctld-work [2] BIT-1177 [3] Bro Bernhard Amann Seth Hall 2014-04-21 2.3 Normal SumStats dynamic updates do not work in cluster mode BIT-1150 [4] Bro Robin Sommer Bernhard Amann 2014-04-25 2.3 Normal X509 updates BIT-1141 [5] Bro Robin Sommer Jon Siwek 2014-04-24 2.3 Normal Investigate further improvements to file analysis performance BIT-348 [6] Bro gregor Robin Sommer 2014-04-24 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ---------- ---------- --------------------------------------------------------------------------- #4 [7] time-machine mareq [8] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [9] #3 [10] time-machine mareq [11] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [12] #2 [13] time-machine mareq [14] 2014-04-09 Query interval start/end is now taken into account. [15] #1 [16] time-machine mareq [17] 2014-03-19 TM-16: Really skip VLAN header for indexing. [18] [1] BIT-1185 https://bro-tracker.atlassian.net/browse/BIT-1185 [2] broctld-work https://github.com/bro/brocontrol/tree/topic/dnthayer/broctld-work [3] BIT-1177 https://bro-tracker.atlassian.net/browse/BIT-1177 [4] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [5] BIT-1141 https://bro-tracker.atlassian.net/browse/BIT-1141 [6] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [7] Pull Request #4 https://github.com/bro/time-machine/pull/4 [8] mareq https://github.com/mareq [9] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [10] Pull Request #3 https://github.com/bro/time-machine/pull/3 [11] mareq https://github.com/mareq [12] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [13] Pull Request #2 https://github.com/bro/time-machine/pull/2 [14] mareq https://github.com/mareq [15] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [16] Pull Request #1 https://github.com/bro/time-machine/pull/1 [17] mareq https://github.com/mareq [18] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From noreply at bro.org Sun Apr 27 00:00:15 2014 From: noreply at bro.org (Merge Tracker) Date: Sun, 27 Apr 2014 00:00:15 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404270700.s3R70FCM008859@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- -------------- ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1185 [1] BroControl Daniel Thayer - 2014-04-25 2.3 Normal topic/dnthayer/broctld-work [2] BIT-1177 [3] Bro Bernhard Amann Seth Hall 2014-04-21 2.3 Normal SumStats dynamic updates do not work in cluster mode BIT-1150 [4] Bro Robin Sommer Bernhard Amann 2014-04-25 2.3 Normal X509 updates BIT-1141 [5] Bro Robin Sommer Jon Siwek 2014-04-24 2.3 Normal Investigate further improvements to file analysis performance BIT-348 [6] Bro gregor Robin Sommer 2014-04-24 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ---------- ---------- --------------------------------------------------------------------------- #4 [7] time-machine mareq [8] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [9] #3 [10] time-machine mareq [11] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [12] #2 [13] time-machine mareq [14] 2014-04-09 Query interval start/end is now taken into account. [15] #1 [16] time-machine mareq [17] 2014-03-19 TM-16: Really skip VLAN header for indexing. [18] [1] BIT-1185 https://bro-tracker.atlassian.net/browse/BIT-1185 [2] broctld-work https://github.com/bro/brocontrol/tree/topic/dnthayer/broctld-work [3] BIT-1177 https://bro-tracker.atlassian.net/browse/BIT-1177 [4] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [5] BIT-1141 https://bro-tracker.atlassian.net/browse/BIT-1141 [6] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [7] Pull Request #4 https://github.com/bro/time-machine/pull/4 [8] mareq https://github.com/mareq [9] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [10] Pull Request #3 https://github.com/bro/time-machine/pull/3 [11] mareq https://github.com/mareq [12] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [13] Pull Request #2 https://github.com/bro/time-machine/pull/2 [14] mareq https://github.com/mareq [15] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [16] Pull Request #1 https://github.com/bro/time-machine/pull/1 [17] mareq https://github.com/mareq [18] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Sun Apr 27 16:36:07 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Sun, 27 Apr 2014 18:36:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1189) merge topic/bernhard/ec-curve In-Reply-To: References: Message-ID: Bernhard Amann created BIT-1189: ----------------------------------- Summary: merge topic/bernhard/ec-curve Key: BIT-1189 URL: https://bro-tracker.atlassian.net/browse/BIT-1189 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.3 topic/bernhard/ec-curve adds support for recognizing which curve was chosen in a connection using ECDH/ECDHE as well as returning the DH parameters for DHE/DH-Anon. Furthermore, it adds a small policy script that warns on weak certificate keys or DH-parameters. Github diff link: https://github.com/bro/bro/compare/topic;bernhard;ec-curve -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Sun Apr 27 16:36:07 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Sun, 27 Apr 2014 18:36:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1189) merge topic/bernhard/ec-curve In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1189: -------------------------------- Status: Merge Request (was: Open) > merge topic/bernhard/ec-curve > ----------------------------- > > Key: BIT-1189 > URL: https://bro-tracker.atlassian.net/browse/BIT-1189 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Fix For: 2.3 > > > topic/bernhard/ec-curve adds support for recognizing which curve was chosen in a connection using ECDH/ECDHE as well as returning the DH parameters for DHE/DH-Anon. > Furthermore, it adds a small policy script that warns on weak certificate keys or DH-parameters. > Github diff link: https://github.com/bro/bro/compare/topic;bernhard;ec-curve -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From jira at bro-tracker.atlassian.net Sun Apr 27 16:39:07 2014 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Sun, 27 Apr 2014 18:39:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1189) merge topic/bernhard/ec-curve In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16229#comment-16229 ] Bernhard Amann commented on BIT-1189: ------------------------------------- I am sorry about the long switch statements in the pac-file. Sadly it is very difficult to avoid that. > merge topic/bernhard/ec-curve > ----------------------------- > > Key: BIT-1189 > URL: https://bro-tracker.atlassian.net/browse/BIT-1189 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Bernhard Amann > Fix For: 2.3 > > > topic/bernhard/ec-curve adds support for recognizing which curve was chosen in a connection using ECDH/ECDHE as well as returning the DH parameters for DHE/DH-Anon. > Furthermore, it adds a small policy script that warns on weak certificate keys or DH-parameters. > Github diff link: https://github.com/bro/bro/compare/topic;bernhard;ec-curve -- This message was sent by Atlassian JIRA (v6.3-OD-02-026#6318) From noreply at bro.org Mon Apr 28 00:00:17 2014 From: noreply at bro.org (Merge Tracker) Date: Mon, 28 Apr 2014 00:00:17 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404280700.s3S70HdS024700@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- -------------- ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1189 [1] Bro Bernhard Amann - 2014-04-27 2.3 Normal merge topic/bernhard/ec-curve BIT-1185 [2] BroControl Daniel Thayer - 2014-04-25 2.3 Normal topic/dnthayer/broctld-work [3] BIT-1177 [4] Bro Bernhard Amann Seth Hall 2014-04-21 2.3 Normal SumStats dynamic updates do not work in cluster mode BIT-1150 [5] Bro Robin Sommer Bernhard Amann 2014-04-25 2.3 Normal X509 updates BIT-1141 [6] Bro Robin Sommer Jon Siwek 2014-04-24 2.3 Normal Investigate further improvements to file analysis performance BIT-348 [7] Bro gregor Robin Sommer 2014-04-24 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ---------- ---------- ---------------------------------------------------------------------------- #4 [8] time-machine mareq [9] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [10] #3 [11] time-machine mareq [12] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [13] #2 [14] time-machine mareq [15] 2014-04-09 Query interval start/end is now taken into account. [16] #1 [17] time-machine mareq [18] 2014-03-19 TM-16: Really skip VLAN header for indexing. [19] [1] BIT-1189 https://bro-tracker.atlassian.net/browse/BIT-1189 [2] BIT-1185 https://bro-tracker.atlassian.net/browse/BIT-1185 [3] broctld-work https://github.com/bro/brocontrol/tree/topic/dnthayer/broctld-work [4] BIT-1177 https://bro-tracker.atlassian.net/browse/BIT-1177 [5] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [6] BIT-1141 https://bro-tracker.atlassian.net/browse/BIT-1141 [7] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [8] Pull Request #4 https://github.com/bro/time-machine/pull/4 [9] mareq https://github.com/mareq [10] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [11] Pull Request #3 https://github.com/bro/time-machine/pull/3 [12] mareq https://github.com/mareq [13] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [14] Pull Request #2 https://github.com/bro/time-machine/pull/2 [15] mareq https://github.com/mareq [16] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [17] Pull Request #1 https://github.com/bro/time-machine/pull/1 [18] mareq https://github.com/mareq [19] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Mon Apr 28 07:19:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 28 Apr 2014 09:19:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1141) Investigate further improvements to file analysis performance In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300#comment-16300 ] Jon Siwek commented on BIT-1141: -------------------------------- {quote} I'm actually wondering about performance here as set/map can potentially be expensive in particular for small sizes (compared to using a vector for example), and these will be instantiated and manipulated quite often. Put differently: I wouldn't be sure that using a set here is necessarily faster overall than a list as long as there's just a few elements in there. Were you able to confirm that? {quote} It can be questionable -- in other places I've tried replacing lists with sets/maps and have measured some performance decrease. But in this case, the difference seemed negligible... I think it was a slight improvement possibly because file signatures will now more commonly have multiple matches where before only a single protocol signature would match. Code-wise, it did simplify things, though I guess that's only a minor/weak argument for the change. {quote} Baseline/tests.m57-long/http.log: some MIME types change from text/html to text/plain, is that expected? (Update: Ah, is that the bof_buffer_size change?) {quote} Yes, that was from the change to restrict how much data may be fed in the the file MIME signature matching stuff to be no greater than the bof_buffer_size field -- as that's the original intent and also the way it's documented. > Investigate further improvements to file analysis performance > ------------------------------------------------------------- > > Key: BIT-1141 > URL: https://bro-tracker.atlassian.net/browse/BIT-1141 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > Some further ideas for measuring and improving the performance of maintaining the handles were floating around. -- This message was sent by Atlassian JIRA (v6.3-OD-03-012#6321) From jira at bro-tracker.atlassian.net Mon Apr 28 07:32:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 28 Apr 2014 09:32:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1156) DNS analyzer parses TXT records imcompletely In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16301#comment-16301 ] Jon Siwek commented on BIT-1156: -------------------------------- {quote} I don't really like the "TXT ddd xxxx" logging but don't have much of a better idea either right now. {quote} Yeah, it was just that the DNS logs for such TXT RRs are pretty ambiguous without doing something like that or overhauling how dns.log is formatted (don't have a fully formed idea, but whenever I try to work w/ those scripts it always seems like the scope of what it's doing is too broad/general to do any particular thing accurately/well). > DNS analyzer parses TXT records imcompletely > -------------------------------------------- > > Key: BIT-1156 > URL: https://bro-tracker.atlassian.net/browse/BIT-1156 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.3 > > > The payload of DNS TXT records can consist of multiple character strings but the DNS analyzer parses out only the first. We should parse them out all and then probably concatenate into a single string to pass to the event, separated with semicolons or something. > I have a trace with an example but it would need anonymization before inclusion into the test suite. -- This message was sent by Atlassian JIRA (v6.3-OD-03-012#6321) From jira at bro-tracker.atlassian.net Mon Apr 28 08:05:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 28 Apr 2014 10:05:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-348) Reassembler integer overflow issues. Data not delivered after 2GB In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16302#comment-16302 ] Jon Siwek commented on BIT-348: ------------------------------- {quote} Alright, I'm giving up on reviewing the changes in TCP.cc, with the refactoring it's really hard to track what's part of the 64-bit changes vs. what's just moved around. I would certainly have preferred having the refactoring separately, though I understand that it's all related. {quote} Sorry, but yeah it was "difficult" to treat that separately (without having worked much on that code beforehand). {quote} I did some more testing on a large trace, and I am seeing differences in the duration of a few connections. Have you seen that as well / do you have an idea where that's coming from? {quote} Not sure, a guess is it's treating the end as the FIN exchange instead of the end being the RST(s). If you can send a pcap, I'll look in to it? > Reassembler integer overflow issues. Data not delivered after 2GB > ----------------------------------------------------------------- > > Key: BIT-348 > URL: https://bro-tracker.atlassian.net/browse/BIT-348 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: gregor > Assignee: Robin Sommer > Priority: High > Labels: inttypes > Fix For: 2.3 > > > {noformat} > #!rst > The TCP Reassembler does not deliver any data to analyzers after the first 2GB due to signed integer overflow (Actually it will deliver again between 4--6GB, etc.) This happens silently, i.e., without content_gap events or Undelivered calls. > This report superseded BIT-315, BIT-137 > The TCP Reassembler (and Reassem) base class use ``int`` to keep track of sequence numbers and ``seq_delta`` to check for differences. If a connection exceeds 2GB, the relative sequence numbers (int) used by the Reassembler become negative. While many parts of the Reassembler still work (because seq_delta still reports the correct difference) some parts do not. In particular ``seq_to_skip`` is broken (and fails silently). There might well be other parts of the Reassembler that fail > silently as well, that I haven't found yet. > See Comments in TCP_Reassembler.cc for more details. > The Reassembler should use int64. However this will require deep changes to the Reassembler and the TCP Analyzer and TCP_Endpoint classes (since we also store sequence numbers there). Also, the analyzer framework will need tweaks as well (e.g., Undelivered uses ``int`` for sequence numbers, also has to go to 64 bit) > As a hotfix that seems to work I disabled the ``seq_to_skip`` features. It wasn't used by any analyzer or policy script (Note, that seq_to_skip is different from skip_deliveries). Hotfix is in > topic/gregor/reassembler-hotfix > {noformat} -- This message was sent by Atlassian JIRA (v6.3-OD-03-012#6321) From noreply at bro.org Tue Apr 29 00:00:18 2014 From: noreply at bro.org (Merge Tracker) Date: Tue, 29 Apr 2014 00:00:18 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404290700.s3T70IK6005346@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- -------------- ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1189 [1] Bro Bernhard Amann - 2014-04-27 2.3 Normal merge topic/bernhard/ec-curve BIT-1185 [2] BroControl Daniel Thayer - 2014-04-25 2.3 Normal topic/dnthayer/broctld-work [3] BIT-1177 [4] Bro Bernhard Amann Seth Hall 2014-04-21 2.3 Normal SumStats dynamic updates do not work in cluster mode BIT-1150 [5] Bro Robin Sommer Bernhard Amann 2014-04-25 2.3 Normal X509 updates BIT-1141 [6] Bro Robin Sommer Jon Siwek 2014-04-28 2.3 Normal Investigate further improvements to file analysis performance BIT-348 [7] Bro gregor Robin Sommer 2014-04-28 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ---------- ---------- ---------------------------------------------------------------------------- #4 [8] time-machine mareq [9] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [10] #3 [11] time-machine mareq [12] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [13] #2 [14] time-machine mareq [15] 2014-04-09 Query interval start/end is now taken into account. [16] #1 [17] time-machine mareq [18] 2014-03-19 TM-16: Really skip VLAN header for indexing. [19] [1] BIT-1189 https://bro-tracker.atlassian.net/browse/BIT-1189 [2] BIT-1185 https://bro-tracker.atlassian.net/browse/BIT-1185 [3] broctld-work https://github.com/bro/brocontrol/tree/topic/dnthayer/broctld-work [4] BIT-1177 https://bro-tracker.atlassian.net/browse/BIT-1177 [5] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [6] BIT-1141 https://bro-tracker.atlassian.net/browse/BIT-1141 [7] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [8] Pull Request #4 https://github.com/bro/time-machine/pull/4 [9] mareq https://github.com/mareq [10] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [11] Pull Request #3 https://github.com/bro/time-machine/pull/3 [12] mareq https://github.com/mareq [13] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [14] Pull Request #2 https://github.com/bro/time-machine/pull/2 [15] mareq https://github.com/mareq [16] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [17] Pull Request #1 https://github.com/bro/time-machine/pull/1 [18] mareq https://github.com/mareq [19] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From noreply at bro.org Wed Apr 30 00:00:16 2014 From: noreply at bro.org (Merge Tracker) Date: Wed, 30 Apr 2014 00:00:16 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201404300700.s3U70GEm025815@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- -------------- ---------- ------------- ---------- ----------------------------------------------------------------- BIT-1189 [1] Bro Bernhard Amann - 2014-04-27 2.3 Normal merge topic/bernhard/ec-curve BIT-1185 [2] BroControl Daniel Thayer - 2014-04-25 2.3 Normal topic/dnthayer/broctld-work [3] BIT-1177 [4] Bro Bernhard Amann Seth Hall 2014-04-21 2.3 Normal SumStats dynamic updates do not work in cluster mode BIT-1150 [5] Bro Robin Sommer Bernhard Amann 2014-04-25 2.3 Normal X509 updates BIT-1141 [6] Bro Robin Sommer Jon Siwek 2014-04-28 2.3 Normal Investigate further improvements to file analysis performance BIT-348 [7] Bro gregor Robin Sommer 2014-04-28 2.3 High Reassembler integer overflow issues. Data not delivered after 2GB Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- --------- ---------- ---------------------------------------------------------- d7d5497 [8] bro Jon Siwek 2014-04-29 Improve/standardize some malloc/realloc return val checks. 4b059ea [9] bro Jon Siwek 2014-04-29 Improve file analysis manager shutdown/cleanup. Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ------------ ---------- ---------- ---------------------------------------------------------------------------- #4 [10] time-machine mareq [11] 2014-04-10 When deleting connections hashtable, delete stored connections as well. [12] #3 [13] time-machine mareq [14] 2014-04-10 Correct handling of Linux SLL header and VLAN headers. [15] #2 [16] time-machine mareq [17] 2014-04-09 Query interval start/end is now taken into account. [18] #1 [19] time-machine mareq [20] 2014-03-19 TM-16: Really skip VLAN header for indexing. [21] [1] BIT-1189 https://bro-tracker.atlassian.net/browse/BIT-1189 [2] BIT-1185 https://bro-tracker.atlassian.net/browse/BIT-1185 [3] broctld-work https://github.com/bro/brocontrol/tree/topic/dnthayer/broctld-work [4] BIT-1177 https://bro-tracker.atlassian.net/browse/BIT-1177 [5] BIT-1150 https://bro-tracker.atlassian.net/browse/BIT-1150 [6] BIT-1141 https://bro-tracker.atlassian.net/browse/BIT-1141 [7] BIT-348 https://bro-tracker.atlassian.net/browse/BIT-348 [8] d7d5497 https://github.com/bro/bro/commit/d7d5497436f8b3bf6101adca4e1d2aa2c1113aa0 [9] 4b059ea https://github.com/bro/bro/commit/4b059ea15ac6bba0c5189532bc1c749ec066b1c6 [10] Pull Request #4 https://github.com/bro/time-machine/pull/4 [11] mareq https://github.com/mareq [12] Merge Pull Request #4 with git pull https://github.com/mareq/time-machine.git topic/mareq/memory-leaks [13] Pull Request #3 https://github.com/bro/time-machine/pull/3 [14] mareq https://github.com/mareq [15] Merge Pull Request #3 with git pull https://github.com/mareq/time-machine.git topic/mareq/linktype-linux-sll [16] Pull Request #2 https://github.com/bro/time-machine/pull/2 [17] mareq https://github.com/mareq [18] Merge Pull Request #2 with git pull https://github.com/mareq/time-machine.git topic/mareq/in-memory-query-interval [19] Pull Request #1 https://github.com/bro/time-machine/pull/1 [20] mareq https://github.com/mareq [21] Merge Pull Request #1 with git pull https://github.com/mareq/time-machine.git topic/mareq/tm-16 From jira at bro-tracker.atlassian.net Wed Apr 30 10:15:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 30 Apr 2014 12:15:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-348) Reassembler integer overflow issues. Data not delivered after 2GB In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16303#comment-16303 ] Jon Siwek commented on BIT-348: ------------------------------- {quote} I did some more testing on a large trace, and I am seeing differences in the duration of a few connections. Have you seen that as well / do you have an idea where that's coming from? Here's an example: {code} before: 1359400833.646398 CHrUQS2wXshH59NEb6 XXXX 35752 YYYY 80 tcp http 3.497443 380 428 SF 429 ShADaFfRR 8 712 4 172 (empty) after: 1359400833.646398 C9yL7F1JnxhPtfoLMi XXXX 35752 YYYY 80 tcp http 0.240429 380 428 SF 429 ShADaFfRR 8 712 4 172 (empty) {code} {quote} This is tricky to explain, hang in there... here's basically what the lead up to the FIN exchange looks like: responder: Seq=1, ACK=381 (everything looks fine up to and including this packet) originator: Seq=381, ACK=430 (ack'd an unseen segment... still "fine", Bro can deal with that) originator: Seq=381, ACK=430, FIN (still "fine") responder: Seq=1, ACK=382, FIN (what? It either backed down the sequence number or the peer's last ACK was wrong) The sequence/ack number tracking in both the old code versus the new code behaves similarly: the responder's TCP reassembler will think that everything before 430 is "old" (in this case it's been reported to analyzers as Undelivered) because of the weird ACK, and that the last sequence seen is 1 (which is true, we only know about the SYN, but have not seen a packet carrying data for this endpoint). The difference in the old code versus the new code that matters is in TCP_Reassembler::DataPending: https://github.com/bro/bro/compare/topic;jsiwek;bit-348#diff-a06b37d4ebafffdc8f9f39cca155b753R649 There's a new check that if the the last sequence seen is less than the last sequence that the reassembler is to treat as "old", then consider that case as "no data pending". The pcap you sent me matches this case in the new code on certain calls and ends up returning a different value than the old code. This matters in determining the connection duration because the connection will stop updating it's "last time" if both endpoints are treated as "closed with no pending data". In the old code, the responder endpoint is prevented from satisfying the "no pending data" requirement and so always updates the "last time" on every packet. In the new code, as soon as the FIN exchange is complete, both endpoints meet the "closed with no pending data" requirement and "last time" of the connection no longer updates with subsequent packets. My justification for adding the additional check in TCP_Reassembler::DataPending was/is that the intention of that method is to tell whether there's any holes in the sequence space that could possibly be filled by a later TCP packet delivered out of order. I didn't anticipate this particular scenario, but the logic still holds: it's never possibly to "deliver" any such data because the reassembler has moved on... if it gets stuff from the sequence space it considers "old", it will be dropped. In that sense, there's "no data pending" and I think the new code is more "correct". And the way in which duration is measured seems a bit finicky/arbitrary to begin with, not sure the difference for cases like this are important? Also, just to note: the same connection but w/ sane seq/ack numbers would be handled the same way between code versions, and that way would produce results that are the same as what the new code produces. > Reassembler integer overflow issues. Data not delivered after 2GB > ----------------------------------------------------------------- > > Key: BIT-348 > URL: https://bro-tracker.atlassian.net/browse/BIT-348 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: gregor > Assignee: Robin Sommer > Priority: High > Labels: inttypes > Fix For: 2.3 > > > {noformat} > #!rst > The TCP Reassembler does not deliver any data to analyzers after the first 2GB due to signed integer overflow (Actually it will deliver again between 4--6GB, etc.) This happens silently, i.e., without content_gap events or Undelivered calls. > This report superseded BIT-315, BIT-137 > The TCP Reassembler (and Reassem) base class use ``int`` to keep track of sequence numbers and ``seq_delta`` to check for differences. If a connection exceeds 2GB, the relative sequence numbers (int) used by the Reassembler become negative. While many parts of the Reassembler still work (because seq_delta still reports the correct difference) some parts do not. In particular ``seq_to_skip`` is broken (and fails silently). There might well be other parts of the Reassembler that fail > silently as well, that I haven't found yet. > See Comments in TCP_Reassembler.cc for more details. > The Reassembler should use int64. However this will require deep changes to the Reassembler and the TCP Analyzer and TCP_Endpoint classes (since we also store sequence numbers there). Also, the analyzer framework will need tweaks as well (e.g., Undelivered uses ``int`` for sequence numbers, also has to go to 64 bit) > As a hotfix that seems to work I disabled the ``seq_to_skip`` features. It wasn't used by any analyzer or policy script (Note, that seq_to_skip is different from skip_deliveries). Hotfix is in > topic/gregor/reassembler-hotfix > {noformat} -- This message was sent by Atlassian JIRA (v6.3-OD-03-012#6321) From jmellander at lbl.gov Wed Apr 30 10:18:47 2014 From: jmellander at lbl.gov (Jim Mellander) Date: Wed, 30 Apr 2014 10:18:47 -0700 Subject: [Bro-Dev] HTTP Sensitive POST bro policy Message-ID: Hi all: For a number of reasons, I elected to write the attached bro policy, which looks at http POSTs and performs regular expression matching on the posted data. The regular expression, by default, looks for the words password or passwd in upper or lower case in an attempt to find HTTP authentications via posted form. Unlike the heartbleed stuff, it does not require a special version of Bro, just @load it, will create notices of what it finds. There are a few knobs that can be adjusted that are documented in the script. The only problem with this script is that it is finding way too much - there are way too many cleartext authentications going on. If you look at outbound traffic ?,? you just might see major corporations with security fails..... ?There's some additional tweaks I want to make to this script, but it is usable as is. I hope if you run this, there aren't too many alarming? results in your traffic. Kudos to the first person who finds the minor inconsistency that I elected not to address. ?Hope this helps, ?Jim Mellander NERSC Cybersecurity -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140430/956c5f01/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: http-sensitive_POSTs.bro Type: application/octet-stream Size: 2830 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140430/956c5f01/attachment.obj From jira at bro-tracker.atlassian.net Wed Apr 30 10:27:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 30 Apr 2014 12:27:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-348) Reassembler integer overflow issues. Data not delivered after 2GB In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16304#comment-16304 ] Jon Siwek commented on BIT-348: ------------------------------- And generally, I suppose maybe Bro might be able to not transition to the closed TCP state when the sequence number on the FIN is weird like that, but I wouldn't expect that to be a trivial change, so maybe save that idea for later on when someone is doing a larger set of updates/rewrites to the TCP analyzer (the reassembly isn't really a part of that "issue"). > Reassembler integer overflow issues. Data not delivered after 2GB > ----------------------------------------------------------------- > > Key: BIT-348 > URL: https://bro-tracker.atlassian.net/browse/BIT-348 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: gregor > Assignee: Robin Sommer > Priority: High > Labels: inttypes > Fix For: 2.3 > > > {noformat} > #!rst > The TCP Reassembler does not deliver any data to analyzers after the first 2GB due to signed integer overflow (Actually it will deliver again between 4--6GB, etc.) This happens silently, i.e., without content_gap events or Undelivered calls. > This report superseded BIT-315, BIT-137 > The TCP Reassembler (and Reassem) base class use ``int`` to keep track of sequence numbers and ``seq_delta`` to check for differences. If a connection exceeds 2GB, the relative sequence numbers (int) used by the Reassembler become negative. While many parts of the Reassembler still work (because seq_delta still reports the correct difference) some parts do not. In particular ``seq_to_skip`` is broken (and fails silently). There might well be other parts of the Reassembler that fail > silently as well, that I haven't found yet. > See Comments in TCP_Reassembler.cc for more details. > The Reassembler should use int64. However this will require deep changes to the Reassembler and the TCP Analyzer and TCP_Endpoint classes (since we also store sequence numbers there). Also, the analyzer framework will need tweaks as well (e.g., Undelivered uses ``int`` for sequence numbers, also has to go to 64 bit) > As a hotfix that seems to work I disabled the ``seq_to_skip`` features. It wasn't used by any analyzer or policy script (Note, that seq_to_skip is different from skip_deliveries). Hotfix is in > topic/gregor/reassembler-hotfix > {noformat} -- This message was sent by Atlassian JIRA (v6.3-OD-03-012#6321) From jsiwek at illinois.edu Wed Apr 30 11:15:49 2014 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Wed, 30 Apr 2014 18:15:49 +0000 Subject: [Bro-Dev] HTTP Sensitive POST bro policy In-Reply-To: References: Message-ID: <5A43272A-B7F0-4BB0-9224-9BFBC15BC9A2@illinois.edu> On Apr 30, 2014, at 12:18 PM, Jim Mellander wrote: > For a number of reasons, I elected to write the attached bro policy, which looks at http POSTs and performs regular expression matching on the posted data. Thanks for sharing. > Kudos to the first person who finds the minor inconsistency that I elected not to address. Maybe not what you were referring to, but I had two concerns: (1) ?connection_end? doesn?t seem to be a defined event, maybe it's meant to be ?connection_state_remove?. (2) Having the global ?POST_entities? and ?POST_requests? tables without &read_expire (or another expiry attribute) makes me nervous. Though I think the clean up in ?http_end_entity? should catch everything, if it doesn?t, that will lead to memory usage issues over time (especially since ?connection_end? won?t be a cleanup safety net as intended). - Jon From jmellander at lbl.gov Wed Apr 30 11:44:06 2014 From: jmellander at lbl.gov (Jim Mellander) Date: Wed, 30 Apr 2014 11:44:06 -0700 Subject: [Bro-Dev] HTTP Sensitive POST bro policy In-Reply-To: <5A43272A-B7F0-4BB0-9224-9BFBC15BC9A2@illinois.edu> References: <5A43272A-B7F0-4BB0-9224-9BFBC15BC9A2@illinois.edu> Message-ID: Thanks for the input. 1. At some point in the past, ISTR bro did have a connection_end event - at least thats what my failing memory tells me. In any event, it seems like a bug to not even give a warning if you have an event handler for a non-existent event - a typo could cause difficult to detect errors. 2. Good catch, probably should have an expiration. The minor inconsistency that I had in mind is the fact that the length of the entity is checked whilst assembling the POST response without unescaping, and checked unescaped in the http_end_entity.. On Wed, Apr 30, 2014 at 11:15 AM, Siwek, Jonathan Luke wrote: > > On Apr 30, 2014, at 12:18 PM, Jim Mellander wrote: > > > For a number of reasons, I elected to write the attached bro policy, > which looks at http POSTs and performs regular expression matching on the > posted data. > > Thanks for sharing. > > > Kudos to the first person who finds the minor inconsistency that I > elected not to address. > > Maybe not what you were referring to, but I had two concerns: > > (1) ?connection_end? doesn?t seem to be a defined event, maybe it's meant > to be ?connection_state_remove?. > > (2) Having the global ?POST_entities? and ?POST_requests? tables without > &read_expire (or another expiry attribute) makes me nervous. Though I > think the clean up in ?http_end_entity? should catch everything, if it > doesn?t, that will lead to memory usage issues over time (especially since > ?connection_end? won?t be a cleanup safety net as intended). > > - Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140430/591c452b/attachment.html From jsiwek at illinois.edu Wed Apr 30 12:30:13 2014 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Wed, 30 Apr 2014 19:30:13 +0000 Subject: [Bro-Dev] HTTP Sensitive POST bro policy In-Reply-To: References: <5A43272A-B7F0-4BB0-9224-9BFBC15BC9A2@illinois.edu> Message-ID: <6C3FD7D0-A85B-40A8-89A9-9A3604B54854@illinois.edu> On Apr 30, 2014, at 1:44 PM, Jim Mellander wrote: > In any event, it seems like a bug to not even give a warning if you have an event handler for a non-existent event - a typo could cause difficult to detect errors. It?s a bit tricky (or maybe noisy) to warn on that because it?s also perfectly valid to define a new event handler like that and then generate the event from a script instead of from Bro?s internals (e.g. there could be some other script that does "event connection_end(c);?). But I agree about it being an error that?s easy to make and hard to find. For what it?s worth, there?s a script-layer option that you can ?redef? called ?check_for_unused_event_handlers? that will give such warnings in reporter.log. - Jon From seth at icir.org Wed Apr 30 13:04:29 2014 From: seth at icir.org (Seth Hall) Date: Wed, 30 Apr 2014 16:04:29 -0400 Subject: [Bro-Dev] HTTP Sensitive POST bro policy In-Reply-To: References: Message-ID: <98C90089-2A07-424C-8CC8-D1A1D239F6DA@icir.org> On Apr 30, 2014, at 1:18 PM, Jim Mellander wrote: > For a number of reasons, I elected to write the attached bro policy, which looks at http POSTs and performs regular expression matching on the posted data. Neat trick Jim. :) This actually brings up something that I've been meaning to work on for quite a while that would make this job *a lot* easier and work better... .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140430/bade1d22/attachment.bin