From noreply at bro.org Mon Feb 1 00:00:30 2016 From: noreply at bro.org (Merge Tracker) Date: Mon, 1 Feb 2016 00:00:30 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602010800.u1180UiC015715@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1531 [1] Bro,BTest Daniel Thayer - 2016-01-28 2.5 Normal Use of mktemp command should be more portable BIT-1527 [2] Bro Johanna Amann - 2016-01-26 2.5 Normal Please merge topic/johanna/cve-2015-3194 BIT-1507 [3] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ---------------------------------------------- be0d2d6 [4] bro-aux Daniel Thayer 2016-01-28 Fix the init-plugin script to be more portable Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ---------- ---------- ------------------------------------- #52 [5] bro J-Gras [6] 2016-01-18 Fixed matching mail address intel [7] [1] BIT-1531 https://bro-tracker.atlassian.net/browse/BIT-1531 [2] BIT-1527 https://bro-tracker.atlassian.net/browse/BIT-1527 [3] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [4] be0d2d6 https://github.com/bro/bro-aux/commit/be0d2d639a0757d0a9664d3e8f22d26a78e2814c [5] Pull Request #52 https://github.com/bro/bro/pull/52 [6] J-Gras https://github.com/J-Gras [7] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 From jira at bro-tracker.atlassian.net Mon Feb 1 12:32:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 1 Feb 2016 14:32:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1531) Use of mktemp command should be more portable In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1531: --------------------------------- Assignee: Robin Sommer > Use of mktemp command should be more portable > --------------------------------------------- > > Key: BIT-1531 > URL: https://bro-tracker.atlassian.net/browse/BIT-1531 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro, BTest > Reporter: Daniel Thayer > Assignee: Robin Sommer > Fix For: 2.5 > > > The use of the mktemp command breaks on some platforms, because > we only use three Xs in our templates, but some platforms require at > least six Xs. -- This message was sent by Atlassian JIRA (v7.1.0-OD-05-006#71001) From jira at bro-tracker.atlassian.net Mon Feb 1 12:45:00 2016 From: jira at bro-tracker.atlassian.net (Tim Yardley (JIRA)) Date: Mon, 1 Feb 2016 14:45:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1531) Use of mktemp command should be more portable In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24007#comment-24007 ] Tim Yardley commented on BIT-1531: ---------------------------------- I would suggest changing the fix to represent the generic template recommendation of XXXXXXXXXX rather than just going to 6. By default, gnu coreutils mktemp uses ?tmp.XXXXXXXXXX? if the template is not specified. > Use of mktemp command should be more portable > --------------------------------------------- > > Key: BIT-1531 > URL: https://bro-tracker.atlassian.net/browse/BIT-1531 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro, BTest > Reporter: Daniel Thayer > Assignee: Robin Sommer > Fix For: 2.5 > > > The use of the mktemp command breaks on some platforms, because > we only use three Xs in our templates, but some platforms require at > least six Xs. -- This message was sent by Atlassian JIRA (v7.1.0-OD-05-006#71001) From jira at bro-tracker.atlassian.net Mon Feb 1 14:09:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 1 Feb 2016 16:09:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1527) Please merge topic/johanna/cve-2015-3194 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1527: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Please merge topic/johanna/cve-2015-3194 > ---------------------------------------- > > Key: BIT-1527 > URL: https://bro-tracker.atlassian.net/browse/BIT-1527 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Fix For: 2.5 > > > Please merge topic/johanna/cve-2015-3194. The branch contains a test that checks if a machine is vulnerable to cve-2015-3194 and - if yes - raises a test error. > Note that we should assure that all our jenkins machines have a current OpenSSL before merging this to master. -- This message was sent by Atlassian JIRA (v7.1.0-OD-05-006#71001) From jira at bro-tracker.atlassian.net Mon Feb 1 14:20:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 1 Feb 2016 16:20:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1530) protocol_confirmation event cannot be hooked by plugin In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24008#comment-24008 ] Robin Sommer commented on BIT-1530: ----------------------------------- Yeah, I think I agree this should be changed. The original motivation was to trigger that event as quickly as possible, but not sure it's really worth going a non-standard route for that; in particular now that we have plugins hooking into the standard route. I'm not sure but there may indeed be a couple more places avoiding the normal event queueing in the same way, might be worth checking them as well. > protocol_confirmation event cannot be hooked by plugin > ------------------------------------------------------ > > Key: BIT-1530 > URL: https://bro-tracker.atlassian.net/browse/BIT-1530 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Jeff Barber > > The way the 'protocol_confirmation' event is raised bypasses the plugin event-hook mechanism. Plugin event hooks are triggered via EventMgr.QueueEvent which is in the usual event generation interface. However, protocol_confirmation is generated via this code in src/analyzer/Analyzer.cc: > {{ > // We immediately raise the event so that the analyzer can quickly > // react if necessary. > ::Event* e = new ::Event(protocol_confirmation, vl, SOURCE_LOCAL); > mgr.Dispatch(e); > }} > The EventMgr.Dispatch method doesn't invoke the hook so at present it's not possible for a plugin to hook this event. It would be nice if this were orthogonal with other events. -- This message was sent by Atlassian JIRA (v7.1.0-OD-05-006#71001) From robin at icir.org Mon Feb 1 14:45:21 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 1 Feb 2016 14:45:21 -0800 Subject: [Bro-Dev] get_event_peer() with Broker Message-ID: <20160201224521.GZ95752@icir.org> I was looking at how to extend the get_event_peer() bif to work with Broker events and realized that there's problem I hadn't thought about so far: when a event comes into Bro through Broker, there's no way right now to tell which peer it was sent from. If I'm not missing anything, the event comes only with event name and arguments, but no meta information of any kind that would point to its source. I think adding such meta information would be quite valuable, however it's actually not trivial to do that, as it would change the signature for incoming events across the whole Broker code base, including language bindings etc. Any ideas? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From noreply at bro.org Tue Feb 2 00:00:25 2016 From: noreply at bro.org (Merge Tracker) Date: Tue, 2 Feb 2016 00:00:25 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602020800.u1280P9w027197@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1531 [1] Bro,BTest Daniel Thayer Robin Sommer 2016-02-01 2.5 Normal Use of mktemp command should be more portable BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ---------- ---------- ------------------------------------- #52 [3] bro J-Gras [4] 2016-01-18 Fixed matching mail address intel [5] [1] BIT-1531 https://bro-tracker.atlassian.net/browse/BIT-1531 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] Pull Request #52 https://github.com/bro/bro/pull/52 [4] J-Gras https://github.com/J-Gras [5] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 From martin.vanhensbergen at fox-it.com Tue Feb 2 02:38:58 2016 From: martin.vanhensbergen at fox-it.com (Martin van Hensbergen) Date: Tue, 2 Feb 2016 10:38:58 +0000 Subject: [Bro-Dev] SMB2 - NTLM GSSAPI messages continued Message-ID: <1454409450932.50647@fox-it.com> Hi all, As you read a while back I'm trying to make the GSSAPI NTLM parsing generic enough for reuse, as it is now intertwined with the SMB parsing code. I have made some progress but before I continue, I would like some feedback from the community what is 'the bro way' on how to proceed. What I did so far is create a separate gssapi.pac file with all the type definitions for GSSAPI*TOKEN* messages. I removed all the smb_header parameters to its parsing functions so that GSSAPI and SMB are decoupled. Yet, I get confused over what information belongs to which protocol. I display my thoughts here hoping that you can either concur my line of reasoning or correct me, before I type something that is completely 'un-brolike' :) As far as I know, the correct?protocol chain is SMB[v1/2] -> GSSAPI -> NTLM. In my opinion, all message descriptions in the pac files from type SMB_NTLM_* should be renamed to NTLM_* as NTLM is not part of SMB. This would result in a separate?NTLM.pac for NTLM parsing only. I am next wondering where and how to generate SMB-NTLM-authenticate BIFevents after the decoupling. As I understand it, the parsing of the types defined in the pac files is done recursively. So originally, the parsing would go something like: type SMB1_session_setup_andx_request_ntlm_extended_security(header: SMB_Header) -> security_blob: SMB_NTLM_SSP(header) --> gssapi: GSSAP_NEG_TOKEN(header) ---> gssapi: GSSAP_NEG_TOKEN_INIT(header) (.......) some more layers (...) ---------> SMB_NTLM_Authenticate: bool = $context.connection.proc_smb_ntlm_authenticate(header, this); -----------> proc_smb_ntlm_authenticate function generates the BifEvent including the SMB header detaIls given down to it as a parameter of each function. In the new situation the header does not get given down to each function, therefor, when reaching the end-of-the-line at 'proc_smb_ntlm_authenticate', it cannot generate a proper BifEvent with the SMB header as it is not available to that function. My guess what is needed is: the ability to raise the BifEvent at the 'top layer', after the whole parsing has been done, and not at the tail of the parsing of the message. If I am not mistaken, if I add a function to the SMB1_session_setup_andx_request then this will gets executed after the complete message has been parsed. I would then be able to reach down the message to get the user/domain/workstation variables in the NTLM message. But this would give me variables like security_blob.gssapi.gssapi_token(...)...domain_name, etc which doesnt make a lot of sense and is actually an eyesore. It is a bit long mail, but basically my two questions are 1) do we all agree that the SMB_NTLM* functions should be renamed to NTLM* or am I missing something? 2) What is the best way to generate a BifEvent with SMB header and all the parsed user/domain/workstation values that were parsed deeper inside the protocol layer? Any help on this is much appreciated; especially if you think I am overlooking a hidden can of worms somewhere ;-) Regards -Martin ? From jira at bro-tracker.atlassian.net Tue Feb 2 07:30:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 2 Feb 2016 09:30:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1531) Use of mktemp command should be more portable In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1531: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) Merged. As this is a portability problem, not a security issue, the fix seems fine. > Use of mktemp command should be more portable > --------------------------------------------- > > Key: BIT-1531 > URL: https://bro-tracker.atlassian.net/browse/BIT-1531 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro, BTest > Reporter: Daniel Thayer > Assignee: Robin Sommer > Fix For: 2.5 > > > The use of the mktemp command breaks on some platforms, because > we only use three Xs in our templates, but some platforms require at > least six Xs. -- This message was sent by Atlassian JIRA (v7.1.0-OD-05-006#71001) From dirk.leinenbach at consistec.de Tue Feb 2 07:33:52 2016 From: dirk.leinenbach at consistec.de (Dirk Leinenbach) Date: Tue, 2 Feb 2016 16:33:52 +0100 Subject: [Bro-Dev] Memory Leak in find_all()? Message-ID: <56B0CC60.5090702@consistec.de> Hi, I've just examined a memory leak in one of our installations and it seems to be caused by find_all() from strings.bif in bro. Running bro with perftools enabled (cf. [1]), I get leak reports, as soon as my call to find_all() returns a non-empty list. When changing find_all() in the following way (inspired by code in IRC.cc), the leak is not reported anymore and my scripts still work as expected: old: function find_all%(str: string, re: pattern%) : string_set %{ TableVal* a = new TableVal(string_set); const u_char* s = str->Bytes(); const u_char* e = s + str->Len(); for ( const u_char* t = s; t < e; ++t ) { int n = re->MatchPrefix(t, e - t); if ( n >= 0 ) { a->Assign(new StringVal(n, (const char*) t), 0); t += n - 1; } } return a; %} new: function find_all%(str: string, re: pattern%) : string_set %{ TableVal* a = new TableVal(string_set); const u_char* s = str->Bytes(); const u_char* e = s + str->Len(); for ( const u_char* t = s; t < e; ++t ) { int n = re->MatchPrefix(t, e - t); if ( n >= 0 ) { Val* ma = new StringVal(n, (const char*) t); a->Assign(ma, 0); Unref(ma); t += n - 1; } } return a; %} Is my observation correct? If so, I could easily send a patch. BTW: there seems to be another instance of the same problem in IRC.cc Best regards, Dirk 1) https://www.bro.org/development/howtos/leaks.html -- Dr.-Ing. Dirk Leinenbach - Leitung Softwareentwicklung consistec Engineering & Consulting GmbH ------------------------------------------------------------------ Europaallee 5 Fon: +49 (0)681 / 959044-0 D-66113 Saarbr?cken Fax: +49 (0)681 / 959044-11 http://www.consistec.de e-mail: dirk.leinenbach at consistec.de Registergericht: Amtsgericht Saarbr?cken Registerblatt: HRB12003 Gesch?ftsf?hrer: Dr. Thomas Sinnwell, Volker Leiendecker, Stefan Sinnwell From robin at icir.org Tue Feb 2 08:21:57 2016 From: robin at icir.org (Robin Sommer) Date: Tue, 2 Feb 2016 08:21:57 -0800 Subject: [Bro-Dev] Memory Leak in find_all()? In-Reply-To: <56B0CC60.5090702@consistec.de> References: <56B0CC60.5090702@consistec.de> Message-ID: <20160202162157.GF45668@icir.org> On Tue, Feb 02, 2016 at 16:33 +0100, you wrote: > Val* ma = new StringVal(n, (const char*) t); > a->Assign(ma, 0); > Unref(ma); > Is my observation correct? It is, good catch. Assign() take ownership of the value (0 in this case), but not of the index. Yes, please send a patch for this one and other instances you find. Thanks, Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From robin at icir.org Tue Feb 2 08:43:21 2016 From: robin at icir.org (Robin Sommer) Date: Tue, 2 Feb 2016 08:43:21 -0800 Subject: [Bro-Dev] Jenkins errors (Re: [Bro-Commits-Internal] UnitTests - Build # 6935 - Failure!) In-Reply-To: <20160129164313.GQ76212@icir.org> References: <1156116149.59.1453973125069.JavaMail.jenkins@brotestbed.ncsa.illinois.edu> <20160128165616.GY61143@icir.org> <99FC1428-6313-4ABF-8E0C-988E26915B03@icir.org> <938E01DB-AF11-4806-9348-EBD2DD03B78A@icir.org> <56AB8949.8000307@illinois.edu> <888F078B-A502-45B9-895B-FF294AB33D5E@icir.org> <20160129164313.GQ76212@icir.org> Message-ID: <20160202164321.GA56862@icir.org> We still have a few Jenkins errors. Can somebody take the action item to go through and see what's going on? Thanks, Robin On Fri, Jan 29, 2016 at 08:43 -0800, I wrote: > I'll fix this, now that I understand what's going on. :) Thanks > everybody, > > Robin > > On Fri, Jan 29, 2016 at 11:25 -0500, you wrote: > > > > > > On Jan 29, 2016, at 10:46 AM, Daniel Thayer wrote: > > > > > > Yes, we need to add this to the CMakeLists.txt file (at least for > > > the elasticsearch and tcprs plugins): > > > > > > include(RequireCXX11) > >  > > I suppose we need to also update the script that generates the plugin skeleton to make sure that this is included there for any future plugins that are created. > > > > .Seth > > > > -- > > Seth Hall > > International Computer Science Institute > > (Bro) because everyone has a network > > http://www.bro.org/ > > > > > > _______________________________________________ > > bro-dev mailing list > > bro-dev at bro.org > > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > > > > -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From seth at icir.org Tue Feb 2 08:44:56 2016 From: seth at icir.org (Seth Hall) Date: Tue, 2 Feb 2016 11:44:56 -0500 Subject: [Bro-Dev] SMB2 - NTLM GSSAPI messages continued In-Reply-To: <1454409450932.50647@fox-it.com> References: <1454409450932.50647@fox-it.com> Message-ID: <02547F35-B93F-4C55-BD22-F3CFBB0F111A@icir.org> > On Feb 2, 2016, at 5:38 AM, Martin van Hensbergen wrote: > > 1) do we all agree that the SMB_NTLM* functions should be renamed to NTLM* or am I missing something? Agreed. > 2) What is the best way to generate a BifEvent with SMB header and all the parsed user/domain/workstation values that were parsed deeper inside the protocol layer? Just generate them with the connection record as an argument and we can tie together the various protocols at the script layer. That gives you the possibility to keep the clean abstraction in the core and all of the messy cross-structure stuff can happen in scripts. > Any help on this is much appreciated; especially if you think I am overlooking a hidden can of worms somewhere ;-) >From what you've described here and in our off-list emails, I think you're on the right track. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Tue Feb 2 08:47:37 2016 From: seth at icir.org (Seth Hall) Date: Tue, 2 Feb 2016 11:47:37 -0500 Subject: [Bro-Dev] get_event_peer() with Broker In-Reply-To: <20160201224521.GZ95752@icir.org> References: <20160201224521.GZ95752@icir.org> Message-ID: > On Feb 1, 2016, at 5:45 PM, Robin Sommer wrote: > > I think adding such meta information would be quite valuable, however > it's actually not trivial to do that, as it would change the signature > for incoming events across the whole Broker code base, including > language bindings etc. > > Any ideas? I was a little concerned that you'd fine something like this. Could we just do what I was suggesting last week where we try to see if we can get rid of the need for that? We don't use the remote peer name for much anyway and I think that it would be good for us if we stopped using it completely. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From dirk.leinenbach at consistec.de Tue Feb 2 08:48:14 2016 From: dirk.leinenbach at consistec.de (Dirk Leinenbach) Date: Tue, 2 Feb 2016 17:48:14 +0100 Subject: [Bro-Dev] Memory Leak in find_all()? In-Reply-To: <20160202162157.GF45668@icir.org> References: <56B0CC60.5090702@consistec.de> <20160202162157.GF45668@icir.org> Message-ID: <56B0DDCE.1090905@consistec.de> Ok, here we go (cf. attachment). With my older state (2.3.XXX) of bro, all tests still are green after the patch. The patch applies cleanly also to the current git head, but there I didn't execute the test suite. Best regards, Dirk On 02.02.2016 17:21, Robin Sommer wrote: > > On Tue, Feb 02, 2016 at 16:33 +0100, you wrote: > >> Val* ma = new StringVal(n, (const char*) t); >> a->Assign(ma, 0); >> Unref(ma); >> Is my observation correct? > It is, good catch. Assign() take ownership of the value (0 in this > case), but not of the index. Yes, please send a patch for this one and > other instances you find. Thanks, > > Robin > -- Dr.-Ing. Dirk Leinenbach - Leitung Softwareentwicklung consistec Engineering & Consulting GmbH ------------------------------------------------------------------ Europaallee 5 Fon: +49 (0)681 / 959044-0 D-66113 Saarbr?cken Fax: +49 (0)681 / 959044-11 http://www.consistec.de e-mail: dirk.leinenbach at consistec.de Registergericht: Amtsgericht Saarbr?cken Registerblatt: HRB12003 Gesch?ftsf?hrer: Dr. Thomas Sinnwell, Volker Leiendecker, Stefan Sinnwell -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-fix-memory-leaks-in-find_all-and-IRC-analyzer.patch Type: text/x-patch Size: 1357 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160202/6e569a35/attachment.bin From robin at icir.org Tue Feb 2 08:54:33 2016 From: robin at icir.org (Robin Sommer) Date: Tue, 2 Feb 2016 08:54:33 -0800 Subject: [Bro-Dev] get_event_peer() with Broker In-Reply-To: References: <20160201224521.GZ95752@icir.org> Message-ID: <20160202165433.GK45668@icir.org> On Tue, Feb 02, 2016 at 11:47 -0500, you wrote: > I was a little concerned that you'd fine something like this. Could > we just do what I was suggesting last week where we try to see if we > can get rid of the need for that? I was thinking about that but do we really want to give up the ability to tell where some information is coming from? If nothing else, wouldn't you want to use it for logging, e.g., to record which Bro reported a notice? Without that, there's no way to trace something on the manager back to its source. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From robin at icir.org Tue Feb 2 08:55:18 2016 From: robin at icir.org (Robin Sommer) Date: Tue, 2 Feb 2016 08:55:18 -0800 Subject: [Bro-Dev] Memory Leak in find_all()? In-Reply-To: <56B0DDCE.1090905@consistec.de> References: <56B0CC60.5090702@consistec.de> <20160202162157.GF45668@icir.org> <56B0DDCE.1090905@consistec.de> Message-ID: <20160202165518.GL45668@icir.org> Mind filing this as a ticket? That ensures it doesn't get lost. Robin On Tue, Feb 02, 2016 at 17:48 +0100, you wrote: > Ok, > > here we go (cf. attachment). > > With my older state (2.3.XXX) of bro, all tests still are green after the > patch. > > The patch applies cleanly also to the current git head, but there I didn't > execute the test suite. > > Best regards, > > Dirk > > On 02.02.2016 17:21, Robin Sommer wrote: > > > >On Tue, Feb 02, 2016 at 16:33 +0100, you wrote: > > > >> Val* ma = new StringVal(n, (const char*) t); > >> a->Assign(ma, 0); > >> Unref(ma); > >>Is my observation correct? > >It is, good catch. Assign() take ownership of the value (0 in this > >case), but not of the index. Yes, please send a patch for this one and > >other instances you find. Thanks, > > > >Robin > > > > -- > > Dr.-Ing. Dirk Leinenbach - Leitung Softwareentwicklung > consistec Engineering & Consulting GmbH > ------------------------------------------------------------------ > > Europaallee 5 Fon: +49 (0)681 / 959044-0 > D-66113 Saarbr?cken Fax: +49 (0)681 / 959044-11 > http://www.consistec.de e-mail: dirk.leinenbach at consistec.de > > Registergericht: Amtsgericht Saarbr?cken > Registerblatt: HRB12003 > Gesch?ftsf?hrer: Dr. Thomas Sinnwell, Volker Leiendecker, Stefan Sinnwell > > From f0f66eedcd236966e1bde04afa95b4cf11cc4328 Mon Sep 17 00:00:00 2001 > From: Dirk Leinenbach > Date: Tue, 2 Feb 2016 17:33:50 +0100 > Subject: [PATCH] fix memory leaks in find_all() and IRC analyzer > > --- > src/analyzer/protocol/irc/IRC.cc | 4 +++- > src/strings.bif | 4 +++- > 2 files changed, 6 insertions(+), 2 deletions(-) > > diff --git a/src/analyzer/protocol/irc/IRC.cc b/src/analyzer/protocol/irc/IRC.cc > index 96449ea..0fe9bcd 100644 > --- a/src/analyzer/protocol/irc/IRC.cc > +++ b/src/analyzer/protocol/irc/IRC.cc > @@ -268,7 +268,9 @@ void IRC_Analyzer::DeliverStream(int length, const u_char* line, bool orig) > { > if ( parts[i][0] == '@' ) > parts[i] = parts[i].substr(1); > - set->Assign(new StringVal(parts[i].c_str()), 0); > + Val* idx = new StringVal(parts[i].c_str()); > + set->Assign(idx, 0); > + Unref(idx); > } > vl->append(set); > > diff --git a/src/strings.bif b/src/strings.bif > index ebee7d9..914baae 100644 > --- a/src/strings.bif > +++ b/src/strings.bif > @@ -1161,7 +1161,9 @@ function find_all%(str: string, re: pattern%) : string_set > int n = re->MatchPrefix(t, e - t); > if ( n >= 0 ) > { > - a->Assign(new StringVal(n, (const char*) t), 0); > + Val* idx = new StringVal(n, (const char*) t); > + a->Assign(idx, 0); > + Unref(idx); > t += n - 1; > } > } -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From seth at icir.org Tue Feb 2 08:58:31 2016 From: seth at icir.org (Seth Hall) Date: Tue, 2 Feb 2016 11:58:31 -0500 Subject: [Bro-Dev] get_event_peer() with Broker In-Reply-To: <20160202165433.GK45668@icir.org> References: <20160201224521.GZ95752@icir.org> <20160202165433.GK45668@icir.org> Message-ID: <8F8940B6-C1B0-4CD3-991D-7C457883531B@icir.org> > On Feb 2, 2016, at 11:54 AM, Robin Sommer wrote: > > I was thinking about that but do we really want to give up the ability > to tell where some information is coming from? We can just include that information in the notices directly on the worker. There is no reason it needs to be done on the manager. I think that giving up that information is reasonable and we do it. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From robin at icir.org Tue Feb 2 09:07:45 2016 From: robin at icir.org (Robin Sommer) Date: Tue, 2 Feb 2016 09:07:45 -0800 Subject: [Bro-Dev] get_event_peer() with Broker In-Reply-To: <8F8940B6-C1B0-4CD3-991D-7C457883531B@icir.org> References: <20160201224521.GZ95752@icir.org> <20160202165433.GK45668@icir.org> <8F8940B6-C1B0-4CD3-991D-7C457883531B@icir.org> Message-ID: <20160202170745.GN45668@icir.org> On Tue, Feb 02, 2016 at 11:58 -0500, you wrote: > We can just include that information in the notices directly on the > worker. There is no reason it needs to be done on the manager. Ok, notices were a bad example. :-) But, yeah, I hear you. For specific cases, one can always send the Information explicitly of course. It just doesn't feel great to me to not have a general way to tell where something is coming from, even without cooperation from the sender. That's said, we indeed have hardly any concrete use cases right now, and if you think we don't need it, I'm fine removing it. The alternative of changing a lot of code isn't very tempting. So, then the action item for 2.5 turns into: remove get_event_peer(). I'll do that, and also then still fix is_local_event() to work with Broker. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From dirk.leinenbach at consistec.de Tue Feb 2 11:00:37 2016 From: dirk.leinenbach at consistec.de (Dirk Leinenbach) Date: Tue, 2 Feb 2016 20:00:37 +0100 Subject: [Bro-Dev] Memory Leak in find_all()? In-Reply-To: <20160202165518.GL45668@icir.org> References: <56B0CC60.5090702@consistec.de> <20160202162157.GF45668@icir.org> <56B0DDCE.1090905@consistec.de> <20160202165518.GL45668@icir.org> Message-ID: <56B0FCD5.8050408@consistec.de> Sure: https://bro-tracker.atlassian.net/projects/BIT/issues/BIT-1532 On 02.02.2016 17:55, Robin Sommer wrote: > Mind filing this as a ticket? That ensures it doesn't get lost. > > Robin > > On Tue, Feb 02, 2016 at 17:48 +0100, you wrote: > >> Ok, >> >> here we go (cf. attachment). >> >> With my older state (2.3.XXX) of bro, all tests still are green after the >> patch. >> >> The patch applies cleanly also to the current git head, but there I didn't >> execute the test suite. >> >> Best regards, >> >> Dirk >> >> On 02.02.2016 17:21, Robin Sommer wrote: >>> On Tue, Feb 02, 2016 at 16:33 +0100, you wrote: >>> >>>> Val* ma = new StringVal(n, (const char*) t); >>>> a->Assign(ma, 0); >>>> Unref(ma); >>>> Is my observation correct? >>> It is, good catch. Assign() take ownership of the value (0 in this >>> case), but not of the index. Yes, please send a patch for this one and >>> other instances you find. Thanks, >>> >>> Robin >>> >> -- >> >> Dr.-Ing. Dirk Leinenbach - Leitung Softwareentwicklung >> consistec Engineering & Consulting GmbH >> ------------------------------------------------------------------ >> >> Europaallee 5 Fon: +49 (0)681 / 959044-0 >> D-66113 Saarbr?cken Fax: +49 (0)681 / 959044-11 >> http://www.consistec.de e-mail: dirk.leinenbach at consistec.de >> >> Registergericht: Amtsgericht Saarbr?cken >> Registerblatt: HRB12003 >> Gesch?ftsf?hrer: Dr. Thomas Sinnwell, Volker Leiendecker, Stefan Sinnwell >> >> From f0f66eedcd236966e1bde04afa95b4cf11cc4328 Mon Sep 17 00:00:00 2001 >> From: Dirk Leinenbach >> Date: Tue, 2 Feb 2016 17:33:50 +0100 >> Subject: [PATCH] fix memory leaks in find_all() and IRC analyzer >> >> --- >> src/analyzer/protocol/irc/IRC.cc | 4 +++- >> src/strings.bif | 4 +++- >> 2 files changed, 6 insertions(+), 2 deletions(-) >> >> diff --git a/src/analyzer/protocol/irc/IRC.cc b/src/analyzer/protocol/irc/IRC.cc >> index 96449ea..0fe9bcd 100644 >> --- a/src/analyzer/protocol/irc/IRC.cc >> +++ b/src/analyzer/protocol/irc/IRC.cc >> @@ -268,7 +268,9 @@ void IRC_Analyzer::DeliverStream(int length, const u_char* line, bool orig) >> { >> if ( parts[i][0] == '@' ) >> parts[i] = parts[i].substr(1); >> - set->Assign(new StringVal(parts[i].c_str()), 0); >> + Val* idx = new StringVal(parts[i].c_str()); >> + set->Assign(idx, 0); >> + Unref(idx); >> } >> vl->append(set); >> >> diff --git a/src/strings.bif b/src/strings.bif >> index ebee7d9..914baae 100644 >> --- a/src/strings.bif >> +++ b/src/strings.bif >> @@ -1161,7 +1161,9 @@ function find_all%(str: string, re: pattern%) : string_set >> int n = re->MatchPrefix(t, e - t); >> if ( n >= 0 ) >> { >> - a->Assign(new StringVal(n, (const char*) t), 0); >> + Val* idx = new StringVal(n, (const char*) t); >> + a->Assign(idx, 0); >> + Unref(idx); >> t += n - 1; >> } >> } > -- Dr.-Ing. Dirk Leinenbach - Leitung Softwareentwicklung consistec Engineering & Consulting GmbH ------------------------------------------------------------------ Europaallee 5 Fon: +49 (0)681 / 959044-0 D-66113 Saarbr?cken Fax: +49 (0)681 / 959044-11 http://www.consistec.de e-mail: dirk.leinenbach at consistec.de Registergericht: Amtsgericht Saarbr?cken Registerblatt: HRB12003 Gesch?ftsf?hrer: Dr. Thomas Sinnwell, Volker Leiendecker, Stefan Sinnwell From jira at bro-tracker.atlassian.net Tue Feb 2 11:01:00 2016 From: jira at bro-tracker.atlassian.net (Dirk Leinenbach (JIRA)) Date: Tue, 2 Feb 2016 13:01:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1532) fix memory leak in find_all() and IRC analyzer In-Reply-To: References: Message-ID: Dirk Leinenbach created BIT-1532: ------------------------------------- Summary: fix memory leak in find_all() and IRC analyzer Key: BIT-1532 URL: https://bro-tracker.atlassian.net/browse/BIT-1532 Project: Bro Issue Tracker Issue Type: Patch Components: Bro Reporter: Dirk Leinenbach Attachments: 0001-fix-memory-leaks-in-find_all-and-IRC-analyzer.patch fix memory leaks in find_all() and IRC analyzer Running bro with perftools enabled (cf. [1]), I get leak reports, as soon as my call to find_all() returns a non-empty list. When changing find_all() in the following way (inspired by code in IRC.cc), the leak is not reported anymore and my scripts still work as expected: old: function find_all%(str: string, re: pattern%) : string_set %{ TableVal* a = new TableVal(string_set); const u_char* s = str->Bytes(); const u_char* e = s + str->Len(); for ( const u_char* t = s; t < e; ++t ) { int n = re->MatchPrefix(t, e - t); if ( n >= 0 ) { a->Assign(new StringVal(n, (const char*) t), 0); t += n - 1; } } return a; %} new: function find_all%(str: string, re: pattern%) : string_set %{ TableVal* a = new TableVal(string_set); const u_char* s = str->Bytes(); const u_char* e = s + str->Len(); for ( const u_char* t = s; t < e; ++t ) { int n = re->MatchPrefix(t, e - t); if ( n >= 0 ) { Val* ma = new StringVal(n, (const char*) t); a->Assign(ma, 0); Unref(ma); t += n - 1; } } return a; %} -- This message was sent by Atlassian JIRA (v7.1.0-OD-05-006#71001) From noreply at bro.org Wed Feb 3 00:00:25 2016 From: noreply at bro.org (Merge Tracker) Date: Wed, 3 Feb 2016 00:00:25 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602030800.u1380PbP000744@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1507 [1] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ---------- ---------- ------------------------------------- #52 [2] bro J-Gras [3] 2016-01-18 Fixed matching mail address intel [4] [1] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [2] Pull Request #52 https://github.com/bro/bro/pull/52 [3] J-Gras https://github.com/J-Gras [4] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 From jira at bro-tracker.atlassian.net Wed Feb 3 08:51:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Wed, 3 Feb 2016 10:51:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1532) fix memory leak in find_all() and IRC analyzer In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1532: ------------------------------ Fix Version/s: 2.5 > fix memory leak in find_all() and IRC analyzer > ---------------------------------------------- > > Key: BIT-1532 > URL: https://bro-tracker.atlassian.net/browse/BIT-1532 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Reporter: Dirk Leinenbach > Fix For: 2.5 > > Attachments: 0001-fix-memory-leaks-in-find_all-and-IRC-analyzer.patch > > > fix memory leaks in find_all() and IRC analyzer > Running bro with perftools enabled (cf. [1]), I get leak reports, as > soon as my call to find_all() returns a non-empty list. > When changing find_all() in the following way (inspired by code in > IRC.cc), the leak is not reported anymore and my scripts still work as > expected: > old: > function find_all%(str: string, re: pattern%) : string_set > %{ > TableVal* a = new TableVal(string_set); > const u_char* s = str->Bytes(); > const u_char* e = s + str->Len(); > for ( const u_char* t = s; t < e; ++t ) > { > int n = re->MatchPrefix(t, e - t); > if ( n >= 0 ) > { > a->Assign(new StringVal(n, (const char*) t), 0); > t += n - 1; > } > } > return a; > %} > new: > function find_all%(str: string, re: pattern%) : string_set > %{ > TableVal* a = new TableVal(string_set); > const u_char* s = str->Bytes(); > const u_char* e = s + str->Len(); > for ( const u_char* t = s; t < e; ++t ) > { > int n = re->MatchPrefix(t, e - t); > if ( n >= 0 ) > { > Val* ma = new StringVal(n, (const char*) t); > a->Assign(ma, 0); > Unref(ma); > t += n - 1; > } > } > return a; > %} -- This message was sent by Atlassian JIRA (v7.1.0-OD-05-006#71001) From jira at bro-tracker.atlassian.net Wed Feb 3 08:51:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Wed, 3 Feb 2016 10:51:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1532) fix memory leak in find_all() and IRC analyzer In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1532: ------------------------------ Status: Merge Request (was: Open) > fix memory leak in find_all() and IRC analyzer > ---------------------------------------------- > > Key: BIT-1532 > URL: https://bro-tracker.atlassian.net/browse/BIT-1532 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Reporter: Dirk Leinenbach > Fix For: 2.5 > > Attachments: 0001-fix-memory-leaks-in-find_all-and-IRC-analyzer.patch > > > fix memory leaks in find_all() and IRC analyzer > Running bro with perftools enabled (cf. [1]), I get leak reports, as > soon as my call to find_all() returns a non-empty list. > When changing find_all() in the following way (inspired by code in > IRC.cc), the leak is not reported anymore and my scripts still work as > expected: > old: > function find_all%(str: string, re: pattern%) : string_set > %{ > TableVal* a = new TableVal(string_set); > const u_char* s = str->Bytes(); > const u_char* e = s + str->Len(); > for ( const u_char* t = s; t < e; ++t ) > { > int n = re->MatchPrefix(t, e - t); > if ( n >= 0 ) > { > a->Assign(new StringVal(n, (const char*) t), 0); > t += n - 1; > } > } > return a; > %} > new: > function find_all%(str: string, re: pattern%) : string_set > %{ > TableVal* a = new TableVal(string_set); > const u_char* s = str->Bytes(); > const u_char* e = s + str->Len(); > for ( const u_char* t = s; t < e; ++t ) > { > int n = re->MatchPrefix(t, e - t); > if ( n >= 0 ) > { > Val* ma = new StringVal(n, (const char*) t); > a->Assign(ma, 0); > Unref(ma); > t += n - 1; > } > } > return a; > %} -- This message was sent by Atlassian JIRA (v7.1.0-OD-05-006#71001) From hwang.jinho at gmail.com Wed Feb 3 20:39:41 2016 From: hwang.jinho at gmail.com (jinho hwang) Date: Wed, 3 Feb 2016 23:39:41 -0500 Subject: [Bro-Dev] Fail to configure CXX when cross compiling Message-ID: Hi, I just started looking at Bro source. When configuring the build with cross-compiler, I set CXX=, but the build tree did not set correctly. It just used c++ in my local OS. CC= works well, though. What I did was CC= CXX= ./configure However, the build summary shows CC: and CXX: /usr/bin/c++ (this is wrong). Do you have any idea where I should put my cross compiler? thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160203/0b4397f4/attachment.html From noreply at bro.org Thu Feb 4 00:00:26 2016 From: noreply at bro.org (Merge Tracker) Date: Thu, 4 Feb 2016 00:00:26 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602040800.u1480QiA014107@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- --------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1532 [1] Bro Dirk Leinenbach - 2016-02-03 2.5 Normal fix memory leak in find_all() and IRC analyzer BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ---------- ---------- ------------------------------------- #52 [3] bro J-Gras [4] 2016-01-18 Fixed matching mail address intel [5] [1] BIT-1532 https://bro-tracker.atlassian.net/browse/BIT-1532 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] Pull Request #52 https://github.com/bro/bro/pull/52 [4] J-Gras https://github.com/J-Gras [5] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 From jira at bro-tracker.atlassian.net Thu Feb 4 11:30:00 2016 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Thu, 4 Feb 2016 13:30:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1533) mysql analyzer does not set service to mysql In-Reply-To: References: Message-ID: Justin Azoff created BIT-1533: --------------------------------- Summary: mysql analyzer does not set service to mysql Key: BIT-1533 URL: https://bro-tracker.atlassian.net/browse/BIT-1533 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.4 Reporter: Justin Azoff Assignee: Vlad Grigorescu The mysql analyzer does not set the service to mysql. The result of this is that conn.log and known_services do not show 'mysql' anywhere. -- This message was sent by Atlassian JIRA (v7.1.0-OD-05-006#71001) From noreply at bro.org Fri Feb 5 00:00:25 2016 From: noreply at bro.org (Merge Tracker) Date: Fri, 5 Feb 2016 00:00:25 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602050800.u1580PIw026264@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- --------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1532 [1] Bro Dirk Leinenbach - 2016-02-03 2.5 Normal fix memory leak in find_all() and IRC analyzer BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ---------------------------------------------------- 2e0c203 [3] bro Johanna Amann 2016-02-04 Add missing break; in StartTLS case of IRC analyzer. Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ---------- ---------- --------------------------------------------------- #52 [4] bro J-Gras [5] 2016-01-18 Fixed matching mail address intel [6] #15 [7] bro-plugins kb1 [8] 2016-02-04 Capitalize Myricom_ROOT_DIR in configure.plugin [9] [1] BIT-1532 https://bro-tracker.atlassian.net/browse/BIT-1532 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] 2e0c203 https://github.com/bro/bro/commit/2e0c2035c9513f2a84e3d242adb22b076e324728 [4] Pull Request #52 https://github.com/bro/bro/pull/52 [5] J-Gras https://github.com/J-Gras [6] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [7] Pull Request #15 https://github.com/bro/bro-plugins/pull/15 [8] kb1 https://github.com/kb1 [9] Merge Pull Request #15 with git pull --no-ff --no-commit https://github.com/kb1/bro-plugins.git master From jsiwek at illinois.edu Fri Feb 5 08:19:01 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Fri, 5 Feb 2016 16:19:01 +0000 Subject: [Bro-Dev] get_event_peer() with Broker In-Reply-To: <20160201224521.GZ95752@icir.org> References: <20160201224521.GZ95752@icir.org> Message-ID: <081582D3-550D-470B-8071-639837B8888A@illinois.edu> > On Feb 1, 2016, at 4:45 PM, Robin Sommer wrote: > > I was looking at how to extend the get_event_peer() bif to work with > Broker events and realized that there's problem I hadn't thought about > so far: when a event comes into Bro through Broker, there's no way > right now to tell which peer it was sent from. If I'm not missing > anything, the event comes only with event name and arguments, but no > meta information of any kind that would point to its source. > > I think adding such meta information would be quite valuable, however > it's actually not trivial to do that, as it would change the signature > for incoming events across the whole Broker code base, including > language bindings etc. Daniel asked about this a few months ago; not sure if he has any progress to share, but yeah, code changes would be needed to support that BIF. My suggested approaches at the time were 1) Change the message format the Bro uses for events. E.g. switch Bro events from { event_name, { arg1, arg2, arg3 ... } }, to { sender_name, event_name, { arg2, arg2, arg3 ... } }. Then internally use a lookup table keyed on sender_name. 2) Change Broker queues to carry a sender/message pair. IIRC a single 32-bit integer is enough to identify CAF actors, so that means an extra 4 bytes per message in all queues (since actor-peer mapping can be done locally, those bytes don?t need to be sent on the wire). You could also have two types of queues ? one type that tracks senders and one type that does not (which is just the current implementation). Then let the application programmer choose the best option for individual use cases. Approach (2) is more robust, but also more work ? probably nothing ?tricky?, just hunting down code that needs changing. - Jon From noreply at bro.org Sat Feb 6 00:00:26 2016 From: noreply at bro.org (Merge Tracker) Date: Sat, 6 Feb 2016 00:00:26 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602060800.u1680Qml007859@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- --------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1532 [1] Bro Dirk Leinenbach - 2016-02-03 2.5 Normal fix memory leak in find_all() and IRC analyzer BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ---------------------------------------------------- 2e0c203 [3] bro Johanna Amann 2016-02-04 Add missing break; in StartTLS case of IRC analyzer. Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- ------------ ---------- ------------------------------------------------------------- #54 [4] bro marktayl [5] 2016-02-06 Removed duplicate parameter for IRC "QUIT" event handler. [6] #52 [7] bro J-Gras [8] 2016-01-18 Fixed matching mail address intel [9] #15 [10] bro-plugins kb1 [11] 2016-02-04 Capitalize Myricom_ROOT_DIR in configure.plugin [12] [1] BIT-1532 https://bro-tracker.atlassian.net/browse/BIT-1532 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] 2e0c203 https://github.com/bro/bro/commit/2e0c2035c9513f2a84e3d242adb22b076e324728 [4] Pull Request #54 https://github.com/bro/bro/pull/54 [5] marktayl https://github.com/marktayl [6] Merge Pull Request #54 with git pull --no-ff --no-commit https://github.com/marktayl/bro.git master [7] Pull Request #52 https://github.com/bro/bro/pull/52 [8] J-Gras https://github.com/J-Gras [9] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [10] Pull Request #15 https://github.com/bro/bro-plugins/pull/15 [11] kb1 https://github.com/kb1 [12] Merge Pull Request #15 with git pull --no-ff --no-commit https://github.com/kb1/bro-plugins.git master From noreply at bro.org Sun Feb 7 00:00:23 2016 From: noreply at bro.org (Merge Tracker) Date: Sun, 7 Feb 2016 00:00:23 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602070800.u1780NiF031413@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- --------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1532 [1] Bro Dirk Leinenbach - 2016-02-03 2.5 Normal fix memory leak in find_all() and IRC analyzer BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ---------------------------------------------------- 2e0c203 [3] bro Johanna Amann 2016-02-04 Add missing break; in StartTLS case of IRC analyzer. Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- ------------ ---------- ------------------------------------------------------------- #54 [4] bro marktayl [5] 2016-02-06 Removed duplicate parameter for IRC "QUIT" event handler. [6] #52 [7] bro J-Gras [8] 2016-01-18 Fixed matching mail address intel [9] #15 [10] bro-plugins kb1 [11] 2016-02-04 Capitalize Myricom_ROOT_DIR in configure.plugin [12] [1] BIT-1532 https://bro-tracker.atlassian.net/browse/BIT-1532 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] 2e0c203 https://github.com/bro/bro/commit/2e0c2035c9513f2a84e3d242adb22b076e324728 [4] Pull Request #54 https://github.com/bro/bro/pull/54 [5] marktayl https://github.com/marktayl [6] Merge Pull Request #54 with git pull --no-ff --no-commit https://github.com/marktayl/bro.git master [7] Pull Request #52 https://github.com/bro/bro/pull/52 [8] J-Gras https://github.com/J-Gras [9] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [10] Pull Request #15 https://github.com/bro/bro-plugins/pull/15 [11] kb1 https://github.com/kb1 [12] Merge Pull Request #15 with git pull --no-ff --no-commit https://github.com/kb1/bro-plugins.git master From robin at icir.org Sun Feb 7 09:01:15 2016 From: robin at icir.org (Robin Sommer) Date: Sun, 7 Feb 2016 09:01:15 -0800 Subject: [Bro-Dev] Fail to configure CXX when cross compiling In-Reply-To: References: Message-ID: <20160207170115.GK90965@icir.org> Please file a bug report with our tracker, looks like something we should fix. Robin On Wed, Feb 03, 2016 at 23:39 -0500, you wrote: > Hi, > > I just started looking at Bro source. When configuring the build with > cross-compiler, I set CXX=, but the build tree did not set > correctly. It just used c++ in my local OS. CC= works well, > though. What I did was > > CC= CXX= ./configure > > However, the build summary shows CC: and CXX: /usr/bin/c++ > (this is wrong). Do you have any idea where I should put my cross compiler? > > thanks, > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From hwang.jinho at gmail.com Sun Feb 7 15:00:07 2016 From: hwang.jinho at gmail.com (jinho hwang) Date: Sun, 7 Feb 2016 18:00:07 -0500 Subject: [Bro-Dev] Fail to configure CXX when cross compiling In-Reply-To: <20160207170115.GK90965@icir.org> References: <20160207170115.GK90965@icir.org> Message-ID: On Sun, Feb 7, 2016 at 12:01 PM, Robin Sommer wrote: > Please file a bug report with our tracker, looks like something we > should fix. > > Robin > > On Wed, Feb 03, 2016 at 23:39 -0500, you wrote: > > > Hi, > > > > I just started looking at Bro source. When configuring the build with > > cross-compiler, I set CXX=, but the build tree did not > set > > correctly. It just used c++ in my local OS. CC= works > well, > > though. What I did was > > > > CC= CXX= ./configure > > > > However, the build summary shows CC: and CXX: > /usr/bin/c++ > > (this is wrong). Do you have any idea where I should put my cross > compiler? > > > > thanks, > > > _______________________________________________ > > bro-dev mailing list > > bro-dev at bro.org > > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > > > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin > It turned out to be my mistake that I did not distclean after I changed the build configuration. The bigger problem is that when I cross-compile the source tree, it actually runs bifcl command in the building host (where compiling is done) with broker-dummy/store.bif.. However, the build is done using target platform, which is different from the building host. I am not very familiar with bro yet.. so I am not sure what this actually does.. can you please help me to understand the process or provide documents if any. Regardless, I will keep looking. thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160207/47f48192/attachment.html From noreply at bro.org Mon Feb 8 00:00:20 2016 From: noreply at bro.org (Merge Tracker) Date: Mon, 8 Feb 2016 00:00:20 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602080800.u1880KEX019634@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- --------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1532 [1] Bro Dirk Leinenbach - 2016-02-03 2.5 Normal fix memory leak in find_all() and IRC analyzer BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ---------------------------------------------------- 2e0c203 [3] bro Johanna Amann 2016-02-04 Add missing break; in StartTLS case of IRC analyzer. Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- ------------ ---------- ------------------------------------------------------------- #55 [4] bro wglodek [5] 2016-02-07 http-evasion [6] #54 [7] bro marktayl [8] 2016-02-06 Removed duplicate parameter for IRC "QUIT" event handler. [9] #52 [10] bro J-Gras [11] 2016-01-18 Fixed matching mail address intel [12] #15 [13] bro-plugins kb1 [14] 2016-02-04 Capitalize Myricom_ROOT_DIR in configure.plugin [15] [1] BIT-1532 https://bro-tracker.atlassian.net/browse/BIT-1532 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] 2e0c203 https://github.com/bro/bro/commit/2e0c2035c9513f2a84e3d242adb22b076e324728 [4] Pull Request #55 https://github.com/bro/bro/pull/55 [5] wglodek https://github.com/wglodek [6] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [7] Pull Request #54 https://github.com/bro/bro/pull/54 [8] marktayl https://github.com/marktayl [9] Merge Pull Request #54 with git pull --no-ff --no-commit https://github.com/marktayl/bro.git master [10] Pull Request #52 https://github.com/bro/bro/pull/52 [11] J-Gras https://github.com/J-Gras [12] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [13] Pull Request #15 https://github.com/bro/bro-plugins/pull/15 [14] kb1 https://github.com/kb1 [15] Merge Pull Request #15 with git pull --no-ff --no-commit https://github.com/kb1/bro-plugins.git master From jira at bro-tracker.atlassian.net Mon Feb 8 13:46:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Mon, 8 Feb 2016 15:46:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1532) fix memory leak in find_all() and IRC analyzer In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann reassigned BIT-1532: ---------------------------------- Assignee: Johanna Amann > fix memory leak in find_all() and IRC analyzer > ---------------------------------------------- > > Key: BIT-1532 > URL: https://bro-tracker.atlassian.net/browse/BIT-1532 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Reporter: Dirk Leinenbach > Assignee: Johanna Amann > Fix For: 2.5 > > Attachments: 0001-fix-memory-leaks-in-find_all-and-IRC-analyzer.patch > > > fix memory leaks in find_all() and IRC analyzer > Running bro with perftools enabled (cf. [1]), I get leak reports, as > soon as my call to find_all() returns a non-empty list. > When changing find_all() in the following way (inspired by code in > IRC.cc), the leak is not reported anymore and my scripts still work as > expected: > old: > function find_all%(str: string, re: pattern%) : string_set > %{ > TableVal* a = new TableVal(string_set); > const u_char* s = str->Bytes(); > const u_char* e = s + str->Len(); > for ( const u_char* t = s; t < e; ++t ) > { > int n = re->MatchPrefix(t, e - t); > if ( n >= 0 ) > { > a->Assign(new StringVal(n, (const char*) t), 0); > t += n - 1; > } > } > return a; > %} > new: > function find_all%(str: string, re: pattern%) : string_set > %{ > TableVal* a = new TableVal(string_set); > const u_char* s = str->Bytes(); > const u_char* e = s + str->Len(); > for ( const u_char* t = s; t < e; ++t ) > { > int n = re->MatchPrefix(t, e - t); > if ( n >= 0 ) > { > Val* ma = new StringVal(n, (const char*) t); > a->Assign(ma, 0); > Unref(ma); > t += n - 1; > } > } > return a; > %} -- This message was sent by Atlassian JIRA (v7.1.0-OD-06-005#71002) From jira at bro-tracker.atlassian.net Mon Feb 8 14:30:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Mon, 8 Feb 2016 16:30:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1532) fix memory leak in find_all() and IRC analyzer In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1532: ------------------------------- Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) Merged in 6b5fd442f09151470c873f57f6a6f31d4c3ac225 > fix memory leak in find_all() and IRC analyzer > ---------------------------------------------- > > Key: BIT-1532 > URL: https://bro-tracker.atlassian.net/browse/BIT-1532 > Project: Bro Issue Tracker > Issue Type: Patch > Components: Bro > Reporter: Dirk Leinenbach > Assignee: Johanna Amann > Fix For: 2.5 > > Attachments: 0001-fix-memory-leaks-in-find_all-and-IRC-analyzer.patch > > > fix memory leaks in find_all() and IRC analyzer > Running bro with perftools enabled (cf. [1]), I get leak reports, as > soon as my call to find_all() returns a non-empty list. > When changing find_all() in the following way (inspired by code in > IRC.cc), the leak is not reported anymore and my scripts still work as > expected: > old: > function find_all%(str: string, re: pattern%) : string_set > %{ > TableVal* a = new TableVal(string_set); > const u_char* s = str->Bytes(); > const u_char* e = s + str->Len(); > for ( const u_char* t = s; t < e; ++t ) > { > int n = re->MatchPrefix(t, e - t); > if ( n >= 0 ) > { > a->Assign(new StringVal(n, (const char*) t), 0); > t += n - 1; > } > } > return a; > %} > new: > function find_all%(str: string, re: pattern%) : string_set > %{ > TableVal* a = new TableVal(string_set); > const u_char* s = str->Bytes(); > const u_char* e = s + str->Len(); > for ( const u_char* t = s; t < e; ++t ) > { > int n = re->MatchPrefix(t, e - t); > if ( n >= 0 ) > { > Val* ma = new StringVal(n, (const char*) t); > a->Assign(ma, 0); > Unref(ma); > t += n - 1; > } > } > return a; > %} -- This message was sent by Atlassian JIRA (v7.1.0-OD-06-005#71002) From jira at bro-tracker.atlassian.net Mon Feb 8 15:43:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Mon, 8 Feb 2016 17:43:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1534) Please merge topic/johanna/stats_smb_leak In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1534: ------------------------------- Status: Merge Request (was: Open) > Please merge topic/johanna/stats_smb_leak > ----------------------------------------- > > Key: BIT-1534 > URL: https://bro-tracker.atlassian.net/browse/BIT-1534 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Fix For: 2.5 > > > Please merge topic/johanna/stats_smb_leak > It fixes a memory leak in stats.cc and smb.cc. A test that triggers the leak in stats.cc is attached. > Due to not having access to test traffic I was not actually able to test the smb.cc case, but I am very confident that this triggers a leak and that the patch should fix it (TableVal->Assign needs an Unref for the index value (first parameter; this is the same case as in stats.cc)). -- This message was sent by Atlassian JIRA (v7.1.0-OD-06-005#71002) From jira at bro-tracker.atlassian.net Mon Feb 8 15:43:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Mon, 8 Feb 2016 17:43:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1534) Please merge topic/johanna/stats_smb_leak In-Reply-To: References: Message-ID: Johanna Amann created BIT-1534: ---------------------------------- Summary: Please merge topic/johanna/stats_smb_leak Key: BIT-1534 URL: https://bro-tracker.atlassian.net/browse/BIT-1534 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: git/master Reporter: Johanna Amann Fix For: 2.5 Please merge topic/johanna/stats_smb_leak It fixes a memory leak in stats.cc and smb.cc. A test that triggers the leak in stats.cc is attached. Due to not having access to test traffic I was not actually able to test the smb.cc case, but I am very confident that this triggers a leak and that the patch should fix it (TableVal->Assign needs an Unref for the index value (first parameter; this is the same case as in stats.cc)). -- This message was sent by Atlassian JIRA (v7.1.0-OD-06-005#71002) From noreply at bro.org Tue Feb 9 00:00:22 2016 From: noreply at bro.org (Merge Tracker) Date: Tue, 9 Feb 2016 00:00:22 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602090800.u1980MY3015589@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1534 [1] Bro Johanna Amann - 2016-02-08 2.5 Normal Please merge topic/johanna/stats_smb_leak BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ---------------------------------------------------- 2e0c203 [3] bro Johanna Amann 2016-02-04 Add missing break; in StartTLS case of IRC analyzer. Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ----------- ---------- ------------------------------------- #55 [4] bro wglodek [5] 2016-02-07 http-evasion [6] #52 [7] bro J-Gras [8] 2016-01-18 Fixed matching mail address intel [9] [1] BIT-1534 https://bro-tracker.atlassian.net/browse/BIT-1534 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] 2e0c203 https://github.com/bro/bro/commit/2e0c2035c9513f2a84e3d242adb22b076e324728 [4] Pull Request #55 https://github.com/bro/bro/pull/55 [5] wglodek https://github.com/wglodek [6] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [7] Pull Request #52 https://github.com/bro/bro/pull/52 [8] J-Gras https://github.com/J-Gras [9] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 From jbarber at computer.org Tue Feb 9 13:04:43 2016 From: jbarber at computer.org (Jeff Barber) Date: Tue, 9 Feb 2016 16:04:43 -0500 Subject: [Bro-Dev] bro and DNP3 decoder Message-ID: Hi I'm trying to use bro for decoding DNP3 traffic and following the logic through its parser to the various dnp3_xxx events. (The documentation on how to use the DNP3 events is a bit light but I think I understand what's happening.) When I try to follow the request objects logic (e.g. as you might get from a DNP3 write command), I can't see how they're getting output to the bro script layer at all. Most of them seem to simply dead-end in the parser with no event generated. I spent a little while looking through the bro branches and came across a branch called topics/hui/dnp3-events that _seems_ to have support for a bunch of additional objects. It was last worked on in February 2014 but I can't find any hint of it in the master branch. Just wondering if anyone can clarify. Am I misunderstanding how it works? Or did the code in dnp3-events branch get lost? Or was it never merged? Or never completed? Thanks! Addressing to Hui Lin but also including bro-dev in case someone else knows the history. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160209/c9503b9a/attachment.html From hlin33 at illinois.edu Tue Feb 9 13:49:25 2016 From: hlin33 at illinois.edu (Hui Lin (Hugo) ) Date: Tue, 9 Feb 2016 15:49:25 -0600 Subject: [Bro-Dev] bro and DNP3 decoder In-Reply-To: References: Message-ID: ?Hi Jeff, I think the master branch should contain what we wrote before. I run some simple DNP3 test cases included in Bro from master branch and I do see the simple print out message. Does running your pcap generate any error message? Do you mind sharing the trace that you are using for me to take a look at what is going on?? Best, Hui Lin On Tue, Feb 9, 2016 at 3:04 PM, Jeff Barber wrote: > Hi > > I'm trying to use bro for decoding DNP3 traffic and following the logic > through its parser to the various dnp3_xxx events. (The documentation on > how to use the DNP3 events is a bit light but I think I understand what's > happening.) When I try to follow the request objects logic (e.g. as you > might get from a DNP3 write command), I can't see how they're getting > output to the bro script layer at all. Most of them seem to simply dead-end > in the parser with no event generated. > > I spent a little while looking through the bro branches and came across a > branch called topics/hui/dnp3-events that _seems_ to have support for a > bunch of additional objects. It was last worked on in February 2014 but I > can't find any hint of it in the master branch. > > Just wondering if anyone can clarify. Am I misunderstanding how it works? > Or did the code in dnp3-events branch get lost? Or was it never merged? Or > never completed? > > Thanks! > > Addressing to Hui Lin but also including bro-dev in case someone else > knows the history. > > -- Hui Lin PhD Candidate (http://hlin33.web.engr.illinois.edu/) DEPEND (http://depend.csl.illinois.edu/) ECE, Uni. of Illinois at Urbana-Champaign -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160209/e7b00e0c/attachment.html From jbarber at computer.org Tue Feb 9 15:19:28 2016 From: jbarber at computer.org (Jeff Barber) Date: Tue, 9 Feb 2016 18:19:28 -0500 Subject: [Bro-Dev] bro and DNP3 decoder In-Reply-To: References: Message-ID: It's only particular object types and especially those in the *request* that I'm referring to. I do see response objects fine. I wish I had better pcaps to share, but I'm having trouble finding those myself. :) I have attached one I found on the web which, according to wireshark, has a single write of object type 50, variation 01. It produces these events: header_block, [orig_h=127.0.0.1, orig_p=37712/tcp, resp_h=127.0.0.1, resp_p=20000/tcp, vlan=0, inner_vlan=0], T, Start, 25605, Len, 18, Ctrl, 196, Dst, 3, Src, 4 application_request_header, [orig_h=127.0.0.1, orig_p=37712/tcp, resp_h=127.0.0.1, resp_p=20000/tcp, vlan=0, inner_vlan=0], T, App, 193, FC, 2 object_header, [orig_h=127.0.0.1, orig_p=37712/tcp, resp_h=127.0.0.1, resp_p=20000/tcp, vlan=0, inner_vlan=0], T, OT, 12801, Qua, 7, Num, 1, RF, 1, 0 object_prefix, [orig_h=127.0.0.1, orig_p=37712/tcp, resp_h=127.0.0.1, resp_p=20000/tcp, vlan=0, inner_vlan=0], T, PREF, 0 (Mnemonics included except for the first two fields which are always c$id and is_orig.) but there's no event giving the content of that object type. I'm not getting any error messages, but just in looking at the .pac files in the dnp3 directory, I see the code apparently parsing all the unique types below, but it doesn't seem to be generating events for any of them. At least some of those *do* seem to have had events generated for them in that dnp3-events branch code. AnaOutStatus32 AnaOutStatus16 AnaOutStatusSP AnaOutStatusDP AnaOut32 AnaOut16 AnaOutSP AnaOutDP AnaOutEve32woTime AnaOutEve16woTime AnaOutEve32wTime AnaOutEve16wTime AnaOutEveSPwoTime AnaOutEveDPwoTime AnaOutEveSPwTime AnaOutEveDPwTime Thanks. On Tue, Feb 9, 2016 at 4:49 PM, Hui Lin (Hugo) wrote: > ?Hi Jeff, > > I think the master branch should contain what we wrote before. I run some > simple DNP3 test cases included in Bro from master branch and I do see the > simple print out message. > > Does running your pcap generate any error message? Do you mind sharing the > trace that you are using for me to take a look at what is going on?? > > Best, > > Hui Lin > > On Tue, Feb 9, 2016 at 3:04 PM, Jeff Barber wrote: > >> Hi >> >> I'm trying to use bro for decoding DNP3 traffic and following the logic >> through its parser to the various dnp3_xxx events. (The documentation on >> how to use the DNP3 events is a bit light but I think I understand what's >> happening.) When I try to follow the request objects logic (e.g. as you >> might get from a DNP3 write command), I can't see how they're getting >> output to the bro script layer at all. Most of them seem to simply dead-end >> in the parser with no event generated. >> >> I spent a little while looking through the bro branches and came across a >> branch called topics/hui/dnp3-events that _seems_ to have support for a >> bunch of additional objects. It was last worked on in February 2014 but I >> can't find any hint of it in the master branch. >> >> Just wondering if anyone can clarify. Am I misunderstanding how it works? >> Or did the code in dnp3-events branch get lost? Or was it never merged? Or >> never completed? >> >> Thanks! >> >> Addressing to Hui Lin but also including bro-dev in case someone else >> knows the history. >> >> > > > -- > Hui Lin > PhD Candidate (http://hlin33.web.engr.illinois.edu/) > DEPEND (http://depend.csl.illinois.edu/) > ECE, Uni. of Illinois at Urbana-Champaign > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160209/f411fa03/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: DNP3-Write.pcap Type: application/octet-stream Size: 610 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160209/f411fa03/attachment.obj From noreply at bro.org Wed Feb 10 00:00:20 2016 From: noreply at bro.org (Merge Tracker) Date: Wed, 10 Feb 2016 00:00:20 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602100800.u1A80Kq7028852@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1534 [1] Bro Johanna Amann - 2016-02-08 2.5 Normal Please merge topic/johanna/stats_smb_leak BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ---------------------------------------------------- 2e0c203 [3] bro Johanna Amann 2016-02-04 Add missing break; in StartTLS case of IRC analyzer. Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- ------------ ---------- ------------------------------------- #55 [4] bro wglodek [5] 2016-02-07 http-evasion [6] #52 [7] bro J-Gras [8] 2016-01-18 Fixed matching mail address intel [9] #18 [10] bro-plugins jshlbrd [11] 2016-02-09 SSDP analyzer [12] [1] BIT-1534 https://bro-tracker.atlassian.net/browse/BIT-1534 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] 2e0c203 https://github.com/bro/bro/commit/2e0c2035c9513f2a84e3d242adb22b076e324728 [4] Pull Request #55 https://github.com/bro/bro/pull/55 [5] wglodek https://github.com/wglodek [6] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [7] Pull Request #52 https://github.com/bro/bro/pull/52 [8] J-Gras https://github.com/J-Gras [9] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [10] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [11] jshlbrd https://github.com/jshlbrd [12] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Wed Feb 10 11:40:00 2016 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Wed, 10 Feb 2016 13:40:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1535) conn.log conn_state field or documentation is wrong In-Reply-To: References: Message-ID: Justin Azoff created BIT-1535: --------------------------------- Summary: conn.log conn_state field or documentation is wrong Key: BIT-1535 URL: https://bro-tracker.atlassian.net/browse/BIT-1535 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.4 Reporter: Justin Azoff There is an issue where the conn.log conn_state field will contain RSTR, which according to the documentation means "Established, responder aborted." The problem that I notice is that I see log entries where conn_state is RSTR, but conn_history does not contain an 'h'. Additionally, the resp_h is absolutely not running a service on resp_p and the orig_h is usually in the process of a tcp scan. Here are the top frequencies of RSTR without an h over about a weeks worth of conn logs: {code} 38193 RSTR Fr 3662 RSTR DFr 1801 RSTR DFdrR 1248 RSTR DRr 432 RSTR DrF 232 RSTR Far 128 RSTR DdAFrR 79 RSTR DFadrR 64 RSTR DrR 58 RSTR DdAFarR {code} Compared to histories that did contain an h: {code} 425398 RSTR ShADadFr 204149 RSTR ShADadFrR 156303 RSTR ShADdFar 141795 RSTR ShADadFRRr 105704 RSTR ShADadfr 79697 RSTR ShADadr 63493 RSTR ShADaFr 51704 RSTR ShADadFrrrr 42075 RSTR ShADdar 37678 RSTR ShADadfRr {code} I don't have a pcap for this, but I believe many of the weird connections are related to scans or backscatter. I'm not sure if the code is wrong or the documentation is wrong, but I don't see how a fin+reset connection could be classified as established. Also, One thing that would be a nice documentation addition is the answer to this question: Given a conn.log entry, how do determine if there was a connection established? I thought it would be if the state was in 'SF S1 S2 S3 RSTO RSTR', but RSTR is problematic... -- This message was sent by Atlassian JIRA (v7.1.0-OD-06-005#71002) From jira at bro-tracker.atlassian.net Wed Feb 10 12:59:00 2016 From: jira at bro-tracker.atlassian.net (Vern Paxson (JIRA)) Date: Wed, 10 Feb 2016 14:59:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1535) conn.log conn_state field or documentation is wrong In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vern Paxson reassigned BIT-1535: -------------------------------- Assignee: Vern Paxson > conn.log conn_state field or documentation is wrong > --------------------------------------------------- > > Key: BIT-1535 > URL: https://bro-tracker.atlassian.net/browse/BIT-1535 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Justin Azoff > Assignee: Vern Paxson > > There is an issue where the conn.log conn_state field will contain RSTR, which according to the documentation means "Established, responder aborted." > The problem that I notice is that I see log entries where conn_state is RSTR, but conn_history does not contain an 'h'. Additionally, the resp_h is absolutely not running a service on resp_p and the orig_h is usually in the process of a tcp scan. > Here are the top frequencies of RSTR without an h over about a weeks worth of conn logs: > {code} > 38193 RSTR Fr > 3662 RSTR DFr > 1801 RSTR DFdrR > 1248 RSTR DRr > 432 RSTR DrF > 232 RSTR Far > 128 RSTR DdAFrR > 79 RSTR DFadrR > 64 RSTR DrR > 58 RSTR DdAFarR > {code} > Compared to histories that did contain an h: > {code} > 425398 RSTR ShADadFr > 204149 RSTR ShADadFrR > 156303 RSTR ShADdFar > 141795 RSTR ShADadFRRr > 105704 RSTR ShADadfr > 79697 RSTR ShADadr > 63493 RSTR ShADaFr > 51704 RSTR ShADadFrrrr > 42075 RSTR ShADdar > 37678 RSTR ShADadfRr > {code} > I don't have a pcap for this, but I believe many of the weird connections are related to scans or backscatter. > I'm not sure if the code is wrong or the documentation is wrong, but I don't see how a fin+reset connection could be classified as established. > Also, One thing that would be a nice documentation addition is the answer to this question: > Given a conn.log entry, how do determine if there was a connection established? I thought it would be if the state was in 'SF S1 S2 S3 RSTO RSTR', but RSTR is problematic... -- This message was sent by Atlassian JIRA (v7.1.0-OD-06-005#71002) From jira at bro-tracker.atlassian.net Wed Feb 10 13:04:00 2016 From: jira at bro-tracker.atlassian.net (Vern Paxson (JIRA)) Date: Wed, 10 Feb 2016 15:04:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1535) conn.log conn_state field or documentation is wrong In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24102#comment-24102 ] Vern Paxson edited comment on BIT-1535 at 2/10/16 3:03 PM: ----------------------------------------------------------- Yeah, historically "RSTR" was defined as established-but-responder-aborted, but that dates way back to a time when "surely" every connection began with a handshake, and RSTR was the observation that that connection ended with a RST sent by the responder. I think the right fix for this ticket is to correct the documentation to state that it simply means that the entity that Bro determined/inferred was the responder sent a RST. was (Author: vern): Yeah, historically "RSTR" was defined as established-but-responder-aborted, but that dates way back to a time when "surely" ever connection began with a handshake, and RSTR was the observation that that connection ended with a RST sent by the responder. I think the right fix for this ticket is to correct the documentation that it simply means that the entity that Bro determined/inferred was the responder sent a RST. > conn.log conn_state field or documentation is wrong > --------------------------------------------------- > > Key: BIT-1535 > URL: https://bro-tracker.atlassian.net/browse/BIT-1535 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Justin Azoff > Assignee: Vern Paxson > > There is an issue where the conn.log conn_state field will contain RSTR, which according to the documentation means "Established, responder aborted." > The problem that I notice is that I see log entries where conn_state is RSTR, but conn_history does not contain an 'h'. Additionally, the resp_h is absolutely not running a service on resp_p and the orig_h is usually in the process of a tcp scan. > Here are the top frequencies of RSTR without an h over about a weeks worth of conn logs: > {code} > 38193 RSTR Fr > 3662 RSTR DFr > 1801 RSTR DFdrR > 1248 RSTR DRr > 432 RSTR DrF > 232 RSTR Far > 128 RSTR DdAFrR > 79 RSTR DFadrR > 64 RSTR DrR > 58 RSTR DdAFarR > {code} > Compared to histories that did contain an h: > {code} > 425398 RSTR ShADadFr > 204149 RSTR ShADadFrR > 156303 RSTR ShADdFar > 141795 RSTR ShADadFRRr > 105704 RSTR ShADadfr > 79697 RSTR ShADadr > 63493 RSTR ShADaFr > 51704 RSTR ShADadFrrrr > 42075 RSTR ShADdar > 37678 RSTR ShADadfRr > {code} > I don't have a pcap for this, but I believe many of the weird connections are related to scans or backscatter. > I'm not sure if the code is wrong or the documentation is wrong, but I don't see how a fin+reset connection could be classified as established. > Also, One thing that would be a nice documentation addition is the answer to this question: > Given a conn.log entry, how do determine if there was a connection established? I thought it would be if the state was in 'SF S1 S2 S3 RSTO RSTR', but RSTR is problematic... -- This message was sent by Atlassian JIRA (v7.1.0-OD-06-005#71002) From jira at bro-tracker.atlassian.net Wed Feb 10 13:04:00 2016 From: jira at bro-tracker.atlassian.net (Vern Paxson (JIRA)) Date: Wed, 10 Feb 2016 15:04:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1535) conn.log conn_state field or documentation is wrong In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24102#comment-24102 ] Vern Paxson commented on BIT-1535: ---------------------------------- Yeah, historically "RSTR" was defined as established-but-responder-aborted, but that dates way back to a time when "surely" ever connection began with a handshake, and RSTR was the observation that that connection ended with a RST sent by the responder. I think the right fix for this ticket is to correct the documentation that it simply means that the entity that Bro determined/inferred was the responder sent a RST. > conn.log conn_state field or documentation is wrong > --------------------------------------------------- > > Key: BIT-1535 > URL: https://bro-tracker.atlassian.net/browse/BIT-1535 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Justin Azoff > Assignee: Vern Paxson > > There is an issue where the conn.log conn_state field will contain RSTR, which according to the documentation means "Established, responder aborted." > The problem that I notice is that I see log entries where conn_state is RSTR, but conn_history does not contain an 'h'. Additionally, the resp_h is absolutely not running a service on resp_p and the orig_h is usually in the process of a tcp scan. > Here are the top frequencies of RSTR without an h over about a weeks worth of conn logs: > {code} > 38193 RSTR Fr > 3662 RSTR DFr > 1801 RSTR DFdrR > 1248 RSTR DRr > 432 RSTR DrF > 232 RSTR Far > 128 RSTR DdAFrR > 79 RSTR DFadrR > 64 RSTR DrR > 58 RSTR DdAFarR > {code} > Compared to histories that did contain an h: > {code} > 425398 RSTR ShADadFr > 204149 RSTR ShADadFrR > 156303 RSTR ShADdFar > 141795 RSTR ShADadFRRr > 105704 RSTR ShADadfr > 79697 RSTR ShADadr > 63493 RSTR ShADaFr > 51704 RSTR ShADadFrrrr > 42075 RSTR ShADdar > 37678 RSTR ShADadfRr > {code} > I don't have a pcap for this, but I believe many of the weird connections are related to scans or backscatter. > I'm not sure if the code is wrong or the documentation is wrong, but I don't see how a fin+reset connection could be classified as established. > Also, One thing that would be a nice documentation addition is the answer to this question: > Given a conn.log entry, how do determine if there was a connection established? I thought it would be if the state was in 'SF S1 S2 S3 RSTO RSTR', but RSTR is problematic... -- This message was sent by Atlassian JIRA (v7.1.0-OD-06-005#71002) From hlin33 at illinois.edu Wed Feb 10 13:46:59 2016 From: hlin33 at illinois.edu (Hui Lin (Hugo) ) Date: Wed, 10 Feb 2016 15:46:59 -0600 Subject: [Bro-Dev] bro and DNP3 decoder In-Reply-To: <7c20d69f993541bda70a06fa7c3135e0@CHIHT4.ad.uillinois.edu> References: <7c20d69f993541bda70a06fa7c3135e0@CHIHT4.ad.uillinois.edu> Message-ID: Hi Jeff, I think in the current master, we do support the function code of "write" in the master branch, but not the type of objects used in the given pcap file. In that development branch you mentioned, I think I added as many as event handlers I can. But we could not merge them into the master branch, as at that time, we could not find any sufficient test pcap files that can trigger the event handlers. Probably it is the time for me to search again for the test pcap files again from the Internet. The pacp that you provide may not be used as a test case, as it looks like a truncated communication, i.e., a request without responses. If you have any test pcap, feel free to share with us if that is appropriate for you. Also, if you need to use these event handlers, I think that you can also go for that development branch. And also feel free to let me know if there is any bug in that branch, I can work to fix them. Best, Hui Lin On Tue, Feb 9, 2016 at 5:19 PM, Jeff Barber wrote: > > It's only particular object types and especially those in the *request* > that I'm referring to. I do see response objects fine. I wish I had better > pcaps to share, but I'm having trouble finding those myself. :) > > > I have attached one I found on the web which, according to wireshark, has > a single write of object type 50, variation 01. It produces these events: > > header_block, [orig_h=127.0.0.1, orig_p=37712/tcp, resp_h=127.0.0.1, > resp_p=20000/tcp, vlan=0, inner_vlan=0], T, Start, 25605, Len, 18, Ctrl, > 196, Dst, 3, Src, 4 > application_request_header, [orig_h=127.0.0.1, orig_p=37712/tcp, > resp_h=127.0.0.1, resp_p=20000/tcp, vlan=0, inner_vlan=0], T, App, 193, FC, > 2 > object_header, [orig_h=127.0.0.1, orig_p=37712/tcp, resp_h=127.0.0.1, > resp_p=20000/tcp, vlan=0, inner_vlan=0], T, OT, 12801, Qua, 7, Num, 1, RF, > 1, 0 > object_prefix, [orig_h=127.0.0.1, orig_p=37712/tcp, resp_h=127.0.0.1, > resp_p=20000/tcp, vlan=0, inner_vlan=0], T, PREF, 0 > > (Mnemonics included except for the first two fields which are always c$id > and is_orig.) > > but there's no event giving the content of that object type. > > > I'm not getting any error messages, but just in looking at the .pac files > in the dnp3 directory, I see the code apparently parsing all the unique > types below, but it doesn't seem to be generating events for any of them. > At least some of those *do* seem to have had events generated for them in > that dnp3-events branch code. > > AnaOutStatus32 > AnaOutStatus16 > AnaOutStatusSP > AnaOutStatusDP > AnaOut32 > AnaOut16 > AnaOutSP > AnaOutDP > AnaOutEve32woTime > AnaOutEve16woTime > AnaOutEve32wTime > AnaOutEve16wTime > AnaOutEveSPwoTime > AnaOutEveDPwoTime > AnaOutEveSPwTime > AnaOutEveDPwTime > > > Thanks. > > > On Tue, Feb 9, 2016 at 4:49 PM, Hui Lin (Hugo) > wrote: > >> ?Hi Jeff, >> >> I think the master branch should contain what we wrote before. I run some >> simple DNP3 test cases included in Bro from master branch and I do see the >> simple print out message. >> >> Does running your pcap generate any error message? Do you mind sharing >> the trace that you are using for me to take a look at what is going on?? >> >> Best, >> >> Hui Lin >> >> On Tue, Feb 9, 2016 at 3:04 PM, Jeff Barber wrote: >> >>> Hi >>> >>> I'm trying to use bro for decoding DNP3 traffic and following the logic >>> through its parser to the various dnp3_xxx events. (The documentation on >>> how to use the DNP3 events is a bit light but I think I understand what's >>> happening.) When I try to follow the request objects logic (e.g. as you >>> might get from a DNP3 write command), I can't see how they're getting >>> output to the bro script layer at all. Most of them seem to simply dead-end >>> in the parser with no event generated. >>> >>> I spent a little while looking through the bro branches and came across >>> a branch called topics/hui/dnp3-events that _seems_ to have support for a >>> bunch of additional objects. It was last worked on in February 2014 but I >>> can't find any hint of it in the master branch. >>> >>> Just wondering if anyone can clarify. Am I misunderstanding how it >>> works? Or did the code in dnp3-events branch get lost? Or was it never >>> merged? Or never completed? >>> >>> Thanks! >>> >>> Addressing to Hui Lin but also including bro-dev in case someone else >>> knows the history. >>> >>> >> >> >> -- >> Hui Lin >> PhD Candidate (http://hlin33.web.engr.illinois.edu/) >> DEPEND (http://depend.csl.illinois.edu/) >> ECE, Uni. of Illinois at Urbana-Champaign >> >> > -- Hui Lin PhD Candidate (http://hlin33.web.engr.illinois.edu/) DEPEND (http://depend.csl.illinois.edu/) ECE, Uni. of Illinois at Urbana-Champaign -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160210/b5f5e814/attachment-0001.html From jbarber at computer.org Wed Feb 10 14:02:05 2016 From: jbarber at computer.org (Jeff Barber) Date: Wed, 10 Feb 2016 17:02:05 -0500 Subject: [Bro-Dev] bro and DNP3 decoder In-Reply-To: References: <7c20d69f993541bda70a06fa7c3135e0@CHIHT4.ad.uillinois.edu> Message-ID: Thanks a lot for the response. That makes sense. If I find some packet captures for this stuff, I will share them if I can. On Wed, Feb 10, 2016 at 4:46 PM, Hui Lin (Hugo) wrote: > Hi Jeff, > > I think in the current master, we do support the function code of "write" > in the master branch, but not the type of objects used in the given pcap > file. In that development branch you mentioned, I think I added as many as > event handlers I can. But we could not merge them into the master branch, > as at that time, we could not find any sufficient test pcap files that can > trigger the event handlers. Probably it is the time for me to search again > for the test pcap files again from the Internet. The pacp that you provide > may not be used as a test case, as it looks like a truncated communication, > i.e., a request without responses. If you have any test pcap, feel free to > share with us if that is appropriate for you. Also, if you need to use > these event handlers, I think that you can also go for that development > branch. And also feel free to let me know if there is any bug in that > branch, I can work to fix them. > > Best, > > Hui Lin > > > > On Tue, Feb 9, 2016 at 5:19 PM, Jeff Barber wrote: > >> >> It's only particular object types and especially those in the *request* >> that I'm referring to. I do see response objects fine. I wish I had better >> pcaps to share, but I'm having trouble finding those myself. :) >> >> >> I have attached one I found on the web which, according to wireshark, has >> a single write of object type 50, variation 01. It produces these events: >> >> header_block, [orig_h=127.0.0.1, orig_p=37712/tcp, resp_h=127.0.0.1, >> resp_p=20000/tcp, vlan=0, inner_vlan=0], T, Start, 25605, Len, 18, Ctrl, >> 196, Dst, 3, Src, 4 >> application_request_header, [orig_h=127.0.0.1, orig_p=37712/tcp, >> resp_h=127.0.0.1, resp_p=20000/tcp, vlan=0, inner_vlan=0], T, App, 193, FC, >> 2 >> object_header, [orig_h=127.0.0.1, orig_p=37712/tcp, resp_h=127.0.0.1, >> resp_p=20000/tcp, vlan=0, inner_vlan=0], T, OT, 12801, Qua, 7, Num, 1, RF, >> 1, 0 >> object_prefix, [orig_h=127.0.0.1, orig_p=37712/tcp, resp_h=127.0.0.1, >> resp_p=20000/tcp, vlan=0, inner_vlan=0], T, PREF, 0 >> >> (Mnemonics included except for the first two fields which are always c$id >> and is_orig.) >> >> but there's no event giving the content of that object type. >> >> >> I'm not getting any error messages, but just in looking at the .pac files >> in the dnp3 directory, I see the code apparently parsing all the unique >> types below, but it doesn't seem to be generating events for any of them. >> At least some of those *do* seem to have had events generated for them in >> that dnp3-events branch code. >> >> AnaOutStatus32 >> AnaOutStatus16 >> AnaOutStatusSP >> AnaOutStatusDP >> AnaOut32 >> AnaOut16 >> AnaOutSP >> AnaOutDP >> AnaOutEve32woTime >> AnaOutEve16woTime >> AnaOutEve32wTime >> AnaOutEve16wTime >> AnaOutEveSPwoTime >> AnaOutEveDPwoTime >> AnaOutEveSPwTime >> AnaOutEveDPwTime >> >> >> Thanks. >> >> >> On Tue, Feb 9, 2016 at 4:49 PM, Hui Lin (Hugo) >> wrote: >> >>> ?Hi Jeff, >>> >>> I think the master branch should contain what we wrote before. I run >>> some simple DNP3 test cases included in Bro from master branch and I do see >>> the simple print out message. >>> >>> Does running your pcap generate any error message? Do you mind sharing >>> the trace that you are using for me to take a look at what is going on?? >>> >>> Best, >>> >>> Hui Lin >>> >>> On Tue, Feb 9, 2016 at 3:04 PM, Jeff Barber >>> wrote: >>> >>>> Hi >>>> >>>> I'm trying to use bro for decoding DNP3 traffic and following the logic >>>> through its parser to the various dnp3_xxx events. (The documentation on >>>> how to use the DNP3 events is a bit light but I think I understand what's >>>> happening.) When I try to follow the request objects logic (e.g. as you >>>> might get from a DNP3 write command), I can't see how they're getting >>>> output to the bro script layer at all. Most of them seem to simply dead-end >>>> in the parser with no event generated. >>>> >>>> I spent a little while looking through the bro branches and came across >>>> a branch called topics/hui/dnp3-events that _seems_ to have support for a >>>> bunch of additional objects. It was last worked on in February 2014 but I >>>> can't find any hint of it in the master branch. >>>> >>>> Just wondering if anyone can clarify. Am I misunderstanding how it >>>> works? Or did the code in dnp3-events branch get lost? Or was it never >>>> merged? Or never completed? >>>> >>>> Thanks! >>>> >>>> Addressing to Hui Lin but also including bro-dev in case someone else >>>> knows the history. >>>> >>>> >>> >>> >>> -- >>> Hui Lin >>> PhD Candidate (http://hlin33.web.engr.illinois.edu/) >>> DEPEND (http://depend.csl.illinois.edu/) >>> ECE, Uni. of Illinois at Urbana-Champaign >>> >>> >> > > > -- > Hui Lin > PhD Candidate (http://hlin33.web.engr.illinois.edu/) > DEPEND (http://depend.csl.illinois.edu/) > ECE, Uni. of Illinois at Urbana-Champaign > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160210/a830ad70/attachment.html From noreply at bro.org Thu Feb 11 00:00:23 2016 From: noreply at bro.org (Merge Tracker) Date: Thu, 11 Feb 2016 00:00:23 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602110800.u1B80NnV015493@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1534 [1] Bro Johanna Amann - 2016-02-08 2.5 Normal Please merge topic/johanna/stats_smb_leak BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ---------------------------------------------------- 2e0c203 [3] bro Johanna Amann 2016-02-04 Add missing break; in StartTLS case of IRC analyzer. Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- ------------ ---------- ------------------------------------- #55 [4] bro wglodek [5] 2016-02-07 http-evasion [6] #52 [7] bro J-Gras [8] 2016-01-18 Fixed matching mail address intel [9] #18 [10] bro-plugins jshlbrd [11] 2016-02-09 SSDP analyzer [12] [1] BIT-1534 https://bro-tracker.atlassian.net/browse/BIT-1534 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] 2e0c203 https://github.com/bro/bro/commit/2e0c2035c9513f2a84e3d242adb22b076e324728 [4] Pull Request #55 https://github.com/bro/bro/pull/55 [5] wglodek https://github.com/wglodek [6] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [7] Pull Request #52 https://github.com/bro/bro/pull/52 [8] J-Gras https://github.com/J-Gras [9] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [10] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [11] jshlbrd https://github.com/jshlbrd [12] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From johanna at icir.org Thu Feb 11 11:13:30 2016 From: johanna at icir.org (Johanna Amann) Date: Thu, 11 Feb 2016 11:13:30 -0800 Subject: [Bro-Dev] Jenkins errors (Re: [Bro-Commits-Internal] UnitTests - Build # 6935 - Failure!) In-Reply-To: <20160202164321.GA56862@icir.org> References: <1156116149.59.1453973125069.JavaMail.jenkins@brotestbed.ncsa.illinois.edu> <20160128165616.GY61143@icir.org> <99FC1428-6313-4ABF-8E0C-988E26915B03@icir.org> <938E01DB-AF11-4806-9348-EBD2DD03B78A@icir.org> <56AB8949.8000307@illinois.edu> <888F078B-A502-45B9-895B-FF294AB33D5E@icir.org> <20160129164313.GQ76212@icir.org> <20160202164321.GA56862@icir.org> Message-ID: <20160211191330.GA36184@Beezling.local> Could someone who has access to the machines please tell me what the exact CentOS version running on jenkins is? And also - the output of openssl version. Johanna On Tue, Feb 02, 2016 at 08:43:21AM -0800, Robin Sommer wrote: > We still have a few Jenkins errors. Can somebody take the action item > to go through and see what's going on? Thanks, > > Robin > > On Fri, Jan 29, 2016 at 08:43 -0800, I wrote: > > > I'll fix this, now that I understand what's going on. :) Thanks > > everybody, > > > > Robin > > > > On Fri, Jan 29, 2016 at 11:25 -0500, you wrote: > > > > > > > > > On Jan 29, 2016, at 10:46 AM, Daniel Thayer wrote: > > > > > > > > Yes, we need to add this to the CMakeLists.txt file (at least for > > > > the elasticsearch and tcprs plugins): > > > > > > > > include(RequireCXX11) > > >  > > > I suppose we need to also update the script that generates the plugin skeleton to make sure that this is included there for any future plugins that are created. > > > > > > .Seth > > > > > > -- > > > Seth Hall > > > International Computer Science Institute > > > (Bro) because everyone has a network > > > http://www.bro.org/ > > > > > > > > > _______________________________________________ > > > bro-dev mailing list > > > bro-dev at bro.org > > > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > > > > > > > > > > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > > From jira at bro-tracker.atlassian.net Thu Feb 11 12:45:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 11 Feb 2016 14:45:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1536) elasticsearch tests using nc fail on some systems In-Reply-To: References: Message-ID: Daniel Thayer created BIT-1536: ---------------------------------- Summary: elasticsearch tests using nc fail on some systems Key: BIT-1536 URL: https://bro-tracker.atlassian.net/browse/BIT-1536 Project: Bro Issue Tracker Issue Type: Task Components: Bro Reporter: Daniel Thayer Some of the elasticsearch tests use the nc command, and these tests fail on some systems (such as debian 7 and 8) because there are at least two different incompatible implementations of nc. A quick fix is to use a shell script wrapper that chooses the correct command-line arguments. -- This message was sent by Atlassian JIRA (v7.1.0-OD-06-005#71002) From jira at bro-tracker.atlassian.net Thu Feb 11 12:45:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 11 Feb 2016 14:45:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1536) elasticsearch tests using nc fail on some systems In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer reassigned BIT-1536: ---------------------------------- Assignee: Daniel Thayer > elasticsearch tests using nc fail on some systems > ------------------------------------------------- > > Key: BIT-1536 > URL: https://bro-tracker.atlassian.net/browse/BIT-1536 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro > Reporter: Daniel Thayer > Assignee: Daniel Thayer > > Some of the elasticsearch tests use the nc command, and these tests > fail on some systems (such as debian 7 and 8) because there are at > least two different incompatible implementations of nc. A quick fix > is to use a shell script wrapper that chooses the correct command-line > arguments. -- This message was sent by Atlassian JIRA (v7.1.0-OD-06-005#71002) From jira at bro-tracker.atlassian.net Thu Feb 11 13:15:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 11 Feb 2016 15:15:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1536) elasticsearch tests using nc fail on some systems In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1536: ------------------------------- Status: Merge Request (was: Open) Assignee: (was: Daniel Thayer) > elasticsearch tests using nc fail on some systems > ------------------------------------------------- > > Key: BIT-1536 > URL: https://bro-tracker.atlassian.net/browse/BIT-1536 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro > Reporter: Daniel Thayer > > Some of the elasticsearch tests use the nc command, and these tests > fail on some systems (such as debian 7 and 8) because there are at > least two different incompatible implementations of nc. A quick fix > is to use a shell script wrapper that chooses the correct command-line > arguments. -- This message was sent by Atlassian JIRA (v7.1.0-OD-06-005#71002) From jira at bro-tracker.atlassian.net Thu Feb 11 13:15:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 11 Feb 2016 15:15:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1536) elasticsearch tests using nc fail on some systems In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24104#comment-24104 ] Daniel Thayer commented on BIT-1536: ------------------------------------ In the "bro-plugins" repo, in branch "topic/dnthayer/ticket1536", I've added a shell script wrapper for the nc command which should work on all systems. > elasticsearch tests using nc fail on some systems > ------------------------------------------------- > > Key: BIT-1536 > URL: https://bro-tracker.atlassian.net/browse/BIT-1536 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro > Reporter: Daniel Thayer > Assignee: Daniel Thayer > > Some of the elasticsearch tests use the nc command, and these tests > fail on some systems (such as debian 7 and 8) because there are at > least two different incompatible implementations of nc. A quick fix > is to use a shell script wrapper that chooses the correct command-line > arguments. -- This message was sent by Atlassian JIRA (v7.1.0-OD-06-005#71002) From noreply at bro.org Fri Feb 12 00:00:26 2016 From: noreply at bro.org (Merge Tracker) Date: Fri, 12 Feb 2016 00:00:26 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602120800.u1C80QYA004770@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1536 [1] Bro Daniel Thayer - 2016-02-11 - Normal elasticsearch tests using nc fail on some systems BIT-1534 [2] Bro Johanna Amann - 2016-02-08 2.5 Normal Please merge topic/johanna/stats_smb_leak BIT-1507 [3] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ---------------------------------------------------- 2e0c203 [4] bro Johanna Amann 2016-02-04 Add missing break; in StartTLS case of IRC analyzer. Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- ------------ ---------- -------------------------------------- #55 [5] bro wglodek [6] 2016-02-07 http-evasion [7] #52 [8] bro J-Gras [9] 2016-01-18 Fixed matching mail address intel [10] #18 [11] bro-plugins jshlbrd [12] 2016-02-09 SSDP analyzer [13] [1] BIT-1536 https://bro-tracker.atlassian.net/browse/BIT-1536 [2] BIT-1534 https://bro-tracker.atlassian.net/browse/BIT-1534 [3] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [4] 2e0c203 https://github.com/bro/bro/commit/2e0c2035c9513f2a84e3d242adb22b076e324728 [5] Pull Request #55 https://github.com/bro/bro/pull/55 [6] wglodek https://github.com/wglodek [7] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [8] Pull Request #52 https://github.com/bro/bro/pull/52 [9] J-Gras https://github.com/J-Gras [10] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [11] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [12] jshlbrd https://github.com/jshlbrd [13] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From noreply at bro.org Sat Feb 13 00:00:34 2016 From: noreply at bro.org (Merge Tracker) Date: Sat, 13 Feb 2016 00:00:34 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602130800.u1D80Yfr002356@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1536 [1] Bro Daniel Thayer - 2016-02-11 - Normal elasticsearch tests using nc fail on some systems BIT-1534 [2] Bro Johanna Amann - 2016-02-08 2.5 Normal Please merge topic/johanna/stats_smb_leak BIT-1507 [3] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ---------------------------------------------------- 2e0c203 [4] bro Johanna Amann 2016-02-04 Add missing break; in StartTLS case of IRC analyzer. Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- ------------ ---------- -------------------------------------- #55 [5] bro wglodek [6] 2016-02-07 http-evasion [7] #52 [8] bro J-Gras [9] 2016-01-18 Fixed matching mail address intel [10] #18 [11] bro-plugins jshlbrd [12] 2016-02-12 SSDP analyzer [13] [1] BIT-1536 https://bro-tracker.atlassian.net/browse/BIT-1536 [2] BIT-1534 https://bro-tracker.atlassian.net/browse/BIT-1534 [3] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [4] 2e0c203 https://github.com/bro/bro/commit/2e0c2035c9513f2a84e3d242adb22b076e324728 [5] Pull Request #55 https://github.com/bro/bro/pull/55 [6] wglodek https://github.com/wglodek [7] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [8] Pull Request #52 https://github.com/bro/bro/pull/52 [9] J-Gras https://github.com/J-Gras [10] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [11] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [12] jshlbrd https://github.com/jshlbrd [13] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Sat Feb 13 10:39:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Sat, 13 Feb 2016 12:39:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1534) Please merge topic/johanna/stats_smb_leak In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1534: --------------------------------- Assignee: Robin Sommer > Please merge topic/johanna/stats_smb_leak > ----------------------------------------- > > Key: BIT-1534 > URL: https://bro-tracker.atlassian.net/browse/BIT-1534 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > Fix For: 2.5 > > > Please merge topic/johanna/stats_smb_leak > It fixes a memory leak in stats.cc and smb.cc. A test that triggers the leak in stats.cc is attached. > Due to not having access to test traffic I was not actually able to test the smb.cc case, but I am very confident that this triggers a leak and that the patch should fix it (TableVal->Assign needs an Unref for the index value (first parameter; this is the same case as in stats.cc)). -- This message was sent by Atlassian JIRA (v7.1.0-OD-06-005#71002) From jira at bro-tracker.atlassian.net Sat Feb 13 10:42:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Sat, 13 Feb 2016 12:42:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1536) elasticsearch tests using nc fail on some systems In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1536: --------------------------------- Assignee: Robin Sommer > elasticsearch tests using nc fail on some systems > ------------------------------------------------- > > Key: BIT-1536 > URL: https://bro-tracker.atlassian.net/browse/BIT-1536 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro > Reporter: Daniel Thayer > Assignee: Robin Sommer > > Some of the elasticsearch tests use the nc command, and these tests > fail on some systems (such as debian 7 and 8) because there are at > least two different incompatible implementations of nc. A quick fix > is to use a shell script wrapper that chooses the correct command-line > arguments. -- This message was sent by Atlassian JIRA (v7.1.0-OD-06-005#71002) From noreply at bro.org Sun Feb 14 00:00:21 2016 From: noreply at bro.org (Merge Tracker) Date: Sun, 14 Feb 2016 00:00:21 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602140800.u1E80LFZ026929@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1536 [1] Bro Daniel Thayer Robin Sommer 2016-02-13 - Normal elasticsearch tests using nc fail on some systems BIT-1534 [2] Bro Johanna Amann Robin Sommer 2016-02-13 2.5 Normal Please merge topic/johanna/stats_smb_leak BIT-1507 [3] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ---------------------------------------------------- 2e0c203 [4] bro Johanna Amann 2016-02-04 Add missing break; in StartTLS case of IRC analyzer. Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- ------------ ---------- -------------------------------------- #55 [5] bro wglodek [6] 2016-02-07 http-evasion [7] #52 [8] bro J-Gras [9] 2016-01-18 Fixed matching mail address intel [10] #18 [11] bro-plugins jshlbrd [12] 2016-02-12 SSDP analyzer [13] [1] BIT-1536 https://bro-tracker.atlassian.net/browse/BIT-1536 [2] BIT-1534 https://bro-tracker.atlassian.net/browse/BIT-1534 [3] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [4] 2e0c203 https://github.com/bro/bro/commit/2e0c2035c9513f2a84e3d242adb22b076e324728 [5] Pull Request #55 https://github.com/bro/bro/pull/55 [6] wglodek https://github.com/wglodek [7] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [8] Pull Request #52 https://github.com/bro/bro/pull/52 [9] J-Gras https://github.com/J-Gras [10] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [11] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [12] jshlbrd https://github.com/jshlbrd [13] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From noreply at bro.org Mon Feb 15 00:00:21 2016 From: noreply at bro.org (Merge Tracker) Date: Mon, 15 Feb 2016 00:00:21 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602150800.u1F80LBm019996@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1536 [1] Bro Daniel Thayer Robin Sommer 2016-02-13 - Normal elasticsearch tests using nc fail on some systems BIT-1534 [2] Bro Johanna Amann Robin Sommer 2016-02-13 2.5 Normal Please merge topic/johanna/stats_smb_leak BIT-1507 [3] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ---------------------------------------------------- 2e0c203 [4] bro Johanna Amann 2016-02-04 Add missing break; in StartTLS case of IRC analyzer. Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- ------------ ---------- -------------------------------------- #55 [5] bro wglodek [6] 2016-02-07 http-evasion [7] #52 [8] bro J-Gras [9] 2016-01-18 Fixed matching mail address intel [10] #18 [11] bro-plugins jshlbrd [12] 2016-02-12 SSDP analyzer [13] [1] BIT-1536 https://bro-tracker.atlassian.net/browse/BIT-1536 [2] BIT-1534 https://bro-tracker.atlassian.net/browse/BIT-1534 [3] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [4] 2e0c203 https://github.com/bro/bro/commit/2e0c2035c9513f2a84e3d242adb22b076e324728 [5] Pull Request #55 https://github.com/bro/bro/pull/55 [6] wglodek https://github.com/wglodek [7] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [8] Pull Request #52 https://github.com/bro/bro/pull/52 [9] J-Gras https://github.com/J-Gras [10] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [11] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [12] jshlbrd https://github.com/jshlbrd [13] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Mon Feb 15 11:09:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 15 Feb 2016 13:09:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1534) Please merge topic/johanna/stats_smb_leak In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1534: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Please merge topic/johanna/stats_smb_leak > ----------------------------------------- > > Key: BIT-1534 > URL: https://bro-tracker.atlassian.net/browse/BIT-1534 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > Fix For: 2.5 > > > Please merge topic/johanna/stats_smb_leak > It fixes a memory leak in stats.cc and smb.cc. A test that triggers the leak in stats.cc is attached. > Due to not having access to test traffic I was not actually able to test the smb.cc case, but I am very confident that this triggers a leak and that the patch should fix it (TableVal->Assign needs an Unref for the index value (first parameter; this is the same case as in stats.cc)). -- This message was sent by Atlassian JIRA (v7.1.0-OD-06-005#71002) From jira at bro-tracker.atlassian.net Mon Feb 15 11:09:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 15 Feb 2016 13:09:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1536) elasticsearch tests using nc fail on some systems In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1536: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > elasticsearch tests using nc fail on some systems > ------------------------------------------------- > > Key: BIT-1536 > URL: https://bro-tracker.atlassian.net/browse/BIT-1536 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro > Reporter: Daniel Thayer > Assignee: Robin Sommer > > Some of the elasticsearch tests use the nc command, and these tests > fail on some systems (such as debian 7 and 8) because there are at > least two different incompatible implementations of nc. A quick fix > is to use a shell script wrapper that chooses the correct command-line > arguments. -- This message was sent by Atlassian JIRA (v7.1.0-OD-06-005#71002) From noreply at bro.org Tue Feb 16 00:00:27 2016 From: noreply at bro.org (Merge Tracker) Date: Tue, 16 Feb 2016 00:00:27 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602160800.u1G80RR6005230@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1507 [1] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ----------- ---------- ------------------------------------- #55 [2] bro wglodek [3] 2016-02-07 http-evasion [4] #52 [5] bro J-Gras [6] 2016-01-18 Fixed matching mail address intel [7] #18 [8] bro-plugins jshlbrd [9] 2016-02-12 SSDP analyzer [10] [1] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [2] Pull Request #55 https://github.com/bro/bro/pull/55 [3] wglodek https://github.com/wglodek [4] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [5] Pull Request #52 https://github.com/bro/bro/pull/52 [6] J-Gras https://github.com/J-Gras [7] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [8] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [9] jshlbrd https://github.com/jshlbrd [10] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From noreply at bro.org Wed Feb 17 00:00:24 2016 From: noreply at bro.org (Merge Tracker) Date: Wed, 17 Feb 2016 00:00:24 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602170800.u1H80Op6020310@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1507 [1] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ----------- ---------- ------------------------------------- #55 [2] bro wglodek [3] 2016-02-07 http-evasion [4] #52 [5] bro J-Gras [6] 2016-01-18 Fixed matching mail address intel [7] #18 [8] bro-plugins jshlbrd [9] 2016-02-12 SSDP analyzer [10] [1] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [2] Pull Request #55 https://github.com/bro/bro/pull/55 [3] wglodek https://github.com/wglodek [4] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [5] Pull Request #52 https://github.com/bro/bro/pull/52 [6] J-Gras https://github.com/J-Gras [7] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [8] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [9] jshlbrd https://github.com/jshlbrd [10] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Wed Feb 17 09:14:00 2016 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Wed, 17 Feb 2016 11:14:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1521) known services should probably ignore gridftp-data In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24200#comment-24200 ] Justin Azoff commented on BIT-1521: ----------------------------------- topic/jazoff/ticket1521 contains a branch that I believe fixes most of the issues with known-services. I think there may still be one outstanding bug (but it is something that is broken worse in the current code). The current code tracks services by (addr,port). If no service is detected on a port it will log it as (ip, port, empty). If a service is later detected on that port, nothing will be logged. This branch WILL log it, but it will also log twice in the opposite order, which is possibly not desired. So, this will work and is an improvement: {code} ip, port, empty # time passes ip, port, HTTP {code} But it may also log {code} ip, port, HTTP # time passes ip, port, empty {code} To fix that it would need to keep track of a separate (ip, port) set that had a non empty service detected. Once something like HTTP was detected the (ip, port) would be added, and then it would skip logging (ip, port, empty) > known services should probably ignore gridftp-data > -------------------------------------------------- > > Key: BIT-1521 > URL: https://bro-tracker.atlassian.net/browse/BIT-1521 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.4 > Reporter: Justin Azoff > Assignee: Justin Azoff > Priority: Low > Fix For: 2.5 > > > known services script does > {code} > if ( ! addr_matches_host(id$resp_h, service_tracking) || > "ftp-data" in c$service || # don't include ftp data sessions > ("DNS" in c$service && c$resp$size == 0) ) # for dns, require that the server talks. > return; > {code} > but should probably also ignore gridftp-data. Probably a good idea to add a set of services that behave like ftp for it to check. -- This message was sent by Atlassian JIRA (v7.2.0-OD-01-031#72000) From jira at bro-tracker.atlassian.net Wed Feb 17 09:23:00 2016 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Wed, 17 Feb 2016 11:23:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1521) known services should probably ignore gridftp-data In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24201#comment-24201 ] Seth Hall commented on BIT-1521: -------------------------------- What if you change known_services to... {code} table[addr, port] of set[string] {code} Then you could avoid the case you laid out here, which I suspect in production would be pretty annoying. > known services should probably ignore gridftp-data > -------------------------------------------------- > > Key: BIT-1521 > URL: https://bro-tracker.atlassian.net/browse/BIT-1521 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.4 > Reporter: Justin Azoff > Assignee: Justin Azoff > Priority: Low > Fix For: 2.5 > > > known services script does > {code} > if ( ! addr_matches_host(id$resp_h, service_tracking) || > "ftp-data" in c$service || # don't include ftp data sessions > ("DNS" in c$service && c$resp$size == 0) ) # for dns, require that the server talks. > return; > {code} > but should probably also ignore gridftp-data. Probably a good idea to add a set of services that behave like ftp for it to check. -- This message was sent by Atlassian JIRA (v7.2.0-OD-01-031#72000) From jira at bro-tracker.atlassian.net Wed Feb 17 12:20:00 2016 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Wed, 17 Feb 2016 14:20:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1521) known services should probably ignore gridftp-data In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24202#comment-24202 ] Justin Azoff commented on BIT-1521: ----------------------------------- Ok, I implemented that. I'm working on generating some pcaps for these test cases. One thing I think i ran into is I think this check (that was in the older version too) is not quite right: {code} # Handle the connection ending in case no protocol was ever detected. event connection_state_remove(c: connection) &priority=-5 { if (c$resp$state == TCP_ESTABLISHED ) event known_services_done(c); } {code} It seems (in my testing) that TCP_CLOSED occurs much more often. I think this check intends to filter connections that completed a handshake, but is not really doing so. I honestly have no idea why a connection would be TCP_ESTABLISHED inside connection_state_removed. This is a similar issue as in BIT-1535.. how to determine if a connection succeeded. Vlad also mentioned that starting a timer for each protocol confirmation might not be the best idea. We could rely 100% on connection_state_removed if we did not mind delaying a known_service log entry until the connection closed. I think if I can fix that if statement, the timers could be removed and the entire thing could be greatly simplified. Vlad also brought up something I was thinking about as well that c$service should probably be an ordered vector.. since now that I fixed the multiple-service-logging issue you see things like {code} FTP,SSL,gridftp gridftp,FTP,SSL SSL,gridftp,FTP or SSL,SMTP {code} but i believe chronologically they should all be {code} FTP,SSL,gridftp SMTP,SSL {code} > known services should probably ignore gridftp-data > -------------------------------------------------- > > Key: BIT-1521 > URL: https://bro-tracker.atlassian.net/browse/BIT-1521 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.4 > Reporter: Justin Azoff > Assignee: Justin Azoff > Priority: Low > Fix For: 2.5 > > > known services script does > {code} > if ( ! addr_matches_host(id$resp_h, service_tracking) || > "ftp-data" in c$service || # don't include ftp data sessions > ("DNS" in c$service && c$resp$size == 0) ) # for dns, require that the server talks. > return; > {code} > but should probably also ignore gridftp-data. Probably a good idea to add a set of services that behave like ftp for it to check. -- This message was sent by Atlassian JIRA (v7.2.0-OD-01-031#72000) From noreply at bro.org Thu Feb 18 00:00:26 2016 From: noreply at bro.org (Merge Tracker) Date: Thu, 18 Feb 2016 00:00:26 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602180800.u1I80Quk005192@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1507 [1] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ----------- ---------- ------------------------------------- #55 [2] bro wglodek [3] 2016-02-07 http-evasion [4] #52 [5] bro J-Gras [6] 2016-01-18 Fixed matching mail address intel [7] #18 [8] bro-plugins jshlbrd [9] 2016-02-12 SSDP analyzer [10] [1] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [2] Pull Request #55 https://github.com/bro/bro/pull/55 [3] wglodek https://github.com/wglodek [4] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [5] Pull Request #52 https://github.com/bro/bro/pull/52 [6] J-Gras https://github.com/J-Gras [7] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [8] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [9] jshlbrd https://github.com/jshlbrd [10] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From noreply at bro.org Fri Feb 19 00:00:25 2016 From: noreply at bro.org (Merge Tracker) Date: Fri, 19 Feb 2016 00:00:25 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602190800.u1J80P5f031733@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1507 [1] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ----------- ---------- ------------------------------------- #55 [2] bro wglodek [3] 2016-02-07 http-evasion [4] #52 [5] bro J-Gras [6] 2016-01-18 Fixed matching mail address intel [7] #18 [8] bro-plugins jshlbrd [9] 2016-02-12 SSDP analyzer [10] [1] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [2] Pull Request #55 https://github.com/bro/bro/pull/55 [3] wglodek https://github.com/wglodek [4] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [5] Pull Request #52 https://github.com/bro/bro/pull/52 [6] J-Gras https://github.com/J-Gras [7] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [8] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [9] jshlbrd https://github.com/jshlbrd [10] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Fri Feb 19 05:36:00 2016 From: jira at bro-tracker.atlassian.net (=?UTF-8?Q?Carlos_Terr=C3=B3n_=28JIRA=29?=) Date: Fri, 19 Feb 2016 07:36:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1537) bro segfaults after compile in MacOS X 10.11 El Capitan In-Reply-To: References: Message-ID: Carlos Terr?n created BIT-1537: ---------------------------------- Summary: bro segfaults after compile in MacOS X 10.11 El Capitan Key: BIT-1537 URL: https://bro-tracker.atlassian.net/browse/BIT-1537 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.4 Reporter: Carlos Terr?n Priority: High After compile with {code} ./configure --prefix=/usr/local make make install {code} And try to execute bro with: {code} bro -i en4 local {code} bro segfaults with {code} Program received signal SIGSEGV, Segmentation fault. 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( cert_val=, fid=) at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 175 char *exponent = BN_bn2dec(pkey->pkey.rsa->e); (gdb) bt #0 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( cert_val=, fid=) at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 #1 0x0000000100303e5d in file_analysis::X509::EndOfFile (this=0x105f8b710) at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:56 #2 0x000000010033f57a in file_analysis::File::EndOfFile (this=0x100961090) at /Users/terron/tmp/bro-2.4.1/src/file_analysis/File.cc:522 #3 0x000000010033bc6e in file_analysis::Manager::RemoveFile ( this=0x105f8b710, file_id=...) at /Users/terron/tmp/bro-2.4.1/src/file_analysis/Manager.cc:395 #4 0x00000001002d910a in binpac::TLSHandshake::Handshake_Conn::proc_certificate (this=0x105f8a220, is_orig=false, certificates=0x100961f90) at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:180 #5 0x00000001002d99d4 in binpac::TLSHandshake::Handshake_Conn::proc_v3_certificate (this=0x105f8b710, is_orig=16, cl=) at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:323 #6 0x00000001002dc430 in binpac::TLSHandshake::Certificate::Parse ( this=0x105f8a220, t_begin_of_data=, t_end_of_data=0x101022f2e "", t_context=0x10095e480) at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:1977 {code} -- This message was sent by Atlassian JIRA (v7.2.0-OD-01-031#72000) From jira at bro-tracker.atlassian.net Fri Feb 19 05:40:00 2016 From: jira at bro-tracker.atlassian.net (=?UTF-8?Q?Carlos_Terr=C3=B3n_=28JIRA=29?=) Date: Fri, 19 Feb 2016 07:40:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1537) bro segfaults after compile in MacOS X 10.11 El Capitan In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24204#comment-24204 ] Carlos Terr?n commented on BIT-1537: ------------------------------------ The full stack trace with lldb {code} * thread #1: tid = 0x58b6c8, 0x00000001003045d2 bro`file_analysis::X509::ParseCertificate(cert_val=, fid=) + 1282 at X509.cc:175, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x28) * frame #0: 0x00000001003045d2 bro`file_analysis::X509::ParseCertificate(cert_val=, fid=) + 1282 at X509.cc:175 frame #1: 0x0000000100303e5d bro`file_analysis::X509::EndOfFile(this=0x00000001064d44b0) + 221 at X509.cc:56 frame #2: 0x000000010033f57a bro`file_analysis::File::EndOfFile(this=0x00000001064c14c0) + 170 at File.cc:522 frame #3: 0x000000010033bc6e bro`file_analysis::Manager::RemoveFile(this=0x0000000100d01f60, file_id=) + 78 at Manager.cc:395 frame #4: 0x00000001002d910a bro`binpac::TLSHandshake::Handshake_Conn::proc_certificate(this=0x00000001064a65a0, is_orig=false, certificates=0x00000001064bdf60 size=3) + 554 at tls-handshake_pac.cc:180 frame #5: 0x00000001002d99d4 bro`binpac::TLSHandshake::Handshake_Conn::proc_v3_certificate(this=0x00000001064a65a0, is_orig=false, cl=) + 164 at tls-handshake_pac.cc:323 frame #6: 0x00000001002dc430 bro`binpac::TLSHandshake::Certificate::Parse(this=0x000000010645d840, t_begin_of_data=, t_end_of_data="", t_context=0x00000001064bb5a0) + 336 at tls-handshake_pac.cc:1977 frame #7: 0x00000001002da599 bro`binpac::TLSHandshake::Handshake::Parse(this=0x00000001064c0c10, t_begin_of_data="", t_end_of_data="", t_context=0x00000001064bb5a0, t_byteorder=) + 793 at tls-handshake_pac.cc:796 frame #8: 0x00000001002da16a bro`binpac::TLSHandshake::HandshakeRecord::ParseBuffer(this=0x00000001064bea10, t_flow_buffer=, t_context=0x00000001064bb5a0, t_byteorder=0) + 442 at tls-handshake_pac.cc:586 frame #9: 0x00000001002dd2cf bro`binpac::TLSHandshake::HandshakePDU::ParseBuffer(this=0x00000001064be9c0, t_flow_buffer=0x00000001064be200, t_context=0x00000001064bb5a0) + 191 at tls-handshake_pac.cc:964 frame #10: 0x00000001002e0c51 bro`binpac::TLSHandshake::Handshake_Flow::NewData(this=, t_begin_of_data=, t_end_of_data=) + 193 at tls-handshake_pac.cc:3776 frame #11: 0x00000001002d62d9 bro`analyzer::ssl::SSL_Analyzer::SendHandshake(this=, begin=, end=, orig=) + 41 at SSL.cc:71 frame #12: 0x00000001002e3c6f bro`binpac::SSL::Handshake::Parse(unsigned char const*, unsigned char const*, binpac::SSL::ContextSSL*) [inlined] binpac::SSL::SSL_Conn::proc_handshake(this=) + 127 at ssl_pac.cc:384 frame #13: 0x00000001002e3c5f bro`binpac::SSL::Handshake::Parse(this=0x00000001064a8660, t_begin_of_data="\v", t_end_of_data=, t_context=0x00000001064bfc50) + 111 at ssl_pac.cc:1099 frame #14: 0x00000001002e51be bro`binpac::SSL::SSLRecord::ParseBuffer(this=0x00000001064bfb00, t_flow_buffer=, t_context=0x00000001064bfc50) + 462 at ssl_pac.cc:982 frame #15: 0x00000001002e5afb bro`binpac::SSL::SSLPDU::ParseBuffer(this=0x00000001064bfad0, t_flow_buffer=0x00000001064be380, t_context=0x00000001064bfc50) + 203 at ssl_pac.cc:1698 frame #16: 0x00000001002e5d61 bro`binpac::SSL::SSL_Flow::NewData(this=, t_begin_of_data=, t_end_of_data=) + 193 at ssl_pac.cc:1769 frame #17: 0x00000001002d623a bro`analyzer::ssl::SSL_Analyzer::DeliverStream(this=, len=, data=, orig=) + 106 at SSL.cc:59 frame #18: 0x0000000100353352 bro`analyzer::Analyzer::NextStream(this=, len=, data=, is_orig=) + 98 at Analyzer.cc:245 frame #19: 0x000000010035376a bro`analyzer::Analyzer::ForwardStream(this=0x0\x01\x05", is_orig=) + 218 at Analyzer.cc:331\"0\r\x06\t*\x86H\x86? frame #20: 0x00000001002f2141 bro`analyzer::tcp::TCP_Reassembler::DeliverBlock(this=0x00000001064bf380, seq=, len=, data=) + 305 at TCP_Reassembler.cc:647 frame #21: 0x00000001002f1f93 bro`analyzer::tcp::TCP_Reassembler::BlockInserted(this=0x00000001064bf380, start_block=) + 115 at TCP_Reassembler.cc:393 frame #22: 0x00000001002f2472 bro`analyzer::tcp::TCP_Reassembler::DataSent(this=0x00000001064bf380, t=, seq=, len=, data=, replaying=) + 114 at TCP_Reassembler.cc:492 frame #23: 0x00000001002f1010 bro`analyzer::tcp::TCP_Endpoint::DataSent(this=0x00000001064bf6f0, t=, seq=2837, len=1047, caplen=1047, data="oTr\x01\x05", ip=, tp=0x000000010083bc48) + 96 at TCP_Endpoint.cc:205 frame #24: 0x00000001002eec69 bro`analyzer::tcp::TCP_Analyzer::DeliverPacket(int, unsigned char const*, bool, unsigned long long, IP_Hdr const*, int) [inlined] analyzer::tcp::TCP_Analyzer::DeliverData(t=, data="oTrust Globa\x01\x05", len=, caplen=, ip=0x00007fff5fbff220, endpoint=, rel_data_seq=2837) + 45 at TCP.cc:982 frame #25: 0x00000001002eec3c bro`analyzer::tcp::TCP_Analyzer::DeliverPacket(this=0x00000001064bdfc0, len=1047, data="oTrust Global CA0\x82\x01\"0\r\x06\t*\\x01\x05", is_orig=false, seq=, ip=0x00007fff5fbff220, caplen=1047) + 6748 at TCP.cc:1381 frame #26: 0x0000000100353206 bro`analyzer::Analyzer::NextPacket(this=, len=, data=, is_orig=, seq=, ip=, caplen=) + 102 at Analyzer.cc:222 frame #27: 0x0000000100037c50 bro`Connection::NextPacket(this=0x00000001064bed50, t=, is_orig=, ip=, len=, caplen=, data=, record_packet=0x00007fff5fbfefec, record_content=, hdr=0x00000001060390a8, pkt="h[5??3", hdr_size=14) + 192 at Conn.cc:260 frame #28: 0x00000001000cec68 bro`NetSessions::DoNextPacket(this=0x0000000106441c10, t=1455889030.109709, hdr=0x00000001060390a8, ip_hdr=, pkt="h[5??3", hdr_size=14, encapsulation=0x0000000000000000) + 3960 at Sessions.cc:758 frame #29: 0x00000001000cd9ae bro`NetSessions::NextPacket(this=0x0000000106441c10, t=1455889030.109709, hdr=0x00000001060390a8, pkt="h[5??3", hdr_size=) + 430 at Sessions.cc:231 frame #30: 0x000000010009d86f bro`net_packet_dispatch(t=1455889030.109709, hdr=0x00000001060390a8, pkt="h[5??3", hdr_size=14, src_ps=0x0000000106038f30) + 431 at Net.cc:281 frame #31: 0x000000010032fafe bro`iosource::PktSrc::Process(this=0x0000000106038f30) + 926 at PktSrc.cc:456 frame #32: 0x000000010009d9b3 bro`net_run() + 163 at Net.cc:329 frame #33: 0x0000000100022695 bro`main(argc=, argv=0x41d5b1c6a0349e23) + 6437 at main.cc:1190 frame #34: 0x00007fff900dd5ad libdyld.dylib`start + 1 {code} > bro segfaults after compile in MacOS X 10.11 El Capitan > ------------------------------------------------------- > > Key: BIT-1537 > URL: https://bro-tracker.atlassian.net/browse/BIT-1537 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Carlos Terr?n > Priority: High > > After compile with > {code} > ./configure --prefix=/usr/local > make > make install > {code} > And try to execute bro with: > {code} > bro -i en4 local > {code} > bro segfaults with > {code} > Program received signal SIGSEGV, Segmentation fault. > 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > 175 char *exponent = BN_bn2dec(pkey->pkey.rsa->e); > (gdb) bt > #0 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > #1 0x0000000100303e5d in file_analysis::X509::EndOfFile (this=0x105f8b710) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:56 > #2 0x000000010033f57a in file_analysis::File::EndOfFile (this=0x100961090) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/File.cc:522 > #3 0x000000010033bc6e in file_analysis::Manager::RemoveFile ( > this=0x105f8b710, file_id=...) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/Manager.cc:395 > #4 0x00000001002d910a in binpac::TLSHandshake::Handshake_Conn::proc_certificate (this=0x105f8a220, is_orig=false, certificates=0x100961f90) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:180 > #5 0x00000001002d99d4 in binpac::TLSHandshake::Handshake_Conn::proc_v3_certificate (this=0x105f8b710, is_orig=16, cl=) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:323 > #6 0x00000001002dc430 in binpac::TLSHandshake::Certificate::Parse ( > this=0x105f8a220, t_begin_of_data=, > t_end_of_data=0x101022f2e "", t_context=0x10095e480) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:1977 > {code} -- This message was sent by Atlassian JIRA (v7.2.0-OD-01-031#72000) From jira at bro-tracker.atlassian.net Fri Feb 19 07:10:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 19 Feb 2016 09:10:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1537) bro segfaults after compile in MacOS X 10.11 El Capitan In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann reassigned BIT-1537: ---------------------------------- Assignee: Johanna Amann > bro segfaults after compile in MacOS X 10.11 El Capitan > ------------------------------------------------------- > > Key: BIT-1537 > URL: https://bro-tracker.atlassian.net/browse/BIT-1537 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Carlos Terr?n > Assignee: Johanna Amann > Priority: High > > After compile with > {code} > ./configure --prefix=/usr/local > make > make install > {code} > And try to execute bro with: > {code} > bro -i en4 local > {code} > bro segfaults with > {code} > Program received signal SIGSEGV, Segmentation fault. > 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > 175 char *exponent = BN_bn2dec(pkey->pkey.rsa->e); > (gdb) bt > #0 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > #1 0x0000000100303e5d in file_analysis::X509::EndOfFile (this=0x105f8b710) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:56 > #2 0x000000010033f57a in file_analysis::File::EndOfFile (this=0x100961090) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/File.cc:522 > #3 0x000000010033bc6e in file_analysis::Manager::RemoveFile ( > this=0x105f8b710, file_id=...) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/Manager.cc:395 > #4 0x00000001002d910a in binpac::TLSHandshake::Handshake_Conn::proc_certificate (this=0x105f8a220, is_orig=false, certificates=0x100961f90) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:180 > #5 0x00000001002d99d4 in binpac::TLSHandshake::Handshake_Conn::proc_v3_certificate (this=0x105f8b710, is_orig=16, cl=) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:323 > #6 0x00000001002dc430 in binpac::TLSHandshake::Certificate::Parse ( > this=0x105f8a220, t_begin_of_data=, > t_end_of_data=0x101022f2e "", t_context=0x10095e480) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:1977 > {code} -- This message was sent by Atlassian JIRA (v7.2.0-OD-01-031#72000) From jira at bro-tracker.atlassian.net Fri Feb 19 09:06:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 19 Feb 2016 11:06:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1537) bro segfaults after compile in MacOS X 10.11 El Capitan In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1537: ------------------------------- Priority: Normal (was: High) > bro segfaults after compile in MacOS X 10.11 El Capitan > ------------------------------------------------------- > > Key: BIT-1537 > URL: https://bro-tracker.atlassian.net/browse/BIT-1537 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Carlos Terr?n > Assignee: Johanna Amann > > After compile with > {code} > ./configure --prefix=/usr/local > make > make install > {code} > And try to execute bro with: > {code} > bro -i en4 local > {code} > bro segfaults with > {code} > Program received signal SIGSEGV, Segmentation fault. > 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > 175 char *exponent = BN_bn2dec(pkey->pkey.rsa->e); > (gdb) bt > #0 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > #1 0x0000000100303e5d in file_analysis::X509::EndOfFile (this=0x105f8b710) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:56 > #2 0x000000010033f57a in file_analysis::File::EndOfFile (this=0x100961090) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/File.cc:522 > #3 0x000000010033bc6e in file_analysis::Manager::RemoveFile ( > this=0x105f8b710, file_id=...) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/Manager.cc:395 > #4 0x00000001002d910a in binpac::TLSHandshake::Handshake_Conn::proc_certificate (this=0x105f8a220, is_orig=false, certificates=0x100961f90) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:180 > #5 0x00000001002d99d4 in binpac::TLSHandshake::Handshake_Conn::proc_v3_certificate (this=0x105f8b710, is_orig=16, cl=) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:323 > #6 0x00000001002dc430 in binpac::TLSHandshake::Certificate::Parse ( > this=0x105f8a220, t_begin_of_data=, > t_end_of_data=0x101022f2e "", t_context=0x10095e480) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:1977 > {code} -- This message was sent by Atlassian JIRA (v7.2.0-OD-01-031#72000) From jira at bro-tracker.atlassian.net Fri Feb 19 09:09:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 19 Feb 2016 11:09:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1537) bro segfaults after compile in MacOS X 10.11 El Capitan In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24205#comment-24205 ] Johanna Amann commented on BIT-1537: ------------------------------------ Ok, I think I figured this one out - and it is a bit mean. I think Bro uses the header files of a new version of OpenSSL (probably installed via hombrew, manually, etc). but then links against the system version of OpenSSL, where the format of the in-memory data structures changed a bit. Could you please send me the output of otool -L [path-to-bro]. Also could you try to recompile bro by first doing a {code} make distclean {code} and then {code} LDFLAGS="-L[path-to-openssl-lib] ./configure {code} I expect the path to be /opt/local/lib, /usr/local/lib or similar :) > bro segfaults after compile in MacOS X 10.11 El Capitan > ------------------------------------------------------- > > Key: BIT-1537 > URL: https://bro-tracker.atlassian.net/browse/BIT-1537 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Carlos Terr?n > Assignee: Johanna Amann > > After compile with > {code} > ./configure --prefix=/usr/local > make > make install > {code} > And try to execute bro with: > {code} > bro -i en4 local > {code} > bro segfaults with > {code} > Program received signal SIGSEGV, Segmentation fault. > 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > 175 char *exponent = BN_bn2dec(pkey->pkey.rsa->e); > (gdb) bt > #0 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > #1 0x0000000100303e5d in file_analysis::X509::EndOfFile (this=0x105f8b710) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:56 > #2 0x000000010033f57a in file_analysis::File::EndOfFile (this=0x100961090) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/File.cc:522 > #3 0x000000010033bc6e in file_analysis::Manager::RemoveFile ( > this=0x105f8b710, file_id=...) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/Manager.cc:395 > #4 0x00000001002d910a in binpac::TLSHandshake::Handshake_Conn::proc_certificate (this=0x105f8a220, is_orig=false, certificates=0x100961f90) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:180 > #5 0x00000001002d99d4 in binpac::TLSHandshake::Handshake_Conn::proc_v3_certificate (this=0x105f8b710, is_orig=16, cl=) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:323 > #6 0x00000001002dc430 in binpac::TLSHandshake::Certificate::Parse ( > this=0x105f8a220, t_begin_of_data=, > t_end_of_data=0x101022f2e "", t_context=0x10095e480) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:1977 > {code} -- This message was sent by Atlassian JIRA (v7.2.0-OD-01-031#72000) From jira at bro-tracker.atlassian.net Fri Feb 19 10:13:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 19 Feb 2016 12:13:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1537) bro segfaults after compile in MacOS X 10.11 El Capitan In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1537: ------------------------------- Fix Version/s: 2.5 > bro segfaults after compile in MacOS X 10.11 El Capitan > ------------------------------------------------------- > > Key: BIT-1537 > URL: https://bro-tracker.atlassian.net/browse/BIT-1537 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Carlos Terr?n > Assignee: Johanna Amann > Fix For: 2.5 > > > After compile with > {code} > ./configure --prefix=/usr/local > make > make install > {code} > And try to execute bro with: > {code} > bro -i en4 local > {code} > bro segfaults with > {code} > Program received signal SIGSEGV, Segmentation fault. > 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > 175 char *exponent = BN_bn2dec(pkey->pkey.rsa->e); > (gdb) bt > #0 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > #1 0x0000000100303e5d in file_analysis::X509::EndOfFile (this=0x105f8b710) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:56 > #2 0x000000010033f57a in file_analysis::File::EndOfFile (this=0x100961090) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/File.cc:522 > #3 0x000000010033bc6e in file_analysis::Manager::RemoveFile ( > this=0x105f8b710, file_id=...) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/Manager.cc:395 > #4 0x00000001002d910a in binpac::TLSHandshake::Handshake_Conn::proc_certificate (this=0x105f8a220, is_orig=false, certificates=0x100961f90) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:180 > #5 0x00000001002d99d4 in binpac::TLSHandshake::Handshake_Conn::proc_v3_certificate (this=0x105f8b710, is_orig=16, cl=) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:323 > #6 0x00000001002dc430 in binpac::TLSHandshake::Certificate::Parse ( > this=0x105f8a220, t_begin_of_data=, > t_end_of_data=0x101022f2e "", t_context=0x10095e480) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:1977 > {code} -- This message was sent by Atlassian JIRA (v7.2.0-OD-01-031#72000) From noreply at bro.org Sat Feb 20 00:00:22 2016 From: noreply at bro.org (Merge Tracker) Date: Sat, 20 Feb 2016 00:00:22 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602200800.u1K80MNB024363@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1507 [1] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ----------- ---------- ------------------------------------- #55 [2] bro wglodek [3] 2016-02-07 http-evasion [4] #52 [5] bro J-Gras [6] 2016-01-18 Fixed matching mail address intel [7] #18 [8] bro-plugins jshlbrd [9] 2016-02-12 SSDP analyzer [10] [1] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [2] Pull Request #55 https://github.com/bro/bro/pull/55 [3] wglodek https://github.com/wglodek [4] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [5] Pull Request #52 https://github.com/bro/bro/pull/52 [6] J-Gras https://github.com/J-Gras [7] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [8] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [9] jshlbrd https://github.com/jshlbrd [10] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Sat Feb 20 07:27:00 2016 From: jira at bro-tracker.atlassian.net (LiJinmiao (JIRA)) Date: Sat, 20 Feb 2016 09:27:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1538) capture loss and notice of `Conn::Content_Gap` are too many In-Reply-To: References: Message-ID: LiJinmiao created BIT-1538: ------------------------------ Summary: capture loss and notice of `Conn::Content_Gap` are too many Key: BIT-1538 URL: https://bro-tracker.atlassian.net/browse/BIT-1538 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.4 Environment: Dell R620 Ubuntu 14.04.04 Server LTS + kde desktop pf_ring 6.2.0 bro 2.4 with pf_ring Reporter: LiJinmiao Priority: High Attachments: capture_loss.log, files.log, http.log, notice.log, prof.log, reporter.log, stats.log, weird.log I use the bro to extract files from http stream. I installed bro with source code on ubuntu server 14.04. And I want to use virtual machine sandbox based on VirtualBox, so I installed desktop finally. My installation document is `https://www.bro.org/sphinx/configuration/index.html#installing-pf-ring`. But I didn't start bro by `broctl`. I just started bro with command `bro -i eth0 filextraction.bro`. And there are too many packet loss. Then I test the bro on another machine that has the same environment without desktop. And there is no packet loss. So I'm confused. By the way, the driver of nic is `igb`. And I can't understand the document of `https://www.bro.org/documentation/faq.html#how-can-i-reduce-the-amount-of-captureloss-or-dropped-packets-notices` clearly. Thanks for any help. -- This message was sent by Atlassian JIRA (v7.2.0-OD-01-031#72000) From jira at bro-tracker.atlassian.net Sat Feb 20 16:57:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Sat, 20 Feb 2016 18:57:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1538) capture loss and notice of `Conn::Content_Gap` are too many In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1538: ------------------------------- Resolution: Invalid Status: Closed (was: Open) Hello, Thanks for contacting us. I am closing this ticket, since this is not a bug but a Bro usage question. Our mailing list is the best place to receive answers to these kinds of questions. It's followed by Bro developers and enthusiasts. You can join here: http://mailman.icsi.berkeley.edu/mailman/listinfo/bro > capture loss and notice of `Conn::Content_Gap` are too many > ------------------------------------------------------------ > > Key: BIT-1538 > URL: https://bro-tracker.atlassian.net/browse/BIT-1538 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Environment: Dell R620 > Ubuntu 14.04.04 Server LTS + kde desktop > pf_ring 6.2.0 > bro 2.4 with pf_ring > Reporter: LiJinmiao > Priority: High > Labels: analyzer > Attachments: capture_loss.log, files.log, http.log, notice.log, prof.log, reporter.log, stats.log, weird.log > > > I use the bro to extract files from http stream. > I installed bro with source code on ubuntu server 14.04. And I want to use virtual machine sandbox based on VirtualBox, so I installed desktop finally. > My installation document is `https://www.bro.org/sphinx/configuration/index.html#installing-pf-ring`. But I didn't start bro by `broctl`. I just started bro with command `bro -i eth0 filextraction.bro`. And there are too many packet loss. > Then I test the bro on another machine that has the same environment without desktop. And there is no packet loss. > So I'm confused. > By the way, the driver of nic is `igb`. > And I can't understand the document of `https://www.bro.org/documentation/faq.html#how-can-i-reduce-the-amount-of-captureloss-or-dropped-packets-notices` clearly. > Thanks for any help. -- This message was sent by Atlassian JIRA (v7.2.0-OD-01-031#72000) From jira at bro-tracker.atlassian.net Sat Feb 20 17:57:00 2016 From: jira at bro-tracker.atlassian.net (LiJinmiao (JIRA)) Date: Sat, 20 Feb 2016 19:57:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1538) capture loss and notice of `Conn::Content_Gap` are too many In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24207#comment-24207 ] LiJinmiao commented on BIT-1538: -------------------------------- Oh, I'm sorry for my behaviour. Thanks for you help > capture loss and notice of `Conn::Content_Gap` are too many > ------------------------------------------------------------ > > Key: BIT-1538 > URL: https://bro-tracker.atlassian.net/browse/BIT-1538 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Environment: Dell R620 > Ubuntu 14.04.04 Server LTS + kde desktop > pf_ring 6.2.0 > bro 2.4 with pf_ring > Reporter: LiJinmiao > Priority: High > Labels: analyzer > Attachments: capture_loss.log, files.log, http.log, notice.log, prof.log, reporter.log, stats.log, weird.log > > > I use the bro to extract files from http stream. > I installed bro with source code on ubuntu server 14.04. And I want to use virtual machine sandbox based on VirtualBox, so I installed desktop finally. > My installation document is `https://www.bro.org/sphinx/configuration/index.html#installing-pf-ring`. But I didn't start bro by `broctl`. I just started bro with command `bro -i eth0 filextraction.bro`. And there are too many packet loss. > Then I test the bro on another machine that has the same environment without desktop. And there is no packet loss. > So I'm confused. > By the way, the driver of nic is `igb`. > And I can't understand the document of `https://www.bro.org/documentation/faq.html#how-can-i-reduce-the-amount-of-captureloss-or-dropped-packets-notices` clearly. > Thanks for any help. -- This message was sent by Atlassian JIRA (v7.2.0-OD-01-031#72000) From noreply at bro.org Sun Feb 21 00:00:18 2016 From: noreply at bro.org (Merge Tracker) Date: Sun, 21 Feb 2016 00:00:18 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602210800.u1L80Itb005686@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1507 [1] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ----------- ---------- ------------------------------------- #55 [2] bro wglodek [3] 2016-02-07 http-evasion [4] #52 [5] bro J-Gras [6] 2016-01-18 Fixed matching mail address intel [7] #18 [8] bro-plugins jshlbrd [9] 2016-02-12 SSDP analyzer [10] [1] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [2] Pull Request #55 https://github.com/bro/bro/pull/55 [3] wglodek https://github.com/wglodek [4] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [5] Pull Request #52 https://github.com/bro/bro/pull/52 [6] J-Gras https://github.com/J-Gras [7] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [8] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [9] jshlbrd https://github.com/jshlbrd [10] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From seth at icir.org Sun Feb 21 07:00:21 2016 From: seth at icir.org (Seth Hall) Date: Sun, 21 Feb 2016 10:00:21 -0500 Subject: [Bro-Dev] Broker updates? Message-ID: <3A669448-2C68-4CBF-8D90-F865BD517BE6@icir.org> If there are updates to a Broker store, is there a way that I can get evented notification that a key was modified? I'm not seeing anything that provides that functionality yet, but I need it for something I'm working on. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From noreply at bro.org Mon Feb 22 00:00:32 2016 From: noreply at bro.org (Merge Tracker) Date: Mon, 22 Feb 2016 00:00:32 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602220800.u1M80Wu9020397@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1507 [1] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ----------- ---------- ------------------------------------- #55 [2] bro wglodek [3] 2016-02-07 http-evasion [4] #52 [5] bro J-Gras [6] 2016-01-18 Fixed matching mail address intel [7] #18 [8] bro-plugins jshlbrd [9] 2016-02-12 SSDP analyzer [10] [1] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [2] Pull Request #55 https://github.com/bro/bro/pull/55 [3] wglodek https://github.com/wglodek [4] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [5] Pull Request #52 https://github.com/bro/bro/pull/52 [6] J-Gras https://github.com/J-Gras [7] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [8] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [9] jshlbrd https://github.com/jshlbrd [10] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From noreply at bro.org Tue Feb 23 00:00:40 2016 From: noreply at bro.org (Merge Tracker) Date: Tue, 23 Feb 2016 00:00:40 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602230800.u1N80e36006184@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1507 [1] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ----------- ---------- ------------------------------------- #55 [2] bro wglodek [3] 2016-02-07 http-evasion [4] #52 [5] bro J-Gras [6] 2016-01-18 Fixed matching mail address intel [7] #18 [8] bro-plugins jshlbrd [9] 2016-02-12 SSDP analyzer [10] [1] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [2] Pull Request #55 https://github.com/bro/bro/pull/55 [3] wglodek https://github.com/wglodek [4] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [5] Pull Request #52 https://github.com/bro/bro/pull/52 [6] J-Gras https://github.com/J-Gras [7] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [8] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [9] jshlbrd https://github.com/jshlbrd [10] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Tue Feb 23 14:51:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Tue, 23 Feb 2016 16:51:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1537) bro segfaults after compile in MacOS X 10.11 El Capitan In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24300#comment-24300 ] Johanna Amann commented on BIT-1537: ------------------------------------ This is fixed in topic/johanna/openssl in master and cmake. The branch adds a new configure time check that tests if the OpenSSL headers and libraries match. I tested that this check works and identifies the problem if present. Furthermore, we switch from our self-made macros of finding OpenSSL to the ones provided by cmake. These do a better job of identifying the correct library and include paths and probably will prevent the error from occurring nearly at all times. > bro segfaults after compile in MacOS X 10.11 El Capitan > ------------------------------------------------------- > > Key: BIT-1537 > URL: https://bro-tracker.atlassian.net/browse/BIT-1537 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Carlos Terr?n > Assignee: Johanna Amann > Fix For: 2.5 > > > After compile with > {code} > ./configure --prefix=/usr/local > make > make install > {code} > And try to execute bro with: > {code} > bro -i en4 local > {code} > bro segfaults with > {code} > Program received signal SIGSEGV, Segmentation fault. > 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > 175 char *exponent = BN_bn2dec(pkey->pkey.rsa->e); > (gdb) bt > #0 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > #1 0x0000000100303e5d in file_analysis::X509::EndOfFile (this=0x105f8b710) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:56 > #2 0x000000010033f57a in file_analysis::File::EndOfFile (this=0x100961090) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/File.cc:522 > #3 0x000000010033bc6e in file_analysis::Manager::RemoveFile ( > this=0x105f8b710, file_id=...) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/Manager.cc:395 > #4 0x00000001002d910a in binpac::TLSHandshake::Handshake_Conn::proc_certificate (this=0x105f8a220, is_orig=false, certificates=0x100961f90) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:180 > #5 0x00000001002d99d4 in binpac::TLSHandshake::Handshake_Conn::proc_v3_certificate (this=0x105f8b710, is_orig=16, cl=) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:323 > #6 0x00000001002dc430 in binpac::TLSHandshake::Certificate::Parse ( > this=0x105f8a220, t_begin_of_data=, > t_end_of_data=0x101022f2e "", t_context=0x10095e480) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:1977 > {code} -- This message was sent by Atlassian JIRA (v7.2.0-OD-02-009#72000) From jira at bro-tracker.atlassian.net Tue Feb 23 14:51:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Tue, 23 Feb 2016 16:51:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1537) bro segfaults after compile in MacOS X 10.11 El Capitan In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1537: ------------------------------- Status: Merge Request (was: Open) Assignee: (was: Johanna Amann) > bro segfaults after compile in MacOS X 10.11 El Capitan > ------------------------------------------------------- > > Key: BIT-1537 > URL: https://bro-tracker.atlassian.net/browse/BIT-1537 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Carlos Terr?n > Fix For: 2.5 > > > After compile with > {code} > ./configure --prefix=/usr/local > make > make install > {code} > And try to execute bro with: > {code} > bro -i en4 local > {code} > bro segfaults with > {code} > Program received signal SIGSEGV, Segmentation fault. > 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > 175 char *exponent = BN_bn2dec(pkey->pkey.rsa->e); > (gdb) bt > #0 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > #1 0x0000000100303e5d in file_analysis::X509::EndOfFile (this=0x105f8b710) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:56 > #2 0x000000010033f57a in file_analysis::File::EndOfFile (this=0x100961090) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/File.cc:522 > #3 0x000000010033bc6e in file_analysis::Manager::RemoveFile ( > this=0x105f8b710, file_id=...) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/Manager.cc:395 > #4 0x00000001002d910a in binpac::TLSHandshake::Handshake_Conn::proc_certificate (this=0x105f8a220, is_orig=false, certificates=0x100961f90) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:180 > #5 0x00000001002d99d4 in binpac::TLSHandshake::Handshake_Conn::proc_v3_certificate (this=0x105f8b710, is_orig=16, cl=) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:323 > #6 0x00000001002dc430 in binpac::TLSHandshake::Certificate::Parse ( > this=0x105f8a220, t_begin_of_data=, > t_end_of_data=0x101022f2e "", t_context=0x10095e480) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:1977 > {code} -- This message was sent by Atlassian JIRA (v7.2.0-OD-02-009#72000) From jira at bro-tracker.atlassian.net Tue Feb 23 15:08:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Tue, 23 Feb 2016 17:08:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1537) bro segfaults after compile in MacOS X 10.11 El Capitan In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24302#comment-24302 ] Johanna Amann commented on BIT-1537: ------------------------------------ Please note that I pushed a second update to the cmake branch and the bro repository branch does not point to the current submodule commit anymore. > bro segfaults after compile in MacOS X 10.11 El Capitan > ------------------------------------------------------- > > Key: BIT-1537 > URL: https://bro-tracker.atlassian.net/browse/BIT-1537 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Carlos Terr?n > Fix For: 2.5 > > > After compile with > {code} > ./configure --prefix=/usr/local > make > make install > {code} > And try to execute bro with: > {code} > bro -i en4 local > {code} > bro segfaults with > {code} > Program received signal SIGSEGV, Segmentation fault. > 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > 175 char *exponent = BN_bn2dec(pkey->pkey.rsa->e); > (gdb) bt > #0 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > #1 0x0000000100303e5d in file_analysis::X509::EndOfFile (this=0x105f8b710) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:56 > #2 0x000000010033f57a in file_analysis::File::EndOfFile (this=0x100961090) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/File.cc:522 > #3 0x000000010033bc6e in file_analysis::Manager::RemoveFile ( > this=0x105f8b710, file_id=...) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/Manager.cc:395 > #4 0x00000001002d910a in binpac::TLSHandshake::Handshake_Conn::proc_certificate (this=0x105f8a220, is_orig=false, certificates=0x100961f90) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:180 > #5 0x00000001002d99d4 in binpac::TLSHandshake::Handshake_Conn::proc_v3_certificate (this=0x105f8b710, is_orig=16, cl=) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:323 > #6 0x00000001002dc430 in binpac::TLSHandshake::Certificate::Parse ( > this=0x105f8a220, t_begin_of_data=, > t_end_of_data=0x101022f2e "", t_context=0x10095e480) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:1977 > {code} -- This message was sent by Atlassian JIRA (v7.2.0-OD-02-009#72000) From vallentin at icir.org Tue Feb 23 19:07:06 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Tue, 23 Feb 2016 19:07:06 -0800 Subject: [Bro-Dev] Broker updates? In-Reply-To: <3A669448-2C68-4CBF-8D90-F865BD517BE6@icir.org> References: <3A669448-2C68-4CBF-8D90-F865BD517BE6@icir.org> Message-ID: <20160224030706.GC34610@shogun> > If there are updates to a Broker store, is there a way that I can get > evented notification that a key was modified? I'm not seeing anything > that provides that functionality yet, but I need it for something I'm > working on. That's currently not supported, but it's on the wish list [1]. Matthias [1] https://www.bro.org/development/projects/broker-extensions.html, Bullet 5 From noreply at bro.org Wed Feb 24 00:00:25 2016 From: noreply at bro.org (Merge Tracker) Date: Wed, 24 Feb 2016 00:00:25 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602240800.u1O80PoJ020768@bro-ids.icir.org> Open Merge Requests =================== From seth at icir.org Wed Feb 24 07:02:41 2016 From: seth at icir.org (Seth Hall) Date: Wed, 24 Feb 2016 10:02:41 -0500 Subject: [Bro-Dev] Broker updates? In-Reply-To: <20160224030706.GC34610@shogun> References: <3A669448-2C68-4CBF-8D90-F865BD517BE6@icir.org> <20160224030706.GC34610@shogun> Message-ID: <09E07E94-0C60-4285-9374-15C5CF7DF090@icir.org> > On Feb 23, 2016, at 10:07 PM, Matthias Vallentin wrote: > > That's currently not supported, but it's on the wish list [1]. Ah, I haven't read through the broker extensions page in much detail yet. Thanks! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From vallentin at icir.org Wed Feb 24 08:10:50 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Wed, 24 Feb 2016 08:10:50 -0800 Subject: [Bro-Dev] Broker raw throughput Message-ID: <20160224161050.GE42006@shogun> I'm taking a discussion about Broker performance public which we previously had on our internal mailing list. Below, I'm pasting in Dominik's answer to my email. Dominik, we've white-listed your email address on bro-dev, but free to subscribe there to get future responses. Apologies for the inconvenience this may have caused on your end. While I'm at it: there's small but important difference: gperf != gperftools The former is GNU profiler whereas the latter Google's instrumentation framework. Unfortunately the naming jungle out there is convoluted. Matthias > (Context for Dominik: we are measuring the maximum throughput CAF can > deliver over a TCP connection and find it performs and order of > magnitude below 0mq.) > > I've attempted to reproduce the same benchmark with pure CAF, without > Broker, and can confirm the same order-of-magnitude difference. In fact, > the rates I observe with CAF are in the same range as Justin's > measurements. We can get one level deeper by using brokers (I mean caf::broker actors) directly to get rid of serialization entirely. I don't think that would change the performance much, but it makes analyzing profiler output a bit easier by removing unrelated components and functions from the graph. I'm going to write a small test program for CAF in the next couple of days. > From looking at the plots, I suspect that the client is the bottleneck, > which spends a third of its cycles in the function _sendto via > caf::io::network::stream::handle_event. The multiplexer and BASP broker > share the rest of CPU time. > > Dominik, does this make sense to you? Have you performed similar > measurements in the past? So far we tested primarily in comparison to other messaging frameworks for specific test cases such as distributed computations with CAF vs. MPI/Scala/Erlang. However, I ran a small benchmark comparing raw throughput measured via iperf to a CAF setup a while ago and noticed that receive performance was ok, but send performance was lacking. This corresponds to your findings, but I don't remember it being factor 5-6 worse. Thanks for sending me the gperf graphs. I will come back to you after running a test series under Linux and digging through the code a bit. Dominik ----- Forwarded message from Matthias Vallentin ----- From: Matthias Vallentin Sender: Matthias Vallentin To: Dominik Charousset Cc: "Azoff, Justin S" , bro-blue Subject: Re: [Bro-Blue] Broker raw throughput Date: Tue, 23 Feb 2016 17:23:09 -0800 User-Agent: Mutt/1.5.24 (2015-08-30) X-Label: (Context for Dominik: we are measuring the maximum throughput CAF can deliver over a TCP connection and find it performs and order of magnitude below 0mq.) I've attempted to reproduce the same benchmark with pure CAF, without Broker, and can confirm the same order-of-magnitude difference. In fact, the rates I observe with CAF are in the same range as Justin's measurements. Then I tried to figure out two things: (i) whether the CAF profiler throughput setting makes a difference, and (ii) where the bottleneck is. Regarding (i), I tested the client with values 10, 100, 1K, and 10K, 100K, and 1M. Up to 10K, no difference in the rate. Starting at 100K occasionally, and always at 1M, the benchmark no longer terminated. I suspect that I simply overloaded CAF's I/O subsystem. I've tried to figure out where CAF spends its time (because it's clearly not I/O-bound), and produced two gperftools plots: http://www.icir.org/matthias/tmp/caf-client-freebsd.pdf http://www.icir.org/matthias/tmp/caf-server-freebsd.pdf http://www.icir.org/matthias/tmp/caf-client-mac.pdf http://www.icir.org/matthias/tmp/caf-server-mac.pdf (There's an issue with the C++ standard library on Macs, which boils down to a pthreads condition variable issue, so only the FreeBSD plots are valuable. Still, I'm showing the Mac plots for completeness. I have been running into this issue many times before.) >From looking at the plots, I suspect that the client is the bottleneck, which spends a third of its cycles in the function _sendto via caf::io::network::stream::handle_event. The multiplexer and BASP broker share the rest of CPU time. Dominik, does this make sense to you? Have you performed similar measurements in the past? Matthias On Wed, Feb 03, 2016 at 06:56:22PM +0000, Azoff, Justin S wrote: > Hi, > > I've tried writing some simple programs using broker to see what kind of raw message rate it can do. > > I have them in a fork of the broker repo: > > https://github.com/JustinAzoff/broker/blob/master/tests/test_listen.cc > > https://github.com/JustinAzoff/broker/blob/master/tests/test_send.cc > > Starting test_listen and then test_send gives this output: > > Received [150000] duration: 0.832057 rate: 60092/sec > Received [200000] duration: 0.981312 rate: 50952.2/sec > Received [250000] duration: 0.984688 rate: 50777.5/sec > Received [300000] duration: 0.888988 rate: 56243.7/sec > Received [350000] duration: 0.99591 rate: 50205.3/sec > Received [400000] duration: 0.824133 rate: 60669.8/sec > > I can't get it to go any faster, if I remove or decrease the second sleep in the sender, something breaks and nothing is received. If I start up 6+ senders, the listener seems to grind to a halt as well. > > I ran a quick test using pyzmq (basically the example on https://learning-0mq-with-pyzmq.readthedocs.org/en/latest/pyzmq/patterns/pubsub.html) with a counter added, and that does: > > 13200000 Received 'message 1771511' duration 1.3138718605 rate: 304443.69198 > 13600000 Received 'message 2172740' duration 1.26631116867 rate: 315878.126875 > 14000000 Received 'message 2575818' duration 1.24697780609 rate: 320775.556747 > 14400000 Received 'message 2979093' duration 1.32109212875 rate: 302779.792033 > 14800000 Received 'message 3423906' duration 1.69283390045 rate: 236290.164022 > > I know it's not apples-to-apples since broker is doing a bunch of serialization stuff, but I wouldn't expect the serialization code to add that much overhead. > > > import zmq > import time > > port = "5556" > > context = zmq.Context() > socket = context.socket(zmq.SUB) > > socket.bind("tcp://*:%s" % port) > > socket.setsockopt(zmq.SUBSCRIBE, "message") > s = time.time() > i = 0 > while True: > string = socket.recv() > i += 1 > if i % 400000 == 0: > e = time.time() > print i, "Received", repr(string), "duration", e-s, "rate:", (400000/(e-s)) > s = time.time() > > import zmq > > port = "5556" > context = zmq.Context() > socket = context.socket(zmq.PUB) > socket.connect("tcp://localhost:%s" % port) > > i = 0 > while True: > socket.send("message %d" % (i)) > i += 1 > > -- > - Justin Azoff > > > _______________________________________________ > bro-blue mailing list > bro-blue at bro.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro-blue #include #include #include using namespace caf; using namespace caf::io; using namespace std; int main(int argc, char** argv) { // Set scheduler. size_t throughput = -1; if (argc > 1) { stringstream ss; ss << argv[1]; ss >> throughput; } cerr << "using scheduler with throughput " << throughput << endl; set_scheduler(2, throughput); // Run test code. auto server = remote_actor("127.0.0.1", 6666); cerr << "connected to 127.0.0.1:6666, blasting out data" << endl; auto i = 0; scoped_actor self; self->monitor(server); for (auto i = 0; i < 10000000; ++i) self->send(server, i++); self->receive( [&](down_msg const& msg) { cerr << "server terminated" << endl; } ); self->await_all_other_actors_done(); } #include #include #include using namespace caf; using namespace caf::io; using namespace std; using namespace std::chrono; behavior server(event_based_actor* self, size_t n = 10) { auto counter = make_shared(); auto iterations = make_shared(n); self->send(self, *counter, high_resolution_clock::now()); return { [=](int i) { ++*counter; }, [=](int last, high_resolution_clock::time_point prev) { auto now = high_resolution_clock::now(); auto secs = duration_cast(now - prev); auto rate = (*counter - last) / static_cast(secs.count()); cout << rate << endl; if (rate > 0 && --*iterations == 0) // Count only when we have data. self->quit(); else self->delayed_send(self, seconds(1), *counter, now); } }; } int main(int argc, char** argv) { // Set scheduler. auto iterations = size_t{0}; if (argc > 1) { stringstream ss; ss << argv[1]; ss >> iterations; } cerr << "spawning server for " << iterations << " iterations" << endl; // Run test code. scoped_actor self; auto s = self->spawn(server, iterations); publish(s, 6666, "127.0.0.1"); self->await_all_other_actors_done(); } ----- End forwarded message ----- -------------- next part -------------- #include #include #include using namespace caf; using namespace caf::io; using namespace std; int main(int argc, char** argv) { // Set scheduler. size_t throughput = -1; if (argc > 1) { stringstream ss; ss << argv[1]; ss >> throughput; } cerr << "using scheduler with throughput " << throughput << endl; set_scheduler(2, throughput); // Run test code. auto server = remote_actor("127.0.0.1", 6666); cerr << "connected to 127.0.0.1:6666, blasting out data" << endl; auto i = 0; scoped_actor self; self->monitor(server); for (auto i = 0; i < 10000000; ++i) self->send(server, i++); self->receive( [&](down_msg const& msg) { cerr << "server terminated" << endl; } ); self->await_all_other_actors_done(); } -------------- next part -------------- #include #include #include using namespace caf; using namespace caf::io; using namespace std; using namespace std::chrono; behavior server(event_based_actor* self, size_t n = 10) { auto counter = make_shared(); auto iterations = make_shared(n); self->send(self, *counter, high_resolution_clock::now()); return { [=](int i) { ++*counter; }, [=](int last, high_resolution_clock::time_point prev) { auto now = high_resolution_clock::now(); auto secs = duration_cast(now - prev); auto rate = (*counter - last) / static_cast(secs.count()); cout << rate << endl; if (rate > 0 && --*iterations == 0) // Count only when we have data. self->quit(); else self->delayed_send(self, seconds(1), *counter, now); } }; } int main(int argc, char** argv) { // Set scheduler. auto iterations = size_t{0}; if (argc > 1) { stringstream ss; ss << argv[1]; ss >> iterations; } cerr << "spawning server for " << iterations << " iterations" << endl; // Run test code. scoped_actor self; auto s = self->spawn(server, iterations); publish(s, 6666, "127.0.0.1"); self->await_all_other_actors_done(); } From seth at icir.org Wed Feb 24 08:20:04 2016 From: seth at icir.org (Seth Hall) Date: Wed, 24 Feb 2016 11:20:04 -0500 Subject: [Bro-Dev] Broker raw throughput In-Reply-To: <20160224161050.GE42006@shogun> References: <20160224161050.GE42006@shogun> Message-ID: And more from the other mailing list, I had given some flame graphs of the client and server running.. -------------- next part -------------- A non-text attachment was scrubbed... Name: client.svg Type: image/svg+xml Size: 471194 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160224/eb978008/attachment-0002.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: server.svg Type: image/svg+xml Size: 405275 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160224/eb978008/attachment-0003.bin -------------- next part -------------- .Seth > On Feb 24, 2016, at 11:10 AM, Matthias Vallentin wrote: > > I'm taking a discussion about Broker performance public which we > previously had on our internal mailing list. Below, I'm pasting in > Dominik's answer to my email. Dominik, we've white-listed your email > address on bro-dev, but free to subscribe there to get future responses. > Apologies for the inconvenience this may have caused on your end. > > While I'm at it: there's small but important difference: > > gperf != gperftools > > The former is GNU profiler whereas the latter Google's instrumentation > framework. Unfortunately the naming jungle out there is convoluted. > > Matthias > >> (Context for Dominik: we are measuring the maximum throughput CAF can >> deliver over a TCP connection and find it performs and order of >> magnitude below 0mq.) >> >> I've attempted to reproduce the same benchmark with pure CAF, without >> Broker, and can confirm the same order-of-magnitude difference. In fact, >> the rates I observe with CAF are in the same range as Justin's >> measurements. > > We can get one level deeper by using brokers (I mean caf::broker actors) > directly to get rid of serialization entirely. I don't think that would > change the performance much, but it makes analyzing profiler output a > bit easier by removing unrelated components and functions from the > graph. I'm going to write a small test program for CAF in the next > couple of days. > >> From looking at the plots, I suspect that the client is the bottleneck, >> which spends a third of its cycles in the function _sendto via >> caf::io::network::stream::handle_event. The multiplexer and BASP broker >> share the rest of CPU time. >> >> Dominik, does this make sense to you? Have you performed similar >> measurements in the past? > > So far we tested primarily in comparison to other messaging frameworks > for specific test cases such as distributed computations with CAF vs. > MPI/Scala/Erlang. However, I ran a small benchmark comparing raw > throughput measured via iperf to a CAF setup a while ago and noticed > that receive performance was ok, but send performance was lacking. This > corresponds to your findings, but I don't remember it being factor 5-6 > worse. > > Thanks for sending me the gperf graphs. I will come back to you after > running a test series under Linux and digging through the code a bit. > > Dominik > > ----- Forwarded message from Matthias Vallentin ----- > > From: Matthias Vallentin > Sender: Matthias Vallentin > To: Dominik Charousset > Cc: "Azoff, Justin S" , bro-blue > Subject: Re: [Bro-Blue] Broker raw throughput > Date: Tue, 23 Feb 2016 17:23:09 -0800 > User-Agent: Mutt/1.5.24 (2015-08-30) > X-Label: > > (Context for Dominik: we are measuring the maximum throughput CAF can > deliver over a TCP connection and find it performs and order of > magnitude below 0mq.) > > I've attempted to reproduce the same benchmark with pure CAF, without > Broker, and can confirm the same order-of-magnitude difference. In fact, > the rates I observe with CAF are in the same range as Justin's > measurements. > > Then I tried to figure out two things: (i) whether the CAF profiler > throughput setting makes a difference, and (ii) where the bottleneck is. > Regarding (i), I tested the client with values 10, 100, 1K, and 10K, > 100K, and 1M. Up to 10K, no difference in the rate. Starting at 100K > occasionally, and always at 1M, the benchmark no longer terminated. I > suspect that I simply overloaded CAF's I/O subsystem. > > I've tried to figure out where CAF spends its time (because it's clearly > not I/O-bound), and produced two gperftools plots: > > http://www.icir.org/matthias/tmp/caf-client-freebsd.pdf > http://www.icir.org/matthias/tmp/caf-server-freebsd.pdf > http://www.icir.org/matthias/tmp/caf-client-mac.pdf > http://www.icir.org/matthias/tmp/caf-server-mac.pdf > > (There's an issue with the C++ standard library on Macs, which boils down > to a pthreads condition variable issue, so only the FreeBSD plots are > valuable. Still, I'm showing the Mac plots for completeness. I have been > running into this issue many times before.) > >> From looking at the plots, I suspect that the client is the bottleneck, > which spends a third of its cycles in the function _sendto via > caf::io::network::stream::handle_event. The multiplexer and BASP broker > share the rest of CPU time. > > Dominik, does this make sense to you? Have you performed similar > measurements in the past? > > Matthias > > On Wed, Feb 03, 2016 at 06:56:22PM +0000, Azoff, Justin S wrote: >> Hi, >> >> I've tried writing some simple programs using broker to see what kind of raw message rate it can do. >> >> I have them in a fork of the broker repo: >> >> https://github.com/JustinAzoff/broker/blob/master/tests/test_listen.cc >> >> https://github.com/JustinAzoff/broker/blob/master/tests/test_send.cc >> >> Starting test_listen and then test_send gives this output: >> >> Received [150000] duration: 0.832057 rate: 60092/sec >> Received [200000] duration: 0.981312 rate: 50952.2/sec >> Received [250000] duration: 0.984688 rate: 50777.5/sec >> Received [300000] duration: 0.888988 rate: 56243.7/sec >> Received [350000] duration: 0.99591 rate: 50205.3/sec >> Received [400000] duration: 0.824133 rate: 60669.8/sec >> >> I can't get it to go any faster, if I remove or decrease the second sleep in the sender, something breaks and nothing is received. If I start up 6+ senders, the listener seems to grind to a halt as well. >> >> I ran a quick test using pyzmq (basically the example on https://learning-0mq-with-pyzmq.readthedocs.org/en/latest/pyzmq/patterns/pubsub.html) with a counter added, and that does: >> >> 13200000 Received 'message 1771511' duration 1.3138718605 rate: 304443.69198 >> 13600000 Received 'message 2172740' duration 1.26631116867 rate: 315878.126875 >> 14000000 Received 'message 2575818' duration 1.24697780609 rate: 320775.556747 >> 14400000 Received 'message 2979093' duration 1.32109212875 rate: 302779.792033 >> 14800000 Received 'message 3423906' duration 1.69283390045 rate: 236290.164022 >> >> I know it's not apples-to-apples since broker is doing a bunch of serialization stuff, but I wouldn't expect the serialization code to add that much overhead. >> >> >> import zmq >> import time >> >> port = "5556" >> >> context = zmq.Context() >> socket = context.socket(zmq.SUB) >> >> socket.bind("tcp://*:%s" % port) >> >> socket.setsockopt(zmq.SUBSCRIBE, "message") >> s = time.time() >> i = 0 >> while True: >> string = socket.recv() >> i += 1 >> if i % 400000 == 0: >> e = time.time() >> print i, "Received", repr(string), "duration", e-s, "rate:", (400000/(e-s)) >> s = time.time() >> >> import zmq >> >> port = "5556" >> context = zmq.Context() >> socket = context.socket(zmq.PUB) >> socket.connect("tcp://localhost:%s" % port) >> >> i = 0 >> while True: >> socket.send("message %d" % (i)) >> i += 1 >> >> -- >> - Justin Azoff >> >> >> _______________________________________________ >> bro-blue mailing list >> bro-blue at bro.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro-blue > > #include > > #include > #include > > using namespace caf; > using namespace caf::io; > using namespace std; > > int main(int argc, char** argv) { > // Set scheduler. > size_t throughput = -1; > if (argc > 1) { > stringstream ss; > ss << argv[1]; > ss >> throughput; > } > cerr << "using scheduler with throughput " << throughput << endl; > set_scheduler(2, throughput); > // Run test code. > auto server = remote_actor("127.0.0.1", 6666); > cerr << "connected to 127.0.0.1:6666, blasting out data" << endl; > auto i = 0; > scoped_actor self; > self->monitor(server); > for (auto i = 0; i < 10000000; ++i) > self->send(server, i++); > self->receive( > [&](down_msg const& msg) { > cerr << "server terminated" << endl; > } > ); > self->await_all_other_actors_done(); > } > > #include > > #include > #include > > using namespace caf; > using namespace caf::io; > using namespace std; > using namespace std::chrono; > > behavior server(event_based_actor* self, size_t n = 10) { > auto counter = make_shared(); > auto iterations = make_shared(n); > self->send(self, *counter, high_resolution_clock::now()); > return { > [=](int i) { > ++*counter; > }, > [=](int last, high_resolution_clock::time_point prev) { > auto now = high_resolution_clock::now(); > auto secs = duration_cast(now - prev); > auto rate = (*counter - last) / static_cast(secs.count()); > cout << rate << endl; > if (rate > 0 && --*iterations == 0) // Count only when we have data. > self->quit(); > else > self->delayed_send(self, seconds(1), *counter, now); > } > }; > } > > int main(int argc, char** argv) { > // Set scheduler. > auto iterations = size_t{0}; > if (argc > 1) { > stringstream ss; > ss << argv[1]; > ss >> iterations; > } > cerr << "spawning server for " << iterations << " iterations" << endl; > // Run test code. > scoped_actor self; > auto s = self->spawn(server, iterations); > publish(s, 6666, "127.0.0.1"); > self->await_all_other_actors_done(); > } > > > ----- End forwarded message ----- > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jira at bro-tracker.atlassian.net Wed Feb 24 12:30:00 2016 From: jira at bro-tracker.atlassian.net (Lu Goon (JIRA)) Date: Wed, 24 Feb 2016 14:30:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1539) Adding intel to intel framework Bro is not loading the file In-Reply-To: References: Message-ID: Lu Goon created BIT-1539: ---------------------------- Summary: Adding intel to intel framework Bro is not loading the file Key: BIT-1539 URL: https://bro-tracker.atlassian.net/browse/BIT-1539 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.4 Environment: CentOS 7.2. 1511 kernel version 3.10 Reporter: Lu Goon We wanted to get our intel ( bad IPs) in to bro for alerting using the intel framework. I crafted a file of BAD IPs based on the documentation on the site. Also based this on the critical stack implementation as well. I provided the following fields: indicator, indicator_type, meta.source, meta.desc, meta.do_notice. thus a sample entry would be 1.2.3.4 \t Intel::ADDR \t MY INTEL \t My bad IP list \t F Per the documentation it should write all that into the intel.log file if activated in the local.bro file either using broctl or bro -i ens33 local.bro. There is no indication in loaded scripts that the files loads. Also in my local.bro file I include. @load policy/frameworks/intel/seen @load policy/frameworks/intel/do_notice redef Intel::read_files += { "/usr/local/bro/upload/intel.dat"}; Any help on debugging why this file is not loading or indication of if it is loaded? -- This message was sent by Atlassian JIRA (v7.2.0-OD-02-009#72000) From noreply at bro.org Thu Feb 25 00:00:23 2016 From: noreply at bro.org (Merge Tracker) Date: Thu, 25 Feb 2016 00:00:23 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602250800.u1P80NAP013705@bro-ids.icir.org> Open Merge Requests =================== From vallentin at icir.org Thu Feb 25 08:19:47 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Thu, 25 Feb 2016 08:19:47 -0800 Subject: [Bro-Dev] Broker raw throughput In-Reply-To: <20160224161050.GE42006@shogun> References: <20160224161050.GE42006@shogun> Message-ID: <20160225161947.GI42006@shogun> For better reproducibility, here's the Makefile that I used to drive the experiments: CC = cc CXX = c++ FLAGS = -O3 -g -std=c++11 -stdlib=libc++ LIBS = -lcaf_core -lcaf_io -ltcmalloc -lprofiler caf-client: caf-client.cpp $(CXX) $(FLAGS) $< -o $@ $(LIBS) caf-server: caf-server.cpp $(CXX) $(FLAGS) $< -o $@ $(LIBS) bench-caf-client: CPUPROFILE=caf-client.prof ./caf-client 1000 bench-caf-server: CPUPROFILE=caf-server.prof ./caf-server 10 bench-caf-pprof: caf-client.prof caf-server.prof pprof --pdf caf-client caf-client.prof > caf-client.pdf pprof --pdf caf-server caf-server.prof > caf-server.pdf On my FreeBSD box, I had to add /usr/local/include to -I and -L, because I installed CAF and gperftools via ports. Since it's a headless machine without ps2pdf, we need extra level of indirection: (1) pprof --raw caf-client caf-client.prof > caf-client.raw (2) copy raw profile to desktop (3) pprof --pdf caf-client.raw > caf-client.pdf Hope this helps, Matthias From noreply at bro.org Fri Feb 26 00:00:19 2016 From: noreply at bro.org (Merge Tracker) Date: Fri, 26 Feb 2016 00:00:19 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602260800.u1Q80JWD017044@bro-ids.icir.org> Open Merge Requests =================== From jira at bro-tracker.atlassian.net Fri Feb 26 09:08:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 26 Feb 2016 11:08:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1540) Ifconfig is hardcoded in BroControl In-Reply-To: References: Message-ID: Johanna Amann created BIT-1540: ---------------------------------- Summary: Ifconfig is hardcoded in BroControl Key: BIT-1540 URL: https://bro-tracker.atlassian.net/browse/BIT-1540 Project: Bro Issue Tracker Issue Type: Problem Components: BroControl Affects Versions: git/master Reporter: Johanna Amann Fix For: 2.5 >From the mailing list: {quote} Hi Folks, On later versions of Linux distros iproute2 replaces ifconfig with ip Starting at line 601 at https://github.com/bro/broctl/blob/master/BroControl/config.py It looks like ifconfig is hard-written into the logic. Probably needs a patch to check for the ip command. Cheers, Harry {quote} We should probably check for the presence of the ip utility and use that, if present. -- This message was sent by Atlassian JIRA (v7.2.0-OD-02-009#72000) From jira at bro-tracker.atlassian.net Fri Feb 26 12:26:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 26 Feb 2016 14:26:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1539) Adding intel to intel framework Bro is not loading the file In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24400#comment-24400 ] Johanna Amann commented on BIT-1539: ------------------------------------ Generally - intelligence files (or any other external data files) that are loaded with the Bro input framework do not appear in loaded_scripts.bro. Unless there is an error in reporter.log, you can assume that a file has been loaded correctly. If you want to check that a file was completely read, you can catch the end_of_data event of the Input framework and check the name of the source that was completely read (intel sources have a name starting with "intel-"). > Adding intel to intel framework Bro is not loading the file > ----------------------------------------------------------- > > Key: BIT-1539 > URL: https://bro-tracker.atlassian.net/browse/BIT-1539 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Environment: CentOS 7.2. 1511 kernel version 3.10 > Reporter: Lu Goon > Labels: Framework, IP, Intel, addresses, data, files, text > > We wanted to get our intel ( bad IPs) in to bro for alerting using the intel framework. I crafted a file of BAD IPs based on the documentation on the site. Also based this on the critical stack implementation as well. > I provided the following fields: indicator, indicator_type, meta.source, meta.desc, meta.do_notice. > thus a sample entry would be > 1.2.3.4 \t Intel::ADDR \t MY INTEL \t My bad IP list \t F > Per the documentation it should write all that into the intel.log file if activated in the local.bro file > either using broctl or bro -i ens33 local.bro. There is no indication in loaded scripts that the files loads. > Also in my local.bro file I include. > @load policy/frameworks/intel/seen > @load policy/frameworks/intel/do_notice > redef Intel::read_files += { "/usr/local/bro/upload/intel.dat"}; > Any help on debugging why this file is not loading or indication of if it is loaded? -- This message was sent by Atlassian JIRA (v7.2.0-OD-02-009#72000) From jira at bro-tracker.atlassian.net Fri Feb 26 12:28:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 26 Feb 2016 14:28:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1529) Base SIP scripts missing SUBSCRIBE and NOTIFY In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1529: ------------------------------- Fix Version/s: 2.5 > Base SIP scripts missing SUBSCRIBE and NOTIFY > --------------------------------------------- > > Key: BIT-1529 > URL: https://bro-tracker.atlassian.net/browse/BIT-1529 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Seth Hall > Fix For: 2.5 > > > The base/protocols/sip/main.bro script has a set in `sip_methods` which needs to have SUBSCRIBE and NOTIFY added. They're defined in RFC 3265. -- This message was sent by Atlassian JIRA (v7.2.0-OD-02-009#72000) From jira at bro-tracker.atlassian.net Fri Feb 26 12:28:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 26 Feb 2016 14:28:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1530) protocol_confirmation event cannot be hooked by plugin In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1530: ------------------------------- Fix Version/s: 2.5 > protocol_confirmation event cannot be hooked by plugin > ------------------------------------------------------ > > Key: BIT-1530 > URL: https://bro-tracker.atlassian.net/browse/BIT-1530 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Jeff Barber > Fix For: 2.5 > > > The way the 'protocol_confirmation' event is raised bypasses the plugin event-hook mechanism. Plugin event hooks are triggered via EventMgr.QueueEvent which is in the usual event generation interface. However, protocol_confirmation is generated via this code in src/analyzer/Analyzer.cc: > {{ > // We immediately raise the event so that the analyzer can quickly > // react if necessary. > ::Event* e = new ::Event(protocol_confirmation, vl, SOURCE_LOCAL); > mgr.Dispatch(e); > }} > The EventMgr.Dispatch method doesn't invoke the hook so at present it's not possible for a plugin to hook this event. It would be nice if this were orthogonal with other events. -- This message was sent by Atlassian JIRA (v7.2.0-OD-02-009#72000) From noreply at bro.org Sat Feb 27 00:00:21 2016 From: noreply at bro.org (Merge Tracker) Date: Sat, 27 Feb 2016 00:00:21 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602270800.u1R80Lho014499@bro-ids.icir.org> Open Merge Requests =================== From noreply at bro.org Sun Feb 28 00:00:20 2016 From: noreply at bro.org (Merge Tracker) Date: Sun, 28 Feb 2016 00:00:20 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602280800.u1S80KL2032533@bro-ids.icir.org> Open Merge Requests =================== From noreply at bro.org Mon Feb 29 00:00:19 2016 From: noreply at bro.org (Merge Tracker) Date: Mon, 29 Feb 2016 00:00:19 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201602290800.u1T80JVG011496@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------- BIT-1537 [1] Bro Carlos Terr??n - 2016-02-23 2.5 Normal bro segfaults after compile in MacOS X 10.11 El Capitan BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ------------ ---------- ------------------------------------- #55 [3] bro wglodek [4] 2016-02-07 http-evasion [5] #52 [6] bro J-Gras [7] 2016-01-18 Fixed matching mail address intel [8] #18 [9] bro-plugins jshlbrd [10] 2016-02-12 SSDP analyzer [11] [1] BIT-1537 https://bro-tracker.atlassian.net/browse/BIT-1537 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] Pull Request #55 https://github.com/bro/bro/pull/55 [4] wglodek https://github.com/wglodek [5] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [6] Pull Request #52 https://github.com/bro/bro/pull/52 [7] J-Gras https://github.com/J-Gras [8] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [9] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [10] jshlbrd https://github.com/jshlbrd [11] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Mon Feb 29 00:50:00 2016 From: jira at bro-tracker.atlassian.net (=?UTF-8?Q?Carlos_Terr=C3=B3n_=28JIRA=29?=) Date: Mon, 29 Feb 2016 02:50:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1537) bro segfaults after compile in MacOS X 10.11 El Capitan In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24500#comment-24500 ] Carlos Terr?n commented on BIT-1537: ------------------------------------ Hi Johanna Sorry for the delay {code} baldurgate:bin terron$otool -L bro bro: /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8) /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8) /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5) /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1) {code} (the binary that segfaults). I'm going to try compile with an openssl library and check that the include headers are not mixed with system ones. > bro segfaults after compile in MacOS X 10.11 El Capitan > ------------------------------------------------------- > > Key: BIT-1537 > URL: https://bro-tracker.atlassian.net/browse/BIT-1537 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Carlos Terr?n > Fix For: 2.5 > > > After compile with > {code} > ./configure --prefix=/usr/local > make > make install > {code} > And try to execute bro with: > {code} > bro -i en4 local > {code} > bro segfaults with > {code} > Program received signal SIGSEGV, Segmentation fault. > 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > 175 char *exponent = BN_bn2dec(pkey->pkey.rsa->e); > (gdb) bt > #0 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > #1 0x0000000100303e5d in file_analysis::X509::EndOfFile (this=0x105f8b710) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:56 > #2 0x000000010033f57a in file_analysis::File::EndOfFile (this=0x100961090) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/File.cc:522 > #3 0x000000010033bc6e in file_analysis::Manager::RemoveFile ( > this=0x105f8b710, file_id=...) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/Manager.cc:395 > #4 0x00000001002d910a in binpac::TLSHandshake::Handshake_Conn::proc_certificate (this=0x105f8a220, is_orig=false, certificates=0x100961f90) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:180 > #5 0x00000001002d99d4 in binpac::TLSHandshake::Handshake_Conn::proc_v3_certificate (this=0x105f8b710, is_orig=16, cl=) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:323 > #6 0x00000001002dc430 in binpac::TLSHandshake::Certificate::Parse ( > this=0x105f8a220, t_begin_of_data=, > t_end_of_data=0x101022f2e "", t_context=0x10095e480) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:1977 > {code} -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-005#72000)