From noreply at bro.org Tue Mar 1 00:00:34 2016 From: noreply at bro.org (Merge Tracker) Date: Tue, 1 Mar 2016 00:00:34 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603010800.u2180YKG026184@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------- BIT-1537 [1] Bro Carlos Terr??n - 2016-02-29 2.5 Normal bro segfaults after compile in MacOS X 10.11 El Capitan BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ------------ ---------- ------------------------------------- #55 [3] bro wglodek [4] 2016-02-07 http-evasion [5] #52 [6] bro J-Gras [7] 2016-01-18 Fixed matching mail address intel [8] #18 [9] bro-plugins jshlbrd [10] 2016-02-12 SSDP analyzer [11] [1] BIT-1537 https://bro-tracker.atlassian.net/browse/BIT-1537 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] Pull Request #55 https://github.com/bro/bro/pull/55 [4] wglodek https://github.com/wglodek [5] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [6] Pull Request #52 https://github.com/bro/bro/pull/52 [7] J-Gras https://github.com/J-Gras [8] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [9] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [10] jshlbrd https://github.com/jshlbrd [11] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From dominik.charousset at haw-hamburg.de Tue Mar 1 07:05:46 2016 From: dominik.charousset at haw-hamburg.de (Dominik Charousset) Date: Tue, 1 Mar 2016 16:05:46 +0100 Subject: [Bro-Dev] Broker raw throughput In-Reply-To: <20160225161947.GI42006@shogun> References: <20160224161050.GE42006@shogun> <20160225161947.GI42006@shogun> Message-ID: Thanks for providing build scripts and sharing results. Just a quick heads-up from me: I have implemented a simple sender/receiver pair using C sockets as well as CAF brokers (attached, but works only with the current actor-system topic branch). Both sending and receiving are slower with CAF (as expected), although the performance is slightly better when using the ASIO backend [1]. I'm still investigating and hopefully come back to you guys later this week. Dominik [1] e.g. ./caf_impl --caf#middleman.network-backend=asio -s > On Feb 25, 2016, at 17:19, Matthias Vallentin wrote: > > For better reproducibility, here's the Makefile that I used to drive the > experiments: > > CC = cc > CXX = c++ > FLAGS = -O3 -g -std=c++11 -stdlib=libc++ > LIBS = -lcaf_core -lcaf_io -ltcmalloc -lprofiler > > caf-client: caf-client.cpp > $(CXX) $(FLAGS) $< -o $@ $(LIBS) > > caf-server: caf-server.cpp > $(CXX) $(FLAGS) $< -o $@ $(LIBS) > > bench-caf-client: > CPUPROFILE=caf-client.prof ./caf-client 1000 > > bench-caf-server: > CPUPROFILE=caf-server.prof ./caf-server 10 > > bench-caf-pprof: caf-client.prof caf-server.prof > pprof --pdf caf-client caf-client.prof > caf-client.pdf > pprof --pdf caf-server caf-server.prof > caf-server.pdf > > On my FreeBSD box, I had to add /usr/local/include to -I and -L, because > I installed CAF and gperftools via ports. Since it's a headless machine > without ps2pdf, we need extra level of indirection: > > (1) pprof --raw caf-client caf-client.prof > caf-client.raw > (2) copy raw profile to desktop > (3) pprof --pdf caf-client.raw > caf-client.pdf > > Hope this helps, > > Matthias -------------- next part -------------- A non-text attachment was scrubbed... Name: caf_impl.cpp Type: application/octet-stream Size: 3055 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160301/34b30017/attachment.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: Makefile Type: application/octet-stream Size: 384 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160301/34b30017/attachment-0001.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: native_impl.cpp Type: application/octet-stream Size: 3286 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160301/34b30017/attachment-0002.obj From jira at bro-tracker.atlassian.net Tue Mar 1 10:22:00 2016 From: jira at bro-tracker.atlassian.net (Aaron Eppert (JIRA)) Date: Tue, 1 Mar 2016 12:22:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1541) Crash in SocketComm::Run - RemoteSerializer.cc:3493 In-Reply-To: References: Message-ID: Aaron Eppert created BIT-1541: --------------------------------- Summary: Crash in SocketComm::Run - RemoteSerializer.cc:3493 Key: BIT-1541 URL: https://bro-tracker.atlassian.net/browse/BIT-1541 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.4 Reporter: Aaron Eppert This has been happening on a few sensors, both standalone and not. On each sensor, there is broctl cron running as well as periodic polling being the Python interface to the netstats data. {quote}#0 0x0000000000607d47 in SocketComm::Run (this=0x1) at /mnt/hgfs/src/psdev/bro/src/RemoteSerializer.cc:3493 #1 0x0000000000608021 in RemoteSerializer::Fork (this=0x2590000) at /mnt/hgfs/src/psdev/bro/src/RemoteSerializer.cc:687 #2 0x000000000060813f in RemoteSerializer::Enable (this=0x2590000) at /mnt/hgfs/src/psdev/bro/src/RemoteSerializer.cc:575 #3 0x00000000005d52b3 in BifFunc::bro_enable_communication (frame=, BiF_ARGS=) at bro.bif:4480 #4 0x00000000005d2cdd in BuiltinFunc::Call (this=0x2ae1180, args=0x16255be0, parent=0x4ada990) at /mnt/hgfs/src/psdev/bro/src/Func.cc:586 #5 0x00000000005b7af6 in CallExpr::Eval (this=0x315e900, f=0x4ada990) at /mnt/hgfs/src/psdev/bro/src/Expr.cc:4544 #6 0x000000000062b8d4 in ExprStmt::Exec (this=0x315e8b0, f=0x4ada990, flow=@0x7ffe64462c50: FLOW_NEXT) at /mnt/hgfs/src/psdev/bro/src/Stmt.cc:352 #7 0x0000000000629b94 in IfStmt::DoExec (this=0x31533c0, f=0x4ada990, v=, flow=@0x7ffe64462c50: FLOW_NEXT) at /mnt/hgfs/src/psdev/bro/src/Stmt.cc:456 #8 0x000000000062b8f1 in ExprStmt::Exec (this=0x31533c0, f=0x4ada990, flow=@0x7ffe64462c50: FLOW_NEXT) at /mnt/hgfs/src/psdev/bro/src/Stmt.cc:356 #9 0x0000000000629c31 in StmtList::Exec (this=0x31534e0, f=0x4ada990, flow=@0x7ffe64462c50: FLOW_NEXT) at /mnt/hgfs/src/psdev/bro/src/Stmt.cc:1696 #10 0x0000000000629c31 in StmtList::Exec (this=0x3153120, f=0x4ada990, flow=@0x7ffe64462c50: FLOW_NEXT) at /mnt/hgfs/src/psdev/bro/src/Stmt.cc:1696 #11 0x00000000005decfe in BroFunc::Call (this=0x2743b80, args=, parent=0x0) at /mnt/hgfs/src/psdev/bro/src/Func.cc:403 #12 0x000000000059d95a in EventHandler::Call (this=0x2476c80, vl=0x15db0b60, no_remote=no_remote at entry=false) at /mnt/hgfs/src/psdev/bro/src/EventHandler.cc:130 #13 0x000000000059cb65 in Dispatch (no_remote=false, this=0x16193120) at /mnt/hgfs/src/psdev/bro/src/Event.h:50 #14 EventMgr::Dispatch (this=this at entry=0xc07840 ) at /mnt/hgfs/src/psdev/bro/src/Event.cc:111 #15 0x000000000059cd00 in EventMgr::Drain (this=0xc07840 ) at /mnt/hgfs/src/psdev/bro/src/Event.cc:128 #16 0x000000000054c659 in main (argc=, argv=) at /mnt/hgfs/src/psdev/bro/src/main.cc:1147 {quote} -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Tue Mar 1 11:26:00 2016 From: jira at bro-tracker.atlassian.net (Jon Schipp (JIRA)) Date: Tue, 1 Mar 2016 13:26:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1540) Ifconfig is hardcoded in BroControl In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Schipp reassigned BIT-1540: ------------------------------- Assignee: Jon Schipp > Ifconfig is hardcoded in BroControl > ----------------------------------- > > Key: BIT-1540 > URL: https://bro-tracker.atlassian.net/browse/BIT-1540 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Jon Schipp > Fix For: 2.5 > > > From the mailing list: > {quote} > Hi Folks, > On later versions of Linux distros iproute2 replaces ifconfig with ip > Starting at line 601 at > https://github.com/bro/broctl/blob/master/BroControl/config.py > It looks like ifconfig is hard-written into the logic. Probably needs a > patch to check for the ip command. > Cheers, > Harry > {quote} > We should probably check for the presence of the ip utility and use that, if present. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Tue Mar 1 14:01:01 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Tue, 1 Mar 2016 16:01:01 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1529) Base SIP scripts missing SUBSCRIBE and NOTIFY In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1529: ------------------------------- Status: Merge Request (was: Open) > Base SIP scripts missing SUBSCRIBE and NOTIFY > --------------------------------------------- > > Key: BIT-1529 > URL: https://bro-tracker.atlassian.net/browse/BIT-1529 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Seth Hall > Fix For: 2.5 > > > The base/protocols/sip/main.bro script has a set in `sip_methods` which needs to have SUBSCRIBE and NOTIFY added. They're defined in RFC 3265. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Tue Mar 1 14:01:01 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Tue, 1 Mar 2016 16:01:01 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1529) Base SIP scripts missing SUBSCRIBE and NOTIFY In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24600#comment-24600 ] Johanna Amann commented on BIT-1529: ------------------------------------ I added subscribe in topic/johanna/bit-1529. Notify was already added in 4a56a17817fc4eed2a3a6c10ecb5140df4f2dfc5 No tests since I don't have traffic (but since this is only used to generate weirds, it should be ok). > Base SIP scripts missing SUBSCRIBE and NOTIFY > --------------------------------------------- > > Key: BIT-1529 > URL: https://bro-tracker.atlassian.net/browse/BIT-1529 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Seth Hall > Fix For: 2.5 > > > The base/protocols/sip/main.bro script has a set in `sip_methods` which needs to have SUBSCRIBE and NOTIFY added. They're defined in RFC 3265. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Tue Mar 1 14:04:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Tue, 1 Mar 2016 16:04:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1539) Adding intel to intel framework Bro is not loading the file In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1539: ------------------------------- Resolution: Solved Status: Closed (was: Open) Since there was no further comment on this, I assumed that solved your problem. Feel free to re-open if you still think there is anything wrong in Bro. > Adding intel to intel framework Bro is not loading the file > ----------------------------------------------------------- > > Key: BIT-1539 > URL: https://bro-tracker.atlassian.net/browse/BIT-1539 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Environment: CentOS 7.2. 1511 kernel version 3.10 > Reporter: Lu Goon > Labels: Framework, IP, Intel, addresses, data, files, text > > We wanted to get our intel ( bad IPs) in to bro for alerting using the intel framework. I crafted a file of BAD IPs based on the documentation on the site. Also based this on the critical stack implementation as well. > I provided the following fields: indicator, indicator_type, meta.source, meta.desc, meta.do_notice. > thus a sample entry would be > 1.2.3.4 \t Intel::ADDR \t MY INTEL \t My bad IP list \t F > Per the documentation it should write all that into the intel.log file if activated in the local.bro file > either using broctl or bro -i ens33 local.bro. There is no indication in loaded scripts that the files loads. > Also in my local.bro file I include. > @load policy/frameworks/intel/seen > @load policy/frameworks/intel/do_notice > redef Intel::read_files += { "/usr/local/bro/upload/intel.dat"}; > Any help on debugging why this file is not loading or indication of if it is loaded? -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Tue Mar 1 14:11:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Tue, 1 Mar 2016 16:11:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1535) conn.log conn_state field or documentation is wrong In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24602#comment-24602 ] Johanna Amann commented on BIT-1535: ------------------------------------ topic/johanna/bit-1535 updates the documentation of RSTR to "Responder sent a RST" > conn.log conn_state field or documentation is wrong > --------------------------------------------------- > > Key: BIT-1535 > URL: https://bro-tracker.atlassian.net/browse/BIT-1535 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Justin Azoff > Assignee: Vern Paxson > > There is an issue where the conn.log conn_state field will contain RSTR, which according to the documentation means "Established, responder aborted." > The problem that I notice is that I see log entries where conn_state is RSTR, but conn_history does not contain an 'h'. Additionally, the resp_h is absolutely not running a service on resp_p and the orig_h is usually in the process of a tcp scan. > Here are the top frequencies of RSTR without an h over about a weeks worth of conn logs: > {code} > 38193 RSTR Fr > 3662 RSTR DFr > 1801 RSTR DFdrR > 1248 RSTR DRr > 432 RSTR DrF > 232 RSTR Far > 128 RSTR DdAFrR > 79 RSTR DFadrR > 64 RSTR DrR > 58 RSTR DdAFarR > {code} > Compared to histories that did contain an h: > {code} > 425398 RSTR ShADadFr > 204149 RSTR ShADadFrR > 156303 RSTR ShADdFar > 141795 RSTR ShADadFRRr > 105704 RSTR ShADadfr > 79697 RSTR ShADadr > 63493 RSTR ShADaFr > 51704 RSTR ShADadFrrrr > 42075 RSTR ShADdar > 37678 RSTR ShADadfRr > {code} > I don't have a pcap for this, but I believe many of the weird connections are related to scans or backscatter. > I'm not sure if the code is wrong or the documentation is wrong, but I don't see how a fin+reset connection could be classified as established. > Also, One thing that would be a nice documentation addition is the answer to this question: > Given a conn.log entry, how do determine if there was a connection established? I thought it would be if the state was in 'SF S1 S2 S3 RSTO RSTR', but RSTR is problematic... -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Tue Mar 1 14:11:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Tue, 1 Mar 2016 16:11:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1535) conn.log conn_state field or documentation is wrong In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1535: ------------------------------- Status: Merge Request (was: Open) Assignee: (was: Vern Paxson) > conn.log conn_state field or documentation is wrong > --------------------------------------------------- > > Key: BIT-1535 > URL: https://bro-tracker.atlassian.net/browse/BIT-1535 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Justin Azoff > > There is an issue where the conn.log conn_state field will contain RSTR, which according to the documentation means "Established, responder aborted." > The problem that I notice is that I see log entries where conn_state is RSTR, but conn_history does not contain an 'h'. Additionally, the resp_h is absolutely not running a service on resp_p and the orig_h is usually in the process of a tcp scan. > Here are the top frequencies of RSTR without an h over about a weeks worth of conn logs: > {code} > 38193 RSTR Fr > 3662 RSTR DFr > 1801 RSTR DFdrR > 1248 RSTR DRr > 432 RSTR DrF > 232 RSTR Far > 128 RSTR DdAFrR > 79 RSTR DFadrR > 64 RSTR DrR > 58 RSTR DdAFarR > {code} > Compared to histories that did contain an h: > {code} > 425398 RSTR ShADadFr > 204149 RSTR ShADadFrR > 156303 RSTR ShADdFar > 141795 RSTR ShADadFRRr > 105704 RSTR ShADadfr > 79697 RSTR ShADadr > 63493 RSTR ShADaFr > 51704 RSTR ShADadFrrrr > 42075 RSTR ShADdar > 37678 RSTR ShADadfRr > {code} > I don't have a pcap for this, but I believe many of the weird connections are related to scans or backscatter. > I'm not sure if the code is wrong or the documentation is wrong, but I don't see how a fin+reset connection could be classified as established. > Also, One thing that would be a nice documentation addition is the answer to this question: > Given a conn.log entry, how do determine if there was a connection established? I thought it would be if the state was in 'SF S1 S2 S3 RSTO RSTR', but RSTR is problematic... -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Tue Mar 1 14:13:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Tue, 1 Mar 2016 16:13:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1518) SSH analyzer doesn't handle non-conformant client version strings In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1518: ------------------------------- Fix Version/s: 2.5 > SSH analyzer doesn't handle non-conformant client version strings > ----------------------------------------------------------------- > > Key: BIT-1518 > URL: https://bro-tracker.atlassian.net/browse/BIT-1518 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Vlad Grigorescu > Assignee: Vlad Grigorescu > Fix For: 2.5 > > > Received a report that some SSH clients send a version identification string similar to 'SSH-2.0-FooBar_Client\n' which causes a protocol violation in the SSH analyzer. RFC 4253 states that this must be terminated by '\r\n', but that's not what's being observed. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Tue Mar 1 15:28:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Tue, 1 Mar 2016 17:28:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1542) Please merge topic/johanna/freebsd9 In-Reply-To: References: Message-ID: Johanna Amann created BIT-1542: ---------------------------------- Summary: Please merge topic/johanna/freebsd9 Key: BIT-1542 URL: https://bro-tracker.atlassian.net/browse/BIT-1542 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: git/master Reporter: Johanna Amann Fix For: 2.5 Please merke topic/johanna/freebsd9 in bro and cmake. It adds a bit of text to the installation instructions on how to install Bro on FreeBSD 9.X. It also adds tests to cmake that check if C++11 header files are usable; this prevents issues where a new compiler uses the includes of an older one, which apparently can easily happen on old versions of FreeBSD. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Tue Mar 1 15:28:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Tue, 1 Mar 2016 17:28:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1542) Please merge topic/johanna/freebsd9 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1542: ------------------------------- Status: Merge Request (was: Open) > Please merge topic/johanna/freebsd9 > ----------------------------------- > > Key: BIT-1542 > URL: https://bro-tracker.atlassian.net/browse/BIT-1542 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Fix For: 2.5 > > > Please merke topic/johanna/freebsd9 in bro and cmake. > It adds a bit of text to the installation instructions on how to install Bro on FreeBSD 9.X. It also adds tests to cmake that check if C++11 header files are usable; this prevents issues where a new compiler uses the includes of an older one, which apparently can easily happen on old versions of FreeBSD. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From noreply at bro.org Wed Mar 2 00:00:29 2016 From: noreply at bro.org (Merge Tracker) Date: Wed, 2 Mar 2016 00:00:29 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603020800.u2280TZ6030915@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------- BIT-1542 [1] Bro Johanna Amann - 2016-03-01 2.5 Normal Please merge topic/johanna/freebsd9 BIT-1537 [2] Bro Carlos Terr??n - 2016-02-29 2.5 Normal bro segfaults after compile in MacOS X 10.11 El Capitan BIT-1535 [3] Bro Justin Azoff - 2016-03-01 - Normal conn.log conn_state field or documentation is wrong BIT-1529 [4] Bro Seth Hall - 2016-03-01 2.5 Normal Base SIP scripts missing SUBSCRIBE and NOTIFY BIT-1507 [5] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- ------------ ---------- -------------------------------------- #55 [6] bro wglodek [7] 2016-02-07 http-evasion [8] #52 [9] bro J-Gras [10] 2016-01-18 Fixed matching mail address intel [11] #18 [12] bro-plugins jshlbrd [13] 2016-02-12 SSDP analyzer [14] [1] BIT-1542 https://bro-tracker.atlassian.net/browse/BIT-1542 [2] BIT-1537 https://bro-tracker.atlassian.net/browse/BIT-1537 [3] BIT-1535 https://bro-tracker.atlassian.net/browse/BIT-1535 [4] BIT-1529 https://bro-tracker.atlassian.net/browse/BIT-1529 [5] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [6] Pull Request #55 https://github.com/bro/bro/pull/55 [7] wglodek https://github.com/wglodek [8] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [9] Pull Request #52 https://github.com/bro/bro/pull/52 [10] J-Gras https://github.com/J-Gras [11] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [12] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [13] jshlbrd https://github.com/jshlbrd [14] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From dominik.charousset at haw-hamburg.de Wed Mar 2 04:53:49 2016 From: dominik.charousset at haw-hamburg.de (Dominik Charousset) Date: Wed, 2 Mar 2016 13:53:49 +0100 Subject: [Bro-Dev] Broker raw throughput In-Reply-To: References: <20160224161050.GE42006@shogun> <20160225161947.GI42006@shogun> Message-ID: With most noise like serialization etc. out of the way, this is what I measured on Linux: native sender -> native receiver 567520085 Bytes/s CAF sender -> native receiver 511333973 Bytes/s native sender -> CAF receiver 229689173 Bytes/s CAF sender -> CAF receiver 222102755 Bytes/s Send performance is OK, but performance drops significantly once CAF is used at the receiver. The profiler output (attached) doesn't point to a particular function that consumes an inappropriate amount of time. So it's either the sum of the many little functions called for each received chunk or the epoll_wait loop itself. I have created a ticket for further progress tracking / discussion [1] as this is clearly not a Bro/Broker problem. Thank you all for reporting this and all the input you have provided. @Matthias: FYI, I have used a new feature in CAF that allows senders to get feedback from the I/O layer for not overloading it. This allows the sender to adapt to the send rate of the network. Dominik [1] https://github.com/actor-framework/actor-framework/issues/432 > On Mar 1, 2016, at 16:05, Dominik Charousset wrote: > > Thanks for providing build scripts and sharing results. > > Just a quick heads-up from me: I have implemented a simple sender/receiver pair using C sockets as well as CAF brokers (attached, but works only with the current actor-system topic branch). Both sending and receiving are slower with CAF (as expected), although the performance is slightly better when using the ASIO backend [1]. I'm still investigating and hopefully come back to you guys later this week. > > Dominik > > [1] e.g. ./caf_impl --caf#middleman.network-backend=asio -s > > > On Feb 25, 2016, at 17:19, Matthias Vallentin wrote: > > > > For better reproducibility, here's the Makefile that I used to drive the > > experiments: > > > > CC = cc > > CXX = c++ > > FLAGS = -O3 -g -std=c++11 -stdlib=libc++ > > LIBS = -lcaf_core -lcaf_io -ltcmalloc -lprofiler > > > > caf-client: caf-client.cpp > > $(CXX) $(FLAGS) $< -o $@ $(LIBS) > > > > caf-server: caf-server.cpp > > $(CXX) $(FLAGS) $< -o $@ $(LIBS) > > > > bench-caf-client: > > CPUPROFILE=caf-client.prof ./caf-client 1000 > > > > bench-caf-server: > > CPUPROFILE=caf-server.prof ./caf-server 10 > > > > bench-caf-pprof: caf-client.prof caf-server.prof > > pprof --pdf caf-client caf-client.prof > caf-client.pdf > > pprof --pdf caf-server caf-server.prof > caf-server.pdf > > > > On my FreeBSD box, I had to add /usr/local/include to -I and -L, because > > I installed CAF and gperftools via ports. Since it's a headless machine > > without ps2pdf, we need extra level of indirection: > > > > (1) pprof --raw caf-client caf-client.prof > caf-client.raw > > (2) copy raw profile to desktop > > (3) pprof --pdf caf-client.raw > caf-client.pdf > > > > Hope this helps, > > > > Matthias > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160302/cc67db69/attachment-0003.html -------------- next part -------------- A non-text attachment was scrubbed... Name: caf-client.pdf Type: application/pdf Size: 24820 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160302/cc67db69/attachment-0002.pdf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160302/cc67db69/attachment-0004.html -------------- next part -------------- A non-text attachment was scrubbed... Name: caf-server.pdf Type: application/pdf Size: 25593 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160302/cc67db69/attachment-0003.pdf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160302/cc67db69/attachment-0005.html From jira at bro-tracker.atlassian.net Wed Mar 2 06:56:00 2016 From: jira at bro-tracker.atlassian.net (Nick Allen (JIRA)) Date: Wed, 2 Mar 2016 08:56:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1543) Kafka Logger - Writes Bro Logs to Kafka In-Reply-To: References: Message-ID: Nick Allen created BIT-1543: ------------------------------- Summary: Kafka Logger - Writes Bro Logs to Kafka Key: BIT-1543 URL: https://bro-tracker.atlassian.net/browse/BIT-1543 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Reporter: Nick Allen As part of the Apache Metron project, we needed a way to send Bro logs to Kafka. From my research it seems like this is a common request. I'd rather give this code back to the Bro community than maintain it as part of Apache Metron. This Bro plugin logs all Bro output to Kafka. Configuring this plugin is as simple as adding the following Bro script. {{ @load Bro/Kafka/logs-to-kafka.bro redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); redef Kafka::topic_name = "bro"; redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "localhost:9092" ); }} This plugin has the following features. * The user can specify a subset of all logs that should be sent to kafka. For example, to only send conn, http, and dns logs, specify the following. {{redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); }} * Full configurability of Kafka connectivity. Any configuration setting accepted by the librdkafka library can be passed to the plugin to tune how the logs are sent to Kafka. {{redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "localhost:9092", ["client.id"] = "bro" ); }} * The plugin will wait a configurable period of time (for example, 3 seconds) after shutdown to attempt to send any queued messages to Kafka. {{redef Kafka::max_wait_on_shutdown = 3000; }} * There are two message formats to choose from. By default, the standard Bro JSON format is used. There is an alternative 'tagged JSON' format that is provided by the plugin. Currently, all messages are sent to a single Bro topic. This 'tagged JSON' format helps a Kafka consumer distinguish which log stream the message originated from. This format prepends the log stream identifier to the JSON message. {{{'conn': { ... }} {'http': { ... }} {'dns': { ... }}}} To enable this alternative format, simply specify the following. {{redef Kafka::tag_json = T;}} -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Wed Mar 2 06:57:00 2016 From: jira at bro-tracker.atlassian.net (Nick Allen (JIRA)) Date: Wed, 2 Mar 2016 08:57:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1543) Kafka Logger - Writes Bro Logs to Kafka In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24603#comment-24603 ] Nick Allen commented on BIT-1543: --------------------------------- Created PR: https://github.com/bro/bro-plugins/pull/19 > Kafka Logger - Writes Bro Logs to Kafka > --------------------------------------- > > Key: BIT-1543 > URL: https://bro-tracker.atlassian.net/browse/BIT-1543 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Nick Allen > > As part of the Apache Metron project, we needed a way to send Bro logs to Kafka. From my research it seems like this is a common request. I'd rather give this code back to the Bro community than maintain it as part of Apache Metron. > This Bro plugin logs all Bro output to Kafka. Configuring this plugin is as simple as adding the following Bro script. > {{ > @load Bro/Kafka/logs-to-kafka.bro > redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); > redef Kafka::topic_name = "bro"; > redef Kafka::kafka_conf = table( > ["metadata.broker.list"] = "localhost:9092" > ); > }} > This plugin has the following features. > * The user can specify a subset of all logs that should be sent to kafka. For example, to only send conn, http, and dns logs, specify the following. > {{redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); > }} > * Full configurability of Kafka connectivity. Any configuration setting accepted by the librdkafka library can be passed to the plugin to tune how the logs are sent to Kafka. > {{redef Kafka::kafka_conf = table( > ["metadata.broker.list"] = "localhost:9092", > ["client.id"] = "bro" > ); > }} > * The plugin will wait a configurable period of time (for example, 3 seconds) after shutdown to attempt to send any queued messages to Kafka. > {{redef Kafka::max_wait_on_shutdown = 3000; > }} > * There are two message formats to choose from. By default, the standard Bro JSON format is used. There is an alternative 'tagged JSON' format that is provided by the plugin. Currently, all messages are sent to a single Bro topic. This 'tagged JSON' format helps a Kafka consumer distinguish which log stream the message originated from. This format prepends the log stream identifier to the JSON message. > {{{'conn': { ... }} > {'http': { ... }} > {'dns': { ... }}}} > To enable this alternative format, simply specify the following. > {{redef Kafka::tag_json = T;}} -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From nick at nickallen.org Wed Mar 2 06:58:00 2016 From: nick at nickallen.org (Nick Allen) Date: Wed, 2 Mar 2016 09:58:00 -0500 Subject: [Bro-Dev] Bro Kafka Logger Message-ID: FYI - I created a pull request and associated JIRA for a plugin that sends logs to Kafka. https://github.com/bro/bro-plugins/pull/19 https://bro-tracker.atlassian.net/browse/BIT-1543 Hope this is helpful. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160302/f88e201e/attachment.html From jira at bro-tracker.atlassian.net Wed Mar 2 06:59:00 2016 From: jira at bro-tracker.atlassian.net (Nick Allen (JIRA)) Date: Wed, 2 Mar 2016 08:59:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1543) Kafka Logger - Writes Bro Logs to Kafka In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated BIT-1543: ---------------------------- Status: In Progress (was: Open) > Kafka Logger - Writes Bro Logs to Kafka > --------------------------------------- > > Key: BIT-1543 > URL: https://bro-tracker.atlassian.net/browse/BIT-1543 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Nick Allen > > As part of the Apache Metron project, we needed a way to send Bro logs to Kafka. From my research it seems like this is a common request. I'd rather give this code back to the Bro community than maintain it as part of Apache Metron. > This Bro plugin logs all Bro output to Kafka. Configuring this plugin is as simple as adding the following Bro script. > {{ > @load Bro/Kafka/logs-to-kafka.bro > redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); > redef Kafka::topic_name = "bro"; > redef Kafka::kafka_conf = table( > ["metadata.broker.list"] = "localhost:9092" > ); > }} > This plugin has the following features. > * The user can specify a subset of all logs that should be sent to kafka. For example, to only send conn, http, and dns logs, specify the following. > {{redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); > }} > * Full configurability of Kafka connectivity. Any configuration setting accepted by the librdkafka library can be passed to the plugin to tune how the logs are sent to Kafka. > {{redef Kafka::kafka_conf = table( > ["metadata.broker.list"] = "localhost:9092", > ["client.id"] = "bro" > ); > }} > * The plugin will wait a configurable period of time (for example, 3 seconds) after shutdown to attempt to send any queued messages to Kafka. > {{redef Kafka::max_wait_on_shutdown = 3000; > }} > * There are two message formats to choose from. By default, the standard Bro JSON format is used. There is an alternative 'tagged JSON' format that is provided by the plugin. Currently, all messages are sent to a single Bro topic. This 'tagged JSON' format helps a Kafka consumer distinguish which log stream the message originated from. This format prepends the log stream identifier to the JSON message. > {{{'conn': { ... }} > {'http': { ... }} > {'dns': { ... }}}} > To enable this alternative format, simply specify the following. > {{redef Kafka::tag_json = T;}} -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Wed Mar 2 07:00:00 2016 From: jira at bro-tracker.atlassian.net (Nick Allen (JIRA)) Date: Wed, 2 Mar 2016 09:00:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1543) Kafka Logger - Writes Bro Logs to Kafka In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated BIT-1543: ---------------------------- Status: Merge Request (was: In Progress) > Kafka Logger - Writes Bro Logs to Kafka > --------------------------------------- > > Key: BIT-1543 > URL: https://bro-tracker.atlassian.net/browse/BIT-1543 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Nick Allen > > As part of the Apache Metron project, we needed a way to send Bro logs to Kafka. From my research it seems like this is a common request. I'd rather give this code back to the Bro community than maintain it as part of Apache Metron. > This Bro plugin logs all Bro output to Kafka. Configuring this plugin is as simple as adding the following Bro script. > {{ > @load Bro/Kafka/logs-to-kafka.bro > redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); > redef Kafka::topic_name = "bro"; > redef Kafka::kafka_conf = table( > ["metadata.broker.list"] = "localhost:9092" > ); > }} > This plugin has the following features. > * The user can specify a subset of all logs that should be sent to kafka. For example, to only send conn, http, and dns logs, specify the following. > {{redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); > }} > * Full configurability of Kafka connectivity. Any configuration setting accepted by the librdkafka library can be passed to the plugin to tune how the logs are sent to Kafka. > {{redef Kafka::kafka_conf = table( > ["metadata.broker.list"] = "localhost:9092", > ["client.id"] = "bro" > ); > }} > * The plugin will wait a configurable period of time (for example, 3 seconds) after shutdown to attempt to send any queued messages to Kafka. > {{redef Kafka::max_wait_on_shutdown = 3000; > }} > * There are two message formats to choose from. By default, the standard Bro JSON format is used. There is an alternative 'tagged JSON' format that is provided by the plugin. Currently, all messages are sent to a single Bro topic. This 'tagged JSON' format helps a Kafka consumer distinguish which log stream the message originated from. This format prepends the log stream identifier to the JSON message. > {{{'conn': { ... }} > {'http': { ... }} > {'dns': { ... }}}} > To enable this alternative format, simply specify the following. > {{redef Kafka::tag_json = T;}} -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From vallentin at icir.org Wed Mar 2 12:47:57 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Wed, 2 Mar 2016 12:47:57 -0800 Subject: [Bro-Dev] Broker raw throughput In-Reply-To: References: <20160224161050.GE42006@shogun> <20160225161947.GI42006@shogun> Message-ID: <20160302204757.GI76786@samurai.ICIR.org> > @Matthias: FYI, I have used a new feature in CAF that allows senders > to get feedback from the I/O layer for not overloading it. This allows > the sender to adapt to the send rate of the network. Great, it sounds like this would fix the stall/hang issues. I expect to port Broker to the actor-system branch by the end of the month. Matthias From noreply at bro.org Thu Mar 3 00:00:21 2016 From: noreply at bro.org (Merge Tracker) Date: Thu, 3 Mar 2016 00:00:21 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603030800.u2380LQs017957@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------- BIT-1543 [1] Bro Nick Allen - 2016-03-02 - Normal Kafka Logger - Writes Bro Logs to Kafka BIT-1542 [2] Bro Johanna Amann - 2016-03-01 2.5 Normal Please merge topic/johanna/freebsd9 BIT-1537 [3] Bro Carlos Terr??n - 2016-02-29 2.5 Normal bro segfaults after compile in MacOS X 10.11 El Capitan BIT-1535 [4] Bro Justin Azoff - 2016-03-01 - Normal conn.log conn_state field or documentation is wrong BIT-1529 [5] Bro Seth Hall - 2016-03-01 2.5 Normal Base SIP scripts missing SUBSCRIBE and NOTIFY BIT-1507 [6] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- --------------- ---------- ------------------------------------------------------ #55 [7] bro wglodek [8] 2016-02-07 http-evasion [9] #52 [10] bro J-Gras [11] 2016-01-18 Fixed matching mail address intel [12] #19 [13] bro-plugins nickwallen [14] 2016-03-02 [BIT-1543] Kafka Logger - Write Bro Logs to Kafka [15] #18 [16] bro-plugins jshlbrd [17] 2016-03-02 SSDP analyzer [18] #1 [19] try-bro t0b0 [20] 2016-03-03 fixed link in readme.markdown for Redefinitions [21] [1] BIT-1543 https://bro-tracker.atlassian.net/browse/BIT-1543 [2] BIT-1542 https://bro-tracker.atlassian.net/browse/BIT-1542 [3] BIT-1537 https://bro-tracker.atlassian.net/browse/BIT-1537 [4] BIT-1535 https://bro-tracker.atlassian.net/browse/BIT-1535 [5] BIT-1529 https://bro-tracker.atlassian.net/browse/BIT-1529 [6] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [7] Pull Request #55 https://github.com/bro/bro/pull/55 [8] wglodek https://github.com/wglodek [9] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [10] Pull Request #52 https://github.com/bro/bro/pull/52 [11] J-Gras https://github.com/J-Gras [12] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [13] Pull Request #19 https://github.com/bro/bro-plugins/pull/19 [14] nickwallen https://github.com/nickwallen [15] Merge Pull Request #19 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git add-kafka-plugin [16] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [17] jshlbrd https://github.com/jshlbrd [18] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp [19] Pull Request #1 https://github.com/bro/try-bro/pull/1 [20] t0b0 https://github.com/t0b0 [21] Merge Pull Request #1 with git pull --no-ff --no-commit https://github.com/t0b0/try-bro.git patch-1 From jira at bro-tracker.atlassian.net Thu Mar 3 07:47:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 3 Mar 2016 09:47:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1542) Please merge topic/johanna/freebsd9 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1542: --------------------------------- Assignee: Robin Sommer > Please merge topic/johanna/freebsd9 > ----------------------------------- > > Key: BIT-1542 > URL: https://bro-tracker.atlassian.net/browse/BIT-1542 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > Fix For: 2.5 > > > Please merke topic/johanna/freebsd9 in bro and cmake. > It adds a bit of text to the installation instructions on how to install Bro on FreeBSD 9.X. It also adds tests to cmake that check if C++11 header files are usable; this prevents issues where a new compiler uses the includes of an older one, which apparently can easily happen on old versions of FreeBSD. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Thu Mar 3 07:47:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 3 Mar 2016 09:47:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1543) Kafka Logger - Writes Bro Logs to Kafka In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1543: --------------------------------- Assignee: Seth Hall > Kafka Logger - Writes Bro Logs to Kafka > --------------------------------------- > > Key: BIT-1543 > URL: https://bro-tracker.atlassian.net/browse/BIT-1543 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Nick Allen > Assignee: Seth Hall > > As part of the Apache Metron project, we needed a way to send Bro logs to Kafka. From my research it seems like this is a common request. I'd rather give this code back to the Bro community than maintain it as part of Apache Metron. > This Bro plugin logs all Bro output to Kafka. Configuring this plugin is as simple as adding the following Bro script. > {{ > @load Bro/Kafka/logs-to-kafka.bro > redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); > redef Kafka::topic_name = "bro"; > redef Kafka::kafka_conf = table( > ["metadata.broker.list"] = "localhost:9092" > ); > }} > This plugin has the following features. > * The user can specify a subset of all logs that should be sent to kafka. For example, to only send conn, http, and dns logs, specify the following. > {{redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); > }} > * Full configurability of Kafka connectivity. Any configuration setting accepted by the librdkafka library can be passed to the plugin to tune how the logs are sent to Kafka. > {{redef Kafka::kafka_conf = table( > ["metadata.broker.list"] = "localhost:9092", > ["client.id"] = "bro" > ); > }} > * The plugin will wait a configurable period of time (for example, 3 seconds) after shutdown to attempt to send any queued messages to Kafka. > {{redef Kafka::max_wait_on_shutdown = 3000; > }} > * There are two message formats to choose from. By default, the standard Bro JSON format is used. There is an alternative 'tagged JSON' format that is provided by the plugin. Currently, all messages are sent to a single Bro topic. This 'tagged JSON' format helps a Kafka consumer distinguish which log stream the message originated from. This format prepends the log stream identifier to the JSON message. > {{{'conn': { ... }} > {'http': { ... }} > {'dns': { ... }}}} > To enable this alternative format, simply specify the following. > {{redef Kafka::tag_json = T;}} -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Thu Mar 3 07:57:01 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 3 Mar 2016 09:57:01 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1542) Please merge topic/johanna/freebsd9 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24604#comment-24604 ] Robin Sommer commented on BIT-1542: ----------------------------------- About the cmake change: Wouldn't this new header check better be located in {{RequireCXX11}}? > Please merge topic/johanna/freebsd9 > ----------------------------------- > > Key: BIT-1542 > URL: https://bro-tracker.atlassian.net/browse/BIT-1542 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > Fix For: 2.5 > > > Please merke topic/johanna/freebsd9 in bro and cmake. > It adds a bit of text to the installation instructions on how to install Bro on FreeBSD 9.X. It also adds tests to cmake that check if C++11 header files are usable; this prevents issues where a new compiler uses the includes of an older one, which apparently can easily happen on old versions of FreeBSD. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Thu Mar 3 07:58:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 3 Mar 2016 09:58:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1535) conn.log conn_state field or documentation is wrong In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1535: --------------------------------- Assignee: Robin Sommer > conn.log conn_state field or documentation is wrong > --------------------------------------------------- > > Key: BIT-1535 > URL: https://bro-tracker.atlassian.net/browse/BIT-1535 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Justin Azoff > Assignee: Robin Sommer > > There is an issue where the conn.log conn_state field will contain RSTR, which according to the documentation means "Established, responder aborted." > The problem that I notice is that I see log entries where conn_state is RSTR, but conn_history does not contain an 'h'. Additionally, the resp_h is absolutely not running a service on resp_p and the orig_h is usually in the process of a tcp scan. > Here are the top frequencies of RSTR without an h over about a weeks worth of conn logs: > {code} > 38193 RSTR Fr > 3662 RSTR DFr > 1801 RSTR DFdrR > 1248 RSTR DRr > 432 RSTR DrF > 232 RSTR Far > 128 RSTR DdAFrR > 79 RSTR DFadrR > 64 RSTR DrR > 58 RSTR DdAFarR > {code} > Compared to histories that did contain an h: > {code} > 425398 RSTR ShADadFr > 204149 RSTR ShADadFrR > 156303 RSTR ShADdFar > 141795 RSTR ShADadFRRr > 105704 RSTR ShADadfr > 79697 RSTR ShADadr > 63493 RSTR ShADaFr > 51704 RSTR ShADadFrrrr > 42075 RSTR ShADdar > 37678 RSTR ShADadfRr > {code} > I don't have a pcap for this, but I believe many of the weird connections are related to scans or backscatter. > I'm not sure if the code is wrong or the documentation is wrong, but I don't see how a fin+reset connection could be classified as established. > Also, One thing that would be a nice documentation addition is the answer to this question: > Given a conn.log entry, how do determine if there was a connection established? I thought it would be if the state was in 'SF S1 S2 S3 RSTO RSTR', but RSTR is problematic... -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Thu Mar 3 07:59:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 3 Mar 2016 09:59:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1529) Base SIP scripts missing SUBSCRIBE and NOTIFY In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1529: --------------------------------- Assignee: Robin Sommer > Base SIP scripts missing SUBSCRIBE and NOTIFY > --------------------------------------------- > > Key: BIT-1529 > URL: https://bro-tracker.atlassian.net/browse/BIT-1529 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Seth Hall > Assignee: Robin Sommer > Fix For: 2.5 > > > The base/protocols/sip/main.bro script has a set in `sip_methods` which needs to have SUBSCRIBE and NOTIFY added. They're defined in RFC 3265. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Thu Mar 3 08:00:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 3 Mar 2016 10:00:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1537) bro segfaults after compile in MacOS X 10.11 El Capitan In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1537: --------------------------------- Assignee: Robin Sommer > bro segfaults after compile in MacOS X 10.11 El Capitan > ------------------------------------------------------- > > Key: BIT-1537 > URL: https://bro-tracker.atlassian.net/browse/BIT-1537 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Carlos Terr?n > Assignee: Robin Sommer > Fix For: 2.5 > > > After compile with > {code} > ./configure --prefix=/usr/local > make > make install > {code} > And try to execute bro with: > {code} > bro -i en4 local > {code} > bro segfaults with > {code} > Program received signal SIGSEGV, Segmentation fault. > 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > 175 char *exponent = BN_bn2dec(pkey->pkey.rsa->e); > (gdb) bt > #0 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > #1 0x0000000100303e5d in file_analysis::X509::EndOfFile (this=0x105f8b710) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:56 > #2 0x000000010033f57a in file_analysis::File::EndOfFile (this=0x100961090) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/File.cc:522 > #3 0x000000010033bc6e in file_analysis::Manager::RemoveFile ( > this=0x105f8b710, file_id=...) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/Manager.cc:395 > #4 0x00000001002d910a in binpac::TLSHandshake::Handshake_Conn::proc_certificate (this=0x105f8a220, is_orig=false, certificates=0x100961f90) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:180 > #5 0x00000001002d99d4 in binpac::TLSHandshake::Handshake_Conn::proc_v3_certificate (this=0x105f8b710, is_orig=16, cl=) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:323 > #6 0x00000001002dc430 in binpac::TLSHandshake::Certificate::Parse ( > this=0x105f8a220, t_begin_of_data=, > t_end_of_data=0x101022f2e "", t_context=0x10095e480) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:1977 > {code} -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Thu Mar 3 09:23:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Thu, 3 Mar 2016 11:23:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1542) Please merge topic/johanna/freebsd9 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24605#comment-24605 ] Johanna Amann commented on BIT-1542: ------------------------------------ You are completely right about that. Do you just want to move it while merging or should i? > Please merge topic/johanna/freebsd9 > ----------------------------------- > > Key: BIT-1542 > URL: https://bro-tracker.atlassian.net/browse/BIT-1542 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > Fix For: 2.5 > > > Please merke topic/johanna/freebsd9 in bro and cmake. > It adds a bit of text to the installation instructions on how to install Bro on FreeBSD 9.X. It also adds tests to cmake that check if C++11 header files are usable; this prevents issues where a new compiler uses the includes of an older one, which apparently can easily happen on old versions of FreeBSD. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Thu Mar 3 09:34:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 3 Mar 2016 11:34:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1542) Please merge topic/johanna/freebsd9 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24606#comment-24606 ] Robin Sommer commented on BIT-1542: ----------------------------------- sure, I'll do it. > Please merge topic/johanna/freebsd9 > ----------------------------------- > > Key: BIT-1542 > URL: https://bro-tracker.atlassian.net/browse/BIT-1542 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > Fix For: 2.5 > > > Please merke topic/johanna/freebsd9 in bro and cmake. > It adds a bit of text to the installation instructions on how to install Bro on FreeBSD 9.X. It also adds tests to cmake that check if C++11 header files are usable; this prevents issues where a new compiler uses the includes of an older one, which apparently can easily happen on old versions of FreeBSD. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Thu Mar 3 10:04:01 2016 From: jira at bro-tracker.atlassian.net (Adam Slagell (JIRA)) Date: Thu, 3 Mar 2016 12:04:01 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1521) known services should probably ignore gridftp-data In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Slagell updated BIT-1521: ------------------------------ Priority: Normal (was: Low) > known services should probably ignore gridftp-data > -------------------------------------------------- > > Key: BIT-1521 > URL: https://bro-tracker.atlassian.net/browse/BIT-1521 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.4 > Reporter: Justin Azoff > Assignee: Justin Azoff > Fix For: 2.5 > > > known services script does > {code} > if ( ! addr_matches_host(id$resp_h, service_tracking) || > "ftp-data" in c$service || # don't include ftp data sessions > ("DNS" in c$service && c$resp$size == 0) ) # for dns, require that the server talks. > return; > {code} > but should probably also ignore gridftp-data. Probably a good idea to add a set of services that behave like ftp for it to check. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Thu Mar 3 10:05:00 2016 From: jira at bro-tracker.atlassian.net (Adam Slagell (JIRA)) Date: Thu, 3 Mar 2016 12:05:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1533) mysql analyzer does not set service to mysql In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Slagell updated BIT-1533: ------------------------------ Priority: Low (was: Normal) > mysql analyzer does not set service to mysql > -------------------------------------------- > > Key: BIT-1533 > URL: https://bro-tracker.atlassian.net/browse/BIT-1533 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Justin Azoff > Assignee: Vlad Grigorescu > Priority: Low > > The mysql analyzer does not set the service to mysql. The result of this is that conn.log and known_services do not show 'mysql' anywhere. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Thu Mar 3 10:05:00 2016 From: jira at bro-tracker.atlassian.net (Adam Slagell (JIRA)) Date: Thu, 3 Mar 2016 12:05:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1518) SSH analyzer doesn't handle non-conformant client version strings In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Slagell updated BIT-1518: ------------------------------ Priority: Low (was: Normal) > SSH analyzer doesn't handle non-conformant client version strings > ----------------------------------------------------------------- > > Key: BIT-1518 > URL: https://bro-tracker.atlassian.net/browse/BIT-1518 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Vlad Grigorescu > Assignee: Vlad Grigorescu > Priority: Low > Fix For: 2.5 > > > Received a report that some SSH clients send a version identification string similar to 'SSH-2.0-FooBar_Client\n' which causes a protocol violation in the SSH analyzer. RFC 4253 states that this must be terminated by '\r\n', but that's not what's being observed. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Thu Mar 3 12:55:00 2016 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 3 Mar 2016 14:55:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1544) File analysis code fails due to CheckString In-Reply-To: References: Message-ID: Seth Hall created BIT-1544: ------------------------------ Summary: File analysis code fails due to CheckString Key: BIT-1544 URL: https://bro-tracker.atlassian.net/browse/BIT-1544 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: git/master Reporter: Seth Hall If a `get_file_handle` function returns a string with NULL bytes in it, it causes Bro to do a reporter error due to the file analysis using CheckString to generate the file ids. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From noreply at bro.org Fri Mar 4 00:00:29 2016 From: noreply at bro.org (Merge Tracker) Date: Fri, 4 Mar 2016 00:00:29 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603040800.u2480TRE021083@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------- BIT-1543 [1] Bro Nick Allen Seth Hall 2016-03-03 - Normal Kafka Logger - Writes Bro Logs to Kafka BIT-1542 [2] Bro Johanna Amann Robin Sommer 2016-03-03 2.5 Normal Please merge topic/johanna/freebsd9 BIT-1537 [3] Bro Carlos Terr??n Robin Sommer 2016-03-03 2.5 Normal bro segfaults after compile in MacOS X 10.11 El Capitan BIT-1535 [4] Bro Justin Azoff Robin Sommer 2016-03-03 - Normal conn.log conn_state field or documentation is wrong BIT-1529 [5] Bro Seth Hall Robin Sommer 2016-03-03 2.5 Normal Base SIP scripts missing SUBSCRIBE and NOTIFY BIT-1507 [6] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- --------------- ---------- ------------------------------------------------------ #55 [7] bro wglodek [8] 2016-02-07 http-evasion [9] #52 [10] bro J-Gras [11] 2016-01-18 Fixed matching mail address intel [12] #19 [13] bro-plugins nickwallen [14] 2016-03-02 [BIT-1543] Kafka Logger - Write Bro Logs to Kafka [15] #18 [16] bro-plugins jshlbrd [17] 2016-03-03 SSDP analyzer [18] [1] BIT-1543 https://bro-tracker.atlassian.net/browse/BIT-1543 [2] BIT-1542 https://bro-tracker.atlassian.net/browse/BIT-1542 [3] BIT-1537 https://bro-tracker.atlassian.net/browse/BIT-1537 [4] BIT-1535 https://bro-tracker.atlassian.net/browse/BIT-1535 [5] BIT-1529 https://bro-tracker.atlassian.net/browse/BIT-1529 [6] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [7] Pull Request #55 https://github.com/bro/bro/pull/55 [8] wglodek https://github.com/wglodek [9] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [10] Pull Request #52 https://github.com/bro/bro/pull/52 [11] J-Gras https://github.com/J-Gras [12] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [13] Pull Request #19 https://github.com/bro/bro-plugins/pull/19 [14] nickwallen https://github.com/nickwallen [15] Merge Pull Request #19 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git add-kafka-plugin [16] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [17] jshlbrd https://github.com/jshlbrd [18] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Fri Mar 4 08:37:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Mar 2016 10:37:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1537) bro segfaults after compile in MacOS X 10.11 El Capitan In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24607#comment-24607 ] Robin Sommer commented on BIT-1537: ----------------------------------- Is it possible that broccoli needs some tweaking here to? After merging, I get lots of these: {{{ ../src/libbroccoli.so.5.1.0: undefined reference to `RAND_seed' ../src/libbroccoli.so.5.1.0: undefined reference to `RAND_pseudo_bytes' ../src/libbroccoli.so.5.1.0: undefined reference to `X509_STORE_CTX_get_error_depth' ../src/libbroccoli.so.5.1.0: undefined reference to `SSL_CTX_use_PrivateKey_file' ../src/libbroccoli.so.5.1.0: undefined reference to `SSL_CTX_set_cipher_list' ../src/libbroccoli.so.5.1.0: undefined reference to `CRYPTO_set_id_callback' ../src/libbroccoli.so.5.1.0: undefined reference to `SSL_CTX_free' ../src/libbroccoli.so.5.1.0: undefined reference to `SSL_load_error_strings' ../src/libbroccoli.so.5.1.0: undefined reference to `CRYPTO_set_dynlock_destroy_call }}} I pushed the cmake merge, without yet moving the submodules (so master won't pull it in yet). Can you try pulling those cmake updates into the all the submodules and see if it compiles fine for you then? > bro segfaults after compile in MacOS X 10.11 El Capitan > ------------------------------------------------------- > > Key: BIT-1537 > URL: https://bro-tracker.atlassian.net/browse/BIT-1537 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Carlos Terr?n > Assignee: Robin Sommer > Fix For: 2.5 > > > After compile with > {code} > ./configure --prefix=/usr/local > make > make install > {code} > And try to execute bro with: > {code} > bro -i en4 local > {code} > bro segfaults with > {code} > Program received signal SIGSEGV, Segmentation fault. > 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > 175 char *exponent = BN_bn2dec(pkey->pkey.rsa->e); > (gdb) bt > #0 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > #1 0x0000000100303e5d in file_analysis::X509::EndOfFile (this=0x105f8b710) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:56 > #2 0x000000010033f57a in file_analysis::File::EndOfFile (this=0x100961090) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/File.cc:522 > #3 0x000000010033bc6e in file_analysis::Manager::RemoveFile ( > this=0x105f8b710, file_id=...) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/Manager.cc:395 > #4 0x00000001002d910a in binpac::TLSHandshake::Handshake_Conn::proc_certificate (this=0x105f8a220, is_orig=false, certificates=0x100961f90) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:180 > #5 0x00000001002d99d4 in binpac::TLSHandshake::Handshake_Conn::proc_v3_certificate (this=0x105f8b710, is_orig=16, cl=) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:323 > #6 0x00000001002dc430 in binpac::TLSHandshake::Certificate::Parse ( > this=0x105f8a220, t_begin_of_data=, > t_end_of_data=0x101022f2e "", t_context=0x10095e480) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:1977 > {code} -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Fri Mar 4 08:38:01 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Mar 2016 10:38:01 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1537) bro segfaults after compile in MacOS X 10.11 El Capitan In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24607#comment-24607 ] Robin Sommer edited comment on BIT-1537 at 3/4/16 10:37 AM: ------------------------------------------------------------ Is it possible that broccoli needs some tweaking here to? After merging, I get lots of these: {code} ../src/libbroccoli.so.5.1.0: undefined reference to `RAND_seed' ../src/libbroccoli.so.5.1.0: undefined reference to `RAND_pseudo_bytes' ../src/libbroccoli.so.5.1.0: undefined reference to `X509_STORE_CTX_get_error_depth' ../src/libbroccoli.so.5.1.0: undefined reference to `SSL_CTX_use_PrivateKey_file' ../src/libbroccoli.so.5.1.0: undefined reference to `SSL_CTX_set_cipher_list' ../src/libbroccoli.so.5.1.0: undefined reference to `CRYPTO_set_id_callback' ../src/libbroccoli.so.5.1.0: undefined reference to `SSL_CTX_free' ../src/libbroccoli.so.5.1.0: undefined reference to `SSL_load_error_strings' ../src/libbroccoli.so.5.1.0: undefined reference to `CRYPTO_set_dynlock_destroy_call {code} I pushed the cmake merge, without yet moving the submodules (so master won't pull it in yet). Can you try pulling those cmake updates into the all the submodules and see if it compiles fine for you then? was (Author: robin): Is it possible that broccoli needs some tweaking here to? After merging, I get lots of these: {{{ ../src/libbroccoli.so.5.1.0: undefined reference to `RAND_seed' ../src/libbroccoli.so.5.1.0: undefined reference to `RAND_pseudo_bytes' ../src/libbroccoli.so.5.1.0: undefined reference to `X509_STORE_CTX_get_error_depth' ../src/libbroccoli.so.5.1.0: undefined reference to `SSL_CTX_use_PrivateKey_file' ../src/libbroccoli.so.5.1.0: undefined reference to `SSL_CTX_set_cipher_list' ../src/libbroccoli.so.5.1.0: undefined reference to `CRYPTO_set_id_callback' ../src/libbroccoli.so.5.1.0: undefined reference to `SSL_CTX_free' ../src/libbroccoli.so.5.1.0: undefined reference to `SSL_load_error_strings' ../src/libbroccoli.so.5.1.0: undefined reference to `CRYPTO_set_dynlock_destroy_call }}} I pushed the cmake merge, without yet moving the submodules (so master won't pull it in yet). Can you try pulling those cmake updates into the all the submodules and see if it compiles fine for you then? > bro segfaults after compile in MacOS X 10.11 El Capitan > ------------------------------------------------------- > > Key: BIT-1537 > URL: https://bro-tracker.atlassian.net/browse/BIT-1537 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Carlos Terr?n > Assignee: Robin Sommer > Fix For: 2.5 > > > After compile with > {code} > ./configure --prefix=/usr/local > make > make install > {code} > And try to execute bro with: > {code} > bro -i en4 local > {code} > bro segfaults with > {code} > Program received signal SIGSEGV, Segmentation fault. > 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > 175 char *exponent = BN_bn2dec(pkey->pkey.rsa->e); > (gdb) bt > #0 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > #1 0x0000000100303e5d in file_analysis::X509::EndOfFile (this=0x105f8b710) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:56 > #2 0x000000010033f57a in file_analysis::File::EndOfFile (this=0x100961090) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/File.cc:522 > #3 0x000000010033bc6e in file_analysis::Manager::RemoveFile ( > this=0x105f8b710, file_id=...) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/Manager.cc:395 > #4 0x00000001002d910a in binpac::TLSHandshake::Handshake_Conn::proc_certificate (this=0x105f8a220, is_orig=false, certificates=0x100961f90) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:180 > #5 0x00000001002d99d4 in binpac::TLSHandshake::Handshake_Conn::proc_v3_certificate (this=0x105f8b710, is_orig=16, cl=) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:323 > #6 0x00000001002dc430 in binpac::TLSHandshake::Certificate::Parse ( > this=0x105f8a220, t_begin_of_data=, > t_end_of_data=0x101022f2e "", t_context=0x10095e480) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:1977 > {code} -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Fri Mar 4 10:47:00 2016 From: jira at bro-tracker.atlassian.net (Jason Carr (JIRA)) Date: Fri, 4 Mar 2016 12:47:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly In-Reply-To: References: Message-ID: Jason Carr created BIT-1545: ------------------------------- Summary: SSH connection not recording entire flow correctly Key: BIT-1545 URL: https://bro-tracker.atlassian.net/browse/BIT-1545 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: git/master, 2.4 Environment: Ubuntu 14.04 LTS, myricom 10g capture card Reporter: Jason Carr Attachments: ssh-port22.pcap Making a connection out to a server via ssh does not write to conn.log while running with broctl but it does log to weird.log and ssh.log but nothing to conn.log. While running bro -C -r ssh-port22.pcap, a partial log entry is listed with an incorrect and very low number of packets and bytes. It was determined that disabling the SSH analyzer gets the correct conn.log output. Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it works as expected. Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Fri Mar 4 10:49:00 2016 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Fri, 4 Mar 2016 12:49:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24608#comment-24608 ] Justin Azoff commented on BIT-1545: ----------------------------------- I also realized it does the same thing on our standard ssh.pcap, so this problem must exist for everyone since 2.4 :-( > SSH connection not recording entire flow correctly > -------------------------------------------------- > > Key: BIT-1545 > URL: https://bro-tracker.atlassian.net/browse/BIT-1545 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.4 > Environment: Ubuntu 14.04 LTS, myricom 10g capture card > Reporter: Jason Carr > Labels: logging > Attachments: ssh-port22.pcap > > > Making a connection out to a server via ssh does not write to conn.log while running with broctl but it does log to weird.log and ssh.log but nothing to conn.log. > While running bro -C -r ssh-port22.pcap, a partial log entry is listed with an incorrect and very low number of packets and bytes. > It was determined that disabling the SSH analyzer gets the correct conn.log output. > Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); > Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it works as expected. > Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Fri Mar 4 11:06:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 4 Mar 2016 13:06:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1545: ------------------------------- Fix Version/s: 2.5 > SSH connection not recording entire flow correctly > -------------------------------------------------- > > Key: BIT-1545 > URL: https://bro-tracker.atlassian.net/browse/BIT-1545 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.4 > Environment: Ubuntu 14.04 LTS, myricom 10g capture card > Reporter: Jason Carr > Labels: logging > Fix For: 2.5 > > Attachments: ssh-port22.pcap > > > Making a connection out to a server via ssh does not write to conn.log while running with broctl but it does log to weird.log and ssh.log but nothing to conn.log. > While running bro -C -r ssh-port22.pcap, a partial log entry is listed with an incorrect and very low number of packets and bytes. > It was determined that disabling the SSH analyzer gets the correct conn.log output. > Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); > Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it works as expected. > Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Fri Mar 4 11:21:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 4 Mar 2016 13:21:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1537) bro segfaults after compile in MacOS X 10.11 El Capitan In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24609#comment-24609 ] Johanna Amann commented on BIT-1537: ------------------------------------ Sorry, this is fixed now. Could you also merge topic/johanna/openssl-osx in cmake topic/johanna/openssl in binpac, broccoli, bro-aux and master (updated there)? After that everything should build fine. > bro segfaults after compile in MacOS X 10.11 El Capitan > ------------------------------------------------------- > > Key: BIT-1537 > URL: https://bro-tracker.atlassian.net/browse/BIT-1537 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Carlos Terr?n > Assignee: Robin Sommer > Fix For: 2.5 > > > After compile with > {code} > ./configure --prefix=/usr/local > make > make install > {code} > And try to execute bro with: > {code} > bro -i en4 local > {code} > bro segfaults with > {code} > Program received signal SIGSEGV, Segmentation fault. > 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > 175 char *exponent = BN_bn2dec(pkey->pkey.rsa->e); > (gdb) bt > #0 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > #1 0x0000000100303e5d in file_analysis::X509::EndOfFile (this=0x105f8b710) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:56 > #2 0x000000010033f57a in file_analysis::File::EndOfFile (this=0x100961090) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/File.cc:522 > #3 0x000000010033bc6e in file_analysis::Manager::RemoveFile ( > this=0x105f8b710, file_id=...) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/Manager.cc:395 > #4 0x00000001002d910a in binpac::TLSHandshake::Handshake_Conn::proc_certificate (this=0x105f8a220, is_orig=false, certificates=0x100961f90) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:180 > #5 0x00000001002d99d4 in binpac::TLSHandshake::Handshake_Conn::proc_v3_certificate (this=0x105f8b710, is_orig=16, cl=) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:323 > #6 0x00000001002dc430 in binpac::TLSHandshake::Certificate::Parse ( > this=0x105f8a220, t_begin_of_data=, > t_end_of_data=0x101022f2e "", t_context=0x10095e480) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:1977 > {code} -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Fri Mar 4 12:10:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 4 Mar 2016 14:10:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1546) Please merge topic/johanna/str-functions In-Reply-To: References: Message-ID: Johanna Amann created BIT-1546: ---------------------------------- Summary: Please merge topic/johanna/str-functions Key: BIT-1546 URL: https://bro-tracker.atlassian.net/browse/BIT-1546 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: Johanna Amann Fix For: 2.5 topic/johanna/str-functions replaces a few string functions in Bro with functions provided by the standard operating system libraries. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Fri Mar 4 12:10:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 4 Mar 2016 14:10:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1546) Please merge topic/johanna/str-functions In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1546: ------------------------------- Status: Merge Request (was: Open) > Please merge topic/johanna/str-functions > ---------------------------------------- > > Key: BIT-1546 > URL: https://bro-tracker.atlassian.net/browse/BIT-1546 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Fix For: 2.5 > > > topic/johanna/str-functions replaces a few string functions in Bro with functions provided by the standard operating system libraries. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Fri Mar 4 15:00:00 2016 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Fri, 4 Mar 2016 17:00:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1547) broctl sets the same state variables over and over In-Reply-To: References: Message-ID: Justin Azoff created BIT-1547: --------------------------------- Summary: broctl sets the same state variables over and over Key: BIT-1547 URL: https://bro-tracker.atlassian.net/browse/BIT-1547 Project: Bro Issue Tracker Issue Type: Problem Components: BroControl Affects Versions: git/master Reporter: Justin Azoff Assignee: Daniel Thayer I happened to notice broctl check on one of our test boxes was slow. traced it to sqlite commits() being very slow. Then noticed that broctl seems to call set_state() with the same key, val over and over again... once for each worker.. so a few thousand sets just to run broctl check. Changing set_state to {code} # Set a dynamic state variable. def set_state(self, key, val): key = key.lower() if self.state.get(key) == val: return self.state[key] = val self.state_store.set(key, val) {code} Seemed to mostly fix it, aside from this: {code} Set manager-port to 47760 Set manager-port to 47761 Set manager-port to 47760 Set manager-port to 47761 Set manager-port to 47760 Set manager-port to 47761 Set manager-port to 47760 Set manager-port to 47761 Set manager-port to 47760 Set manager-port to 47761 Set manager-port to 47760 Set manager-port to 47761 {code} any idea why that is flipping around like that? We should possibly add a way for broctl to update state vars without calling commit where it knows it will be setting a large number of state vars in a loop. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Fri Mar 4 17:19:00 2016 From: jira at bro-tracker.atlassian.net (Aaron Eppert (JIRA)) Date: Fri, 4 Mar 2016 19:19:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24610#comment-24610 ] Aaron Eppert commented on BIT-1545: ----------------------------------- {{SSH::skip_processing_after_detection}} defaults to T and is {{&redef}}'able. With that set {{skip_further_processing()}} is called in {{event ssh_auth_successful(c: connection, auth_method_none: bool) &priority=5}}, I assume this would be the culprit in the matter. Try: {{redef SSH::skip_processing_after_detection = T;}} and see if that fixes the issue. > SSH connection not recording entire flow correctly > -------------------------------------------------- > > Key: BIT-1545 > URL: https://bro-tracker.atlassian.net/browse/BIT-1545 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.4 > Environment: Ubuntu 14.04 LTS, myricom 10g capture card > Reporter: Jason Carr > Labels: logging > Fix For: 2.5 > > Attachments: ssh-port22.pcap > > > Making a connection out to a server via ssh does not write to conn.log while running with broctl but it does log to weird.log and ssh.log but nothing to conn.log. > While running bro -C -r ssh-port22.pcap, a partial log entry is listed with an incorrect and very low number of packets and bytes. > It was determined that disabling the SSH analyzer gets the correct conn.log output. > Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); > Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it works as expected. > Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Fri Mar 4 18:37:00 2016 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Fri, 4 Mar 2016 20:37:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24611#comment-24611 ] Justin Azoff commented on BIT-1545: ----------------------------------- Oh, right... I knew that option existed but I missed that the default was changed. I looked at all the changes in src/analyzer/protocol/ssh/ but of course that is inside scripts/ That sure explains a lot. You need to redef it to F though, not T, to fix the bytes reported in the conn.log :-) > SSH connection not recording entire flow correctly > -------------------------------------------------- > > Key: BIT-1545 > URL: https://bro-tracker.atlassian.net/browse/BIT-1545 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.4 > Environment: Ubuntu 14.04 LTS, myricom 10g capture card > Reporter: Jason Carr > Labels: logging > Fix For: 2.5 > > Attachments: ssh-port22.pcap > > > Making a connection out to a server via ssh does not write to conn.log while running with broctl but it does log to weird.log and ssh.log but nothing to conn.log. > While running bro -C -r ssh-port22.pcap, a partial log entry is listed with an incorrect and very low number of packets and bytes. > It was determined that disabling the SSH analyzer gets the correct conn.log output. > Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); > Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it works as expected. > Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Fri Mar 4 18:39:00 2016 From: jira at bro-tracker.atlassian.net (Aaron Eppert (JIRA)) Date: Fri, 4 Mar 2016 20:39:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24610#comment-24610 ] Aaron Eppert edited comment on BIT-1545 at 3/4/16 8:38 PM: ----------------------------------------------------------- {{SSH::skip_processing_after_detection}} defaults to T and is {{&redef}}'able. With that set {{skip_further_processing()}} is called in {{event ssh_auth_successful(c: connection, auth_method_none: bool) &priority=5}}, I assume this would be the culprit in the matter. Try: {{redef SSH::skip_processing_after_detection = F;}} and see if that fixes the issue. (Per Justin... T and F on a Friday look the same :) ) was (Author: aeppert): {{SSH::skip_processing_after_detection}} defaults to T and is {{&redef}}'able. With that set {{skip_further_processing()}} is called in {{event ssh_auth_successful(c: connection, auth_method_none: bool) &priority=5}}, I assume this would be the culprit in the matter. Try: {{redef SSH::skip_processing_after_detection = T;}} and see if that fixes the issue. > SSH connection not recording entire flow correctly > -------------------------------------------------- > > Key: BIT-1545 > URL: https://bro-tracker.atlassian.net/browse/BIT-1545 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.4 > Environment: Ubuntu 14.04 LTS, myricom 10g capture card > Reporter: Jason Carr > Labels: logging > Fix For: 2.5 > > Attachments: ssh-port22.pcap > > > Making a connection out to a server via ssh does not write to conn.log while running with broctl but it does log to weird.log and ssh.log but nothing to conn.log. > While running bro -C -r ssh-port22.pcap, a partial log entry is listed with an incorrect and very low number of packets and bytes. > It was determined that disabling the SSH analyzer gets the correct conn.log output. > Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); > Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it works as expected. > Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Fri Mar 4 20:25:00 2016 From: jira at bro-tracker.atlassian.net (Jamshid Karimi (JIRA)) Date: Fri, 4 Mar 2016 22:25:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1548) SendMail parameter is missing from broctl.cfg file in Debian binary installation In-Reply-To: References: Message-ID: Jamshid Karimi created BIT-1548: ----------------------------------- Summary: SendMail parameter is missing from broctl.cfg file in Debian binary installation Key: BIT-1548 URL: https://bro-tracker.atlassian.net/browse/BIT-1548 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: 2.4 Environment: Date tested: 2016-03-05 Operating system: Debian 8.2 Repository: Open Build System Packages: bro 2.4.1-0 bro-core 2.4.1-0 broctl 2.4.1-0 libbroccoli 2.4.1-0 Reporter: Jamshid Karimi The Debian binary packages from Open Build Service have sendmail binary location set to SENDMAIL-NOTFOUND by default but provide no SendMail parameter in broctl.cfg to set the correct location. This means, out of the box, Bro does not send any summary connection reports to the configured email recipient. For a recent binary installation, I had to manually add the following line to broctl.cfg file to resolve the issue: SendMail = /usr/sbin/sendmail Here is the output of broctl config right after installation: Hint: Run the broctl "deploy" command to get started. bindir = /opt/bro/bin broargs = brobase = /opt/bro broctlconfigdir = /opt/bro/spool broport = 47760 broscriptdir = /opt/bro/share/bro capstatspath = /opt/bro/bin/capstats cfgdir = /opt/bro/etc cflowaddress = cflowpassword = cflowuser = commandtimeout = 60 commtimeout = 10 compresscmd = gzip -9 compressextension = gz compresslogs = 1 cron = 0 croncmd = debug = 0 debuglog = /opt/bro/spool/debug.log env_vars = havenfs = 0 helperdir = /opt/bro/share/broctl/scripts/helpers ipv6comm = 1 keeplogs = libdir = /opt/bro/lib libdirinternal = /opt/bro/lib/broctl localnetscfg = /opt/bro/etc/networks.cfg lockfile = /opt/bro/spool/lock logdir = /opt/bro/logs logexpireinterval = 0 logrotationinterval = 3600 mailalarmsinterval = 86400 mailalarmsto = root at localhost mailconnectionsummary = 1 mailfrom = Big Brother mailhostupdown = 1 mailreplyto = mailsubjectprefix = [Bro] mailto = root at localhost makearchivename = /opt/bro/share/broctl/scripts/make-archive-name memlimit = unlimited mindiskspace = 5 nodecfg = /opt/bro/etc/node.cfg os = linux pfringclusterid = 0 pfringclustertype = 4-tuple pfringfirstappinstance = 0 pin_command = taskset -c plugindir = /opt/bro/lib/broctl/plugins policydir = /opt/bro/share/bro policydirsiteinstall = /opt/bro/spool/installed-scripts-do-not-touch/site policydirsiteinstallauto = /opt/bro/spool/installed-scripts-do-not-touch/auto postprocdir = /opt/bro/share/broctl/scripts/postprocessors prefixes = local savetraces = 0 scriptsdir = /opt/bro/share/broctl/scripts sendmail = SENDMAIL-NOTFOUND sigint = 0 sitepluginpath = sitepolicymanager = local-manager.bro sitepolicypath = /opt/bro/share/bro/site sitepolicystandalone = local.bro sitepolicyworker = local-worker.bro spooldir = /opt/bro/spool standalone = 1 statefile = /opt/bro/spool/state.db staticdir = /opt/bro/share/broctl statsdir = /opt/bro/logs/stats statslog = /opt/bro/spool/stats.log statslogenable = 1 statslogexpireinterval = 0 statuscmdshowall = 1 stoptimeout = 60 test.enabled = 0 test.foo = 1 time = timefmt = %d %b %H:%M:%S timemachinehost = timemachineport = 47757/tcp tmpdir = /opt/bro/spool/tmp tmpexecdir = /opt/bro/spool/tmp tracesummary = /opt/bro/bin/trace-summary version = 1.4 zoneid = -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Fri Mar 4 20:34:01 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Mar 2016 22:34:01 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1542) Please merge topic/johanna/freebsd9 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1542: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Please merge topic/johanna/freebsd9 > ----------------------------------- > > Key: BIT-1542 > URL: https://bro-tracker.atlassian.net/browse/BIT-1542 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > Fix For: 2.5 > > > Please merke topic/johanna/freebsd9 in bro and cmake. > It adds a bit of text to the installation instructions on how to install Bro on FreeBSD 9.X. It also adds tests to cmake that check if C++11 header files are usable; this prevents issues where a new compiler uses the includes of an older one, which apparently can easily happen on old versions of FreeBSD. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Fri Mar 4 20:34:01 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Mar 2016 22:34:01 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1535) conn.log conn_state field or documentation is wrong In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1535: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > conn.log conn_state field or documentation is wrong > --------------------------------------------------- > > Key: BIT-1535 > URL: https://bro-tracker.atlassian.net/browse/BIT-1535 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Justin Azoff > Assignee: Robin Sommer > > There is an issue where the conn.log conn_state field will contain RSTR, which according to the documentation means "Established, responder aborted." > The problem that I notice is that I see log entries where conn_state is RSTR, but conn_history does not contain an 'h'. Additionally, the resp_h is absolutely not running a service on resp_p and the orig_h is usually in the process of a tcp scan. > Here are the top frequencies of RSTR without an h over about a weeks worth of conn logs: > {code} > 38193 RSTR Fr > 3662 RSTR DFr > 1801 RSTR DFdrR > 1248 RSTR DRr > 432 RSTR DrF > 232 RSTR Far > 128 RSTR DdAFrR > 79 RSTR DFadrR > 64 RSTR DrR > 58 RSTR DdAFarR > {code} > Compared to histories that did contain an h: > {code} > 425398 RSTR ShADadFr > 204149 RSTR ShADadFrR > 156303 RSTR ShADdFar > 141795 RSTR ShADadFRRr > 105704 RSTR ShADadfr > 79697 RSTR ShADadr > 63493 RSTR ShADaFr > 51704 RSTR ShADadFrrrr > 42075 RSTR ShADdar > 37678 RSTR ShADadfRr > {code} > I don't have a pcap for this, but I believe many of the weird connections are related to scans or backscatter. > I'm not sure if the code is wrong or the documentation is wrong, but I don't see how a fin+reset connection could be classified as established. > Also, One thing that would be a nice documentation addition is the answer to this question: > Given a conn.log entry, how do determine if there was a connection established? I thought it would be if the state was in 'SF S1 S2 S3 RSTO RSTR', but RSTR is problematic... -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Fri Mar 4 20:34:01 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Mar 2016 22:34:01 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1537) bro segfaults after compile in MacOS X 10.11 El Capitan In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1537: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > bro segfaults after compile in MacOS X 10.11 El Capitan > ------------------------------------------------------- > > Key: BIT-1537 > URL: https://bro-tracker.atlassian.net/browse/BIT-1537 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Carlos Terr?n > Assignee: Robin Sommer > Fix For: 2.5 > > > After compile with > {code} > ./configure --prefix=/usr/local > make > make install > {code} > And try to execute bro with: > {code} > bro -i en4 local > {code} > bro segfaults with > {code} > Program received signal SIGSEGV, Segmentation fault. > 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > 175 char *exponent = BN_bn2dec(pkey->pkey.rsa->e); > (gdb) bt > #0 0x00000001003045d2 in file_analysis::X509::ParseCertificate ( > cert_val=, fid=) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:175 > #1 0x0000000100303e5d in file_analysis::X509::EndOfFile (this=0x105f8b710) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/analyzer/x509/X509.cc:56 > #2 0x000000010033f57a in file_analysis::File::EndOfFile (this=0x100961090) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/File.cc:522 > #3 0x000000010033bc6e in file_analysis::Manager::RemoveFile ( > this=0x105f8b710, file_id=...) > at /Users/terron/tmp/bro-2.4.1/src/file_analysis/Manager.cc:395 > #4 0x00000001002d910a in binpac::TLSHandshake::Handshake_Conn::proc_certificate (this=0x105f8a220, is_orig=false, certificates=0x100961f90) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:180 > #5 0x00000001002d99d4 in binpac::TLSHandshake::Handshake_Conn::proc_v3_certificate (this=0x105f8b710, is_orig=16, cl=) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:323 > #6 0x00000001002dc430 in binpac::TLSHandshake::Certificate::Parse ( > this=0x105f8a220, t_begin_of_data=, > t_end_of_data=0x101022f2e "", t_context=0x10095e480) > at /Users/terron/tmp/bro-2.4.1/build/src/analyzer/protocol/ssl/tls-handshake_pac.cc:1977 > {code} -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From jira at bro-tracker.atlassian.net Fri Mar 4 20:34:01 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 4 Mar 2016 22:34:01 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1529) Base SIP scripts missing SUBSCRIBE and NOTIFY In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1529: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Base SIP scripts missing SUBSCRIBE and NOTIFY > --------------------------------------------- > > Key: BIT-1529 > URL: https://bro-tracker.atlassian.net/browse/BIT-1529 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Seth Hall > Assignee: Robin Sommer > Fix For: 2.5 > > > The base/protocols/sip/main.bro script has a set in `sip_methods` which needs to have SUBSCRIBE and NOTIFY added. They're defined in RFC 3265. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From noreply at bro.org Sat Mar 5 00:00:21 2016 From: noreply at bro.org (Merge Tracker) Date: Sat, 5 Mar 2016 00:00:21 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603050800.u2580LmZ007437@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1546 [1] Bro Johanna Amann - 2016-03-04 2.5 Normal Please merge topic/johanna/str-functions BIT-1543 [2] Bro Nick Allen Seth Hall 2016-03-03 - Normal Kafka Logger - Writes Bro Logs to Kafka BIT-1507 [3] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- --------------- ---------- ------------------------------------------------------ #55 [4] bro wglodek [5] 2016-03-04 http-evasion [6] #52 [7] bro J-Gras [8] 2016-01-18 Fixed matching mail address intel [9] #19 [10] bro-plugins nickwallen [11] 2016-03-02 [BIT-1543] Kafka Logger - Write Bro Logs to Kafka [12] #18 [13] bro-plugins jshlbrd [14] 2016-03-03 SSDP analyzer [15] [1] BIT-1546 https://bro-tracker.atlassian.net/browse/BIT-1546 [2] BIT-1543 https://bro-tracker.atlassian.net/browse/BIT-1543 [3] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [4] Pull Request #55 https://github.com/bro/bro/pull/55 [5] wglodek https://github.com/wglodek [6] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [7] Pull Request #52 https://github.com/bro/bro/pull/52 [8] J-Gras https://github.com/J-Gras [9] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [10] Pull Request #19 https://github.com/bro/bro-plugins/pull/19 [11] nickwallen https://github.com/nickwallen [12] Merge Pull Request #19 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git add-kafka-plugin [13] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [14] jshlbrd https://github.com/jshlbrd [15] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Sat Mar 5 09:07:00 2016 From: jira at bro-tracker.atlassian.net (Aaron Eppert (JIRA)) Date: Sat, 5 Mar 2016 11:07:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24612#comment-24612 ] Aaron Eppert commented on BIT-1545: ----------------------------------- https://github.com/bro/bro/pull/58 Is my proposal for dealing with this matter. Once we know it's SSH and have everything, disable the analyzer, and continue collecting data appropriately for writing into conn.log. > SSH connection not recording entire flow correctly > -------------------------------------------------- > > Key: BIT-1545 > URL: https://bro-tracker.atlassian.net/browse/BIT-1545 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.4 > Environment: Ubuntu 14.04 LTS, myricom 10g capture card > Reporter: Jason Carr > Labels: logging > Fix For: 2.5 > > Attachments: ssh-port22.pcap > > > Making a connection out to a server via ssh does not write to conn.log while running with broctl but it does log to weird.log and ssh.log but nothing to conn.log. > While running bro -C -r ssh-port22.pcap, a partial log entry is listed with an incorrect and very low number of packets and bytes. > It was determined that disabling the SSH analyzer gets the correct conn.log output. > Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); > Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it works as expected. > Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) From noreply at bro.org Sun Mar 6 00:00:21 2016 From: noreply at bro.org (Merge Tracker) Date: Sun, 6 Mar 2016 00:00:21 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603060800.u2680LZO017343@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1546 [1] Bro Johanna Amann - 2016-03-04 2.5 Normal Please merge topic/johanna/str-functions BIT-1543 [2] Bro Nick Allen Seth Hall 2016-03-03 - Normal Kafka Logger - Writes Bro Logs to Kafka BIT-1507 [3] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- --------------- ---------- -------------------------------------------------------------------------- #58 [4] bro aeppert [5] 2016-03-05 (BIT-1545) Add "disable_analyzer_after_detection" en lieu of "skip_pr??? [6] #55 [7] bro wglodek [8] 2016-03-04 http-evasion [9] #52 [10] bro J-Gras [11] 2016-01-18 Fixed matching mail address intel [12] #19 [13] bro-plugins nickwallen [14] 2016-03-02 [BIT-1543] Kafka Logger - Write Bro Logs to Kafka [15] #18 [16] bro-plugins jshlbrd [17] 2016-03-03 SSDP analyzer [18] [1] BIT-1546 https://bro-tracker.atlassian.net/browse/BIT-1546 [2] BIT-1543 https://bro-tracker.atlassian.net/browse/BIT-1543 [3] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [4] Pull Request #58 https://github.com/bro/bro/pull/58 [5] aeppert https://github.com/aeppert [6] Merge Pull Request #58 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-4 [7] Pull Request #55 https://github.com/bro/bro/pull/55 [8] wglodek https://github.com/wglodek [9] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [10] Pull Request #52 https://github.com/bro/bro/pull/52 [11] J-Gras https://github.com/J-Gras [12] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [13] Pull Request #19 https://github.com/bro/bro-plugins/pull/19 [14] nickwallen https://github.com/nickwallen [15] Merge Pull Request #19 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git add-kafka-plugin [16] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [17] jshlbrd https://github.com/jshlbrd [18] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From noreply at bro.org Mon Mar 7 00:00:19 2016 From: noreply at bro.org (Merge Tracker) Date: Mon, 7 Mar 2016 00:00:19 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603070800.u2780J1h003061@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1546 [1] Bro Johanna Amann - 2016-03-04 2.5 Normal Please merge topic/johanna/str-functions BIT-1543 [2] Bro Nick Allen Seth Hall 2016-03-03 - Normal Kafka Logger - Writes Bro Logs to Kafka BIT-1507 [3] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- --------------- ---------- -------------------------------------------------------------------------- #58 [4] bro aeppert [5] 2016-03-05 (BIT-1545) Add "disable_analyzer_after_detection" en lieu of "skip_pr??? [6] #55 [7] bro wglodek [8] 2016-03-04 http-evasion [9] #52 [10] bro J-Gras [11] 2016-01-18 Fixed matching mail address intel [12] #19 [13] bro-plugins nickwallen [14] 2016-03-02 [BIT-1543] Kafka Logger - Write Bro Logs to Kafka [15] #18 [16] bro-plugins jshlbrd [17] 2016-03-03 SSDP analyzer [18] [1] BIT-1546 https://bro-tracker.atlassian.net/browse/BIT-1546 [2] BIT-1543 https://bro-tracker.atlassian.net/browse/BIT-1543 [3] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [4] Pull Request #58 https://github.com/bro/bro/pull/58 [5] aeppert https://github.com/aeppert [6] Merge Pull Request #58 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-4 [7] Pull Request #55 https://github.com/bro/bro/pull/55 [8] wglodek https://github.com/wglodek [9] Merge Pull Request #55 with git pull --no-ff --no-commit https://github.com/0xcc-labs/bro.git topic/http-evasion [10] Pull Request #52 https://github.com/bro/bro/pull/52 [11] J-Gras https://github.com/J-Gras [12] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [13] Pull Request #19 https://github.com/bro/bro-plugins/pull/19 [14] nickwallen https://github.com/nickwallen [15] Merge Pull Request #19 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git add-kafka-plugin [16] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [17] jshlbrd https://github.com/jshlbrd [18] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Mon Mar 7 05:51:01 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Mon, 7 Mar 2016 07:51:01 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1547) broctl sets the same state variables over and over In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24700#comment-24700 ] Daniel Thayer commented on BIT-1547: ------------------------------------ How much faster does it get after adding the "if self.state.get(key)" check? And how many workers are you using? > broctl sets the same state variables over and over > -------------------------------------------------- > > Key: BIT-1547 > URL: https://bro-tracker.atlassian.net/browse/BIT-1547 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Reporter: Justin Azoff > Assignee: Daniel Thayer > > I happened to notice broctl check on one of our test boxes was slow. traced it to sqlite commits() being very slow. Then noticed that broctl seems to call set_state() with the same key, val over and over again... once for each worker.. so a few thousand sets just to run broctl check. > Changing set_state to > {code} > # Set a dynamic state variable. > def set_state(self, key, val): > key = key.lower() > if self.state.get(key) == val: > return > self.state[key] = val > self.state_store.set(key, val) > {code} > Seemed to mostly fix it, aside from this: > {code} > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > {code} > any idea why that is flipping around like that? > We should possibly add a way for broctl to update state vars without calling commit where it knows it will be setting a large number of state vars in a loop. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From jira at bro-tracker.atlassian.net Mon Mar 7 07:13:00 2016 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Mon, 7 Mar 2016 09:13:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1547) broctl sets the same state variables over and over In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24701#comment-24701 ] Justin Azoff commented on BIT-1547: ----------------------------------- Before the change: real 1m32.978s After the change: real 0m6.413s It set options 1192 times. this is what it sets (count + key, value): {code} 1 commiting set of configchksum to "8a94f1078550bc5bcfbb78228b673d75" 1 commiting set of confignodechksum to "761eb1e973f7bdbc94120ae522b76db5" 34 commiting set of manager-port to 47760 34 commiting set of manager-port to 47761 34 commiting set of nids-test1a-1-port to 47763 34 commiting set of nids-test1a-2-port to 47764 34 commiting set of nids-test1a-3-port to 47765 34 commiting set of nids-test1a-4-port to 47766 34 commiting set of nids-test1a-5-port to 47767 34 commiting set of nids-test1a-6-port to 47768 34 commiting set of nids-test1a-7-port to 47769 34 commiting set of nids-test1a-8-port to 47770 34 commiting set of nids-test1b-1-port to 47771 34 commiting set of nids-test1b-2-port to 47772 34 commiting set of nids-test1b-3-port to 47773 34 commiting set of nids-test1b-4-port to 47774 34 commiting set of nids-test1b-5-port to 47775 34 commiting set of nids-test1b-6-port to 47776 34 commiting set of nids-test1b-7-port to 47777 34 commiting set of nids-test1b-8-port to 47778 34 commiting set of nids-test2a-1-port to 47779 34 commiting set of nids-test2a-2-port to 47780 34 commiting set of nids-test2a-3-port to 47781 34 commiting set of nids-test2a-4-port to 47782 34 commiting set of nids-test2a-5-port to 47783 34 commiting set of nids-test2a-6-port to 47784 34 commiting set of nids-test2a-7-port to 47785 34 commiting set of nids-test2a-8-port to 47786 34 commiting set of nids-test2b-1-port to 47787 34 commiting set of nids-test2b-2-port to 47788 34 commiting set of nids-test2b-3-port to 47789 34 commiting set of nids-test2b-4-port to 47790 34 commiting set of nids-test2b-5-port to 47791 34 commiting set of nids-test2b-6-port to 47792 34 commiting set of nids-test2b-7-port to 47793 34 commiting set of nids-test2b-8-port to 47794 34 commiting set of proxy-1-port to 47762 {code} After the change it just does {code} 1 commiting set of configchksum to "8a94f1078550bc5bcfbb78228b673d75" 1 commiting set of confignodechksum to "761eb1e973f7bdbc94120ae522b76db5" 34 commiting set of manager-port to 47760 34 commiting set of manager-port to 47761 {code} > broctl sets the same state variables over and over > -------------------------------------------------- > > Key: BIT-1547 > URL: https://bro-tracker.atlassian.net/browse/BIT-1547 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Reporter: Justin Azoff > Assignee: Daniel Thayer > > I happened to notice broctl check on one of our test boxes was slow. traced it to sqlite commits() being very slow. Then noticed that broctl seems to call set_state() with the same key, val over and over again... once for each worker.. so a few thousand sets just to run broctl check. > Changing set_state to > {code} > # Set a dynamic state variable. > def set_state(self, key, val): > key = key.lower() > if self.state.get(key) == val: > return > self.state[key] = val > self.state_store.set(key, val) > {code} > Seemed to mostly fix it, aside from this: > {code} > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > {code} > any idea why that is flipping around like that? > We should possibly add a way for broctl to update state vars without calling commit where it knows it will be setting a large number of state vars in a loop. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From jira at bro-tracker.atlassian.net Mon Mar 7 08:12:02 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Mon, 7 Mar 2016 10:12:02 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1547) broctl sets the same state variables over and over In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24702#comment-24702 ] Daniel Thayer commented on BIT-1547: ------------------------------------ In branch "topic/dnthayer/ticket1547" in the broctl repo, I've eliminated unnecessary writes to the state db. > broctl sets the same state variables over and over > -------------------------------------------------- > > Key: BIT-1547 > URL: https://bro-tracker.atlassian.net/browse/BIT-1547 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Reporter: Justin Azoff > Assignee: Daniel Thayer > > I happened to notice broctl check on one of our test boxes was slow. traced it to sqlite commits() being very slow. Then noticed that broctl seems to call set_state() with the same key, val over and over again... once for each worker.. so a few thousand sets just to run broctl check. > Changing set_state to > {code} > # Set a dynamic state variable. > def set_state(self, key, val): > key = key.lower() > if self.state.get(key) == val: > return > self.state[key] = val > self.state_store.set(key, val) > {code} > Seemed to mostly fix it, aside from this: > {code} > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > {code} > any idea why that is flipping around like that? > We should possibly add a way for broctl to update state vars without calling commit where it knows it will be setting a large number of state vars in a loop. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From jira at bro-tracker.atlassian.net Mon Mar 7 08:13:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Mon, 7 Mar 2016 10:13:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1547) broctl sets the same state variables over and over In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1547: ------------------------------- Status: Merge Request (was: Open) Assignee: (was: Daniel Thayer) > broctl sets the same state variables over and over > -------------------------------------------------- > > Key: BIT-1547 > URL: https://bro-tracker.atlassian.net/browse/BIT-1547 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Reporter: Justin Azoff > > I happened to notice broctl check on one of our test boxes was slow. traced it to sqlite commits() being very slow. Then noticed that broctl seems to call set_state() with the same key, val over and over again... once for each worker.. so a few thousand sets just to run broctl check. > Changing set_state to > {code} > # Set a dynamic state variable. > def set_state(self, key, val): > key = key.lower() > if self.state.get(key) == val: > return > self.state[key] = val > self.state_store.set(key, val) > {code} > Seemed to mostly fix it, aside from this: > {code} > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > {code} > any idea why that is flipping around like that? > We should possibly add a way for broctl to update state vars without calling commit where it knows it will be setting a large number of state vars in a loop. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From jira at bro-tracker.atlassian.net Mon Mar 7 08:14:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Mon, 7 Mar 2016 10:14:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1547) broctl sets the same state variables over and over In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1547: ------------------------------- Fix Version/s: 2.5 > broctl sets the same state variables over and over > -------------------------------------------------- > > Key: BIT-1547 > URL: https://bro-tracker.atlassian.net/browse/BIT-1547 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Reporter: Justin Azoff > Fix For: 2.5 > > > I happened to notice broctl check on one of our test boxes was slow. traced it to sqlite commits() being very slow. Then noticed that broctl seems to call set_state() with the same key, val over and over again... once for each worker.. so a few thousand sets just to run broctl check. > Changing set_state to > {code} > # Set a dynamic state variable. > def set_state(self, key, val): > key = key.lower() > if self.state.get(key) == val: > return > self.state[key] = val > self.state_store.set(key, val) > {code} > Seemed to mostly fix it, aside from this: > {code} > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > {code} > any idea why that is flipping around like that? > We should possibly add a way for broctl to update state vars without calling commit where it knows it will be setting a large number of state vars in a loop. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From jira at bro-tracker.atlassian.net Mon Mar 7 11:09:01 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Mon, 7 Mar 2016 13:09:01 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24703#comment-24703 ] Johanna Amann commented on BIT-1545: ------------------------------------ This actually is an interesting bug with a few larger implications. I was not aware that setting the skip flag on a connection will completely disable processing in the sense that even byte counts are not updated anymore. While this might be obvious when thinking about it (no reassembly is performed anymore), that means that we might also have to change a few other analyzers to do things differently. Or - what might be preferable - change the way that skipping works, and still let it increase the byte counters. For reference, SetSkip is currently called in these circumstances: - When an analyzer reports an error (in Reporter::AnalyzerError) - by the SSL analyzer when encountering a number of conditions that do not allow it to confinue further parsing - by the SMB analyzer (the old one, so that might not be a problem) - by the login analyzer - by the DNP3 analyzer when encountering problems - by the DCE_RPC analyzer when encountering problems - and by the gridftp script We probably currently get wrong byte counts in all these instances. > SSH connection not recording entire flow correctly > -------------------------------------------------- > > Key: BIT-1545 > URL: https://bro-tracker.atlassian.net/browse/BIT-1545 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.4 > Environment: Ubuntu 14.04 LTS, myricom 10g capture card > Reporter: Jason Carr > Labels: logging > Fix For: 2.5 > > Attachments: ssh-port22.pcap > > > Making a connection out to a server via ssh does not write to conn.log while running with broctl but it does log to weird.log and ssh.log but nothing to conn.log. > While running bro -C -r ssh-port22.pcap, a partial log entry is listed with an incorrect and very low number of packets and bytes. > It was determined that disabling the SSH analyzer gets the correct conn.log output. > Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); > Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it works as expected. > Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From jira at bro-tracker.atlassian.net Mon Mar 7 11:32:01 2016 From: jira at bro-tracker.atlassian.net (Aaron Eppert (JIRA)) Date: Mon, 7 Mar 2016 13:32:01 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24704#comment-24704 ] Aaron Eppert commented on BIT-1545: ----------------------------------- There may be an instance where completely disabling processing still makes sense, in an extreme shunting scenario for instance. A thin wrapper around disable_analyzer would be likely the cleanest way of implementing this with the appropriate desired result, which is why I implemented the suggested change that way. I do that the same for several internal shunting mechanisms. I still want byte counts, clearly, but the analyzer in those instances are no longer helpful. It is definitely a global issue, though. > SSH connection not recording entire flow correctly > -------------------------------------------------- > > Key: BIT-1545 > URL: https://bro-tracker.atlassian.net/browse/BIT-1545 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.4 > Environment: Ubuntu 14.04 LTS, myricom 10g capture card > Reporter: Jason Carr > Labels: logging > Fix For: 2.5 > > Attachments: ssh-port22.pcap > > > Making a connection out to a server via ssh does not write to conn.log while running with broctl but it does log to weird.log and ssh.log but nothing to conn.log. > While running bro -C -r ssh-port22.pcap, a partial log entry is listed with an incorrect and very low number of packets and bytes. > It was determined that disabling the SSH analyzer gets the correct conn.log output. > Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); > Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it works as expected. > Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From jira at bro-tracker.atlassian.net Mon Mar 7 12:33:02 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Mon, 7 Mar 2016 14:33:02 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24705#comment-24705 ] Johanna Amann commented on BIT-1545: ------------------------------------ I talked to Robin - and was mistaken, most of these cases are actually not a problem because the analyzers only disable themselves, not the root-analyzer. We still should fix the current behavior someday - for example by adding a field to the connection history that the size counting was disabled for the rest of this connection. This will potentially become even more interesting with the addition of the netcontrol framework, which also should somehow signal that connections have been shunted (currently, it is not really doing that). > SSH connection not recording entire flow correctly > -------------------------------------------------- > > Key: BIT-1545 > URL: https://bro-tracker.atlassian.net/browse/BIT-1545 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.4 > Environment: Ubuntu 14.04 LTS, myricom 10g capture card > Reporter: Jason Carr > Labels: logging > Fix For: 2.5 > > Attachments: ssh-port22.pcap > > > Making a connection out to a server via ssh does not write to conn.log while running with broctl but it does log to weird.log and ssh.log but nothing to conn.log. > While running bro -C -r ssh-port22.pcap, a partial log entry is listed with an incorrect and very low number of packets and bytes. > It was determined that disabling the SSH analyzer gets the correct conn.log output. > Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); > Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it works as expected. > Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From jira at bro-tracker.atlassian.net Mon Mar 7 12:33:02 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Mon, 7 Mar 2016 14:33:02 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1545: ------------------------------- Status: Merge Request (was: Open) > SSH connection not recording entire flow correctly > -------------------------------------------------- > > Key: BIT-1545 > URL: https://bro-tracker.atlassian.net/browse/BIT-1545 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.4 > Environment: Ubuntu 14.04 LTS, myricom 10g capture card > Reporter: Jason Carr > Labels: logging > Fix For: 2.5 > > Attachments: ssh-port22.pcap > > > Making a connection out to a server via ssh does not write to conn.log while running with broctl but it does log to weird.log and ssh.log but nothing to conn.log. > While running bro -C -r ssh-port22.pcap, a partial log entry is listed with an incorrect and very low number of packets and bytes. > It was determined that disabling the SSH analyzer gets the correct conn.log output. > Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); > Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it works as expected. > Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From jira at bro-tracker.atlassian.net Mon Mar 7 12:33:02 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Mon, 7 Mar 2016 14:33:02 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann reassigned BIT-1545: ---------------------------------- Assignee: Johanna Amann > SSH connection not recording entire flow correctly > -------------------------------------------------- > > Key: BIT-1545 > URL: https://bro-tracker.atlassian.net/browse/BIT-1545 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.4 > Environment: Ubuntu 14.04 LTS, myricom 10g capture card > Reporter: Jason Carr > Assignee: Johanna Amann > Labels: logging > Fix For: 2.5 > > Attachments: ssh-port22.pcap > > > Making a connection out to a server via ssh does not write to conn.log while running with broctl but it does log to weird.log and ssh.log but nothing to conn.log. > While running bro -C -r ssh-port22.pcap, a partial log entry is listed with an incorrect and very low number of packets and bytes. > It was determined that disabling the SSH analyzer gets the correct conn.log output. > Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); > Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it works as expected. > Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From jira at bro-tracker.atlassian.net Mon Mar 7 12:46:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Mon, 7 Mar 2016 14:46:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1548) SendMail parameter is missing from broctl.cfg file in Debian binary installation In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann reassigned BIT-1548: ---------------------------------- Assignee: Johanna Amann > SendMail parameter is missing from broctl.cfg file in Debian binary installation > -------------------------------------------------------------------------------- > > Key: BIT-1548 > URL: https://bro-tracker.atlassian.net/browse/BIT-1548 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.4 > Environment: Date tested: 2016-03-05 > Operating system: Debian 8.2 > Repository: Open Build System > Packages: > bro 2.4.1-0 > bro-core 2.4.1-0 > broctl 2.4.1-0 > libbroccoli 2.4.1-0 > Reporter: Jamshid Karimi > Assignee: Johanna Amann > Labels: file > > The Debian binary packages from Open Build Service have sendmail binary location set to SENDMAIL-NOTFOUND by default but provide no SendMail parameter in broctl.cfg to set the correct location. This means, out of the box, Bro does not send any summary connection reports to the configured email recipient. > For a recent binary installation, I had to manually add the following line to broctl.cfg file to resolve the issue: > SendMail = /usr/sbin/sendmail > Here is the output of broctl config right after installation: > Hint: Run the broctl "deploy" command to get started. > bindir = /opt/bro/bin > broargs = > brobase = /opt/bro > broctlconfigdir = /opt/bro/spool > broport = 47760 > broscriptdir = /opt/bro/share/bro > capstatspath = /opt/bro/bin/capstats > cfgdir = /opt/bro/etc > cflowaddress = > cflowpassword = > cflowuser = > commandtimeout = 60 > commtimeout = 10 > compresscmd = gzip -9 > compressextension = gz > compresslogs = 1 > cron = 0 > croncmd = > debug = 0 > debuglog = /opt/bro/spool/debug.log > env_vars = > havenfs = 0 > helperdir = /opt/bro/share/broctl/scripts/helpers > ipv6comm = 1 > keeplogs = > libdir = /opt/bro/lib > libdirinternal = /opt/bro/lib/broctl > localnetscfg = /opt/bro/etc/networks.cfg > lockfile = /opt/bro/spool/lock > logdir = /opt/bro/logs > logexpireinterval = 0 > logrotationinterval = 3600 > mailalarmsinterval = 86400 > mailalarmsto = root at localhost > mailconnectionsummary = 1 > mailfrom = Big Brother > mailhostupdown = 1 > mailreplyto = > mailsubjectprefix = [Bro] > mailto = root at localhost > makearchivename = /opt/bro/share/broctl/scripts/make-archive-name > memlimit = unlimited > mindiskspace = 5 > nodecfg = /opt/bro/etc/node.cfg > os = linux > pfringclusterid = 0 > pfringclustertype = 4-tuple > pfringfirstappinstance = 0 > pin_command = taskset -c > plugindir = /opt/bro/lib/broctl/plugins > policydir = /opt/bro/share/bro > policydirsiteinstall = /opt/bro/spool/installed-scripts-do-not-touch/site > policydirsiteinstallauto = /opt/bro/spool/installed-scripts-do-not-touch/auto > postprocdir = /opt/bro/share/broctl/scripts/postprocessors > prefixes = local > savetraces = 0 > scriptsdir = /opt/bro/share/broctl/scripts > sendmail = SENDMAIL-NOTFOUND > sigint = 0 > sitepluginpath = > sitepolicymanager = local-manager.bro > sitepolicypath = /opt/bro/share/bro/site > sitepolicystandalone = local.bro > sitepolicyworker = local-worker.bro > spooldir = /opt/bro/spool > standalone = 1 > statefile = /opt/bro/spool/state.db > staticdir = /opt/bro/share/broctl > statsdir = /opt/bro/logs/stats > statslog = /opt/bro/spool/stats.log > statslogenable = 1 > statslogexpireinterval = 0 > statuscmdshowall = 1 > stoptimeout = 60 > test.enabled = 0 > test.foo = 1 > time = > timefmt = %d %b %H:%M:%S > timemachinehost = > timemachineport = 47757/tcp > tmpdir = /opt/bro/spool/tmp > tmpexecdir = /opt/bro/spool/tmp > tracesummary = /opt/bro/bin/trace-summary > version = 1.4 > zoneid = -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From jira at bro-tracker.atlassian.net Mon Mar 7 13:42:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Mon, 7 Mar 2016 15:42:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1545: ------------------------------- Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > SSH connection not recording entire flow correctly > -------------------------------------------------- > > Key: BIT-1545 > URL: https://bro-tracker.atlassian.net/browse/BIT-1545 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.4 > Environment: Ubuntu 14.04 LTS, myricom 10g capture card > Reporter: Jason Carr > Assignee: Johanna Amann > Labels: logging > Fix For: 2.5 > > Attachments: ssh-port22.pcap > > > Making a connection out to a server via ssh does not write to conn.log while running with broctl but it does log to weird.log and ssh.log but nothing to conn.log. > While running bro -C -r ssh-port22.pcap, a partial log entry is listed with an incorrect and very low number of packets and bytes. > It was determined that disabling the SSH analyzer gets the correct conn.log output. > Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); > Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it works as expected. > Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From vallentin at icir.org Mon Mar 7 17:10:09 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Mon, 7 Mar 2016 17:10:09 -0800 Subject: [Bro-Dev] Broker raw throughput In-Reply-To: References: <20160224161050.GE42006@shogun> <20160225161947.GI42006@shogun> Message-ID: <20160308011009.GF33456@shogun> > I have created a ticket for further progress tracking / discussion [1] > as this is clearly not a Bro/Broker problem. Thank you all for > reporting this and all the input you have provided. It's good to see the new commit improves performance. But I want to take again the perspective of Broker, where we're measuring throughput in number of messages per second. Before the changes, we could blast around 80K messages/sec through two remotely connected CAF nodes. After your changes, I am now measuring peak rate of up to 190K/sec on my FreeBSD box. That's more than double. Really cool! But: the benchmark no longer terminates and the server quickly stops getting data, and I would like to know why. Here is the modified actor-system code: // Client using namespace caf; using namespace caf::io; using namespace std; int main(int argc, char** argv) { actor_system_config cfg{argc, argv}; cfg.load(); actor_system system{cfg}; auto server = system.middleman().remote_actor("127.0.0.1", 6666); cerr << "connected to 127.0.0.1:6666, blasting out data" << endl; auto i = 0; scoped_actor self{system}; self->monitor(server); for (auto i = 0; i < 1000000; ++i) self->send(server, i++); self->receive( [&](down_msg const& msg) { cerr << "server terminated" << endl; } ); self->await_all_other_actors_done(); } // Server using namespace caf; using namespace caf::io; using namespace std; using namespace std::chrono; CAF_ALLOW_UNSAFE_MESSAGE_TYPE(high_resolution_clock::time_point) behavior server(event_based_actor* self, int n = 10) { auto counter = make_shared(); auto iterations = make_shared(n); self->send(self, *counter, high_resolution_clock::now()); return { [=](int i) { ++*counter; }, [=](int last, high_resolution_clock::time_point prev) { auto now = high_resolution_clock::now(); auto secs = duration_cast(now - prev); auto rate = (*counter - last) / static_cast(secs.count()); cout << rate << endl; if (rate > 0 && --*iterations == 0) // Count only when we have data. self->quit(); else self->delayed_send(self, seconds(1), *counter, now); } }; } I invoke the server as follows: CPUPROFILE=caf-server.prof ./caf-server --caf#scheduler.scheduler-max-threads=4 And the client like this: CPUPROFILE=caf-client.prof ./caf-client --caf#scheduler.scheduler-max-threads=4 --caf#scheduler.max-throughput=10000 I've tried various parameters for the scheduler throughput, but they do not seem to make a difference. Would you mind taking a look at what's going on here? It looks like the "sender overload protection" you mentioned is not working as expected. I'm also attaching a new gperftools profiler output from the client and server. The server is not too telling, because it was spinning idle for a bit until I ran the client, hence the high CPU load in nanosleep. Looking at the client, it seems that only 67.3% of time is spent in local_actor::resume, which would mean that the runtime adds 33.7% overhead. That's not correct, because gperftools cannot link the second tree on the right properly. (When compiling with -O0 instead of -O3, it looks even worse.) Still, why is intrusive_ptr::get consuming 27.9%? Looking on the left tree, it looks like this workload stresses the allocator heavily: - 20.4% tc_malloc_skip_new_handler - 7% std::vector::insert in the BASP broker - 13.5% CAF serialization (adding two out-edges from basp::instance::write, 5.8 + 7.5) Perhaps this helps you to see some more optimization opportunities. Switching gears to your own performance measurements: it sounded like that you got gains at the order 400% when comparing just raw byte throughput (as opposed to message throughput). Can you give us an intuition how that relates to the throughput measurements we have been doing? Matthias -------------- next part -------------- A non-text attachment was scrubbed... Name: caf-client-freebsd.pdf Type: application/pdf Size: 16369 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160307/dc017798/attachment-0002.pdf -------------- next part -------------- A non-text attachment was scrubbed... Name: caf-server-freebsd.pdf Type: application/pdf Size: 18170 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160307/dc017798/attachment-0003.pdf From jira at bro-tracker.atlassian.net Mon Mar 7 17:54:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Mon, 7 Mar 2016 19:54:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1548) SendMail parameter is missing from broctl.cfg file in Debian binary installation In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1548: ------------------------------- Resolution: Fixed Fix Version/s: 2.5 Status: Closed (was: Open) Thank you for this report - this should be fixed for our nightly builds and will be fixed for the rest of the packages with the next Bro release. > SendMail parameter is missing from broctl.cfg file in Debian binary installation > -------------------------------------------------------------------------------- > > Key: BIT-1548 > URL: https://bro-tracker.atlassian.net/browse/BIT-1548 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.4 > Environment: Date tested: 2016-03-05 > Operating system: Debian 8.2 > Repository: Open Build System > Packages: > bro 2.4.1-0 > bro-core 2.4.1-0 > broctl 2.4.1-0 > libbroccoli 2.4.1-0 > Reporter: Jamshid Karimi > Assignee: Johanna Amann > Labels: file > Fix For: 2.5 > > > The Debian binary packages from Open Build Service have sendmail binary location set to SENDMAIL-NOTFOUND by default but provide no SendMail parameter in broctl.cfg to set the correct location. This means, out of the box, Bro does not send any summary connection reports to the configured email recipient. > For a recent binary installation, I had to manually add the following line to broctl.cfg file to resolve the issue: > SendMail = /usr/sbin/sendmail > Here is the output of broctl config right after installation: > Hint: Run the broctl "deploy" command to get started. > bindir = /opt/bro/bin > broargs = > brobase = /opt/bro > broctlconfigdir = /opt/bro/spool > broport = 47760 > broscriptdir = /opt/bro/share/bro > capstatspath = /opt/bro/bin/capstats > cfgdir = /opt/bro/etc > cflowaddress = > cflowpassword = > cflowuser = > commandtimeout = 60 > commtimeout = 10 > compresscmd = gzip -9 > compressextension = gz > compresslogs = 1 > cron = 0 > croncmd = > debug = 0 > debuglog = /opt/bro/spool/debug.log > env_vars = > havenfs = 0 > helperdir = /opt/bro/share/broctl/scripts/helpers > ipv6comm = 1 > keeplogs = > libdir = /opt/bro/lib > libdirinternal = /opt/bro/lib/broctl > localnetscfg = /opt/bro/etc/networks.cfg > lockfile = /opt/bro/spool/lock > logdir = /opt/bro/logs > logexpireinterval = 0 > logrotationinterval = 3600 > mailalarmsinterval = 86400 > mailalarmsto = root at localhost > mailconnectionsummary = 1 > mailfrom = Big Brother > mailhostupdown = 1 > mailreplyto = > mailsubjectprefix = [Bro] > mailto = root at localhost > makearchivename = /opt/bro/share/broctl/scripts/make-archive-name > memlimit = unlimited > mindiskspace = 5 > nodecfg = /opt/bro/etc/node.cfg > os = linux > pfringclusterid = 0 > pfringclustertype = 4-tuple > pfringfirstappinstance = 0 > pin_command = taskset -c > plugindir = /opt/bro/lib/broctl/plugins > policydir = /opt/bro/share/bro > policydirsiteinstall = /opt/bro/spool/installed-scripts-do-not-touch/site > policydirsiteinstallauto = /opt/bro/spool/installed-scripts-do-not-touch/auto > postprocdir = /opt/bro/share/broctl/scripts/postprocessors > prefixes = local > savetraces = 0 > scriptsdir = /opt/bro/share/broctl/scripts > sendmail = SENDMAIL-NOTFOUND > sigint = 0 > sitepluginpath = > sitepolicymanager = local-manager.bro > sitepolicypath = /opt/bro/share/bro/site > sitepolicystandalone = local.bro > sitepolicyworker = local-worker.bro > spooldir = /opt/bro/spool > standalone = 1 > statefile = /opt/bro/spool/state.db > staticdir = /opt/bro/share/broctl > statsdir = /opt/bro/logs/stats > statslog = /opt/bro/spool/stats.log > statslogenable = 1 > statslogexpireinterval = 0 > statuscmdshowall = 1 > stoptimeout = 60 > test.enabled = 0 > test.foo = 1 > time = > timefmt = %d %b %H:%M:%S > timemachinehost = > timemachineport = 47757/tcp > tmpdir = /opt/bro/spool/tmp > tmpexecdir = /opt/bro/spool/tmp > tracesummary = /opt/bro/bin/trace-summary > version = 1.4 > zoneid = -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From noreply at bro.org Tue Mar 8 00:00:17 2016 From: noreply at bro.org (Merge Tracker) Date: Tue, 8 Mar 2016 00:00:17 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603080800.u2880HKC023957@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ---------- ---------- ------------- ---------- ------------------------------------------------------ BIT-1547 [1] BroControl Justin Azoff - 2016-03-07 2.5 Normal broctl sets the same state variables over and over BIT-1546 [2] Bro Johanna Amann - 2016-03-04 2.5 Normal Please merge topic/johanna/str-functions BIT-1543 [3] Bro Nick Allen Seth Hall 2016-03-03 - Normal Kafka Logger - Writes Bro Logs to Kafka BIT-1507 [4] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- -------------- ---------- ------------------------------------------------------ #52 [5] bro J-Gras [6] 2016-01-18 Fixed matching mail address intel [7] #19 [8] bro-plugins nickwallen [9] 2016-03-07 [BIT-1543] Kafka Logger - Write Bro Logs to Kafka [10] #18 [11] bro-plugins jshlbrd [12] 2016-03-03 SSDP analyzer [13] [1] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [2] BIT-1546 https://bro-tracker.atlassian.net/browse/BIT-1546 [3] BIT-1543 https://bro-tracker.atlassian.net/browse/BIT-1543 [4] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [5] Pull Request #52 https://github.com/bro/bro/pull/52 [6] J-Gras https://github.com/J-Gras [7] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [8] Pull Request #19 https://github.com/bro/bro-plugins/pull/19 [9] nickwallen https://github.com/nickwallen [10] Merge Pull Request #19 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git add-kafka-plugin [11] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [12] jshlbrd https://github.com/jshlbrd [13] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From dominik.charousset at haw-hamburg.de Tue Mar 8 07:09:30 2016 From: dominik.charousset at haw-hamburg.de (Dominik Charousset) Date: Tue, 8 Mar 2016 16:09:30 +0100 Subject: [Bro-Dev] Broker raw throughput In-Reply-To: <20160308011009.GF33456@shogun> References: <20160224161050.GE42006@shogun> <20160225161947.GI42006@shogun> <20160308011009.GF33456@shogun> Message-ID: <2C500E33-8402-474E-9EAE-E1C612A9E2DC@haw-hamburg.de> > the benchmark no longer > terminates and the server quickly stops getting data, and I would like > to know why. I'll have a look at it. > I've tried various parameters for the scheduler throughput, but they do > not seem to make a difference. Would you mind taking a look at what's > going on here? The throughput parameter does not apply to network inputs, so you only modify how many integers per scheduler run the server receives. You could additionally try to tweak caf#middleman.max_consecutive_reads, which configures how many new_data_msg messages a broker receives from the backend in a single shot. It makes sense to have the two separated, because one configures fairness in the scheduling and the other fairness of connection multiplexing. > It looks like the "sender overload protection" you > mentioned is not working as expected. The new feature "merely" allows (CAF) brokers to receive messages from the backend when data is transferred. This basically uplifts TCP's backpressure. When blindly throwing messages at remote actors, there's nothing CAF could do about it. However, the new broker feedback will be one piece in the puzzle when implementing flow control in CAF later on. > I'm also attaching a new gperftools profiler output from the client and > server. The server is not too telling, because it was spinning idle for > a bit until I ran the client, hence the high CPU load in nanosleep. > Looking at the client, it seems that only 67.3% of time is spent in > local_actor::resume, which would mean that the runtime adds 33.7% > overhead. The call to resume() happens in the BASP broker which dumps the messages to its output buffer. So the 67% load include serialization, etc. 28.3% of the remaining load are accumulated in main(). > Still, why is intrusive_ptr::get consuming 27.9%? The 27.9% is accumulating all load down the path, isn't it? intrusive_ptr::get itself simply returns a pointer: https://github.com/actor-framework/actor-framework/blob/d5f43de65c42a74afa4c979ae4f60292f71e371f/libcaf_core/caf/intrusive_ptr.hpp#L128 > Looking on the left tree, it looks like this workload stresses the > allocator heavily: > > - 20.4% tc_malloc_skip_new_handler > - 7% std::vector::insert in the BASP broker > - 13.5% CAF serialization (adding two out-edges from > basp::instance::write, 5.8 + 7.5) Not really surprising. You are sending integers around. Each integer has to be wrapped in a heap-allocated message which gets enqueued to an actor's mailbox. By using many small messages, you basically maximize the messaging overhead. > Switching gears to your own performance measurements: it sounded like > that you got gains at the order 400% when comparing just raw byte > throughput (as opposed to message throughput). Can you give us an > intuition how that relates to the throughput measurements we have been > doing? At the lowest level, a framework like CAF ultimately needs to efficiently manage buffers and events provided by the OS. That's the functionality of recv/send/poll/epoll and friends. That's what I was looking at, since you can't get good performance if you have problems at that level (which, as it turned out, CAF had). Moving a few layers up, some overhead is inherent in a messaging framework. Stressing the heap (see 20% load in tc_malloc_skip_new_handler) when sending many small messages, for example. >From the gperf output (just looking at the client), I don't see that much CPU time spent in CAF itself. If I sum up CPU load from std::vector (6.2%), tcmalloc (20.4%), atomics (8%) and serialization (12.4%), I'm already at 47% out of 70% total for the multiplexer (default_multiplexer::run). Pattern Matching (caf::detail::try_match) cause less than 6% CPU load, so that seems not to be an issue. Serialization has 12% CPU load, which probably mostly results from std::copy (cut out after std::function unfortunately). So, I don't see that many optimization opportunities in these components. Tackling the "many small messages problem" isn't going to be easy. CAF could try to wrap multiple messages from the network into a single heap-allocated storage that is then shipped to an actor as a whole, but this optimization would have a high complexity. That's of course just some thoughts after looking at the gperf output you provided. I'll hopefully have new insights after looking at the termination problem in detail. Dominik From jira at bro-tracker.atlassian.net Tue Mar 8 07:18:03 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 8 Mar 2016 09:18:03 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1547) broctl sets the same state variables over and over In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1547: --------------------------------- Assignee: Justin Azoff > broctl sets the same state variables over and over > -------------------------------------------------- > > Key: BIT-1547 > URL: https://bro-tracker.atlassian.net/browse/BIT-1547 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Reporter: Justin Azoff > Assignee: Justin Azoff > Fix For: 2.5 > > > I happened to notice broctl check on one of our test boxes was slow. traced it to sqlite commits() being very slow. Then noticed that broctl seems to call set_state() with the same key, val over and over again... once for each worker.. so a few thousand sets just to run broctl check. > Changing set_state to > {code} > # Set a dynamic state variable. > def set_state(self, key, val): > key = key.lower() > if self.state.get(key) == val: > return > self.state[key] = val > self.state_store.set(key, val) > {code} > Seemed to mostly fix it, aside from this: > {code} > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > {code} > any idea why that is flipping around like that? > We should possibly add a way for broctl to update state vars without calling commit where it knows it will be setting a large number of state vars in a loop. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From jira at bro-tracker.atlassian.net Tue Mar 8 07:19:02 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 8 Mar 2016 09:19:02 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1546) Please merge topic/johanna/str-functions In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1546: --------------------------------- Assignee: Robin Sommer > Please merge topic/johanna/str-functions > ---------------------------------------- > > Key: BIT-1546 > URL: https://bro-tracker.atlassian.net/browse/BIT-1546 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > Fix For: 2.5 > > > topic/johanna/str-functions replaces a few string functions in Bro with functions provided by the standard operating system libraries. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From jira at bro-tracker.atlassian.net Tue Mar 8 08:03:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 8 Mar 2016 10:03:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1543) Kafka Logger - Writes Bro Logs to Kafka In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1543: --------------------------------- Assignee: Robin Sommer (was: Seth Hall) > Kafka Logger - Writes Bro Logs to Kafka > --------------------------------------- > > Key: BIT-1543 > URL: https://bro-tracker.atlassian.net/browse/BIT-1543 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Nick Allen > Assignee: Robin Sommer > > As part of the Apache Metron project, we needed a way to send Bro logs to Kafka. From my research it seems like this is a common request. I'd rather give this code back to the Bro community than maintain it as part of Apache Metron. > This Bro plugin logs all Bro output to Kafka. Configuring this plugin is as simple as adding the following Bro script. > {{ > @load Bro/Kafka/logs-to-kafka.bro > redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); > redef Kafka::topic_name = "bro"; > redef Kafka::kafka_conf = table( > ["metadata.broker.list"] = "localhost:9092" > ); > }} > This plugin has the following features. > * The user can specify a subset of all logs that should be sent to kafka. For example, to only send conn, http, and dns logs, specify the following. > {{redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); > }} > * Full configurability of Kafka connectivity. Any configuration setting accepted by the librdkafka library can be passed to the plugin to tune how the logs are sent to Kafka. > {{redef Kafka::kafka_conf = table( > ["metadata.broker.list"] = "localhost:9092", > ["client.id"] = "bro" > ); > }} > * The plugin will wait a configurable period of time (for example, 3 seconds) after shutdown to attempt to send any queued messages to Kafka. > {{redef Kafka::max_wait_on_shutdown = 3000; > }} > * There are two message formats to choose from. By default, the standard Bro JSON format is used. There is an alternative 'tagged JSON' format that is provided by the plugin. Currently, all messages are sent to a single Bro topic. This 'tagged JSON' format helps a Kafka consumer distinguish which log stream the message originated from. This format prepends the log stream identifier to the JSON message. > {{{'conn': { ... }} > {'http': { ... }} > {'dns': { ... }}}} > To enable this alternative format, simply specify the following. > {{redef Kafka::tag_json = T;}} -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From jira at bro-tracker.atlassian.net Tue Mar 8 08:06:01 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 8 Mar 2016 10:06:01 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1546) Please merge topic/johanna/str-functions In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1546: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Please merge topic/johanna/str-functions > ---------------------------------------- > > Key: BIT-1546 > URL: https://bro-tracker.atlassian.net/browse/BIT-1546 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > Fix For: 2.5 > > > topic/johanna/str-functions replaces a few string functions in Bro with functions provided by the standard operating system libraries. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From jira at bro-tracker.atlassian.net Tue Mar 8 10:15:00 2016 From: jira at bro-tracker.atlassian.net (Aaron Eppert (JIRA)) Date: Tue, 8 Mar 2016 12:15:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1541) Crash in SocketComm::Run - RemoteSerializer.cc:3493 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24707#comment-24707 ] Aaron Eppert commented on BIT-1541: ----------------------------------- I do have a, rather large, core file associated with this event, again. Attaching it isn't really an option though. > Crash in SocketComm::Run - RemoteSerializer.cc:3493 > --------------------------------------------------- > > Key: BIT-1541 > URL: https://bro-tracker.atlassian.net/browse/BIT-1541 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Aaron Eppert > > This has been happening on a few sensors, both standalone and not. On each sensor, there is broctl cron running as well as periodic polling being the Python interface to the netstats data. > {quote}#0 0x0000000000607d47 in SocketComm::Run (this=0x1) at /mnt/hgfs/src/psdev/bro/src/RemoteSerializer.cc:3493 > #1 0x0000000000608021 in RemoteSerializer::Fork (this=0x2590000) > at /mnt/hgfs/src/psdev/bro/src/RemoteSerializer.cc:687 > #2 0x000000000060813f in RemoteSerializer::Enable (this=0x2590000) > at /mnt/hgfs/src/psdev/bro/src/RemoteSerializer.cc:575 > #3 0x00000000005d52b3 in BifFunc::bro_enable_communication (frame=, BiF_ARGS=) > at bro.bif:4480 > #4 0x00000000005d2cdd in BuiltinFunc::Call (this=0x2ae1180, args=0x16255be0, parent=0x4ada990) > at /mnt/hgfs/src/psdev/bro/src/Func.cc:586 > #5 0x00000000005b7af6 in CallExpr::Eval (this=0x315e900, f=0x4ada990) > at /mnt/hgfs/src/psdev/bro/src/Expr.cc:4544 > #6 0x000000000062b8d4 in ExprStmt::Exec (this=0x315e8b0, f=0x4ada990, flow=@0x7ffe64462c50: FLOW_NEXT) > at /mnt/hgfs/src/psdev/bro/src/Stmt.cc:352 > #7 0x0000000000629b94 in IfStmt::DoExec (this=0x31533c0, f=0x4ada990, v=, > flow=@0x7ffe64462c50: FLOW_NEXT) at /mnt/hgfs/src/psdev/bro/src/Stmt.cc:456 > #8 0x000000000062b8f1 in ExprStmt::Exec (this=0x31533c0, f=0x4ada990, flow=@0x7ffe64462c50: FLOW_NEXT) > at /mnt/hgfs/src/psdev/bro/src/Stmt.cc:356 > #9 0x0000000000629c31 in StmtList::Exec (this=0x31534e0, f=0x4ada990, flow=@0x7ffe64462c50: FLOW_NEXT) > at /mnt/hgfs/src/psdev/bro/src/Stmt.cc:1696 > #10 0x0000000000629c31 in StmtList::Exec (this=0x3153120, f=0x4ada990, flow=@0x7ffe64462c50: FLOW_NEXT) > at /mnt/hgfs/src/psdev/bro/src/Stmt.cc:1696 > #11 0x00000000005decfe in BroFunc::Call (this=0x2743b80, args=, parent=0x0) > at /mnt/hgfs/src/psdev/bro/src/Func.cc:403 > #12 0x000000000059d95a in EventHandler::Call (this=0x2476c80, vl=0x15db0b60, no_remote=no_remote at entry=false) > at /mnt/hgfs/src/psdev/bro/src/EventHandler.cc:130 > #13 0x000000000059cb65 in Dispatch (no_remote=false, this=0x16193120) at /mnt/hgfs/src/psdev/bro/src/Event.h:50 > #14 EventMgr::Dispatch (this=this at entry=0xc07840 ) at /mnt/hgfs/src/psdev/bro/src/Event.cc:111 > #15 0x000000000059cd00 in EventMgr::Drain (this=0xc07840 ) at /mnt/hgfs/src/psdev/bro/src/Event.cc:128 > #16 0x000000000054c659 in main (argc=, argv=) > at /mnt/hgfs/src/psdev/bro/src/main.cc:1147 > {quote} -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From vallentin at icir.org Tue Mar 8 12:54:17 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Tue, 8 Mar 2016 12:54:17 -0800 Subject: [Bro-Dev] Broker raw throughput In-Reply-To: <2C500E33-8402-474E-9EAE-E1C612A9E2DC@haw-hamburg.de> References: <20160224161050.GE42006@shogun> <20160225161947.GI42006@shogun> <20160308011009.GF33456@shogun> <2C500E33-8402-474E-9EAE-E1C612A9E2DC@haw-hamburg.de> Message-ID: <20160308205417.GI33456@shogun> > You could additionally try to tweak > caf#middleman.max_consecutive_reads, which configures how many > new_data_msg messages a broker receives from the backend in a single > shot. It makes sense to have the two separated, because one configures > fairness in the scheduling and the other fairness of connection > multiplexing. Good to know about this tuning knob. I played with a few values, from 1 to 1K, but could not find an improvement by tweaking this value alone. Have you already performed some measurements to find the optimal combination of parameters? > The 27.9% is accumulating all load down the path, isn't it? Yeah, right, I must have confused the absolute vs. cumulative numbers in this case. :-/ > By using many small messages, you basically maximize the messaging > overhead. Exactly. That is the worse-case scenario I'm trying to benchmark :-). > Tackling the "many small messages problem" isn't going to be easy. CAF > could try to wrap multiple messages from the network into a single > heap-allocated storage that is then shipped to an actor as a whole, > but this optimization would have a high complexity. A common strategy to reduce high heap pressure involves custom allocators, and memory pools in particular. Assuming that a single actor produces a fixed number of message types (e.g., <= 10), one could create one memory pool for each message type. What do you think about such a strategy? Matthias From noreply at bro.org Wed Mar 9 00:00:18 2016 From: noreply at bro.org (Merge Tracker) Date: Wed, 9 Mar 2016 00:00:18 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603090800.u2980ITP030885@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1547 [1] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1543 [2] Bro Nick Allen Robin Sommer 2016-03-08 - Normal Kafka Logger - Writes Bro Logs to Kafka BIT-1507 [3] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- -------------- ---------- ----------------------------------------------------- #52 [4] bro J-Gras [5] 2016-01-18 Fixed matching mail address intel [6] #19 [7] bro-plugins nickwallen [8] 2016-03-08 [BIT-1543] Kafka Logger - Write Bro Logs to Kafka [9] #18 [10] bro-plugins jshlbrd [11] 2016-03-03 SSDP analyzer [12] [1] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [2] BIT-1543 https://bro-tracker.atlassian.net/browse/BIT-1543 [3] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [4] Pull Request #52 https://github.com/bro/bro/pull/52 [5] J-Gras https://github.com/J-Gras [6] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [7] Pull Request #19 https://github.com/bro/bro-plugins/pull/19 [8] nickwallen https://github.com/nickwallen [9] Merge Pull Request #19 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git add-kafka-plugin [10] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [11] jshlbrd https://github.com/jshlbrd [12] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From dominik.charousset at haw-hamburg.de Wed Mar 9 02:23:28 2016 From: dominik.charousset at haw-hamburg.de (Dominik Charousset) Date: Wed, 9 Mar 2016 11:23:28 +0100 Subject: [Bro-Dev] Broker raw throughput In-Reply-To: <20160308205417.GI33456@shogun> References: <20160224161050.GE42006@shogun> <20160225161947.GI42006@shogun> <20160308011009.GF33456@shogun> <2C500E33-8402-474E-9EAE-E1C612A9E2DC@haw-hamburg.de> <20160308205417.GI33456@shogun> Message-ID: <887CF7AE-A8BC-425A-A026-7EC022049622@haw-hamburg.de> >> You could additionally try to tweak >> caf#middleman.max_consecutive_reads, which configures how many >> new_data_msg messages a broker receives from the backend in a single >> shot. It makes sense to have the two separated, because one configures >> fairness in the scheduling and the other fairness of connection >> multiplexing. > > Good to know about this tuning knob. I played with a few values, from 1 > to 1K, but could not find an improvement by tweaking this value alone. > Have you already performed some measurements to find the optimal > combination of parameters? I don't think there is an optimal combination for all use cases. You are always trading between fairness and throughput. The question is whether your application needs to stay responsive to multiple clients or if your workload is some form of non-interactive batch processing. Any default value is arbitrary at the end of the day. As long as messages are distributed more-or-less evenly among actors and no actor receives hundreds of messages between scheduling cycles, the parameters don't matter anyway. >> Tackling the "many small messages problem" isn't going to be easy. CAF >> could try to wrap multiple messages from the network into a single >> heap-allocated storage that is then shipped to an actor as a whole, >> but this optimization would have a high complexity. > > A common strategy to reduce high heap pressure involves custom > allocators, and memory pools in particular. Assuming that a single actor > produces a fixed number of message types (e.g., <= 10), one could create > one memory pool for each message type. What do you think about such a > strategy? This is exactly what CAF does. A few years ago, this was absolutely necessary to get decent performance. Recently, however, standard heap allocators were getting much better (at least on Linux). You can build CAF with --no-memory-management to see if it makes a difference on BSD. The optimization I meant is to not wrap each integer in its own message object, but rather make one message which then contains X integers that are transparently interpreted by the receiver as X messages. But this requires some form of "output queue" or lookahead mechanism. Dominik From vallentin at icir.org Wed Mar 9 08:54:53 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Wed, 9 Mar 2016 08:54:53 -0800 Subject: [Bro-Dev] Broker raw throughput In-Reply-To: <887CF7AE-A8BC-425A-A026-7EC022049622@haw-hamburg.de> References: <20160224161050.GE42006@shogun> <20160225161947.GI42006@shogun> <20160308011009.GF33456@shogun> <2C500E33-8402-474E-9EAE-E1C612A9E2DC@haw-hamburg.de> <20160308205417.GI33456@shogun> <887CF7AE-A8BC-425A-A026-7EC022049622@haw-hamburg.de> Message-ID: <20160309165453.GR33456@shogun> > The optimization I meant is to not wrap each integer in its own > message object, but rather make one message which then contains X > integers that are transparently interpreted by the receiver as X > messages. But this requires some form of "output queue" or lookahead > mechanism. I can see that being a nice intrinsic performance gain. At this point, we have too little experience with Broker to warrant such a specific optimization, but I like this "SIMD approach" in general. I already have some ideas regarding transparent compression/coding that could be interesting to explore in the future. Matthias From jira at bro-tracker.atlassian.net Wed Mar 9 13:00:02 2016 From: jira at bro-tracker.atlassian.net (Aaron Eppert (JIRA)) Date: Wed, 9 Mar 2016 15:00:02 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1482) Crash from: "tcmalloc: large alloc" In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Eppert updated BIT-1482: ------------------------------ Resolution: No longer applies Status: Closed (was: Open) > Crash from: "tcmalloc: large alloc" > ----------------------------------- > > Key: BIT-1482 > URL: https://bro-tracker.atlassian.net/browse/BIT-1482 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Aaron Eppert > Attachments: redacted-crash-diag.log.bz2 > > > core.91861 > [New Thread 91861] > [New Thread 91871] > [New Thread 91872] > [New Thread 91873] > [Thread debugging using libthread_db enabled] > Core was generated by `/usr/local/bro/bin/bro -i eth1 -U .status -p broctl -p broctl-live -p local -p'. > Program terminated with signal 11, Segmentation fault. > #0 0x000000000081816b in Serializer::Write (this=0x7ffde1aa2d00, v=35329, tag=0xb752df "stype") at /mnt/hgfs/src/psdev/bro/src/Serializer.h:57 > in /mnt/hgfs/src/psdev/bro/src/Serializer.h > Thread 4 (Thread 0x7fb7ce219700 (LWP 91873)): > #0 0x0000003b8f00ba0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 > #1 0x000000000086f551 in threading::Queue::Get (this=0x3e10c38) at /mnt/hgfs/src/psdev/bro/src/threading/Queue.h:173 > #2 0x000000000086dcfb in threading::MsgThread::RetrieveIn (this=0x3e10c00) at /mnt/hgfs/src/psdev/bro/src/threading/MsgThread.cc:349 > #3 0x000000000086de02 in threading::MsgThread::Run (this=0x3e10c00) at /mnt/hgfs/src/psdev/bro/src/threading/MsgThread.cc:366 > #4 0x000000000086a2c6 in threading::BasicThread::launcher (arg=0x3e10c00) at /mnt/hgfs/src/psdev/bro/src/threading/BasicThread.cc:201 > #5 0x0000003b8f007a51 in start_thread () from /lib64/libpthread.so.0 > #6 0x0000003b8ece89ad in clone () from /lib64/libc.so.6 > Thread 3 (Thread 0x7fb7cec1a700 (LWP 91872)): > #0 0x0000003b8f00ba0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 > #1 0x000000000086f551 in threading::Queue::Get (this=0x3e11838) at /mnt/hgfs/src/psdev/bro/src/threading/Queue.h:173 > #2 0x000000000086dcfb in threading::MsgThread::RetrieveIn (this=0x3e11800) at /mnt/hgfs/src/psdev/bro/src/threading/MsgThread.cc:349 > #3 0x000000000086de02 in threading::MsgThread::Run (this=0x3e11800) at /mnt/hgfs/src/psdev/bro/src/threading/MsgThread.cc:366 > #4 0x000000000086a2c6 in threading::BasicThread::launcher (arg=0x3e11800) at /mnt/hgfs/src/psdev/bro/src/threading/BasicThread.cc:201 > #5 0x0000003b8f007a51 in start_thread () from /lib64/libpthread.so.0 > #6 0x0000003b8ece89ad in clone () from /lib64/libc.so.6 > Thread 2 (Thread 0x7fb7cf61b700 (LWP 91871)): > #0 0x0000003b8f00ba0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 > #1 0x000000000086f551 in threading::Queue::Get (this=0x3e12438) at /mnt/hgfs/src/psdev/bro/src/threading/Queue.h:173 > #2 0x000000000086dcfb in threading::MsgThread::RetrieveIn (this=0x3e12400) at /mnt/hgfs/src/psdev/bro/src/threading/MsgThread.cc:349 > #3 0x000000000086de02 in threading::MsgThread::Run (this=0x3e12400) at /mnt/hgfs/src/psdev/bro/src/threading/MsgThread.cc:366 > #4 0x000000000086a2c6 in threading::BasicThread::launcher (arg=0x3e12400) at /mnt/hgfs/src/psdev/bro/src/threading/BasicThread.cc:201 > #5 0x0000003b8f007a51 in start_thread () from /lib64/libpthread.so.0 > #6 0x0000003b8ece89ad in clone () from /lib64/libc.so.6 > Thread 1 (Thread 0x7fb84fc06800 (LWP 91861)): > #0 0x000000000081816b in Serializer::Write (this=0x7ffde1aa2d00, v=35329, tag=0xb752df "stype") at /mnt/hgfs/src/psdev/bro/src/Serializer.h:57 > #1 0x0000000000817fb4 in SerialObj::DoSerialize (this=0x2c2a400, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/SerialObj.cc:268 > #2 0x00000000007e1be2 in BroObj::DoSerialize (this=0x2c2a400, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Obj.cc:226 > #3 0x00000000008459b4 in BroType::DoSerialize (this=0x2c2a400, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:283 > #4 0x000000000081788a in SerialObj::Serialize (this=0x2c2a400, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/SerialObj.cc:121 > #5 0x0000000000845670 in BroType::Serialize (this=0x2c2a400, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:212 > #6 0x0000000000742c72 in Attributes::DoSerialize (this=0x2c2afc0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Attr.cc:516 > #7 0x000000000081788a in SerialObj::Serialize (this=0x2c2afc0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/SerialObj.cc:121 > #8 0x0000000000742b1b in Attributes::Serialize (this=0x2c2afc0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Attr.cc:500 > #9 0x0000000000848ab5 in TypeDecl::Serialize (this=0x2c05ec0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:929 > #10 0x000000000084a01a in RecordType::DoSerialize (this=0x2aea340, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:1250 > #11 0x000000000081788a in SerialObj::Serialize (this=0x2aea340, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/SerialObj.cc:121 > ... (pattern repeats .... ) > ... > #116924 0x0000000000845670 in BroType::Serialize (this=0x4740480, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:212 > #116925 0x0000000000742c72 in Attributes::DoSerialize (this=0x4808e00, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Attr.cc:516 > #116926 0x000000000081788a in SerialObj::Serialize (this=0x4808e00, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/SerialObj.cc:121 > #116927 0x0000000000742b1b in Attributes::Serialize (this=0x4808e00, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Attr.cc:500 > #116928 0x0000000000848ab5 in TypeDecl::Serialize (this=0x47eae00, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:929 > #116929 0x000000000084a01a in RecordType::DoSerialize (this=0x4847e60, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:1250 > #116930 0x000000000081788a in SerialObj::Serialize (this=0x4847e60, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/SerialObj.cc:121 > #116931 0x0000000000845670 in BroType::Serialize (this=0x4847e60, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:212 > #116932 0x0000000000742c72 in Attributes::DoSerialize (this=0x48081c0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Attr.cc:516 > #116933 0x000000000081788a in SerialObj::Serialize (this=0x48081c0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/SerialObj.cc:121 > #116934 0x0000000000742b1b in Attributes::Serialize (this=0x48081c0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Attr.cc:500 > #116935 0x0000000000848ab5 in TypeDecl::Serialize (this=0x47e81c0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:929 > #116936 0x000000000084a01a in RecordType::DoSerialize (this=0x2aec4a0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:1250 > #116937 0x000000000081788a in SerialObj::Serialize (this=0x2aec4a0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/SerialObj.cc:121 > #116938 0x0000000000845670 in BroType::Serialize (this=0x2aec4a0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:212 > #116939 0x0000000000854a9e in Val::DoSerialize (this=0x6b92760, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Val.cc:188 > #116940 0x00000000008562bc in MutableVal::DoSerialize (this=0x6b92760, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Val.cc:656 > #116941 0x000000000085efb2 in RecordVal::DoSerialize (this=0x6b92760, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Val.cc:2813 > #116942 0x000000000081788a in SerialObj::Serialize (this=0x6b92760, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/SerialObj.cc:121 > #116943 0x0000000000854643 in Val::Serialize (this=0x6b92760, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Val.cc:100 > #116944 0x0000000000854511 in Val::Clone (this=0x6b92760) at /mnt/hgfs/src/psdev/bro/src/Val.cc:83 > #116945 0x00000000007a4d91 in Frame::Clone (this=0x8b612d0) at /mnt/hgfs/src/psdev/bro/src/Frame.cc:78 > #116946 0x0000000000841676 in Trigger::Trigger (this=0x2b79dc0, arg_cond=0x4ae81c0, arg_body=0x4af3600, arg_timeout_stmts=0x0, arg_timeout=0x0, arg_frame=0x8b612d0, arg_is_return=false, arg_location=0x4b4d280) at /mnt/hgfs/src/psdev/bro/src/Trigger.cc:108 > #116947 0x000000000083db0e in WhenStmt::Exec (this=0x4b3eba0, f=0x8b612d0, flow=@0x7ffde1aa3064) at /mnt/hgfs/src/psdev/bro/src/Stmt.cc:2166 > #116948 0x000000000083c17b in StmtList::Exec (this=0x4af4260, f=0x8b612d0, flow=@0x7ffde1aa3064) at /mnt/hgfs/src/psdev/bro/src/Stmt.cc:1764 > #116949 0x000000000083c17b in StmtList::Exec (this=0x4b56540, f=0x8b612d0, flow=@0x7ffde1aa3064) at /mnt/hgfs/src/psdev/bro/src/Stmt.cc:1764 > #116950 0x00000000007a649b in BroFunc::Call (this=0x3099030, args=0x82c33e0, parent=0x0) at /mnt/hgfs/src/psdev/bro/src/Func.cc:386 > #116951 0x000000000077f12e in EventHandler::Call (this=0x3084600, vl=0x82c33e0, no_remote=false) at /mnt/hgfs/src/psdev/bro/src/EventHandler.cc:80 > #116952 0x0000000000732965 in Event::Dispatch (this=0xb5004e0, no_remote=false) at /mnt/hgfs/src/psdev/bro/src/Event.h:50 > #116953 0x000000000077e85d in EventMgr::Dispatch (this=0xf66ee0) at /mnt/hgfs/src/psdev/bro/src/Event.cc:111 > #116954 0x000000000077e968 in EventMgr::Drain (this=0xf66ee0) at /mnt/hgfs/src/psdev/bro/src/Event.cc:128 > #116955 0x00000000007ddd66 in net_packet_dispatch (t=1442838074.400739, hdr=0x4d73140, pkt=0x7fb7db8622fc
, hdr_size=14, src_ps=0x4d73000) at /mnt/hgfs/src/psdev/bro/src/Net.cc:278 > #116956 0x0000000000af1ed6 in iosource::PktSrc::Process (this=0x4d73000) at /mnt/hgfs/src/psdev/bro/src/iosource/PktSrc.cc:411 > #116957 0x00000000007ddf6f in net_run () at /mnt/hgfs/src/psdev/bro/src/Net.cc:320 > #116958 0x00000000007319aa in main (argc=18, argv=0x7ffde1aa3af8) at /mnt/hgfs/src/psdev/bro/src/main.cc:1200 > ==== No reporter.log > ==== stderr.log > internal warning in /usr/local/bro/share/bro/base/frameworks/control/./main.bro, line 1: Discarded extraneous Broxygen comment: MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > internal warning in /usr/local/bro/share/bro/base/frameworks/control/./main.bro, line 1: Discarded extraneous Broxygen comment: GNU General Public License for more details. > internal warning in /usr/local/bro/share/bro/base/frameworks/control/./main.bro, line 1: Discarded extraneous Broxygen comment: You should have received a copy of the GNU General Public License > internal warning in /usr/local/bro/share/bro/base/frameworks/control/./main.bro, line 1: Discarded extraneous Broxygen comment: along with tcplog. If not, see . > listening on eth1, capture length 65535 bytes > processing suspended > processing continued > tcmalloc: large alloc 1562509312 bytes == 0x498f0000 @ 0x7fb85004b4ac 0x7fb85006b22c 0x73b0e5 0x815270 0x81627e 0x7437f8 0x742ddd 0x81788a 0x742b1b 0x848ab5 0x84a01a 0x81788a 0x845670 0x848b3b 0x84a01a 0x81788a 0x845670 0x846db0 0x84759e 0x81788a 0x845670 0x742c72 0x81788a 0x742b1b 0x848ab5 0x84a01a 0x81788a 0x845670 0x742c72 0x81788a 0x742b1b > /usr/local/bro/share/broctl/scripts/run-bro: line 85: 91861 Segmentation fault (core dumped) nohup ${pin_command} $pin_cpu $mybro "$@" > ---- > (gdb) frame 0 > #0 0x000000000081816b in Serializer::Write (this=0x7ffde1aa2d00, v=35329, tag=0xb752df "stype") > at /mnt/hgfs/src/psdev/bro/src/Serializer.h:57 > 57 DECLARE_IO(uint16) > (gdb) print *this > $8 = {_vptr.Serializer = 0xb7dc50, static MAGIC = 1112691540, static DATA_FORMAT_VERSION = 25, io = 0x0, format = 0x89def00, > current_cache = 0x0, error_descr = 0x0} > (gdb) print *this > $10 = {_vptr.Serializer = 0xb7dc50, static MAGIC = 1112691540, static DATA_FORMAT_VERSION = 25, io = 0x0, > format = 0x89def00, current_cache = 0x0, error_descr = 0x0} > (gdb) print *this->format > $11 = {_vptr.SerializationFormat = 0xb74dd0, static INITIAL_SIZE = 65536, static GROWTH_FACTOR = 2.5, > output = 0x498f0000 "\001", output_size = 1562499968, output_pos = 852829181, input = 0x0, input_len = 0, input_pos = 0, > bytes_written = 852829181, bytes_read = 0} > The stack trace and the problem seems to be similar to: > http://mailman.icsi.berkeley.edu/pipermail/bro/2015-March/008241.html -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-012#72000) From noreply at bro.org Thu Mar 10 00:00:18 2016 From: noreply at bro.org (Merge Tracker) Date: Thu, 10 Mar 2016 00:00:18 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603100800.u2A80Ixh017316@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1547 [1] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1543 [2] Bro Nick Allen Robin Sommer 2016-03-08 - Normal Kafka Logger - Writes Bro Logs to Kafka BIT-1507 [3] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- -------------- ---------- ----------------------------------------------------- #52 [4] bro J-Gras [5] 2016-01-18 Fixed matching mail address intel [6] #19 [7] bro-plugins nickwallen [8] 2016-03-09 [BIT-1543] Kafka Logger - Write Bro Logs to Kafka [9] #18 [10] bro-plugins jshlbrd [11] 2016-03-03 SSDP analyzer [12] [1] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [2] BIT-1543 https://bro-tracker.atlassian.net/browse/BIT-1543 [3] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [4] Pull Request #52 https://github.com/bro/bro/pull/52 [5] J-Gras https://github.com/J-Gras [6] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [7] Pull Request #19 https://github.com/bro/bro-plugins/pull/19 [8] nickwallen https://github.com/nickwallen [9] Merge Pull Request #19 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git add-kafka-plugin [10] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [11] jshlbrd https://github.com/jshlbrd [12] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Thu Mar 10 07:45:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 10 Mar 2016 09:45:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1498) add '-q' to ssh execution in ssh_runner.py In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer reassigned BIT-1498: ---------------------------------- Assignee: (was: Daniel Thayer) > add '-q' to ssh execution in ssh_runner.py > ------------------------------------------ > > Key: BIT-1498 > URL: https://bro-tracker.atlassian.net/browse/BIT-1498 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Affects Versions: 2.4 > Reporter: scampbell > Priority: Trivial > Labels: broctl > Fix For: 2.5 > > > When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. > The patch is trivial: > --- a/BroControl/ssh_runner.py > +++ b/BroControl/ssh_runner.py > @@ -108,6 +108,7 @@ class SSHMaster: > self.base_cmd = [ > "ssh", > "-o", "BatchMode=yes", > + "-q", > host, > ] > self.need_connect = True -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 10 08:07:00 2016 From: jira at bro-tracker.atlassian.net (Jon Schipp (JIRA)) Date: Thu, 10 Mar 2016 10:07:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1498) add '-q' to ssh execution in ssh_runner.py In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Schipp reassigned BIT-1498: ------------------------------- Assignee: Jon Schipp > add '-q' to ssh execution in ssh_runner.py > ------------------------------------------ > > Key: BIT-1498 > URL: https://bro-tracker.atlassian.net/browse/BIT-1498 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Affects Versions: 2.4 > Reporter: scampbell > Assignee: Jon Schipp > Priority: Trivial > Labels: broctl > Fix For: 2.5 > > > When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. > The patch is trivial: > --- a/BroControl/ssh_runner.py > +++ b/BroControl/ssh_runner.py > @@ -108,6 +108,7 @@ class SSHMaster: > self.base_cmd = [ > "ssh", > "-o", "BatchMode=yes", > + "-q", > host, > ] > self.need_connect = True -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 10 08:10:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 10 Mar 2016 10:10:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1549) broctl top command doesn't work on OS X 10.10 or newer In-Reply-To: References: Message-ID: Daniel Thayer created BIT-1549: ---------------------------------- Summary: broctl top command doesn't work on OS X 10.10 or newer Key: BIT-1549 URL: https://bro-tracker.atlassian.net/browse/BIT-1549 Project: Bro Issue Tracker Issue Type: Task Components: BroControl Reporter: Daniel Thayer On OS X Mavericks, the broctl top command was working, but on Yosemite (and El Capitan), it no longer works. The reason is that the "-stats vprvt" option of the top command always prints "N/A". -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 10 08:11:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 10 Mar 2016 10:11:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1549) broctl top command doesn't work on OS X 10.10 or newer In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1549: ------------------------------- Fix Version/s: 2.5 > broctl top command doesn't work on OS X 10.10 or newer > ------------------------------------------------------ > > Key: BIT-1549 > URL: https://bro-tracker.atlassian.net/browse/BIT-1549 > Project: Bro Issue Tracker > Issue Type: Task > Components: BroControl > Reporter: Daniel Thayer > Fix For: 2.5 > > > On OS X Mavericks, the broctl top command was working, but on Yosemite > (and El Capitan), it no longer works. The reason is that the > "-stats vprvt" option of the top command always prints "N/A". -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 10 08:13:02 2016 From: jira at bro-tracker.atlassian.net (Jon Schipp (JIRA)) Date: Thu, 10 Mar 2016 10:13:02 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1498) add '-q' to ssh execution in ssh_runner.py In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Schipp updated BIT-1498: ---------------------------- Description: When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. The patch is trivial: --- a/BroControl/ssh_runner.py +++ b/BroControl/ssh_runner.py @@ -108,6 +108,7 @@ class SSHMaster: self.base_cmd = [ "ssh", "-o", "BatchMode=yes", + "-q", host, ] self.need_connect = True was: When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. *test* The patch is trivial: --- a/BroControl/ssh_runner.py +++ b/BroControl/ssh_runner.py @@ -108,6 +108,7 @@ class SSHMaster: self.base_cmd = [ "ssh", "-o", "BatchMode=yes", + "-q", host, ] self.need_connect = True > add '-q' to ssh execution in ssh_runner.py > ------------------------------------------ > > Key: BIT-1498 > URL: https://bro-tracker.atlassian.net/browse/BIT-1498 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Affects Versions: 2.4 > Reporter: scampbell > Assignee: Jon Schipp > Priority: Trivial > Labels: broctl > Fix For: 2.5 > > > When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. > The patch is trivial: > --- a/BroControl/ssh_runner.py > +++ b/BroControl/ssh_runner.py > @@ -108,6 +108,7 @@ class SSHMaster: > self.base_cmd = [ > "ssh", > "-o", "BatchMode=yes", > + "-q", > host, > ] > self.need_connect = True -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 10 08:13:02 2016 From: jira at bro-tracker.atlassian.net (Jon Schipp (JIRA)) Date: Thu, 10 Mar 2016 10:13:02 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1498) add '-q' to ssh execution in ssh_runner.py In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Schipp updated BIT-1498: ---------------------------- Description: When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. *test* The patch is trivial: --- a/BroControl/ssh_runner.py +++ b/BroControl/ssh_runner.py @@ -108,6 +108,7 @@ class SSHMaster: self.base_cmd = [ "ssh", "-o", "BatchMode=yes", + "-q", host, ] self.need_connect = True was: When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. The patch is trivial: --- a/BroControl/ssh_runner.py +++ b/BroControl/ssh_runner.py @@ -108,6 +108,7 @@ class SSHMaster: self.base_cmd = [ "ssh", "-o", "BatchMode=yes", + "-q", host, ] self.need_connect = True > add '-q' to ssh execution in ssh_runner.py > ------------------------------------------ > > Key: BIT-1498 > URL: https://bro-tracker.atlassian.net/browse/BIT-1498 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Affects Versions: 2.4 > Reporter: scampbell > Assignee: Jon Schipp > Priority: Trivial > Labels: broctl > Fix For: 2.5 > > > When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. > *test* > The patch is trivial: > --- a/BroControl/ssh_runner.py > +++ b/BroControl/ssh_runner.py > @@ -108,6 +108,7 @@ class SSHMaster: > self.base_cmd = [ > "ssh", > "-o", "BatchMode=yes", > + "-q", > host, > ] > self.need_connect = True -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 10 08:27:00 2016 From: jira at bro-tracker.atlassian.net (Vern Paxson (JIRA)) Date: Thu, 10 Mar 2016 10:27:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24800#comment-24800 ] Vern Paxson commented on BIT-1545: ---------------------------------- I'm definitely a fan of at least adding transparency that the value has not been properly tracked! It would also be good to understand in what shunting situations one can still afford to track such values; and I would hope that even if there's full (blind) shunting, the FIN/RSTs that terminate the connection are still captured, so one can make a guess based on sequence numbers. (Likewise, we'd want this annotated as a guess and not a directly measured value.) > SSH connection not recording entire flow correctly > -------------------------------------------------- > > Key: BIT-1545 > URL: https://bro-tracker.atlassian.net/browse/BIT-1545 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.4 > Environment: Ubuntu 14.04 LTS, myricom 10g capture card > Reporter: Jason Carr > Assignee: Johanna Amann > Labels: logging > Fix For: 2.5 > > Attachments: ssh-port22.pcap > > > Making a connection out to a server via ssh does not write to conn.log while running with broctl but it does log to weird.log and ssh.log but nothing to conn.log. > While running bro -C -r ssh-port22.pcap, a partial log entry is listed with an incorrect and very low number of packets and bytes. > It was determined that disabling the SSH analyzer gets the correct conn.log output. > Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); > Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it works as expected. > Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 10 09:33:01 2016 From: jira at bro-tracker.atlassian.net (Jon Schipp (JIRA)) Date: Thu, 10 Mar 2016 11:33:01 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1540) Ifconfig is hardcoded in BroControl In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24801#comment-24801 ] Jon Schipp commented on BIT-1540: --------------------------------- I added support for the ip tool in branch topic/jschipp/broctl-ip-support > Ifconfig is hardcoded in BroControl > ----------------------------------- > > Key: BIT-1540 > URL: https://bro-tracker.atlassian.net/browse/BIT-1540 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Jon Schipp > Fix For: 2.5 > > > From the mailing list: > {quote} > Hi Folks, > On later versions of Linux distros iproute2 replaces ifconfig with ip > Starting at line 601 at > https://github.com/bro/broctl/blob/master/BroControl/config.py > It looks like ifconfig is hard-written into the logic. Probably needs a > patch to check for the ip command. > Cheers, > Harry > {quote} > We should probably check for the presence of the ip utility and use that, if present. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 10 09:34:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Thu, 10 Mar 2016 11:34:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24802#comment-24802 ] Johanna Amann commented on BIT-1545: ------------------------------------ Yes, that was pretty much the outcome of our discussion. The SSH case is fixed now (the merged patch only removes the SSH analyzer - all counting stays intact), and I was mistaken about the other protocols, they do not do it. For external shunting (which is not part of Bro yet, but will be soon), we have a way to get some information from the switches (if they support that). I just have to get that into conn log. We also discussed that adding a character to the connection history for "connection was shunted" would be a good idea, to indicate that the numbers are only a guess. > SSH connection not recording entire flow correctly > -------------------------------------------------- > > Key: BIT-1545 > URL: https://bro-tracker.atlassian.net/browse/BIT-1545 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.4 > Environment: Ubuntu 14.04 LTS, myricom 10g capture card > Reporter: Jason Carr > Assignee: Johanna Amann > Labels: logging > Fix For: 2.5 > > Attachments: ssh-port22.pcap > > > Making a connection out to a server via ssh does not write to conn.log while running with broctl but it does log to weird.log and ssh.log but nothing to conn.log. > While running bro -C -r ssh-port22.pcap, a partial log entry is listed with an incorrect and very low number of packets and bytes. > It was determined that disabling the SSH analyzer gets the correct conn.log output. > Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); > Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it works as expected. > Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 10 10:58:00 2016 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Thu, 10 Mar 2016 12:58:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24803#comment-24803 ] Justin Azoff commented on BIT-1545: ----------------------------------- The other thing to keep in mind is how this affects missed_bytes and capture loss. When I do shunting with the Arista I allow control packets through which lets most counters work, the only issue is the missed_bytes ends up being huge because bro thinks we are dropping all the packets. > SSH connection not recording entire flow correctly > -------------------------------------------------- > > Key: BIT-1545 > URL: https://bro-tracker.atlassian.net/browse/BIT-1545 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master, 2.4 > Environment: Ubuntu 14.04 LTS, myricom 10g capture card > Reporter: Jason Carr > Assignee: Johanna Amann > Labels: logging > Fix For: 2.5 > > Attachments: ssh-port22.pcap > > > Making a connection out to a server via ssh does not write to conn.log while running with broctl but it does log to weird.log and ssh.log but nothing to conn.log. > While running bro -C -r ssh-port22.pcap, a partial log entry is listed with an incorrect and very low number of packets and bytes. > It was determined that disabling the SSH analyzer gets the correct conn.log output. > Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); > Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it works as expected. > Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 10 13:40:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 10 Mar 2016 15:40:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1543) Kafka Logger - Writes Bro Logs to Kafka In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1543: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Kafka Logger - Writes Bro Logs to Kafka > --------------------------------------- > > Key: BIT-1543 > URL: https://bro-tracker.atlassian.net/browse/BIT-1543 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Nick Allen > Assignee: Robin Sommer > > As part of the Apache Metron project, we needed a way to send Bro logs to Kafka. From my research it seems like this is a common request. I'd rather give this code back to the Bro community than maintain it as part of Apache Metron. > This Bro plugin logs all Bro output to Kafka. Configuring this plugin is as simple as adding the following Bro script. > {{ > @load Bro/Kafka/logs-to-kafka.bro > redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); > redef Kafka::topic_name = "bro"; > redef Kafka::kafka_conf = table( > ["metadata.broker.list"] = "localhost:9092" > ); > }} > This plugin has the following features. > * The user can specify a subset of all logs that should be sent to kafka. For example, to only send conn, http, and dns logs, specify the following. > {{redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); > }} > * Full configurability of Kafka connectivity. Any configuration setting accepted by the librdkafka library can be passed to the plugin to tune how the logs are sent to Kafka. > {{redef Kafka::kafka_conf = table( > ["metadata.broker.list"] = "localhost:9092", > ["client.id"] = "bro" > ); > }} > * The plugin will wait a configurable period of time (for example, 3 seconds) after shutdown to attempt to send any queued messages to Kafka. > {{redef Kafka::max_wait_on_shutdown = 3000; > }} > * There are two message formats to choose from. By default, the standard Bro JSON format is used. There is an alternative 'tagged JSON' format that is provided by the plugin. Currently, all messages are sent to a single Bro topic. This 'tagged JSON' format helps a Kafka consumer distinguish which log stream the message originated from. This format prepends the log stream identifier to the JSON message. > {{{'conn': { ... }} > {'http': { ... }} > {'dns': { ... }}}} > To enable this alternative format, simply specify the following. > {{redef Kafka::tag_json = T;}} -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From gc355804 at ohio.edu Thu Mar 10 13:49:34 2016 From: gc355804 at ohio.edu (Clark, Gilbert) Date: Thu, 10 Mar 2016 21:49:34 +0000 Subject: [Bro-Dev] Fw: Broker raw throughput In-Reply-To: <20160310184610.GG76756@samurai.ICIR.org> References: <20160224161050.GE42006@shogun> <20160225161947.GI42006@shogun> <20160308011009.GF33456@shogun> <2C500E33-8402-474E-9EAE-E1C612A9E2DC@haw-hamburg.de> <20160308205417.GI33456@shogun> <887CF7AE-A8BC-425A-A026-7EC022049622@haw-hamburg.de> , <20160310184610.GG76756@samurai.ICIR.org> Message-ID: Hi: Forwarding reply to the bro-dev list: original was mistakenly posted elsewhere (sorry about that). Leaving original message content inline for context. ________________________________________ From: Matthias Vallentin on behalf of Matthias Vallentin Sent: Thursday, March 10, 2016 1:46 PM To: Clark, Gilbert Subject: Re: [Bro-Dev] Broker raw throughput > > Sorry if this is a stupid question, but what are the performance > > requirements for broker, exactly? > >We have too little experience to tell what we need Right, this was my question. The less that broker's requirements are documented and understood, the more difficult it becomes to evaluate whether or not broker will fit with an intended use-case, I think. To me, the performance numbers themselves don't matter as much as managing expectations does: should I *expect* to be able to pass all of my events through broker? >and where we hit bottlenecks. > > This is why we compare a worst-case scenario across >multiple libraries and see where we find low-hanging fruits for >optimization potential. Got it. I think that's what confused me ... >> That's *good enough* for most things, but it's also still quite >> possible to do better: I've built DSP applications on top of DPDK that >> do fine with ~14 Mpps, which is itself pretty slow when compared to >> results reported by e.g. [2]. > >It's not about going as fast as possible. We're looking to achieve good >performance in the common case, *without* reducing a high level of >abstraction. ... because some of those messaging libraries operate at different levels of abstraction than others, which is going to drive the performance to some extent ... > >> Realistically, depending on what broker is intended to support, maybe >> 200k messages / second is fine: > >Agreed. At this point, this rate is certainly fine for our >worst-case-single-int-per-message benchmarks. > >> TL/DR: I'm of the opinion that optimization is fun, but I also would >> feel kind of bad watching CAF go too far down a (very scary) rabbit >> hole just to support one (albeit very large and rather cool) >> application ... > >I don't think we want to do that either, this must be a >misunderstanding. The optimizations we've been looking at are >application-independent. ... so it looks like it was indeed a misunderstanding on my part. Sorry about that. Trying to express things a slightly different way, I was concerned that the different numbers from the different libraries were being interpreted as an apples-to-apples comparison. Modifying CAF to achieve the same results as e.g. 0mq would, at some point and in some way, eventually require modifying CAF to be more like 0mq. I don't think that would be good, because 0mq and CAF aren't (and shouldn't be, in my humble opinion) the same thing. > We have looked at a very specific workload to >bound worst-case performance. I'm very happy with the recent >improvements that will ship with CAF 0.15. Definitely. Performance improvements are always good :) Cheers, Gilbert From noreply at bro.org Fri Mar 11 00:00:18 2016 From: noreply at bro.org (Merge Tracker) Date: Fri, 11 Mar 2016 00:00:18 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603110800.u2B80IhM018722@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1547 [1] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- -------------- ---------- ----------------------------------------------------- #52 [3] bro J-Gras [4] 2016-01-18 Fixed matching mail address intel [5] #19 [6] bro-plugins nickwallen [7] 2016-03-09 [BIT-1543] Kafka Logger - Write Bro Logs to Kafka [8] #18 [9] bro-plugins jshlbrd [10] 2016-03-03 SSDP analyzer [11] [1] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] Pull Request #52 https://github.com/bro/bro/pull/52 [4] J-Gras https://github.com/J-Gras [5] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [6] Pull Request #19 https://github.com/bro/bro-plugins/pull/19 [7] nickwallen https://github.com/nickwallen [8] Merge Pull Request #19 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git add-kafka-plugin [9] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [10] jshlbrd https://github.com/jshlbrd [11] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Fri Mar 11 08:30:00 2016 From: jira at bro-tracker.atlassian.net (Jon Schipp (JIRA)) Date: Fri, 11 Mar 2016 10:30:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1498) add '-q' to ssh execution in ssh_runner.py In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24804#comment-24804 ] Jon Schipp commented on BIT-1498: --------------------------------- I think the ssh messages are helpful in debugging problems but you're right we probably don't want to see the banner. Some notes: The banner prints when sshd_config is set to use the Banner option. Banner happens before authentication. The motd is not printed when the PrintMotd option is used. root at manager:~# broctl start starting manager ... starting proxy-1 ... starting worker-1 ... starting worker-2 ... This BANNER is displaying /etc/issue.net Ubuntu 14.04.1 LTS Stopping sshd on node1 will show banner and the informational ssh messages: root at manager:~# broctl start manager still running proxy-1 still running ssh: connect to host 10.1.1.20 port 22: Connection refused This BANNER is displaying /etc/issue.net Ubuntu 14.04.1 LTS Error: cannot connect to worker-1 worker-2 still running Stopping sshd on node1 while -q is set in ssh_runner.py yields a "Error: cannot connect", not the ssh errors. root at manager:~# broctl start ... Error: cannot connect to worker-1 worker-2 still running Shutting down the node will yield root at manager:~# broctl start ... ssh: connect to host 10.1.1.20 port 22: No route to host Error: cannot connect to worker-1 > add '-q' to ssh execution in ssh_runner.py > ------------------------------------------ > > Key: BIT-1498 > URL: https://bro-tracker.atlassian.net/browse/BIT-1498 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Affects Versions: 2.4 > Reporter: scampbell > Assignee: Jon Schipp > Priority: Trivial > Labels: broctl > Fix For: 2.5 > > > When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. > The patch is trivial: > --- a/BroControl/ssh_runner.py > +++ b/BroControl/ssh_runner.py > @@ -108,6 +108,7 @@ class SSHMaster: > self.base_cmd = [ > "ssh", > "-o", "BatchMode=yes", > + "-q", > host, > ] > self.need_connect = True -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 11 08:31:00 2016 From: jira at bro-tracker.atlassian.net (Jon Schipp (JIRA)) Date: Fri, 11 Mar 2016 10:31:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1498) add '-q' to ssh execution in ssh_runner.py In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24805#comment-24805 ] Jon Schipp commented on BIT-1498: --------------------------------- Using -o LogLevel=error will suppress the banner but still print the error messages. A happy medium I say. Going to push a new branch with it > add '-q' to ssh execution in ssh_runner.py > ------------------------------------------ > > Key: BIT-1498 > URL: https://bro-tracker.atlassian.net/browse/BIT-1498 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Affects Versions: 2.4 > Reporter: scampbell > Assignee: Jon Schipp > Priority: Trivial > Labels: broctl > Fix For: 2.5 > > > When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. > The patch is trivial: > --- a/BroControl/ssh_runner.py > +++ b/BroControl/ssh_runner.py > @@ -108,6 +108,7 @@ class SSHMaster: > self.base_cmd = [ > "ssh", > "-o", "BatchMode=yes", > + "-q", > host, > ] > self.need_connect = True -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 11 09:05:00 2016 From: jira at bro-tracker.atlassian.net (Jon Schipp (JIRA)) Date: Fri, 11 Mar 2016 11:05:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1498) add '-q' to ssh execution in ssh_runner.py In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24806#comment-24806 ] Jon Schipp commented on BIT-1498: --------------------------------- [~dnthayer] I pushed topic/jschipp/broctl-quiet-ssh-banner > add '-q' to ssh execution in ssh_runner.py > ------------------------------------------ > > Key: BIT-1498 > URL: https://bro-tracker.atlassian.net/browse/BIT-1498 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Affects Versions: 2.4 > Reporter: scampbell > Assignee: Jon Schipp > Priority: Trivial > Labels: broctl > Fix For: 2.5 > > > When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. > The patch is trivial: > --- a/BroControl/ssh_runner.py > +++ b/BroControl/ssh_runner.py > @@ -108,6 +108,7 @@ class SSHMaster: > self.base_cmd = [ > "ssh", > "-o", "BatchMode=yes", > + "-q", > host, > ] > self.need_connect = True -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 11 09:08:02 2016 From: jira at bro-tracker.atlassian.net (Jon Schipp (JIRA)) Date: Fri, 11 Mar 2016 11:08:02 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1540) Ifconfig is hardcoded in BroControl In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Schipp reassigned BIT-1540: ------------------------------- Assignee: Jon Schipp > Ifconfig is hardcoded in BroControl > ----------------------------------- > > Key: BIT-1540 > URL: https://bro-tracker.atlassian.net/browse/BIT-1540 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Jon Schipp > Fix For: 2.5 > > > From the mailing list: > {quote} > Hi Folks, > On later versions of Linux distros iproute2 replaces ifconfig with ip > Starting at line 601 at > https://github.com/bro/broctl/blob/master/BroControl/config.py > It looks like ifconfig is hard-written into the logic. Probably needs a > patch to check for the ip command. > Cheers, > Harry > {quote} > We should probably check for the presence of the ip utility and use that, if present. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 11 09:08:02 2016 From: jira at bro-tracker.atlassian.net (Jon Schipp (JIRA)) Date: Fri, 11 Mar 2016 11:08:02 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1540) Ifconfig is hardcoded in BroControl In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Schipp updated BIT-1540: ---------------------------- Status: Merge Request (was: Open) Assignee: (was: Jon Schipp) > Ifconfig is hardcoded in BroControl > ----------------------------------- > > Key: BIT-1540 > URL: https://bro-tracker.atlassian.net/browse/BIT-1540 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Reporter: Johanna Amann > Fix For: 2.5 > > > From the mailing list: > {quote} > Hi Folks, > On later versions of Linux distros iproute2 replaces ifconfig with ip > Starting at line 601 at > https://github.com/bro/broctl/blob/master/BroControl/config.py > It looks like ifconfig is hard-written into the logic. Probably needs a > patch to check for the ip command. > Cheers, > Harry > {quote} > We should probably check for the presence of the ip utility and use that, if present. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 11 09:08:02 2016 From: jira at bro-tracker.atlassian.net (Jon Schipp (JIRA)) Date: Fri, 11 Mar 2016 11:08:02 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1498) add '-q' to ssh execution in ssh_runner.py In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Schipp reassigned BIT-1498: ------------------------------- Assignee: Jon Schipp > add '-q' to ssh execution in ssh_runner.py > ------------------------------------------ > > Key: BIT-1498 > URL: https://bro-tracker.atlassian.net/browse/BIT-1498 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Affects Versions: 2.4 > Reporter: scampbell > Assignee: Jon Schipp > Priority: Trivial > Labels: broctl > Fix For: 2.5 > > > When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. > The patch is trivial: > --- a/BroControl/ssh_runner.py > +++ b/BroControl/ssh_runner.py > @@ -108,6 +108,7 @@ class SSHMaster: > self.base_cmd = [ > "ssh", > "-o", "BatchMode=yes", > + "-q", > host, > ] > self.need_connect = True -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 11 09:08:01 2016 From: jira at bro-tracker.atlassian.net (Jon Schipp (JIRA)) Date: Fri, 11 Mar 2016 11:08:01 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1498) add '-q' to ssh execution in ssh_runner.py In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Schipp updated BIT-1498: ---------------------------- Status: Merge Request (was: Open) Assignee: (was: Jon Schipp) > add '-q' to ssh execution in ssh_runner.py > ------------------------------------------ > > Key: BIT-1498 > URL: https://bro-tracker.atlassian.net/browse/BIT-1498 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Affects Versions: 2.4 > Reporter: scampbell > Priority: Trivial > Labels: broctl > Fix For: 2.5 > > > When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. > The patch is trivial: > --- a/BroControl/ssh_runner.py > +++ b/BroControl/ssh_runner.py > @@ -108,6 +108,7 @@ class SSHMaster: > self.base_cmd = [ > "ssh", > "-o", "BatchMode=yes", > + "-q", > host, > ] > self.need_connect = True -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 11 09:09:00 2016 From: jira at bro-tracker.atlassian.net (Jon Schipp (JIRA)) Date: Fri, 11 Mar 2016 11:09:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1540) Ifconfig is hardcoded in BroControl In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Schipp reassigned BIT-1540: ------------------------------- Assignee: (was: Jon Schipp) > Ifconfig is hardcoded in BroControl > ----------------------------------- > > Key: BIT-1540 > URL: https://bro-tracker.atlassian.net/browse/BIT-1540 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Reporter: Johanna Amann > Fix For: 2.5 > > > From the mailing list: > {quote} > Hi Folks, > On later versions of Linux distros iproute2 replaces ifconfig with ip > Starting at line 601 at > https://github.com/bro/broctl/blob/master/BroControl/config.py > It looks like ifconfig is hard-written into the logic. Probably needs a > patch to check for the ip command. > Cheers, > Harry > {quote} > We should probably check for the presence of the ip utility and use that, if present. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 11 09:09:00 2016 From: jira at bro-tracker.atlassian.net (Jon Schipp (JIRA)) Date: Fri, 11 Mar 2016 11:09:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1498) add '-q' to ssh execution in ssh_runner.py In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Schipp reassigned BIT-1498: ------------------------------- Assignee: (was: Jon Schipp) > add '-q' to ssh execution in ssh_runner.py > ------------------------------------------ > > Key: BIT-1498 > URL: https://bro-tracker.atlassian.net/browse/BIT-1498 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Affects Versions: 2.4 > Reporter: scampbell > Priority: Trivial > Labels: broctl > Fix For: 2.5 > > > When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. > The patch is trivial: > --- a/BroControl/ssh_runner.py > +++ b/BroControl/ssh_runner.py > @@ -108,6 +108,7 @@ class SSHMaster: > self.base_cmd = [ > "ssh", > "-o", "BatchMode=yes", > + "-q", > host, > ] > self.need_connect = True -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 11 10:37:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 11 Mar 2016 12:37:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1550) Please merge topic/johanna/netcontrol In-Reply-To: References: Message-ID: Johanna Amann created BIT-1550: ---------------------------------- Summary: Please merge topic/johanna/netcontrol Key: BIT-1550 URL: https://bro-tracker.atlassian.net/browse/BIT-1550 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: Johanna Amann Fix For: 2.5 Please merge topic/johanna/netcontrol, which contains the NetControl framework and some small core changes necessary for it. The core changes are: - add support for the PrefixTable and patricia tree to dump lists of covered IP addresses - add a number of bifs - add tracking of recursive types to prevent crash when a function contains a record as an argument in which the function is a member of The framework will get a few small updates in the future. However, these mostly should be small missing features and either not affect the API at all, or only contain minor changes. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 11 10:37:01 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 11 Mar 2016 12:37:01 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1550) Please merge topic/johanna/netcontrol In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1550: ------------------------------- Status: Merge Request (was: Open) > Please merge topic/johanna/netcontrol > ------------------------------------- > > Key: BIT-1550 > URL: https://bro-tracker.atlassian.net/browse/BIT-1550 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Fix For: 2.5 > > > Please merge topic/johanna/netcontrol, which contains the NetControl framework and some small core changes necessary for it. > The core changes are: > - add support for the PrefixTable and patricia tree to dump lists of covered IP addresses > - add a number of bifs > - add tracking of recursive types to prevent crash when a function contains a record as an argument in which the function is a member of > The framework will get a few small updates in the future. However, these mostly should be small missing features and either not affect the API at all, or only contain minor changes. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From vallentin at icir.org Fri Mar 11 10:54:18 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Fri, 11 Mar 2016 10:54:18 -0800 Subject: [Bro-Dev] Fw: Broker raw throughput In-Reply-To: References: <20160225161947.GI42006@shogun> <20160308011009.GF33456@shogun> <2C500E33-8402-474E-9EAE-E1C612A9E2DC@haw-hamburg.de> <20160308205417.GI33456@shogun> <887CF7AE-A8BC-425A-A026-7EC022049622@haw-hamburg.de> <20160310184610.GG76756@samurai.ICIR.org> Message-ID: <20160311185418.GB717@shogun> > To me, the performance numbers themselves don't matter as much as > managing expectations does: should I *expect* to be able to pass all > of my events through broker? This question depends on the event type and your concrete topology, and is hard to answer in general. We can say "in our point-to-point test scenario, our measurements show and upper bound of X events/sec for a workload consisting of message type Y." As Broker gets more traction, I assume we will get much more data points and a better understanding on the performance boundaries. > Trying to express things a slightly different way, I was concerned > that the different numbers from the different libraries were being > interpreted as an apples-to-apples comparison. Modifying CAF to > achieve the same results as e.g. 0mq would, at some point and in some > way, eventually require modifying CAF to be more like 0mq. I don't > think that would be good, because 0mq and CAF aren't (and shouldn't > be, in my humble opinion) the same thing. 0mq/nanomsg are only a thin wrapper around a blob of bytes, whereas CAF provides much more than that. However, I don't think the comparison we did was unrealistic: we looked at the overhead of sending a stream of simple (nearly empty) messages between two remote endpoints. This "dumbs down" CAF to a point where we're primarily stressing the messaging subsystem, without using much of the higher-level abstractions (CAF still has to go through its serialization layer). After Dominik's performance tweaks, the two libraries operate in the same order of magnitude, which strikes me as reasonable. 0mq still outperforms CAF in terms of maximum message rate for this specific workload, but this is also not surprising at this point, because it has received a lot of attention and optimizations over the past years specifically targeting high-throughput scenarios. Matthias From jira at bro-tracker.atlassian.net Fri Mar 11 12:55:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Fri, 11 Mar 2016 14:55:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1540) Ifconfig is hardcoded in BroControl In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer reassigned BIT-1540: ---------------------------------- Assignee: Daniel Thayer > Ifconfig is hardcoded in BroControl > ----------------------------------- > > Key: BIT-1540 > URL: https://bro-tracker.atlassian.net/browse/BIT-1540 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Daniel Thayer > Fix For: 2.5 > > > From the mailing list: > {quote} > Hi Folks, > On later versions of Linux distros iproute2 replaces ifconfig with ip > Starting at line 601 at > https://github.com/bro/broctl/blob/master/BroControl/config.py > It looks like ifconfig is hard-written into the logic. Probably needs a > patch to check for the ip command. > Cheers, > Harry > {quote} > We should probably check for the presence of the ip utility and use that, if present. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 11 13:05:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 11 Mar 2016 15:05:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1550) Please merge topic/johanna/netcontrol In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1550: --------------------------------- Assignee: Robin Sommer > Please merge topic/johanna/netcontrol > ------------------------------------- > > Key: BIT-1550 > URL: https://bro-tracker.atlassian.net/browse/BIT-1550 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > Fix For: 2.5 > > > Please merge topic/johanna/netcontrol, which contains the NetControl framework and some small core changes necessary for it. > The core changes are: > - add support for the PrefixTable and patricia tree to dump lists of covered IP addresses > - add a number of bifs > - add tracking of recursive types to prevent crash when a function contains a record as an argument in which the function is a member of > The framework will get a few small updates in the future. However, these mostly should be small missing features and either not affect the API at all, or only contain minor changes. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 11 14:25:00 2016 From: jira at bro-tracker.atlassian.net (Vlad Grigorescu (JIRA)) Date: Fri, 11 Mar 2016 16:25:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1551) Broctl plugins in Bro plugins In-Reply-To: References: Message-ID: Vlad Grigorescu created BIT-1551: ------------------------------------ Summary: Broctl plugins in Bro plugins Key: BIT-1551 URL: https://bro-tracker.atlassian.net/browse/BIT-1551 Project: Bro Issue Tracker Issue Type: New Feature Components: bro-aux, BroControl, Documentation Reporter: Vlad Grigorescu Right now, the Bro plugin skeleton creates: /scripts /src /tests I propose that a new directory, /broctl-plugins be created and that broctl adds the following directories to the search path: /lib/bro/plugins/*/broctl-plugins $BRO_PLUGIN_PATH/*/broctl-plugins The documentation here should also be updated: https://www.bro.org/sphinx-git/devel/plugins.html -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 11 14:25:00 2016 From: jira at bro-tracker.atlassian.net (Vlad Grigorescu (JIRA)) Date: Fri, 11 Mar 2016 16:25:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1551) Broctl plugins in Bro plugins In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Grigorescu reassigned BIT-1551: ------------------------------------ Assignee: Daniel Thayer > Broctl plugins in Bro plugins > ----------------------------- > > Key: BIT-1551 > URL: https://bro-tracker.atlassian.net/browse/BIT-1551 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: bro-aux, BroControl, Documentation > Reporter: Vlad Grigorescu > Assignee: Daniel Thayer > > Right now, the Bro plugin skeleton creates: > /scripts > /src > /tests > I propose that a new directory, /broctl-plugins be created and that broctl adds the following directories to the search path: > /lib/bro/plugins/*/broctl-plugins > $BRO_PLUGIN_PATH/*/broctl-plugins > The documentation here should also be updated: https://www.bro.org/sphinx-git/devel/plugins.html -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 11 14:25:00 2016 From: jira at bro-tracker.atlassian.net (Vlad Grigorescu (JIRA)) Date: Fri, 11 Mar 2016 16:25:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1551) Broctl plugins in Bro plugins In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24807#comment-24807 ] Vlad Grigorescu commented on BIT-1551: -------------------------------------- Assigning to Daniel for the broctl piece. > Broctl plugins in Bro plugins > ----------------------------- > > Key: BIT-1551 > URL: https://bro-tracker.atlassian.net/browse/BIT-1551 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: bro-aux, BroControl, Documentation > Reporter: Vlad Grigorescu > > Right now, the Bro plugin skeleton creates: > /scripts > /src > /tests > I propose that a new directory, /broctl-plugins be created and that broctl adds the following directories to the search path: > /lib/bro/plugins/*/broctl-plugins > $BRO_PLUGIN_PATH/*/broctl-plugins > The documentation here should also be updated: https://www.bro.org/sphinx-git/devel/plugins.html -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From robin at icir.org Fri Mar 11 14:33:23 2016 From: robin at icir.org (Robin Sommer) Date: Fri, 11 Mar 2016 14:33:23 -0800 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Files transferred over FTP were showing incorrect sizes. (08399da) In-Reply-To: <201603111757.u2BHvAhB024449@bro-ids.icir.org> References: <201603111757.u2BHvAhB024449@bro-ids.icir.org> Message-ID: <20160311223323.GA11677@icir.org> On Fri, Mar 11, 2016 at 12:56 -0500, Seth Hall wrote: > Files transferred over FTP were showing incorrect sizes. This seems to be causing a number of baseline mismatches in the external test suite. I can't tell if they are legitimate, did you run the tests? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jira at bro-tracker.atlassian.net Fri Mar 11 14:35:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 11 Mar 2016 16:35:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1550) Please merge topic/johanna/netcontrol In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1550: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Please merge topic/johanna/netcontrol > ------------------------------------- > > Key: BIT-1550 > URL: https://bro-tracker.atlassian.net/browse/BIT-1550 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > Fix For: 2.5 > > > Please merge topic/johanna/netcontrol, which contains the NetControl framework and some small core changes necessary for it. > The core changes are: > - add support for the PrefixTable and patricia tree to dump lists of covered IP addresses > - add a number of bifs > - add tracking of recursive types to prevent crash when a function contains a record as an argument in which the function is a member of > The framework will get a few small updates in the future. However, these mostly should be small missing features and either not affect the API at all, or only contain minor changes. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 11 14:40:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Fri, 11 Mar 2016 16:40:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1540) Ifconfig is hardcoded in BroControl In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1540: ------------------------------- Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) Merged into master. > Ifconfig is hardcoded in BroControl > ----------------------------------- > > Key: BIT-1540 > URL: https://bro-tracker.atlassian.net/browse/BIT-1540 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Daniel Thayer > Fix For: 2.5 > > > From the mailing list: > {quote} > Hi Folks, > On later versions of Linux distros iproute2 replaces ifconfig with ip > Starting at line 601 at > https://github.com/bro/broctl/blob/master/BroControl/config.py > It looks like ifconfig is hard-written into the logic. Probably needs a > patch to check for the ip command. > Cheers, > Harry > {quote} > We should probably check for the presence of the ip utility and use that, if present. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From vallentin at icir.org Fri Mar 11 14:58:45 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Fri, 11 Mar 2016 14:58:45 -0800 Subject: [Bro-Dev] Coding style enforcement Message-ID: <20160311225845.GN76756@samurai.ICIR.org> While porting Broker to the latest CAF version, I am realizing that the current pre C++11 coding style is not very conducive. Since the introduction of lambdas, and in particular with CAF's asynchronous and template-heavy programming model, the Whitesmiths style isn't very practical. Once can consider Broker a separate project, and perhaps a style change wouldn't be as complicated as in the main Bro code, I still wanted to check in with you whether anyone would object to changing the style. In particular, I'm planning to use CAF's coding style [1], which provides a unified style for meta programming as well as "regular" programming. There exists also a clang-format style file for this [2], which makes it really easy to enforce this style globally. Unfortunately, clang-format currently doesn't support the Whitesmiths style, so using this tool for Bro is not (yet) an option. (There exists an unmerged patch that needs some cleanup [3], if anyone wants to go for it.) On a related note: I'd also like to see stricter git commit message guidelines, at least putting strict rules on the first line [4]. Would you be in favor of such rules on commit messages? Matthias [1] https://github.com/actor-framework/actor-framework/blob/master/CONTRIBUTING.md [2] https://github.com/actor-framework/actor-framework/blob/master/.clang-format [3] http://reviews.llvm.org/D6833 [4] https://github.com/agis-/git-style-guide#messages From noreply at bro.org Sat Mar 12 00:00:17 2016 From: noreply at bro.org (Merge Tracker) Date: Sat, 12 Mar 2016 00:00:17 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603120800.u2C80HJB018190@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1547 [1] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [3] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- -------------- ---------- ----------------------------------------------------- #52 [4] bro J-Gras [5] 2016-01-18 Fixed matching mail address intel [6] #19 [7] bro-plugins nickwallen [8] 2016-03-09 [BIT-1543] Kafka Logger - Write Bro Logs to Kafka [9] #18 [10] bro-plugins jshlbrd [11] 2016-03-03 SSDP analyzer [12] [1] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [4] Pull Request #52 https://github.com/bro/bro/pull/52 [5] J-Gras https://github.com/J-Gras [6] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [7] Pull Request #19 https://github.com/bro/bro-plugins/pull/19 [8] nickwallen https://github.com/nickwallen [9] Merge Pull Request #19 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git add-kafka-plugin [10] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [11] jshlbrd https://github.com/jshlbrd [12] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From seth at icir.org Sat Mar 12 07:41:09 2016 From: seth at icir.org (Seth Hall) Date: Sat, 12 Mar 2016 10:41:09 -0500 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Files transferred over FTP were showing incorrect sizes. (08399da) In-Reply-To: <20160311223323.GA11677@icir.org> References: <201603111757.u2BHvAhB024449@bro-ids.icir.org> <20160311223323.GA11677@icir.org> Message-ID: <8EE42C23-4A32-4422-93F5-106178045040@icir.org> > On Mar 11, 2016, at 5:33 PM, Robin Sommer wrote: > > > On Fri, Mar 11, 2016 at 12:56 -0500, Seth Hall wrote: > >> Files transferred over FTP were showing incorrect sizes. > > This seems to be causing a number of baseline mismatches in the > external test suite. I can't tell if they are legitimate, did you run > the tests? Ugh, I didn't run those tests. I'll look over those soon and do any updates necessary. I'm sure that the test baselines are improved though. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Sat Mar 12 08:16:17 2016 From: seth at icir.org (Seth Hall) Date: Sat, 12 Mar 2016 11:16:17 -0500 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Files transferred over FTP were showing incorrect sizes. (08399da) In-Reply-To: <20160311223323.GA11677@icir.org> References: <201603111757.u2BHvAhB024449@bro-ids.icir.org> <20160311223323.GA11677@icir.org> Message-ID: <2778443D-4A7C-4D8D-B42D-A9960028DD96@icir.org> > On Mar 11, 2016, at 5:33 PM, Robin Sommer wrote: > > This seems to be causing a number of baseline mismatches in the > external test suite. I can't tell if they are legitimate, did you run > the tests? Fixed. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jira at bro-tracker.atlassian.net Sat Mar 12 14:23:00 2016 From: jira at bro-tracker.atlassian.net (Jan Grashoefer (JIRA)) Date: Sat, 12 Mar 2016 16:23:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1550) Please merge topic/johanna/netcontrol In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24809#comment-24809 ] Jan Grashoefer commented on BIT-1550: ------------------------------------- That framework looks awesome! It's a pity, but last week I realized that the subnet-functionality would be nice to have to extend the intel-framework, so I did something similar to your changes in patricia tree and PrefixTable (see [https://github.com/J-Gras/bro/compare/master...J-Gras:topic/jgras/subnet]). My approach was different in the way that I did not return a list of subnets. I returned a table including only the subnets that contain the queried one, so you would not need to query each item again based on the list of matching subnets. Would you mind to have a look and tell me your opinion on this approach (excluding the function names... yours are much better :) )? > Please merge topic/johanna/netcontrol > ------------------------------------- > > Key: BIT-1550 > URL: https://bro-tracker.atlassian.net/browse/BIT-1550 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > Fix For: 2.5 > > > Please merge topic/johanna/netcontrol, which contains the NetControl framework and some small core changes necessary for it. > The core changes are: > - add support for the PrefixTable and patricia tree to dump lists of covered IP addresses > - add a number of bifs > - add tracking of recursive types to prevent crash when a function contains a record as an argument in which the function is a member of > The framework will get a few small updates in the future. However, these mostly should be small missing features and either not affect the API at all, or only contain minor changes. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Sat Mar 12 20:48:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Sat, 12 Mar 2016 22:48:00 -0600 (CST) Subject: [Bro-Dev] [JIRA] (BIT-1550) Please merge topic/johanna/netcontrol In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24810#comment-24810 ] Johanna Amann commented on BIT-1550: ------------------------------------ I think both approaches are valid - it just depends on your use-case which is more appropriate. For what I am using them for, I really only needed the list of subnets (and not constructing the whole table has a lower overhead). I will try to just fold that into my current patch sometime end-of-next-week-ish. I will probably do it a bit different, but that mostly is a style thing (and on a first glance, your code has a few gotchas - I think constructing the table as having the type table[subnet] of any can lead to a few interesting issues, e.g.). > Please merge topic/johanna/netcontrol > ------------------------------------- > > Key: BIT-1550 > URL: https://bro-tracker.atlassian.net/browse/BIT-1550 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > Fix For: 2.5 > > > Please merge topic/johanna/netcontrol, which contains the NetControl framework and some small core changes necessary for it. > The core changes are: > - add support for the PrefixTable and patricia tree to dump lists of covered IP addresses > - add a number of bifs > - add tracking of recursive types to prevent crash when a function contains a record as an argument in which the function is a member of > The framework will get a few small updates in the future. However, these mostly should be small missing features and either not affect the API at all, or only contain minor changes. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From noreply at bro.org Sun Mar 13 00:00:16 2016 From: noreply at bro.org (Merge Tracker) Date: Sun, 13 Mar 2016 00:00:16 -0800 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603130800.u2D80GEw001486@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1547 [1] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [3] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- -------------- ---------- ----------------------------------------------------- #52 [4] bro J-Gras [5] 2016-01-18 Fixed matching mail address intel [6] #19 [7] bro-plugins nickwallen [8] 2016-03-09 [BIT-1543] Kafka Logger - Write Bro Logs to Kafka [9] #18 [10] bro-plugins jshlbrd [11] 2016-03-03 SSDP analyzer [12] [1] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [4] Pull Request #52 https://github.com/bro/bro/pull/52 [5] J-Gras https://github.com/J-Gras [6] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [7] Pull Request #19 https://github.com/bro/bro-plugins/pull/19 [8] nickwallen https://github.com/nickwallen [9] Merge Pull Request #19 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git add-kafka-plugin [10] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [11] jshlbrd https://github.com/jshlbrd [12] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From noreply at bro.org Mon Mar 14 00:00:23 2016 From: noreply at bro.org (Merge Tracker) Date: Mon, 14 Mar 2016 00:00:23 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603140700.u2E70N1G029177@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1547 [1] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [3] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- -------------- ---------- ----------------------------------------------------- #52 [4] bro J-Gras [5] 2016-01-18 Fixed matching mail address intel [6] #19 [7] bro-plugins nickwallen [8] 2016-03-09 [BIT-1543] Kafka Logger - Write Bro Logs to Kafka [9] #18 [10] bro-plugins jshlbrd [11] 2016-03-03 SSDP analyzer [12] [1] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [4] Pull Request #52 https://github.com/bro/bro/pull/52 [5] J-Gras https://github.com/J-Gras [6] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [7] Pull Request #19 https://github.com/bro/bro-plugins/pull/19 [8] nickwallen https://github.com/nickwallen [9] Merge Pull Request #19 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git add-kafka-plugin [10] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [11] jshlbrd https://github.com/jshlbrd [12] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Mon Mar 14 07:09:01 2016 From: jira at bro-tracker.atlassian.net (Jan Grashoefer (JIRA)) Date: Mon, 14 Mar 2016 09:09:01 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1550) Please merge topic/johanna/netcontrol In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24900#comment-24900 ] Jan Grashoefer commented on BIT-1550: ------------------------------------- My idea was to keep it generic so you could call it with any table, which will fall back to normal lookup for everything except subnet-indexed tables. Therefore the result tables should copy the type of the input tables. Probably I am doing it wrong, as it was a hack without full knowledge about how the typesystem really works. In case you think it's really worth to keep both approaches, I am happy to help. > Please merge topic/johanna/netcontrol > ------------------------------------- > > Key: BIT-1550 > URL: https://bro-tracker.atlassian.net/browse/BIT-1550 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > Fix For: 2.5 > > > Please merge topic/johanna/netcontrol, which contains the NetControl framework and some small core changes necessary for it. > The core changes are: > - add support for the PrefixTable and patricia tree to dump lists of covered IP addresses > - add a number of bifs > - add tracking of recursive types to prevent crash when a function contains a record as an argument in which the function is a member of > The framework will get a few small updates in the future. However, these mostly should be small missing features and either not affect the API at all, or only contain minor changes. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From noreply at bro.org Tue Mar 15 00:00:19 2016 From: noreply at bro.org (Merge Tracker) Date: Tue, 15 Mar 2016 00:00:19 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603150700.u2F70JbB022456@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1547 [1] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [3] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- -------------- ---------- ----------------------------------------------------- #52 [4] bro J-Gras [5] 2016-01-18 Fixed matching mail address intel [6] #19 [7] bro-plugins nickwallen [8] 2016-03-09 [BIT-1543] Kafka Logger - Write Bro Logs to Kafka [9] #18 [10] bro-plugins jshlbrd [11] 2016-03-03 SSDP analyzer [12] [1] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [4] Pull Request #52 https://github.com/bro/bro/pull/52 [5] J-Gras https://github.com/J-Gras [6] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [7] Pull Request #19 https://github.com/bro/bro-plugins/pull/19 [8] nickwallen https://github.com/nickwallen [9] Merge Pull Request #19 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git add-kafka-plugin [10] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [11] jshlbrd https://github.com/jshlbrd [12] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From noreply at bro.org Wed Mar 16 00:00:23 2016 From: noreply at bro.org (Merge Tracker) Date: Wed, 16 Mar 2016 00:00:23 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603160700.u2G70NEr006515@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1547 [1] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [3] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- -------------- ---------- ----------------------------------------------------- #52 [4] bro J-Gras [5] 2016-01-18 Fixed matching mail address intel [6] #19 [7] bro-plugins nickwallen [8] 2016-03-15 [BIT-1543] Kafka Logger - Write Bro Logs to Kafka [9] #18 [10] bro-plugins jshlbrd [11] 2016-03-03 SSDP analyzer [12] [1] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [4] Pull Request #52 https://github.com/bro/bro/pull/52 [5] J-Gras https://github.com/J-Gras [6] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [7] Pull Request #19 https://github.com/bro/bro-plugins/pull/19 [8] nickwallen https://github.com/nickwallen [9] Merge Pull Request #19 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git add-kafka-plugin [10] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [11] jshlbrd https://github.com/jshlbrd [12] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Wed Mar 16 07:53:00 2016 From: jira at bro-tracker.atlassian.net (Lu Goon (JIRA)) Date: Wed, 16 Mar 2016 09:53:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1552) User Agent Strings to Intel Framework In-Reply-To: References: Message-ID: Lu Goon created BIT-1552: ---------------------------- Summary: User Agent Strings to Intel Framework Key: BIT-1552 URL: https://bro-tracker.atlassian.net/browse/BIT-1552 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.4 Environment: Linux - CentOS - ubuntu 14.04 Reporter: Lu Goon Does anyone know how to add user agent strings to the Intel Framework? We have a list of about 100 UAs that we want to track and alert on. I have been good at adding IP addresses to INTEL however when I add using HTTP::IN_USER_AGENT_HEADER then I think bro does not understand that or I have my fields wrong. So far I have #fields indicator indicator_type meta.source meta.desc meta.url and for indicator I use the HTTP::IN_USER_AGENT_HEADER Any help with importing this via the INTEL framework, is greatly appreciated. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From robin at icir.org Wed Mar 16 09:26:40 2016 From: robin at icir.org (Robin Sommer) Date: Wed, 16 Mar 2016 09:26:40 -0700 Subject: [Bro-Dev] Coding style enforcement In-Reply-To: <20160311225845.GN76756@samurai.ICIR.org> References: <20160311225845.GN76756@samurai.ICIR.org> Message-ID: <20160316162640.GL40337@icir.org> As I had mentioned to Matthias already, I don't have strong feelings regarding Broker coding style. Changing that to match CAF sounds reasonable to me, as a lot of the code's structure is driven by CAF as well. Jon is the one most invested into the style, so as long as he's ok with it, I don't see a problem. (For the record, for Bro itself my preference remains staying with the current style. Ideally somebody will be able to teach that to clang-format eventually.). Regrading commit messages, these are our current guidelines: https://www.bro.org/development/howtos/process.html#writing-commit-messages I wouldn't want to be too religious about that, but having a succinct first line certainly makes sense. Not sure I'd remember not to put a period at the end though. :-) Robin On Fri, Mar 11, 2016 at 14:58 -0800, you wrote: > While porting Broker to the latest CAF version, I am realizing that the > current pre C++11 coding style is not very conducive. Since the > introduction of lambdas, and in particular with CAF's asynchronous and > template-heavy programming model, the Whitesmiths style isn't very > practical. > > Once can consider Broker a separate project, and perhaps a style change > wouldn't be as complicated as in the main Bro code, I still wanted to > check in with you whether anyone would object to changing the style. In > particular, I'm planning to use CAF's coding style [1], which provides a > unified style for meta programming as well as "regular" programming. > > There exists also a clang-format style file for this [2], which makes it > really easy to enforce this style globally. Unfortunately, clang-format > currently doesn't support the Whitesmiths style, so using this tool for > Bro is not (yet) an option. (There exists an unmerged patch that needs > some cleanup [3], if anyone wants to go for it.) > > On a related note: I'd also like to see stricter git commit message > guidelines, at least putting strict rules on the first line [4]. Would > you be in favor of such rules on commit messages? > > Matthias > > [1] https://github.com/actor-framework/actor-framework/blob/master/CONTRIBUTING.md > [2] https://github.com/actor-framework/actor-framework/blob/master/.clang-format > [3] http://reviews.llvm.org/D6833 > [4] https://github.com/agis-/git-style-guide#messages > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jira at bro-tracker.atlassian.net Wed Mar 16 10:09:01 2016 From: jira at bro-tracker.atlassian.net (Jan Grashoefer (JIRA)) Date: Wed, 16 Mar 2016 12:09:01 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1552) User Agent Strings to Intel Framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25000#comment-25000 ] Jan Grashoefer commented on BIT-1552: ------------------------------------- The intel framework treats user agent strings as software. Thus the type to use should be Intel::SOFTWARE (see [http-headers.bro|https://github.com/bro/bro/blob/master/scripts/policy/frameworks/intel/seen/http-headers.bro#L45]). HTTP::IN_USER_AGENT_HEADER can be used as where-location. > User Agent Strings to Intel Framework > ------------------------------------- > > Key: BIT-1552 > URL: https://bro-tracker.atlassian.net/browse/BIT-1552 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Environment: Linux - CentOS - ubuntu 14.04 > Reporter: Lu Goon > Labels: agent, intel-framework, user > > Does anyone know how to add user agent strings to the Intel Framework? We have a list of about 100 UAs that we want to track and alert on. I have been good at adding IP addresses to INTEL however when I add using HTTP::IN_USER_AGENT_HEADER then I think bro does not understand that or I have my fields wrong. > So far I have #fields indicator indicator_type meta.source meta.desc meta.url > and for indicator I use the HTTP::IN_USER_AGENT_HEADER > Any help with importing this via the INTEL framework, is greatly appreciated. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Wed Mar 16 16:00:02 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Wed, 16 Mar 2016 18:00:02 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1550) Please merge topic/johanna/netcontrol In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25001#comment-25001 ] Johanna Amann commented on BIT-1550: ------------------------------------ I added the feature of just filtering tables to my current implementation in the topic/johanna/filter_subnet_table branch. Could you perhaps take a look at that and see if that works for you? The implementation is quite a bit different from how you did it. I also opted to keep the bif specific to table/set[subnet] - my reasoning was that if we ever have other types that support that kind of matching, we will probably want to convert that into a language feature instead of having a bif that works for all types. > Please merge topic/johanna/netcontrol > ------------------------------------- > > Key: BIT-1550 > URL: https://bro-tracker.atlassian.net/browse/BIT-1550 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > Fix For: 2.5 > > > Please merge topic/johanna/netcontrol, which contains the NetControl framework and some small core changes necessary for it. > The core changes are: > - add support for the PrefixTable and patricia tree to dump lists of covered IP addresses > - add a number of bifs > - add tracking of recursive types to prevent crash when a function contains a record as an argument in which the function is a member of > The framework will get a few small updates in the future. However, these mostly should be small missing features and either not affect the API at all, or only contain minor changes. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Wed Mar 16 20:13:02 2016 From: jira at bro-tracker.atlassian.net (Lu Goon (JIRA)) Date: Wed, 16 Mar 2016 22:13:02 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1552) User Agent Strings to Intel Framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lu Goon updated BIT-1552: ------------------------- Resolution: Fixed Status: Closed (was: Open) The suggestion provided worked! > User Agent Strings to Intel Framework > ------------------------------------- > > Key: BIT-1552 > URL: https://bro-tracker.atlassian.net/browse/BIT-1552 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Environment: Linux - CentOS - ubuntu 14.04 > Reporter: Lu Goon > Labels: agent, intel-framework, user > > Does anyone know how to add user agent strings to the Intel Framework? We have a list of about 100 UAs that we want to track and alert on. I have been good at adding IP addresses to INTEL however when I add using HTTP::IN_USER_AGENT_HEADER then I think bro does not understand that or I have my fields wrong. > So far I have #fields indicator indicator_type meta.source meta.desc meta.url > and for indicator I use the HTTP::IN_USER_AGENT_HEADER > Any help with importing this via the INTEL framework, is greatly appreciated. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Wed Mar 16 20:13:01 2016 From: jira at bro-tracker.atlassian.net (Lu Goon (JIRA)) Date: Wed, 16 Mar 2016 22:13:01 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1552) User Agent Strings to Intel Framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25002#comment-25002 ] Lu Goon commented on BIT-1552: ------------------------------ Thanks that worked!!! > User Agent Strings to Intel Framework > ------------------------------------- > > Key: BIT-1552 > URL: https://bro-tracker.atlassian.net/browse/BIT-1552 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Environment: Linux - CentOS - ubuntu 14.04 > Reporter: Lu Goon > Labels: agent, intel-framework, user > > Does anyone know how to add user agent strings to the Intel Framework? We have a list of about 100 UAs that we want to track and alert on. I have been good at adding IP addresses to INTEL however when I add using HTTP::IN_USER_AGENT_HEADER then I think bro does not understand that or I have my fields wrong. > So far I have #fields indicator indicator_type meta.source meta.desc meta.url > and for indicator I use the HTTP::IN_USER_AGENT_HEADER > Any help with importing this via the INTEL framework, is greatly appreciated. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From noreply at bro.org Thu Mar 17 00:00:20 2016 From: noreply at bro.org (Merge Tracker) Date: Thu, 17 Mar 2016 00:00:20 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603170700.u2H70K6r003326@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1547 [1] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1507 [2] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [3] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ----------- ---------- ------------------------------------- #52 [4] bro J-Gras [5] 2016-01-18 Fixed matching mail address intel [6] #18 [7] bro-plugins jshlbrd [8] 2016-03-03 SSDP analyzer [9] [1] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [2] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [3] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [4] Pull Request #52 https://github.com/bro/bro/pull/52 [5] J-Gras https://github.com/J-Gras [6] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [7] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [8] jshlbrd https://github.com/jshlbrd [9] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jsiwek at illinois.edu Thu Mar 17 05:07:43 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Thu, 17 Mar 2016 12:07:43 +0000 Subject: [Bro-Dev] Coding style enforcement In-Reply-To: <20160316162640.GL40337@icir.org> References: <20160311225845.GN76756@samurai.ICIR.org> <20160316162640.GL40337@icir.org> Message-ID: > On Mar 16, 2016, at 11:26 AM, Robin Sommer wrote: > > As I had mentioned to Matthias already, I don't have strong feelings > regarding Broker coding style. Changing that to match CAF sounds > reasonable to me, as a lot of the code's structure is driven by CAF as > well. Jon is the one most invested into the style, so as long as he's > ok with it, I don't see a problem. I?m fine w/ any style or naming convention changes in order to cause less friction for Matthias/others. - Jon From jira at bro-tracker.atlassian.net Thu Mar 17 10:56:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Thu, 17 Mar 2016 12:56:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1553) Please merge topic/johanna/filter_subnet_table In-Reply-To: References: Message-ID: Johanna Amann created BIT-1553: ---------------------------------- Summary: Please merge topic/johanna/filter_subnet_table Key: BIT-1553 URL: https://bro-tracker.atlassian.net/browse/BIT-1553 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: Johanna Amann Please merge topic/johanna/filter_subnet_table This branch adds the filter_subnet_table bif. This bif works similar to the matching_subnet bif. The difference is that, instead of returning a vector of the subnets that match, we return a filtered view of the original set/table only containing the changed subnets. The branch also fixes a small bug in TableVal::UpdateTimestamp (ReadOperation only has to be called when LoggingAccess() is true). -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 17 10:57:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Thu, 17 Mar 2016 12:57:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1553) Please merge topic/johanna/filter_subnet_table In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1553: ------------------------------- Status: Merge Request (was: Open) > Please merge topic/johanna/filter_subnet_table > ---------------------------------------------- > > Key: BIT-1553 > URL: https://bro-tracker.atlassian.net/browse/BIT-1553 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > > Please merge topic/johanna/filter_subnet_table > This branch adds the filter_subnet_table bif. This bif works similar to the matching_subnet bif. The difference is that, instead of returning a vector of the subnets that match, we return a filtered view of the original set/table only containing the changed subnets. > The branch also fixes a small bug in TableVal::UpdateTimestamp (ReadOperation only has to be called when LoggingAccess() is true). -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 17 11:15:01 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Thu, 17 Mar 2016 13:15:01 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1550) Please merge topic/johanna/netcontrol In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25004#comment-25004 ] Johanna Amann commented on BIT-1550: ------------------------------------ FYI - merge request for this is in BIT-1553 > Please merge topic/johanna/netcontrol > ------------------------------------- > > Key: BIT-1550 > URL: https://bro-tracker.atlassian.net/browse/BIT-1550 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > Fix For: 2.5 > > > Please merge topic/johanna/netcontrol, which contains the NetControl framework and some small core changes necessary for it. > The core changes are: > - add support for the PrefixTable and patricia tree to dump lists of covered IP addresses > - add a number of bifs > - add tracking of recursive types to prevent crash when a function contains a record as an argument in which the function is a member of > The framework will get a few small updates in the future. However, these mostly should be small missing features and either not affect the API at all, or only contain minor changes. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 17 11:53:01 2016 From: jira at bro-tracker.atlassian.net (Vlad Grigorescu (JIRA)) Date: Thu, 17 Mar 2016 13:53:01 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1533) mysql analyzer does not set service to mysql In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Grigorescu updated BIT-1533: --------------------------------- Status: Merge Request (was: Open) Assignee: (was: Vlad Grigorescu) > mysql analyzer does not set service to mysql > -------------------------------------------- > > Key: BIT-1533 > URL: https://bro-tracker.atlassian.net/browse/BIT-1533 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Justin Azoff > Priority: Low > > The mysql analyzer does not set the service to mysql. The result of this is that conn.log and known_services do not show 'mysql' anywhere. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 17 11:53:01 2016 From: jira at bro-tracker.atlassian.net (Vlad Grigorescu (JIRA)) Date: Thu, 17 Mar 2016 13:53:01 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1533) mysql analyzer does not set service to mysql In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25005#comment-25005 ] Vlad Grigorescu commented on BIT-1533: -------------------------------------- Fixed in topic/vladg/bit-1533 > mysql analyzer does not set service to mysql > -------------------------------------------- > > Key: BIT-1533 > URL: https://bro-tracker.atlassian.net/browse/BIT-1533 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Justin Azoff > Assignee: Vlad Grigorescu > Priority: Low > > The mysql analyzer does not set the service to mysql. The result of this is that conn.log and known_services do not show 'mysql' anywhere. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 17 12:06:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Thu, 17 Mar 2016 14:06:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1553) Please merge topic/johanna/filter_subnet_table In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1553: ------------------------------- Status: Open (was: Merge Request) > Please merge topic/johanna/filter_subnet_table > ---------------------------------------------- > > Key: BIT-1553 > URL: https://bro-tracker.atlassian.net/browse/BIT-1553 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > > Please merge topic/johanna/filter_subnet_table > This branch adds the filter_subnet_table bif. This bif works similar to the matching_subnet bif. The difference is that, instead of returning a vector of the subnets that match, we return a filtered view of the original set/table only containing the changed subnets. > The branch also fixes a small bug in TableVal::UpdateTimestamp (ReadOperation only has to be called when LoggingAccess() is true). -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 17 12:06:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Thu, 17 Mar 2016 14:06:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1553) Please merge topic/johanna/filter_subnet_table In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann reassigned BIT-1553: ---------------------------------- Assignee: Johanna Amann > Please merge topic/johanna/filter_subnet_table > ---------------------------------------------- > > Key: BIT-1553 > URL: https://bro-tracker.atlassian.net/browse/BIT-1553 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Johanna Amann > > Please merge topic/johanna/filter_subnet_table > This branch adds the filter_subnet_table bif. This bif works similar to the matching_subnet bif. The difference is that, instead of returning a vector of the subnets that match, we return a filtered view of the original set/table only containing the changed subnets. > The branch also fixes a small bug in TableVal::UpdateTimestamp (ReadOperation only has to be called when LoggingAccess() is true). -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 17 12:07:00 2016 From: jira at bro-tracker.atlassian.net (Jan Grashoefer (JIRA)) Date: Thu, 17 Mar 2016 14:07:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1553) Please merge topic/johanna/filter_subnet_table In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25006#comment-25006 ] Jan Grashoefer commented on BIT-1553: ------------------------------------- I have checked that against my testcase, fixed a small bug regarding &read_expire and opened a pull request for that branch. > Please merge topic/johanna/filter_subnet_table > ---------------------------------------------- > > Key: BIT-1553 > URL: https://bro-tracker.atlassian.net/browse/BIT-1553 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Johanna Amann > > Please merge topic/johanna/filter_subnet_table > This branch adds the filter_subnet_table bif. This bif works similar to the matching_subnet bif. The difference is that, instead of returning a vector of the subnets that match, we return a filtered view of the original set/table only containing the changed subnets. > The branch also fixes a small bug in TableVal::UpdateTimestamp (ReadOperation only has to be called when LoggingAccess() is true). -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 17 14:52:00 2016 From: jira at bro-tracker.atlassian.net (Adam Slagell (JIRA)) Date: Thu, 17 Mar 2016 16:52:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1551) Broctl plugins in Bro plugins In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Slagell updated BIT-1551: ------------------------------ Priority: Low (was: Normal) > Broctl plugins in Bro plugins > ----------------------------- > > Key: BIT-1551 > URL: https://bro-tracker.atlassian.net/browse/BIT-1551 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: bro-aux, BroControl, Documentation > Reporter: Vlad Grigorescu > Assignee: Daniel Thayer > Priority: Low > > Right now, the Bro plugin skeleton creates: > /scripts > /src > /tests > I propose that a new directory, /broctl-plugins be created and that broctl adds the following directories to the search path: > /lib/bro/plugins/*/broctl-plugins > $BRO_PLUGIN_PATH/*/broctl-plugins > The documentation here should also be updated: https://www.bro.org/sphinx-git/devel/plugins.html -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 17 15:16:00 2016 From: jira at bro-tracker.atlassian.net (M.B. (JIRA)) Date: Thu, 17 Mar 2016 17:16:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1554) broker (bro 2.4.1) fails to build against Python 3.{3, 4, 5} In-Reply-To: References: Message-ID: M.B. created BIT-1554: ------------------------- Summary: broker (bro 2.4.1) fails to build against Python 3.{3,4,5} Key: BIT-1554 URL: https://bro-tracker.atlassian.net/browse/BIT-1554 Project: Bro Issue Tracker Issue Type: Problem Components: Broker Affects Versions: 2.4 Environment: Trying to compile Bro 2.4.1 on Gentoo Linux (x86_64) with broker enabled, against CAF 0.13.2, with python, using GCC support. Reporter: M.B. Attachments: bro-2.4.1.ebuild, build.log Bro fails to build. Details (in particular the options cmake gets called with) can be seen from the build.log. For completeness I included the .ebuild. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Thu Mar 17 15:39:00 2016 From: jira at bro-tracker.atlassian.net (M.B. (JIRA)) Date: Thu, 17 Mar 2016 17:39:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1555) aux/broker/bindings/python/CMakeLists.txt doesn't respect -DINSTALL_LIB_DIR In-Reply-To: References: Message-ID: M.B. created BIT-1555: ------------------------- Summary: aux/broker/bindings/python/CMakeLists.txt doesn't respect -DINSTALL_LIB_DIR Key: BIT-1555 URL: https://bro-tracker.atlassian.net/browse/BIT-1555 Project: Bro Issue Tracker Issue Type: Problem Components: Broker Affects Versions: 2.4 Environment: Building on Gentoo Linux (x86_64) Reporter: M.B. Attachments: bro-2.4.1.ebuild, bro-2.4.1-fix-python-install-dir.patch During a normal build, this is a non-issue, as files get installed to .../lib/... However, in a multilib environment this may become an issue. Hence it should respect INSTALL_LIB_DIR, propagated from the top-level CMakeLists.txt. I wrote a simple patch that simply removes the logic for re-setting PY_MOD_INSTALL_DIR, as I use this var to circumvent the issue. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From noreply at bro.org Fri Mar 18 00:00:24 2016 From: noreply at bro.org (Merge Tracker) Date: Fri, 18 Mar 2016 00:00:24 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603180700.u2I70OJh001866@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1547 [1] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1533 [2] Bro Justin Azoff - 2016-03-17 - Low mysql analyzer does not set service to mysql BIT-1507 [3] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [4] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ----------- ---------- ------------------------------------- #52 [5] bro J-Gras [6] 2016-01-18 Fixed matching mail address intel [7] #18 [8] bro-plugins jshlbrd [9] 2016-03-03 SSDP analyzer [10] [1] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [2] BIT-1533 https://bro-tracker.atlassian.net/browse/BIT-1533 [3] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [4] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [5] Pull Request #52 https://github.com/bro/bro/pull/52 [6] J-Gras https://github.com/J-Gras [7] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [8] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [9] jshlbrd https://github.com/jshlbrd [10] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From vallentin at icir.org Fri Mar 18 08:24:31 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Fri, 18 Mar 2016 08:24:31 -0700 Subject: [Bro-Dev] Coding style enforcement In-Reply-To: References: <20160311225845.GN76756@samurai.ICIR.org> <20160316162640.GL40337@icir.org> Message-ID: <20160318152431.GD8458@shogun> > I?m fine w/ any style or naming convention changes in order to cause > less friction for Matthias/others. Good to know Jon, thanks for chiming in. My goal is to leverage clang-format to the best degree possible such switching styles is not a big undertaking. Since no tool has (yet) good support for lambdas and template meta programming, I'll make sure to document the style for these constructs explicitly. Ideally this gives us some experience for future application throughout the entire Bro code base. Matthias From jira at bro-tracker.atlassian.net Fri Mar 18 11:11:01 2016 From: jira at bro-tracker.atlassian.net (Jon Schipp (JIRA)) Date: Fri, 18 Mar 2016 13:11:01 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-274) Finding lines where redefs occurred In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Schipp reassigned BIT-274: ------------------------------ Assignee: Jon Schipp > Finding lines where redefs occurred > ----------------------------------- > > Key: BIT-274 > URL: https://bro-tracker.atlassian.net/browse/BIT-274 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: 1.5.1 > Reporter: Seth Hall > Assignee: Jon Schipp > > First, support would need added to Bro for finding all of the lines and scripts where redef's against a certain variable occurred. I would also like to see this support added through broctl. > Here's the scenario... > {noformat} > [BroControl] > find redef ignore_checksums > /usr/local/bro/share/bro/bro.init:360 const ignore_checksums = F &redef; > /usr/local/bro/share/bro/site/local.bro:133 redef ignore_checksums = T; > {noformat} > This is relating to a discussion I had about trouble people have with starting with Bro and the gotcha's encountered from enabling the cluster support. There are so many redef's happening and potentially without the user realizing it. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 18 12:50:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 18 Mar 2016 14:50:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1553) Please merge topic/johanna/filter_subnet_table In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25010#comment-25010 ] Johanna Amann commented on BIT-1553: ------------------------------------ I added Jans branch and one more commit. Now this branch also fixes read_expire for set/table[subnet], which did not work before (this is all the work of Jan). Furthermore, I added a check that throws a syntax error when one tries to define several out of read, write and create_expire. > Please merge topic/johanna/filter_subnet_table > ---------------------------------------------- > > Key: BIT-1553 > URL: https://bro-tracker.atlassian.net/browse/BIT-1553 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Johanna Amann > > Please merge topic/johanna/filter_subnet_table > This branch adds the filter_subnet_table bif. This bif works similar to the matching_subnet bif. The difference is that, instead of returning a vector of the subnets that match, we return a filtered view of the original set/table only containing the changed subnets. > The branch also fixes a small bug in TableVal::UpdateTimestamp (ReadOperation only has to be called when LoggingAccess() is true). -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 18 12:50:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 18 Mar 2016 14:50:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1553) Please merge topic/johanna/filter_subnet_table In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1553: ------------------------------- Status: Merge Request (was: Open) Assignee: (was: Johanna Amann) > Please merge topic/johanna/filter_subnet_table > ---------------------------------------------- > > Key: BIT-1553 > URL: https://bro-tracker.atlassian.net/browse/BIT-1553 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > > Please merge topic/johanna/filter_subnet_table > This branch adds the filter_subnet_table bif. This bif works similar to the matching_subnet bif. The difference is that, instead of returning a vector of the subnets that match, we return a filtered view of the original set/table only containing the changed subnets. > The branch also fixes a small bug in TableVal::UpdateTimestamp (ReadOperation only has to be called when LoggingAccess() is true). -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 18 13:08:01 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 18 Mar 2016 15:08:01 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-274) Finding lines where redefs occurred In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25011#comment-25011 ] Johanna Amann commented on BIT-274: ----------------------------------- Jon, if you have not already done that you might want to talk to Seth before you start any work on this - this might no longer be applicable with the config framework (or should be included in it). > Finding lines where redefs occurred > ----------------------------------- > > Key: BIT-274 > URL: https://bro-tracker.atlassian.net/browse/BIT-274 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: 1.5.1 > Reporter: Seth Hall > Assignee: Jon Schipp > > First, support would need added to Bro for finding all of the lines and scripts where redef's against a certain variable occurred. I would also like to see this support added through broctl. > Here's the scenario... > {noformat} > [BroControl] > find redef ignore_checksums > /usr/local/bro/share/bro/bro.init:360 const ignore_checksums = F &redef; > /usr/local/bro/share/bro/site/local.bro:133 redef ignore_checksums = T; > {noformat} > This is relating to a discussion I had about trouble people have with starting with Bro and the gotcha's encountered from enabling the cluster support. There are so many redef's happening and potentially without the user realizing it. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 18 13:27:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 18 Mar 2016 15:27:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1555) aux/broker/bindings/python/CMakeLists.txt doesn't respect -DINSTALL_LIB_DIR In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1555: ------------------------------- Fix Version/s: 2.5 > aux/broker/bindings/python/CMakeLists.txt doesn't respect -DINSTALL_LIB_DIR > --------------------------------------------------------------------------- > > Key: BIT-1555 > URL: https://bro-tracker.atlassian.net/browse/BIT-1555 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Broker > Affects Versions: 2.4 > Environment: Building on Gentoo Linux (x86_64) > Reporter: M.B. > Labels: build > Fix For: 2.5 > > Attachments: bro-2.4.1.ebuild, bro-2.4.1-fix-python-install-dir.patch > > > During a normal build, this is a non-issue, as files get installed to .../lib/... > However, in a multilib environment this may become an issue. Hence it should respect INSTALL_LIB_DIR, propagated from the top-level CMakeLists.txt. > I wrote a simple patch that simply removes the logic for re-setting PY_MOD_INSTALL_DIR, as I use this var to circumvent the issue. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From jira at bro-tracker.atlassian.net Fri Mar 18 13:27:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 18 Mar 2016 15:27:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1554) broker (bro 2.4.1) fails to build against Python 3.{3, 4, 5} In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1554: ------------------------------- Fix Version/s: 2.5 > broker (bro 2.4.1) fails to build against Python 3.{3,4,5} > ---------------------------------------------------------- > > Key: BIT-1554 > URL: https://bro-tracker.atlassian.net/browse/BIT-1554 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Broker > Affects Versions: 2.4 > Environment: Trying to compile Bro 2.4.1 on Gentoo Linux (x86_64) with broker enabled, against CAF 0.13.2, with python, using GCC support. > Reporter: M.B. > Labels: build > Fix For: 2.5 > > Attachments: bro-2.4.1.ebuild, build.log > > > Bro fails to build. Details (in particular the options cmake gets called with) can be seen from the build.log. > For completeness I included the .ebuild. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From vallentin at icir.org Fri Mar 18 20:20:57 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Fri, 18 Mar 2016 20:20:57 -0700 Subject: [Bro-Dev] Broker & CAF includes Message-ID: <20160319032057.GJ8458@shogun> During Broker refactoring, I noticed the following: all headers in broker/* include either standard library headers or Broker headers. This appears to be by design, which makes sense to me. As a library writer, one faces the tricky question of exposing headers from dependencies. For example, Broker currently has it's own broker::util::optional, which ships as a (now outdated) copy of the corresponding CAF source. I am inclined to change this copy to an include that points directly into CAF headers, with the following rationale: Broker already depends on CAF, and a system that has CAF installed always ships with CAF headers. (Strictly speaking, we're not copying the code of into broker either, but relying on it via an include.) >From a user perspective, nothing changes here. A user will never include a CAF header, but may rely on it during compilation. Here's what I an example of what I want broker/util/optional.hh to look like: #include "caf/optional.hpp" // <--- New include. namespace broker { namespace util { using caf::optional; }} Currently we have: // Note the absence of a CAF include. namespace broker { namespace util { template class optional { // code copied from CAF }; }} Relying on the former form is more maintainable, and allows us to stay in sync with upstream fixes and improvements on the CAF side. I'm checking in here on the list to see whether anyone has objections. Matthias From leres at ee.lbl.gov Fri Mar 18 21:37:38 2016 From: leres at ee.lbl.gov (Craig Leres) Date: Fri, 18 Mar 2016 21:37:38 -0700 Subject: [Bro-Dev] Broker & CAF includes In-Reply-To: <20160319032057.GJ8458@shogun> References: <20160319032057.GJ8458@shogun> Message-ID: <56ECD792.3090404@ee.lbl.gov> This all makes sense to me; what you describe as the current situation (two packages defining the same data structure) seems broken to me. And I see caf/config.hpp defines CAF_VERSION so you're set if broker now or in the future requires a minimum version of caf. Craig From noreply at bro.org Sat Mar 19 00:00:19 2016 From: noreply at bro.org (Merge Tracker) Date: Sat, 19 Mar 2016 00:00:19 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603190700.u2J70J32025552@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1553 [1] Bro Johanna Amann - 2016-03-18 - Normal Please merge topic/johanna/filter_subnet_table BIT-1547 [2] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1533 [3] Bro Justin Azoff - 2016-03-17 - Low mysql analyzer does not set service to mysql BIT-1507 [4] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [5] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ------------ ---------- ------------------------------------- #52 [6] bro J-Gras [7] 2016-01-18 Fixed matching mail address intel [8] #18 [9] bro-plugins jshlbrd [10] 2016-03-03 SSDP analyzer [11] [1] BIT-1553 https://bro-tracker.atlassian.net/browse/BIT-1553 [2] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [3] BIT-1533 https://bro-tracker.atlassian.net/browse/BIT-1533 [4] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [5] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [6] Pull Request #52 https://github.com/bro/bro/pull/52 [7] J-Gras https://github.com/J-Gras [8] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [9] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [10] jshlbrd https://github.com/jshlbrd [11] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Sat Mar 19 19:55:00 2016 From: jira at bro-tracker.atlassian.net (M.B. (JIRA)) Date: Sat, 19 Mar 2016 21:55:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1556) bro-2.4.1 fails to compile broker with -march=i686 In-Reply-To: References: Message-ID: M.B. created BIT-1556: ------------------------- Summary: bro-2.4.1 fails to compile broker with -march=i686 Key: BIT-1556 URL: https://bro-tracker.atlassian.net/browse/BIT-1556 Project: Bro Issue Tracker Issue Type: Problem Components: Bro, Broker Affects Versions: 2.4 Environment: x86 chroot on Gentoo, with generic settings. E.g. -march=i686. Reporter: M.B. Attachments: build.log Build fails due to missing SSE support. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From noreply at bro.org Sun Mar 20 00:00:14 2016 From: noreply at bro.org (Merge Tracker) Date: Sun, 20 Mar 2016 00:00:14 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603200700.u2K70EXr007441@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1553 [1] Bro Johanna Amann - 2016-03-18 - Normal Please merge topic/johanna/filter_subnet_table BIT-1547 [2] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1533 [3] Bro Justin Azoff - 2016-03-17 - Low mysql analyzer does not set service to mysql BIT-1507 [4] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [5] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ------------ ---------- ------------------------------------- #52 [6] bro J-Gras [7] 2016-01-18 Fixed matching mail address intel [8] #18 [9] bro-plugins jshlbrd [10] 2016-03-03 SSDP analyzer [11] [1] BIT-1553 https://bro-tracker.atlassian.net/browse/BIT-1553 [2] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [3] BIT-1533 https://bro-tracker.atlassian.net/browse/BIT-1533 [4] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [5] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [6] Pull Request #52 https://github.com/bro/bro/pull/52 [7] J-Gras https://github.com/J-Gras [8] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [9] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [10] jshlbrd https://github.com/jshlbrd [11] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Sun Mar 20 11:22:01 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Sun, 20 Mar 2016 13:22:01 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1556) bro-2.4.1 fails to compile broker with -march=i686 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1556: ------------------------------- Resolution: No longer applies Status: Closed (was: Open) I think this one was fixed a while ago in master - which has special provisions for SSE2 support. That fix will make it into 2.5. If you need it backported into 2.4.1, you should be able to just use the alternative pure c implementation in the ifdef of the master version of 2.5. If I am mistaken and this is still a problem, feel free to reopen :) For completeness - the code in the current version is: {code} #ifdef BROKER_USE_SSE2 // Compare the key to all 16 stored keys __m128i cmp = _mm_cmpeq_epi8(_mm_set1_epi8(c), _mm_loadu_si128((__m128i*)p->keys.data())); // Use a mask to ignore children that don't exist int mask = (1 << n->num_children) - 1; int bitfield = _mm_movemask_epi8(cmp) & mask; if ( bitfield ) { auto i = __builtin_ctz(bitfield); return {&p->children[i], i}; } #else for ( int i = 0; i < n->num_children; ++i ) if ( p->keys[i] == c ) return {&p->children[i], i}; #endif {code} > bro-2.4.1 fails to compile broker with -march=i686 > -------------------------------------------------- > > Key: BIT-1556 > URL: https://bro-tracker.atlassian.net/browse/BIT-1556 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro, Broker > Affects Versions: 2.4 > Environment: x86 chroot on Gentoo, with generic settings. E.g. -march=i686. > Reporter: M.B. > Labels: build > Attachments: build.log > > > Build fails due to missing SSE support. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) From noreply at bro.org Mon Mar 21 00:00:24 2016 From: noreply at bro.org (Merge Tracker) Date: Mon, 21 Mar 2016 00:00:24 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603210700.u2L70OGZ024394@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1553 [1] Bro Johanna Amann - 2016-03-18 - Normal Please merge topic/johanna/filter_subnet_table BIT-1547 [2] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1533 [3] Bro Justin Azoff - 2016-03-17 - Low mysql analyzer does not set service to mysql BIT-1507 [4] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [5] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ------------ ---------- ------------------------------------- #52 [6] bro J-Gras [7] 2016-01-18 Fixed matching mail address intel [8] #18 [9] bro-plugins jshlbrd [10] 2016-03-03 SSDP analyzer [11] [1] BIT-1553 https://bro-tracker.atlassian.net/browse/BIT-1553 [2] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [3] BIT-1533 https://bro-tracker.atlassian.net/browse/BIT-1533 [4] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [5] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [6] Pull Request #52 https://github.com/bro/bro/pull/52 [7] J-Gras https://github.com/J-Gras [8] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [9] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [10] jshlbrd https://github.com/jshlbrd [11] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Mon Mar 21 07:20:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Mon, 21 Mar 2016 09:20:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1557) broccoli code examples don't compile In-Reply-To: References: Message-ID: Daniel Thayer created BIT-1557: ---------------------------------- Summary: broccoli code examples don't compile Key: BIT-1557 URL: https://bro-tracker.atlassian.net/browse/BIT-1557 Project: Bro Issue Tracker Issue Type: Task Components: Broccoli Reporter: Daniel Thayer Priority: Low In the broccoli manual, there are code examples, and some of them contain errors that prevent the code from compiling. -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jira at bro-tracker.atlassian.net Mon Mar 21 07:21:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Mon, 21 Mar 2016 09:21:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1557) broccoli code examples don't compile In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer reassigned BIT-1557: ---------------------------------- Assignee: Daniel Thayer > broccoli code examples don't compile > ------------------------------------ > > Key: BIT-1557 > URL: https://bro-tracker.atlassian.net/browse/BIT-1557 > Project: Bro Issue Tracker > Issue Type: Task > Components: Broccoli > Reporter: Daniel Thayer > Assignee: Daniel Thayer > Priority: Low > > In the broccoli manual, there are code examples, and some of them contain > errors that prevent the code from compiling. -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jsiwek at illinois.edu Mon Mar 21 09:43:08 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Mon, 21 Mar 2016 16:43:08 +0000 Subject: [Bro-Dev] Broker & CAF includes In-Reply-To: <20160319032057.GJ8458@shogun> References: <20160319032057.GJ8458@shogun> Message-ID: <29FF882F-3D9A-4E9D-9A29-A59169DDD8BD@illinois.edu> > On Mar 18, 2016, at 10:20 PM, Matthias Vallentin wrote: > > During Broker refactoring, I noticed the following: all headers in > broker/* include either standard library headers or Broker headers. This > appears to be by design, which makes sense to me. > > As a library writer, one faces the tricky question of exposing headers > from dependencies. For example, Broker currently has it's own > broker::util::optional, which ships as a (now outdated) copy of the > corresponding CAF source. I am inclined to change this copy to an > include that points directly into CAF headers, with the following > rationale: Broker already depends on CAF, and a system that has CAF > installed always ships with CAF headers. (Strictly speaking, we're not > copying the code of into broker either, but relying on it via > an include.) Just thought I?d share the logic behind the original decision. It was by design not to expose any CAF features directly in the public API of Broker, sort of as a proof that Broker?s interface would be sound enough to still work with arbitrary messaging back-ends/libraries besides CAF ? i.e. treat the use of CAF an implementation detail. Though, now I think this a case where including a CAF header might be an improvement and doesn?t defeat the original intention of treating CAF like an implementation detail since ?optional? types aren?t a CAF-specific concept to begin with. The problem was just that they don?t come standard until c++17, I needed to get them some place, but the arbitrary rule I had for myself at the time said to generally not include CAF things in Broker?s public API. At the time, the risk of a copied version getting outdated seemed a lower priority to me than keeping Broker?s interface/design more simple/coherent in my head. I think this is the only instance of this type of situation popping up when I was working on Broker and I can?t recall any other reasoning that led me handle it that way, so you?re proposed change looks good to me. Hope the explanation helps. - Jon From vallentin at icir.org Mon Mar 21 10:36:35 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Mon, 21 Mar 2016 10:36:35 -0700 Subject: [Bro-Dev] Broker & CAF includes In-Reply-To: <29FF882F-3D9A-4E9D-9A29-A59169DDD8BD@illinois.edu> References: <20160319032057.GJ8458@shogun> <29FF882F-3D9A-4E9D-9A29-A59169DDD8BD@illinois.edu> Message-ID: <20160321173635.GK43666@samurai.ICIR.org> Thanks for chiming in, Jon. > [..] i.e. treat the use of CAF an implementation detail. This is the clean way to think about layering and creating abstractions. It applies to the API perspective, though. As long as CAF internals are hidden from a Broker user, we are good. The "implementation detail" maxim lead to artifacts like PIMPL. This certainly made sense at the time where we considered multiple messaging backends. At this point, we are invested into CAF, and I don't think switching will happen anytime soon. Therefore, I don't think we need to keep up the implementation-hiding abstractions, such as PIMPL, which come at the cost of development productivity and performance (they are essentially a compiler firewall due to type erasure). Moving forward, I plan to remove the PIMPL design while keeping CAF hidden from the Broker API, but we'll see more CAF code in Broker headers. That's fine in my thinking, because anyone developing and compiling a Broker application must have CAF installed anyway. > At the time, the risk of a copied version getting outdated seemed a > lower priority to me than keeping Broker?s interface/design more > simple/coherent in my head. And to be clear: that rationale totally makes sense in this context and at the time of writing. Matthias From robin at icir.org Mon Mar 21 11:29:35 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 21 Mar 2016 11:29:35 -0700 Subject: [Bro-Dev] Broker & CAF includes In-Reply-To: <20160321173635.GK43666@samurai.ICIR.org> References: <20160319032057.GJ8458@shogun> <29FF882F-3D9A-4E9D-9A29-A59169DDD8BD@illinois.edu> <20160321173635.GK43666@samurai.ICIR.org> Message-ID: <20160321182935.GI13137@icir.org> On Mon, Mar 21, 2016 at 10:36 -0700, you wrote: > That's fine in my thinking, because anyone developing and compiling a > Broker application must have CAF installed anyway. Yeah, I agree, sounds like the right strategy at this point. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jira at bro-tracker.atlassian.net Mon Mar 21 12:02:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 21 Mar 2016 14:02:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1553) Please merge topic/johanna/filter_subnet_table In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1553: --------------------------------- Assignee: Robin Sommer > Please merge topic/johanna/filter_subnet_table > ---------------------------------------------- > > Key: BIT-1553 > URL: https://bro-tracker.atlassian.net/browse/BIT-1553 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > > Please merge topic/johanna/filter_subnet_table > This branch adds the filter_subnet_table bif. This bif works similar to the matching_subnet bif. The difference is that, instead of returning a vector of the subnets that match, we return a filtered view of the original set/table only containing the changed subnets. > The branch also fixes a small bug in TableVal::UpdateTimestamp (ReadOperation only has to be called when LoggingAccess() is true). -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jira at bro-tracker.atlassian.net Mon Mar 21 12:22:00 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 21 Mar 2016 14:22:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1533) mysql analyzer does not set service to mysql In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1533: --------------------------------- Assignee: Robin Sommer > mysql analyzer does not set service to mysql > -------------------------------------------- > > Key: BIT-1533 > URL: https://bro-tracker.atlassian.net/browse/BIT-1533 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Justin Azoff > Assignee: Robin Sommer > Priority: Low > > The mysql analyzer does not set the service to mysql. The result of this is that conn.log and known_services do not show 'mysql' anywhere. -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jira at bro-tracker.atlassian.net Mon Mar 21 13:04:01 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Mon, 21 Mar 2016 15:04:01 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1557) broccoli code examples don't compile In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1557: ------------------------------- Fix Version/s: 2.5 > broccoli code examples don't compile > ------------------------------------ > > Key: BIT-1557 > URL: https://bro-tracker.atlassian.net/browse/BIT-1557 > Project: Bro Issue Tracker > Issue Type: Task > Components: Broccoli > Reporter: Daniel Thayer > Assignee: Daniel Thayer > Priority: Low > Fix For: 2.5 > > > In the broccoli manual, there are code examples, and some of them contain > errors that prevent the code from compiling. -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jira at bro-tracker.atlassian.net Mon Mar 21 13:04:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Mon, 21 Mar 2016 15:04:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1557) broccoli code examples don't compile In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25101#comment-25101 ] Daniel Thayer commented on BIT-1557: ------------------------------------ In branch "topic/dnthayer/ticket1557" in the broccoli repo, I've fixed the errors that prevent the code examples from compiling (along with a few other obvious errors that I noticed). > broccoli code examples don't compile > ------------------------------------ > > Key: BIT-1557 > URL: https://bro-tracker.atlassian.net/browse/BIT-1557 > Project: Bro Issue Tracker > Issue Type: Task > Components: Broccoli > Reporter: Daniel Thayer > Assignee: Daniel Thayer > Priority: Low > > In the broccoli manual, there are code examples, and some of them contain > errors that prevent the code from compiling. -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jira at bro-tracker.atlassian.net Mon Mar 21 13:04:01 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Mon, 21 Mar 2016 15:04:01 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1557) broccoli code examples don't compile In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1557: ------------------------------- Status: Merge Request (was: Open) Assignee: (was: Daniel Thayer) > broccoli code examples don't compile > ------------------------------------ > > Key: BIT-1557 > URL: https://bro-tracker.atlassian.net/browse/BIT-1557 > Project: Bro Issue Tracker > Issue Type: Task > Components: Broccoli > Reporter: Daniel Thayer > Priority: Low > Fix For: 2.5 > > > In the broccoli manual, there are code examples, and some of them contain > errors that prevent the code from compiling. -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From asharma at lbl.gov Mon Mar 21 15:44:15 2016 From: asharma at lbl.gov (Aashish Sharma) Date: Mon, 21 Mar 2016 15:44:15 -0700 Subject: [Bro-Dev] MOTS and bro ? Message-ID: <20160321224412.GB12327@yaksha.lbl.gov> I got a query from ANL about Bro's capability to detect MOTS: "I had a question for you ? I was at a talk last week, and someone was talking about a Man on the Side attack. The presenter had indicated that suricata was currently the only tool doing this detection, but that they thought an update to bro was in work ? that would add that capability into bro as well. Was the speaker correct ? Do you know if bro currently can detect MOTS ? " Wondering is MOTS detection this something we worry about in bro world and Any feedback for my reply ? Aashish From mfischer at ICSI.Berkeley.EDU Mon Mar 21 16:24:12 2016 From: mfischer at ICSI.Berkeley.EDU (Mathias Fischer) Date: Tue, 22 Mar 2016 00:24:12 +0100 Subject: [Bro-Dev] Broker bug: routing loops Message-ID: <56F0829C.5000401@icsi.berkeley.edu> I found a bug in the bro-part of broker in bro/src/EventHandler.cc and bro/src/Event.h. I came across this when integrating my multi-hop capable broker into bro in a deep-cluster setup. The bug causes routing loops in between two directly peered bros when both have subscribed to exactly the same prefix. The current broker-integration branch is also affected by this. This remained unnoticed until now, because peered bros in a cluster-setup always use distinct subscription prefixes.However, that might not be the case with future (deep cluster) deployments anymore. I created a new branch of the broker-integration branch (topic/mfischer/broker-fix) that fixes this bug. I also added another test for it: bro/testing/btest/broker/remote_same_prefix.bro. If there are no objections, I (or Daniel?) will merge it into the broker-integration branch. Mathias From robin at icir.org Mon Mar 21 18:12:53 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 21 Mar 2016 18:12:53 -0700 Subject: [Bro-Dev] Broker bug: routing loops In-Reply-To: <56F0829C.5000401@icsi.berkeley.edu> References: <56F0829C.5000401@icsi.berkeley.edu> Message-ID: <20160322011253.GO13137@icir.org> On Tue, Mar 22, 2016 at 00:24 +0100, you wrote: > If there are no objections, I (or Daniel?) will merge it into the > broker-integration branch. Let's merge it into master first, master can then be merged into the integration branch. It's better to keep things as separate as possible (I realize that the integration already has quite a bit in there ...) Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From noreply at bro.org Tue Mar 22 00:00:20 2016 From: noreply at bro.org (Merge Tracker) Date: Tue, 22 Mar 2016 00:00:20 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603220700.u2M70KVf010480@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1557 [1] Broccoli Daniel Thayer - 2016-03-21 2.5 Low broccoli code examples don't compile BIT-1553 [2] Bro Johanna Amann Robin Sommer 2016-03-21 - Normal Please merge topic/johanna/filter_subnet_table BIT-1547 [3] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1533 [4] Bro Justin Azoff Robin Sommer 2016-03-21 - Low mysql analyzer does not set service to mysql BIT-1507 [5] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [6] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- ------------ ---------- ------------------------------------- #52 [7] bro J-Gras [8] 2016-01-18 Fixed matching mail address intel [9] #18 [10] bro-plugins jshlbrd [11] 2016-03-03 SSDP analyzer [12] [1] BIT-1557 https://bro-tracker.atlassian.net/browse/BIT-1557 [2] BIT-1553 https://bro-tracker.atlassian.net/browse/BIT-1553 [3] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [4] BIT-1533 https://bro-tracker.atlassian.net/browse/BIT-1533 [5] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [6] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [7] Pull Request #52 https://github.com/bro/bro/pull/52 [8] J-Gras https://github.com/J-Gras [9] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [10] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [11] jshlbrd https://github.com/jshlbrd [12] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Tue Mar 22 08:00:01 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 22 Mar 2016 10:00:01 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1553) Please merge topic/johanna/filter_subnet_table In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1553: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Please merge topic/johanna/filter_subnet_table > ---------------------------------------------- > > Key: BIT-1553 > URL: https://bro-tracker.atlassian.net/browse/BIT-1553 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > > Please merge topic/johanna/filter_subnet_table > This branch adds the filter_subnet_table bif. This bif works similar to the matching_subnet bif. The difference is that, instead of returning a vector of the subnets that match, we return a filtered view of the original set/table only containing the changed subnets. > The branch also fixes a small bug in TableVal::UpdateTimestamp (ReadOperation only has to be called when LoggingAccess() is true). -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jira at bro-tracker.atlassian.net Tue Mar 22 08:00:01 2016 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 22 Mar 2016 10:00:01 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1533) mysql analyzer does not set service to mysql In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1533: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > mysql analyzer does not set service to mysql > -------------------------------------------- > > Key: BIT-1533 > URL: https://bro-tracker.atlassian.net/browse/BIT-1533 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Justin Azoff > Assignee: Robin Sommer > Priority: Low > > The mysql analyzer does not set the service to mysql. The result of this is that conn.log and known_services do not show 'mysql' anywhere. -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jira at bro-tracker.atlassian.net Tue Mar 22 12:42:00 2016 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Tue, 22 Mar 2016 14:42:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1558) Bro's ascii formatter writing out scientific notation In-Reply-To: References: Message-ID: Seth Hall created BIT-1558: ------------------------------ Summary: Bro's ascii formatter writing out scientific notation Key: BIT-1558 URL: https://bro-tracker.atlassian.net/browse/BIT-1558 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: git/master Reporter: Seth Hall >From the mailing list: ``` Hello, in the x509.log normally the values regarding certificate.not_valid_before & certificate.not_valid_after look like: 1444082400.000000 1475791199.000000 I found some value like this: -3600.000000 2.153226e+09 Is it possible to modify something in order to have 2153226000 instead 2.153226e+09 ? ``` Bro's formatter's shouldn't use scientific notation because it complicates parsing of the data. -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jira at bro-tracker.atlassian.net Tue Mar 22 13:51:01 2016 From: jira at bro-tracker.atlassian.net (Matthias Vallentin (JIRA)) Date: Tue, 22 Mar 2016 15:51:01 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1558) Bro's ascii formatter writing out scientific notation In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25102#comment-25102 ] Matthias Vallentin commented on BIT-1558: ----------------------------------------- Looks like a missing `std::fixed` in an I/O stream, or missing conversion from double to (un)signed integer. > Bro's ascii formatter writing out scientific notation > ----------------------------------------------------- > > Key: BIT-1558 > URL: https://bro-tracker.atlassian.net/browse/BIT-1558 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Seth Hall > > From the mailing list: > ``` > Hello, > in the x509.log normally the values regarding certificate.not_valid_before & certificate.not_valid_after look like: > 1444082400.000000 1475791199.000000 > I found some value like this: > -3600.000000 2.153226e+09 > Is it possible to modify something in order to have 2153226000 instead 2.153226e+09 ? > ``` > Bro's formatter's shouldn't use scientific notation because it complicates parsing of the data. -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From noreply at bro.org Wed Mar 23 00:00:19 2016 From: noreply at bro.org (Merge Tracker) Date: Wed, 23 Mar 2016 00:00:19 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603230700.u2N70J9s029238@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1557 [1] Broccoli Daniel Thayer - 2016-03-21 2.5 Low broccoli code examples don't compile BIT-1547 [2] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1507 [3] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [4] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- ------------ ---------- ---------------------------------------------- #62 [5] bro aeppert [6] 2016-03-22 Add a means of disabling the SMB files log [7] #61 [8] bro aeppert [9] 2016-03-22 Add uid to smb_ntlm_authenticate [10] #60 [11] bro aeppert [12] 2016-03-22 Add uid to AuthInfo [13] #52 [14] bro J-Gras [15] 2016-01-18 Fixed matching mail address intel [16] #18 [17] bro-plugins jshlbrd [18] 2016-03-03 SSDP analyzer [19] [1] BIT-1557 https://bro-tracker.atlassian.net/browse/BIT-1557 [2] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [3] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [4] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [5] Pull Request #62 https://github.com/bro/bro/pull/62 [6] aeppert https://github.com/aeppert [7] Merge Pull Request #62 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-7 [8] Pull Request #61 https://github.com/bro/bro/pull/61 [9] aeppert https://github.com/aeppert [10] Merge Pull Request #61 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-6 [11] Pull Request #60 https://github.com/bro/bro/pull/60 [12] aeppert https://github.com/aeppert [13] Merge Pull Request #60 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-5 [14] Pull Request #52 https://github.com/bro/bro/pull/52 [15] J-Gras https://github.com/J-Gras [16] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [17] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [18] jshlbrd https://github.com/jshlbrd [19] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Wed Mar 23 13:05:00 2016 From: jira at bro-tracker.atlassian.net (Nick Allen (JIRA)) Date: Wed, 23 Mar 2016 15:05:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1559) Bro-Plugins - Send Each Log Stream to Different Kafka Topic In-Reply-To: References: Message-ID: Nick Allen created BIT-1559: ------------------------------- Summary: Bro-Plugins - Send Each Log Stream to Different Kafka Topic Key: BIT-1559 URL: https://bro-tracker.atlassian.net/browse/BIT-1559 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Reporter: Nick Allen The current Kafka log writer sends all log streams (Conn, Http, Dns) to the same Kafka topic. Allow the user to configure a separate topic for each log stream. -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jira at bro-tracker.atlassian.net Wed Mar 23 13:54:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Wed, 23 Mar 2016 15:54:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1560) BroControl unhappy when host dies during shutdown In-Reply-To: References: Message-ID: Johanna Amann created BIT-1560: ---------------------------------- Summary: BroControl unhappy when host dies during shutdown Key: BIT-1560 URL: https://bro-tracker.atlassian.net/browse/BIT-1560 Project: Bro Issue Tracker Issue Type: Problem Components: BroControl Affects Versions: git/master Reporter: Johanna Amann Fix For: 2.5 BroControl currently seems to get rather unhappy if a node crashes while Bro is being shut down. The output is something along these lines (it retries quite a few times and takes a while): {code} Error: failed to send stop signal to worker-19-1 Error: failed to send stop signal to worker-19-2 Error: cannot connect to worker-19-1 Error: cannot connect to worker-19-2 ssh: connect to host 10.0.1.69 port 22: Connection refused ssh: connect to host 10.0.1.83 port 22: Host is down Error: cannot connect to worker-19-1 Error: cannot connect to worker-19-2 ssh: connect to host 10.0.1.83 port 22: Host is down ssh: connect to host 10.0.1.83 port 22: Host is down Error: cannot connect to worker-19-1 Error: cannot connect to worker-19-2 ssh: connect to host 10.0.1.83 port 22: Host is down ssh: connect to host 10.0.1.83 port 22: Host is down Error: cannot connect to worker-19-1 Error: cannot connect to worker-19-2 ssh: connect to host 10.0.1.83 port 22: Host is down ssh: connect to host 10.0.1.83 port 22: Host is down Error: cannot connect to worker-19-1 Error: cannot connect to worker-19-2 ssh: connect to host 10.0.1.83 port 22: Host is down ssh: connect to host 10.0.1.83 port 22: Host is down Error: cannot connect to worker-19-1 Error: cannot connect to worker-19-2 ssh: connect to host 10.0.1.83 port 22: Host is down ssh: connect to host 10.0.1.83 port 22: Host is down ... ssh: connect to host 10.0.1.83 port 22: Host is down Error: cannot connect to worker-19-1 Error: cannot connect to worker-19-2 Error: 'str' object has no attribute 'type' [BroControl] > {code} -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jira at bro-tracker.atlassian.net Wed Mar 23 14:01:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Wed, 23 Mar 2016 16:01:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1560) BroControl unhappy when host dies during shutdown In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25200#comment-25200 ] Johanna Amann commented on BIT-1560: ------------------------------------ For completeness sake - apparently not all nodes are shut down in this scenario - brocontrol stopped shutting down nodes before trying to shut down the manager and proxies. broctl version is: BroControl Version 1.4-77 bro version is: bro version 2.4-284 > BroControl unhappy when host dies during shutdown > ------------------------------------------------- > > Key: BIT-1560 > URL: https://bro-tracker.atlassian.net/browse/BIT-1560 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Reporter: Johanna Amann > Fix For: 2.5 > > > BroControl currently seems to get rather unhappy if a node crashes while Bro is being shut down. The output is something along these lines (it retries quite a few times and takes a while): > {code} > Error: failed to send stop signal to worker-19-1 > Error: failed to send stop signal to worker-19-2 > Error: cannot connect to worker-19-1 > Error: cannot connect to worker-19-2 > ssh: connect to host 10.0.1.69 port 22: Connection refused > ssh: connect to host 10.0.1.83 port 22: Host is down > Error: cannot connect to worker-19-1 > Error: cannot connect to worker-19-2 > ssh: connect to host 10.0.1.83 port 22: Host is down > ssh: connect to host 10.0.1.83 port 22: Host is down > Error: cannot connect to worker-19-1 > Error: cannot connect to worker-19-2 > ssh: connect to host 10.0.1.83 port 22: Host is down > ssh: connect to host 10.0.1.83 port 22: Host is down > Error: cannot connect to worker-19-1 > Error: cannot connect to worker-19-2 > ssh: connect to host 10.0.1.83 port 22: Host is down > ssh: connect to host 10.0.1.83 port 22: Host is down > Error: cannot connect to worker-19-1 > Error: cannot connect to worker-19-2 > ssh: connect to host 10.0.1.83 port 22: Host is down > ssh: connect to host 10.0.1.83 port 22: Host is down > Error: cannot connect to worker-19-1 > Error: cannot connect to worker-19-2 > ssh: connect to host 10.0.1.83 port 22: Host is down > ssh: connect to host 10.0.1.83 port 22: Host is down > ... > ssh: connect to host 10.0.1.83 port 22: Host is down > Error: cannot connect to worker-19-1 > Error: cannot connect to worker-19-2 > Error: 'str' object has no attribute 'type' > [BroControl] > > {code} -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From noreply at bro.org Thu Mar 24 00:00:23 2016 From: noreply at bro.org (Merge Tracker) Date: Thu, 24 Mar 2016 00:00:23 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603240700.u2O70N8w012654@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1557 [1] Broccoli Daniel Thayer - 2016-03-21 2.5 Low broccoli code examples don't compile BIT-1547 [2] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1507 [3] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [4] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- --------------- ---------- ----------------------------------------------------------------------- #61 [5] bro aeppert [6] 2016-03-22 Add uid to smb_ntlm_authenticate [7] #60 [8] bro aeppert [9] 2016-03-22 Add uid to AuthInfo [10] #52 [11] bro J-Gras [12] 2016-01-18 Fixed matching mail address intel [13] #22 [14] bro-plugins nickwallen [15] 2016-03-23 BIT-1559 Bro-Plugins Send each log stream to different kafka topic [16] #18 [17] bro-plugins jshlbrd [18] 2016-03-03 SSDP analyzer [19] [1] BIT-1557 https://bro-tracker.atlassian.net/browse/BIT-1557 [2] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [3] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [4] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [5] Pull Request #61 https://github.com/bro/bro/pull/61 [6] aeppert https://github.com/aeppert [7] Merge Pull Request #61 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-6 [8] Pull Request #60 https://github.com/bro/bro/pull/60 [9] aeppert https://github.com/aeppert [10] Merge Pull Request #60 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-5 [11] Pull Request #52 https://github.com/bro/bro/pull/52 [12] J-Gras https://github.com/J-Gras [13] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [14] Pull Request #22 https://github.com/bro/bro-plugins/pull/22 [15] nickwallen https://github.com/nickwallen [16] Merge Pull Request #22 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git support-many-kafka-topics [17] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [18] jshlbrd https://github.com/jshlbrd [19] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From mfischer at ICSI.Berkeley.EDU Thu Mar 24 05:45:44 2016 From: mfischer at ICSI.Berkeley.EDU (Mathias Fischer) Date: Thu, 24 Mar 2016 13:45:44 +0100 Subject: [Bro-Dev] Broker bug: routing loops In-Reply-To: <20160322011253.GO13137@icir.org> References: <56F0829C.5000401@icsi.berkeley.edu> <20160322011253.GO13137@icir.org> Message-ID: <56F3E178.40802@icsi.berkeley.edu> Ok, fine with that. I will integrate my bugfix into master the next days. Mathias Am 22.03.2016 um 02:12 schrieb Robin Sommer: > > On Tue, Mar 22, 2016 at 00:24 +0100, you wrote: > >> If there are no objections, I (or Daniel?) will merge it into the >> broker-integration branch. > Let's merge it into master first, master can then be merged into the > integration branch. It's better to keep things as separate as possible > (I realize that the integration already has quite a bit in there ...) > > Robin > -- Mathias Fischer International Computer Science Institute Berkeley, USA http://www.icsi.berkeley.edu/~mfischer/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 884 bytes Desc: OpenPGP digital signature Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160324/292a0fe2/attachment.bin From jsiwek at illinois.edu Thu Mar 24 05:50:09 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Thu, 24 Mar 2016 12:50:09 +0000 Subject: [Bro-Dev] Broker & CAF includes In-Reply-To: <20160321173635.GK43666@samurai.ICIR.org> References: <20160319032057.GJ8458@shogun> <29FF882F-3D9A-4E9D-9A29-A59169DDD8BD@illinois.edu> <20160321173635.GK43666@samurai.ICIR.org> Message-ID: > On Mar 21, 2016, at 12:36 PM, Matthias Vallentin wrote: > > The "implementation detail" maxim lead to artifacts like PIMPL. This > certainly made sense at the time where we considered multiple messaging > backends. At this point, we are invested into CAF, and I don't think > switching will happen anytime soon. Therefore, I don't think we need to > keep up the implementation-hiding abstractions, such as PIMPL, which > come at the cost of development productivity and performance (they are > essentially a compiler firewall due to type erasure). A possible benefit to PIMPL is improving ability to maintain binary compatibility across releases (i.e. programs dynamically linked to Broker don?t need a recompile if a new release is binary compatible). Providing stable-ish ABIs seems like something libraries often do, so I tried to plan that in to Broker. Don?t know if I did that well, or there?s better strategies to use, or I was the only one worried about that to begin with, but thought I?d mention it just in case it wasn?t even on your radar. - Jon From jira at bro-tracker.atlassian.net Thu Mar 24 09:29:03 2016 From: jira at bro-tracker.atlassian.net (Vlad Grigorescu (JIRA)) Date: Thu, 24 Mar 2016 11:29:03 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1528) SNMP and SIP scans show up in known services. In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25201#comment-25201 ] Vlad Grigorescu commented on BIT-1528: -------------------------------------- Completed in topic/vladg/bit-1528. > SNMP and SIP scans show up in known services. > --------------------------------------------- > > Key: BIT-1528 > URL: https://bro-tracker.atlassian.net/browse/BIT-1528 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Justin Azoff > Assignee: Vlad Grigorescu > Fix For: 2.5 > > > It appears that single packet SIP and SNMP scans cause the destination host to end up in known_services as running a SIP or SNMP service, even though they are not running that service and did not respond to the packet. -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jira at bro-tracker.atlassian.net Thu Mar 24 09:29:03 2016 From: jira at bro-tracker.atlassian.net (Vlad Grigorescu (JIRA)) Date: Thu, 24 Mar 2016 11:29:03 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1528) SNMP and SIP scans show up in known services. In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Grigorescu updated BIT-1528: --------------------------------- Status: Merge Request (was: Open) Assignee: (was: Vlad Grigorescu) > SNMP and SIP scans show up in known services. > --------------------------------------------- > > Key: BIT-1528 > URL: https://bro-tracker.atlassian.net/browse/BIT-1528 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Reporter: Justin Azoff > Fix For: 2.5 > > > It appears that single packet SIP and SNMP scans cause the destination host to end up in known_services as running a SIP or SNMP service, even though they are not running that service and did not respond to the packet. -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jira at bro-tracker.atlassian.net Thu Mar 24 14:02:02 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Thu, 24 Mar 2016 16:02:02 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1446) Remove the dummy Broker framework In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25202#comment-25202 ] Johanna Amann commented on BIT-1446: ------------------------------------ In the same line - currently there is an @if around the openflow and netcontrol frameworks that disables them when broker is not enabled. Once broker is required, this should be removed. > Remove the dummy Broker framework > --------------------------------- > > Key: BIT-1446 > URL: https://bro-tracker.atlassian.net/browse/BIT-1446 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Environment: For unit testing with Broker disabled, there's currently a dummy script-level framework to fill in. > Unfortunately that dummy framework is the one that ends up getting documented, overriding the the actual one. > Now that Broker is mandatory, we should just remove the dummy. > Reporter: Robin Sommer > Fix For: 2.5 > > -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From noreply at bro.org Fri Mar 25 00:00:21 2016 From: noreply at bro.org (Merge Tracker) Date: Fri, 25 Mar 2016 00:00:21 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603250700.u2P70LXB019660@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1557 [1] Broccoli Daniel Thayer - 2016-03-21 2.5 Low broccoli code examples don't compile BIT-1547 [2] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1528 [3] Bro Justin Azoff - 2016-03-24 2.5 Normal SNMP and SIP scans show up in known services. BIT-1507 [4] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [5] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- --------------- ---------- ----------------------------------------------------------------------- #61 [6] bro aeppert [7] 2016-03-22 Add uid to smb_ntlm_authenticate [8] #60 [9] bro aeppert [10] 2016-03-22 Add uid to AuthInfo [11] #52 [12] bro J-Gras [13] 2016-01-18 Fixed matching mail address intel [14] #22 [15] bro-plugins nickwallen [16] 2016-03-23 BIT-1559 Bro-Plugins Send each log stream to different kafka topic [17] #18 [18] bro-plugins jshlbrd [19] 2016-03-03 SSDP analyzer [20] [1] BIT-1557 https://bro-tracker.atlassian.net/browse/BIT-1557 [2] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [3] BIT-1528 https://bro-tracker.atlassian.net/browse/BIT-1528 [4] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [5] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [6] Pull Request #61 https://github.com/bro/bro/pull/61 [7] aeppert https://github.com/aeppert [8] Merge Pull Request #61 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-6 [9] Pull Request #60 https://github.com/bro/bro/pull/60 [10] aeppert https://github.com/aeppert [11] Merge Pull Request #60 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-5 [12] Pull Request #52 https://github.com/bro/bro/pull/52 [13] J-Gras https://github.com/J-Gras [14] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [15] Pull Request #22 https://github.com/bro/bro-plugins/pull/22 [16] nickwallen https://github.com/nickwallen [17] Merge Pull Request #22 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git support-many-kafka-topics [18] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [19] jshlbrd https://github.com/jshlbrd [20] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jira at bro-tracker.atlassian.net Fri Mar 25 08:37:00 2016 From: jira at bro-tracker.atlassian.net (Aaron Eppert (JIRA)) Date: Fri, 25 Mar 2016 10:37:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1561) Pull Request to fix resource leaking in BroControl Python API In-Reply-To: References: Message-ID: Aaron Eppert created BIT-1561: --------------------------------- Summary: Pull Request to fix resource leaking in BroControl Python API Key: BIT-1561 URL: https://bro-tracker.atlassian.net/browse/BIT-1561 Project: Bro Issue Tracker Issue Type: Patch Components: BroControl Reporter: Aaron Eppert https://github.com/bro/broctl/pull/4 -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jira at bro-tracker.atlassian.net Fri Mar 25 08:57:00 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 25 Mar 2016 10:57:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1561) Pull Request to fix resource leaking in BroControl Python API In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1561: ------------------------------- Status: Merge Request (was: Open) > Pull Request to fix resource leaking in BroControl Python API > ------------------------------------------------------------- > > Key: BIT-1561 > URL: https://bro-tracker.atlassian.net/browse/BIT-1561 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Reporter: Aaron Eppert > Labels: broctl > > https://github.com/bro/broctl/pull/4 -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From vallentin at icir.org Fri Mar 25 09:16:53 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Fri, 25 Mar 2016 09:16:53 -0700 Subject: [Bro-Dev] Broker & CAF includes In-Reply-To: References: <20160319032057.GJ8458@shogun> <29FF882F-3D9A-4E9D-9A29-A59169DDD8BD@illinois.edu> <20160321173635.GK43666@samurai.ICIR.org> Message-ID: <20160325161653.GR43666@samurai.ICIR.org> > Providing stable-ish ABIs seems like something libraries often do, so > I tried to plan that in to Broker. Don?t know if I did that well, or > there?s better strategies to use, or I was the only one worried about > that to begin with, but thought I?d mention it just in case it wasn?t > even on your radar. Indeed, it wasn't on my radar that you employed PIMPL to achieve ABI compatibility. At this point, I'm inclined to move towards a more light-weight model that is less robust against ABI changes. I believe we still need more experience with the API. Once the API matures, hiding central implementation aspects to increase ABI stability becomes the next priority to improve medium- to long-term release compatibility. Does that sound reasonable? Matthias From noreply at bro.org Sat Mar 26 00:00:23 2016 From: noreply at bro.org (Merge Tracker) Date: Sat, 26 Mar 2016 00:00:23 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603260700.u2Q70Np9014450@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------------- BIT-1561 [1] BroControl Aaron Eppert - 2016-03-25 - Normal Pull Request to fix resource leaking in BroControl Python API BIT-1557 [2] Broccoli Daniel Thayer - 2016-03-21 2.5 Low broccoli code examples don't compile BIT-1547 [3] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1528 [4] Bro Justin Azoff - 2016-03-24 2.5 Normal SNMP and SIP scans show up in known services. BIT-1507 [5] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [6] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- --------------- ---------- ------------------------------------------------------------------------- #61 [7] bro aeppert [8] 2016-03-22 Add uid to smb_ntlm_authenticate [9] #60 [10] bro aeppert [11] 2016-03-22 Add uid to AuthInfo [12] #52 [13] bro J-Gras [14] 2016-01-18 Fixed matching mail address intel [15] #22 [16] bro-plugins nickwallen [17] 2016-03-23 BIT-1559 Bro-Plugins Send each log stream to different kafka topic [18] #18 [19] bro-plugins jshlbrd [20] 2016-03-03 SSDP analyzer [21] #4 [22] broctl aeppert [23] 2016-03-25 Fix leaking resources from never closing out the Broccoli connection [24] [1] BIT-1561 https://bro-tracker.atlassian.net/browse/BIT-1561 [2] BIT-1557 https://bro-tracker.atlassian.net/browse/BIT-1557 [3] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [4] BIT-1528 https://bro-tracker.atlassian.net/browse/BIT-1528 [5] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [6] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [7] Pull Request #61 https://github.com/bro/bro/pull/61 [8] aeppert https://github.com/aeppert [9] Merge Pull Request #61 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-6 [10] Pull Request #60 https://github.com/bro/bro/pull/60 [11] aeppert https://github.com/aeppert [12] Merge Pull Request #60 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-5 [13] Pull Request #52 https://github.com/bro/bro/pull/52 [14] J-Gras https://github.com/J-Gras [15] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [16] Pull Request #22 https://github.com/bro/bro-plugins/pull/22 [17] nickwallen https://github.com/nickwallen [18] Merge Pull Request #22 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git support-many-kafka-topics [19] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [20] jshlbrd https://github.com/jshlbrd [21] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp [22] Pull Request #4 https://github.com/bro/broctl/pull/4 [23] aeppert https://github.com/aeppert [24] Merge Pull Request #4 with git pull --no-ff --no-commit https://github.com/aeppert/broctl.git patch-1 From jira at bro-tracker.atlassian.net Sat Mar 26 08:59:00 2016 From: jira at bro-tracker.atlassian.net (rmkml (JIRA)) Date: Sat, 26 Mar 2016 10:59:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1562) Bro v2.4.1 lock with a old pcap file In-Reply-To: References: Message-ID: rmkml created BIT-1562: -------------------------- Summary: Bro v2.4.1 lock with a old pcap file Key: BIT-1562 URL: https://bro-tracker.atlassian.net/browse/BIT-1562 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.4 Environment: Tested on Ubuntu v14.04.4 LTS Reporter: rmkml Attachments: bro241lock.pcap.gz Hi, I have replayed many pcap files on Bro v2.4.1, but discovered one pcap "lock" bro process (bro never quit). Simply replay: bro241 -r bro241lock.pcap Added -C or -b have same pb. Joigned gziped pcap file. (This partial file sended to bro list around 2009 not by me...) Regards @Rmkml -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From noreply at bro.org Sun Mar 27 00:00:19 2016 From: noreply at bro.org (Merge Tracker) Date: Sun, 27 Mar 2016 00:00:19 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603270700.u2R70JKM007822@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------------- BIT-1561 [1] BroControl Aaron Eppert - 2016-03-25 - Normal Pull Request to fix resource leaking in BroControl Python API BIT-1557 [2] Broccoli Daniel Thayer - 2016-03-21 2.5 Low broccoli code examples don't compile BIT-1547 [3] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1528 [4] Bro Justin Azoff - 2016-03-24 2.5 Normal SNMP and SIP scans show up in known services. BIT-1507 [5] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [6] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- --------------- ---------- ------------------------------------------------------------------------- #63 [7] bro WilliamTom [8] 2016-03-26 Wrong regex literal in scripting doc [9] #61 [10] bro aeppert [11] 2016-03-22 Add uid to smb_ntlm_authenticate [12] #60 [13] bro aeppert [14] 2016-03-22 Add uid to AuthInfo [15] #52 [16] bro J-Gras [17] 2016-01-18 Fixed matching mail address intel [18] #22 [19] bro-plugins nickwallen [20] 2016-03-23 BIT-1559 Bro-Plugins Send each log stream to different kafka topic [21] #18 [22] bro-plugins jshlbrd [23] 2016-03-03 SSDP analyzer [24] #4 [25] broctl aeppert [26] 2016-03-25 Fix leaking resources from never closing out the Broccoli connection [27] [1] BIT-1561 https://bro-tracker.atlassian.net/browse/BIT-1561 [2] BIT-1557 https://bro-tracker.atlassian.net/browse/BIT-1557 [3] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [4] BIT-1528 https://bro-tracker.atlassian.net/browse/BIT-1528 [5] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [6] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [7] Pull Request #63 https://github.com/bro/bro/pull/63 [8] WilliamTom https://github.com/WilliamTom [9] Merge Pull Request #63 with git pull --no-ff --no-commit https://github.com/WilliamTom/bro.git master [10] Pull Request #61 https://github.com/bro/bro/pull/61 [11] aeppert https://github.com/aeppert [12] Merge Pull Request #61 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-6 [13] Pull Request #60 https://github.com/bro/bro/pull/60 [14] aeppert https://github.com/aeppert [15] Merge Pull Request #60 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-5 [16] Pull Request #52 https://github.com/bro/bro/pull/52 [17] J-Gras https://github.com/J-Gras [18] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [19] Pull Request #22 https://github.com/bro/bro-plugins/pull/22 [20] nickwallen https://github.com/nickwallen [21] Merge Pull Request #22 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git support-many-kafka-topics [22] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [23] jshlbrd https://github.com/jshlbrd [24] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp [25] Pull Request #4 https://github.com/bro/broctl/pull/4 [26] aeppert https://github.com/aeppert [27] Merge Pull Request #4 with git pull --no-ff --no-commit https://github.com/aeppert/broctl.git patch-1 From noreply at bro.org Mon Mar 28 00:00:22 2016 From: noreply at bro.org (Merge Tracker) Date: Mon, 28 Mar 2016 00:00:22 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603280700.u2S70MVq026896@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------------- BIT-1561 [1] BroControl Aaron Eppert - 2016-03-25 - Normal Pull Request to fix resource leaking in BroControl Python API BIT-1557 [2] Broccoli Daniel Thayer - 2016-03-21 2.5 Low broccoli code examples don't compile BIT-1547 [3] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1528 [4] Bro Justin Azoff - 2016-03-24 2.5 Normal SNMP and SIP scans show up in known services. BIT-1507 [5] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [6] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- --------------- ---------- ------------------------------------------------------------------------- #63 [7] bro WilliamTom [8] 2016-03-26 Wrong regex literal in scripting doc [9] #61 [10] bro aeppert [11] 2016-03-22 Add uid to smb_ntlm_authenticate [12] #60 [13] bro aeppert [14] 2016-03-22 Add uid to AuthInfo [15] #52 [16] bro J-Gras [17] 2016-01-18 Fixed matching mail address intel [18] #22 [19] bro-plugins nickwallen [20] 2016-03-23 BIT-1559 Bro-Plugins Send each log stream to different kafka topic [21] #18 [22] bro-plugins jshlbrd [23] 2016-03-03 SSDP analyzer [24] #4 [25] broctl aeppert [26] 2016-03-25 Fix leaking resources from never closing out the Broccoli connection [27] [1] BIT-1561 https://bro-tracker.atlassian.net/browse/BIT-1561 [2] BIT-1557 https://bro-tracker.atlassian.net/browse/BIT-1557 [3] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [4] BIT-1528 https://bro-tracker.atlassian.net/browse/BIT-1528 [5] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [6] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [7] Pull Request #63 https://github.com/bro/bro/pull/63 [8] WilliamTom https://github.com/WilliamTom [9] Merge Pull Request #63 with git pull --no-ff --no-commit https://github.com/WilliamTom/bro.git master [10] Pull Request #61 https://github.com/bro/bro/pull/61 [11] aeppert https://github.com/aeppert [12] Merge Pull Request #61 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-6 [13] Pull Request #60 https://github.com/bro/bro/pull/60 [14] aeppert https://github.com/aeppert [15] Merge Pull Request #60 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-5 [16] Pull Request #52 https://github.com/bro/bro/pull/52 [17] J-Gras https://github.com/J-Gras [18] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [19] Pull Request #22 https://github.com/bro/bro-plugins/pull/22 [20] nickwallen https://github.com/nickwallen [21] Merge Pull Request #22 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git support-many-kafka-topics [22] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [23] jshlbrd https://github.com/jshlbrd [24] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp [25] Pull Request #4 https://github.com/bro/broctl/pull/4 [26] aeppert https://github.com/aeppert [27] Merge Pull Request #4 with git pull --no-ff --no-commit https://github.com/aeppert/broctl.git patch-1 From jira at bro-tracker.atlassian.net Mon Mar 28 12:25:00 2016 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Mon, 28 Mar 2016 14:25:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1562) Bro v2.4.1 lock with a old pcap file In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25203#comment-25203 ] Justin Azoff commented on BIT-1562: ----------------------------------- This looks like BIT-1443. {code} $ tcpdump -n -r bro241lock.pcap |cut -d ' ' -f 1|head reading from file bro241lock.pcap, link-type EN10MB (Ethernet) 08:49:46.182640 08:49:46.182865 08:49:46.182889 08:49:46.182950 08:49:46.182970 08:49:46.183275 08:49:46.183942 -19:-28:-39.000718 -13:-6:-47.000718 08:49:46.183964 {code} > Bro v2.4.1 lock with a old pcap file > ------------------------------------ > > Key: BIT-1562 > URL: https://bro-tracker.atlassian.net/browse/BIT-1562 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.4 > Environment: Tested on Ubuntu v14.04.4 LTS > Reporter: rmkml > Attachments: bro241lock.pcap.gz > > > Hi, > I have replayed many pcap files on Bro v2.4.1, > but discovered one pcap "lock" bro process (bro never quit). > Simply replay: bro241 -r bro241lock.pcap > > Added -C or -b have same pb. > Joigned gziped pcap file. > (This partial file sended to bro list around 2009 not by me...) > Regards > @Rmkml -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From noreply at bro.org Tue Mar 29 00:00:22 2016 From: noreply at bro.org (Merge Tracker) Date: Tue, 29 Mar 2016 00:00:22 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603290700.u2T70MCx018637@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------------- BIT-1561 [1] BroControl Aaron Eppert - 2016-03-25 - Normal Pull Request to fix resource leaking in BroControl Python API BIT-1557 [2] Broccoli Daniel Thayer - 2016-03-21 2.5 Low broccoli code examples don't compile BIT-1547 [3] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1528 [4] Bro Justin Azoff - 2016-03-24 2.5 Normal SNMP and SIP scans show up in known services. BIT-1507 [5] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [6] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- --------------- ---------- ------------------------------------------------------------------------- #63 [7] bro WilliamTom [8] 2016-03-26 Wrong regex literal in scripting doc [9] #61 [10] bro aeppert [11] 2016-03-22 Add uid to smb_ntlm_authenticate [12] #60 [13] bro aeppert [14] 2016-03-22 Add uid to AuthInfo [15] #52 [16] bro J-Gras [17] 2016-01-18 Fixed matching mail address intel [18] #22 [19] bro-plugins nickwallen [20] 2016-03-23 BIT-1559 Bro-Plugins Send each log stream to different kafka topic [21] #18 [22] bro-plugins jshlbrd [23] 2016-03-03 SSDP analyzer [24] #4 [25] broctl aeppert [26] 2016-03-25 Fix leaking resources from never closing out the Broccoli connection [27] [1] BIT-1561 https://bro-tracker.atlassian.net/browse/BIT-1561 [2] BIT-1557 https://bro-tracker.atlassian.net/browse/BIT-1557 [3] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [4] BIT-1528 https://bro-tracker.atlassian.net/browse/BIT-1528 [5] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [6] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [7] Pull Request #63 https://github.com/bro/bro/pull/63 [8] WilliamTom https://github.com/WilliamTom [9] Merge Pull Request #63 with git pull --no-ff --no-commit https://github.com/WilliamTom/bro.git master [10] Pull Request #61 https://github.com/bro/bro/pull/61 [11] aeppert https://github.com/aeppert [12] Merge Pull Request #61 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-6 [13] Pull Request #60 https://github.com/bro/bro/pull/60 [14] aeppert https://github.com/aeppert [15] Merge Pull Request #60 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-5 [16] Pull Request #52 https://github.com/bro/bro/pull/52 [17] J-Gras https://github.com/J-Gras [18] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [19] Pull Request #22 https://github.com/bro/bro-plugins/pull/22 [20] nickwallen https://github.com/nickwallen [21] Merge Pull Request #22 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git support-many-kafka-topics [22] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [23] jshlbrd https://github.com/jshlbrd [24] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp [25] Pull Request #4 https://github.com/bro/broctl/pull/4 [26] aeppert https://github.com/aeppert [27] Merge Pull Request #4 with git pull --no-ff --no-commit https://github.com/aeppert/broctl.git patch-1 From noreply at bro.org Wed Mar 30 00:00:20 2016 From: noreply at bro.org (Merge Tracker) Date: Wed, 30 Mar 2016 00:00:20 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603300700.u2U70KoU021783@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------------- BIT-1561 [1] BroControl Aaron Eppert - 2016-03-25 - Normal Pull Request to fix resource leaking in BroControl Python API BIT-1557 [2] Broccoli Daniel Thayer - 2016-03-21 2.5 Low broccoli code examples don't compile BIT-1547 [3] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1528 [4] Bro Justin Azoff - 2016-03-24 2.5 Normal SNMP and SIP scans show up in known services. BIT-1507 [5] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [6] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- --------------- ---------- ------------------------------------------------------------------------- #63 [7] bro WilliamTom [8] 2016-03-26 Wrong regex literal in scripting doc [9] #61 [10] bro aeppert [11] 2016-03-22 Add uid to smb_ntlm_authenticate [12] #60 [13] bro aeppert [14] 2016-03-22 Add uid to AuthInfo [15] #52 [16] bro J-Gras [17] 2016-01-18 Fixed matching mail address intel [18] #22 [19] bro-plugins nickwallen [20] 2016-03-23 BIT-1559 Bro-Plugins Send each log stream to different kafka topic [21] #18 [22] bro-plugins jshlbrd [23] 2016-03-03 SSDP analyzer [24] #4 [25] broctl aeppert [26] 2016-03-25 Fix leaking resources from never closing out the Broccoli connection [27] [1] BIT-1561 https://bro-tracker.atlassian.net/browse/BIT-1561 [2] BIT-1557 https://bro-tracker.atlassian.net/browse/BIT-1557 [3] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [4] BIT-1528 https://bro-tracker.atlassian.net/browse/BIT-1528 [5] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [6] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [7] Pull Request #63 https://github.com/bro/bro/pull/63 [8] WilliamTom https://github.com/WilliamTom [9] Merge Pull Request #63 with git pull --no-ff --no-commit https://github.com/WilliamTom/bro.git master [10] Pull Request #61 https://github.com/bro/bro/pull/61 [11] aeppert https://github.com/aeppert [12] Merge Pull Request #61 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-6 [13] Pull Request #60 https://github.com/bro/bro/pull/60 [14] aeppert https://github.com/aeppert [15] Merge Pull Request #60 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-5 [16] Pull Request #52 https://github.com/bro/bro/pull/52 [17] J-Gras https://github.com/J-Gras [18] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [19] Pull Request #22 https://github.com/bro/bro-plugins/pull/22 [20] nickwallen https://github.com/nickwallen [21] Merge Pull Request #22 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git support-many-kafka-topics [22] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [23] jshlbrd https://github.com/jshlbrd [24] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp [25] Pull Request #4 https://github.com/bro/broctl/pull/4 [26] aeppert https://github.com/aeppert [27] Merge Pull Request #4 with git pull --no-ff --no-commit https://github.com/aeppert/broctl.git patch-1 From jira at bro-tracker.atlassian.net Wed Mar 30 08:30:01 2016 From: jira at bro-tracker.atlassian.net (Jon Schipp (JIRA)) Date: Wed, 30 Mar 2016 10:30:01 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-274) Finding lines where redefs occurred In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25204#comment-25204 ] Jon Schipp commented on BIT-274: -------------------------------- [~johanna], thanks I'll ask him. > Finding lines where redefs occurred > ----------------------------------- > > Key: BIT-274 > URL: https://bro-tracker.atlassian.net/browse/BIT-274 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: 1.5.1 > Reporter: Seth Hall > Assignee: Jon Schipp > > First, support would need added to Bro for finding all of the lines and scripts where redef's against a certain variable occurred. I would also like to see this support added through broctl. > Here's the scenario... > {noformat} > [BroControl] > find redef ignore_checksums > /usr/local/bro/share/bro/bro.init:360 const ignore_checksums = F &redef; > /usr/local/bro/share/bro/site/local.bro:133 redef ignore_checksums = T; > {noformat} > This is relating to a discussion I had about trouble people have with starting with Bro and the gotcha's encountered from enabling the cluster support. There are so many redef's happening and potentially without the user realizing it. -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jira at bro-tracker.atlassian.net Wed Mar 30 11:25:02 2016 From: jira at bro-tracker.atlassian.net (Aashish Sharma (JIRA)) Date: Wed, 30 Mar 2016 13:25:02 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1472) Bif for a new function to calculates haversine distance between two geoip locations In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25205#comment-25205 ] Aashish Sharma commented on BIT-1472: ------------------------------------- Until you are set to update libGeoIP2 API, could you add this bif to bro.bif you can later eliminate this from bro.bif or reintegrate as you see fit. > Bif for a new function to calculates haversine distance between two geoip locations > ----------------------------------------------------------------------------------- > > Key: BIT-1472 > URL: https://bro-tracker.atlassian.net/browse/BIT-1472 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: 2.4 > Reporter: Aashish Sharma > Assignee: Justin Azoff > Priority: Low > Labels: bif, function > Fix For: 2.5 > > > Merge request for: > topic/aashish/haversine > ## ## Calculates haversine distance between two geoip locations > ## > ## > ## lat1, long1, lat2, long2 > ## > ## Returns: distance in miles > ## function haversine_distance%(lat1:double, long1:double, lat2:double, long2:double %): double > accompanying bro policy in base/utils/haversine_distance_ip.bro > module GLOBAL; > ## Returns the haversine distance between two IP addresses based on GeoIP > ## database locations > ## > ## > ## orig: the address of orig connection > ## resp: the address of resp server > ## Returns: the GeoIP distance between orig and resp in miles > function haversine_distance_ip(orig: addr, resp: addr): double -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jira at bro-tracker.atlassian.net Wed Mar 30 18:50:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Wed, 30 Mar 2016 20:50:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1563) BrokerComm and BrokerStore namespaces should be combined In-Reply-To: References: Message-ID: Daniel Thayer created BIT-1563: ---------------------------------- Summary: BrokerComm and BrokerStore namespaces should be combined Key: BIT-1563 URL: https://bro-tracker.atlassian.net/browse/BIT-1563 Project: Bro Issue Tracker Issue Type: Task Components: Bro Reporter: Daniel Thayer The BrokerComm and BrokerStore namespaces should be combined to just "Broker". -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jira at bro-tracker.atlassian.net Wed Mar 30 18:51:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Wed, 30 Mar 2016 20:51:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1563) BrokerComm and BrokerStore namespaces should be combined In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1563: ------------------------------- Fix Version/s: 2.5 > BrokerComm and BrokerStore namespaces should be combined > -------------------------------------------------------- > > Key: BIT-1563 > URL: https://bro-tracker.atlassian.net/browse/BIT-1563 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro > Reporter: Daniel Thayer > Fix For: 2.5 > > > The BrokerComm and BrokerStore namespaces should be combined to > just "Broker". -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jira at bro-tracker.atlassian.net Wed Mar 30 18:54:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Wed, 30 Mar 2016 20:54:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1563) BrokerComm and BrokerStore namespaces should be combined In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25206#comment-25206 ] Daniel Thayer commented on BIT-1563: ------------------------------------ Branch "topic/dnthayer/broker-namespace" in the bro git repo contains this change. I also split the broker main.bro into two scripts, because they will become much bigger when BIF script wrappers are added to them. > BrokerComm and BrokerStore namespaces should be combined > -------------------------------------------------------- > > Key: BIT-1563 > URL: https://bro-tracker.atlassian.net/browse/BIT-1563 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro > Reporter: Daniel Thayer > Fix For: 2.5 > > > The BrokerComm and BrokerStore namespaces should be combined to > just "Broker". -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From jira at bro-tracker.atlassian.net Wed Mar 30 18:55:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Wed, 30 Mar 2016 20:55:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1563) BrokerComm and BrokerStore namespaces should be combined In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1563: ------------------------------- Status: Merge Request (was: Open) > BrokerComm and BrokerStore namespaces should be combined > -------------------------------------------------------- > > Key: BIT-1563 > URL: https://bro-tracker.atlassian.net/browse/BIT-1563 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro > Reporter: Daniel Thayer > Fix For: 2.5 > > > The BrokerComm and BrokerStore namespaces should be combined to > just "Broker". -- This message was sent by Atlassian JIRA (v7.2.0-OD-04-029#72002) From noreply at bro.org Thu Mar 31 00:00:37 2016 From: noreply at bro.org (Merge Tracker) Date: Thu, 31 Mar 2016 00:00:37 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201603310700.u2V70b50026831@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------------- BIT-1563 [1] Bro Daniel Thayer - 2016-03-30 2.5 Normal BrokerComm and BrokerStore namespaces should be combined BIT-1561 [2] BroControl Aaron Eppert - 2016-03-25 - Normal Pull Request to fix resource leaking in BroControl Python API BIT-1557 [3] Broccoli Daniel Thayer - 2016-03-21 2.5 Low broccoli code examples don't compile BIT-1547 [4] BroControl Justin Azoff Justin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1528 [5] Bro Justin Azoff - 2016-03-24 2.5 Normal SNMP and SIP scans show up in known services. BIT-1507 [6] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly BIT-1498 [7] BroControl scampbell - 2016-03-11 2.5 Trivial add '-q' to ssh execution in ssh_runner.py Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- --------------- ---------- ------------------------------------------------------------------------- #64 [8] bro aeppert [9] 2016-03-30 Add disable_all_analyzers for connection shunting options [10] #63 [11] bro WilliamTom [12] 2016-03-26 Wrong regex literal in scripting doc [13] #61 [14] bro aeppert [15] 2016-03-22 Add uid to smb_ntlm_authenticate [16] #60 [17] bro aeppert [18] 2016-03-22 Add uid to AuthInfo [19] #52 [20] bro J-Gras [21] 2016-01-18 Fixed matching mail address intel [22] #22 [23] bro-plugins nickwallen [24] 2016-03-23 BIT-1559 Bro-Plugins Send each log stream to different kafka topic [25] #18 [26] bro-plugins jshlbrd [27] 2016-03-03 SSDP analyzer [28] #4 [29] broctl aeppert [30] 2016-03-25 Fix leaking resources from never closing out the Broccoli connection [31] [1] BIT-1563 https://bro-tracker.atlassian.net/browse/BIT-1563 [2] BIT-1561 https://bro-tracker.atlassian.net/browse/BIT-1561 [3] BIT-1557 https://bro-tracker.atlassian.net/browse/BIT-1557 [4] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [5] BIT-1528 https://bro-tracker.atlassian.net/browse/BIT-1528 [6] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [7] BIT-1498 https://bro-tracker.atlassian.net/browse/BIT-1498 [8] Pull Request #64 https://github.com/bro/bro/pull/64 [9] aeppert https://github.com/aeppert [10] Merge Pull Request #64 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git master [11] Pull Request #63 https://github.com/bro/bro/pull/63 [12] WilliamTom https://github.com/WilliamTom [13] Merge Pull Request #63 with git pull --no-ff --no-commit https://github.com/WilliamTom/bro.git master [14] Pull Request #61 https://github.com/bro/bro/pull/61 [15] aeppert https://github.com/aeppert [16] Merge Pull Request #61 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-6 [17] Pull Request #60 https://github.com/bro/bro/pull/60 [18] aeppert https://github.com/aeppert [19] Merge Pull Request #60 with git pull --no-ff --no-commit https://github.com/aeppert/bro.git patch-5 [20] Pull Request #52 https://github.com/bro/bro/pull/52 [21] J-Gras https://github.com/J-Gras [22] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [23] Pull Request #22 https://github.com/bro/bro-plugins/pull/22 [24] nickwallen https://github.com/nickwallen [25] Merge Pull Request #22 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git support-many-kafka-topics [26] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [27] jshlbrd https://github.com/jshlbrd [28] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp [29] Pull Request #4 https://github.com/bro/broctl/pull/4 [30] aeppert https://github.com/aeppert [31] Merge Pull Request #4 with git pull --no-ff --no-commit https://github.com/aeppert/broctl.git patch-1 From jira at bro-tracker.atlassian.net Thu Mar 31 05:23:01 2016 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Thu, 31 Mar 2016 07:23:01 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1558) Bro's ascii formatter writing out scientific notation In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1558: ------------------------------- Fix Version/s: 2.5 > Bro's ascii formatter writing out scientific notation > ----------------------------------------------------- > > Key: BIT-1558 > URL: https://bro-tracker.atlassian.net/browse/BIT-1558 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Seth Hall > Fix For: 2.5 > > > From the mailing list: > ``` > Hello, > in the x509.log normally the values regarding certificate.not_valid_before & certificate.not_valid_after look like: > 1444082400.000000 1475791199.000000 > I found some value like this: > -3600.000000 2.153226e+09 > Is it possible to modify something in order to have 2153226000 instead 2.153226e+09 ? > ``` > Bro's formatter's shouldn't use scientific notation because it complicates parsing of the data. -- This message was sent by Atlassian JIRA (v7.2.0-OD-05-022#72002) From jsiwek at illinois.edu Thu Mar 31 05:29:17 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Thu, 31 Mar 2016 12:29:17 +0000 Subject: [Bro-Dev] Broker & CAF includes In-Reply-To: <20160325161653.GR43666@samurai.ICIR.org> References: <20160319032057.GJ8458@shogun> <29FF882F-3D9A-4E9D-9A29-A59169DDD8BD@illinois.edu> <20160321173635.GK43666@samurai.ICIR.org> <20160325161653.GR43666@samurai.ICIR.org> Message-ID: > On Mar 25, 2016, at 11:16 AM, Matthias Vallentin wrote: > > At this point, I'm inclined to move towards a more light-weight model > that is less robust against ABI changes. I believe we still need more > experience with the API. Once the API matures, hiding central > implementation aspects to increase ABI stability becomes the next > priority to improve medium- to long-term release compatibility. > > Does that sound reasonable? Yeah, that makes sense. - Jon From jira at bro-tracker.atlassian.net Thu Mar 31 07:43:00 2016 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Thu, 31 Mar 2016 09:43:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1547) broctl sets the same state variables over and over In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25303#comment-25303 ] Justin Azoff commented on BIT-1547: ----------------------------------- Merged.. Can you see about merging master back into topic/mfischer/broctl-broker? This latest change conflicts. I was able to just apply the config.py diff but the install.py change has issues. > broctl sets the same state variables over and over > -------------------------------------------------- > > Key: BIT-1547 > URL: https://bro-tracker.atlassian.net/browse/BIT-1547 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Reporter: Justin Azoff > Assignee: Justin Azoff > Fix For: 2.5 > > > I happened to notice broctl check on one of our test boxes was slow. traced it to sqlite commits() being very slow. Then noticed that broctl seems to call set_state() with the same key, val over and over again... once for each worker.. so a few thousand sets just to run broctl check. > Changing set_state to > {code} > # Set a dynamic state variable. > def set_state(self, key, val): > key = key.lower() > if self.state.get(key) == val: > return > self.state[key] = val > self.state_store.set(key, val) > {code} > Seemed to mostly fix it, aside from this: > {code} > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > {code} > any idea why that is flipping around like that? > We should possibly add a way for broctl to update state vars without calling commit where it knows it will be setting a large number of state vars in a loop. -- This message was sent by Atlassian JIRA (v7.2.0-OD-05-022#72002) From jira at bro-tracker.atlassian.net Thu Mar 31 07:44:00 2016 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Thu, 31 Mar 2016 09:44:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1547) broctl sets the same state variables over and over In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Azoff updated BIT-1547: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > broctl sets the same state variables over and over > -------------------------------------------------- > > Key: BIT-1547 > URL: https://bro-tracker.atlassian.net/browse/BIT-1547 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl > Affects Versions: git/master > Reporter: Justin Azoff > Assignee: Justin Azoff > Fix For: 2.5 > > > I happened to notice broctl check on one of our test boxes was slow. traced it to sqlite commits() being very slow. Then noticed that broctl seems to call set_state() with the same key, val over and over again... once for each worker.. so a few thousand sets just to run broctl check. > Changing set_state to > {code} > # Set a dynamic state variable. > def set_state(self, key, val): > key = key.lower() > if self.state.get(key) == val: > return > self.state[key] = val > self.state_store.set(key, val) > {code} > Seemed to mostly fix it, aside from this: > {code} > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > Set manager-port to 47760 > Set manager-port to 47761 > {code} > any idea why that is flipping around like that? > We should possibly add a way for broctl to update state vars without calling commit where it knows it will be setting a large number of state vars in a loop. -- This message was sent by Atlassian JIRA (v7.2.0-OD-05-022#72002) From jira at bro-tracker.atlassian.net Thu Mar 31 09:59:01 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 31 Mar 2016 11:59:01 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1498) add '-q' to ssh execution in ssh_runner.py In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer reassigned BIT-1498: ---------------------------------- Assignee: Daniel Thayer > add '-q' to ssh execution in ssh_runner.py > ------------------------------------------ > > Key: BIT-1498 > URL: https://bro-tracker.atlassian.net/browse/BIT-1498 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Affects Versions: 2.4 > Reporter: scampbell > Assignee: Daniel Thayer > Priority: Trivial > Labels: broctl > Fix For: 2.5 > > > When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. > The patch is trivial: > --- a/BroControl/ssh_runner.py > +++ b/BroControl/ssh_runner.py > @@ -108,6 +108,7 @@ class SSHMaster: > self.base_cmd = [ > "ssh", > "-o", "BatchMode=yes", > + "-q", > host, > ] > self.need_connect = True -- This message was sent by Atlassian JIRA (v7.2.0-OD-05-022#72002) From jira at bro-tracker.atlassian.net Thu Mar 31 11:31:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 31 Mar 2016 13:31:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1549) broctl top command doesn't work on OS X 10.10 or newer In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25304#comment-25304 ] Daniel Thayer commented on BIT-1549: ------------------------------------ As a quick fix, I will just change "vprvt" to "mem". This means that both memory columns in the top command output will contain the same data. > broctl top command doesn't work on OS X 10.10 or newer > ------------------------------------------------------ > > Key: BIT-1549 > URL: https://bro-tracker.atlassian.net/browse/BIT-1549 > Project: Bro Issue Tracker > Issue Type: Task > Components: BroControl > Reporter: Daniel Thayer > Fix For: 2.5 > > > On OS X Mavericks, the broctl top command was working, but on Yosemite > (and El Capitan), it no longer works. The reason is that the > "-stats vprvt" option of the top command always prints "N/A". -- This message was sent by Atlassian JIRA (v7.2.0-OD-05-022#72002) From jira at bro-tracker.atlassian.net Thu Mar 31 11:59:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 31 Mar 2016 13:59:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1498) Suppress ssh login banner from broctl output In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1498: ------------------------------- Summary: Suppress ssh login banner from broctl output (was: add '-q' to ssh execution in ssh_runner.py) > Suppress ssh login banner from broctl output > -------------------------------------------- > > Key: BIT-1498 > URL: https://bro-tracker.atlassian.net/browse/BIT-1498 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Affects Versions: 2.4 > Reporter: scampbell > Assignee: Daniel Thayer > Priority: Trivial > Labels: broctl > Fix For: 2.5 > > > When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. > The patch is trivial: > --- a/BroControl/ssh_runner.py > +++ b/BroControl/ssh_runner.py > @@ -108,6 +108,7 @@ class SSHMaster: > self.base_cmd = [ > "ssh", > "-o", "BatchMode=yes", > + "-q", > host, > ] > self.need_connect = True -- This message was sent by Atlassian JIRA (v7.2.0-OD-05-022#72002) From jira at bro-tracker.atlassian.net Thu Mar 31 12:00:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 31 Mar 2016 14:00:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1498) Suppress ssh login banner from broctl output In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1498: ------------------------------- Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Suppress ssh login banner from broctl output > -------------------------------------------- > > Key: BIT-1498 > URL: https://bro-tracker.atlassian.net/browse/BIT-1498 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Affects Versions: 2.4 > Reporter: scampbell > Assignee: Daniel Thayer > Priority: Trivial > Labels: broctl > Fix For: 2.5 > > > When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. > The patch is trivial: > --- a/BroControl/ssh_runner.py > +++ b/BroControl/ssh_runner.py > @@ -108,6 +108,7 @@ class SSHMaster: > self.base_cmd = [ > "ssh", > "-o", "BatchMode=yes", > + "-q", > host, > ] > self.need_connect = True -- This message was sent by Atlassian JIRA (v7.2.0-OD-05-022#72002) From jira at bro-tracker.atlassian.net Thu Mar 31 12:42:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 31 Mar 2016 14:42:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1549) broctl top command doesn't work on OS X 10.10 or newer In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1549: ------------------------------- Status: Merge Request (was: Open) > broctl top command doesn't work on OS X 10.10 or newer > ------------------------------------------------------ > > Key: BIT-1549 > URL: https://bro-tracker.atlassian.net/browse/BIT-1549 > Project: Bro Issue Tracker > Issue Type: Task > Components: BroControl > Reporter: Daniel Thayer > Fix For: 2.5 > > > On OS X Mavericks, the broctl top command was working, but on Yosemite > (and El Capitan), it no longer works. The reason is that the > "-stats vprvt" option of the top command always prints "N/A". -- This message was sent by Atlassian JIRA (v7.2.0-OD-05-022#72002) From jira at bro-tracker.atlassian.net Thu Mar 31 12:42:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 31 Mar 2016 14:42:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1549) broctl top command doesn't work on OS X 10.10 or newer In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer reassigned BIT-1549: ---------------------------------- Assignee: Justin Azoff > broctl top command doesn't work on OS X 10.10 or newer > ------------------------------------------------------ > > Key: BIT-1549 > URL: https://bro-tracker.atlassian.net/browse/BIT-1549 > Project: Bro Issue Tracker > Issue Type: Task > Components: BroControl > Reporter: Daniel Thayer > Assignee: Justin Azoff > Fix For: 2.5 > > > On OS X Mavericks, the broctl top command was working, but on Yosemite > (and El Capitan), it no longer works. The reason is that the > "-stats vprvt" option of the top command always prints "N/A". -- This message was sent by Atlassian JIRA (v7.2.0-OD-05-022#72002) From jira at bro-tracker.atlassian.net Thu Mar 31 12:42:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 31 Mar 2016 14:42:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1549) broctl top command doesn't work on OS X 10.10 or newer In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25306#comment-25306 ] Daniel Thayer commented on BIT-1549: ------------------------------------ Branch "topic/dnthayer/ticket1549" in the broctl repo contains the fix for this issue. > broctl top command doesn't work on OS X 10.10 or newer > ------------------------------------------------------ > > Key: BIT-1549 > URL: https://bro-tracker.atlassian.net/browse/BIT-1549 > Project: Bro Issue Tracker > Issue Type: Task > Components: BroControl > Reporter: Daniel Thayer > Fix For: 2.5 > > > On OS X Mavericks, the broctl top command was working, but on Yosemite > (and El Capitan), it no longer works. The reason is that the > "-stats vprvt" option of the top command always prints "N/A". -- This message was sent by Atlassian JIRA (v7.2.0-OD-05-022#72002) From jira at bro-tracker.atlassian.net Thu Mar 31 12:54:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 31 Mar 2016 14:54:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1561) Pull Request to fix resource leaking in BroControl Python API In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1561: ------------------------------- Description: connDelete() is never called in broctl, which is generally not an issue if it is being used via the command line. However, given the Python interface for calling into broctl, if a service is written around broctl and any parallel events are sent (peerstatus, netstats, etc.) then a TCP connection will be maintained given a connDelete() is not called. After a fairly certain interval, resources become an issue and things fail. https://github.com/bro/broctl/pull/4 was:https://github.com/bro/broctl/pull/4 > Pull Request to fix resource leaking in BroControl Python API > ------------------------------------------------------------- > > Key: BIT-1561 > URL: https://bro-tracker.atlassian.net/browse/BIT-1561 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Reporter: Aaron Eppert > Labels: broctl > > connDelete() is never called in broctl, which is generally not an issue if it is being used via the command line. However, given the Python interface for calling into broctl, if a service is written around broctl and any parallel events are sent (peerstatus, netstats, etc.) then a TCP connection will be maintained given a connDelete() is not called. After a fairly certain interval, resources become an issue and things fail. > https://github.com/bro/broctl/pull/4 -- This message was sent by Atlassian JIRA (v7.2.0-OD-05-022#72002) From jira at bro-tracker.atlassian.net Thu Mar 31 13:24:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 31 Mar 2016 15:24:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1561) Pull Request to fix resource leaking in BroControl Python API In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1561: ------------------------------- Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Pull Request to fix resource leaking in BroControl Python API > ------------------------------------------------------------- > > Key: BIT-1561 > URL: https://bro-tracker.atlassian.net/browse/BIT-1561 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Reporter: Aaron Eppert > Labels: broctl > > connDelete() is never called in broctl, which is generally not an issue if it is being used via the command line. However, given the Python interface for calling into broctl, if a service is written around broctl and any parallel events are sent (peerstatus, netstats, etc.) then a TCP connection will be maintained given a connDelete() is not called. After a fairly certain interval, resources become an issue and things fail. > https://github.com/bro/broctl/pull/4 -- This message was sent by Atlassian JIRA (v7.2.0-OD-05-022#72002) From jira at bro-tracker.atlassian.net Thu Mar 31 13:24:00 2016 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 31 Mar 2016 15:24:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1561) Pull Request to fix resource leaking in BroControl Python API In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25307#comment-25307 ] Daniel Thayer commented on BIT-1561: ------------------------------------ Each time a broctl command is run that uses broccoli, a new TCP connection is established. The connection is not closed until one either exits from broctl, or stops the Bro nodes. Ideally, it would reuse the TCP connection, but at least your fix is better than the current situation. > Pull Request to fix resource leaking in BroControl Python API > ------------------------------------------------------------- > > Key: BIT-1561 > URL: https://bro-tracker.atlassian.net/browse/BIT-1561 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Reporter: Aaron Eppert > Labels: broctl > > connDelete() is never called in broctl, which is generally not an issue if it is being used via the command line. However, given the Python interface for calling into broctl, if a service is written around broctl and any parallel events are sent (peerstatus, netstats, etc.) then a TCP connection will be maintained given a connDelete() is not called. After a fairly certain interval, resources become an issue and things fail. > https://github.com/bro/broctl/pull/4 -- This message was sent by Atlassian JIRA (v7.2.0-OD-05-022#72002) From vallentin at icir.org Thu Mar 31 21:00:41 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Thu, 31 Mar 2016 21:00:41 -0700 Subject: [Bro-Dev] Broker: use of broker::peering Message-ID: <20160401040041.GH26101@shogun> In Broker, what is the use case for having an explicit peering between two endpoints? Would it maybe suffice to provide endpoint introspection, i.e., the ability to iterate over an endpoint's peers? At least Bro doesn't use broker::endpoint for anything but that, but I was wondering if I am missing a use case that involves explicit peerings. If there's no clear use case, I was thinking to make broker::peering an implementation detail and remove it from the public API. Matthias