From Izik.Birka at hot.net.il Thu Jun 1 03:10:04 2017 From: Izik.Birka at hot.net.il (Izik Birka) Date: Thu, 1 Jun 2017 10:10:04 +0000 Subject: [Bro] BRO - Ransomware script (fox-it) Message-ID: <592228F4D0C8504187F2F76658040CB6E0028DB4@HOT-MAILBOX-02.HOT.NET.IL> Hi How can I check that the script is working ? Any idea where I can find PCAP with malicious SMB activity ? Thanks This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain materials protected by copyright or information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or agreement. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication by error, notify the sender immediately and delete this message immediately. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170601/2846e305/attachment.html From philosnef at gmail.com Thu Jun 1 03:49:52 2017 From: philosnef at gmail.com (erik clark) Date: Thu, 1 Jun 2017 06:49:52 -0400 Subject: [Bro] bro rpm compiled with af-packet plugin? Message-ID: Is the bro rpm compiled with the af-packet plugin? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170601/22fd4ca1/attachment.html From philosnef at gmail.com Thu Jun 1 04:46:44 2017 From: philosnef at gmail.com (erik clark) Date: Thu, 1 Jun 2017 07:46:44 -0400 Subject: [Bro] af_packet plugin does not compile Message-ID: Af_packet plugin does not compile with same dependencies as Bro. Bro compiles cleanly (2.5). Af_packet plugin states: CMake Error at ...RequireCXX11.cmake:31 C++11 headers cannot be used for compilation This is from the bro 2.5 release from last year and not a github update.Af_packet plugin documentation has nothing to say about why this would occur. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170601/46cbbbca/attachment.html From tomas.bortoli at sit.fraunhofer.de Thu Jun 1 04:57:25 2017 From: tomas.bortoli at sit.fraunhofer.de (Bortoli, Tomas) Date: Thu, 1 Jun 2017 11:57:25 +0000 Subject: [Bro] binpac to bro script types In-Reply-To: References: <201705261008.v4QA8Rep016155@vladg.net> <201705291113.v4TBDVoa027976@vladg.net>, Message-ID: Thank you very much Vlad! I finally also solve it in a very similar way in the end (conversion + offset) Tomas ________________________________________ From: Vlad Grigorescu [vladg at illinois.edu] Sent: Wednesday, May 31, 2017 9:13 PM To: Bortoli, Tomas; bro at bro.org Subject: RE: [Bro] binpac to bro script types Well, that's protocol specific, but I did some digging: > >>> TIME_FIXUP_CONSTANT > 11644473600 > >>> hex(filetime) > '0x01d238cc0f66a007' > >>> filetime/10000000. > 13122978809.960194 > >>> _-TIME_FIXUP_CONSTANT > 1478505209.9601936 > >>> datetime.datetime.fromtimestamp(1478505209.9601936).strftime('%Y-%m-%d %H:%M:%S') > '2016-11-07 01:53:29' This is already implemented in smb-time.pac: https://github.com/bro/bro/blob/master/src/analyzer/protocol/smb/smb-time.pac#L13 You could try just adding this to your PAC file and then you'll be able to use that function: > %include ../smb/smb-time.pac Check out krb-asn1.pac for an example of including another PAC file: https://github.com/bro/bro/blob/master/src/analyzer/protocol/krb/krb-asn1.pac --Vlad From philosnef at gmail.com Thu Jun 1 05:00:36 2017 From: philosnef at gmail.com (erik clark) Date: Thu, 1 Jun 2017 08:00:36 -0400 Subject: [Bro] af_packet plugin does not compile In-Reply-To: References: Message-ID: Scrap this. I deleted the CMake directories and this compiled. On Thu, Jun 1, 2017 at 7:46 AM, erik clark wrote: > Af_packet plugin does not compile with same dependencies as Bro. Bro > compiles cleanly (2.5). Af_packet plugin states: > > CMake Error at ...RequireCXX11.cmake:31 > C++11 headers cannot be used for compilation > > This is from the bro 2.5 release from last year and not a github > update.Af_packet plugin documentation has nothing to say about why this > would occur. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170601/f5f3a853/attachment-0001.html From philosnef at gmail.com Thu Jun 1 06:53:44 2017 From: philosnef at gmail.com (erik clark) Date: Thu, 1 Jun 2017 09:53:44 -0400 Subject: [Bro] problem with bro in container Message-ID: When I run my container like so: docker run -v /data:/data --name bro -it --net=host bro /opt/bro/bin/broctl deploy it seems to fail immediately. There is no useful information in the logs in spool or current. When I run it as docker run -v /data:/data --name bro -it --net=host bro /bin/bash and start broctl deploy manually, it works fine and my logs are mapped back like they should be. Why is bro bailing like this? I've confirmed that /opt/bro/spool is mapped to /data/bro/spool in broctl.cfg, but even when I leave it mapped back to /opt/bro/spool, it still fails to stay up when deployed with /opt/bro/bin/broctl deploy Please advise -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170601/38eb7190/attachment.html From jdopheid at illinois.edu Thu Jun 1 08:06:11 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Thu, 1 Jun 2017 15:06:11 +0000 Subject: [Bro] Bro Package Manager Questionnaire Message-ID: <7CE20DCD-27F9-4FCA-901D-8889A702497D@illinois.edu> Friendly reminder to fill out this questionnaire. Thanks to those who have responded so far! ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign On 5/30/17, 1:09 PM, "bro-bounces at bro.org on behalf of Dopheide, Jeannette M" wrote: The Bro team would like to encourage the development of Bro scripts and plugins by creating a website front-end for the Bro Package Manager, with additional functionality to be determined. We are seeking input from the Bro user community as to what features would be desirable. Please let us know what features you would like to see by filling out our questionnaire: https://goo.gl/forms/VyVH1aRIBB2qdZF53 ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From wren3 at illinois.edu Thu Jun 1 15:16:59 2017 From: wren3 at illinois.edu (Ren, Wenyu) Date: Thu, 1 Jun 2017 22:16:59 +0000 Subject: [Bro] "conn" field not present in connection Message-ID: Hi Everyone, I have a problem using the "conn" field in the connection record. The reference states that "conn" should exist if "base/protocols/conn/main.bro" is loaded. I have it loaded and the "conn.log" is generated. However, the "conn" field is not there. I got the connection record from the new_packet event. Anyone have any idea? Thanks a lot. Best, Wenyu Wenyu Ren Ph.D. Candidate Department of Computer Science University of Illinois at Urbana-Champaign From dnthayer at illinois.edu Thu Jun 1 17:40:36 2017 From: dnthayer at illinois.edu (Daniel Thayer) Date: Thu, 1 Jun 2017 19:40:36 -0500 Subject: [Bro] "conn" field not present in connection In-Reply-To: References: Message-ID: On 6/1/17 5:16 PM, Ren, Wenyu wrote: > Hi Everyone, > > I have a problem using the "conn" field in the connection record. The reference states that "conn" should exist if "base/protocols/conn/main.bro" is loaded. I have it loaded and the "conn.log" is generated. However, the "conn" field is not there. I got the connection record from the new_packet event. Anyone have any idea? Thanks a lot. > > Best, > Wenyu The documentation states that the "conn" field exists if base/protocols/conn/main.bro is loaded. Since the "conn" field has the "&optional" attribute, it is not required to have a value. At the time that the new_packet event is handled, not all of the necessary information has been gathered, so the "conn" field doesn't have a value yet. If you try checking in a subsequent event, such as connection_state_remove, then you should be able to see a value assigned to the "conn" field (and if you look at the base/protocols/conn/main.bro script, you can see where it assigns a value to the conn field). From byublakemoss12 at gmail.com Fri Jun 2 08:19:12 2017 From: byublakemoss12 at gmail.com (Blake Moss) Date: Fri, 2 Jun 2017 09:19:12 -0600 Subject: [Bro] Custom Script for log field addition. Message-ID: <593181f0.c86e630a.2138a.052f@mx.google.com> Hi all, I have a question regarding deploying custom scripts across a distributed bro cluster (manager, multiple worker nodes, etc.). I have a particular custom script which add an extra field to the ?conn.log?. When I load this script in my local.bro (via @load myscript) on my manager node and use broctl to deploy this across the cluster I do not get an error. However the extra field in my ?conn.log? does not appear in the /usr/local/bro/logs/current/conn.log. I did some looking around and found that the field was however being added to the /usr/local/bro/spool/bro/conn.log. I have tried loading this script in the local-worker.bro, and local-manager.bro but have had no success. Here is my script: module MyScript.bro ----------------------------- addWorker; export { redef record Conn::Info += { ??????? worker_id: string &default="unknown" &log; }; event connection_state_remove(c: connection) { ???????? c$conn$worker_id = peer_description; } } Thanks for your help! -Blake Sent from Mail for Windows 10 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170602/3a6bd3a6/attachment.html From jazoff at illinois.edu Fri Jun 2 08:34:41 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 2 Jun 2017 15:34:41 +0000 Subject: [Bro] Custom Script for log field addition. In-Reply-To: <593181f0.c86e630a.2138a.052f@mx.google.com> References: <593181f0.c86e630a.2138a.052f@mx.google.com> Message-ID: > On Jun 2, 2017, at 11:19 AM, Blake Moss wrote: > > Hi all, > I have a question regarding deploying custom scripts across a distributed bro cluster (manager, multiple worker nodes, etc.). I have a particular custom script which add an extra field to the ?conn.log?. When I load this script in my local.bro (via @load myscript) on my manager node and use broctl to deploy this across the cluster I do not get an error. However the extra field in my ?conn.log? does not appear in the /usr/local/bro/logs/current/conn.log. I did some looking around and found that the field was however being added to the /usr/local/bro/spool/bro/conn.log. I have tried loading this script in the local-worker.bro, and local-manager.bro but have had no success. Here is my script: module > > MyScript.bro > ----------------------------- > addWorker; > export > { > redef record Conn::Info += { > worker_id: string &default="unknown" &log; > }; > > event connection_state_remove(c: connection) > { > c$conn$worker_id = peer_description; > } > } > Looks like my script :-) /usr/local/bro/logs/current/conn.log and /usr/local/bro/spool/bro/conn.log should be the same file /usr/local/bro/logs/current should be a symlink to /usr/local/bro/spool/bro However, on a cluster the log files should really be under spool/manager or spool/logger, unless you have something like this in node.cfg [bro] type=manager host=.. instead of [manager] type=manager host=.. In any case, you should never add things to local-worker.bro or local-manager.bro. -- - Justin Azoff From kgoldman at us.ibm.com Fri Jun 2 09:53:22 2017 From: kgoldman at us.ibm.com (Kenneth Goldman) Date: Fri, 2 Jun 2017 12:53:22 -0400 Subject: [Bro] Missing notice.log, have weird.log In-Reply-To: References: <914dea0f-8cbd-00ea-ed7e-b0b2e60f1650@illinois.edu> Message-ID: > From: Seth Hall > To: Kenneth Goldman > Cc: bro > Date: 05/31/2017 03:20 PM > Subject: Re: [Bro] Missing notice.log, have weird.log > > On Wed, May 31, 2017 at 2:31 PM, Kenneth Goldman wrote: > > The quick starter refers to a notice.log file. It's not being created. > > Logs in Bro are created when they are written to. It's like that none > of the scripts you have loaded are generating notices. Should it generate notices "out of the box"? I have not done any customization. notice/main.bro says this, which I read to mean ignore nothing. const ignored_types: set[Notice::Type] = {} &redef; If I have to customize something to get notices, is there a tutorial on how to do that? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170602/f9ee0677/attachment-0001.html From chris at ceegeebee.com Sun Jun 4 03:08:55 2017 From: chris at ceegeebee.com (Chris Bennett) Date: Sun, 4 Jun 2017 19:38:55 +0930 Subject: [Bro] JSON logging of datasource or 'path' value Message-ID: Hi there, I'm experimenting with the JSON output and wanting to manually feed logs to logstash via 'cat | nc'. Is it possible to have the JSON output write the datatype or 'path' value similar to what is wrtiten as a metadata field at the top of ascii logs, but include it in each record for easy parsing in Logstash? Thanks, Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170604/965fcc2d/attachment.html From bill.de.ping at gmail.com Sun Jun 4 08:06:53 2017 From: bill.de.ping at gmail.com (william de ping) Date: Sun, 4 Jun 2017 18:06:53 +0300 Subject: [Bro] - Skip Weird or ProtocolViolation analyzer Message-ID: Hi all, I am trying to save bro unnecessary events, weird is has quit a few hits that are not relevant to me. I see that under HTTP.cc or DNS.cc I have some redirection to WEIRD or ProtocolViolation analyzers. How can I delete the connection at this stage instead of sending it to another costly analyzer ? can I just comment it out ? Thank you, B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170604/99aa6ad9/attachment.html From bro at pingtrip.com Sun Jun 4 08:47:54 2017 From: bro at pingtrip.com (Dave Crawford) Date: Sun, 4 Jun 2017 11:47:54 -0400 Subject: [Bro] &expire_function Index data type on multi-index table Message-ID: With the following table: test_expire_table: table[string, addr] of set[string] &write_expire=5sec &expire_func=test_expire_func What is the data type for the index variable when the expire function is called? function test_expire_func(t: table[string, addr] of set[string], idx: ???): interval If I set the idx type to any or string then print idx: Test, 1.2.3.4 But trying to use it as an index produces an error: print t[idx]; not an index type (t[idx]) Thanks, -Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170604/840849d6/attachment.html From jan.grashoefer at gmail.com Sun Jun 4 10:47:56 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Sun, 4 Jun 2017 19:47:56 +0200 Subject: [Bro] &expire_function Index data type on multi-index table In-Reply-To: References: Message-ID: Hi Dave, > With the following table: > > test_expire_table: table[string, addr] of set[string] &write_expire=5sec &expire_func=test_expire_func > > What is the data type for the index variable when the expire function is called? > > function test_expire_func(t: table[string, addr] of set[string], idx: ???): interval > > If I set the idx type to any or string then print idx: > > Test, 1.2.3.4 > > But trying to use it as an index produces an error: you can use a tuple-like syntax to get the index: [i, j] = idx Then you can use each part of the index separately or use both to access the table (c.f. https://github.com/bro/bro/blob/master/scripts/base/frameworks/intel/main.bro#L254). Jan From hosom at battelle.org Mon Jun 5 10:36:52 2017 From: hosom at battelle.org (Hosom, Stephen M) Date: Mon, 5 Jun 2017 17:36:52 +0000 Subject: [Bro] - Skip Weird or ProtocolViolation analyzer In-Reply-To: References: Message-ID: I don?t think weird can cleanly be disabled. Is there a particular reason that you?re trying this hard to optimize? Even if you could turn off weird, it would be a bad idea to do so. That?s where a lot of the good troubleshooting data comes from for Bro. It can be a great way to find problems with your Bro deployment and your environment. From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of william de ping Sent: Sunday, June 4, 2017 11:07 AM To: bro at bro.org Subject: [Bro] - Skip Weird or ProtocolViolation analyzer Message received from outside the Battelle network. Carefully examine it before you open any links or attachments. Hi all, I am trying to save bro unnecessary events, weird is has quit a few hits that are not relevant to me. I see that under HTTP.cc or DNS.cc I have some redirection to WEIRD or ProtocolViolation analyzers. How can I delete the connection at this stage instead of sending it to another costly analyzer ? can I just comment it out ? Thank you, B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170605/a7e77819/attachment.html From cbvpandiraj at gmail.com Mon Jun 5 11:32:03 2017 From: cbvpandiraj at gmail.com (Rajkumar) Date: Mon, 5 Jun 2017 12:32:03 -0600 Subject: [Bro] Source code for reassembly Message-ID: Hi, Can anyone help me in locating the source code for tcp payload reassembly process in bro? Raj -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170605/d4888f99/attachment.bin From zhangxu1115 at gmail.com Mon Jun 5 13:30:38 2017 From: zhangxu1115 at gmail.com (Xu Zhang) Date: Mon, 5 Jun 2017 13:30:38 -0700 Subject: [Bro] Text editor plugin for bro indent Message-ID: Hi, I'm new to Bro. I'm wondering if there is a plugin for bro indent? I'm currently using Vim with setting "set autoindent and smartindent", but it is not the same indent as in the other bro scripts. This git repo https://github.com/mephux/bro.vim has plugin for bro syntax highlighting, but has stopped updating the indent part from 4 years ago. -- Sincerely, Xu Zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170605/77eb9715/attachment.html From bill.de.ping at gmail.com Tue Jun 6 05:12:56 2017 From: bill.de.ping at gmail.com (william de ping) Date: Tue, 6 Jun 2017 15:12:56 +0300 Subject: [Bro] - Skip Weird or ProtocolViolation analyzer In-Reply-To: References: Message-ID: Hi, Yes I am well aware of my input traffic and I would like to save bro as much processing as I can. If I know that all my traffic is SMTP related, I have no need for other analyzers. I would even like bro to delete a packet that have some malformed data instead of forwarding it to another analyzer. The thing is that I would like to make Bro as bare as possible so it can work as fast as Suricata Thanks B On Mon, Jun 5, 2017 at 8:36 PM, Hosom, Stephen M wrote: > I don?t think weird can cleanly be disabled. Is there a particular reason > that you?re trying this hard to optimize? Even if you could turn off weird, > it would be a bad idea to do so. That?s where a lot of the good > troubleshooting data comes from for Bro. It can be a great way to find > problems with your Bro deployment and your environment. > > > > *From:* bro-bounces at bro.org [mailto:bro-bounces at bro.org] *On Behalf Of *william > de ping > *Sent:* Sunday, June 4, 2017 11:07 AM > *To:* bro at bro.org > *Subject:* [Bro] - Skip Weird or ProtocolViolation analyzer > > > > Message received from outside the Battelle network. Carefully examine it > before you open any links or attachments. > > Hi all, > > I am trying to save bro unnecessary events, weird is has quit a few hits > that are not relevant to me. > I see that under HTTP.cc or DNS.cc I have some redirection to WEIRD or > ProtocolViolation analyzers. > > How can I delete the connection at this stage instead of sending it to > another costly analyzer ? > > can I just comment it out ? > > Thank you, > > B > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170606/e18ea55c/attachment.html From bill.de.ping at gmail.com Tue Jun 6 06:09:49 2017 From: bill.de.ping at gmail.com (william de ping) Date: Tue, 6 Jun 2017 16:09:49 +0300 Subject: [Bro] Text editor plugin for bro indent In-Reply-To: References: Message-ID: you can use Sublime with the Bro language addition : https://github.com/bro/bro-sublime On Mon, Jun 5, 2017 at 11:30 PM, Xu Zhang wrote: > Hi, > I'm new to Bro. I'm wondering if there is a plugin for bro indent? I'm > currently using Vim with setting "set autoindent and smartindent", but it > is not the same indent as in the other bro scripts. > > This git repo > https://github.com/mephux/bro.vim > has plugin for bro syntax highlighting, but has stopped updating the > indent part from 4 years ago. > > -- > Sincerely, > Xu Zhang > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170606/e3f72831/attachment.html From tbi28430 at gmail.com Tue Jun 6 06:23:18 2017 From: tbi28430 at gmail.com (Tom B) Date: Tue, 6 Jun 2017 09:23:18 -0400 Subject: [Bro] saving packet content to disk Message-ID: Is there a way to save the packet content to disk, for example, when detecting the http_header event matching certain criteria (host name), how can I get the packet payload (http_entity_data ?) and save to disk in binary form ? Do I need to create custom writer ? My apology for any confusion since I am new to BRO. Thanks, Tom -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170606/953aff10/attachment.html From johanna at icir.org Tue Jun 6 09:21:22 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 6 Jun 2017 09:21:22 -0700 Subject: [Bro] bro rpm compiled with af-packet plugin? In-Reply-To: References: Message-ID: <20170606162122.ufpohen5ygosyfqb@wifi109.sys.ICSI.Berkeley.EDU> > Is the bro rpm compiled with the af-packet plugin? Thanks! No, it is not. Johanna From jdopheid at illinois.edu Tue Jun 6 09:23:15 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Tue, 6 Jun 2017 16:23:15 +0000 Subject: [Bro] Reminder BroCon CFP expires Friday! Message-ID: <7EFD7D614A2BB84ABEA19B2CEDD246583B49CAA8@CITESMBX5.ad.uillinois.edu> Are you planning on presenting at BroCon this year? The Call for Presentations expires on Friday! ************************** BroCon ?17 is accepting presentation proposals. We are looking for talks to represent the many applications of Bro. Suitable topics include, but are not limited to: * as a tool for solving problems; * interesting user stories, solutions, or research projects; * a postmortem analysis of a security incident, emphasizing Bro?s contribution; * the value Bro brings to your professional work; * and, using Bro for more than intrusion detection. * Please, no product presentations Criteria for evaluating proposals include whether the topic is applicable to multiple types of organizations, gives people ideas to take home and use, can be understood by a broad audience, or is novel to many in the audience. Scrolling through our YouTube Channel may provide some insight into the types of presentations we wish to feature. Plan on limiting your talk to 30-35 minutes with an additional 10 minutes for questions/comments. Send abstracts (max 500 words) to: info at bro.org Subject: BroCon 2017 Call for Presentations Submission due date: Friday, June 9th Target date for announcing speakers: Friday June 30th CFPs are selected by the Bro Leadership Team: * Johanna Amann, International Computer Science Institute * Seth Hall, International Computer Science Institute * Keith Lehigh, Indiana University * Vern Paxson, University of California at Berkeley * Michal Purzynski, Mozilla Foundation * Aashish Sharma, Lawrence Berkeley Lab * Adam Slagell, National Center for Supercomputing Applications * Robin Sommer, International Computer Science Institute * Martin van Hensbergen, Fox-IT ------ Jeannette M. Dopheide Bro Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170606/fc539702/attachment-0001.html From johanna at icir.org Tue Jun 6 09:27:18 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 6 Jun 2017 09:27:18 -0700 Subject: [Bro] JSON logging of datasource or 'path' value In-Reply-To: References: Message-ID: <20170606162718.jfia2p4hqbuiuxfa@wifi109.sys.ICSI.Berkeley.EDU> Hi Chris, > I'm experimenting with the JSON output and wanting to manually feed logs to > logstash via 'cat | nc'. Is it possible to have the JSON output write the > datatype or 'path' value similar to what is wrtiten as a metadata field at > the top of ascii logs, but include it in each record for easy parsing in > Logstash? yes, this is possible using log extension functions, more specifically by redefining Log::default_ext_func (https://www.bro.org/sphinx/scripts/base/frameworks/logging/main.bro.html#id-Log::default_ext_func). Bro actually contains a testcase that has a script that basically does exactly what you want: https://github.com/bro/bro/blob/master/testing/btest/scripts/base/frameworks/logging/field-extension.bro That script adds three fields to each logfile (_write_ts, _stream, and _system_name). For your case, you only want _stream, but apart from that this approach should directly work for you. I hope this helps, Johanna From johanna at icir.org Tue Jun 6 09:31:34 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 6 Jun 2017 09:31:34 -0700 Subject: [Bro] Source code for reassembly In-Reply-To: References: Message-ID: <20170606163134.fowheo3dvrswkecb@wifi109.sys.ICSI.Berkeley.EDU> > Can anyone help me in locating the source code for tcp payload > reassembly process in bro? It mainly is in src/analyzer/protocol/tcp/, files TCP_Reassembler.h/cc, as well as src/Reassem.h/cc I hope that helps, Johanna From johanna at icir.org Tue Jun 6 09:36:34 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 6 Jun 2017 09:36:34 -0700 Subject: [Bro] - Skip Weird or ProtocolViolation analyzer In-Reply-To: References: Message-ID: <20170606163634.2ah5oqxapldvkjlv@wifi109.sys.ICSI.Berkeley.EDU> Hi, Weird and ProtocolViolation are no analyzers, and because of that they are not especially costly. Weird is generally called when one of the protocol analyzers notices something "weird" happening in the protocol; this is then logged directly to weird.log. While you can disable this function call, I don't really think you will see significant performance gains by this. ProtocolViolation is a bit different; this is called when a analyzer encounters data in a protocol that it cannot parse (i.e. it is a violation of how we think that the protocol should work). This is generally logged into dpd.log, and the analyzer stops processing the connection after that. You definitely should not just delete this function call, as it might mess with what happens during protocol detection. If you want a Bro installation that does not instantiate most protocol analyzers, you can just start Bro in bare mode (using -b), and only load the scripts that you are interested in. By default Bro will not parse any application layer protocols in bare mode (you should not even see conn.log generated). Johanna On Sun, Jun 04, 2017 at 06:06:53PM +0300, william de ping wrote: > Hi all, > > I am trying to save bro unnecessary events, weird is has quit a few hits > that are not relevant to me. > I see that under HTTP.cc or DNS.cc I have some redirection to WEIRD or > ProtocolViolation analyzers. > How can I delete the connection at this stage instead of sending it to > another costly analyzer ? > > can I just comment it out ? > > Thank you, > B > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From asharma at lbl.gov Tue Jun 6 11:12:28 2017 From: asharma at lbl.gov (Aashish Sharma) Date: Tue, 6 Jun 2017 11:12:28 -0700 Subject: [Bro] Job Posting: Cyber Security Engineer at Berkeley Lab Message-ID: <20170606181227.GD95838@mac-822.local> Colleagues: Lawrence Berkeley National Lab, a University of California managed lab, has an immediate opening for a cyber security engineer: http://go.lbl.gov/cyber-position A few reasons why it's awesome to work at Berkeley Lab. - Mission - http://go.lbl.gov/mission (pdf) - Smart colleagues - you will teach and learn - Location - famous California Bay Area weather, activities, and food - Work Environment - science driven environment, less politics than usual - Fun and challenging - we enjoy the challenge of hard problems - Bro IDS - if you like Bro, we are the birthplace - Benefits - excellent benefits and retirement - Prior experience in an .edu environment is highly relevant Feel free to contact me directly if you're interested. Aashish From wren3 at illinois.edu Tue Jun 6 13:44:01 2017 From: wren3 at illinois.edu (Ren, Wenyu) Date: Tue, 6 Jun 2017 20:44:01 +0000 Subject: [Bro] Is the "service" field of the connection record unreliable? Message-ID: Hi everyone, I have a question for the "service" field in the connection record. When I run the "testing/btest/Traces/modbus/modbus.trace" in the bro repo, it contains "MODBUS" for most of the connections except for a few. However, when I run the "testing/btest/Traces/modbus/modbusBig.pcap" trace, all of the connections have empty service fields although they are all using Modbus. The connection record I used is from the new_packet event. Does this mean the service field is quite unreliable and cannot be used to tell the service of the connection? If I need to directly use the destination port to identify the service type, there might be other problems. For example, sometimes the destination port contained in the "id" tuple in the connection record is actually the source port. This is probably due to the connection re-establishment from the receiver side. An example can be seen in the highlighted packet in the attached screenshot (which is from the "modbus.trace" in the repo). So my question is what's the best way to get the service of the connection from Bro. Any help and idea are appreciated. Thanks in advance. Best, Wenyu Wenyu Ren Ph.D. Candidate Department of Computer Science University of Illinois at Urbana-Champaign -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2017-06-06 14-52-51.png Type: image/png Size: 65453 bytes Desc: Screenshot from 2017-06-06 14-52-51.png Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170606/2dc68400/attachment-0001.bin From johanna at icir.org Wed Jun 7 16:29:23 2017 From: johanna at icir.org (Johanna Amann) Date: Wed, 7 Jun 2017 16:29:23 -0700 Subject: [Bro] Bro 2.5.1 Beta available Message-ID: <20170607232923.anudfvg57un2xusk@wifi109.sys.ICSI.Berkeley.EDU> The beta version for Bro 2.5.1 is now available for testing and can be downloaded at: https://bro.org/download/index.html Binary packages also are available at: https://bro.org/download/beta-packages.html This release contains a number of bug fixes. Fixes include: - Better file analysis memory management - Less cluster node communication - Correct expiration of intelligence items after reinsertion - A bug in the OCSP validation code This point-release also includes a number of new features, including new file handling BIFS, support for ERSPAN, and new BroControl options. For more information see the NEWS and CHANGES files: https://www.bro.org/documentation/beta/NEWS.bro.html https://www.bro.org/documentation/beta/CHANGES.bro.txt Feel free to use this mailing list or the bug tracker (tracker.bro.org) to provide feedback or report problems. Johanna From Izik.Birka at hot.net.il Thu Jun 8 01:51:24 2017 From: Izik.Birka at hot.net.il (Izik Birka) Date: Thu, 8 Jun 2017 08:51:24 +0000 Subject: [Bro] no sha1\md5 for some logs in files.log Message-ID: <592228F4D0C8504187F2F76658040CB6E004C31A@HOT-MAILBOX-02.HOT.NET.IL> Hi Why there some logs in files.log that not contains the sha1 or md5 value ? For example : Jun 8 11:32:39 127.0.0.1 bro_files: 1496910758.272740|FIMpTB242jcRsKCCYj|x.x.x.x|x.x.x.x|CAuOUv3lwBwigjH7mk|SMB|0|MD5,SHA1|-|test\test111\bro\go.pdf|0.021820|F|F|581600|1040352|458752|65536|F|-|-|-|-|- [Enjoy] ????? ????? ??? ???? ????? ???? ?????? ???? ????? ?????? ???? 077-7077790 | 053-6064571 P ???? ?? ?????? ???? ?????? ???? ?? [Enjoy] This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain materials protected by copyright or information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or agreement. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication by error, notify the sender immediately and delete this message immediately. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170608/b662bf9e/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 18831 bytes Desc: image001.jpg Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170608/b662bf9e/attachment-0002.jpg -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 43264 bytes Desc: image002.jpg Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170608/b662bf9e/attachment-0003.jpg From seth at corelight.com Thu Jun 8 06:31:38 2017 From: seth at corelight.com (Seth Hall) Date: Thu, 8 Jun 2017 09:31:38 -0400 Subject: [Bro] no sha1\md5 for some logs in files.log In-Reply-To: <592228F4D0C8504187F2F76658040CB6E004C31A@HOT-MAILBOX-02.HOT.NET.IL> References: <592228F4D0C8504187F2F76658040CB6E004C31A@HOT-MAILBOX-02.HOT.NET.IL> Message-ID: Hi Izik, Your file had a content gap (458752 bytes missing). Since the file was transferred over SMB, it's very possible that only part of the file was actually transferred due to offset reads or writes. It's one of the downsides of monitoring file system protocols since it's very common for software to only read or write a portion of a file after seeking. The reason that no hashes are provided in that case is that the hash wouldn't mean anything since it would just be a hash of some fairly arbitrary portion of the file. .Seth On Thu, Jun 8, 2017 at 4:51 AM, Izik Birka wrote: > Hi > > Why there some logs in files.log that not contains the sha1 or md5 value > ? > > > > For example : > > > > Jun 8 11:32:39 127.0.0.1 bro_files: 1496910758.272740| > FIMpTB242jcRsKCCYj|x.x.x.x|x.x.x.x|CAuOUv3lwBwigjH7mk|SMB| > 0|MD5,SHA1|-|test\test111\bro\go.pdf|0.021820|F|F|581600| > 1040352|458752|65536|F|-|-|-|-|- > > > > > > > > [image: Enjoy] > > ????? ????? > ??? ???? ????? ???? ?????? ???? > ????? ?????? ???? > 077-7077790 | 053-6064571 > > P ???? ?? ?????? ???? ?????? ???? ?? > > > [image: Enjoy] > > This message (including any attachments) is intended only for the use of > the individual or entity to which it is addressed and may contain materials > protected by copyright or information that is non-public, proprietary, > privileged, confidential, and exempt from disclosure under applicable law > or agreement. If you are not the intended recipient, you are hereby > notified that any use, dissemination, distribution, or copying of this > communication is strictly prohibited. If you have received this > communication by error, notify the sender immediately and delete this > message immediately. Thank you. > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170608/4401aee2/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 18831 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170608/4401aee2/attachment-0002.jpg -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 43264 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170608/4401aee2/attachment-0003.jpg From seth at corelight.com Thu Jun 8 06:37:58 2017 From: seth at corelight.com (Seth Hall) Date: Thu, 8 Jun 2017 09:37:58 -0400 Subject: [Bro] Is the "service" field of the connection record unreliable? In-Reply-To: References: Message-ID: For the service field to be filled out, there are two things that are required. First, the relevant analyzer needs to be attached to attached to the connection, but the modbus analyzer is only port based. There is no signature for attaching the modbus analyzer to connections (i.e., DPD/dynamic protocol detection) because I wasn't able to write one that would identify modbus reliably. Secondly, the analyzer itself has to confirm that it does in fact appear to be analyzing the protocol. The modbus analyzer doesn't identify the protocol as being modbus until both sides have sent modbus messages. I suspect that you aren't seeing modbus show up as a service because both sides aren't speaking modbus (scanners will have this problem since they just connect to random stuff and send modbus messages). We can certainly debate if that's the right way to do it or if only one side sending a valid modbus message should be enough to identify modbus. I can see the argument there for treating it that way. .Seth On Tue, Jun 6, 2017 at 4:44 PM, Ren, Wenyu wrote: > Hi everyone, > > I have a question for the "service" field in the connection record. When I run the "testing/btest/Traces/modbus/modbus.trace" in the bro repo, it contains "MODBUS" for most of the connections except for a few. However, when I run the "testing/btest/Traces/modbus/modbusBig.pcap" trace, all of the connections have empty service fields although they are all using Modbus. The connection record I used is from the new_packet event. Does this mean the service field is quite unreliable and cannot be used to tell the service of the connection? > > If I need to directly use the destination port to identify the service type, there might be other problems. For example, sometimes the destination port contained in the "id" tuple in the connection record is actually the source port. This is probably due to the connection re-establishment from the receiver side. An example can be seen in the highlighted packet in the attached screenshot (which is from the "modbus.trace" in the repo). > > So my question is what's the best way to get the service of the connection from Bro. Any help and idea are appreciated. Thanks in advance. > > Best, > Wenyu > > > > Wenyu Ren > Ph.D. Candidate > Department of Computer Science > University of Illinois at Urbana-Champaign > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From bill.de.ping at gmail.com Thu Jun 8 07:55:41 2017 From: bill.de.ping at gmail.com (william de ping) Date: Thu, 8 Jun 2017 17:55:41 +0300 Subject: [Bro] - Skip Weird or ProtocolViolation analyzer In-Reply-To: <20170606163634.2ah5oqxapldvkjlv@wifi109.sys.ICSI.Berkeley.EDU> References: <20170606163634.2ah5oqxapldvkjlv@wifi109.sys.ICSI.Berkeley.EDU> Message-ID: Hi, Yes I do see better results with bare mode. However, is it possible to run Broctl in bare mode ? Thanks, B On Tue, Jun 6, 2017 at 7:36 PM, Johanna Amann wrote: > Hi, > > Weird and ProtocolViolation are no analyzers, and because of that they are > not especially costly. Weird is generally called when one of the protocol > analyzers notices something "weird" happening in the protocol; this is > then logged directly to weird.log. While you can disable this function > call, I don't really think you will see significant performance gains by > this. > > ProtocolViolation is a bit different; this is called when a analyzer > encounters data in a protocol that it cannot parse (i.e. it is a violation > of how we think that the protocol should work). This is generally logged > into dpd.log, and the analyzer stops processing the connection after that. > You definitely should not just delete this function call, as it might mess > with what happens during protocol detection. > > If you want a Bro installation that does not instantiate most protocol > analyzers, you can just start Bro in bare mode (using -b), and only load > the scripts that you are interested in. By default Bro will not parse any > application layer protocols in bare mode (you should not even see conn.log > generated). > > Johanna > > On Sun, Jun 04, 2017 at 06:06:53PM +0300, william de ping wrote: > > Hi all, > > > > I am trying to save bro unnecessary events, weird is has quit a few hits > > that are not relevant to me. > > I see that under HTTP.cc or DNS.cc I have some redirection to WEIRD or > > ProtocolViolation analyzers. > > How can I delete the connection at this stage instead of sending it to > > another costly analyzer ? > > > > can I just comment it out ? > > > > Thank you, > > B > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170608/8f31f5c6/attachment.html From hongdal at g.clemson.edu Thu Jun 8 13:02:56 2017 From: hongdal at g.clemson.edu (Hongda Li) Date: Thu, 8 Jun 2017 16:02:56 -0400 Subject: [Bro] How to check the length of NDS request packets? Message-ID: Hi All, I am going to write a script that detects DNS tunneling. First the script checks all DNS request packets to see the length. If the length of a DNS request packet exceeds a threshold, say, 255 bytes, then this packet will be sent for DPI to check the requested domain name. The problem is the "dns_request" event does not provide packet length, which means, for every DNS request, I have to check the requested domain name. This is expensive. If I use "raw_packet" or "new_packet" events, then every new packet will trigger an event, which is also expensive. Is there a way that only triggers an event for a DNS request packet (e.g., based on the protocol and port number), and I can determine whether DPI is necessary for this DNS request packet based on its length? I am appreciate for any inputs! Best regards, Hongda ---------------------- Hongda Li, Graduate Research Assistant Division of Computer Science, School of Computing Clemson University Email: hongdal at clemson.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170608/dd6cbdf5/attachment.html From jazoff at illinois.edu Thu Jun 8 13:17:29 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 8 Jun 2017 20:17:29 +0000 Subject: [Bro] How to check the length of NDS request packets? In-Reply-To: References: Message-ID: > On Jun 8, 2017, at 4:02 PM, Hongda Li wrote: > > The problem is the "dns_request" event does not provide packet length, which means, for every DNS request, I have to check the requested domain name. This is expensive. Why do you say that it is expensive? Getting the length of a string in bro is an O(1) operation. -- - Justin Azoff From hongdal at g.clemson.edu Thu Jun 8 13:37:59 2017 From: hongdal at g.clemson.edu (Hongda Li) Date: Thu, 8 Jun 2017 16:37:59 -0400 Subject: [Bro] How to check the length of NDS request packets? In-Reply-To: References: Message-ID: >> On Jun 8, 2017, at 4:02 PM, Hongda Li wrote: >> >> The problem is the "dns_request" event does not provide packet length, which means, for every DNS request, I have to check the requested domain name. This is expensive. >Why do you say that it is expensive? Getting the length of a string in bro is an O(1) operation. Thanks, Justin. But I am worrying about the cost of DPI, since dns_request event contains dns_msg, query string and many other information that are not necessary when I only look at the length of the packet. For most of the DNS request packets, I would like to check the length. Only those packets with greater length will be checked for querying strings. Can I specify a filter that only checks the length of DNS request, like BPF, to the live traffic in my policy script? Best regards, Hongda ---------------------- Hongda Li, Graduate Research Assistant Division of Computer Science, School of Computing Clemson University Email: hongdal at clemson.edu On Thu, Jun 8, 2017 at 4:17 PM, Azoff, Justin S wrote: > > > On Jun 8, 2017, at 4:02 PM, Hongda Li wrote: > > > > The problem is the "dns_request" event does not provide packet length, > which means, for every DNS request, I have to check the requested domain > name. This is expensive. > > Why do you say that it is expensive? Getting the length of a string in > bro is an O(1) operation. > > -- > - Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170608/69ace87a/attachment.html From jdopheid at illinois.edu Fri Jun 9 06:56:39 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Fri, 9 Jun 2017 13:56:39 +0000 Subject: [Bro] Reminder BroCon CFP expires Friday! Message-ID: Reminder to submit your presentation proposal today! ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign ************************** BroCon ?17 is accepting presentation proposals. We are looking for talks to represent the many applications of Bro. Suitable topics include, but are not limited to: ? as a tool for solving problems; ? interesting user stories, solutions, or research projects; ? a postmortem analysis of a security incident, emphasizing Bro?s contribution; ? the value Bro brings to your professional work; ? and, using Bro for more than intrusion detection. ? Please, no product presentations Criteria for evaluating proposals include whether the topic is applicable to multiple types of organizations, gives people ideas to take home and use, can be understood by a broad audience, or is novel to many in the audience. Scrolling through our YouTube Channel may provide some insight into the types of presentations we wish to feature. Plan on limiting your talk to 30-35 minutes with an additional 10 minutes for questions/comments. Send abstracts (max 500 words) to: info at bro.org Subject: BroCon 2017 Call for Presentations Submission due date: Friday, June 9th Target date for announcing speakers: Friday June 30th CFPs are selected by the Bro Leadership Team: ? Johanna Amann, International Computer Science Institute ? Seth Hall, International Computer Science Institute ? Keith Lehigh, Indiana University ? Vern Paxson, University of California at Berkeley ? Michal Purzynski, Mozilla Foundation ? Aashish Sharma, Lawrence Berkeley Lab ? Adam Slagell, National Center for Supercomputing Applications ? Robin Sommer, International Computer Science Institute ? Martin van Hensbergen, Fox-IT ------ Jeannette M. Dopheide Bro Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170609/8ab65754/attachment-0001.html From vern at berkeley.edu Fri Jun 9 19:05:24 2017 From: vern at berkeley.edu (Vern Paxson) Date: Fri, 09 Jun 2017 19:05:24 -0700 Subject: [Bro] How to check the length of NDS request packets? In-Reply-To: (Thu, 08 Jun 2017 16:02:56 EDT). Message-ID: <20170610020524.5C4422C4104@rock.ICSI.Berkeley.EDU> > I am going to write a script that detects DNS tunneling. BTW, if it's not on your radar you should check out our paper on doing this: http://www.icir.org/vern/papers/covert-dns-usec13.pdf In generally, finding tunneling is much more involved than looking for long lookups, for example. Vern From oelnaggar04 at gmail.com Fri Jun 9 19:23:53 2017 From: oelnaggar04 at gmail.com (Osama Elnaggar) Date: Fri, 9 Jun 2017 19:23:53 -0700 Subject: [Bro] HTTPS Decryption Message-ID: Hi, I noticed the issue of decrypting HTTPS was mentioned several times over the years (with the last time back in 2015 I think - http://mailman.icsi.berkeley.edu/pipermail/bro/2015-June/008568.html) and was wondering if this feature was ever added or if anyone was able to successfully implement it. Thanks a lot. -- Osama Elnaggar -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170609/646b9ac6/attachment.html From johanna at icir.org Fri Jun 9 20:04:03 2017 From: johanna at icir.org (Johanna Amann) Date: Fri, 9 Jun 2017 20:04:03 -0700 Subject: [Bro] HTTPS Decryption In-Reply-To: References: Message-ID: <20170610030403.7jllup2xcforuz7c@Beezling.local> On Fri, Jun 09, 2017 at 07:23:53PM -0700, Osama Elnaggar wrote: > I noticed the issue of decrypting HTTPS was mentioned several times over > the years (with the last time back in 2015 I think - > http://mailman.icsi.berkeley.edu/pipermail/bro/2015-June/008568.html) and > was wondering if this feature was ever added or if anyone was able to > successfully implement it. No, not to my knowledge. There were several people who wanted to implement it over the years - if someone did it, they never open-sourced it. That being said - due to the prevalence of perfectly forward secure ciphers, TLS decryption is not really an option anymore in most use-cases. Johanna From oelnaggar04 at gmail.com Fri Jun 9 20:15:28 2017 From: oelnaggar04 at gmail.com (Osama Elnaggar) Date: Fri, 9 Jun 2017 20:15:28 -0700 Subject: [Bro] HTTPS Decryption In-Reply-To: <20170610030403.7jllup2xcforuz7c@Beezling.local> References: <20170610030403.7jllup2xcforuz7c@Beezling.local> Message-ID: Thanks Johanna. But I was actually looking at the use case where you terminated PFS at a load balancer (or other device at the perimeter) and used upstream SSL (non PFS) to the backend servers. Would it be possible to forward SSL packets to viewssld - https://github.com/plashchynski/viewssld - and then back to Bro? Thanks. -- Osama Elnaggar On June 10, 2017 at 1:04:05 PM, Johanna Amann (johanna at icir.org) wrote: On Fri, Jun 09, 2017 at 07:23:53PM -0700, Osama Elnaggar wrote: > I noticed the issue of decrypting HTTPS was mentioned several times over > the years (with the last time back in 2015 I think - > http://mailman.icsi.berkeley.edu/pipermail/bro/2015-June/008568.html) and > was wondering if this feature was ever added or if anyone was able to > successfully implement it. No, not to my knowledge. There were several people who wanted to implement it over the years - if someone did it, they never open-sourced it. That being said - due to the prevalence of perfectly forward secure ciphers, TLS decryption is not really an option anymore in most use-cases. Johanna -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170609/b6365131/attachment.html From dnj0496 at gmail.com Fri Jun 9 21:21:25 2017 From: dnj0496 at gmail.com (Dk Jack) Date: Fri, 9 Jun 2017 21:21:25 -0700 Subject: [Bro] string_to_pattern Message-ID: Hi, I am trying to solve a problem, where I am analyzing some http traffic using bro. To limit the bro log sizes, I want to capture only those http events which which have certain string patterns in their bodies. The string patterns will be unique for each host + uri pair. I am putting this info a file host, uri, regex in a file and loading it into bro using file input framework. I want to apply the regex on the http body if the host and uri matches. When I try to use to search the body using: find_all(body, string_to_pattern(regex_string_from_file, T)); I get some very weird behavior. The code in the entire block after string_to_pattern statement is not executed (and I don't get any error). This happens when I run it on command line against a pcap. To understand the problem better, I tried reproduce the problem on try.bro.org. I get the following error when I use string_to_pattern in my script on the try.bro.org website. Would like to understand the reason behind this restriction? Also, Would like to know there any alternative solutions I can pursue to solve my problem? Any help is appreciated. Thanks. Dk. 1320279566.452687 error in ././trybro.bro, line 17: string_to_pattern can only be called at init time (string_to_pattern(Hello, World, T)) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170609/8211454f/attachment.html From bill.de.ping at gmail.com Sat Jun 10 23:01:49 2017 From: bill.de.ping at gmail.com (william de ping) Date: Sun, 11 Jun 2017 09:01:49 +0300 Subject: [Bro] - Skip Weird or ProtocolViolation analyzer In-Reply-To: References: <20170606163634.2ah5oqxapldvkjlv@wifi109.sys.ICSI.Berkeley.EDU> Message-ID: Thank you very much ! I was not aware of that option On Thu, Jun 8, 2017 at 11:50 PM, Daniel Thayer wrote: > When running Bro from broctl, you can pass command-line > options to bro by setting a value for the "broargs" option > in your etc/broctl.cfg file. > > For example, you can add this line to your etc/broctl.cfg file: > broargs = -b > > > On 6/8/17 9:55 AM, william de ping wrote: > >> Hi, >> >> Yes I do see better results with bare mode. >> >> However, is it possible to run Broctl in bare mode ? >> >> Thanks, >> B >> >> On Tue, Jun 6, 2017 at 7:36 PM, Johanna Amann > > wrote: >> >> Hi, >> >> Weird and ProtocolViolation are no analyzers, and because of that >> they are >> not especially costly. Weird is generally called when one of the >> protocol >> analyzers notices something "weird" happening in the protocol; this is >> then logged directly to weird.log. While you can disable this function >> call, I don't really think you will see significant performance gains >> by >> this. >> >> ProtocolViolation is a bit different; this is called when a analyzer >> encounters data in a protocol that it cannot parse (i.e. it is a >> violation >> of how we think that the protocol should work). This is generally >> logged >> into dpd.log, and the analyzer stops processing the connection after >> that. >> You definitely should not just delete this function call, as it >> might mess >> with what happens during protocol detection. >> >> If you want a Bro installation that does not instantiate most protocol >> analyzers, you can just start Bro in bare mode (using -b), and only >> load >> the scripts that you are interested in. By default Bro will not >> parse any >> application layer protocols in bare mode (you should not even see >> conn.log >> generated). >> >> Johanna >> >> On Sun, Jun 04, 2017 at 06:06:53PM +0300, william de ping wrote: >> > Hi all, >> > >> > I am trying to save bro unnecessary events, weird is has quit a >> few hits >> > that are not relevant to me. >> > I see that under HTTP.cc or DNS.cc I have some redirection to WEIRD >> or >> > ProtocolViolation analyzers. >> > How can I delete the connection at this stage instead of sending it >> to >> > another costly analyzer ? >> > >> > can I just comment it out ? >> > >> > Thank you, >> > B >> >> > _______________________________________________ >> > Bro mailing list >> > bro at bro-ids.org >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > mailman.ICSI.Berkeley.EDU_mailman_listinfo_bro&d=DwMFaQ&c= >> 8hUWFZcy2Z-Za5rBPlktOQ&r=Bi5qPBnY0NmYPqnRTPj_AfXQKpfQTZUpCzp >> fFBcawv0&m=-guHXAqiujhzCMbLN4s2XOGSKix7YchSqGOtzEKMOkg&s= >> 5OUdqDoBzW6-MO4SWOpKaShy3Hf6f1xdpYGHg3p7e9A&e=> >> >> >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170611/70b6c66b/attachment.html From Izik.Birka at hot.net.il Sun Jun 11 02:02:05 2017 From: Izik.Birka at hot.net.il (Izik Birka) Date: Sun, 11 Jun 2017 09:02:05 +0000 Subject: [Bro] actions in smb_files.log Message-ID: <592228F4D0C8504187F2F76658040CB6E005B645@HOT-MAILBOX-02.HOT.NET.IL> Hi What is the reason that I don't get action SMB::FILE_WRITE? I try to copy files \ create new files \ change files but still no SMB::FILE_WRITE I only have open,rename,delete Thanks Izik Birka This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain materials protected by copyright or information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or agreement. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication by error, notify the sender immediately and delete this message immediately. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170611/9ec11081/attachment.html From anthony.kasza at gmail.com Sun Jun 11 05:23:55 2017 From: anthony.kasza at gmail.com (anthony kasza) Date: Sun, 11 Jun 2017 06:23:55 -0600 Subject: [Bro] actions in smb_files.log In-Reply-To: <592228F4D0C8504187F2F76658040CB6E005B645@HOT-MAILBOX-02.HOT.NET.IL> References: <592228F4D0C8504187F2F76658040CB6E005B645@HOT-MAILBOX-02.HOT.NET.IL> Message-ID: Hi Izik, Without an example pcap its difficult to troubleshoot your issue. Are you using a custom Bro script? -AK On Jun 11, 2017 5:04 AM, "Izik Birka" wrote: > Hi > > What is the reason that I don?t get action SMB::FILE_WRITE? > > I try to copy files \ create new files \ change files but still no > SMB::FILE_WRITE > > > > I only have open,rename,delete > > > > Thanks > > Izik Birka > > This message (including any attachments) is intended only for the use of > the individual or entity to which it is addressed and may contain materials > protected by copyright or information that is non-public, proprietary, > privileged, confidential, and exempt from disclosure under applicable law > or agreement. If you are not the intended recipient, you are hereby > notified that any use, dissemination, distribution, or copying of this > communication is strictly prohibited. If you have received this > communication by error, notify the sender immediately and delete this > message immediately. Thank you. > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170611/350520d7/attachment.html From zeutech at gmail.com Sun Jun 11 06:44:56 2017 From: zeutech at gmail.com (iraj norouzi) Date: Sun, 11 Jun 2017 18:14:56 +0430 Subject: [Bro] GUI interface Message-ID: hi does bro give GUI Dashboard for analysis data? does bro have any GUI for reporting? *Regards,Iraj Norouzi* *+989122494558* -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170611/6b0a667e/attachment-0001.html From blackhole.em at gmail.com Sun Jun 11 07:17:09 2017 From: blackhole.em at gmail.com (Joe Blow) Date: Sun, 11 Jun 2017 10:17:09 -0400 Subject: [Bro] GUI interface In-Reply-To: Message-ID: <593d50e8.c9d3240a.e0e15.ea41@mx.google.com> An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170611/bb9eb968/attachment.html From dave.a.florek at gmail.com Sun Jun 11 11:06:35 2017 From: dave.a.florek at gmail.com (Dave Florek) Date: Sun, 11 Jun 2017 14:06:35 -0400 Subject: [Bro] Bro Digest, Vol 134, Issue 14 In-Reply-To: References: Message-ID: Iraj, try BroTop. > Message: 4 > Date: Sun, 11 Jun 2017 18:14:56 +0430 > From: iraj norouzi > Subject: [Bro] GUI interface > To: bro at bro.org > Message-ID: > gmail.com> > Content-Type: text/plain; charset="utf-8" > > hi > does bro give GUI Dashboard for analysis data? > does bro have any GUI for reporting? > > *Regards,Iraj Norouzi* > *+989122494558* > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/ > 20170611/6b0a667e/attachment.html > > ------------------------------ > > _______________________________________________ > Bro mailing list > Bro at bro.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > End of Bro Digest, Vol 134, Issue 14 > ************************************ > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170611/5cb8a747/attachment.html From ernestfarias at gmail.com Tue Jun 13 00:57:37 2017 From: ernestfarias at gmail.com (Ernest Farias) Date: Tue, 13 Jun 2017 09:57:37 +0200 Subject: [Bro] bro scripts global vars Message-ID: <1497340657.6910.38.camel@gmail.com> Hi all What's the best way to know the value of globas vars on my loaded bro scripts? I need to know if some vars redefs are in fact in place. Thanks, Regards Ernest From alkenepan at gmail.com Tue Jun 13 06:09:24 2017 From: alkenepan at gmail.com (Alkene Pan) Date: Tue, 13 Jun 2017 21:09:24 +0800 Subject: [Bro] Question about Bro manager write data to kafka Message-ID: Hi Bro, i'm encountered a performance issue about Bro manager write data to kafka. Can anyone help me please? System details: Operation System: CentOS 7.2 CPU: - Model name: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz - CPU(s): 32 - CPU MHz: 2334.445 Memory: - 64GB Network Interface: - 03:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) Disks: Operation system is running with SSD drive. The Kafka log will write into the RAID0(Two HDD). Bro Cluster Config details: [manager] type=manager host=localhost [proxy-1] type=proxy host=localhost [worker-1] type=worker host=localhost interface=eno1 aux_scripts= -C lb_method=pf_ring lb_procs=15 pin_cpus=3,5,7,9,11,13,15,17,19,21,23,25,27,29,31 Kakfa Config details: 1 broker listeners=PLAINTEXT://10.0.81.60:9091 advertised.listeners=PLAINTEXT://10.0.81.60:9091 num.network.threads=60 num.io.threads=120 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=1024000 socket.request.max.bytes=104857600 log.dirs=/data-kafka/kafka-logs num.partitions=10 num.recovery.threads.per.data.dir=1 log.flush.interval.messages=10000 log.flush.interval.ms=1000 log.retention.hours=5 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 (Bro & kafka) are installed and running with a single machine. Bro Kafka plugin: https://github.com/bro/bro-plugins/tree/master/kafka librdkafka: librdkafka-0.9.4.tar.gz Kafka: http://ftp.cuhk.edu.hk/pub/packages/apache.org/kafka/0.10.0.0/kafka_2.11-0.10.0.0.tgz Bro load the custom scripts, which will reassembly the http data(include full request and response), the data format as below: {"ts":1497353836.648655,"sip":"10.0.85.9","sport":60484,"dip":"10.0.81.48","dport":80,"protocol":"http","sensor":"10.0.81.60","worker":"worker-1-3","http.request.host":"10.0.81.48","http.request.uri":"/xab.html","http.request.method":"GET","http.request.body":"","http.request.body_len":0,"http.request.header_names":["HOST","USER-AGENT","ACCEPT"],"http.request.header_values":["10.0.81.48","curl/7.51.0","*/*"],"http.request.range_request":false,"http.request.username":"","http.request.password":"","http.response.status_code":200,"http.response.status_msg":"OK","http.response.body":"\u000a\u000atest\u000a\u000a\u000a","http.response.body_len":35,"http.response.header_names":["SERVER","DATE","CONTENT-TYPE","CONTENT-LENGTH","LAST-MODIFIED","CONNECTION","ETAG","ACCEPT-RANGES"],"http.response.header_values":["nginx/1.10.2","Tue, 13 Jun 2017 11:37:16 GMT","text/html","35","Tue, 13 Jun 2017 11:25:40 GMT","keep-alive","\u0022593fcbb4-23\u0022","bytes"]} We set up a const variable to limit http response length(like `const http_max_body_len: count = 10240;`). Scenario: The details of configuration as below: - Bro capture network flow average size: 700mbps - the single http response length: 100kb - set the limit of http response length: http.response.body=51200(50kb) System performance status as below: - Loopback network flow average size: 1gbps - Total disk write: 100MB/s Test Results: We used `htop` and `watch /usr/local/bro/bin/broctl top` to watch the system and bro status. The OS memory usage grow with time until fill up. The operation system will be unstable. Bro Manager used 40G+ memory. But each worker memory usage size: 200MB. So we performed another test which reduce the limit of http response length to 5kb in bro script. After testing, the bro manager memory usage will remain around 130MB. In other test, we load 500mbps on NIC and Bro Manager use 4G memory(with 40960 http response data limit), but when we stop the performance test, the Manager memory usage is not reduce, just keep in 4G(we use vmstat durning in test). In summary, we assume the write rate(Bro Manager write data to Kafka) less than Bro Manager generate the data rate. Which leads to the Bro Manager high memory usage. The mechanism is correct? Or Bro Manager exist performance issue about write a huge data into Kafka? Or incorrect configuration? Please kindly let me know if you have any recommendation. Thank you so much. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170613/43f84fde/attachment.html From jazoff at illinois.edu Tue Jun 13 06:24:45 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 13 Jun 2017 13:24:45 +0000 Subject: [Bro] Question about Bro manager write data to kafka In-Reply-To: References: Message-ID: > On Jun 13, 2017, at 9:09 AM, Alkene Pan wrote: > > Hi Bro, i'm encountered a performance issue about Bro manager write data to kafka. Can anyone help me please? ... > > Bro Cluster Config details: > [manager] > type=manager > host=localhost > > [proxy-1] > type=proxy > host=localhost > > [worker-1] > type=worker > host=localhost > interface=eno1 > aux_scripts= -C > lb_method=pf_ring > lb_procs=15 > pin_cpus=3,5,7,9,11,13,15,17,19,21,23,25,27,29,31 > ... > The mechanism is correct? Or Bro Manager exist performance issue about write a huge data into Kafka? Or incorrect configuration? Please kindly let me know if you have any recommendation. Thank you so much. You're not running a logger process which will easily double the performance of your cluster. Add [logger] type=logger host=localhost to your node.cfg If you install the bro 2.5.1 beta you can have two or more loggers defined: [logger-1] type=logger host=localhost [logger-2] type=logger host=localhost (This is specifically intended for things like kafka) -- - Justin Azoff From zeolla at gmail.com Tue Jun 13 06:47:28 2017 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Tue, 13 Jun 2017 13:47:28 +0000 Subject: [Bro] Question about Bro manager write data to kafka In-Reply-To: References: Message-ID: I am working on a newer version of the Kafka writer plugin (as a part of the Apache Metron project, which is where the plugin was initially created) which has support for sending to kerberized Kafka, some bug fixes, better debug logging, etc. It currently exists here , but I'm going to be turning it into a bro package and moving it here eventually (once it has more testing). If you're willing to beta test a bit, perhaps it's worth giving a shot, in addition to Justin's comments? Jon On Tue, Jun 13, 2017 at 9:27 AM Azoff, Justin S wrote: > > > On Jun 13, 2017, at 9:09 AM, Alkene Pan wrote: > > > > Hi Bro, i'm encountered a performance issue about Bro manager write data > to kafka. Can anyone help me please? > ... > > > > Bro Cluster Config details: > > [manager] > > type=manager > > host=localhost > > > > [proxy-1] > > type=proxy > > host=localhost > > > > [worker-1] > > type=worker > > host=localhost > > interface=eno1 > > aux_scripts= -C > > lb_method=pf_ring > > lb_procs=15 > > pin_cpus=3,5,7,9,11,13,15,17,19,21,23,25,27,29,31 > > > ... > > > The mechanism is correct? Or Bro Manager exist performance issue about > write a huge data into Kafka? Or incorrect configuration? Please kindly let > me know if you have any recommendation. Thank you so much. > > You're not running a logger process which will easily double the > performance of your cluster. Add > > [logger] > type=logger > host=localhost > > to your node.cfg > > If you install the bro 2.5.1 beta you can have two or more loggers defined: > > > [logger-1] > type=logger > host=localhost > > [logger-2] > type=logger > host=localhost > > (This is specifically intended for things like kafka) > > -- > - Justin Azoff > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170613/8c3019fb/attachment-0001.html From archeldeeb at gmail.com Tue Jun 13 07:43:54 2017 From: archeldeeb at gmail.com (Sherif Eldeeb) Date: Tue, 13 Jun 2017 17:43:54 +0300 Subject: [Bro] Allowing only certain log types Message-ID: We are planning to only use the "logging" features of Bro, and for certain types, on a 10G link. I'd appreciate pointing me to right direction to only enable (conn.log, dns.log, http.log and ssl.log) while disabling all the others (to save processing cycles and storage) for the types that we won't use/need. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170613/97cdb2fd/attachment.html From craig.edgmand at okstate.edu Tue Jun 13 07:59:18 2017 From: craig.edgmand at okstate.edu (Edgmand, Craig) Date: Tue, 13 Jun 2017 14:59:18 +0000 Subject: [Bro] Bro restrict filters question Message-ID: Hello, I am running Bro 2.5 and I am trying to set up some restrict_filters to drop certain hosts and types of traffic. I have the following entries in my local.bro.. redef PacketFilter::enable_auto_protocol_capture_filters = F; redef capture_filters = { ["packets-like-this"] = "ip or not ip" }; redef restrict_filters = { ["no-data-like-this"] = "not host 192.168.2.1" }; I had something similar in earlier versions of Bro that seemed to work but this doesn't work at all. When I run ./broctl print restrict_filters it shows that the workers have that filter. Any ideas? Thanks, Craig Edgmand Oklahoma State University -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170613/b6b679e1/attachment.html From jazoff at illinois.edu Tue Jun 13 08:12:41 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 13 Jun 2017 15:12:41 +0000 Subject: [Bro] Bro restrict filters question In-Reply-To: References: Message-ID: <6C4D5D51-1D0B-46AC-AA42-B5B2F4E6A806@illinois.edu> > On Jun 13, 2017, at 10:59 AM, Edgmand, Craig wrote: > > Hello, > > I am running Bro 2.5 and I am trying to set up some restrict_filters to drop certain hosts and types of traffic. > I have the following entries in my local.bro.. > > redef PacketFilter::enable_auto_protocol_capture_filters = F; > redef capture_filters = { ["packets-like-this"] = "ip or not ip" }; > redef restrict_filters = { ["no-data-like-this"] = "not host 192.168.2.1" }; > > > I had something similar in earlier versions of Bro that seemed to work but this doesn?t work at all. > > When I run ./broctl print restrict_filters it shows that the workers have that filter. > > Any ideas? Is your traffic vlan tagged? You may need to use redef restrict_filters = { ["no-data-like-this"] = "vlan and not host 192.168.2.1" }; -- - Justin Azoff From vikrambasu059 at gmail.com Tue Jun 13 08:29:34 2017 From: vikrambasu059 at gmail.com (Vikram Basu) Date: Tue, 13 Jun 2017 20:59:34 +0530 Subject: [Bro] bro_beacons.bro - bad conversion to count Message-ID: <594004de.c59d630a.d382e.bec4@mx.google.com> Hi, I am using the bro_beacons.bro script and it generating multiple errors like 1486782334.477425 error in /usr/local/bro/share/bro/site/bro_beacons.bro, line 46: bad conversion to count (double_to_count(interval_to_double(BEACON::collection[BEACON::i + (coerce 1 to int)] - BEACON::collection[BEACON::i])) and -3.57259) Here is the script in question #Author: Nick Hoffman / securitykitten.github.io / @infoseckitten #Description: A bro script to find beacons module BEACON; @load base/protocols/http #this is our master collection, we'll use this to store all our information global master_collection: table[addr,addr] of vector of time &synchronized; export { redef enum Log::ID += { LOG }; type Info: record { ts: time &log; #id: conn_id &log; local_host: addr &log; remote_host: addr &log; entropy: double &log; }; global log_beacon: event(rec: Info); # Add hosts to ignore with: # redef BEACON::whitelist += {192.168.0.1/32, 192.168.1.0/24} const whitelist: set [subnet] = set() &redef; } event bro_init() { Log::create_stream(BEACON::LOG, [$columns=Info, $ev=log_beacon]); } function calculate_entropy(host: addr, server: addr): double { local collection = master_collection[host,server]; local entropy: count; local length = |collection|; local intervals = vector(); local pmf: table[time] of double; local probs: table[time] of double; local sum: double; sum = 0; for (i in collection) { if ( i+1 >= length ) break; else { intervals[i] = double_to_interval(double_to_count(interval_to_double(collection[i+1] - collection[i]))); } } #i don't like this solution, oh well for (i in intervals) { if ( intervals[i] !in pmf ) pmf[intervals[i]] = 1; else pmf[intervals[i]] += 1; } #calculate the probabilities for (i in intervals) { probs[intervals[i]] = pmf[intervals[i]] / |intervals|; } for (k in probs) { sum += probs[k] * (log10(probs[k]) / log10(2.0)); } if (double_to_time(0.0) in probs) { if (probs[double_to_time(0.0)] > 0.3) sum = 4; } #debug statement #print fmt("host:%s,server:%s,entropy:%s,interval:%s",host,server,|sum|,intervals); return |sum|; } #we'll start with http posts, in the case that event http_request(c: connection, method: string, original_URI: string, unescaped_URI: string, version: string) { #declare variables local host: addr; local server: addr; local ts: time; local uid: string; local entropy_result: double; for (sn in whitelist) { if (c$id$resp_h in sn || c$id$orig_h in sn ) return; } if ( method == "POST" || method == "GET" ) { #grab the relevant information host = c$id$orig_h; server = c$id$resp_h; ts = c$start_time; uid = c$uid; if ( [host,server] !in master_collection ){ master_collection[host,server] = vector(ts) ; } else { master_collection[host,server][|master_collection[host,server]|] = ts; if ( |master_collection[host,server]| > 12) { entropy_result = calculate_entropy(host,server); if (entropy_result < 0.75 ) { print fmt("%s - beacon %s and %s", ts, host, server); local rec: BEACON::Info = [$ts=ts, $entropy=entropy_result,$local_host=host,$remote_host=server]; Log::write(BEACON::LOG, rec); } master_collection[host,server] = vector(); } } } } Can anyone tell me how to solve this ? The issue seems to be this line intervals[i] = double_to_interval(double_to_count(interval_to_double(collection[i+1] - collection[i]))); Help? Regards Vikram Basu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170613/60325c1a/attachment-0001.html From johanna at icir.org Tue Jun 13 10:05:57 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 13 Jun 2017 10:05:57 -0700 Subject: [Bro] HTTPS Decryption In-Reply-To: References: <20170610030403.7jllup2xcforuz7c@Beezling.local> Message-ID: <20170613170557.om4vvfshfa6imf3w@Beezling.local> Oh - sorry, I misunderstood the question. In any case - no, as far as I know, no one has done exactly what I said in the original thread (stripping encryption while keeping the framing intact). That would need modifications to Bro; nothing changed since the thread you linked to. I don't jnow viewssld; if it outputs just a decrypted HTTP stream, Bro will pick it up by itself. There are a number of people that just use Bro behind a SSL terminator, which is kind of similar conceptually. If it outputs some other format, you will have to adjust the Bro protocol parsers. Johanna On Fri, Jun 09, 2017 at 08:15:28PM -0700, Osama Elnaggar wrote: > Thanks Johanna. But I was actually looking at the use case where you > terminated PFS at a load balancer (or other device at the perimeter) and > used upstream SSL (non PFS) to the backend servers. > > Would it be possible to forward SSL packets to viewssld - > https://github.com/plashchynski/viewssld - and then back to Bro? > > Thanks. > > -- > Osama Elnaggar > > On June 10, 2017 at 1:04:05 PM, Johanna Amann (johanna at icir.org) wrote: > > On Fri, Jun 09, 2017 at 07:23:53PM -0700, Osama Elnaggar wrote: > > I noticed the issue of decrypting HTTPS was mentioned several times over > > the years (with the last time back in 2015 I think - > > http://mailman.icsi.berkeley.edu/pipermail/bro/2015-June/008568.html) and > > was wondering if this feature was ever added or if anyone was able to > > successfully implement it. > > No, not to my knowledge. There were several people who wanted to implement > it over the years - if someone did it, they never open-sourced it. > > That being said - due to the prevalence of perfectly forward secure > ciphers, TLS decryption is not really an option anymore in most use-cases. > > Johanna From johanna at icir.org Tue Jun 13 10:11:13 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 13 Jun 2017 10:11:13 -0700 Subject: [Bro] bro_beacons.bro - bad conversion to count In-Reply-To: <594004de.c59d630a.d382e.bec4@mx.google.com> References: <594004de.c59d630a.d382e.bec4@mx.google.com> Message-ID: <20170613171113.e3mxnyvexi4uccwp@Beezling.local> The problem is that a negative number was passed to double_to_count; count is an unsigned integer value and cannot represent that. On a first glance, I am not quite sure why all those conversions take place in the first place - it seems to convert from an interval to double, then to count, and then back to interval. I would just try to either skip the conversions altogether, or if the idea is to get rid of the positions after the decimal place, to convert to int instead. I hope that helps, Johanna On Tue, Jun 13, 2017 at 08:59:34PM +0530, Vikram Basu wrote: > Hi, > > I am using the bro_beacons.bro script and it generating multiple errors like > > 1486782334.477425 error in /usr/local/bro/share/bro/site/bro_beacons.bro, line 46: bad conversion to count (double_to_count(interval_to_double(BEACON::collection[BEACON::i + (coerce 1 to int)] - BEACON::collection[BEACON::i])) and -3.57259) > > Here is the script in question > > > #Author: Nick Hoffman / securitykitten.github.io / @infoseckitten > #Description: A bro script to find beacons > > module BEACON; > > @load base/protocols/http > > #this is our master collection, we'll use this to store all our information > global master_collection: table[addr,addr] of vector of time &synchronized; > > export { > redef enum Log::ID += { LOG }; > type Info: record { > ts: time &log; > #id: conn_id &log; > local_host: addr &log; > remote_host: addr &log; > entropy: double &log; > }; > global log_beacon: event(rec: Info); > > # Add hosts to ignore with: > # redef BEACON::whitelist += {192.168.0.1/32, 192.168.1.0/24} > const whitelist: set [subnet] = set() &redef; > > } > event bro_init() > { > Log::create_stream(BEACON::LOG, [$columns=Info, $ev=log_beacon]); > } > > function calculate_entropy(host: addr, server: addr): double > { > local collection = master_collection[host,server]; > local entropy: count; > local length = |collection|; > local intervals = vector(); > local pmf: table[time] of double; > local probs: table[time] of double; > local sum: double; > sum = 0; > for (i in collection) { > if ( i+1 >= length ) > break; > else { > intervals[i] = double_to_interval(double_to_count(interval_to_double(collection[i+1] - collection[i]))); > } > } > > #i don't like this solution, oh well > for (i in intervals) { > if ( intervals[i] !in pmf ) > pmf[intervals[i]] = 1; > else > pmf[intervals[i]] += 1; > } > #calculate the probabilities > for (i in intervals) { > probs[intervals[i]] = pmf[intervals[i]] / |intervals|; > } > for (k in probs) { > sum += probs[k] * (log10(probs[k]) / log10(2.0)); > } > if (double_to_time(0.0) in probs) { > if (probs[double_to_time(0.0)] > 0.3) > sum = 4; > } > #debug statement > #print fmt("host:%s,server:%s,entropy:%s,interval:%s",host,server,|sum|,intervals); > return |sum|; > } > > #we'll start with http posts, in the case that > event http_request(c: connection, method: string, original_URI: string, unescaped_URI: string, version: string) { > #declare variables > local host: addr; > local server: addr; > local ts: time; > local uid: string; > local entropy_result: double; > > for (sn in whitelist) { > if (c$id$resp_h in sn || c$id$orig_h in sn ) > return; > } > > if ( method == "POST" || method == "GET" ) { > #grab the relevant information > host = c$id$orig_h; > server = c$id$resp_h; > ts = c$start_time; > uid = c$uid; > if ( [host,server] !in master_collection ){ > master_collection[host,server] = vector(ts) ; > } > else { > master_collection[host,server][|master_collection[host,server]|] = ts; > if ( |master_collection[host,server]| > 12) { > entropy_result = calculate_entropy(host,server); > if (entropy_result < 0.75 ) { > print fmt("%s - beacon %s and %s", ts, host, server); > local rec: BEACON::Info = [$ts=ts, $entropy=entropy_result,$local_host=host,$remote_host=server]; > Log::write(BEACON::LOG, rec); > } > master_collection[host,server] = vector(); > } > } > } > } > > > > Can anyone tell me how to solve this ? > > The issue seems to be this line > intervals[i] = double_to_interval(double_to_count(interval_to_double(collection[i+1] - collection[i]))); > > Help? > > Regards > > Vikram Basu > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Tue Jun 13 10:12:23 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 13 Jun 2017 10:12:23 -0700 Subject: [Bro] bro scripts global vars In-Reply-To: <1497340657.6910.38.camel@gmail.com> References: <1497340657.6910.38.camel@gmail.com> Message-ID: <20170613171223.k4xkbu22bbpegrnd@Beezling.local> Hi, > What's the best way to know the value of globas vars on my loaded bro > scripts? The easiest way probably is to just check their values in a bro_init event. Unless there is a reason that you can't do that? Johanna From craig.edgmand at okstate.edu Tue Jun 13 12:05:29 2017 From: craig.edgmand at okstate.edu (Edgmand, Craig) Date: Tue, 13 Jun 2017 19:05:29 +0000 Subject: [Bro] Bro restrict filters question In-Reply-To: <6C4D5D51-1D0B-46AC-AA42-B5B2F4E6A806@illinois.edu> References: <6C4D5D51-1D0B-46AC-AA42-B5B2F4E6A806@illinois.edu> Message-ID: Oddly enough it works with tcpdump but not with Bro. -----Original Message----- From: Azoff, Justin S [mailto:jazoff at illinois.edu] Sent: Tuesday, June 13, 2017 10:13 AM To: Edgmand, Craig Cc: bro at bro.org Subject: Re: [Bro] Bro restrict filters question > On Jun 13, 2017, at 10:59 AM, Edgmand, Craig wrote: > > Hello, > > I am running Bro 2.5 and I am trying to set up some restrict_filters to drop certain hosts and types of traffic. > I have the following entries in my local.bro.. > > redef PacketFilter::enable_auto_protocol_capture_filters = F; redef > capture_filters = { ["packets-like-this"] = "ip or not ip" }; redef > restrict_filters = { ["no-data-like-this"] = "not host 192.168.2.1" }; > > > I had something similar in earlier versions of Bro that seemed to work but this doesn?t work at all. > > When I run ./broctl print restrict_filters it shows that the workers have that filter. > > Any ideas? Is your traffic vlan tagged? You may need to use redef restrict_filters = { ["no-data-like-this"] = "vlan and not host 192.168.2.1" }; -- - Justin Azoff From matt.clemons at gmail.com Tue Jun 13 13:02:54 2017 From: matt.clemons at gmail.com (Matt Clemons) Date: Tue, 13 Jun 2017 15:02:54 -0500 Subject: [Bro] ICMP crashing bro Message-ID: Has anyone had issues similar to below? They are crashing one of my workers several times per day. Sifting through PCAP, I don't see anything unusual. Checked logs for segfaults, and tried debugging bro with gdb, but that never works for me. Any insight? ==== stderr.log 1497373293.094293 analyzer error: unexpected IP proto in ICMP analyzer: 94 1497373315.914487 analyzer error: unexpected IP proto in ICMP analyzer: 140 1497376098.060558 analyzer error: unexpected IP proto in ICMP analyzer: 6 1497376877.285181 analyzer error: unexpected IP proto in ICMP analyzer: 120 1497377211.129670 analyzer error: unexpected IP proto in ICMP analyzer: 105 1497377776.944610 analyzer error: unexpected IP proto in ICMP analyzer: 5 1497378391.755041 analyzer error: unexpected IP proto in ICMP analyzer: 5 1497379522.608902 analyzer error: unexpected IP proto in ICMP analyzer: 76 1497380228.950978 analyzer error: unexpected IP proto in ICMP analyzer: 196 /opt/bro/share/broctl/scripts/run-bro: line 107: 128262 Segmentation fault (core dumped) nohup ${pin_command} $pin_cpu "$mybro" "$@" ====messages log 2017-06-12T19:08:08.651585+00:00 hostname kernel: bro[128194]: segfault at de1004a6 ip 00007fb37127d569 sp 00007ffe0e78f830 error 4 in libtcmalloc.so.4.1.0[7fb371259000+47000] 2017-06-12T19:10:40.117580+00:00 hostname kernel: bro[128192]: segfault at 714226d4 ip 00000000008cd944 sp 00007ffe80937740 error 4 in bro[400000+624000] 2017-06-12T20:14:42.590509+00:00 hostname kernel: bro[128103]: segfault at 5a4f02a6 ip 00007f6d161c3e6c sp 00007ffdba13e9d0 error 4 in libtcmalloc.so.4.1.0[7f6d161ac000+47000] 2017-06-13T12:36:24.514458+00:00 hostname kernel: bro[128283]: segfault at c7322eca ip 00007f175f2d5569 sp 00007ffe25199000 error 4 in libtcmalloc.so.4.1.0[7f175f2b1000+47000] 2017-06-13T12:37:22.767631+00:00 hostname kernel: bro[128279]: segfault at 82fb85c7 ip 00007f4c1a6f9569 sp 00007ffda0594120 error 4 in libtcmalloc.so.4.1.0[7f4c1a6d5000+47000] 2017-06-13T14:14:33.137432+00:00 hostname kernel: bro[128114]: segfault at b413ac8c ip 00007f4c4d89f569 sp 00007fff1b1267b0 error 4 in libtcmalloc.so.4.1.0[7f4c4d87b000+47000] -- Regards, Matt Clemons (816) 200-0789 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170613/12f981b2/attachment.html From fatema.bannatwala at gmail.com Tue Jun 13 14:28:33 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Tue, 13 Jun 2017 17:28:33 -0400 Subject: [Bro] AddressScan numbers and actual log number mismatch? Message-ID: Hi All, So we had an incident today where an IP got blocked because of doing Address Scan, as reported by Bro. But when asked to corroborate the activity with actual logs, I couldn't find the relevant logs or number of distinct IPs the scanner connected to. To clarify: Here is the log that reported an Address Scan: 1497360944.102926 Reporter::INFO AddressScan NOTICE 71.162.229.81 has *scanned 30 hosts (4282/tcp)* manager But when did a quick grep through the conn logs, only 5 distinct IPs showed up as oppose to 30: $ zcat conn.09:00:00-10:00:00.log.gz | grep "71.162.229.81" | grep "4282" | awk -F'\t' '{if ($6 == "4282") print $5}' | sort | uniq -c | sort -rn 38 128.x.x.x 26 128.y.y.y 20 128.z.z.z 2 128.k.k.k 2 128.j.j.j Even looked at last all conn logs, but still couldn't get "30 IPs", as reported by the notice log: $ zcat conn.*.log.gz | grep "71.162.229.81" | grep "4282" | awk -F'\t' '{if ($6 == "4282") print $5}' | sort | uniq | sort -rn 128.x.x.x 128.y.y.y 128.z.z.z 128.k.k.k 128.j.j.j Not sure why the numbers don't match up, also to mention, I am using the check-addressscan.bro script from Scan-NG scripts folder. Any idea? or if I am interpreting the logs correctly. Thanks, Fatema. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170613/3c8e211e/attachment.html From jazoff at illinois.edu Tue Jun 13 14:51:51 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 13 Jun 2017 21:51:51 +0000 Subject: [Bro] AddressScan numbers and actual log number mismatch? In-Reply-To: References: Message-ID: <633B2F2A-E88F-4AD9-B129-D511E69446EA@illinois.edu> > On Jun 13, 2017, at 5:28 PM, fatema bannatwala wrote: > > Hi All, > > So we had an incident today where an IP got blocked because of doing Address Scan, as reported by Bro. > ... > > Not sure why the numbers don't match up, also to mention, I am using the check-addressscan.bro script from Scan-NG scripts folder. > > Any idea? or if I am interpreting the logs correctly. The tables that it uses are: global distinct_peers: table[addr] of set[addr] &read_expire = 1 days &expire_func=scan_sum &redef; or (depending on mode) global c_distinct_peers: table[addr] of opaque of cardinality &default = function(n: any): opaque of cardinality { return hll_cardinality_init(0.1, 0.99); } &read_expire = 1 day ; for 30 hosts, the logs related to this scan could go as far back as 30 days. If the src ip was flagged as scanning one new IP every 12 hours the total length of the scan would be 15 days. So.. you are probably looking at the right logs, you just did not search far back enough in time. -- - Justin Azoff From jazoff at illinois.edu Tue Jun 13 15:32:48 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 13 Jun 2017 22:32:48 +0000 Subject: [Bro] AddressScan numbers and actual log number mismatch? In-Reply-To: References: <9EBC6833-9FED-4096-9A1F-8357A007F9B3@illinois.edu> Message-ID: <2B9FB252-356E-45C1-A7B5-4A14A42848FD@illinois.edu> > > On Jun 13, 2017, at 6:21 PM, fatema bannatwala wrote: > > Thanks Justin, quick search through the data for past 23 days still showed up only 5 IPs, all belonging to today's logs. > Hence, was thinking, that the port/service in the Notice is one of the several services Bro notices an address scan on, and only reports the last one? > or the address scan was actually performed on that service only. > > Looking at the script, I think the service port (4282 for ex.) is the port for which Address Scans get reported, but just wanted to verify, > as I still not able to see more than 5 IPs hit on that port by 71.162.229.81. Ah yes, I see now that you were filtering for the port. The policy counts scans across all ports. You'd need to look for failed connections on any port. You still may have to go back days to find the entire scan though. -- - Justin Azoff From valerio.click at gmx.com Tue Jun 13 16:04:14 2017 From: valerio.click at gmx.com (Valerio) Date: Wed, 14 Jun 2017 01:04:14 +0200 Subject: [Bro] Best way to contribute to existing analyzer Message-ID: Hi all, I'd like to ask guidance on how to contribute to BRO by proposing extensions to existing protocol analyzers. For instance, suppose that I realize a patch to the DHCP analyzer that includes new unsupported options. Such patch would impact on multiple files like those in src/analyzer/protocol/dhcp, scripts/base/protocols/dhcp as well as new types to be included in init-bare.bro. What would be the best procedure (and format) to submit such a patch? best, Valerio From robin at icir.org Tue Jun 13 16:28:32 2017 From: robin at icir.org (Robin Sommer) Date: Tue, 13 Jun 2017 16:28:32 -0700 Subject: [Bro] Best way to contribute to existing analyzer In-Reply-To: References: Message-ID: <20170613232832.GP10430@icir.org> On Wed, Jun 14, 2017 at 01:04 +0200, Valerio wrote: > What would be the best procedure (and format) to submit such a patch? Easiest is to prepare a pull request on GitHub. We have some guidelines here: https://www.bro.org/development/contribute.html#submitting-patches Looking forward to your patches! Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From hckim at narusec.com Tue Jun 13 18:40:22 2017 From: hckim at narusec.com (=?UTF-8?B?6rmA7Z2s7LKg?=) Date: Wed, 14 Jun 2017 10:40:22 +0900 Subject: [Bro] Allowing only certain log types Message-ID: Hi you could disable log by using Log::disable_stream to my knowledge it only stop writing to log. It does not going to save process reference --sample -- add this to local.bro or separate bro file event bro_init() { Log::disable_stream(Syslog::LOG); Log::disable_stream(PE::LOG); Log::disable_stream(X509::LOG); Log::disable_stream(SIP::LOG); Log::disable_stream(SNMP::LOG); Log::disable_stream(mysql::LOG); Log::disable_stream(Syslog::LOG); } -- ------------------------------------------------------ Hichul Kim ??? ?? ??? Naru Security (?)?????? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170614/eb95cdfa/attachment.html From bill.de.ping at gmail.com Wed Jun 14 01:23:21 2017 From: bill.de.ping at gmail.com (william de ping) Date: Wed, 14 Jun 2017 11:23:21 +0300 Subject: [Bro] - send syslog message via Bro Message-ID: Hello everyone, I see that there is a built in function named syslog that sends a string. I cannot provide it with any parameters. Is it possible to send a syslog message from a bro script to a specific host ? Thanks B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170614/e5c35a55/attachment.html From ernestfarias at gmail.com Wed Jun 14 02:51:25 2017 From: ernestfarias at gmail.com (Ernest Farias) Date: Wed, 14 Jun 2017 11:51:25 +0200 Subject: [Bro] bro scripts global vars In-Reply-To: <20170613171223.k4xkbu22bbpegrnd@Beezling.local> References: <1497340657.6910.38.camel@gmail.com> <20170613171223.k4xkbu22bbpegrnd@Beezling.local> Message-ID: <1497433885.6910.50.camel@gmail.com> Thanks Johanna! But now it arise another question it works fine when test on cmd line but using broctl and I supposed it would go to my ?/log/current/stdout.log (?) , but it only contains this, I don't know what I'm doing wrong "max memory size?????????(kbytes, -m) unlimited data seg size???????????(kbytes, -d) unlimited virtual memory??????????(kbytes, -v) unlimited core file size??????????(blocks, -c) unlimited" Thanks Ernest On Tue, 2017-06-13 at 10:12 -0700, Johanna Amann wrote: > Hi, > > > > > What's the best way to know the value of globas vars on my loaded > > bro > > scripts? > The easiest way probably is to just check their values in a bro_init > event. Unless there is a reason that you can't do that? > > Johanna From fatema.bannatwala at gmail.com Wed Jun 14 04:59:31 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Wed, 14 Jun 2017 07:59:31 -0400 Subject: [Bro] AddressScan numbers and actual log number mismatch? In-Reply-To: <2B9FB252-356E-45C1-A7B5-4A14A42848FD@illinois.edu> References: <9EBC6833-9FED-4096-9A1F-8357A007F9B3@illinois.edu> <2B9FB252-356E-45C1-A7B5-4A14A42848FD@illinois.edu> Message-ID: Thanks Justin! When I looked for all ports, excluding "SF" connections from conn log, I did get more than 30 IPs (31 in total). I think that would be it, causing Bro to mark that IP as Scanning addresses, if I am doing filtering correctly. $ nice zgrep --no-filename 71.162.229.81 conn.0[8-9]* | egrep -v "SF" | awk -F'\t' '{if ($1 < 1497360944) print $5, $6}' | sort | uniq -c On Tue, Jun 13, 2017 at 6:32 PM, Azoff, Justin S wrote: > > > > On Jun 13, 2017, at 6:21 PM, fatema bannatwala < > fatema.bannatwala at gmail.com> wrote: > > > > Thanks Justin, quick search through the data for past 23 days still > showed up only 5 IPs, all belonging to today's logs. > > Hence, was thinking, that the port/service in the Notice is one of the > several services Bro notices an address scan on, and only reports the last > one? > > or the address scan was actually performed on that service only. > > > > Looking at the script, I think the service port (4282 for ex.) is the > port for which Address Scans get reported, but just wanted to verify, > > as I still not able to see more than 5 IPs hit on that port by > 71.162.229.81. > > Ah yes, I see now that you were filtering for the port. The policy counts > scans across all ports. You'd need to look for failed connections on any > port. You still may have to go back days to find the entire scan though. > > > -- > - Justin Azoff > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170614/8a84e8d9/attachment.html From zhangxu1115 at gmail.com Wed Jun 14 11:49:34 2017 From: zhangxu1115 at gmail.com (Xu Zhang) Date: Wed, 14 Jun 2017 11:49:34 -0700 Subject: [Bro] exponential operation Message-ID: Hi all, 1. Is there an exponential operation in bro to do something like a**b, the built in exp(k) calculate e to the power of k. 2. Is there something like python range(1, 10) in bro? 3. how to initialize the size of a vector, in python e.g. [0 for i in range(5)], is there a similar approach in bro? I cannot find these in the bro documentation. -- Sincerely, Xu Zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170614/76f3d53d/attachment-0001.html From robin at icir.org Thu Jun 15 07:39:42 2017 From: robin at icir.org (Robin Sommer) Date: Thu, 15 Jun 2017 07:39:42 -0700 Subject: [Bro] string_to_pattern In-Reply-To: References: Message-ID: <20170615143942.GA41893@icir.org> On Fri, Jun 09, 2017 at 21:21 -0700, Dk Jack wrote: > 1320279566.452687 error in ././trybro.bro, line 17: string_to_pattern > can only be called at init time (string_to_pattern(Hello, World, T)) Bro currently doesn't support creating regular expressions values dynamically during runtime. The reason is an unfortunate internal deficiency: regexp values don't fully clean up their memory when deleted, meaning that frequent instantiating/destroying would lead to significant memory leaks. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From bill.de.ping at gmail.com Sun Jun 18 06:27:48 2017 From: bill.de.ping at gmail.com (william de ping) Date: Sun, 18 Jun 2017 16:27:48 +0300 Subject: [Bro] - BROKER - send messages from bro workers to python listener Message-ID: Hi all, I have been experiencing some difficulties in understanding the usage of Broker. I have a cluster and I wish bro workers to notify a single python script with a costume text message. On the python side I use pybroker. Do I need to use listen for the python side and send on the bro side ? How does the publishing works ? how can I implement it ? Thank you B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170618/093982de/attachment.html From zach.rogers at oregonstate.edu Mon Jun 19 11:47:27 2017 From: zach.rogers at oregonstate.edu (Zach Rogers) Date: Mon, 19 Jun 2017 11:47:27 -0700 Subject: [Bro] Reporter Error: No Such Index Message-ID: <1497898047.2802.4.camel@oregonstate.edu> Hello, When looking at my current reporter.log file, I noticed this error message: "Reporter::ERROR no such index (Cluster::nodes[Intel::p $descr]) /usr/local/bro/share/bro/base/frameworks/intel/./cluster.bro, line 35" I did some searching online and couldn't find much information regarding this error message. Does anyone know what the cause might be? In case this is at all helpful, I have a cluster of five machines for my current Bro configuration: One is acting as the Master & Proxy node, and the other four are Worker nodes. The four worker nodes have Myricom 10g NICs and are using the Bro::Myricom plugin. Everything seems to be working, and data is being logged. Any information would be greatly appreciated! Regards, -- Zach Rogers Security Analyst, Office of Information Security Information Services | Oregon State University http://is.oregonstate.edu/ois From jan.grashoefer at gmail.com Tue Jun 20 01:17:09 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Tue, 20 Jun 2017 10:17:09 +0200 Subject: [Bro] Reporter Error: No Such Index In-Reply-To: <1497898047.2802.4.camel@oregonstate.edu> References: <1497898047.2802.4.camel@oregonstate.edu> Message-ID: Hi Zach, > "Reporter::ERROR no such index (Cluster::nodes[Intel::p > $descr]) /usr/local/bro/share/bro/base/frameworks/intel/./cluster.bro, > line 35" > > I did some searching online and couldn't find much information regarding > this error message. Does anyone know what the cause might be? I remember others reporting the same issue but unfortunately I don't remember a solution. It might be a timing issue during the setup of the cluster. To have a closer look you could start by printing p$descr in that handshake event. Which Bro version are you using? Jan From cchiaverini at bnl.gov Tue Jun 20 06:27:51 2017 From: cchiaverini at bnl.gov (Chris Chiaverini) Date: Tue, 20 Jun 2017 09:27:51 -0400 Subject: [Bro] Bro node.cfg not setting Myricom Sniffer10G environment variables Message-ID: <30abafd5-fae4-f1d2-0df5-c62e82e777c4@bnl.gov> It seems that bro 2.5.1 is not taking the SNF_DATARING_SIZE variable, no matter what I set it to. When at the defaults in the /etc/bro/node.cfg and with nothing set at the shell, it still reports it is set via "userset" instead of "default" like SNF_DESCRING_SIZE. Here is the defaults: - Nothing at shell #env | grep SNF # - node.cfg with defaults #fgrep -A 12 worker-1 node.cfg [worker-1] type=worker host= interface=snf0 lb_method=myricom lb_procs=8 #lb_procs=7 ### Keep it on one NUMA node ## NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22 ## NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23 pin_cpus=2,4,6,8,10,12,14,16 env_vars=SNF_APP_ID=10,SNF_FLAGS=0x1,SNF_NUM_RINGS=8,SNF_DEBUG_MASK=3 # - defaults show "userset" for SNF_DATARING_SIZE #fgrep SNF_D /data01/bro/spool/worker-1-1/stderr.log 121273 snf.0.-1 P (*userset*) * SNF_DATARING_SIZE = 134217728 (0x8000000) (128.0 MiB)* 121273 snf.0.-1 P (*default*) *SNF_DESCRING_SIZE = 67108864 (0x4000000) (64.0 MiB)* 121273 snf.0.-1 P (environ) SNF_DEBUG_MASK = 3 (0x3) 121273 snf.0.-1 P (default) SNF_DEBUG_FILENAME = stderr 121273 snf.0.-1 P SNF_DEBUG_MASK=0x3 for modes WARN=0x1, PARAM=0x2 QSTATS=0x4 TIMESYNC=0x8 IOCTL=0x10 QEVENTS=0x20 ARISTA=0x40 # ------------------------------------------- - Manually setting *SNF_DATARING_SIZE* and *SNF_DESCRING_SIZE* #fgrep -A 12 worker-1 node.cfg [worker-1] type=worker host= interface=snf0 lb_method=myricom lb_procs=8 #lb_procs=7 ### Keep it on one NUMA node ## NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22 ## NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23 pin_cpus=2,4,6,8,10,12,14,16 #env_vars=SNF_APP_ID=10,SNF_FLAGS=0x1,SNF_NUM_RINGS=8,SNF_DEBUG_MASK=3 env_vars=SNF_APP_ID=10,SNF_FLAGS=0x1,SNF_NUM_RINGS=8,SNF_DEBUG_MASK=3,*SNF_DATARING_SIZE=4294967296,SNF_DESCRING_SIZE=1073741824* # - deploy new config with restart #/opt/bro/bin/broctl deploy - SNF_DATARING_SIZE still set to 128MB like in default and still reports that it was set via "userset" (should be 4GB set via "environ") #fgrep SNF_D /data01/bro/spool/worker-1-1/stderr.log 34852 snf.0.-1 P *(userset) SNF_DATARING_SIZE = 134217728 (0x8000000) (128.0 MiB)* 34852 snf.0.-1 P *(environ) SNF_DESCRING_SIZE = 1073741824 (0x40000000) (1024.0 MiB)* 34852 snf.0.-1 P (environ) SNF_DEBUG_MASK = 3 (0x3) 34852 snf.0.-1 P (default) SNF_DEBUG_FILENAME = stderr 34852 snf.0.-1 P SNF_DEBUG_MASK=0x3 for modes WARN=0x1, PARAM=0x2 QSTATS=0x4 TIMESYNC=0x8 IOCTL=0x10 QEVENTS=0x20 ARISTA=0x40 # -- Regards, Chris Chiaverini Cyber Security Operations Brookhaven National Laboratory Upton, New York 11973 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170620/49c40f30/attachment.html From jazoff at illinois.edu Tue Jun 20 06:46:55 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 20 Jun 2017 13:46:55 +0000 Subject: [Bro] Bro node.cfg not setting Myricom Sniffer10G environment variables In-Reply-To: <30abafd5-fae4-f1d2-0df5-c62e82e777c4@bnl.gov> References: <30abafd5-fae4-f1d2-0df5-c62e82e777c4@bnl.gov> Message-ID: <2831AF5A-E4E7-4C8C-B4FF-9838A580FA22@illinois.edu> > On Jun 20, 2017, at 9:27 AM, Chris Chiaverini wrote: > > It seems that bro 2.5.1 is not taking the SNF_DATARING_SIZE variable, no matter what I set it to. > > When at the defaults in the /etc/bro/node.cfg and with nothing set at the shell, it still reports it is set via "userset" instead of "default" like SNF_DESCRING_SIZE. Can you do this quick test using tcpdump to verify the problem is with bro/broctl or something with the myricom driver/library? SNF_APP_ID=10 SNF_FLAGS=0x1 SNF_NUM_RINGS=8 SNF_DEBUG_MASK=3 SNF_DATARING_SIZE=4294967296 SNF_DESCRING_SIZE=1073741824 tcpdump -n -i snf0 -c 1 When I run that I get 23681 snf.0.-1 P (userset) SNF_PORTNUM = 0 23681 snf.0.-1 P (default) SNF_RING_ID = -1 (0xffffffff) 23681 snf.0.-1 P (environ) SNF_NUM_RINGS = 8 (0x8) 23681 snf.0.-1 P (default) SNF_RSS_FLAGS = 49 (0x31) 23681 snf.0.-1 P (environ) SNF_DATARING_SIZE = 4294967296 (0x100000000) (4096.0 MiB) 23681 snf.0.-1 P (environ) SNF_DESCRING_SIZE = 1073741824 (0x40000000) (1024.0 MiB) 23681 snf.0.-1 P (userset) SNF_FLAGS = 1 (0x1) 23681 snf.0.-1 P (environ) SNF_DEBUG_MASK = 3 (0x3) 23681 snf.0.-1 P (default) SNF_DEBUG_FILENAME = stderr 23681 snf.0.-1 P (environ) SNF_APP_ID = 10 (0xa) -- - Justin Azoff From acarreno at ucsb.edu Tue Jun 20 07:09:15 2017 From: acarreno at ucsb.edu (Alejandro Carreno) Date: Tue, 20 Jun 2017 14:09:15 +0000 Subject: [Bro] Bro node.cfg not setting Myricom Sniffer10G environment variables In-Reply-To: <2831AF5A-E4E7-4C8C-B4FF-9838A580FA22@illinois.edu> References: <30abafd5-fae4-f1d2-0df5-c62e82e777c4@bnl.gov> <2831AF5A-E4E7-4C8C-B4FF-9838A580FA22@illinois.edu> Message-ID: I noticed this behavior as well a while back after upgrading SNF from 3.0.10 to 3.0.11. Downgrading back to 3.0.10 would return the ring sizes to the expected values. -Alex On Tue, Jun 20, 2017 at 6:47 AM Azoff, Justin S wrote: > > > On Jun 20, 2017, at 9:27 AM, Chris Chiaverini > wrote: > > > > It seems that bro 2.5.1 is not taking the SNF_DATARING_SIZE variable, no > matter what I set it to. > > > > When at the defaults in the /etc/bro/node.cfg and with nothing set at > the shell, it still reports it is set via "userset" instead of "default" > like SNF_DESCRING_SIZE. > > Can you do this quick test using tcpdump to verify the problem is with > bro/broctl or something with the myricom driver/library? > > SNF_APP_ID=10 SNF_FLAGS=0x1 SNF_NUM_RINGS=8 SNF_DEBUG_MASK=3 > SNF_DATARING_SIZE=4294967296 SNF_DESCRING_SIZE=1073741824 tcpdump -n -i > snf0 -c 1 > > When I run that I get > > 23681 snf.0.-1 P (userset) SNF_PORTNUM = 0 > 23681 snf.0.-1 P (default) SNF_RING_ID = -1 (0xffffffff) > 23681 snf.0.-1 P (environ) SNF_NUM_RINGS = 8 (0x8) > 23681 snf.0.-1 P (default) SNF_RSS_FLAGS = 49 (0x31) > 23681 snf.0.-1 P (environ) SNF_DATARING_SIZE = 4294967296 > (0x100000000) (4096.0 MiB) > 23681 snf.0.-1 P (environ) SNF_DESCRING_SIZE = 1073741824 > (0x40000000) (1024.0 MiB) > 23681 snf.0.-1 P (userset) SNF_FLAGS = 1 (0x1) > 23681 snf.0.-1 P (environ) SNF_DEBUG_MASK = 3 (0x3) > 23681 snf.0.-1 P (default) SNF_DEBUG_FILENAME = stderr > 23681 snf.0.-1 P (environ) SNF_APP_ID = 10 (0xa) > > > > -- > - Justin Azoff > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170620/5a7ccc4d/attachment.html From cchiaverini at bnl.gov Tue Jun 20 08:00:08 2017 From: cchiaverini at bnl.gov (Chris Chiaverini) Date: Tue, 20 Jun 2017 11:00:08 -0400 Subject: [Bro] Bro node.cfg not setting Myricom Sniffer10G environment variables In-Reply-To: <2831AF5A-E4E7-4C8C-B4FF-9838A580FA22@illinois.edu> References: <30abafd5-fae4-f1d2-0df5-c62e82e777c4@bnl.gov> <2831AF5A-E4E7-4C8C-B4FF-9838A580FA22@illinois.edu> Message-ID: <063fb6ec-7c55-bd59-f4f2-244ce931c385@bnl.gov> Justin, Looks like it was able to set it successfully at the shell (disregard the last line, I did not change the APP_ID from running process): # SNF_APP_ID=10 SNF_FLAGS=0x1 SNF_NUM_RINGS=8 SNF_DEBUG_MASK=3 SNF_DATARING_SIZE=4294967296 SNF_DESCRING_SIZE=1073741824 tcpdump -n -i snf0 -c 1 61474 snf.0.-1 P (userset) SNF_PORTNUM = 0 61474 snf.0.-1 P (default) SNF_RING_ID = -1 (0xffffffff) 61474 snf.0.-1 P (environ) SNF_NUM_RINGS = 8 (0x8) 61474 snf.0.-1 P (default) SNF_RSS_FLAGS = 49 (0x31) 61474 snf.0.-1 P*(environ) SNF_DATARING_SIZE = 4294967296 (0x100000000) (4096.0 MiB)* 61474 snf.0.-1 P (environ) SNF_DESCRING_SIZE = 1073741824 (0x40000000) (1024.0 MiB) 61474 snf.0.-1 P (userset) SNF_FLAGS = 1 (0x1) 61474 snf.0.-1 P (environ) SNF_DEBUG_MASK = 3 (0x3) 61474 snf.0.-1 P (default) SNF_DEBUG_FILENAME = stderr 61474 snf.0.-1 P (environ) SNF_APP_ID = 10 (0xa) 61474 snf.0.-1 P SNF_DEBUG_MASK=0x3 for modes WARN=0x1, PARAM=0x2 QSTATS=0x4 TIMESYNC=0x8 IOCTL=0x10 QEVENTS=0x20 ARISTA=0x40 61474 snf.0.-1 P lib version=3.0.11.50818 build=snf-3.0.11.50818_07ecd3440 03/16/17_08:43 07ecd3440 61474 snf.0.-1 P kernel version=3.0.11.50818 build=snf-3.0.11.50818_07ecd3440 03/16/17_08:43 07ecd3440 61474 snf.0.-1 P pqstate [ 0x7fabade7e000.. 0x7fabade80000) size 8 KiB 8192 (0x2000) 61474 snf.0.-1 P desc_ring [ 0x7fab9c824000.. 0x7fabac824000) size 256 MiB 268435456 (0x10000000) 61474 snf.0.-1 P data_ring [ 0x7fab94c14000.. 0x7fab9c824000) size 124 MiB 130088960 (0x7c10000) 61474 snf.0.-1 P pq_init: desc[seq=216,ev_idx=222869,cnt=105364809365] tcpdump: snf_ring_open_id(ring=-1) failed: Device or resource busy # Regards, Chris Chiaverini Cyber Security Operations Brookhaven National Laboratory Upton, New York 11973 On 06/20/2017 09:46 AM, Azoff, Justin S wrote: >> On Jun 20, 2017, at 9:27 AM, Chris Chiaverini wrote: >> >> It seems that bro 2.5.1 is not taking the SNF_DATARING_SIZE variable, no matter what I set it to. >> >> When at the defaults in the /etc/bro/node.cfg and with nothing set at the shell, it still reports it is set via "userset" instead of "default" like SNF_DESCRING_SIZE. > Can you do this quick test using tcpdump to verify the problem is with bro/broctl or something with the myricom driver/library? > > SNF_APP_ID=10 SNF_FLAGS=0x1 SNF_NUM_RINGS=8 SNF_DEBUG_MASK=3 SNF_DATARING_SIZE=4294967296 SNF_DESCRING_SIZE=1073741824 tcpdump -n -i snf0 -c 1 > > When I run that I get > > 23681 snf.0.-1 P (userset) SNF_PORTNUM = 0 > 23681 snf.0.-1 P (default) SNF_RING_ID = -1 (0xffffffff) > 23681 snf.0.-1 P (environ) SNF_NUM_RINGS = 8 (0x8) > 23681 snf.0.-1 P (default) SNF_RSS_FLAGS = 49 (0x31) > 23681 snf.0.-1 P (environ) SNF_DATARING_SIZE = 4294967296 (0x100000000) (4096.0 MiB) > 23681 snf.0.-1 P (environ) SNF_DESCRING_SIZE = 1073741824 (0x40000000) (1024.0 MiB) > 23681 snf.0.-1 P (userset) SNF_FLAGS = 1 (0x1) > 23681 snf.0.-1 P (environ) SNF_DEBUG_MASK = 3 (0x3) > 23681 snf.0.-1 P (default) SNF_DEBUG_FILENAME = stderr > 23681 snf.0.-1 P (environ) SNF_APP_ID = 10 (0xa) > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170620/a6cd9071/attachment-0001.html From cchiaverini at bnl.gov Tue Jun 20 08:01:03 2017 From: cchiaverini at bnl.gov (Chris Chiaverini) Date: Tue, 20 Jun 2017 11:01:03 -0400 Subject: [Bro] Bro node.cfg not setting Myricom Sniffer10G environment variables In-Reply-To: References: <30abafd5-fae4-f1d2-0df5-c62e82e777c4@bnl.gov> <2831AF5A-E4E7-4C8C-B4FF-9838A580FA22@illinois.edu> Message-ID: <51cec005-586b-26d8-42e8-6f1815dc9594@bnl.gov> I have a support case open with them in parallel. I will report this to them too. Maybe we'll get a fix in next minor release. Regards, Chris Chiaverini Cyber Security Operations Brookhaven National Laboratory Upton, New York 11973 On 06/20/2017 10:09 AM, Alejandro Carreno wrote: > I noticed this behavior as well a while back after upgrading SNF from > 3.0.10 to 3.0.11. Downgrading back to 3.0.10 would return the ring > sizes to the expected values. > > -Alex > > On Tue, Jun 20, 2017 at 6:47 AM Azoff, Justin S > wrote: > > > > On Jun 20, 2017, at 9:27 AM, Chris Chiaverini > > wrote: > > > > It seems that bro 2.5.1 is not taking the SNF_DATARING_SIZE > variable, no matter what I set it to. > > > > When at the defaults in the /etc/bro/node.cfg and with nothing > set at the shell, it still reports it is set via "userset" instead > of "default" like SNF_DESCRING_SIZE. > > Can you do this quick test using tcpdump to verify the problem is > with bro/broctl or something with the myricom driver/library? > > SNF_APP_ID=10 SNF_FLAGS=0x1 SNF_NUM_RINGS=8 SNF_DEBUG_MASK=3 > SNF_DATARING_SIZE=4294967296 SNF_DESCRING_SIZE=1073741824 tcpdump > -n -i snf0 -c 1 > > When I run that I get > > 23681 snf.0.-1 P (userset) SNF_PORTNUM = 0 > 23681 snf.0.-1 P (default) SNF_RING_ID = -1 (0xffffffff) > 23681 snf.0.-1 P (environ) SNF_NUM_RINGS = 8 (0x8) > 23681 snf.0.-1 P (default) SNF_RSS_FLAGS = 49 (0x31) > 23681 snf.0.-1 P (environ) SNF_DATARING_SIZE = 4294967296 > (0x100000000) (4096.0 MiB) > 23681 snf.0.-1 P (environ) SNF_DESCRING_SIZE = 1073741824 > (0x40000000) (1024.0 MiB) > 23681 snf.0.-1 P (userset) SNF_FLAGS = 1 (0x1) > 23681 snf.0.-1 P (environ) SNF_DEBUG_MASK = 3 (0x3) > 23681 snf.0.-1 P (default) SNF_DEBUG_FILENAME = stderr > 23681 snf.0.-1 P (environ) SNF_APP_ID = 10 (0xa) > > > > -- > - Justin Azoff > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170620/1408d0e9/attachment.html From gary.w.weasel2.civ at mail.mil Tue Jun 20 12:14:03 2017 From: gary.w.weasel2.civ at mail.mil (Weasel, Gary W Jr CIV DISA RE (US)) Date: Tue, 20 Jun 2017 19:14:03 +0000 Subject: [Bro] Digging through Source Code Message-ID: <0C34D9CA9B9DBB45B1C51871C177B4B286AAEF06@UMECHPA68.easf.csd.disa.mil> All, I've been digging through the Bro source code, and there's been something that's mystifying me for a while now. type Array = record { array_meta: ASN1EncodingMeta; data: ASN1Encoding[]; }; As from https://github.com/bro/bro/blob/57da2d091b30aad52d52fce8018feeb2cdf8ff1f/src/analyzer/protocol/asn1/asn1.pac I have no clue what "record" is in this context. I suspect it has other attributes that are being inherited, but I haven't found anything to indicate what this is. Does anyone have any insight into this? Thanks, - Gary From jazoff at illinois.edu Tue Jun 20 12:28:22 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 20 Jun 2017 19:28:22 +0000 Subject: [Bro] Digging through Source Code In-Reply-To: <0C34D9CA9B9DBB45B1C51871C177B4B286AAEF06@UMECHPA68.easf.csd.disa.mil> References: <0C34D9CA9B9DBB45B1C51871C177B4B286AAEF06@UMECHPA68.easf.csd.disa.mil> Message-ID: > On Jun 20, 2017, at 3:14 PM, Weasel, Gary W Jr CIV DISA RE (US) wrote: > > All, > > I've been digging through the Bro source code, and there's been something that's mystifying me for a while now. > > type Array = record { > array_meta: ASN1EncodingMeta; > data: ASN1Encoding[]; > }; > > As from https://github.com/bro/bro/blob/57da2d091b30aad52d52fce8018feeb2cdf8ff1f/src/analyzer/protocol/asn1/asn1.pac > > I have no clue what "record" is in this context. I suspect it has other attributes that are being inherited, but I haven't found anything to indicate what this is. Does anyone have any insight into this? > > Thanks, > - Gary Does this help? https://www.bro.org/sphinx/script-reference/types.html#type-record -- - Justin Azoff From gary.w.weasel2.civ at mail.mil Tue Jun 20 13:42:35 2017 From: gary.w.weasel2.civ at mail.mil (Weasel, Gary W Jr CIV DISA RE (US)) Date: Tue, 20 Jun 2017 20:42:35 +0000 Subject: [Bro] Digging through Source Code Message-ID: <0C34D9CA9B9DBB45B1C51871C177B4B286AAEF1D@UMECHPA68.easf.csd.disa.mil> Yes, but there's something that's still stumping me. Looking at line 70 from https://github.com/bro/bro/blob/master/src/analyzer/protocol/krb/krb-analyzer.pac case 8: if ( element->data()->etype()->data()->size() ) rv->Assign(11, proc_cipher_list(element->data()->etype())); Following the breadcrumb trail in the if statement here... element is type KRB_REQ_Arg (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/krb/krb-protocol.pac) -> data is type KRB_REQ_Arg_Data (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/krb/krb-protocol.pac) -> etype is type Array (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/asn1/asn1.pac) -> data is type ASN1Encoding (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/asn1/asn1.pac) -> size is type ? Following this line of thought, I'm a little confused by what "size()" is supposed to mean here, since it's not an attribute. I can infer that it's simply returning the size of the record, but I don't have any information as to how or where that would be defined. I've also tried looking through the source of BinPAC (https://www.bro.org/sphinx/components/binpac/README.html) but have come up empty so far. I have a sample of kerberos pcap that populates the msg$pa_data$encryption_type vector (from event krb_tgs_request), so I know that the aforementioned if statement is returning true - - but the other two vectors "host_addrs" and "additional"tickets" (that from documentation seem to imply they're parallel with the encryption_type vector) come up as . This made me question that maybe there was something wrong with the code that was causing it to miss the host_addr and ticket data, I clearly find this data in my pcap sample under padata. This is my current theory anyway, and wanted to see if I'm making a bad assumption somewhere or if someone can shed light on what's going on here. -----Original Message----- From: Azoff, Justin S [mailto:jazoff at illinois.edu] Sent: Tuesday, June 20, 2017 3:28 PM To: Weasel, Gary W Jr CIV DISA RE (US) Cc: bro at bro.org Subject: Re: [Bro] Digging through Source Code All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. ---- > On Jun 20, 2017, at 3:14 PM, Weasel, Gary W Jr CIV DISA RE (US) wrote: > > All, > > I've been digging through the Bro source code, and there's been something that's mystifying me for a while now. > > type Array = record { > array_meta: ASN1EncodingMeta; > data: ASN1Encoding[]; > }; > > As from https://github.com/bro/bro/blob/57da2d091b30aad52d52fce8018feeb2cdf8ff1f/src/analyzer/protocol/asn1/asn1.pac > > I have no clue what "record" is in this context. I suspect it has other attributes that are being inherited, but I haven't found anything to indicate what this is. Does anyone have any insight into this? > > Thanks, > - Gary Does this help? https://www.bro.org/sphinx/script-reference/types.html#type-record -- - Justin Azoff From daniel.guerra69 at gmail.com Tue Jun 20 14:43:36 2017 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Tue, 20 Jun 2017 23:43:36 +0200 Subject: [Bro] Digging through Source Code In-Reply-To: <0C34D9CA9B9DBB45B1C51871C177B4B286AAEF1D@UMECHPA68.easf.csd.disa.mil> References: <0C34D9CA9B9DBB45B1C51871C177B4B286AAEF1D@UMECHPA68.easf.csd.disa.mil> Message-ID: <43326d22-6731-ce2d-1d5b-28b537852fc1@gmail.com> Talking about ASN1. Would bro be able to read ETSI standard files ? Op 20/06/2017 om 22:42 schreef Weasel, Gary W Jr CIV DISA RE (US): > Yes, but there's something that's still stumping me. > > Looking at line 70 from https://github.com/bro/bro/blob/master/src/analyzer/protocol/krb/krb-analyzer.pac > > case 8: > if ( element->data()->etype()->data()->size() ) > rv->Assign(11, proc_cipher_list(element->data()->etype())); > > Following the breadcrumb trail in the if statement here... > > element is type KRB_REQ_Arg (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/krb/krb-protocol.pac) > -> data is type KRB_REQ_Arg_Data (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/krb/krb-protocol.pac) > -> etype is type Array (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/asn1/asn1.pac) > -> data is type ASN1Encoding (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/asn1/asn1.pac) > -> size is type ? > > Following this line of thought, I'm a little confused by what "size()" is supposed to mean here, since it's not an attribute. I can infer that it's simply returning the size of the record, but I don't have any information as to how or where that would be defined. I've also tried looking through the source of BinPAC (https://www.bro.org/sphinx/components/binpac/README.html) but have come up empty so far. > > I have a sample of kerberos pcap that populates the msg$pa_data$encryption_type vector (from event krb_tgs_request), so I know that the aforementioned if statement is returning true - - but the other two vectors "host_addrs" and "additional"tickets" (that from documentation seem to imply they're parallel with the encryption_type vector) come up as . > > This made me question that maybe there was something wrong with the code that was causing it to miss the host_addr and ticket data, I clearly find this data in my pcap sample under padata. This is my current theory anyway, and wanted to see if I'm making a bad assumption somewhere or if someone can shed light on what's going on here. > > > -----Original Message----- > From: Azoff, Justin S [mailto:jazoff at illinois.edu] > Sent: Tuesday, June 20, 2017 3:28 PM > To: Weasel, Gary W Jr CIV DISA RE (US) > Cc: bro at bro.org > Subject: Re: [Bro] Digging through Source Code > > All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. > > > > > ---- > > >> On Jun 20, 2017, at 3:14 PM, Weasel, Gary W Jr CIV DISA RE (US) wrote: >> >> All, >> >> I've been digging through the Bro source code, and there's been something that's mystifying me for a while now. >> >> type Array = record { >> array_meta: ASN1EncodingMeta; >> data: ASN1Encoding[]; >> }; >> >> As from https://github.com/bro/bro/blob/57da2d091b30aad52d52fce8018feeb2cdf8ff1f/src/analyzer/protocol/asn1/asn1.pac >> >> I have no clue what "record" is in this context. I suspect it has other attributes that are being inherited, but I haven't found anything to indicate what this is. Does anyone have any insight into this? >> >> Thanks, >> - Gary > Does this help? > > https://www.bro.org/sphinx/script-reference/types.html#type-record > > > -- > - Justin Azoff > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From egoant495 at gmail.com Wed Jun 21 05:45:32 2017 From: egoant495 at gmail.com (Anton Egorov) Date: Wed, 21 Jun 2017 15:45:32 +0300 Subject: [Bro] Bro doesn't detect SSH version in local network Message-ID: Hi, Bro somehow doesn't detect the SSH client version when listening on a local network interface. The machine with installed Bro has two network interfaces. One is in the company common network and the other is in the small test network. Small network has address in a 192.168.0.0/16 space. Other machines in the small network has the two interfaces for intranet and test network as well. When ssh connection is established from test machine and Bro is listening on eth0 interface the ssh client version gets detected. But if ssh connection targets the eth1 interface which Bro is listening nothing gets detected. Here are the interfaces on machine with installed bro: # ifconfig eth0 Link encap:Ethernet HWaddr 00:50:56:99:76:5f inet addr:10.31.10.190 Bcast:10.31.10.255 Mask:255.255.255.0 UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:346628470 errors:0 dropped:1417 overruns:0 frame:0 TX packets:327889 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:104910129783 (97.7 GiB) TX bytes:77220087 (73.6 MiB) eth1 Link encap:Ethernet HWaddr 00:50:56:99:74:81 inet addr:192.168.99.90 Bcast:192.168.99.255 Mask:255.255.255.0 UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:1648090595 errors:0 dropped:20 overruns:0 frame:0 TX packets:645 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:98885922776 (92.0 GiB) TX bytes:93928 (91.7 KiB) Bro is started like that # bro -i eth0 os-app-detect.bro local or for a local interface # bro -i eth1 os-app-detect.bro local The output that shows in the first case is: OpenSSH OpenSSH_6.0p1 Debian-4+deb7u3 The connections from a test machine runs like that On eth0 interface (Bro detects it) # ssh root at 10.31.10.190 On eht1 interface (Bro doesn't detect it) # ssh root at 192.168.99.90 The .bro script for printing SSH client version: ----- # cat os-app-detect.bro global os_detect: event(host: addr, os_name: string); global app_detect: event(host: addr, app_name: string); event OS_version_found(c: connection, host_addr: addr, OS: OS_version) { local rec_value = OS$genre + " " + OS$detail; print rec_value; event os_detect(host_addr, rec_value); } event Software::log_software(rec: Software::Info) { local app_name_ver = rec$name + " " + rec$unparsed_version; print app_name_ver; event app_detect(rec$host, app_name_ver); } ----- Info about system # uname -a Linux evm190 4.2.0-23-generic # /usr/local/bro/bin/bro -v /usr/local/bro/bin/bro version 2.4.1 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170621/3ff8b2e3/attachment-0001.html From daniel.manzo at bayer.com Wed Jun 21 06:20:28 2017 From: daniel.manzo at bayer.com (Daniel Manzo) Date: Wed, 21 Jun 2017 13:20:28 +0000 Subject: [Bro] Network tap issues Message-ID: <94e07e566040490e9d33be0b995ebd5f@moxde9.na.bayer.cnb> Hi all, I have Bro 2.5 configured on a RHEL 7.3 server and have a network tap question, which I know isn't totally Bro related, but I figured the Bro community would be able to advise. The tap I have is a passive fiber tap (OM3/4, 850mm, 50/50) enabled for up to 10Gb throughput. The connection in port A is coming from Level 3 internet and the connection in port B is going to a network switch. The monitor port is connected to my Bro server. The problem is that I am seeing no traffic at all coming from the monitor, and the light on the server NIC doesn't even light up. However, I am still able to access the internet from my server, despite receiving no traffic from the monitor. Basically the connection from A to B works, but the monitor is not mirroring traffic. We have tested the tap before in other areas of our network, and it was working, so I'm not sure why it is not working in this location. Any and all help is appreciated! Thank you, Dan Manzo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170621/8a824c57/attachment.html From daniel.manzo at bayer.com Wed Jun 21 07:07:18 2017 From: daniel.manzo at bayer.com (Daniel Manzo) Date: Wed, 21 Jun 2017 14:07:18 +0000 Subject: [Bro] Network tap issues In-Reply-To: References: <94e07e566040490e9d33be0b995ebd5f@moxde9.na.bayer.cnb> Message-ID: Thanks for the response! Unfortunately, we have tried that, but still no luck. I?m not sure what else could be wrong. From: Mark Buchanan [mailto:mabuchan at gmail.com] Sent: Wednesday, June 21, 2017 9:46 AM To: Daniel Manzo Subject: Re: [Bro] Network tap issues Flip both TX and RX around. The tap is in "backwards" meaning the light is not flowing the right direction to hit the optical splitter and get to your sensor. It as acting more as a "combiner", which could be bad if someone pushes light from your tap to the circuit. -- Mark Buchanan On Jun 21, 2017, at 07:20, Daniel Manzo > wrote: Hi all, I have Bro 2.5 configured on a RHEL 7.3 server and have a network tap question, which I know isn?t totally Bro related, but I figured the Bro community would be able to advise. The tap I have is a passive fiber tap (OM3/4, 850mm, 50/50) enabled for up to 10Gb throughput. The connection in port A is coming from Level 3 internet and the connection in port B is going to a network switch. The monitor port is connected to my Bro server. The problem is that I am seeing no traffic at all coming from the monitor, and the light on the server NIC doesn?t even light up. However, I am still able to access the internet from my server, despite receiving no traffic from the monitor. Basically the connection from A to B works, but the monitor is not mirroring traffic. We have tested the tap before in other areas of our network, and it was working, so I?m not sure why it is not working in this location. Any and all help is appreciated! Thank you, Dan Manzo _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170621/6facd7bc/attachment.html From jazoff at illinois.edu Wed Jun 21 07:21:52 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 21 Jun 2017 14:21:52 +0000 Subject: [Bro] Bro doesn't detect SSH version in local network In-Reply-To: References: Message-ID: > On Jun 21, 2017, at 8:45 AM, Anton Egorov wrote: > > Hi, > > Bro somehow doesn't detect the SSH client version when listening on a local network interface. see https://www.bro.org/documentation/faq.html#why-isn-t-bro-producing-the-logs-i-expect-a-note-about-checksums -- - Justin Azoff From jazoff at illinois.edu Wed Jun 21 07:26:49 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 21 Jun 2017 14:26:49 +0000 Subject: [Bro] Network tap issues In-Reply-To: <94e07e566040490e9d33be0b995ebd5f@moxde9.na.bayer.cnb> References: <94e07e566040490e9d33be0b995ebd5f@moxde9.na.bayer.cnb> Message-ID: <844D3984-C194-4C60-8B53-451C23FE8E06@illinois.edu> > On Jun 21, 2017, at 9:20 AM, Daniel Manzo wrote: > > The connection in port A is coming from Level 3 internet and the connection in port B is going to a network switch. The monitor port I'm confused on what you mean by "the monitor port". A passive fiber tap has two rx/tx input ports and two tx only output ports, one for each direction. The two outputs need to be connected to the rx ports on two separate NICs, you can't just connect them to a single nic, as you'd be connecting one of the tx ports from the tap to a tx port on the NIC. -- - Justin Azoff From egoant495 at gmail.com Wed Jun 21 07:37:12 2017 From: egoant495 at gmail.com (Anton Egorov) Date: Wed, 21 Jun 2017 17:37:12 +0300 Subject: [Bro] Bro doesn't detect SSH version in local network In-Reply-To: References: Message-ID: The offloading is disabled on both NIC's and the -C option also doesn't do the trick. While reading pcap of a saved ssh traffic bro outputs a warning: # /usr/local/bro/bin/bro -C -r /root/eth1-ssh.cap /usr/local/bro/share/bro/pluton/os-app-detect.bro local UNKNOWN 1497975118.771257 warning: Stream SOrfileNrXm8iGmlR6 is already queued for removal. Ignoring remove. while on a pcap from the other interface: # /usr/local/bro/bin/bro -C -r /root/eth0-ssh.cap /usr/local/bro/share/bro/pluton/os-app-detect.bro local UNKNOWN OpenSSH OpenSSH_6.0p1 Debian-4+deb7u3 Thank you 2017-06-21 17:21 GMT+03:00 Azoff, Justin S : > > > On Jun 21, 2017, at 8:45 AM, Anton Egorov wrote: > > > > Hi, > > > > Bro somehow doesn't detect the SSH client version when listening on a > local network interface. > > see > > https://www.bro.org/documentation/faq.html#why- > isn-t-bro-producing-the-logs-i-expect-a-note-about-checksums > > -- > - Justin Azoff > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170621/df4b20be/attachment-0001.html From jazoff at illinois.edu Wed Jun 21 07:44:58 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 21 Jun 2017 14:44:58 +0000 Subject: [Bro] Bro doesn't detect SSH version in local network In-Reply-To: References: Message-ID: <3694DCA6-36DE-47A2-9549-54615A7FD110@illinois.edu> > On Jun 21, 2017, at 10:37 AM, Anton Egorov wrote: > > The offloading is disabled on both NIC's and the -C option also doesn't do the trick. > > While reading pcap of a saved ssh traffic bro outputs a warning: > > # /usr/local/bro/bin/bro -C -r /root/eth1-ssh.cap /usr/local/bro/share/bro/pluton/os-app-detect.bro local > UNKNOWN > 1497975118.771257 warning: Stream SOrfileNrXm8iGmlR6 is already queued for removal. Ignoring remove. > > while on a pcap from the other interface: > > # /usr/local/bro/bin/bro -C -r /root/eth0-ssh.cap /usr/local/bro/share/bro/pluton/os-app-detect.bro local > UNKNOWN > OpenSSH OpenSSH_6.0p1 Debian-4+deb7u3 What does the full conn.log entry show for the SSH connection in these two cases? Can you upgrade bro to 2.5 or the 2.5.1 beta? 2.4.1 is almost a year old at this point. -- - Justin Azoff From daniel.manzo at bayer.com Wed Jun 21 08:09:09 2017 From: daniel.manzo at bayer.com (Daniel Manzo) Date: Wed, 21 Jun 2017 15:09:09 +0000 Subject: [Bro] Network tap issues In-Reply-To: <844D3984-C194-4C60-8B53-451C23FE8E06@illinois.edu> References: <94e07e566040490e9d33be0b995ebd5f@moxde9.na.bayer.cnb> <844D3984-C194-4C60-8B53-451C23FE8E06@illinois.edu> Message-ID: <641a63a081954eb0b71341bc26589334@moxde9.na.bayer.cnb> Sorry for the confusion, I am referring to the two TX output ports as a duplex "monitor" port. We have a duplex cable going from the two TX output ports to the NIC, so perhaps we can split the cable and plug each individual fiber into its own SFP. However, shouldn't I be seeing some sort of traffic on the NIC? Since both fibers of the duplex are transmitting, either one of them has to be in the rx port on the NIC, so I would then be seeing some packets, correct? If you think splitting the duplex cable from the TX output ports would fix the problem, then I will do so. -----Original Message----- From: Azoff, Justin S [mailto:jazoff at illinois.edu] Sent: Wednesday, June 21, 2017 10:27 AM To: Daniel Manzo Cc: Bro-IDS Subject: Re: [Bro] Network tap issues > On Jun 21, 2017, at 9:20 AM, Daniel Manzo wrote: > > The connection in port A is coming from Level 3 internet and the connection in port B is going to a network switch. The monitor port I'm confused on what you mean by "the monitor port". A passive fiber tap has two rx/tx input ports and two tx only output ports, one for each direction. The two outputs need to be connected to the rx ports on two separate NICs, you can't just connect them to a single nic, as you'd be connecting one of the tx ports from the tap to a tx port on the NIC. -- - Justin Azoff From kevin at branchnetconsulting.com Wed Jun 21 08:29:18 2017 From: kevin at branchnetconsulting.com (Kevin Branch) Date: Wed, 21 Jun 2017 11:29:18 -0400 Subject: [Bro] Bro 2.5 appears to be ignoring redefs of Pcap::snaplen Message-ID: For a long time I have used "redef Pcap::snaplen = 1600;" in local.bro to make Bro drop its default snaplen from 8192 to 1600. This is helpful for conserving memory when using Bro in conjunction with PF_RING and a high number of ring slots. Today I just noticed that while Bro does not complain about "redef Pcap::snaplen = 1600;" when I run a "broctl check", that Bro appears to be ignoring the redef. All my Bro instances are actually using a snaplen of 8192. I use Bro on the latest Security Onion Ubuntu 14.04 platform, and have observed this problem with both PF_RING 6.4.1 (SO stable) and PF_RING 6.6.0 (SO test). The "Bucket Len" in the below PF_RING status file corresponds to the snaplen of the app that allocated the ring. root at nsm.xyz.org:~# cat /proc/net/pf_ring/15028-dmz.9 Bound Device(s) : dmz Active : 1 Breed : Standard Appl. Name : bro-dmz Socket Mode : RX+TX Capture Direction : RX+TX Sampling Rate : 1 IP Defragment : No BPF Filtering : Enabled Sw Filt Hash Rules : 0 Sw Filt WC Rules : 0 Hw Filt Rules : 0 Sw Filt Hash Match : 0 Sw Filt Hash Miss : 0 Poll Pkt Watermark : 1 Num Poll Calls : 345386919 Channel Id Mask : 0xFFFFFFFFFFFFFFFF Cluster Id : 21 Slot Version : 16 [6.4.1] Min Num Slots : 128000 Bucket Len : 8192 Slot Len : 8248 [bucket+header] Tot Memory : 1055756288 Tot Packets : 1966471960 Tot Pkt Lost : 3 Tot Insert : 1966471957 Tot Read : 1966471957 Insert Offset : 809944608 Remove Offset : 809944608 Num Free Slots : 128000 TX: Send Ok : 0 TX: Send Errors : 0 Reflect: Fwd Ok : 0 Reflect: Fwd Errors: 0 Please advise me about how to successfully change the snaplen used by Bro 2.5 at this time, Can anyone reproduce this problem? I don't know if this issue applies across the board or only comes up with PF_RING. Let me know if there is anything I can do to help test this issue. Thanks! Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170621/4646ca6c/attachment.html From bill.de.ping at gmail.com Wed Jun 21 09:29:56 2017 From: bill.de.ping at gmail.com (william de ping) Date: Wed, 21 Jun 2017 19:29:56 +0300 Subject: [Bro] - http$host diff between bro and broctl Message-ID: Hi all, Scenario 1 : bro instance on my local interface + browsing to www.bbc.com Scenario 2 : bro cluster with a single Worker on my local interface + browsing to www.cnn.com in http.log, on the 1st scenario, the host field is initialized with www.bbc.com on the 2nd scenario, the host field is NOT initialized I'm running bro 2.5 Is there any explanation for the diff ? thank you B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170621/7973e635/attachment.html From hovsep.sanjay.levi at gmail.com Wed Jun 21 09:42:50 2017 From: hovsep.sanjay.levi at gmail.com (Hovsep Levi) Date: Wed, 21 Jun 2017 16:42:50 +0000 Subject: [Bro] Network tap issues In-Reply-To: <94e07e566040490e9d33be0b995ebd5f@moxde9.na.bayer.cnb> References: <94e07e566040490e9d33be0b995ebd5f@moxde9.na.bayer.cnb> Message-ID: I've dealt with this before although I don't really understand it technically. Some sort of layer1-ish protocol.. the only way I can explain it is something like Cisco's UDLD.. the NIC detects a fault in the circuit due to a half connected state and shuts down the laser/LED for safety reasons. You might find some obscure low-level setting for the NIC to force it into a special monitor mode similar to how wifi monitor mode works. What happens if you force the interface up and run tcpdump ? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170621/2df51d64/attachment.html From daniel.manzo at bayer.com Wed Jun 21 10:54:25 2017 From: daniel.manzo at bayer.com (Daniel Manzo) Date: Wed, 21 Jun 2017 17:54:25 +0000 Subject: [Bro] Network tap issues In-Reply-To: References: <94e07e566040490e9d33be0b995ebd5f@moxde9.na.bayer.cnb> Message-ID: <9d37a3af308a483983d3fb609845996d@moxde9.na.bayer.cnb> Based on what I?ve seen, I think you might be right about the NIC detecting a fault due to a half connected state. I forced the interface up and put it in ?promiscuous? mode, then ran tcp dump. Unfortunately, it reported back with no packets captured ? From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Hovsep Levi Sent: Wednesday, June 21, 2017 12:43 PM To: Bro-IDS Subject: Re: [Bro] Network tap issues I've dealt with this before although I don't really understand it technically. Some sort of layer1-ish protocol.. the only way I can explain it is something like Cisco's UDLD.. the NIC detects a fault in the circuit due to a half connected state and shuts down the laser/LED for safety reasons. You might find some obscure low-level setting for the NIC to force it into a special monitor mode similar to how wifi monitor mode works. What happens if you force the interface up and run tcpdump ? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170621/6e9a588b/attachment-0001.html From hovsep.sanjay.levi at gmail.com Wed Jun 21 11:43:06 2017 From: hovsep.sanjay.levi at gmail.com (Hovsep Levi) Date: Wed, 21 Jun 2017 18:43:06 +0000 Subject: [Bro] Network tap issues In-Reply-To: <9d37a3af308a483983d3fb609845996d@moxde9.na.bayer.cnb> References: <94e07e566040490e9d33be0b995ebd5f@moxde9.na.bayer.cnb> <9d37a3af308a483983d3fb609845996d@moxde9.na.bayer.cnb> Message-ID: I don't think dedicated capture cards have this problem. What card/driver/sfp is it ? There might be a setting with ethtool you can toggle, try disabling auto-negotiation and changing the duplex settings. Then again, I think the specs for 10G claim autoneg is required so stock Linux drivers might not allow this. Seems your options are buy a dedicated capture card like Myricom or hack the driver. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170621/7b7e27a5/attachment.html From hovsep.sanjay.levi at gmail.com Wed Jun 21 11:49:03 2017 From: hovsep.sanjay.levi at gmail.com (Hovsep Levi) Date: Wed, 21 Jun 2017 18:49:03 +0000 Subject: [Bro] Network tap issues In-Reply-To: References: <94e07e566040490e9d33be0b995ebd5f@moxde9.na.bayer.cnb> <9d37a3af308a483983d3fb609845996d@moxde9.na.bayer.cnb> Message-ID: > > > Seems your options are buy a dedicated capture card like Myricom or hack > the driver. > > Or run the tap connections to a 10G switch that can do magic tap aggregation like SPAN. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170621/8d2bfc8b/attachment.html From ssakai at sdsc.edu Wed Jun 21 12:20:12 2017 From: ssakai at sdsc.edu (Scott Sakai) Date: Wed, 21 Jun 2017 12:20:12 -0700 Subject: [Bro] Network tap issues In-Reply-To: References: <94e07e566040490e9d33be0b995ebd5f@moxde9.na.bayer.cnb> Message-ID: <3e7d52aa-ac6b-1cac-bf29-dd8484897fc5@sdsc.edu> Hi Mark, As others have mentioned, the connection from the output of your tap to the capture nic needs some attention. Unlike a switch port, both sides of the duplex output port are outputs (light comes out). If you plug this into a nic with a duplex fiber, you'll blast light into the internet <-> switch link, which is definitely not going to do you any favors. You'll have to split the capture side of the fiber pair, and plug the single fiber into the RX port on the. For now, leave the other end dangling until you decide how to aggregate the two; this is just for testing. A reminder to never look into the end of a fiber, or into a port, even if you think it's off or shut down. You can do some permanent damage to your retinas, especially with LR optics, which use an invisible laser. Been there, done that, still got the scarring. These days, I use the camera on my cell phone, which has no direct optical path to the screen, plus it picks up near infra-red wavelengths, used in LR optics. Not that I suggest using this technique; the proper tool for light-path diagnostics is a light meter. With that in mind, do the optics in the capture nic match the optics in the switch behind the tap? In most cases, an SR optic won't respond to LR light and vice-versa. The link led will come on if the interface is up (ifconfig up) and the RX side receives properly-coded light of sufficient brightness. Thus, assuming the interface is up and the light-path is otherwise good, you might have a mismatch, or a bad optic in the capture nic. Good luck! On 06/21/2017 07:07 AM, Daniel Manzo wrote: > Thanks for the response! Unfortunately, we have tried that, but still no > luck. I?m not sure what else could be wrong. > > > > *From:*Mark Buchanan [mailto:mabuchan at gmail.com] > *Sent:* Wednesday, June 21, 2017 9:46 AM > *To:* Daniel Manzo > *Subject:* Re: [Bro] Network tap issues > > > > Flip both TX and RX around. The tap is in "backwards" meaning the light > is not flowing the right direction to hit the optical splitter and get to > your sensor. It as acting more as a "combiner", which could be bad if > someone pushes light from your tap to the circuit. > > -- > > Mark Buchanan > > > On Jun 21, 2017, at 07:20, Daniel Manzo > wrote: > > Hi all, > > > > I have Bro 2.5 configured on a RHEL 7.3 server and have a network tap > question, which I know isn?t totally Bro related, but I figured the Bro > community would be able to advise. The tap I have is a passive fiber > tap (OM3/4, 850mm, 50/50) enabled for up to 10Gb throughput. The > connection in port A is coming from Level 3 internet and the connection > in port B is going to a network switch. The monitor port is connected > to my Bro server. The problem is that I am seeing no traffic at all > coming from the monitor, and the light on the server NIC doesn?t even > light up. However, I am still able to access the internet from my > server, despite receiving no traffic from the monitor. Basically the > connection from A to B works, but the monitor is not mirroring traffic. > We have tested the tap before in other areas of our network, and it was > working, so I?m not sure why it is not working in this location. Any > and all help is appreciated! > > > > Thank you, > > Dan Manzo > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Scott Sakai Security Analyst San Diego Supercomputer Center ssakai at sdsc.edu +1-858-822-0851 From bmixonb1 at cs.unm.edu Wed Jun 21 12:51:26 2017 From: bmixonb1 at cs.unm.edu (Ben Mixon-Baca) Date: Wed, 21 Jun 2017 13:51:26 -0600 Subject: [Bro] SSL/TLS decryption Message-ID: <1c0846f3-fc75-a372-6cbe-388e06d9f3cc@cs.unm.edu> Hello, Is bro capable of decrypting SSL/TLS flows if I provide it a private key for the flow? -- Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 473 bytes Desc: OpenPGP digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170621/ee6d1648/attachment.bin From oelnaggar04 at gmail.com Wed Jun 21 13:55:11 2017 From: oelnaggar04 at gmail.com (Osama Elnaggar) Date: Wed, 21 Jun 2017 16:55:11 -0400 Subject: [Bro] SSL/TLS decryption In-Reply-To: <1c0846f3-fc75-a372-6cbe-388e06d9f3cc@cs.unm.edu> References: <1c0846f3-fc75-a372-6cbe-388e06d9f3cc@cs.unm.edu> Message-ID: No. Here is the last thread that discussed this back in 2015 - http://mailman.icsi.berkeley.edu/pipermail/bro/2015-June/008568.html However, some people were able to successfully extend Bro to do this but none of these implementations was released. -- Osama Elnaggar On June 22, 2017 at 5:56:21 AM, Ben Mixon-Baca (bmixonb1 at cs.unm.edu) wrote: Hello, Is bro capable of decrypting SSL/TLS flows if I provide it a private key for the flow? -- Ben _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170621/c54e97cb/attachment.html From jazoff at illinois.edu Wed Jun 21 14:17:05 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 21 Jun 2017 21:17:05 +0000 Subject: [Bro] - http$host diff between bro and broctl In-Reply-To: References: Message-ID: <05B2AB4A-A639-40D2-AB5D-67421D122EC0@illinois.edu> > On Jun 21, 2017, at 12:29 PM, william de ping wrote: > > Hi all, > > Scenario 1 : bro instance on my local interface + browsing to www.bbc.com > Scenario 2 : bro cluster with a single Worker on my local interface + browsing to www.cnn.com > > in http.log, > on the 1st scenario, the host field is initialized with www.bbc.com > on the 2nd scenario, the host field is NOT initialized > > I'm running bro 2.5 > > Is there any explanation for the diff ? You're probably starting bro differently in the two cases. https://www.bro.org/documentation/faq.html#why-isn-t-bro-producing-the-logs-i-expect-a-note-about-checksums -- - Justin Azoff From cchiaverini at bnl.gov Wed Jun 21 15:31:50 2017 From: cchiaverini at bnl.gov (Chris Chiaverini) Date: Wed, 21 Jun 2017 18:31:50 -0400 Subject: [Bro] Bro node.cfg not setting Myricom Sniffer10G environment variables In-Reply-To: <51cec005-586b-26d8-42e8-6f1815dc9594@bnl.gov> References: <30abafd5-fae4-f1d2-0df5-c62e82e777c4@bnl.gov> <2831AF5A-E4E7-4C8C-B4FF-9838A580FA22@illinois.edu> <51cec005-586b-26d8-42e8-6f1815dc9594@bnl.gov> Message-ID: <58b4bc70-bb8a-685e-b7cc-a28a327b3b5e@bnl.gov> Alex, Thank you for this. I confirmed on my end too... rolled back to 3.0.10 and it worked. I will let you know what Myricom comes up with, if they will fix in next release. Regards, Chris Chiaverini Cyber Security Operations Brookhaven National Laboratory Upton, New York 11973 On 06/20/2017 11:01 AM, Chris Chiaverini wrote: > > I have a support case open with them in parallel. I will report this > to them too. Maybe we'll get a fix in next minor release. > > Regards, > > Chris Chiaverini > Cyber Security Operations > Brookhaven National Laboratory > Upton, New York 11973 > On 06/20/2017 10:09 AM, Alejandro Carreno wrote: >> I noticed this behavior as well a while back after upgrading SNF from >> 3.0.10 to 3.0.11. Downgrading back to 3.0.10 would return the ring >> sizes to the expected values. >> >> -Alex >> >> On Tue, Jun 20, 2017 at 6:47 AM Azoff, Justin S > > wrote: >> >> >> > On Jun 20, 2017, at 9:27 AM, Chris Chiaverini >> > wrote: >> > >> > It seems that bro 2.5.1 is not taking the SNF_DATARING_SIZE >> variable, no matter what I set it to. >> > >> > When at the defaults in the /etc/bro/node.cfg and with nothing >> set at the shell, it still reports it is set via "userset" >> instead of "default" like SNF_DESCRING_SIZE. >> >> Can you do this quick test using tcpdump to verify the problem is >> with bro/broctl or something with the myricom driver/library? >> >> SNF_APP_ID=10 SNF_FLAGS=0x1 SNF_NUM_RINGS=8 SNF_DEBUG_MASK=3 >> SNF_DATARING_SIZE=4294967296 SNF_DESCRING_SIZE=1073741824 tcpdump >> -n -i snf0 -c 1 >> >> When I run that I get >> >> 23681 snf.0.-1 P (userset) SNF_PORTNUM = 0 >> 23681 snf.0.-1 P (default) SNF_RING_ID = -1 (0xffffffff) >> 23681 snf.0.-1 P (environ) SNF_NUM_RINGS = 8 (0x8) >> 23681 snf.0.-1 P (default) SNF_RSS_FLAGS = 49 (0x31) >> 23681 snf.0.-1 P (environ) SNF_DATARING_SIZE = 4294967296 >> (0x100000000) (4096.0 MiB) >> 23681 snf.0.-1 P (environ) SNF_DESCRING_SIZE = 1073741824 >> (0x40000000) (1024.0 MiB) >> 23681 snf.0.-1 P (userset) SNF_FLAGS = 1 (0x1) >> 23681 snf.0.-1 P (environ) SNF_DEBUG_MASK = 3 (0x3) >> 23681 snf.0.-1 P (default) SNF_DEBUG_FILENAME = stderr >> 23681 snf.0.-1 P (environ) SNF_APP_ID = 10 (0xa) >> >> >> >> -- >> - Justin Azoff >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170621/af9afded/attachment-0001.html From asharma at lbl.gov Wed Jun 21 16:59:08 2017 From: asharma at lbl.gov (Aashish Sharma) Date: Wed, 21 Jun 2017 16:59:08 -0700 Subject: [Bro] Bro node.cfg not setting Myricom Sniffer10G environment variables In-Reply-To: <58b4bc70-bb8a-685e-b7cc-a28a327b3b5e@bnl.gov> References: <30abafd5-fae4-f1d2-0df5-c62e82e777c4@bnl.gov> <2831AF5A-E4E7-4C8C-B4FF-9838A580FA22@illinois.edu> <51cec005-586b-26d8-42e8-6f1815dc9594@bnl.gov> <58b4bc70-bb8a-685e-b7cc-a28a327b3b5e@bnl.gov> Message-ID: <20170621235906.GS75741@mac-822.local> Doh! I just upgraded the myricom drivers to 3.0.11 today only :) Aashish On Wed, Jun 21, 2017 at 06:31:50PM -0400, Chris Chiaverini wrote: > Alex, > > Thank you for this. I confirmed on my end too... rolled back to 3.0.10 and > it worked. I will let you know what Myricom comes up with, if they will > fix in next release. > > > Regards, > > Chris Chiaverini > Cyber Security Operations > Brookhaven National Laboratory > Upton, New York 11973 > > On 06/20/2017 11:01 AM, Chris Chiaverini wrote: > > > >I have a support case open with them in parallel. I will report this to > >them too. Maybe we'll get a fix in next minor release. > > > >Regards, > > > >Chris Chiaverini > >Cyber Security Operations > >Brookhaven National Laboratory > >Upton, New York 11973 > >On 06/20/2017 10:09 AM, Alejandro Carreno wrote: > >>I noticed this behavior as well a while back after upgrading SNF from > >>3.0.10 to 3.0.11. Downgrading back to 3.0.10 would return the ring sizes > >>to the expected values. > >> > >>-Alex > >> > >>On Tue, Jun 20, 2017 at 6:47 AM Azoff, Justin S >>> wrote: > >> > >> > >> > On Jun 20, 2017, at 9:27 AM, Chris Chiaverini > >> > wrote: > >> > > >> > It seems that bro 2.5.1 is not taking the SNF_DATARING_SIZE > >> variable, no matter what I set it to. > >> > > >> > When at the defaults in the /etc/bro/node.cfg and with nothing > >> set at the shell, it still reports it is set via "userset" > >> instead of "default" like SNF_DESCRING_SIZE. > >> > >> Can you do this quick test using tcpdump to verify the problem is > >> with bro/broctl or something with the myricom driver/library? > >> > >> SNF_APP_ID=10 SNF_FLAGS=0x1 SNF_NUM_RINGS=8 SNF_DEBUG_MASK=3 > >> SNF_DATARING_SIZE=4294967296 SNF_DESCRING_SIZE=1073741824 tcpdump > >> -n -i snf0 -c 1 > >> > >> When I run that I get > >> > >> 23681 snf.0.-1 P (userset) SNF_PORTNUM = 0 > >> 23681 snf.0.-1 P (default) SNF_RING_ID = -1 (0xffffffff) > >> 23681 snf.0.-1 P (environ) SNF_NUM_RINGS = 8 (0x8) > >> 23681 snf.0.-1 P (default) SNF_RSS_FLAGS = 49 (0x31) > >> 23681 snf.0.-1 P (environ) SNF_DATARING_SIZE = 4294967296 > >> (0x100000000) (4096.0 MiB) > >> 23681 snf.0.-1 P (environ) SNF_DESCRING_SIZE = 1073741824 > >> (0x40000000) (1024.0 MiB) > >> 23681 snf.0.-1 P (userset) SNF_FLAGS = 1 (0x1) > >> 23681 snf.0.-1 P (environ) SNF_DEBUG_MASK = 3 (0x3) > >> 23681 snf.0.-1 P (default) SNF_DEBUG_FILENAME = stderr > >> 23681 snf.0.-1 P (environ) SNF_APP_ID = 10 (0xa) > >> > >> > >> > >> -- > >> - Justin Azoff > >> > >> > >> _______________________________________________ > >> Bro mailing list > >> bro at bro-ids.org > >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > >> > > > > > > > >_______________________________________________ > >Bro mailing list > >bro at bro-ids.org > >http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From shaananc at hotmail.com Thu Jun 22 00:42:01 2017 From: shaananc at hotmail.com (Shaanan Cohney) Date: Thu, 22 Jun 2017 07:42:01 +0000 Subject: [Bro] SSL Events not being triggered Message-ID: I am having trouble getting any SSL events to trigger. I've tested scripts against the snakeoil pcap from the wireshark sample pages, as well as my own data. I've also tried scripts ranging from the one I actually plan to use, along with a few barebones ones. When I use the -d flag and break on an connection I can see port 443 in the four tuple, but the SSL field remains uninitialized. Here's a sample of something I tried event bro_init(){ #Analyzer::enable_analyzer(Analyzer::ANALYZER_SSL); #Analyzer::register_for_port(Analyzer::ANALYZER_SSL, 443/tcp); } event ssl_established(c: connection) &priority=3 { print c; } I've also tried my own compiled version of bro, and the apt package on the latest version of ubuntu. I'm running bro as follows bro -r a.pcap ./nameofscript.bro Any help would be much appreciated! Thanks, snc -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170622/c89f74ce/attachment.html From shaananc at hotmail.com Thu Jun 22 01:29:07 2017 From: shaananc at hotmail.com (Shaanan Cohney) Date: Thu, 22 Jun 2017 08:29:07 +0000 Subject: [Bro] SSL Events not being triggered In-Reply-To: References: Message-ID: Solved it after a while. All the pcaps I was using had invalid checksums. If anyone comes across this thread with the same problem see https://www.bro.org/documentation/faq.html#why-isn-t-bro-producing-the-logs-i-expect-a-note-about-checksums Frequently Asked Questions - Bro www.bro.org What is Bro? Bro provides a comprehensive platform for network traffic analysis, with a particular focus on semantic security monitoring at scale. Thanks! ________________________________ From: bro-bounces at bro.org on behalf of Shaanan Cohney Sent: Thursday, June 22, 2017 3:42:01 AM To: bro at bro.org Subject: [Bro] SSL Events not being triggered I am having trouble getting any SSL events to trigger. I've tested scripts against the snakeoil pcap from the wireshark sample pages, as well as my own data. I've also tried scripts ranging from the one I actually plan to use, along with a few barebones ones. When I use the -d flag and break on an connection I can see port 443 in the four tuple, but the SSL field remains uninitialized. Here's a sample of something I tried event bro_init(){ #Analyzer::enable_analyzer(Analyzer::ANALYZER_SSL); #Analyzer::register_for_port(Analyzer::ANALYZER_SSL, 443/tcp); } event ssl_established(c: connection) &priority=3 { print c; } I've also tried my own compiled version of bro, and the apt package on the latest version of ubuntu. I'm running bro as follows bro -r a.pcap ./nameofscript.bro Any help would be much appreciated! Thanks, snc -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170622/d26dfeff/attachment.html From shaananc at hotmail.com Thu Jun 22 01:47:56 2017 From: shaananc at hotmail.com (Shaanan Cohney) Date: Thu, 22 Jun 2017 08:47:56 +0000 Subject: [Bro] SSL Events not being triggered In-Reply-To: References: , Message-ID: Still encountering the problem with another pcap. Sorry for the extra message! Any ideas? ________________________________ From: bro-bounces at bro.org on behalf of Shaanan Cohney Sent: Thursday, June 22, 2017 4:29:07 AM To: bro at bro.org Subject: Re: [Bro] SSL Events not being triggered Solved it after a while. All the pcaps I was using had invalid checksums. If anyone comes across this thread with the same problem see https://www.bro.org/documentation/faq.html#why-isn-t-bro-producing-the-logs-i-expect-a-note-about-checksums Frequently Asked Questions - Bro www.bro.org What is Bro? Bro provides a comprehensive platform for network traffic analysis, with a particular focus on semantic security monitoring at scale. Thanks! ________________________________ From: bro-bounces at bro.org on behalf of Shaanan Cohney Sent: Thursday, June 22, 2017 3:42:01 AM To: bro at bro.org Subject: [Bro] SSL Events not being triggered I am having trouble getting any SSL events to trigger. I've tested scripts against the snakeoil pcap from the wireshark sample pages, as well as my own data. I've also tried scripts ranging from the one I actually plan to use, along with a few barebones ones. When I use the -d flag and break on an connection I can see port 443 in the four tuple, but the SSL field remains uninitialized. Here's a sample of something I tried event bro_init(){ #Analyzer::enable_analyzer(Analyzer::ANALYZER_SSL); #Analyzer::register_for_port(Analyzer::ANALYZER_SSL, 443/tcp); } event ssl_established(c: connection) &priority=3 { print c; } I've also tried my own compiled version of bro, and the apt package on the latest version of ubuntu. I'm running bro as follows bro -r a.pcap ./nameofscript.bro Any help would be much appreciated! Thanks, snc -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170622/7a398afd/attachment-0001.html From egoant495 at gmail.com Thu Jun 22 03:02:41 2017 From: egoant495 at gmail.com (Anton Egorov) Date: Thu, 22 Jun 2017 13:02:41 +0300 Subject: [Bro] Bro doesn't detect SSH version in local network In-Reply-To: <8C3C7C97-ECAF-48F3-9F2C-C94CAFF51AFE@illinois.edu> References: <3694DCA6-36DE-47A2-9549-54615A7FD110@illinois.edu> <8C3C7C97-ECAF-48F3-9F2C-C94CAFF51AFE@illinois.edu> Message-ID: Connection entries differs only in ` local_orig local_resp` fields. What is the meaning of these connection parameters? 2017-06-21 18:56 GMT+03:00 Azoff, Justin S : > > > > On Jun 21, 2017, at 11:48 AM, Anton Egorov wrote: > > > > Thank you, I'll upgrade to the 2.5 but the results will be later. > > > > Here the connection logs. > > Both of those conn log entries show that your pcaps only have packets from > the source and no packets from the destination. Bro needs to see both > halves of the conversation in order to work right. I'm surprised the pcap > from eth0 reports an ssh version at all. > > > For command: > > > > # /usr/local/bro/bin/bro -C -r /root/eth0-ssh.cap > /usr/local/bro/share/bro/pluton/os-app-detect.bro local > > > > conn.log is: > > > > #separator \x09 > > #set_separator , > > #empty_field (empty) > > #unset_field - > > #path conn > > #open 2017-06-21-18-39-12 > > #fields ts uid id.orig_h id.orig_p id.resp_h > id.resp_p proto service duration orig_bytes > resp_bytes conn_state local_orig local_resp > missed_bytes history orig_pkts orig_ip_bytes resp_pkts > resp_ip_bytes tunnel_parents orig_cc resp_cc sensorname > > #types time string addr port addr port enum string > interval count count string bool bool count string > count count count count set[string] string string string > > 1497975205.750554 CEKFhs8sv3uEuTO6e 10.31.10.189 34496 > 10.31.10.192 22 tcp ssh 1.860624 3151 0 > SH T T 0 SADF 24 4407 0 0 > (empty) - - (empty) > > #close 2017-06-21-18-39-12 > > > > And for command > > > > # /usr/local/bro/bin/bro -C -r /root/eth1-ssh.cap > /usr/local/bro/share/bro/pluton/os-app-detect.bro local > > > > conn.log is: > > > > #separator \x09 > > #set_separator , > > #empty_field (empty) > > #unset_field - > > #path conn > > #open 2017-06-21-18-40-03 > > #fields ts uid id.orig_h id.orig_p id.resp_h > id.resp_p proto service duration orig_bytes > resp_bytes conn_state local_orig local_resp > missed_bytes history orig_pkts orig_ip_bytes resp_pkts > resp_ip_bytes tunnel_parents orig_cc resp_cc sensorname > > #types time string addr port addr port enum string > interval count count string bool bool count string > count count count count set[string] string string string > > 1497975114.740641 CX1F8jnuI7WceEAP6 192.168.99.189 33384 > 192.168.99.192 22 tcp ssh 4.030616 3151 0 > SH F F 0 SADF 25 4459 0 0 > (empty) - - (empty) > > #close 2017-06-21-18-40-03 > > > > > > 2017-06-21 17:44 GMT+03:00 Azoff, Justin S : > > > > > On Jun 21, 2017, at 10:37 AM, Anton Egorov > wrote: > > > > > > The offloading is disabled on both NIC's and the -C option also > doesn't do the trick. > > > > > > While reading pcap of a saved ssh traffic bro outputs a warning: > > > > > > # /usr/local/bro/bin/bro -C -r /root/eth1-ssh.cap > /usr/local/bro/share/bro/pluton/os-app-detect.bro local > > > UNKNOWN > > > 1497975118.771257 warning: Stream SOrfileNrXm8iGmlR6 is already queued > for removal. Ignoring remove. > > > > > > while on a pcap from the other interface: > > > > > > # /usr/local/bro/bin/bro -C -r /root/eth0-ssh.cap > /usr/local/bro/share/bro/pluton/os-app-detect.bro local > > > UNKNOWN > > > OpenSSH OpenSSH_6.0p1 Debian-4+deb7u3 > > > > What does the full conn.log entry show for the SSH connection in these > two cases? > > > > Can you upgrade bro to 2.5 or the 2.5.1 beta? 2.4.1 is almost a year > old at this point. > > > > -- > > - Justin Azoff > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170622/c23a0c63/attachment.html From jazoff at illinois.edu Thu Jun 22 06:44:48 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 22 Jun 2017 13:44:48 +0000 Subject: [Bro] Bro doesn't detect SSH version in local network In-Reply-To: References: <3694DCA6-36DE-47A2-9549-54615A7FD110@illinois.edu> <8C3C7C97-ECAF-48F3-9F2C-C94CAFF51AFE@illinois.edu> Message-ID: > On Jun 22, 2017, at 6:02 AM, Anton Egorov wrote: > > Connection entries differs only in ` local_orig local_resp` fields. What is the meaning of these connection parameters? Ah, so you have 2 separate problems here. Your first problem was that bro was only seeing half of the traffic. Note, this does not have anything to do with wether or not you ran an ls command. The TCP 3 way handshake and the ssh negotiation would include traffic from both sides. Your latest conn log entry shows a proper record with packets from both directions of the connection, so whatever the issue you were having with that has been resolved. Your second problem is that you are using the Software::log_software event. By default this will only log software seen on local ip addresses. For a bro installation that is using broctl this is controlled by /usr/local/bro/etc/networks.cfg. If you're normally using broctl just ensure that 192.168.99.0/24 and 10.31.10.0/24 (or whatever larger block you are using) is present in that file. If you're not using broctl just use another script that includes redef Site::local_nets = { 10.0.0.0/8, # Private IP space 192.168.0.0/16, # Private IP space }; -- - Justin Azoff From bill.de.ping at gmail.com Thu Jun 22 06:46:14 2017 From: bill.de.ping at gmail.com (william de ping) Date: Thu, 22 Jun 2017 16:46:14 +0300 Subject: [Bro] - http$host diff between bro and broctl In-Reply-To: <05B2AB4A-A639-40D2-AB5D-67421D122EC0@illinois.edu> References: <05B2AB4A-A639-40D2-AB5D-67421D122EC0@illinois.edu> Message-ID: Thank you ! it turns out to be checksum B On Thu, Jun 22, 2017 at 12:17 AM, Azoff, Justin S wrote: > > > On Jun 21, 2017, at 12:29 PM, william de ping > wrote: > > > > Hi all, > > > > Scenario 1 : bro instance on my local interface + browsing to > www.bbc.com > > Scenario 2 : bro cluster with a single Worker on my local interface + > browsing to www.cnn.com > > > > in http.log, > > on the 1st scenario, the host field is initialized with www.bbc.com > > on the 2nd scenario, the host field is NOT initialized > > > > I'm running bro 2.5 > > > > Is there any explanation for the diff ? > > You're probably starting bro differently in the two cases. > > https://www.bro.org/documentation/faq.html#why- > isn-t-bro-producing-the-logs-i-expect-a-note-about-checksums > > -- > - Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170622/8921b8d0/attachment.html From valerio.click at gmx.com Thu Jun 22 08:39:26 2017 From: valerio.click at gmx.com (Valerio) Date: Thu, 22 Jun 2017 17:39:26 +0200 Subject: [Bro] Problems in writing an analyzer for custom TCP-based protocol Message-ID: Hi all, I am trying to write a simple analyzer with BinPAC for a custom binary TCP protocol with the following structure +------------+---------------------------------------------------------+ |FIXED | MESSAGE_1 # MESSAGE_2 # .......# MESSAGE_N | |HEX PREFIX | | +------------+---------------------------------------------------------+ The problem is that the above structure doesn't have a fixed lenght and, therefore, it can be spread across several TCP packets. At the moment, my prot_protocol.pac file has the following structure: type PROT_Message = record { entry: uint8[] &until($input.length() == 0); }; type PROT_PDU(is_orig: bool) = record { entry : bytestring &restofdata; } &byteorder=bigendian; While the analyzer processes every packet by calling the following function (as defined in prot_analyzer.pac refine flow PROT_FLOW += { function proc_prot_message(msg: PROT_PDU): bool [...] The analyzer is activated with a dpd signature that matches the FIXED HEX PREFIX. Once activated, the function proc_prot_message is called for each packet in the session exposing in msg its payload. What is the best way to feed the analyzer with the whole reassembled TCP payload so that I can process it once without having to keep state while processing each single packet? many thanks in advance, Valerio From egoant495 at gmail.com Thu Jun 22 08:39:53 2017 From: egoant495 at gmail.com (Anton Egorov) Date: Thu, 22 Jun 2017 18:39:53 +0300 Subject: [Bro] Bro doesn't detect SSH version in local network In-Reply-To: References: <3694DCA6-36DE-47A2-9549-54615A7FD110@illinois.edu> <8C3C7C97-ECAF-48F3-9F2C-C94CAFF51AFE@illinois.edu> Message-ID: Thank you very much. After setting proper local IP space it is working. 2017-06-22 16:44 GMT+03:00 Azoff, Justin S : > > > On Jun 22, 2017, at 6:02 AM, Anton Egorov wrote: > > > > Connection entries differs only in ` local_orig local_resp` fields. > What is the meaning of these connection parameters? > > Ah, so you have 2 separate problems here. > > Your first problem was that bro was only seeing half of the traffic. > Note, this does not have anything to do with wether or not you ran an ls > command. The TCP 3 way handshake and the ssh negotiation would include > traffic from both sides. > > Your latest conn log entry shows a proper record with packets from both > directions of the connection, so whatever the issue you were having with > that has been resolved. > > Your second problem is that you are using the Software::log_software > event. By default this will only log software seen on local ip addresses. > For a bro installation that is using broctl this is controlled by > /usr/local/bro/etc/networks.cfg. If you're normally using broctl just > ensure that 192.168.99.0/24 and 10.31.10.0/24 (or whatever larger block > you are using) is present in that file. If you're not using broctl just > use another script that includes > > redef Site::local_nets = { > 10.0.0.0/8, # Private IP space > 192.168.0.0/16, # Private IP space > }; > > > -- > - Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170622/1f28adb4/attachment.html From johanna at icir.org Thu Jun 22 12:14:15 2017 From: johanna at icir.org (Johanna Amann) Date: Thu, 22 Jun 2017 12:14:15 -0700 Subject: [Bro] Allowing only certain log types In-Reply-To: References: Message-ID: <20170622191415.xohowm7immuwqbsl@wifi180.sys.ICSI.Berkeley.EDU> Hi, in addition to disabling log files (which you can do using Log::disable_stream, as was already pointed out), you can start Bro in bare mode. This will not enable any analyzers by default, you will have to load them manually, wich can save a bit of processing. Note however that bare mode comes with its own complications - you have to be sure that you load everything that is required (it is easy to, for example, forget to load the dynamic protocol detection scripts); this is not an approach I would generally recommend. Johanna On Tue, Jun 13, 2017 at 05:43:54PM +0300, Sherif Eldeeb wrote: > We are planning to only use the "logging" features of Bro, and for certain > types, on a 10G link. > > I'd appreciate pointing me to right direction to only enable (conn.log, > dns.log, http.log and ssl.log) while disabling all the others (to save > processing cycles and storage) for the types that we won't use/need. > > Thanks. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Thu Jun 22 12:20:57 2017 From: johanna at icir.org (Johanna Amann) Date: Thu, 22 Jun 2017 12:20:57 -0700 Subject: [Bro] Bro restrict filters question In-Reply-To: References: <6C4D5D51-1D0B-46AC-AA42-B5B2F4E6A806@illinois.edu> Message-ID: <20170622192057.vhyv5mr7v6gljsjv@wifi180.sys.ICSI.Berkeley.EDU> Can you check if "broctl "print PacketFilter::current_filter" looks reasonable, and if the exact filter it returns works for you with tcpdump? Johanna On Tue, Jun 13, 2017 at 07:05:29PM +0000, Edgmand, Craig wrote: > Oddly enough it works with tcpdump but not with Bro. > > -----Original Message----- > From: Azoff, Justin S [mailto:jazoff at illinois.edu] > Sent: Tuesday, June 13, 2017 10:13 AM > To: Edgmand, Craig > Cc: bro at bro.org > Subject: Re: [Bro] Bro restrict filters question > > > > On Jun 13, 2017, at 10:59 AM, Edgmand, Craig wrote: > > > > Hello, > > > > I am running Bro 2.5 and I am trying to set up some restrict_filters to drop certain hosts and types of traffic. > > I have the following entries in my local.bro.. > > > > redef PacketFilter::enable_auto_protocol_capture_filters = F; redef > > capture_filters = { ["packets-like-this"] = "ip or not ip" }; redef > > restrict_filters = { ["no-data-like-this"] = "not host 192.168.2.1" }; > > > > > > I had something similar in earlier versions of Bro that seemed to work but this doesn?t work at all. > > > > When I run ./broctl print restrict_filters it shows that the workers have that filter. > > > > Any ideas? > > Is your traffic vlan tagged? You may need to use > > redef restrict_filters = { ["no-data-like-this"] = "vlan and not host 192.168.2.1" }; > > -- > - Justin Azoff > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From johanna at icir.org Thu Jun 22 12:25:20 2017 From: johanna at icir.org (Johanna Amann) Date: Thu, 22 Jun 2017 12:25:20 -0700 Subject: [Bro] - send syslog message via Bro In-Reply-To: References: Message-ID: <20170622192520.53po2b7sdgsdghii@wifi180.sys.ICSI.Berkeley.EDU> Hello, > Is it possible to send a syslog message from a bro script to a specific > host ? Bro internally just uses the vsyslog function to send the data to syslog (https://linux.die.net/man/3/vsyslog), which also just takes a string. It is currently not possible to configure bro to send a message to a specific host - that is something that you have to change in your syslog configuration. I hope this helps, Johanna From johanna at icir.org Thu Jun 22 12:27:56 2017 From: johanna at icir.org (Johanna Amann) Date: Thu, 22 Jun 2017 12:27:56 -0700 Subject: [Bro] bro scripts global vars In-Reply-To: <1497433885.6910.50.camel@gmail.com> References: <1497340657.6910.38.camel@gmail.com> <20170613171223.k4xkbu22bbpegrnd@Beezling.local> <1497433885.6910.50.camel@gmail.com> Message-ID: <20170622192756.f3msbjz7m6qhi6ap@wifi180.sys.ICSI.Berkeley.EDU> Stdout is a bit special. Especially if you are running in a cluster environment, it is probably easier to just create a new log-file and write your data to it. Apart from that, you should find the stdout.log for your worker nodes in spool/worker-[x]/stdout.log. Johanna On Wed, Jun 14, 2017 at 11:51:25AM +0200, Ernest Farias wrote: > Thanks Johanna! > But now it arise another question it works fine when test on cmd line > but using broctl and I supposed it would go to my > ?/log/current/stdout.log (?) , but it only contains this, I don't know > what I'm doing wrong > > "max memory size?????????(kbytes, -m) unlimited > data seg size???????????(kbytes, -d) unlimited > virtual memory??????????(kbytes, -v) unlimited > core file size??????????(blocks, -c) unlimited" > > Thanks > > Ernest > > > On Tue, 2017-06-13 at 10:12 -0700, Johanna Amann wrote: > > Hi, > > > > > > > > What's the best way to know the value of globas vars on my loaded > > > bro > > > scripts? > > The easiest way probably is to just check their values in a bro_init > > event. Unless there is a reason that you can't do that? > > > > Johanna > From johanna at icir.org Thu Jun 22 12:34:39 2017 From: johanna at icir.org (Johanna Amann) Date: Thu, 22 Jun 2017 12:34:39 -0700 Subject: [Bro] - BROKER - send messages from bro workers to python listener In-Reply-To: References: Message-ID: <20170622193439.pndggpkdu7nctd7i@wifi180.sys.ICSI.Berkeley.EDU> Hi, > I have a cluster and I wish bro workers to notify a single python script > with a costume text message. > > On the python side I use pybroker. > > Do I need to use listen for the python side and send on the bro side ? > How does the publishing works ? This is a twofold question. First - broker has to connect python and Bro. For that it actually is completely up to you; you can either have python listen for connections of Bro or vice-versa. The connection is bidirectional afterwards, it does not matter who established it. After that, the python side has to subscribe to the correct events, and the Bro side has to send them. The best way is to probably look at examples; the best example nowadays are probably the netcontrol connectors. https://github.com/bro/bro/blob/master/scripts/base/frameworks/netcontrol/plugins/broker.bro is the script that sends out netcontrol events, https://github.com/bro/bro-netcontrol/blob/master/netcontrol/api.py it the script that implements the API for listening to events. Note that the python and Bro API for Broker are not fixed yet and will change in the future. I hope this helps a bit, Johanna From johanna at icir.org Thu Jun 22 12:42:08 2017 From: johanna at icir.org (Johanna Amann) Date: Thu, 22 Jun 2017 12:42:08 -0700 Subject: [Bro] Digging through Source Code In-Reply-To: <0C34D9CA9B9DBB45B1C51871C177B4B286AAEF1D@UMECHPA68.easf.csd.disa.mil> References: <0C34D9CA9B9DBB45B1C51871C177B4B286AAEF1D@UMECHPA68.easf.csd.disa.mil> Message-ID: <20170622194208.6yxzm55qm6lokcrg@wifi180.sys.ICSI.Berkeley.EDU> On Tue, Jun 20, 2017 at 08:42:35PM +0000, Weasel, Gary W Jr CIV DISA RE (US) wrote: > element is type KRB_REQ_Arg (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/krb/krb-protocol.pac) > -> data is type KRB_REQ_Arg_Data (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/krb/krb-protocol.pac) > -> etype is type Array (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/asn1/asn1.pac) > -> data is type ASN1Encoding (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/asn1/asn1.pac) > -> size is type ? Actually, data is of type ASN1Encoding[] - so it is an array; size returns the size of that array (i.e. the number of array elements). I hope that helps, Johanna From johanna at icir.org Thu Jun 22 12:45:25 2017 From: johanna at icir.org (Johanna Amann) Date: Thu, 22 Jun 2017 12:45:25 -0700 Subject: [Bro] Digging through Source Code In-Reply-To: <43326d22-6731-ce2d-1d5b-28b537852fc1@gmail.com> References: <0C34D9CA9B9DBB45B1C51871C177B4B286AAEF1D@UMECHPA68.easf.csd.disa.mil> <43326d22-6731-ce2d-1d5b-28b537852fc1@gmail.com> Message-ID: <20170622194525.mndtxwhnpyhk6dqz@wifi180.sys.ICSI.Berkeley.EDU> I don't know ETSI standard files, but just assuming they are some kind of ASN.1 data: While Bro has a bit of ASN.1 parsing capability (meaning that there is a binpac definition for parts of ASN.1), the implementation is limited to a small subset of ASN.1. Furthermore it is no generic parser - one still has to implement the actual parsing logic for the specific ASN.1 data on top of the existing primitives. So - no, not currently. Johanna On Tue, Jun 20, 2017 at 11:43:36PM +0200, Daniel Guerra wrote: > Talking about ASN1. Would bro be able to read ETSI standard files ? > > > Op 20/06/2017 om 22:42 schreef Weasel, Gary W Jr CIV DISA RE (US): > > Yes, but there's something that's still stumping me. > > > > Looking at line 70 from https://github.com/bro/bro/blob/master/src/analyzer/protocol/krb/krb-analyzer.pac > > > > case 8: > > if ( element->data()->etype()->data()->size() ) > > rv->Assign(11, proc_cipher_list(element->data()->etype())); > > > > Following the breadcrumb trail in the if statement here... > > > > element is type KRB_REQ_Arg (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/krb/krb-protocol.pac) > > -> data is type KRB_REQ_Arg_Data (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/krb/krb-protocol.pac) > > -> etype is type Array (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/asn1/asn1.pac) > > -> data is type ASN1Encoding (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/asn1/asn1.pac) > > -> size is type ? > > > > Following this line of thought, I'm a little confused by what "size()" is supposed to mean here, since it's not an attribute. I can infer that it's simply returning the size of the record, but I don't have any information as to how or where that would be defined. I've also tried looking through the source of BinPAC (https://www.bro.org/sphinx/components/binpac/README.html) but have come up empty so far. > > > > I have a sample of kerberos pcap that populates the msg$pa_data$encryption_type vector (from event krb_tgs_request), so I know that the aforementioned if statement is returning true - - but the other two vectors "host_addrs" and "additional"tickets" (that from documentation seem to imply they're parallel with the encryption_type vector) come up as . > > > > This made me question that maybe there was something wrong with the code that was causing it to miss the host_addr and ticket data, I clearly find this data in my pcap sample under padata. This is my current theory anyway, and wanted to see if I'm making a bad assumption somewhere or if someone can shed light on what's going on here. > > > > > > -----Original Message----- > > From: Azoff, Justin S [mailto:jazoff at illinois.edu] > > Sent: Tuesday, June 20, 2017 3:28 PM > > To: Weasel, Gary W Jr CIV DISA RE (US) > > Cc: bro at bro.org > > Subject: Re: [Bro] Digging through Source Code > > > > All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. > > > > > > > > > > ---- > > > > > >> On Jun 20, 2017, at 3:14 PM, Weasel, Gary W Jr CIV DISA RE (US) wrote: > >> > >> All, > >> > >> I've been digging through the Bro source code, and there's been something that's mystifying me for a while now. > >> > >> type Array = record { > >> array_meta: ASN1EncodingMeta; > >> data: ASN1Encoding[]; > >> }; > >> > >> As from https://github.com/bro/bro/blob/57da2d091b30aad52d52fce8018feeb2cdf8ff1f/src/analyzer/protocol/asn1/asn1.pac > >> > >> I have no clue what "record" is in this context. I suspect it has other attributes that are being inherited, but I haven't found anything to indicate what this is. Does anyone have any insight into this? > >> > >> Thanks, > >> - Gary > > Does this help? > > > > https://www.bro.org/sphinx/script-reference/types.html#type-record > > > > > > -- > > - Justin Azoff > > > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From johanna at icir.org Thu Jun 22 12:47:54 2017 From: johanna at icir.org (Johanna Amann) Date: Thu, 22 Jun 2017 12:47:54 -0700 Subject: [Bro] Bro 2.5 appears to be ignoring redefs of Pcap::snaplen In-Reply-To: References: Message-ID: <20170622194754.pizvdxexyfpzw76r@wifi180.sys.ICSI.Berkeley.EDU> Are you using pf_ring through libpcap, or are you using the pf-ring plugin? In case you are using it through libpcap - Bro just calls pcap_set_snaplen; if it does not work anymore it is probably that this is an issue with PF_RING or the pfring libpcap. Johanna On Wed, Jun 21, 2017 at 11:29:18AM -0400, Kevin Branch wrote: > For a long time I have used "redef Pcap::snaplen = 1600;" in local.bro to > make Bro drop its default snaplen from 8192 to 1600. This is helpful for > conserving memory when using Bro in conjunction with PF_RING and a high > number of ring slots. > > Today I just noticed that while Bro does not complain about "redef > Pcap::snaplen = 1600;" when I run a "broctl check", that Bro appears to be > ignoring the redef. All my Bro instances are actually using a snaplen of > 8192. > > I use Bro on the latest Security Onion Ubuntu 14.04 platform, and have > observed this problem with both PF_RING 6.4.1 (SO stable) and PF_RING 6.6.0 > (SO test). > > The "Bucket Len" in the below PF_RING status file corresponds to the > snaplen of the app that allocated the ring. > > root at nsm.xyz.org:~# cat /proc/net/pf_ring/15028-dmz.9 > Bound Device(s) : dmz > Active : 1 > Breed : Standard > Appl. Name : bro-dmz > Socket Mode : RX+TX > Capture Direction : RX+TX > Sampling Rate : 1 > IP Defragment : No > BPF Filtering : Enabled > Sw Filt Hash Rules : 0 > Sw Filt WC Rules : 0 > Hw Filt Rules : 0 > Sw Filt Hash Match : 0 > Sw Filt Hash Miss : 0 > Poll Pkt Watermark : 1 > Num Poll Calls : 345386919 > Channel Id Mask : 0xFFFFFFFFFFFFFFFF > Cluster Id : 21 > Slot Version : 16 [6.4.1] > Min Num Slots : 128000 > Bucket Len : 8192 > Slot Len : 8248 [bucket+header] > Tot Memory : 1055756288 > Tot Packets : 1966471960 > Tot Pkt Lost : 3 > Tot Insert : 1966471957 > Tot Read : 1966471957 > Insert Offset : 809944608 > Remove Offset : 809944608 > Num Free Slots : 128000 > TX: Send Ok : 0 > TX: Send Errors : 0 > Reflect: Fwd Ok : 0 > Reflect: Fwd Errors: 0 > > > Please advise me about how to successfully change the snaplen used by Bro > 2.5 at this time, Can anyone reproduce this problem? I don't know if this > issue applies across the board or only comes up with PF_RING. Let me know > if there is anything I can do to help test this issue. > > Thanks! > Kevin > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From asharma at lbl.gov Thu Jun 22 13:14:20 2017 From: asharma at lbl.gov (Aashish Sharma) Date: Thu, 22 Jun 2017 13:14:20 -0700 Subject: [Bro] bro scripts global vars In-Reply-To: <20170622192756.f3msbjz7m6qhi6ap@wifi180.sys.ICSI.Berkeley.EDU> References: <1497340657.6910.38.camel@gmail.com> <20170613171223.k4xkbu22bbpegrnd@Beezling.local> <1497433885.6910.50.camel@gmail.com> <20170622192756.f3msbjz7m6qhi6ap@wifi180.sys.ICSI.Berkeley.EDU> Message-ID: <20170622201420.GE51206@mac-822.local> So I use (or misuse) a few different techniques to see whats in globals and tables etc. 1) reporter.log use: event reporter_info(network_time(), msg, peer_description); and msg contains whatever output/result/values I'd like. this way, I can look at debugging within a cluster I actually use a wrapper: function log_reporter(msg: string, debug: count) { if (debug <= 5) { @if ( ! Cluster::is_enabled()) print fmt("%s", msg); @endif event reporter_info(network_time(), msg, peer_description); } } and call it as within the functions. log_reporter(fmt("EVENT: value of variable is %s", my_global_var),1); 2) in some-cases I'd schedule a print_stats event and dump information periodically using (1) above 3) using broctl print you can also try: broctl print :: eg. broctl print Site::local_nets however if your tables are > N this may take a long time to work or may not work. I have been unable to understand why but some timeouts happen. There is a broctl.cfg setting which can increase the timeouts but that's also best effort - may or may not work. But for all practical purposes broctl print :: works Hope this helps, Aashish On Thu, Jun 22, 2017 at 12:27:56PM -0700, Johanna Amann wrote: > Stdout is a bit special. Especially if you are running in a cluster > environment, it is probably easier to just create a new log-file and write > your data to it. > > Apart from that, you should find the stdout.log for your worker nodes in > spool/worker-[x]/stdout.log. > > Johanna > > On Wed, Jun 14, 2017 at 11:51:25AM +0200, Ernest Farias wrote: > > Thanks Johanna! > > But now it arise another question it works fine when test on cmd line > > but using broctl and I supposed it would go to my > > ?/log/current/stdout.log (?) , but it only contains this, I don't know > > what I'm doing wrong > > > > "max memory size?????????(kbytes, -m) unlimited > > data seg size???????????(kbytes, -d) unlimited > > virtual memory??????????(kbytes, -v) unlimited > > core file size??????????(blocks, -c) unlimited" > > > > Thanks > > > > Ernest > > > > > > On Tue, 2017-06-13 at 10:12 -0700, Johanna Amann wrote: > > > Hi, > > > > > > > > > > > What's the best way to know the value of globas vars on my loaded > > > > bro > > > > scripts? > > > The easiest way probably is to just check their values in a bro_init > > > event. Unless there is a reason that you can't do that? > > > > > > Johanna > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jazoff at illinois.edu Thu Jun 22 13:41:20 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 22 Jun 2017 20:41:20 +0000 Subject: [Bro] bro scripts global vars In-Reply-To: <20170622201420.GE51206@mac-822.local> References: <1497340657.6910.38.camel@gmail.com> <20170613171223.k4xkbu22bbpegrnd@Beezling.local> <1497433885.6910.50.camel@gmail.com> <20170622192756.f3msbjz7m6qhi6ap@wifi180.sys.ICSI.Berkeley.EDU> <20170622201420.GE51206@mac-822.local> Message-ID: > On Jun 22, 2017, at 4:14 PM, Aashish Sharma wrote: > > you can also try: > > broctl print :: > > eg. broctl print Site::local_nets > > > however if your tables are > N this may take a long time to work or may not work. I have been unable to understand why but some timeouts happen. There is a broctl.cfg setting which can increase the timeouts but that's also best effort - may or may not work. This is one thing that is multiple orders of magnitude better on the broker branch. With the broker branch, broctl print against tables with hundreds of thousands of items in them finishes in a second or two. -- - Justin Azoff From cchiaverini at bnl.gov Thu Jun 22 14:23:21 2017 From: cchiaverini at bnl.gov (Chris Chiaverini) Date: Thu, 22 Jun 2017 17:23:21 -0400 Subject: [Bro] Bro node.cfg not setting Myricom Sniffer10G environment variables In-Reply-To: <20170621235906.GS75741@mac-822.local> References: <30abafd5-fae4-f1d2-0df5-c62e82e777c4@bnl.gov> <2831AF5A-E4E7-4C8C-B4FF-9838A580FA22@illinois.edu> <51cec005-586b-26d8-42e8-6f1815dc9594@bnl.gov> <58b4bc70-bb8a-685e-b7cc-a28a327b3b5e@bnl.gov> <20170621235906.GS75741@mac-822.local> Message-ID: Rollback!!!! Myricom opened an internal ticket on their end so hopefully we will see a bugfix soon. Regards, Chris Chiaverini Cyber Security Operations Brookhaven National Laboratory Upton, New York 11973 On 06/21/2017 07:59 PM, Aashish Sharma wrote: > Doh! I just upgraded the myricom drivers to 3.0.11 today only :) > > Aashish > > On Wed, Jun 21, 2017 at 06:31:50PM -0400, Chris Chiaverini wrote: >> Alex, >> >> Thank you for this. I confirmed on my end too... rolled back to 3.0.10 and >> it worked. I will let you know what Myricom comes up with, if they will >> fix in next release. >> >> >> Regards, >> >> Chris Chiaverini >> Cyber Security Operations >> Brookhaven National Laboratory >> Upton, New York 11973 >> >> On 06/20/2017 11:01 AM, Chris Chiaverini wrote: >>> I have a support case open with them in parallel. I will report this to >>> them too. Maybe we'll get a fix in next minor release. >>> >>> Regards, >>> >>> Chris Chiaverini >>> Cyber Security Operations >>> Brookhaven National Laboratory >>> Upton, New York 11973 >>> On 06/20/2017 10:09 AM, Alejandro Carreno wrote: >>>> I noticed this behavior as well a while back after upgrading SNF from >>>> 3.0.10 to 3.0.11. Downgrading back to 3.0.10 would return the ring sizes >>>> to the expected values. >>>> >>>> -Alex >>>> >>>> On Tue, Jun 20, 2017 at 6:47 AM Azoff, Justin S >>> > wrote: >>>> >>>> >>>> > On Jun 20, 2017, at 9:27 AM, Chris Chiaverini >>>> > wrote: >>>> > >>>> > It seems that bro 2.5.1 is not taking the SNF_DATARING_SIZE >>>> variable, no matter what I set it to. >>>> > >>>> > When at the defaults in the /etc/bro/node.cfg and with nothing >>>> set at the shell, it still reports it is set via "userset" >>>> instead of "default" like SNF_DESCRING_SIZE. >>>> >>>> Can you do this quick test using tcpdump to verify the problem is >>>> with bro/broctl or something with the myricom driver/library? >>>> >>>> SNF_APP_ID=10 SNF_FLAGS=0x1 SNF_NUM_RINGS=8 SNF_DEBUG_MASK=3 >>>> SNF_DATARING_SIZE=4294967296 SNF_DESCRING_SIZE=1073741824 tcpdump >>>> -n -i snf0 -c 1 >>>> >>>> When I run that I get >>>> >>>> 23681 snf.0.-1 P (userset) SNF_PORTNUM = 0 >>>> 23681 snf.0.-1 P (default) SNF_RING_ID = -1 (0xffffffff) >>>> 23681 snf.0.-1 P (environ) SNF_NUM_RINGS = 8 (0x8) >>>> 23681 snf.0.-1 P (default) SNF_RSS_FLAGS = 49 (0x31) >>>> 23681 snf.0.-1 P (environ) SNF_DATARING_SIZE = 4294967296 >>>> (0x100000000) (4096.0 MiB) >>>> 23681 snf.0.-1 P (environ) SNF_DESCRING_SIZE = 1073741824 >>>> (0x40000000) (1024.0 MiB) >>>> 23681 snf.0.-1 P (userset) SNF_FLAGS = 1 (0x1) >>>> 23681 snf.0.-1 P (environ) SNF_DEBUG_MASK = 3 (0x3) >>>> 23681 snf.0.-1 P (default) SNF_DEBUG_FILENAME = stderr >>>> 23681 snf.0.-1 P (environ) SNF_APP_ID = 10 (0xa) >>>> >>>> >>>> >>>> -- >>>> - Justin Azoff >>>> >>>> >>>> _______________________________________________ >>>> Bro mailing list >>>> bro at bro-ids.org >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>>> >>> >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From dnthayer at illinois.edu Thu Jun 22 21:59:52 2017 From: dnthayer at illinois.edu (Thayer, Daniel N) Date: Fri, 23 Jun 2017 04:59:52 +0000 Subject: [Bro] Bro 2.5 appears to be ignoring redefs of Pcap::snaplen In-Reply-To: References: Message-ID: <8F865DA62E66F543B6104A2835719CF969D20A0C@CITESMBX5.ad.uillinois.edu> You might want to try setting this value in your etc/broctl.cfg file: pcapsnaplen=1600 ________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Kevin Branch [kevin at branchnetconsulting.com] Sent: Wednesday, June 21, 2017 10:29 AM To: bro at bro.org Subject: [Bro] Bro 2.5 appears to be ignoring redefs of Pcap::snaplen For a long time I have used "redef Pcap::snaplen = 1600;" in local.bro to make Bro drop its default snaplen from 8192 to 1600. This is helpful for conserving memory when using Bro in conjunction with PF_RING and a high number of ring slots. Today I just noticed that while Bro does not complain about "redef Pcap::snaplen = 1600;" when I run a "broctl check", that Bro appears to be ignoring the redef. All my Bro instances are actually using a snaplen of 8192. I use Bro on the latest Security Onion Ubuntu 14.04 platform, and have observed this problem with both PF_RING 6.4.1 (SO stable) and PF_RING 6.6.0 (SO test). The "Bucket Len" in the below PF_RING status file corresponds to the snaplen of the app that allocated the ring. root at nsm.xyz.org:~# cat /proc/net/pf_ring/15028-dmz.9 Bound Device(s) : dmz Active : 1 Breed : Standard Appl. Name : bro-dmz Socket Mode : RX+TX Capture Direction : RX+TX Sampling Rate : 1 IP Defragment : No BPF Filtering : Enabled Sw Filt Hash Rules : 0 Sw Filt WC Rules : 0 Hw Filt Rules : 0 Sw Filt Hash Match : 0 Sw Filt Hash Miss : 0 Poll Pkt Watermark : 1 Num Poll Calls : 345386919 Channel Id Mask : 0xFFFFFFFFFFFFFFFF Cluster Id : 21 Slot Version : 16 [6.4.1] Min Num Slots : 128000 Bucket Len : 8192 Slot Len : 8248 [bucket+header] Tot Memory : 1055756288 Tot Packets : 1966471960 Tot Pkt Lost : 3 Tot Insert : 1966471957 Tot Read : 1966471957 Insert Offset : 809944608 Remove Offset : 809944608 Num Free Slots : 128000 TX: Send Ok : 0 TX: Send Errors : 0 Reflect: Fwd Ok : 0 Reflect: Fwd Errors: 0 Please advise me about how to successfully change the snaplen used by Bro 2.5 at this time, Can anyone reproduce this problem? I don't know if this issue applies across the board or only comes up with PF_RING. Let me know if there is anything I can do to help test this issue. Thanks! Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170623/bd6c0844/attachment.html From zhangxu1115 at gmail.com Fri Jun 23 15:47:45 2017 From: zhangxu1115 at gmail.com (Xu Zhang) Date: Fri, 23 Jun 2017 15:47:45 -0700 Subject: [Bro] get TCP payload of first ACK from client Message-ID: Hello, I'm writing a bro script to output TCP payload of first ack from client (is_orig = True), I'm currently using tcp_packet event, check the ack flag and payload length as well as if it is the first ack. I'm wondering if there is a cheaper way to achieve this, since tcp_packet is pretty expensive. I cannot use connection_first_ACK event because it does not give me the actual TCP payload. I cannot use ssl_client_hello because i want to handle not only ssl. Does anyone have suggestions? Thanks for the help! -- Sincerely, Xu Zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170623/afee466f/attachment.html From seth at corelight.com Fri Jun 23 18:23:05 2017 From: seth at corelight.com (Seth Hall) Date: Fri, 23 Jun 2017 21:23:05 -0400 Subject: [Bro] Bro node.cfg not setting Myricom Sniffer10G environment variables In-Reply-To: References: <30abafd5-fae4-f1d2-0df5-c62e82e777c4@bnl.gov> <2831AF5A-E4E7-4C8C-B4FF-9838A580FA22@illinois.edu> <51cec005-586b-26d8-42e8-6f1815dc9594@bnl.gov> <58b4bc70-bb8a-685e-b7cc-a28a327b3b5e@bnl.gov> <20170621235906.GS75741@mac-822.local> Message-ID: You could also try the bro-myricom plugin from the Bro package repository. If you have bro-pkg set up, you should be able to do this... bro-pkg refresh bro-pkg install sethhall/bro-myricom There is documentation on how to use it here: https://github.com/sethhall/bro-myricom You only configure the data ring size in it. For some reason they don't expose the desc ring size option through their native SNF api, only the data ring size. I'm going to go out on a limb here and guess that you may be experiencing weird behavior because they probably want to get rid of the desc ring size option. It makes more sense if they just auto adjust that based on the chosen data ring size. .Seth On Thu, Jun 22, 2017 at 5:23 PM, Chris Chiaverini wrote: > Rollback!!!! > > Myricom opened an internal ticket on their end so hopefully we will see > a bugfix soon. > > Regards, > > Chris Chiaverini > Cyber Security Operations > Brookhaven National Laboratory > Upton, New York 11973 > > On 06/21/2017 07:59 PM, Aashish Sharma wrote: >> Doh! I just upgraded the myricom drivers to 3.0.11 today only :) >> >> Aashish >> >> On Wed, Jun 21, 2017 at 06:31:50PM -0400, Chris Chiaverini wrote: >>> Alex, >>> >>> Thank you for this. I confirmed on my end too... rolled back to 3.0.10 and >>> it worked. I will let you know what Myricom comes up with, if they will >>> fix in next release. >>> >>> >>> Regards, >>> >>> Chris Chiaverini >>> Cyber Security Operations >>> Brookhaven National Laboratory >>> Upton, New York 11973 >>> >>> On 06/20/2017 11:01 AM, Chris Chiaverini wrote: >>>> I have a support case open with them in parallel. I will report this to >>>> them too. Maybe we'll get a fix in next minor release. >>>> >>>> Regards, >>>> >>>> Chris Chiaverini >>>> Cyber Security Operations >>>> Brookhaven National Laboratory >>>> Upton, New York 11973 >>>> On 06/20/2017 10:09 AM, Alejandro Carreno wrote: >>>>> I noticed this behavior as well a while back after upgrading SNF from >>>>> 3.0.10 to 3.0.11. Downgrading back to 3.0.10 would return the ring sizes >>>>> to the expected values. >>>>> >>>>> -Alex >>>>> >>>>> On Tue, Jun 20, 2017 at 6:47 AM Azoff, Justin S >>>> > wrote: >>>>> >>>>> >>>>> > On Jun 20, 2017, at 9:27 AM, Chris Chiaverini >>>>> > wrote: >>>>> > >>>>> > It seems that bro 2.5.1 is not taking the SNF_DATARING_SIZE >>>>> variable, no matter what I set it to. >>>>> > >>>>> > When at the defaults in the /etc/bro/node.cfg and with nothing >>>>> set at the shell, it still reports it is set via "userset" >>>>> instead of "default" like SNF_DESCRING_SIZE. >>>>> >>>>> Can you do this quick test using tcpdump to verify the problem is >>>>> with bro/broctl or something with the myricom driver/library? >>>>> >>>>> SNF_APP_ID=10 SNF_FLAGS=0x1 SNF_NUM_RINGS=8 SNF_DEBUG_MASK=3 >>>>> SNF_DATARING_SIZE=4294967296 SNF_DESCRING_SIZE=1073741824 tcpdump >>>>> -n -i snf0 -c 1 >>>>> >>>>> When I run that I get >>>>> >>>>> 23681 snf.0.-1 P (userset) SNF_PORTNUM = 0 >>>>> 23681 snf.0.-1 P (default) SNF_RING_ID = -1 (0xffffffff) >>>>> 23681 snf.0.-1 P (environ) SNF_NUM_RINGS = 8 (0x8) >>>>> 23681 snf.0.-1 P (default) SNF_RSS_FLAGS = 49 (0x31) >>>>> 23681 snf.0.-1 P (environ) SNF_DATARING_SIZE = 4294967296 >>>>> (0x100000000) (4096.0 MiB) >>>>> 23681 snf.0.-1 P (environ) SNF_DESCRING_SIZE = 1073741824 >>>>> (0x40000000) (1024.0 MiB) >>>>> 23681 snf.0.-1 P (userset) SNF_FLAGS = 1 (0x1) >>>>> 23681 snf.0.-1 P (environ) SNF_DEBUG_MASK = 3 (0x3) >>>>> 23681 snf.0.-1 P (default) SNF_DEBUG_FILENAME = stderr >>>>> 23681 snf.0.-1 P (environ) SNF_APP_ID = 10 (0xa) >>>>> >>>>> >>>>> >>>>> -- >>>>> - Justin Azoff >>>>> >>>>> >>>>> _______________________________________________ >>>>> Bro mailing list >>>>> bro at bro-ids.org >>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Bro mailing list >>>> bro at bro-ids.org >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From craig.edgmand at okstate.edu Mon Jun 26 07:12:42 2017 From: craig.edgmand at okstate.edu (Edgmand, Craig) Date: Mon, 26 Jun 2017 14:12:42 +0000 Subject: [Bro] Bro node.cfg not setting Myricom Sniffer10G environment variables In-Reply-To: References: <30abafd5-fae4-f1d2-0df5-c62e82e777c4@bnl.gov> <2831AF5A-E4E7-4C8C-B4FF-9838A580FA22@illinois.edu> <51cec005-586b-26d8-42e8-6f1815dc9594@bnl.gov> <58b4bc70-bb8a-685e-b7cc-a28a327b3b5e@bnl.gov> <20170621235906.GS75741@mac-822.local> Message-ID: Does this only impact Bro 2.5.1? -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Seth Hall Sent: Friday, June 23, 2017 8:23 PM To: Chris Chiaverini Cc: bro Subject: Re: [Bro] Bro node.cfg not setting Myricom Sniffer10G environment variables You could also try the bro-myricom plugin from the Bro package repository. If you have bro-pkg set up, you should be able to do this... bro-pkg refresh bro-pkg install sethhall/bro-myricom There is documentation on how to use it here: https://github.com/sethhall/bro-myricom You only configure the data ring size in it. For some reason they don't expose the desc ring size option through their native SNF api, only the data ring size. I'm going to go out on a limb here and guess that you may be experiencing weird behavior because they probably want to get rid of the desc ring size option. It makes more sense if they just auto adjust that based on the chosen data ring size. .Seth On Thu, Jun 22, 2017 at 5:23 PM, Chris Chiaverini wrote: > Rollback!!!! > > Myricom opened an internal ticket on their end so hopefully we will > see a bugfix soon. > > Regards, > > Chris Chiaverini > Cyber Security Operations > Brookhaven National Laboratory > Upton, New York 11973 > > On 06/21/2017 07:59 PM, Aashish Sharma wrote: >> Doh! I just upgraded the myricom drivers to 3.0.11 today only :) >> >> Aashish >> >> On Wed, Jun 21, 2017 at 06:31:50PM -0400, Chris Chiaverini wrote: >>> Alex, >>> >>> Thank you for this. I confirmed on my end too... rolled back to 3.0.10 and >>> it worked. I will let you know what Myricom comes up with, if they will >>> fix in next release. >>> >>> >>> Regards, >>> >>> Chris Chiaverini >>> Cyber Security Operations >>> Brookhaven National Laboratory >>> Upton, New York 11973 >>> >>> On 06/20/2017 11:01 AM, Chris Chiaverini wrote: >>>> I have a support case open with them in parallel. I will report >>>> this to them too. Maybe we'll get a fix in next minor release. >>>> >>>> Regards, >>>> >>>> Chris Chiaverini >>>> Cyber Security Operations >>>> Brookhaven National Laboratory >>>> Upton, New York 11973 >>>> On 06/20/2017 10:09 AM, Alejandro Carreno wrote: >>>>> I noticed this behavior as well a while back after upgrading SNF >>>>> from >>>>> 3.0.10 to 3.0.11. Downgrading back to 3.0.10 would return the ring >>>>> sizes to the expected values. >>>>> >>>>> -Alex >>>>> >>>>> On Tue, Jun 20, 2017 at 6:47 AM Azoff, Justin S >>>>> > wrote: >>>>> >>>>> >>>>> > On Jun 20, 2017, at 9:27 AM, Chris Chiaverini >>>>> > wrote: >>>>> > >>>>> > It seems that bro 2.5.1 is not taking the SNF_DATARING_SIZE >>>>> variable, no matter what I set it to. >>>>> > >>>>> > When at the defaults in the /etc/bro/node.cfg and with nothing >>>>> set at the shell, it still reports it is set via "userset" >>>>> instead of "default" like SNF_DESCRING_SIZE. >>>>> >>>>> Can you do this quick test using tcpdump to verify the problem is >>>>> with bro/broctl or something with the myricom driver/library? >>>>> >>>>> SNF_APP_ID=10 SNF_FLAGS=0x1 SNF_NUM_RINGS=8 SNF_DEBUG_MASK=3 >>>>> SNF_DATARING_SIZE=4294967296 SNF_DESCRING_SIZE=1073741824 tcpdump >>>>> -n -i snf0 -c 1 >>>>> >>>>> When I run that I get >>>>> >>>>> 23681 snf.0.-1 P (userset) SNF_PORTNUM = 0 >>>>> 23681 snf.0.-1 P (default) SNF_RING_ID = -1 (0xffffffff) >>>>> 23681 snf.0.-1 P (environ) SNF_NUM_RINGS = 8 (0x8) >>>>> 23681 snf.0.-1 P (default) SNF_RSS_FLAGS = 49 (0x31) >>>>> 23681 snf.0.-1 P (environ) SNF_DATARING_SIZE = 4294967296 >>>>> (0x100000000) (4096.0 MiB) >>>>> 23681 snf.0.-1 P (environ) SNF_DESCRING_SIZE = 1073741824 >>>>> (0x40000000) (1024.0 MiB) >>>>> 23681 snf.0.-1 P (userset) SNF_FLAGS = 1 (0x1) >>>>> 23681 snf.0.-1 P (environ) SNF_DEBUG_MASK = 3 (0x3) >>>>> 23681 snf.0.-1 P (default) SNF_DEBUG_FILENAME = stderr >>>>> 23681 snf.0.-1 P (environ) SNF_APP_ID = 10 (0xa) >>>>> >>>>> >>>>> >>>>> -- >>>>> - Justin Azoff >>>>> >>>>> >>>>> _______________________________________________ >>>>> Bro mailing list >>>>> bro at bro-ids.org >>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Bro mailing list >>>> bro at bro-ids.org >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From acarreno at ucsb.edu Mon Jun 26 07:41:41 2017 From: acarreno at ucsb.edu (Alejandro Carreno) Date: Mon, 26 Jun 2017 14:41:41 +0000 Subject: [Bro] Bro node.cfg not setting Myricom Sniffer10G environment variables In-Reply-To: References: <30abafd5-fae4-f1d2-0df5-c62e82e777c4@bnl.gov> <2831AF5A-E4E7-4C8C-B4FF-9838A580FA22@illinois.edu> <51cec005-586b-26d8-42e8-6f1815dc9594@bnl.gov> <58b4bc70-bb8a-685e-b7cc-a28a327b3b5e@bnl.gov> <20170621235906.GS75741@mac-822.local> Message-ID: Negative, noticed in 2.5 when 3.0.11 was released. -Alex On Mon, Jun 26, 2017 at 7:12 AM Edgmand, Craig wrote: > Does this only impact Bro 2.5.1? > > -----Original Message----- > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Seth > Hall > Sent: Friday, June 23, 2017 8:23 PM > To: Chris Chiaverini > Cc: bro > Subject: Re: [Bro] Bro node.cfg not setting Myricom Sniffer10G environment > variables > > You could also try the bro-myricom plugin from the Bro package > repository. If you have bro-pkg set up, you should be able to do this... > > bro-pkg refresh > bro-pkg install sethhall/bro-myricom > > There is documentation on how to use it here: > https://github.com/sethhall/bro-myricom > > You only configure the data ring size in it. For some reason they don't > expose the desc ring size option through their native SNF api, only the > data ring size. I'm going to go out on a limb here and guess that you may > be experiencing weird behavior because they probably want to get rid of the > desc ring size option. It makes more sense if they just auto adjust that > based on the chosen data ring size. > > .Seth > > > On Thu, Jun 22, 2017 at 5:23 PM, Chris Chiaverini > wrote: > > Rollback!!!! > > > > Myricom opened an internal ticket on their end so hopefully we will > > see a bugfix soon. > > > > Regards, > > > > Chris Chiaverini > > Cyber Security Operations > > Brookhaven National Laboratory > > Upton, New York 11973 > > > > On 06/21/2017 07:59 PM, Aashish Sharma wrote: > >> Doh! I just upgraded the myricom drivers to 3.0.11 today only :) > >> > >> Aashish > >> > >> On Wed, Jun 21, 2017 at 06:31:50PM -0400, Chris Chiaverini wrote: > >>> Alex, > >>> > >>> Thank you for this. I confirmed on my end too... rolled back to > 3.0.10 and > >>> it worked. I will let you know what Myricom comes up with, if they > will > >>> fix in next release. > >>> > >>> > >>> Regards, > >>> > >>> Chris Chiaverini > >>> Cyber Security Operations > >>> Brookhaven National Laboratory > >>> Upton, New York 11973 > >>> > >>> On 06/20/2017 11:01 AM, Chris Chiaverini wrote: > >>>> I have a support case open with them in parallel. I will report > >>>> this to them too. Maybe we'll get a fix in next minor release. > >>>> > >>>> Regards, > >>>> > >>>> Chris Chiaverini > >>>> Cyber Security Operations > >>>> Brookhaven National Laboratory > >>>> Upton, New York 11973 > >>>> On 06/20/2017 10:09 AM, Alejandro Carreno wrote: > >>>>> I noticed this behavior as well a while back after upgrading SNF > >>>>> from > >>>>> 3.0.10 to 3.0.11. Downgrading back to 3.0.10 would return the ring > >>>>> sizes to the expected values. > >>>>> > >>>>> -Alex > >>>>> > >>>>> On Tue, Jun 20, 2017 at 6:47 AM Azoff, Justin S > >>>>> > wrote: > >>>>> > >>>>> > >>>>> > On Jun 20, 2017, at 9:27 AM, Chris Chiaverini > >>>>> > wrote: > >>>>> > > >>>>> > It seems that bro 2.5.1 is not taking the SNF_DATARING_SIZE > >>>>> variable, no matter what I set it to. > >>>>> > > >>>>> > When at the defaults in the /etc/bro/node.cfg and with nothing > >>>>> set at the shell, it still reports it is set via "userset" > >>>>> instead of "default" like SNF_DESCRING_SIZE. > >>>>> > >>>>> Can you do this quick test using tcpdump to verify the problem is > >>>>> with bro/broctl or something with the myricom driver/library? > >>>>> > >>>>> SNF_APP_ID=10 SNF_FLAGS=0x1 SNF_NUM_RINGS=8 SNF_DEBUG_MASK=3 > >>>>> SNF_DATARING_SIZE=4294967296 SNF_DESCRING_SIZE=1073741824 tcpdump > >>>>> -n -i snf0 -c 1 > >>>>> > >>>>> When I run that I get > >>>>> > >>>>> 23681 snf.0.-1 P (userset) SNF_PORTNUM = 0 > >>>>> 23681 snf.0.-1 P (default) SNF_RING_ID = -1 > (0xffffffff) > >>>>> 23681 snf.0.-1 P (environ) SNF_NUM_RINGS = 8 (0x8) > >>>>> 23681 snf.0.-1 P (default) SNF_RSS_FLAGS = 49 (0x31) > >>>>> 23681 snf.0.-1 P (environ) SNF_DATARING_SIZE = 4294967296 > >>>>> (0x100000000) (4096.0 MiB) > >>>>> 23681 snf.0.-1 P (environ) SNF_DESCRING_SIZE = 1073741824 > >>>>> (0x40000000) (1024.0 MiB) > >>>>> 23681 snf.0.-1 P (userset) SNF_FLAGS = 1 (0x1) > >>>>> 23681 snf.0.-1 P (environ) SNF_DEBUG_MASK = 3 (0x3) > >>>>> 23681 snf.0.-1 P (default) SNF_DEBUG_FILENAME = stderr > >>>>> 23681 snf.0.-1 P (environ) SNF_APP_ID = 10 (0xa) > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> - Justin Azoff > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> Bro mailing list > >>>>> bro at bro-ids.org > >>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > >>>>> > >>>> > >>>> > >>>> _______________________________________________ > >>>> Bro mailing list > >>>> bro at bro-ids.org > >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > >>> _______________________________________________ > >>> Bro mailing list > >>> bro at bro-ids.org > >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > -- > Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170626/57bd6833/attachment-0001.html From valerio.click at gmx.com Tue Jun 27 09:14:46 2017 From: valerio.click at gmx.com (Valerio) Date: Tue, 27 Jun 2017 18:14:46 +0200 Subject: [Bro] Relationship between custom protocol analyzer and weird log Message-ID: Hi all, I am experiencing a strange behaviour in BRO that I am not able to troubleshoot autonomously. I developed a simple binary protocol analyzer that produces a log file of type prot1.log. If I run bro offline on a dedicated pcap it correctly outputs prot1.log with the proper record. If I run bro sniffing on an interface and I tcpreplay the pcap on the sniffed interface I get weird.log with SYN_inside_connection warning. Is weird preemting the application of my analyzer? many thanks in advance, Valerio From kir215 at email.vccs.edu Tue Jun 27 13:30:45 2017 From: kir215 at email.vccs.edu (Kyle Reidell) Date: Tue, 27 Jun 2017 16:30:45 -0400 Subject: [Bro] ERSPAN & Missing Logs Message-ID: Hello all, I am attempting to monitor a Cisco CSR1000v within AWS via ERSPAN. Through my research, I am running Bro version 2.5-147 on an AWS Linux AMI and have uploaded a pcap containing ERSPAN data which I have been able to read; however, the only log files that are being created from Bro/live traffic are the following: capture_loss stats stderr stdout weird communication As a test, I have used tcpdump to capture packets on the configured interface (mon0) which sees plenty of traffic, however, I still cannot see the corresponding logs from Bro. Any help would be greatly appreciated!! Thank you, Planearium -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170627/13684ce9/attachment.html From johanna at icir.org Tue Jun 27 14:34:41 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 27 Jun 2017 14:34:41 -0700 Subject: [Bro] Bro 2.5.1 release Message-ID: <20170627213441.u5y4wisc3kh7xbt3@wifi180.sys.ICSI.Berkeley.EDU> We are very happy to announce the release of Bro v2.51. The new version is now available for download at: https://bro.org/download/index.html Binary packages also are available at: https://bro.org/download/packages.html This release contains a number of bug fixes. Fixes include: - Better file analysis memory management - Less cluster node communication - Correct expiration of intelligence items after reinsertion - A bug in the OCSP validation code This point-release also includes a number of new features, including new file handling BIFS, support for ERSPAN, and new BroControl options. For more information see the NEWS and CHANGES files: https://www.bro.org/download/NEWS.bro.html https://www.bro.org/download/CHANGES.bro.txt Thanks to everyone who helped make this release possible. We extend special thanks to the community for their feedback and support. Johanna -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 842 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170627/e51c335b/attachment.bin From jdopheid at illinois.edu Wed Jun 28 11:42:22 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Wed, 28 Jun 2017 18:42:22 +0000 Subject: [Bro] v2.5 Bro Log Cheat Sheets Message-ID: <110199BA-43EB-4C6B-B91E-CB413946092E@illinois.edu> Corelight has made its Bro Log cheat sheets public on their Github: https://github.com/corelight/bro-cheatsheets ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From jdopheid at illinois.edu Wed Jun 28 12:11:37 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Wed, 28 Jun 2017 19:11:37 +0000 Subject: [Bro] Bro Package Manager: list of packages Message-ID: Attention Bro Community, While we?re in the process of developing a web site for the Bro Package Manager project, we?d like to share the packages we have collected so far. The package names and a short description are listed below: bro/0xxon/bro-postgresql - A PostgreSQL reader and writer for Bro. bro/0xxon/bro-sumstats-counttable - Two-dimensional buckets for sumstats (count occurences per $str). bro/corelight/bro-long-connections - Find and log long-lived connections into a "conn_long" log. bro/dopheide/bro_notice_correlation - Adds support for multi-notice correlation. bro/dopheide/venom (installed: master) - https://security.web.cern.ch/security/venom.shtml bro/hhzzk/dns-tunnels - Detect DNS Tunnels attack. bro/initconf/CVE-2017-5638_struts.git bro/initconf/phish-analysis.git bro/initconf/scan-NG bro/j-gras/add-json - Additional JSON-logging for Bro. bro/j-gras/bro-af_packet-plugin - This plugin provides native AF_Packet support for Bro. bro/j-gras/intel-extensions - Extensions for Bro's intelligence framework. bro/joesecurity/Joe-Sandbox-Bro - JoeSandbox-Bro extracts files from your internet connection and analyzes them automatically on Joe Sandbox. bro/jonzeolla/scan-sampling - Modified version of scan.bro to add destination IP sampling. bro/jsiwek/bro-test-package - An example Bro package for testing purposes. bro/jswaro/tcprs - TCP Retransmission and State Analyzer plugin for Bro. bro/ncsa/bro-interface-setup - A broctl plugin that helps you setup capture interfaces bro/pgaulon/bro-notice-slack - Bro Notices through Slack webhook bro/scebro/ldap-analyzer - LDAP write operations analyzer for Bro. bro/sethhall/bro-myricom - Packet source plugin that provides native Myricom SNF v3+v4 support. bro/sethhall/credit-card-exposure - Detect credit card numbers in HTTP and SMTP with Bro. bro/sethhall/domain-tld bro/sethhall/ssn-exposure - Detect US Social Security numbers in HTTP and SMTP with Bro. bro/srozb/dns_axfr - Find and notice DNS zone transfer attempts. bro/theflakes/bro-large_uploads - Raise notices on outgoing files over X bytes in size. To learn how to use the Package Manager, see our documentation here: http://bro-package-manager.readthedocs.io/en/stable/index.html ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From seth at corelight.com Fri Jun 30 07:51:46 2017 From: seth at corelight.com (Seth Hall) Date: Fri, 30 Jun 2017 10:51:46 -0400 Subject: [Bro] ERSPAN & Missing Logs In-Reply-To: References: Message-ID: If you could send me a few packets of traffic captured with tcpdump I could take a look for you (I wrote the RSPAN support). Sometimes it's hard to verify that parsers will always work with all versions of protocols and all usage of a protocol. .Seth On Tue, Jun 27, 2017 at 4:30 PM, Kyle Reidell wrote: > Hello all, > > I am attempting to monitor a Cisco CSR1000v within AWS via ERSPAN. Through > my research, I am running Bro version 2.5-147 on an AWS Linux AMI and have > uploaded a pcap containing ERSPAN data which I have been able to read; > however, the only log files that are being created from Bro/live traffic are > the following: > > capture_loss > stats > stderr > stdout > weird > communication > > As a test, I have used tcpdump to capture packets on the configured > interface (mon0) which sees plenty of traffic, however, I still cannot see > the corresponding logs from Bro. > > Any help would be greatly appreciated!! > > > Thank you, > Planearium > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com