From wren3 at illinois.edu Tue Nov 3 13:57:13 2015 From: wren3 at illinois.edu (Ren, Wenyu) Date: Tue, 3 Nov 2015 21:57:13 +0000 Subject: [Bro] Is there existing signatures for modbus and dnp3 Message-ID: Dear all, I am wondering whether there is some existing signatures of Bro for Modbus and DNP3. I found something named quickdraw, which are signatures for Modbus and DNP3. But it is for Snort and Bro does not support signature for Snort anymore as far as I know. Does anyone know some similar signature available for Bro? Thanks a lot. Wenyu From anthony.kasza at gmail.com Tue Nov 3 15:31:58 2015 From: anthony.kasza at gmail.com (anthony kasza) Date: Tue, 3 Nov 2015 15:31:58 -0800 Subject: [Bro] Is there existing signatures for modbus and dnp3 In-Reply-To: References: Message-ID: Here are the sigs for dnp3 < https://github.com/bro/bro/blob/master/scripts/base/protocols/dnp3/dpd.sig>. -AK On Nov 3, 2015 2:04 PM, "Ren, Wenyu" wrote: > Dear all, > > I am wondering whether there is some existing signatures of Bro for Modbus > and DNP3. I found something named quickdraw, which are signatures for > Modbus and DNP3. But it is for Snort and Bro does not support signature for > Snort anymore as far as I know. Does anyone know some similar signature > available for Bro? Thanks a lot. > > Wenyu > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151103/0dcc31de/attachment.html From daniel.guerra69 at gmail.com Tue Nov 3 16:59:05 2015 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Wed, 4 Nov 2015 01:59:05 +0100 Subject: [Bro] Bro -> Elasticsearch -> Kibana4beta -> GeoLocation In-Reply-To: References: <1DA686BF-8B1F-4E85-B6CF-826E4F7A3E4D@gmail.com> <8129DD67-D968-415D-A564-1F4AF96C70C9@icir.org> Message-ID: Hi All, The problem was solved like this. The geoip script adds the geo_location With the proper mapping kibana shows geoip data ;). Check https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/ ##! Add geo_location for the originator and responder of a connection ##! to the connection logs. module Conn; export { redef record Conn::Info += { geo_location: string &optional &log; }; } event connection_state_remove(c: connection) { local resp_loc = lookup_location(c$id$resp_h); if (resp_loc?$longitude && resp_loc?$latitude) #geo location is just a cat lat,long c$conn$geo_location= cat(resp_loc$latitude,",",resp_loc$longitude); } add the mapping before reading data with curl -XPUT elasticsearch:9200/_template/fixstrings_bro -d '{ "template": "bro-*", "mappings" : { "conn" : { "geo_location" : { "type" : "geo_point? } } }}' > On 30 Oct 2015, at 19:25, Daniel Guerra wrote: > > The funny thing is that elasticsearch stores the data internal > like the bro output is. > > quote from the object document > Internally, this document is indexed as a simple, flat list of key-value pairs, something like this: > > { > "region": "US", > "manager.age": 30, > "manager.name.first": "John", > "manager.name.last": "Smith" > } > Maybe this is an elasticsearch problem ? > To make it all work ElasticSearch.cc has to change to do > the geopoint mapping. And maybe stop analyse strings like > user_agent to avoid chopping of the result in the first word. > This could be solved by using url formatted strings you want > to show in graphs etc (no spaces). > The last thing is some naming collisions elasticsearch is > confused about, like version in ssh & socks, but thats easy > to change in their main scripts. > > Daniel >> On 30 Oct 2015, at 14:46, Seth Hall > wrote: >> >> >>> On Oct 29, 2015, at 9:33 PM, Daniel Guerra > wrote: >>> >>> I use the elasticsearch plugin in bro. I know logstash works fine but its >>> very cpu intensive. Thanx anyway. >> >> Technically it can be done, but it would require changes to the JSON formatter (in the core). This is actually a pretty reasonable request (and I like the idea a lot!). It might not be too much work to implement it, it just needs to be done. >> >> .Seth >> >> -- >> Seth Hall >> International Computer Science Institute >> (Bro) because everyone has a network >> http://www.bro.org/ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151104/fd7f7ade/attachment.html From wren3 at illinois.edu Tue Nov 3 18:23:35 2015 From: wren3 at illinois.edu (Ren, Wenyu) Date: Wed, 4 Nov 2015 02:23:35 +0000 Subject: [Bro] Is there existing signatures for modbus and dnp3 In-Reply-To: References: , Message-ID: Hi Anthony, Thanks for your reply. I am sorry but I probably did not make myself clear. I am not looking for signature that help you to identify Modbus and DNP3 packets. Instead, I am looking for signatures that help you to detect attacks on Modbus and DNP3. Do you know any signature like that available for Bro? Thanks a lot. Wenyu ________________________________ From: anthony kasza [anthony.kasza at gmail.com] Sent: Tuesday, November 03, 2015 5:31 PM To: Ren, Wenyu Cc: bro at bro.org Subject: Re: [Bro] Is there existing signatures for modbus and dnp3 Here are the sigs for dnp3 >. -AK On Nov 3, 2015 2:04 PM, "Ren, Wenyu" > wrote: Dear all, I am wondering whether there is some existing signatures of Bro for Modbus and DNP3. I found something named quickdraw, which are signatures for Modbus and DNP3. But it is for Snort and Bro does not support signature for Snort anymore as far as I know. Does anyone know some similar signature available for Bro? Thanks a lot. Wenyu _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151104/db0c1646/attachment-0001.html From ajamshed at ICSI.Berkeley.EDU Tue Nov 3 21:44:36 2015 From: ajamshed at ICSI.Berkeley.EDU (Muhammad Asim Jamshed) Date: Tue, 3 Nov 2015 21:44:36 -0800 Subject: [Bro] Packet bricks Message-ID: Hi, Packet bricks is a Linux/FreeBSD networking tool that can be used to load balance or duplicate ingress traffic across applications and/or clusters. Packet bricks uses netmap I/O module for capturing traffic and netmap pipes to forward packets to target applications. The experimental version of packet-bricks is now available on github (https://github.com/bro/packet-bricks ). You can find more details in the README file. Please email any feedback or bug reports to ajamshed at icsi.berkeley.edu Thanks, ?Asim -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151103/77ebd7f6/attachment.html From daniel.guerra69 at gmail.com Wed Nov 4 05:42:22 2015 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Wed, 4 Nov 2015 14:42:22 +0100 Subject: [Bro] Bro -> Elasticsearch -> Kibana4beta -> GeoLocation In-Reply-To: References: <1DA686BF-8B1F-4E85-B6CF-826E4F7A3E4D@gmail.com> <8129DD67-D968-415D-A564-1F4AF96C70C9@icir.org> Message-ID: <44F2E6F0-EDDA-45C7-B4D5-C110A7C84F47@gmail.com> Hi, The docker image is currently not working (should have checked first). It works fine on my laptop but docker has some troubles with it. You can build the dockerfile yourself https://github.com/danielguerra69/bro-debian-elasticsearch > On 04 Nov 2015, at 01:59, Daniel Guerra wrote: > > Hi All, > > The problem was solved like this. > The geoip script adds the geo_location > With the proper mapping kibana shows > geoip data ;). > Check > https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/ > > ##! Add geo_location for the originator and responder of a connection > ##! to the connection logs. > > module Conn; > > export > { > redef record Conn::Info += > { > geo_location: string &optional &log; > }; > } > > event connection_state_remove(c: connection) > { > local resp_loc = lookup_location(c$id$resp_h); > if (resp_loc?$longitude && resp_loc?$latitude) > #geo location is just a cat lat,long > c$conn$geo_location= cat(resp_loc$latitude,",",resp_loc$longitude); > } > > add the mapping before reading data with > curl -XPUT elasticsearch:9200/_template/fixstrings_bro -d '{ > "template": "bro-*", > "mappings" : { > "conn" : { > "geo_location" : { "type" : "geo_point? } > } > }}' > >> On 30 Oct 2015, at 19:25, Daniel Guerra > wrote: >> >> The funny thing is that elasticsearch stores the data internal >> like the bro output is. >> >> quote from the object document >> Internally, this document is indexed as a simple, flat list of key-value pairs, something like this: >> >> { >> "region": "US", >> "manager.age": 30, >> "manager.name.first": "John", >> "manager.name.last": "Smith" >> } >> Maybe this is an elasticsearch problem ? >> To make it all work ElasticSearch.cc has to change to do >> the geopoint mapping. And maybe stop analyse strings like >> user_agent to avoid chopping of the result in the first word. >> This could be solved by using url formatted strings you want >> to show in graphs etc (no spaces). >> The last thing is some naming collisions elasticsearch is >> confused about, like version in ssh & socks, but thats easy >> to change in their main scripts. >> >> Daniel >>> On 30 Oct 2015, at 14:46, Seth Hall > wrote: >>> >>> >>>> On Oct 29, 2015, at 9:33 PM, Daniel Guerra > wrote: >>>> >>>> I use the elasticsearch plugin in bro. I know logstash works fine but its >>>> very cpu intensive. Thanx anyway. >>> >>> Technically it can be done, but it would require changes to the JSON formatter (in the core). This is actually a pretty reasonable request (and I like the idea a lot!). It might not be too much work to implement it, it just needs to be done. >>> >>> .Seth >>> >>> -- >>> Seth Hall >>> International Computer Science Institute >>> (Bro) because everyone has a network >>> http://www.bro.org/ >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151104/4c7ef25d/attachment.html From anthony.kasza at gmail.com Wed Nov 4 10:38:12 2015 From: anthony.kasza at gmail.com (anthony kasza) Date: Wed, 4 Nov 2015 10:38:12 -0800 Subject: [Bro] Is there existing signatures for modbus and dnp3 In-Reply-To: References: Message-ID: In general, Bro doesn't use signatures for attack determinations. If you look at the way Bro identifies things like SQLi you can see Bro uses signatures only to assist its behavioral approach (implemented in policy scripts) to identify attack situations. https://www.bro.org/sphinx/_downloads/detect-sqli.bro -AK On Nov 3, 2015 6:23 PM, "Ren, Wenyu" wrote: > Hi Anthony, > > Thanks for your reply. I am sorry but I probably did not make myself > clear. I am not looking for signature that help you to identify Modbus and > DNP3 packets. Instead, I am looking for signatures that help you to detect > attacks on Modbus and DNP3. Do you know any signature like that available > for Bro? Thanks a lot. > > Wenyu > ------------------------------ > *From:* anthony kasza [anthony.kasza at gmail.com] > *Sent:* Tuesday, November 03, 2015 5:31 PM > *To:* Ren, Wenyu > *Cc:* bro at bro.org > *Subject:* Re: [Bro] Is there existing signatures for modbus and dnp3 > > Here are the sigs for dnp3 < > https://github.com/bro/bro/blob/master/scripts/base/protocols/dnp3/dpd.sig > > >. > > -AK > On Nov 3, 2015 2:04 PM, "Ren, Wenyu" wrote: > >> Dear all, >> >> I am wondering whether there is some existing signatures of Bro for >> Modbus and DNP3. I found something named quickdraw, which are signatures >> for Modbus and DNP3. But it is for Snort and Bro does not support signature >> for Snort anymore as far as I know. Does anyone know some similar signature >> available for Bro? Thanks a lot. >> >> Wenyu >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151104/ea2b2654/attachment-0001.html From monahbaki at gmail.com Thu Nov 5 04:52:04 2015 From: monahbaki at gmail.com (Monah Baki) Date: Thu, 5 Nov 2015 07:52:04 -0500 Subject: [Bro] Bro +Splunk Message-ID: Hi all, Any good documentation for newbies as to how to send bro logs to a remote splunk server? What's the requirements on both sides and what files needs to be touched on the bro to send the logs to the remote splunk server. I know I installed from the splunk app the "Splunk add on for bro ids" Thanks Monah From pkelley at hyperionavenue.com Thu Nov 5 07:18:36 2015 From: pkelley at hyperionavenue.com (Patrick Kelley) Date: Thu, 5 Nov 2015 07:18:36 -0800 Subject: [Bro] Bro +Splunk In-Reply-To: References: Message-ID: Yes! Use the Splunk Universal Forwarder and monitor the "/usr/local/bro/logs/current" folder. Make sure you configure Splunk to receive the data. This can be done under settings. *Instructions* Install the forwarder - http://www.splunk.com/en_us/download/universal-forwarder.html Add the location of your Splunk server - ./splunk add forward-server 172.0.0.20:9997 Add the monitor command - ./splunk add monitor //usr/local/bro/logs/current That's it. On Thu, Nov 5, 2015 at 4:52 AM, Monah Baki wrote: > Hi all, > > Any good documentation for newbies as to how to send bro logs to a > remote splunk server? > What's the requirements on both sides and what files needs to be > touched on the bro to send the logs to the remote splunk server. > I know I installed from the splunk app the "Splunk add on for bro ids" > > Thanks > Monah > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Patrick Kelley, CEH Hyperion Avenue Labs http://www.hyperionavenue.com 951.291.8310 *The limit to which you have accepted being comfortable is the limit to which you have grown. Accept new challenges as an opportunity to enrich yourself and not as a point of potential failure.* [image: hal_logo] -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151105/fce3d0e4/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 12155 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151105/fce3d0e4/attachment-0002.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2015-11-05 at 7.18.03 AM.png Type: image/png Size: 29553 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151105/fce3d0e4/attachment-0003.bin From latt0050 at umn.edu Thu Nov 5 07:21:10 2015 From: latt0050 at umn.edu (Brandon Lattin) Date: Thu, 5 Nov 2015 09:21:10 -0600 Subject: [Bro] Bro +Splunk In-Reply-To: References: Message-ID: Splunk universal forwarders monitoring the files you're interested (those in the ./current directory) on the Bro cluster master. The Splunk TA for Bro on the search head(s), indexer(s), and on the Bro cluster master (https://splunkbase.splunk.com/app/1617/). I highly suggest forking it and removing the Splunk_TA_Bro/default/inputs.conf and building your own. That should get you started. On Thu, Nov 5, 2015 at 6:52 AM, Monah Baki wrote: > Hi all, > > Any good documentation for newbies as to how to send bro logs to a > remote splunk server? > What's the requirements on both sides and what files needs to be > touched on the bro to send the logs to the remote splunk server. > I know I installed from the splunk app the "Splunk add on for bro ids" > > Thanks > Monah > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Brandon Lattin Security Analyst University of Minnesota - University Information Security Office: 612-626-6672 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151105/e64de3c7/attachment.html From monahbaki at gmail.com Thu Nov 5 07:31:24 2015 From: monahbaki at gmail.com (Monah Baki) Date: Thu, 5 Nov 2015 10:31:24 -0500 Subject: [Bro] Bro +Splunk In-Reply-To: References: Message-ID: Hi Patrick, http://www.splunk.com/en_us/download/universal-forwarder.html Can't foind a link to download the universal forwarder. Is it free? Thanks Monah On Thu, Nov 5, 2015 at 10:18 AM, Patrick Kelley wrote: > Yes! Use the Splunk Universal Forwarder and monitor the > "/usr/local/bro/logs/current" folder. Make sure you configure Splunk to > receive the data. This can be done under settings. > > *Instructions* > > Install the forwarder - > > http://www.splunk.com/en_us/download/universal-forwarder.html > > Add the location of your Splunk server - > ./splunk add forward-server 172.0.0.20:9997 > > Add the monitor command - > ./splunk add monitor //usr/local/bro/logs/current > > That's it. > > On Thu, Nov 5, 2015 at 4:52 AM, Monah Baki wrote: > >> Hi all, >> >> Any good documentation for newbies as to how to send bro logs to a >> remote splunk server? >> What's the requirements on both sides and what files needs to be >> touched on the bro to send the logs to the remote splunk server. >> I know I installed from the splunk app the "Splunk add on for bro ids" >> >> Thanks >> Monah >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > > > -- > > Patrick Kelley, CEH > Hyperion Avenue Labs > http://www.hyperionavenue.com > 951.291.8310 > > *The limit to which you have accepted being comfortable is the limit to > which you have grown. Accept new challenges as an opportunity to enrich > yourself and not as a point of potential failure.* > > [image: hal_logo] > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151105/bb95bece/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 12155 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151105/bb95bece/attachment-0001.bin From pkelley at hyperionavenue.com Thu Nov 5 07:33:19 2015 From: pkelley at hyperionavenue.com (Patrick Kelley) Date: Thu, 5 Nov 2015 07:33:19 -0800 Subject: [Bro] Bro +Splunk In-Reply-To: References: Message-ID: Yes, it is. Just register for a Splunk account, which is free. https://www.splunk.com/page/sign_up?redirecturl=http://www.splunk.com/en_us/download/universal-forwarder.html Once you register using the link above, it should send you to the free download. On Thu, Nov 5, 2015 at 7:31 AM, Monah Baki wrote: > Hi Patrick, > > http://www.splunk.com/en_us/download/universal-forwarder.html Can't foind > a link to download the universal forwarder. Is it free? > > Thanks > Monah > > On Thu, Nov 5, 2015 at 10:18 AM, Patrick Kelley < > pkelley at hyperionavenue.com> wrote: > >> Yes! Use the Splunk Universal Forwarder and monitor the >> "/usr/local/bro/logs/current" folder. Make sure you configure Splunk to >> receive the data. This can be done under settings. >> >> *Instructions* >> >> Install the forwarder - >> >> http://www.splunk.com/en_us/download/universal-forwarder.html >> >> Add the location of your Splunk server - >> ./splunk add forward-server 172.0.0.20:9997 >> >> Add the monitor command - >> ./splunk add monitor //usr/local/bro/logs/current >> >> That's it. >> >> On Thu, Nov 5, 2015 at 4:52 AM, Monah Baki wrote: >> >>> Hi all, >>> >>> Any good documentation for newbies as to how to send bro logs to a >>> remote splunk server? >>> What's the requirements on both sides and what files needs to be >>> touched on the bro to send the logs to the remote splunk server. >>> I know I installed from the splunk app the "Splunk add on for bro ids" >>> >>> Thanks >>> Monah >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >> >> >> >> -- >> >> Patrick Kelley, CEH >> Hyperion Avenue Labs >> http://www.hyperionavenue.com >> 951.291.8310 >> >> *The limit to which you have accepted being comfortable is the limit to >> which you have grown. Accept new challenges as an opportunity to enrich >> yourself and not as a point of potential failure.* >> >> [image: hal_logo] >> >> > -- Patrick Kelley, CEH Hyperion Avenue Labs http://www.hyperionavenue.com 951.291.8310 *The limit to which you have accepted being comfortable is the limit to which you have grown. Accept new challenges as an opportunity to enrich yourself and not as a point of potential failure.* [image: hal_logo] -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151105/9972bf7f/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 12155 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151105/9972bf7f/attachment.bin From bengen--bro at hilluzination.de Thu Nov 5 08:22:48 2015 From: bengen--bro at hilluzination.de (Hilko Bengen) Date: Thu, 05 Nov 2015 17:22:48 +0100 Subject: [Bro] SSL/TLS parameters used in Bro, Broccoli Message-ID: <87d1voe7qf.fsf@msgid.hilluzination.de> Hi, in Debian/unstable (and probably elsewhere) support for SSLv3 has been dropped: SSLv3_method(), SSLv3_server_method(), SSLv3_client_method() are no longer available. This leads to build failures with at least bro and broccoli (see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=804104). What should SSLv3_method() in SSL_CTX_new(SSLv3_method()) be replaced with? Would SSLv23_method() which is still available and (according to the OpenSSL documentation) means support for SSLv2, 3 and TLSv1, 1.1, 1.2 appropriate? Cheers, -Hilko From seth at icir.org Thu Nov 5 09:09:47 2015 From: seth at icir.org (Seth Hall) Date: Thu, 5 Nov 2015 12:09:47 -0500 Subject: [Bro] SSL/TLS parameters used in Bro, Broccoli In-Reply-To: <87d1voe7qf.fsf@msgid.hilluzination.de> References: <87d1voe7qf.fsf@msgid.hilluzination.de> Message-ID: <94089807-5588-4BDF-875D-E3CF3AAA8ECF@icir.org> > On Nov 5, 2015, at 11:22 AM, Hilko Bengen wrote: > > What should SSLv3_method() in SSL_CTX_new(SSLv3_method()) be replaced > with? Would SSLv23_method() which is still available and (according to > the OpenSSL documentation) means support for SSLv2, 3 and TLSv1, 1.1, > 1.2 appropriate? We have a ticket related to this: https://bro-tracker.atlassian.net/browse/BIT-1499 .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From bengen--bro at hilluzination.de Thu Nov 5 09:19:46 2015 From: bengen--bro at hilluzination.de (Hilko Bengen) Date: Thu, 05 Nov 2015 18:19:46 +0100 Subject: [Bro] SSL/TLS parameters used in Bro, Broccoli In-Reply-To: <94089807-5588-4BDF-875D-E3CF3AAA8ECF@icir.org> (Seth Hall's message of "Thu, 5 Nov 2015 12:09:47 -0500") References: <87d1voe7qf.fsf@msgid.hilluzination.de> <94089807-5588-4BDF-875D-E3CF3AAA8ECF@icir.org> Message-ID: <87r3k4jrd9.fsf@msgid.hilluzination.de> * Seth Hall: > We have a ticket related to this: > https://bro-tracker.atlassian.net/browse/BIT-1499 Alright, I will be using those patches for the Debian packages of bro and broccoli. Thank you for answering so quickly! Cheers, -Hilko From troyj at maine.edu Thu Nov 5 20:10:54 2015 From: troyj at maine.edu (Troy Jordan) Date: Thu, 5 Nov 2015 23:10:54 -0500 Subject: [Bro] spicy docker image - type parsing oddity Message-ID: <563C284E.6050606@maine.edu> Hello, I'm uncertain if I've run into an issue peculiar to the spicy docker image (which should be the latest - 247ea5070b15), or if this is syntax problem. In a basic modbus parser (attached : .pac2, .evt, .bro and modbus trace file) , the Message definition throws an error when executing as: root# bro -r modbus_part1.3.pcap modbus.evt modbus.bro >>> struct.set __self "data" ref() :: error, operand type ref is not compatible with type ref [pass::hilti::Validator] However, ALT Message definition works fine. In fact, if I parse the data field with the same type (ie both with type CoilsPdu or both with type DinputsPdu) it works, which is puzzling. The idea is to parse different modbus function codes as different types to enable raising type-specific events. Any insights appreciated. - Troy -- Troy Jordan t r o y j @ m a i n e . e d u GIAC GCIH,GCIA ------------------------------------------------------------ Network Systems Security Analyst Information Technology Security Office University of Maine System ------------------------------------------------------------ 233 Science Building | voice: 207.561.3590 Portland, ME 04103 | fax: 509.351.3650 "As you all know, Security Is Mortals chiefest Enemy" William Shakespeare, Macbeth -------------- next part -------------- A non-text attachment was scrubbed... Name: modbus-parse-fail.tar Type: application/octet-stream Size: 10240 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151105/baaf507c/attachment.obj From troyj at maine.edu Fri Nov 6 12:39:35 2015 From: troyj at maine.edu (Troy Jordan) Date: Fri, 6 Nov 2015 15:39:35 -0500 Subject: [Bro] spicy docker image - type parsing oddity In-Reply-To: <563C284E.6050606@maine.edu> References: <563C284E.6050606@maine.edu> Message-ID: <563D1007.5040102@maine.edu> So, this is easy to fix. I figured out that once a field name gets typed, it can only appear again with the same type. Makes perfect sense. So this snippet would work: switch ( self.fcode ) { b"0x01" -> data1: CoilPDU; b"0x03" -> data2: HregisterPDU; } and so would this: switch ( self.fcode ) { b"0x01" -> data1: CoilPDU; b"0x03" -> data1: CoilPDU; } but not this one which reuses the 'data' fieldname but with different types: switch ( self.fcode ) { b"0x01" -> data: CoilPDU; b"0x03" -> data: HregisterPDU; } - Troy On 11/5/2015 11:10 PM, Troy Jordan wrote: > Hello, > > I'm uncertain if I've run into an issue peculiar to the spicy docker > image (which should be the latest - 247ea5070b15), or if this is syntax > problem. > > In a basic modbus parser (attached : .pac2, .evt, .bro and modbus trace > file) , the Message definition throws an error when executing as: > > root# bro -r modbus_part1.3.pcap modbus.evt modbus.bro > >>>> struct.set __self "data" ref() > :: error, operand type ref is not > compatible with type ref [pass::hilti::Validator] > > However, ALT Message definition works fine. In fact, if I parse the > data field with the same type (ie both with type CoilsPdu or both with > type DinputsPdu) it works, which is puzzling. > > The idea is to parse different modbus function codes as different types > to enable raising type-specific events. > > Any insights appreciated. > > - Troy > > > > > > > -- Troy Jordan t r o y j @ m a i n e . e d u GIAC GCIH,GCIA ------------------------------------------------------------ Network Systems Security Analyst Information Technology Security Office University of Maine System ------------------------------------------------------------ 233 Science Building | voice: 207.561.3590 Portland, ME 04103 | fax: 509.351.3650 "As you all know, Security Is Mortals chiefest Enemy" William Shakespeare, Macbeth --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus From tgdesrochers at gmail.com Fri Nov 6 13:13:32 2015 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Fri, 06 Nov 2015 16:13:32 -0500 Subject: [Bro] Bro with elasticsearch 2.0 Message-ID: <563d17fe.4439810a.34448.6560@mx.google.com> This may not be a question for this forum but I have raised it in the elasticsearch forum with no answers. I just upgraded my ES cluster to? elasticsearch 2.0 and it seems that elasticsearch no longer allows for dot (.) In field names and will not send that data into the cluster. This means that any info from the Intel log, x509 log, and other fields will no longer be indexed. Is there a work around for this. Is there a way to have bro print fields with underscores instead of periods or, and this may be easier, is there a way to have logstash look for any field name with dot and replace them with an underscore. As with may things upgrades in one product drives changes in others. I'm not sure the reason ES 2.0 decided that field names cannot include dots but I'd love to find a way to make this work with bro once again. Thanks From robin at icir.org Fri Nov 6 13:23:17 2015 From: robin at icir.org (Robin Sommer) Date: Fri, 6 Nov 2015 13:23:17 -0800 Subject: [Bro] spicy docker image - type parsing oddity In-Reply-To: <563D1007.5040102@maine.edu> References: <563C284E.6050606@maine.edu> <563D1007.5040102@maine.edu> Message-ID: <20151106212317.GG95541@icir.org> On Fri, Nov 06, 2015 at 15:39 -0500, Troy Jordan wrote: > So, this is easy to fix. I figured out that once a field name gets > typed, it can only appear again with the same type. Yeah, exactly. That's something the compiler should catch and give you a decent error message for but validation is still a weak spot of Spicy. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From daniel.guerra69 at gmail.com Fri Nov 6 14:28:33 2015 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Fri, 6 Nov 2015 23:28:33 +0100 Subject: [Bro] Bro with elasticsearch 2.0 In-Reply-To: <563d17fe.4439810a.34448.6560@mx.google.com> References: <563d17fe.4439810a.34448.6560@mx.google.com> Message-ID: <48FFB98B-3E32-4E1F-B880-ECFD4323454B@gmail.com> Hi, I have no problems with elastic 2.0. I have made a docker container for this check. It runs the git version and i patched some things (check the dockerfile). https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/ I just added a elasticsearch data volume that has all indexes for bro and kibana. https://hub.docker.com/r/danielguerra/bro-elasticsearch-kibana-volume/ > On 06 Nov 2015, at 22:13, Tim Desrochers wrote: > > This may not be a question for this forum but I have raised it in the elasticsearch forum with no answers. > > I just upgraded my ES cluster to elasticsearch 2.0 and it seems that elasticsearch no longer allows for dot (.) In field names and will not send that data into the cluster. This means that any info from the Intel log, x509 log, and other fields will no longer be indexed. > > Is there a work around for this. Is there a way to have bro print fields with underscores instead of periods or, and this may be easier, is there a way to have logstash look for any field name with dot and replace them with an underscore. > > As with may things upgrades in one product drives changes in others. I'm not sure the reason ES 2.0 decided that field names cannot include dots but I'd love to find a way to make this work with bro once again. > > Thanks > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151106/e328a708/attachment.html From tgdesrochers at gmail.com Fri Nov 6 14:56:07 2015 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Fri, 06 Nov 2015 17:56:07 -0500 Subject: [Bro] Bro with elasticsearch 2.0 Message-ID: <563d3009.c4c70d0a.8d7cb.7a2f@mx.google.com> An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151106/233ee5ad/attachment.html From daniel.guerra69 at gmail.com Fri Nov 6 14:59:28 2015 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Fri, 6 Nov 2015 23:59:28 +0100 Subject: [Bro] Bro with elasticsearch 2.0 In-Reply-To: <563d3009.c4c70d0a.8d7cb.7a2f@mx.google.com> References: <563d3009.c4c70d0a.8d7cb.7a2f@mx.google.com> Message-ID: <2552D9D3-4211-4116-8802-A70E330894F4@gmail.com> Check the mapping script i use. All the id.orig etc are all there. https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/elasticsearchMapping.sh > On 06 Nov 2015, at 23:56, Tim Desrochers wrote: > > Both links reference the no dot issue. And state they are working on a plugin (I hadn't seen a plugin mentioned before now) so that will most likely fix the issue. But I still do not get any logs with dots in the label. > I see errors in the /var/log/elasticsearch directory stating cannot index field error no dot in label allowed. So I know the info is not making it into the cluster it get stopped at my indexers. > > https://github.com/logstash-plugins/logstash-filter-mutate/issues/54 > > https://discuss.elastic.co/t/please-read-upgrading-logstash-and-elasticsearch-to-2-0/33791 > > Any ideas on a temp fix before a plugin can be issued? > > On Nov 6, 2015 5:28 PM, Daniel Guerra wrote: > Hi, > I have no problems with elastic 2.0. I have made a docker container for this > check. It runs the git version and i patched some things (check the dockerfile). > https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/ > I just added a elasticsearch data volume that has all indexes for bro and kibana. > https://hub.docker.com/r/danielguerra/bro-elasticsearch-kibana-volume/ > > >> On 06 Nov 2015, at 22:13, Tim Desrochers > wrote: >> >> This may not be a question for this forum but I have raised it in the elasticsearch forum with no answers. >> >> I just upgraded my ES cluster to elasticsearch 2.0 and it seems that elasticsearch no longer allows for dot (.) In field names and will not send that data into the cluster. This means that any info from the Intel log, x509 log, and other fields will no longer be indexed. >> >> Is there a work around for this. Is there a way to have bro print fields with underscores instead of periods or, and this may be easier, is there a way to have logstash look for any field name with dot and replace them with an underscore. >> >> As with may things upgrades in one product drives changes in others. I'm not sure the reason ES 2.0 decided that field names cannot include dots but I'd love to find a way to make this work with bro once again. >> >> Thanks >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151106/cce3dee0/attachment-0001.html From tgdesrochers at gmail.com Fri Nov 6 15:22:40 2015 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Fri, 06 Nov 2015 18:22:40 -0500 Subject: [Bro] Bro with elasticsearch 2.0 Message-ID: <563d3642.8b38810a.dbd69.7f86@mx.google.com> An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151106/f43d7a2c/attachment.html From dnj0496 at gmail.com Fri Nov 6 16:50:07 2015 From: dnj0496 at gmail.com (Dk Jack) Date: Fri, 6 Nov 2015 16:50:07 -0800 Subject: [Bro] counters Message-ID: Hi, I am new to bro and I am writing a plugin. How does one go about adding counters. In my c++ code, I want to add counters, for things like number connections, bytes rcvd/sent, protocol parsing errors etc. I want to access them on demand i.e. this is mainly for debugging. I don't want to write disk. I couldn't find any examples that do this and my google search didn't yield any satisfactory answers either. Would appreciate it if someone can point me the correct direction on how to do this in bro. Thanks. Dnj -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151106/1923d94f/attachment.html From anthony.kasza at gmail.com Fri Nov 6 17:45:02 2015 From: anthony.kasza at gmail.com (anthony kasza) Date: Fri, 6 Nov 2015 17:45:02 -0800 Subject: [Bro] counters In-Reply-To: References: Message-ID: Most of the things you mentioned are already available to scriptland from the C++ core. A little Bro script can accomplish what you are asking. For example, declare a global variable and increment it inside a new_connection event, then periodically print to screen. -AK On Nov 6, 2015 4:52 PM, "Dk Jack" wrote: > Hi, > I am new to bro and I am writing a plugin. How does one go about adding > counters. > In my c++ code, I want to add counters, for things like number > connections, bytes rcvd/sent, > protocol parsing errors etc. I want to access them on demand i.e. this is > mainly for debugging. > I don't want to write disk. I couldn't find any examples that do this and > my google search didn't > yield any satisfactory answers either. Would appreciate it if someone can > point me the correct > direction on how to do this in bro. Thanks. > > Dnj > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151106/3d0db893/attachment.html From sanjuanswan at gmail.com Sat Nov 7 07:27:16 2015 From: sanjuanswan at gmail.com (Jay Swan) Date: Sat, 7 Nov 2015 08:27:16 -0700 Subject: [Bro] Bro with elasticsearch 2.0 In-Reply-To: <563d3642.8b38810a.dbd69.7f86@mx.google.com> References: <563d3642.8b38810a.dbd69.7f86@mx.google.com> Message-ID: This isn't directly relevant to the original question, but it bit me: Elasticsearch field names that contain a "." require extra work if you're querying them from one of the language APIs that uses the dot as a dereference character. I wrote a blog post about this a while back: https://jayswan.github.io/2015/03/01/searching-raw-fields-with-python/ If you plan to do a lot of work with Bro logs in Elasticsearch from the language clients, it's a good thing to be aware of. Jay On Fri, Nov 6, 2015 at 4:22 PM, Tim Desrochers wrote: > Yes mine are there too. I should clarify more. It seems to be an issue > with fields that have a dot as the last special character. So "seen.where" > from Intel or "basic_constraints.ca" from x509. Those fields don't seem > to be indexed in the new version. > On Nov 6, 2015 5:59 PM, Daniel Guerra wrote: > > Check the mapping script i use. All the id.orig etc are all there. > > > https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/elasticsearchMapping.sh > > On 06 Nov 2015, at 23:56, Tim Desrochers wrote: > > Both links reference the no dot issue. And state they are working on a > plugin (I hadn't seen a plugin mentioned before now) so that will most > likely fix the issue. But I still do not get any logs with dots in the > label. > I see errors in the /var/log/elasticsearch directory stating cannot index > field error no dot in label allowed. So I know the info is not making it > into the cluster it get stopped at my indexers. > > https://github.com/logstash-plugins/logstash-filter-mutate/issues/54 > > > https://discuss.elastic.co/t/please-read-upgrading-logstash-and-elasticsearch-to-2-0/33791 > > Any ideas on a temp fix before a plugin can be issued? > On Nov 6, 2015 5:28 PM, Daniel Guerra wrote: > > Hi, > I have no problems with elastic 2.0. I have made a docker container for > this > check. It runs the git version and i patched some things (check the > dockerfile). > https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/ > I just added a elasticsearch data volume that has all indexes for bro and > kibana. > https://hub.docker.com/r/danielguerra/bro-elasticsearch-kibana-volume/ > > > On 06 Nov 2015, at 22:13, Tim Desrochers wrote: > > This may not be a question for this forum but I have raised it in the > elasticsearch forum with no answers. > > I just upgraded my ES cluster to elasticsearch 2.0 and it seems that > elasticsearch no longer allows for dot (.) In field names and will not send > that data into the cluster. This means that any info from the Intel log, > x509 log, and other fields will no longer be indexed. > > Is there a work around for this. Is there a way to have bro print fields > with underscores instead of periods or, and this may be easier, is there a > way to have logstash look for any field name with dot and replace them with > an underscore. > > As with may things upgrades in one product drives changes in others. I'm > not sure the reason ES 2.0 decided that field names cannot include dots but > I'd love to find a way to make this work with bro once again. > > Thanks > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151107/2f06ccec/attachment-0001.html From troyj at maine.edu Sun Nov 8 13:12:24 2015 From: troyj at maine.edu (Troy Jordan) Date: Sun, 8 Nov 2015 16:12:24 -0500 Subject: [Bro] exercising binpac++/spicy parsers In-Reply-To: <20150313143919.GB64511@Beezling.local> References: <5502CC83.9030701@maine.edu> <20150313143919.GB64511@Beezling.local> Message-ID: <563FBAB8.1000408@maine.edu> (reviving this old thread) Is the method for generating log output the same with spicy parsers as it is for binpac parsers? Would the code to do so reside in .bro file? - Troy On 3/13/2015 10:39 AM, Johanna Amann wrote: > Hello Troy, > >> There are some hilti-based parsers in the Bro docker image. When I run >> the pcaps for BACnet (/opt/hilti/bro/tests/Traces/bacnet/*.pcap) through >> Bro (eg bro -r NPDU.pcap) , no event logs are produced in >> /usr/local/bro/logs). > > You have to load the applicable scripts and pac files for Bro to be able > to parse these protocols. Which files you have to load depends a bit on > the protocol. The easiest way is to look at the tests that should be there > for each of the protocols that is to look into the tests directory in > hilti/bro/tests. In there, pac2/bacnet/npdu_nlmessages.bro shows that to > parse the NPDUs, it loads bacnet.evt and bacnet.bro out of the hilti > distribution and then defines a few custom events for output. > > Generally, none of the spicy parsers come with the Bro scripts to generate > log output -- the parsers just create events. For some of the parsers > replacing protocols (like DNS or HTTP), the events might be similar enough > to the events emitted by the binpac parsers to already generate logs. For > all new protocols implemented by spicy, you would first have to create > such scripts. > > I hope that helps, > Johanna > -- Troy Jordan t r o y j @ m a i n e . e d u GIAC GCIH,GCIA ------------------------------------------------------------ Network Systems Security Analyst Information Technology Security Office University of Maine System ------------------------------------------------------------ 233 Science Building | voice: 207.561.3590 Portland, ME 04103 | fax: 509.351.3650 "As you all know, Security Is Mortals chiefest Enemy" William Shakespeare, Macbeth From daniel.guerra69 at gmail.com Mon Nov 9 13:24:37 2015 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Mon, 9 Nov 2015 22:24:37 +0100 Subject: [Bro] Bro with elasticsearch 2.0 In-Reply-To: References: <563d3642.8b38810a.dbd69.7f86@mx.google.com> Message-ID: <36AD3DB4-19C9-42EE-B989-A001D104118C@gmail.com> It has some interesting changes https://www.elastic.co/downloads/past-releases/elasticsearch-2-0-0 I?m having trouble with 2.0 too but it works great with 1.7. > On 07 Nov 2015, at 16:27, Jay Swan wrote: > > This isn't directly relevant to the original question, but it bit me: Elasticsearch field names that contain a "." require extra work if you're querying them from one of the language APIs that uses the dot as a dereference character. I wrote a blog post about this a while back: > > https://jayswan.github.io/2015/03/01/searching-raw-fields-with-python/ > > If you plan to do a lot of work with Bro logs in Elasticsearch from the language clients, it's a good thing to be aware of. > > Jay > > On Fri, Nov 6, 2015 at 4:22 PM, Tim Desrochers > wrote: > Yes mine are there too. I should clarify more. It seems to be an issue with fields that have a dot as the last special character. So "seen.where" from Intel or "basic_constraints.ca " from x509. Those fields don't seem to be indexed in the new version. > > On Nov 6, 2015 5:59 PM, Daniel Guerra > wrote: > Check the mapping script i use. All the id.orig etc are all there. > > https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/elasticsearchMapping.sh > >> On 06 Nov 2015, at 23:56, Tim Desrochers > wrote: >> >> Both links reference the no dot issue. And state they are working on a plugin (I hadn't seen a plugin mentioned before now) so that will most likely fix the issue. But I still do not get any logs with dots in the label. >> I see errors in the /var/log/elasticsearch directory stating cannot index field error no dot in label allowed. So I know the info is not making it into the cluster it get stopped at my indexers. >> >> https://github.com/logstash-plugins/logstash-filter-mutate/issues/54 >> https://discuss.elastic.co/t/please-read-upgrading-logstash-and-elasticsearch-to-2-0/33791 >> Any ideas on a temp fix before a plugin can be issued? >> >> On Nov 6, 2015 5:28 PM, Daniel Guerra > wrote: >> Hi, >> I have no problems with elastic 2.0. I have made a docker container for this >> check. It runs the git version and i patched some things (check the dockerfile). >> https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/ >> I just added a elasticsearch data volume that has all indexes for bro and kibana. >> https://hub.docker.com/r/danielguerra/bro-elasticsearch-kibana-volume/ >> >> >>> On 06 Nov 2015, at 22:13, Tim Desrochers > wrote: >>> >>> This may not be a question for this forum but I have raised it in the elasticsearch forum with no answers. >>> >>> I just upgraded my ES cluster to elasticsearch 2.0 and it seems that elasticsearch no longer allows for dot (.) In field names and will not send that data into the cluster. This means that any info from the Intel log, x509 log, and other fields will no longer be indexed. >>> >>> Is there a work around for this. Is there a way to have bro print fields with underscores instead of periods or, and this may be easier, is there a way to have logstash look for any field name with dot and replace them with an underscore. >>> >>> As with may things upgrades in one product drives changes in others. I'm not sure the reason ES 2.0 decided that field names cannot include dots but I'd love to find a way to make this work with bro once again. >>> >>> Thanks >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151109/81458619/attachment.html From jdopheid at illinois.edu Thu Nov 12 12:40:10 2015 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Thu, 12 Nov 2015 20:40:10 +0000 Subject: [Bro] Announcing the Bro Future Fund Message-ID: <7EFD7D614A2BB84ABEA19B2CEDD2465809B7C6FB@CITESMBX5.ad.uillinois.edu> Bro Community, We're launching the Bro Future Fund. This initiative will enable us to collect more stable funding for the Bro Project, further support to the community, and better plan for the future. To learn more about our proposal and sponsorship opportunities, read our blog: http://blog.bro.org/2015/11/announcing-bro-future-fund.html Thank you for your continued support, The Bro Development Team -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151112/d30b4b46/attachment.html From muthm at upenn.edu Fri Nov 13 11:50:20 2015 From: muthm at upenn.edu (Melissa Muth) Date: Fri, 13 Nov 2015 14:50:20 -0500 Subject: [Bro] Bro performance & sizing question Message-ID: <56463EFC.2020000@upenn.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 We have a Bro cluster currently attempting to process up to 13Gbps (1.4Mpps) partitioned over two 10Gbps Gigamon network taps. Capture loss currently averages 44% - but before buying more hardware, we'd like to sanity-check our plans with folks who have already successfully sized their own installations. Currently there are two Bro hosts in the cluster, each with 20 CPU cores (3.1Ghz), 128GB memory, and Myricom cards with the Sniffer V3 driver. Each host runs a proxy, and 17 workers pinned to CPUs. The manager is running on one of the worker hosts, and logs are being written to SSD drives. We're using restrict_filters to ignore (large) flows generated by four hosts. The current plan is to buy 2 more worker hosts (same specs), as well as a NAS for storing logs after each hourly rotation. If we're capturing 56% of 13Gbps, that's 7454Mbps. Given the 34 cores used by bro, that works out to 219Mbps/core and about 3.6Gbps/host. Does that seem like expected performance, or might there be something broken somewhere? Does it seem reasonable to buy two more worker hosts (at least to handle current needs)? Any thoughts or recommendations would be much appreciated. Cheers, Melissa -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iEYEARECAAYFAlZGPvwACgkQjGIGZe3KNcl6GgCgijm+F4zbDC0rnuP8VMRa2YSi Tz8AoIPAvHBeF/R1e/C+HEIkSv2XO//L =p+4P -----END PGP SIGNATURE----- From dnj0496 at gmail.com Sat Nov 14 14:15:11 2015 From: dnj0496 at gmail.com (Dk Jack) Date: Sat, 14 Nov 2015 14:15:11 -0800 Subject: [Bro] pcap replay issue Message-ID: Hi, I am try to replay small ssl pcap and have bro analyze the packets. When I do tcpreplay on the pcap, the first time I see five ssl connections in bro ssl log. When I replay same pcap within a minute or so of the first replay, then I only see 3 connections. If I give a gap of say 5mins between the replays, then I see 5 connections in the 2nd replay too. If I use tcpreplay-edit with -s option i.e. where the source is randomized, then I see 5 connections both times even if I don't have a large delay between the two replays. Also, I see some messages in the weird.log. I've attached the pcap, ssl_with_delay.log (shows all 10 connections from two replays), ssl_no_delay.log(shows only 8 connections from 2 replays) and weird.log. Could someone explain what's going on and if there is a work around for this issue. Thanks. Dnj. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151114/5a61a04a/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: iis.pcap Type: application/octet-stream Size: 27548 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151114/5a61a04a/attachment-0004.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: ssl_no_delay.log Type: application/octet-stream Size: 2124 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151114/5a61a04a/attachment-0005.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: ssl_with_delay.log Type: application/octet-stream Size: 2508 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151114/5a61a04a/attachment-0006.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: weird.log Type: application/octet-stream Size: 656 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151114/5a61a04a/attachment-0007.obj From seth at icir.org Mon Nov 16 06:39:55 2015 From: seth at icir.org (Seth Hall) Date: Mon, 16 Nov 2015 09:39:55 -0500 Subject: [Bro] pcap replay issue In-Reply-To: References: Message-ID: > On Nov 14, 2015, at 5:15 PM, Dk Jack wrote: > > I am try to replay small ssl pcap and have bro analyze the packets. When I do tcpreplay on the pcap, the first time I see five ssl connections in bro ssl log. When I replay same pcap within a minute or so of the first replay, then I only see 3 connections. If I give a gap of say 5mins between the replays, then I see 5 connections in the 2nd replay too. I haven?t looked at your pcap, but from what you are saying and from looking at your weird.log it appears that your connections with the ephemeral ports 54169 and 54167 aren?t being shutdown correctly. Make sure they have correct tcp shutdown sequences (FINs or RSTs, etc). You are seeing things correctly after 5 minutes because that?s Bro?s default TCP timeout. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From zied.turki at outlook.com Mon Nov 16 12:28:17 2015 From: zied.turki at outlook.com (Zied Turki) Date: Mon, 16 Nov 2015 20:28:17 +0000 Subject: [Bro] Bro to detect Ransomware Message-ID: Hi all, I am new to Bro scripts and I am trying to build a platform to detect Ransomware like CyptoLocker using Bro IDS. I am wondering whether Bro mechanisms and Frameworks can be useful to detect this kind of malware. Please, has anyone tried to built some scripts to do that before ? Any ideas, please ? Many thanks, BR, Zied -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151116/928599a2/attachment.html From anthony.kasza at gmail.com Mon Nov 16 14:31:06 2015 From: anthony.kasza at gmail.com (anthony kasza) Date: Mon, 16 Nov 2015 14:31:06 -0800 Subject: [Bro] Bro to detect Ransomware In-Reply-To: References: Message-ID: Most ransomware indicators are host based. >From a network monitoring perspective there are three things I can think of which you can look for with Bro. 1) Some families of ransomware will contact STUN services to geolocate themselves so they can display a ransom message in a native language. Look for connections to these services. 2) Some families of ransomware use tor for beaconing after initial execution. Looks for connections to Tor. 3) Email spam and exploit kits are known distribution mechanisms for a good amount of ransomware. Check hashes from inbound emails against VT and ensure your users aren't visiting known EK URLs. -AK On Nov 16, 2015 3:30 PM, "Zied Turki" wrote: > Hi all, > > I am new to Bro scripts and I am trying to build a platform to detect > Ransomware like CyptoLocker using Bro IDS. > I am wondering whether Bro mechanisms and Frameworks can be useful to > detect this kind of malware. Please, has anyone tried to built some scripts > to do that before ? Any ideas, please ? > > Many thanks, > > BR, > Zied > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151116/dc7851b9/attachment.html From jazoff at illinois.edu Mon Nov 16 14:42:07 2015 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 16 Nov 2015 22:42:07 +0000 Subject: [Bro] Bro performance & sizing question In-Reply-To: <56463EFC.2020000@upenn.edu> References: <56463EFC.2020000@upenn.edu> Message-ID: <146E6019-CB71-4E02-9D8A-6A7AF05DEAA8@illinois.edu> > On Nov 13, 2015, at 2:50 PM, Melissa Muth wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > We have a Bro cluster currently attempting to process up to 13Gbps > (1.4Mpps) partitioned over two 10Gbps Gigamon network taps. > > Capture loss currently averages 44% - but before buying more hardware, > we'd like to sanity-check our plans with folks who have already > successfully sized their own installations. > > Currently there are two Bro hosts in the cluster, each with 20 CPU > cores (3.1Ghz), 128GB memory, and Myricom cards with the Sniffer V3 > driver. Each host runs a proxy, and 17 workers pinned to CPUs. The > manager is running on one of the worker hosts, and logs are being > written to SSD drives. We're using restrict_filters to ignore (large) > flows generated by four hosts. > Are those 20 real cores or 10 cores with hyperthreading? We have some tests planned to further test this, but I think most people disable hypherthreading or don't pin workers to the 'extra' cores. If you are running 17 workers on 10 real cores, that could lead to problems. > The current plan is to buy 2 more worker hosts (same specs), as well > as a NAS for storing logs after each hourly rotation. > > If we're capturing 56% of 13Gbps, that's 7454Mbps. Given the 34 cores > used by bro, that works out to 219Mbps/core and about 3.6Gbps/host. That's not that an extreme amount of traffic, but 44% loss does sound a bit high. What does broctl netstats report? One thing to watch out for is that the myricom driver reports capture loss across the entire ring, so the dropped amount needs to be divided by the number of worker processes. Step one should be to see if netstats reports a similar level of loss. If netstats is reporting something closer to 1-5% loss, you could have a problem elsewhere. If netstats agrees with capstats, then the workers are definitely not keeping up. > Does that seem like expected performance, or might there be something > broken somewhere? Does it seem reasonable to buy two more worker hosts > (at least to handle current needs)? Hard to say.. More boxes always helps, but it can't hurt to see if things can be optimized a bit with your current hardware. If the gigamon you have is the kind that does aggregation/load balancing, you may be able to do something like send 50% as much traffic to each box to see how they would behave if you had 2 other boxes helping out. -- - Justin Azoff From robin at icir.org Mon Nov 16 14:45:04 2015 From: robin at icir.org (Robin Sommer) Date: Mon, 16 Nov 2015 14:45:04 -0800 Subject: [Bro] Spicy TR Message-ID: <20151116224504.GL91128@icir.org> FYI, a Technical Report on Spicy is now online at http://www.icir.org/robin/papers/Spicy-TR-15-004.pdf Spicy (formerly known as BinPAC++) is a next-generation parser generator that makes it easy to build parsers for network protocols, file formats, and more. It comes with a Bro interface, and is currently in prototype state. See http://www.icir.org/hilti for more. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From tgdesrochers at gmail.com Mon Nov 16 15:35:17 2015 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Mon, 16 Nov 2015 18:35:17 -0500 Subject: [Bro] [bro] known-devices.log Message-ID: I am trying to generate the known devices logs that is included in bro. In my /site/local.bro file I currently have @load policy/misc/known-devices Do I also need to include a statement to load policy/protocols/dhcp/known-devices-and-hostnames.bro? If so is it as simple as @load policy/protocols/dhcp/known-devices-and-hostnames.bro in the same local.bro file. Currently I generate a known-devices.log file. I'd like to create it to start to track mac addresses and devices. Thanks in advance -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151116/1e89e959/attachment.html From daniel.guerra69 at gmail.com Mon Nov 16 17:55:28 2015 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Tue, 17 Nov 2015 02:55:28 +0100 Subject: [Bro] Elasticsearch 2.0 dot problem Message-ID: Elasticsearch 2.0 doesn?t accept dots in fieldnames. Bro writes fieldnames with dots. As a result bro data can not be written to Elasticsearch 2.0. I have made 2 very small patches to bro/src/threading/formatters/JSON.h and bro/src/threading/formatters/JSON.cc that solves this problem. -------------- next part -------------- A non-text attachment was scrubbed... Name: JSON.cc.patch Type: application/octet-stream Size: 120 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151117/9e225306/attachment.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: JSON.h.patch Type: application/octet-stream Size: 31 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151117/9e225306/attachment-0001.obj -------------- next part -------------- Regards, Daniel From zied.turki at outlook.com Tue Nov 17 01:19:40 2015 From: zied.turki at outlook.com (Zied Turki) Date: Tue, 17 Nov 2015 09:19:40 +0000 Subject: [Bro] Bro to detect Ransomware In-Reply-To: References: , Message-ID: Hello, Thank you for the prompt reply. I will try to do that. Kind regards, Zied Date: Mon, 16 Nov 2015 14:31:06 -0800 Subject: Re: [Bro] Bro to detect Ransomware From: anthony.kasza at gmail.com To: zied.turki at outlook.com CC: bro at bro.org Most ransomware indicators are host based. From a network monitoring perspective there are three things I can think of which you can look for with Bro. 1) Some families of ransomware will contact STUN services to geolocate themselves so they can display a ransom message in a native language. Look for connections to these services. 2) Some families of ransomware use tor for beaconing after initial execution. Looks for connections to Tor. 3) Email spam and exploit kits are known distribution mechanisms for a good amount of ransomware. Check hashes from inbound emails against VT and ensure your users aren't visiting known EK URLs. -AK On Nov 16, 2015 3:30 PM, "Zied Turki" wrote: Hi all, I am new to Bro scripts and I am trying to build a platform to detect Ransomware like CyptoLocker using Bro IDS. I am wondering whether Bro mechanisms and Frameworks can be useful to detect this kind of malware. Please, has anyone tried to built some scripts to do that before ? Any ideas, please ? Many thanks, BR, Zied _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151117/0b86dc0f/attachment-0001.html From vikrant.nook at aim.com Tue Nov 17 02:05:06 2015 From: vikrant.nook at aim.com (vikrant.nook at aim.com) Date: Tue, 17 Nov 2015 21:05:06 +1100 Subject: [Bro] Bro Whitelists and Integration Message-ID: <20151117100510.DC90B2C4067@rock.ICSI.Berkeley.EDU> Hi Team, I am new to bro and starting to look at platform from various detection and policy point of view. I have following queries : 1) How can i add domains Whitelist ,in controlled enviroment i know which domains are allowed anything else Trigger alert or log message/ email would be good as well. 2) Same as 1 expect look for URL or part of URL like /?var=32532part (basically any URI) 3) Software which are not approved trigger alert. 4) Integrate for Cuckoo or sandbox ? Any scripting guide /videos (paid or free ) which can help in basics and can ramp up modules script writing. Thanks for support and creating wonderful software. Regards, Vijay --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151117/8d01c633/attachment.html From robin at icir.org Tue Nov 17 07:54:14 2015 From: robin at icir.org (Robin Sommer) Date: Tue, 17 Nov 2015 07:54:14 -0800 Subject: [Bro] Elasticsearch 2.0 dot problem In-Reply-To: References: Message-ID: <20151117155414.GR91128@icir.org> Mind filing this as a ticket on tracker.bro.org with the pathces attached? Thanks, Robin On Tue, Nov 17, 2015 at 02:55 +0100, Daniel Guerra wrote: > Elasticsearch 2.0 doesn?t accept dots in fieldnames. Bro writes fieldnames with dots. > As a result bro data can not be written to Elasticsearch 2.0. > I have made 2 very small patches to bro/src/threading/formatters/JSON.h and > bro/src/threading/formatters/JSON.cc that solves this problem. > > > Regards, > > Daniel > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From Jimenez-Ramirez_Cesar at bah.com Tue Nov 17 09:34:59 2015 From: Jimenez-Ramirez_Cesar at bah.com (Jimenez-Ramirez, Cesar [USA]) Date: Tue, 17 Nov 2015 17:34:59 +0000 Subject: [Bro] bro Message-ID: <4C31E87932C1F749A7FD17A163AAC781013C08C6@ASHBDAG3M3.resource.ds.bah.com> Hello, I am trying to lear bro from scratch. i don't know where to start. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151117/53d090fe/attachment.html From tgdesrochers at gmail.com Tue Nov 17 11:11:12 2015 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Tue, 17 Nov 2015 14:11:12 -0500 Subject: [Bro] bro_json-logs Message-ID: I am using the following to output bro logs to JSON: #!bin/bash ## Configure bro to write JSON logs mkdir -p /opt/bro/share/bro/site/scripts sudo tee /opt/bro/share/bro/site/scripts/json-logs.bro << EOF @load tuning/json-logs redef LogAscii::json_timestamps = JSON::TS_ISO8601; redef LogAscii::use_json = T; EOF sudo tee -a /opt/bro/share/bro/site/local.bro << EOF # Load policy for JSON output @load scripts/json-logs EOF I ship my logs with logstash into my ELK stack. In logstash I am using the date match ts ISO8601. This is reading my dates correctly the issue is, I think, the bro to json function of bro is not parsing the date correctly. I begin to get records in logs with dates all over the map. Some dates are in the past, 1970, some are in the future, 2024, and some are correct. I have debugged my ELK stack down to the output of bro logs on the sensor and it appears that dates printed in bro logs are getting printed incorrectly and causing strange indices down stream in my ELK stack. Has anyone had this issue? Is there a bug in the way bro is reading UNIX and translating to ISO8601? Any thoughts on correcting this. I don't mind having TS in UNIX and having Logstash do the translating for me, but how would I do that with bro. Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151117/4cee3f93/attachment.html From jazoff at illinois.edu Tue Nov 17 11:31:07 2015 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 17 Nov 2015 19:31:07 +0000 Subject: [Bro] bro_json-logs In-Reply-To: References: Message-ID: <94A95258-AB15-4214-97F2-F6255370EE6D@illinois.edu> > On Nov 17, 2015, at 2:11 PM, Tim Desrochers wrote: > > it appears that dates printed in bro logs are getting printed incorrectly and causing strange indices down stream in my ELK stack. > Do you have an example of one of these log entries? -- - Justin Azoff From daniel.guerra69 at gmail.com Tue Nov 17 11:39:03 2015 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Tue, 17 Nov 2015 20:39:03 +0100 Subject: [Bro] Elasticsearch 2.0 dot problem In-Reply-To: <20151117155414.GR91128@icir.org> References: <20151117155414.GR91128@icir.org> Message-ID: Do I have access to that ? > On 17 Nov 2015, at 16:54, Robin Sommer wrote: > > Mind filing this as a ticket on tracker.bro.org with the pathces > attached? Thanks, > > Robin > > On Tue, Nov 17, 2015 at 02:55 +0100, Daniel Guerra wrote: > >> Elasticsearch 2.0 doesn?t accept dots in fieldnames. Bro writes fieldnames with dots. >> As a result bro data can not be written to Elasticsearch 2.0. >> I have made 2 very small patches to bro/src/threading/formatters/JSON.h and >> bro/src/threading/formatters/JSON.cc that solves this problem. > > > >> >> >> Regards, >> >> Daniel >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From daniel.guerra69 at gmail.com Tue Nov 17 11:45:12 2015 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Tue, 17 Nov 2015 20:45:12 +0100 Subject: [Bro] bro_json-logs In-Reply-To: <94A95258-AB15-4214-97F2-F6255370EE6D@illinois.edu> References: <94A95258-AB15-4214-97F2-F6255370EE6D@illinois.edu> Message-ID: copied it from my kibana _source in the json view (I use a patch to change id.orig_h in id_orig_h for elastic 2.0) { "ts": "2015-11-17T18:23:15.519645Z", "uid": "CtqA6r1V0ob769KrQh", "id_orig_h": "192.168.1.122", "id_orig_p": 123, "id_resp_h": "17.253.84.253", "id_resp_p": 123, "proto": "udp", "duration": 0.393113, "orig_bytes": 48, "resp_bytes": 48, "conn_state": "SF", "missed_bytes": 0, "history": "Dd", "orig_pkts": 1, "orig_ip_bytes": 76, "resp_pkts": 1, "resp_ip_bytes": 76, "tunnel_parents": [], "resp_location": "37.323002,-122.032204" } > On 17 Nov 2015, at 20:31, Azoff, Justin S wrote: > >> On Nov 17, 2015, at 2:11 PM, Tim Desrochers wrote: >> >> it appears that dates printed in bro logs are getting printed incorrectly and causing strange indices down stream in my ELK stack. >> > > Do you have an example of one of these log entries? > > -- > - Justin Azoff > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From doris at bro.org Tue Nov 17 12:12:02 2015 From: doris at bro.org (Doris Schioberg) Date: Tue, 17 Nov 2015 12:12:02 -0800 Subject: [Bro] bro In-Reply-To: <4C31E87932C1F749A7FD17A163AAC781013C08C6@ASHBDAG3M3.resource.ds.bah.com> References: <4C31E87932C1F749A7FD17A163AAC781013C08C6@ASHBDAG3M3.resource.ds.bah.com> Message-ID: <564B8A12.9060807@bro.org> Hello Cesar, you have already started by using this mailing list :) This is a first contact point for every question that you can't get answered through documentation. Depending on how you learn you might find the right type of tool on the Bro documentation page. https://www.bro.org/documentation/index.html There you can find classic how-tos, exercises, Videos, and other sources. Whenever you get stuck with a topic please come back to this mailing list! Welcome to the Bro world! Doris Schioberg On 11/17/15 9:34 AM, Jimenez-Ramirez, Cesar [USA] wrote: > Hello, > > > I am trying to lear bro from scratch. i don't know where to start. > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Doris Schioberg Bro Outreach, Training, and Education Coordinator International Computer Science Institute (ICSI Berkeley) Phone: +1 (510) 289-8406 * doris at bro.org From sven at dreyer-net.de Tue Nov 17 12:38:56 2015 From: sven at dreyer-net.de (Sven Dreyer) Date: Tue, 17 Nov 2015 21:38:56 +0100 Subject: [Bro] Problem with connections in S1 and SF state Message-ID: <564B9060.1080304@dreyer-net.de> Dear list, I'm having trouble understanding some log entries from my conn.log. I already learned from this mailing list that bro cannot surely detect who initiated a connection if it does not see the initial connection setup, which seems logical to me. But if I look to my conn.log file, I find entries like these: 1446190221.687738 Cbu3fj3FYdODxvLF1h 87.152.221.xxx 50993 192.168.100.yyy 36709 tcp ssl 122.745965 1238 5340 S1 F T 0 ShAD ad 20 2050 19 6112 (empty) 1446190138.746769 CykNrp4VEfzbrJ2vm6 87.152.221.xxx 50993 192.168.100.yyy 36679 tcp ssl 223.406750 1384 18908 S1 F T 0 ShAD ad 39 2956 36 20360 (empty) It looks like our IMAP server (87.152.221.xxx running on port 50993) initiated a connection to my notebook (192.168.100.yyy). That should not be possible due to lack of port forwarding for this connection. So my first guess is that bro didn't see the initial connection setup (midstream traffic, OTH state). But I took a look into the documentation on https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html regarding the reported states (S1), which says: S1 Connection established, not terminated. This looks to me like bro saw the connection setup. Or did I get something wrong here? Oh and by the way: the next paragraph reads: SF Normal establishment and termination. Note that this is the same symbol as for state S1. You can tell the two apart because for S1 there will not be any byte counts in the summary, while for SF there will be. I don't understand this. Do S1 and SF really only differ in byte count zero or non-zero? It seems to me that they also differ in "connection still alive" and "connection was terminated". Looking further trough the logs, I also find entries with "SF" flag in whuch source and destination seem twisted: 1445338094.186121 C9uuKp4dE9nrHo46bd 87.152.220.xxx 50993 192.168.100.yyy 20108 tcp -462.348551 401 754 SF F T 0 DdAfFa 13 921 12 1234 (empty) Does anybody have a hint? Did I misunderstand something? I'm running bro 2.4.1. Thanks a lot! Sven From jazoff at illinois.edu Tue Nov 17 12:53:19 2015 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 17 Nov 2015 20:53:19 +0000 Subject: [Bro] Problem with connections in S1 and SF state In-Reply-To: <564B9060.1080304@dreyer-net.de> References: <564B9060.1080304@dreyer-net.de> Message-ID: <5A214812-FF4A-4360-B643-EFCC1516F869@illinois.edu> You should really be looking at the history field: history: string &log &optional Records the state history of connections as a string of letters. The meaning of those letters is: Letter Meaning s a SYN w/o the ACK bit set h a SYN+ACK (?handshake?) a a pure ACK d packet with payload (?data?) f packet with FIN bit set r packet with RST bit set c packet with a bad checksum i inconsistent packet (e.g. SYN+RST bits both set) If the event comes from the originator, the letter is in upper-case; if it comes from the responder, it?s in lower-case. Multiple packets of the same type will only be noted once (e.g. we only record one ?d? in each direction, regardless of how many data packets were seen.) So any connection that starts with D or d means bro missed the initial syn handshake (Sh) -- - Justin Azoff > On Nov 17, 2015, at 3:38 PM, Sven Dreyer wrote: > > Dear list, > > I'm having trouble understanding some log entries from my conn.log. I > already learned from this mailing list that bro cannot surely detect who > initiated a connection if it does not see the initial connection setup, > which seems logical to me. > > But if I look to my conn.log file, I find entries like these: > > 1446190221.687738 Cbu3fj3FYdODxvLF1h 87.152.221.xxx 50993 > 192.168.100.yyy 36709 tcp ssl 122.745965 1238 5340 > S1 F T 0 ShAD > ad 20 2050 19 6112 (empty) > 1446190138.746769 CykNrp4VEfzbrJ2vm6 87.152.221.xxx 50993 > 192.168.100.yyy 36679 tcp ssl 223.406750 1384 18908 > S1 F T 0 ShAD ad 39 2956 36 > 20360 (empty) > > It looks like our IMAP server (87.152.221.xxx running on port 50993) > initiated a connection to my notebook (192.168.100.yyy). That should not > be possible due to lack of port forwarding for this connection. > > So my first guess is that bro didn't see the initial connection setup > (midstream traffic, OTH state). But I took a look into the documentation > on https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html > regarding the reported states (S1), which says: > > S1 Connection established, not terminated. > > This looks to me like bro saw the connection setup. Or did I get > something wrong here? > > Oh and by the way: the next paragraph reads: > > SF Normal establishment and termination. Note that this is the same > symbol as for state S1. You can tell the two apart because for S1 there > will not be any byte counts in the summary, while for SF there will be. > > I don't understand this. Do S1 and SF really only differ in byte count > zero or non-zero? It seems to me that they also differ in "connection > still alive" and "connection was terminated". > > Looking further trough the logs, I also find entries with "SF" flag in > whuch source and destination seem twisted: > > 1445338094.186121 C9uuKp4dE9nrHo46bd 87.152.220.xxx 50993 > 192.168.100.yyy 20108 tcp -462.348551 401 754 SF > F T 0 DdAfFa 13 921 12 1234 (empty) > > Does anybody have a hint? Did I misunderstand something? > > I'm running bro 2.4.1. > > Thanks a lot! > Sven > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From asharma at lbl.gov Wed Nov 18 11:13:26 2015 From: asharma at lbl.gov (Aashish Sharma) Date: Wed, 18 Nov 2015 11:13:26 -0800 Subject: [Bro] Fwd: current_time() vs network_time() In-Reply-To: <20151118185851.GC15059@yaksha.lbl.gov> References: <20151118185851.GC15059@yaksha.lbl.gov> Message-ID: So, I am trying to have bro send me report/alerts at specific time-slots. Given current_time is the wall-clock time, I am relying on current_time() function to get time and then, my code is : if (hh:mm:ss == desired time), run a report. I noticed inconsistencies so here is more detailed debug log: I notice, jumps in the current_time: Report time is 1447869593.121702, report hour is 9:59:53 Report time is 1447869595.234395, report hour is 9:59:55 Report time is 1447869596.45385, report hour is 9:59:56 Report time is 1447869597.636261, report hour is 9:59:57 Report time is 1447869598.597632, report hour is 9:59:58 Report time is 1447869599.628088, report hour is 9:59:59 Report time is 1447869601.926001, report hour is 10:0:1 <----- no 10:0:0 ? Report time is 1447869603.182218, report hour is 10:0:3 <--- jump Report time is 1447869604.166191, report hour is 10:0:4 Report time is 1447869605.647308, report hour is 10:0:5 Report time is 1447869606.499426, report hour is 10:0:6 Report time is 1447869607.383869, report hour is 10:0:7 Report time is 1447869617.52706, report hour is 10:0:17 <----- big jump Report time is 1447869618.188414, report hour is 10:0:18 Report time is 1447869619.04252, report hour is 10:0:19 <- stall ? Report time is 1447869619.733979, report hour is 10:0:19 <--- stall ? Report time is 1447869622.635545, report hour is 10:0:22 Report time is 1447869623.28335, report hour is 10:0:23 I believe network_time would be somewhat better probably and will try to see how that fares for my use case. Any idea why I see such jumps on the wall-clock times ? I'd think this should be rather more reliable ? Thanks, Aashish From brianallen at wustl.edu Wed Nov 18 13:25:07 2015 From: brianallen at wustl.edu (Allen, Brian) Date: Wed, 18 Nov 2015 21:25:07 +0000 Subject: [Bro] Bro performance & sizing question In-Reply-To: <56463EFC.2020000@upenn.edu> References: <56463EFC.2020000@upenn.edu> Message-ID: Hi Melissa, I built our cluster with the following specs: Two bro hosts in the cluster + a separate server which runs the bro manager and stores the logs + the arista switch. Each worker server has 20 cores, 128GB memory, with a 10Gig Intel network card with Pf_Ring. We have now pinned 18 bro processes to the cores, which leaves 2 for OS tasks. Regarding campus bandwidth, we are pushing 4Gb during normal peak times on I1. We are pushing up to 2Gb on I2 at random times. So it is very possible to be pushing between 4 and 6Gb of traffic at any one time. I installed the cluster in a datacenter nearby to test it out before moving it downtown next to our edge routers. The edge routers span the traffic to our bro cluster arista switch. While the cluster was in the the datacenter it saw live traffic, but it was much less than the overall campus bandwidth, but I thought it would be a good place to test it out near my office and make sure it was working correctly. The packet loss in the data center was very low, the cluster was running great, logs looked complete, so we moved it downtown. We immediately had a problem with capture loss of 60%-90%. The CPUs, though, weren't all pegged at 100% like they were with our old underpowered bro box. Only 6-7 processors were pegged at 100% and the rest were down around 30-40%. We added a bpf filter to only process packets to/from our campus subnets. We haven't tried a filter for large flows yet. We double checked pf_ring settings, double checked the arista switch settings, and tuned the network card, but the main thing we did was turn off hyper threading. That immediately dropped the capture loss down to 0.0% I guess we didn't catch that earlier because the data center didn't have as much traffic going through the cluster. In the datacenter test we had pinned bro to 36 of the 40 hyper threaded cpus on each worker, so when we got rid of that, it worked great. We had been running eight bro proxies on the manager when it was hyper threaded, and I think I can drop that down to four proxies now, I just haven't tried that yet since it is working at the moment. After we got the packet loss to 0.0%, we actually ran a test where we set up a separate instance of BRO on just one of our worker servers and it was able to handle the entire load (at least during a short test). All the processors jumped from 20-40% usage to 40-60% when we ran it on a single box, but the capture loss was still basically zero. We are going to run with two boxes, though, because we expect our bandwidth needs to grow and the cluster will be able to keep up for a while. Hope that helps. Brian Allen Information Security Manager Washington University From: > on behalf of Melissa Muth > Date: Friday, November 13, 2015 at 1:50 PM To: Bro-Mailinglist > Subject: [Bro] Bro performance & sizing question -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 We have a Bro cluster currently attempting to process up to 13Gbps (1.4Mpps) partitioned over two 10Gbps Gigamon network taps. Capture loss currently averages 44% - but before buying more hardware, we'd like to sanity-check our plans with folks who have already successfully sized their own installations. Currently there are two Bro hosts in the cluster, each with 20 CPU cores (3.1Ghz), 128GB memory, and Myricom cards with the Sniffer V3 driver. Each host runs a proxy, and 17 workers pinned to CPUs. The manager is running on one of the worker hosts, and logs are being written to SSD drives. We're using restrict_filters to ignore (large) flows generated by four hosts. The current plan is to buy 2 more worker hosts (same specs), as well as a NAS for storing logs after each hourly rotation. If we're capturing 56% of 13Gbps, that's 7454Mbps. Given the 34 cores used by bro, that works out to 219Mbps/core and about 3.6Gbps/host. Does that seem like expected performance, or might there be something broken somewhere? Does it seem reasonable to buy two more worker hosts (at least to handle current needs)? Any thoughts or recommendations would be much appreciated. Cheers, Melissa -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iEYEARECAAYFAlZGPvwACgkQjGIGZe3KNcl6GgCgijm+F4zbDC0rnuP8VMRa2YSi Tz8AoIPAvHBeF/R1e/C+HEIkSv2XO//L =p+4P -----END PGP SIGNATURE----- _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151118/4ebc7ba1/attachment.html From daniel.guerra69 at gmail.com Thu Nov 19 06:33:24 2015 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Thu, 19 Nov 2015 15:33:24 +0100 Subject: [Bro] git clone error Message-ID: Hi, I?m trying to do a git clone ?recursive but end up with this error. Unable to checkout '71a1e3efc437aa9f981be71affa1c4615e8d98a5' in submodule path 'aux/btest? Any idea how to solve this ? Regards, Daniel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151119/15f82250/attachment.html From robin at icir.org Thu Nov 19 08:01:23 2015 From: robin at icir.org (Robin Sommer) Date: Thu, 19 Nov 2015 08:01:23 -0800 Subject: [Bro] git clone error In-Reply-To: References: Message-ID: <20151119160123.GG78027@icir.org> On Thu, Nov 19, 2015 at 15:33 +0100, Daniel Guerra wrote: > Unable to checkout '71a1e3efc437aa9f981be71affa1c4615e8d98a5' in submodule path 'aux/btest? Just fixed, try again. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From sven at dreyer-net.de Thu Nov 19 08:21:57 2015 From: sven at dreyer-net.de (Sven Dreyer) Date: Thu, 19 Nov 2015 17:21:57 +0100 Subject: [Bro] Problem with connections in S1 and SF state In-Reply-To: <5A214812-FF4A-4360-B643-EFCC1516F869@illinois.edu> References: <564B9060.1080304@dreyer-net.de> <5A214812-FF4A-4360-B643-EFCC1516F869@illinois.edu> Message-ID: <564DF725.6010604@dreyer-net.de> Justin, thanks for the hint. I should indeed have checked the history field. But even for connections that do not start with d or D in the history field, I see the same behaviour. Source and destination is still twisted: 1447675087.121817 CjRCD61gNErucciPb8 87.144.16.xxx 50993 192.168.100.yyy 26577 tcp ssl 83.596659 1432 2619 S1 F T 0 ShADad 18 2164 15 3231 (empty) Bro is configured to listen to a bridge interface (br0). But I also have running a dumpcap process writing all packets to pcap files. Interestingly, if I feed the corresponding pcap file to bro (bro -r file.pcap), I get source and destination in the right order: 1447675087.121817 C2AvJf3WgcdiBlYfS4 192.168.100.yyy 26577 87.144.16.xxx 50993 tcp ssl 83.596659 1432 2619 S1 - - 0 ShADad 18 2164 15 3231 (empty) Does anybody have an explanation for this? Thanks, Sven Am 17.11.2015 um 21:53 schrieb Azoff, Justin S: > You should really be looking at the history field: > > history: string &log &optional > Records the state history of connections as a string of letters. The meaning of those letters is: > > Letter Meaning > s a SYN w/o the ACK bit set > h a SYN+ACK (?handshake?) > a a pure ACK > d packet with payload (?data?) > f packet with FIN bit set > r packet with RST bit set > c packet with a bad checksum > i inconsistent packet (e.g. SYN+RST bits both set) > If the event comes from the originator, the letter is in upper-case; if it comes from the responder, it?s in lower-case. Multiple packets of the same type will only be noted once (e.g. we only record one ?d? in each direction, regardless of how many data packets were seen.) > > So any connection that starts with D or d means bro missed the initial syn handshake (Sh) > > From chris at cwalsh.org Thu Nov 19 08:45:25 2015 From: chris at cwalsh.org (Chris Walsh) Date: Thu, 19 Nov 2015 10:45:25 -0600 Subject: [Bro] OT: thanks In-Reply-To: <20151119160123.GG78027@icir.org> References: <20151119160123.GG78027@icir.org> Message-ID: <37C81D09-9B6D-473B-8BE8-ED0FFCEA195E@cwalsh.org> I hope folks will pardon my going meta, and thus possibly OT, but I just wanted to say that the responsiveness of the folks on the list to questions and issues is really (really!) awesome. cw On Nov 19, 2015, at 10:01 AM, Robin Sommer wrote: > > On Thu, Nov 19, 2015 at 15:33 +0100, Daniel Guerra wrote: > >> Unable to checkout '71a1e3efc437aa9f981be71affa1c4615e8d98a5' in submodule path 'aux/btest? > > Just fixed, try again. From tgdesrochers at gmail.com Thu Nov 19 17:29:18 2015 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Thu, 19 Nov 2015 20:29:18 -0500 Subject: [Bro] [bro] bro_json-logs In-Reply-To: References: <00DDF6FD-0B64-44C4-973C-3E8D46446EB2@gmail.com> <79C2B85D-AF7E-4007-8D3D-FBEF296AF47C@gmail.com> <44536B9E-E195-43DC-8A91-890CBFAC2D9B@gmail.com> <49ED6397-0174-4AE4-9A23-585C3E64AFEC@gmail.com> <82710E34-E7A9-4425-A2B0-7D7B71DE04F7@gmail.com> <09785155-8802-4A84-A363-C09DCA183AF9@gmail.com> <55FE1167-0B02-49C4-967C-78DE350921CE@gmail.com> <01407BDA-C164-42E9-9369-A49317FD3B92@gmail.com> Message-ID: I didn't realize I hadn't posted this on the forum and I had just responded to one person. First, thank you Daniel for your help. I'm running bro 2.4.1 on Ubuntu 12.04 and collecting live traffic. I have bro configured to print logs in in json with timestamps in ISO8601. My problem is some timestamps in the logs are printed with days and years in the future and past. Below is an example of logs with the TS in 2016. I have many other examples I'm happy to share if it would help people. So here are my configs: for bro I use the following to change the logs to json: 

@load tuning/json-logs

redef LogAscii::json_timestamps = JSON::TS_ISO8601;

redef LogAscii::use_json = T;
If I perform a search of the printed logs on my sensor for dates that are in the future I see the following: zcat -r /nsm/bro/logs/* | grep -r \"ts\"\:\"2016 

{"ts":"2016-08-27T16:10:11.269010Z","uid":"CfxBj3Sb7mz0Dmv8i","id.orig_h":"192.168.10.100","id.orig_p":62662,"id.resp_h":"104.85.62.152","id.resp_p":443,"name":"window_recision","notice":false,"peer":"satcon99-eth2-2"}

{"ts":"2016-02-06T19:13:33.416241Z","fuid":"FJsBOM25KVPMKSxu07","tx_hosts":["23.31.220.33"],"rx_hosts":["17.110.224.215"],"conn_uids":["CIxuLC3WAExVyJisll"],"source":"SSL","depth":0,"analyzers":["SHA1","MD5","X509"],"mime_type":"application/pkix-cert","duration":0.0,"local_orig":true,"is_orig":true,"seen_bytes":674,"missing_bytes":0,"overflow_bytes":0,"timedout":false,"md5":"2a465a2f7f29fc9677683337114d5cb0","sha1":"2f6854f72d69c47ebadd2a859cade1f7492190a2"}

{"ts":"2016-01-30T17:12:50.116824Z","uid":"Cw4Sfa3mwiDWVraPq5","id.orig_h":"23.31.220.33","id.orig_p":12912,"id.resp_h":"17.164.1.22","id.resp_p":443,"proto":"tcp","duration":0.042421,"orig_bytes":0,"resp_bytes":0,"conn_state":"SH","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"FA","orig_pkts":2,"orig_ip_bytes":80,"resp_pkts":0,"resp_ip_bytes":0,"tunnel_parents":[],"resp_cc":"US","sensorname":"satcon99-eth3"}

{"ts":"2016-01-19T00:00:00.078926Z","uid":"CPrS9S3LLs3oGVRAQ2","id.orig_h":"192.168.10.115","id.orig_p":64606,"id.resp_h":"50.31.185.42","id.resp_p":80,"name":"unescaped_special_URI_char","notice":false,"peer":"satcon99-eth2-2"}

{"ts":"2016-10-20T23:59:59.681405Z","fuid":"FQsWce4xIknwzbuBCg","tx_hosts":["54.187.59.44"],"rx_hosts":["192.168.10.143"],"conn_uids":["C8EhsO2mX4IVRshw07"],"source":"HTTP","depth":0,"analyzers":["MD5","SHA1"],"mime_type":"image/gif","duration":0.0,"local_orig":false,"is_orig":false,"seen_bytes":43,"total_bytes":43,"missing_bytes":0,"overflow_bytes":0,"timedout":false,"md5":"325472601571f31e1bf00674c368d335","sha1":"2daeaa8b5f19f0bc209d976c02bd6acb51b00b0a"}

{"ts":"2016-06-09T12:00:00.819970Z","uid":"Con3l44xdVoLgXarc3","id.orig_h":"23.31.220.33","id.orig_p":12721,"id.resp_h":"66.119.33.141","id.resp_p":80,"name":"data_before_established","notice":false,"peer":"satcon99-eth3-1"}

{"ts":"2016-04-20T23:59:59.918852Z","uid":"CBXuzu4OTSZyy7OHS7","id.orig_h":"23.31.220.33","id.orig_p":5109,"id.resp_h":"173.194.123.104","id.resp_p":443,"name":"possible_split_routing","notice":false,"peer":"satcon99-eth3-1"}

{"ts":"2016-06-16T23:59:59.011608Z","uid":"CCmxue4UZiXJPZTcG4","id.orig_h":"192.168.10.104","id.orig_p":57418,"id.resp_h":"10.1.72.2","id.resp_p":5601,"proto":"tcp","duration":3.001216,"orig_bytes":0,"resp_bytes":0,"conn_state":"S0","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"S","orig_pkts":4,"orig_ip_bytes":208,"resp_pkts":0,"resp_ip_bytes":0,"tunnel_parents":[],"sensorname":"satcon99-eth2"}
My assumption is that there is an error in the way bro is parsing the unix time and converting it to JSON. Is this a bug?? Why would the sensor be outputting logs with TS in the future. It has been suggested to try redef LogAscii::json_timestamps = TS_MILLIS (Thanks Daniel) I am trying that in the morning but I wanted to know if there were any other who have experienced this issue when printing json logs with ISO8601 timestamps. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151119/e6be93c5/attachment.html From tgdesrochers at gmail.com Fri Nov 20 08:19:26 2015 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Fri, 20 Nov 2015 11:19:26 -0500 Subject: [Bro] [bro] bro_json-logs In-Reply-To: References: <00DDF6FD-0B64-44C4-973C-3E8D46446EB2@gmail.com> <79C2B85D-AF7E-4007-8D3D-FBEF296AF47C@gmail.com> <44536B9E-E195-43DC-8A91-890CBFAC2D9B@gmail.com> <49ED6397-0174-4AE4-9A23-585C3E64AFEC@gmail.com> <82710E34-E7A9-4425-A2B0-7D7B71DE04F7@gmail.com> <09785155-8802-4A84-A363-C09DCA183AF9@gmail.com> <55FE1167-0B02-49C4-967C-78DE350921CE@gmail.com> <01407BDA-C164-42E9-9369-A49317FD3B92@gmail.com> Message-ID: Thank you Daniel and Derek, it appears that changing the bro ts to TS_MILLIS and using a logstash date match UNIX_MS filter has fixed the strange date issue. This leads me to believe there is some issue how bro interprets prints ISO8601 timestamps. But for now this work around will suffice. Thanks again On Thu, Nov 19, 2015 at 8:29 PM, Tim Desrochers wrote: > I didn't realize I hadn't posted this on the forum and I had just > responded to one person. > > First, thank you Daniel for your help. > > I'm running bro 2.4.1 on Ubuntu 12.04 and collecting live traffic. > > I have bro configured to print logs in in json with timestamps in ISO8601. > My problem is some timestamps in the logs are printed with days and years > in the future and past. Below is an example of logs with the TS in 2016. I > have many other examples I'm happy to share if it would help people. > > So here are my configs: > > for bro I use the following to change the logs to json: > > 
>
> @load tuning/json-logs
>
> redef LogAscii::json_timestamps = JSON::TS_ISO8601;
>
> redef LogAscii::use_json = T;
>
>
> > If I perform a search of the printed logs on my sensor for dates that are > in the future I see the following: > > zcat -r /nsm/bro/logs/* | grep -r \"ts\"\:\"2016 > > 
>
>
> {"ts":"2016-08-27T16:10:11.269010Z","uid":"CfxBj3Sb7mz0Dmv8i","id.orig_h":"192.168.10.100","id.orig_p":62662,"id.resp_h":"104.85.62.152","id.resp_p":443,"name":"window_recision","notice":false,"peer":"satcon99-eth2-2"}
>
>
> {"ts":"2016-02-06T19:13:33.416241Z","fuid":"FJsBOM25KVPMKSxu07","tx_hosts":["23.31.220.33"],"rx_hosts":["17.110.224.215"],"conn_uids":["CIxuLC3WAExVyJisll"],"source":"SSL","depth":0,"analyzers":["SHA1","MD5","X509"],"mime_type":"application/pkix-cert","duration":0.0,"local_orig":true,"is_orig":true,"seen_bytes":674,"missing_bytes":0,"overflow_bytes":0,"timedout":false,"md5":"2a465a2f7f29fc9677683337114d5cb0","sha1":"2f6854f72d69c47ebadd2a859cade1f7492190a2"}
>
>
> {"ts":"2016-01-30T17:12:50.116824Z","uid":"Cw4Sfa3mwiDWVraPq5","id.orig_h":"23.31.220.33","id.orig_p":12912,"id.resp_h":"17.164.1.22","id.resp_p":443,"proto":"tcp","duration":0.042421,"orig_bytes":0,"resp_bytes":0,"conn_state":"SH","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"FA","orig_pkts":2,"orig_ip_bytes":80,"resp_pkts":0,"resp_ip_bytes":0,"tunnel_parents":[],"resp_cc":"US","sensorname":"satcon99-eth3"}
>
>
> {"ts":"2016-01-19T00:00:00.078926Z","uid":"CPrS9S3LLs3oGVRAQ2","id.orig_h":"192.168.10.115","id.orig_p":64606,"id.resp_h":"50.31.185.42","id.resp_p":80,"name":"unescaped_special_URI_char","notice":false,"peer":"satcon99-eth2-2"}
>
>
> {"ts":"2016-10-20T23:59:59.681405Z","fuid":"FQsWce4xIknwzbuBCg","tx_hosts":["54.187.59.44"],"rx_hosts":["192.168.10.143"],"conn_uids":["C8EhsO2mX4IVRshw07"],"source":"HTTP","depth":0,"analyzers":["MD5","SHA1"],"mime_type":"image/gif","duration":0.0,"local_orig":false,"is_orig":false,"seen_bytes":43,"total_bytes":43,"missing_bytes":0,"overflow_bytes":0,"timedout":false,"md5":"325472601571f31e1bf00674c368d335","sha1":"2daeaa8b5f19f0bc209d976c02bd6acb51b00b0a"}
>
>
> {"ts":"2016-06-09T12:00:00.819970Z","uid":"Con3l44xdVoLgXarc3","id.orig_h":"23.31.220.33","id.orig_p":12721,"id.resp_h":"66.119.33.141","id.resp_p":80,"name":"data_before_established","notice":false,"peer":"satcon99-eth3-1"}
>
>
> {"ts":"2016-04-20T23:59:59.918852Z","uid":"CBXuzu4OTSZyy7OHS7","id.orig_h":"23.31.220.33","id.orig_p":5109,"id.resp_h":"173.194.123.104","id.resp_p":443,"name":"possible_split_routing","notice":false,"peer":"satcon99-eth3-1"}
>
>
> {"ts":"2016-06-16T23:59:59.011608Z","uid":"CCmxue4UZiXJPZTcG4","id.orig_h":"192.168.10.104","id.orig_p":57418,"id.resp_h":"10.1.72.2","id.resp_p":5601,"proto":"tcp","duration":3.001216,"orig_bytes":0,"resp_bytes":0,"conn_state":"S0","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"S","orig_pkts":4,"orig_ip_bytes":208,"resp_pkts":0,"resp_ip_bytes":0,"tunnel_parents":[],"sensorname":"satcon99-eth2"}
>
> > My assumption is that there is an error in the way bro is parsing the unix > time and converting it to JSON. Is this a bug?? Why would the sensor be > outputting logs with TS in the future. > > It has been suggested to try > redef LogAscii::json_timestamps = TS_MILLIS > (Thanks Daniel) > > I am trying that in the morning but I wanted to know if there were any > other who have experienced this issue when printing json logs with ISO8601 > timestamps. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151120/fc4584a3/attachment-0001.html From jazoff at illinois.edu Fri Nov 20 08:34:33 2015 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 20 Nov 2015 16:34:33 +0000 Subject: [Bro] [bro] bro_json-logs In-Reply-To: References: <00DDF6FD-0B64-44C4-973C-3E8D46446EB2@gmail.com> <79C2B85D-AF7E-4007-8D3D-FBEF296AF47C@gmail.com> <44536B9E-E195-43DC-8A91-890CBFAC2D9B@gmail.com> <49ED6397-0174-4AE4-9A23-585C3E64AFEC@gmail.com> <82710E34-E7A9-4425-A2B0-7D7B71DE04F7@gmail.com> <09785155-8802-4A84-A363-C09DCA183AF9@gmail.com> <55FE1167-0B02-49C4-967C-78DE350921CE@gmail.com> <01407BDA-C164-42E9-9369-A49317FD3B92@gmail.com> Message-ID: > On Nov 20, 2015, at 11:19 AM, Tim Desrochers wrote: > > Thank you Daniel and Derek, it appears that changing the bro ts to TS_MILLIS and using a logstash date match UNIX_MS filter has fixed the strange date issue. > > This leads me to believe there is some issue how bro interprets prints ISO8601 timestamps. But for now this work around will suffice. > > Thanks again That's really odd because it is not interpreting them. The timestamp is already in seconds, it just passes it on to strftime: time_t t = time_t(val->val.double_val); if ( strftime(buffer, sizeof(buffer), "%Y-%m-%dT%H:%M:%S", gmtime(&t)) > 0 ) the TS_MILLIS setting just outputs val->val.double_val * 1000 as-is -- - Justin Azoff From dbora at isightpartners.com Mon Nov 23 09:12:23 2015 From: dbora at isightpartners.com (Disha Bora) Date: Mon, 23 Nov 2015 17:12:23 +0000 Subject: [Bro] Intel Framework Issues Message-ID: Hi, I have been using Bro's intel framework to input my intelligence feed and get matches in intel.log. I seem to only be getting hits on domains but not IPs or URLs. I have also tried it on the mal-dnssearch feeds with the same results. Is there any particular reason why this would happen? How can I fix it? Thanks! Disha Bora Associate Product Manager iSIGHT Partners -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151123/cfe28397/attachment.html From anthony.kasza at gmail.com Mon Nov 23 09:50:51 2015 From: anthony.kasza at gmail.com (anthony kasza) Date: Mon, 23 Nov 2015 09:50:51 -0800 Subject: [Bro] Intel Framework Issues In-Reply-To: References: Message-ID: Hi Disha, Can you include your intel.dat file and the script you are using to load it please? -AK On Nov 23, 2015 9:34 AM, "Disha Bora" wrote: > Hi, > > I have been using Bro's intel framework to input my intelligence feed and > get matches in intel.log. I seem to only be getting hits on domains but not > IPs or URLs. I have also tried it on the mal-dnssearch feeds with the same > results. Is there any particular reason why this would happen? How can I > fix it? > > Thanks! > > Disha Bora > Associate Product Manager > iSIGHT Partners > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151123/ad1350cc/attachment.html From jan.grashofer at cern.ch Mon Nov 23 09:53:34 2015 From: jan.grashofer at cern.ch (Jan Grashofer) Date: Mon, 23 Nov 2015 17:53:34 +0000 Subject: [Bro] Intel Framework Issues In-Reply-To: References: Message-ID: Hi, for URLs there is an important detail I missed the first time, when I used the intel framework. The documentation says: Intel::URL - A complete URL _without_ the prefix "http://". However, IPs worked for me without any problem. Did you see any errors in the logs regarding the intel-files you use? Depending on how you generate your feeds the intel linter (https://github.com/packetsled/bro_intel_linter) might be helpful for you. Best regards, Jan ________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Disha Bora [dbora at isightpartners.com] Sent: Monday, November 23, 2015 18:12 To: bro at bro.org Subject: [Bro] Intel Framework Issues Hi, I have been using Bro's intel framework to input my intelligence feed and get matches in intel.log. I seem to only be getting hits on domains but not IPs or URLs. I have also tried it on the mal-dnssearch feeds with the same results. Is there any particular reason why this would happen? How can I fix it? Thanks! Disha Bora Associate Product Manager iSIGHT Partners -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151123/e9f2afdb/attachment.html From dbora at isightpartners.com Mon Nov 23 10:42:13 2015 From: dbora at isightpartners.com (Disha Bora) Date: Mon, 23 Nov 2015 18:42:13 +0000 Subject: [Bro] Intel Framework Issues In-Reply-To: References: Message-ID: Hi, There are no errors as far as I can tell. I use a python script to generate my feeds but I also tried it on the mal-dnssearch feeds which are pre-formatted. In both cases, I am only successful on domains not IPs. Both have the same tab separation and header delineation. Thanks, Disha From: Jan Grashofer > Date: Monday, November 23, 2015 at 12:53 PM To: Disha Bora >, "bro at bro.org" > Subject: RE: Intel Framework Issues Hi, for URLs there is an important detail I missed the first time, when I used the intel framework. The documentation says: Intel::URL - A complete URL _without_ the prefix "http://". However, IPs worked for me without any problem. Did you see any errors in the logs regarding the intel-files you use? Depending on how you generate your feeds the intel linter (https://github.com/packetsled/bro_intel_linter) might be helpful for you. Best regards, Jan ________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Disha Bora [dbora at isightpartners.com] Sent: Monday, November 23, 2015 18:12 To: bro at bro.org Subject: [Bro] Intel Framework Issues Hi, I have been using Bro's intel framework to input my intelligence feed and get matches in intel.log. I seem to only be getting hits on domains but not IPs or URLs. I have also tried it on the mal-dnssearch feeds with the same results. Is there any particular reason why this would happen? How can I fix it? Thanks! Disha Bora Associate Product Manager iSIGHT Partners -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151123/234efa7f/attachment-0001.html From liburdi.joshua at gmail.com Mon Nov 23 10:47:57 2015 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Mon, 23 Nov 2015 13:47:57 -0500 Subject: [Bro] Intel Framework Issues In-Reply-To: References: Message-ID: <8F46B608-6C8C-4C9A-AA69-845B50879CBF@gmail.com> I think the most common gotcha for IP addresses is that they will only appear in intel.log when they are a part of a successful TCP connection. Unsuccessful TCP connections and non-TCP connections will not appear in the log. Josh > On Nov 23, 2015, at 12:53 PM, Jan Grashofer wrote: > > Hi, > > for URLs there is an important detail I missed the first time, when I used the intel framework. The documentation says: Intel::URL <> - A complete URL _without_ the prefix "http://". <> > > However, IPs worked for me without any problem. Did you see any errors in the logs regarding the intel-files you use? Depending on how you generate your feeds the intel linter (https://github.com/packetsled/bro_intel_linter ) might be helpful for you. > > Best regards, > Jan > > From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Disha Bora [dbora at isightpartners.com] > Sent: Monday, November 23, 2015 18:12 > To: bro at bro.org > Subject: [Bro] Intel Framework Issues > > Hi, > > I have been using Bro's intel framework to input my intelligence feed and get matches in intel.log. I seem to only be getting hits on domains but not IPs or URLs. I have also tried it on the mal-dnssearch feeds with the same results. Is there any particular reason why this would happen? How can I fix it? > > Thanks! > > Disha Bora > Associate Product Manager > iSIGHT Partners > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151123/38c2d1ef/attachment.html From pkelley at hyperionavenue.com Mon Nov 23 11:20:15 2015 From: pkelley at hyperionavenue.com (Patrick Kelley) Date: Mon, 23 Nov 2015 11:20:15 -0800 Subject: [Bro] Intel Framework Issues In-Reply-To: <8F46B608-6C8C-4C9A-AA69-845B50879CBF@gmail.com> References: <8F46B608-6C8C-4C9A-AA69-845B50879CBF@gmail.com> Message-ID: This is true. I had to write a new script that will match for uncompleted connections. However and as you might expect, it is costly with performance. On Mon, Nov 23, 2015 at 10:47 AM, Josh Liburdi wrote: > I think the most common gotcha for IP addresses is that they will only > appear in intel.log when they are a part of a successful TCP connection. > Unsuccessful TCP connections and non-TCP connections will not appear in the > log. > > Josh > > On Nov 23, 2015, at 12:53 PM, Jan Grashofer wrote: > > Hi, > > > for URLs there is an important detail I missed the first time, when I used > the intel framework. The documentation says: Intel::URL - A complete URL > _without_ the prefix "http://". > > > However, IPs worked for me without any problem. Did you see any errors in > the logs regarding the intel-files you use? Depending on how you generate > your feeds the intel linter ( > https://github.com/packetsled/bro_intel_linter) might be helpful for you. > > > Best regards, > Jan > > > ------------------------------ > *From:* bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Disha Bora > [dbora at isightpartners.com] > *Sent:* Monday, November 23, 2015 18:12 > *To:* bro at bro.org > *Subject:* [Bro] Intel Framework Issues > > Hi, > > I have been using Bro's intel framework to input my intelligence feed and > get matches in intel.log. I seem to only be getting hits on domains but not > IPs or URLs. I have also tried it on the mal-dnssearch feeds with the same > results. Is there any particular reason why this would happen? How can I > fix it? > > Thanks! > > Disha Bora > Associate Product Manager > iSIGHT Partners > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Patrick Kelley, CEH Hyperion Avenue Labs http://www.hyperionavenue.com 951.291.8310 *The limit to which you have accepted being comfortable is the limit to which you have grown. Accept new challenges as an opportunity to enrich yourself and not as a point of potential failure.* [image: hal_logo] -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151123/a1c027fa/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 12155 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151123/a1c027fa/attachment-0001.bin From seth at icir.org Mon Nov 23 12:48:17 2015 From: seth at icir.org (Seth Hall) Date: Mon, 23 Nov 2015 15:48:17 -0500 Subject: [Bro] Intel Framework Issues In-Reply-To: <8F46B608-6C8C-4C9A-AA69-845B50879CBF@gmail.com> References: <8F46B608-6C8C-4C9A-AA69-845B50879CBF@gmail.com> Message-ID: > On Nov 23, 2015, at 1:47 PM, Josh Liburdi wrote: > > I think the most common gotcha for IP addresses is that they will only appear in intel.log when they are a part of a successful TCP connection. Unsuccessful TCP connections and non-TCP connections will not appear in the log. Thanks for pointing that out Josh! I do want to point out the reasoning behind that as a default decision. My thinking was that spoofed packets could cause false hits and generally non-responded-to probes coming from intelligence hosts aren?t all that interesting. Is there general agreement that that?s the right approach or should single packets hit on intelligence items by default? I think there is a high bar to clear with adding that as default behavior, but I?d like to hear from people actively doing incident response on their thoughts. I should also point out that we can certainly include a script that matches on non-complete connections with Bro even if we don?t load it by default. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From liburdi.joshua at gmail.com Mon Nov 23 13:03:54 2015 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Mon, 23 Nov 2015 16:03:54 -0500 Subject: [Bro] Intel Framework Issues In-Reply-To: References: <8F46B608-6C8C-4C9A-AA69-845B50879CBF@gmail.com> Message-ID: I think you?re right, the default behavior is probably the best for most users. There are a few cases where I?d want to see an indicator match regardless of the connection status? for example, if the IP indicator in question is associated with a known command and control server or an attack group who may utilize the server as a C2, then I?d want to see any connection attempt from my network to that IP address. It also depends on how users are using intel.log. For example, if a team treats every indicator in intel.log as a traditional alert (that someone then has to triage), then you want to minimize the resource impact that has on your team (e.g., I don?t want my team chasing down indicators that amount to nothing)? in this case, the default behavior works to our favor. If a team uses intel.log as a source of indicator-related information, then you may want to see every IP indicator match regardless of connection status? seeing every IP indicator match increases your chances of catching things like attacker infrastructure changes or infrastructure roll over. There?s a lot of granularity that can be worked into how indicators are handled (the intel-ext script is great for that), but it?s very dependent on the objectives of any one particular team. Adding a policy script to enable IP indicator matches for any connection would be a good start, but the default behavior is a great approach! Josh > On Nov 23, 2015, at 3:48 PM, Seth Hall wrote: > > >> On Nov 23, 2015, at 1:47 PM, Josh Liburdi wrote: >> >> I think the most common gotcha for IP addresses is that they will only appear in intel.log when they are a part of a successful TCP connection. Unsuccessful TCP connections and non-TCP connections will not appear in the log. > > Thanks for pointing that out Josh! I do want to point out the reasoning behind that as a default decision. My thinking was that spoofed packets could cause false hits and generally non-responded-to probes coming from intelligence hosts aren?t all that interesting. Is there general agreement that that?s the right approach or should single packets hit on intelligence items by default? > > I think there is a high bar to clear with adding that as default behavior, but I?d like to hear from people actively doing incident response on their thoughts. > > I should also point out that we can certainly include a script that matches on non-complete connections with Bro even if we don?t load it by default. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > From pkelley at hyperionavenue.com Mon Nov 23 13:14:40 2015 From: pkelley at hyperionavenue.com (Patrick Kelley) Date: Mon, 23 Nov 2015 13:14:40 -0800 Subject: [Bro] Intel Framework Issues In-Reply-To: References: <8F46B608-6C8C-4C9A-AA69-845B50879CBF@gmail.com> Message-ID: I perform IR daily as a core function. I find immense value in seeing all attempts, so long as the performance cost isn't prohibitive. Even if the connection doesn't complete, it shows intent. There may not be a great deal to look at, but it puts it on my radar and generally inspires me to look at other traffic on that local host. It is possible that something else of interest is happening, but the IOCs haven't made it into a feed. On Mon, Nov 23, 2015 at 12:48 PM, Seth Hall wrote: > > > On Nov 23, 2015, at 1:47 PM, Josh Liburdi > wrote: > > > > I think the most common gotcha for IP addresses is that they will only > appear in intel.log when they are a part of a successful TCP connection. > Unsuccessful TCP connections and non-TCP connections will not appear in the > log. > > Thanks for pointing that out Josh! I do want to point out the reasoning > behind that as a default decision. My thinking was that spoofed packets > could cause false hits and generally non-responded-to probes coming from > intelligence hosts aren?t all that interesting. Is there general agreement > that that?s the right approach or should single packets hit on intelligence > items by default? > > I think there is a high bar to clear with adding that as default behavior, > but I?d like to hear from people actively doing incident response on their > thoughts. > > I should also point out that we can certainly include a script that > matches on non-complete connections with Bro even if we don?t load it by > default. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Patrick Kelley, CEH Hyperion Avenue Labs http://www.hyperionavenue.com 951.291.8310 *The limit to which you have accepted being comfortable is the limit to which you have grown. Accept new challenges as an opportunity to enrich yourself and not as a point of potential failure.* [image: hal_logo] -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151123/aee5e95f/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 12155 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151123/aee5e95f/attachment-0001.bin From bkellogg at dresser-rand.com Mon Nov 23 13:20:52 2015 From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN)) Date: Mon, 23 Nov 2015 21:20:52 +0000 Subject: [Bro] Intel Framework Issues In-Reply-To: References: <8F46B608-6C8C-4C9A-AA69-845B50879CBF@gmail.com> Message-ID: I'm interested on non-complete connections being logged; CrowdStrike has a script to enable this already and I use it. It?s a way of finding compromised hosts out there that have lost contact with their CnC for whatever reason. But, I do not run a lot of indicators in Bro. Most open source intel lists create too many FPs to be used as a good primary indicator of badness in my experience. An option to enable it per indicator/indicator type or both may be a nice option. -Brian -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Seth Hall Sent: Monday, November 23, 2015 3:48 PM To: Josh Liburdi Cc: Disha Bora; bro at bro.org Subject: Re: [Bro] Intel Framework Issues > On Nov 23, 2015, at 1:47 PM, Josh Liburdi wrote: > > I think the most common gotcha for IP addresses is that they will only appear in intel.log when they are a part of a successful TCP connection. Unsuccessful TCP connections and non-TCP connections will not appear in the log. Thanks for pointing that out Josh! I do want to point out the reasoning behind that as a default decision. My thinking was that spoofed packets could cause false hits and generally non-responded-to probes coming from intelligence hosts aren?t all that interesting. Is there general agreement that that?s the right approach or should single packets hit on intelligence items by default? I think there is a high bar to clear with adding that as default behavior, but I?d like to hear from people actively doing incident response on their thoughts. I should also point out that we can certainly include a script that matches on non-complete connections with Bro even if we don?t load it by default. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jazoff at illinois.edu Mon Nov 23 13:40:16 2015 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 23 Nov 2015 21:40:16 +0000 Subject: [Bro] Intel Framework Issues In-Reply-To: References: <8F46B608-6C8C-4C9A-AA69-845B50879CBF@gmail.com> Message-ID: > On Nov 23, 2015, at 3:48 PM, Seth Hall wrote: > >> >> On Nov 23, 2015, at 1:47 PM, Josh Liburdi wrote: >> >> I think the most common gotcha for IP addresses is that they will only appear in intel.log when they are a part of a successful TCP connection. Unsuccessful TCP connections and non-TCP connections will not appear in the log. > > Thanks for pointing that out Josh! I do want to point out the reasoning behind that as a default decision. My thinking was that spoofed packets could cause false hits and generally non-responded-to probes coming from intelligence hosts aren?t all that interesting. Is there general agreement that that?s the right approach or should single packets hit on intelligence items by default? > > I think there is a high bar to clear with adding that as default behavior, but I?d like to hear from people actively doing incident response on their thoughts. > > I should also point out that we can certainly include a script that matches on non-complete connections with Bro even if we don?t load it by default. > > .Seth IN_ORIG / IN_RESP may help with this seen IN_RESP in a failed outbound connection to a known phishing site, useful to know seen IN_ORIG in a failed incoming port 22 connection from a known ssh scanner, probably just noise. seen in IN_RESP in a failed outbound port 22 connection to that same known ssh scanner, useful to know Which I guess would mean something like event connection_i_forget(c: connection) { if(!Site::is_local_addr(c$id$resp_h)) { Intel::seen([$host=c$id$resp_h, $conn=c, $where=Conn::IN_RESP]); } } -- - Justin Azoff From tgdesrochers at gmail.com Mon Nov 23 13:58:15 2015 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Mon, 23 Nov 2015 16:58:15 -0500 Subject: [Bro] Intel Framework Issues In-Reply-To: References: <8F46B608-6C8C-4C9A-AA69-845B50879CBF@gmail.com> Message-ID: I would have to agree with the majority here. I'd like the availability to turn it on per sensor or not. Some places are just more sensitive than others. Also I think looking for every connection outbound is very useful because it can show a compromised host on the network. On Nov 23, 2015 4:48 PM, "Azoff, Justin S" wrote: > > > On Nov 23, 2015, at 3:48 PM, Seth Hall wrote: > > > >> > >> On Nov 23, 2015, at 1:47 PM, Josh Liburdi > wrote: > >> > >> I think the most common gotcha for IP addresses is that they will only > appear in intel.log when they are a part of a successful TCP connection. > Unsuccessful TCP connections and non-TCP connections will not appear in the > log. > > > > Thanks for pointing that out Josh! I do want to point out the reasoning > behind that as a default decision. My thinking was that spoofed packets > could cause false hits and generally non-responded-to probes coming from > intelligence hosts aren?t all that interesting. Is there general agreement > that that?s the right approach or should single packets hit on intelligence > items by default? > > > > I think there is a high bar to clear with adding that as default > behavior, but I?d like to hear from people actively doing incident response > on their thoughts. > > > > I should also point out that we can certainly include a script that > matches on non-complete connections with Bro even if we don?t load it by > default. > > > > .Seth > > > IN_ORIG / IN_RESP may help with this > > seen IN_RESP in a failed outbound connection to a known phishing site, > useful to know > > seen IN_ORIG in a failed incoming port 22 connection from a known ssh > scanner, probably just noise. > seen in IN_RESP in a failed outbound port 22 connection to that same known > ssh scanner, useful to know > > Which I guess would mean something like > > event connection_i_forget(c: connection) { > if(!Site::is_local_addr(c$id$resp_h)) { > Intel::seen([$host=c$id$resp_h, $conn=c, $where=Conn::IN_RESP]); > } > } > > > > -- > - Justin Azoff > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151123/96fd5be9/attachment.html From mabuchan at gmail.com Mon Nov 23 14:17:22 2015 From: mabuchan at gmail.com (Mark Buchanan) Date: Mon, 23 Nov 2015 16:17:22 -0600 Subject: [Bro] Intel Framework Issues In-Reply-To: References: <8F46B608-6C8C-4C9A-AA69-845B50879CBF@gmail.com> Message-ID: Could it be added as a flag in the intel feed data file? Included in the Metadata items - granted, this is a little more actionable than metadata, but there is already meta.do_notice. Could there be a meta.log_any_conn (or something like that). It could default to the full connection state and then those who wanted to see any hits, could turn change the option for those items they want any notification regardless of state. I think it's valuable to provide the options, but can see both sides of the fence. Just like others, however, I'm not sure what the performance penalty would be on enabling something like this by default. Mark On Mon, Nov 23, 2015 at 3:58 PM, Tim Desrochers wrote: > I would have to agree with the majority here. I'd like the availability to > turn it on per sensor or not. Some places are just more sensitive than > others. > > Also I think looking for every connection outbound is very useful because > it can show a compromised host on the network. > On Nov 23, 2015 4:48 PM, "Azoff, Justin S" wrote: > >> >> > On Nov 23, 2015, at 3:48 PM, Seth Hall wrote: >> > >> >> >> >> On Nov 23, 2015, at 1:47 PM, Josh Liburdi >> wrote: >> >> >> >> I think the most common gotcha for IP addresses is that they will only >> appear in intel.log when they are a part of a successful TCP connection. >> Unsuccessful TCP connections and non-TCP connections will not appear in the >> log. >> > >> > Thanks for pointing that out Josh! I do want to point out the >> reasoning behind that as a default decision. My thinking was that spoofed >> packets could cause false hits and generally non-responded-to probes coming >> from intelligence hosts aren?t all that interesting. Is there general >> agreement that that?s the right approach or should single packets hit on >> intelligence items by default? >> > >> > I think there is a high bar to clear with adding that as default >> behavior, but I?d like to hear from people actively doing incident response >> on their thoughts. >> > >> > I should also point out that we can certainly include a script that >> matches on non-complete connections with Bro even if we don?t load it by >> default. >> > >> > .Seth >> >> >> IN_ORIG / IN_RESP may help with this >> >> seen IN_RESP in a failed outbound connection to a known phishing site, >> useful to know >> >> seen IN_ORIG in a failed incoming port 22 connection from a known ssh >> scanner, probably just noise. >> seen in IN_RESP in a failed outbound port 22 connection to that same >> known ssh scanner, useful to know >> >> Which I guess would mean something like >> >> event connection_i_forget(c: connection) { >> if(!Site::is_local_addr(c$id$resp_h)) { >> Intel::seen([$host=c$id$resp_h, $conn=c, $where=Conn::IN_RESP]); >> } >> } >> >> >> >> -- >> - Justin Azoff >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Mark Buchanan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151123/8882b635/attachment-0001.html From audrjon at gmail.com Mon Nov 23 21:58:49 2015 From: audrjon at gmail.com (AJ) Date: Tue, 24 Nov 2015 07:58:49 +0200 Subject: [Bro] Intel Framework Issues Message-ID: Hi, The one thing that's not working really - is EMAIL indicators. On my previous installation it worked well (2.2), but after that it broke. You can easily test it in http://try.bro.org/ (just upload your own pcap file with smtp connection). Somebody pointed here this issue early this year, but it is still not resolved. Regards, Audrius -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151124/57718211/attachment.html From jan.grashofer at cern.ch Tue Nov 24 03:58:35 2015 From: jan.grashofer at cern.ch (Jan Grashofer) Date: Tue, 24 Nov 2015 11:58:35 +0000 Subject: [Bro] Intel Framework Issues In-Reply-To: References: Message-ID: Hi, I like the idea of using the $where, because it was the first thing that came to my mind, too. We just need to make sure that the behavior is well documented. Regards, Jan P.S.: I've already opened a ticket for the mail issue (https://bro-tracker.atlassian.net/browse/BIT-1507). -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151124/6d797ecf/attachment.html From abhall1 at yahoo.com Tue Nov 24 07:13:24 2015 From: abhall1 at yahoo.com (Adam Hall) Date: Tue, 24 Nov 2015 15:13:24 +0000 (UTC) Subject: [Bro] Writing Bro logs as a different user References: <1867229615.10002789.1448378004896.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <1867229615.10002789.1448378004896.JavaMail.yahoo@mail.yahoo.com> Good Evening Bro Community, I have an issue I need resolving about running bro as a user or group or at least writing logs as a certain user or group. I have a script that performs a search through my bro logs, groups the results by file location(conn,http,dns),applies the headers of each file type to each group, and then formats everything in csv. ?Next, I pull down the data through a simple web interface. The problem I am having is that apache can't access my bro logs and I don't see a way that I can just tell broto write all it's logs as a certain user. 1) Is there an option to tell bro to write logs as user somebody or group somebody?2) Is there a way to run bro as a different user other than root? I am using bro-2.4.I have looked through the ./configure and there are no options for group or user in there. Thanks, Adam Hall -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151124/c9476200/attachment.html From audrjon at gmail.com Tue Nov 24 09:48:52 2015 From: audrjon at gmail.com (AJ) Date: Tue, 24 Nov 2015 19:48:52 +0200 Subject: [Bro] Intel Framework Issues In-Reply-To: References: Message-ID: <6CBEF602-92C3-4906-8616-DF30542F8630@gmail.com> Thanks a lot! Regards, Audrius > On 24 Nov 2015, at 13:58, Jan Grashofer wrote: > > Hi, > > I like the idea of using the $where, because it was the first thing that came to my mind, too. We just need to make sure that the behavior is well documented. > > Regards, > Jan > > P.S.: I've already opened a ticket for the mail issue (https://bro-tracker.atlassian.net/browse/BIT-1507). -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151124/aa683806/attachment.html From sven at dreyer-net.de Tue Nov 24 12:03:29 2015 From: sven at dreyer-net.de (Sven Dreyer) Date: Tue, 24 Nov 2015 21:03:29 +0100 Subject: [Bro] IP address parsing l10n problem Message-ID: <5654C291.6030202@dreyer-net.de> Dear list, I run bro 2.4.1 in a manager+proxy+worker configuration. When running "broctl status" on the manager, I get: Error: must run broctl only on manager node Although I am on the manager node. This seems to arise from around line 163 in /usr/lib/broctl/BroControl/execute.py, which tries to eliminate the string "addr:" in ifconfig's output. Unfortunately, the german localized ifconfig output contains "Adresse:", so in my case this prefix does not match "addr:" and is not removed. Running LANG= broctl status makes broctl run as expected. Just wanted to let you know about it, since it took some time to figure out the problem is l10n related. Thanks, Sven From dnthayer at illinois.edu Tue Nov 24 13:19:53 2015 From: dnthayer at illinois.edu (Daniel Thayer) Date: Tue, 24 Nov 2015 15:19:53 -0600 Subject: [Bro] IP address parsing l10n problem In-Reply-To: <5654C291.6030202@dreyer-net.de> References: <5654C291.6030202@dreyer-net.de> Message-ID: <5654D479.4010304@illinois.edu> There is already a ticket for this issue: https://bro-tracker.atlassian.net/browse/BIT-1511 If you run this: LANG=C /sbin/ifconfig -a Does it still show the German localized output? -Daniel On 11/24/2015 02:03 PM, Sven Dreyer wrote: > Dear list, > > I run bro 2.4.1 in a manager+proxy+worker configuration. When running > "broctl status" on the manager, I get: > > Error: must run broctl only on manager node > > Although I am on the manager node. > > This seems to arise from around line 163 in > /usr/lib/broctl/BroControl/execute.py, which tries to eliminate the > string "addr:" in ifconfig's output. > > Unfortunately, the german localized ifconfig output contains "Adresse:", > so in my case this prefix does not match "addr:" and is not removed. > > Running > > LANG= broctl status > > makes broctl run as expected. > > Just wanted to let you know about it, since it took some time to figure > out the problem is l10n related. > > Thanks, > Sven From derek at criticalstack.com Tue Nov 24 13:28:14 2015 From: derek at criticalstack.com (Derek Ditch) Date: Tue, 24 Nov 2015 15:28:14 -0600 Subject: [Bro] Intel Framework Issues In-Reply-To: References: <8F46B608-6C8C-4C9A-AA69-845B50879CBF@gmail.com> Message-ID: On 2015-11-23, 15:40, "Azoff, Justin S" wrote: >IN_ORIG / IN_RESP may help with this > >seen IN_RESP in a failed outbound connection to a known phishing site, useful to know > >seen IN_ORIG in a failed incoming port 22 connection from a known ssh scanner, probably just noise. >seen in IN_RESP in a failed outbound port 22 connection to that same known ssh scanner, useful to know > >Which I guess would mean something like > >event connection_i_forget(c: connection) { > if(!Site::is_local_addr(c$id$resp_h)) { > Intel::seen([$host=c$id$resp_h, $conn=c, $where=Conn::IN_RESP]); > } >} Something that I?d add to Justin?s approach is quasi-state for non-TCP. I?d have to think how best to write the event for a bit, but basically apply the same logic to ICMP/UDP. But also catch if seen IN_ORIG of a UDP connection and there is any response at all. Maybe extend the Crowdstrike script and apply is_local_addr filter. Something like (could be glitches, haven?t tested this yet): # Source: https://gist.github.com/dcode/dfe6026fd9865fb8e1ab @load base/frameworks/intel @load policy/frameworks/intel/seen/where-locations event connection_state_remove(c: connection) { if ( c$conn?$proto && ( c$conn$proto != tcp || ( c$conn?$history && c$conn$proto == tcp && "h" !in c$conn$history ) ) ) { if ( !Site::is_local_addr(c$id$resp_h) ) { Intel::seen([$host=c$id$resp_h, $conn=c, $where=Conn::IN_RESP]); } else if ( Site::is_local_addr(c$id$orig_h) && c$resp_pkts > 0 ) { Intel::seen([$host=c$id$orig_h, $conn=c, $where=Conn::IN_ORIG]); } } } Of course, you could definitely parameterize this behavior like known-hosts so it?s easier to configure for incident responders. ? Derek Ditch derek at criticalstack.com GPG: 0x2543A3B5 From liburdi.joshua at gmail.com Tue Nov 24 15:49:27 2015 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Tue, 24 Nov 2015 18:49:27 -0500 Subject: [Bro] Intel Framework Issues In-Reply-To: References: <8F46B608-6C8C-4C9A-AA69-845B50879CBF@gmail.com> Message-ID: Generally speaking, I favor grabbing more data over grabbing a portion of the data, so I prefer to see all of the IP indicators in every connection, then process the output outside of Bro. That said, the suggestions of checking for non-TCP / unestablished TCP IN_RESP connections makes sense if a sensor is resource constrained. I also like the idea of getting more granular with indicator-related activity (I do a bit of this, just not shared publicly), but I wonder if those specific use cases should be their own policy scripts instead of default behavior. More indicator-related policy scripts could give users better options. Josh > On Nov 24, 2015, at 4:28 PM, Derek Ditch wrote: > > On 2015-11-23, 15:40, "Azoff, Justin S" wrote: > > >> IN_ORIG / IN_RESP may help with this >> >> seen IN_RESP in a failed outbound connection to a known phishing site, useful to know >> >> seen IN_ORIG in a failed incoming port 22 connection from a known ssh scanner, probably just noise. >> seen in IN_RESP in a failed outbound port 22 connection to that same known ssh scanner, useful to know >> >> Which I guess would mean something like >> >> event connection_i_forget(c: connection) { >> if(!Site::is_local_addr(c$id$resp_h)) { >> Intel::seen([$host=c$id$resp_h, $conn=c, $where=Conn::IN_RESP]); >> } >> } > > Something that I?d add to Justin?s approach is quasi-state for non-TCP. I?d have to think how best to write the event for a bit, but basically apply the same logic to ICMP/UDP. But also catch if seen IN_ORIG of a UDP connection and there is any response at all. Maybe extend the Crowdstrike script and apply is_local_addr filter. > > Something like (could be glitches, haven?t tested this yet): > > # Source: https://gist.github.com/dcode/dfe6026fd9865fb8e1ab > @load base/frameworks/intel > @load policy/frameworks/intel/seen/where-locations > > event connection_state_remove(c: connection) > { > if ( c$conn?$proto && ( c$conn$proto != tcp || ( c$conn?$history && c$conn$proto == tcp && "h" !in c$conn$history ) ) ) > { > if ( !Site::is_local_addr(c$id$resp_h) ) > { > Intel::seen([$host=c$id$resp_h, $conn=c, $where=Conn::IN_RESP]); > } > else if ( Site::is_local_addr(c$id$orig_h) && c$resp_pkts > 0 ) > { > Intel::seen([$host=c$id$orig_h, $conn=c, $where=Conn::IN_ORIG]); > } > } > } > > > Of course, you could definitely parameterize this behavior like known-hosts so it?s easier to configure for incident responders. > > ? > Derek Ditch > derek at criticalstack.com > GPG: 0x2543A3B5 > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From sven at dreyer-net.de Tue Nov 24 22:49:33 2015 From: sven at dreyer-net.de (Sven Dreyer) Date: Wed, 25 Nov 2015 07:49:33 +0100 Subject: [Bro] IP address parsing l10n problem In-Reply-To: <5654D479.4010304@illinois.edu> References: <5654C291.6030202@dreyer-net.de> <5654D479.4010304@illinois.edu> Message-ID: <565559FD.5040406@dreyer-net.de> Am 24.11.2015 um 22:19 schrieb Daniel Thayer: > If you run this: > LANG=C /sbin/ifconfig -a > > Does it still show the German localized output? No, setting LANG to C gives the same result as setting an empty locale, I get "inet addr:" in both cases. Thanks, Sven From derek at criticalstack.com Wed Nov 25 05:08:39 2015 From: derek at criticalstack.com (derek at criticalstack.com) Date: Wed, 25 Nov 2015 13:08:39 +0000 (UTC) Subject: [Bro] Problem with connections in S1 and SF state In-Reply-To: <564DF725.6010604@dreyer-net.de> References: <564B9060.1080304@dreyer-net.de> <5A214812-FF4A-4360-B643-EFCC1516F869@illinois.edu> <564DF725.6010604@dreyer-net.de> Message-ID: Sven, Try running the pcap through your local policy scripts and check the output: bro -r file.pcap local I don't otherwise have a specific clue why this could happen, but it's best to compare the same process. -Derek From: Sven Dreyer Sent: Thursday, November 19, 10:34 Subject: Re: [Bro] Problem with connections in S1 and SF state To: bro at bro.org, Azoff, Justin S Justin, thanks for the hint. I should indeed have checked the history field. But even for connections that do not start with d or D in the history field, I see the same behaviour. Source and destination is still twisted: 1447675087.121817 CjRCD61gNErucciPb8 87.144.16.xxx 50993 192.168.100.yyy 26577 tcp ssl 83.596659 1432 2619 S1 F T 0 ShADad 18 2164 15 3231 (empty) Bro is configured to listen to a bridge interface (br0). But I also have running a dumpcap process writing all packets to pcap files. Interestingly, if I feed the corresponding pcap file to bro (bro -r file.pcap), I get source and destination in the right order: 1447675087.121817 C2AvJf3WgcdiBlYfS4 192.168.100.yyy 26577 87.144.16.xxx 50993 tcp ssl 83.596659 1432 2619 S1 - - 0 ShADad 18 2164 15 3231 (empty) Does anybody have an explanation for this? Thanks, Sven Am 17.11.2015 um 21:53 schrieb Azoff, Justin S: > You should really be looking at the history field: > > history: string &log &optional > Records the state history of connections as a string of letters. The meaning of those letters is: > > Letter Meaning > s a SYN w/o the ACK bit set > h a SYN+ACK (?handshake?) > a a pure ACK > d packet with payload (?data?) > f packet with FIN bit set > r packet with RST bit set > c packet with a bad checksum > i inconsistent packet (e.g. SYN+RST bits both set) > If the event comes from the originator, the letter is in upper-case; if it comes from the responder, it?s in lower-case. Multiple packets of the same type will only be noted once (e.g. we only record one ?d? in each direction, regardless of how many data packets were seen.) > > So any connection that starts with D or d means bro missed the initial syn handshake (Sh) > > _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151125/3c211a8a/attachment-0001.html From thomastan81 at gmail.com Thu Nov 26 01:18:37 2015 From: thomastan81 at gmail.com (Thomas Tan) Date: Thu, 26 Nov 2015 10:18:37 +0100 Subject: [Bro] TCP options of a SYN packet Message-ID: Dear All, Just wondering if anyone knows a way (an event) to obtain TCP options of a SYN packet? Your help will be very much appreciated. Thank you. Best regards, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151126/f7d96d65/attachment.html From jan.grashofer at cern.ch Thu Nov 26 03:16:02 2015 From: jan.grashofer at cern.ch (Jan Grashofer) Date: Thu, 26 Nov 2015 11:16:02 +0000 Subject: [Bro] TCP options of a SYN packet In-Reply-To: References: Message-ID: Hi Thomas, there is the tcp_option event, that might help you (see https://www.bro.org/sphinx/scripts/base/bif/plugins/Bro_TCP.events.bif.bro.html#id-tcp_option). If that does not fit for you, you might have a look into the TCPRS-plugin (https://github.com/bro/bro-plugins/tree/master/tcprs/scripts/Bro/TCPRS). I have never used it but I think it also parses some TCP options and thus might be a good starting point. Best regards, Jan ________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Thomas Tan [thomastan81 at gmail.com] Sent: Thursday, November 26, 2015 10:18 To: bro at bro.org Subject: [Bro] TCP options of a SYN packet Dear All, Just wondering if anyone knows a way (an event) to obtain TCP options of a SYN packet? Your help will be very much appreciated. Thank you. Best regards, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151126/525be524/attachment.html From thomastan81 at gmail.com Thu Nov 26 03:29:00 2015 From: thomastan81 at gmail.com (Thomas Tan) Date: Thu, 26 Nov 2015 12:29:00 +0100 Subject: [Bro] TCP options of a SYN packet In-Reply-To: References: Message-ID: Dear Jan, Many thanks for you reply. I am using tcp_option event. However, it seems to me that the event can't tell which TCP options are from the SYN packet of a connection and which ones are from other packets of the connection. I think I will look into the TCPRS-plugin. Best regards, Thomas On 26 November 2015 at 12:16, Jan Grashofer wrote: > Hi Thomas, > > > > there is the tcp_option event, that might help you (see > https://www.bro.org/sphinx/scripts/base/bif/plugins/Bro_TCP.events.bif.bro.html#id-tcp_option). > If that does not fit for you, you might have a look into the TCPRS-plugin ( > https://github.com/bro/bro-plugins/tree/master/tcprs/scripts/Bro/TCPRS). > I have never used it but I think it also parses some TCP options and thus > might be a good starting point. > > > > Best regards, > > Jan > > > ------------------------------ > *From:* bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Thomas Tan > [thomastan81 at gmail.com] > *Sent:* Thursday, November 26, 2015 10:18 > *To:* bro at bro.org > *Subject:* [Bro] TCP options of a SYN packet > > Dear All, > > Just wondering if anyone knows a way (an event) to obtain TCP options of a > SYN packet? > > Your help will be very much appreciated. > > Thank you. > > Best regards, > > Thomas > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151126/5b2cfe1a/attachment.html From zied.turki at outlook.com Fri Nov 27 02:54:25 2015 From: zied.turki at outlook.com (Zied Turki) Date: Fri, 27 Nov 2015 10:54:25 +0000 Subject: [Bro] Duqu script Message-ID: Dear all, I tried to test the bro script to log the Duqu attack published through Github. The broctl check failed and here below the output : "error in /usr/local/bro/share/bro/policy/bro-scripts/duqu.bro, line 81: no such field in record (HTTP::c$http$mime_types)" I've understood that $mime_type has changed in the new bro version and I've tried to change it in the script with "resp_mime_types" . Here below the new output : error in /usr/local/bro/share/bro/base/protocols/http/./entities.bro, line 27 and /usr/local/bro/share/bro/policy/bro-scripts/duqu.bro, line 81: pattern requires string index (vector of string and /^?(image\/jpeg)$?/) I am new to bro scripts. Please, I need your help to understand how to manage this kind of errors. Anyone could help please ? Please find below the link to the original script : https://github.com/mavam/brospects/blob/master/bro/duqu.bro Many thanks, BR, Zied -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151127/228cd354/attachment.html From jan.grashofer at cern.ch Fri Nov 27 04:10:04 2015 From: jan.grashofer at cern.ch (Jan Grashofer) Date: Fri, 27 Nov 2015 12:10:04 +0000 Subject: [Bro] Duqu script In-Reply-To: References: Message-ID: Hi Zied, while mime_types was a string, resp_mime_types is a vector of strings. If I am not mistaken it contains all matching mime types ordered by probability. Unfortunately I am not able to find proof for this in the documentation. However, to fix your issue you could loop through the vector or just use the first element resp_mime_types[0]. Best regards, Jan ________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Zied Turki [zied.turki at outlook.com] Sent: Friday, November 27, 2015 11:54 To: bro at bro.org Subject: [Bro] Duqu script Dear all, I tried to test the bro script to log the Duqu attack published through Github. The broctl check failed and here below the output : "error in /usr/local/bro/share/bro/policy/bro-scripts/duqu.bro, line 81: no such field in record (HTTP::c$http$mime_types)" I've understood that $mime_type has changed in the new bro version and I've tried to change it in the script with "resp_mime_types" . Here below the new output : error in /usr/local/bro/share/bro/base/protocols/http/./entities.bro, line 27 and /usr/local/bro/share/bro/policy/bro-scripts/duqu.bro, line 81: pattern requires string index (vector of string and /^?(image\/jpeg)$?/) I am new to bro scripts. Please, I need your help to understand how to manage this kind of errors. Anyone could help please ? Please find below the link to the original script : https://github.com/mavam/brospects/blob/master/bro/duqu.bro Many thanks, BR, Zied -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151127/a338622e/attachment.html From daniel.guerra69 at gmail.com Fri Nov 27 13:14:09 2015 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Fri, 27 Nov 2015 22:14:09 +0100 Subject: [Bro] Bro Elasticsearch 2+ Message-ID: <7F86A961-4F10-44B5-BE6D-7E3EE2480F62@gmail.com> Hi, I?ve been working a while on the elasticsearch integration with bro. There have been some issues like timestamp, the elstic 2.0 no dot and the name/type changes in the logging (version ?). See my changes in https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/Dockerfile It was made pragmatic, some changes where just a quick hack. The latest release is stable. https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/ Regards, Daniel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151127/d993eb1a/attachment.html From jlay at slave-tothe-box.net Sat Nov 28 05:16:57 2015 From: jlay at slave-tothe-box.net (James Lay) Date: Sat, 28 Nov 2015 06:16:57 -0700 Subject: [Bro] Bro Elasticsearch 2+ In-Reply-To: <7F86A961-4F10-44B5-BE6D-7E3EE2480F62@gmail.com> References: <7F86A961-4F10-44B5-BE6D-7E3EE2480F62@gmail.com> Message-ID: <1448716617.3583.1.camel@JamesiMac> On Fri, 2015-11-27 at 22:14 +0100, Daniel Guerra wrote: > Hi, > > > > I?ve been working a while on the elasticsearch integration with bro. > There have been some issues like timestamp, the elstic 2.0 no dot > and the name/type changes in the logging (version ?). See my changes > in https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/Dockerfile > It was made pragmatic, some changes where just a quick hack. > The latest release is stable. > https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/ > > > Regards, > > > Daniel > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro Thanks for this Daniel....I've been looking at the new ES as well....seems like a large pain now...this will help me out. James -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151128/7b483167/attachment.html From daniel.guerra69 at gmail.com Sat Nov 28 05:47:11 2015 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Sat, 28 Nov 2015 14:47:11 +0100 Subject: [Bro] Bro Elasticsearch 2+ In-Reply-To: <1448716617.3583.1.camel@JamesiMac> References: <7F86A961-4F10-44B5-BE6D-7E3EE2480F62@gmail.com> <1448716617.3583.1.camel@JamesiMac> Message-ID: <4DA8C05B-9653-4BAD-BF5E-F50B1788F4AE@gmail.com> Hi, Diving deeper in the problem, beside the dot & timestamp, it can be solved with bro config and elastic mapping. I didn?t find the exact place where the dot is placed in the fieldnames, but I found the point it was writing the json and changes JSON.cc (ugly but pragmatic). About the bro script script structure there is a need for naming conventions and type. Like the version field which changes type all the time (ssl ssh socks etc.). Check /scripts/bro-map.sh for geo_point and not analyzed fields (when you let elastic index the data it cuts the results into words). In this script also the shards and copies are set. Mapping needs to be done before writing. > On 28 Nov 2015, at 14:16, James Lay wrote: > > On Fri, 2015-11-27 at 22:14 +0100, Daniel Guerra wrote: >> Hi, >> >> >> I?ve been working a while on the elasticsearch integration with bro. >> There have been some issues like timestamp, the elstic 2.0 no dot >> and the name/type changes in the logging (version ?). See my changes >> in https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/Dockerfile It was made pragmatic, some changes where just a quick hack. >> The latest release is stable. >> https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/ >> >> Regards, >> >> >> Daniel >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > Thanks for this Daniel....I've been looking at the new ES as well....seems like a large pain now...this will help me out. > > James > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151128/bba3e91d/attachment.html From daniel.guerra69 at gmail.com Sat Nov 28 14:14:30 2015 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Sat, 28 Nov 2015 23:14:30 +0100 Subject: [Bro] Duqu script In-Reply-To: References: Message-ID: Or you could use this from line 77 if ( c$id$orig_h in duqus && c$http?$resp_mime_types ) { local mime_num:count; for (mime_num in c$http$resp_mime_types) { if ( duqus[c$id$orig_h] == JPEG_REQUEST && c$http$status_code == 200 && /image\/jpeg/ in c$http$resp_mime_types[mime_num] ) { duqus[c$id$orig_h] = JPEG_REPLY; NOTICE([$note=Potential_Duqu_Infection, $msg=fmt("Initial Duqu JPEG exchange"), $conn=c]); } else delete duqus[c$id$orig_h]; # Purge unnecessary state early. } } > On 27 Nov 2015, at 13:10, Jan Grashofer wrote: > > Hi Zied, > > while mime_types was a string, resp_mime_types is a vector of strings. If I am not mistaken it contains all matching mime types ordered by probability. Unfortunately I am not able to find proof for this in the documentation. However, to fix your issue you could loop through the vector or just use the first element resp_mime_types[0]. > > Best regards, > Jan > > From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Zied Turki [zied.turki at outlook.com] > Sent: Friday, November 27, 2015 11:54 > To: bro at bro.org > Subject: [Bro] Duqu script > > Dear all, > > I tried to test the bro script to log the Duqu attack published through Github. The broctl check failed and here below the output : > > "error in /usr/local/bro/share/bro/policy/bro-scripts/duqu.bro, line 81: no such field in record (HTTP::c$http$mime_types)" > > I've understood that $mime_type has changed in the new bro version and I've tried to change it in the script with "resp_mime_types" . Here below the new output : > > error in /usr/local/bro/share/bro/base/protocols/http/./entities.bro, line 27 and /usr/local/bro/share/bro/policy/bro-scripts/duqu.bro, line 81: pattern requires string index (vector of string and /^?(image\/jpeg)$?/) > > I am new to bro scripts. Please, I need your help to understand how to manage this kind of errors. Anyone could help please ? > > Please find below the link to the original script : > https://github.com/mavam/brospects/blob/master/bro/duqu.bro > > > Many thanks, > > BR, > Zied > > > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151128/5f6cb85a/attachment.html From seth at icir.org Sat Nov 28 21:27:06 2015 From: seth at icir.org (Seth Hall) Date: Sun, 29 Nov 2015 00:27:06 -0500 Subject: [Bro] Bro Elasticsearch 2+ In-Reply-To: <7F86A961-4F10-44B5-BE6D-7E3EE2480F62@gmail.com> References: <7F86A961-4F10-44B5-BE6D-7E3EE2480F62@gmail.com> Message-ID: > On Nov 27, 2015, at 4:14 PM, Daniel Guerra wrote: > > There have been some issues like timestamp, the elstic 2.0 no dot > and the name/type changes in the logging There may be more updates later, but I just pushed a fix for the fields in the SIP log that were accidentally strings instead of counts (along with some other cleanup). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From josh.guild at morphick.com Sun Nov 29 18:27:37 2015 From: josh.guild at morphick.com (Josh Guild) Date: Mon, 30 Nov 2015 02:27:37 +0000 Subject: [Bro] Reverse Lookups? Message-ID: Hi all, Does anyone know if Bro signatures support reverse lookups? I attempted one but it failed to compile. Just wanted to check before I pull my hair out messing with the syntax :) Thanks! Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151130/4dcc1dc5/attachment.html From anthony.kasza at gmail.com Sun Nov 29 19:07:57 2015 From: anthony.kasza at gmail.com (anthony kasza) Date: Sun, 29 Nov 2015 19:07:57 -0800 Subject: [Bro] Reverse Lookups? In-Reply-To: References: Message-ID: See here: https://www.bro.org/sphinx-git/frameworks/signatures.html And here: http://flex.sourceforge.net/manual/Patterns.html -AK On Nov 29, 2015 6:32 PM, "Josh Guild" wrote: > Hi all, > > Does anyone know if Bro signatures support reverse lookups? I attempted > one but it failed to compile. Just wanted to check before I pull my hair > out messing with the syntax :) > > Thanks! > > Josh > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151129/fa1bf882/attachment.html From josh.guild at morphick.com Sun Nov 29 19:14:20 2015 From: josh.guild at morphick.com (Josh Guild) Date: Mon, 30 Nov 2015 03:14:20 +0000 Subject: [Bro] Reverse Lookups? In-Reply-To: References: Message-ID: Cool, thanks for the reference. I'll poke at this tomorrow and see what I come up with. On Sun, Nov 29, 2015, 22:08 anthony kasza wrote: > See here: > https://www.bro.org/sphinx-git/frameworks/signatures.html > And here: > http://flex.sourceforge.net/manual/Patterns.html > > -AK > On Nov 29, 2015 6:32 PM, "Josh Guild" wrote: > >> Hi all, >> >> Does anyone know if Bro signatures support reverse lookups? I attempted >> one but it failed to compile. Just wanted to check before I pull my hair >> out messing with the syntax :) >> >> Thanks! >> >> Josh >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151130/766effc9/attachment.html From zied.turki at outlook.com Mon Nov 30 02:43:36 2015 From: zied.turki at outlook.com (Zied Turki) Date: Mon, 30 Nov 2015 10:43:36 +0000 Subject: [Bro] Duqu script In-Reply-To: References: , Message-ID: Hello, Thank you for your help, I prefer your version. Many thanks, BR, Zied Subject: Re: [Bro] Duqu script From: daniel.guerra69 at gmail.com Date: Sat, 28 Nov 2015 23:14:30 +0100 CC: zied.turki at outlook.com; bro at bro.org To: jan.grashofer at cern.ch Or you could use this from line 77if ( c$id$orig_h in duqus && c$http?$resp_mime_types ){ local mime_num:count; for (mime_num in c$http$resp_mime_types) { if ( duqus[c$id$orig_h] == JPEG_REQUEST && c$http$status_code == 200 && /image\/jpeg/ in c$http$resp_mime_types[mime_num] ) { duqus[c$id$orig_h] = JPEG_REPLY; NOTICE([$note=Potential_Duqu_Infection, $msg=fmt("Initial Duqu JPEG exchange"), $conn=c]); } else delete duqus[c$id$orig_h]; # Purge unnecessary state early. }}On 27 Nov 2015, at 13:10, Jan Grashofer wrote:Hi Zied, while mime_types was a string, resp_mime_types is a vector of strings. If I am not mistaken it contains all matching mime types ordered by probability. Unfortunately I am not able to find proof for this in the documentation. However, to fix your issue you could loop through the vector or just use the first element resp_mime_types[0]. Best regards,Jan From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Zied Turki [zied.turki at outlook.com] Sent: Friday, November 27, 2015 11:54 To: bro at bro.org Subject: [Bro] Duqu script Dear all, I tried to test the bro script to log the Duqu attack published through Github. The broctl check failed and here below the output : "error in /usr/local/bro/share/bro/policy/bro-scripts/duqu.bro, line 81: no such field in record (HTTP::c$http$mime_types)" I've understood that $mime_type has changed in the new bro version and I've tried to change it in the script with "resp_mime_types" . Here below the new output : error in /usr/local/bro/share/bro/base/protocols/http/./entities.bro, line 27 and /usr/local/bro/share/bro/policy/bro-scripts/duqu.bro, line 81: pattern requires string index (vector of string and /^?(image\/jpeg)$?/) I am new to bro scripts. Please, I need your help to understand how to manage this kind of errors. Anyone could help please ? Please find below the link to the original script : https://github.com/mavam/brospects/blob/master/bro/duqu.bro Many thanks, BR, Zied _______________________________________________Bro mailing listbro at bro-ids.orghttp://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151130/db4f1d34/attachment.html From zied.turki at outlook.com Mon Nov 30 02:48:54 2015 From: zied.turki at outlook.com (Zied Turki) Date: Mon, 30 Nov 2015 10:48:54 +0000 Subject: [Bro] SMB connections Message-ID: Hello Bro Community, I am working on the data exfiltration and I have just tested the Exfil Framework. I have noticed, that the script failed to detect file uploads from the file server using SMB protocol. Looking to the connections logs (conn.log), the SMB connections are unfortunately not logged. Would it be a known issue ? or should I tune some params ? Please note that the trafic arrives to Bro machine (I have checked using tcpdump). Many thanks, BR, Zied -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151130/d70a9257/attachment-0001.html From vladg at illinois.edu Mon Nov 30 09:08:17 2015 From: vladg at illinois.edu (Vlad Grigorescu) Date: Mon, 30 Nov 2015 11:08:17 -0600 Subject: [Bro] SMB connections In-Reply-To: References: Message-ID: Hi, The Exfil Framework is developed by someone from Reservoir Labs. Please contact them with any questions. That being said, note that SMB support in Bro is a best-effort implementation of part of the specification (and very different from what's actually seen on the wire), so detecting exfil over SMB likely won't work at all. --Vlad Zied Turki writes: > Hello Bro Community, > > I am working on the data exfiltration and I have just tested the Exfil Framework. > I have noticed, that the script failed to detect file uploads from the file server using SMB protocol. Looking to the connections logs (conn.log), the SMB connections are unfortunately not logged. > Would it be a known issue ? or should I tune some params ? > Please note that the trafic arrives to Bro machine (I have checked using tcpdump). > > Many thanks, > > BR, > Zied > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 800 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151130/d4c05488/attachment.bin From bglaze at gmail.com Mon Nov 30 09:50:27 2015 From: bglaze at gmail.com (Brandon Glaze) Date: Mon, 30 Nov 2015 09:50:27 -0800 Subject: [Bro] surgical file extraction Message-ID: Good day all, I am new to creating custom bro scripts, so please bear with me. I am looking to perform file extraction only on HTTP conversations to either a specific URL or destination IP address. I have some command and control traffic POST traffic that I want to capture with bro, as opposed to what I am using now. I am currently having to run TCPDumps on my sensors for traffic destined to a specific IP, then manually cut out the POST requests, redo the pcap, then run it through bro? tedious. Here is what I have, but cant get it to work (mainly due to a lack of bro script knowledge): global mime_to_ext: table[string] of string = { ["text/plain"] = "txt", ["text/html"] = "html", }; Event file_sniff(f: fa_file, meta: fa_metadata) { if ( f$source != "HTTP" ) return; if ( meta$rx_hosts != "123.123.123.123" ) return; if ( ! meta?$mime_type ) return; if ( meta$mime_type !in mime_to_ext ) return; local fname = fmt("%s-%s.%s", f$source, f$id, mime_to_ext[meta$mime_type]); print fmt("Extracting file %s", fname); Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=fname]); } I get the warning that the meta$rx_hosts isn?t valid. I have tried a ?redef record? statement, but it made it much worse. Am I even in the ballpark here? I wasnt sure if I needed to use some other means to identify the URL, then start file extraction. Appreciate any help. ===================== Brandon Glaze bglaze at gmail.com "Lead me, follow me, or get the hell out of my way." - General George Patton Jr -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151130/636da68a/attachment.html From liburdi.joshua at gmail.com Mon Nov 30 10:12:24 2015 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Mon, 30 Nov 2015 13:12:24 -0500 Subject: [Bro] surgical file extraction In-Reply-To: References: Message-ID: <51BCA40C-1E34-4FF0-876A-F8BBDFFB8374@gmail.com> You?re close! The field rx_hosts doesn?t exist in the meta record. If you want to get the destination IP, then you can do something similar to the first example on this page: https://www.bro.org/sphinx-git/frameworks/file-analysis.html In your event, you?d do something like this ? for ( cid in f$conns ) { if ( f$conns[cid]$id$resp_h == 123.123.123.123 ) { print "found IP 123.123.123.123"; } } This should get you started: http://try.bro.org/#/trybro/saved/27817 Josh > On Nov 30, 2015, at 12:50 PM, Brandon Glaze wrote: > > Good day all, > > I am new to creating custom bro scripts, so please bear with me. > > > I am looking to perform file extraction only on HTTP conversations to either a specific URL or destination IP address. I have some command and control traffic POST traffic that I want to capture with bro, as opposed to what I am using now. I am currently having to run TCPDumps on my sensors for traffic destined to a specific IP, then manually cut out the POST requests, redo the pcap, then run it through bro? tedious. > > > Here is what I have, but cant get it to work (mainly due to a lack of bro script knowledge): > > > global mime_to_ext: table[string] of string = { > > ["text/plain"] = "txt", > > ["text/html"] = "html", > > }; > > > Event file_sniff(f: fa_file, meta: fa_metadata) > > { > > if ( f$source != "HTTP" ) > > return; > > > if ( meta$rx_hosts != "123.123.123.123" ) > > return; > > > if ( ! meta?$mime_type ) > > return; > > > if ( meta$mime_type !in mime_to_ext ) > > return; > > > local fname = fmt("%s-%s.%s", f$source, f$id, mime_to_ext[meta$mime_type]); > > print fmt("Extracting file %s", fname); > > Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=fname]); > > } > > > I get the warning that the meta$rx_hosts isn?t valid. I have tried a ?redef record? statement, but it made it much worse. Am I even in the ballpark here? I wasnt sure if I needed to use some other means to identify the URL, then start file extraction. > > > Appreciate any help. > > ===================== > Brandon Glaze > bglaze at gmail.com > > "Lead me, follow me, or get the hell out of my way." > - General George Patton Jr > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151130/d38cfd3d/attachment.html From rrotsted at gmail.com Mon Nov 30 11:44:13 2015 From: rrotsted at gmail.com (Robert Rotsted) Date: Mon, 30 Nov 2015 11:44:13 -0800 Subject: [Bro] SMB connections In-Reply-To: References: Message-ID: Hi Zied, By default, the Exfil framework will only attach to flows originated by addresses in 10.0.0.0/8 that have a non-local responder. Try setting "ignore_local_dest_conn" to F in app-exfil-conn.bro. --bob On Mon, Nov 30, 2015 at 2:48 AM, Zied Turki wrote: > Hello Bro Community, > > I am working on the data exfiltration and I have just tested the Exfil > Framework. > I have noticed, that the script failed to detect file uploads from the file > server using SMB protocol. Looking to the connections logs (conn.log), the > SMB connections are unfortunately not logged. > Would it be a known issue ? or should I tune some params ? > Please note that the trafic arrives to Bro machine (I have checked using > tcpdump). > > Many thanks, > > BR, > Zied > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From bglaze at gmail.com Mon Nov 30 16:14:07 2015 From: bglaze at gmail.com (Brandon Glaze) Date: Mon, 30 Nov 2015 16:14:07 -0800 Subject: [Bro] surgical file extraction In-Reply-To: <51BCA40C-1E34-4FF0-876A-F8BBDFFB8374@gmail.com> References: <51BCA40C-1E34-4FF0-876A-F8BBDFFB8374@gmail.com> Message-ID: Josh, That worked great. Thanks a ton for the help. Any idea why when I load my new file extraction script, that I verified works great on standalone packet captures, I get the below error: error in /usr/local/bro/share/bro/policy/frameworks/control/controllee.bro, line 15: syntax error, at or near "module" I validated that if I comment out my @load line for the new file extraction script this error goes away in a "broctl check" check. ===================== Brandon Glaze bglaze at gmail.com "Lead me, follow me, or get the hell out of my way." - General George Patton Jr On Mon, Nov 30, 2015 at 10:12 AM, Josh Liburdi wrote: > You?re close! The field rx_hosts doesn?t exist in the meta record. If you > want to get the destination IP, then you can do something similar to the > first example on this page: > https://www.bro.org/sphinx-git/frameworks/file-analysis.html > > In your event, you?d do something like this ? > > for ( cid in f$conns ) > { > if ( f$conns[cid]$id$resp_h == 123.123.123.123 ) > { > print "found IP 123.123.123.123"; > } > } > > This should get you started: http://try.bro.org/#/trybro/saved/27817 > > Josh > > On Nov 30, 2015, at 12:50 PM, Brandon Glaze wrote: > > Good day all, > > I am new to creating custom bro scripts, so please bear with me. > > > I am looking to perform file extraction only on HTTP conversations to > either a specific URL or destination IP address. I have some command and > control traffic POST traffic that I want to capture with bro, as opposed to > what I am using now. I am currently having to run TCPDumps on my sensors > for traffic destined to a specific IP, then manually cut out the POST > requests, redo the pcap, then run it through bro? tedious. > > > Here is what I have, but cant get it to work (mainly due to a lack of bro > script knowledge): > > > global mime_to_ext: table[string] of string = { > > ["text/plain"] = "txt", > > ["text/html"] = "html", > > }; > > > Event file_sniff(f: fa_file, meta: fa_metadata) > > { > > if ( f$source != "HTTP" ) > > return; > > > if ( meta$rx_hosts != "123.123.123.123" ) > > return; > > > if ( ! meta?$mime_type ) > > return; > > > if ( meta$mime_type !in mime_to_ext ) > > return; > > > local fname = fmt("%s-%s.%s", f$source, f$id, > mime_to_ext[meta$mime_type]); > > print fmt("Extracting file %s", fname); > > Files::add_analyzer(f, Files::ANALYZER_EXTRACT, > [$extract_filename=fname]); > > } > > > I get the warning that the meta$rx_hosts isn?t valid. I have tried a > ?redef record? statement, but it made it much worse. Am I even in the > ballpark here? I wasnt sure if I needed to use some other means to identify > the URL, then start file extraction. > > > Appreciate any help. > ===================== > Brandon Glaze > bglaze at gmail.com > > "Lead me, follow me, or get the hell out of my way." > - General George Patton Jr > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151130/6f99e1ed/attachment.html From dnj0496 at gmail.com Mon Nov 30 16:21:16 2015 From: dnj0496 at gmail.com (Dk Jack) Date: Mon, 30 Nov 2015 16:21:16 -0800 Subject: [Bro] conditional loading Message-ID: Hi, Is it possible to perform conditional loading of bro script files ? In my local.bro file, I'd like to do something like this: if file 'filters.bro' exists { @load filters.bro } The file filters.bro may or may not exist because it's getting generated from an external program. Thanks for your help in advance. Dnj -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151130/58bccb2c/attachment.html From anthony.kasza at gmail.com Mon Nov 30 16:41:42 2015 From: anthony.kasza at gmail.com (anthony kasza) Date: Mon, 30 Nov 2015 16:41:42 -0800 Subject: [Bro] conditional loading In-Reply-To: References: Message-ID: See here: https://www.bro.org/sphinx/script-reference/directives.html -AK On Nov 30, 2015 4:36 PM, "Dk Jack" wrote: > Hi, > Is it possible to perform conditional loading of bro script files ? > In my local.bro file, I'd like to do something like this: > > if file 'filters.bro' exists > { > @load filters.bro > } > > The file filters.bro may or may not exist because it's getting generated > from an external program. > Thanks for your help in advance. > > Dnj > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151130/d758d7f0/attachment.html From dnj0496 at gmail.com Mon Nov 30 18:43:49 2015 From: dnj0496 at gmail.com (Dk Jack) Date: Mon, 30 Nov 2015 18:43:49 -0800 Subject: [Bro] conditional loading In-Reply-To: References: Message-ID: the directives don't seem to have file test operators. I tried to do the following... local c = "test -f filters.bro"; local cmd = Exec::Command($cmd=c); when (local res = Exec::run(cmd)) { @if (res$exit_code == 0) { @load filters.bro; } } However, I am getting parsing errors... --------------------------------------------------------------- referencing a local name in @if (res) invalid expression in @if (res$exit_code == 0) --------------------------------------------------------------- Dnj On Mon, Nov 30, 2015 at 4:41 PM, anthony kasza wrote: > See here: > https://www.bro.org/sphinx/script-reference/directives.html > > -AK > On Nov 30, 2015 4:36 PM, "Dk Jack" wrote: > >> Hi, >> Is it possible to perform conditional loading of bro script files ? >> In my local.bro file, I'd like to do something like this: >> >> if file 'filters.bro' exists >> { >> @load filters.bro >> } >> >> The file filters.bro may or may not exist because it's getting generated >> from an external program. >> Thanks for your help in advance. >> >> Dnj >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151130/01645876/attachment.html From anthony.kasza at gmail.com Mon Nov 30 19:16:33 2015 From: anthony.kasza at gmail.com (anthony kasza) Date: Mon, 30 Nov 2015 19:16:33 -0800 Subject: [Bro] conditional loading In-Reply-To: References: Message-ID: I have no clue if it would work but... You could try to make a global variable in the script you're testing for (something like global is_loaded: bool) then check if it is defined with @ifdef. If so, the script was loaded. -AK On Nov 30, 2015 6:43 PM, "Dk Jack" wrote: > the directives don't seem to have file test operators. I tried to do the > following... > > local c = "test -f filters.bro"; > local cmd = Exec::Command($cmd=c); > > when (local res = Exec::run(cmd)) > { > @if (res$exit_code == 0) > { > @load filters.bro; > } > } > > > However, I am getting parsing errors... > > --------------------------------------------------------------- > referencing a local name in @if (res) > invalid expression in @if (res$exit_code == 0) > --------------------------------------------------------------- > > Dnj > > > On Mon, Nov 30, 2015 at 4:41 PM, anthony kasza > wrote: > >> See here: >> https://www.bro.org/sphinx/script-reference/directives.html >> >> -AK >> On Nov 30, 2015 4:36 PM, "Dk Jack" wrote: >> >>> Hi, >>> Is it possible to perform conditional loading of bro script files ? >>> In my local.bro file, I'd like to do something like this: >>> >>> if file 'filters.bro' exists >>> { >>> @load filters.bro >>> } >>> >>> The file filters.bro may or may not exist because it's getting generated >>> from an external program. >>> Thanks for your help in advance. >>> >>> Dnj >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151130/995fd21c/attachment-0001.html From grant at grantstavely.com Mon Nov 30 19:42:17 2015 From: grant at grantstavely.com (Grant Stavely) Date: Mon, 30 Nov 2015 19:42:17 -0800 Subject: [Bro] conditional loading In-Reply-To: References: Message-ID: Innelegant but I just have the same process that creates populated files create empty ones for bro to @load in environments where they aren't applicable and e.g. a puppet template would be overkill. Grant > On Nov 30, 2015, at 19:16, anthony kasza wrote: > > I have no clue if it would work but... You could try to make a global variable in the script you're testing for (something like global is_loaded: bool) then check if it is defined with @ifdef. If so, the script was loaded. > > -AK > >> On Nov 30, 2015 6:43 PM, "Dk Jack" wrote: >> the directives don't seem to have file test operators. I tried to do the following... >> >> local c = "test -f filters.bro"; >> local cmd = Exec::Command($cmd=c); >> >> when (local res = Exec::run(cmd)) >> { >> @if (res$exit_code == 0) >> { >> @load filters.bro; >> } >> } >> >> >> However, I am getting parsing errors... >> >> --------------------------------------------------------------- >> referencing a local name in @if (res) >> invalid expression in @if (res$exit_code == 0) >> --------------------------------------------------------------- >> >> Dnj >> >> >>> On Mon, Nov 30, 2015 at 4:41 PM, anthony kasza wrote: >>> See here: >>> https://www.bro.org/sphinx/script-reference/directives.html >>> >>> -AK >>> >>>> On Nov 30, 2015 4:36 PM, "Dk Jack" wrote: >>>> Hi, >>>> Is it possible to perform conditional loading of bro script files ? >>>> In my local.bro file, I'd like to do something like this: >>>> >>>> if file 'filters.bro' exists >>>> { >>>> @load filters.bro >>>> } >>>> >>>> The file filters.bro may or may not exist because it's getting generated from an external program. >>>> Thanks for your help in advance. >>>> >>>> Dnj >>>> >>>> _______________________________________________ >>>> Bro mailing list >>>> bro at bro-ids.org >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151130/b7a64d36/attachment.html From dnj0496 at gmail.com Mon Nov 30 21:00:11 2015 From: dnj0496 at gmail.com (Dnj) Date: Mon, 30 Nov 2015 21:00:11 -0800 Subject: [Bro] conditional loading In-Reply-To: References: Message-ID: <9DD3C771-3488-4D11-8689-7622A10A38F6@gmail.com> Thanks Athiny, Grant, I'll try your suggestions. > On Nov 30, 2015, at 7:42 PM, Grant Stavely wrote: > > Innelegant but I just have the same process that creates populated files create empty ones for bro to @load in environments where they aren't applicable and e.g. a puppet template would be overkill. > > Grant > >> On Nov 30, 2015, at 19:16, anthony kasza wrote: >> >> I have no clue if it would work but... You could try to make a global variable in the script you're testing for (something like global is_loaded: bool) then check if it is defined with @ifdef. If so, the script was loaded. >> >> -AK >> >>> On Nov 30, 2015 6:43 PM, "Dk Jack" wrote: >>> the directives don't seem to have file test operators. I tried to do the following... >>> >>> local c = "test -f filters.bro"; >>> local cmd = Exec::Command($cmd=c); >>> >>> when (local res = Exec::run(cmd)) >>> { >>> @if (res$exit_code == 0) >>> { >>> @load filters.bro; >>> } >>> } >>> >>> >>> However, I am getting parsing errors... >>> >>> --------------------------------------------------------------- >>> referencing a local name in @if (res) >>> invalid expression in @if (res$exit_code == 0) >>> --------------------------------------------------------------- >>> >>> Dnj >>> >>> >>>> On Mon, Nov 30, 2015 at 4:41 PM, anthony kasza wrote: >>>> See here: >>>> https://www.bro.org/sphinx/script-reference/directives.html >>>> >>>> -AK >>>> >>>>> On Nov 30, 2015 4:36 PM, "Dk Jack" wrote: >>>>> Hi, >>>>> Is it possible to perform conditional loading of bro script files ? >>>>> In my local.bro file, I'd like to do something like this: >>>>> >>>>> if file 'filters.bro' exists >>>>> { >>>>> @load filters.bro >>>>> } >>>>> >>>>> The file filters.bro may or may not exist because it's getting generated from an external program. >>>>> Thanks for your help in advance. >>>>> >>>>> Dnj >>>>> >>>>> _______________________________________________ >>>>> Bro mailing list >>>>> bro at bro-ids.org >>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20151130/cff1a339/attachment.html