From johanna at icir.org Fri Apr 1 01:52:28 2016 From: johanna at icir.org (Johanna Amann) Date: Fri, 1 Apr 2016 11:52:28 +0300 Subject: [Bro] SFTP analysis In-Reply-To: References: Message-ID: <20160401085228.GA48993@Beezling.local> Hello John, On Thu, Mar 31, 2016 at 09:25:42AM -0700, john smith wrote: > Does anyone know if Bro supports SFTP? Thanks in advance. Bro supports and gives information about SSH; since SFTP traffic is just encapsulated inside the encrypted SSH session, there is not really much more that we can do. I hope this helps, Johanna From johanna at icir.org Fri Apr 1 01:59:10 2016 From: johanna at icir.org (Johanna Amann) Date: Fri, 1 Apr 2016 11:59:10 +0300 Subject: [Bro] Question : How can I change a particular log file format? In-Reply-To: <266584401.1260510.1459068973002.JavaMail.yahoo@mail.yahoo.com> References: <266584401.1260510.1459068973002.JavaMail.yahoo.ref@mail.yahoo.com> <266584401.1260510.1459068973002.JavaMail.yahoo@mail.yahoo.com> Message-ID: <20160401085910.GB48993@Beezling.local> Hello Aneela, On Sun, Mar 27, 2016 at 08:56:12AM +0000, Aneela Safdar wrote: > I am a newbie at bro and wanted to change log format of http.log file to > json.?Currently I have made changes in ascii.bro and now I am getting > all logs format in json but what I have to do if I only want http.log to > have that format and others keep default? you can do something like this in bro_init: local f = Log::get_filter(HTTP::LOG, "default"); # get default filter f$config = table(["use_json"] = "T"); # set json config option Log::add_filter(HTTP::LOG, f); # replace default filter > Also my log files have got still .log extension, how can I change it o .json? You sadly cannot change it for a single log file, just for all of them (by setting the BRO_LOG_SUFFIX environment variable). I hope this helps, Johanna From twaller at bivio.net Fri Apr 1 07:38:24 2016 From: twaller at bivio.net (Tony Waller) Date: Fri, 1 Apr 2016 14:38:24 +0000 Subject: [Bro] Performing file analysis with bro Message-ID: Hello, I am trying to write a bro event that matches this example content conditions: Payload string 1: content: "eaio" - This is example hex string Payload string 2: content: ".exe HTTP" When I look at file analysis it talk about the framework but leaves me to believe that it will only look at hash information. Any suggestions? Sincerely, Tony Tony Waller, CISSP Director, Systems Engineering Bivio Networks, Inc. ?Powering Advanced Cyber Operations? (TM) Mobile (443) 994-0936 [cid:9D7F8072-7FA8-45B5-AC0F-E8D71AFDE3BB] *Note: The information contained in this email confidential. This information is intended only for the individual, individuals or entity to whom it is addressed. If you are not the intended recipient(s), the employee or agent responsible for delivering it to the intended recipient(s), you are hereby notified that any use, dissemination, distribution or copying of this communication is strictly prohibited. If you have received this email in error, please return the original message to us by email and delete or destroy any copies. Please note any views or opinions expressed or presented in this email are solely those of the author and do not necessarily represent those of Bivio Networks, Inc. The recipient should check this email or any attachments for the presence of viruses or malware. Bivio Networks, Inc. accepts no responsibility for any damage caused by any virus or malware transmitted by this email. Thank you. Think Green when printing -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160401/7bea071a/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: CB97AD91-2E15-43EE-A2B4-B3542935CCB5[8].png Type: image/png Size: 8127 bytes Desc: CB97AD91-2E15-43EE-A2B4-B3542935CCB5[8].png Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160401/7bea071a/attachment-0001.bin From daniel.guerra69 at gmail.com Fri Apr 1 12:08:01 2016 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Fri, 1 Apr 2016 21:08:01 +0200 Subject: [Bro] Performing file analysis with bro In-Reply-To: References: Message-ID: <8E9B6345-23AF-445A-8A56-71BC7DD7C1E0@gmail.com> Hi Tony, This is a nice example https://www.bro.org/sphinx/_downloads/detect-MHR.bro Regards, Daniel > On 01 Apr 2016, at 16:38, Tony Waller wrote: > > Hello, > > I am trying to write a bro event that matches this example content conditions: > > Payload string 1: content: "eaio" - This is example hex string > Payload string 2: content: ".exe HTTP" > > When I look at file analysis it talk about the framework but leaves me to believe that it will only look at hash information. Any suggestions? > > Sincerely, > > Tony > > > Tony Waller, CISSP > Director, Systems Engineering > Bivio Networks, Inc. > ?Powering Advanced Cyber Operations? (TM) > Mobile (443) 994-0936 > > > > *Note: The information contained in this email confidential. This information is intended only for the individual, individuals or entity to whom it is addressed. If you are not the intended recipient(s), the employee or agent responsible for delivering it to the intended recipient(s), you are hereby notified that any use, dissemination, distribution or copying of this communication is strictly prohibited. If you have received this email in error, please return the original message to us by email and delete or destroy any copies. Please note any views or opinions expressed or presented in this email are solely those of the author and do not necessarily represent those of Bivio Networks, Inc. The recipient should check this email or any attachments for the presence of viruses or malware. Bivio Networks, Inc. accepts no responsibility for any damage caused by any virus or malware transmitted by this email. Thank you. > > Think Green when printing > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160401/def5573e/attachment.html From tgdesrochers at gmail.com Sun Apr 3 05:57:53 2016 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Sun, 3 Apr 2016 08:57:53 -0400 Subject: [Bro] [bro] ftp & file hash Message-ID: Does bro hash files it sees being uploaded and/or downloaded via FTP. When I see traffic in the ftp.log I never see a fuid so I assume the file analyzer is not being executed against the traffic. Am I correct in my assumption that by default bro does not hash files it sees over FTP? How can I invoke the file analyzer to have it hash all files seen over FTP? Thanks Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160403/d48c5a83/attachment.html From seth at icir.org Sun Apr 3 11:50:49 2016 From: seth at icir.org (Seth Hall) Date: Sun, 3 Apr 2016 14:50:49 -0400 Subject: [Bro] Question : How can I change a particular log file format? In-Reply-To: <266584401.1260510.1459068973002.JavaMail.yahoo@mail.yahoo.com> References: <266584401.1260510.1459068973002.JavaMail.yahoo.ref@mail.yahoo.com> <266584401.1260510.1459068973002.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1CA1B0C4-1294-4C1C-83D2-A728CF05A0CE@icir.org> > On Mar 27, 2016, at 4:56 AM, Aneela Safdar wrote: > > I am a newbie at bro and wanted to change log format of http.log file to json. > Currently I have made changes in ascii.bro and now I am getting all logs format in json but what I have to do if I only want http.log to have that format and others keep default? Alternately, you can just add the following line in a script (to avoid changing base scripts)... redef LogAscii::use_json = T; Of course, Johanna's reply still applies since it sounds like you don't want to make that change to all of your logs. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Sun Apr 3 11:52:43 2016 From: seth at icir.org (Seth Hall) Date: Sun, 3 Apr 2016 14:52:43 -0400 Subject: [Bro] [bro] ftp & file hash In-Reply-To: References: Message-ID: > On Apr 3, 2016, at 8:57 AM, Tim Desrochers wrote: > > Does bro hash files it sees being uploaded and/or downloaded via FTP. When I see traffic in the ftp.log I never see a fuid so I assume the file analyzer is not being executed against the traffic. > > Am I correct in my assumption that by default bro does not hash files it sees over FTP?  This is due to a race decision in the FTP analyzer. Your control session and data session are likely being load balanced to separate workers due to them being separate TCP connections and the information that the Data analyzer should expect a connection on a separate worker isn't being communicated quickly enough. We've discussed a few solutions to this problem, but still have not tackled it unfortunately. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Sun Apr 3 12:05:30 2016 From: seth at icir.org (Seth Hall) Date: Sun, 3 Apr 2016 15:05:30 -0400 Subject: [Bro] Bro email notice question In-Reply-To: <56FAAA2D.9010705@gmail.com> References: <56F372EE.9060909@gmail.com> <56F4046E.5070301@gmail.com> <56FA0771.6020909@gmail.com> <56FAAA2D.9010705@gmail.com> Message-ID: <41CF5C15-4505-485D-9232-B9E79B4C31B8@icir.org> > On Mar 29, 2016, at 12:15 PM, Jan Grash?fer wrote: > > If I remember correctly, the intention of do_notice.bro was to provide > an example how the intel-framework could be used in this context. I > think the example somehow became the default. Yep, that script is really only meant as an example and it's not loaded by default in Bro. I believe that criticalstack has chosen to load that script though. We certainly aren't against fixing up any scripts in Bro to make them more generally useful though, and from a quick skim it looks like those are totally reasonable changes which I apparently missed when I was writing that script. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Sun Apr 3 12:08:00 2016 From: seth at icir.org (Seth Hall) Date: Sun, 3 Apr 2016 15:08:00 -0400 Subject: [Bro] logs-to-elasticsearch.bro error In-Reply-To: <7A63317E-EAC2-4612-867F-E01FB67736DD@gmail.com> References: <004c01d1867a$968de9f0$c3a9bdd0$@126.com> <7A63317E-EAC2-4612-867F-E01FB67736DD@gmail.com> Message-ID: > On Mar 25, 2016, at 5:50 AM, Daniel Guerra wrote: > > To make this work you need some patches > or use an elasticsearch version lower than 2 (1.7) Daniel (and others), thank you for persisting with getting data into data stores that are currently having trouble with Bro data. I have some changes queued up and I'm hoping to get a bit more work done in the upcoming week or two which should make you very happy and make it possible to use mainline Bro again. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Sun Apr 3 12:12:10 2016 From: seth at icir.org (Seth Hall) Date: Sun, 3 Apr 2016 15:12:10 -0400 Subject: [Bro] [bro] smtp log strangeness In-Reply-To: <56F56C20.4050902@gmail.com> References: <56f52d54.8953810a.43002.ffffca33@mx.google.com> <56F56C20.4050902@gmail.com> Message-ID: <6BA06226-CFEA-4753-BD8D-68C4D9F987E8@icir.org> > On Mar 25, 2016, at 12:49 PM, Jan Grash?fer wrote: > >> Why am I getting all of this extra info in these fields? > > The subject headers seem to look that strange to support other encodings > than ASCII (see Yep! There is a hacky script I wrote a while to deal with this stuff too (we need to integrate it into the analyzer at some point though) https://github.com/sethhall/bro-junk-drawer/blob/master/smtp-decode-encoded-word-subjects.bro If you load that script, it adds another field to smtp.log named "decoded_subject". .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From daniel.guerra69 at gmail.com Sun Apr 3 13:46:56 2016 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Sun, 3 Apr 2016 22:46:56 +0200 Subject: [Bro] logs-to-elasticsearch.bro error In-Reply-To: References: <004c01d1867a$968de9f0$c3a9bdd0$@126.com> <7A63317E-EAC2-4612-867F-E01FB67736DD@gmail.com> Message-ID: Hi Seth, That?s great news because I think elasticsearch is very useful in combination with kibana. I wanted to use the latest version of E/K because of the map projection. With the plugin I can do continuous processing instead of batch processing with logstash. Regards, Daniel > On 03 Apr 2016, at 21:08, Seth Hall wrote: > > >> On Mar 25, 2016, at 5:50 AM, Daniel Guerra wrote: >> >> To make this work you need some patches >> or use an elasticsearch version lower than 2 (1.7) > > Daniel (and others), thank you for persisting with getting data into data stores that are currently having trouble with Bro data. I have some changes queued up and I'm hoping to get a bit more work done in the upcoming week or two which should make you very happy and make it possible to use mainline Bro again. :) > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > From daniel.guerra69 at gmail.com Sun Apr 3 13:50:03 2016 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Sun, 3 Apr 2016 22:50:03 +0200 Subject: [Bro] logs-to-elasticsearch.bro error In-Reply-To: <000801d187ce$f62eccd0$e28c6670$@126.com> References: <004c01d1867a$968de9f0$c3a9bdd0$@126.com> <7A63317E-EAC2-4612-867F-E01FB67736DD@gmail.com> <000801d187ce$f62eccd0$e28c6670$@126.com> Message-ID: <73D176B1-E461-4C38-920C-49FE7298BCA3@gmail.com> Use this patch for the elasticsearch plugin ElasticSearch.cc.patch > On 27 Mar 2016, at 04:18, mz wrote: > > HI > I installed the patch you provide, and the emergence of new error.By the way: I did not find this file krb-main bro source and elasticsearch plugins source code, so krb-main.patch I did not install the patch > > [2016-03-27 10:06:31,295][DEBUG][action.bulk ] [node-1] [mzh-201603190900][0] failed to execute bulk item (index) index {[mzh-201603190900][http][AVO10phhcNJqDxEYDvYi], source[{"ts":"2016-03-19T09:48:21.250090Z","uid":"CU0uvD4pYTE2YeoKh","id.orig_h":"222.246.191.234","id.orig_p":11325,"id.resp_h":"119.143.122.225","id.resp_p":80,"trans_depth":1,"method":"GET","host":"xxxxxx.com.cn ","uri":"/img/xxxxxx/Uploads/2015-08-24/55daef788341b.jpg","user_agent":"WeChat/6.3.13.17 CFNetwork/758.2.8 Darwin/15.0.0","request_body_len":0,"response_body_len":15201,"status_code":200,"status_msg":"OK","tags":[],"resp_fuids":["F3xb6m3Ffqs0QW1AI4"],"resp_mime_types":["image/jpeg"]}]} > MapperParsingException[Field name [id.orig_h] cannot contain '.'] > at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:276) > at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:221) > at org.elasticsearch.index.mapper.object.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:138) > at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:119) > at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:100) > at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:435) > at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:257) > at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230) > at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:458) > at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:762) > at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231) > at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > ???: Daniel Guerra [mailto:daniel.guerra69 at gmail.com] > ????: 2016?3?25? 17:51 > ???: mz > ??: bro at bro.org > ??: Re: [Bro] logs-to-elasticsearch.bro error > > Hi, > > To make this work you need some patches > or use an elasticsearch version lower than 2 (1.7) > > I made a docker image for this > https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/ > In the git there is a map bro-patch > https://github.com/danielguerra69/bro-debian-elasticsearch.git > > Regards, > > Daniel > >> On 25 Mar 2016, at 10:42, mz > wrote: >> >> Dear >> Use logs-to-elasticsearch.bro send logs to ES. Is now work. >> >> ES error logs: >> [2016-03-25 17:30:52,957][DEBUG][action.bulk ] [node-1] [whbro-201603251500][1] failed to execute bulk item (index) index {[whbro-201603251500][dns][AVOtHLQHooGOx5uLgLSQ], source[{"_timestamp":1458898236411,"ts":1458898206267,"uid":"ClbNI74bIcRQ8Gs6Wc","id.orig_h":"10.100.78.88","id.orig_p":137,"id.resp_h":"10.100.79.255","id.resp_p":137,"proto":"udp","trans_id":47282,"query":"ISATAP","qclass":1,"qclass_name":"C_INTERNET","qtype":32,"qtype_name":"NB","AA":false,"TC":false,"RD":true,"RA":false,"Z":1,"rejected":false}]} >> MapperParsingException[Field [_timestamp] is a metadata field and cannot be added inside a document. Use the index API request parameters.] >> at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:213) >> at org.elasticsearch.index.mapper.DocumentParser.innerParseDocument(DocumentParser.java:131) >> at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:79) >> at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:304) >> at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:500) >> at org.elasticsearch.index.shard.IndexShard.prepareCreateOnPrimary(IndexShard.java:481) >> at org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:214) >> at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:223) >> at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326) >> at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119) >> at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68) >> at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:595) >> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) >> at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:263) >> at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:260) >> at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350) >> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) >> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> >> Bro config file: >> /usr/local/bro/lib/bro/plugins/Bro_ElasticSearch/scripts/init.bro >> module LogElasticSearch; >> >> export { >> ## Destination for the ES logs. Valid options are >> ## "direct" to directly connect to ES and "nsq" to >> ## transfer the logs into an nsqd instance. >> const destination = "direct" &redef; >> >> ## Name of the ES cluster. >> const cluster_name = "my-application" &redef; >> >> ## ES server. >> const server_host = "10.100.79.10" &redef; >> >> ## ES port. >> const server_port = 9200 &redef; >> >> ## Name of the ES index. >> const index_prefix = "testooo" &redef; >> >> ## Should the index names be in UTC or in local time? >> ## Setting this to true would be more compatible with Kibana and other tools. >> const index_name_in_utc = F &redef; >> >> ## Format for the index names. >> ## Setting this to "%Y.%m.%d-%H" would be more compatible Kibana and other tools. >> #const index_name_fmt = "%Y%m%d" &redef; >> const index_name_fmt = "%Y%m%d%H%M" &redef; >> ## The ES type prefix comes before the name of the related log. >> ## e.g. prefix = "bro\_" would create types of bro_dns, bro_software, etc. >> const type_prefix = "" &redef; >> >> ## The time before an ElasticSearch transfer will timeout. Note that >> ## the fractional part of the timeout will be ignored. In particular, >> ## time specifications less than a second result in a timeout value of >> ## 0, which means "no timeout." >> const transfer_timeout = 2secs; >> >> ## The batch size is the number of messages that will be queued up before >> ## they are sent to be bulk indexed. >> const max_batch_size = 1000 &redef; >> >> ## The maximum amount of wall-clock time that is allowed to pass without >> ## finishing a bulk log send. This represents the maximum delay you >> ## would like to have with your logs before they are sent to ElasticSearch. >> const max_batch_interval = 1min &redef; >> >> ## The maximum byte size for a buffered JSON string to send to the bulk >> ## insert API. >> const max_byte_size = 1024 * 1024 &redef; >> >> ## If the "nsq" destination is given, this is the topic >> ## that Bro will push logs into. >> const nsq_topic = "bro_logs" &redef; >> } >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160403/43f5caaa/attachment-0001.html From scotty.b.brown at gmail.com Sun Apr 3 14:52:48 2016 From: scotty.b.brown at gmail.com (Scotty Brown) Date: Mon, 4 Apr 2016 07:52:48 +1000 Subject: [Bro] Bro email notice question In-Reply-To: <41CF5C15-4505-485D-9232-B9E79B4C31B8@icir.org> References: <56F372EE.9060909@gmail.com> <56F4046E.5070301@gmail.com> <56FA0771.6020909@gmail.com> <56FAAA2D.9010705@gmail.com> <41CF5C15-4505-485D-9232-B9E79B4C31B8@icir.org> Message-ID: <570190B0.40307@gmail.com> Hi Seth, It's saved us a BUNCH of time already - just having the notice source in the email by default. Would def +1 having these rolled back in at some point :) Cheers, Scotty On 04/04/16 05:05, Seth Hall wrote: >> On Mar 29, 2016, at 12:15 PM, Jan Grash?fer wrote: >> >> If I remember correctly, the intention of do_notice.bro was to provide >> an example how the intel-framework could be used in this context. I >> think the example somehow became the default. > Yep, that script is really only meant as an example and it's not loaded by default in Bro. I believe that criticalstack has chosen to load that script though. > > We certainly aren't against fixing up any scripts in Bro to make them more generally useful though, and from a quick skim it looks like those are totally reasonable changes which I apparently missed when I was writing that script. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From scotty.b.brown at gmail.com Sun Apr 3 14:53:25 2016 From: scotty.b.brown at gmail.com (Scotty Brown) Date: Mon, 4 Apr 2016 07:53:25 +1000 Subject: [Bro] Bro email notice question In-Reply-To: <56FAAA2D.9010705@gmail.com> References: <56F372EE.9060909@gmail.com> <56F4046E.5070301@gmail.com> <56FA0771.6020909@gmail.com> <56FAAA2D.9010705@gmail.com> Message-ID: <570190D5.4020301@gmail.com> If we are ever in the same place Jan - beers on me! Thanks. Scotty On 30/03/16 02:15, Jan Grash?fer wrote: > Hi Scotty, > >> Thank you! This is exactly what I was after. I did have to add a >> missing closing bracket ) to line 39. > You are welcome! I fixed the bracket as well as the misleading > indentation of the script. > >> Did you ever have any discussion on getting this added/changed to the >> default do_notice that is distributed with bro? > If I remember correctly, the intention of do_notice.bro was to provide > an example how the intel-framework could be used in this context. I > think the example somehow became the default. Therefore I was not sure > whether these changes would be suited for the do_notice.bro shipped with > Bro. > > Regards, > Jan > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From tkrabec at gmail.com Sun Apr 3 16:31:19 2016 From: tkrabec at gmail.com (Tim Krabec) Date: Sun, 3 Apr 2016 19:31:19 -0400 Subject: [Bro] odd data in my bro dns.log Message-ID: I'm working through mydns logs and I'm seeing entries like this .13 & .14 are AD domain servers not sure what to make of these, google has led me no where. TIA 192.168.1.13bp\x95\xff\x7f 192.168.1.13 192.168.1.13bp\x95\xff\x7f 192.168.1.138 192.168.1.13bp\x95\xff\x7f 192.168.1.14 192.168.1.13bp\x95\xff\x7f 192.168.1.143 192.168.1.13b\x16\x91\xff\x7f 192.168.1.138 192.168.1.13b\x16\x91\xff\x7f 192.168.1.14 192.168.1.13b\x16\x91\xff\x7f 192.168.1.143 192.168.1.13b\xcf\x8f\xff\x7f 192.168.1.14 192.168.1.13!ek\xff\x7f 192.168.1.14 192.168.1.13!ek\xff\x7f 192.168.1.143 192.168.1.13\x17\x96\x96\xff\x7f 192.168.1.14 192.168.1.13\x81 192.168.1.138 192.168.1.13\x81 192.168.1.14 192.168.1.13\x81 192.168.1.143 192.168.1.13\x82d\x8a\xff\x7f 192.168.1.138 192.168.1.13\x82d\x8a\xff\x7f 192.168.1.14 192.168.1.13\x82d\x8a\xff\x7f 192.168.1.143 192.168.1.13\xa1\xf6a\xff\x7f 192.168.1.138 192.168.1.13\xa1\xf6a\xff\x7f 192.168.1.14 192.168.1.13\xa1\xf6a\xff\x7f 192.168.1.143 192.168.1.13"\xb0\x94\xff\x7f 192.168.1.14 192.168.1.13\xb1+m\xff\x7f 192.168.1.138 192.168.1.13\xb1+m\xff\x7f 192.168.1.14 192.168.1.13\xb1+m\xff\x7f 192.168.1.143 192.168.1.13\xb2\xc6\x8b\xff\x7f 192.168.1.138 192.168.1.13\xb2\xc6\x8b\xff\x7f 192.168.1.14 192.168.1.13\xb2\xc6\x8b\xff\x7f 192.168.1.143 192.168.1.13\xb2\xd4\x8b\xff\x7f 192.168.1.138 192.168.1.13\xb2\xd4\x8b\xff\x7f 192.168.1.14 192.168.1.13\xb2\xd4\x8b\xff\x7f 192.168.1.143 192.168.1.13\xc0\x80}\xff\x7f 192.168.1.138 192.168.1.13\xc0\x80}\xff\x7f 192.168.1.14 192.168.1.13\xc0\x80}\xff\x7f 192.168.1.143 192.168.1.13\xc2\x07\x93\xff\x7f 192.168.1.13 192.168.1.13\xc2\x07\x93\xff\x7f 192.168.1.138 192.168.1.13\xc2\x07\x93\xff\x7f 192.168.1.14 192.168.1.13\xc2\x07\x93\xff\x7f 192.168.1.143 192.168.1.13\xd2\x1c\x95\xff\x7f 192.168.1.13 192.168.1.13\xd2\x1c\x95\xff\x7f 192.168.1.138 192.168.1.13\xd2\x1c\x95\xff\x7f 192.168.1.14 192.168.1.13\xd2\x1c\x95\xff\x7f 192.168.1.143 192.168.1.13\xe1\xb7i\xff\x7f 192.168.1.14 192.168.1.13\xff 192.168.1.14 192.168.1.14bp\x95\xff\x7f 192.168.1.14 192.168.1.14b\x16\x91\xff\x7f 192.168.1.14 192.168.1.14b\xcf\x8f\xff\x7f 192.168.1.14 192.168.1.14!ek\xff\x7f 192.168.1.138 192.168.1.14!ek\xff\x7f 192.168.1.14 192.168.1.14!ek\xff\x7f 192.168.1.143 192.168.1.14\x17\x96\x96\xff\x7f 192.168.1.14 192.168.1.14\x81 192.168.1.14 192.168.1.14\x82d\x8a\xff\x7f 192.168.1.14 192.168.1.14\xa1\xf6a\xff\x7f 192.168.1.14 192.168.1.14"\xb0\x94\xff\x7f 192.168.1.14 192.168.1.14\xb1+m\xff\x7f 192.168.1.14 Tim Krabec tkrabec.com Bio -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160403/f8635e2b/attachment.html From jan.grashoefer at gmail.com Mon Apr 4 05:01:28 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Mon, 4 Apr 2016 14:01:28 +0200 Subject: [Bro] Bro email notice question In-Reply-To: <570190D5.4020301@gmail.com> References: <56F372EE.9060909@gmail.com> <56F4046E.5070301@gmail.com> <56FA0771.6020909@gmail.com> <56FAAA2D.9010705@gmail.com> <570190D5.4020301@gmail.com> Message-ID: <57025798.8010906@gmail.com> Hi, > If we are ever in the same place Jan - beers on me! > > Thanks. I am happy that I could help! :) I have a few other small improvements for the intel-framework, which I plan to commit. I will include the do_notice.bro once I open a pull request. Best regards, Jan From aniketpsavanand at gmail.com Mon Apr 4 15:15:49 2016 From: aniketpsavanand at gmail.com (Aniket Savanand) Date: Mon, 4 Apr 2016 15:15:49 -0700 Subject: [Bro] Integrating AMQP in BRO via BINPAC Message-ID: <736BEE5E-E272-4974-94DB-239FD58770E4@gmail.com> Hi all I am trying to integrate AMQP protocol in BRO. I have followed all steps from these videos on YouTube. https://youtu.be/eZAgqSFd9-c http://youtu.be/o2XdMdyJHzk I am getting conn.log generated in usr/local/bro/logs/current when I run AMQP traffic via SwiftMQ server and client with messages. I can see 5672 port(AMQP dedicated port) connections on localhost in conn.log file. What else need to do to make sure that AMQP is correctly integrated into BRO.? I am not able to see actual AMQP messages in any of the log file though. Thanks Aniket -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160404/f204cbc5/attachment.html From finalstatic at gmail.com Mon Apr 4 18:59:31 2016 From: finalstatic at gmail.com (Carl Forsythe) Date: Mon, 4 Apr 2016 18:59:31 -0700 Subject: [Bro] Logging in multiple formats Message-ID: Hi All, I've seen examples in how you can log everything in JSON, individual logs in JSON, or everything in the regular log format. I have a use case in which I would like to be able to log output in both JSON and the regular log format as I have a set of consumers that easily ingest JSON and a set that doesn't enjoy it quite so much. Is there a way to log in both formats easily similar to how Suricata can emit both JSON and other log types all at once? Thanks, Carl -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160404/c6981da6/attachment.html From jan.grashoefer at gmail.com Tue Apr 5 04:42:13 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Tue, 5 Apr 2016 13:42:13 +0200 Subject: [Bro] Logging in multiple formats In-Reply-To: References: Message-ID: <5703A495.30300@gmail.com> Hi Carl, > Is there a way to log in both formats easily similar to how Suricata can > emit both JSON and other log types all at once? once I ran into the same problem and wrote a small script that might help: https://gist.github.com/J-Gras/f9f86828f9e9d9c0b8f0908bc3573bb0 It adds a filter to generate additional JSON logs for the configured log streams. Hope this helps, Jan From michael.fry at morphick.com Tue Apr 5 07:04:18 2016 From: michael.fry at morphick.com (Michael Fry) Date: Tue, 05 Apr 2016 14:04:18 +0000 Subject: [Bro] Assertion failure, Tag.cc Message-ID: Hi All, Has anyone encountered the assertion failure below with Bro 2.4.1? This is happening with live traffic at least a couple times a day since upgrading from Bro 2.3.2. I didn't see anything the bug tracker, so thought I'd float it here first. listening on zc:99 at 1, capture length 8192 bytes 1459786307.312525 processing suspended 1459786307.312525 processing continued 1459802619.911190 Failed to open GeoIP City database: /usr/share/GeoIP/GeoIPCity.dat 1459802619.911190 Fell back to GeoIP Country database 1459802619.911190 Failed to open GeoIP Cityv6 database: /usr/share/GeoIP/GeoIPCityv6.dat bro: /home/mfry/dev/bro24/bro-2.4.1/src/Tag.cc:72: EnumVal* Tag::AsEnumVal(EnumType*) const: Assertion `type == 0 && subtype == 0' failed. /opt/bro/share/broctl/scripts/run-bro: line 100: 11312 Aborted (core dumped) nohup ${pin_command} $pin_cpu "$mybro" "$@" Regards, Michael -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160405/ec0c57f8/attachment.html From jazoff at illinois.edu Tue Apr 5 07:18:44 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 5 Apr 2016 14:18:44 +0000 Subject: [Bro] Assertion failure, Tag.cc In-Reply-To: References: Message-ID: I've not seen that before. Are you running any custom local scripts? Do you still have the core file that it dumped? There's a good chance gdb can be used to figure out what script triggered that assertion. https://www.bro.org/support/reporting-problems.html outlines the process on how to use gdb against a bro core file. -- - Justin Azoff > On Apr 5, 2016, at 10:04 AM, Michael Fry wrote: > > Hi All, > > Has anyone encountered the assertion failure below with Bro 2.4.1? This is happening with live traffic at least a couple times a day since upgrading from Bro 2.3.2. I didn't see anything the bug tracker, so thought I'd float it here first. > > listening on zc:99 at 1, capture length 8192 bytes > > 1459786307.312525 processing suspended > 1459786307.312525 processing continued > 1459802619.911190 Failed to open GeoIP City database: /usr/share/GeoIP/GeoIPCity.dat > 1459802619.911190 Fell back to GeoIP Country database > 1459802619.911190 Failed to open GeoIP Cityv6 database: /usr/share/GeoIP/GeoIPCityv6.dat > bro: /home/mfry/dev/bro24/bro-2.4.1/src/Tag.cc:72: EnumVal* Tag::AsEnumVal(EnumType*) const: Assertion `type == 0 && subtype == 0' failed. > /opt/bro/share/broctl/scripts/run-bro: line 100: 11312 Aborted (core dumped) nohup ${pin_command} $pin_cpu "$mybro" "$@" > > Regards, > Michael > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From michael.fry at morphick.com Tue Apr 5 08:00:55 2016 From: michael.fry at morphick.com (Michael Fry) Date: Tue, 05 Apr 2016 15:00:55 +0000 Subject: [Bro] Assertion failure, Tag.cc In-Reply-To: References: Message-ID: Thanks for the response. We do have a custom script running that chooses files to extract based on mime type. Here is the stack trace that I'm seeing. It's always the same. Regards #0 0x00007ffb28a680d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffb28a6b83b in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007ffb28a60d9e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x00007ffb28a60e42 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x0000000000852850 in Tag::AsEnumVal (this=0x7fff27c7a8e0, etype=0x246a700) at /home/mfry/dev/bro24/bro-2.4.1/src/Tag.cc:72 #5 0x0000000000b51b7b in file_analysis::Tag::AsEnumVal (this=0x7fff27c7a8e0) at /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/Tag.cc:23 #6 0x0000000000b4d95b in file_analysis::AnalyzerSet::GetKey (this=0xb74e850, t=..., args=0xff59ca0) at /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/AnalyzerSet.cc:155 #7 0x0000000000b4d7c7 in file_analysis::AnalyzerSet::QueueRemove (this=0xb74e850, tag=..., args=0xff59ca0) at /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/AnalyzerSet.cc:140 #8 0x0000000000b4b3c9 in file_analysis::File::DeliverStream (this=0xb74e820, data=0xc02f4f0 "\217\020\272\214\355\361\301\272p\200\371\355\f\301\305\066\247Bj\216\027v\217^\303\t\221.6Fgrv\250\347Clcc\264\212\224\210\230!\336;\271Y\336!\355?\210\332$]\b\217l\372[\240\362O\261K'\234b\v\376", len=1460) at /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/File.cc:405 #9 0x0000000000b4c995 in file_analysis::FileReassembler::BlockInserted (this=0x10a0e400, start_block=0x11637850) at /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/FileReassembler.cc:66 #10 0x0000000000b4cabd in file_analysis::FileReassembler::Undelivered (this=0x10a0e400, up_to_seq=70713344) at /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/FileReassembler.cc:96 #11 0x0000000000806342 in Reassembler::TrimToSeq (this=0x10a0e400, seq=70713344) at /home/mfry/dev/bro24/bro-2.4.1/src/Reassem.cc:101 #12 0x0000000000b4c879 in file_analysis::FileReassembler::Flush (this=0x10a0e400) at /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/FileReassembler.cc:33 #13 0x0000000000b4b533 in file_analysis::File::DeliverChunk (this=0xb74e820, data=0xf0b2870 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", len=1460, offset=1147283) at /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/File.cc:421 #14 0x0000000000b4b9dc in file_analysis::File::DataIn (this=0xb74e820, data=0xf0b2870 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", len=1460, offset=1147283) at /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/File.cc:481 #15 0x0000000000b45722 in file_analysis::Manager::DataIn (this=0x246a440, data=0xf0b2870 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", len=1460, offset=1147283, tag=..., conn=0x10fb36f0, is_orig=false, precomputed_id=...) at /home/mfry/dev/bro24/bro-2.4.1/src/file_analysis/Manager.cc:121 #16 0x00000000009bea07 in analyzer::http::HTTP_Entity::SubmitData (this=0xa3eb330, len=1460, buf=0xf0b2870 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y") at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/http/HTTP.cc:322 #17 0x0000000000a2140b in analyzer::mime::MIME_Entity::FlushData (this=0xa3eb330) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/mime/MIME.cc:1217 #18 0x0000000000a20a97 in analyzer::mime::MIME_Entity::DecodeDataLine (this=0xa3eb330, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", trailing_CRLF=0) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/mime/MIME.cc:1013 #19 0x0000000000a1fe3c in analyzer::mime::MIME_Entity::NewDataLine (this=0xa3eb330, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", trailing_CRLF=0) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/mime/MIME.cc:698 #20 0x0000000000a1fbd3 in analyzer::mime::MIME_Entity::Deliver (this=0xa3eb330, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", trailing_CRLF=0) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/mime/MIME.cc:617 #21 0x00000000009be2cb in analyzer::http::HTTP_Entity::DeliverBodyClear (this=0xa3eb330, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", trailing_CRLF=0) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/http/HTTP.cc:217 #22 0x00000000009be218 in analyzer::http::HTTP_Entity::DeliverBody (this=0xa3eb330, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", trailing_CRLF=0) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/http/HTTP.cc:205 #23 0x00000000009be086 in analyzer::http::HTTP_Entity::Deliver (this=0xa3eb330, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", trailing_CRLF=0) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/http/HTTP.cc:159 #24 0x0000000000a1fe1a in analyzer::mime::MIME_Entity::NewDataLine (this=0x12b37890, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", trailing_CRLF=0) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/mime/MIME.cc:693 #25 0x0000000000a1fbd3 in analyzer::mime::MIME_Entity::Deliver (this=0x12b37890, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", trailing_CRLF=0) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/mime/MIME.cc:617 #26 0x00000000009be2cb in analyzer::http::HTTP_Entity::DeliverBodyClear (this=0x12b37890, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", trailing_CRLF=0) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/http/HTTP.cc:217 #27 0x00000000009be218 in analyzer::http::HTTP_Entity::DeliverBody (this=0x12b37890, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", trailing_CRLF=0) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/http/HTTP.cc:205 #28 0x00000000009bddfe in analyzer::http::HTTP_Entity::Deliver (this=0x12b37890, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", trailing_CRLF=0) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/http/HTTP.cc:107 #29 0x00000000009c43ea in analyzer::mime::MIME_Message::Deliver (this=0xbbf60b0, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", trailing_CRLF=0) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/mime/MIME.h:207 #30 0x00000000009c10e0 in analyzer::http::HTTP_Analyzer::DeliverStream (this=0xaba9ed0, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", is_orig=false) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/http/HTTP.cc:908 #31 0x0000000000b74c0b in analyzer::SupportAnalyzer::ForwardStream (this=0x96e9790, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", is_orig=false) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/Analyzer.cc:835 #32 0x0000000000adcd94 in analyzer::tcp::ContentLine_Analyzer::DoDeliver (this=0x96e9790, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y") at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/tcp/ContentLine.cc:168 #33 0x0000000000adcb44 in analyzer::tcp::ContentLine_Analyzer::DeliverStream (this=0x96e9790, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", is_orig=false) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/tcp/ContentLine.cc:108 #34 0x0000000000b72383 in analyzer::Analyzer::NextStream (this=0x96e9790, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", is_orig=false) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/Analyzer.cc:245 #35 0x0000000000b7235e in analyzer::Analyzer::NextStream (this=0xaba9ed0, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", is_orig=false) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/Analyzer.cc:239 #36 0x0000000000b72784 in analyzer::Analyzer::ForwardStream (this=0x77aa210, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y", is_orig=false) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/Analyzer.cc:331 #37 0x0000000000adb93f in analyzer::tcp::TCP_Reassembler::Deliver (this=0xd8f75b0, seq=6422408, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y") at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/tcp/TCP_Reassembler.cc:457 #38 0x0000000000adc279 in analyzer::tcp::TCP_Reassembler::DeliverBlock (this=0xd8f75b0, seq=6422408, len=1460, data=0xc15aab0 "\030\\\240\357\224\017\001<\207\n8\212\322W\273\023\034C<}@\210\242\031\245\206f'\212\367\254\211\213\227c\303\326k\341\305,\333\034\226\355y") at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/tcp/TCP_Reassembler.cc:647 #39 0x0000000000adb599 in analyzer::tcp::TCP_Reassembler::BlockInserted (this=0xd8f75b0, start_block=0x117aba30) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/tcp/TCP_Reassembler.cc:393 #40 0x0000000000806256 in Reassembler::NewBlock (this=0xd8f75b0, t=1459362263.205771, seq=6422408, len=1460, data=0x7ffb2bd72536
) at /home/mfry/dev/bro24/bro-2.4.1/src/Reassem.cc:73 #41 0x0000000000adba48 in analyzer::tcp::TCP_Reassembler::DataSent (this=0xd8f75b0, t=1459362263.205771, seq=6422408, len=1460, data=0x7ffb2bd72536
, replaying=true) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/tcp/TCP_Reassembler.cc:492 #42 0x0000000000ad9eb1 in analyzer::tcp::TCP_Endpoint::DataSent (this=0xbd1a7f0, t=1459362263.205771, seq=6422408, len=1460, caplen=1460, data=0x7ffb2bd72536
, ip=0x7fff27c7bd60, tp=0x7ffb2bd72522) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/tcp/TCP_Endpoint.cc:205 #43 0x0000000000ad4eac in analyzer::tcp::TCP_Analyzer::DeliverData (this=0x77aa210, t=1459362263.205771, data=0x7ffb2bd72536
, len=1460, caplen=1460, ip=0x7fff27c7bd60, tp=0x7ffb2bd72522, endpoint=0xbd1a7f0, rel_data_seq=6422408, is_orig=0, flags=...) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/tcp/TCP.cc:982 #44 0x0000000000ad5f82 in analyzer::tcp::TCP_Analyzer::DeliverPacket (this=0x77aa210, len=1460, data=0x7ffb2bd72536
, is_orig=false, seq=18446744073709551615, ip=0x7fff27c7bd60, caplen=1460) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/protocol/tcp/TCP.cc:1382 #45 0x0000000000b7229e in analyzer::Analyzer::NextPacket (this=0x77aa210, len=1480, data=0x7ffb2bd72522
, is_orig=false, seq=18446744073709551615, ip=0x7fff27c7bd60, caplen=1480) at /home/mfry/dev/bro24/bro-2.4.1/src/analyzer/Analyzer.cc:222 #46 0x000000000076c0c9 in Connection::NextPacket (this=0x10fb36f0, t=1459362263.205771, is_orig=0, ip=0x7fff27c7bd60, len=1480, caplen=1480, data=@0x7fff27c7b9a0: 0x7ffb2bd72522
, record_packet=@0x7fff27c7ba10: 1, record_content=@0x7fff27c7ba14: 1, hdr=0x48d3bb8, pkt=0x7ffb2bd72500
, hdr_size=14) at /home/mfry/dev/bro24/bro-2.4.1/src/Conn.cc:260 #47 0x0000000000837be5 in NetSessions::DoNextPacket (this=0x5f2db20, t=1459362263.205771, hdr=0x48d3bb8, ip_hdr=0x7fff27c7bd60, pkt=0x7ffb2bd72500
, hdr_size=14, encapsulation=0x0) at /home/mfry/dev/bro24/bro-2.4.1/src/Sessions.cc:760 #48 0x00000000008361e9 in NetSessions::NextPacket (this=0x5f2db20, t=1459362263.205771, hdr=0x48d3bb8, pkt=0x7ffb2bd72500
, hdr_size=14) at /home/mfry/dev/bro24/bro-2.4.1/src/Sessions.cc:231 #49 0x0000000000835ff7 in NetSessions::DispatchPacket (this=0x5f2db20, t=1459362263.205771, hdr=0x48d3bb8, pkt=0x7ffb2bd72500
, hdr_size=14, src_ps=0x48d3a70) at /home/mfry/dev/bro24/bro-2.4.1/src/Sessions.cc:187 #50 0x00000000007f2082 in net_packet_dispatch (t=1459362263.205771, hdr=0x48d3bb8, pkt=0x7ffb2bd72500
, hdr_size=14, src_ps=0x48d3a70) at /home/mfry/dev/bro24/bro-2.4.1/src/Net.cc:281 #51 0x0000000000b33fc2 in iosource::PktSrc::Process (this=0x48d3a70) at /home/mfry/dev/bro24/bro-2.4.1/src/iosource/PktSrc.cc:456 #52 0x00000000007f2290 in net_run () at /home/mfry/dev/bro24/bro-2.4.1/src/Net.cc:329 #53 0x000000000074815e in main (argc=18, argv=0x7fff27c7c628) at /home/mfry/dev/bro24/bro-2.4.1/src/main.cc:1190 On Tue, Apr 5, 2016 at 10:37 AM Azoff, Justin S wrote: > I've not seen that before. Are you running any custom local scripts? > > Do you still have the core file that it dumped? There's a good chance gdb > can be used to figure out what script triggered that assertion. > > https://www.bro.org/support/reporting-problems.html outlines the process > on how to use gdb against a bro core file. > -- > - Justin Azoff > > > On Apr 5, 2016, at 10:04 AM, Michael Fry > wrote: > > > > Hi All, > > > > Has anyone encountered the assertion failure below with Bro 2.4.1? This > is happening with live traffic at least a couple times a day since > upgrading from Bro 2.3.2. I didn't see anything the bug tracker, so thought > I'd float it here first. > > > > listening on zc:99 at 1, capture length 8192 bytes > > > > 1459786307.312525 processing suspended > > 1459786307.312525 processing continued > > 1459802619.911190 Failed to open GeoIP City database: > /usr/share/GeoIP/GeoIPCity.dat > > 1459802619.911190 Fell back to GeoIP Country database > > 1459802619.911190 Failed to open GeoIP Cityv6 database: > /usr/share/GeoIP/GeoIPCityv6.dat > > bro: /home/mfry/dev/bro24/bro-2.4.1/src/Tag.cc:72: EnumVal* > Tag::AsEnumVal(EnumType*) const: Assertion `type == 0 && subtype == 0' > failed. > > /opt/bro/share/broctl/scripts/run-bro: line 100: 11312 Aborted > (core dumped) nohup ${pin_command} $pin_cpu "$mybro" "$@" > > > > Regards, > > Michael > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160405/ab5f2e21/attachment-0001.html From youn1614 at umn.edu Tue Apr 5 09:20:01 2016 From: youn1614 at umn.edu (Luke Young) Date: Tue, 5 Apr 2016 11:20:01 -0500 Subject: [Bro] Bro (4.2.1) &synchronized attribute doesn't seem to be working with default cluster configuration Message-ID: I seem to be having an issue with the &synchronized attribute when running in a cluster configuration on the latest stable release, Bro 4.2.1. I've condensed it down into a small re-produceable proof of concept. I took a standard bro install, appended the following POC to local.bro: sync.bro I configured a manager, 2 proxies and 2 workers: node.cfg Finally I deployed the configuration and scripts via broctl and checked the output: broctl.stdout Since the "pids" set has the "&synchronized" attribute one would expect that as each node appends it's own pid the "pids" set would grow, however this is not the behavior observed. Am I missing something obvious for this functionality to work? Thanks, Luke Young -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160405/5d27cc75/attachment.html From josh.guild at morphick.com Thu Apr 7 11:39:18 2016 From: josh.guild at morphick.com (Josh Guild) Date: Thu, 7 Apr 2016 14:39:18 -0400 Subject: [Bro] Bro and APCON Message-ID: Hi all, We have a few deployments that utilize an APCON for traffic aggregation. We've noticed in these environments that Bro has trouble reassembling the traffic correctly and there is a significant amount of capture loss (based on the script). We've tried different hashing algorithms on the APCON to no effect. Has anyone else seen anything similar to this or have any insight? Thanks! -- Josh Guild Network Intelligence Analyst -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160407/bcaeba58/attachment.html From pawelec93 at googlemail.com Thu Apr 7 15:46:53 2016 From: pawelec93 at googlemail.com (=?UTF-8?Q?Pawe=C5=82_Piszczatowski?=) Date: Thu, 7 Apr 2016 23:46:53 +0100 Subject: [Bro] Bro not producing a notice.log Message-ID: I have a Bro cluster setup in the AWS cloud, currently just with one node. My problem is that Bro is not producing the notice.log, it should just log successful SSH logins but it doesn't. I have tried SSH and FTP bruteforcing the worker node and exceeding the limit of failed connections, again no notice.log. I can see the detect-bruteforcing.bro scripts loaded in the loaded_scripts.log. I am pretty new to Bro, so I am not sure what I am doing wrong. Regards, -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160407/e70ff85e/attachment.html From dopheide at gmail.com Thu Apr 7 16:04:39 2016 From: dopheide at gmail.com (Mike Dopheide) Date: Thu, 7 Apr 2016 18:04:39 -0500 Subject: [Bro] Bro not producing a notice.log In-Reply-To: References: Message-ID: I want to say that's likely because AWS disables promiscuous mode so getting Bro to work requires some additional tricks. Can anyone verify? On Thursday, April 7, 2016, Pawe? Piszczatowski wrote: > I have a Bro cluster setup in the AWS cloud, currently just with one node. > My problem is that Bro is not producing the notice.log, it should just log > successful SSH logins but it doesn't. I have tried SSH and FTP bruteforcing > the worker node and exceeding the limit of failed connections, again no > notice.log. I can see the detect-bruteforcing.bro scripts loaded in the > loaded_scripts.log. I am pretty new to Bro, so I am not sure what I am > doing wrong. > > Regards, > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160407/0fbff160/attachment.html From jeff.geiger at gmail.com Thu Apr 7 19:04:04 2016 From: jeff.geiger at gmail.com (Jeff Geiger) Date: Thu, 7 Apr 2016 21:04:04 -0500 Subject: [Bro] Bro not producing a notice.log In-Reply-To: References: Message-ID: I don't know if anything has changed in the last few years, but I know it used to be the case that you could not put an AWS interface into promiscuous mode. To get around this, you had to use a tool like daemonlogger to dump packets from the external interface to a tap, tun, or bridge interface and monitor that. For larger scale implementations, you can use openvpn internally to route all the traffic back to your sensor. I set up a PoC doing similar with Snort a few years back. ( https://github.com/jeffgeiger/CloudSnort) Hopefully that helps, if this is still the case. Best, Jeff Geiger On Thu, Apr 7, 2016 at 6:04 PM, Mike Dopheide wrote: > I want to say that's likely because AWS disables promiscuous mode so > getting Bro to work requires some additional tricks. Can anyone verify? > > > On Thursday, April 7, 2016, Pawe? Piszczatowski > wrote: > >> I have a Bro cluster setup in the AWS cloud, currently just with one >> node. My problem is that Bro is not producing the notice.log, it should >> just log successful SSH logins but it doesn't. I have tried SSH and FTP >> bruteforcing the worker node and exceeding the limit of failed connections, >> again no notice.log. I can see the detect-bruteforcing.bro scripts loaded >> in the loaded_scripts.log. I am pretty new to Bro, so I am not sure what I >> am doing wrong. >> >> Regards, >> >> > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160407/817f4f15/attachment.html From pawelec93 at googlemail.com Fri Apr 8 01:34:45 2016 From: pawelec93 at googlemail.com (Pawel) Date: Fri, 8 Apr 2016 08:34:45 +0000 (UTC) Subject: [Bro] Bro not producing a notice.log References: Message-ID: Jeff Geiger gmail.com> writes: > > > > > I don't know if anything has changed in the last few years, but I know it used to be the case that you could not put an AWS interface into promiscuous mode.? To get around this, you had to use a tool like daemonlogger to dump packets from the external interface to a tap, tun, or bridge interface and monitor that.? For larger scale implementations, you can use openvpn internally to route all the traffic back to your sensor.? I set up a PoC doing similar with Snort a few years back. (https://github.com/jeffgeiger/CloudSnort) ?Hopefully that helps, if this is still the case. > > Best, > > Jeff Geiger > > > On Thu, Apr 7, 2016 at 6:04 PM, Mike Dopheide gmail.com> wrote:I want to say that's likely because AWS disables promiscuous mode so getting Bro to work requires some additional tricks. ? Can anyone verify? > On Thursday, April 7, 2016, Pawe? Piszczatowski googlemail.com> wrote: > > I have a Bro cluster setup in the AWS cloud, currently just with one node. My problem is that Bro is not producing the notice.log, it should just log successful SSH logins but it doesn't. I have tried SSH and FTP bruteforcing the worker node and exceeding the limit of failed connections, again no notice.log. I can see the detect-bruteforcing.bro scripts loaded in the loaded_scripts.log. I am pretty new to Bro, so I am not sure what I am doing wrong. > Regards, > > > > _______________________________________________ > Bro mailing listbro bro-ids.orghttp://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > > > > _______________________________________________ > Bro mailing list > bro bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro So lets say I bridge an interface and start to monitor br0 instead of eth0, the br0 interface will not be in promiscuous mode and will allow to produce the notice.log? I am asking because you said you have done it few years ago, therefore wondering whether anything has changed in AWS. I still can't understand it why is it not producing the notice.log. I have all the other logs (conn, http, ssl, x509 etc) and they are working fine. I am not sure how Bro exactly works but if there is more than 30 rejected FTP requests in the conn.log why doesn't it raise a notice if that's what the FTP-bruteforce is looking for. Regards From jazoff at illinois.edu Fri Apr 8 04:57:33 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 8 Apr 2016 11:57:33 +0000 Subject: [Bro] Bro not producing a notice.log In-Reply-To: References: Message-ID: > On Apr 7, 2016, at 6:46 PM, Pawe? Piszczatowski wrote: > > I have a Bro cluster setup in the AWS cloud, currently just with one node. My problem is that Bro is not producing the notice.log, it should just log successful SSH logins but it doesn't. I have tried SSH and FTP bruteforcing the worker node and exceeding the limit of failed connections, again no notice.log. I can see the detect-bruteforcing.bro scripts loaded in the loaded_scripts.log. I am pretty new to Bro, so I am not sure what I am doing wrong. > > Regards, > I still can't understand it why is it not producing the notice.log. I have > all the other logs (conn, http, ssl, x509 etc) and they are working fine. Are you running Bro on the machine that is running the ssh and ftp server? Your logs are likely broken, but you haven't looked closely enough at them. Just because a log exists doesn't mean bro is seeing both sides of the connection. https://www.bro.org/documentation/faq.html#why-isn-t-bro-producing-the-logs-i-expect-a-note-about-checksums -- - Justin Azoff From wayward710 at gmail.com Fri Apr 8 12:27:01 2016 From: wayward710 at gmail.com (Wendy Edwards) Date: Fri, 8 Apr 2016 14:27:01 -0500 Subject: [Bro] Bro support in PrismJS Message-ID: Bro support has been added to PrismJS (which is used in Google Polymer). You can try it out at http://prismjs.com/test.html . If you click the circles on the right side of the page, you can see it rendered in different styles. (Copy-paste some Bro code into the box and select "Bro" as the language.) Also, there's an Ace online editor with (limited) Bro syntax highlighting. I created it by importing an existing Bro Textmate file. It's at https://github.com/open-nsm/ace . It could probably use some manual tweaking (which may happen at some point in the future). Wendy -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160408/2635ebd2/attachment.html From jazoff at illinois.edu Fri Apr 8 12:42:44 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 8 Apr 2016 19:42:44 +0000 Subject: [Bro] Bro support in PrismJS In-Reply-To: References: Message-ID: <6D6313A7-D054-48EB-AC9A-91E7C0BB751D@illinois.edu> > On Apr 8, 2016, at 3:27 PM, Wendy Edwards wrote: > > Also, there's an Ace online editor with (limited) Bro syntax highlighting. I created it by importing an existing Bro Textmate file. It's at https://github.com/open-nsm/ace . It could probably use some manual tweaking (which may happen at some point in the future). > > Wendy oooh! Ace is what try.bro.org uses. I have it in 'perl' mode which usually is close enough, but more accurate highlighting would be better.. I'll have to see about integrating that. -- - Justin Azoff From wayward710 at gmail.com Fri Apr 8 12:46:26 2016 From: wayward710 at gmail.com (Wendy Edwards) Date: Fri, 8 Apr 2016 14:46:26 -0500 Subject: [Bro] Bro support in PrismJS In-Reply-To: <6D6313A7-D054-48EB-AC9A-91E7C0BB751D@illinois.edu> References: <6D6313A7-D054-48EB-AC9A-91E7C0BB751D@illinois.edu> Message-ID: Cool. I submitted a PR to the Ace project, though it's possible that the Bro mode for Ace could use some manual tweaking. It's in the open-nsm Github space, so you should also be able to make changes to it if you want -- you're part of that group. On Fri, Apr 8, 2016 at 2:42 PM, Azoff, Justin S wrote: > > > On Apr 8, 2016, at 3:27 PM, Wendy Edwards wrote: > > > > Also, there's an Ace online editor with (limited) Bro syntax > highlighting. I created it by importing an existing Bro Textmate file. > It's at https://github.com/open-nsm/ace . It could probably use some > manual tweaking (which may happen at some point in the future). > > > > Wendy > > oooh! Ace is what try.bro.org uses. I have it in 'perl' mode which > usually is close enough, but more accurate highlighting would be better.. > I'll have to see about integrating that. > > > -- > - Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160408/1534121a/attachment.html From pawelec93 at googlemail.com Sat Apr 9 07:56:04 2016 From: pawelec93 at googlemail.com (Pawel) Date: Sat, 9 Apr 2016 14:56:04 +0000 (UTC) Subject: [Bro] Bro not producing a notice.log References: Message-ID: That worked perfectly, thanks !! However, now I am seeing a lot of possible_split_routing and data_before_established. Any ideas? From santiago.fonseca at accelogic.com Mon Apr 11 12:26:10 2016 From: santiago.fonseca at accelogic.com (Santiago Fonseca) Date: Mon, 11 Apr 2016 15:26:10 -0400 Subject: [Bro] Consulting services Message-ID: Hello everyone, My company is new to Bro. We are working on a project for which we are evaluating integrating our development into Bro. However, we are not interested in becoming Bro experts to do so. We prefer to use consultants that can do the job. We are therefore interested in knowing if the developers/maintainers of Bro would be able to provide such type of consulting services. If this is indeed a possibility we would like to assess the following: 1. Are there different "tiers" of service depending on the need? If so, what are the different consulting rates. 2. Is it possible that the consultants travel to our offices in Florida for a 1 - 2 day kick-off meeting? If this is not possible, then would the people from my R&D team be able to travel to the offices of the Bro team for this kick-off? 3. Is it possible to hire the consulting service on a "retainer-type" contract such that we can use the time as the project evolves and we see it necessary? Thank you for your consideration in replying to this post, Best, -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160411/11f61948/attachment.html From js688886 at gmail.com Mon Apr 11 13:21:39 2016 From: js688886 at gmail.com (john smith) Date: Mon, 11 Apr 2016 13:21:39 -0700 Subject: [Bro] SFTP analysis In-Reply-To: <20160401085228.GA48993@Beezling.local> References: <20160401085228.GA48993@Beezling.local> Message-ID: Thanks Johanna! There is a builtin script /bro/frameworks/logging/postprocessors/sftp.bro. Can that be used and how? Thanks. John On Fri, Apr 1, 2016 at 1:52 AM, Johanna Amann wrote: > Hello John, > > On Thu, Mar 31, 2016 at 09:25:42AM -0700, john smith wrote: > > Does anyone know if Bro supports SFTP? Thanks in advance. > > Bro supports and gives information about SSH; since SFTP traffic is just > encapsulated inside the encrypted SSH session, there is not really much > more that we can do. > > I hope this helps, > Johanna > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160411/eabb922f/attachment.html From sudo.darkstar at gmail.com Mon Apr 11 13:26:01 2016 From: sudo.darkstar at gmail.com (John B. Althouse III) Date: Mon, 11 Apr 2016 16:26:01 -0400 Subject: [Bro] Shockwave Flash Analyzer Message-ID: Does anyone know of a Shockwave Flash analyzer for Bro? It would be useful to gather the metadata in the header like version, width, hight, frame rate, frame count, compression ratio, ect. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160411/65ddbe5a/attachment.html From johanna at icir.org Mon Apr 11 13:46:49 2016 From: johanna at icir.org (Johanna Amann) Date: Mon, 11 Apr 2016 22:46:49 +0200 Subject: [Bro] SFTP analysis In-Reply-To: References: <20160401085228.GA48993@Beezling.local> Message-ID: <79CE17BF-A4FB-4507-8B27-EC45EB9FA60B@icir.org> That is for copying Bro log files to another host via sftp, not for parsing sftp sessions, sorry. Unless that is what you originally asked for :) Johanna On 11 Apr 2016, at 22:21, john smith wrote: > Thanks Johanna! > > There is a builtin script > /bro/frameworks/logging/postprocessors/sftp.bro. > Can that be used and how? Thanks. > > John > > On Fri, Apr 1, 2016 at 1:52 AM, Johanna Amann > wrote: > >> Hello John, >> >> On Thu, Mar 31, 2016 at 09:25:42AM -0700, john smith wrote: >>> Does anyone know if Bro supports SFTP? Thanks in advance. >> >> Bro supports and gives information about SSH; since SFTP traffic is >> just >> encapsulated inside the encrypted SSH session, there is not really >> much >> more that we can do. >> >> I hope this helps, >> Johanna >> From js688886 at gmail.com Tue Apr 12 10:29:06 2016 From: js688886 at gmail.com (john smith) Date: Tue, 12 Apr 2016 10:29:06 -0700 Subject: [Bro] Telnet log Message-ID: Hello, With 2.4.1, is there any way to generate Telnet logs? Thanks in advance! John -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160412/a79459fe/attachment.html From Rich.Giesige at oregonstate.edu Tue Apr 12 13:23:49 2016 From: Rich.Giesige at oregonstate.edu (Giesige, Rich) Date: Tue, 12 Apr 2016 20:23:49 +0000 Subject: [Bro] Question about network cards Message-ID: <40481EA2-57C7-4D7C-97C5-A27D80AEBC96@oregonstate.edu> Hello, I?m wondering what people are using for network cards in their bro clusters that are not using the Myricom Network Cards. We don?t have a $1,000 dollars per a card + license to spend on the cards. Is anyone using Intel or other brands that aren?t as expensive to capture their traffic? We are looking at doing all 10 Gig connections into the Bro Cluster. Thanks for all your answers. -- Richard Giesige IT Security Analyst Office of Information Security Oregon State University "OSU staff will NEVER ask for you password. Never email or share your password with anyone." -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160412/79c5163e/attachment.html From slagell at illinois.edu Tue Apr 12 13:32:47 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Tue, 12 Apr 2016 20:32:47 +0000 Subject: [Bro] Question about network cards In-Reply-To: <40481EA2-57C7-4D7C-97C5-A27D80AEBC96@oregonstate.edu> References: <40481EA2-57C7-4D7C-97C5-A27D80AEBC96@oregonstate.edu> Message-ID: A lot of people use Intel cards with pf_ring. We have both. On Apr 12, 2016, at 3:31 PM, Giesige, Rich > wrote: Hello, I?m wondering what people are using for network cards in their bro clusters that are not using the Myricom Network Cards. We don?t have a $1,000 dollars per a card + license to spend on the cards. Is anyone using Intel or other brands that aren?t as expensive to capture their traffic? We are looking at doing all 10 Gig connections into the Bro Cluster. Thanks for all your answers. -- Richard Giesige IT Security Analyst Office of Information Security Oregon State University "OSU staff will NEVER ask for you password. Never email or share your password with anyone." _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160412/05b76091/attachment.html From BLMILLER at comerica.com Tue Apr 12 13:41:37 2016 From: BLMILLER at comerica.com (Miller, Brad L) Date: Tue, 12 Apr 2016 20:41:37 +0000 Subject: [Bro] Question about network cards In-Reply-To: <40481EA2-57C7-4D7C-97C5-A27D80AEBC96@oregonstate.edu> References: <40481EA2-57C7-4D7C-97C5-A27D80AEBC96@oregonstate.edu> Message-ID: We are using Endace cards which are quite a bit more pricey, but we are actively looking at the Myricom cards now. My advice ? get the Myricom cards. While you can do pfring using standard cards, nothing beats the low to no capture loss hardware. The ability to do onboard load distribution with multiple sub interfaces is a killer feature and your Bro config is greatly simplified. We use a patched version of libpacap for Endace.. but I hear that 2.5 may incorporate native Myricom support. Without cards like these it is like getting a new mustang but skimping on the powertrain options. From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Giesige, Rich Sent: Tuesday, April 12, 2016 4:24 PM To: bro at bro.org Subject: [Bro] Question about network cards Hello, I?m wondering what people are using for network cards in their bro clusters that are not using the Myricom Network Cards. We don?t have a $1,000 dollars per a card + license to spend on the cards. Is anyone using Intel or other brands that aren?t as expensive to capture their traffic? We are looking at doing all 10 Gig connections into the Bro Cluster. Thanks for all your answers. -- Richard Giesige IT Security Analyst Office of Information Security Oregon State University "OSU staff will NEVER ask for you password. Never email or share your password with anyone." Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160412/4fc59c95/attachment-0001.html From michalpurzynski1 at gmail.com Tue Apr 12 15:01:25 2016 From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=) Date: Wed, 13 Apr 2016 00:01:25 +0200 Subject: [Bro] Question about network cards In-Reply-To: References: <40481EA2-57C7-4D7C-97C5-A27D80AEBC96@oregonstate.edu> Message-ID: <96727EE0-7D0C-400D-A37D-77E97E1B0BED@gmail.com> Another voice for myricoms. Single port with the sniffer v3 license was nowhere close to 1000, but much cheaper. Maintaining that, comparing to pfring, is day and night. Netmap with Intel should be the future, I don't have much experience with that yet. Another option is afpacket and intels, works well. > On 12 Apr 2016, at 22:41, Miller, Brad L wrote: > > We are using Endace cards which are quite a bit more pricey, but we are actively looking at the Myricom cards now. > > My advice ? get the Myricom cards. While you can do pfring using standard cards, nothing beats the low to no capture loss hardware. The ability to do onboard load distribution with multiple sub interfaces is a killer feature and your Bro config is greatly simplified. We use a patched version of libpacap for Endace.. but I hear that 2.5 may incorporate native Myricom support. > > Without cards like these it is like getting a new mustang but skimping on the powertrain options. > > > > > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Giesige, Rich > Sent: Tuesday, April 12, 2016 4:24 PM > To: bro at bro.org > Subject: [Bro] Question about network cards > > Hello, > > I?m wondering what people are using for network cards in their bro clusters that are not using the Myricom Network Cards. We don?t have a $1,000 dollars per a card + license to spend on the cards. Is anyone using Intel or other brands that aren?t as expensive to capture their traffic? We are looking at doing all 10 Gig connections into the Bro Cluster. > > Thanks for all your answers. > > -- > Richard Giesige > IT Security Analyst > Office of Information Security > Oregon State University > > "OSU staff will NEVER ask for you password. > Never email or share your password with anyone." > > > Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160413/aaf83a91/attachment.html From hosom at battelle.org Wed Apr 13 04:46:29 2016 From: hosom at battelle.org (Hosom, Stephen M) Date: Wed, 13 Apr 2016 11:46:29 +0000 Subject: [Bro] Question about network cards References: <40481EA2-57C7-4D7C-97C5-A27D80AEBC96@oregonstate.edu> <96727EE0-7D0C-400D-A37D-77E97E1B0BED@gmail.com> Message-ID: Intel x520s work fine with both af_packet and pf_ring. On 04/12/2016 06:03 PM, Micha? Purzy?ski wrote: Another voice for myricoms. Single port with the sniffer v3 license was nowhere close to 1000, but much cheaper. Maintaining that, comparing to pfring, is day and night. Netmap with Intel should be the future, I don't have much experience with that yet. Another option is afpacket and intels, works well. On 12 Apr 2016, at 22:41, Miller, Brad L <BLMILLER at comerica.com> wrote: We are using Endace cards which are quite a bit more pricey, but we are actively looking at the Myricom cards now. My advice ? get the Myricom cards. While you can do pfring using standard cards, nothing beats the low to no capture loss hardware. The ability to do onboard load distribution with multiple sub interfaces is a killer feature and your Bro config is greatly simplified. We use a patched version of libpacap for Endace.. but I hear that 2.5 may incorporate native Myricom support. Without cards like these it is like getting a new mustang but skimping on the powertrain options. From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Giesige, Rich Sent: Tuesday, April 12, 2016 4:24 PM To: bro at bro.org Subject: [Bro] Question about network cards Hello, I?m wondering what people are using for network cards in their bro clusters that are not using the Myricom Network Cards. We don?t have a $1,000 dollars per a card + license to spend on the cards. Is anyone using Intel or other brands that aren?t as expensive to capture their traffic? We are looking at doing all 10 Gig connections into the Bro Cluster. Thanks for all your answers. -- Richard Giesige IT Security Analyst Office of Information Security Oregon State University "OSU staff will NEVER ask for you password. Never email or share your password with anyone." Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email. _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From gfaulkner.nsm at gmail.com Wed Apr 13 07:39:50 2016 From: gfaulkner.nsm at gmail.com (Gary Faulkner) Date: Wed, 13 Apr 2016 09:39:50 -0500 Subject: [Bro] Question about network cards In-Reply-To: References: <40481EA2-57C7-4D7C-97C5-A27D80AEBC96@oregonstate.edu> <96727EE0-7D0C-400D-A37D-77E97E1B0BED@gmail.com> Message-ID: <570E5A36.5050301@gmail.com> What are folks thoughts on Intel Cards with the fully licensed PF_RING DNA+Libzero or ZC drivers and libraries, which NTOP typically offers to EDUs at no cost. Shouldn't these perform much more closely to the Myricoms with Sniffer v3 than standard PF_RING drivers and libraries? On 4/13/16 6:46 AM, Hosom, Stephen M wrote: > Intel x520s work fine with both af_packet and pf_ring. > > > On 04/12/2016 06:03 PM, Micha? Purzy?ski wrote: > Another voice for myricoms. Single port with the sniffer v3 license was nowhere close to 1000, but much cheaper. > > Maintaining that, comparing to pfring, is day and night. > > Netmap with Intel should be the future, I don't have much experience with that yet. > > Another option is afpacket and intels, works well. > > On 12 Apr 2016, at 22:41, Miller, Brad L <BLMILLER at comerica.com> wrote: > > We are using Endace cards which are quite a bit more pricey, but we are actively looking at the Myricom cards now. > > My advice ? get the Myricom cards. While you can do pfring using standard cards, nothing beats the low to no capture loss hardware. The ability to do onboard load distribution with multiple sub interfaces is a killer feature and your Bro config is greatly simplified. We use a patched version of libpacap for Endace.. but I hear that 2.5 may incorporate native Myricom support. > > Without cards like these it is like getting a new mustang but skimping on the powertrain options. > > > > > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Giesige, Rich > Sent: Tuesday, April 12, 2016 4:24 PM > To: bro at bro.org > Subject: [Bro] Question about network cards > > Hello, > > I?m wondering what people are using for network cards in their bro clusters that are not using the Myricom Network Cards. We don?t have a $1,000 dollars per a card + license to spend on the cards. Is anyone using Intel or other brands that aren?t as expensive to capture their traffic? We are looking at doing all 10 Gig connections into the Bro Cluster. > > Thanks for all your answers. > > -- > Richard Giesige > IT Security Analyst > Office of Information Security > Oregon State University > > "OSU staff will NEVER ask for you password. > Never email or share your password with anyone." > > > Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From michalpurzynski1 at gmail.com Wed Apr 13 08:27:43 2016 From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=) Date: Wed, 13 Apr 2016 17:27:43 +0200 Subject: [Bro] Question about network cards In-Reply-To: <570E5A36.5050301@gmail.com> References: <40481EA2-57C7-4D7C-97C5-A27D80AEBC96@oregonstate.edu> <96727EE0-7D0C-400D-A37D-77E97E1B0BED@gmail.com> <570E5A36.5050301@gmail.com> Message-ID: Of only I had enough patience for ZC. When it worked, I saw some packed loss that wasn't there when I used Myricom on the same sensor. A nice alternative would be an Intel plus NetMap. > On 13 Apr 2016, at 16:39, Gary Faulkner wrote: > > What are folks thoughts on Intel Cards with the fully licensed PF_RING DNA+Libzero or ZC drivers and libraries, which NTOP typically offers to EDUs at no cost. Shouldn't these perform much more closely to the Myricoms with Sniffer v3 than standard PF_RING drivers and libraries? > >> On 4/13/16 6:46 AM, Hosom, Stephen M wrote: >> Intel x520s work fine with both af_packet and pf_ring. >> >> >> On 04/12/2016 06:03 PM, Micha? Purzy?ski wrote: >> Another voice for myricoms. Single port with the sniffer v3 license was nowhere close to 1000, but much cheaper. >> >> Maintaining that, comparing to pfring, is day and night. >> >> Netmap with Intel should be the future, I don't have much experience with that yet. >> >> Another option is afpacket and intels, works well. >> >> On 12 Apr 2016, at 22:41, Miller, Brad L <BLMILLER at comerica.com> wrote: >> >> We are using Endace cards which are quite a bit more pricey, but we are actively looking at the Myricom cards now. >> >> My advice ? get the Myricom cards. While you can do pfring using standard cards, nothing beats the low to no capture loss hardware. The ability to do onboard load distribution with multiple sub interfaces is a killer feature and your Bro config is greatly simplified. We use a patched version of libpacap for Endace.. but I hear that 2.5 may incorporate native Myricom support. >> >> Without cards like these it is like getting a new mustang but skimping on the powertrain options. >> >> >> >> >> From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Giesige, Rich >> Sent: Tuesday, April 12, 2016 4:24 PM >> To: bro at bro.org >> Subject: [Bro] Question about network cards >> >> Hello, >> >> I?m wondering what people are using for network cards in their bro clusters that are not using the Myricom Network Cards. We don?t have a $1,000 dollars per a card + license to spend on the cards. Is anyone using Intel or other brands that aren?t as expensive to capture their traffic? We are looking at doing all 10 Gig connections into the Bro Cluster. >> >> Thanks for all your answers. >> >> -- >> Richard Giesige >> IT Security Analyst >> Office of Information Security >> Oregon State University >> >> "OSU staff will NEVER ask for you password. >> Never email or share your password with anyone." >> >> >> Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email. >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From bro at pingtrip.com Wed Apr 13 16:03:19 2016 From: bro at pingtrip.com (Dave Crawford) Date: Wed, 13 Apr 2016 19:03:19 -0400 Subject: [Bro] High-CPU on just a single worker in the cluster Message-ID: <45D4C014-4574-49F6-ACEF-97ADE67B94B4@pingtrip.com> I'm in the process of trying to debug an odd high-cpu issue and looking for guidance. The deployment is a follows: - Cluster has with two nodes, each with 10 workers and the workers are pinned to specific cpu cores. - x520 with PF_RING - Traffic to each node is load balanced equally The issue is that one worker on one of the nodes is always at 100% CPU while all other workers are around 50%. If I restart Bro a different worker will pin to 100%, but always on the same node. I ran 'strace' on both a "bad" and "good" worker and one anomaly I spotted was that the "bad" worker never called 'nanosleep', whereas the "good" worker had about 84,000 'nanosleep' calls in the same amount of time. I'm wondering if its possible for a queue to go bad on the x520, which might explain why its a random worker on the same node after restarting. Is there a way to determine which x520 queue a specific worker is reading from? Thanks, -Dave From jazoff at illinois.edu Wed Apr 13 18:43:26 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 14 Apr 2016 01:43:26 +0000 Subject: [Bro] High-CPU on just a single worker in the cluster In-Reply-To: <45D4C014-4574-49F6-ACEF-97ADE67B94B4@pingtrip.com> References: <45D4C014-4574-49F6-ACEF-97ADE67B94B4@pingtrip.com> Message-ID: Can you load this script that will add a node column to the conn.log that says which node handled that connection: https://github.com/broala/bro-snippets/blob/master/add-node-to-conn.bro also, what 'broctl netstats' outputs would be useful to see. -- - Justin Azoff > On Apr 13, 2016, at 7:03 PM, Dave Crawford wrote: > > I'm in the process of trying to debug an odd high-cpu issue and looking for guidance. > > The deployment is a follows: > - Cluster has with two nodes, each with 10 workers and the workers are pinned to specific cpu cores. > - x520 with PF_RING > - Traffic to each node is load balanced equally > > The issue is that one worker on one of the nodes is always at 100% CPU while all other workers are around 50%. If I restart Bro a different worker will pin to 100%, but always on the same node. > > I ran 'strace' on both a "bad" and "good" worker and one anomaly I spotted was that the "bad" worker never called 'nanosleep', whereas the "good" worker had about 84,000 'nanosleep' calls in the same amount of time. > > I'm wondering if its possible for a queue to go bad on the x520, which might explain why its a random worker on the same node after restarting. > > Is there a way to determine which x520 queue a specific worker is reading from? > > Thanks, > -Dave > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From bro at pingtrip.com Thu Apr 14 05:26:43 2016 From: bro at pingtrip.com (Dave Crawford) Date: Thu, 14 Apr 2016 08:26:43 -0400 Subject: [Bro] High-CPU on just a single worker in the cluster In-Reply-To: References: <45D4C014-4574-49F6-ACEF-97ADE67B94B4@pingtrip.com> Message-ID: <448CE5C2-AB2F-411C-9EA6-AED7490F52B2@pingtrip.com> I'm already capturing the node in conn.log but haven't been able to spot anything out off the ordinary compared to the other nodes. Below is a fresh 'netstats' from this morning (WIN_INT-9 is obviously the culprit). MID_INT-1: 1460636199.609909 recvd=327807625 dropped=1 link=327807625 MID_INT-10: 1460636199.813720 recvd=339804389 dropped=4 link=339804389 MID_INT-2: 1460636200.010047 recvd=313901304 dropped=0 link=313901304 MID_INT-3: 1460636200.210033 recvd=323507786 dropped=1 link=323507786 MID_INT-4: 1460636200.413951 recvd=322338069 dropped=0 link=322338069 MID_INT-5: 1460636200.613996 recvd=314681107 dropped=1 link=314681107 MID_INT-6: 1460636200.814761 recvd=325488973 dropped=1 link=325488973 MID_INT-7: 1460636201.017945 recvd=328830658 dropped=3 link=328830658 MID_INT-8: 1460636201.218113 recvd=338250015 dropped=0 link=338250015 MID_INT-9: 1460636201.417949 recvd=387979776 dropped=0 link=387979776 WIN_INT-1: 1460636288.903341 recvd=142474122 dropped=1 link=142474122 WIN_INT-10: 1460636289.103648 recvd=232076131 dropped=1 link=232076131 WIN_INT-2: 1460636289.303290 recvd=145451659 dropped=2 link=145451659 WIN_INT-3: 1460636289.507242 recvd=182345947 dropped=0 link=182345947 WIN_INT-4: 1460636289.707591 recvd=140378820 dropped=1 link=140378820 WIN_INT-5: 1460636289.911410 recvd=140342198 dropped=0 link=140342198 WIN_INT-6: 1460636290.111178 recvd=138961706 dropped=0 link=138961706 WIN_INT-7: 1460636290.315433 recvd=198792251 dropped=0 link=198792251 WIN_INT-8: 1460636290.515158 recvd=170824302 dropped=3 link=170824302 WIN_INT-9: 1460636287.108095 recvd=2414368833 dropped=438939600 link=2414368833 > On Apr 13, 2016, at 9:43 PM, Azoff, Justin S wrote: > > Can you load this script that will add a node column to the conn.log that says which node handled that connection: > > https://github.com/broala/bro-snippets/blob/master/add-node-to-conn.bro > > also, what 'broctl netstats' outputs would be useful to see. > > > -- > - Justin Azoff > >> On Apr 13, 2016, at 7:03 PM, Dave Crawford wrote: >> >> I'm in the process of trying to debug an odd high-cpu issue and looking for guidance. >> >> The deployment is a follows: >> - Cluster has with two nodes, each with 10 workers and the workers are pinned to specific cpu cores. >> - x520 with PF_RING >> - Traffic to each node is load balanced equally >> >> The issue is that one worker on one of the nodes is always at 100% CPU while all other workers are around 50%. If I restart Bro a different worker will pin to 100%, but always on the same node. >> >> I ran 'strace' on both a "bad" and "good" worker and one anomaly I spotted was that the "bad" worker never called 'nanosleep', whereas the "good" worker had about 84,000 'nanosleep' calls in the same amount of time. >> >> I'm wondering if its possible for a queue to go bad on the x520, which might explain why its a random worker on the same node after restarting. >> >> Is there a way to determine which x520 queue a specific worker is reading from? >> >> Thanks, >> -Dave >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From jazoff at illinois.edu Thu Apr 14 06:25:37 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 14 Apr 2016 13:25:37 +0000 Subject: [Bro] High-CPU on just a single worker in the cluster In-Reply-To: <448CE5C2-AB2F-411C-9EA6-AED7490F52B2@pingtrip.com> References: <45D4C014-4574-49F6-ACEF-97ADE67B94B4@pingtrip.com> <448CE5C2-AB2F-411C-9EA6-AED7490F52B2@pingtrip.com> Message-ID: <536AD81E-BA9C-4A47-ADB1-06185FD71200@illinois.edu> > On Apr 14, 2016, at 8:26 AM, Dave Crawford wrote: > > I'm already capturing the node in conn.log but haven't been able to spot anything out off the ordinary compared to the other nodes. > > Below is a fresh 'netstats' from this morning (WIN_INT-9 is obviously the culprit). > > MID_INT-1: 1460636199.609909 recvd=327807625 dropped=1 link=327807625 > MID_INT-10: 1460636199.813720 recvd=339804389 dropped=4 link=339804389 > MID_INT-2: 1460636200.010047 recvd=313901304 dropped=0 link=313901304 > MID_INT-3: 1460636200.210033 recvd=323507786 dropped=1 link=323507786 > MID_INT-4: 1460636200.413951 recvd=322338069 dropped=0 link=322338069 > MID_INT-5: 1460636200.613996 recvd=314681107 dropped=1 link=314681107 > MID_INT-6: 1460636200.814761 recvd=325488973 dropped=1 link=325488973 > MID_INT-7: 1460636201.017945 recvd=328830658 dropped=3 link=328830658 > MID_INT-8: 1460636201.218113 recvd=338250015 dropped=0 link=338250015 > MID_INT-9: 1460636201.417949 recvd=387979776 dropped=0 link=387979776 > WIN_INT-1: 1460636288.903341 recvd=142474122 dropped=1 link=142474122 > WIN_INT-10: 1460636289.103648 recvd=232076131 dropped=1 link=232076131 > WIN_INT-2: 1460636289.303290 recvd=145451659 dropped=2 link=145451659 > WIN_INT-3: 1460636289.507242 recvd=182345947 dropped=0 link=182345947 > WIN_INT-4: 1460636289.707591 recvd=140378820 dropped=1 link=140378820 > WIN_INT-5: 1460636289.911410 recvd=140342198 dropped=0 link=140342198 > WIN_INT-6: 1460636290.111178 recvd=138961706 dropped=0 link=138961706 > WIN_INT-7: 1460636290.315433 recvd=198792251 dropped=0 link=198792251 > WIN_INT-8: 1460636290.515158 recvd=170824302 dropped=3 link=170824302 > WIN_INT-9: 1460636287.108095 recvd=2414368833 dropped=438939600 link=2414368833 > Ahh, well that last worker is seeing almost double the number of packets than all of the other workers on that host combined, so that explains the CPU usage. What does a frequency distribution of the node column from your conn.log from around that time show? zcat conn.....gz | bro-cut node | sort | uniq -c Adding this to local.bro @load misc/stats If you don't have it already will give you a stats.log which will contain some helpful information too. If I had to guess, there's probably something going on traffic wise.. if I had to guess the WIN box is seeing a ton of non-ip traffic that all gets load balanced to the same worker. Can you run capstats on the two boxes and compare the output? # capstats -i p1p1 -I 2 1460639895.723665 pkts=6642 kpps=2.8 kbytes=2100 mbps=7.2 nic_pkts=6642 nic_drops=0 u=0 t=0 i=0 o=0 nonip=6642 I just noticed that capstats doesn't properly handle vlan encapsulated packets, so all of our traffic shows up as nonip.. I'll look into fixing that, but if you are not using vlans looking at the breakdown of udp,tcp,ip,other,nonip (u,t,i,o,nonip) would help. -- - Justin Azoff From bro at pingtrip.com Thu Apr 14 06:55:09 2016 From: bro at pingtrip.com (Dave Crawford) Date: Thu, 14 Apr 2016 09:55:09 -0400 Subject: [Bro] High-CPU on just a single worker in the cluster In-Reply-To: <536AD81E-BA9C-4A47-ADB1-06185FD71200@illinois.edu> References: <45D4C014-4574-49F6-ACEF-97ADE67B94B4@pingtrip.com> <448CE5C2-AB2F-411C-9EA6-AED7490F52B2@pingtrip.com> <536AD81E-BA9C-4A47-ADB1-06185FD71200@illinois.edu> Message-ID: <437C06EF-0962-4D66-9513-0E952F30740B@pingtrip.com> Here is the frequency distribution (I'm logging in JSON so different command but same results): $ zcat 2016-04-14/conn.08\:00\:00-09\:00\:00.log.gz | jq -r '.node' | sort | uniq -c 192470 MID_GLR-1 192325 MID_GLR-2 193491 MID_GLR-3 192444 MID_GLR-4 192288 MID_GLR-5 654908 MID_INT-1 655252 MID_INT-10 652749 MID_INT-2 657361 MID_INT-3 655236 MID_INT-4 656477 MID_INT-5 654518 MID_INT-6 656199 MID_INT-7 656069 MID_INT-8 655798 MID_INT-9 770362 WIN_INT-1 772679 WIN_INT-10 773520 WIN_INT-2 772197 WIN_INT-3 768066 WIN_INT-4 771089 WIN_INT-5 771734 WIN_INT-6 772599 WIN_INT-7 772359 WIN_INT-8 721526 WIN_INT-9 You may be on to something with the non-ip traffic... there is a drastic difference between the two datacenters: WIN 1460641772.239436 pkts=10414545 kpps=208.2 kbytes=5732528 mbps=938.6 nic_pkts=10414545 nic_drops=0 u=104675 t=3627503 i=307 o=405 nonip=6681655 MID 1460641723.573448 pkts=9553569 kpps=178.9 kbytes=6561123 mbps=1006.6 nic_pkts=9553569 nic_drops=0 u=174140 t=9373195 i=267 o=934 nonip=5033 -Dave > On Apr 14, 2016, at 9:25 AM, Azoff, Justin S wrote: > >> On Apr 14, 2016, at 8:26 AM, Dave Crawford wrote: >> >> I'm already capturing the node in conn.log but haven't been able to spot anything out off the ordinary compared to the other nodes. >> >> Below is a fresh 'netstats' from this morning (WIN_INT-9 is obviously the culprit). >> >> MID_INT-1: 1460636199.609909 recvd=327807625 dropped=1 link=327807625 >> MID_INT-10: 1460636199.813720 recvd=339804389 dropped=4 link=339804389 >> MID_INT-2: 1460636200.010047 recvd=313901304 dropped=0 link=313901304 >> MID_INT-3: 1460636200.210033 recvd=323507786 dropped=1 link=323507786 >> MID_INT-4: 1460636200.413951 recvd=322338069 dropped=0 link=322338069 >> MID_INT-5: 1460636200.613996 recvd=314681107 dropped=1 link=314681107 >> MID_INT-6: 1460636200.814761 recvd=325488973 dropped=1 link=325488973 >> MID_INT-7: 1460636201.017945 recvd=328830658 dropped=3 link=328830658 >> MID_INT-8: 1460636201.218113 recvd=338250015 dropped=0 link=338250015 >> MID_INT-9: 1460636201.417949 recvd=387979776 dropped=0 link=387979776 >> WIN_INT-1: 1460636288.903341 recvd=142474122 dropped=1 link=142474122 >> WIN_INT-10: 1460636289.103648 recvd=232076131 dropped=1 link=232076131 >> WIN_INT-2: 1460636289.303290 recvd=145451659 dropped=2 link=145451659 >> WIN_INT-3: 1460636289.507242 recvd=182345947 dropped=0 link=182345947 >> WIN_INT-4: 1460636289.707591 recvd=140378820 dropped=1 link=140378820 >> WIN_INT-5: 1460636289.911410 recvd=140342198 dropped=0 link=140342198 >> WIN_INT-6: 1460636290.111178 recvd=138961706 dropped=0 link=138961706 >> WIN_INT-7: 1460636290.315433 recvd=198792251 dropped=0 link=198792251 >> WIN_INT-8: 1460636290.515158 recvd=170824302 dropped=3 link=170824302 >> WIN_INT-9: 1460636287.108095 recvd=2414368833 dropped=438939600 link=2414368833 >> > > Ahh, well that last worker is seeing almost double the number of packets than all of the other workers on that host combined, so that explains the CPU usage. > > What does a frequency distribution of the node column from your conn.log from around that time show? > > zcat conn.....gz | bro-cut node | sort | uniq -c > > Adding this to local.bro > > @load misc/stats > > If you don't have it already will give you a stats.log which will contain some helpful information too. > > If I had to guess, there's probably something going on traffic wise.. if I had to guess the WIN box is seeing a ton of non-ip traffic that all gets load balanced to the same worker. > > Can you run capstats on the two boxes and compare the output? > > # capstats -i p1p1 -I 2 > 1460639895.723665 pkts=6642 kpps=2.8 kbytes=2100 mbps=7.2 nic_pkts=6642 nic_drops=0 u=0 t=0 i=0 o=0 nonip=6642 > > I just noticed that capstats doesn't properly handle vlan encapsulated packets, so all of our traffic shows up as nonip.. I'll look into fixing that, but if you are not using vlans looking at the breakdown of udp,tcp,ip,other,nonip (u,t,i,o,nonip) would help. > > -- > - Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160414/c8caf449/attachment-0001.html From seth at icir.org Thu Apr 14 06:58:36 2016 From: seth at icir.org (Seth Hall) Date: Thu, 14 Apr 2016 09:58:36 -0400 Subject: [Bro] High-CPU on just a single worker in the cluster In-Reply-To: <448CE5C2-AB2F-411C-9EA6-AED7490F52B2@pingtrip.com> References: <45D4C014-4574-49F6-ACEF-97ADE67B94B4@pingtrip.com> <448CE5C2-AB2F-411C-9EA6-AED7490F52B2@pingtrip.com> Message-ID: > On Apr 14, 2016, at 8:26 AM, Dave Crawford wrote: > > Below is a fresh 'netstats' from this morning (WIN_INT-9 is obviously the culprit). You are seeing the side effects of hot spotting. It's likely that you have a connection which is massively larger than all others that your host is seeing. You could filter your traffic to stop seeing that flow which would cut down on the problems you are seeing. I just pushed a new repository that might help you address your issue. It will still require a lot of clean up and documentation but if you just load the module as-is, it might automatically help you. :) It automatically identifies large and fast flows and places BPF filters to automatically shunt them while still watching for SYN/ACK/FIN/RST flags. https://github.com/sethhall/bro-bulkflows There are a couple of configuration options you'll want to pay attention to, so I'll just give you an example blurb you could put into your local.bro: @load bro-bulkflows # Set a minimum flow size before considering shunting (32MB) redef BulkFlows::size_threshold = 33554432; # Set a minimum flow data rate before considering shunting (50Mbps) BulkFlows::speed_threshold = 6250000; # If you have VLAN tags on your traffic, you will also have to add the following line because packet filtering with BPF is generally broken otherwise... #redef PacketFilter::restricted_filter="vlan"; .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From mabuchan at gmail.com Thu Apr 14 07:33:52 2016 From: mabuchan at gmail.com (Mark Buchanan) Date: Thu, 14 Apr 2016 09:33:52 -0500 Subject: [Bro] High-CPU on just a single worker in the cluster In-Reply-To: References: <45D4C014-4574-49F6-ACEF-97ADE67B94B4@pingtrip.com> <448CE5C2-AB2F-411C-9EA6-AED7490F52B2@pingtrip.com> Message-ID: <365CD418-8457-4E42-B7F4-C8BCF5238E42@gmail.com> I'm not 100% familiar with the load balancing methods, but know if your doing a 5 tuple load balance algorithm, I have seen IPsec stick to a single location, as there are effectively only three tuple available (Src IP, Dst IP and proto), which don't change for most static IPSec connections. Which results in the traffic not changing and sliding from worker to worker like normal connections would. Could there be a high use IPSec tunnel between that Datacenter and another location? -- Mark Buchanan > On Apr 14, 2016, at 08:58, Seth Hall wrote: > > >> On Apr 14, 2016, at 8:26 AM, Dave Crawford wrote: >> >> Below is a fresh 'netstats' from this morning (WIN_INT-9 is obviously the culprit). > > You are seeing the side effects of hot spotting. It's likely that you have a connection which is massively larger than all others that your host is seeing. You could filter your traffic to stop seeing that flow which would cut down on the problems you are seeing. > > I just pushed a new repository that might help you address your issue. It will still require a lot of clean up and documentation but if you just load the module as-is, it might automatically help you. :) It automatically identifies large and fast flows and places BPF filters to automatically shunt them while still watching for SYN/ACK/FIN/RST flags. > > https://github.com/sethhall/bro-bulkflows > > There are a couple of configuration options you'll want to pay attention to, so I'll just give you an example blurb you could put into your local.bro: > > @load bro-bulkflows > # Set a minimum flow size before considering shunting (32MB) > redef BulkFlows::size_threshold = 33554432; > # Set a minimum flow data rate before considering shunting (50Mbps) > BulkFlows::speed_threshold = 6250000; > > # If you have VLAN tags on your traffic, you will also have to add the following line because packet filtering with BPF is generally broken otherwise... > #redef PacketFilter::restricted_filter="vlan"; > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2182 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160414/9f885ef5/attachment.bin From jazoff at illinois.edu Thu Apr 14 07:41:00 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 14 Apr 2016 14:41:00 +0000 Subject: [Bro] High-CPU on just a single worker in the cluster In-Reply-To: <437C06EF-0962-4D66-9513-0E952F30740B@pingtrip.com> References: <45D4C014-4574-49F6-ACEF-97ADE67B94B4@pingtrip.com> <448CE5C2-AB2F-411C-9EA6-AED7490F52B2@pingtrip.com> <536AD81E-BA9C-4A47-ADB1-06185FD71200@illinois.edu> <437C06EF-0962-4D66-9513-0E952F30740B@pingtrip.com> Message-ID: <1534307A-D400-4380-98CE-0D5D34BF26B7@illinois.edu> > On Apr 14, 2016, at 9:55 AM, Dave Crawford wrote: > > > You may be on to something with the non-ip traffic... there is a drastic difference between the two datacenters: > > WIN > 1460641772.239436 pkts=10414545 kpps=208.2 kbytes=5732528 mbps=938.6 nic_pkts=10414545 nic_drops=0 u=104675 t=3627503 i=307 o=405 nonip=6681655 > > MID > 1460641723.573448 pkts=9553569 kpps=178.9 kbytes=6561123 mbps=1006.6 nic_pkts=9553569 nic_drops=0 u=174140 t=9373195 i=267 o=934 nonip=5033 > Great.. just what I was thinking. At this point you should be able to just run something like tcpdump -n -c 1000 'not ip' on the WIN box and see exactly what this traffic is.. then we can figure out what to do about it... -- - Justin Azoff From bro at pingtrip.com Thu Apr 14 08:18:17 2016 From: bro at pingtrip.com (Dave Crawford) Date: Thu, 14 Apr 2016 11:18:17 -0400 Subject: [Bro] High-CPU on just a single worker in the cluster In-Reply-To: <1534307A-D400-4380-98CE-0D5D34BF26B7@illinois.edu> References: <45D4C014-4574-49F6-ACEF-97ADE67B94B4@pingtrip.com> <448CE5C2-AB2F-411C-9EA6-AED7490F52B2@pingtrip.com> <536AD81E-BA9C-4A47-ADB1-06185FD71200@illinois.edu> <437C06EF-0962-4D66-9513-0E952F30740B@pingtrip.com> <1534307A-D400-4380-98CE-0D5D34BF26B7@illinois.edu> Message-ID: $ sudo tcpdump -n -i eth6 not ip and not arp -c10000 | grep ethertype | cut -f 2 -d ',' | sort | uniq -c 9980 ethertype Unknown (0x8903) A quick Google points to Cisco FabricPath Switching ( http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/fabricpath/configuration/guide/fp_switching.html ) "The FabricPath hierarchical MAC address carries the reserved EtherType 0x8903." I suppose now is a good time to reach out to the Network Engineering team and ask about the SPAN placement in that datacenter. Thanks for helping me quickly navigate this issue! -Dave > On Apr 14, 2016, at 10:41 AM, Azoff, Justin S wrote: > > >> On Apr 14, 2016, at 9:55 AM, Dave Crawford wrote: >> >> >> You may be on to something with the non-ip traffic... there is a drastic difference between the two datacenters: >> >> WIN >> 1460641772.239436 pkts=10414545 kpps=208.2 kbytes=5732528 mbps=938.6 nic_pkts=10414545 nic_drops=0 u=104675 t=3627503 i=307 o=405 nonip=6681655 >> >> MID >> 1460641723.573448 pkts=9553569 kpps=178.9 kbytes=6561123 mbps=1006.6 nic_pkts=9553569 nic_drops=0 u=174140 t=9373195 i=267 o=934 nonip=5033 >> > > Great.. just what I was thinking. At this point you should be able to just run something like > > tcpdump -n -c 1000 'not ip' > > on the WIN box > > and see exactly what this traffic is.. then we can figure out what to do about it... > > > -- > - Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160414/356f3415/attachment.html From seth at icir.org Thu Apr 14 09:50:05 2016 From: seth at icir.org (Seth Hall) Date: Thu, 14 Apr 2016 12:50:05 -0400 Subject: [Bro] High-CPU on just a single worker in the cluster In-Reply-To: References: <45D4C014-4574-49F6-ACEF-97ADE67B94B4@pingtrip.com> <448CE5C2-AB2F-411C-9EA6-AED7490F52B2@pingtrip.com> Message-ID: > On Apr 14, 2016, at 9:58 AM, Seth Hall wrote: > > https://github.com/sethhall/bro-bulkflows This repository will probably go away at some point. I need to just integrate some small changes into Justin's bro-react[1] repository to make it work with BPF filters. Apparently I had gotten that code from Justin a long time ago and subsequently forgotten about it. Makes more sense to just integrate it into his, but this is functionality that I think could make sense to integrate into Bro eventually as a nice core pairing with the upcoming NetControl framework. .Seth 1. https://github.com/JustinAzoff/bro-react -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jazoff at illinois.edu Thu Apr 14 10:11:25 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 14 Apr 2016 17:11:25 +0000 Subject: [Bro] High-CPU on just a single worker in the cluster In-Reply-To: References: <45D4C014-4574-49F6-ACEF-97ADE67B94B4@pingtrip.com> <448CE5C2-AB2F-411C-9EA6-AED7490F52B2@pingtrip.com> <536AD81E-BA9C-4A47-ADB1-06185FD71200@illinois.edu> <437C06EF-0962-4D66-9513-0E952F30740B@pingtrip.com> <1534307A-D400-4380-98CE-0D5D34BF26B7@illinois.edu> Message-ID: <1FEBC31E-FA38-484D-8EA8-66153CE8E8F5@illinois.edu> > On Apr 14, 2016, at 11:18 AM, Dave Crawford wrote: > > > $ sudo tcpdump -n -i eth6 not ip and not arp -c10000 | grep ethertype | cut -f 2 -d ',' | sort | uniq -c > 9980 ethertype Unknown (0x8903) > > > A quick Google points to Cisco FabricPath Switching ( http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/fabricpath/configuration/guide/fp_switching.html) > > "The FabricPath hierarchical MAC address carries the reserved EtherType 0x8903." > > I suppose now is a good time to reach out to the Network Engineering team and ask about the SPAN placement in that datacenter. > > Thanks for helping me quickly navigate this issue! > > -Dave Ah.. so there are probably two issues here: bro and pf_ring Based on this image from your link https://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/fabricpath/configuration/guide/fp_switching_files/fp_switching-1.jpg the FP header is fixed size, so adding support for it to bro should be easy enough: handle that ether type, skip the right number of bytes (see iosource/Packet.cc) That won't help with the pf_ring issue though, but you're probably best reaching out to the pf_ring people about this issue. It's entirely possibly you can fix the issue with a different span configuration though. -- - Justin Azoff From jlamps at sandia.gov Thu Apr 14 12:06:23 2016 From: jlamps at sandia.gov (Lamps, Jereme) Date: Thu, 14 Apr 2016 19:06:23 +0000 Subject: [Bro] Determining when the Conn::log_conn event executes Message-ID: Does anyone have an idea of when the Conn::log_conn event executes? Does it execute whenever Bro believes the connection is finished, or can it execute earlier than that? Thanks in advance -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160414/6be6f552/attachment.html From jlay at slave-tothe-box.net Thu Apr 14 12:33:52 2016 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 14 Apr 2016 13:33:52 -0600 Subject: [Bro] Determining when the Conn::log_conn event executes In-Reply-To: References: Message-ID: <1d1aa11e244d9b4ee8360f75bc73b718@localhost> On 2016-04-14 13:06, Lamps, Jereme wrote: > Does anyone have an idea of when the Conn::log_conn event executes? > Does it execute whenever Bro believes the connection is finished, or > can it execute earlier than that? > > Thanks in advance > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro My understanding is once the session has completed. James From josh.guild at morphick.com Thu Apr 14 13:18:22 2016 From: josh.guild at morphick.com (Josh Guild) Date: Thu, 14 Apr 2016 16:18:22 -0400 Subject: [Bro] weird.log help Message-ID: Howdy all, I'm trying to debug some traffic that is coming off an aggregator right now. I was pointed to this helpful set of slides from Vlad on how to troubleshoot and verify a network ( https://speakerdeck.com/vladg/bro-deployment-verification-and-troubleshooting ). Looking at the weird.log from a ~2 min pcap on a network with ~6 Gbps throughput, I've noticed these entries in the weird.log (top 10 or so). 5454 line_terminated_with_single_CR 4012 above_hole_data_without_any_acks 2827 TCP_ack_underflow_or_misorder 2601 SYN_seq_jump 2395 TCP_seq_underflow_or_misorder 2192 FIN_advanced_last_seq 1330 HTTP_version_mismatch 570 bad_HTTP_request 333 unescaped_special_URI_char 205 window_recision 151 dns_unmatched_msg Now my questions are these - 1) That seems like a lot of errors for a small sample set but I don't have a reference point for a network of this size. Does anyone else have an equivalent network that they could sanity check for me? 2) Is there a good reference for these weird.log entries that I can look at to try to pin down what is going wrong in the network? I'm particularly interested in the HTTP_version_mismatch and a few other that Vlad mentioned in his presentation. The main reason I'm interested in the details on HTTP_version_mismatch is because I have two pcaps from two separate ports off the aggregator and, for some reason, one is showing as HTTP2 (but only in the OSX version of Wireshark) and Bro can't read pcap properly. The other pcap is read just fine. Sorry for the wall of text but if anyone can point me in the right direction, I'd be much obliged. Thanks! -- Josh Guild Network Intelligence Analyst -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160414/0d04e74e/attachment.html From daniel.guerra69 at gmail.com Thu Apr 14 15:07:17 2016 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Fri, 15 Apr 2016 00:07:17 +0200 Subject: [Bro] weird.log help In-Reply-To: References: Message-ID: <3B302039-AC80-4CC2-8B0E-A37D080A6647@gmail.com> I don?t know your situation but this looks like reordering problem. All tools expect a time order. Timeout increase might help. > On 14 Apr 2016, at 22:18, Josh Guild wrote: > > Howdy all, > > I'm trying to debug some traffic that is coming off an aggregator right now. I was pointed to this helpful set of slides from Vlad on how to troubleshoot and verify a network (https://speakerdeck.com/vladg/bro-deployment-verification-and-troubleshooting ). > > Looking at the weird.log from a ~2 min pcap on a network with ~6 Gbps throughput, I've noticed these entries in the weird.log (top 10 or so). > > 5454 line_terminated_with_single_CR > 4012 above_hole_data_without_any_acks > 2827 TCP_ack_underflow_or_misorder > 2601 SYN_seq_jump > 2395 TCP_seq_underflow_or_misorder > 2192 FIN_advanced_last_seq > 1330 HTTP_version_mismatch > 570 bad_HTTP_request > 333 unescaped_special_URI_char > 205 window_recision > 151 dns_unmatched_msg > > Now my questions are these - 1) That seems like a lot of errors for a small sample set but I don't have a reference point for a network of this size. Does anyone else have an equivalent network that they could sanity check for me? 2) Is there a good reference for these weird.log entries that I can look at to try to pin down what is going wrong in the network? I'm particularly interested in the HTTP_version_mismatch and a few other that Vlad mentioned in his presentation. > > The main reason I'm interested in the details on HTTP_version_mismatch is because I have two pcaps from two separate ports off the aggregator and, for some reason, one is showing as HTTP2 (but only in the OSX version of Wireshark) and Bro can't read pcap properly. The other pcap is read just fine. > > Sorry for the wall of text but if anyone can point me in the right direction, I'd be much obliged. Thanks! > > > -- > Josh Guild > Network Intelligence Analyst > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160415/23a93e10/attachment.html From josh.guild at morphick.com Fri Apr 15 05:35:19 2016 From: josh.guild at morphick.com (Josh Guild) Date: Fri, 15 Apr 2016 08:35:19 -0400 Subject: [Bro] weird.log help In-Reply-To: <3B302039-AC80-4CC2-8B0E-A37D080A6647@gmail.com> References: <3B302039-AC80-4CC2-8B0E-A37D080A6647@gmail.com> Message-ID: Thanks, Dan, I'll look into this. When I analyze the pcap in Wireshark I see a lot of "port reuse" errors as well which I think it indicative of this as well. On Thu, Apr 14, 2016 at 6:07 PM, Daniel Guerra wrote: > I don?t know your situation but this looks like reordering problem. All > tools expect a time order. > > Timeout increase might help. > > On 14 Apr 2016, at 22:18, Josh Guild wrote: > > Howdy all, > > I'm trying to debug some traffic that is coming off an aggregator right > now. I was pointed to this helpful set of slides from Vlad on how to > troubleshoot and verify a network ( > https://speakerdeck.com/vladg/bro-deployment-verification-and-troubleshooting > ). > > Looking at the weird.log from a ~2 min pcap on a network with ~6 Gbps > throughput, I've noticed these entries in the weird.log (top 10 or so). > > 5454 line_terminated_with_single_CR > 4012 above_hole_data_without_any_acks > 2827 TCP_ack_underflow_or_misorder > 2601 SYN_seq_jump > 2395 TCP_seq_underflow_or_misorder > 2192 FIN_advanced_last_seq > 1330 HTTP_version_mismatch > 570 bad_HTTP_request > 333 unescaped_special_URI_char > 205 window_recision > 151 dns_unmatched_msg > > Now my questions are these - 1) That seems like a lot of errors for a > small sample set but I don't have a reference point for a network of this > size. Does anyone else have an equivalent network that they could sanity > check for me? 2) Is there a good reference for these weird.log entries that > I can look at to try to pin down what is going wrong in the network? I'm > particularly interested in the HTTP_version_mismatch and a few other that > Vlad mentioned in his presentation. > > The main reason I'm interested in the details on HTTP_version_mismatch is > because I have two pcaps from two separate ports off the aggregator and, > for some reason, one is showing as HTTP2 (but only in the OSX version of > Wireshark) and Bro can't read pcap properly. The other pcap is read just > fine. > > Sorry for the wall of text but if anyone can point me in the right > direction, I'd be much obliged. Thanks! > > > -- > Josh Guild > Network Intelligence Analyst > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > -- Josh Guild Network Intelligence Analyst -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160415/68c8b42b/attachment-0001.html From daniel.guerra69 at gmail.com Fri Apr 15 15:43:33 2016 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Sat, 16 Apr 2016 00:43:33 +0200 Subject: [Bro] weird.log help In-Reply-To: References: <3B302039-AC80-4CC2-8B0E-A37D080A6647@gmail.com> Message-ID: Use reorderpcap https://www.wireshark.org/docs/man-pages/reordercap.html Or it might help to install the tcprs plugin. You could could some timeout tweaking too. > On 15 Apr 2016, at 14:35, Josh Guild wrote: > > Thanks, Dan, I'll look into this. > When I analyze the pcap in Wireshark I see a lot of "port reuse" errors as well which I think it indicative of this as well. > > > On Thu, Apr 14, 2016 at 6:07 PM, Daniel Guerra > wrote: > I don?t know your situation but this looks like reordering problem. All tools expect a time order. > > Timeout increase might help. > >> On 14 Apr 2016, at 22:18, Josh Guild > wrote: >> >> Howdy all, >> >> I'm trying to debug some traffic that is coming off an aggregator right now. I was pointed to this helpful set of slides from Vlad on how to troubleshoot and verify a network (https://speakerdeck.com/vladg/bro-deployment-verification-and-troubleshooting ). >> >> Looking at the weird.log from a ~2 min pcap on a network with ~6 Gbps throughput, I've noticed these entries in the weird.log (top 10 or so). >> >> 5454 line_terminated_with_single_CR >> 4012 above_hole_data_without_any_acks >> 2827 TCP_ack_underflow_or_misorder >> 2601 SYN_seq_jump >> 2395 TCP_seq_underflow_or_misorder >> 2192 FIN_advanced_last_seq >> 1330 HTTP_version_mismatch >> 570 bad_HTTP_request >> 333 unescaped_special_URI_char >> 205 window_recision >> 151 dns_unmatched_msg >> >> Now my questions are these - 1) That seems like a lot of errors for a small sample set but I don't have a reference point for a network of this size. Does anyone else have an equivalent network that they could sanity check for me? 2) Is there a good reference for these weird.log entries that I can look at to try to pin down what is going wrong in the network? I'm particularly interested in the HTTP_version_mismatch and a few other that Vlad mentioned in his presentation. >> >> The main reason I'm interested in the details on HTTP_version_mismatch is because I have two pcaps from two separate ports off the aggregator and, for some reason, one is showing as HTTP2 (but only in the OSX version of Wireshark) and Bro can't read pcap properly. The other pcap is read just fine. >> >> Sorry for the wall of text but if anyone can point me in the right direction, I'd be much obliged. Thanks! >> >> >> -- >> Josh Guild >> Network Intelligence Analyst >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > -- > Josh Guild > Network Intelligence Analyst > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160416/34ef87e5/attachment.html From beikejinmiao at gmail.com Thu Apr 14 05:02:10 2016 From: beikejinmiao at gmail.com (=?UTF-8?B?5p2O6YeR6IuX?=) Date: Thu, 14 Apr 2016 20:02:10 +0800 Subject: [Bro] Traffic just into a single worker in the cluster Message-ID: The deployment is a follows: - Cluster has with two nodes, each with 2 workers and the workers are pinned to specific cpu cores. - dell r620 and centos7 with PF_RING [root at centos soft]# uname -a Linux centos 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux [root at centos~]# cat /proc/net/pf_ring/info PF_RING Version : 6.3.0 (unknown) Total rings : 4 Standard (non DNA/ZC) Options Ring slots : 32768 Slot version : 16 Capture TX : Yes [RX+TX] IP Defragment : No Socket Mode : Standard Total plugins : 0 Cluster Fragment Queue : 0 Cluster Fragment Discard : 0 broctl capture [BroControl] > netstats worker-1-1: 1460663117.341926 recvd=5317 dropped=0 link=5317 worker-1-2: 1460663117.542621 recvd=524 dropped=0 link=524 worker-2-1: 1460663117.751165 recvd=17 dropped=0 link=17 worker-2-2: 1460663117.945068 recvd=14417 dropped=0 link=14417 You can see almost all of the traffic capture by the woker-2-2. And I downloaded 100 files, and only 25 files that could be extracted. ?I found an very interesting phenomenon? If I start 1 workers, all of the files can be extracted. If I start 2 workers, 50% of the files can be extracted. If I start 4 workers, 25% of the files can be extracted. ....... I don't know why . Could any one give me some advice.Thanks very much. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160414/47fd76b4/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: bro-cluster1.PNG Type: image/png Size: 77889 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160414/47fd76b4/attachment-0001.bin From bradjcox at gmail.com Sat Apr 16 13:05:18 2016 From: bradjcox at gmail.com (Brad Cox) Date: Sat, 16 Apr 2016 16:05:18 -0400 Subject: [Bro] How to parse bro decimal timestamps? Message-ID: How do I turn the timestamp (ts) field in this example into a standard date format (java or unix dates for example?) set_separator , #empty_field (empty) #unset_field - #path conn #open 2016-04-04-09-00-04 #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig local_resp missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents #types time string addr port addr port enum string interval count count string bool bool count string count count count count set[string] 1459774793.429104 CZgDTe31Z6ynNuzgN7 fe80::c874:93f:5b4e:c1e1 64648 ff02::1:3 5355 udp dns 0.412428 44 0 S0 F F 0 D 2 140 0 0 (empty) 1459774793.429113 Ci77TT3Kp4dNmhAYc1 172.16.2.33 64648 224.0.0.252 5355 udp dns 0.412434 44 0 S0 F F 0 D 2 100 0 0 (empty) Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox From chris at cwalsh.org Sat Apr 16 13:31:13 2016 From: chris at cwalsh.org (Chris Walsh) Date: Sat, 16 Apr 2016 15:31:13 -0500 Subject: [Bro] How to parse bro decimal timestamps? In-Reply-To: References: Message-ID: <796F7815-804D-4A0C-8818-B36741EB84FE@cwalsh.org> Depends on what you?re reading the logs with. You could use bro-cut with the ?-d? flag, to do the conversion for you. If you just need to do one-off date conversion: Using GNU date (takes date as is): $ date --date='@1459774793.429104? Mon Apr 4 12:59:53 UTC 2016 OSX (wants the date as an integer) $ foobar=`echo 1459774793.429104 | cut -d. -f1` $ date -r $foobar Mon Apr 4 07:59:53 CDT 2016 If you?re snarfing the timestamps into your own code, then it depends on what language/libraries you?re using. > On Apr 16, 2016, at 3:05 PM, Brad Cox wrote: > > How do I turn the timestamp (ts) field in this example into a standard date format (java or unix dates for example?) > > set_separator , > #empty_field (empty) > #unset_field - > #path conn > #open 2016-04-04-09-00-04 > #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig local_resp missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents > #types time string addr port addr port enum string interval count count string bool bool count string count count count count set[string] > 1459774793.429104 CZgDTe31Z6ynNuzgN7 fe80::c874:93f:5b4e:c1e1 64648 ff02::1:3 5355 udp dns 0.412428 44 0 S0 F F 0 D 2 140 0 0 (empty) > 1459774793.429113 Ci77TT3Kp4dNmhAYc1 172.16.2.33 64648 224.0.0.252 5355 udp dns 0.412434 44 0 S0 F F 0 D 2 100 0 0 (empty) > From bradjcox at gmail.com Sat Apr 16 13:44:06 2016 From: bradjcox at gmail.com (Brad Cox) Date: Sat, 16 Apr 2016 16:44:06 -0400 Subject: [Bro] How to parse bro decimal timestamps? In-Reply-To: <796F7815-804D-4A0C-8818-B36741EB84FE@cwalsh.org> References: <796F7815-804D-4A0C-8818-B36741EB84FE@cwalsh.org> Message-ID: Need to parse dates in java; using this in a spark streaming analytics pipeline. Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox > On Apr 16, 2016, at 4:31 PM, Chris Walsh wrote: > > Depends on what you?re reading the logs with. > > You could use bro-cut with the ?-d? flag, to do the conversion for you. > > If you just need to do one-off date conversion: > > Using GNU date (takes date as is): > > $ date --date='@1459774793.429104? > Mon Apr 4 12:59:53 UTC 2016 > > OSX (wants the date as an integer) > > $ foobar=`echo 1459774793.429104 | cut -d. -f1` > $ date -r $foobar > Mon Apr 4 07:59:53 CDT 2016 > > > If you?re snarfing the timestamps into your own code, then it depends on what language/libraries you?re using. > > > >> On Apr 16, 2016, at 3:05 PM, Brad Cox wrote: >> >> How do I turn the timestamp (ts) field in this example into a standard date format (java or unix dates for example?) >> >> set_separator , >> #empty_field (empty) >> #unset_field - >> #path conn >> #open 2016-04-04-09-00-04 >> #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig local_resp missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents >> #types time string addr port addr port enum string interval count count string bool bool count string count count count count set[string] >> 1459774793.429104 CZgDTe31Z6ynNuzgN7 fe80::c874:93f:5b4e:c1e1 64648 ff02::1:3 5355 udp dns 0.412428 44 0 S0 F F 0 D 2 140 0 0 (empty) >> 1459774793.429113 Ci77TT3Kp4dNmhAYc1 172.16.2.33 64648 224.0.0.252 5355 udp dns 0.412434 44 0 S0 F F 0 D 2 100 0 0 (empty) >> > From bradjcox at gmail.com Sat Apr 16 15:25:13 2016 From: bradjcox at gmail.com (Brad Cox) Date: Sat, 16 Apr 2016 18:25:13 -0400 Subject: [Bro] How to parse bro decimal timestamps? In-Reply-To: References: <796F7815-804D-4A0C-8818-B36741EB84FE@cwalsh.org> Message-ID: Java code would be nice, but a ordinary description of how a decimal date relates to standard dates would do. I'm familiar with Java/Unix conventions where a long integer specifies seconds since the Unix epoch (Jan 1970). But I've tried converting the bro decimal to long and converting that to a date. That gives a date sometime in 1970 which clearly isn't right. And what do the fractional values mean? Milliseconds perhaps? Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox > On Apr 16, 2016, at 4:44 PM, Brad Cox wrote: > > Need to parse dates in java; using this in a spark streaming analytics pipeline. > > Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox > > > > >> On Apr 16, 2016, at 4:31 PM, Chris Walsh wrote: >> >> Depends on what you?re reading the logs with. >> >> You could use bro-cut with the ?-d? flag, to do the conversion for you. >> >> If you just need to do one-off date conversion: >> >> Using GNU date (takes date as is): >> >> $ date --date='@1459774793.429104? >> Mon Apr 4 12:59:53 UTC 2016 >> >> OSX (wants the date as an integer) >> >> $ foobar=`echo 1459774793.429104 | cut -d. -f1` >> $ date -r $foobar >> Mon Apr 4 07:59:53 CDT 2016 >> >> >> If you?re snarfing the timestamps into your own code, then it depends on what language/libraries you?re using. >> >> >> >>> On Apr 16, 2016, at 3:05 PM, Brad Cox wrote: >>> >>> How do I turn the timestamp (ts) field in this example into a standard date format (java or unix dates for example?) >>> >>> set_separator , >>> #empty_field (empty) >>> #unset_field - >>> #path conn >>> #open 2016-04-04-09-00-04 >>> #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig local_resp missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents >>> #types time string addr port addr port enum string interval count count string bool bool count string count count count count set[string] >>> 1459774793.429104 CZgDTe31Z6ynNuzgN7 fe80::c874:93f:5b4e:c1e1 64648 ff02::1:3 5355 udp dns 0.412428 44 0 S0 F F 0 D 2 140 0 0 (empty) >>> 1459774793.429113 Ci77TT3Kp4dNmhAYc1 172.16.2.33 64648 224.0.0.252 5355 udp dns 0.412434 44 0 S0 F F 0 D 2 100 0 0 (empty) >>> >> > From chris at cwalsh.org Sat Apr 16 15:43:05 2016 From: chris at cwalsh.org (Chris Walsh) Date: Sat, 16 Apr 2016 17:43:05 -0500 Subject: [Bro] How to parse bro decimal timestamps? In-Reply-To: References: <796F7815-804D-4A0C-8818-B36741EB84FE@cwalsh.org> Message-ID: <2180E3D0-8819-4AA1-9D06-5732166053F8@cwalsh.org> The value is the number of seconds since the epoch, including a fractional portion. Your problem with Java dealing with it is likely that Java wants the number of *milliseconds* since the epoch, so you should multiply what Bro gives you by 1000 (and dropping the fractional part) before converting it with whatever Java code you?re working with. Chris > On Apr 16, 2016, at 5:25 PM, Brad Cox wrote: > > Java code would be nice, but a ordinary description of how a decimal date relates to standard dates would do. I'm familiar with Java/Unix conventions where a long integer specifies seconds since the Unix epoch (Jan 1970). But I've tried converting the bro decimal to long and converting that to a date. That gives a date sometime in 1970 which clearly isn't right. And what do the fractional values mean? Milliseconds perhaps? > > Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox > > > > >> On Apr 16, 2016, at 4:44 PM, Brad Cox wrote: >> >> Need to parse dates in java; using this in a spark streaming analytics pipeline. >> >> Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox >> >> >> >> >>> On Apr 16, 2016, at 4:31 PM, Chris Walsh wrote: >>> >>> Depends on what you?re reading the logs with. >>> >>> You could use bro-cut with the ?-d? flag, to do the conversion for you. >>> >>> If you just need to do one-off date conversion: >>> >>> Using GNU date (takes date as is): >>> >>> $ date --date='@1459774793.429104? >>> Mon Apr 4 12:59:53 UTC 2016 >>> >>> OSX (wants the date as an integer) >>> >>> $ foobar=`echo 1459774793.429104 | cut -d. -f1` >>> $ date -r $foobar >>> Mon Apr 4 07:59:53 CDT 2016 >>> >>> >>> If you?re snarfing the timestamps into your own code, then it depends on what language/libraries you?re using. >>> >>> >>> >>>> On Apr 16, 2016, at 3:05 PM, Brad Cox wrote: >>>> >>>> How do I turn the timestamp (ts) field in this example into a standard date format (java or unix dates for example?) >>>> >>>> set_separator , >>>> #empty_field (empty) >>>> #unset_field - >>>> #path conn >>>> #open 2016-04-04-09-00-04 >>>> #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig local_resp missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents >>>> #types time string addr port addr port enum string interval count count string bool bool count string count count count count set[string] >>>> 1459774793.429104 CZgDTe31Z6ynNuzgN7 fe80::c874:93f:5b4e:c1e1 64648 ff02::1:3 5355 udp dns 0.412428 44 0 S0 F F 0 D 2 140 0 0 (empty) >>>> 1459774793.429113 Ci77TT3Kp4dNmhAYc1 172.16.2.33 64648 224.0.0.252 5355 udp dns 0.412434 44 0 S0 F F 0 D 2 100 0 0 (empty) >>>> >>> >> > From bradjcox at gmail.com Sat Apr 16 16:09:57 2016 From: bradjcox at gmail.com (Brad Cox) Date: Sat, 16 Apr 2016 19:09:57 -0400 Subject: [Bro] How to parse bro decimal timestamps? In-Reply-To: <2180E3D0-8819-4AA1-9D06-5732166053F8@cwalsh.org> References: <796F7815-804D-4A0C-8818-B36741EB84FE@cwalsh.org> <2180E3D0-8819-4AA1-9D06-5732166053F8@cwalsh.org> Message-ID: <64000B73-3099-4782-ADA1-EDEF5E7BDE0B@gmail.com> Perfect. Thanks so much! Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox > On Apr 16, 2016, at 6:43 PM, Chris Walsh wrote: > > The value is the number of seconds since the epoch, including a fractional portion. Your problem with Java dealing with it is likely that Java wants the number of *milliseconds* since the epoch, so you should multiply what Bro gives you by 1000 (and dropping the fractional part) before converting it with whatever Java code you?re working with. > > > Chris > > >> On Apr 16, 2016, at 5:25 PM, Brad Cox wrote: >> >> Java code would be nice, but a ordinary description of how a decimal date relates to standard dates would do. I'm familiar with Java/Unix conventions where a long integer specifies seconds since the Unix epoch (Jan 1970). But I've tried converting the bro decimal to long and converting that to a date. That gives a date sometime in 1970 which clearly isn't right. And what do the fractional values mean? Milliseconds perhaps? >> >> Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox >> >> >> >> >>> On Apr 16, 2016, at 4:44 PM, Brad Cox wrote: >>> >>> Need to parse dates in java; using this in a spark streaming analytics pipeline. >>> >>> Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox >>> >>> >>> >>> >>>> On Apr 16, 2016, at 4:31 PM, Chris Walsh wrote: >>>> >>>> Depends on what you?re reading the logs with. >>>> >>>> You could use bro-cut with the ?-d? flag, to do the conversion for you. >>>> >>>> If you just need to do one-off date conversion: >>>> >>>> Using GNU date (takes date as is): >>>> >>>> $ date --date='@1459774793.429104? >>>> Mon Apr 4 12:59:53 UTC 2016 >>>> >>>> OSX (wants the date as an integer) >>>> >>>> $ foobar=`echo 1459774793.429104 | cut -d. -f1` >>>> $ date -r $foobar >>>> Mon Apr 4 07:59:53 CDT 2016 >>>> >>>> >>>> If you?re snarfing the timestamps into your own code, then it depends on what language/libraries you?re using. >>>> >>>> >>>> >>>>> On Apr 16, 2016, at 3:05 PM, Brad Cox wrote: >>>>> >>>>> How do I turn the timestamp (ts) field in this example into a standard date format (java or unix dates for example?) >>>>> >>>>> set_separator , >>>>> #empty_field (empty) >>>>> #unset_field - >>>>> #path conn >>>>> #open 2016-04-04-09-00-04 >>>>> #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig local_resp missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents >>>>> #types time string addr port addr port enum string interval count count string bool bool count string count count count count set[string] >>>>> 1459774793.429104 CZgDTe31Z6ynNuzgN7 fe80::c874:93f:5b4e:c1e1 64648 ff02::1:3 5355 udp dns 0.412428 44 0 S0 F F 0 D 2 140 0 0 (empty) >>>>> 1459774793.429113 Ci77TT3Kp4dNmhAYc1 172.16.2.33 64648 224.0.0.252 5355 udp dns 0.412434 44 0 S0 F F 0 D 2 100 0 0 (empty) >>>>> >>>> >>> >> > From tgdesrochers at gmail.com Sun Apr 17 04:19:01 2016 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Sun, 17 Apr 2016 07:19:01 -0400 Subject: [Bro] [bro] misp2bro Message-ID: Anyone using MISP? I installed MISP as a test and it seems pretty useful. What I can't seem to get working is the misp2bro script written to export indicators in MISP to bro format. https://github.com/unusedPhD/misp2bro When I run the script it appears to crash and give the error: Traceback (most recent call last): File "misp2bro.py", line 288, in if makeBroFiles(parseXML(EXPORT_FILE)): File "misp2bro.py", line 168, in makeBroFiles if int(event.find('attribute_count').text): AttributeError: 'NoneType' object has no attribute 'text' If I run it again there is no crash but that is because the md5 it generates matches the previous hash so no action is taken on the downloaded xml. Has anyone used this, I could use a hand getting it working. Thanks Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160417/a5bc8409/attachment.html From elhoim at gmail.com Sun Apr 17 07:28:42 2016 From: elhoim at gmail.com (=?UTF-8?B?RGF2aWQgQW5kcsOp?=) Date: Sun, 17 Apr 2016 09:28:42 -0500 Subject: [Bro] [bro] misp2bro In-Reply-To: References: Message-ID: Is there an error message in the xml file? If yes, could you post it? If you want to write your own script to download IOCs, there is the PyMISP library @ https://github.com/MISP/PyMISP/ This library is really great because it abstracts most of the details needed to create a script for interacting with a MISP instance. Then you can just grep your bro logs, or generate bro IOCs lists that can be used to match. On Sun, Apr 17, 2016 at 6:19 AM, Tim Desrochers wrote: > Anyone using MISP? I installed MISP as a test and it seems pretty useful. > What I can't seem to get working is the misp2bro script written to export > indicators in MISP to bro format. > > https://github.com/unusedPhD/misp2bro > > When I run the script it appears to crash and give the error: > Traceback (most recent call last): > File "misp2bro.py", line 288, in > if makeBroFiles(parseXML(EXPORT_FILE)): > File "misp2bro.py", line 168, in makeBroFiles > if int(event.find('attribute_count').text): > AttributeError: 'NoneType' object has no attribute 'text' > > If I run it again there is no crash but that is because the md5 it generates > matches the previous hash so no action is taken on the downloaded xml. > > Has anyone used this, I could use a hand getting it working. > > Thanks > Tim > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From elhoim at gmail.com Sun Apr 17 07:50:28 2016 From: elhoim at gmail.com (=?UTF-8?B?RGF2aWQgQW5kcsOp?=) Date: Sun, 17 Apr 2016 09:50:28 -0500 Subject: [Bro] [bro] misp2bro In-Reply-To: References: Message-ID: Then it is probably the fact that the misp2bro script is exepcting an old format of XML from MISP. Given the date of creation of the script, I would say it expects v2.3 file format, while you are probably using a v2.4 MISP. On Sun, Apr 17, 2016 at 9:35 AM, Tim Desrochers wrote: > I've attached the error log and the xml. I don't see any issues with the > XML and the error log just shows that it started processing the XML. The > script errors out after beginning to process the XML so thats why I assume > there are no more entries in the log. > > Thanks for the pointer to the PyMISP. I will look into it > > > > On Sun, Apr 17, 2016 at 10:28 AM, David Andr? wrote: >> >> Is there an error message in the xml file? >> If yes, could you post it? >> >> If you want to write your own script to download IOCs, there is the >> PyMISP library @ https://github.com/MISP/PyMISP/ >> This library is really great because it abstracts most of the details >> needed to create a script for interacting with a MISP instance. >> >> Then you can just grep your bro logs, or generate bro IOCs lists that >> can be used to match. >> >> On Sun, Apr 17, 2016 at 6:19 AM, Tim Desrochers >> wrote: >> > Anyone using MISP? I installed MISP as a test and it seems pretty >> > useful. >> > What I can't seem to get working is the misp2bro script written to >> > export >> > indicators in MISP to bro format. >> > >> > https://github.com/unusedPhD/misp2bro >> > >> > When I run the script it appears to crash and give the error: >> > Traceback (most recent call last): >> > File "misp2bro.py", line 288, in >> > if makeBroFiles(parseXML(EXPORT_FILE)): >> > File "misp2bro.py", line 168, in makeBroFiles >> > if int(event.find('attribute_count').text): >> > AttributeError: 'NoneType' object has no attribute 'text' >> > >> > If I run it again there is no crash but that is because the md5 it >> > generates >> > matches the previous hash so no action is taken on the downloaded xml. >> > >> > Has anyone used this, I could use a hand getting it working. >> > >> > Thanks >> > Tim >> > >> > _______________________________________________ >> > Bro mailing list >> > bro at bro-ids.org >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > From tgdesrochers at gmail.com Sun Apr 17 08:15:51 2016 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Sun, 17 Apr 2016 11:15:51 -0400 Subject: [Bro] [bro] misp2bro In-Reply-To: References: Message-ID: Thanks, I'll try to hit up the creator again and see if it can be fixed On Sun, Apr 17, 2016 at 10:50 AM, David Andr? wrote: > Then it is probably the fact that the misp2bro script is exepcting an > old format of XML from MISP. > Given the date of creation of the script, I would say it expects v2.3 > file format, while you are probably using a v2.4 MISP. > > On Sun, Apr 17, 2016 at 9:35 AM, Tim Desrochers > wrote: > > I've attached the error log and the xml. I don't see any issues with the > > XML and the error log just shows that it started processing the XML. The > > script errors out after beginning to process the XML so thats why I > assume > > there are no more entries in the log. > > > > Thanks for the pointer to the PyMISP. I will look into it > > > > > > > > On Sun, Apr 17, 2016 at 10:28 AM, David Andr? wrote: > >> > >> Is there an error message in the xml file? > >> If yes, could you post it? > >> > >> If you want to write your own script to download IOCs, there is the > >> PyMISP library @ https://github.com/MISP/PyMISP/ > >> This library is really great because it abstracts most of the details > >> needed to create a script for interacting with a MISP instance. > >> > >> Then you can just grep your bro logs, or generate bro IOCs lists that > >> can be used to match. > >> > >> On Sun, Apr 17, 2016 at 6:19 AM, Tim Desrochers > > >> wrote: > >> > Anyone using MISP? I installed MISP as a test and it seems pretty > >> > useful. > >> > What I can't seem to get working is the misp2bro script written to > >> > export > >> > indicators in MISP to bro format. > >> > > >> > https://github.com/unusedPhD/misp2bro > >> > > >> > When I run the script it appears to crash and give the error: > >> > Traceback (most recent call last): > >> > File "misp2bro.py", line 288, in > >> > if makeBroFiles(parseXML(EXPORT_FILE)): > >> > File "misp2bro.py", line 168, in makeBroFiles > >> > if int(event.find('attribute_count').text): > >> > AttributeError: 'NoneType' object has no attribute 'text' > >> > > >> > If I run it again there is no crash but that is because the md5 it > >> > generates > >> > matches the previous hash so no action is taken on the downloaded xml. > >> > > >> > Has anyone used this, I could use a hand getting it working. > >> > > >> > Thanks > >> > Tim > >> > > >> > _______________________________________________ > >> > Bro mailing list > >> > bro at bro-ids.org > >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160417/111e1a26/attachment.html From elhoim at gmail.com Sun Apr 17 08:52:48 2016 From: elhoim at gmail.com (=?UTF-8?B?RGF2aWQgQW5kcsOp?=) Date: Sun, 17 Apr 2016 10:52:48 -0500 Subject: [Bro] [bro] misp2bro In-Reply-To: References: Message-ID: I took a peak, there is no field "attribute_count" under event in XML. Commenting out line #168 to stop checking for that, and it works for me. To get it working, I also had to manually create the tmp directory since it is used to indicate the filename for logs, but if it does not exists, it is created after the logger object creation which fails because it does not exist. :( I created a pull request for it: https://github.com/thnyheim/misp2bro/pull/1 On Sun, Apr 17, 2016 at 10:15 AM, Tim Desrochers wrote: > Thanks, I'll try to hit up the creator again and see if it can be fixed > > On Sun, Apr 17, 2016 at 10:50 AM, David Andr? wrote: >> >> Then it is probably the fact that the misp2bro script is exepcting an >> old format of XML from MISP. >> Given the date of creation of the script, I would say it expects v2.3 >> file format, while you are probably using a v2.4 MISP. >> >> On Sun, Apr 17, 2016 at 9:35 AM, Tim Desrochers >> wrote: >> > I've attached the error log and the xml. I don't see any issues with >> > the >> > XML and the error log just shows that it started processing the XML. >> > The >> > script errors out after beginning to process the XML so thats why I >> > assume >> > there are no more entries in the log. >> > >> > Thanks for the pointer to the PyMISP. I will look into it >> > >> > >> > >> > On Sun, Apr 17, 2016 at 10:28 AM, David Andr? wrote: >> >> >> >> Is there an error message in the xml file? >> >> If yes, could you post it? >> >> >> >> If you want to write your own script to download IOCs, there is the >> >> PyMISP library @ https://github.com/MISP/PyMISP/ >> >> This library is really great because it abstracts most of the details >> >> needed to create a script for interacting with a MISP instance. >> >> >> >> Then you can just grep your bro logs, or generate bro IOCs lists that >> >> can be used to match. >> >> >> >> On Sun, Apr 17, 2016 at 6:19 AM, Tim Desrochers >> >> >> >> wrote: >> >> > Anyone using MISP? I installed MISP as a test and it seems pretty >> >> > useful. >> >> > What I can't seem to get working is the misp2bro script written to >> >> > export >> >> > indicators in MISP to bro format. >> >> > >> >> > https://github.com/unusedPhD/misp2bro >> >> > >> >> > When I run the script it appears to crash and give the error: >> >> > Traceback (most recent call last): >> >> > File "misp2bro.py", line 288, in >> >> > if makeBroFiles(parseXML(EXPORT_FILE)): >> >> > File "misp2bro.py", line 168, in makeBroFiles >> >> > if int(event.find('attribute_count').text): >> >> > AttributeError: 'NoneType' object has no attribute 'text' >> >> > >> >> > If I run it again there is no crash but that is because the md5 it >> >> > generates >> >> > matches the previous hash so no action is taken on the downloaded >> >> > xml. >> >> > >> >> > Has anyone used this, I could use a hand getting it working. >> >> > >> >> > Thanks >> >> > Tim >> >> > >> >> > _______________________________________________ >> >> > Bro mailing list >> >> > bro at bro-ids.org >> >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > >> > > > From jlamps at sandia.gov Mon Apr 18 06:48:04 2016 From: jlamps at sandia.gov (Lamps, Jereme) Date: Mon, 18 Apr 2016 13:48:04 +0000 Subject: [Bro] Logging at end of connections Message-ID: I would like to take some action on a connection before it is written to conn.log. I added some code to the event Conn::log_conn and it works as intended when running through pcaps. However, when I try to run the script live on a network interface, it appears that log_conn is not getting called until I hit ctrl-c.. Is there another event I need to use?? Thanks in advance, Jereme -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160418/c3dee841/attachment.html From chegov at gmail.com Mon Apr 18 07:59:55 2016 From: chegov at gmail.com (Shiran Aizik) Date: Mon, 18 Apr 2016 17:59:55 +0300 Subject: [Bro] Updating table in SumStats - not recognized in outer event Message-ID: Hello, I have a global synced table which I update on threshold_crossed in SumStats. The update is successful, yet when I try to access this table from a different event (dns query reply) it is empty. I work wih bro 2.4.1 in cluster mode. The table is defined as global, synced and mergable. My goal is that all workers will send their observations to SumStats and based on a threshold, update a table for all the workers to know. Any ideas what can I try ? Thank you Chegov -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160418/f5135eda/attachment.html From hhoffman at ip-solutions.net Mon Apr 18 09:51:59 2016 From: hhoffman at ip-solutions.net (Harry Hoffman) Date: Mon, 18 Apr 2016 09:51:59 -0700 Subject: [Bro] Git checkout OSX Python 3.x Message-ID: <571510AF.2020203@ip-solutions.net> Hi Folks, Decided to have a go installing Bro on a OS X laptop. It seems as maybe there's a problem with later versions of Python. Anyone run into this? Cheers, Harry canning dependencies of target _pybroker [ 8%] Building CXX object aux/broker/bindings/python/CMakeFiles/_pybroker.dir/pybrokerPYTHON_wrap.cxx.o /var/tmp/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:39657:20: error: use of undeclared identifier 'PyInt_AsSsize_t'; did you mean 'PyLong_AsSsize_t'? arg4 = (void*)(PyInt_AsSsize_t(obj3)); ^~~~~~~~~~~~~~~ PyLong_AsSsize_t /opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m/longobject.h:25:24: note: 'PyLong_AsSsize_t' declared here PyAPI_FUNC(Py_ssize_t) PyLong_AsSsize_t(PyObject *); ^ /var/tmp/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:39846:20: error: use of undeclared identifier 'PyInt_AsSsize_t'; did you mean 'PyLong_AsSsize_t'? arg4 = (void*)(PyInt_AsSsize_t(obj3)); ^~~~~~~~~~~~~~~ PyLong_AsSsize_t /opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m/longobject.h:25:24: note: 'PyLong_AsSsize_t' declared here PyAPI_FUNC(Py_ssize_t) PyLong_AsSsize_t(PyObject *); ^ /var/tmp/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:40035:20: error: use of undeclared identifier 'PyInt_AsSsize_t'; did you mean 'PyLong_AsSsize_t'? arg4 = (void*)(PyInt_AsSsize_t(obj3)); ^~~~~~~~~~~~~~~ PyLong_AsSsize_t /opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m/longobject.h:25:24: note: 'PyLong_AsSsize_t' declared here PyAPI_FUNC(Py_ssize_t) PyLong_AsSsize_t(PyObject *); ^ /var/tmp/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:40224:20: error: use of undeclared identifier 'PyInt_AsSsize_t'; did you mean 'PyLong_AsSsize_t'? arg4 = (void*)(PyInt_AsSsize_t(obj3)); ^~~~~~~~~~~~~~~ PyLong_AsSsize_t /opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m/longobject.h:25:24: note: 'PyLong_AsSsize_t' declared here PyAPI_FUNC(Py_ssize_t) PyLong_AsSsize_t(PyObject *); ^ /var/tmp/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:40413:20: error: use of undeclared identifier 'PyInt_AsSsize_t'; did you mean 'PyLong_AsSsize_t'? arg4 = (void*)(PyInt_AsSsize_t(obj3)); ^~~~~~~~~~~~~~~ PyLong_AsSsize_t /opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m/longobject.h:25:24: note: 'PyLong_AsSsize_t' declared here PyAPI_FUNC(Py_ssize_t) PyLong_AsSsize_t(PyObject *); ^ /var/tmp/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:40585:20: error: use of undeclared identifier 'PyInt_AsSsize_t'; did you mean 'PyLong_AsSsize_t'? arg3 = (void*)(PyInt_AsSsize_t(obj2)); ^~~~~~~~~~~~~~~ PyLong_AsSsize_t /opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m/longobject.h:25:24: note: 'PyLong_AsSsize_t' declared here PyAPI_FUNC(Py_ssize_t) PyLong_AsSsize_t(PyObject *); ^ /var/tmp/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:40728:20: error: use of undeclared identifier 'PyInt_AsSsize_t'; did you mean 'PyLong_AsSsize_t'? arg3 = (void*)(PyInt_AsSsize_t(obj2)); ^~~~~~~~~~~~~~~ PyLong_AsSsize_t /opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m/longobject.h:25:24: note: 'PyLong_AsSsize_t' declared here PyAPI_FUNC(Py_ssize_t) PyLong_AsSsize_t(PyObject *); ^ 7 errors generated. make[3]: *** [aux/broker/bindings/python/CMakeFiles/_pybroker.dir/pybrokerPYTHON_wrap.cxx.o] Error 1 make[2]: *** [aux/broker/bindings/python/CMakeFiles/_pybroker.dir/all] Error 2 make[1]: *** [all] Error 2 make: *** [all] Error 2 From johanna at icir.org Mon Apr 18 10:41:52 2016 From: johanna at icir.org (Johanna Amann) Date: Mon, 18 Apr 2016 10:41:52 -0700 Subject: [Bro] Git checkout OSX Python 3.x In-Reply-To: <571510AF.2020203@ip-solutions.net> References: <571510AF.2020203@ip-solutions.net> Message-ID: <560D7DBE-6CD6-4CEA-8077-EB4CCEFFA897@icir.org> Yes, there currently is a known issue about Broker not compiling with python 3. For the moment, you can either compile against python 2.x or use --disable-broker; I hope we will fix this soon. Johanna On 18 Apr 2016, at 9:51, Harry Hoffman wrote: > Hi Folks, > > Decided to have a go installing Bro on a OS X laptop. It seems as > maybe > there's a problem with later versions of Python. Anyone run into this? > > Cheers, > Harry > > > canning dependencies of target _pybroker > [ 8%] Building CXX object > aux/broker/bindings/python/CMakeFiles/_pybroker.dir/pybrokerPYTHON_wrap.cxx.o > /var/tmp/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:39657:20: > error: > use of undeclared identifier 'PyInt_AsSsize_t'; did you mean > 'PyLong_AsSsize_t'? > arg4 = (void*)(PyInt_AsSsize_t(obj3)); > ^~~~~~~~~~~~~~~ > PyLong_AsSsize_t > /opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m/longobject.h:25:24: > note: > 'PyLong_AsSsize_t' declared here > PyAPI_FUNC(Py_ssize_t) PyLong_AsSsize_t(PyObject *); > ^ > /var/tmp/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:39846:20: > error: > use of undeclared identifier 'PyInt_AsSsize_t'; did you mean > 'PyLong_AsSsize_t'? > arg4 = (void*)(PyInt_AsSsize_t(obj3)); > ^~~~~~~~~~~~~~~ > PyLong_AsSsize_t > /opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m/longobject.h:25:24: > note: > 'PyLong_AsSsize_t' declared here > PyAPI_FUNC(Py_ssize_t) PyLong_AsSsize_t(PyObject *); > ^ > /var/tmp/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:40035:20: > error: > use of undeclared identifier 'PyInt_AsSsize_t'; did you mean > 'PyLong_AsSsize_t'? > arg4 = (void*)(PyInt_AsSsize_t(obj3)); > ^~~~~~~~~~~~~~~ > PyLong_AsSsize_t > /opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m/longobject.h:25:24: > note: > 'PyLong_AsSsize_t' declared here > PyAPI_FUNC(Py_ssize_t) PyLong_AsSsize_t(PyObject *); > ^ > /var/tmp/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:40224:20: > error: > use of undeclared identifier 'PyInt_AsSsize_t'; did you mean > 'PyLong_AsSsize_t'? > arg4 = (void*)(PyInt_AsSsize_t(obj3)); > ^~~~~~~~~~~~~~~ > PyLong_AsSsize_t > /opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m/longobject.h:25:24: > note: > 'PyLong_AsSsize_t' declared here > PyAPI_FUNC(Py_ssize_t) PyLong_AsSsize_t(PyObject *); > ^ > /var/tmp/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:40413:20: > error: > use of undeclared identifier 'PyInt_AsSsize_t'; did you mean > 'PyLong_AsSsize_t'? > arg4 = (void*)(PyInt_AsSsize_t(obj3)); > ^~~~~~~~~~~~~~~ > PyLong_AsSsize_t > /opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m/longobject.h:25:24: > note: > 'PyLong_AsSsize_t' declared here > PyAPI_FUNC(Py_ssize_t) PyLong_AsSsize_t(PyObject *); > ^ > /var/tmp/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:40585:20: > error: > use of undeclared identifier 'PyInt_AsSsize_t'; did you mean > 'PyLong_AsSsize_t'? > arg3 = (void*)(PyInt_AsSsize_t(obj2)); > ^~~~~~~~~~~~~~~ > PyLong_AsSsize_t > /opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m/longobject.h:25:24: > note: > 'PyLong_AsSsize_t' declared here > PyAPI_FUNC(Py_ssize_t) PyLong_AsSsize_t(PyObject *); > ^ > /var/tmp/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:40728:20: > error: > use of undeclared identifier 'PyInt_AsSsize_t'; did you mean > 'PyLong_AsSsize_t'? > arg3 = (void*)(PyInt_AsSsize_t(obj2)); > ^~~~~~~~~~~~~~~ > PyLong_AsSsize_t > /opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m/longobject.h:25:24: > note: > 'PyLong_AsSsize_t' declared here > PyAPI_FUNC(Py_ssize_t) PyLong_AsSsize_t(PyObject *); > ^ > 7 errors generated. > make[3]: *** > [aux/broker/bindings/python/CMakeFiles/_pybroker.dir/pybrokerPYTHON_wrap.cxx.o] > Error 1 > make[2]: *** [aux/broker/bindings/python/CMakeFiles/_pybroker.dir/all] > Error 2 > make[1]: *** [all] Error 2 > make: *** [all] Error 2 > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From franky.meier.1 at gmx.de Mon Apr 18 23:29:54 2016 From: franky.meier.1 at gmx.de (Frank Meier) Date: Tue, 19 Apr 2016 08:29:54 +0200 Subject: [Bro] Git checkout OSX Python 3.x In-Reply-To: <571510AF.2020203@ip-solutions.net> References: <571510AF.2020203@ip-solutions.net> Message-ID: <20160419082954.64727a20@NB181106> Hi Harry, On Mon, 18 Apr 2016 09:51:59 -0700 Harry Hoffman wrote: > Decided to have a go installing Bro on a OS X laptop. It seems as > maybe there's a problem with later versions of Python. Anyone run > into this? > > > canning dependencies of target _pybroker > [ 8%] Building CXX object > aux/broker/bindings/python/CMakeFiles/_pybroker.dir/pybrokerPYTHON_wrap.cxx.o > /var/tmp/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:39657:20: > error: > use of undeclared identifier 'PyInt_AsSsize_t'; did you mean > 'PyLong_AsSsize_t'? > arg4 = (void*)(PyInt_AsSsize_t(obj3)); > ^~~~~~~~~~~~~~~ > PyLong_AsSsize_t you could try my solution from BIT-528, but as noted there it causes some warning. Franky From jlamps at sandia.gov Tue Apr 19 08:55:17 2016 From: jlamps at sandia.gov (Lamps, Jereme) Date: Tue, 19 Apr 2016 15:55:17 +0000 Subject: [Bro] ConnPolling::watch not working in cluster mode Message-ID: Hey guys, Would anyone know why ConnPolling::watch does not seem to be working when I start it up with broctl (in cluster mode). When I start it up with just bro (standalone) it works fine. The line of code is: ConnPolling::watch(s$conn, is_active, 0, 0sec); I have a log statement right before this line which I am seeing, and I have a log statement as the first instruction in the is_active function that I am NOT seeing. Thanks in advance, Jereme -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160419/3239e52a/attachment.html From hosom at battelle.org Tue Apr 19 11:13:04 2016 From: hosom at battelle.org (Hosom, Stephen M) Date: Tue, 19 Apr 2016 18:13:04 +0000 Subject: [Bro] ConnPolling::watch not working in cluster mode References: Message-ID: For what it's worth, these questions are usually significantly easier to answer if you provide the code that isn't working. On 04/19/2016 12:04 PM, Lamps, Jereme wrote: Hey guys, Would anyone know why ConnPolling::watch does not seem to be working when I start it up with broctl (in cluster mode). When I start it up with just bro (standalone) it works fine. The line of code is: ConnPolling::watch(s$conn, is_active, 0, 0sec); I have a log statement right before this line which I am seeing, and I have a log statement as the first instruction in the is_active function that I am NOT seeing. Thanks in advance, Jereme From heiland at iu.edu Tue Apr 19 13:32:02 2016 From: heiland at iu.edu (Heiland, Randy) Date: Tue, 19 Apr 2016 20:32:02 +0000 Subject: [Bro] build error on OSX ElC; OpenSSL Message-ID: Hi, Trying to build Bro on OSX (10.11), I get an error about not being able to find my OpenSSL. Any ideas? Before attempting the config, I set the following env vars: OPENSSL_INCLUDE_DIR=/usr/local/ssl/include OPENSSL_LIBRARIES=/usr/local/ssl/lib OPENSSL_ROOT_DIR=/usr/local/ssl ~/git/bro$ ./configure --with-openssl=/usr/local/ssl Build Directory : build Source Directory: /Users/heiland/git/bro -- Found sed: /usr/local/bin/sed -- Found PythonInterp: /Users/heiland/anaconda3/bin/python (found version "3.5.1") -- Found FLEX: /usr/bin/flex (found version "2.5.35") -- Found BISON: /usr/local/bin/bison -- Found PCAP: /usr/lib/libpcap.dylib -- Performing Test PCAP_LINKS_SOLO -- Performing Test PCAP_LINKS_SOLO - Success -- Looking for pcap_get_pfring_id -- Looking for pcap_get_pfring_id - not found -- Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the system variable OPENSSL_ROOT_DIR (missing: OPENSSL_INCLUDE_DIR) ... CMake Error at aux/binpac/CMakeLists.txt:17 (message): Could not find prerequisite package 'OpenSSL' CMake Error at aux/binpac/CMakeLists.txt:19 (message): Configuration aborted due to missing prerequisites thanks, Randy p.s. what?s recommended to search your mail archive - gmane? From slagell at illinois.edu Tue Apr 19 13:37:00 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Tue, 19 Apr 2016 20:37:00 +0000 Subject: [Bro] build error on OSX ElC; OpenSSL In-Reply-To: References: Message-ID: <8B770ECD-699F-42BD-AA71-2BE912A44324@illinois.edu> Apple deprecated openssl. https://www.bro.org/documentation/faq.html#why-does-v2-4-fail-to-build-on-mac-os-x-10-11 > On Apr 19, 2016, at 3:32 PM, Heiland, Randy wrote: > > Hi, > > Trying to build Bro on OSX (10.11), I get an error about not being able to find my OpenSSL. Any ideas? > > Before attempting the config, I set the following env vars: > OPENSSL_INCLUDE_DIR=/usr/local/ssl/include > OPENSSL_LIBRARIES=/usr/local/ssl/lib > OPENSSL_ROOT_DIR=/usr/local/ssl > > > ~/git/bro$ ./configure --with-openssl=/usr/local/ssl > Build Directory : build > Source Directory: /Users/heiland/git/bro > -- Found sed: /usr/local/bin/sed > -- Found PythonInterp: /Users/heiland/anaconda3/bin/python (found version "3.5.1") > -- Found FLEX: /usr/bin/flex (found version "2.5.35") > -- Found BISON: /usr/local/bin/bison > -- Found PCAP: /usr/lib/libpcap.dylib > -- Performing Test PCAP_LINKS_SOLO > -- Performing Test PCAP_LINKS_SOLO - Success > -- Looking for pcap_get_pfring_id > -- Looking for pcap_get_pfring_id - not found > -- Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the system variable OPENSSL_ROOT_DIR (missing: OPENSSL_INCLUDE_DIR) > ... > CMake Error at aux/binpac/CMakeLists.txt:17 (message): > Could not find prerequisite package 'OpenSSL' > > > CMake Error at aux/binpac/CMakeLists.txt:19 (message): > Configuration aborted due to missing prerequisites > > thanks, > Randy > > p.s. what?s recommended to search your mail archive - gmane? > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro ------ Adam J. Slagell Chief Information Security Officer Director, Cybersecurity Division National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." From heiland at iu.edu Tue Apr 19 13:48:33 2016 From: heiland at iu.edu (Heiland, Randy) Date: Tue, 19 Apr 2016 20:48:33 +0000 Subject: [Bro] build error on OSX ElC; OpenSSL In-Reply-To: <8B770ECD-699F-42BD-AA71-2BE912A44324@illinois.edu> References: <8B770ECD-699F-42BD-AA71-2BE912A44324@illinois.edu> Message-ID: Yes, but I built it. Turns out, I?ve got multiple versions lying around which might be giving me grief. Sigh. ~/git/bro$ /usr/local/ssl/bin/openssl OpenSSL> version OpenSSL 1.0.2-beta2-dev xx XXX xxxx OpenSSL> quit ~/git/bro$ which openssl /Users/heiland/anaconda3/bin/openssl ~/git/bro$ /Users/heiland/anaconda3/bin/openssl OpenSSL> version OpenSSL 1.0.2g 1 Mar 2016 OpenSSL> quit ~/git/bro$ /usr/bin/openssl OpenSSL> version OpenSSL 0.9.8zh 14 Jan 2016 I?ll dig around? > On Apr 19, 2016, at 4:37 PM, Slagell, Adam J wrote: > > Apple deprecated openssl. > > https://www.bro.org/documentation/faq.html#why-does-v2-4-fail-to-build-on-mac-os-x-10-11 > > >> On Apr 19, 2016, at 3:32 PM, Heiland, Randy wrote: >> >> Hi, >> >> Trying to build Bro on OSX (10.11), I get an error about not being able to find my OpenSSL. Any ideas? >> >> Before attempting the config, I set the following env vars: >> OPENSSL_INCLUDE_DIR=/usr/local/ssl/include >> OPENSSL_LIBRARIES=/usr/local/ssl/lib >> OPENSSL_ROOT_DIR=/usr/local/ssl >> >> >> ~/git/bro$ ./configure --with-openssl=/usr/local/ssl >> Build Directory : build >> Source Directory: /Users/heiland/git/bro >> -- Found sed: /usr/local/bin/sed >> -- Found PythonInterp: /Users/heiland/anaconda3/bin/python (found version "3.5.1") >> -- Found FLEX: /usr/bin/flex (found version "2.5.35") >> -- Found BISON: /usr/local/bin/bison >> -- Found PCAP: /usr/lib/libpcap.dylib >> -- Performing Test PCAP_LINKS_SOLO >> -- Performing Test PCAP_LINKS_SOLO - Success >> -- Looking for pcap_get_pfring_id >> -- Looking for pcap_get_pfring_id - not found >> -- Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the system variable OPENSSL_ROOT_DIR (missing: OPENSSL_INCLUDE_DIR) >> ... >> CMake Error at aux/binpac/CMakeLists.txt:17 (message): >> Could not find prerequisite package 'OpenSSL' >> >> >> CMake Error at aux/binpac/CMakeLists.txt:19 (message): >> Configuration aborted due to missing prerequisites >> >> thanks, >> Randy >> >> p.s. what?s recommended to search your mail archive - gmane? >> >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > ------ > > Adam J. Slagell > Chief Information Security Officer > Director, Cybersecurity Division > National Center for Supercomputing Applications > University of Illinois at Urbana-Champaign > www.slagell.info > > "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." > > > > > > > > From mz89924 at 126.com Wed Apr 20 01:49:16 2016 From: mz89924 at 126.com (126) Date: Wed, 20 Apr 2016 16:49:16 +0800 Subject: [Bro] logs-to-elasticsearch.bro authentication information Message-ID: <004f01d19ae1$851dd920$8f598b60$@126.com> Hi guys I use logs-to-elasticsearch.bro script to log directly to the ES server, but my ES server is certified, will configure the authentication information? Regards, Siemon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160420/1e32ef42/attachment.html From chegov at gmail.com Wed Apr 20 05:36:49 2016 From: chegov at gmail.com (Shiran Aizik) Date: Wed, 20 Apr 2016 15:36:49 +0300 Subject: [Bro] Updating table in SumStats - not recognized in outer event In-Reply-To: References: Message-ID: Hello, I have a global synced table which I update on threshold_crossed in SumStats. The update is successful, yet when I try to access this table from a different event (dns query reply) it is empty. I work wih bro 2.4.1 in cluster mode. The table is defined as global, synced and mergable. My goal is that all workers will send their observations to SumStats and based on a threshold, update a table for all the workers to know. Any ideas what can I try ? Thank you Chegov -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160420/950936b3/attachment.html From martin.liras at gmail.com Wed Apr 20 06:22:46 2016 From: martin.liras at gmail.com (Luis Martin Liras) Date: Wed, 20 Apr 2016 15:22:46 +0200 Subject: [Bro] Undefined symbol error while writing an analyzer Message-ID: <571782A6.3040000@gmail.com> Hi, I'm quite new in bro, and I'm trying to develop an analyzer as a plugin (Not an easy task), using the modbus analyzer as a starting point. I have passed through all the steps and everything seem to be OK: It compiled OK and Bro detects the plugin and its events when run with the -NN option However when I try to register my analyzer: ... Analyzer::register_for_ports(Analyzer::ANALYZER_UMAS, ports); ... I get the following error: ../build/src/bro: symbol lookup error: /tools/pruebas_bro/bro/bro/umas_plugin/build//lib/UMAS-UNITY.linux-x86_64.so : undefined symbol: _ZN8analyzer4umas16UmasTCP_AnalyzerC1EP10Connection I have tried to make it simpler and simpler, but always crash against the same error. Even started a new analyzer from scratch, following the quickstarter guide (the SAMPLE one)... and ended up in the same issue. How could I debug this error?. Anyone has an idea of what could be happening? Thanks ! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160420/de574a4c/attachment-0001.html From heiland at iu.edu Wed Apr 20 10:41:36 2016 From: heiland at iu.edu (Heiland, Randy) Date: Wed, 20 Apr 2016 17:41:36 +0000 Subject: [Bro] build error on OSX ElC; OpenSSL In-Reply-To: References: <8B770ECD-699F-42BD-AA71-2BE912A44324@illinois.edu> Message-ID: <70130452-23B4-410E-B069-ED10E8D2E174@iu.edu> Got this working and will just report for the archive. Not entirely sure what the problem/fix was, but I: 1) upgraded cmake (to 3.5.2; previous was 2.8.12) 2) upgraded openssl: OpenSSL 1.0.2g 1 Mar 2016 3) made sure I wasn?t using the Anaconda Python distro (which also included openssl); rather using system Python (2.7.10) 4) upgraded couple other libs after bro warnings - CAF and swig 4) config?d via: ~/git/bro$ ./configure --with-openssl=/usr/local/ssl and seemed to finally build OK: ~$ bro --version bro version 2.4-454 -Randy > On Apr 19, 2016, at 4:48 PM, Heiland, Randy wrote: > > Yes, but I built it. Turns out, I?ve got multiple versions lying around which might be giving me grief. Sigh. > > ~/git/bro$ /usr/local/ssl/bin/openssl > OpenSSL> version > OpenSSL 1.0.2-beta2-dev xx XXX xxxx > OpenSSL> quit > > ~/git/bro$ which openssl > /Users/heiland/anaconda3/bin/openssl > ~/git/bro$ /Users/heiland/anaconda3/bin/openssl > OpenSSL> version > OpenSSL 1.0.2g 1 Mar 2016 > OpenSSL> quit > > ~/git/bro$ /usr/bin/openssl > OpenSSL> version > OpenSSL 0.9.8zh 14 Jan 2016 > > I?ll dig around? > > > >> On Apr 19, 2016, at 4:37 PM, Slagell, Adam J wrote: >> >> Apple deprecated openssl. >> >> https://www.bro.org/documentation/faq.html#why-does-v2-4-fail-to-build-on-mac-os-x-10-11 >> >> >>> On Apr 19, 2016, at 3:32 PM, Heiland, Randy wrote: >>> >>> Hi, >>> >>> Trying to build Bro on OSX (10.11), I get an error about not being able to find my OpenSSL. Any ideas? >>> >>> Before attempting the config, I set the following env vars: >>> OPENSSL_INCLUDE_DIR=/usr/local/ssl/include >>> OPENSSL_LIBRARIES=/usr/local/ssl/lib >>> OPENSSL_ROOT_DIR=/usr/local/ssl >>> >>> >>> ~/git/bro$ ./configure --with-openssl=/usr/local/ssl >>> Build Directory : build >>> Source Directory: /Users/heiland/git/bro >>> -- Found sed: /usr/local/bin/sed >>> -- Found PythonInterp: /Users/heiland/anaconda3/bin/python (found version "3.5.1") >>> -- Found FLEX: /usr/bin/flex (found version "2.5.35") >>> -- Found BISON: /usr/local/bin/bison >>> -- Found PCAP: /usr/lib/libpcap.dylib >>> -- Performing Test PCAP_LINKS_SOLO >>> -- Performing Test PCAP_LINKS_SOLO - Success >>> -- Looking for pcap_get_pfring_id >>> -- Looking for pcap_get_pfring_id - not found >>> -- Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the system variable OPENSSL_ROOT_DIR (missing: OPENSSL_INCLUDE_DIR) >>> ... >>> CMake Error at aux/binpac/CMakeLists.txt:17 (message): >>> Could not find prerequisite package 'OpenSSL' >>> >>> >>> CMake Error at aux/binpac/CMakeLists.txt:19 (message): >>> Configuration aborted due to missing prerequisites >>> >>> thanks, >>> Randy >>> >>> p.s. what?s recommended to search your mail archive - gmane? >>> >>> >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> ------ >> >> Adam J. Slagell >> Chief Information Security Officer >> Director, Cybersecurity Division >> National Center for Supercomputing Applications >> University of Illinois at Urbana-Champaign >> www.slagell.info >> >> "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." >> >> >> >> >> >> >> >> > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From josh.guild at morphick.com Thu Apr 21 07:30:32 2016 From: josh.guild at morphick.com (Josh Guild) Date: Thu, 21 Apr 2016 10:30:32 -0400 Subject: [Bro] Bro and APCON In-Reply-To: <4BEC04C5-8DF5-4804-987A-165230A55F84@pingtrip.com> References: <4BEC04C5-8DF5-4804-987A-165230A55F84@pingtrip.com> Message-ID: Hi Dave, We bypassed the APCON in one of the environments and it helped a little with capture loss (about a 10% drop) and errors in the weird.log. Unfortunately, this was during a weekend so it's tough to say how much of an impact it made. Another network we're in fixed some load balancing issues upstream and this help significantly with loss (though weird.logs remain about where they were). I think the APCON may have been a red herring in this instance but I'd be curious to see how your network looks before and after implementation if you'd like to keep in touch. The main things I've been looking at are capture loss and weird.log errors (specifically HTTP_version_mismatch, SYN_seq_jump, TCP_seq_underflow_or_misorder) these may lean towards traffic being mangled. This presentation is pretty helpful in showing you what to look for: https://speakerdeck.com/vladg/bro-deployment-verification-and-troubleshooting Thanks! On Thu, Apr 21, 2016 at 10:08 AM, Dave Crawford wrote: > Josh, > > Were you able to solve this issue? We just started swapping out our > current solution with Apcon?s and wondering if we?ll run into the same > issue. > > -Dave > > On Apr 7, 2016, at 2:39 PM, Josh Guild wrote: > > Hi all, > > We have a few deployments that utilize an APCON for traffic aggregation. > We've noticed in these environments that Bro has trouble reassembling the > traffic correctly and there is a significant amount of capture loss (based > on the script). We've tried different hashing algorithms on the APCON to no > effect. > > Has anyone else seen anything similar to this or have any insight? > > Thanks! > > -- > Josh Guild > Network Intelligence Analyst > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > -- Josh Guild Network Intelligence Analyst -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160421/de09b33a/attachment.html From bro at pingtrip.com Thu Apr 21 07:45:24 2016 From: bro at pingtrip.com (Dave Crawford) Date: Thu, 21 Apr 2016 10:45:24 -0400 Subject: [Bro] Bro and APCON In-Reply-To: References: <4BEC04C5-8DF5-4804-987A-165230A55F84@pingtrip.com> Message-ID: <7E88231B-4421-4C5E-8758-4804255130E9@pingtrip.com> Thanks Josh, We?re also trying to determine if the Apcon is a red herring since (unfortunately) two changes were made at the same time. While we swapped our Anues for Apcons the network team was also upgrading to Nexus switches. Our weird log started filling with with ?data_before_established? and ?possible_split_routing? events right after the changes. -Dave > On Apr 21, 2016, at 10:30 AM, Josh Guild wrote: > > Hi Dave, > > We bypassed the APCON in one of the environments and it helped a little with capture loss (about a 10% drop) and errors in the weird.log. Unfortunately, this was during a weekend so it's tough to say how much of an impact it made. Another network we're in fixed some load balancing issues upstream and this help significantly with loss (though weird.logs remain about where they were). I think the APCON may have been a red herring in this instance but I'd be curious to see how your network looks before and after implementation if you'd like to keep in touch. > > The main things I've been looking at are capture loss and weird.log errors (specifically HTTP_version_mismatch, SYN_seq_jump, TCP_seq_underflow_or_misorder) these may lean towards traffic being mangled. This presentation is pretty helpful in showing you what to look for: https://speakerdeck.com/vladg/bro-deployment-verification-and-troubleshooting > > Thanks! > > On Thu, Apr 21, 2016 at 10:08 AM, Dave Crawford > wrote: > Josh, > > Were you able to solve this issue? We just started swapping out our current solution with Apcon?s and wondering if we?ll run into the same issue. > > -Dave > >> On Apr 7, 2016, at 2:39 PM, Josh Guild > wrote: >> >> Hi all, >> >> We have a few deployments that utilize an APCON for traffic aggregation. We've noticed in these environments that Bro has trouble reassembling the traffic correctly and there is a significant amount of capture loss (based on the script). We've tried different hashing algorithms on the APCON to no effect. >> >> Has anyone else seen anything similar to this or have any insight? >> >> Thanks! >> >> -- >> Josh Guild >> Network Intelligence Analyst >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > -- > Josh Guild > Network Intelligence Analyst > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160421/cace80fc/attachment.html From martin.liras at gmail.com Thu Apr 21 08:46:43 2016 From: martin.liras at gmail.com (Luis Martin Liras) Date: Thu, 21 Apr 2016 17:46:43 +0200 Subject: [Bro] Fwd: Undefined symbol while writing an analyzer In-Reply-To: References: Message-ID: <5718F5E3.3080800@gmail.com> Hey guys, I would really appreciate some help on this. I think I can reduce the source of the problem to two points: - A namespace problem. I have revised my code and does not find anything wrong. I'll keep checking - A linkage problem: To compile the plugin I followed the instructions of the "Writing Bro Plugins" documentation guide. In the Plugins.cc file instantiated of the Analyzer and included "Myprotocol.h", which had been previously created with binpac_Quickstart's start.py However the Makefile and configure files were created with the "init-plugin" script. Do you know if I have to modify anything in the Makefile to build a protocol analyzer plugin? Unfortunately in the "Writing Bro plugins" documentation page, the "Protocol Analyzers" section is empty... Hope you can help me. Best reagards! -------- Forwarded Message -------- Subject: Undefined symbol while writing an analyzer Date: Wed, 20 Apr 2016 11:48:12 +0200 From: Luis Martin To: bro at bro.org Hi, I'm quite new in bro, but I'm trying to develop an analyzer as a plugin (Not an easy task), using the modbus analyzer as a starting point. I have passed through all the steps and everything seem to be OK: It compiled OK and Bro detects the plugin and its events when run with the -NN option However when I try to register my analyzer: ... Analyzer::register_for_ports(Analyzer::ANALYZER_UMAS, ports); ... I get the following error: ../build/src/bro: symbol lookup error: /tools/pruebas_bro/bro/bro/umas_plugin/build//lib/UMAS-UNITY.linux-x86_64.so : undefined symbol: _ZN8analyzer4umas16UmasTCP_AnalyzerC1EP10Connection Is necessary to declare ANALYZER_UMAS anywhere?. How could I debug this error?. Anyone who has an idea could help me ? Thanks ! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160421/766fa08b/attachment-0001.html From josh.guild at morphick.com Thu Apr 21 09:22:34 2016 From: josh.guild at morphick.com (Josh Guild) Date: Thu, 21 Apr 2016 12:22:34 -0400 Subject: [Bro] Bro and APCON In-Reply-To: <7E88231B-4421-4C5E-8758-4804255130E9@pingtrip.com> References: <4BEC04C5-8DF5-4804-987A-165230A55F84@pingtrip.com> <7E88231B-4421-4C5E-8758-4804255130E9@pingtrip.com> Message-ID: Yep, in one of the environments, we're getting a ton of "possible_split_routing" and "data_before_established" both with and without the APCON in the mix. Thinking it has to do with how Bro is handling the load balancing to workers. On Thu, Apr 21, 2016 at 10:45 AM, Dave Crawford wrote: > Thanks Josh, > > We?re also trying to determine if the Apcon is a red herring since > (unfortunately) two changes were made at the same time. While we swapped > our Anues for Apcons the network team was also upgrading to Nexus switches. > > Our weird log started filling with with ?data_before_established? and > ?possible_split_routing? events right after the changes. > > -Dave > > > On Apr 21, 2016, at 10:30 AM, Josh Guild wrote: > > Hi Dave, > > We bypassed the APCON in one of the environments and it helped a little > with capture loss (about a 10% drop) and errors in the weird.log. > Unfortunately, this was during a weekend so it's tough to say how much of > an impact it made. Another network we're in fixed some load balancing > issues upstream and this help significantly with loss (though weird.logs > remain about where they were). I think the APCON may have been a red > herring in this instance but I'd be curious to see how your network looks > before and after implementation if you'd like to keep in touch. > > The main things I've been looking at are capture loss and weird.log errors > (specifically HTTP_version_mismatch, SYN_seq_jump, > TCP_seq_underflow_or_misorder) these may lean towards traffic being > mangled. This presentation is pretty helpful in showing you what to look > for: > https://speakerdeck.com/vladg/bro-deployment-verification-and-troubleshooting > > Thanks! > > On Thu, Apr 21, 2016 at 10:08 AM, Dave Crawford wrote: > >> Josh, >> >> Were you able to solve this issue? We just started swapping out our >> current solution with Apcon?s and wondering if we?ll run into the same >> issue. >> >> -Dave >> >> On Apr 7, 2016, at 2:39 PM, Josh Guild wrote: >> >> Hi all, >> >> We have a few deployments that utilize an APCON for traffic aggregation. >> We've noticed in these environments that Bro has trouble reassembling the >> traffic correctly and there is a significant amount of capture loss (based >> on the script). We've tried different hashing algorithms on the APCON to no >> effect. >> >> Has anyone else seen anything similar to this or have any insight? >> >> Thanks! >> >> -- >> Josh Guild >> Network Intelligence Analyst >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> >> >> > > > -- > Josh Guild > Network Intelligence Analyst > > > > -- Josh Guild Network Intelligence Analyst -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160421/c58fac37/attachment.html From bro at pingtrip.com Thu Apr 21 16:57:57 2016 From: bro at pingtrip.com (Dave Crawford) Date: Thu, 21 Apr 2016 19:57:57 -0400 Subject: [Bro] Bro and APCON In-Reply-To: References: <4BEC04C5-8DF5-4804-987A-165230A55F84@pingtrip.com> <7E88231B-4421-4C5E-8758-4804255130E9@pingtrip.com> Message-ID: <82C4FCF2-6052-47D2-BF67-7D080B08A690@pingtrip.com> Just providing a quick update that think we solved our issue. Our SPANs began getting FabricPath packets after the network team upgraded to Nexus switches (see my separate thread about workers at 100% CPU). Long story short, Apcon has a "Protocol Stripping" configuration screen where you can enable stripping of the FP encapsulation layer. After enabling that option the "data_before_established" and "inappropriate_FIN" messages stopped. -Dave > On Apr 21, 2016, at 12:22 PM, Josh Guild wrote: > > Yep, in one of the environments, we're getting a ton of "possible_split_routing" and "data_before_established" both with and without the APCON in the mix. > Thinking it has to do with how Bro is handling the load balancing to workers. > > On Thu, Apr 21, 2016 at 10:45 AM, Dave Crawford > wrote: > Thanks Josh, > > We?re also trying to determine if the Apcon is a red herring since (unfortunately) two changes were made at the same time. While we swapped our Anues for Apcons the network team was also upgrading to Nexus switches. > > Our weird log started filling with with ?data_before_established? and ?possible_split_routing? events right after the changes. > > -Dave > > >> On Apr 21, 2016, at 10:30 AM, Josh Guild > wrote: >> >> Hi Dave, >> >> We bypassed the APCON in one of the environments and it helped a little with capture loss (about a 10% drop) and errors in the weird.log. Unfortunately, this was during a weekend so it's tough to say how much of an impact it made. Another network we're in fixed some load balancing issues upstream and this help significantly with loss (though weird.logs remain about where they were). I think the APCON may have been a red herring in this instance but I'd be curious to see how your network looks before and after implementation if you'd like to keep in touch. >> >> The main things I've been looking at are capture loss and weird.log errors (specifically HTTP_version_mismatch, SYN_seq_jump, TCP_seq_underflow_or_misorder) these may lean towards traffic being mangled. This presentation is pretty helpful in showing you what to look for: https://speakerdeck.com/vladg/bro-deployment-verification-and-troubleshooting >> >> Thanks! >> >> On Thu, Apr 21, 2016 at 10:08 AM, Dave Crawford > wrote: >> Josh, >> >> Were you able to solve this issue? We just started swapping out our current solution with Apcon?s and wondering if we?ll run into the same issue. >> >> -Dave >> >>> On Apr 7, 2016, at 2:39 PM, Josh Guild > wrote: >>> >>> Hi all, >>> >>> We have a few deployments that utilize an APCON for traffic aggregation. We've noticed in these environments that Bro has trouble reassembling the traffic correctly and there is a significant amount of capture loss (based on the script). We've tried different hashing algorithms on the APCON to no effect. >>> >>> Has anyone else seen anything similar to this or have any insight? >>> >>> Thanks! >>> >>> -- >>> Josh Guild >>> Network Intelligence Analyst >>> >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> >> >> -- >> Josh Guild >> Network Intelligence Analyst >> >> > > > > > -- > Josh Guild > Network Intelligence Analyst > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160421/60615c01/attachment.html From josh.guild at morphick.com Thu Apr 21 17:07:23 2016 From: josh.guild at morphick.com (Josh Guild) Date: Fri, 22 Apr 2016 00:07:23 +0000 Subject: [Bro] Bro and APCON In-Reply-To: <82C4FCF2-6052-47D2-BF67-7D080B08A690@pingtrip.com> References: <4BEC04C5-8DF5-4804-987A-165230A55F84@pingtrip.com> <7E88231B-4421-4C5E-8758-4804255130E9@pingtrip.com> <82C4FCF2-6052-47D2-BF67-7D080B08A690@pingtrip.com> Message-ID: That's awesome that you got it worked out! I'll take a look and see what the configs are for the APCONs we're dealing with and maybe get to the bottom of our issue. Thanks! On Thu, Apr 21, 2016, 19:58 Dave Crawford wrote: > Just providing a quick update that think we solved our issue. Our SPANs > began getting FabricPath packets after the network team upgraded to Nexus > switches (see my separate thread about workers at 100% CPU). > > Long story short, Apcon has a "Protocol Stripping" configuration screen > where you can enable stripping of the FP encapsulation layer. After > enabling that option the "data_before_established" and "inappropriate_FIN" > messages stopped. > > -Dave > > > On Apr 21, 2016, at 12:22 PM, Josh Guild wrote: > > Yep, in one of the environments, we're getting a ton of > "possible_split_routing" and "data_before_established" both with and > without the APCON in the mix. > Thinking it has to do with how Bro is handling the load balancing to > workers. > > On Thu, Apr 21, 2016 at 10:45 AM, Dave Crawford wrote: > >> Thanks Josh, >> >> We?re also trying to determine if the Apcon is a red herring since >> (unfortunately) two changes were made at the same time. While we swapped >> our Anues for Apcons the network team was also upgrading to Nexus switches. >> >> Our weird log started filling with with ?data_before_established? and >> ?possible_split_routing? events right after the changes. >> >> -Dave >> >> >> On Apr 21, 2016, at 10:30 AM, Josh Guild wrote: >> >> Hi Dave, >> >> We bypassed the APCON in one of the environments and it helped a little >> with capture loss (about a 10% drop) and errors in the weird.log. >> Unfortunately, this was during a weekend so it's tough to say how much of >> an impact it made. Another network we're in fixed some load balancing >> issues upstream and this help significantly with loss (though weird.logs >> remain about where they were). I think the APCON may have been a red >> herring in this instance but I'd be curious to see how your network looks >> before and after implementation if you'd like to keep in touch. >> >> The main things I've been looking at are capture loss and weird.log >> errors (specifically HTTP_version_mismatch, SYN_seq_jump, >> TCP_seq_underflow_or_misorder) these may lean towards traffic being >> mangled. This presentation is pretty helpful in showing you what to look >> for: >> https://speakerdeck.com/vladg/bro-deployment-verification-and-troubleshooting >> >> Thanks! >> >> On Thu, Apr 21, 2016 at 10:08 AM, Dave Crawford >> wrote: >> >>> Josh, >>> >>> Were you able to solve this issue? We just started swapping out our >>> current solution with Apcon?s and wondering if we?ll run into the same >>> issue. >>> >>> -Dave >>> >>> On Apr 7, 2016, at 2:39 PM, Josh Guild wrote: >>> >>> Hi all, >>> >>> We have a few deployments that utilize an APCON for traffic aggregation. >>> We've noticed in these environments that Bro has trouble reassembling the >>> traffic correctly and there is a significant amount of capture loss (based >>> on the script). We've tried different hashing algorithms on the APCON to no >>> effect. >>> >>> Has anyone else seen anything similar to this or have any insight? >>> >>> Thanks! >>> >>> -- >>> Josh Guild >>> Network Intelligence Analyst >>> >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >>> >>> >>> >> >> >> -- >> Josh Guild >> Network Intelligence Analyst >> >> >> >> > > > -- > Josh Guild > Network Intelligence Analyst > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160422/6a56572b/attachment-0001.html From vladg at illinois.edu Fri Apr 22 09:33:25 2016 From: vladg at illinois.edu (Vlad Grigorescu) Date: Fri, 22 Apr 2016 11:33:25 -0500 Subject: [Bro] Fwd: Undefined symbol while writing an analyzer In-Reply-To: <5718F5E3.3080800@gmail.com> References: <5718F5E3.3080800@gmail.com> Message-ID: Hi Luis, binpac_quickstart has a --plugin option which should set up the skeleton in much the same way init-plugin does. Try that, and please let me know if you encounter issues with it. --Vlad Luis Martin Liras writes: > Hey guys, > > I would really appreciate some help on this. > > I think I can reduce the source of the problem to two points: > > - A namespace problem. I have revised my code and does not find anything > wrong. I'll keep checking > - A linkage problem: > > To compile the plugin I followed the instructions of the "Writing Bro > Plugins" documentation guide. In the Plugins.cc file instantiated of the > Analyzer and included "Myprotocol.h", which had been previously created > with binpac_Quickstart's start.py > > However the Makefile and configure files were created with the > "init-plugin" script. Do you know if I have to modify anything in the > Makefile to build a protocol analyzer plugin? > > Unfortunately in the "Writing Bro plugins" documentation page, the > "Protocol Analyzers" section is empty... > > Hope you can help me. > > Best reagards! > > > -------- Forwarded Message -------- > Subject: Undefined symbol while writing an analyzer > Date: Wed, 20 Apr 2016 11:48:12 +0200 > From: Luis Martin > To: bro at bro.org > > > > Hi, > > I'm quite new in bro, but I'm trying to develop an analyzer as a plugin > (Not an easy task), using the modbus analyzer as a starting point. > > I have passed through all the steps and everything seem to be OK: It > compiled OK and Bro detects the plugin and its events when run with the > -NN option > > However when I try to register my analyzer: > > ... > Analyzer::register_for_ports(Analyzer::ANALYZER_UMAS, ports); > ... > > I get the following error: > > ../build/src/bro: symbol lookup error: > /tools/pruebas_bro/bro/bro/umas_plugin/build//lib/UMAS-UNITY.linux-x86_64.so > : undefined symbol: > _ZN8analyzer4umas16UmasTCP_AnalyzerC1EP10Connection > > Is necessary to declare ANALYZER_UMAS anywhere?. > > > How could I debug this error?. > > Anyone who has an idea could help me ? > > Thanks ! > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 800 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160422/f7b733bc/attachment.bin From vladg at illinois.edu Fri Apr 22 09:54:47 2016 From: vladg at illinois.edu (Vlad Grigorescu) Date: Fri, 22 Apr 2016 11:54:47 -0500 Subject: [Bro] Telnet log In-Reply-To: References: Message-ID: Mayyyybe...? There's an older Login analyzer which claims to have support for Telnet, but I personally haven't used it, so I don't know how well it would work. I believe that it was pretty thorough, it just hasn't been used in a while. The events you're looking for are: https://www.bro.org/sphinx/scripts/base/bif/plugins/Bro_Login.events.bif.bro.html#id-login_output_line https://www.bro.org/sphinx/scripts/base/bif/plugins/Bro_Login.events.bif.bro.html#id-login_input_line However, as the TODO states, this protocol analyzer does not get activated in Bro 2.x. A copy of the script that Bro used to use is here, though it's a mess and would require porting and cleanup: https://github.com/grigorescu/bro-scripts/blob/master/scripts/todo/needs_review/login.bro I might start with something like this, run it against some PCAPs and then start building a log from there: > const ports = { 23/tcp }; > > event bro_init() &priority=5 > { > Analyzer::register_for_ports(Analyzer::ANALYZER_LOGIN, ports); > } > > event login_input_line(c: connection, line: string) > { > print "<-", line; > } > > event login_output_line(c: connection, line: string) > { > print "->", line; > } --Vlad john smith writes: > Hello, > > With 2.4.1, is there any way to generate Telnet logs? Thanks in advance! > > John > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 800 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160422/6212500f/attachment.bin From vladg at illinois.edu Fri Apr 22 09:57:45 2016 From: vladg at illinois.edu (Vlad Grigorescu) Date: Fri, 22 Apr 2016 11:57:45 -0500 Subject: [Bro] Shockwave Flash Analyzer In-Reply-To: References: Message-ID: I do not, but it's a thought I've kicked around before. If there are specific indicators in the headers that could be linked to malicious Flash files, I think that would provide more incentive to write such an analyzer. Do you (or someone else) know if that's the case? I've seen some malicious Flash files that claim they're just 1x1 pixels (or maybe even 0x0?), but I'm not sure if that's common for files which are... given that it's Flash, I'll say "less malicious" rather than "benign." :-) --Vlad "John B. Althouse III" writes: > Does anyone know of a Shockwave Flash analyzer for Bro? It would be useful > to gather the metadata in the header like version, width, hight, frame > rate, frame count, compression ratio, ect. > > Thanks! > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 800 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160422/7f00152c/attachment.bin From martin.liras at gmail.com Mon Apr 25 00:04:54 2016 From: martin.liras at gmail.com (Luis Martin Liras) Date: Mon, 25 Apr 2016 09:04:54 +0200 Subject: [Bro] Fwd: Undefined symbol while writing an analyzer In-Reply-To: References: <5718F5E3.3080800@gmail.com> Message-ID: <571DC196.70201@gmail.com> Thank you for your reply Vlad. Unfortunately that was not the problem as I've been using this option from the beginning. The problem was related with the linkage options. I was using the configure and Makefile files that the init-plugin --plugin script was giving me. BUT these 'configure' and 'Makefile' files are prepared for a simple plugin with simple functions. If you need to write an analyzer with thi --plugin option you need to modify the resulting CMakeLists.txt. This is how it comes: cmake_minimum_required(VERSION 2.8) project(Plugin) include(BroPlugin) bro_plugin_begin(MyProt MyProt) bro_plugin_cc(src/Plugin.cc) bro_plugin_bif(src/events.bif) bro_plugin_dist_files(README CHANGES COPYING VERSION) bro_plugin_end() and this is how it must be: cmake_minimum_required(VERSION 2.8) project(Plugin) include(BroPlugin) bro_plugin_begin(MyProt MyProt) bro_plugin_bif(src/types.bif src/events.bif) <--- bro_plugin_cc(src/Plugin.cc src/MyProt.cc src/MyProt_pac.cc) <--- bro_plugin_dist_files(README CHANGES COPYING VERSION) bro_plugin_end() Apart from that, I had to add the different records in the init-bare.bro file and, weird enough, I had to modify the build/src/types.bif.netvar_h file to add the records I was using, maybe someone can explain me that. Ah!, and do not reuse a type name that other analyzer is already using... it will give you a segmentation fault. Now it works fine. Thank you! On 22/04/16 18:33, Vlad Grigorescu wrote: > Hi Luis, > > binpac_quickstart has a --plugin option which should set up the skeleton > in much the same way init-plugin does. Try that, and please let me know > if you encounter issues with it. > > --Vlad > > Luis Martin Liras writes: > >> Hey guys, >> >> I would really appreciate some help on this. >> >> I think I can reduce the source of the problem to two points: >> >> - A namespace problem. I have revised my code and does not find anything >> wrong. I'll keep checking >> - A linkage problem: >> >> To compile the plugin I followed the instructions of the "Writing Bro >> Plugins" documentation guide. In the Plugins.cc file instantiated of the >> Analyzer and included "Myprotocol.h", which had been previously created >> with binpac_Quickstart's start.py >> >> However the Makefile and configure files were created with the >> "init-plugin" script. Do you know if I have to modify anything in the >> Makefile to build a protocol analyzer plugin? >> >> Unfortunately in the "Writing Bro plugins" documentation page, the >> "Protocol Analyzers" section is empty... >> >> Hope you can help me. >> >> Best reagards! >> >> >> -------- Forwarded Message -------- >> Subject: Undefined symbol while writing an analyzer >> Date: Wed, 20 Apr 2016 11:48:12 +0200 >> From: Luis Martin >> To: bro at bro.org >> >> >> >> Hi, >> >> I'm quite new in bro, but I'm trying to develop an analyzer as a plugin >> (Not an easy task), using the modbus analyzer as a starting point. >> >> I have passed through all the steps and everything seem to be OK: It >> compiled OK and Bro detects the plugin and its events when run with the >> -NN option >> >> However when I try to register my analyzer: >> >> ... >> Analyzer::register_for_ports(Analyzer::ANALYZER_UMAS, ports); >> ... >> >> I get the following error: >> >> ../build/src/bro: symbol lookup error: >> /tools/pruebas_bro/bro/bro/umas_plugin/build//lib/UMAS-UNITY.linux-x86_64.so >> : undefined symbol: >> _ZN8analyzer4umas16UmasTCP_AnalyzerC1EP10Connection >> >> Is necessary to declare ANALYZER_UMAS anywhere?. >> >> >> How could I debug this error?. >> >> Anyone who has an idea could help me ? >> >> Thanks ! >> >> >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From cody.shepherd at apcon.com Mon Apr 25 16:31:42 2016 From: cody.shepherd at apcon.com (Cody Shepherd) Date: Mon, 25 Apr 2016 23:31:42 +0000 Subject: [Bro] outputting only a single log Message-ID: Hi there, Do you have a brief set of instructions for how to have a command like bro --iface output only one of the default logs? E.g. the conn.log. ________________________________ Cody Shepherd Security Engineer Intern cody.shepherd at apcon.com 503.682.4050 tel 503.682.4059 fax [http://utility2012.apcon.com/signature/logo.png] APCON, Inc. 9255 SW Pioneer Court Wilsonville, OR 97070 www.apcon.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160425/c861ac43/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 1146 bytes Desc: image001.png Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160425/c861ac43/attachment.bin From vallentin at icir.org Mon Apr 25 19:43:00 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Mon, 25 Apr 2016 19:43:00 -0700 Subject: [Bro] outputting only a single log In-Reply-To: References: Message-ID: <20160426024300.GB1218@shogun> > Do you have a brief set of instructions for how to have a command like > bro --iface output only one of the default logs? E.g. the > conn.log. Per the similar stackoverflow post [1], you can do this with: bro -i -b base/protocols/conn The flag -b runs Bro in "bare mode." This disables all default scripts. You can then manually turn on only the scripts you need. Matthias [1] http://stackoverflow.com/q/36853106/1170277 From scott.e.knick.ctr at mail.mil Tue Apr 26 00:27:24 2016 From: scott.e.knick.ctr at mail.mil (Knick, Scott E CTR (US)) Date: Tue, 26 Apr 2016 07:27:24 +0000 Subject: [Bro] BroCtl plugin - Hooking into install command (UNCLASSIFIED) Message-ID: CLASSIFICATION: UNCLASSIFIED I've developed a custom BroCtl plugin which attempts to hook into the install command before it executes (i.e., I'm overriding cmd_install_pre()) so that a configuration defined elsewhere in /usr/local/etc can be read and the various Bro configuration files (e.g., node.cfg, networks.cfg, etc.) can be adjusted as a result. This basically works, but I have noticed that it seems like I have to run broctl install *twice* in order to make BroCtl and/or Bro realize the new configuration. Otherwise, Bro crashes and BroCtl tells me to look at the diagnostics using the diag command when I do a broctl start. The actual error messages vary but they all seem to suggest that something in Bro isn't reading in my new configuration as defined in the various Bro configuration files. I have verified that those files are actually updated to what I want prior to running broctl start. Any ideas what might be going on? Am I missing something? -- Scott Knick CLASSIFICATION: UNCLASSIFIED -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5633 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160426/854b1fc4/attachment-0001.bin From tgdesrochers at gmail.com Tue Apr 26 06:07:32 2016 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Tue, 26 Apr 2016 09:07:32 -0400 Subject: [Bro] [bro] Extending intel.log Message-ID: <571f6819.90dc8c0a.23157.ffffb97a@mx.google.com> Is there an easy way to extend the intel.log file to include the meta.url field. I ingest these logs into ELK and having the meta.url would be extremely helpful. Right now when my logs print I get seen_indicator, seen_indicator_type, seen_node, seen_where, and sources, but I?d like to have the meta URL come through and print in the log to make it easy for an analyst to find the source documentation for the referenced intel alert. Thanks Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160426/b4b84450/attachment.html From hosom at battelle.org Tue Apr 26 06:46:44 2016 From: hosom at battelle.org (Hosom, Stephen M) Date: Tue, 26 Apr 2016 13:46:44 +0000 Subject: [Bro] [bro] Extending intel.log References: <571f6819.90dc8c0a.23157.ffffb97a@mx.google.com> Message-ID: The easiest thing to do at the moment is to place the url in the source field. There are some custom scripts that add custom fields to the intel.log, but if I remember right, in order to do that they replace parts of the intel framework, which would likely be prone to breaking without notice on version changes. On 04/26/2016 09:31 AM, Tim Desrochers wrote: Is there an easy way to extend the intel.log file to include the meta.url field. I ingest these logs into ELK and having the meta.url would be extremely helpful. Right now when my logs print I get seen_indicator, seen_indicator_type, seen_node, seen_where, and sources, but I?d like to have the meta URL come through and print in the log to make it easy for an analyst to find the source documentation for the referenced intel alert. Thanks Tim From dnthayer at illinois.edu Tue Apr 26 07:46:52 2016 From: dnthayer at illinois.edu (Thayer, Daniel N) Date: Tue, 26 Apr 2016 14:46:52 +0000 Subject: [Bro] BroCtl plugin - Hooking into install command (UNCLASSIFIED) In-Reply-To: References: Message-ID: <8F865DA62E66F543B6104A2835719CF93913E9BD@CITESMBX5.ad.uillinois.edu> The problem here is that broctl reads the config files when broctl starts up, not when the "install" command is run, so by the time your pre-install plugin runs, the config files have already been read. ________________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Knick, Scott E CTR (US) [scott.e.knick.ctr at mail.mil] Sent: Tuesday, April 26, 2016 2:27 AM To: bro at bro.org Subject: [Bro] BroCtl plugin - Hooking into install command (UNCLASSIFIED) CLASSIFICATION: UNCLASSIFIED I've developed a custom BroCtl plugin which attempts to hook into the install command before it executes (i.e., I'm overriding cmd_install_pre()) so that a configuration defined elsewhere in /usr/local/etc can be read and the various Bro configuration files (e.g., node.cfg, networks.cfg, etc.) can be adjusted as a result. This basically works, but I have noticed that it seems like I have to run broctl install *twice* in order to make BroCtl and/or Bro realize the new configuration. Otherwise, Bro crashes and BroCtl tells me to look at the diagnostics using the diag command when I do a broctl start. The actual error messages vary but they all seem to suggest that something in Bro isn't reading in my new configuration as defined in the various Bro configuration files. I have verified that those files are actually updated to what I want prior to running broctl start. Any ideas what might be going on? Am I missing something? -- Scott Knick CLASSIFICATION: UNCLASSIFIED From seth at icir.org Tue Apr 26 11:00:37 2016 From: seth at icir.org (Seth Hall) Date: Tue, 26 Apr 2016 14:00:37 -0400 Subject: [Bro] [bro] Extending intel.log In-Reply-To: <571f6819.90dc8c0a.23157.ffffb97a@mx.google.com> References: <571f6819.90dc8c0a.23157.ffffb97a@mx.google.com> Message-ID: > On Apr 26, 2016, at 9:07 AM, Tim Desrochers wrote: > > Right now when my logs print I get seen_indicator, seen_indicator_type, seen_node, seen_where, and sources, but I?d like to have the meta URL come through and print in the log to make it easy for an analyst to find the source documentation for the referenced intel alert. I have some extensions for the intel framework here: https://github.com/sethhall/intel-ext Look at the tests to see how to use it: https://github.com/sethhall/intel-ext/tree/master/testing/intel-ext I'm hoping that some changes will be coming to the intel framework in the next release that should incorporate changes like these and more too. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From tgdesrochers at gmail.com Tue Apr 26 11:15:38 2016 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Tue, 26 Apr 2016 14:15:38 -0400 Subject: [Bro] [bro] Extending intel.log In-Reply-To: References: <571f6819.90dc8c0a.23157.ffffb97a@mx.google.com> Message-ID: <571fb04a.85b38c0a.be0f5.0272@mx.google.com> Thanks Seth From: Seth Hall Sent: Tuesday, April 26, 2016 2:00 PM To: Tim Desrochers Cc: bro at bro.org Subject: Re: [Bro] [bro] Extending intel.log > On Apr 26, 2016, at 9:07 AM, Tim Desrochers wrote: > > Right now when my logs print I get seen_indicator, seen_indicator_type, seen_node, seen_where, and sources, but I?d like to have the meta URL come through and print in the log to make it easy for an analyst to find the source documentation for the referenced intel alert. I have some extensions for the intel framework here: https://github.com/sethhall/intel-ext Look at the tests to see how to use it: https://github.com/sethhall/intel-ext/tree/master/testing/intel-ext I'm hoping that some changes will be coming to the intel framework in the next release that should incorporate changes like these and more too. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160426/98a0e8b2/attachment.html From scott.e.knick.ctr at mail.mil Tue Apr 26 23:45:04 2016 From: scott.e.knick.ctr at mail.mil (Knick, Scott E CTR (US)) Date: Wed, 27 Apr 2016 06:45:04 +0000 Subject: [Bro] BroCtl plugin - Hooking into install command (UNCLASSIFIED) In-Reply-To: <8F865DA62E66F543B6104A2835719CF93913E9BD@CITESMBX5.ad.uillinois.edu> References: <8F865DA62E66F543B6104A2835719CF93913E9BD@CITESMBX5.ad.uillinois.edu> Message-ID: A non-text attachment was scrubbed... Name: smime.p7m Type: application/pkcs7-mime Size: 13382 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160427/c14f5f0e/attachment.p7c From sven at dreyer-net.de Wed Apr 27 04:40:39 2016 From: sven at dreyer-net.de (Sven Dreyer) Date: Wed, 27 Apr 2016 13:40:39 +0200 Subject: [Bro] Problem with connections in S1 and SF state In-Reply-To: <564B9060.1080304@dreyer-net.de> References: <564B9060.1080304@dreyer-net.de> Message-ID: <5720A537.1010203@dreyer-net.de> Dear list, we are still having problems with messed up src and dst ip/port of some connections. What we did up to now: - Disabled bridging and used a single interface (eth1) for packet capturing, as suggested by Seth Hall - Disabled offloading by running "ethtool -K eth1 tso off ufo off gso off gro off lro off" in the interface's post-up script - Changed hardware from a "PC Engines APU 1C4" embedded board to a "Thomas Krenn LESv2" system - Check the history field in conn.log for "ShA" flags, as suggested by Justin S. Azoff I found several entries like this in conn.log: 1457953693.259152 CaUcf02Z9xpSSHPz24 10.85.1.1 50993 10.85.1.104 41023 tcp ssl 302.736749 987 7059 SF T T 0 ShADadFfR 23 2167 16 7899 (empty) According to conn.log, this is a connection from 10.85.1.1 (internal IP of our server) port 50993 (the IMAP-TLS port the servers uses) to 10.85.1.104 (a notebook computer) port 41023, lasting for 302 seconds. My understanding is that bro saw the whole connection from establishment to termination (according to the history field). In fact, the notebook established a connection to the IMAP service of our server. So src ip and port and dst ip and port are twisted in conn.log. The problem is reproducible (at least in our environment). I captured a pcap file on the same machine running bro, using the same interface and stripped it down so that it contains only the above mentioned connection - 39 frames and about 15 kB (see attachment). If I inject this pcap file in the network using tcpreplay -i eth1 twisted.pcapng bro shows a connection from 10.85.1.1 => 10.85.1.104 (wrong!) in conn.log. If instead I read the pcap file using "bro -r", conn.log shows a connection from 10.85.1.104 => 10.85.1.1 (correct!). Does anyboby have a further idea what we could do to track the problem down? Thanks and best regards, Sven Am 17.11.2015 um 21:38 schrieb Sven Dreyer: > Dear list, > > I'm having trouble understanding some log entries from my conn.log. I > already learned from this mailing list that bro cannot surely detect who > initiated a connection if it does not see the initial connection setup, > which seems logical to me. > > But if I look to my conn.log file, I find entries like these: > > 1446190221.687738 Cbu3fj3FYdODxvLF1h 87.152.221.xxx 50993 > 192.168.100.yyy 36709 tcp ssl 122.745965 1238 5340 > S1 F T 0 ShAD > ad 20 2050 19 6112 (empty) > 1446190138.746769 CykNrp4VEfzbrJ2vm6 87.152.221.xxx 50993 > 192.168.100.yyy 36679 tcp ssl 223.406750 1384 18908 > S1 F T 0 ShAD ad 39 2956 36 > 20360 (empty) > > It looks like our IMAP server (87.152.221.xxx running on port 50993) > initiated a connection to my notebook (192.168.100.yyy). That should not > be possible due to lack of port forwarding for this connection. > > So my first guess is that bro didn't see the initial connection setup > (midstream traffic, OTH state). But I took a look into the documentation > on https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html > regarding the reported states (S1), which says: > > S1 Connection established, not terminated. > > This looks to me like bro saw the connection setup. Or did I get > something wrong here? > > Oh and by the way: the next paragraph reads: > > SF Normal establishment and termination. Note that this is the same > symbol as for state S1. You can tell the two apart because for S1 there > will not be any byte counts in the summary, while for SF there will be. > > I don't understand this. Do S1 and SF really only differ in byte count > zero or non-zero? It seems to me that they also differ in "connection > still alive" and "connection was terminated". > > Looking further trough the logs, I also find entries with "SF" flag in > whuch source and destination seem twisted: > > 1445338094.186121 C9uuKp4dE9nrHo46bd 87.152.220.xxx 50993 > 192.168.100.yyy 20108 tcp -462.348551 401 754 SF > F T 0 DdAfFa 13 921 12 1234 (empty) > > Does anybody have a hint? Did I misunderstand something? > > I'm running bro 2.4.1. > > Thanks a lot! > Sven > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- A non-text attachment was scrubbed... Name: twisted.pcapng Type: application/octet-stream Size: 14856 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160427/c57d736c/attachment-0001.obj From sven at dreyer-net.de Wed Apr 27 05:02:21 2016 From: sven at dreyer-net.de (Sven Dreyer) Date: Wed, 27 Apr 2016 14:02:21 +0200 Subject: [Bro] Using workers without SSH possible? Message-ID: <5720AA4D.8020901@dreyer-net.de> Dear list, I'd like to operate 3 workers in different subnets. Unluckily, the network connection between manager/proxy and these subnets is really slow (around 500 kBit/s) and has frequent outages. Is there a possibility to synchronize the nodes without the necessity to use SSH? Perhaps by a frequent file transfer using rsync? Thanks and best regards, Sven From hosom at battelle.org Wed Apr 27 05:51:29 2016 From: hosom at battelle.org (Hosom, Stephen M) Date: Wed, 27 Apr 2016 12:51:29 +0000 Subject: [Bro] Using workers without SSH possible? References: <5720AA4D.8020901@dreyer-net.de> Message-ID: I don't think I would feel comfortable running a cluster in the way you describe. Generally speaking, you want cluster nodes to all exist on the same layer 2 network. On 04/27/2016 08:44 AM, Sven Dreyer wrote: > Dear list, > > I'd like to operate 3 workers in different subnets. Unluckily, the > network connection between manager/proxy and these subnets is really > slow (around 500 kBit/s) and has frequent outages. > > Is there a possibility to synchronize the nodes without the necessity to > use SSH? Perhaps by a frequent file transfer using rsync? > > Thanks and best regards, > Sven > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From gl89 at cornell.edu Wed Apr 27 05:57:12 2016 From: gl89 at cornell.edu (Glenn Forbes Fleming Larratt) Date: Wed, 27 Apr 2016 08:57:12 -0400 (EDT) Subject: [Bro] Using workers without SSH possible? In-Reply-To: <5720AA4D.8020901@dreyer-net.de> References: <5720AA4D.8020901@dreyer-net.de> Message-ID: Sven, Doesn't rsync default to using ssh as its transport? Also, I'm not sure how using rsync vs. ssh improves things in the face of slow and unreliable networking between nodes; can you elaborate? Best, -g -- Glenn Forbes Fleming Larratt Cornell University IT Security Office On Wed, 27 Apr 2016, Sven Dreyer wrote: > Dear list, > > I'd like to operate 3 workers in different subnets. Unluckily, the > network connection between manager/proxy and these subnets is really > slow (around 500 kBit/s) and has frequent outages. > > Is there a possibility to synchronize the nodes without the necessity to > use SSH? Perhaps by a frequent file transfer using rsync? > > Thanks and best regards, > Sven > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From josh.guild at morphick.com Wed Apr 27 07:14:41 2016 From: josh.guild at morphick.com (Josh Guild) Date: Wed, 27 Apr 2016 10:14:41 -0400 Subject: [Bro] Bro and APCON In-Reply-To: References: <4BEC04C5-8DF5-4804-987A-165230A55F84@pingtrip.com> <7E88231B-4421-4C5E-8758-4804255130E9@pingtrip.com> <82C4FCF2-6052-47D2-BF67-7D080B08A690@pingtrip.com> Message-ID: Hey Dave, Still doing some preliminary analysis but we had one of our clients strip the VLAN protocol on our egress ports from the APCON and it allowed Bro/PF_Ring to sessionize the traffic properly (we had tried different -tuples in our set up previously). We have a test we run that checks each level of our framework to make sure we have the proper visibility. This was always a scattershot of what was logged by Bro but, after we stripped the VLAN protocol going to our box, it was cleaned up and looked good. I'm going to have them strip the FabricPath protocol and see how that affects the traffic as well (can only strip one protocol at a time). Odd thing is, the weird.log entries were still roughly the same with the VLANs stripped, so it was something with how Bro or PF_Ring was handling the incoming packets from the APCON. Research continues! On Thu, Apr 21, 2016 at 8:07 PM, Josh Guild wrote: > That's awesome that you got it worked out! I'll take a look and see what > the configs are for the APCONs we're dealing with and maybe get to the > bottom of our issue. > > Thanks! > > On Thu, Apr 21, 2016, 19:58 Dave Crawford wrote: > >> Just providing a quick update that think we solved our issue. Our SPANs >> began getting FabricPath packets after the network team upgraded to Nexus >> switches (see my separate thread about workers at 100% CPU). >> >> Long story short, Apcon has a "Protocol Stripping" configuration screen >> where you can enable stripping of the FP encapsulation layer. After >> enabling that option the "data_before_established" and "inappropriate_FIN" >> messages stopped. >> >> -Dave >> >> >> On Apr 21, 2016, at 12:22 PM, Josh Guild wrote: >> >> Yep, in one of the environments, we're getting a ton of >> "possible_split_routing" and "data_before_established" both with and >> without the APCON in the mix. >> Thinking it has to do with how Bro is handling the load balancing to >> workers. >> >> On Thu, Apr 21, 2016 at 10:45 AM, Dave Crawford wrote: >> >>> Thanks Josh, >>> >>> We?re also trying to determine if the Apcon is a red herring since >>> (unfortunately) two changes were made at the same time. While we swapped >>> our Anues for Apcons the network team was also upgrading to Nexus switches. >>> >>> Our weird log started filling with with ?data_before_established? and >>> ?possible_split_routing? events right after the changes. >>> >>> -Dave >>> >>> >>> On Apr 21, 2016, at 10:30 AM, Josh Guild >>> wrote: >>> >>> Hi Dave, >>> >>> We bypassed the APCON in one of the environments and it helped a little >>> with capture loss (about a 10% drop) and errors in the weird.log. >>> Unfortunately, this was during a weekend so it's tough to say how much of >>> an impact it made. Another network we're in fixed some load balancing >>> issues upstream and this help significantly with loss (though weird.logs >>> remain about where they were). I think the APCON may have been a red >>> herring in this instance but I'd be curious to see how your network looks >>> before and after implementation if you'd like to keep in touch. >>> >>> The main things I've been looking at are capture loss and weird.log >>> errors (specifically HTTP_version_mismatch, SYN_seq_jump, >>> TCP_seq_underflow_or_misorder) these may lean towards traffic being >>> mangled. This presentation is pretty helpful in showing you what to look >>> for: >>> https://speakerdeck.com/vladg/bro-deployment-verification-and-troubleshooting >>> >>> Thanks! >>> >>> On Thu, Apr 21, 2016 at 10:08 AM, Dave Crawford >>> wrote: >>> >>>> Josh, >>>> >>>> Were you able to solve this issue? We just started swapping out our >>>> current solution with Apcon?s and wondering if we?ll run into the same >>>> issue. >>>> >>>> -Dave >>>> >>>> On Apr 7, 2016, at 2:39 PM, Josh Guild wrote: >>>> >>>> Hi all, >>>> >>>> We have a few deployments that utilize an APCON for traffic >>>> aggregation. We've noticed in these environments that Bro has trouble >>>> reassembling the traffic correctly and there is a significant amount of >>>> capture loss (based on the script). We've tried different hashing >>>> algorithms on the APCON to no effect. >>>> >>>> Has anyone else seen anything similar to this or have any insight? >>>> >>>> Thanks! >>>> >>>> -- >>>> Josh Guild >>>> Network Intelligence Analyst >>>> >>>> >>>> _______________________________________________ >>>> Bro mailing list >>>> bro at bro-ids.org >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>>> >>>> >>>> >>>> >>> >>> >>> -- >>> Josh Guild >>> Network Intelligence Analyst >>> >>> >>> >>> >> >> >> -- >> Josh Guild >> Network Intelligence Analyst >> >> >> >> -- Josh Guild Network Intelligence Analyst -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160427/291c89fd/attachment-0001.html From jan.grashoefer at gmail.com Wed Apr 27 07:49:06 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Wed, 27 Apr 2016 16:49:06 +0200 Subject: [Bro] Problem with connections in S1 and SF state In-Reply-To: <5720A537.1010203@dreyer-net.de> References: <564B9060.1080304@dreyer-net.de> <5720A537.1010203@dreyer-net.de> Message-ID: <5720D162.6020408@gmail.com> Hi Sven, > bro shows a connection from 10.85.1.1 => 10.85.1.104 (wrong!) in > conn.log. If instead I read the pcap file using "bro -r", conn.log shows > a connection from 10.85.1.104 => 10.85.1.1 (correct!). Do both log lines differ only in receiver/originator? If there are packets missing in your replayed test it is likely that there is an issue with capturing the traffic. Best regards, Jan From jlay at slave-tothe-box.net Wed Apr 27 07:59:23 2016 From: jlay at slave-tothe-box.net (James Lay) Date: Wed, 27 Apr 2016 08:59:23 -0600 Subject: [Bro] Problem with connections in S1 and SF state In-Reply-To: <5720A537.1010203@dreyer-net.de> References: <564B9060.1080304@dreyer-net.de> <5720A537.1010203@dreyer-net.de> Message-ID: On 2016-04-27 05:40, Sven Dreyer wrote: > Dear list, > > we are still having problems with messed up src and dst ip/port of > some connections. > > What we did up to now: > - Disabled bridging and used a single interface (eth1) for packet > capturing, as suggested by Seth Hall > - Disabled offloading by running "ethtool -K eth1 tso off ufo off gso > off gro off lro off" in the interface's post-up script > - Changed hardware from a "PC Engines APU 1C4" embedded board to a > "Thomas Krenn LESv2" system > - Check the history field in conn.log for "ShA" flags, as suggested by > Justin S. Azoff > > I found several entries like this in conn.log: > 1457953693.259152 CaUcf02Z9xpSSHPz24 10.85.1.1 50993 > 10.85.1.104 41023 tcp ssl 302.736749 987 7059 > SF T T 0 ShADadFfR 23 2167 16 > 7899 (empty) > > According to conn.log, this is a connection from 10.85.1.1 (internal > IP of our server) port 50993 (the IMAP-TLS port the servers uses) to > 10.85.1.104 (a notebook computer) port 41023, lasting for 302 seconds. > My understanding is that bro saw the whole connection from > establishment to termination (according to the history field). > > In fact, the notebook established a connection to the IMAP service of > our server. So src ip and port and dst ip and port are twisted in > conn.log. > > The problem is reproducible (at least in our environment). I captured > a pcap file on the same machine running bro, using the same interface > and stripped it down so that it contains only the above mentioned > connection - 39 frames and about 15 kB (see attachment). If I inject > this pcap file in the network using > > tcpreplay -i eth1 twisted.pcapng > > bro shows a connection from 10.85.1.1 => 10.85.1.104 (wrong!) in > conn.log. If instead I read the pcap file using "bro -r", conn.log > shows a connection from 10.85.1.104 => 10.85.1.1 (correct!). > > Does anyboby have a further idea what we could do to track the problem > down? > > Thanks and best regards, Sven > > > Am 17.11.2015 um 21:38 schrieb Sven Dreyer: >> Dear list, >> >> I'm having trouble understanding some log entries from my conn.log. I >> already learned from this mailing list that bro cannot surely detect >> who >> initiated a connection if it does not see the initial connection >> setup, >> which seems logical to me. >> >> But if I look to my conn.log file, I find entries like these: >> >> 1446190221.687738 Cbu3fj3FYdODxvLF1h 87.152.221.xxx 50993 >> 192.168.100.yyy 36709 tcp ssl 122.745965 1238 5340 >> S1 F T 0 ShAD >> ad 20 2050 19 6112 (empty) >> 1446190138.746769 CykNrp4VEfzbrJ2vm6 87.152.221.xxx 50993 >> 192.168.100.yyy 36679 tcp ssl 223.406750 1384 18908 >> S1 F T 0 ShAD ad 39 2956 36 >> 20360 (empty) >> >> It looks like our IMAP server (87.152.221.xxx running on port 50993) >> initiated a connection to my notebook (192.168.100.yyy). That should >> not >> be possible due to lack of port forwarding for this connection. >> >> So my first guess is that bro didn't see the initial connection setup >> (midstream traffic, OTH state). But I took a look into the >> documentation >> on >> https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html >> regarding the reported states (S1), which says: >> >> S1 Connection established, not terminated. >> >> This looks to me like bro saw the connection setup. Or did I get >> something wrong here? >> >> Oh and by the way: the next paragraph reads: >> >> SF Normal establishment and termination. Note that this is the same >> symbol as for state S1. You can tell the two apart because for S1 >> there >> will not be any byte counts in the summary, while for SF there will >> be. >> >> I don't understand this. Do S1 and SF really only differ in byte count >> zero or non-zero? It seems to me that they also differ in "connection >> still alive" and "connection was terminated". >> >> Looking further trough the logs, I also find entries with "SF" flag in >> whuch source and destination seem twisted: >> >> 1445338094.186121 C9uuKp4dE9nrHo46bd 87.152.220.xxx 50993 >> 192.168.100.yyy 20108 tcp -462.348551 401 754 SF >> F T 0 DdAfFa 13 921 12 1234 >> (empty) >> >> Does anybody have a hint? Did I misunderstand something? >> >> I'm running bro 2.4.1. >> >> Thanks a lot! >> Sven >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro Read this thread: http://thread.gmane.org/gmane.comp.security.detection.bro/9211 It might help. James From bro at pingtrip.com Wed Apr 27 09:41:22 2016 From: bro at pingtrip.com (Dave Crawford) Date: Wed, 27 Apr 2016 12:41:22 -0400 Subject: [Bro] Bro and APCON In-Reply-To: References: <4BEC04C5-8DF5-4804-987A-165230A55F84@pingtrip.com> <7E88231B-4421-4C5E-8758-4804255130E9@pingtrip.com> <82C4FCF2-6052-47D2-BF67-7D080B08A690@pingtrip.com> Message-ID: <7995EACA-FC0A-42AE-84A4-B913F959C0D1@pingtrip.com> Thanks for the update Josh! Another hard lesson learned today was that the Apcon requires a power-cycle if any physical changes are made. For us it was removing/replacing a SFP during testing. Were were seeing an extremely high number of ?ethertype unknown? with arbitrary values (in a 1000 packet sample tcpdump reported 340 unique unknown ethertype values) this issue cleared after power-cycling the Apcon. -Dave > On Apr 27, 2016, at 10:14 AM, Josh Guild wrote: > > Hey Dave, > > Still doing some preliminary analysis but we had one of our clients strip the VLAN protocol on our egress ports from the APCON and it allowed Bro/PF_Ring to sessionize the traffic properly (we had tried different -tuples in our set up previously). We have a test we run that checks each level of our framework to make sure we have the proper visibility. This was always a scattershot of what was logged by Bro but, after we stripped the VLAN protocol going to our box, it was cleaned up and looked good. > > I'm going to have them strip the FabricPath protocol and see how that affects the traffic as well (can only strip one protocol at a time). > > Odd thing is, the weird.log entries were still roughly the same with the VLANs stripped, so it was something with how Bro or PF_Ring was handling the incoming packets from the APCON. > > Research continues! From josh.guild at morphick.com Wed Apr 27 09:46:24 2016 From: josh.guild at morphick.com (Josh Guild) Date: Wed, 27 Apr 2016 12:46:24 -0400 Subject: [Bro] Bro and APCON In-Reply-To: <7995EACA-FC0A-42AE-84A4-B913F959C0D1@pingtrip.com> References: <4BEC04C5-8DF5-4804-987A-165230A55F84@pingtrip.com> <7E88231B-4421-4C5E-8758-4804255130E9@pingtrip.com> <82C4FCF2-6052-47D2-BF67-7D080B08A690@pingtrip.com> <7995EACA-FC0A-42AE-84A4-B913F959C0D1@pingtrip.com> Message-ID: That's interesting, I'll be sure to add that to my debugging cheatsheet for sure! On Wed, Apr 27, 2016 at 12:41 PM, Dave Crawford wrote: > Thanks for the update Josh! Another hard lesson learned today was that the > Apcon requires a power-cycle if any physical changes are made. For us it > was removing/replacing a SFP during testing. Were were seeing an extremely > high number of ?ethertype unknown? with arbitrary values (in a 1000 packet > sample tcpdump reported 340 unique unknown ethertype values) this issue > cleared after power-cycling the Apcon. > > -Dave > > > On Apr 27, 2016, at 10:14 AM, Josh Guild > wrote: > > > > Hey Dave, > > > > Still doing some preliminary analysis but we had one of our clients > strip the VLAN protocol on our egress ports from the APCON and it allowed > Bro/PF_Ring to sessionize the traffic properly (we had tried different > -tuples in our set up previously). We have a test we run that checks each > level of our framework to make sure we have the proper visibility. This was > always a scattershot of what was logged by Bro but, after we stripped the > VLAN protocol going to our box, it was cleaned up and looked good. > > > > I'm going to have them strip the FabricPath protocol and see how that > affects the traffic as well (can only strip one protocol at a time). > > > > Odd thing is, the weird.log entries were still roughly the same with the > VLANs stripped, so it was something with how Bro or PF_Ring was handling > the incoming packets from the APCON. > > > > Research continues! > > -- Josh Guild Network Intelligence Analyst -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160427/939a0043/attachment.html From dnthayer at illinois.edu Wed Apr 27 12:55:22 2016 From: dnthayer at illinois.edu (Daniel Thayer) Date: Wed, 27 Apr 2016 14:55:22 -0500 Subject: [Bro] BroCtl plugin - Hooking into install command (UNCLASSIFIED) In-Reply-To: References: <8F865DA62E66F543B6104A2835719CF93913E9BD@CITESMBX5.ad.uillinois.edu> Message-ID: <5721192A.4010408@illinois.edu> Have you tried a pre-deploy plugin (i.e., cmd_deploy_pre)? That might be a better fit for what you're trying to accomplish, unless there's some reason why you would prefer to not use the deploy command. On 04/27/2016 01:45 AM, Knick, Scott E CTR (US) wrote: > CLASSIFICATION: UNCLASSIFIED > > CLASSIFICATION: UNCLASSIFIED > > Thanks. This seems non-intuitive, but regardless, can anyone offer any suggestions to implement what I'm describing using a BroCtl plugin, or am I purely barking up the wrong tree? Thanks. > From bro at pingtrip.com Wed Apr 27 17:21:39 2016 From: bro at pingtrip.com (Dave Crawford) Date: Wed, 27 Apr 2016 20:21:39 -0400 Subject: [Bro] DNSSEC Support Message-ID: It doesn't appear that there is full support for DNSSEC RR types in the current release and I'm looking for the best option in the meantime. For example, answers that include RRSIG's will produce a vector similar to ["192.168.1.1",""] with a corresponding event in weird.log of "DNS_RR_unknown_type". In protocols/dns/consts.bro I see type 46 is included in the query_type map but based on the variable name I assume its not applied to answers? -Dave From scott.e.knick.ctr at mail.mil Thu Apr 28 06:28:59 2016 From: scott.e.knick.ctr at mail.mil (Knick, Scott E CTR (US)) Date: Thu, 28 Apr 2016 13:28:59 +0000 Subject: [Bro] [Non-DoD Source] Re: BroCtl plugin - Hooking into install command (UNCLASSIFIED) In-Reply-To: <5721192A.4010408@illinois.edu> References: <8F865DA62E66F543B6104A2835719CF93913E9BD@CITESMBX5.ad.uillinois.edu> <5721192A.4010408@illinois.edu> Message-ID: CLASSIFICATION: UNCLASSIFIED Thanks for the recommendation! I'll take a look. -----Original Message----- From: Daniel Thayer [mailto:dnthayer at illinois.edu] Sent: Wednesday, April 27, 2016 9:55 PM To: Knick, Scott E CTR (US) ; bro at bro.org Subject: [Non-DoD Source] Re: [Bro] BroCtl plugin - Hooking into install command (UNCLASSIFIED) Have you tried a pre-deploy plugin (i.e., cmd_deploy_pre)? That might be a better fit for what you're trying to accomplish, unless there's some reason why you would prefer to not use the deploy command. On 04/27/2016 01:45 AM, Knick, Scott E CTR (US) wrote: > CLASSIFICATION: UNCLASSIFIED > > CLASSIFICATION: UNCLASSIFIED > > Thanks. This seems non-intuitive, but regardless, can anyone offer any suggestions to implement what I'm describing using a BroCtl plugin, or am I purely barking up the wrong tree? Thanks. > CLASSIFICATION: UNCLASSIFIED -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5633 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160428/10f12d14/attachment.bin From sven at dreyer-net.de Thu Apr 28 06:39:45 2016 From: sven at dreyer-net.de (Sven Dreyer) Date: Thu, 28 Apr 2016 15:39:45 +0200 Subject: [Bro] Using workers without SSH possible? In-Reply-To: References: <5720AA4D.8020901@dreyer-net.de> Message-ID: <572212A1.3010906@dreyer-net.de> Am 27.04.2016 um 14:51 schrieb Hosom, Stephen M: > I don't think I would feel comfortable running a cluster in the way you > describe. Generally speaking, you want cluster nodes to all exist on the > same layer 2 network. Thanks Stephen, do you know of a better way to have one bro instance that monitors multiple subnets? Best regards, Sven From sven at dreyer-net.de Thu Apr 28 06:43:10 2016 From: sven at dreyer-net.de (Sven Dreyer) Date: Thu, 28 Apr 2016 15:43:10 +0200 Subject: [Bro] Using workers without SSH possible? In-Reply-To: References: <5720AA4D.8020901@dreyer-net.de> Message-ID: <5722136E.6090705@dreyer-net.de> Glenn, Am 27.04.2016 um 14:57 schrieb Glenn Forbes Fleming Larratt: > Doesn't rsync default to using ssh as its transport? Also, I'm not sure > how using rsync vs. ssh improves things in the face of slow and > unreliable networking between nodes; can you elaborate? I thought of locally collecting bro logs and have a cron job transferring the log file(s) in regular intervals. If the network is down for 5 minutes, no problem, the log files will be transferred the next time the cronjob runs. if you use "rsync -e ssh", rsync uses SSH as transport, that's correct. But rsync has a standalone daemon mode and does not need SSH to be used. Thanks, Sven From sven at dreyer-net.de Thu Apr 28 06:48:36 2016 From: sven at dreyer-net.de (Sven Dreyer) Date: Thu, 28 Apr 2016 15:48:36 +0200 Subject: [Bro] Problem with connections in S1 and SF state In-Reply-To: <5720D162.6020408@gmail.com> References: <564B9060.1080304@dreyer-net.de> <5720A537.1010203@dreyer-net.de> <5720D162.6020408@gmail.com> Message-ID: <572214B4.1020906@dreyer-net.de> Hi Jan, Am 27.04.2016 um 16:49 schrieb Jan Grash?fer: > Do both log lines differ only in receiver/originator? If there are > packets missing in your replayed test it is likely that there is an > issue with capturing the traffic. You are right, they only differ in receiver/originator. Thanks for the hint, but I don't think that packet loss is the problem here, because conn.log says that bro saw the initial connection setup (ShA flags in history field). That should be sufficient to tell who's the originator. I also repeated playback several times with different speeds, the result is reproducible. Thanks, Sven From sven at dreyer-net.de Thu Apr 28 06:54:34 2016 From: sven at dreyer-net.de (Sven Dreyer) Date: Thu, 28 Apr 2016 15:54:34 +0200 Subject: [Bro] Problem with connections in S1 and SF state In-Reply-To: References: <564B9060.1080304@dreyer-net.de> <5720A537.1010203@dreyer-net.de> Message-ID: <5722161A.7060508@dreyer-net.de> Am 27.04.2016 um 16:59 schrieb James Lay: > Read this thread: > > http://thread.gmane.org/gmane.comp.security.detection.bro/9211 > > It might help. Thanks James. In my case, it's not UDP, but TCP. Due to the handshake at the beginning of a TCP connection, initiator and receiver of the connection can be distinguished, and conn.log says that bro saw the initial connection setup (ShA flags in history field). Thanks, Sven From robin at icir.org Thu Apr 28 08:13:59 2016 From: robin at icir.org (Robin Sommer) Date: Thu, 28 Apr 2016 08:13:59 -0700 Subject: [Bro] Using workers without SSH possible? In-Reply-To: <5722136E.6090705@dreyer-net.de> References: <5720AA4D.8020901@dreyer-net.de> <5722136E.6090705@dreyer-net.de> Message-ID: <20160428151359.GA65121@icir.org> Actually BroControl is already using rsync over SSH, but it needs SSH for other stuff as well, as it runs commands on the worker nodes. The rsync is used for transferring the Bro setup over to the workers. The logs on the other hand are sent back via Bro's internal communication, neither SSH nor rsync involved there. Changing any of this remains tricky currently. However, we are planing to switch to a different deployment model eventually where each node maintains its Bro setup itself (so no rsync necessary anymore) and also keeps a persistent broctld running for inter-node communication (so no SSH executing commands anymore). With regards of other approaches to monitor subnets, some folks run a single-machine Bro cluster with multiple interfaces and then send each subnet's traffic to one interface. That can work pretty well in practice, but might not apply to your situation. Robin On Thu, Apr 28, 2016 at 15:43 +0200, Sven Dreyer wrote: > Glenn, > > Am 27.04.2016 um 14:57 schrieb Glenn Forbes Fleming Larratt: > > Doesn't rsync default to using ssh as its transport? Also, I'm not sure > > how using rsync vs. ssh improves things in the face of slow and > > unreliable networking between nodes; can you elaborate? > > I thought of locally collecting bro logs and have a cron job > transferring the log file(s) in regular intervals. If the network is > down for 5 minutes, no problem, the log files will be transferred the > next time the cronjob runs. > > if you use "rsync -e ssh", rsync uses SSH as transport, that's correct. > But rsync has a standalone daemon mode and does not need SSH to be used. > > Thanks, > Sven > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jan.grashoefer at gmail.com Thu Apr 28 15:28:15 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Fri, 29 Apr 2016 00:28:15 +0200 Subject: [Bro] Problem with connections in S1 and SF state In-Reply-To: <572214B4.1020906@dreyer-net.de> References: <564B9060.1080304@dreyer-net.de> <5720A537.1010203@dreyer-net.de> <5720D162.6020408@gmail.com> <572214B4.1020906@dreyer-net.de> Message-ID: <24995e7e-ce04-b2b3-91e0-978175e8c840@gmail.com> Hi Sven, > I also repeated playback several times with different speeds, the result > is reproducible. I ran Bro on your pcap and replayed it into my test environment. I could not reproduce the receiver/originator issue you described. Best regards, Jan From mpselab at gmail.com Thu Apr 28 15:49:35 2016 From: mpselab at gmail.com (M P) Date: Fri, 29 Apr 2016 01:49:35 +0300 Subject: [Bro] Unable to recursively clone Bro Message-ID: Hello all, This afternoon recursively cloning Bro was successful. A couple of hours later, cloning does not successfully complete at the "aux" submodule. Hence dependencies are not met. This occurs on both Bro' git and github. Below is a snippet of the errors: $ git clone --recursive git://git.bro.org/bro ... fatal: reference is not a tree: edbbe445d92cc6a5c2557661195f486b784769db ... fatal: reference is not a tree: cb771a3cf592d46643eea35d206b9f3e1a0758f7 ... ... Unable to checkout 'edbbe445d92cc6a5c2557661195f486b784769db' in submodule path 'aux/binpac' Unable to checkout 'cb771a3cf592d46643eea35d206b9f3e1a0758f7' in submodule path 'aux/bro-aux' Unable to checkout 'b4d1686cdd3f5505e405667b1083e8335cae6928' in submodule path 'aux/broccoli' Unable to checkout '6583b0a84b59a90e671d6405613c35f8502ce023' in submodule path 'aux/broctl' Unable to checkout 'bb3f55f198f9cfd5e545345dd6425dd08ca1d45e' in submodule path 'aux/broker' I'm not sure if it's just me or if there is generic issue somewhere. MP -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160429/a70662af/attachment.html From jules.rogers at gmail.com Thu Apr 28 16:07:08 2016 From: jules.rogers at gmail.com (Jules Rogers) Date: Thu, 28 Apr 2016 18:07:08 -0500 Subject: [Bro] Unable to recursively clone Bro In-Reply-To: References: Message-ID: Hi, I just saw the same errors here too. ~ Jules Rogers On Thu, Apr 28, 2016 at 5:49 PM, M P wrote: > Hello all, > > This afternoon recursively cloning Bro was successful. A couple of hours > later, cloning does not successfully complete at the "aux" submodule. Hence > dependencies are not met. This occurs on both Bro' git and github. Below is > a snippet of the errors: > > $ git clone --recursive git://git.bro.org/bro > > ... > > fatal: reference is not a tree: edbbe445d92cc6a5c2557661195f486b784769db > > ... > > fatal: reference is not a tree: cb771a3cf592d46643eea35d206b9f3e1a0758f7 > > ... > > ... > > Unable to checkout 'edbbe445d92cc6a5c2557661195f486b784769db' in submodule > path 'aux/binpac' > Unable to checkout 'cb771a3cf592d46643eea35d206b9f3e1a0758f7' in submodule > path 'aux/bro-aux' > Unable to checkout 'b4d1686cdd3f5505e405667b1083e8335cae6928' in submodule > path 'aux/broccoli' > Unable to checkout '6583b0a84b59a90e671d6405613c35f8502ce023' in submodule > path 'aux/broctl' > Unable to checkout 'bb3f55f198f9cfd5e545345dd6425dd08ca1d45e' in submodule > path 'aux/broker' > > > I'm not sure if it's just me or if there is generic issue somewhere. > > > MP > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160428/6a90384a/attachment.html From robin at icir.org Thu Apr 28 20:16:21 2016 From: robin at icir.org (Robin Sommer) Date: Thu, 28 Apr 2016 20:16:21 -0700 Subject: [Bro] Unable to recursively clone Bro In-Reply-To: References: Message-ID: <20160429031621.GA49291@icir.org> On Fri, Apr 29, 2016 at 01:49 +0300, M P wrote: > This afternoon recursively cloning Bro was successful. A couple of hours > later, cloning does not successfully complete at the "aux" submodule. Should be fixed now, thanks for reporting. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jules.rogers at gmail.com Thu Apr 28 20:40:39 2016 From: jules.rogers at gmail.com (Jules Rogers) Date: Thu, 28 Apr 2016 22:40:39 -0500 Subject: [Bro] Unable to recursively clone Bro In-Reply-To: <20160429031621.GA49291@icir.org> References: <20160429031621.GA49291@icir.org> Message-ID: Looks good, thanks. ~ Jules Rogers On Thu, Apr 28, 2016 at 10:16 PM, Robin Sommer wrote: > On Fri, Apr 29, 2016 at 01:49 +0300, M P wrote: > > > This afternoon recursively cloning Bro was successful. A couple of hours > > later, cloning does not successfully complete at the "aux" submodule. > > Should be fixed now, thanks for reporting. > > Robin > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160428/82e3e081/attachment.html From mpselab at gmail.com Fri Apr 29 01:02:44 2016 From: mpselab at gmail.com (M P) Date: Fri, 29 Apr 2016 11:02:44 +0300 Subject: [Bro] Unable to recursively clone Bro In-Reply-To: <20160429031621.GA49291@icir.org> References: <20160429031621.GA49291@icir.org> Message-ID: Great, thank you Robin. MP On Friday, April 29, 2016, Robin Sommer wrote: > On Fri, Apr 29, 2016 at 01:49 +0300, M P wrote: > > > This afternoon recursively cloning Bro was successful. A couple of hours > > later, cloning does not successfully complete at the "aux" submodule. > > Should be fixed now, thanks for reporting. > > Robin > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * > www.icir.org/robin > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160429/35277d1f/attachment-0001.html From martin.liras at gmail.com Fri Apr 29 05:19:59 2016 From: martin.liras at gmail.com (Luis Martin) Date: Fri, 29 Apr 2016 14:19:59 +0200 Subject: [Bro] Global array in context? Message-ID: Hi, I'm trying to develop an application level analyzer. My protocol has a request/response architectutrte, and the request/response message structure depends on a Function Code. This Function Code is sent in the request message, but NOT in the response message. So, when I receive a response I don't know if it belongs to the request A or the request B. The only way to know that is checking the transport level ID. So, I need to maintain any kind of array that relate transport id values and function codes. My question is how to do that. I need to read and write that global value in the MyProt-protocol.pac file. But I don't know how to maintain a global variable in binpac. I've been trying to do it within the $context, but I don't know how to write a value in a type added to the context. Any idea? Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160429/077d15c8/attachment.html From martin.liras at gmail.com Fri Apr 29 07:21:10 2016 From: martin.liras at gmail.com (Luis Martin) Date: Fri, 29 Apr 2016 16:21:10 +0200 Subject: [Bro] Global array in context? In-Reply-To: References: Message-ID: Hi all, I will repond to myself. I finally got it to work modifying the connection class and adding the array: refine connection UmasTCP_Conn += { %member{ int previous_fcs[256]; %} }; Adding a RESPONSE type that call to a function to recover the FC data from the array: type XXX{ (...) } &let { (...) ufc: uint8 = $context.connection.get_Previous_FC(header.Transport_id); }; And writing two functions in the analyzer.pac file: refine connection MyProtTCP_Conn += { (...) function get_Previous_FC(Transport_id: int): int %{ return previous_fcs[tid%256]; %} function SetTID_FC(transport_id:int, ufc:int): bool %{ previous_fcs[transport_id%256]=ufc; return true; %} The second one is called every time a message event is detected: function deliver_message(header: BinPAC_TCP_Header, MYPROTheader: BinPAC_MYPROT_header): bool %{ if ( ::MyProt_message ) { connection()->SetTID_FC(${header.transport_id}, ${MYPROTheader.myprot_fc}); Works fine. Thanks anyway! 2016-04-29 14:19 GMT+02:00 Luis Martin : > Hi, > > I'm trying to develop an application level analyzer. > > My protocol has a request/response architectutrte, and the > request/response message structure depends on a Function Code. > > This Function Code is sent in the request message, but NOT in the response > message. > > So, when I receive a response I don't know if it belongs to the request A > or the request B. > > The only way to know that is checking the transport level ID. > > So, I need to maintain any kind of array that relate transport id values > and function codes. > > My question is how to do that. > > I need to read and write that global value in the MyProt-protocol.pac > file. But I don't know how to maintain a global variable in binpac. > > I've been trying to do it within the $context, but I don't know how to > write a value in a type added to the context. > > Any idea? > > Thank you! > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160429/fa51a807/attachment.html From vallentin at icir.org Fri Apr 29 12:57:45 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Fri, 29 Apr 2016 12:57:45 -0700 Subject: [Bro] Experimental Bro community chat - Gitter Message-ID: <20160429195745.GN1304@samurai.ICIR.org> We're in the process (re)evaluating our team/dev communication and would like to loop you, the community, in to this process. We just started trying out Gitter: - https://gitter.im/bro/bro (general Bro discussion) - https://gitter.im/bro/broker (Broker-specific discussion) In addition to the web interface, Gitter has an IRC bridge and native clients. We do not want to compromise our current IRC outlet, #bro on Freenode. Rather, we see our Gitter test drive as a complimentary experiment to allow a broader audience to tune in, and everything that develops from there we just take organically. Feel free to swing by and let us know what you think, Matthias From damian.barlow at gmail.com Sat Apr 30 12:36:56 2016 From: damian.barlow at gmail.com (Damian Barlow) Date: Sun, 1 May 2016 07:36:56 +1200 Subject: [Bro] SSLv3 Installation errors installingBro on Raspberry Pi 2 running Arch Linux Message-ID: Hi There. Hope this is ok to ask this here...? SSLv3 Installation errors installing bro on Raspberry Pi 2 running Arch Linux Versions: Archlinux: 4.4.8-2-ARCH (latest 11:04:2016) Bro: Both 2.4.1 and 2.3.2 tested Broccoli: 1.97 packaged with Bro 2.4.1 When running the following order of install commands for Bro ./configure make *First Error: *Which I seem to get past ...but not sure if it contribute to the second error. /home/bro/bro-2.3.2/src/ChunkedIO.cc:692:34: error: 'SSLv3_method' was not declared in this scope. *Workaround:* Editing using nano the file ChunkedIO.cc, I am able to change the SSL version from 3 to 23 and rerun make and it will continue past this section. *Second Error:* It then fails at [ 95%] Linking C executable brohose ../src/libbroccoli.so.5.1.0: undefined reference to `SSLv3_method' collect2: error: ld returned 1 exit status aux/broccoli/test/CMakeFiles/brohose.dir/build.make:98: recipe for target 'aux/broccoli/test/brohose' failed This looks more like a compiled file than a script and if I edit it the MAKE the process fails to read the file. I do see at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=804104 there is a bug reported fixed: Found in version broccoli/1.97-2 Fixed in version broccoli/1.97-3 I've found broccoli folder with 1.97-3 downloads for Ubuntu but not sue if and how it can be incorporated to the MAKE process. http://www10.frugalware.org/pub/linux/ftp.ubuntu.com/ubuntu/pool/universe/b/broccoli/ Looking for advice Kind Regards Damian Barlow -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160501/89c92b24/attachment.html From damian.barlow at gmail.com Sat Apr 30 19:06:47 2016 From: damian.barlow at gmail.com (Damian Barlow) Date: Sun, 1 May 2016 14:06:47 +1200 Subject: [Bro] SSLv3 Installation errors installingBro on Raspberry Pi 2 running Arch Linux In-Reply-To: References: Message-ID: I've just found a patch from the bro archives which I missed earlier. I will see if I can work out how to use it... Can anyone advise how to install? https://bro-tracker.atlassian.net/browse/BIT-1499 > On 1/05/2016, at 7:36 AM, Damian Barlow wrote: > > Hi There. > > Hope this is ok to ask this here...? > > SSLv3 Installation errors installing bro on Raspberry Pi 2 running Arch Linux > > Versions: > Archlinux: 4.4.8-2-ARCH (latest 11:04:2016) > Bro: Both 2.4.1 and 2.3.2 tested > Broccoli: 1.97 packaged with Bro 2.4.1 > > When running the following order of install commands for Bro > > ./configure > make > > First Error: Which I seem to get past > ...but not sure if it contribute to the second error. > /home/bro/bro-2.3.2/src/ChunkedIO.cc:692:34: error: 'SSLv3_method' was not declared in this scope. > Workaround: > Editing using nano the file ChunkedIO.cc, I am able to change the SSL version from 3 to 23 and rerun make and it will continue past this section. > > Second Error: > It then fails at > [ 95%] Linking C executable brohose > ../src/libbroccoli.so.5.1.0: undefined reference to `SSLv3_method' > collect2: error: ld returned 1 exit status > aux/broccoli/test/CMakeFiles/brohose.dir/build.make:98: recipe for target 'aux/broccoli/test/brohose' failed > > This looks more like a compiled file than a script and if I edit it the MAKE the process fails to read the file. > > > I do see at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=804104 there is a bug reported fixed: > Found in version broccoli/1.97-2 > Fixed in version broccoli/1.97-3 > > I've found broccoli folder with 1.97-3 downloads for Ubuntu but not sue if and how it can be incorporated to the MAKE process. > http://www10.frugalware.org/pub/linux/ftp.ubuntu.com/ubuntu/pool/universe/b/broccoli/ > > Looking for advice > > Kind Regards > > Damian Barlow > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160501/1d1eb8f7/attachment.html From damian.barlow at gmail.com Sat Apr 30 19:31:07 2016 From: damian.barlow at gmail.com (Damian Barlow) Date: Sun, 1 May 2016 14:31:07 +1200 Subject: [Bro] SSLv3 Installation errors installingBro on Raspberry Pi 2 running Arch Linux In-Reply-To: References: Message-ID: Sorry... Rookie mistake. I found the patches on my mobile so didn't investigate further before sending the second email. Now home I can see that I can edit another file mentioned in the patches, manually. I've done that now and the install has finished. One positive is that I've learned to use the archive repository. Will continue on now with further configuring. Regards > On 1/05/2016, at 2:06 PM, Damian Barlow wrote: > > I've just found a patch from the bro archives which I missed earlier. > I will see if I can work out how to use it... > Can anyone advise how to install? > > https://bro-tracker.atlassian.net/browse/BIT-1499 > >> On 1/05/2016, at 7:36 AM, Damian Barlow wrote: >> >> Hi There. >> >> Hope this is ok to ask this here...? >> >> SSLv3 Installation errors installing bro on Raspberry Pi 2 running Arch Linux >> >> Versions: >> Archlinux: 4.4.8-2-ARCH (latest 11:04:2016) >> Bro: Both 2.4.1 and 2.3.2 tested >> Broccoli: 1.97 packaged with Bro 2.4.1 >> >> When running the following order of install commands for Bro >> >> ./configure >> make >> >> First Error: Which I seem to get past >> ...but not sure if it contribute to the second error. >> /home/bro/bro-2.3.2/src/ChunkedIO.cc:692:34: error: 'SSLv3_method' was not declared in this scope. >> Workaround: >> Editing using nano the file ChunkedIO.cc, I am able to change the SSL version from 3 to 23 and rerun make and it will continue past this section. >> >> Second Error: >> It then fails at >> [ 95%] Linking C executable brohose >> ../src/libbroccoli.so.5.1.0: undefined reference to `SSLv3_method' >> collect2: error: ld returned 1 exit status >> aux/broccoli/test/CMakeFiles/brohose.dir/build.make:98: recipe for target 'aux/broccoli/test/brohose' failed >> >> This looks more like a compiled file than a script and if I edit it the MAKE the process fails to read the file. >> >> >> I do see at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=804104 there is a bug reported fixed: >> Found in version broccoli/1.97-2 >> Fixed in version broccoli/1.97-3 >> >> I've found broccoli folder with 1.97-3 downloads for Ubuntu but not sue if and how it can be incorporated to the MAKE process. >> http://www10.frugalware.org/pub/linux/ftp.ubuntu.com/ubuntu/pool/universe/b/broccoli/ >> >> Looking for advice >> >> Kind Regards >> >> Damian Barlow >> >> >> >> >> >> >> >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160501/939965f8/attachment-0001.html From tgdesrochers at gmail.com Sun Apr 17 07:35:38 2016 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Sun, 17 Apr 2016 14:35:38 -0000 Subject: [Bro] [bro] misp2bro In-Reply-To: References: Message-ID: I've attached the error log and the xml. I don't see any issues with the XML and the error log just shows that it started processing the XML. The script errors out after beginning to process the XML so thats why I assume there are no more entries in the log. Thanks for the pointer to the PyMISP. I will look into it On Sun, Apr 17, 2016 at 10:28 AM, David Andr? wrote: > Is there an error message in the xml file? > If yes, could you post it? > > If you want to write your own script to download IOCs, there is the > PyMISP library @ https://github.com/MISP/PyMISP/ > This library is really great because it abstracts most of the details > needed to create a script for interacting with a MISP instance. > > Then you can just grep your bro logs, or generate bro IOCs lists that > can be used to match. > > On Sun, Apr 17, 2016 at 6:19 AM, Tim Desrochers > wrote: > > Anyone using MISP? I installed MISP as a test and it seems pretty > useful. > > What I can't seem to get working is the misp2bro script written to export > > indicators in MISP to bro format. > > > > https://github.com/unusedPhD/misp2bro > > > > When I run the script it appears to crash and give the error: > > Traceback (most recent call last): > > File "misp2bro.py", line 288, in > > if makeBroFiles(parseXML(EXPORT_FILE)): > > File "misp2bro.py", line 168, in makeBroFiles > > if int(event.find('attribute_count').text): > > AttributeError: 'NoneType' object has no attribute 'text' > > > > If I run it again there is no crash but that is because the md5 it > generates > > matches the previous hash so no action is taken on the downloaded xml. > > > > Has anyone used this, I could use a hand getting it working. > > > > Thanks > > Tim > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160417/5e712ba2/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: misp2bro.log Type: application/octet-stream Size: 351 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160417/5e712ba2/attachment-0001.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: misp-export.xml Type: text/xml Size: 141039 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160417/5e712ba2/attachment-0001.xml