From josh.guild at morphick.com Fri Jul 1 06:27:47 2016 From: josh.guild at morphick.com (Josh Guild) Date: Fri, 01 Jul 2016 13:27:47 +0000 Subject: [Bro] rdp.log result column Message-ID: Hi all, I have a quick question on the different entries for the "result" column in the rdp.log. What's the difference between an "encrypted" v. "Success RDP" result and is there a source with explanations of different results? My Google-Fu is failing :) Any help would be much obliged, thanks! Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160701/53bf82f2/attachment.html From liburdi.joshua at gmail.com Fri Jul 1 06:32:40 2016 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Fri, 1 Jul 2016 09:32:40 -0400 Subject: [Bro] rdp.log result column In-Reply-To: References: Message-ID: Success means that the RDP server successfully accepted the RDP client's setup parameters. (Note that it doesn't mean the RDP connection was successful.) Encrypted means that the RDP session setup was already encrypted and the analyzer can't determine the result. IIRC if the result is encrypted, you will have little to no metadata in the log entry-- maybe just a cookie value. Josh On Fri, Jul 1, 2016 at 9:27 AM, Josh Guild wrote: > Hi all, > > I have a quick question on the different entries for the "result" column > in the rdp.log. > > What's the difference between an "encrypted" v. "Success RDP" result and > is there a source with explanations of different results? My Google-Fu is > failing :) > > Any help would be much obliged, thanks! > > Josh > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160701/a2b1b52e/attachment.html From josh.guild at morphick.com Fri Jul 1 06:38:37 2016 From: josh.guild at morphick.com (Josh Guild) Date: Fri, 01 Jul 2016 13:38:37 +0000 Subject: [Bro] rdp.log result column In-Reply-To: References: Message-ID: Yep, that's what it looks like. On the encrypted sessions it just has the cookie, result, and security_protocol value. Is there a way to see if the connection was actually established and successful? (vice just accepting the setup params) Just enabled the rdp.log and getting used to reading it. Ha. Thanks a bunch for the help! On Fri, Jul 1, 2016 at 9:33 AM Josh Liburdi wrote: > Success means that the RDP server successfully accepted the RDP client's > setup parameters. (Note that it doesn't mean the RDP connection was > successful.) Encrypted means that the RDP session setup was already > encrypted and the analyzer can't determine the result. IIRC if the result > is encrypted, you will have little to no metadata in the log entry-- maybe > just a cookie value. > > Josh > > On Fri, Jul 1, 2016 at 9:27 AM, Josh Guild > wrote: > >> Hi all, >> >> I have a quick question on the different entries for the "result" column >> in the rdp.log. >> >> What's the difference between an "encrypted" v. "Success RDP" result and >> is there a source with explanations of different results? My Google-Fu is >> failing :) >> >> Any help would be much obliged, thanks! >> >> Josh >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160701/616338d0/attachment.html From liburdi.joshua at gmail.com Fri Jul 1 06:52:19 2016 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Fri, 1 Jul 2016 09:52:19 -0400 Subject: [Bro] rdp.log result column In-Reply-To: References: Message-ID: <991A1A2D-B75C-49E5-ADB3-41898C3E97F1@gmail.com> Unfortunately there's no way to prove an RDP connection was established using Bro. You could possibly infer it from the length of the connection and the amount of bytes transferred, but I wouldn't stake your life on that. :) Your best bet at verifying establishment is to pull authentication records from the endpoint in question. You could also decrypt the RDP session if you have full packet capture and the private key using this method: http://www.contextis.com/resources/blog/rdp-replay-code-release/ Josh Sent from my iPhone > On Jul 1, 2016, at 9:38 AM, Josh Guild wrote: > > Yep, that's what it looks like. On the encrypted sessions it just has the cookie, result, and security_protocol value. > Is there a way to see if the connection was actually established and successful? (vice just accepting the setup params) > > Just enabled the rdp.log and getting used to reading it. Ha. > > Thanks a bunch for the help! > >> On Fri, Jul 1, 2016 at 9:33 AM Josh Liburdi wrote: >> Success means that the RDP server successfully accepted the RDP client's setup parameters. (Note that it doesn't mean the RDP connection was successful.) Encrypted means that the RDP session setup was already encrypted and the analyzer can't determine the result. IIRC if the result is encrypted, you will have little to no metadata in the log entry-- maybe just a cookie value. >> >> Josh >> >>> On Fri, Jul 1, 2016 at 9:27 AM, Josh Guild wrote: >> >>> Hi all, >>> >>> I have a quick question on the different entries for the "result" column in the rdp.log. >>> >>> What's the difference between an "encrypted" v. "Success RDP" result and is there a source with explanations of different results? My Google-Fu is failing :) >>> >>> Any help would be much obliged, thanks! >>> >>> Josh >>> >> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160701/bb1603d3/attachment.html From josh.guild at morphick.com Fri Jul 1 06:58:06 2016 From: josh.guild at morphick.com (Josh Guild) Date: Fri, 01 Jul 2016 13:58:06 +0000 Subject: [Bro] rdp.log result column In-Reply-To: <991A1A2D-B75C-49E5-ADB3-41898C3E97F1@gmail.com> References: <991A1A2D-B75C-49E5-ADB3-41898C3E97F1@gmail.com> Message-ID: Sweet, we were thinking the same thing about bytes and connection length. Glad to know we weren't far off. Unfortunately, we don't have access to the endpoints right now but we can reach out to the customer and see. Full pcaps exist as well but no private key (that I know of). Thanks for the quick answers! On Fri, Jul 1, 2016 at 9:52 AM Josh Liburdi wrote: > Unfortunately there's no way to prove an RDP connection was established > using Bro. You could possibly infer it from the length of the connection > and the amount of bytes transferred, but I wouldn't stake your life on > that. :) > > Your best bet at verifying establishment is to pull authentication records > from the endpoint in question. You could also decrypt the RDP session if > you have full packet capture and the private key using this method: > http://www.contextis.com/resources/blog/rdp-replay-code-release/ > > Josh > > Sent from my iPhone > > On Jul 1, 2016, at 9:38 AM, Josh Guild wrote: > > Yep, that's what it looks like. On the encrypted sessions it just has the > cookie, result, and security_protocol value. > Is there a way to see if the connection was actually established and > successful? (vice just accepting the setup params) > > Just enabled the rdp.log and getting used to reading it. Ha. > > Thanks a bunch for the help! > > On Fri, Jul 1, 2016 at 9:33 AM Josh Liburdi > wrote: > >> Success means that the RDP server successfully accepted the RDP client's >> setup parameters. (Note that it doesn't mean the RDP connection was >> successful.) Encrypted means that the RDP session setup was already >> encrypted and the analyzer can't determine the result. IIRC if the result >> is encrypted, you will have little to no metadata in the log entry-- maybe >> just a cookie value. >> >> Josh >> >> On Fri, Jul 1, 2016 at 9:27 AM, Josh Guild >> wrote: >> >>> Hi all, >>> >>> I have a quick question on the different entries for the "result" column >>> in the rdp.log. >>> >>> What's the difference between an "encrypted" v. "Success RDP" result and >>> is there a source with explanations of different results? My Google-Fu is >>> failing :) >>> >>> Any help would be much obliged, thanks! >>> >>> Josh >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160701/a749af63/attachment-0001.html From liburdi.joshua at gmail.com Fri Jul 1 07:02:12 2016 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Fri, 1 Jul 2016 10:02:12 -0400 Subject: [Bro] rdp.log result column In-Reply-To: References: <991A1A2D-B75C-49E5-ADB3-41898C3E97F1@gmail.com> Message-ID: Happy to help! If you all think of an alternate way to infer the establishment, I'd be curious to hear it. Josh On Fri, Jul 1, 2016 at 9:58 AM, Josh Guild wrote: > Sweet, we were thinking the same thing about bytes and connection length. > Glad to know we weren't far off. > Unfortunately, we don't have access to the endpoints right now but we can > reach out to the customer and see. > Full pcaps exist as well but no private key (that I know of). > > Thanks for the quick answers! > > On Fri, Jul 1, 2016 at 9:52 AM Josh Liburdi > wrote: > >> Unfortunately there's no way to prove an RDP connection was established >> using Bro. You could possibly infer it from the length of the connection >> and the amount of bytes transferred, but I wouldn't stake your life on >> that. :) >> >> Your best bet at verifying establishment is to pull authentication >> records from the endpoint in question. You could also decrypt the RDP >> session if you have full packet capture and the private key using this >> method: http://www.contextis.com/resources/blog/rdp-replay-code-release/ >> >> Josh >> >> Sent from my iPhone >> >> On Jul 1, 2016, at 9:38 AM, Josh Guild wrote: >> >> Yep, that's what it looks like. On the encrypted sessions it just has the >> cookie, result, and security_protocol value. >> Is there a way to see if the connection was actually established and >> successful? (vice just accepting the setup params) >> >> Just enabled the rdp.log and getting used to reading it. Ha. >> >> Thanks a bunch for the help! >> >> On Fri, Jul 1, 2016 at 9:33 AM Josh Liburdi >> wrote: >> >>> Success means that the RDP server successfully accepted the RDP client's >>> setup parameters. (Note that it doesn't mean the RDP connection was >>> successful.) Encrypted means that the RDP session setup was already >>> encrypted and the analyzer can't determine the result. IIRC if the result >>> is encrypted, you will have little to no metadata in the log entry-- maybe >>> just a cookie value. >>> >>> Josh >>> >>> On Fri, Jul 1, 2016 at 9:27 AM, Josh Guild >>> wrote: >>> >>>> Hi all, >>>> >>>> I have a quick question on the different entries for the "result" >>>> column in the rdp.log. >>>> >>>> What's the difference between an "encrypted" v. "Success RDP" result >>>> and is there a source with explanations of different results? My Google-Fu >>>> is failing :) >>>> >>>> Any help would be much obliged, thanks! >>>> >>>> Josh >>>> >>>> _______________________________________________ >>>> Bro mailing list >>>> bro at bro-ids.org >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160701/70979770/attachment.html From josh.guild at morphick.com Fri Jul 1 07:15:52 2016 From: josh.guild at morphick.com (Josh Guild) Date: Fri, 01 Jul 2016 14:15:52 +0000 Subject: [Bro] rdp.log result column In-Reply-To: References: <991A1A2D-B75C-49E5-ADB3-41898C3E97F1@gmail.com> Message-ID: Will do! On Fri, Jul 1, 2016 at 10:02 AM Josh Liburdi wrote: > Happy to help! If you all think of an alternate way to infer the > establishment, I'd be curious to hear it. > > Josh > > On Fri, Jul 1, 2016 at 9:58 AM, Josh Guild > wrote: > >> Sweet, we were thinking the same thing about bytes and connection length. >> Glad to know we weren't far off. >> Unfortunately, we don't have access to the endpoints right now but we can >> reach out to the customer and see. >> Full pcaps exist as well but no private key (that I know of). >> >> Thanks for the quick answers! >> >> On Fri, Jul 1, 2016 at 9:52 AM Josh Liburdi >> wrote: >> >>> Unfortunately there's no way to prove an RDP connection was established >>> using Bro. You could possibly infer it from the length of the connection >>> and the amount of bytes transferred, but I wouldn't stake your life on >>> that. :) >>> >>> Your best bet at verifying establishment is to pull authentication >>> records from the endpoint in question. You could also decrypt the RDP >>> session if you have full packet capture and the private key using this >>> method: http://www.contextis.com/resources/blog/rdp-replay-code-release/ >>> >>> Josh >>> >>> Sent from my iPhone >>> >>> On Jul 1, 2016, at 9:38 AM, Josh Guild wrote: >>> >>> Yep, that's what it looks like. On the encrypted sessions it just has >>> the cookie, result, and security_protocol value. >>> Is there a way to see if the connection was actually established and >>> successful? (vice just accepting the setup params) >>> >>> Just enabled the rdp.log and getting used to reading it. Ha. >>> >>> Thanks a bunch for the help! >>> >>> On Fri, Jul 1, 2016 at 9:33 AM Josh Liburdi >>> wrote: >>> >>>> Success means that the RDP server successfully accepted the RDP >>>> client's setup parameters. (Note that it doesn't mean the RDP connection >>>> was successful.) Encrypted means that the RDP session setup was already >>>> encrypted and the analyzer can't determine the result. IIRC if the result >>>> is encrypted, you will have little to no metadata in the log entry-- maybe >>>> just a cookie value. >>>> >>>> Josh >>>> >>>> On Fri, Jul 1, 2016 at 9:27 AM, Josh Guild >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I have a quick question on the different entries for the "result" >>>>> column in the rdp.log. >>>>> >>>>> What's the difference between an "encrypted" v. "Success RDP" result >>>>> and is there a source with explanations of different results? My Google-Fu >>>>> is failing :) >>>>> >>>>> Any help would be much obliged, thanks! >>>>> >>>>> Josh >>>>> >>>>> _______________________________________________ >>>>> Bro mailing list >>>>> bro at bro-ids.org >>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>>>> >>>> >>>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160701/fc06dc94/attachment.html From fatema.bannatwala at gmail.com Fri Jul 1 14:28:49 2016 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Fri, 1 Jul 2016 17:28:49 -0400 Subject: [Bro] Notice.log logs a Password_Guessing attempt but no logs in conn.log Message-ID: Hi, So I had a weird situation at work today. The notice.log file logged an IP for "SSH::Password_Guessing" with note as "50.123.48.2 appears to be guessing SSH passwords (seen in 53 connections)". But when I check conn.log file during that time period and grep that IP, I just see single ssh established connection from that IP. I was assuming to get 53 bad ssh connections logged in conn.lo file. What am I missing here? How can I confirm whether that IP was actually doing a SSH password guessing attempt? Thanks, Fatema. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160701/f578495a/attachment.html From raj at bivio.net Fri Jul 1 15:56:58 2016 From: raj at bivio.net (Raj Srinivasan) Date: Fri, 1 Jul 2016 22:56:58 +0000 Subject: [Bro] Question on "BROBASE" Message-ID: Hello, We have a test environment where we want to run multiple bro instances. Thus, we would like to have /usr/local/bro1, /usr/local/bro2, etc. Is there a way I can change the "BROBASE" variable for copies of the first instance, or do I have to generate the other instances with a "make install" for each? "BROBASE" is an internal variable, and the documentation seems to discourage changing it. This seems to be the cleanest way for us to ensure that we can configure each instance totally independently of the others. Thanks, Raj -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160701/ef36f2dd/attachment.html From aidaros.dev at gmail.com Sat Jul 2 06:23:46 2016 From: aidaros.dev at gmail.com (Hashem Alaidaros) Date: Sat, 2 Jul 2016 21:23:46 +0800 Subject: [Bro] How to Set N-byte of Payload to be Processed by Bro? Message-ID: Hi everyone, I use signature framework in Bro. I want Bro to capture and processes only the first N-bytes of each packet received. In other words, only the first N-bytes will be compared with the signature. How to do that? Aidaros -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160702/522a7b6e/attachment.html From slawek at rozbicki.eu Mon Jul 4 02:26:28 2016 From: slawek at rozbicki.eu (=?UTF-8?Q?S=C5=82awek_Rozbicki?=) Date: Mon, 4 Jul 2016 11:26:28 +0200 Subject: [Bro] missing p$descr in cluster configuration Message-ID: Hi, I'm using Bro 2.4.1 in cluster mode with 10 workers spread around 2 worker machines + proxy and manager node. Everything seems to work fine except intel framework (especially criticalstack client intelligence feed). There is error in Reporter.log that might correspond this issue: Reporter::ERROR no such index (Cluster::nodes[Intel::p$descr]) /opt/bro/share/bro/base/frameworks/intel/./cluster.bro thats because when 'event remote_connection_handshake_done(p: event_peer)' is triggered, event_peer lacks most of fields, in my case 'p' looks like this: [id=10012, host=10.151.0.20, p=45783/tcp, is_local=F, descr=, class=control] and I have no clue what is this peer (maybe a CONTROL type?) That error is triggered every time I execute 'broctl status'. Intel framework is loaded: # broctl scripts | grep intel /opt/bro/share/bro/base/frameworks/intel/__load__.bro /opt/bro/share/bro/base/frameworks/intel/main.bro /opt/bro/share/bro/base/frameworks/intel/cluster.bro /opt/bro/share/bro/base/frameworks/intel/input.bro ... # broctl print Intel::read_files manager manager Intel::read_files = { /opt/critical-stack/frameworks/intel/master-public.bro.dat } # head /opt/critical-stack/frameworks/intel/master-public.bro.dat #fields indicator indicator_type meta.source meta.do_notice unionsystems.com Intel::DOMAIN from http://mirror1.malwaredomains.com/files/domains.txt,phishing via intel.criticalstack.com F bmwclubavto.ru/p/p/Conflict/Resolution/www/PP-009-176-639-801/7mXF56vGr5onWk5syWMn2wU/ISCNjQqH5WKey51F/ Intel::URL from http://www.phishtank.com/phish_detail.php?phish_id=4112321 via intel.criticalstack.com F and feed format seems to be alright. Yet I have the second issue: 0.000000 Reporter::ERROR /opt/critical-stack/frameworks/intel/master-public.bro.dat/Input::READER_ASCII: could not read first line (empty) which is also strange. This results in no intel.log even though there is my query logged in dns.log, and host value present in intel feed. Is this bro bug or my deployment misconfiguration? What would be the best way to debug this issue? -- BR S?awomir Rozbicki -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160704/a3cc8e2d/attachment.html From maxime.lambert at insa-cvl.fr Mon Jul 4 03:53:52 2016 From: maxime.lambert at insa-cvl.fr (Maxime Lambert) Date: Mon, 4 Jul 2016 12:53:52 +0200 (CEST) Subject: [Bro] Host key verification failed. In-Reply-To: <128183266.120541.1467628925390.JavaMail.zimbra@insa-cvl.fr> Message-ID: <1188696904.124843.1467629632665.JavaMail.zimbra@insa-cvl.fr> Hi ! I don't understand why Bro didn't work,could you help me ? Ubuntu at Ubuntu:~/bro-2.4.1$ sudo /opt/bro/bin/broctl Warning: state database needs updating (run the broctl "deploy" command) Welcome to BroControl 1.4 Type "help" for help. [BroControl] > install removing old policies in /opt/bro/spool/installed-scripts-do-not-touch/site ... removing old policies in /opt/bro/spool/installed-scripts-do-not-touch/auto ... creating policy directories ... installing site policies ... generating standalone-layout.bro ... generating local-networks.bro ... generating broctl-config.bro ... generating broctl-config.sh ... updating nodes ... Host key verification failed. Error: cannot create (some of the) directories /opt/bro,/var/log/bro,/opt/bro/spool,/opt/bro/spool/tmp on node bro I've installed Bro from tar.gz with : Ubuntu at Ubuntu:~/bro-2.4.1$ ./configure --prefix=/opt/bro --logdir=/var/log/bro Ubuntu at Ubuntu:~/bro-2.4.1$ make Ubuntu at Ubuntu:~/bro-2.4.1$ sudo make install And my conf is : networks.cfg W.W.W.W/24 Private IP space Y.Y.Y.Y/24 Private IP space nodes.cfg [bro] type=standalone host=localhost interface=eth1 Thank you for your help, Regards, Maxime Lambert -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160704/12d35295/attachment.html From slawek at rozbicki.eu Mon Jul 4 04:16:07 2016 From: slawek at rozbicki.eu (=?UTF-8?Q?S=C5=82awek_Rozbicki?=) Date: Mon, 4 Jul 2016 13:16:07 +0200 Subject: [Bro] missing p$descr in cluster configuration In-Reply-To: References: Message-ID: SOLVED - it turned out Intel::match events were not generated on workers because no 'policy/frameworks/intel/seen' script was loaded. Still lack of Intel::p$descr in remote_connection_handshake_done event is a mystery to me (possibly a bro bug). 2016-07-04 11:26 GMT+02:00 S?awek Rozbicki : > Hi, > I'm using Bro 2.4.1 in cluster mode with 10 workers spread around 2 worker > machines + proxy and manager node. Everything seems to work fine except > intel framework (especially criticalstack client intelligence feed). There > is error in Reporter.log that might correspond this issue: > > Reporter::ERROR no such index (Cluster::nodes[Intel::p$descr]) > /opt/bro/share/bro/base/frameworks/intel/./cluster.bro > > thats because when 'event remote_connection_handshake_done(p: event_peer)' > is triggered, event_peer lacks most of fields, in my case 'p' looks like > this: > > [id=10012, host=10.151.0.20, p=45783/tcp, is_local=F, descr=, > class=control] > > and I have no clue what is this peer (maybe a CONTROL type?) > > That error is triggered every time I execute 'broctl status'. > > Intel framework is loaded: > > # broctl scripts | grep intel > /opt/bro/share/bro/base/frameworks/intel/__load__.bro > /opt/bro/share/bro/base/frameworks/intel/main.bro > /opt/bro/share/bro/base/frameworks/intel/cluster.bro > /opt/bro/share/bro/base/frameworks/intel/input.bro > ... > > # broctl print Intel::read_files manager > manager Intel::read_files = { > /opt/critical-stack/frameworks/intel/master-public.bro.dat > } > > # head /opt/critical-stack/frameworks/intel/master-public.bro.dat > #fields indicator indicator_type meta.source meta.do_notice > unionsystems.com Intel::DOMAIN from > http://mirror1.malwaredomains.com/files/domains.txt,phishing via > intel.criticalstack.com F > > bmwclubavto.ru/p/p/Conflict/Resolution/www/PP-009-176-639-801/7mXF56vGr5onWk5syWMn2wU/ISCNjQqH5WKey51F/ > Intel::URL from http://www.phishtank.com/phish_detail.php?phish_id=4112321 > via intel.criticalstack.com F > > and feed format seems to be alright. Yet I have the second issue: > > 0.000000 Reporter::ERROR /opt/critical-stack/frameworks/intel/master-public.bro.dat/Input::READER_ASCII: > could not read first line (empty) > > which is also strange. > > This results in no intel.log even though there is my query logged in > dns.log, and host value present in intel feed. > > Is this bro bug or my deployment misconfiguration? What would be the best > way to debug this issue? > > -- > BR > S?awomir Rozbicki > -- Pozdrawiam S?awomir Rozbicki -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160704/f63ea684/attachment.html From simone.rotondo at aizoongroup.com Tue Jul 5 08:30:15 2016 From: simone.rotondo at aizoongroup.com (Rotondo Simone) Date: Tue, 5 Jul 2016 15:30:15 +0000 Subject: [Bro] Connection lasts huge time Message-ID: <3f8d13efec52440c8c43630d44891128@SRVEX03.aizoon.local> Hi, in my Bro logs, I have some connections that lasts 6 hours and more. Those conns use different services: ------------------------ conn.13:00:00-14:00:00.log.gz:{"ts":1466403937.471482,"uid":"CNLOdrb4ss9hRDDgg","id.orig_h":"XXX.XXX.XXX.XXX","id.orig_p":XXXXX,"id.resp_h":"XXX.XXX.XXX.XXX","id.resp_p":3268,"proto":"tcp","duration":16980.700023,"orig_bytes":299358,"resp_bytes":258817,"conn_state":"S1","local_resp":false,"missed_bytes":58394,"history":"ShADad","orig_pkts":485,"orig_ip_bytes":287130,"resp_pkts":243,"resp_ip_bytes":241795,"tunnel_parents":[],"local_origi":"T4","local_respo":"F4"} ------------------------ conn.14:00:00-15:00:00.log.gz:{"ts":1466404357.492809,"uid":"CA7q9dl7q5ZbTDRXa","id.orig_h":"XXX.XXX.XXX.XXX","id.orig_p":XXXXX,"id.resp_h":" XXX.XXX.XXX.XXX","id.resp_p":443,"proto":"tcp","service":"ssl","duration":22774.467724,"orig_bytes":341462,"resp_bytes":402631,"conn_state":"S1","local_resp":true,"missed_bytes":51675,"history":"ShADda","orig_pkts":921,"orig_ip_bytes":353314,"resp_pkts":2058,"resp_ip_bytes":458288,"tunnel_parents":[],"from_known_services":["SSL"],"local_origi":"T4","local_respo":"T4"} +++++ ssl.08:00:00-09:00:00.log.gz:{"ts":1466404357.495789,"uid":"CA7q9dl7q5ZbTDRXa","id.orig_h":"XXX.XXX.XXX.XXX","id.orig_p":XXXXX,"id.resp_h":"XXX.XXX.XXX.XXX","id.resp_p":443,"version":"TLSv12","cipher":"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","server_name":"mail.xxxxxx.xxx","resumed":true,"established":true} ------------------------ conn.14:00:00-15:00:00.log.gz:{"ts":1466404268.700607,"uid":"Czvp3U1saSEh9powDh","id.orig_h":"XXX.XXX.XXX.XXX","id.orig_p":XXXXX,"id.resp_h":"XXX.XXX.XXX.XXX","id.resp_p":443,"proto":"tcp","service":"ssl","duration":21422.058832,"orig_bytes":2158,"resp_bytes":122049,"conn_state":"RSTO","local_resp":true,"missed_bytes":3254,"history":"ShADdaR","orig_pkts":411,"orig_ip_bytes":18610,"resp_pkts":836,"resp_ip_bytes":152247,"tunnel_parents":[],"from_known_services":["SSL"],"local_origi":"T4","local_respo":"T4"} +++++ ssl.08:00:00-09:00:00.log.gz:{"ts":1466404268.70101,"uid":"Czvp3U1saSEh9powDh","id.orig_h":"XXX.XXX.XXX.XXX","id.orig_p":XXXXX,"id.resp_h":"XXX.XXX.XXX.XXX","id.resp_p":443,"version":"TLSv10","cipher":"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","curve":"secp384r1","server_name":"mail.xxxxx.xxx","resumed":false,"established":true,"cert_chain_fuids":["XXXXXXXXXXXXXXXXX","XXXXXXXXXXXXXXXXXXX"],"client_cert_chain_fuids":[],"subject":"emailAddress=hostmaster at xxxxxx.xx,CN=mail.xxxxxxx.xxx,O=XXXXXXX,...} Have you got any idea about this issue? BR Simone -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160705/a682af1b/attachment.html From jdopheid at illinois.edu Tue Jul 5 09:28:01 2016 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Tue, 5 Jul 2016 16:28:01 +0000 Subject: [Bro] =?utf-8?q?BroCon_=E2=80=9916=3A_Announcing_a_few_speakers?= Message-ID: <354D9603-38DF-461A-8967-D0156BD59E63@illinois.edu> Hello Bro Community, We have a few ?BroCon ?16? speakers and topics to announce: ? Aashish Sharma, LBNL, "How-to-clusterize : Using scan-detection with netcontrol framework as case study" ? Mike "Dop" Dopheide, ESNet, "Notice Correlation and Other Randomness" ? Atomic Mole, "Detecting malicious SSL certificates using Bro" We'll announce more speakers and topics as we receive confirmations. Don't forget to register: https://www.bro.org/community/brocon2016.html#registration Early bird registration expires August 1st. See you in September, The Bro Team ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From dnthayer at illinois.edu Tue Jul 5 11:21:10 2016 From: dnthayer at illinois.edu (Daniel Thayer) Date: Tue, 5 Jul 2016 13:21:10 -0500 Subject: [Bro] Host key verification failed. In-Reply-To: <1188696904.124843.1467629632665.JavaMail.zimbra@insa-cvl.fr> References: <1188696904.124843.1467629632665.JavaMail.zimbra@insa-cvl.fr> Message-ID: <577BFA96.9050400@illinois.edu> In BroControl, what is the output of the "nodes" command? Also, what is the output of the "status" command? On 07/04/2016 05:53 AM, Maxime Lambert wrote: > Hi ! > > > I don't understand why Bro didn't work,could you help me ? > > Ubuntu at Ubuntu:~/bro-2.4.1$ sudo /opt/bro/bin/broctl > Warning: state database needs updating (run the broctl "deploy" command) > > Welcome to BroControl 1.4 > > Type "help" for help. > > [BroControl] > install > removing old policies in > /opt/bro/spool/installed-scripts-do-not-touch/site ... > removing old policies in > /opt/bro/spool/installed-scripts-do-not-touch/auto ... > creating policy directories ... > installing site policies ... > generating standalone-layout.bro ... > generating local-networks.bro ... > generating broctl-config.bro ... > generating broctl-config.sh ... > updating nodes ... > *Host key verification failed.* > *Error: cannot create (some of the) directories > /opt/bro,/var/log/bro,/opt/bro/spool,/opt/bro/spool/tmp on node bro* > From sven at dreyer-net.de Tue Jul 5 11:26:59 2016 From: sven at dreyer-net.de (Sven Dreyer) Date: Tue, 5 Jul 2016 20:26:59 +0200 Subject: [Bro] Host key verification failed. In-Reply-To: <1188696904.124843.1467629632665.JavaMail.zimbra@insa-cvl.fr> References: <1188696904.124843.1467629632665.JavaMail.zimbra@insa-cvl.fr> Message-ID: <577BFBF3.1060009@dreyer-net.de> Am 04.07.2016 um 12:53 schrieb Maxime Lambert: > Ubuntu at Ubuntu:~/bro-2.4.1$ sudo /opt/bro/bin/broctl Did you try unsetting the LANG variable? sudo LANG= /opt/bro/bin/broctl (mind the blank after the "=".) Best regards, Sven From dnthayer at illinois.edu Tue Jul 5 12:30:53 2016 From: dnthayer at illinois.edu (Daniel Thayer) Date: Tue, 5 Jul 2016 14:30:53 -0500 Subject: [Bro] Host key verification failed. In-Reply-To: <577BFBF3.1060009@dreyer-net.de> References: <1188696904.124843.1467629632665.JavaMail.zimbra@insa-cvl.fr> <577BFBF3.1060009@dreyer-net.de> Message-ID: <577C0AED.4050906@illinois.edu> On 07/05/2016 01:26 PM, Sven Dreyer wrote: > Am 04.07.2016 um 12:53 schrieb Maxime Lambert: >> Ubuntu at Ubuntu:~/bro-2.4.1$ sudo /opt/bro/bin/broctl > > Did you try unsetting the LANG variable? > > sudo LANG= /opt/bro/bin/broctl > > (mind the blank after the "=".) > This issue was fixed a long time ago, but I just remembered that the fix was not included in the Bro 2.4.1 release. It will be included in the next release, however. From maxime.lambert at insa-cvl.fr Tue Jul 5 23:23:20 2016 From: maxime.lambert at insa-cvl.fr (Maxime Lambert) Date: Wed, 6 Jul 2016 08:23:20 +0200 (CEST) Subject: [Bro] Host key verification failed. In-Reply-To: <577BFA96.9050400@illinois.edu> References: <1188696904.124843.1467629632665.JavaMail.zimbra@insa-cvl.fr> <577BFA96.9050400@illinois.edu> Message-ID: <832230354.501531.1467786200148.JavaMail.zimbra@insa-cvl.fr> Hi Daniel, That's : [BroControl] > nodes bro - addr=::1 aux_scripts= brobase= count=1 env_vars= ether= host=localhost interface=eth1 lb_interfaces= lb_method= lb_procs= name=bro pin_cpus= test_mykey= type=standalone zone_id= And : [BroControl] > status Getting process status ... Getting peer status ... Name Type Host Status Pid Peers Started bro standalone localhost stopped Best regards, Maxime Lambert ----- Mail original ----- De: "Daniel Thayer" ?: "Maxime Lambert" , bro at bro.org Envoy?: Mardi 5 Juillet 2016 20:21:10 Objet: Re: [Bro] Host key verification failed. In BroControl, what is the output of the "nodes" command? Also, what is the output of the "status" command? On 07/04/2016 05:53 AM, Maxime Lambert wrote: > Hi ! > > > I don't understand why Bro didn't work,could you help me ? > > Ubuntu at Ubuntu:~/bro-2.4.1$ sudo /opt/bro/bin/broctl > Warning: state database needs updating (run the broctl "deploy" command) > > Welcome to BroControl 1.4 > > Type "help" for help. > > [BroControl] > install > removing old policies in > /opt/bro/spool/installed-scripts-do-not-touch/site ... > removing old policies in > /opt/bro/spool/installed-scripts-do-not-touch/auto ... > creating policy directories ... > installing site policies ... > generating standalone-layout.bro ... > generating local-networks.bro ... > generating broctl-config.bro ... > generating broctl-config.sh ... > updating nodes ... > *Host key verification failed.* > *Error: cannot create (some of the) directories > /opt/bro,/var/log/bro,/opt/bro/spool,/opt/bro/spool/tmp on node bro* > From maxime.lambert at insa-cvl.fr Tue Jul 5 23:25:35 2016 From: maxime.lambert at insa-cvl.fr (Maxime Lambert) Date: Wed, 6 Jul 2016 08:25:35 +0200 (CEST) Subject: [Bro] Host key verification failed. In-Reply-To: <577BFBF3.1060009@dreyer-net.de> References: <1188696904.124843.1467629632665.JavaMail.zimbra@insa-cvl.fr> <577BFBF3.1060009@dreyer-net.de> Message-ID: <1791023881.501912.1467786335862.JavaMail.zimbra@insa-cvl.fr> Hi Sven, That's the result : Ubuntu at Ubuntu:~$ sudo LANG= /opt/bro/bin/broctl Warning: state database needs updating (run the broctl "deploy" command) Welcome to BroControl 1.4 Type "help" for help. [BroControl] > deploy checking configurations ... installing ... removing old policies in /opt/bro/spool/installed-scripts-do-not-touch/site ... removing old policies in /opt/bro/spool/installed-scripts-do-not-touch/auto ... creating policy directories ... installing site policies ... generating standalone-layout.bro ... generating local-networks.bro ... generating broctl-config.bro ... generating broctl-config.sh ... updating nodes ... stopping ... bro not running starting ... starting bro ... bro terminated immediately after starting; check output with "diag" [BroControl] > status Getting process status ... Getting peer status ... Name Type Host Status Pid Peers Started bro standalone localhost crashed Best regards, Maxime Lambert ----- Mail original ----- De: "Sven Dreyer" ?: "Maxime Lambert" , bro at bro.org Envoy?: Mardi 5 Juillet 2016 20:26:59 Objet: Re: [Bro] Host key verification failed. Am 04.07.2016 um 12:53 schrieb Maxime Lambert: > Ubuntu at Ubuntu:~/bro-2.4.1$ sudo /opt/bro/bin/broctl Did you try unsetting the LANG variable? sudo LANG= /opt/bro/bin/broctl (mind the blank after the "=".) Best regards, Sven From maxime.lambert at insa-cvl.fr Tue Jul 5 23:28:56 2016 From: maxime.lambert at insa-cvl.fr (Maxime Lambert) Date: Wed, 6 Jul 2016 08:28:56 +0200 (CEST) Subject: [Bro] Host key verification failed. In-Reply-To: <577C0AED.4050906@illinois.edu> References: <1188696904.124843.1467629632665.JavaMail.zimbra@insa-cvl.fr> <577BFBF3.1060009@dreyer-net.de> <577C0AED.4050906@illinois.edu> Message-ID: <513908158.502677.1467786536156.JavaMail.zimbra@insa-cvl.fr> Sorry, I forget : After the sudo LANG= ... command : [BroControl] > diag [bro] Bro 2.4.1 Linux 4.4.0-21-generic No gdb installed. ==== No reporter.log ==== stderr.log fatal error: problem with interface eth1 (eth1: SIOCETHTOOL(ETHTOOL_GET_TS_INFO) ioctl failed: No such device) ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i eth1 -U .status -p broctl -p broctl-live -p standalone -p local -p bro local. bro broctl broctl/standalone broctl/auto ==== .env_vars PATH=/opt/bro/bin:/opt/bro/share/broctl/scripts:/usr/local/sbin:/usr/local/bin:/ usr/sbin:/usr/bin:/sbin:/bin:/opt/bro/bin BROPATH=/opt/bro/spool/installed-scripts-do-not-touch/site::/opt/bro/spool/insta lled-scripts-do-not-touch/auto:/opt/bro/share/bro:/opt/bro/share/bro/policy:/opt /bro/share/bro/site CLUSTER_NODE= ==== .status TERMINATED [atexit] ==== No prof.log ==== No packet_filter.log ==== No loaded_scripts.log Gest regards, Maxime Lambert ----- Mail original ----- De: "Daniel Thayer" ?: "Sven Dreyer" , "Maxime Lambert" , bro at bro.org Envoy?: Mardi 5 Juillet 2016 21:30:53 Objet: Re: [Bro] Host key verification failed. On 07/05/2016 01:26 PM, Sven Dreyer wrote: > Am 04.07.2016 um 12:53 schrieb Maxime Lambert: >> Ubuntu at Ubuntu:~/bro-2.4.1$ sudo /opt/bro/bin/broctl > > Did you try unsetting the LANG variable? > > sudo LANG= /opt/bro/bin/broctl > > (mind the blank after the "=".) > This issue was fixed a long time ago, but I just remembered that the fix was not included in the Bro 2.4.1 release. It will be included in the next release, however. From jazoff at illinois.edu Wed Jul 6 07:13:43 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 6 Jul 2016 14:13:43 +0000 Subject: [Bro] Host key verification failed. In-Reply-To: <513908158.502677.1467786536156.JavaMail.zimbra@insa-cvl.fr> References: <1188696904.124843.1467629632665.JavaMail.zimbra@insa-cvl.fr> <577BFBF3.1060009@dreyer-net.de> <577C0AED.4050906@illinois.edu> <513908158.502677.1467786536156.JavaMail.zimbra@insa-cvl.fr> Message-ID: <47C69343-CC2F-456F-85F2-A12DD1F322B8@illinois.edu> > On Jul 6, 2016, at 2:28 AM, Maxime Lambert wrote: > > ==== stderr.log > fatal error: problem with interface eth1 (eth1: SIOCETHTOOL(ETHTOOL_GET_TS_INFO) ioctl failed: No such device) You have 'interface=eth1' in your node.cfg but you don't actually have an eth1 interface. -- - Justin Azoff From maxime.lambert at insa-cvl.fr Wed Jul 6 07:28:12 2016 From: maxime.lambert at insa-cvl.fr (Maxime Lambert) Date: Wed, 6 Jul 2016 16:28:12 +0200 (CEST) Subject: [Bro] Host key verification failed. In-Reply-To: <47C69343-CC2F-456F-85F2-A12DD1F322B8@illinois.edu> References: <1188696904.124843.1467629632665.JavaMail.zimbra@insa-cvl.fr> <577BFBF3.1060009@dreyer-net.de> <577C0AED.4050906@illinois.edu> <513908158.502677.1467786536156.JavaMail.zimbra@insa-cvl.fr> <47C69343-CC2F-456F-85F2-A12DD1F322B8@illinois.edu> Message-ID: <1673403437.817147.1467815292103.JavaMail.zimbra@insa-cvl.fr> Hi everyone, Mistake, from my interface ! And thanks to sudo LANG= /opt/bro/bin/broctl all is OK ! Thanks you very much ! Best regards, Maxime Lambert ----- Mail original ----- De: "Justin S Azoff" ?: "Maxime Lambert" Cc: "Daniel N Thayer" , bro at bro.org Envoy?: Mercredi 6 Juillet 2016 16:13:43 Objet: Re: [Bro] Host key verification failed. > On Jul 6, 2016, at 2:28 AM, Maxime Lambert wrote: > > ==== stderr.log > fatal error: problem with interface eth1 (eth1: SIOCETHTOOL(ETHTOOL_GET_TS_INFO) ioctl failed: No such device) You have 'interface=eth1' in your node.cfg but you don't actually have an eth1 interface. -- - Justin Azoff From vladg at illinois.edu Wed Jul 6 09:41:09 2016 From: vladg at illinois.edu (Vlad Grigorescu) Date: Wed, 06 Jul 2016 11:41:09 -0500 Subject: [Bro] How to Set N-byte of Payload to be Processed by Bro? In-Reply-To: References: Message-ID: Hi, There's a variable you can redefine for this, snaplen[1]. The module that this variable is in changed from 2.4.1 to the current git master. If you're on 2.4.1 or older, use "snaplen" otherwise use "Pcap::snaplen". If running Bro in standalone mode, you can do something like: > bro -i eth0 Pcap::snaplen=1024 Otherwise, you can add this to site/local.bro: > redef Pcap::snaplen=1024; One thing to note is that this only applies to capturing from live interfaces, and not reading from PCAPs. --Vlad [1] - Hashem Alaidaros writes: > Hi everyone, > I use signature framework in Bro. I want Bro to capture and processes only > the first N-bytes of each packet received. In other words, only the first > N-bytes will be compared with the signature. > How to do that? > Aidaros > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 800 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160706/2895971d/attachment.bin From k2 at korrosivesecurity.com Thu Jul 7 06:19:57 2016 From: k2 at korrosivesecurity.com (K2) Date: Thu, 07 Jul 2016 08:19:57 -0500 Subject: [Bro] Onsite Training Options? Message-ID: <1467897597.2878480.659499673.56DE6795@webmail.messagingengine.com> Anyone have any leads on anybody available for onsite Bro training in the US? From florent.monjalet at gmail.com Thu Jul 7 06:45:00 2016 From: florent.monjalet at gmail.com (Florent Monjalet) Date: Thu, 07 Jul 2016 13:45:00 +0000 Subject: [Bro] NTLM parsing in DCE RPC Message-ID: Hi, As this is my first interaction with the bro community, first things first: thanks a lot for working on such an awesome project. We are currently working on IVRE (https://ivre.rocks, https://github.com/cea-sec/ivre), an open source framework aimed at exploring and querying results from various network related tools (such as nmap scans, bro logs, argus or netflow files), through either a web interface, CLI tools or a python API. As part of this work, I was very interested in Seth's work on SMB, so this mail is about the topic/seth/smb branch. Here again, thanks a lot for the huge work on these protocols. It turns out that for DCE RPC, the NTLM decoding seems broken: the NTLM analyzer is called, but the decoding fails to recognize the message type, and no `ntlm.log` log is produced. It works very well for SMB, though. After debugging a bit, it turned out that the `meta: ASN1EncodingMeta` field of `NTLM_SSP_Token` in `ntlm-protocol.pac` should in fact be part of the underlying GSSAPI layer. Therefore, when directly decoding NTLM without GSSAPI, the fields are shifted and have no meaning. Since SMB uses GSSAPI in the examples I found and DCE RPC did not, NTLM over GSSAPI over SMB worked great, but not NTLM over DCE RPC. The first field of NTLM should actually be the "NTLMSSP\x00" magic (according to: http://davenport.sourceforge.net/ntlm.html#theNtlmMessageHeaderLayout and wireshark dissectors). Moving the `meta` field to the `GSSAPI` layer will allow to properlly decode NTLM over DCE RPC and maybe HTLM NTLM Authentication later on. I am attaching a naive (git) patch to this mail; this fixes the problem for me but might not be the exact way you want to do this. Also, I might just have missed something from the start, maybe the way it is currently done was on purpose. Please tell me if you have any comments on the patch and/or want me to submit it to the bug tracker or on github. Sample captures the NTLM log generation can be tested on: - DCE RPC: Not easy to find an open example capture, but this one is ok https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=mapi.cap (from packet 711 in wireshark). You'll have to register DCE RPC on port 4997 (mapi) in bro. - SMB: I tested on https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=smbtorture.cap.gz but I guess you already have samples. Cheers, Florent -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160707/1b0ea233/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Fix-GSSAPI-mech_token-to-include-ASN1EncodingMeta.patch Type: text/x-patch Size: 3120 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160707/1b0ea233/attachment.bin From seth at icir.org Thu Jul 7 08:39:16 2016 From: seth at icir.org (Seth Hall) Date: Thu, 7 Jul 2016 11:39:16 -0400 Subject: [Bro] NTLM parsing in DCE RPC In-Reply-To: References: Message-ID: > On Jul 7, 2016, at 9:45 AM, Florent Monjalet wrote: > > As part of this work, I was very interested in Seth's work on SMB, so > this mail is about the topic/seth/smb branch. Here again, thanks a lot > for the huge work on these protocols. This is a good time to reach out about that branch. We are preparing to merge it into the master branch soon once we do a bit more review. > The first field of NTLM should actually be the "NTLMSSP\x00" magic > (according to: > http://davenport.sourceforge.net/ntlm.html#theNtlmMessageHeaderLayout > and wireshark dissectors). Moving the `meta` field to the `GSSAPI` > layer will allow to properlly decode NTLM over DCE RPC and maybe HTLM > NTLM Authentication later on. Ugh, I'm not surprised that there is yet another case where this is done wrong. I'll review the change you proposed and the look at the pcap. > - DCE RPC: Not easy to find an open example capture, but this one > is ok > https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=mapi.cap > (from packet 711 in wireshark). You'll have to register DCE RPC > on port 4997 (mapi) in bro. Just for clarity, you're saying that this pcap should write out an ntlm log yet isn't? > - SMB: I tested on > https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=smbtorture.cap.gz  So many people point to this pcap, but I tend to avoid it because it doesn't seem to represent a normal smb client and server very well. It's too hard to understand how that pcap should map into logs. Maybe someday. :) Thanks! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From florent.monjalet at gmail.com Thu Jul 7 09:26:54 2016 From: florent.monjalet at gmail.com (Florent Monjalet) Date: Thu, 07 Jul 2016 16:26:54 +0000 Subject: [Bro] NTLM parsing in DCE RPC In-Reply-To: References: Message-ID: Le jeu. 7 juil. 2016 ? 17:39, Seth Hall a ?crit : > > > On Jul 7, 2016, at 9:45 AM, Florent Monjalet > wrote: > > > > As part of this work, I was very interested in Seth's work on SMB, so > > this mail is about the topic/seth/smb branch. Here again, thanks a lot > > for the huge work on these protocols. > > This is a good time to reach out about that branch. We are preparing to > merge it into the master branch soon once we do a bit more review. > > > The first field of NTLM should actually be the "NTLMSSP\x00" magic > > (according to: > > http://davenport.sourceforge.net/ntlm.html#theNtlmMessageHeaderLayout > > and wireshark dissectors). Moving the `meta` field to the `GSSAPI` > > layer will allow to properlly decode NTLM over DCE RPC and maybe HTLM > > NTLM Authentication later on. > > Ugh, I'm not surprised that there is yet another case where this is done > wrong. I'll review the change you proposed and the look at the pcap. > > > - DCE RPC: Not easy to find an open example capture, but this one > > is ok > > > https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=mapi.cap > > (from packet 711 in wireshark). You'll have to register DCE RPC > > on port 4997 (mapi) in bro. > > Just for clarity, you're saying that this pcap should write out an ntlm > log yet isn't? > Exactly (provided that you enable DCE RPC decoding on port 4997). Actually, I found and debugged the issue on private captures and just looked for public pcap where I could reproduce the issue. The expected ntlm log body for this capture is: 1056991898.902392 CUwb2m3ZV4I6liX6Ba 192.168.0.173 1068 192.168.0.2 4997 ALeonard ALEONARD-XP CNAMIS - - (Success/failure for NTLM authentication on DCE RPC is not implemented yet, but I guess it is rather non trivial to do.) > > > - SMB: I tested on > > > https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=smbtorture.cap.gz > > So many people point to this pcap, but I tend to avoid it because it > doesn't seem to represent a normal smb client and server very well. It's > too hard to understand how that pcap should map into logs. Maybe someday. > :) > Well, as previously mentionned, I just took the first matching public pcap in google for my issue, I was just interested in the SMB/GSSAPI/NTLM auth packets. I think your test samples are perfect for testing the issue. > > Thanks! > .Seth > Thanks for the quick answer! Florent > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160707/d617a2a1/attachment-0001.html From seth at icir.org Thu Jul 7 10:06:29 2016 From: seth at icir.org (Seth Hall) Date: Thu, 7 Jul 2016 13:06:29 -0400 Subject: [Bro] NTLM parsing in DCE RPC In-Reply-To: References: Message-ID: <19D9C465-4625-406C-9A70-4B3D9A65041C@icir.org> > On Jul 7, 2016, at 9:45 AM, Florent Monjalet wrote: > > It turns out that for DCE RPC, the NTLM decoding seems broken: the > NTLM analyzer is called, but the decoding fails to recognize the > message type, and no `ntlm.log` log is produced. It works very well > for SMB, though. I just merged your patch into the topic/seth/smb branch. I also verified that the change doesn't impact the public tests or a private test I'm maintaining. I also did another fix to actually load the DPD signature for DCE-RPC. It makes the port 4997/tcp stuff from that mapi.cap file show up automatically. Thanks! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From florent.monjalet at gmail.com Thu Jul 7 10:12:21 2016 From: florent.monjalet at gmail.com (Florent Monjalet) Date: Thu, 07 Jul 2016 17:12:21 +0000 Subject: [Bro] NTLM parsing in DCE RPC In-Reply-To: <19D9C465-4625-406C-9A70-4B3D9A65041C@icir.org> References: <19D9C465-4625-406C-9A70-4B3D9A65041C@icir.org> Message-ID: Le jeu. 7 juil. 2016 ? 19:06, Seth Hall a ?crit : > > > On Jul 7, 2016, at 9:45 AM, Florent Monjalet > wrote: > > > > It turns out that for DCE RPC, the NTLM decoding seems broken: the > > NTLM analyzer is called, but the decoding fails to recognize the > > message type, and no `ntlm.log` log is produced. It works very well > > for SMB, though. > > I just merged your patch into the topic/seth/smb branch. I also verified > that the change doesn't impact the public tests or a private test I'm > maintaining. > > I also did another fix to actually load the DPD signature for DCE-RPC. It > makes the port 4997/tcp stuff from that mapi.cap file show up automatically. > Great, thanks again for your great work and reactivity! Florent > Thanks! > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160707/ac39912d/attachment.html From bro at pingtrip.com Thu Jul 7 15:25:54 2016 From: bro at pingtrip.com (Dave Crawford) Date: Thu, 7 Jul 2016 18:25:54 -0400 Subject: [Bro] PF_RING ZC Config In-Reply-To: References: Message-ID: Just wanted to update the list that I quit spending cycles on this and for the time being reverted back to running our clusters with the non-commercial version of pf_ring. I can only comment on my experience, but I discovered there is an extreme lack of quality documentation and the "commercial support" that came with the 10 licenses was nearly non-existent. Lessons have been learned and when the need to expand comes we'll be looking at other commercial solutions to replace our X520's with. -Dave > On Jun 24, 2016, at 8:28 AM, Dave Crawford wrote: > > Would anyone happen to have documentation for configuring ZC and Bro? I have NTop's PF_RING and ixgbe driver packages installed, the proper license in /etc/pf_ring, and have compiled Bro with the NTop libraries but I'm seeing the kernel error below along with a ton of ?split routing? messages in weird.conf, so I suspect the flows aren?t being load balanced correctly. > > Jun 22 15:10:03 win-csignsm-01 kernel: [11060.244524] [PF_RING] Unable to activate two or more ZC sockets on the same interface eth6/link direction > > The monitored NIC is an Intel X520-LR1. > > Contents of /etc/pf_ring/zc/ixgbe/ixgbe.conf: > RSS=10 allow_unsupported_sfp=0 > > Contents of /etc/pf_ring/hugepages.conf > node=1 hugepages=1024 > > > And Bro is configured as: > [MID_INT] > type=worker > host=10.20.30.123 > interface=zc:eth6 > lb_method=pf_ring > lb_procs=10 > pin_cpus=10,11,12,13,14,15,16,17,18,19 > > Thanks! > -Dave > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160707/4f063c7e/attachment.html From bro at pingtrip.com Thu Jul 7 15:36:55 2016 From: bro at pingtrip.com (Dave Crawford) Date: Thu, 7 Jul 2016 18:36:55 -0400 Subject: [Bro] Onsite Training Options? In-Reply-To: <1467897597.2878480.659499673.56DE6795@webmail.messagingengine.com> References: <1467897597.2878480.659499673.56DE6795@webmail.messagingengine.com> Message-ID: <095DD75D-6F2E-4D91-93B9-6DAA0446D38F@pingtrip.com> Not sure of his current status but we had Liam from CriticalStack onsite multiple times and it was phenomenal experience. > On Jul 7, 2016, at 9:19 AM, K2 wrote: > > Anyone have any leads on anybody available for onsite Bro training in > the US? > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From cardigliano at ntop.org Fri Jul 8 00:51:48 2016 From: cardigliano at ntop.org (Alfredo Cardigliano) Date: Fri, 8 Jul 2016 09:51:48 +0200 Subject: [Bro] PF_RING ZC Config In-Reply-To: References: Message-ID: Hi Dave how did you ask support? I do not see any related issue in our ticketing system nor in email threads. Alfredo > On 08 Jul 2016, at 00:25, Dave Crawford wrote: > > Just wanted to update the list that I quit spending cycles on this and for the time being reverted back to running our clusters with the non-commercial version of pf_ring. > > I can only comment on my experience, but I discovered there is an extreme lack of quality documentation and the "commercial support" that came with the 10 licenses was nearly non-existent. > > Lessons have been learned and when the need to expand comes we'll be looking at other commercial solutions to replace our X520's with. > > -Dave > >> On Jun 24, 2016, at 8:28 AM, Dave Crawford > wrote: >> >> Would anyone happen to have documentation for configuring ZC and Bro? I have NTop's PF_RING and ixgbe driver packages installed, the proper license in /etc/pf_ring, and have compiled Bro with the NTop libraries but I'm seeing the kernel error below along with a ton of ?split routing? messages in weird.conf, so I suspect the flows aren?t being load balanced correctly. >> >> Jun 22 15:10:03 win-csignsm-01 kernel: [11060.244524] [PF_RING] Unable to activate two or more ZC sockets on the same interface eth6/link direction >> >> The monitored NIC is an Intel X520-LR1. >> >> Contents of /etc/pf_ring/zc/ixgbe/ixgbe.conf: >> RSS=10 allow_unsupported_sfp=0 >> >> Contents of /etc/pf_ring/hugepages.conf >> node=1 hugepages=1024 >> >> >> And Bro is configured as: >> [MID_INT] >> type=worker >> host=10.20.30.123 >> interface=zc:eth6 >> lb_method=pf_ring >> lb_procs=10 >> pin_cpus=10,11,12,13,14,15,16,17,18,19 >> >> Thanks! >> -Dave >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160708/c54dc796/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160708/c54dc796/attachment.bin From johanna at icir.org Fri Jul 8 09:18:22 2016 From: johanna at icir.org (Johanna Amann) Date: Fri, 8 Jul 2016 09:18:22 -0700 Subject: [Bro] Notice.log logs a Password_Guessing attempt but no logs in conn.log In-Reply-To: References: Message-ID: <20160708161818.GA42940@wifi241.sys.ICSI.Berkeley.EDU> Hello Fatema, you actually managed to stumble accross a bug here - apparently the event that we use to determine when password guessing occurs can be raised several times in the same connection (which probably is an error). I filed a ticket for this, if you want you can track the progress at https://bro-tracker.atlassian.net/browse/BIT-1641. Thank you, Johanna On Fri, Jul 01, 2016 at 05:28:49PM -0400, fatema bannatwala wrote: > Hi, > > So I had a weird situation at work today. > The notice.log file logged an IP for "SSH::Password_Guessing" with note as > "50.123.48.2 appears to be guessing SSH passwords (seen in 53 connections)". > > But when I check conn.log file during that time period and grep that IP, I > just see single ssh established connection from that IP. I was assuming to > get 53 bad ssh connections logged in conn.lo file. > > What am I missing here? > How can I confirm whether that IP was actually doing a SSH password > guessing attempt? > > Thanks, > Fatema. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Fri Jul 8 09:30:02 2016 From: johanna at icir.org (Johanna Amann) Date: Fri, 8 Jul 2016 09:30:02 -0700 Subject: [Bro] Question on "BROBASE" In-Reply-To: References: Message-ID: <20160708163002.GB42940@wifi241.sys.ICSI.Berkeley.EDU> Hello Raj, I think BROBASE is only used by broctl internally and is not used by Bro itself at all; Bro uses different environment variable to find its script called BROPATH. In any case, it is probably not advisable to change these manually, and the functionality of them might change in the future (especially when they might interplay with the bro package manager that is being written at the moment). So - I would recommend just doing completely separate installations. Johanna On Fri, Jul 01, 2016 at 10:56:58PM +0000, Raj Srinivasan wrote: > Hello, > > We have a test environment where we want to run multiple bro instances. Thus, we would like to have /usr/local/bro1, /usr/local/bro2, etc. Is there a way I can change the "BROBASE" variable for copies of the first instance, or do I have to generate the other instances with a "make install" for each? "BROBASE" is an internal variable, and the documentation seems to discourage changing it. This seems to be the cleanest way for us to ensure that we can configure each instance totally independently of the others. > > Thanks, > Raj > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Fri Jul 8 10:03:04 2016 From: johanna at icir.org (Johanna Amann) Date: Fri, 8 Jul 2016 10:03:04 -0700 Subject: [Bro] Connection lasts huge time In-Reply-To: <3f8d13efec52440c8c43630d44891128@SRVEX03.aizoon.local> References: <3f8d13efec52440c8c43630d44891128@SRVEX03.aizoon.local> Message-ID: <20160708170304.GA67565@wifi241.sys.ICSI.Berkeley.EDU> Hello Simone, Are you sure that these connections did not just last several hours? If you are sure - the only possible way that I can think of for these values getting messed up is libpcap (and thus probably the kernel) delivering wrong timestamps for packets. Bro just determines the duration by sybstracting the timestamp of the first packet that it saw from the timestamp of the last packet that it saw. Additionally, Bro will expire connections where it has not seen any packets, generally after a few minutes. I have seen wrong timestamps being delivered for a couple of packets before, but in those cases they were off by years, not just by hours, so I consider that unlikely. Johanna On Tue, Jul 05, 2016 at 03:30:15PM +0000, Rotondo Simone wrote: > Hi, > in my Bro logs, I have some connections that lasts 6 hours and more. > Those conns use different services: > > ------------------------ > conn.13:00:00-14:00:00.log.gz:{"ts":1466403937.471482,"uid":"CNLOdrb4ss9hRDDgg","id.orig_h":"XXX.XXX.XXX.XXX","id.orig_p":XXXXX,"id.resp_h":"XXX.XXX.XXX.XXX","id.resp_p":3268,"proto":"tcp","duration":16980.700023,"orig_bytes":299358,"resp_bytes":258817,"conn_state":"S1","local_resp":false,"missed_bytes":58394,"history":"ShADad","orig_pkts":485,"orig_ip_bytes":287130,"resp_pkts":243,"resp_ip_bytes":241795,"tunnel_parents":[],"local_origi":"T4","local_respo":"F4"} > > ------------------------ > conn.14:00:00-15:00:00.log.gz:{"ts":1466404357.492809,"uid":"CA7q9dl7q5ZbTDRXa","id.orig_h":"XXX.XXX.XXX.XXX","id.orig_p":XXXXX,"id.resp_h":" XXX.XXX.XXX.XXX","id.resp_p":443,"proto":"tcp","service":"ssl","duration":22774.467724,"orig_bytes":341462,"resp_bytes":402631,"conn_state":"S1","local_resp":true,"missed_bytes":51675,"history":"ShADda","orig_pkts":921,"orig_ip_bytes":353314,"resp_pkts":2058,"resp_ip_bytes":458288,"tunnel_parents":[],"from_known_services":["SSL"],"local_origi":"T4","local_respo":"T4"} > +++++ > ssl.08:00:00-09:00:00.log.gz:{"ts":1466404357.495789,"uid":"CA7q9dl7q5ZbTDRXa","id.orig_h":"XXX.XXX.XXX.XXX","id.orig_p":XXXXX,"id.resp_h":"XXX.XXX.XXX.XXX","id.resp_p":443,"version":"TLSv12","cipher":"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","server_name":"mail.xxxxxx.xxx","resumed":true,"established":true} > > ------------------------ > conn.14:00:00-15:00:00.log.gz:{"ts":1466404268.700607,"uid":"Czvp3U1saSEh9powDh","id.orig_h":"XXX.XXX.XXX.XXX","id.orig_p":XXXXX,"id.resp_h":"XXX.XXX.XXX.XXX","id.resp_p":443,"proto":"tcp","service":"ssl","duration":21422.058832,"orig_bytes":2158,"resp_bytes":122049,"conn_state":"RSTO","local_resp":true,"missed_bytes":3254,"history":"ShADdaR","orig_pkts":411,"orig_ip_bytes":18610,"resp_pkts":836,"resp_ip_bytes":152247,"tunnel_parents":[],"from_known_services":["SSL"],"local_origi":"T4","local_respo":"T4"} > +++++ > ssl.08:00:00-09:00:00.log.gz:{"ts":1466404268.70101,"uid":"Czvp3U1saSEh9powDh","id.orig_h":"XXX.XXX.XXX.XXX","id.orig_p":XXXXX,"id.resp_h":"XXX.XXX.XXX.XXX","id.resp_p":443,"version":"TLSv10","cipher":"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","curve":"secp384r1","server_name":"mail.xxxxx.xxx","resumed":false,"established":true,"cert_chain_fuids":["XXXXXXXXXXXXXXXXX","XXXXXXXXXXXXXXXXXXX"],"client_cert_chain_fuids":[],"subject":"emailAddress=hostmaster at xxxxxx.xx,CN=mail.xxxxxxx.xxx,O=XXXXXXX,...} > > > Have you got any idea about this issue? > > BR > Simone > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From gfaulkner.nsm at gmail.com Fri Jul 8 10:26:22 2016 From: gfaulkner.nsm at gmail.com (Gary Faulkner) Date: Fri, 8 Jul 2016 12:26:22 -0500 Subject: [Bro] PF_RING ZC Config In-Reply-To: References: Message-ID: <9c5209a7-1601-6ae9-b4e0-5713e3316c1a@gmail.com> Related to Dave's query, but not really an answer, sorry Dave... It might be worth revisiting this doc and updating for ZC: https://www.bro.org/documentation/load-balancing.html A few things have changed on the PF_RING DNA side in broctl in regards to naming support "dnacl" instead of "dnacluster" due to problems with name length for dnaclusters with greater than 10 queues, and with the most recent releases of PF_RING (6.4+), DNA appears to have been removed finally in favor of the newer ZC according to the change notes. From what I recall reading I don't believe it is terribly different outside of substituting ZC drivers (and tweaking huge-pages in the driver load script) in favor of DNA, and using zbalance_ipc instead of pfdnacluster_master. I want to say the naming in node.cfg becomes zc: instead of dnacl:. Also, speaking of ZC, NTOP has a blog post that might be worth taking a look at concerning alternate ways of implementing ZC / zbalance_ipc with bro to work around a problem that can occur when bro workers crash and get automatically restarted. http://www.ntop.org/pf_ring/best-practices-for-using-bro_ids-with-pf_ring-zc-reliably/ I haven't quite made the transition to ZC from DNA yet, otherwise I'd take a stab at submitting updated docs and trying to assist more here. I have plans to make the switch later this summer though. ~Gary On 7/7/16 5:25 PM, Dave Crawford wrote: > Just wanted to update the list that I quit spending cycles on this and for the time being reverted back to running our clusters with the non-commercial version of pf_ring. > > I can only comment on my experience, but I discovered there is an extreme lack of quality documentation and the "commercial support" that came with the 10 licenses was nearly non-existent. > > Lessons have been learned and when the need to expand comes we'll be looking at other commercial solutions to replace our X520's with. > > -Dave > >> On Jun 24, 2016, at 8:28 AM, Dave Crawford wrote: >> >> Would anyone happen to have documentation for configuring ZC and Bro? I have NTop's PF_RING and ixgbe driver packages installed, the proper license in /etc/pf_ring, and have compiled Bro with the NTop libraries but I'm seeing the kernel error below along with a ton of ?split routing? messages in weird.conf, so I suspect the flows aren?t being load balanced correctly. >> >> Jun 22 15:10:03 win-csignsm-01 kernel: [11060.244524] [PF_RING] Unable to activate two or more ZC sockets on the same interface eth6/link direction >> >> The monitored NIC is an Intel X520-LR1. >> >> Contents of /etc/pf_ring/zc/ixgbe/ixgbe.conf: >> RSS=10 allow_unsupported_sfp=0 >> >> Contents of /etc/pf_ring/hugepages.conf >> node=1 hugepages=1024 >> >> >> And Bro is configured as: >> [MID_INT] >> type=worker >> host=10.20.30.123 >> interface=zc:eth6 >> lb_method=pf_ring >> lb_procs=10 >> pin_cpus=10,11,12,13,14,15,16,17,18,19 >> >> Thanks! >> -Dave >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160708/8e49d090/attachment.html From fatema.bannatwala at gmail.com Fri Jul 8 10:29:29 2016 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Fri, 8 Jul 2016 13:29:29 -0400 Subject: [Bro] Notice.log logs a Password_Guessing attempt but no logs in conn.log In-Reply-To: <20160708161818.GA42940@wifi241.sys.ICSI.Berkeley.EDU> References: <20160708161818.GA42940@wifi241.sys.ICSI.Berkeley.EDU> Message-ID: Hi Johanna, Thanks for looking into the issue, because currently we were blocking all the IPs reported by BRO doing SSH:Password_Guessing (from notice.log). And we have been doing that for almost 6 months now, but came across this situation when a legit IP got blocked for doing a Password_guessing according to BRO and we were asked to produce the log files reporting that it was actually doing guessing, but we couldn't because we didn't find any logs to prove it.. I hope it gets fixed in the new version, because it's a really cool feature to check to see all password guessing IPs and take necessary action against them. Thanks for working on it, appreciate it. Thanks, Fatema. On Fri, Jul 8, 2016 at 12:18 PM, Johanna Amann wrote: > Hello Fatema, > > you actually managed to stumble accross a bug here - apparently the event > that we use to determine when password guessing occurs can be raised > several times in the same connection (which probably is an error). > > I filed a ticket for this, if you want you can track the progress at > https://bro-tracker.atlassian.net/browse/BIT-1641. > > Thank you, > Johanna > > On Fri, Jul 01, 2016 at 05:28:49PM -0400, fatema bannatwala wrote: > > Hi, > > > > So I had a weird situation at work today. > > The notice.log file logged an IP for "SSH::Password_Guessing" with note > as > > "50.123.48.2 appears to be guessing SSH passwords (seen in 53 > connections)". > > > > But when I check conn.log file during that time period and grep that IP, > I > > just see single ssh established connection from that IP. I was assuming > to > > get 53 bad ssh connections logged in conn.lo file. > > > > What am I missing here? > > How can I confirm whether that IP was actually doing a SSH password > > guessing attempt? > > > > Thanks, > > Fatema. > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160708/179187ca/attachment.html From slagell at illinois.edu Fri Jul 8 10:35:58 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Fri, 8 Jul 2016 17:35:58 +0000 Subject: [Bro] PF_RING ZC Config In-Reply-To: <9c5209a7-1601-6ae9-b4e0-5713e3316c1a@gmail.com> References: <9c5209a7-1601-6ae9-b4e0-5713e3316c1a@gmail.com> Message-ID: Could you create a ticket for this in the tracker? On Jul 8, 2016, at 12:26 PM, Gary Faulkner > wrote: Related to Dave's query, but not really an answer, sorry Dave... It might be worth revisiting this doc and updating for ZC: https://www.bro.org/documentation/load-balancing.html A few things have changed on the PF_RING DNA side in broctl in regards to naming support "dnacl" instead of "dnacluster" due to problems with name length for dnaclusters with greater than 10 queues, and with the most recent releases of PF_RING (6.4+), DNA appears to have been removed finally in favor of the newer ZC according to the change notes. From what I recall reading I don't believe it is terribly different outside of substituting ZC drivers (and tweaking huge-pages in the driver load script) in favor of DNA, and using zbalance_ipc instead of pfdnacluster_master. I want to say the naming in node.cfg becomes zc: instead of dnacl:. Also, speaking of ZC, NTOP has a blog post that might be worth taking a look at concerning alternate ways of implementing ZC / zbalance_ipc with bro to work around a problem that can occur when bro workers crash and get automatically restarted. http://www.ntop.org/pf_ring/best-practices-for-using-bro_ids-with-pf_ring-zc-reliably/ I haven't quite made the transition to ZC from DNA yet, otherwise I'd take a stab at submitting updated docs and trying to assist more here. I have plans to make the switch later this summer though. ~Gary On 7/7/16 5:25 PM, Dave Crawford wrote: Just wanted to update the list that I quit spending cycles on this and for the time being reverted back to running our clusters with the non-commercial version of pf_ring. I can only comment on my experience, but I discovered there is an extreme lack of quality documentation and the "commercial support" that came with the 10 licenses was nearly non-existent. Lessons have been learned and when the need to expand comes we'll be looking at other commercial solutions to replace our X520's with. -Dave On Jun 24, 2016, at 8:28 AM, Dave Crawford wrote: Would anyone happen to have documentation for configuring ZC and Bro? I have NTop's PF_RING and ixgbe driver packages installed, the proper license in /etc/pf_ring, and have compiled Bro with the NTop libraries but I'm seeing the kernel error below along with a ton of ?split routing? messages in weird.conf, so I suspect the flows aren?t being load balanced correctly. Jun 22 15:10:03 win-csignsm-01 kernel: [11060.244524] [PF_RING] Unable to activate two or more ZC sockets on the same interface eth6/link direction The monitored NIC is an Intel X520-LR1. Contents of /etc/pf_ring/zc/ixgbe/ixgbe.conf: RSS=10 allow_unsupported_sfp=0 Contents of /etc/pf_ring/hugepages.conf node=1 hugepages=1024 And Bro is configured as: [MID_INT] type=worker host=10.20.30.123 interface=zc:eth6 lb_method=pf_ring lb_procs=10 pin_cpus=10,11,12,13,14,15,16,17,18,19 Thanks! -Dave _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro ------ Adam J. Slagell Chief Information Security Officer Director, Cybersecurity Division National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160708/cee92c86/attachment-0001.html From gfaulkner.nsm at gmail.com Fri Jul 8 10:57:18 2016 From: gfaulkner.nsm at gmail.com (Gary Faulkner) Date: Fri, 8 Jul 2016 12:57:18 -0500 Subject: [Bro] PF_RING ZC Config In-Reply-To: References: <9c5209a7-1601-6ae9-b4e0-5713e3316c1a@gmail.com> Message-ID: <4085de6a-b70d-d18a-cb21-33ffa95b84aa@gmail.com> https://bro-tracker.atlassian.net/browse/BIT-1642 On 7/8/16 12:35 PM, Slagell, Adam J wrote: > Could you create a ticket for this in the tracker? > > On Jul 8, 2016, at 12:26 PM, Gary Faulkner > wrote: > > > Related to Dave's query, but not really an answer, sorry Dave... > > It might be worth revisiting this doc and updating for ZC: > > https://www.bro.org/documentation/load-balancing.html > > A few things have changed on the PF_RING DNA side in broctl in regards to naming support "dnacl" instead of "dnacluster" due to problems with name length for dnaclusters with greater than 10 queues, and with the most recent releases of PF_RING (6.4+), DNA appears to have been removed finally in favor of the newer ZC according to the change notes. From what I recall reading I don't believe it is terribly different outside of substituting ZC drivers (and tweaking huge-pages in the driver load script) in favor of DNA, and using zbalance_ipc instead of pfdnacluster_master. I want to say the naming in node.cfg becomes zc: instead of dnacl:. > > Also, speaking of ZC, NTOP has a blog post that might be worth taking a look at concerning alternate ways of implementing ZC / zbalance_ipc with bro to work around a problem that can occur when bro workers crash and get automatically restarted. > > http://www.ntop.org/pf_ring/best-practices-for-using-bro_ids-with-pf_ring-zc-reliably/ > > I haven't quite made the transition to ZC from DNA yet, otherwise I'd take a stab at submitting updated docs and trying to assist more here. I have plans to make the switch later this summer though. > > ~Gary > > On 7/7/16 5:25 PM, Dave Crawford wrote: > > Just wanted to update the list that I quit spending cycles on this and for the time being reverted back to running our clusters with the non-commercial version of pf_ring. > > I can only comment on my experience, but I discovered there is an extreme lack of quality documentation and the "commercial support" that came with the 10 licenses was nearly non-existent. > > Lessons have been learned and when the need to expand comes we'll be looking at other commercial solutions to replace our X520's with. > > -Dave > > > > On Jun 24, 2016, at 8:28 AM, Dave Crawford wrote: > > Would anyone happen to have documentation for configuring ZC and Bro? I have NTop's PF_RING and ixgbe driver packages installed, the proper license in /etc/pf_ring, and have compiled Bro with the NTop libraries but I'm seeing the kernel error below along with a ton of ?split routing? messages in weird.conf, so I suspect the flows aren?t being load balanced correctly. > > Jun 22 15:10:03 win-csignsm-01 kernel: [11060.244524] [PF_RING] Unable to activate two or more ZC sockets on the same interface eth6/link direction > > The monitored NIC is an Intel X520-LR1. > > Contents of /etc/pf_ring/zc/ixgbe/ixgbe.conf: > RSS=10 allow_unsupported_sfp=0 > > Contents of /etc/pf_ring/hugepages.conf > node=1 hugepages=1024 > > > And Bro is configured as: > [MID_INT] > type=worker > host=10.20.30.123 > interface=zc:eth6 > lb_method=pf_ring > lb_procs=10 > pin_cpus=10,11,12,13,14,15,16,17,18,19 > > Thanks! > -Dave > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > ------ > > Adam J. Slagell > Chief Information Security Officer > Director, Cybersecurity Division > National Center for Supercomputing Applications > University of Illinois at Urbana-Champaign > www.slagell.info > > "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." > > > > > > > > > From slagell at illinois.edu Fri Jul 8 12:23:39 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Fri, 8 Jul 2016 19:23:39 +0000 Subject: [Bro] PF_RING ZC Config In-Reply-To: <4085de6a-b70d-d18a-cb21-33ffa95b84aa@gmail.com> References: <9c5209a7-1601-6ae9-b4e0-5713e3316c1a@gmail.com> <4085de6a-b70d-d18a-cb21-33ffa95b84aa@gmail.com> Message-ID: <62F96348-E24A-4514-9109-0FB7B39D9F86@illinois.edu> Thanks. I don?t want to forget to come back to this. > On Jul 8, 2016, at 12:57 PM, Gary Faulkner wrote: > > https://bro-tracker.atlassian.net/browse/BIT-1642 > > > On 7/8/16 12:35 PM, Slagell, Adam J wrote: >> Could you create a ticket for this in the tracker? >> >> On Jul 8, 2016, at 12:26 PM, Gary Faulkner > wrote: >> >> >> Related to Dave's query, but not really an answer, sorry Dave... >> >> It might be worth revisiting this doc and updating for ZC: >> >> https://www.bro.org/documentation/load-balancing.html >> >> A few things have changed on the PF_RING DNA side in broctl in regards to naming support "dnacl" instead of "dnacluster" due to problems with name length for dnaclusters with greater than 10 queues, and with the most recent releases of PF_RING (6.4+), DNA appears to have been removed finally in favor of the newer ZC according to the change notes. From what I recall reading I don't believe it is terribly different outside of substituting ZC drivers (and tweaking huge-pages in the driver load script) in favor of DNA, and using zbalance_ipc instead of pfdnacluster_master. I want to say the naming in node.cfg becomes zc: instead of dnacl:. >> >> Also, speaking of ZC, NTOP has a blog post that might be worth taking a look at concerning alternate ways of implementing ZC / zbalance_ipc with bro to work around a problem that can occur when bro workers crash and get automatically restarted. >> >> http://www.ntop.org/pf_ring/best-practices-for-using-bro_ids-with-pf_ring-zc-reliably/ >> >> I haven't quite made the transition to ZC from DNA yet, otherwise I'd take a stab at submitting updated docs and trying to assist more here. I have plans to make the switch later this summer though. >> >> ~Gary >> >> On 7/7/16 5:25 PM, Dave Crawford wrote: >> >> Just wanted to update the list that I quit spending cycles on this and for the time being reverted back to running our clusters with the non-commercial version of pf_ring. >> >> I can only comment on my experience, but I discovered there is an extreme lack of quality documentation and the "commercial support" that came with the 10 licenses was nearly non-existent. >> >> Lessons have been learned and when the need to expand comes we'll be looking at other commercial solutions to replace our X520's with. >> >> -Dave >> >> >> >> On Jun 24, 2016, at 8:28 AM, Dave Crawford wrote: >> >> Would anyone happen to have documentation for configuring ZC and Bro? I have NTop's PF_RING and ixgbe driver packages installed, the proper license in /etc/pf_ring, and have compiled Bro with the NTop libraries but I'm seeing the kernel error below along with a ton of ?split routing? messages in weird.conf, so I suspect the flows aren?t being load balanced correctly. >> >> Jun 22 15:10:03 win-csignsm-01 kernel: [11060.244524] [PF_RING] Unable to activate two or more ZC sockets on the same interface eth6/link direction >> >> The monitored NIC is an Intel X520-LR1. >> >> Contents of /etc/pf_ring/zc/ixgbe/ixgbe.conf: >> RSS=10 allow_unsupported_sfp=0 >> >> Contents of /etc/pf_ring/hugepages.conf >> node=1 hugepages=1024 >> >> >> And Bro is configured as: >> [MID_INT] >> type=worker >> host=10.20.30.123 >> interface=zc:eth6 >> lb_method=pf_ring >> lb_procs=10 >> pin_cpus=10,11,12,13,14,15,16,17,18,19 >> >> Thanks! >> -Dave >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> >> >> >> >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> ------ >> >> Adam J. Slagell >> Chief Information Security Officer >> Director, Cybersecurity Division >> National Center for Supercomputing Applications >> University of Illinois at Urbana-Champaign >> www.slagell.info >> >> "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." >> >> >> >> >> >> >> >> >> > ------ Adam J. Slagell Chief Information Security Officer Director, Cybersecurity Division National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." From raj at bivio.net Fri Jul 8 12:29:17 2016 From: raj at bivio.net (Raj Srinivasan) Date: Fri, 8 Jul 2016 19:29:17 +0000 Subject: [Bro] Question on SSL logs Message-ID: Hello, First, the background info... we are in the process of upgrading from Bro v2.3.2 to v2.4.1. The older version runs on a slower system which experiences more packet loss than the newer version, which is running on a faster system (which has mostly no loss at all). Both systems are seeing the same network traffic. What we are seeing is that the SSL logs from v2.3.2 are consistently larger (by 20% to 25%) than the logs produced by v2.4.1. I see that there are a lot of improvements in the handling of SSL, and many that might actually impact log information, but we are unable to quantify how the logs are being affected even after a visual inspection of the logs. Is it reasonable to expect the new log files to be more compact (using the default SSL policies in both cases)? Just as a data point, the HTTP logs are comparable in size. Would highly appreciate a response from the Bro SSL experts. Thanks! Raj -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160708/87f86079/attachment.html From johanna at icir.org Fri Jul 8 12:54:44 2016 From: johanna at icir.org (Johanna Amann) Date: Fri, 8 Jul 2016 12:54:44 -0700 Subject: [Bro] Question on SSL logs In-Reply-To: References: Message-ID: <20160708195444.GA74936@wifi241.sys.ICSI.Berkeley.EDU> Hello Raj, on the top of my head, I am not aware of any reason why the 2.4.1 SSL logs should be more compact then the 2.3.2 logs; if anything, they should be larger. There were some changes to make the logs more compact, but thost were from 2.2 to 2.3; in these cases, a better metric would be the number of lines. In any case if you can potentially take a small sample of your traffic and run both 2.3.2 and 2.4.1 against it, and notice any changes (especially missing lines in 2.4.1), I would appreciate it you could let me know. Johanna On Fri, Jul 08, 2016 at 07:29:17PM +0000, Raj Srinivasan wrote: > Hello, > > First, the background info... we are in the process of upgrading from Bro v2.3.2 to v2.4.1. The older version runs on a slower system which experiences more packet loss than the newer version, which is running on a faster system (which has mostly no loss at all). Both systems are seeing the same network traffic. > > What we are seeing is that the SSL logs from v2.3.2 are consistently larger (by 20% to 25%) than the logs produced by v2.4.1. I see that there are a lot of improvements in the handling of SSL, and many that might actually impact log information, but we are unable to quantify how the logs are being affected even after a visual inspection of the logs. Is it reasonable to expect the new log files to be more compact (using the default SSL policies in both cases)? Just as a data point, the HTTP logs are comparable in size. > > Would highly appreciate a response from the Bro SSL experts. > > Thanks! > Raj > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jlay at slave-tothe-box.net Fri Jul 8 13:13:09 2016 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 08 Jul 2016 14:13:09 -0600 Subject: [Bro] More crypto ID Message-ID: FYI: 2016-07-01T12:35:15-0600 CyqleS3tHf607yRdrj 192.168.1.101 38151 31.13.76.102 443 TLSv12 unknown-52393 - graph.facebook.com F- h2 T Fq3gsi3bxz1RdtYqej,FiQmMNkbUAqhiOOkk (empty) CN=*.facebook.com,O=Facebook\\, Inc.,L=Menlo Park,ST=CA,C=US CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US - - ok unkonwn-52393 is apparently QUIC crypto. James From johanna at icir.org Fri Jul 8 13:21:21 2016 From: johanna at icir.org (Johanna Amann) Date: Fri, 08 Jul 2016 13:21:21 -0700 Subject: [Bro] More crypto ID In-Reply-To: References: Message-ID: Hello James, it is TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 and should be correctly identified by master. The use of that number is newer than Bro 2.4, which is why it is not present there. That cipher is specified in RFC7905. Thanks, Johanna On 8 Jul 2016, at 13:13, James Lay wrote: > FYI: > > 2016-07-01T12:35:15-0600 CyqleS3tHf607yRdrj 192.168.1.101 > 38151 31.13.76.102 443 TLSv12 unknown-52393 - > graph.facebook.com F- h2 T > Fq3gsi3bxz1RdtYqej,FiQmMNkbUAqhiOOkk (empty) > CN=*.facebook.com,O=Facebook\\, Inc.,L=Menlo Park,ST=CA,C=US > CN=DigiCert SHA2 High Assurance Server > CA,OU=www.digicert.com,O=DigiCert > Inc,C=US - - ok > > unkonwn-52393 is apparently QUIC crypto. > > James > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jlay at slave-tothe-box.net Fri Jul 8 13:30:57 2016 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 08 Jul 2016 14:30:57 -0600 Subject: [Bro] More crypto ID In-Reply-To: References: Message-ID: <937ebc5656dad0812cff33279c11adbc@localhost> Argh...yea you're right wrong stream. I am including a QUIC crypto session that bro does not seem to recognize. Only thing I have for bro seeing this stream is: 2016-07-02T14:46:30-0600 CWaKhQ3UAvIEem73fj 192.168.1.101 38848 31.13.76.102 443 tcp - 0.026353 1725 0 RSTR TF 0 ShADar 5 1993 5 268 (empty) Thank you. James On 2016-07-08 14:21, Johanna Amann wrote: > Hello James, > > it is TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 and should be > correctly identified by master. The use of that number is newer than > Bro 2.4, which is why it is not present there. That cipher is > specified in RFC7905. > > Thanks, > Johanna > > On 8 Jul 2016, at 13:13, James Lay wrote: > >> FYI: >> >> 2016-07-01T12:35:15-0600 CyqleS3tHf607yRdrj 192.168.1.101 >> 38151 31.13.76.102 443 TLSv12 unknown-52393 - >> graph.facebook.com F- h2 T >> Fq3gsi3bxz1RdtYqej,FiQmMNkbUAqhiOOkk (empty) >> CN=*.facebook.com,O=Facebook\\, Inc.,L=Menlo Park,ST=CA,C=US >> CN=DigiCert SHA2 High Assurance Server >> CA,OU=www.digicert.com,O=DigiCert >> Inc,C=US - - ok >> >> unkonwn-52393 is apparently QUIC crypto. >> >> James >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/octet-stream Size: 3548 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160708/a035314f/attachment-0001.obj From johanna at icir.org Fri Jul 8 14:09:32 2016 From: johanna at icir.org (Johanna Amann) Date: Fri, 08 Jul 2016 14:09:32 -0700 Subject: [Bro] More crypto ID In-Reply-To: <937ebc5656dad0812cff33279c11adbc@localhost> References: <937ebc5656dad0812cff33279c11adbc@localhost> Message-ID: <0DC4B153-DE2E-443D-998C-663AC9BCB662@icir.org> Bro currently does not support parsing QUIC at all - so you are correct - you won't get any data outside of conn.log for QUIC sessions. Johanna On 8 Jul 2016, at 13:30, James Lay wrote: > Argh...yea you're right wrong stream. I am including a QUIC crypto > session that bro does not seem to recognize. Only thing I have for > bro seeing this stream is: > > 2016-07-02T14:46:30-0600 CWaKhQ3UAvIEem73fj 192.168.1.101 > 38848 31.13.76.102 443 tcp - 0.026353 1725 > 0 RSTR TF 0 ShADar 5 1993 5 268 > (empty) > > Thank you. > > James > > On 2016-07-08 14:21, Johanna Amann wrote: >> Hello James, >> >> it is TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 and should be >> correctly identified by master. The use of that number is newer than >> Bro 2.4, which is why it is not present there. That cipher is >> specified in RFC7905. >> >> Thanks, >> Johanna >> >> On 8 Jul 2016, at 13:13, James Lay wrote: >> >>> FYI: >>> >>> 2016-07-01T12:35:15-0600 CyqleS3tHf607yRdrj >>> 192.168.1.101 >>> 38151 31.13.76.102 443 TLSv12 unknown-52393 - >>> graph.facebook.com F- h2 T >>> Fq3gsi3bxz1RdtYqej,FiQmMNkbUAqhiOOkk (empty) >>> CN=*.facebook.com,O=Facebook\\, Inc.,L=Menlo Park,ST=CA,C=US >>> CN=DigiCert SHA2 High Assurance Server >>> CA,OU=www.digicert.com,O=DigiCert >>> Inc,C=US - - ok >>> >>> unkonwn-52393 is apparently QUIC crypto. >>> >>> James >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jlay at slave-tothe-box.net Fri Jul 8 14:12:45 2016 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 08 Jul 2016 15:12:45 -0600 Subject: [Bro] More crypto ID In-Reply-To: <0DC4B153-DE2E-443D-998C-663AC9BCB662@icir.org> References: <937ebc5656dad0812cff33279c11adbc@localhost> <0DC4B153-DE2E-443D-998C-663AC9BCB662@icir.org> Message-ID: <2198919e63eca430ab91a25f0fc59a5f@localhost> Ok cool...I haven't seen many tools that do support QUIC crypto yet...thanks Johanna! James On 2016-07-08 15:09, Johanna Amann wrote: > Bro currently does not support parsing QUIC at all - so you are > correct - you won't get any data outside of conn.log for QUIC > sessions. > > Johanna > > On 8 Jul 2016, at 13:30, James Lay wrote: > >> Argh...yea you're right wrong stream. I am including a QUIC crypto >> session that bro does not seem to recognize. Only thing I have for >> bro seeing this stream is: >> >> 2016-07-02T14:46:30-0600 CWaKhQ3UAvIEem73fj 192.168.1.101 >> 38848 31.13.76.102 443 tcp - 0.026353 1725 >> 0 RSTR TF 0 ShADar 5 1993 5 268 >> (empty) >> >> Thank you. >> >> James >> >> On 2016-07-08 14:21, Johanna Amann wrote: >>> Hello James, >>> >>> it is TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 and should be >>> correctly identified by master. The use of that number is newer than >>> Bro 2.4, which is why it is not present there. That cipher is >>> specified in RFC7905. >>> >>> Thanks, >>> Johanna >>> >>> On 8 Jul 2016, at 13:13, James Lay wrote: >>> >>>> FYI: >>>> >>>> 2016-07-01T12:35:15-0600 CyqleS3tHf607yRdrj >>>> 192.168.1.101 >>>> 38151 31.13.76.102 443 TLSv12 unknown-52393 - >>>> graph.facebook.com F- h2 T >>>> Fq3gsi3bxz1RdtYqej,FiQmMNkbUAqhiOOkk (empty) >>>> CN=*.facebook.com,O=Facebook\\, Inc.,L=Menlo Park,ST=CA,C=US >>>> CN=DigiCert SHA2 High Assurance Server >>>> CA,OU=www.digicert.com,O=DigiCert >>>> Inc,C=US - - ok >>>> >>>> unkonwn-52393 is apparently QUIC crypto. >>>> >>>> James >>>> _______________________________________________ >>>> Bro mailing list >>>> bro at bro-ids.org >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From bglaze at gmail.com Fri Jul 8 14:35:14 2016 From: bglaze at gmail.com (Brandon Glaze) Date: Fri, 8 Jul 2016 14:35:14 -0700 Subject: [Bro] Bro Azure HDInsight HIVE table scripts and Azure python uplod Message-ID: Good day everyone, I wanted to share what I have done with regards to uploading my bro clusters logs to Azure HDInsight Blob storage, then each day creating HIVE tables from the logs. My bro cluster averages around 55-60Gbps, so sorting through logs via zgrep, or even Elastic, is far from ideal. I found that looking for a specific file ID took around 30 minutes to search all logs for the day, but when using HDInsight it was under a minute. Now I am sure there are some big data scientists on this forum, so forgive my newbness on Hadoop (HDInsight is Hortonworks Hadoop), as well as a basic python programmer, so that script is simple. It is my hope that this work helps someone else, or at least gets them started. I will see if I can sanitize my bro cluster build documents and send that out in the hope that it also helps. Here we go: First I modify each bro log to be Hadoop name convention friendly (doesnt like colons), then each hour I upload my log files with the below python script: ================= /etc/BRO-Azure-Upload.sh #!/bin/bash # Script to run each hour, which modifies the file names to be Azure Hadoop friendly, then calls a Python script to upload # each file to Azure blob storage DIR=/DATA/bro-logs DATE=`date +%Y-%m-%d "--date=last hour"` TIME=`date +%H "--date=last hour"` # Change to working directory cd $DIR/$DATE # Remove all ":" and replace with "_" in file names ls -1 *.`date +%H "--date=last hour"`* | while read FILE ; do newfile="$(echo ${FILE} | sed -e 's/\:/\_/g')" ; mv "${FILE}" "${newfile}" ; done # Add the building name to each file by finding the _([[:digit:]]).log.gz files, in case the first buildings logs have already been added ls -1 *.`date +%H "--date=last hour"`_*_[0-9][0-9].log.gz | while read FILE ; do newfile="$(echo ${FILE} | sed -e 's/.log.gz/.B31.log.gz/')" ; mv "${FILE}" "${newfile}" ; done # Call python script to upload last hours logs: for LOG in $(ls -1 *.`date +%H "--date=last hour"`_*.B01.log.gz ) ; do /DATA/Azure-Blob-PROMPT.py $LOG $DATE ; done *Here is the Python script I use to upload the logs:* ======================= Azure-Blob-PROMPT.py ======================= REQUIRES THAT YOU INSTALL PYTHON 2.7 on CentOS/RHEL #!/usr/local/python2.7/bin/python2.7 import sys import azure.storage from azure.storage.blob import BlockBlobService FILE = sys.argv[1] DIR = sys.argv[2] storage_account_name = '' # storage key in file in parent directory called .storagekey storage_account_key = '' block_blob_service = BlockBlobService(account_name=storage_account_name, account_key=storage_account_key) storage_container_name = '' block_blob_service.create_container(storage_container_name) #storage_dir = (storage_container_name,'/', DIR) storage_dir = storage_container_name+"/"+DIR from azure.storage.blob import ContentSettings block_blob_service.create_blob_from_path( storage_dir, FILE, FILE, content_settings=ContentSettings(content_type='application/octet-stream'), progress_callback=None, max_connections=5, max_retries=5, retry_wait=2 ) ============== Here are the commands I use to create the HIVE tables. I create a directory with the date via a cron job, then make a table with the entire days logs in it. ============= SSL: ================= DROP TABLE IF EXISTS SSLDAILY; CREATE EXTERNAL TABLE SSLDAILY (ts TIMESTAMP, uid STRING, orig_h STRING, orig_p STRING, resp_h STRING, resp_p STRING, version STRING, cipher STRING, curve STRING, server_name STRING, resumed STRING, last_alert STRING, next_protocol STRING, established STRING, cert_chain_fuids STRING, client_cert_chain_fuids STRING, subject STRING, issuer STRING, client_subject STRING, client_issuer STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' LOAD DATA INPATH 'wasb://@.blob.core.windows.net//ssl.*' INTO TABLE SSLDAILY; ================= CONN: ================= DROP TABLE IF EXISTS CONNDAILY; CREATE EXTERNAL TABLE CONNDAILY (ts TIMESTAMP, uid STRING, orig_h STRING, orig_p STRING, resp_h STRING, resp_p STRING, proto STRING, service STRING, duration DECIMAL, orig_bytes INT, resp_bytes INT, conn_state STRING, local_orig STRING, local_resp STRING, missed_bytes INT, history STRING, orig_pkts INT, orig_ip_bytes INT, resp_pkts INT, resp_ip_bytes INT, tunnel_parents STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'; LOAD DATA INPATH 'wasb://@.blob.core.windows.net//conn.*' INTO TABLE CONNDAILY; ================== DNS: ================== DROP TABLE IF EXISTS DNSDAILY; CREATE EXTERNAL TABLE DNSDAILY (ts TIMESTAMP, uid STRING, orig_h STRING, orig_p STRING, resp_h STRING, resp_p STRING, proto STRING, trans_id STRING, query STRING, qclass STRING, qclass_name STRING, qtype STRING, qtype_name STRING, rcode STRING, rcode_name STRING, AA STRING, TC STRING, RD STRING, RA STRING, Z STRING, answers STRING, TTLs STRING, rejected STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'; LOAD DATA INPATH 'wasb://@.blob.core.windows.net//dns.*' INTO TABLE DNSDAILY; ================================== DPD (Dynamic Protocol Detection): ================================== DROP TABLE IF EXISTS DPDDAILY; CREATE EXTERNAL TABLE DPDDAILY (ts TIMESTAMP, uid STRING, orig_h STRING, orig_p STRING, resp_h STRING, resp_p STRING, proto STRING, analyzer STRING, failure_reason STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'; LOAD DATA INPATH 'wasb://@.blob.core.windows.net//dpd.*' INTO TABLE DPDDAILY; ================== FILES: ================== DROP TABLE IF EXISTS FILESDAILY; CREATE EXTERNAL TABLE FILESDAILY (ts TIMESTAMP, fuid STRING, tx_hosts STRING, rx_hosts STRING, conn_uids STRING, source STRING, depth STRING, analyzers STRING, mime_type STRING, filename STRING, duration DECIMAL, local_orig STRING, is_orig STRING, seen_bytes INT, total_bytes INT, missing_bytes INT, overflow_bytes INT, timedout STRING, parent_fuid STRING, md5 STRING, sha1 STRING, sha256 STRING, extracted STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'; LOAD DATA INPATH 'wasb://@.blob.core.windows.net//files.*' INTO TABLE FILESDAILY; ================== FTP: ================== DROP TABLE IF EXISTS FTPDAILY; CREATE EXTERNAL TABLE FTPDAILY (ts TIMESTAMP, uid STRING, orig_h STRING, orig_p STRING, resp_h STRING, resp_p STRING, user STRING, password STRING, command STRING, arg STRING, mime_type STRING, file_size STRING, reply_code STRING, reply_msg STRING, data_channel_passive STRING, data_channel_orig_h STRING, data_channel_resp_h STRING, data_channel_resp_p STRING, fuid STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'; LOAD DATA INPATH 'wasb://@.blob.core.windows.net//ftp.*' INTO TABLE FTPDAILY; ================== HTTP: ================== DROP TABLE IF EXISTS HTTPDAILY; CREATE EXTERNAL TABLE HTTPDAILY (ts TIMESTAMP, uid STRING, orig_h STRING, orig_p STRING, resp_h STRING, resp_p STRING, trans_depth STRING, method STRING, host STRING, uri STRING, referrer STRING, user_agent STRING, request_body_len INT, status_code STRING, status_msg STRING, info_code STRING, info_msg STRING, filename STRING, tags STRING, username STRING, password STRING, proxied STRING, orig_fuids STRING, orig_mime_types STRING, resp_fuids STRING, resp_mime_types STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'; LOAD DATA INPATH 'wasb://@.blob.core.windows.net//http.*' INTO TABLE HTTPDAILY; ================== INTEL: ================== DROP TABLE IF EXISTS INTELDAILY; CREATE EXTERNAL TABLE INTELDAILY (ts TIMESTAMP, uid STRING, orig_h STRING, orig_p STRING, resp_h STRING, resp_p STRING, fuid STRING, file_mime_type STRING, file_desc STRING, seen_indicator STRING, seen_indicator_type STRING, seen_where STRING, seen_node STRING, sources STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'; LOAD DATA INPATH 'wasb://@.blob.core.windows.net//intel.*' INTO TABLE INTELDAILY; ================== IRC: ================== DROP TABLE IF EXISTS IRCDAILY; CREATE EXTERNAL TABLE IRCDAILY (ts TIMESTAMP, uid STRING, orig_h STRING, orig_p STRING, resp_h STRING, resp_p STRING, nick STRING, user STRING, command STRING, value STRING, addl STRING, dcc_file_name STRING, dcc_file_size INT, dcc_mime_type STRING, fuid STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'; LOAD DATA INPATH 'wasb://@.blob.core.windows.net//irc.*' INTO TABLE IRCDAILY; ================== KERBEROS: ================== DROP TABLE IF EXISTS KERBEROSDAILY; CREATE EXTERNAL TABLE KERBEROSDAILY (ts TIMESTAMP, uid STRING, orig_h STRING, orig_p STRING, resp_h STRING, resp_p STRING, request_type STRING, client STRING, service STRING, success STRING, error_msg STRING, from STRING, till STRING, cipher STRING, forwardable STRING, renewable STRING, client_cert_subject STRING, client_cert_fuid STRING, server_cert_subject STRING, server_cert_fuid STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'; LOAD DATA INPATH 'wasb://@.blob.core.windows.net//kerberos.*' INTO TABLE KERBEROSDAILY; ================== INTEL: ================== DROP TABLE IF EXISTS INTELDAILY; CREATE EXTERNAL TABLE INTELDAILY (ts TIMESTAMP, uid STRING, orig_h STRING, orig_p STRING, resp_h STRING, resp_p STRING, fuid STRING, file_mime_type STRING, file_desc STRING, seen_indicator STRING, seen_indicator_type STRING, seen_where STRING, seen_node STRING, sources STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'; LOAD DATA INPATH 'wasb://@.blob.core.windows.net//intel.*' INTO TABLE INTELDAILY; I welcome any feedback or questions. ===================== Brandon Glaze bglaze at gmail.com "Lead me, follow me, or get the hell out of my way." - General George Patton Jr -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160708/84115484/attachment-0001.html From brianallen at wustl.edu Mon Jul 11 06:58:26 2016 From: brianallen at wustl.edu (Allen, Brian) Date: Mon, 11 Jul 2016 13:58:26 +0000 Subject: [Bro] Onsite Training Options? In-Reply-To: <1467897597.2878480.659499673.56DE6795@webmail.messagingengine.com> References: <1467897597.2878480.659499673.56DE6795@webmail.messagingengine.com> Message-ID: Liam Randall from Critical Stack came to our town for a conference and offered a day long BRO training session. It was great. I highly recommend it if you can get him. -Brian From: > on behalf of K2 > Date: Thursday, July 7, 2016 at 8:19 AM To: Bro-Mailinglist > Subject: [Bro] Onsite Training Options? Anyone have any leads on anybody available for onsite Bro training in the US? _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160711/6f0ea1cf/attachment.html From jmlynch at gmail.com Mon Jul 11 07:14:34 2016 From: jmlynch at gmail.com (Jason Lynch) Date: Mon, 11 Jul 2016 10:14:34 -0400 Subject: [Bro] Onsite Training Options? In-Reply-To: References: <1467897597.2878480.659499673.56DE6795@webmail.messagingengine.com> Message-ID: Liam is on list and is absolutely fantastic. If he's busy due to the recent acquisition / figuring out what to spend his mad $$$ on: we're currently in the process of scheduling someone that I think is going to be the next best thing. Happy to provide feedback in the next month or two if it helps or provide a contact if interested off list. On Mon, Jul 11, 2016 at 9:58 AM, Allen, Brian wrote: > Liam Randall from Critical Stack came to our town for a conference and > offered a day long BRO training session. It was great. I highly recommend > it if you can get him. > -Brian > > From: on behalf of K2 > Date: Thursday, July 7, 2016 at 8:19 AM > To: Bro-Mailinglist > Subject: [Bro] Onsite Training Options? > > Anyone have any leads on anybody available for onsite Bro training in > the US? > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160711/77cd0e2d/attachment.html From giralt at reservoir.com Mon Jul 11 09:55:39 2016 From: giralt at reservoir.com (Jordi Ros-Giralt) Date: Mon, 11 Jul 2016 18:55:39 +0200 Subject: [Bro] Onsite Training Options? In-Reply-To: <1467897597.2878480.659499673.56DE6795@webmail.messagingengine.com> References: <1467897597.2878480.659499673.56DE6795@webmail.messagingengine.com> Message-ID: Reservoir Labs is also providing Bro training courses in case you are interested. Thanks, Jordi Ros-Giralt, PhD Reservoir Labs +1 212 780 0527 x110 On Thu, Jul 7, 2016 at 3:19 PM, K2 wrote: > Anyone have any leads on anybody available for onsite Bro training in > the US? > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160711/8db9f7e0/attachment.html From liam.randall at gmail.com Mon Jul 11 10:39:59 2016 From: liam.randall at gmail.com (Liam Randall) Date: Mon, 11 Jul 2016 13:39:59 -0400 Subject: [Bro] Onsite Training Options? In-Reply-To: References: <1467897597.2878480.659499673.56DE6795@webmail.messagingengine.com> Message-ID: Hey Guys, Thanks for all of the great feedback. We have not scheduled any new onsite training although I am still volunteering at conferences and doing the full day crash courses; I perform them for free although sometime the conference organizers offset the cost to cover the rooms, etc. I *think* I will be at BSides DC again this year (4 years running!); it will be easier this year as I'm relocating to Northern Virginia. We've done 2 days of classes there the last few years. If you run a conference hit me up privately and we'll see if we can get scheduled, I really enjoy helping folks get into NSM. Working on getting some additional options out into the community, including the long discussed NSM Con- already have commits from a bunch of folks! I'll make sure to announce them on twitter. Thanks, Liam Randall On Mon, Jul 11, 2016 at 12:55 PM, Jordi Ros-Giralt wrote: > Reservoir Labs is also providing Bro training courses in case you are > interested. > > Thanks, > > Jordi Ros-Giralt, PhD > Reservoir Labs > +1 212 780 0527 x110 > > > On Thu, Jul 7, 2016 at 3:19 PM, K2 wrote: > >> Anyone have any leads on anybody available for onsite Bro training in >> the US? >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160711/def720dd/attachment-0001.html From af7 at umbc.edu Tue Jul 12 08:16:06 2016 From: af7 at umbc.edu (Arash Fallah) Date: Tue, 12 Jul 2016 11:16:06 -0400 Subject: [Bro] Bro Not Extracting Host Fields from HTTP Traffic Message-ID: I'm having an issue where Bro is not extracting the host field correctly from captured HTTP traffic (in the form of a PCAP). I've verified it has nothing to do with split-routing. I also manually examined the PCAP file using Wireshark and found the host field to be present in all instances. I am a bit puzzled. This is significant for our use case because we will be using Bro to monitor for malicious URLs and the like. I have my http.log, weird.log, and the PCAP file itself. Unfortunately, I cannot attach the PCAP due to its size and the mail list rejecting the message. Please reply and I will send the PCAP individually. Any advice is appreciated. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160712/05cf5d18/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: http.log Type: text/x-log Size: 13961 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160712/05cf5d18/attachment-0002.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: weird.log Type: text/x-log Size: 36202 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160712/05cf5d18/attachment-0003.bin From jazoff at illinois.edu Tue Jul 12 08:24:11 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 12 Jul 2016 15:24:11 +0000 Subject: [Bro] Bro Not Extracting Host Fields from HTTP Traffic In-Reply-To: References: Message-ID: <490B7C10-C5CF-4411-86AD-CE30EF458A15@illinois.edu> It looks like it is missing the entire request. It's not missing the host field, it's missing every single field from the client request. method,host,uri,referrer,user_agent,request_body_len are all missing. Are you running bro on the machine making the outbound connections? I'm guessing that 130.85.70.132 is your desktop machine. If you look inside your reporter.log, is there a warning about tcp checksums? -- - Justin Azoff > On Jul 12, 2016, at 11:16 AM, Arash Fallah wrote: > > I'm having an issue where Bro is not extracting the host field correctly from captured HTTP traffic (in the form of a PCAP). I've verified it has nothing to do with split-routing. I also manually examined the PCAP file using Wireshark and found the host field to be present in all instances. I am a bit puzzled. This is significant for our use case because we will be using Bro to monitor for malicious URLs and the like. > > I have my http.log, weird.log, and the PCAP file itself. Unfortunately, I cannot attach the PCAP due to its size and the mail list rejecting the message. Please reply and I will send the PCAP individually. > > Any advice is appreciated. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From af7 at umbc.edu Tue Jul 12 09:26:25 2016 From: af7 at umbc.edu (Arash Fallah) Date: Tue, 12 Jul 2016 12:26:25 -0400 Subject: [Bro] Bro Not Extracting Host Fields from HTTP Traffic In-Reply-To: <490B7C10-C5CF-4411-86AD-CE30EF458A15@illinois.edu> References: <490B7C10-C5CF-4411-86AD-CE30EF458A15@illinois.edu> Message-ID: Thanks Justin, that was the problem. I have two follow-up questions. Can a NIC card handle calculating checksums for all packets instead of offloading to the CPU or would disabling offloading result in dropped packets? Is it preferable to have Bro ignore the checksums instead? I understand this is a general question but I'm having trouble benchmarking a 10Gb/s capture card. On Tue, Jul 12, 2016 at 11:24 AM, Azoff, Justin S wrote: > It looks like it is missing the entire request. It's not missing the host > field, it's missing every single field from the client request. > > method,host,uri,referrer,user_agent,request_body_len are all missing. > > Are you running bro on the machine making the outbound connections? I'm > guessing that 130.85.70.132 is your desktop machine. > > If you look inside your reporter.log, is there a warning about tcp > checksums? > > -- > - Justin Azoff > > > On Jul 12, 2016, at 11:16 AM, Arash Fallah wrote: > > > > I'm having an issue where Bro is not extracting the host field correctly > from captured HTTP traffic (in the form of a PCAP). I've verified it has > nothing to do with split-routing. I also manually examined the PCAP file > using Wireshark and found the host field to be present in all instances. I > am a bit puzzled. This is significant for our use case because we will be > using Bro to monitor for malicious URLs and the like. > > > > I have my http.log, weird.log, and the PCAP file itself. Unfortunately, > I cannot attach the PCAP due to its size and the mail list rejecting the > message. Please reply and I will send the PCAP individually. > > > > Any advice is appreciated. > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160712/99760d96/attachment.html From jazoff at illinois.edu Tue Jul 12 09:29:19 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 12 Jul 2016 16:29:19 +0000 Subject: [Bro] Bro Not Extracting Host Fields from HTTP Traffic In-Reply-To: References: <490B7C10-C5CF-4411-86AD-CE30EF458A15@illinois.edu> Message-ID: <460F8137-64A7-4027-AEDC-797C0EF98004@illinois.edu> > On Jul 12, 2016, at 12:26 PM, Arash Fallah wrote: > > Thanks Justin, that was the problem. > > I have two follow-up questions. Can a NIC card handle calculating checksums for all packets instead of offloading to the CPU or would disabling offloading result in dropped packets? Is it preferable to have Bro ignore the checksums instead? > > I understand this is a general question but I'm having trouble benchmarking a 10Gb/s capture card. This is only an issue when you are running bro on the same machine that is generating the traffic. Hook up bro to a tap or a span port and the checksums will be correct. -- - Justin Azoff From borchgrevink at aggienetwork.com Tue Jul 12 12:26:34 2016 From: borchgrevink at aggienetwork.com (Connor Borchgrevink) Date: Tue, 12 Jul 2016 14:26:34 -0500 Subject: [Bro] Implementing a new script Message-ID: <5785446A.7060103@aggienetwork.com> Howdy all, I recently began poking around Bro and had my first attempt of writing a script. The purpose of it was to detect whether or not traffic on port 443 used SSL, the moment I implemented the script my CPU usage was at 100% and the same happened to my memory. Without the script enabled, the machine runs fine and bro only uses about 50-60% of the resources. I also tried to turn off every other thing in local.bro but it was the same result. The code is as follows (be warned its pretty rough): @load base/protocols/ssl @load base/frameworks/notice @load base/protocols/conn @load base/utils/directions-and-hosts module conn; export { redef enum Notice::Type += { Unencrypted_Traffic }; const List_of_Hosts = LOCAL_HOSTS &redef; const Encryption = "SSL" &redef; } event new_connection(c: connection) &priority=3 { if ( ! addr_matches_host(c$id$resp_h, List_of_Hosts) ) return; local port_number=c$conn$id$resp_p; local ip_address=c$conn$id$resp_h; local encrypted=c$conn$service; if ( port_number != 443/tcp ) return; if ( encrypted != Encryption ) NOTICE([$note=Unencrypted_Traffic, $msg=fmt("Unencrypted traffic"), $conn=c, $identifier=cat(c$id$resp_h, c$id$resp_p) ]); } It is probably something I am not catching in my code or a limited knowledge of bro, but any help is much appreciated. Thanks Connor -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160712/2976a084/attachment.html From k2 at korrosivesecurity.com Tue Jul 12 13:36:50 2016 From: k2 at korrosivesecurity.com (K2) Date: Tue, 12 Jul 2016 15:36:50 -0500 Subject: [Bro] Onsite Training Options? In-Reply-To: References: <1467897597.2878480.659499673.56DE6795@webmail.messagingengine.com> Message-ID: <1468355810.2900747.664404153.7AB1C61C@webmail.messagingengine.com> Thanks for the recommendations everyone. I've gotten a couple of promising leads and will be checking into those in addition to looking to attend SOcon and BroCon. Kory On Mon, Jul 11, 2016, at 12:39 PM, Liam Randall wrote: > Hey Guys, > > Thanks for all of the great feedback. > > We have not scheduled any new onsite training although I am still > volunteering at conferences and doing the full day crash courses; I > perform them for free although sometime the conference organizers > offset the cost to cover the rooms, etc. > > I *think* I will be at BSides DC again this year (4 years running!); > it will be easier this year as I'm relocating to Northern Virginia. > We've done 2 days of classes there the last few years. If you run a > conference hit me up privately and we'll see if we can get scheduled, > I really enjoy helping folks get into NSM. > > Working on getting some additional options out into the community, > including the long discussed NSM Con- already have commits from a > bunch of folks! > > I'll make sure to announce them on twitter. > > Thanks, > > Liam Randall > > > > > > On Mon, Jul 11, 2016 at 12:55 PM, Jordi Ros-Giralt > wrote: >> Reservoir Labs is also providing Bro training courses in case you are >> interested. >> >> Thanks, >> >> >> Jordi Ros-Giralt, PhD Reservoir Labs +1 212 780 0527 x110[1] >> >> >> On Thu, Jul 7, 2016 at 3:19 PM, K2 wrote: >>> Anyone have any leads on anybody available for onsite Bro >>> training in >>> the US? >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro ? Links: 1. tel:%2B1%20212%20780%200527%20x110 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160712/f8a69965/attachment.html From seth at icir.org Wed Jul 13 08:25:57 2016 From: seth at icir.org (Seth Hall) Date: Wed, 13 Jul 2016 11:25:57 -0400 Subject: [Bro] Implementing a new script In-Reply-To: <5785446A.7060103@aggienetwork.com> References: <5785446A.7060103@aggienetwork.com> Message-ID: <5A8BD3E4-DB28-416A-B5AF-597AEB52E6D5@icir.org> Hi Connor, This is a nice job at a first script. I'll point out a few issues I noticed with it, but generally I think you got pretty close to what you wanted.  Bro won't have detected the protocol in the connection yet in the new_connection event. Generally you can assume that no payload has been seen when the new_connection event is generated which means that we don't yet know if the traffic is encrypted or not. :) You could use the protocol_violation event to watch for the SSL analyzer to be violated. Port 443/tcp always gets the SSL analyzer attached as a heuristic but it will fail if the traffic is not actually SSL. This might actually be the most straightforward mechanism. https://www.bro.org/sphinx/scripts/base/bif/event.bif.bro.html#id-protocol_violation One other tiny mistake is that you're comparing the c$conn$service field against "SSL" but the value in the service field will be "ssl". You just need to fix your casing. Nice script! .Seth > On Jul 12, 2016, at 3:26 PM, Connor Borchgrevink wrote: > > Howdy all, > > I recently began poking around Bro and had my first attempt of writing a script. The purpose of it was to detect whether or not traffic on port 443 used SSL, the moment I implemented the script my CPU usage was at 100% and the same happened to my memory. Without the script enabled, the machine runs fine and bro only uses about 50-60% of the resources. I also tried to turn off every other thing in local.bro but it was the same result. The code is as follows (be warned its pretty rough): > > @load base/protocols/ssl > @load base/frameworks/notice > @load base/protocols/conn > @load base/utils/directions-and-hosts > > module conn; > > export { > redef enum Notice::Type += { > Unencrypted_Traffic > }; > > const List_of_Hosts = LOCAL_HOSTS &redef; > const Encryption = "SSL" &redef; > } > > event new_connection(c: connection) &priority=3 > { > if ( ! addr_matches_host(c$id$resp_h, List_of_Hosts) ) > return; > > local port_number=c$conn$id$resp_p; > local ip_address=c$conn$id$resp_h; > local encrypted=c$conn$service; > > if ( port_number != 443/tcp ) > return; > > if ( encrypted != Encryption ) > NOTICE([$note=Unencrypted_Traffic, > $msg=fmt("Unencrypted traffic"), > $conn=c, > $identifier=cat(c$id$resp_h, c$id$resp_p) > ]); > } > > > It is probably something I am not catching in my code or a limited knowledge of bro, but any help is much appreciated. > > Thanks > > Connor > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From obdnanr at gmail.com Wed Jul 13 11:00:33 2016 From: obdnanr at gmail.com (Obndnar smith) Date: Wed, 13 Jul 2016 18:00:33 +0000 Subject: [Bro] Distributed deployment high cpu low network traffic Message-ID: We're getting around 30-160 mbps worth of traffic on our sensor and when we use multiple workers they are all pegged 96% to 99% cpu utilization. I've tried disabling our custom scripts and that wasn't the culprit. I've tried 4 workers and 8 workers, I've tried pinning them to different CPUs and nothing has changed anything. We're using PF_RING. Any idea how I can trouble shoot this issue? We're having packet loss, but with the low speed of traffic I wouldn't expect to see any loss. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 32027 root 20 0 1527600 1.430g 11696 R 99.9 4.7 10:37.82 bro 32018 root 20 0 1520192 1.423g 11792 R 99.6 4.6 10:38.83 bro 32019 root 20 0 1525308 1.428g 11780 R 99.6 4.7 10:39.06 bro 32032 root 20 0 1522564 1.425g 11712 R 99.6 4.6 10:39.20 bro 32002 root 20 0 1571588 1.472g 11716 R 98.9 4.8 10:36.93 bro 32029 root 20 0 1529472 1.432g 11788 R 98.9 4.7 10:38.50 bro 32016 root 20 0 1341716 1.252g 11712 R 97.9 4.1 10:33.21 bro 32006 root 20 0 1422616 1.330g 11712 R 96.0 4.3 10:20.19 bro Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160713/219269c5/attachment.html From jazoff at illinois.edu Wed Jul 13 11:08:10 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 13 Jul 2016 18:08:10 +0000 Subject: [Bro] Distributed deployment high cpu low network traffic In-Reply-To: References: Message-ID: <19471C08-D299-422C-A5EB-B033A11AD481@illinois.edu> First verify that you are truly using pf_ring.. $ ldd `which bro`|grep -i pcap libpcap.so.1 => /opt/pfring/lib/libpcap.so.1 (0x00007f70757a6000) If bro is not actually linked against pf_ring you will end up running 8 workers that are all seeing 100% of the traffic. While bro is running you should also have a bunch of files in /proc/net/pf_ring/, one for each worker+nic. -- - Justin Azoff > On Jul 13, 2016, at 2:00 PM, Obndnar smith wrote: > > We're getting around 30-160 mbps worth of traffic on our sensor and when we use multiple workers they are all pegged 96% to 99% cpu utilization. I've tried disabling our custom scripts and that wasn't the culprit. I've tried 4 workers and 8 workers, I've tried pinning them to different CPUs and nothing has changed anything. We're using PF_RING. Any idea how I can trouble shoot this issue? We're having packet loss, but with the low speed of traffic I wouldn't expect to see any loss. > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 32027 root 20 0 1527600 1.430g 11696 R 99.9 4.7 10:37.82 bro > 32018 root 20 0 1520192 1.423g 11792 R 99.6 4.6 10:38.83 bro > 32019 root 20 0 1525308 1.428g 11780 R 99.6 4.7 10:39.06 bro > 32032 root 20 0 1522564 1.425g 11712 R 99.6 4.6 10:39.20 bro > 32002 root 20 0 1571588 1.472g 11716 R 98.9 4.8 10:36.93 bro > 32029 root 20 0 1529472 1.432g 11788 R 98.9 4.7 10:38.50 bro > 32016 root 20 0 1341716 1.252g 11712 R 97.9 4.1 10:33.21 bro > 32006 root 20 0 1422616 1.330g 11712 R 96.0 4.3 10:20.19 bro > > Thanks > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jazoff at illinois.edu Wed Jul 13 13:14:57 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 13 Jul 2016 20:14:57 +0000 Subject: [Bro] Distributed deployment high cpu low network traffic In-Reply-To: References: <19471C08-D299-422C-A5EB-B033A11AD481@illinois.edu> Message-ID: > On Jul 13, 2016, at 4:08 PM, Obndnar smith wrote: > > It's not using pf_ring! I recompiled it like the directions said, any advice on getting it to work? > > libpcap.so.0.8 => /usr/lib/x86_64-linux-gnu/libpcap.so.0.8 (0x00007f90c13c7000) try deleting the build/ directory in the bro source tree and then running ./configure again with the right options. As part of the configure output it will say which pcap it is using. We install pf_ring into /opt/pfring so we build bro using --with-pcap=/opt/pfring -- - Justin Azoff From bmixonb1 at cs.unm.edu Wed Jul 13 15:36:16 2016 From: bmixonb1 at cs.unm.edu (Ben Mixon-Baca) Date: Wed, 13 Jul 2016 15:36:16 -0700 Subject: [Bro] First orig_h packet after 3 way handshake Message-ID: <5786C260.4000500@cs.unm.edu> Does Bro have an event that will get fired for the first packet after the tcp 3-way handshake, or is there a way to get at that easily or does it require a lot of state to be maintained in the script? I am trying to get at this first packet following the 3 way handshake because that is where the client hello in the ssl handshake should be. -- Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: OpenPGP digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160713/f6035abc/attachment.bin From jazoff at illinois.edu Wed Jul 13 15:58:32 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 13 Jul 2016 22:58:32 +0000 Subject: [Bro] First orig_h packet after 3 way handshake In-Reply-To: <5786C260.4000500@cs.unm.edu> References: <5786C260.4000500@cs.unm.edu> Message-ID: > On Jul 13, 2016, at 6:36 PM, Ben Mixon-Baca wrote: > > Does Bro have an event that will get fired for the first packet after > the tcp 3-way handshake, or is there a way to get at that easily or does > it require a lot of state to be maintained in the script? > > I am trying to get at this first packet following the 3 way handshake > because that is where the client hello in the ssl handshake should be. Can you use the ssl_client_hello event? event ssl_client_hello(c: connection, version: count, possible_ts: time, client_random: string, session_id: string, ciphers: index_vec) -- - Justin Azoff From bmixonb1 at cs.unm.edu Wed Jul 13 16:04:05 2016 From: bmixonb1 at cs.unm.edu (Ben Mixon-Baca) Date: Wed, 13 Jul 2016 16:04:05 -0700 Subject: [Bro] First orig_h packet after 3 way handshake In-Reply-To: References: <5786C260.4000500@cs.unm.edu> Message-ID: <5786C8E5.8000909@cs.unm.edu> Unfortunately for what I am doing, I cannot. On 07/13/2016 03:58 PM, Azoff, Justin S wrote: > >> On Jul 13, 2016, at 6:36 PM, Ben Mixon-Baca wrote: >> >> Does Bro have an event that will get fired for the first packet after >> the tcp 3-way handshake, or is there a way to get at that easily or does >> it require a lot of state to be maintained in the script? >> >> I am trying to get at this first packet following the 3 way handshake >> because that is where the client hello in the ssl handshake should be. > > Can you use the ssl_client_hello event? > > event ssl_client_hello(c: connection, version: count, possible_ts: time, client_random: string, session_id: string, ciphers: index_vec) > -- Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: OpenPGP digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160713/18508c16/attachment.bin From johanna at icir.org Wed Jul 13 17:17:08 2016 From: johanna at icir.org (Johanna Amann) Date: Wed, 13 Jul 2016 17:17:08 -0700 Subject: [Bro] First orig_h packet after 3 way handshake In-Reply-To: <5786C8E5.8000909@cs.unm.edu> References: <5786C260.4000500@cs.unm.edu> <5786C8E5.8000909@cs.unm.edu> Message-ID: <5902E9B6-2801-4A67-AAC1-A385AE5728F2@icir.org> Out of curiosity - what are you trying to do? (I am always curious what people try to get from the SSL handshake that we do not parse out yet...) Johanna On 13 Jul 2016, at 16:04, Ben Mixon-Baca wrote: > Unfortunately for what I am doing, I cannot. > > On 07/13/2016 03:58 PM, Azoff, Justin S wrote: >> >>> On Jul 13, 2016, at 6:36 PM, Ben Mixon-Baca >>> wrote: >>> >>> Does Bro have an event that will get fired for the first packet >>> after >>> the tcp 3-way handshake, or is there a way to get at that easily or >>> does >>> it require a lot of state to be maintained in the script? >>> >>> I am trying to get at this first packet following the 3 way >>> handshake >>> because that is where the client hello in the ssl handshake should >>> be. >> >> Can you use the ssl_client_hello event? >> >> event ssl_client_hello(c: connection, version: count, possible_ts: >> time, client_random: string, session_id: string, ciphers: index_vec) >> > > -- > Ben > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From philosnef at yahoo.com Thu Jul 14 06:33:50 2016 From: philosnef at yahoo.com (philosnef) Date: Thu, 14 Jul 2016 13:33:50 +0000 (UTC) Subject: [Bro] problem ingesting bro json logs into splunk References: <1840447497.3042597.1468503230969.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <1840447497.3042597.1468503230969.JavaMail.yahoo@mail.yahoo.com> We are getting a spurious sourcetype when ingesting bro json logs into splunk. Specifically, we are getting a sourcetype of bro_00. There is no log file named this, and the splunkforwarder is just pushing the raw logs for indexing into splunk. There is no massaging of the log data. Anyone know why this sourcetype is popping up? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/ec923f54/attachment.html From lattin at umn.edu Thu Jul 14 07:14:25 2016 From: lattin at umn.edu (Brandon Lattin) Date: Thu, 14 Jul 2016 09:14:25 -0500 Subject: [Bro] problem ingesting bro json logs into splunk In-Reply-To: <1840447497.3042597.1468503230969.JavaMail.yahoo@mail.yahoo.com> References: <1840447497.3042597.1468503230969.JavaMail.yahoo.ref@mail.yahoo.com> <1840447497.3042597.1468503230969.JavaMail.yahoo@mail.yahoo.com> Message-ID: Do you have the Splunk installed? (https://splunkbase.splunk.com/app/1617/) The TA will dynamically create sourcetypes based on the log name. # Dynamic source typing based on log filename # Match: conn.log, bro.conn.log, # md5.bro.conn.log, whatever.conn.log [BroAutoType] DEST_KEY = MetaData:Sourcetype SOURCE_KEY = MetaData:Source REGEX = ([a-zA-Z0-9-]+)(?:\.[0-9-]*)?(?:\.[0-9\:-]*)?\.log FORMAT = sourcetype::bro_$1 WRITE_META = true On Thu, Jul 14, 2016 at 8:33 AM, philosnef wrote: > We are getting a spurious sourcetype when ingesting bro json logs into > splunk. > > Specifically, we are getting a sourcetype of bro_00. There is no log file > named this, and the splunkforwarder is just pushing the raw logs for > indexing into splunk. There is no massaging of the log data. Anyone know > why this sourcetype is popping up? > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Brandon Lattin Security Analyst University of Minnesota - University Information Security Office: 612-626-6672 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/b73d7058/attachment-0001.html From philosnef at yahoo.com Thu Jul 14 07:23:32 2016 From: philosnef at yahoo.com (philosnef) Date: Thu, 14 Jul 2016 14:23:32 +0000 (UTC) Subject: [Bro] problem ingesting bro json logs into splunk In-Reply-To: References: <1840447497.3042597.1468503230969.JavaMail.yahoo.ref@mail.yahoo.com> <1840447497.3042597.1468503230969.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1483283772.3249400.1468506212328.JavaMail.yahoo@mail.yahoo.com> There are no 00.log files in Bro, so the automatic generation of the sourcetype bro_00 makes no sense. It does not follow the standard sourcetype pinning that all the other log files generate. find . -name "00*" in the parent logs directory reports zero logs of this type. This only occured when we moved off of Bro standard log format to JSON format. On Thursday, July 14, 2016 10:14 AM, Brandon Lattin wrote: Do you have the Splunk installed? (https://splunkbase.splunk.com/app/1617/) The TA will dynamically create sourcetypes based on the log name. # Dynamic source typing based on log filename# Match: conn.log, bro.conn.log,?# md5.bro.conn.log, whatever.conn.log[BroAutoType]DEST_KEY = MetaData:SourcetypeSOURCE_KEY = MetaData:SourceREGEX = ([a-zA-Z0-9-]+)(?:\.[0-9-]*)?(?:\.[0-9\:-]*)?\.logFORMAT = sourcetype::bro_$1WRITE_META = true On Thu, Jul 14, 2016 at 8:33 AM, philosnef wrote: We are getting a spurious sourcetype when ingesting bro json logs into splunk. Specifically, we are getting a sourcetype of bro_00. There is no log file named this, and the splunkforwarder is just pushing the raw logs for indexing into splunk. There is no massaging of the log data. Anyone know why this sourcetype is popping up? _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Brandon LattinSecurity Analyst University of Minnesota - University Information Security Office: 612-626-6672 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/397f5d77/attachment.html From gross.b at ghc.org Thu Jul 14 08:33:57 2016 From: gross.b at ghc.org (Gross, Brett) Date: Thu, 14 Jul 2016 15:33:57 +0000 Subject: [Bro] problem ingesting bro json logs into splunk In-Reply-To: <1483283772.3249400.1468506212328.JavaMail.yahoo@mail.yahoo.com> References: <1840447497.3042597.1468503230969.JavaMail.yahoo.ref@mail.yahoo.com> <1840447497.3042597.1468503230969.JavaMail.yahoo@mail.yahoo.com> <1483283772.3249400.1468506212328.JavaMail.yahoo@mail.yahoo.com> Message-ID: We?ve used bro and splunk at our organization for a couple years now. We utilize the Splunk props and transforms configs to ingest the bro log in the format we want or with the additional attributes and aliases. transforms.conf: [remove_hash_comments] REGEX = ^#.* DEST_KEY = queue FORMAT = nullQueue [bro_conn_extractions] DELIMS = "\t" FIELDS = ts,uid,id.orig_h,id.orig_p,id.resp_h,id.resp_p,proto,service,duration,orig_bytes,resp_bytes,conn_state,local_orig,local_resp,missed_bytes,history,orig_pkts,orig_ip_bytes,resp_pkts,resp_ip_bytes,tunnel_parents,orig_cc,resp_cc,sensorname Props.conf: [bro_conn] REPORT-bro_conn_extract = bro_conn_extractions TRANSFORMS-sourcetype = remove_hash_comments SHOULD_LINEMERGE = false TRUNCATE = 0 KV_MODE = none MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s.%6N Inputs.conf: [monitor:///your_log_path/log/bro/conn.log] index = bro_conn sourcetype = bro_conn _TCP_ROUTING = primary_indexers brett From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of philosnef Sent: Thursday, July 14, 2016 7:24 AM Cc: bro at bro.org Subject: Re: [Bro] problem ingesting bro json logs into splunk There are no 00.log files in Bro, so the automatic generation of the sourcetype bro_00 makes no sense. It does not follow the standard sourcetype pinning that all the other log files generate. find . -name "00*" in the parent logs directory reports zero logs of this type. This only occured when we moved off of Bro standard log format to JSON format. On Thursday, July 14, 2016 10:14 AM, Brandon Lattin > wrote: Do you have the Splunk installed? (https://splunkbase.splunk.com/app/1617/) The TA will dynamically create sourcetypes based on the log name. # Dynamic source typing based on log filename # Match: conn.log, bro.conn.log, # md5.bro.conn.log, whatever.conn.log [BroAutoType] DEST_KEY = MetaData:Sourcetype SOURCE_KEY = MetaData:Source REGEX = ([a-zA-Z0-9-]+)(?:\.[0-9-]*)?(?:\.[0-9\:-]*)?\.log FORMAT = sourcetype::bro_$1 WRITE_META = true On Thu, Jul 14, 2016 at 8:33 AM, philosnef > wrote: We are getting a spurious sourcetype when ingesting bro json logs into splunk. Specifically, we are getting a sourcetype of bro_00. There is no log file named this, and the splunkforwarder is just pushing the raw logs for indexing into splunk. There is no massaging of the log data. Anyone know why this sourcetype is popping up? _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Brandon Lattin Security Analyst University of Minnesota - University Information Security Office: 612-626-6672 ________________________________ GHC Confidentiality Statement This message and any attached files might contain confidential information protected by federal and state law. The information is intended only for the use of the individual(s) or entities originally named as addressees. The improper disclosure of such information may be subject to civil or criminal penalties. If this message reached you in error, please contact the sender and destroy this message. Disclosing, copying, forwarding, or distributing the information by unauthorized individuals or entities is strictly prohibited by law. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/05d86df4/attachment-0001.html From jazoff at illinois.edu Thu Jul 14 08:59:48 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 14 Jul 2016 15:59:48 +0000 Subject: [Bro] problem ingesting bro json logs into splunk In-Reply-To: References: <1840447497.3042597.1468503230969.JavaMail.yahoo.ref@mail.yahoo.com> <1840447497.3042597.1468503230969.JavaMail.yahoo@mail.yahoo.com> <1483283772.3249400.1468506212328.JavaMail.yahoo@mail.yahoo.com> Message-ID: <5E8320E9-58A8-42F6-8097-B0E7066FF7AF@illinois.edu> > On Jul 14, 2016, at 11:33 AM, Gross, Brett wrote: > > We?ve used bro and splunk at our organization for a couple years now. We utilize the Splunk props and transforms configs to ingest the bro log in the format we want or with the additional attributes and aliases. Ah, that's for the tab delimited logs, not the json logs though. I actually did it that way for years, I even have a python program that helps you generate the config: https://github.com/JustinAzoff/bro_scripts/blob/2.0/generate_splunk_configs.py But, I wouldn't use this method - the splunk TA app for bro is better. As far as I know the transforms/props method only does the field lookups at search time, not at index time like the TA app configures. Whenever the bro logs change and a column is added or removed, all those search time field lookups break. -- - Justin Azoff From lattin at umn.edu Thu Jul 14 09:06:43 2016 From: lattin at umn.edu (Brandon Lattin) Date: Thu, 14 Jul 2016 11:06:43 -0500 Subject: [Bro] problem ingesting bro json logs into splunk In-Reply-To: <1483283772.3249400.1468506212328.JavaMail.yahoo@mail.yahoo.com> References: <1840447497.3042597.1468503230969.JavaMail.yahoo.ref@mail.yahoo.com> <1840447497.3042597.1468503230969.JavaMail.yahoo@mail.yahoo.com> <1483283772.3249400.1468506212328.JavaMail.yahoo@mail.yahoo.com> Message-ID: The Bro TA is assuming TSV extractions. The move to JSON probably is causing the Splunk auto-sourcetyper to do some funky things. [source::...bro.*.log] SHOULD_LINEMERGE = false TRUNCATE = 0 MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s.%6N TRANSFORMS-BroAutoType = BroAutoType, TrashComments INDEXED_EXTRACTIONS = TSV FIELD_HEADER_REGEX = ^#fields\t(.*) FIELD_DELIMITER = \t FIELD_QUOTE = \t On Thu, Jul 14, 2016 at 9:23 AM, philosnef wrote: > There are no 00.log files in Bro, so the automatic generation of the > sourcetype bro_00 makes no sense. It does not follow the standard > sourcetype pinning that all the other log files generate. find . -name > "00*" in the parent logs directory reports zero logs of this type. This > only occured when we moved off of Bro standard log format to JSON format. > > > On Thursday, July 14, 2016 10:14 AM, Brandon Lattin > wrote: > > > Do you have the Splunk installed? (https://splunkbase.splunk.com/app/1617/ > ) > > The TA will dynamically create sourcetypes based on the log name. > > # Dynamic source typing based on log filename > # Match: conn.log, bro.conn.log, > # md5.bro.conn.log, whatever.conn.log > [BroAutoType] > DEST_KEY = MetaData:Sourcetype > SOURCE_KEY = MetaData:Source > REGEX = ([a-zA-Z0-9-]+)(?:\.[0-9-]*)?(?:\.[0-9\:-]*)?\.log > FORMAT = sourcetype::bro_$1 > WRITE_META = true > > > On Thu, Jul 14, 2016 at 8:33 AM, philosnef wrote: > > We are getting a spurious sourcetype when ingesting bro json logs into > splunk. > > Specifically, we are getting a sourcetype of bro_00. There is no log file > named this, and the splunkforwarder is just pushing the raw logs for > indexing into splunk. There is no massaging of the log data. Anyone know > why this sourcetype is popping up? > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > > > -- > Brandon Lattin > Security Analyst > University of Minnesota - University Information Security > Office: 612-626-6672 > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Brandon Lattin Security Analyst University of Minnesota - University Information Security Office: 612-626-6672 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/051c6aa3/attachment.html From dwdixon at umich.edu Thu Jul 14 09:14:44 2016 From: dwdixon at umich.edu (Drew Dixon) Date: Thu, 14 Jul 2016 12:14:44 -0400 Subject: [Bro] problem ingesting bro json logs into splunk In-Reply-To: <5E8320E9-58A8-42F6-8097-B0E7066FF7AF@illinois.edu> References: <1840447497.3042597.1468503230969.JavaMail.yahoo.ref@mail.yahoo.com> <1840447497.3042597.1468503230969.JavaMail.yahoo@mail.yahoo.com> <1483283772.3249400.1468506212328.JavaMail.yahoo@mail.yahoo.com> <5E8320E9-58A8-42F6-8097-B0E7066FF7AF@illinois.edu> Message-ID: Sorry hope I'm not hijacking- quick question very closely related to this...is the Splunk app for Bro that Brandon linked to here supposed to parse out all the various bro 2.4.1 log types' fields correctly? In other words, is the latest version of the Splunk app fro Bro/TA supposed to work properly for parsing out Bro log fields with they way the log fields/columns etc. are now in Bro 2.4.1? I think the Splunk Add-on for Bro IDS was written for Bro 2.1 or 2.2...do changes that were made in subsequent versions of Bro such as 2.4.1 break the fields being parsed out in Splunk when using this Splunk Add-on for Bro/Bro TA in Splunkbase? Or does Splunk need to update the add-on to work properly with Bro 2.4.1? Thank you, -Drew On Thu, Jul 14, 2016 at 11:59 AM, Azoff, Justin S wrote: > > > On Jul 14, 2016, at 11:33 AM, Gross, Brett wrote: > > > > We?ve used bro and splunk at our organization for a couple years now. We > utilize the Splunk props and transforms configs to ingest the bro log in > the format we want or with the additional attributes and aliases. > > Ah, that's for the tab delimited logs, not the json logs though. I > actually did it that way for years, I even have a python program that helps > you generate the config: > > > https://github.com/JustinAzoff/bro_scripts/blob/2.0/generate_splunk_configs.py > > But, I wouldn't use this method - the splunk TA app for bro is better. > > As far as I know the transforms/props method only does the field lookups > at search time, not at index time like the TA app configures. > > Whenever the bro logs change and a column is added or removed, all those > search time field lookups break. > > > -- > - Justin Azoff > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/9e2cc4ae/attachment.html From philosnef at yahoo.com Thu Jul 14 09:16:24 2016 From: philosnef at yahoo.com (philosnef) Date: Thu, 14 Jul 2016 16:16:24 +0000 (UTC) Subject: [Bro] problem ingesting bro json logs into splunk In-Reply-To: References: <1840447497.3042597.1468503230969.JavaMail.yahoo.ref@mail.yahoo.com> <1840447497.3042597.1468503230969.JavaMail.yahoo@mail.yahoo.com> <1483283772.3249400.1468506212328.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1305592882.3102885.1468512984462.JavaMail.yahoo@mail.yahoo.com> The problem with the Spunk app is that indexing is occuring at time of ingest. This causes the indices of the Bro data to grow extremely fast. Using json and not the Bro app means that the data is indexed by Splunk, resulting in far smaller indices on the splunk indexing servers. This is specifically why we moved away from TSV and to JSON, since it was nuking disk storage for those indices... On Thursday, July 14, 2016 12:07 PM, Brandon Lattin wrote: The Bro TA is assuming TSV extractions. The move to JSON probably is causing the Splunk auto-sourcetyper to do some funky things. [source::...bro.*.log]SHOULD_LINEMERGE = falseTRUNCATE = 0MAX_TIMESTAMP_LOOKAHEAD = 20TIME_FORMAT = %s.%6NTRANSFORMS-BroAutoType = BroAutoType, TrashCommentsINDEXED_EXTRACTIONS = TSVFIELD_HEADER_REGEX = ^#fields\t(.*)FIELD_DELIMITER = \tFIELD_QUOTE = \t On Thu, Jul 14, 2016 at 9:23 AM, philosnef wrote: There are no 00.log files in Bro, so the automatic generation of the sourcetype bro_00 makes no sense. It does not follow the standard sourcetype pinning that all the other log files generate. find . -name "00*" in the parent logs directory reports zero logs of this type. This only occured when we moved off of Bro standard log format to JSON format. On Thursday, July 14, 2016 10:14 AM, Brandon Lattin wrote: Do you have the Splunk installed? (https://splunkbase.splunk.com/app/1617/) The TA will dynamically create sourcetypes based on the log name. # Dynamic source typing based on log filename# Match: conn.log, bro.conn.log,?# md5.bro.conn.log, whatever.conn.log[BroAutoType]DEST_KEY = MetaData:SourcetypeSOURCE_KEY = MetaData:SourceREGEX = ([a-zA-Z0-9-]+)(?:\.[0-9-]*)?(?:\.[0-9\:-]*)?\.logFORMAT = sourcetype::bro_$1WRITE_META = true On Thu, Jul 14, 2016 at 8:33 AM, philosnef wrote: We are getting a spurious sourcetype when ingesting bro json logs into splunk. Specifically, we are getting a sourcetype of bro_00. There is no log file named this, and the splunkforwarder is just pushing the raw logs for indexing into splunk. There is no massaging of the log data. Anyone know why this sourcetype is popping up? _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Brandon LattinSecurity Analyst University of Minnesota - University Information Security Office: 612-626-6672 _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Brandon LattinSecurity Analyst University of Minnesota - University Information Security Office: 612-626-6672 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/08eaeda9/attachment-0001.html From jazoff at illinois.edu Thu Jul 14 09:30:34 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 14 Jul 2016 16:30:34 +0000 Subject: [Bro] problem ingesting bro json logs into splunk In-Reply-To: <1305592882.3102885.1468512984462.JavaMail.yahoo@mail.yahoo.com> References: <1840447497.3042597.1468503230969.JavaMail.yahoo.ref@mail.yahoo.com> <1840447497.3042597.1468503230969.JavaMail.yahoo@mail.yahoo.com> <1483283772.3249400.1468506212328.JavaMail.yahoo@mail.yahoo.com> <1305592882.3102885.1468512984462.JavaMail.yahoo@mail.yahoo.com> Message-ID: <82D0EDF6-7064-4EC8-8679-7220C372569A@illinois.edu> > On Jul 14, 2016, at 12:16 PM, philosnef wrote: > > The problem with the Spunk app is that indexing is occuring at time of ingest. This causes the indices of the Bro data to grow extremely fast. Using json and not the Bro app means that the data is indexed by Splunk, resulting in far smaller indices on the splunk indexing servers. This is specifically why we moved away from TSV and to JSON, since it was nuking disk storage for those indices... Odd, I'd expect it to be about the same. The indexed data should be the same, and even though every json record includes the field names, they compress well. It's possible that the bro app indexing the fields individually is what makes the indexes larger... if you do something like id_resp_p=6379 (or whatever the field shows up as for you) does that find the records immediately, or does it have to scan through all the data? without individual field indexes you would have to do something like 6379 id_resp_p=6379 and hope that speeds it up, if you're trying to do something like id_orig_p=80 Then this will be pretty slow: 80 id_orig_p=80 -- - Justin Azoff From lattin at umn.edu Thu Jul 14 09:33:55 2016 From: lattin at umn.edu (Brandon Lattin) Date: Thu, 14 Jul 2016 11:33:55 -0500 Subject: [Bro] problem ingesting bro json logs into splunk In-Reply-To: References: <1840447497.3042597.1468503230969.JavaMail.yahoo.ref@mail.yahoo.com> <1840447497.3042597.1468503230969.JavaMail.yahoo@mail.yahoo.com> <1483283772.3249400.1468506212328.JavaMail.yahoo@mail.yahoo.com> <5E8320E9-58A8-42F6-8097-B0E7066FF7AF@illinois.edu> Message-ID: Drew, It should work just fine, assuming TSV headers are present, as it keys off the headers for the extractions. On Thu, Jul 14, 2016 at 11:14 AM, Drew Dixon wrote: > Sorry hope I'm not hijacking- quick question very closely related to > this...is the Splunk app for Bro that Brandon linked to here supposed to > parse out all the various bro 2.4.1 log types' fields correctly? > In other words, is the latest version of the Splunk app fro Bro/TA > supposed to work properly for parsing out Bro log fields with they way the > log fields/columns etc. are now in Bro 2.4.1? I think the Splunk Add-on for > Bro IDS was written for Bro 2.1 or 2.2...do changes that were made in > subsequent versions of Bro such as 2.4.1 break the fields being parsed out > in Splunk when using this Splunk Add-on for Bro/Bro TA in Splunkbase? Or > does Splunk need to update the add-on to work properly with Bro 2.4.1? > > Thank you, > > -Drew > > On Thu, Jul 14, 2016 at 11:59 AM, Azoff, Justin S > wrote: > >> >> > On Jul 14, 2016, at 11:33 AM, Gross, Brett wrote: >> > >> > We?ve used bro and splunk at our organization for a couple years now. >> We utilize the Splunk props and transforms configs to ingest the bro log in >> the format we want or with the additional attributes and aliases. >> >> Ah, that's for the tab delimited logs, not the json logs though. I >> actually did it that way for years, I even have a python program that helps >> you generate the config: >> >> >> https://github.com/JustinAzoff/bro_scripts/blob/2.0/generate_splunk_configs.py >> >> But, I wouldn't use this method - the splunk TA app for bro is better. >> >> As far as I know the transforms/props method only does the field lookups >> at search time, not at index time like the TA app configures. >> >> Whenever the bro logs change and a column is added or removed, all those >> search time field lookups break. >> >> >> -- >> - Justin Azoff >> >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Brandon Lattin Security Analyst University of Minnesota - University Information Security Office: 612-626-6672 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/3068c362/attachment.html From lattin at umn.edu Thu Jul 14 09:38:35 2016 From: lattin at umn.edu (Brandon Lattin) Date: Thu, 14 Jul 2016 11:38:35 -0500 Subject: [Bro] problem ingesting bro json logs into splunk In-Reply-To: <1305592882.3102885.1468512984462.JavaMail.yahoo@mail.yahoo.com> References: <1840447497.3042597.1468503230969.JavaMail.yahoo.ref@mail.yahoo.com> <1840447497.3042597.1468503230969.JavaMail.yahoo@mail.yahoo.com> <1483283772.3249400.1468506212328.JavaMail.yahoo@mail.yahoo.com> <1305592882.3102885.1468512984462.JavaMail.yahoo@mail.yahoo.com> Message-ID: I would assume your index volume (license usage) is significantly greater though? You're right, the raw to indexed ratio is abysmal with TSV (we're getting 0.29:1). On Thu, Jul 14, 2016 at 11:16 AM, philosnef wrote: > The problem with the Spunk app is that indexing is occuring at time of > ingest. This causes the indices of the Bro data to grow extremely fast. > Using json and not the Bro app means that the data is indexed by Splunk, > resulting in far smaller indices on the splunk indexing servers. This is > specifically why we moved away from TSV and to JSON, since it was nuking > disk storage for those indices... > > > On Thursday, July 14, 2016 12:07 PM, Brandon Lattin > wrote: > > > The Bro TA is assuming TSV extractions. The move to JSON probably is > causing the Splunk auto-sourcetyper to do some funky things. > > [source::...bro.*.log] > SHOULD_LINEMERGE = false > TRUNCATE = 0 > MAX_TIMESTAMP_LOOKAHEAD = 20 > TIME_FORMAT = %s.%6N > TRANSFORMS-BroAutoType = BroAutoType, TrashComments > INDEXED_EXTRACTIONS = TSV > FIELD_HEADER_REGEX = ^#fields\t(.*) > FIELD_DELIMITER = \t > FIELD_QUOTE = \t > > On Thu, Jul 14, 2016 at 9:23 AM, philosnef wrote: > > There are no 00.log files in Bro, so the automatic generation of the > sourcetype bro_00 makes no sense. It does not follow the standard > sourcetype pinning that all the other log files generate. find . -name > "00*" in the parent logs directory reports zero logs of this type. This > only occured when we moved off of Bro standard log format to JSON format. > > > On Thursday, July 14, 2016 10:14 AM, Brandon Lattin > wrote: > > > Do you have the Splunk installed? (https://splunkbase.splunk.com/app/1617/ > ) > > The TA will dynamically create sourcetypes based on the log name. > > # Dynamic source typing based on log filename > # Match: conn.log, bro.conn.log, > # md5.bro.conn.log, whatever.conn.log > [BroAutoType] > DEST_KEY = MetaData:Sourcetype > SOURCE_KEY = MetaData:Source > REGEX = ([a-zA-Z0-9-]+)(?:\.[0-9-]*)?(?:\.[0-9\:-]*)?\.log > FORMAT = sourcetype::bro_$1 > WRITE_META = true > > > On Thu, Jul 14, 2016 at 8:33 AM, philosnef wrote: > > We are getting a spurious sourcetype when ingesting bro json logs into > splunk. > > Specifically, we are getting a sourcetype of bro_00. There is no log file > named this, and the splunkforwarder is just pushing the raw logs for > indexing into splunk. There is no massaging of the log data. Anyone know > why this sourcetype is popping up? > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > > > -- > Brandon Lattin > Security Analyst > University of Minnesota - University Information Security > Office: 612-626-6672 > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > > > -- > Brandon Lattin > Security Analyst > University of Minnesota - University Information Security > Office: 612-626-6672 > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Brandon Lattin Security Analyst University of Minnesota - University Information Security Office: 612-626-6672 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/74ff59f8/attachment-0001.html From philosnef at yahoo.com Thu Jul 14 09:44:01 2016 From: philosnef at yahoo.com (philosnef) Date: Thu, 14 Jul 2016 16:44:01 +0000 (UTC) Subject: [Bro] problem ingesting bro json logs into splunk In-Reply-To: <82D0EDF6-7064-4EC8-8679-7220C372569A@illinois.edu> References: <1840447497.3042597.1468503230969.JavaMail.yahoo.ref@mail.yahoo.com> <1840447497.3042597.1468503230969.JavaMail.yahoo@mail.yahoo.com> <1483283772.3249400.1468506212328.JavaMail.yahoo@mail.yahoo.com> <1305592882.3102885.1468512984462.JavaMail.yahoo@mail.yahoo.com> <82D0EDF6-7064-4EC8-8679-7220C372569A@illinois.edu> Message-ID: <1431536404.3205916.1468514641628.JavaMail.yahoo@mail.yahoo.com> It is the TS IDX files in Splunk that grow out of control when using the Bro TSV app. Hope this helps for anyone interested. On Thursday, July 14, 2016 12:30 PM, "Azoff, Justin S" wrote: > On Jul 14, 2016, at 12:16 PM, philosnef wrote: > > The problem with the Spunk app is that indexing is occuring at time of ingest. This causes the indices of the Bro data to grow extremely fast. Using json and not the Bro app means that the data is indexed by Splunk, resulting in far smaller indices on the splunk indexing servers. This is specifically why we moved away from TSV and to JSON, since it was nuking disk storage for those indices... Odd, I'd expect it to be about the same.? The indexed data should be the same, and even though every json record includes the field names, they compress well. It's possible that the bro app indexing the fields individually is what makes the indexes larger... if you do something like ? ? id_resp_p=6379 (or whatever the field shows up as for you) does that find the records immediately, or does it have to scan through all the data? without individual field indexes you would have to do something like ? ? 6379 id_resp_p=6379 and hope that speeds it up, if you're trying to do something like ? ? id_orig_p=80 Then this will be pretty slow: ? ? 80 id_orig_p=80 -- - Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/61c8ad85/attachment.html From lattin at umn.edu Thu Jul 14 09:54:49 2016 From: lattin at umn.edu (Brandon Lattin) Date: Thu, 14 Jul 2016 11:54:49 -0500 Subject: [Bro] problem ingesting bro json logs into splunk In-Reply-To: <1431536404.3205916.1468514641628.JavaMail.yahoo@mail.yahoo.com> References: <1840447497.3042597.1468503230969.JavaMail.yahoo.ref@mail.yahoo.com> <1840447497.3042597.1468503230969.JavaMail.yahoo@mail.yahoo.com> <1483283772.3249400.1468506212328.JavaMail.yahoo@mail.yahoo.com> <1305592882.3102885.1468512984462.JavaMail.yahoo@mail.yahoo.com> <82D0EDF6-7064-4EC8-8679-7220C372569A@illinois.edu> <1431536404.3205916.1468514641628.JavaMail.yahoo@mail.yahoo.com> Message-ID: I do wonder if it's even faster having the pre-search-time extractions in the tsidx files. I suppose if you're going for a specific IP, the bloom filters may help? I've been really hesitant to move to JSON, simply because of the added raw volume impact on licensing. Bro is already over 250GB/day for us using TSV files. On Thu, Jul 14, 2016 at 11:44 AM, philosnef wrote: > It is the TS IDX files in Splunk that grow out of control when using the > Bro TSV app. Hope this helps for anyone interested. > > > On Thursday, July 14, 2016 12:30 PM, "Azoff, Justin S" < > jazoff at illinois.edu> wrote: > > > > > On Jul 14, 2016, at 12:16 PM, philosnef wrote: > > > > The problem with the Spunk app is that indexing is occuring at time of > ingest. This causes the indices of the Bro data to grow extremely fast. > Using json and not the Bro app means that the data is indexed by Splunk, > resulting in far smaller indices on the splunk indexing servers. This is > specifically why we moved away from TSV and to JSON, since it was nuking > disk storage for those indices... > > > Odd, I'd expect it to be about the same. The indexed data should be the > same, and even though every json record includes the field names, they > compress well. > > It's possible that the bro app indexing the fields individually is what > makes the indexes larger... if you do something like > > id_resp_p=6379 > > (or whatever the field shows up as for you) > > does that find the records immediately, or does it have to scan through > all the data? > > without individual field indexes you would have to do something like > > 6379 id_resp_p=6379 > > and hope that speeds it up, if you're trying to do something like > > id_orig_p=80 > > Then this will be pretty slow: > > 80 id_orig_p=80 > > > -- > - Justin Azoff > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Brandon Lattin Security Analyst University of Minnesota - University Information Security Office: 612-626-6672 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/e841a123/attachment.html From philosnef at yahoo.com Thu Jul 14 10:03:13 2016 From: philosnef at yahoo.com (philosnef) Date: Thu, 14 Jul 2016 17:03:13 +0000 (UTC) Subject: [Bro] problem ingesting bro json logs into splunk In-Reply-To: References: <1840447497.3042597.1468503230969.JavaMail.yahoo.ref@mail.yahoo.com> <1840447497.3042597.1468503230969.JavaMail.yahoo@mail.yahoo.com> <1483283772.3249400.1468506212328.JavaMail.yahoo@mail.yahoo.com> <1305592882.3102885.1468512984462.JavaMail.yahoo@mail.yahoo.com> <82D0EDF6-7064-4EC8-8679-7220C372569A@illinois.edu> <1431536404.3205916.1468514641628.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1110371812.3247111.1468515793459.JavaMail.yahoo@mail.yahoo.com> Heh. We have a multi-TB license for splunk... Bro is one of the largest consumers of that license.... On Thursday, July 14, 2016 12:54 PM, Brandon Lattin wrote: I do wonder if it's even faster having the pre-search-time extractions in the tsidx files. I suppose if you're going for a specific IP, the bloom filters may help? I've been really hesitant to move to JSON, simply because of the added raw volume impact on licensing. Bro is already over 250GB/day for us using TSV files. On Thu, Jul 14, 2016 at 11:44 AM, philosnef wrote: It is the TS IDX files in Splunk that grow out of control when using the Bro TSV app. Hope this helps for anyone interested. On Thursday, July 14, 2016 12:30 PM, "Azoff, Justin S" wrote: > On Jul 14, 2016, at 12:16 PM, philosnef wrote: > > The problem with the Spunk app is that indexing is occuring at time of ingest. This causes the indices of the Bro data to grow extremely fast. Using json and not the Bro app means that the data is indexed by Splunk, resulting in far smaller indices on the splunk indexing servers. This is specifically why we moved away from TSV and to JSON, since it was nuking disk storage for those indices... Odd, I'd expect it to be about the same.? The indexed data should be the same, and even though every json record includes the field names, they compress well. It's possible that the bro app indexing the fields individually is what makes the indexes larger... if you do something like ? ? id_resp_p=6379 (or whatever the field shows up as for you) does that find the records immediately, or does it have to scan through all the data? without individual field indexes you would have to do something like ? ? 6379 id_resp_p=6379 and hope that speeds it up, if you're trying to do something like ? ? id_orig_p=80 Then this will be pretty slow: ? ? 80 id_orig_p=80 -- - Justin Azoff _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Brandon LattinSecurity Analyst University of Minnesota - University Information Security Office: 612-626-6672 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/dcaeefd9/attachment-0001.html From bmixonb1 at cs.unm.edu Thu Jul 14 10:26:33 2016 From: bmixonb1 at cs.unm.edu (Ben Mixon-Baca) Date: Thu, 14 Jul 2016 10:26:33 -0700 Subject: [Bro] First orig_h packet after 3 way handshake In-Reply-To: <5902E9B6-2801-4A67-AAC1-A385AE5728F2@icir.org> References: <5786C260.4000500@cs.unm.edu> <5786C8E5.8000909@cs.unm.edu> <5902E9B6-2801-4A67-AAC1-A385AE5728F2@icir.org> Message-ID: <5787CB49.801@cs.unm.edu> I'm looking at Tor+obfs4. Normally, everything parsed out using the events in the SSL module would be perfect but since the handshake is obfuscated, none of those events fire. I was trying to look at the packet that _should_ be the client hello in order to see if there is anything regular about that particular payload. On 07/13/2016 05:17 PM, Johanna Amann wrote: > Out of curiosity - what are you trying to do? > > (I am always curious what people try to get from the SSL handshake that > we do not parse out yet...) > > Johanna > > On 13 Jul 2016, at 16:04, Ben Mixon-Baca wrote: > >> Unfortunately for what I am doing, I cannot. >> >> On 07/13/2016 03:58 PM, Azoff, Justin S wrote: >>> >>>> On Jul 13, 2016, at 6:36 PM, Ben Mixon-Baca >>>> wrote: >>>> >>>> Does Bro have an event that will get fired for the first packet after >>>> the tcp 3-way handshake, or is there a way to get at that easily or >>>> does >>>> it require a lot of state to be maintained in the script? >>>> >>>> I am trying to get at this first packet following the 3 way handshake >>>> because that is where the client hello in the ssl handshake should be. >>> >>> Can you use the ssl_client_hello event? >>> >>> event ssl_client_hello(c: connection, version: count, possible_ts: >>> time, client_random: string, session_id: string, ciphers: index_vec) >>> >> >> -- >> Ben >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: OpenPGP digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/54a8507e/attachment.bin From johanna at icir.org Thu Jul 14 10:30:37 2016 From: johanna at icir.org (Johanna Amann) Date: Thu, 14 Jul 2016 10:30:37 -0700 Subject: [Bro] First orig_h packet after 3 way handshake In-Reply-To: <5787CB49.801@cs.unm.edu> References: <5786C260.4000500@cs.unm.edu> <5786C8E5.8000909@cs.unm.edu> <5902E9B6-2801-4A67-AAC1-A385AE5728F2@icir.org> <5787CB49.801@cs.unm.edu> Message-ID: Oh, interesting. You should be able to use the new_packet/packet_contents events and add some counter to the connection record to let you count at which place in the handshake you are. But - these are very expensive events, so you might get into problems when trying to run this on a link that has any real volume on it. Johanna On 14 Jul 2016, at 10:26, Ben Mixon-Baca wrote: > I'm looking at Tor+obfs4. Normally, everything parsed out using the > events in the SSL module would be perfect but since the handshake is > obfuscated, none of those events fire. I was trying to look at the > packet that _should_ be the client hello in order to see if there is > anything regular about that particular payload. > > On 07/13/2016 05:17 PM, Johanna Amann wrote: >> Out of curiosity - what are you trying to do? >> >> (I am always curious what people try to get from the SSL handshake >> that >> we do not parse out yet...) >> >> Johanna >> >> On 13 Jul 2016, at 16:04, Ben Mixon-Baca wrote: >> >>> Unfortunately for what I am doing, I cannot. >>> >>> On 07/13/2016 03:58 PM, Azoff, Justin S wrote: >>>> >>>>> On Jul 13, 2016, at 6:36 PM, Ben Mixon-Baca >>>>> wrote: >>>>> >>>>> Does Bro have an event that will get fired for the first packet >>>>> after >>>>> the tcp 3-way handshake, or is there a way to get at that easily >>>>> or >>>>> does >>>>> it require a lot of state to be maintained in the script? >>>>> >>>>> I am trying to get at this first packet following the 3 way >>>>> handshake >>>>> because that is where the client hello in the ssl handshake should >>>>> be. >>>> >>>> Can you use the ssl_client_hello event? >>>> >>>> event ssl_client_hello(c: connection, version: count, possible_ts: >>>> time, client_random: string, session_id: string, ciphers: >>>> index_vec) >>>> >>> >>> -- >>> Ben >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- > Ben From johanna at icir.org Thu Jul 14 10:31:39 2016 From: johanna at icir.org (Johanna Amann) Date: Thu, 14 Jul 2016 10:31:39 -0700 Subject: [Bro] First orig_h packet after 3 way handshake In-Reply-To: References: <5786C260.4000500@cs.unm.edu> <5786C8E5.8000909@cs.unm.edu> <5902E9B6-2801-4A67-AAC1-A385AE5728F2@icir.org> <5787CB49.801@cs.unm.edu> Message-ID: <38690D92-EDC0-47B3-9D20-1BA86E164131@icir.org> Actually, thinking a bit more about it - tcp_packet might be the best event for this. On 14 Jul 2016, at 10:30, Johanna Amann wrote: > Oh, interesting. > > You should be able to use the new_packet/packet_contents events and > add some counter to the connection record to let you count at which > place in the handshake you are. > > But - these are very expensive events, so you might get into problems > when trying to run this on a link that has any real volume on it. > > Johanna > > On 14 Jul 2016, at 10:26, Ben Mixon-Baca wrote: > >> I'm looking at Tor+obfs4. Normally, everything parsed out using the >> events in the SSL module would be perfect but since the handshake is >> obfuscated, none of those events fire. I was trying to look at the >> packet that _should_ be the client hello in order to see if there is >> anything regular about that particular payload. >> >> On 07/13/2016 05:17 PM, Johanna Amann wrote: >>> Out of curiosity - what are you trying to do? >>> >>> (I am always curious what people try to get from the SSL handshake >>> that >>> we do not parse out yet...) >>> >>> Johanna >>> >>> On 13 Jul 2016, at 16:04, Ben Mixon-Baca wrote: >>> >>>> Unfortunately for what I am doing, I cannot. >>>> >>>> On 07/13/2016 03:58 PM, Azoff, Justin S wrote: >>>>> >>>>>> On Jul 13, 2016, at 6:36 PM, Ben Mixon-Baca >>>>>> wrote: >>>>>> >>>>>> Does Bro have an event that will get fired for the first packet >>>>>> after >>>>>> the tcp 3-way handshake, or is there a way to get at that easily >>>>>> or >>>>>> does >>>>>> it require a lot of state to be maintained in the script? >>>>>> >>>>>> I am trying to get at this first packet following the 3 way >>>>>> handshake >>>>>> because that is where the client hello in the ssl handshake >>>>>> should be. >>>>> >>>>> Can you use the ssl_client_hello event? >>>>> >>>>> event ssl_client_hello(c: connection, version: count, possible_ts: >>>>> time, client_random: string, session_id: string, ciphers: >>>>> index_vec) >>>>> >>>> >>>> -- >>>> Ben >>>> >>>> _______________________________________________ >>>> Bro mailing list >>>> bro at bro-ids.org >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> -- >> Ben From bmixonb1 at cs.unm.edu Thu Jul 14 10:39:34 2016 From: bmixonb1 at cs.unm.edu (Ben Mixon-Baca) Date: Thu, 14 Jul 2016 10:39:34 -0700 Subject: [Bro] First orig_h packet after 3 way handshake In-Reply-To: <38690D92-EDC0-47B3-9D20-1BA86E164131@icir.org> References: <5786C260.4000500@cs.unm.edu> <5786C8E5.8000909@cs.unm.edu> <5902E9B6-2801-4A67-AAC1-A385AE5728F2@icir.org> <5787CB49.801@cs.unm.edu> <38690D92-EDC0-47B3-9D20-1BA86E164131@icir.org> Message-ID: <5787CE56.40306@cs.unm.edu> Cool, thanks Johanna! I had started to use the tcp_packet event but was concerned about the amount of state I would need to keep, I hadn't even thought to add to the connection record, thanks! Fortunately, all of the analysis I am doing is on pcaps so I don't need to worry about running my script on live traffic. On 07/14/2016 10:31 AM, Johanna Amann wrote: > Actually, thinking a bit more about it - tcp_packet might be the best > event for this. > > On 14 Jul 2016, at 10:30, Johanna Amann wrote: > >> Oh, interesting. >> >> You should be able to use the new_packet/packet_contents events and >> add some counter to the connection record to let you count at which >> place in the handshake you are. >> >> But - these are very expensive events, so you might get into problems >> when trying to run this on a link that has any real volume on it. >> >> Johanna >> >> On 14 Jul 2016, at 10:26, Ben Mixon-Baca wrote: >> >>> I'm looking at Tor+obfs4. Normally, everything parsed out using the >>> events in the SSL module would be perfect but since the handshake is >>> obfuscated, none of those events fire. I was trying to look at the >>> packet that _should_ be the client hello in order to see if there is >>> anything regular about that particular payload. >>> >>> On 07/13/2016 05:17 PM, Johanna Amann wrote: >>>> Out of curiosity - what are you trying to do? >>>> >>>> (I am always curious what people try to get from the SSL handshake that >>>> we do not parse out yet...) >>>> >>>> Johanna >>>> >>>> On 13 Jul 2016, at 16:04, Ben Mixon-Baca wrote: >>>> >>>>> Unfortunately for what I am doing, I cannot. >>>>> >>>>> On 07/13/2016 03:58 PM, Azoff, Justin S wrote: >>>>>> >>>>>>> On Jul 13, 2016, at 6:36 PM, Ben Mixon-Baca >>>>>>> wrote: >>>>>>> >>>>>>> Does Bro have an event that will get fired for the first packet >>>>>>> after >>>>>>> the tcp 3-way handshake, or is there a way to get at that easily or >>>>>>> does >>>>>>> it require a lot of state to be maintained in the script? >>>>>>> >>>>>>> I am trying to get at this first packet following the 3 way >>>>>>> handshake >>>>>>> because that is where the client hello in the ssl handshake >>>>>>> should be. >>>>>> >>>>>> Can you use the ssl_client_hello event? >>>>>> >>>>>> event ssl_client_hello(c: connection, version: count, possible_ts: >>>>>> time, client_random: string, session_id: string, ciphers: index_vec) >>>>>> >>>>> >>>>> -- >>>>> Ben >>>>> >>>>> _______________________________________________ >>>>> Bro mailing list >>>>> bro at bro-ids.org >>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >>> -- >>> Ben -- Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: OpenPGP digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/fb1d5c29/attachment.bin From castle1126 at yahoo.com Thu Jul 14 10:46:13 2016 From: castle1126 at yahoo.com (Stephen Castellarin) Date: Thu, 14 Jul 2016 17:46:13 +0000 (UTC) Subject: [Bro] Help with missed_bytes affecting hash creations in files.log References: <37325130.3263906.1468518373774.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <37325130.3263906.1468518373774.JavaMail.yahoo@mail.yahoo.com> Hi, I have Bro 2.4.1 running on an older system (2 Intel X5550 processors giving 8 CPUs), 48Gb memory running 64 bit Ubuntu (14.04.4) server, using PF_Ring with an Intel 82571EB Ethernet card (1gb copper).? This system is sitting on a network tap that is just seeing SMTP traffic between our outer mail gateway and our inside mail infrastructure.? My Bro configuration is relatively simple, with a nodes.cfg being: [manager] type=manager host=localhost # [proxy-1] type=proxy host=localhost # [worker-1] type=worker host=localhost interface=eth5 lb_method=pf_ring lb_procs=8 When I look at the files.log file I see instances of files that have missing_bytes, which causes the hashes to not be calculated.? Running an IFCONFIG I don't see any drops, errors, etc.? Same with running broctl netstats, no drops.? SAR reports on that system show the CPUs running at 73% IDLE. Is there something I'm missing in tuning or tweaking our configuration?? Can I get to a point where I have zero files with no missed_bytes, or will there always be something or things with missed_bytes.? A hardware upgrade can be in our future, but I'm trying to prove this concept by using this setup to help get funding for upgrading. Thanks all,Steve -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/c69eda1d/attachment.html From steve at brant.nu Thu Jul 14 16:07:41 2016 From: steve at brant.nu (Steve Brant) Date: Thu, 14 Jul 2016 17:07:41 -0600 Subject: [Bro] problem ingesting bro json logs into splunk In-Reply-To: References: <1840447497.3042597.1468503230969.JavaMail.yahoo.ref@mail.yahoo.com> <1840447497.3042597.1468503230969.JavaMail.yahoo@mail.yahoo.com> Message-ID: I've attached a modified version of the Splunk TA for bro, that accommodates bro logs in json format. Let me know if you have any problem with it. Thanks, Steve ~/SB On Thu, Jul 14, 2016 at 8:14 AM, Brandon Lattin wrote: > Do you have the Splunk installed? (https://splunkbase.splunk.com/app/1617/ > ) > > The TA will dynamically create sourcetypes based on the log name. > > # Dynamic source typing based on log filename > # Match: conn.log, bro.conn.log, > # md5.bro.conn.log, whatever.conn.log > [BroAutoType] > DEST_KEY = MetaData:Sourcetype > SOURCE_KEY = MetaData:Source > REGEX = ([a-zA-Z0-9-]+)(?:\.[0-9-]*)?(?:\.[0-9\:-]*)?\.log > FORMAT = sourcetype::bro_$1 > WRITE_META = true > > > On Thu, Jul 14, 2016 at 8:33 AM, philosnef wrote: > >> We are getting a spurious sourcetype when ingesting bro json logs into >> splunk. >> >> Specifically, we are getting a sourcetype of bro_00. There is no log file >> named this, and the splunkforwarder is just pushing the raw logs for >> indexing into splunk. There is no massaging of the log data. Anyone know >> why this sourcetype is popping up? >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > > > -- > Brandon Lattin > Security Analyst > University of Minnesota - University Information Security > Office: 612-626-6672 > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/ea658fc8/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: TA-bro_json.tar.gz Type: application/x-gzip Size: 4241 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/ea658fc8/attachment.gz From cardigliano at ntop.org Fri Jul 15 02:55:50 2016 From: cardigliano at ntop.org (Alfredo Cardigliano) Date: Fri, 15 Jul 2016 11:55:50 +0200 Subject: [Bro] PF_RING ZC Config In-Reply-To: <62F96348-E24A-4514-9109-0FB7B39D9F86@illinois.edu> References: <9c5209a7-1601-6ae9-b4e0-5713e3316c1a@gmail.com> <4085de6a-b70d-d18a-cb21-33ffa95b84aa@gmail.com> <62F96348-E24A-4514-9109-0FB7B39D9F86@illinois.edu> Message-ID: <9B8FC355-9897-4FCC-9B77-6799DA201088@ntop.org> Hi guys I drafted a REAME based on PF_RING ZC at https://github.com/ntop/PF_RING/blob/dev/doc/README.bro feel free to edit it (sending pull requests). Thank you Alfredo > On 08 Jul 2016, at 21:23, Slagell, Adam J wrote: > > Thanks. I don?t want to forget to come back to this. > >> On Jul 8, 2016, at 12:57 PM, Gary Faulkner wrote: >> >> https://bro-tracker.atlassian.net/browse/BIT-1642 >> >> >> On 7/8/16 12:35 PM, Slagell, Adam J wrote: >>> Could you create a ticket for this in the tracker? >>> >>> On Jul 8, 2016, at 12:26 PM, Gary Faulkner > wrote: >>> >>> >>> Related to Dave's query, but not really an answer, sorry Dave... >>> >>> It might be worth revisiting this doc and updating for ZC: >>> >>> https://www.bro.org/documentation/load-balancing.html >>> >>> A few things have changed on the PF_RING DNA side in broctl in regards to naming support "dnacl" instead of "dnacluster" due to problems with name length for dnaclusters with greater than 10 queues, and with the most recent releases of PF_RING (6.4+), DNA appears to have been removed finally in favor of the newer ZC according to the change notes. From what I recall reading I don't believe it is terribly different outside of substituting ZC drivers (and tweaking huge-pages in the driver load script) in favor of DNA, and using zbalance_ipc instead of pfdnacluster_master. I want to say the naming in node.cfg becomes zc: instead of dnacl:. >>> >>> Also, speaking of ZC, NTOP has a blog post that might be worth taking a look at concerning alternate ways of implementing ZC / zbalance_ipc with bro to work around a problem that can occur when bro workers crash and get automatically restarted. >>> >>> http://www.ntop.org/pf_ring/best-practices-for-using-bro_ids-with-pf_ring-zc-reliably/ >>> >>> I haven't quite made the transition to ZC from DNA yet, otherwise I'd take a stab at submitting updated docs and trying to assist more here. I have plans to make the switch later this summer though. >>> >>> ~Gary >>> >>> On 7/7/16 5:25 PM, Dave Crawford wrote: >>> >>> Just wanted to update the list that I quit spending cycles on this and for the time being reverted back to running our clusters with the non-commercial version of pf_ring. >>> >>> I can only comment on my experience, but I discovered there is an extreme lack of quality documentation and the "commercial support" that came with the 10 licenses was nearly non-existent. >>> >>> Lessons have been learned and when the need to expand comes we'll be looking at other commercial solutions to replace our X520's with. >>> >>> -Dave >>> >>> >>> >>> On Jun 24, 2016, at 8:28 AM, Dave Crawford wrote: >>> >>> Would anyone happen to have documentation for configuring ZC and Bro? I have NTop's PF_RING and ixgbe driver packages installed, the proper license in /etc/pf_ring, and have compiled Bro with the NTop libraries but I'm seeing the kernel error below along with a ton of ?split routing? messages in weird.conf, so I suspect the flows aren?t being load balanced correctly. >>> >>> Jun 22 15:10:03 win-csignsm-01 kernel: [11060.244524] [PF_RING] Unable to activate two or more ZC sockets on the same interface eth6/link direction >>> >>> The monitored NIC is an Intel X520-LR1. >>> >>> Contents of /etc/pf_ring/zc/ixgbe/ixgbe.conf: >>> RSS=10 allow_unsupported_sfp=0 >>> >>> Contents of /etc/pf_ring/hugepages.conf >>> node=1 hugepages=1024 >>> >>> >>> And Bro is configured as: >>> [MID_INT] >>> type=worker >>> host=10.20.30.123 >>> interface=zc:eth6 >>> lb_method=pf_ring >>> lb_procs=10 >>> pin_cpus=10,11,12,13,14,15,16,17,18,19 >>> >>> Thanks! >>> -Dave >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >>> ------ >>> >>> Adam J. Slagell >>> Chief Information Security Officer >>> Director, Cybersecurity Division >>> National Center for Supercomputing Applications >>> University of Illinois at Urbana-Champaign >>> www.slagell.info >>> >>> "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." >>> >>> >>> >>> >>> >>> >>> >>> >>> >> > > ------ > > Adam J. Slagell > Chief Information Security Officer > Director, Cybersecurity Division > National Center for Supercomputing Applications > University of Illinois at Urbana-Champaign > www.slagell.info > > "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." > > > > > > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160715/955bdb0d/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160715/955bdb0d/attachment-0001.bin From cbarbaro at cert.unlp.edu.ar Fri Jul 15 07:33:31 2016 From: cbarbaro at cert.unlp.edu.ar (Cristian Daniel Barbaro) Date: Fri, 15 Jul 2016 11:33:31 -0300 Subject: [Bro] Telnet analyzer Message-ID: <5788F43B.7000503@cert.unlp.edu.ar> Hello, people. I'm trying detect when someone connects to Telnet and login is sucessful. But Bro never detects this event, only login_confused or login_confused_text events was detected. Any idea? Thanks. Regards... -- Cristian Daniel Barbaro CERTUNLP -- From fatema.bannatwala at gmail.com Fri Jul 15 10:03:56 2016 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Fri, 15 Jul 2016 13:03:56 -0400 Subject: [Bro] Bro Digest, Vol 123, Issue 24 In-Reply-To: References: Message-ID: Well regarding Splunk add-on for BRO-IDS, I asked following question on Splunkbase and still waiting for an answer so thought might be worth sharing it here as well: Starting with the environment, I have an indexer cluster of 3 indexers, two independent search heads, and one Universal forwarder. My question is where the BRO IDS app goes and how it works? What I have done is - I have installed the app on both of my search heads (as per general convention while dealing with apps), and my Universal Forwarder is monitoring the Bro log directory (yes I have installed UF on my Bro sensor machine). I am getting the monitored Bro logs in my indexers and am able to search them via search heads, but the app is just sitting there doing nothing it seems. The documentation I have read so far says that you need to install app on the heavy forwarder that is monitoring your log dir and have to set the inputs path in the app instead of heavy forwarder's input. (So I think it's stupid for the people who just want to have a forwarder installed on their bro sensor for just forwarding bro logs and for that we need to install heavy forwarder with the app, and that too app will be doing all the forwarding and parsing and heavy forwarder will be just sitting there providing Python support to the app to do its stuff). So my question is: is my above configuration even workable with Bro IDS add-on or do I have to just chuck the idea of using the add-on because I don't want to run a heavy forwarder on my Bro machines? Thanks, Fatema. On Fri, Jul 15, 2016 at 5:55 AM, wrote: > Send Bro mailing list submissions to > bro at bro.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > or, via email, send a message with subject or body 'help' to > bro-request at bro.org > > You can reach the person managing the list at > bro-owner at bro.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Bro digest..." > > > Today's Topics: > > 1. Re: problem ingesting bro json logs into splunk (Steve Brant) > 2. Re: PF_RING ZC Config (Alfredo Cardigliano) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 14 Jul 2016 17:07:41 -0600 > From: Steve Brant > Subject: Re: [Bro] problem ingesting bro json logs into splunk > To: Brandon Lattin > Cc: philosnef , "bro at bro.org" > Message-ID: > spH96fjZWyMduzDmmT+HCgUufdY2aNaOkf00zD07ZmnmONg at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > I've attached a modified version of the Splunk TA for bro, that > accommodates bro logs in json format. Let me know if you have any problem > with it. > > Thanks, > Steve > > ~/SB > > On Thu, Jul 14, 2016 at 8:14 AM, Brandon Lattin wrote: > > > Do you have the Splunk installed? ( > https://splunkbase.splunk.com/app/1617/ > > ) > > > > The TA will dynamically create sourcetypes based on the log name. > > > > # Dynamic source typing based on log filename > > # Match: conn.log, bro.conn.log, > > # md5.bro.conn.log, whatever.conn.log > > [BroAutoType] > > DEST_KEY = MetaData:Sourcetype > > SOURCE_KEY = MetaData:Source > > REGEX = ([a-zA-Z0-9-]+)(?:\.[0-9-]*)?(?:\.[0-9\:-]*)?\.log > > FORMAT = sourcetype::bro_$1 > > WRITE_META = true > > > > > > On Thu, Jul 14, 2016 at 8:33 AM, philosnef wrote: > > > >> We are getting a spurious sourcetype when ingesting bro json logs into > >> splunk. > >> > >> Specifically, we are getting a sourcetype of bro_00. There is no log > file > >> named this, and the splunkforwarder is just pushing the raw logs for > >> indexing into splunk. There is no massaging of the log data. Anyone know > >> why this sourcetype is popping up? > >> > >> _______________________________________________ > >> Bro mailing list > >> bro at bro-ids.org > >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > >> > > > > > > > > -- > > Brandon Lattin > > Security Analyst > > University of Minnesota - University Information Security > > Office: 612-626-6672 > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/ea658fc8/attachment-0001.html > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: TA-bro_json.tar.gz > Type: application/x-gzip > Size: 4241 bytes > Desc: not available > Url : > http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/ea658fc8/attachment-0001.gz > > ------------------------------ > > Message: 2 > Date: Fri, 15 Jul 2016 11:55:50 +0200 > From: Alfredo Cardigliano > Subject: Re: [Bro] PF_RING ZC Config > To: "Slagell, Adam J" > Cc: "bro at bro.org" > Message-ID: <9B8FC355-9897-4FCC-9B77-6799DA201088 at ntop.org> > Content-Type: text/plain; charset="windows-1252" > > Hi guys > I drafted a REAME based on PF_RING ZC at > https://github.com/ntop/PF_RING/blob/dev/doc/README.bro > feel free to edit it (sending pull requests). > > Thank you > Alfredo > > > On 08 Jul 2016, at 21:23, Slagell, Adam J wrote: > > > > Thanks. I don?t want to forget to come back to this. > > > >> On Jul 8, 2016, at 12:57 PM, Gary Faulkner > wrote: > >> > >> https://bro-tracker.atlassian.net/browse/BIT-1642 > >> > >> > >> On 7/8/16 12:35 PM, Slagell, Adam J wrote: > >>> Could you create a ticket for this in the tracker? > >>> > >>> On Jul 8, 2016, at 12:26 PM, Gary Faulkner > wrote: > >>> > >>> > >>> Related to Dave's query, but not really an answer, sorry Dave... > >>> > >>> It might be worth revisiting this doc and updating for ZC: > >>> > >>> https://www.bro.org/documentation/load-balancing.html< > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.bro.org_documentation_load-2Dbalancing.html&d=CwMD-g&c=8hUWFZcy2Z-Za5rBPlktOQ&r=gMEsgy9kNQo7aTfyIJsOSuw4Z57hfQyz6uV2H4S9PvE&m=jKm5KEaN1h0UfG6EGbAMSpW5NbXR4pzULAvbXFWNnvg&s=x-uS-7-lC174enHZ7TiRa3RLswl6nRsyBvGQ2_T2W-E&e= > > > >>> > >>> A few things have changed on the PF_RING DNA side in broctl in regards > to naming support "dnacl" instead of "dnacluster" due to problems with name > length for dnaclusters with greater than 10 queues, and with the most > recent releases of PF_RING (6.4+), DNA appears to have been removed finally > in favor of the newer ZC according to the change notes. From what I recall > reading I don't believe it is terribly different outside of substituting ZC > drivers (and tweaking huge-pages in the driver load script) in favor of > DNA, and using zbalance_ipc instead of pfdnacluster_master. I want to say > the naming in node.cfg becomes zc: instead of dnacl:. > >>> > >>> Also, speaking of ZC, NTOP has a blog post that might be worth taking > a look at concerning alternate ways of implementing ZC / zbalance_ipc with > bro to work around a problem that can occur when bro workers crash and get > automatically restarted. > >>> > >>> > http://www.ntop.org/pf_ring/best-practices-for-using-bro_ids-with-pf_ring-zc-reliably/ > < > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ntop.org_pf-5Fring_best-2Dpractices-2Dfor-2Dusing-2Dbro-5Fids-2Dwith-2Dpf-5Fring-2Dzc-2Dreliably_&d=CwMD-g&c=8hUWFZcy2Z-Za5rBPlktOQ&r=gMEsgy9kNQo7aTfyIJsOSuw4Z57hfQyz6uV2H4S9PvE&m=jKm5KEaN1h0UfG6EGbAMSpW5NbXR4pzULAvbXFWNnvg&s=oOBYvlJMigTXYIzqgtcGz3iNzZpTQrMlSPBWRYkOFA4&e= > > > >>> > >>> I haven't quite made the transition to ZC from DNA yet, otherwise I'd > take a stab at submitting updated docs and trying to assist more here. I > have plans to make the switch later this summer though. > >>> > >>> ~Gary > >>> > >>> On 7/7/16 5:25 PM, Dave Crawford wrote: > >>> > >>> Just wanted to update the list that I quit spending cycles on this and > for the time being reverted back to running our clusters with the > non-commercial version of pf_ring. > >>> > >>> I can only comment on my experience, but I discovered there is an > extreme lack of quality documentation and the "commercial support" that > came with the 10 licenses was nearly non-existent. > >>> > >>> Lessons have been learned and when the need to expand comes we'll be > looking at other commercial solutions to replace our X520's with. > >>> > >>> -Dave > >>> > >>> > >>> > >>> On Jun 24, 2016, at 8:28 AM, Dave Crawford bro at pingtrip.com> wrote: > >>> > >>> Would anyone happen to have documentation for configuring ZC and Bro? > I have NTop's PF_RING and ixgbe driver packages installed, the proper > license in /etc/pf_ring, and have compiled Bro with the NTop libraries but > I'm seeing the kernel error below along with a ton of ?split routing? > messages in weird.conf, so I suspect the flows aren?t being load balanced > correctly. > >>> > >>> Jun 22 15:10:03 win-csignsm-01 kernel: [11060.244524] [PF_RING] Unable > to activate two or more ZC sockets on the same interface eth6/link direction > >>> > >>> The monitored NIC is an Intel X520-LR1. > >>> > >>> Contents of /etc/pf_ring/zc/ixgbe/ixgbe.conf: > >>> RSS=10 allow_unsupported_sfp=0 > >>> > >>> Contents of /etc/pf_ring/hugepages.conf > >>> node=1 hugepages=1024 > >>> > >>> > >>> And Bro is configured as: > >>> [MID_INT] > >>> type=worker > >>> host=10.20.30.123 > >>> interface=zc:eth6 > >>> lb_method=pf_ring > >>> lb_procs=10 > >>> pin_cpus=10,11,12,13,14,15,16,17,18,19 > >>> > >>> Thanks! > >>> -Dave > >>> _______________________________________________ > >>> Bro mailing list > >>> bro at bro-ids.org > >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro< > https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.ICSI.Berkeley.EDU_mailman_listinfo_bro&d=CwMD-g&c=8hUWFZcy2Z-Za5rBPlktOQ&r=gMEsgy9kNQo7aTfyIJsOSuw4Z57hfQyz6uV2H4S9PvE&m=jKm5KEaN1h0UfG6EGbAMSpW5NbXR4pzULAvbXFWNnvg&s=V2Ec5cOcnYqN7P3EpauWtYZUCrRNMUzy_pvRrgdH_C8&e= > > > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> _______________________________________________ > >>> Bro mailing list > >>> bro at bro-ids.org > >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro< > https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.ICSI.Berkeley.EDU_mailman_listinfo_bro&d=CwMD-g&c=8hUWFZcy2Z-Za5rBPlktOQ&r=gMEsgy9kNQo7aTfyIJsOSuw4Z57hfQyz6uV2H4S9PvE&m=jKm5KEaN1h0UfG6EGbAMSpW5NbXR4pzULAvbXFWNnvg&s=V2Ec5cOcnYqN7P3EpauWtYZUCrRNMUzy_pvRrgdH_C8&e= > > > >>> > >>> _______________________________________________ > >>> Bro mailing list > >>> bro at bro-ids.org > >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > >>> > >>> ------ > >>> > >>> Adam J. Slagell > >>> Chief Information Security Officer > >>> Director, Cybersecurity Division > >>> National Center for Supercomputing Applications > >>> University of Illinois at Urbana-Champaign > >>> www.slagell.info > >>> > >>> "Under the Illinois Freedom of Information Act (FOIA), any written > communication to or from University employees regarding University business > is a public record and may be subject to public disclosure." > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >> > > > > ------ > > > > Adam J. Slagell > > Chief Information Security Officer > > Director, Cybersecurity Division > > National Center for Supercomputing Applications > > University of Illinois at Urbana-Champaign > > www.slagell.info > > > > "Under the Illinois Freedom of Information Act (FOIA), any written > communication to or from University employees regarding University business > is a public record and may be subject to public disclosure." > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro < > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro> > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160715/955bdb0d/attachment.html > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: signature.asc > Type: application/pgp-signature > Size: 842 bytes > Desc: Message signed with OpenPGP using GPGMail > Url : > http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160715/955bdb0d/attachment.bin > > ------------------------------ > > _______________________________________________ > Bro mailing list > Bro at bro.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > End of Bro Digest, Vol 123, Issue 24 > ************************************ > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160715/499d93cd/attachment-0001.html From dnthayer at illinois.edu Fri Jul 15 16:02:31 2016 From: dnthayer at illinois.edu (Daniel Thayer) Date: Fri, 15 Jul 2016 18:02:31 -0500 Subject: [Bro] PF_RING ZC Config In-Reply-To: <9B8FC355-9897-4FCC-9B77-6799DA201088@ntop.org> References: <9c5209a7-1601-6ae9-b4e0-5713e3316c1a@gmail.com> <4085de6a-b70d-d18a-cb21-33ffa95b84aa@gmail.com> <62F96348-E24A-4514-9109-0FB7B39D9F86@illinois.edu> <9B8FC355-9897-4FCC-9B77-6799DA201088@ntop.org> Message-ID: <06e8d50f-e9b0-5377-6bf1-83472c142a48@illinois.edu> Looks like there is a mistake on line 64 of the README. The text "-g 9" should probably be "-g 8". On 7/15/16 4:55 AM, Alfredo Cardigliano wrote: > Hi guys > I drafted a REAME based on PF_RING ZC at > https://github.com/ntop/PF_RING/blob/dev/doc/README.bro > feel free to edit it (sending pull requests). > > Thank you > Alfredo > From steve at brant.nu Sat Jul 16 22:53:34 2016 From: steve at brant.nu (Steve Brant) Date: Sat, 16 Jul 2016 23:53:34 -0600 Subject: [Bro] Bro Digest, Vol 123, Issue 24 In-Reply-To: References: Message-ID: Hi Fatema- I've answered your posting at https://answers.splunk.com/answers/427161/where-to-install-and-configure-the-splunk-add-on-f.html#answer-430982 Please respond there if you need some additional detail. Thanks, Steve ~/SB On Fri, Jul 15, 2016 at 11:03 AM, fatema bannatwala < fatema.bannatwala at gmail.com> wrote: > Well regarding Splunk add-on for BRO-IDS, I asked following question on > Splunkbase and still waiting for an answer so thought might be worth > sharing it here as well: > Starting with the environment, I have an indexer cluster of 3 indexers, > two independent search heads, and one Universal forwarder. > My question is where the BRO IDS app goes and how it works? > What I have done is - I have installed the app on both of my search heads > (as per general convention while dealing with apps), and my Universal > Forwarder is monitoring the Bro log directory (yes I have installed UF on > my Bro sensor machine). > I am getting the monitored Bro logs in my indexers and am able to search > them via search heads, but the app is just sitting there doing nothing it > seems. > > The documentation I have read so far says that you need to install app on > the heavy forwarder that is monitoring your log dir and have to set the > inputs path in the app instead of heavy forwarder's input. (So I think it's > stupid for the people who just want to have a forwarder installed on their > bro sensor for just forwarding bro logs and for that we need to install > heavy forwarder with the app, and that too app will be doing all the > forwarding and parsing and heavy forwarder will be just sitting there > providing Python support to the app to do its stuff). > > So my question is: is my above configuration even workable with Bro IDS > add-on or do I have to just chuck the idea of using the add-on because I > don't want to run a heavy forwarder on my Bro machines? > > Thanks, > Fatema. > > On Fri, Jul 15, 2016 at 5:55 AM, wrote: > >> Send Bro mailing list submissions to >> bro at bro.org >> >> To subscribe or unsubscribe via the World Wide Web, visit >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> or, via email, send a message with subject or body 'help' to >> bro-request at bro.org >> >> You can reach the person managing the list at >> bro-owner at bro.org >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of Bro digest..." >> >> >> Today's Topics: >> >> 1. Re: problem ingesting bro json logs into splunk (Steve Brant) >> 2. Re: PF_RING ZC Config (Alfredo Cardigliano) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Thu, 14 Jul 2016 17:07:41 -0600 >> From: Steve Brant >> Subject: Re: [Bro] problem ingesting bro json logs into splunk >> To: Brandon Lattin >> Cc: philosnef , "bro at bro.org" >> Message-ID: >> > spH96fjZWyMduzDmmT+HCgUufdY2aNaOkf00zD07ZmnmONg at mail.gmail.com> >> Content-Type: text/plain; charset="utf-8" >> >> I've attached a modified version of the Splunk TA for bro, that >> accommodates bro logs in json format. Let me know if you have any problem >> with it. >> >> Thanks, >> Steve >> >> ~/SB >> >> On Thu, Jul 14, 2016 at 8:14 AM, Brandon Lattin wrote: >> >> > Do you have the Splunk installed? ( >> https://splunkbase.splunk.com/app/1617/ >> > ) >> > >> > The TA will dynamically create sourcetypes based on the log name. >> > >> > # Dynamic source typing based on log filename >> > # Match: conn.log, bro.conn.log, >> > # md5.bro.conn.log, whatever.conn.log >> > [BroAutoType] >> > DEST_KEY = MetaData:Sourcetype >> > SOURCE_KEY = MetaData:Source >> > REGEX = ([a-zA-Z0-9-]+)(?:\.[0-9-]*)?(?:\.[0-9\:-]*)?\.log >> > FORMAT = sourcetype::bro_$1 >> > WRITE_META = true >> > >> > >> > On Thu, Jul 14, 2016 at 8:33 AM, philosnef wrote: >> > >> >> We are getting a spurious sourcetype when ingesting bro json logs into >> >> splunk. >> >> >> >> Specifically, we are getting a sourcetype of bro_00. There is no log >> file >> >> named this, and the splunkforwarder is just pushing the raw logs for >> >> indexing into splunk. There is no massaging of the log data. Anyone >> know >> >> why this sourcetype is popping up? >> >> >> >> _______________________________________________ >> >> Bro mailing list >> >> bro at bro-ids.org >> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> >> > >> > >> > >> > -- >> > Brandon Lattin >> > Security Analyst >> > University of Minnesota - University Information Security >> > Office: 612-626-6672 >> > >> > _______________________________________________ >> > Bro mailing list >> > bro at bro-ids.org >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/ea658fc8/attachment-0001.html >> -------------- next part -------------- >> A non-text attachment was scrubbed... >> Name: TA-bro_json.tar.gz >> Type: application/x-gzip >> Size: 4241 bytes >> Desc: not available >> Url : >> http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/ea658fc8/attachment-0001.gz >> >> ------------------------------ >> >> Message: 2 >> Date: Fri, 15 Jul 2016 11:55:50 +0200 >> From: Alfredo Cardigliano >> Subject: Re: [Bro] PF_RING ZC Config >> To: "Slagell, Adam J" >> Cc: "bro at bro.org" >> Message-ID: <9B8FC355-9897-4FCC-9B77-6799DA201088 at ntop.org> >> Content-Type: text/plain; charset="windows-1252" >> >> Hi guys >> I drafted a REAME based on PF_RING ZC at >> https://github.com/ntop/PF_RING/blob/dev/doc/README.bro >> feel free to edit it (sending pull requests). >> >> Thank you >> Alfredo >> >> > On 08 Jul 2016, at 21:23, Slagell, Adam J wrote: >> > >> > Thanks. I don?t want to forget to come back to this. >> > >> >> On Jul 8, 2016, at 12:57 PM, Gary Faulkner >> wrote: >> >> >> >> https://bro-tracker.atlassian.net/browse/BIT-1642 >> >> >> >> >> >> On 7/8/16 12:35 PM, Slagell, Adam J wrote: >> >>> Could you create a ticket for this in the tracker? >> >>> >> >>> On Jul 8, 2016, at 12:26 PM, Gary Faulkner > > wrote: >> >>> >> >>> >> >>> Related to Dave's query, but not really an answer, sorry Dave... >> >>> >> >>> It might be worth revisiting this doc and updating for ZC: >> >>> >> >>> https://www.bro.org/documentation/load-balancing.html< >> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.bro.org_documentation_load-2Dbalancing.html&d=CwMD-g&c=8hUWFZcy2Z-Za5rBPlktOQ&r=gMEsgy9kNQo7aTfyIJsOSuw4Z57hfQyz6uV2H4S9PvE&m=jKm5KEaN1h0UfG6EGbAMSpW5NbXR4pzULAvbXFWNnvg&s=x-uS-7-lC174enHZ7TiRa3RLswl6nRsyBvGQ2_T2W-E&e= >> > >> >>> >> >>> A few things have changed on the PF_RING DNA side in broctl in >> regards to naming support "dnacl" instead of "dnacluster" due to problems >> with name length for dnaclusters with greater than 10 queues, and with the >> most recent releases of PF_RING (6.4+), DNA appears to have been removed >> finally in favor of the newer ZC according to the change notes. From what I >> recall reading I don't believe it is terribly different outside of >> substituting ZC drivers (and tweaking huge-pages in the driver load script) >> in favor of DNA, and using zbalance_ipc instead of pfdnacluster_master. I >> want to say the naming in node.cfg becomes zc: instead of >> dnacl:. >> >>> >> >>> Also, speaking of ZC, NTOP has a blog post that might be worth taking >> a look at concerning alternate ways of implementing ZC / zbalance_ipc with >> bro to work around a problem that can occur when bro workers crash and get >> automatically restarted. >> >>> >> >>> >> http://www.ntop.org/pf_ring/best-practices-for-using-bro_ids-with-pf_ring-zc-reliably/ >> < >> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ntop.org_pf-5Fring_best-2Dpractices-2Dfor-2Dusing-2Dbro-5Fids-2Dwith-2Dpf-5Fring-2Dzc-2Dreliably_&d=CwMD-g&c=8hUWFZcy2Z-Za5rBPlktOQ&r=gMEsgy9kNQo7aTfyIJsOSuw4Z57hfQyz6uV2H4S9PvE&m=jKm5KEaN1h0UfG6EGbAMSpW5NbXR4pzULAvbXFWNnvg&s=oOBYvlJMigTXYIzqgtcGz3iNzZpTQrMlSPBWRYkOFA4&e= >> > >> >>> >> >>> I haven't quite made the transition to ZC from DNA yet, otherwise I'd >> take a stab at submitting updated docs and trying to assist more here. I >> have plans to make the switch later this summer though. >> >>> >> >>> ~Gary >> >>> >> >>> On 7/7/16 5:25 PM, Dave Crawford wrote: >> >>> >> >>> Just wanted to update the list that I quit spending cycles on this >> and for the time being reverted back to running our clusters with the >> non-commercial version of pf_ring. >> >>> >> >>> I can only comment on my experience, but I discovered there is an >> extreme lack of quality documentation and the "commercial support" that >> came with the 10 licenses was nearly non-existent. >> >>> >> >>> Lessons have been learned and when the need to expand comes we'll be >> looking at other commercial solutions to replace our X520's with. >> >>> >> >>> -Dave >> >>> >> >>> >> >>> >> >>> On Jun 24, 2016, at 8:28 AM, Dave Crawford > bro at pingtrip.com> wrote: >> >>> >> >>> Would anyone happen to have documentation for configuring ZC and Bro? >> I have NTop's PF_RING and ixgbe driver packages installed, the proper >> license in /etc/pf_ring, and have compiled Bro with the NTop libraries but >> I'm seeing the kernel error below along with a ton of ?split routing? >> messages in weird.conf, so I suspect the flows aren?t being load balanced >> correctly. >> >>> >> >>> Jun 22 15:10:03 win-csignsm-01 kernel: [11060.244524] [PF_RING] >> Unable to activate two or more ZC sockets on the same interface eth6/link >> direction >> >>> >> >>> The monitored NIC is an Intel X520-LR1. >> >>> >> >>> Contents of /etc/pf_ring/zc/ixgbe/ixgbe.conf: >> >>> RSS=10 allow_unsupported_sfp=0 >> >>> >> >>> Contents of /etc/pf_ring/hugepages.conf >> >>> node=1 hugepages=1024 >> >>> >> >>> >> >>> And Bro is configured as: >> >>> [MID_INT] >> >>> type=worker >> >>> host=10.20.30.123 >> >>> interface=zc:eth6 >> >>> lb_method=pf_ring >> >>> lb_procs=10 >> >>> pin_cpus=10,11,12,13,14,15,16,17,18,19 >> >>> >> >>> Thanks! >> >>> -Dave >> >>> _______________________________________________ >> >>> Bro mailing list >> >>> bro at bro-ids.org >> >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro< >> https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.ICSI.Berkeley.EDU_mailman_listinfo_bro&d=CwMD-g&c=8hUWFZcy2Z-Za5rBPlktOQ&r=gMEsgy9kNQo7aTfyIJsOSuw4Z57hfQyz6uV2H4S9PvE&m=jKm5KEaN1h0UfG6EGbAMSpW5NbXR4pzULAvbXFWNnvg&s=V2Ec5cOcnYqN7P3EpauWtYZUCrRNMUzy_pvRrgdH_C8&e= >> > >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> _______________________________________________ >> >>> Bro mailing list >> >>> bro at bro-ids.org >> >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro< >> https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.ICSI.Berkeley.EDU_mailman_listinfo_bro&d=CwMD-g&c=8hUWFZcy2Z-Za5rBPlktOQ&r=gMEsgy9kNQo7aTfyIJsOSuw4Z57hfQyz6uV2H4S9PvE&m=jKm5KEaN1h0UfG6EGbAMSpW5NbXR4pzULAvbXFWNnvg&s=V2Ec5cOcnYqN7P3EpauWtYZUCrRNMUzy_pvRrgdH_C8&e= >> > >> >>> >> >>> _______________________________________________ >> >>> Bro mailing list >> >>> bro at bro-ids.org >> >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >>> >> >>> ------ >> >>> >> >>> Adam J. Slagell >> >>> Chief Information Security Officer >> >>> Director, Cybersecurity Division >> >>> National Center for Supercomputing Applications >> >>> University of Illinois at Urbana-Champaign >> >>> www.slagell.info >> >>> >> >>> "Under the Illinois Freedom of Information Act (FOIA), any written >> communication to or from University employees regarding University business >> is a public record and may be subject to public disclosure." >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >> >> > >> > ------ >> > >> > Adam J. Slagell >> > Chief Information Security Officer >> > Director, Cybersecurity Division >> > National Center for Supercomputing Applications >> > University of Illinois at Urbana-Champaign >> > www.slagell.info >> > >> > "Under the Illinois Freedom of Information Act (FOIA), any written >> communication to or from University employees regarding University business >> is a public record and may be subject to public disclosure." >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > _______________________________________________ >> > Bro mailing list >> > bro at bro-ids.org >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro < >> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro> >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160715/955bdb0d/attachment.html >> -------------- next part -------------- >> A non-text attachment was scrubbed... >> Name: signature.asc >> Type: application/pgp-signature >> Size: 842 bytes >> Desc: Message signed with OpenPGP using GPGMail >> Url : >> http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160715/955bdb0d/attachment.bin >> >> ------------------------------ >> >> _______________________________________________ >> Bro mailing list >> Bro at bro.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> >> End of Bro Digest, Vol 123, Issue 24 >> ************************************ >> > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160716/ed44dc11/attachment-0001.html From tgdesrochers at gmail.com Sun Jul 17 17:10:53 2016 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Sun, 17 Jul 2016 20:10:53 -0400 Subject: [Bro] [bro] intel framework Message-ID: Is there a way to use the intel framework to alert on something like this /templates/nivoslider/loading.php I don't care about the domain I just care about the URI. The adversary keeps using DGA domains but the rest stays the same. I read the intel framework section online and I don't see anything that appears it would match this type of intel. Thanks Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160717/7b763d4c/attachment.html From anthony.kasza at gmail.com Sun Jul 17 17:42:29 2016 From: anthony.kasza at gmail.com (anthony kasza) Date: Sun, 17 Jul 2016 17:42:29 -0700 Subject: [Bro] [bro] intel framework In-Reply-To: References: Message-ID: This should work: https://github.com/bro/bro/blob/master/scripts/policy/frameworks/intel/seen/http-url.bro The Intel frameworks works on a plugin system. You should be able to add some protocol fields by writing a new scripts if what you need isn't already there. -AK On Jul 17, 2016 7:19 PM, "Tim Desrochers" wrote: > Is there a way to use the intel framework to alert on something like this > > /templates/nivoslider/loading.php > > I don't care about the domain I just care about the URI. The adversary > keeps using DGA domains but the rest stays the same. > > I read the intel framework section online and I don't see anything that > appears it would match this type of intel. > > Thanks > Tim > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160717/034d8950/attachment.html From hosom at battelle.org Mon Jul 18 05:00:24 2016 From: hosom at battelle.org (Hosom, Stephen M) Date: Mon, 18 Jul 2016 12:00:24 +0000 Subject: [Bro] [bro] intel framework In-Reply-To: References: , Message-ID: You could also use signatures for this. https://www.bro.org/sphinx/frameworks/signatures.html ________________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of anthony kasza [anthony.kasza at gmail.com] Sent: Sunday, July 17, 2016 8:42 PM To: Tim Desrochers Cc: bro at bro.org Subject: Re: [Bro] [bro] intel framework This should work: https://github.com/bro/bro/blob/master/scripts/policy/frameworks/intel/seen/http-url.bro The Intel frameworks works on a plugin system. You should be able to add some protocol fields by writing a new scripts if what you need isn't already there. -AK On Jul 17, 2016 7:19 PM, "Tim Desrochers" > wrote: Is there a way to use the intel framework to alert on something like this /templates/nivoslider/loading.php I don't care about the domain I just care about the URI. The adversary keeps using DGA domains but the rest stays the same. I read the intel framework section online and I don't see anything that appears it would match this type of intel. Thanks Tim _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jazoff at illinois.edu Mon Jul 18 05:16:33 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 18 Jul 2016 12:16:33 +0000 Subject: [Bro] [bro] intel framework In-Reply-To: References: Message-ID: <961DBF2F-B2E3-4A3D-886C-081CACCB8940@illinois.edu> > On Jul 18, 2016, at 2:42 AM, anthony kasza wrote: > > This should work: > > https://github.com/bro/bro/blob/master/scripts/policy/frameworks/intel/seen/http-url.bro > > The Intel frameworks works on a plugin system. You should be able to add some protocol fields by writing a new scripts if what you need isn't already there. That is close, but won't work for this. the http-url script uses $indicator=HTTP::build_url(c$http) build_url basically does host + uri with some extra smarts for all the edge cases. To have the intel framework just flag the path, you would need a variation of that script that only sets the indicator to the path: event http_message_done(c: connection, is_orig: bool, stat: http_message_stat) { if ( is_orig && c?$http && c$http$?uri) Intel::seen([$indicator=c$http$uri), $indicator_type=Intel::URL, $conn=c, $where=HTTP::IN_URL]); } You may need/want to remove any query string at the end of the path. I don't think using Intel::URL for something that is not really a URL will cause a problem, but it is slightly confusing :-) If you do not want that you can add a new intel type called URL_PATH. -- - Justin Azoff From tgdesrochers at gmail.com Mon Jul 18 06:25:14 2016 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Mon, 18 Jul 2016 09:25:14 -0400 Subject: [Bro] [bro] intel framework In-Reply-To: References: Message-ID: signatures seems to be what I was looking for, thanks for the tip On Mon, Jul 18, 2016 at 8:00 AM, Hosom, Stephen M wrote: > You could also use signatures for this. > > https://www.bro.org/sphinx/frameworks/signatures.html > > ________________________________________ > From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of anthony > kasza [anthony.kasza at gmail.com] > Sent: Sunday, July 17, 2016 8:42 PM > To: Tim Desrochers > Cc: bro at bro.org > Subject: Re: [Bro] [bro] intel framework > > This should work: > > > https://github.com/bro/bro/blob/master/scripts/policy/frameworks/intel/seen/http-url.bro > > The Intel frameworks works on a plugin system. You should be able to add > some protocol fields by writing a new scripts if what you need isn't > already there. > > -AK > > On Jul 17, 2016 7:19 PM, "Tim Desrochers" tgdesrochers at gmail.com>> wrote: > Is there a way to use the intel framework to alert on something like this > > /templates/nivoslider/loading.php > > I don't care about the domain I just care about the URI. The adversary > keeps using DGA domains but the rest stays the same. > > I read the intel framework section online and I don't see anything that > appears it would match this type of intel. > > Thanks > Tim > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160718/973792e8/attachment.html From jdopheid at illinois.edu Mon Jul 18 08:16:52 2016 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Mon, 18 Jul 2016 15:16:52 +0000 Subject: [Bro] =?windows-1252?q?BroCon_=9216=3A_Don=27t_forget_to_book_you?= =?windows-1252?q?r_hotel?= Message-ID: <7EFD7D614A2BB84ABEA19B2CEDD246581C2E5AC1@CITESMBX5.ad.uillinois.edu> Bro Community, Are you planning to attend BroCon this year? If so, don't forget to book your hotel: https://www.bro.org/community/brocon2016.html#hotelinformation We reserved a block of rooms less than a half a mile away from the conference site at the Lone Star Court. One thing that might not be clear from the website is they do not charge your card to book the reservation, in case that detail is causing you to delay booking your stay. Our hold on the rooms expires August 22nd. See you in August, The Bro Team -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160718/b031181c/attachment-0001.html From jdopheid at illinois.edu Mon Jul 18 08:28:22 2016 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Mon, 18 Jul 2016 15:28:22 +0000 Subject: [Bro] =?windows-1252?q?BroCon_=9216=3A_Don=27t_forget_to_book_you?= =?windows-1252?q?r_hotel?= In-Reply-To: <7EFD7D614A2BB84ABEA19B2CEDD246581C2E5AC1@CITESMBX5.ad.uillinois.edu> References: <7EFD7D614A2BB84ABEA19B2CEDD246581C2E5AC1@CITESMBX5.ad.uillinois.edu> Message-ID: <7EFD7D614A2BB84ABEA19B2CEDD246581C2E5AD8@CITESMBX5.ad.uillinois.edu> Oops, actually, we'll see you in September. :) ------ Jeannette M. Dopheide Bro Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign ________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Dopheide, Jeannette M [jdopheid at illinois.edu] Sent: Monday, July 18, 2016 10:16 AM To: bro at bro.org Subject: [Bro] BroCon ?16: Don't forget to book your hotel Bro Community, Are you planning to attend BroCon this year? If so, don't forget to book your hotel: https://www.bro.org/community/brocon2016.html#hotelinformation We reserved a block of rooms less than a half a mile away from the conference site at the Lone Star Court. One thing that might not be clear from the website is they do not charge your card to book the reservation, in case that detail is causing you to delay booking your stay. Our hold on the rooms expires August 22nd. See you in August September, The Bro Team -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160718/b71c446d/attachment.html From obdnanr at gmail.com Mon Jul 18 08:43:32 2016 From: obdnanr at gmail.com (Obndnar smith) Date: Mon, 18 Jul 2016 15:43:32 +0000 Subject: [Bro] Bro and PF_Ring Message-ID: I've followed the steps to get Bro to use pf_ring and it even shows that it's using the pf_ring/lib, but as soon as I install from my manager it reverts back to libpcap. Any ideas? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160718/ff8d0bb3/attachment.html From wulfyh at gmail.com Mon Jul 18 09:58:41 2016 From: wulfyh at gmail.com (Wulfy H) Date: Mon, 18 Jul 2016 09:58:41 -0700 Subject: [Bro] Insider Abuse Use Case Message-ID: Hello all, On slide 11 of this presentation: https://www.bro.org/bro-workshop-2011/slides/network-forensics.pdf There is a Use Case for Insider Abuse, I am interested in this and am a beginner to Bro IDS scripting. Is there any existing script dealing with some form of Insider Abuse that I can use as an example? Thanks, W. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160718/6b26a19a/attachment.html From hosom at battelle.org Mon Jul 18 11:39:23 2016 From: hosom at battelle.org (Hosom, Stephen M) Date: Mon, 18 Jul 2016 18:39:23 +0000 Subject: [Bro] Insider Abuse Use Case In-Reply-To: References: Message-ID: "Insider Abuse" is a very wide topic. Is there something a little more specific that you're looking to do? ________________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Wulfy H [wulfyh at gmail.com] Sent: Monday, July 18, 2016 12:58 PM To: bro at bro.org Subject: [Bro] Insider Abuse Use Case Hello all, On slide 11 of this presentation: https://www.bro.org/bro-workshop-2011/slides/network-forensics.pdf There is a Use Case for Insider Abuse, I am interested in this and am a beginner to Bro IDS scripting. Is there any existing script dealing with some form of Insider Abuse that I can use as an example? Thanks, W. From jazoff at illinois.edu Mon Jul 18 13:02:39 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 18 Jul 2016 20:02:39 +0000 Subject: [Bro] Bro and PF_Ring In-Reply-To: References: Message-ID: > On Jul 18, 2016, at 5:43 PM, Obndnar smith wrote: > > I've followed the steps to get Bro to use pf_ring and it even shows that it's using the pf_ring/lib, but as soon as I install from my manager it reverts back to libpcap. Any ideas? It sounds like you are building and installing bro on the worker nodes a well as on the manager nodes. You only need to install bro on the manager node. broctl copies the bro installation to each worker node for you. The process for using pf_ring on a bro cluster would be: 1) install pf_ring kernel module and libraries on each worker 2) install pf_ring libraries on the manager - You can install the kernel modules if you wanted to, but nothing will use them. 3) install bro on the manager If you are missing the pf_ring libraries on the manager that will cause the manager binary to not be linked against pf_ring. -- - Justin Azoff From vladg at illinois.edu Mon Jul 18 14:06:23 2016 From: vladg at illinois.edu (Vlad Grigorescu) Date: Mon, 18 Jul 2016 16:06:23 -0500 Subject: [Bro] Help with missed_bytes affecting hash creations in files.log In-Reply-To: <37325130.3263906.1468518373774.JavaMail.yahoo@mail.yahoo.com> References: <37325130.3263906.1468518373774.JavaMail.yahoo.ref@mail.yahoo.com> <37325130.3263906.1468518373774.JavaMail.yahoo@mail.yahoo.com> Message-ID: Try adding this to local.bro: @load misc/capture-loss And then checking the capture_loss.log file which it will generate (will take 15 minutes to get it to appear initially). For more information about capture loss, see: https://www.bro.org/documentation/faq.html#how-can-i-reduce-the-amount-of-captureloss-or-dropped-packets-notices --Vlad Stephen Castellarin writes: > Hi, > I have Bro 2.4.1 running on an older system (2 Intel X5550 processors giving 8 CPUs), 48Gb memory running 64 bit Ubuntu (14.04.4) server, using PF_Ring with an Intel 82571EB Ethernet card (1gb copper).? This system is sitting on a network tap that is just seeing SMTP traffic between our outer mail gateway and our inside mail infrastructure.? My Bro configuration is relatively simple, with a nodes.cfg being: > > [manager] > type=manager > host=localhost > # > [proxy-1] > type=proxy > host=localhost > # > [worker-1] > type=worker > host=localhost > interface=eth5 > lb_method=pf_ring > lb_procs=8 > > When I look at the files.log file I see instances of files that have missing_bytes, which causes the hashes to not be calculated.? Running an IFCONFIG I don't see any drops, errors, etc.? Same with running broctl netstats, no drops.? SAR reports on that system show the CPUs running at 73% IDLE. > Is there something I'm missing in tuning or tweaking our configuration?? Can I get to a point where I have zero files with no missed_bytes, or will there always be something or things with missed_bytes.? A hardware upgrade can be in our future, but I'm trying to prove this concept by using this setup to help get funding for upgrading. > > Thanks all,Steve > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 800 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160718/9986a45d/attachment.bin From cgaylord at vt.edu Tue Jul 19 06:54:29 2016 From: cgaylord at vt.edu (Clark Gaylord) Date: Tue, 19 Jul 2016 09:54:29 -0400 Subject: [Bro] Bro and PF_Ring In-Reply-To: References: Message-ID: On Jul 18, 2016 16:18, "Azoff, Justin S" wrote: > > It sounds like you are building and installing bro on the worker nodes a well as on the manager nodes. You only need to install bro on the manager node. broctl copies the bro installation to each worker node for you. ... > 3) install bro on the manager > > If you are missing the pf_ring libraries on the manager that will cause the manager binary to not be linked against pf_ring. Making sure I follow you: And hence the binary distributed to the workers by the manager will not pick up those libraries, even though they do exist on the worker? -- Clark Gaylord cgaylord at vt.edu ... autocorrect may have improved this message brevity should not imply curtness ... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160719/83ec6087/attachment.html From castle1126 at yahoo.com Tue Jul 19 07:49:25 2016 From: castle1126 at yahoo.com (Stephen Castellarin) Date: Tue, 19 Jul 2016 14:49:25 +0000 (UTC) Subject: [Bro] Help with missed_bytes affecting hash creations in files.log In-Reply-To: References: <37325130.3263906.1468518373774.JavaMail.yahoo.ref@mail.yahoo.com> <37325130.3263906.1468518373774.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1404551269.1841999.1468939765733.JavaMail.yahoo@mail.yahoo.com> Hi Vlad, Yes I've had the capture_loss script enabled for some time on my system.? Looking at today's entries - I've seen percent loss in? 154 of 344 entries, with the largest percentage being 2.9%.? Checking "broctl netstats" I'm showing 0 dropped by my workers. Steve On Monday, July 18, 2016 5:06 PM, Vlad Grigorescu wrote: Try adding this to local.bro: @load misc/capture-loss And then checking the capture_loss.log file which it will generate (will take 15 minutes to get it to appear initially). For more information about capture loss, see: https://www.bro.org/documentation/faq.html#how-can-i-reduce-the-amount-of-captureloss-or-dropped-packets-notices ? --Vlad Stephen Castellarin writes: > Hi, > I have Bro 2.4.1 running on an older system (2 Intel X5550 processors giving 8 CPUs), 48Gb memory running 64 bit Ubuntu (14.04.4) server, using PF_Ring with an Intel 82571EB Ethernet card (1gb copper).? This system is sitting on a network tap that is just seeing SMTP traffic between our outer mail gateway and our inside mail infrastructure.? My Bro configuration is relatively simple, with a nodes.cfg being: > > [manager] > type=manager > host=localhost > # > [proxy-1] > type=proxy > host=localhost > # > [worker-1] > type=worker > host=localhost > interface=eth5 > lb_method=pf_ring > lb_procs=8 > > When I look at the files.log file I see instances of files that have missing_bytes, which causes the hashes to not be calculated.? Running an IFCONFIG I don't see any drops, errors, etc.? Same with running broctl netstats, no drops.? SAR reports on that system show the CPUs running at 73% IDLE. > Is there something I'm missing in tuning or tweaking our configuration?? Can I get to a point where I have zero files with no missed_bytes, or will there always be something or things with missed_bytes.? A hardware upgrade can be in our future, but I'm trying to prove this concept by using this setup to help get funding for upgrading. > > Thanks all,Steve > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160719/d355c1d5/attachment.html From johanna at icir.org Tue Jul 19 10:58:06 2016 From: johanna at icir.org (Johanna Amann) Date: Tue, 19 Jul 2016 10:58:06 -0700 Subject: [Bro] Telnet analyzer In-Reply-To: <5788F43B.7000503@cert.unlp.edu.ar> References: <5788F43B.7000503@cert.unlp.edu.ar> Message-ID: <20160719175806.GA16214@wifi241.sys.ICSI.Berkeley.EDU> Hello Christian, could you perhaps send me an example trace for this (or make a ticket on tracker.bro.org and attach it there)? I could take a short look if this is eomething that is easy to fix. Johanna On Fri, Jul 15, 2016 at 11:33:31AM -0300, Cristian Daniel Barbaro wrote: > Hello, people. > I'm trying detect when someone connects to Telnet and login is > sucessful. But Bro never detects this event, only login_confused or > login_confused_text events was detected. > > Any idea? > > Thanks. Regards... > > -- > Cristian Daniel Barbaro > CERTUNLP > -- > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From wulfyh at gmail.com Tue Jul 19 12:14:01 2016 From: wulfyh at gmail.com (Wulfy H) Date: Tue, 19 Jul 2016 15:14:01 -0400 Subject: [Bro] syntax highlighting for bro Message-ID: I've been trying to set up Bro syntax highlighting for Sublime text 2 using Liam Randall's guide: http://liamrandall.com/syntax-highlighting-for-bro-network-programming-language/ But I'm running in to trouble on the last step and can't seem to find the Windows 10 equivalent of the config directory that I'm supposed to put the bro.tmbundle. I see a folder called "Pristine Packages" the C directory folder for sublime but I don't believe this is it. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160719/758911ee/attachment.html From jan.grashoefer at gmail.com Tue Jul 19 13:08:28 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Tue, 19 Jul 2016 22:08:28 +0200 Subject: [Bro] syntax highlighting for bro In-Reply-To: References: Message-ID: Hi, > But I'm running in to trouble on the last step and can't seem to find the > Windows 10 equivalent of the config directory that I'm supposed to put the > bro.tmbundle. I see a folder called "Pristine Packages" the C directory > folder for sublime but I don't believe this is it. Have a look at https://github.com/bro/bro-sublime. I think that should be the most recent version. For me "Preferences -> Browse Packages" leads to "C:\Users\\AppData\Roaming\Sublime Text 3\Packages" on Windows 7. Jan From vallentin at icir.org Thu Jul 21 08:40:33 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Thu, 21 Jul 2016 08:40:33 -0700 Subject: [Bro] Insider Abuse Use Case In-Reply-To: References: Message-ID: <20160721154033.GH38625@ninja.local> > There is a Use Case for Insider Abuse, I am interested in this and am a > beginner to Bro IDS scripting. Is there any existing script dealing with > some form of Insider Abuse that I can use as an example? Rather than having a script for one particular instance of insider abuse, I wanted to highlight overall approach towards in this talk. What makes insider abuse hard to detect, is that often each individual action in isolation is legit, but only constitute a policy violation when analyzed in sequence. The challenge lies in analyzing chains of actions. Doing so live (i.e., while analyzing traffic in real time) may not be feasible because such actions often manifest over longer time periods. Therefore, detection angles often rely on summaries of past activity, such as behavior profiles. But this goes quickly into distilling patterns of normality and then flagging deviations (with all its pitfalls [1]). That said, I'm sure there are simpler, concrete instances of insider abuse which can be readily coded up in Bro. It all depends on the policy of your site and the assets you're trying to protect. Matthias [1] http://www.icir.org/robin/papers/oakland10-ml.pdf From tgdesrochers at gmail.com Thu Jul 21 09:35:01 2016 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Thu, 21 Jul 2016 12:35:01 -0400 Subject: [Bro] originator/responder possibly swapped Message-ID: Is it possible for bro to swap the originator and responder IP addresses in its logs. Below you will see the only conn.log I have for a IP address. The originator is the external host and responder is the internal IP of my org. Then every other log I have ssh and notice where this IP shows up the originator is my internal host and the responder is the external IP. conn.15: 00: 00-16: 00: 00.log.gz: { "ts": 1469114110842, "uid": "ClvNzi2EnWQnmykMZ7", "id.orig_h": "EXTERNAL IP", "id.orig_p": 15000, "id.resp_h": "INTERNAL HOST", "id.resp_p": 1043, "proto": "tcp", "duration": 0.658319, "orig_bytes": 416, "resp_bytes": 976, "conn_state": "SF", "local_orig": false, "local_resp": true, "missed_bytes": 0, "history": "DadAfF", "orig_pkts": 12, "orig_ip_bytes": 1040, "resp_pkts": 10, "resp_ip_bytes": 1496, "tunnel_parents": [], "orig_cc": "US", "sensorname": "SENSOR-1" }notice.15: 00: 00-16: 00: 00.log.gz: { "ts": 1469113976350, "uid": "CseS0Q2AQ0biwoE97g", "id.orig_h": "INTERNAL HOST", "id.orig_p": 1024, "id.resp_h": "EXTERNAL IP", "id.resp_p": 15000, "proto": "tcp", "note": "SSH::Interesting_Hostname_Login", "msg": "Possible SSH login involving a remote server with an interesting hostname.", "sub": "EXTERNAL DOMAIL", "src": "10.21.4.124", "dst": "EXTERNAL IP", "p": 15000, "peer_descr": "SENSOR-1", "actions": ["Notice::ACTION_EMAIL", "Notice::ACTION_LOG"], "suppress_for": 3600.0, "dropped": false }ssh.15: 00: 00-16: 00: 00.log.gz: { "ts": 1469116787409, "uid": "CWQdeiL75K07BRtb4", "id.orig_h": "INTERNAL HOST", "id.orig_p": 1427, "id.resp_h": "EXTERNAL IP", "id.resp_p": 15000, "version": 2, "auth_success": true, "direction": "OUTBOUND", "client": "SSH-2.0-OpenSSH_3.1p1", "server": "SSH-2.0-OpenSSH_6.0p1 Debian-4+deb7u2", "cipher_alg": "aes128-cbc", "mac_alg": "hmac-md5", "compression_alg": "none", "kex_alg": "diffie-hellman-group-exchange-sha1", "host_key_alg": "ssh-rsa", "host_key": "REMOVED", "remote_location.country_code": "US" } Thoughts as to why? Also, I know I saw this come up before but it has been burried, does auth_success:true indicate that ssh authentication was successful -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160721/569354b2/attachment.html From jdopheid at illinois.edu Thu Jul 21 09:45:40 2016 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Thu, 21 Jul 2016 16:45:40 +0000 Subject: [Bro] =?utf-8?q?BroCon_=E2=80=9916=3A_Agenda_is_now_available?= Message-ID: <5468A762-0177-428B-916C-DD062DFDEED2@illinois.edu> Hello Bro Community, We have posted the BroCon agenda to our site: https://www.bro.org/community/brocon2016.html#agenda Note this year we extended the lunch and break sessions to allow for more mingling. We have a few more pending presentations to announce, check back periodically for updates. Don?t forget to register and book your hotel. See you in September, The Bro Team ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From dhoelzer at sans.org Thu Jul 21 11:47:31 2016 From: dhoelzer at sans.org (Hoelzer, Dave) Date: Thu, 21 Jul 2016 18:47:31 +0000 Subject: [Bro] originator/responder possibly swapped In-Reply-To: References: Message-ID: It could be that Bro missed the first packet in the exchange. The originator will be whoever it sees first, afaik. From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Tim Desrochers Sent: Thursday, July 21, 2016 12:35 PM To: bro at bro.org Subject: [Bro] originator/responder possibly swapped Is it possible for bro to swap the originator and responder IP addresses in its logs. Below you will see the only conn.log I have for a IP address. The originator is the external host and responder is the internal IP of my org. Then every other log I have ssh and notice where this IP shows up the originator is my internal host and the responder is the external IP. conn.15: 00: 00-16: 00: 00.log.gz: { "ts": 1469114110842, "uid": "ClvNzi2EnWQnmykMZ7", "id.orig_h": "EXTERNAL IP", "id.orig_p": 15000, "id.resp_h": "INTERNAL HOST", "id.resp_p": 1043, "proto": "tcp", "duration": 0.658319, "orig_bytes": 416, "resp_bytes": 976, "conn_state": "SF", "local_orig": false, "local_resp": true, "missed_bytes": 0, "history": "DadAfF", "orig_pkts": 12, "orig_ip_bytes": 1040, "resp_pkts": 10, "resp_ip_bytes": 1496, "tunnel_parents": [], "orig_cc": "US", "sensorname": "SENSOR-1" }notice.15: 00: 00-16: 00: 00.log.gz: { "ts": 1469113976350, "uid": "CseS0Q2AQ0biwoE97g", "id.orig_h": "INTERNAL HOST", "id.orig_p": 1024, "id.resp_h": "EXTERNAL IP", "id.resp_p": 15000, "proto": "tcp", "note": "SSH::Interesting_Hostname_Login", "msg": "Possible SSH login involving a remote server with an interesting hostname.", "sub": "EXTERNAL DOMAIL", "src": "10.21.4.124", "dst": "EXTERNAL IP", "p": 15000, "peer_descr": "SENSOR-1", "actions": ["Notice::ACTION_EMAIL", "Notice::ACTION_LOG"], "suppress_for": 3600.0, "dropped": false }ssh.15: 00: 00-16: 00: 00.log.gz: { "ts": 1469116787409, "uid": "CWQdeiL75K07BRtb4", "id.orig_h": "INTERNAL HOST", "id.orig_p": 1427, "id.resp_h": "EXTERNAL IP", "id.resp_p": 15000, "version": 2, "auth_success": true, "direction": "OUTBOUND", "client": "SSH-2.0-OpenSSH_3.1p1", "server": "SSH-2.0-OpenSSH_6.0p1 Debian-4+deb7u2", "cipher_alg": "aes128-cbc", "mac_alg": "hmac-md5", "compression_alg": "none", "kex_alg": "diffie-hellman-group-exchange-sha1", "host_key_alg": "ssh-rsa", "host_key": "REMOVED", "remote_location.country_code": "US" } Thoughts as to why? Also, I know I saw this come up before but it has been burried, does auth_success:true indicate that ssh authentication was successful -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160721/0ce92f16/attachment-0001.html From ansaf_130 at yahoo.com Thu Jul 21 23:37:32 2016 From: ansaf_130 at yahoo.com (Aneela Safdar) Date: Fri, 22 Jul 2016 06:37:32 +0000 (UTC) Subject: [Bro] ftp.log file isn't logging ftp related requests. In-Reply-To: <1342536298.2807940.1469164239793.JavaMail.yahoo@mail.yahoo.com> References: <1342536298.2807940.1469164239793.JavaMail.yahoo.ref@mail.yahoo.com> <1342536298.2807940.1469164239793.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1116978397.3000825.1469169452604.JavaMail.yahoo@mail.yahoo.com> Hi, I have bro and proftpd server installed on linux. I tried to connect with this server through a intentionally brute force attack with random usernames and passwords thinking ftp.log will record these attempts but it didn't. Instead ftp.log I am getting these requests logged in weird.log file. I did same with ssh service but i logged all requests and related information in ssh.log file.How come I can make ftp.log file to log all ftp related information which I genuinely think should be a default setting. Isn't so? Regards, Aneela Safdar -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160722/f07618e8/attachment.html From bmixonb1 at cs.unm.edu Fri Jul 22 13:50:32 2016 From: bmixonb1 at cs.unm.edu (Ben Mixon-Baca) Date: Fri, 22 Jul 2016 13:50:32 -0700 Subject: [Bro] Issues with data structures Message-ID: Hi, I am trying to build a 2D histogram based byte values in a tcp payload. I look at each byte independently. So I get 2^8 possible byte values which I use as a row index. the first 256 or so bytes of the payload as passed from the tcp_packet function are where I get this, and the specific byte position in the payload is suppose to give me the column index. I initially was trying the following using the with and without the SumStats framwork. The code below is with SumStats: module BytePatterns; export { global other_counts: vector of table[string] of count; } $epoch_result(ts: time, key: SumStats::Key, result: SumStats::Result) = { local tmp: table[string] of count; if ("prefix" in result) { local i = 0; local payload = SumStats::key2str(key); for (bytei in payload) { bytei = string_to_ascii_hex(bytei); if (i !in BytePatterns::other_counts) { BytePatterns::other_counts[i] = tmp; } if (bytei !in BytePatterns::other_counts[i]) { BytePatterns::other_counts[i][bytei] = 0; } BytePatterns::other_counts[i][bytei] += 1; i += 1; } } event tcp_packet(c: connection , is_orig: bool , flags: string , seq: count , ack: count , len: count , payload: string ) { if (is_orig) { if (c$seen_syn == T) { c$acks += 1; if (c$acks == 2 && len > 0) { print fmt("%s", BytePatterns::eseen); SumStats::observe("prefix",[$str=payload], [$str=payload]); } } } } Based on other state, this event should only be getting 3 payloads with a trace file I made for testing. The column sum should therefore be 3 for all columns, however, when I run this code, I am getting significantly larger counts. I am wondering if the way I am adding tables to the other_counts, data structure is somehow causing this behavior. Or possibly something with the way I am using SumStats and it is not doing what I think it is. Thank you in advance. -- Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: OpenPGP digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160722/13b24855/attachment.bin From bmixonb1 at cs.unm.edu Fri Jul 22 14:18:00 2016 From: bmixonb1 at cs.unm.edu (Ben Mixon-Baca) Date: Fri, 22 Jul 2016 14:18:00 -0700 Subject: [Bro] Issues with data structures In-Reply-To: References: Message-ID: I figured it out. I wasn't performing a deep copy. I changed the code to this, and now it is giving me what I was expoecting: $epoch_result(ts: time, key: SumStats::Key, result: SumStats::Result) = { local tmp: table[string] of count; if ("prefix" in result) { local i = 0; local payload = SumStats::key2str(key); for (bytei in payload) { bytei = string_to_ascii_hex(bytei); if (i !in BytePatterns::other_counts) { BytePatterns::other_counts[i] = copy(tmp); } if (bytei !in BytePatterns::other_counts[i]) { BytePatterns::other_counts[i][bytei] = 0; } BytePatterns::other_counts[i][bytei] += 1; i += 1; } } On 07/22/2016 01:50 PM, Ben Mixon-Baca wrote: > Hi, > > I am trying to build a 2D histogram based byte values in a tcp payload. > I look at each byte independently. So I get 2^8 possible byte values > which I use as a row index. the first 256 or so bytes of the payload as > passed from the tcp_packet function are where I get this, and the > specific byte position in the payload is suppose to give me the column > index. I initially was trying the following using the with and without > the SumStats framwork. The code below is with SumStats: > > module BytePatterns; > > export { > global other_counts: vector of table[string] of count; > } > > > $epoch_result(ts: time, key: SumStats::Key, result: SumStats::Result) = > { > local tmp: table[string] of count; > if ("prefix" in result) > { > local i = 0; > local payload = SumStats::key2str(key); > for (bytei in payload) > { > bytei = string_to_ascii_hex(bytei); > if (i !in BytePatterns::other_counts) > { > BytePatterns::other_counts[i] = tmp; > } > if (bytei !in BytePatterns::other_counts[i]) > { > BytePatterns::other_counts[i][bytei] = 0; > } > BytePatterns::other_counts[i][bytei] += 1; > i += 1; > } > } > > > > event tcp_packet(c: connection , is_orig: bool , flags: string , seq: > count , ack: count , len: count , payload: string ) > { > if (is_orig) > { > if (c$seen_syn == T) > { > c$acks += 1; > if (c$acks == 2 && len > 0) > { > print fmt("%s", BytePatterns::eseen); > SumStats::observe("prefix",[$str=payload], [$str=payload]); > } > } > } > } > > > Based on other state, this event should only be getting 3 payloads with > a trace file I made for testing. The column sum should therefore be 3 > for all columns, however, when I run this code, I am getting > significantly larger counts. > > I am wondering if the way I am adding tables to the other_counts, data > structure is somehow causing this behavior. Or possibly something with > the way I am using SumStats and it is not doing what I think it is. > > Thank you in advance. > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: OpenPGP digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160722/6f3624f5/attachment.bin From bmixonb1 at cs.unm.edu Fri Jul 22 19:25:16 2016 From: bmixonb1 at cs.unm.edu (Ben Mixon-Baca) Date: Fri, 22 Jul 2016 19:25:16 -0700 Subject: [Bro] Inconsistent results Message-ID: <053b9c2d-f77d-9c49-d76a-4b3cabee0a67@cs.unm.edu> Is there any reason to expect Bro to give inconsistent output when running a script on trace files? I have a script that uses SumStats to fire off an observation in the tcp_packet event to save the payload for later analysis. I seem to be running into an issue where I count a smaller number of observed tcp_packet events than I should actually be seeing. -- Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: OpenPGP digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160722/e3214c09/attachment.bin From bmixonb1 at cs.unm.edu Sat Jul 23 13:36:24 2016 From: bmixonb1 at cs.unm.edu (Ben Mixon-Baca) Date: Sat, 23 Jul 2016 13:36:24 -0700 Subject: [Bro] Weird behavior Message-ID: <63b68e4f-a5f4-1067-5cfc-a9454b3b82b9@cs.unm.edu> Hi, I have been trying to find trace a bug in my code. I put print statements in several events including connection_SYN_packet. I am seeing this event getting fired off twice for every SYN packet seen on the wire. When I inspect the pcap with wireshark however, I have only found a single SYN packet. So I am wondering if there is something special happening in the event engine when using low level functions like connect_SYN_packet, that might cause this behavior. -- Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: OpenPGP digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160723/45190dad/attachment.bin From dhoelzer at sans.org Sun Jul 24 04:34:21 2016 From: dhoelzer at sans.org (Hoelzer, Dave) Date: Sun, 24 Jul 2016 11:34:21 +0000 Subject: [Bro] Weird behavior In-Reply-To: <63b68e4f-a5f4-1067-5cfc-a9454b3b82b9@cs.unm.edu> References: <63b68e4f-a5f4-1067-5cfc-a9454b3b82b9@cs.unm.edu> Message-ID: I have not looked, but might you be seeing the SYN-ACK from the respondent trigger the rule as well? ??????????????????? David Hoelzer Fellow, SANS Institute Dean of Faculty, SANS Technology Institute On July 23, 2016 at 4:39:13 PM, Ben Mixon-Baca (bmixonb1 at cs.unm.edu) wrote: Hi, I have been trying to find trace a bug in my code. I put print statements in several events including connection_SYN_packet. I am seeing this event getting fired off twice for every SYN packet seen on the wire. When I inspect the pcap with wireshark however, I have only found a single SYN packet. So I am wondering if there is something special happening in the event engine when using low level functions like connect_SYN_packet, that might cause this behavior. -- Ben _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160724/e44e2216/attachment.html From tgdesrochers at gmail.com Sun Jul 24 08:01:37 2016 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Sun, 24 Jul 2016 11:01:37 -0400 Subject: [Bro] SYN/ACK Attack Message-ID: Bro Community I have been seeing A LOT of SYN/ACK attacks lately on my net and it seems that every time Bro is switching the orig and resp IP's. Luckily the history column has helped me determine which IP is the true src. Also, I have SiLK running on my bro sensors and the netflow traffic I get from SiLK is identifying the true src and resp IP addresses but Bro seems to switch the IP addresses. Is this because Bro thinks it missed the SYN packet so it switches the IP's? Should it be doing this? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160724/18d3235d/attachment.html From michael.fry at morphick.com Mon Jul 25 08:10:23 2016 From: michael.fry at morphick.com (Michael Fry) Date: Mon, 25 Jul 2016 15:10:23 +0000 Subject: [Bro] Assertion failure, Tag.cc In-Reply-To: <36D97979-14C9-4BA6-988B-715A4B79E205@icir.org> References: <36D97979-14C9-4BA6-988B-715A4B79E205@icir.org> Message-ID: Apologies for the delay. Below is the minimal script that will reproduce the core dump. As described earlier, files.log shows the same FUID with different connections multiple times. This causes the file_extraction_limit() to be called multiple times for the same file. Removing my event handler side-steps the core dump. redef FileExtract::prefix = "/tmp/bro"; # Files of sizes outside this range will be discarded global min_file_size: count = 1024; global max_file_size: count = 5000000; event bro_init() { mkdir(FileExtract::prefix); } event file_sniff(f: fa_file, meta: fa_metadata) { if (f$source != "HTTP") return; Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_limit=max_file_size]); Files::add_analyzer(f, Files::ANALYZER_MD5); } event file_state_remove(f: fa_file) { if (f$info?$extracted) { print(fmt("Deleting %s, size=%s", f$info$extracted, f$seen_bytes)); system(fmt("rm \"/tmp/bro/%s\"", str_shell_escape(f$info$extracted))); } } event file_extraction_limit(f: fa_file , args: any , limit: count , len: count) { print "file_extraction_limit():", f; flush_all(); } Here is an example that led to a crash. Note the same FUID is logged multiple times with different connection UIDs. root at redacted:/opt/cbts/log/bro/2016-06-28# zgrep FBzliCXoEipv0oEM8 * files.2016-06-28-18-00-00.log.dz:1467138209.682287 FBzliCXoEipv0oEM8 13.107.4.50 10.100.162.133 CapHn31QDgLbeGCqIh HTTP 0 EXTRACT,SHA1,MD5 application/zip Microsoft.Getstarted_3.11.1.0_neutral_~_8wekyb3d8bbwe.AppxBundle 0.010122 F F 22610 8811168 0 0 F - 02af20e0f97904b4c9da2c5a0071c324 b09760ed7e7f64e389eb709abf97cbbc395cd22a - HTTP-FBzliCXoEipv0oEM8-Microsoft.Getstarted_3.11.1.0_neutral_~_8wekyb3d8bbwe.AppxBundle.file files.2016-06-28-18-00-00.log.dz:1467138211.315952 FBzliCXoEipv0oEM8 13.107.4.50 10.100.162.133 CbGwtH2ajMS3nq8oR9 HTTP 0 MD5,SHA1,EXTRACT application/zip Microsoft.Getstarted_3.11.1.0_neutral_~_8wekyb3d8bbwe.AppxBundle 0.007328 F F 13850 8811168 0 0 F - 7d8f20dc6941d42cb735bfbb1f5dee1d dccb1e3e513ab377ee909823c538ba196140a6df - HTTP-FBzliCXoEipv0oEM8-Microsoft.Getstarted_3.11.1.0_neutral_~_8wekyb3d8bbwe.AppxBundle.file files.2016-06-28-18-00-00.log.dz:1467138211.234745 FBzliCXoEipv0oEM8 13.107.4.50 10.100.162.133 ClZAlx4VvTPCGXWnGf HTTP 0 SHA1,EXTRACT,MD5 application/zip Microsoft.Getstarted_3.11.1.0_neutral_~_8wekyb3d8bbwe.AppxBundle 0.010127 F F 22610 8811168 0 0 F - 02af20e0f97904b4c9da2c5a0071c324 b09760ed7e7f64e389eb709abf97cbbc395cd22a - HTTP-FBzliCXoEipv0oEM8-Microsoft.Getstarted_3.11.1.0_neutral_~_8wekyb3d8bbwe.AppxBundle.file files.2016-06-28-18-00-00.log.dz:1467138211.395275 FBzliCXoEipv0oEM8 13.107.4.50 10.100.162.133 C1PiIFx6hRC32DES9 HTTP 0 SHA1,MD5,EXTRACT application/zip Microsoft.Getstarted_3.11.1.0_neutral_~_8wekyb3d8bbwe.AppxBundle 0.008257 F F 16770 8811168 0 0 F - c6c39cc6b8a6773a8b4b416a919cfca6 0ae53083b1fc02d7710e9c2b12ad930990e73c8e - HTTP-FBzliCXoEipv0oEM8-Microsoft.Getstarted_3.11.1.0_neutral_~_8wekyb3d8bbwe.AppxBundle.file http.2016-06-28-18-00-00.log.dz:1467138209.662449 CapHn31QDgLbeGCqIh 10.100.162.133 61074 13.107.4.50 80 2 GET tlu.dl.delivery.mp.microsoft.com /filestreamingservice/files/9f5dbc38-f074-45be-8346-ca45b940a576?P1=1467138990&P2=301&P3=2&P4=M61a7Qj+Xwl55JKMXfAzkJPsnDstij/KQXct1zwrcGI= - Microsoft BITS/7.8 0 22610 200 OK - -- (empty) - - - - - FBzliCXoEipv0oEM8 application/zip http.2016-06-28-18-00-00.log.dz:1467138209.748598 CalgEr4pL03r1nZKql 10.100.162.133 61075 13.107.4.50 80 1 GET tlu.dl.delivery.mp.microsoft.com /filestreamingservice/files/9f5dbc38-f074-45be-8346-ca45b940a576?P1=1467138990&P2=301&P3=2&P4=M61a7Qj+Xwl55JKMXfAzkJPsnDstij/KQXct1zwrcGI= - Microsoft BITS/7.8 0 94880 206 Partial Content - - - (empty) - - - - - FBzliCXoEipv0oEM8 - http.2016-06-28-18-00-00.log.dz:1467138211.295292 CbGwtH2ajMS3nq8oR9 10.100.162.133 61077 13.107.4.50 80 1 GET tlu.dl.delivery.mp.microsoft.com /filestreamingservice/files/9f5dbc38-f074-45be-8346-ca45b940a576?P1=1467138990&P2=301&P3=2&P4=M61a7Qj+Xwl55JKMXfAzkJPsnDstij/KQXct1zwrcGI= - Microsoft BITS/7.8 0 13850 200 OK - -- (empty) - - - - - FBzliCXoEipv0oEM8 application/zip http.2016-06-28-18-00-00.log.dz:1467138211.214133 ClZAlx4VvTPCGXWnGf 10.100.162.133 61076 13.107.4.50 80 1 GET tlu.dl.delivery.mp.microsoft.com /filestreamingservice/files/9f5dbc38-f074-45be-8346-ca45b940a576?P1=1467138990&P2=301&P3=2&P4=M61a7Qj+Xwl55JKMXfAzkJPsnDstij/KQXct1zwrcGI= - Microsoft BITS/7.8 0 22610 200 OK - -- (empty) - - - - - FBzliCXoEipv0oEM8 application/zip http.2016-06-28-18-00-00.log.dz:1467138211.373513 C1PiIFx6hRC32DES9 10.100.162.133 61078 13.107.4.50 80 1 GET tlu.dl.delivery.mp.microsoft.com /filestreamingservice/files/9f5dbc38-f074-45be-8346-ca45b940a576?P1=1467138990&P2=301&P3=2&P4=M61a7Qj+Xwl55JKMXfAzkJPsnDstij/KQXct1zwrcGI= - Microsoft BITS/7.8 0 16770 200 OK - -- (empty) - - - - - FBzliCXoEipv0oEM8 application/zip Thanks, Mike On Thu, Jun 30, 2016 at 12:02 AM Seth Hall wrote: > > > On Jun 29, 2016, at 6:58 PM, Michael Fry > wrote: > > > > I've been able to narrow down the circumstances where we see this core > dump. In the dozens of times that I've seen it, the file being extracted is > always delivered over HTTP via the BITS client. > > Could you send the script that you are using which causes this problem? > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160725/422fc7b5/attachment.html From robin at icir.org Mon Jul 25 08:18:16 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 25 Jul 2016 08:18:16 -0700 Subject: [Bro] SYN/ACK Attack In-Reply-To: References: Message-ID: <20160725151816.GF27478@icir.org> On Sun, Jul 24, 2016 at 11:01 -0400, Tim Desrochers wrote: > I have been seeing A LOT of SYN/ACK attacks lately on my net and it seems > that every time Bro is switching the orig and resp IP's. Bro has a few heuristics when to flip the endpoints. The main one is that when it misses the initial SYN but the SYN/ACK is coming *from* a well-known server port, it assumes it's seeing a responder-side packet. This table determines what Bro considers a server port: https://www.bro.org/sphinx/scripts/base/init-bare.bro.html#id-likely_server_ports To help spot cases where the direction got slipped, master recently got a feature that now adds a '^' flag to the connection history in these cases. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From robin at icir.org Mon Jul 25 08:26:54 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 25 Jul 2016 08:26:54 -0700 Subject: [Bro] Weird behavior In-Reply-To: <63b68e4f-a5f4-1067-5cfc-a9454b3b82b9@cs.unm.edu> References: <63b68e4f-a5f4-1067-5cfc-a9454b3b82b9@cs.unm.edu> Message-ID: <20160725152654.GG27478@icir.org> On Sat, Jul 23, 2016 at 13:36 -0700, Ben Mixon-Baca wrote: > special happening in the event engine when using low level functions > like connect_SYN_packet, that might cause this behavior. Generally, there shouldn't. It's hard to say what's happening without seeing the packets. If you can send a small trace exhibiting the problem and the Bro script/command line you're using, we can probably figure it out pretty quickly. And just to confirm what Dave wrote: yes, SYN/ACKs will trigger the event as well, pkt$is_orig says which side the packet came from. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jan.grashoefer at gmail.com Mon Jul 25 12:06:34 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Mon, 25 Jul 2016 21:06:34 +0200 Subject: [Bro] SYN/ACK Attack In-Reply-To: <20160725151816.GF27478@icir.org> References: <20160725151816.GF27478@icir.org> Message-ID: <6b300590-919a-810d-4115-7727937a7e30@gmail.com> Since originator/responder depends on the protocol logic, another solution would be to log the source address of the first packet of the connection. I've just written a small script (https://gist.github.com/J-Gras/f6bfb6092d29aa0e9c53eb98e23a7955) that should achieve this. As it uses the new bif "get_current_packet_header()", the script only works with master (see http://try.bro.org/#/trybro/saved/80298). Best regards, Jan From seth at broala.com Mon Jul 25 13:53:33 2016 From: seth at broala.com (Seth Hall) Date: Mon, 25 Jul 2016 16:53:33 -0400 Subject: [Bro] Broala job openings Message-ID: Hi all! I'm posting from my Broala address this time because I wanted to let everyone know that we have a couple of job openings at Broala. If anyone is interested in working with us as we develop the BroBox platform, please reach out, we'd love to hear from you! Here are the links to the job postings: https://www.broala.com/platform-engineer.html https://www.broala.com/platform-quality-engineer.html Thanks, .Seth -- Seth Hall * Broala * seth at broala.com * www.broala.com From daniel.manzo at bayer.com Tue Jul 26 06:24:53 2016 From: daniel.manzo at bayer.com (Daniel Manzo) Date: Tue, 26 Jul 2016 13:24:53 +0000 Subject: [Bro] Network taps necessary for Bro Message-ID: <2C7473428EFB4348960ACC47FDC529457F2590@MOXCXR.na.bayer.cnb> Hi all, My team is looking into using the Bro IDS for monitoring of a science DMZ with a 100 Gbps network. I was wondering how to choose which network tap(s) is necessary for this type of connection and if you have any recommendations/methods for setting up the hardware for Bro. I have been looking at the passive Ixia Flex taps, but after reading the paper on bro.org about the 100G connection in Berkeley Labs, I'm not so sure this is the right direction. Thanks for the help, Daniel Manzo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160726/6ff05f31/attachment.html From 55sjp55 at gmail.com Tue Jul 26 07:08:57 2016 From: 55sjp55 at gmail.com (Scott P) Date: Tue, 26 Jul 2016 10:08:57 -0400 Subject: [Bro] Newbie question Extract Binaries from traffic Message-ID: Newbie question added the following to my local.bro file #Extract EXEs redef HTTP::extract_file_types += /application\/x-dosexec/; redef FTP::extract_file_types += /application\/x-dosexec/; #Extract files to /nsm/bro/extracted redef HTTP::extraction_prefix = "/nsm/bro/extracted/http/http-item"; redef FTP::extraction_prefix = "/nsm/bro/extracted/ftp/ftp-file"; But when I test against the file I am getting: sudo bro -r http-putty.pcap /opt/bro/share/bro/site/local.bro error in /opt/bro/share/bro/site/local.bro, line 105: "redef" used but not previously defined (HTTP::extract_file_types) internal warning in /opt/bro/share/bro/site/local.bro, line 105: Can't document redef of HTTP::extract_file_types, identifier lookup failed error in /opt/bro/share/bro/site/local.bro, line 106: "redef" used but not previously defined (FTP::extract_file_types) internal warning in /opt/bro/share/bro/site/local.bro, line 106: Can't document redef of FTP::extract_file_types, identifier lookup failed error in /opt/bro/share/bro/site/local.bro, line 109: "redef" used but not previously defined (HTTP::extraction_prefix) internal warning in /opt/bro/share/bro/site/local.bro, line 109: Can't document redef of HTTP::extraction_prefix, identifier lookup failed error in /opt/bro/share/bro/site/local.bro, line 110: "redef" used but not previously defined (FTP::extraction_prefix) internal warning in /opt/bro/share/bro/site/local.bro, line 110: Can't document redef of FTP::extraction_prefix, identifier lookup failed Any insight would be helpful. -- *Read, *pause, *think,* pause, *write*, pause, (perhaps erase), pause, *read, *pause, (perhaps *go back*), pause, *write, ....* -- *Alan Turing (1936)* -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160726/6b6ed389/attachment.html From ansaf_130 at yahoo.com Tue Jul 26 09:18:27 2016 From: ansaf_130 at yahoo.com (Aneela Safdar) Date: Tue, 26 Jul 2016 16:18:27 +0000 (UTC) Subject: [Bro] Configuration of timing when to move log file from current folder References: <562083842.4600603.1469549907967.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <562083842.4600603.1469549907967.JavaMail.yahoo@mail.yahoo.com> Hi, I need to know where I can specified the time interval to move log files of bro from current folder to the respected day date named folder? Can I change it??Regards, Aneela Safdar -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160726/b0c4896b/attachment.html From dopheide at gmail.com Tue Jul 26 09:57:38 2016 From: dopheide at gmail.com (Mike Dopheide) Date: Tue, 26 Jul 2016 11:57:38 -0500 Subject: [Bro] Network taps necessary for Bro In-Reply-To: <2C7473428EFB4348960ACC47FDC529457F2590@MOXCXR.na.bayer.cnb> References: <2C7473428EFB4348960ACC47FDC529457F2590@MOXCXR.na.bayer.cnb> Message-ID: Granted, budget will enter into the equation, but I would highly recommend following LBL's model. I'd feed a passive 100G tap into a smarter tap/agg switch before your Bro cluster. If you setup shunting for elephant flows you'll likely be able to get by with a small Bro cluster simply by filtering out that large traffic. This is especially true for typical ScienceDMZ traffic. You'll want a tap/agg switch that can also load balance to the tool ports. If you were to choose an Arista switch, the shunting code already exists. https://github.com/ncsa/dumbno -Dop On Tue, Jul 26, 2016 at 8:24 AM, Daniel Manzo wrote: > Hi all, > > > > My team is looking into using the Bro IDS for monitoring of a science DMZ > with a 100 Gbps network. I was wondering how to choose which network tap(s) > is necessary for this type of connection and if you have any > recommendations/methods for setting up the hardware for Bro. I have been > looking at the passive Ixia Flex taps, but after reading the paper on > bro.org about the 100G connection in Berkeley Labs, I?m not so sure this > is the right direction. > > > > Thanks for the help, > > Daniel Manzo > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160726/5ba8cf82/attachment-0001.html From enno.rosinger at hpe.com Tue Jul 26 13:23:20 2016 From: enno.rosinger at hpe.com (Rosinger, Enno (DualStudy)) Date: Tue, 26 Jul 2016 20:23:20 +0000 Subject: [Bro] Issue: load balancer PF_RING drops 25% of incoming packets Message-ID: Hey All, I am running Bro with configured PF_RING load balancing. Strangely only 16 Million of my 21 Million packet input pass through the PF_RING kernel module. Nevertheless they are then distributed correctly on the Bro processes. How can I avoid this loss of 5 Million packets and how can I verify that PF_RING is configured correctly? I use Intel Corporation I350 Gigabit Network Connection as NICs. They work with the igb drivers. The input rate is 0.5Gb/s = 60k to 80k packets/s and currently I am working without the ZeroCopy drivers It is verified that all of my 21 Million packets are received by my NIC's driver. The PF_Ring module itself exists and BRO is running with load balancing. Looking forward to your response and hope to solve this problem with you. Below you will find more detailed information about my system. If you need something else let me know. Best, Enno Additional information: One interesting fact: I cannot run "make" in "PF_RING/userland/examples", because gcc: error: ../libpcap/libpcap.a: No such file or directory PF_RING/userland looks like this. Indeed "libpcap" is missing c++ examples examples_zc fast_bpf go lib libpcap-1.7.4 Makefile snort tcpdump-4.7.4 PF_RING module [root at slinky-3-4 ~]# cat /proc/net/pf_ring/info PF_RING Version : 6.5.0 (dev:9e221bc0b91040afee98f3e3c22ce83226f63f3e) Total rings : 0 Standard (non ZC) Options Ring slots : 32768 Slot version : 16 Capture TX : No [RX only] IP Defragment : No Socket Mode : Standard Total plugins : 0 Cluster Fragment Queue : 0 Cluster Fragment Discard : 0 Used NICs [rosinger at slinky-3-4 ~]$ lspci | egrep -i --color 'network|ethernet' 02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) 02:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) NICs Drivers [rosinger @slinky-3-4 ~]# ethtool -i eno2 driver: igb version: 5.2.15-k firmware-version: 1.61, 0x80000cd5, 1.1067.0 bus-info: 0000:02:00.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: no Enno Rosinger Student DualStudy Business Informatics - Application Management +49 617 22685124 Office Hewlett-Packard-Stra?e 1| 61352 Bad Homburg | Germany enno.rosinger at hpe.com [HPE_logoemail] -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160726/143899b1/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 5715 bytes Desc: image001.png Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160726/143899b1/attachment.bin From jazoff at illinois.edu Tue Jul 26 14:08:51 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 26 Jul 2016 21:08:51 +0000 Subject: [Bro] Issue: load balancer PF_RING drops 25% of incoming packets In-Reply-To: References: Message-ID: <58CE6483-1836-4070-996C-E46FBD34B978@illinois.edu> > On Jul 26, 2016, at 4:23 PM, Rosinger, Enno (DualStudy) wrote: > > Strangely only 16 Million of my 21 Million packet input pass through the PF_RING kernel module. Nevertheless they are then distributed correctly on the Bro processes. > How can I avoid this loss of 5 Million packets and how can I verify that PF_RING is configured correctly? What are you using to measure the difference in packet counts? Where is the 21 and 16 coming from? Can you add this to your local.bro and see what it logs to capture_loss.log after 30 minutes or so? @load misc/capture-loss > > I use Intel Corporation I350 Gigabit Network Connection as NICs. They work with the igb drivers. > The input rate is 0.5Gb/s = 60k to 80k packets/s and currently I am working without the ZeroCopy drivers > It is verified that all of my 21 Million packets are received by my NIC?s driver. > The PF_Ring module itself exists and BRO is running with load balancing. > > Looking forward to your response and hope to solve this problem with you. Below you will find more detailed information about my system. > If you need something else let me know. > > Best, > Enno > > Additional information: > > One interesting fact: I cannot run ?make? in ?PF_RING/userland/examples?, because > gcc: error: ../libpcap/libpcap.a: No such file or directory > > PF_RING/userland looks like this. Indeed ?libpcap? is missing > c++ examples examples_zc fast_bpf go lib libpcap-1.7.4 Makefile snort tcpdump-4.7.4 This should fix your build issue: cd PF_RING/userland ln -s libpcap-1.7.4 libpcap -- - Justin Azoff From enno.rosinger at hpe.com Tue Jul 26 17:27:43 2016 From: enno.rosinger at hpe.com (Rosinger, Enno (DualStudy)) Date: Wed, 27 Jul 2016 00:27:43 +0000 Subject: [Bro] FW: Issue: load balancer PF_RING drops 25% of incoming packets References: <58CE6483-1836-4070-996C-E46FBD34B978@illinois.edu> Message-ID: Hi Justin, Thank you for the fast reply. 21 Million received packets: Bro receives it's traffic on an isolated network (where the traffic is generated another server by TCPreplay). I manually take the stats of received packets of the NIC before and after a replaying by issuing "ifconfig eno2(interface-name)" . 16 Million handled packets: I use broctl and issue the command "netstats" to see the number of each worker process' received packets. If you make a sum out of that you will come to 16 Million (NOTE: now 18 Million, as I upgraded to Zero Copy drivers since the last mail). ###Ifconfig on Bro system### ###Before replaying### [root at slinky-3-4 kernel]# ifconfig eno2 [...] RX packets 25758824 bytes 20353552393 (18.9 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 182 bytes 36558 (35.7 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [...] ###After replaying### [root at slinky-3-4 kernel]# ifconfig eno2 [...] RX packets 47447181 bytes 37400251832 (34.8 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 268 bytes 54486 (53.2 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [...] That makes 47447181 - 25758824 = 21.688.357 received packets ###netstats in broctl on Bro system### ### after replaying ### [BroControl] > netstats worker-1-1: 1469577816.953862 recvd=5088052 dropped=0 link=5088052 worker-1-2: 1469577817.153796 recvd=4205599 dropped=0 link=4205599 worker-1-3: 1469577817.353889 recvd=4562288 dropped=0 link=4562288 worker-1-4: 1469577817.554795 recvd=4546975 dropped=0 link=4546975 The sum of this is 18.402.914 packets, which are seen by BRO as "on the link". Thanks to your help on the build issue I can also support this number with the stats of pfcount (NOTE: This is another run - slightly different numbers ) ##PFcount result Absolute Stats: [18'416'555 pkts total][0 pkts dropped][0.0% dropped] [18'416'555 pkts rcvd][17'225'248'719 bytes rcvd][58'886.73 pkt/sec][440.62 Mbit/sec] ========================= Actual Stats: [0 pkts rcvd][722.14 ms][0.00 pps][0.00 Gbps] As you requeted the capture_loss stats. I currently do not understand what the issue is with this. I hope you can help me track down the cause for this numbers ... ###first caputer loss file### #path capture_loss #open 2016-07-26-16-48-14 #fields ts ts_delta peer gaps acks percent_lost #types time interval string count count double 1469576894.926898 900.000078 worker-1-4 1156978 1683888 68.708726 1469576894.926602 900.000073 worker-1-1 1396713 1911004 73.087916 1469576894.977632 900.000080 worker-1-2 1055436 1544723 68.32526 1469576895.027647 900.000080 worker-1-3 1218489 1710519 71.235046 #close 2016-07-26-17-00-0 ###second caputer loss file### #open 2016-07-26-17-03-37 #fields ts ts_delta peer gaps acks percent_lost #types time interval string count count double 1469577794.926695 900.000093 worker-1-1 0 0 0.0 1469577794.977721 900.000089 worker-1-2 0 0 0.0 1469577795.027754 900.000107 worker-1-3 0 0 0.0 1469577794.927012 900.000114 worker-1-4 0 0 0.0 #close 2016-07-26-17-05-030 Looking forward to your response. It already helps me a lot to have more support on this issue. Best, Enno -----Original Message----- From: Azoff, Justin S [mailto:jazoff at illinois.edu] Sent: Dienstag, 26. Juli 2016 14:09 To: Rosinger, Enno (DualStudy) Cc: bro at bro.org Subject: Re: [Bro] Issue: load balancer PF_RING drops 25% of incoming packets > On Jul 26, 2016, at 4:23 PM, Rosinger, Enno (DualStudy) wrote: > > Strangely only 16 Million of my 21 Million packet input pass through the PF_RING kernel module. Nevertheless they are then distributed correctly on the Bro processes. > How can I avoid this loss of 5 Million packets and how can I verify that PF_RING is configured correctly? What are you using to measure the difference in packet counts? Where is the 21 and 16 coming from? Can you add this to your local.bro and see what it logs to capture_loss.log after 30 minutes or so? @load misc/capture-loss > > I use Intel Corporation I350 Gigabit Network Connection as NICs. They work with the igb drivers. > The input rate is 0.5Gb/s = 60k to 80k packets/s and currently I am > working without the ZeroCopy drivers It is verified that all of my 21 Million packets are received by my NIC?s driver. > The PF_Ring module itself exists and BRO is running with load balancing. > > Looking forward to your response and hope to solve this problem with you. Below you will find more detailed information about my system. > If you need something else let me know. > > Best, > Enno > > Additional information: > > One interesting fact: I cannot run ?make? in > ?PF_RING/userland/examples?, because > gcc: error: ../libpcap/libpcap.a: No such file or directory > > PF_RING/userland looks like this. Indeed ?libpcap? is missing > c++ examples examples_zc fast_bpf go lib libpcap-1.7.4 Makefile > c++ snort tcpdump-4.7.4 This should fix your build issue: cd PF_RING/userland ln -s libpcap-1.7.4 libpcap -- - Justin Azoff From johanna at icir.org Tue Jul 26 18:13:57 2016 From: johanna at icir.org (Johanna Amann) Date: Tue, 26 Jul 2016 18:13:57 -0700 Subject: [Bro] Newbie question Extract Binaries from traffic In-Reply-To: References: Message-ID: <20160727011353.GA31128@Beezling.local> Hi Scott, I think the syntax you are using there was retired with Bro 2.2 (or potentially earlier). Newer versions of Bro use the file analysis framework; Documentation for it is available at https://www.bro.org/sphinx-git/frameworks/file-analysis.html To see an example of someone using the framework, see e.g. the email thread at http://mailman.icsi.berkeley.edu/pipermail/bro/2015-July/008715.html I hope this helps, Johanna On Tue, Jul 26, 2016 at 10:08:57AM -0400, Scott P wrote: > Newbie question added the following to my local.bro file > > #Extract EXEs > redef HTTP::extract_file_types += /application\/x-dosexec/; > redef FTP::extract_file_types += /application\/x-dosexec/; > > #Extract files to /nsm/bro/extracted > redef HTTP::extraction_prefix = "/nsm/bro/extracted/http/http-item"; > redef FTP::extraction_prefix = "/nsm/bro/extracted/ftp/ftp-file"; > > But when I test against the file I am getting: > > sudo bro -r http-putty.pcap /opt/bro/share/bro/site/local.bro > > > error in /opt/bro/share/bro/site/local.bro, line 105: "redef" used but not > previously defined (HTTP::extract_file_types) > internal warning in /opt/bro/share/bro/site/local.bro, line 105: Can't > document redef of HTTP::extract_file_types, identifier lookup failed > error in /opt/bro/share/bro/site/local.bro, line 106: "redef" used but not > previously defined (FTP::extract_file_types) > internal warning in /opt/bro/share/bro/site/local.bro, line 106: Can't > document redef of FTP::extract_file_types, identifier lookup failed > error in /opt/bro/share/bro/site/local.bro, line 109: "redef" used but not > previously defined (HTTP::extraction_prefix) > internal warning in /opt/bro/share/bro/site/local.bro, line 109: Can't > document redef of HTTP::extraction_prefix, identifier lookup failed > error in /opt/bro/share/bro/site/local.bro, line 110: "redef" used but not > previously defined (FTP::extraction_prefix) > internal warning in /opt/bro/share/bro/site/local.bro, line 110: Can't > document redef of FTP::extraction_prefix, identifier lookup failed > > > Any insight would be helpful. > > -- > *Read, *pause, *think,* pause, *write*, pause, (perhaps erase), pause, *read, > *pause, (perhaps *go back*), pause, *write, ....* -- *Alan Turing (1936)* > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Tue Jul 26 18:15:27 2016 From: johanna at icir.org (Johanna Amann) Date: Tue, 26 Jul 2016 18:15:27 -0700 Subject: [Bro] ftp.log file isn't logging ftp related requests. In-Reply-To: <1116978397.3000825.1469169452604.JavaMail.yahoo@mail.yahoo.com> References: <1342536298.2807940.1469164239793.JavaMail.yahoo.ref@mail.yahoo.com> <1342536298.2807940.1469164239793.JavaMail.yahoo@mail.yahoo.com> <1116978397.3000825.1469169452604.JavaMail.yahoo@mail.yahoo.com> Message-ID: <20160727011527.GB31128@Beezling.local> Hi Aneela, would it potentially be possible to send me a trace of one of the sessions that Bro does not recognize correctly? Or, alternatively, can you create a ticket on tracker.bro.org and upload a trace there? Thank you, Johanna On Fri, Jul 22, 2016 at 06:37:32AM +0000, Aneela Safdar wrote: > Hi, > > > I have bro and proftpd server installed on linux. I tried to connect with this server through a intentionally brute force attack with random usernames and passwords thinking ftp.log will record these attempts but it didn't. Instead ftp.log I am getting these requests logged in weird.log file. > I did same with ssh service but i logged all requests and related information in ssh.log file.How come I can make ftp.log file to log all ftp related information which I genuinely think should be a default setting. Isn't so? > > > Regards, Aneela Safdar > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Tue Jul 26 18:16:41 2016 From: johanna at icir.org (Johanna Amann) Date: Tue, 26 Jul 2016 18:16:41 -0700 Subject: [Bro] Configuration of timing when to move log file from current folder In-Reply-To: <562083842.4600603.1469549907967.JavaMail.yahoo@mail.yahoo.com> References: <562083842.4600603.1469549907967.JavaMail.yahoo.ref@mail.yahoo.com> <562083842.4600603.1469549907967.JavaMail.yahoo@mail.yahoo.com> Message-ID: <20160727011641.GC31128@Beezling.local> Hello Aneela, You are probably searching for https://www.bro.org/sphinx-git/scripts/base/frameworks/logging/main.bro.html#id-Log::default_rotation_interval Log file rotation is also documented at https://www.bro.org/sphinx-git/frameworks/logging.html#rotation I hope this helps, Johanna On Tue, Jul 26, 2016 at 04:18:27PM +0000, Aneela Safdar wrote: > Hi, > I need to know where I can specified the time interval to move log files of bro from current folder to the respected day date named folder? Can I change it??Regards, Aneela Safdar > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Tue Jul 26 18:18:26 2016 From: johanna at icir.org (Johanna Amann) Date: Tue, 26 Jul 2016 18:18:26 -0700 Subject: [Bro] Inconsistent results In-Reply-To: <053b9c2d-f77d-9c49-d76a-4b3cabee0a67@cs.unm.edu> References: <053b9c2d-f77d-9c49-d76a-4b3cabee0a67@cs.unm.edu> Message-ID: <20160727011826.GD31128@Beezling.local> Hello Ben, no, there is no reason to expect Bro to give inconsistent output when running a script on a trace file. Do you potentially have a minimal example that you construct where this happens, either with a small trace, or with one of the traces in the Bro test suite? Without an example it is a bit hard to guess what exactly might be going wrong. Johanna On Fri, Jul 22, 2016 at 07:25:16PM -0700, Ben Mixon-Baca wrote: > Is there any reason to expect Bro to give inconsistent output when > running a script on trace files? I have a script that uses SumStats to > fire off an observation in the tcp_packet event to save the payload for > later analysis. I seem to be running into an issue where I count a > smaller number of observed tcp_packet events than I should actually be > seeing. > > > -- > Ben > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Tue Jul 26 19:20:44 2016 From: johanna at icir.org (Johanna Amann) Date: Tue, 26 Jul 2016 19:20:44 -0700 Subject: [Bro] ftp.log file isn't logging ftp related requests. In-Reply-To: <20160727011527.GB31128@Beezling.local> References: <1342536298.2807940.1469164239793.JavaMail.yahoo.ref@mail.yahoo.com> <1342536298.2807940.1469164239793.JavaMail.yahoo@mail.yahoo.com> <1116978397.3000825.1469169452604.JavaMail.yahoo@mail.yahoo.com> <20160727011527.GB31128@Beezling.local> Message-ID: <20160727022043.GA32244@Beezling.local> Hi Aneela, Justin just reminded you that if you are running Bro and your ftp server on the same box, you might potentially have to either disable some of the acceleration features of your NIC, or tell Bro to disable checksumming. You can try to either run bro with the -C command line flag, or set redef ignore_checksums = T;. in local.bro, if you are running broctl to see if that fixes the issue. Johanna On Tue, Jul 26, 2016 at 06:15:27PM -0700, Johanna Amann wrote: > Hi Aneela, > > would it potentially be possible to send me a trace of one of the sessions > that Bro does not recognize correctly? Or, alternatively, can you create a > ticket on tracker.bro.org and upload a trace there? > > Thank you, > Johanna > > On Fri, Jul 22, 2016 at 06:37:32AM +0000, Aneela Safdar wrote: > > Hi, > > > > > > I have bro and proftpd server installed on linux. I tried to connect with this server through a intentionally brute force attack with random usernames and passwords thinking ftp.log will record these attempts but it didn't. Instead ftp.log I am getting these requests logged in weird.log file. > > I did same with ssh service but i logged all requests and related information in ssh.log file.How come I can make ftp.log file to log all ftp related information which I genuinely think should be a default setting. Isn't so? > > > > > > Regards, Aneela Safdar > > > > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From hosom at battelle.org Wed Jul 27 05:22:29 2016 From: hosom at battelle.org (Hosom, Stephen M) Date: Wed, 27 Jul 2016 12:22:29 +0000 Subject: [Bro] Newbie question Extract Binaries from traffic In-Reply-To: <20160727011353.GA31128@Beezling.local> References: , <20160727011353.GA31128@Beezling.local> Message-ID: Scott, I have an example of how file extraction is usually done on modern Bro versions here: https://github.com/hosom/bro-file-extraction I'm assuming based on what it looks like you were trying to do that you want to extract PE files that appear in HTTP and FTP? You might try loading the extract-pe.bro script from the plugins directory in that repo. It won't limit the extraction to just HTTP and FTP though. You'd have to modify the script to get it to do that. -Stephen ________________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Johanna Amann [johanna at icir.org] Sent: Tuesday, July 26, 2016 9:13 PM To: Scott P Cc: bro at bro.org Subject: Re: [Bro] Newbie question Extract Binaries from traffic Hi Scott, I think the syntax you are using there was retired with Bro 2.2 (or potentially earlier). Newer versions of Bro use the file analysis framework; Documentation for it is available at https://www.bro.org/sphinx-git/frameworks/file-analysis.html To see an example of someone using the framework, see e.g. the email thread at http://mailman.icsi.berkeley.edu/pipermail/bro/2015-July/008715.html I hope this helps, Johanna On Tue, Jul 26, 2016 at 10:08:57AM -0400, Scott P wrote: > Newbie question added the following to my local.bro file > > #Extract EXEs > redef HTTP::extract_file_types += /application\/x-dosexec/; > redef FTP::extract_file_types += /application\/x-dosexec/; > > #Extract files to /nsm/bro/extracted > redef HTTP::extraction_prefix = "/nsm/bro/extracted/http/http-item"; > redef FTP::extraction_prefix = "/nsm/bro/extracted/ftp/ftp-file"; > > But when I test against the file I am getting: > > sudo bro -r http-putty.pcap /opt/bro/share/bro/site/local.bro > > > error in /opt/bro/share/bro/site/local.bro, line 105: "redef" used but not > previously defined (HTTP::extract_file_types) > internal warning in /opt/bro/share/bro/site/local.bro, line 105: Can't > document redef of HTTP::extract_file_types, identifier lookup failed > error in /opt/bro/share/bro/site/local.bro, line 106: "redef" used but not > previously defined (FTP::extract_file_types) > internal warning in /opt/bro/share/bro/site/local.bro, line 106: Can't > document redef of FTP::extract_file_types, identifier lookup failed > error in /opt/bro/share/bro/site/local.bro, line 109: "redef" used but not > previously defined (HTTP::extraction_prefix) > internal warning in /opt/bro/share/bro/site/local.bro, line 109: Can't > document redef of HTTP::extraction_prefix, identifier lookup failed > error in /opt/bro/share/bro/site/local.bro, line 110: "redef" used but not > previously defined (FTP::extraction_prefix) > internal warning in /opt/bro/share/bro/site/local.bro, line 110: Can't > document redef of FTP::extraction_prefix, identifier lookup failed > > > Any insight would be helpful. > > -- > *Read, *pause, *think,* pause, *write*, pause, (perhaps erase), pause, *read, > *pause, (perhaps *go back*), pause, *write, ....* -- *Alan Turing (1936)* > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From philosnef at yahoo.com Wed Jul 27 06:51:34 2016 From: philosnef at yahoo.com (philosnef) Date: Wed, 27 Jul 2016 13:51:34 +0000 (UTC) Subject: [Bro] Issue: load balancer PF_RING drops 25% of incoming packets References: <835178894.6408853.1469627494141.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <835178894.6408853.1469627494141.JavaMail.yahoo@mail.yahoo.com> Did you compile both tcpdump and libpcap on the box inside of pfring source directory and confirm ldd shows that those libraries are actually called in bro? If you built the kernel module, tcpdump, and libpcap, but ldd shows that pf_ring's libpcap isnt compiled into bro, you will see issues. Big ones. Also, did you run set_irq_affinity $interface? The only reason I can see Bro dropping packets on a pcap file would be if bro isn't properly compiled against the correct libpcap. In theory, that shouldn't happen either. But for sure on a live link, you will see this kind of drop rate in that situation. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160727/9cbefa3a/attachment.html From jazoff at illinois.edu Wed Jul 27 07:06:08 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 27 Jul 2016 14:06:08 +0000 Subject: [Bro] Issue: load balancer PF_RING drops 25% of incoming packets In-Reply-To: References: <58CE6483-1836-4070-996C-E46FBD34B978@illinois.edu> Message-ID: > On Jul 26, 2016, at 8:27 PM, Rosinger, Enno (DualStudy) wrote: > > Hi Justin, > > Thank you for the fast reply. > > 21 Million received packets: Bro receives it's traffic on an isolated network (where the traffic is generated another server by TCPreplay). I manually take the stats of received packets of the NIC before and after a replaying by issuing "ifconfig eno2(interface-name)" . > 16 Million handled packets: I use broctl and issue the command "netstats" to see the number of each worker process' received packets. If you make a sum out of that you will come to 16 Million (NOTE: now 18 Million, as I upgraded to Zero Copy drivers since the last mail). > > ###Ifconfig on Bro system### > ###Before replaying### > [root at slinky-3-4 kernel]# ifconfig eno2 > [...] > RX packets 25758824 bytes 20353552393 (18.9 GiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 182 bytes 36558 (35.7 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [...] > > ###After replaying### > [root at slinky-3-4 kernel]# ifconfig eno2 > [...] > RX packets 47447181 bytes 37400251832 (34.8 GiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 268 bytes 54486 (53.2 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [...] > > That makes 47447181 - 25758824 = 21.688.357 received packets > > ###netstats in broctl on Bro system### > ### after replaying ### > [BroControl] > netstats > worker-1-1: 1469577816.953862 recvd=5088052 dropped=0 link=5088052 > worker-1-2: 1469577817.153796 recvd=4205599 dropped=0 link=4205599 > worker-1-3: 1469577817.353889 recvd=4562288 dropped=0 link=4562288 > worker-1-4: 1469577817.554795 recvd=4546975 dropped=0 link=4546975 > > The sum of this is 18.402.914 packets, which are seen by BRO as "on the link". > > Thanks to your help on the build issue I can also support this number with the stats of pfcount (NOTE: This is another run - slightly different numbers ) ##PFcount result Absolute Stats: [18'416'555 pkts total][0 pkts dropped][0.0% dropped] > [18'416'555 pkts rcvd][17'225'248'719 bytes rcvd][58'886.73 pkt/sec][440.62 Mbit/sec] ========================= Actual Stats: [0 pkts rcvd][722.14 ms][0.00 pps][0.00 Gbps] Ah, do I understand that to mean that pfcount is also showing that only 18 million packets are received as well? If that is the case you should probably reach out to the pf_ring people and see if they have any ideas. If pfcount and bro both agree on the number of packets received, the problem is probably not within bro. The one thing I can think of is that you have not disabled offloading with something like for i in rx tx sg tso ufo gso gro lro; do ethtool -K en02 $i off; done So that while the system is receiving 21 million packets, they are being reassembled into only 18 million. -- - Justin Azoff From jdopheid at illinois.edu Wed Jul 27 09:21:49 2016 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Wed, 27 Jul 2016 16:21:49 +0000 Subject: [Bro] =?utf-8?q?BroCon_=E2=80=9916=3A_Early_bird_registration_exp?= =?utf-8?q?ires_Monday?= Message-ID: <5317690E-4898-4BE8-AD74-2A62A75943AE@illinois.edu> Bro Community, A friendly reminder that BroCon?s early bird registration expires on Monday August 1st. Save yourself $50 and register now! https://www.bro.org/community/brocon2016.html#registration See you in September, The Bro Team ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From enno.rosinger at hpe.com Wed Jul 27 11:00:10 2016 From: enno.rosinger at hpe.com (Rosinger, Enno (DualStudy)) Date: Wed, 27 Jul 2016 18:00:10 +0000 Subject: [Bro] Issue: load balancer PF_RING drops 25% of incoming packets In-Reply-To: References: <58CE6483-1836-4070-996C-E46FBD34B978@illinois.edu> Message-ID: Hey Justin, Thanks for these good advices. I'll test if it helps turn off the offloading, because it could indeed be, that the packets are reassembled. Do you think the PF_RINGs packet difference could also be caused by a bad configured irq-affinity? Philosnef suggested that in a separate mail. And I agree that BRO seems to be working fine, since the numbers match pretty well. I am looking forward to the pf-ring people's response and see what their opinion is. Thanks for the support again. I'll let you know if I could fix it. Best, Enno -----Original Message----- From: Azoff, Justin S [mailto:jazoff at illinois.edu] Sent: Mittwoch, 27. Juli 2016 07:06 To: Rosinger, Enno (DualStudy) Cc: bro at bro.org Subject: Re: [Bro] Issue: load balancer PF_RING drops 25% of incoming packets > On Jul 26, 2016, at 8:27 PM, Rosinger, Enno (DualStudy) wrote: > > Hi Justin, > > Thank you for the fast reply. > > 21 Million received packets: Bro receives it's traffic on an isolated network (where the traffic is generated another server by TCPreplay). I manually take the stats of received packets of the NIC before and after a replaying by issuing "ifconfig eno2(interface-name)" . > 16 Million handled packets: I use broctl and issue the command "netstats" to see the number of each worker process' received packets. If you make a sum out of that you will come to 16 Million (NOTE: now 18 Million, as I upgraded to Zero Copy drivers since the last mail). > > ###Ifconfig on Bro system### > ###Before replaying### > [root at slinky-3-4 kernel]# ifconfig eno2 [...] > RX packets 25758824 bytes 20353552393 (18.9 GiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 182 bytes 36558 (35.7 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > [...] > > ###After replaying### > [root at slinky-3-4 kernel]# ifconfig eno2 [...] > RX packets 47447181 bytes 37400251832 (34.8 GiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 268 bytes 54486 (53.2 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > [...] > > That makes 47447181 - 25758824 = 21.688.357 received packets > > ###netstats in broctl on Bro system### ### after replaying ### > [BroControl] > netstats > worker-1-1: 1469577816.953862 recvd=5088052 dropped=0 link=5088052 > worker-1-2: 1469577817.153796 recvd=4205599 dropped=0 link=4205599 > worker-1-3: 1469577817.353889 recvd=4562288 dropped=0 link=4562288 > worker-1-4: 1469577817.554795 recvd=4546975 dropped=0 link=4546975 > > The sum of this is 18.402.914 packets, which are seen by BRO as "on the link". > > Thanks to your help on the build issue I can also support this number > with the stats of pfcount (NOTE: This is another run - slightly > different numbers ) ##PFcount result Absolute Stats: [18'416'555 pkts > total][0 pkts dropped][0.0% dropped] > [18'416'555 pkts rcvd][17'225'248'719 bytes rcvd][58'886.73 > pkt/sec][440.62 Mbit/sec] ========================= Actual Stats: [0 > pkts rcvd][722.14 ms][0.00 pps][0.00 Gbps] Ah, do I understand that to mean that pfcount is also showing that only 18 million packets are received as well? If that is the case you should probably reach out to the pf_ring people and see if they have any ideas. If pfcount and bro both agree on the number of packets received, the problem is probably not within bro. The one thing I can think of is that you have not disabled offloading with something like for i in rx tx sg tso ufo gso gro lro; do ethtool -K en02 $i off; done So that while the system is receiving 21 million packets, they are being reassembled into only 18 million. -- - Justin Azoff From philosnef at yahoo.com Wed Jul 27 11:24:54 2016 From: philosnef at yahoo.com (philosnef) Date: Wed, 27 Jul 2016 18:24:54 +0000 (UTC) Subject: [Bro] problems with geo scripts in phirelight repository References: <1938041593.6681649.1469643894595.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <1938041593.6681649.1469643894595.JavaMail.yahoo@mail.yahoo.com> I am trying to get the geo scripts in the phirelight git repository to work. The readme says the scripts have to be explicitly enabled. I have: @load ./set-country.bro enabled in my geo/__load__.bro. However, my conn.log is not showing any country information. There are no errors and bro deploys cleanly. Why exactly is the country information not being inserted into the conn.log? I load geo/conn as well, which includes the 4 add-X-conn.bro scripts. This SHOULD put in the country, but is not doing so. What am I missing. The loadled_scripts.log says the add-X-conn.bro scripts are loaded. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160727/f6bcdbdc/attachment-0001.html From enno.rosinger at hpe.com Wed Jul 27 11:31:34 2016 From: enno.rosinger at hpe.com (Rosinger, Enno (DualStudy)) Date: Wed, 27 Jul 2016 18:31:34 +0000 Subject: [Bro] Issue: load balancer PF_RING drops 25% of incoming packets In-Reply-To: <835178894.6408853.1469627494141.JavaMail.yahoo@mail.yahoo.com> References: <835178894.6408853.1469627494141.JavaMail.yahoo.ref@mail.yahoo.com> <835178894.6408853.1469627494141.JavaMail.yahoo@mail.yahoo.com> Message-ID: Hi Philosnef, I installed PF_RING from source and compiled libpcap and tcpdump from that folder. Ldd supposes that the correct libpcap is used. Is there another way to verify this? [rosinger at slinky-3-4 examples]$ ldd /usr/local/bro/bin/bro | grep pcap libpcap.so.1 => /opt/pfring/lib/libpcap.so.1 (0x00007f7d78387000) As BRO is distributing the packets correctly and the number of packets match the number that PF_RING reports, I assume Bro is running correctly. PF_RING seems to be the bigger issue .. Thanks for the IRQ affinity hint. After some investigation it turns out, that all my cores (28 logical) handle the interrupts of my interface: [root at slinky-3-4 examples]# grep eno2 /proc/interrupts 43: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eno2 44: 49241082 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eno2-TxRx-0 [root at slinky-3-4 examples]# cat /proc/irq/43/smp_affinity fffffff [root at slinky-3-4 examples]# cat /proc/irq/44/smp_affinity fffffff Do you think that this is a good/correct configuration? Best, Enno From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of philosnef Sent: Mittwoch, 27. Juli 2016 06:52 To: bro at bro.org Subject: [Bro] Issue: load balancer PF_RING drops 25% of incoming packets Did you compile both tcpdump and libpcap on the box inside of pfring source directory and confirm ldd shows that those libraries are actually called in bro? If you built the kernel module, tcpdump, and libpcap, but ldd shows that pf_ring's libpcap isnt compiled into bro, you will see issues. Big ones. Also, did you run set_irq_affinity $interface? The only reason I can see Bro dropping packets on a pcap file would be if bro isn't properly compiled against the correct libpcap. In theory, that shouldn't happen either. But for sure on a live link, you will see this kind of drop rate in that situation. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160727/82491f6e/attachment.html From jazoff at illinois.edu Wed Jul 27 11:56:11 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 27 Jul 2016 18:56:11 +0000 Subject: [Bro] problems with geo scripts in phirelight repository In-Reply-To: <1938041593.6681649.1469643894595.JavaMail.yahoo@mail.yahoo.com> References: <1938041593.6681649.1469643894595.JavaMail.yahoo.ref@mail.yahoo.com> <1938041593.6681649.1469643894595.JavaMail.yahoo@mail.yahoo.com> Message-ID: <6F8B8216-9AE0-47A6-A1D7-435046C0EA75@illinois.edu> > On Jul 27, 2016, at 2:24 PM, philosnef wrote: > > I am trying to get the geo scripts in the phirelight git repository to work. The readme says the scripts have to be explicitly enabled. > > I have: > > @load ./set-country.bro > > enabled in my geo/__load__.bro. However, my conn.log is not showing any country information. There are no errors and bro deploys cleanly. Why exactly is the country information not being inserted into the conn.log? I load geo/conn as well, which includes the 4 add-X-conn.bro scripts. This SHOULD put in the country, but is not doing so. What am I missing. The loadled_scripts.log says the add-X-conn.bro scripts are loaded. These scripts https://github.com/phirelight/bro-scripts/tree/master/geo ? "Scripts to set geoip/asn info for conn. Note, these will not be logged as is. There are addional scripts to log the variables for each log type." You need to load the scripts under conn,dns,files, or ssl if you want the fields to be logged. By default they are just making the fields available. In your case what you are missing is loading geo/conn/add-country-conn.bro -- - Justin Azoff From philosnef at yahoo.com Wed Jul 27 12:02:04 2016 From: philosnef at yahoo.com (philosnef) Date: Wed, 27 Jul 2016 19:02:04 +0000 (UTC) Subject: [Bro] problems with geo scripts in phirelight repository In-Reply-To: <6F8B8216-9AE0-47A6-A1D7-435046C0EA75@illinois.edu> References: <1938041593.6681649.1469643894595.JavaMail.yahoo.ref@mail.yahoo.com> <1938041593.6681649.1469643894595.JavaMail.yahoo@mail.yahoo.com> <6F8B8216-9AE0-47A6-A1D7-435046C0EA75@illinois.edu> Message-ID: <199513467.6702059.1469646124170.JavaMail.yahoo@mail.yahoo.com> Sorry, meant to reply to the list....From my original message: The loadled_scripts.log says the add-X-conn.bro scripts are loaded. All the add scripts are already loaded, or loaded_scripts.log from the log directory would not say so? If the scripts weren't being loaded, loaded_scripts.log would not indicate they are.... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160727/c1ac53e6/attachment.html From jazoff at illinois.edu Wed Jul 27 12:05:35 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 27 Jul 2016 19:05:35 +0000 Subject: [Bro] problems with geo scripts in phirelight repository In-Reply-To: <1938041593.6681649.1469643894595.JavaMail.yahoo@mail.yahoo.com> References: <1938041593.6681649.1469643894595.JavaMail.yahoo.ref@mail.yahoo.com> <1938041593.6681649.1469643894595.JavaMail.yahoo@mail.yahoo.com> Message-ID: > > Begin forwarded message: > > From: philosnef > Subject: Re: [Bro] problems with geo scripts in phirelight repository > Date: July 27, 2016 at 3:00:31 PM EDT > To: "Azoff, Justin S" > Reply-To: philosnef > > From my original message: > > The loadled_scripts.log says the add-X-conn.bro scripts are loaded. > > All the add scripts are already loaded, or loaded_scripts.log from the log directory would not say so? > Oh, derp. I can read.. really.. :-) When you say "conn.log is not showing any country information" do you have the new columns? add-country-conn.bro adds an "orig_country" and "resp_country" column. Do you have those 2 columns but they are blank, or do you not even have those columns? If you don't have the columns at all that would be very odd. If you have them but they are all blank, that would point to an issue with the geoip bindings or databases. -- - Justin Azoff From philosnef at yahoo.com Wed Jul 27 12:08:51 2016 From: philosnef at yahoo.com (philosnef) Date: Wed, 27 Jul 2016 19:08:51 +0000 (UTC) Subject: [Bro] problems with geo scripts in phirelight repository In-Reply-To: References: <1938041593.6681649.1469643894595.JavaMail.yahoo.ref@mail.yahoo.com> <1938041593.6681649.1469643894595.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1620680645.6617182.1469646531822.JavaMail.yahoo@mail.yahoo.com> Yeah, no new columns at all. I am logging in json format, but they should still show up, right? On Wednesday, July 27, 2016 3:05 PM, "Azoff, Justin S" wrote: > > Begin forwarded message: > > From: philosnef > Subject: Re: [Bro] problems with geo scripts in phirelight repository > Date: July 27, 2016 at 3:00:31 PM EDT > To: "Azoff, Justin S" > Reply-To: philosnef > > From my original message: > > The loadled_scripts.log says the add-X-conn.bro scripts are loaded. > > All the add scripts are already loaded, or loaded_scripts.log from the log directory would not say so? > Oh, derp. I can read.. really.. :-) When you say "conn.log is not showing any country information" do you have the new columns? add-country-conn.bro adds an "orig_country" and "resp_country" column. Do you have those 2 columns but they are blank, or do you not even have those columns? If you don't have the columns at all that would be very odd.? If you have them but they are all blank, that would point to an issue with the geoip bindings or? databases. -- - Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160727/03b4611c/attachment.html From jazoff at illinois.edu Wed Jul 27 12:15:22 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 27 Jul 2016 19:15:22 +0000 Subject: [Bro] problems with geo scripts in phirelight repository In-Reply-To: <1620680645.6617182.1469646531822.JavaMail.yahoo@mail.yahoo.com> References: <1938041593.6681649.1469643894595.JavaMail.yahoo.ref@mail.yahoo.com> <1938041593.6681649.1469643894595.JavaMail.yahoo@mail.yahoo.com> <1620680645.6617182.1469646531822.JavaMail.yahoo@mail.yahoo.com> Message-ID: <34A2B298-D27B-49A3-AF7A-220E94122562@illinois.edu> > On Jul 27, 2016, at 3:08 PM, philosnef wrote: > > Yeah, no new columns at all. I am logging in json format, but they should still show up, right? Ah, that complicates things because optional fields are not logged in json format. unlike the TSV logs it doesn't need a fixed column layout, so fields can only show up when needed.. geoip is probably broken for you in general. I'd try this experiment. Some errors are ok since it tries some fallbacks, but you should get the result in the end: $ cat test.bro print lookup_location(8.8.8.8); $ bro test.bro Failed to open GeoIP Cityv6 database: /usr/local/var/GeoIP/GeoIPCityv6.dat Failed to open GeoIPv6 Country database: /usr/local/var/GeoIP/GeoIPv6.dat error in ./test.bro, line 1: Can't open GeoIPv6 City/Country database (lookup_location(8.8.8.8)) [country_code=US, region=CA, city=Mountain View, latitude=37.386002, longitude=-122.083801] -- - Justin Azoff From philosnef at yahoo.com Wed Jul 27 12:39:02 2016 From: philosnef at yahoo.com (philosnef) Date: Wed, 27 Jul 2016 19:39:02 +0000 (UTC) Subject: [Bro] problems with geo scripts in phirelight repository In-Reply-To: <34A2B298-D27B-49A3-AF7A-220E94122562@illinois.edu> References: <1938041593.6681649.1469643894595.JavaMail.yahoo.ref@mail.yahoo.com> <1938041593.6681649.1469643894595.JavaMail.yahoo@mail.yahoo.com> <1620680645.6617182.1469646531822.JavaMail.yahoo@mail.yahoo.com> <34A2B298-D27B-49A3-AF7A-220E94122562@illinois.edu> Message-ID: <976767223.6675243.1469648342721.JavaMail.yahoo@mail.yahoo.com> Justin, I get a bit where it says init-bare.bro problem initializing NB-DNS, but other than that it properly reports the exact same information you have there (US, CA, Mountain View). For some reason, it seems to work fine now. We just pushed a new version of geo from the phirelight repository and that seems to have fixed it. On Wednesday, July 27, 2016 3:15 PM, "Azoff, Justin S" wrote: > On Jul 27, 2016, at 3:08 PM, philosnef wrote: > > Yeah, no new columns at all. I am logging in json format, but they should still show up, right? Ah, that complicates things because optional fields are not logged in json format.? unlike the TSV logs it doesn't need a fixed column layout, so fields can only show up when needed.. geoip is probably broken for you in general. I'd try this experiment.? Some errors are ok since it tries some fallbacks, but you should get the result in the end: $ cat test.bro print lookup_location(8.8.8.8); $ bro test.bro Failed to open GeoIP Cityv6 database: /usr/local/var/GeoIP/GeoIPCityv6.dat Failed to open GeoIPv6 Country database: /usr/local/var/GeoIP/GeoIPv6.dat error in ./test.bro, line 1: Can't open GeoIPv6 City/Country database (lookup_location(8.8.8.8)) [country_code=US, region=CA, city=Mountain View, latitude=37.386002, longitude=-122.083801] -- - Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160727/c390dc57/attachment-0001.html From lagoon7 at gmail.com Thu Jul 28 07:53:45 2016 From: lagoon7 at gmail.com (Ludwig Goon) Date: Thu, 28 Jul 2016 10:53:45 -0400 Subject: [Bro] Revisiting CEF formatted BRO Logs Message-ID: Can someone from the community provide more information or examples of using log writer to create CEF formatted logs for consumption with Arcsight SIEMs? it seems that we can not customize arcsight connectors for BRO logs however since arcsight can accept CEF events directly I would like to experiment with directly sending CEF formatted BRO events from the standard log set. Additionally I have 5 BRO sensors and would like to tag each event with the BRO sensor's hostname before sending it to arc sight. The default logs do not allow that modification and documentation is not the greatest. If you want to do this in Arcsight via the connector, which is a version or two behind, the connector will not allow the adding of the hostname. So I have attempted to write PERL and PYTHON converters but the performance of tailing logs and sending all events is challenging. Also using brocut requires scripting and again not sure if I am sending ALL log events. In previous questions to the forum the answer was using the logging framework however I have not seen anymore content on this subject. Thus here is my formal request: Can someone show how to use the logging framework to convert or have bro output the http.log into CEF format? Also can I add custom fields such as sensor-name and the end of the event or at the beginning near CEF:0. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160728/d80f6a39/attachment.html From jason.carr at gmail.com Thu Jul 28 08:36:49 2016 From: jason.carr at gmail.com (Jason Carr) Date: Thu, 28 Jul 2016 15:36:49 +0000 Subject: [Bro] Revisiting CEF formatted BRO Logs In-Reply-To: References: Message-ID: It might be a little work but utilizing the JSON output ( https://www.bro.org/sphinx/scripts/policy/tuning/json-logs.bro.html) and sending it to logstash and in turn using mutate/filter in the logstash config may get you to where you want to be at. On Thu, Jul 28, 2016 at 10:56 AM Ludwig Goon wrote: > Can someone from the community provide more information or examples of > using log writer to create CEF formatted logs for consumption with Arcsight > SIEMs? > > it seems that we can not customize arcsight connectors for BRO logs > however since arcsight can accept CEF events directly I would like to > experiment with directly sending CEF formatted BRO events from the standard > log set. > > Additionally I have 5 BRO sensors and would like to tag each event with > the BRO sensor's hostname before sending it to arc sight. The default logs > do not allow that modification and documentation is not the greatest. If > you want to do this in Arcsight via the connector, which is a version or > two behind, the connector will not allow the adding of the hostname. > > So I have attempted to write PERL and PYTHON converters but the > performance of tailing logs and sending all events is challenging. > > Also using brocut requires scripting and again not sure if I am sending > ALL log events. > > > In previous questions to the forum the answer was using the logging > framework however I have not seen anymore content on this subject. Thus > here is my formal request: > > Can someone show how to use the logging framework to convert or have bro > output the http.log into CEF format? Also can I add custom fields such as > sensor-name and the end of the event or at the beginning near CEF:0. > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160728/452710d0/attachment.html From enno.rosinger at hpe.com Thu Jul 28 16:21:45 2016 From: enno.rosinger at hpe.com (Rosinger, Enno (DualStudy)) Date: Thu, 28 Jul 2016 23:21:45 +0000 Subject: [Bro] Issue: load balancer PF_RING drops 25% of incoming packets References: <58CE6483-1836-4070-996C-E46FBD34B978@illinois.edu> Message-ID: Hey Everyone, With your hint to disable the offloading features, I can now see all my packets as required and Bro is distributing them evenly. I consider this Issue fixed for now. Thanks for all the help and advices - I appreciate it. Best, Enno -----Original Message----- From: Rosinger, Enno (DualStudy) Sent: Mittwoch, 27. Juli 2016 11:00 To: 'Azoff, Justin S' Cc: 'bro at bro.org' Subject: RE: [Bro] Issue: load balancer PF_RING drops 25% of incoming packets Hey Justin, Thanks for these good advices. I'll test if it helps turn off the offloading, because it could indeed be, that the packets are reassembled. Do you think the PF_RINGs packet difference could also be caused by a bad configured irq-affinity? Philosnef suggested that in a separate mail. And I agree that BRO seems to be working fine, since the numbers match pretty well. I am looking forward to the pf-ring people's response and see what their opinion is. Thanks for the support again. I'll let you know if I could fix it. Best, Enno -----Original Message----- From: Azoff, Justin S [mailto:jazoff at illinois.edu] Sent: Mittwoch, 27. Juli 2016 07:06 To: Rosinger, Enno (DualStudy) Cc: bro at bro.org Subject: Re: [Bro] Issue: load balancer PF_RING drops 25% of incoming packets > On Jul 26, 2016, at 8:27 PM, Rosinger, Enno (DualStudy) wrote: > > Hi Justin, > > Thank you for the fast reply. > > 21 Million received packets: Bro receives it's traffic on an isolated network (where the traffic is generated another server by TCPreplay). I manually take the stats of received packets of the NIC before and after a replaying by issuing "ifconfig eno2(interface-name)" . > 16 Million handled packets: I use broctl and issue the command "netstats" to see the number of each worker process' received packets. If you make a sum out of that you will come to 16 Million (NOTE: now 18 Million, as I upgraded to Zero Copy drivers since the last mail). > > ###Ifconfig on Bro system### > ###Before replaying### > [root at slinky-3-4 kernel]# ifconfig eno2 [...] > RX packets 25758824 bytes 20353552393 (18.9 GiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 182 bytes 36558 (35.7 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > [...] > > ###After replaying### > [root at slinky-3-4 kernel]# ifconfig eno2 [...] > RX packets 47447181 bytes 37400251832 (34.8 GiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 268 bytes 54486 (53.2 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > [...] > > That makes 47447181 - 25758824 = 21.688.357 received packets > > ###netstats in broctl on Bro system### ### after replaying ### > [BroControl] > netstats > worker-1-1: 1469577816.953862 recvd=5088052 dropped=0 link=5088052 > worker-1-2: 1469577817.153796 recvd=4205599 dropped=0 link=4205599 > worker-1-3: 1469577817.353889 recvd=4562288 dropped=0 link=4562288 > worker-1-4: 1469577817.554795 recvd=4546975 dropped=0 link=4546975 > > The sum of this is 18.402.914 packets, which are seen by BRO as "on the link". > > Thanks to your help on the build issue I can also support this number > with the stats of pfcount (NOTE: This is another run - slightly > different numbers ) ##PFcount result Absolute Stats: [18'416'555 pkts > total][0 pkts dropped][0.0% dropped] > [18'416'555 pkts rcvd][17'225'248'719 bytes rcvd][58'886.73 > pkt/sec][440.62 Mbit/sec] ========================= Actual Stats: [0 > pkts rcvd][722.14 ms][0.00 pps][0.00 Gbps] Ah, do I understand that to mean that pfcount is also showing that only 18 million packets are received as well? If that is the case you should probably reach out to the pf_ring people and see if they have any ideas. If pfcount and bro both agree on the number of packets received, the problem is probably not within bro. The one thing I can think of is that you have not disabled offloading with something like for i in rx tx sg tso ufo gso gro lro; do ethtool -K en02 $i off; done So that while the system is receiving 21 million packets, they are being reassembled into only 18 million. -- - Justin Azoff From ansaf_130 at yahoo.com Fri Jul 29 05:20:59 2016 From: ansaf_130 at yahoo.com (Aneela Safdar) Date: Fri, 29 Jul 2016 12:20:59 +0000 (UTC) Subject: [Bro] What will happen to Log files rotation if they are monitored/in used by another background program References: <1438472040.6395439.1469794860001.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <1438472040.6395439.1469794860001.JavaMail.yahoo@mail.yahoo.com> Hi, I am working on? a project to monitor bro log files for instance ssh.log to send any new entry that has been appended to it, to a server by a background java process. As rotation period moves file from current directory to a another and create new file when there are records to be logged, I am curious what could happen if background monitoring program wont let those files to move somewhere as they are obviously in use by it. I ran my program on a test directory with some test records and when I tried to move that file to simulate the real scenario, it prompted me that I cant move or delete this file as its already open in java process. ?Regards, Aneela Safdar -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160729/9ef9ed70/attachment.html From jazoff at illinois.edu Fri Jul 29 06:04:40 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 29 Jul 2016 13:04:40 +0000 Subject: [Bro] What will happen to Log files rotation if they are monitored/in used by another background program In-Reply-To: <1438472040.6395439.1469794860001.JavaMail.yahoo@mail.yahoo.com> References: <1438472040.6395439.1469794860001.JavaMail.yahoo.ref@mail.yahoo.com> <1438472040.6395439.1469794860001.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1D30FBF9-B1B5-4F5B-93C6-9E98A54F935F@illinois.edu> > On Jul 29, 2016, at 8:20 AM, Aneela Safdar wrote: > > I ran my program on a test directory with some test records and when I tried to move that file to simulate the real scenario, it prompted me that I cant move or delete this file as its already open in java process. Was this by any chance on a windows system? There is no mechanism on a unix system that would generate such an error message. -- - Justin Azoff From al.brocino1 at gmail.com Fri Jul 29 06:42:32 2016 From: al.brocino1 at gmail.com (al brocino) Date: Fri, 29 Jul 2016 08:42:32 -0500 Subject: [Bro] File Extraction Message-ID: Hello Bro community, I'm new to Bro and using version 2.3.2 and want to extract all the exe's seen on the network. In bro-file-extract we are using the file-extract.bro script to try to parse for the exe's (partial of script): global ext_map:table[string] of string = { ["application/x/dosexec"] = "exe", redef FileExtract::prefix="/var/log/netlogs/bro/file-extracts.bro"; redef FileExtract::default_limit = 314572800; redef FileExtract::prefix = "/var/log/netlogs/bro/file-extracts/"; We also have the file-extract-http-local.bro set to extract on our network: global http_extract_file_ignore: set [subnet] = { 10.0.0.0/8, }; We think the problem is that _load_.bro has the file extract commented out under bro-icmp: #@load ./file-extract-http-local.bro #@load ./file-extract-types.bro @load ./bro-file-extract When I tried to enable these Bro failed the scripts check with errors like: internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7:can't find base/protocols/http/file-ident I continued to receive these errors and had to back out of removing the comments Under bro-file-extract _load_.bro looks correct: @load ./file-extract What I'm getting in /var/log/netlogs/bro/file-extracts are entries like: HTTP-F7K52nSzN3h7GNM31.exe These files occur occasionally I'm not sure what they are. Thanks for your help, Al -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160729/fe89e2a0/attachment.html From fatema.bannatwala at gmail.com Fri Jul 29 11:17:37 2016 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Fri, 29 Jul 2016 14:17:37 -0400 Subject: [Bro] Determining remote proxy servers using Bro. Message-ID: Hi, Recently we have seen an uptick in use of proxy servers to login to the accounts from people living in China. And since the connection appears to come from US based IP address (probably a proxy) they go un-flagged by the IDS/IPS devices, as they see normal logins from United States IP addresses. So my question is, is there a way to determine that the incoming connection from an IP is actually a proxy server's IP, by looking at some unique patterns in data collected by IDS/IPS devices? and if so can we do it using Bro? Thanks, Fatema. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160729/8e979f7e/attachment-0001.html From wulfyh at gmail.com Fri Jul 29 13:08:34 2016 From: wulfyh at gmail.com (Wulfy H) Date: Fri, 29 Jul 2016 16:08:34 -0400 Subject: [Bro] BroTop Web GUI Message-ID: Hi, Why is the BroTop repo gone from github? I was hoping to try it out! I checked criticalstack's github repo directory, no BroTop there. Thanks, W. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160729/f1af3204/attachment.html From slagell at illinois.edu Fri Jul 29 13:47:34 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Fri, 29 Jul 2016 20:47:34 +0000 Subject: [Bro] BroTop Web GUI In-Reply-To: References: Message-ID: <0994D769-3C15-4F8B-B2A8-380BB85E6554@illinois.edu> That is not something official from the Bro team. You should ask Critical Stack or Liam Randall. On Jul 29, 2016, at 3:08 PM, Wulfy H > wrote: Hi, Why is the BroTop repo gone from github? I was hoping to try it out! I checked criticalstack's github repo directory, no BroTop there. Thanks, W. _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro ------ Adam J. Slagell Chief Information Security Officer Director, Cybersecurity Division National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160729/d276d983/attachment.html From liam.randall at gmail.com Fri Jul 29 14:07:54 2016 From: liam.randall at gmail.com (Liam Randall) Date: Fri, 29 Jul 2016 17:07:54 -0400 Subject: [Bro] BroTop Web GUI In-Reply-To: <0994D769-3C15-4F8B-B2A8-380BB85E6554@illinois.edu> References: <0994D769-3C15-4F8B-B2A8-380BB85E6554@illinois.edu> Message-ID: It got moved with the aquisition. Let me look into it. Liam On Jul 29, 2016 4:57 PM, "Slagell, Adam J" wrote: > That is not something official from the Bro team. You should ask Critical > Stack or Liam Randall. > > On Jul 29, 2016, at 3:08 PM, Wulfy H wrote: > > Hi, > > Why is the BroTop repo gone from github? I was hoping to try it out! I > checked criticalstack's github repo directory, no BroTop there. > > Thanks, > > W. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > ------ > > Adam J. Slagell > Chief Information Security Officer > Director, Cybersecurity Division > National Center for Supercomputing Applications > University of Illinois at Urbana-Champaign > www.slagell.info > > "Under the Illinois Freedom of Information Act (FOIA), any written > communication to or from University employees regarding University business > is a public record and may be subject to public disclosure." > > > > > > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160729/56435a30/attachment.html From wsladekjr at hotmail.com Sat Jul 30 08:15:10 2016 From: wsladekjr at hotmail.com (Ward Sladek) Date: Sat, 30 Jul 2016 15:15:10 +0000 Subject: [Bro] Revisiting CEF formatted BRO Logs In-Reply-To: References: Message-ID: If I were in your shoes and assuming it's possible to add the sensor ID/name to the bro logs, I would just add that one field (keeping the same format, etc) and not rewrite everything for CEF. Then I would press HP support to give me the encrypted bro parser (they have given me several parsers in the past) and write a parser override to account for the new sensor/worker field. Sorry this doesn't answer your question directly, but maybe this route is an option for you. ________________________________ From: bro-bounces at bro.org on behalf of Ludwig Goon Sent: Thursday, July 28, 2016 9:53 AM To: bro at bro.org Subject: [Bro] Revisiting CEF formatted BRO Logs Can someone from the community provide more information or examples of using log writer to create CEF formatted logs for consumption with Arcsight SIEMs? it seems that we can not customize arcsight connectors for BRO logs however since arcsight can accept CEF events directly I would like to experiment with directly sending CEF formatted BRO events from the standard log set. Additionally I have 5 BRO sensors and would like to tag each event with the BRO sensor's hostname before sending it to arc sight. The default logs do not allow that modification and documentation is not the greatest. If you want to do this in Arcsight via the connector, which is a version or two behind, the connector will not allow the adding of the hostname. So I have attempted to write PERL and PYTHON converters but the performance of tailing logs and sending all events is challenging. Also using brocut requires scripting and again not sure if I am sending ALL log events. In previous questions to the forum the answer was using the logging framework however I have not seen anymore content on this subject. Thus here is my formal request: Can someone show how to use the logging framework to convert or have bro output the http.log into CEF format? Also can I add custom fields such as sensor-name and the end of the event or at the beginning near CEF:0. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160730/663b67a9/attachment.html From neslog at gmail.com Sat Jul 30 08:29:09 2016 From: neslog at gmail.com (Neslog) Date: Sat, 30 Jul 2016 11:29:09 -0400 Subject: [Bro] Revisiting CEF formatted BRO Logs In-Reply-To: References: Message-ID: Back in the days of the old bro 1.9 log formats I generated CEF logging. Not the logging framework but it worked perfectly. Was just a bro script output to a file. It's another option. On Jul 30, 2016 11:25, "Ward Sladek" wrote: > If I were in your shoes and assuming it's possible to add the sensor > ID/name to the bro logs, I would just add that one field (keeping the same > format, etc) and not rewrite everything for CEF. > > Then I would press HP support to give me the encrypted bro parser (they > have given me several parsers in the past) and write a parser override to > account for the new sensor/worker field. > > Sorry this doesn't answer your question directly, but maybe this route is > an option for you. > > > > ------------------------------ > *From:* bro-bounces at bro.org on behalf of Ludwig > Goon > *Sent:* Thursday, July 28, 2016 9:53 AM > *To:* bro at bro.org > *Subject:* [Bro] Revisiting CEF formatted BRO Logs > > Can someone from the community provide more information or examples of > using log writer to create CEF formatted logs for consumption with Arcsight > SIEMs? > > it seems that we can not customize arcsight connectors for BRO logs > however since arcsight can accept CEF events directly I would like to > experiment with directly sending CEF formatted BRO events from the standard > log set. > > Additionally I have 5 BRO sensors and would like to tag each event with > the BRO sensor's hostname before sending it to arc sight. The default logs > do not allow that modification and documentation is not the greatest. If > you want to do this in Arcsight via the connector, which is a version or > two behind, the connector will not allow the adding of the hostname. > > So I have attempted to write PERL and PYTHON converters but the > performance of tailing logs and sending all events is challenging. > > Also using brocut requires scripting and again not sure if I am sending > ALL log events. > > > In previous questions to the forum the answer was using the logging > framework however I have not seen anymore content on this subject. Thus > here is my formal request: > > Can someone show how to use the logging framework to convert or have bro > output the http.log into CEF format? Also can I add custom fields such as > sensor-name and the end of the event or at the beginning near CEF:0. > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160730/0b9556ec/attachment-0001.html