From seth at corelight.com Fri Sep 1 06:05:42 2017 From: seth at corelight.com (Seth Hall) Date: Fri, 01 Sep 2017 09:05:42 -0400 Subject: [Bro] http response timeout In-Reply-To: References: Message-ID: On 31 Aug 2017, at 19:57, Dk Jack wrote: > In my http.log, I am seeing some lines being written without response > code > etc. What could be the reason for this? One reason I could think of > was, > what if the server or some entity between bro and the server that > dropped > the request/response thus preventing the response from reaching bro or > the > connection is closed on receiving the request by a downstream security > device. How does bro react in such cases? could one of these scenarios > explain why the response fields are missing from the log? You seem to have a pretty good handle on what could be causing the problem. One additional thing you didn't list is if you have load balancing happening incorrectly. That could cause the same problem because the request could have gone to a different process than the reply. What would help most at this point is if you could send a conn.log entry for a connection where you saw the http.log missing the response code (feel free to redact IP addresses, they don't matter). .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From reswob10 at gmail.com Fri Sep 1 06:34:07 2017 From: reswob10 at gmail.com (craig bowser) Date: Fri, 1 Sep 2017 09:34:07 -0400 Subject: [Bro] Change location of log files? Message-ID: I've been looking thru the docs, but I don't see ( and perhaps I don't understand) if there is an option to change the location where bro writes all the log files. The default is /usr/local/bro/logs and I would like them to be written to a partition I created called /data /usr/local/bro/logs/current can stay where it is, but I'd like everything else to be moved. Thanks Craig -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170901/3410db24/attachment.html From seth at corelight.com Fri Sep 1 07:00:20 2017 From: seth at corelight.com (Seth Hall) Date: Fri, 01 Sep 2017 10:00:20 -0400 Subject: [Bro] Change location of log files? In-Reply-To: References: Message-ID: <5399B427-27E1-4958-A1A0-6AD97B1E3DE9@corelight.com> On 1 Sep 2017, at 9:34, craig bowser wrote: > I've been looking thru the docs, but I don't see ( and perhaps I don't > understand) if there is an option to change the location where bro writes > all the log files. > > The default is /usr/local/bro/logs and I would like them to be written to a > partition I created called /data In broctl.cfg... logdir = /usr/local/bro/logs .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From maerzsa at ornl.gov Fri Sep 1 07:05:43 2017 From: maerzsa at ornl.gov (Maerz, Stefan A.) Date: Fri, 1 Sep 2017 14:05:43 +0000 Subject: [Bro] Change location of log files? In-Reply-To: References: Message-ID: <6DCB4E5C-3A66-4D09-ACC9-782888E36A90@ornl.gov> Broctl.conf file has a logging location. Scroll down to find it. Default place for it is /usr/local/bro/etc/broctl.conf You can specify where both the current and rotated data is stored separately. This is what I have, the defaults are commented out: # Location of the log directory where log files will be archived each rotation # interval. ##LogDir = /usr/local/bro/logs LogDir = /data/log # Location of the spool directory where files and data that are currently being # written are stored. ##SpoolDir = /usr/local/bro/spool SpoolDir = /data/spool Best Regards, -Stefan -- Stefan Maerz HPC Cyber Security Engineer Oak Ridge National Laboratory National Center for Computational Sciences Oak Ridge Leadership Computing Facility maerzsa at ornl.gov linkedin.com/in/stefanmaerz > On Sep 1, 2017, at 9:34 AM, craig bowser wrote: > > > > I've been looking thru the docs, but I don't see ( and perhaps I don't understand) if there is an option to change the location where bro writes all the log files. > > The default is /usr/local/bro/logs and I would like them to be written to a partition I created called /data > > /usr/local/bro/logs/current can stay where it is, but I'd like everything else to be moved. > > Thanks > > Craig > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170901/96a8fd25/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170901/96a8fd25/attachment.bin From reswob10 at gmail.com Fri Sep 1 08:13:47 2017 From: reswob10 at gmail.com (craig bowser) Date: Fri, 1 Sep 2017 11:13:47 -0400 Subject: [Bro] Change location of log files? In-Reply-To: <6DCB4E5C-3A66-4D09-ACC9-782888E36A90@ornl.gov> References: <6DCB4E5C-3A66-4D09-ACC9-782888E36A90@ornl.gov> Message-ID: Thanks! On 1 Sep 2017 10:05 am, "Maerz, Stefan A." wrote: > Broctl.conf file has a logging location. Scroll down to find it. Default > place for it is /usr/local/bro/etc/broctl.conf > > You can specify where both the current and rotated data is stored > separately. This is what I have, the defaults are commented out: > > # Location of the log directory where log files will be archived each > rotation > # interval. > ##LogDir = /usr/local/bro/logs > LogDir = /data/log > > # Location of the spool directory where files and data that are currently > being > # written are stored. > ##SpoolDir = /usr/local/bro/spool > SpoolDir = /data/spool > > > Best Regards, > -Stefan > > > -- > Stefan Maerz > HPC Cyber Security Engineer > Oak Ridge National Laboratory > National Center for Computational Sciences > Oak Ridge Leadership Computing Facility > maerzsa at ornl.gov > linkedin.com/in/stefanmaerz > > On Sep 1, 2017, at 9:34 AM, craig bowser wrote: > > > > I've been looking thru the docs, but I don't see ( and perhaps I don't > understand) if there is an option to change the location where bro writes > all the log files. > > The default is /usr/local/bro/logs and I would like them to be written to > a partition I created called /data > > /usr/local/bro/logs/current can stay where it is, but I'd like everything > else to be moved. > > Thanks > > Craig > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170901/3c52f89b/attachment-0001.html From dnj0496 at gmail.com Fri Sep 1 09:44:03 2017 From: dnj0496 at gmail.com (Dk Jack) Date: Fri, 1 Sep 2017 09:44:03 -0700 Subject: [Bro] http response timeout In-Reply-To: References: Message-ID: <2B4D1F17-D9E8-42A4-9427-1CE9217C14CF@gmail.com> Thanks Seth, Unfortunately don't have the conn.log. Will continue to investigate. Thanks > On Sep 1, 2017, at 6:05 AM, Seth Hall wrote: > > > >> On 31 Aug 2017, at 19:57, Dk Jack wrote: >> >> In my http.log, I am seeing some lines being written without response code >> etc. What could be the reason for this? One reason I could think of was, >> what if the server or some entity between bro and the server that dropped >> the request/response thus preventing the response from reaching bro or the >> connection is closed on receiving the request by a downstream security >> device. How does bro react in such cases? could one of these scenarios >> explain why the response fields are missing from the log? > > You seem to have a pretty good handle on what could be causing the problem. One additional thing you didn't list is if you have load balancing happening incorrectly. That could cause the same problem because the request could have gone to a different process than the reply. > > What would help most at this point is if you could send a conn.log entry for a connection where you saw the http.log missing the response code (feel free to redact IP addresses, they don't matter). > > .Seth > > -- > Seth Hall * Corelight, Inc * www.corelight.com From jdopheid at illinois.edu Tue Sep 5 06:28:32 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Tue, 5 Sep 2017 13:28:32 +0000 Subject: [Bro] BroCon is next week Message-ID: <8175176D-6E83-42E7-9108-A8F0F02B5D7B@illinois.edu> Reminder, BroCon is next week. If you haven?t registered, tickets are still available: https://www.bro.org/community/brocon2017.html Looking forward to seeing you all there. The Bro Team ------ Jeannette Dopheide Sr. Education, Outreach, and Training Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From brianallen at wustl.edu Tue Sep 5 11:54:24 2017 From: brianallen at wustl.edu (Allen, Brian) Date: Tue, 5 Sep 2017 18:54:24 +0000 Subject: [Bro] caret and the stick Message-ID: I upgraded one of my clusters recently and I noticed that the history column in the conn.log has a caret symbol now. The docs say: ^ = "connection direction was flipped by Bro?s heuristic? I was wondering what exactly this means. Which part exactly was flipped? And which heuristic is it referring to? Here is a line from our conn.log showing what I think is backscatter. (Our network is 128.252.0.0/16.) 128.252.X.Y 57756 111.29.2.3 80 tcp - - - - OTH T F 0 ^h 0 0 1 44 So in this example, what was flipped exactly? Thanks, -Brian From: > on behalf of "Dopheide, Jeannette M" > Date: Tuesday, September 5, 2017 at 8:28 AM To: Bro-Mailinglist > Subject: [Bro] BroCon is next week Reminder, BroCon is next week. If you haven?t registered, tickets are still available: https://www.bro.org/community/brocon2017.html Looking forward to seeing you all there. The Bro Team ------ Jeannette Dopheide Sr. Education, Outreach, and Training Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170905/8ce49993/attachment.html From seth at corelight.com Tue Sep 5 12:23:55 2017 From: seth at corelight.com (Seth Hall) Date: Tue, 05 Sep 2017 15:23:55 -0400 Subject: [Bro] caret and the stick In-Reply-To: References: Message-ID: On 5 Sep 2017, at 14:54, Allen, Brian wrote: > Here is a line from our conn.log showing what I think is backscatter. > (Our network is 128.252.0.0/16.) > > > 128.252.X.Y 57756 111.29.2.3 80 tcp - - - > - OTH T F 0 ^h 0 0 > 1 44 > > So in this example, what was flipped exactly? Good question! For background, Bro "flips" connections in there case that it thinks it has orig and resp backwards. You nailed a very common case where this will be true. Since backscatter will frequently have a server port as the src port the "correct" way to view that connection (if it was an actual full connection) would be to "flip" it and swap the orig and resp. In the case that you outlined, 111.29.2.3 sent a single packet (a syn-ack based on the history field) with src port 80 and dst port 57756 (the likely actual ephemeral port). Since Bro initially had no context, it viewed 111.29.2.3 as the originator since it was the first host that seemed to send a packet. But, 80/tcp is registered as a likely server port and no other analyzers attached to the connection so Bro flipped it so that the likely server port was the resp_p. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From bro at pingtrip.com Wed Sep 6 06:48:50 2017 From: bro at pingtrip.com (Dave Crawford) Date: Wed, 6 Sep 2017 09:48:50 -0400 Subject: [Bro] Debian Packages Message-ID: Does anyone know why there are so many dependencies for the ?broctl? Debian package (mysql as an example)? -Dave From johanna at icir.org Thu Sep 7 13:13:10 2017 From: johanna at icir.org (Johanna Amann) Date: Thu, 7 Sep 2017 13:13:10 -0700 Subject: [Bro] Debian Packages In-Reply-To: References: Message-ID: <20170907201310.3qgccqmccvk2ul6i@user143.sys.ICSI.Berkeley.EDU> Hi, > Does anyone know why there are so many dependencies for the ?broctl? > Debian package (mysql as an example)? this is caused by recursive dependencies of the software that broctl depends on. Specifically, broctl has a dependency on a mail server; if none is installed on debian, the default is exim, which has a dependency on some mysql/mariadb libraries. The direct dependency list of broctl is quite short: bro, capstats, trace-summary, a mail transport agent, python, and python-broccoli. due to recursive dependencies this gets quite blown up - for a full dependency graph on debian, see https://icir.org/johanna/tmp/broctl-dependencies.pdf Johanna From johanna at icir.org Thu Sep 7 13:17:34 2017 From: johanna at icir.org (Johanna Amann) Date: Thu, 7 Sep 2017 13:17:34 -0700 Subject: [Bro] debug prints In-Reply-To: References: Message-ID: <20170907201734.wchb7ghut5yi4l5z@user143.sys.ICSI.Berkeley.EDU> Hi, is it possible that you are running this on a broctl cluster? In this case, each worker has its own stderr.log/stdout.log in spool/[workername]. Johanna On Fri, Aug 25, 2017 at 01:52:27AM +0000, Dk Jack wrote: > Hi All, > I am noticing a weird issue. I am not seeing debug prints from my scripts. > Initially some debug prints show up in stdout.log. Debug prints I make when > analyzing traffic i.e. http_header event etc are not showing up. I am using > flush_all() in my scripts and yet the debugs are not showing up in > stdout.log. Could someone comment on what could be going here? Thanks. > > Dk > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jazoff at illinois.edu Thu Sep 7 13:33:18 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 7 Sep 2017 20:33:18 +0000 Subject: [Bro] Debian Packages In-Reply-To: References: Message-ID: > On Sep 6, 2017, at 8:48 AM, Dave Crawford wrote: > > Does anyone know why there are so many dependencies for the ?broctl? Debian package (mysql as an example)? > > -Dave You can workaround this using some options to apt. You can use apt install broctl mysql-common- to install broctl but not pull in the mysql libs. This gets the package install from 76 packages and 94MB to 48 packages and 65MB. Using apt install --no-install-recommends broctl gets it down to 37 packages and 48MB, but that probably leaves out some things that you'd want like the geoip stuff. -- - Justin Azoff From dnj0496 at gmail.com Thu Sep 7 14:04:20 2017 From: dnj0496 at gmail.com (Dk Jack) Date: Thu, 7 Sep 2017 14:04:20 -0700 Subject: [Bro] debug prints In-Reply-To: <20170907201734.wchb7ghut5yi4l5z@user143.sys.ICSI.Berkeley.EDU> References: <20170907201734.wchb7ghut5yi4l5z@user143.sys.ICSI.Berkeley.EDU> Message-ID: Yes, I am running in a cluster. I'll check the worker logs. Thanks > On Sep 7, 2017, at 1:17 PM, Johanna Amann wrote: > > Hi, > > is it possible that you are running this on a broctl cluster? In this > case, each worker has its own stderr.log/stdout.log in spool/[workername]. > > Johanna > >> On Fri, Aug 25, 2017 at 01:52:27AM +0000, Dk Jack wrote: >> Hi All, >> I am noticing a weird issue. I am not seeing debug prints from my scripts. >> Initially some debug prints show up in stdout.log. Debug prints I make when >> analyzing traffic i.e. http_header event etc are not showing up. I am using >> flush_all() in my scripts and yet the debugs are not showing up in >> stdout.log. Could someone comment on what could be going here? Thanks. >> >> Dk > >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From jdopheid at illinois.edu Mon Sep 11 08:06:06 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Mon, 11 Sep 2017 15:06:06 +0000 Subject: [Bro] =?utf-8?q?BroCon_=E2=80=9917_panel_talk=3A_Submit_questions?= =?utf-8?q?_to_our_form?= Message-ID: <41E2ABE5-D41A-4789-BE97-26F2AAD5BAD1@illinois.edu> Hello BroCon attendees, We are soliciting questions from the audience for Thursday?s panel. You can submit your question here: https://goo.gl/forms/7ET7t2uOTX2TlMFa2 Our panel this year is: Leadership Team members Vern Paxson, Robin Sommer, Seth Hall, Keith Lehigh, Adam Slagell (moderator), and NSF Program Director Anita Nikolich. ------ Jeannette Dopheide Sr. Education, Outreach, and Training Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From vikrambasu059 at gmail.com Tue Sep 12 02:44:35 2017 From: vikrambasu059 at gmail.com (Vikram Basu) Date: Tue, 12 Sep 2017 15:14:35 +0530 Subject: [Bro] Keyword matching in documents Message-ID: <59b7ac85.caa56b0a.e353c.1554@mx.google.com> Hi, Is it possible for Bro to perform keyword matching on document files (such as text, open office, pdf etc.) and generate notices when the keyword is found. Regards Vikram Basu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170912/5eb51667/attachment.html From vikrambasu059 at gmail.com Tue Sep 12 04:28:38 2017 From: vikrambasu059 at gmail.com (Vikram Basu) Date: Tue, 12 Sep 2017 16:58:38 +0530 Subject: [Bro] Keyword matching in documents In-Reply-To: <59b7ac85.caa56b0a.e353c.1554@mx.google.com> References: <59b7ac85.caa56b0a.e353c.1554@mx.google.com> Message-ID: <59b7c4e7.a2a86b0a.826c5.4b84@mx.google.com> I have made a sample Bro script after looking into the ssn-exposure and credit-card-exposure scripts. But I am getting error ?{"ts":1505214009.989112,"level":"Reporter::ERROR","message":"string without NUL terminator: \u0022CONFIDENTIAL\u005cx0a\u0022","location":""}? in reporter.log How would I fix this ? Regards Vikram Here is the script #Keyword Matching Basic script @load base/frameworks/notice module KeywordMatch; export { ## Keyword Matching Log ID definition redef enum Log::ID += { LOG }; redef enum Notice::Type += { Matched }; type Info: record { ts: time &log; uid: string &log; id: conn_id &log; word: string &log &optional; data: string &log; }; ## The Keyword that is being matched const keyword = "CONFIDENTIAL" &redef; } event bro_init() &priority=5 { Log::create_stream(KeywordMatch::LOG, [$columns=Info]); } function check_keyword(c: connection, data: string): bool { local it_matched = F; if ( keyword in data ) { it_matched = T; } if ( it_matched ) { local log: Info = [$ts=network_time(), $uid=c$uid, $id=c$id, $word=keyword, $data=data]; Log::write(KeywordMatch::LOG, log); NOTICE([$note=Matched,$conn=c, $msg=fmt("Keyword Matched %s",keyword), $sub=data,$identifier=cat(c$id$orig_h,c$id$resp_h)]); return T; } return F; } event KeywordMatch::stream_data(f: fa_file, data: string) { local c: connection; for ( id in f$conns ) { c = f$conns[id]; break; } if ( c$start_time > network_time()-20secs ) check_keyword(c, data); } event file_new (f: fa_file) { if ( f$source =="HTTP" ) { Files::add_analyzer(f, Files::ANALYZER_DATA_EVENT, [$stream_event=KeywordMatch::stream_data]); } } From: Vikram Basu Sent: 12 September 2017 03:14 PM To: bro at bro.org Subject: Keyword matching in documents Hi, Is it possible for Bro to perform keyword matching on document files (such as text, open office, pdf etc.) and generate notices when the keyword is found. Regards Vikram Basu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170912/cf2ca195/attachment.html From zeolla at gmail.com Wed Sep 13 07:45:12 2017 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Wed, 13 Sep 2017 14:45:12 +0000 Subject: [Bro] `bro-pkg install .` issue Message-ID: Under step 6 of this documentation it shows that you can install a package with `bro-pkg install .`, but I'm having some issues doing that. I've attached a screenshot - anybody know why this would be happening? Jon [image: local bro-pkg install.png] -- Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170913/97bf1856/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: local bro-pkg install.png Type: image/png Size: 72512 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170913/97bf1856/attachment-0001.bin From jsiwek at illinois.edu Wed Sep 13 10:12:19 2017 From: jsiwek at illinois.edu (Siwek, Jon) Date: Wed, 13 Sep 2017 17:12:19 +0000 Subject: [Bro] `bro-pkg install .` issue In-Reply-To: References: Message-ID: <290344B2-4582-4BF1-85D1-5BA5E41322C8@illinois.edu> > On Sep 13, 2017, at 9:45 AM, Zeolla at GMail.com wrote: > > Under step 6 of this documentation it shows that you can install a package with `bro-pkg install .`, but I'm having some issues doing that. I've attached a screenshot - anybody know why this would be happening? At a glance, I don?t know what would cause that. You could try going through the full steps of the walkthrough you linked with a dummy package just to see if it works with a simpler package. If it doesn?t work with a simpler dummy package, that?s clearly a bug. Or generally feel free to open a github issue for this. - Jon From jazoff at illinois.edu Wed Sep 13 11:03:13 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 13 Sep 2017 18:03:13 +0000 Subject: [Bro] `bro-pkg install .` issue In-Reply-To: <290344B2-4582-4BF1-85D1-5BA5E41322C8@illinois.edu> References: <290344B2-4582-4BF1-85D1-5BA5E41322C8@illinois.edu> Message-ID: > On Sep 13, 2017, at 12:12 PM, Siwek, Jon wrote: > > >> On Sep 13, 2017, at 9:45 AM, Zeolla at GMail.com wrote: >> >> Under step 6 of this documentation it shows that you can install a package with `bro-pkg install .`, but I'm having some issues doing that. I've attached a screenshot - anybody know why this would be happening? > > At a glance, I don?t know what would cause that. You could try going through the full steps of the walkthrough you linked with a dummy package just to see if it works with a simpler package. If it doesn?t work with a simpler dummy package, that?s clearly a bug. Or generally feel free to open a github issue for this. I think I ran into the same issue when the local git repo was in a weird state.. like I hadn't committed yet, or hadn't created a tag yet.. so the git stuff that bro-pkg does got confused. Problem went away on its own once I had things more fleshed out. -- - Justin Azoff From zeolla at gmail.com Wed Sep 13 12:53:43 2017 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Wed, 13 Sep 2017 19:53:43 +0000 Subject: [Bro] `bro-pkg install .` issue In-Reply-To: References: <290344B2-4582-4BF1-85D1-5BA5E41322C8@illinois.edu> Message-ID: Problem solved - user error, but with some follow-up. At first, I thought bro-pkg install . was leveraging the local (./) directory structure, but instead it was using the local git repo, and assumed the master branch (not the current branch). If I run bro-pkg install . --version bro-pkg (which is the name of my git branch with the right bro-pkg.meta file) it works just fine. Per a chat with Jon, I've opened this issue as a follow-on. Jon On Wed, Sep 13, 2017 at 1:03 PM Azoff, Justin S wrote: > > > On Sep 13, 2017, at 12:12 PM, Siwek, Jon wrote: > > > > > >> On Sep 13, 2017, at 9:45 AM, Zeolla at GMail.com wrote: > >> > >> Under step 6 of this documentation it shows that you can install a > package with `bro-pkg install .`, but I'm having some issues doing that. > I've attached a screenshot - anybody know why this would be happening? > > > > At a glance, I don?t know what would cause that. You could try going > through the full steps of the walkthrough you linked with a dummy package > just to see if it works with a simpler package. If it doesn?t work with a > simpler dummy package, that?s clearly a bug. Or generally feel free to > open a github issue for this. > > I think I ran into the same issue when the local git repo was in a weird > state.. like I hadn't committed yet, or hadn't created a tag yet.. so the > git stuff that bro-pkg does got confused. Problem went away on its own > once I had things more fleshed out. > > > > -- > - Justin Azoff > > -- Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170913/7f94305e/attachment.html From bill.de.ping at gmail.com Thu Sep 14 00:55:43 2017 From: bill.de.ping at gmail.com (william de ping) Date: Thu, 14 Sep 2017 10:55:43 +0300 Subject: [Bro] - change default MTU for pcaps processing Message-ID: Hi, Does anyone know how can I change the default MTU for bro ? This is relevant for pcap parsing and not for interface monitoring (for the latter, I assume bro will use the MTU for this NIC itself). Thanks B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170914/a0e8cb5d/attachment.html From map at cvm.com Fri Sep 15 07:54:03 2017 From: map at cvm.com (Marco A. Patritti) Date: Fri, 15 Sep 2017 10:54:03 -0400 (EDT) Subject: [Bro] Bro IDS Question Message-ID: <000c01d32e33$0be4a520$23adef60$@cvm.com> Hello, I am researching different IDS's to integrate to my company's infrastructure and would need to know how Bro specifically identify intrusions? Please note that the contact form on bro.org site is unresponsive and this is why I subscribed to the mailing list and found this contact email. Your assistance will be greatly appreciated. Thank you very much, Marco -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170915/8b37aae1/attachment.html From johanna at icir.org Fri Sep 15 12:23:41 2017 From: johanna at icir.org (Johanna Amann) Date: Fri, 15 Sep 2017 12:23:41 -0700 Subject: [Bro] - change default MTU for pcaps processing In-Reply-To: References: Message-ID: <20170915192341.occj5hkehyiavnd5@user203.sys.ICSI.Berkeley.EDU> Hi, the way to do this is to redef Pcap::snaplen to a desired value. If you use broctl, you might have to set pcapsnaplen in broctl.cfg (I think broctl might overwrite the value otherwhise). Note that this is also used for interface monitoring - as far as I am aware, Bro does not just use the NIC MTU. The default snaplen of Bro is 8192. Johanna On Thu, Sep 14, 2017 at 07:55:43AM +0000, william de ping wrote: > Hi, > > Does anyone know how can I change the default MTU for bro ? > > This is relevant for pcap parsing and not for interface monitoring (for the > latter, I assume bro will use the MTU for this NIC itself). > > Thanks > B > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Mon Sep 18 12:23:33 2017 From: johanna at icir.org (Johanna Amann) Date: Mon, 18 Sep 2017 12:23:33 -0700 Subject: [Bro] Keyword matching in documents In-Reply-To: <59b7c4e7.a2a86b0a.826c5.4b84@mx.google.com> References: <59b7ac85.caa56b0a.e353c.1554@mx.google.com> <59b7c4e7.a2a86b0a.826c5.4b84@mx.google.com> Message-ID: <20170918192333.mz4schm3m35vwls4@Beezling.local> Hi Vikram, it turns out that you found a small bug (or at least gotcha) in Bro. Bro has a few functions that do not deal very well with binary data. "in" happens to be one of them. I wrote a small patch to Bro that should fix this problems. It is in the branch topic/johanna/in-binary. If you want to manually apply it, you only need the single line change in Expr.cc: https://github.com/bro/bro/compare/topic/johanna/in-binary I also created a merge request for this at https://bro-tracker.atlassian.net/browse/BIT-1845 if you are interested in tracking this. Johanna On Tue, Sep 12, 2017 at 11:28:38AM +0000, Vikram Basu wrote: > I have made a sample Bro script after looking into the ssn-exposure and credit-card-exposure scripts. But I am getting error > ?{"ts":1505214009.989112,"level":"Reporter::ERROR","message":"string without NUL terminator: \u0022CONFIDENTIAL\u005cx0a\u0022","location":""}? in reporter.log > > How would I fix this ? > > Regards > > Vikram > > Here is the script > > #Keyword Matching Basic script > @load base/frameworks/notice > > module KeywordMatch; > > export { > ## Keyword Matching Log ID definition > redef enum Log::ID += { LOG }; > > redef enum Notice::Type += { > Matched > }; > > type Info: record { > ts: time &log; > uid: string &log; > id: conn_id &log; > word: string &log &optional; > data: string &log; > }; > > ## The Keyword that is being matched > const keyword = "CONFIDENTIAL" &redef; > > } > > event bro_init() &priority=5 > { > Log::create_stream(KeywordMatch::LOG, [$columns=Info]); > } > > function check_keyword(c: connection, data: string): bool > { > local it_matched = F; > if ( keyword in data ) > { > it_matched = T; > } > > if ( it_matched ) > { > local log: Info = [$ts=network_time(), > $uid=c$uid, $id=c$id, > $word=keyword, $data=data]; > > Log::write(KeywordMatch::LOG, log); > > NOTICE([$note=Matched,$conn=c, > $msg=fmt("Keyword Matched %s",keyword), > $sub=data,$identifier=cat(c$id$orig_h,c$id$resp_h)]); > > return T; > } > return F; > } > > event KeywordMatch::stream_data(f: fa_file, data: string) > { > local c: connection; > for ( id in f$conns ) > { > c = f$conns[id]; > break; > } > if ( c$start_time > network_time()-20secs ) > check_keyword(c, data); > } > > event file_new (f: fa_file) > { > if ( f$source =="HTTP" ) > { > Files::add_analyzer(f, Files::ANALYZER_DATA_EVENT, > [$stream_event=KeywordMatch::stream_data]); > } > } > > From: Vikram Basu > Sent: 12 September 2017 03:14 PM > To: bro at bro.org > Subject: Keyword matching in documents > > > Hi, > > Is it possible for Bro to perform keyword matching on document files (such as text, open office, pdf etc.) and generate notices when the keyword is found. > > Regards > > Vikram Basu > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From briford.wylie at gmail.com Tue Sep 19 15:12:32 2017 From: briford.wylie at gmail.com (Brian Wylie) Date: Tue, 19 Sep 2017 16:12:32 -0600 Subject: [Bro] installation issue with Bro doctor Message-ID: Hi All, - Ubuntu 16.04 - Bro installed from Source (from repo updated today) - Using pyenv for python virtualenv - pip install bro-pkg (bro3) bro-user at bro:~$ bro-pkg install bro-doctor The following packages will be INSTALLED: bro/ncsa/bro-doctor (1.9.0) Proceed? [Y/n] Y Skipping unit tests for "bro/ncsa/bro-doctor": no test_command in metadata Installing "bro/ncsa/bro-doctor" Installed "bro/ncsa/bro-doctor" (1.9.0) (bro3) bro-user at bro:/opt/bro/share$ bro-pkg list bro/ncsa/bro-doctor (installed: 1.9.0) - A broctl plugin that helps you troubleshoot common problems (bro3) bro-user at bro:/opt/bro/share$ broctl doctor.bro Error: unknown command 'doctor.bro' BroControl Version 1.7-7 ... So... I did a find under /opt for doctor.bro or bro-doctor* etc... I didn't find anything... so I'm sure it's just pilot error... can someone point out where I went wrong ? Cheers, -Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170919/5c7d4d6e/attachment.html From jazoff at illinois.edu Tue Sep 19 15:28:34 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 19 Sep 2017 22:28:34 +0000 Subject: [Bro] installation issue with Bro doctor In-Reply-To: References: Message-ID: <0B80FAAF-C0E6-475F-8F9F-2717581EF392@illinois.edu> I think you need an initial bro-pkg autoconfig That will cause bro-pkg to figure out where your bro installation is. I think otherwise it installs to your home directory. Also, there?s a bug in bro-pkg (that should be fixed very soon) that is causing bro-pkg to install an older version of bro-doctor, so for now you?ll want to do bro-pkg install bro-doctor --version 1.13.0 To force it to install the latest version. ? Justin Azoff > On Sep 19, 2017, at 6:12 PM, Brian Wylie wrote: > > Hi All, > > - Ubuntu 16.04 > - Bro installed from Source (from repo updated today) > - Using pyenv for python virtualenv > - pip install bro-pkg > > (bro3) bro-user at bro:~$ bro-pkg install bro-doctor > > The following packages will be INSTALLED: > > bro/ncsa/bro-doctor (1.9.0) > > > > Proceed? [Y/n] Y > > Skipping unit tests for "bro/ncsa/bro-doctor": no test_command in metadata > > Installing "bro/ncsa/bro-doctor" > > Installed "bro/ncsa/bro-doctor" (1.9.0) > > > > (bro3) bro-user at bro:/opt/bro/share$ bro-pkg list > > bro/ncsa/bro-doctor (installed: 1.9.0) - A broctl plugin that helps you troubleshoot common problems > > > > (bro3) bro-user at bro:/opt/bro/share$ broctl doctor.bro > > Error: unknown command 'doctor.bro' > > BroControl Version 1.7-7 > > ... > > > > So... I did a find under /opt for doctor.bro or bro-doctor* etc... I didn't find anything... so I'm sure it's just pilot error... can someone point out where I went wrong ? > > Cheers, > > -Brian > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From seth at corelight.com Wed Sep 20 06:43:54 2017 From: seth at corelight.com (Seth Hall) Date: Wed, 20 Sep 2017 09:43:54 -0400 Subject: [Bro] installation issue with Bro doctor In-Reply-To: <0B80FAAF-C0E6-475F-8F9F-2717581EF392@illinois.edu> References: <0B80FAAF-C0E6-475F-8F9F-2717581EF392@illinois.edu> Message-ID: <8B010B30-0F37-4AE6-B3FD-E5222449A2CC@corelight.com> On 19 Sep 2017, at 18:28, Azoff, Justin S wrote: > I think you need an initial > > bro-pkg autoconfig You need to write the output from that in ~/.bro-pkg/config too. I don't think it automatically puts the config in there. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From jazoff at illinois.edu Wed Sep 20 06:45:06 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 20 Sep 2017 13:45:06 +0000 Subject: [Bro] installation issue with Bro doctor In-Reply-To: <8B010B30-0F37-4AE6-B3FD-E5222449A2CC@corelight.com> References: <0B80FAAF-C0E6-475F-8F9F-2717581EF392@illinois.edu> <8B010B30-0F37-4AE6-B3FD-E5222449A2CC@corelight.com> Message-ID: <7597F116-BFFE-45D6-ACDD-74425132586A@illinois.edu> > On Sep 20, 2017, at 9:43 AM, Seth Hall wrote: > > > > On 19 Sep 2017, at 18:28, Azoff, Justin S wrote: > >> I think you need an initial >> >> bro-pkg autoconfig > > You need to write the output from that in ~/.bro-pkg/config too. I don't think it automatically puts the config in there. > > .Seth > > -- > Seth Hall * Corelight, Inc * www.corelight.com jazoff at mbp ~ $ bro-pkg autoconfig Successfully wrote config file to /Users/jazoff/.bro-pkg/config :-) ? Justin Azoff From seth at corelight.com Wed Sep 20 07:05:34 2017 From: seth at corelight.com (Seth Hall) Date: Wed, 20 Sep 2017 10:05:34 -0400 Subject: [Bro] installation issue with Bro doctor In-Reply-To: <7597F116-BFFE-45D6-ACDD-74425132586A@illinois.edu> References: <0B80FAAF-C0E6-475F-8F9F-2717581EF392@illinois.edu> <8B010B30-0F37-4AE6-B3FD-E5222449A2CC@corelight.com> <7597F116-BFFE-45D6-ACDD-74425132586A@illinois.edu> Message-ID: <0E3D0403-49E9-4D9A-BAA0-4D85D004EDAA@corelight.com> On 20 Sep 2017, at 9:45, Azoff, Justin S wrote: > Successfully wrote config file to /Users/jazoff/.bro-pkg/config > > :-) Ah, that behavior is different than it used to be then. Pretty good to see actually! I like that better than having to write to the file yourself. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From tomas.bortoli at sit.fraunhofer.de Thu Sep 21 07:12:27 2017 From: tomas.bortoli at sit.fraunhofer.de (Bortoli, Tomas) Date: Thu, 21 Sep 2017 14:12:27 +0000 Subject: [Bro] testing performances of binpac's parser Message-ID: Hello everybody, I would like to test a binpac's generated parser that I wrote. Is there a handy way to get the C++ code running on some payload of my choice? I guess there is but it would be helpful if somebody gives a hint on where to start! thanks in advance Cheers, Tomas -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170921/70280204/attachment.html From tomas.bortoli at sit.fraunhofer.de Thu Sep 21 07:15:09 2017 From: tomas.bortoli at sit.fraunhofer.de (Bortoli, Tomas) Date: Thu, 21 Sep 2017 14:15:09 +0000 Subject: [Bro] Binpac bug Message-ID: Hello, If there's any Binpac maintainer here I would kindly invite to take a look at the bug I found and please review the patch I provided if you can. The issue is sitting there from kind of a long time. Cheers, Tomas -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170921/5a84b9ce/attachment.html From johanna at icir.org Thu Sep 21 09:02:43 2017 From: johanna at icir.org (Johanna Amann) Date: Thu, 21 Sep 2017 09:02:43 -0700 Subject: [Bro] testing performances of binpac's parser In-Reply-To: References: Message-ID: <20170921160239.mtdbc6kusw6o3gqs@Beezling.local> Hi, > I would like to test a binpac's generated parser that I wrote. Is there > a handy way to get the C++ code running on some payload of my choice? I > guess there is but it would be helpful if somebody gives a hint on where > to start! I actually recently did this for a research project where I used binpac to parse a X.509 data structure in a standalone project. The source code is up here: https://github.com/0xxon/sct-utils The interesting utility is extractSCT. Note that to be able to use binpac completely without Bro you need to patch binpac.h a little bit; if I remember it correctly it does include Bro header files by default (that are not really necessary in a standalone utility). I hope this helps, Johanna From bluebike.sjlee at gmail.com Thu Sep 21 18:22:25 2017 From: bluebike.sjlee at gmail.com (SJ Lee) Date: Thu, 21 Sep 2017 21:22:25 -0400 Subject: [Bro] Question about GCC version error while run configure Message-ID: Hello everyone, I recently need to re-build a bro package, but keep failed to run configure command because of GCC version. By default does not have a high version, so I installed devtoolset and did symbolic link like below, Can anyone kindly help this issue? /usr/bin/c++ -> /opt/rh/devtoolset-3/root/usr/bin/c++ /usr/bin/gcc -> /opt/rh/devtoolset-3/root/usr/bin/gcc. [Here is details] CMake Error at cmake/RequireCXX11.cmake:40 (message): GCC version must be at least 4.8 for C++11 support, detected: 4.4.7 Call Stack (most recent call first): CMakeLists.txt:168 (include) -- Configuring incomplete, errors occurred! See also "/home/bluebike/bro-2.5.1/build/CMakeFiles/CMakeOutput.log". See also "/home/bluebike/bro-2.5.1/build/CMakeFiles/CMakeError.log". bash-4.1$ gcc --version gcc (GCC) 4.9.2 20150212 (Red Hat 4.9.2-6) Copyright (C) 2014 Free Software Foundation, Inc. This is free software; see the source for copying conditions. bash-4.1$ c++ --version c++ (GCC) 4.9.2 20150212 (Red Hat 4.9.2-6) Copyright (C) 2014 Free Software Foundation, Inc. This is free software; see the source for copying conditions. Thank you in advance, Regards, SJ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170921/361498e1/attachment.html From jazoff at illinois.edu Thu Sep 21 18:37:19 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 22 Sep 2017 01:37:19 +0000 Subject: [Bro] Question about GCC version error while run configure In-Reply-To: References: Message-ID: <483BA663-F552-4D3F-B75A-B069997201DE@illinois.edu> > On Sep 21, 2017, at 9:22 PM, SJ Lee wrote: > > Hello everyone, > I recently need to re-build a bro package, but keep failed to run configure command because of GCC version. The compiler used gets cached inside build/ If you've installed a new one it won't get picked up automatically, rm -r build/ ./configure Should fix that. ? Justin Azoff From tomas.bortoli at sit.fraunhofer.de Fri Sep 22 02:30:57 2017 From: tomas.bortoli at sit.fraunhofer.de (Bortoli, Tomas) Date: Fri, 22 Sep 2017 09:30:57 +0000 Subject: [Bro] testing performances of binpac's parser In-Reply-To: <20170921160239.mtdbc6kusw6o3gqs@Beezling.local> References: , <20170921160239.mtdbc6kusw6o3gqs@Beezling.local> Message-ID: Thanks, I'll check that out! Tomas ________________________________________ From: Johanna Amann [johanna at icir.org] Sent: Thursday, September 21, 2017 6:02 PM To: Bortoli, Tomas Cc: bro at bro.org Subject: Re: [Bro] testing performances of binpac's parser Hi, > I would like to test a binpac's generated parser that I wrote. Is there > a handy way to get the C++ code running on some payload of my choice? I > guess there is but it would be helpful if somebody gives a hint on where > to start! I actually recently did this for a research project where I used binpac to parse a X.509 data structure in a standalone project. The source code is up here: https://github.com/0xxon/sct-utils The interesting utility is extractSCT. Note that to be able to use binpac completely without Bro you need to patch binpac.h a little bit; if I remember it correctly it does include Bro header files by default (that are not really necessary in a standalone utility). I hope this helps, Johanna From franky.meier.1 at gmx.de Fri Sep 22 04:15:06 2017 From: franky.meier.1 at gmx.de (Frank Meier) Date: Fri, 22 Sep 2017 13:15:06 +0200 Subject: [Bro] optimize running bro from PCAPs / advantage of cluster mode Message-ID: <20170922131506.3883ecfa@NB181106> Hello! In contrast to the normal use case I run Bro mostly from pcaps. When huge amounts of data (~20 TB) have to be processed, bro in standalone mode becomes a real bottleneck. So I thought about using the bro cluster mode. In the past I thought, the bro workers would communicate with each other, so when for example one worker sees upstream and the other downstream, they would combine the information to one log. Seth told me at BroCon, that Bro needs to be fed complete streams. To do this some kind of load balancer is needed in front of bro. When I need to split the flows with a load balancer anyway, is there any advantage of running bro in cluster mode at all? I do not need any shared data like tables. Are there any parsers which combine the information seen by different workers in different flows? If cluster mode has no added value in my case, I could just load balance my pcaps to independent bro instances which would make my setup much easier. Have a nice weekend! Franky From bluebike.sjlee at gmail.com Fri Sep 22 06:40:57 2017 From: bluebike.sjlee at gmail.com (SJ Lee) Date: Fri, 22 Sep 2017 09:40:57 -0400 Subject: [Bro] Question about GCC version error while run configure In-Reply-To: <483BA663-F552-4D3F-B75A-B069997201DE@illinois.edu> References: <483BA663-F552-4D3F-B75A-B069997201DE@illinois.edu> Message-ID: Thanks a lot, finally issues are resolved. --SJ On Thu, Sep 21, 2017 at 9:37 PM, Azoff, Justin S wrote: > > > On Sep 21, 2017, at 9:22 PM, SJ Lee wrote: > > > > Hello everyone, > > I recently need to re-build a bro package, but keep failed to run > configure command because of GCC version. > > The compiler used gets cached inside build/ > > If you've installed a new one it won't get picked up automatically, > > rm -r build/ > ./configure > > Should fix that. > > > ? > Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170922/7694879f/attachment.html From dopheide at gmail.com Fri Sep 22 08:33:55 2017 From: dopheide at gmail.com (Mike Dopheide) Date: Fri, 22 Sep 2017 10:33:55 -0500 Subject: [Bro] optimize running bro from PCAPs / advantage of cluster mode In-Reply-To: <20170922131506.3883ecfa@NB181106> References: <20170922131506.3883ecfa@NB181106> Message-ID: Frank, I would argue that using Bro's cluster configuration ends up making it a lot easier for you in the long run. 1) To start, you only have one logger node so all of your logs will be in one place and you don't have to worry about trying to consolidate them later. 2) broctl provides an easy way to check the status of all of your nodes without having to write anything custom. 3) Sync'ing all of your bro binaries and policies across all workers is also done for you. 4) I question not needing to have shared tables, but I also don't know your environment and your end goals. That's how most of the scan detection scripts work, by counting the number of anomalies over time across all of your traffic. If an attacker scans you ten times which are split across ten bro nodes that aren't communicating with each other, you may miss it. A lot of the malware detection policies also look for the inbound connection and then a separate outbound connection. Also, using broctl puts you in the same place as a lot of other other installations so it's easier for people on this list to help troubleshoot. -Dop On Fri, Sep 22, 2017 at 6:15 AM, Frank Meier wrote: > Hello! > > In contrast to the normal use case I run Bro mostly from pcaps. When > huge amounts of data (~20 TB) have to be processed, bro in standalone > mode becomes a real bottleneck. So I thought about using the bro cluster > mode. > > In the past I thought, the bro workers would communicate with each > other, so when for example one worker sees upstream and the other > downstream, they would combine the information to one log. Seth told me > at BroCon, that Bro needs to be fed complete streams. To do this some > kind of load balancer is needed in front of bro. > > When I need to split the flows with a load balancer anyway, is there any > advantage of running bro in cluster mode at all? I do not need any > shared data like tables. Are there any parsers which combine the > information seen by different workers in different flows? > > If cluster mode has no added value in my case, I could just load > balance my pcaps to independent bro instances which would make my setup > much easier. > > Have a nice weekend! > > Franky > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170922/aeb83c52/attachment-0001.html From glallen01 at gmail.com Sat Sep 23 08:39:33 2017 From: glallen01 at gmail.com (George Allen) Date: Sat, 23 Sep 2017 15:39:33 +0000 Subject: [Bro] Netflow and bro Message-ID: Is there a decoder for Netflow, such that one could use bro to collect and log Netflow packets seen by a hardware tap, from multiple sources, in a similar fashion to how Bro handles syslog? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170923/ea9a7614/attachment.html From briford.wylie at gmail.com Sun Sep 24 14:52:31 2017 From: briford.wylie at gmail.com (Brian Wylie) Date: Sun, 24 Sep 2017 15:52:31 -0600 Subject: [Bro] traffic vs log size Message-ID: Hi All, I know these questions have lots of variables and 'it depends' but modulo that, I'm looking for anecdotal information on the 'data reduction' that happens with bro logs. Example: - The tap/span sees 2TBytes of traffic per day. - All the bro logs files for that day are approx 4GBytes on disk. So in this case the log files are giving about a 500x reduction in data. Again I know there are lots of factors.. just looking for a few data points from folks running Bro on a daily basis. In particular I'd like to get numbers for uncompressed log sizes. Thanks in advance, -Bri -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170924/9e416e75/attachment.html From zeolla at gmail.com Mon Sep 25 04:49:15 2017 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Mon, 25 Sep 2017 11:49:15 +0000 Subject: [Bro] traffic vs log size In-Reply-To: References: Message-ID: My bro sensors are sent about 56TB/day and log around 600GB/day uncompressed. Jon On Sun, Sep 24, 2017, 18:02 Brian Wylie wrote: > Hi All, > > I know these questions have lots of variables and 'it depends' but modulo > that, I'm looking for anecdotal information on the 'data reduction' that > happens with bro logs. > > Example: > - The tap/span sees 2TBytes of traffic per day. > - All the bro logs files for that day are approx 4GBytes on disk. > > So in this case the log files are giving about a 500x reduction in data. > Again I know there are lots of factors.. just looking for a few data points > from folks running Bro on a daily basis. In particular I'd like to get > numbers for uncompressed log sizes. > > Thanks in advance, > -Bri > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170925/37525788/attachment.html From landy-bible at utulsa.edu Mon Sep 25 06:13:13 2017 From: landy-bible at utulsa.edu (Landy Bible) Date: Mon, 25 Sep 2017 13:13:13 +0000 Subject: [Bro] traffic vs log size In-Reply-To: References: Message-ID: Sample size of one day... 138.5 GB of traffic, 12.6 GB of logs. On Mon, Sep 25, 2017 at 6:57 AM Zeolla at GMail.com wrote: > My bro sensors are sent about 56TB/day and log around 600GB/day > uncompressed. > > Jon > > On Sun, Sep 24, 2017, 18:02 Brian Wylie wrote: > >> Hi All, >> >> I know these questions have lots of variables and 'it depends' but modulo >> that, I'm looking for anecdotal information on the 'data reduction' that >> happens with bro logs. >> >> Example: >> - The tap/span sees 2TBytes of traffic per day. >> - All the bro logs files for that day are approx 4GBytes on disk. >> >> So in this case the log files are giving about a 500x reduction in data. >> Again I know there are lots of factors.. just looking for a few data points >> from folks running Bro on a daily basis. In particular I'd like to get >> numbers for uncompressed log sizes. >> >> Thanks in advance, >> -Bri >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- > > Jon > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Landy Bible Information Security Analyst The University of Tulsa -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170925/1faa5087/attachment.html From ptcnop at gmail.com Mon Sep 25 06:55:26 2017 From: ptcnop at gmail.com (Patrick Copeland) Date: Mon, 25 Sep 2017 09:55:26 -0400 Subject: [Bro] Bro MITM Detection Message-ID: Hello, I have a question about Bro MITM detection. Here's the general scenario I'm curious about: Bro sensor is fed off switch SPAN port. Adversary has MITM on LAN using ARP cache poisoning with the goal of modifying responses. From packet capture you see that for every request, there are two responses (1) server->adversary (good) and (2) adversary->host (bad). The modified packet is identical except that it has a different src mac addr and the application layer has been modified. Right now Bro is parsing the original response but is ignoring the modified response. I can?t find anything in weird.log / notice.log to know that it is processing the second packet at all. Questions: - Would you expect Bro to parse and log both good resp and the modified bad resp? - Is it application layer dependent? - Any thoughts about how switch SPAN configuration might affect Bro output? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170925/668b51aa/attachment.html From briford.wylie at gmail.com Mon Sep 25 11:12:26 2017 From: briford.wylie at gmail.com (Brian Wylie) Date: Mon, 25 Sep 2017 18:12:26 +0000 Subject: [Bro] traffic vs log size In-Reply-To: References: Message-ID: Thanks for the info guys... On Mon, Sep 25, 2017 at 8:13 AM Landy Bible wrote: > Sample size of one day... 138.5 GB of traffic, 12.6 GB of logs. > > On Mon, Sep 25, 2017 at 6:57 AM Zeolla at GMail.com wrote: > >> My bro sensors are sent about 56TB/day and log around 600GB/day >> uncompressed. >> >> Jon >> >> On Sun, Sep 24, 2017, 18:02 Brian Wylie wrote: >> >>> Hi All, >>> >>> I know these questions have lots of variables and 'it depends' but >>> modulo that, I'm looking for anecdotal information on the 'data reduction' >>> that happens with bro logs. >>> >>> Example: >>> - The tap/span sees 2TBytes of traffic per day. >>> - All the bro logs files for that day are approx 4GBytes on disk. >>> >>> So in this case the log files are giving about a 500x reduction in data. >>> Again I know there are lots of factors.. just looking for a few data points >>> from folks running Bro on a daily basis. In particular I'd like to get >>> numbers for uncompressed log sizes. >>> >>> Thanks in advance, >>> -Bri >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> -- >> >> Jon >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- > Landy Bible > Information Security Analyst > The University of Tulsa > -- Sent from iPhone -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170925/3d636c43/attachment-0001.html From jmellander at lbl.gov Mon Sep 25 11:48:55 2017 From: jmellander at lbl.gov (Jim Mellander) Date: Mon, 25 Sep 2017 11:48:55 -0700 Subject: [Bro] Bro MITM Detection In-Reply-To: References: Message-ID: A couple of thoughts: It seems that you would want to be looking for ARP poisoning directly, with the MAC address changing. IIRC, I've seen that happening under normal circumstances with Apple TV, when it puts both the MAC address for both hardwired & wireless on the wire.... Either a bug, or a feature, you decide..... Although Bro can generate events for ARP traffic (see Bro_ARP.events.bif.bro), there are no policies in the current distro - bro 1.5 has ARP policies but they haven't been ported. So the other part of your question is in relation to spurious TCP acks containing different payload. I haven't examined the bro source code in detail for that circumstance (although I suspect it would be a bookkeeping problem to maintain acked packets in the off-chance that a valid ack packet with different contents is received), but the end host should be silently rejecting acks for packets that already have been accepted - the first valid packet (with correct seq #s) should win. Also, with the randomization of ISNs in all modern OSs, this attack would be difficult to accomplish successfully anyway, unless the attacker had access to the span port in order to sniff the actual ISNs from the traffic. Of interest, however, would be if an attacker could spoof the router's IP via ARP poisoning, and convince a host to route its traffic through the attacker's box which would allow for payload modifications. The span port would probably see a bunch of ARP flooding at that point. Hope this helps, Jim On Mon, Sep 25, 2017 at 6:55 AM, Patrick Copeland wrote: > Hello, > > I have a question about Bro MITM detection. Here's the general scenario > I'm curious about: > > Bro sensor is fed off switch SPAN port. Adversary has MITM on LAN using > ARP cache poisoning with the goal of modifying responses. From packet > capture you see that for every request, there are two responses (1) > server->adversary (good) and (2) adversary->host (bad). The modified packet > is identical except that it has a different src mac addr and the > application layer has been modified. > > Right now Bro is parsing the original response but is ignoring the > modified response. I can?t find anything in weird.log / notice.log to know > that it is processing the second packet at all. > > Questions: > - Would you expect Bro to parse and log both good resp and the modified > bad resp? > - Is it application layer dependent? > - Any thoughts about how switch SPAN configuration might affect Bro output? > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170925/a402274e/attachment.html From matt.clemons at gmail.com Tue Sep 26 07:18:35 2017 From: matt.clemons at gmail.com (Matt Clemons) Date: Tue, 26 Sep 2017 09:18:35 -0500 Subject: [Bro] slides Message-ID: Any news on when the brocon slides and presentations will be up? I really would like the Deloitte slides and ones by Mark Krenz. -- Regards, Matt Clemons 816-200-0789 GPG: https://mattclemons.com/GPG/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170926/18c5f00c/attachment.html From jdopheid at illinois.edu Tue Sep 26 07:28:32 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Tue, 26 Sep 2017 14:28:32 +0000 Subject: [Bro] slides In-Reply-To: References: Message-ID: <7EFD7D614A2BB84ABEA19B2CEDD246583B55ED63@CITESMBX5.ad.uillinois.edu> I've been posting slides to the agenda as they are available (and I have time to do so). I'm working on the videos this week but don't want to make any guarantees, the videos will probably be uploaded within a couple weeks. ------ Jeannette M. Dopheide Bro Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign ________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Matt Clemons [matt.clemons at gmail.com] Sent: Tuesday, September 26, 2017 9:18 AM To: bro at bro.org Subject: [Bro] slides Any news on when the brocon slides and presentations will be up? I really would like the Deloitte slides and ones by Mark Krenz. -- Regards, Matt Clemons 816-200-0789 GPG: https://mattclemons.com/GPG/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170926/cc707d1d/attachment.html From haoscs at gmail.com Tue Sep 26 08:38:20 2017 From: haoscs at gmail.com (Shuai Hao) Date: Tue, 26 Sep 2017 11:38:20 -0400 Subject: [Bro] Losing events associated with Signature Matching Message-ID: Hi All, My Bro program shows a wired behavior. We leverage the signature framework to capture embedded components in HTTP replies (http-reply-body) as well as the file download (tcp payload). However, we lose many events associated with the signature (only around 1/3 shown). The exactly same program actually runs well on another desktop (capturing all signature matching we issued). I would be appreciate if anyone can have a clue on the problem. The machine running bro is fanless computer with Intel Atom and Ubuntu 16.04. It is almost dedicated to the Bro monitoring so it shouldn't be performance issue. The signature matching is quite straightforward: we define some simple signature patterns, load those signatures to BroControl, and pull some fields from corresponding log files via a broccoli python client. We do capture some signature matching events, but also lose many that should be captured. Those events are not shown in signatures.log; it means that they are either failure of capturing or dropped by Bro Control, rather than the problem of python client. BTW, we use File Analysis to capture the file downloads, it works well as expected. Thanks very much for any comments~ Cheers, -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170926/1259a337/attachment.html From radoslawc at gmail.com Tue Sep 26 15:07:24 2017 From: radoslawc at gmail.com (radek) Date: Wed, 27 Sep 2017 00:07:24 +0200 Subject: [Bro] bro and pf_ring zc configuration success stories Message-ID: Hi! Anyone care to share bro + pfring success story? What's the speed, what NIC, what's the configuration. I'm running bro 2.5.1 built with jemalloc and gperftools and against pf_ring 6.6.0 with ixgbe_zc on CentOS 7.2. In ZeroCopy mode with zbalance_ipc dividing NIC to 20 application rings (-n 20) I'm getting each CPU core loaded at 100% and around 50% packet drop (reported by netstats in broctl). When redirecting from zc to 20 dummy interfaces (zbalance_ipc -r 0:dummy0 and so on) I'm getting around 50% load on each core and a lot less of packet drop (10% - 15%). This is with traffic around 700 - 800 Mbit/s All input will be highly appreciated. Best regards Rado -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170927/0410e2ef/attachment.html From dopheide at gmail.com Wed Sep 27 13:12:29 2017 From: dopheide at gmail.com (Mike Dopheide) Date: Wed, 27 Sep 2017 15:12:29 -0500 Subject: [Bro] careful w/ myricom plugin switchover on cluster Message-ID: Howdy, This is just a heads-up for anyone using the myricom plugin that was shipped with Bro who is getting ready to upgrade and switch to the bro-pkg'd version. This is an upcoming change that Seth mentioned at BroCon. Long story short, if you're going to do that, wait just a little bit or let me know and I'll help with some details. It caused our cluster to be down for several hours while we figured it out and I'd hate for others to have to debug it again. We think we've identified a bug in one of the core distributed cmake files that causes plugins to create a symlink in the installed build environment back to the .bro-pkg directory. The result of this is when you push Bro to the cluster w/ 'broctl deploy', the plugin breaks on the workers. We haven't noticed the issue with anything other than the myricom plugin, but I believe it could impact others. (Shout out to Sam Oehlert @ ESnet that figured out the make file issue was in the Bro core and not part of the plugin itself!) -Dop -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170927/801c9b77/attachment.html From jazoff at illinois.edu Wed Sep 27 14:53:22 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 27 Sep 2017 21:53:22 +0000 Subject: [Bro] bro and pf_ring zc configuration success stories In-Reply-To: References: Message-ID: <212F32B7-0DAA-48D9-AD0F-4A308BD57271@illinois.edu> > On Sep 26, 2017, at 6:07 PM, radek wrote: > > Hi! > > Anyone care to share bro + pfring success story? > > What's the speed, what NIC, what's the configuration. > > I'm running bro 2.5.1 built with jemalloc and gperftools and against pf_ring 6.6.0 with ixgbe_zc on CentOS 7.2. You can't be using both jemalloc and gperftools(tcmalloc).. they are both malloc implementations. > In ZeroCopy mode with zbalance_ipc dividing NIC to 20 application rings (-n 20) I'm getting each CPU core loaded at 100% and around 50% packet drop (reported by netstats in broctl). Sounds like the load balancing is not working right and you are just analyzing all of your traffic 20 times. What does your node.cfg contain? > When redirecting from zc to 20 dummy interfaces (zbalance_ipc -r 0:dummy0 and so on) I'm getting around 50% load on each core and a lot less of packet drop (10% - 15%). > > This is with traffic around 700 - 800 Mbit/s A few workers should be able to handle this load, not to mention 20.. > All input will be highly appreciated. > Can you install bro-pkg (http://bro-package-manager.readthedocs.io/en/stable/quickstart.html) and then do bro-pkg install bro-doctor --version 1.16.1 broctl doctor.bro And share the results. ? Justin Azoff From radoslawc at gmail.com Thu Sep 28 02:52:41 2017 From: radoslawc at gmail.com (radek) Date: Thu, 28 Sep 2017 11:52:41 +0200 Subject: [Bro] bro and pf_ring zc configuration success stories In-Reply-To: <212F32B7-0DAA-48D9-AD0F-4A308BD57271@illinois.edu> References: <212F32B7-0DAA-48D9-AD0F-4A308BD57271@illinois.edu> Message-ID: Hi! Thank you for your reply. In 'full zerocopy' mode: zbalance_ipc cluster-27.conf: https://gist.github.com/radoslawc/afa7293fde9ba5bc9f51640d5fc63005 node.cfg: https://gist.github.com/radoslawc/c7406452f01c14caa43c729c164d701b bro doctor output for above setup: https://gist.github.com/radoslawc/bb3e608dfa7ceca97378c26e98520fae Bro doctor states that bro binary is not linked against pfring (which is correct, as configure doesn't give this option) instead I've used pf_ring plugin from aux: Bro-PF_RING.linux-x86_64.so user at u1604:/opt/bro/lib/bro/plugins/Bro_PF_RING/lib$ ldd Bro-PF_RING.linux-x86_64.so linux-vdso.so.1 => (0x00007ffdd37f1000) libpfring.so => /usr/local/lib/libpfring.so (0x00007f85dbd5e000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f85db9dc000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f85db7c6000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f85db3fc000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f85db1df000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f85dafd7000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f85dadd3000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f85daaca000) /lib64/ld-linux-x86-64.so.2 (0x00007f85dc1dc000) I'll rebuild bro with gperftools only, thank you for pointing that out. Best regard Rado On 27 September 2017 at 23:53, Azoff, Justin S wrote: > > > On Sep 26, 2017, at 6:07 PM, radek wrote: > > > > Hi! > > > > Anyone care to share bro + pfring success story? > > > > What's the speed, what NIC, what's the configuration. > > > > I'm running bro 2.5.1 built with jemalloc and gperftools and against > pf_ring 6.6.0 with ixgbe_zc on CentOS 7.2. > > You can't be using both jemalloc and gperftools(tcmalloc).. they are both > malloc implementations. > > > In ZeroCopy mode with zbalance_ipc dividing NIC to 20 application rings > (-n 20) I'm getting each CPU core loaded at 100% and around 50% packet drop > (reported by netstats in broctl). > > Sounds like the load balancing is not working right and you are just > analyzing all of your traffic 20 times. What does your node.cfg contain? > > > When redirecting from zc to 20 dummy interfaces (zbalance_ipc -r > 0:dummy0 and so on) I'm getting around 50% load on each core and a lot less > of packet drop (10% - 15%). > > > > This is with traffic around 700 - 800 Mbit/s > > A few workers should be able to handle this load, not to mention 20.. > > > All input will be highly appreciated. > > > > Can you install bro-pkg (http://bro-package-manager. > readthedocs.io/en/stable/quickstart.html) and then do > > bro-pkg install bro-doctor --version 1.16.1 > broctl doctor.bro > > And share the results. > > ? > Justin Azoff > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170928/d4f5b7e7/attachment.html From reswob10 at gmail.com Thu Sep 28 05:55:51 2017 From: reswob10 at gmail.com (craig bowser) Date: Thu, 28 Sep 2017 08:55:51 -0400 Subject: [Bro] Bro won't start after kernel upgrade In-Reply-To: References: Message-ID: I am running Bro on Ubuntu LTS 16.04 and I just upgraded the kernel from linux-image-4.4.0-62-generic_4.4.0-62 to linux-image-4.4.0-62-generic_4.4.0-96 along with the headers (I first stopped bro before upgrading via broctl). After I rebooted, I went into broctl and tried to re-start bro, but if failed. It said it couldn?t find /run-bro. So after checking to make sure all the files were where they were supposed to be, I thought that maybe the new image install had overwritten something. So I went back into broctl and ran the install command. Then I tried to start bro again. This time when it failed to start, the diag said it couldn?t find the interface of eth0. What was weird since I knew a/ didn?t have an interface with that name and b/ had already configured bro to use the correctly named interface. After some poking around, I realized that a new node.cfg had been created at /etc/bro along with new broccoli.conf, broctl.cfg and networks.cfg (well, I don?t know if they are new since they are dated back to 2015). These are different than the originally installed and correct files located at /usr/local/bro/etc. How do I get broctl to point back to the correct cfg files? Thanks. Craig Note: please excuse any delays in response as I currently don't have constant access to Gmail -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170928/15d762b0/attachment.html From jazoff at illinois.edu Thu Sep 28 06:14:53 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 28 Sep 2017 13:14:53 +0000 Subject: [Bro] bro and pf_ring zc configuration success stories In-Reply-To: References: <212F32B7-0DAA-48D9-AD0F-4A308BD57271@illinois.edu> Message-ID: <0578457A-01D5-4D69-8D63-C02BFBA5DBAF@illinois.edu> > On Sep 28, 2017, at 5:52 AM, radek wrote: > > Hi! > Thank you for your reply. > > In 'full zerocopy' mode: > > zbalance_ipc cluster-27.conf: > > https://gist.github.com/radoslawc/afa7293fde9ba5bc9f51640d5fc63005 > > node.cfg: > > https://gist.github.com/radoslawc/c7406452f01c14caa43c729c164d701b > > bro doctor output for above setup: > > https://gist.github.com/radoslawc/bb3e608dfa7ceca97378c26e98520fae Ah.. so this is not good: error: 99.17%, 7562 out of 7625 connections are half duplex And this is not great either: ok, only 0.00%, 0 out of 13 connections appear to be duplicate It only looked at 13 connections because there were only 13 bidirectional connections in the log. I think your problem is this: interface=zc:27 That should not actually work with the pf_ring plugin.. in order to use the pf_ring plugin the interface needs to start with pf_ring:: I believe you need interface=pf_ring::zc:27 So try that and see if that fixes everything. If not, can you remove lb_procs and move to one worker for now to at least verify that that configuration works. > Bro doctor states that bro binary is not linked against pfring (which is correct, as configure doesn't give this option) instead I've used pf_ring plugin from aux: > > Bro-PF_RING.linux-x86_64.so > user at u1604:/opt/bro/lib/bro/plugins/Bro_PF_RING/lib$ ldd Bro-PF_RING.linux-x86_64.so > linux-vdso.so.1 => (0x00007ffdd37f1000) > libpfring.so => /usr/local/lib/libpfring.so (0x00007f85dbd5e000) > libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f85db9dc000) > libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f85db7c6000) > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f85db3fc000) > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f85db1df000) > librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f85dafd7000) > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f85dadd3000) > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f85daaca000) > /lib64/ld-linux-x86-64.so.2 (0x00007f85dc1dc000) Ah, that is correct. I need to have it separately check to see if bro -N lists the pf_ring plugin. If the pf_ring::zc thing fixes things, I'll fix bro-doctor to check for that. I think the check needs to be that if bro -N lists the pf_ring plugin, the interface MUST start with pf_ring:: The bro pf_ring plugin should probably do the same check.. I think there are a few issues with the pf_ring plugin. I'm working on fixing one issue that causes the plugin to be broken if you are not using ZC. > I'll rebuild bro with gperftools only, thank you for pointing that out. > > Best regard > Rado ? Justin Azoff From radoslawc at gmail.com Thu Sep 28 06:43:22 2017 From: radoslawc at gmail.com (radek) Date: Thu, 28 Sep 2017 15:43:22 +0200 Subject: [Bro] bro and pf_ring zc configuration success stories In-Reply-To: <0578457A-01D5-4D69-8D63-C02BFBA5DBAF@illinois.edu> References: <212F32B7-0DAA-48D9-AD0F-4A308BD57271@illinois.edu> <0578457A-01D5-4D69-8D63-C02BFBA5DBAF@illinois.edu> Message-ID: Hi! I've rebuilt bro with gperftools only. With worker defined like this: [worker-1] type=worker host=localhost interface=pf_ring::zc:27 lb_method=pf_ring lb_procs=20 all worker threads fail with below message: ==== stderr.log fatal error: problem with interface pf_ring::zc:27 (No such device) with zbalance_ipc stopped and using NIC device: [worker-1] type=worker host=localhost interface=pf_ring::zc:enp5s0f0 lb_method=pf_ring lb_procs=20 only one worker thread starts: [BroControl] > status Name Type Host Status Pid Started logger logger localhost running 3886 28 Sep 09:38:30 manager manager localhost running 4063 28 Sep 09:38:32 proxy-1 proxy localhost running 4384 28 Sep 09:38:34 proxy-2 proxy localhost running 4386 28 Sep 09:38:34 worker-1-1 worker localhost stopped worker-1-2 worker localhost stopped worker-1-3 worker localhost running 4751 28 Sep 09:38:36 worker-1-4 worker localhost stopped worker-1-5 worker localhost stopped worker-1-6 worker localhost stopped worker-1-7 worker localhost stopped worker-1-8 worker localhost stopped worker-1-9 worker localhost stopped worker-1-10 worker localhost stopped worker-1-11 worker localhost stopped worker-1-12 worker localhost stopped worker-1-13 worker localhost stopped worker-1-14 worker localhost stopped worker-1-15 worker localhost stopped worker-1-16 worker localhost stopped worker-1-17 worker localhost stopped worker-1-18 worker localhost stopped worker-1-19 worker localhost stopped worker-1-20 worker localhost stopped rest of them are failing with message: ==== stderr.log fatal error: problem with interface pf_ring::zc:enp5s0f0 (Bad address) Best regards Rado On 28 September 2017 at 15:14, Azoff, Justin S wrote: > > > On Sep 28, 2017, at 5:52 AM, radek wrote: > > > > Hi! > > Thank you for your reply. > > > > In 'full zerocopy' mode: > > > > zbalance_ipc cluster-27.conf: > > > > https://gist.github.com/radoslawc/afa7293fde9ba5bc9f51640d5fc63005 > > > > node.cfg: > > > > https://gist.github.com/radoslawc/c7406452f01c14caa43c729c164d701b > > > > bro doctor output for above setup: > > > > https://gist.github.com/radoslawc/bb3e608dfa7ceca97378c26e98520fae > > Ah.. so this is not good: > > error: 99.17%, 7562 out of 7625 connections are half duplex > > And this is not great either: > > ok, only 0.00%, 0 out of 13 connections appear to be duplicate > > It only looked at 13 connections because there were only 13 bidirectional > connections in the log. > > I think your problem is this: > > interface=zc:27 > > That should not actually work with the pf_ring plugin.. in order to use > the pf_ring plugin the interface needs to start with pf_ring:: I believe > you need > > interface=pf_ring::zc:27 > > So try that and see if that fixes everything. If not, can you remove > lb_procs and move to one worker for now to at least verify that that > configuration works. > > > > Bro doctor states that bro binary is not linked against pfring (which is > correct, as configure doesn't give this option) instead I've used pf_ring > plugin from aux: > > > > Bro-PF_RING.linux-x86_64.so > > user at u1604:/opt/bro/lib/bro/plugins/Bro_PF_RING/lib$ ldd > Bro-PF_RING.linux-x86_64.so > > linux-vdso.so.1 => (0x00007ffdd37f1000) > > libpfring.so => /usr/local/lib/libpfring.so (0x00007f85dbd5e000) > > libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 > (0x00007f85db9dc000) > > libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 > (0x00007f85db7c6000) > > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 > (0x00007f85db3fc000) > > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 > (0x00007f85db1df000) > > librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 > (0x00007f85dafd7000) > > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 > (0x00007f85dadd3000) > > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 > (0x00007f85daaca000) > > /lib64/ld-linux-x86-64.so.2 (0x00007f85dc1dc000) > > Ah, that is correct. I need to have it separately check to see if bro -N > lists the pf_ring plugin. > > If the pf_ring::zc thing fixes things, I'll fix bro-doctor to check for > that. > > I think the check needs to be that if bro -N lists the pf_ring plugin, the > interface MUST start with pf_ring:: > > The bro pf_ring plugin should probably do the same check.. I think there > are a few issues with the pf_ring plugin. I'm working on fixing one issue > that causes the plugin to be broken if you are not using ZC. > > > > > I'll rebuild bro with gperftools only, thank you for pointing that out. > > > > Best regard > > Rado > > ? > Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170928/60e38bf2/attachment.html From jazoff at illinois.edu Thu Sep 28 06:46:22 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 28 Sep 2017 13:46:22 +0000 Subject: [Bro] bro and pf_ring zc configuration success stories In-Reply-To: References: <212F32B7-0DAA-48D9-AD0F-4A308BD57271@illinois.edu> <0578457A-01D5-4D69-8D63-C02BFBA5DBAF@illinois.edu> Message-ID: <49EF131F-C806-4896-B721-2B50788604ED@illinois.edu> Do you have the pf_ring plugin installed. Do you see this output? $ bro -N | grep -v built-in Bro::PF_RING - Packet acquisition via PF_RING (dynamic, version 1.0) ? Justin Azoff > On Sep 28, 2017, at 9:43 AM, radek wrote: > > Hi! > > I've rebuilt bro with gperftools only. > > With worker defined like this: > > [worker-1] > type=worker > host=localhost > interface=pf_ring::zc:27 > lb_method=pf_ring > lb_procs=20 > > all worker threads fail with below message: > ==== stderr.log > > fatal error: problem with interface pf_ring::zc:27 (No such device) > > with zbalance_ipc stopped and using NIC device: > > [worker-1] > type=worker > host=localhost > interface=pf_ring::zc:enp5s0f0 > lb_method=pf_ring > lb_procs=20 > > only one worker thread starts: > > [BroControl] > status > Name Type Host Status Pid Started > logger logger localhost running 3886 28 Sep 09:38:30 > manager manager localhost running 4063 28 Sep 09:38:32 > proxy-1 proxy localhost running 4384 28 Sep 09:38:34 > proxy-2 proxy localhost running 4386 28 Sep 09:38:34 > worker-1-1 worker localhost stopped > worker-1-2 worker localhost stopped > worker-1-3 worker localhost running 4751 28 Sep 09:38:36 > worker-1-4 worker localhost stopped > worker-1-5 worker localhost stopped > worker-1-6 worker localhost stopped > worker-1-7 worker localhost stopped > worker-1-8 worker localhost stopped > worker-1-9 worker localhost stopped > worker-1-10 worker localhost stopped > worker-1-11 worker localhost stopped > worker-1-12 worker localhost stopped > worker-1-13 worker localhost stopped > worker-1-14 worker localhost stopped > worker-1-15 worker localhost stopped > worker-1-16 worker localhost stopped > worker-1-17 worker localhost stopped > worker-1-18 worker localhost stopped > worker-1-19 worker localhost stopped > worker-1-20 worker localhost stopped > > rest of them are failing with message: > > ==== stderr.log > > fatal error: problem with interface pf_ring::zc:enp5s0f0 (Bad address) > > > > Best regards > > Rado > > > On 28 September 2017 at 15:14, Azoff, Justin S wrote: > > > On Sep 28, 2017, at 5:52 AM, radek wrote: > > > > Hi! > > Thank you for your reply. > > > > In 'full zerocopy' mode: > > > > zbalance_ipc cluster-27.conf: > > > > https://gist.github.com/radoslawc/afa7293fde9ba5bc9f51640d5fc63005 > > > > node.cfg: > > > > https://gist.github.com/radoslawc/c7406452f01c14caa43c729c164d701b > > > > bro doctor output for above setup: > > > > https://gist.github.com/radoslawc/bb3e608dfa7ceca97378c26e98520fae > > Ah.. so this is not good: > > error: 99.17%, 7562 out of 7625 connections are half duplex > > And this is not great either: > > ok, only 0.00%, 0 out of 13 connections appear to be duplicate > > It only looked at 13 connections because there were only 13 bidirectional connections in the log. > > I think your problem is this: > > interface=zc:27 > > That should not actually work with the pf_ring plugin.. in order to use the pf_ring plugin the interface needs to start with pf_ring:: I believe you need > > interface=pf_ring::zc:27 > > So try that and see if that fixes everything. If not, can you remove lb_procs and move to one worker for now to at least verify that that configuration works. > > > > Bro doctor states that bro binary is not linked against pfring (which is correct, as configure doesn't give this option) instead I've used pf_ring plugin from aux: > > > > Bro-PF_RING.linux-x86_64.so > > user at u1604:/opt/bro/lib/bro/plugins/Bro_PF_RING/lib$ ldd Bro-PF_RING.linux-x86_64.so > > linux-vdso.so.1 => (0x00007ffdd37f1000) > > libpfring.so => /usr/local/lib/libpfring.so (0x00007f85dbd5e000) > > libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f85db9dc000) > > libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f85db7c6000) > > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f85db3fc000) > > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f85db1df000) > > librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f85dafd7000) > > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f85dadd3000) > > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f85daaca000) > > /lib64/ld-linux-x86-64.so.2 (0x00007f85dc1dc000) > > Ah, that is correct. I need to have it separately check to see if bro -N lists the pf_ring plugin. > > If the pf_ring::zc thing fixes things, I'll fix bro-doctor to check for that. > > I think the check needs to be that if bro -N lists the pf_ring plugin, the interface MUST start with pf_ring:: > > The bro pf_ring plugin should probably do the same check.. I think there are a few issues with the pf_ring plugin. I'm working on fixing one issue that causes the plugin to be broken if you are not using ZC. > > > > > I'll rebuild bro with gperftools only, thank you for pointing that out. > > > > Best regard > > Rado > > ? > Justin Azoff > > From radoslawc at gmail.com Thu Sep 28 06:49:29 2017 From: radoslawc at gmail.com (radek) Date: Thu, 28 Sep 2017 15:49:29 +0200 Subject: [Bro] bro and pf_ring zc configuration success stories In-Reply-To: <49EF131F-C806-4896-B721-2B50788604ED@illinois.edu> References: <212F32B7-0DAA-48D9-AD0F-4A308BD57271@illinois.edu> <0578457A-01D5-4D69-8D63-C02BFBA5DBAF@illinois.edu> <49EF131F-C806-4896-B721-2B50788604ED@illinois.edu> Message-ID: Yes, plugin is installed, root at u1604:~# /opt/bro/bin/bro -N | grep -v built-in Bro::PF_RING - Packet acquisition via PF_RING (dynamic, version 1.0) with worker definition: [worker-1] type=worker host=localhost interface=zc:27 lb_method=pf_ring lb_procs=20 I've double checked now and I'm able to start and all 20 threads are reported to be running in broctl. Best regards Rado On 28 September 2017 at 15:46, Azoff, Justin S wrote: > Do you have the pf_ring plugin installed. Do you see this output? > > $ bro -N | grep -v built-in > Bro::PF_RING - Packet acquisition via PF_RING (dynamic, version 1.0) > > > ? > Justin Azoff > > > On Sep 28, 2017, at 9:43 AM, radek wrote: > > > > Hi! > > > > I've rebuilt bro with gperftools only. > > > > With worker defined like this: > > > > [worker-1] > > type=worker > > host=localhost > > interface=pf_ring::zc:27 > > lb_method=pf_ring > > lb_procs=20 > > > > all worker threads fail with below message: > > ==== stderr.log > > > > fatal error: problem with interface pf_ring::zc:27 (No such device) > > > > with zbalance_ipc stopped and using NIC device: > > > > [worker-1] > > type=worker > > host=localhost > > interface=pf_ring::zc:enp5s0f0 > > lb_method=pf_ring > > lb_procs=20 > > > > only one worker thread starts: > > > > [BroControl] > status > > Name Type Host Status Pid Started > > logger logger localhost running 3886 28 Sep 09:38:30 > > manager manager localhost running 4063 28 Sep 09:38:32 > > proxy-1 proxy localhost running 4384 28 Sep 09:38:34 > > proxy-2 proxy localhost running 4386 28 Sep 09:38:34 > > worker-1-1 worker localhost stopped > > worker-1-2 worker localhost stopped > > worker-1-3 worker localhost running 4751 28 Sep 09:38:36 > > worker-1-4 worker localhost stopped > > worker-1-5 worker localhost stopped > > worker-1-6 worker localhost stopped > > worker-1-7 worker localhost stopped > > worker-1-8 worker localhost stopped > > worker-1-9 worker localhost stopped > > worker-1-10 worker localhost stopped > > worker-1-11 worker localhost stopped > > worker-1-12 worker localhost stopped > > worker-1-13 worker localhost stopped > > worker-1-14 worker localhost stopped > > worker-1-15 worker localhost stopped > > worker-1-16 worker localhost stopped > > worker-1-17 worker localhost stopped > > worker-1-18 worker localhost stopped > > worker-1-19 worker localhost stopped > > worker-1-20 worker localhost stopped > > > > rest of them are failing with message: > > > > ==== stderr.log > > > > fatal error: problem with interface pf_ring::zc:enp5s0f0 (Bad address) > > > > > > > > Best regards > > > > Rado > > > > > > On 28 September 2017 at 15:14, Azoff, Justin S > wrote: > > > > > On Sep 28, 2017, at 5:52 AM, radek wrote: > > > > > > Hi! > > > Thank you for your reply. > > > > > > In 'full zerocopy' mode: > > > > > > zbalance_ipc cluster-27.conf: > > > > > > https://gist.github.com/radoslawc/afa7293fde9ba5bc9f51640d5fc63005 > > > > > > node.cfg: > > > > > > https://gist.github.com/radoslawc/c7406452f01c14caa43c729c164d701b > > > > > > bro doctor output for above setup: > > > > > > https://gist.github.com/radoslawc/bb3e608dfa7ceca97378c26e98520fae > > > > Ah.. so this is not good: > > > > error: 99.17%, 7562 out of 7625 connections are half duplex > > > > And this is not great either: > > > > ok, only 0.00%, 0 out of 13 connections appear to be duplicate > > > > It only looked at 13 connections because there were only 13 > bidirectional connections in the log. > > > > I think your problem is this: > > > > interface=zc:27 > > > > That should not actually work with the pf_ring plugin.. in order to use > the pf_ring plugin the interface needs to start with pf_ring:: I believe > you need > > > > interface=pf_ring::zc:27 > > > > So try that and see if that fixes everything. If not, can you remove > lb_procs and move to one worker for now to at least verify that that > configuration works. > > > > > > > Bro doctor states that bro binary is not linked against pfring (which > is correct, as configure doesn't give this option) instead I've used > pf_ring plugin from aux: > > > > > > Bro-PF_RING.linux-x86_64.so > > > user at u1604:/opt/bro/lib/bro/plugins/Bro_PF_RING/lib$ ldd > Bro-PF_RING.linux-x86_64.so > > > linux-vdso.so.1 => (0x00007ffdd37f1000) > > > libpfring.so => /usr/local/lib/libpfring.so > (0x00007f85dbd5e000) > > > libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 > (0x00007f85db9dc000) > > > libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 > (0x00007f85db7c6000) > > > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 > (0x00007f85db3fc000) > > > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 > (0x00007f85db1df000) > > > librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 > (0x00007f85dafd7000) > > > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 > (0x00007f85dadd3000) > > > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 > (0x00007f85daaca000) > > > /lib64/ld-linux-x86-64.so.2 (0x00007f85dc1dc000) > > > > Ah, that is correct. I need to have it separately check to see if bro > -N lists the pf_ring plugin. > > > > If the pf_ring::zc thing fixes things, I'll fix bro-doctor to check for > that. > > > > I think the check needs to be that if bro -N lists the pf_ring plugin, > the interface MUST start with pf_ring:: > > > > The bro pf_ring plugin should probably do the same check.. I think there > are a few issues with the pf_ring plugin. I'm working on fixing one issue > that causes the plugin to be broken if you are not using ZC. > > > > > > > > > I'll rebuild bro with gperftools only, thank you for pointing that out. > > > > > > Best regard > > > Rado > > > > ? > > Justin Azoff > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170928/ab72902d/attachment-0001.html From jazoff at illinois.edu Thu Sep 28 07:12:25 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 28 Sep 2017 14:12:25 +0000 Subject: [Bro] bro and pf_ring zc configuration success stories In-Reply-To: References: <212F32B7-0DAA-48D9-AD0F-4A308BD57271@illinois.edu> <0578457A-01D5-4D69-8D63-C02BFBA5DBAF@illinois.edu> <49EF131F-C806-4896-B721-2B50788604ED@illinois.edu> Message-ID: <1703643E-5A0D-48AF-B194-6F55A3780BB6@illinois.edu> > On Sep 28, 2017, at 9:49 AM, radek wrote: > > Yes, plugin is installed, > root at u1604:~# /opt/bro/bin/bro -N | grep -v built-in > Bro::PF_RING - Packet acquisition via PF_RING (dynamic, version 1.0) > > with worker definition: > [worker-1] > type=worker > host=localhost > interface=zc:27 > lb_method=pf_ring > lb_procs=20 > > I've double checked now and I'm able to start and all 20 threads are reported to be running in broctl. Yes, but the plugin is only actually used when you have interface=pf_ring::... If you are using interface=zc:27 then you're just opening the zc: interfaces using libpcap. According to http://www.ntop.org/pf_ring/best-practices-for-using-bro_ids-with-pf_ring-zc-reliably/. You should run zbalance_ipc using dummy interfaces like -r 0:dummy0 -r 1:dummy1 -r 2:dummy2 -r 3:dummy3 Then you would configure bro like [worker-0] type=worker host=localhost interface=pf_ring::dummy0 pin_cpus=1 [worker-1] type=worker host=localhost interface=pf_ring::dummy1 pin_cpus=2 [worker-2] type=worker host=localhost interface=pf_ring::dummy2 pin_cpus=3 [worker-3] type=worker host=localhost interface=pf_ring::dummy3 pin_cpus=4 ? Justin Azoff From radoslawc at gmail.com Thu Sep 28 08:12:46 2017 From: radoslawc at gmail.com (radek) Date: Thu, 28 Sep 2017 17:12:46 +0200 Subject: [Bro] bro and pf_ring zc configuration success stories In-Reply-To: <1703643E-5A0D-48AF-B194-6F55A3780BB6@illinois.edu> References: <212F32B7-0DAA-48D9-AD0F-4A308BD57271@illinois.edu> <0578457A-01D5-4D69-8D63-C02BFBA5DBAF@illinois.edu> <49EF131F-C806-4896-B721-2B50788604ED@illinois.edu> <1703643E-5A0D-48AF-B194-6F55A3780BB6@illinois.edu> Message-ID: Hi! Yes this was my initial setup (with dummy interfaces), I've used worker definition like you've suggested (pf_ring::dummy{0..19}) - before I was using interface=dummy{0..19} It works, with the same traffic replayed, netstats: https://gist.github.com/radoslawc/4ca4d2f8bb0e7a2e5763d53eb31b59de so almost no drops, capstats returns nothing with interface=pf_ring::dummy{0..19}, with interface=dummy{0..19} it worked, but that's not the issue. Here's htop btw: https://imgur.com/a/99ETo My question is with using dummy interfaces, doesn't it defeat purpose of zerocopy? It has to pass packets trough kernel to dummy interface. Also I've used worker definition for 20 of them: [worker-0] type=worker host=localhost interface=pf_ring::zc:27 at 0 pin_cpus=1 and result was identical as with using: [worker-0] type=worker host=localhost interface=zc:27 lb_method=pf_ring lb_procs=20 meaning all used cores loaded at 100% and instant high packet drop: netstats from broctl: https://gist.github.com/radoslawc/c7d5c97fe443b1bed62ca4025249a342 Best regards Rado On 28 September 2017 at 16:12, Azoff, Justin S wrote: > > > On Sep 28, 2017, at 9:49 AM, radek wrote: > > > > Yes, plugin is installed, > > root at u1604:~# /opt/bro/bin/bro -N | grep -v built-in > > Bro::PF_RING - Packet acquisition via PF_RING (dynamic, version 1.0) > > > > with worker definition: > > [worker-1] > > type=worker > > host=localhost > > interface=zc:27 > > lb_method=pf_ring > > lb_procs=20 > > > > I've double checked now and I'm able to start and all 20 threads are > reported to be running in broctl. > > Yes, but the plugin is only actually used when you have > interface=pf_ring::... > > If you are using interface=zc:27 then you're just opening the zc: > interfaces using libpcap. > > According to http://www.ntop.org/pf_ring/best-practices-for-using-bro_ > ids-with-pf_ring-zc-reliably/. You should run zbalance_ipc using dummy > interfaces like > > -r 0:dummy0 -r 1:dummy1 -r 2:dummy2 -r 3:dummy3 > > Then you would configure bro like > > [worker-0] > type=worker > host=localhost > interface=pf_ring::dummy0 > pin_cpus=1 > > [worker-1] > type=worker > host=localhost > interface=pf_ring::dummy1 > pin_cpus=2 > > [worker-2] > type=worker > host=localhost > interface=pf_ring::dummy2 > pin_cpus=3 > > [worker-3] > type=worker > host=localhost > interface=pf_ring::dummy3 > pin_cpus=4 > > > > ? > Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170928/2c91475a/attachment.html From jazoff at illinois.edu Thu Sep 28 08:49:23 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 28 Sep 2017 15:49:23 +0000 Subject: [Bro] bro and pf_ring zc configuration success stories In-Reply-To: References: <212F32B7-0DAA-48D9-AD0F-4A308BD57271@illinois.edu> <0578457A-01D5-4D69-8D63-C02BFBA5DBAF@illinois.edu> <49EF131F-C806-4896-B721-2B50788604ED@illinois.edu> <1703643E-5A0D-48AF-B194-6F55A3780BB6@illinois.edu> Message-ID: <37EA893A-FCC7-467F-AC3F-ED8F2F52B785@illinois.edu> > On Sep 28, 2017, at 11:12 AM, radek wrote: > > Hi! > Yes this was my initial setup (with dummy interfaces), I've used worker definition like you've suggested (pf_ring::dummy{0..19}) - before I was using interface=dummy{0..19} > It works, with the same traffic replayed, netstats: > > https://gist.github.com/radoslawc/4ca4d2f8bb0e7a2e5763d53eb31b59de > > so almost no drops, > > capstats returns nothing with interface=pf_ring::dummy{0..19}, with interface=dummy{0..19} it worked, but that's not the issue. > > Here's htop btw: > https://imgur.com/a/99ETo > Initially you said > This is with traffic around 700 - 800 Mbit/s Did you mean 700 megabits/sec or megabytes/sec ? At 700 Mbits/sec I'd expect the load on 20 workers to be almost nothing. What model CPU is in this box? > My question is with using dummy interfaces, doesn't it defeat purpose of zerocopy? It has to pass packets trough kernel to dummy interface. It's what they recommend, so it's probably fine... Another issue I see with your configuration is that you are passing -g=2 to zbalance_ipc, which tells it to bind to core 2. You should specifically bind zbalance_ipc and bro to different cores. I'm also not sure what the -n=20,1 does and if that should just be -n=20. > Also I've used worker definition for 20 of them: > > [worker-0] > type=worker > host=localhost > interface=pf_ring::zc:27 at 0 > pin_cpus=1 > > and result was identical as with using: > > [worker-0] > type=worker > host=localhost > interface=zc:27 > lb_method=pf_ring > lb_procs=20 > Can you just run 4 workers and see how it works? You don't need 20 cores to handle 700mbit. I just checked one of our worker boxes that is currently getting around 4000mbit with 14 workers and the cpus are at about 70% ? Justin Azoff From radoslawc at gmail.com Thu Sep 28 09:33:34 2017 From: radoslawc at gmail.com (radek) Date: Thu, 28 Sep 2017 18:33:34 +0200 Subject: [Bro] bro and pf_ring zc configuration success stories In-Reply-To: <37EA893A-FCC7-467F-AC3F-ED8F2F52B785@illinois.edu> References: <212F32B7-0DAA-48D9-AD0F-4A308BD57271@illinois.edu> <0578457A-01D5-4D69-8D63-C02BFBA5DBAF@illinois.edu> <49EF131F-C806-4896-B721-2B50788604ED@illinois.edu> <1703643E-5A0D-48AF-B194-6F55A3780BB6@illinois.edu> <37EA893A-FCC7-467F-AC3F-ED8F2F52B785@illinois.edu> Message-ID: That's megabits, as reported by capstats total. cpu is: https://gist.github.com/radoslawc/376ddb061354aec40e376214f6d830cc nic is: 05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) -n=20,1 creates two applications, one with all traffic divided to 20 rings (so effectively you've got 20 interfaces to attach 20 processes/threads whatever you've got) to read packets, and another application to attach one process to consume the same traffic (for example zcount). Netstats is reporting 700 megabits per second (which I've assumed is amount of traffic bro handles and drops rest, or am I wrong?), traffic that this sensor receives is from 2 to 3.5 Gbit/s from Ixia's Breaking Point traffic generator. I've moved zbalance_ipc to core #1 and started bro with 4 workers: [worker-0] type=worker host=localhost interface=pf_ring::zc:27 at 0 pin_cpus=2 bound to cpu # 2,3,4,5 maybe 30 seconds into test: [BroControl] > capstats Interface kpps mbps (10s average) ---------------------------------------- [BroControl] > netstats worker-0: 1506615620.937720 recvd=3019521 dropped=23608217 link=3019521 worker-1: 1506615621.137749 recvd=2785062 dropped=18098988 link=2785062 worker-2: 1506615621.337785 recvd=3285288 dropped=18621993 link=3285288 worker-3: 1506615621.537699 recvd=2746902 dropped=17934732 link=2746902 all 4 cores at 100% with zc -> dummy on 4 interfaces: [worker-0] type=worker host=localhost interface=pf_ring::dummy0 pin_cpus=2 traffic per dummy device is now: ========================= Absolute Stats: [550'822 pkts total][0 pkts dropped][0.0% dropped] [550'822 pkts rcvd][172'616'004 bytes rcvd][550'799.41 pkt/sec][1'380.87 Mbit/sec] ========================= Actual Stats: [274'414 pkts rcvd][1'000.04 ms][274'402.74 pps][0.69 Gbps] ========================= and drop rate: [BroControl] > netstats worker-0: 1506616144.443402 recvd=435806 dropped=2060714 link=435806 worker-1: 1506616148.260893 recvd=769123 dropped=1168980 link=769123 worker-2: 1506616148.374244 recvd=774666 dropped=1157254 link=774666 worker-3: 1506616148.639837 recvd=768808 dropped=1180893 link=768808 Best regards Rado On 28 September 2017 at 17:49, Azoff, Justin S wrote: > > On Sep 28, 2017, at 11:12 AM, radek wrote: > > > > Hi! > > Yes this was my initial setup (with dummy interfaces), I've used worker > definition like you've suggested (pf_ring::dummy{0..19}) - before I was > using interface=dummy{0..19} > > It works, with the same traffic replayed, netstats: > > > > https://gist.github.com/radoslawc/4ca4d2f8bb0e7a2e5763d53eb31b59de > > > > so almost no drops, > > > > capstats returns nothing with interface=pf_ring::dummy{0..19}, with > interface=dummy{0..19} it worked, but that's not the issue. > > > > Here's htop btw: > > https://imgur.com/a/99ETo > > > > > Initially you said > > > This is with traffic around 700 - 800 Mbit/s > > Did you mean 700 megabits/sec or megabytes/sec ? > > At 700 Mbits/sec I'd expect the load on 20 workers to be almost nothing. > What model CPU is in this box? > > > My question is with using dummy interfaces, doesn't it defeat purpose of > zerocopy? It has to pass packets trough kernel to dummy interface. > > It's what they recommend, so it's probably fine... > > Another issue I see with your configuration is that you are passing -g=2 > to zbalance_ipc, which tells it to bind to core 2. You should specifically > bind zbalance_ipc and bro to different cores. > > I'm also not sure what the -n=20,1 does and if that should just be -n=20. > > > > > Also I've used worker definition for 20 of them: > > > > [worker-0] > > type=worker > > host=localhost > > interface=pf_ring::zc:27 at 0 > > pin_cpus=1 > > > > and result was identical as with using: > > > > [worker-0] > > type=worker > > host=localhost > > interface=zc:27 > > lb_method=pf_ring > > lb_procs=20 > > > > Can you just run 4 workers and see how it works? You don't need 20 cores > to handle 700mbit. I just checked one of our worker boxes that is > currently getting around 4000mbit with 14 workers and the cpus are at about > 70% > > > > ? > Justin Azoff > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170928/decdc530/attachment-0001.html From jazoff at illinois.edu Thu Sep 28 09:38:32 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 28 Sep 2017 16:38:32 +0000 Subject: [Bro] bro and pf_ring zc configuration success stories In-Reply-To: References: <212F32B7-0DAA-48D9-AD0F-4A308BD57271@illinois.edu> <0578457A-01D5-4D69-8D63-C02BFBA5DBAF@illinois.edu> <49EF131F-C806-4896-B721-2B50788604ED@illinois.edu> <1703643E-5A0D-48AF-B194-6F55A3780BB6@illinois.edu> <37EA893A-FCC7-467F-AC3F-ED8F2F52B785@illinois.edu> Message-ID: <553EEDF8-51CC-438D-A293-DFA6A5EEE9F1@illinois.edu> > On Sep 28, 2017, at 12:33 PM, radek wrote: > > Netstats is reporting 700 megabits per second (which I've assumed is amount of traffic bro handles and drops rest, or am I wrong?), traffic that this sensor receives is from 2 to 3.5 Gbit/s from Ixia's Breaking Point traffic generator. > Ah... if you are sending bro random traffic from a traffic generator then it is not going to work well at all. Can you configure your traffic generator to send it "real" traffic? ? Justin Azoff From radoslawc at gmail.com Thu Sep 28 09:55:16 2017 From: radoslawc at gmail.com (radek) Date: Thu, 28 Sep 2017 18:55:16 +0200 Subject: [Bro] bro and pf_ring zc configuration success stories In-Reply-To: <553EEDF8-51CC-438D-A293-DFA6A5EEE9F1@illinois.edu> References: <212F32B7-0DAA-48D9-AD0F-4A308BD57271@illinois.edu> <0578457A-01D5-4D69-8D63-C02BFBA5DBAF@illinois.edu> <49EF131F-C806-4896-B721-2B50788604ED@illinois.edu> <1703643E-5A0D-48AF-B194-6F55A3780BB6@illinois.edu> <37EA893A-FCC7-467F-AC3F-ED8F2F52B785@illinois.edu> <553EEDF8-51CC-438D-A293-DFA6A5EEE9F1@illinois.edu> Message-ID: > Can you configure your traffic generator to send it "real" traffic? that's the setup, it is even called Real-World Traffic (TM) by vendor. currently that's the only way for me to have somewhat reproducible test results in my setup. On 28 September 2017 at 18:38, Azoff, Justin S wrote: > > > On Sep 28, 2017, at 12:33 PM, radek wrote: > > > > Netstats is reporting 700 megabits per second (which I've assumed is > amount of traffic bro handles and drops rest, or am I wrong?), traffic that > this sensor receives is from 2 to 3.5 Gbit/s from Ixia's Breaking Point > traffic generator. > > > > Ah... if you are sending bro random traffic from a traffic generator then > it is not going to work well at all. > > Can you configure your traffic generator to send it "real" traffic? > > ? > Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170928/095f5d7a/attachment.html From jazoff at illinois.edu Thu Sep 28 10:13:32 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 28 Sep 2017 17:13:32 +0000 Subject: [Bro] bro and pf_ring zc configuration success stories In-Reply-To: References: <212F32B7-0DAA-48D9-AD0F-4A308BD57271@illinois.edu> <0578457A-01D5-4D69-8D63-C02BFBA5DBAF@illinois.edu> <49EF131F-C806-4896-B721-2B50788604ED@illinois.edu> <1703643E-5A0D-48AF-B194-6F55A3780BB6@illinois.edu> <37EA893A-FCC7-467F-AC3F-ED8F2F52B785@illinois.edu> <553EEDF8-51CC-438D-A293-DFA6A5EEE9F1@illinois.edu> Message-ID: > On Sep 28, 2017, at 12:55 PM, radek wrote: > > > Can you configure your traffic generator to send it "real" traffic? > > that's the setup, it is even called Real-World Traffic (TM) by vendor. currently that's the only way for me to have somewhat reproducible test results in my setup. Can you set the rate to 200mbit then for a bit? You need to get things to a point where the workers are running properly without drops. Then once the configuration looks correct and bro is logging proper connections you can start ramping the rate back up. Based on the "error: 99.17%, 7562 out of 7625 connections are half duplex" from before, nothing was working properly... and 50% drops alone wouldn't cause that. ? Justin Azoff From radoslawc at gmail.com Thu Sep 28 10:17:21 2017 From: radoslawc at gmail.com (radek) Date: Thu, 28 Sep 2017 19:17:21 +0200 Subject: [Bro] bro and pf_ring zc configuration success stories In-Reply-To: References: <212F32B7-0DAA-48D9-AD0F-4A308BD57271@illinois.edu> <0578457A-01D5-4D69-8D63-C02BFBA5DBAF@illinois.edu> <49EF131F-C806-4896-B721-2B50788604ED@illinois.edu> <1703643E-5A0D-48AF-B194-6F55A3780BB6@illinois.edu> <37EA893A-FCC7-467F-AC3F-ED8F2F52B785@illinois.edu> <553EEDF8-51CC-438D-A293-DFA6A5EEE9F1@illinois.edu> Message-ID: Will do, I'll get back with results tomorrow as my day ended. Thanks for your help so far. On 28 September 2017 at 19:13, Azoff, Justin S wrote: > > > On Sep 28, 2017, at 12:55 PM, radek wrote: > > > > > Can you configure your traffic generator to send it "real" traffic? > > > > that's the setup, it is even called Real-World Traffic (TM) by vendor. > currently that's the only way for me to have somewhat reproducible test > results in my setup. > > > Can you set the rate to 200mbit then for a bit? You need to get things to > a point where the workers are running properly without drops. > > Then once the configuration looks correct and bro is logging proper > connections you can start ramping the rate back up. > > Based on the "error: 99.17%, 7562 out of 7625 connections are half duplex" > from before, nothing was working properly... and 50% drops alone wouldn't > cause that. > > ? > Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170928/ba35e176/attachment.html From rsreese at gmail.com Thu Sep 28 10:57:16 2017 From: rsreese at gmail.com (Stephen Reese) Date: Thu, 28 Sep 2017 13:57:16 -0400 Subject: [Bro] lack of seen_bytes Message-ID: I have been experiencing hash misses so to speak with PE files due to lack of seen_bytes verse total_bytes. Is this indication of a performance problem which the sensor is overwhelmed therefore cannot parse the entire file? e.g. I have a file that's 300832 in which seen_bytes consistently matches total_bytes and then a hash is provided. Another file is 774200 total_bytes but the seen_bytes usually does not amount to the total_bytes (sometimes it does). From dnthayer at illinois.edu Thu Sep 28 12:41:29 2017 From: dnthayer at illinois.edu (Daniel Thayer) Date: Thu, 28 Sep 2017 14:41:29 -0500 Subject: [Bro] Bro won't start after kernel upgrade In-Reply-To: References: Message-ID: <7d1e0cce-e909-efa8-38a8-10e2589c8357@illinois.edu> Did you check if there is more than one broctl installed on your system? On 9/28/17 7:55 AM, craig bowser wrote: > I am running Bro on Ubuntu LTS 16.04 and I just upgraded the kernel from > linux-image-4.4.0-62-generic_4.4.0-62 to > linux-image-4.4.0-62-generic_4.4.0-96 along with the headers (I first > stopped bro before upgrading via broctl). > After I rebooted, I went into broctl and tried to re-start bro, but if > failed.? It said it couldn?t find /run-bro.? So after checking to make > sure all the files were where they were supposed to be, I thought that > maybe the new image install had overwritten something.? So I went back > into broctl and ran the install command.? Then I tried to start bro > again.? This time when it failed to start, the diag said it couldn?t > find the interface of eth0.? What was weird since I knew a/ didn?t have > an interface with that name and b/ had already configured bro to use the > correctly named interface.? After some poking around, I realized that a > new node.cfg had been created at /etc/bro along with new broccoli.conf, > broctl.cfg and networks.cfg (well, I don?t know if they are new since > they are dated back to 2015).? These are different than the originally > installed and correct files located at /usr/local/bro/etc. > How do I get broctl to point back to the correct cfg files? > Thanks. > > Craig > > Note: please excuse any delays in response as I currently don't have > constant access to Gmail > From franky.meier.1 at gmx.de Fri Sep 29 00:22:24 2017 From: franky.meier.1 at gmx.de (Frank Meier) Date: Fri, 29 Sep 2017 09:22:24 +0200 Subject: [Bro] optimize running bro from PCAPs / advantage of cluster mode In-Reply-To: References: <20170922131506.3883ecfa@NB181106> Message-ID: <20170929092224.5f61c79e@NB181106> Hi Mike, thanks for your reply! On Fri, 22 Sep 2017 10:33:55 -0500 Mike Dopheide wrote: > > I would argue that using Bro's cluster configuration ends up making > it a lot easier for you in the long run. > > 1) To start, you only have one logger node so all of your logs will > be in one place and you don't have to worry about trying to > consolidate them later. This is true, but you could also argue that you might get better throughput, if multiple loggers write to for example a cluster of elastic or kafka servers. > 4) I question not needing to have shared tables, but I also don't > know your environment and your end goals. That's how most of the > scan detection scripts work, by counting the number of anomalies over > time across all of your traffic. If an attacker scans you ten times > which are split across ten bro nodes that aren't communicating with > each other, you may miss it. A lot of the malware detection policies > also look for the inbound connection and then a separate outbound > connection. I should clarify, that I run bro mainly as a source of meta data about pcaps. As all data is from the past, scanner detection is no priority. > Also, using broctl puts you in the same place as a lot of other other > installations so it's easier for people on this list to help > troubleshoot. > That's a good point. My original question still stands: Are there any parsers which combine the information seen by different workers in different flows? Martin > -Dop > > > > On Fri, Sep 22, 2017 at 6:15 AM, Frank Meier > wrote: > > > Hello! > > > > In contrast to the normal use case I run Bro mostly from pcaps. When > > huge amounts of data (~20 TB) have to be processed, bro in > > standalone mode becomes a real bottleneck. So I thought about using > > the bro cluster mode. > > > > In the past I thought, the bro workers would communicate with each > > other, so when for example one worker sees upstream and the other > > downstream, they would combine the information to one log. Seth > > told me at BroCon, that Bro needs to be fed complete streams. To do > > this some kind of load balancer is needed in front of bro. > > > > When I need to split the flows with a load balancer anyway, is > > there any advantage of running bro in cluster mode at all? I do not > > need any shared data like tables. Are there any parsers which > > combine the information seen by different workers in different > > flows? > > > > If cluster mode has no added value in my case, I could just load > > balance my pcaps to independent bro instances which would make my > > setup much easier. > > > > Have a nice weekend! > > > > Franky > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > From seth at corelight.com Fri Sep 29 07:14:52 2017 From: seth at corelight.com (Seth Hall) Date: Fri, 29 Sep 2017 10:14:52 -0400 Subject: [Bro] lack of seen_bytes In-Reply-To: References: Message-ID: <806B5595-8D6B-44FC-9024-B28B0F58768B@corelight.com> On 28 Sep 2017, at 13:57, Stephen Reese wrote: > I have been experiencing hash misses so to speak with PE files due to > lack of seen_bytes verse total_bytes. Is this indication of a > performance problem which the sensor is overwhelmed therefore cannot > parse the entire file? Those numbers can be really tricky. If a protocol indicates how much data it's going to transfer or how big the file is, Bro will know the total_bytes. There are a number of cases where total_bytes isn't even known. It's also possible that Bro is tracking files that aren't even being transferred in their entirety. Over SMB, you will very frequently see portions of files transferred where Bro never even had an opportunity to see the whole file. What may help next is if you look at the conn log for the connections where you are seeing files transferred to see if the missed_bytes on that connection is greater than zero. That should tell you if there was any packet loss in the connection which could also cause some bizarre behavior as you're describing. If you could provide a conn log entry and files log entry where you are seeing the problem, that would be the fastest way to figure out what happening (please just mask out ip addresses). .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From radoslawc at gmail.com Fri Sep 29 08:44:03 2017 From: radoslawc at gmail.com (radek) Date: Fri, 29 Sep 2017 17:44:03 +0200 Subject: [Bro] bro and pf_ring zc configuration success stories In-Reply-To: References: <212F32B7-0DAA-48D9-AD0F-4A308BD57271@illinois.edu> <0578457A-01D5-4D69-8D63-C02BFBA5DBAF@illinois.edu> <49EF131F-C806-4896-B721-2B50788604ED@illinois.edu> <1703643E-5A0D-48AF-B194-6F55A3780BB6@illinois.edu> <37EA893A-FCC7-467F-AC3F-ED8F2F52B785@illinois.edu> <553EEDF8-51CC-438D-A293-DFA6A5EEE9F1@illinois.edu> Message-ID: Hi! I'm back with results. I've created new test and ran 200 Mbits, 600 Mbit, 1Gbit then went all in with 8 Gbits. 1. You were right with traffic generator, previous test had some parameters changed and was doing something funky with TCP. I've removed this and above issues are to some extent gone. 2. With zbalance_ipc -n 20 and worker definition: [worker-0] type=worker host=localhost interface=pf_ring::zc:27 at 0 pin_cpus=1 I'm able to process 4.5 Gbit/s with all 20 cores loaded at 60 - 70 % with minimal drop at bro [BroControl] > netstats worker-0: 1506695586.298096 recvd=5465310 dropped=30118 link=5465310 worker-1: 1506695586.497686 recvd=5438281 dropped=9041 link=5438281 worker-2: 1506695586.701504 recvd=5498208 dropped=8756 link=5498208 worker-3: 1506695586.901398 recvd=5457893 dropped=9326 link=5457893 worker-4: 1506695587.101722 recvd=5472315 dropped=8877 link=5472315 worker-5: 1506695587.301448 recvd=5541810 dropped=10604 link=5541810 worker-6: 1506695587.501405 recvd=5556953 dropped=2022 link=5556953 worker-7: 1506695587.705590 recvd=5508997 dropped=2149 link=5508997 worker-8: 1506695587.905592 recvd=5526052 dropped=1955 link=5526052 worker-9: 1506695588.105445 recvd=5506942 dropped=2751 link=5506942 worker-10: 1506695588.305863 recvd=5597609 dropped=7534 link=5597609 worker-11: 1506695588.505499 recvd=5550657 dropped=4975 link=5550657 worker-12: 1506695588.705426 recvd=5578005 dropped=1152 link=5578005 worker-13: 1506695588.905554 recvd=5541178 dropped=90 link=5541178 worker-14: 1506695589.109446 recvd=5561273 dropped=3568 link=5561273 worker-15: 1506695589.309585 recvd=5552211 dropped=2850 link=5552211 worker-16: 1506695589.509799 recvd=5524173 dropped=7896 link=5524173 worker-17: 1506695589.709838 recvd=5565320 dropped=10923 link=5565320 worker-18: 1506695589.910352 recvd=5632122 dropped=9169 link=5632122 worker-19: 1506695590.113969 recvd=5603647 dropped=10448 link=5603647 this drop occured at the beginning of test and stayed like this until end (20 minutes) with zbalance_ipc - n 20 -r 0:dummy0 and so on for 20 workers defined like this: [worker-0] type=worker host=localhost interface=pf_ring::dummy0 pin_cpus=1 I can process at around 3 Gbit/s and around 36 % of packets are dropped at zbalance_ipc ingress (ixgbe NIC) (so it seems that bottleneck here is zc - > dummy packets processing) Core designated for zbalance_ipc is loaded 100% during test , I'll look into it next. So so far so good. I'll be posting updates on my findings I'm very grateful for your help. Thank you. Best regards Rado On 28 September 2017 at 19:17, radek wrote: > Will do, I'll get back with results tomorrow as my day ended. Thanks for > your help so far. > > On 28 September 2017 at 19:13, Azoff, Justin S > wrote: > >> >> > On Sep 28, 2017, at 12:55 PM, radek wrote: >> > >> > > Can you configure your traffic generator to send it "real" traffic? >> > >> > that's the setup, it is even called Real-World Traffic (TM) by vendor. >> currently that's the only way for me to have somewhat reproducible test >> results in my setup. >> >> >> Can you set the rate to 200mbit then for a bit? You need to get things >> to a point where the workers are running properly without drops. >> >> Then once the configuration looks correct and bro is logging proper >> connections you can start ramping the rate back up. >> >> Based on the "error: 99.17%, 7562 out of 7625 connections are half >> duplex" from before, nothing was working properly... and 50% drops alone >> wouldn't cause that. >> >> ? >> Justin Azoff >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170929/395a64ea/attachment-0001.html From jdopheid at illinois.edu Fri Sep 29 11:54:42 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Fri, 29 Sep 2017 18:54:42 +0000 Subject: [Bro] BroCon 2017 slides and videos are posted! Message-ID: The BroCon 2017 list of approved videos have been posted to our YouTube channel. You can view the playlist here: https://www.youtube.com/watch?v=b6lrNaKRgmA&list=PL2EYTX8UVCMjx7DdqfPEOjCmih3nrtqxv BroCon slides (and links to videos) are available on our agenda page: https://www.bro.org/community/brocon2017.html#agenda If you do not see a link to a slideshow or video it has not been approved for release. Thanks again to all the speakers, attendees, and sponsors who made this another successful BroCon. We hope to see you again, either in the audience or on stage, next year. ------ Jeannette Dopheide Sr. Education, Outreach, and Training Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From dopheide at gmail.com Fri Sep 29 12:14:32 2017 From: dopheide at gmail.com (Mike Dopheide) Date: Fri, 29 Sep 2017 14:14:32 -0500 Subject: [Bro] optimize running bro from PCAPs / advantage of cluster mode In-Reply-To: <20170929092224.5f61c79e@NB181106> References: <20170922131506.3883ecfa@NB181106> <20170929092224.5f61c79e@NB181106> Message-ID: > My original question still stands: Are there any parsers which combine > the information seen by different workers in different flows? > Policies yes, parsers I'm not sure, but I don't believe so. -Dop -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170929/ee8fb7aa/attachment.html