From bogus@does.not.exist.com Tue Dec 16 23:17:26 2014 From: bogus@does.not.exist.com () Date: Wed, 17 Dec 2014 07:17:26 -0000 Subject: No subject Message-ID: lot of packets - and given 4 seconds elapsed time, this is happening at a pretty modest load, about 6Kpps, even when you're not capturing full packets. That all boils down to this certainly looking like a problem with the packet filter itself rather than Bro. Vern From bogus@does.not.exist.com Tue Dec 16 23:17:26 2014 From: bogus@does.not.exist.com () Date: Wed, 17 Dec 2014 07:17:26 -0000 Subject: No subject Message-ID: The SACK option is advisory, in that, while it notifies the data sender that the data receiver has received the indicated segments, the data receiver is permitted to later discard data which have been reported in a SACK option. - Vern From bogus@does.not.exist.com Tue Dec 16 23:17:26 2014 From: bogus@does.not.exist.com () Date: Wed, 17 Dec 2014 07:17:26 -0000 Subject: No subject Message-ID: files, our CPU utilization is generally 99%, and the packet filter seems to be dropping a high percentage of packets. We are going to re-design our Bro architecture and are seeking recommendations for hardware and OS. We are currently considering running FreeBSD 6.0 instead of RedHat if that will provide better performance. We are also considering splitting the collecting and initial log creation from the subsequent log processing we perform to retain data in our database. We suspect we will need stronger machines for the initial collection/log creation than for the subsequent processing, which is primarily parsing the various log files. We are looking at Sun Fire X4100 servers with our existing SK-9844 cards for the "collector" systems. However, it appears that we cannot run FreeBSD on the X4100 machines due a lack of support for the LSI SAS (serial attached SCSI) HBA. So, we would instead keep RedHat. As an alternative, we could use Sun Fire X2100 servers with SK-9E92 cards for the collectors, running FreeBSD, as long as these would provide sufficient performance. We may run 4 collector machines, each listening to its own tap. We were also thinking of using the Sun Fire X2100s for the secondary log parsing. I suppose our questions are: 1) Which OS should we use - FreeBSD or RedHat? 2) Can anyone recommend using the Sun Fire X2100s or X4100s? 3) Does anyone have advice regarding the Syskonnect SK-9844 or SK-9E92 cards? 4) Is it reasonable to assume that the most intensive part of this process is the initial collection and analysis by Bro which results in the various Bro log files? 5) Are there other hardware or OS recommendations? I'm sure I omitting something, but this is a good start. Thanks in advance for your advice! Joncarlo Ruggieri University of CA, Davis Data Center & Client Services jruggieri at ucdavis.edu From blackhole.em at gmail.com Mon Dec 1 12:56:31 2014 From: blackhole.em at gmail.com (Joe Blow) Date: Mon, 1 Dec 2014 15:56:31 -0500 Subject: [Bro] SSL Decrypt + SMTP Message-ID: Hey Bro folks, I was hoping someone has done this before, but we're trying to log our SMTP connections which we decrypt. In the packets, you'll see STARTTLS, but everything after that is clear text. Does anyone have a good mechanism for allowing the indexing to go past the STARTTLS? I was thinking about changing/removing this, but wanted to consult the list before going too far: // If an TLS transaction has been initiated, forward to child and abort. if ( state == SMTP_IN_TLS ) { ForwardStream(length, line, orig); return; } The underlying packets are the same as regular SMTP, they're just being decrypted by an appliance. Any ideas? Cheers, JB -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141201/bfca6738/attachment.html From johanna at icir.org Mon Dec 1 13:28:21 2014 From: johanna at icir.org (Johanna Amann) Date: Mon, 1 Dec 2014 13:28:21 -0800 Subject: [Bro] SSL Decrypt + SMTP In-Reply-To: References: Message-ID: <20141201212821.GA21772@user149.sys.icsi.berkeley.edu> Hello Joe, > I was hoping someone has done this before, but we're trying to log our SMTP > connections which we decrypt. In the packets, you'll see STARTTLS, but > everything after that is clear text. > > Does anyone have a good mechanism for allowing the indexing to go past the > STARTTLS? I was thinking about changing/removing this, but wanted to > consult the list before going too far: > > // If an TLS transaction has been initiated, forward to child and abort. if ( > state == SMTP_IN_TLS ) { ForwardStream(length, line, orig); return; } > The underlying packets are the same as regular SMTP, they're just being > decrypted by an appliance. The easy fix should be to remove the call to StartTLS() in line 768. This will prevent the state to be set to SMTP_IN_TLS, and also keep the support analyzers that the call removes if a TLS connection is started. Johanna From slawek at rozbicki.eu Wed Dec 3 00:52:45 2014 From: slawek at rozbicki.eu (=?UTF-8?Q?S=C5=82awek_Rozbicki?=) Date: Wed, 3 Dec 2014 09:52:45 +0100 Subject: [Bro] string to (huge) int/count convert Message-ID: Hi, In order to make IBAN checksum checker I need to make modulo 97 operation on a very high value that cannot be stored in int or count type. Is there any way I could make this possible? there is no 'long' type in bro? This is exactly what I'm tring to accomplish: http://www.ibancalculator.com/calculation.html -- Pozdrawiam S?awomir Rozbicki From jsiwek at illinois.edu Wed Dec 3 08:03:25 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Wed, 3 Dec 2014 16:03:25 +0000 Subject: [Bro] string to (huge) int/count convert In-Reply-To: References: Message-ID: > On Dec 3, 2014, at 2:52 AM, S?awek Rozbicki wrote: > > In order to make IBAN checksum checker I need to make modulo 97 > operation on a very high value that cannot be stored in int or count > type. Is there any way I could make this possible? there is no 'long' > type in bro? > > This is exactly what I'm tring to accomplish: > http://www.ibancalculator.com/calculation.html The maximum integer storage in Bro is the unsigned 64-bit ?count? type. The numbers in the example seem to overflow that, so maybe an approach to try is to find/port an arbitrary precision modulus algorithm (e.g. one that operates on a string of digits and thus avoids overflowing fixed size storage). It may be easier to add such a thing as a Built-In Function (using C/C++) rather than entirely in the Bro scripting language. - Jon From seth at icir.org Wed Dec 3 08:39:01 2014 From: seth at icir.org (Seth Hall) Date: Wed, 3 Dec 2014 11:39:01 -0500 Subject: [Bro] string to (huge) int/count convert In-Reply-To: References: Message-ID: > On Dec 3, 2014, at 11:03 AM, Siwek, Jon wrote: > It may be easier to add such a thing as a Built-In Function (using C/C++) rather than entirely in the Bro scripting language. That?s probably the right approach. That said, I?m playing around now with writing an arbitrary precision library with at least enough functionality to validate IBAN numbers completely as a Bro script. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jdopheid at illinois.edu Wed Dec 3 12:37:36 2014 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Wed, 3 Dec 2014 20:37:36 +0000 Subject: [Bro] BroCon '15: August 4th - 6th at MIT Message-ID: We are happy to announce that BroCon ?15 will occur on Tuesday, August 4th ? Thursday, August 6th, at the campus of the Massachusetts Institute of Technology in Cambridge, Massachusetts. Updates will be posted to the event page: https://www.bro.org/community/brocon2015.html Our registration link will be available soon. Interested in sponsoring BroCon? Contact us at info at bro.org Thank you for your continued support, and see you in August! Regards, The Bro Development Team From hhoffman at ip-solutions.net Wed Dec 3 12:50:01 2014 From: hhoffman at ip-solutions.net (Harry Hoffman) Date: Wed, 03 Dec 2014 15:50:01 -0500 Subject: [Bro] BroCon '15: August 4th - 6th at MIT In-Reply-To: References: Message-ID: <547F7779.1000505@ip-solutions.net> Yay!!! On 12/3/14 3:37 PM, Dopheide, Jeannette M wrote: > We are happy to announce that BroCon ?15 will occur on Tuesday, August 4th > ? Thursday, August 6th, at the campus of the Massachusetts Institute of > Technology in Cambridge, Massachusetts. > > Updates will be posted to the event page: > https://www.bro.org/community/brocon2015.html > > > Our registration link will be available soon. > > Interested in sponsoring BroCon? Contact us at info at bro.org > > Thank you for your continued support, and see you in August! > > Regards, > The Bro Development Team > > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From anthony.kasza at gmail.com Wed Dec 3 13:18:23 2014 From: anthony.kasza at gmail.com (anthony kasza) Date: Wed, 3 Dec 2014 13:18:23 -0800 Subject: [Bro] BroCon '15: August 4th - 6th at MIT In-Reply-To: <547F7779.1000505@ip-solutions.net> References: <547F7779.1000505@ip-solutions.net> Message-ID: Not to be confused with the Limerick games & anime convention. -AK On Dec 3, 2014 1:00 PM, "Harry Hoffman" wrote: > Yay!!! > > On 12/3/14 3:37 PM, Dopheide, Jeannette M wrote: > > We are happy to announce that BroCon ?15 will occur on Tuesday, August > 4th > > ? Thursday, August 6th, at the campus of the Massachusetts Institute of > > Technology in Cambridge, Massachusetts. > > > > Updates will be posted to the event page: > > https://www.bro.org/community/brocon2015.html > > > > > > Our registration link will be available soon. > > > > Interested in sponsoring BroCon? Contact us at info at bro.org > > > > Thank you for your continued support, and see you in August! > > > > Regards, > > The Bro Development Team > > > > > > > > > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141203/8e631a13/attachment.html From jlay at slave-tothe-box.net Wed Dec 3 14:52:17 2014 From: jlay at slave-tothe-box.net (jlay) Date: Wed, 03 Dec 2014 15:52:17 -0700 Subject: [Bro] Scripts Message-ID: Not mine, but found online (thank you Cyber News RSS feed). https://github.com/sooshie/bro-scripts/tree/master/exploitkit James From jonathon.s.wright at gmail.com Wed Dec 3 17:58:02 2014 From: jonathon.s.wright at gmail.com (Jonathon Wright) Date: Wed, 3 Dec 2014 15:58:02 -1000 Subject: [Bro] File Extraction Message-ID: Hey Bro members, Some questions about File Extraction for Bro on my Red Hat 6.5 server. File Extraction 1. I've configured Bro appropriately to extract "exe" mime types from the HTTP protocol. It works great. However, the "files.log" only contains MD5 and SHA1 entries for some of the files, not all of them. How do I fix this so that all of the extracted files have the MD5 and SHA1 entries? 2. I have analysts that need access to the files (/var/data/bro/extracted), but I've noticed that bro creates the files with random permissions, either 644 or 600... so they can only access the ones with 644. How do I ensure bro extracts the file with the 644 permission set on all of them? (see below example) -rw-r--r--. 1 root root 703736 Nov 7 04:29 HTTP-FzedfU1k233I0Kiwn8.exe.dead -rw-------. 1 root root 358799 Nov 5 04:07 HTTP-FzFPDF3EF77DEUSjdf.exe.dead -rw-------. 1 root root 26121658 Nov 6 03:17 HTTP-FzhwqG33dNtUHZraZ4.exe.dead -rw-------. 1 root root 249856 Nov 5 00:00 HTTP-FZi4XxyXiaoBquRu.exe.dead -rw-r--r--. 1 root root 332536 Nov 28 14:21 HTTP-FZikQY3r8a7gXtLlee.exe.dead -rw-r--r--. 1 root root 24306 Nov 12 05:02 HTTP-FzjIxe2MR9Uj8S8j27.exe.dead -rw-------. 1 root root 94568 Nov 6 04:00 HTTP-FzJjxg23F3HPqtRbC2.exe.dead 3. Is there a way to tell bro to run as a different user / group other than root? I didn't see any options for it in the bro --help. I would assume I would have to give broctl and bro binaries / modules the ownership and executable rights by another user, then have bro start up as that new user, but wanted to see if there was an easier way. Otherwise I'd have to change the default install configuration each time I upgrade. Thanks! JW -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141203/43b19606/attachment.html From sconzo at visiblerisk.com Wed Dec 3 21:37:36 2014 From: sconzo at visiblerisk.com (Mike Sconzo) Date: Wed, 3 Dec 2014 23:37:36 -0600 Subject: [Bro] Scripts In-Reply-To: References: Message-ID: Glad you like the script! All updates/suggestions welcome. On Wed, Dec 3, 2014 at 4:52 PM, jlay wrote: > Not mine, but found online (thank you Cyber News RSS feed). > > https://github.com/sooshie/bro-scripts/tree/master/exploitkit > > James > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- cat ~/.bash_history > documentation.txt From jdonnelly at dyn.com Thu Dec 4 05:28:51 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Thu, 4 Dec 2014 07:28:51 -0600 Subject: [Bro] File Extraction In-Reply-To: References: Message-ID: I doubt. Bro needs to run as root because it captures network traffic using libpcap. On Wed, Dec 3, 2014 at 7:58 PM, Jonathon Wright wrote: > Hey Bro members, > > Some questions about File Extraction for Bro on my Red Hat 6.5 server. > > File Extraction > 1. I've configured Bro appropriately to extract "exe" mime types from the > HTTP protocol. It works great. However, the "files.log" only contains MD5 > and SHA1 entries for some of the files, not all of them. How do I fix this > so that all of the extracted files have the MD5 and SHA1 entries? > > 2. I have analysts that need access to the files > (/var/data/bro/extracted), but I've noticed that bro creates the files with > random permissions, either 644 or 600... so they can only access the ones > with 644. How do I ensure bro extracts the file with the 644 permission set > on all of them? (see below example) > > -rw-r--r--. 1 root root 703736 Nov 7 04:29 > HTTP-FzedfU1k233I0Kiwn8.exe.dead > -rw-------. 1 root root 358799 Nov 5 04:07 > HTTP-FzFPDF3EF77DEUSjdf.exe.dead > -rw-------. 1 root root 26121658 Nov 6 03:17 > HTTP-FzhwqG33dNtUHZraZ4.exe.dead > -rw-------. 1 root root 249856 Nov 5 00:00 > HTTP-FZi4XxyXiaoBquRu.exe.dead > -rw-r--r--. 1 root root 332536 Nov 28 14:21 > HTTP-FZikQY3r8a7gXtLlee.exe.dead > -rw-r--r--. 1 root root 24306 Nov 12 05:02 > HTTP-FzjIxe2MR9Uj8S8j27.exe.dead > -rw-------. 1 root root 94568 Nov 6 04:00 > HTTP-FzJjxg23F3HPqtRbC2.exe.dead > 3. Is there a way to tell bro to run as a different user / group other > than root? I didn't see any options for it in the bro --help. I would > assume I would have to give broctl and bro binaries / modules the ownership > and executable rights by another user, then have bro start up as that new > user, but wanted to see if there was an easier way. Otherwise I'd have to > change the default install configuration each time I upgrade. > > Thanks! > JW > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141204/1e494bdd/attachment.html From seth at icir.org Thu Dec 4 05:48:29 2014 From: seth at icir.org (Seth Hall) Date: Thu, 4 Dec 2014 08:48:29 -0500 Subject: [Bro] File Extraction In-Reply-To: References: Message-ID: <33913C4B-A819-4311-BB43-97970ACC0CF1@icir.org> > On Dec 3, 2014, at 8:58 PM, Jonathon Wright wrote: > > 1. I've configured Bro appropriately to extract "exe" mime types from the HTTP protocol. It works great. However, the "files.log" only contains MD5 and SHA1 entries for some of the files, not all of them. How do I fix this so that all of the extracted files have the MD5 and SHA1 entries? Hm. Could you show a line from files.log where a file was extracted but you did get hashes? I suspect this is because the file ended up having trouble being transferred which will cause the hash analyzers to stop hashing (could be due to a packet drop even). > 2. I have analysts that need access to the files (/var/data/bro/extracted), but I've noticed that bro creates the files with random permissions, either 644 or 600... so they can only access the ones with 644. How do I ensure bro extracts the file with the 644 permission set on all of them? (see below example) This is weird. I haven?t seen Bro creating extracted files with different permissions before. Are these all dropped directly into the location by Bro or copied there later? All files from the same Bro process or from multiple Bro processes? > 3. Is there a way to tell bro to run as a different user / group other than root? I didn't see any options for it in the bro --help. I would assume I would have to give broctl and bro binaries / modules the ownership and executable rights by another user, then have bro start up as that new user, but wanted to see if there was an easier way. Otherwise I'd have to change the default install configuration each time I upgrade. Unfortunately, this is an area that hasn?t seen much attention for too long. We?ve been meaning to spend some time adding privilege dropping to Bro but it hasn?t happened yet. This would make Bro similar to many other tools that are run by root, but then drop their privileges to another user account at runtime, and in our case this would happen after Bro opens any interfaces that it needs to sniff on. Is there anyone out there in Bro-land that is interested in digging into the core enough to add privilege dropping? :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From marcus at randomhack.org Thu Dec 4 06:05:47 2014 From: marcus at randomhack.org (Marcus LaFerrera) Date: Thu, 4 Dec 2014 09:05:47 -0500 Subject: [Bro] File Extraction In-Reply-To: <33913C4B-A819-4311-BB43-97970ACC0CF1@icir.org> References: <33913C4B-A819-4311-BB43-97970ACC0CF1@icir.org> Message-ID: As for running bro as non-root, I've always created a bro user/group, chown'd the bro directory and files to that user and group, and use setcap as below. setcap cap_net_raw,cap_net_admin=eip /path/to/bro/bin Though not privilege dropping, it will still give you the added security and peace of mind that you aren't running as root. I've been doing this for several years now and never had any issues with it. Albeit, this has always been on a linux based server. ? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141204/6db31d9c/attachment.html From seth at icir.org Thu Dec 4 06:14:00 2014 From: seth at icir.org (Seth Hall) Date: Thu, 4 Dec 2014 09:14:00 -0500 Subject: [Bro] File Extraction In-Reply-To: References: <33913C4B-A819-4311-BB43-97970ACC0CF1@icir.org> Message-ID: <9CF83991-90D7-4A81-980C-6A89A4476D1E@icir.org> > On Dec 4, 2014, at 9:05 AM, Marcus LaFerrera wrote: > > Though not privilege dropping, it will still give you the added security and peace of mind that you aren't running as root. I've been doing this for several years now and never had any issues with it. Albeit, this has always been on a linux based server. That?s definitely the alternative. I used to do the same thing on FreeBSD, but on there you make changes to the devd.conf so that your lower-privileged user has permission to open the /dev/bpf* devices. Ultimately I think that using a privilege dropping mechanism is probably the easiest way for most people because it doesn?t require any special configuration to the OS to make it work. You would just configure broctl to run Bro as a certain user or run Bro with a certain flag (depending on if you use BroControl or run Bro directly). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From tiburcesotohou at yahoo.fr Thu Dec 4 08:47:39 2014 From: tiburcesotohou at yahoo.fr (SOTOHOU Osince Tiburce) Date: Thu, 4 Dec 2014 16:47:39 +0000 (UTC) Subject: [Bro] communication script between bro instances Message-ID: <1441702650.5599267.1417711659053.JavaMail.yahoo@jws11159.mail.ir2.yahoo.com> Hi;I have installed and configured two instances of bro on two different LAN.each instance monitors a LAN on which it is deployed. Now, i want to share some events between these instances.I would like to know what kind of events i can share between them.Could you give me an exemple of script to share important events?What kind of events can i consider as important to share? thanks for your response. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141204/c6f6e993/attachment.html From tiburcesotohou at yahoo.fr Thu Dec 4 08:47:40 2014 From: tiburcesotohou at yahoo.fr (SOTOHOU Osince Tiburce) Date: Thu, 4 Dec 2014 16:47:40 +0000 (UTC) Subject: [Bro] communication script between bro instances Message-ID: <1898022768.5588475.1417711660723.JavaMail.yahoo@jws11107.mail.ir2.yahoo.com> Hi;I have installed and configured two instances of bro on two different LAN.each instance monitors a LAN on which it is deployed. Now, i want to share some events between these instances.I would like to know what kind of events i can share between them.Could you give me an exemple of script to share important events?What kind of events can i consider as important to share? thanks for your response. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141204/39be907a/attachment.html From jonathon.s.wright at gmail.com Thu Dec 4 12:44:35 2014 From: jonathon.s.wright at gmail.com (Jonathon Wright) Date: Thu, 4 Dec 2014 10:44:35 -1000 Subject: [Bro] File Extraction In-Reply-To: <33913C4B-A819-4311-BB43-97970ACC0CF1@icir.org> References: <33913C4B-A819-4311-BB43-97970ACC0CF1@icir.org> Message-ID: 1.) Here is a sample of the files.log where there are entries that have no MD5 or SHA1: 1417723198.683744 FAvAKu2jFR7eGbRsH9 23.235.40.193 143.45.2.4 CYzgUS3Y6uS9dk3YM9 HTTP 0 MD5,SHA1 image/png - 2.677866 F F 60072 7 731951 131224 0 F - - - - - 1417723198.684489 FBn4gNzWGTRPkoTm3 23.235.40.193 143.45.97.186 CHjRXfnjhGudwL2re HTTP 0 MD5,SHA1 image/png - 2.677447 F F 59842 3 731951 133528 0 F - - - - - 1417723201.725784 FTegh14C2bs0OBTXil 143.45.170.74 184.75.174.163 Ch6xdv40tNxq4gWOb7 HTTP 0 MD5,SHA1 text/plain - 0.000000 T T 44 44 0 0 F - bc443e340953993a069985719f1cac76 8b1c00142caf938a917ef5cd04a2977993c7edd3 - - Packet drop might be an issue, this sensor is hit pretty hard, and we see roughly 1-5% packets dropped on tcpdump for example. One thing that sticks out now as I look at this log is that it says "image/png" and "text/plain" for mime_type. I configured the script I wrote to only extract exe's: #make a lookup hash of mimetypes to file extensions global ext_map: table[string] of string = { [?application/x-dosexec?] = ?exe?, [?text/plain?] = ?txt?, [?image/jpeg?] = ?jpg?, [?text/html?] = ?html?, } &default=??; #create an event to handle new files event file_new(f: fa_file) { #ignore files with no mimetype and get only exe?s if (! f?$mime_type || f$mime_type != ?application/x-dosexec?) return; #otherwise it passes and continues on, so it IS an exe file #lets capture it and put it in a directory, name it,& analyze it local ext = ??; if ( f?$mime_type ) ext = ext_map[f$mime_type]; #add .dead to the end to ensure file cannot be executed accidentally local fname = fmt(?/var/data/bro/extracted/%s-%s.%s.dead?, f$source, f$id, ext); Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=fname]); } 2.) All are sent to the directory by bro (based on script above that has the file_new event handler). As far as the number of bro processes... I honestly don't know. I start bro via broctl via cron: @reboot root sleep 90 ; /opt/bro/bin/broctl start However, if I look at the process table, it shows 3 processes: ]# ps -ef | grep -i bro root 2733 1 0 Nov13 ? 00:00:00 bash /opt/bro/share/broctl/scripts/run-bro -1 -i em1 -U .status -p broctl -p broctl-live -p standalone -p local -p bro local.bro broctl broctl/standalone broctl/auto root 3425 2733 39 Nov13 ? 8-06:46:02 /opt/bro/bin/bro -i em1 -U .status -p broctl -p broctl-live -p standalone -p local -p bro local.bro broctl broctl/standalone broctl/auto root 4073 3425 13 Nov13 ? 2-19:47:05 /opt/bro/bin/bro -i em1 -U .status -p broctl -p broctl-live -p standalone -p local -p bro local.bro broctl broctl/standalone broctl/auto 3.) Sounds good, I'll keep an eye out. Let me know on the other 2, thanks! JW On Thu, Dec 4, 2014 at 3:48 AM, Seth Hall wrote: > > > On Dec 3, 2014, at 8:58 PM, Jonathon Wright > wrote: > > > > 1. I've configured Bro appropriately to extract "exe" mime types from > the HTTP protocol. It works great. However, the "files.log" only contains > MD5 and SHA1 entries for some of the files, not all of them. How do I fix > this so that all of the extracted files have the MD5 and SHA1 entries? > > Hm. Could you show a line from files.log where a file was extracted but > you did get hashes? I suspect this is because the file ended up having > trouble being transferred which will cause the hash analyzers to stop > hashing (could be due to a packet drop even). > > > 2. I have analysts that need access to the files > (/var/data/bro/extracted), but I've noticed that bro creates the files with > random permissions, either 644 or 600... so they can only access the ones > with 644. How do I ensure bro extracts the file with the 644 permission set > on all of them? (see below example) > > This is weird. I haven?t seen Bro creating extracted files with different > permissions before. Are these all dropped directly into the location by > Bro or copied there later? All files from the same Bro process or from > multiple Bro processes? > > > 3. Is there a way to tell bro to run as a different user / group other > than root? I didn't see any options for it in the bro --help. I would > assume I would have to give broctl and bro binaries / modules the ownership > and executable rights by another user, then have bro start up as that new > user, but wanted to see if there was an easier way. Otherwise I'd have to > change the default install configuration each time I upgrade. > > Unfortunately, this is an area that hasn?t seen much attention for too > long. We?ve been meaning to spend some time adding privilege dropping to > Bro but it hasn?t happened yet. This would make Bro similar to many other > tools that are run by root, but then drop their privileges to another user > account at runtime, and in our case this would happen after Bro opens any > interfaces that it needs to sniff on. > > Is there anyone out there in Bro-land that is interested in digging into > the core enough to add privilege dropping? :) > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141204/5b47886d/attachment.html From jonathon.s.wright at gmail.com Thu Dec 4 12:45:42 2014 From: jonathon.s.wright at gmail.com (Jonathon Wright) Date: Thu, 4 Dec 2014 10:45:42 -1000 Subject: [Bro] File Extraction In-Reply-To: References: <33913C4B-A819-4311-BB43-97970ACC0CF1@icir.org> Message-ID: Very interesting Marcus, I'll take a look at the setcap option, thanks! On Thu, Dec 4, 2014 at 4:05 AM, Marcus LaFerrera wrote: > As for running bro as non-root, I've always created a bro user/group, > chown'd the bro directory and files to that user and group, and use setcap > as below. > > setcap cap_net_raw,cap_net_admin=eip /path/to/bro/bin > > Though not privilege dropping, it will still give you the added security > and peace of mind that you aren't running as root. I've been doing this for > several years now and never had any issues with it. Albeit, this has always > been on a linux based server. > ? > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141204/8d30017f/attachment.html From openjaf at gmail.com Sun Dec 7 09:02:16 2014 From: openjaf at gmail.com (James Feister) Date: Sun, 7 Dec 2014 12:02:16 -0500 Subject: [Bro] broccoli not processing events Message-ID: I am creating an application based on the broccoli test/broconn source. It seems when running the broconn example and the application I am writing using bro versions 2.0 and 2.1, I can subscribe to "new_connection" events and process them in a callback; everything is great. But when using bro version 2.3 there is no processing of the events sending them to the callback. This goes for the broconn example and my app. The ping example does work for version 2.3 which is using custom events. I can capture the client server traffic and see the 'new_connection' events being passed back to the client but the client side code seems to not do anything for bro v2.3, this goes for my app and the broconn example. I was not able to try bro v2.2, when executing bro it would display a bunch of warnings for magic files and the following message, then quit "internal error: can't load magic file /usr/local/bro/share/bro/magic: could not find any valid magic files!" This is being on fedora 20 and arch linux, tried all versions of bro (v2.0,2.1,2.2,2.3) using the src tarballs and the git repo. -Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141207/0ff622b9/attachment.html From bkellogg at dresser-rand.com Mon Dec 8 08:21:05 2014 From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN)) Date: Mon, 8 Dec 2014 16:21:05 +0000 Subject: [Bro] Scripts In-Reply-To: References: Message-ID: Excellent, and thanks -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of jlay Sent: Wednesday, December 03, 2014 5:52 PM To: bro at bro.org Subject: [Bro] Scripts Not mine, but found online (thank you Cyber News RSS feed). https://github.com/sooshie/bro-scripts/tree/master/exploitkit James _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jsiwek at illinois.edu Mon Dec 8 08:56:30 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Mon, 8 Dec 2014 16:56:30 +0000 Subject: [Bro] broccoli not processing events In-Reply-To: References: Message-ID: > On Dec 7, 2014, at 11:02 AM, James Feister wrote: > > I am creating an application based on the broccoli test/broconn source. It seems when running the broconn example and the application I am writing using bro versions 2.0 and 2.1, I can subscribe to "new_connection" events and process them in a callback; everything is great. But when using bro version 2.3 there is no processing of the events sending them to the callback. This goes for the broconn example and my app. The ?connection? record in Bro has evolved over time to include data types that Broccoli clients can no longer process (I think it?s ?opaque? types that are now problematic, but may be misremembering). Broccoli silently (unless debug output is turned out) fails to invoke callbacks for events that use those data types. As a general suggestion, try writing custom Bro events to pick and choose particular things to send to Broccoli clients. E.g.: global my_custom_new_connection: event(cid: conn_id); event new_connection(c: connection) { event my_custom_new_connection(c$id); } Then have the Broccoli client register a callback for ?my_custom_new_connection? instead of ?new_connection?. I think this should workaround the bug in Broccoli as well as be more efficient (unless the Broccoli client actually intended to make use of every last bit of data in the connection record). Also, the new library being developed to replace Broccoli/Bro communication, called Broker, is planned to require being explicit about the content of remotely communicated messages. So writing code in this way should make it easier to adapt to the new library when available. - Jon From robin at broala.com Mon Dec 8 09:15:51 2014 From: robin at broala.com (Robin Sommer) Date: Mon, 8 Dec 2014 09:15:51 -0800 Subject: [Bro] [Announcement] Professional Bro Support Message-ID: <20141208171551.GA93583@icir.org> As some might have seen already, Broala has recently announced providing professional Bro support in the form of annual subscription services. With Bro's proliferation across corporate and government environments, we are excited to offer enterprises a reliable point of contact for any phase of their Bro deployment: planning, validation, and full production. Please see our web page at http://www.broala.com for more information. If you'd like more specifics, feel free to contact us. Robin From openjaf at gmail.com Mon Dec 8 10:55:21 2014 From: openjaf at gmail.com (James Feister) Date: Mon, 8 Dec 2014 13:55:21 -0500 Subject: [Bro] broccoli not processing events In-Reply-To: References: Message-ID: On Mon, Dec 8, 2014 at 11:56 AM, Siwek, Jon wrote: > > > On Dec 7, 2014, at 11:02 AM, James Feister wrote: > > > > I am creating an application based on the broccoli test/broconn source. > It seems when running the broconn example and the application I am writing > using bro versions 2.0 and 2.1, I can subscribe to "new_connection" events > and process them in a callback; everything is great. But when using bro > version 2.3 there is no processing of the events sending them to the > callback. This goes for the broconn example and my app. > > The ?connection? record in Bro has evolved over time to include data types > that Broccoli clients can no longer process (I think it?s ?opaque? types > that are now problematic, but may be misremembering). Broccoli silently > (unless debug output is turned out) fails to invoke callbacks for events > that use those data types. > > As a general suggestion, try writing custom Bro events to pick and choose > particular things to send to Broccoli clients. E.g.: > > global my_custom_new_connection: event(cid: conn_id); > > event new_connection(c: connection) > { > event my_custom_new_connection(c$id); > } > > Then have the Broccoli client register a callback for > ?my_custom_new_connection? instead of ?new_connection?. > > I think this should workaround the bug in Broccoli as well as be more > efficient (unless the Broccoli client actually intended to make use of > every last bit of data in the connection record). Also, the new library > being developed to replace Broccoli/Bro communication, called Broker, is > planned to require being explicit about the content of remotely > communicated messages. So writing code in this way should make it easier > to adapt to the new library when available. > > - Jon Thank you, I will go that path with my development. - Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141208/1fd6414c/attachment.html From brianallen at wustl.edu Mon Dec 8 14:56:01 2014 From: brianallen at wustl.edu (Allen, Brian) Date: Mon, 8 Dec 2014 22:56:01 +0000 Subject: [Bro] building a new bro server In-Reply-To: References: Message-ID: Hi All- I currently have a server running BRO, and we are seeing a lot of packet loss. I am getting quotes for a new server to replace it, and I wanted to run some of the options by this group to see what would be better. Current server specs: -2 Processors, 8 cores each at 2.4GHz, so 16 total. We run 14 bro processes, one per core. And they run at 100% utilization all the time. -128G memory -Intel IXGBE 10Gig network card with pfring We are seeing 3-4 Gig traffic pretty much constantly, and we spike to 5 Gig. The bro packet-loss file shows 30+% packet loss most of the time, but during the early morning hours, when traffic drops considerably it will fall to 0.01%. For one test, we used a bpf filter to block all traffic going to bro except for a one /24 subnet of campus traffic for about 15 minutes and the packet-loss dropped to 0.01%. So we think our processors are too few and too slow to handle this amount of bandwidth. Our question as we get a quote to buy a new box is, which is more important for BRO, having the roughly same number of cores but get faster ones, or get more cores at the same or slower speed? I'm looking at the following two Dell server options, although I can adjust this to add other better possibilities: Option1: -Intel Xeon E5-2699, two processors, 18 cores each at 2.3GHz for 36 total -256Gig RAM -Intel IXGBE 10Gig network card with pfring Option2: -Intel Xeon E5-2687 two processors, 10 cores each at 3.1GHz for 20 total -256Gig RAM -Intel IXGBE 10Gig network card with pfring I'm assuming the first option would be much better but I've never researched this to know for sure, or how much better it would actually be. I think the difference in price is around $2,400. I'd like to get one box to handle our bandwidth as it grows over the next couple years, take the current underpowered box and use it is a BRO test box/elastic search server, and build the infrastructure to move to a BRO cluster in a couple years. Right now a single box would be better for space issues. I would be really interested to talk to other companies/universities who are running bro in the 3-7 Gig bandwidth range right now so I can see what hardware works for you. Thanks for your help, Brian Allen, CISSP Information Security Manager Washington University brianallen at wustl.edu 314-935-5380 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141208/e2fe9577/attachment.html From gfaulkner.nsm at gmail.com Mon Dec 8 16:52:39 2014 From: gfaulkner.nsm at gmail.com (Gary Faulkner) Date: Mon, 08 Dec 2014 18:52:39 -0600 Subject: [Bro] building a new bro server In-Reply-To: References: Message-ID: <548647D7.9050305@gmail.com> A couple thoughts that might help the list better understand your topology/situation. 1. Are the manager and/or proxies on the same host? 2. What are you using to determine packet loss? (ex. Bro capture loss script, broctl netstat, pf_ring counters, etc) 3. Are you running PF_RING using any of the enhanced drivers (DNA/ZC) and/or zero copy scripts(Libzero/ZC)? 4. Are you pinning your worker processes to individual cores (via node.cfg) or are you letting the OS handle things? I saw a marked improvement in average loss as measured by the bro capture loss script simply by pinning CPU cores on a server very similar to yours with similar traffic per host. Bursty traffic, and mega-flows, will still cause higher loss levels for individual workers at times though. Also, if you are running the manager and proxies on the same host they could be competing for the same cores that one or more workers are running on. Running htop might give you an idea of workers are being bounced between cores (if not pinned) as well as whether other processes are clobbering one or more of the cores your workers are on. Either could be an issue with workers running at 100% CPU usage. Regards, Gary On 12/8/2014 4:56 PM, Allen, Brian wrote: > Hi All- > I currently have a server running BRO, and we are seeing a lot of > packet loss. I am getting quotes for a new server to replace it, and > I wanted to run some of the options by this group to see what would be > better. > > Current server specs: > > -2 Processors, 8 cores each at 2.4GHz, so 16 total. We run 14 bro > processes, one per core. And they run at 100% utilization all the time. > -128G memory > -Intel IXGBE 10Gig network card with pfring > > We are seeing 3-4 Gig traffic pretty much constantly, and we spike to > 5 Gig. The bro packet-loss file shows 30+% packet loss most of the > time, but during the early morning hours, when traffic drops > considerably it will fall to 0.01%. > > For one test, we used a bpf filter to block all traffic going to bro > except for a one /24 subnet of campus traffic for about 15 minutes and > the packet-loss dropped to 0.01%. > > So we think our processors are too few and too slow to handle this > amount of bandwidth. > > Our question as we get a quote to buy a new box is, which is more > important for BRO, having the roughly same number of cores but get > faster ones, or get more cores at the same or slower speed? > > I?m looking at the following two Dell server options, although I can > adjust this to add other better possibilities: > > Option1: > -Intel Xeon E5-2699, two processors, 18 cores each at 2.3GHz for 36 total > -256Gig RAM > -Intel IXGBE 10Gig network card with pfring > > Option2: > -Intel Xeon E5-2687 two processors, 10 cores each at 3.1GHz for 20 total > -256Gig RAM > -Intel IXGBE 10Gig network card with pfring > > I?m assuming the first option would be much better but I?ve never > researched this to know for sure, or how much better it would actually > be. I think the difference in price is around $2,400. > > I?d like to get one box to handle our bandwidth as it grows over the > next couple years, take the current underpowered box and use it is a > BRO test box/elastic search server, and build the infrastructure to > move to a BRO cluster in a couple years. Right now a single box would > be better for space issues. > > I would be really interested to talk to other companies/universities > who are running bro in the 3-7 Gig bandwidth range right now so I can > see what hardware works for you. > > Thanks for your help, > Brian Allen, CISSP > Information Security Manager > Washington University > brianallen at wustl.edu > 314-935-5380 > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141208/4b1e28f0/attachment.html From brianallen at wustl.edu Mon Dec 8 19:57:45 2014 From: brianallen at wustl.edu (Allen, Brian) Date: Tue, 9 Dec 2014 03:57:45 +0000 Subject: [Bro] building a new bro server In-Reply-To: <548647D7.9050305@gmail.com> References: <548647D7.9050305@gmail.com> Message-ID: Good questions and suggestions. 1. The manager and the workers are all on the same server. 2. We have looked at all of those metrics, but the bro capture loss file is what we use most. That is the one saying we have 30+% packet loss. 3. We got a license and tried PF_Ring with DNA/Zero copy but it didn't make a noticeable difference. 4. We do use the node.cfg file to pin the 14 worker processes to the individual cores. That leaves 2 free cores for OS/System tasks. We saw a huge improvement when we went from 16Gig RAM to 128Gig RAM. (That one was pretty obvious so we did that first). We also saw improvement when we pinned the processes to the cores. Thanks, -Brian From: Gary Faulkner > Date: Monday, December 8, 2014 at 6:52 PM To: Brian Allen > Cc: Bro-Mailinglist > Subject: Re: [Bro] building a new bro server A couple thoughts that might help the list better understand your topology/situation. 1. Are the manager and/or proxies on the same host? 2. What are you using to determine packet loss? (ex. Bro capture loss script, broctl netstat, pf_ring counters, etc) 3. Are you running PF_RING using any of the enhanced drivers (DNA/ZC) and/or zero copy scripts(Libzero/ZC)? 4. Are you pinning your worker processes to individual cores (via node.cfg) or are you letting the OS handle things? I saw a marked improvement in average loss as measured by the bro capture loss script simply by pinning CPU cores on a server very similar to yours with similar traffic per host. Bursty traffic, and mega-flows, will still cause higher loss levels for individual workers at times though. Also, if you are running the manager and proxies on the same host they could be competing for the same cores that one or more workers are running on. Running htop might give you an idea of workers are being bounced between cores (if not pinned) as well as whether other processes are clobbering one or more of the cores your workers are on. Either could be an issue with workers running at 100% CPU usage. Regards, Gary On 12/8/2014 4:56 PM, Allen, Brian wrote: Hi All- I currently have a server running BRO, and we are seeing a lot of packet loss. I am getting quotes for a new server to replace it, and I wanted to run some of the options by this group to see what would be better. Current server specs: -2 Processors, 8 cores each at 2.4GHz, so 16 total. We run 14 bro processes, one per core. And they run at 100% utilization all the time. -128G memory -Intel IXGBE 10Gig network card with pfring We are seeing 3-4 Gig traffic pretty much constantly, and we spike to 5 Gig. The bro packet-loss file shows 30+% packet loss most of the time, but during the early morning hours, when traffic drops considerably it will fall to 0.01%. For one test, we used a bpf filter to block all traffic going to bro except for a one /24 subnet of campus traffic for about 15 minutes and the packet-loss dropped to 0.01%. So we think our processors are too few and too slow to handle this amount of bandwidth. Our question as we get a quote to buy a new box is, which is more important for BRO, having the roughly same number of cores but get faster ones, or get more cores at the same or slower speed? I'm looking at the following two Dell server options, although I can adjust this to add other better possibilities: Option1: -Intel Xeon E5-2699, two processors, 18 cores each at 2.3GHz for 36 total -256Gig RAM -Intel IXGBE 10Gig network card with pfring Option2: -Intel Xeon E5-2687 two processors, 10 cores each at 3.1GHz for 20 total -256Gig RAM -Intel IXGBE 10Gig network card with pfring I'm assuming the first option would be much better but I've never researched this to know for sure, or how much better it would actually be. I think the difference in price is around $2,400. I'd like to get one box to handle our bandwidth as it grows over the next couple years, take the current underpowered box and use it is a BRO test box/elastic search server, and build the infrastructure to move to a BRO cluster in a couple years. Right now a single box would be better for space issues. I would be really interested to talk to other companies/universities who are running bro in the 3-7 Gig bandwidth range right now so I can see what hardware works for you. Thanks for your help, Brian Allen, CISSP Information Security Manager Washington University brianallen at wustl.edu 314-935-5380 _______________________________________________ Bro mailing list bro at bro-ids.orghttp://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141209/76c14a97/attachment.html From michal at rsbac.org Mon Dec 8 20:39:07 2014 From: michal at rsbac.org (Michal Purzynski) Date: Mon, 08 Dec 2014 20:39:07 -0800 Subject: [Bro] building a new bro server In-Reply-To: References: <548647D7.9050305@gmail.com> Message-ID: <54867CEB.6060108@rsbac.org> I run a cluster with 12 workers on 16 physical cores. 64GB RAM each (mandatory!!). I pin workers to physical cores, otherwise OS will end up assigning two different workers to the same physical cores but different virtual (HT) and they end up competing for the same resources. Are you looking at "netstats" or capture loss from a script (the one, that does heuristics with ACKs)? If you later and you can see a lot of drops and the number is more or less even accross the workers, than the loss might be happening before Bro. You said that using DNA didn't make a difference, so that might be worth looking into. How are you mirroring traffic - using taps or switch? On 08/12/14 19:57, Allen, Brian wrote: > Good questions and suggestions. > > 1. The manager and the workers are all on the same server. > 2. We have looked at all of those metrics, but the bro capture loss > file is what we use most. That is the one saying we have 30+% > packet loss. > 3. We got a license and tried PF_Ring with DNA/Zero copy but it > didn?t make a noticeable difference. > 4. We do use the node.cfg file to pin the 14 worker processes to the > individual cores. That leaves 2 free cores for OS/System tasks. > > We saw a huge improvement when we went from 16Gig RAM to 128Gig RAM. > (That one was pretty obvious so we did that first). We also saw > improvement when we pinned the processes to the cores. > > Thanks, > -Brian > > > From: Gary Faulkner > > Date: Monday, December 8, 2014 at 6:52 PM > To: Brian Allen > > Cc: Bro-Mailinglist > > Subject: Re: [Bro] building a new bro server > > A couple thoughts that might help the list better understand your > topology/situation. > > 1. Are the manager and/or proxies on the same host? > 2. What are you using to determine packet loss? (ex. Bro capture loss > script, broctl netstat, pf_ring counters, etc) > 3. Are you running PF_RING using any of the enhanced drivers (DNA/ZC) > and/or zero copy scripts(Libzero/ZC)? > 4. Are you pinning your worker processes to individual cores (via > node.cfg) or are you letting the OS handle things? > > I saw a marked improvement in average loss as measured by the bro > capture loss script simply by pinning CPU cores on a server very > similar to yours with similar traffic per host. Bursty traffic, and > mega-flows, will still cause higher loss levels for individual workers > at times though. Also, if you are running the manager and proxies on > the same host they could be competing for the same cores that one or > more workers are running on. Running htop might give you an idea of > workers are being bounced between cores (if not pinned) as well as > whether other processes are clobbering one or more of the cores your > workers are on. Either could be an issue with workers running at 100% > CPU usage. > > Regards, > Gary > > On 12/8/2014 4:56 PM, Allen, Brian wrote: >> Hi All- >> I currently have a server running BRO, and we are seeing a lot of >> packet loss. I am getting quotes for a new server to replace it, and >> I wanted to run some of the options by this group to see what would >> be better. >> >> Current server specs: >> >> -2 Processors, 8 cores each at 2.4GHz, so 16 total. We run 14 bro >> processes, one per core. And they run at 100% utilization all the time. >> -128G memory >> -Intel IXGBE 10Gig network card with pfring >> >> We are seeing 3-4 Gig traffic pretty much constantly, and we spike to >> 5 Gig. The bro packet-loss file shows 30+% packet loss most of the >> time, but during the early morning hours, when traffic drops >> considerably it will fall to 0.01%. >> >> For one test, we used a bpf filter to block all traffic going to bro >> except for a one /24 subnet of campus traffic for about 15 minutes >> and the packet-loss dropped to 0.01%. >> >> So we think our processors are too few and too slow to handle this >> amount of bandwidth. >> >> Our question as we get a quote to buy a new box is, which is more >> important for BRO, having the roughly same number of cores but get >> faster ones, or get more cores at the same or slower speed? >> >> I?m looking at the following two Dell server options, although I can >> adjust this to add other better possibilities: >> >> Option1: >> -Intel Xeon E5-2699, two processors, 18 cores each at 2.3GHz for 36 total >> -256Gig RAM >> -Intel IXGBE 10Gig network card with pfring >> >> Option2: >> -Intel Xeon E5-2687 two processors, 10 cores each at 3.1GHz for 20 total >> -256Gig RAM >> -Intel IXGBE 10Gig network card with pfring >> >> I?m assuming the first option would be much better but I?ve never >> researched this to know for sure, or how much better it would >> actually be. I think the difference in price is around $2,400. >> >> I?d like to get one box to handle our bandwidth as it grows over the >> next couple years, take the current underpowered box and use it is a >> BRO test box/elastic search server, and build the infrastructure to >> move to a BRO cluster in a couple years. Right now a single box >> would be better for space issues. >> >> I would be really interested to talk to other companies/universities >> who are running bro in the 3-7 Gig bandwidth range right now so I can >> see what hardware works for you. >> >> Thanks for your help, >> Brian Allen, CISSP >> Information Security Manager >> Washington University >> brianallen at wustl.edu >> 314-935-5380 >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.orghttp://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141208/3c1df469/attachment.html From babbitmail at gmail.com Tue Dec 9 01:38:09 2014 From: babbitmail at gmail.com (BabbitMail) Date: Tue, 9 Dec 2014 09:38:09 +0000 Subject: [Bro] Yara analyser Message-ID: <8CDFF69F-3AB2-4FDD-8B70-1B9FBB6C49E1@gmail.com> Hi all, This is not really a question, more just to see if anybody had any strong opinions, or good suggestions about how to integrate yard into bro. I read something on this mailing list about integrating bro with yara, and hadn?t seen anything since so I?ve developed a yara analyser for bro. https://github.com/hempnall/broyara . The code seems to work well for small pcaps - but I wondered about memory exhaustion using std::ostringstream to store files in larger deployments. I just wondered whether this was something that you might consider including in the bro source - i?d be happy to tidy it up if there was enough enthusiasm. This only took me about three hours - (thanks to Bro?s extensibility and Yara?s excellent docs) Regards James -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141209/007a9a5d/attachment.html From seth at icir.org Tue Dec 9 06:43:41 2014 From: seth at icir.org (Seth Hall) Date: Tue, 9 Dec 2014 09:43:41 -0500 Subject: [Bro] Yara analyser In-Reply-To: <8CDFF69F-3AB2-4FDD-8B70-1B9FBB6C49E1@gmail.com> References: <8CDFF69F-3AB2-4FDD-8B70-1B9FBB6C49E1@gmail.com> Message-ID: <9F0AAFA0-BCCE-42C6-976E-5CA62898B99C@icir.org> > On Dec 9, 2014, at 4:38 AM, BabbitMail wrote: > > The code seems to work well for small pcaps - but I wondered about memory exhaustion using std::ostringstream to store files in larger deployments. I just wondered whether this was something that you might consider including in the bro source - i?d be happy to tidy it up if there was enough enthusiasm. Unfortunately that?s unlikely to work well on live traffic and it could be abused easily. I?ve actually spent quite a bit of time on making some API updates to Yara to introduce an incremental API and I have a Yara analyzer laying around somewhere that uses the incremental API (it only took about an hour to create the analyzer after I made the API extension in Yara). I?ve been in contact a bit with Victor Alvarez about getting an incremental analysis API into Yara and I showed him my code. He responded well but he hasn?t merged my code or update his to add an incremental API yet. I?ll follow up with him again soon to get his thoughts on it. In case anyone here wants to take a look at what I?ve done, you can see my Yara branch with an incremental API here: https://github.com/sethhall/yara/tree/incremental-parsing .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jdopheid at illinois.edu Tue Dec 9 08:29:58 2014 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Tue, 9 Dec 2014 16:29:58 +0000 Subject: [Bro] building a new bro server In-Reply-To: <54867CEB.6060108@rsbac.org> Message-ID: Archiving to thread on behalf of Jusin Azoff: On 12/9/14, 10:23 AM, "Azoff, Justin S" > wrote: The 2699 is definitely faster. You can see some benchmarks here http://www.cpubenchmark.net/high_end_cpus.html One thing I know about intel CPUS is that the model numbers are mostly BS and hard to tell which should be faster, unless only the last two digits differ.. so for E5-26XX compared to E5-26YY, if YY is greater than XX, it's a better CPU. -- - Justin Azoff ------ Jeannette M. Dopheide Bro Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From: Michal Purzynski > Date: Mon, 8 Dec 2014 20:39:07 -0800 To: > Subject: Re: [Bro] building a new bro server I run a cluster with 12 workers on 16 physical cores. 64GB RAM each (mandatory!!). I pin workers to physical cores, otherwise OS will end up assigning two different workers to the same physical cores but different virtual (HT) and they end up competing for the same resources. Are you looking at "netstats" or capture loss from a script (the one, that does heuristics with ACKs)? If you later and you can see a lot of drops and the number is more or less even accross the workers, than the loss might be happening before Bro. You said that using DNA didn't make a difference, so that might be worth looking into. How are you mirroring traffic - using taps or switch? On 08/12/14 19:57, Allen, Brian wrote: Good questions and suggestions. 1. The manager and the workers are all on the same server. 2. We have looked at all of those metrics, but the bro capture loss file is what we use most. That is the one saying we have 30+% packet loss. 3. We got a license and tried PF_Ring with DNA/Zero copy but it didn?t make a noticeable difference. 4. We do use the node.cfg file to pin the 14 worker processes to the individual cores. That leaves 2 free cores for OS/System tasks. We saw a huge improvement when we went from 16Gig RAM to 128Gig RAM. (That one was pretty obvious so we did that first). We also saw improvement when we pinned the processes to the cores. Thanks, -Brian From: Gary Faulkner > Date: Monday, December 8, 2014 at 6:52 PM To: Brian Allen > Cc: Bro-Mailinglist > Subject: Re: [Bro] building a new bro server A couple thoughts that might help the list better understand your topology/situation. 1. Are the manager and/or proxies on the same host? 2. What are you using to determine packet loss? (ex. Bro capture loss script, broctl netstat, pf_ring counters, etc) 3. Are you running PF_RING using any of the enhanced drivers (DNA/ZC) and/or zero copy scripts(Libzero/ZC)? 4. Are you pinning your worker processes to individual cores (via node.cfg) or are you letting the OS handle things? I saw a marked improvement in average loss as measured by the bro capture loss script simply by pinning CPU cores on a server very similar to yours with similar traffic per host. Bursty traffic, and mega-flows, will still cause higher loss levels for individual workers at times though. Also, if you are running the manager and proxies on the same host they could be competing for the same cores that one or more workers are running on. Running htop might give you an idea of workers are being bounced between cores (if not pinned) as well as whether other processes are clobbering one or more of the cores your workers are on. Either could be an issue with workers running at 100% CPU usage. Regards, Gary On 12/8/2014 4:56 PM, Allen, Brian wrote: Hi All- I currently have a server running BRO, and we are seeing a lot of packet loss. I am getting quotes for a new server to replace it, and I wanted to run some of the options by this group to see what would be better. Current server specs: -2 Processors, 8 cores each at 2.4GHz, so 16 total. We run 14 bro processes, one per core. And they run at 100% utilization all the time. -128G memory -Intel IXGBE 10Gig network card with pfring We are seeing 3-4 Gig traffic pretty much constantly, and we spike to 5 Gig. The bro packet-loss file shows 30+% packet loss most of the time, but during the early morning hours, when traffic drops considerably it will fall to 0.01%. For one test, we used a bpf filter to block all traffic going to bro except for a one /24 subnet of campus traffic for about 15 minutes and the packet-loss dropped to 0.01%. So we think our processors are too few and too slow to handle this amount of bandwidth. Our question as we get a quote to buy a new box is, which is more important for BRO, having the roughly same number of cores but get faster ones, or get more cores at the same or slower speed? I?m looking at the following two Dell server options, although I can adjust this to add other better possibilities: Option1: -Intel Xeon E5-2699, two processors, 18 cores each at 2.3GHz for 36 total -256Gig RAM -Intel IXGBE 10Gig network card with pfring Option2: -Intel Xeon E5-2687 two processors, 10 cores each at 3.1GHz for 20 total -256Gig RAM -Intel IXGBE 10Gig network card with pfring I?m assuming the first option would be much better but I?ve never researched this to know for sure, or how much better it would actually be. I think the difference in price is around $2,400. I?d like to get one box to handle our bandwidth as it grows over the next couple years, take the current underpowered box and use it is a BRO test box/elastic search server, and build the infrastructure to move to a BRO cluster in a couple years. Right now a single box would be better for space issues. I would be really interested to talk to other companies/universities who are running bro in the 3-7 Gig bandwidth range right now so I can see what hardware works for you. Thanks for your help, Brian Allen, CISSP Information Security Manager Washington University brianallen at wustl.edu 314-935-5380 _______________________________________________ Bro mailing list bro at bro-ids.orghttp://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.orghttp://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141209/c9b02da0/attachment.html From bkellogg at dresser-rand.com Tue Dec 9 08:50:26 2014 From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN)) Date: Tue, 9 Dec 2014 16:50:26 +0000 Subject: [Bro] Scripts FastFlux Message-ID: https://github.com/sooshie/bro-scripts/ I've initiated a pull request for jlay's rep on github to include the ffluxDNS.bro script. This is an update of Seth Hall's original script to work in Bro 2.3. It is also attached. I replaced the custom log file with notices. More work needs to be done with it, but it is working in my environment. Has anyone written a script to detect oversized DNS requests? If not, which event/hook would be the best method for checking for these? thanks -------------- next part -------------- A non-text attachment was scrubbed... Name: ffluxDNS.bro Type: application/octet-stream Size: 4158 bytes Desc: ffluxDNS.bro Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141209/445231dd/attachment.obj From gl89 at CORNELL.EDU Tue Dec 9 09:02:30 2014 From: gl89 at CORNELL.EDU (Glenn Forbes Fleming Larratt) Date: Tue, 9 Dec 2014 12:02:30 -0500 (EST) Subject: [Bro] Followup Re: "hash-all-files", er, doesn't? In-Reply-To: <7C8F3BB6-C454-4B97-A838-7A99EC4427DD@icir.org> References: <845DB223-51BF-4DF6-B3AD-A2094DF877AC@icir.org> <7C8F3BB6-C454-4B97-A838-7A99EC4427DD@icir.org> Message-ID: Dear Seth and everyone, Thanks for the assistance. It turned out that the configuration was seemingly correct, it's just that my installation doesn't really have the horsepower to use hash-all-files: over time, I got the several results: - (frequent) normal function, no evidence of hash attempts as previously reported; - (frequent) crashes every 5 minutes, reducing usefulness to zero; - (occasional) checksums in the log files as expected. For now, I've turned off global file hashing - I may revisit it more selectively as I learn. Thanks again, -- Glenn Forbes Fleming Larratt Cornell University IT Security Office On Fri, 21 Nov 2014, Seth Hall wrote: > >> On Nov 21, 2014, at 11:42 AM, Glenn Forbes Fleming Larratt wrote: >> >> I did - my process for rule changes goes: >> >> broctl check manager proxy{rnd} bro{rnd}-{rnd} >> broctl install >> broctl restart > > Have you looked at your loaded_scripts.log to see if the script isn't being loaded for some reason? > >>> That script should already be loaded by local.bro too so I'm actually kind of surprised that it wasn't already working? The result is that out of the box, Bro should be doing MD5 and SHA1 hashes by default when run with BroControl. >>> >> A previous maintainer had commented out hash-all-files for performance reasons. > > Ah. For the record, turning that feature off really doesn't change performance all that much (in my informal testing). > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > From seth at icir.org Tue Dec 9 10:00:48 2014 From: seth at icir.org (Seth Hall) Date: Tue, 9 Dec 2014 13:00:48 -0500 Subject: [Bro] building a new bro server In-Reply-To: References: <548647D7.9050305@gmail.com> Message-ID: > On Dec 8, 2014, at 10:57 PM, Allen, Brian wrote: > > We saw a huge improvement when we went from 16Gig RAM to 128Gig RAM. (That one was pretty obvious so we did that first). We also saw improvement when we pinned the processes to the cores. I think I had also suggested that you move to tcmalloc. Have you tried that yet? It?s not going to fix your issue with 30% packet loss, but I expect it would cut it down a bit further. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From pemberson at solarflare.com Tue Dec 9 10:06:33 2014 From: pemberson at solarflare.com (Paul Emberson) Date: Tue, 9 Dec 2014 18:06:33 +0000 Subject: [Bro] Using Bro without selectable fd support in libpcap Message-ID: <630C366ECD14F646B58D133C4C827B73B0EC3443@ocex02.SolarFlarecom.com> Hi, I am experimenting with Bro v2.3.1 using a version of libpcap that does not support an fd to call select() on. I noticed that earlier versions had a compile time option --disable-select-loop to disable the use of this within Bro. The only way I could find to use the recent version of Bro was to modify the code in a similar way to the conditional compilation used by --disable-select-loop (forcing 'idle = false' in PktSrc::ExtractNextPacket). Have I missed any Bro configuration (compile or run time) that will allow it to busy wait in recent versions? Is the intention for current and future versions of Bro to rely on selectable fd support in libpcap? Thanks in advance for any advice. Regards, Paul The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error, please notify the sender immediately and delete the message. Unless you are an addressee (or authorized to receive for an addressee), you may not use, copy or disclose to anyone this message or any information contained in this message. The unauthorized use, disclosure, copying or alteration of this message is strictly prohibited. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141209/2c4992d9/attachment.html From gc355804 at ohio.edu Tue Dec 9 10:22:13 2014 From: gc355804 at ohio.edu (Clark, Gilbert) Date: Tue, 9 Dec 2014 18:22:13 +0000 Subject: [Bro] building a new bro server In-Reply-To: References: <548647D7.9050305@gmail.com> , Message-ID: <1418149335192.91337@ohio.edu> A few other memory-related things, for what they're worth: * Make sure vm.swappiness is turned way down * numactl / numastat could be useful to play with: memory locality can make a difference * Related to memory locality: try tweaking vm.zone_reclaim_mode If you're into lower-level tuning / analysis, I also like: https://software.intel.com/en-us/articles/intel-performance-counter-monitor-a-better-way-to-measure-cpu-utilization HTH, Gilbert ________________________________________ From: bro-bounces at Bro.ORG on behalf of Seth Hall Sent: Tuesday, December 09, 2014 1:00 PM To: Allen, Brian Cc: bro at bro.org List Subject: Re: [Bro] building a new bro server > On Dec 8, 2014, at 10:57 PM, Allen, Brian wrote: > > We saw a huge improvement when we went from 16Gig RAM to 128Gig RAM. (That one was pretty obvious so we did that first). We also saw improvement when we pinned the processes to the cores. I think I had also suggested that you move to tcmalloc. Have you tried that yet? It?s not going to fix your issue with 30% packet loss, but I expect it would cut it down a bit further. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From gfaulkner.nsm at gmail.com Tue Dec 9 13:25:04 2014 From: gfaulkner.nsm at gmail.com (Gary Faulkner) Date: Tue, 09 Dec 2014 15:25:04 -0600 Subject: [Bro] building a new bro server In-Reply-To: References: <548647D7.9050305@gmail.com> Message-ID: <548768B0.1060604@gmail.com> For perspective I currently have a bro cluster comprised of 3 physical hosts. The first host runs the manager, proxies, and has storage to handle lots of bro logs and keep them for several months, the other two are dedicated to workers with relatively little storage. We have a hardware load-balancer to distribute traffic as evenly as possible between the worker nodes, and some effort has been made to limit having to process really large uninteresting flows before they reach the cluster. I looked at one of our typically busier blocks of time today (10:00-14:00) and during that time the cluster was seeing an average of 10Gbps of traffic with peaks as high as 15Gbps. Looking at our traffic graphs and capstats showed each host typically was seeing around 50% of that load, or around 5Gbps on average. During this time we saw an average capture loss of around 0.47%, with a max loss of 22.53%. During that same time-frame I had 18 snapshots where individual workers reported loss over 5%, and 2 over 10% out of 748. So, I'd say each host is probably seeing about the same amount of traffic as you have described, but loaded scripts etc may vary from your configuration. We have 22 workers per host for a total of 44 workers, and I believe the capture loss script is sampling traffic over 15 minute intervals by default, so there are roughly 17 time slices for each worker. Here are some details of how those nodes are configured in terms of hardware and bro. 2 worker hosts each with: 2xE5-2697v2 (12 Cores / 24 HT) 2.7Ghz/3.5Ghz Turbo 256GB RAM (probably overkill, but I used to have the manager and proxies running on one of the hosts and it skewed my memory use quite a bit) Intel X520-DA2 NIC Bro 2.3-7 (git master at the time I last updated) 22 workers PF_RING 5.6.2 using DNA IXGBE drivers, and pfdnacluster_master script CPU's pinned (used OS to verify which core presented to the OS mapped to each physical core to avoid mapping 2 workers to the same physical cores, and didn't use the 1st core on each CPU) HT is not disabled on these hosts and I'm still using the OS malloc. Worker configs like this: [worker-1] type=worker host=10.10.10.10 interface=dnacluster:21 lb_procs=22 lb_method=pf_ring pin_cpus=2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23 I suspect the faster CPUs will handle bursty flows better such as when a large volume of traffic load balances to a single worker, while more cores will probably help when you can better distribute the workload more evenly. This led me to try to pick something that balanced the 2 options (more cores vs higher clock speed. Naturally YMMV, and your traffic may not look like mine. Hope this helps. Regards, Gary On 12/9/2014 12:00 PM, Seth Hall wrote: >> On Dec 8, 2014, at 10:57 PM, Allen, Brian wrote: >> >> We saw a huge improvement when we went from 16Gig RAM to 128Gig RAM. (That one was pretty obvious so we did that first). We also saw improvement when we pinned the processes to the cores. > I think I had also suggested that you move to tcmalloc. Have you tried that yet? It?s not going to fix your issue with 30% packet loss, but I expect it would cut it down a bit further. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > From alexwis at gmail.com Tue Dec 9 14:55:52 2014 From: alexwis at gmail.com (Alex Waher) Date: Tue, 9 Dec 2014 14:55:52 -0800 Subject: [Bro] building a new bro server In-Reply-To: <548768B0.1060604@gmail.com> References: <548647D7.9050305@gmail.com> <548768B0.1060604@gmail.com> Message-ID: Bear in mind that there is a 32 application limit for the number of bro workers/slaves that can attach to a single cluster ID with the pf_ring dna/zc drivers. Or you can get really crafty and bounce traffic from one ring to another interface/ring and have up to 64 workers on a single box, provided you have the cores to work with :) Looking at the current Intel chips, I'd say the 8-core high-clock (+3.3Ghz) speed procs are a good option in a quad-socket system build and not break the bank. Would give you 32-cores to pin workers upon at a nice high clockspeed, which bro seems to greatly appreciate. The E5-2687W v2 or E5-2667 v2 or E5-4627 v2, some of which can turbo up to 4Ghz for traffic spikes (if you manage the power modes correctly! https://communities.intel.com/community/itpeernetwork/datastack/blog/2013/08/05/how-to-maximise-cpu-performance-for-the-oracle-database-on-linux ) -Alex On Tue, Dec 9, 2014 at 1:25 PM, Gary Faulkner wrote: > For perspective I currently have a bro cluster comprised of 3 physical > hosts. The first host runs the manager, proxies, and has storage to > handle lots of bro logs and keep them for several months, the other two > are dedicated to workers with relatively little storage. We have a > hardware load-balancer to distribute traffic as evenly as possible > between the worker nodes, and some effort has been made to limit having > to process really large uninteresting flows before they reach the > cluster. I looked at one of our typically busier blocks of time today > (10:00-14:00) and during that time the cluster was seeing an average of > 10Gbps of traffic with peaks as high as 15Gbps. Looking at our traffic > graphs and capstats showed each host typically was seeing around 50% of > that load, or around 5Gbps on average. During this time we saw an > average capture loss of around 0.47%, with a max loss of 22.53%. During > that same time-frame I had 18 snapshots where individual workers > reported loss over 5%, and 2 over 10% out of 748. So, I'd say each host > is probably seeing about the same amount of traffic as you have > described, but loaded scripts etc may vary from your configuration. We > have 22 workers per host for a total of 44 workers, and I believe the > capture loss script is sampling traffic over 15 minute intervals by > default, so there are roughly 17 time slices for each worker. Here are > some details of how those nodes are configured in terms of hardware and > bro. > > 2 worker hosts each with: > 2xE5-2697v2 (12 Cores / 24 HT) 2.7Ghz/3.5Ghz Turbo > 256GB RAM (probably overkill, but I used to have the manager and proxies > running on one of the hosts and it skewed my memory use quite a bit) > Intel X520-DA2 NIC > Bro 2.3-7 (git master at the time I last updated) > 22 workers > PF_RING 5.6.2 using DNA IXGBE drivers, and pfdnacluster_master script > CPU's pinned (used OS to verify which core presented to the OS mapped to > each physical core to avoid mapping 2 workers to the same physical > cores, and didn't use the 1st core on each CPU) > HT is not disabled on these hosts and I'm still using the OS malloc. > > Worker configs like this: > [worker-1] > type=worker > host=10.10.10.10 > interface=dnacluster:21 > lb_procs=22 > lb_method=pf_ring > pin_cpus=2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23 > > I suspect the faster CPUs will handle bursty flows better such as when a > large volume of traffic load balances to a single worker, while more > cores will probably help when you can better distribute the workload > more evenly. This led me to try to pick something that balanced the 2 > options (more cores vs higher clock speed. Naturally YMMV, and your > traffic may not look like mine. > > Hope this helps. > > Regards, > Gary > > On 12/9/2014 12:00 PM, Seth Hall wrote: > >> On Dec 8, 2014, at 10:57 PM, Allen, Brian wrote: > >> > >> We saw a huge improvement when we went from 16Gig RAM to 128Gig RAM. > (That one was pretty obvious so we did that first). We also saw > improvement when we pinned the processes to the cores. > > I think I had also suggested that you move to tcmalloc. Have you tried > that yet? It?s not going to fix your issue with 30% packet loss, but I > expect it would cut it down a bit further. > > > > .Seth > > > > -- > > Seth Hall > > International Computer Science Institute > > (Bro) because everyone has a network > > http://www.bro.org/ > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141209/08d70050/attachment.html From robin at icir.org Tue Dec 9 16:03:46 2014 From: robin at icir.org (Robin Sommer) Date: Tue, 9 Dec 2014 16:03:46 -0800 Subject: [Bro] Using Bro without selectable fd support in libpcap In-Reply-To: <630C366ECD14F646B58D133C4C827B73B0EC3443@ocex02.SolarFlarecom.com> References: <630C366ECD14F646B58D133C4C827B73B0EC3443@ocex02.SolarFlarecom.com> Message-ID: <20141210000346.GF2714@icir.org> On Tue, Dec 09, 2014 at 18:06 +0000, Paul Emberson wrote: > I noticed that earlier versions had a compile time option > --disable-select-loop to disable the use of this within Bro. It's been a while since we removed that. I believe it was primarily a case of bit rot: nobody had been using it in a while and hence it wasn't clear if it was still working; and it didn't seem necessary anymore anyways. I'd be reluctant to bring it back though, in the spirit of avoiding configure time options where possible. One alternative might be a writing plugin: current git master supports adding new packets sources in the form of external plugins. You could turn the the pcap code into a new plugin that adapts the necessary pieces, and then use that instead of the standard version. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From hhoffman at ip-solutions.net Tue Dec 9 16:46:39 2014 From: hhoffman at ip-solutions.net (Harry Hoffman) Date: Tue, 09 Dec 2014 19:46:39 -0500 Subject: [Bro] building a new bro server Message-ID: <20141210004643.AD17325666@pb-smtp1.pobox.com> So, slightly off-topic but since myself and several of you seem to be going through this would anyone be willing to collaborate on a paper/presentation to submit for Brocon 2015 that details the various methodologies folks are using to capture at X rate? We're somewhere around 5-6Gbps average but burst as high as 9. I've been through many iterations to get the "perfect" recipe and it might prove useful to others. However there are many different options on the network and system side so there are probably a few "perfect" recipes depending upon budget and equipment. Thoughts? Cheers, Harry On Dec 9, 2014 5:55 PM, Alex Waher wrote: > > Bear in mind that there is a 32 application limit for the number of bro workers/slaves that can attach to a single cluster ID with the pf_ring dna/zc drivers. Or you can get really crafty and bounce traffic from one ring to another interface/ring and have up to 64 workers on a single box, provided you have the cores to work with :) > > Looking at the current Intel chips, I'd say the 8-core high-clock (+3.3Ghz) speed procs are a good option in a quad-socket system build and not break the bank. Would give you 32-cores to pin workers upon at a nice high clockspeed, which bro seems to greatly appreciate.? The?E5-2687W v2 or?E5-2667 v2 or?E5-4627 v2, some of which can turbo up to 4Ghz for traffic spikes (if you manage the power modes correctly!?https://communities.intel.com/community/itpeernetwork/datastack/blog/2013/08/05/how-to-maximise-cpu-performance-for-the-oracle-database-on-linux ) > > -Alex > > On Tue, Dec 9, 2014 at 1:25 PM, Gary Faulkner wrote: >> >> For perspective I currently have a bro cluster comprised of 3 physical >> hosts. The first host runs the manager, proxies, and has storage to >> handle lots of bro logs and keep them for several months, the other two >> are dedicated to workers with relatively little storage. We have a >> hardware load-balancer to distribute traffic as evenly as possible >> between the worker nodes, and some effort has been made to limit having >> to process really large uninteresting flows before they reach the >> cluster. I looked at one of our typically busier blocks of time today >> (10:00-14:00) and during that time the cluster was seeing an average of >> 10Gbps of traffic with peaks as high as 15Gbps. Looking at our traffic >> graphs and capstats showed each host typically was seeing around 50% of >> that load, or around 5Gbps on average. During this time we saw an >> average capture loss of around 0.47%, with a max loss of 22.53%. During >> that same time-frame I had 18 snapshots where individual workers >> reported loss over 5%, and 2 over 10% out of 748. So, I'd say each host >> is probably seeing about the same amount of traffic as you have >> described, but loaded scripts etc may vary from your configuration. We >> have 22 workers per host for a total of 44 workers, and I believe the >> capture loss script is sampling traffic over 15 minute intervals by >> default, so there are roughly 17 time slices for each worker. Here are >> some details of how those nodes are configured in terms of hardware and bro. >> >> 2 worker hosts each with: >> 2xE5-2697v2 (12 Cores / 24 HT) 2.7Ghz/3.5Ghz Turbo >> 256GB RAM (probably overkill, but I used to have the manager and proxies >> running on one of the hosts and it skewed my memory use quite a bit) >> Intel X520-DA2 NIC >> Bro 2.3-7 (git master at the time I last updated) >> 22 workers >> PF_RING 5.6.2 using DNA IXGBE drivers, and pfdnacluster_master script >> CPU's pinned (used OS to verify which core presented to the OS mapped to >> each physical core to avoid mapping 2 workers to the same physical >> cores, and didn't use the 1st core on each CPU) >> HT is not disabled on these hosts and I'm still using the OS malloc. >> >> Worker configs like this: >> [worker-1] >> type=worker >> host=10.10.10.10 >> interface=dnacluster:21 >> lb_procs=22 >> lb_method=pf_ring >> pin_cpus=2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23 >> >> I suspect the faster CPUs will handle bursty flows better such as when a >> large volume of traffic load balances to a single worker, while more >> cores will probably help when you can better distribute the workload >> more evenly. This led me to try to pick something that balanced the 2 >> options (more cores vs higher clock speed. Naturally YMMV, and your >> traffic may not look like mine. >> >> Hope this helps. >> >> Regards, >> Gary >> >> On 12/9/2014 12:00 PM, Seth Hall wrote: >> >> On Dec 8, 2014, at 10:57 PM, Allen, Brian wrote: >> >> >> >> We saw a huge improvement when we went from 16Gig RAM to 128Gig RAM. (That one was pretty obvious so we did that first).? We also saw improvement when we pinned the processes to the cores. >> > I think I had also suggested that you move to tcmalloc.? Have you tried that yet?? It?s not going to fix your issue with 30% packet loss, but I expect it would cut it down a bit further. >> > >> >? ? .Seth >> > >> > -- >> > Seth Hall >> > International Computer Science Institute >> > (Bro) because everyone has a network >> > http://www.bro.org/ >> > >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > From scampbell at lbl.gov Tue Dec 9 17:17:12 2014 From: scampbell at lbl.gov (Scott Campbell) Date: Tue, 09 Dec 2014 20:17:12 -0500 Subject: [Bro] building a new bro server In-Reply-To: <20141210004643.AD17325666@pb-smtp1.pobox.com> References: <20141210004643.AD17325666@pb-smtp1.pobox.com> Message-ID: <54879F18.4070108@lbl.gov> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I second it, and will be able to provide some docs/tuning/configs regarding what we are currently doing. scott On 12/9/14 7:46 PM, Harry Hoffman wrote: > So, slightly off-topic but since myself and several of you seem to > be going through this would anyone be willing to collaborate on a > paper/presentation to submit for Brocon 2015 that details the > various methodologies folks are using to capture at X rate? > > We're somewhere around 5-6Gbps average but burst as high as 9. > > I've been through many iterations to get the "perfect" recipe and > it might prove useful to others. > > However there are many different options on the network and system > side so there are probably a few "perfect" recipes depending upon > budget and equipment. > > Thoughts? > > Cheers, Harry > > On Dec 9, 2014 5:55 PM, Alex Waher wrote: >> >> Bear in mind that there is a 32 application limit for the number >> of bro workers/slaves that can attach to a single cluster ID with >> the pf_ring dna/zc drivers. Or you can get really crafty and >> bounce traffic from one ring to another interface/ring and have >> up to 64 workers on a single box, provided you have the cores to >> work with :) >> >> Looking at the current Intel chips, I'd say the 8-core high-clock >> (+3.3Ghz) speed procs are a good option in a quad-socket system >> build and not break the bank. Would give you 32-cores to pin >> workers upon at a nice high clockspeed, which bro seems to >> greatly appreciate. The E5-2687W v2 or E5-2667 v2 or E5-4627 v2, >> some of which can turbo up to 4Ghz for traffic spikes (if you >> manage the power modes correctly! >> https://communities.intel.com/community/itpeernetwork/datastack/blog/2013/08/05/how-to-maximise-cpu-performance-for-the-oracle-database-on-linux >> ) >> >> -Alex >> >> On Tue, Dec 9, 2014 at 1:25 PM, Gary Faulkner >> wrote: >>> >>> For perspective I currently have a bro cluster comprised of 3 >>> physical hosts. The first host runs the manager, proxies, and >>> has storage to handle lots of bro logs and keep them for >>> several months, the other two are dedicated to workers with >>> relatively little storage. We have a hardware load-balancer to >>> distribute traffic as evenly as possible between the worker >>> nodes, and some effort has been made to limit having to process >>> really large uninteresting flows before they reach the cluster. >>> I looked at one of our typically busier blocks of time today >>> (10:00-14:00) and during that time the cluster was seeing an >>> average of 10Gbps of traffic with peaks as high as 15Gbps. >>> Looking at our traffic graphs and capstats showed each host >>> typically was seeing around 50% of that load, or around 5Gbps >>> on average. During this time we saw an average capture loss of >>> around 0.47%, with a max loss of 22.53%. During that same >>> time-frame I had 18 snapshots where individual workers reported >>> loss over 5%, and 2 over 10% out of 748. So, I'd say each host >>> is probably seeing about the same amount of traffic as you >>> have described, but loaded scripts etc may vary from your >>> configuration. We have 22 workers per host for a total of 44 >>> workers, and I believe the capture loss script is sampling >>> traffic over 15 minute intervals by default, so there are >>> roughly 17 time slices for each worker. Here are some details >>> of how those nodes are configured in terms of hardware and >>> bro. >>> >>> 2 worker hosts each with: 2xE5-2697v2 (12 Cores / 24 HT) >>> 2.7Ghz/3.5Ghz Turbo 256GB RAM (probably overkill, but I used to >>> have the manager and proxies running on one of the hosts and it >>> skewed my memory use quite a bit) Intel X520-DA2 NIC Bro 2.3-7 >>> (git master at the time I last updated) 22 workers PF_RING >>> 5.6.2 using DNA IXGBE drivers, and pfdnacluster_master script >>> CPU's pinned (used OS to verify which core presented to the OS >>> mapped to each physical core to avoid mapping 2 workers to the >>> same physical cores, and didn't use the 1st core on each CPU) >>> HT is not disabled on these hosts and I'm still using the OS >>> malloc. >>> >>> Worker configs like this: [worker-1] type=worker >>> host=10.10.10.10 interface=dnacluster:21 lb_procs=22 >>> lb_method=pf_ring >>> pin_cpus=2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23 >>> >>> >>> I suspect the faster CPUs will handle bursty flows better such as when a >>> large volume of traffic load balances to a single worker, while >>> more cores will probably help when you can better distribute >>> the workload more evenly. This led me to try to pick something >>> that balanced the 2 options (more cores vs higher clock speed. >>> Naturally YMMV, and your traffic may not look like mine. >>> >>> Hope this helps. >>> >>> Regards, Gary >>> >>> On 12/9/2014 12:00 PM, Seth Hall wrote: >>>>> On Dec 8, 2014, at 10:57 PM, Allen, Brian >>>>> wrote: >>>>> >>>>> We saw a huge improvement when we went from 16Gig RAM to >>>>> 128Gig RAM. (That one was pretty obvious so we did that >>>>> first). We also saw improvement when we pinned the >>>>> processes to the cores. >>>> I think I had also suggested that you move to tcmalloc. Have >>>> you tried that yet? It?s not going to fix your issue with >>>> 30% packet loss, but I expect it would cut it down a bit >>>> further. >>>> >>>> .Seth >>>> >>>> -- Seth Hall International Computer Science Institute (Bro) >>>> because everyone has a network http://www.bro.org/ >>>> >>> >>> _______________________________________________ Bro mailing >>> list bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> > > _______________________________________________ Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlSHnxcACgkQK2Plq8B7ZBwvFgCg5VH6AzbPFAQaiIMlT5IptCwX hlUAoLp8+D0rNXF43Cgaspn9uOU9jO7H =vWU4 -----END PGP SIGNATURE----- From rotsted at reservoir.com Wed Dec 10 10:37:30 2014 From: rotsted at reservoir.com (Robert Rotsted) Date: Wed, 10 Dec 2014 10:37:30 -0800 Subject: [Bro] Netflow ingest with Bro? Message-ID: Hi all, Is anyone using Bro's Netflow ingest capabilities? If so, what is the output? Does Bro generate TCP and UDP events? Does it create a "conn" log? Some context from the Bro 1.4 release notes (https://www.bro.org/sphinx/install/changes.html?highlight=netflow): Bro now supports analyzing NetFlow v5 data, i.e., from Cisco routers (Bernhard Ager). NetFlow can be useful for intrusion detection as it allows analysis of traffic from many different points in the network. Bro can now read NetFlow data from a UDP socket, as well as (mostly for debugging purposes) from a file in a specialized format. You can create these files with the programs given in aux/nftools. Best, Bob -- Bob Rotsted Senior Engineer Reservoir Labs, Inc. From robin at icir.org Wed Dec 10 11:45:31 2014 From: robin at icir.org (Robin Sommer) Date: Wed, 10 Dec 2014 11:45:31 -0800 Subject: [Bro] Netflow ingest with Bro? In-Reply-To: References: Message-ID: <20141210194531.GP18367@icir.org> On Wed, Dec 10, 2014 at 10:37 -0800, Robert Rotsted wrote: > Is anyone using Bro's Netflow ingest capabilities? If so, what is the > output? Does Bro generate TCP and UDP events? Does it create a "conn" > log? Two netflow events: event netflow_v5_header(h: nf_v5_header); event netflow_v5_record(r: nf_v5_record); I don't think we ever had a standard script doing something further with these. Note, the Netflow support has been removed in current git master along with some of the restructuring, as it was neither much used nor tested at all. But it's not inconceivable to bring it back before the next release if there's demand for it. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From rotsted at reservoir.com Wed Dec 10 13:29:01 2014 From: rotsted at reservoir.com (Robert Rotsted) Date: Wed, 10 Dec 2014 13:29:01 -0800 Subject: [Bro] Netflow ingest with Bro? In-Reply-To: <20141210194531.GP18367@icir.org> References: <20141210194531.GP18367@icir.org> Message-ID: Good to know. Thanks for the quick reply Robin! On Wed, Dec 10, 2014 at 11:45 AM, Robin Sommer wrote: > > > On Wed, Dec 10, 2014 at 10:37 -0800, Robert Rotsted wrote: > >> Is anyone using Bro's Netflow ingest capabilities? If so, what is the >> output? Does Bro generate TCP and UDP events? Does it create a "conn" >> log? > > Two netflow events: > > event netflow_v5_header(h: nf_v5_header); > event netflow_v5_record(r: nf_v5_record); > > I don't think we ever had a standard script doing something further > with these. > > Note, the Netflow support has been removed in current git master along > with some of the restructuring, as it was neither much used nor tested > at all. But it's not inconceivable to bring it back before the next > release if there's demand for it. > > Robin > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin -- Bob Rotsted Senior Engineer Reservoir Labs, Inc. 503-225-0583 x138 From sovakah at gmail.com Wed Dec 10 14:25:16 2014 From: sovakah at gmail.com (Savakh S) Date: Wed, 10 Dec 2014 23:25:16 +0100 Subject: [Bro] Bro's capabilities Message-ID: Hi all, I have a question about the Bro's capabilities. Could you please detail how Bro works ? I know Bro has "protocol analysis" capabilities for some protocols and is a "behavior-based" IDS. If I understand well, Bro can learn the way a network is used (like a machine learning) and then dissect all the protocols he can parse (http, ftp, ...) to see if the fields' values of these procoles were recorded at the learning phase ? Thanks for your answers. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141210/c4d0a170/attachment.html From wren3 at illinois.edu Wed Dec 10 23:14:33 2014 From: wren3 at illinois.edu (Ren, Wenyu) Date: Thu, 11 Dec 2014 07:14:33 +0000 Subject: [Bro] Question about how to transfer bro data to java Message-ID: Hi all, I have a question about transferring data in Bro script to java. I currently have a tree-like data structure kept by multi-level tables in Bro script. I would like to transfer this data structure to java environment for further use (maybe using multi-level hashmaps in java to keep it). What is the most efficient way to do this? The way I can come up with is to simply traverse the tree and write all the data in the tree in some certain formats to a text file and read that file in java to construct the structure again from scratches. But this seems extremely inefficient. Do you know any better way to do this? Thanks a lot for your answers. Wenyu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141211/044c3306/attachment.html From vitologrillo at gmail.com Thu Dec 11 02:56:15 2014 From: vitologrillo at gmail.com (Vito Logrillo) Date: Thu, 11 Dec 2014 11:56:15 +0100 Subject: [Bro] Known-services log file not created Message-ID: Hi, i've downloaded the latest Bro version from git yesterday: i've noticed that known-services.log is no more generated even if known-services.bro is present in loaded_scrips.log. I've used the below commands to build bro: git clone --recursive git://git.bro.org/bro ./configure make make install What's wrong? Thanks From seth at icir.org Thu Dec 11 06:13:40 2014 From: seth at icir.org (Seth Hall) Date: Thu, 11 Dec 2014 09:13:40 -0500 Subject: [Bro] Question about how to transfer bro data to java In-Reply-To: References: Message-ID: > On Dec 11, 2014, at 2:14 AM, Ren, Wenyu wrote: > > The way I can come up with is to simply traverse the tree and write all the data in the tree in some certain formats to a text file and read that file in java to construct the structure again from scratches. But this seems extremely inefficient. Do you know any better way to do this? That?s probably not a bad way to go. No matter what you do you?re going to end up with Bro serializing and then your java code deserializing all of your data. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Thu Dec 11 06:14:38 2014 From: seth at icir.org (Seth Hall) Date: Thu, 11 Dec 2014 09:14:38 -0500 Subject: [Bro] Known-services log file not created In-Reply-To: References: Message-ID: > On Dec 11, 2014, at 5:56 AM, Vito Logrillo wrote: > git clone --recursive git://git.bro.org/bro > ./configure > make > make install Are you configuring your Site::local_nets? With broctl, this is typically done by configuring your local networks in networks.cfg. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From tiburcesotohou at yahoo.fr Thu Dec 11 07:04:54 2014 From: tiburcesotohou at yahoo.fr (SOTOHOU Osince Tiburce) Date: Thu, 11 Dec 2014 15:04:54 +0000 (UTC) Subject: [Bro] the exolits events Message-ID: <692298722.8913039.1418310294253.JavaMail.yahoo@jws11168.mail.ir2.yahoo.com> hi,Where can i see the list of the events that can be generated by Bro IDS?Particularly, i want to see the events related to the known exploits. thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141211/2664b2e6/attachment.html From gc355804 at ohio.edu Fri Dec 12 18:15:00 2014 From: gc355804 at ohio.edu (Clark, Gilbert) Date: Sat, 13 Dec 2014 02:15:00 +0000 Subject: [Bro] Question about how to transfer bro data to java In-Reply-To: References: , Message-ID: <1418436903507.76014@ohio.edu> Just a few more thoughts, for what they're worth. Might be a reason the following are not good ideas: haven't thought through them too much. Just listing the first few things that come to mind. One alternative might be to build a plugin to define a bro script method that would serialize the table data into something Java could understand, send it to Java somehow, and then have Java deserialize it on the receiving end. If you're looking for simplicity, I think JSON would definitely be easy to deserialize into nested HashMaps on the Java end, and would *probably* be pretty straightforward to serialize on the Bro end. That's not going to be the fastest format to work with, however. Something that might be even faster (and harder) would be to build a shared memory region, then build a plugin method that would serialize the data into a form that made sense for Java, then bulk copy the table data into that shared memory space. From there, the Java application could map the shared memory segment into its own space directly, so there wouldn't need to be a deserialization step involved. The synchronization could be a challenge, though, I think. Also, pulling tabular data from Java mapped ByteBuffer objects (or equivalent) could be a challenge, depending on the type of data contained in the table. A slightly more complex twist might be to extend bro's tables to support being backed by shared memory, the idea being that they could be mapped into other processes directly. I do think this would be *very* hard to get exactly right ... and there would still be some additional overhead in the form of copying / synchronization required to make this kind of IPC work in the first place. It's hard to say exactly how well (or not) this would perform, I think. If the Java environment can reasonably run in the bro process space, then another option might be to launch a JVM from within bro. Write a script extension that would write the table data into a queue, and let a separate Java thread read / process the data being fed to it through said queue. Note that, last I knew, Java doesn't support shared memory directly ... [1] and [2] might be interesting reading that would be related to this subject and offer practicable workarounds, however. If the intermediate file option is the most attractive, might consider looking into trying to write to / read from a RAM disk (if that hasn't been done already). The performance *could* be better ... or not. Anyway, hope something in there is interesting / useful, and good luck :) -Gilbert [1] http://javaforu.blogspot.com/2011/09/offloading-data-from-jvm-heap-little.html [2] http://javaforu.blogspot.com/2011/09/ram-disk-is-already-in-linux-and-nobody.html ________________________________________ From: bro-bounces at bro.org on behalf of Seth Hall Sent: Thursday, December 11, 2014 9:13 AM To: Ren, Wenyu Cc: bro at bro.org Subject: Re: [Bro] Question about how to transfer bro data to java > On Dec 11, 2014, at 2:14 AM, Ren, Wenyu wrote: > > The way I can come up with is to simply traverse the tree and write all the data in the tree in some certain formats to a text file and read that file in java to construct the structure again from scratches. But this seems extremely inefficient. Do you know any better way to do this? That?s probably not a bad way to go. No matter what you do you?re going to end up with Bro serializing and then your java code deserializing all of your data. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From wren3 at illinois.edu Fri Dec 12 18:50:09 2014 From: wren3 at illinois.edu (Ren, Wenyu) Date: Sat, 13 Dec 2014 02:50:09 +0000 Subject: [Bro] Question about how to transfer bro data to java In-Reply-To: <1418436903507.76014@ohio.edu> References: , , <1418436903507.76014@ohio.edu> Message-ID: Hi Gilbert and Seth, Thanks a lot for so many thoughts. I guess I will start with writing the data into a file and see how it works. Best, Wenyu ________________________________________ From: Clark, Gilbert [gc355804 at ohio.edu] Sent: Friday, December 12, 2014 8:15 PM To: Seth Hall; Ren, Wenyu Cc: bro at bro.org Subject: Re: [Bro] Question about how to transfer bro data to java Just a few more thoughts, for what they're worth. Might be a reason the following are not good ideas: haven't thought through them too much. Just listing the first few things that come to mind. One alternative might be to build a plugin to define a bro script method that would serialize the table data into something Java could understand, send it to Java somehow, and then have Java deserialize it on the receiving end. If you're looking for simplicity, I think JSON would definitely be easy to deserialize into nested HashMaps on the Java end, and would *probably* be pretty straightforward to serialize on the Bro end. That's not going to be the fastest format to work with, however. Something that might be even faster (and harder) would be to build a shared memory region, then build a plugin method that would serialize the data into a form that made sense for Java, then bulk copy the table data into that shared memory space. From there, the Java application could map the shared memory segment into its own space directly, so there wouldn't need to be a deserialization step involved. The synchronization could be a challenge, though, I think. Also, pulling tabular data from Java mapped ByteBuffer objects (or equivalent) could be a challenge, depending on the type of data contained in the table. A slightly more complex twist might be to extend bro's tables to support being backed by shared memory, the idea being that they could be mapped into other processes directly. I do think this would be *very* hard to get exactly right ... and there would still be some additional overhead in the form of copying / synchronization required to make this kind of IPC work in the first place. It's hard to say exactly how well (or not) this would perform, I think. If the Java environment can reasonably run in the bro process space, then another option might be to launch a JVM from within bro. Write a script extension that would write the table data into a queue, and let a separate Java thread read / process the data being fed to it through said queue. Note that, last I knew, Java doesn't support shared memory directly ... [1] and [2] might be interesting reading that would be related to this subject and offer practicable workarounds, however. If the intermediate file option is the most attractive, might consider looking into trying to write to / read from a RAM disk (if that hasn't been done already). The performance *could* be better ... or not. Anyway, hope something in there is interesting / useful, and good luck :) -Gilbert [1] http://javaforu.blogspot.com/2011/09/offloading-data-from-jvm-heap-little.html [2] http://javaforu.blogspot.com/2011/09/ram-disk-is-already-in-linux-and-nobody.html ________________________________________ From: bro-bounces at bro.org on behalf of Seth Hall Sent: Thursday, December 11, 2014 9:13 AM To: Ren, Wenyu Cc: bro at bro.org Subject: Re: [Bro] Question about how to transfer bro data to java > On Dec 11, 2014, at 2:14 AM, Ren, Wenyu wrote: > > The way I can come up with is to simply traverse the tree and write all the data in the tree in some certain formats to a text file and read that file in java to construct the structure again from scratches. But this seems extremely inefficient. Do you know any better way to do this? That?s probably not a bad way to go. No matter what you do you?re going to end up with Bro serializing and then your java code deserializing all of your data. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From joshcwa at live.com Sat Dec 13 13:56:09 2014 From: joshcwa at live.com (Josh Carlson) Date: Sat, 13 Dec 2014 21:56:09 +0000 Subject: [Bro] Hardware Guidence Message-ID: I'm looking at specing out HP PL boxes for monitoring 300Mbps and 600Mbps links separately. Does bro benefit from having more cores on the processor? Or should I get multiple processors each with a lower number of cores? Looking at a DL180 w/ a Xeon 1.8 with 12-cores and 196GB of RAM for now. Thanks for your help! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141213/415bf034/attachment.html From jdonnelly at dyn.com Sun Dec 14 13:44:55 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Sun, 14 Dec 2014 15:44:55 -0600 Subject: [Bro] Bro eliminating banner in log files . Message-ID: Hi. Is there a way to eliminate the descriptive record banner that is the first line of the log file? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141214/8fa237b8/attachment.html From psm8337 at naver.com Mon Dec 15 00:43:38 2014 From: psm8337 at naver.com (=?UTF-8?B?67CV7ISg66+8?=) Date: Mon, 15 Dec 2014 17:43:38 +0900 (KST) Subject: [Bro] =?utf-8?q?Hello=2C_I=27m_the_beginner_of_Bro=2C_Seonmin=2E?= In-Reply-To: References: Message-ID: <4cd184a443819327eae6363ae305057@cweb08.nm.nhnsystem.com> Dear Sir or Madam. Hi, Nice to meet you. My name is Seonmin and I'm South Korean. I'm studying Bro and I saw Bro User Manual.(version 0.9, 12-1-2004, DRAFT) That's really helpful for me to understand the Bro. Thank you so much. All these days, I've searched far and wide the bro's config files and other documents on the Internet. However I couldn't find the best one what I need... I understand the sketchy knowledge. For example, The keynote of Bro's benefits and simple flow.( Network -> libpcap -> event engine -> PolicyScriptInterpreter) But that kind of information is not enough for me to understand the Bro perfectly. I want to know how to work the Bro accurately.(Some information of Bro's architecture) I need like a Bro hierarchy structure. ( EX. To run the "bro/base/protocols" file, Bro needs the "bro/base/framewalk/analyzer" file. And it also needs the "bro/base/framewalk/packet_filter.util" file.) bro/base/framewalk/packet_filter.util ? bro/base/framewalk/analyzer ? bro/base/protocols So, I refered to the Bro User Manual's "Appendix A Bro Directory and Files". But It is corrupted like below. If you don?t mind me asking, Would you send me the original file or picture of that? * based on that file, I will try to do the simplification of Bro. (To inspect only http packets) I'm sorry for the sentences are not neat... If you are not sure what I mean, Please tell me. Yours Faithfully, Seonmin. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141215/6b6b389f/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: ??? 1.png Type: image/x-png Size: 24204 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141215/6b6b389f/attachment.png From hckim at narusec.com Mon Dec 15 03:08:15 2014 From: hckim at narusec.com (=?UTF-8?B?6rmA7Z2s7LKg?=) Date: Mon, 15 Dec 2014 20:08:15 +0900 Subject: [Bro] Exclude IPS - only src ip Message-ID: HI I used to filter ip by adding this command to local.bro redef restrict_filters = { ["not-hosts"] = "not host X.X.X.X" }; but now I want to filter out only src_ip(in bro id.orig_h) I tried redef restrict_filters = { ["not-nets"] = "not src net X.X.X.X" }; redef restrict_filters = { ["not-nets"] = "!src net X.X.X.X" }; redef restrict_filters = { ["not-nets"] = "not(src net X.X.X.X)" }; but it does not filter a ip I want from src_ip it there a way to filter out only a src_ip? Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141215/5c2be198/attachment.html From hosom at battelle.org Mon Dec 15 04:53:38 2014 From: hosom at battelle.org (Hosom, Stephen M) Date: Mon, 15 Dec 2014 12:53:38 +0000 Subject: [Bro] Hello, I'm the beginner of Bro, Seonmin. In-Reply-To: <4cd184a443819327eae6363ae305057@cweb08.nm.nhnsystem.com> References: <4cd184a443819327eae6363ae305057@cweb08.nm.nhnsystem.com> Message-ID: Seonmin, The document you?re referencing is quite old, and I?m not sure that it would be of great benefit to you. It seems like what you want to do is instruct Bro to only monitor port 80. Normally, I would advise people to do this at their traffic aggregation point, but that isn?t always available. You can do this within bro by using redef to change the value of PacketFilter::default_capture_filter. Do this by adding the following line to your local.bro: redef PacketFilter::default_capture_filter = ?port 80?; That having been said, I think what you actually want is to only analyze the HTTP protocol, which can occur on many ports. That?s a bit trickier. Let me know if this is what you want, and I?m certain that there is a way to do that as well, but I would need a minute to look into the best way to do it. Thanks, Stephen From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of ??? Sent: Monday, December 15, 2014 3:44 AM To: bro at bro.org Subject: [Bro] Hello, I'm the beginner of Bro, Seonmin. Dear Sir or Madam. Hi, Nice to meet you. My name is Seonmin and I'm South Korean. I'm studying Bro and I saw Bro User Manual.(version 0.9, 12-1-2004, DRAFT) That's really helpful for me to understand the Bro. Thank you so much. All these days, I've searched far and wide the bro's config files and other documents on the Internet. However I couldn't find the best one what I need... I understand the sketchy knowledge. For example, The keynote of Bro's benefits and simple flow.( Network -> libpcap -> event engine -> PolicyScriptInterpreter) But that kind of information is not enough for me to understand the Bro perfectly. I want to know how to work the Bro accurately.(Some information of Bro's architecture) I need like a Bro hierarchy structure. ( EX. To run the "bro/base/protocols" file, Bro needs the "bro/base/framewalk/analyzer" file. And it also needs the "bro/base/framewalk/packet_filter.util" file.) bro/base/framewalk/packet_filter.util ? bro/base/framewalk/analyzer ? bro/base/protocols So, I refered to the Bro User Manual's "Appendix A Bro Directory and Files". But It is corrupted like below. [cid:image001.png at 01D0183C.3C90C790] If you don?t mind me asking, Would you send me the original file or picture of that? * based on that file, I will try to do the simplification of Bro. (To inspect only http packets) I'm sorry for the sentences are not neat... If you are not sure what I mean, Please tell me. Yours Faithfully, Seonmin. [http://mail.naver.com/readReceipt/notify/?img=5mnqKqCRbXgwpzKdhAnXaA3CFAtqaAulKqUXp6J0FoKrKrJ0Kqg9KAEXtzFXp6UmaVl5WLl51zlqDBFdp6d5MreRhoRSb4egMNiT74eZpm%3D%3D.gif] -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141215/5ac1a4a8/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 24204 bytes Desc: image001.png Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141215/5ac1a4a8/attachment.bin From monahbaki at gmail.com Mon Dec 15 05:22:17 2014 From: monahbaki at gmail.com (Monah Baki) Date: Mon, 15 Dec 2014 08:22:17 -0500 Subject: [Bro] Question regarding conn-summary reports Message-ID: Hi all, I am trying to view the conn-summary.07:00:00-08:00:00.log.gz file, they way I understood it, this is for a 1 hour summary of what's going on in our network. However, at the beginning of the output, it displays >== Total Connections=== 2014-12-12-14-54-42 - 2014-12-15-07-59-55 Then >== Incoming Connections=== 2014-12-14-18-59-08 - 2014-12-15-07-59-55 Why does it show a span of 3 days and the time is confusing. Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141215/f51193bd/attachment.html From seth at icir.org Mon Dec 15 06:59:36 2014 From: seth at icir.org (Seth Hall) Date: Mon, 15 Dec 2014 09:59:36 -0500 Subject: [Bro] Bro eliminating banner in log files . In-Reply-To: References: Message-ID: <1626C33E-214B-45E0-B1C7-06BD8D0BDF27@icir.org> > On Dec 14, 2014, at 4:44 PM, John Donnelly wrote: > > Hi. Is there a way to eliminate the descriptive record banner that is the first line of the log file? You can do this per-log or globally. Globally: redef LogAscii::include_meta=F; Depending on what this is for, it may be beneficial to just output your log as JSON: redef LogAscii::use_json=T; If you want to do either of these settings only in a single log, you can do this? event bro_init() { Log::remove_default_filter(HTTP::LOG); local filt = Log::Filter($name="default", $writer=Log::WRITER_ASCII, $config=table(["include_meta"] = "F")); Log::add_filter(HTTP::LOG, filt); } You should be able to configure any of the other global ascii log settings the same as I did in that example. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Mon Dec 15 07:09:06 2014 From: seth at icir.org (Seth Hall) Date: Mon, 15 Dec 2014 10:09:06 -0500 Subject: [Bro] Exclude IPS - only src ip In-Reply-To: References: Message-ID: <53774992-05B6-4921-928A-41CFF1C095FF@icir.org> > On Dec 15, 2014, at 6:08 AM, ??? wrote: > > it there a way to filter out only a src_ip? Are you sure you really want to filter a src address? Because Bro typically needs full duplex traffic to work correctly, it rarely makes sense to filter with a src or dst. Do you also have multiple ?redef restrict_filters? line as you showed? You are doing full value assignment by using ?=? instead of extending the table with ?+=? which will definitely cause you trouble if that?s happening. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From mike.patterson at uwaterloo.ca Mon Dec 15 07:25:32 2014 From: mike.patterson at uwaterloo.ca (Mike Patterson) Date: Mon, 15 Dec 2014 15:25:32 +0000 Subject: [Bro] Bro eliminating banner in log files . In-Reply-To: <1626C33E-214B-45E0-B1C7-06BD8D0BDF27@icir.org> References: <1626C33E-214B-45E0-B1C7-06BD8D0BDF27@icir.org> Message-ID: <60175EA7-9C3C-4044-94AE-BD5811C4E5AA@uwaterloo.ca> > On Dec 15, 2014, at 9:59 AM, Seth Hall wrote: > > Depending on what this is for, it may be beneficial to just output your log as JSON: > redef LogAscii::use_json=T; Tangentially but related: I just had a look at the code for ascii.bro. It seems that I can't have both JSON and "normal" text, is that correct? ("But why would you want to?" Because I want to have my cake and eat it too :) ) Mike -- Whoever is out of patience is out of possession of his soul. Men must not turn into bees, and kill themselves in stinging others. - Sir Francis Bacon From seth at icir.org Mon Dec 15 07:32:39 2014 From: seth at icir.org (Seth Hall) Date: Mon, 15 Dec 2014 10:32:39 -0500 Subject: [Bro] Bro eliminating banner in log files . In-Reply-To: <60175EA7-9C3C-4044-94AE-BD5811C4E5AA@uwaterloo.ca> References: <1626C33E-214B-45E0-B1C7-06BD8D0BDF27@icir.org> <60175EA7-9C3C-4044-94AE-BD5811C4E5AA@uwaterloo.ca> Message-ID: > On Dec 15, 2014, at 10:25 AM, Mike Patterson wrote: > > ("But why would you want to?" Because I want to have my cake and eat it too :) ) Sure you can! That use_json name is available for use in the $config table that I used in my filter configuration. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jdonnelly at dyn.com Mon Dec 15 07:48:27 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Mon, 15 Dec 2014 09:48:27 -0600 Subject: [Bro] Bro eliminating banner in log files . In-Reply-To: <1626C33E-214B-45E0-B1C7-06BD8D0BDF27@icir.org> References: <1626C33E-214B-45E0-B1C7-06BD8D0BDF27@icir.org> Message-ID: Thank you. Th global setting did not appear to work, but the individual LOG filter line did . On Mon, Dec 15, 2014 at 8:59 AM, Seth Hall wrote: > > > > On Dec 14, 2014, at 4:44 PM, John Donnelly wrote: > > > > Hi. Is there a way to eliminate the descriptive record banner that is > the first line of the log file? > > You can do this per-log or globally. > > Globally: > redef LogAscii::include_meta=F; > > Depending on what this is for, it may be beneficial to just output your > log as JSON: > redef LogAscii::use_json=T; > > If you want to do either of these settings only in a single log, you can > do this? > > event bro_init() > { > Log::remove_default_filter(HTTP::LOG); > local filt = Log::Filter($name="default", > $writer=Log::WRITER_ASCII, > $config=table(["include_meta"] = "F")); > Log::add_filter(HTTP::LOG, filt); > } > > You should be able to configure any of the other global ascii log settings > the same as I did in that example. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141215/a2f7cda0/attachment.html From hckim at narusec.com Mon Dec 15 17:55:07 2014 From: hckim at narusec.com (=?UTF-8?B?6rmA7Z2s7LKg?=) Date: Tue, 16 Dec 2014 10:55:07 +0900 Subject: [Bro] Exclude IPS - only src ip In-Reply-To: <53774992-05B6-4921-928A-41CFF1C095FF@icir.org> References: <53774992-05B6-4921-928A-41CFF1C095FF@icir.org> Message-ID: src_ip I want to filter out is a 'proxy web server ip'. I want to watch only local net work log. There is to much proxy_src_ip log that we do not need, other reason is to reduce log amount (I am getting live traffic by mirror which our customer is doing, so I do not have any choice) I did not write multiple ?redef restrict_filters? line. I ran one line at a time. Thank you On Tue, Dec 16, 2014 at 12:09 AM, Seth Hall wrote: > > > > On Dec 15, 2014, at 6:08 AM, ??? wrote: > > > > it there a way to filter out only a src_ip? > > Are you sure you really want to filter a src address? Because Bro > typically needs full duplex traffic to work correctly, it rarely makes > sense to filter with a src or dst. > > Do you also have multiple ?redef restrict_filters? line as you showed? > You are doing full value assignment by using ?=? instead of extending the > table with ?+=? which will definitely cause you trouble if that?s happening. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141216/48b74eaf/attachment.html From seth at icir.org Tue Dec 16 07:42:14 2014 From: seth at icir.org (Seth Hall) Date: Tue, 16 Dec 2014 10:42:14 -0500 Subject: [Bro] Exclude IPS - only src ip In-Reply-To: References: <53774992-05B6-4921-928A-41CFF1C095FF@icir.org> Message-ID: <96902473-46DA-4541-A2BA-EC334C1E58B9@icir.org> > On Dec 15, 2014, at 8:55 PM, ??? wrote: > > src_ip I want to filter out is a 'proxy web server ip'. I want to watch only local net work log. > There is to much proxy_src_ip log that we do not need, other reason is to reduce log amount > (I am getting live traffic by mirror which our customer is doing, so I do not have any choice)  I believe you?re over-thinking this. Just remove the ?src? from your expressions. Try something like this? redef restrict_filters += { ["not-nets"] = "not net 1.2.3.0/24? }; .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From bkellogg at dresser-rand.com Wed Dec 17 07:23:24 2014 From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN)) Date: Wed, 17 Dec 2014 15:23:24 +0000 Subject: [Bro] Supress notices Message-ID: Do notice suppressions only suppress that notice type based upon the identifier? So if the identifier can match across multiple notice types will it suppress those other notice types as well? Dumb question probably, but just want to verify. Thanks, Brian From jdonnelly at dyn.com Wed Dec 17 14:45:57 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Wed, 17 Dec 2014 16:45:57 -0600 Subject: [Bro] Moving a old plugin.cc to new format Message-ID: Hi, I see the plugin format has changed .. Any suggestions what needs to done to move a older one to it: Plugin.cc BRO_PLUGIN_BEGIN(Bro, DNS_TELEMETRY) BRO_PLUGIN_DESCRIPTION("DNS Telemetry analyzer"); BRO_PLUGIN_ANALYZER("DNS_TELEMETRY", dns_telemetry::DNS_Telemetry_Analyzer); BRO_PLUGIN_SUPPORT_ANALYZER("Contents_DNS_Telemetry"); BRO_PLUGIN_BIF_FILE(events); BRO_PLUGIN_END Thanks ! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141217/251365a5/attachment.html From jdonnelly at dyn.com Wed Dec 17 15:32:20 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Wed, 17 Dec 2014 17:32:20 -0600 Subject: [Bro] Moving a old plugin.cc to new format In-Reply-To: References: Message-ID: I got it working ;-) On Wed, Dec 17, 2014 at 4:45 PM, John Donnelly wrote: > > Hi, > > I see the plugin format has changed .. Any suggestions what needs to done > to move a older one to it: > > Plugin.cc > > > BRO_PLUGIN_BEGIN(Bro, DNS_TELEMETRY) > BRO_PLUGIN_DESCRIPTION("DNS Telemetry analyzer"); > BRO_PLUGIN_ANALYZER("DNS_TELEMETRY", > dns_telemetry::DNS_Telemetry_Analyzer); > BRO_PLUGIN_SUPPORT_ANALYZER("Contents_DNS_Telemetry"); > BRO_PLUGIN_BIF_FILE(events); > BRO_PLUGIN_END > > > Thanks ! > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141217/eb5c6d0c/attachment.html From seth at icir.org Thu Dec 18 06:29:38 2014 From: seth at icir.org (Seth Hall) Date: Thu, 18 Dec 2014 09:29:38 -0500 Subject: [Bro] Supress notices In-Reply-To: References: Message-ID: > On Dec 17, 2014, at 10:23 AM, Kellogg, Brian D (OLN) wrote: > > Do notice suppressions only suppress that notice type based upon the identifier? So if the identifier can match across multiple notice types will it suppress those other notice types as well? Dumb question probably, but just want to verify.  It suppresses based on the combination of the $note field and the $identifier field. If your notice doesn?t have an $identifier field defined for it, no suppression will occur. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From doris at bro.org Thu Dec 18 15:47:29 2014 From: doris at bro.org (Doris Schioberg) Date: Thu, 18 Dec 2014 15:47:29 -0800 Subject: [Bro] Bro in 2014: the annual newsletter is her Message-ID: <54936791.8060603@bro.org> Bro had an eventful year. Our December Monthly is replaced by a summary of the whole year 2014. Find out about past events, research highlights, the tools of the year, and more in our annual newsletter. http://blog.bro.org/2014/12/bro-rewind-2014.html Happy Holidays! - The Bro Team -- Doris Schioberg Bro Outreach, Training, and Education Coordinator International Computer Science Institute (ICSI Berkeley) Phone: +1 (510) 289-8406 * doris at bro.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 882 bytes Desc: OpenPGP digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141218/a900f98c/attachment.bin From shrtsns at gmail.com Fri Dec 19 03:52:50 2014 From: shrtsns at gmail.com (Sharath SN) Date: Fri, 19 Dec 2014 17:22:50 +0530 Subject: [Bro] signature configuration Message-ID: hi, Can anyone suggest me how to add our own signatures. I had tried with below code place it in policy/framework/signature/ssl_renegotiation.sig signature ssl_renegotiation { ip-proto == tcp dst-port == 443 event "ssl renegotiation" payload /.*\x14\x03/ } but signature logs are not getting generated. could you please suggest me what I'm doing wrong with this. dont mind if it is silly q'n.. I'm new this bro.. Thanks for ur time, shrtsns -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141219/92138dc9/attachment.html From jdonnelly at dyn.com Fri Dec 19 05:37:48 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Fri, 19 Dec 2014 07:37:48 -0600 Subject: [Bro] Restructing of PktSrc.cc - missing FileName Message-ID: Hi, We have been using 2.2.135 fork of bro and notice a lot of changes in the PktSrc class in the current implementation; Some of fields we have been using have disappeared in the recent code base: ( FileName ) Old : class PktDumper {public: PktDumper(const char* file = 0, bool append = false ); ~PktDumper() { Close(); } bool Open(const char* file = 0); bool Close(); bool Dump(const struct pcap_pkthdr* hdr, const u_char* pkt); pcap_dumper_t* PcapDumper() { return dumper; } const char* FileName() const { return filename; } bro.bif: In function ?Val* BifFunc::bro_rotate_file_to_name(Frame*, val_list*)?: bro.bif:4216:39: error: ?class iosource::PktDumper? has no member named ?FileName? bro.bif:4223:30: error: ?class iosource::PktDumper? has no member named ?FileName? Suggestions welcome. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141219/3c6038c2/attachment.html From jsiwek at illinois.edu Fri Dec 19 07:24:32 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Fri, 19 Dec 2014 15:24:32 +0000 Subject: [Bro] signature configuration In-Reply-To: References: Message-ID: <8B4D1F9E-12CE-4777-BC41-202AD4BAB071@illinois.edu> > On Dec 19, 2014, at 5:52 AM, Sharath SN wrote: > > Can anyone suggest me how to add our own signatures. Are you using the ?@load-sigs? directive in a script or giving the ?-s? flag to bro on the command line to tell it to use the custom signature file? More documentation on signatures here: https://www.bro.org/sphinx/frameworks/signatures.html - Jon From luke at geekempire.com Fri Dec 19 07:52:52 2014 From: luke at geekempire.com (Mike Reeves) Date: Fri, 19 Dec 2014 10:52:52 -0500 Subject: [Bro] Bro SMB and segfaulting Message-ID: Hey all! I compiled Vlad?s topic from github to try it out. It runs fine on low speed environments but when I drop it on a high speed sensor it blows up. The link the sensor is on runs at between 600Mbit and 2.5Gbit. When I was doing the testing it was running at around 700Mbit and 1.7M PPS. Normal Bro 2.3.1 runs fine with no traffic being dropped at the ring. I am running pf_ring vanilla. The box runs 1 manager, 2 proxies, and 10 workers. The box is a dual 10 core HT with 128GB of RAM. All workers are pinned to real processors. The sensor starts and begins writing logs and then the disk IO goes to 100% and stops writing. It also starts dropping packets from the ring immediatly. Then the workers segfault and I have to stop it because when they go into crazy town they tie up the disk IO. The conn log and the syslog.log are much larger than the smb logs. I tried turning off logging on some of the other busy log files in case it is a disk IO problem. It didn?t make a difference. I write a LOT of logs on normal 2.3.1 and the IO usage is ver low. Has anyone had any luck running the SMB analyzer on high a high speed link? Is there anything I can provide to help figure out the root cause? Thanks Mike @TOoSmOotH From robin at icir.org Fri Dec 19 23:05:11 2014 From: robin at icir.org (Robin Sommer) Date: Fri, 19 Dec 2014 23:05:11 -0800 Subject: [Bro] Restructing of PktSrc.cc - missing FileName In-Reply-To: References: Message-ID: <20141220070511.GB94434@icir.org> On Fri, Dec 19, 2014 at 07:37 -0600, John Donnelly wrote: > bro.bif:4216:39: error: ?class iosource::PktDumper? has no member named > ?FileName? PktDumper::Path() is the new equivalent. The PktDumper class is now in src/iosource/PktDumper.h. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From michalpurzynski1 at gmail.com Sat Dec 20 02:38:50 2014 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Sat, 20 Dec 2014 11:38:50 +0100 Subject: [Bro] Bro SMB and segfaulting In-Reply-To: References: Message-ID: Hint - do you send syslog somewhere else i.e. to some external logging server? Is it possible that Bro can see the syslog packets? I'm asking, because you said syslog.log is large. Make sure you're not creating a positive feedback loop - Bro produces logs, logs gets send on the wire, Bro sees logs, logs these logs ;-) and that creates new logs about the logs, etc. On Fri, Dec 19, 2014 at 4:52 PM, Mike Reeves wrote: > Hey all! > > I compiled Vlad?s topic from github to try it out. It runs fine on low speed environments but when I drop it on a high speed sensor it blows up. The link the sensor is on runs at between 600Mbit and 2.5Gbit. When I was doing the testing it was running at around 700Mbit and 1.7M PPS. Normal Bro 2.3.1 runs fine with no traffic being dropped at the ring. I am running pf_ring vanilla. The box runs 1 manager, 2 proxies, and 10 workers. The box is a dual 10 core HT with 128GB of RAM. All workers are pinned to real processors. The sensor starts and begins writing logs and then the disk IO goes to 100% and stops writing. It also starts dropping packets from the ring immediatly. Then the workers segfault and I have to stop it because when they go into crazy town they tie up the disk IO. The conn log and the syslog.log are much larger than the smb logs. I tried turning off logging on some of the other busy log files in case it is a disk IO problem. It didn?t make a difference. I write a LOT of logs on normal 2.3.1 and the IO usage is ver low. > > Has anyone had any luck running the SMB analyzer on high a high speed link? Is there anything I can provide to help figure out the root cause? > > Thanks > > Mike > @TOoSmOotH > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jdonnelly at dyn.com Sat Dec 20 05:49:43 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Sat, 20 Dec 2014 07:49:43 -0600 Subject: [Bro] Bro SMB and segfaulting In-Reply-To: References: Message-ID: You could build Bro with debug. rm -rf build. ./configure --enable-debug make install Start bro with your setup and use gdb to attach to the running process : gdb -p When it segfaults .. gdb will wake up and you can post the trace using "t" command (stack trace) . On Fri, Dec 19, 2014 at 9:52 AM, Mike Reeves wrote: > > Hey all! > > I compiled Vlad?s topic from github to try it out. It runs fine on > low speed environments but when I drop it on a high speed sensor it blows > up. The link the sensor is on runs at between 600Mbit and 2.5Gbit. When I > was doing the testing it was running at around 700Mbit and 1.7M PPS. Normal > Bro 2.3.1 runs fine with no traffic being dropped at the ring. I am running > pf_ring vanilla. The box runs 1 manager, 2 proxies, and 10 workers. The box > is a dual 10 core HT with 128GB of RAM. All workers are pinned to real > processors. The sensor starts and begins writing logs and then the disk IO > goes to 100% and stops writing. It also starts dropping packets from the > ring immediatly. Then the workers segfault and I have to stop it because > when they go into crazy town they tie up the disk IO. The conn log and the > syslog.log are much larger than the smb logs. I tried turning off logging > on some of the other busy log files in case it is a disk IO problem. It > didn?t make a difference. I write a LOT of logs on normal 2.3.1 and the IO > usage is ver low. > > Has anyone had any luck running the SMB analyzer on high a high speed > link? Is there anything I can provide to help figure out the root cause? > > Thanks > > Mike > @TOoSmOotH > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141220/7089b82d/attachment.html From shrtsns at gmail.com Mon Dec 22 00:25:16 2014 From: shrtsns at gmail.com (Sharath SN) Date: Mon, 22 Dec 2014 13:55:16 +0530 Subject: [Bro] signature configuration In-Reply-To: <8B4D1F9E-12CE-4777-BC41-202AD4BAB071@illinois.edu> References: <8B4D1F9E-12CE-4777-BC41-202AD4BAB071@illinois.edu> Message-ID: Hi Jon, Thanks for quick reply. Now I could able to generate signatures in signature log files by using @load-sigs. But I got stuck with another problem i.e., *specifing an payload for "raw packet data"..*? I had tried with following simple code "*payload /\x14\x03/* " but logs are not generating even if our traffics are having same payloads. Can you suggect me how to over come this issue.. On Fri, Dec 19, 2014 at 8:54 PM, Siwek, Jon wrote: > > > On Dec 19, 2014, at 5:52 AM, Sharath SN wrote: > > > > Can anyone suggest me how to add our own signatures. > > Are you using the ?@load-sigs? directive in a script or giving the ?-s? > flag to bro on the command line to tell it to use the custom signature > file? More documentation on signatures here: > > https://www.bro.org/sphinx/frameworks/signatures.html > > - Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141222/2e3bc791/attachment.html From vitologrillo at gmail.com Mon Dec 22 03:46:26 2014 From: vitologrillo at gmail.com (Vito Logrillo) Date: Mon, 22 Dec 2014 12:46:26 +0100 Subject: [Bro] How to manage a dynamic vector Message-ID: Hi, i have some problems with a code like this: ------- code.bro ----- type known_services_buffer: record { host: addr; port_num: port; port_proto: transport_proto; service: set[string]; }; type known_services_buffer_vec: vector of known_services_buffer; global known_services_buf_vec = known_services_buffer_vec(); global count_a: int = 0; event Known::log_known_services(rec: Known::ServicesInfo) &priority=5 { ...do something..... known_services_buf_vec[count_a]$host = rec$host; known_services_buf_vec[count_a]$port_num = rec$port_num; known_services_buf_vec[count_a]$port_proto = rec$port_proto; known_services_buf_vec[count_a]$service = rec$service; count_a = count_a + 1; } event Conn::log_conn(rec: Conn::Info) { print (count_a); for (idx in known_services_buf_vec) { ...do something.... } } ------ code.bro ------- My variable count_a is incremented each time the event Known::log_known_services is executed, but the last "for" in Conn::log_conn is never executed because known_services_buf_vec is always a void vector. What's wrong? Any suggestion? Thanks From anthony.kasza at gmail.com Mon Dec 22 08:19:21 2014 From: anthony.kasza at gmail.com (anthony kasza) Date: Mon, 22 Dec 2014 08:19:21 -0800 Subject: [Bro] How to manage a dynamic vector In-Reply-To: References: Message-ID: Where are you defining your new type and declaring your global variables? Across multiple files? Within export blocks? -AK On Dec 22, 2014 6:56 AM, "Vito Logrillo" wrote: > Hi, > i have some problems with a code like this: > > ------- code.bro ----- > > type known_services_buffer: record { > host: addr; > port_num: port; > port_proto: transport_proto; > service: set[string]; > }; > > type known_services_buffer_vec: vector of known_services_buffer; > global known_services_buf_vec = known_services_buffer_vec(); > global count_a: int = 0; > > event Known::log_known_services(rec: Known::ServicesInfo) &priority=5 > { > ...do something..... > known_services_buf_vec[count_a]$host = rec$host; > known_services_buf_vec[count_a]$port_num = rec$port_num; > known_services_buf_vec[count_a]$port_proto = rec$port_proto; > known_services_buf_vec[count_a]$service = rec$service; > count_a = count_a + 1; > } > > event Conn::log_conn(rec: Conn::Info) > { > print (count_a); > for (idx in known_services_buf_vec) > { > ...do something.... > } > } > > > ------ code.bro ------- > > My variable count_a is incremented each time the event > Known::log_known_services is executed, but the last "for" in > Conn::log_conn is never executed because known_services_buf_vec is > always a void vector. > What's wrong? Any suggestion? > Thanks > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141222/c73179ee/attachment.html From seth at icir.org Mon Dec 22 08:38:46 2014 From: seth at icir.org (Seth Hall) Date: Mon, 22 Dec 2014 11:38:46 -0500 Subject: [Bro] How to manage a dynamic vector In-Reply-To: References: Message-ID: > On Dec 22, 2014, at 6:46 AM, Vito Logrillo wrote: > > My variable count_a is incremented each time the event > Known::log_known_services is executed, but the last "for" in > Conn::log_conn is never executed because known_services_buf_vec is > always a void vector. Vectors are a little goofy in Bro still. They are statically sized so you are creating an instance of a zero length vector. There is a built in function named ?resize? that you may want to use to provide a size to your vector. https://www.bro.org/sphinx-git/scripts/base/bif/bro.bif.bro.html#id-resize Alternately, you may want to use something like a table[count] of known_services_buffer .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From sovakah at gmail.com Sun Dec 28 08:44:24 2014 From: sovakah at gmail.com (Savakh S) Date: Sun, 28 Dec 2014 17:44:24 +0100 Subject: [Bro] General questions about Bro's capabilities Message-ID: Hello, How can Bro be used to set a network profile and works as a behavior IDS ? Does Bro need to understand L7 protocols ? Is it compulsory ? If not, what would be its capabilities ? Thank for your answers. Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141228/b7e862e5/attachment.html From anthony.kasza at gmail.com Sun Dec 28 15:43:24 2014 From: anthony.kasza at gmail.com (anthony kasza) Date: Sun, 28 Dec 2014 15:43:24 -0800 Subject: [Bro] General questions about Bro's capabilities In-Reply-To: References: Message-ID: Bro has an understanding of many of the more popular layer 7 protocols and is able to generate logs based on what it sees. One rather simple way to create a baseline for a network could be to monitor the network for a period of time and observe the logs Bro generates. -AK On Dec 28, 2014 9:01 AM, "Savakh S" wrote: > Hello, > > How can Bro be used to set a network profile and works as a behavior IDS ? > Does Bro need to understand L7 protocols ? Is it compulsory ? If not, what > would be its capabilities ? > > Thank for your answers. > > Regards. > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141228/92eca383/attachment.html From hlin33 at illinois.edu Sun Dec 28 19:01:21 2014 From: hlin33 at illinois.edu (Hui Lin (Hugo) ) Date: Sun, 28 Dec 2014 21:01:21 -0600 Subject: [Bro] General questions about Bro's capabilities In-Reply-To: <02e3c02d57f74fbaa7db2bc0ae0f4e6a@CHIHT2.ad.uillinois.edu> References: <02e3c02d57f74fbaa7db2bc0ae0f4e6a@CHIHT2.ad.uillinois.edu> Message-ID: Hi Savakh, The analyzer for specific protocols are included in src\analyzer\protocol of the source code repository . You may also take a look at Binpac ( https://www.bro.org/sphinx/components/binpac/README.html). Most current application layer protocols based on TCP or UDP are added in Bro by using Binpac. With the help of Binpac, it is also easy to add new protocol analyzer in Bro. Hope this helps. Hugo On Sun, Dec 28, 2014 at 5:43 PM, anthony kasza wrote: > Bro has an understanding of many of the more popular layer 7 protocols > and is able to generate logs based on what it sees. > One rather simple way to create a baseline for a network could be to > monitor the network for a period of time and observe the logs Bro generates. > > -AK > On Dec 28, 2014 9:01 AM, "Savakh S" wrote: > >> Hello, >> >> How can Bro be used to set a network profile and works as a behavior IDS >> ? >> Does Bro need to understand L7 protocols ? Is it compulsory ? If not, >> what would be its capabilities ? >> >> Thank for your answers. >> >> Regards. >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > -- Hui Lin PhD Candidate, Research Assistant Electrical and Computer Engineering Department University of Illinois at Urbana-Champaign -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141228/f91f7a85/attachment.html From jdonnelly at dyn.com Tue Dec 30 13:09:13 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Tue, 30 Dec 2014 15:09:13 -0600 Subject: [Bro] Hex and bit expressions in Bro scripts ? Message-ID: Hi, Can bro scripts not handle binary and bit expressions ? local temp = 32; If ( temp & 0x20) { } I didn't see a list of expressions in the online guide. Also , I tried to add binary variables in the Ascii (tab separated files) it reads and it complained too: #field count 1 0x20 Pointers welcomed. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141230/a7b870b4/attachment.html From anthony.kasza at gmail.com Tue Dec 30 14:07:31 2014 From: anthony.kasza at gmail.com (anthony kasza) Date: Tue, 30 Dec 2014 14:07:31 -0800 Subject: [Bro] Hex and bit expressions in Bro scripts ? In-Reply-To: References: Message-ID: Nope. Neither are supported in the bro language. You could write a wrapper function and compile it into the core or a core plugin. -AK On Dec 30, 2014 1:14 PM, "John Donnelly" wrote: > Hi, > > Can bro scripts not handle binary and bit expressions ? > > local temp = 32; > > If ( temp & 0x20) { > > } > > I didn't see a list of expressions in the online guide. > > Also , I tried to add binary variables in the Ascii (tab separated > files) it reads and it complained too: > > > #field count > 1 0x20 > > > Pointers welcomed. > > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141230/e5474482/attachment.html