From fatema.bannatwala at gmail.com Wed Mar 1 09:19:22 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Wed, 1 Mar 2017 12:19:22 -0500 Subject: [Bro] Various OSs detection using Bro Message-ID: I was wondering if anyone has tried detecting different OSs using Bro. I know Bro ships with windows version detection script, and to add to the OS detection, I have written two more scripts to detect MacOS and iOS. Next, was trying to write something to detect Linux OSs, for Android phones and other PCs that might be running some kind of Linux OS. So, before trying to re-invent the wheel, wanted to ask if some one trying to address similar use-case, and if would like to share the scripts, or if someone has any pointers to any specific way of detection and have any pointers in that direction! :) Thanks, Fatema. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170301/9bf40d67/attachment.html From zeolla at gmail.com Wed Mar 1 10:27:18 2017 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Wed, 01 Mar 2017 18:27:18 +0000 Subject: [Bro] Various OSs detection using Bro In-Reply-To: References: Message-ID: https://github.com/bro/bro/blob/master/scripts/base/misc/p0f.fp ? We used it before and it was very false positive prone, which is actually why Vlad worked on the Windows detection script in the first place. Jon On Wed, Mar 1, 2017 at 12:20 PM fatema bannatwala < fatema.bannatwala at gmail.com> wrote: > I was wondering if anyone has tried detecting different OSs using Bro. > I know Bro ships with windows version detection script, and to add to the > OS > detection, I have written two more scripts to detect MacOS and iOS. > > Next, was trying to write something to detect Linux OSs, for Android phones > and other PCs that might be running some kind of Linux OS. > > So, before trying to re-invent the wheel, wanted to ask if some one trying > to address > similar use-case, and if would like to share the scripts, or if someone > has any pointers to > any specific way of detection and have any pointers in that direction! :) > > Thanks, > Fatema. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Jon Sent from my mobile device -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170301/d4662326/attachment.html From deneaulp at bc.edu Wed Mar 1 11:30:19 2017 From: deneaulp at bc.edu (Phillip Deneault) Date: Wed, 1 Mar 2017 14:30:19 -0500 Subject: [Bro] Trying to RTFM Message-ID: I know someone will make fun of me for this but... I'm _trying_ to read the manual... and I'm getting a 404. https://www.bro.org/sphinx/index.html Thanks, Phil -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170301/3b259c66/attachment-0001.html From graham at american.edu Wed Mar 1 11:37:48 2017 From: graham at american.edu (Isabelle Grey) Date: Wed, 1 Mar 2017 19:37:48 +0000 Subject: [Bro] Trying to RTFM In-Reply-To: References: Message-ID: Also getting a 404 here. --- Isabelle Grey pronouns: she/her Information Security Engineer American University ________________________________________ From: bro-bounces at bro.org on behalf of Phillip Deneault Sent: 01 March 2017 14:30:19 To: bro at bro.org Subject: [Bro] Trying to RTFM I know someone will make fun of me for this but... I'm _trying_ to read the manual... and I'm getting a 404. https://www.bro.org/sphinx/index.html Thanks, Phil From robin at icir.org Wed Mar 1 14:38:17 2017 From: robin at icir.org (Robin Sommer) Date: Wed, 1 Mar 2017 14:38:17 -0800 Subject: [Bro] Trying to RTFM In-Reply-To: References: Message-ID: <20170301223816.GE32611@icir.org> On Wed, Mar 01, 2017 at 19:37 +0000, Isabelle Grey wrote: > Also getting a 404 here. Working on it, should be fixed shortly. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From bill.de.ping at gmail.com Thu Mar 2 01:33:46 2017 From: bill.de.ping at gmail.com (william de ping) Date: Thu, 2 Mar 2017 11:33:46 +0200 Subject: [Bro] feeding bro cluster with parameters without restarting it Message-ID: Hello all, I know that I can update bro parameters using the INPUT framework (reading input files and updating a table for instance). The thing is that the INPUT framework (STREAM) and generally reading from files is relatively slow. Can I add elements to a table inside bro from lets say a syslog message or any other faster method ? thanks B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170302/1da1815f/attachment.html From bill.de.ping at gmail.com Thu Mar 2 02:45:20 2017 From: bill.de.ping at gmail.com (william de ping) Date: Thu, 2 Mar 2017 12:45:20 +0200 Subject: [Bro] feeding bro cluster with parameters without restarting it In-Reply-To: References: Message-ID: I am also now reading about PYBROKER. Can anyone provide me with an example of how can I use PYBROKER and a python script to update a table inside a running bro cluster (bro workers to be exact) ? Thanks B On Thu, Mar 2, 2017 at 11:33 AM, william de ping wrote: > Hello all, > > I know that I can update bro parameters using the INPUT framework (reading > input files and updating a table for instance). > > The thing is that the INPUT framework (STREAM) and generally reading from > files is relatively slow. > > Can I add elements to a table inside bro from lets say a syslog message or > any other faster method ? > > thanks > B > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170302/db34eb89/attachment.html From jan.grashoefer at gmail.com Thu Mar 2 02:45:57 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Thu, 2 Mar 2017 11:45:57 +0100 Subject: [Bro] feeding bro cluster with parameters without restarting it In-Reply-To: References: Message-ID: > Can I add elements to a table inside bro from lets say a syslog message or > any other faster method ? There is a syslog analyzer you could theoretically use ( https://www.bro.org/sphinx/script-reference/proto-analyzers.html#bro-syslog) but I would strongly discourage mixing monitored traffic and control traffic. If you want to interact with Bro, broker might be of interest for you (https://www.bro.org/sphinx/components/broker/broker-manual.html). For example, I have used broker to write a python script that allows to delete intel items. Jan From seth at corelight.com Thu Mar 2 07:13:59 2017 From: seth at corelight.com (Seth Hall) Date: Thu, 2 Mar 2017 10:13:59 -0500 Subject: [Bro] feeding bro cluster with parameters without restarting it In-Reply-To: References: Message-ID: > On Mar 2, 2017, at 5:45 AM, william de ping wrote: > > I am also now reading about PYBROKER. > > Can anyone provide me with an example of how can I use PYBROKER and a python script to update a table inside a running bro cluster (bro workers to be exact) ? There is still ongoing work on the upcoming config framework which will enable dynamic reconfiguration of lots of parts of Bro. We built it within Corelight and an initial implementation has already been pushed into a branch in the Bro repository but it's going to be improved before being merged into master. I would still recommend playing with pybroker if you're interested in it though. If you can solve a point solution more easily and quickly for yourself you shouldn't let upcoming yet imcomplete work stop you! :) .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From jazoff at illinois.edu Thu Mar 2 07:27:06 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 2 Mar 2017 15:27:06 +0000 Subject: [Bro] feeding bro cluster with parameters without restarting it In-Reply-To: References: Message-ID: > On Mar 2, 2017, at 4:33 AM, william de ping wrote: > > The thing is that the INPUT framework (STREAM) and generally reading from files is relatively slow. What exactly do you mean by relatively slow? How large are these tables that you are reading? -- - Justin Azoff From johanna at icir.org Thu Mar 2 07:33:28 2017 From: johanna at icir.org (Johanna Amann) Date: Thu, 02 Mar 2017 07:33:28 -0800 Subject: [Bro] feeding bro cluster with parameters without restarting it In-Reply-To: References: Message-ID: <13172D2B-6C27-49ED-8015-A02F88951191@icir.org> Indeed, I was also going to ask that. We did some performance measurements when we first wrote it - and it actually is quite fast. There only is a relatively low amount of components between the input reader and it storing things in a table; I cannot be 100% sure, but I doubt that other ingestion methods can be much faster. (I actually doubt that they will be faster at all). Johanna On 2 Mar 2017, at 7:27, Azoff, Justin S wrote: >> On Mar 2, 2017, at 4:33 AM, william de ping >> wrote: >> >> The thing is that the INPUT framework (STREAM) and generally reading >> from files is relatively slow. > > What exactly do you mean by relatively slow? How large are these > tables that you are reading? > > -- > - Justin Azoff > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jdopheid at illinois.edu Thu Mar 2 09:10:19 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Thu, 2 Mar 2017 17:10:19 +0000 Subject: [Bro] =?utf-8?q?BroCon_=E2=80=9917=3A_Registration_is_open!?= Message-ID: <876DB55E-1A3A-4CEC-BB48-C75F1BEB02BD@illinois.edu> Bro Community, BroCon ?17 will occur on Tuesday, September 12th - Thursday, September 14th at the National Center for Supercomputing Applications in Urbana, IL. See our event page: https://www.bro.org/community/brocon2017.html Early bird registration is open! CFP is open! Don't forget to book your hotel. Interested in sponsoring BroCon? Contact us at info at bro.org for more information. Thank you for your continued support, and see you in September! Regards, The Bro Project ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From dwdixon at umich.edu Thu Mar 2 14:20:37 2017 From: dwdixon at umich.edu (Drew Dixon) Date: Thu, 2 Mar 2017 17:20:37 -0500 Subject: [Bro] Issue with Bro reporting dropped packets In-Reply-To: References: Message-ID: First I think the recommended number of workings is something like number of *real* cores (not counting hyperthreading) -2 so for 8 *real* cores you would use 6 workers, if you have 16 *real* cores you probably want closer to 14 workers if this is a dedicated bro box. Maybe try bumping up your number of workers and enabling cpu pinning if you haven't done so. Have you reviewed everything located here? : https://www.bro.org/documentation/faq.html#how-can-i-reduce- the-amount-of-captureloss-or-dropped-packets-notices Specifically a few things come to mind...I know you mentioned NIC settings but are you sure you disabled all the NIC offloading features using ethtool?, more detail on that at this link: http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not-full.html Also, wouldn't hurt to double check the the pf_ring kernel module is loaded/loading staying loaded? If you patch the server and the kernel gets updated unless you have something automated to reload/reinstall the pf_ring module you will probably need to reload the pf_ring module for the new kernel... Also, did you configure the number of ring slots for PF_RING ? Check to be sure that /etc/modprobe.d/pf_ring.conf exists for your PF_RING installation...this is where you will configure the number of ring slots for PF_RING, the default is 4096 I believe but on busy networks this needs to be increased as appropriate (in increments of 4096)...the max value is 65534. I would try that if you've tried everything else at the first link above to no avail... This is also a great resource re: PF_RING and number of ring slots: https://groups.google.com/forum/#!topic/security-onion/zu7U7U9pBT8 Hope this helps, -Drew On Tue, Feb 28, 2017 at 3:20 PM, Espresso Beanies wrote: > Hi, > > I'm trying to troubleshoot a Bro IDS that is experiencing capture loss > with dropped packets. The machine I'm using has a 16-core Intel Xeon > processor, 96Gb RAM, and an Intel NIC. I have 3 Bro workers with CPU > affinity enabled and I'm using the pf_ring module on CentOS with no custom > Bro scripts running. All of my processors are running at 99% utilization. > > According to my operating system, I'm dropping about 8000 packets over the > course of a day on a 300-400Mbps network. According to Bro capstats, I am > dropping about the same number of packets I'm receiving, sometimes more > than I receive. My capture_loss.log shows my workers lose about 30-50% > packets and my manager and proxy, 70-90%. I can provide any configurations > or screenshots if necessary. > > I'm trying to troubleshoot where the issue lies. I initially installed Bro > with all the recommended packages (tcmalloc, etc...) and the pf_ring module > and I can see that Bro is using it. At this point, everything I see is > pointing to an application issue and I'm running Bro version 2.5. I had the > same issue with Bro v.2.4 as well. > > Short of tweaking OS kernel and NIC card settings, I'm not sure where else > I could try to reduce my packet drop count in Bro. Any recommendations? > > Thanks, > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170302/fc0d2531/attachment.html From espressobeanies at gmail.com Thu Mar 2 15:26:21 2017 From: espressobeanies at gmail.com (Espresso Beanies) Date: Thu, 2 Mar 2017 18:26:21 -0500 Subject: [Bro] Issue with Bro reporting dropped packets In-Reply-To: References: Message-ID: Hi Drew, I definitely did. I tried asking earlier if there was a difference between adding more Bro workers via [worker-1],[worker-2],[worker-3],etc... vs lb_procs 'N' but didn't receive a response. I tried both methods in the node.cfg file with little to no noticeable performance impact. I'm definitely using CPU affinity. Yep, I tried disabling as many NIC offloading features as I could and I'd like to mention I found a more comprehensive list of NIC offload disabling for Suricata that may be applied to Bro as well: http://pevma.blogspot.com/2014/03/suricata-prepearing-10gbps-network.html Right now I'm working with ntop on an issue with PF_RING because their repo and rpm packages are not correctly loading the pf_ring module into the kernel and I get errors when attempting to run this following command to validate my PF_RING install after a successful Bro installation: bro -N Bro::PF_RING I also have an issue where the pf_ring repo packages are interfering with a Bro reinstall because Bro no longer recognizes the libpcap library. Still working that one out with ntop. > Have you reviewed everything located here? : > > https://www.bro.org/documentation/faq.html#how-can-i-reduce- > the-amount-of-captureloss-or-dropped-packets-notices > > Specifically a few things come to mind...I know you mentioned NIC settings > but are you sure you disabled all the NIC offloading features using > ethtool?, more detail on that at this link: > > http://securityonion.blogspot.com/2011/10/when-is-full- > packet-capture-not-full.html > > Also, wouldn't hurt to double check the the pf_ring kernel module is > loaded/loading staying loaded? If you patch the server and the kernel gets > updated unless you have something automated to reload/reinstall the pf_ring > module you will probably need to reload the pf_ring module for the new > kernel... > > Also, did you configure the number of ring slots for PF_RING ? > > Check to be sure that /etc/modprobe.d/pf_ring.conf exists for your PF_RING > installation...this is where you will configure the number of ring slots > for PF_RING, the default is 4096 I believe but on busy networks this needs > to be increased as appropriate (in increments of 4096)...the max value is > 65534. I would try that if you've tried everything else at the first link > above to no avail... > > This is also a great resource re: PF_RING and number of ring slots: > > https://groups.google.com/forum/#!topic/security-onion/zu7U7U9pBT8 > > Hope this helps, > > -Drew > > On Tue, Feb 28, 2017 at 3:20 PM, Espresso Beanies < > espressobeanies at gmail.com> wrote: > >> Hi, >> >> I'm trying to troubleshoot a Bro IDS that is experiencing capture loss >> with dropped packets. The machine I'm using has a 16-core Intel Xeon >> processor, 96Gb RAM, and an Intel NIC. I have 3 Bro workers with CPU >> affinity enabled and I'm using the pf_ring module on CentOS with no custom >> Bro scripts running. All of my processors are running at 99% utilization. >> >> According to my operating system, I'm dropping about 8000 packets over >> the course of a day on a 300-400Mbps network. According to Bro capstats, I >> am dropping about the same number of packets I'm receiving, sometimes more >> than I receive. My capture_loss.log shows my workers lose about 30-50% >> packets and my manager and proxy, 70-90%. I can provide any configurations >> or screenshots if necessary. >> >> I'm trying to troubleshoot where the issue lies. I initially installed >> Bro with all the recommended packages (tcmalloc, etc...) and the pf_ring >> module and I can see that Bro is using it. At this point, everything I see >> is pointing to an application issue and I'm running Bro version 2.5. I had >> the same issue with Bro v.2.4 as well. >> >> Short of tweaking OS kernel and NIC card settings, I'm not sure where >> else I could try to reduce my packet drop count in Bro. Any recommendations? >> >> Thanks, >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170302/dbe70c9d/attachment.html From seth at corelight.com Fri Mar 3 05:50:35 2017 From: seth at corelight.com (Seth Hall) Date: Fri, 3 Mar 2017 08:50:35 -0500 Subject: [Bro] Issue with Bro reporting dropped packets In-Reply-To: References: Message-ID: > On Mar 2, 2017, at 6:26 PM, Espresso Beanies wrote: > > I definitely did. I tried asking earlier if there was a difference between adding more Bro workers via [worker-1],[worker-2],[worker-3],etc... vs lb_procs 'N' but didn't receive a response. I tried both methods in the node.cfg file with little to no noticeable performance impact. I'm definitely using CPU affinity. There is no difference. The whole lb_procs thing arose because we used to have people with huge node.cfg files because they were running a lot of workers. Adding the lb_procs mechanism gave them a short hand to not have to configure each and every worker. .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From espressobeanies at gmail.com Fri Mar 3 08:26:07 2017 From: espressobeanies at gmail.com (Espresso Beanies) Date: Fri, 3 Mar 2017 11:26:07 -0500 Subject: [Bro] Issue with Bro reporting dropped packets In-Reply-To: References: Message-ID: Hi Seth, That makes more sense. Thank you for the background. I was able to find and resolve the issue I was experiencing with ntop. Thank you both. Sincerely, On Fri, Mar 3, 2017 at 8:50 AM, Seth Hall wrote: > > > On Mar 2, 2017, at 6:26 PM, Espresso Beanies > wrote: > > > > I definitely did. I tried asking earlier if there was a difference > between adding more Bro workers via [worker-1],[worker-2],[worker-3],etc... > vs lb_procs 'N' but didn't receive a response. I tried both methods in the > node.cfg file with little to no noticeable performance impact. I'm > definitely using CPU affinity. > > There is no difference. The whole lb_procs thing arose because we used to > have people with huge node.cfg files because they were running a lot of > workers. Adding the lb_procs mechanism gave them a short hand to not have > to configure each and every worker. > > .Seth > > -- > Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170303/b660d1b4/attachment-0001.html From bond.masuda at jlbond.com Fri Mar 3 14:00:21 2017 From: bond.masuda at jlbond.com (Bond Masuda) Date: Fri, 3 Mar 2017 14:00:21 -0800 Subject: [Bro] building bro w/ dynamic libpcap? Message-ID: Hello, Trying to build bro with pf_ring's libpcap, but the 'configure' script seems to choose static libpcap.a over libpcap.so: $ ./configure --with-pcap=/usr/local Build Directory : build Source Directory: /home/bmasuda/RPMBUILD/BUILD/bro-2.4.1 -- Found sed: /usr/bin/sed -- Found Perl: /usr/bin/perl (found version "5.16.3") -- Found FLEX: 2.5.37 -- Found BISON: /usr/bin/bison -- Found PCAP: /usr/local/lib/libpcap.a pf_ring is installed in /usr/local/lib: $ ls -l /usr/local/lib total 3536 drwxr-xr-x. 2 root root 96 Mar 3 07:35 daq -rw-r--r--. 1 root root 461618 Mar 2 21:13 libpcap.a lrwxrwxrwx. 1 root root 16 Mar 3 07:35 libpcap.so.1 -> libpcap.so.1.7.4 -rwxr-xr-x. 1 root root 1452912 Mar 2 21:13 libpcap.so.1.7.4 -rw-r--r--. 1 root root 692966 Mar 2 21:13 libpfring.a -rwxr-xr-x. 1 root root 517664 Mar 2 21:13 libpfring.so lrwxrwxrwx. 1 root root 17 Mar 3 07:35 libsfbpf.so.0 -> libsfbpf.so.0.0.1 -rwxrwxr-x. 1 root root 486933 Oct 10 2015 libsfbpf.so.0.0.1 I would like for bro to be linked dynamically with the libpcap.so instead of libpcap.a. How can I specify this? Above example is with bro 2.4.1, but same thing with 2.5 as well. Thanks, Bond From johanna at icir.org Fri Mar 3 15:12:41 2017 From: johanna at icir.org (Johanna Amann) Date: Fri, 3 Mar 2017 15:12:41 -0800 Subject: [Bro] building bro w/ dynamic libpcap? In-Reply-To: References: Message-ID: <20170303231241.ix6t36tksildb6ka@Beezling.local> You seem to be missing the libcap.so -> libpcap.so.1(.7.4) symlink. So that would be expected behavior in this case (I am not even sure you can easily tell a linker to link against a file with a .so.1 file extension). Johanna On Fri, Mar 03, 2017 at 02:00:21PM -0800, Bond Masuda wrote: > Hello, > > Trying to build bro with pf_ring's libpcap, but the 'configure' script > seems to choose static libpcap.a over libpcap.so: > > $ ./configure --with-pcap=/usr/local > Build Directory : build > Source Directory: /home/bmasuda/RPMBUILD/BUILD/bro-2.4.1 > -- Found sed: /usr/bin/sed > -- Found Perl: /usr/bin/perl (found version "5.16.3") > -- Found FLEX: 2.5.37 > -- Found BISON: /usr/bin/bison > -- Found PCAP: /usr/local/lib/libpcap.a > > pf_ring is installed in /usr/local/lib: > > $ ls -l /usr/local/lib > total 3536 > drwxr-xr-x. 2 root root 96 Mar 3 07:35 daq > -rw-r--r--. 1 root root 461618 Mar 2 21:13 libpcap.a > lrwxrwxrwx. 1 root root 16 Mar 3 07:35 libpcap.so.1 -> > libpcap.so.1.7.4 > -rwxr-xr-x. 1 root root 1452912 Mar 2 21:13 libpcap.so.1.7.4 > -rw-r--r--. 1 root root 692966 Mar 2 21:13 libpfring.a > -rwxr-xr-x. 1 root root 517664 Mar 2 21:13 libpfring.so > lrwxrwxrwx. 1 root root 17 Mar 3 07:35 libsfbpf.so.0 -> > libsfbpf.so.0.0.1 > -rwxrwxr-x. 1 root root 486933 Oct 10 2015 libsfbpf.so.0.0.1 > > I would like for bro to be linked dynamically with the libpcap.so > instead of libpcap.a. How can I specify this? > > Above example is with bro 2.4.1, but same thing with 2.5 as well. > > Thanks, > Bond > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From johanna at icir.org Fri Mar 3 15:16:15 2017 From: johanna at icir.org (Johanna Amann) Date: Fri, 3 Mar 2017 15:16:15 -0800 Subject: [Bro] Bro Detections and Compliance Questions In-Reply-To: References: Message-ID: <20170303231615.c2b6ing3shtof2oe@Beezling.local> On Thu, Feb 23, 2017 at 02:20:37PM +0000, Andrew Dellana wrote: > When a bro script detects something, how can you go about resolving the > issues that caused it (assuming it wasn't noise that caused it)? Is > there something that I change in Bro or is this something that would be > covered in the corporate compliance / security? You have to handle that either outside of Bro, or use something like netcontrol to change your network settings (if appropriate). > Following up with that what is the best practice to analyze the packet > captures from Bro to determine if there is an actual issue? I am > currently looking into Splunk as a log parser. There is a wide variety of tools used for the job, but Splunk is certainly popular. Others just operate directly on the logfiles; an ELK stack might be another solution. Johanna From johanna at icir.org Fri Mar 3 15:18:51 2017 From: johanna at icir.org (Johanna Amann) Date: Fri, 3 Mar 2017 15:18:51 -0800 Subject: [Bro] Netmap Seg faults In-Reply-To: <0555C89E-4F7B-4FE0-AAEC-5EFBF06C7FCF@pingtrip.com> References: <0555C89E-4F7B-4FE0-AAEC-5EFBF06C7FCF@pingtrip.com> Message-ID: <20170303231851.p4stzwntxqgcwxha@Beezling.local> Is this on FreeBSD or on Linux? :) In any case, on FreeBSD I hae not seen segfaults with Bro & Netmap on a broctl stop; however, some machines of us still like to crash when starting up Bro in the same configuration. (Crash as in apparently kernel-crash and reboot). Johanna On Thu, Feb 16, 2017 at 11:13:41PM -0500, Dave Crawford wrote: > > > On Feb 16, 2017, at 9:50 PM, Dave Crawford wrote: > > > > Has anyone experienced segfaults with Bro + Netmap when executing a ?broctrl stop'? > > > > 1487299650.431866 818913 packets received on interface bro}3, 0 dropped > > /opt/bro/share/broctl/scripts/run-bro: line 107: 4821 Segmentation fault nohup "$mybro" "$@" > > > > > > Also seeing these messages in dmesg: > > [ 8113.725495] bro[2098]: segfault at 0 ip 00007f7695f360f7 sp 00007fffe8969360 error 4 in libtcmalloc.so.4.2.2[7f7695eeb000+98000] > [ 8113.766085] bro[2088]: segfault at 0 ip 00007f6584ff40f7 sp 00007ffc40fcf7f0 error 4 in libtcmalloc.so.4.2.2[7f6584fa9000+98000] > [29084.773876] bro[4823]: segfault at 0 ip 00007f116704d0f7 sp 00007ffe93723430 error 4 in libtcmalloc.so.4.2.2[7f1167002000+98000] > [29084.787171] bro[4821]: segfault at 0 ip 00007f013f9610f7 sp 00007ffe0d0a9950 error 4 in libtcmalloc.so.4.2.2[7f013f916000+98000] > > -Dave > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Fri Mar 3 15:20:03 2017 From: johanna at icir.org (Johanna Amann) Date: Fri, 3 Mar 2017 15:20:03 -0800 Subject: [Bro] Question on PacketFilter::DroppedPackets In-Reply-To: References: Message-ID: <20170303232003.gdadydjsr3hxrnuc@Beezling.local> On Thu, Feb 16, 2017 at 02:05:19PM -0500, Espresso Beanies wrote: > Hi, > > It seems no matter what I do, I still get these notices > "PacketFilter::DroppedPackets". I created more workers, but I have a > question about creating new workers via using an existing worker to capture > on the same interface using the "lb_procs" method to up the number of > "sub-threads?" for multi-CPU processing. What advantage does a new worker > give me over "lb_procs"? It is the same, lb_procs also basically just creates new workers; the advantage is that it does things automatically for you that you would have to configure by hand otherwhise. Johanna From zeolla at gmail.com Fri Mar 3 15:28:37 2017 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Fri, 03 Mar 2017 23:28:37 +0000 Subject: [Bro] Bro Detections and Compliance Questions In-Reply-To: <20170303231615.c2b6ing3shtof2oe@Beezling.local> References: <20170303231615.c2b6ing3shtof2oe@Beezling.local> Message-ID: Another solution could be Apache Metron (previously OpenSOC). It handles pcap and bro logs natively, among other things. Jon On Fri, Mar 3, 2017, 6:24 PM Johanna Amann wrote: > On Thu, Feb 23, 2017 at 02:20:37PM +0000, Andrew Dellana wrote: > > When a bro script detects something, how can you go about resolving the > > issues that caused it (assuming it wasn't noise that caused it)? Is > > there something that I change in Bro or is this something that would be > > covered in the corporate compliance / security? > > You have to handle that either outside of Bro, or use something like > netcontrol to change your network settings (if appropriate). > > > Following up with that what is the best practice to analyze the packet > > captures from Bro to determine if there is an actual issue? I am > > currently looking into Splunk as a log parser. > > There is a wide variety of tools used for the job, but Splunk is certainly > popular. Others just operate directly on the logfiles; an ELK stack might > be another solution. > > Johanna > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Jon Sent from my mobile device -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170303/5bd9ca6b/attachment.html From johanna at icir.org Fri Mar 3 15:34:31 2017 From: johanna at icir.org (Johanna Amann) Date: Fri, 3 Mar 2017 15:34:31 -0800 Subject: [Bro] SMB In-Reply-To: <592228F4D0C8504187F2F76658040CB6DFE39D1A@HOT-MAILBOX-02.HOT.NET.IL> References: <592228F4D0C8504187F2F76658040CB6DFE23DFF@HOT-MAILBOX-02.HOT.NET.IL> <592228F4D0C8504187F2F76658040CB6DFE39D1A@HOT-MAILBOX-02.HOT.NET.IL> Message-ID: <20170303233421.xdrwwipopoaroftp@Beezling.local> Hi, I might be mistaken here, but I think that datastreams in smb can use multiple tcp connections. For individual files, you should be able to look at files log; if you want an aggregate, you will probably have to script that yourself. Johanna On Thu, Feb 16, 2017 at 07:35:58AM +0000, Izik Birka wrote: > Hi > Any idea ? > > > > > > > > > > > > From: Izik Birka > Sent: Tuesday, February 14, 2017 9:15 AM > To: 'Martin, Eric J' > Subject: RE: SMB > > Hi > I enable them and it's great but I'm looking for SMB bytes statistics , like in conn.log file > For example if someone downloaded 300 MB with SMB protocol (form network share) , is there any file that hold this statistics ? > > with http protocol , I can find it in conn.log file > > > thanks > > > > From: Martin, Eric J [mailto:ejmartin2 at wpi.edu] > Sent: Tuesday, February 14, 2017 12:09 AM > To: Izik Birka > > Subject: Re: SMB > > > There's smb_files and smb_mappings that need to be enabled. When you say 'stats', what are you looking for? > > > -- > > Eric Martin > > ejmartin2 at wpi.edu > > > Information Security Analyst > > Office: (508) 831-6070 > > > Worcester Polytechnic Institute > > www.wpi.edu > > PGP: C74F 1EBF 2E80 7984 8CB5 064E BF17 D34C C704 B30F > For security purposes, this message has been double ROT13 encoded > > ________________________________ > From: bro-bounces at bro.org > on behalf of Izik Birka > > Sent: Monday, February 13, 2017 3:34:29 AM > To: bro at bro.org > Subject: [Bro] SMB > > Hi > Is there any logs that contains SMB stats ? why conn.log doesn't contains SMB connection ? > > I have bro 2.5 > > Thanks > Izik Birka > > > This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain materials protected by copyright or information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or agreement. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication by error, notify the sender immediately and delete this message immediately. Thank you. > > This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain materials protected by copyright or information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or agreement. > > If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication by error, notify the sender immediately and delete this message immediately. > > Thank you. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From al.kefallonitis at gmail.com Sat Mar 4 17:57:54 2017 From: al.kefallonitis at gmail.com (Alex Kefallonitis) Date: Sun, 5 Mar 2017 03:57:54 +0200 Subject: [Bro] bro elasticsearch plugin + kibana indexing Message-ID: ELK + Kibana not indexing bro logs Succesfully installed the plugin and ELK but when i add indexing bro-* , index time-field appears empty (@timestamp) so i cannot use bro logs with kibana search. Anyone have same issue? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170305/22c3eefa/attachment.html From bill.de.ping at gmail.com Sat Mar 4 23:44:10 2017 From: bill.de.ping at gmail.com (william de ping) Date: Sun, 5 Mar 2017 09:44:10 +0200 Subject: [Bro] feeding bro cluster with parameters without restarting it In-Reply-To: <13172D2B-6C27-49ED-8015-A02F88951191@icir.org> References: <13172D2B-6C27-49ED-8015-A02F88951191@icir.org> Message-ID: Hi and thank you for your answers ! By slow I mean that writing to a file on a remote machine will have network and IO (read and write) strains. I suppose having something like ZeroMQ or some syslog messaging framework will be more efficient. On my case, I have a file that is being updated with 3+ lines per sec (each line has 3 fields). This file is being mapped to a table (&create_expire=10min). Upon a new connection I check if orig_h is in this table and assign a field accordingly. I see that many orig_h's are not recognized even though they exist in the file. Seth, can you please address me to a branch that includes this reconfigurable bro framework ? thanks again B On Thu, Mar 2, 2017 at 5:33 PM, Johanna Amann wrote: > Indeed, I was also going to ask that. We did some performance measurements > when we first wrote it - and it actually is quite fast. There only is a > relatively low amount of components between the input reader and it storing > things in a table; I cannot be 100% sure, but I doubt that other ingestion > methods can be much faster. (I actually doubt that they will be faster at > all). > > Johanna > > > On 2 Mar 2017, at 7:27, Azoff, Justin S wrote: > > On Mar 2, 2017, at 4:33 AM, william de ping >>> wrote: >>> >>> The thing is that the INPUT framework (STREAM) and generally reading >>> from files is relatively slow. >>> >> >> What exactly do you mean by relatively slow? How large are these tables >> that you are reading? >> >> -- >> - Justin Azoff >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170305/979e7ff6/attachment.html From daniel.guerra69 at gmail.com Sun Mar 5 00:27:06 2017 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Sun, 5 Mar 2017 09:27:06 +0100 Subject: [Bro] bro elasticsearch plugin + kibana indexing In-Reply-To: References: Message-ID: <0385FC7C-66B4-4778-B154-6EAFE12B2E3B@gmail.com> Try this https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/bro-patch/ElasticSearch.cc.patch > On 05 Mar 2017, at 02:57, Alex Kefallonitis wrote: > > ELK + Kibana not indexing bro logs > > Succesfully installed the plugin and ELK but when i add indexing bro-* , index time-field appears empty (@timestamp) so i cannot use bro logs with kibana search. Anyone have same issue? > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From al.kefallonitis at gmail.com Sun Mar 5 02:14:46 2017 From: al.kefallonitis at gmail.com (Alex Kefallonitis) Date: Sun, 5 Mar 2017 12:14:46 +0200 Subject: [Bro] bro elasticsearch plugin + kibana indexing In-Reply-To: <0385FC7C-66B4-4778-B154-6EAFE12B2E3B@gmail.com> References: <0385FC7C-66B4-4778-B154-6EAFE12B2E3B@gmail.com> Message-ID: I try the patch too but still no timestamp appears i am using ELK 5.2.2 2017-03-05 10:27 GMT+02:00 Daniel Guerra : > Try this > > https://github.com/danielguerra69/bro-debian- > elasticsearch/blob/master/bro-patch/ElasticSearch.cc.patch > > > > On 05 Mar 2017, at 02:57, Alex Kefallonitis > wrote: > > > > ELK + Kibana not indexing bro logs > > > > Succesfully installed the plugin and ELK but when i add indexing bro-* , > index time-field appears empty (@timestamp) so i cannot use bro logs with > kibana search. Anyone have same issue? > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170305/4d291c6a/attachment.html From daniel.guerra69 at gmail.com Sun Mar 5 04:28:46 2017 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Sun, 5 Mar 2017 13:28:46 +0100 Subject: [Bro] bro elasticsearch plugin + kibana indexing In-Reply-To: References: <0385FC7C-66B4-4778-B154-6EAFE12B2E3B@gmail.com> Message-ID: <1969E16C-8139-40B9-8633-D182D7C4368A@gmail.com> It does work. You have to send data first. Can you show your json output ? > On 05 Mar 2017, at 11:22, Alex Kefallonitis wrote: > > I do patch src/ElasticSearch.cc ./ElasticSearch.cc.patch ./configure && make && make install . Load bro elasticsearch script and restart bro open kibana > > > > 2017-03-05 12:14 GMT+02:00 Alex Kefallonitis >: > I try the patch too but still no timestamp appears i am using ELK 5.2.2 > > 2017-03-05 10:27 GMT+02:00 Daniel Guerra >: > Try this > > https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/bro-patch/ElasticSearch.cc.patch > > > > On 05 Mar 2017, at 02:57, Alex Kefallonitis > wrote: > > > > ELK + Kibana not indexing bro logs > > > > Succesfully installed the plugin and ELK but when i add indexing bro-* , index time-field appears empty (@timestamp) so i cannot use bro logs with kibana search. Anyone have same issue? > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170305/d5b00e40/attachment-0001.html From daniel.guerra69 at gmail.com Sun Mar 5 04:32:12 2017 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Sun, 5 Mar 2017 13:32:12 +0100 Subject: [Bro] bro elasticsearch plugin + kibana indexing In-Reply-To: References: <0385FC7C-66B4-4778-B154-6EAFE12B2E3B@gmail.com> Message-ID: <9032382A-62D9-485F-9D5B-500B0CE05583@gmail.com> Don?t forget this in the bro script that starts elasticsearch in the export part redef Log::default_scope_sep = "_?; > On 05 Mar 2017, at 11:22, Alex Kefallonitis wrote: > > I do patch src/ElasticSearch.cc ./ElasticSearch.cc.patch ./configure && make && make install . Load bro elasticsearch script and restart bro open kibana > > > > 2017-03-05 12:14 GMT+02:00 Alex Kefallonitis >: > I try the patch too but still no timestamp appears i am using ELK 5.2.2 > > 2017-03-05 10:27 GMT+02:00 Daniel Guerra >: > Try this > > https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/bro-patch/ElasticSearch.cc.patch > > > > On 05 Mar 2017, at 02:57, Alex Kefallonitis > wrote: > > > > ELK + Kibana not indexing bro logs > > > > Succesfully installed the plugin and ELK but when i add indexing bro-* , index time-field appears empty (@timestamp) so i cannot use bro logs with kibana search. Anyone have same issue? > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170305/8fefa77e/attachment.html From al.kefallonitis at gmail.com Sun Mar 5 04:56:48 2017 From: al.kefallonitis at gmail.com (Alex Kefallonitis) Date: Sun, 5 Mar 2017 14:56:48 +0200 Subject: [Bro] bro elasticsearch plugin + kibana indexing In-Reply-To: <9032382A-62D9-485F-9D5B-500B0CE05583@gmail.com> References: <0385FC7C-66B4-4778-B154-6EAFE12B2E3B@gmail.com> <9032382A-62D9-485F-9D5B-500B0CE05583@gmail.com> Message-ID: Where do i put this? redef Log::default_scope_sep = "_?; Do i have to enable json output to ascii.bro? 2017-03-05 14:32 GMT+02:00 Daniel Guerra : > Don?t forget this in the bro script that starts elasticsearch in the > export part > > redef Log::default_scope_sep = "_?; > > > On 05 Mar 2017, at 11:22, Alex Kefallonitis > wrote: > > I do patch src/ElasticSearch.cc > ./ElasticSearch.cc.patch ./configure && make && make install . Load bro > elasticsearch script and restart bro open kibana > > > > 2017-03-05 12:14 GMT+02:00 Alex Kefallonitis : > >> I try the patch too but still no timestamp appears i am using ELK 5.2.2 >> >> 2017-03-05 10:27 GMT+02:00 Daniel Guerra : >> >>> Try this >>> >>> https://github.com/danielguerra69/bro-debian-elasticsearch/b >>> lob/master/bro-patch/ElasticSearch.cc.patch >>> >>> >>> > On 05 Mar 2017, at 02:57, Alex Kefallonitis >>> wrote: >>> > >>> > ELK + Kibana not indexing bro logs >>> > >>> > Succesfully installed the plugin and ELK but when i add indexing bro-* >>> , index time-field appears empty (@timestamp) so i cannot use bro logs with >>> kibana search. Anyone have same issue? >>> > _______________________________________________ >>> > Bro mailing list >>> > bro at bro-ids.org >>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >>> >>> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170305/0cc71b77/attachment.html From daniel.guerra69 at gmail.com Sun Mar 5 04:57:36 2017 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Sun, 5 Mar 2017 13:57:36 +0100 Subject: [Bro] bro elasticsearch plugin + kibana indexing In-Reply-To: References: <0385FC7C-66B4-4778-B154-6EAFE12B2E3B@gmail.com> <9032382A-62D9-485F-9D5B-500B0CE05583@gmail.com> Message-ID: ##! Load this script to enable global log output to an ElasticSearch database. module LogElasticSearch; export { ## An elasticsearch specific rotation interval. const rotation_interval = 1hr &redef; ## Optionally ignore any :bro:type:`Log::ID` from being sent to ## ElasticSearch with this script. const excluded_log_ids: set[Log::ID] &redef; ## If you want to explicitly only send certain :bro:type:`Log::ID` ## streams, add them to this set. If the set remains empty, all will ## be sent. The :bro:id:`LogElasticSearch::excluded_log_ids` option ## will remain in effect as well. const send_logs: set[Log::ID] &redef; ## Set the separator redef Log::default_scope_sep = "_"; } > On 05 Mar 2017, at 13:56, Alex Kefallonitis wrote: > > Where do i put this? > > redef Log::default_scope_sep = "_?; > > Do i have to enable json output to ascii.bro? > > 2017-03-05 14:32 GMT+02:00 Daniel Guerra >: > Don?t forget this in the bro script that starts elasticsearch in the export part > > redef Log::default_scope_sep = "_?; > > >> On 05 Mar 2017, at 11:22, Alex Kefallonitis > wrote: >> >> I do patch src/ElasticSearch.cc ./ElasticSearch.cc.patch ./configure && make && make install . Load bro elasticsearch script and restart bro open kibana >> >> >> >> 2017-03-05 12:14 GMT+02:00 Alex Kefallonitis >: >> I try the patch too but still no timestamp appears i am using ELK 5.2.2 >> >> 2017-03-05 10:27 GMT+02:00 Daniel Guerra >: >> Try this >> >> https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/bro-patch/ElasticSearch.cc.patch >> >> >> > On 05 Mar 2017, at 02:57, Alex Kefallonitis > wrote: >> > >> > ELK + Kibana not indexing bro logs >> > >> > Succesfully installed the plugin and ELK but when i add indexing bro-* , index time-field appears empty (@timestamp) so i cannot use bro logs with kibana search. Anyone have same issue? >> > _______________________________________________ >> > Bro mailing list >> > bro at bro-ids.org >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170305/61d852e9/attachment-0001.html From daniel.guerra69 at gmail.com Sun Mar 5 05:41:45 2017 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Sun, 5 Mar 2017 14:41:45 +0100 Subject: [Bro] bro elasticsearch plugin + kibana indexing In-Reply-To: References: <0385FC7C-66B4-4778-B154-6EAFE12B2E3B@gmail.com> <9032382A-62D9-485F-9D5B-500B0CE05583@gmail.com> Message-ID: <1E0D4AA4-40E8-4AD6-BB7B-F54CDC6F416B@gmail.com> The patch wasn?t used, your timestamp is not in TS_ISO8601 but in TS_MILIS And your separator is a ?.? not a ?_" Check my docker https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/ > On 05 Mar 2017, at 14:12, Alex Kefallonitis wrote: > > Nothing changed > > > > > Although i have logs > > tail -f /opt/bro/logs/current/conn.log > {"ts":1488719244.873684,"uid":"CCToVE1JzVl9n5zDnj","id.orig_h":"10.0.0.31","id.orig_p":123,"id.resp_h":"194.177.210.54","id.resp_p":123,"proto":"udp","duration":0.021199,"orig_bytes":0,"resp_bytes":48,"conn_state":"SHR","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"Cd","orig_pkts":0,"orig_ip_bytes":0,"resp_pkts":1,"resp_ip_bytes":76,"tunnel_parents":[],"orig_l2_addr":"b8:27:eb:68:1a:49","resp_l2_addr":"36:34:64:31:64:39"} > {"ts":1488719252.873686,"uid":"ChHXxg3NsigjS6QwXg","id.orig_h":"10.0.0.31","id.orig_p":123,"id.resp_h":"62.1.45.120","id.resp_p":123,"proto":"udp","duration":0.020193,"orig_bytes":0,"resp_bytes":48,"conn_state":"SHR","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"Cd","orig_pkts":0,"orig_ip_bytes":0,"resp_pkts":1,"resp_ip_bytes":76,"tunnel_parents":[],"orig_l2_addr":"b8:27:eb:68:1a:49","resp_l2_addr":"36:34:64:31:64:39"} > {"ts":1488719249.686949,"uid":"CIrHst2VsHafEIR4vk","id.orig_h":"10.0.0.3","id.orig_p":123,"id.resp_h":"91.189.89.198","id.resp_p":123,"proto":"udp","duration":0.066331,"orig_bytes":48,"resp_bytes":48,"conn_state":"SF","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"Dd","orig_pkts":1,"orig_ip_bytes":76,"resp_pkts":1,"resp_ip_bytes":76,"tunnel_parents":[],"orig_l2_addr":"32:38:66:64:64:62","resp_l2_addr":"36:34:64:31:64:39"} > {"ts":1488719306.835847,"uid":"CqrMPYWEW543RxOX5","id.orig_h":"10.0.0.33","id.orig_p":51666,"id.resp_h":"10.0.0.1","id.resp_p":53,"proto":"udp","service":"dns","duration":0.000744,"orig_bytes":39,"resp_bytes":98,"conn_state":"SF","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"Dd","orig_pkts":1,"orig_ip_bytes":67,"resp_pkts":1,"resp_ip_bytes":126,"tunnel_parents":[],"orig_l2_addr":"36:33:62:63:39:61","resp_l2_addr":"36:34:64:31:64:39"} > {"ts":1488719314.06168,"uid":"CYw1dj2WUBXn6ua8O1","id.orig_h":"10.0.0.31","id.orig_p":37456,"id.resp_h":"10.0.0.1","id.resp_p":53,"proto":"udp","service":"dns","duration":0.000596,"orig_bytes":0,"resp_bytes":172,"conn_state":"SHR","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"Cd","orig_pkts":0,"orig_ip_bytes":0,"resp_pkts":2,"resp_ip_bytes":228,"tunnel_parents":[],"orig_l2_addr":"b8:27:eb:68:1a:49","resp_l2_addr":"36:34:64:31:64:39"} > > Do i have to change serator to init.bro also? > > > 2017-03-05 14:57 GMT+02:00 Daniel Guerra >: > ##! Load this script to enable global log output to an ElasticSearch database. > > module LogElasticSearch; > > export { > ## An elasticsearch specific rotation interval. > const rotation_interval = 1hr &redef; > > ## Optionally ignore any :bro:type:`Log::ID` from being sent to > ## ElasticSearch with this script. > const excluded_log_ids: set[Log::ID] &redef; > > ## If you want to explicitly only send certain :bro:type:`Log::ID` > ## streams, add them to this set. If the set remains empty, all will > ## be sent. The :bro:id:`LogElasticSearch::excluded_log_ids` option > ## will remain in effect as well. > const send_logs: set[Log::ID] &redef; > > ## Set the separator > redef Log::default_scope_sep = "_"; > } > > >> On 05 Mar 2017, at 13:56, Alex Kefallonitis > wrote: >> >> Where do i put this? >> >> redef Log::default_scope_sep = "_?; >> >> Do i have to enable json output to ascii.bro? >> >> 2017-03-05 14:32 GMT+02:00 Daniel Guerra >: >> Don?t forget this in the bro script that starts elasticsearch in the export part >> >> redef Log::default_scope_sep = "_?; >> >> >>> On 05 Mar 2017, at 11:22, Alex Kefallonitis > wrote: >>> >>> I do patch src/ElasticSearch.cc ./ElasticSearch.cc.patch ./configure && make && make install . Load bro elasticsearch script and restart bro open kibana >>> >>> >>> >>> 2017-03-05 12:14 GMT+02:00 Alex Kefallonitis >: >>> I try the patch too but still no timestamp appears i am using ELK 5.2.2 >>> >>> 2017-03-05 10:27 GMT+02:00 Daniel Guerra >: >>> Try this >>> >>> https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/bro-patch/ElasticSearch.cc.patch >>> >>> >>> > On 05 Mar 2017, at 02:57, Alex Kefallonitis > wrote: >>> > >>> > ELK + Kibana not indexing bro logs >>> > >>> > Succesfully installed the plugin and ELK but when i add indexing bro-* , index time-field appears empty (@timestamp) so i cannot use bro logs with kibana search. Anyone have same issue? >>> > _______________________________________________ >>> > Bro mailing list >>> > bro at bro-ids.org >>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170305/2a40ecf5/attachment.html From jazoff at illinois.edu Sun Mar 5 09:10:09 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Sun, 5 Mar 2017 17:10:09 +0000 Subject: [Bro] feeding bro cluster with parameters without restarting it In-Reply-To: References: <13172D2B-6C27-49ED-8015-A02F88951191@icir.org> Message-ID: > On Mar 5, 2017, at 2:44 AM, william de ping wrote: > > On my case, I have a file that is being updated with 3+ lines per sec (each line has 3 fields). This file is being mapped to a table (&create_expire=10min). > Upon a new connection I check if orig_h is in this table and assign a field accordingly. > I see that many orig_h's are not recognized even though they exist in the file. What is the time difference between when the file is updated and the table is checked? -- - Justin Azoff From iitsukas at nttdata.co.jp Mon Mar 6 01:45:21 2017 From: iitsukas at nttdata.co.jp (iitsukas at nttdata.co.jp) Date: Mon, 6 Mar 2017 09:45:21 +0000 Subject: [Bro] several questions for introducing Bro to commercial system Message-ID: <69140395ca8142e18f9344a7b75e90eb@MP-MSGSS-MBX007.msg.nttdata.co.jp> Hello, I am trying to introduce Bro to the enterprise system for the security enhancement purpose. I have several questions. Could you please answer the following questions? 1. Bro stores captured data into XXX.log files(XXX is http for example). In this case, how much data does Haka store into local file system per transaction? If you have any reference data, please let me know. 2. When Bro introduced machine has broken and fixed it, is it possible to continue the process(packet capturing process and storing data process into local file system) using the fixed machine without any problems? 3. What is the market share in the network forensic domain? Best regards, -- Satoshi Iitsuka -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170306/d3a56d79/attachment.html From bill.de.ping at gmail.com Mon Mar 6 03:42:23 2017 From: bill.de.ping at gmail.com (william de ping) Date: Mon, 6 Mar 2017 13:42:23 +0200 Subject: [Bro] feeding bro cluster with parameters without restarting it In-Reply-To: References: <13172D2B-6C27-49ED-8015-A02F88951191@icir.org> Message-ID: Well, its hard to provide you with this information As a process, writing to a remote file and reading from that remote file into a bro table, it is not the most efficient way to perform such a task. I do see events that have recognized their orig_h as part of the updated table, but they are very infrequent. Thanks On Sun, Mar 5, 2017 at 7:10 PM, Azoff, Justin S wrote: > > > On Mar 5, 2017, at 2:44 AM, william de ping > wrote: > > > > On my case, I have a file that is being updated with 3+ lines per sec > (each line has 3 fields). This file is being mapped to a table > (&create_expire=10min). > > Upon a new connection I check if orig_h is in this table and assign a > field accordingly. > > I see that many orig_h's are not recognized even though they exist in > the file. > > What is the time difference between when the file is updated and the table > is checked? > > -- > - Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170306/69ea38ec/attachment.html From jazoff at illinois.edu Mon Mar 6 08:25:20 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 6 Mar 2017 16:25:20 +0000 Subject: [Bro] feeding bro cluster with parameters without restarting it In-Reply-To: References: <13172D2B-6C27-49ED-8015-A02F88951191@icir.org> Message-ID: <1376954E-54E3-4330-89C3-53634BA85245@illinois.edu> > On Mar 6, 2017, at 6:42 AM, william de ping wrote: > > Well, its hard to provide you with this information Right but you are saying things are not efficient and slow, so I would expect that you have numbers to back up these statements. How fast are you expecting this table to update? 1ms? 1s? 30s? 'slow' is relative, and you haven't said what your expectations are, just that things aren't working. > As a process, writing to a remote file and reading from that remote file into a bro table, it is not the most efficient way to perform such a task. What remote file? Are you using NFS or something? Again you are saying things are not efficient, but you have not provided any information that proves this. You said you are writing 3 lines per second and that each line has 3 fields. This means that you are writing maybe 300 bytes/second to a file. This is not an extreme amount of data and it's not really possible to be inefficient with 3 lines per second. -- - Justin Azoff From wren3 at illinois.edu Mon Mar 6 11:48:55 2017 From: wren3 at illinois.edu (Ren, Wenyu) Date: Mon, 6 Mar 2017 19:48:55 +0000 Subject: [Bro] Question about Broker-Enabled Communication Framework Message-ID: Hi everyone, Recently I am learning to use the Broker-Enabled Communication Framework. When I tried to run an example in the document, I encountered a problem saying "value used but not set". Might be a silly question, but anyone have any idea? Here is my codes: const broker_port: port = 9999/tcp &redef; redef exit_only_after_terminate = T; redef Broker::endpoint_name = "connector"; global my_auto_event: event(msg: string, c: count); event bro_init() { Broker::enable(); Broker::connect("127.0.0.1", broker_port, 1sec); Broker::auto_event("bro/event/my_auto_event", my_auto_event); } event Broker::outgoing_connection_established(peer_address: string, peer_port: port, peer_name: string) { print "Broker::outgoing_connection_established", peer_address, peer_port, peer_name; event my_auto_event("stuff", 88); event my_auto_event("more stuff", 51); } event Broker::outgoing_connection_broken(peer_address: string, peer_port: port) { terminate(); } And here is the problem: bro events_connector.bro error in ./events_connector.bro, line 8: value used but not set (Broker::enable) error in ./events_connector.bro, line 9: value used but not set (Broker::connect) error in ./events_connector.bro, line 10: value used but not set (Broker::auto_event) Thanks a lot. Wenyu From dnthayer at illinois.edu Mon Mar 6 11:58:34 2017 From: dnthayer at illinois.edu (Daniel Thayer) Date: Mon, 6 Mar 2017 13:58:34 -0600 Subject: [Bro] Question about Broker-Enabled Communication Framework In-Reply-To: References: Message-ID: <2b58ab49-5717-39e1-bc07-afc3220013a4@illinois.edu> Did you build Bro with the "--enable-broker" option? ./configure --enable-broker make make install On 3/6/17 1:48 PM, Ren, Wenyu wrote: > Hi everyone, > > Recently I am learning to use the Broker-Enabled Communication Framework. When I tried to run an example in the document, I encountered a problem saying "value used but not set". Might be a silly question, but anyone have any idea? > > Here is my codes: > > const broker_port: port = 9999/tcp &redef; > redef exit_only_after_terminate = T; > redef Broker::endpoint_name = "connector"; > global my_auto_event: event(msg: string, c: count); > > event bro_init() > { > Broker::enable(); > Broker::connect("127.0.0.1", broker_port, 1sec); > Broker::auto_event("bro/event/my_auto_event", my_auto_event); > } > > event Broker::outgoing_connection_established(peer_address: string, > peer_port: port, > peer_name: string) > { > print "Broker::outgoing_connection_established", > peer_address, peer_port, peer_name; > event my_auto_event("stuff", 88); > event my_auto_event("more stuff", 51); > } > > event Broker::outgoing_connection_broken(peer_address: string, > peer_port: port) > { > terminate(); > } > > > And here is the problem: > > bro events_connector.bro > error in ./events_connector.bro, line 8: value used but not set (Broker::enable) > error in ./events_connector.bro, line 9: value used but not set (Broker::connect) > error in ./events_connector.bro, line 10: value used but not set (Broker::auto_event) > > > Thanks a lot. > > Wenyu > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From jazoff at illinois.edu Mon Mar 6 12:02:14 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 6 Mar 2017 20:02:14 +0000 Subject: [Bro] Question about Broker-Enabled Communication Framework In-Reply-To: References: Message-ID: <2910EBC1-D083-44C8-88C1-96B2D1B96251@illinois.edu> > On Mar 6, 2017, at 2:48 PM, Ren, Wenyu wrote: > > Hi everyone, > > Recently I am learning to use the Broker-Enabled Communication Framework. When I tried to run an example in the document, I encountered a problem saying "value used but not set". Might be a silly question, but anyone have any idea? Are you using a bro installation built with broker support ? ./configure --enable-broker ... -- - Justin Azoff From wren3 at illinois.edu Mon Mar 6 12:04:33 2017 From: wren3 at illinois.edu (Ren, Wenyu) Date: Mon, 6 Mar 2017 20:04:33 +0000 Subject: [Bro] Question about Broker-Enabled Communication Framework In-Reply-To: <2b58ab49-5717-39e1-bc07-afc3220013a4@illinois.edu> References: , <2b58ab49-5717-39e1-bc07-afc3220013a4@illinois.edu> Message-ID: I believe I had broker installed separately. Does that mean I still need to build bro again with the option? Thanks. ________________________________________ From: Thayer, Daniel N Sent: Monday, March 06, 2017 1:58 PM To: Ren, Wenyu; bro at bro.org Subject: Re: [Bro] Question about Broker-Enabled Communication Framework Did you build Bro with the "--enable-broker" option? ./configure --enable-broker make make install On 3/6/17 1:48 PM, Ren, Wenyu wrote: > Hi everyone, > > Recently I am learning to use the Broker-Enabled Communication Framework. When I tried to run an example in the document, I encountered a problem saying "value used but not set". Might be a silly question, but anyone have any idea? > > Here is my codes: > > const broker_port: port = 9999/tcp &redef; > redef exit_only_after_terminate = T; > redef Broker::endpoint_name = "connector"; > global my_auto_event: event(msg: string, c: count); > > event bro_init() > { > Broker::enable(); > Broker::connect("127.0.0.1", broker_port, 1sec); > Broker::auto_event("bro/event/my_auto_event", my_auto_event); > } > > event Broker::outgoing_connection_established(peer_address: string, > peer_port: port, > peer_name: string) > { > print "Broker::outgoing_connection_established", > peer_address, peer_port, peer_name; > event my_auto_event("stuff", 88); > event my_auto_event("more stuff", 51); > } > > event Broker::outgoing_connection_broken(peer_address: string, > peer_port: port) > { > terminate(); > } > > > And here is the problem: > > bro events_connector.bro > error in ./events_connector.bro, line 8: value used but not set (Broker::enable) > error in ./events_connector.bro, line 9: value used but not set (Broker::connect) > error in ./events_connector.bro, line 10: value used but not set (Broker::auto_event) > > > Thanks a lot. > > Wenyu > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From dnthayer at illinois.edu Mon Mar 6 12:15:41 2017 From: dnthayer at illinois.edu (Daniel Thayer) Date: Mon, 6 Mar 2017 14:15:41 -0600 Subject: [Bro] Question about Broker-Enabled Communication Framework In-Reply-To: References: <2b58ab49-5717-39e1-bc07-afc3220013a4@illinois.edu> Message-ID: Yes, you need to build Bro with "--enable-broker" in order for it to use broker. On 3/6/17 2:04 PM, Ren, Wenyu wrote: > I believe I had broker installed separately. Does that mean I still need to build bro again with the option? > > Thanks. > ________________________________________ > From: Thayer, Daniel N > Sent: Monday, March 06, 2017 1:58 PM > To: Ren, Wenyu; bro at bro.org > Subject: Re: [Bro] Question about Broker-Enabled Communication Framework > > Did you build Bro with the "--enable-broker" option? > > ./configure --enable-broker > make > make install > > > On 3/6/17 1:48 PM, Ren, Wenyu wrote: >> Hi everyone, >> >> Recently I am learning to use the Broker-Enabled Communication Framework. When I tried to run an example in the document, I encountered a problem saying "value used but not set". Might be a silly question, but anyone have any idea? >> >> Here is my codes: >> >> const broker_port: port = 9999/tcp &redef; >> redef exit_only_after_terminate = T; >> redef Broker::endpoint_name = "connector"; >> global my_auto_event: event(msg: string, c: count); >> >> event bro_init() >> { >> Broker::enable(); >> Broker::connect("127.0.0.1", broker_port, 1sec); >> Broker::auto_event("bro/event/my_auto_event", my_auto_event); >> } >> >> event Broker::outgoing_connection_established(peer_address: string, >> peer_port: port, >> peer_name: string) >> { >> print "Broker::outgoing_connection_established", >> peer_address, peer_port, peer_name; >> event my_auto_event("stuff", 88); >> event my_auto_event("more stuff", 51); >> } >> >> event Broker::outgoing_connection_broken(peer_address: string, >> peer_port: port) >> { >> terminate(); >> } >> >> >> And here is the problem: >> >> bro events_connector.bro >> error in ./events_connector.bro, line 8: value used but not set (Broker::enable) >> error in ./events_connector.bro, line 9: value used but not set (Broker::connect) >> error in ./events_connector.bro, line 10: value used but not set (Broker::auto_event) >> >> >> Thanks a lot. >> >> Wenyu >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> From bill.de.ping at gmail.com Tue Mar 7 03:20:14 2017 From: bill.de.ping at gmail.com (william de ping) Date: Tue, 7 Mar 2017 13:20:14 +0200 Subject: [Bro] BROKER enable fails Message-ID: Hi all, I'm trying to send my bro cluster some messages via pybroker. On the bro side I run the a.bro script and I get the following error : line 12: value used but not set (Broker::enable) line 13: value used but not set (Broker::listen) Is there anything else I should load to get this script working ? Any ideas about the source of the problem ? here is a.bro : @load base/frameworks/broker module test; const broker_port: port = 9999/tcp &redef;redef exit_only_after_terminate = T;redef Broker::endpoint_name = "listener"; event bro_init() { Broker::enable(); Broker::listen(broker_port, "127.0.0.1"); } event Broker::incoming_connection_established(peer_name: string) { print "Broker::incoming_connection_established", peer_name; } event Broker::incoming_connection_broken(peer_name: string) { print "Broker::incoming_connection_broken", peer_name; terminate(); } Thanks, B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170307/3fe3657f/attachment.html From jan.grashoefer at gmail.com Tue Mar 7 05:13:38 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Tue, 7 Mar 2017 14:13:38 +0100 Subject: [Bro] BROKER enable fails In-Reply-To: References: Message-ID: <4341475d-a14c-22f3-5846-a99e8f4f1a9d@gmail.com> > Hi all, > > I'm trying to send my bro cluster some messages via pybroker. > > On the bro side I run the a.bro script and I get the following error : > line 12: value used but not set (Broker::enable) > line 13: value used but not set (Broker::listen) > > Is there anything else I should load to get this script working ? > Any ideas about the source of the problem ? > > here is a.bro : > > @load base/frameworks/broker > > module test; > > const broker_port: port = 9999/tcp &redef;redef > exit_only_after_terminate = T;redef Broker::endpoint_name = > "listener"; > event bro_init() > { > Broker::enable(); > Broker::listen(broker_port, "127.0.0.1"); > } > event Broker::incoming_connection_established(peer_name: string) > { > print "Broker::incoming_connection_established", peer_name; > } > event Broker::incoming_connection_broken(peer_name: string) > { > print "Broker::incoming_connection_broken", peer_name; > terminate(); > } > > Thanks, > > B Sounds like yesterday: http://mailman.icsi.berkeley.edu/pipermail/bro/2017-March/011652.html From af7 at umbc.edu Tue Mar 7 07:34:23 2017 From: af7 at umbc.edu (Arash Fallah) Date: Tue, 7 Mar 2017 10:34:23 -0500 Subject: [Bro] Capture Loss Message-ID: I'm running Bro in a clustered configuration using PF_RING to have 8 separate workers on one box. Additionally, I have commented out almost everything in the default local.bro to run in Bro as efficiently as possible. Together, these 8 workers are using less than 20% of total CPU capacity. However, we are experiencing capture loss consistently in the 50% range, even though CPUs are idle 80% of the time on average. Does anyone have any experience with this? I would greatly appreciate the help. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170307/57b77539/attachment-0001.html From bill.de.ping at gmail.com Tue Mar 7 07:38:00 2017 From: bill.de.ping at gmail.com (william de ping) Date: Tue, 7 Mar 2017 17:38:00 +0200 Subject: [Bro] BROKER enable fails In-Reply-To: <1dff35de-7eda-539e-67f4-9ec555d34494@gmail.com> References: <4341475d-a14c-22f3-5846-a99e8f4f1a9d@gmail.com> <1dff35de-7eda-539e-67f4-9ec555d34494@gmail.com> Message-ID: OK I can now send messages and it works :) thanks How do you implement it in a clustered environment ? Every instance of bro (manager and workers) will try to listen to the same port I assume (because broctl will not start while a single bro instance does work) Is there a way of updating a table in a cluster environment for only one bro instance (manager) and make sure it will be synchronized for instances in the cluster ? Thanks B On Tue, Mar 7, 2017 at 4:37 PM, Jan Grash?fer wrote: > > Yes , I'm glad to see that the BROKER framework is getting headlines :) > > > > I still cannot figure out how to update a table in bro via pybroker. > > For now I'm just trying to send "hello" from pybroker and print it in a > bro > > script. > > > > Any suggestions on how to achieve that ? > > I sent an event and did stuff in there: > https://github.com/J-Gras/intel-extensions > > Hope that helps, > Jan > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170307/8a7e4227/attachment.html From espressobeanies at gmail.com Tue Mar 7 07:53:51 2017 From: espressobeanies at gmail.com (Espresso Beanies) Date: Tue, 7 Mar 2017 10:53:51 -0500 Subject: [Bro] Question on PacketFilter::DroppedPackets In-Reply-To: <20170303232003.gdadydjsr3hxrnuc@Beezling.local> References: <20170303232003.gdadydjsr3hxrnuc@Beezling.local> Message-ID: Hi Johanna, Thanks for the clarification. I wasn't sure if one method had an advantage over another. On Fri, Mar 3, 2017 at 6:20 PM, Johanna Amann wrote: > On Thu, Feb 16, 2017 at 02:05:19PM -0500, Espresso Beanies wrote: > > Hi, > > > > It seems no matter what I do, I still get these notices > > "PacketFilter::DroppedPackets". I created more workers, but I have a > > question about creating new workers via using an existing worker to > capture > > on the same interface using the "lb_procs" method to up the number of > > "sub-threads?" for multi-CPU processing. What advantage does a new worker > > give me over "lb_procs"? > > It is the same, lb_procs also basically just creates new workers; the > advantage is that it does things automatically for you that you would have > to configure by hand otherwhise. > > Johanna > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170307/ed468ed2/attachment.html From bill.de.ping at gmail.com Wed Mar 8 02:22:15 2017 From: bill.de.ping at gmail.com (william de ping) Date: Wed, 8 Mar 2017 12:22:15 +0200 Subject: [Bro] BROKER + CLUSTER - stuck Message-ID: Hello all, Short version : On bro2.5 I cannot seem to get the cluster working with a broker script. that same script is working on a single instance of bro. Detailed version : a.bro has a Broker listener linked to myevent. Once myevent is occurring, the string it received is added to a table named tb. On a pybroker remote machine I send updates to the listener. This scenario works with a single bro instance. However when I try to run broctl deploy, its stuck on "checking configurations ..." and never finish executing. I do see 7 bro processes that are up (1 manager, 1 proxy + 4 workers). Any ideas on why is this occurring ? Also, if I state in a.bro that the Broker should listen to 9999/tcp, what happens in the case of a cluster ? do all instances try to bind to the same address+port ? How can I send one update from my pybroker script to several bro instances (workers) ? Thanks in advance B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/08231520/attachment.html From dan.ecott at gmail.com Wed Mar 8 03:50:59 2017 From: dan.ecott at gmail.com (Dan Ecott) Date: Wed, 08 Mar 2017 11:50:59 +0000 Subject: [Bro] BRO on the endpoint, how to manage. Message-ID: Hello. I am exploring whether Bro can work for my company in a particular use case. What I would like to do is run Bro sensors on developer laptops, centrally manage the Bro scripts that run on those end points and ensure the Bro process is always running. What is the best way to run a deployment like this? Has it been done before? Bro Cluster doesn't look like the right solution. As far as managing the scripts, I was thinking of building an AWS code pipeline where I can promote scripts through a Git repo, then have a process whereby approved scripts get pushed out to the end points quickly. Any help on this would be appreciated. Dan. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/4c0e2af4/attachment.html From dopheide at gmail.com Wed Mar 8 07:11:15 2017 From: dopheide at gmail.com (Mike Dopheide) Date: Wed, 08 Mar 2017 15:11:15 +0000 Subject: [Bro] BROKER + CLUSTER - stuck In-Reply-To: References: Message-ID: For just the hanging issue... Do you perhaps have "exit_only_after_terminate" set in your policy for testing? I've seen that hang the config checker. Dop On Wed, Mar 8, 2017 at 4:24 AM william de ping wrote: > Hello all, > > Short version : On bro2.5 I cannot seem to get the cluster working with a > broker script. that same script is working on a single instance of bro. > > Detailed version : > > a.bro has a Broker listener linked to myevent. Once myevent is occurring, > the string it received is added to a table named tb. On a pybroker remote > machine I send updates to the listener. > > This scenario works with a single bro instance. > However when I try to run broctl deploy, its stuck on "checking > configurations ..." and never finish executing. > I do see 7 bro processes that are up (1 manager, 1 proxy + 4 workers). > > Any ideas on why is this occurring ? > > Also, if I state in a.bro that the Broker should listen to 9999/tcp, what > happens in the case of a cluster ? do all instances try to bind to the same > address+port ? > > How can I send one update from my pybroker script to several bro instances > (workers) ? > > Thanks in advance > B > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/3f5df439/attachment.html From jazoff at illinois.edu Wed Mar 8 07:39:33 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 8 Mar 2017 15:39:33 +0000 Subject: [Bro] BROKER + CLUSTER - stuck In-Reply-To: References: Message-ID: > On Mar 8, 2017, at 5:22 AM, william de ping wrote: > > How can I send one update from my pybroker script to several bro instances (workers) ? Have your pybroker script listen for connections and have all the workers connect to it. Or if you just send updates to the manager but mark the table as &synchronized updates will propagate to the other workers. -- - Justin Azoff From bill.de.ping at gmail.com Wed Mar 8 08:01:08 2017 From: bill.de.ping at gmail.com (william de ping) Date: Wed, 8 Mar 2017 18:01:08 +0200 Subject: [Bro] BROKER + CLUSTER - stuck In-Reply-To: References: Message-ID: Thanks, the exit_only_after_terminate helped about the &synchronized, only the manager is listening and all nodes are familiar with tb: table[string] of string &create_expire=10min &synchronized. manager successfully updates tb but it is not synchronized across cluster : broctl print TEST::tb manager TEST:tb = { ["hi"] = [name="bye"] } proxy-1 TEST:tb = { } worker-0 TEST:tb = { } do I need to invoke something after adding an element to tb, or &synchronized should be taking care of it ? Thanks B On Wed, Mar 8, 2017 at 5:39 PM, Azoff, Justin S wrote: > > > On Mar 8, 2017, at 5:22 AM, william de ping > wrote: > > > > How can I send one update from my pybroker script to several bro > instances (workers) ? > > Have your pybroker script listen for connections and have all the workers > connect to it. Or if you just send updates to the manager but mark the > table as &synchronized updates will propagate to the other workers. > > > > -- > - Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/4d41b2a8/attachment.html From dopheide at gmail.com Wed Mar 8 08:55:41 2017 From: dopheide at gmail.com (Mike Dopheide) Date: Wed, 8 Mar 2017 10:55:41 -0600 Subject: [Bro] BRO on the endpoint, how to manage. In-Reply-To: References: Message-ID: I don't know anyone else that's tried this, but it's an interesting thought experiment. A few initial thoughts in no particular order... (1) Given that Bro can be relatively CPU intensive, your developers will likely hate you for having something like that running on the same system where they're trying to do their work. I'd suggest setting up a one-off example and getting some real data on performance impact. (2) Cool idea! (3) I'd definitely run these as one-off Bro instances rather than trying to make it a cluster. To start cluster communication doesn't traverse secure protocols. However, that means you'll have to build up your own means of getting the log data, alerts, and checking on process status. (4) Related to (3) most of us use Bro to passively monitor network links. If your Bro process is sending data back out over the same network connection that it's monitoring you'll need to be very careful not to build a snowball effect. (5) We've been tracking our Bro policies in git for some time now, works great. (6) Do your developers run a fairly standard system configuration on their endpoints or would you have to potentially build Bro for a lot of different environments? (7) Maybe you could have Bro running on the endpoint only when the developer is traveling or otherwise on a less trusted (unmonitored) network? -Dop On Wed, Mar 8, 2017 at 5:50 AM, Dan Ecott wrote: > Hello. > > I am exploring whether Bro can work for my company in a particular use > case. What I would like to do is run Bro sensors on developer laptops, > centrally manage the Bro scripts that run on those end points and ensure > the Bro process is always running. > > What is the best way to run a deployment like this? Has it been done > before? Bro Cluster doesn't look like the right solution. > > As far as managing the scripts, I was thinking of building an AWS code > pipeline where I can promote scripts through a Git repo, then have a > process whereby approved scripts get pushed out to the end points quickly. > > Any help on this would be appreciated. > > Dan. > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/1a8bc084/attachment-0001.html From jlay at slave-tothe-box.net Wed Mar 8 09:15:04 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Wed, 08 Mar 2017 10:15:04 -0700 Subject: [Bro] Disabling an analyzer in weird Message-ID: <0d865d16f64276e3ff61c41b4d134a1e@localhost> Topic :) I'd like to have bro not dump non-rfc compliant syslog messages in the weird file. How can I go about doing that? Thank you. James From fatema.bannatwala at gmail.com Wed Mar 8 09:51:21 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Wed, 8 Mar 2017 12:51:21 -0500 Subject: [Bro] Bro error: "too many values to unpack" Message-ID: I usually once in a while run into an error, when I do a restart on the bro cluster. The restart successes but not sure what those error lines mean, as I don't find anything abnormal after bro cluster restarts. Does anyone have a clue? [fatema at mng site]$ /usr/local/bin/restart-bro removing old policies in /mnt/brolog/spool/installed-scripts-do-not-touch/site ... removing old policies in /mnt/brolog/spool/installed-scripts-do-not-touch/auto ... creating policy directories ... installing site policies ... generating cluster-layout.bro ... generating local-networks.bro ... generating broctl-config.bro ... generating broctl-config.sh ... updating nodes ... *Error: cannot create a directory on node proxy-3* *Error: Failed to establish ssh connection to host 10.10.24.211 : too many values to unpack* stopping ... stopping worker-1-1 ... stopping worker-1-10 ... stopping worker-1-11 ... stopping worker-1-12 ... And SO ON ... starting ... starting logger ... starting manager ... starting proxy-1 ... starting proxy-2 ... starting proxy-3 ... starting proxy-4 ... starting worker-1-1 ... starting worker-1-10 ... starting worker-1-11 ... And SO ON The restart-bro script looks something like this: #!/bin/sh sudo -u bro /usr/local/bro/default/bin/broctl install sudo /usr/local/bro/bin/fix-perms sudo -u bro /usr/local/bro/default/bin/broctl restart sudo /usr/local/bro/bin/restart-bro-dependents Thanks, Fatema. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/08827ec3/attachment.html From jan.grashoefer at gmail.com Wed Mar 8 10:17:57 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Wed, 8 Mar 2017 19:17:57 +0100 Subject: [Bro] Disabling an analyzer in weird In-Reply-To: <0d865d16f64276e3ff61c41b4d134a1e@localhost> References: <0d865d16f64276e3ff61c41b4d134a1e@localhost> Message-ID: <8e515a6f-1a50-c607-bf92-7d1034da08d1@gmail.com> > Topic :) I'd like to have bro not dump non-rfc compliant syslog > messages in the weird file. How can I go about doing that? Thank you. Add a filter for the log might be an option: https://www.bro.org/sphinx-git/frameworks/logging.html#filter-log-records Jan From promero at cenic.org Wed Mar 8 10:59:08 2017 From: promero at cenic.org (Philip Romero) Date: Wed, 8 Mar 2017 10:59:08 -0800 Subject: [Bro] Try.Bro.Org Table Creation Inquiry Message-ID: <9f8f4f5c-93d9-a437-b45b-228c30e145fc@cenic.org> All, I am trying to test a script to create a table at try.bro.org that I am having some trouble getting to work and was hoping to get some insight as to how to fix the issue. The intention is to create a table that I can read from for some additional monitoring. The testing I am doing is for what I believe is a simple use case, but the long term intention is to have a foundation for which I might create additional monitoring triggers. BTW - I have not gotten to the actual trigger script yet. I am still at the table creation portion of this process. The intention of the table is to log "active" scanner IP sources, timestamps, and notes from the notice.log and compare these to the conn.log to trigger if a scanner actually reaches and gets a response from an internal host. I have 2 scripts that independently work on the try.bro.org site, but I am trying to get them to work at the same time. The second script only works if I manually add the file (space delimited) in try.bro.org before I run the script. Ideally I'd like to have the table build live by reading in fields from either the current/notice.log or current/scanners.log I created from the first script, or even native tables created by the core bro environment. When I have these both run at the same time from try.bro.org site, I get an error opening the file source called in the second script being created by the first script. Any pointers or help on this would be greatly appreciated. SCRIPT 1: creates a new log file called scanners.log (btw - I have this running fine in my local dev environment) event bro_init() { # Add a new filter to the Notice::LOG stream that logs only # timestamp, note, and scanner address. local scanner_filter: Log::Filter = [$name="active-scanners", $path="scanners", $include=set("ts", "src", "note"), $pred(rec: Notice::Info) = { return rec?$sub && rec$sub == "remote" && rec$note == Scan::Port_Scan || rec$note == Scan::Address_Scan && rec$sub == "remote"; }]; Log::add_filter(Notice::LOG, scanner_filter); } SCRIPT 2: reads in the scanners.log file, creates the scanners table, and prints it for confirmation that it worked. redef InputAscii::separator = " "; type Idx: record { src: addr; }; type Val: record { ts: time; note: string; }; global scanners: table[addr] of Val = table(); event bro_init() { Input::add_table([$source="scanners.log", $name="scanners", $idx=Idx, $val=Val, $destination=scanners]); Input::remove("scanners"); } event Input::end_of_data(name: string, source: string) { # now all data is in the table print scanners; } -- Philip Romero, CISSP, CISA Sr. Information Security Analyst CENIC promero at cenic.org Phone: (714) 220-3430 Mobile: (562) 237-9290 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/b4ff9af7/attachment.html From fatema.bannatwala at gmail.com Wed Mar 8 11:06:04 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Wed, 8 Mar 2017 14:06:04 -0500 Subject: [Bro] BROKER + CLUSTER - stuck (Mike Dopheide) Message-ID: >However when I try to run broctl deploy, its stuck on "checking >configurations ..." and never finish executing. I have exact same issue, but not using any part of Broker, hence no " exit_only_after_terminate" flag solution for me. The only work around I have currently on our Bro cluster is to use: broctl install broctl restart instead of directly using "broctl deploy", because for some reason it just hangs on the "checking configuration.." and never finishes, and hence eventually I have to kill the process. :( $ /usr/local/bro/2.5/bin/broctl deploy checking configurations ... ^C $ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/0f1f7c21/attachment-0001.html From espressobeanies at gmail.com Wed Mar 8 11:11:05 2017 From: espressobeanies at gmail.com (Espresso Beanies) Date: Wed, 8 Mar 2017 14:11:05 -0500 Subject: [Bro] Question on cutting down on number of conn.log entries Message-ID: Hi, I'm realizing my conn.log is eating up most of my performance and I'm trying to cut down the number of times Bro makes a duplicate entry in the conn.log file. I don't necessarily need to see the same simultaneous traffic from a specific set of IP addresses and I'm trying to see if there's a way to exempt them or at least cut down on the number of times they are entered in my conn.log. Does anyone have any recommendations? I'm also trying to do it in a way that also cuts down on my CPU performance if possible. Thanks in advance, -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/fc797043/attachment.html From dopheide at gmail.com Wed Mar 8 11:11:57 2017 From: dopheide at gmail.com (Mike Dopheide) Date: Wed, 8 Mar 2017 13:11:57 -0600 Subject: [Bro] BROKER + CLUSTER - stuck (Mike Dopheide) In-Reply-To: References: Message-ID: Just a clarification, "exit_only_after_terminate" doesn't depend on Broker, it could just be buried in a script picked up from someone else as it's very common for debugging. -Dop On Wed, Mar 8, 2017 at 1:06 PM, fatema bannatwala < fatema.bannatwala at gmail.com> wrote: > >However when I try to run broctl deploy, its stuck on "checking > >configurations ..." and never finish executing. > > I have exact same issue, but not using any part of Broker, hence no " > exit_only_after_terminate" > flag solution for me. > The only work around I have currently on our Bro cluster is to use: > broctl install > broctl restart > instead of directly using "broctl deploy", because for some reason it just > hangs on the "checking configuration.." and never finishes, and hence > eventually I have to kill the process. :( > > $ /usr/local/bro/2.5/bin/broctl deploy > checking configurations ... > ^C > $ > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/40d3fb77/attachment.html From jazoff at illinois.edu Wed Mar 8 11:22:39 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 8 Mar 2017 19:22:39 +0000 Subject: [Bro] Question on cutting down on number of conn.log entries In-Reply-To: References: Message-ID: <872FCBEA-EDAF-41C1-8065-0926B5CEAFC1@illinois.edu> > On Mar 8, 2017, at 2:11 PM, Espresso Beanies wrote: > > Hi, > > I'm realizing my conn.log is eating up most of my performance and I'm trying to cut down the number of times Bro makes a duplicate entry in the conn.log file. What do you mean by duplicate entries? Are you seeing the same exact connection(same 5 tuple) logged multiple times? -- - Justin Azoff From fatema.bannatwala at gmail.com Wed Mar 8 11:32:37 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Wed, 8 Mar 2017 14:32:37 -0500 Subject: [Bro] BROKER + CLUSTER - stuck (Mike Dopheide) In-Reply-To: References: Message-ID: Thanks Mike for clarification. Does it means I can use it and redef it in my local.bro ? I was thinking is there anyone else encountering same issues with deploy cmd, and if it can be added to Bro documentation on how to solve the hanging issue. Because thinking of it as a normal user perspective: I upgrade the Bro cluster to 2.5, and when use broctl deploy, it just hangs there, while user having no clue why it would happen. Thanks, Fatema. On Wed, Mar 8, 2017 at 2:11 PM, Mike Dopheide wrote: > Just a clarification, "exit_only_after_terminate" doesn't depend on > Broker, it could just be buried in a script picked up from someone else as > it's very common for debugging. > > -Dop > > On Wed, Mar 8, 2017 at 1:06 PM, fatema bannatwala < > fatema.bannatwala at gmail.com> wrote: > >> >However when I try to run broctl deploy, its stuck on "checking >> >configurations ..." and never finish executing. >> >> I have exact same issue, but not using any part of Broker, hence no " >> exit_only_after_terminate" >> flag solution for me. >> The only work around I have currently on our Bro cluster is to use: >> broctl install >> broctl restart >> instead of directly using "broctl deploy", because for some reason it >> just hangs on the "checking configuration.." and never finishes, and hence >> eventually I have to kill the process. :( >> >> $ /usr/local/bro/2.5/bin/broctl deploy >> checking configurations ... >> ^C >> $ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/0aeef177/attachment.html From jazoff at illinois.edu Wed Mar 8 11:37:30 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 8 Mar 2017 19:37:30 +0000 Subject: [Bro] Try.Bro.Org Table Creation Inquiry In-Reply-To: <9f8f4f5c-93d9-a437-b45b-228c30e145fc@cenic.org> References: <9f8f4f5c-93d9-a437-b45b-228c30e145fc@cenic.org> Message-ID: <6715D779-DBC1-4817-860A-5BC8BC733CBF@illinois.edu> > On Mar 8, 2017, at 1:59 PM, Philip Romero wrote: > > All, > > I am trying to test a script to create a table at try.bro.org that I am having some trouble getting to work and was hoping to get some insight as to how to fix the issue. The intention is to create a table that I can read from for some additional monitoring. The testing I am doing is for what I believe is a simple use case, but the long term intention is to have a foundation for which I might create additional monitoring triggers. BTW - I have not gotten to the actual trigger script yet. I am still at the table creation portion of this process. > > The intention of the table is to log "active" scanner IP sources, timestamps, and notes from the notice.log and compare these to the conn.log to trigger if a scanner actually reaches and gets a response from an internal host. I have 2 scripts that independently work on the try.bro.org site, but I am trying to get them to work at the same time. The second script only works if I manually add the file (space delimited) in try.bro.org before I run the script. > > Ideally I'd like to have the table build live by reading in fields from either the current/notice.log or current/scanners.log I created from the first script, or even native tables created by the core bro environment. When I have these both run at the same time from try.bro.org site, I get an error opening the file source called in the second script being created by the first script. Any pointers or help on this would be greatly appreciated. It isn't working on try.bro.org because it's running against a pcap and the bro process only runs for a fraction of a second before exiting. At startup the file doesn't exist yet and the initial read will fail. This won't work properly on a live cluster though due to issues that only recently got fixed with the input framework. The next version of bro will re-try input files that couldn't be read at startup. Previously, the input framework would stop trying after the initial failure. In any case.. what you are trying to do doesn't actually require the use of files. You can just add a Notice::policy hook and add the scanner ip directly to the scanners table. -- - Justin Azoff From jazoff at illinois.edu Wed Mar 8 11:42:14 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 8 Mar 2017 19:42:14 +0000 Subject: [Bro] BROKER + CLUSTER - stuck (Mike Dopheide) In-Reply-To: References: Message-ID: <433339B7-7067-436D-BD2C-05E14736D116@illinois.edu> > On Mar 8, 2017, at 2:32 PM, fatema bannatwala wrote: > > Thanks Mike for clarification. Does it means I can use it and redef it in my local.bro ? > I was thinking is there anyone else encountering same issues with deploy cmd, and > if it can be added to Bro documentation on how to solve the hanging issue. > Because thinking of it as a normal user perspective: I upgrade the Bro cluster to 2.5, and when use broctl deploy, it just hangs there, while user having no clue why it would happen. > > Thanks, > Fatema. I used to see this issue on heavily overloaded clusters, but haven't been able to reproduce it in a while. All check does is run a sc One thing you could try is changing the check.bro script that comes with broctl to do terminate(); instead of terminate_communication(); I think Daniel also had a branch of broctl that just used bro -a instead of the check.bro script. -- - Justin Azoff From mus3 at lehigh.edu Wed Mar 8 11:50:36 2017 From: mus3 at lehigh.edu (Munroe Sollog) Date: Wed, 8 Mar 2017 14:50:36 -0500 Subject: [Bro] redef plugin variables Message-ID: <69d7cfdc-8979-3fe9-a05b-c61c10365fb7@lehigh.edu> I am using the elasticsearch plugin with NSQ and I am trying to set the following: redef destination = "nsq"; redef server_port = 4151; redef nsq_topic = "bro_logs"; These statements when put in plugins/Bro_Elasticsearch/scripts/init.bro causes everything to work as expected. However that file gets overwritten when the plugin gets rebuilt. I am trying to figure out how to put these statements in my local.bro. when I include the above in local.bro I get the following errors: worker-1-1 scripts failed. error in /usr/local/bro/share/bro/site/local.bro, line 98: "redef" used but not previously defined (destination) internal warning in /usr/local/bro/share/bro/site/local.bro, line 98: Can't document redef of destination, identifier lookup failed I'm assuming I have to prepend these variables with something like "Bro::ElasticSearch" but I can't find any docs to clarify. -- Munroe Sollog LTS - Senior Network Engineer x85002 From fatema.bannatwala at gmail.com Wed Mar 8 11:59:01 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Wed, 8 Mar 2017 14:59:01 -0500 Subject: [Bro] BROKER + CLUSTER - stuck (Mike Dopheide) In-Reply-To: <433339B7-7067-436D-BD2C-05E14736D116@illinois.edu> References: <433339B7-7067-436D-BD2C-05E14736D116@illinois.edu> Message-ID: Thanks Justin for the input! Yeah, you are right, tested the deploy cmd on a standalone node, and it does not hang there. I will test out the check.bro suggestions on the prod cluster. The cluster nodes use an average of ~30-35Gigs of memory (having ~125G in total) And the capture loss also doesn't report any loss i.e 0.025% etc Hence thought that the nodes were doing Ok, not sure if they are getting loads of traffic and hence getting overloaded. Also, I have noticed that when doing a restart on the cluster, it takes longer now (in 2.5) than it used to take when running the old version (2.4.1), maybe the custom scripts can be the culprit, but had same scripts in the old version as well. On Wed, Mar 8, 2017 at 2:42 PM, Azoff, Justin S wrote: > > > On Mar 8, 2017, at 2:32 PM, fatema bannatwala < > fatema.bannatwala at gmail.com> wrote: > > > > Thanks Mike for clarification. Does it means I can use it and redef it > in my local.bro ? > > I was thinking is there anyone else encountering same issues with deploy > cmd, and > > if it can be added to Bro documentation on how to solve the hanging > issue. > > Because thinking of it as a normal user perspective: I upgrade the Bro > cluster to 2.5, and when use broctl deploy, it just hangs there, while user > having no clue why it would happen. > > > > Thanks, > > Fatema. > > I used to see this issue on heavily overloaded clusters, but haven't been > able to reproduce it in a while. All check does is run a sc > > One thing you could try is changing the check.bro script that comes with > broctl to do > > terminate(); > > instead of > > terminate_communication(); > > > I think Daniel also had a branch of broctl that just used bro -a instead > of the check.bro script. > > -- > - Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/b704d84c/attachment.html From jazoff at illinois.edu Wed Mar 8 12:20:16 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 8 Mar 2017 20:20:16 +0000 Subject: [Bro] BROKER + CLUSTER - stuck (Mike Dopheide) In-Reply-To: References: <433339B7-7067-436D-BD2C-05E14736D116@illinois.edu> Message-ID: <8DB42B0C-608B-431F-85F4-A3A0DA656EB5@illinois.edu> > On Mar 8, 2017, at 2:59 PM, fatema bannatwala wrote: > > Thanks Justin for the input! > Yeah, you are right, tested the deploy cmd on a standalone node, and it does not hang there. > I will test out the check.bro suggestions on the prod cluster. > > The cluster nodes use an average of ~30-35Gigs of memory (having ~125G in total) > And the capture loss also doesn't report any loss i.e 0.025% etc > Hence thought that the nodes were doing Ok, not sure if they are getting loads of traffic and hence getting overloaded. > > Also, I have noticed that when doing a restart on the cluster, it takes longer now (in 2.5) than it used to take when running the old version (2.4.1), > maybe the custom scripts can be the culprit, but had same scripts in the old version as well. > Ah, I should have said manager not cluster. Check actually runs 100% on the manager. I think the hang is due to a race condition of some sort that prevents it from exiting like it is supposed to. It seems to only occur when the load is high, which is why deploy has an issue but stop first+check works ok. -- - Justin Azoff From fatema.bannatwala at gmail.com Wed Mar 8 12:39:01 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Wed, 8 Mar 2017 15:39:01 -0500 Subject: [Bro] BROKER + CLUSTER - stuck (Mike Dopheide) In-Reply-To: <8DB42B0C-608B-431F-85F4-A3A0DA656EB5@illinois.edu> References: <433339B7-7067-436D-BD2C-05E14736D116@illinois.edu> <8DB42B0C-608B-431F-85F4-A3A0DA656EB5@illinois.edu> Message-ID: Hmm, looks like the manager is also running with low memory: $ free -g total used free shared buff/cache available Mem: 70 9 46 0 14 60 Swap: 7 4 3 $ top top - 15:30:04 up 47 days, 22:33, 8 users, load average: 1.26, 1.25, 1.37 Tasks: 495 total, 2 running, 491 sleeping, 2 stopped, 0 zombie %Cpu(s): 2.9 us, 1.7 sy, 0.4 ni, 94.9 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 73949688 total, 48927592 free, 9836872 used, 15185224 buff/cache KiB Swap: 8388600 total, 4192868 free, 4195732 used. 63369176 avail Mem Anyways, not going into that rabbit hole :) So the correct sequence to deploy any config changes in a cluster would be: stop -> check -> install -> start I was looking at the cmds available and looks like "restart --clean" would do the trick? or I can just script the above sequence in my restart-bro script :) Thanks, Fatema. On Wed, Mar 8, 2017 at 3:20 PM, Azoff, Justin S wrote: > > > On Mar 8, 2017, at 2:59 PM, fatema bannatwala < > fatema.bannatwala at gmail.com> wrote: > > > > Thanks Justin for the input! > > Yeah, you are right, tested the deploy cmd on a standalone node, and it > does not hang there. > > I will test out the check.bro suggestions on the prod cluster. > > > > The cluster nodes use an average of ~30-35Gigs of memory (having ~125G > in total) > > And the capture loss also doesn't report any loss i.e 0.025% etc > > Hence thought that the nodes were doing Ok, not sure if they are getting > loads of traffic and hence getting overloaded. > > > > Also, I have noticed that when doing a restart on the cluster, it takes > longer now (in 2.5) than it used to take when running the old version > (2.4.1), > > maybe the custom scripts can be the culprit, but had same scripts in the > old version as well. > > > Ah, I should have said manager not cluster. > > Check actually runs 100% on the manager. I think the hang is due to a > race condition of some sort that prevents it from exiting like it is > supposed to. It seems to only occur when the load is high, which is why > deploy has an issue but stop first+check works ok. > > -- > - Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/771ee153/attachment.html From al.kefallonitis at gmail.com Wed Mar 8 12:42:29 2017 From: al.kefallonitis at gmail.com (Alex Kefallonitis) Date: Wed, 8 Mar 2017 22:42:29 +0200 Subject: [Bro] bro elasticsearch plugin + kibana indexing In-Reply-To: <1E0D4AA4-40E8-4AD6-BB7B-F54CDC6F416B@gmail.com> References: <0385FC7C-66B4-4778-B154-6EAFE12B2E3B@gmail.com> <9032382A-62D9-485F-9D5B-500B0CE05583@gmail.com> <1E0D4AA4-40E8-4AD6-BB7B-F54CDC6F416B@gmail.com> Message-ID: How TS_ISO8601 timestamp looks like? I ll try to recompile maybe i didn't apply the patch correctly. Thanks again a lot for your help 2017-03-05 15:41 GMT+02:00 Daniel Guerra : > The patch wasn?t used, your timestamp is not in TS_ISO8601 but in TS_MILIS > And your separator is a ?.? not a ?_" > > Check my docker > > https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/ > > On 05 Mar 2017, at 14:12, Alex Kefallonitis > wrote: > > Nothing changed > > > > > Although i have logs > > tail -f /opt/bro/logs/current/conn.log > {"ts":1488719244.873684,"uid":"CCToVE1JzVl9n5zDnj","id.orig_ > h":"10.0.0.31","id.orig_p":123,"id.resp_h":"194.177.210. > 54","id.resp_p":123,"proto":"udp","duration":0.021199," > orig_bytes":0,"resp_bytes":48,"conn_state":"SHR","local_ > orig":true,"local_resp":false,"missed_bytes":0,"history":" > Cd","orig_pkts":0,"orig_ip_bytes":0,"resp_pkts":1,"resp_ > ip_bytes":76,"tunnel_parents":[],"orig_l2_addr":"b8:27:eb: > 68:1a:49","resp_l2_addr":"36:34:64:31:64:39"} > {"ts":1488719252.873686,"uid":"ChHXxg3NsigjS6QwXg","id.orig_ > h":"10.0.0.31","id.orig_p":123,"id.resp_h":"62.1.45.120", > "id.resp_p":123,"proto":"udp","duration":0.020193,"orig_ > bytes":0,"resp_bytes":48,"conn_state":"SHR","local_orig" > :true,"local_resp":false,"missed_bytes":0,"history":"Cd" > ,"orig_pkts":0,"orig_ip_bytes":0,"resp_pkts":1,"resp_ip_ > bytes":76,"tunnel_parents":[],"orig_l2_addr":"b8:27:eb:68: > 1a:49","resp_l2_addr":"36:34:64:31:64:39"} > {"ts":1488719249.686949,"uid":"CIrHst2VsHafEIR4vk","id.orig_ > h":"10.0.0.3","id.orig_p":123,"id.resp_h":"91.189.89.198"," > id.resp_p":123,"proto":"udp","duration":0.066331,"orig_ > bytes":48,"resp_bytes":48,"conn_state":"SF","local_orig": > true,"local_resp":false,"missed_bytes":0,"history":"Dd" > ,"orig_pkts":1,"orig_ip_bytes":76,"resp_pkts":1,"resp_ip_ > bytes":76,"tunnel_parents":[],"orig_l2_addr":"32:38:66:64: > 64:62","resp_l2_addr":"36:34:64:31:64:39"} > {"ts":1488719306.835847,"uid":"CqrMPYWEW543RxOX5","id.orig_ > h":"10.0.0.33","id.orig_p":51666,"id.resp_h":"10.0.0.1"," > id.resp_p":53,"proto":"udp","service":"dns","duration":0. > 000744,"orig_bytes":39,"resp_bytes":98,"conn_state":"SF"," > local_orig":true,"local_resp":true,"missed_bytes":0," > history":"Dd","orig_pkts":1,"orig_ip_bytes":67,"resp_pkts": > 1,"resp_ip_bytes":126,"tunnel_parents":[],"orig_l2_addr":" > 36:33:62:63:39:61","resp_l2_addr":"36:34:64:31:64:39"} > {"ts":1488719314.06168,"uid":"CYw1dj2WUBXn6ua8O1","id.orig_ > h":"10.0.0.31","id.orig_p":37456,"id.resp_h":"10.0.0.1"," > id.resp_p":53,"proto":"udp","service":"dns","duration":0. > 000596,"orig_bytes":0,"resp_bytes":172,"conn_state":"SHR", > "local_orig":true,"local_resp":true,"missed_bytes":0," > history":"Cd","orig_pkts":0,"orig_ip_bytes":0,"resp_pkts": > 2,"resp_ip_bytes":228,"tunnel_parents":[],"orig_l2_addr":" > b8:27:eb:68:1a:49","resp_l2_addr":"36:34:64:31:64:39"} > > Do i have to change serator to init.bro also? > > > 2017-03-05 14:57 GMT+02:00 Daniel Guerra : > >> ##! Load this script to enable global log output to an ElasticSearch >> database. >> >> module LogElasticSearch; >> >> export { >> ## An elasticsearch specific rotation interval. >> const rotation_interval = 1hr &redef; >> >> ## Optionally ignore any :bro:type:`Log::ID` from being sent to >> ## ElasticSearch with this script. >> const excluded_log_ids: set[Log::ID] &redef; >> >> ## If you want to explicitly only send certain :bro:type:`Log::ID` >> ## streams, add them to this set. If the set remains empty, all will >> ## be sent. The :bro:id:`LogElasticSearch::excluded_log_ids` option >> ## will remain in effect as well. >> const send_logs: set[Log::ID] &redef; >> >> ## Set the separator >> redef Log::default_scope_sep = "_"; >> } >> >> >> On 05 Mar 2017, at 13:56, Alex Kefallonitis >> wrote: >> >> Where do i put this? >> >> redef Log::default_scope_sep = "_?; >> >> Do i have to enable json output to ascii.bro? >> >> 2017-03-05 14:32 GMT+02:00 Daniel Guerra : >> >>> Don?t forget this in the bro script that starts elasticsearch in the >>> export part >>> >>> redef Log::default_scope_sep = "_?; >>> >>> >>> On 05 Mar 2017, at 11:22, Alex Kefallonitis >>> wrote: >>> >>> I do patch src/ElasticSearch.cc >>> ./ElasticSearch.cc.patch ./configure && make && make install . Load bro >>> elasticsearch script and restart bro open kibana >>> >>> >>> >>> 2017-03-05 12:14 GMT+02:00 Alex Kefallonitis >>> : >>> >>>> I try the patch too but still no timestamp appears i am using ELK 5.2.2 >>>> >>>> 2017-03-05 10:27 GMT+02:00 Daniel Guerra : >>>> >>>>> Try this >>>>> >>>>> https://github.com/danielguerra69/bro-debian-elasticsearch/b >>>>> lob/master/bro-patch/ElasticSearch.cc.patch >>>>> >>>>> >>>>> > On 05 Mar 2017, at 02:57, Alex Kefallonitis < >>>>> al.kefallonitis at gmail.com> wrote: >>>>> > >>>>> > ELK + Kibana not indexing bro logs >>>>> > >>>>> > Succesfully installed the plugin and ELK but when i add indexing >>>>> bro-* , index time-field appears empty (@timestamp) so i cannot use bro >>>>> logs with kibana search. Anyone have same issue? >>>>> > _______________________________________________ >>>>> > Bro mailing list >>>>> > bro at bro-ids.org >>>>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>>>> >>>>> >>>>> >>>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/3fd09c8d/attachment-0001.html From jazoff at illinois.edu Wed Mar 8 12:56:32 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 8 Mar 2017 20:56:32 +0000 Subject: [Bro] BROKER + CLUSTER - stuck (Mike Dopheide) In-Reply-To: References: <433339B7-7067-436D-BD2C-05E14736D116@illinois.edu> <8DB42B0C-608B-431F-85F4-A3A0DA656EB5@illinois.edu> Message-ID: <5E1E1CD1-68D6-4E2C-B2EB-C5BD5CF97B60@illinois.edu> > On Mar 8, 2017, at 3:39 PM, fatema bannatwala wrote: > > Hmm, looks like the manager is also running with low memory: > $ free -g > total used free shared buff/cache available > Mem: 70 9 46 0 14 60 > Swap: 7 4 3 > That's fine.. you have 46G of ram free. > $ top > top - 15:30:04 up 47 days, 22:33, 8 users, load average: 1.26, 1.25, 1.37 > Tasks: 495 total, 2 running, 491 sleeping, 2 stopped, 0 zombie > %Cpu(s): 2.9 us, 1.7 sy, 0.4 ni, 94.9 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st > KiB Mem : 73949688 total, 48927592 free, 9836872 used, 15185224 buff/cache > KiB Swap: 8388600 total, 4192868 free, 4195732 used. 63369176 avail Mem > > > Anyways, not going into that rabbit hole :) > So the correct sequence to deploy any config changes in a cluster would be: > stop -> check -> install -> start > I was looking at the cmds available and looks like "restart --clean" would do the trick? > or I can just script the above sequence in my restart-bro script :) Yeah.. deploy basically automates check+install+restart. stop+check+install+start is basically the same as stop+deploy. The reason why deploy does check first is because if you accidentally broke your configuration, you need to start it again with the old configuration, fix the configuration, and then retry - effectively wasting a restart. Doing check first means that you don't stop bro unless you know the new configuration will work. The worst thing you can do is stop+install+check . If you do that, you can end up with a broken bro installation that needs to be fixed before you can start it up again. A lot of people were doing this which is why added deploy. -- - Justin Azoff From espressobeanies at gmail.com Wed Mar 8 13:40:56 2017 From: espressobeanies at gmail.com (Espresso Beanies) Date: Wed, 8 Mar 2017 16:40:56 -0500 Subject: [Bro] Question on cutting down on number of conn.log entries In-Reply-To: <872FCBEA-EDAF-41C1-8065-0926B5CEAFC1@illinois.edu> References: <872FCBEA-EDAF-41C1-8065-0926B5CEAFC1@illinois.edu> Message-ID: Yep On Wed, Mar 8, 2017 at 2:22 PM, Azoff, Justin S wrote: > > > On Mar 8, 2017, at 2:11 PM, Espresso Beanies > wrote: > > > > Hi, > > > > I'm realizing my conn.log is eating up most of my performance and I'm > trying to cut down the number of times Bro makes a duplicate entry in the > conn.log file. > > What do you mean by duplicate entries? Are you seeing the same exact > connection(same 5 tuple) logged multiple times? > > -- > - Justin Azoff > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/f7fc87a9/attachment.html From jazoff at illinois.edu Wed Mar 8 13:47:09 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 8 Mar 2017 21:47:09 +0000 Subject: [Bro] Question on cutting down on number of conn.log entries In-Reply-To: References: <872FCBEA-EDAF-41C1-8065-0926B5CEAFC1@illinois.edu> Message-ID: <5C77A6AB-6785-4D36-AB0B-FA78EBAC579C@illinois.edu> Ok.. if you are seeing the exact same connection repeated multiple times that would point to an issue with your deployment. Are you running multiple bro workers using lb_procs? If you run multiple workers but the load balancing is not functioning properly, you'll see multiple entries as you described. -- - Justin Azoff > On Mar 8, 2017, at 4:40 PM, Espresso Beanies wrote: > > Yep > > On Wed, Mar 8, 2017 at 2:22 PM, Azoff, Justin S wrote: > > > On Mar 8, 2017, at 2:11 PM, Espresso Beanies wrote: > > > > Hi, > > > > I'm realizing my conn.log is eating up most of my performance and I'm trying to cut down the number of times Bro makes a duplicate entry in the conn.log file. > > What do you mean by duplicate entries? Are you seeing the same exact connection(same 5 tuple) logged multiple times? > > -- > - Justin Azoff > > > From fatema.bannatwala at gmail.com Wed Mar 8 13:52:31 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Wed, 8 Mar 2017 16:52:31 -0500 Subject: [Bro] BROKER + CLUSTER - stuck (Mike Dopheide) In-Reply-To: <5E1E1CD1-68D6-4E2C-B2EB-C5BD5CF97B60@illinois.edu> References: <433339B7-7067-436D-BD2C-05E14736D116@illinois.edu> <8DB42B0C-608B-431F-85F4-A3A0DA656EB5@illinois.edu> <5E1E1CD1-68D6-4E2C-B2EB-C5BD5CF97B60@illinois.edu> Message-ID: Cool! Thanks Justin for the explanation :) Tried using stop first+deploy but hangs and never completes. Also tried doing stop+check+install+start, but hangs again. Hence, ended up doing install+restart, to bring back the cluster up. :( The manager doesn't look overloaded though. I think next will try out the check.bro change you suggested at the beginning, and see if that helps.. Thanks! On Wed, Mar 8, 2017 at 3:56 PM, Azoff, Justin S wrote: > > > On Mar 8, 2017, at 3:39 PM, fatema bannatwala < > fatema.bannatwala at gmail.com> wrote: > > > > Hmm, looks like the manager is also running with low memory: > > $ free -g > > total used free shared buff/cache > available > > Mem: 70 9 46 0 14 > 60 > > Swap: 7 4 3 > > > > That's fine.. you have 46G of ram free. > > > $ top > > top - 15:30:04 up 47 days, 22:33, 8 users, load average: 1.26, 1.25, > 1.37 > > Tasks: 495 total, 2 running, 491 sleeping, 2 stopped, 0 zombie > > %Cpu(s): 2.9 us, 1.7 sy, 0.4 ni, 94.9 id, 0.1 wa, 0.0 hi, 0.0 si, > 0.0 st > > KiB Mem : 73949688 total, 48927592 free, 9836872 used, 15185224 > buff/cache > > KiB Swap: 8388600 total, 4192868 free, 4195732 used. 63369176 avail > Mem > > > > > > Anyways, not going into that rabbit hole :) > > So the correct sequence to deploy any config changes in a cluster would > be: > > stop -> check -> install -> start > > I was looking at the cmds available and looks like "restart --clean" > would do the trick? > > or I can just script the above sequence in my restart-bro script :) > > Yeah.. deploy basically automates check+install+restart. > > stop+check+install+start is basically the same as stop+deploy. > > The reason why deploy does check first is because if you accidentally > broke your configuration, you need to start it again with the old > configuration, fix the configuration, and then retry - effectively wasting > a restart. > > Doing check first means that you don't stop bro unless you know the new > configuration will work. > > The worst thing you can do is stop+install+check . If you do that, you > can end up with a broken bro installation that needs to be fixed before you > can start it up again. A lot of people were doing this which is why added > deploy. > > > > > -- > - Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/f4f8ed32/attachment.html From dnthayer at illinois.edu Wed Mar 8 14:03:57 2017 From: dnthayer at illinois.edu (Daniel Thayer) Date: Wed, 8 Mar 2017 16:03:57 -0600 Subject: [Bro] BROKER + CLUSTER - stuck (Mike Dopheide) In-Reply-To: References: <433339B7-7067-436D-BD2C-05E14736D116@illinois.edu> <8DB42B0C-608B-431F-85F4-A3A0DA656EB5@illinois.edu> <5E1E1CD1-68D6-4E2C-B2EB-C5BD5CF97B60@illinois.edu> Message-ID: When "broctl deploy" hangs, were there any bro processes running (before you ran "deploy")? Also, when you ran "stop+check+install+start", which command hangs? Also, in this case, did you verify that all bro processes were stopped by the "stop" command? On 3/8/17 3:52 PM, fatema bannatwala wrote: > Cool! Thanks Justin for the explanation :) > > Tried using stop first+deploy but hangs and never completes. > Also tried doing stop+check+install+start, but hangs again. > > Hence, ended up doing install+restart, to bring back the cluster up. :( > The manager doesn't look overloaded though. > I think next will try out the check.bro change you suggested at the > beginning, and see if that helps.. > > Thanks! > > On Wed, Mar 8, 2017 at 3:56 PM, Azoff, Justin S > wrote: > > > > On Mar 8, 2017, at 3:39 PM, fatema bannatwala > wrote: > > > > Hmm, looks like the manager is also running with low memory: > > $ free -g > > total used free shared buff/cache available > > Mem: 70 9 46 0 14 60 > > Swap: 7 4 3 > > > > That's fine.. you have 46G of ram free. > > > $ top > > top - 15:30:04 up 47 days, 22:33, 8 users, load average: 1.26, 1.25, 1.37 > > Tasks: 495 total, 2 running, 491 sleeping, 2 stopped, 0 zombie > > %Cpu(s): 2.9 us, 1.7 sy, 0.4 ni, 94.9 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st > > KiB Mem : 73949688 total, 48927592 free, 9836872 used, 15185224 buff/cache > > KiB Swap: 8388600 total, 4192868 free, 4195732 used. 63369176 avail Mem > > > > > > Anyways, not going into that rabbit hole :) > > So the correct sequence to deploy any config changes in a cluster would be: > > stop -> check -> install -> start > > I was looking at the cmds available and looks like "restart --clean" would do the trick? > > or I can just script the above sequence in my restart-bro script :) > > Yeah.. deploy basically automates check+install+restart. > > stop+check+install+start is basically the same as stop+deploy. > > The reason why deploy does check first is because if you > accidentally broke your configuration, you need to start it again > with the old configuration, fix the configuration, and then retry - > effectively wasting a restart. > > Doing check first means that you don't stop bro unless you know the > new configuration will work. > > The worst thing you can do is stop+install+check . If you do that, > you can end up with a broken bro installation that needs to be fixed > before you can start it up again. A lot of people were doing this > which is why added deploy. > > > > > -- > - Justin Azoff > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From fatema.bannatwala at gmail.com Wed Mar 8 14:26:49 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Wed, 8 Mar 2017 17:26:49 -0500 Subject: [Bro] BROKER + CLUSTER - stuck (Mike Dopheide) In-Reply-To: References: <433339B7-7067-436D-BD2C-05E14736D116@illinois.edu> <8DB42B0C-608B-431F-85F4-A3A0DA656EB5@illinois.edu> <5E1E1CD1-68D6-4E2C-B2EB-C5BD5CF97B60@illinois.edu> Message-ID: There were bro processes running before I used deploy cmd ( manager acts as Proxies-4, logger as well as manager, so around 5-6 bro processes run on manager machine). When I ran stop+check+install+start, all the bro processes stopped properly (atleast that's what console output said), and it hung on check. After restoring the working cluster, I ran 'broctl check' on manager just to see what it does when run standalone, while the cluster is running: it just never completed, and turned out, after couple of minutes, that the manager started swapping and hung, so I had to run 'sudo kill -9 bro' to get back the hold of machine, then restarted bro normally by doing a 'install and restart'.. :/ Thanks, Fatema. On Wed, Mar 8, 2017 at 5:03 PM, Daniel Thayer wrote: > When "broctl deploy" hangs, were there any bro processes > running (before you ran "deploy")? > > Also, when you ran "stop+check+install+start", which > command hangs? Also, in this case, did you verify that > all bro processes were stopped by the "stop" command? > > > > On 3/8/17 3:52 PM, fatema bannatwala wrote: > >> Cool! Thanks Justin for the explanation :) >> >> Tried using stop first+deploy but hangs and never completes. >> Also tried doing stop+check+install+start, but hangs again. >> >> Hence, ended up doing install+restart, to bring back the cluster up. :( >> The manager doesn't look overloaded though. >> I think next will try out the check.bro change you suggested at the >> beginning, and see if that helps.. >> >> Thanks! >> >> On Wed, Mar 8, 2017 at 3:56 PM, Azoff, Justin S > > wrote: >> >> >> >> > On Mar 8, 2017, at 3:39 PM, fatema bannatwala < >> fatema.bannatwala at gmail.com > wrote: >> > >> > Hmm, looks like the manager is also running with low memory: >> > $ free -g >> > total used free shared >> buff/cache available >> > Mem: 70 9 46 0 >> 14 60 >> > Swap: 7 4 3 >> > >> >> That's fine.. you have 46G of ram free. >> >> > $ top >> > top - 15:30:04 up 47 days, 22:33, 8 users, load average: 1.26, >> 1.25, 1.37 >> > Tasks: 495 total, 2 running, 491 sleeping, 2 stopped, 0 zombie >> > %Cpu(s): 2.9 us, 1.7 sy, 0.4 ni, 94.9 id, 0.1 wa, 0.0 hi, 0.0 >> si, 0.0 st >> > KiB Mem : 73949688 total, 48927592 free, 9836872 used, 15185224 >> buff/cache >> > KiB Swap: 8388600 total, 4192868 free, 4195732 used. 63369176 >> avail Mem >> > >> > >> > Anyways, not going into that rabbit hole :) >> > So the correct sequence to deploy any config changes in a cluster >> would be: >> > stop -> check -> install -> start >> > I was looking at the cmds available and looks like "restart >> --clean" would do the trick? >> > or I can just script the above sequence in my restart-bro script :) >> >> Yeah.. deploy basically automates check+install+restart. >> >> stop+check+install+start is basically the same as stop+deploy. >> >> The reason why deploy does check first is because if you >> accidentally broke your configuration, you need to start it again >> with the old configuration, fix the configuration, and then retry - >> effectively wasting a restart. >> >> Doing check first means that you don't stop bro unless you know the >> new configuration will work. >> >> The worst thing you can do is stop+install+check . If you do that, >> you can end up with a broken bro installation that needs to be fixed >> before you can start it up again. A lot of people were doing this >> which is why added deploy. >> >> >> >> >> -- >> - Justin Azoff >> >> >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/af602150/attachment.html From dan.ecott at gmail.com Wed Mar 8 15:53:06 2017 From: dan.ecott at gmail.com (Dan Ecott) Date: Wed, 08 Mar 2017 23:53:06 +0000 Subject: [Bro] BRO on the endpoint, how to manage. In-Reply-To: References: Message-ID: Some responses inline. I may try and leverage AWS lambda triggers on git commits and Jamf to get new scripts down to the laptops somehow. Thanks for your responses. I will check back in if I make any good progress. Dan. On Wed, Mar 8, 2017 at 11:55 AM Mike Dopheide wrote: > I don't know anyone else that's tried this, but it's an interesting > thought experiment. A few initial thoughts in no particular order... > > (1) Given that Bro can be relatively CPU intensive, your developers will > likely hate you for having something like that running on the same system > where they're trying to do their work. I'd suggest setting up a one-off > example and getting some real data on performance impact. > Definitely going to look at how this impacts the use of the laptops. Initial assessments on my own show that the bro processes don't use anything more than 2% CPU with the out of the box scripts. > > > (2) Cool idea! > If it works. :-) > > (3) I'd definitely run these as one-off Bro instances rather than trying > to make it a cluster. To start cluster communication doesn't traverse > secure protocols. However, that means you'll have to build up your own > means of getting the log data, alerts, and checking on process status. > Yes. We run splunk forwarder on the laptops too and I sym link the bro logs into /var/log which splunk forwards out for indexing. Process checking is another thing I have to figure out. > > > (4) Related to (3) most of us use Bro to passively monitor network links. > If your Bro process is sending data back out over the same network > connection that it's monitoring you'll need to be very careful not to build > a snowball effect. > Agreed. Gotcha. > > > (5) We've been tracking our Bro policies in git for some time now, works > great. > cool. > > > (6) Do your developers run a fairly standard system configuration on their > endpoints or would you have to potentially build Bro for a lot of different > environments? > Yes. This is going to be running on modern MacBook Pro laptops fully loaded. Should be no issues. > > > (7) Maybe you could have Bro running on the endpoint only when the > developer is traveling or otherwise on a less trusted (unmonitored) network? > Worth thinking about. > > -Dop > > > > On Wed, Mar 8, 2017 at 5:50 AM, Dan Ecott wrote: > > Hello. > > I am exploring whether Bro can work for my company in a particular use > case. What I would like to do is run Bro sensors on developer laptops, > centrally manage the Bro scripts that run on those end points and ensure > the Bro process is always running. > > What is the best way to run a deployment like this? Has it been done > before? Bro Cluster doesn't look like the right solution. > > As far as managing the scripts, I was thinking of building an AWS code > pipeline where I can promote scripts through a Git repo, then have a > process whereby approved scripts get pushed out to the end points quickly. > > Any help on this would be appreciated. > > Dan. > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/10d56934/attachment.html From dnthayer at illinois.edu Wed Mar 8 15:55:20 2017 From: dnthayer at illinois.edu (Daniel Thayer) Date: Wed, 8 Mar 2017 17:55:20 -0600 Subject: [Bro] BROKER + CLUSTER - stuck (Mike Dopheide) In-Reply-To: References: <433339B7-7067-436D-BD2C-05E14736D116@illinois.edu> <8DB42B0C-608B-431F-85F4-A3A0DA656EB5@illinois.edu> <5E1E1CD1-68D6-4E2C-B2EB-C5BD5CF97B60@illinois.edu> Message-ID: <5cec25a1-f0da-0a6c-1d92-62205fe9d7d7@illinois.edu> Although I couldn't reproduce this problem, I have a possible fix. If you decide to try it, let me know if it fixes the problem. Apply the following patch to /usr/local/bro/share/bro/broctl/check.bro --- check.bro.orig 2017-03-08 17:49:53.000000000 -0600 +++ check.bro 2017-03-08 17:49:37.000000000 -0600 @@ -17,3 +17,6 @@ Log::remove_filter(LoadedScripts::LOG, "default"); Log::add_filter(LoadedScripts::LOG, f); } + +# This prevents "broctl scripts" from hanging. +redef exit_only_after_terminate = F; And apply the following patch to /usr/local/bro/share/broctl/scripts/check-config --- check-config.orig 2017-03-08 17:51:54.000000000 -0600 +++ check-config 2017-03-08 17:52:05.000000000 -0600 @@ -45,7 +45,13 @@ echo $@ >.cmdline touch .checking -"${bro}" "$@" +check_option="-a" +if [ "$print_loaded_scripts" = "1" ]; then + # No bro logs are created with "-a", so don't use it with "broctl scripts". + check_option= +fi + +"${bro}" $check_option "$@" rc=$? if [ $rc -eq 0 -a "$print_loaded_scripts" = "1" ]; then From fatema.bannatwala at gmail.com Wed Mar 8 16:07:49 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Wed, 8 Mar 2017 19:07:49 -0500 Subject: [Bro] BROKER + CLUSTER - stuck (Mike Dopheide) In-Reply-To: <5cec25a1-f0da-0a6c-1d92-62205fe9d7d7@illinois.edu> References: <433339B7-7067-436D-BD2C-05E14736D116@illinois.edu> <8DB42B0C-608B-431F-85F4-A3A0DA656EB5@illinois.edu> <5E1E1CD1-68D6-4E2C-B2EB-C5BD5CF97B60@illinois.edu> <5cec25a1-f0da-0a6c-1d92-62205fe9d7d7@illinois.edu> Message-ID: Deploy works fine when I run it on our standalone test instance, but hangs when run on prod cluster, so it might be the cluster specific issue? Thanks Dan for providing the patch, will try it on Bro cluster and then let you know. :) On Wed, Mar 8, 2017 at 6:55 PM, Daniel Thayer wrote: > Although I couldn't reproduce this problem, I have a > possible fix. If you decide to try it, let me know > if it fixes the problem. > > Apply the following patch to > /usr/local/bro/share/bro/broctl/check.bro > > --- check.bro.orig 2017-03-08 17:49:53.000000000 -0600 > +++ check.bro 2017-03-08 17:49:37.000000000 -0600 > @@ -17,3 +17,6 @@ > Log::remove_filter(LoadedScripts::LOG, "default"); > Log::add_filter(LoadedScripts::LOG, f); > } > + > +# This prevents "broctl scripts" from hanging. > +redef exit_only_after_terminate = F; > > > > And apply the following patch to > /usr/local/bro/share/broctl/scripts/check-config > > --- check-config.orig 2017-03-08 17:51:54.000000000 -0600 > +++ check-config 2017-03-08 17:52:05.000000000 -0600 > @@ -45,7 +45,13 @@ > echo $@ >.cmdline > touch .checking > > -"${bro}" "$@" > +check_option="-a" > +if [ "$print_loaded_scripts" = "1" ]; then > + # No bro logs are created with "-a", so don't use it with "broctl > scripts". > + check_option= > +fi > + > +"${bro}" $check_option "$@" > rc=$? > > if [ $rc -eq 0 -a "$print_loaded_scripts" = "1" ]; then > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170308/bec40f2f/attachment.html From soehlert at es.net Thu Mar 9 00:23:03 2017 From: soehlert at es.net (Samuel Oehlert) Date: Thu, 9 Mar 2017 02:23:03 -0600 Subject: [Bro] BRO on the endpoint, how to manage. In-Reply-To: References: Message-ID: Are you attempting to monitor traffic coming to and from the hosts? If so, wouldn't it be easier to keep bro at the network level and require VPN connections for remote employees? Another possible option if you're just looking for some general endpoint information is to try out the bro-osquery connection: ( https://github.com/bro/bro-osquery) Just a little more food for thought. -Sam On Wed, Mar 8, 2017 at 5:53 PM, Dan Ecott wrote: > Some responses inline. I may try and leverage AWS lambda triggers on git > commits and Jamf to get new scripts down to the laptops somehow. > > Thanks for your responses. I will check back in if I make any good > progress. > > Dan. > > On Wed, Mar 8, 2017 at 11:55 AM Mike Dopheide wrote: > >> I don't know anyone else that's tried this, but it's an interesting >> thought experiment. A few initial thoughts in no particular order... >> >> (1) Given that Bro can be relatively CPU intensive, your developers will >> likely hate you for having something like that running on the same system >> where they're trying to do their work. I'd suggest setting up a one-off >> example and getting some real data on performance impact. >> > > Definitely going to look at how this impacts the use of the laptops. > Initial assessments on my own show that the bro processes don't use > anything more than 2% CPU with the out of the box scripts. > >> >> >> (2) Cool idea! >> > > If it works. :-) > >> >> (3) I'd definitely run these as one-off Bro instances rather than trying >> to make it a cluster. To start cluster communication doesn't traverse >> secure protocols. However, that means you'll have to build up your own >> means of getting the log data, alerts, and checking on process status. >> > > Yes. We run splunk forwarder on the laptops too and I sym link the bro > logs into /var/log which splunk forwards out for indexing. Process checking > is another thing I have to figure out. > >> >> >> (4) Related to (3) most of us use Bro to passively monitor network >> links. If your Bro process is sending data back out over the same network >> connection that it's monitoring you'll need to be very careful not to build >> a snowball effect. >> > > Agreed. Gotcha. > >> >> >> (5) We've been tracking our Bro policies in git for some time now, works >> great. >> > > cool. > >> >> >> (6) Do your developers run a fairly standard system configuration on >> their endpoints or would you have to potentially build Bro for a lot of >> different environments? >> > > Yes. This is going to be running on modern MacBook Pro laptops fully > loaded. Should be no issues. > >> >> >> (7) Maybe you could have Bro running on the endpoint only when the >> developer is traveling or otherwise on a less trusted (unmonitored) network? >> > > Worth thinking about. > >> >> -Dop >> >> >> >> On Wed, Mar 8, 2017 at 5:50 AM, Dan Ecott wrote: >> >> Hello. >> >> I am exploring whether Bro can work for my company in a particular use >> case. What I would like to do is run Bro sensors on developer laptops, >> centrally manage the Bro scripts that run on those end points and ensure >> the Bro process is always running. >> >> What is the best way to run a deployment like this? Has it been done >> before? Bro Cluster doesn't look like the right solution. >> >> As far as managing the scripts, I was thinking of building an AWS code >> pipeline where I can promote scripts through a Git repo, then have a >> process whereby approved scripts get pushed out to the end points quickly. >> >> Any help on this would be appreciated. >> >> Dan. >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> >> > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170309/96605812/attachment.html From matt.clemons at gmail.com Thu Mar 9 09:40:18 2017 From: matt.clemons at gmail.com (Matt Clemons) Date: Thu, 9 Mar 2017 11:40:18 -0600 Subject: [Bro] bro master crashing Message-ID: And I can't tell why. One master. 26 worker systems. Total of 200 worker processes. All centos6. Bro 2.5. Crashes just started happening last night. System has been running since the release of 2.5 with 0 issues. Any way to tell why it's crashing? So far, all i have is the email from broctl and it's not very helpful. --------- This crash report does not include a backtrace. In order for crash reports to be useful when Bro crashes, a backtrace is needed. No core file found. Bro 2.5 Linux 2.6.32-573.3.1.el6.x86_64 Bro plugins: (none found) ==== No reporter.log ==== stderr.log received termination signal ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -U .status -p broctl -p broctl-live -p local -p SERVERNAME local.bro broctl base/frameworks/cluster local-manager.bro broctl/auto ==== .env_vars PATH=/opt/bro/bin:/opt/bro/share/broctl/scripts:/opt/bro/bin:/opt/rh/devtoolset-3/root/usr/bin:/opt/bro/bin:/sbin:/bin:/usr/sbin:/usr/bin BROPATH=/data/bro/spool/installed-scripts-do-not-touch/site::/data/bro/spool/installed-scripts-do-not-touch/auto:/opt/bro/share/bro:/opt/bro/share/bro/policy:/opt/bro/share/bro/site CLUSTER_NODE=SERVERNAME ==== .status TERMINATED [atexit] ==== No prof.log ==== No packet_filter.log ==== No loaded_scripts.log -- [Automatically generated.] -- Regards, Matt Clemons (816) 200-0789 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170309/efabd069/attachment-0001.html From jazoff at illinois.edu Thu Mar 9 09:55:00 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 9 Mar 2017 17:55:00 +0000 Subject: [Bro] bro master crashing In-Reply-To: References: Message-ID: > On Mar 9, 2017, at 12:40 PM, Matt Clemons wrote: > > And I can't tell why. > > One master. 26 worker systems. Total of 200 worker processes. All centos6. Bro 2.5. > > Crashes just started happening last night. System has been running since the release of 2.5 with 0 issues. I'm actually surprised that works at all. Because bro currently (but not for much longer) uses select for handling connections from all the workers, the manager will fail as soon as it gets enough connections for a file descriptor to hit above 1024. You used to hit that limit around 175 workers. Though now that I think of it, we fixed a .bro script leak in 2.5, so I think the new limit may be around 220 for bro 2.5. The next version of bro should hopefully not have a limit :-) > Any way to tell why it's crashing? So far, all i have is the email from broctl and it's not very helpful. > This message: > received termination signal > Means something killed it, probably the kernel OOM killer. Does syslog show anything? -- - Justin Azoff From matt.clemons at gmail.com Thu Mar 9 09:59:03 2017 From: matt.clemons at gmail.com (Matt Clemons) Date: Thu, 9 Mar 2017 11:59:03 -0600 Subject: [Bro] bro master crashing In-Reply-To: References: Message-ID: I've tried commenting out all workers in node.cfg except for the one master, one proxy, and one worker system using 6 worker processes. Still crashes after around 15 seconds. On Thu, Mar 9, 2017 at 11:55 AM, Azoff, Justin S wrote: > > On Mar 9, 2017, at 12:40 PM, Matt Clemons > wrote: > > > > And I can't tell why. > > > > One master. 26 worker systems. Total of 200 worker processes. All > centos6. Bro 2.5. > > > > Crashes just started happening last night. System has been running > since the release of 2.5 with 0 issues. > > I'm actually surprised that works at all. Because bro currently (but not > for much longer) uses select for handling connections from all the workers, > the manager will fail as soon as it gets enough connections for a file > descriptor to hit above 1024. You used to hit that limit around 175 > workers. Though now that I think of it, we fixed a .bro script leak in > 2.5, so I think the new limit may be around 220 for bro 2.5. The next > version of bro should hopefully not have a limit :-) > > > Any way to tell why it's crashing? So far, all i have is the email from > broctl and it's not very helpful. > > > > This message: > > > received termination signal > > > > Means something killed it, probably the kernel OOM killer. Does syslog > show anything? > > > -- > - Justin Azoff > > > -- Regards, Matt Clemons (816) 200-0789 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170309/4e4d3470/attachment.html From jazoff at illinois.edu Thu Mar 9 10:01:54 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 9 Mar 2017 18:01:54 +0000 Subject: [Bro] bro master crashing In-Reply-To: References: Message-ID: <019AE829-183E-46B4-B06D-A618F088F8B6@illinois.edu> > On Mar 9, 2017, at 12:59 PM, Matt Clemons wrote: > > I've tried commenting out all workers in node.cfg except for the one master, one proxy, and one worker system using 6 worker processes. Still crashes after around 15 seconds. > If it says "received termination signal " it's not crashing, something is killing it. -- - Justin Azoff From matt.clemons at gmail.com Thu Mar 9 10:57:36 2017 From: matt.clemons at gmail.com (Matt Clemons) Date: Thu, 9 Mar 2017 12:57:36 -0600 Subject: [Bro] bro master crashing In-Reply-To: <019AE829-183E-46B4-B06D-A618F088F8B6@illinois.edu> References: <019AE829-183E-46B4-B06D-A618F088F8B6@illinois.edu> Message-ID: Ran several manual bro start commands, and i always get "received termination signal." Writes to a nohup.out files that contains that string. broctl diag says... ==== .status TERMINATING [done_with_network] Not sure what to do here. The manager doesn't have a sniffer nic. It's purpose is accepting data in from the worker nodes. On Thu, Mar 9, 2017 at 12:01 PM, Azoff, Justin S wrote: > > > On Mar 9, 2017, at 12:59 PM, Matt Clemons > wrote: > > > > I've tried commenting out all workers in node.cfg except for the one > master, one proxy, and one worker system using 6 worker processes. Still > crashes after around 15 seconds. > > > > If it says "received termination signal " it's not crashing, something is > killing it. > > -- > - Justin Azoff > > -- Regards, Matt Clemons (816) 200-0789 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170309/990e4097/attachment.html From jazoff at illinois.edu Thu Mar 9 11:28:59 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 9 Mar 2017 19:28:59 +0000 Subject: [Bro] bro master crashing In-Reply-To: References: <019AE829-183E-46B4-B06D-A618F088F8B6@illinois.edu> Message-ID: > On Mar 9, 2017, at 1:57 PM, Matt Clemons wrote: > > Ran several manual bro start commands, and i always get "received termination signal." Writes to a nohup.out files that contains that string. Do you have an agent or cron job or something running on the machine that could be killing bro for some reason? -- - Justin Azoff From promero at cenic.org Thu Mar 9 11:32:49 2017 From: promero at cenic.org (Philip Romero) Date: Thu, 9 Mar 2017 11:32:49 -0800 Subject: [Bro] Try.Bro.Org Table Creation Inquiry In-Reply-To: <6715D779-DBC1-4817-860A-5BC8BC733CBF@illinois.edu> References: <9f8f4f5c-93d9-a437-b45b-228c30e145fc@cenic.org> <6715D779-DBC1-4817-860A-5BC8BC733CBF@illinois.edu> Message-ID: Justin, Thanks for the use of hook pointer. I'm able to retrieve the data elements directly from the hook, but I am still struggling for feeding this data into a table. All the searching I find only speaks of feeding data in from a file. I can't find any source for how to feed this "live" data into a table. UPDATED CODE: hook Notice::policy(n: Notice::Info) { if ( n$sub == "remote" && n$note == Scan::Port_Scan || n$note == Scan::Address_Scan && n$sub == "remote") { local ssrc= n$src; local sts= n$ts; local snote = n$note; print ssrc, sts, snote; } } On 3/8/17 11:37 AM, Azoff, Justin S wrote: > It isn't working on try.bro.org because it's running against a pcap > and the bro process only runs for a fraction of a second before > exiting. At startup the file doesn't exist yet and the initial read > will fail. > This won't work properly on a live cluster though due to issues that only recently got fixed with the input framework. The next version of bro will re-try input files that couldn't be read at startup. Previously, the input framework would stop trying after the initial failure. > > In any case.. what you are trying to do doesn't actually require the use of files. You can just add a Notice::policy hook and add the scanner ip directly to the scanners table. > > > -- Philip Romero, CISSP, CISA Sr. Information Security Analyst CENIC promero at cenic.org Phone: (714) 220-3430 Mobile: (562) 237-9290 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170309/642add01/attachment-0001.html From jazoff at illinois.edu Thu Mar 9 11:39:26 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 9 Mar 2017 19:39:26 +0000 Subject: [Bro] Try.Bro.Org Table Creation Inquiry In-Reply-To: References: <9f8f4f5c-93d9-a437-b45b-228c30e145fc@cenic.org> <6715D779-DBC1-4817-860A-5BC8BC733CBF@illinois.edu> Message-ID: <51CBEE17-F664-4E58-B13C-2EC5E883F1D6@illinois.edu> > On Mar 9, 2017, at 2:32 PM, Philip Romero wrote: > > Justin, > Thanks for the use of hook pointer. I'm able to retrieve the data elements directly from the hook, but I am still struggling for feeding this data into a table. All the searching I find only speaks of feeding data in from a file. I can't find any source for how to feed this "live" data into a table. > > UPDATED CODE: > hook Notice::policy(n: Notice::Info) > { > if ( n$sub == "remote" && n$note == Scan::Port_Scan || n$note == Scan::Address_Scan && n$sub == "remote") > { > local ssrc = n$src; > local sts = n$ts; > local snote = n$note; > print ssrc, sts, snote; > } > } Try this hook Notice::policy(n: Notice::Info) { if ( n$sub == "remote" && (n$note == Scan::Port_Scan || n$note == Scan::Address_Scan)) scanners[n$src] = [$ts=n$ts, $note=cat(n$note)]; } n$note is actually a notice type enum so to fit in the table you described in your previous email you need to stringify it. -- - Justin Azoff From dwdixon at umich.edu Thu Mar 9 13:27:55 2017 From: dwdixon at umich.edu (Drew Dixon) Date: Thu, 9 Mar 2017 16:27:55 -0500 Subject: [Bro] Capture Loss In-Reply-To: References: Message-ID: Did you search the email list already or did you just join the list? Are you capturing the traffic from a SPAN port or a Tap? Is your network full of asymmetrical traffic/routing? Answers to these two questions first is pretty important IMO. I responded to a very similar question around 6 days ago or so on list...here's what I said again: _____________________________ First I think the recommended number of workers is something like number of *real* cores (not counting hyperthreading) -2 so for 8 *real* cores you would use 6 workers, if you have 16 *real* cores you probably want closer to 14 workers if this is a dedicated bro box. Maybe try bumping up your number of workers and enabling cpu pinning if you haven't done so. Have you reviewed everything located here? : https://www.bro.org/documentation/faq.html#how-can-i-reduce- the-amount-of-captureloss-or-dropped-packets-notices Specifically a few things come to mind...I know you mentioned NIC settings but are you sure you disabled all the NIC offloading features using ethtool?, more detail on that at this link: http://securityonion.blogspot.com/2011/10/when-is-full-packe t-capture-not-full.html Also, wouldn't hurt to double check the the pf_ring kernel module is loaded/loading staying loaded? If you patch the server and the kernel gets updated unless you have something automated to reload/reinstall the pf_ring module you will probably need to reload the pf_ring module for the new kernel... Also, did you configure the number of ring slots for PF_RING ? Check to be sure that /etc/modprobe.d/pf_ring.conf exists for your PF_RING installation...this is where you will configure the number of ring slots for PF_RING, the default is 4096 I believe but on busy networks this needs to be increased as appropriate (in increments of 4096)...the max value is 65534. I would try that if you've tried everything else at the first link above to no avail... This is also a great resource re: PF_RING and number of ring slots: https://groups.google.com/forum/#!topic/security-onion/zu7U7U9pBT8 Hope this helps, -Drew ____________________________ On Tue, Mar 7, 2017 at 10:34 AM, Arash Fallah wrote: > I'm running Bro in a clustered configuration using PF_RING to have 8 > separate workers on one box. Additionally, I have commented out almost > everything in the default local.bro to run in Bro as efficiently as > possible. Together, these 8 workers are using less than 20% of total CPU > capacity. > > However, we are experiencing capture loss consistently in the 50% range, > even though CPUs are idle 80% of the time on average. > > Does anyone have any experience with this? I would greatly appreciate the > help. > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170309/2dc57927/attachment.html From matt.clemons at gmail.com Thu Mar 9 15:29:47 2017 From: matt.clemons at gmail.com (Matt Clemons) Date: Thu, 9 Mar 2017 17:29:47 -0600 Subject: [Bro] bro master crashing In-Reply-To: References: <019AE829-183E-46B4-B06D-A618F088F8B6@illinois.edu> Message-ID: Lots of these. 0.000000 Reporter::ERROR no such index (Cluster::nodes[Intel::p$descr]) /opt/bro/share/bro/base/frameworks/intel/./cluster.bro, line 35 0.000000 Reporter::ERROR no such index (Cluster::nodes[Intel::p$descr]) /opt/bro/share/bro/base/frameworks/intel/./cluster.bro, line 35 0.000000 Reporter::ERROR no such index (Cluster::nodes[Intel::p$descr]) /opt/bro/share/bro/base/frameworks/intel/./cluster.bro, line 35 0.000000 Reporter::ERROR no such index (Cluster::nodes[Intel::p$descr]) /opt/bro/share/bro/base/frameworks/intel/./cluster.bro, line 35 So I commented out that section just for grins, and it still crashes. [mclemons at bromaster-kcc:~/logs/current ] $ tail -f reporter.log 1489101446.599386 Reporter::INFO processing continued (empty) 1489101446.582511 Reporter::INFO processing continued (empty) 1489101446.565019 Reporter::INFO processing suspended (empty) 1489101446.565019 Reporter::INFO processing continued (empty) 1489101446.637924 Reporter::INFO processing suspended (empty) 1489101446.637924 Reporter::INFO processing continued (empty) 1489101446.728349 Reporter::INFO processing continued (empty) 1489101446.681030 Reporter::INFO processing continued (empty) 1489101446.751914 Reporter::INFO processing continued (empty) 1489101446.755815 Reporter::INFO processing continued (empty) 0.000000 Reporter::INFO received termination signal (empty) #close 2017-03-09-23-19-16 Child died in the communication.log. And a segfault: 2017-03-09T18:34:06.409225+00:00 HOSTNAME kernel: bro[60506]: segfault at 0 ip 00000000005fcf8d sp 00007fffaf9d2f40 error 6 in bro[400000+624000] On Thu, Mar 9, 2017 at 5:06 PM, Azoff, Justin S wrote: > > > On Mar 9, 2017, at 5:11 PM, Matt Clemons wrote: > > > > I've disabled cron. > > > > Still getting "received termination signal." and "child died" in the > communications.log. > > Ah! "child died" makes things interesting. That's literally the only > thing that can cause bro to say 'received termination signal' for an > internal reason. I completely forgot about this case :-( > > When the child process that handles communication dies, the parent can't > continue without it so it kills itself so the whole thing can be restarted > in a known working state. > > Is there anything that shows up in your reporter.log or communication.log > right before this happens? > > Is the kernel logging any segfaults to syslog? > > -- > - Justin Azoff > > -- Regards, Matt Clemons (816) 200-0789 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170309/bd7e37c0/attachment.html From sunari1031 at gmail.com Thu Mar 9 20:15:54 2017 From: sunari1031 at gmail.com (=?UTF-8?B?6rmA7IiY66Co?=) Date: Fri, 10 Mar 2017 13:15:54 +0900 Subject: [Bro] All file extraction Message-ID: Hi All, I am new to Bro and want to extract all files on my network. (smb, http, and whatever all protocols) I probably need to set up proper server spec and bro cluster to extract all files. However I don't know what the spec I need. On my network traffic is below. - maximum throughput is around 55m bits per second. - maximum packets are around 6k packets per second. Please give me some advice to build bro. And I have one more question. Some extracted files' hash isn't same to origin file' hash when I tested bro on virtual machine before setting up bro on real network. Is it because of the server spec? (lost some packets?) Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170310/2bf5b0c9/attachment.html From daniel.guerra69 at gmail.com Fri Mar 10 00:54:04 2017 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Fri, 10 Mar 2017 09:54:04 +0100 Subject: [Bro] bro elasticsearch plugin + kibana indexing In-Reply-To: References: <0385FC7C-66B4-4778-B154-6EAFE12B2E3B@gmail.com> <9032382A-62D9-485F-9D5B-500B0CE05583@gmail.com> <1E0D4AA4-40E8-4AD6-BB7B-F54CDC6F416B@gmail.com> Message-ID: <491F6D2A-2453-4D71-A245-D6F00AD82713@gmail.com> Google for it :D > On 08 Mar 2017, at 21:42, Alex Kefallonitis wrote: > > How TS_ISO8601 timestamp looks like? I ll try to recompile maybe i didn't apply the patch correctly. Thanks again a lot for your help > > 2017-03-05 15:41 GMT+02:00 Daniel Guerra >: > The patch wasn?t used, your timestamp is not in TS_ISO8601 but in TS_MILIS > And your separator is a ?.? not a ?_" > > Check my docker > > https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/ > >> On 05 Mar 2017, at 14:12, Alex Kefallonitis > wrote: >> >> Nothing changed >> >> >> >> >> Although i have logs >> >> tail -f /opt/bro/logs/current/conn.log >> {"ts":1488719244.873684,"uid":"CCToVE1JzVl9n5zDnj","id.orig_h":"10.0.0.31","id.orig_p":123,"id.resp_h":"194.177.210.54","id.resp_p":123,"proto":"udp","duration":0.021199,"orig_bytes":0,"resp_bytes":48,"conn_state":"SHR","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"Cd","orig_pkts":0,"orig_ip_bytes":0,"resp_pkts":1,"resp_ip_bytes":76,"tunnel_parents":[],"orig_l2_addr":"b8:27:eb:68:1a:49","resp_l2_addr":"36:34:64:31:64:39"} >> {"ts":1488719252.873686,"uid":"ChHXxg3NsigjS6QwXg","id.orig_h":"10.0.0.31","id.orig_p":123,"id.resp_h":"62.1.45.120","id.resp_p":123,"proto":"udp","duration":0.020193,"orig_bytes":0,"resp_bytes":48,"conn_state":"SHR","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"Cd","orig_pkts":0,"orig_ip_bytes":0,"resp_pkts":1,"resp_ip_bytes":76,"tunnel_parents":[],"orig_l2_addr":"b8:27:eb:68:1a:49","resp_l2_addr":"36:34:64:31:64:39"} >> {"ts":1488719249.686949,"uid":"CIrHst2VsHafEIR4vk","id.orig_h":"10.0.0.3","id.orig_p":123,"id.resp_h":"91.189.89.198","id.resp_p":123,"proto":"udp","duration":0.066331,"orig_bytes":48,"resp_bytes":48,"conn_state":"SF","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"Dd","orig_pkts":1,"orig_ip_bytes":76,"resp_pkts":1,"resp_ip_bytes":76,"tunnel_parents":[],"orig_l2_addr":"32:38:66:64:64:62","resp_l2_addr":"36:34:64:31:64:39"} >> {"ts":1488719306.835847,"uid":"CqrMPYWEW543RxOX5","id.orig_h":"10.0.0.33","id.orig_p":51666,"id.resp_h":"10.0.0.1","id.resp_p":53,"proto":"udp","service":"dns","duration":0.000744,"orig_bytes":39,"resp_bytes":98,"conn_state":"SF","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"Dd","orig_pkts":1,"orig_ip_bytes":67,"resp_pkts":1,"resp_ip_bytes":126,"tunnel_parents":[],"orig_l2_addr":"36:33:62:63:39:61","resp_l2_addr":"36:34:64:31:64:39"} >> {"ts":1488719314.06168,"uid":"CYw1dj2WUBXn6ua8O1","id.orig_h":"10.0.0.31","id.orig_p":37456,"id.resp_h":"10.0.0.1","id.resp_p":53,"proto":"udp","service":"dns","duration":0.000596,"orig_bytes":0,"resp_bytes":172,"conn_state":"SHR","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"Cd","orig_pkts":0,"orig_ip_bytes":0,"resp_pkts":2,"resp_ip_bytes":228,"tunnel_parents":[],"orig_l2_addr":"b8:27:eb:68:1a:49","resp_l2_addr":"36:34:64:31:64:39"} >> >> Do i have to change serator to init.bro also? >> >> >> 2017-03-05 14:57 GMT+02:00 Daniel Guerra >: >> ##! Load this script to enable global log output to an ElasticSearch database. >> >> module LogElasticSearch; >> >> export { >> ## An elasticsearch specific rotation interval. >> const rotation_interval = 1hr &redef; >> >> ## Optionally ignore any :bro:type:`Log::ID` from being sent to >> ## ElasticSearch with this script. >> const excluded_log_ids: set[Log::ID] &redef; >> >> ## If you want to explicitly only send certain :bro:type:`Log::ID` >> ## streams, add them to this set. If the set remains empty, all will >> ## be sent. The :bro:id:`LogElasticSearch::excluded_log_ids` option >> ## will remain in effect as well. >> const send_logs: set[Log::ID] &redef; >> >> ## Set the separator >> redef Log::default_scope_sep = "_"; >> } >> >> >>> On 05 Mar 2017, at 13:56, Alex Kefallonitis > wrote: >>> >>> Where do i put this? >>> >>> redef Log::default_scope_sep = "_?; >>> >>> Do i have to enable json output to ascii.bro? >>> >>> 2017-03-05 14:32 GMT+02:00 Daniel Guerra >: >>> Don?t forget this in the bro script that starts elasticsearch in the export part >>> >>> redef Log::default_scope_sep = "_?; >>> >>> >>>> On 05 Mar 2017, at 11:22, Alex Kefallonitis > wrote: >>>> >>>> I do patch src/ElasticSearch.cc ./ElasticSearch.cc.patch ./configure && make && make install . Load bro elasticsearch script and restart bro open kibana >>>> >>>> >>>> >>>> 2017-03-05 12:14 GMT+02:00 Alex Kefallonitis >: >>>> I try the patch too but still no timestamp appears i am using ELK 5.2.2 >>>> >>>> 2017-03-05 10:27 GMT+02:00 Daniel Guerra >: >>>> Try this >>>> >>>> https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/bro-patch/ElasticSearch.cc.patch >>>> >>>> >>>> > On 05 Mar 2017, at 02:57, Alex Kefallonitis > wrote: >>>> > >>>> > ELK + Kibana not indexing bro logs >>>> > >>>> > Succesfully installed the plugin and ELK but when i add indexing bro-* , index time-field appears empty (@timestamp) so i cannot use bro logs with kibana search. Anyone have same issue? >>>> > _______________________________________________ >>>> > Bro mailing list >>>> > bro at bro-ids.org >>>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>>> >>>> >>>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170310/af813506/attachment-0001.html From jlay at slave-tothe-box.net Fri Mar 10 11:18:45 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 10 Mar 2017 12:18:45 -0700 Subject: [Bro] Disabling an analyzer in weird In-Reply-To: <8e515a6f-1a50-c607-bf92-7d1034da08d1@gmail.com> References: <0d865d16f64276e3ff61c41b4d134a1e@localhost> <8e515a6f-1a50-c607-bf92-7d1034da08d1@gmail.com> Message-ID: <9875d66bb3aefb85d6918022d33e2778@localhost> On 2017-03-08 11:17, Jan Grash?fer wrote: >> Topic :) I'd like to have bro not dump non-rfc compliant syslog >> messages in the weird file. How can I go about doing that? Thank >> you. > > Add a filter for the log might be an option: > https://www.bro.org/sphinx-git/frameworks/logging.html#filter-log-records > > Jan > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro Thanks Jan. So I did more digging...this used to work in 2.4.1: http://mailman.icsi.berkeley.edu/pipermail/bro/2014-July/007178.html But now no longer...I guess I don't want to see binpac exceptions in weird. Any folks have any thoughts on this? Thank you. James From jan.grashoefer at gmail.com Fri Mar 10 11:30:05 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Fri, 10 Mar 2017 20:30:05 +0100 Subject: [Bro] Disabling an analyzer in weird In-Reply-To: <9875d66bb3aefb85d6918022d33e2778@localhost> References: <0d865d16f64276e3ff61c41b4d134a1e@localhost> <8e515a6f-1a50-c607-bf92-7d1034da08d1@gmail.com> <9875d66bb3aefb85d6918022d33e2778@localhost> Message-ID: <6848d391-41eb-ef36-7893-2b808ec1dd9f@gmail.com> > Thanks Jan. So I did more digging...this used to work in 2.4.1: > > http://mailman.icsi.berkeley.edu/pipermail/bro/2014-July/007178.html > > But now no longer...I guess I don't want to see binpac exceptions in > weird. Any folks have any thoughts on this? Thank you. So if disabling the syslog analyzer completely is ok for you that should just work fine with 2.5. Do you see any errors? Jan From bro at pingtrip.com Fri Mar 10 12:03:48 2017 From: bro at pingtrip.com (Dave Crawford) Date: Fri, 10 Mar 2017 15:03:48 -0500 Subject: [Bro] ASN Lookups Message-ID: <8F3EEC1D-FE81-45C4-8D42-DEAB8C4E2FEA@pingtrip.com> Can someone point out the errors in my script to add ASNs to the conn log? The fields are always ?0? in the log but GeoIP is working as expected. MaxMind?s ASN database is here: $ ls -l /usr/share/GeoIP/GeoIPASNum.dat -rw-r--r-- 1 dcrawford dcrawford 4361995 Mar 6 10:14 /usr/share/GeoIP/GeoIPASNum.dat And my add_geo-asn.bro script: redef record Conn::Info += { orig_cc: string &optional &log; resp_cc: string &optional &log; orig_asn: count &optional &log; resp_asn: count &optional &log; }; event connection_state_remove(c: connection) { c$conn$orig_asn = lookup_asn(c$id$orig_h); local orig_loc = lookup_location(c$id$orig_h); if ( orig_loc?$country_code ) c$conn$orig_cc = orig_loc$country_code; c$conn$resp_asn = lookup_asn(c$id$resp_h); local resp_loc = lookup_location(c$id$resp_h); if ( resp_loc?$country_code ) c$conn$resp_cc = resp_loc$country_code; } -Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170310/2305792b/attachment.html From seth at corelight.com Fri Mar 10 12:52:00 2017 From: seth at corelight.com (Seth Hall) Date: Fri, 10 Mar 2017 15:52:00 -0500 Subject: [Bro] ASN Lookups In-Reply-To: <8F3EEC1D-FE81-45C4-8D42-DEAB8C4E2FEA@pingtrip.com> References: <8F3EEC1D-FE81-45C4-8D42-DEAB8C4E2FEA@pingtrip.com> Message-ID: <0AA5BF80-C7BC-4623-82D4-91261FD34D64@corelight.com> Your script looks fine to me. Is it possible you?re seeing messages like "Can't open GeoIP ASNUM database? in your reporter log? .Seth > On Mar 10, 2017, at 3:03 PM, Dave Crawford wrote: > > Can someone point out the errors in my script to add ASNs to the conn log? The fields are always ?0? in the log but GeoIP is working as expected. > > MaxMind?s ASN database is here: > > $ ls -l /usr/share/GeoIP/GeoIPASNum.dat > -rw-r--r-- 1 dcrawford dcrawford 4361995 Mar 6 10:14 /usr/share/GeoIP/GeoIPASNum.dat > > And my add_geo-asn.bro script: > > redef record Conn::Info += { > orig_cc: string &optional &log; > resp_cc: string &optional &log; > orig_asn: count &optional &log; > resp_asn: count &optional &log; > }; > > event connection_state_remove(c: connection) > { > c$conn$orig_asn = lookup_asn(c$id$orig_h); > > local orig_loc = lookup_location(c$id$orig_h); > if ( orig_loc?$country_code ) > c$conn$orig_cc = orig_loc$country_code; > > > c$conn$resp_asn = lookup_asn(c$id$resp_h); > > local resp_loc = lookup_location(c$id$resp_h); > if ( resp_loc?$country_code ) > c$conn$resp_cc = resp_loc$country_code; > } > > -Dave > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From bro at pingtrip.com Fri Mar 10 13:01:33 2017 From: bro at pingtrip.com (Dave Crawford) Date: Fri, 10 Mar 2017 16:01:33 -0500 Subject: [Bro] ASN Lookups In-Reply-To: <0AA5BF80-C7BC-4623-82D4-91261FD34D64@corelight.com> References: <8F3EEC1D-FE81-45C4-8D42-DEAB8C4E2FEA@pingtrip.com> <0AA5BF80-C7BC-4623-82D4-91261FD34D64@corelight.com> Message-ID: Ahh yes, there is an error: Reporter::ERROR Can't open GeoIP ASNUM database: /usr/share/GeoIP/GeoIPASNum.dat (lookup_asn(c$id$orig_h)) But the permissions look correct: $ ls -l /usr/share/GeoIP/GeoIPASNum.dat -rw-r--r-- 1 dcrawford dcrawford 4361995 Mar 6 10:14 /usr/share/GeoIP/GeoIPASNum.dat Perhaps I grabbed the wrong version of the MaxMind ASN DB? This is the one I installed: http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz > On Mar 10, 2017, at 3:52 PM, Seth Hall wrote: > > Your script looks fine to me. Is it possible you?re seeing messages like "Can't open GeoIP ASNUM database? in your reporter log? > > .Seth -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170310/4bc830b5/attachment.html From jlay at slave-tothe-box.net Fri Mar 10 13:22:49 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 10 Mar 2017 14:22:49 -0700 Subject: [Bro] Disabling an analyzer in weird In-Reply-To: <6848d391-41eb-ef36-7893-2b808ec1dd9f@gmail.com> References: <0d865d16f64276e3ff61c41b4d134a1e@localhost> <8e515a6f-1a50-c607-bf92-7d1034da08d1@gmail.com> <9875d66bb3aefb85d6918022d33e2778@localhost> <6848d391-41eb-ef36-7893-2b808ec1dd9f@gmail.com> Message-ID: <08b0b3ccad2384ccef9c9032190ca56d@localhost> Thanks Jan, I got this to fly with disabling the analyzer, but as I look at the weird.log there are several items I'd like to filter out. For example: dns_unmatched_msg inappropriate_FIN and others. I've looked at the code snippet as shown below: function http_only(rec: Conn::Info) : bool { # Record only connections with successfully analyzed HTTP traffic return rec?$service && rec$service == "http"; } event bro_init() { local filter: Log::Filter = [$name="http-only", $path="conn-http", $pred=http_only]; Log::add_filter(Conn::LOG, filter); } and, as usual when I stare at bro code snippets, I'm completely lost. I get that the above creates a new log and only http from conn.log, but I have no idea how to tweak this to filter out things from weird.log. I've looked at: http://try.bro.org/#/?example=logs-filter-logs http://blog.bro.org/2012/02/filtering-logs-with-bro.html https://www.bro.org/development/projects/logging-api.html I see a lot of these are about splitting into new logs or filtering out fields...none of which I want to do. Any additional guidance on negating entries from logs would be excellent. Thank you...bro always makes me feel stupid 8-/ James On 2017-03-10 12:30, Jan Grash?fer wrote: >> Thanks Jan. So I did more digging...this used to work in 2.4.1: >> >> http://mailman.icsi.berkeley.edu/pipermail/bro/2014-July/007178.html >> >> But now no longer...I guess I don't want to see binpac exceptions in >> weird. Any folks have any thoughts on this? Thank you. > > So if disabling the syslog analyzer completely is ok for you that should > just work fine with 2.5. Do you see any errors? > > Jan > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170310/ee046aa9/attachment.html From filus at psc.edu Fri Mar 10 13:45:11 2017 From: filus at psc.edu (Shane Filus) Date: Fri, 10 Mar 2017 16:45:11 -0500 Subject: [Bro] Disabling an analyzer in weird In-Reply-To: <08b0b3ccad2384ccef9c9032190ca56d@localhost> References: <0d865d16f64276e3ff61c41b4d134a1e@localhost> <8e515a6f-1a50-c607-bf92-7d1034da08d1@gmail.com> <9875d66bb3aefb85d6918022d33e2778@localhost> <6848d391-41eb-ef36-7893-2b808ec1dd9f@gmail.com> <08b0b3ccad2384ccef9c9032190ca56d@localhost> Message-ID: <919237e7-f397-d989-3bd2-cbd0100441a7@psc.edu> On 3/10/17 4:22 PM, James Lay wrote: > Thanks Jan, > > I got this to fly with disabling the analyzer, but as I look at the > weird.log there are several items I'd like to filter out. For example: > > dns_unmatched_msg > inappropriate_FIN Hi James, Specifically to weird logging, you can redef individual messages: redef Weird::actions["dns_unmatched_msg"] = Weird::ACTION_IGNORE; redef Weird::actions["dns_unmatched_reply"] = Weird::ACTION_IGNORE; https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html Re-reading, didn't realize there were more actions than IGNORE(and LOG). Smart. Thanks! Shane From jan.grashoefer at gmail.com Fri Mar 10 14:05:20 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Fri, 10 Mar 2017 23:05:20 +0100 Subject: [Bro] Disabling an analyzer in weird In-Reply-To: <919237e7-f397-d989-3bd2-cbd0100441a7@psc.edu> References: <0d865d16f64276e3ff61c41b4d134a1e@localhost> <8e515a6f-1a50-c607-bf92-7d1034da08d1@gmail.com> <9875d66bb3aefb85d6918022d33e2778@localhost> <6848d391-41eb-ef36-7893-2b808ec1dd9f@gmail.com> <08b0b3ccad2384ccef9c9032190ca56d@localhost> <919237e7-f397-d989-3bd2-cbd0100441a7@psc.edu> Message-ID: > Specifically to weird logging, you can redef individual messages: > > redef Weird::actions["dns_unmatched_msg"] = Weird::ACTION_IGNORE; > redef Weird::actions["dns_unmatched_reply"] = Weird::ACTION_IGNORE; Just remembered that as I read "dns_unmatched_reply". Thanks for helping out, Shane! > Re-reading, didn't realize there were more actions than IGNORE(and LOG). > Smart. That's the reason why this mechanism would be preferred for filtering weird. Thanks, Jan From jlay at slave-tothe-box.net Fri Mar 10 15:11:21 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 10 Mar 2017 16:11:21 -0700 Subject: [Bro] Disabling an analyzer in weird In-Reply-To: References: <0d865d16f64276e3ff61c41b4d134a1e@localhost> <8e515a6f-1a50-c607-bf92-7d1034da08d1@gmail.com> <9875d66bb3aefb85d6918022d33e2778@localhost> <6848d391-41eb-ef36-7893-2b808ec1dd9f@gmail.com> <08b0b3ccad2384ccef9c9032190ca56d@localhost> <919237e7-f397-d989-3bd2-cbd0100441a7@psc.edu> Message-ID: Perfect...thanks Shane and Jan...I'll give it a go and report my findings. James On 2017-03-10 15:05, Jan Grash?fer wrote: >> Specifically to weird logging, you can redef individual messages: >> >> redef Weird::actions["dns_unmatched_msg"] = Weird::ACTION_IGNORE; >> redef Weird::actions["dns_unmatched_reply"] = >> Weird::ACTION_IGNORE; > > Just remembered that as I read "dns_unmatched_reply". Thanks for > helping > out, Shane! > >> Re-reading, didn't realize there were more actions than IGNORE(and >> LOG). >> Smart. > > That's the reason why this mechanism would be preferred for filtering > weird. > > Thanks, > Jan > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jlay at slave-tothe-box.net Fri Mar 10 15:28:02 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 10 Mar 2017 16:28:02 -0700 Subject: [Bro] Disabling an analyzer in weird In-Reply-To: References: <0d865d16f64276e3ff61c41b4d134a1e@localhost> <8e515a6f-1a50-c607-bf92-7d1034da08d1@gmail.com> <9875d66bb3aefb85d6918022d33e2778@localhost> <6848d391-41eb-ef36-7893-2b808ec1dd9f@gmail.com> <08b0b3ccad2384ccef9c9032190ca56d@localhost> <919237e7-f397-d989-3bd2-cbd0100441a7@psc.edu> Message-ID: Well I'm certainly close. Thanks to the redef I'm able to squelch out a lot of noise, but alas, not the binpac exception. If I disable the analyzer I don't get any syslog.log file, so that's not what I need in this case. I'll keep digging..thanks again for all the help. James On 2017-03-10 16:11, James Lay wrote: > Perfect...thanks Shane and Jan...I'll give it a go and report my > findings. > > James > > On 2017-03-10 15:05, Jan Grash?fer wrote: >>> Specifically to weird logging, you can redef individual messages: >>> >>> redef Weird::actions["dns_unmatched_msg"] = Weird::ACTION_IGNORE; >>> redef Weird::actions["dns_unmatched_reply"] = >>> Weird::ACTION_IGNORE; >> >> Just remembered that as I read "dns_unmatched_reply". Thanks for >> helping >> out, Shane! >> >>> Re-reading, didn't realize there were more actions than IGNORE(and >>> LOG). >>> Smart. >> >> That's the reason why this mechanism would be preferred for filtering >> weird. >> >> Thanks, >> Jan >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From bro at pingtrip.com Fri Mar 10 16:18:20 2017 From: bro at pingtrip.com (Dave Crawford) Date: Fri, 10 Mar 2017 19:18:20 -0500 Subject: [Bro] ASN Lookups In-Reply-To: References: <8F3EEC1D-FE81-45C4-8D42-DEAB8C4E2FEA@pingtrip.com> <0AA5BF80-C7BC-4623-82D4-91261FD34D64@corelight.com> Message-ID: Closing the loop on this? totally self-inflicted. I deployed the MaxMind database to the manger but forgot to also deploy to all the sensors. Everything is working as intended now. > On Mar 10, 2017, at 4:01 PM, Dave Crawford wrote: > > Ahh yes, there is an error: > > Reporter::ERROR Can't open GeoIP ASNUM database: /usr/share/GeoIP/GeoIPASNum.dat (lookup_asn(c$id$orig_h)) > > But the permissions look correct: > > $ ls -l /usr/share/GeoIP/GeoIPASNum.dat > -rw-r--r-- 1 dcrawford dcrawford 4361995 Mar 6 10:14 /usr/share/GeoIP/GeoIPASNum.dat > > Perhaps I grabbed the wrong version of the MaxMind ASN DB? This is the one I installed: > > http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz > > >> On Mar 10, 2017, at 3:52 PM, Seth Hall > wrote: >> >> Your script looks fine to me. Is it possible you?re seeing messages like "Can't open GeoIP ASNUM database? in your reporter log? >> >> .Seth > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170310/de0c22df/attachment.html From jan.grashoefer at gmail.com Sat Mar 11 12:46:53 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Sat, 11 Mar 2017 21:46:53 +0100 Subject: [Bro] Disabling an analyzer in weird In-Reply-To: References: <0d865d16f64276e3ff61c41b4d134a1e@localhost> <8e515a6f-1a50-c607-bf92-7d1034da08d1@gmail.com> <9875d66bb3aefb85d6918022d33e2778@localhost> <6848d391-41eb-ef36-7893-2b808ec1dd9f@gmail.com> <08b0b3ccad2384ccef9c9032190ca56d@localhost> <919237e7-f397-d989-3bd2-cbd0100441a7@psc.edu> Message-ID: <0f822295-a9d5-4225-7879-0653aab70567@gmail.com> Hi James, > Well I'm certainly close. Thanks to the redef I'm able to squelch out a > lot of noise, but alas, not the binpac exception. If I disable the > analyzer I don't get any syslog.log file, so that's not what I need in > this case. I'll keep digging..thanks again for all the help. if that particular notice is not listed in Weird::actions you can still just filter manually. Something like that might work for you: http://try.bro.org/#/trybro/saved/130377 Jan From jlay at slave-tothe-box.net Sat Mar 11 15:36:20 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Sat, 11 Mar 2017 16:36:20 -0700 Subject: [Bro] Disabling an analyzer in weird In-Reply-To: <0f822295-a9d5-4225-7879-0653aab70567@gmail.com> References: <0d865d16f64276e3ff61c41b4d134a1e@localhost> <8e515a6f-1a50-c607-bf92-7d1034da08d1@gmail.com> <9875d66bb3aefb85d6918022d33e2778@localhost> <6848d391-41eb-ef36-7893-2b808ec1dd9f@gmail.com> <08b0b3ccad2384ccef9c9032190ca56d@localhost> <919237e7-f397-d989-3bd2-cbd0100441a7@psc.edu> <0f822295-a9d5-4225-7879-0653aab70567@gmail.com> Message-ID: <1489275380.2551.0.camel@slave-tothe-box.net> Thanks a bunch Jan...I'll give that a test and report my findings ? James On Sat, 2017-03-11 at 21:46 +0100, Jan Grash?fer wrote: > Hi James, > > > > > Well I'm certainly close.??Thanks to the redef I'm able to squelch > > out a? > > lot of noise, but alas, not the binpac exception.??If I disable > > the? > > analyzer I don't get any syslog.log file, so that's not what I need > > in? > > this case. I'll keep digging..thanks again for all the help. > if that particular notice is not listed in Weird::actions you can > still > just filter manually. Something like that might work for you: > http://try.bro.org/#/trybro/saved/130377 > > Jan > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170311/d60ee067/attachment.html From bro at pingtrip.com Sat Mar 11 15:43:09 2017 From: bro at pingtrip.com (Dave Crawford) Date: Sat, 11 Mar 2017 18:43:09 -0500 Subject: [Bro] Running specific scripts on specific workers Message-ID: <3231C0D1-CFFA-472D-B6E7-C73F66E4FBA2@pingtrip.com> I have a cluster that has three workers configured in node.cfg and I?m looking for the best approach for limiting the scripts on each. For example, with v2.4 this style config in local.bro worked great: # CONDITIONAL SCRIPT LOADING # @if ( Cluster::is_enabled() ) # INTERNAL ONLY - Matches on workers (MID_INT-1), proxies (MID_INT_PXY_1), and manager (MGR_INT). @if ( /^.{3,3}_INT.*/ in Cluster::node) # load internal specific scripts here @endif # GLR ONLY - Matches on workers (MID_GLR-1), proxies (MID_INT_PXY), and manager (MGR_INT). @if ( /^(MID_GLR|[DIMNW]{3,3}_INT_PXY|MGR_INT).*/ in Cluster::node ) # Load GLR specifc scripts @endif # DNS ONLY - Matches on workers (MID_GLR-1), proxies (MID_INT_PXY), and manager (MGR_INT). @if ( /^(MID_DNS|[DIMNW]{3,3}_INT_PXY|MGR_INT).*/ in Cluster::node ) # Load DNS specifc scripts @endif @endif However, I?ve started seeing an oddity since moving to v2.5 where some events in notice.log have an entirely unrelated ?note? value. If I remove the conditional script loading, and load all scripts everywhere, the problem goes away. I did limited testing with ?aux_scripts? in nod.cfg but was unsure of the proper config. I vaguely recall reading that if scripts weren?t loaded on the proxies and manager, as well as the worker, things could malfunction. Would a better approach be to move conditional logic into the specific scripts themselves? For example, if node ==?GLR? then exit. -Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170311/f5f2d001/attachment.html From bill.de.ping at gmail.com Sun Mar 12 05:56:37 2017 From: bill.de.ping at gmail.com (william de ping) Date: Sun, 12 Mar 2017 14:56:37 +0200 Subject: [Bro] adding fields to HTTP log - cluster environment Message-ID: Hi everyone, I am trying to add a new field to HTTP log. I want to check if orig_h is in a table, if true then add the value from that table to the record. I have a script that works in a single bro instance, but does not work in a cluster environment: @load base/protocol/http redef record HTTP::Info += { field: string &log &optional; } event http_message_done(c: connection, is_orig: bool, stat: http_message_stat) { if ( c$http$id$orig_h in test_table ) { c$http$field = test_table[c$http$id$orig_h]; } } I am not sure why this script works with bro in a single instance mode but not in cluster mode. Also, giving a higher priority to http_message_done event will override the actual event in main.bro under http ? thanks B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170312/8ea76818/attachment.html From bro at pingtrip.com Sun Mar 12 09:03:17 2017 From: bro at pingtrip.com (Dave Crawford) Date: Sun, 12 Mar 2017 12:03:17 -0400 Subject: [Bro] adding fields to HTTP log - cluster environment In-Reply-To: References: Message-ID: <01DFC18E-EADA-4012-9647-7E7E05D94730@pingtrip.com> Do you have ?test_table? set as ?&synchronized?? > On Mar 12, 2017, at 8:56 AM, william de ping wrote: > > Hi everyone, > > I am trying to add a new field to HTTP log. > I want to check if orig_h is in a table, if true then add the value from that table to the record. > > I have a script that works in a single bro instance, but does not work in a cluster environment: > > @load base/protocol/http > > redef record HTTP::Info += { > field: string &log &optional; > } > > event http_message_done(c: connection, is_orig: bool, stat: http_message_stat) > { > if ( c$http$id$orig_h in test_table ) > { > c$http$field = test_table[c$http$id$orig_h]; > } > } > > I am not sure why this script works with bro in a single instance mode but not in cluster mode. > Also, giving a higher priority to http_message_done event will override the actual event in main.bro under http ? > > thanks > B > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From dan.ecott at gmail.com Mon Mar 13 03:34:38 2017 From: dan.ecott at gmail.com (Dan Ecott) Date: Mon, 13 Mar 2017 06:34:38 -0400 Subject: [Bro] Best way to autostart local BRO on Mac OS X Message-ID: Hello. I have had a couple of attempts at writing a plist file that auto starts BRO on my developer Mac but havent been able to get any to work yet. I am deploying the plist file in the /Library/LaunchAgents and LaunchDaemons directory but it doesn't seem to want to work. Does anyone have something like this working? Goal is to ensure the sensor is always running from system startup and wake. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170313/8bf69f27/attachment.html From bill.de.ping at gmail.com Mon Mar 13 07:00:37 2017 From: bill.de.ping at gmail.com (william de ping) Date: Mon, 13 Mar 2017 16:00:37 +0200 Subject: [Bro] adding fields to HTTP log - cluster environment In-Reply-To: <01DFC18E-EADA-4012-9647-7E7E05D94730@pingtrip.com> References: <01DFC18E-EADA-4012-9647-7E7E05D94730@pingtrip.com> Message-ID: Hi, my mistake, another script ran and removed the default fields on HTTP :) Thanks anyways B On Sun, Mar 12, 2017 at 6:03 PM, Dave Crawford wrote: > Do you have ?test_table? set as ?&synchronized?? > > > On Mar 12, 2017, at 8:56 AM, william de ping > wrote: > > > > Hi everyone, > > > > I am trying to add a new field to HTTP log. > > I want to check if orig_h is in a table, if true then add the value from > that table to the record. > > > > I have a script that works in a single bro instance, but does not work > in a cluster environment: > > > > @load base/protocol/http > > > > redef record HTTP::Info += { > > field: string &log &optional; > > } > > > > event http_message_done(c: connection, is_orig: bool, stat: > http_message_stat) > > { > > if ( c$http$id$orig_h in test_table ) > > { > > c$http$field = test_table[c$http$id$orig_h]; > > } > > } > > > > I am not sure why this script works with bro in a single instance mode > but not in cluster mode. > > Also, giving a higher priority to http_message_done event will override > the actual event in main.bro under http ? > > > > thanks > > B > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170313/1f3ba18a/attachment.html From fatema.bannatwala at gmail.com Mon Mar 13 08:10:18 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Mon, 13 Mar 2017 11:10:18 -0400 Subject: [Bro] BROKER + CLUSTER - stuck (Mike Dopheide) In-Reply-To: References: <433339B7-7067-436D-BD2C-05E14736D116@illinois.edu> <8DB42B0C-608B-431F-85F4-A3A0DA656EB5@illinois.edu> <5E1E1CD1-68D6-4E2C-B2EB-C5BD5CF97B60@illinois.edu> <5cec25a1-f0da-0a6c-1d92-62205fe9d7d7@illinois.edu> Message-ID: Hi Dan, Thanks for the patch, finally tested on production cluster and seems to be working fine. Thanks, Fatema. On Wed, Mar 8, 2017 at 7:07 PM, fatema bannatwala < fatema.bannatwala at gmail.com> wrote: > Deploy works fine when I run it on our standalone test instance, but hangs > when run on prod cluster, so it might be the cluster specific issue? > Thanks Dan for providing the patch, will try it on Bro cluster and then > let you know. :) > > > > > On Wed, Mar 8, 2017 at 6:55 PM, Daniel Thayer > wrote: > >> Although I couldn't reproduce this problem, I have a >> possible fix. If you decide to try it, let me know >> if it fixes the problem. >> >> Apply the following patch to >> /usr/local/bro/share/bro/broctl/check.bro >> >> --- check.bro.orig 2017-03-08 17:49:53.000000000 -0600 >> +++ check.bro 2017-03-08 17:49:37.000000000 -0600 >> @@ -17,3 +17,6 @@ >> Log::remove_filter(LoadedScripts::LOG, "default"); >> Log::add_filter(LoadedScripts::LOG, f); >> } >> + >> +# This prevents "broctl scripts" from hanging. >> +redef exit_only_after_terminate = F; >> >> >> >> And apply the following patch to >> /usr/local/bro/share/broctl/scripts/check-config >> >> --- check-config.orig 2017-03-08 17:51:54.000000000 -0600 >> +++ check-config 2017-03-08 17:52:05.000000000 -0600 >> @@ -45,7 +45,13 @@ >> echo $@ >.cmdline >> touch .checking >> >> -"${bro}" "$@" >> +check_option="-a" >> +if [ "$print_loaded_scripts" = "1" ]; then >> + # No bro logs are created with "-a", so don't use it with "broctl >> scripts". >> + check_option= >> +fi >> + >> +"${bro}" $check_option "$@" >> rc=$? >> >> if [ $rc -eq 0 -a "$print_loaded_scripts" = "1" ]; then >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170313/b51fa53e/attachment.html From jlay at slave-tothe-box.net Mon Mar 13 10:26:42 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Mon, 13 Mar 2017 11:26:42 -0600 Subject: [Bro] Disabling an analyzer in weird In-Reply-To: <1489275380.2551.0.camel@slave-tothe-box.net> References: <0d865d16f64276e3ff61c41b4d134a1e@localhost> <8e515a6f-1a50-c607-bf92-7d1034da08d1@gmail.com> <9875d66bb3aefb85d6918022d33e2778@localhost> <6848d391-41eb-ef36-7893-2b808ec1dd9f@gmail.com> <08b0b3ccad2384ccef9c9032190ca56d@localhost> <919237e7-f397-d989-3bd2-cbd0100441a7@psc.edu> <0f822295-a9d5-4225-7879-0653aab70567@gmail.com> <1489275380.2551.0.camel@slave-tothe-box.net> Message-ID: <47bfafd89500a666e4a8c4f243866516@localhost> Well I gave it a shot...no go though: 1489425830.509505 CD8sYx3dttq6ynlg2c x.x.x.x 51132 x.x.x.x 514 binpac exception: string mismatch at /home/build/bro-2.5/src/analyzer/protocol/syslog/syslog-protocol.pac:8: \x0aexpected pattern: "[[:digit:]]+"\x0aactual data: "x09MSWinEventLog\x091\x09Application\x09674838\x09Mon Mar 13 11:23:50 \x0a" - F worker-3-5 Ok Seth...how does stop either a) weird from analyzing a protocol, or b) logging binpac errors? Thanks. James On 2017-03-11 16:36, James Lay wrote: > Thanks a bunch Jan...I'll give that a test and report my findings ? > > James > > On Sat, 2017-03-11 at 21:46 +0100, Jan Grash?fer wrote: > >> Hi James, >> >>> Well I'm certainly close. Thanks to the redef I'm able to squelch >>> out a lot of noise, but alas, not the binpac exception. If I >>> disable the analyzer I don't get any syslog.log file, so that's >>> not what I need in this case. I'll keep digging..thanks again for >>> all the help. >> if that particular notice is not listed in Weird::actions you can >> still just filter manually. Something like that might work for you: >> http://try.bro.org/#/trybro/saved/130377 Jan >> _______________________________________________ Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From espressobeanies at gmail.com Mon Mar 13 12:18:35 2017 From: espressobeanies at gmail.com (Espresso Beanies) Date: Mon, 13 Mar 2017 15:18:35 -0400 Subject: [Bro] Question on Bro efficiency and bonded interfaces running async traffic Message-ID: My Bro setup has two hard links, each running uplink and downlink traffic separately. Would it be more efficient for Bro to define each hard link in the node.cfg or do a soft-bond that merges both hard links into a virtual interface, that channels into Bro? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170313/571c7633/attachment.html From jan.grashoefer at gmail.com Mon Mar 13 12:33:55 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Mon, 13 Mar 2017 20:33:55 +0100 Subject: [Bro] Disabling an analyzer in weird In-Reply-To: <47bfafd89500a666e4a8c4f243866516@localhost> References: <0d865d16f64276e3ff61c41b4d134a1e@localhost> <8e515a6f-1a50-c607-bf92-7d1034da08d1@gmail.com> <9875d66bb3aefb85d6918022d33e2778@localhost> <6848d391-41eb-ef36-7893-2b808ec1dd9f@gmail.com> <08b0b3ccad2384ccef9c9032190ca56d@localhost> <919237e7-f397-d989-3bd2-cbd0100441a7@psc.edu> <0f822295-a9d5-4225-7879-0653aab70567@gmail.com> <1489275380.2551.0.camel@slave-tothe-box.net> <47bfafd89500a666e4a8c4f243866516@localhost> Message-ID: <4509dfb4-27b5-ca7a-ddc8-013e811fc5d5@gmail.com> Hi James, > Well I gave it a shot...no go though: > > 1489425830.509505 CD8sYx3dttq6ynlg2c x.x.x.x 51132 > x.x.x.x 514 binpac exception: string mismatch at > /home/build/bro-2.5/src/analyzer/protocol/syslog/syslog-protocol.pac:8: > \x0aexpected pattern: "[[:digit:]]+"\x0aactual data: > "x09MSWinEventLog\x091\x09Application\x09674838\x09Mon Mar 13 > 11:23:50 \x0a" - F worker-3-5 How did you customize the filter_weird function to match that line? Looks like the name field also contains some context-dependent info, so that you might need a regex. However, if you see a lot of this, it might be a good idea to dig deeper into the analyzer. Can you provide a pcap for testing? Jan From seth at corelight.com Mon Mar 13 12:46:58 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 13 Mar 2017 15:46:58 -0400 Subject: [Bro] Question on Bro efficiency and bonded interfaces running async traffic In-Reply-To: References: Message-ID: > On Mar 13, 2017, at 3:18 PM, Espresso Beanies wrote: > > My Bro setup has two hard links, each running uplink and downlink traffic separately. Would it be more efficient for Bro to define each hard link in the node.cfg or do a soft-bond that merges both hard links into a virtual interface, that channels into Bro? You will need to merge the interfaces. You can?t monitor them separately because a Bro process needs to see both sides of a connection, but if you run with each interface on a different Bro process, each process will only see a single direction of traffic. If you merge/bond interfaces, it?s very possible that some of your connections will be messed up as well because there is no synchronization between how packets are received from the separate interfaces and you could receive traffic out of order. I typically recommend that people merge traffic in a switch (SPAN port) or through a packet broker because those will merge the packets from different interfaces correctly. .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From espressobeanies at gmail.com Mon Mar 13 12:55:09 2017 From: espressobeanies at gmail.com (Espresso Beanies) Date: Mon, 13 Mar 2017 15:55:09 -0400 Subject: [Bro] Question on Bro efficiency and bonded interfaces running async traffic In-Reply-To: References: Message-ID: I see. Thanks Seth! On Mon, Mar 13, 2017 at 3:46 PM, Seth Hall wrote: > > > On Mar 13, 2017, at 3:18 PM, Espresso Beanies > wrote: > > > > My Bro setup has two hard links, each running uplink and downlink > traffic separately. Would it be more efficient for Bro to define each hard > link in the node.cfg or do a soft-bond that merges both hard links into a > virtual interface, that channels into Bro? > > You will need to merge the interfaces. You can?t monitor them separately > because a Bro process needs to see both sides of a connection, but if you > run with each interface on a different Bro process, each process will only > see a single direction of traffic. > > If you merge/bond interfaces, it?s very possible that some of your > connections will be messed up as well because there is no synchronization > between how packets are received from the separate interfaces and you > could receive traffic out of order. I typically recommend that people > merge traffic in a switch (SPAN port) or through a packet broker because > those will merge the packets from different interfaces correctly. > > .Seth > > -- > Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170313/ebce5233/attachment.html From jlay at slave-tothe-box.net Mon Mar 13 13:02:05 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Mon, 13 Mar 2017 14:02:05 -0600 Subject: [Bro] Disabling an analyzer in weird In-Reply-To: <4509dfb4-27b5-ca7a-ddc8-013e811fc5d5@gmail.com> References: <0d865d16f64276e3ff61c41b4d134a1e@localhost> <8e515a6f-1a50-c607-bf92-7d1034da08d1@gmail.com> <9875d66bb3aefb85d6918022d33e2778@localhost> <6848d391-41eb-ef36-7893-2b808ec1dd9f@gmail.com> <08b0b3ccad2384ccef9c9032190ca56d@localhost> <919237e7-f397-d989-3bd2-cbd0100441a7@psc.edu> <0f822295-a9d5-4225-7879-0653aab70567@gmail.com> <1489275380.2551.0.camel@slave-tothe-box.net> <47bfafd89500a666e4a8c4f243866516@localhost> <4509dfb4-27b5-ca7a-ddc8-013e811fc5d5@gmail.com> Message-ID: Hi Jan, Thanks for looking at this...I don't want to be a pest and we can take this off list if we need to so as not to drive all the smart people crazy :) Here's what I added: function filter_weird (rec: Weird::Info) : bool { return rec$name ! in set("binpac exception"); } event bro_init() { local filter: Log::Filter = Log::get_filter(Weird::LOG, "default"); filter$pred=filter_weird; Log::add_filter(Weird::LOG, filter); } This is getting "syslogs" from Windows machines via a third party app. Clearly not adhering to the RFC. As for pcap, I cannot as this is sensitive data :( I can share more details off list if needed. Thank you. James On 2017-03-13 13:33, Jan Grash?fer wrote: > Hi James, > >> Well I gave it a shot...no go though: >> >> 1489425830.509505 CD8sYx3dttq6ynlg2c x.x.x.x 51132 >> x.x.x.x 514 binpac exception: string mismatch at >> /home/build/bro-2.5/src/analyzer/protocol/syslog/syslog-protocol.pac:8: >> \x0aexpected pattern: "[[:digit:]]+"\x0aactual data: >> "x09MSWinEventLog\x091\x09Application\x09674838\x09Mon Mar 13 >> 11:23:50 \x0a" - F worker-3-5 > > How did you customize the filter_weird function to match that line? > Looks like the name field also contains some context-dependent info, so > that you might need a regex. However, if you see a lot of this, it > might > be a good idea to dig deeper into the analyzer. Can you provide a pcap > for testing? > > Jan From dwdixon at umich.edu Mon Mar 13 13:32:55 2017 From: dwdixon at umich.edu (Drew Dixon) Date: Mon, 13 Mar 2017 16:32:55 -0400 Subject: [Bro] Best way to autostart local BRO on Mac OS X In-Reply-To: References: Message-ID: Is there some reason you want to make it more complicated by using plist's/launchd/launchctl etc.? Why not just use a good ol' cron job on your Mac, should work just fine. FYI the standard bro cron job to clean bro up periodically will autostart bro if it's not running AFAIK... */5 * * * * /usr/local/bin/broctl cleanup or if 5 minutes on boot is too long after you boot up make it more frequent replacing the 5 with a 2 or test to see if the following works in your crontab as expected (on boot): @reboot /usr/local/bin/broctl deploy Hope this helps, -Drew On Mon, Mar 13, 2017 at 6:34 AM, Dan Ecott > wrote: > Hello. > > I have had a couple of attempts at writing a plist file that auto starts > BRO on my developer Mac but havent been able to get any to work yet. I am > deploying the plist file in the /Library/LaunchAgents and LaunchDaemons > directory but it doesn't seem to want to work. > > Does anyone have something like this working? > > Goal is to ensure the sensor is always running from system startup and > wake. > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170313/82811399/attachment-0001.html From jlay at slave-tothe-box.net Mon Mar 13 13:49:11 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Mon, 13 Mar 2017 14:49:11 -0600 Subject: [Bro] Disabling an analyzer in weird In-Reply-To: <4509dfb4-27b5-ca7a-ddc8-013e811fc5d5@gmail.com> References: <0d865d16f64276e3ff61c41b4d134a1e@localhost> <8e515a6f-1a50-c607-bf92-7d1034da08d1@gmail.com> <9875d66bb3aefb85d6918022d33e2778@localhost> <6848d391-41eb-ef36-7893-2b808ec1dd9f@gmail.com> <08b0b3ccad2384ccef9c9032190ca56d@localhost> <919237e7-f397-d989-3bd2-cbd0100441a7@psc.edu> <0f822295-a9d5-4225-7879-0653aab70567@gmail.com> <1489275380.2551.0.camel@slave-tothe-box.net> <47bfafd89500a666e4a8c4f243866516@localhost> <4509dfb4-27b5-ca7a-ddc8-013e811fc5d5@gmail.com> Message-ID: <4e7da6c36c73d8c5d4e74a3634b0999d@localhost> Big thanks to Jan...I have so much to learn about bro 8-| Anyway solution below for filtering out binpac exception: function filter_weird (rec: Weird::Info) : bool { return /binpac exception/ ! in rec$name; } event bro_init() { local filter: Log::Filter = Log::get_filter(Weird::LOG, "default"); filter$pred=filter_weird; Log::add_filter(Weird::LOG, filter); } Thanks again Jan! James On 2017-03-13 13:33, Jan Grash?fer wrote: > Hi James, > >> Well I gave it a shot...no go though: >> >> 1489425830.509505 CD8sYx3dttq6ynlg2c x.x.x.x 51132 >> x.x.x.x 514 binpac exception: string mismatch at >> /home/build/bro-2.5/src/analyzer/protocol/syslog/syslog-protocol.pac:8: >> \x0aexpected pattern: "[[:digit:]]+"\x0aactual data: >> "x09MSWinEventLog\x091\x09Application\x09674838\x09Mon Mar 13 >> 11:23:50 \x0a" - F worker-3-5 > > How did you customize the filter_weird function to match that line? > Looks like the name field also contains some context-dependent info, so > that you might need a regex. However, if you see a lot of this, it > might > be a good idea to dig deeper into the analyzer. Can you provide a pcap > for testing? > > Jan From jedwards2728 at gmail.com Mon Mar 13 23:56:31 2017 From: jedwards2728 at gmail.com (John Edwards) Date: Tue, 14 Mar 2017 06:56:31 +0000 Subject: [Bro] Apache struts exploit detection In-Reply-To: References: Message-ID: Hi all For the likes of the apache struts web application attack that the actual exploit is contained within a web http GET request. Or let's say any web app attack that is embedded within the referer field like embedded JavaScript can bro actually view or log that level of info? I can see bro will see things like http user agent fields and get or post request but for the actual malicious code embedded further in the request I'm assuming isn't captured? My ips obviously captures that alert data and I can see the the exploit but the bro data from the http log I'll only see "GET / HTTP1.1" and that's all Cheers John -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170314/f60e5b67/attachment.html From zeolla at gmail.com Tue Mar 14 03:08:02 2017 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Tue, 14 Mar 2017 10:08:02 +0000 Subject: [Bro] Apache struts exploit detection In-Reply-To: References: Message-ID: Here's an example script that will detect CVE-2017-5638 exploit attempts and log the contents of the header. https://github.com/set-element/misc-scripts/blob/master/CVE-2017-5638_struts.bro For future reference the key component is: event http_header(c: connection, is_orig: bool, name: string, value: string) &priority=5 { # look if the connection is from offsite and the value is content-type if ( !Site::is_local_addr(c$id$orig_h) && name == "CONTENT-TYPE" && detection_string in value ) { NOTICE([$note=HTTP_StrutsAttack, $src=c$id$orig_h, $msg=fmt("CVE-2017-5638/Struts attack from %s seen: %s", c$id$orig_h, value)]); } } Please note that this is not my script, it is set-element's. Depending on the situation you may want to check the src/dst to add exemptions (vulnerability scanning boxes?), ignore or specifically monitor Site::is_private_addr src/dsts, add $identifier/$suppress_for to the NOTICE, replace $src=... with $conn=c to get more details in the notice log, etc. All depends on what you want, those are just things I would do. Jon On Tue, Mar 14, 2017, 3:04 AM John Edwards wrote: > Hi all > > For the likes of the apache struts web application attack that the actual > exploit is contained within a web http GET request. Or let's say any web > app attack that is embedded within the referer field like embedded > JavaScript can bro actually view or log that level of info? > > I can see bro will see things like http user agent fields and get or post > request but for the actual malicious code embedded further in the request > I'm assuming isn't captured? > > My ips obviously captures that alert data and I can see the the exploit > but the bro data from the http log I'll only see "GET / HTTP1.1" and that's > all > > Cheers > John > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170314/333077d8/attachment.html From dwdixon at umich.edu Tue Mar 14 09:09:52 2017 From: dwdixon at umich.edu (Drew Dixon) Date: Tue, 14 Mar 2017 12:09:52 -0400 Subject: [Bro] Apache struts exploit detection In-Reply-To: References: Message-ID: Deploy this outstanding bro detection script for this vulnerability: https://github.com/initconf/CVE-2017-5638_struts.git On Tue, Mar 14, 2017 at 6:08 AM, Zeolla at GMail.com wrote: > Here's an example script that will detect CVE-2017-5638 exploit attempts > and log the contents of the header. > > https://github.com/set-element/misc-scripts/blob/ > master/CVE-2017-5638_struts.bro > > For future reference the key component is: > > event http_header(c: connection, is_orig: bool, name: string, value: > string) &priority=5 > > { > > # look if the connection is from offsite and the value is content-type > > if ( !Site::is_local_addr(c$id$orig_h) && name == "CONTENT-TYPE" > && detection_string in value ) > > { > > NOTICE([$note=HTTP_StrutsAttack, $src=c$id$orig_h, > $msg=fmt("CVE-2017-5638/Struts attack from %s seen: %s", c$id$orig_h, > value)]); > > } > > } > > Please note that this is not my script, it is set-element's. Depending on > the situation you may want to check the src/dst to add exemptions > (vulnerability scanning boxes?), ignore or specifically monitor > Site::is_private_addr src/dsts, add $identifier/$suppress_for to the > NOTICE, replace $src=... with $conn=c to get more details in the notice > log, etc. All depends on what you want, those are just things I would do. > > Jon > > On Tue, Mar 14, 2017, 3:04 AM John Edwards wrote: > >> Hi all >> >> For the likes of the apache struts web application attack that the actual >> exploit is contained within a web http GET request. Or let's say any web >> app attack that is embedded within the referer field like embedded >> JavaScript can bro actually view or log that level of info? >> >> I can see bro will see things like http user agent fields and get or post >> request but for the actual malicious code embedded further in the request >> I'm assuming isn't captured? >> >> My ips obviously captures that alert data and I can see the the exploit >> but the bro data from the http log I'll only see "GET / HTTP1.1" and that's >> all >> >> Cheers >> John >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- > > Jon > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170314/f27075b8/attachment.html From johanna at icir.org Tue Mar 14 10:55:13 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 14 Mar 2017 10:55:13 -0700 Subject: [Bro] Apache struts exploit detection In-Reply-To: References: Message-ID: <20170314175513.l52k4yslu7lazir5@Beezling.local> And note that Aashish made it so you can install it using bro-pkg :) Johanna On Tue, Mar 14, 2017 at 12:09:52PM -0400, Drew Dixon wrote: > Deploy this outstanding bro detection script for this vulnerability: > > https://github.com/initconf/CVE-2017-5638_struts.git > > On Tue, Mar 14, 2017 at 6:08 AM, Zeolla at GMail.com wrote: > > > Here's an example script that will detect CVE-2017-5638 exploit attempts > > and log the contents of the header. > > > > https://github.com/set-element/misc-scripts/blob/ > > master/CVE-2017-5638_struts.bro > > > > For future reference the key component is: > > > > event http_header(c: connection, is_orig: bool, name: string, value: > > string) &priority=5 > > > > { > > > > # look if the connection is from offsite and the value is content-type > > > > if ( !Site::is_local_addr(c$id$orig_h) && name == "CONTENT-TYPE" > > && detection_string in value ) > > > > { > > > > NOTICE([$note=HTTP_StrutsAttack, $src=c$id$orig_h, > > $msg=fmt("CVE-2017-5638/Struts attack from %s seen: %s", c$id$orig_h, > > value)]); > > > > } > > > > } > > > > Please note that this is not my script, it is set-element's. Depending on > > the situation you may want to check the src/dst to add exemptions > > (vulnerability scanning boxes?), ignore or specifically monitor > > Site::is_private_addr src/dsts, add $identifier/$suppress_for to the > > NOTICE, replace $src=... with $conn=c to get more details in the notice > > log, etc. All depends on what you want, those are just things I would do. > > > > Jon > > > > On Tue, Mar 14, 2017, 3:04 AM John Edwards wrote: > > > >> Hi all > >> > >> For the likes of the apache struts web application attack that the actual > >> exploit is contained within a web http GET request. Or let's say any web > >> app attack that is embedded within the referer field like embedded > >> JavaScript can bro actually view or log that level of info? > >> > >> I can see bro will see things like http user agent fields and get or post > >> request but for the actual malicious code embedded further in the request > >> I'm assuming isn't captured? > >> > >> My ips obviously captures that alert data and I can see the the exploit > >> but the bro data from the http log I'll only see "GET / HTTP1.1" and that's > >> all > >> > >> Cheers > >> John > >> _______________________________________________ > >> Bro mailing list > >> bro at bro-ids.org > >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > -- > > > > Jon > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Tue Mar 14 11:02:14 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 14 Mar 2017 11:02:14 -0700 Subject: [Bro] Running specific scripts on specific workers In-Reply-To: <3231C0D1-CFFA-472D-B6E7-C73F66E4FBA2@pingtrip.com> References: <3231C0D1-CFFA-472D-B6E7-C73F66E4FBA2@pingtrip.com> Message-ID: <20170314180214.bdbvmfj33fhay4km@Beezling.local> Hi, what exactly do you mean by "an entirely unrelated note value"? Do they have the wrong type associated with them? In general, the most significant change between Bro 2.5 and 2.4 in regards to logging is the introduction of the logger node, which might impact your deployments. If you are using the logger node, you probably have to change your scripts in a way that the parts of the scripts that previously were running on the manager are also running on the logger now. Without really looking into details of how enums are currently serialized, I expect that the different nodes of your cluster currently might extend Notice::Type with different values, depending on which scripts were loaded on which nodes. Which might lead to issues like this. Is is possible that that happens? If yes, the solution probably is to run the parts of the script that perform the data type initialization and redef Notice::Type, define data types, open log files, etc. on all cluster nodes, and just use your conditionals around the parts of the scripts that actually perform works (like catch events). I hope this helps :) Johanna On Sat, Mar 11, 2017 at 06:43:09PM -0500, Dave Crawford wrote: > I have a cluster that has three workers configured in node.cfg and I?m looking for the best approach for limiting the scripts on each. For example, with v2.4 this style config in local.bro worked great: > > # CONDITIONAL SCRIPT LOADING # > @if ( Cluster::is_enabled() ) > # INTERNAL ONLY - Matches on workers (MID_INT-1), proxies (MID_INT_PXY_1), and manager (MGR_INT). > @if ( /^.{3,3}_INT.*/ in Cluster::node) > # load internal specific scripts here > @endif > > # GLR ONLY - Matches on workers (MID_GLR-1), proxies (MID_INT_PXY), and manager (MGR_INT). > @if ( /^(MID_GLR|[DIMNW]{3,3}_INT_PXY|MGR_INT).*/ in Cluster::node ) > # Load GLR specifc scripts > @endif > > # DNS ONLY - Matches on workers (MID_GLR-1), proxies (MID_INT_PXY), and manager (MGR_INT). > @if ( /^(MID_DNS|[DIMNW]{3,3}_INT_PXY|MGR_INT).*/ in Cluster::node ) > # Load DNS specifc scripts > @endif > @endif > > However, I?ve started seeing an oddity since moving to v2.5 where some events in notice.log have an entirely unrelated ?note? value. If I remove the conditional script loading, and load all scripts everywhere, the problem goes away. > > I did limited testing with ?aux_scripts? in nod.cfg but was unsure of the proper config. I vaguely recall reading that if scripts weren?t loaded on the proxies and manager, as well as the worker, things could malfunction. > > Would a better approach be to move conditional logic into the specific scripts themselves? For example, if node ==?GLR? then exit. > > -Dave > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Tue Mar 14 11:07:20 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 14 Mar 2017 11:07:20 -0700 Subject: [Bro] redef plugin variables In-Reply-To: <69d7cfdc-8979-3fe9-a05b-c61c10365fb7@lehigh.edu> References: <69d7cfdc-8979-3fe9-a05b-c61c10365fb7@lehigh.edu> Message-ID: <20170314180720.z3sqm5th3e4iqy3t@Beezling.local> Try redef LogElasticSearch::destination = ... basically you have to specify the module name explicitly when redef-ing outside of the module in question. Johanna On Wed, Mar 08, 2017 at 02:50:36PM -0500, Munroe Sollog wrote: > I am using the elasticsearch plugin with NSQ and I am trying to set the following: > > redef destination = "nsq"; > redef server_port = 4151; > redef nsq_topic = "bro_logs"; > > These statements when put in plugins/Bro_Elasticsearch/scripts/init.bro causes everything to work as > expected. However that file gets overwritten when the plugin gets rebuilt. I am trying to figure > out how to put these statements in my local.bro. when I include the above in local.bro I get the > following errors: > > > worker-1-1 scripts failed. > error in /usr/local/bro/share/bro/site/local.bro, line 98: "redef" used but not previously defined > (destination) > internal warning in /usr/local/bro/share/bro/site/local.bro, line 98: Can't document redef of > destination, identifier lookup failed > > I'm assuming I have to prepend these variables with something like "Bro::ElasticSearch" but I can't > find any docs to clarify. > -- > Munroe Sollog > LTS - Senior Network Engineer > x85002 > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From johanna at icir.org Tue Mar 14 11:08:19 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 14 Mar 2017 11:08:19 -0700 Subject: [Bro] Bro error: "too many values to unpack" In-Reply-To: References: Message-ID: <20170314180819.dxm666hb536sgzpk@Beezling.local> Random guess - can you search if you specified http://... instead of just the IP address in node.cfg? Johanna On Wed, Mar 08, 2017 at 12:51:21PM -0500, fatema bannatwala wrote: > I usually once in a while run into an error, when I do a restart on the bro > cluster. > The restart successes but not sure what those error lines mean, as I don't > find > anything abnormal after bro cluster restarts. > > Does anyone have a clue? > > [fatema at mng site]$ /usr/local/bin/restart-bro > removing old policies in > /mnt/brolog/spool/installed-scripts-do-not-touch/site ... > removing old policies in > /mnt/brolog/spool/installed-scripts-do-not-touch/auto ... > creating policy directories ... > installing site policies ... > generating cluster-layout.bro ... > generating local-networks.bro ... > generating broctl-config.bro ... > generating broctl-config.sh ... > updating nodes ... > *Error: cannot create a directory on node proxy-3* > *Error: Failed to establish ssh connection to host 10.10.24.211 > : too many values to unpack* > stopping ... > stopping worker-1-1 ... > stopping worker-1-10 ... > stopping worker-1-11 ... > stopping worker-1-12 ... And SO ON > ... > starting ... > starting logger ... > starting manager ... > starting proxy-1 ... > starting proxy-2 ... > starting proxy-3 ... > starting proxy-4 ... > starting worker-1-1 ... > starting worker-1-10 ... > starting worker-1-11 ... And SO ON > > The restart-bro script looks something like this: > > #!/bin/sh > > sudo -u bro /usr/local/bro/default/bin/broctl install > sudo /usr/local/bro/bin/fix-perms > sudo -u bro /usr/local/bro/default/bin/broctl restart > sudo /usr/local/bro/bin/restart-bro-dependents > > Thanks, > Fatema. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Tue Mar 14 11:13:42 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 14 Mar 2017 11:13:42 -0700 Subject: [Bro] several questions for introducing Bro to commercial system In-Reply-To: <69140395ca8142e18f9344a7b75e90eb@MP-MSGSS-MBX007.msg.nttdata.co.jp> References: <69140395ca8142e18f9344a7b75e90eb@MP-MSGSS-MBX007.msg.nttdata.co.jp> Message-ID: <20170314181342.x4b2gckqqliugta2@Beezling.local> Hi, > 1. Bro stores captured data into XXX.log files(XXX is http for example). > In this case, how much data does Haka store into local file system per > transaction? If you have any reference data, please let me know. I think the best way to answer this is to just try it out for yourself with some Bro log files. The size of log files generally also differs a lot; some of them have much londer lines than others. > 2. When Bro introduced machine has broken and fixed it, is it possible > to continue the process(packet capturing process and storing data > process into local file system) using the fixed machine without any > problems? I am not 100% sure what you mean here. If a maching running a few worker processes fails, they can be restarted later and will just resume sending data to the manager (assuming the installation is still intact). Local held state will be lost however (Bro does not tend to write internal variables to disk). > 3. What is the market share in the network forensic domain? I don't think we have any information on this. Johanna From fatema.bannatwala at gmail.com Tue Mar 14 12:26:14 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Tue, 14 Mar 2017 15:26:14 -0400 Subject: [Bro] Bro error: "too many values to unpack" In-Reply-To: <20170314180819.dxm666hb536sgzpk@Beezling.local> References: <20170314180819.dxm666hb536sgzpk@Beezling.local> Message-ID: Hi Johanna, The proxies, workers, logger and manager, all defined with the host field as their fqdn, in node.cfg. For ex: [logger] type=logger host=manager.udel.edu [manager] type=manager host=manager.udel.edu [proxy-1] type=proxy host=worker1.udel.edu [worker-1] type=worker host=worker1.udel.edu interface=eth1 lb_method=pf_ring lb_procs=22 pin_cpus=4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46 I run into this issue once in a while, and think maybe because the manager or workers might be overloaded during restart and hence manager isn't able to connect to one of the workers (proxy), and connection times out? but didn't really got to know the reason of the other part of the error ("too many values.."). Thanks for looking into it. Appreciate it. Thanks, Fatema. On Tue, Mar 14, 2017 at 2:08 PM, Johanna Amann wrote: > Random guess - can you search if you specified http://... instead of just > the IP address in node.cfg? > > Johanna > > On Wed, Mar 08, 2017 at 12:51:21PM -0500, fatema bannatwala wrote: > > I usually once in a while run into an error, when I do a restart on the > bro > > cluster. > > The restart successes but not sure what those error lines mean, as I > don't > > find > > anything abnormal after bro cluster restarts. > > > > Does anyone have a clue? > > > > [fatema at mng site]$ /usr/local/bin/restart-bro > > removing old policies in > > /mnt/brolog/spool/installed-scripts-do-not-touch/site ... > > removing old policies in > > /mnt/brolog/spool/installed-scripts-do-not-touch/auto ... > > creating policy directories ... > > installing site policies ... > > generating cluster-layout.bro ... > > generating local-networks.bro ... > > generating broctl-config.bro ... > > generating broctl-config.sh ... > > updating nodes ... > > *Error: cannot create a directory on node proxy-3* > > *Error: Failed to establish ssh connection to host 10.10.24.211 > > : too many values to unpack* > > stopping ... > > stopping worker-1-1 ... > > stopping worker-1-10 ... > > stopping worker-1-11 ... > > stopping worker-1-12 ... And SO ON > > ... > > starting ... > > starting logger ... > > starting manager ... > > starting proxy-1 ... > > starting proxy-2 ... > > starting proxy-3 ... > > starting proxy-4 ... > > starting worker-1-1 ... > > starting worker-1-10 ... > > starting worker-1-11 ... And SO ON > > > > The restart-bro script looks something like this: > > > > #!/bin/sh > > > > sudo -u bro /usr/local/bro/default/bin/broctl install > > sudo /usr/local/bro/bin/fix-perms > > sudo -u bro /usr/local/bro/default/bin/broctl restart > > sudo /usr/local/bro/bin/restart-bro-dependents > > > > Thanks, > > Fatema. > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170314/90197888/attachment.html From af7 at umbc.edu Tue Mar 14 14:47:27 2017 From: af7 at umbc.edu (Arash Fallah) Date: Tue, 14 Mar 2017 17:47:27 -0400 Subject: [Bro] Capture Loss In-Reply-To: References: Message-ID: Hey Drew, I've been on the list for over a year, I tried searching to see similar issues but I didn't find it. We are capturing from a span port, we have 3 edge routers and tons of asymmetrical routing. We are experiencing packet loss at such a high rate, we believe the error might be upstream (thanks to Seth)! We are going to try passive taps instead of capturing from SPAN ports. PF_RING is installed with DKMS. All offloading has been disabled and I have been checking reporter.log for invalid checksums (none so far). CPU pinning is enabled. Though I did I did not know about ring slots for PF_RING, I do not think our network at 3Gbps requires increasing the threshold from my research. Thanks so much, you were on point with your questions. On Thu, Mar 9, 2017 at 4:27 PM, Drew Dixon wrote: > Did you search the email list already or did you just join the list? Are > you capturing the traffic from a SPAN port or a Tap? Is your network full > of asymmetrical traffic/routing? Answers to these two questions first is > pretty important IMO. I responded to a very similar question around 6 days > ago or so on list...here's what I said again: > > _____________________________ > > First I think the recommended number of workers is something like number > of *real* cores (not counting hyperthreading) -2 so for 8 *real* cores you > would use 6 workers, if you have 16 *real* cores you probably want closer > to 14 workers if this is a dedicated bro box. Maybe try bumping up your > number of workers and enabling cpu pinning if you haven't done so. > > Have you reviewed everything located here? : > > https://www.bro.org/documentation/faq.html#how-can-i-reduce- > the-amount-of-captureloss-or-dropped-packets-notices > > Specifically a few things come to mind...I know you mentioned NIC settings > but are you sure you disabled all the NIC offloading features using > ethtool?, more detail on that at this link: > > http://securityonion.blogspot.com/2011/10/when-is-full-packe > t-capture-not-full.html > > Also, wouldn't hurt to double check the the pf_ring kernel module is > loaded/loading staying loaded? If you patch the server and the kernel gets > updated unless you have something automated to reload/reinstall the pf_ring > module you will probably need to reload the pf_ring module for the new > kernel... > > Also, did you configure the number of ring slots for PF_RING ? > > Check to be sure that /etc/modprobe.d/pf_ring.conf exists for your PF_RING > installation...this is where you will configure the number of ring slots > for PF_RING, the default is 4096 I believe but on busy networks this needs > to be increased as appropriate (in increments of 4096)...the max value is > 65534. I would try that if you've tried everything else at the first link > above to no avail... > > This is also a great resource re: PF_RING and number of ring slots: > > https://groups.google.com/forum/#!topic/security-onion/zu7U7U9pBT8 > > Hope this helps, > > -Drew > ____________________________ > > On Tue, Mar 7, 2017 at 10:34 AM, Arash Fallah wrote: > >> I'm running Bro in a clustered configuration using PF_RING to have 8 >> separate workers on one box. Additionally, I have commented out almost >> everything in the default local.bro to run in Bro as efficiently as >> possible. Together, these 8 workers are using less than 20% of total CPU >> capacity. >> >> However, we are experiencing capture loss consistently in the 50% range, >> even though CPUs are idle 80% of the time on average. >> >> Does anyone have any experience with this? I would greatly appreciate the >> help. >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170314/2dfea899/attachment.html From dwdixon at umich.edu Tue Mar 14 19:49:17 2017 From: dwdixon at umich.edu (Drew Dixon) Date: Tue, 14 Mar 2017 22:49:17 -0400 Subject: [Bro] Capture Loss In-Reply-To: References: Message-ID: Glad to hear you're now on the right track! You're very welcome. FWIW I think the other person on the other similar thread I copied my reply from might not have known about installing PF_RING with DKMS so wanted to cover possible kernel module issues etc.. I was going to guess your issue was upstream based on what you described in your first email but I didn't want to speculate too much heh. Taps are def. the way to go if you have the option to use them instead of SPAN ports, for sure. Best regards, -Drew On Tue, Mar 14, 2017 at 5:47 PM, Arash Fallah wrote: > Hey Drew, > > I've been on the list for over a year, I tried searching to see similar > issues but I didn't find it. We are capturing from a span port, we have 3 > edge routers and tons of asymmetrical routing. We are experiencing packet > loss at such a high rate, we believe the error might be upstream (thanks to > Seth)! We are going to try passive taps instead of capturing from SPAN > ports. > > PF_RING is installed with DKMS. All offloading has been disabled and I > have been checking reporter.log for invalid checksums (none so far). CPU > pinning is enabled. Though I did I did not know about ring slots for > PF_RING, I do not think our network at 3Gbps requires increasing the > threshold from my research. > > Thanks so much, you were on point with your questions. > > On Thu, Mar 9, 2017 at 4:27 PM, Drew Dixon wrote: > >> Did you search the email list already or did you just join the list? Are >> you capturing the traffic from a SPAN port or a Tap? Is your network full >> of asymmetrical traffic/routing? Answers to these two questions first is >> pretty important IMO. I responded to a very similar question around 6 days >> ago or so on list...here's what I said again: >> >> _____________________________ >> >> First I think the recommended number of workers is something like number >> of *real* cores (not counting hyperthreading) -2 so for 8 *real* cores you >> would use 6 workers, if you have 16 *real* cores you probably want closer >> to 14 workers if this is a dedicated bro box. Maybe try bumping up your >> number of workers and enabling cpu pinning if you haven't done so. >> >> Have you reviewed everything located here? : >> >> https://www.bro.org/documentation/faq.html#how-can-i-reduce- >> the-amount-of-captureloss-or-dropped-packets-notices >> >> Specifically a few things come to mind...I know you mentioned NIC >> settings but are you sure you disabled all the NIC offloading features >> using ethtool?, more detail on that at this link: >> >> http://securityonion.blogspot.com/2011/10/when-is-full-packe >> t-capture-not-full.html >> >> Also, wouldn't hurt to double check the the pf_ring kernel module is >> loaded/loading staying loaded? If you patch the server and the kernel gets >> updated unless you have something automated to reload/reinstall the pf_ring >> module you will probably need to reload the pf_ring module for the new >> kernel... >> >> Also, did you configure the number of ring slots for PF_RING ? >> >> Check to be sure that /etc/modprobe.d/pf_ring.conf exists for your >> PF_RING installation...this is where you will configure the number of ring >> slots for PF_RING, the default is 4096 I believe but on busy networks this >> needs to be increased as appropriate (in increments of 4096)...the max >> value is 65534. I would try that if you've tried everything else at the >> first link above to no avail... >> >> This is also a great resource re: PF_RING and number of ring slots: >> >> https://groups.google.com/forum/#!topic/security-onion/zu7U7U9pBT8 >> >> Hope this helps, >> >> -Drew >> ____________________________ >> >> On Tue, Mar 7, 2017 at 10:34 AM, Arash Fallah wrote: >> >>> I'm running Bro in a clustered configuration using PF_RING to have 8 >>> separate workers on one box. Additionally, I have commented out almost >>> everything in the default local.bro to run in Bro as efficiently as >>> possible. Together, these 8 workers are using less than 20% of total CPU >>> capacity. >>> >>> However, we are experiencing capture loss consistently in the 50% range, >>> even though CPUs are idle 80% of the time on average. >>> >>> Does anyone have any experience with this? I would greatly appreciate >>> the help. >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170314/c91a8fd5/attachment-0001.html From al.kefallonitis at gmail.com Wed Mar 15 03:00:35 2017 From: al.kefallonitis at gmail.com (Alex Kefallonitis) Date: Wed, 15 Mar 2017 12:00:35 +0200 Subject: [Bro] broctl write output pcap Message-ID: I know that i can run bro -i eth0 -w .pcap . Is there a way broctl to also write to pcap file? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170315/538b29c4/attachment.html From jdopheid at illinois.edu Wed Mar 15 11:21:45 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Wed, 15 Mar 2017 18:21:45 +0000 Subject: [Bro] =?utf-8?q?=5BBro-Dev=5D_BroCon_=E2=80=9917=3A_Registration_?= =?utf-8?q?is_open!?= Message-ID: Friendly reminder that BroCon CFP is open, are you planning on sending us a proposal? ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign On 3/2/17, 11:10 AM, "bro-dev-bounces at bro.org on behalf of Dopheide, Jeannette M" wrote: Bro Community, BroCon ?17 will occur on Tuesday, September 12th - Thursday, September 14th at the National Center for Supercomputing Applications in Urbana, IL. See our event page: https://www.bro.org/community/brocon2017.html Early bird registration is open! CFP is open! Don't forget to book your hotel. Interested in sponsoring BroCon? Contact us at info at bro.org for more information. Thank you for your continued support, and see you in September! Regards, The Bro Project ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign _______________________________________________ bro-dev mailing list bro-dev at bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From andrew.dellana at bayer.com Thu Mar 16 07:47:00 2017 From: andrew.dellana at bayer.com (Andrew Dellana) Date: Thu, 16 Mar 2017 14:47:00 +0000 Subject: [Bro] NetControl configuration Message-ID: <9d900f340345412d8c545f2ced222966@moxde9.na.bayer.cnb> Hello, Is it easier to have a NetControl action in each script or to have one file that contains all the NetControl actions. I want to do one that has all the NetControl actions contained in one script, but am unsure of how / if it is possible to import information from one script to another. And if it is possible to import information to a single NetControl Script would someone be kind enough to provide a template. Freundliche Gr??e / Best regards, Andrew Dellana Intern ________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170316/d2ca55ae/attachment.html From jazoff at illinois.edu Thu Mar 16 07:56:13 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 16 Mar 2017 14:56:13 +0000 Subject: [Bro] NetControl configuration In-Reply-To: <9d900f340345412d8c545f2ced222966@moxde9.na.bayer.cnb> References: <9d900f340345412d8c545f2ced222966@moxde9.na.bayer.cnb> Message-ID: > On Mar 16, 2017, at 10:47 AM, Andrew Dellana wrote: > > Hello, > > Is it easier to have a NetControl action in each script or to have one file that contains all the NetControl actions. I want to do one that has all the NetControl actions contained in one script, but am unsure of how / if it is possible to import information from one script to another. > > And if it is possible to import information to a single NetControl Script would someone be kind enough to provide a template. > > Freundliche Gr??e / Best regards, > > Andrew Dellana > Intern What sort of actions are you talking about? If you are triggering these actions based on a NOTICE being raised, then you can use a notice hook to trigger the netcontrol actions when certain notices are raised. If you just want to store helpers in a file, you just need to do something like # my-netcontrol-actions.bro @load base/frameworks/netcontrol function do_block(ip: addr) { NetControl::drop_address(ip , 20sec, "No internet for you!"); } And then in any other script # my-script.bro @load my-net-control-actions event ... { do_block(id$orig_h); } -- - Justin Azoff From andrew.dellana at bayer.com Thu Mar 16 08:04:40 2017 From: andrew.dellana at bayer.com (Andrew Dellana) Date: Thu, 16 Mar 2017 15:04:40 +0000 Subject: [Bro] NetControl configuration In-Reply-To: References: <9d900f340345412d8c545f2ced222966@moxde9.na.bayer.cnb> Message-ID: Yes, I do want to make the NetControl actions based on what is alerted in Notices. Can all the helpers be stored in one file and only call the helper that is needed? Freundliche Gr??e / Best regards, Andrew Dellana Intern -----Original Message----- From: Azoff, Justin S [mailto:jazoff at illinois.edu] Sent: Thursday, March 16, 2017 10:56 AM To: Andrew Dellana Cc: bro at bro.org Subject: Re: [Bro] NetControl configuration > On Mar 16, 2017, at 10:47 AM, Andrew Dellana wrote: > > Hello, > > Is it easier to have a NetControl action in each script or to have one file that contains all the NetControl actions. I want to do one that has all the NetControl actions contained in one script, but am unsure of how / if it is possible to import information from one script to another. > > And if it is possible to import information to a single NetControl Script would someone be kind enough to provide a template. > > Freundliche Gr??e / Best regards, > > Andrew Dellana > Intern What sort of actions are you talking about? If you are triggering these actions based on a NOTICE being raised, then you can use a notice hook to trigger the netcontrol actions when certain notices are raised. If you just want to store helpers in a file, you just need to do something like # my-netcontrol-actions.bro @load base/frameworks/netcontrol function do_block(ip: addr) { NetControl::drop_address(ip , 20sec, "No internet for you!"); } And then in any other script # my-script.bro @load my-net-control-actions event ... { do_block(id$orig_h); } -- - Justin Azoff From jazoff at illinois.edu Thu Mar 16 08:08:23 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 16 Mar 2017 15:08:23 +0000 Subject: [Bro] NetControl configuration In-Reply-To: References: <9d900f340345412d8c545f2ced222966@moxde9.na.bayer.cnb> Message-ID: > On Mar 16, 2017, at 11:04 AM, Andrew Dellana wrote: > > Yes, I do want to make the NetControl actions based on what is alerted in Notices. Can all the helpers be stored in one file and only call the helper that is needed? Yep, you can do exactly that. -- - Justin Azoff From darkheaven1983 at gmail.com Fri Mar 17 23:18:20 2017 From: darkheaven1983 at gmail.com (duhang) Date: Sat, 18 Mar 2017 14:18:20 +0800 Subject: [Bro] Different behavior between online and offline for http keepalive reqeusts Message-ID: Hi, I'm trying to capture the http request between client and a http proxy which is using keepalive to send multiple requests within one connection. I tried to start a pf_ring cluster and a standalone bro worker using broctl, and also start bro from command line, I saved the pcap file in the meantime. I got incomplete http request logged, also observe url as http method in the log. Then I tried to use offline mode to load pcap file from command line, I got all requests logged without any issue. What's the difference between online and offline mode? Using broctl is even worse than using command line to launch online capture. What's the difference? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170318/4c7af6ff/attachment.html From bro at pingtrip.com Sat Mar 18 10:07:39 2017 From: bro at pingtrip.com (Dave Crawford) Date: Sat, 18 Mar 2017 13:07:39 -0400 Subject: [Bro] PacketFilter Message-ID: <75914B54-6E8A-475C-955F-4FA79E31BBBD@pingtrip.com> I'm attemtpting to impement a packet filter to drop multicast traffic but I?m not having success. This is what I have in local.bro: @load base/frameworks/packet-filter redef capture_filters += { ["ip"] = "ip", ["non-ip"] = "not ip" }; redef restrict_filters += { ["not-multicast"] = "net 224.0.0.0/4" }; Which according to the FAQ (https://www.bro.org/documentation/faq.html) should produce a BPF like: ((ip) or (not ip)) and (not net 224.0.0.0/4) But I'm still seeing multicast in the conn log: 1489855468.534667 CM5Ehj4nefU23EOeyj 192.168.20.8 41340 239.254.127.63 60000 udp It looks like the filters are being implemented: [BroControl] > print capture_filters ext-1 capture_filters = { [non-ip] = not ip, [ip] = ip } [BroControl] > print restrict_filters ext-1 restrict_filters = { [not-multicast] = net 224.0.0.0/4 } Am I missing a step? -Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170318/64b1b596/attachment.html From jlay at slave-tothe-box.net Sat Mar 18 10:21:22 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Sat, 18 Mar 2017 11:21:22 -0600 Subject: [Bro] PacketFilter In-Reply-To: <75914B54-6E8A-475C-955F-4FA79E31BBBD@pingtrip.com> References: <75914B54-6E8A-475C-955F-4FA79E31BBBD@pingtrip.com> Message-ID: <09f4ba432bd5b9d89a007858e30563b5@localhost> On 2017-03-18 11:07, Dave Crawford wrote: > I'm attemtpting to impement a packet filter to drop multicast traffic > but I?m not having success. > > This is what I have in local.bro: > > @load base/frameworks/packet-filter > redef capture_filters += { > ["ip"] = "ip", > ["non-ip"] = "not ip" > }; > > redef restrict_filters += { ["not-multicast"] = "net 224.0.0.0/4" }; > > Which according to the FAQ > (https://www.bro.org/documentation/faq.html) should produce a BPF > like: > > ((ip) or (not ip)) and (not net 224.0.0.0/4) > > But I'm still seeing multicast in the conn log: > > 1489855468.534667 CM5Ehj4nefU23EOeyj 192.168.20.8 41340 > 239.254.127.63 60000 udp > > It looks like the filters are being implemented: > > [BroControl] > print capture_filters > ext-1 capture_filters = { > [non-ip] = not ip, > [ip] = ip > } > > > [BroControl] > print restrict_filters > ext-1 restrict_filters = { > [not-multicast] = net 224.0.0.0/4 > } > > > Am I missing a step? > > -Dave > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro You could always just add it to your broctl.conf like so: broargs = --filter 'your bpf here' James From bro at pingtrip.com Sat Mar 18 11:00:48 2017 From: bro at pingtrip.com (Dave Crawford) Date: Sat, 18 Mar 2017 14:00:48 -0400 Subject: [Bro] PacketFilter In-Reply-To: <09f4ba432bd5b9d89a007858e30563b5@localhost> References: <75914B54-6E8A-475C-955F-4FA79E31BBBD@pingtrip.com> <09f4ba432bd5b9d89a007858e30563b5@localhost> Message-ID: <2BE918E4-4A5D-4C0E-B55A-0745674158BC@pingtrip.com> That method worked perfect, thanks James. I am curious if I was doing something wrong or if PacketFilter is buggy. > On Mar 18, 2017, at 1:21 PM, James Lay wrote: > > You could always just add it to your broctl.conf like so: > > broargs = --filter 'your bpf here' > From bro at pingtrip.com Sat Mar 18 11:01:35 2017 From: bro at pingtrip.com (Dave Crawford) Date: Sat, 18 Mar 2017 14:01:35 -0400 Subject: [Bro] PacketFilter In-Reply-To: <2BE918E4-4A5D-4C0E-B55A-0745674158BC@pingtrip.com> References: <75914B54-6E8A-475C-955F-4FA79E31BBBD@pingtrip.com> <09f4ba432bd5b9d89a007858e30563b5@localhost> <2BE918E4-4A5D-4C0E-B55A-0745674158BC@pingtrip.com> Message-ID: <6ED11E34-3F30-4C36-AE75-115C8FAC63B9@pingtrip.com> Damnit. I spoke too soon: 1489860004.749780 C7LM4TvxWGSWhxOL1 192.168.20.8 40972 239.254.127.63 60000 > On Mar 18, 2017, at 2:00 PM, Dave Crawford wrote: > > That method worked perfect, thanks James. > > I am curious if I was doing something wrong or if PacketFilter is buggy. > >> On Mar 18, 2017, at 1:21 PM, James Lay wrote: >> >> You could always just add it to your broctl.conf like so: >> >> broargs = --filter 'your bpf here' >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170318/7c416151/attachment.html From bro at pingtrip.com Sat Mar 18 12:20:06 2017 From: bro at pingtrip.com (Dave Crawford) Date: Sat, 18 Mar 2017 15:20:06 -0400 Subject: [Bro] PacketFilter In-Reply-To: <6ED11E34-3F30-4C36-AE75-115C8FAC63B9@pingtrip.com> References: <75914B54-6E8A-475C-955F-4FA79E31BBBD@pingtrip.com> <09f4ba432bd5b9d89a007858e30563b5@localhost> <2BE918E4-4A5D-4C0E-B55A-0745674158BC@pingtrip.com> <6ED11E34-3F30-4C36-AE75-115C8FAC63B9@pingtrip.com> Message-ID: <0A3BFD72-3C7C-4C1C-99CF-7617C2823C2A@pingtrip.com> tcpdump doesn?t enforce the filter either. $ sudo tcpdump -nn -i netmap:eth2/Rz not net 224.0.0.0/4 | grep 60000 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on netmap:eth2/Rz, link-type EN10MB (Ethernet), capture size 262144 bytes 15:11:26.286104 IP 192.168.20.8.40364 > 239.254.127.63.60000: UDP, length 44 15:11:26.497024 IP 192.168.20.8.47779 > 239.254.127.63.60000: UDP, length 44 15:11:26.950899 IP 192.168.20.8.38593 > 239.254.127.63.60000: UDP, length 44 I?m at a loss now. > On Mar 18, 2017, at 2:01 PM, Dave Crawford wrote: > > Damnit. I spoke too soon: > > 1489860004.749780 C7LM4TvxWGSWhxOL1 192.168.20.8 40972 239.254.127.63 60000 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170318/a121607b/attachment.html From jlay at slave-tothe-box.net Sat Mar 18 12:30:31 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Sat, 18 Mar 2017 13:30:31 -0600 Subject: [Bro] PacketFilter In-Reply-To: <0A3BFD72-3C7C-4C1C-99CF-7617C2823C2A@pingtrip.com> References: <75914B54-6E8A-475C-955F-4FA79E31BBBD@pingtrip.com> <09f4ba432bd5b9d89a007858e30563b5@localhost> <2BE918E4-4A5D-4C0E-B55A-0745674158BC@pingtrip.com> <6ED11E34-3F30-4C36-AE75-115C8FAC63B9@pingtrip.com> <0A3BFD72-3C7C-4C1C-99CF-7617C2823C2A@pingtrip.com> Message-ID: <1489865431.2426.1.camel@slave-tothe-box.net> That's weird....I can't reproduce that here...on Ubuntu 16 across the board here. ?Maybe libpcap or interface issue? ?My only guess. On Sat, 2017-03-18 at 15:20 -0400, Dave Crawford wrote: > tcpdump doesn?t enforce the filter either. > > $ sudo tcpdump -nn -i netmap:eth2/Rz not net 224.0.0.0/4 | grep 60000 > > tcpdump: verbose output suppressed, use -v or -vv for full protocol > decode > listening on netmap:eth2/Rz, link-type EN10MB (Ethernet), capture > size 262144 bytes > 15:11:26.286104 IP 192.168.20.8.40364 > 239.254.127.63.60000: UDP, > length 44 > 15:11:26.497024 IP 192.168.20.8.47779 > 239.254.127.63.60000: UDP, > length 44 > 15:11:26.950899 IP 192.168.20.8.38593 > 239.254.127.63.60000: UDP, > length 44 > > I?m at a loss now. > > > On Mar 18, 2017, at 2:01 PM, Dave Crawford > > wrote: > > > > Damnit. I spoke too soon: > > > > 1489860004.749780 C7LM4TvxWGSWhxOL1 192.168.20.8 > > 40972 239.254.127.63 60000 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170318/ccee50ad/attachment.html From bro at pingtrip.com Sat Mar 18 12:34:48 2017 From: bro at pingtrip.com (Dave Crawford) Date: Sat, 18 Mar 2017 15:34:48 -0400 Subject: [Bro] PacketFilter In-Reply-To: <1489865431.2426.1.camel@slave-tothe-box.net> References: <75914B54-6E8A-475C-955F-4FA79E31BBBD@pingtrip.com> <09f4ba432bd5b9d89a007858e30563b5@localhost> <2BE918E4-4A5D-4C0E-B55A-0745674158BC@pingtrip.com> <6ED11E34-3F30-4C36-AE75-115C8FAC63B9@pingtrip.com> <0A3BFD72-3C7C-4C1C-99CF-7617C2823C2A@pingtrip.com> <1489865431.2426.1.camel@slave-tothe-box.net> Message-ID: Thanks for validating James. I?m running netmap + netmap-libpcap and then compiled tcpdump 4.9.0. So looking like a netmap bug. > On Mar 18, 2017, at 3:30 PM, James Lay wrote: > > That's weird....I can't reproduce that here...on Ubuntu 16 across the board here. Maybe libpcap or interface issue? My only guess. > From jlay at slave-tothe-box.net Sat Mar 18 12:48:55 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Sat, 18 Mar 2017 13:48:55 -0600 Subject: [Bro] PacketFilter In-Reply-To: References: <75914B54-6E8A-475C-955F-4FA79E31BBBD@pingtrip.com> <09f4ba432bd5b9d89a007858e30563b5@localhost> <2BE918E4-4A5D-4C0E-B55A-0745674158BC@pingtrip.com> <6ED11E34-3F30-4C36-AE75-115C8FAC63B9@pingtrip.com> <0A3BFD72-3C7C-4C1C-99CF-7617C2823C2A@pingtrip.com> <1489865431.2426.1.camel@slave-tothe-box.net> Message-ID: <1489866535.2426.2.camel@slave-tothe-box.net> You bet....good luck with the fix...I'd be curious to know what the fix is. James On Sat, 2017-03-18 at 15:34 -0400, Dave Crawford wrote: > Thanks for validating James. I?m running netmap + netmap-libpcap and > then compiled tcpdump 4.9.0. So looking like a netmap bug. > > > > > > On Mar 18, 2017, at 3:30 PM, James Lay > > wrote: > > > > That's weird....I can't reproduce that here...on Ubuntu 16 across > > the board here.??Maybe libpcap or interface issue???My only guess. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170318/892c86a5/attachment.html From jazoff at illinois.edu Sun Mar 19 07:37:54 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Sun, 19 Mar 2017 14:37:54 +0000 Subject: [Bro] PacketFilter In-Reply-To: <0A3BFD72-3C7C-4C1C-99CF-7617C2823C2A@pingtrip.com> References: <75914B54-6E8A-475C-955F-4FA79E31BBBD@pingtrip.com> <09f4ba432bd5b9d89a007858e30563b5@localhost> <2BE918E4-4A5D-4C0E-B55A-0745674158BC@pingtrip.com> <6ED11E34-3F30-4C36-AE75-115C8FAC63B9@pingtrip.com> <0A3BFD72-3C7C-4C1C-99CF-7617C2823C2A@pingtrip.com> Message-ID: <902A835A-1ACC-49C0-801E-CBDF0C56EEBB@illinois.edu> > On Mar 18, 2017, at 3:20 PM, Dave Crawford wrote: > > tcpdump doesn?t enforce the filter either. > > $ sudo tcpdump -nn -i netmap:eth2/Rz not net 224.0.0.0/4 | grep 60000 > > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on netmap:eth2/Rz, link-type EN10MB (Ethernet), capture size 262144 bytes > 15:11:26.286104 IP 192.168.20.8.40364 > 239.254.127.63.60000: UDP, length 44 > 15:11:26.497024 IP 192.168.20.8.47779 > 239.254.127.63.60000: UDP, length 44 > 15:11:26.950899 IP 192.168.20.8.38593 > 239.254.127.63.60000: UDP, length 44 > > I?m at a loss now. Does tcpdump -ve show any encapsulation like vlans is in use? You may need to use sudo tcpdump -nn -i netmap:eth2/Rz vlan and not net 224.0.0.0/4 Or it's a bug in netmap :-) -- - Justin Azoff From troyj at maine.edu Sun Mar 19 14:46:05 2017 From: troyj at maine.edu (Troy Jordan) Date: Sun, 19 Mar 2017 17:46:05 -0400 Subject: [Bro] does Spicy have a BNF? Message-ID: <6e3887ca-9bcc-1247-81a0-a3d64e0bba2a@maine.edu> Does Spicy/BinPAC++ have a BNF? I couldn't find one, if it exists. Thanks! - Troy -- Troy Jordan t r o y j @ m a i n e . e d u GIAC GCIH,GCIA ------------------------------------------------------------ Network Systems Security Analyst Information Technology Security Office University of Maine System ------------------------------------------------------------ 233 Science Building | voice: 207.561.3590 Portland, ME 04103 | fax: 509.351.3650 "As you all know, Security Is Mortals chiefest Enemy" William Shakespeare, Macbeth From bro at pingtrip.com Sun Mar 19 16:36:15 2017 From: bro at pingtrip.com (Dave Crawford) Date: Sun, 19 Mar 2017 19:36:15 -0400 Subject: [Bro] PacketFilter In-Reply-To: <902A835A-1ACC-49C0-801E-CBDF0C56EEBB@illinois.edu> References: <75914B54-6E8A-475C-955F-4FA79E31BBBD@pingtrip.com> <09f4ba432bd5b9d89a007858e30563b5@localhost> <2BE918E4-4A5D-4C0E-B55A-0745674158BC@pingtrip.com> <6ED11E34-3F30-4C36-AE75-115C8FAC63B9@pingtrip.com> <0A3BFD72-3C7C-4C1C-99CF-7617C2823C2A@pingtrip.com> <902A835A-1ACC-49C0-801E-CBDF0C56EEBB@illinois.edu> Message-ID: <71F603E2-FA7D-430F-ADE3-0045033F7359@pingtrip.com> > > Does tcpdump -ve show any encapsulation like vlans is in use? You may need to use > > sudo tcpdump -nn -i netmap:eth2/Rz vlan and not net 224.0.0.0/4 > > Or it's a bug in netmap :-) > > -- > - Justin Azoff > I built a new Bro cluster without Netmap (standard libpcap-dev libraries for Debian 8.7) and the BPF works as expected: $ sudo tcpdump -nn -i eth2 net 224.0.0.0/4 | grep 60000 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth2, link-type EN10MB (Ethernet), capture size 262144 bytes 14:38:37.656784 IP 192.168.20.4.34697 > 239.254.127.63.60000: UDP, length 44 14:38:37.656799 IP 192.168.20.4.34697 > 239.254.127.63.60000: UDP, length 44 14:38:37.656974 IP 192.168.20.4.45799 > 239.254.127.63.60000: UDP, length 44 AND $ sudo tcpdump -nn -i eth2 not net 224.0.0.0/4 | grep 60000 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth2, link-type EN10MB (Ethernet), capture size 262144 bytes 4866 packets received by filter 0 packets dropped by kernel -Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170319/ef2805b1/attachment.html From jlay at slave-tothe-box.net Sun Mar 19 16:46:53 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Sun, 19 Mar 2017 17:46:53 -0600 Subject: [Bro] PacketFilter In-Reply-To: <71F603E2-FA7D-430F-ADE3-0045033F7359@pingtrip.com> References: <75914B54-6E8A-475C-955F-4FA79E31BBBD@pingtrip.com> <09f4ba432bd5b9d89a007858e30563b5@localhost> <2BE918E4-4A5D-4C0E-B55A-0745674158BC@pingtrip.com> <6ED11E34-3F30-4C36-AE75-115C8FAC63B9@pingtrip.com> <0A3BFD72-3C7C-4C1C-99CF-7617C2823C2A@pingtrip.com> <902A835A-1ACC-49C0-801E-CBDF0C56EEBB@illinois.edu> <71F603E2-FA7D-430F-ADE3-0045033F7359@pingtrip.com> Message-ID: <1489967213.2481.5.camel@slave-tothe-box.net> And there you go....I think I attempted netmap a couple months ago...didn't have good results, so stuck with af_packet. ?Looks like netmap needs a massage. James On Sun, 2017-03-19 at 19:36 -0400, Dave Crawford wrote: > > > > > Does tcpdump -ve show any encapsulation like vlans is in use? ?You > > may need to use > > > > sudo tcpdump -nn -i netmap:eth2/Rz vlan and not net 224.0.0.0/4? > > > > Or it's a bug in netmap :-) > > > > --? > > - Justin Azoff > > > I built a new Bro cluster without Netmap (standard libpcap-dev > libraries for Debian 8.7) and the BPF works as expected: > > $ sudo tcpdump -nn -i eth2 net 224.0.0.0/4 | grep 60000 > tcpdump: verbose output suppressed, use -v or -vv for full protocol > decode > listening on eth2, link-type EN10MB (Ethernet), capture size 262144 > bytes > > > 14:38:37.656784 IP 192.168.20.4.34697 > 239.254.127.63.60000: UDP, > length 44 > 14:38:37.656799 IP 192.168.20.4.34697 > 239.254.127.63.60000: UDP, > length 44 > 14:38:37.656974 IP 192.168.20.4.45799 > 239.254.127.63.60000: UDP, > length 44 > > > AND > > $ sudo tcpdump -nn -i eth2 not net 224.0.0.0/4 | grep 60000 > tcpdump: verbose output suppressed, use -v or -vv for full protocol > decode > listening on eth2, link-type EN10MB (Ethernet), capture size 262144 > bytes > > > > > 4866 packets received by filter > 0 packets dropped by kernel > > -Dave > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170319/1544efb4/attachment.html From obdnanr at gmail.com Mon Mar 20 06:17:57 2017 From: obdnanr at gmail.com (Obdnanr smith) Date: Mon, 20 Mar 2017 13:17:57 +0000 Subject: [Bro] Logging to syslog Message-ID: I'm on bro 2.5-101 and bro is logging event data to the system's syslog. Any idea what could cause this? Ubuntu 16.04 cluster configuration. Is this something new with the logger node? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170320/3f977c54/attachment.html From matt.clemons at gmail.com Mon Mar 20 11:34:20 2017 From: matt.clemons at gmail.com (Matt Clemons) Date: Mon, 20 Mar 2017 13:34:20 -0500 Subject: [Bro] bro master crashing In-Reply-To: References: <019AE829-183E-46B4-B06D-A618F088F8B6@illinois.edu> Message-ID: Just wanted to give an update to show how crazy this has been. The segfaults made me think "memory issue", so i ran memtest on the system. It has a lot of mems so this took many hours to complete, and with 0 errors. Pulled power on the system and upon boot, everything came up fine with a limited set of workers. I added all 200+ worker processes back in, and now it's running like a champ again. The only other thing that it could have been was a power outage on one of the 10 gig worker boxes. It kept blipping and coming back up. Bro cron was starting processes, and then that worker system was crashing due to lack of power. This could have caused the manager to fail. But i can't really tell what the root cause was. Thanks for the responses. On Thu, Mar 9, 2017 at 5:29 PM, Matt Clemons wrote: > Lots of these. > > 0.000000 Reporter::ERROR no such index (Cluster::nodes[Intel::p$ > descr]) /opt/bro/share/bro/base/frameworks/intel/./cluster.bro, line 35 > 0.000000 Reporter::ERROR no such index (Cluster::nodes[Intel::p$ > descr]) /opt/bro/share/bro/base/frameworks/intel/./cluster.bro, line 35 > 0.000000 Reporter::ERROR no such index (Cluster::nodes[Intel::p$ > descr]) /opt/bro/share/bro/base/frameworks/intel/./cluster.bro, line 35 > 0.000000 Reporter::ERROR no such index (Cluster::nodes[Intel::p$ > descr]) /opt/bro/share/bro/base/frameworks/intel/./cluster.bro, line 35 > > So I commented out that section just for grins, and it still crashes. > > [mclemons at bromaster-kcc:~/logs/current ] $ tail -f reporter.log > 1489101446.599386 Reporter::INFO processing continued (empty) > 1489101446.582511 Reporter::INFO processing continued (empty) > 1489101446.565019 Reporter::INFO processing suspended (empty) > 1489101446.565019 Reporter::INFO processing continued (empty) > 1489101446.637924 Reporter::INFO processing suspended (empty) > 1489101446.637924 Reporter::INFO processing continued (empty) > 1489101446.728349 Reporter::INFO processing continued (empty) > 1489101446.681030 Reporter::INFO processing continued (empty) > 1489101446.751914 Reporter::INFO processing continued (empty) > 1489101446.755815 Reporter::INFO processing continued (empty) > 0.000000 Reporter::INFO received termination signal (empty) > #close 2017-03-09-23-19-16 > > Child died in the communication.log. > > And a segfault: > 2017-03-09T18:34:06.409225+00:00 HOSTNAME kernel: bro[60506]: segfault at > 0 ip 00000000005fcf8d sp 00007fffaf9d2f40 error 6 in bro[400000+624000] > > On Thu, Mar 9, 2017 at 5:06 PM, Azoff, Justin S > wrote: > >> >> > On Mar 9, 2017, at 5:11 PM, Matt Clemons >> wrote: >> > >> > I've disabled cron. >> > >> > Still getting "received termination signal." and "child died" in the >> communications.log. >> >> Ah! "child died" makes things interesting. That's literally the only >> thing that can cause bro to say 'received termination signal' for an >> internal reason. I completely forgot about this case :-( >> >> When the child process that handles communication dies, the parent can't >> continue without it so it kills itself so the whole thing can be restarted >> in a known working state. >> >> Is there anything that shows up in your reporter.log or communication.log >> right before this happens? >> >> Is the kernel logging any segfaults to syslog? >> >> -- >> - Justin Azoff >> >> > > > -- > Regards, > > Matt Clemons > (816) 200-0789 > -- Regards, Matt Clemons (816) 200-0789 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170320/9c5fa3e7/attachment-0001.html From jazoff at illinois.edu Mon Mar 20 11:42:26 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 20 Mar 2017 18:42:26 +0000 Subject: [Bro] bro master crashing In-Reply-To: References: <019AE829-183E-46B4-B06D-A618F088F8B6@illinois.edu> Message-ID: > On Mar 20, 2017, at 2:34 PM, Matt Clemons wrote: > > Just wanted to give an update to show how crazy this has been. > > The segfaults made me think "memory issue", so i ran memtest on the system. It has a lot of mems so this took many hours to complete, and with 0 errors. Pulled power on the system and upon boot, everything came up fine with a limited set of workers. I added all 200+ worker processes back in, and now it's running like a champ again. > > The only other thing that it could have been was a power outage on one of the 10 gig worker boxes. It kept blipping and coming back up. Bro cron was starting processes, and then that worker system was crashing due to lack of power. This could have caused the manager to fail. But i can't really tell what the root cause was. > > Thanks for the responses. Ah.. I dropped the ball on this, sorry. That's really interesting that a full restart fixed things. One thing I was thinking could have caused it was a stray/hung bro process somehow still listening on the port, but that usually shows up as a much more explicit issue in the logs. It may be possible to use gdb to see where this is in the bro binary: 2017-03-09T18:34:06.409225+00:00 HOSTNAME kernel: bro[60506]: segfault at 0 ip 00000000005fcf8d sp 00007fffaf9d2f40 error 6 in bro[400000+624000] I'm not sure if the usual method would work, but you can try gdb `which bro` and then at the (gdb) prompt, see if info symbol 0x00000000005fcf8d info symbol 0x00007fffaf9d2f40 show anything useful. There may be a more correct command to get gdb to tell you where in the bro binary the segfault occurred. -- - Justin Azoff From brot212 at googlemail.com Mon Mar 20 12:12:04 2017 From: brot212 at googlemail.com (D. W.) Date: Mon, 20 Mar 2017 20:12:04 +0100 Subject: [Bro] BinPAC &check Message-ID: <8f247c70-695a-7921-1649-6fed31801959@googlemail.com> Hi there, can someone explain me how the &check attribute in binpac is suppose to work, or does it even work at all? I checked my protocol analyzer after the make process and I couldn't find any impact in the code, like other attributes does (like byteorder etc... ) Thanks From seth at corelight.com Mon Mar 20 12:16:45 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 20 Mar 2017 15:16:45 -0400 Subject: [Bro] PacketFilter In-Reply-To: <71F603E2-FA7D-430F-ADE3-0045033F7359@pingtrip.com> References: <75914B54-6E8A-475C-955F-4FA79E31BBBD@pingtrip.com> <09f4ba432bd5b9d89a007858e30563b5@localhost> <2BE918E4-4A5D-4C0E-B55A-0745674158BC@pingtrip.com> <6ED11E34-3F30-4C36-AE75-115C8FAC63B9@pingtrip.com> <0A3BFD72-3C7C-4C1C-99CF-7617C2823C2A@pingtrip.com> <902A835A-1ACC-49C0-801E-CBDF0C56EEBB@illinois.edu> <71F603E2-FA7D-430F-ADE3-0045033F7359@pingtrip.com> Message-ID: <27F9CACC-BA2C-4DA3-B798-8B41272D78F4@corelight.com> > On Mar 19, 2017, at 7:36 PM, Dave Crawford wrote: > > I built a new Bro cluster without Netmap (standard libpcap-dev libraries for Debian 8.7) and the BPF works as expected: Could you try using the netmap plugin for Bro instead of the modified libpcap? The filtering should work correctly there. .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From bro at pingtrip.com Mon Mar 20 14:27:39 2017 From: bro at pingtrip.com (Dave Crawford) Date: Mon, 20 Mar 2017 17:27:39 -0400 Subject: [Bro] PacketFilter In-Reply-To: <27F9CACC-BA2C-4DA3-B798-8B41272D78F4@corelight.com> References: <75914B54-6E8A-475C-955F-4FA79E31BBBD@pingtrip.com> <09f4ba432bd5b9d89a007858e30563b5@localhost> <2BE918E4-4A5D-4C0E-B55A-0745674158BC@pingtrip.com> <6ED11E34-3F30-4C36-AE75-115C8FAC63B9@pingtrip.com> <0A3BFD72-3C7C-4C1C-99CF-7617C2823C2A@pingtrip.com> <902A835A-1ACC-49C0-801E-CBDF0C56EEBB@illinois.edu> <71F603E2-FA7D-430F-ADE3-0045033F7359@pingtrip.com> <27F9CACC-BA2C-4DA3-B798-8B41272D78F4@corelight.com> Message-ID: <4C44A328-AB4E-4EE5-A748-2AE3A8239538@pingtrip.com> Sure, I?ll uninstall netmap-libpcap, install the standard Debian libpcap-dev and recompile Bro. Will respond back with observations. > On Mar 20, 2017, at 3:16 PM, Seth Hall wrote: > > >> On Mar 19, 2017, at 7:36 PM, Dave Crawford wrote: >> >> I built a new Bro cluster without Netmap (standard libpcap-dev libraries for Debian 8.7) and the BPF works as expected: > > Could you try using the netmap plugin for Bro instead of the modified libpcap? The filtering should work correctly there. > > .Seth > > -- > Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com > From seth at corelight.com Mon Mar 20 19:45:27 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 20 Mar 2017 22:45:27 -0400 Subject: [Bro] PacketFilter In-Reply-To: <4C44A328-AB4E-4EE5-A748-2AE3A8239538@pingtrip.com> References: <75914B54-6E8A-475C-955F-4FA79E31BBBD@pingtrip.com> <09f4ba432bd5b9d89a007858e30563b5@localhost> <2BE918E4-4A5D-4C0E-B55A-0745674158BC@pingtrip.com> <6ED11E34-3F30-4C36-AE75-115C8FAC63B9@pingtrip.com> <0A3BFD72-3C7C-4C1C-99CF-7617C2823C2A@pingtrip.com> <902A835A-1ACC-49C0-801E-CBDF0C56EEBB@illinois.edu> <71F603E2-FA7D-430F-ADE3-0045033F7359@pingtrip.com> <27F9CACC-BA2C-4DA3-B798-8B41272D78F4@corelight.com> <4C44A328-AB4E-4EE5-A748-2AE3A8239538@pingtrip.com> Message-ID: > On Mar 20, 2017, at 5:27 PM, Dave Crawford wrote: > > Sure, I?ll uninstall netmap-libpcap, install the standard Debian libpcap-dev and recompile Bro. Will respond back with observations. You don?t need to do that if you don?t want to. Just compile and install the netmap plugin that ships with Bro 2.5. Check out the README that comes with it too because it explains how to configure a cluster with the netmap plugin. .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From darkheaven1983 at gmail.com Tue Mar 21 05:05:15 2017 From: darkheaven1983 at gmail.com (duhang) Date: Tue, 21 Mar 2017 20:05:15 +0800 Subject: [Bro] Significant slow for smtp traffic Message-ID: Hello, I am trying to use bro to monitor smtp requests in my network. Before putting it to production, I simulated the smtp traffic between clients and smtp server using avalanche as the rate of 100 emails/second to test the performance of bro. The size of the attachment is random between a few KBs to 8MB. I was running bro cluster using pf_ring as load balance and launching 20 workers pinned on different CPU. The average network bandwidth is about 200M - 300M. I observed significant slow to get smtp requests showing in the log. The CPU usage is pretty high(100% for every cpu I pinned) and is busy doing memcpy in BroString.cc:concatenate. After a few minutes, I can see a significant drop in the statistic of pf_ring. Is there any suggestion how can I cope with this traffic? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170321/46127950/attachment.html From kingsleyluoxin at hotmail.com Tue Mar 21 18:28:42 2017 From: kingsleyluoxin at hotmail.com (Luo Xin) Date: Wed, 22 Mar 2017 01:28:42 +0000 Subject: [Bro] How bro create an event from a packet? Message-ID: My puzzles mainly exist in the states managing of bro. I have noticed that there are C++ code for the implemention of DFA and NFA. Nevertheless, I could not find where it is invoked. So I was wondering if you could tell me where I can find the use of state machine. In addition, I also want to know about how bro transfer low level pcap file into high level event. I have read some information about that of protocol based on TCP or UDP and been aware that they are implemented by means of binpac tool. But I still want to know how lower level protocol such as IP or TCP can transfer pcap packets into bro events. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170322/6e200494/attachment.html From kingsleyluoxin at hotmail.com Tue Mar 21 18:35:50 2017 From: kingsleyluoxin at hotmail.com (Luo Xin) Date: Wed, 22 Mar 2017 01:35:50 +0000 Subject: [Bro] How bro create an event from a packet? Message-ID: My puzzles mainly exist in the states managing of bro. I have noticed that there are C++ code for the implemention of DFA and NFA. Nevertheless, I could not find where it is invoked. So I was wondering if anyone could tell me where I can find the use of state machine. In addition, I also want to know about how bro transfer low level pcap file into high level event. I have read some information about that of protocol based on TCP or UDP and been aware that they are implemented by means of binpac tool. But I still want to know how lower level protocol such as IP or TCP can transfer pcap packets into bro events. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170322/f1d29849/attachment.html From darkheaven1983 at gmail.com Wed Mar 22 04:48:28 2017 From: darkheaven1983 at gmail.com (duhang) Date: Wed, 22 Mar 2017 19:48:28 +0800 Subject: [Bro] Significant slow for smtp traffic In-Reply-To: References: Message-ID: Just find out that it is event smtp_data which causes the slowness. What's the suggested event to capture smtp body and save it as eml file? 2017-03-21 20:05 GMT+08:00 duhang : > Hello, > > I am trying to use bro to monitor smtp requests in my network. Before > putting it to production, I simulated the smtp traffic between clients and > smtp server using avalanche as the rate of 100 emails/second to test the > performance of bro. The size of the attachment is random between a few KBs > to 8MB. I was running bro cluster using pf_ring as load balance and > launching 20 workers pinned on different CPU. The average network bandwidth > is about 200M - 300M. I observed significant slow to get smtp requests > showing in the log. The CPU usage is pretty high(100% for every cpu I > pinned) and is busy doing memcpy in BroString.cc:concatenate. After a few > minutes, I can see a significant drop in the statistic of pf_ring. > > Is there any suggestion how can I cope with this traffic? > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170322/9961f091/attachment.html From seth at corelight.com Wed Mar 22 06:28:51 2017 From: seth at corelight.com (Seth Hall) Date: Wed, 22 Mar 2017 09:28:51 -0400 Subject: [Bro] Significant slow for smtp traffic In-Reply-To: References: Message-ID: <47A8D4DF-5546-4FCE-8017-9479D12C4563@corelight.com> > On Mar 22, 2017, at 7:48 AM, duhang wrote: > > Just find out that it is event smtp_data which causes the slowness. What's the suggested event to capture smtp body and save it as eml file? Using the file analysis framework is the best way. Are you just trying to save the box from any body transferred over SMTP or is there some particular things you?re looking for? Regardless, the event you?ll want to use is probably file_sniff. Something like this... event file_sniff(f: fa_file, meta: fa_metadata) { if ( f$source == ?SMTP" ) { Files::add_analyzer(f, Files::ANALYZER_EXTRACT); } } .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From fatema.bannatwala at gmail.com Wed Mar 22 08:05:24 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Wed, 22 Mar 2017 11:05:24 -0400 Subject: [Bro] Manager swapping.. Message-ID: Hey all, We have logger and manager running on the same node, and it started to use complete swap and bro logs in current dir stopped rotating. We have run in this type of issue before when running Bro2.4, and it turned out that moving proxies to the worker nodes solved the high load issue on manager, and things started working normally. Now, we have all the proxies on the worker nodes (4 in total) and logger is running on the same node as manager, so my guess would be, that might be causing the high load on manager. The bro processes are really big on the manager: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 104772 bro 20 0 24.926g 0.017t 1300 S 45.7 25.0 4542:04 bro 125346 bro 20 0 0.221t 0.027t 3444 S 40.4 39.4 187:28.80 bro 125366 bro 25 5 1510856 275516 728 R 40.1 0.4 222:22.58 bro 104776 bro 25 5 540736 228920 360 S 8.9 0.3 893:42.05 bro Also, the free -g output looks like this: $ free -g total used free shared buff/cache available Mem: 70 47 0 0 22 21 Swap: 7 7 0 Next thing I am going to try is to disable some of the protocols from logging (don't know how much help it would be) and restart Bro. Any other suggestions/Best practices to follow, to avoid this situation in future (really not looking forward to the quick and dirty fix of restarting Bro whenever this happens :) )? Also, I have proper ethtool settings (tso off gso off gro off rx off tx off sg off) on the manager as well (as suggested in some of the posts for better performance). Thanks, Fatema. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170322/e9a8d27f/attachment-0001.html From fatema.bannatwala at gmail.com Wed Mar 22 08:24:19 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Wed, 22 Mar 2017 11:24:19 -0400 Subject: [Bro] Manager swapping.. In-Reply-To: References: Message-ID: Was just brainstorming, and thinking if multi-threading can be used for logger as well, just like worker threads? As a single Bro logger process is becoming big, why not to distribute the work load across multiple logger processes. Is it possible to do? and if it impacts manager on the same node? Anybody tried that? On Wed, Mar 22, 2017 at 11:05 AM, fatema bannatwala < fatema.bannatwala at gmail.com> wrote: > Hey all, > > We have logger and manager running on the same node, and it started to use > complete swap and bro logs in current dir stopped rotating. > > We have run in this type of issue before when running Bro2.4, and it > turned out that moving proxies to the worker nodes solved the high load > issue on manager, and things started working normally. > > Now, we have all the proxies on the worker nodes (4 in total) and logger > is running on the same node as manager, so my guess would be, that might be > causing the high load on manager. > > The bro processes are really big on the manager: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 104772 bro 20 0 24.926g 0.017t 1300 S 45.7 25.0 4542:04 bro > 125346 bro 20 0 0.221t 0.027t 3444 S 40.4 39.4 187:28.80 bro > 125366 bro 25 5 1510856 275516 728 R 40.1 0.4 222:22.58 bro > 104776 bro 25 5 540736 228920 360 S 8.9 0.3 893:42.05 bro > > Also, the free -g output looks like this: > $ free -g > total used free shared buff/cache > available > Mem: 70 47 0 0 22 > 21 > Swap: 7 7 0 > > Next thing I am going to try is to disable some of the protocols from > logging (don't know how much help it would be) and restart Bro. > > Any other suggestions/Best practices to follow, to avoid this situation in > future (really not looking forward to the quick and dirty fix of restarting > Bro whenever this happens :) )? > > Also, I have proper ethtool settings (tso off gso off gro off rx off tx > off sg off) on the manager as well (as suggested in some of the posts for > better performance). > > Thanks, > Fatema. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170322/19f43239/attachment.html From robin at icir.org Wed Mar 22 10:53:51 2017 From: robin at icir.org (Robin Sommer) Date: Wed, 22 Mar 2017 10:53:51 -0700 Subject: [Bro] does Spicy have a BNF? In-Reply-To: <6e3887ca-9bcc-1247-81a0-a3d64e0bba2a@maine.edu> References: <6e3887ca-9bcc-1247-81a0-a3d64e0bba2a@maine.edu> Message-ID: <20170322175351.GH42999@icir.org> On Sun, Mar 19, 2017 at 17:46 -0400, Troy Jordan wrote: > Does Spicy/BinPAC++ have a BNF? Not quite sure what you're asking for. If you're looking for a grammar for the Spicy language itself, that's here: https://github.com/rsmmr/hilti/blob/master/spicy/parser/parser.yy Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From josh.guild at morphick.com Wed Mar 22 11:30:11 2017 From: josh.guild at morphick.com (Josh Guild) Date: Wed, 22 Mar 2017 14:30:11 -0400 Subject: [Bro] Blank HTTP logs Message-ID: Howdy all, I've been running into an issue with the http.log not populating fields (method, host, uri, referrer, UA) when spanned. I'm still getting the status_code and status_msg populated in the http.log and I've read an ancient article where Seth says this may be because of TCP checksum offloadin. ( https://groups.google.com/forum/#!topic/security-onion/12jqLwMShUo). We currently have rx/tx-checksumming disabled on the ports we're monitoring but rx/tx-vlan-offload is enabled, could this be the culprit? The largest entries in the weird.log are windo_recision, data_before_established, and possible_split_routing. Any help would be much appreciated! -- Josh Guild Network Intelligence Analyst -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170322/f1bc04fc/attachment.html From seth at corelight.com Wed Mar 22 12:07:45 2017 From: seth at corelight.com (Seth Hall) Date: Wed, 22 Mar 2017 19:07:45 +0000 Subject: [Bro] Blank HTTP logs In-Reply-To: References: Message-ID: I suspect that your span port is only capturing one direction of the traffic. All of the fields that you said are missing are from the client Check your conn log to see if you're seeing orig_pkts or resp_pkts frequently set to zero. .Seth On Wed, Mar 22, 2017 at 2:32 PM Josh Guild wrote: > Howdy all, > > I've been running into an issue with the http.log not populating fields > (method, host, uri, referrer, UA) when spanned. I'm still getting the > status_code and status_msg populated in the http.log and I've read an > ancient article where Seth says this may be because of TCP checksum > offloadin. ( > https://groups.google.com/forum/#!topic/security-onion/12jqLwMShUo). > > We currently have rx/tx-checksumming disabled on the ports we're > monitoring but rx/tx-vlan-offload is enabled, could this be the culprit? > > The largest entries in the weird.log are windo_recision, > data_before_established, and possible_split_routing. > > Any help would be much appreciated! > > -- > Josh Guild > Network Intelligence Analyst > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170322/2cbf5709/attachment.html From josh.guild at morphick.com Wed Mar 22 12:35:07 2017 From: josh.guild at morphick.com (Josh Guild) Date: Wed, 22 Mar 2017 15:35:07 -0400 Subject: [Bro] Blank HTTP logs In-Reply-To: References: Message-ID: Cool. that's what I was thinking as well since we're only seeing resp or orig in the history of the conn.log as well. I'm thinking they spanned and have RX on one port with TX on the other. Thanks for the help! On Wed, Mar 22, 2017 at 3:07 PM, Seth Hall wrote: > I suspect that your span port is only capturing one direction of the > traffic. All of the fields that you said are missing are from the client > > Check your conn log to see if you're seeing orig_pkts or resp_pkts > frequently set to zero. > > .Seth > > On Wed, Mar 22, 2017 at 2:32 PM Josh Guild > wrote: > >> Howdy all, >> >> I've been running into an issue with the http.log not populating fields >> (method, host, uri, referrer, UA) when spanned. I'm still getting the >> status_code and status_msg populated in the http.log and I've read an >> ancient article where Seth says this may be because of TCP checksum >> offloadin. (https://groups.google.com/forum/#!topic/security-onion/ >> 12jqLwMShUo). >> >> We currently have rx/tx-checksumming disabled on the ports we're >> monitoring but rx/tx-vlan-offload is enabled, could this be the culprit? >> >> The largest entries in the weird.log are windo_recision, >> data_before_established, and possible_split_routing. >> >> Any help would be much appreciated! >> >> -- >> Josh Guild >> Network Intelligence Analyst >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- Josh Guild Network Intelligence Analyst -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170322/412fa63c/attachment.html From jazoff at illinois.edu Wed Mar 22 16:41:18 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 22 Mar 2017 23:41:18 +0000 Subject: [Bro] Manager swapping.. In-Reply-To: References: Message-ID: <9D6F6FFF-A711-43BC-8FEA-E436F507D6E4@illinois.edu> > On Mar 22, 2017, at 11:05 AM, fatema bannatwala wrote: > > The bro processes are really big on the manager: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 104772 bro 20 0 24.926g 0.017t 1300 S 45.7 25.0 4542:04 bro > 125346 bro 20 0 0.221t 0.027t 3444 S 40.4 39.4 187:28.80 bro > 125366 bro 25 5 1510856 275516 728 R 40.1 0.4 222:22.58 bro > 104776 bro 25 5 540736 228920 360 S 8.9 0.3 893:42.05 bro Which process is which in this output? Can you use broctl top manager logger instead? that will include the output about which process is which along with the cpu/mem usage. How to troubleshoot the issue depends a lot on if it is the manage process or the logger process causing the problems. > Also, the free -g output looks like this: > $ free -g > total used free shared buff/cache available > Mem: 70 47 0 0 22 21 > Swap: 7 7 0 Looks like you have some headroom there, but not much. > Next thing I am going to try is to disable some of the protocols from logging (don't know how much help it would be) and restart Bro. Well, if it's the logger node, reducing the log volume can help. Depending on how many cpu cores you have one thing that can help is using logging filters to split logs out into multiple files. That lets the logger node dedicate more threads to writing logs. > Any other suggestions/Best practices to follow, to avoid this situation in future (really not looking forward to the quick and dirty fix of restarting Bro whenever this happens :) )? > > Also, I have proper ethtool settings (tso off gso off gro off rx off tx off sg off) on the manager as well (as suggested in some of the posts for better performance). That shouldn't really matter on the manager, but it can't hurt. > Was just brainstorming, and thinking if multi-threading can be used for logger as well, just like worker threads? > As a single Bro logger process is becoming big, why not to distribute the work load across multiple logger processes. > Is it possible to do? and if it impacts manager on the same node? > Anybody tried that? The logger does distribute the work across multiple threads, but it has a central component that has to receive all the messages. Someone else on the mailing list was having issues with logger scaling and I pointed them to the parts of broctl that needed to be tweaked to let you run multiple logger nodes. If you're currently using something like the kafka log writer or something like logstash to ship bro logs off to another system it will kind of work. The 'issue' is that you end up with 2 log directories that each contain the logs from half the workers. As long as you have something else that can merge them back together and correlate everything that's not a problem. Hopefully multiple logger nodes can be supported officially at some point. -- - Justin Azoff From jazoff at illinois.edu Wed Mar 22 16:51:50 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 22 Mar 2017 23:51:50 +0000 Subject: [Bro] Manager swapping.. In-Reply-To: <9D6F6FFF-A711-43BC-8FEA-E436F507D6E4@illinois.edu> References: <9D6F6FFF-A711-43BC-8FEA-E436F507D6E4@illinois.edu> Message-ID: > On Mar 22, 2017, at 7:41 PM, Azoff, Justin S wrote: > Hopefully multiple logger nodes can be supported officially at some point. And right after I send this I see that Daniel has a branch of broctl with the initial changes needed to make this work. -- - Justin Azoff From zeolla at gmail.com Thu Mar 23 05:54:10 2017 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Thu, 23 Mar 2017 12:54:10 +0000 Subject: [Bro] Apache struts exploit detection In-Reply-To: <20170314175513.l52k4yslu7lazir5@Beezling.local> References: <20170314175513.l52k4yslu7lazir5@Beezling.local> Message-ID: I just wanted to add onto this thread and mention that it appears there is a new way to exploit CVE-2017-5638 which neither of the prior scripts are currently looking for. I've opened #3 on Aashish's repo to provide more details. Jon On Tue, Mar 14, 2017 at 1:55 PM Johanna Amann wrote: And note that Aashish made it so you can install it using bro-pkg :) Johanna On Tue, Mar 14, 2017 at 12:09:52PM -0400, Drew Dixon wrote: > Deploy this outstanding bro detection script for this vulnerability: > > https://github.com/initconf/CVE-2017-5638_struts.git > > On Tue, Mar 14, 2017 at 6:08 AM, Zeolla at GMail.com wrote: > > > Here's an example script that will detect CVE-2017-5638 exploit attempts > > and log the contents of the header. > > > > https://github.com/set-element/misc-scripts/blob/ > > master/CVE-2017-5638_struts.bro > > > > For future reference the key component is: > > > > event http_header(c: connection, is_orig: bool, name: string, value: > > string) &priority=5 > > > > { > > > > # look if the connection is from offsite and the value is content-type > > > > if ( !Site::is_local_addr(c$id$orig_h) && name == "CONTENT-TYPE" > > && detection_string in value ) > > > > { > > > > NOTICE([$note=HTTP_StrutsAttack, $src=c$id$orig_h, > > $msg=fmt("CVE-2017-5638/Struts attack from %s seen: %s", c$id$orig_h, > > value)]); > > > > } > > > > } > > > > Please note that this is not my script, it is set-element's. Depending on > > the situation you may want to check the src/dst to add exemptions > > (vulnerability scanning boxes?), ignore or specifically monitor > > Site::is_private_addr src/dsts, add $identifier/$suppress_for to the > > NOTICE, replace $src=... with $conn=c to get more details in the notice > > log, etc. All depends on what you want, those are just things I would do. > > > > Jon > > > > On Tue, Mar 14, 2017, 3:04 AM John Edwards wrote: > > > >> Hi all > >> > >> For the likes of the apache struts web application attack that the actual > >> exploit is contained within a web http GET request. Or let's say any web > >> app attack that is embedded within the referer field like embedded > >> JavaScript can bro actually view or log that level of info? > >> > >> I can see bro will see things like http user agent fields and get or post > >> request but for the actual malicious code embedded further in the request > >> I'm assuming isn't captured? > >> > >> My ips obviously captures that alert data and I can see the the exploit > >> but the bro data from the http log I'll only see "GET / HTTP1.1" and that's > >> all > >> > >> Cheers > >> John > >> _______________________________________________ > >> Bro mailing list > >> bro at bro-ids.org > >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > -- > > > > Jon > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170323/f3e92977/attachment-0001.html From fatema.bannatwala at gmail.com Thu Mar 23 06:40:07 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Thu, 23 Mar 2017 09:40:07 -0400 Subject: [Bro] Manager swapping.. In-Reply-To: References: <9D6F6FFF-A711-43BC-8FEA-E436F507D6E4@illinois.edu> Message-ID: Thanks Justin for the input :) I restarted Bro after disabling some of the protocols logging (like rdp, syslog, snmp etc) yesterday afternoon, as the machine is in production and needed to be fixed kind of "ASAP". Hence couldn't get a chance to run the broctl top while having the issue, I know you have mentioned it couple of times in past to use "broctl top" instead of normal "top", but magically I keep forgetting to do that, I think I should come up with by BRO troubleshoot guide, which should list some basic troubleshooting commands that you guys suggest in these emails :) Anyways, I did run the command today, and it looks like the manager process is overwhelmed, hmm I thought that it might logger that might be having issues catching up on the load, but I was wrong: $ sudo -u bro /usr/local/bro/2.5/bin/broctl top manager logger Name Type Host Pid Proc VSize Rss Cpu Cmd logger logger IDS 60928 parent 2G 90M 17% bro logger logger IDS 60932 child 522M 246M 5% bro manager manager IDS 60990 child 1G 257M 35% bro *manager manager IDS 60973 parent 222G 31G 23% bro* It makes me think, if there is some memory leak issue with manager. *Thanks,* *Fatema.* On Wed, Mar 22, 2017 at 7:51 PM, Azoff, Justin S wrote: > > > On Mar 22, 2017, at 7:41 PM, Azoff, Justin S > wrote: > > Hopefully multiple logger nodes can be supported officially at some > point. > > And right after I send this I see that Daniel has a branch of broctl with > the initial changes needed to make this work. > > > > > -- > - Justin Azoff > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170323/4c826053/attachment.html From jazoff at illinois.edu Thu Mar 23 07:43:39 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 23 Mar 2017 14:43:39 +0000 Subject: [Bro] Manager swapping.. In-Reply-To: References: <9D6F6FFF-A711-43BC-8FEA-E436F507D6E4@illinois.edu> Message-ID: <8A24D1AD-53B3-46D5-B6D1-A8D3F80FC8B8@illinois.edu> > On Mar 23, 2017, at 7:40 AM, fatema bannatwala wrote: > > Thanks Justin for the input :) > > I restarted Bro after disabling some of the protocols logging (like rdp, syslog, snmp etc) yesterday afternoon, > as the machine is in production and needed to be fixed kind of "ASAP". Hence couldn't get a chance to run > the broctl top while having the issue, I know you have mentioned it couple of times in past to use "broctl top" > instead of normal "top", but magically I keep forgetting to do that, I think I should come up with by BRO troubleshoot > guide, which should list some basic troubleshooting commands that you guys suggest in these emails :) > > Anyways, I did run the command today, and it looks like the manager process is overwhelmed, > hmm I thought that it might logger that might be having issues catching up on the load, but I was wrong: > > $ sudo -u bro /usr/local/bro/2.5/bin/broctl top manager logger > Name Type Host Pid Proc VSize Rss Cpu Cmd > logger logger IDS 60928 parent 2G 90M 17% bro > logger logger IDS 60932 child 522M 246M 5% bro > manager manager IDS 60990 child 1G 257M 35% bro > manager manager IDS 60973 parent 222G 31G 23% bro > > It makes me think, if there is some memory leak issue with manager. Are you loading misc/detect-traceroute or misc/scan in your local.bro? -- - Justin Azoff From daniel.manzo at bayer.com Thu Mar 23 07:52:46 2017 From: daniel.manzo at bayer.com (Daniel Manzo) Date: Thu, 23 Mar 2017 14:52:46 +0000 Subject: [Bro] No email notices after updating to 2.5 Message-ID: <7b09877dc0114fc0ac533ac13808c940@moxde9.na.bayer.cnb> Hi all, After installing Bro 2.5 via rpm on RHEL 7.3 (and running bro with broctl), I no longer receive "Dropped Packets" and "Invalid_Server_Cert" email notices that I would receive almost daily when running Bro 2.4. I still receive connection summaries every hour, which is the same as 2.4. While looking into this problem, I noticed that Bro is no longer generating the notice.log or reporter.log nearly as often as it was before. I understand that it could be possible that these problems are no longer being triggered, but I find it very hard to believe that there are no dropped packets or invalid server certs anymore. A custom script that sends an email notice when Bro is started and when Bro is stopped works fine, so I'm not sure why the other alerts wouldn't be working .Any and all help is appreciated. Best regards, Dan Manzo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170323/f30934b3/attachment.html From fatema.bannatwala at gmail.com Thu Mar 23 07:56:25 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Thu, 23 Mar 2017 10:56:25 -0400 Subject: [Bro] Manager swapping.. In-Reply-To: <8A24D1AD-53B3-46D5-B6D1-A8D3F80FC8B8@illinois.edu> References: <9D6F6FFF-A711-43BC-8FEA-E436F507D6E4@illinois.edu> <8A24D1AD-53B3-46D5-B6D1-A8D3F80FC8B8@illinois.edu> Message-ID: Nope, based on our previous discussion in another thread, I disabled the misc/scan, and loaded scan-NG-master script. I always thought that the scripts would have more load on workers than manager. When I was seeing memory issues on workers, I stopped using misc/scan and switched to the scan-NG script. Didn't know that it would impact manager performance as well, hmm. On Thu, Mar 23, 2017 at 10:43 AM, Azoff, Justin S wrote: > > > On Mar 23, 2017, at 7:40 AM, fatema bannatwala < > fatema.bannatwala at gmail.com> wrote: > > > > Thanks Justin for the input :) > > > > I restarted Bro after disabling some of the protocols logging (like rdp, > syslog, snmp etc) yesterday afternoon, > > as the machine is in production and needed to be fixed kind of "ASAP". > Hence couldn't get a chance to run > > the broctl top while having the issue, I know you have mentioned it > couple of times in past to use "broctl top" > > instead of normal "top", but magically I keep forgetting to do that, I > think I should come up with by BRO troubleshoot > > guide, which should list some basic troubleshooting commands that you > guys suggest in these emails :) > > > > Anyways, I did run the command today, and it looks like the manager > process is overwhelmed, > > hmm I thought that it might logger that might be having issues catching > up on the load, but I was wrong: > > > > $ sudo -u bro /usr/local/bro/2.5/bin/broctl top manager logger > > Name Type Host Pid Proc VSize Rss Cpu Cmd > > logger logger IDS 60928 parent 2G 90M 17% bro > > logger logger IDS 60932 child 522M 246M 5% bro > > manager manager IDS 60990 child 1G 257M 35% bro > > manager manager IDS 60973 parent 222G 31G 23% bro > > > > It makes me think, if there is some memory leak issue with manager. > > Are you loading misc/detect-traceroute or misc/scan in your local.bro? > > -- > - Justin Azoff > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170323/97f93b4a/attachment.html From hovsep.sanjay.levi at gmail.com Thu Mar 23 08:40:32 2017 From: hovsep.sanjay.levi at gmail.com (Hovsep Levi) Date: Thu, 23 Mar 2017 15:40:32 +0000 Subject: [Bro] Manager swapping.. In-Reply-To: References: <9D6F6FFF-A711-43BC-8FEA-E436F507D6E4@illinois.edu> <8A24D1AD-53B3-46D5-B6D1-A8D3F80FC8B8@illinois.edu> Message-ID: Try disabling the SSL/TLS cert verification. I'm not sure why but that helped, without it the manager would slowly climb to massive memory usage. Now it works fine for one or two weeks before unexpectedly using all memory. #@load protocols/ssl/validate-certs Good: Name Type Host Pid Proc VSize Rss Cpu Cmd logger-1 logger 10.1.1.1 6241 parent 701M 163M 20% bro logger-1 logger 10.1.1.1 6261 child 458M 69M 3% bro manager manager 10.1.1.1 6345 child 510M 377M 100% bro manager manager 10.1.1.1 6292 parent 890M 804M 24% bro Bad: Name Type Host Pid Proc VSize Rss Cpu Cmd logger-1 logger 10.1.1.1 52731 parent 1G 806M 0% bro logger-1 logger 10.1.1.1 52951 child 8G 8G 0% bro manager manager 10.1.1.1 53127 child 1G 742M 0% bro manager manager 10.1.1.1 52979 parent 1573G 100G 0% bro From fatema.bannatwala at gmail.com Thu Mar 23 10:24:58 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Thu, 23 Mar 2017 13:24:58 -0400 Subject: [Bro] Manager swapping.. In-Reply-To: References: <9D6F6FFF-A711-43BC-8FEA-E436F507D6E4@illinois.edu> <8A24D1AD-53B3-46D5-B6D1-A8D3F80FC8B8@illinois.edu> Message-ID: Thanks Sanjay for suggestions.I already have the @load protocols/ssl/validate-certs disabled in local.bro. :) I was looking into the reporter logs and see some logs like this: Some INFO logs: 1490288453.884071 Reporter::INFO Got counters: [new_conn_counter=4394103, is_catch_release_active=7433937, known_scanners_counter=0, not_scanner=2439888, darknet_counter=64358, not_darknet_counter=3114626, already_scanner_counter=0, filteration_entry=0, filteration_success=1543038, c_knock_filterate=3548445, c_knock_checkscan=0, c_knock_core=0, c_land_filterate=22317, c_land_checkscan=0, c_land_core=0, c_backscat_filterate=3548445, c_backscat_checkscan=0, c_backscat_core=0, c_addressscan_filterate=3548445, c_addressscan_checkscan=0, c_addressscan_core=0, check_scan_counter=0, worker_to_manager_counter=0, run_scan_detection=0, check_scan_cache=1543038, event_peer=worker-1-15] manager 1490288454.925040 Reporter::INFO known_scanners_inactive: [scanner=94.51.38.120, status=T, detection=KnockKnockScan, detect_ts=1490202054.11266, event_peer=manager, expire=F] manager 1490288454.925040 Reporter::INFO known_scanners_inactive: [scanner=171.249.5.188, status=T, detection=KnockKnockScan, detect_ts=1490202053.07045, event_peer=manager, expire=F] manager Ans these error logs: 0.000000 Reporter::ERROR field value missing [Scan::geoip_info$country_code] /usr/local/bro/2.5/share/bro/site/scan-NG-master/scripts/./scan-summary.bro, line 292 0.000000 Reporter::ERROR value used but not set (Scan::c_landmine_scan_summary) /usr/local/bro/2.5/share/bro/site/scan-NG-master/scripts/./check-landmine.bro, line 33 0.000000 Reporter::ERROR value used but not set (Scan::c_landmine_scan_summary) /usr/local/bro/2.5/share/bro/site/scan-NG-master/scripts/./check-landmine.bro, line 33 Are they anywhere related to the issue? Thanks, Fatema. On Thu, Mar 23, 2017 at 10:56 AM, fatema bannatwala < fatema.bannatwala at gmail.com> wrote: > Nope, based on our previous discussion in another thread, > I disabled the misc/scan, and loaded scan-NG-master script. > I always thought that the scripts would have more load on workers than > manager. > When I was seeing memory issues on workers, I stopped using misc/scan and > switched to > the scan-NG script. > Didn't know that it would impact manager performance as well, hmm. > > On Thu, Mar 23, 2017 at 10:43 AM, Azoff, Justin S > wrote: > >> >> > On Mar 23, 2017, at 7:40 AM, fatema bannatwala < >> fatema.bannatwala at gmail.com> wrote: >> > >> > Thanks Justin for the input :) >> > >> > I restarted Bro after disabling some of the protocols logging (like >> rdp, syslog, snmp etc) yesterday afternoon, >> > as the machine is in production and needed to be fixed kind of "ASAP". >> Hence couldn't get a chance to run >> > the broctl top while having the issue, I know you have mentioned it >> couple of times in past to use "broctl top" >> > instead of normal "top", but magically I keep forgetting to do that, I >> think I should come up with by BRO troubleshoot >> > guide, which should list some basic troubleshooting commands that you >> guys suggest in these emails :) >> > >> > Anyways, I did run the command today, and it looks like the manager >> process is overwhelmed, >> > hmm I thought that it might logger that might be having issues catching >> up on the load, but I was wrong: >> > >> > $ sudo -u bro /usr/local/bro/2.5/bin/broctl top manager logger >> > Name Type Host Pid Proc VSize Rss Cpu Cmd >> > logger logger IDS 60928 parent 2G 90M 17% bro >> > logger logger IDS 60932 child 522M 246M 5% bro >> > manager manager IDS 60990 child 1G 257M 35% bro >> > manager manager IDS 60973 parent 222G 31G 23% bro >> > >> > It makes me think, if there is some memory leak issue with manager. >> >> Are you loading misc/detect-traceroute or misc/scan in your local.bro? >> >> -- >> - Justin Azoff >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170323/5c5f8f0f/attachment.html From briford.wylie at gmail.com Thu Mar 23 12:40:35 2017 From: briford.wylie at gmail.com (Brian Wylie) Date: Thu, 23 Mar 2017 13:40:35 -0600 Subject: [Bro] Getting 'standard' Bro events into Python Message-ID: Hi All, I'm fairly new to Bro and I have a question very similar to this one ' http://mailman.icsi.berkeley.edu/pipermail/bro/2017-January/011389.html'. Basically I want the easiest/best path to get standard Bro events (conn, http, dns, ssl, weird..etc) into Python. 1) Is broctl / python-broccoli the best path? - Note: in my testing I had to use broctl> start . in order for my python Connection() to work - If this isn't necessary and I can do the same with just running Bro standalone pls let me know 2) If broctl/python-broccoli IS the best path then how do I 'subscribe' to the standard events? - Is there a list of the standard events? - If so do I just @event with a method that has the same name as the event? Sorry if these are naive questions, but so far my googling/trying/testing has been a bit hit-miss :) Cheers, -Brian Wylie -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170323/ba6f75dd/attachment.html From jackhcovington at gmail.com Thu Mar 23 13:56:49 2017 From: jackhcovington at gmail.com (Jack Covington) Date: Thu, 23 Mar 2017 15:56:49 -0500 Subject: [Bro] Disable broctl port-terminate script Message-ID: Hello, Is there was a way to disable the bro/share/broctl/scripts/post-terminate script that is called on "broctl stop" from control.py? I have a program that ingests a couple bro logs in spool/logger so I want to keep everything in that directory, but the post-terminate script moves them out to spool/tmp/post-terminate* directories. So far I just added an exit at the beginning of the script since I am not concerned with any archiving, but is there a more correct way to do it without modifying the bro script, maybe some option I can redefine in my local.bro? I already set LogRotationInterval to 0 in broctl.config. Thank you, Jack -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170323/4a04107d/attachment.html From dhoelzer at enclaveforensics.com Thu Mar 23 13:59:25 2017 From: dhoelzer at enclaveforensics.com (=?UTF-8?Q?David_Hoelzer?=) Date: Thu, 23 Mar 2017 20:59:25 +0000 Subject: [Bro] Disable broctl port-terminate script In-Reply-To: References: Message-ID: <0100015afcf78af4-9421f625-6559-4af3-96c0-6c97be4d543f-000000@email.amazonses.com> Just modify the python source code; comment it out. > On Mar 23, 2017, at 4:56 PM, Jack Covington wrote: > > Hello, > > Is there was a way to disable the bro/share/broctl/scripts/post-terminate script that is called on "broctl stop" from control.py? > > I have a program that ingests a couple bro logs in spool/logger so I want to keep everything in that directory, but the post-terminate script moves them out to spool/tmp/post-terminate* directories. So far I just added an exit at the beginning of the script since I am not concerned with any archiving, but is there a more correct way to do it without modifying the bro script, maybe some option I can redefine in my local.bro? > > I already set LogRotationInterval to 0 in broctl.config. > > Thank you, > Jack > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From briford.wylie at gmail.com Fri Mar 24 10:54:52 2017 From: briford.wylie at gmail.com (Brian Wylie) Date: Fri, 24 Mar 2017 11:54:52 -0600 Subject: [Bro] Getting 'standard' Bro events into Python In-Reply-To: References: Message-ID: Okay, after a bit more hunting I see the new Broker communications docs. - https://www.bro.org/sphinx/components/broker/README.html - https://www.bro.org/sphinx/components/broker/broker-manual.html I see that you can wrap the broker API with SWIG, so this is all good new. Anyway happen to have/make/point me to a small example python script that maybe subscribes to all connection events (events that go into conn.long)? Thanks a bunch, -Brian Wyli On Thu, Mar 23, 2017 at 1:40 PM, Brian Wylie wrote: > Hi All, > > I'm fairly new to Bro and I have a question very similar to this one ' > http://mailman.icsi.berkeley.edu/pipermail/bro/2017-January/011389.html'. > > Basically I want the easiest/best path to get standard Bro events (conn, > http, dns, ssl, weird..etc) into Python. > > 1) Is broctl / python-broccoli the best path? > - Note: in my testing I had to use broctl> start . in order for my > python Connection() to work > - If this isn't necessary and I can do the same with just running > Bro standalone pls let me know > > 2) If broctl/python-broccoli IS the best path then how do I 'subscribe' to > the standard events? > - Is there a list of the standard events? > - If so do I just @event with a method that has the same name as the > event? > > Sorry if these are naive questions, but so far my googling/trying/testing > has been a bit hit-miss :) > > Cheers, > -Brian Wylie > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170324/a0716493/attachment.html From Hafiz.Ul-Asad.1 at city.ac.uk Sat Mar 25 02:39:28 2017 From: Hafiz.Ul-Asad.1 at city.ac.uk (Ul Asad, Hafiz) Date: Sat, 25 Mar 2017 09:39:28 +0000 Subject: [Bro] multiple tables in SQLite Database Message-ID: Bro Users, I have been trying to have multiple logs in a single sqlite database but I am getting the "the database is locked error". This problem was previously raised here, https://bro-tracker.atlassian.net/browse/BIT-1325?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aworklog-tabpanel. I wonder if there has been any solution for it in the Bro 2.5? Regards Asad -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170325/e0022478/attachment.html From asharma at lbl.gov Sat Mar 25 07:24:55 2017 From: asharma at lbl.gov (Aashish Sharma) Date: Sat, 25 Mar 2017 07:24:55 -0700 Subject: [Bro] multiple tables in SQLite Database In-Reply-To: References: Message-ID: <20170325142454.GH2071@mac-822.local> Asad, You'd need to use postgres instead. SQLite + BRO is good for readonly operations. If you have a lot of reads/writes Postgres works fantastic. It should be fairly straight forward to port your current bro SQLITE policy to use postgres code. I have been using postgres instead as well. Don't use sqlite. Aashish On Sat, Mar 25, 2017 at 09:39:28AM +0000, Ul Asad, Hafiz wrote: > Bro Users, > > I have been trying to have multiple logs in a single sqlite database but I am getting the "the database is locked error". This problem was previously raised here, https://bro-tracker.atlassian.net/browse/BIT-1325?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aworklog-tabpanel. I wonder if there has been any solution for it in the Bro 2.5? > > Regards > Asad > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From Hafiz.Ul-Asad.1 at city.ac.uk Sat Mar 25 07:39:19 2017 From: Hafiz.Ul-Asad.1 at city.ac.uk (Ul Asad, Hafiz) Date: Sat, 25 Mar 2017 14:39:19 +0000 Subject: [Bro] multiple tables in SQLite Database In-Reply-To: <20170325142454.GH2071@mac-822.local> References: <20170325142454.GH2071@mac-822.local> Message-ID: Thanks Aashish, So you mean the following script, event bro_init() { local filter: Log::Filter = [ $name="sqlite", $path="/var/db/conn", $config=table(["tablename"] = "conn"), $writer=Log::WRITER_SQLITE ]; Log::add_filter(Conn::LOG, filter); } Would write conn.log to a "postgres" database if we make what changes?? Asad -----Original Message----- From: Aashish Sharma [mailto:asharma at lbl.gov] Sent: 25 March 2017 14:25 To: Ul Asad, Hafiz Cc: bro at bro.org Subject: Re: [Bro] multiple tables in SQLite Database Asad, You'd need to use postgres instead. SQLite + BRO is good for readonly operations. If you have a lot of reads/writes Postgres works fantastic. It should be fairly straight forward to port your current bro SQLITE policy to use postgres code. I have been using postgres instead as well. Don't use sqlite. Aashish On Sat, Mar 25, 2017 at 09:39:28AM +0000, Ul Asad, Hafiz wrote: > Bro Users, > > I have been trying to have multiple logs in a single sqlite database but I am getting the "the database is locked error". This problem was previously raised here, https://bro-tracker.atlassian.net/browse/BIT-1325?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aworklog-tabpanel. I wonder if there has been any solution for it in the Bro 2.5? > > Regards > Asad > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From asharma at lbl.gov Sat Mar 25 07:46:39 2017 From: asharma at lbl.gov (Aashish Sharma) Date: Sat, 25 Mar 2017 07:46:39 -0700 Subject: [Bro] multiple tables in SQLite Database In-Reply-To: References: <20170325142454.GH2071@mac-822.local> Message-ID: <20170325144637.GI2071@mac-822.local> This page should help: https://www.bro.org/sphinx/components/bro-plugins/postgresql/README.html basically, event bro_init() { local filter: Log::Filter = [ $name="postgres", $path="conn", $writer=Log::WRITER_POSTGRESQL, $config=table(["dbname"]="testdb") ]; Log::add_filter(Conn::LOG, filter); } On Sat, Mar 25, 2017 at 02:39:19PM +0000, Ul Asad, Hafiz wrote: > Thanks Aashish, > > So you mean the following script, > > event bro_init() > { > local filter: Log::Filter = > [ > $name="sqlite", > $path="/var/db/conn", > $config=table(["tablename"] = "conn"), > $writer=Log::WRITER_SQLITE > ]; > > Log::add_filter(Conn::LOG, filter); > } > > Would write conn.log to a "postgres" database if we make what changes?? > > Asad > > -----Original Message----- > From: Aashish Sharma [mailto:asharma at lbl.gov] > Sent: 25 March 2017 14:25 > To: Ul Asad, Hafiz > Cc: bro at bro.org > Subject: Re: [Bro] multiple tables in SQLite Database > > Asad, > > You'd need to use postgres instead. SQLite + BRO is good for readonly operations. If you have a lot of reads/writes Postgres works fantastic. It should be fairly straight forward to port your current bro SQLITE policy to use postgres code. I have been using postgres instead as well. Don't use sqlite. > > Aashish > > On Sat, Mar 25, 2017 at 09:39:28AM +0000, Ul Asad, Hafiz wrote: > > Bro Users, > > > > I have been trying to have multiple logs in a single sqlite database but I am getting the "the database is locked error". This problem was previously raised here, https://bro-tracker.atlassian.net/browse/BIT-1325?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aworklog-tabpanel. I wonder if there has been any solution for it in the Bro 2.5? > > > > Regards > > Asad > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From Hafiz.Ul-Asad.1 at city.ac.uk Sat Mar 25 07:51:57 2017 From: Hafiz.Ul-Asad.1 at city.ac.uk (Ul Asad, Hafiz) Date: Sat, 25 Mar 2017 14:51:57 +0000 Subject: [Bro] multiple tables in SQLite Database In-Reply-To: <20170325144637.GI2071@mac-822.local> References: <20170325142454.GH2071@mac-822.local> <20170325144637.GI2071@mac-822.local> Message-ID: Thanks, And have you tried multiple tables? And if yes, how to add multiple tables? Asad -----Original Message----- From: Aashish Sharma [mailto:asharma at lbl.gov] Sent: 25 March 2017 14:47 To: Ul Asad, Hafiz Cc: bro at bro.org Subject: Re: [Bro] multiple tables in SQLite Database This page should help: https://www.bro.org/sphinx/components/bro-plugins/postgresql/README.html basically, event bro_init() { local filter: Log::Filter = [ $name="postgres", $path="conn", $writer=Log::WRITER_POSTGRESQL, $config=table(["dbname"]="testdb") ]; Log::add_filter(Conn::LOG, filter); } On Sat, Mar 25, 2017 at 02:39:19PM +0000, Ul Asad, Hafiz wrote: > Thanks Aashish, > > So you mean the following script, > > event bro_init() > { > local filter: Log::Filter = > [ > $name="sqlite", > $path="/var/db/conn", > $config=table(["tablename"] = "conn"), > $writer=Log::WRITER_SQLITE > ]; > > Log::add_filter(Conn::LOG, filter); > } > > Would write conn.log to a "postgres" database if we make what changes?? > > Asad > > -----Original Message----- > From: Aashish Sharma [mailto:asharma at lbl.gov] > Sent: 25 March 2017 14:25 > To: Ul Asad, Hafiz > Cc: bro at bro.org > Subject: Re: [Bro] multiple tables in SQLite Database > > Asad, > > You'd need to use postgres instead. SQLite + BRO is good for readonly operations. If you have a lot of reads/writes Postgres works fantastic. It should be fairly straight forward to port your current bro SQLITE policy to use postgres code. I have been using postgres instead as well. Don't use sqlite. > > Aashish > > On Sat, Mar 25, 2017 at 09:39:28AM +0000, Ul Asad, Hafiz wrote: > > Bro Users, > > > > I have been trying to have multiple logs in a single sqlite database but I am getting the "the database is locked error". This problem was previously raised here, https://bro-tracker.atlassian.net/browse/BIT-1325?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aworklog-tabpanel. I wonder if there has been any solution for it in the Bro 2.5? > > > > Regards > > Asad > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From init.conf at gmail.com Sat Mar 25 08:03:35 2017 From: init.conf at gmail.com (Init Conf) Date: Sat, 25 Mar 2017 08:03:35 -0700 Subject: [Bro] multiple tables in SQLite Database In-Reply-To: References: <20170325142454.GH2071@mac-822.local> <20170325144637.GI2071@mac-822.local> Message-ID: You create a new filter for each table. local conn_filter: LOG::Filter = [ ?. ] local dns_filter: LOG::Filter = [ ?.. ] then set $path and $name for each individual table as you see fit. If tables don?t exist in postgres, bro creates them for you. then depending on log stream: Log::add_filter(Conn::LOG, conn_filter); Log::add_filter(DNS::LOG, dns_filter); > On Mar 25, 2017, at 7:51 AM, Ul Asad, Hafiz wrote: > > Thanks, > > And have you tried multiple tables? And if yes, how to add multiple tables? > > > Asad > > -----Original Message----- > From: Aashish Sharma [mailto:asharma at lbl.gov] > Sent: 25 March 2017 14:47 > To: Ul Asad, Hafiz > Cc: bro at bro.org > Subject: Re: [Bro] multiple tables in SQLite Database > > This page should help: > > https://www.bro.org/sphinx/components/bro-plugins/postgresql/README.html > > basically, > > event bro_init() > { > local filter: Log::Filter = > [ > $name="postgres", > $path="conn", > $writer=Log::WRITER_POSTGRESQL, > $config=table(["dbname"]="testdb") > ]; > > Log::add_filter(Conn::LOG, filter); > } > > On Sat, Mar 25, 2017 at 02:39:19PM +0000, Ul Asad, Hafiz wrote: >> Thanks Aashish, >> >> So you mean the following script, >> >> event bro_init() >> { >> local filter: Log::Filter = >> [ >> $name="sqlite", >> $path="/var/db/conn", >> $config=table(["tablename"] = "conn"), >> $writer=Log::WRITER_SQLITE >> ]; >> >> Log::add_filter(Conn::LOG, filter); >> } >> >> Would write conn.log to a "postgres" database if we make what changes?? >> >> Asad >> >> -----Original Message----- >> From: Aashish Sharma [mailto:asharma at lbl.gov] >> Sent: 25 March 2017 14:25 >> To: Ul Asad, Hafiz >> Cc: bro at bro.org >> Subject: Re: [Bro] multiple tables in SQLite Database >> >> Asad, >> >> You'd need to use postgres instead. SQLite + BRO is good for readonly operations. If you have a lot of reads/writes Postgres works fantastic. It should be fairly straight forward to port your current bro SQLITE policy to use postgres code. I have been using postgres instead as well. Don't use sqlite. >> >> Aashish >> >> On Sat, Mar 25, 2017 at 09:39:28AM +0000, Ul Asad, Hafiz wrote: >>> Bro Users, >>> >>> I have been trying to have multiple logs in a single sqlite database but I am getting the "the database is locked error". This problem was previously raised here, https://bro-tracker.atlassian.net/browse/BIT-1325?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aworklog-tabpanel. I wonder if there has been any solution for it in the Bro 2.5? >>> >>> Regards >>> Asad >> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From Hafiz.Ul-Asad.1 at city.ac.uk Sun Mar 26 11:18:03 2017 From: Hafiz.Ul-Asad.1 at city.ac.uk (Ul Asad, Hafiz) Date: Sun, 26 Mar 2017 18:18:03 +0000 Subject: [Bro] log rotation Message-ID: Hi, I am analysing a large number of "pcap" files using, bro -r *.pcap my_bro.script The problem is that for each new pcap file, bro over-writes the previous *.log files if I don't change my working directory. Is there a way of controlling the rotation of log files? I know that "broctl" has this time base rotation, but is there any sort of rotation control when bro is run from command line? I can change the working directory, but I want to have all my results in a single a log file (files) so that it is easy to query them. Regards Asad -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170326/6047d649/attachment.html From dotwayland at gmail.com Mon Mar 27 05:52:59 2017 From: dotwayland at gmail.com (Wayland Morgan) Date: Mon, 27 Mar 2017 07:52:59 -0500 Subject: [Bro] Position Announcement - IT Security Analyst or Senior IT Security Analyst Message-ID: Good morning, We are looking for one or more analysts to join our team within the Privacy and Security Office on a full time basis. The Analyst role is within our Cyber Security Operations Center (CSOC) and will have focus in the areas of Incident Response, Threat Intelligence, and Vulnerability Assessment. The deadline to apply is March 29th, 2017. Please contact me if you have any questions. If you know anyone who might be interested, please pass this along. https://jobs.illinois.edu/academic-job-board/job-details?jobID=76728&job=it- security-analyst-or-senior-it-security-analyst-a1700148 Thanks, Wayland *IT Security Analyst or Senior IT Security Analyst* *Office of the CIO* * Technology Services at Illinois* *University of Illinois at Urbana-Champaign* Technology Services at Illinois is the provider of campus-wide computing, networking, communications technology, and instructional technology services supporting academic and research activities of faculty, staff, and students at the University of Illinois at Urbana-Champaign. Our mission is to provide a strong, agile, and customer needs-oriented organization by providing cost-effective, value-added IT services and solutions. Technology Services has 2 openings available for IT Security Analyst and/or Senior IT Security Analyst. These positions are primarily focused on the proactive and reactive systems and response to cybersecurity attacks on the campus and University systems and computing resources. This includes analyzing and responding to attacks and developing defensive measures as well as handling data breach incidents and system compromises. Both the minimum requirements and desired qualifications will be considered in choosing the most qualified candidates. These positions reports to the Assistant Director, Privacy and IT Security Infrastructure who reports to the Chief Security and Privacy Officer. *Primary Position Function* Respond to and handle cybersecurity attacks upon the University. Analyze and respond to attacks, compromises, and breaches. Proactively develop defensive security measures and capabilities. Handle information security incidents and other adverse information security events. *Major Duties and Responsibilities* ? Use forensic, incident response, and process expertise to respond to and investigate system, service, or network attacks and breaches. ? Work in one or more of the following domains including but not limited to Data Security, Digital Forensics, Incident Response and Analysis, IT Systems and Operations, Network Security, Systems and Applications Security or Vulnerability Management. ? Apply professional information security knowledge, skills, and abilities with supervision on projects and programs. ? Work with IT Security Engineers to plan and build proactive defenses, automation, and event detection into the University IT infrastructure. ? Work with local, State, and Federal law enforcement as well as with University staff on personnel and other investigations involving regulated private information. ? Maintain a professional expertise by attending outside seminars/courses and thorough review of published literature. ? Participate in team discussions to formulate new or enhance existing processes, policies, and standards. ? Review existing procedures and practices with operational staff across the University and implement University standards and industry best practices for security. ? Provide excellent quality of Customer Service on behalf of the IT Security Office. ? Set and manage customer expectations through partnership with Technology Services Service Center. ? Advocate for Technology Services clients in service planning and deployment across the organization. ? Resolve customer satisfaction issues. ? Understand the overall processes and procedures of the organization and make recommendations in the continual improvement of those processes and procedures, providing for management analysis and recommendations on continual improvement. ? Work non-traditional hours and respond to on-call requests in a 24 x 7 service environment. *In addition, Senior IT Security Analyst Major Duties and Responsibilities* ? Represent the security office in collaborative initiatives. ? Draft procedural documentation. ? Generate analysis documents for technical security issues and present to both technical and mid to higher-level executive leadership audiences. ? Participate and manage internal and University meetings. ? Participate in team discussions to formulate new or enhance existing processes, policies, and standards. Drive discussions as needed to represent the incident response or forensic functions. ? Collaborate with peers to complete complex technical solutions with limited supervision. ? Review existing procedures and practices with operational staff across the University and implement University standards and industry best practices for security. *Organizational Chart* Vice Chancellor for Academic Affairs and Provost Chief Information Officer Chief Security and Privacy Officer Assistant Director, Privacy and IT Security Infrastructure IT Security Analyst and/or Senior Security Analyst *Position Requirements and Qualifications* *Education* Bachelor's degree. *Preferred Education* Bachelor?s degree in related field. *Experience ? IT Security Analyst* ? One or more years in an IT security role or IT support role with significant security responsibilities. ? Demonstrated expertise in one or more of the following domains Data Security, Digital Forensics, Incident Response and Analysis, IT Systems and Operations, Network Security, Systems and Applications Security or Vulnerability Management. ? Excellent oral and written communication skills. ? Individuals will be required to submit to a background examination. ? Demonstrated ability in effective communication and collaborating in a high performance team environment. ? Demonstrated commitment to customer service. ? Experience functioning in diverse workgroups. *Experience ? Senior IT Security Analyst* ? Two or more years in an IT security role or IT support role with significant security responsibilities. ? Demonstrated expertise in one or more of the following domains Data Security, Digital Forensics, Incident Response and Analysis, IT Systems and Operations, Network Security, Systems and Applications Security or Vulnerability Management. ? Excellent oral and written communication skills. ? Individuals will be required to submit to a background examination. ? Demonstrated ability in effective communication and collaborating in a high performance team environment. ? Demonstrated commitment to customer service. ? Experience functioning in diverse workgroups. *Preferred Experience - IT Security Analyst* ? One or more years of experience in an academic campus IT environment. ? Experience working with or for a dedicated IT security office. ? Experience working with IT Security Incident Response. ? Experience performing malware analyses. ? Experience performing vulnerability scans in a professional environment. *Preferred Experience ? Senior IT Security Analyst* ? One or more years of experience in an academic campus IT environment. ? Experience working with IT Security Incident Response. ? Experience performing malware analyses. ? Experience performing vulnerability scans in a professional environment. *Knowledge Requirements - Senior IT Security Analyst* ? Familiarity working with a Security Event Management product. ? Familiarity with security event triage ? Familiarity with enterprise forensic tools. *Preferred Knowledge Requirements ? IT Security Analyst* ? Familiarity working with a Security Event Management product. ? Familiarity with security event triage. *Training Requirements -* *Senior IT Security Analyst* ? Completion of at least one information security certification is required after working one year. *Preferred Training -* *IT Security Analyst & Senior IT Security Analyst* ? GSEC, GCIH, GCIA, CISSP, CEA, MCA, CCSA, CISA or similar certifications highly desired. *Appointment Status and Salary* This position is a regular, full-time, twelve-month academic professional appointment and includes salary and a benefits package (24 vacation days, 12 sick days, health, dental, vision, SURS retirement). For other University provided benefits, please go to: https://nessie.uihr. uillinois.edu. Salary is competitive and commensurate with qualifications and experience. There will be no relocation costs included in this package. Applicants should have a current, valid legal authorization to work in the United States. The start date will be as soon as possible after the close date. *To Apply* For full consideration, applications should be received by March 29, 2017. Interviews and hires may take place before the closing date, but all applications received by the closing date will receive full consideration. Please create a candidate profile at https://jobs.illinois.edu and upload in one file a letter of application, resume, and the names of three professional references including address, telephone, and email information. The online application will require names and contact information for the three professional references. Three letters of recommendation will be required for Search Finalists. Employment requires a criminal background check. For further information about this position, please contact Lori Oakes at TechSvc-HR at mx.uillinois.edu or call (217) 333-4222. The University of Illinois conducts criminal background checks on all job candidates upon acceptance of a contingent offer. *Illinois is an equal opportunity employer and all qualified applicants will receive consideration for employment without regard to race, religion, color, national origin, sex, sexual orientation, gender identity, age, status as a protected veteran, status as a qualified individual with a disability, or criminal conviction history. Illinois welcomes individuals with diverse backgrounds, experiences, and ideas who embrace and value diversity and inclusivity. (**www.inclusiveillinois.illinois.edu* *).* -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170327/2bbe3d53/attachment-0001.html From philosnef at gmail.com Mon Mar 27 09:07:13 2017 From: philosnef at gmail.com (erik clark) Date: Mon, 27 Mar 2017 12:07:13 -0400 Subject: [Bro] dpdk Message-ID: Any idea if this will be supported? I can not find any reference in the past year indicating this one way or another. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170327/a424f174/attachment.html From al.kefallonitis at gmail.com Mon Mar 27 12:27:27 2017 From: al.kefallonitis at gmail.com (Alex Kefallonitis) Date: Mon, 27 Mar 2017 22:27:27 +0300 Subject: [Bro] bro intel notice log Message-ID: Critical stack module is up and running and generates intel logs. I want bro to send email when an indicator is seen . Although i receive mail from bro for notices and i added do_notice.bro to local.bro i never seen a notice intel email or log. Any advice? I also try adding these to local.bro redef Notice::emailed_types += { Intel::Notice, Intel::DOMAIN, TeamCymruMalwareHashRegistry::Match, Software::Vulnerable_Version, Traceroute::Detected, Scan::Address_Scan, Scan::Port_Scan, Conn::Content_Gap, DNS::External_Name, FTP::Bruteforcing, FTP::Site_Exec_Success, HTTP::SQL_Injection_Attacker, HTTP::SQL_Injection_Victim, SMTP::Blocklist_Error_Message, SMTP::Blocklist_Blocked_Host, SMTP::Suspicious_Origination, SSH::Password_Guessing, SSH::Login_By_Password_Guesser, SSH::Watched_Country_Login, SSH::Interesting_Hostname_Login, SSL::Certificate_Expired, SSL::Certificate_Expires_Soon, SSL::Certificate_Not_Valid_Yet, Heartbleed::SSL_Heartbeat_Attack, Heartbleed::SSL_Heartbeat_Attack_Success, Heartbleed::SSL_Heartbeat_Odd_Length, Heartbleed::SSL_Heartbeat_Many_Requests, }; hook Notice::policy(n: Notice::Info) { add n$actions[Notice::ACTION_EMAIL]; } but nothing changed -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170327/b9ab7ab4/attachment.html From al.kefallonitis at gmail.com Tue Mar 28 00:24:53 2017 From: al.kefallonitis at gmail.com (Alex Kefallonitis) Date: Tue, 28 Mar 2017 10:24:53 +0300 Subject: [Bro] bro intel notice log In-Reply-To: References: Message-ID: I want bro to send email notification on all notices...but i can't get intel to notice log i can't figure out what i am doing wrong 2017-03-27 22:27 GMT+03:00 Alex Kefallonitis : > Critical stack module is up and running and generates intel logs. I want > bro to send email when an indicator is seen . Although i receive mail from > bro for notices and i added do_notice.bro to local.bro i never seen a > notice intel email or log. Any advice? > > I also try adding these to local.bro > > redef Notice::emailed_types += { > Intel::Notice, > Intel::DOMAIN, > TeamCymruMalwareHashRegistry::Match, > Software::Vulnerable_Version, > Traceroute::Detected, > Scan::Address_Scan, > Scan::Port_Scan, > Conn::Content_Gap, > DNS::External_Name, > FTP::Bruteforcing, > FTP::Site_Exec_Success, > HTTP::SQL_Injection_Attacker, > HTTP::SQL_Injection_Victim, > SMTP::Blocklist_Error_Message, > SMTP::Blocklist_Blocked_Host, > SMTP::Suspicious_Origination, > SSH::Password_Guessing, > SSH::Login_By_Password_Guesser, > SSH::Watched_Country_Login, > SSH::Interesting_Hostname_Login, > SSL::Certificate_Expired, > SSL::Certificate_Expires_Soon, > SSL::Certificate_Not_Valid_Yet, > Heartbleed::SSL_Heartbeat_Attack, > Heartbleed::SSL_Heartbeat_Attack_Success, > Heartbleed::SSL_Heartbeat_Odd_Length, > Heartbleed::SSL_Heartbeat_Many_Requests, > }; > > hook Notice::policy(n: Notice::Info) > { > add n$actions[Notice::ACTION_EMAIL]; > } > > > but nothing changed > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170328/daa2b4fb/attachment.html From al.kefallonitis at gmail.com Tue Mar 28 02:04:36 2017 From: al.kefallonitis at gmail.com (Alex Kefallonitis) Date: Tue, 28 Mar 2017 12:04:36 +0300 Subject: [Bro] bro intel notice log In-Reply-To: References: Message-ID: Update i also tried loading critical stack manually on local.bro hook Notice::policy(n: Notice::Info) { add n$actions[Notice::ACTION_EMAIL]; } @load base/frameworks/intel @load frameworks/intel/seen @load frameworks/intel/do_notice redef Intel::read_files += { "/opt/critical-stack/frameworks/intel/master-public.bro.dat" }; I get no errors i have intel logs but no notice and emails like do_notice doesn't work... 2017-03-28 10:24 GMT+03:00 Alex Kefallonitis : > I want bro to send email notification on all notices...but i can't get > intel to notice log i can't figure out what i am doing wrong > > 2017-03-27 22:27 GMT+03:00 Alex Kefallonitis : > >> Critical stack module is up and running and generates intel logs. I want >> bro to send email when an indicator is seen . Although i receive mail from >> bro for notices and i added do_notice.bro to local.bro i never seen a >> notice intel email or log. Any advice? >> >> I also try adding these to local.bro >> >> redef Notice::emailed_types += { >> Intel::Notice, >> Intel::DOMAIN, >> TeamCymruMalwareHashRegistry::Match, >> Software::Vulnerable_Version, >> Traceroute::Detected, >> Scan::Address_Scan, >> Scan::Port_Scan, >> Conn::Content_Gap, >> DNS::External_Name, >> FTP::Bruteforcing, >> FTP::Site_Exec_Success, >> HTTP::SQL_Injection_Attacker, >> HTTP::SQL_Injection_Victim, >> SMTP::Blocklist_Error_Message, >> SMTP::Blocklist_Blocked_Host, >> SMTP::Suspicious_Origination, >> SSH::Password_Guessing, >> SSH::Login_By_Password_Guesser, >> SSH::Watched_Country_Login, >> SSH::Interesting_Hostname_Login, >> SSL::Certificate_Expired, >> SSL::Certificate_Expires_Soon, >> SSL::Certificate_Not_Valid_Yet, >> Heartbleed::SSL_Heartbeat_Attack, >> Heartbleed::SSL_Heartbeat_Attack_Success, >> Heartbleed::SSL_Heartbeat_Odd_Length, >> Heartbleed::SSL_Heartbeat_Many_Requests, >> }; >> >> hook Notice::policy(n: Notice::Info) >> { >> add n$actions[Notice::ACTION_EMAIL]; >> } >> >> >> but nothing changed >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170328/db028e2e/attachment.html From bobharrelsons at gmail.com Tue Mar 28 10:33:51 2017 From: bobharrelsons at gmail.com (Robert Harrelson) Date: Tue, 28 Mar 2017 13:33:51 -0400 Subject: [Bro] &log cert_chain attribute (vector of Files::info) in ssl.log file Message-ID: &log cert_chain attribute (vector of Files::info) in ssl.log file. I would like to list the server's chain of certificates in ssl.log (log of handshake data) alongside each handshake. In ssl.log, the cert_chain attribute (certificate chain of the server) is not being logged, and is of type *vector of **Files::info*. When I tried to add "&log" attribute to cert_chain in files.bro, it gave an error that: ".... cert_chain is of type that cannot be logged." When I tried changing the type from *vector of Files::info* to *vector of string*, it sprang up some different errors since cert_chain is referenced as a *vector of Files::info* in other parts of files.bro script. Please tell me how I can log the cert_chain attribute in ssl.log file. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170328/bed0e745/attachment.html From philosnef at gmail.com Wed Mar 29 07:17:56 2017 From: philosnef at gmail.com (erik clark) Date: Wed, 29 Mar 2017 10:17:56 -0400 Subject: [Bro] is vlan bpf broken in bro Message-ID: Per this thread: http://serverfault.com/questions/544651/vlan-tags-not-shown-in-packet-capture-linux-via-tcpdump tcpdump can't process vlan filters. Testing confirms this. >From link: tcpdump -i eth0 -Uw - | tcpdump -en -r - vlan 4 This works and displays only vlan 4 stuff. The reverse does not: tcpdump -i eth0 -Uw - "vlan 4" |tcpdump -en -r - This displays ALL vlans tagged in the traffic, and not just vlan 4. This is on RHEL 7. Apparently there are some issues with x86_64 vlan acceleration. The short of it: Will bro respect vlan filters, or does it have the same issue that tcpdump and libpcap seem to have? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170329/e5b51c27/attachment.html From seth at corelight.com Wed Mar 29 08:14:42 2017 From: seth at corelight.com (Seth Hall) Date: Wed, 29 Mar 2017 11:14:42 -0400 Subject: [Bro] is vlan bpf broken in bro In-Reply-To: References: Message-ID: <5300ABA1-CC5A-4A65-B93F-94D96E0BEF1F@corelight.com> > On Mar 29, 2017, at 10:17 AM, erik clark wrote: > > The short of it: Will bro respect vlan filters, or does it have the same issue that tcpdump and libpcap seem to have? If it's acquiring packets through straight libpcap on linux and linux has an issue with vlan handling, then yes, you will have the same problem. If you are using some alternate packet handling mechanism then the problem with likely not be there. Are you using the default libpcap on your distro? .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From philosnef at gmail.com Wed Mar 29 08:28:59 2017 From: philosnef at gmail.com (erik clark) Date: Wed, 29 Mar 2017 11:28:59 -0400 Subject: [Bro] is vlan bpf broken in bro In-Reply-To: <5300ABA1-CC5A-4A65-B93F-94D96E0BEF1F@corelight.com> References: <5300ABA1-CC5A-4A65-B93F-94D96E0BEF1F@corelight.com> Message-ID: Libpcap from ntop for pf_ring, on a vmxnet3 interface. ixgbe nics have rxvlan option in ethtool which disables acceleration, but these are all virtual nics; I cant set rx-vlan-offload (bad command, despite being listed in ethtool -k), so I think I am out of luck for filtering. On Wed, Mar 29, 2017 at 11:14 AM, Seth Hall wrote: > > > On Mar 29, 2017, at 10:17 AM, erik clark wrote: > > > > The short of it: Will bro respect vlan filters, or does it have the same > issue that tcpdump and libpcap seem to have? > > If it's acquiring packets through straight libpcap on linux and linux has > an issue with vlan handling, then yes, you will have the same problem. If > you are using some alternate packet handling mechanism then the problem > with likely not be there. Are you using the default libpcap on your distro? > > .Seth > > -- > Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170329/5c13a840/attachment.html From seth at corelight.com Wed Mar 29 08:31:54 2017 From: seth at corelight.com (Seth Hall) Date: Wed, 29 Mar 2017 11:31:54 -0400 Subject: [Bro] is vlan bpf broken in bro In-Reply-To: References: <5300ABA1-CC5A-4A65-B93F-94D96E0BEF1F@corelight.com> Message-ID: > On Mar 29, 2017, at 11:28 AM, erik clark wrote: > > Libpcap from ntop for pf_ring, on a vmxnet3 interface. ixgbe nics have rxvlan option in ethtool which disables acceleration, but these are all virtual nics; I cant set rx-vlan-offload (bad command, despite being listed in ethtool -k), so I think I am out of luck for filtering. Ah, that's a much more narrow case than I thought you were referring to. You might be out of luck without deeper changes to things (or you could use netmap, it might work there!). .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From philosnef at gmail.com Wed Mar 29 08:50:50 2017 From: philosnef at gmail.com (erik clark) Date: Wed, 29 Mar 2017 11:50:50 -0400 Subject: [Bro] is vlan bpf broken in bro In-Reply-To: References: <5300ABA1-CC5A-4A65-B93F-94D96E0BEF1F@corelight.com> Message-ID: Ive actually tried this with stock libpcap, and have the same results. I just checked an ixgbe nic I have laying around, and I get the same results, without any option to set rxvlan... >From the various boxes I have tried so far (e1000e, ixgbe, vmxnet3), all of these exhibit the same filtering problem with vlans. Maybe netmap can handle it, but it doesnt appear that libpcap (pf_ring or not) can properly handle vlan filtering... Since we are going with af_packet, netmap is unfortunately off the table. On Wed, Mar 29, 2017 at 11:31 AM, Seth Hall wrote: > > > On Mar 29, 2017, at 11:28 AM, erik clark wrote: > > > > Libpcap from ntop for pf_ring, on a vmxnet3 interface. ixgbe nics have > rxvlan option in ethtool which disables acceleration, but these are all > virtual nics; I cant set rx-vlan-offload (bad command, despite being listed > in ethtool -k), so I think I am out of luck for filtering. > > Ah, that's a much more narrow case than I thought you were referring to. > You might be out of luck without deeper changes to things (or you could use > netmap, it might work there!). > > .Seth > > -- > Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170329/dd8af67f/attachment.html From jan.grashoefer at gmail.com Wed Mar 29 09:06:22 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Wed, 29 Mar 2017 18:06:22 +0200 Subject: [Bro] is vlan bpf broken in bro In-Reply-To: References: <5300ABA1-CC5A-4A65-B93F-94D96E0BEF1F@corelight.com> Message-ID: <32ff84f0-0975-697d-4895-9e237aaa5b7f@gmail.com> > Since we are going with af_packet, netmap is > unfortunately off the table. In case you are using AF_Packet, unfortunately VLANs won't be available in Bro. See https://github.com/J-Gras/bro-af_packet-plugin#limitations Jan From bobharrelsons at gmail.com Wed Mar 29 13:20:43 2017 From: bobharrelsons at gmail.com (Robert Harrelson) Date: Wed, 29 Mar 2017 16:20:43 -0400 Subject: [Bro] Log serial number in ssl.log Message-ID: How do I log the serial number of the certificate in ssl.log? I tried to perform this in protocols/ssl/files.bro file at the event ssl_established(), but this event is almost never called. This means that issuer and subject also almost never get logged. Is this because the handshake happens at line speed, but the certificate does not get processed as fast, so the certificate details are almost never available to Bro when it logs the handshake data in ssl.log? Thanks, Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170329/4536ffef/attachment.html From jazoff at illinois.edu Wed Mar 29 13:29:07 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 29 Mar 2017 20:29:07 +0000 Subject: [Bro] Log serial number in ssl.log In-Reply-To: References: Message-ID: > On Mar 29, 2017, at 4:20 PM, Robert Harrelson wrote: > > How do I log the serial number of the certificate in ssl.log? > > I tried to perform this in protocols/ssl/files.bro file at the event ssl_established(), but this event is almost never called. This means that issuer and subject also almost never get logged. > > Is this because the handshake happens at line speed, but the certificate does not get processed as fast, so the certificate details are almost never available to Bro when it logs the handshake data in ssl.log? > > Thanks, > > Robert It sounds like your bro installation is not functioning properly. ssl_established is raised on every ssl connection and includes all of the information about the handshake. What does the ssl.log and conn.log entry look like for one of the ssl connections that is missing the issuer and subject fields? -- - Justin Azoff From jazoff at illinois.edu Wed Mar 29 14:07:16 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 29 Mar 2017 21:07:16 +0000 Subject: [Bro] Log serial number in ssl.log In-Reply-To: References: Message-ID: <88C0CF1E-E1DE-4503-8238-BB9CB43733D4@illinois.edu> > On Mar 29, 2017, at 4:58 PM, Robert Harrelson wrote: > > Thank you for the quick reply. > > ssl.log > > #close 2017-03-22-16-35-24 > > conn.log > > > #close 2017-03-27-16-40-01 These logs are from 5 days apart and do not match up with each other. I'd need to see the conn.log entires that correspond to connections like CrfdsS268VpsZjzjV2, C3PoFJ0FJ51yFKOm7, CoxsGYKjELt5oTZUj, C5EqDA4f0hY9BlCR2d etc -- - Justin Azoff From jazoff at illinois.edu Wed Mar 29 14:44:03 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 29 Mar 2017 21:44:03 +0000 Subject: [Bro] Log serial number in ssl.log In-Reply-To: References: <88C0CF1E-E1DE-4503-8238-BB9CB43733D4@illinois.edu> Message-ID: <025CF9C9-07F2-4E55-9996-332C01DF0D5C@illinois.edu> > On Mar 29, 2017, at 5:38 PM, Robert Harrelson wrote: > > Dear Justin, > > Sorry for that mistake. I may have mixed up the files. I just re-ran bro and have copied below the results of ssl.log and conn.log. > Thanks again for your help! > > --Robert > > > > conn.log > > #separator \x09 > #set_separator , > #empty_field (empty) > #unset_field - > #path conn > #open 2017-03-29-17-27-40 > #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig local_resp missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents > #types time string addr port addr port enum string interval count count string bool bool count string count count count count set[string] > > 1490822851.106865 Ckk89B3l4i616mbQx6 10.245.44.33 61486 216.58.219.100 443 tcp - 12.846213 0 4118 SHR - - 0 ^hadf 0 0 9 4594 (empty) > Ah yes... the hadf for all of your connection histories shows that Bro is only seeing half of your connections Are you running bro on 10.245.44.33 itself? https://www.bro.org/documentation/faq.html#why-isn-t-bro-producing-the-logs-i-expect-a-note-about-checksums -- - Justin Azoff From bobharrelsons at gmail.com Wed Mar 29 15:52:54 2017 From: bobharrelsons at gmail.com (Robert Harrelson) Date: Wed, 29 Mar 2017 18:52:54 -0400 Subject: [Bro] Log serial number in ssl.log In-Reply-To: <025CF9C9-07F2-4E55-9996-332C01DF0D5C@illinois.edu> References: <88C0CF1E-E1DE-4503-8238-BB9CB43733D4@illinois.edu> <025CF9C9-07F2-4E55-9996-332C01DF0D5C@illinois.edu> Message-ID: Yes, I am running bro on an iMac having IP address 10.245.44.33 . I will try out the workarounds for ignoring checksums tomorrow, and let you know how it went. Let me know if you have any more advice, I am all ears. Thank you so much! --Robert On Wed, Mar 29, 2017 at 5:44 PM, Azoff, Justin S wrote: > > > On Mar 29, 2017, at 5:38 PM, Robert Harrelson > wrote: > > > > Dear Justin, > > > > Sorry for that mistake. I may have mixed up the files. I just re-ran bro > and have copied below the results of ssl.log and conn.log. > > Thanks again for your help! > > > > --Robert > > > > > > > > conn.log > > > > #separator \x09 > > #set_separator , > > #empty_field (empty) > > #unset_field - > > #path conn > > #open 2017-03-29-17-27-40 > > #fields ts uid id.orig_h id.orig_p id.resp_h > id.resp_p proto service duration orig_bytes > resp_bytes conn_state local_orig local_resp > missed_bytes history orig_pkts orig_ip_bytes resp_pkts > resp_ip_bytes tunnel_parents > > #types time string addr port addr port enum > string interval count count string bool bool count > string count count count count set[string] > > > > 1490822851.106865 Ckk89B3l4i616mbQx6 10.245.44.33 61486 > 216.58.219.100 443 tcp - 12.846213 0 4118 > SHR - - 0 ^hadf 0 0 9 4594 > (empty) > > > > Ah yes... the hadf for all of your connection histories shows that Bro is > only seeing half of your connections > > Are you running bro on 10.245.44.33 itself? > > https://www.bro.org/documentation/faq.html#why- > isn-t-bro-producing-the-logs-i-expect-a-note-about-checksums > > > -- > - Justin Azoff > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170329/363cec9d/attachment.html From vladg at illinois.edu Thu Mar 30 07:17:48 2017 From: vladg at illinois.edu (Vlad Grigorescu) Date: Thu, 30 Mar 2017 09:17:48 -0500 Subject: [Bro] BinPAC &check In-Reply-To: <201703201913.v2KJDi3I020462@vladg.net> References: <201703201913.v2KJDi3I020462@vladg.net> Message-ID: No, unfortunately this hasn't been implemented. I believe that this is the current code snippet: > src/pac_type.h: void AddCheck(Expr *expr) { /* TODO */ } We should update that README to make that clearer -- or just implement it, of course. :-) --Vlad "D. W." writes: > Hi there, > > can someone explain me how the &check attribute in binpac is suppose to > work, or does it even work at all? > > I checked my protocol analyzer after the make process and I couldn't > find any impact in the code, like other attributes does (like byteorder > etc... ) > > Thanks > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 800 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170330/51fba0d7/attachment-0001.bin From andrew.dellana at bayer.com Thu Mar 30 07:32:51 2017 From: andrew.dellana at bayer.com (Andrew Dellana) Date: Thu, 30 Mar 2017 14:32:51 +0000 Subject: [Bro] NetControl configuration In-Reply-To: References: <9d900f340345412d8c545f2ced222966@moxde9.na.bayer.cnb> Message-ID: Got around to adding net control to all the scripts, and now they are failing. The script is FoxIT's ransomware script. Any idea how I can get this to work? event NetControl::init() { NetControl::drop_connection (conn_id, 0, "Cyrpto Blocked") } hook Notice::policy(n: Notice::Info) { if fox_entropy=T Then add n$actions[Notice::ACTION_DROP] add n$actions[Notice::ACTION_EMAIL]; } error in /opt/bro/share/bro/base/init-bare.bro, lines 123-127 and /opt/bro/share/bro/base/misc/CryptoRansomCheck.bro, line 127: type clash (conn_id and conn_id) error in /opt/bro/share/bro/base/misc/CryptoRansomCheck.bro, line 127 and /opt/bro/share/bro/base/init-bare.bro, lines 123-127: type mismatch (conn_id and conn_id) error in /opt/bro/share/bro/base/misc/CryptoRansomCheck.bro, line 127: argument type mismatch in function call (NetControl::drop_connection(conn_id, 0, Cyrpto Blocked)) error in /opt/bro/share/bro/base/misc/CryptoRansomCheck.bro, line 128: syntax error, at or near "}" Freundliche Gr??e / Best regards, Andrew Dellana Intern ________________________ -----Original Message----- From: Azoff, Justin S [mailto:jazoff at illinois.edu] Sent: Thursday, March 16, 2017 11:08 AM To: Andrew Dellana Cc: bro at bro.org Subject: Re: [Bro] NetControl configuration > On Mar 16, 2017, at 11:04 AM, Andrew Dellana wrote: > > Yes, I do want to make the NetControl actions based on what is alerted in Notices. Can all the helpers be stored in one file and only call the helper that is needed? Yep, you can do exactly that. -- - Justin Azoff From bobharrelsons at gmail.com Thu Mar 30 08:11:19 2017 From: bobharrelsons at gmail.com (Robert Harrelson) Date: Thu, 30 Mar 2017 11:11:19 -0400 Subject: [Bro] Log serial number in ssl.log In-Reply-To: References: <88C0CF1E-E1DE-4503-8238-BB9CB43733D4@illinois.edu> <025CF9C9-07F2-4E55-9996-332C01DF0D5C@illinois.edu> Message-ID: The workaround is working. Thank you On Wed, Mar 29, 2017 at 6:52 PM, Robert Harrelson wrote: > Yes, I am running bro on an iMac having IP address 10.245.44.33 . > > I will try out the workarounds for ignoring checksums tomorrow, and let > you know how it went. Let me know if you have any more advice, I am all > ears. > > Thank you so much! > > --Robert > > On Wed, Mar 29, 2017 at 5:44 PM, Azoff, Justin S > wrote: > >> >> > On Mar 29, 2017, at 5:38 PM, Robert Harrelson >> wrote: >> > >> > Dear Justin, >> > >> > Sorry for that mistake. I may have mixed up the files. I just re-ran >> bro and have copied below the results of ssl.log and conn.log. >> > Thanks again for your help! >> > >> > --Robert >> > >> > >> > >> > conn.log >> > >> > #separator \x09 >> > #set_separator , >> > #empty_field (empty) >> > #unset_field - >> > #path conn >> > #open 2017-03-29-17-27-40 >> > #fields ts uid id.orig_h id.orig_p >> id.resp_h id.resp_p proto service duration >> orig_bytes resp_bytes conn_state local_orig local_resp >> missed_bytes history orig_pkts orig_ip_bytes resp_pkts >> resp_ip_bytes tunnel_parents >> > #types time string addr port addr port enum >> string interval count count string bool bool count >> string count count count count set[string] >> > >> > 1490822851.106865 Ckk89B3l4i616mbQx6 10.245.44.33 61486 >> 216.58.219.100 443 tcp - 12.846213 0 4118 >> SHR - - 0 ^hadf 0 0 9 4594 >> (empty) >> > >> >> Ah yes... the hadf for all of your connection histories shows that Bro is >> only seeing half of your connections >> >> Are you running bro on 10.245.44.33 itself? >> >> https://www.bro.org/documentation/faq.html#why-isn-t-bro- >> producing-the-logs-i-expect-a-note-about-checksums >> >> >> -- >> - Justin Azoff >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170330/ebb6b343/attachment.html From troyj at maine.edu Thu Mar 30 18:41:25 2017 From: troyj at maine.edu (Troy Jordan) Date: Thu, 30 Mar 2017 21:41:25 -0400 Subject: [Bro] does Spicy have a BNF? In-Reply-To: <20170322175351.GH42999@icir.org> References: <6e3887ca-9bcc-1247-81a0-a3d64e0bba2a@maine.edu> <20170322175351.GH42999@icir.org> Message-ID: Yes, thanks! - Troy On 3/22/2017 1:53 PM, Robin Sommer wrote: > > On Sun, Mar 19, 2017 at 17:46 -0400, Troy Jordan wrote: > >> Does Spicy/BinPAC++ have a BNF? > Not quite sure what you're asking for. If you're looking for a grammar > for the Spicy language itself, that's here: > https://github.com/rsmmr/hilti/blob/master/spicy/parser/parser.yy > > Robin > -- Troy Jordan t r o y j @ m a i n e . e d u GIAC GCIH,GCIA ------------------------------------------------------------ Network Systems Security Analyst Information Technology Security Office University of Maine System ------------------------------------------------------------ 233 Science Building | voice: 207.561.3590 Portland, ME 04103 | fax: 509.351.3650 "As you all know, Security Is Mortals chiefest Enemy" William Shakespeare, Macbeth