From ansaf_130 at yahoo.com Tue Aug 2 00:55:52 2016 From: ansaf_130 at yahoo.com (Aneela Safdar) Date: Tue, 2 Aug 2016 07:55:52 +0000 (UTC) Subject: [Bro] Changing timezone in log's timestamps References: <1238696709.8007446.1470124552618.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <1238696709.8007446.1470124552618.JavaMail.yahoo@mail.yahoo.com> Hi, What settings are to do in order to change the timezone of log files??Regards, Aneela Safdar -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160802/6237d717/attachment.html From ansaf_130 at yahoo.com Tue Aug 2 01:10:09 2016 From: ansaf_130 at yahoo.com (Aneela Safdar) Date: Tue, 2 Aug 2016 08:10:09 +0000 (UTC) Subject: [Bro] Changing timezone in log's timestamps In-Reply-To: <1238696709.8007446.1470124552618.JavaMail.yahoo@mail.yahoo.com> References: <1238696709.8007446.1470124552618.JavaMail.yahoo.ref@mail.yahoo.com> <1238696709.8007446.1470124552618.JavaMail.yahoo@mail.yahoo.com> Message-ID: <418105455.7992730.1470125409583.JavaMail.yahoo@mail.yahoo.com> Hi, What settings are to do in order to change the timezone of log files??Regards, Aneela Safdar -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160802/13f5ec8e/attachment.html From 55sjp55 at gmail.com Tue Aug 2 02:34:40 2016 From: 55sjp55 at gmail.com (Scott P) Date: Tue, 2 Aug 2016 05:34:40 -0400 Subject: [Bro] Newbie question Extract Binaries from traffic In-Reply-To: References: <20160727011353.GA31128@Beezling.local> Message-ID: Thank you both. Exactly what I was looking for On Jul 27, 2016 08:22, "Hosom, Stephen M" wrote: > Scott, > > I have an example of how file extraction is usually done on modern Bro > versions here: > > https://github.com/hosom/bro-file-extraction > > I'm assuming based on what it looks like you were trying to do that you > want to extract PE files that appear in HTTP and FTP? > > You might try loading the extract-pe.bro script from the plugins directory > in that repo. It won't limit the extraction to just HTTP and FTP though. > You'd have to modify the script to get it to do that. > > -Stephen > > ________________________________________ > From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Johanna > Amann [johanna at icir.org] > Sent: Tuesday, July 26, 2016 9:13 PM > To: Scott P > Cc: bro at bro.org > Subject: Re: [Bro] Newbie question Extract Binaries from traffic > > Hi Scott, > > I think the syntax you are using there was retired with Bro 2.2 (or > potentially earlier). Newer versions of Bro use the file analysis > framework; Documentation for it is available at > https://www.bro.org/sphinx-git/frameworks/file-analysis.html > > To see an example of someone using the framework, see e.g. the email > thread at > http://mailman.icsi.berkeley.edu/pipermail/bro/2015-July/008715.html > > I hope this helps, > Johanna > > On Tue, Jul 26, 2016 at 10:08:57AM -0400, Scott P wrote: > > Newbie question added the following to my local.bro file > > > > #Extract EXEs > > redef HTTP::extract_file_types += /application\/x-dosexec/; > > redef FTP::extract_file_types += /application\/x-dosexec/; > > > > #Extract files to /nsm/bro/extracted > > redef HTTP::extraction_prefix = "/nsm/bro/extracted/http/http-item"; > > redef FTP::extraction_prefix = "/nsm/bro/extracted/ftp/ftp-file"; > > > > But when I test against the file I am getting: > > > > sudo bro -r http-putty.pcap /opt/bro/share/bro/site/local.bro > > > > > > error in /opt/bro/share/bro/site/local.bro, line 105: "redef" used but > not > > previously defined (HTTP::extract_file_types) > > internal warning in /opt/bro/share/bro/site/local.bro, line 105: Can't > > document redef of HTTP::extract_file_types, identifier lookup failed > > error in /opt/bro/share/bro/site/local.bro, line 106: "redef" used but > not > > previously defined (FTP::extract_file_types) > > internal warning in /opt/bro/share/bro/site/local.bro, line 106: Can't > > document redef of FTP::extract_file_types, identifier lookup failed > > error in /opt/bro/share/bro/site/local.bro, line 109: "redef" used but > not > > previously defined (HTTP::extraction_prefix) > > internal warning in /opt/bro/share/bro/site/local.bro, line 109: Can't > > document redef of HTTP::extraction_prefix, identifier lookup failed > > error in /opt/bro/share/bro/site/local.bro, line 110: "redef" used but > not > > previously defined (FTP::extraction_prefix) > > internal warning in /opt/bro/share/bro/site/local.bro, line 110: Can't > > document redef of FTP::extraction_prefix, identifier lookup failed > > > > > > Any insight would be helpful. > > > > -- > > *Read, *pause, *think,* pause, *write*, pause, (perhaps erase), pause, > *read, > > *pause, (perhaps *go back*), pause, *write, ....* -- *Alan Turing (1936)* > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160802/649a5ae2/attachment.html From philosnef at yahoo.com Tue Aug 2 05:43:23 2016 From: philosnef at yahoo.com (philosnef) Date: Tue, 2 Aug 2016 12:43:23 +0000 (UTC) Subject: [Bro] question about intel files References: <1610342229.9668660.1470141803800.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <1610342229.9668660.1470141803800.JavaMail.yahoo@mail.yahoo.com> Are intel files loaded into memory or statically evaluated? We have a 7.4 meg intel file we are looking to push; however, out of 400 gigs of ram, we are using 400 gigs, with a load average well over 10.... This is only a 3.5 Gb/s sustained link. We have about 2000 lines of intel (cert hash, file hash, domain) currently. This new addition would drive this up to ~35,000 lines of intel. We are trying to determine if this is practical given our current load on the box. Also, why does bro continuously chew ram up? When first started, bro eats about 80 gigs, then moves up through the day to about 120-175. However, if we leave it running for a few days, it ends up at the max of the memory allowed for the system... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160802/fdcf39d6/attachment.html From jazoff at illinois.edu Tue Aug 2 06:04:35 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 2 Aug 2016 13:04:35 +0000 Subject: [Bro] question about intel files In-Reply-To: <1610342229.9668660.1470141803800.JavaMail.yahoo@mail.yahoo.com> References: <1610342229.9668660.1470141803800.JavaMail.yahoo.ref@mail.yahoo.com> <1610342229.9668660.1470141803800.JavaMail.yahoo@mail.yahoo.com> Message-ID: > On Aug 2, 2016, at 8:43 AM, philosnef wrote: > > Are intel files loaded into memory or statically evaluated? We have a 7.4 meg intel file we are looking to push; however, out of 400 gigs of ram, we are using 400 gigs, with a load average well over 10.... This is only a 3.5 Gb/s sustained link. We have about 2000 lines of intel (cert hash, file hash, domain) currently. This new addition would drive this up to ~35,000 lines of intel. We are trying to determine if this is practical given our current load on the box. > > Also, why does bro continuously chew ram up? When first started, bro eats about 80 gigs, then moves up through the day to about 120-175. However, if we leave it running for a few days, it ends up at the max of the memory allowed for the system... What process is using memory? Workers? Proxies? Manager? If you can include the output of 'broctl top' that would be helpful. -- - Justin Azoff From seth at icir.org Tue Aug 2 07:22:46 2016 From: seth at icir.org (Seth Hall) Date: Tue, 2 Aug 2016 10:22:46 -0400 Subject: [Bro] question about intel files In-Reply-To: <1610342229.9668660.1470141803800.JavaMail.yahoo@mail.yahoo.com> References: <1610342229.9668660.1470141803800.JavaMail.yahoo.ref@mail.yahoo.com> <1610342229.9668660.1470141803800.JavaMail.yahoo@mail.yahoo.com> Message-ID: <5ECE6972-9029-41C7-BBB4-A1CD39624FD3@icir.org> > On Aug 2, 2016, at 8:43 AM, philosnef wrote: > > Are intel files loaded into memory or statically evaluated? It's loaded into memory. It's just using normal Bro data types which have some overhead. > We have about 2000 lines of intel (cert hash, file hash, domain) currently. This new addition would drive this up to ~35,000 lines of intel. We are trying to determine if this is practical given our current load on the box. Generally I would expect that amount of intelligence to be fine. It seems as though you may have some other trouble in your deployment though. > Also, why does bro continuously chew ram up? When first started, bro eats about 80 gigs, then moves up through the day to about 120-175.] How many workers are you running? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at broala.com Tue Aug 2 07:23:56 2016 From: seth at broala.com (Seth Hall) Date: Tue, 2 Aug 2016 10:23:56 -0400 Subject: [Bro] Changing timezone in log's timestamps In-Reply-To: <418105455.7992730.1470125409583.JavaMail.yahoo@mail.yahoo.com> References: <1238696709.8007446.1470124552618.JavaMail.yahoo.ref@mail.yahoo.com> <1238696709.8007446.1470124552618.JavaMail.yahoo@mail.yahoo.com> <418105455.7992730.1470125409583.JavaMail.yahoo@mail.yahoo.com> Message-ID: > On Aug 2, 2016, at 4:10 AM, Aneela Safdar wrote: > > What settings are to do in order to change the timezone of log files? Are you referring to the normal Bro ascii logs? If you are, those are written out in UTC (as UNIX epoch time always is) .Seth -- Seth Hall * Broala * seth at broala.com * www.broala.com From philosnef at yahoo.com Tue Aug 2 07:33:09 2016 From: philosnef at yahoo.com (philosnef) Date: Tue, 2 Aug 2016 14:33:09 +0000 (UTC) Subject: [Bro] question about intel files In-Reply-To: <5ECE6972-9029-41C7-BBB4-A1CD39624FD3@icir.org> References: <1610342229.9668660.1470141803800.JavaMail.yahoo.ref@mail.yahoo.com> <1610342229.9668660.1470141803800.JavaMail.yahoo@mail.yahoo.com> <5ECE6972-9029-41C7-BBB4-A1CD39624FD3@icir.org> Message-ID: <555731861.10052936.1470148389402.JavaMail.yahoo@mail.yahoo.com> We are running pfring with lb_procs=20. We have 40 cores on the box.? On Tuesday, August 2, 2016 10:22 AM, Seth Hall wrote: > On Aug 2, 2016, at 8:43 AM, philosnef wrote: > > Are intel files loaded into memory or statically evaluated? It's loaded into memory.? It's just using normal Bro data types which have some overhead. > We have about 2000 lines of intel (cert hash, file hash, domain) currently. This new addition would drive this up to ~35,000 lines of intel. We are trying to determine if this is practical given our current load on the box. Generally I would expect that amount of intelligence to be fine.? It seems as though you may have some other trouble in your deployment though. > Also, why does bro continuously chew ram up? When first started, bro eats about 80 gigs, then moves up through the day to about 120-175.] How many workers are you running? ? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160802/250a548c/attachment.html From seth at icir.org Tue Aug 2 20:11:08 2016 From: seth at icir.org (Seth Hall) Date: Tue, 2 Aug 2016 23:11:08 -0400 Subject: [Bro] question about intel files In-Reply-To: <555731861.10052936.1470148389402.JavaMail.yahoo@mail.yahoo.com> References: <1610342229.9668660.1470141803800.JavaMail.yahoo.ref@mail.yahoo.com> <1610342229.9668660.1470141803800.JavaMail.yahoo@mail.yahoo.com> <5ECE6972-9029-41C7-BBB4-A1CD39624FD3@icir.org> <555731861.10052936.1470148389402.JavaMail.yahoo@mail.yahoo.com> Message-ID: <76310CD9-93DF-445A-91C4-C9520F483E3C@icir.org> > On Aug 2, 2016, at 10:33 AM, philosnef wrote: > > We are running pfring with lb_procs=20. We have 40 cores on the box. Is that 40 cores with hyper threading or without? It's possible you're overwhelming the system if 20 of those cores are hyper threaded (this is definitely a guess though since there are so many things that could cause trouble). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From brot212 at googlemail.com Wed Aug 3 03:59:53 2016 From: brot212 at googlemail.com (Dane Wullen) Date: Wed, 3 Aug 2016 12:59:53 +0200 Subject: [Bro] Host Key Verification Failed Message-ID: Hello there, I've tried to install Bro on a Ubuntu 16.04 virtual machine (VirtualBox) with following guide: http://knowm.org/how-to-install-bro-network-security-monitor-on-ubuntu/ After the installation, I started broctl and tryped "install", but I reveiced an error message: Host key verification failed. Error: cannot create (some of the) directories /usr/local/bro,/usr/local/bro/logs,/usr/local/bro/spool,/usr/local/bro/spool/tmp on node bro I want to run Bro on a single machine (so no cluster at all), I checked the node.cfg, it looks like this: [bro] type=standalone host=localhost interface=eth0 Of course I installed a SSH Server (apt-get install openssh-server), and successfully connected to my VM with several divices. There was a common problem in this Mailing-List, but unfortunatly it remaind unsolved... http://mailman.icsi.berkeley.edu/pipermail/bro/2015-July/008697.html I'm new to linux and bro, also, english is not my native language, so please forgive me my faults. :) I would be glad to hear from you guys! Thanks alot! brot -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160803/05d785c5/attachment.html From philosnef at yahoo.com Wed Aug 3 04:22:08 2016 From: philosnef at yahoo.com (philosnef) Date: Wed, 3 Aug 2016 11:22:08 +0000 (UTC) Subject: [Bro] question about intel files In-Reply-To: <76310CD9-93DF-445A-91C4-C9520F483E3C@icir.org> References: <1610342229.9668660.1470141803800.JavaMail.yahoo.ref@mail.yahoo.com> <1610342229.9668660.1470141803800.JavaMail.yahoo@mail.yahoo.com> <5ECE6972-9029-41C7-BBB4-A1CD39624FD3@icir.org> <555731861.10052936.1470148389402.JavaMail.yahoo@mail.yahoo.com> <76310CD9-93DF-445A-91C4-C9520F483E3C@icir.org> Message-ID: <798158920.10439922.1470223328602.JavaMail.yahoo@mail.yahoo.com> We have 2 10 physical core systems with 20 logical cores for a total of 40. Bro has a capture loss of sub .5% across all workers, so it seems unlikely that the box is overloaded. The capture rate of the box, per pfring is about 3.5Gb/s. We reported memory issues in the past, but those were written off as not related to the memory leak recently patched in the 24 branch and the 25 branch. On Tuesday, August 2, 2016 11:11 PM, Seth Hall wrote: > On Aug 2, 2016, at 10:33 AM, philosnef wrote: > > We are running pfring with lb_procs=20. We have 40 cores on the box. Is that 40 cores with hyper threading or without?? It's possible you're overwhelming the system if 20 of those cores are hyper threaded (this is definitely a guess though since there are so many things that could cause trouble). ? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160803/0245ef93/attachment.html From jazoff at illinois.edu Wed Aug 3 06:42:54 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 3 Aug 2016 13:42:54 +0000 Subject: [Bro] question about intel files In-Reply-To: <798158920.10439922.1470223328602.JavaMail.yahoo@mail.yahoo.com> References: <1610342229.9668660.1470141803800.JavaMail.yahoo.ref@mail.yahoo.com> <1610342229.9668660.1470141803800.JavaMail.yahoo@mail.yahoo.com> <5ECE6972-9029-41C7-BBB4-A1CD39624FD3@icir.org> <555731861.10052936.1470148389402.JavaMail.yahoo@mail.yahoo.com> <76310CD9-93DF-445A-91C4-C9520F483E3C@icir.org> <798158920.10439922.1470223328602.JavaMail.yahoo@mail.yahoo.com> Message-ID: <0D5EEEF4-EB3C-4631-A130-B64D3F4EAF80@illinois.edu> > On Aug 3, 2016, at 7:22 AM, philosnef wrote: > > We have 2 10 physical core systems with 20 logical cores for a total of 40. Bro has a capture loss of sub .5% across all workers, so it seems unlikely that the box is overloaded. The capture rate of the box, per pfring is about 3.5Gb/s. We reported memory issues in the past, but those were written off as not related to the memory leak recently patched in the 24 branch and the 25 branch. What process is using memory? Workers? Proxies? Manager? If you can include the output of 'broctl top' that would be helpful. Otherwise it is pretty hard to determine what the issue may even be. If you have a dual 10 core system and are running 20 workers then that leaves no room for the manager or for any tasks like log rotation. For a 20 core system I would run at most 18 workers. -- - Justin Azoff From philosnef at yahoo.com Wed Aug 3 06:56:11 2016 From: philosnef at yahoo.com (philosnef) Date: Wed, 3 Aug 2016 13:56:11 +0000 (UTC) Subject: [Bro] question about intel files In-Reply-To: <0D5EEEF4-EB3C-4631-A130-B64D3F4EAF80@illinois.edu> References: <1610342229.9668660.1470141803800.JavaMail.yahoo.ref@mail.yahoo.com> <1610342229.9668660.1470141803800.JavaMail.yahoo@mail.yahoo.com> <5ECE6972-9029-41C7-BBB4-A1CD39624FD3@icir.org> <555731861.10052936.1470148389402.JavaMail.yahoo@mail.yahoo.com> <76310CD9-93DF-445A-91C4-C9520F483E3C@icir.org> <798158920.10439922.1470223328602.JavaMail.yahoo@mail.yahoo.com> <0D5EEEF4-EB3C-4631-A130-B64D3F4EAF80@illinois.edu> Message-ID: <1934841513.10998600.1470232571597.JavaMail.yahoo@mail.yahoo.com> With ?hyperthreading that's actually 40 cores, not 20. Running 20 workers with 40 cores available should be more than sufficient. At the time brotop was run, 355 out of 390 gigs of ram are in use. The only things running on this box are bro, and a splunk forwarder. The splunk forwarder is only using about 15 gigs of ram. This excessive memory consumption is on all of our bro boxes, no matter the input stream. Even on boxes only getting 500Mb/s, we see this memory creep until it is exhausted. At no point is oomkiller called however, so it is not exceeding available memory, just consuming all of the available memory. brotop---Name ? ? ? ? Type ? ?Host ? ? ? ? ? ? Pid ? ? Proc ? ?VSize ?Rss ?Cpu ? Cmdmanager ? ? ?manager localhost ? ? ? ?67408 ? parent ?884M ? 343M 136% ?bromanager ? ? ?manager localhost ? ? ? ?67442 ? child ? 346M ? 179M ?24% ?broproxy-1 ? ? ?proxy ? localhost ? ? ? ?67512 ? parent ?366M ? 284M ? 3% ?broproxy-1 ? ? ?proxy ? localhost ? ? ? ?67542 ? child ? 201M ? 114M ? 3% ?broproxy-2 ? ? ?proxy ? localhost ? ? ? ?67543 ? child ? 201M ? 107M ? 3% ?broproxy-2 ? ? ?proxy ? localhost ? ? ? ?67513 ? parent ?366M ? 284M ? 1% ?broworker-1-1 ? worker ?localhost ? ? ? ?67683 ? parent ? ?1G ? ? 1G 100% ?broworker-1-1 ? worker ?localhost ? ? ? ?68236 ? child ? 716M ? 625M ? 3% ?broworker-1-10 ?worker ?localhost ? ? ? ?67688 ? parent ? ?1G ? ? 1G ?96% ?broworker-1-10 ?worker ?localhost ? ? ? ?68278 ? child ? 716M ? 629M ? 1% ?broworker-1-11 ?worker ?localhost ? ? ? ?67697 ? parent ? ?2G ? ? 2G 100% ?broworker-1-11 ?worker ?localhost ? ? ? ?68229 ? child ? 716M ? 628M ? 0% ?broworker-1-12 ?worker ?localhost ? ? ? ?67712 ? parent ? ?1G ? ? 1G ?83% ?broworker-1-12 ?worker ?localhost ? ? ? ?68264 ? child ? 716M ? 629M ? 1% ?broworker-1-13 ?worker ?localhost ? ? ? ?67717 ? parent ? ?4G ? ? 4G 100% ?broworker-1-13 ?worker ?localhost ? ? ? ?68233 ? child ? 716M ? 627M ? 1% ?broworker-1-14 ?worker ?localhost ? ? ? ?67737 ? parent ? ?1G ? ? 1G ?98% ?broworker-1-14 ?worker ?localhost ? ? ? ?68223 ? child ? 716M ? 626M ? 1% ?broworker-1-15 ?worker ?localhost ? ? ? ?67752 ? parent ? ?2G ? ? 2G 100% ?broworker-1-15 ?worker ?localhost ? ? ? ?68269 ? child ? 716M ? 626M ? 0% ?broworker-1-16 ?worker ?localhost ? ? ? ?67749 ? parent ? ?1G ? ? 1G ?72% ?broworker-1-16 ?worker ?localhost ? ? ? ?68228 ? child ? 716M ? 630M ? 0% ?broworker-1-17 ?worker ?localhost ? ? ? ?67758 ? parent ? ?2G ? ? 2G ?87% ?broworker-1-17 ?worker ?localhost ? ? ? ?68263 ? child ? 716M ? 627M ? 1% ?broworker-1-18 ?worker ?localhost ? ? ? ?67764 ? parent ? ?1G ? ? 1G ?98% ?broworker-1-18 ?worker ?localhost ? ? ? ?68254 ? child ? 716M ? 626M ? 1% ?broworker-1-19 ?worker ?localhost ? ? ? ?67767 ? parent ? ?1G ? ? 1G ?66% ?broworker-1-19 ?worker ?localhost ? ? ? ?68239 ? child ? 716M ? 629M ? 0% ?broworker-1-2 ? worker ?localhost ? ? ? ?67774 ? parent ? ?1G ? ? 1G ?98% ?broworker-1-2 ? worker ?localhost ? ? ? ?68230 ? child ? 716M ? 625M ? 0% ?broworker-1-20 ?worker ?localhost ? ? ? ?67794 ? parent ? ?3G ? ? 3G ?98% ?broworker-1-20 ?worker ?localhost ? ? ? ?68245 ? child ? 716M ? 629M ? 3% ?broworker-1-3 ? worker ?localhost ? ? ? ?67792 ? parent ? ?1G ? ? 1G ?91% ?broworker-1-3 ? worker ?localhost ? ? ? ?68265 ? child ? 716M ? 627M ? 3% ?broworker-1-4 ? worker ?localhost ? ? ? ?67800 ? parent ? ?1G ? ? 1G ?83% ?broworker-1-4 ? worker ?localhost ? ? ? ?68248 ? child ? 716M ? 628M ? 1% ?broworker-1-5 ? worker ?localhost ? ? ? ?67799 ? parent ? ?1G ? ? 1G ?98% ?broworker-1-5 ? worker ?localhost ? ? ? ?68277 ? child ? 716M ? 626M ? 0% ?broworker-1-6 ? worker ?localhost ? ? ? ?67801 ? parent ? ?1G ? ? 1G ?85% ?broworker-1-6 ? worker ?localhost ? ? ? ?68279 ? child ? 716M ? 626M ? 1% ?broworker-1-7 ? worker ?localhost ? ? ? ?67813 ? parent ? ?1G ? ? 1G 100% ?broworker-1-7 ? worker ?localhost ? ? ? ?68251 ? child ? 716M ? 628M ? 1% ?broworker-1-8 ? worker ?localhost ? ? ? ?67812 ? parent ? ?1G ? ? 1G ?79% ?broworker-1-8 ? worker ?localhost ? ? ? ?68244 ? child ? 716M ? 629M ? 0% ?broworker-1-9 ? worker ?localhost ? ? ? ?67814 ? parent ? ?1G ? ? 1G ?96% ?broworker-1-9 ? worker ?localhost ? ? ? ?68266 ? child ? 716M ? 626M ? 1% ?bro On Wednesday, August 3, 2016 9:43 AM, "Azoff, Justin S" wrote: > On Aug 3, 2016, at 7:22 AM, philosnef wrote: > > We have 2 10 physical core systems with 20 logical cores for a total of 40. Bro has a capture loss of sub .5% across all workers, so it seems unlikely that the box is overloaded. The capture rate of the box, per pfring is about 3.5Gb/s. We reported memory issues in the past, but those were written off as not related to the memory leak recently patched in the 24 branch and the 25 branch. What process is using memory?? Workers? Proxies? Manager?? If you can include the output of 'broctl top' that would be helpful.? Otherwise it is pretty hard to determine what the issue may even be. If you have a dual 10 core system and are running 20 workers then that leaves no room for the manager or for any tasks like log rotation.? For a 20 core system I would run at most 18 workers. -- - Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160803/dfb40020/attachment-0001.html From seth at icir.org Wed Aug 3 06:57:27 2016 From: seth at icir.org (Seth Hall) Date: Wed, 3 Aug 2016 09:57:27 -0400 Subject: [Bro] question about intel files In-Reply-To: <798158920.10439922.1470223328602.JavaMail.yahoo@mail.yahoo.com> References: <1610342229.9668660.1470141803800.JavaMail.yahoo.ref@mail.yahoo.com> <1610342229.9668660.1470141803800.JavaMail.yahoo@mail.yahoo.com> <5ECE6972-9029-41C7-BBB4-A1CD39624FD3@icir.org> <555731861.10052936.1470148389402.JavaMail.yahoo@mail.yahoo.com> <76310CD9-93DF-445A-91C4-C9520F483E3C@icir.org> <798158920.10439922.1470223328602.JavaMail.yahoo@mail.yahoo.com> Message-ID: <47A7BA64-C91F-49C3-B323-E61EDD73A009@icir.org> > On Aug 3, 2016, at 7:22 AM, philosnef wrote: > > We have 2 10 physical core systems with 20 logical cores for a total of 40. Bro has a capture loss of sub .5% across all workers, so it seems unlikely that the box is overloaded. The capture rate of the box, per pfring is about 3.5Gb/s. We reported memory issues in the past, but those were written off as not related to the memory leak recently patched in the 24 branch and the 25 branch. Cool, thanks for the details. If you don't load your intelligence data, do you see any memory trouble? That seems like the next logical step to take for testing. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jazoff at illinois.edu Wed Aug 3 06:58:31 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 3 Aug 2016 13:58:31 +0000 Subject: [Bro] question about intel files In-Reply-To: <1934841513.10998600.1470232571597.JavaMail.yahoo@mail.yahoo.com> References: <1610342229.9668660.1470141803800.JavaMail.yahoo.ref@mail.yahoo.com> <1610342229.9668660.1470141803800.JavaMail.yahoo@mail.yahoo.com> <5ECE6972-9029-41C7-BBB4-A1CD39624FD3@icir.org> <555731861.10052936.1470148389402.JavaMail.yahoo@mail.yahoo.com> <76310CD9-93DF-445A-91C4-C9520F483E3C@icir.org> <798158920.10439922.1470223328602.JavaMail.yahoo@mail.yahoo.com> <0D5EEEF4-EB3C-4631-A130-B64D3F4EAF80@illinois.edu> <1934841513.10998600.1470232571597.JavaMail.yahoo@mail.yahoo.com> Message-ID: <5C02FAFF-C73C-47AD-AE97-7F1FB056B2E9@illinois.edu> > On Aug 3, 2016, at 9:56 AM, philosnef wrote: > > With hyperthreading that's actually 40 cores, not 20. Running 20 workers with 40 cores available should be more than sufficient. At the time brotop was run, 355 out of 390 gigs of ram are in use. The only things running on this box are bro, and a splunk forwarder. The splunk forwarder is only using about 15 gigs of ram. This excessive memory consumption is on all of our bro boxes, no matter the input stream. Even on boxes only getting 500Mb/s, we see this memory creep until it is exhausted. At no point is oomkiller called however, so it is not exceeding available memory, just consuming all of the available memory. Can you show the output of free -m -- - Justin Azoff From philosnef at yahoo.com Wed Aug 3 07:03:40 2016 From: philosnef at yahoo.com (philosnef) Date: Wed, 3 Aug 2016 14:03:40 +0000 (UTC) Subject: [Bro] question about intel files In-Reply-To: <5C02FAFF-C73C-47AD-AE97-7F1FB056B2E9@illinois.edu> References: <1610342229.9668660.1470141803800.JavaMail.yahoo.ref@mail.yahoo.com> <1610342229.9668660.1470141803800.JavaMail.yahoo@mail.yahoo.com> <5ECE6972-9029-41C7-BBB4-A1CD39624FD3@icir.org> <555731861.10052936.1470148389402.JavaMail.yahoo@mail.yahoo.com> <76310CD9-93DF-445A-91C4-C9520F483E3C@icir.org> <798158920.10439922.1470223328602.JavaMail.yahoo@mail.yahoo.com> <0D5EEEF4-EB3C-4631-A130-B64D3F4EAF80@illinois.edu> <1934841513.10998600.1470232571597.JavaMail.yahoo@mail.yahoo.com> <5C02FAFF-C73C-47AD-AE97-7F1FB056B2E9@illinois.edu> Message-ID: <1815715631.221972.1470233020637@mail.yahoo.com> ? ? ? ? ? ? ?total ? ? ? used ? ? ? free ? ? shared ? ?buffers ? ? cachedMem: ? ? ? ?371336 ? ? 340383 ? ? ?30952 ? ? ? ? ?0 ? ? ? ?300 ? ? 111823-/+ buffers/cache: ? ? 228259 ? ? 143076?Swap: ? ? ? ?15999 ? ? ? ?191 ? ? ?15808? On Wednesday, August 3, 2016 9:59 AM, "Azoff, Justin S" wrote: > On Aug 3, 2016, at 9:56 AM, philosnef wrote: > > With? hyperthreading that's actually 40 cores, not 20. Running 20 workers with 40 cores available should be more than sufficient. At the time brotop was run, 355 out of 390 gigs of ram are in use. The only things running on this box are bro, and a splunk forwarder. The splunk forwarder is only using about 15 gigs of ram. This excessive memory consumption is on all of our bro boxes, no matter the input stream. Even on boxes only getting 500Mb/s, we see this memory creep until it is exhausted. At no point is oomkiller called however, so it is not exceeding available memory, just consuming all of the available memory. Can you show the output of free -m -- - Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160803/4870c1b3/attachment.html From ansaf_130 at yahoo.com Wed Aug 3 07:19:22 2016 From: ansaf_130 at yahoo.com (Aneela Safdar) Date: Wed, 3 Aug 2016 14:19:22 +0000 (UTC) Subject: [Bro] Ignore_checksum causes weird.log to stop logging unusual login attempts References: <393368653.8849843.1470233962418.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <393368653.8849843.1470233962418.JavaMail.yahoo@mail.yahoo.com> Hi, I am monitoring weird.log file to look for unusual login attempts on different services running like SMB. But when I added ignore_checksum=T in local.bro weird.log stopped recording those login attempts. I am also in parallel reading ssh login requests which only logged by ssh.log if checksum is ignored. Is there a way I could log attempts on both SMB and SSH services? How can I make a separate file for SMB related requests just login attempts would be fine coz weird.log doesnot log usernames and other essential info related to attack. ssh.log file content, only logged when checksum is ignored: {"ts":"2016-08-03T13:37:44.054012Z","uid":"CftFQ54On2aEMWTxe2","id.orig_h":"192.168.227.102","id.orig_p":41146,"id.resp_h":"192.168.227.101","id.resp_p":22,"version":2,"client":"SSH-2.0-OpenSSH_5.0","server":"SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6","cipher_alg":"arcfour256","mac_alg":"hmac-md5","compression_alg":"none","kex_alg":"diffie-hellman-group-exchange-sha1","host_key_alg":"ssh-rsa","host_key":"8d:df:71:ac:29:1f:67:6f:f3:dd:c3:e5:2e:5f:3e:b4"} {"ts":"2016-08-03T13:37:46.403884Z","uid":"CiPQlY3yKBXpFNAZy7","id.orig_h":"192.168.227.102","id.orig_p":38431,"id.resp_h":"192.168.227.101","id.resp_p":22,"version":2,"client":"SSH-2.0-OpenSSH_5.0","server":"SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6","cipher_alg":"arcfour256","mac_alg":"hmac-md5","compression_alg":"none","kex_alg":"diffie-hellman-group-exchange-sha1","host_key_alg":"ssh-rsa","host_key":"8d:df:71:ac:29:1f:67:6f:f3:dd:c3:e5:2e:5f:3e:b4"} {"ts":"2016-08-03T13:37:53.591712Z","uid":"CrBgS9RnVLTqoJ0Ch","id.orig_h":"192.168.227.102","id.orig_p":42909,"id.resp_h":"192.168.227.101","id.resp_p":22,"version":2,"auth_success":true,"client":"SSH-2.0-OpenSSH_5.0","server":"SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6","cipher_alg":"arcfour256","mac_alg":"hmac-md5","compression_alg":"none","kex_alg":"diffie-hellman-group-exchange-sha1","host_key_alg":"ssh-rsa","host_key":"8d:df:71:ac:29:1f:67:6f:f3:dd:c3:e5:2e:5f:3e:b4"} {"ts":"2016-08-03T13:37:48.727616Z","uid":"Cl8KRP2oeWFBeEu1c8","id.orig_h":"192.168.227.102","id.orig_p":36868,"id.resp_h":"192.168.227.101","id.resp_p":22,"version":2,"client":"SSH-2.0-OpenSSH_5.0","server":"SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6","cipher_alg":"arcfour256","mac_alg":"hmac-md5","compression_alg":"none","kex_alg":"diffie-hellman-group-exchange-sha1","host_key_alg":"ssh-rsa","host_key":"8d:df:71:ac:29:1f:67:6f:f3:dd:c3:e5:2e:5f:3e:b4"} {"ts":"2016-08-03T13:37:51.030760Z","uid":"CihwTS2fBKkKOnLQmh","id.orig_h":"192.168.227.102","id.orig_p":34020,"id.resp_h":"192.168.227.101","id.resp_p":22,"version":2,"client":"SSH-2.0-OpenSSH_5.0","server":"SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6","cipher_alg":"arcfour256","mac_alg":"hmac-md5","compression_alg":"none","kex_alg":"diffie-hellman-group-exchange-sha1","host_key_alg":"ssh-rsa","host_key":"8d:df:71:ac:29:1f:67:6f:f3:dd:c3:e5:2e:5f:3e:b4"} {"ts":"2016-08-03T13:37:54.514701Z","uid":"Cy9JZh7rnAmkUopic","id.orig_h":"192.168.227.102","id.orig_p":46764,"id.resp_h":"192.168.227.101","id.resp_p":22,"version":2,"client":"SSH-2.0-OpenSSH_5.0","server":"SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6","cipher_alg":"arcfour256","mac_alg":"hmac-md5","compression_alg":"none","kex_alg":"diffie-hellman-group-exchange-sha1","host_key_alg":"ssh-rsa","host_key":"8d:df:71:ac:29:1f:67:6f:f3:dd:c3:e5:2e:5f:3e:b4"} {"ts":"2016-08-03T13:37:56.157141Z","uid":"CPlFiq1B98W54N2CHb","id.orig_h":"192.168.227.102","id.orig_p":39147,"id.resp_h":"192.168.227.101","id.resp_p":22,"version":2,"client":"SSH-2.0-OpenSSH_5.0","server":"SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6","cipher_alg":"arcfour256","mac_alg":"hmac-md5","compression_alg":"none","kex_alg":"diffie-hellman-group-exchange-sha1","host_key_alg":"ssh-rsa","host_key":"8d:df:71:ac:29:1f:67:6f:f3:dd:c3:e5:2e:5f:3e:b4"} {"ts":"2016-08-03T13:37:58.399253Z","uid":"CIUjNm1YN5VOCh2kMj","id.orig_h":"192.168.227.102","id.orig_p":33347,"id.resp_h":"192.168.227.101","id.resp_p":22,"version":2,"client":"SSH-2.0-OpenSSH_5.0","server":"SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6","cipher_alg":"arcfour256","mac_alg":"hmac-md5","compression_alg":"none","kex_alg":"diffie-hellman-group-exchange-sha1","host_key_alg":"ssh-rsa","host_key":"8d:df:71:ac:29:1f:67:6f:f3:dd:c3:e5:2e:5f:3e:b4"} weird.log file content for SMB service login attempts, logged when checksum is not ignored {"ts":"2016-08-03T12:58:27.310293Z","uid":"CX7tYC3dcJRhr7JHQf","id.orig_h":"192.168.227.102","id.orig_p":34040,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"data_before_established","notice":false,"peer":"bro"} {"ts":"2016-08-03T12:58:27.379358Z","uid":"CX7tYC3dcJRhr7JHQf","id.orig_h":"192.168.227.102","id.orig_p":34040,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"inappropriate_FIN","notice":false,"peer":"bro"} {"ts":"2016-08-03T12:58:27.383344Z","uid":"CU8OtK24mBy3xArCUf","id.orig_h":"192.168.227.102","id.orig_p":35751,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"data_before_established","notice":false,"peer":"bro"} {"ts":"2016-08-03T12:58:27.434387Z","uid":"CU8OtK24mBy3xArCUf","id.orig_h":"192.168.227.102","id.orig_p":35751,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"inappropriate_FIN","notice":false,"peer":"bro"} {"ts":"2016-08-03T12:58:27.437407Z","uid":"CJxYrM2ZDvbfXrOOMg","id.orig_h":"192.168.227.102","id.orig_p":37063,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"data_before_established","notice":false,"peer":"bro"} {"ts":"2016-08-03T12:58:27.493461Z","uid":"CJxYrM2ZDvbfXrOOMg","id.orig_h":"192.168.227.102","id.orig_p":37063,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"inappropriate_FIN","notice":false,"peer":"bro"} {"ts":"2016-08-03T12:58:27.496109Z","uid":"CmyLjl40RuIbPyIfGg","id.orig_h":"192.168.227.102","id.orig_p":37447,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"data_before_established","notice":false,"peer":"bro"} {"ts":"2016-08-03T12:58:27.560012Z","uid":"CmyLjl40RuIbPyIfGg","id.orig_h":"192.168.227.102","id.orig_p":37447,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"inappropriate_FIN","notice":false,"peer":"bro"} {"ts":"2016-08-03T12:58:27.567962Z","uid":"Cdkk3l4VSBL9hHfMyc","id.orig_h":"192.168.227.102","id.orig_p":38688,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"data_before_established","notice":false,"peer":"bro"} {"ts":"2016-08-03T12:58:27.629859Z","uid":"Cdkk3l4VSBL9hHfMyc","id.orig_h":"192.168.227.102","id.orig_p":38688,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"inappropriate_FIN","notice":false,"peer":"bro"} {"ts":"2016-08-03T12:58:27.633006Z","uid":"CWPSxs3IcmDxCuZlFc","id.orig_h":"192.168.227.102","id.orig_p":39016,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"data_before_established","notice":false,"peer":"bro"} {"ts":"2016-08-03T12:58:27.696545Z","uid":"CWPSxs3IcmDxCuZlFc","id.orig_h":"192.168.227.102","id.orig_p":39016,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"inappropriate_FIN","notice":false,"peer":"bro"} {"ts":"2016-08-03T12:58:27.712067Z","uid":"CVBsOs3XLLccpSJBZe","id.orig_h":"192.168.227.102","id.orig_p":42692,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"data_before_established","notice":false,"peer":"bro"} {"ts":"2016-08-03T12:58:27.803202Z","uid":"CVBsOs3XLLccpSJBZe","id.orig_h":"192.168.227.102","id.orig_p":42692,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"inappropriate_FIN","notice":false,"peer":"bro"} {"ts":"2016-08-03T12:58:27.805073Z","uid":"CJgitl2mjXr4YEnw3f","id.orig_h":"192.168.227.102","id.orig_p":42910,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"data_before_established","notice":false,"peer":"bro"} {"ts":"2016-08-03T12:58:27.871340Z","uid":"CJgitl2mjXr4YEnw3f","id.orig_h":"192.168.227.102","id.orig_p":42910,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"inappropriate_FIN","notice":false,"peer":"bro"} {"ts":"2016-08-03T12:58:27.896425Z","uid":"CAFrrn2rYMKZtslpVl","id.orig_h":"192.168.227.102","id.orig_p":35664,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"data_before_established","notice":false,"peer":"bro"} Thanks, Regards, Aneela Safdar -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160803/65ce53dc/attachment-0001.html From jazoff at illinois.edu Wed Aug 3 07:24:18 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 3 Aug 2016 14:24:18 +0000 Subject: [Bro] question about intel files In-Reply-To: <1815715631.221972.1470233020637@mail.yahoo.com> References: <1610342229.9668660.1470141803800.JavaMail.yahoo.ref@mail.yahoo.com> <1610342229.9668660.1470141803800.JavaMail.yahoo@mail.yahoo.com> <5ECE6972-9029-41C7-BBB4-A1CD39624FD3@icir.org> <555731861.10052936.1470148389402.JavaMail.yahoo@mail.yahoo.com> <76310CD9-93DF-445A-91C4-C9520F483E3C@icir.org> <798158920.10439922.1470223328602.JavaMail.yahoo@mail.yahoo.com> <0D5EEEF4-EB3C-4631-A130-B64D3F4EAF80@illinois.edu> <1934841513.10998600.1470232571597.JavaMail.yahoo@mail.yahoo.com> <5C02FAFF-C73C-47AD-AE97-7F1FB056B2E9@illinois.edu> <1815715631.221972.1470233020637@mail.yahoo.com> Message-ID: > On Aug 3, 2016, at 10:03 AM, philosnef wrote: > > total used free shared buffers cached > Mem: 371336 340383 30952 0 300 111823 > -/+ buffers/cache: 228259 143076 > Swap: 15999 191 15808 > > Ah, I think you have been looking at the wrong numbers. You are only using 228259M, (~222G, not 355G) 111823M is unallocated and currently used for buffer/disk cache. This amount will always grow until it ends up using almost all the 'free' memory on the machine. The reason why the OOM killer isn't killing anything is because you still have over 128G of ram free. I added up all the ram usage from the output of bro top, and adding some overhead for the rounded amounts measured in gigs, came to 56184M. Minus splunk, that does still leave about 150G unaccounted for. I believe some of that will be used by packet buffers in the kernel, depending on how you have configured pf_ring. But even at a huge 1G buffer for each of 20 workers (which I think is much much more than it uses by default) that is only another 20G. -- - Justin Azoff From philosnef at yahoo.com Wed Aug 3 07:33:25 2016 From: philosnef at yahoo.com (philosnef) Date: Wed, 3 Aug 2016 14:33:25 +0000 (UTC) Subject: [Bro] question about intel files In-Reply-To: References: <1610342229.9668660.1470141803800.JavaMail.yahoo.ref@mail.yahoo.com> <1610342229.9668660.1470141803800.JavaMail.yahoo@mail.yahoo.com> <5ECE6972-9029-41C7-BBB4-A1CD39624FD3@icir.org> <555731861.10052936.1470148389402.JavaMail.yahoo@mail.yahoo.com> <76310CD9-93DF-445A-91C4-C9520F483E3C@icir.org> <798158920.10439922.1470223328602.JavaMail.yahoo@mail.yahoo.com> <0D5EEEF4-EB3C-4631-A130-B64D3F4EAF80@illinois.edu> <1934841513.10998600.1470232571597.JavaMail.yahoo@mail.yahoo.com> <5C02FAFF-C73C-47AD-AE97-7F1FB056B2E9@illinois.edu> <1815715631.221972.1470233020637@mail.yahoo.com> Message-ID: <245006581.10288553.1470234805813.JavaMail.yahoo@mail.yahoo.com> This was about 2 hours after Bro was rebooted. Here is the output of a bro box with nearly identical throughput that has had bro up and running for the past 48 hours. As you can see, the buffer at this rate has shot up to 4g per parent. If I did not reboot the first box reported every 8 hours or so, we would see the same result there. $ free -m? ? ? ? ? ? ?total ? ? ? used ? ? ? free ? ? shared ? ?buffers ? ? cachedMem: ? ? ? ?387495 ? ? 375736 ? ? ?11758 ? ? ? ? ?0 ? ? ? 1073 ? ? 248216-/+ buffers/cache: ? ? 126446 ? ? 261048Swap: ? ? ? ?15999 ? ? ? ?267 ? ? ?15732$ /opt/bro/bin/broctl topwaiting for lock (owned by PID 71999) ...Name ? ? ? ? Type ? ?Host ? ? ? ? ? ? Pid ? ? Proc ? ?VSize ?Rss ?Cpu ? Cmdmanager ? ? ?manager localhost ? ? ? ?69286 ? parent ?853M ? 325M 108% ?bromanager ? ? ?manager localhost ? ? ? ?69313 ? child ? 381M ? 171M ?21% ?broproxy-1 ? ? ?proxy ? localhost ? ? ? ?69363 ? parent ? ?1G ? 845M ?34% ?broproxy-1 ? ? ?proxy ? localhost ? ? ? ?69394 ? child ? 215M ? 112M ? 3% ?broproxy-2 ? ? ?proxy ? localhost ? ? ? ?69391 ? child ? 210M ? ?94M ? 7% ?broproxy-2 ? ? ?proxy ? localhost ? ? ? ?69364 ? parent ?935M ? 829M ? 3% ?broworker-1-1 ? worker ?localhost ? ? ? ?69517 ? parent ? ?4G ? ? 4G ?79% ?broworker-1-1 ? worker ?localhost ? ? ? ?70127 ? child ? 712M ? 627M ? 1% ?broworker-1-10 ?worker ?localhost ? ? ? ?69526 ? parent ? ?4G ? ? 4G ?98% ?broworker-1-10 ?worker ?localhost ? ? ? ?70123 ? child ? 712M ? 626M ? 1% ?broworker-1-11 ?worker ?localhost ? ? ? ?69537 ? parent ? ?4G ? ? 4G ?83% ?broworker-1-11 ?worker ?localhost ? ? ? ?70095 ? child ? 712M ? 627M ? 1% ?broworker-1-12 ?worker ?localhost ? ? ? ?69545 ? parent ? ?4G ? ? 4G ?86% ?broworker-1-12 ?worker ?localhost ? ? ? ?70098 ? child ? 712M ? 628M ? 1% ?broworker-1-13 ?worker ?localhost ? ? ? ?69563 ? parent ? ?4G ? ? 4G ?92% ?broworker-1-13 ?worker ?localhost ? ? ? ?70027 ? child ? 712M ? 628M ? 1% ?broworker-1-14 ?worker ?localhost ? ? ? ?69564 ? parent ? ?4G ? ? 4G ?98% ?broworker-1-14 ?worker ?localhost ? ? ? ?70140 ? child ? 712M ? 626M ? 1% ?broworker-1-15 ?worker ?localhost ? ? ? ?69582 ? parent ? ?4G ? ? 4G ?98% ?broworker-1-15 ?worker ?localhost ? ? ? ?70143 ? child ? 712M ? 628M ? 1% ?broworker-1-16 ?worker ?localhost ? ? ? ?69577 ? parent ? ?4G ? ? 4G 100% ?broworker-1-16 ?worker ?localhost ? ? ? ?70125 ? child ? 712M ? 628M ? 0% ?broworker-1-17 ?worker ?localhost ? ? ? ?69595 ? parent ? ?4G ? ? 4G ?98% ?broworker-1-17 ?worker ?localhost ? ? ? ?70135 ? child ? 712M ? 629M ? 1% ?broworker-1-18 ?worker ?localhost ? ? ? ?69600 ? parent ? ?4G ? ? 4G ?79% ?broworker-1-18 ?worker ?localhost ? ? ? ?70141 ? child ? 712M ? 628M ? 0% ?broworker-1-19 ?worker ?localhost ? ? ? ?69618 ? parent ? ?4G ? ? 4G ?77% ?broworker-1-19 ?worker ?localhost ? ? ? ?70106 ? child ? 712M ? 624M ? 1% ?broworker-1-2 ? worker ?localhost ? ? ? ?69615 ? parent ? ?4G ? ? 4G ?79% ?broworker-1-2 ? worker ?localhost ? ? ? ?70138 ? child ? 712M ? 628M ? 0% ?broworker-1-20 ?worker ?localhost ? ? ? ?69620 ? parent ? ?4G ? ? 4G ?88% ?broworker-1-20 ?worker ?localhost ? ? ? ?70131 ? child ? 712M ? 628M ? 1% ?broworker-1-3 ? worker ?localhost ? ? ? ?69631 ? parent ? ?4G ? ? 4G ?81% ?broworker-1-3 ? worker ?localhost ? ? ? ?70025 ? child ? 712M ? 626M ? 1% ?broworker-1-4 ? worker ?localhost ? ? ? ?69639 ? parent ? ?4G ? ? 4G ?86% ?broworker-1-4 ? worker ?localhost ? ? ? ?70139 ? child ? 712M ? 628M ? 1% ?broworker-1-5 ? worker ?localhost ? ? ? ?69636 ? parent ? ?4G ? ? 4G ?98% ?broworker-1-5 ? worker ?localhost ? ? ? ?70108 ? child ? 712M ? 626M ? 1% ?broworker-1-6 ? worker ?localhost ? ? ? ?69646 ? parent ? ?4G ? ? 4G 100% ?broworker-1-6 ? worker ?localhost ? ? ? ?70107 ? child ? 712M ? 625M ? 1% ?broworker-1-7 ? worker ?localhost ? ? ? ?69647 ? parent ? ?4G ? ? 4G ?67% ?broworker-1-7 ? worker ?localhost ? ? ? ?70097 ? child ? 712M ? 622M ? 1% ?broworker-1-8 ? worker ?localhost ? ? ? ?69649 ? parent ? ?4G ? ? 4G ?84% ?broworker-1-8 ? worker ?localhost ? ? ? ?70026 ? child ? 712M ? 626M ? 1% ?broworker-1-9 ? worker ?localhost ? ? ? ?69651 ? parent ? ?4G ? ? 4G ?67% ?broworker-1-9 ? worker ?localhost ? ? ? ?70134 ? child ? 712M ? 628M ? 1% ?bro On Wednesday, August 3, 2016 10:24 AM, "Azoff, Justin S" wrote: > On Aug 3, 2016, at 10:03 AM, philosnef wrote: > >? ? ? ? ? ? ? total? ? ? used? ? ? free? ? shared? ? buffers? ? cached > Mem:? ? ? ? 371336? ? 340383? ? ? 30952? ? ? ? ? 0? ? ? ? 300? ? 111823 > -/+ buffers/cache:? ? 228259? ? 143076 > Swap:? ? ? ? 15999? ? ? ? 191? ? ? 15808 > > Ah, I think you have been looking at the wrong numbers. You are only using 228259M, (~222G, not 355G) 111823M is unallocated and currently used for buffer/disk cache. This amount will always grow until it ends up using almost all the 'free' memory on the machine. The reason why the OOM killer isn't killing anything is because you still have over 128G of ram free. I added up all the ram usage from the output of bro top, and adding some overhead for the rounded amounts measured in gigs, came to 56184M. Minus splunk, that does still leave about 150G unaccounted for. I believe some of that will be used by packet buffers in the kernel, depending on how you have configured pf_ring. But even at a huge 1G buffer for each of 20 workers (which I think is much much more than it uses by default) that is only another 20G. -- - Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160803/47f54425/attachment-0001.html From jazoff at illinois.edu Wed Aug 3 07:39:49 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 3 Aug 2016 14:39:49 +0000 Subject: [Bro] question about intel files In-Reply-To: <245006581.10288553.1470234805813.JavaMail.yahoo@mail.yahoo.com> References: <1610342229.9668660.1470141803800.JavaMail.yahoo.ref@mail.yahoo.com> <1610342229.9668660.1470141803800.JavaMail.yahoo@mail.yahoo.com> <5ECE6972-9029-41C7-BBB4-A1CD39624FD3@icir.org> <555731861.10052936.1470148389402.JavaMail.yahoo@mail.yahoo.com> <76310CD9-93DF-445A-91C4-C9520F483E3C@icir.org> <798158920.10439922.1470223328602.JavaMail.yahoo@mail.yahoo.com> <0D5EEEF4-EB3C-4631-A130-B64D3F4EAF80@illinois.edu> <1934841513.10998600.1470232571597.JavaMail.yahoo@mail.yahoo.com> <5C02FAFF-C73C-47AD-AE97-7F1FB056B2E9@illinois.edu> <1815715631.221972.1470233020637@mail.yahoo.com> <245006581.10288553.1470234805813.JavaMail.yahoo@mail.yahoo.com> Message-ID: <75844ADE-4F86-4447-B622-68447FF3649E@illinois.edu> > On Aug 3, 2016, at 10:33 AM, philosnef wrote: > > This was about 2 hours after Bro was rebooted. Here is the output of a bro box with nearly identical throughput that has had bro up and running for the past 48 hours. As you can see, the buffer at this rate has shot up to 4g per parent. If I did not reboot the first box reported every 8 hours or so, we would see the same result there. > > $ free -m > total used free shared buffers cached > Mem: 387495 375736 11758 0 1073 248216 > -/+ buffers/cache: 126446 261048 > Swap: 15999 267 15732 This box has 256G of ram free. I'm sorry but I just don't see where you have a problem here. -- - Justin Azoff From philosnef at yahoo.com Wed Aug 3 07:42:22 2016 From: philosnef at yahoo.com (philosnef) Date: Wed, 3 Aug 2016 14:42:22 +0000 (UTC) Subject: [Bro] question about intel files In-Reply-To: <75844ADE-4F86-4447-B622-68447FF3649E@illinois.edu> References: <1610342229.9668660.1470141803800.JavaMail.yahoo.ref@mail.yahoo.com> <1610342229.9668660.1470141803800.JavaMail.yahoo@mail.yahoo.com> <5ECE6972-9029-41C7-BBB4-A1CD39624FD3@icir.org> <555731861.10052936.1470148389402.JavaMail.yahoo@mail.yahoo.com> <76310CD9-93DF-445A-91C4-C9520F483E3C@icir.org> <798158920.10439922.1470223328602.JavaMail.yahoo@mail.yahoo.com> <0D5EEEF4-EB3C-4631-A130-B64D3F4EAF80@illinois.edu> <1934841513.10998600.1470232571597.JavaMail.yahoo@mail.yahoo.com> <5C02FAFF-C73C-47AD-AE97-7F1FB056B2E9@illinois.edu> <1815715631.221972.1470233020637@mail.yahoo.com> <245006581.10288553.1470234805813.JavaMail.yahoo@mail.yahoo.com> <75844ADE-4F86-4447-B622-68447FF3649E@illinois.edu> Message-ID: <1736110784.10393815.1470235342910.JavaMail.yahoo@mail.yahoo.com> Because, on boxes where we arent consistently rebooting bro, we are having oomkiller nuking splunk and bro. On Wednesday, August 3, 2016 10:39 AM, "Azoff, Justin S" wrote: > On Aug 3, 2016, at 10:33 AM, philosnef wrote: > > This was about 2 hours after Bro was rebooted. Here is the output of a bro box with nearly identical throughput that has had bro up and running for the past 48 hours. As you can see, the buffer at this rate has shot up to 4g per parent. If I did not reboot the first box reported every 8 hours or so, we would see the same result there. > > $ free -m >? ? ? ? ? ? ? total? ? ? used? ? ? free? ? shared? ? buffers? ? cached > Mem:? ? ? ? 387495? ? 375736? ? ? 11758? ? ? ? ? 0? ? ? 1073? ? 248216 > -/+ buffers/cache:? ? 126446? ? 261048 > Swap:? ? ? ? 15999? ? ? ? 267? ? ? 15732 This box has 256G of ram free. I'm sorry but I just don't see where you have a problem here. -- - Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160803/73865e17/attachment.html From jazoff at illinois.edu Wed Aug 3 07:51:04 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 3 Aug 2016 14:51:04 +0000 Subject: [Bro] question about intel files In-Reply-To: <1736110784.10393815.1470235342910.JavaMail.yahoo@mail.yahoo.com> References: <1610342229.9668660.1470141803800.JavaMail.yahoo.ref@mail.yahoo.com> <1610342229.9668660.1470141803800.JavaMail.yahoo@mail.yahoo.com> <5ECE6972-9029-41C7-BBB4-A1CD39624FD3@icir.org> <555731861.10052936.1470148389402.JavaMail.yahoo@mail.yahoo.com> <76310CD9-93DF-445A-91C4-C9520F483E3C@icir.org> <798158920.10439922.1470223328602.JavaMail.yahoo@mail.yahoo.com> <0D5EEEF4-EB3C-4631-A130-B64D3F4EAF80@illinois.edu> <1934841513.10998600.1470232571597.JavaMail.yahoo@mail.yahoo.com> <5C02FAFF-C73C-47AD-AE97-7F1FB056B2E9@illinois.edu> <1815715631.221972.1470233020637@mail.yahoo.com> <245006581.10288553.1470234805813.JavaMail.yahoo@mail.yahoo.com> <75844ADE-4F86-4447-B622-68447FF3649E@illinois.edu> <1736110784.10393815.1470235342910.JavaMail.yahoo@mail.yahoo.com> Message-ID: <2C7540E8-0005-4663-91ED-BF851D33EAEF@illinois.edu> > On Aug 3, 2016, at 10:42 AM, philosnef wrote: > > Because, on boxes where we arent consistently rebooting bro, we are having oomkiller nuking splunk and bro. > Ok.. because before you said "At no point is oomkiller called" I'm assuming that you have a cron job or something running broctl restart every 8 hours. Can you add a script that does this, once per hour or so (and set to run at a particular minute so it runs before the job that restarts bro runs) date free -m top -a -b -n 1 broctl top and sends that to a file, then show us what that says after a day or so? If you've been showing us system information from immediately after bro is restarted and not while the problem is occurring then that data isn't very useful. -- - Justin Azoff From dnthayer at illinois.edu Wed Aug 3 08:03:08 2016 From: dnthayer at illinois.edu (Daniel Thayer) Date: Wed, 3 Aug 2016 10:03:08 -0500 Subject: [Bro] Host Key Verification Failed In-Reply-To: References: Message-ID: <737daada-57db-afc4-b71d-d27d53af8633@illinois.edu> Could you try running this command: LANG=C broctl install Let me know if that works or not. On 8/3/16 5:59 AM, Dane Wullen wrote: > Hello there, > > I've tried to install Bro on a Ubuntu 16.04 virtual machine (VirtualBox) > with following guide: > > http://knowm.org/how-to-install-bro-network-security-monitor-on-ubuntu/ > > > After the installation, I started broctl and tryped "install", but I > reveiced an error message: > > Host key verification failed. > Error: cannot create (some of the) directories /usr/local/bro,/usr/local/bro/logs,/usr/local/bro/spool,/usr/local/bro/spool/tmp on node bro > > I want to run Bro on a single machine (so no cluster at all), I checked the node.cfg, it looks like this: > > [bro] > type=standalone > host=localhost > interface=eth0 > > Of course I installed a SSH Server (apt-get install openssh-server), and > successfully connected to my VM with several divices. > > There was a common problem in this Mailing-List, but unfortunatly it > remaind unsolved... > > http://mailman.icsi.berkeley.edu/pipermail/bro/2015-July/008697.html > > > I'm new to linux and bro, also, english is not my native language, so > please forgive me my faults. :) > > I would be glad to hear from you guys! > > Thanks alot! > > brot > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From jdopheid at illinois.edu Wed Aug 3 08:49:48 2016 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Wed, 3 Aug 2016 15:49:48 +0000 Subject: [Bro] Reminder to book BroCon travel, hotel Message-ID: <379EA04F-64BF-4BA3-945D-0B28726A050C@illinois.edu> Bro Community, If you?re planning on attending BroCon this year, consider taking advantage of our reserved room block at the Lone Star Court before August 22nd, details here: https://www.bro.org/community/brocon2016.html#hotelinformation And don?t forget to use the group code even if you?re booking over the phone. We need the community?s help to meet our hotel contract obligations to continue to negotiate special rates in the future. Thanks, The Bro Team ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From brot212 at googlemail.com Wed Aug 3 09:39:17 2016 From: brot212 at googlemail.com (Dane Wullen) Date: Wed, 3 Aug 2016 18:39:17 +0200 Subject: [Bro] Host Key Verification Failed In-Reply-To: <737daada-57db-afc4-b71d-d27d53af8633@illinois.edu> References: <737daada-57db-afc4-b71d-d27d53af8633@illinois.edu> Message-ID: Hey Daniel, well it worked. I was able to submit the command "install", "start" and "stop", but everytime with LANG=C. How can I avoid that I have to type in LANG=C all the time? Could you explain me what I did wrong or what the command "LANG=C" does? Thanks alot. :) Dane Am 03.08.2016 um 17:03 schrieb Daniel Thayer: > Could you try running this command: > LANG=C broctl install > > Let me know if that works or not. > > > On 8/3/16 5:59 AM, Dane Wullen wrote: >> Hello there, >> >> I've tried to install Bro on a Ubuntu 16.04 virtual machine (VirtualBox) >> with following guide: >> >> http://knowm.org/how-to-install-bro-network-security-monitor-on-ubuntu/ >> >> >> >> After the installation, I started broctl and tryped "install", but I >> reveiced an error message: >> >> Host key verification failed. >> Error: cannot create (some of the) directories >> /usr/local/bro,/usr/local/bro/logs,/usr/local/bro/spool,/usr/local/bro/spool/tmp >> on node bro >> >> I want to run Bro on a single machine (so no cluster at all), I >> checked the node.cfg, it looks like this: >> >> [bro] >> type=standalone >> host=localhost >> interface=eth0 >> >> Of course I installed a SSH Server (apt-get install openssh-server), and >> successfully connected to my VM with several divices. >> >> There was a common problem in this Mailing-List, but unfortunatly it >> remaind unsolved... >> >> http://mailman.icsi.berkeley.edu/pipermail/bro/2015-July/008697.html >> >> >> >> I'm new to linux and bro, also, english is not my native language, so >> please forgive me my faults. :) >> >> I would be glad to hear from you guys! >> >> Thanks alot! >> >> brot >> >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> From dnthayer at illinois.edu Wed Aug 3 10:00:18 2016 From: dnthayer at illinois.edu (Daniel Thayer) Date: Wed, 3 Aug 2016 12:00:18 -0500 Subject: [Bro] Host Key Verification Failed In-Reply-To: References: <737daada-57db-afc4-b71d-d27d53af8633@illinois.edu> Message-ID: <570a9dae-68f7-db52-0539-9194aa98b182@illinois.edu> BroControl runs the "ifconfig" command, and then tries to read IP addresses from the output. This might fail depending on which locale your system is configured to use. Here are two simple workarounds: 1) create a shell script wrapper that sets LANG and runs broctl, or 2) patch the broctl source to set LANG when it runs ifconfig (to do this, edit $PREFIX/lib/broctl/BroControl/execute.py, where $PREFIX is your bro install prefix directory, and then look for PATH, and add LANG=C right before the PATH=...) These workarounds won't be needed for the next Bro release. On 8/3/16 11:39 AM, Dane Wullen wrote: > Hey Daniel, > > well it worked. I was able to submit the command "install", "start" and > "stop", but everytime with LANG=C. How can I avoid that I have to type > in LANG=C all the time? > > Could you explain me what I did wrong or what the command "LANG=C" does? > Thanks alot. :) > > Dane > > > Am 03.08.2016 um 17:03 schrieb Daniel Thayer: >> Could you try running this command: >> LANG=C broctl install >> >> Let me know if that works or not. >> >> >> On 8/3/16 5:59 AM, Dane Wullen wrote: >>> Hello there, >>> >>> I've tried to install Bro on a Ubuntu 16.04 virtual machine (VirtualBox) >>> with following guide: >>> >>> http://knowm.org/how-to-install-bro-network-security-monitor-on-ubuntu/ >>> >>> >>> >>> After the installation, I started broctl and tryped "install", but I >>> reveiced an error message: >>> >>> Host key verification failed. >>> Error: cannot create (some of the) directories >>> /usr/local/bro,/usr/local/bro/logs,/usr/local/bro/spool,/usr/local/bro/spool/tmp >>> on node bro >>> >>> I want to run Bro on a single machine (so no cluster at all), I >>> checked the node.cfg, it looks like this: >>> >>> [bro] >>> type=standalone >>> host=localhost >>> interface=eth0 >>> >>> Of course I installed a SSH Server (apt-get install openssh-server), and >>> successfully connected to my VM with several divices. >>> >>> There was a common problem in this Mailing-List, but unfortunatly it >>> remaind unsolved... >>> >>> http://mailman.icsi.berkeley.edu/pipermail/bro/2015-July/008697.html >>> >>> >>> >>> I'm new to linux and bro, also, english is not my native language, so >>> please forgive me my faults. :) >>> >>> I would be glad to hear from you guys! >>> >>> Thanks alot! >>> >>> brot >>> >>> >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> > From jwc3f at virginia.edu Wed Aug 3 11:22:34 2016 From: jwc3f at virginia.edu (Collyer, Jeffrey W. (jwc3f)) Date: Wed, 3 Aug 2016 18:22:34 +0000 Subject: [Bro] BroCon rideshare? Message-ID: In booking my travel, I queried the hotel about travel to/from TACC. They say its a good 20 minute walk. Googling the weather in Austin mid September, but it seems that 90s and Thunderstorms are reasonably normal. With that in mind - is there a rideshare? or does someone know a clever way/app to organize one. It seems a shame for everyone to rent a car for such a short daily ride. Jeffrey Collyer -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160803/9d5ea7db/attachment.html From daniel.manzo at bayer.com Wed Aug 3 11:37:03 2016 From: daniel.manzo at bayer.com (Daniel Manzo) Date: Wed, 3 Aug 2016 18:37:03 +0000 Subject: [Bro] Network taps for Bro Message-ID: <2C7473428EFB4348960ACC47FDC529457F4F38@MOXCXR.na.bayer.cnb> Hi all, My team is looking into using the Bro IDS for monitoring of a science DMZ with a 10 Gbps network. I was wondering how to choose which network tap(s) is necessary for this type of connection and if you have any recommendations/methods for setting up the hardware for Bro. I have been looking at the passive Ixia Flex taps, specifically the LC 10G SM 50/50 split tap. Will single mode (SM) versus multi-mode (MM) make a difference for Bro? And does Bro require a 50/50 ratio, or would I be able to get away with a different ratio? Thanks for the help, Daniel Manzo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160803/f450dca9/attachment.html From eyrich at illinois.edu Wed Aug 3 12:02:06 2016 From: eyrich at illinois.edu (James Eyrich) Date: Wed, 3 Aug 2016 14:02:06 -0500 Subject: [Bro] Network taps for Bro In-Reply-To: <2C7473428EFB4348960ACC47FDC529457F4F38@MOXCXR.na.bayer.cnb> References: <2C7473428EFB4348960ACC47FDC529457F4F38@MOXCXR.na.bayer.cnb> Message-ID: Bro doesnt care about any of that. The optics going into your tap aggregator or direct into to the bro nodes need to match what ever you are using for the connection same for the splitter regarding splitter ratios - it depends what your light budget regarding the receive sensitivity on the ends of the actual connection and the optics feeding the bro system Off the top of my head I was thinking 50/50 is good for data center and 70/30 for WAN if you are running out of light once the splitter is in place you might have to move to higher powered optics all around. One thing we ran into is some of the "lite" optics for use in data centers also have reduced sensitivity in addition to lower send power. On 8/3/2016 1:37 PM, Daniel Manzo wrote: > > Hi all, > > > > My team is looking into using the Bro IDS for monitoring of a science > DMZ with a 10 Gbps network. I was wondering how to choose which > network tap(s) is necessary for this type of connection and if you > have any recommendations/methods for setting up the hardware for Bro. > I have been looking at the passive Ixia Flex taps, specifically the LC > 10G SM 50/50 split tap. Will single mode (SM) versus multi-mode (MM) > make a difference for Bro? And does Bro require a 50/50 ratio, or > would I be able to get away with a different ratio? > > > > Thanks for the help, > > Daniel Manzo > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- ---------------------------------------- James Eyrich Manager - Incident Response and Security National Center for Supercomputer Applications University of Illinois at Urbana-Champaign eyrich at illinois.edu 217-265-6867 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160803/36dee5a4/attachment.html From gfaulkner.nsm at gmail.com Wed Aug 3 12:29:13 2016 From: gfaulkner.nsm at gmail.com (Gary Faulkner) Date: Wed, 3 Aug 2016 14:29:13 -0500 Subject: [Bro] Network taps for Bro In-Reply-To: References: <2C7473428EFB4348960ACC47FDC529457F4F38@MOXCXR.na.bayer.cnb> Message-ID: Another thing to consider is if it is a single 10G connection you may be able to go right to the bro box from the tap, but if you have multiple 10G connections, or need to send the signal to monitoring tools on multiple boxes you may also need to look into a tap aggregator/ load-balancer as well. If the connection is running on a specific CWDM/DWDM wavelength you may also need to check that your NICs and/or tap aggregator support the proper optics as not all do. ~Gary On 8/3/16 2:02 PM, James Eyrich wrote: > Bro doesnt care about any of that. > The optics going into your tap aggregator or direct into to the bro > nodes need to match what ever you are using for the connection > same for the splitter > regarding splitter ratios - it depends what your light budget regarding > the receive sensitivity on the ends of the actual connection and the > optics feeding the bro system > Off the top of my head I was thinking 50/50 is good for data center and > 70/30 for WAN > if you are running out of light once the splitter is in place you might > have to move to higher powered optics all around. > > One thing we ran into is some of the "lite" optics for use in data > centers also have reduced sensitivity in addition to lower send power. > > On 8/3/2016 1:37 PM, Daniel Manzo wrote: >> Hi all, >> >> >> >> My team is looking into using the Bro IDS for monitoring of a science >> DMZ with a 10 Gbps network. I was wondering how to choose which >> network tap(s) is necessary for this type of connection and if you >> have any recommendations/methods for setting up the hardware for Bro. >> I have been looking at the passive Ixia Flex taps, specifically the LC >> 10G SM 50/50 split tap. Will single mode (SM) versus multi-mode (MM) >> make a difference for Bro? And does Bro require a 50/50 ratio, or >> would I be able to get away with a different ratio? >> >> >> >> Thanks for the help, >> >> Daniel Manzo >> >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160803/49693e47/attachment.html From johanna at icir.org Wed Aug 3 12:39:25 2016 From: johanna at icir.org (Johanna Amann) Date: Wed, 3 Aug 2016 12:39:25 -0700 Subject: [Bro] Ignore_checksum causes weird.log to stop logging unusual login attempts In-Reply-To: <393368653.8849843.1470233962418.JavaMail.yahoo@mail.yahoo.com> References: <393368653.8849843.1470233962418.JavaMail.yahoo.ref@mail.yahoo.com> <393368653.8849843.1470233962418.JavaMail.yahoo@mail.yahoo.com> Message-ID: <20160803193925.GA6821@wifi154.sys.ICSI.Berkeley.EDU> Hi Aneela, weird.log is not really the place to look for unusual login attemps; in this case, all the messages you see are caused by problems of the TCP traffic that Bro sees; due to issues with your NIC, the checksums are not correct, Bro discards packets with incorrect checksums, and the remaining traffic looks broken. When you set ignore_checksum to true, Bro ignores broken checksums and sees all the traffic - which makes the weirds that were reported because of TCP oddities go away. You will still find these connections in conn.log. There is no smb.log (or similar), because Bro currently does not ship with a working SMB analyzer; however, we are working on this and an SMB analyzer should be merged into the Bro master within the next weeks (the branch is accessible on git). Depending on your traffic, this might then log the information that you want. I hope this helps, Johanna On Wed, Aug 03, 2016 at 02:19:22PM +0000, Aneela Safdar wrote: > Hi, > I am monitoring weird.log file to look for unusual login attempts on different services running like SMB. But when I added ignore_checksum=T in local.bro weird.log stopped recording those login attempts. I am also in parallel reading ssh login requests which only logged by ssh.log if checksum is ignored. > Is there a way I could log attempts on both SMB and SSH services? How can I make a separate file for SMB related requests just login attempts would be fine coz weird.log doesnot log usernames and other essential info related to attack. > ssh.log file content, only logged when checksum is ignored: > {"ts":"2016-08-03T13:37:44.054012Z","uid":"CftFQ54On2aEMWTxe2","id.orig_h":"192.168.227.102","id.orig_p":41146,"id.resp_h":"192.168.227.101","id.resp_p":22,"version":2,"client":"SSH-2.0-OpenSSH_5.0","server":"SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6","cipher_alg":"arcfour256","mac_alg":"hmac-md5","compression_alg":"none","kex_alg":"diffie-hellman-group-exchange-sha1","host_key_alg":"ssh-rsa","host_key":"8d:df:71:ac:29:1f:67:6f:f3:dd:c3:e5:2e:5f:3e:b4"} > {"ts":"2016-08-03T13:37:46.403884Z","uid":"CiPQlY3yKBXpFNAZy7","id.orig_h":"192.168.227.102","id.orig_p":38431,"id.resp_h":"192.168.227.101","id.resp_p":22,"version":2,"client":"SSH-2.0-OpenSSH_5.0","server":"SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6","cipher_alg":"arcfour256","mac_alg":"hmac-md5","compression_alg":"none","kex_alg":"diffie-hellman-group-exchange-sha1","host_key_alg":"ssh-rsa","host_key":"8d:df:71:ac:29:1f:67:6f:f3:dd:c3:e5:2e:5f:3e:b4"} > {"ts":"2016-08-03T13:37:53.591712Z","uid":"CrBgS9RnVLTqoJ0Ch","id.orig_h":"192.168.227.102","id.orig_p":42909,"id.resp_h":"192.168.227.101","id.resp_p":22,"version":2,"auth_success":true,"client":"SSH-2.0-OpenSSH_5.0","server":"SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6","cipher_alg":"arcfour256","mac_alg":"hmac-md5","compression_alg":"none","kex_alg":"diffie-hellman-group-exchange-sha1","host_key_alg":"ssh-rsa","host_key":"8d:df:71:ac:29:1f:67:6f:f3:dd:c3:e5:2e:5f:3e:b4"} > {"ts":"2016-08-03T13:37:48.727616Z","uid":"Cl8KRP2oeWFBeEu1c8","id.orig_h":"192.168.227.102","id.orig_p":36868,"id.resp_h":"192.168.227.101","id.resp_p":22,"version":2,"client":"SSH-2.0-OpenSSH_5.0","server":"SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6","cipher_alg":"arcfour256","mac_alg":"hmac-md5","compression_alg":"none","kex_alg":"diffie-hellman-group-exchange-sha1","host_key_alg":"ssh-rsa","host_key":"8d:df:71:ac:29:1f:67:6f:f3:dd:c3:e5:2e:5f:3e:b4"} > {"ts":"2016-08-03T13:37:51.030760Z","uid":"CihwTS2fBKkKOnLQmh","id.orig_h":"192.168.227.102","id.orig_p":34020,"id.resp_h":"192.168.227.101","id.resp_p":22,"version":2,"client":"SSH-2.0-OpenSSH_5.0","server":"SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6","cipher_alg":"arcfour256","mac_alg":"hmac-md5","compression_alg":"none","kex_alg":"diffie-hellman-group-exchange-sha1","host_key_alg":"ssh-rsa","host_key":"8d:df:71:ac:29:1f:67:6f:f3:dd:c3:e5:2e:5f:3e:b4"} > {"ts":"2016-08-03T13:37:54.514701Z","uid":"Cy9JZh7rnAmkUopic","id.orig_h":"192.168.227.102","id.orig_p":46764,"id.resp_h":"192.168.227.101","id.resp_p":22,"version":2,"client":"SSH-2.0-OpenSSH_5.0","server":"SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6","cipher_alg":"arcfour256","mac_alg":"hmac-md5","compression_alg":"none","kex_alg":"diffie-hellman-group-exchange-sha1","host_key_alg":"ssh-rsa","host_key":"8d:df:71:ac:29:1f:67:6f:f3:dd:c3:e5:2e:5f:3e:b4"} > {"ts":"2016-08-03T13:37:56.157141Z","uid":"CPlFiq1B98W54N2CHb","id.orig_h":"192.168.227.102","id.orig_p":39147,"id.resp_h":"192.168.227.101","id.resp_p":22,"version":2,"client":"SSH-2.0-OpenSSH_5.0","server":"SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6","cipher_alg":"arcfour256","mac_alg":"hmac-md5","compression_alg":"none","kex_alg":"diffie-hellman-group-exchange-sha1","host_key_alg":"ssh-rsa","host_key":"8d:df:71:ac:29:1f:67:6f:f3:dd:c3:e5:2e:5f:3e:b4"} > {"ts":"2016-08-03T13:37:58.399253Z","uid":"CIUjNm1YN5VOCh2kMj","id.orig_h":"192.168.227.102","id.orig_p":33347,"id.resp_h":"192.168.227.101","id.resp_p":22,"version":2,"client":"SSH-2.0-OpenSSH_5.0","server":"SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6","cipher_alg":"arcfour256","mac_alg":"hmac-md5","compression_alg":"none","kex_alg":"diffie-hellman-group-exchange-sha1","host_key_alg":"ssh-rsa","host_key":"8d:df:71:ac:29:1f:67:6f:f3:dd:c3:e5:2e:5f:3e:b4"} > > weird.log file content for SMB service login attempts, logged when checksum is not ignored > {"ts":"2016-08-03T12:58:27.310293Z","uid":"CX7tYC3dcJRhr7JHQf","id.orig_h":"192.168.227.102","id.orig_p":34040,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"data_before_established","notice":false,"peer":"bro"} > {"ts":"2016-08-03T12:58:27.379358Z","uid":"CX7tYC3dcJRhr7JHQf","id.orig_h":"192.168.227.102","id.orig_p":34040,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"inappropriate_FIN","notice":false,"peer":"bro"} > {"ts":"2016-08-03T12:58:27.383344Z","uid":"CU8OtK24mBy3xArCUf","id.orig_h":"192.168.227.102","id.orig_p":35751,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"data_before_established","notice":false,"peer":"bro"} > {"ts":"2016-08-03T12:58:27.434387Z","uid":"CU8OtK24mBy3xArCUf","id.orig_h":"192.168.227.102","id.orig_p":35751,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"inappropriate_FIN","notice":false,"peer":"bro"} > {"ts":"2016-08-03T12:58:27.437407Z","uid":"CJxYrM2ZDvbfXrOOMg","id.orig_h":"192.168.227.102","id.orig_p":37063,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"data_before_established","notice":false,"peer":"bro"} > {"ts":"2016-08-03T12:58:27.493461Z","uid":"CJxYrM2ZDvbfXrOOMg","id.orig_h":"192.168.227.102","id.orig_p":37063,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"inappropriate_FIN","notice":false,"peer":"bro"} > {"ts":"2016-08-03T12:58:27.496109Z","uid":"CmyLjl40RuIbPyIfGg","id.orig_h":"192.168.227.102","id.orig_p":37447,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"data_before_established","notice":false,"peer":"bro"} > {"ts":"2016-08-03T12:58:27.560012Z","uid":"CmyLjl40RuIbPyIfGg","id.orig_h":"192.168.227.102","id.orig_p":37447,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"inappropriate_FIN","notice":false,"peer":"bro"} > {"ts":"2016-08-03T12:58:27.567962Z","uid":"Cdkk3l4VSBL9hHfMyc","id.orig_h":"192.168.227.102","id.orig_p":38688,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"data_before_established","notice":false,"peer":"bro"} > {"ts":"2016-08-03T12:58:27.629859Z","uid":"Cdkk3l4VSBL9hHfMyc","id.orig_h":"192.168.227.102","id.orig_p":38688,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"inappropriate_FIN","notice":false,"peer":"bro"} > {"ts":"2016-08-03T12:58:27.633006Z","uid":"CWPSxs3IcmDxCuZlFc","id.orig_h":"192.168.227.102","id.orig_p":39016,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"data_before_established","notice":false,"peer":"bro"} > {"ts":"2016-08-03T12:58:27.696545Z","uid":"CWPSxs3IcmDxCuZlFc","id.orig_h":"192.168.227.102","id.orig_p":39016,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"inappropriate_FIN","notice":false,"peer":"bro"} > {"ts":"2016-08-03T12:58:27.712067Z","uid":"CVBsOs3XLLccpSJBZe","id.orig_h":"192.168.227.102","id.orig_p":42692,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"data_before_established","notice":false,"peer":"bro"} > {"ts":"2016-08-03T12:58:27.803202Z","uid":"CVBsOs3XLLccpSJBZe","id.orig_h":"192.168.227.102","id.orig_p":42692,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"inappropriate_FIN","notice":false,"peer":"bro"} > {"ts":"2016-08-03T12:58:27.805073Z","uid":"CJgitl2mjXr4YEnw3f","id.orig_h":"192.168.227.102","id.orig_p":42910,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"data_before_established","notice":false,"peer":"bro"} > {"ts":"2016-08-03T12:58:27.871340Z","uid":"CJgitl2mjXr4YEnw3f","id.orig_h":"192.168.227.102","id.orig_p":42910,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"inappropriate_FIN","notice":false,"peer":"bro"} > {"ts":"2016-08-03T12:58:27.896425Z","uid":"CAFrrn2rYMKZtslpVl","id.orig_h":"192.168.227.102","id.orig_p":35664,"id.resp_h":"192.168.227.101","id.resp_p":445,"name":"data_before_established","notice":false,"peer":"bro"} > > > Thanks, > Regards, > Aneela Safdar > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From daniel.manzo at bayer.com Wed Aug 3 12:39:33 2016 From: daniel.manzo at bayer.com (Daniel Manzo) Date: Wed, 3 Aug 2016 19:39:33 +0000 Subject: [Bro] Network taps for Bro In-Reply-To: References: <2C7473428EFB4348960ACC47FDC529457F4F38@MOXCXR.na.bayer.cnb> Message-ID: <2C7473428EFB4348960ACC47FDC529457F50B3@MOXCXR.na.bayer.cnb> It is a single 10G connection right now, but possibly expanding in the future. I'm just focusing on the single 10G at the moment, so I think I would be able to connect right to the bro box, like you mentioned. I'll look more into tap aggregation/load-balancing later on. Thanks, Daniel From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Gary Faulkner Sent: Wednesday, August 03, 2016 3:29 PM To: bro at bro.org Subject: Re: [Bro] Network taps for Bro Another thing to consider is if it is a single 10G connection you may be able to go right to the bro box from the tap, but if you have multiple 10G connections, or need to send the signal to monitoring tools on multiple boxes you may also need to look into a tap aggregator/ load-balancer as well. If the connection is running on a specific CWDM/DWDM wavelength you may also need to check that your NICs and/or tap aggregator support the proper optics as not all do. ~Gary On 8/3/16 2:02 PM, James Eyrich wrote: Bro doesnt care about any of that. The optics going into your tap aggregator or direct into to the bro nodes need to match what ever you are using for the connection same for the splitter regarding splitter ratios - it depends what your light budget regarding the receive sensitivity on the ends of the actual connection and the optics feeding the bro system Off the top of my head I was thinking 50/50 is good for data center and 70/30 for WAN if you are running out of light once the splitter is in place you might have to move to higher powered optics all around. One thing we ran into is some of the "lite" optics for use in data centers also have reduced sensitivity in addition to lower send power. On 8/3/2016 1:37 PM, Daniel Manzo wrote: Hi all, My team is looking into using the Bro IDS for monitoring of a science DMZ with a 10 Gbps network. I was wondering how to choose which network tap(s) is necessary for this type of connection and if you have any recommendations/methods for setting up the hardware for Bro. I have been looking at the passive Ixia Flex taps, specifically the LC 10G SM 50/50 split tap. Will single mode (SM) versus multi-mode (MM) make a difference for Bro? And does Bro require a 50/50 ratio, or would I be able to get away with a different ratio? Thanks for the help, Daniel Manzo _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160803/10c1e2f8/attachment-0001.html From johanna at icir.org Wed Aug 3 12:42:32 2016 From: johanna at icir.org (Johanna Amann) Date: Wed, 3 Aug 2016 12:42:32 -0700 Subject: [Bro] Determining remote proxy servers using Bro. In-Reply-To: References: Message-ID: <20160803194232.GB6821@wifi154.sys.ICSI.Berkeley.EDU> Hi Fatema, one idea would be to look if the used proxy servers set a header like, X-Forwarded-For (https://en.wikipedia.org/wiki/X-Forwarded-For). If such a header is present, you already might have an entry in the proxied column of http.log. I hope this helps, Johanna On Fri, Jul 29, 2016 at 02:17:37PM -0400, fatema bannatwala wrote: > Hi, > > Recently we have seen an uptick in use of proxy servers to login to the > accounts from people living in China. And since the connection appears to > come from US based IP address (probably a proxy) they go un-flagged by the > IDS/IPS devices, as they see normal logins from United States IP addresses. > So my question is, is there a way to determine that the incoming connection > from an IP is actually a proxy server's IP, by looking at some unique > patterns in data collected by IDS/IPS devices? and if so can we do it using > Bro? > > Thanks, > Fatema. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Wed Aug 3 12:47:32 2016 From: johanna at icir.org (Johanna Amann) Date: Wed, 3 Aug 2016 12:47:32 -0700 Subject: [Bro] File Extraction In-Reply-To: References: Message-ID: <20160803194732.GA7211@wifi154.sys.ICSI.Berkeley.EDU> Hi Al, > I'm new to Bro and using version 2.3.2 and want to extract all the exe's > seen on the network. In bro-file-extract we are using the file-extract.bro > script to try to parse for the exe's (partial of script): First - is there any reason for you to still use 2.3.2? File handling (and a lot of other things) have become more robust in 2.4. In any case... > global ext_map:table[string] of string = { > ["application/x/dosexec"] = "exe", you probably want application/x-dosexec here, not x/dosexec. That might already be enough to fix this. > redef FileExtract::prefix="/var/log/netlogs/bro/file-extracts.bro"; This line seems superfluous and wrong, especially since it is redef-ed again two lines later. > redef FileExtract::default_limit = 314572800; > redef FileExtract::prefix = "/var/log/netlogs/bro/file-extracts/"; > > We also have the file-extract-http-local.bro set to extract on our network: > > global http_extract_file_ignore: set [subnet] = { > 10.0.0.0/8, > }; > The following seems to talk about files that you modified locally and that do not ship with the Bro distribution. As such, it is really hard to give feedback about it. > We think the problem is that _load_.bro has the file extract commented out > under bro-icmp: > #@load ./file-extract-http-local.bro > #@load ./file-extract-types.bro > @load ./bro-file-extract > When I tried to enable these Bro failed the scripts check with errors like: > internal warning in > /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line > 6: Discarded extraneous Broxygen comment: Modified from base scripts to > extract only from external hosts > fatal error in > /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line > 7:can't find base/protocols/http/file-ident > I continued to receive these errors and had to back out of removing the > comments > > Under bro-file-extract _load_.bro looks correct: > @load ./file-extract > > What I'm getting in /var/log/netlogs/bro/file-extracts are entries like: > HTTP-F7K52nSzN3h7GNM31.exe > These files occur occasionally I'm not sure what they are. I hope this helps, Johanna From mike.patterson at uwaterloo.ca Wed Aug 3 12:54:28 2016 From: mike.patterson at uwaterloo.ca (Mike Patterson) Date: Wed, 3 Aug 2016 19:54:28 +0000 Subject: [Bro] Network taps for Bro In-Reply-To: <2C7473428EFB4348960ACC47FDC529457F50B3@MOXCXR.na.bayer.cnb> References: <2C7473428EFB4348960ACC47FDC529457F4F38@MOXCXR.na.bayer.cnb> <2C7473428EFB4348960ACC47FDC529457F50B3@MOXCXR.na.bayer.cnb> Message-ID: <12F8667E-8A24-4D69-BDCC-37A7EF497056@uwaterloo.ca> Depending on your actual load, you'll definitely need load balancing, whether or not you're plugged in directly. Depending on the NIC, there's various solutions - PF_RING drivers for various platforms (Intel X520 is popular), Endace DAG, Myricom - I probably left somebody out - that can do this for varying costs. DAG is fantastically expensive, but is kinda magic (except when it isn't, kernel upgrades can hose you). PF_RING is cheap - free for certain folks - but I find it a bit more annoying to configure and maintain than the DAG. Can't argue with the price though. I can't speak for the Myricom options, but I gather they're a middle ground - more expensive than X520 + PF_RING, much less expensive than a DAG. All perform reasonably well. My own environment started out with a single Dell R710 with a DAG 9.2X2, into which I plugged a couple SPAN ports, merged them, then load balanced them back out again. For a while I ran both Snort and Bro on the same box. Later, I acquired an Arista 7150S and 720 with Intel gear, put my SPANs into that, then have it just merge my two inputs into single outputs on a couple of tap ports - an upgraded box contains the DAG for Bro, and the new 720 contains an X520 with PF_RING, which does similar load balancing for Snort. Be prepared to spend a certain amount of time up front configuring hardware + software just so. Having the Arista in the mix is nice because I can easily add more tap ports for a test environment, one-off snooping, that sort of thing. Mike -- The question, "Will a key with more bits give me better security?" is a lot like the question, "Will more cylinders in my car engine make me go faster?" - Jon Callas > On Aug 3, 2016, at 15:39, Daniel Manzo wrote: > > It is a single 10G connection right now, but possibly expanding in the future. I?m just focusing on the single 10G at the moment, so I think I would be able to connect right to the bro box, like you mentioned. I?ll look more into tap aggregation/load-balancing later on. > > Thanks, > Daniel > > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Gary Faulkner > Sent: Wednesday, August 03, 2016 3:29 PM > To: bro at bro.org > Subject: Re: [Bro] Network taps for Bro > > Another thing to consider is if it is a single 10G connection you may be able to go right to the bro box from the tap, but if you have multiple 10G connections, or need to send the signal to monitoring tools on multiple boxes you may also need to look into a tap aggregator/ load-balancer as well. If the connection is running on a specific CWDM/DWDM wavelength you may also need to check that your NICs and/or tap aggregator support the proper optics as not all do. > > ~Gary > > On 8/3/16 2:02 PM, James Eyrich wrote: > Bro doesnt care about any of that. > The optics going into your tap aggregator or direct into to the bro > nodes need to match what ever you are using for the connection > same for the splitter > regarding splitter ratios - it depends what your light budget regarding > the receive sensitivity on the ends of the actual connection and the > optics feeding the bro system > Off the top of my head I was thinking 50/50 is good for data center and > 70/30 for WAN > if you are running out of light once the splitter is in place you might > have to move to higher powered optics all around. > > One thing we ran into is some of the "lite" optics for use in data > centers also have reduced sensitivity in addition to lower send power. > > On 8/3/2016 1:37 PM, Daniel Manzo wrote: > > Hi all, > > > > My team is looking into using the Bro IDS for monitoring of a science > DMZ with a 10 Gbps network. I was wondering how to choose which > network tap(s) is necessary for this type of connection and if you > have any recommendations/methods for setting up the hardware for Bro. > I have been looking at the passive Ixia Flex taps, specifically the LC > 10G SM 50/50 split tap. Will single mode (SM) versus multi-mode (MM) > make a difference for Bro? And does Bro require a 50/50 ratio, or > would I be able to get away with a different ratio? > > > > Thanks for the help, > > Daniel Manzo > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From dopheide at gmail.com Wed Aug 3 17:24:58 2016 From: dopheide at gmail.com (Mike Dopheide) Date: Wed, 3 Aug 2016 19:24:58 -0500 Subject: [Bro] BroCon rideshare? In-Reply-To: References: Message-ID: I was in Austin last year for SuperComputing'15 and had very good luck getting around with Uber. I've never actually tried Uber Pool though. -Dop On Wed, Aug 3, 2016 at 1:22 PM, Collyer, Jeffrey W. (jwc3f) < jwc3f at virginia.edu> wrote: > In booking my travel, I queried the hotel about travel to/from TACC. They > say its a good 20 minute walk. Googling the weather in Austin mid > September, but it seems that 90s and Thunderstorms are reasonably normal. > > With that in mind - is there a rideshare? or does someone know a clever > way/app to organize one. It seems a shame for everyone to rent a car for > such a short daily ride. > > Jeffrey Collyer > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160803/b0fedb39/attachment.html From ansaf_130 at yahoo.com Wed Aug 3 21:52:54 2016 From: ansaf_130 at yahoo.com (Aneela Safdar) Date: Thu, 4 Aug 2016 04:52:54 +0000 (UTC) Subject: [Bro] Changing location of log files References: <726144474.9422636.1470286374614.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <726144474.9422636.1470286374614.JavaMail.yahoo@mail.yahoo.com> Hi, Is there a way I could change the default location of log files. I am interested in some of the bro generated logs and was wondering if I could make them being create and update in a separate directory instead of 'current'??Regards, Aneela Safdar -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160804/477a8c62/attachment.html From dnj0496 at gmail.com Wed Aug 3 22:11:06 2016 From: dnj0496 at gmail.com (Dk Jack) Date: Wed, 3 Aug 2016 22:11:06 -0700 Subject: [Bro] debugging script Message-ID: Hi, I am trying to debug a bro script. In my script I am trying to load a table and reference the table contents. This is working correctly as expected. I want to verify if the re-reading of the table is working correctly or not. I've set the mode to REREAD in the add_table call. After changing the file contents, the new data doesn't seem to be making it into the table. I tried adding print statements into the 'event entry' but not sure where those prints are going. Where do the output of the print statements go on a running system (i.e. a cluster)? Is there a way for me add some debugging info into scripts i.e. printf like debugging? Any pointers are much appreciated... thanks. Dk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160803/73eb6eac/attachment-0001.html From hckim at narusec.com Wed Aug 3 22:44:25 2016 From: hckim at narusec.com (=?UTF-8?B?6rmA7Z2s7LKg?=) Date: Thu, 4 Aug 2016 14:44:25 +0900 Subject: [Bro] Changing location of log files Message-ID: Hi you could change default location directory by config file ../etc/broctl.conf SpoolDir = PATH here is the user options page https://www.bro.org/sphinx/components/broctl/README.html and you could also change this when you are installing from source ./configure --spooldir=PATH make make install >Hi, >Is there a way I could change the default location of log files. I am interested in some of the bro generated logs and was wondering if I >could make them being create and update in a separate directory instead of 'current'??Regards, >Aneela Safdar -- ------------------------------------------------------ Hichul Kim ??? ?? ??? Naru Security (?)?????? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160804/8840838f/attachment.html From johanna at icir.org Thu Aug 4 09:21:40 2016 From: johanna at icir.org (Johanna Amann) Date: Thu, 4 Aug 2016 09:21:40 -0700 Subject: [Bro] debugging script In-Reply-To: References: Message-ID: <20160804162136.GA10845@Beezling.local> Hi, > After changing the file contents, the new data doesn't seem to be making it > into the table. I tried adding print statements into the 'event entry' but > not sure where those prints are going. Where do the output of the print > statements go on a running system (i.e. a cluster)? Is there a way for me > add some debugging info into scripts i.e. printf like debugging? Any > pointers are much appreciated... thanks. When you are running with broctl, I think the print output goes into [install-base]/spool/[nodename]/stdout.log. So, e.g. [base]/spool/worker-1/stdout.log. Generally, if something odd happens with the input framework, looking at reporter.log also always is a good idea; if the input framework stops reading from a specified stream, it always logs the errors there. I hope this helps, Johanna From jazoff at illinois.edu Thu Aug 4 09:56:11 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 4 Aug 2016 16:56:11 +0000 Subject: [Bro] debugging script In-Reply-To: <20160804162136.GA10845@Beezling.local> References: <20160804162136.GA10845@Beezling.local> Message-ID: <6DD80255-BD6D-4A78-B963-F6536BAA960A@illinois.edu> > On Aug 4, 2016, at 12:21 PM, Johanna Amann wrote: > > Hi, > >> After changing the file contents, the new data doesn't seem to be making it >> into the table. I tried adding print statements into the 'event entry' but >> not sure where those prints are going. Where do the output of the print >> statements go on a running system (i.e. a cluster)? Is there a way for me >> add some debugging info into scripts i.e. printf like debugging? Any >> pointers are much appreciated... thanks. > > When you are running with broctl, I think the print output goes into > [install-base]/spool/[nodename]/stdout.log. > > So, e.g. [base]/spool/worker-1/stdout.log. One gotcha with this (that has tripped me up an embarrassing number of times and as most recently as yesterday) is that those files are buffered. If you only print a few lines, nothing will be written to stdout.log until bro stops. To fix that, you can just do print("whatever"); flush_all(); Or if you are doing a lot of testing, have this in place: event flush() { flush_all(); schedule 5sec { flush() }; } event bro_init() { schedule 5sec { flush() }; } I vaguely remember there is a way to just set all files to be non-buffered.. though hard flushing every few seconds probably is better for performance. -- - Justin Azoff From soehlert at es.net Thu Aug 4 10:11:44 2016 From: soehlert at es.net (Samuel Oehlert) Date: Thu, 4 Aug 2016 12:11:44 -0500 Subject: [Bro] BroCon rideshare? In-Reply-To: References: Message-ID: Due to some crazy politics in Austin, Uber/Lyft are no longer there, but Austin has some other ride share start ups. http://kxan.com/2016/06/02/travelers-guide-ridesharing-apps-offer-rides-from-austin-airport/ On Wed, Aug 3, 2016 at 7:24 PM, Mike Dopheide wrote: > I was in Austin last year for SuperComputing'15 and had very good luck > getting around with Uber. I've never actually tried Uber Pool though. > > -Dop > > On Wed, Aug 3, 2016 at 1:22 PM, Collyer, Jeffrey W. (jwc3f) < > jwc3f at virginia.edu> wrote: > >> In booking my travel, I queried the hotel about travel to/from TACC. >> They say its a good 20 minute walk. Googling the weather in Austin mid >> September, but it seems that 90s and Thunderstorms are reasonably normal. >> >> With that in mind - is there a rideshare? or does someone know a clever >> way/app to organize one. It seems a shame for everyone to rent a car for >> such a short daily ride. >> >> Jeffrey Collyer >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160804/b1ba0774/attachment.html From slagell at illinois.edu Thu Aug 4 10:54:22 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Thu, 4 Aug 2016 17:54:22 +0000 Subject: [Bro] BroCon rideshare? In-Reply-To: References: , Message-ID: The hotel is probably also thinking of the old TACC building. It is only 0.4 miles. On Aug 4, 2016, at 12:19 PM, Samuel Oehlert > wrote: Due to some crazy politics in Austin, Uber/Lyft are no longer there, but Austin has some other ride share start ups. http://kxan.com/2016/06/02/travelers-guide-ridesharing-apps-offer-rides-from-austin-airport/ On Wed, Aug 3, 2016 at 7:24 PM, Mike Dopheide > wrote: I was in Austin last year for SuperComputing'15 and had very good luck getting around with Uber. I've never actually tried Uber Pool though. -Dop On Wed, Aug 3, 2016 at 1:22 PM, Collyer, Jeffrey W. (jwc3f) > wrote: In booking my travel, I queried the hotel about travel to/from TACC. They say its a good 20 minute walk. Googling the weather in Austin mid September, but it seems that 90s and Thunderstorms are reasonably normal. With that in mind - is there a rideshare? or does someone know a clever way/app to organize one. It seems a shame for everyone to rent a car for such a short daily ride. Jeffrey Collyer _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160804/ab58a2aa/attachment-0001.html From fatema.bannatwala at gmail.com Thu Aug 4 13:07:15 2016 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Thu, 4 Aug 2016 16:07:15 -0400 Subject: [Bro] Determining remote proxy servers using Bro. In-Reply-To: <20160803194232.GB6821@wifi154.sys.ICSI.Berkeley.EDU> References: <20160803194232.GB6821@wifi154.sys.ICSI.Berkeley.EDU> Message-ID: Thanks Johanna, Didn't realized that the "Proxied" field in http.log serves the purpose. Thanks for the suggestion. -Fatema On Wed, Aug 3, 2016 at 3:42 PM, Johanna Amann wrote: > Hi Fatema, > > one idea would be to look if the used proxy servers set a header like, > X-Forwarded-For (https://en.wikipedia.org/wiki/X-Forwarded-For). If such a > header is present, you already might have an entry in the proxied column > of http.log. > > I hope this helps, > Johanna > > On Fri, Jul 29, 2016 at 02:17:37PM -0400, fatema bannatwala wrote: > > Hi, > > > > Recently we have seen an uptick in use of proxy servers to login to the > > accounts from people living in China. And since the connection appears to > > come from US based IP address (probably a proxy) they go un-flagged by > the > > IDS/IPS devices, as they see normal logins from United States IP > addresses. > > So my question is, is there a way to determine that the incoming > connection > > from an IP is actually a proxy server's IP, by looking at some unique > > patterns in data collected by IDS/IPS devices? and if so can we do it > using > > Bro? > > > > Thanks, > > Fatema. > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160804/46e39f01/attachment.html From dnj0496 at gmail.com Thu Aug 4 14:19:06 2016 From: dnj0496 at gmail.com (Dk Jack) Date: Thu, 4 Aug 2016 14:19:06 -0700 Subject: [Bro] debugging script In-Reply-To: <6DD80255-BD6D-4A78-B963-F6536BAA960A@illinois.edu> References: <20160804162136.GA10845@Beezling.local> <6DD80255-BD6D-4A78-B963-F6536BAA960A@illinois.edu> Message-ID: Thanks Johanna, Justin, flush_all helped. Bhasker. > On Aug 4, 2016, at 9:56 AM, "Azoff, Justin S" wrote: > > >> On Aug 4, 2016, at 12:21 PM, Johanna Amann wrote: >> >> Hi, >> >>> After changing the file contents, the new data doesn't seem to be making it >>> into the table. I tried adding print statements into the 'event entry' but >>> not sure where those prints are going. Where do the output of the print >>> statements go on a running system (i.e. a cluster)? Is there a way for me >>> add some debugging info into scripts i.e. printf like debugging? Any >>> pointers are much appreciated... thanks. >> >> When you are running with broctl, I think the print output goes into >> [install-base]/spool/[nodename]/stdout.log. >> >> So, e.g. [base]/spool/worker-1/stdout.log. > > One gotcha with this (that has tripped me up an embarrassing number of times and as most recently as yesterday) is that those files are buffered. > > If you only print a few lines, nothing will be written to stdout.log until bro stops. To fix that, you can just do > > print("whatever"); > flush_all(); > > > Or if you are doing a lot of testing, have this in place: > > event flush() { > flush_all(); > schedule 5sec { flush() }; > } > > event bro_init() { > schedule 5sec { flush() }; > } > > I vaguely remember there is a way to just set all files to be non-buffered.. though hard flushing every few seconds probably is better for performance. > > -- > - Justin Azoff > From Ben.McDowall at spark.co.nz Thu Aug 4 18:38:26 2016 From: Ben.McDowall at spark.co.nz (Ben McDowall) Date: Fri, 5 Aug 2016 01:38:26 +0000 Subject: [Bro] http.log stops logging Message-ID: I have a weird condition going on in which Bro stops loging after the filesize of http hits 100 odd kb it just started happening the other morning (5am) -rw-r--r-- 1 root root 107K Aug 5 00:00 http.23:00:00-00:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 23:00 http.22:00:00-23:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 22:00 http.21:00:00-22:00:00.log.gz -rw-r--r-- 1 root root 106K Aug 4 21:00 http.20:00:00-21:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 20:00 http.19:00:00-20:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 19:00 http.18:00:00-19:00:00.log.gz -rw-r--r-- 1 root root 108K Aug 4 18:00 http.17:00:00-18:00:00.log.gz -rw-r--r-- 1 root root 108K Aug 4 17:00 http.16:00:00-17:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 16:00 http.15:00:00-16:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 15:00 http.14:00:00-15:00:00.log.gz -rw-r--r-- 1 root root 106K Aug 4 14:00 http.13:00:00-14:00:00.log.gz -rw-r--r-- 1 root root 106K Aug 4 13:00 http.12:00:00-13:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 12:00 http.11:00:00-12:00:00.log.gz -rw-r--r-- 1 root root 109K Aug 4 11:00 http.10:00:00-11:00:00.log.gz -rw-r--r-- 1 root root 110K Aug 4 10:00 http.09:00:00-10:00:00.log.gz -rw-r--r-- 1 root root 110K Aug 4 09:00 http.08:00:00-09:00:00.log.gz -rw-r--r-- 1 root root 112K Aug 4 08:00 http.07:00:00-08:00:00.log.gz -rw-r--r-- 1 root root 110K Aug 4 07:00 http.06:00:00-07:00:00.log.gz -rw-r--r-- 1 root root 476K Aug 4 06:00 http.05:00:00-06:00:00.log.gz -rw-r--r-- 1 root root 30M Aug 4 05:00 http.04:00:00-05:00:00.log.gz -rw-r--r-- 1 root root 34M Aug 4 04:00 http.03:00:00-04:00:00.log.gz -rw-r--r-- 1 root root 34M Aug 4 03:00 http.02:00:00-03:00:00.log.gz -rw-r--r-- 1 root root 40M Aug 4 02:00 http.01:00:00-02:00:00.log.gz -rw-r--r-- 1 root root 45M Aug 4 01:00 http.00:00:00-01:00:00.log.gz Has anyone else encountered this before? I have 3 workers as I load balance the traffic going to my server. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160805/c90782a3/attachment.html From jazoff at illinois.edu Thu Aug 4 18:52:14 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 5 Aug 2016 01:52:14 +0000 Subject: [Bro] http.log stops logging In-Reply-To: References: Message-ID: Does your reporter.log contain anything? Is that the only log file that is having this problem? What do those log files contain? Is it normal logs up until a certain port, or is the only thing in the http.log a certain kind of request? does the conn.log contain entries for all the http traffic you are missing? -- - Justin Azoff > On Aug 4, 2016, at 9:38 PM, Ben McDowall wrote: > > I have a weird condition going on in which Bro stops loging after the filesize of http hits 100 odd kb it just started happening the other morning (5am) > > -rw-r--r-- 1 root root 107K Aug 5 00:00 http.23:00:00-00:00:00.log.gz > -rw-r--r-- 1 root root 107K Aug 4 23:00 http.22:00:00-23:00:00.log.gz > -rw-r--r-- 1 root root 107K Aug 4 22:00 http.21:00:00-22:00:00.log.gz > -rw-r--r-- 1 root root 106K Aug 4 21:00 http.20:00:00-21:00:00.log.gz > -rw-r--r-- 1 root root 107K Aug 4 20:00 http.19:00:00-20:00:00.log.gz > -rw-r--r-- 1 root root 107K Aug 4 19:00 http.18:00:00-19:00:00.log.gz > -rw-r--r-- 1 root root 108K Aug 4 18:00 http.17:00:00-18:00:00.log.gz > -rw-r--r-- 1 root root 108K Aug 4 17:00 http.16:00:00-17:00:00.log.gz > -rw-r--r-- 1 root root 107K Aug 4 16:00 http.15:00:00-16:00:00.log.gz > -rw-r--r-- 1 root root 107K Aug 4 15:00 http.14:00:00-15:00:00.log.gz > -rw-r--r-- 1 root root 106K Aug 4 14:00 http.13:00:00-14:00:00.log.gz > -rw-r--r-- 1 root root 106K Aug 4 13:00 http.12:00:00-13:00:00.log.gz > -rw-r--r-- 1 root root 107K Aug 4 12:00 http.11:00:00-12:00:00.log.gz > -rw-r--r-- 1 root root 109K Aug 4 11:00 http.10:00:00-11:00:00.log.gz > -rw-r--r-- 1 root root 110K Aug 4 10:00 http.09:00:00-10:00:00.log.gz > -rw-r--r-- 1 root root 110K Aug 4 09:00 http.08:00:00-09:00:00.log.gz > -rw-r--r-- 1 root root 112K Aug 4 08:00 http.07:00:00-08:00:00.log.gz > -rw-r--r-- 1 root root 110K Aug 4 07:00 http.06:00:00-07:00:00.log.gz > -rw-r--r-- 1 root root 476K Aug 4 06:00 http.05:00:00-06:00:00.log.gz > -rw-r--r-- 1 root root 30M Aug 4 05:00 http.04:00:00-05:00:00.log.gz > -rw-r--r-- 1 root root 34M Aug 4 04:00 http.03:00:00-04:00:00.log.gz > -rw-r--r-- 1 root root 34M Aug 4 03:00 http.02:00:00-03:00:00.log.gz > -rw-r--r-- 1 root root 40M Aug 4 02:00 http.01:00:00-02:00:00.log.gz > -rw-r--r-- 1 root root 45M Aug 4 01:00 http.00:00:00-01:00:00.log.gz > > Has anyone else encountered this before? I have 3 workers as I load balance the traffic going to my server. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From dhoelzer at sans.org Fri Aug 5 04:13:28 2016 From: dhoelzer at sans.org (Hoelzer, Dave) Date: Fri, 5 Aug 2016 11:13:28 +0000 Subject: [Bro] http.log stops logging In-Reply-To: References: Message-ID: Just a thought? Are you sure that no one has changed the network around and that the HTTP traffic is still passing the bro sensor? If it creates the log then it sees something and is working (for the moment, assume correctly). If it saw nothing, no log.. Could someone have changed a path on you? ??????????????????? David Hoelzer Fellow, SANS Institute Dean of Faculty, SANS Technology Institute On August 4, 2016 at 9:41:08 PM, Ben McDowall (ben.mcdowall at spark.co.nz) wrote: I have a weird condition going on in which Bro stops loging after the filesize of http hits 100 odd kb it just started happening the other morning (5am) -rw-r--r-- 1 root root 107K Aug 5 00:00 http.23:00:00-00:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 23:00 http.22:00:00-23:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 22:00 http.21:00:00-22:00:00.log.gz -rw-r--r-- 1 root root 106K Aug 4 21:00 http.20:00:00-21:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 20:00 http.19:00:00-20:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 19:00 http.18:00:00-19:00:00.log.gz -rw-r--r-- 1 root root 108K Aug 4 18:00 http.17:00:00-18:00:00.log.gz -rw-r--r-- 1 root root 108K Aug 4 17:00 http.16:00:00-17:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 16:00 http.15:00:00-16:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 15:00 http.14:00:00-15:00:00.log.gz -rw-r--r-- 1 root root 106K Aug 4 14:00 http.13:00:00-14:00:00.log.gz -rw-r--r-- 1 root root 106K Aug 4 13:00 http.12:00:00-13:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 12:00 http.11:00:00-12:00:00.log.gz -rw-r--r-- 1 root root 109K Aug 4 11:00 http.10:00:00-11:00:00.log.gz -rw-r--r-- 1 root root 110K Aug 4 10:00 http.09:00:00-10:00:00.log.gz -rw-r--r-- 1 root root 110K Aug 4 09:00 http.08:00:00-09:00:00.log.gz -rw-r--r-- 1 root root 112K Aug 4 08:00 http.07:00:00-08:00:00.log.gz -rw-r--r-- 1 root root 110K Aug 4 07:00 http.06:00:00-07:00:00.log.gz -rw-r--r-- 1 root root 476K Aug 4 06:00 http.05:00:00-06:00:00.log.gz -rw-r--r-- 1 root root 30M Aug 4 05:00 http.04:00:00-05:00:00.log.gz -rw-r--r-- 1 root root 34M Aug 4 04:00 http.03:00:00-04:00:00.log.gz -rw-r--r-- 1 root root 34M Aug 4 03:00 http.02:00:00-03:00:00.log.gz -rw-r--r-- 1 root root 40M Aug 4 02:00 http.01:00:00-02:00:00.log.gz -rw-r--r-- 1 root root 45M Aug 4 01:00 http.00:00:00-01:00:00.log.gz Has anyone else encountered this before? I have 3 workers as I load balance the traffic going to my server. _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160805/b10c6251/attachment-0001.html From daniel.manzo at bayer.com Fri Aug 5 08:36:23 2016 From: daniel.manzo at bayer.com (Daniel Manzo) Date: Fri, 5 Aug 2016 15:36:23 +0000 Subject: [Bro] Blocking packets Message-ID: <2C7473428EFB4348960ACC47FDC529450EA4E2C5@MOXCXN.na.bayer.cnb> Hi all, Can Bro block packets or part of traffic, in addition to logging? Or is this something that needs to be configured on an aggregator or tap? I apologize if this is a very simple topic, as I'm a Bro noob. Best regards, Daniel Manzo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160805/eca0292c/attachment.html From Ben.McDowall at spark.co.nz Fri Aug 5 23:14:05 2016 From: Ben.McDowall at spark.co.nz (Ben McDowall) Date: Sat, 6 Aug 2016 06:14:05 +0000 Subject: [Bro] http.log stops logging In-Reply-To: References: , Message-ID: Sorted now. Rebooted my guest that didn't work. Rebooted my host platform now all working. Strange as. :) As you all were Sent from my Samsung Galaxy smartphone. -------- Original message -------- From: "Hoelzer, Dave" Date: 5/08/16 11:13 PM (GMT+12:00) To: Ben McDowall , bro at bro.org Subject: Re: [Bro] http.log stops logging Just a thought... Are you sure that no one has changed the network around and that the HTTP traffic is still passing the bro sensor? If it creates the log then it sees something and is working (for the moment, assume correctly). If it saw nothing, no log.. Could someone have changed a path on you? ------------------- David Hoelzer Fellow, SANS Institute Dean of Faculty, SANS Technology Institute On August 4, 2016 at 9:41:08 PM, Ben McDowall (ben.mcdowall at spark.co.nz) wrote: I have a weird condition going on in which Bro stops loging after the filesize of http hits 100 odd kb it just started happening the other morning (5am) -rw-r--r-- 1 root root 107K Aug 5 00:00 http.23:00:00-00:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 23:00 http.22:00:00-23:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 22:00 http.21:00:00-22:00:00.log.gz -rw-r--r-- 1 root root 106K Aug 4 21:00 http.20:00:00-21:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 20:00 http.19:00:00-20:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 19:00 http.18:00:00-19:00:00.log.gz -rw-r--r-- 1 root root 108K Aug 4 18:00 http.17:00:00-18:00:00.log.gz -rw-r--r-- 1 root root 108K Aug 4 17:00 http.16:00:00-17:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 16:00 http.15:00:00-16:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 15:00 http.14:00:00-15:00:00.log.gz -rw-r--r-- 1 root root 106K Aug 4 14:00 http.13:00:00-14:00:00.log.gz -rw-r--r-- 1 root root 106K Aug 4 13:00 http.12:00:00-13:00:00.log.gz -rw-r--r-- 1 root root 107K Aug 4 12:00 http.11:00:00-12:00:00.log.gz -rw-r--r-- 1 root root 109K Aug 4 11:00 http.10:00:00-11:00:00.log.gz -rw-r--r-- 1 root root 110K Aug 4 10:00 http.09:00:00-10:00:00.log.gz -rw-r--r-- 1 root root 110K Aug 4 09:00 http.08:00:00-09:00:00.log.gz -rw-r--r-- 1 root root 112K Aug 4 08:00 http.07:00:00-08:00:00.log.gz -rw-r--r-- 1 root root 110K Aug 4 07:00 http.06:00:00-07:00:00.log.gz -rw-r--r-- 1 root root 476K Aug 4 06:00 http.05:00:00-06:00:00.log.gz -rw-r--r-- 1 root root 30M Aug 4 05:00 http.04:00:00-05:00:00.log.gz -rw-r--r-- 1 root root 34M Aug 4 04:00 http.03:00:00-04:00:00.log.gz -rw-r--r-- 1 root root 34M Aug 4 03:00 http.02:00:00-03:00:00.log.gz -rw-r--r-- 1 root root 40M Aug 4 02:00 http.01:00:00-02:00:00.log.gz -rw-r--r-- 1 root root 45M Aug 4 01:00 http.00:00:00-01:00:00.log.gz Has anyone else encountered this before? I have 3 workers as I load balance the traffic going to my server. _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160806/fb6aca36/attachment.html From johndbabio at gmail.com Sun Aug 7 07:04:07 2016 From: johndbabio at gmail.com (John Babio) Date: Sun, 7 Aug 2016 10:04:07 -0400 Subject: [Bro] broker make error Message-ID: Following this guide http://www.icir.org/johanna/netcontrol/ I built and install the actor framework from github Broker: make -C build all make[1]: Entering directory '/home/john/broker/build' make[2]: Entering directory '/home/john/broker/build' make[3]: Entering directory '/home/john/broker/build' Scanning dependencies of target broker make[3]: Leaving directory '/home/john/broker/build' make[3]: Entering directory '/home/john/broker/build' [ 1%] Building CXX object CMakeFiles/broker.dir/src/address.cc.o [ 2%] Building CXX object CMakeFiles/broker.dir/src/broker.cc.o In file included from /home/john/broker/src/broker.cc:9:0: /home/john/broker/src/store/result_type_info.hh:5:46: fatal error: caf/abstract_uniform_type_info.hpp: No such file or directory -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160807/c6b0ef39/attachment.html From dhoelzer at sans.org Sun Aug 7 09:38:46 2016 From: dhoelzer at sans.org (Hoelzer, Dave) Date: Sun, 7 Aug 2016 16:38:46 +0000 Subject: [Bro] broker make error In-Reply-To: References: Message-ID: The issue is the version of CAF. 0.14 will work, I believe. If you do the research you?ll see that there is supposedly a patch that was added into the Bro code in 2015, but there?s no evidence of that since the compilation continues to fail. CAF made a change that removed some types, which has lead to this issue. Nothing?s really gone, it?s just moved and Bro hasn?t kept up. ??????????????????? David Hoelzer Fellow, SANS Institute Dean of Faculty, SANS Technology Institute On August 7, 2016 at 10:13:15 AM, John Babio (johndbabio at gmail.com) wrote: Following this guide http://www.icir.org/johanna/netcontrol/ I built and install the actor framework from github Broker: make -C build all make[1]: Entering directory '/home/john/broker/build' make[2]: Entering directory '/home/john/broker/build' make[3]: Entering directory '/home/john/broker/build' Scanning dependencies of target broker make[3]: Leaving directory '/home/john/broker/build' make[3]: Entering directory '/home/john/broker/build' [ 1%] Building CXX object CMakeFiles/broker.dir/src/address.cc.o [ 2%] Building CXX object CMakeFiles/broker.dir/src/broker.cc.o In file included from /home/john/broker/src/broker.cc:9:0: /home/john/broker/src/store/result_type_info.hh:5:46: fatal error: caf/abstract_uniform_type_info.hpp: No such file or directory _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160807/127e2dab/attachment.html From johanna at icir.org Sun Aug 7 09:59:53 2016 From: johanna at icir.org (Johanna Amann) Date: Sun, 07 Aug 2016 09:59:53 -0700 Subject: [Bro] broker make error In-Reply-To: References: Message-ID: <073B9213-B640-404F-9B1F-7C8BA94D5208@icir.org> Yup, that is exactly it. There currently is a rewrite of Broker underway, which will use the newer library versions, but it is not quite done yet. Also note - you are probably better served with the instructions at https://github.com/bro/bro-netcontrol; you do no longer need to use a branch of Bro, NetControl is in master now. Johanna On 7 Aug 2016, at 9:38, Hoelzer, Dave wrote: > The issue is the version of CAF. 0.14 will work, I believe. If you > do the research you?ll see that there is supposedly a patch that was > added into the Bro code in 2015, but there?s no evidence of that > since the compilation continues to fail. > > CAF made a change that removed some types, which has lead to this > issue. Nothing?s really gone, it?s just moved and Bro hasn?t > kept up. > > ??????????????????? > David Hoelzer > Fellow, SANS Institute > Dean of Faculty, SANS Technology Institute > > > On August 7, 2016 at 10:13:15 AM, John Babio > (johndbabio at gmail.com) wrote: > > Following this guide > > http://www.icir.org/johanna/netcontrol/ > > I built and install the actor framework from github > > Broker: > make -C build all > make[1]: Entering directory '/home/john/broker/build' > make[2]: Entering directory '/home/john/broker/build' > make[3]: Entering directory '/home/john/broker/build' > Scanning dependencies of target broker > make[3]: Leaving directory '/home/john/broker/build' > make[3]: Entering directory '/home/john/broker/build' > [ 1%] Building CXX object CMakeFiles/broker.dir/src/address.cc.o > [ 2%] Building CXX object CMakeFiles/broker.dir/src/broker.cc.o > In file included from /home/john/broker/src/broker.cc:9:0: > /home/john/broker/src/store/result_type_info.hh:5:46: fatal error: > caf/abstract_uniform_type_info.hpp: No such file or directory > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johndbabio at gmail.com Sun Aug 7 10:36:09 2016 From: johndbabio at gmail.com (John Babio) Date: Sun, 7 Aug 2016 13:36:09 -0400 Subject: [Bro] broker make error In-Reply-To: <073B9213-B640-404F-9B1F-7C8BA94D5208@icir.org> References: <073B9213-B640-404F-9B1F-7C8BA94D5208@icir.org> Message-ID: Using Ubuntu 16 as the base os So until everything is sorted with the new rewrite: 1. Download caf 0.14 and config, make, make install 2. Download bro from https://www.bro.org/download/index.html or do i have to pull from git? 3. Config bro with --enable-broker On Sun, Aug 7, 2016 at 12:59 PM, Johanna Amann wrote: > Yup, that is exactly it. There currently is a rewrite of Broker underway, > which will use the newer library versions, but it is not quite done yet. > > Also note - you are probably better served with the instructions at > https://github.com/bro/bro-netcontrol; you do no longer need to use a > branch of Bro, NetControl is in master now. > > Johanna > > On 7 Aug 2016, at 9:38, Hoelzer, Dave wrote: > > The issue is the version of CAF. 0.14 will work, I believe. If you do >> the research you?ll see that there is supposedly a patch that was added >> into the Bro code in 2015, but there?s no evidence of that since the >> compilation continues to fail. >> >> CAF made a change that removed some types, which has lead to this issue. >> Nothing?s really gone, it?s just moved and Bro hasn?t kept up. >> >> ??????????????????? >> David Hoelzer >> Fellow, SANS Institute >> Dean of Faculty, SANS Technology Institute >> >> >> On August 7, 2016 at 10:13:15 AM, John Babio (johndbabio at gmail.com >> ) wrote: >> >> Following this guide >> >> http://www.icir.org/johanna/netcontrol/ >> >> I built and install the actor framework from github >> >> Broker: >> make -C build all >> make[1]: Entering directory '/home/john/broker/build' >> make[2]: Entering directory '/home/john/broker/build' >> make[3]: Entering directory '/home/john/broker/build' >> Scanning dependencies of target broker >> make[3]: Leaving directory '/home/john/broker/build' >> make[3]: Entering directory '/home/john/broker/build' >> [ 1%] Building CXX object CMakeFiles/broker.dir/src/address.cc.o >> [ 2%] Building CXX object CMakeFiles/broker.dir/src/broker.cc.o >> In file included from /home/john/broker/src/broker.cc:9:0: >> /home/john/broker/src/store/result_type_info.hh:5:46: fatal error: >> caf/abstract_uniform_type_info.hpp: No such file or directory >> >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160807/72e99e7d/attachment-0001.html From johanna at icir.org Sun Aug 7 13:07:40 2016 From: johanna at icir.org (Johanna Amann) Date: Sun, 07 Aug 2016 13:07:40 -0700 Subject: [Bro] broker make error In-Reply-To: References: <073B9213-B640-404F-9B1F-7C8BA94D5208@icir.org> Message-ID: Download can 0.14, and pull Bro from git; configure with --enable-broker and --with-libcaf=[location] (if caf is installed outside the standard paths). You also will probably have to set pythonpath like described in https://github.com/bro/bro-netcontrol if you want to use any of the connectors. Johanna On 7 Aug 2016, at 10:36, John Babio wrote: > Using Ubuntu 16 as the base os > > So until everything is sorted with the new rewrite: > 1. Download caf 0.14 and config, make, make install > 2. Download bro from https://www.bro.org/download/index.html or do i > have > to pull from git? > 3. Config bro with --enable-broker > > > > > > On Sun, Aug 7, 2016 at 12:59 PM, Johanna Amann > wrote: > >> Yup, that is exactly it. There currently is a rewrite of Broker >> underway, >> which will use the newer library versions, but it is not quite done >> yet. >> >> Also note - you are probably better served with the instructions at >> https://github.com/bro/bro-netcontrol; you do no longer need to use a >> branch of Bro, NetControl is in master now. >> >> Johanna >> >> On 7 Aug 2016, at 9:38, Hoelzer, Dave wrote: >> >> The issue is the version of CAF. 0.14 will work, I believe. If you >> do >>> the research you?ll see that there is supposedly a patch that was >>> added >>> into the Bro code in 2015, but there?s no evidence of that since >>> the >>> compilation continues to fail. >>> >>> CAF made a change that removed some types, which has lead to this >>> issue. >>> Nothing?s really gone, it?s just moved and Bro hasn?t kept up. >>> >>> ??????????????????? >>> David Hoelzer >>> Fellow, SANS Institute >>> Dean of Faculty, SANS Technology Institute >>> >>> >>> On August 7, 2016 at 10:13:15 AM, John Babio (johndbabio at gmail.com >>> ) wrote: >>> >>> Following this guide >>> >>> http://www.icir.org/johanna/netcontrol/ >>> >>> I built and install the actor framework from github >>> >>> Broker: >>> make -C build all >>> make[1]: Entering directory '/home/john/broker/build' >>> make[2]: Entering directory '/home/john/broker/build' >>> make[3]: Entering directory '/home/john/broker/build' >>> Scanning dependencies of target broker >>> make[3]: Leaving directory '/home/john/broker/build' >>> make[3]: Entering directory '/home/john/broker/build' >>> [ 1%] Building CXX object CMakeFiles/broker.dir/src/address.cc.o >>> [ 2%] Building CXX object CMakeFiles/broker.dir/src/broker.cc.o >>> In file included from /home/john/broker/src/broker.cc:9:0: >>> /home/john/broker/src/store/result_type_info.hh:5:46: fatal error: >>> caf/abstract_uniform_type_info.hpp: No such file or directory >>> >>> >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >> From johndbabio at gmail.com Sun Aug 7 13:23:14 2016 From: johndbabio at gmail.com (John Babio) Date: Sun, 7 Aug 2016 16:23:14 -0400 Subject: [Bro] broker make error In-Reply-To: References: <073B9213-B640-404F-9B1F-7C8BA94D5208@icir.org> Message-ID: Thank you much! On Sun, Aug 7, 2016 at 4:07 PM, Johanna Amann wrote: > Download can 0.14, and pull Bro from git; configure with --enable-broker > and --with-libcaf=[location] (if caf is installed outside the standard > paths). > > You also will probably have to set pythonpath like described in > https://github.com/bro/bro-netcontrol if you want to use any of the > connectors. > > Johanna > > > On 7 Aug 2016, at 10:36, John Babio wrote: > > Using Ubuntu 16 as the base os >> >> So until everything is sorted with the new rewrite: >> 1. Download caf 0.14 and config, make, make install >> 2. Download bro from https://www.bro.org/download/index.html or do i have >> to pull from git? >> 3. Config bro with --enable-broker >> >> >> >> >> >> On Sun, Aug 7, 2016 at 12:59 PM, Johanna Amann wrote: >> >> Yup, that is exactly it. There currently is a rewrite of Broker underway, >>> which will use the newer library versions, but it is not quite done yet. >>> >>> Also note - you are probably better served with the instructions at >>> https://github.com/bro/bro-netcontrol; you do no longer need to use a >>> branch of Bro, NetControl is in master now. >>> >>> Johanna >>> >>> On 7 Aug 2016, at 9:38, Hoelzer, Dave wrote: >>> >>> The issue is the version of CAF. 0.14 will work, I believe. If you do >>> >>>> the research you?ll see that there is supposedly a patch that was added >>>> into the Bro code in 2015, but there?s no evidence of that since the >>>> compilation continues to fail. >>>> >>>> CAF made a change that removed some types, which has lead to this issue. >>>> Nothing?s really gone, it?s just moved and Bro hasn?t kept up. >>>> >>>> ??????????????????? >>>> David Hoelzer >>>> Fellow, SANS Institute >>>> Dean of Faculty, SANS Technology Institute >>>> >>>> >>>> On August 7, 2016 at 10:13:15 AM, John Babio (johndbabio at gmail.com >>>> ) wrote: >>>> >>>> Following this guide >>>> >>>> http://www.icir.org/johanna/netcontrol/ >>>> >>>> I built and install the actor framework from github >>>> >>>> Broker: >>>> make -C build all >>>> make[1]: Entering directory '/home/john/broker/build' >>>> make[2]: Entering directory '/home/john/broker/build' >>>> make[3]: Entering directory '/home/john/broker/build' >>>> Scanning dependencies of target broker >>>> make[3]: Leaving directory '/home/john/broker/build' >>>> make[3]: Entering directory '/home/john/broker/build' >>>> [ 1%] Building CXX object CMakeFiles/broker.dir/src/address.cc.o >>>> [ 2%] Building CXX object CMakeFiles/broker.dir/src/broker.cc.o >>>> In file included from /home/john/broker/src/broker.cc:9:0: >>>> /home/john/broker/src/store/result_type_info.hh:5:46: fatal error: >>>> caf/abstract_uniform_type_info.hpp: No such file or directory >>>> >>>> >>>> >>>> _______________________________________________ >>>> Bro mailing list >>>> bro at bro-ids.org >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>>> _______________________________________________ >>>> Bro mailing list >>>> bro at bro-ids.org >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>>> >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160807/0bc25dff/attachment.html From johndbabio at gmail.com Sun Aug 7 15:01:10 2016 From: johndbabio at gmail.com (John Babio) Date: Sun, 7 Aug 2016 18:01:10 -0400 Subject: [Bro] bro netcontrol acld for use with Cisco ASA acl's Message-ID: Has anyone been able to accomplish connecting netcontrol into an ASA firewall? For use with shun? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160807/4a5ce0b9/attachment.html From johanna at icir.org Mon Aug 8 10:07:06 2016 From: johanna at icir.org (Johanna Amann) Date: Mon, 8 Aug 2016 10:07:06 -0700 Subject: [Bro] Blocking packets In-Reply-To: <2C7473428EFB4348960ACC47FDC529450EA4E2C5@MOXCXN.na.bayer.cnb> References: <2C7473428EFB4348960ACC47FDC529450EA4E2C5@MOXCXN.na.bayer.cnb> Message-ID: <20160808170706.GB37301@wifi154.sys.ICSI.Berkeley.EDU> Hello Daniel, to interact with the traffic on your network, e.g. by installing blocking rules into your hardware, you can use the NetControl framework, which is part of our current development version and will be part of 2.5. Documentation is available at https://www.bro.org/sphinx-git/frameworks/netcontrol.html and https://github.com/bro/bro-netcontrol Apart from that, Bro by itself can not block traffic; it depends on outside hardware or software to do that, but it can be used to push rules out depending on the traffic that you see. I hope that helps, Johanna On Fri, Aug 05, 2016 at 03:36:23PM +0000, Daniel Manzo wrote: > Hi all, > > Can Bro block packets or part of traffic, in addition to logging? Or is this something that needs to be configured on an aggregator or tap? I apologize if this is a very simple topic, as I'm a Bro noob. > > Best regards, > > Daniel Manzo > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Mon Aug 8 10:11:29 2016 From: johanna at icir.org (Johanna Amann) Date: Mon, 8 Aug 2016 10:11:29 -0700 Subject: [Bro] bro netcontrol acld for use with Cisco ASA acl's In-Reply-To: References: Message-ID: <20160808171129.GC37301@wifi154.sys.ICSI.Berkeley.EDU> Hi John, since the NetControl framework is still rather new, I assume that no one has done that and that you would have to write your own connectors. Just to give you a few pointers - if you use the netcontrol broker plugin, which uses broker to push out the netcontrol rules, you can use the python API at https://github.com/bro/bro-netcontrol/blob/master/netcontrol/api.py to get access to the commands without having to do all the python-side parsing yourself. https://github.com/bro/bro-netcontrol/blob/master/command-line/command-line.py is an application that takes that route and uses the broker plugin on the NetControl side and the python API. If you need more complex rules that you need to change on the Bro side, before you push them out via broker or another mechanism, you will probably need to write your own NetControl plugin; instructions for that are available at https://www.bro.org/sphinx-git/frameworks/netcontrol.html#writing-plugins I hope this helps a bit, Johanna On Sun, Aug 07, 2016 at 06:01:10PM -0400, John Babio wrote: > Has anyone been able to accomplish connecting netcontrol into an ASA > firewall? For use with shun? > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From dnj0496 at gmail.com Mon Aug 8 19:05:49 2016 From: dnj0496 at gmail.com (Dk Jack) Date: Mon, 8 Aug 2016 19:05:49 -0700 Subject: [Bro] question about plugin init Message-ID: Hi, I am trying write a bro analyzer plugin. My directory structure for my plugin is shown below. My scripts __load__.bro loads my init.bro and analyzer/test/test.bro. My test.bro file registers for 'event bro_init' in which I am registering for some ports. Even though I can see both my bro scripts are loaded in loaded_scripts.log, my bro_init event is never triggered and hence I never register for my ports. I can it's not being invoked because, my print in that function don't show and up and it doesn't complain even if I put syntactically incorrect code in my bro_init function. Could someone shed some light on why my bro_init event is not getting triggered or if I am doing something wrong. Thanks. Dk. |-- __bro_plugin__ |-- lib | |-- analyzer-test.linux-x86_64.so | `-- bif | |-- __load__.bro | |-- events.bif.bro | `-- test.bif.bro `-- scripts |-- __load__.bro |-- analyzer | `-- test | |-- __load__.bro | `-- test.bro `-- init.bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160808/769e4434/attachment.html From dave.a.florek at gmail.com Tue Aug 9 07:49:16 2016 From: dave.a.florek at gmail.com (Dave Florek) Date: Tue, 9 Aug 2016 10:49:16 -0400 Subject: [Bro] PF_RING integration with Bro IDS post-install Message-ID: Hi, Is there a way to enable PF_RING after a Bro IDS install? From what I saw, the Bro IDS documentation ( https://www.bro.org/documentation/load-balancing.html) mentions the following pre-configuration to enable "pf_ring" pre-install but I wanted to know if it can be enabled post-install: ./configure --with-pcap=/opt/pfring make make install -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160809/b83d88b4/attachment.html From johanna at icir.org Tue Aug 9 16:54:24 2016 From: johanna at icir.org (Johanna Amann) Date: Tue, 9 Aug 2016 16:54:24 -0700 Subject: [Bro] PF_RING integration with Bro IDS post-install In-Reply-To: References: Message-ID: <20160809235419.GA7797@wifi154.sys.ICSI.Berkeley.EDU> Hello Dave, > Is there a way to enable PF_RING after a Bro IDS install? From what I saw, > the Bro IDS documentation ( > https://www.bro.org/documentation/load-balancing.html) mentions the > following pre-configuration to enable "pf_ring" pre-install but I wanted to > know if it can be enabled post-install: since Bro has to be compiled against the correct pfring libraries, this sadly cannot be enabled without re-compiling and re-installing Bro. Johanna From dnj0496 at gmail.com Tue Aug 9 16:57:27 2016 From: dnj0496 at gmail.com (Dk Jack) Date: Tue, 9 Aug 2016 16:57:27 -0700 Subject: [Bro] question about plugin init In-Reply-To: References: Message-ID: could some please comment on my query. Another related question, can two protocol analyzer modules register for the same dpd signature? thanks. On Mon, Aug 8, 2016 at 7:05 PM, Dk Jack wrote: > Hi, > I am trying write a bro analyzer plugin. My directory structure for my > plugin is shown below. > My scripts __load__.bro loads my init.bro and analyzer/test/test.bro. My > test.bro file registers > for 'event bro_init' in which I am registering for some ports. I can see > both my bro scripts are > loaded in loaded_scripts.log. However, my bro_init event is never triggered > and hence my > ports are never registered. I can say my ports are not registered, because > I put a print in my bro_init > function and I never see it. It doesn't complain even if I put > syntactically incorrect code in my bro_init > function. Could someone shed some light on why my bro_init event is not > getting triggered or if I am > doing something wrong. Thanks. > > Dk. > > |-- __bro_plugin__ > |-- lib > | |-- analyzer-test.linux-x86_64.so > | `-- bif > | |-- __load__.bro > | |-- events.bif.bro > | `-- test.bif.bro > `-- scripts > |-- __load__.bro > |-- analyzer > | `-- test > | |-- __load__.bro > | `-- test.bro > `-- init.bro > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160809/fbf36d4d/attachment.html From michalpurzynski1 at gmail.com Tue Aug 9 18:12:39 2016 From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=) Date: Wed, 10 Aug 2016 03:12:39 +0200 Subject: [Bro] PF_RING integration with Bro IDS post-install In-Reply-To: <20160809235419.GA7797@wifi154.sys.ICSI.Berkeley.EDU> References: <20160809235419.GA7797@wifi154.sys.ICSI.Berkeley.EDU> Message-ID: Use a bro plugin for pfring. No need to rebuild anything, just that plugin. Avoids libpcap. https://github.com/bro/bro-plugins > On 10 Aug 2016, at 01:54, Johanna Amann wrote: > > Hello Dave, > >> Is there a way to enable PF_RING after a Bro IDS install? From what I saw, >> the Bro IDS documentation ( >> https://www.bro.org/documentation/load-balancing.html) mentions the >> following pre-configuration to enable "pf_ring" pre-install but I wanted to >> know if it can be enabled post-install: > > since Bro has to be compiled against the correct pfring libraries, this > sadly cannot be enabled without re-compiling and re-installing Bro. > > Johanna > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160810/c16c5c25/attachment.html From robin at icir.org Wed Aug 10 09:17:33 2016 From: robin at icir.org (Robin Sommer) Date: Wed, 10 Aug 2016 09:17:33 -0700 Subject: [Bro] question about plugin init In-Reply-To: References: Message-ID: <20160810161733.GA32926@icir.org> > Could someone shed some light on why my bro_init event is not getting > triggered or if I am doing something wrong. This would be easiest to debug with the actual code. Can you send me a tar file of your plugin off-list? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From robin at icir.org Wed Aug 10 09:20:30 2016 From: robin at icir.org (Robin Sommer) Date: Wed, 10 Aug 2016 09:20:30 -0700 Subject: [Bro] question about plugin init In-Reply-To: References: Message-ID: <20160810162030.GB32926@icir.org> On Tue, Aug 09, 2016 at 16:57 -0700, Dk Jack wrote: > Another related question, can two protocol analyzer modules register for > the same dpd signature? Yes, if memory serves me right, adding two separate 'enable "..."' statements to a signature should work fine. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From dave.a.florek at gmail.com Wed Aug 10 09:26:49 2016 From: dave.a.florek at gmail.com (Dave Florek) Date: Wed, 10 Aug 2016 12:26:49 -0400 Subject: [Bro] PF_RING integration with Bro IDS post-install In-Reply-To: References: <20160809235419.GA7797@wifi154.sys.ICSI.Berkeley.EDU> Message-ID: Thank you. What 'configure' option would I use to activate PF_RING if I installed pf_ring from the ntop repo and not a source package? I'm not sure if /opt/pfring is referencing a program or directory for it to see certain files. On Tue, Aug 9, 2016 at 9:12 PM, Micha? Purzy?ski wrote: > Use a bro plugin for pfring. No need to rebuild anything, just that > plugin. Avoids libpcap. > > https://github.com/bro/bro-plugins > > On 10 Aug 2016, at 01:54, Johanna Amann wrote: > > Hello Dave, > > Is there a way to enable PF_RING after a Bro IDS install? From what I saw, > > the Bro IDS documentation ( > > https://www.bro.org/documentation/load-balancing.html) mentions the > > following pre-configuration to enable "pf_ring" pre-install but I wanted to > > know if it can be enabled post-install: > > > since Bro has to be compiled against the correct pfring libraries, this > sadly cannot be enabled without re-compiling and re-installing Bro. > > Johanna > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160810/bb3a237c/attachment-0001.html From tjtaubit at mtu.edu Wed Aug 10 09:59:45 2016 From: tjtaubit at mtu.edu (Trevor Taubitz) Date: Wed, 10 Aug 2016 10:59:45 -0600 Subject: [Bro] SMTP Data Fragmentation Problems Message-ID: So I'm having an odd problem that I can't seem to find any documentation on. I'm trying to use Bro to do some stuff with email monitoring, but I'm having some issues when it comes to data fragmentation. The test setup that I have is three servers: one DNS server for MX resolution, a sending SMTP server/client, and a receiving SMTP server with Bro running on it. The Bro server is using the default configuration. I'm sending emails to the receiving server, and they are showing up in the test user's mail just fine. Most of the time, Bro picks up this traffic no problem and puts the necessary log entries into smtp.log and files.log. The problem is that any time I try sending a large attachment (which amounts to any time that the SMTP data field needs to be fragmented across multiple packets), Bro doesn't seem to be picking it up. It will catch extremely small attachments, but won't even log emails that have to fragment. Is there any insight someone could give me about this? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160810/4e5cb2bc/attachment.html From jazoff at illinois.edu Wed Aug 10 10:22:40 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 10 Aug 2016 17:22:40 +0000 Subject: [Bro] SMTP Data Fragmentation Problems In-Reply-To: References: Message-ID: <5BC82DF8-36CA-422A-A5F5-79A6108F5F8C@illinois.edu> > On Aug 10, 2016, at 12:59 PM, Trevor Taubitz wrote: > > a receiving SMTP server with Bro running on it. This sounds like it could be the common invalid checksum issue. Is your reporter.log complaining about checksum errors? See this link for more info and some possible fixes: https://www.bro.org/documentation/faq.html#why-isn-t-bro-producing-the-logs-i-expect-a-note-about-checksums -- - Justin Azoff From hlin33 at illinois.edu Wed Aug 10 11:59:57 2016 From: hlin33 at illinois.edu (Hui Lin (Hugo) ) Date: Wed, 10 Aug 2016 13:59:57 -0500 Subject: [Bro] Can I make Bro ignore retransmitted TCP packets Message-ID: Hi I am not quite sure how Bro is handling retransmitted TCP packets at this moment. Is there a way for me to ignore retransmitted TCP packets? I am analysing an offline trace. Best, Hui Lin -- Hui Lin Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/) DEPEND (http://depend.csl.illinois.edu/) ECE, Uni. of Illinois at Urbana-Champaign -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160810/4c9d34b1/attachment.html From philosnef at yahoo.com Thu Aug 11 04:55:31 2016 From: philosnef at yahoo.com (philosnef) Date: Thu, 11 Aug 2016 11:55:31 +0000 (UTC) Subject: [Bro] ssdeep hashing References: <944518874.14707001.1470916531800.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <944518874.14707001.1470916531800.JavaMail.yahoo@mail.yahoo.com> Is there anything out there Bro wise that can do ssdeep hashing? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160811/1240d751/attachment.html From dhoelzer at enclaveforensics.com Thu Aug 11 05:30:18 2016 From: dhoelzer at enclaveforensics.com (=?UTF-8?Q?David_Hoelzer?=) Date: Thu, 11 Aug 2016 12:30:18 +0000 Subject: [Bro] ssdeep hashing In-Reply-To: <944518874.14707001.1470916531800.JavaMail.yahoo@mail.yahoo.com> References: <944518874.14707001.1470916531800.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <010001567994ede7-5e58a9a0-f9d2-4fff-aa4e-e7742e545ac4-000000@email.amazonses.com> Sounds like an interesting plugin to write. ? From: [mailto:bro-bounces at bro.org] On Behalf Of philosnef Sent: Thursday, August 11, 2016 7:56 AM To: bro at bro.org Subject: [Bro] ssdeep hashing ? Is there anything out there Bro wise that can do ssdeep hashing? Thanks. ? _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160811/7262146a/attachment.html From mabuchan at gmail.com Thu Aug 11 06:02:30 2016 From: mabuchan at gmail.com (Mark Buchanan) Date: Thu, 11 Aug 2016 08:02:30 -0500 Subject: [Bro] ssdeep hashing In-Reply-To: <010001567994ede7-5e58a9a0-f9d2-4fff-aa4e-e7742e545ac4-000000@email.amazonses.com> References: <944518874.14707001.1470916531800.JavaMail.yahoo.ref@mail.yahoo.com> <010001567994ede7-5e58a9a0-f9d2-4fff-aa4e-e7742e545ac4-000000@email.amazonses.com> Message-ID: I'm curious (and will admit, I haven't checked source), but is there a framework for handling hashing/file analysis, to allow for extensibility/experimentation with different mechanisms? Or is all the current hashing "hard coded"? Is this something that Bro threads, so it scales better? -- Mark Buchanan > On Aug 11, 2016, at 07:30, David Hoelzer wrote: > > Sounds like an interesting plugin to write. > > From: [mailto:bro-bounces at bro.org] On Behalf Of philosnef > Sent: Thursday, August 11, 2016 7:56 AM > To: bro at bro.org > Subject: [Bro] ssdeep hashing > > Is there anything out there Bro wise that can do ssdeep hashing? Thanks. > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160811/c4344f00/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2182 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160811/c4344f00/attachment.bin From philosnef at yahoo.com Thu Aug 11 08:18:34 2016 From: philosnef at yahoo.com (philosnef) Date: Thu, 11 Aug 2016 15:18:34 +0000 (UTC) Subject: [Bro] tcp off-path exploit References: <1556453793.14736351.1470928714319.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <1556453793.14736351.1470928714319.JavaMail.yahoo@mail.yahoo.com> Is it possible to flag these exploit attempts? From the look of things, it seems reasonable to think that the connection information in conn.log could be used to trace this, do to the very particular way it hands syn/ack requests.? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160811/ba30dbcd/attachment-0001.html From dave.a.florek at gmail.com Fri Aug 12 08:54:18 2016 From: dave.a.florek at gmail.com (Dave Florek) Date: Fri, 12 Aug 2016 11:54:18 -0400 Subject: [Bro] Is there a way to use "bro-cut -d" automatically during log parsing? Message-ID: Hi, Is there a way to use "bro-cut -d" automatically during log parsing so when I open the Bro Logs, they're all in EST instead of Epoch? Thanks in advance, -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160812/f09665cb/attachment.html From dnthayer at illinois.edu Fri Aug 12 10:34:33 2016 From: dnthayer at illinois.edu (Daniel Thayer) Date: Fri, 12 Aug 2016 12:34:33 -0500 Subject: [Bro] Is there a way to use "bro-cut -d" automatically during log parsing? In-Reply-To: References: Message-ID: <57AE08A9.90104@illinois.edu> You could create a shell script wrapper that contains this: bro-cut -d "$@" And then just use that script instead of bro-cut. On 08/12/2016 10:54 AM, Dave Florek wrote: > Hi, > > Is there a way to use "bro-cut -d" automatically during log parsing so > when I open the Bro Logs, they're all in EST instead of Epoch? > > Thanks in advance, > > From dave.a.florek at gmail.com Fri Aug 12 12:14:51 2016 From: dave.a.florek at gmail.com (Dave Florek) Date: Fri, 12 Aug 2016 15:14:51 -0400 Subject: [Bro] Is there a way to use "bro-cut -d" automatically during log parsing? In-Reply-To: <57AE08A9.90104@illinois.edu> References: <57AE08A9.90104@illinois.edu> Message-ID: Thank you, I'll take a look at this. On Fri, Aug 12, 2016 at 1:34 PM, Daniel Thayer wrote: > You could create a shell script wrapper that contains this: > bro-cut -d "$@" > > And then just use that script instead of bro-cut. > > > > On 08/12/2016 10:54 AM, Dave Florek wrote: > >> Hi, >> >> Is there a way to use "bro-cut -d" automatically during log parsing so >> when I open the Bro Logs, they're all in EST instead of Epoch? >> >> Thanks in advance, >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160812/e7746d1d/attachment.html From jdopheid at illinois.edu Sun Aug 14 15:37:28 2016 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Sun, 14 Aug 2016 22:37:28 +0000 Subject: [Bro] Reminder to book BroCon travel, hotel In-Reply-To: <379EA04F-64BF-4BA3-945D-0B28726A050C@illinois.edu> References: <379EA04F-64BF-4BA3-945D-0B28726A050C@illinois.edu> Message-ID: <7EFD7D614A2BB84ABEA19B2CEDD246581C310EA1@CITESMBX5.ad.uillinois.edu> Bro Community, Friendly reminder to book your hotel if you haven't done so yet. Also, Lone Star's site is showing some of the dates as "sold out" in the calendar. We still have rooms available for booking (Group code "1609BROCON"). The sold out status is confusing, but if you look further down the page under "Available Rooms" you'll see a reservation is still available for those dates. Click "Book Room" to proceed with your reservation. We apologize for the confusion. Email us at info at bro.org if you have any more questions. Thanks, Jeannette ------ Jeannette M. Dopheide Bro Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign ________________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Dopheide, Jeannette M [jdopheid at illinois.edu] Sent: Wednesday, August 03, 2016 10:49 AM To: bro at bro.org Subject: [Bro] Reminder to book BroCon travel, hotel Bro Community, If you?re planning on attending BroCon this year, consider taking advantage of our reserved room block at the Lone Star Court before August 22nd, details here: https://www.bro.org/community/brocon2016.html#hotelinformation And don?t forget to use the group code even if you?re booking over the phone. We need the community?s help to meet our hotel contract obligations to continue to negotiate special rates in the future. Thanks, The Bro Team ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From ansaf_130 at yahoo.com Mon Aug 15 00:34:54 2016 From: ansaf_130 at yahoo.com (Aneela Safdar) Date: Mon, 15 Aug 2016 07:34:54 +0000 (UTC) Subject: [Bro] Bro timestamp JSON::TS_ISO8601 - how to get system time in log files References: <45417989.14955581.1471246494265.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <45417989.14955581.1471246494265.JavaMail.yahoo@mail.yahoo.com> Hi, I am using JSON::TS_ISO8601 as time stamp format for json formatted logs. The date part is working fine, i.e. receiving current date but is there any settings for GMT offset as I am not getting the correct time of my timezone. Its even not taking it from my system. Any suggestions??Regards, Aneela Safdar -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160815/8371d100/attachment.html From seth at icir.org Mon Aug 15 06:21:53 2016 From: seth at icir.org (Seth Hall) Date: Mon, 15 Aug 2016 09:21:53 -0400 Subject: [Bro] Bro timestamp JSON::TS_ISO8601 - how to get system time in log files In-Reply-To: <45417989.14955581.1471246494265.JavaMail.yahoo@mail.yahoo.com> References: <45417989.14955581.1471246494265.JavaMail.yahoo.ref@mail.yahoo.com> <45417989.14955581.1471246494265.JavaMail.yahoo@mail.yahoo.com> Message-ID: > On Aug 15, 2016, at 3:34 AM, Aneela Safdar wrote: > > I am using JSON::TS_ISO8601 as time stamp format for json formatted logs. The date part is working fine, i.e. receiving current date but is there any settings for GMT offset as I am not getting the correct time of my timezone. Its even not taking it from my system. There is no mechanism right now to do anything but UTC in the json output. Are you sure that you want data with timezones? Most people are not served well with this approach since most systems prefer to store everything in UTC internally and change the output time to match whatever the viewer would like. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From vladg at illinois.edu Tue Aug 16 06:30:45 2016 From: vladg at illinois.edu (Vlad Grigorescu) Date: Tue, 16 Aug 2016 09:30:45 -0400 Subject: [Bro] ssdeep hashing In-Reply-To: References: <944518874.14707001.1470916531800.JavaMail.yahoo.ref@mail.yahoo.com> <010001567994ede7-5e58a9a0-f9d2-4fff-aa4e-e7742e545ac4-000000@email.amazonses.com> Message-ID: Yes and no. :-) There's a way to do this in C++, but there's no script framework for it. The main reason is performance -- this is really something that needs to happen in the core (that is, in C++) as opposed to in a script. Relevant examples would be: https://github.com/bro/bro/blob/master/src/OpaqueVal.cc https://github.com/bro/bro/blob/master/src/file_analysis/analyzer/hash/Hash.h --Vlad Mark Buchanan writes: > I'm curious (and will admit, I haven't checked source), but is there a framework for handling hashing/file analysis, to allow for extensibility/experimentation with different mechanisms? Or is all the current hashing "hard coded"? Is this something that Bro threads, so it scales better? > > -- > Mark Buchanan > >> On Aug 11, 2016, at 07:30, David Hoelzer wrote: >> >> Sounds like an interesting plugin to write. >> >> From: [mailto:bro-bounces at bro.org] On Behalf Of philosnef >> Sent: Thursday, August 11, 2016 7:56 AM >> To: bro at bro.org >> Subject: [Bro] ssdeep hashing >> >> Is there anything out there Bro wise that can do ssdeep hashing? Thanks. >> >> _______________________________________________ >> >> Bro mailing list >> >> bro at bro-ids.org >> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 800 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160816/b1ff94ba/attachment.bin From philosnef at yahoo.com Tue Aug 16 08:18:54 2016 From: philosnef at yahoo.com (philosnef) Date: Tue, 16 Aug 2016 15:18:54 +0000 (UTC) Subject: [Bro] secure boot key hash References: <602732918.17176426.1471360734775.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <602732918.17176426.1471360734775.JavaMail.yahoo@mail.yahoo.com> Does anyone have the hash for the MS secure boot key? I want to put this in an intel feed. That is borderline useless, but knowing who is pulling that key is of value however. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160816/031204b6/attachment.html From jdopheid at illinois.edu Thu Aug 18 08:40:50 2016 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Thu, 18 Aug 2016 15:40:50 +0000 Subject: [Bro] Reminder to book BroCon travel, hotel In-Reply-To: <7EFD7D614A2BB84ABEA19B2CEDD246581C310EA1@CITESMBX5.ad.uillinois.edu> References: <379EA04F-64BF-4BA3-945D-0B28726A050C@illinois.edu>, <7EFD7D614A2BB84ABEA19B2CEDD246581C310EA1@CITESMBX5.ad.uillinois.edu> Message-ID: <7EFD7D614A2BB84ABEA19B2CEDD246581C313870@CITESMBX5.ad.uillinois.edu> Bro Community, If you're planning on staying at the Lone Star Court, be sure to book your hotel before Monday, August 22nd. The hotel will release our block of rooms for general sale. Still on the fence about attending? Check out our agenda: https://www.bro.org/community/brocon2016.html#agenda ------ Jeannette M. Dopheide Bro Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign ________________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Dopheide, Jeannette M [jdopheid at illinois.edu] Sent: Sunday, August 14, 2016 5:37 PM To: bro at bro.org Subject: Re: [Bro] Reminder to book BroCon travel, hotel Bro Community, Friendly reminder to book your hotel if you haven't done so yet. Also, Lone Star's site is showing some of the dates as "sold out" in the calendar. We still have rooms available for booking (Group code "1609BROCON"). The sold out status is confusing, but if you look further down the page under "Available Rooms" you'll see a reservation is still available for those dates. Click "Book Room" to proceed with your reservation. We apologize for the confusion. Email us at info at bro.org if you have any more questions. Thanks, Jeannette ------ Jeannette M. Dopheide Bro Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign ________________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Dopheide, Jeannette M [jdopheid at illinois.edu] Sent: Wednesday, August 03, 2016 10:49 AM To: bro at bro.org Subject: [Bro] Reminder to book BroCon travel, hotel Bro Community, If you?re planning on attending BroCon this year, consider taking advantage of our reserved room block at the Lone Star Court before August 22nd, details here: https://www.bro.org/community/brocon2016.html#hotelinformation And don?t forget to use the group code even if you?re booking over the phone. We need the community?s help to meet our hotel contract obligations to continue to negotiate special rates in the future. Thanks, The Bro Team ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Thu Aug 18 14:12:14 2016 From: johanna at icir.org (Johanna Amann) Date: Thu, 18 Aug 2016 14:12:14 -0700 Subject: [Bro] Bro 2.5 Beta available Message-ID: <20160818211214.GA86796@Beezling.local> The beta version for Bro 2.5 is now available for testing and can be downloaded at: https://bro.org/download/index.html Binary packages also are available at: https://bro.org/download/beta-packages.html Some of the notable changes since the 2.4 release are: - Bro now includes the NetControl framework. This framework allows easy interaction with hard- and software switches, firewalls, etc. - Support for the SMB protocol (SMB1 and SMB2), including GSSAPI and NTLM. - Support for the remote framebuffer protocol (RFB), that is used by VNC servers for remote graphical display. - The Intelligence framework was refactored and extended. It now supports, for example subnet indicators and item deletion/expiration. For more information see the NEWS and CHANGES files: https://www.bro.org/documentation/beta/NEWS.bro.html https://www.bro.org/documentation/beta/CHANGES.bro.txt Feel free to use this mailing list or the bug tracker (tracker.bro.org) to provide feedback or report problems. Johanna From lutfioduncuoglu at gmail.com Fri Aug 19 05:15:34 2016 From: lutfioduncuoglu at gmail.com (Lutfi Oduncuoglu) Date: Fri, 19 Aug 2016 15:15:34 +0300 Subject: [Bro] Bro Phishnig Message-ID: Hello, I am newbie on BRO and I am trying to analyze phishing events with BRO. I did some googling and found the link below. https://github.com/hosom/bro-phishing I extract the file put into /bro/bae/protocols folder. I edited the init-default.bro script and added @load base/protocols/phishing. I can see the phishing when I check with scripts command in broctl environment. Since It isn't a standlane script I do not know how to test it with command line. Now is it working or not how can I test it? Best Regards, Lutfi -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160819/766ba6ee/attachment.html From jdopheid at illinois.edu Fri Aug 19 12:29:08 2016 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Fri, 19 Aug 2016 19:29:08 +0000 Subject: [Bro] BroCon hotel: 3 reservations short of our commitment Message-ID: Hello again Bro community, To those of you who have booked your stay, thank you! As of earlier today we are 9 room nights, i.e. about 3 reservations, short of meeting our minimum requirement for our hotel block of rooms. If any of you are planning on staying at the Lone Star Court but haven?t booked yet, please do so by Monday the 22nd. This will really help us meet our commitment without the Bro Project having to pay penalties. See you in September, The Bro Team ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign On 8/18/16, 10:40 AM, "bro-bounces at bro.org on behalf of Dopheide, Jeannette M" wrote: Bro Community, If you're planning on staying at the Lone Star Court, be sure to book your hotel before Monday, August 22nd. The hotel will release our block of rooms for general sale. Still on the fence about attending? Check out our agenda: https://www.bro.org/community/brocon2016.html#agenda ------ Jeannette M. Dopheide Bro Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign ________________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Dopheide, Jeannette M [jdopheid at illinois.edu] Sent: Sunday, August 14, 2016 5:37 PM To: bro at bro.org Subject: Re: [Bro] Reminder to book BroCon travel, hotel Bro Community, Friendly reminder to book your hotel if you haven't done so yet. Also, Lone Star's site is showing some of the dates as "sold out" in the calendar. We still have rooms available for booking (Group code "1609BROCON"). The sold out status is confusing, but if you look further down the page under "Available Rooms" you'll see a reservation is still available for those dates. Click "Book Room" to proceed with your reservation. We apologize for the confusion. Email us at info at bro.org if you have any more questions. Thanks, Jeannette ------ Jeannette M. Dopheide Bro Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign ________________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Dopheide, Jeannette M [jdopheid at illinois.edu] Sent: Wednesday, August 03, 2016 10:49 AM To: bro at bro.org Subject: [Bro] Reminder to book BroCon travel, hotel Bro Community, If you?re planning on attending BroCon this year, consider taking advantage of our reserved room block at the Lone Star Court before August 22nd, details here: https://www.bro.org/community/brocon2016.html#hotelinformation And don?t forget to use the group code even if you?re booking over the phone. We need the community?s help to meet our hotel contract obligations to continue to negotiate special rates in the future. Thanks, The Bro Team ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From newfire.bw at gmail.com Sat Aug 20 02:47:42 2016 From: newfire.bw at gmail.com (Bowen Li) Date: Sat, 20 Aug 2016 17:47:42 +0800 Subject: [Bro] Get interface from bro scripts Message-ID: Hey all, I am running a bro cluster and I need to distinguish different interface in bro scripts. Is there any way or build in functions to get the interface used by the current thread in bro scripts? Any insight would be helpful. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160820/27968c95/attachment.html From rmkml at ligfy.org Sat Aug 20 14:38:54 2016 From: rmkml at ligfy.org (rmkml) Date: Sat, 20 Aug 2016 23:38:54 +0200 (CEST) Subject: [Bro] Bro 2.5 Beta available In-Reply-To: <20160818211214.GA86796@Beezling.local> References: <20160818211214.GA86796@Beezling.local> Message-ID: Thx all for awesome Bro project! Could you check if BIT-1562 (lock on old pcap file) fix is present on 2.5 beta version please ? (because first test repeat lock) Best Regards @Rmkml On Thu, 18 Aug 2016, Johanna Amann wrote: > The beta version for Bro 2.5 is now available for testing and can be downloaded at: > > https://bro.org/download/index.html > > Binary packages also are available at: > > https://bro.org/download/beta-packages.html > > Some of the notable changes since the 2.4 release are: > > - Bro now includes the NetControl framework. This framework allows easy > interaction with hard- and software switches, firewalls, etc. > > - Support for the SMB protocol (SMB1 and SMB2), including GSSAPI and NTLM. > > - Support for the remote framebuffer protocol (RFB), that is used by VNC > servers for remote graphical display. > > - The Intelligence framework was refactored and extended. It now supports, for > example subnet indicators and item deletion/expiration. > > For more information see the NEWS and CHANGES files: > https://www.bro.org/documentation/beta/NEWS.bro.html > https://www.bro.org/documentation/beta/CHANGES.bro.txt > > Feel free to use this mailing list or the bug tracker (tracker.bro.org) to > provide feedback or report problems. > > Johanna > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From jazoff at illinois.edu Sat Aug 20 19:08:03 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Sun, 21 Aug 2016 02:08:03 +0000 Subject: [Bro] Bro 2.5 Beta available In-Reply-To: References: <20160818211214.GA86796@Beezling.local> Message-ID: > On Aug 20, 2016, at 5:38 PM, rmkml wrote: > > Thx all for awesome Bro project! > > Could you check if BIT-1562 (lock on old pcap file) fix is present on 2.5 beta version please ? > (because first test repeat lock) > > Best Regards > @Rmkml Yes.. this was fixed back in May. What do you mean by "first test repeat lock" ? Which pcap did you test on which bro version? [jazoff at bro-test tmp]$ bro --version bro version 2.5-beta [jazoff at bro-test tmp]$ time bro -r bro241lock.pcap 1243601416.209199 warning in /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54: Your trace file likely has invalid IP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command-line option or toggling the 'ignore_checksums' variable. Alternatively, disable checksum offloading by the network adapter to ensure Bro analyzes the actual checksums that are transmitted. real 0m1.157s user 0m1.079s sys 0m0.050s [jazoff at bro-test tmp]$ time bro -C -r bro241lock.pcap real 0m0.936s user 0m0.863s sys 0m0.048s -- - Justin Azoff From Ben.McDowall at spark.co.nz Sun Aug 21 02:58:07 2016 From: Ben.McDowall at spark.co.nz (Ben McDowall) Date: Sun, 21 Aug 2016 09:58:07 +0000 Subject: [Bro] Get interface from bro scripts In-Reply-To: References: Message-ID: <1471773487688.80723@spark.co.nz> ?I too am looking for this type of functionality. Currently I have one box that has about 8 interfaces on it, all looking at different network segments, its important I can identify the interface it came from, ideally I could split the logs to sit in different folders too. Is there any plans for this in the roadmap? ________________________________ From: bro-bounces at bro.org on behalf of Bowen Li Sent: Saturday, 20 August 2016 9:47 p.m. To: bro at bro.org Subject: [Bro] Get interface from bro scripts Hey all, I am running a bro cluster and I need to distinguish different interface in bro scripts. Is there any way or build in functions to get the interface used by the current thread in bro scripts? Any insight would be helpful. This communication, including any attachments, is confidential. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160821/72317530/attachment.html From rmkml at ligfy.org Mon Aug 22 12:19:39 2016 From: rmkml at ligfy.org (rmkml) Date: Mon, 22 Aug 2016 21:19:39 +0200 (CEST) Subject: [Bro] Bro 2.5 Beta available In-Reply-To: References: <20160818211214.GA86796@Beezling.local> Message-ID: Thx Justin, I am wrong, it's my bad, sorry for noise. Happy Bro Testing @Rmkml On Sun, 21 Aug 2016, Azoff, Justin S wrote: > >> On Aug 20, 2016, at 5:38 PM, rmkml wrote: >> >> Thx all for awesome Bro project! >> >> Could you check if BIT-1562 (lock on old pcap file) fix is present on 2.5 beta version please ? >> (because first test repeat lock) >> >> Best Regards >> @Rmkml > > Yes.. this was fixed back in May. What do you mean by "first test repeat lock" ? Which pcap did you test on which bro version? > > [jazoff at bro-test tmp]$ bro --version > bro version 2.5-beta > > [jazoff at bro-test tmp]$ time bro -r bro241lock.pcap > 1243601416.209199 warning in /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54: Your trace file likely has invalid IP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command-line option or toggling the 'ignore_checksums' variable. Alternatively, disable checksum offloading by the network adapter to ensure Bro analyzes the actual checksums that are transmitted. > > real 0m1.157s > user 0m1.079s > sys 0m0.050s > > [jazoff at bro-test tmp]$ time bro -C -r bro241lock.pcap > > real 0m0.936s > user 0m0.863s > sys 0m0.048s > > -- > - Justin Azoff > > From navraj42 at gmail.com Wed Aug 24 11:29:21 2016 From: navraj42 at gmail.com (Navraj Singh) Date: Wed, 24 Aug 2016 12:29:21 -0600 Subject: [Bro] Bro connections v. NetFlow Message-ID: Hi, I was wondering what some major differences are between the concept of a 'connection' in Bro and a a 'flow' in NetFlow. Or are they essentially the same concept? If this requires a detailed answer, a reference would be very helpful! Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160824/0f752ebf/attachment.html From dhoelzer at sans.org Wed Aug 24 17:48:09 2016 From: dhoelzer at sans.org (Hoelzer, Dave) Date: Thu, 25 Aug 2016 00:48:09 +0000 Subject: [Bro] Bro connections v. NetFlow In-Reply-To: References: Message-ID: Netflow connections are generally logged and a new connection recorded if they exceed 30 minutes. That?s one. ??????????????????? David Hoelzer Fellow, SANS Institute Dean of Faculty, SANS Technology Institute On August 24, 2016 at 1:45:07 PM, Navraj Singh (navraj42 at gmail.com) wrote: Hi, I was wondering what some major differences are between the concept of a 'connection' in Bro and a a 'flow' in NetFlow. Or are they essentially the same concept? If this requires a detailed answer, a reference would be very helpful! Thank you! _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160825/a5fa7025/attachment.html From Alec.Waters at dataline.co.uk Thu Aug 25 03:16:04 2016 From: Alec.Waters at dataline.co.uk (Alec Waters) Date: Thu, 25 Aug 2016 10:16:04 +0000 Subject: [Bro] Bro connections v. NetFlow In-Reply-To: References: Message-ID: <8350146BADDCE04480B969B36967473D13F090CB@ZEUS.olympus.dataline.co.uk> We set our routers to export flows after one minute if they?re still in progress (it?ll continue to send a flow export every minute until it?s complete). More info here: https://www.manageengine.com/products/netflow/help/cisco-netflow/cisco-ios-netflow.html (?ip flow-cache timeout active 1? is the command to use) This means that, AFAIUI, Netflow can be made to be more timely than Bro. Bro will only output a bro_conn when the flow has been deemed to have finished. Also, Netflow exports are unidirectional ? you get separate flow exports for A->B and B->A. With Bro, a bro_conn logs traffic in both directions. alec From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Hoelzer, Dave Sent: 25 August 2016 01:48 To: Navraj Singh; bro at bro.org Subject: Re: [Bro] Bro connections v. NetFlow Netflow connections are generally logged and a new connection recorded if they exceed 30 minutes. That?s one. ??????????????????? David Hoelzer Fellow, SANS Institute Dean of Faculty, SANS Technology Institute On August 24, 2016 at 1:45:07 PM, Navraj Singh (navraj42 at gmail.com) wrote: Hi, I was wondering what some major differences are between the concept of a 'connection' in Bro and a a 'flow' in NetFlow. Or are they essentially the same concept? If this requires a detailed answer, a reference would be very helpful! Thank you! _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160825/38b049db/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3901 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160825/38b049db/attachment.bin From jeb446 at msstate.edu Thu Aug 25 14:47:45 2016 From: jeb446 at msstate.edu (John Bradley) Date: Thu, 25 Aug 2016 16:47:45 -0500 (CDT) Subject: [Bro] Duplicate Entries - using PF_RING Message-ID: I've been trying to optimize our bro cluster and noticed that, even after building PF_RING and Bro according to various documentations, I still get duplicated entries. The kernel modules are loaded, BRO is built to support PF_RING. This is what I see when I check /pro/net/pf_ring/dev/eth2/info Name: eth2 Index: 4 Address: [mac address] Polling Mode: NAPI Type: Ethernet Family: Standard NIC # Bound Sockets: 0 Max # TX Queues: 40 # Used RX Queues: 40 Help would be appreciated. --- John Bradley ITS Network Services Systems Programmer 5-0298 From jazoff at illinois.edu Thu Aug 25 15:06:23 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 25 Aug 2016 22:06:23 +0000 Subject: [Bro] Duplicate Entries - using PF_RING In-Reply-To: References: Message-ID: <0BED0F4C-7162-458B-91FD-E56CF491A888@illinois.edu> > On Aug 25, 2016, at 5:47 PM, John Bradley wrote: > > # Bound Sockets: 0 Bro is not using pf_ring. This is what it looks like when bro is using pf_ring: [root at nids-dev3 ~]# ldd `which bro`|grep pcap libpcap.so.1 => /opt/pfring/lib/libpcap.so.1 (0x00007f91abc5a000) [root at nids-dev3 ~]# cat /proc/net/pf_ring/dev/p1p1/info Name: p1p1 Index: 4 Address: A0:36:9F:27:4C:48 Polling Mode: NAPI Type: Ethernet Family: Standard NIC # Bound Sockets: 7 Max # TX Queues: 32 # Used RX Queues: 32 [root at nids-dev3 ~]# grep App /proc/net/pf_ring/*p1p1* /proc/net/pf_ring/32471-p1p1.7619:Appl. Name : bro-p1p1 /proc/net/pf_ring/32506-p1p1.7620:Appl. Name : bro-p1p1 /proc/net/pf_ring/32523-p1p1.7623:Appl. Name : bro-p1p1 /proc/net/pf_ring/32535-p1p1.7622:Appl. Name : bro-p1p1 /proc/net/pf_ring/32537-p1p1.7624:Appl. Name : bro-p1p1 /proc/net/pf_ring/32548-p1p1.7614:Appl. Name : bro-p1p1 /proc/net/pf_ring/32563-p1p1.7616:Appl. Name : bro-p1p1 [root at bro-dev bro]# tail etc/node.cfg [nids-dev3a] type=worker interface=p1p1 lb_method=pf_ring lb_procs=7 pin_cpus=2,3,4,5,6,7,8 ... -- - Justin Azoff From joshi.pradyumna at gmail.com Fri Aug 26 04:14:51 2016 From: joshi.pradyumna at gmail.com (Pradyumna Joshi) Date: Fri, 26 Aug 2016 16:44:51 +0530 Subject: [Bro] Enable/Disable (Turn on/off) Bro crash reports Message-ID: Is it possible to turn on/off Bro crash reports on demand? We are having a peculiar situation where-in power goes off frequently and bro instances are crashed often. Over a period of time, a lot of space is occupied by bro crash reports. Of course, there are other options to reclaim the crash report space. But, still, I am curious to know if Bro supports turning on/off bro crash reports on demand. I very much appreciate Bro crash reports as these offer great insights into the issue. So, I would not like to compile bro without "--enable-debug" option. -- Pradyumna Joshi -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160826/ec450096/attachment.html From al.brocino1 at gmail.com Fri Aug 26 09:10:33 2016 From: al.brocino1 at gmail.com (al brocino) Date: Fri, 26 Aug 2016 11:10:33 -0500 Subject: [Bro] Fwd: File Extraction In-Reply-To: References: <20160803194732.GA7211@wifi154.sys.ICSI.Berkeley.EDU> Message-ID: Thanks Johanna, I made your recommended change and am still getting the error, see detail below: file-extract.bro script > global ext_map:table[string] of string = { ["application/x/dosexec"] = > "exe", you probably want application/x-dosexec here, not x/dosexec. That might already be enough to fix this. Changed: file-extract.bro global ext_map: table[string] of string = { ["application/x-dosexec"] = "exe", ["text/plain"] = "txt", ["image/jpeg"] = "jpg", ["image/png"] = "png", ["text/html"] = "html", } &default =""; Uncomment #@load ./file-extract-http-local.bro and #@load ./file-extract-types.bro: _load_.bro # File extractions (/application\/.*) -- This has changed significantly in 2.2 @load ./file-extract-http-local.bro @load ./file-extract-types.bro @load ./bro-file-extract I get this error again: manager scripts failed. internal warning in /usr/local/bro/share/bro/site/ ./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts fatal error in /usr/local/bro/share/bro/site/ ./custom/./file-extract-http-local.bro, line 7: can't find base/protocols/http/file-ident proxy scripts failed. internal warning in /usr/local/bro/share/bro/site/ ./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts fatal error in /usr/local/bro/share/bro/site/ ./custom/./file-extract-http-local.bro, line 7: can't find base/protocols/http/file-ident enm1-eth1-httpproxy scripts failed. internal warning in /usr/local/bro/share/bro/site/ ./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts fatal error in /usr/local/bro/share/bro/site/ ./custom/./file-extract-http-local.bro, line 7: can't find base/protocols/http/file-ident enm2-eth2-httpinternal scripts failed. internal warning in /usr/local/bro/share/bro/site/ ./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts fatal error in /usr/local/bro/share/bro/site/ ./custom/./file-extract-http-local.bro, line 7: can't find base/protocols/http/file-ident enm3-eth3-collector scripts failed. internal warning in /usr/local/bro/share/bro/site/ ./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts fatal error in /usr/local/bro/share/bro/site/ ./custom/./file-extract-http-local.bro, line 7: can't find base/protocols/http/file-ident enm4-eth5-dns scripts failed. internal warning in /usr/local/bro/share/bro/site/ ./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts fatal error in /usr/local/bro/share/bro/site/ ./custom/./file-extract-http-local.bro, line 7: can't find base/protocols/http/file-ident enm5-eth6-syslog scripts failed. internal warning in /usr/local/bro/share/bro/site/ ./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts fatal error in /usr/local/bro/share/bro/site/ ./custom/./file-extract-http-local.bro, line 7: can't find base/protocols/http/file-ident Here's the script that it's failing on: file-extract-http-local.bro @load base/protocols/http/main @load base/protocols/http/file-ident @load base/utils/files module HTTP; export { ## Pattern of file mime types to extract from HTTP response entity bodies. const extract_file_types_local = /NO_DEFAULT/ &redef; ## The on-disk prefix for files to be extracted from HTTP entity bodies. const extraction_prefix_local = "http-item" &redef; redef record Info += { ## On-disk file where the response body was extracted to. extraction_file_local: file &log &optional; ## Indicates if the response body is to be extracted or not. Must be ## set before or by the first :bro:id:`http_entity_data` event for the ## content. extract_file_local: bool &default=F; }; } # Define local sources to ignore file extract global http_extract_file_ignore: set[subnet] = { 172.16.0.0/12, # Internal FRS, trusted destination 10.0.0.0/8, # Internal FRS, trusted destination }; event http_entity_data(c: connection, is_orig: bool, length: count, data: string) &priority=-5 { # Client body extraction is not currently supported in this script. if ( is_orig ) return; # We do not want to extract files from internal to internal hosts if ( c$id$resp_h in http_extract_file_ignore ) return; if ( c$http$first_chunk ) { if ( c$http?$mime_type && extract_file_types_local in c$http$mime_type ) { c$http$extract_file_local = T; } if ( c$http$extract_file_local ) { local suffix = fmt("%s_%d.dat", is_orig ? "orig" : "resp", c$http_state$current_response); local fname = generate_extraction_filename(extraction_prefix_local, c, suffix); c$http$extraction_file_local = open(fname); enable_raw_output(c$http$ extraction_file_local); } } if ( c$http?$extraction_file_local ) print c$http$extraction_file_local, data; } event http_end_entity(c: connection, is_orig: bool) { if ( c$http?$extraction_file_local ) close(c$http$extraction_file_local); } Ideas? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160826/d83a6cc9/attachment-0001.html From dnthayer at illinois.edu Fri Aug 26 12:08:30 2016 From: dnthayer at illinois.edu (Daniel Thayer) Date: Fri, 26 Aug 2016 14:08:30 -0500 Subject: [Bro] Enable/Disable (Turn on/off) Bro crash reports In-Reply-To: References: Message-ID: <2f1f0c85-a873-ca7e-d48e-9821da7ce247@illinois.edu> BroControl doesn't have an option to turn off crash reports. However, one easy way to reclaim disk space is to run the command: broctl cleanup --all This will remove all files and directories in PREFIX/spool/tmp/ on all machines in your Bro cluster (PREFIX is your Bro install directory, such as /usr/local/bro). -Daniel On 8/26/16 6:14 AM, Pradyumna Joshi wrote: > Is it possible to turn on/off Bro crash reports on demand? > > We are having a peculiar situation where-in power goes off frequently > and bro instances are crashed often. Over a period of time, a lot of > space is occupied by bro crash reports. > > Of course, there are other options to reclaim the crash report space. > But, still, I am curious to know if Bro supports turning on/off bro > crash reports on demand. > > I very much appreciate Bro crash reports as these offer great insights > into the issue. So, I would not like to compile bro without > "--enable-debug" option. > > > -- > Pradyumna Joshi From pcain at coopercain.com Sat Aug 27 09:22:19 2016 From: pcain at coopercain.com (Patrick Cain) Date: Sat, 27 Aug 2016 12:22:19 -0400 Subject: [Bro] Revisiting CEF formatted BRO Logs In-Reply-To: References: Message-ID: <13b601d2007f$30c32e30$92498a90$@coopercain.com> Hi, In Arcsight speak: You could also just create a flex connector to read the ascii/json bro logs and spit out CEF. The easy way is to create one flex for each file you're reading; the much more fun way is to craft one big one that handles all the different files using a multi-file reader (There is an old project on github that did this.). I took the one-flex-per-filetype approach. Took a few hours to get logs flowing. Pat From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Ward Sladek Sent: Saturday, July 30, 2016 11:15 AM To: Ludwig Goon ; bro at bro.org Subject: Re: [Bro] Revisiting CEF formatted BRO Logs If I were in your shoes and assuming it's possible to add the sensor ID/name to the bro logs, I would just add that one field (keeping the same format, etc) and not rewrite everything for CEF. Then I would press HP support to give me the encrypted bro parser (they have given me several parsers in the past) and write a parser override to account for the new sensor/worker field. Sorry this doesn't answer your question directly, but maybe this route is an option for you. _____ From: bro-bounces at bro.org > on behalf of Ludwig Goon > Sent: Thursday, July 28, 2016 9:53 AM To: bro at bro.org Subject: [Bro] Revisiting CEF formatted BRO Logs Can someone from the community provide more information or examples of using log writer to create CEF formatted logs for consumption with Arcsight SIEMs? it seems that we can not customize arcsight connectors for BRO logs however since arcsight can accept CEF events directly I would like to experiment with directly sending CEF formatted BRO events from the standard log set. Additionally I have 5 BRO sensors and would like to tag each event with the BRO sensor's hostname before sending it to arc sight. The default logs do not allow that modification and documentation is not the greatest. If you want to do this in Arcsight via the connector, which is a version or two behind, the connector will not allow the adding of the hostname. So I have attempted to write PERL and PYTHON converters but the performance of tailing logs and sending all events is challenging. Also using brocut requires scripting and again not sure if I am sending ALL log events. In previous questions to the forum the answer was using the logging framework however I have not seen anymore content on this subject. Thus here is my formal request: Can someone show how to use the logging framework to convert or have bro output the http.log into CEF format? Also can I add custom fields such as sensor-name and the end of the event or at the beginning near CEF:0. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160827/623fcd94/attachment.html From al.brocino1 at gmail.com Sat Aug 27 10:37:35 2016 From: al.brocino1 at gmail.com (al brocino) Date: Sat, 27 Aug 2016 12:37:35 -0500 Subject: [Bro] File Extraction In-Reply-To: <20160803194732.GA7211@wifi154.sys.ICSI.Berkeley.EDU> References: <20160803194732.GA7211@wifi154.sys.ICSI.Berkeley.EDU> Message-ID: Thanks Johanna, *Adding additional information:* We are going to upgrade from 2.3.2 but have not yet. *I made your recommended change and am still getting the error, see detail below:* file-extract.bro script > global ext_map:table[string] of string = { ["application/x/dosexec"] = > "exe", you probably want application/x-dosexec here, not x/dosexec. That might already be enough to fix this. *Changed: * file-extract.bro global ext_map: table[string] of string = { ["application/x-dosexec"] = "exe", ["text/plain"] = "txt", ["image/jpeg"] = "jpg", ["image/png"] = "png", ["text/html"] = "html", } &default =""; *Un-comment #@load ./file-extract-http-local.bro and #@load ./file-extract-types.bro:* _load_.bro # File extractions (/application\/.*) -- This has changed significantly in 2.2 @load ./file-extract-http-local.bro @load ./file-extract-types.bro @load ./bro-file-extract * I get this error again:* manager scripts failed. internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can't find base/protocols/http/file-ident proxy scripts failed. internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can't find base/protocols/http/file-ident enm1-eth1-httpproxy scripts failed. internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can't find base/protocols/http/file-ident enm2-eth2-httpinternal scripts failed. internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can't find base/protocols/http/file-ident enm3-eth3-collector scripts failed. internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can't find base/protocols/http/file-ident enm4-eth5-dns scripts failed. internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can't find base/protocols/http/file-ident enm5-eth6-syslog scripts failed. internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can't find base/protocols/http/file-ident *Here's the script that it's failing on:* file-extract-http-local.bro @load base/protocols/http/main @load base/protocols/http/file-ident @load base/utils/files module HTTP; export { ## Pattern of file mime types to extract from HTTP response entity bodies. const extract_file_types_local = /NO_DEFAULT/ &redef; ## The on-disk prefix for files to be extracted from HTTP entity bodies. const extraction_prefix_local = "http-item" &redef; redef record Info += { ## On-disk file where the response body was extracted to. extraction_file_local: file &log &optional; ## Indicates if the response body is to be extracted or not. Must be ## set before or by the first :bro:id:`http_entity_data` event for the ## content. extract_file_local: bool &default=F; }; } # Define local sources to ignore file extract global http_extract_file_ignore: set[subnet] = { 192.168.2.0.0/24, # Internal Seminal1, trusted destination 192.168.1.0/24, # Internal Seminal2, trusted destination }; event http_entity_data(c: connection, is_orig: bool, length: count, data: string) &priority=-5 { # Client body extraction is not currently supported in this script. if ( is_orig ) return; # We do not want to extract files from internal to internal hosts if ( c$id$resp_h in http_extract_file_ignore ) return; if ( c$http$first_chunk ) { if ( c$http?$mime_type && extract_file_types_local in c$http$mime_type ) { c$http$extract_file_local = T; } if ( c$http$extract_file_local ) { local suffix = fmt("%s_%d.dat", is_orig ? "orig" : "resp", c$http_state$current_response); local fname = generate_extraction_filename(extraction_prefix_local, c, suffix); c$http$extraction_file_local = open(fname); enable_raw_output(c$http$extraction_file_local); } } if ( c$http?$extraction_file_local ) print c$http$extraction_file_local, data; } event http_end_entity(c: connection, is_orig: bool) { if ( c$http?$extraction_file_local ) close(c$http$extraction_file_local); } *Ideas? Thanks!* *Al B.* *Seminal Networks* On Wed, Aug 3, 2016 at 2:47 PM, Johanna Amann wrote: > Hi Al, > > > I'm new to Bro and using version 2.3.2 and want to extract all the exe's > > seen on the network. In bro-file-extract we are using the > file-extract.bro > > script to try to parse for the exe's (partial of script): > > First - is there any reason for you to still use 2.3.2? File handling (and > a lot of other things) have become more robust in 2.4. > > In any case... > > > global ext_map:table[string] of string = { > > ["application/x/dosexec"] = "exe", > > you probably want application/x-dosexec here, not x/dosexec. That might > already be enough to fix this. > > > redef FileExtract::prefix="/var/log/netlogs/bro/file-extracts.bro"; > > This line seems superfluous and wrong, especially since it is redef-ed > again two lines later. > > > redef FileExtract::default_limit = 314572800; > > redef FileExtract::prefix = "/var/log/netlogs/bro/file-extracts/"; > > > > We also have the file-extract-http-local.bro set to extract on our > network: > > > > global http_extract_file_ignore: set [subnet] = { > > 10.0.0.0/8, > > }; > > > > The following seems to talk about files that you modified locally and that > do not ship with the Bro distribution. As such, it is really hard to give > feedback about it. > > > We think the problem is that _load_.bro has the file extract commented > out > > under bro-icmp: > > #@load ./file-extract-http-local.bro > > #@load ./file-extract-types.bro > > @load ./bro-file-extract > > When I tried to enable these Bro failed the scripts check with errors > like: > > internal warning in > > /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, > line > > 6: Discarded extraneous Broxygen comment: Modified from base scripts to > > extract only from external hosts > > fatal error in > > /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, > line > > 7:can't find base/protocols/http/file-ident > > I continued to receive these errors and had to back out of removing the > > comments > > > > Under bro-file-extract _load_.bro looks correct: > > @load ./file-extract > > > > What I'm getting in /var/log/netlogs/bro/file-extracts are entries like: > > HTTP-F7K52nSzN3h7GNM31.exe > > These files occur occasionally I'm not sure what they are. > > I hope this helps, > Johanna > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160827/b91ded93/attachment-0001.html From fatema.bannatwala at gmail.com Mon Aug 29 06:30:21 2016 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Mon, 29 Aug 2016 09:30:21 -0400 Subject: [Bro] IOCs data for hashes. Message-ID: Hi, I am working with BRO, trying to add the capability of malware detection using Bro. I am already using the intel framework provided by Bro and feeding IOC data into it. It successfully detects and logs the connection having bad IPs and domains in intel.log file. The functionality I would like to add is to detect any malware downloaded by any of the endpoints, and for that I need some good IOC data of hashes. I searched the internet for IOCs hashes but couldn't fine any good source for it. Does anyone have any pointers in the same direction? or any other magic that can be used to accomplish the same purpose? Thanks, Fatema. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160829/d762ecb3/attachment.html From ed.sealing at sealingtech.org Mon Aug 29 07:39:26 2016 From: ed.sealing at sealingtech.org (Ed Sealing) Date: Mon, 29 Aug 2016 10:39:26 -0400 Subject: [Bro] IOCs data for hashes. In-Reply-To: References: Message-ID: MITRE and NIST have been putting some efforts into the "Malware Attribute Enumeration and Characterization (MAEC)" standard. I haven't done much work with it, but it's worth looking into. They have a list of datasets at https://github.com/MAECProject/datasets. Sending the hashes out to services like VirusTotal or Team CYMRU is another widely used option. This is all covered under the Bro File Extraction Exercise on the website (https://www.bro.org/current/exercises/faf/) If you are trying to do this without sending any information over the internet, there are in-house implementations that are available for commercial use. Opswat Meta-defender is an example of a commercially available multi-AV platform with an API that Bro can interface with. https://www.opswat.com/metadefender-core Hope this helps. ~Ed On Mon, Aug 29, 2016 at 9:30 AM, fatema bannatwala < fatema.bannatwala at gmail.com> wrote: > Hi, > > I am working with BRO, trying to add the capability of malware detection > using Bro. > I am already using the intel framework provided by Bro and feeding IOC > data into it. > It successfully detects and logs the connection having bad IPs and domains > in intel.log file. > The functionality I would like to add is to detect any malware downloaded > by any of the endpoints, and for that I need some good IOC data of hashes. > I searched the internet for IOCs hashes but couldn't fine any good source > for it. > Does anyone have any pointers in the same direction? or any other magic > that can be used to accomplish the same purpose? > > Thanks, > Fatema. > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- R/S *Ed Sealing President / CEO* *CISSP, CEH, RHCSA* 7226 Lee Deforest Dr. Columbia, MD 21046 Mobile: (301) 885-6947 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160829/fa879dab/attachment.html From chris at cwalsh.org Mon Aug 29 07:42:29 2016 From: chris at cwalsh.org (Chris Walsh) Date: Mon, 29 Aug 2016 09:42:29 -0500 Subject: [Bro] IOCs data for hashes. In-Reply-To: References: Message-ID: Have you looked at https://www.bro.org/sphinx/scripts/policy/frameworks/files/detect-MHR.bro.html ? If I am understanding your goal, this seems to be a good fit for what you?re trying to do. Chris > On Aug 29, 2016, at 8:30 AM, fatema bannatwala wrote: > > Hi, > > I am working with BRO, trying to add the capability of malware detection using Bro. > I am already using the intel framework provided by Bro and feeding IOC data into it. > It successfully detects and logs the connection having bad IPs and domains in intel.log file. > The functionality I would like to add is to detect any malware downloaded by any of the endpoints, and for that I need some good IOC data of hashes. I searched the internet for IOCs hashes but couldn't fine any good source for it. > Does anyone have any pointers in the same direction? or any other magic that can be used to accomplish the same purpose? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160829/b755d3ed/attachment.html From dani.nicolo at gmail.com Mon Aug 29 10:01:17 2016 From: dani.nicolo at gmail.com (=?UTF-8?Q?Danilo_Nicol=C3=B2?=) Date: Mon, 29 Aug 2016 19:01:17 +0200 Subject: [Bro] High orig_bytes value Message-ID: Hello guys, I'm testing Bro 2.5 beta with netmap, and I noticed this row: {"ts":1472467151.681244,"uid":"CgoIaB3GxSCIEgWea7","id.orig_h":"192.168.181.107","id.orig_p":11328,"id.resp_h":"172.16.1.60","id.resp_p":9997,"proto":"tcp","duration":0.362595,"orig_bytes":4294967296,"resp_bytes":4294967296,"conn_state":"SF","local_resp":true,"missed_bytes":1168863602,"history":"ShAFFff","orig_pkts":7,"orig_ip_bytes":292,"resp_pkts":4,"resp_ip_bytes":184,"tunnel_parents":[],"local_origi":"T4","local_respo":"T4"} If you look at this log, you can see that there was 4gb data exchanged in 0sec, that's impossible. I followed the netmap installation guide by patching the igb intel driver and so using libpcap system (version 0.8) Anyone has had this kind of problem? Might it be a netmap problem? Should I use pf_ring instead? Thanks in advance, Danilo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160829/d2c53787/attachment.html From fatema.bannatwala at gmail.com Mon Aug 29 11:00:47 2016 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Mon, 29 Aug 2016 14:00:47 -0400 Subject: [Bro] IOCs data for hashes. In-Reply-To: References: Message-ID: Hi Chris, Thank you for the suggestions. The detect-MHR.bro script is already enabled in the local.bro file, but I don't get any logging in notice.log file corresponding to Malware hash registry. I looked at the script and the notice_threshold is set to 10 (10% min detection rate) which is reasonable, but as I was analyzing a malware hash, detected by other IDS device and when checked on team cymru's lookup: https://hash.cymru.com had 26% as detection rate, realized that there were no logs in files.log and notice.log files corresponding to that hash. Bro didn't log any hash for the file transfer that transpired. 1472425280.047247 Fs9rse1xsQgD2TIADa 220.243.237.153 x.x.x.x CJFssC2o2RqHx6PJY8 HTTP 0 MD5,PE,SHA1 application/x-dosexec - 11.101799 F F 2122412 20265152 18142740 0 Also, when I checked, the Content-type reported by the IDS device was: application/x-www-form-urlencoded and guessing that maybe files with this mime-type are not hashed by bro probably. I don't know why I am not able to find the corresponding hash in bro logs. Thanks, Fatema. On Mon, Aug 29, 2016 at 10:42 AM, Chris Walsh wrote: > Have you looked at https://www.bro.org/sphinx/scripts/policy/frameworks/ > files/detect-MHR.bro.html ? > > If I am understanding your goal, this seems to be a good fit for what > you?re trying to do. > > > Chris > > On Aug 29, 2016, at 8:30 AM, fatema bannatwala < > fatema.bannatwala at gmail.com> wrote: > > Hi, > > I am working with BRO, trying to add the capability of malware detection > using Bro. > I am already using the intel framework provided by Bro and feeding IOC > data into it. > It successfully detects and logs the connection having bad IPs and domains > in intel.log file. > The functionality I would like to add is to detect any malware downloaded > by any of the endpoints, and for that I need some good IOC data of hashes. > I searched the internet for IOCs hashes but couldn't fine any good source > for it. > Does anyone have any pointers in the same direction? or any other magic > that can be used to accomplish the same purpose? > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160829/12bdd928/attachment-0001.html From jazoff at illinois.edu Mon Aug 29 11:13:49 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 29 Aug 2016 18:13:49 +0000 Subject: [Bro] IOCs data for hashes. In-Reply-To: References: Message-ID: <5DB1A6D9-5318-4B2B-8CA0-50E5D8CB3DC2@illinois.edu> > On Aug 29, 2016, at 2:00 PM, fatema bannatwala wrote: > > Hi Chris, > > Thank you for the suggestions. > The detect-MHR.bro script is already enabled in the local.bro file, but I don't get any > logging in notice.log file corresponding to Malware hash registry. > > I looked at the script and the notice_threshold is set to 10 (10% min detection rate) which is reasonable, > but as I was analyzing a malware hash, detected by other IDS device and when checked on team cymru's lookup: https://hash.cymru.com had 26% as detection rate, realized that there were no logs in > files.log and notice.log files corresponding to that hash. > Bro didn't log any hash for the file transfer that transpired. > 1472425280.047247 Fs9rse1xsQgD2TIADa 220.243.237.153 x.x.x.x CJFssC2o2RqHx6PJY8 HTTP 0 MD5,PE,SHA1 application/x-dosexec - 11.101799 F F 2122412 20265152 18142740 0 Those last 3 numbers are seen_bytes = 2122412 total_bytes = 20265152 missing_bytes = 18142740 Bro did not see 90% of the bytes of the file, it can't hash what it didn't see. -- - Justin Azoff From philosnef at yahoo.com Mon Aug 29 13:31:14 2016 From: philosnef at yahoo.com (philosnef) Date: Mon, 29 Aug 2016 20:31:14 +0000 (UTC) Subject: [Bro] IOCs data for hashes. In-Reply-To: References: Message-ID: <81812374.1718282.1472502674195@mail.yahoo.com> Ok, so if you have a highly active network pushing many gigs of traffic, then it seems like hash based iocs are not likely to be reliable.... However, ssdeep hashing by bro would likely not be as severely impacted, correct? On Monday, August 29, 2016 3:00 PM, "bro-request at bro.org" wrote: Send Bro mailing list submissions to ??? bro at bro.org To subscribe or unsubscribe via the World Wide Web, visit ??? http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro or, via email, send a message with subject or body 'help' to ??? bro-request at bro.org You can reach the person managing the list at ??? bro-owner at bro.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Bro digest..." Today's Topics: ? 1. Re: IOCs data for hashes. (Azoff, Justin S) ---------------------------------------------------------------------- Message: 1 Date: Mon, 29 Aug 2016 18:13:49 +0000 From: "Azoff, Justin S" Subject: Re: [Bro] IOCs data for hashes. To: fatema bannatwala Cc: "bro at bro.org" Message-ID: <5DB1A6D9-5318-4B2B-8CA0-50E5D8CB3DC2 at illinois.edu> Content-Type: text/plain; charset="us-ascii" > On Aug 29, 2016, at 2:00 PM, fatema bannatwala wrote: > > Hi Chris, > > Thank you for the suggestions. > The detect-MHR.bro script is already enabled in the local.bro file, but I don't get any > logging in notice.log file corresponding to Malware hash registry. > > I looked at the script and the notice_threshold is set to 10 (10% min detection rate) which is reasonable, > but as I was analyzing a malware hash, detected by other IDS device and when checked on team cymru's lookup: https://hash.cymru.com had 26% as detection rate, realized that there were no logs in > files.log and notice.log files corresponding to that hash. > Bro didn't log any hash for the file transfer that transpired. > 1472425280.047247? ? ? Fs9rse1xsQgD2TIADa? ? ? 220.243.237.153? x.x.x.x? ? ? CJFssC2o2RqHx6PJY8? ? ? HTTP? ? 0? ? ? MD5,PE,SHA1? ? application/x-dosexec? -? ? ? 11.101799? ? ? F? ? ? F? ? ? 2122412 20265152? ? ? 18142740 0 Those last 3 numbers are seen_bytes? ? = 2122412 total_bytes? = 20265152 missing_bytes = 18142740 Bro did not see 90% of the bytes of the file, it can't hash what it didn't see. -- - Justin Azoff ------------------------------ _______________________________________________ Bro mailing list Bro at bro.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro End of Bro Digest, Vol 124, Issue 34 ************************************ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160829/270c7a85/attachment.html From seth at icir.org Tue Aug 30 07:09:18 2016 From: seth at icir.org (Seth Hall) Date: Tue, 30 Aug 2016 10:09:18 -0400 Subject: [Bro] Bro connections v. NetFlow In-Reply-To: <8350146BADDCE04480B969B36967473D13F090CB@ZEUS.olympus.dataline.co.uk> References: <8350146BADDCE04480B969B36967473D13F090CB@ZEUS.olympus.dataline.co.uk> Message-ID: > On Aug 25, 2016, at 6:16 AM, Alec Waters wrote: > > We set our routers to export flows after one minute if they?re still in progress (it?ll continue to send a flow export every minute until it?s complete). More info here:  The fun part about Bro is that it's a scripting language and we can do whatever we want! :) Here's a script that I wrote in Broala a while ago that we're releasing under the BSD license. https://github.com/broala/bro-long-connections I think I will need to do a bit more work on this to make it more like flow cutting, but at the very least it now makes active connections visible. Any feedback would be appreciated. Thanks! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160830/eb717f06/attachment.bin From michalpurzynski1 at gmail.com Tue Aug 30 09:44:35 2016 From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=) Date: Tue, 30 Aug 2016 18:44:35 +0200 Subject: [Bro] Bro connections v. NetFlow In-Reply-To: References: <8350146BADDCE04480B969B36967473D13F090CB@ZEUS.olympus.dataline.co.uk> Message-ID: Have you tested it with loooots of connections? How hard it is on the memory and CPU? > On 30 Aug 2016, at 16:09, Seth Hall wrote: > > >> On Aug 25, 2016, at 6:16 AM, Alec Waters wrote: >> >> We set our routers to export flows after one minute if they?re still in progress (it?ll continue to send a flow export every minute until it?s complete). More info here: >  > The fun part about Bro is that it's a scripting language and we can do whatever we want! :) > > Here's a script that I wrote in Broala a while ago that we're releasing under the BSD license. > https://github.com/broala/bro-long-connections > > I think I will need to do a bit more work on this to make it more like flow cutting, but at the very least it now makes active connections visible. Any feedback would be appreciated. > > Thanks! > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From seth at icir.org Tue Aug 30 09:53:10 2016 From: seth at icir.org (Seth Hall) Date: Tue, 30 Aug 2016 12:53:10 -0400 Subject: [Bro] Bro connections v. NetFlow In-Reply-To: References: <8350146BADDCE04480B969B36967473D13F090CB@ZEUS.olympus.dataline.co.uk> Message-ID: > On Aug 30, 2016, at 12:44 PM, Micha? Purzy?ski wrote: > > Have you tested it with loooots of connections? How hard it is on the memory and CPU? It hasn't been tested very extensively, but I wouldn't expect it to have much trouble with either memory or CPU since it's just riding on top of the existing connection state mechanism. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From slagell at illinois.edu Tue Aug 30 09:59:40 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Tue, 30 Aug 2016 16:59:40 +0000 Subject: [Bro] Where do I find broctl error logs? Message-ID: I am running the beta on Mac OS, and I noticed broctl cron is not working anymore. I can run it manually, and other cron jobs run fine, but it seems to be failing in some way I can?t see in syslogs. I suspect something in broctl needs to set the path [1], but I have no information on the failure to investigate further. :Adam [1] http://apple.stackexchange.com/questions/116566/cron-script-not-executing-on-mavericks ------ Adam J. Slagell Chief Information Security Officer Director, Cybersecurity Division National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." From jlay at slave-tothe-box.net Tue Aug 30 10:03:17 2016 From: jlay at slave-tothe-box.net (James Lay) Date: Tue, 30 Aug 2016 11:03:17 -0600 Subject: [Bro] Bro connections v. NetFlow In-Reply-To: References: <8350146BADDCE04480B969B36967473D13F090CB@ZEUS.olympus.dataline.co.uk> Message-ID: On 2016-08-30 10:53, Seth Hall wrote: >> On Aug 30, 2016, at 12:44 PM, Micha? Purzy?ski >> wrote: >> >> Have you tested it with loooots of connections? How hard it is on the >> memory and CPU? > > It hasn't been tested very extensively, but I wouldn't expect it to > have much trouble with either memory or CPU since it's just riding on > top of the existing connection state mechanism. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro Imma test this out Seth thank you...I'll report findings here. James From asharma at lbl.gov Tue Aug 30 10:06:02 2016 From: asharma at lbl.gov (Aashish Sharma) Date: Tue, 30 Aug 2016 10:06:02 -0700 Subject: [Bro] Where do I find broctl error logs? In-Reply-To: References: Message-ID: <20160830170558.GF64515@mac-4.local> > I noticed broctl cron is not working anymore. I am encountering the same issue. I first noticed that if workers are dying, broctl cron won't kick those back in - finally it appear that broctl cron itself might not be running. (This is on FreeBSD) Aashish On Tue, Aug 30, 2016 at 04:59:40PM +0000, Slagell, Adam J wrote: > I am running the beta on Mac OS, and I noticed broctl cron is not working anymore. I can run it manually, and other cron jobs run fine, but it seems to be failing in some way I can?t see in syslogs. > > I suspect something in broctl needs to set the path [1], but I have no information on the failure to investigate further. > > :Adam > > [1] http://apple.stackexchange.com/questions/116566/cron-script-not-executing-on-mavericks > ------ > > Adam J. Slagell > Chief Information Security Officer > Director, Cybersecurity Division > National Center for Supercomputing Applications > University of Illinois at Urbana-Champaign > www.slagell.info > > "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." > > > > > > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From seth at icir.org Tue Aug 30 10:43:10 2016 From: seth at icir.org (Seth Hall) Date: Tue, 30 Aug 2016 13:43:10 -0400 Subject: [Bro] Bro connections v. NetFlow In-Reply-To: References: <8350146BADDCE04480B969B36967473D13F090CB@ZEUS.olympus.dataline.co.uk> Message-ID: > On Aug 30, 2016, at 1:03 PM, James Lay wrote: > > Imma test this out Seth thank you...I'll report findings here. Awesome! Thanks. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From fatema.bannatwala at gmail.com Tue Aug 30 10:46:35 2016 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Tue, 30 Aug 2016 13:46:35 -0400 Subject: [Bro] IOCs data for hashes. In-Reply-To: <5DB1A6D9-5318-4B2B-8CA0-50E5D8CB3DC2@illinois.edu> References: <5DB1A6D9-5318-4B2B-8CA0-50E5D8CB3DC2@illinois.edu> Message-ID: Thanks Justin for the answer. Yeah, we realized that we were having some capture loss with our BRO sensors, it's fixed now. Thanks, Fatema. On Mon, Aug 29, 2016 at 2:13 PM, Azoff, Justin S wrote: > > > On Aug 29, 2016, at 2:00 PM, fatema bannatwala < > fatema.bannatwala at gmail.com> wrote: > > > > Hi Chris, > > > > Thank you for the suggestions. > > The detect-MHR.bro script is already enabled in the local.bro file, but > I don't get any > > logging in notice.log file corresponding to Malware hash registry. > > > > I looked at the script and the notice_threshold is set to 10 (10% min > detection rate) which is reasonable, > > but as I was analyzing a malware hash, detected by other IDS device and > when checked on team cymru's lookup: https://hash.cymru.com had 26% as > detection rate, realized that there were no logs in > > files.log and notice.log files corresponding to that hash. > > Bro didn't log any hash for the file transfer that transpired. > > 1472425280.047247 Fs9rse1xsQgD2TIADa 220.243.237.153 > x.x.x.x CJFssC2o2RqHx6PJY8 HTTP 0 MD5,PE,SHA1 > application/x-dosexec - 11.101799 F F 2122412 > 20265152 18142740 0 > > Those last 3 numbers are > > seen_bytes = 2122412 > total_bytes = 20265152 > missing_bytes = 18142740 > > Bro did not see 90% of the bytes of the file, it can't hash what it didn't > see. > > -- > - Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160830/8ac904c8/attachment.html From slagell at illinois.edu Tue Aug 30 10:52:49 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Tue, 30 Aug 2016 17:52:49 +0000 Subject: [Bro] Where do I find broctl error logs? In-Reply-To: <20160830170558.GF64515@mac-4.local> References: , <20160830170558.GF64515@mac-4.local> Message-ID: <0BA2EE19-1AA8-4CD6-A7E7-F6E012E509DB@illinois.edu> On Aug 30, 2016, at 12:06 PM, Aashish Sharma wrote: >> I noticed broctl cron is not working anymore. > > I am encountering the same issue. I first noticed that if workers are dying, broctl cron won't kick those back in - finally it appear that broctl cron itself might not be running. (This is on FreeBSD) Perhaps there is a BSD only issue with broctl cron From hosom at battelle.org Tue Aug 30 18:47:50 2016 From: hosom at battelle.org (Hosom, Stephen M) Date: Wed, 31 Aug 2016 01:47:50 +0000 Subject: [Bro] Where do I find broctl error logs? In-Reply-To: <0BA2EE19-1AA8-4CD6-A7E7-F6E012E509DB@illinois.edu> References: , <20160830170558.GF64515@mac-4.local> <0BA2EE19-1AA8-4CD6-A7E7-F6E012E509DB@illinois.edu> Message-ID: I don't know anything about the issue, but if you enable debug in broctl's configuration, you should be able to see a debug.log in your spool directory. -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Slagell, Adam J Sent: Tuesday, August 30, 2016 1:53 PM To: Aashish Sharma Cc: #bro Subject: Re: [Bro] Where do I find broctl error logs? On Aug 30, 2016, at 12:06 PM, Aashish Sharma wrote: >> I noticed broctl cron is not working anymore. > > I am encountering the same issue. I first noticed that if workers are > dying, broctl cron won't kick those back in - finally it appear that > broctl cron itself might not be running. (This is on FreeBSD) Perhaps there is a BSD only issue with broctl cron _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From crharwood at gmail.com Tue Aug 30 23:55:03 2016 From: crharwood at gmail.com (Chris Harwood) Date: Tue, 30 Aug 2016 23:55:03 -0700 Subject: [Bro] Newbie question: Bro logs to OSSIM Message-ID: Hi, New to the list and working with Bro. Interested in sending Bro logs to Alienvault OSSIM to consolidate two separate IDS and Argus systems. Has anyone else done this or have a resource to share? The one article describing the rsyslog config to output the logs is dated 2011 so thought I'd ask while taking a look at the OSSIM Bro plugin. Thanks in advance, Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160830/4d15021d/attachment.html From neslog at gmail.com Wed Aug 31 03:41:43 2016 From: neslog at gmail.com (Neslog) Date: Wed, 31 Aug 2016 06:41:43 -0400 Subject: [Bro] Where do I find broctl error logs? In-Reply-To: References: <20160830170558.GF64515@mac-4.local> <0BA2EE19-1AA8-4CD6-A7E7-F6E012E509DB@illinois.edu> Message-ID: I'd like to see some basic broctl error logging also. Right now I'm running additional con jobs for status. I'd also like to get broctl output logging as it happens. My scripts have to include 2>&1 piped to grep/awk to make it useful. On Aug 30, 2016 9:50 PM, "Hosom, Stephen M" wrote: > I don't know anything about the issue, but if you enable debug in broctl's > configuration, you should be able to see a debug.log in your spool > directory. > > -----Original Message----- > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of > Slagell, Adam J > Sent: Tuesday, August 30, 2016 1:53 PM > To: Aashish Sharma > Cc: #bro > Subject: Re: [Bro] Where do I find broctl error logs? > > > > On Aug 30, 2016, at 12:06 PM, Aashish Sharma wrote: > > >> I noticed broctl cron is not working anymore. > > > > I am encountering the same issue. I first noticed that if workers are > > dying, broctl cron won't kick those back in - finally it appear that > > broctl cron itself might not be running. (This is on FreeBSD) > > Perhaps there is a BSD only issue with broctl cron > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160831/d23ad7c7/attachment.html From jazoff at illinois.edu Wed Aug 31 06:09:12 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 31 Aug 2016 13:09:12 +0000 Subject: [Bro] Where do I find broctl error logs? In-Reply-To: References: <20160830170558.GF64515@mac-4.local> <0BA2EE19-1AA8-4CD6-A7E7-F6E012E509DB@illinois.edu> Message-ID: > On Aug 31, 2016, at 6:41 AM, Neslog wrote: > > I'd like to see some basic broctl error logging also. Right now I'm running additional con jobs for status. I'd also like to get broctl output logging as it happens. My scripts have to include 2>&1 piped to grep/awk to make it useful. Starting in 2.4 broctl has an API, there's no reason to pipe it to grep/awk for getting status # PYTHONPATH=/usr/local/bro/lib/broctl python >>> from BroControl import broctl >>> b = broctl.BroCtl() >>> res = b.status() >>> res.ok True >>> res.success_count 60 >>> for node in res.nodes[:3]: print node ... (, True, {'status': 'running', 'name': 'nids-dev2a-1', 'started': '29 Aug 21:41:04', 'pid': 18025, 'host': '10.1.1.40', 'type': 'worker'}) (, True, {'status': 'running', 'name': 'nids-dev2a-2', 'started': '29 Aug 21:41:04', 'pid': 18037, 'host': '10.1.1.40', 'type': 'worker'}) (, True, {'status': 'running', 'name': 'nids-dev2a-3', 'started': '29 Aug 21:41:04', 'pid': 18076, 'host': '10.1.1.40', 'type': 'worker'}) -- - Justin Azoff From fatema.bannatwala at gmail.com Wed Aug 31 06:36:43 2016 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Wed, 31 Aug 2016 09:36:43 -0400 Subject: [Bro] IOCs data for hashes. In-Reply-To: References: <5DB1A6D9-5318-4B2B-8CA0-50E5D8CB3DC2@illinois.edu> Message-ID: Hi , I wanted to ask that is there a framework that is available to integrate bro with virusTotal as well, just like it uses TeamCymru's MHR to check the hashes against? The issue is that we get a very low detection rate with MHR, i.e we see a good number of hashes that are detected as malware/Trojan by other IDS devices, and same hashes when checked against MHR by bro, we do not get any records in notice.log as they come out clean by MHR. I realized that virusTotal has a pretty decent detection rate for those hashes. I haven't looked into details of the framework used to integrate BRO with MHR, but thinking if the same can be done with virusTotal, hence wanted to confirm whether this is something achievable using BRO or not, before diving deep into it. Appreciate the help. Thanks, Fatema. On Tue, Aug 30, 2016 at 1:46 PM, fatema bannatwala < fatema.bannatwala at gmail.com> wrote: > Thanks Justin for the answer. > Yeah, we realized that we were having some capture loss with our BRO > sensors, it's fixed now. > > Thanks, > Fatema. > > On Mon, Aug 29, 2016 at 2:13 PM, Azoff, Justin S > wrote: > >> >> > On Aug 29, 2016, at 2:00 PM, fatema bannatwala < >> fatema.bannatwala at gmail.com> wrote: >> > >> > Hi Chris, >> > >> > Thank you for the suggestions. >> > The detect-MHR.bro script is already enabled in the local.bro file, but >> I don't get any >> > logging in notice.log file corresponding to Malware hash registry. >> > >> > I looked at the script and the notice_threshold is set to 10 (10% min >> detection rate) which is reasonable, >> > but as I was analyzing a malware hash, detected by other IDS device and >> when checked on team cymru's lookup: https://hash.cymru.com had 26% as >> detection rate, realized that there were no logs in >> > files.log and notice.log files corresponding to that hash. >> > Bro didn't log any hash for the file transfer that transpired. >> > 1472425280.047247 Fs9rse1xsQgD2TIADa 220.243.237.153 >> x.x.x.x CJFssC2o2RqHx6PJY8 HTTP 0 MD5,PE,SHA1 >> application/x-dosexec - 11.101799 F F 2122412 >> 20265152 18142740 0 >> >> Those last 3 numbers are >> >> seen_bytes = 2122412 >> total_bytes = 20265152 >> missing_bytes = 18142740 >> >> Bro did not see 90% of the bytes of the file, it can't hash what it >> didn't see. >> >> -- >> - Justin Azoff >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160831/15a65b8f/attachment-0001.html From jwc3f at virginia.edu Wed Aug 31 08:23:59 2016 From: jwc3f at virginia.edu (Collyer, Jeffrey W. (jwc3f)) Date: Wed, 31 Aug 2016 15:23:59 +0000 Subject: [Bro] Bro Splunk file size and removal interaction Message-ID: So I?m logging my Bro in JSON format on my manager node. I have Splunk ingesting the log files through the Splunk TA from Github : https://github.com/jahshuah/splunk-ta-bro-json Everything is working fine except I?m only getting sporadic http.log entries. Looking in the Splunk logs, it appears that the http.log file is large enough that Splunk isn?t finished indexing it, when it gets rotated/compressed out and the new 1/2 hour files starts to fill. Splunk doesn?t seem to do any file locking(a good thing), but the file goes away before its finished with it. The machine seems to have plenty of resources, and I?ve turned off the index thruput limit on the splunk heavy forwarder. So I?m not sure if I can make Splunk go any faster. Are there any bro settings that would help here? I thought about rotating the logs more frequently but if volume is the issue that won?t really help. Is there a way to have bro not compress/remove the file immediately? Or anyone tackled this problem and found a different/splunk solution? Jeffrey Collyer Information Security Engineer University of Virginia 434-297-6317 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160831/3a985822/attachment.html From neslog at gmail.com Wed Aug 31 08:33:46 2016 From: neslog at gmail.com (Neslog) Date: Wed, 31 Aug 2016 11:33:46 -0400 Subject: [Bro] Where do I find broctl error logs? In-Reply-To: References: <20160830170558.GF64515@mac-4.local> <0BA2EE19-1AA8-4CD6-A7E7-F6E012E509DB@illinois.edu> Message-ID: Alright. I'll have to see about how to instrument that. Thanks! On Aug 31, 2016 9:09 AM, "Azoff, Justin S" wrote: > > > On Aug 31, 2016, at 6:41 AM, Neslog wrote: > > > > I'd like to see some basic broctl error logging also. Right now I'm > running additional con jobs for status. I'd also like to get broctl output > logging as it happens. My scripts have to include 2>&1 piped to grep/awk to > make it useful. > > Starting in 2.4 broctl has an API, there's no reason to pipe it to > grep/awk for getting status > > # PYTHONPATH=/usr/local/bro/lib/broctl python > >>> from BroControl import broctl > >>> b = broctl.BroCtl() > >>> res = b.status() > >>> res.ok > True > >>> res.success_count > 60 > >>> for node in res.nodes[:3]: print node > ... > (, True, {'status': 'running', > 'name': 'nids-dev2a-1', 'started': '29 Aug 21:41:04', 'pid': 18025, 'host': > '10.1.1.40', 'type': 'worker'}) > (, True, {'status': 'running', > 'name': 'nids-dev2a-2', 'started': '29 Aug 21:41:04', 'pid': 18037, 'host': > '10.1.1.40', 'type': 'worker'}) > (, True, {'status': 'running', > 'name': 'nids-dev2a-3', 'started': '29 Aug 21:41:04', 'pid': 18076, 'host': > '10.1.1.40', 'type': 'worker'}) > > -- > - Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160831/f7c91137/attachment.html From jazoff at illinois.edu Wed Aug 31 08:38:34 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 31 Aug 2016 15:38:34 +0000 Subject: [Bro] Bro Splunk file size and removal interaction In-Reply-To: References: Message-ID: <8B9285C9-E628-41DA-B5F9-A3AB22DABCAA@illinois.edu> I ran into similar issues a while ago, I now use these settings in limits.conf: min_batch_size_bytes = 1048576000 [thruput] maxKBps = 0 # means unlimited -- - Justin Azoff > On Aug 31, 2016, at 11:23 AM, Collyer, Jeffrey W. (jwc3f) wrote: > > So I?m logging my Bro in JSON format on my manager node. I have Splunk ingesting the log files through the Splunk TA from Github : https://github.com/jahshuah/splunk-ta-bro-json > > Everything is working fine except I?m only getting sporadic http.log entries. Looking in the Splunk logs, it appears that the http.log file is large enough that Splunk isn?t finished indexing it, when it gets rotated/compressed out and the new 1/2 hour files starts to fill. > > Splunk doesn?t seem to do any file locking(a good thing), but the file goes away before its finished with it. The machine seems to have plenty of resources, and I?ve turned off the index thruput limit on the splunk heavy forwarder. So I?m not sure if I can make Splunk go any faster. > > Are there any bro settings that would help here? I thought about rotating the logs more frequently but if volume is the issue that won?t really help. Is there a way to have bro not compress/remove the file immediately? > > Or anyone tackled this problem and found a different/splunk solution? > > Jeffrey Collyer > Information Security Engineer > University of Virginia > 434-297-6317 > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro