From christopher.p.crawford at gmail.com Thu Mar 1 06:02:18 2012 From: christopher.p.crawford at gmail.com (Chris Crawford) Date: Thu, 1 Mar 2012 09:02:18 -0500 Subject: [Bro] Bro2 Random Crashes Message-ID: When initially I set up Bro2, it ran for a few days with no problems. I recently noticed recently that Bro2 has started to crash randomly. Typically it will take Bro2 5 or 6 hours before a crash, but sometimes it crashes immediately. I installed Bro2 on Ubuntu 10.04 LTS. Ubuntu is a xen VM on a Citrix Xen server. The throughput on the network is < 400 Mb/s. After taking a look at one of my crash reports, Seth suggested that I'm running out of memory. The VM I have initially started out with 4GB of RAM, but I bumped it up to 8GB of RAM, only to get the same results. There was no time difference, and there is no pattern as to why or when Bro2 crashes. The only thing I've been able to key in on, is that over time Bro2 eventually causes all free memory in Ubuntu to change to cached memory. From the Citrix Xen Console, that cached memory shows up as used memory. So, maybe Xen interprets cached memory as being used? Also -- when Xen senses that most of the memory is "used" (but, really, it's cached inside Ubuntu), the percent utilization in one of the CPUs in the VM spikes. After Bro2 crashes, CPU utilization returns to normal, but memory is never freed -- it remains cached forever. I can dump the cache using echo 1 > /proc/sys/vm/drop_caches Which converts all the memory allocated as cached to free. Any chance that this is related? Anyone have ideas on what is causing Bro2 to crash? -Chris I have included some additional information below. Here are the steps I used to install Bro2: sudo aptitude -y install swig libmagic-dev libgeoip-dev cmake build-essential flex bison libpcap-dev libssl-dev python-dev gawk cd /tmp wget http://www.bro-ids.org/downloads/release/bro-2.0.tar.gz tar xvzf bro-2.0.tar.gz cd bro-2.0 ./configure make sudo make install sudo chmod a+w /etc/bash.bashrc sudo echo '' >> /etc/bash.bashrc sudo echo 'export PATH=/usr/local/bro/bin:$PATH' >> /etc/bash.bashrc sudo chmod go-w /etc/bash.bashrc # Add to /usr/local/bro/etc/networks.cfg: [...] broctl install broctl start ### End Installation ### Here is a recent crash report: core [New Thread 5101] Core was generated by `/usr/local/bro/bin/bro -i eth1 -U .status -p broctl -p broctl-live -p standalon'. Program terminated with signal 6, Aborted. #0 0x00007f72aefcaa75 in raise () from /lib/libc.so.6 ==== reporter.log #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path reporter #fields ts level message location #types time enum string string 1330462476.662662 Reporter::ERROR bro wasn't compiled with IPv6 support (empty) ==== stderr.log listening on eth1, capture length 8192 bytes terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc /usr/local/bro/share/broctl/scripts/run-bro: line 60: 5101 Aborted (core dumped) nohup $mybro $@ ==== stdout.log unlimited unlimited unlimited ==== .cmdline -i eth1 -U .status -p broctl -p broctl-live -p standalone -p local -p bro local broctl broctl/standalone broctl/auto ==== .env_vars PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games BROPATH=/logs/bro/spool/policy/site::/logs/bro/spool/policy/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site CLUSTER_NODE= ==== .status RUNNING [net_run] ==== No prof.log ==== packet_filter.log #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path packet_filter #fields ts node filter init success #types time string string bool bool 1330462468.693819 - not ip6 T T ==== loaded_scripts.log #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path loaded_scripts #fields name #types string /usr/local/bro/share/bro/base/init-bare.bro /usr/local/bro/share/bro/base/const.bif.bro /usr/local/bro/share/bro/base/types.bif.bro /usr/local/bro/share/bro/base/strings.bif.bro /usr/local/bro/share/bro/base/bro.bif.bro /usr/local/bro/share/bro/base/reporter.bif.bro /usr/local/bro/share/bro/base/event.bif.bro /usr/local/bro/share/bro/base/frameworks/logging/__load__.bro /usr/local/bro/share/bro/base/frameworks/logging/./main.bro /usr/local/bro/share/bro/base/logging.bif.bro /usr/local/bro/share/bro/base/frameworks/logging/./postprocessors/__load__.bro /usr/local/bro/share/bro/base/frameworks/logging/./postprocessors/./scp.bro /usr/local/bro/share/bro/base/frameworks/logging/./postprocessors/./sftp.bro /usr/local/bro/share/bro/base/frameworks/logging/./writers/ascii.bro /usr/local/bro/share/bro/base/init-default.bro /usr/local/bro/share/bro/base/utils/site.bro /usr/local/bro/share/bro/base/utils/./patterns.bro /usr/local/bro/share/bro/base/utils/addrs.bro /usr/local/bro/share/bro/base/utils/conn-ids.bro /usr/local/bro/share/bro/base/utils/directions-and-hosts.bro /usr/local/bro/share/bro/base/utils/files.bro /usr/local/bro/share/bro/base/utils/numbers.bro /usr/local/bro/share/bro/base/utils/paths.bro /usr/local/bro/share/bro/base/utils/strings.bro /usr/local/bro/share/bro/base/utils/thresholds.bro /usr/local/bro/share/bro/base/frameworks/notice/__load__.bro /usr/local/bro/share/bro/base/frameworks/notice/./main.bro /usr/local/bro/share/bro/base/frameworks/notice/./weird.bro /usr/local/bro/share/bro/base/frameworks/notice/./actions/drop.bro /usr/local/bro/share/bro/base/frameworks/notice/./actions/email_admin.bro /usr/local/bro/share/bro/base/frameworks/notice/./actions/page.bro /usr/local/bro/share/bro/base/frameworks/notice/./actions/add-geodata.bro /usr/local/bro/share/bro/base/frameworks/notice/./extend-email/hostnames.bro /usr/local/bro/share/bro/base/frameworks/cluster/__load__.bro /usr/local/bro/share/bro/base/frameworks/cluster/./main.bro /usr/local/bro/share/bro/base/frameworks/control/__load__.bro /usr/local/bro/share/bro/base/frameworks/control/./main.bro /usr/local/bro/share/bro/base/frameworks/notice/./actions/pp-alarms.bro /usr/local/bro/share/bro/base/frameworks/dpd/__load__.bro /usr/local/bro/share/bro/base/frameworks/dpd/./main.bro /usr/local/bro/share/bro/base/frameworks/signatures/__load__.bro /usr/local/bro/share/bro/base/frameworks/signatures/./main.bro /usr/local/bro/share/bro/base/frameworks/packet-filter/__load__.bro /usr/local/bro/share/bro/base/frameworks/packet-filter/./main.bro /usr/local/bro/share/bro/base/frameworks/packet-filter/./netstats.bro /usr/local/bro/share/bro/base/frameworks/software/__load__.bro /usr/local/bro/share/bro/base/frameworks/software/./main.bro /usr/local/bro/share/bro/base/frameworks/communication/__load__.bro /usr/local/bro/share/bro/base/frameworks/communication/./main.bro /usr/local/bro/share/bro/base/frameworks/metrics/__load__.bro /usr/local/bro/share/bro/base/frameworks/metrics/./main.bro /usr/local/bro/share/bro/base/frameworks/metrics/./non-cluster.bro /usr/local/bro/share/bro/base/frameworks/intel/__load__.bro /usr/local/bro/share/bro/base/frameworks/intel/./main.bro /usr/local/bro/share/bro/base/frameworks/reporter/__load__.bro /usr/local/bro/share/bro/base/frameworks/reporter/./main.bro /usr/local/bro/share/bro/base/protocols/conn/__load__.bro /usr/local/bro/share/bro/base/protocols/conn/./main.bro /usr/local/bro/share/bro/base/protocols/conn/./contents.bro /usr/local/bro/share/bro/base/protocols/conn/./inactivity.bro /usr/local/bro/share/bro/base/protocols/dns/__load__.bro /usr/local/bro/share/bro/base/protocols/dns/./consts.bro /usr/local/bro/share/bro/base/protocols/dns/./main.bro /usr/local/bro/share/bro/base/protocols/ftp/__load__.bro /usr/local/bro/share/bro/base/protocols/ftp/./utils-commands.bro /usr/local/bro/share/bro/base/protocols/ftp/./main.bro /usr/local/bro/share/bro/base/protocols/ftp/./file-extract.bro /usr/local/bro/share/bro/base/protocols/http/__load__.bro /usr/local/bro/share/bro/base/protocols/http/./main.bro /usr/local/bro/share/bro/base/protocols/http/./utils.bro /usr/local/bro/share/bro/base/protocols/http/./file-ident.bro /usr/local/bro/share/bro/base/protocols/http/./file-hash.bro /usr/local/bro/share/bro/base/protocols/http/./file-extract.bro /usr/local/bro/share/bro/base/protocols/irc/__load__.bro /usr/local/bro/share/bro/base/protocols/irc/./main.bro /usr/local/bro/share/bro/base/protocols/irc/./dcc-send.bro /usr/local/bro/share/bro/base/protocols/smtp/__load__.bro /usr/local/bro/share/bro/base/protocols/smtp/./main.bro /usr/local/bro/share/bro/base/protocols/smtp/./entities.bro /usr/local/bro/share/bro/base/protocols/smtp/./entities-excerpt.bro /usr/local/bro/share/bro/base/protocols/ssh/__load__.bro /usr/local/bro/share/bro/base/protocols/ssh/./main.bro /usr/local/bro/share/bro/base/protocols/ssl/__load__.bro /usr/local/bro/share/bro/base/protocols/ssl/./consts.bro /usr/local/bro/share/bro/base/protocols/ssl/./main.bro /usr/local/bro/share/bro/base/protocols/ssl/./mozilla-ca-list.bro /usr/local/bro/share/bro/base/protocols/syslog/__load__.bro /usr/local/bro/share/bro/base/protocols/syslog/./consts.bro /usr/local/bro/share/bro/base/protocols/syslog/./main.bro /logs/bro/spool/policy/site/local.bro /usr/local/bro/share/bro/policy/misc/loaded-scripts.bro /usr/local/bro/share/bro/policy/tuning/defaults/__load__.bro /usr/local/bro/share/bro/policy/tuning/defaults/./packet-fragments.bro /usr/local/bro/share/bro/policy/tuning/defaults/./warnings.bro /usr/local/bro/share/bro/policy/frameworks/software/vulnerable.bro /usr/local/bro/share/bro/policy/frameworks/software/version-changes.bro /usr/local/bro/share/bro/policy/protocols/ftp/software.bro /usr/local/bro/share/bro/policy/protocols/smtp/software.bro /usr/local/bro/share/bro/policy/protocols/ssh/software.bro /usr/local/bro/share/bro/policy/protocols/http/software.bro /usr/local/bro/share/bro/policy/protocols/dns/detect-external-names.bro /usr/local/bro/share/bro/policy/protocols/ftp/detect.bro /usr/local/bro/share/bro/policy/protocols/conn/known-hosts.bro /usr/local/bro/share/bro/policy/protocols/conn/known-services.bro /usr/local/bro/share/bro/policy/protocols/ssl/known-certs.bro /usr/local/bro/share/bro/policy/protocols/ssl/cert-hash.bro /usr/local/bro/share/bro/policy/protocols/ssl/validate-certs.bro /usr/local/bro/share/bro/policy/protocols/ssh/geo-data.bro /usr/local/bro/share/bro/policy/protocols/ssh/detect-bruteforcing.bro /usr/local/bro/share/bro/policy/protocols/ssh/interesting-hostnames.bro /usr/local/bro/share/bro/policy/protocols/http/detect-MHR.bro /usr/local/bro/share/bro/policy/protocols/http/detect-sqli.bro /usr/local/bro/share/bro/broctl/__load__.bro /usr/local/bro/share/bro/broctl/./main.bro /usr/local/bro/share/bro/policy/frameworks/control/controllee.bro /usr/local/bro/share/bro/policy/frameworks/communication/listen.bro /usr/local/bro/share/bro/broctl/standalone.bro /logs/bro/spool/policy/auto/standalone-layout.bro /usr/local/bro/share/bro/policy/misc/trim-trace-file.bro /usr/local/bro/share/bro/broctl/auto.bro /logs/bro/spool/policy/auto/local-networks.bro /logs/bro/spool/policy/auto/broctl-config.bro From seth at icir.org Thu Mar 1 06:16:21 2012 From: seth at icir.org (Seth Hall) Date: Thu, 1 Mar 2012 09:16:21 -0500 Subject: [Bro] Bro2 Random Crashes In-Reply-To: References: Message-ID: <4349CEFF-76DB-46F6-8892-C63621C06A34@icir.org> On Mar 1, 2012, at 9:02 AM, Chris Crawford wrote: > After taking a look at one of my crash reports, Seth suggested that > I'm running out of memory. ?snip... > ==== stderr.log > listening on eth1, capture length 8192 bytes > > terminate called after throwing an instance of 'std::bad_alloc' > what(): std::bad_alloc For a bit of extra context, this was the line that made me think he was running out of memory. I used to see this more often than I wish I ever did when I ran an extremely memory starved cluster (and before many of the memory leaks were fixed for 2.0). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From christopher.p.crawford at gmail.com Thu Mar 1 06:30:42 2012 From: christopher.p.crawford at gmail.com (Chris Crawford) Date: Thu, 1 Mar 2012 09:30:42 -0500 Subject: [Bro] Bro2 Random Crashes In-Reply-To: <4349CEFF-76DB-46F6-8892-C63621C06A34@icir.org> References: <4349CEFF-76DB-46F6-8892-C63621C06A34@icir.org> Message-ID: I am using broctl to manage a single, local instance of bro. Would it make a difference if I just ran bro by hand, without broctl? -Chris On Thu, Mar 1, 2012 at 9:16 AM, Seth Hall wrote: > > On Mar 1, 2012, at 9:02 AM, Chris Crawford wrote: > >> After taking a look at one of my crash reports, Seth suggested that >> I'm running out of memory. > ?snip... >> ==== stderr.log >> listening on eth1, capture length 8192 bytes >> >> terminate called after throwing an instance of 'std::bad_alloc' >> what(): ?std::bad_alloc > > For a bit of extra context, this was the line that made me think he was running out of memory. ?I used to see this more often than I wish I ever did when I ran an extremely memory starved cluster (and before many of the memory leaks were fixed for 2.0). > > ?.Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > From nweaver at ICSI.Berkeley.EDU Thu Mar 1 06:32:37 2012 From: nweaver at ICSI.Berkeley.EDU (Nicholas Weaver) Date: Thu, 1 Mar 2012 06:32:37 -0800 Subject: [Bro] Bro2 Random Crashes In-Reply-To: References: Message-ID: <83F128A4-E797-484D-9326-20E33DC839B2@icsi.berkeley.edu> On Mar 1, 2012, at 6:02 AM, Chris Crawford wrote: > The only thing I've been able to key in on, is that over time Bro2 > eventually causes all free memory in Ubuntu to change to cached > memory. From the Citrix Xen Console, that cached memory shows up as > used memory. So, maybe Xen interprets cached memory as being used? > Also -- when Xen senses that most of the memory is "used" (but, > really, it's cached inside Ubuntu), the percent utilization in one of > the CPUs in the VM spikes. After Bro2 crashes, CPU utilization > returns to normal, but memory is never freed -- it remains cached > forever. Cached memory IS used memory by the OS. Modern operating systems are very aggressive about using all available "physical" memory as a disk cache when otherwise not being used, especially Linux variants. But at the same time, it should also aggressively evict those entries when it think that there are limits on physical memory. [1] So naturally, a virtual machine should see the disk cache as being "used" memory, since it is being used by the guest OS. But at the same time, this should not necessarily be a problem, since the guest OS in the virtual machine is managing this memory and will evict it for other, higher priority uses on demand. But it does bring up a question: within the virtual machine, how much physical memory does the VM think it has? How does that compare to the actual allocation to the VM environment? From seth at icir.org Thu Mar 1 06:48:49 2012 From: seth at icir.org (Seth Hall) Date: Thu, 1 Mar 2012 09:48:49 -0500 Subject: [Bro] Bro2 Random Crashes In-Reply-To: <83F128A4-E797-484D-9326-20E33DC839B2@icsi.berkeley.edu> References: <83F128A4-E797-484D-9326-20E33DC839B2@icsi.berkeley.edu> Message-ID: <7BAB9ADC-4FFB-47C4-A941-D49CBAE4935C@icir.org> On Mar 1, 2012, at 9:32 AM, Nicholas Weaver wrote: > But it does bring up a question: within the virtual machine, how much physical memory does the VM think it has? How does that compare to the actual allocation to the VM environment? That's a good point. I wonder if Xen is telling the guest OS that it still has memory available but then it eventually rejects a request for more memory? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From gregor at icir.org Fri Mar 2 11:54:51 2012 From: gregor at icir.org (Gregor Maier) Date: Fri, 02 Mar 2012 11:54:51 -0800 Subject: [Bro] Bro2 Random Crashes In-Reply-To: <83F128A4-E797-484D-9326-20E33DC839B2@icsi.berkeley.edu> References: <83F128A4-E797-484D-9326-20E33DC839B2@icsi.berkeley.edu> Message-ID: <4F51258B.7080102@icir.org> On 3/1/12 6:32 , Nicholas Weaver wrote: > > > On Mar 1, 2012, at 6:02 AM, Chris Crawford wrote: >> The only thing I've been able to key in on, is that over time Bro2 >> eventually causes all free memory in Ubuntu to change to cached >> memory. From the Citrix Xen Console, that cached memory shows up as >> used memory. So, maybe Xen interprets cached memory as being used? >> Also -- when Xen senses that most of the memory is "used" (but, >> really, it's cached inside Ubuntu), the percent utilization in one of >> the CPUs in the VM spikes. After Bro2 crashes, CPU utilization >> returns to normal, but memory is never freed -- it remains cached >> forever. > > Cached memory IS used memory by the OS. Modern operating systems are very aggressive about using all available "physical" memory as a disk cache when otherwise not being used, especially Linux variants. But at the same time, it should also aggressively evict those entries when it think that there are limits on physical memory. [1] Indeed. I found in several other cases that Linux caching strategy can be really annoying. I'm regularly having performance problem on with high throughput workloads.... cu Gregor From gc355804 at ohio.edu Sun Mar 4 12:27:38 2012 From: gc355804 at ohio.edu (G. Clark) Date: Sun, 04 Mar 2012 15:27:38 -0500 Subject: [Bro] Bro2 Random Crashes In-Reply-To: <4F51258B.7080102@icir.org> References: <83F128A4-E797-484D-9326-20E33DC839B2@icsi.berkeley.edu> <4F51258B.7080102@icir.org> Message-ID: <4F53D03A.7050803@ohio.edu> http://www.westnet.com/~gsmith/content/linux-pdflush.htm Specifically: dirty_background_ratio: Primary tunable to adjust, probably downward. If your goal is to reduce the amount of data Linux keeps cached in memory, so that it writes it more consistently to the disk rather than in a batch, lowering dirty_background_ratio is the most effective way to do that. It is more likely the default is too large in situations where the system has large amounts of memory and/or slow physical I/O. You might also take a look at: http://www.mjmwired.net/kernel/Documentation/cgroups/memory.txt HTH --Gilbert On 3/2/12 2:54 PM, Gregor Maier wrote: > On 3/1/12 6:32 , Nicholas Weaver wrote: >> >> >> On Mar 1, 2012, at 6:02 AM, Chris Crawford wrote: >>> The only thing I've been able to key in on, is that over time Bro2 >>> eventually causes all free memory in Ubuntu to change to cached >>> memory. From the Citrix Xen Console, that cached memory shows up as >>> used memory. So, maybe Xen interprets cached memory as being used? >>> Also -- when Xen senses that most of the memory is "used" (but, >>> really, it's cached inside Ubuntu), the percent utilization in one of >>> the CPUs in the VM spikes. After Bro2 crashes, CPU utilization >>> returns to normal, but memory is never freed -- it remains cached >>> forever. >> >> Cached memory IS used memory by the OS. Modern operating systems are very aggressive about using all available "physical" memory as a disk cache when otherwise not being used, especially Linux variants. But at the same time, it should also aggressively evict those entries when it think that there are limits on physical memory. [1] > > Indeed. I found in several other cases that Linux caching strategy can > be really annoying. I'm regularly having performance problem on with > high throughput workloads.... > > > > cu > Gregor > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From himself at louruppert.com Tue Mar 6 07:07:49 2012 From: himself at louruppert.com (Lou RUPPERT) Date: Tue, 06 Mar 2012 10:07:49 -0500 Subject: [Bro] Disabling mailed trace-summary reports Message-ID: <4F562845.5020007@louruppert.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hey, Has anyone found a way to disable the trace-summary traffic reports? The only way I've been able to do it, short of modifying the script code itself, is to unset the SendMail variable in broctl.cfg. There must be a better way. I also wonder how good the IPV6 support is in 2.1. Maybe someone can comment on that. On the system I do use the reports with, I had to write my own wrappers to the inet_aton and inet_ntoa functions and then comment out the IPV6 subnets in networks.cfg. That seems a lot hackier than I like, but I understand that v6 isn't officially supported in 2.0. - -Lou -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9WKEUACgkQEn9NahFdz2KjagCg1Ytr96jk/4fOJzE5M732Rjv3 6T0AoLNoiOzV7lBiBWuVRrE1JeW3eMcE =Tu/1 -----END PGP SIGNATURE----- From relevantusername at gmail.com Tue Mar 6 07:23:43 2012 From: relevantusername at gmail.com (relevant username) Date: Tue, 6 Mar 2012 09:23:43 -0600 Subject: [Bro] Ignoring hosts or ranges? Message-ID: I was wondering what the best way to ignore certain hosts or ranges would be. I found some documentation from 2004 on this, but it doesn't look like it's applicable any more. The reason for this is that we're working to extract certain data from the connection log but our network scanners are creating a lot of entries in conn.log that we don't care about. We can of course filter this all out after it's in the log, but for the sake of simplicity I was hoping there would be a way to do this in bro. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120306/893a6107/attachment.html From robin at icir.org Tue Mar 6 07:30:02 2012 From: robin at icir.org (Robin Sommer) Date: Tue, 6 Mar 2012 07:30:02 -0800 Subject: [Bro] Disabling mailed trace-summary reports In-Reply-To: <4F562845.5020007@louruppert.com> References: <4F562845.5020007@louruppert.com> Message-ID: <20120306153002.GA16607@icir.org> On Tue, Mar 06, 2012 at 10:07 -0500, Lou RUPPERT wrote: > Has anyone found a way to disable the trace-summary traffic reports? Adding a line "TraceSummary=" to broctl.cfg (i.e., clearing that option) should disable it. > I also wonder how good the IPV6 support is in 2.1. Maybe someone can > comment on that. On the system I do use the reports with, The reports will fully support IPv6 in 2.1. The main piece of that (in addition to the inet_aton changes) is adding support to the underlying Python library doing subnet lookups. We have a contributed patch for that already but not yet merged it in (http://tracker.bro-ids.org/bro/ticket/750). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From JAzoff at albany.edu Tue Mar 6 08:07:24 2012 From: JAzoff at albany.edu (Justin Azoff) Date: Tue, 6 Mar 2012 11:07:24 -0500 Subject: [Bro] Ignoring hosts or ranges? In-Reply-To: References: Message-ID: <20120306160724.GH15524@datacomm.albany.edu> On Tue, Mar 06, 2012 at 09:23:43AM -0600, relevant username wrote: > I was wondering what the best way to ignore certain hosts or ranges would be. > I found some documentation from 2004 on this, but it doesn't look like it's > applicable any more. Give something like this a try: redef PacketFilter::all_packets = F; # don't capture all packets redef capture_filters = [[ "all"] = "ip or not ip"]; redef restrict_filters += [ ["not-scanners"] = "not host 192.168.1.100 and not host 192.168.2.100"]; -- -- Justin Azoff -- Network Security & Performance Analyst From seth at icir.org Tue Mar 6 08:18:47 2012 From: seth at icir.org (Seth Hall) Date: Tue, 6 Mar 2012 11:18:47 -0500 Subject: [Bro] Ignoring hosts or ranges? In-Reply-To: <20120306160724.GH15524@datacomm.albany.edu> References: <20120306160724.GH15524@datacomm.albany.edu> Message-ID: <6383ED75-03ED-41F1-A85A-2BD4B0C0BADF@icir.org> On Mar 6, 2012, at 11:07 AM, Justin Azoff wrote: > Give something like this a try: Thanks for that reply Justin! I was in the middle of writing exactly the same thing. Just for the record I want to point out that this (incredibly common!) scenario is being made much easier for 2.1. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From christopher.p.crawford at gmail.com Tue Mar 6 13:05:43 2012 From: christopher.p.crawford at gmail.com (Chris Crawford) Date: Tue, 6 Mar 2012 16:05:43 -0500 Subject: [Bro] Bro2 Random Crashes In-Reply-To: <4F53D03A.7050803@ohio.edu> References: <83F128A4-E797-484D-9326-20E33DC839B2@icsi.berkeley.edu> <4F51258B.7080102@icir.org> <4F53D03A.7050803@ohio.edu> Message-ID: Thanks for the suggestions. I ended up using tcpdump to do the network capture. I wrote a cron job to regularly run bro over the the files generated by tcpdump. Definitely not as ideal as having bro run continuously, but it gets the job done reliably without crashing. This might be an interesting issue for Bro in the future, though. Running bro continuously from a xen VM would be incredibly useful. I'd be curious if anybody can reproduce the crash conditions, or if it is just something weird about my setup or environment. -Chris On Sun, Mar 4, 2012 at 3:27 PM, G. Clark wrote: > http://www.westnet.com/~gsmith/content/linux-pdflush.htm > > Specifically: > > dirty_background_ratio: Primary tunable to adjust, probably downward. If > your goal is to reduce the amount of data Linux keeps cached in memory, > so that it writes it more consistently to the disk rather than in a > batch, lowering dirty_background_ratio is the most effective way to do > that. It is more likely the default is too large in situations where the > system has large amounts of memory and/or slow physical I/O. > > You might also take a look at: > > http://www.mjmwired.net/kernel/Documentation/cgroups/memory.txt > > HTH > > --Gilbert > > On 3/2/12 2:54 PM, Gregor Maier wrote: >> On 3/1/12 6:32 , Nicholas Weaver wrote: >>> >>> >>> On Mar 1, 2012, at 6:02 AM, Chris Crawford wrote: >>>> The only thing I've been able to key in on, is that over time Bro2 >>>> eventually causes all free memory in Ubuntu to change to cached >>>> memory. ?From the Citrix Xen Console, that cached memory shows up as >>>> used memory. ?So, maybe Xen interprets cached memory as being used? >>>> Also -- when Xen senses that most of the memory is "used" (but, >>>> really, it's cached inside Ubuntu), the percent utilization in one of >>>> the CPUs in the VM spikes. ?After Bro2 crashes, CPU utilization >>>> returns to normal, but memory is never freed -- it remains cached >>>> forever. >>> >>> Cached memory IS used memory by the OS. ?Modern operating systems are very aggressive about using all available "physical" memory as a disk cache when otherwise not being used, especially Linux variants. ?But at the same time, it should also aggressively evict those entries when it think that there are limits on physical memory. [1] >> >> Indeed. I found in several other cases that Linux caching strategy can >> be really annoying. I'm regularly having performance problem on with >> high throughput workloads.... >> >> >> >> cu >> Gregor >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jmellander at lbl.gov Tue Mar 6 13:34:34 2012 From: jmellander at lbl.gov (Jim Mellander) Date: Tue, 6 Mar 2012 13:34:34 -0800 Subject: [Bro] Access to DNS TTL in lookup_hostname() Message-ID: Hi: I'm looking for a clean way to perform special processing based on the resolved ip address of specified hostnames - what I was thinking was using lookup_hostname() in a when statement to perform the lookup, but I really would like to access the TTL to schedule another lookup when it expires, so that we only perform the lookups as needed. Thanks in advance for your thoughts. From hlein at korelogic.com Wed Mar 7 11:53:53 2012 From: hlein at korelogic.com (Hank Leininger) Date: Wed, 7 Mar 2012 14:53:53 -0500 Subject: [Bro] Issues with DESTDIR support Message-ID: <20120307195353.GE28047@marklar.spinoli.org> Hello, I'm encountering some issues with DESTDIR support. I'm trying to create an updated Gentoo .ebuild for bro 2.0. Gentoo's packaging process basically compiles and installs into a sandbox directory under /var/tmp/portage/blahblahblah, using 'make DESTDIR=/var/tmp/portage/blahblahblah/image/ install', and then merges the contents of the sandbox into the real filesystem. Most of bro's build system supports DESTDIR correctly; lib/, share/, and bin/ all get populated properly under the sandbox directory. But, it seems that all config-file-writing code does not. For instance, any time a cmake_install.cmake file contains a section like: if (EXISTS /opt/bro/etc/something.cfg) ... ... cmake -E compare_files new_file /opt/bro/etc/something.cfg if not equal configure_file(new_file /opt/bro/etc/something.cfg.example) else ... configure_file(new_file /opt/bro/etc/something.cfg) I believe the right thing to do there is to leave the 'compare_files' call alone (looking in the real FS for something.cfg), but then the configure_file calls ought to write to $ENV{DESTDIR}/... I've tried manually tweaking CMakeLists.txt, cmake_install.cmake etc files to make that change, and I get... closer, but not quite. I'm a cmake n00b. Any advice, or patches that people have done to package for other distros with similar requirements would be much appreciated! Thanks, -- Hank Leininger D24D 2C2A F3AC B9AE CD03 B506 2D57 32E1 686B 6DB3 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 447 bytes Desc: Digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120307/58e0756a/attachment.bin From jsiwek at illinois.edu Wed Mar 7 13:54:09 2012 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Wed, 7 Mar 2012 21:54:09 +0000 Subject: [Bro] Issues with DESTDIR support In-Reply-To: <20120307195353.GE28047@marklar.spinoli.org> References: <20120307195353.GE28047@marklar.spinoli.org> Message-ID: > Most of bro's build system supports DESTDIR correctly; lib/, share/, and > bin/ all get populated properly under the sandbox directory. But, it > seems that all config-file-writing code does not. You're right, the InstallPackageConfigFile.cmake script does extra precautionary checks so that user-modified config files don't get clobbered at install time, but, in doing that, it worked around the normal file install process and ends up not respecting DESTDIR. If you can work from our git repository, I made a change on the "topic/jsiwek/destdir-fix" branch of our "cmake" repository that looks like it fixes the problem. To test it out, you'd do something like: git clone --recursive git://bro-ids.org/bro cd bro for repo in `find -d . -name cmake`; do (cd $repo && git fetch && git checkout topic/jsiwek/destdir-fix); done And then the usual configure, make, install stuff. Let me know if it works for you and I'll get it merged into the master branch. +Jon From hlein at korelogic.com Wed Mar 7 14:40:04 2012 From: hlein at korelogic.com (Hank Leininger) Date: Wed, 7 Mar 2012 17:40:04 -0500 Subject: [Bro] Issues with DESTDIR support In-Reply-To: References: <20120307195353.GE28047@marklar.spinoli.org> Message-ID: <20120307224003.GF28047@marklar.spinoli.org> On Wed, Mar 07, 2012 at 09:54:09PM +0000, Siwek, Jonathan Luke wrote: > > You're right, the InstallPackageConfigFile.cmake script does extra > precautionary checks so that user-modified config files don't get > clobbered at install time, but, in doing that, it worked around the > normal file install process and ends up not respecting DESTDIR. > > If you can work from our git repository, I made a change on the > "topic/jsiwek/destdir-fix" branch of our "cmake" repository that looks > like it fixes the problem. To test it out, you'd do something like: Sweet! This looks like it's about 99% there: install completes without error. The main broctl.cfg, broccoli.conf, etc config files are installed in /etc/. However, it looks like scripts/site/local*.bro files are installed into /share/bro/site/ The other scripts/ dir (base/, policy/) contents are installed under /opt/bro/share/bro/, which makes sense. I'm not sure where the site/ files are supposed to go -- I guess any of /etc/bro/site/, /var/opt/bro/site/, or even /opt/bro/share/bro/site/ would work. Thanks, -- Hank Leininger D24D 2C2A F3AC B9AE CD03 B506 2D57 32E1 686B 6DB3 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 447 bytes Desc: Digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120307/819a47a9/attachment.bin From will.havlovick at zenimax.com Thu Mar 8 07:03:47 2012 From: will.havlovick at zenimax.com (Will Havlovick) Date: Thu, 8 Mar 2012 10:03:47 -0500 Subject: [Bro] HTTP Post data Message-ID: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA279@usrkvexch01.zenimax.com> Hi all, Is there a way to write the data(body) of a HTTP Post request to the http.log? Or another log file? Thank you, Will -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120308/9ab4a90f/attachment.html From jsiwek at illinois.edu Thu Mar 8 08:17:37 2012 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Thu, 8 Mar 2012 16:17:37 +0000 Subject: [Bro] Issues with DESTDIR support In-Reply-To: <20120307224003.GF28047@marklar.spinoli.org> References: <20120307195353.GE28047@marklar.spinoli.org> <20120307224003.GF28047@marklar.spinoli.org> Message-ID: <93953F2F-8497-4DAB-80AC-D8E9367186E1@illinois.edu> > Sweet! This looks like it's about 99% there: install completes without > error. The main broctl.cfg, broccoli.conf, etc config files are > installed in /etc/. > > However, it looks like scripts/site/local*.bro files are installed into > /share/bro/site/ Seems in that change I did yesterday I was mistakenly replacing the install prefix with DESTDIR instead of just prepending to it. I fixed that in the same branch which you can try out by doing this command in your bro repository (assuming it's the same one from yesterday with the branch already checked out): for repo in `find . -type d -name cmake`; do (cd $repo && git pull); done And then try your build/install and let me know if looks alright to you now? +Jon From wirtz at in.tum.de Thu Mar 8 08:56:55 2012 From: wirtz at in.tum.de (Arne Wirtz) Date: Thu, 08 Mar 2012 17:56:55 +0100 Subject: [Bro] SSH-enhancement Message-ID: <4F58E4D7.8090007@in.tum.de> Hi all, I'm currently working on capturing and logging of further SSH-traffic to analyze used kex-algorithms: the negotiation which algorithm to use directly after the initial message, e.g. the Client send a SSH-Version-CLIENT - request to the server, the server answers with a SSH-Version-SERVER and directly afterwards the available kex-algorithms are exchanged. So I enhanced the SSH.cc (src/SSH.cc) and began logging. The log output: 192.168.1.50 59521 192.168.1.51 22 failure INBOUND SSH-2.0-OpenSSH_5.8p1 Debian-7ubuntu1 SSH-2.0-OpenSSH_5.8 \x16\x04\xd3\xef\x82\xa6/\x07\xb4\xecZA\xb5{\x98\xea\xee\x99\x7f\x04\xfe\xd8"\x9b{\xaf\x86\xbd\xd0\xe6y\x09\x1b\x0b\x9dg\xe7*a\x96\xc0\x09U\x89\xaf\xe5S\x0eoO\xfbD%x\xc4\x11\xda\x08\xc8qca\xffZ\x096\xe2rcZ#I"\x1f/?\xdfo\xdf\x88q\xf7\xb2\x0f\xc3\x99\xbf \xbe\xdd\x99\xf6\xec\x92\xbd~\xbb\x04\x91\xba\xcbIafi\xcf\xf6'I\x81|\xda!\xc4\xd7\x1c%9b\xf8\xe5\xaf\xc2\xfd}w\x87\xa0\xf5\xe4\xa3k\x91-\xc0qY\x0e\x84\xd9\x1ah\x19\x9e\xf5\xfc\xa52\x89n\xda\xee\x08\x0f\xfb\xde\xfbA*\xbd\x82\xfd\x17\x9f\xc6\xba\x04\x91\xcb\x86\xdb\x0e\xaa\xc26\x82 k\xd8%cU\x89\xbe\x10\x90kb\xc9\xe7A/sR:\x0a\x82\xa2\xe7\xb1c\xb6@\xcd\xcd\xa20T\xfe\xf2e\xaf\x8b\x04\xbc\xd3\xbb\x98\x84p\x97\x9c[\xfc\xed\x1a\xa5?W\x85\x9d;\xdf\x81\xf6\x03\xe8d\xeaWA*9\xf8\xc6 1999 - - - - - the relevant SSH.cc: SSH_Analyzer::SSH_Analyzer(Connection* c) : TCP_ApplicationAnalyzer(AnalyzerTag::SSH, c) { state = 0; //these two are global key = ""; orig = new ContentLine_Analyzer(c, true); orig->SetSkipPartial(false); orig->SetCRLFAsEOL(LF_as_EOL); AddSupportAnalyzer(orig); resp = new ContentLine_Analyzer(c, false); resp->SetSkipPartial(false); resp->SetCRLFAsEOL(LF_as_EOL); AddSupportAnalyzer(resp); } void SSH_Analyzer::DeliverStream(int length, const u_char* data, bool is_orig) { TCP_ApplicationAnalyzer::DeliverStream(length, data, is_orig); state=state+1; if (state < 3) { //here is the part with the ssh_server_version and ssh_client_version, I left it out because it works } else { if (TCP()) { event = ssh_add3; char tmp[length+strlen(key)]; memcpy (tmp,key,strlen(key)); memcpy (tmp,data,length); // here I concatenate old string with the new data and override the old data in the log key = tmp; StringVal* kex = new StringVal(key); val_list* vl = new val_list; vl->append(BuildConnVal()); vl->append(kex); ConnectionEvent(event, vl); return; } } I have 2 questions : 1 ) is it possible to change the logging in a more ascii style the way the first two exchanged packets are logged ? ( I tested different options for the ContentAnalyzer from ContentLine.cc, e.g. SetPlainDelivery and SetCRLFAsEOL, but all I got was hex style logging for the first packets. ) 2 ) I think the delivered data are not all there is, wireshark shows more package content, am I missing something ? Thanks for all your help, Arne From seth at icir.org Thu Mar 8 09:14:18 2012 From: seth at icir.org (Seth Hall) Date: Thu, 8 Mar 2012 12:14:18 -0500 Subject: [Bro] SSH-enhancement In-Reply-To: <4F58E4D7.8090007@in.tum.de> References: <4F58E4D7.8090007@in.tum.de> Message-ID: <2BC60433-F753-4BB4-A051-731B84DA8BFE@icir.org> On Mar 8, 2012, at 11:56 AM, Arne Wirtz wrote: > I have 2 questions : > 1 ) is it possible to change the logging in a more ascii style the way > the first two exchanged packets are logged ? ( I tested different > options for the ContentAnalyzer from ContentLine.cc, e.g. > SetPlainDelivery and SetCRLFAsEOL, but all I got was hex style logging > for the first packets. ) I'm a little unclear about the changes you made. If you could work with our repository and send us a diff that would be much more helpful. I do think that part of your problem is that you aren't actually parsing those fields. You're just shoving the data after the version exchange into a string but there is a lot of structure to it which you are just directly including in your output. > 2 ) I think the delivered data are not all there is, wireshark shows > more package content, am I missing something ? It's funny that you are looking into this. I've been planning on overhauling the SSH analyzer very soon myself. I was going to turn the whole analyzer into a binpac based analyzer and my plan was to extract a lot more data than is currently extracted. It should address what you are trying to do at least.  .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From vallentin at icir.org Thu Mar 8 09:29:54 2012 From: vallentin at icir.org (Matthias Vallentin) Date: Thu, 8 Mar 2012 09:29:54 -0800 Subject: [Bro] HTTP Post data In-Reply-To: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA279@usrkvexch01.zenimax.com> References: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA279@usrkvexch01.zenimax.com> Message-ID: > Is there a way to write the data(body) of a HTTP Post request to the > http.log? Or another log file? Yes, that's possible. You would have to reassemble the data from the body across the http_entity_* events. Here is an example of how one could do it: https://github.com/mavam/brospects/blob/master/bro/bodies.bro Matthias From will.havlovick at zenimax.com Fri Mar 9 06:35:03 2012 From: will.havlovick at zenimax.com (Will Havlovick) Date: Fri, 9 Mar 2012 09:35:03 -0500 Subject: [Bro] HTTP Post data In-Reply-To: References: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA279@usrkvexch01.zenimax.com> Message-ID: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA27C@usrkvexch01.zenimax.com> Very cool! I will check this out. We have had some interesting data in forms that are being submitted. Thank you, Will -----Original Message----- From: matthias at vallentin.net [mailto:matthias at vallentin.net] On Behalf Of Matthias Vallentin Sent: Thursday, March 08, 2012 12:30 PM To: Will Havlovick Cc: bro at bro-ids.org Subject: Re: [Bro] HTTP Post data > Is there a way to write the data(body) of a HTTP Post request to the > http.log? Or another log file? Yes, that's possible. You would have to reassemble the data from the body across the http_entity_* events. Here is an example of how one could do it: https://github.com/mavam/brospects/blob/master/bro/bodies.bro Matthias From mcholste at gmail.com Fri Mar 9 07:57:37 2012 From: mcholste at gmail.com (Martin Holste) Date: Fri, 9 Mar 2012 09:57:37 -0600 Subject: [Bro] HTTP Post data In-Reply-To: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA27C@usrkvexch01.zenimax.com> References: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA279@usrkvexch01.zenimax.com> <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA27C@usrkvexch01.zenimax.com> Message-ID: This is important enough that the Bro team might want to work on something that's on by default. Specifically, many attackers hide SQLi in POST params, so auto-extracting and logging some default, finite limit of POST params into the HTTP log would be a big win for the community. On Fri, Mar 9, 2012 at 8:35 AM, Will Havlovick wrote: > Very cool! > > I will check this out. ?We have had some interesting data in forms that are being submitted. > > Thank you, > > Will > > -----Original Message----- > From: matthias at vallentin.net [mailto:matthias at vallentin.net] On Behalf Of Matthias Vallentin > Sent: Thursday, March 08, 2012 12:30 PM > To: Will Havlovick > Cc: bro at bro-ids.org > Subject: Re: [Bro] HTTP Post data > >> Is there a way to write the data(body) of a HTTP Post request to the >> http.log? Or another log file? > > Yes, that's possible. You would have to reassemble the data from the body across the http_entity_* events. Here is an example of how one could do it: > > https://github.com/mavam/brospects/blob/master/bro/bodies.bro > > ? ?Matthias > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From seth at icir.org Fri Mar 9 09:11:26 2012 From: seth at icir.org (Seth Hall) Date: Fri, 9 Mar 2012 12:11:26 -0500 Subject: [Bro] HTTP Post data In-Reply-To: References: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA279@usrkvexch01.zenimax.com> <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA27C@usrkvexch01.zenimax.com> Message-ID: On Mar 9, 2012, at 10:57 AM, Martin Holste wrote: > This is important enough that the Bro team might want to work on > something that's on by default. Specifically, many attackers hide > SQLi in POST params, so auto-extracting and logging some default, > finite limit of POST params into the HTTP log would be a big win for > the community. Yep, I've done that before and (again!) it's another source of perspective change on network traffic. Regarding the SQLi detection, I've been planning on extending the SQLi detection script to cover POST data for a long time. Adding post data to the logs is at least easy. I attached a script which will just blindly add a configurable amount of data to your http.log. I'm not so sure it would ever be turned on by default, but we can certainly consider including a script that does this. It's a load statement away from being enabled that way. ;) -------------- next part -------------- A non-text attachment was scrubbed... Name: http-extract-post.bro Type: application/octet-stream Size: 574 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120309/a012cc40/attachment.obj -------------- next part -------------- .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From hlein at korelogic.com Fri Mar 9 18:17:59 2012 From: hlein at korelogic.com (Hank Leininger) Date: Fri, 9 Mar 2012 21:17:59 -0500 Subject: [Bro] Issues with DESTDIR support In-Reply-To: <93953F2F-8497-4DAB-80AC-D8E9367186E1@illinois.edu> References: <20120307195353.GE28047@marklar.spinoli.org> <20120307224003.GF28047@marklar.spinoli.org> <93953F2F-8497-4DAB-80AC-D8E9367186E1@illinois.edu> Message-ID: <20120310021759.GM28047@marklar.spinoli.org> On Thu, Mar 08, 2012 at 04:17:37PM +0000, Siwek, Jonathan Luke wrote: > > Seems in that change I did yesterday I was mistakenly replacing the > install prefix with DESTDIR instead of just prepending to it. I fixed > that in the same branch which you can try out by doing this command in > your bro repository (assuming it's the same one from yesterday with > the branch already checked out): I think this is getting closer. But it throws an error at: [snip] -- Installing: /var/tmp/portage/net-analyzer/bro-2.0.20120309/image/var/opt/bro/spool/scripts -- Installing: /var/tmp/portage/net-analyzer/bro-2.0.20120309/image/var/opt/bro/logs ACCESS DENIED symlink: /opt/bro/share/broctl/scripts/broctl-config.sh -- Installing: /var/tmp/portage/net-analyzer/bro-2.0.20120309/image///opt/bro/etc/broctl.cfg -- Installing: /var/tmp/portage/net-analyzer/bro-2.0.20120309/image///opt/bro/etc/networks.cfg [snip] It looks like the command that's failing is: /usr/bin/cmake -E create_symlink /var/opt/bro/spool/broctl-config.sh /opt/bro/share/broctl/scripts/broctl-config.sh That's in ./build/aux/broctl/cmake_install.cmake. When I manually add $ENV{DESTDIR} as a prefix for the destination and re-merge, it works cleanly, and I end up with a sane-looking install--except that /var/opt/bro/spool/broctl-config.sh does not exist, so the symlink at /opt/bro/share/broctl/scripts/broctl-config.sh is dangling (could be a side effect of my manual hack?) Thanks, Hank -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 447 bytes Desc: Digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120309/575ccf8a/attachment.bin From jsiwek at illinois.edu Mon Mar 12 12:17:51 2012 From: jsiwek at illinois.edu (Siwek, Jon) Date: Mon, 12 Mar 2012 14:17:51 -0500 Subject: [Bro] Issues with DESTDIR support In-Reply-To: <20120310021759.GM28047@marklar.spinoli.org> References: <20120307195353.GE28047@marklar.spinoli.org> <20120307224003.GF28047@marklar.spinoli.org> <93953F2F-8497-4DAB-80AC-D8E9367186E1@illinois.edu> <20120310021759.GM28047@marklar.spinoli.org> Message-ID: <4F5E4BDF.2030008@illinois.edu> On 3/9/2012 8:17 PM, Hank Leininger wrote: > > It looks like the command that's failing is: > > /usr/bin/cmake -E create_symlink /var/opt/bro/spool/broctl-config.sh /opt/bro/share/broctl/scripts/broctl-config.sh Yeah, sorry, there was another place in InstallSymlink.cmake that does that at install-time. I changed that to prepend DESTDIR if it's set. Can you pull from "topic/jsiwek/destdir-fix" again like you did before: for repo in `find . -type d -name cmake`; do (cd $repo&& git pull); done And let me know how it works out for you? > so the symlink at > /opt/bro/share/broctl/scripts/broctl-config.sh is dangling (could be a > side effect of my manual hack?) That's fine, the file is generated later by `broctl install`. +Jon From christopher.p.crawford at gmail.com Tue Mar 13 11:22:38 2012 From: christopher.p.crawford at gmail.com (Chris Crawford) Date: Tue, 13 Mar 2012 14:22:38 -0400 Subject: [Bro] MD5 Hashing Message-ID: What is the correct way to turn on MD5 hashing in SMTP and HTTP logs? Which variables do I need to set in my share/bro/site/local.bro ? -Chris From seth at icir.org Tue Mar 13 11:55:42 2012 From: seth at icir.org (Seth Hall) Date: Tue, 13 Mar 2012 14:55:42 -0400 Subject: [Bro] MD5 Hashing In-Reply-To: References: Message-ID: <7B34ECAB-B151-4AA3-9D8B-A25A7365D63D@icir.org> On Mar 13, 2012, at 2:22 PM, Chris Crawford wrote: > What is the correct way to turn on MD5 hashing in SMTP and HTTP logs? > Which variables do I need to set in my share/bro/site/local.bro ? # Windows executables are hashed by default (it's a regex matching the mime type of the file) redef HTTP::generate_md5 += /image.*/; redef SMTP::generate_md5 += /image.*/; Those were pulled from these pages in our docs? http://www.bro-ids.org/documentation/scripts/base/protocols/http/file-hash.html#id-HTTP::generate_md5 http://www.bro-ids.org/documentation/scripts/base/protocols/smtp/entities.html#id-SMTP::generate_md5 This is being seriously reworked for 2.1 right now too. There is going to be a file analysis policy where you will be able to be declare more easily with much better granularity when you'd like to do certain analyses. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From christopher.p.crawford at gmail.com Tue Mar 13 12:24:05 2012 From: christopher.p.crawford at gmail.com (Chris Crawford) Date: Tue, 13 Mar 2012 15:24:05 -0400 Subject: [Bro] MD5 Hashing In-Reply-To: <7B34ECAB-B151-4AA3-9D8B-A25A7365D63D@icir.org> References: <7B34ECAB-B151-4AA3-9D8B-A25A7365D63D@icir.org> Message-ID: Sounds simple enough. So, hypothetically, if I wanted SMTP to MD5 hash all mime types that are image.* or application.*, I would add the lines below to my local.bro? redef SMTP::generate_md5 += /image.*/; redef SMTP::generate_md5 += /application.*/; I'm assuming that the += operator appends new regular expressions. Is that correct? -Chris On Tue, Mar 13, 2012 at 2:55 PM, Seth Hall wrote: > > On Mar 13, 2012, at 2:22 PM, Chris Crawford wrote: > >> What is the correct way to turn on MD5 hashing in SMTP and HTTP logs? >> Which variables do I need to set in my share/bro/site/local.bro ? > > > # Windows executables are hashed by default (it's a regex matching the mime type of the file) > redef HTTP::generate_md5 += /image.*/; > redef SMTP::generate_md5 += /image.*/; > > Those were pulled from these pages in our docs? > http://www.bro-ids.org/documentation/scripts/base/protocols/http/file-hash.html#id-HTTP::generate_md5 > http://www.bro-ids.org/documentation/scripts/base/protocols/smtp/entities.html#id-SMTP::generate_md5 > > This is being seriously reworked for 2.1 right now too. ?There is going to be a file analysis policy where you will be able to be declare more easily with much better granularity when you'd like to do certain analyses. > > ?.Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > From seth at icir.org Tue Mar 13 12:30:30 2012 From: seth at icir.org (Seth Hall) Date: Tue, 13 Mar 2012 15:30:30 -0400 Subject: [Bro] MD5 Hashing In-Reply-To: References: <7B34ECAB-B151-4AA3-9D8B-A25A7365D63D@icir.org> Message-ID: <4137BB78-94A2-4D96-8C7D-3FD74EDC8A4E@icir.org> On Mar 13, 2012, at 3:24 PM, Chris Crawford wrote: > So, hypothetically, if I wanted SMTP to MD5 hash all mime types that > are image.* or application.*, I would add the lines below to my > local.bro? > > redef SMTP::generate_md5 += /image.*/; > redef SMTP::generate_md5 += /application.*/; Yep, just keeping in mind that the PDF mime type falls within application/ too (and a number of others). > I'm assuming that the += operator appends new regular expressions. Is > that correct? Correct. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From sconzo at visiblerisk.com Tue Mar 13 12:54:27 2012 From: sconzo at visiblerisk.com (Mike Sconzo) Date: Tue, 13 Mar 2012 14:54:27 -0500 Subject: [Bro] MD5 Hashing In-Reply-To: <4137BB78-94A2-4D96-8C7D-3FD74EDC8A4E@icir.org> References: <7B34ECAB-B151-4AA3-9D8B-A25A7365D63D@icir.org> <4137BB78-94A2-4D96-8C7D-3FD74EDC8A4E@icir.org> Message-ID: Will the changes in 2.1 allow for passing of data to an MD5 function? Or will it (the file analysis policy) use protocol knowledge + magic number to determine if it should be MD5'd or not? I only ask because seeing an exe downloaded with a mime type of image/jpg is not completely uncommon. On Tue, Mar 13, 2012 at 2:30 PM, Seth Hall wrote: > > On Mar 13, 2012, at 3:24 PM, Chris Crawford wrote: > >> So, hypothetically, if I wanted SMTP to MD5 hash all mime types that >> are image.* or application.*, I would add the lines below to my >> local.bro? >> >> redef SMTP::generate_md5 += /image.*/; >> redef SMTP::generate_md5 += /application.*/; > > Yep, just keeping in mind that the PDF mime type falls within application/ too (and a number of others). > >> I'm assuming that the += operator appends new regular expressions. ?Is >> that correct? > > > Correct. > > ?.Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- cat ~/.bash_history > documentation.txt From seth at icir.org Tue Mar 13 13:10:12 2012 From: seth at icir.org (Seth Hall) Date: Tue, 13 Mar 2012 16:10:12 -0400 Subject: [Bro] MD5 Hashing In-Reply-To: References: <7B34ECAB-B151-4AA3-9D8B-A25A7365D63D@icir.org> <4137BB78-94A2-4D96-8C7D-3FD74EDC8A4E@icir.org> Message-ID: <9C6753A2-2F55-497A-AFBA-27BE133FD719@icir.org> On Mar 13, 2012, at 3:54 PM, Mike Sconzo wrote: > Will the changes in 2.1 allow for passing of data to an MD5 function? > Or will it (the file analysis policy) use protocol knowledge + magic > number to determine if it should be MD5'd or not? That's only a cheat mechanism I put in place. You actually have a lot more flexibility than that if you write a bit of code. The HTTP::Info data structure is extended in the scripts/base/protocols/http/file-hash.bro script to get a field named "calc_md5". If you set that field to true (T) before the first chunk of data is seen Bro will calculate an MD5 sum for the transfer. If you handle the http_header event for example, you would just do your condition and then set the field to T. Here's a short and dumb example? event http_header(c: connection, is_orig: bool, name: string, value: string) { if ( ! is_orig && name == "CONTENT-TYPE" && value == "IMAGE/JPG" ) c$http$calc_md5 = T; } This will make Bro calculate md5 sums for any HTTP transfer where the server sent jpg as the content type (this is not what would be matched with the generate_md5 variable as I mention below). > I only ask because seeing an exe downloaded with a mime type of > image/jpg is not completely uncommon. Those mime types are sniffed (we ignore the content-type header). If it's a windows executable it will be detected as such. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From baxterw3232 at gmail.com Wed Mar 14 08:12:16 2012 From: baxterw3232 at gmail.com (Will) Date: Wed, 14 Mar 2012 10:12:16 -0500 Subject: [Bro] Adding SSL certs to Bro 2.0 In-Reply-To: References: <4EC3F2C7.50206@vanderbilt.edu> <3BCA38E4-AA36-4D3D-8DE0-E826D44C2B64@icir.org> <20120208175412.GD12752@yahoo.fr> Message-ID: Quick question. I am getting a timeout when trying to print the variable containing the root_certs. I am just wondering if this is due to having too many trusted certs loaded or if this is just a limitation of the broctl print function. My primary concern is whether bro was actually able to load all the root_certs on startup. I have over 500 certs redef'd in site/mytrustedcerts.bro. Is this too many for bro to handle? broctl print SSL::root_certs manager proxy-1 worker-1 worker-2 worker-3 worker-4 worker-5 worker-6 worker-7 worker-8 Thanks! -will On Wed, Feb 8, 2012 at 1:57 PM, Seth Hall wrote: > > On Feb 8, 2012, at 12:54 PM, Stephane Chazelas wrote: > >> In case it may be of some help to anyone, here is a script to >> convert a PEM CA cert bundle such as >> /etc/ssl/certs/ca-certificates.crt as found on debian based >> system to bro's format: > > > Cool, thanks. ?We have a script in bro-aux that generates the CA list Bro script directly from the Mozilla repository too. ?If you have a copy of our source tree, the script is here: > ? ? ? ?aux/bro-aux/devel-tools/gen-mozilla-ca-list.rb > > ?.Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From hlein at korelogic.com Wed Mar 14 20:29:42 2012 From: hlein at korelogic.com (Hank Leininger) Date: Wed, 14 Mar 2012 23:29:42 -0400 Subject: [Bro] Issues with DESTDIR support In-Reply-To: <4F5E4BDF.2030008@illinois.edu> References: <20120307195353.GE28047@marklar.spinoli.org> <20120307224003.GF28047@marklar.spinoli.org> <93953F2F-8497-4DAB-80AC-D8E9367186E1@illinois.edu> <20120310021759.GM28047@marklar.spinoli.org> <4F5E4BDF.2030008@illinois.edu> Message-ID: <20120315032942.GV28047@marklar.spinoli.org> On Mon, Mar 12, 2012 at 02:17:51PM -0500, Siwek, Jon wrote: > Yeah, sorry, there was another place in InstallSymlink.cmake that does > that at install-time. I changed that to prepend DESTDIR if it's set. > Can you pull from "topic/jsiwek/destdir-fix" again like you did before: > > for repo in `find . -type d -name cmake`; do (cd $repo&& git pull); done > > And let me know how it works out for you? This new version looks perfect. Ship it! =) Thanks, Hank -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 447 bytes Desc: Digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120314/b3552fcc/attachment.bin From robin at icir.org Thu Mar 15 08:26:18 2012 From: robin at icir.org (Robin Sommer) Date: Thu, 15 Mar 2012 08:26:18 -0700 Subject: [Bro] Adding SSL certs to Bro 2.0 In-Reply-To: References: <4EC3F2C7.50206@vanderbilt.edu> <3BCA38E4-AA36-4D3D-8DE0-E826D44C2B64@icir.org> <20120208175412.GD12752@yahoo.fr> Message-ID: <20120315152618.GI44856@icir.org> On Wed, Mar 14, 2012 at 10:12 -0500, you wrote: > Quick question. I am getting a timeout when trying to print the > variable containing the root_certs. I am just wondering if this is due > to having too many trusted certs loaded or if this is just a > limitation of the broctl print function. Pretty certainly the latter, the table itself shouldn't have a problem with many entries. Bro's current git version has a new option to increase the BroControl timeout for print, but here's another thing you can do to see the output: Add this to your local.bro: event bro_init() { print SSL::root_certs; } Get a small trace and run it through broctl's "process" command: broctl process path/to/trace This will print all the certs to stdout (and then process the trace). (Trace content doesn't matter; one of Bro's test traces will do, like bro/testing/btest/Traces/web.trace). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From baxterw3232 at gmail.com Thu Mar 15 12:24:08 2012 From: baxterw3232 at gmail.com (Will) Date: Thu, 15 Mar 2012 14:24:08 -0500 Subject: [Bro] Adding SSL certs to Bro 2.0 In-Reply-To: <20120315152618.GI44856@icir.org> References: <4EC3F2C7.50206@vanderbilt.edu> <3BCA38E4-AA36-4D3D-8DE0-E826D44C2B64@icir.org> <20120208175412.GD12752@yahoo.fr> <20120315152618.GI44856@icir.org> Message-ID: On Thu, Mar 15, 2012 at 10:26 AM, Robin Sommer wrote: > > On Wed, Mar 14, 2012 at 10:12 -0500, you wrote: > >> Quick question. I am getting a timeout when trying to print the >> variable containing the root_certs. I am just wondering if this is due >> to having too many trusted certs loaded or if this is just a >> limitation of the broctl print function. > > Pretty certainly the latter, the table itself shouldn't have a problem > with many entries. Bro's current git version has a new option to > increase the BroControl timeout for print, but here's another thing > you can do to see the output: > Excellent! Thanks Robin! > Add this to your local.bro: > > ? ?event bro_init() > ? ?{ > ? ? ? ?print SSL::root_certs; > ? ?} > > Get a small trace and run it through broctl's "process" command: > > ? ?broctl process path/to/trace > > This will print all the certs to stdout (and then process the trace). > > (Trace content doesn't matter; one of Bro's test traces will do, like > bro/testing/btest/Traces/web.trace). > Much appreciated! > Robin > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL ? ?* Fax ? +1 (510) 666-2956 * ? www.icir.org From hlein at korelogic.com Sat Mar 17 11:03:50 2012 From: hlein at korelogic.com (Hank Leininger) Date: Sat, 17 Mar 2012 14:03:50 -0400 Subject: [Bro] Support for gpg-encrypting email reports removed? Message-ID: <20120317180350.GI28047@marklar.spinoli.org> I'd like to encrypt and sign emails generated by Bro. It looks like up through 1.5.3, Bro supported PGP-encrypting emails it generated, but that support was dropped with the 2.0 release, although I don't see it listed in upgrade.html#removed-functionality so maybe nobody missed it. Was this dropped deliberately due to lack of user interest, or the code was known to be buggy, or something? Or just hasn't been re-added in 2.x yet, but it is planned? I'll probably make some local patches based on how 1.5.3's mail_reports.sh worked (will try to add --sign support while I'm at it); I'll post here if there's interest. Thanks, -- Hank Leininger D24D 2C2A F3AC B9AE CD03 B506 2D57 32E1 686B 6DB3 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 447 bytes Desc: Digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120317/1c915958/attachment.bin From robin at icir.org Tue Mar 20 17:17:00 2012 From: robin at icir.org (Robin Sommer) Date: Tue, 20 Mar 2012 17:17:00 -0700 Subject: [Bro] Support for gpg-encrypting email reports removed? In-Reply-To: <20120317180350.GI28047@marklar.spinoli.org> References: <20120317180350.GI28047@marklar.spinoli.org> Message-ID: <20120321001700.GA81568@icir.org> On Sat, Mar 17, 2012 at 14:03 -0400, you wrote: > It looks like up through 1.5.3, Bro supported PGP-encrypting emails it This must have been part of the old Bro Lite framework, which was in fact already deprecated in 1.5.x (and hence no further mentioning in the upgrade notes). BroControl (the replacemewnt for the Bro Lite scripts) indeed doesn't have PGP support yet. I would be a nice addition, but I don't think anybody is working on it yet. > I'll post here if there's interest. Sure, please do. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From rrotsted at pdx.edu Wed Mar 21 09:34:49 2012 From: rrotsted at pdx.edu (Bob Rotsted) Date: Wed, 21 Mar 2012 09:34:49 -0700 Subject: [Bro] Blacklist DNS alerting Message-ID: <4F6A0329.3010305@pdx.edu> Hello all, I recently spun up my first Bro instance and I'm trying to find the most elegant way to alert any time there is a query for a particular set of malicious domains (ex. https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist) . Would this be best accomplished with a signature? Would I be better off writing a hook for Bro's core DNS script? Any input will be greatly appreciated, Bob -- Bob Rotsted Network Security Analyst Portland State University Desk: 503-725-6215 Cell: 503-208-6575 314B D581 A8CD E28A A690 7E9D 5B43 4B28 0EB6 A21A From himself at louruppert.com Wed Mar 21 09:46:22 2012 From: himself at louruppert.com (Lou RUPPERT) Date: Wed, 21 Mar 2012 12:46:22 -0400 Subject: [Bro] Blacklist DNS alerting In-Reply-To: <4F6A0329.3010305@pdx.edu> References: <4F6A0329.3010305@pdx.edu> Message-ID: <4F6A05DE.8010408@louruppert.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/21/2012 12:34 PM, Bob Rotsted wrote: > Hello all, > > I recently spun up my first Bro instance and I'm trying to find the most > elegant way to alert any time there is a query for a particular set of > malicious domains (ex. > https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist) . > > Would this be best accomplished with a signature? Would I be better off > writing a hook for Bro's core DNS script? > > Any input will be greatly appreciated, I do it with a simple hook based on the 1.5.x DNS code. I then use a perl script to parse a list of domains into a data structure this can use. Lines in the output of that script look like this: redef DNS::hostile_domain_list += { "torpig-sinkhole.org", "riaa.com", "mpaa.org", }; And the bro hook code looks like this: module DNS; export { const hostile_domain_list: set[string] &redef; const okay_to_lookup_hostile_domains: set[addr] &redef; } redef okay_to_lookup_hostile_domains = { 192.168.1.1, 192.168.1.2, }; redef enum Notice::Type += { DNS_Malicious_Domain }; function second_level_domain(name: string): string { local split_on_dots = split(name, /\./); local num_dots = length(split_on_dots); if ( num_dots <= 1 ) return name; return fmt("%s.%s", split_on_dots[num_dots-1], split_on_dots[num_dots]); } event dns_request(c: connection, msg: dns_msg, query: string, qtype: count, qcla ss: count) &priority=0 { if (c$id$orig_h !in okay_to_lookup_hostile_domains) if (second_level_domain(query) in hostile_domain_list) local message=fmt("Test: Malware domain %s",query); NOTICE([$note=DNS_Malicious_Domain, $msg=message, $conn=c]); } -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEUEARECAAYFAk9qBd4ACgkQEn9NahFdz2LgHACYn0PsX2IsnD9iYoudCVCx/4mJ 8gCg5dUo9t3eWhtcCL6nhEzMrhVcqkk= =Bhtk -----END PGP SIGNATURE----- From qhu009 at aucklanduni.ac.nz Tue Mar 27 01:42:50 2012 From: qhu009 at aucklanduni.ac.nz (Qinwen Hu) Date: Tue, 27 Mar 2012 16:42:50 +0800 Subject: [Bro] How should Bro to read wireshark trace file Message-ID: Hi All: I am a new user, I have some trace files, which captured by using wireshark, when I use Bro to read it, I got the error message, which shows "fatal error: ./bro: problem with trace file alert1.pcap - alert1.pcap: No such file or directory" , or if I run "bro alert1 local", I got "fatal error: ./bro: problem with trace file alert1 - unknown file format" is anyone know how to solve this problem? Thank you very much Regards Steven -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120327/8483df16/attachment.html From seth at icir.org Tue Mar 27 06:05:33 2012 From: seth at icir.org (Seth Hall) Date: Tue, 27 Mar 2012 09:05:33 -0400 Subject: [Bro] How should Bro to read wireshark trace file In-Reply-To: References: Message-ID: On Mar 27, 2012, at 4:42 AM, Qinwen Hu wrote: > is anyone know how to solve this problem? You need to supply the pcap formatted trace file with the "-r" flag. bro -r alert1 local .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From sheharbano.k at gmail.com Wed Mar 28 04:24:41 2012 From: sheharbano.k at gmail.com (Sheharbano Khattak) Date: Wed, 28 Mar 2012 16:24:41 +0500 Subject: [Bro] Blacklist querying using Bro script Message-ID: Dear Bro Team, I maintain blacklists of botnet C&C servers, spam sources etc. These are usually distributed as text files. Every once in a while, i need to update these by re-downloading them or better yet, by using rsync. In other cases, the database is too large to be locally maintained e.g. DNSBL and i would rather make an online query. I want this process to be completely automated. That is to say, i want to provide Bro with a list of URL's from where these lists can be obtained at the time of invocation. In my Bro script, i want to handle reading these files and also 'refresh' the lists say every 24 hours. Occasionally, i want to be able to make online queries about the 'sanity' of certain IP addresses. Can i do this using Bro Script? If not, how do i go about doing this? Regards, -- Sheharbano Khattak Research Engineer / MS student NUST, Pakistan http://etheryell.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120328/d13f9467/attachment.html From mcholste at gmail.com Wed Mar 28 06:01:09 2012 From: mcholste at gmail.com (Martin Holste) Date: Wed, 28 Mar 2012 08:01:09 -0500 Subject: [Bro] Blacklist querying using Bro script In-Reply-To: References: Message-ID: I highly recommend checking out the Collective Intelligence Framework ( http://code.google.com/p/collective-intelligence-framework/) as a way to manage your blacklists. Of particular importance is its ability to store and share to authorized parties your org's own custom blacklists in a seamless way with other blacklists. On Wed, Mar 28, 2012 at 6:24 AM, Sheharbano Khattak wrote: > Dear Bro Team, > > I maintain blacklists of botnet C&C servers, spam sources etc. These are > usually distributed as text files. Every once in a while, i need to update > these by re-downloading them or better yet, by using rsync. In other cases, > the database is too large to be locally maintained e.g. DNSBL and i would > rather make an online query. > > I want this process to be completely automated. That is to say, i want to > provide Bro with a list of URL's from where these lists can be obtained at > the time of invocation. In my Bro script, i want to handle reading these > files and also 'refresh' the lists say every 24 hours. Occasionally, i want > to be able to make online queries about the 'sanity' of certain IP > addresses. > > Can i do this using Bro Script? If not, how do i go about doing this? > > Regards, > -- > Sheharbano Khattak > > Research Engineer / MS student > > NUST, Pakistan > > http://etheryell.com > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120328/19987e23/attachment.html From robin at icir.org Wed Mar 28 08:07:39 2012 From: robin at icir.org (Robin Sommer) Date: Wed, 28 Mar 2012 08:07:39 -0700 Subject: [Bro] Blacklist querying using Bro script In-Reply-To: References: Message-ID: <20120328150739.GB39215@icir.org> On Wed, Mar 28, 2012 at 16:24 +0500, Sheharbano Khattak wrote: > Can i do this using Bro Script? If not, how do i go about doing this? Sounds like there are two parts to this: (1) downloading/organizing/maintaining the information, and (2) then getting it into Bro. The former might be best done outside of Bro with something like CIF (as Martin wrote) or even just some simple shell scripts. For the latter, we're working on much better support than is currenty available: the next version will have a new input framework that can read and parse files dynamically at runtime (including dynamic updatess) and map the content into tables or events. It also supports querying other sources like a DB or doing external queries. There's a working prototype in git if you want to give it a try: branch topic/bernhard/input-threads. Feel free to send questions and feedback to the development list, we're still working on finalizing the specifics. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org