From christopher.p.crawford at gmail.com Wed Sep 5 11:12:09 2012 From: christopher.p.crawford at gmail.com (Chris Crawford) Date: Wed, 5 Sep 2012 14:12:09 -0400 Subject: [Bro] broctl Email Reports In-Reply-To: <20120531144414.GE48884@icir.org> References: <20120530182755.GA97952@icir.org> <20120531144414.GE48884@icir.org> Message-ID: What is the recommended way to completely disable hourly reports? On Thu, May 31, 2012 at 10:44 AM, Robin Sommer wrote: > > On Thu, May 31, 2012 at 09:22 -0400, Chris Crawford wrote: > >> Hmm...I restarted bro and it's still sending Connection Summary >> reports every hour. > > Ah, ok, I thought your question was only about the alarm summaries > (they should now come once a day). The connection summaries can't > really be detached from the rotation because that's a post-processor > working on the conn.log file at the time it's closed and archived. If > you want them daily (but keep rotating conn.log hourly), you'd need to > do that externally, like with a cron job running over the archived > conn.logs. > > (The tool that generates the summaries is "trace-summary", it can be > used standalone as well). > > Robin > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From Huiping.Song at ultra-3eti.com Thu Sep 6 13:27:06 2012 From: Huiping.Song at ultra-3eti.com (Huiping Song) Date: Thu, 6 Sep 2012 20:27:06 +0000 Subject: [Bro] How to set the common log file directory for Bro 2.0? Message-ID: <394E2A9510AB1642987C5AADE63C5476075869BD@RockMX01.rock.corp> When starting up bro from the command line (for example: bro -i eth0 local ), the log files are always written to the directory from which bro utility is invoked. When not using BroControl, is there any way to configure bro to write log files to a common directory like "/usr/local/bro/logs"? Any command line option or a quick script customization for doing so? Thank you, Huiping -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120906/a62bf0d7/attachment.html From robin at icir.org Thu Sep 6 21:29:15 2012 From: robin at icir.org (Robin Sommer) Date: Thu, 6 Sep 2012 21:29:15 -0700 Subject: [Bro] broctl Email Reports In-Reply-To: References: <20120530182755.GA97952@icir.org> <20120531144414.GE48884@icir.org> Message-ID: <20120907042915.GG26051@icir.org> On Wed, Sep 05, 2012 at 14:12 -0400, you wrote: > What is the recommended way to completely disable hourly reports? In broctl.cfg, add "TraceSummary=" (i.e., set it to empty) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From kkamin at 21ct.com Mon Sep 10 14:35:46 2012 From: kkamin at 21ct.com (Karl Kamin) Date: Mon, 10 Sep 2012 16:35:46 -0500 Subject: [Bro] SSH Message-ID: <04920BD67C651C469D0387704CD7692A742086436D@21ct-exg07.21technologies.com> We are setting up a cluster for Bro. I have setup two machines that have 2 worker threads each. The first machine runs the manager and proxy functions too. When I run broctl install it is apparent that the ssh connection fails. I have used password-less rsa logins for years and am familiar with creating rsa keys, and configuring ssh to use the keys. What I cannot figure out is how the bro user (bro) is configured to find the key. Here is the output from my install. Permissions for the user bro should be correct on both systems. (chown bro:bro /usr/local/bro -R) ++++++++++++++ [bro at payshuntzero bro]$ /usr/local/bro/bin/broctl check manager is ok. proxy-1 is ok. worker-1 is ok. worker-2 is ok. worker-3 is ok. worker-4 is ok. ++++++++++++++ [bro at payshuntzero bro]$ /usr/local/bro/bin/broctl install waiting for lock ....... ok removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/site ... done. removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/auto ... done. creating policy directories ... done. installing site policies ... done. generating cluster-layout.bro ... done. generating local-networks.bro ... done. generating broctl-config.bro ... done. updating nodes ... warning: host patientone is not alive warning: cannot create directory /usr/local/bro/spool/tmp on worker-3 warning: cannot create directory /usr/local/bro/spool/tmp on worker-3 warning: cannot create directory /usr/local/bro/spool/tmp on worker-3 warning: cannot create directory /usr/local/bro/spool/tmp on worker-3 warning: error rsyncing to patientone: ['Host key verification failed.\r', 'rsync: connection unexpectedly closed (0 bytes received so far) [sender]', 'rsync error: unexplained error (code 255) at io.c(600) [sender=3.0.6]'] done. [bro at payshuntzero bro]$ ++++++++++++++ Karl -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120910/1ca646be/attachment.html From jsiwek at illinois.edu Mon Sep 10 15:18:16 2012 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Mon, 10 Sep 2012 22:18:16 +0000 Subject: [Bro] SSH In-Reply-To: <04920BD67C651C469D0387704CD7692A742086436D@21ct-exg07.21technologies.com> References: <04920BD67C651C469D0387704CD7692A742086436D@21ct-exg07.21technologies.com> Message-ID: > What I cannot figure out is how the bro user (bro) is configured to find the key. > BroControl doesn't have any special options to tweak that, it would just be whatever is configured in ssh_config(5) on the local system or sshd_config(5) on the remote in the case you need to do something other than the standards/defaults for ssh. > warning: error rsyncing to patientone: ['Host key verification failed.\r', 'rsync: connection unexpectedly closed (0 bytes received so far) [sender]', 'rsync error: unexplained error (code 255) at io.c(600) [sender=3.0.6]'] > done. Are you able to just ssh into it on the command line? I'm guessing either 1) no entry is in ~/.ssh/known_hosts and rsync isn't going to automatically trust a host for you or 2) an orphaned entry is in ~/.ssh/known_hosts (maybe because the remote OS was re-installed fresh). Or there's a MITM. Jon From tritium.cat at gmail.com Mon Sep 10 17:51:39 2012 From: tritium.cat at gmail.com (Tritium Cat) Date: Tue, 11 Sep 2012 00:51:39 +0000 Subject: [Bro] Troubleshooting crashes In-Reply-To: References: Message-ID: Finally getting back to this. On Fri, Aug 31, 2012 at 1:18 AM, Seth Hall wrote: > > On Aug 30, 2012, at 5:46 PM, Tritium Cat wrote: > > > What's the best way to disable Bro in a systematic way to isolate > crashes ? > > Sending us the diag output from broctl is best since it will include a > back trace. ==== No reporter.log ==== stderr.log listening on eth5, capture length 8192 bytes /usr/local/3rd-party/bro/share/broctl/scripts/run-bro: line 60: 15452 Segmentation fault nohup $mybro $@ ==== stdout.log unlimited unlimited unlimited ==== .cmdline -i eth5 -U .status -p broctl -p broctl-live -p local -p worker-5-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/usr/local/3rd-party/bro/bin:/usr/local/3rd-party/bro/share/broctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games BROPATH=/usr/local/3rd-party/bro/spool/installed-scripts-do-not-touch/site::/usr/local/3rd-party/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/3rd-party/bro/share/bro:/usr/local/3rd-party/bro/share/bro/policy:/usr/local/3rd-party/bro/share/bro/site CLUSTER_NODE=worker-5-9 ==== .status RUNNING [net_run] ==== No prof.log ==== No packet_filter.log ==== No loaded_scripts.log -- [Automatically generated.] > > After ~12 hours I returned to find many of the worker nodes had > crashed. I forgot to look at the diag for the crashed workers before > stopping the cluster. > > Do you have the cron command setup correctly? The workers should have > been restart automatically after they crashed and a diagnostic email sent > to you. > Mentioned in this section: > http://bro-ids.org/documentation/quickstart.html#a-minimal-starting-configuration I did not; it's working properly now. (...) > Total rings : 10 > > How many CPU cores do you have? 48 per server. > > -rw-r--r-- 1 bro bro 10323 Aug 30 21:15 reporter.log > > -rw-r--r-- 1 bro bro 52846117 Aug 30 21:27 weird.log > > I'm curious about what's in reporter.log, normally that shouldn't have too > much in it. Also, that's an astonishingly large weird.log. Is there > anything that stands out in those two? > reporter.log -- looks like I need to setup GeoIPV6 database: /usr/share/GeoIP/GeoIPCityv6.dat (empty) 50 Reporter::INFO processing continued (empty) 50 Reporter::INFO Failed to open GeoIP database: 29 Reporter::INFO processing suspended (empty) weird.log -- bro at bc : [12:33am] : 2012-08-30 : ls -l weird.* | tail -5 -rw-r--r-- 1 bro bro 16757363 Aug 30 21:00 weird.20:00:00-21:00:00.log.gz -rw-r--r-- 1 bro bro 304697 Aug 30 21:02 weird.21:00:00-21:02:10.log.gz -rw-r--r-- 1 bro bro 39351508 Aug 30 22:00 weird.21:12:53-22:00:00.log.gz -rw-r--r-- 1 bro bro 55141105 Aug 30 23:00 weird.22:00:00-23:00:00.log.gz -rw-r--r-- 1 bro bro 38190282 Aug 31 00:00 weird.23:00:00-00:00:00.log.gz bro at bc : [12:33am] : 2012-08-30 : gzcat weird.23:00:00-00:00:00.log.gz | awk '{print $7}' | sort | uniq -c | sort -rn | head -10 614589 data_before_established 585445 possible_split_routing 260703 window_recision 190652 SYN_seq_jump 100211 inappropriate_FIN 64533 above_hole_data_without_any_acks 37882 connection_originator_SYN_ack 33611 data_after_reset 19106 Teredo_bubble_with_payload 11510 SYN_after_reset bro at bc : [12:34am] : current : awk '{print $7}' weird.log | sort | uniq -c | sort -rn | head -10 51561 window_recision 49218 possible_split_routing 47776 data_before_established 24526 Teredo_bubble_with_payload 19894 connection_originator_SYN_ack 11718 SYN_seq_jump 8938 inappropriate_FIN 8701 data_after_reset 7523 above_hole_data_without_any_acks 5765 inner_IP_payload_length_mismatch > Could you show me your node.cfg configuration too? > bro at bc : [12:41am] : bro : cat etc/node.cfg [manager] type=manager host=z.z.z.M [proxy-1] type=proxy host=z.z.z.M [worker-1] type=worker host=z.z.z.A interface=eth5 lb_procs=10 lb_method=pf_ring [worker-2] type=worker host=z.z.z.B interface=eth5 lb_procs=10 lb_method=pf_ring [worker-3] type=worker host=z.z.z.C interface=eth5 lb_procs=10 lb_method=pf_ring [worker-4] type=worker host=z.z.z.D interface=eth5 lb_procs=10 lb_method=pf_ring [worker-5] type=worker host=z.z.z.E interface=eth5 lb_procs=10 lb_method=pf_ring > Oh, and one last thing, have you made sure to disable all of special NIC > features? > > http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not-full.html Yeah, I've used those recommendations from the start with one exception; the Intel X520-DA2 cards I'm using do not support disabling "ufo" (UDP large send offload). # Adjust interface features # # Disable features on network card that may deliver super packets # # http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not-full.html # ethtool -K eth5 rx off ethtool -K eth5 tx off ethtool -K eth5 sg off ethtool -K eth5 tso off #ethtool -K eth5 ufo off ethtool -K eth5 gso off ethtool -K eth5 gro off ethtool -K eth5 lro off ethtool -K eth5 rxvlan off ethtool -K eth5 txvlan off ethtool -K eth5 ntuple on # ethtool -s eth5 speed 10000 duplex full ifconfig eth5 mtu 9600 ifconfig eth5 up -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120911/5a85d6b3/attachment.html From seth at icir.org Tue Sep 11 06:16:06 2012 From: seth at icir.org (Seth Hall) Date: Tue, 11 Sep 2012 09:16:06 -0400 Subject: [Bro] Troubleshooting crashes In-Reply-To: References: Message-ID: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> On Sep 10, 2012, at 8:51 PM, Tritium Cat wrote: > listening on eth5, capture length 8192 bytes I see later in the email that you have the MTU on your NIC set to 9600. You may want to add the following line to local.bro to make Bro's snap length match that. redef snaplen = 9600; > /usr/local/3rd-party/bro/share/broctl/scripts/run-bro: line 60: 15452 Segmentation fault nohup $mybro $@ Hm, looks like you aren't getting stack traces. Your OS is probably not keeping core dumps or writing them to some OS-wide core dump directory. Change the sysctl variable for your OS to dump core files and make sure they're being dropped into the CWD prefixed with "core". Daniel, do you think that's something that you could add to the documentation somewhere? > bro at bc : [12:33am] : 2012-08-30 : gzcat weird.23:00:00-00:00:00.log.gz | awk '{print $7}' | sort | uniq -c | sort -rn | head -10 > 614589 data_before_established > 585445 possible_split_routing I'm a little curious about these two. Normally lots of these lines indicates that something is wrong with how Bro is collecting packets. I'm interested to find out if these go away when you adapt the snap length. Is the MTU of your network actually 9600 or did you just set that MTU for the interface to the maximum it would allow? > [worker-1] > type=worker > host=z.z.z.A > interface=eth5 > lb_procs=10 > lb_method=pf_ring Nice, I don't think that many people are using the load balancing feature of BroControl yet since I don't think we have it documented. Daniel, did that end up getting documented anywhere? > Yeah, I've used those recommendations from the start with one exception; the Intel X520-DA2 cards I'm using do not support disabling "ufo" (UDP large send offload). I would think that's fine. Thanks for all of the debugging information, it's really helpful. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From will.havlovick at zenimax.com Tue Sep 11 07:56:47 2012 From: will.havlovick at zenimax.com (Will Havlovick) Date: Tue, 11 Sep 2012 14:56:47 +0000 Subject: [Bro] Internship Message-ID: <6D84EF2B987E124F9BD63225968320A803E7FA@usrkvexchmbx01.zenimax.com> Hi all, My apologies to send an non-technical email to the list. I am having a difficult time finding interns for a fall/winter internship here. We are currently looking to hire an intern or interns here at ZeniMax Inc. The position is in Rockville MD (about 20min outside of DC). About the position: http://jobs.zenimax.com/requisitions/view/200 You will be working with Bro about 98% of the time. Full time or part time. It is a paid internship. Initially for 3 months with a possible 6 month extension. It will be working with 18+ Bro sensors located all over the world. Please email me if you or someone you know is interested. Also, if you apply directly to the link above, let me know and I will give our HR department a heads up. Thank you, Will -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120911/70a0641a/attachment.html From tritium.cat at gmail.com Tue Sep 11 13:34:16 2012 From: tritium.cat at gmail.com (Tritium Cat) Date: Tue, 11 Sep 2012 13:34:16 -0700 Subject: [Bro] Troubleshooting crashes In-Reply-To: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> References: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> Message-ID: On Tue, Sep 11, 2012 at 1:16 PM, Seth Hall wrote: > > On Sep 10, 2012, at 8:51 PM, Tritium Cat wrote: > (...cut...) > > /usr/local/3rd-party/bro/share/broctl/scripts/run-bro: line 60: 15452 > Segmentation fault nohup $mybro $@ > > Hm, looks like you aren't getting stack traces. Your OS is probably not > keeping core dumps or writing them to some OS-wide core dump directory. > Change the sysctl variable for your OS to dump core files and make sure > they're being dropped into the CWD prefixed with "core". > I'll make the change and send the info when it's available. > > bro at bc : [12:33am] : 2012-08-30 : gzcat weird.23:00:00-00:00:00.log.gz > | awk '{print $7}' | sort | uniq -c | sort -rn | head -10 > > 614589 data_before_established > > 585445 possible_split_routing > > I'm a little curious about these two. Normally lots of these lines > indicates that something is wrong with how Bro is collecting packets. I'm > interested to find out if these go away when you adapt the snap length. Is > the MTU of your network actually 9600 or did you just set that MTU for the > interface to the maximum it would allow? The MTU in use is actually 9600. I made the snaplen change you recommended but I'm still seeing more or less the same results in weird.log. I checked the source to see what "possible_split_routing" represents and that leads me to think our tap/aggregation setup may be incorrectly load balancing traffic; I'll get back to you on that later. 417369 possible_split_routing 353346 data_before_established 258014 window_recision 121325 active_connection_reuse 91851 connection_originator_SYN_ack 48796 Teredo_bubble_with_payload 48324 bad_SYN_ack 44451 inappropriate_FIN 44451 above_hole_data_without_any_acks 31832 SYN_seq_jump -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120911/d23c37c0/attachment.html From tritium.cat at gmail.com Tue Sep 11 17:38:12 2012 From: tritium.cat at gmail.com (Tritium Cat) Date: Tue, 11 Sep 2012 17:38:12 -0700 Subject: [Bro] Non-interactive cron task never finishes Message-ID: After enabling the broctl crontab I've noticed the cron task never finishes. As time passes this leaves many "broctl cron" tasks running and prevents other broctl administrative tasks due to lock file contention. I'm guessing they are all waiting for the first task to complete and release the lock file. If I run broctl cron interactively it seems to work fine. TIA. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120911/ba44f2d6/attachment.html From tritium.cat at gmail.com Thu Sep 13 07:28:34 2012 From: tritium.cat at gmail.com (Tritium Cat) Date: Thu, 13 Sep 2012 14:28:34 +0000 Subject: [Bro] Troubleshooting crashes In-Reply-To: References: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> Message-ID: On Tue, Sep 11, 2012 at 8:34 PM, Tritium Cat wrote: > > On Tue, Sep 11, 2012 at 1:16 PM, Seth Hall wrote: > >> >> On Sep 10, 2012, at 8:51 PM, Tritium Cat wrote: >> > > > bro at bc : [12:33am] : 2012-08-30 : gzcat weird.23:00:00-00:00:00.log.gz >> | awk '{print $7}' | sort | uniq -c | sort -rn | head -10 >> > 614589 data_before_established >> > 585445 possible_split_routing >> >> I'm a little curious about these two. Normally lots of these lines >> indicates that something is wrong with how Bro is collecting packets. I'm >> interested to find out if these go away when you adapt the snap length. Is >> the MTU of your network actually 9600 or did you just set that MTU for the >> interface to the maximum it would allow? > > > The MTU in use is actually 9600. I made the snaplen change you > recommended but I'm still seeing more or less the same results in > weird.log. I checked the source to see what "possible_split_routing" > represents and that leads me to think our tap/aggregation setup may be > incorrectly load balancing traffic; I'll get back to you on that later. > The front-end setup is working ok. I was missing PFRINGClusterID in broctl.conf; fixing that seems to have helped with memory and cpu usage. I still have lots of entries in weird.log with split_routing being the most common. broctl's "netstats" command does not show any drops, nor does the PF_RING info from /proc/net/pf_ring/. The count of "split_routing" events is about equal across all workers so I think it's something to do with the load-balancing via PF_RING. The traffic is 802.1Q tagged so maybe pf_ring is using 6-tuple load balancing for the cluster. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120913/365ad0e6/attachment.html From seth at icir.org Thu Sep 13 07:47:41 2012 From: seth at icir.org (Seth Hall) Date: Thu, 13 Sep 2012 10:47:41 -0400 Subject: [Bro] Troubleshooting crashes In-Reply-To: References: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> Message-ID: On Sep 13, 2012, at 10:28 AM, Tritium Cat wrote: > The front-end setup is working ok. I was missing PFRINGClusterID in broctl.conf; fixing that seems to have helped with memory and cpu usage. Oh, that should have been set already. Well, I suppose it might not have been if you upgraded this installation from a previous non-pf_ring enabled installation. It may be time to revisit our decision to only set that variable when building against a pf_ring enabled libpcap since this "upgrading to pf_ring" problem exposes itself. Daniel, Jon, what do you guys think? > The count of "split_routing" events is about equal across all workers so I think it's something to do with the load-balancing via PF_RING. That sounds like the culprit. > The traffic is 802.1Q tagged so maybe pf_ring is using 6-tuple load balancing for the cluster. They made that configurable a while back for me. I would recommend trying 2-tuple or 4-tuple balancing (I don't remember their default). If you figure out how to configure it, could you let us know how so we don't have to go look it up? :) Are you loading the misc/capture-loss script too? I would recommend loading that once you get this pf_ring issue all sorted out. That should be the final (or nearly final) measurement to see if you are getting all of your traffic correctly. http://www.bro-ids.org/documentation/scripts/policy/misc/capture-loss.html Thanks! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From jsiwek at illinois.edu Thu Sep 13 09:37:31 2012 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Thu, 13 Sep 2012 16:37:31 +0000 Subject: [Bro] Troubleshooting crashes In-Reply-To: References: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> Message-ID: > >> The front-end setup is working ok. I was missing PFRINGClusterID in broctl.conf; fixing that seems to have helped with memory and cpu usage. > > Oh, that should have been set already. Well, I suppose it might not have been if you upgraded this installation from a previous non-pf_ring enabled installation. > > It may be time to revisit our decision to only set that variable when building against a pf_ring enabled libpcap since this "upgrading to pf_ring" problem exposes itself. Daniel, Jon, what do you guys think? The PF_RING user's guide has a note about recompiling existing applications in the section about libpfring and libpcap installation. As long as that happens, I think Bro should already automatically upgrade to use it because the default value for PFRINGClusterID in $prefix/lib/broctl/BroControl/options.py gets overwritten by the new installation (no change to broctl.conf is technically necessary, setting it there is probably for advanced users or maybe if the default value happens to conflict with another application). Jon From keqhe at cs.wisc.edu Thu Sep 13 15:56:26 2012 From: keqhe at cs.wisc.edu (keqhe at cs.wisc.edu) Date: Thu, 13 Sep 2012 17:56:26 -0500 Subject: [Bro] what application layer protocols could Bro-2.1 identify using its default configuration? Message-ID: Hello Everyone: I set up Bro-2.1 and DataSeries to do trace analysis. I am not sure whether Bro-2.1 can identify (using default configuration)application layer protocols such as DEC_PRC, DNS, Finger, Gnutella, FTP, HTTP, Ident, IRC, NetbiosSSN, NCP, NFS, NTP, POP3, Portmapper, PRC, RSH, Rlogin, SMB, SSH, SSL, SMTP, Telnet as specified on Bro IDS' WIKI ? Or it can only identify some of the listed protocols. Could you please help me? Thank you! From klehigh at iupui.edu Fri Sep 14 15:40:58 2012 From: klehigh at iupui.edu (Keith Lehigh) Date: Fri, 14 Sep 2012 18:40:58 -0400 Subject: [Bro] compiling bro with tcmalloc Message-ID: <20120914224058.B2DF9C0056@rijndael.uits.iupui.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 All, I'm taking a stab at using tcmalloc with Bro. I'm running FreeBSD-8.3. I'v e got tcmalloc built and installed from gperftools : $ file /usr/local/lib/libtcmalloc.so.5 /usr/local/lib/libtcmalloc.so.5: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, not stripped When I run configure, Bro says it finds GooglePerfTools, but shows "false" fo r tcmalloc : - -- Found GooglePerftools: /usr/local/lib/libtcmalloc.so ... gperftools found: true tcmalloc: false debugging: false When I compile Bro and check to see if tcmalloc is there, ldd doesn't show i t in the shared libraries. Am I misunderstanding something about how Bro is compiling in/using tcmalloc or Bro's configure output? Thanks! - - Keith -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) iEYEARECAAYFAlBTsnUACgkQW5AQrvjB4meTVACfYoZk4Q1Be7V4PGdxfUg51TJW 3kYAniZGwCRnmadFCCLene/oP4VnKalg =3PF/ -----END PGP SIGNATURE----- From jmellander at lbl.gov Fri Sep 14 16:07:03 2012 From: jmellander at lbl.gov (Jim Mellander) Date: Fri, 14 Sep 2012 16:07:03 -0700 Subject: [Bro] compiling bro with tcmalloc In-Reply-To: <20120914224058.B2DF9C0056@rijndael.uits.iupui.edu> References: <20120914224058.B2DF9C0056@rijndael.uits.iupui.edu> Message-ID: Hi Keith: Might need to do: setenv LDFLAGS -L/usr/local/lib setenv CPPFLAGS -I/usr/local/include or the moral equivalent for your shell, before running configure Hope this helps. On Fri, Sep 14, 2012 at 3:40 PM, Keith Lehigh wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > All, > I'm taking a stab at using tcmalloc with Bro. I'm running FreeBSD-8.3. I'v > e got tcmalloc built and installed from gperftools : > > $ file /usr/local/lib/libtcmalloc.so.5 > /usr/local/lib/libtcmalloc.so.5: ELF 64-bit LSB shared object, x86-64, version > 1 (FreeBSD), dynamically linked, not stripped > > When I run configure, Bro says it finds GooglePerfTools, but shows "false" fo > r tcmalloc : > > - -- Found GooglePerftools: /usr/local/lib/libtcmalloc.so > ... > gperftools found: true > tcmalloc: false > debugging: false > > When I compile Bro and check to see if tcmalloc is there, ldd doesn't show i > t in the shared libraries. > Am I misunderstanding something about how Bro is compiling in/using tcmalloc > or Bro's configure output? > > Thanks! > > - - Keith > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > > iEYEARECAAYFAlBTsnUACgkQW5AQrvjB4meTVACfYoZk4Q1Be7V4PGdxfUg51TJW > 3kYAniZGwCRnmadFCCLene/oP4VnKalg > =3PF/ > -----END PGP SIGNATURE----- > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From vallentin at icir.org Fri Sep 14 16:22:14 2012 From: vallentin at icir.org (Matthias Vallentin) Date: Fri, 14 Sep 2012 16:22:14 -0700 Subject: [Bro] compiling bro with tcmalloc In-Reply-To: <20120914224058.B2DF9C0056@rijndael.uits.iupui.edu> References: <20120914224058.B2DF9C0056@rijndael.uits.iupui.edu> Message-ID: > Am I misunderstanding something about how Bro is compiling in/using tcmalloc > or Bro's configure output? Since 2.1, you need to explicitly pass --enable-perftools for Bro to use tcmalloc. We changed this because non-Linux platforms have less reliable tcmalloc support. Matthias From klehigh at iupui.edu Fri Sep 14 16:32:31 2012 From: klehigh at iupui.edu (Keith Lehigh) Date: Fri, 14 Sep 2012 19:32:31 -0400 Subject: [Bro] compiling bro with tcmalloc In-Reply-To: <10146_1347665073_q8ENOUIv025686_CADTpMNZm-uKeLfxFzHX-RciY+0QsershdvyE60RBO_UxDHJL+g@mail.gmail.com> References: <20120914224058.B2DF9C0056@rijndael.uits.iupui.edu> <10146_1347665073_q8ENOUIv025686_CADTpMNZm-uKeLfxFzHX-RciY+0QsershdvyE60RBO_UxDHJL+g@mail.gmail.com> Message-ID: <20120914233231.EB0A6C0056@rijndael.uits.iupui.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > > Am I misunderstanding something about how Bro is compiling in/using tcmall > oc > > or Bro's configure output? > > Since 2.1, you need to explicitly pass --enable-perftools for Bro to > use tcmalloc. We changed this because non-Linux platforms have less > reliable tcmalloc support. > That did the trick. Might I ask what "less reliable tcmalloc support" entails? Anything specific or am I just in for random crashes, etc.? - - Keith -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) iEYEARECAAYFAlBTvoQACgkQW5AQrvjB4mckagCdEqb6nD7FXCono5aQvQ/EsB/c /YAAnjQqLfadje5olpjr5Bqayasc/XBY =luX/ -----END PGP SIGNATURE----- From vallentin at icir.org Fri Sep 14 21:20:22 2012 From: vallentin at icir.org (Matthias Vallentin) Date: Fri, 14 Sep 2012 21:20:22 -0700 Subject: [Bro] compiling bro with tcmalloc In-Reply-To: <20120914233231.EB0A6C0056@rijndael.uits.iupui.edu> References: <20120914224058.B2DF9C0056@rijndael.uits.iupui.edu> <10146_1347665073_q8ENOUIv025686_CADTpMNZm-uKeLfxFzHX-RciY+0QsershdvyE60RBO_UxDHJL+g@mail.gmail.com> <20120914233231.EB0A6C0056@rijndael.uits.iupui.edu> Message-ID: > Might I ask what "less reliable tcmalloc support" entails? Anything specific or am I just in for random crashes, etc.? For example, we have encountered a Darwin (10.7) setup with MacPorts where Bro just segfaults immediately as it starts, you could not even get to the usage information. Similar segfault issues apparently also exist for FreeBSD 8.2. This does not mean that tcmalloc does not work with BSD in general, just some specific configurations turn out to be complicated. Matthias From tyler.schoenke at colorado.edu Mon Sep 17 12:56:16 2012 From: tyler.schoenke at colorado.edu (Tyler T. Schoenke) Date: Mon, 17 Sep 2012 13:56:16 -0600 Subject: [Bro] Troubleshooting crashes In-Reply-To: References: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> Message-ID: <50578060.50005@colorado.edu> On 9/11/12 2:34 PM, Tritium Cat wrote: > > 614589 data_before_established > > 585445 possible_split_routing > > I'm a little curious about these two. Normally lots of these lines > indicates that something is wrong with how Bro is collecting > packets. I'm interested to find out if these go away when you adapt > the snap length. Is the MTU of your network actually 9600 or did > you just set that MTU for the interface to the maximum it would allow? > I was seeing a lot of these as well. I am mirroring two ports, hence a lot of duplicate traffic. Are you doing something similar? When I had my networking engineer turn off one of the mirrored ports, I saw a 60% reduction in data_before_established and 66% decrease in possible_split_routing. I'm comparing data between the same hour on Thursday and Friday, so some of that drop is related to a normal drop in traffic, but most is probably turning off the mirror. Tyler From tritium.cat at gmail.com Mon Sep 17 17:18:38 2012 From: tritium.cat at gmail.com (Tritium Cat) Date: Tue, 18 Sep 2012 00:18:38 +0000 Subject: [Bro] Troubleshooting crashes In-Reply-To: <50578060.50005@colorado.edu> References: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> <50578060.50005@colorado.edu> Message-ID: On Mon, Sep 17, 2012 at 7:56 PM, Tyler T. Schoenke < tyler.schoenke at colorado.edu> wrote: > > On 9/11/12 2:34 PM, Tritium Cat wrote: > > > 614589 data_before_established > > > 585445 possible_split_routing > > > > I'm a little curious about these two. Normally lots of these lines > > indicates that something is wrong with how Bro is collecting > > packets. I'm interested to find out if these go away when you adapt > > the snap length. Is the MTU of your network actually 9600 or did > > you just set that MTU for the interface to the maximum it would > allow? > > > > I was seeing a lot of these as well. I am mirroring two ports, hence a > lot of duplicate traffic. Are you doing something similar? When I had > my networking engineer turn off one of the mirrored ports, I saw a 60% > reduction in data_before_established and 66% decrease in > possible_split_routing. I'm comparing data between the same hour on > Thursday and Friday, so some of that drop is related to a normal drop in > traffic, but most is probably turning off the mirror. > > No. It was due to the traffic being 802.1Q tagged and the default hashing algorithm for PF_RING. The default hashing included the VLAN id and in this network the traffic is tagged according to peering session and direction of flow; as a result a "5-tuple" flow is really two "6-tuple" flows so the flow ends up split among different workers. I think the fix should be as easy as including another PF_RING environment variable when starting bro. The way I understand possible_split_routing means bro is missing some packets so you should check the front-end setup. grep -n -B5 possible_split src/TCP.cc 521- // We've already sent a SYN, but that 522- // hasn't roused the other end, yet we're 523- // ack'ing their data. 524- 525- if ( ! Conn()->DidWeird() ) 526: Weird("possible_split_routing"); /tc -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120918/a1897b2e/attachment.html From seth at icir.org Mon Sep 17 18:10:36 2012 From: seth at icir.org (Seth Hall) Date: Mon, 17 Sep 2012 21:10:36 -0400 Subject: [Bro] Troubleshooting crashes In-Reply-To: References: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> <50578060.50005@colorado.edu> Message-ID: <8B992C56-A78A-478B-BE47-AA14ADD3793B@icir.org> On Sep 17, 2012, at 8:18 PM, Tritium Cat wrote: > I think the fix should be as easy as including another PF_RING environment variable when starting bro. Oh! I didn't realize this was settable through an environment variable when using their libpcap wrappers. Do you happen to know the variable? That's something we definitely need to be setting, almost everyone with VLAN tagged traffic has trouble with the default PF_RING setting. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From tritium.cat at gmail.com Mon Sep 17 18:34:07 2012 From: tritium.cat at gmail.com (Tritium Cat) Date: Tue, 18 Sep 2012 01:34:07 +0000 Subject: [Bro] Troubleshooting crashes In-Reply-To: <8B992C56-A78A-478B-BE47-AA14ADD3793B@icir.org> References: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> <50578060.50005@colorado.edu> <8B992C56-A78A-478B-BE47-AA14ADD3793B@icir.org> Message-ID: On Tue, Sep 18, 2012 at 1:10 AM, Seth Hall wrote: > > On Sep 17, 2012, at 8:18 PM, Tritium Cat wrote: > > > I think the fix should be as easy as including another PF_RING > environment variable when starting bro. > > Oh! I didn't realize this was settable through an environment variable > when using their libpcap wrappers. Do you happen to know the variable? > That's something we definitely need to be setting, almost everyone with > VLAN tagged traffic has trouble with the default PF_RING setting. I checked and bro already has the right env variables. >From lib/broctl/plugins/lb_pf_ring.py: 23: if BroControl.config.Config.pfringclusterid != "0": 24: nn.env_vars += ["PCAP_PF_RING_USE_CLUSTER_PER_FLOW=1"] 25: nn.env_vars += ["PCAP_PF_RING_CLUSTER_ID=%s" % BroControl.config.Config.pfringclusterid] The problem is the changes this triggers inside PF_RING does not work as expected, so I'm still working to prove that to the developer and find out why. For the time being I'm using a slight mod to pf_ring as a workaround. /tc -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120918/f9659313/attachment.html From seth at icir.org Mon Sep 17 18:51:36 2012 From: seth at icir.org (Seth Hall) Date: Mon, 17 Sep 2012 21:51:36 -0400 Subject: [Bro] Troubleshooting crashes In-Reply-To: References: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> <50578060.50005@colorado.edu> <8B992C56-A78A-478B-BE47-AA14ADD3793B@icir.org> Message-ID: On Sep 17, 2012, at 9:34 PM, Tritium Cat wrote: > The problem is the changes this triggers inside PF_RING does not work as expected, so I'm still working to prove that to the developer and find out why. For the time being I'm using a slight mod to pf_ring as a workaround. There should be some other PF_RING configuration option for setting the tuple's to use for load balancing too. It's a relatively new feature, I'll look into the variable soon. You shouldn't have to make a modification to pf_ring. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From abhishek.lists at gmail.com Mon Sep 17 19:04:04 2012 From: abhishek.lists at gmail.com (Abhishek Chanda) Date: Mon, 17 Sep 2012 19:04:04 -0700 Subject: [Bro] Trying to extract HTTP payload In-Reply-To: References: Message-ID: Hi, I am trying to extract HTTP payload and bro throws an error: achanda at achanda-OptiPlex-780:~/bro/scripts$ bro -i eth0 http-reply error in ./site, line 1: read failed with "Is a directory" achanda at achanda-OptiPlex-780:~/bro/scripts$ bro -i eth0 contents error in ./site, line 1: read failed with "Is a directory" achanda at achanda-OptiPlex-780:~/bro/scripts$ I tried to run bro from the top level installation directory but that failed since it could not find the scripts. What am I missing? Thanks From seth at icir.org Mon Sep 17 19:12:53 2012 From: seth at icir.org (Seth Hall) Date: Mon, 17 Sep 2012 22:12:53 -0400 Subject: [Bro] Trying to extract HTTP payload In-Reply-To: References: Message-ID: <89A74C7D-701E-45FC-B859-04C29792F1FD@icir.org> On Sep 17, 2012, at 10:04 PM, Abhishek Chanda wrote: > achanda at achanda-OptiPlex-780:~/bro/scripts$ bro -i eth0 http-reply > error in ./site, line 1: read failed with "Is a directory" > achanda at achanda-OptiPlex-780:~/bro/scripts$ bro -i eth0 contents > error in ./site, line 1: read failed with "Is a directory" > achanda at achanda-OptiPlex-780:~/bro/scripts$ What version of Bro are you running? There is not http-reply script anymore (it was removed in 2.0). 2.0 and 2.1 can extract payloads in several ways. There is currently only one mechanism builtin for doing it though by matching the sniffed mime type of the response body. This will do it if you are just interested in running from the command line... bro -r somepackets.pcap "HTTP::extract_file_types=/.*/" .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From tritium.cat at gmail.com Mon Sep 17 23:52:24 2012 From: tritium.cat at gmail.com (Tritium Cat) Date: Tue, 18 Sep 2012 06:52:24 +0000 Subject: [Bro] Troubleshooting crashes In-Reply-To: References: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> <50578060.50005@colorado.edu> <8B992C56-A78A-478B-BE47-AA14ADD3793B@icir.org> Message-ID: On Tue, Sep 18, 2012 at 1:51 AM, Seth Hall wrote: > > On Sep 17, 2012, at 9:34 PM, Tritium Cat wrote: > > > The problem is the changes this triggers inside PF_RING does not work as > expected, so I'm still working to prove that to the developer and find out > why. For the time being I'm using a slight mod to pf_ring as a workaround. > > There should be some other PF_RING configuration option for setting the > tuple's to use for load balancing too. It's a relatively new feature, I'll > look into the variable soon. You shouldn't have to make a modification to > pf_ring. Yeah I know, and I've read the PF_RING source for the changes you're referring to. You can select 2-tuple, 4-tuple, 5-tuple-tcp only with 2-tuple for all other traffic, regular 5-tuple, or the default which is 6-tuple (if the vlan_id is present, otherwise it is essentially 5-tuple). The point I'm trying to make is the knobs within PF_RING that control that behavior are not working properly when 802.1Q tags are involved. My custom patch just prevents the parsed vlan from being assigned. All other suggestions, including the recommended approach, do not work; bro spews split_routing alarms for everything except my patched version. /tc -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120918/bc35c510/attachment.html From abhishek.lists at gmail.com Tue Sep 18 10:08:51 2012 From: abhishek.lists at gmail.com (Abhishek Chanda) Date: Tue, 18 Sep 2012 10:08:51 -0700 Subject: [Bro] Trying to extract HTTP payload In-Reply-To: <89A74C7D-701E-45FC-B859-04C29792F1FD@icir.org> References: <89A74C7D-701E-45FC-B859-04C29792F1FD@icir.org> Message-ID: Hi, Thanks for the reply. This is bro 2.1. Now, I ran this: sudo ./bro -i eth0 "HTTP::extract_file_types=/.jpg/" But no file gets saved in the current directory. The entry appears in http.log though with a 200 OK 1347988043.663837 SWYFHjGx0x6 192.168.10.185 58146 74.200.247.186 80 0 - - - - - 0 7240 200 OK - - - (empty) - - - image/jpeg - - 1347988052.178112 BVcSiCSyzA4 192.168.10.185 46424 54.240.160.141 80 0 - - - - - 0 31225 200 OK - - - (empty) - - - image/jpeg - - #close 2012-09-18-10-07-40 Is there something else I need to do? Thanks On Mon, Sep 17, 2012 at 7:12 PM, Seth Hall wrote: > > On Sep 17, 2012, at 10:04 PM, Abhishek Chanda wrote: > >> achanda at achanda-OptiPlex-780:~/bro/scripts$ bro -i eth0 http-reply >> error in ./site, line 1: read failed with "Is a directory" >> achanda at achanda-OptiPlex-780:~/bro/scripts$ bro -i eth0 contents >> error in ./site, line 1: read failed with "Is a directory" >> achanda at achanda-OptiPlex-780:~/bro/scripts$ > > > What version of Bro are you running? There is not http-reply script anymore (it was removed in 2.0). > > 2.0 and 2.1 can extract payloads in several ways. There is currently only one mechanism builtin for doing it though by matching the sniffed mime type of the response body. > > This will do it if you are just interested in running from the command line... > bro -r somepackets.pcap "HTTP::extract_file_types=/.*/" > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > From seth at icir.org Tue Sep 18 10:14:34 2012 From: seth at icir.org (Seth Hall) Date: Tue, 18 Sep 2012 13:14:34 -0400 Subject: [Bro] Trying to extract HTTP payload In-Reply-To: References: <89A74C7D-701E-45FC-B859-04C29792F1FD@icir.org> Message-ID: <548F4343-6612-4E16-B34D-E58011ED1104@icir.org> On Sep 18, 2012, at 1:08 PM, Abhishek Chanda wrote: > sudo ./bro -i eth0 "HTTP::extract_file_types=/.jpg/" sudo ./bro -i eth0 "HTTP::extract_file_types=/.*\.jpg/" .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From abhishek.lists at gmail.com Tue Sep 18 10:23:56 2012 From: abhishek.lists at gmail.com (Abhishek Chanda) Date: Tue, 18 Sep 2012 10:23:56 -0700 Subject: [Bro] Trying to extract HTTP payload In-Reply-To: <548F4343-6612-4E16-B34D-E58011ED1104@icir.org> References: <89A74C7D-701E-45FC-B859-04C29792F1FD@icir.org> <548F4343-6612-4E16-B34D-E58011ED1104@icir.org> Message-ID: Hi, It still does not seem to work, there is nothing in the current directory. Here is an entry from http.log 1347988766.291078 t3VZX9hEzl7 192.168.10.185 48299 184.172.154.91 80 0 - - - - - 0 1131 200 OK - - - (empty) - - - image/jpeg - - There are similar entries which do not have a file name. Thanks On Tue, Sep 18, 2012 at 10:14 AM, Seth Hall wrote: > > On Sep 18, 2012, at 1:08 PM, Abhishek Chanda wrote: > >> sudo ./bro -i eth0 "HTTP::extract_file_types=/.jpg/" > > > sudo ./bro -i eth0 "HTTP::extract_file_types=/.*\.jpg/" > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > From seth at icir.org Tue Sep 18 10:40:48 2012 From: seth at icir.org (Seth Hall) Date: Tue, 18 Sep 2012 13:40:48 -0400 Subject: [Bro] Trying to extract HTTP payload In-Reply-To: References: <89A74C7D-701E-45FC-B859-04C29792F1FD@icir.org> <548F4343-6612-4E16-B34D-E58011ED1104@icir.org> Message-ID: On Sep 18, 2012, at 1:23 PM, Abhishek Chanda wrote: >> sudo ./bro -i eth0 "HTTP::extract_file_types=/.*\.jpg/" sudo ./bro -i eth0 "HTTP::extract_file_types=/.*\.jpeg/" :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From doug.burks at gmail.com Tue Sep 18 10:43:30 2012 From: doug.burks at gmail.com (Doug Burks) Date: Tue, 18 Sep 2012 13:43:30 -0400 Subject: [Bro] Trying to extract HTTP payload In-Reply-To: References: <89A74C7D-701E-45FC-B859-04C29792F1FD@icir.org> <548F4343-6612-4E16-B34D-E58011ED1104@icir.org> Message-ID: The blank fields in http.log could be the result of checksum offloading: http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not-full.html Doug On Tue, Sep 18, 2012 at 1:23 PM, Abhishek Chanda wrote: > Hi, > > It still does not seem to work, there is nothing in the current > directory. Here is an entry from http.log > > 1347988766.291078 t3VZX9hEzl7 192.168.10.185 48299 184.172.154.91 80 0 - - - - - 0 1131 200 OK - - - (empty) - - - image/jpeg - - > > There are similar entries which do not have a file name. > > Thanks > > On Tue, Sep 18, 2012 at 10:14 AM, Seth Hall wrote: >> >> On Sep 18, 2012, at 1:08 PM, Abhishek Chanda wrote: >> >>> sudo ./bro -i eth0 "HTTP::extract_file_types=/.jpg/" >> >> >> sudo ./bro -i eth0 "HTTP::extract_file_types=/.*\.jpg/" >> >> .Seth >> >> -- >> Seth Hall >> International Computer Science Institute >> (Bro) because everyone has a network >> http://www.bro-ids.org/ >> > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Doug Burks http://securityonion.blogspot.com From seth at icir.org Tue Sep 18 10:53:53 2012 From: seth at icir.org (Seth Hall) Date: Tue, 18 Sep 2012 13:53:53 -0400 Subject: [Bro] Trying to extract HTTP payload In-Reply-To: References: <89A74C7D-701E-45FC-B859-04C29792F1FD@icir.org> <548F4343-6612-4E16-B34D-E58011ED1104@icir.org> Message-ID: On Sep 18, 2012, at 1:43 PM, Doug Burks wrote: > The blank fields in http.log could be the result of checksum offloading: > http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not-full.html > Doug Hah! Good catch Doug. Ironically, the file extraction as he's doing it will still work fine. Abhishek, you can have Bro ignore checksums with the -C command line argument, but you definitely do not want to run Bro in production with that argument because it opens the door to easy evasions. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From abhishek.lists at gmail.com Tue Sep 18 11:27:40 2012 From: abhishek.lists at gmail.com (Abhishek Chanda) Date: Tue, 18 Sep 2012 11:27:40 -0700 Subject: [Bro] Trying to extract HTTP payload In-Reply-To: References: <89A74C7D-701E-45FC-B859-04C29792F1FD@icir.org> <548F4343-6612-4E16-B34D-E58011ED1104@icir.org> Message-ID: Hi Seth and Doug, Thanks for the replies. I still could not get Bro to work though. I am trying to save a gif file since I thought this would cause less confusion with the file MIME and extension. I disabled TCP checksum offloading as Doug suggested. I ran Bro as: sudo ./bro -C -i eth1 "HTTP::extract_file_types=/.*\.gif/" I then pointed my browser to a gif image. The entry for the image appears in http.log but the image does not get saved. I am sure that the interface is correct. What else can go wrong? Thanks On Tue, Sep 18, 2012 at 10:53 AM, Seth Hall wrote: > > On Sep 18, 2012, at 1:43 PM, Doug Burks wrote: > >> The blank fields in http.log could be the result of checksum offloading: >> http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not-full.html >> Doug > > > Hah! Good catch Doug. Ironically, the file extraction as he's doing it will still work fine. > > Abhishek, you can have Bro ignore checksums with the -C command line argument, but you definitely do not want to run Bro in production with that argument because it opens the door to easy evasions. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > From seth at icir.org Tue Sep 18 11:32:24 2012 From: seth at icir.org (Seth Hall) Date: Tue, 18 Sep 2012 14:32:24 -0400 Subject: [Bro] Trying to extract HTTP payload In-Reply-To: References: <89A74C7D-701E-45FC-B859-04C29792F1FD@icir.org> <548F4343-6612-4E16-B34D-E58011ED1104@icir.org> Message-ID: On Sep 18, 2012, at 2:27 PM, Abhishek Chanda wrote: > I then pointed my browser to a gif image. The entry for the image > appears in http.log but the image does not get saved. I am sure that > the interface is correct. What else can go wrong? What's the line in http.log? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From abhishek.lists at gmail.com Tue Sep 18 11:37:02 2012 From: abhishek.lists at gmail.com (Abhishek Chanda) Date: Tue, 18 Sep 2012 11:37:02 -0700 Subject: [Bro] Trying to extract HTTP payload In-Reply-To: References: <89A74C7D-701E-45FC-B859-04C29792F1FD@icir.org> <548F4343-6612-4E16-B34D-E58011ED1104@icir.org> Message-ID: Here: 1347993371.841877 J6Gs3YxcaZ3 10.0.3.15 33554 216.92.99.29 80 1 GET www.effetech.com /images/msn2_full.gif - Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:13.0) Gecko/20100101 Firefox/13.0.1 0 47818 200 OK - - - (empty) - - - image/gif - - I cleared my browser cache before I tried to get the image. Thanks On Tue, Sep 18, 2012 at 11:32 AM, Seth Hall wrote: > > On Sep 18, 2012, at 2:27 PM, Abhishek Chanda wrote: > >> I then pointed my browser to a gif image. The entry for the image >> appears in http.log but the image does not get saved. I am sure that >> the interface is correct. What else can go wrong? > > > What's the line in http.log? > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > From hlin33 at illinois.edu Wed Sep 19 07:05:42 2012 From: hlin33 at illinois.edu (Hui Lin (Hugo) ) Date: Wed, 19 Sep 2012 09:05:42 -0500 Subject: [Bro] what application layer protocols could Bro-2.1 identify using its default configuration? In-Reply-To: <9f2a9201fb454523a60625b681974798@CITESHT1.ad.uillinois.edu> References: <9f2a9201fb454523a60625b681974798@CITESHT1.ad.uillinois.edu> Message-ID: The answer to your question can be very complex. First, Bro's application layer analyzer can be written by binpac or directly by c++ (in src, *.pac codes are the binpac scripts, so you can know what analyzers are written by binpac). Most analyzer developed in the early stage is directly written by C++, for those codes, how they are enabled, I am not quite sure. For binpac analyzer, there are three ways of enabling a analyzer, which can be found in http://www.bro-ids.org/development/dpd.html (Determining Analyzer Activation ). Even if the binpac analyzer is always enabled, it may not be working if you don't define any event handler related to this analyzer. As a result, you have to check what policies are loaded by default, which can be found in /share/bro/base under bro's installation directory (not source code directory). On Thu, Sep 13, 2012 at 5:56 PM, keqhe at cs.wisc.edu wrote: > Hello Everyone: > > I set up Bro-2.1 and DataSeries to do trace analysis. I am not sure > whether Bro-2.1 can identify (using default configuration)application > layer protocols such as DEC_PRC, DNS, Finger, Gnutella, FTP, HTTP, Ident, > IRC, NetbiosSSN, NCP, NFS, NTP, POP3, Portmapper, PRC, RSH, Rlogin, SMB, > SSH, SSL, SMTP, Telnet as specified on Bro IDS' WIKI ? Or it can only > identify some of the listed protocols. > > Could you please help me? > Thank you! > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Hui Lin PhD Candidate, Research Assistant Electrical and Computer Engineering Department University of Illinois at Urbana-Champaign -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120919/e7a40c2c/attachment.html From keqhe at cs.wisc.edu Wed Sep 19 09:03:32 2012 From: keqhe at cs.wisc.edu (keqhe at cs.wisc.edu) Date: Wed, 19 Sep 2012 11:03:32 -0500 Subject: [Bro] what application layer protocols could Bro-2.1 identify using its default configuration? In-Reply-To: References: <9f2a9201fb454523a60625b681974798@CITESHT1.ad.uillinois.edu> Message-ID: HI, Hui: Thank you very much for your information! > > On Wed, Sep 19, 2012 at 9:05 AM, Hui Lin (Hugo) > wrote: > >> The answer to your question can be very complex. >> >> First, Bro's application layer analyzer can be written by binpac or >> directly by c++ (in src, *.pac codes are the binpac scripts, so you can >> know what analyzers are written by binpac). Most analyzer developed in >> the >> early stage is directly written by C++, for those codes, how they are >> enabled, I am not quite sure. For binpac analyzer, there are three ways >> of >> enabling a analyzer, which can be found in >> http://www.bro-ids.org/development/dpd.html (Determining Analyzer >> Activation ). >> >> Even if the binpac analyzer is always enabled, it may not be working if >> you don't define any event handler related to this analyzer. As a >> result, >> you have to check what policies are loaded by default, which can be >> found >> in /share/bro/base under bro's installation directory (not source code >> directory). >> >> >> On Thu, Sep 13, 2012 at 5:56 PM, keqhe at cs.wisc.edu >> wrote: >> >>> Hello Everyone: >>> >>> I set up Bro-2.1 and DataSeries to do trace analysis. I am not sure >>> whether Bro-2.1 can identify (using default configuration)application >>> layer protocols such as DEC_PRC, DNS, Finger, Gnutella, FTP, HTTP, >>> Ident, >>> IRC, NetbiosSSN, NCP, NFS, NTP, POP3, Portmapper, PRC, RSH, Rlogin, >>> SMB, >>> SSH, SSL, SMTP, Telnet as specified on Bro IDS' WIKI ? Or it can only >>> identify some of the listed protocols. >>> >>> Could you please help me? >>> Thank you! >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >> >> >> >> -- >> Hui Lin >> PhD Candidate, Research Assistant >> Electrical and Computer Engineering Department >> University of Illinois at Urbana-Champaign >> >> > > > -- > Keqiang He > Dept. of Computer Sciences, University of Wisconsin-Madison > Madison, WI 53706 > From will.havlovick at zenimax.com Wed Sep 19 09:35:04 2012 From: will.havlovick at zenimax.com (Will Havlovick) Date: Wed, 19 Sep 2012 16:35:04 +0000 Subject: [Bro] PPPoE support Message-ID: <6D84EF2B987E124F9BD63225968320A804D631@usrkvexchmbx01.zenimax.com> One of my sites is monitoring a link that has all PPPoE traffic on it. The Bro Sensor was not logging the traffic due to it being PPPoE. The bro version was 2.1 stable. I found this(thank you Seth for writing this) : https://github.com/bro-ids/bro/commit/908b1a17d1b08a8473695316e56eb98f7b005cbd and added it to the PktSrc.cc of the Bro2.1 Stable release source. Then recompiled and now my sensor seems to be logging the traffic properly. This is probably not the best way to do things. My question is: Will PPPoE support be in 2.2? Will -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120919/ea16ecd0/attachment.html From seth at icir.org Wed Sep 19 12:19:56 2012 From: seth at icir.org (Seth Hall) Date: Wed, 19 Sep 2012 15:19:56 -0400 Subject: [Bro] Troubleshooting crashes In-Reply-To: References: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> <50578060.50005@colorado.edu> <8B992C56-A78A-478B-BE47-AA14ADD3793B@icir.org> Message-ID: <102DE684-DA06-42E3-BBC4-05BD5DC4CD9B@icir.org> On Sep 18, 2012, at 2:52 AM, Tritium Cat wrote: > Yeah I know, and I've read the PF_RING source for the changes you're referring to. You can select 2-tuple, 4-tuple, 5-tuple-tcp only with 2-tuple for all other traffic, regular 5-tuple, or the default which is 6-tuple (if the vlan_id is present, otherwise it is essentially 5-tuple). Ok, I found the problem. We got tripped up when they added the configuration option and we may have to do a 2.1.1 release. We were setting the PCAP_PF_RING_USE_CLUSTER_PER_FLOW env var to "1" just to have it set to something. They started using that value to set how the load balancing is done and "1" was chosen to be round-robin. We probably need to make that configurable (from broctl.cfg) with a default of 2 (2-tuple? it causes less trouble). Here's the structure that defines the different load balancing approaches with that variable? typedef enum { cluster_per_flow = 0, /* 6-tuple: */ cluster_round_robin, cluster_per_flow_2_tuple, /* 2-tuple: */ cluster_per_flow_4_tuple, /* 4-tuple: */ cluster_per_flow_5_tuple, /* 5-tuple: */ } cluster_type; Daniel could you make this change? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From tritium.cat at gmail.com Wed Sep 19 14:19:38 2012 From: tritium.cat at gmail.com (Tritium Cat) Date: Wed, 19 Sep 2012 21:19:38 +0000 Subject: [Bro] Troubleshooting crashes In-Reply-To: <102DE684-DA06-42E3-BBC4-05BD5DC4CD9B@icir.org> References: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> <50578060.50005@colorado.edu> <8B992C56-A78A-478B-BE47-AA14ADD3793B@icir.org> <102DE684-DA06-42E3-BBC4-05BD5DC4CD9B@icir.org> Message-ID: On Wed, Sep 19, 2012 at 7:19 PM, Seth Hall wrote: > > Ok, I found the problem. We got tripped up when they added the > configuration option and we may have to do a 2.1.1 release. We were > setting the PCAP_PF_RING_USE_CLUSTER_PER_FLOW env var to "1" just to have > it set to something. They started using that value to set how the load > balancing is done and "1" was chosen to be round-robin. > > We probably need to make that configurable (from broctl.cfg) with a > default of 2 (2-tuple? it causes less trouble). > > Here's the structure that defines the different load balancing approaches > with that variable? > > typedef enum { > cluster_per_flow = 0, /* 6-tuple: port, proto, vlan> */ > cluster_round_robin, > cluster_per_flow_2_tuple, /* 2-tuple: > */ > cluster_per_flow_4_tuple, /* 4-tuple: port > */ > cluster_per_flow_5_tuple, /* 5-tuple: port, proto > */ > } cluster_type; > Ok that's good to hear, however I still think the problem is with PF_RING. I'm not sure your assessment of that environment variable is correct. See the last three emails to the pf_ring developers here: [1]. >From pf_ring libpcap.... pcap-linux.c.orig:1183: if(getenv("PCAP_PF_RING_USE_CLUSTER_PER_FLOW")) pcap-linux.c.orig-1184- pfring_set_cluster(handle->ring, atoi(clusterId), cluster_per_flow); pcap-linux.c.orig-1185- else pcap-linux.c.orig-1186- pfring_set_cluster(handle->ring, atoi(clusterId), cluster_round_robin); Notice the call to pfring_set_cluster. "cluster_per_flow" is the default 5-tuple hashing mode. (Actually 6-tuple due to the vlan problem). Below I've included the relevant code from pf_ring that shows the only difference between cluster_per_flow and cluster_per_flow_5_tuple is the call to hash_pkt_header(). "hash_pkt_header" is called from "hash_pkt_cluster" and "default_rehash_rss_func". Inside kernel/pf_ring.c, the function "hash_pkt_cluster" will determine the hashing mode and call another function "hash_pkt_header". The last argument to "hash_pkt_header" instructs PF_RING to mask/prevent the VLAN from being used in the hash calculation. It looks like this --> idx = hash_pkt_header(hdr, 0, 0, 0, 0, 1); Notice every call to hash_pkt_header() except the default case of "cluster_per_flow" attempts to mask the vlan. (Up until a few days ago, all calls to hash_pkt_header() did not mask the vlan). I patched PF_RING in various places to force hash_pkt_header() to always mask the vlan.. regardless of what environment variable was set. This did not work so I took it a step further. Of course it's always possible I'm highly confused, time will tell :o /tc 3889:static u_int hash_pkt_cluster(ring_cluster_element *cluster_ptr, 3890- struct pfring_pkthdr *hdr) 3891-{ 3892- u_int idx; 3893- 3894- switch(cluster_ptr->cluster.hashing_mode) { 3895- case cluster_round_robin: 3896- idx = cluster_ptr->cluster.hashing_id++; 3897- break; 3898- 3899- case cluster_per_flow_2_tuple: 3900- idx = hash_pkt_header(hdr, 0, 0, 1, 1, 1); 3901- break; 3902- 3903- case cluster_per_flow_4_tuple: 3904- idx = hash_pkt_header(hdr, 0, 0, 0, 1, 1); 3905- break; 3906- 3907- case cluster_per_flow_tcp_5_tuple: 3908- if(((hdr->extended_hdr.parsed_pkt.tunnel.tunnel_id == NO_TUNNEL_ID) ? 3909- hdr->extended_hdr.parsed_pkt.l3_proto : hdr->extended_hdr.parsed_pkt.tunnel.tunneled_proto) == IPPROTO_TCP) 3910- idx = hash_pkt_header(hdr, 0, 0, 0, 0, 1); /* 5 tuple */ 3911- else 3912- idx = hash_pkt_header(hdr, 0, 0, 1, 1, 1); /* 2 tuple */ 3913- break; 3914- 3915- case cluster_per_flow_5_tuple: 3916- idx = hash_pkt_header(hdr, 0, 0, 0, 0, 1); 3917- break; 3918- 3919- case cluster_per_flow: 3920- default: 3921- idx = hash_pkt_header(hdr, 0, 0, 0, 0, 0); 3922- break; 3923- } 3924- 3925- return(idx % cluster_ptr->cluster.num_cluster_elements); 3926-} -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120919/cc80ccb0/attachment.html From tritium.cat at gmail.com Wed Sep 19 14:21:07 2012 From: tritium.cat at gmail.com (Tritium Cat) Date: Wed, 19 Sep 2012 21:21:07 +0000 Subject: [Bro] Troubleshooting crashes In-Reply-To: References: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> <50578060.50005@colorado.edu> <8B992C56-A78A-478B-BE47-AA14ADD3793B@icir.org> <102DE684-DA06-42E3-BBC4-05BD5DC4CD9B@icir.org> Message-ID: On Wed, Sep 19, 2012 at 9:19 PM, Tritium Cat wrote: (...) > Ok that's good to hear, however I still think the problem is with PF_RING. > I'm not sure your assessment of that environment variable is correct. See > the last three emails to the pf_ring developers here: [1]. > I forgot to include the link for [1]. http://www.mail-archive.com/ntop-misc at listgateway.unipi.it/msg02944.html /tc -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120919/52a04193/attachment.html From seth at icir.org Thu Sep 20 04:30:36 2012 From: seth at icir.org (Seth Hall) Date: Thu, 20 Sep 2012 07:30:36 -0400 Subject: [Bro] Troubleshooting crashes In-Reply-To: References: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> <50578060.50005@colorado.edu> <8B992C56-A78A-478B-BE47-AA14ADD3793B@icir.org> <102DE684-DA06-42E3-BBC4-05BD5DC4CD9B@icir.org> Message-ID: On Sep 19, 2012, at 5:19 PM, Tritium Cat wrote: > Ok that's good to hear, however I still think the problem is with PF_RING. I'm not sure your assessment of that environment variable is correct. See the last three emails to the pf_ring developers here: [1]. Arg! You're right, I didn't read through that code closely enough. I guess we're stuck in a position still where can't configure that since they haven't exposed it through their libpcap wrappers yet. I'll try and bring that up with them soon. Daniel, you can either just delete that branch or wait until we have more configurability and make the correct modification then. Thanks, .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Thu Sep 20 13:54:46 2012 From: seth at icir.org (Seth Hall) Date: Thu, 20 Sep 2012 16:54:46 -0400 Subject: [Bro] URI decoding Message-ID: <5374BABB-6ED5-4EC3-A72D-D36A841D9CBB@icir.org> We received a question privately about the HTTP logs and if there was a setting to stop URL decoding the "uri" field. It turns out there isn't a setting for this, but the base scripts have been designed in a way that makes this very easy to do. Here's the script to do it in case anyone else is interested? event http_request(c: connection, method: string, original_URI: string, unescaped_URI: string, version: string) { c$http$uri = original_URI; } What it's doing is overwriting the c$http$uri field with the original_URI value instead of the unescaped_URI value which the base script uses. It ends up being overwritten because the http_request handler in the base HTTP scripts is handled at a higher priority and is executed first, that way you are assured that your handler with no explicit priority (priority zero) will be executed second. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From tritium.cat at gmail.com Tue Sep 25 04:03:19 2012 From: tritium.cat at gmail.com (Tritium Cat) Date: Tue, 25 Sep 2012 11:03:19 +0000 Subject: [Bro] Troubleshooting crashes In-Reply-To: References: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> <50578060.50005@colorado.edu> <8B992C56-A78A-478B-BE47-AA14ADD3793B@icir.org> <102DE684-DA06-42E3-BBC4-05BD5DC4CD9B@icir.org> Message-ID: On Thu, Sep 20, 2012 at 11:30 AM, Seth Hall wrote: > > Arg! You're right, I didn't read through that code closely enough. I > guess we're stuck in a position still where can't configure that since they > haven't exposed it through their libpcap wrappers yet. I'll try and bring > that up with them soon. > > Daniel, you can either just delete that branch or wait until we have more > configurability and make the correct modification then. > The PF_RING issue has been fixed and the API now allows selecting cluster mode via API (with environment variables). http://www.mail-archive.com/ntop-misc at listgateway.unipi.it/msg02972.html I made the following patch for Bro. To set the cluster mode from broctl.cnf, set "PFRINGClusterType" to one of the following: - "2tuple" - "4tuple" - "5tupletcp" - "5tuple" If PFRINGClusterType is not defined then the default will be "cluster_per_flow", which is essentially 5-tuple unless the 802.1Q header is present. /tc --- ../bro-2012-09-22/aux/broctl/BroControl/plugins/lb_pf_ring.py 2012-09-22 08:43:01.000000000 +0000 +++ aux/broctl/BroControl/plugins/lb_pf_ring.py 2012-09-25 10:48:54.000000000 +0000 @@ -21,5 +21,19 @@ if nn.lb_method == "pf_ring": if BroControl.config.Config.pfringclusterid != "0": - nn.env_vars += ["PCAP_PF_RING_USE_CLUSTER_PER_FLOW=1"] - nn.env_vars += ["PCAP_PF_RING_CLUSTER_ID=%s" % BroControl.config.Config.pfringclusterid] + if BroControl.config.Config.pfringclustertype == "2tuple": + nn.env_vars += ["PCAP_PF_RING_USE_CLUSTER_PER_FLOW_2_TUPLE=1"] + nn.env_vars += ["PCAP_PF_RING_CLUSTER_ID=%s" % BroControl.config.Config.pfringclusterid] + elif BroControl.config.Config.pfringclustertype == "4tuple": + nn.env_vars += ["PCAP_PF_RING_USE_CLUSTER_PER_FLOW_4_TUPLE=1"] + nn.env_vars += ["PCAP_PF_RING_CLUSTER_ID=%s" % BroControl.config.Config.pfringclusterid] + elif BroControl.config.Config.pfringclustertype == "5tupletcp": + nn.env_vars += ["PCAP_PF_RING_USE_CLUSTER_PER_FLOW_TCP_5_TUPLE=1"] + nn.env_vars += ["PCAP_PF_RING_CLUSTER_ID=%s" % BroControl.config.Config.pfringclusterid] + elif BroControl.config.Config.pfringclustertype == "5tuple": + nn.env_vars += ["PCAP_PF_RING_USE_CLUSTER_PER_FLOW_5_TUPLE=1"] + nn.env_vars += ["PCAP_PF_RING_CLUSTER_ID=%s" % BroControl.config.Config.pfringclusterid] + else: + nn.env_vars += ["PCAP_PF_RING_USE_CLUSTER_PER_FLOW=1"] + nn.env_vars += ["PCAP_PF_RING_CLUSTER_ID=%s" % BroControl.config.Config.pfringclusterid] + -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120925/c654c948/attachment.html From kkamin at 21ct.com Tue Sep 25 06:57:28 2012 From: kkamin at 21ct.com (Karl Kamin) Date: Tue, 25 Sep 2012 08:57:28 -0500 Subject: [Bro] Cookies Message-ID: <04920BD67C651C469D0387704CD7692A7420A0F8FA@21ct-exg07.21technologies.com> Has anyone extended bro to retrieve the cookie name/value/attributes ? I have added the var-extraction-cookies.bro and see names in the bro logs, but would like to capture the value/attributes. Karl -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120925/be3a638a/attachment.html From vallentin at icir.org Tue Sep 25 08:45:43 2012 From: vallentin at icir.org (Matthias Vallentin) Date: Tue, 25 Sep 2012 08:45:43 -0700 Subject: [Bro] Cookies In-Reply-To: <04920BD67C651C469D0387704CD7692A7420A0F8FA@21ct-exg07.21technologies.com> References: <04920BD67C651C469D0387704CD7692A7420A0F8FA@21ct-exg07.21technologies.com> Message-ID: > Has anyone extended bro to retrieve the cookie name/value/attributes ? Our contributed scripts repository [1] contains a script sidejack.bro, which extracts cookies as you describe it. Essentially, you have to perform two nested splits: one to get the key-value pairs and one to separate keys from values. Matthias [1] http://git.bro-ids.org/bro-scripts.git/tree From seth at icir.org Tue Sep 25 08:55:07 2012 From: seth at icir.org (Seth Hall) Date: Tue, 25 Sep 2012 11:55:07 -0400 Subject: [Bro] Troubleshooting crashes In-Reply-To: References: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> <50578060.50005@colorado.edu> <8B992C56-A78A-478B-BE47-AA14ADD3793B@icir.org> <102DE684-DA06-42E3-BBC4-05BD5DC4CD9B@icir.org> Message-ID: <024CAAF4-676D-41C9-8432-2705E978C81F@icir.org> On Sep 25, 2012, at 7:03 AM, Tritium Cat wrote: > The PF_RING issue has been fixed and the API now allows selecting cluster mode via API (with environment variables). Nice! > If PFRINGClusterType is not defined then the default will be "cluster_per_flow", which is essentially 5-tuple unless the 802.1Q header is present. Cool! Daniel, can you merge this into your branch where you had fixed this previously (when I was wrong about how to fix it)? Thanks, .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From doug.burks at gmail.com Tue Sep 25 10:56:56 2012 From: doug.burks at gmail.com (Doug Burks) Date: Tue, 25 Sep 2012 13:56:56 -0400 Subject: [Bro] Troubleshooting crashes In-Reply-To: <024CAAF4-676D-41C9-8432-2705E978C81F@icir.org> References: <247BDA91-E4D0-4C0A-8A4B-51FE14A145A9@icir.org> <50578060.50005@colorado.edu> <8B992C56-A78A-478B-BE47-AA14ADD3793B@icir.org> <102DE684-DA06-42E3-BBC4-05BD5DC4CD9B@icir.org> <024CAAF4-676D-41C9-8432-2705E978C81F@icir.org> Message-ID: Great job, Tritium Cat and all! Will there be a 2.1.1 release for this? Thanks, Doug On Tue, Sep 25, 2012 at 11:55 AM, Seth Hall wrote: > > On Sep 25, 2012, at 7:03 AM, Tritium Cat wrote: > >> The PF_RING issue has been fixed and the API now allows selecting cluster mode via API (with environment variables). > > Nice! > >> If PFRINGClusterType is not defined then the default will be "cluster_per_flow", which is essentially 5-tuple unless the 802.1Q header is present. > > Cool! Daniel, can you merge this into your branch where you had fixed this previously (when I was wrong about how to fix it)? > > Thanks, > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Doug Burks http://securityonion.blogspot.com From hlin33 at illinois.edu Sun Sep 30 18:22:09 2012 From: hlin33 at illinois.edu (Hui Lin (Hugo) ) Date: Sun, 30 Sep 2012 20:22:09 -0500 Subject: [Bro] From a Mobile Security Article Message-ID: Just read a mobile security article ( http://www.itworld.com/security/298633/5-mobile-security-lessons-department-defense?page=0,2&source=ITWNLE_nlt_today_2012-09-28), it includes somethings like this "On the one hand, the DoD requires the ability to remotely wipe and disable lost devices, an example of a key centralized management capability. *On the other hand, it's also counting on its extensive user base to understand and implement mobile security policies in the field. *As a result, training and human management are central elements of the new strategy." Kind of happy to see this type of thinking. -- Hui Lin PhD Candidate, Research Assistant Electrical and Computer Engineering Department University of Illinois at Urbana-Champaign -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120930/391fcdbd/attachment.html From hlein at korelogic.com Sun Sep 30 19:34:54 2012 From: hlein at korelogic.com (Hank Leininger) Date: Sun, 30 Sep 2012 22:34:54 -0400 Subject: [Bro] Minor install issue with 2.1 Message-ID: <20121001023454.GN30612@marklar.spinoli.org> Hello, When rolling a Gentoo ebuild for 2.1, I ran into a parse error in the auto-generated build/cmake_install.cmake file, due to insufficient quoting (so empty strings disappeared from the argument list in an if() statement). Sorry if this is a known issue and/or only occurs because of some peculiarity of my build environment. The below patch, applied between compilation and installation, fixes it. But it's not the right fix--I did not trace backward to the creation of the build/cmake_install.cmake file and fix that. --- build/cmake_install.cmake.orig 2012-09-30 18:49:31.669030129 -0400 +++ build/cmake_install.cmake 2012-09-30 18:49:07.145028834 -0400 @@ -34,7 +34,7 @@ IF(NOT CMAKE_INSTALL_COMPONENT OR "${CMAKE_INSTALL_COMPONENT}" STREQUAL "Unspecified") - if (NOT STREQUAL $ENV{USER} AND NOT $ENV{USER} STREQUAL root) + if (NOT "" STREQUAL "$ENV{USER}" AND NOT "$ENV{USER}" STREQUAL root) message(STATUS "WARNING: Install is being performed by user " "'$ENV{USER}', but the build directory was configured by " "user ''. This may result in a permissions error " -- Hank Leininger D24D 2C2A F3AC B9AE CD03 B506 2D57 32E1 686B 6DB3 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 447 bytes Desc: Digital signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120930/62f65d5a/attachment.bin