From slagell at illinois.edu Tue Apr 2 12:14:48 2013 From: slagell at illinois.edu (Slagell, Adam J) Date: Tue, 2 Apr 2013 19:14:48 +0000 Subject: [Bro] A chance to play on a 100+ GB Bro playground, and call it work Message-ID: <558D23D33781EF45A69229CDAC6BF15110F84A11@CITESMBX6.ad.uillinois.edu> We are looking for some new security engineers & analysts at the NCSA [1], a part of the University of Illinois at Urbana-Champaign. This person will be on the team responsible for day-to-day security operations, performing incident response, and running some really innovative technologies. Bro experience is a highly desired skill for us. Bro is a key part of our security infrastructure, where we currently use it to monitor two dozen 10G links, with plans to add another 100G link and more. If you are interested, you can follow the link below to apply. [1] http://www.ncsa.illinois.edu/AboutUs/Employment/A1300136.html From mbower2 at gmail.com Tue Apr 2 17:19:13 2013 From: mbower2 at gmail.com (Michael Bower) Date: Tue, 02 Apr 2013 20:19:13 -0400 Subject: [Bro] Help with searching logs In-Reply-To: References: Message-ID: <515B7581.9080403@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Im still learning, so bare with me. I ran the following command: bro-cut id.orig_h orig_bytes < conn.log \ | sort \ | awk '{ if (host != $1) { \ if (size != 0) \ print $1, size; \ host=$1; \ size=0 \ } else \ size += $2 \ } \ END { \ if (size != 0) \ print $1, size \ }' \ | sort -rnk 2 \ | head -n 10 This worked well to show me the top 10 hosts (originators). What Im trying to do is show the top 10 hosts and the time (ts). Maybe show the resp_bytes field too, if that is possible. Any help would be greatly appreciated. Thanks! - -- Mike -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBCgAGBQJRW3WBAAoJEIAKCPjZh/yXUF4H/RhFuVQy6bT3Z8Z1k2oMDBGt TYFAfsyeXcnf9dOl3NFGEIlifjDMZ/gK5kBVWo/FYSHGWHrCT0+ICcsjwLroRP/E rn1StPS7ozlSiY2ZJSG0UAUCZX0HZ0ujvmNo8UvnoAR75cORq8Y08cU4XpLjqhxc d4xu3G+HnhyzjKAiG6xtqDpK2Z3bnjJzyWEqZCoYDzNqtcYnrxYjcKa0kX9rBhUr uV6upZ9OHIdf25EYCVfjDrKPSUaRhSAnTVtYE0+OQRA0OPpnG3rLWFSK2yjkTbNG AzKXfhJZ0PWmUWkeD6Bzf2TCNqfyLchNSScm2atA/dhTRBV3JhHIhwIcejXr6sk= =23Kd -----END PGP SIGNATURE----- From siliang.cao at gmail.com Tue Apr 2 18:32:44 2013 From: siliang.cao at gmail.com (siliang.cao) Date: Wed, 3 Apr 2013 09:32:44 +0800 Subject: [Bro] Mail Sending Problem for Bro Message-ID: <201304030932423458973@gmail.com> Hello, everyone! I am trying to Bro IDS as a simple IDS. Now, I want to use the MAIL Notice feature, but I am stucked here. I redefined Notice::emailed_types to add my interested Notice::Type, and installed sendmail program, which worked well through command-line usage. But it still can not work any way. Maybe I have made some wrong settings, but how can I get debug information of mail processing? Or, are there any special settings I may forget to do? Your help would be appreciated. -------------- siliang.cao From soehlert at illinois.edu Tue Apr 2 19:23:41 2013 From: soehlert at illinois.edu (Oehlert, Samuel J) Date: Wed, 3 Apr 2013 02:23:41 +0000 Subject: [Bro] Help with searching logs In-Reply-To: <515B7581.9080403@gmail.com> References: <515B7581.9080403@gmail.com> Message-ID: You cut those fields out when you did your bro-cut. The id.orig_h and orig_bytes are both fields in the log. If you want other fields, you just add those to your list. E.g.: bro-cut ts id.orig_h orig_bytes resp_bytes < conn.log ?.etc -Sam ------- Sam Oehlert (217) 300-1076 Security Engineer National Center for Supercomputing Applications On Apr 2, 2013, at 7:19 PM, Michael Bower wrote: > Signed PGP part > Im still learning, so bare with me. I ran the following command: > > bro-cut id.orig_h orig_bytes < conn.log \ > | sort \ > | awk '{ if (host != $1) { \ > if (size != 0) \ > print $1, size; \ > host=$1; \ > size=0 \ > } else \ > size += $2 \ > } \ > END { \ > if (size != 0) \ > print $1, size \ > }' \ > | sort -rnk 2 \ > | head -n 10 > > > This worked well to show me the top 10 hosts (originators). What Im > trying to do is show the top 10 hosts and the time (ts). Maybe show > the resp_bytes field too, if that is possible. Any help would be > greatly appreciated. > > Thanks! > - -- > > Mike > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130403/8da6dfb1/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130403/8da6dfb1/attachment.bin From mbower2 at gmail.com Tue Apr 2 19:40:41 2013 From: mbower2 at gmail.com (Michael Bower) Date: Tue, 02 Apr 2013 22:40:41 -0400 Subject: [Bro] Help with searching logs In-Reply-To: References: <515B7581.9080403@gmail.com> Message-ID: <515B96A9.7060502@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 I had tried that, but it only outputs the ts field. I need something else in that awk statement, just don't know what. On 4/2/13 8:43 PM, Oehlert, Samuel J wrote: > You cut those fields out when you did your bro-cut. The id.orig_h > and orig_bytes are both fields in the log. If you want other > fields, you just add those to your list. E.g.: bro-cut ts id.orig_h > orig_bytes resp_bytes < conn.log ?.etc > > -Sam ------- Sam Oehlert > (217) 300-1076 Security Engineer > National Center for Supercomputing Applications > > On Apr 2, 2013, at 7:19 PM, Michael Bower > wrote: > >> Signed PGP part Im still learning, so bare with me. I ran the >> following command: >> >> bro-cut id.orig_h orig_bytes < conn.log \ | sort >> \ | awk '{ if (host != $1) { \ if (size != >> 0) \ print $1, size; \ >> host=$1; \ size=0 >> \ } else \ size += $2 >> \ } \ END { >> \ if (size != 0) \ print $1, size >> \ }' \ | sort -rnk 2 >> \ | head -n 10 >> >> >> This worked well to show me the top 10 hosts (originators). What >> Im trying to do is show the top 10 hosts and the time (ts). >> Maybe show the resp_bytes field too, if that is possible. Any >> help would be greatly appreciated. >> >> Thanks! - -- >> >> Mike >> >> >> >> _______________________________________________ Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > - -- Mike -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBCgAGBQJRW5apAAoJEIAKCPjZh/yXPuAH/1PataeTEIhRa3S6juEkr4m0 jmrq/YvfsvZ/Mu1bdAGXTLjihwy9KFmciRPndT1jINgkd0ixsnW2NmNqIC3zgOF9 BWYXDuq5WfedaZKORzgpj00tMv6BFrY1KSkmNeUMCLVyviahs/cdhTZKvGg6hphQ PyhiU1Fb8Zl3LPgL5Kp/72DIS3IjoJtbRl71QkdjI8eXlnZzJNYUDtRlVMpbCo7l b5qh68RbP/MsKVyUXxaRwm7b33AtJ3/ZRVMNM1wHKGDugTUkxKYoy5VrkEUK7PBt gMhtf5OvcxR1QHbzqvUQ2Xpb+8qLSVPx09bJ/6hYqVPPCROXieNhUNA5uo+ros4= =m5+H -----END PGP SIGNATURE----- From scastle at bouldercounty.org Wed Apr 3 07:31:12 2013 From: scastle at bouldercounty.org (Castle, Shane) Date: Wed, 3 Apr 2013 14:31:12 +0000 Subject: [Bro] Help with searching logs In-Reply-To: <515B7581.9080403@gmail.com> References: <515B7581.9080403@gmail.com> Message-ID: <21DD7C64179C9843B756C6DD491634DB5385146A@Mailbox1.boco.co.boulder.co.us> The script is lying to you. Here's the correct script: bro-cut id.orig_h orig_bytes < conn.log \ | sort -t '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n \ | awk 'BEGIN { size=0;host="" } \ { if (host != $1) { \ if (size != 0) \ print host, size; \ host=$1; \ if ($2 != "-") \ size=$2 \ else \ size=0 \ } else \ if ($2 != "-") \ size += $2 \ } \ END { \ if (size != 0) \ print host, size \ }' \ | sort -rnk 2 \ | head -n 10 Since this script summarizes, having timestamps in there would not be useful. And, if you want to change the logic to responder, change "orig" to "resp" in the first line. Yes, it would be nice to have the top conversations, not just the top talkers, which would combine both orig and resp, but I'm not sure the result would justify the work. A few months ago I went through this and we were hoping that the doc would have been changed to show a correct script, but it has not been, apparently. -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Michael Bower Sent: Tuesday, April 02, 2013 18:19 To: bro at bro.org Subject: [Bro] Help with searching logs -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Im still learning, so bare with me. I ran the following command: bro-cut id.orig_h orig_bytes < conn.log \ | sort \ | awk '{ if (host != $1) { \ if (size != 0) \ print $1, size; \ host=$1; \ size=0 \ } else \ size += $2 \ } \ END { \ if (size != 0) \ print $1, size \ }' \ | sort -rnk 2 \ | head -n 10 This worked well to show me the top 10 hosts (originators). What Im trying to do is show the top 10 hosts and the time (ts). Maybe show the resp_bytes field too, if that is possible. Any help would be greatly appreciated. Thanks! - -- Mike -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBCgAGBQJRW3WBAAoJEIAKCPjZh/yXUF4H/RhFuVQy6bT3Z8Z1k2oMDBGt TYFAfsyeXcnf9dOl3NFGEIlifjDMZ/gK5kBVWo/FYSHGWHrCT0+ICcsjwLroRP/E rn1StPS7ozlSiY2ZJSG0UAUCZX0HZ0ujvmNo8UvnoAR75cORq8Y08cU4XpLjqhxc d4xu3G+HnhyzjKAiG6xtqDpK2Z3bnjJzyWEqZCoYDzNqtcYnrxYjcKa0kX9rBhUr uV6upZ9OHIdf25EYCVfjDrKPSUaRhSAnTVtYE0+OQRA0OPpnG3rLWFSK2yjkTbNG AzKXfhJZ0PWmUWkeD6Bzf2TCNqfyLchNSScm2atA/dhTRBV3JhHIhwIcejXr6sk= =23Kd -----END PGP SIGNATURE----- _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From scastle at bouldercounty.org Wed Apr 3 07:58:42 2013 From: scastle at bouldercounty.org (Castle, Shane) Date: Wed, 3 Apr 2013 14:58:42 +0000 Subject: [Bro] Help with searching logs In-Reply-To: <21DD7C64179C9843B756C6DD491634DB5385146A@Mailbox1.boco.co.boulder.co.us> References: <515B7581.9080403@gmail.com> <21DD7C64179C9843B756C6DD491634DB5385146A@Mailbox1.boco.co.boulder.co.us> Message-ID: <21DD7C64179C9843B756C6DD491634DB53851519@Mailbox1.boco.co.boulder.co.us> Hm, I get a syntax error in that script now. Let me figure this out... -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Castle, Shane Sent: Wednesday, April 03, 2013 08:31 To: 'Michael Bower'; 'bro at bro.org' Subject: Re: [Bro] Help with searching logs The script is lying to you. Here's the correct script: bro-cut id.orig_h orig_bytes < conn.log \ | sort -t '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n \ | awk 'BEGIN { size=0;host="" } \ { if (host != $1) { \ if (size != 0) \ print host, size; \ host=$1; \ if ($2 != "-") \ size=$2 \ else \ size=0 \ } else \ if ($2 != "-") \ size += $2 \ } \ END { \ if (size != 0) \ print host, size \ }' \ | sort -rnk 2 \ | head -n 10 Since this script summarizes, having timestamps in there would not be useful. And, if you want to change the logic to responder, change "orig" to "resp" in the first line. Yes, it would be nice to have the top conversations, not just the top talkers, which would combine both orig and resp, but I'm not sure the result would justify the work. A few months ago I went through this and we were hoping that the doc would have been changed to show a correct script, but it has not been, apparently. -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Michael Bower Sent: Tuesday, April 02, 2013 18:19 To: bro at bro.org Subject: [Bro] Help with searching logs -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Im still learning, so bare with me. I ran the following command: bro-cut id.orig_h orig_bytes < conn.log \ | sort \ | awk '{ if (host != $1) { \ if (size != 0) \ print $1, size; \ host=$1; \ size=0 \ } else \ size += $2 \ } \ END { \ if (size != 0) \ print $1, size \ }' \ | sort -rnk 2 \ | head -n 10 This worked well to show me the top 10 hosts (originators). What Im trying to do is show the top 10 hosts and the time (ts). Maybe show the resp_bytes field too, if that is possible. Any help would be greatly appreciated. Thanks! - -- Mike -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBCgAGBQJRW3WBAAoJEIAKCPjZh/yXUF4H/RhFuVQy6bT3Z8Z1k2oMDBGt TYFAfsyeXcnf9dOl3NFGEIlifjDMZ/gK5kBVWo/FYSHGWHrCT0+ICcsjwLroRP/E rn1StPS7ozlSiY2ZJSG0UAUCZX0HZ0ujvmNo8UvnoAR75cORq8Y08cU4XpLjqhxc d4xu3G+HnhyzjKAiG6xtqDpK2Z3bnjJzyWEqZCoYDzNqtcYnrxYjcKa0kX9rBhUr uV6upZ9OHIdf25EYCVfjDrKPSUaRhSAnTVtYE0+OQRA0OPpnG3rLWFSK2yjkTbNG AzKXfhJZ0PWmUWkeD6Bzf2TCNqfyLchNSScm2atA/dhTRBV3JhHIhwIcejXr6sk= =23Kd -----END PGP SIGNATURE----- _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From scastle at bouldercounty.org Wed Apr 3 08:30:03 2013 From: scastle at bouldercounty.org (Castle, Shane) Date: Wed, 3 Apr 2013 15:30:03 +0000 Subject: [Bro] Help with searching logs In-Reply-To: <21DD7C64179C9843B756C6DD491634DB53851519@Mailbox1.boco.co.boulder.co.us> References: <515B7581.9080403@gmail.com> <21DD7C64179C9843B756C6DD491634DB5385146A@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851519@Mailbox1.boco.co.boulder.co.us> Message-ID: <21DD7C64179C9843B756C6DD491634DB5385157E@Mailbox1.boco.co.boulder.co.us> Seems as though an awk update tightened some of the syntax restrictions. This script works: bro-cut id.orig_h orig_bytes < conn.log \ | sort -t '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n \ | awk 'BEGIN { size=0;host="" } \ { if (host != $1) { \ if (size != 0) \ print host, size; \ host=$1; \ if ($2 != "-") { \ size=$2 } \ else { \ size=0; } \ } else { \ if ($2 != "-") \ size += $2 } \ } \ END { \ if (size != 0) \ print host, size \ }' \ | sort -rnk 2 \ | head -n 10 The conversations could be done by a script that takes conn.log as input, merges orig_h and resp_h into one field while adding together their bytes, sorting on that, then putting the result through the same awk script. The output might look something like this: 1.2.3.4|5.6.7.8 123456789 ... depending on what you choose for your conjoining character. Also, note that if there are multiple conversations between two IP addresses then you will be adding up all those conversations and presenting the result as one line - that might not be exactly what you are looking for. Welcome to the world of IDS data mining. Sharpen your awk/sort/sql/perl/bash skills - they will come in very handy. -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Castle, Shane Sent: Wednesday, April 03, 2013 08:59 To: 'Michael Bower'; 'bro at bro.org' Subject: Re: [Bro] Help with searching logs Hm, I get a syntax error in that script now. Let me figure this out... -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Castle, Shane Sent: Wednesday, April 03, 2013 08:31 To: 'Michael Bower'; 'bro at bro.org' Subject: Re: [Bro] Help with searching logs The script is lying to you. Here's the correct script: bro-cut id.orig_h orig_bytes < conn.log \ | sort -t '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n \ | awk 'BEGIN { size=0;host="" } \ { if (host != $1) { \ if (size != 0) \ print host, size; \ host=$1; \ if ($2 != "-") \ size=$2 \ else \ size=0 \ } else \ if ($2 != "-") \ size += $2 \ } \ END { \ if (size != 0) \ print host, size \ }' \ | sort -rnk 2 \ | head -n 10 Since this script summarizes, having timestamps in there would not be useful. And, if you want to change the logic to responder, change "orig" to "resp" in the first line. Yes, it would be nice to have the top conversations, not just the top talkers, which would combine both orig and resp, but I'm not sure the result would justify the work. A few months ago I went through this and we were hoping that the doc would have been changed to show a correct script, but it has not been, apparently. -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Michael Bower Sent: Tuesday, April 02, 2013 18:19 To: bro at bro.org Subject: [Bro] Help with searching logs -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Im still learning, so bare with me. I ran the following command: bro-cut id.orig_h orig_bytes < conn.log \ | sort \ | awk '{ if (host != $1) { \ if (size != 0) \ print $1, size; \ host=$1; \ size=0 \ } else \ size += $2 \ } \ END { \ if (size != 0) \ print $1, size \ }' \ | sort -rnk 2 \ | head -n 10 This worked well to show me the top 10 hosts (originators). What Im trying to do is show the top 10 hosts and the time (ts). Maybe show the resp_bytes field too, if that is possible. Any help would be greatly appreciated. Thanks! - -- Mike -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBCgAGBQJRW3WBAAoJEIAKCPjZh/yXUF4H/RhFuVQy6bT3Z8Z1k2oMDBGt TYFAfsyeXcnf9dOl3NFGEIlifjDMZ/gK5kBVWo/FYSHGWHrCT0+ICcsjwLroRP/E rn1StPS7ozlSiY2ZJSG0UAUCZX0HZ0ujvmNo8UvnoAR75cORq8Y08cU4XpLjqhxc d4xu3G+HnhyzjKAiG6xtqDpK2Z3bnjJzyWEqZCoYDzNqtcYnrxYjcKa0kX9rBhUr uV6upZ9OHIdf25EYCVfjDrKPSUaRhSAnTVtYE0+OQRA0OPpnG3rLWFSK2yjkTbNG AzKXfhJZ0PWmUWkeD6Bzf2TCNqfyLchNSScm2atA/dhTRBV3JhHIhwIcejXr6sk= =23Kd -----END PGP SIGNATURE----- _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From mbower2 at gmail.com Wed Apr 3 08:34:47 2013 From: mbower2 at gmail.com (Michael Bower) Date: Wed, 3 Apr 2013 11:34:47 -0400 Subject: [Bro] Help with searching logs In-Reply-To: <21DD7C64179C9843B756C6DD491634DB5385157E@Mailbox1.boco.co.boulder.co.us> References: <515B7581.9080403@gmail.com> <21DD7C64179C9843B756C6DD491634DB5385146A@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851519@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB5385157E@Mailbox1.boco.co.boulder.co.us> Message-ID: Thanks! This will give me something to go on. On Apr 3, 2013 11:30 AM, "Castle, Shane" wrote: > Seems as though an awk update tightened some of the syntax restrictions. > This script works: > > bro-cut id.orig_h orig_bytes < conn.log \ > | sort -t '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n \ > | awk 'BEGIN { size=0;host="" } \ > { if (host != $1) { \ > if (size != 0) \ > print host, size; \ > host=$1; \ > if ($2 != "-") { \ > size=$2 } \ > else { \ > size=0; } \ > } else { \ > if ($2 != "-") \ > size += $2 } \ > } \ > END { \ > if (size != 0) \ > print host, size \ > }' \ > | sort -rnk 2 \ > | head -n 10 > > The conversations could be done by a script that takes conn.log as input, > merges orig_h and resp_h into one field while adding together their bytes, > sorting on that, then putting the result through the same awk script. The > output might look something like this: > > 1.2.3.4|5.6.7.8 123456789 > ... > > depending on what you choose for your conjoining character. Also, note > that if there are multiple conversations between two IP addresses then you > will be adding up all those conversations and presenting the result as one > line - that might not be exactly what you are looking for. > > Welcome to the world of IDS data mining. Sharpen your > awk/sort/sql/perl/bash skills - they will come in very handy. > > -- > Shane Castle > Data Security Mgr, Boulder County IT > > > -----Original Message----- > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of > Castle, Shane > Sent: Wednesday, April 03, 2013 08:59 > To: 'Michael Bower'; 'bro at bro.org' > Subject: Re: [Bro] Help with searching logs > > Hm, I get a syntax error in that script now. Let me figure this out... > > -- > Shane Castle > Data Security Mgr, Boulder County IT > > > -----Original Message----- > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of > Castle, Shane > Sent: Wednesday, April 03, 2013 08:31 > To: 'Michael Bower'; 'bro at bro.org' > Subject: Re: [Bro] Help with searching logs > > The script is lying to you. Here's the correct script: > > bro-cut id.orig_h orig_bytes < conn.log \ > | sort -t '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n \ > | awk 'BEGIN { size=0;host="" } \ > { if (host != $1) { \ > if (size != 0) \ > print host, size; \ > host=$1; \ > if ($2 != "-") \ > size=$2 \ > else \ > size=0 \ > } else \ > if ($2 != "-") \ > size += $2 \ > } \ > END { \ > if (size != 0) \ > print host, size \ > }' \ > | sort -rnk 2 \ > | head -n 10 > > Since this script summarizes, having timestamps in there would not be > useful. And, if you want to change the logic to responder, change "orig" to > "resp" in the first line. Yes, it would be nice to have the top > conversations, not just the top talkers, which would combine both orig and > resp, but I'm not sure the result would justify the work. > > A few months ago I went through this and we were hoping that the doc would > have been changed to show a correct script, but it has not been, apparently. > > -- > Shane Castle > Data Security Mgr, Boulder County IT > > -----Original Message----- > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of > Michael Bower > Sent: Tuesday, April 02, 2013 18:19 > To: bro at bro.org > Subject: [Bro] Help with searching logs > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > > Im still learning, so bare with me. I ran the following command: > > bro-cut id.orig_h orig_bytes < conn.log \ > | sort \ > | awk '{ if (host != $1) { \ > if (size != 0) \ > print $1, size; \ > host=$1; \ > size=0 \ > } else \ > size += $2 \ > } \ > END { \ > if (size != 0) \ > print $1, size \ > }' \ > | sort -rnk 2 \ > | head -n 10 > > > This worked well to show me the top 10 hosts (originators). What Im > trying to do is show the top 10 hosts and the time (ts). Maybe show > the resp_bytes field too, if that is possible. Any help would be > greatly appreciated. > > Thanks! > - -- > > Mike > > > -----BEGIN PGP SIGNATURE----- > Comment: GPGTools - http://gpgtools.org > > iQEcBAEBCgAGBQJRW3WBAAoJEIAKCPjZh/yXUF4H/RhFuVQy6bT3Z8Z1k2oMDBGt > TYFAfsyeXcnf9dOl3NFGEIlifjDMZ/gK5kBVWo/FYSHGWHrCT0+ICcsjwLroRP/E > rn1StPS7ozlSiY2ZJSG0UAUCZX0HZ0ujvmNo8UvnoAR75cORq8Y08cU4XpLjqhxc > d4xu3G+HnhyzjKAiG6xtqDpK2Z3bnjJzyWEqZCoYDzNqtcYnrxYjcKa0kX9rBhUr > uV6upZ9OHIdf25EYCVfjDrKPSUaRhSAnTVtYE0+OQRA0OPpnG3rLWFSK2yjkTbNG > AzKXfhJZ0PWmUWkeD6Bzf2TCNqfyLchNSScm2atA/dhTRBV3JhHIhwIcejXr6sk= > =23Kd > -----END PGP SIGNATURE----- > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130403/1a604a30/attachment.html From scastle at bouldercounty.org Wed Apr 3 13:15:18 2013 From: scastle at bouldercounty.org (Castle, Shane) Date: Wed, 3 Apr 2013 20:15:18 +0000 Subject: [Bro] Help with searching logs In-Reply-To: References: <515B7581.9080403@gmail.com> <21DD7C64179C9843B756C6DD491634DB5385146A@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851519@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB5385157E@Mailbox1.boco.co.boulder.co.us> Message-ID: <21DD7C64179C9843B756C6DD491634DB53851CB7@Mailbox1.boco.co.boulder.co.us> Just for grins, I worked on the conversation angle a bit. Here's what I came up with as a first hack. $ bro-cut id.orig_h id.resp_h orig_bytes resp_bytes ", then I get $ bro-cut id.orig_h id.resp_h orig_bytes resp_bytes wrote: Seems as though an awk update tightened some of the syntax restrictions. This script works: bro-cut id.orig_h orig_bytes < conn.log \ | sort -t '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n \ | awk 'BEGIN { size=0;host="" } \ { if (host != $1) { \ if (size != 0) \ print host, size; \ host=$1; \ if ($2 != "-") { \ size=$2 } \ else { \ size=0; } \ } else { \ if ($2 != "-") \ size += $2 } \ } \ END { \ if (size != 0) \ print host, size \ }' \ | sort -rnk 2 \ | head -n 10 The conversations could be done by a script that takes conn.log as input, merges orig_h and resp_h into one field while adding together their bytes, sorting on that, then putting the result through the same awk script. The output might look something like this: 1.2.3.4|5.6.7.8 123456789 ... depending on what you choose for your conjoining character. Also, note that if there are multiple conversations between two IP addresses then you will be adding up all those conversations and presenting the result as one line - that might not be exactly what you are looking for. Welcome to the world of IDS data mining. Sharpen your awk/sort/sql/perl/bash skills - they will come in very handy. -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Castle, Shane Sent: Wednesday, April 03, 2013 08:59 To: 'Michael Bower'; 'bro at bro.org' Subject: Re: [Bro] Help with searching logs Hm, I get a syntax error in that script now. Let me figure this out... -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Castle, Shane Sent: Wednesday, April 03, 2013 08:31 To: 'Michael Bower'; 'bro at bro.org' Subject: Re: [Bro] Help with searching logs The script is lying to you. Here's the correct script: bro-cut id.orig_h orig_bytes < conn.log \ | sort -t '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n \ | awk 'BEGIN { size=0;host="" } \ { if (host != $1) { \ if (size != 0) \ print host, size; \ host=$1; \ if ($2 != "-") \ size=$2 \ else \ size=0 \ } else \ if ($2 != "-") \ size += $2 \ } \ END { \ if (size != 0) \ print host, size \ }' \ | sort -rnk 2 \ | head -n 10 Since this script summarizes, having timestamps in there would not be useful. And, if you want to change the logic to responder, change "orig" to "resp" in the first line. Yes, it would be nice to have the top conversations, not just the top talkers, which would combine both orig and resp, but I'm not sure the result would justify the work. A few months ago I went through this and we were hoping that the doc would have been changed to show a correct script, but it has not been, apparently. -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Michael Bower Sent: Tuesday, April 02, 2013 18:19 To: bro at bro.org Subject: [Bro] Help with searching logs -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Im still learning, so bare with me. I ran the following command: bro-cut id.orig_h orig_bytes < conn.log \ | sort \ | awk '{ if (host != $1) { \ if (size != 0) \ print $1, size; \ host=$1; \ size=0 \ } else \ size += $2 \ } \ END { \ if (size != 0) \ print $1, size \ }' \ | sort -rnk 2 \ | head -n 10 This worked well to show me the top 10 hosts (originators). What Im trying to do is show the top 10 hosts and the time (ts). Maybe show the resp_bytes field too, if that is possible. Any help would be greatly appreciated. Thanks! - -- Mike -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBCgAGBQJRW3WBAAoJEIAKCPjZh/yXUF4H/RhFuVQy6bT3Z8Z1k2oMDBGt TYFAfsyeXcnf9dOl3NFGEIlifjDMZ/gK5kBVWo/FYSHGWHrCT0+ICcsjwLroRP/E rn1StPS7ozlSiY2ZJSG0UAUCZX0HZ0ujvmNo8UvnoAR75cORq8Y08cU4XpLjqhxc d4xu3G+HnhyzjKAiG6xtqDpK2Z3bnjJzyWEqZCoYDzNqtcYnrxYjcKa0kX9rBhUr uV6upZ9OHIdf25EYCVfjDrKPSUaRhSAnTVtYE0+OQRA0OPpnG3rLWFSK2yjkTbNG AzKXfhJZ0PWmUWkeD6Bzf2TCNqfyLchNSScm2atA/dhTRBV3JhHIhwIcejXr6sk= =23Kd -----END PGP SIGNATURE----- _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From scastle at bouldercounty.org Wed Apr 3 13:47:45 2013 From: scastle at bouldercounty.org (Castle, Shane) Date: Wed, 3 Apr 2013 20:47:45 +0000 Subject: [Bro] Help with searching logs In-Reply-To: <21DD7C64179C9843B756C6DD491634DB53851CB7@Mailbox1.boco.co.boulder.co.us> References: <515B7581.9080403@gmail.com> <21DD7C64179C9843B756C6DD491634DB5385146A@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851519@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB5385157E@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851CB7@Mailbox1.boco.co.boulder.co.us> Message-ID: <21DD7C64179C9843B756C6DD491634DB53851D50@Mailbox1.boco.co.boulder.co.us> No, not really. I discovered that Bro is not reporting the bytes correctly. That makes these numbers totally suspect. I wonder what's going on there. scastle at nsm:~/scripts$ zcat /nsm/bro/logs/2013-04-03/conn.19:00:00-20:00:00.log.gz | bro-cut -d ts id.orig_h id.resp_h orig_bytes resp_bytes | grep 192.168.56.166.23.7.65.224 2013-04-03T19:21:28+0000 192.168.56.166 23.7.65.224 0 1200242775 2013-04-03T19:21:28+0000 192.168.56.166 23.7.65.224 0 1214734459 2013-04-03T19:21:33+0000 192.168.56.166 23.7.65.224 540 3920 2013-04-03T19:21:51+0000 192.168.56.166 23.7.65.224 0 1214734460 2013-04-03T19:21:40+0000 192.168.56.166 23.7.65.224 0 1200242776 2013-04-03T19:21:28+0000 192.168.56.166 23.7.65.224 1112 1200442670 2013-04-03T19:21:33+0000 192.168.56.166 23.7.65.224 1270 4325 2013-04-03T19:24:42+0000 192.168.56.166 23.7.65.224 0 0 2013-04-03T19:24:41+0000 192.168.56.166 23.7.65.224 0 0 2013-04-03T19:24:46+0000 192.168.56.166 23.7.65.224 572 3920 2013-04-03T19:24:47+0000 192.168.56.166 23.7.65.224 572 3920 2013-04-03T19:24:42+0000 192.168.56.166 23.7.65.224 1105 405 2013-04-03T19:24:41+0000 192.168.56.166 23.7.65.224 504 26430 2013-04-03T19:21:28+0000 192.168.56.166 23.7.65.224 0 3067067778 Looks like resp_bytes is not being properly shown sometimes. Hmm, missed_bytes seems to be large here, too. Sigh - I still don't know what's going on. If missed_bytes is nonzero, the orig and resp bytes can't be trusted. More work and research. Here's the unfiltered output: scastle at nsm:~/scripts$ zcat /nsm/bro/logs/2013-04-03/conn.19:00:00-20:00:00.log.gz | bro-cut -d | grep 192.168.56.166.\*23.7.65.224 2013-04-03T19:21:28+0000 YTOQlbbx1Pg 192.168.56.166 64878 23.7.65.224 80 tcp - 6.655300 0 1200242775 SF T 1200242775 ShhhhAFf 14 788 8 472 (empty) 2013-04-03T19:21:28+0000 bLfZjXnY29h 192.168.56.166 64888 23.7.65.224 80 tcp - 6.284761 0 1214734459 SF T 1214734459 ShhhAhFfr 15 840 10 572 (empty) 2013-04-03T19:21:33+0000 zs1xf4L3iZ6 192.168.56.166 64906 23.7.65.224 443 tcp ssl 5.754741 540 3920 SF T 0 ShADadFfRR 16 1372 12 4568 (empty) 2013-04-03T19:21:51+0000 ramah8M2Oc1 192.168.56.166 64888 23.7.65.224 80 tcp - 24.008354 0 1214734460 RSTR T 0 hArR 2 92 3 160 (empty) 2013-04-03T19:21:40+0000 JObupHqiDjl 192.168.56.166 64878 23.7.65.224 80 tcp - 36.214781 0 1200242776 RSTR T 0 hArR 3 144 5 260 (empty) 2013-04-03T19:21:28+0000 XkrirkTDGzk 192.168.56.166 64876 23.7.65.224 80 tcp http 113.531073 1112 1200442670 SF T 1200442265 ShhhADadhfF 20 2212 15 1273 (empty) 2013-04-03T19:21:33+0000 PtKQpijzTff 192.168.56.166 64905 23.7.65.224 443 tcp ssl 116.851227 1270 4325 SF T 0 ShADadFfRR 18 2206 13 5025 (empty) 2013-04-03T19:24:42+0000 tAwhU1nQ8e8 192.168.56.166 65044 23.7.65.224 80 tcp - 7.372079 0 0 SF T 0 ShAFfar 6 324 5 268 (empty) 2013-04-03T19:24:41+0000 wWH8h7RtYg6 192.168.56.166 65003 23.7.65.224 80 tcp - 8.491360 0 0 SF T 0 ShAFfar 6 324 5 268 (empty) 2013-04-03T19:24:46+0000 lsIG9TsJ0Ti 192.168.56.166 65081 23.7.65.224 443 tcp ssl 6.057423 572 3920 SF T 0 ShADadFfR 18 1532 12 4584 (empty) 2013-04-03T19:24:47+0000 TXZBimJtLN6 192.168.56.166 65083 23.7.65.224 443 tcp ssl 5.903914 572 3920 SF T 0 ShADadFfR 18 1532 12 4584 (empty) 2013-04-03T19:24:42+0000 qtKgUBFlRNa 192.168.56.166 65041 23.7.65.224 80 tcp http 96.772052 1105 405 SF T 0 ShADadfF 8 1533 7 1587 (empty) 2013-04-03T19:24:41+0000 KmKMWDZ7Kk2 192.168.56.166 65001 23.7.65.224 80 tcp http 104.511714 504 26430 SF T 0 ShADadfF 18 1452 24 29134 (empty) 2013-04-03T19:21:28+0000 R4eSKC00lqe 192.168.56.166 64879 23.7.65.224 80 tcp - 46.186418 0 3067067778 SF T 0 ShhAFfarR 17 932 15 872 (empty) -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Castle, Shane Sent: Wednesday, April 03, 2013 14:15 To: 'Michael Bower' Cc: 'Bro Mailing List' Subject: Re: [Bro] Help with searching logs Just for grins, I worked on the conversation angle a bit. Here's what I came up with as a first hack. $ bro-cut id.orig_h id.resp_h orig_bytes resp_bytes ", then I get $ bro-cut id.orig_h id.resp_h orig_bytes resp_bytes wrote: Seems as though an awk update tightened some of the syntax restrictions. This script works: bro-cut id.orig_h orig_bytes < conn.log \ | sort -t '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n \ | awk 'BEGIN { size=0;host="" } \ { if (host != $1) { \ if (size != 0) \ print host, size; \ host=$1; \ if ($2 != "-") { \ size=$2 } \ else { \ size=0; } \ } else { \ if ($2 != "-") \ size += $2 } \ } \ END { \ if (size != 0) \ print host, size \ }' \ | sort -rnk 2 \ | head -n 10 The conversations could be done by a script that takes conn.log as input, merges orig_h and resp_h into one field while adding together their bytes, sorting on that, then putting the result through the same awk script. The output might look something like this: 1.2.3.4|5.6.7.8 123456789 ... depending on what you choose for your conjoining character. Also, note that if there are multiple conversations between two IP addresses then you will be adding up all those conversations and presenting the result as one line - that might not be exactly what you are looking for. Welcome to the world of IDS data mining. Sharpen your awk/sort/sql/perl/bash skills - they will come in very handy. -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Castle, Shane Sent: Wednesday, April 03, 2013 08:59 To: 'Michael Bower'; 'bro at bro.org' Subject: Re: [Bro] Help with searching logs Hm, I get a syntax error in that script now. Let me figure this out... -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Castle, Shane Sent: Wednesday, April 03, 2013 08:31 To: 'Michael Bower'; 'bro at bro.org' Subject: Re: [Bro] Help with searching logs The script is lying to you. Here's the correct script: bro-cut id.orig_h orig_bytes < conn.log \ | sort -t '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n \ | awk 'BEGIN { size=0;host="" } \ { if (host != $1) { \ if (size != 0) \ print host, size; \ host=$1; \ if ($2 != "-") \ size=$2 \ else \ size=0 \ } else \ if ($2 != "-") \ size += $2 \ } \ END { \ if (size != 0) \ print host, size \ }' \ | sort -rnk 2 \ | head -n 10 Since this script summarizes, having timestamps in there would not be useful. And, if you want to change the logic to responder, change "orig" to "resp" in the first line. Yes, it would be nice to have the top conversations, not just the top talkers, which would combine both orig and resp, but I'm not sure the result would justify the work. A few months ago I went through this and we were hoping that the doc would have been changed to show a correct script, but it has not been, apparently. -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Michael Bower Sent: Tuesday, April 02, 2013 18:19 To: bro at bro.org Subject: [Bro] Help with searching logs -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Im still learning, so bare with me. I ran the following command: bro-cut id.orig_h orig_bytes < conn.log \ | sort \ | awk '{ if (host != $1) { \ if (size != 0) \ print $1, size; \ host=$1; \ size=0 \ } else \ size += $2 \ } \ END { \ if (size != 0) \ print $1, size \ }' \ | sort -rnk 2 \ | head -n 10 This worked well to show me the top 10 hosts (originators). What Im trying to do is show the top 10 hosts and the time (ts). Maybe show the resp_bytes field too, if that is possible. Any help would be greatly appreciated. Thanks! - -- Mike -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBCgAGBQJRW3WBAAoJEIAKCPjZh/yXUF4H/RhFuVQy6bT3Z8Z1k2oMDBGt TYFAfsyeXcnf9dOl3NFGEIlifjDMZ/gK5kBVWo/FYSHGWHrCT0+ICcsjwLroRP/E rn1StPS7ozlSiY2ZJSG0UAUCZX0HZ0ujvmNo8UvnoAR75cORq8Y08cU4XpLjqhxc d4xu3G+HnhyzjKAiG6xtqDpK2Z3bnjJzyWEqZCoYDzNqtcYnrxYjcKa0kX9rBhUr uV6upZ9OHIdf25EYCVfjDrKPSUaRhSAnTVtYE0+OQRA0OPpnG3rLWFSK2yjkTbNG AzKXfhJZ0PWmUWkeD6Bzf2TCNqfyLchNSScm2atA/dhTRBV3JhHIhwIcejXr6sk= =23Kd -----END PGP SIGNATURE----- _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From Keith_Schoenefeld at baylor.edu Wed Apr 3 14:58:26 2013 From: Keith_Schoenefeld at baylor.edu (Schoenefeld, Keith P.) Date: Wed, 3 Apr 2013 21:58:26 +0000 Subject: [Bro] Help with searching logs In-Reply-To: <21DD7C64179C9843B756C6DD491634DB53851D50@Mailbox1.boco.co.boulder.co.us> References: <515B7581.9080403@gmail.com> <21DD7C64179C9843B756C6DD491634DB5385146A@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851519@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB5385157E@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851CB7@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851D50@Mailbox1.boco.co.boulder.co.us> Message-ID: This may be unrelated (or even something I'm doing terribly wrong), but I noticed what I think is a similar issue when working with the capture-loss script that led me to believe that the underlying code in Bro may not be properly dealing with it when the 32 bit sequence number in a TCP packet rolls over. I spoke briefly to Seth about it, and he asked me to provide the .bro code I was using as well as a pcap demonstrating my concerns -- I failed to do either. The modified capture_loss script I'm running results in: #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path capture_loss #open 2013-04-03-21-00-08 #fields ts ts_delta peer gaps acks percent_lost gap_bytes ack_bytes percent_bytes_lost #types time interval string count count string count count string 1365022809.033023 60.000027 worker-4-4 3 170598 0.002 4239898009 4652853950 91.125 1365022809.059350 60.000047 worker-1-1 0 129786 0.000 0 301416506 0.000 1365022808.638408 60.000054 worker-2-1 1 143234 0.001 1611842887 1956581780 82.381 1365022808.885606 60.000046 worker-3-4 1 144122 0.001 1350348978 1680367485 80.360 1365022809.328350 60.000060 worker-3-3 0 148226 0.000 0 364927843 0.000 1365022809.292290 60.000081 worker-4-2 1 118706 0.001 1609627515 1900369040 84.701 I'll try to throw something together and get it submitted. -- KS Keith Schoenefeld Information Security Analyst Baylor University 254-710-6667 -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Castle, Shane Sent: Wednesday, April 03, 2013 3:48 PM To: 'Michael Bower' Cc: 'Bro Mailing List' Subject: Re: [Bro] Help with searching logs No, not really. I discovered that Bro is not reporting the bytes correctly. That makes these numbers totally suspect. I wonder what's going on there. scastle at nsm:~/scripts$ zcat /nsm/bro/logs/2013-04-03/conn.19:00:00-20:00:00.log.gz | bro-cut -d ts id.orig_h id.resp_h orig_bytes resp_bytes | grep 192.168.56.166.23.7.65.224 2013-04-03T19:21:28+0000 192.168.56.166 23.7.65.224 0 1200242775 2013-04-03T19:21:28+0000 192.168.56.166 23.7.65.224 0 1214734459 2013-04-03T19:21:33+0000 192.168.56.166 23.7.65.224 540 3920 2013-04-03T19:21:51+0000 192.168.56.166 23.7.65.224 0 1214734460 2013-04-03T19:21:40+0000 192.168.56.166 23.7.65.224 0 1200242776 2013-04-03T19:21:28+0000 192.168.56.166 23.7.65.224 1112 1200442670 2013-04-03T19:21:33+0000 192.168.56.166 23.7.65.224 1270 4325 2013-04-03T19:24:42+0000 192.168.56.166 23.7.65.224 0 0 2013-04-03T19:24:41+0000 192.168.56.166 23.7.65.224 0 0 2013-04-03T19:24:46+0000 192.168.56.166 23.7.65.224 572 3920 2013-04-03T19:24:47+0000 192.168.56.166 23.7.65.224 572 3920 2013-04-03T19:24:42+0000 192.168.56.166 23.7.65.224 1105 405 2013-04-03T19:24:41+0000 192.168.56.166 23.7.65.224 504 26430 2013-04-03T19:21:28+0000 192.168.56.166 23.7.65.224 0 3067067778 Looks like resp_bytes is not being properly shown sometimes. Hmm, missed_bytes seems to be large here, too. Sigh - I still don't know what's going on. If missed_bytes is nonzero, the orig and resp bytes can't be trusted. More work and research. Here's the unfiltered output: scastle at nsm:~/scripts$ zcat /nsm/bro/logs/2013-04-03/conn.19:00:00-20:00:00.log.gz | bro-cut -d | grep 192.168.56.166.\*23.7.65.224 2013-04-03T19:21:28+0000 YTOQlbbx1Pg 192.168.56.166 64878 23.7.65.224 80 tcp - 6.655300 0 1200242775 SF T 1200242775 ShhhhAFf 14 788 8 472 (empty) 2013-04-03T19:21:28+0000 bLfZjXnY29h 192.168.56.166 64888 23.7.65.224 80 tcp - 6.284761 0 1214734459 SF T 1214734459 ShhhAhFfr 15 840 10 572 (empty) 2013-04-03T19:21:33+0000 zs1xf4L3iZ6 192.168.56.166 64906 23.7.65.224 443 tcp ssl 5.754741 540 3920 SF T 0 ShADadFfRR 16 1372 12 4568 (empty) 2013-04-03T19:21:51+0000 ramah8M2Oc1 192.168.56.166 64888 23.7.65.224 80 tcp - 24.008354 0 1214734460 RSTR T 0 hArR 2 92 3 160 (empty) 2013-04-03T19:21:40+0000 JObupHqiDjl 192.168.56.166 64878 23.7.65.224 80 tcp - 36.214781 0 1200242776 RSTR T 0 hArR 3 144 5 260 (empty) 2013-04-03T19:21:28+0000 XkrirkTDGzk 192.168.56.166 64876 23.7.65.224 80 tcp http 113.531073 1112 1200442670 SF T 1200442265 ShhhADadhfF 20 2212 15 1273 (empty) 2013-04-03T19:21:33+0000 PtKQpijzTff 192.168.56.166 64905 23.7.65.224 443 tcp ssl 116.851227 1270 4325 SF T 0 ShADadFfRR 18 2206 13 5025 (empty) 2013-04-03T19:24:42+0000 tAwhU1nQ8e8 192.168.56.166 65044 23.7.65.224 80 tcp - 7.372079 0 0 SF T 0 ShAFfar 6 324 5 268 (empty) 2013-04-03T19:24:41+0000 wWH8h7RtYg6 192.168.56.166 65003 23.7.65.224 80 tcp - 8.491360 0 0 SF T 0 ShAFfar 6 324 5 268 (empty) 2013-04-03T19:24:46+0000 lsIG9TsJ0Ti 192.168.56.166 65081 23.7.65.224 443 tcp ssl 6.057423 572 3920 SF T 0 ShADadFfR 18 1532 12 4584 (empty) 2013-04-03T19:24:47+0000 TXZBimJtLN6 192.168.56.166 65083 23.7.65.224 443 tcp ssl 5.903914 572 3920 SF T 0 ShADadFfR 18 1532 12 4584 (empty) 2013-04-03T19:24:42+0000 qtKgUBFlRNa 192.168.56.166 65041 23.7.65.224 80 tcp http 96.772052 1105 405 SF T 0 ShADadfF 8 1533 7 1587 (empty) 2013-04-03T19:24:41+0000 KmKMWDZ7Kk2 192.168.56.166 65001 23.7.65.224 80 tcp http 104.511714 504 26430 SF T 0 ShADadfF 18 1452 24 29134 (empty) 2013-04-03T19:21:28+0000 R4eSKC00lqe 192.168.56.166 64879 23.7.65.224 80 tcp - 46.186418 0 3067067778 SF T 0 ShhAFfarR 17 932 15 872 (empty) -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Castle, Shane Sent: Wednesday, April 03, 2013 14:15 To: 'Michael Bower' Cc: 'Bro Mailing List' Subject: Re: [Bro] Help with searching logs Just for grins, I worked on the conversation angle a bit. Here's what I came up with as a first hack. $ bro-cut id.orig_h id.resp_h orig_bytes resp_bytes ", then I get $ bro-cut id.orig_h id.resp_h orig_bytes resp_bytes wrote: Seems as though an awk update tightened some of the syntax restrictions. This script works: bro-cut id.orig_h orig_bytes < conn.log \ | sort -t '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n \ | awk 'BEGIN { size=0;host="" } \ { if (host != $1) { \ if (size != 0) \ print host, size; \ host=$1; \ if ($2 != "-") { \ size=$2 } \ else { \ size=0; } \ } else { \ if ($2 != "-") \ size += $2 } \ } \ END { \ if (size != 0) \ print host, size \ }' \ | sort -rnk 2 \ | head -n 10 The conversations could be done by a script that takes conn.log as input, merges orig_h and resp_h into one field while adding together their bytes, sorting on that, then putting the result through the same awk script. The output might look something like this: 1.2.3.4|5.6.7.8 123456789 ... depending on what you choose for your conjoining character. Also, note that if there are multiple conversations between two IP addresses then you will be adding up all those conversations and presenting the result as one line - that might not be exactly what you are looking for. Welcome to the world of IDS data mining. Sharpen your awk/sort/sql/perl/bash skills - they will come in very handy. -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Castle, Shane Sent: Wednesday, April 03, 2013 08:59 To: 'Michael Bower'; 'bro at bro.org' Subject: Re: [Bro] Help with searching logs Hm, I get a syntax error in that script now. Let me figure this out... -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Castle, Shane Sent: Wednesday, April 03, 2013 08:31 To: 'Michael Bower'; 'bro at bro.org' Subject: Re: [Bro] Help with searching logs The script is lying to you. Here's the correct script: bro-cut id.orig_h orig_bytes < conn.log \ | sort -t '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n \ | awk 'BEGIN { size=0;host="" } \ { if (host != $1) { \ if (size != 0) \ print host, size; \ host=$1; \ if ($2 != "-") \ size=$2 \ else \ size=0 \ } else \ if ($2 != "-") \ size += $2 \ } \ END { \ if (size != 0) \ print host, size \ }' \ | sort -rnk 2 \ | head -n 10 Since this script summarizes, having timestamps in there would not be useful. And, if you want to change the logic to responder, change "orig" to "resp" in the first line. Yes, it would be nice to have the top conversations, not just the top talkers, which would combine both orig and resp, but I'm not sure the result would justify the work. A few months ago I went through this and we were hoping that the doc would have been changed to show a correct script, but it has not been, apparently. -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Michael Bower Sent: Tuesday, April 02, 2013 18:19 To: bro at bro.org Subject: [Bro] Help with searching logs -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Im still learning, so bare with me. I ran the following command: bro-cut id.orig_h orig_bytes < conn.log \ | sort \ | awk '{ if (host != $1) { \ if (size != 0) \ print $1, size; \ host=$1; \ size=0 \ } else \ size += $2 \ } \ END { \ if (size != 0) \ print $1, size \ }' \ | sort -rnk 2 \ | head -n 10 This worked well to show me the top 10 hosts (originators). What Im trying to do is show the top 10 hosts and the time (ts). Maybe show the resp_bytes field too, if that is possible. Any help would be greatly appreciated. Thanks! - -- Mike -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBCgAGBQJRW3WBAAoJEIAKCPjZh/yXUF4H/RhFuVQy6bT3Z8Z1k2oMDBGt TYFAfsyeXcnf9dOl3NFGEIlifjDMZ/gK5kBVWo/FYSHGWHrCT0+ICcsjwLroRP/E rn1StPS7ozlSiY2ZJSG0UAUCZX0HZ0ujvmNo8UvnoAR75cORq8Y08cU4XpLjqhxc d4xu3G+HnhyzjKAiG6xtqDpK2Z3bnjJzyWEqZCoYDzNqtcYnrxYjcKa0kX9rBhUr uV6upZ9OHIdf25EYCVfjDrKPSUaRhSAnTVtYE0+OQRA0OPpnG3rLWFSK2yjkTbNG AzKXfhJZ0PWmUWkeD6Bzf2TCNqfyLchNSScm2atA/dhTRBV3JhHIhwIcejXr6sk= =23Kd -----END PGP SIGNATURE----- _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From seth at icir.org Wed Apr 3 18:12:16 2013 From: seth at icir.org (Seth Hall) Date: Wed, 3 Apr 2013 21:12:16 -0400 Subject: [Bro] Help with searching logs In-Reply-To: <21DD7C64179C9843B756C6DD491634DB53851D50@Mailbox1.boco.co.boulder.co.us> References: <515B7581.9080403@gmail.com> <21DD7C64179C9843B756C6DD491634DB5385146A@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851519@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB5385157E@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851CB7@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851D50@Mailbox1.boco.co.boulder.co.us> Message-ID: On Apr 3, 2013, at 4:47 PM, "Castle, Shane" wrote: > Looks like resp_bytes is not being properly shown sometimes. Hmm, missed_bytes seems to be large here, too. Sigh - I still don't know what's going on. If missed_bytes is nonzero, the orig and resp bytes can't be trusted. More work and research.  The orig_bytes and resp_bytes fields can still be trusted even with missed_bytes being something greater than zero. Those two fields are calculated with TCP sequence number counting and so they can cope with packet loss. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Wed Apr 3 18:23:03 2013 From: seth at icir.org (Seth Hall) Date: Wed, 3 Apr 2013 21:23:03 -0400 Subject: [Bro] Help with searching logs In-Reply-To: <21DD7C64179C9843B756C6DD491634DB53851D50@Mailbox1.boco.co.boulder.co.us> References: <515B7581.9080403@gmail.com> <21DD7C64179C9843B756C6DD491634DB5385146A@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851519@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB5385157E@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851CB7@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851D50@Mailbox1.boco.co.boulder.co.us> Message-ID: On Apr 3, 2013, at 4:47 PM, "Castle, Shane" wrote: > Looks like resp_bytes is not being properly shown sometimes. Hmm, missed_bytes seems to be large here, too. Sigh - I still don't know what's going on. If missed_bytes is nonzero, the orig and resp bytes can't be trusted. More work and research. > > Here's the unfiltered output: > scastle at nsm:~/scripts$ zcat /nsm/bro/logs/2013-04-03/conn.19:00:00-20:00:00.log.gz | bro-cut -d | grep 192.168.56.166.\*23.7.65.224 > 2013-04-03T19:21:51+0000 ramah8M2Oc1 192.168.56.166 64888 23.7.65.224 80 tcp - 24.008354 0 1214734460 RSTR T 0 hArR 2 92 3 160 (empty) This is what I get for not reading the whole email. Bro has/had an issue with middle boxes sending RST packets to kill TCP connections (great firewall of China being a primary offender) where it would use the sequence number from the RST packet instead of the sequence number from the initial syn or syn-ack. It resulted in these connections like you have here with very few packets and huge data sizes. It's fixed in master and if you want more context to the problem you can refer to the ticket where we tracked the issue and fix: http://tracker.bro.org/bro/ticket/730 .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From Eric.Asselin at usherbrooke.ca Thu Apr 4 02:15:06 2013 From: Eric.Asselin at usherbrooke.ca (Eric Asselin) Date: Thu, 04 Apr 2013 11:15:06 +0200 Subject: [Bro] segmentation fault by adding an analyzer to port 80 Message-ID: <515D449A.6090705@usherbrooke.ca> When I force the anaylzer to be activated by modifying the DPM.cc, everything works well. But when I try to activate it by using a Bro script, like this: global foo_ports: set[port] = { 80/tcp } &redef; redef dpd_config += { [ANALYZER_MYPROTO] = [$ports = foo_ports] }; a segmentation fault appear at the end of the script. There is the command I use to execute it (note that the browse.pcap file came from a Bro workshop): bro -b -r browse.pcap my_proto.bro Here's the complete code of the analyzer and the Bro script: MyProto.h ---------------------------------- #ifndef myproto_h #define myproto_h #include "Analyzer.h" class MyProto_Analyzer : public Analyzer { public: MyProto_Analyzer(Connection* conn); virtual ~MyProto_Analyzer(); virtual void Init(); static Analyzer* InstantiateAnalyzer(Connection* conn) { return new MyProto_Analyzer(conn); } static bool Available() { return true; } protected: virtual void Done(); virtual void DeliverPacket(int len, const u_char* data, bool orig, int seq, const IP_Hdr* ip, int caplen); }; #endif MyProto.cc ---------------------------------- #include "MyProto.h" MyProto_Analyzer::MyProto_Analyzer(Connection* c) : Analyzer(AnalyzerTag::MyProto, c) {} MyProto_Analyzer::~MyProto_Analyzer() {} void MyProto_Analyzer::Init() { val_list* vl = new val_list; vl->append(BuildConnVal()); vl->append(new StringVal("A message from MyProto_Analyzer!")); Conn()->ConnectionEvent(MyEvent, this, vl); } void MyProto_Analyzer::Done() { Analyzer::Done(); } void MyProto_Analyzer::DeliverPacket(int len, const u_char* data, bool is_orig, int seq, const IP_Hdr* ip, int caplen) { Analyzer::DeliverPacket(len, data, is_orig, seq, ip, caplen); } my_proto.bro ---------------------------------- global foo_ports: set[port] = { 80/tcp } &redef; redef dpd_config += { [ANALYZER_MYPROTO] = [$ports = foo_ports] }; event bro_init() { print ("Hello!"); } event MyEvent(c: connection, msg: string) { print msg; } event.bif ---------------------------------- event MyEvent%(c: connection, msg: string%); Thanks, Eric. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130404/5154de30/attachment.html From Eric.Asselin at usherbrooke.ca Thu Apr 4 04:41:03 2013 From: Eric.Asselin at usherbrooke.ca (Eric Asselin) Date: Thu, 04 Apr 2013 13:41:03 +0200 Subject: [Bro] segmentation fault by adding an analyzer to port 80 In-Reply-To: <515D449A.6090705@usherbrooke.ca> References: <515D449A.6090705@usherbrooke.ca> Message-ID: <515D66CF.8010405@usherbrooke.ca> Additional information from gdb: Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? () (gdb) bt #0 0x0000000000000000 in ?? () #1 0x000000000082b156 in TCP_Analyzer::EndpointEOF (this=0xed24b0, endp=0xed2920) at /home/easselin/bro/src/TCP.cc:1756 #2 0x000000000082f8b5 in TCP_Reassembler::CheckEOF (this=0xed2920) at /home/easselin/bro/src/TCP_Reassembler.cc:574 #3 0x000000000082d4db in TCP_Endpoint::CheckEOF (this=0xed2570) at /home/easselin/bro/src/TCP_Endpoint.cc:98 #4 0x0000000000828e7f in TCP_Analyzer::DeliverPacket (this=0xed24b0, len=0, data=0xec1a06 "\r\n", is_orig=true, seq=-1, ip=0x7fffffffe310, caplen=0) at /home/easselin/bro/src/TCP.cc:1051 #5 0x0000000000686c2f in Analyzer::NextPacket (this=0xed24b0, len=20, data=0xec19f2 "\300\203", is_orig=true, seq=-1, ip=0x7fffffffe310, caplen=20) at /home/easselin/bro/src/Analyzer.cc:348 #6 0x00000000006b3733 in Connection::NextPacket (this=0xed23b0, t=1320329772.929872, is_orig=1, ip=0x7fffffffe310, len=20, caplen=20, data=@0x7fffffffe088, record_packet=@0x7fffffffe084, record_content=@0x7fffffffe080, hdr=0xebdc90, pkt=0xec19d0 "RT", hdr_size=14) at /home/easselin/bro/src/Conn.cc:259 #7 0x0000000000805a09 in NetSessions::DoNextPacket (this=0xebe740, t=1320329772.929872, hdr=0xebdc90, ip_hdr=0x7fffffffe310, pkt=0xec19d0 "RT", hdr_size=14, encapsulation=0x0) at /home/easselin/bro/src/Sessions.cc:706 #8 0x0000000000804011 in NetSessions::NextPacket (this=0xebe740, t=1320329772.929872, hdr=0xebdc90, pkt=0xec19d0 "RT", hdr_size=14, pkt_elem=0x0) at /home/easselin/bro/src/Sessions.cc:244 #9 0x0000000000803d56 in NetSessions::DispatchPacket (this=0xebe740, t=1320329772.929872, hdr=0xebdc90, pkt=0xec19d0 "RT", hdr_size=14, src_ps=0xebdc50, pkt_elem=0x0) at /home/easselin/bro/src/Sessions.cc:186 #10 0x000000000079f00c in net_packet_dispatch (t=1320329772.929872, hdr=0xebdc90, pkt=0xec19d0 "RT", hdr_size=14, src_ps=0xebdc50, pkt_elem=0x0) at /home/easselin/bro/src/Net.cc:353 #11 0x000000000079f25f in net_packet_arrival (t=1320329772.929872, hdr=0xebdc90, pkt=0xec19d0 "RT", hdr_size=14, src_ps=0xebdc50) at /home/easselin/bro/src/Net.cc:416 #12 0x00000000007b6ea1 in PktSrc::Process (this=0xebdc50) at /home/easselin/bro/src/PktSrc.cc:312 #13 0x000000000079f397 in net_run () at /home/easselin/bro/src/Net.cc:447 #14 0x000000000067df27 in main (argc=5, argv=0x7fffffffebd8) at /home/easselin/bro/src/main.cc:1077 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130404/4aba3ada/attachment.html From scastle at bouldercounty.org Thu Apr 4 07:51:02 2013 From: scastle at bouldercounty.org (Castle, Shane) Date: Thu, 4 Apr 2013 14:51:02 +0000 Subject: [Bro] Help with searching logs In-Reply-To: References: <515B7581.9080403@gmail.com> <21DD7C64179C9843B756C6DD491634DB5385146A@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851519@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB5385157E@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851CB7@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851D50@Mailbox1.boco.co.boulder.co.us> Message-ID: <21DD7C64179C9843B756C6DD491634DB53852527@Mailbox1.boco.co.boulder.co.us> So, this was not included in 2.1? I'm thinking "git clone --recursive git://git.bro.org/bro.git" will get me a new copy that includes this, right? -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: Seth Hall [mailto:seth at icir.org] Sent: Wednesday, April 03, 2013 19:23 To: Castle, Shane Cc: 'Michael Bower'; 'Bro Mailing List' Subject: Re: [Bro] Help with searching logs On Apr 3, 2013, at 4:47 PM, "Castle, Shane" wrote: > Looks like resp_bytes is not being properly shown sometimes. Hmm, missed_bytes seems to be large here, too. Sigh - I still don't know what's going on. If missed_bytes is nonzero, the orig and resp bytes can't be trusted. More work and research. > > Here's the unfiltered output: > scastle at nsm:~/scripts$ zcat /nsm/bro/logs/2013-04-03/conn.19:00:00-20:00:00.log.gz | bro-cut -d | grep 192.168.56.166.\*23.7.65.224 > 2013-04-03T19:21:51+0000 ramah8M2Oc1 192.168.56.166 64888 23.7.65.224 80 tcp - 24.008354 0 1214734460 RSTR T 0 hArR 2 92 3 160 (empty) This is what I get for not reading the whole email. Bro has/had an issue with middle boxes sending RST packets to kill TCP connections (great firewall of China being a primary offender) where it would use the sequence number from the RST packet instead of the sequence number from the initial syn or syn-ack. It resulted in these connections like you have here with very few packets and huge data sizes. It's fixed in master and if you want more context to the problem you can refer to the ticket where we tracked the issue and fix: http://tracker.bro.org/bro/ticket/730 .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jep at g-c-i.net Thu Apr 4 08:25:20 2013 From: jep at g-c-i.net (Parker, Jonathan E.) Date: Thu, 4 Apr 2013 15:25:20 +0000 Subject: [Bro] Snort Rules Message-ID: <36C06B73C5A9D845A5435F7AAE88802B40062794@Mail10.Corporate.net> I?m a Bro newbie and I?ve been tasked to look at using Bro to perform analysis on Pcap files. We?d like to utilize some existing Snort rules in this analysis. A number of the Snort rules contain ?offset? and ?depth? parameters. I?d appreciate some advice on how to accomplish doing these Snort alerts in Bro. Thanks ? Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130404/fa05c395/attachment.html From anthony.kasza at gmail.com Thu Apr 4 08:33:36 2013 From: anthony.kasza at gmail.com (anthony kasza) Date: Thu, 4 Apr 2013 10:33:36 -0500 Subject: [Bro] Snort Rules In-Reply-To: <36C06B73C5A9D845A5435F7AAE88802B40062794@Mail10.Corporate.net> References: <36C06B73C5A9D845A5435F7AAE88802B40062794@Mail10.Corporate.net> Message-ID: As a fellow newbie, I feel the best answer is "don't". To me, Bro seems better suited for flow like analysis, not byte-by-byte packet analysis. On Thu, Apr 4, 2013 at 10:25 AM, Parker, Jonathan E. wrote: > I?m a Bro newbie and I?ve been tasked to look at using Bro to perform > analysis on Pcap files. We?d like to utilize some existing Snort rules > in this analysis. A number of the Snort rules contain ?offset? and > ?depth? parameters. I?d appreciate some advice on how to accomplish > doing these Snort alerts in Bro. > > > Thanks ? Jon > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130404/8e1e5627/attachment.html From jsiwek at illinois.edu Thu Apr 4 08:48:19 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Thu, 4 Apr 2013 15:48:19 +0000 Subject: [Bro] segmentation fault by adding an analyzer to port 80 In-Reply-To: <515D449A.6090705@usherbrooke.ca> References: <515D449A.6090705@usherbrooke.ca> Message-ID: On Apr 4, 2013, at 4:15 AM, Eric Asselin wrote: > When I force the anaylzer to be activated by modifying the DPM.cc, everything works well. But when I try to activate it by using a Bro script, like this: > > global foo_ports: set[port] = { 80/tcp } &redef; > redef dpd_config += { [ANALYZER_MYPROTO] = [$ports = foo_ports] }; > > a segmentation fault appear at the end of the script. I don't think there's a way to attach packet-based analyzers to TCP connections like that, so manually adding it in DPM.cc via TCP_Analyzer::AddChildPacketAnalyzer() for the ports you want may be the right thing. But if what you really wanted is a stream-based analyzer (it only sees the content after TCP reassembly), you can derive from TCP_ApplicationAnalyzer instead of Analyzer and override DeliverStream() instead of DeliverPacket(). Then the dpd_config redef you had should work. - Jon From seth at icir.org Thu Apr 4 13:37:35 2013 From: seth at icir.org (Seth Hall) Date: Thu, 4 Apr 2013 16:37:35 -0400 Subject: [Bro] Help with searching logs In-Reply-To: <21DD7C64179C9843B756C6DD491634DB53852527@Mailbox1.boco.co.boulder.co.us> References: <515B7581.9080403@gmail.com> <21DD7C64179C9843B756C6DD491634DB5385146A@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851519@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB5385157E@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851CB7@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53851D50@Mailbox1.boco.co.boulder.co.us> <21DD7C64179C9843B756C6DD491634DB53852527@Mailbox1.boco.co.boulder.co.us> Message-ID: On Apr 4, 2013, at 10:51 AM, "Castle, Shane" wrote: > So, this was not included in 2.1? I'm thinking "git clone --recursive git://git.bro.org/bro.git" will get me a new copy that includes this, right? Correct :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130404/b357e2dc/attachment.bin From jlay at slave-tothe-box.net Fri Apr 5 09:03:06 2013 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 05 Apr 2013 10:03:06 -0600 Subject: [Bro] Command line bro on two interfaces Message-ID: Hey all...so here's my line: /usr/sbin/bro -i eth4 -i eth5 local Site::local_nets += { externalip/mask } I was thinking this was working, but alas, I see in my email packet captures that I think it's only listening on one interface as bro has missed an email that came by..I have no record of it anywhere. Is this not the right way to get this to work? I don't want to use broctl since the cpu usage is far too high. Thank you for any tips you may have. James From jlay at slave-tothe-box.net Fri Apr 5 09:08:53 2013 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 05 Apr 2013 10:08:53 -0600 Subject: [Bro] Command line bro on two interfaces Message-ID: <5f6b693d10530d3bf204b8d74e2f4315@localhost> Hey all...so here's my line: /usr/sbin/bro -i eth4 -i eth5 local Site::local_nets += { externalip/mask } I was thinking this was working, but alas, I see in my email packet captures that I think it's only listening on one interface as bro has missed an email that came by..I have no record of it anywhere. Is this not the right way to get this to work? I don't want to use broctl since the cpu usage is far too high. Thank you for any tips you may have. James From mcholste at gmail.com Fri Apr 5 09:41:08 2013 From: mcholste at gmail.com (Martin Holste) Date: Fri, 5 Apr 2013 11:41:08 -0500 Subject: [Bro] Snort Rules In-Reply-To: References: <36C06B73C5A9D845A5435F7AAE88802B40062794@Mail10.Corporate.net> Message-ID: Yes, use Suricata or Snort for Snort rule analysis, and combine the output there with Bro output. That will give you great data to supplement the IDS alerts and will be most efficient both in CPU time and human time. On Thu, Apr 4, 2013 at 10:33 AM, anthony kasza wrote: > As a fellow newbie, I feel the best answer is "don't". > To me, Bro seems better suited for flow like analysis, not byte-by-byte > packet analysis. > > > On Thu, Apr 4, 2013 at 10:25 AM, Parker, Jonathan E. wrote: > >> I?m a Bro newbie and I?ve been tasked to look at using Bro to perform >> analysis on Pcap files. We?d like to utilize some existing Snort rules >> in this analysis. A number of the Snort rules contain ?offset? and >> ?depth? parameters. I?d appreciate some advice on how to accomplish >> doing these Snort alerts in Bro. >> >> >> Thanks ? Jon >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130405/bf081224/attachment.html From Eric.Asselin at usherbrooke.ca Fri Apr 5 14:05:07 2013 From: Eric.Asselin at usherbrooke.ca (Eric Asselin) Date: Fri, 05 Apr 2013 23:05:07 +0200 Subject: [Bro] segmentation fault by adding an analyzer to port 80 In-Reply-To: References: <515D449A.6090705@usherbrooke.ca> Message-ID: <515F3C83.7070509@usherbrooke.ca> I try the same code but with the use of a signature to trigger my analyzer which worked, but again at the end I have the same "segmentation fault". If I derive the analyzer with TCP_ApplicationAnalyzer instead of just Analyzer and still use DeliverPacket(), the "seg fault" disappear. But in fact, I don't care about TCP connection stream, I just want the analyzer to be triggered regardless of the transport layer... and process the packet. Le 13-04-04 17:48, Siwek, Jonathan Luke a ?crit : > On Apr 4, 2013, at 4:15 AM, Eric Asselin > wrote: > >> When I force the anaylzer to be activated by modifying the DPM.cc, everything works well. But when I try to activate it by using a Bro script, like this: >> >> global foo_ports: set[port] = { 80/tcp } &redef; >> redef dpd_config += { [ANALYZER_MYPROTO] = [$ports = foo_ports] }; >> >> a segmentation fault appear at the end of the script. > I don't think there's a way to attach packet-based analyzers to TCP connections like that, so manually adding it in DPM.cc via TCP_Analyzer::AddChildPacketAnalyzer() for the ports you want may be the right thing. > > But if what you really wanted is a stream-based analyzer (it only sees the content after TCP reassembly), you can derive from TCP_ApplicationAnalyzer instead of Analyzer and override DeliverStream() instead of DeliverPacket(). Then the dpd_config redef you had should work. > > - Jon From sconzo at visiblerisk.com Mon Apr 8 11:33:31 2013 From: sconzo at visiblerisk.com (Mike Sconzo) Date: Mon, 8 Apr 2013 13:33:31 -0500 Subject: [Bro] signature question Message-ID: I've got 2 signatures that I've defined, and I'm trying to figure out the best way to see if both have fired an event in the same connection. Specifically, I'm curious if signature A hits before signature B. Below is some [pseudo]code to describe the scenario. I'm confused on what set_state() does (I just copied this from an example I found). I can see that both signatures hit, however the "if ( c$sig_a_seen )" is always False. It seems that the assignment of "c$sig_a_seen = T;" doesn't get added to the connection to get referenced in other events. What am I missing to get this value to persist through the entire connection? Thanks for the input. -=Mike export { redef record Info += {sig_a_seen: bool &default=F; }; } event signature_match(state: signature_state, msg: string, data: string) { local c = state$conn; set_state(c, F, F); local message = ""; if ( /^signature/ in state$sig_id ) { message = fmt("%s signature found", state$sig_id); print message c$sig_a_seen = T; } # if I do a print c, here then I can see that if it's signature A it's set T, but signature B is set F. if ( /signatureB/ in state$sig_id ) { if ( c$sig_a_seen ) { print "I've seen A followed by B"; } } -- cat ~/.bash_history > documentation.txt From nicolas.retrain at cea.fr Wed Apr 10 00:34:36 2013 From: nicolas.retrain at cea.fr (nicolas.retrain at cea.fr) Date: Wed, 10 Apr 2013 09:34:36 +0200 Subject: [Bro] login_success event Message-ID: <5165160C.60803@cea.fr> Hi, I apologize in advance for the rough English. Is someone has already successfully make the login_success (or login_failure) event work? The event always returns : user= , password=. Am i forgetting something? I have take a look into the Analyzer code. It seems that when the Login Analyzer get the first server command : "OpenBSD/i386 (oof) (ttyp2)" It tries to parse it like an authentication, and go in a LOGIN_STATE_CONFUSED. Then, when the client is authenticating the analyzer does not see it. I also asked for a tracker account, but have no return yet. I just want to share an Imap analyzer. Best regards, Nicolas From Eric.Asselin at usherbrooke.ca Wed Apr 10 01:05:28 2013 From: Eric.Asselin at usherbrooke.ca (Eric Asselin) Date: Wed, 10 Apr 2013 10:05:28 +0200 Subject: [Bro] segmentation fault by adding an analyzer to port 80 In-Reply-To: <515F3C83.7070509@usherbrooke.ca> References: <515D449A.6090705@usherbrooke.ca> <515F3C83.7070509@usherbrooke.ca> Message-ID: <51651D48.2000402@usherbrooke.ca> I think I've found why it doesn't work. When the inital tree of the DPD is build, the transport layer is hardcoded within a switch so further analyzer started by it's corresponding PIA (for UDP and TCP only) should derive from the corresponding class to work properly. So an analyzer itself can't act regardless of the transport layer. In this case, for an analyzer to support both transport layer protocols, it should derive TCP_ApplicationAnalyzer class and "extract or compute" packet like the DNS Analyzer does. Le 13-04-05 23:05, Eric Asselin a ?crit : > I try the same code but with the use of a signature to trigger my > analyzer which worked, but again at the end I have the same > "segmentation fault". If I derive the analyzer with > TCP_ApplicationAnalyzer instead of just Analyzer and still use > DeliverPacket(), the "seg fault" disappear. > > But in fact, I don't care about TCP connection stream, I just want the > analyzer to be triggered regardless of the transport layer... and > process the packet. > > > Le 13-04-04 17:48, Siwek, Jonathan Luke a ?crit : >> On Apr 4, 2013, at 4:15 AM, Eric Asselin >> wrote: >> >>> When I force the anaylzer to be activated by modifying the DPM.cc, everything works well. But when I try to activate it by using a Bro script, like this: >>> >>> global foo_ports: set[port] = { 80/tcp } &redef; >>> redef dpd_config += { [ANALYZER_MYPROTO] = [$ports = foo_ports] }; >>> >>> a segmentation fault appear at the end of the script. >> I don't think there's a way to attach packet-based analyzers to TCP connections like that, so manually adding it in DPM.cc via TCP_Analyzer::AddChildPacketAnalyzer() for the ports you want may be the right thing. >> >> But if what you really wanted is a stream-based analyzer (it only sees the content after TCP reassembly), you can derive from TCP_ApplicationAnalyzer instead of Analyzer and override DeliverStream() instead of DeliverPacket(). Then the dpd_config redef you had should work. >> >> - Jon > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From david at mandelberg.org Wed Apr 10 08:24:24 2013 From: david at mandelberg.org (David Mandelberg) Date: Wed, 10 Apr 2013 11:24:24 -0400 Subject: [Bro] more &synchronized questions Message-ID: <4f71618f672adfe8a8231d4289e82e5b@mail.mandelberg.org> Hi, How does &synchronized work with records? E.g., in the below code, is the effect of two cluster nodes simultaneously running ++foos[0]$foo similar to simultaneously running ++bars[0]? Or would there be a conflict causing foos[0]$foo to only be incremented once? type Foo: record { foo: count &default=0; } global foos: table[count] of Foo &default=[] &synchronized; global bars: table[count] of count &default=0 &synchronized; Also, does a "X += N" instruction propagate to all nodes as "increment by N" or as "set to X+N?" -- David Eric Mandelberg / dseomn http://david.mandelberg.org/ From robin at icir.org Wed Apr 10 13:37:53 2013 From: robin at icir.org (Robin Sommer) Date: Wed, 10 Apr 2013 13:37:53 -0700 Subject: [Bro] more &synchronized questions In-Reply-To: <4f71618f672adfe8a8231d4289e82e5b@mail.mandelberg.org> References: <4f71618f672adfe8a8231d4289e82e5b@mail.mandelberg.org> Message-ID: <20130410203753.GF58245@icir.org> On Wed, Apr 10, 2013 at 11:24 -0400, you wrote: > effect of two cluster nodes simultaneously running ++foos[0]$foo similar to > simultaneously running ++bars[0]? Yes, it's similar. The operation is propated for the specific record instance being modified, along with the record field where it applies to. > Also, does a "X += N" instruction propagate to all nodes as "increment by > N" or as "set to X+N?" The former, it's an actual increment operation. If multiple increments are sent, nodes may diverge briefly while not everybody has seen all updates yet, but eventually they'll all arrive at the same value. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From nicolas.retrain at cea.fr Wed Apr 10 23:52:18 2013 From: nicolas.retrain at cea.fr (nicolas.retrain at cea.fr) Date: Thu, 11 Apr 2013 08:52:18 +0200 Subject: [Bro] [PROVENANCE INTERNET] login_success event In-Reply-To: <16047_1365579428_r3A7b7s5008495_5165160C.60803@cea.fr> References: <16047_1365579428_r3A7b7s5008495_5165160C.60803@cea.fr> Message-ID: <51665DA2.50708@cea.fr> Le 10/04/2013 09:34, nicolas.retrain at cea.fr a ?crit : > Hi, > I apologize in advance for the rough English. > > Is someone has already successfully make the login_success (or > login_failure) event work? > The event always returns : user= , password=. > Am i forgetting something? If anyone wants the solution : I fix it out using a copy paste of the old login.bro script from 1.5 BRO IDS. Only the part to set : * skip_authentication * direct_login_prompts * login_prompts * login_non_failure_msgs * login_failure_msgs * login_success_msgs * login_timeout Now, login events work well :) > > > I have take a look into the Analyzer code. > It seems that when the Login Analyzer get the first server command : > "OpenBSD/i386 (oof) (ttyp2)" > It tries to parse it like an authentication, and go in a > LOGIN_STATE_CONFUSED. > Then, when the client is authenticating the analyzer does not see it. > > I also asked for a tracker account, but have no return yet. I just want > to share an Imap analyzer. Thanks for the account > > Best regards, > Nicolas > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130411/f0caae1e/attachment.html From jlay at slave-tothe-box.net Thu Apr 11 10:08:48 2013 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 11 Apr 2013 11:08:48 -0600 Subject: [Bro] Bro not seeing certain FTP transfers Message-ID: Topic says it...here's what I have from conn.log: 2013-04-08T06:00:25-0600 rTIHfQrsHgh x.x.x.x 26519 x.x.x.x 21 tcp ftp 22.117093 1141 4128 RSTR T 0 ShAdDaFr 111 5601 71 6972 (empty) And from my other logs: Apr 8 06:00:31 x.x.x.x FTP connection from interface:x.x.x.x/26519 to x.x.x.x/21, user Stored file filename ftp.log has no record at all of either the filename or the IP address. I am my own ISP and I peer with two other ISP's over two separate interfaces, meaning a packet can go out one interface, but come in the other. I'm running bro with: bro -i eth4 -i eth5 local Site::local_nets += { ipspace/mask, ipspace/mask } Any hints on where to look for a solution to this? I suspect I'm going to end up bridging these interfaces. Thank you. James From scastle at bouldercounty.org Thu Apr 11 10:25:43 2013 From: scastle at bouldercounty.org (Castle, Shane) Date: Thu, 11 Apr 2013 17:25:43 +0000 Subject: [Bro] Bro not seeing certain FTP transfers In-Reply-To: References: Message-ID: <21DD7C64179C9843B756C6DD491634DB5C8C287D@Mailbox1.boco.co.boulder.co.us> I wonder if it's because the conversation ended with an RST - the originator sent a FIN and got back RST. I assume the line you quoted corresponds with the actual transfer. -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of James Lay Sent: Thursday, April 11, 2013 11:09 To: bro at bro-ids.org Subject: [Bro] Bro not seeing certain FTP transfers Topic says it...here's what I have from conn.log: 2013-04-08T06:00:25-0600 rTIHfQrsHgh x.x.x.x 26519 x.x.x.x 21 tcp ftp 22.117093 1141 4128 RSTR T 0 ShAdDaFr 111 5601 71 6972 (empty) And from my other logs: Apr 8 06:00:31 x.x.x.x FTP connection from interface:x.x.x.x/26519 to x.x.x.x/21, user Stored file filename ftp.log has no record at all of either the filename or the IP address. I am my own ISP and I peer with two other ISP's over two separate interfaces, meaning a packet can go out one interface, but come in the other. I'm running bro with: bro -i eth4 -i eth5 local Site::local_nets += { ipspace/mask, ipspace/mask } Any hints on where to look for a solution to this? I suspect I'm going to end up bridging these interfaces. Thank you. James _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jlay at slave-tothe-box.net Thu Apr 11 10:30:07 2013 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 11 Apr 2013 11:30:07 -0600 Subject: [Bro] Bro not seeing certain FTP transfers In-Reply-To: <21DD7C64179C9843B756C6DD491634DB5C8C287D@Mailbox1.boco.co.boulder.co.us> References: <21DD7C64179C9843B756C6DD491634DB5C8C287D@Mailbox1.boco.co.boulder.co.us> Message-ID: <23ce0a0067c37afb470b543ad8998cd7@localhost> On 2013-04-11 11:25, Castle, Shane wrote: > I wonder if it's because the conversation ended with an RST - the > originator sent a FIN and got back RST. I assume the line you quoted > corresponds with the actual transfer. > > -- > Shane Castle > Data Security Mgr, Boulder County IT > > > -----Original Message----- > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of > James Lay > Sent: Thursday, April 11, 2013 11:09 > To: bro at bro-ids.org > Subject: [Bro] Bro not seeing certain FTP transfers > > Topic says it...here's what I have from conn.log: > > 2013-04-08T06:00:25-0600 rTIHfQrsHgh x.x.x.x 26519 > x.x.x.x 21 tcp ftp 22.117093 1141 4128 > RSTR > T 0 ShAdDaFr 111 5601 71 6972 > (empty) > > And from my other logs: > Apr 8 06:00:31 x.x.x.x FTP connection from interface:x.x.x.x/26519 > to > x.x.x.x/21, user Stored file filename > > ftp.log has no record at all of either the filename or the IP > address. > I am my own ISP and I peer with two other ISP's over two separate > interfaces, meaning a packet can go out one interface, but come in > the > other. I'm running bro with: > > bro -i eth4 -i eth5 local Site::local_nets += { ipspace/mask, > ipspace/mask } > > Any hints on where to look for a solution to this? I suspect I'm > going > to end up bridging these interfaces. Thank you. > > James Indeed it does. Thanks Shane. James From vern at icir.org Thu Apr 11 22:52:03 2013 From: vern at icir.org (Vern Paxson) Date: Thu, 11 Apr 2013 22:52:03 -0700 Subject: [Bro] login_success event In-Reply-To: <5165160C.60803@cea.fr> (Wed, 10 Apr 2013 09:34:36 +0200). Message-ID: <20130412055203.A3D942C4003@rock.ICSI.Berkeley.EDU> > Is someone has already successfully make the login_success (or > login_failure) event work? > The event always returns : user= , password=. (1) What version of Bro are you running? (2) Do you really have plaintext telnet/rlogin traffic? (That's what the analyzer focuses on. It's very old.) Vern From nicolas.retrain at cea.fr Fri Apr 12 00:46:25 2013 From: nicolas.retrain at cea.fr (nicolas.retrain at cea.fr) Date: Fri, 12 Apr 2013 09:46:25 +0200 Subject: [Bro] login_success event In-Reply-To: <20130412055203.A3D942C4003@rock.ICSI.Berkeley.EDU> References: <20130412055203.A3D942C4003@rock.ICSI.Berkeley.EDU> Message-ID: <5167BBD1.9090105@cea.fr> Le 12/04/2013 07:52, Vern Paxson a ?crit : >> Is someone has already successfully make the login_success (or >> login_failure) event work? >> The event always returns : user= , password=. > (1) What version of Bro are you running? I am using the 2.1 > > (2) Do you really have plaintext telnet/rlogin traffic? (That's what the > analyzer focuses on. It's very old.) yes, in fact I think the problem is: -the login analyzer try to match the coming command with some key words. If it matches, the analyzer raises an event or change the current state (or both). -these key words have to be defined in a bro script, in lists like "login_success_msgs", "login_failure_msgs"... -the login.bro doesn't exist anymore. So lists are not re-defined, and the matching method always return T (empty word match with every input line). -consequence : the analyzer raises the login_success event for the first input line (whatever it is), and turn the current state to AUTHENTICATE. -the login_success event leaves default user and password because it can not find theme into the input line. -the bro login_success event is launched at a bad time with user=, and password= -end of story :) It is a shame that so many bro scripts have disappeared after the version 1.5, what has happened? Nicolas > Vern From seth at icir.org Fri Apr 12 05:46:28 2013 From: seth at icir.org (Seth Hall) Date: Fri, 12 Apr 2013 08:46:28 -0400 Subject: [Bro] login_success event In-Reply-To: <5167BBD1.9090105@cea.fr> References: <20130412055203.A3D942C4003@rock.ICSI.Berkeley.EDU> <5167BBD1.9090105@cea.fr> Message-ID: <2D626A01-EC95-47F4-A06B-465280A148BD@icir.org> On Apr 12, 2013, at 3:46 AM, nicolas.retrain at cea.fr wrote: > It is a shame that so many bro scripts have disappeared after the > version 1.5, what has happened? I only had time and motivation to get as many ported as I did. The intent is to port many of the old ones over time, but it takes a lot of effort. If you would like to port the login script over to git master (creating a scripts/base/protocols/telnet directory perhaps?) I would certainly appreciate it. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From nicolas.retrain at cea.fr Fri Apr 12 06:12:36 2013 From: nicolas.retrain at cea.fr (nicolas.retrain at cea.fr) Date: Fri, 12 Apr 2013 15:12:36 +0200 Subject: [Bro] login_success event In-Reply-To: <2D626A01-EC95-47F4-A06B-465280A148BD@icir.org> References: <20130412055203.A3D942C4003@rock.ICSI.Berkeley.EDU> <5167BBD1.9090105@cea.fr> <2D626A01-EC95-47F4-A06B-465280A148BD@icir.org> Message-ID: <51680844.20501@cea.fr> Le 12/04/2013 14:46, Seth Hall a ?crit : > On Apr 12, 2013, at 3:46 AM, nicolas.retrain at cea.fr wrote: > >> It is a shame that so many bro scripts have disappeared after the >> version 1.5, what has happened? > > I only had time and motivation to get as many ported as I did. The intent is to port many of the old ones over time, but it takes a lot of effort. If you would like to port the login script over to git master (creating a scripts/base/protocols/telnet directory perhaps?) I would certainly appreciate it. :) > > .Seth I have quickly tried to make the whole script work, but I had a lot of dependency errors. For now my bro script only makes the analyzer work. I do not promise anything, but if I find time I will take a look once more. From david at mandelberg.org Mon Apr 15 13:19:56 2013 From: david at mandelberg.org (David Mandelberg) Date: Mon, 15 Apr 2013 16:19:56 -0400 Subject: [Bro] propagation of &mergeable Message-ID: Hi, In the below data structures, is there any way to make &mergeable apply to the set[port]s? If I make empty_smtp_servers &mergeable, I get an error. If I make data &mergeable, will it propagate all the way down to the smtp_servers tables? type SMTPServers: table[addr] of set[port]; const empty_set_of_ports: set[port] = {}; const empty_smtp_servers: SMTPServers = table() &default=copy(empty_set_of_ports) &write_expire=too_many_smtp_servers_expire; type HostData: record { ... ## SMTP servers the host has successfully connected to. smtp_servers: SMTPServers &default=copy(empty_smtp_servers); ... }; global data: table[addr] of HostData = {} &default=[] &synchronized &persistent; Just to be clear, what I want is the below two lines--run simultaneously on two nodes--to result in data[192.0.2.1]$smtp_servers[192.0.2.2] containing both 25/tcp and 587/tcp. add data[192.0.2.1]$smtp_servers[192.0.2.2][25/tcp]; add data[192.0.2.1]$smtp_servers[192.0.2.2][587/tcp]; -- David Eric Mandelberg / dseomn http://david.mandelberg.org/ From carlopmart at gmail.com Tue Apr 16 01:35:34 2013 From: carlopmart at gmail.com (C. L. Martinez) Date: Tue, 16 Apr 2013 08:35:34 +0000 Subject: [Bro] Is www.bro.org down?? Message-ID: -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130416/516d3bbe/attachment.html From carlopmart at gmail.com Tue Apr 16 02:02:33 2013 From: carlopmart at gmail.com (C. L. Martinez) Date: Tue, 16 Apr 2013 09:02:33 +0000 Subject: [Bro] Recommendations for install Bro under FreeBSD Message-ID: Hi all, Because we have made ??changes in our perimeter networks, I need to migrate my two Bro-IDS sensors based on CentOS to FreeBSD OS but it raises some doubts. a) What is the best option: FreeBSD 9.1 or 8.4?? b) Kernel options to monitor 1 GiB networks (ipc,buffers,etc)?? Many thanks for your help in advance. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130416/192a6927/attachment.html From bernhard at ICSI.Berkeley.EDU Tue Apr 16 05:12:05 2013 From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann) Date: Tue, 16 Apr 2013 05:12:05 -0700 Subject: [Bro] Is www.bro.org down?? In-Reply-To: References: Message-ID: bro.org looks fine for me? On Apr 16, 2013, at 1:35 AM, C. L. Martinez wrote: > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jessebowling at gmail.com Tue Apr 16 06:37:06 2013 From: jessebowling at gmail.com (Jesse Bowling) Date: Tue, 16 Apr 2013 09:37:06 -0400 Subject: [Bro] Take action on a notice? Message-ID: I'm regularly seeing PacketFilter::Dropped_Packets notices in my logs, which I believe are related to an issue with the version of PF_RING that I'm using. I'm in the midst of getting it upgraded, but in the meantime I'd love to be able to take an automated action on these notices (i.e., automatically restart the worker process that's dropping packets). I know all the parts for doing this are in the archives somewhere, but would someone mind giving me at least the high-level steps? My brogramming is nascent at best... Cheers, Jesse -- Jesse Bowling -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130416/ba854e32/attachment.html From JAzoff at albany.edu Tue Apr 16 06:49:43 2013 From: JAzoff at albany.edu (Justin Azoff) Date: Tue, 16 Apr 2013 09:49:43 -0400 Subject: [Bro] Take action on a notice? In-Reply-To: References: Message-ID: <20130416134943.GQ32624@datacomm.albany.edu> On Tue, Apr 16, 2013 at 09:37:06AM -0400, Jesse Bowling wrote: > I'm regularly seeing PacketFilter::Dropped_Packets notices in my logs, which I > believe are related to an issue with the version of PF_RING that I'm using. I'm > in the midst of getting it upgraded, but in the meantime I'd love to be able to > take an automated action on these notices (i.e., automatically restart the > worker process that's dropping packets). > > I know all the parts for doing this are in the archives somewhere, but would > someone mind giving me at least the high-level steps? My brogramming is nascent > at best... I use this, but it is for restarting workers that have completely stopped processing packets: */5 * * * * root sleep 5 ; grep -s -P "\t0\t0\t0" /usr/local/bro/logs/current/capture_loss.log && restart_bro restart_bro is just a script that uses broctl to restart bro and sends notifications. -- -- Justin Azoff -- Network Security & Performance Analyst From jessebowling at gmail.com Tue Apr 16 08:29:16 2013 From: jessebowling at gmail.com (Jesse Bowling) Date: Tue, 16 Apr 2013 11:29:16 -0400 Subject: [Bro] Take action on a notice? In-Reply-To: <20130416134943.GQ32624@datacomm.albany.edu> References: <20130416134943.GQ32624@datacomm.albany.edu> Message-ID: Hi Justin, In fact my workers have completely stopped processing packets as well; I'll look at this incantation and see if that works for me. Although I need to just solve the problem, I thought it might also be a good problem to force me into delving into the scripting language itself. :) Cheers, Jesse On Tue, Apr 16, 2013 at 9:49 AM, Justin Azoff wrote: > On Tue, Apr 16, 2013 at 09:37:06AM -0400, Jesse Bowling wrote: > > I'm regularly seeing PacketFilter::Dropped_Packets notices in my logs, > which I > > believe are related to an issue with the version of PF_RING that I'm > using. I'm > > in the midst of getting it upgraded, but in the meantime I'd love to be > able to > > take an automated action on these notices (i.e., automatically restart > the > > worker process that's dropping packets). > > > > I know all the parts for doing this are in the archives somewhere, but > would > > someone mind giving me at least the high-level steps? My brogramming is > nascent > > at best... > > I use this, but it is for restarting workers that have completely > stopped processing packets: > > */5 * * * * root sleep 5 ; grep -s -P "\t0\t0\t0" > /usr/local/bro/logs/current/capture_loss.log && restart_bro > > > restart_bro is just a script that uses broctl to restart bro and sends > notifications. > > > > -- > -- Justin Azoff > -- Network Security & Performance Analyst > -- Jesse Bowling -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130416/f70be4e7/attachment.html From david at mandelberg.org Tue Apr 16 15:03:29 2013 From: david at mandelberg.org (David Mandelberg) Date: Tue, 16 Apr 2013 18:03:29 -0400 Subject: [Bro] =?utf-8?q?connection=5Festablished_for_udp?= Message-ID: Is there an equivalent of event connection_established for UDP? I.e., an event that is raised once datagrams are seen in both directions for a given 4-tuple. -- David Eric Mandelberg / dseomn http://david.mandelberg.org/ From carlopmart at gmail.com Tue Apr 16 22:59:21 2013 From: carlopmart at gmail.com (C. L. Martinez) Date: Wed, 17 Apr 2013 05:59:21 +0000 Subject: [Bro] Recommendations for install Bro under FreeBSD In-Reply-To: References: Message-ID: On Tue, Apr 16, 2013 at 9:02 AM, C. L. Martinez wrote: > Hi all, > > Because we have made ??changes in our perimeter networks, I need to > migrate my two Bro-IDS sensors based on CentOS to FreeBSD OS but it raises > some doubts. > > a) What is the best option: FreeBSD 9.1 or 8.4?? > b) Kernel options to monitor 1 GiB networks (ipc,buffers,etc)?? > > Many thanks for your help in advance. > > Please, any tip?? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130417/26d45fc9/attachment.html From jlay at slave-tothe-box.net Wed Apr 17 05:36:23 2013 From: jlay at slave-tothe-box.net (James Lay) Date: Wed, 17 Apr 2013 06:36:23 -0600 Subject: [Bro] Bro not seeing certain FTP transfers In-Reply-To: <23ce0a0067c37afb470b543ad8998cd7@localhost> References: <21DD7C64179C9843B756C6DD491634DB5C8C287D@Mailbox1.boco.co.boulder.co.us> <23ce0a0067c37afb470b543ad8998cd7@localhost> Message-ID: <16BC0A65-FB20-4DD9-8721-85FF6A38B9D7@slave-tothe-box.net> No more thoughts on this all? James On Apr 11, 2013, at 11:30 AM, James Lay wrote: > On 2013-04-11 11:25, Castle, Shane wrote: >> I wonder if it's because the conversation ended with an RST - the >> originator sent a FIN and got back RST. I assume the line you quoted >> corresponds with the actual transfer. >> >> -- >> Shane Castle >> Data Security Mgr, Boulder County IT >> >> >> -----Original Message----- >> From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of >> James Lay >> Sent: Thursday, April 11, 2013 11:09 >> To: bro at bro-ids.org >> Subject: [Bro] Bro not seeing certain FTP transfers >> >> Topic says it...here's what I have from conn.log: >> >> 2013-04-08T06:00:25-0600 rTIHfQrsHgh x.x.x.x 26519 >> x.x.x.x 21 tcp ftp 22.117093 1141 4128 >> RSTR >> T 0 ShAdDaFr 111 5601 71 6972 >> (empty) >> >> And from my other logs: >> Apr 8 06:00:31 x.x.x.x FTP connection from interface:x.x.x.x/26519 >> to >> x.x.x.x/21, user Stored file filename >> >> ftp.log has no record at all of either the filename or the IP >> address. >> I am my own ISP and I peer with two other ISP's over two separate >> interfaces, meaning a packet can go out one interface, but come in >> the >> other. I'm running bro with: >> >> bro -i eth4 -i eth5 local Site::local_nets += { ipspace/mask, >> ipspace/mask } >> >> Any hints on where to look for a solution to this? I suspect I'm >> going >> to end up bridging these interfaces. Thank you. >> >> James > > Indeed it does. Thanks Shane. > > James > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From carlopmart at gmail.com Wed Apr 17 06:14:16 2013 From: carlopmart at gmail.com (C. L. Martinez) Date: Wed, 17 Apr 2013 13:14:16 +0000 Subject: [Bro] error: can't open base/init-bare.bro Message-ID: Hi all, After install bro-ids in a FreeBSD and configure cfg files and scripts that I need to load, when I try to execute 'broctl check', this error appears: error: can't open base/init-bare.bro Where is init-bare.bro?? There is not exists in my installation ... but exists in other bro ids linux based sensor ... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130417/f1773b17/attachment.html From carlopmart at gmail.com Wed Apr 17 06:16:59 2013 From: carlopmart at gmail.com (C. L. Martinez) Date: Wed, 17 Apr 2013 13:16:59 +0000 Subject: [Bro] error: can't open base/init-bare.bro In-Reply-To: References: Message-ID: On Wed, Apr 17, 2013 at 1:14 PM, C. L. Martinez wrote: > Hi all, > > After install bro-ids in a FreeBSD and configure cfg files and scripts > that I need to load, when I try to execute 'broctl check', this error > appears: > > error: can't open base/init-bare.bro > > Where is init-bare.bro?? There is not exists in my installation ... but > exists in other bro ids linux based sensor ... > Yep, sorry, exists: root at bsd01nm:/data/config/etc/bro/scripts/base# ls -la /data/config/etc/bro/scripts/base/init-bare.bro -rw-r--r-- 1 root wheel 98954 Aug 29 2012 /data/config/etc/bro/scripts/base/init-bare.bro Maybe is a problem with my configure options: "./configure --prefix=/opt/bro --enable-perftools --disable-ruby --conf-files-dir=/data/config/etc/bro/conf --scriptdir=/data/config/etc/bro/scripts"?? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130417/43f7da88/attachment.html From carlopmart at gmail.com Wed Apr 17 06:21:38 2013 From: carlopmart at gmail.com (C. L. Martinez) Date: Wed, 17 Apr 2013 13:21:38 +0000 Subject: [Bro] error: can't open base/init-bare.bro In-Reply-To: References: Message-ID: On Wed, Apr 17, 2013 at 1:16 PM, C. L. Martinez wrote: > > > > On Wed, Apr 17, 2013 at 1:14 PM, C. L. Martinez wrote: > >> Hi all, >> >> After install bro-ids in a FreeBSD and configure cfg files and scripts >> that I need to load, when I try to execute 'broctl check', this error >> appears: >> >> error: can't open base/init-bare.bro >> >> Where is init-bare.bro?? There is not exists in my installation ... but >> exists in other bro ids linux based sensor ... >> > > Yep, sorry, exists: > > root at bsd01nm:/data/config/etc/bro/scripts/base# ls -la > /data/config/etc/bro/scripts/base/init-bare.bro > -rw-r--r-- 1 root wheel 98954 Aug 29 2012 > /data/config/etc/bro/scripts/base/init-bare.bro > > Maybe is a problem with my configure options: "./configure > --prefix=/opt/bro --enable-perftools --disable-ruby > --conf-files-dir=/data/config/etc/bro/conf > --scriptdir=/data/config/etc/bro/scripts"?? > > Ok, problem is solved doing the correct softlinks: root at bsd01nm:/opt/bro/share/bro# ls -al total 12 drwxr-xr-x 3 root wheel 512 Apr 17 13:19 . drwxr-xr-x 4 root wheel 512 Apr 17 10:31 .. lrwxr-xr-x 1 root wheel 44 Apr 17 13:19 base -> ../../../../data/config/etc/bro/scripts/base drwxr-xr-x 2 root wheel 512 Apr 17 10:31 broctl lrwxr-xr-x 1 root wheel 46 Apr 17 13:19 policy -> ../../../../data/config/etc/bro/scripts/policy lrwxr-xr-x 1 root wheel 44 Apr 17 13:19 site -> ../../../../data/config/etc/bro/scripts/site root at plzfnsm01:/opt/bro/share/bro# but is this the correct solution?? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130417/38090332/attachment.html From pschmehl_lists at tx.rr.com Wed Apr 17 09:31:38 2013 From: pschmehl_lists at tx.rr.com (Paul Schmehl) Date: Wed, 17 Apr 2013 11:31:38 -0500 Subject: [Bro] Recommendations for install Bro under FreeBSD In-Reply-To: References: Message-ID: <7BB9184C83CB6B34A25155AD@localhost> --On April 17, 2013 5:59:21 AM +0000 "C. L. Martinez" wrote: > > On Tue, Apr 16, 2013 at 9:02 AM, C. L. Martinez > wrote: > > > Hi all, > > > ?Because we have made ??changes in our perimeter networks, I need to > migrate my two Bro-IDS sensors based on CentOS to FreeBSD OS but it > raises some doubts. > > > a) What is the best option: FreeBSD 9.1 or 8.4?? > b) Kernel options to monitor 1 GiB networks (ipc,buffers,etc)?? > > > Many thanks for your help in advance. > > > > > Please, any tip?? I would use FreeBSD 9.1. It has built in support for gpart (the default for the new installer) so it can handle disks larger than 2TB (which you will likely have) and has numerous improvements to the network stack as well as many of the network card drivers. I can't answer the second question since I use the defaults, but there are pages on the internet describing how to turn the kernel for high speed network performance - like this one - -- Paul Schmehl, Senior Infosec Analyst As if it wasn't already obvious, my opinions are my own and not those of my employer. ******************************************* "It is as useless to argue with those who have renounced the use of reason as to administer medication to the dead." Thomas Jefferson "There are some ideas so wrong that only a very intelligent person could believe in them." George Orwell From carlopmart at gmail.com Wed Apr 17 22:44:50 2013 From: carlopmart at gmail.com (C. L. Martinez) Date: Thu, 18 Apr 2013 05:44:50 +0000 Subject: [Bro] Recommendations for install Bro under FreeBSD In-Reply-To: <7BB9184C83CB6B34A25155AD@localhost> References: <7BB9184C83CB6B34A25155AD@localhost> Message-ID: On Wed, Apr 17, 2013 at 4:31 PM, Paul Schmehl wrote: > --On April 17, 2013 5:59:21 AM +0000 "C. L. Martinez" < > carlopmart at gmail.com> wrote: > > >> On Tue, Apr 16, 2013 at 9:02 AM, C. L. Martinez >> wrote: >> >> >> Hi all, >> >> >> Because we have made ??changes in our perimeter networks, I need to >> migrate my two Bro-IDS sensors based on CentOS to FreeBSD OS but it >> raises some doubts. >> >> >> a) What is the best option: FreeBSD 9.1 or 8.4?? >> b) Kernel options to monitor 1 GiB networks (ipc,buffers,etc)?? >> >> >> Many thanks for your help in advance. >> >> >> >> >> Please, any tip?? >> > > I would use FreeBSD 9.1. It has built in support for gpart (the default > for the new installer) so it can handle disks larger than 2TB (which you > will likely have) and has numerous improvements to the network stack as > well as many of the network card drivers. > > I can't answer the second question since I use the defaults, but there are > pages on the internet describing how to turn the kernel for high speed > network performance - like this one - NetworkPerformanceTuning > > > > Many thanks Paul. I am testing Fbsd 9.1 amd64 now and I will see how it goes ... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130418/8d52c65d/attachment.html From dn1nj4 at gmail.com Thu Apr 18 08:15:39 2013 From: dn1nj4 at gmail.com (. .) Date: Thu, 18 Apr 2013 11:15:39 -0400 Subject: [Bro] Email Link Extraction Message-ID: Hi list, Is there an easy way to extract links from emails in a method similar to smtp_entities processing of attachments? Thanks in advance! Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130418/73b1146c/attachment.html From jlay at slave-tothe-box.net Thu Apr 18 08:31:16 2013 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 18 Apr 2013 09:31:16 -0600 Subject: [Bro] Email Link Extraction In-Reply-To: References: Message-ID: On 2013-04-18 09:15, . . wrote: > Hi list, > > Is there an easy way to extract links from emails in a method similar > to smtp_entities processing of attachments? > > Thanks in advance! > Jason Yea I'll second that...email packet captures make finding links a challenge as quoted emails split the links..this would really help to correlate a user click to actual email in a fraction of the time. Thank you. James From jessebowling at gmail.com Thu Apr 18 08:50:30 2013 From: jessebowling at gmail.com (Jesse Bowling) Date: Thu, 18 Apr 2013 11:50:30 -0400 Subject: [Bro] Email Link Extraction In-Reply-To: References: Message-ID: +1 on that. On Thu, Apr 18, 2013 at 11:31 AM, James Lay wrote: > On 2013-04-18 09:15, . . wrote: > > Hi list, > > > > Is there an easy way to extract links from emails in a method similar > > to smtp_entities processing of attachments? > > > > Thanks in advance! > > Jason > > Yea I'll second that...email packet captures make finding links a > challenge as quoted emails split the links..this would really help to > correlate a user click to actual email in a fraction of the time. Thank > you. > > James > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Jesse Bowling -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130418/00f18753/attachment.html From scastle at bouldercounty.org Thu Apr 18 08:58:09 2013 From: scastle at bouldercounty.org (Castle, Shane) Date: Thu, 18 Apr 2013 15:58:09 +0000 Subject: [Bro] Email Link Extraction In-Reply-To: References: Message-ID: <21DD7C64179C9843B756C6DD491634DB5C8D006E@Mailbox1.boco.co.boulder.co.us> At first glance this seems like all it needs is an appropriate regex. But then consider: any string containing both "." and "/" might be a candidate. (Actually, just a string containing "." with no space around it.) So, this might range from the full regex to detect '.+' to just '\s.+\..+\s' (Perl regex used). I'd welcome attempts to work on this. And, even if the result does not catch everything, if it gets anything at all it'd be better than what we have now. -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of James Lay Sent: Thursday, April 18, 2013 09:31 To: bro at bro.org Subject: Re: [Bro] Email Link Extraction On 2013-04-18 09:15, . . wrote: > Hi list, > > Is there an easy way to extract links from emails in a method similar > to smtp_entities processing of attachments? > > Thanks in advance! > Jason Yea I'll second that...email packet captures make finding links a challenge as quoted emails split the links..this would really help to correlate a user click to actual email in a fraction of the time. Thank you. James _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From david at mandelberg.org Thu Apr 18 09:05:25 2013 From: david at mandelberg.org (David Mandelberg) Date: Thu, 18 Apr 2013 12:05:25 -0400 Subject: [Bro] =?utf-8?q?x_=3D_x_and_=26synchronized?= Message-ID: How does the workaround I found in http://tracker.bro.org/bro/ticket/980#comment:1 interact with &synchronized? In the below case, can the last line undo the line before it? node-1> foo[0]$x += 42; node-2> foo[0]$x += 42; node-1> foo[0] = foo[0]; Alternatively, is there a better workaround or will that bug be fixed soon? -- David Eric Mandelberg / dseomn http://david.mandelberg.org/ From jlay at slave-tothe-box.net Thu Apr 18 09:24:25 2013 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 18 Apr 2013 10:24:25 -0600 Subject: [Bro] Email Link Extraction In-Reply-To: <21DD7C64179C9843B756C6DD491634DB5C8D006E@Mailbox1.boco.co.boulder.co.us> References: <21DD7C64179C9843B756C6DD491634DB5C8D006E@Mailbox1.boco.co.boulder.co.us> Message-ID: <5611aa119b81f6c3379603ce822d3d5d@localhost> On 2013-04-18 09:58, Castle, Shane wrote: > At first glance this seems like all it needs is an appropriate regex. > But then consider: any string containing both "." and "/" might be a > candidate. (Actually, just a string containing "." with no space > around it.) > > So, this might range from the full regex to detect ' href=".+">.+' to just '\s.+\..+\s' (Perl regex used). > > I'd welcome attempts to work on this. And, even if the result does > not catch everything, if it gets anything at all it'd be better than > what we have now. > > -- > Shane Castle > Data Security Mgr, Boulder County IT Here's a special just from this morning (xx's added): Hello, Please view the document i uploaded for you using Google docs. *VIEW HERE *just sign in with your email to view the document its very important Regards And the quoted-printable content (it's a hoot): VIEW=A0 guessing that some normalization will be needed to nuke the 3D's and possible ='s within links, or just match on "http://" and call it good. Hope the above shows up right. James From seth at icir.org Thu Apr 18 11:27:43 2013 From: seth at icir.org (Seth Hall) Date: Thu, 18 Apr 2013 14:27:43 -0400 Subject: [Bro] Email Link Extraction In-Reply-To: References: Message-ID: <367AE717-F777-4DD8-8B69-954A8275788A@icir.org> On Apr 18, 2013, at 11:31 AM, James Lay wrote: > Yea I'll second that...email packet captures make finding links a > challenge as quoted emails split the link This is far from perfect due to the reason you pointed out, but it's a start and this code snippet is from the next release of Bro (you just call find_all_urls_without_scheme with the string that you want to extract urls from): const url_regex = /^([a-zA-Z\-]{3,5})(:\/\/[^\/?#"'\r\n><]*)([^?#"'\r\n><]*)([^[:blank:]\r\n"'><]*|\??[^"'\r\n><]*)/ &redef; ## Extracts URLs discovered in arbitrary text. function find_all_urls(s: string): string_set { return find_all(s, url_regex); } ## Extracts URLs discovered in arbitrary text without ## the URL scheme included. function find_all_urls_without_scheme(s: string): string_set { local urls = find_all_urls(s); local return_urls: set[string] = set(); for ( url in urls ) { local no_scheme = sub(url, /^([a-zA-Z\-]{3,5})(:\/\/)/, ""); add return_urls[no_scheme]; } return return_urls; } .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jlay at slave-tothe-box.net Thu Apr 18 12:13:18 2013 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 18 Apr 2013 13:13:18 -0600 Subject: [Bro] Email Link Extraction In-Reply-To: <367AE717-F777-4DD8-8B69-954A8275788A@icir.org> References: <367AE717-F777-4DD8-8B69-954A8275788A@icir.org> Message-ID: On 2013-04-18 12:27, Seth Hall wrote: > On Apr 18, 2013, at 11:31 AM, James Lay > wrote: > >> Yea I'll second that...email packet captures make finding links a >> challenge as quoted emails split the link > > This is far from perfect due to the reason you pointed out, but it's > a start and this code snippet is from the next release of Bro (you > just call find_all_urls_without_scheme with the string that you want > to extract urls from): > > > const url_regex = > > /^([a-zA-Z\-]{3,5})(:\/\/[^\/?#"'\r\n><]*)([^?#"'\r\n><]*)([^[:blank:]\r\n"'><]*|\??[^"'\r\n><]*)/ > &redef; > > ## Extracts URLs discovered in arbitrary text. > function find_all_urls(s: string): string_set > { > return find_all(s, url_regex); > } > > ## Extracts URLs discovered in arbitrary text without > ## the URL scheme included. > function find_all_urls_without_scheme(s: string): string_set > { > local urls = find_all_urls(s); > local return_urls: set[string] = set(); > for ( url in urls ) > { > local no_scheme = sub(url, /^([a-zA-Z\-]{3,5})(:\/\/)/, ""); > add return_urls[no_scheme]; > } > > return return_urls; > } > > > > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ Thanks Seth...as I'm still horrifically newb with Bro, I'm guessing the above can go in local.bro? Thank you. James From seth at icir.org Fri Apr 19 08:04:29 2013 From: seth at icir.org (Seth Hall) Date: Fri, 19 Apr 2013 11:04:29 -0400 Subject: [Bro] x = x and &synchronized In-Reply-To: References: Message-ID: On Apr 18, 2013, at 12:05 PM, David Mandelberg wrote: > Alternatively, is there a better workaround or will that bug be fixed > soon? I've been moving away from the &synchronized attribute for quite a while and moving toward manual, explicit data sharing through events. My experience with using it was like you're discovering, there are so many complicated edge cases that we don't even reasonably have a good mechanism of handing well. At some point we may have better primitives for synchronizing data across nodes easily, but we don't have anything concrete in progress yet. Just something to think about. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From david at mandelberg.org Fri Apr 19 08:34:01 2013 From: david at mandelberg.org (David Mandelberg) Date: Fri, 19 Apr 2013 11:34:01 -0400 Subject: [Bro] x = x and &synchronized In-Reply-To: References: Message-ID: <1366385641.3473.1.camel@solaria> On Fri, 2013-04-19 at 11:04 -0400, Seth Hall wrote: > I've been moving away from the &synchronized attribute for quite a while and moving toward manual, explicit data sharing through events. Ok. Can you point me to an example script that does explicit data sharing? -- David Eric Mandelberg / dseomn http://david.mandelberg.org/ Fri Apr 19 11:32:12 EDT 2013 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130419/1b6b2267/attachment.bin From seth at icir.org Fri Apr 19 08:42:19 2013 From: seth at icir.org (Seth Hall) Date: Fri, 19 Apr 2013 11:42:19 -0400 Subject: [Bro] x = x and &synchronized In-Reply-To: <1366385641.3473.1.camel@solaria> References: <1366385641.3473.1.camel@solaria> Message-ID: On Apr 19, 2013, at 11:34 AM, David Mandelberg wrote: > Ok. Can you point me to an example script that does explicit data > sharing?  The metrics framework in 2.1 does it. In 2.2 the intel framework and the sumstats framework does it (sumstats is the renamed and expanded metrics framework). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130419/34f72bf6/attachment.bin From kim at blackcatsec.net Fri Apr 19 08:44:50 2013 From: kim at blackcatsec.net (Kim Halavakoski) Date: Fri, 19 Apr 2013 18:44:50 +0300 Subject: [Bro] Connection summary email Message-ID: <94FB64C1-56E3-4887-A461-1727B7BA5DAC@blackcatsec.net> Hello Bro-folks! I have just setup bro and are getting aquainted with Bro. Bro feels very good and the information that can be gathered eith Bro is impressive. I have one issue though that is simple and easy but haven't had the time to figure it out. I am receiving these connection summary emails via the Bro cron feature. The mail however complains about time command that cannot be found: [Bro] Connection summary from 16:00:00-17:00:00 nice: which:: No such file or directory -- [Automatically generated.] The "time" command is not available as a separate package under RedHat so I am curious about what that script does, where it is and how I could modify it to work without "time"? Best regards, Kim Halavakoski Sent from my mobile device, excuse my clawfingerness! PGP S#: 0BFA A910 9AA7 94A5 A323 53F5 4151 4CE4 33BE 35FA http://www.blackcatsec.net -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130419/c9faa571/attachment.html From jones at tacc.utexas.edu Fri Apr 19 08:59:09 2013 From: jones at tacc.utexas.edu (William Jones) Date: Fri, 19 Apr 2013 15:59:09 +0000 Subject: [Bro] A explaination of In-Reply-To: <5138ADD7-0004405F@straylight.g-clef.net> References: <5137B696-00043FBD@straylight.g-clef.net> <5138ADD7-0004405F@straylight.g-clef.net> Message-ID: # Someone's going to have to explain what this is and add a field to the Info record. #info$util = 100.0*((res$user_time + res$system_time) - (last_res$user_time + last_res$system_time))/(now-last_ts); This is just an estimate of the cpu utilization to process the packets. If you seeing lost packets and this number is a 100% then you run out of the processing power for that worker. Most of my dropped packets are due to running out of cpu. This mostly happens during denial of serves attach when some is try to use TACC resole to generate icmp replies. Were carefully not generate icmp messages in most cases. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130419/1793acaf/attachment.html From robin at icir.org Fri Apr 19 12:34:00 2013 From: robin at icir.org (Robin Sommer) Date: Fri, 19 Apr 2013 12:34:00 -0700 Subject: [Bro] x = x and &synchronized In-Reply-To: References: Message-ID: <20130419193400.GJ57001@icir.org> On Thu, Apr 18, 2013 at 12:05 -0400, you wrote: > node-1> foo[0]$x += 42; > node-2> foo[0]$x += 42; > node-1> foo[0] = foo[0]; Yes, the 3rd line can undo the 2nd. For more consistency we'd need to have some explicit synchronization/locking across nodes, which we (deliberately) don't do. &synchronized provides a best-effort solution, but it's not great if you need strict semantics. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From robin at icir.org Fri Apr 19 12:39:04 2013 From: robin at icir.org (Robin Sommer) Date: Fri, 19 Apr 2013 12:39:04 -0700 Subject: [Bro] connection_established for udp In-Reply-To: References: Message-ID: <20130419193904.GL57001@icir.org> On Tue, Apr 16, 2013 at 18:03 -0400, you wrote: > Is there an equivalent of event connection_established for UDP? > I.e., an event that is raised once datagrams are seen in both > directions for a given 4-tuple. No, and I wouldn't be sure about the semantics of seeing something in both direction as that's not required for UDP. What are you trying to do? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From david at mandelberg.org Fri Apr 19 12:51:07 2013 From: david at mandelberg.org (David Mandelberg) Date: Fri, 19 Apr 2013 15:51:07 -0400 Subject: [Bro] x = x and &synchronized In-Reply-To: <20130419193400.GJ57001@icir.org> References: <20130419193400.GJ57001@icir.org> Message-ID: <1366401067.3473.33.camel@solaria> On Fri, 2013-04-19 at 12:34 -0700, Robin Sommer wrote: > For more consistency we'd need to > have some explicit synchronization/locking across nodes, It is possible to get more consistency without locking if node-1 sends a message like "foo[0] = foo[0]" (instead of the "foo[0] = " that I assume it does now) to all other nodes and they execute that locally. This adds complexity that may not be worthwhile though. Anyway, after the discussion of #981 on bro-dev, I realized that this workaround is also generally incorrect. Maybe #980 and #981 should be merged since they're caused by the same thing that I initially misunderstood? -- David Eric Mandelberg / dseomn http://david.mandelberg.org/ Fri Apr 19 15:42:54 EDT 2013 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130419/cc4de13a/attachment.bin From david at mandelberg.org Fri Apr 19 13:12:43 2013 From: david at mandelberg.org (David Mandelberg) Date: Fri, 19 Apr 2013 16:12:43 -0400 Subject: [Bro] connection_established for udp In-Reply-To: <20130419193904.GL57001@icir.org> References: <20130419193904.GL57001@icir.org> Message-ID: <1366402363.3473.45.camel@solaria> On Fri, 2013-04-19 at 12:39 -0700, Robin Sommer wrote: > On Tue, Apr 16, 2013 at 18:03 -0400, you wrote: > > > Is there an equivalent of event connection_established for UDP? > > I.e., an event that is raised once datagrams are seen in both > > directions for a given 4-tuple. > > No, and I wouldn't be sure about the semantics of seeing something in > both direction as that's not required for UDP. What are you trying to > do? I'd like to know whenever a new server appears on a network. Ideally, this would be whenever a host calls listen() on a connection-oriented socket or bind() on a datagram socket. In practice, it seems to work well enough to track the responding hosts and ports of established connections or datagram pseudo-connections where the "server" has responded. This doesn't work for UDP servers that don't respond using the same 4-tuple, but it works for DNS and a few other common UDP server types. -- David Eric Mandelberg / dseomn http://david.mandelberg.org/ Fri Apr 19 16:02:16 EDT 2013 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130419/8aba9f72/attachment.bin From dnthayer at illinois.edu Fri Apr 19 13:07:22 2013 From: dnthayer at illinois.edu (Daniel Thayer) Date: Fri, 19 Apr 2013 15:07:22 -0500 Subject: [Bro] Connection summary email In-Reply-To: <94FB64C1-56E3-4887-A461-1727B7BA5DAC@blackcatsec.net> References: <94FB64C1-56E3-4887-A461-1727B7BA5DAC@blackcatsec.net> Message-ID: <5171A3FA.9010702@illinois.edu> The time command is used by broctl to time how long it takes to generate a connection summary report (the output of the time command is visible at the end of each connection summary report). There is currently a bug in broctl that prevents connection summary reports from being generated if the time command is not available (this bug should be fixed in the next release). On RHEL5 and RHEL6, the "time" command is contained in the "time" rpm. So, to fix this, you could install the "time" rpm, then do a "broctl install" to update the broctl configuration so that it knows where the time command is located. If you are unable to install "time", then an alternative solution is to patch the broctl source so that connection summary reports can be generated whether or not "time" is installed (let me know if you want instructions on how to do this). -Daniel On 04/19/2013 10:44 AM, Kim Halavakoski wrote: > Hello Bro-folks! > > I have just setup bro and are getting aquainted with Bro. Bro feels very > good and the information that can be gathered eith Bro is impressive. > > I have one issue though that is simple and easy but haven't had the time > to figure it out. > > I am receiving these connection summary emails via the Bro cron feature. > The mail however complains about time command that cannot be found: > > [Bro] Connection summary from 16:00:00-17:00:00 > > nice: which:: No such file or directory > > -- > [Automatically generated.] > > > The "time" command is not available as a separate package under RedHat > so I am curious about what that script does, where it is and how I could > modify it to work without "time"? > > Best regards, > > Kim Halavakoski > > Sent from my mobile device, excuse my clawfingerness! > > PGP S#: 0BFA A910 9AA7 94A5 A323 53F5 4151 4CE4 33BE 35FA > http://www.blackcatsec.net > > > From jsiwek at illinois.edu Fri Apr 19 13:33:55 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Fri, 19 Apr 2013 20:33:55 +0000 Subject: [Bro] connection_established for udp In-Reply-To: <1366402363.3473.45.camel@solaria> References: <20130419193904.GL57001@icir.org> <1366402363.3473.45.camel@solaria> Message-ID: > I'd like to know whenever a new server appears on a network. Ideally, > this would be whenever a host calls listen() on a connection-oriented > socket or bind() on a datagram socket. In practice, it seems to work > well enough to track the responding hosts and ports of established > connections or datagram pseudo-connections where the "server" has > responded. This doesn't work for UDP servers that don't respond using > the same 4-tuple, but it works for DNS and a few other common UDP server > types. If you can wait until the internal state of UDP "connections" in Bro times out due to inactivity (default of "udp_inactivity_timeout" variable is 1 min), would it work to handle the "connection_state_remove" event and check for a non-zero c$resp$size ? - Jon From david at mandelberg.org Fri Apr 19 13:46:32 2013 From: david at mandelberg.org (David Mandelberg) Date: Fri, 19 Apr 2013 16:46:32 -0400 Subject: [Bro] connection_established for udp In-Reply-To: References: <20130419193904.GL57001@icir.org> <1366402363.3473.45.camel@solaria> Message-ID: <1366404392.3473.51.camel@solaria> On Fri, 2013-04-19 at 20:33 +0000, Siwek, Jonathan Luke wrote: > > I'd like to know whenever a new server appears on a network. Ideally, > > this would be whenever a host calls listen() on a connection-oriented > > socket or bind() on a datagram socket. In practice, it seems to work > > well enough to track the responding hosts and ports of established > > connections or datagram pseudo-connections where the "server" has > > responded. This doesn't work for UDP servers that don't respond using > > the same 4-tuple, but it works for DNS and a few other common UDP server > > types. > > If you can wait until the internal state of UDP "connections" in Bro times out due to inactivity (default of "udp_inactivity_timeout" variable is 1 min), would it work to handle the "connection_state_remove" event and check for a non-zero c$resp$size ? That would work ok for a protocol like DNS, but not as well when the 4-tuples are long-lived. It also makes IDS evasion really easy for a custom protocol running over UDP. -- David Eric Mandelberg / dseomn http://david.mandelberg.org/ Fri Apr 19 16:37:38 EDT 2013 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130419/eb4ec557/attachment.bin From Olivier.Bonaventure at uclouvain.be Fri Apr 19 13:57:42 2013 From: Olivier.Bonaventure at uclouvain.be (Olivier Bonaventure) Date: Fri, 19 Apr 2013 22:57:42 +0200 Subject: [Bro] Multipath TCP and bro Message-ID: <5171AFC6.9040902@uclouvain.be> Hello, Multipath TCP is a recent extension to TCP (RFC6824) that allows the utilisation of multiple paths for a single TCP connection. This TCP extension has several use cases including smartphones that could use WiFi at 3G for the same TCP connection, dual-stack hosts that would use IPv6 and IPv4 packets for the same TCP connection, ... These benefits could also impact the middleboxes, such as those running bro, that usually expect to receive all packets from a given TCP bytestream inside a single TCP connection identified by the classical four-tuple. This is not necessarily always the case anymore with Multipath TCP. The utilisation of Multipath TCP could have an impact on the algorithms used by bro but also on bro deployments. An implementation of Multipath TCP exists in the Linux kernel, but it is not yet part of the official kernel. To enable middlebox developpers and users to test the interoperability between Multipath TCP and their middlebox, we have developped a special test suite. This test suite is implemented as a set of scritps running on a virtualbox image containing a modified Multipath TCP kernel. This image interact with Multipath TCP capable servers and uses applications like http, ftp, scp on top of Multipath TCP. To ease the debugging of possible problems, all packets sent and received by the test suite are collected and the trace is available at the end of the test. Usually, the test runs in about 15 minutes. You can download it from http://multipath-tcp.org/pmwiki.php?n=Users.AboutMeasures It currently runs on Linux and Mac. Best regards, Olivier Bonaventure -- INL, ICTEAM, UCLouvain, Belgium, http://inl.info.ucl.ac.be From jsiwek at illinois.edu Fri Apr 19 14:05:49 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Fri, 19 Apr 2013 21:05:49 +0000 Subject: [Bro] connection_established for udp In-Reply-To: <1366404392.3473.51.camel@solaria> References: <20130419193904.GL57001@icir.org> <1366402363.3473.45.camel@solaria> <1366404392.3473.51.camel@solaria> Message-ID: >> If you can wait until the internal state of UDP "connections" in Bro times out due to inactivity (default of "udp_inactivity_timeout" variable is 1 min), would it work to handle the "connection_state_remove" event and check for a non-zero c$resp$size ? > > That would work ok for a protocol like DNS, but not as well when the > 4-tuples are long-lived. It also makes IDS evasion really easy for a > custom protocol running over UDP. How about also handling "new_connection" and polling that connection periodically for a non-zero c$resp$size ? That may be more expensive, though. The base/protocols/conn/polling.bro script provides a generic way to do polling via events, and an example use of it can be found in base/protocols/ftp/gridftp.bro. - Jon From robin at icir.org Fri Apr 19 14:24:37 2013 From: robin at icir.org (Robin Sommer) Date: Fri, 19 Apr 2013 14:24:37 -0700 Subject: [Bro] connection_established for udp In-Reply-To: References: <20130419193904.GL57001@icir.org> <1366402363.3473.45.camel@solaria> <1366404392.3473.51.camel@solaria> Message-ID: <20130419212437.GE64806@icir.org> On Fri, Apr 19, 2013 at 21:05 +0000, Jonathan Siwek wrote: > How about also handling "new_connection" and polling that connection > periodically for a non-zero c$resp$size ? That may be more expensive, > though. It would actually be useful to have a "new_udp_connection" (or so) event that triggers only for new UDP flows. I don't think I'd tie the bi-directionality in there, but that along with the polling could work. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From kim at blackcatsec.net Sat Apr 20 03:52:24 2013 From: kim at blackcatsec.net (Kim Halavakoski) Date: Sat, 20 Apr 2013 13:52:24 +0300 Subject: [Bro] Weird stuff in weird.log? Message-ID: <51727368.6000604@blackcatsec.net> Hi, testing out bro and here is my next question to the bro-list. I am seeing some weird stuff in weird.log(yeah! doh!).I am getting alot of stuff like nappropriate_FIN, data_before_established, possible_split_routing, active_connection_reuse etc. Where can I control / diasable these weird-messages and control what gets logged? I suspect that it is due to the fact that I am spanning multiple VLANs that Bro sees, with traffic both before and after loabalancers and NATs etc. so it kind-of sees the whole chain of packets from outside the firewall, before / after loadbalancer behind firewall and finally the traffic behind the loadbalancers/firewalls...would that in some way explain the weird.log stuff shown here? 1366445558.221865 sMz3SRMvJ1f 57597 30390 inappropriate_FIN - F bro 1366445558.778349 28ucePIK0f5 389 47584 data_before_established - F bro 1366445558.803882 C1cexKcJdhd 389 34297 data_before_established - F bro 1366445558.823156 C1cexKcJdhd 389 34297 inappropriate_FIN - F bro 1366445559.364974 C2jakgfKyh6 389 34300 data_before_established - F bro 1366445559.368926 M4MZKG1MEtk 41978 5666 possible_split_routing - F bro 1366445559.368936 M4MZKG1MEtk 41978 5666 data_before_established - F bro 1366445559.369958 II9kgjUU29i 33842 5666 possible_split_routing - F bro 1366445559.370258 II9kgjUU29i 33842 5666 data_before_established - F bro 1366445559.458949 C2jakgfKyh6 389 34300 inappropriate_FIN - F bro 1366445559.590081 xzP6usa0oXd 9997 63028 inappropriate_FIN - F bro 1366445559.590095 5XYkXUNzYFd 9997 47501 inappropriate_FIN - F bro 1366445559.775448 6wzuYevBpef 37270 30310 possible_split_routing - F bro 1366445559.775824 6wzuYevBpef 37270 30310 data_before_established - F bro 1366445559.779481 6wzuYevBpef 37270 30310 inappropriate_FIN - F bro 1366445560.873320 NM2tVOQW5Nb 60618 22 active_connection_reuse - F bro 1366445560.874857 P5qPMPaCUuf 60618 22 possible_split_routing - F bro 1366445560.875449 P5qPMPaCUuf 60618 22 inappropriate_FIN - F bro 1366445561.235377 n4cK9P7xS14 47275 30310 possible_split_routing - F bro 1366445561.235455 n4cK9P7xS14 47275 30310 data_before_established - F bro 1366445561.238828 n4cK9P7xS14 47275 30310 inappropriate_FIN - F bro 1366445562.716152 b7hwy5hPMx3 34664 30390 possible_split_routing - F bro 1366445562.716642 b7hwy5hPMx3 34664 30390 data_before_established - F bro 1366445562.720299 b7hwy5hPMx3 34664 30390 inappropriate_FIN - F bro 1366445563.029829 3JCdOtzkBL1 1523 35631 data_before_established - F bro 1366445563.030430 3JCdOtzkBL1 1523 35631 active_connection_reuse - F bro 1366445563.039064 PuQti8prAgc 35631 1523 data_after_reset - F bro 1366445563.244717 yUB4KmfF8hi 57618 30390 possible_split_routing - F bro 1366445563.245336 yUB4KmfF8hi 57618 30390 data_before_established - F bro 1366445563.250505 yUB4KmfF8hi 57618 30390 inappropriate_FIN - F bro 1366445563.698203 UupX3wx5GT9 52198 22 active_connection_reuse - F bro 1366445564.737450 AnRairldCP3 37289 30310 possible_split_routing - F bro 1366445564.737835 AnRairldCP3 37289 30310 data_before_established - F bro 1366445564.742235 AnRairldCP3 37289 30310 inappropriate_FIN - F bro 1366445565.118886 ITn36vbahlj 52576 5666 possible_split_routing - F bro 1366445565.119184 ITn36vbahlj 52576 5666 data_before_established - F bro 1366445565.148984 shxX8UNt1a3 59902 5666 possible_split_routing - F bro 1366445565.149235 shxX8UNt1a3 59902 5666 data_before_established - F bro 1366445565.885159 Dj3WAdR8v5b 60728 22 active_connection_reuse - F bro 1366445565.885595 rDy60hYCOdg 60728 22 possible_split_routing - F bro 1366445565.886074 rDy60hYCOdg 60728 22 inappropriate_FIN - F bro 1366445566.158385 5B8TeJKxXi3 47294 30310 possible_split_routing - F bro 1366445566.158716 5B8TeJKxXi3 47294 30310 data_before_established - F bro 1366445566.162550 5B8TeJKxXi3 47294 30310 inappropriate_FIN - F bro 1366445567.751813 vC30wQV7K7j 34685 30390 possible_split_routing - F bro 1366445567.752999 vC30wQV7K7j 34685 30390 data_before_established - F bro 1366445567.757998 vC30wQV7K7j 34685 30390 inappropriate_FIN - F bro 1366445568.176397 RYxS3He4jqa 57637 30390 possible_split_routing - F bro 1366445568.176683 RYxS3He4jqa 57637 30390 data_before_established - F bro 1366445568.180986 RYxS3He4jqa 57637 30390 inappropriate_FIN - F bro 1366445568.779173 xatDJTfRKT5 9997 32941 inappropriate_FIN - F bro 1366445568.779185 iShp45vkZB5 9997 47413 inappropriate_FIN - F bro 1366445569.125537 Ibw0hED2Fob 36384 9997 possible_split_routing - F bro 1366445569.125548 uUEy466ylTf 47952 9997 possible_split_routing - F bro 1366445569.692394 eXp0EbWU2Ka 63605 9997 possible_split_routing - F bro 1366445569.692469 c12zuBPumRj 14686 9997 possible_split_routing - F bro 1366445569.763472 sA21Rb4LVz5 37310 30310 possible_split_routing - F bro 1366445569.764000 sA21Rb4LVz5 37310 30310 data_before_established - F bro 1366445569.767989 sA21Rb4LVz5 37310 30310 inappropriate_FIN - F bro 1366445570.908940 GD1tiVPOnRf 60836 22 active_connection_reuse - F bro 1366445570.909404 kveiMvxFbZ2 60836 22 possible_split_routing - F bro 1366445570.909679 kveiMvxFbZ2 60836 22 inappropriate_FIN - F bro 1366445571.216116 GmMDwe1BuF2 47317 30310 possible_split_routing - F bro 1366445571.216465 GmMDwe1BuF2 47317 30310 data_before_established - F bro 1366445571.220271 GmMDwe1BuF2 47317 30310 inappropriate_FIN - F bro 1366445572.701760 3nU4LDW2lm3 34704 30390 possible_split_routing - F bro 1366445572.702540 3nU4LDW2lm3 34704 30390 data_before_established - F bro 1366445572.710165 3nU4LDW2lm3 34704 30390 inappropriate_FIN - F bro 1366445573.232767 BjdVV5GUd2b 57660 30390 possible_split_routing - F bro 1366445573.233064 BjdVV5GUd2b 57660 30390 data_before_established - F bro 1366445573.237140 BjdVV5GUd2b 57660 30390 inappropriate_FIN - F bro 1366445574.510449 r1qXhtWW1bj 9997 44302 inappropriate_FIN - F bro 1366445574.510660 tObvNwyOY2g 9997 35674 inappropriate_FIN - F bro 1366445574.730389 TcPdkyidOv2 37329 30310 possible_split_routing - F bro 1366445574.730846 TcPdkyidOv2 37329 30310 data_before_established - F bro 1366445574.735051 TcPdkyidOv2 37329 30310 inappropriate_FIN - F bro 1366445575.913782 yDxehi4Al72 60946 22 active_connection_reuse - F bro 1366445577.063401 otQNSnT4EN6 5666 51277 active_connection_reuse - F bro 1366445577.063728 UWRLM9DTaji 51277 5666 data_before_established - F bro 1366445577.743207 Mwz8weyGyR3 34725 30390 possible_split_routing - F bro 1366445577.743652 Mwz8weyGyR3 34725 30390 data_before_established - F bro 1366445577.747345 Mwz8weyGyR3 34725 30390 inappropriate_FIN - F bro 1366445578.059666 iswlXZ1p4Ub 9997 40619 inappropriate_FIN - F bro 1366445578.059827 E20A0zNnKr1 9997 44455 inappropriate_FIN - F bro 1366445578.183760 5Y37osqnKkf 57679 30390 possible_split_routing - F bro 1366445578.184349 5Y37osqnKkf 57679 30390 data_before_established - F bro 1366445578.187890 5Y37osqnKkf 57679 30390 inappropriate_FIN - F bro 1366445578.204042 NC98VDwhyic 60005 5666 possible_split_routing - F bro 1366445578.204270 NC98VDwhyic 60005 5666 data_before_established - F bro 1366445578.244628 jj6AoPGcuda 1522 28963 data_before_established - F bro 1366445578.244639 jj6AoPGcuda 1522 28963 inappropriate_FIN - F bro 1366445578.706908 A1Zs7s7nXwk 52529 22 active_connection_reuse - F bro 1366445579.765173 o2wIRfp99bf 37350 30310 possible_split_routing - F bro 1366445579.765499 o2wIRfp99bf 37350 30310 data_before_established - F bro 1366445579.770880 o2wIRfp99bf 37350 30310 inappropriate_FIN - F bro 1366445580.926279 kbXAp4Y6A02 32821 22 active_connection_reuse - F bro 1366445580.926714 PisVwAmRZme 32821 22 possible_split_routing - F bro 1366445580.926772 PisVwAmRZme 32821 22 inappropriate_FIN - F bro 1366445581.207573 LiB2ITRfHhd 47357 30310 possible_split_routing - F bro 1366445581.207883 LiB2ITRfHhd 47357 30310 data_before_established - F bro 1366445581.214265 LiB2ITRfHhd 47357 30310 inappropriate_FIN - F bro 1366445582.778903 L3DPgYmunWg 34746 30390 possible_split_routing - F bro 1366445582.779365 L3DPgYmunWg 34746 30390 data_before_established - F bro 1366445582.783030 L3DPgYmunWg 34746 30390 inappropriate_FIN - F bro 1366445583.224117 Kj2jZLpcHQ8 57700 30390 possible_split_routing - F bro 1366445583.224643 Kj2jZLpcHQ8 57700 30390 data_before_established - F bro 1366445583.227992 Kj2jZLpcHQ8 57700 30390 inappropriate_FIN - F bro 1366445583.243559 e7hVXNWAcXd 5666 59937 data_before_established - F bro 1366445583.243600 e7hVXNWAcXd 5666 59937 active_connection_reuse - F bro 1366445583.243729 IQ7JRPICo5h 59937 5666 possible_split_routing - F bro 1366445583.243797 IQ7JRPICo5h 59937 5666 data_before_established - F bro 1366445583.701833 bCOWiFYHCD 52635 22 active_connection_reuse - F bro 1366445584.687427 JGJvoLHFEB7 37373 30310 possible_split_routing - F bro 1366445584.687809 JGJvoLHFEB7 37373 30310 data_before_established - F bro 1366445584.691962 JGJvoLHFEB7 37373 30310 inappropriate_FIN - F bro 1366445586.153302 3Ob4kGkMRh1 47376 30310 possible_split_routing - F bro 1366445586.153590 3Ob4kGkMRh1 47376 30310 data_before_established - F bro 1366445586.157429 3Ob4kGkMRh1 47376 30310 inappropriate_FIN - F bro 1366445587.725699 tc0YRXXhLte 34771 30390 possible_split_routing - F bro 1366445587.726034 tc0YRXXhLte 34771 30390 data_before_established - F bro 1366445587.731125 tc0YRXXhLte 34771 30390 inappropriate_FIN - F bro 1366445588.637477 Q02XJs56kOh 53571 22 active_connection_reuse - F bro 1366445588.638637 6ZMW9bImJPk 53571 22 possible_split_routing - F bro 1366445588.638854 6ZMW9bImJPk 53571 22 inappropriate_FIN - F bro 1366445588.739231 0TqgtuL9f09 52745 22 active_connection_reuse - F bro 1366445589.240728 eGBBdQdpsa4 51904 9997 possible_split_routing - F bro 1366445589.241435 GuQiSBjJV81 52907 9997 possible_split_routing - F bro 1366445589.620204 OQ9yx4BafSe 9997 63043 inappropriate_FIN - F bro 1366445589.620212 mJB9omzWSJ4 9997 47518 inappropriate_FIN - F bro 1366445589.669002 yP5EYio24W 9997 41757 inappropriate_FIN - F bro 1366445589.669023 W7CKWaOEVy9 9997 33300 inappropriate_FIN - F bro 1366445589.746727 CrufwTKL6Ge 37396 30310 possible_split_routing - F bro 1366445589.747158 CrufwTKL6Ge 37396 30310 data_before_established - F bro 1366445589.750101 CrufwTKL6Ge 37396 30310 inappropriate_FIN - F bro 1366445590.927663 DsNA8pbfsBc 33044 22 active_connection_reuse - F bro 1366445590.929037 zhl0AijYsK 33044 22 possible_split_routing - F bro 1366445590.929271 zhl0AijYsK 33044 22 inappropriate_FIN - F bro 1366445591.201280 1eOWlJTZgA4 47397 30310 possible_split_routing - F bro 1366445591.201483 1eOWlJTZgA4 47397 30310 data_before_established - F bro 1366445591.205162 1eOWlJTZgA4 47397 30310 inappropriate_FIN - F bro 1366445592.084380 HzDFhdIAHGk 40657 5666 possible_split_routing - F bro 1366445592.084641 HzDFhdIAHGk 40657 5666 data_before_established - F bro 1366445592.131622 xOlqn2UHYBh 36773 5666 possible_split_routing - F bro 1366445592.131814 xOlqn2UHYBh 36773 5666 data_before_established - F bro 1366445592.132775 9RwZo1ZqMu7 36776 5666 possible_split_routing - F bro 1366445592.133552 9RwZo1ZqMu7 36776 5666 data_before_established - F bro 1366445592.134698 t5FrtDo8b99 36777 5666 possible_split_routing - F bro 1366445592.135041 t5FrtDo8b99 36777 5666 data_before_established - F bro 1366445592.165969 5ZAeOoyBTw4 5666 40250 active_connection_reuse - F bro 1366445592.166997 nwlzihurpM 40250 5666 data_before_established - F bro 1366445592.769294 HtIFXqFKxc9 34790 30390 possible_split_routing - F bro 1366445592.769819 HtIFXqFKxc9 34790 30390 data_before_established - F bro 1366445592.773547 HtIFXqFKxc9 34790 30390 inappropriate_FIN - F bro 1366445593.220485 LCp8FbOMFyh 57740 30390 possible_split_routing - F bro 1366445593.220765 LCp8FbOMFyh 57740 30390 data_before_established - F bro 1366445593.224419 LCp8FbOMFyh 57740 30390 inappropriate_FIN - F bro 1366445594.687187 CuOQQ7zeFv9 37415 30310 possible_split_routing - F bro 1366445594.687523 CuOQQ7zeFv9 37415 30310 data_before_established - F bro 1366445594.691361 CuOQQ7zeFv9 37415 30310 inappropriate_FIN - F bro 1366445596.150750 SBPXzXc2Ld4 47416 30310 possible_split_routing - F bro 1366445596.151009 SBPXzXc2Ld4 47416 30310 data_before_established - F bro 1366445596.154921 SBPXzXc2Ld4 47416 30310 inappropriate_FIN - F bro 1366445597.184565 qEiR2l1TFRk 56920 389 possible_split_routing - F bro 1366445597.184651 qEiR2l1TFRk 56920 389 data_before_established - F bro 1366445597.710011 SOoWl6RKWKh 34811 30390 possible_split_routing - F bro 1366445597.710502 SOoWl6RKWKh 34811 30390 data_before_established - F bro 1366445597.714152 SOoWl6RKWKh 34811 30390 inappropriate_FIN - F bro 1366445598.056099 uggbZJSe6Y 38292 22 active_connection_reuse - F bro 1366445598.166435 8sGmATv0hQe 57759 30390 possible_split_routing - F bro 1366445598.166768 8sGmATv0hQe 57759 30390 data_before_established - F bro 1366445598.171075 8sGmATv0hQe 57759 30390 inappropriate_FIN - F bro 1366445598.314109 Dagu32rJNve 42335 30090 possible_split_routing - F bro 1366445598.314407 Dagu32rJNve 42335 30090 data_before_established - F bro 1366445598.317858 pUHgaskr6q4 43443 5666 possible_split_routing - F bro 1366445598.318003 pUHgaskr6q4 43443 5666 data_before_established - F bro 1366445598.337726 h9SmXzOmYj9 34044 5666 possible_split_routing - F bro 1366445598.337961 h9SmXzOmYj9 34044 5666 data_before_established - F bro 1366445598.342370 ze6jFUsuRrd 42337 30090 possible_split_routing - F bro 1366445598.348767 ze6jFUsuRrd 42337 30090 data_before_established - F bro 1366445598.420624 lW5R8mIZ1Rl 5666 51409 active_connection_reuse - F bro 1366445598.420785 tbjrXcZo2F8 51409 5666 data_before_established - F bro 1366445598.833697 FbWPTvKfWE7 9997 47414 inappropriate_FIN - F bro 1366445598.833866 mPmGNK0x3Y7 9997 32944 inappropriate_FIN - F bro 1366445599.718553 qOMrTniwJO8 37436 30310 possible_split_routing - F bro 1366445599.718580 qOMrTniwJO8 37436 30310 data_before_established - F bro 1366445599.721075 qOMrTniwJO8 37436 30310 inappropriate_FIN - F bro 1366445601.197471 A6hBUcHyYn 47438 30310 possible_split_routing - F bro 1366445601.197857 A6hBUcHyYn 47438 30310 data_before_established - F bro 1366445601.201627 A6hBUcHyYn 47438 30310 inappropriate_FIN - F bro 1366445602.737133 fMC2czM4ad5 34830 30390 possible_split_routing - F bro 1366445602.737195 fMC2czM4ad5 34830 30390 data_before_established - F bro 1366445602.739803 fMC2czM4ad5 34830 30390 inappropriate_FIN - F bro 1366445603.217926 5KQYxY7FEjd 57781 30390 possible_split_routing - F bro 1366445603.218318 5KQYxY7FEjd 57781 30390 data_before_established - F bro 1366445603.222483 5KQYxY7FEjd 57781 30390 inappropriate_FIN - F bro 1366445604.158883 ND8g7Onxkol 9997 35676 data_before_established - F bro 1366445604.598523 5BsmxtcRqUg 9997 44304 inappropriate_FIN - F bro 1366445604.769609 mkfqXaTHoYd 37455 30310 possible_split_routing - F bro 1366445604.769945 mkfqXaTHoYd 37455 30310 data_before_established - F bro 1366445604.773733 mkfqXaTHoYd 37455 30310 inappropriate_FIN - F bro 1366445606.233186 2YpIesZO8S8 47457 30310 possible_split_routing - F bro 1366445606.233483 2YpIesZO8S8 47457 30310 data_before_established - F bro 1366445606.237583 2YpIesZO8S8 47457 30310 inappropriate_FIN - F bro 1366445607.696371 pEFuDb9cZb8 34851 30390 possible_split_routing - F bro 1366445607.696856 pEFuDb9cZb8 34851 30390 data_before_established - F bro 1366445607.700532 pEFuDb9cZb8 34851 30390 inappropriate_FIN - F bro 1366445608.078190 VrW91Y7Afml 9997 40622 inappropriate_FIN - F bro 1366445608.078379 T3BCgECYnt8 9997 44458 inappropriate_FIN - F bro 1366445608.161685 djiIIUpoL2 57800 30390 possible_split_routing - F bro 1366445608.162008 djiIIUpoL2 57800 30390 data_before_established - F bro -- -- kimh -------------------------------------------------------- Kim Halavakoski - Chief Everything Officer - GCIH Black Cat Security - http://www.blackcatsec.net email: kim at blackcatsec.net Mobile: +358 xxx xxxxxxx PGP: 0BFA A910 9AA7 94A5 A323 53F5 4151 4CE4 -------------------------------------------------------- From vern at icir.org Sun Apr 21 02:05:49 2013 From: vern at icir.org (Vern Paxson) Date: Sun, 21 Apr 2013 02:05:49 -0700 Subject: [Bro] Weird stuff in weird.log? In-Reply-To: <51727368.6000604@blackcatsec.net> (Sat, 20 Apr 2013 13:52:24 +0300). Message-ID: <20130421090549.617BF2C4003@rock.ICSI.Berkeley.EDU> > I suspect that it is due to the fact that I am spanning > multiple VLANs that Bro sees, with traffic both before and after > loabalancers and NATs etc. so it kind-of sees the whole chain of packets > from outside the firewall, before / after loadbalancer behind firewall > and finally the traffic behind the loadbalancers/firewalls...would that > in some way explain the weird.log stuff shown here? That for sure would explain these sorts of "weird" messages, since they all relate to Bro reporting that it's not seeing a single consistent picture of (bidirectional) network flows. Vern From pfranzel at t-online.de Sun Apr 21 03:23:20 2013 From: pfranzel at t-online.de (Peter Franzel) Date: Sun, 21 Apr 2013 12:23:20 +0200 Subject: [Bro] Weird stuff in weird.log? In-Reply-To: <20130421090549.617BF2C4003@rock.ICSI.Berkeley.EDU> References: <20130421090549.617BF2C4003@rock.ICSI.Berkeley.EDU> Message-ID: <5173BE18.9020901@t-online.de> I am experiencing the same problem in the weired.log here as using one interface defined for the WAN and one for the LAN traffic (between there is a firewall and a loadbalancer with ssl-offload). I am using the following node configuration: [manager] type=manager host=10.XX.XX.11 [proxy-1] type=proxy host=10.XX.XX.11 [worker-1] --> WAN Connection type=worker host=10.XX.XX.11 interface=p6p1 lb_method=pf_ring lb_procs=8 [worker-2] --> LAN Connection type=worker host=10.XX.XX.11 interface=p6p2 lb_method=pf_ring lb_procs=8 [worker-3] --> dedicated line between two DCs type=worker host=10.XX.XX.11 interface=bond0 ... Question: What I meaningfully should do to get rid of this: --> Running one bro cluster/instance for each interface? --> Or is there are way to do it by an other configuration change? Peter Am 21.04.2013 11:05, schrieb Vern Paxson: >> I suspect that it is due to the fact that I am spanning >> multiple VLANs that Bro sees, with traffic both before and after >> loabalancers and NATs etc. so it kind-of sees the whole chain of packets >> from outside the firewall, before / after loadbalancer behind firewall >> and finally the traffic behind the loadbalancers/firewalls...would that >> in some way explain the weird.log stuff shown here? > That for sure would explain these sorts of "weird" messages, since they > all relate to Bro reporting that it's not seeing a single consistent > picture of (bidirectional) network flows. > > Vern > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130421/b909545b/attachment.html From kim at blackcatsec.net Sun Apr 21 04:15:38 2013 From: kim at blackcatsec.net (Kim Halavakoski) Date: Sun, 21 Apr 2013 14:15:38 +0300 Subject: [Bro] Weird stuff in weird.log? In-Reply-To: <5173BE18.9020901@t-online.de> References: <20130421090549.617BF2C4003@rock.ICSI.Berkeley.EDU> <5173BE18.9020901@t-online.de> Message-ID: Peter, I was about to ask the list if splitting up interfaces / VLANs / snooping-points in different workers would fix it, but it seems you have done that and are still seeing the same weirdness. May I ask what was your reason for having 3 workers and a proxy? I am still new to how to design and setup Bro with all the features it has... Best regards, Kim Halavakoski Sent from my mobile device, excuse my clawfingerness! Mobile N#: +358 [0] 40 702 7844 PGP S#: 0BFA A910 9AA7 94A5 A323 53F5 4151 4CE4 33BE 35FA http://www.blackcatsec.net On 21 apr 2013, at 13:23, Peter Franzel wrote: > I am experiencing the same problem in the weired.log here as using one interface defined for the WAN and one for the LAN traffic (between there is a firewall and a loadbalancer with ssl-offload). > I am using the following node configuration: > > [manager] > type=manager > host=10.XX.XX.11 > > [proxy-1] > type=proxy > host=10.XX.XX.11 > > [worker-1] --> WAN Connection > type=worker > host=10.XX.XX.11 > interface=p6p1 > lb_method=pf_ring > lb_procs=8 > > [worker-2] --> LAN Connection > type=worker > host=10.XX.XX.11 > interface=p6p2 > lb_method=pf_ring > lb_procs=8 > > [worker-3] --> dedicated line between two DCs > type=worker > host=10.XX.XX.11 > interface=bond0 > ... > > Question: What I meaningfully should do to get rid of this: > --> Running one bro cluster/instance for each interface? > --> Or is there are way to do it by an other configuration change? > > Peter > > Am 21.04.2013 11:05, schrieb Vern Paxson: >>> I suspect that it is due to the fact that I am spanning >>> multiple VLANs that Bro sees, with traffic both before and after >>> loabalancers and NATs etc. so it kind-of sees the whole chain of packets >>> from outside the firewall, before / after loadbalancer behind firewall >>> and finally the traffic behind the loadbalancers/firewalls...would that >>> in some way explain the weird.log stuff shown here? >> That for sure would explain these sorts of "weird" messages, since they >> all relate to Bro reporting that it's not seeing a single consistent >> picture of (bidirectional) network flows. >> >> Vern >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130421/7d8c7a04/attachment.html From kim at blackcatsec.net Sun Apr 21 08:08:24 2013 From: kim at blackcatsec.net (Kim Halavakoski) Date: Sun, 21 Apr 2013 18:08:24 +0300 Subject: [Bro] Analysis tool ideas/tips? Message-ID: Hello Bro-Jedis, The more I experiment with Bro and start to see the wast possibilities with the collected data, I also feel the need for some serious log analysis / correlation capabilities besides the self-evident Unixy-way with grep, sort and [awk|sed]... I really like splunk and I will for sure try feeding splunk with the bro-logs, and I am sure it will work perfectly. And there is even an app for that: http://eyeis.net/2012/04/splunking-the-onion/ Have you guys tried out any other log analysis tools? Greylog2? Logstash+Kibana? Any ideas/tips/discussions welcome... PS. I saw somewhere a presentation about saving executables flying over the network. Any pointers to scripts for doing that? I have a vision of sending those to cuckoosandbox for automatic malware analysis, ehich would be cool, kind-of like FireEye! Best regards, Kim Halavakoski Sent from my mobile device, excuse my clawfingerness! PGP S#: 0BFA A910 9AA7 94A5 A323 53F5 4151 4CE4 33BE 35FA http://www.blackcatsec.net -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130421/d18840e0/attachment.html From lysemose at gmail.com Sun Apr 21 08:56:36 2013 From: lysemose at gmail.com (Heine Lysemose) Date: Sun, 21 Apr 2013 17:56:36 +0200 Subject: [Bro] Analysis tool ideas/tips? In-Reply-To: References: Message-ID: Hi Take a look at ELSA, https://code.google.com/p/enterprise-log-search-and-archive/ Regards, Lysemose On Apr 21, 2013 5:10 PM, "Kim Halavakoski" wrote: > > Hello Bro-Jedis, > The more I experiment with Bro and start to see the wast possibilities > with the collected data, I also feel the need for some serious log analysis > / correlation capabilities besides the self-evident Unixy-way with grep, > sort and [awk|sed]... > > I really like splunk and I will for sure try feeding splunk with the > bro-logs, and I am sure it will work perfectly. And there is even an app > for that: http://eyeis.net/2012/04/splunking-the-onion/ > > Have you guys tried out any other log analysis tools? Greylog2? > Logstash+Kibana? > > Any ideas/tips/discussions welcome... > > PS. I saw somewhere a presentation about saving executables flying over > the network. Any pointers to scripts for doing that? I have a vision of > sending those to cuckoosandbox for automatic malware analysis, ehich would > be cool, kind-of like FireEye! > > > Best regards, > > Kim Halavakoski > > *Sent from my mobile device, excuse my clawfingerness!* > > PGP S#: 0BFA A910 9AA7 94A5 A323 53F5 4151 4CE4 33BE 35FA > http://www.blackcatsec.net > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130421/e485119a/attachment.html From michal at rsbac.org Mon Apr 22 12:07:18 2013 From: michal at rsbac.org (Michal Purzynski) Date: Mon, 22 Apr 2013 21:07:18 +0200 Subject: [Bro] BRO performance in a real world Message-ID: <51758A66.2000609@rsbac.org> Hi. How's the BRO real world performance? You know, 10Gbit links and up. How many workers do I need for every 1Gbit of traffic (sure, it depends on the rules heavily)? Or just how much traffic can I expect a single worker to handle? How about the memory? That's what I have here: Intel(R) Xeon(R) CPU E5-2620 @ 2.00GHz x 2 so it gives 24 threads with HT enabled. Also, I have 64GB of RAM in each NSM sensor. Expected traffc? A few Gbit/sec, depending on a sensor location. Do you have some real world examples, such as "we have server with and and it handles Gbit/sec of traffic on average/peak" I know that's a lot of questions, but trying to establish a baseline and do some capacity planning here :) And there's nothing in google, apart from some (i guess old) statement, that a single bro process can handle up to 80Mbit/sec. From bernhard at ICSI.Berkeley.EDU Mon Apr 22 12:14:35 2013 From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann) Date: Mon, 22 Apr 2013 12:14:35 -0700 Subject: [Bro] BRO performance in a real world In-Reply-To: <51758A66.2000609@rsbac.org> References: <51758A66.2000609@rsbac.org> Message-ID: On Apr 22, 2013, at 12:07 PM, Michal Purzynski wrote: > How's the BRO real world performance? You know, 10Gbit links and up. How > many workers do I need for every 1Gbit of traffic (sure, it depends on > the rules heavily)? [?] > > Do you have some real world examples, such as "we have server with > and and it handles Gbit/sec of traffic on average/peak" There was a thread about exactly this on here just a few weeks ago - to cite a bit from it: On Mar 19, 2013, at 11:20 AM, Mike Patterson wrote: [?] > I keep meaning to write this up, but on *my* configuration: > * 16 cores of model name : Intel(R) Xeon(R) CPU X5677 @ 3.47GHz > * 72GB of RAM > * Endace DAG (9.2) > * some config magic by Seth, which I'd be happy to share. > > 6 workers keep up with ~2.5-3Gbps peaks, no problem. [?] > It doesn't actually consume all of the above resources - I'm running other things on the box too - but bro itself consumes ~4.5GB resident per worker, and can be counted on to pin most of its allocated cores at peak loads. On Mar 19, 2013, at 11:35 AM, Vlad Grigorescu wrote: > Just to throw another data point out there: > * 16 physical cores of model name : Intel(R) Xeon(R) CPU E5-2680 @ 2.70 GHz > * 96GB of RAM > * Myricom NIC > > 28 workers (I have Hyperthreading turned on) keep up with a 6-7 Gbps average, and I've seen them do fine with short peaks of 9 Gbps or so. The Myricom cards definitely won't break the bank: card + SR optics + perpetual license is $895. [?] Full thread at: http://mailman.icsi.berkeley.edu/pipermail/bro/2013-March/006242.html I hope that helps, Bernhard From michal at rsbac.org Mon Apr 22 12:19:51 2013 From: michal at rsbac.org (Michal Purzynski) Date: Mon, 22 Apr 2013 21:19:51 +0200 Subject: [Bro] BRO performance in a real world In-Reply-To: References: <51758A66.2000609@rsbac.org> Message-ID: <51758D57.8010503@rsbac.org> On 4/22/13 9:14 PM, Bernhard Amann wrote: > On Apr 22, 2013, at 12:07 PM, Michal Purzynski wrote: > >> How's the BRO real world performance? You know, 10Gbit links and up. How >> many workers do I need for every 1Gbit of traffic (sure, it depends on >> the rules heavily)? (...) > Full thread at: > http://mailman.icsi.berkeley.edu/pipermail/bro/2013-March/006242.html > I hope that helps, > Bernhard Thanks, now I'm like stupid because I should have checked ;) Now that's what I call the real world numbers, awesome! From seth at icir.org Mon Apr 22 12:23:36 2013 From: seth at icir.org (Seth Hall) Date: Mon, 22 Apr 2013 15:23:36 -0400 Subject: [Bro] BRO performance in a real world In-Reply-To: <51758A66.2000609@rsbac.org> References: <51758A66.2000609@rsbac.org> Message-ID: On Apr 22, 2013, at 3:07 PM, Michal Purzynski wrote: > I know that's a lot of questions, but trying to establish a baseline and > do some capacity planning here :) And there's nothing in google, apart > from some (i guess old) statement, that a single bro process can handle > up to 80Mbit/sec. Yeah, I begrudgingly wrote that because the question came up so frequently. It was based on old estimates and doesn't seem to be as relevant anymore. I know of sites doing everything from 100Mbps/core to >500Mbps/core, it depends heavily on the clock rate of the CPU and how you are capturing packets. In the case of the site with >500Mbps/core, they are using an Endace DAG card and skipping the OS nearly completely to acquire packets and their per-core clock rate is 3.7Ghz I believe. With 2GHz cores, you likely won't hit that speed, but it will almost certainly be faster than that horribly documented 80Mbps. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jlay at slave-tothe-box.net Mon Apr 22 12:28:54 2013 From: jlay at slave-tothe-box.net (James Lay) Date: Mon, 22 Apr 2013 13:28:54 -0600 Subject: [Bro] BRO performance in a real world In-Reply-To: References: <51758A66.2000609@rsbac.org> Message-ID: <8ff8b6a59e7dfbb090e0c08628fed583@localhost> On 2013-04-22 13:14, Bernhard Amann wrote: > On Apr 22, 2013, at 12:07 PM, Michal Purzynski > wrote: > >> How's the BRO real world performance? You know, 10Gbit links and up. >> How >> many workers do I need for every 1Gbit of traffic (sure, it depends >> on >> the rules heavily)? > [?] >> >> Do you have some real world examples, such as "we have server with >> >> and and it handles Gbit/sec of traffic on average/peak" > > There was a thread about exactly this on here just a few weeks ago - > to cite a > bit from it: > > On Mar 19, 2013, at 11:20 AM, Mike Patterson > wrote: > [?] >> I keep meaning to write this up, but on *my* configuration: >> * 16 cores of model name : Intel(R) Xeon(R) CPU X5677 @ >> 3.47GHz >> * 72GB of RAM >> * Endace DAG (9.2) >> * some config magic by Seth, which I'd be happy to share. >> >> 6 workers keep up with ~2.5-3Gbps peaks, no problem. > > [?] >> It doesn't actually consume all of the above resources - I'm running >> other things on the box too - but bro itself consumes ~4.5GB resident >> per worker, and can be counted on to pin most of its allocated cores >> at peak loads. > > > On Mar 19, 2013, at 11:35 AM, Vlad Grigorescu wrote: >> Just to throw another data point out there: >> * 16 physical cores of model name : Intel(R) Xeon(R) CPU E5-2680 @ >> 2.70 GHz >> * 96GB of RAM >> * Myricom NIC >> >> 28 workers (I have Hyperthreading turned on) keep up with a 6-7 Gbps >> average, and I've seen them do fine with short peaks of 9 Gbps or so. >> The Myricom cards definitely won't break the bank: card + SR optics + >> perpetual license is $895. > [?] > > Full thread at: > http://mailman.icsi.berkeley.edu/pipermail/bro/2013-March/006242.html > > I hope that helps, > Bernhard Also try running just bro command line instead of using broctl in your tests. James From mike.patterson at uwaterloo.ca Mon Apr 22 12:34:39 2013 From: mike.patterson at uwaterloo.ca (Mike Patterson) Date: Mon, 22 Apr 2013 15:34:39 -0400 Subject: [Bro] BRO performance in a real world In-Reply-To: References: <51758A66.2000609@rsbac.org> Message-ID: <394FBC43-C58F-4DAF-90C3-5C5A293829A0@uwaterloo.ca> I just got my filthy paws on another host similar to the one I specced earlier. It will be getting similar-but-different loads to that one, on a slower CPU and with an Intel NIC instead of the DAG. Once I've got some performance numbers, I'll post those too. I don't mind if people want to contact me, either on or off-list, to see how things are running and what I'm doing. Seth, have you considered collecting these so they're not stashed in the mailing list archives? A "here's some performance numbers from real installations" kind of page. Maybe link it off http://bro.org/community/index.html ? Mike -- The most difficult thing in the world is to know how to do a thing and to watch someone else doing it wrong, without commenting. - T.H. White On 2013-04-22, at 3:23 PM, Seth Hall wrote: > > On Apr 22, 2013, at 3:07 PM, Michal Purzynski wrote: > >> I know that's a lot of questions, but trying to establish a baseline and >> do some capacity planning here :) And there's nothing in google, apart >> from some (i guess old) statement, that a single bro process can handle >> up to 80Mbit/sec. > > Yeah, I begrudgingly wrote that because the question came up so frequently. It was based on old estimates and doesn't seem to be as relevant anymore. I know of sites doing everything from 100Mbps/core to >500Mbps/core, it depends heavily on the clock rate of the CPU and how you are capturing packets. > > In the case of the site with >500Mbps/core, they are using an Endace DAG card and skipping the OS nearly completely to acquire packets and their per-core clock rate is 3.7Ghz I believe. > > With 2GHz cores, you likely won't hit that speed, but it will almost certainly be faster than that horribly documented 80Mbps. :) > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From seth at icir.org Mon Apr 22 12:50:07 2013 From: seth at icir.org (Seth Hall) Date: Mon, 22 Apr 2013 15:50:07 -0400 Subject: [Bro] BRO performance in a real world In-Reply-To: <394FBC43-C58F-4DAF-90C3-5C5A293829A0@uwaterloo.ca> References: <51758A66.2000609@rsbac.org> <394FBC43-C58F-4DAF-90C3-5C5A293829A0@uwaterloo.ca> Message-ID: <8A7C6506-D17F-4AA3-BBD2-CEE591B59284@icir.org> On Apr 22, 2013, at 3:34 PM, Mike Patterson wrote: > Seth, have you considered collecting these so they're not stashed in the mailing list archives?  Keep bugging us about it. That's a good idea. Oh yeah, for the benefit of the list, most sites run pretty stock Bro so performance differences are mediated by that. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From michal at rsbac.org Mon Apr 22 13:04:42 2013 From: michal at rsbac.org (Michal Purzynski) Date: Mon, 22 Apr 2013 22:04:42 +0200 Subject: [Bro] Bro workers die Message-ID: <517597DA.6090005@rsbac.org> Hi, me again. Bro is a new one, from the SVN, but I had the same results with 2.1 stable. broctl start starting manager ... starting proxy ... starting nsm1-eth4-1 ... starting nsm1-eth5-1 ... starting nsm1-eth5-10 ... starting nsm1-eth5-11 ... starting nsm1-eth5-12 ... starting nsm1-eth5-2 ... starting nsm1-eth5-3 ... starting nsm1-eth5-4 ... starting nsm1-eth5-5 ... starting nsm1-eth5-6 ... starting nsm1-eth5-7 ... starting nsm1-eth5-8 ... starting nsm1-eth5-9 ... (nsm1-eth5-12 still initializing) (nsm1-eth5-9 still initializing) (nsm1-eth5-10 still initializing) (nsm1-eth5-11 still initializing) (nsm1-eth4-1 still initializing) And after a while Name Type Host Status Pid Peers Started nsm1-eth4-1 worker crashed nsm1-eth5-10 worker crashed nsm1-eth5-11 worker crashed nsm1-eth5-12 worker crashed nsm1-eth5-9 worker crashed manager manager running 44798 9 22 Apr 19:27:37 proxy proxy running 44845 9 22 Apr 19:27:39 nsm1-eth5-1 worker running 45048 2 22 Apr 19:27:41 nsm1-eth5-2 worker running 45057 2 22 Apr 19:27:41 nsm1-eth5-3 worker running 45060 2 22 Apr 19:27:41 nsm1-eth5-4 worker running 45063 2 22 Apr 19:27:41 nsm1-eth5-5 worker running 45066 2 22 Apr 19:27:41 nsm1-eth5-6 worker running 45067 2 22 Apr 19:27:41 nsm1-eth5-7 worker running 45068 2 22 Apr 19:27:41 nsm1-eth5-8 worker running 45069 2 22 Apr 19:27:41 Two more questions: 1. does Bro use pf_ring by default with a configuration like this? 2. how can i change the load balancing method? I need to spread things more evenly. cat /opt/bro/etc/node.cfg [manager] type=manager host= [proxy] type=proxy host= [nsm1-eth4] type=worker host= interface=eth4 lb_method=pf_ring lb_procs=1 [nsm1-eth5] type=worker host= interface=eth5 lb_method=pf_ring lb_procs=12 From seth at icir.org Mon Apr 22 13:21:30 2013 From: seth at icir.org (Seth Hall) Date: Mon, 22 Apr 2013 16:21:30 -0400 Subject: [Bro] Bro workers die In-Reply-To: <517597DA.6090005@rsbac.org> References: <517597DA.6090005@rsbac.org> Message-ID: On Apr 22, 2013, at 4:04 PM, Michal Purzynski wrote: > 1. does Bro use pf_ring by default with a configuration like this? Yes, it's the lb_method=pf_ring that enables it. > 2. how can i change the load balancing method? I need to spread things > more evenly. What do you want to change it to? I think it's doing 4-tuple or 5-tuple by default right now.  One problem you will encounter is a issue with pf_ring cluster_id choice. You will be running two pf_ring clusters on the same host (i'm assuming that nsm1 is the same physical host) and pf_ring doesn't like that. It does something weird like trying to stick packets from both NICs into the same queue. We have it fixed for our next release (that did get merged into master, right Daniel?) but it's a problem right now. You are sending us enough information to determine why you're seeing crashes though. Could you send the output from broctl diag nsm1-eth5-1 (assuming that's a host that is currently crashed)? Thanks, .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jones at tacc.utexas.edu Mon Apr 22 13:24:07 2013 From: jones at tacc.utexas.edu (William Jones) Date: Mon, 22 Apr 2013 20:24:07 +0000 Subject: [Bro] BRO performance in a real world In-Reply-To: <8ff8b6a59e7dfbb090e0c08628fed583@localhost> References: <51758A66.2000609@rsbac.org> <8ff8b6a59e7dfbb090e0c08628fed583@localhost> Message-ID: What sort of packet rate can you handle per worker? -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of James Lay Sent: Monday, April 22, 2013 2:29 PM To: bro at bro.org Subject: Re: [Bro] BRO performance in a real world On 2013-04-22 13:14, Bernhard Amann wrote: > On Apr 22, 2013, at 12:07 PM, Michal Purzynski > wrote: > >> How's the BRO real world performance? You know, 10Gbit links and up. >> How >> many workers do I need for every 1Gbit of traffic (sure, it depends >> on the rules heavily)? > [?] >> >> Do you have some real world examples, such as "we have server with >> >> and and it handles Gbit/sec of traffic on average/peak" > > There was a thread about exactly this on here just a few weeks ago - > to cite a > bit from it: > > On Mar 19, 2013, at 11:20 AM, Mike Patterson > wrote: > [?] >> I keep meaning to write this up, but on *my* configuration: >> * 16 cores of model name : Intel(R) Xeon(R) CPU X5677 @ >> 3.47GHz >> * 72GB of RAM >> * Endace DAG (9.2) >> * some config magic by Seth, which I'd be happy to share. >> >> 6 workers keep up with ~2.5-3Gbps peaks, no problem. > > [?] >> It doesn't actually consume all of the above resources - I'm running >> other things on the box too - but bro itself consumes ~4.5GB resident >> per worker, and can be counted on to pin most of its allocated cores >> at peak loads. > > > On Mar 19, 2013, at 11:35 AM, Vlad Grigorescu wrote: >> Just to throw another data point out there: >> * 16 physical cores of model name : Intel(R) Xeon(R) CPU E5-2680 @ >> 2.70 GHz >> * 96GB of RAM >> * Myricom NIC >> >> 28 workers (I have Hyperthreading turned on) keep up with a 6-7 Gbps >> average, and I've seen them do fine with short peaks of 9 Gbps or so. >> The Myricom cards definitely won't break the bank: card + SR optics + >> perpetual license is $895. > [?] > > Full thread at: > http://mailman.icsi.berkeley.edu/pipermail/bro/2013-March/006242.html > > I hope that helps, > Bernhard Also try running just bro command line instead of using broctl in your tests. James _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jlay at slave-tothe-box.net Mon Apr 22 13:28:44 2013 From: jlay at slave-tothe-box.net (James Lay) Date: Mon, 22 Apr 2013 14:28:44 -0600 Subject: [Bro] BRO performance in a real world In-Reply-To: References: <51758A66.2000609@rsbac.org> <8ff8b6a59e7dfbb090e0c08628fed583@localhost> Message-ID: <641cb8f057860f682bf773119b12c636@localhost> On 2013-04-22 14:24, William Jones wrote: > What sort of packet rate can you handle per worker? > I've maxed out gig links with only minimal impact. I use bro command line though and not broctl. James From michal at rsbac.org Mon Apr 22 13:51:41 2013 From: michal at rsbac.org (Michal Purzynski) Date: Mon, 22 Apr 2013 22:51:41 +0200 Subject: [Bro] Bro workers die In-Reply-To: References: <517597DA.6090005@rsbac.org> Message-ID: <5175A2DD.1020708@rsbac.org> On 4/22/13 10:21 PM, Seth Hall wrote: > On Apr 22, 2013, at 4:04 PM, Michal Purzynski wrote: > >> 1. does Bro use pf_ring by default with a configuration like this? > Yes, it's the lb_method=pf_ring that enables it. > >> 2. how can i change the load balancing method? I need to spread things >> more evenly. > What do you want to change it to? I think it's doing 4-tuple or 5-tuple by default right now. OK, I might be wrong on that, it has helped in a big way for snort. >  > One problem you will encounter is a issue with pf_ring cluster_id choice. You will be running two pf_ring clusters on the same host (i'm assuming that nsm1 is the same physical host) and pf_ring doesn't like that. It does something weird like trying to stick packets from both NICs into the same queue. We have it fixed for our next release (that did get merged into master, right Daniel?) but it's a problem right now. I'm running the SVN code, so you think it does not choose a unique cluster id for eth4 and another for eth5? How can i fix it? > > You are sending us enough information to determine why you're seeing crashes though. Could you send the output from broctl diag nsm1-eth5-1 (assuming that's a host that is currently crashed)? > > Thanks, > > .Seth broctl diag nsm1-eth5-1 [nsm1-eth5-1] Bro 2.1-386 ==== No reporter.log ==== stderr.log listening on eth5, capture length 8192 bytes 1366658863.663940 processing suspended 1366658863.664006 processing continued 1366658869.682828 Failed to open GeoIP database: /usr/share/GeoIP/GeoIPCity.dat 1366658869.682828 Fell back to GeoIP Country database 1366658869.682828 Failed to open GeoIP database: /usr/share/GeoIP/GeoIPCityv6.dat ==== stdout.log unlimited unlimited unlimited ==== .cmdline -i eth5 -U .status -p broctl -p broctl-live -p local -p nsm1-eth5-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/opt/bro/bin:/opt/bro/share/broctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bro/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/opt/bro/share/bro:/opt/bro/share/bro/policy:/opt/bro/share/bro/site CLUSTER_NODE=nsm1-eth5-1 ==== .status RUNNING [net_run] ==== No prof.log ==== No packet_filter.log ==== No loaded_scripts.log > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > From seth at icir.org Mon Apr 22 13:58:19 2013 From: seth at icir.org (Seth Hall) Date: Mon, 22 Apr 2013 16:58:19 -0400 Subject: [Bro] Bro workers die In-Reply-To: <5175A2DD.1020708@rsbac.org> References: <517597DA.6090005@rsbac.org> <5175A2DD.1020708@rsbac.org> Message-ID: On Apr 22, 2013, at 4:51 PM, Michal Purzynski wrote: > I'm running the SVN code, so you think it does not choose a unique cluster id for eth4 and another for eth5? How can i fix it? I don't know if that's fixed in master yet (i'm assuming you're running git master). > broctl diag nsm1-eth5-1 That shows the process is running fine. You need to do that for a worker that is crashed. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From michal at rsbac.org Mon Apr 22 14:01:50 2013 From: michal at rsbac.org (Michal Purzynski) Date: Mon, 22 Apr 2013 23:01:50 +0200 Subject: [Bro] Bro workers die In-Reply-To: References: <517597DA.6090005@rsbac.org> <5175A2DD.1020708@rsbac.org> Message-ID: <5175A53E.8000608@rsbac.org> On 4/22/13 10:58 PM, Seth Hall wrote: > On Apr 22, 2013, at 4:51 PM, Michal Purzynski wrote: > >> I'm running the SVN code, so you think it does not choose a unique cluster id for eth4 and another for eth5? How can i fix it? > > I don't know if that's fixed in master yet (i'm assuming you're running git master). > >> broctl diag nsm1-eth5-1 > That shows the process is running fine. You need to do that for a worker that is crashed. > > Right, just noticed the stack traces. root at nsm1:/nsm/bro/logs/current# broctl diag nsm1-eth5-9 [nsm1-eth5-9] Bro 2.1-386 core [New LWP 54717] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/opt/bro/bin/bro -i eth5 -U .status -p broctl -p broctl-live -p local -p nsm1-e'. Program terminated with signal 11, Segmentation fault. #0 AsBool (this=0x0) at scan.l:1074 Thread 1 (Thread 0x7f3af913d740 (LWP 54717)): #0 AsBool (this=0x0) at scan.l:1074 #1 do_atif (expr=) at scan.l:686 #2 0x000000000051c95e in yyparse () at parse.y:1203 #3 0x00000000004c615e in main (argc=18, argv=) at /home/michal/bro/src/main.cc:801 ==== No reporter.log ==== stderr.log error in /opt/bro/share/bro/base/frameworks/cluster/./main.bro, line 136: no such index (Cluster::nodes[Cluster::node]) warning in /opt/bro/share/bro/base/frameworks/notice/./cluster.bro, line 23: non-void function returns without a value: Cluster::local_node_type /opt/bro/share/broctl/scripts/run-bro: line 60: 54717 Segmentation fault (core dumped) nohup $mybro $@ ==== stdout.log unlimited unlimited unlimited ==== .cmdline -i eth5 -U .status -p broctl -p broctl-live -p local -p nsm1-eth5-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/opt/bro/bin:/opt/bro/share/broctl/scripts:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/opt/bro/share/bro:/opt/bro/share/bro/policy:/opt/bro/share/bro/site CLUSTER_NODE=nsm1-eth5-9 ==== .status INITIALIZING [main] ==== No prof.log ==== No packet_filter.log ==== No loaded_scripts.log You have new mail in /var/mail/root root at nsm1:/nsm/bro/logs/current# broctl diag nsm1-eth4-1 [nsm1-eth4-1] Bro 2.1-386 core [New LWP 54008] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/opt/bro/bin/bro -i eth4 -U .status -p broctl -p broctl-live -p local -p nsm1-e'. Program terminated with signal 11, Segmentation fault. #0 AsBool (this=0x0) at scan.l:1074 Thread 1 (Thread 0x7ff6c1d0f740 (LWP 54008)): #0 AsBool (this=0x0) at scan.l:1074 #1 do_atif (expr=) at scan.l:686 #2 0x000000000051c95e in yyparse () at parse.y:1203 #3 0x00000000004c615e in main (argc=18, argv=) at /home/michal/bro/src/main.cc:801 ==== No reporter.log ==== stderr.log error in /opt/bro/share/bro/base/frameworks/cluster/./main.bro, line 136: no such index (Cluster::nodes[Cluster::node]) warning in /opt/bro/share/bro/base/frameworks/notice/./cluster.bro, line 23: non-void function returns without a value: Cluster::local_node_type /opt/bro/share/broctl/scripts/run-bro: line 60: 54008 Segmentation fault (core dumped) nohup $mybro $@ ==== stdout.log unlimited unlimited unlimited ==== .cmdline -i eth4 -U .status -p broctl -p broctl-live -p local -p nsm1-eth4-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/opt/bro/bin:/opt/bro/share/broctl/scripts:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/opt/bro/share/bro:/opt/bro/share/bro/policy:/opt/bro/share/bro/site CLUSTER_NODE=nsm1-eth4-1 ==== .status INITIALIZING [main] ==== No prof.log ==== No packet_filter.log ==== No loaded_scripts.log root at nsm1:/nsm/bro/logs/current# From seth at icir.org Mon Apr 22 14:05:59 2013 From: seth at icir.org (Seth Hall) Date: Mon, 22 Apr 2013 17:05:59 -0400 Subject: [Bro] Bro workers die In-Reply-To: <5175A53E.8000608@rsbac.org> References: <517597DA.6090005@rsbac.org> <5175A2DD.1020708@rsbac.org> <5175A53E.8000608@rsbac.org> Message-ID: On Apr 22, 2013, at 5:01 PM, Michal Purzynski wrote: > error in /opt/bro/share/bro/base/frameworks/cluster/./main.bro, line 136: no such index (Cluster::nodes[Cluster::node]) > warning in /opt/bro/share/bro/base/frameworks/notice/./cluster.bro, line 23: non-void function returns without a value: Cluster::local_node_type Have you run "broctl install" since you last changed your node.cfg file? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From michal at rsbac.org Mon Apr 22 14:15:37 2013 From: michal at rsbac.org (Michal Purzynski) Date: Mon, 22 Apr 2013 23:15:37 +0200 Subject: [Bro] Bro workers die In-Reply-To: References: <517597DA.6090005@rsbac.org> <5175A2DD.1020708@rsbac.org> <5175A53E.8000608@rsbac.org> Message-ID: <5175A879.1070504@rsbac.org> On 4/22/13 11:05 PM, Seth Hall wrote: > On Apr 22, 2013, at 5:01 PM, Michal Purzynski wrote: > >> error in /opt/bro/share/bro/base/frameworks/cluster/./main.bro, line 136: no such index (Cluster::nodes[Cluster::node]) >> warning in /opt/bro/share/bro/base/frameworks/notice/./cluster.bro, line 23: non-void function returns without a value: Cluster::local_node_type > Have you run "broctl install" since you last changed your node.cfg file? > > root at nsm1:~# broctl status 2>&1 | grep nsm1 | grep worker | wc -l 13 root at nsm1:~# broctl status 2>&1 | grep nsm1 | grep running | wc -l 13 Awesome! Thank you, i didn't know I'm supposed to :) Now on to the traffic filtering (which is ignored) but I've separated that into another post. From jones at tacc.utexas.edu Tue Apr 23 08:38:41 2013 From: jones at tacc.utexas.edu (William Jones) Date: Tue, 23 Apr 2013 15:38:41 +0000 Subject: [Bro] Packet scans drops In-Reply-To: References: <517597DA.6090005@rsbac.org> <5175A2DD.1020708@rsbac.org> Message-ID: Seth, The only time I am seeing dropped packets are during attempts to us TACC to amplify dos attach very aggressive port scans. In both cases bro workers are being overloaded by 500kk to 1000k incoming packets. It looks like a single worker can only handle 30K packets/sec before it reaches 100 percent cpu usage. Is there any effort going into bro development to handle these cases. My only work around that I have now is to block aces to common ports at the boarder router and opening host to vetted hosts. Bill Jones From david at mandelberg.org Wed Apr 24 12:26:53 2013 From: david at mandelberg.org (David Mandelberg) Date: Wed, 24 Apr 2013 15:26:53 -0400 Subject: [Bro] =?utf-8?q?x_=3D_x_and_=26synchronized?= In-Reply-To: References: <1366385641.3473.1.camel@solaria> Message-ID: On Fri, 19 Apr 2013 11:42:19 -0400, Seth Hall wrote: > On Apr 19, 2013, at 11:34 AM, David Mandelberg > wrote: > >> Ok. Can you point me to an example script that does explicit data >> sharing? > >  > The metrics framework in 2.1 does it. In 2.2 the intel framework and the > sumstats framework does it (sumstats is the renamed and expanded metrics > framework). Just to make sure I'm understanding this correctly, if the name of an event (lets call it foo) matches the Cluster::worker2manager_events pattern, then "event foo();" on a worker node will queue up the event handlers for foo on both that node and on the manager. Is that right? -- David Eric Mandelberg / dseomn http://david.mandelberg.org/ From carlopmart at gmail.com Wed Apr 24 23:45:55 2013 From: carlopmart at gmail.com (C. L. Martinez) Date: Thu, 25 Apr 2013 06:45:55 +0000 Subject: [Bro] broctl cron locks all bro processes Message-ID: Hi all, Recently, I have installed bro 2.1 in a FreeBSD 9.1 amd64 host (compiled from source). All work as expected except for "broctl cron" job ... I have configured this job to run every 5 min (like Bro's documentation says), but every time locks all bro processes: 41561 ?? I 0:00.01 bash /opt/bro/share/broctl/scripts/run-bro -i em4 -U .status -p broctl -p broctl-live -p standalone -p local -p bro local.bro broctl broctl/standalone broctl/auto 41572 ?? R 215:33.48 /opt/bro/bin/bro -i em4 -U .status -p broctl -p broctl-live -p standalone -p local -p bro local.bro broctl broctl/standalone broctl/auto 41590 ?? SN 0:48.43 /opt/bro/bin/bro -i em4 -U .status -p broctl -p broctl-live -p standalone -p local -p bro local.bro broctl broctl/standalone broctl/auto 41695 ?? I 0:00.00 cron: running job (cron) 41699 ?? Is 0:00.22 /usr/local/bin/python /usr/local/bin/broctl cron (python2.7) 41731 ?? Is 0:00.01 sh 41745 ?? I 0:00.00 cron: running job (cron) 41748 ?? Is 0:00.17 /usr/local/bin/python /usr/local/bin/broctl cron (python2.7) 41774 ?? Is 0:00.01 /usr/sbin/sendmail -FCronDaemon -odi -oem -oi -t 41775 ?? Is 0:00.01 sh 41777 ?? I 0:00.00 cron: running job (cron) 41780 ?? Is 0:00.20 /usr/local/bin/python /usr/local/bin/broctl cron (python2.7) 41813 ?? Is 0:00.01 /usr/sbin/sendmail -FCronDaemon -odi -oem -oi -t 41814 ?? Is 0:00.01 sh 41828 ?? I 0:00.00 cron: running job (cron) 41831 ?? Is 0:00.19 /usr/local/bin/python /usr/local/bin/broctl cron (python2.7) 41857 ?? Is 0:00.01 /usr/sbin/sendmail -FCronDaemon -odi -oem -oi -t 41858 ?? Is 0:00.01 sh 41862 ?? I 0:00.00 cron: running job (cron) 41866 ?? Is 0:00.19 /usr/local/bin/python /usr/local/bin/broctl cron (python2.7) 41898 ?? Is 0:00.01 /usr/sbin/sendmail -FCronDaemon -odi -oem -oi -t 41899 ?? Is 0:00.01 sh 41901 ?? I 0:00.00 cron: running job (cron) 41905 ?? Is 0:00.19 /usr/local/bin/python /usr/local/bin/broctl cron (python2.7) 41942 ?? Is 0:00.01 /usr/sbin/sendmail -FCronDaemon -odi -oem -oi -t 41943 ?? Is 0:00.01 sh 42935 ?? I 0:00.00 cron: running job (cron) 42939 ?? Is 0:00.15 /usr/local/bin/python /usr/local/bin/broctl cron (python2.7) 43003 ?? Is 0:00.01 /usr/sbin/sendmail -FCronDaemon -odi -oem -oi -t 43004 ?? Is 0:00.01 sh If I try to run any broctl command: Welcome to BroControl 1.1 Type "help" for help. [BroControl] > status waiting for lock ..................................cannot get lock How can I debug this?? Is it safe to run "kill -9" over these processes?? And how can I unlock using broctl command?? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130425/bdf6b21a/attachment.html From dnthayer at illinois.edu Thu Apr 25 15:03:58 2013 From: dnthayer at illinois.edu (Daniel Thayer) Date: Thu, 25 Apr 2013 17:03:58 -0500 Subject: [Bro] broctl cron locks all bro processes In-Reply-To: References: Message-ID: <5179A84E.7020106@illinois.edu> Did you check that your cron jobs are running as the same user that you're using to run broctl manually? What happens if you try to run "broctl cron" manually? (you'll first need to kill all of the hung processes and temporarily remove broctl from your crontab) On 04/25/2013 01:45 AM, C. L. Martinez wrote: > Hi all, > > Recently, I have installed bro 2.1 in a FreeBSD 9.1 amd64 host > (compiled from source). All work as expected except for "broctl cron" > job ... I have configured this job to run every 5 min (like Bro's > documentation says), but every time locks all bro processes: > > 41561 ?? I 0:00.01 bash /opt/bro/share/broctl/scripts/run-bro -i > em4 -U .status -p broctl -p broctl-live -p standalone -p local -p bro > local.bro broctl broctl/standalone broctl/auto > 41572 ?? R 215:33.48 /opt/bro/bin/bro -i em4 -U .status -p broctl > -p broctl-live -p standalone -p local -p bro local.bro broctl > broctl/standalone broctl/auto > 41590 ?? SN 0:48.43 /opt/bro/bin/bro -i em4 -U .status -p broctl > -p broctl-live -p standalone -p local -p bro local.bro broctl > broctl/standalone broctl/auto > 41695 ?? I 0:00.00 cron: running job (cron) > 41699 ?? Is 0:00.22 /usr/local/bin/python /usr/local/bin/broctl > cron (python2.7) > 41731 ?? Is 0:00.01 sh > 41745 ?? I 0:00.00 cron: running job (cron) > 41748 ?? Is 0:00.17 /usr/local/bin/python /usr/local/bin/broctl > cron (python2.7) > 41774 ?? Is 0:00.01 /usr/sbin/sendmail -FCronDaemon -odi -oem -oi -t > 41775 ?? Is 0:00.01 sh > 41777 ?? I 0:00.00 cron: running job (cron) > 41780 ?? Is 0:00.20 /usr/local/bin/python /usr/local/bin/broctl > cron (python2.7) > 41813 ?? Is 0:00.01 /usr/sbin/sendmail -FCronDaemon -odi -oem -oi -t > 41814 ?? Is 0:00.01 sh > 41828 ?? I 0:00.00 cron: running job (cron) > 41831 ?? Is 0:00.19 /usr/local/bin/python /usr/local/bin/broctl > cron (python2.7) > 41857 ?? Is 0:00.01 /usr/sbin/sendmail -FCronDaemon -odi -oem -oi -t > 41858 ?? Is 0:00.01 sh > 41862 ?? I 0:00.00 cron: running job (cron) > 41866 ?? Is 0:00.19 /usr/local/bin/python /usr/local/bin/broctl > cron (python2.7) > 41898 ?? Is 0:00.01 /usr/sbin/sendmail -FCronDaemon -odi -oem -oi -t > 41899 ?? Is 0:00.01 sh > 41901 ?? I 0:00.00 cron: running job (cron) > 41905 ?? Is 0:00.19 /usr/local/bin/python /usr/local/bin/broctl > cron (python2.7) > 41942 ?? Is 0:00.01 /usr/sbin/sendmail -FCronDaemon -odi -oem -oi -t > 41943 ?? Is 0:00.01 sh > 42935 ?? I 0:00.00 cron: running job (cron) > 42939 ?? Is 0:00.15 /usr/local/bin/python /usr/local/bin/broctl > cron (python2.7) > 43003 ?? Is 0:00.01 /usr/sbin/sendmail -FCronDaemon -odi -oem -oi -t > 43004 ?? Is 0:00.01 sh > > If I try to run any broctl command: > > Welcome to BroControl 1.1 > > Type "help" for help. > > [BroControl] > status > waiting for lock ..................................cannot get lock > > How can I debug this?? Is it safe to run "kill -9" over these > processes?? And how can I unlock using broctl command?? > > Thanks. > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From shabbir1282 at gmail.com Fri Apr 26 02:49:05 2013 From: shabbir1282 at gmail.com (Shabbir Ahmed) Date: Fri, 26 Apr 2013 14:49:05 +0500 Subject: [Bro] XML-interface Message-ID: Hi, I want to know a communication interface which i can use to recieve triggered messages from bro, bro has detected a malicious traffic and it sends me information via some xml-rpc or other means and then i do action of blocking the traffic. please help, Shabbir. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130426/1bf026a7/attachment.html From robin at icir.org Fri Apr 26 07:21:39 2013 From: robin at icir.org (Robin Sommer) Date: Fri, 26 Apr 2013 07:21:39 -0700 Subject: [Bro] XML-interface In-Reply-To: References: Message-ID: <20130426142139.GJ89784@icir.org> There's a library to exchange events with Bro: http://www.bro.org/documentation/components/broccoli/broccoli-manual.html Robin On Fri, Apr 26, 2013 at 14:49 +0500, Shabbir Ahmed wrote: > Hi, I want to know a communication interface which i can use to recieve > triggered messages from bro, bro has detected a malicious traffic and it > sends me information via some xml-rpc or other means and then i do action > of blocking the traffic. > > please help, > > Shabbir. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From michael.brandeis at ucr.edu Fri Apr 26 09:11:14 2013 From: michael.brandeis at ucr.edu (Michael Brandeis) Date: Fri, 26 Apr 2013 16:11:14 +0000 Subject: [Bro] XML-interface In-Reply-To: <20130426142139.GJ89784@icir.org> References: <20130426142139.GJ89784@icir.org> Message-ID: <2590BD399E5AA74D811796B5DAC9F43328EC9C86@EXCH-MBOX-2.exch.ucr.edu> That link is 404. This may be the right link now http://www.bro.org/sphinx/components/broccoli/README.html http://www.bro.org/sphinx/components/broccoli/broccoli-manual.html http://www.bro.org/sphinx/broccoli-api/broccoli_8h.html -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Robin Sommer Sent: Friday, April 26, 2013 7:22 AM To: Shabbir Ahmed Cc: bro at bro.org Subject: Re: [Bro] XML-interface There's a library to exchange events with Bro: http://www.bro.org/documentation/components/broccoli/broccoli-manual.html Robin On Fri, Apr 26, 2013 at 14:49 +0500, Shabbir Ahmed wrote: > Hi, I want to know a communication interface which i can use to > recieve triggered messages from bro, bro has detected a malicious > traffic and it sends me information via some xml-rpc or other means > and then i do action of blocking the traffic. > > please help, > > Shabbir. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From robin at icir.org Fri Apr 26 10:04:08 2013 From: robin at icir.org (Robin Sommer) Date: Fri, 26 Apr 2013 10:04:08 -0700 Subject: [Bro] XML-interface In-Reply-To: <2590BD399E5AA74D811796B5DAC9F43328EC9C86@EXCH-MBOX-2.exch.ucr.edu> References: <20130426142139.GJ89784@icir.org> <2590BD399E5AA74D811796B5DAC9F43328EC9C86@EXCH-MBOX-2.exch.ucr.edu> Message-ID: <20130426170408.GQ89784@icir.org> Correct. I actually pushed out a modified web site structure shortly after sending that mail, sorry about that. Robin On Fri, Apr 26, 2013 at 16:11 +0000, Michael Brandeis wrote: > That link is 404. This may be the right link now > > http://www.bro.org/sphinx/components/broccoli/README.html > http://www.bro.org/sphinx/components/broccoli/broccoli-manual.html > http://www.bro.org/sphinx/broccoli-api/broccoli_8h.html > > > -----Original Message----- > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Robin Sommer > Sent: Friday, April 26, 2013 7:22 AM > To: Shabbir Ahmed > Cc: bro at bro.org > Subject: Re: [Bro] XML-interface > > There's a library to exchange events with Bro: > > http://www.bro.org/documentation/components/broccoli/broccoli-manual.html > > Robin > > On Fri, Apr 26, 2013 at 14:49 +0500, Shabbir Ahmed wrote: > > > Hi, I want to know a communication interface which i can use to > > recieve triggered messages from bro, bro has detected a malicious > > traffic and it sends me information via some xml-rpc or other means > > and then i do action of blocking the traffic. > > > > please help, > > > > Shabbir. > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From shabbir1282 at gmail.com Sat Apr 27 06:38:28 2013 From: shabbir1282 at gmail.com (Shabbir Ahmed) Date: Sat, 27 Apr 2013 18:38:28 +0500 Subject: [Bro] XML-interface In-Reply-To: <20130426170408.GQ89784@icir.org> References: <20130426142139.GJ89784@icir.org> <2590BD399E5AA74D811796B5DAC9F43328EC9C86@EXCH-MBOX-2.exch.ucr.edu> <20130426170408.GQ89784@icir.org> Message-ID: actually im trying to run bro on openwrt we have successfully ported bro to openwrt but will broccoli work on openwrt? has any one ported it to ...? On Fri, Apr 26, 2013 at 10:04 PM, Robin Sommer wrote: > Correct. I actually pushed out a modified web site structure shortly > after sending that mail, sorry about that. > > Robin > > On Fri, Apr 26, 2013 at 16:11 +0000, Michael Brandeis wrote: > > > That link is 404. This may be the right link now > > > > http://www.bro.org/sphinx/components/broccoli/README.html > > http://www.bro.org/sphinx/components/broccoli/broccoli-manual.html > > http://www.bro.org/sphinx/broccoli-api/broccoli_8h.html > > > > > > -----Original Message----- > > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of > Robin Sommer > > Sent: Friday, April 26, 2013 7:22 AM > > To: Shabbir Ahmed > > Cc: bro at bro.org > > Subject: Re: [Bro] XML-interface > > > > There's a library to exchange events with Bro: > > > > > http://www.bro.org/documentation/components/broccoli/broccoli-manual.html > > > > Robin > > > > On Fri, Apr 26, 2013 at 14:49 +0500, Shabbir Ahmed wrote: > > > > > Hi, I want to know a communication interface which i can use to > > > recieve triggered messages from bro, bro has detected a malicious > > > traffic and it sends me information via some xml-rpc or other means > > > and then i do action of blocking the traffic. > > > > > > please help, > > > > > > Shabbir. > > > > > _______________________________________________ > > > Bro mailing list > > > bro at bro-ids.org > > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > > > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130427/f4cc6dd0/attachment.html From carlopmart at gmail.com Sun Apr 28 23:07:48 2013 From: carlopmart at gmail.com (C. L. Martinez) Date: Mon, 29 Apr 2013 06:07:48 +0000 Subject: [Bro] broctl cron locks all bro processes In-Reply-To: <5179A84E.7020106@illinois.edu> References: <5179A84E.7020106@illinois.edu> Message-ID: On Thu, Apr 25, 2013 at 10:03 PM, Daniel Thayer wrote: > Did you check that your cron jobs are running as the same user > that you're using to run broctl manually? > > What happens if you try to run "broctl cron" manually? (you'll > first need to kill all of the hung processes and temporarily > remove broctl from your crontab) > > > >> First of all. sorry for the later response. I have tried to run broctl cron manually and this is the error message: warning: cannot get list of local IP addresses .. and I don't understand it ... Bro process runs as a root user ... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130429/cae326a7/attachment.html From treed at ultraviolet.org Mon Apr 29 03:33:07 2013 From: treed at ultraviolet.org (Tracy Reed) Date: Mon, 29 Apr 2013 03:33:07 -0700 Subject: [Bro] whitelisting Message-ID: <20130429103151.GA19655@tracyreed.org> Hello all, I am running Bro 2.1 in Security Onion 12.04 and I am very happy with it. This level of detail into what is happening on the network is just amazing! I'm beginning to wonder how I ever did without it for so long. I have an ssh that happens every 5 minutes which causes a lot of noise. I've gone through all of the docs on bro.org and done some googling but can't seem to figure out how to whitelist certain connections so they will not constantly appear in the bro alarm summaries. I did find this, which contains an example for watching ssh to particular hosts which seems related to what I am trying to do: http://www.bro.org/sphinx/quickstart.html#deployment-customization But what I want is somewhat the opposite: I want to ignore/whitelist connections to certain hosts, preferably from certain IP addresses. Can anyone suggest how this would be done? And while I'm writing (and related to another example in the above URL) I get alarms about SSL certs. I would like to add our in-house CA to the list of accepted certs. How can I do this? Thanks for a great tool! -- Tracy Reed -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130429/4d884e4a/attachment.bin From lysemose at gmail.com Mon Apr 29 03:45:06 2013 From: lysemose at gmail.com (Heine Lysemose) Date: Mon, 29 Apr 2013 12:45:06 +0200 Subject: [Bro] whitelisting In-Reply-To: <20130429103151.GA19655@tracyreed.org> References: <20130429103151.GA19655@tracyreed.org> Message-ID: Hi Tracy Have you looked at this, http://code.google.com/p/security-onion/wiki/BPF /Lysemose On Mon, Apr 29, 2013 at 12:33 PM, Tracy Reed wrote: > Hello all, > > I am running Bro 2.1 in Security Onion 12.04 and I am very happy with it. > This > level of detail into what is happening on the network is just amazing! I'm > beginning to wonder how I ever did without it for so long. > > I have an ssh that happens every 5 minutes which causes a lot of noise. > > I've gone through all of the docs on bro.org and done some googling but > can't > seem to figure out how to whitelist certain connections so they will not > constantly appear in the bro alarm summaries. I did find this, which > contains > an example for watching ssh to particular hosts which seems related to > what I > am trying to do: > > http://www.bro.org/sphinx/quickstart.html#deployment-customization > > But what I want is somewhat the opposite: I want to ignore/whitelist > connections to certain hosts, preferably from certain IP addresses. > > Can anyone suggest how this would be done? > > And while I'm writing (and related to another example in the above URL) I > get > alarms about SSL certs. I would like to add our in-house CA to the list of > accepted certs. How can I do this? > > Thanks for a great tool! > > -- > Tracy Reed > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130429/aa127567/attachment.html From scastle at bouldercounty.org Mon Apr 29 07:46:38 2013 From: scastle at bouldercounty.org (Castle, Shane) Date: Mon, 29 Apr 2013 14:46:38 +0000 Subject: [Bro] whitelisting In-Reply-To: References: <20130429103151.GA19655@tracyreed.org> Message-ID: <21DD7C64179C9843B756C6DD491634DB5C8DB5EE@Mailbox1.boco.co.boulder.co.us> Also, the Bro scripting language is very accessible once you learn its syntax. Check out a great repository at https://github.com/bro/ - especially see the "cheat sheet" link there. With this you can roll you own mods without too much trouble. Jump in! -- Shane Castle Data Security Mgr, Boulder County IT -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Heine Lysemose Sent: Monday, April 29, 2013 04:45 To: Tracy Reed Cc: bro at bro.org Subject: Re: [Bro] whitelisting Hi Tracy Have you looked at this, http://code.google.com/p/security-onion/wiki/BPF /Lysemose On Mon, Apr 29, 2013 at 12:33 PM, Tracy Reed wrote: Hello all, I am running Bro 2.1 in Security Onion 12.04 and I am very happy with it. This level of detail into what is happening on the network is just amazing! I'm beginning to wonder how I ever did without it for so long. I have an ssh that happens every 5 minutes which causes a lot of noise. I've gone through all of the docs on bro.org and done some googling but can't seem to figure out how to whitelist certain connections so they will not constantly appear in the bro alarm summaries. I did find this, which contains an example for watching ssh to particular hosts which seems related to what I am trying to do: http://www.bro.org/sphinx/quickstart.html#deployment-customization But what I want is somewhat the opposite: I want to ignore/whitelist connections to certain hosts, preferably from certain IP addresses. Can anyone suggest how this would be done? And while I'm writing (and related to another example in the above URL) I get alarms about SSL certs. I would like to add our in-house CA to the list of accepted certs. How can I do this? Thanks for a great tool! -- Tracy Reed _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From dnthayer at illinois.edu Mon Apr 29 08:25:03 2013 From: dnthayer at illinois.edu (Daniel Thayer) Date: Mon, 29 Apr 2013 10:25:03 -0500 Subject: [Bro] broctl cron locks all bro processes In-Reply-To: References: <5179A84E.7020106@illinois.edu> Message-ID: <517E90CF.9080703@illinois.edu> On 04/29/2013 01:07 AM, C. L. Martinez wrote: > First of all. sorry for the later response. I have tried to run broctl > cron manually and this is the error message: > > warning: cannot get list of local IP addresses > > .. and I don't understand it ... > > Bro process runs as a root user ... > The warning message you saw just means that a script called "local-interfaces" failed for some reason. Try running this script (replace with the bro install prefix directory that you are using): /share/broctl/scripts/local-interfaces and then immediately type "echo $?" From vladg at cmu.edu Mon Apr 29 11:26:25 2013 From: vladg at cmu.edu (Vlad Grigorescu) Date: Mon, 29 Apr 2013 18:26:25 +0000 Subject: [Bro] whitelisting In-Reply-To: <2707_1367231662_r3TAYLth004330_20130429103151.GA19655@tracyreed.org> References: <2707_1367231662_r3TAYLth004330_20130429103151.GA19655@tracyreed.org> Message-ID: <1202BE242E080642B0CD0AD0A03E8552B8DA77@PGH-MSGMB-03.andrew.ad.cmu.edu> Hi Tracy, Here's what my whitelisting looks like in Bro 2.1: > # In my local.bro: > > const external_port_scanners_whitelist = { 8.8.8.8, 8.8.4.4, # Google example > 1.2.3.4 # Another example > }; > > redef Notice::policy += { > [$action = Notice::ACTION_EMAIL, > $pred(n: Notice::Info) = { return n$note == Scan::Port_Scan && n?$src && !(n$src in external_port_scanners_whitelist); } ] > }; It's a bit clunky, but it works. The n?$src clause is used to test whether the src field exists. The Bro Workshop has some great resources for learning about notice handling[1]. Just as a preview, this got cleaned up a bit for the upcoming Bro 2.2: > const external_port_scanners_whitelist = { 8.8.8.8, 8.8.4.4 }; > > hook Notice::policy(n: Notice::Info) &priority=10 > { > if ( n$note == Scan::Port_Scan && n?$src && !(n$src in external_port_scanners_whitelist) ) > { > add n$actions[Notice::ACTION_EMAIL]; > } > > } The new notation might not look like a big benefit in this short example, but it comes in handy as your notice handling becomes more complex. Hope this helps, --Vlad [1] - Exercise 3 at: . On Apr 29, 2013, at 6:33 AM, Tracy Reed wrote: > Hello all, > > I am running Bro 2.1 in Security Onion 12.04 and I am very happy with it. This > level of detail into what is happening on the network is just amazing! I'm > beginning to wonder how I ever did without it for so long. > > I have an ssh that happens every 5 minutes which causes a lot of noise. > > I've gone through all of the docs on bro.org and done some googling but can't > seem to figure out how to whitelist certain connections so they will not > constantly appear in the bro alarm summaries. I did find this, which contains > an example for watching ssh to particular hosts which seems related to what I > am trying to do: > > http://www.bro.org/sphinx/quickstart.html#deployment-customization > > But what I want is somewhat the opposite: I want to ignore/whitelist > connections to certain hosts, preferably from certain IP addresses. > > Can anyone suggest how this would be done? > > And while I'm writing (and related to another example in the above URL) I get > alarms about SSL certs. I would like to add our in-house CA to the list of > accepted certs. How can I do this? > > Thanks for a great tool! > > -- > Tracy Reed > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130429/8227e54a/attachment.bin From christopher.p.crawford at gmail.com Mon Apr 29 12:13:32 2013 From: christopher.p.crawford at gmail.com (Chris Crawford) Date: Mon, 29 Apr 2013 15:13:32 -0400 Subject: [Bro] Syntax Highlighting for Bro IDS Message-ID: It looks like Sublime and Nano support syntax highlighting for Bro IDS: http://liamrandall.com/syntax-highlighting-for-bro-network-programming-language/ http://www.appliednsm.com/syntax-highlighting-for-bro-for-nano/ Is anyone aware of other text editors that also have syntax highlighting support? It would be awesome to see bro language syntax highlighting supported in gVim or notepad++ . -Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130429/c8c51598/attachment.html From seth at icir.org Mon Apr 29 12:23:55 2013 From: seth at icir.org (Seth Hall) Date: Mon, 29 Apr 2013 15:23:55 -0400 Subject: [Bro] Syntax Highlighting for Bro IDS In-Reply-To: References: Message-ID: <36CE4E31-F5EA-48C2-A079-89C6BC5DEDFE@icir.org> On Apr 29, 2013, at 3:13 PM, Chris Crawford wrote: > Is anyone aware of other text editors that also have syntax highlighting support? It would be awesome to see bro language syntax highlighting supported in gVim or notepad++ .  Vim: https://github.com/mephux/bro.vim .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From vallentin at icir.org Mon Apr 29 12:44:00 2013 From: vallentin at icir.org (Matthias Vallentin) Date: Mon, 29 Apr 2013 12:44:00 -0700 Subject: [Bro] Syntax Highlighting for Bro IDS In-Reply-To: <36CE4E31-F5EA-48C2-A079-89C6BC5DEDFE@icir.org> References: <36CE4E31-F5EA-48C2-A079-89C6BC5DEDFE@icir.org> Message-ID: > Vim: https://github.com/mephux/bro.vim We also have an official Pygments lexer [1] which you may use to as a reference. Github uses to it highlight Bro code. Matthias [1] https://bitbucket.org/birkenfeld/pygments-main From srunnels at gmail.com Mon Apr 29 12:56:07 2013 From: srunnels at gmail.com (Scott Runnels) Date: Mon, 29 Apr 2013 15:56:07 -0400 Subject: [Bro] Syntax Highlighting for Bro IDS In-Reply-To: References: <36CE4E31-F5EA-48C2-A079-89C6BC5DEDFE@icir.org> Message-ID: I used Matthias' pygments submission for Bro to do some of the syntax highlighting for the emacs bro-mode https://github.com/srunnels/bro-mode Scott Runnels On Mon, Apr 29, 2013 at 3:44 PM, Matthias Vallentin wrote: > > Vim: https://github.com/mephux/bro.vim > > We also have an official Pygments lexer [1] which you may use to as a > reference. Github uses to it highlight Bro code. > > Matthias > > [1] https://bitbucket.org/birkenfeld/pygments-main > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130429/ac0058db/attachment.html From carlopmart at gmail.com Mon Apr 29 22:55:50 2013 From: carlopmart at gmail.com (C. L. Martinez) Date: Tue, 30 Apr 2013 05:55:50 +0000 Subject: [Bro] broctl cron locks all bro processes In-Reply-To: <517E90CF.9080703@illinois.edu> References: <5179A84E.7020106@illinois.edu> <517E90CF.9080703@illinois.edu> Message-ID: On Mon, Apr 29, 2013 at 3:25 PM, Daniel Thayer wrote: > On 04/29/2013 01:07 AM, C. L. Martinez wrote: > > First of all. sorry for the later response. I have tried to run broctl >> cron manually and this is the error message: >> >> warning: cannot get list of local IP addresses >> >> .. and I don't understand it ... >> >> Bro process runs as a root user ... >> >> > The warning message you saw just means that a script > called "local-interfaces" failed for some reason. > Try running this script (replace with the bro install > prefix directory that you are using): > /share/broctl/scripts/**local-interfaces > > and then immediately type "echo $?" > > Uhmm .. I do not know if this is correct. root at nsm01:~# /opt/bro/share/broctl/scripts/local-interfaces 10.196.0.106 fe80::250:56ff:fe35:22bb 172.17.22.4 fe80::250:56ff:fe2c:47d 172.17.24.4 fe80::250:56ff:fe0e:6fd2 172.17.25.4 fe80::250:56ff:fe1d:a2db fe80::250:56ff:fe1e:94f ::1 fe80::1 127.0.0.1 root at nsm01:~# echo $? 0 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130430/1e3ebf71/attachment.html From blittle766 at gmail.com Tue Apr 30 01:52:26 2013 From: blittle766 at gmail.com (Brian Little) Date: Tue, 30 Apr 2013 09:52:26 +0100 Subject: [Bro] whitelisting In-Reply-To: <1202BE242E080642B0CD0AD0A03E8552B8DA77@PGH-MSGMB-03.andrew.ad.cmu.edu> References: <2707_1367231662_r3TAYLth004330_20130429103151.GA19655@tracyreed.org> <1202BE242E080642B0CD0AD0A03E8552B8DA77@PGH-MSGMB-03.andrew.ad.cmu.edu> Message-ID: Tracy, To add SSL Certificate Authorities to bro, I used this example: http://mailman.icsi.berkeley.edu/pipermail/bro/2012-February/005333.html Bro stores its own list of CAs (generated from Mozilla's list) and the system certificates are ignored. B Little. On Mon, Apr 29, 2013 at 7:26 PM, Vlad Grigorescu wrote: > Hi Tracy, > > Here's what my whitelisting looks like in Bro 2.1: > > > # In my local.bro: > > > > const external_port_scanners_whitelist = { 8.8.8.8, 8.8.4.4, # Google > example > > 1.2.3.4 # Another > example > > }; > > > > redef Notice::policy += { > > [$action = Notice::ACTION_EMAIL, > > $pred(n: Notice::Info) = { return n$note == Scan::Port_Scan && > n?$src && !(n$src in external_port_scanners_whitelist); } ] > > }; > > It's a bit clunky, but it works. The n?$src clause is used to test whether > the src field exists. The Bro Workshop has some great resources for > learning about notice handling[1]. > > Just as a preview, this got cleaned up a bit for the upcoming Bro 2.2: > > > const external_port_scanners_whitelist = { 8.8.8.8, 8.8.4.4 }; > > > > hook Notice::policy(n: Notice::Info) &priority=10 > > { > > if ( n$note == Scan::Port_Scan && n?$src && !(n$src in > external_port_scanners_whitelist) ) > > { > > add n$actions[Notice::ACTION_EMAIL]; > > } > > > > } > > The new notation might not look like a big benefit in this short example, > but it comes in handy as your notice handling becomes more complex. > > Hope this helps, > > --Vlad > > [1] - Exercise 3 at: . > > On Apr 29, 2013, at 6:33 AM, Tracy Reed wrote: > > > Hello all, > > > > I am running Bro 2.1 in Security Onion 12.04 and I am very happy with > it. This > > level of detail into what is happening on the network is just amazing! > I'm > > beginning to wonder how I ever did without it for so long. > > > > I have an ssh that happens every 5 minutes which causes a lot of noise. > > > > I've gone through all of the docs on bro.org and done some googling but > can't > > seem to figure out how to whitelist certain connections so they will not > > constantly appear in the bro alarm summaries. I did find this, which > contains > > an example for watching ssh to particular hosts which seems related to > what I > > am trying to do: > > > > http://www.bro.org/sphinx/quickstart.html#deployment-customization > > > > But what I want is somewhat the opposite: I want to ignore/whitelist > > connections to certain hosts, preferably from certain IP addresses. > > > > Can anyone suggest how this would be done? > > > > And while I'm writing (and related to another example in the above URL) > I get > > alarms about SSL certs. I would like to add our in-house CA to the list > of > > accepted certs. How can I do this? > > > > Thanks for a great tool! > > > > -- > > Tracy Reed > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130430/16b8149d/attachment.html From BrianAllen at wustl.edu Tue Apr 30 13:01:55 2013 From: BrianAllen at wustl.edu (Allen, Brian) Date: Tue, 30 Apr 2013 20:01:55 +0000 Subject: [Bro] quick question In-Reply-To: Message-ID: An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130430/f771c095/attachment.html From scastle at bouldercounty.org Tue Apr 30 13:37:31 2013 From: scastle at bouldercounty.org (Castle, Shane) Date: Tue, 30 Apr 2013 20:37:31 +0000 Subject: [Bro] quick question In-Reply-To: References: Message-ID: <21DD7C64179C9843B756C6DD491634DB5C8DD1B3@Mailbox1.boco.co.boulder.co.us> The Bro documentation area is strangely lacking in some respects. The command you are looking for is bro-cut, a powerful little script that can display a human-readable timestamp and also display only the fields of the log files that you are interested in, and rearrange them if you want. The main thing to remember is that it's a classic stdin->stdout command and does not operate on the filename: "bro-cut -d ts id.orig_h id.resp_h orig_bytes resp_bytes id.resp_p References: <20513_1367354259_r3UKbcNo021279_21DD7C64179C9843B756C6DD491634DB5C8DD1B3@Mailbox1.boco.co.boulder.co.us> Message-ID: <1202BE242E080642B0CD0AD0A03E8552B9260A@PGH-MSGMB-03.andrew.ad.cmu.edu> Hey Brian, As Shane mentioned, bro-cut is one way to make better sense of your logs. There's some additional information about working with your logs available from the Bro Workshop, at: (Exercise 2). If you're new to Bro, I'd definitely recommend checking out the Bro Workshop exercises. There are videos and a lot of hands-on questions and solutions for many day-to-day tasks with Bro. Another way of converting timestamps is simply with the date command (I often resort to this when I just need a single timestamp): > $ date -d @1367064229.605422 > Sat Apr 27 08:03:49 EDT 2013 Hope this helps, --Vlad On Apr 30, 2013, at 4:37 PM, "Castle, Shane" wrote: > The Bro documentation area is strangely lacking in some respects. The command you are looking for is bro-cut, a powerful little script that can display a human-readable timestamp and also display only the fields of the log files that you are interested in, and rearrange them if you want. The main thing to remember is that it's a classic stdin->stdout command and does not operate on the filename: > "bro-cut -d ts id.orig_h id.resp_h orig_bytes resp_bytes id.resp_p > Or, after the archiving has been done: > > ls -1 2013-03-27/conn.*gz | while read fn;do (export TZ=MST7MDT;zcat $fn | bro-cut -d );done | fgrep 192.168.131.135 | less > > This would be if, for instance, your system's clock was running in UTC (which mine is). > > -- > Shane Castle > Data Security Mgr, Boulder County IT > > -----Original Message----- > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Allen, Brian > Sent: Tuesday, April 30, 2013 14:02 > To: bro at bro.org > Subject: [Bro] quick question > > Hi, I installed Bro here and I can already tell it is extremely useful. I'm just learning how to use it so I have lots of questions. Here are a couple quick ones: > > When parsing through the bro log files, how do I turn the timestamp column into something human readable? Where would I go to find this answer on my own? Is there a newbie guide to bro I should be reading? I don't see how to search this mailing list's archives. > > Thanks, > -Brian > > Brian Allen > Network Security Analyst > Washington University > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From dnthayer at illinois.edu Tue Apr 30 13:29:59 2013 From: dnthayer at illinois.edu (Daniel Thayer) Date: Tue, 30 Apr 2013 15:29:59 -0500 Subject: [Bro] quick question In-Reply-To: References: Message-ID: <518029C7.2070407@illinois.edu> On 04/30/2013 03:01 PM, Allen, Brian wrote: > Hi, I installed Bro here and I can already tell it is extremely useful. > I'm just learning how to use it so I have lots of questions. Here are > a couple quick ones: > > When parsing through the bro log files, how do I turn the timestamp > column into something human readable? Where would I go to find this > answer on my own? Is there a newbie guide to bro I should be reading? > I don't see how to search this mailing list's archives. > > Thanks, > -Brian > > Brian Allen > Network Security Analyst > Washington University > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > There's a command "bro-cut" that should be installed in the same directory as "bro". Run bro-cut with an invalid option (such as "bro-cut -x") and it will output a usage message. There are several command-line options to convert timestamps to human-readable format. If you look at any email that was sent out to the mailing list, there is a link to the mailing list archives at the bottom of the message. The Bro documentation is at http://bro.org/documentation/index.html From BrianAllen at wustl.edu Tue Apr 30 13:46:38 2013 From: BrianAllen at wustl.edu (Allen, Brian) Date: Tue, 30 Apr 2013 20:46:38 +0000 Subject: [Bro] quick question In-Reply-To: <21DD7C64179C9843B756C6DD491634DB5C8DD1B3@Mailbox1.boco.co.boulder.co.us> Message-ID: Awesome. Thanks. I'm still wandering around the bro directories learning where everything is. Thanks, -Brian -----Original Message----- From: , Shane Date: Tuesday, April 30, 2013 3:37 PM To: Brian Allen , "'bro at bro.org'" Subject: RE: quick question The Bro documentation area is strangely lacking in some respects. The command you are looking for is bro-cut, a powerful little script that can display a human-readable timestamp and also display only the fields of the log files that you are interested in, and rearrange them if you want. The main thing to remember is that it's a classic stdin->stdout command and does not operate on the filename: "bro-cut -d ts id.orig_h id.resp_h orig_bytes resp_bytes id.resp_p References: Message-ID: <4E8A059B-EC1E-4C1C-B730-577E0093FACA@gmail.com> Brian, bro has some helper utilities called cf and hf ( can't recall if they are already installed with standard dist or not - used to be in aux directory in source) but they are also available here : ftp://ftp.ee.lbl.gov/cf-1.2.4.tar.gz ftp://ftp.ee.lbl.gov/hf-1.3.tar.gz cf converts unix time in human readable format. hf - resolves hostnames one way to search logs is: grep conn.log | cf (make sure cf is your path) Also, from the bro mailing list archives: http://mailman.icsi.berkeley.edu/pipermail/bro/2004-January/001373.html Aashish On Apr 30, 2013, at 1:46 PM, "Allen, Brian" wrote: > Awesome. Thanks. I'm still wandering around the bro directories learning > where everything is. > Thanks, > -Brian > > -----Original Message----- > From: , Shane > Date: Tuesday, April 30, 2013 3:37 PM > To: Brian Allen , "'bro at bro.org'" > Subject: RE: quick question > > The Bro documentation area is strangely lacking in some respects. The > command you are looking for is bro-cut, a powerful little script that can > display a human-readable timestamp and also display only the fields of the > log files that you are interested in, and rearrange them if you want. The > main thing to remember is that it's a classic stdin->stdout command and > does not operate on the filename: > "bro-cut -d ts id.orig_h id.resp_h orig_bytes resp_bytes id.resp_p > > Or, after the archiving has been done: > > ls -1 2013-03-27/conn.*gz | while read fn;do (export TZ=MST7MDT;zcat $fn | > bro-cut -d );done | fgrep 192.168.131.135 | less > > This would be if, for instance, your system's clock was running in UTC > (which mine is). > > -- > Shane Castle > Data Security Mgr, Boulder County IT > > -----Original Message----- > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Allen, > Brian > Sent: Tuesday, April 30, 2013 14:02 > To: bro at bro.org > Subject: [Bro] quick question > > Hi, I installed Bro here and I can already tell it is extremely useful. > I'm just learning how to use it so I have lots of questions. Here are a > couple quick ones: > > When parsing through the bro log files, how do I turn the timestamp column > into something human readable? Where would I go to find this answer on my > own? Is there a newbie guide to bro I should be reading? I don't see how > to search this mailing list's archives. > > Thanks, > -Brian > > Brian Allen > Network Security Analyst > Washington University > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130430/bc52974e/attachment.html From init.conf at gmail.com Tue Apr 30 14:48:22 2013 From: init.conf at gmail.com (Aashish SHARMA) Date: Tue, 30 Apr 2013 14:48:22 -0700 Subject: [Bro] quick question In-Reply-To: References: Message-ID: <6CFE18AD-0A2A-4AF7-930D-2B3931E4119E@gmail.com> Brian, bro has some helper utilities called cf and hf ( can't recall if they are already installed with standard dist or not - used to be in aux directory in source) but they are also available here : ftp://ftp.ee.lbl.gov/cf-1.2.4.tar.gz ftp://ftp.ee.lbl.gov/hf-1.3.tar.gz cf converts unix time in human readable format. hf - resolves hostnames one way to search logs is: grep conn.log | cf (make sure cf is your path) Also, from the bro mailing list archives: http://mailman.icsi.berkeley.edu/pipermail/bro/2004-January/001373.html Aashish On Apr 30, 2013, at 1:46 PM, "Allen, Brian" wrote: > Awesome. Thanks. I'm still wandering around the bro directories learning > where everything is. > Thanks, > -Brian > > -----Original Message----- > From: , Shane > Date: Tuesday, April 30, 2013 3:37 PM > To: Brian Allen , "'bro at bro.org'" > Subject: RE: quick question > > The Bro documentation area is strangely lacking in some respects. The > command you are looking for is bro-cut, a powerful little script that can > display a human-readable timestamp and also display only the fields of the > log files that you are interested in, and rearrange them if you want. The > main thing to remember is that it's a classic stdin->stdout command and > does not operate on the filename: > "bro-cut -d ts id.orig_h id.resp_h orig_bytes resp_bytes id.resp_p > > Or, after the archiving has been done: > > ls -1 2013-03-27/conn.*gz | while read fn;do (export TZ=MST7MDT;zcat $fn | > bro-cut -d );done | fgrep 192.168.131.135 | less > > This would be if, for instance, your system's clock was running in UTC > (which mine is). > > -- > Shane Castle > Data Security Mgr, Boulder County IT > > -----Original Message----- > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Allen, > Brian > Sent: Tuesday, April 30, 2013 14:02 > To: bro at bro.org > Subject: [Bro] quick question > > Hi, I installed Bro here and I can already tell it is extremely useful. > I'm just learning how to use it so I have lots of questions. Here are a > couple quick ones: > > When parsing through the bro log files, how do I turn the timestamp column > into something human readable? Where would I go to find this answer on my > own? Is there a newbie guide to bro I should be reading? I don't see how > to search this mailing list's archives. > > Thanks, > -Brian > > Brian Allen > Network Security Analyst > Washington University > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130430/f65cd650/attachment.html From init.conf at gmail.com Tue Apr 30 14:48:23 2013 From: init.conf at gmail.com (Aashish SHARMA) Date: Tue, 30 Apr 2013 14:48:23 -0700 Subject: [Bro] quick question In-Reply-To: References: Message-ID: Brian, bro has some helper utilities called cf and hf ( can't recall if they are already installed with standard dist or not - used to be in aux directory in source) but they are also available here : ftp://ftp.ee.lbl.gov/cf-1.2.4.tar.gz ftp://ftp.ee.lbl.gov/hf-1.3.tar.gz cf converts unix time in human readable format. hf - resolves hostnames one way to search logs is: grep conn.log | cf (make sure cf is your path) Also, from the bro mailing list archives: http://mailman.icsi.berkeley.edu/pipermail/bro/2004-January/001373.html Aashish On Apr 30, 2013, at 1:46 PM, "Allen, Brian" wrote: > Awesome. Thanks. I'm still wandering around the bro directories learning > where everything is. > Thanks, > -Brian > > -----Original Message----- > From: , Shane > Date: Tuesday, April 30, 2013 3:37 PM > To: Brian Allen , "'bro at bro.org'" > Subject: RE: quick question > > The Bro documentation area is strangely lacking in some respects. The > command you are looking for is bro-cut, a powerful little script that can > display a human-readable timestamp and also display only the fields of the > log files that you are interested in, and rearrange them if you want. The > main thing to remember is that it's a classic stdin->stdout command and > does not operate on the filename: > "bro-cut -d ts id.orig_h id.resp_h orig_bytes resp_bytes id.resp_p > > Or, after the archiving has been done: > > ls -1 2013-03-27/conn.*gz | while read fn;do (export TZ=MST7MDT;zcat $fn | > bro-cut -d );done | fgrep 192.168.131.135 | less > > This would be if, for instance, your system's clock was running in UTC > (which mine is). > > -- > Shane Castle > Data Security Mgr, Boulder County IT > > -----Original Message----- > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Allen, > Brian > Sent: Tuesday, April 30, 2013 14:02 > To: bro at bro.org > Subject: [Bro] quick question > > Hi, I installed Bro here and I can already tell it is extremely useful. > I'm just learning how to use it so I have lots of questions. Here are a > couple quick ones: > > When parsing through the bro log files, how do I turn the timestamp column > into something human readable? Where would I go to find this answer on my > own? Is there a newbie guide to bro I should be reading? I don't see how > to search this mailing list's archives. > > Thanks, > -Brian > > Brian Allen > Network Security Analyst > Washington University > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130430/c379d9bd/attachment.html From jlay at slave-tothe-box.net Tue Apr 30 16:57:04 2013 From: jlay at slave-tothe-box.net (James Lay) Date: Tue, 30 Apr 2013 17:57:04 -0600 Subject: [Bro] quick question In-Reply-To: <518029C7.2070407@illinois.edu> References: <518029C7.2070407@illinois.edu> Message-ID: I made two scripts?findbro and zfindbro?I run findbro in current, and zfindbro in the archive dirs: egrep -h "^#|$1" * | bro-cut -d zegrep -h "^#|$1" * | bro-cut -d Then just put in a host or ip or domain and you'll get everything on it?from dns lookups to connections. James On Apr 30, 2013, at 2:29 PM, Daniel Thayer wrote: > On 04/30/2013 03:01 PM, Allen, Brian wrote: >> Hi, I installed Bro here and I can already tell it is extremely useful. >> I'm just learning how to use it so I have lots of questions. Here are >> a couple quick ones: >> >> When parsing through the bro log files, how do I turn the timestamp >> column into something human readable? Where would I go to find this >> answer on my own? Is there a newbie guide to bro I should be reading? >> I don't see how to search this mailing list's archives. >> >> Thanks, >> -Brian >> >> Brian Allen >> Network Security Analyst >> Washington University >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > There's a command "bro-cut" that should be installed in > the same directory as "bro". Run bro-cut with an invalid > option (such as "bro-cut -x") and it will output a usage > message. There are several command-line options to convert > timestamps to human-readable format. > > If you look at any email that was sent out to the mailing list, > there is a link to the mailing list archives at the bottom > of the message. > > The Bro documentation is at http://bro.org/documentation/index.html > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From shabbir1282 at gmail.com Tue Apr 30 23:57:12 2013 From: shabbir1282 at gmail.com (Shabbir Ahmed) Date: Wed, 1 May 2013 11:57:12 +0500 Subject: [Bro] XML-interface In-Reply-To: References: <20130426142139.GJ89784@icir.org> <2590BD399E5AA74D811796B5DAC9F43328EC9C86@EXCH-MBOX-2.exch.ucr.edu> <20130426170408.GQ89784@icir.org> Message-ID: actually im trying to run bro on openwrt we have successfully ported bro to openwrt but will broccoli work on openwrt? has any one ported it to ...? kindly help im running out of the time on a project. can any one help in porting it to openwrt or any other xml-rpc like interface that is available? thank you, Shabbir. On Sat, Apr 27, 2013 at 6:38 PM, Shabbir Ahmed wrote: > actually im trying to run bro on openwrt we have successfully ported bro > to openwrt but will broccoli work on openwrt? has any one ported it to ...? > > > On Fri, Apr 26, 2013 at 10:04 PM, Robin Sommer wrote: > >> Correct. I actually pushed out a modified web site structure shortly >> after sending that mail, sorry about that. >> >> Robin >> >> On Fri, Apr 26, 2013 at 16:11 +0000, Michael Brandeis wrote: >> >> > That link is 404. This may be the right link now >> > >> > http://www.bro.org/sphinx/components/broccoli/README.html >> > http://www.bro.org/sphinx/components/broccoli/broccoli-manual.html >> > http://www.bro.org/sphinx/broccoli-api/broccoli_8h.html >> > >> > >> > -----Original Message----- >> > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of >> Robin Sommer >> > Sent: Friday, April 26, 2013 7:22 AM >> > To: Shabbir Ahmed >> > Cc: bro at bro.org >> > Subject: Re: [Bro] XML-interface >> > >> > There's a library to exchange events with Bro: >> > >> > >> http://www.bro.org/documentation/components/broccoli/broccoli-manual.html >> > >> > Robin >> > >> > On Fri, Apr 26, 2013 at 14:49 +0500, Shabbir Ahmed wrote: >> > >> > > Hi, I want to know a communication interface which i can use to >> > > recieve triggered messages from bro, bro has detected a malicious >> > > traffic and it sends me information via some xml-rpc or other means >> > > and then i do action of blocking the traffic. >> > > >> > > please help, >> > > >> > > Shabbir. >> > >> > > _______________________________________________ >> > > Bro mailing list >> > > bro at bro-ids.org >> > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > >> > >> >> >> -- >> Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org >> ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130501/22601b84/attachment.html