[Bro] Help with searching logs

Michael Bower mbower2 at gmail.com
Wed Apr 3 08:34:47 PDT 2013


Thanks! This will give me something to go on.
On Apr 3, 2013 11:30 AM, "Castle, Shane" <scastle at bouldercounty.org> wrote:

> Seems as though an awk update tightened some of the syntax restrictions.
> This script works:
>
> bro-cut id.orig_h orig_bytes < conn.log             \
>     | sort -t '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n   \
>     | awk 'BEGIN { size=0;host="" }                 \
>            { if (host != $1) {                      \
>                  if (size != 0)                     \
>                      print host, size;              \
>                   host=$1;                          \
>                   if ($2 != "-") {                  \
>                      size=$2 }                      \
>                   else {                            \
>                      size=0; }                      \
>               } else {                              \
>                   if ($2 != "-")                    \
>                      size += $2 }                   \
>             }                                       \
>             END {                                   \
>                 if (size != 0)                      \
>                      print host, size               \
>                 }'                                  \
>     | sort -rnk 2                                   \
>     | head -n 10
>
> The conversations could be done by a script that takes conn.log as input,
> merges orig_h and resp_h into one field while adding together their bytes,
> sorting on that, then putting the result through the same awk script. The
> output might look something like this:
>
> 1.2.3.4|5.6.7.8 123456789
> ...
>
> depending on what you choose for your conjoining character. Also, note
> that if there are multiple conversations between two IP addresses then you
> will be adding up all those conversations and presenting the result as one
> line - that might not be exactly what you are looking for.
>
> Welcome to the world of IDS data mining. Sharpen your
> awk/sort/sql/perl/bash skills - they will come in very handy.
>
> --
> Shane Castle
> Data Security Mgr, Boulder County IT
>
>
> -----Original Message-----
> From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of
> Castle, Shane
> Sent: Wednesday, April 03, 2013 08:59
> To: 'Michael Bower'; 'bro at bro.org'
> Subject: Re: [Bro] Help with searching logs
>
> Hm, I get a syntax error in that script now. Let me figure this out...
>
> --
> Shane Castle
> Data Security Mgr, Boulder County IT
>
>
> -----Original Message-----
> From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of
> Castle, Shane
> Sent: Wednesday, April 03, 2013 08:31
> To: 'Michael Bower'; 'bro at bro.org'
> Subject: Re: [Bro] Help with searching logs
>
> The script is lying to you. Here's the correct script:
>
> bro-cut id.orig_h orig_bytes < conn.log             \
>     | sort -t '.' -k 1,1n -k 2,2n -k 3,3n -k 4,4n   \
>     | awk 'BEGIN { size=0;host="" }                 \
>            { if (host != $1) {                      \
>                  if (size != 0)                     \
>                      print host, size;              \
>                   host=$1;                          \
>                   if ($2 != "-")                    \
>                      size=$2                        \
>                   else                              \
>                      size=0                         \
>               } else                                \
>                   if ($2 != "-")                    \
>                      size += $2                     \
>             }                                       \
>             END {                                   \
>                 if (size != 0)                      \
>                      print host, size               \
>                 }'                                  \
>     | sort -rnk 2                                   \
>     | head -n 10
>
> Since this script summarizes, having timestamps in there would not be
> useful. And, if you want to change the logic to responder, change "orig" to
> "resp" in the first line. Yes, it would be nice to have the top
> conversations, not just the top talkers, which would combine both orig and
> resp, but I'm not sure the result would justify the work.
>
> A few months ago I went through this and we were hoping that the doc would
> have been changed to show a correct script, but it has not been, apparently.
>
> --
> Shane Castle
> Data Security Mgr, Boulder County IT
>
> -----Original Message-----
> From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of
> Michael Bower
> Sent: Tuesday, April 02, 2013 18:19
> To: bro at bro.org
> Subject: [Bro] Help with searching logs
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
>
> Im still learning, so bare with me.  I ran the following command:
>
> bro-cut id.orig_h orig_bytes < conn.log             \
>     | sort                                          \
>     | awk '{ if (host != $1) {                      \
>                  if (size != 0)                     \
>                      print $1, size;                \
>                   host=$1;                          \
>                   size=0                            \
>               } else                                \
>                   size += $2                        \
>             }                                       \
>             END {                                   \
>                 if (size != 0)                      \
>                      print $1, size                 \
>                 }'                                  \
>     | sort -rnk 2                                     \
>     | head -n 10
>
>
> This worked well to show me the top 10 hosts (originators).  What Im
> trying to do is show the top 10 hosts and the time (ts).  Maybe show
> the resp_bytes field too, if that is possible.  Any help would be
> greatly appreciated.
>
> Thanks!
> - --
>
> Mike
>
>
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
>
> iQEcBAEBCgAGBQJRW3WBAAoJEIAKCPjZh/yXUF4H/RhFuVQy6bT3Z8Z1k2oMDBGt
> TYFAfsyeXcnf9dOl3NFGEIlifjDMZ/gK5kBVWo/FYSHGWHrCT0+ICcsjwLroRP/E
> rn1StPS7ozlSiY2ZJSG0UAUCZX0HZ0ujvmNo8UvnoAR75cORq8Y08cU4XpLjqhxc
> d4xu3G+HnhyzjKAiG6xtqDpK2Z3bnjJzyWEqZCoYDzNqtcYnrxYjcKa0kX9rBhUr
> uV6upZ9OHIdf25EYCVfjDrKPSUaRhSAnTVtYE0+OQRA0OPpnG3rLWFSK2yjkTbNG
> AzKXfhJZ0PWmUWkeD6Bzf2TCNqfyLchNSScm2atA/dhTRBV3JhHIhwIcejXr6sk=
> =23Kd
> -----END PGP SIGNATURE-----
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130403/1a604a30/attachment.html 


More information about the Bro mailing list