[Bro] Simple way to get a combined unique IP list from an arbitrary date range

Azoff, Justin S jazoff at illinois.edu
Thu Jan 19 10:23:32 PST 2017


> On Jan 19, 2017, at 12:56 PM, Charles Fair <charles.a.fair at gmail.com> wrote:
> 
> Help with this would be greatly appreciated.  I am trying to figure out a simple way to get a combined unique ip list from an arbitrary date range.  I want the unique IP addresses as a single list from the conn.log fields ip.orig_h and ip.resp_h.  Answering questions like give me the unique IPs from the past 7/14/30/60/90 days would be quite tedious this way.  
> 
> I can do it manually as the below example using a temp file for the working data.
> 
> Thanks!
> 
> Chuck

This should do it:

zcat 2016-01-0{1,5}/conn.* | bro-cut id.orig_h id.resp_h -F $'\n' | sort | uniq -c | sort -n > /tmp/alluniqip.txt

If you're going to be doing that a lot, it would make sense to process each day individually (but keep them sorted by ip), then reporting on a date range would just involve doing a k-way merge across multiple days of data.

I use this program as a replacement for sort | uniq -c | sort -n, as long as you have the memory it ends up being a lot faster:

#!/usr/bin/env python
import sys
from collections import defaultdict

c = defaultdict(int)

for line in sys.stdin:
    c[line] += 1

top = sorted(c.items(), key=lambda (k,v): v)
for k, v in top:
    print v, k,


-- 
- Justin Azoff





More information about the Bro mailing list