[Bro] Sanity check - Grabbing platform tokens from browser user agents (was p0f)

[Gary Faulkner]

I've tried the below script now on a Bro 2.2 (release version) cluster 
watching about 8Gbps of traffic and it seems to do what I intended which 
is to create a separate log file that keeps track of IP to OS mappings, 
for a handful of Windows Desktop OS versions, as pulled from user agent 
strings sent over HTTP. In the first full day in production I 
successfully logged around 227,000 unique IP to OS mappings. I have 
since slightly modified it to only log IPs represented in local nets to 
reduce log volume as I'm mostly interested in my own networks.

One issue I'm running into is in keeping track of IP to OS mappings and 
only logging them once per day. I've set an expire timer for 1 day, but 
in production it seems to only keep track of those IPs for the duration 
of a log rotation interval which is set to 20 minutes. I have observed 
that without the expire timer each mapping will log continuously, so it 
appears to get used in some way, but just seems to be tied to log 
rotation instead of the explicit value in the script. I'm guessing I 
need to do something different, but not sure what. Thought? Script is 
pasted below.

================= Begin Script ==============
@load base/utils/site

module BrowserPlatform;

export
{
     # The fully resolved name for this log will be BrowserPlatform::LOG
     redef enum Log::ID += { LOG };

     type Info: record {
         ts:                 time    &log &optional;
         uid:                string  &log &optional;
         host:               addr    &log &optional;
         platform_token:     string  &log &optional;
         unparsed_version:   string  &log &optional;
     };

     # A set of seen IP + OS combinations. Used to prevent logging the 
same combo repeatedly.
     global seen_browser_platforms: set[string] &create_expire = 1.0 day 
&synchronized &redef;
}

event bro_init() &priority=5
     {
     Log::create_stream(BrowserPlatform::LOG,[$columns=Info]);
     }

event http_header(c: connection, is_orig: bool, name: string, value: string)
{
     local platform = "Unknown OS";
     if ( is_orig && Site::is_local_addr(c$id$orig_h) )
         {
         if ( name == "USER-AGENT" && /Windows NT 5.1/ in value )
                 {
                 platform = "Windows XP";
                 }
         else if ( name == "USER-AGENT" && /Windows NT 6.0/ in value )
                 {
                 platform = "Windows Vista";
                 }
         else if ( name == "USER-AGENT" && /Windows NT 6.1/ in value )
                 {
                 platform = "Windows 7";
                 }
         else if ( name == "USER-AGENT" && /Windows NT 6.2/ in value )
                 {
                 platform = "Windows 8";
                 }
         else if ( name == "USER-AGENT" && /Windows NT 6.3/ in value )
                 {
                 platform = "Windows 8.1";
                 }
         }
     local saw = cat(c$id$orig_h,platform); #There is probably a less 
ugly way to do this than cat, but it seems to work
     if ( platform != "Unknown OS" && saw !in seen_browser_platforms )
         {
         local rec: BrowserPlatform::Info = [$ts=network_time(), 
$uid=c$uid, $host=c$id$orig_h, $platform_token=platform, 
$unparsed_version=value];
         Log::write(BrowserPlatform::LOG, rec);
         add seen_browser_platforms[saw];
         }
}

================ End script ==================

Thanks,

Gary Faulkner
UW Madison
Office of Campus Information Security

On 1/29/2014 5:35 PM, Gary Faulkner wrote:
> After being asked if Bro could be used to gather passive intelligence 
> on OS usage I started investigating places to try to identify OS. I 
> initially was looking into p0f and Seth showed me a way to invoke the 
> existing p0f fingerprinting functionality within Bro, but also 
> suggested a slew of other data sources to look at. I wasn't terribly 
> excited with the p0f fingerprint output, and while browser user agents 
> may not be the best data source, I decided to start by looking at 
> platform tokens and reporting on those instead of the p0f data. This 
> is my first-ish bro script and it is by no means a complete script (it 
> only matches a handful of Windows OS). I'm wondering if folks see 
> anything in the attached that would misbehave badly if used on live 
> traffic instead of pcaps?


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6257 bytes
Desc: S/MIME Cryptographic Signature
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140131/595bf545/attachment.bin