[Bro] Filter Questions

[Jared Moore]

Hi!
I’m a masters student at the University of Washington and I’m setting up an installation to inform users of a space about digital privacy and teach them about threat modeling by displaying web sites requested in an open wifi network on a few displays. I have an openwrt router using port-mirroring to send a copy of all packets to my linux machine which is running bro to filter the headers and harvest just the source ip, host, uri, and user-agent, but I’m having trouble developing the proper bro code to filter out (ideally) all get requests besides the initial ones when a users clicks a link or types one in the address bar. The solution doesn’t need to be perfect, but I still need to narrow the scope dramatically. The following code is better than nothing, but it doesn’t filter out enough. 

I have a python script extracting the urls from the sql database and loading a few firefox browsers with a new url every couple of seconds and I want the urls queried to be visually similar to what the page a user requests to highlight the vulnerability of unencrypted traffic. I initially tried to extract the files from http connections and then load the html pages in the browsers, but I can’t seem to resolve the original names of the files appropriately. One suggestion I found was to use Xplico <http://xplico.org/>, but I couldn’t get that to work. 

I’m new to bro and appreciate any advice you have!

Thanks,
Jared



@load base/protocols/http

module HttpToSql;

export
{
    redef enum Log::ID += { LOG };
    
    type Request: record
    {
        ts:                 string      &log;
        source:             addr        &log;
        dest:               addr        &log;
        dest_port:          port        &log;
        method:             string      &log &optional;
        host:               string      &log &optional;
        uri:                string      &log &optional;
        url:                string      &log;
        referrer:           string      &log &optional;
        user_agent:         string      &log &optional;
        content_length:     count       &log &optional;
        basic_auth_user:    string      &log &optional;
        trans_depth:   count &log;
    };
}

event bro_init()
{
   Log::create_stream(LOG, [$columns = Request]);
   local sql_filter: Log::Filter =
                   [$name = "http-extracted-sqlite",
		    $path = "/var/db/httptosql",
                    $writer = Log::WRITER_SQLITE,
                    $config = table(["tablename"] = "http")];
    Log::add_filter(LOG, sql_filter);
}

event http_all_headers(c: connection, is_orig: bool, hlist: mime_header_list)
{   
    if (!is_orig)
        return;

    if ( !Site::is_local_addr(c$id$orig_h))
        return;

    if ( !(/^[wW][wW][wW]/ in c$http$host))
        return;

    if ( c$http$trans_depth > 1)
        return;

    local req: Request;
        
    req$ts                                              = strftime("%Y/%m/%d %H:%M:%S", c$http$ts);
    req$trans_depth = c$http$trans_depth;
    req$source                                          = c$id$orig_h;
    req$dest                                            = c$id$resp_h;
    req$dest_port                                       = c$id$resp_p;
    if (c$http?$method) req$method                      = c$http$method;
    if (c$http?$host) req$host                          = c$http$host;
    if (c$http?$uri) req$uri                            = c$http$uri;
    if (c$http?$referrer) req$referrer                  = c$http$referrer;
    if (c$http?$user_agent) req$user_agent              = c$http$user_agent;
    if (c$http?$request_body_len) req$content_length    = c$http$request_body_len;
    if (c$http?$username) req$basic_auth_user           = c$http$username;
     req$url = HTTP::build_url_http(f$http);
    
    Log::write(LOG, req);

}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170802/026e5502/attachment-0001.html