[Bro] File extraction after checking hash.

[Azoff, Justin S]

> On Oct 4, 2016, at 1:39 PM, fatema bannatwala <fatema.bannatwala at gmail.com> wrote:
> 
> 
> And, then I can grep the hashes with uniq_hash=T and query the cif server for analysis.
> Also, can script to get the name of the extracted file from the 'extracted' field in files.log with uniq_hash=F
> and delete that file almost realtime, after Bro has extracted that file.

Do you know that the intel framework supports hashes?  If you export a feed of hashes from CIF you can load that into bro and do the alerting on known hashes bad in real time.

> Before I can test it in production, I want to ask if there is a way I can delete the contents of set uniq_hashes right at the midnight
> so that we can get unique files and hashes on a daily basis logged in files.log? I don't want that variable to grow out of bound, 
> consuming lot of memory, hence thought 1 day should be reasonable period of time to flush the contents of the set and exact time line
> will give an idea of uniq hashes queried daily and no. of execs extracted on daily basis.

You can probably do it using something like this:

global SECONDS_IN_DAY = 60*60*24;

function midnight(): time
{
    local now = network_time();
    local dt = time_to_double(now);
    local mn =  double_to_count(dt / SECONDS_IN_DAY) * SECONDS_IN_DAY;
    return double_to_time(mn);
}

function interval_to_midnight(): interval
{
    return midnight() - network_time();
}
event reset_hashes()
{
    uniq_hashes = set();  #I think this is the proper way to clear a set?
}

event bro_init()
{
    print "Time to midnight:", interval_to_midnight();
    schedule interval_to_midnight() { reset_hashes()};
}

I think that might work properly except for the timezone being in UTC, so it might need to be adjusted, or something different altogether

Seth has this plugin: https://github.com/sethhall/bro-approxidate

which would let you do

local md = approxidate("midnight");

If it was packaged for bro-pkg it would be easier to install though :-)

The known hosts/services/certs scripts need a framework to do things like this, so 2.6 may end up having this as a built in feature.

-- 
- Justin Azoff