[Bro] File extraction after checking hash.

fatema bannatwala fatema.bannatwala at gmail.com
Tue Oct 4 12:49:54 PDT 2016


Hi Justin,

>Do you know that the intel framework supports hashes?  If you export a
feed of hashes from CIF you can load that into bro and do the alerting on
known hashes bad in real time.

Yes. And that was the plan, but unfortunately, I  couldn't get the list of
the feeds (hashes) pulled down from REN-ISAC , that's interesting that they
provide other feeds but hashes (will ask in REN-ISAC mailing list to
confirm).
But I figured out that you can query their database to get information
about a particular hash.
Also,  tried looking for a good open source of feeds for hashes, but
couldn't find it hence don't have any hash feeds currently in intel :(

Thank you for the code, works perfect! :-)
Made a little tweak, replaced network_time()  with current_time() function
at both the places.
For some reason I was getting 0.0 as network_time() value when ran the code
in try.bro.org with sample http pcap.

Also, added "local mn_EST = mn + 14400.0; " in midnight() function to get
local EST in quick and dirty way. :) (I know the best way to do ii to use
Seth's plugin, will try that next).

Hence, the complete script looks like this now:

module Uniq_hashes;

redef record Files::Info += {
    ## Adding a field column of host and uniq_hash to show from where
    ## the file got downloaded and whether seen first time or duplicate.
    host: string &optional &log;
    uniq_hash: bool &optional &log ;
};

global SECONDS_IN_DAY = 60*60*24;
global uniq_hashes: set[string] ;

function midnight(): time
{
    local now = current_time();
    local dt = time_to_double(now);
    local mn =  double_to_count(dt / SECONDS_IN_DAY) * SECONDS_IN_DAY;
    local mn_EST = mn + 14400.0;
    return double_to_time(mn_EST);
}

function interval_to_midnight(): interval
{
    return midnight() - current_time();
}

event reset_hashes()
{
    uniq_hashes = set();  #I think this is the proper way to clear a set?
}

event file_hash(f: fa_file, kind: string, hash: string)
{
    #print "file_hash", f$id, kind, hash;

    if(f?$http && f$http?$host)
      f$info$host = f$http$host;

    if(hash in uniq_hashes)
      f$info$uniq_hash = F;

    else
      {
      add uniq_hashes[hash];
        f$info$uniq_hash = T;
      }

}
event bro_init()
{   #print "current_time", current_time();
    #print "midnight", midnight();
    #print "Time to midnight:", interval_to_midnight();
    schedule interval_to_midnight() { reset_hashes()};
}


Thanks,
Fatema.



On Tue, Oct 4, 2016 at 2:40 PM, Azoff, Justin S <jazoff at illinois.edu> wrote:

>
> > On Oct 4, 2016, at 1:39 PM, fatema bannatwala <
> fatema.bannatwala at gmail.com> wrote:
> >
> >
> > And, then I can grep the hashes with uniq_hash=T and query the cif
> server for analysis.
> > Also, can script to get the name of the extracted file from the
> 'extracted' field in files.log with uniq_hash=F
> > and delete that file almost realtime, after Bro has extracted that file.
>
> Do you know that the intel framework supports hashes?  If you export a
> feed of hashes from CIF you can load that into bro and do the alerting on
> known hashes bad in real time.
>
> > Before I can test it in production, I want to ask if there is a way I
> can delete the contents of set uniq_hashes right at the midnight
> > so that we can get unique files and hashes on a daily basis logged in
> files.log? I don't want that variable to grow out of bound,
> > consuming lot of memory, hence thought 1 day should be reasonable period
> of time to flush the contents of the set and exact time line
> > will give an idea of uniq hashes queried daily and no. of execs
> extracted on daily basis.
>
> You can probably do it using something like this:
>
> global SECONDS_IN_DAY = 60*60*24;
>
> function midnight(): time
> {
>     local now = network_time();
>     local dt = time_to_double(now);
>     local mn =  double_to_count(dt / SECONDS_IN_DAY) * SECONDS_IN_DAY;
>     return double_to_time(mn);
> }
>
> function interval_to_midnight(): interval
> {
>     return midnight() - network_time();
> }
> event reset_hashes()
> {
>     uniq_hashes = set();  #I think this is the proper way to clear a set?
> }
>
> event bro_init()
> {
>     print "Time to midnight:", interval_to_midnight();
>     schedule interval_to_midnight() { reset_hashes()};
> }
>
> I think that might work properly except for the timezone being in UTC, so
> it might need to be adjusted, or something different altogether
>
> Seth has this plugin: https://github.com/sethhall/bro-approxidate
>
> which would let you do
>
> local md = approxidate("midnight");
>
> If it was packaged for bro-pkg it would be easier to install though :-)
>
> The known hosts/services/certs scripts need a framework to do things like
> this, so 2.6 may end up having this as a built in feature.
>
> --
> - Justin Azoff
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161004/20f757b8/attachment.html 


More information about the Bro mailing list