[Bro] File extraction after checking hash.

[fatema bannatwala]

So here's a simple script that will add a column 'uniq_hash' to the
files.log file that will show
whether bro has seen that hash before (in one day duration).

module Uniq_hashes;

redef record Files::Info += {
    ## Adding a field column of host and uniq_hash to show from where
    ## the file got downloaded and whether seen first time or duplicate.
    host: string &optional &log;
    uniq_hash: bool &optional &log ;
};

#global uniq_hashes: set[string] ;
global uniq_hashes: set[string] &create_expire=1day;

event file_hash(f: fa_file, kind: string, hash: string)
    {
    print "file_hash", f$id, kind, hash;

    if(f?$http && f$http?$host)
      f$info$host = f$http$host;

    if(hash in uniq_hashes)
      f$info$uniq_hash = F;

    else
      {
      add uniq_hashes[hash];
        f$info$uniq_hash = T;
      }

    }


And, then I can grep the hashes with uniq_hash=T and query the cif server
for analysis.
Also, can script to get the name of the extracted file from the 'extracted'
field in files.log with uniq_hash=F
and delete that file almost realtime, after Bro has extracted that file.

Before I can test it in production, I want to ask if there is a way I can
delete the contents of set uniq_hashes right at the midnight
so that we can get unique files and hashes on a daily basis logged in
files.log? I don't want that variable to grow out of bound,
consuming lot of memory, hence thought 1 day should be reasonable period of
time to flush the contents of the set and exact time line
will give an idea of uniq hashes queried daily and no. of execs extracted
on daily basis.

Any help appreciated!

Thanks,
Fatema.

On Tue, Oct 4, 2016 at 10:28 AM, Seth Hall <seth at icir.org> wrote:

>
> > On Oct 4, 2016, at 8:34 AM, fatema bannatwala <
> fatema.bannatwala at gmail.com> wrote:
> >
> > I know that we can query the cif server for a given hash, and get back
> the results with cif confidence rate and other respective fields.
> > Hence will be writing some scripts to get unique hashes and malware
> execs from traffic :)
>
> Awesome!  Anything you can do to package what you're doing well enough
> that other people in higher-ed could use it too would be great.  I've just
> seen so many things in higher-ed that people will create but would be so
> difficult to install anywhere else that they never get used anywhere else.
> It's a shame that so much good work goes to waste because it's never made
> generic.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161004/33b1833f/attachment-0001.html