[Bro] File extraction after checking hash.
fatema bannatwala
fatema.bannatwala at gmail.com
Tue Oct 4 12:49:54 PDT 2016
Hi Justin,
>Do you know that the intel framework supports hashes? If you export a
feed of hashes from CIF you can load that into bro and do the alerting on
known hashes bad in real time.
Yes. And that was the plan, but unfortunately, I couldn't get the list of
the feeds (hashes) pulled down from REN-ISAC , that's interesting that they
provide other feeds but hashes (will ask in REN-ISAC mailing list to
confirm).
But I figured out that you can query their database to get information
about a particular hash.
Also, tried looking for a good open source of feeds for hashes, but
couldn't find it hence don't have any hash feeds currently in intel :(
Thank you for the code, works perfect! :-)
Made a little tweak, replaced network_time() with current_time() function
at both the places.
For some reason I was getting 0.0 as network_time() value when ran the code
in try.bro.org with sample http pcap.
Also, added "local mn_EST = mn + 14400.0; " in midnight() function to get
local EST in quick and dirty way. :) (I know the best way to do ii to use
Seth's plugin, will try that next).
Hence, the complete script looks like this now:
module Uniq_hashes;
redef record Files::Info += {
## Adding a field column of host and uniq_hash to show from where
## the file got downloaded and whether seen first time or duplicate.
host: string &optional &log;
uniq_hash: bool &optional &log ;
};
global SECONDS_IN_DAY = 60*60*24;
global uniq_hashes: set[string] ;
function midnight(): time
{
local now = current_time();
local dt = time_to_double(now);
local mn = double_to_count(dt / SECONDS_IN_DAY) * SECONDS_IN_DAY;
local mn_EST = mn + 14400.0;
return double_to_time(mn_EST);
}
function interval_to_midnight(): interval
{
return midnight() - current_time();
}
event reset_hashes()
{
uniq_hashes = set(); #I think this is the proper way to clear a set?
}
event file_hash(f: fa_file, kind: string, hash: string)
{
#print "file_hash", f$id, kind, hash;
if(f?$http && f$http?$host)
f$info$host = f$http$host;
if(hash in uniq_hashes)
f$info$uniq_hash = F;
else
{
add uniq_hashes[hash];
f$info$uniq_hash = T;
}
}
event bro_init()
{ #print "current_time", current_time();
#print "midnight", midnight();
#print "Time to midnight:", interval_to_midnight();
schedule interval_to_midnight() { reset_hashes()};
}
Thanks,
Fatema.
On Tue, Oct 4, 2016 at 2:40 PM, Azoff, Justin S <jazoff at illinois.edu> wrote:
>
> > On Oct 4, 2016, at 1:39 PM, fatema bannatwala <
> fatema.bannatwala at gmail.com> wrote:
> >
> >
> > And, then I can grep the hashes with uniq_hash=T and query the cif
> server for analysis.
> > Also, can script to get the name of the extracted file from the
> 'extracted' field in files.log with uniq_hash=F
> > and delete that file almost realtime, after Bro has extracted that file.
>
> Do you know that the intel framework supports hashes? If you export a
> feed of hashes from CIF you can load that into bro and do the alerting on
> known hashes bad in real time.
>
> > Before I can test it in production, I want to ask if there is a way I
> can delete the contents of set uniq_hashes right at the midnight
> > so that we can get unique files and hashes on a daily basis logged in
> files.log? I don't want that variable to grow out of bound,
> > consuming lot of memory, hence thought 1 day should be reasonable period
> of time to flush the contents of the set and exact time line
> > will give an idea of uniq hashes queried daily and no. of execs
> extracted on daily basis.
>
> You can probably do it using something like this:
>
> global SECONDS_IN_DAY = 60*60*24;
>
> function midnight(): time
> {
> local now = network_time();
> local dt = time_to_double(now);
> local mn = double_to_count(dt / SECONDS_IN_DAY) * SECONDS_IN_DAY;
> return double_to_time(mn);
> }
>
> function interval_to_midnight(): interval
> {
> return midnight() - network_time();
> }
> event reset_hashes()
> {
> uniq_hashes = set(); #I think this is the proper way to clear a set?
> }
>
> event bro_init()
> {
> print "Time to midnight:", interval_to_midnight();
> schedule interval_to_midnight() { reset_hashes()};
> }
>
> I think that might work properly except for the timezone being in UTC, so
> it might need to be adjusted, or something different altogether
>
> Seth has this plugin: https://github.com/sethhall/bro-approxidate
>
> which would let you do
>
> local md = approxidate("midnight");
>
> If it was packaged for bro-pkg it would be easier to install though :-)
>
> The known hosts/services/certs scripts need a framework to do things like
> this, so 2.6 may end up having this as a built in feature.
>
> --
> - Justin Azoff
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161004/20f757b8/attachment.html
More information about the Bro
mailing list