[Bro] File Extraction

Jonathon Wright jonathon.s.wright at gmail.com
Thu Dec 4 12:44:35 PST 2014

 Here is a sample of the files.log where there are entries that have no MD5
or SHA1:

1417723198.683744       FAvAKu2jFR7eGbRsH9      CYzgUS3Y6uS9dk3YM9      HTTP    0       MD5,SHA1
image/png       -       2.677866        F       F       60072
7       731951  131224  0       F       -       -       -       -       -
1417723198.684489       FBn4gNzWGTRPkoTm3   CHjRXfnjhGudwL2re       HTTP    0       MD5,SHA1
image/png       -       2.677447        F       F       59842
3       731951  133528  0       F       -       -       -       -       -
1417723201.725784       FTegh14C2bs0OBTXil  Ch6xdv40tNxq4gWOb7      HTTP    0       MD5,SHA1
text/plain      -       0.000000        T       T       44  44
0       0       F       -       bc443e340953993a069985719f1cac76
8b1c00142caf938a917ef5cd04a2977993c7edd3        -       -

Packet drop might be an issue, this sensor is hit pretty hard, and we see
roughly 1-5% packets dropped on tcpdump for example. One thing that sticks
out now as I look at this log is that it says "image/png" and "text/plain"
for mime_type. I configured the script I wrote to only extract exe's:

#make a lookup hash of mimetypes to file extensions
global ext_map: table[string] of string = {
 [“application/x-dosexec”] = “exe”,
 [“text/plain”] = “txt”,
 [“image/jpeg”] = “jpg”,
 [“text/html”] = “html”,
} &default=””;
#create an event to handle new files
event file_new(f: fa_file)
  #ignore files with no mimetype and get only exe’s
  if (! f?$mime_type || f$mime_type != “application/x-dosexec”)

  #otherwise it passes and continues on, so it IS an exe file
  #lets capture it and put it in a directory, name it,& analyze it
  local ext = “”;
  if ( f?$mime_type )
   ext = ext_map[f$mime_type];
     #add <dot>.dead to the end to ensure file cannot be executed accidentally
  local fname = fmt(“/var/data/bro/extracted/%s-%s.%s.dead”, f$source,
f$id, ext);
  Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=fname]);


All are sent to the directory by bro (based on script above that has the
file_new event handler).
As far as the number of bro processes... I honestly don't know. I start bro
via broctl via cron:

@reboot   root      sleep 90 ; /opt/bro/bin/broctl start

However, if I look at the process table, it shows 3 processes:
]# ps -ef | grep -i bro
root      2733     1  0 Nov13 ?        00:00:00 bash
/opt/bro/share/broctl/scripts/run-bro -1 -i em1 -U .status -p broctl -p
broctl-live -p standalone -p local -p bro local.bro broctl
broctl/standalone broctl/auto
root      3425  2733 39 Nov13 ?        8-06:46:02 /opt/bro/bin/bro -i em1
-U .status -p broctl -p broctl-live -p standalone -p local -p bro local.bro
broctl broctl/standalone broctl/auto
root      4073  3425 13 Nov13 ?        2-19:47:05 /opt/bro/bin/bro -i em1
-U .status -p broctl -p broctl-live -p standalone -p local -p bro local.bro
broctl broctl/standalone broctl/auto

Sounds good, I'll keep an eye out.

Let me know on the other 2, thanks!


On Thu, Dec 4, 2014 at 3:48 AM, Seth Hall <seth at icir.org> wrote:

> > On Dec 3, 2014, at 8:58 PM, Jonathon Wright <jonathon.s.wright at gmail.com>
> wrote:
> >
> > 1. I've configured Bro appropriately to extract "exe" mime types from
> the HTTP protocol. It works great. However, the "files.log" only contains
> MD5 and SHA1 entries for some of the files, not all of them. How do I fix
> this so that all of the extracted files have the MD5 and SHA1 entries?
> Hm.  Could you show a line from files.log where a file was extracted but
> you did get hashes?  I suspect this is because the file ended up having
> trouble being transferred which will cause the hash analyzers to stop
> hashing (could be due to a packet drop even).
> >  2. I have analysts that need access to the files
> (/var/data/bro/extracted), but I've noticed that bro creates the files with
> random permissions, either 644 or 600... so they can only access the ones
> with 644. How do I ensure bro extracts the file with the 644 permission set
> on all of them? (see below example)
> This is weird.  I haven’t seen Bro creating extracted files with different
> permissions before.  Are these all dropped directly into the location by
> Bro or copied there later?  All files from the same Bro process or from
> multiple Bro processes?
> > 3. Is there a way to tell bro to run as a different user / group other
> than root? I didn't see any options for it in the bro --help. I would
> assume I would have to give broctl and bro binaries / modules the ownership
> and executable rights by another user, then have bro start up as that new
> user, but wanted to see if there was an easier way. Otherwise I'd have to
> change the default install configuration each time I upgrade.
> Unfortunately, this is an area that hasn’t seen much attention for too
> long.  We’ve been meaning to spend some time adding privilege dropping to
> Bro but it hasn’t happened yet.  This would make Bro similar to many other
> tools that are run by root, but then drop their privileges to another user
> account at runtime, and in our case this would happen after Bro opens any
> interfaces that it needs to sniff on.
> Is there anyone out there in Bro-land that is interested in digging into
> the core enough to add privilege dropping? :)
>   .Seth
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141204/5b47886d/attachment.html 

More information about the Bro mailing list