[Bro] File Extraction

Thu Sep 22 10:50:10 PDT 2016

I did not look through all of your script - the big reason that Bro
currently complains is that you try to load a nonexisting script (there is
no base/protocols/http/file-ident).

Johanna

On Sat, Aug 27, 2016 at 12:37:35PM -0500, al brocino wrote:
> Thanks Johanna,
> 
> *Adding additional information:*
> 
> We are going to upgrade from 2.3.2 but have not yet.
> 
> *I made your recommended change and am still getting the error, see detail
> below:*
> 
> file-extract.bro script
> > global ext_map:table[string] of string = { ["application/x/dosexec"] =
> > "exe",
> you probably want application/x-dosexec here, not x/dosexec. That might
> already be enough to fix this.
> 
> *Changed: *
> 
> file-extract.bro
> global ext_map: table[string] of string = {
>     ["application/x-dosexec"] = "exe",
>     ["text/plain"] = "txt",
>     ["image/jpeg"] = "jpg",
>     ["image/png"] = "png",
>     ["text/html"] = "html",
> } &default ="";
> 
> *Un-comment #@load ./file-extract-http-local.bro and #@load
> ./file-extract-types.bro:*
> 
> _load_.bro
> # File extractions (/application\/.*) -- This has changed significantly in
> 2.2
> @load ./file-extract-http-local.bro
> @load ./file-extract-types.bro
> @load ./bro-file-extract
> 
> * I get this error again:*
> 
>  manager scripts failed.
>    internal warning in
> /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
> 6: Discarded extraneous Broxygen comment: Modified from base scripts to
> extract only from external hosts
>    fatal error in
> /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
> 7: can't find base/protocols/http/file-ident
> proxy scripts failed.
>    internal warning in
> /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
> 6: Discarded extraneous Broxygen comment: Modified from base scripts to
> extract only from external hosts
>    fatal error in
> /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
> 7: can't find base/protocols/http/file-ident
> enm1-eth1-httpproxy scripts failed.
>    internal warning in
> /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
> 6: Discarded extraneous Broxygen comment: Modified from base scripts to
> extract only from external hosts
>    fatal error in
> /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
> 7: can't find base/protocols/http/file-ident
> enm2-eth2-httpinternal scripts failed.
>    internal warning in
> /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
> 6: Discarded extraneous Broxygen comment: Modified from base scripts to
> extract only from external hosts
>    fatal error in
> /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
> 7: can't find base/protocols/http/file-ident
> enm3-eth3-collector scripts failed.
>    internal warning in
> /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
> 6: Discarded extraneous Broxygen comment: Modified from base scripts to
> extract only from external hosts
>    fatal error in
> /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
> 7: can't find base/protocols/http/file-ident
> enm4-eth5-dns scripts failed.
>    internal warning in
> /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
> 6: Discarded extraneous Broxygen comment: Modified from base scripts to
> extract only from external hosts
>    fatal error in
> /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
> 7: can't find base/protocols/http/file-ident
> enm5-eth6-syslog scripts failed.
>    internal warning in
> /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
> 6: Discarded extraneous Broxygen comment: Modified from base scripts to
> extract only from external hosts
>    fatal error in
> /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
> 7: can't find base/protocols/http/file-ident
> 
> *Here's the script that it's failing on:*
> 
> file-extract-http-local.bro
> @load base/protocols/http/main
> @load base/protocols/http/file-ident
> @load base/utils/files
>  module HTTP;
>  export {
>                 ## Pattern of file mime types to extract from HTTP response
> entity bodies.
>                 const extract_file_types_local = /NO_DEFAULT/ &redef;
>                 ## The on-disk prefix for files to be extracted from HTTP
> entity bodies.
>                 const extraction_prefix_local = "http-item" &redef;
> 
>                 redef record Info += {
>                                 ## On-disk file where the response body was
> extracted to.
>                                 extraction_file_local:  file &log &optional;
> 
>                                 ## Indicates if the response body is to be
> extracted or not.  Must be
>                                 ## set before or by the first
> :bro:id:`http_entity_data` event for the
>                                 ## content.
>                                 extract_file_local:     bool &default=F;
>                 };
> }
> 
> # Define local sources to ignore file extract
> global http_extract_file_ignore: set[subnet] = {
>                                                 192.168.2.0.0/24,
>  # Internal Seminal1, trusted destination
>                                                 192.168.1.0/24,
> # Internal Seminal2, trusted destination
> };
> 
> 
> event http_entity_data(c: connection, is_orig: bool, length: count, data:
> string) &priority=-5
>                 {
>                 # Client body extraction is not currently supported in this
> script.
>                 if ( is_orig )
>                                 return;
> 
>                 # We do not want to extract files from internal to internal
> hosts
>                 if ( c$id$resp_h in http_extract_file_ignore )
>                                 return;
> 
>                 if ( c$http$first_chunk )
>                                 {
>                                 if ( c$http?$mime_type &&
>                                      extract_file_types_local in
> c$http$mime_type )
>                                                 {
>                                                 c$http$extract_file_local =
> T;
>                                                 }
> 
>                                 if ( c$http$extract_file_local )
>                                                 {
>                                                 local suffix =
> fmt("%s_%d.dat", is_orig ? "orig" : "resp", c$http_state$current_response);
>                                                 local fname =
> generate_extraction_filename(extraction_prefix_local, c, suffix);
> 
> 
> c$http$extraction_file_local = open(fname);
> 
> enable_raw_output(c$http$extraction_file_local);
>                                                 }
>                                 }
> 
>                 if ( c$http?$extraction_file_local )
>                                 print c$http$extraction_file_local, data;
>                 }
> 
> event http_end_entity(c: connection, is_orig: bool)
>                 {
>                 if ( c$http?$extraction_file_local )
>                                 close(c$http$extraction_file_local);
>                 }
> 
> *Ideas? Thanks!*
> 
> *Al B.*
> *Seminal Networks*
> 
> On Wed, Aug 3, 2016 at 2:47 PM, Johanna Amann <johanna at icir.org> wrote:
> 
> > Hi Al,
> >
> > > I'm new to Bro and using version 2.3.2 and want to extract all the exe's
> > > seen on the network. In bro-file-extract we are using the
> > file-extract.bro
> > > script to try to parse for the exe's (partial of script):
> >
> > First - is there any reason for you to still use 2.3.2? File handling (and
> > a lot of other things) have become more robust in 2.4.
> >
> > In any case...
> >
> > > global ext_map:table[string] of string = {
> > > ["application/x/dosexec"] = "exe",
> >
> > you probably want application/x-dosexec here, not x/dosexec. That might
> > already be enough to fix this.
> >
> > > redef FileExtract::prefix="/var/log/netlogs/bro/file-extracts.bro";
> >
> > This line seems superfluous and wrong, especially since it is redef-ed
> > again two lines later.
> >
> > > redef FileExtract::default_limit = 314572800;
> > > redef FileExtract::prefix = "/var/log/netlogs/bro/file-extracts/";
> > >
> > > We also have the file-extract-http-local.bro set to extract on our
> > network:
> > >
> > > global http_extract_file_ignore: set [subnet] = {
> > >                  10.0.0.0/8,
> > > };
> > >
> >
> > The following seems to talk about files that you modified locally and that
> > do not ship with the Bro distribution. As such, it is really hard to give
> > feedback about it.
> >
> > > We think the problem is that _load_.bro has the file extract commented
> > out
> > > under bro-icmp:
> > > #@load ./file-extract-http-local.bro
> > > #@load ./file-extract-types.bro
> > > @load ./bro-file-extract
> > > When I tried to enable these Bro failed the scripts check with errors
> > like:
> > > internal warning in
> > > /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro,
> > line
> > > 6: Discarded extraneous Broxygen comment: Modified from base scripts to
> > > extract only from external hosts
> > > fatal error in
> > > /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro,
> > line
> > > 7:can't find base/protocols/http/file-ident
> > > I continued to receive these errors and had to back out of removing the
> > > comments
> > >
> > > Under bro-file-extract _load_.bro looks correct:
> > > @load ./file-extract
> > >
> > > What I'm getting in /var/log/netlogs/bro/file-extracts are entries like:
> > > HTTP-F7K52nSzN3h7GNM31.exe
> > > These files occur occasionally I'm not sure what they are.
> >
> > I hope this helps,
> >  Johanna
> >