[Bro] File Extraction
al brocino
al.brocino1 at gmail.com
Sat Aug 27 10:37:35 PDT 2016
Thanks Johanna,
*Adding additional information:*
We are going to upgrade from 2.3.2 but have not yet.
*I made your recommended change and am still getting the error, see detail
below:*
file-extract.bro script
> global ext_map:table[string] of string = { ["application/x/dosexec"] =
> "exe",
you probably want application/x-dosexec here, not x/dosexec. That might
already be enough to fix this.
*Changed: *
file-extract.bro
global ext_map: table[string] of string = {
["application/x-dosexec"] = "exe",
["text/plain"] = "txt",
["image/jpeg"] = "jpg",
["image/png"] = "png",
["text/html"] = "html",
} &default ="";
*Un-comment #@load ./file-extract-http-local.bro and #@load
./file-extract-types.bro:*
_load_.bro
# File extractions (/application\/.*) -- This has changed significantly in
2.2
@load ./file-extract-http-local.bro
@load ./file-extract-types.bro
@load ./bro-file-extract
* I get this error again:*
manager scripts failed.
internal warning in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
6: Discarded extraneous Broxygen comment: Modified from base scripts to
extract only from external hosts
fatal error in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
7: can't find base/protocols/http/file-ident
proxy scripts failed.
internal warning in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
6: Discarded extraneous Broxygen comment: Modified from base scripts to
extract only from external hosts
fatal error in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
7: can't find base/protocols/http/file-ident
enm1-eth1-httpproxy scripts failed.
internal warning in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
6: Discarded extraneous Broxygen comment: Modified from base scripts to
extract only from external hosts
fatal error in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
7: can't find base/protocols/http/file-ident
enm2-eth2-httpinternal scripts failed.
internal warning in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
6: Discarded extraneous Broxygen comment: Modified from base scripts to
extract only from external hosts
fatal error in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
7: can't find base/protocols/http/file-ident
enm3-eth3-collector scripts failed.
internal warning in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
6: Discarded extraneous Broxygen comment: Modified from base scripts to
extract only from external hosts
fatal error in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
7: can't find base/protocols/http/file-ident
enm4-eth5-dns scripts failed.
internal warning in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
6: Discarded extraneous Broxygen comment: Modified from base scripts to
extract only from external hosts
fatal error in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
7: can't find base/protocols/http/file-ident
enm5-eth6-syslog scripts failed.
internal warning in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
6: Discarded extraneous Broxygen comment: Modified from base scripts to
extract only from external hosts
fatal error in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
7: can't find base/protocols/http/file-ident
*Here's the script that it's failing on:*
file-extract-http-local.bro
@load base/protocols/http/main
@load base/protocols/http/file-ident
@load base/utils/files
module HTTP;
export {
## Pattern of file mime types to extract from HTTP response
entity bodies.
const extract_file_types_local = /NO_DEFAULT/ &redef;
## The on-disk prefix for files to be extracted from HTTP
entity bodies.
const extraction_prefix_local = "http-item" &redef;
redef record Info += {
## On-disk file where the response body was
extracted to.
extraction_file_local: file &log &optional;
## Indicates if the response body is to be
extracted or not. Must be
## set before or by the first
:bro:id:`http_entity_data` event for the
## content.
extract_file_local: bool &default=F;
};
}
# Define local sources to ignore file extract
global http_extract_file_ignore: set[subnet] = {
192.168.2.0.0/24,
# Internal Seminal1, trusted destination
192.168.1.0/24,
# Internal Seminal2, trusted destination
};
event http_entity_data(c: connection, is_orig: bool, length: count, data:
string) &priority=-5
{
# Client body extraction is not currently supported in this
script.
if ( is_orig )
return;
# We do not want to extract files from internal to internal
hosts
if ( c$id$resp_h in http_extract_file_ignore )
return;
if ( c$http$first_chunk )
{
if ( c$http?$mime_type &&
extract_file_types_local in
c$http$mime_type )
{
c$http$extract_file_local =
T;
}
if ( c$http$extract_file_local )
{
local suffix =
fmt("%s_%d.dat", is_orig ? "orig" : "resp", c$http_state$current_response);
local fname =
generate_extraction_filename(extraction_prefix_local, c, suffix);
c$http$extraction_file_local = open(fname);
enable_raw_output(c$http$extraction_file_local);
}
}
if ( c$http?$extraction_file_local )
print c$http$extraction_file_local, data;
}
event http_end_entity(c: connection, is_orig: bool)
{
if ( c$http?$extraction_file_local )
close(c$http$extraction_file_local);
}
*Ideas? Thanks!*
*Al B.*
*Seminal Networks*
On Wed, Aug 3, 2016 at 2:47 PM, Johanna Amann <johanna at icir.org> wrote:
> Hi Al,
>
> > I'm new to Bro and using version 2.3.2 and want to extract all the exe's
> > seen on the network. In bro-file-extract we are using the
> file-extract.bro
> > script to try to parse for the exe's (partial of script):
>
> First - is there any reason for you to still use 2.3.2? File handling (and
> a lot of other things) have become more robust in 2.4.
>
> In any case...
>
> > global ext_map:table[string] of string = {
> > ["application/x/dosexec"] = "exe",
>
> you probably want application/x-dosexec here, not x/dosexec. That might
> already be enough to fix this.
>
> > redef FileExtract::prefix="/var/log/netlogs/bro/file-extracts.bro";
>
> This line seems superfluous and wrong, especially since it is redef-ed
> again two lines later.
>
> > redef FileExtract::default_limit = 314572800;
> > redef FileExtract::prefix = "/var/log/netlogs/bro/file-extracts/";
> >
> > We also have the file-extract-http-local.bro set to extract on our
> network:
> >
> > global http_extract_file_ignore: set [subnet] = {
> > 10.0.0.0/8,
> > };
> >
>
> The following seems to talk about files that you modified locally and that
> do not ship with the Bro distribution. As such, it is really hard to give
> feedback about it.
>
> > We think the problem is that _load_.bro has the file extract commented
> out
> > under bro-icmp:
> > #@load ./file-extract-http-local.bro
> > #@load ./file-extract-types.bro
> > @load ./bro-file-extract
> > When I tried to enable these Bro failed the scripts check with errors
> like:
> > internal warning in
> > /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro,
> line
> > 6: Discarded extraneous Broxygen comment: Modified from base scripts to
> > extract only from external hosts
> > fatal error in
> > /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro,
> line
> > 7:can't find base/protocols/http/file-ident
> > I continued to receive these errors and had to back out of removing the
> > comments
> >
> > Under bro-file-extract _load_.bro looks correct:
> > @load ./file-extract
> >
> > What I'm getting in /var/log/netlogs/bro/file-extracts are entries like:
> > HTTP-F7K52nSzN3h7GNM31.exe
> > These files occur occasionally I'm not sure what they are.
>
> I hope this helps,
> Johanna
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160827/b91ded93/attachment-0001.html
More information about the Bro
mailing list