[Bro] extract jar files from HTTP stream

drum drummachina at tdhack.com
Wed Jan 1 07:26:15 PST 2014


Thank you Doug, that worked. Actually I ended up with following (ugly)
syntax:

root at onion:~# cat /opt/bro/share/bro/file-extraction/extract.bro
global ext_map: table[string] of string = {
    ["application/x-dosexec"] = "exe",
    ["text/plain"] = "txt",
    ["image/jpeg"] = "jpg",
    ["image/png"] = "png",
    ["text/html"] = "html",
} &default ="";

event file_new(f: fa_file)
    {
    #if ( ! f?$mime_type || f$mime_type != "application/x-dosexec" )
    if ( ! f?$mime_type || f$mime_type != "application/jar" )
        return;

    local ext = "";

    if ( f?$mime_type )
        ext = ext_map[f$mime_type];

    local fname = fmt("/nsm/bro/extracted/%s-%s.%s", f$source, f$id, ext);
    Files::add_analyzer(f, Files::ANALYZER_EXTRACT,
[$extract_filename=fname]);
    }

# define file extraction filters
const match_file_source = /HTTP/ |
              /IRC/ |
              /IRC_DATA/ |
              /FTP/ |
              /FTP_DATA/ &redef;

const match_file_mime =   /text\/x-perl/ |
              /text\/x-msdos-batch/ |
              /text\/x-java/ |
              /application\/x-gzip/ |
              /application\/x-bzip2/ |
              /application\/x-dosexec/ |
              /application\/zip/ |
              /application\/jar/ |
              /application\/x-tar/ |
              /application\/x-archive/ |
              /application\/mac-binhex40/ |
              /application\/x-java-keystore/ |
              /application\/x-java-jce-keystore/ |
              /application\/x-executable/ |
              /application\/javascript/ &redef;

# add analyer to file_new event
event file_new(f: fa_file)
    {
    if ( f?$mime_type &&
        match_file_source in f$source &&
        match_file_mime in f$mime_type )
            Files::add_analyzer(f, Files::ANALYZER_EXTRACT);
       }

and I bet it can be written better.

During this excersise I noticed that /nsm/bro/logs/current/files.log was
not present. Found this in google:
https://groups.google.com/forum/#!topic/security-onion/r4eZWOegvsY and
followed suggestions. Indeed, /nsm/bro/logs/current/communication.log file
contained:

1388589086.005591       manager child   -       -       -       error
can't bind to 0.0.0.0:47761, Address already in use

I had to use lsof command to check which process was it and:

root at onion:/nsm/bro/logs/current# lsof -i:47761
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
bro     12253 root    0u  IPv4 300348      0t0  TCP *:47761 (LISTEN)
bro     12253 root    1u  IPv6 300349      0t0  TCP *:47761 (LISTEN)

so I killed it. After doing service nsm restart everything seems to be
working again (logs + file extraction. BTW - jar files are stored without
extension "jar"). But I still can see errors in communication.log:

root at onion:~# cat /nsm/bro/logs/current/communication.log |grep Address
1388589202.005024       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589204.006373       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589235.000845       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589233.001513       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589264.004692       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589266.005739       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589297.004983       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589295.005424       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589328.004598       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589326.005488       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589359.004987       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589357.004749       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589390.004760       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589388.004887       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589419.005759       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589421.005335       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589450.004988       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589452.005818       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589481.001524       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589483.001843       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589512.004547       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589514.004785       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589543.005441       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589545.004584       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589574.005125       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589576.005318       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589605.005628       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589607.004816       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589636.005317       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589638.005756       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589667.005455       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589669.005977       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589700.006115       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589698.004967       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589729.000811       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589731.012333       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589760.005435       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589762.005389       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589791.004834       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589793.005790       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589824.005289       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589822.004770       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use
1388589855.005452       onion-eth1-1    child   -       -       -
error   can't bind to 0.0.0.0:47763, Address already in use
1388589853.006436       proxy   child   -       -       -       error
can't bind to 0.0.0.0:47762, Address already in use


Should I be worried about these errors? I mean, file extraction is working
fine, but maybe other service is not?


PS> Am I mixing two project here? BRO and OnionSecurity?


Pozdrawiam,
drumm


2014/1/1 Doug Burks <doug.burks at gmail.com>

> Hi drum,
>
> Start off with the following:
>
> - edit /opt/bro/share/bro/file-extraction/extract.bro
>
> - change the following line:
>     if ( ! f?$mime_type || f$mime_type != "application/x-dosexec" )
> to:
>     if ( ! f?$mime_type || f$mime_type != "application/jar" )
>
> - run the following:
> sudo broctl install
> sudo broctl restart
>
> Bro should now be extracting jar files to /nsm/bro/extracted/.
>
> Once you have that working, then you should be able to add in Shane's
> match_file_mime to the same script to allow you to extract multiple
> file types.
>
> On Wed, Jan 1, 2014 at 7:39 AM, drum <drummachina at tdhack.com> wrote:
> > Thanks Shane, could you please write step-by-step instruction where
> should I
> > put your code. I have no idea how to port it into my installation. Let's
> > assume I use 'plain' bro (no security onion) installed in /opt/bro. What
> is
> > the next step? BTW - I have these both files (extract.bro and main.bro):
> >
> > root at onion:~# ls -al /opt/bro/share/bro/base/files/extract/main.bro
> > /opt/bro/share/bro/file-extraction/extract.bro
> > -rw-r--r-- 1 root root 2126 Nov  7 18:27
> > /opt/bro/share/bro/base/files/extract/main.bro
> > -rw-r--r-- 1 root root  572 Jan  1 12:26
> > /opt/bro/share/bro/file-extraction/extract.bro
> >
> >
> >
> > Pozdrawiam,
> > drumm
> >
> >
> > 2013/12/30 Shane Filus <filus at psc.edu>
> >>
> >> On 12/29/13 2:25 PM, drum wrote:
> >> > Hello,
> >> >
> >> >
> >> > Is there a tutorial for version 2.2 of BRO? I'd like to understand how
> >> > can I write my own scripts to support extraction of verious files,
> >> > like jar. So far I tried adding "application/jar" (it was logged to
> >> > /nsm/bro/logs/current/files.
> >> > log) as mime type to /opt/bro/share/bro/file-extraction/extract.bro
> >> > file but it seems I have to do something else too as this change is
> >> > not capturing files to /nsm/bro/extracted/ directory.
> >> Hello,
> >>
> >>
> >> Check the second example under 'Adding Analysis'  for a start in file
> >> extraction.
> >> http://www.bro.org/sphinx/frameworks/file-analysis.html#adding-analysis
> >>
> >> Also, not sure how it maps to Sec Onion, but there is
> >> $PREFX/share/bro/base/files/extract/main.bro from a source install.
> >> Might be your 'extract.bro'? I don't that file name in either 2.1 or 2.2
> >> source trees.
> >>
> >> Used the code below to do something similar. There's probably a more
> >> elegant, or efficient solution, but it seems to working as expected,
> >> given the limited testing I've done.
> >>
> >>
> >> # define file extraction filters
> >> const match_file_source = /HTTP/ |
> >>               /IRC/ |
> >>               /IRC_DATA/ |
> >>               /FTP/ |
> >>               /FTP_DATA/ &redef;
> >>
> >> const match_file_mime =   /text\/x-perl/ |
> >>               /text\/x-msdos-batch/ |
> >>               /text\/x-java/ |
> >>               /application\/x-gzip/ |
> >>               /application\/x-bzip2/ |
> >>               /application\/x-dosexec/ |
> >>               /application\/zip/ |
> >>               /application\/jar/ |
> >>               /application\/x-tar/ |
> >>               /application\/x-archive/ |
> >>               /application\/mac-binhex40/ |
> >>               /application\/x-java-keystore/ |
> >>               /application\/x-java-jce-keystore/ |
> >>               /application\/x-executable/ |
> >>               /application\/javascript/ &redef;
> >>
> >> # add analyer to file_new event
> >> event file_new(f: fa_file)
> >>     {
> >>     if ( f?$mime_type &&
> >>         match_file_source in f$source &&
> >>         match_file_mime in f$mime_type )
> >>             Files::add_analyzer(f, Files::ANALYZER_EXTRACT);
> >>        }
> >>
> >>
> >> Thanks,
> >>
> >>
> >> Shane
> >
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> --
> Doug Burks
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140101/754ca399/attachment.html 


More information about the Bro mailing list