[Bro] extract jar files from HTTP stream
Doug Burks
doug.burks at gmail.com
Wed Jan 1 13:20:33 PST 2014
Check that your IP address is correct in /opt/bro/etc/node.cfg and
then run the following:
sudo broctl install
sudo reboot
If you continue to have issues, please start a new thread on the
Security Onion mailing list and we can troubleshoot further there.
Thanks,
Doug
On Wed, Jan 1, 2014 at 10:26 AM, drum <drummachina at tdhack.com> wrote:
> Thank you Doug, that worked. Actually I ended up with following (ugly)
> syntax:
>
> root at onion:~# cat /opt/bro/share/bro/file-extraction/extract.bro
> global ext_map: table[string] of string = {
> ["application/x-dosexec"] = "exe",
> ["text/plain"] = "txt",
> ["image/jpeg"] = "jpg",
> ["image/png"] = "png",
> ["text/html"] = "html",
> } &default ="";
>
> event file_new(f: fa_file)
> {
> #if ( ! f?$mime_type || f$mime_type != "application/x-dosexec" )
>
> if ( ! f?$mime_type || f$mime_type != "application/jar" )
> return;
>
> local ext = "";
>
> if ( f?$mime_type )
> ext = ext_map[f$mime_type];
>
> local fname = fmt("/nsm/bro/extracted/%s-%s.%s", f$source, f$id, ext);
> Files::add_analyzer(f, Files::ANALYZER_EXTRACT,
> [$extract_filename=fname]);
>
> }
>
> # define file extraction filters
> const match_file_source = /HTTP/ |
> /IRC/ |
> /IRC_DATA/ |
> /FTP/ |
> /FTP_DATA/ &redef;
>
> const match_file_mime = /text\/x-perl/ |
> /text\/x-msdos-batch/ |
> /text\/x-java/ |
> /application\/x-gzip/ |
> /application\/x-bzip2/ |
> /application\/x-dosexec/ |
> /application\/zip/ |
> /application\/jar/ |
> /application\/x-tar/ |
> /application\/x-archive/ |
> /application\/mac-binhex40/ |
> /application\/x-java-keystore/ |
> /application\/x-java-jce-keystore/ |
> /application\/x-executable/ |
> /application\/javascript/ &redef;
>
> # add analyer to file_new event
> event file_new(f: fa_file)
> {
> if ( f?$mime_type &&
> match_file_source in f$source &&
> match_file_mime in f$mime_type )
> Files::add_analyzer(f, Files::ANALYZER_EXTRACT);
> }
>
> and I bet it can be written better.
>
> During this excersise I noticed that /nsm/bro/logs/current/files.log was not
> present. Found this in google:
> https://groups.google.com/forum/#!topic/security-onion/r4eZWOegvsY and
> followed suggestions. Indeed, /nsm/bro/logs/current/communication.log file
> contained:
>
> 1388589086.005591 manager child - - - error
> can't bind to 0.0.0.0:47761, Address already in use
>
> I had to use lsof command to check which process was it and:
>
> root at onion:/nsm/bro/logs/current# lsof -i:47761
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> bro 12253 root 0u IPv4 300348 0t0 TCP *:47761 (LISTEN)
> bro 12253 root 1u IPv6 300349 0t0 TCP *:47761 (LISTEN)
>
> so I killed it. After doing service nsm restart everything seems to be
> working again (logs + file extraction. BTW - jar files are stored without
> extension "jar"). But I still can see errors in communication.log:
>
> root at onion:~# cat /nsm/bro/logs/current/communication.log |grep Address
> 1388589202.005024 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589204.006373 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589235.000845 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589233.001513 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589264.004692 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589266.005739 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589297.004983 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589295.005424 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589328.004598 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589326.005488 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589359.004987 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589357.004749 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589390.004760 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589388.004887 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589419.005759 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589421.005335 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589450.004988 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589452.005818 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589481.001524 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589483.001843 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589512.004547 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589514.004785 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589543.005441 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589545.004584 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589574.005125 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589576.005318 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589605.005628 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589607.004816 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589636.005317 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589638.005756 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589667.005455 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589669.005977 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589700.006115 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589698.004967 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589729.000811 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589731.012333 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589760.005435 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589762.005389 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589791.004834 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589793.005790 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589824.005289 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589822.004770 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
> 1388589855.005452 onion-eth1-1 child - - -
> error can't bind to 0.0.0.0:47763, Address already in use
> 1388589853.006436 proxy child - - - error
> can't bind to 0.0.0.0:47762, Address already in use
>
>
> Should I be worried about these errors? I mean, file extraction is working
> fine, but maybe other service is not?
>
>
> PS> Am I mixing two project here? BRO and OnionSecurity?
>
>
> Pozdrawiam,
> drumm
>
>
> 2014/1/1 Doug Burks <doug.burks at gmail.com>
>>
>> Hi drum,
>>
>> Start off with the following:
>>
>> - edit /opt/bro/share/bro/file-extraction/extract.bro
>>
>> - change the following line:
>> if ( ! f?$mime_type || f$mime_type != "application/x-dosexec" )
>> to:
>> if ( ! f?$mime_type || f$mime_type != "application/jar" )
>>
>> - run the following:
>> sudo broctl install
>> sudo broctl restart
>>
>> Bro should now be extracting jar files to /nsm/bro/extracted/.
>>
>> Once you have that working, then you should be able to add in Shane's
>> match_file_mime to the same script to allow you to extract multiple
>> file types.
>>
>> On Wed, Jan 1, 2014 at 7:39 AM, drum <drummachina at tdhack.com> wrote:
>> > Thanks Shane, could you please write step-by-step instruction where
>> > should I
>> > put your code. I have no idea how to port it into my installation. Let's
>> > assume I use 'plain' bro (no security onion) installed in /opt/bro. What
>> > is
>> > the next step? BTW - I have these both files (extract.bro and main.bro):
>> >
>> > root at onion:~# ls -al /opt/bro/share/bro/base/files/extract/main.bro
>> > /opt/bro/share/bro/file-extraction/extract.bro
>> > -rw-r--r-- 1 root root 2126 Nov 7 18:27
>> > /opt/bro/share/bro/base/files/extract/main.bro
>> > -rw-r--r-- 1 root root 572 Jan 1 12:26
>> > /opt/bro/share/bro/file-extraction/extract.bro
>> >
>> >
>> >
>> > Pozdrawiam,
>> > drumm
>> >
>> >
>> > 2013/12/30 Shane Filus <filus at psc.edu>
>> >>
>> >> On 12/29/13 2:25 PM, drum wrote:
>> >> > Hello,
>> >> >
>> >> >
>> >> > Is there a tutorial for version 2.2 of BRO? I'd like to understand
>> >> > how
>> >> > can I write my own scripts to support extraction of verious files,
>> >> > like jar. So far I tried adding "application/jar" (it was logged to
>> >> > /nsm/bro/logs/current/files.
>> >> > log) as mime type to /opt/bro/share/bro/file-extraction/extract.bro
>> >> > file but it seems I have to do something else too as this change is
>> >> > not capturing files to /nsm/bro/extracted/ directory.
>> >> Hello,
>> >>
>> >>
>> >> Check the second example under 'Adding Analysis' for a start in file
>> >> extraction.
>> >> http://www.bro.org/sphinx/frameworks/file-analysis.html#adding-analysis
>> >>
>> >> Also, not sure how it maps to Sec Onion, but there is
>> >> $PREFX/share/bro/base/files/extract/main.bro from a source install.
>> >> Might be your 'extract.bro'? I don't that file name in either 2.1 or
>> >> 2.2
>> >> source trees.
>> >>
>> >> Used the code below to do something similar. There's probably a more
>> >> elegant, or efficient solution, but it seems to working as expected,
>> >> given the limited testing I've done.
>> >>
>> >>
>> >> # define file extraction filters
>> >> const match_file_source = /HTTP/ |
>> >> /IRC/ |
>> >> /IRC_DATA/ |
>> >> /FTP/ |
>> >> /FTP_DATA/ &redef;
>> >>
>> >> const match_file_mime = /text\/x-perl/ |
>> >> /text\/x-msdos-batch/ |
>> >> /text\/x-java/ |
>> >> /application\/x-gzip/ |
>> >> /application\/x-bzip2/ |
>> >> /application\/x-dosexec/ |
>> >> /application\/zip/ |
>> >> /application\/jar/ |
>> >> /application\/x-tar/ |
>> >> /application\/x-archive/ |
>> >> /application\/mac-binhex40/ |
>> >> /application\/x-java-keystore/ |
>> >> /application\/x-java-jce-keystore/ |
>> >> /application\/x-executable/ |
>> >> /application\/javascript/ &redef;
>> >>
>> >> # add analyer to file_new event
>> >> event file_new(f: fa_file)
>> >> {
>> >> if ( f?$mime_type &&
>> >> match_file_source in f$source &&
>> >> match_file_mime in f$mime_type )
>> >> Files::add_analyzer(f, Files::ANALYZER_EXTRACT);
>> >> }
>> >>
>> >>
>> >> Thanks,
>> >>
>> >>
>> >> Shane
>> >
>> >
>> >
>> > _______________________________________________
>> > Bro mailing list
>> > bro at bro-ids.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>>
>> --
>> Doug Burks
>
>
--
Doug Burks
More information about the Bro
mailing list