[Bro] extract jar files from HTTP stream
Michael Logoyda
logoyda at gmail.com
Thu Jan 2 07:02:37 PST 2014
Is there a way to name the extracted files based on what IP or domain
name they originated from? In the event file_new(f: fa_file) section, I'm
not able to access anything from f$conns to use for such naming. That would
make the extracted files much more useful.
On Wed, Jan 1, 2014 at 2:20 PM, Doug Burks <doug.burks at gmail.com> wrote:
> Check that your IP address is correct in /opt/bro/etc/node.cfg and
> then run the following:
> sudo broctl install
> sudo reboot
>
> If you continue to have issues, please start a new thread on the
> Security Onion mailing list and we can troubleshoot further there.
>
> Thanks,
> Doug
>
> On Wed, Jan 1, 2014 at 10:26 AM, drum <drummachina at tdhack.com> wrote:
> > Thank you Doug, that worked. Actually I ended up with following (ugly)
> > syntax:
> >
> > root at onion:~# cat /opt/bro/share/bro/file-extraction/extract.bro
> > global ext_map: table[string] of string = {
> > ["application/x-dosexec"] = "exe",
> > ["text/plain"] = "txt",
> > ["image/jpeg"] = "jpg",
> > ["image/png"] = "png",
> > ["text/html"] = "html",
> > } &default ="";
> >
> > event file_new(f: fa_file)
> > {
> > #if ( ! f?$mime_type || f$mime_type != "application/x-dosexec" )
> >
> > if ( ! f?$mime_type || f$mime_type != "application/jar" )
> > return;
> >
> > local ext = "";
> >
> > if ( f?$mime_type )
> > ext = ext_map[f$mime_type];
> >
> > local fname = fmt("/nsm/bro/extracted/%s-%s.%s", f$source, f$id,
> ext);
> > Files::add_analyzer(f, Files::ANALYZER_EXTRACT,
> > [$extract_filename=fname]);
> >
> > }
> >
> > # define file extraction filters
> > const match_file_source = /HTTP/ |
> > /IRC/ |
> > /IRC_DATA/ |
> > /FTP/ |
> > /FTP_DATA/ &redef;
> >
> > const match_file_mime = /text\/x-perl/ |
> > /text\/x-msdos-batch/ |
> > /text\/x-java/ |
> > /application\/x-gzip/ |
> > /application\/x-bzip2/ |
> > /application\/x-dosexec/ |
> > /application\/zip/ |
> > /application\/jar/ |
> > /application\/x-tar/ |
> > /application\/x-archive/ |
> > /application\/mac-binhex40/ |
> > /application\/x-java-keystore/ |
> > /application\/x-java-jce-keystore/ |
> > /application\/x-executable/ |
> > /application\/javascript/ &redef;
> >
> > # add analyer to file_new event
> > event file_new(f: fa_file)
> > {
> > if ( f?$mime_type &&
> > match_file_source in f$source &&
> > match_file_mime in f$mime_type )
> > Files::add_analyzer(f, Files::ANALYZER_EXTRACT);
> > }
> >
> > and I bet it can be written better.
> >
> > During this excersise I noticed that /nsm/bro/logs/current/files.log was
> not
> > present. Found this in google:
> > https://groups.google.com/forum/#!topic/security-onion/r4eZWOegvsY and
> > followed suggestions. Indeed, /nsm/bro/logs/current/communication.log
> file
> > contained:
> >
> > 1388589086.005591 manager child - - - error
> > can't bind to 0.0.0.0:47761, Address already in use
> >
> > I had to use lsof command to check which process was it and:
> >
> > root at onion:/nsm/bro/logs/current# lsof -i:47761
> > COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> > bro 12253 root 0u IPv4 300348 0t0 TCP *:47761 (LISTEN)
> > bro 12253 root 1u IPv6 300349 0t0 TCP *:47761 (LISTEN)
> >
> > so I killed it. After doing service nsm restart everything seems to be
> > working again (logs + file extraction. BTW - jar files are stored without
> > extension "jar"). But I still can see errors in communication.log:
> >
> > root at onion:~# cat /nsm/bro/logs/current/communication.log |grep Address
> > 1388589202.005024 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589204.006373 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589235.000845 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589233.001513 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589264.004692 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589266.005739 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589297.004983 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589295.005424 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589328.004598 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589326.005488 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589359.004987 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589357.004749 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589390.004760 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589388.004887 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589419.005759 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589421.005335 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589450.004988 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589452.005818 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589481.001524 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589483.001843 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589512.004547 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589514.004785 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589543.005441 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589545.004584 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589574.005125 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589576.005318 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589605.005628 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589607.004816 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589636.005317 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589638.005756 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589667.005455 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589669.005977 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589700.006115 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589698.004967 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589729.000811 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589731.012333 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589760.005435 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589762.005389 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589791.004834 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589793.005790 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589824.005289 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589822.004770 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> > 1388589855.005452 onion-eth1-1 child - - -
> > error can't bind to 0.0.0.0:47763, Address already in use
> > 1388589853.006436 proxy child - - - error
> > can't bind to 0.0.0.0:47762, Address already in use
> >
> >
> > Should I be worried about these errors? I mean, file extraction is
> working
> > fine, but maybe other service is not?
> >
> >
> > PS> Am I mixing two project here? BRO and OnionSecurity?
> >
> >
> > Pozdrawiam,
> > drumm
> >
> >
> > 2014/1/1 Doug Burks <doug.burks at gmail.com>
> >>
> >> Hi drum,
> >>
> >> Start off with the following:
> >>
> >> - edit /opt/bro/share/bro/file-extraction/extract.bro
> >>
> >> - change the following line:
> >> if ( ! f?$mime_type || f$mime_type != "application/x-dosexec" )
> >> to:
> >> if ( ! f?$mime_type || f$mime_type != "application/jar" )
> >>
> >> - run the following:
> >> sudo broctl install
> >> sudo broctl restart
> >>
> >> Bro should now be extracting jar files to /nsm/bro/extracted/.
> >>
> >> Once you have that working, then you should be able to add in Shane's
> >> match_file_mime to the same script to allow you to extract multiple
> >> file types.
> >>
> >> On Wed, Jan 1, 2014 at 7:39 AM, drum <drummachina at tdhack.com> wrote:
> >> > Thanks Shane, could you please write step-by-step instruction where
> >> > should I
> >> > put your code. I have no idea how to port it into my installation.
> Let's
> >> > assume I use 'plain' bro (no security onion) installed in /opt/bro.
> What
> >> > is
> >> > the next step? BTW - I have these both files (extract.bro and
> main.bro):
> >> >
> >> > root at onion:~# ls -al /opt/bro/share/bro/base/files/extract/main.bro
> >> > /opt/bro/share/bro/file-extraction/extract.bro
> >> > -rw-r--r-- 1 root root 2126 Nov 7 18:27
> >> > /opt/bro/share/bro/base/files/extract/main.bro
> >> > -rw-r--r-- 1 root root 572 Jan 1 12:26
> >> > /opt/bro/share/bro/file-extraction/extract.bro
> >> >
> >> >
> >> >
> >> > Pozdrawiam,
> >> > drumm
> >> >
> >> >
> >> > 2013/12/30 Shane Filus <filus at psc.edu>
> >> >>
> >> >> On 12/29/13 2:25 PM, drum wrote:
> >> >> > Hello,
> >> >> >
> >> >> >
> >> >> > Is there a tutorial for version 2.2 of BRO? I'd like to understand
> >> >> > how
> >> >> > can I write my own scripts to support extraction of verious files,
> >> >> > like jar. So far I tried adding "application/jar" (it was logged to
> >> >> > /nsm/bro/logs/current/files.
> >> >> > log) as mime type to /opt/bro/share/bro/file-extraction/extract.bro
> >> >> > file but it seems I have to do something else too as this change is
> >> >> > not capturing files to /nsm/bro/extracted/ directory.
> >> >> Hello,
> >> >>
> >> >>
> >> >> Check the second example under 'Adding Analysis' for a start in file
> >> >> extraction.
> >> >>
> http://www.bro.org/sphinx/frameworks/file-analysis.html#adding-analysis
> >> >>
> >> >> Also, not sure how it maps to Sec Onion, but there is
> >> >> $PREFX/share/bro/base/files/extract/main.bro from a source install.
> >> >> Might be your 'extract.bro'? I don't that file name in either 2.1 or
> >> >> 2.2
> >> >> source trees.
> >> >>
> >> >> Used the code below to do something similar. There's probably a more
> >> >> elegant, or efficient solution, but it seems to working as expected,
> >> >> given the limited testing I've done.
> >> >>
> >> >>
> >> >> # define file extraction filters
> >> >> const match_file_source = /HTTP/ |
> >> >> /IRC/ |
> >> >> /IRC_DATA/ |
> >> >> /FTP/ |
> >> >> /FTP_DATA/ &redef;
> >> >>
> >> >> const match_file_mime = /text\/x-perl/ |
> >> >> /text\/x-msdos-batch/ |
> >> >> /text\/x-java/ |
> >> >> /application\/x-gzip/ |
> >> >> /application\/x-bzip2/ |
> >> >> /application\/x-dosexec/ |
> >> >> /application\/zip/ |
> >> >> /application\/jar/ |
> >> >> /application\/x-tar/ |
> >> >> /application\/x-archive/ |
> >> >> /application\/mac-binhex40/ |
> >> >> /application\/x-java-keystore/ |
> >> >> /application\/x-java-jce-keystore/ |
> >> >> /application\/x-executable/ |
> >> >> /application\/javascript/ &redef;
> >> >>
> >> >> # add analyer to file_new event
> >> >> event file_new(f: fa_file)
> >> >> {
> >> >> if ( f?$mime_type &&
> >> >> match_file_source in f$source &&
> >> >> match_file_mime in f$mime_type )
> >> >> Files::add_analyzer(f, Files::ANALYZER_EXTRACT);
> >> >> }
> >> >>
> >> >>
> >> >> Thanks,
> >> >>
> >> >>
> >> >> Shane
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > Bro mailing list
> >> > bro at bro-ids.org
> >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >>
> >>
> >>
> >> --
> >> Doug Burks
> >
> >
>
>
>
> --
> Doug Burks
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140102/df095165/attachment.html
More information about the Bro
mailing list