[Bro] SMTP attachments and files from other ports/protocols

Adam Hall abhall1 at yahoo.com
Thu Jul 9 15:33:42 PDT 2015


Good Evening Daniel,
   You can use the below to extract files only from SMTP using file_sniff in bro2.4.
   I added in the line (    if( f$source != 'SMTP' ) return;       ) and didn't change the rest of my code as this is live code I use.
  This will ignore anything that is not SMTP and then ignore anything that does not have a mime_type set.
  Please feel free to respond on the boards if you have any problems with the code below.
RedLight

export {
const mimes = /application\/x-dosexec/ | /application\/pdf/ | /application\/java-archive/ &redef;
}

event file_sniff(f: fa_file, meta: fa_metadata)
    {   
        if( f$source != 'SMTP' ) return;
        
        if ( ! meta?$mime_type ) return;

    if ( mimes in meta$mime_type )
        {
        local app = sub( meta$mime_type, /\//, "_");
        local tsp = sub_bytes(cat(f$info$ts), 1, 10);
        local fname = fmt("%s-%s.%s", f$id,app,tsp);
        Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename= fname]);
        }
}


      From: "bro-request at bro.org" <bro-request at bro.org>
 To: bro at bro.org 
 Sent: Thursday, July 9, 2015 5:05 PM
 Subject: Bro Digest, Vol 111, Issue 16
   
Send Bro mailing list submissions to
    bro at bro.org

To subscribe or unsubscribe via the World Wide Web, visit
    http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
or, via email, send a message with subject or body 'help' to
    bro-request at bro.org

You can reach the person managing the list at
    bro-owner at bro.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Bro digest..."


Today's Topics:

  1. SMTP attachments and files from other ports/protocols
      (Sanner, Daniel A)
  2. Re: SMTP attachments and files from other ports/protocols
      (Donaldson, John)
  3. Converting fa_file to file_sniff (James Lay)
  4. Re: Converting fa_file to file_sniff (Josh Liburdi)
  5. Re: Converting fa_file to file_sniff (Doug Burks)
  6. [SOLVED] Re:  Converting fa_file to file_sniff (James Lay)


----------------------------------------------------------------------

Message: 1
Date: Thu, 9 Jul 2015 19:42:03 +0000
From: "Sanner, Daniel A" <daniel.sanner at pnnl.gov>
Subject: [Bro] SMTP attachments and files from other ports/protocols
To: "bro at bro.org" <bro at bro.org>
Message-ID: <D1C41C99.596%daniel.sanner at pnnl.gov>
Content-Type: text/plain; charset="us-ascii"

Is there a script that exists or that can be modified to be able to capture/download attachments that are detected?
Specifically, looking for SMTP attachments in e-mails.  However, files in the Files.log could be helpful too.
Right now, the best I can figure is that the SMTP log is just a copy of e-mail headers and nothing more.
The files.log only has MD5 and/or SHA1 hashes, but no details about file name, type, or even the file itself.
We had heard that there are tools out there like Bro (if not Bro itself) that can do this.
If Bro has this capability, storage media requirements is not an issue.
Any additional information that I can glean and add to the logs could be helpful.

Thanks in advance,
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150709/b7edad5b/attachment-0001.html 

------------------------------

Message: 2
Date: Thu, 9 Jul 2015 20:00:57 +0000
From: "Donaldson, John" <donaldson8 at llnl.gov>
Subject: Re: [Bro] SMTP attachments and files from other
    ports/protocols
To: "Sanner, Daniel A" <daniel.sanner at pnnl.gov>, "bro at bro.org"
    <bro at bro.org>
Message-ID:
    <C623401B7419BD4E8A983814FEFA02F6BFD05DA8 at PRDEXMBX-07.the-lab.llnl.gov>
    
Content-Type: text/plain; charset="us-ascii"

Dan,

If you take a look at the HTTP monitoring example here (https://www.bro.org/sphinx/httpmonitor/index.html) and modify the final example ("Inspecting Files") to use "SMTP" instead of "HTTP", you should be pretty close to getting this to work.

v/r

John Donaldson

From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Sanner, Daniel A
Sent: Thursday, July 09, 2015 2:42 PM
To: bro at bro.org
Subject: [Bro] SMTP attachments and files from other ports/protocols

Is there a script that exists or that can be modified to be able to capture/download attachments that are detected?
Specifically, looking for SMTP attachments in e-mails.  However, files in the Files.log could be helpful too.
Right now, the best I can figure is that the SMTP log is just a copy of e-mail headers and nothing more.
The files.log only has MD5 and/or SHA1 hashes, but no details about file name, type, or even the file itself.
We had heard that there are tools out there like Bro (if not Bro itself) that can do this.
If Bro has this capability, storage media requirements is not an issue.
Any additional information that I can glean and add to the logs could be helpful.

Thanks in advance,
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150709/cd144d3d/attachment-0001.html 

------------------------------

Message: 3
Date: Thu, 09 Jul 2015 14:07:56 -0600
From: James Lay <jlay at slave-tothe-box.net>
Subject: [Bro] Converting fa_file to file_sniff
To: bro at bro.org
Message-ID: <1da719006cbd11ec1336ecfb89e5059d at localhost>
Content-Type: text/plain; charset=UTF-8; format=flowed

So per:

https://www.bro.org/download/NEWS.bro.html

"Removed fa_file record?s mime_type and mime_types fields. The event 
file_sniff has been added which provides the same information. The 
mime_type field of Files::Info also still has this info."

I have a script...smtp-file-extract.bro:

global ext_map: table[string] of string = {
        ["application/x-dosexec"] = "exe",
        ["application/zip"] = "zip",
        ["application/msword"] = "xls",
};

event file_new(f: fa_file)
        {
        if ( f$source != "SMTP" )
                return;

        if ( ! f?$mime_type || f$mime_type !in ext_map )
                return;

        local ext = "";

        if ( f?$mime_type )
                ext = ext_map[f$mime_type];

        local fname = fmt("%s-%s.%s", f$source, f$id, ext);
        Files::add_analyzer(f, Files::ANALYZER_EXTRACT, 
[$extract_filename=fname]);
}

which while not perfects gets what I need done.  This is now broken with 
2.4, as expected, however I'm at a loss on how to fix this.  Again, per 
the NEWS link above:

"The earliest point that new mime type information is available is in 
the file_sniff event which comes after the file_new and 
file_over_new_connection events. Scripts which inspected mime type info 
within those events will need to be adapted. (Note: for users that 
worked w/ versions of Bro from git, for a while there was also an event 
called file_mime_type which is now replaced with the file_sniff event)."

Awesome.  How do I adapt this?  Not sure where to look for changing 
this.  Thank you.

James




------------------------------

Message: 4
Date: Thu, 9 Jul 2015 16:13:20 -0400
From: Josh Liburdi <liburdi.joshua at gmail.com>
Subject: Re: [Bro] Converting fa_file to file_sniff
To: jlay at slave-tothe-box.net
Cc: bro <bro at bro.org>
Message-ID:
    <CANrCRiG7JLU-25bJa4rvOXD4d9g8oetEQLwRD7tn0SsfYqmhQg at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

James,

Take a look at this script, it's almost identical to yours:
https://github.com/bro/bro/blob/2b1cd66f17194a30b90490965cbdffdd71c18c09/doc/httpmonitor/file_extraction.bro

Josh

On Thu, Jul 9, 2015 at 4:07 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> So per:
>
> https://www.bro.org/download/NEWS.bro.html
>
> "Removed fa_file record?s mime_type and mime_types fields. The event
> file_sniff has been added which provides the same information. The
> mime_type field of Files::Info also still has this info."
>
> I have a script...smtp-file-extract.bro:
>
> global ext_map: table[string] of string = {
>          ["application/x-dosexec"] = "exe",
>          ["application/zip"] = "zip",
>          ["application/msword"] = "xls",
> };
>
> event file_new(f: fa_file)
>          {
>          if ( f$source != "SMTP" )
>                  return;
>
>          if ( ! f?$mime_type || f$mime_type !in ext_map )
>                  return;
>
>          local ext = "";
>
>          if ( f?$mime_type )
>                  ext = ext_map[f$mime_type];
>
>          local fname = fmt("%s-%s.%s", f$source, f$id, ext);
>          Files::add_analyzer(f, Files::ANALYZER_EXTRACT,
> [$extract_filename=fname]);
> }
>
> which while not perfects gets what I need done.  This is now broken with
> 2.4, as expected, however I'm at a loss on how to fix this.  Again, per
> the NEWS link above:
>
> "The earliest point that new mime type information is available is in
> the file_sniff event which comes after the file_new and
> file_over_new_connection events. Scripts which inspected mime type info
> within those events will need to be adapted. (Note: for users that
> worked w/ versions of Bro from git, for a while there was also an event
> called file_mime_type which is now replaced with the file_sniff event)."
>
> Awesome.  How do I adapt this?  Not sure where to look for changing
> this.  Thank you.
>
> James
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



------------------------------

Message: 5
Date: Thu, 9 Jul 2015 16:17:51 -0400
From: Doug Burks <doug.burks at gmail.com>
Subject: Re: [Bro] Converting fa_file to file_sniff
To: James Lay <jlay at slave-tothe-box.net>
Cc: "<bro at bro.org>" <bro at bro.org>
Message-ID:
    <CAK8kjrD4O7z1+JV=nhwWGJFHB2v53X5OoQjjbhbuNZSprE50qg at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Hi James,

Here's how I adapted a similar script:
https://github.com/Security-Onion-Solutions/securityonion-bro-scripts/blob/master/file-extraction/extract.bro

On Thu, Jul 9, 2015 at 4:07 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> So per:
>
> https://www.bro.org/download/NEWS.bro.html
>
> "Removed fa_file record?s mime_type and mime_types fields. The event
> file_sniff has been added which provides the same information. The
> mime_type field of Files::Info also still has this info."
>
> I have a script...smtp-file-extract.bro:
>
> global ext_map: table[string] of string = {
>          ["application/x-dosexec"] = "exe",
>          ["application/zip"] = "zip",
>          ["application/msword"] = "xls",
> };
>
> event file_new(f: fa_file)
>          {
>          if ( f$source != "SMTP" )
>                  return;
>
>          if ( ! f?$mime_type || f$mime_type !in ext_map )
>                  return;
>
>          local ext = "";
>
>          if ( f?$mime_type )
>                  ext = ext_map[f$mime_type];
>
>          local fname = fmt("%s-%s.%s", f$source, f$id, ext);
>          Files::add_analyzer(f, Files::ANALYZER_EXTRACT,
> [$extract_filename=fname]);
> }
>
> which while not perfects gets what I need done.  This is now broken with
> 2.4, as expected, however I'm at a loss on how to fix this.  Again, per
> the NEWS link above:
>
> "The earliest point that new mime type information is available is in
> the file_sniff event which comes after the file_new and
> file_over_new_connection events. Scripts which inspected mime type info
> within those events will need to be adapted. (Note: for users that
> worked w/ versions of Bro from git, for a while there was also an event
> called file_mime_type which is now replaced with the file_sniff event)."
>
> Awesome.  How do I adapt this?  Not sure where to look for changing
> this.  Thank you.
>
> James
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com



------------------------------

Message: 6
Date: Thu, 09 Jul 2015 15:05:20 -0600
From: James Lay <jlay at slave-tothe-box.net>
Subject: [Bro] [SOLVED] Re:  Converting fa_file to file_sniff
To: bro at bro.org
Message-ID: <71e055751f7e4f2b655cb61830e78e78 at localhost>
Content-Type: text/plain; charset=UTF-8; format=flowed

On 2015-07-09 02:17 PM, Doug Burks wrote:
> Hi James,
> 
> Here's how I adapted a similar script:
> https://github.com/Security-Onion-Solutions/securityonion-bro-scripts/blob/master/file-extraction/extract.bro
> 
> On Thu, Jul 9, 2015 at 4:07 PM, James Lay <jlay at slave-tothe-box.net> 
> wrote:
>> So per:
>> 
>> https://www.bro.org/download/NEWS.bro.html
>> 
>> "Removed fa_file record?s mime_type and mime_types fields. The event
>> file_sniff has been added which provides the same information. The
>> mime_type field of Files::Info also still has this info."
>> 
>> I have a script...smtp-file-extract.bro:
>> 
>> global ext_map: table[string] of string = {
>>          ["application/x-dosexec"] = "exe",
>>          ["application/zip"] = "zip",
>>          ["application/msword"] = "xls",
>> };
>> 
>> event file_new(f: fa_file)
>>          {
>>          if ( f$source != "SMTP" )
>>                  return;
>> 
>>          if ( ! f?$mime_type || f$mime_type !in ext_map )
>>                  return;
>> 
>>          local ext = "";
>> 
>>          if ( f?$mime_type )
>>                  ext = ext_map[f$mime_type];
>> 
>>          local fname = fmt("%s-%s.%s", f$source, f$id, ext);
>>          Files::add_analyzer(f, Files::ANALYZER_EXTRACT,
>> [$extract_filename=fname]);
>> }
>> 
>> which while not perfects gets what I need done.  This is now broken 
>> with
>> 2.4, as expected, however I'm at a loss on how to fix this.  Again, 
>> per
>> the NEWS link above:
>> 
>> "The earliest point that new mime type information is available is in
>> the file_sniff event which comes after the file_new and
>> file_over_new_connection events. Scripts which inspected mime type 
>> info
>> within those events will need to be adapted. (Note: for users that
>> worked w/ versions of Bro from git, for a while there was also an 
>> event
>> called file_mime_type which is now replaced with the file_sniff 
>> event)."
>> 
>> Awesome.  How do I adapt this?  Not sure where to look for changing
>> this.  Thank you.
>> 
>> James

Thanks Gents...appreciate the fast help.  For those keeping score at 
home here's what works:

global ext_map: table[string] of string = {
        ["application/x-dosexec"] = "exe",
        ["application/zip"] = "zip",
        ["application/msword"] = "xls",
};

event file_sniff(f: fa_file, meta: fa_metadata)
        {
        if ( f$source != "SMTP" )
                return;

        if ( ! meta?$mime_type || meta$mime_type !in ext_map )
                return;

        local ext = "";

        if ( meta?$mime_type )
                ext = ext_map[meta$mime_type];

        local fname = fmt("%s-%s.%s", f$source, f$id, ext);
        Files::add_analyzer(f, Files::ANALYZER_EXTRACT, 
[$extract_filename=fname]);
}


Have to say I would have never figured this out just by reading the Bro 
documentation....thanks to you both again.

James


------------------------------

_______________________________________________
Bro mailing list
Bro at bro.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


End of Bro Digest, Vol 111, Issue 16
************************************


  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150709/e05fe6f4/attachment-0001.html 


More information about the Bro mailing list