[Bro] http analyzer and de-obfuscating the payload

[Seth Hall]

On Oct 12, 2010, at 11:30 PM, Peter Erickson wrote:
> I didn't think of using broccoli to feed it back into the system. I'll have to reconsider my current setup to see if that makes sense. It works now without it, but there is definitely a benefit of having additional information within bro's log files.

It's especially useful when you're using Bro on live network because the information gained from the external analysis could feed back into Bro to change it's behavior if the same thing is seen again.  As a personal exercise, I'm going to start including concrete examples when I talk about techniques in Bro. :)  So, here's my concrete example...

Bro identifies a Windows executable being downloaded over HTTP so it begins calculating an MD5 sum of the bytes being transferred.  It could also save the file to disk.  When the file is done being transferred, the on-disk filename could be sent off to an external process which grabs the file does something like run it through VirusTotal and returns the result of that scan to Bro.  If the file is determined to be malicious an alarm could be raised about the initial transfer and the MD5 sum could be added to a set of malicious MD5 sums.  The URL of the file could also be added to a set of URLs.  In the future, if any host downloads a file with that MD5 sum or from the same URL then an alarm would automatically be raised without waiting for the external analysis to take place.  This full scenario is not currently implemented in Bro, but things are lining up to make this sort of analysis possible.

If you have ideas for analysis scenarios that you'd like to see implemented, I'd really like to hear them!

> All the analysis that I have been (and will be doing) is with tracefiles on a machine that is not connected to a network. I figured that there were chances that I could run out of memory, but was hoping that the memory would be released once the connection was terminated. I did not think about using a table of strings to keep the data... guess I was thinking too deep.

You could either keep a table of strings or concatenate the strings together as new data comes in.  I'll include some examples here.

Using these inputs...
global a = "first string";
global b = "second string";
global output = "";

You can do this...
global stuff: string_array = table();
stuff[|stuff|+1] = a;
stuff[|stuff|+1] = b;
output = cat_string_array(stuff);

Or this...
output = string_cat(a, b);

> I figured that you would accept patches. It has been awhile since I've used C++, but hoping it will come back to me. I have spent a lot of time looking at the source code to better understand how bro works. I would love to see RDP and SSL decryption, but I know that those aren't easy tasks... doesnt mean I wont try eventually.

Bro currently doesn't have any support for RDP but I think that a lot of the support for SSL decryption is already in place.  I've haven't ever done it though so I don't know if it is completely there and working though.

  .Seth