[Bro] "Faking" connections and http records

[Jim Mellander]

Reviving my old thread - this project was on hiatus for a while, but
is now generating useful data, although not yet in production.

What I ended up doing was transport the syslogs of interest (apache
logs) via stunnel to my bro box, which then runs a custom python
script:

1. Parses and extracts the logs into python variables
2. Construct plausible http session conversation.
3. Packetize the session into TCP packets
4. Push tcp packets onto virtual interface
5. bro listening to virtual interface performs normal IDS monitoring of traffic.

Here's an example:

Syslog entry:
Jul  3 09:34:54 128.55.22.194 httpd[4148]: www.nersc.gov 66.249.73.109
- - [03/Jul/2013:09:34:54 -0700] "GET /robots.txt HTTP/1.1" 200 82 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"

Bro http logs:
Jul  3 09:34:58	u3qPWFy8m9	66.249.73.109	64555	128.55.22.194	80	GET	www.nersc.gov	/robots.txt		Mozilla/5.0
(compatible; Googlebot/2.1;
+http://www.google.com/bot.html)	0	0	200	<empty>(empty)	-	-	-	-	-	-

A few points:

1.Tested creating packets to make the conn logs show the correct
amount of data returned by the server, as reported in the log, but
eventually chose to not do that, as that is of limited value.
2. Support both ipv4 & ipv6 - if a host is dual homed, typically the
syslog entry will be from the ipv4 address, but the requesting ip may
be ipv6, in which case we convert both address to ipv6 using several
strategies.

The value of this, from our perspective, is that we can now perform
the usual http IDS functions on https connections to our syslogging
webservers, without having to store the certs in our bro system for
decryption.  We also have visibility on intrasite traffic to those
same webservers.


Alpha quality code available for the asking.

Jim









On Fri, Mar 22, 2013 at 2:18 PM, Jim Mellander <jmellander at lbl.gov> wrote:
> Well, its unfortunate that we can't feed in data from other sources
> and subject it to the same policies that network traffic is subject
> to.
>
> In the meantime, I may just write some code that fakes the data into
> pcap files that can be read by bro directly.
>
> On Fri, Mar 22, 2013 at 1:54 PM, Seth Hall <seth at icir.org> wrote:
>>
>> On Mar 22, 2013, at 4:04 PM, Jim Mellander <jmellander at LBL.GOV> wrote:
>>
>>> Does anyone have suggestions on how to proceed with this?
>>
>>
>> It wouldn't work very well. :)
>>
>> Nearly all of the detections rely on the various http_ events.  I would go down a slightly different route with logs than I would with raw traffic.  This is something that I've been talking about for quite a while and I suspect something related to happen in the next year.
>>
>> I think it's really cool that you're importing logs into Bro!
>>
>>   .Seth
>>
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro.org/
>>