[Bro] Issues with intel framework

[James Gordon]

Thanks Justin!

Adding an entry for www.reddit.com:443 seemed to do the trick. I’ll experiment with removing the port from the http-headers.bro file in a test environment so that we don’t have to add duplicate entries for all https sites. Unfortunately, I added another entry in the intel_domains.dat file for www.linux.com:443, and that still is not firing intel hits. I have '@load intel' in my local.bro file, and I modified intel’s __load__.bro to include the new intel_domains.dat (as found in the original attachment). Is that all that should be necessary to load a new .dat into the intel framework? 

Also, thanks for explaining the way connections are alerted on! I see value in alerting on failed outbound connections that we’re suspicious about.
This is probably a dumb question, but should I replace $bropath/policy/frameworks/intel/seen/conn-established.bro with a conn-attempted.bro, or will it be necessary to use both if I want to be alerted on both successful and attempted connections?

Thanks again,

James Gordon



> On Sep 16, 2016, at 2:23 PM, Azoff, Justin S <jazoff at illinois.edu> wrote:
> 
> 
>> On Sep 16, 2016, at 1:42 PM, James Gordon <gordonjamesr at gmail.com> wrote:
>> 
>> Hi all,
>> 
>> I’m running Bro on a fairly new distributed security onion setup - we have 3 sensors running bro in our environment. We’re running bro version 2.4.1. I’ve been wanting to add Critical Stack’s intel agent into our stack, but wanted to test how the intel framework works before we set it all up. To test the framework, I added a test domain (www.reddit.com) and two test IP addresses to intel.dat. I also downloaded and formatted a list of known malicious domains using the instructions found here: http://www.pintumbler.org/words/broagentforsguil-nowsupportsintellog, and saved this file in $bropath/share/bro/intel/intel_domans.dat. To test this list, I appended an entry for www.linux.com. All these files are tab delimited. We’re having two large reliability issues with the intel framework:
>> 
>> 1 - My rule for the reddit.com domain in intel.dat fires sporadically, and it seems like only for certain subnets / end users. I can not get that alert to trip for me by browsing to reddit.com, despite seeing these connections in http.log and conn.log.
> 
> The good news is that bro appears to be working properly, but intel is just not doing what you want (which is much easier to fix than bro not working properly!)
> 
> It looks like http requests that are not matching are the proxy requests, correct?  There is a small discrepancy in the code that logs http requests and the code that feeds into the intel system:
> 
> The logging code does this:
> 
> event http_header(c: connection, is_orig: bool, name: string, value: string) &priority=5
>    {
>    ...
>        else if ( name == "HOST" )
>            # The split is done to remove the occasional port value that shows up here.
>            c$http$host = split_string1(value, /:/)[0];
> 
> While the intel code uses 'value' as is without stripping the port.  If your client is sending the host header as www.reddit.com:443 that would cause this to not match.
> 
> If you add an entry to your intel file for www.reddit.com:443 does that fix that problem?
> 
> If it does, it's a simple fix to intel/seen/http-headers.bro to remove the port as well. I filed an issue to look into this:
> 
> https://bro-tracker.atlassian.net/browse/BIT-1695
> 
> 
>> The rules for Intel::ADDR in intel.dat never fire, even though we do see connections to those addresses in conn.log.
> 
> The default behavior for connections in intel is that it only alerts for established connections.  The two log entries you have there are failed connections.  You can easily have it alert on failed inbound and failed outbound connections if you wanted though.  I thought we had a script for that somewhere.. It would basically be the same as the ./policy/frameworks/intel/seen/conn-established.bro but using connection_attempt instead of connection_established.. something like
> 
> event connection_attempt(c: connection)
> 	{
> 	Intel::seen([$host=c$id$orig_h, $conn=c, $where=Conn::IN_ORIG]);
> 	Intel::seen([$host=c$id$resp_h, $conn=c, $where=Conn::IN_RESP]);
> 	}
> 
> Or maybe just only one of orig/resp.. it likely depends on exactly what you care about :-)
> 
> 
> -- 
> - Justin Azoff
>