[Bro] Getting info records when log events happen, but where the logging script has no specific log event type

[Gary Faulkner]

In my quest to graph event statistics tied to bro logs I've run across a 
few scripts that seem to break the idiom of logging being a separate 
event from the rest of the events in a script. A couple examples are 
capture-loss and tunnels. Both scripts call the LOG function within some 
other event that doesn't expose the underlying info record to other 
scripts as far as I can tell. A lot of my meta-data collection acts on 
the log events and the data contained within the info records at the 
time those events are logged. I'm wondering if there is another way to 
grab that data without modifying the base scripts or if these scripts 
can be easily made to have a logging event?

Here are a couple examples of things I'd like to be able to do.

  * Increment a counter whenever a new log line is written (useful for
    troubleshooting upstream log aggregator inputs)
  * Send raw data such as percent_lost per peer to an external time
    series database (could be useful for seeing loss over time, or
    identifying problems with load-balancing and filtering of flows).
  * Track number of tunnels seen by tunnel type (knowing how often and
    when traffic is being tunneled could be interesting)

I also find tracking event counters can be useful for identifying things 
that are outside the norm, especially in cases where seeing similar 
trends in a log management system or SEM involves a very expensive 
query. For example a sudden spike in TCP connection attempts / SYNs that 
could indicate DOS participation, spikes in the number of DNS ANY 
queries (probably an open resolver being abused) etc.

Here are a few simplistic examples of some counters I'm already 
collecting that show how the log event and info record are used (These 
rely on JA's statsd plugin and some stats may be borrowed/derived):

# DNS Events

event DNS::log_dns(rec: DNS::Info)
{
     statsd_increment("bro.log.dns", 1); #Track DNS log volume

     if(rec?$rcode && rec$rcode == 3) 
{statsd_increment("bro.dns.error.nxdomain", 1);}

     if(rec?$qtype_name && /query/ !in rec$qtype_name)
     {
         local s = fmt("bro.dns.query.type.%s", rec$qtype_name);
         statsd_increment(s, 1);
     }
}

# Notice Events

event Notice::log_notice(rec: Notice::Info)
{
     statsd_increment("bro.log.notice", 1); #Track Notice log volume

     if(rec?$note)
     {
         local s = fmt("bro.notice.type.%s", rec$note);
         local s2 = sub(s, /::/, "_"); #influxdb doesn't like :: so 
replace it with _
         statsd_increment(s2, 1);
     }
}


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150826/6f1f2ed9/attachment.html