[Bro] sum stats q.

[Dk Jack]

Hi,
I am trying to use Bro sumstats framework. Based on the examples, I came up
with the script shown at the end of the email. In the script, I am counting
the number of http requests for each method+uri combination.

As dictated by the framework, I am calling observe for each request. At the
end, I expected the total sumstats equal to the number of requests in my
pcap. However, this doesn't seem to be the case. I am trying understand if
I made a mistake in how I am using the framework of if something else is
going on.

For example, I ran the script on try.bro.org website using the http.pcap
available there. Per my analysis, there should be 197 requests in the pcap.
However, when I dump each of my stat into a log file, I expected the hits
column from the log to add up to 197. However, that's not the case. Running
the script against my own pcap is giving different numbers from what I
would expect.

Any help understanding the issue is appreciated... Thanks

Dk.

PS: you can copy paste this script in to try.bro.org website and run it
against the http.pcap.

@load base/utils/site
@load base/frameworks/sumstats

module HttpStats;

export {
  redef enum Log::ID += { LOG };

  type Info: record {
    ts:         time   &log;
    method:     string &log;
    uri:        string &log;
    hits:       count  &log;
  };

  global update_http_stats: function(method: string, uri: string);
}

global scount: count = 0;

event bro_init() &priority=5
{
  print "Creating HttpStats log stream and HTTP sumstats";
  flush_all();

  # Create the stream.
  Log::create_stream(HttpStats::LOG, [$columns=Info, $path="http-stats"]);

  local r1 = SumStats::Reducer($stream="http-stats",
$apply=set(SumStats::SUM));

  SumStats::create([$name="http-stats",
                    $epoch=5sec,
                    $reducers=set(r1),
                    $epoch_result(ts: time, key: SumStats::Key, result:
SumStats::Result) =
                    {
                      local r = result["http-stats"];
                      local host_uri_vec = split_string(key$str, /,/);
                      local method = host_uri_vec[0];
                      local uri = host_uri_vec[1];
                      #local hits = double_to_count(floor(r$sum));
                      local hits = double_to_count(floor(r$num));

                      # prep the record
                      local log_rec: Info = [$ts=ts, $method=method,
$uri=uri, $hits=hits];
                      Log::write(HttpStats::LOG, log_rec);
                    }
                    ]);
}

event bro_done()
{
    Reporter::info(fmt("scount=%d", scount));
}

function update_http_stats(method: string, uri: string)
{
  local key = cat_sep(",", "-", method, uri);

  scount += 1;

  # count URI hits.
  SumStats::observe("http-stats", SumStats::Key($str=key),
SumStats::Observation($num=1));
}

event http_request(c: connection, method: string, original_URI: string,
unescaped_URI: string, version: string)
{
    update_http_stats(method, unescaped_URI);
}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181128/09929bd6/attachment.html