[Bro-Dev] Per connection byte and packet counting

[Gregor Maier]

Hi,

I've played around with adding support to count packets and (IP-)bytes
per connection and direction. (Sort of like in addition to conn.log's
osize and rsize, which are based on sequence numbers).
This adds 4 counters per connection.

Basically, there are two ways to implement this:

a) hard coded into the transport analyzers that keep track of the
   connection record (which includes osize and rsize). I've done that
   and added the counters to the "endpoint" record val. (A connection
   record val has an orig and resp "endpoint" record val).

   Advantage: every connection event, i.e., the majority of events, now
   has access to these counters whenever the event is generated.

   Disadvantage: one cannot turn it off and it will result in a memory
   overhead per connection (see below).

b) implement it as a child-analyzer to the transport analyzers and add
   this child-analyzer to the initial analyzer tree. One can configure
   whether to enable/disable this size-analyzer via policy scripts. I've
   done that too.

   Advantage: can be switched on and off. No overhead when off.

   Disadvantage / Questions:
   * How to pass the information to policy scripts? (see below)
   * Slightly slower due to calls to an additional analyzer (and
     virtualized method lookups)

I've implemented both variants to check their performance. Here are
the results using a quite large Time Machine trace and only loading conn
and weird. The trace has some 14.8 M connections total.

  * trunk: 26.5min--27min runtime, 900MB virtual mem, 890MB residual.
  *     a:   27min--28min runtime, 960MB virtual mem, 948MB residual.
  *     b: 29.25min runtime,       952MB virtual mem, 940MB residual.


WRT variant b) (separate analyzer): So far I've implemented an event
when the analyzer is Done, i.e., when the connection ends. So, the
policy script can collect the counter values only after the connection
ends. However, I think it might be helpful, if one could access these
counters at any time.
* Is there any way to do so? E.g., by having a table that's indexed by
  connid that can be read from the policy layer but is updated by the
  event engine? Can this be done from an analyzer, or are there some
  hacks required?
* Other ideas to make the counters available to the policy script.
* Do you guys think it's important to have access to the counters at
  any time, or is it sufficient to just get them when the connection
  ends?


Any thoughts or comments?
Gregor
-- 
Gregor Maier                                             gregor at icir.org
Int. Computer Science Institute (ICSI)          gregor at icsi.berkeley.edu
1947 Center St., Ste. 600                    http://www.icir.org/gregor/
Berkeley, CA 94704
USA