[Bro-Dev] Per connection byte and packet counting

[Gregor Maier]

Hi,

I've played around with packet and byte counting for connections some
more. I think I've got a pretty neat version:

* Counting is done in a separate analyzer that can be added to the
  initial analyzer tree based on a global boolean const.

* Reporting is done via the endpoint records in the connection record (
  c$orig, c$resp). The endpoint records have to new &optional fields for
  number of packets and number of bytes.

* Updating the counters in the endpoint is done by:
  + Removing UpdateEndpointVal() from TransportAnalyzer
  + Adding a UpdateConnVal(conn_val) to Analyzer.cc. When a Conn object
    needs to update the conn_val (e.g., to generate an event) it calls
    the root analyzer's UpdateConnVal(). This call is then "forwarded"
    to every analyzer in the tree. I.e., every analyzer could update the
    conn_val, e.g., by adding additional, optional fields to the
    connection record.

In terms of speed and memory consumption this is surprisingly
lightweight. Runtime as reported by time, memory consumption from top.
I've only loaded conn and weird:

* Baseline. Current git master: (3 runs)
     2m42s  900MB
     2m37s  900MB
     2m37s  900MB

* My version, with counting disabled (i.e., counting analyzer not
  instantiated, no the optional fields in endpoint are NULL):
     2m34s  898MB
     2m36s  898MB
     2m36s  898MB
  It seems to be faster and using less memory(!). There is an
  explanation why it needs slightly less memory: the git-master Conn.cc
  had two RecordVal* for orig_endp, and resp_endp that were passed to
  UpdateEndpointVal. Since I now only pass the conn_val to
  UpdateConnVal, I don't need these pointers in Conn.cc anymore.
  I've checked the testsuite it my version passes it.


* My version with counting enabled:
     2m40s  1000MB
     2m41s  1000MB
     2m41s  1000MB
     2m41s  1000MB
  So: no speed penalty and about 100MB for the analyzer instance and
  the additional fields in endpoint (these are responsible for most of
  the memory. Vals are expensive memory-wise).

* As an additional experiment, I made the history field in conn_val
  &optional, so that it's only allocated, if record_state_history=T.
  Results:
    with    counting:  2m39s  862MB
    wihtout counting:  2m33s  962MB
  So I think it might make sense to also integrate this change.
  (Currently the history is always tracked and always part of the
  connection record, but it is only logged in conn.log if
  record_state_history=T).

Finally, a longer trace with way more data and more policy scripts (conn
weird dpd http-request htt-reply dns):
    master / baseline  88m38   2906MB
    no counting        88m37   2907MB
    with counting      87m47   2963MB


So, all in all I would say: we should go with this version, as this
seems to be the ideal solution: overhead when disabled, only memory
overhead when enabled (and this overhead is inevitable!).

In addition, I would suggest to make c$history &optional and only
allocate and assign a Val* to it if record_state_history=T, since this
seems to safe quite a bit of memory!

cu
Gregor
-- 
Gregor Maier                                             gregor at icir.org
Int. Computer Science Institute (ICSI)          gregor at icsi.berkeley.edu
1947 Center St., Ste. 600                    http://www.icir.org/gregor/
Berkeley, CA 94704
USA