[Bro-Dev] Per connection byte and packet counting

[Gregor Maier]

Hi,

since there were no more comments. I'll wrap up these changes and make
them ready for a merge.

I will also make the connection history in the conn record an optional
field, since this will save quite a bit of memory.

cu
Gregor

On 12/14/10 16:03 , Gregor Maier wrote:
> Hi,
> 
> I've played around with packet and byte counting for connections some
> more. I think I've got a pretty neat version:
> 
> * Counting is done in a separate analyzer that can be added to the
>   initial analyzer tree based on a global boolean const.
> 
> * Reporting is done via the endpoint records in the connection record (
>   c$orig, c$resp). The endpoint records have to new &optional fields for
>   number of packets and number of bytes.
> 
> * Updating the counters in the endpoint is done by:
>   + Removing UpdateEndpointVal() from TransportAnalyzer
>   + Adding a UpdateConnVal(conn_val) to Analyzer.cc. When a Conn object
>     needs to update the conn_val (e.g., to generate an event) it calls
>     the root analyzer's UpdateConnVal(). This call is then "forwarded"
>     to every analyzer in the tree. I.e., every analyzer could update the
>     conn_val, e.g., by adding additional, optional fields to the
>     connection record.
> 
> In terms of speed and memory consumption this is surprisingly
> lightweight. Runtime as reported by time, memory consumption from top.
> I've only loaded conn and weird:
> 
> * Baseline. Current git master: (3 runs)
>      2m42s  900MB
>      2m37s  900MB
>      2m37s  900MB
> 
> * My version, with counting disabled (i.e., counting analyzer not
>   instantiated, no the optional fields in endpoint are NULL):
>      2m34s  898MB
>      2m36s  898MB
>      2m36s  898MB
>   It seems to be faster and using less memory(!). There is an
>   explanation why it needs slightly less memory: the git-master Conn.cc
>   had two RecordVal* for orig_endp, and resp_endp that were passed to
>   UpdateEndpointVal. Since I now only pass the conn_val to
>   UpdateConnVal, I don't need these pointers in Conn.cc anymore.
>   I've checked the testsuite it my version passes it.
> 
> 
> * My version with counting enabled:
>      2m40s  1000MB
>      2m41s  1000MB
>      2m41s  1000MB
>      2m41s  1000MB
>   So: no speed penalty and about 100MB for the analyzer instance and
>   the additional fields in endpoint (these are responsible for most of
>   the memory. Vals are expensive memory-wise).
> 
> * As an additional experiment, I made the history field in conn_val
>   &optional, so that it's only allocated, if record_state_history=T.
>   Results:
>     with    counting:  2m39s  862MB
>     wihtout counting:  2m33s  962MB
>   So I think it might make sense to also integrate this change.
>   (Currently the history is always tracked and always part of the
>   connection record, but it is only logged in conn.log if
>   record_state_history=T).
> 
> Finally, a longer trace with way more data and more policy scripts (conn
> weird dpd http-request htt-reply dns):
>     master / baseline  88m38   2906MB
>     no counting        88m37   2907MB
>     with counting      87m47   2963MB
> 
> 
> So, all in all I would say: we should go with this version, as this
> seems to be the ideal solution: overhead when disabled, only memory
> overhead when enabled (and this overhead is inevitable!).
> 
> In addition, I would suggest to make c$history &optional and only
> allocate and assign a Val* to it if record_state_history=T, since this
> seems to safe quite a bit of memory!
> 
> cu
> Gregor


-- 
Gregor Maier
<gregor at icir.org>  <gregor at icsi.berkeley.edu>
Int. Computer Science Institute (ICSI)
1947 Center St., Ste. 600
Berkeley, CA 94704, USA
http://www.icir.org/gregor/