[Bro] Time Machine RAM usage question

[Gregor Maier]

Hi,

looking at the source code, it seems that the message 'not found in
connection table' is related to subscriptions (i.e., the query request
that all future packets for this connections should be included in the
query results. And subscriptions only work for connections that are
currently active). So this message is ok. (Actually it is commented out
in the current svn-snapshot. Did you uncomment it or was it in your
version of the TM source code).

As others already pointed out the conn_timeout is indeed the idle time
until a connection is expired from the connection table (we only use the
timeout to expire connections). Setting this to high value is
counter-productive: the memory consumption increases significantly.
Furthermore, long timeouts will reduce visibility. No new packets will
be recorded for connections (actually 5-tuples) that aren't expired but
have exceeded the cutoff. So long timeouts can be problematic in the
case of 5-tuple reuse.

To check your current retention times, you can check the classes.tm.log
file. mem_dt and disk_dt will tell you how many seconds of packet data
are currently retained in memory and on disk. Can you check whether the
packets you want to retrieve fall into this time-frame?


cu
Gregor





On 10/26/10 14:04 , Martin Holste wrote:
> That's what I originally thought.  What was throwing me was when I
> would try to find packets any older than the cutoff, the queries would
> come up empty, the log showing something like "query not found in
> connection table."  So I ran "show conn sample" to see the connections
> table, and the oldest connections were always at the cutoff.  When I
> looked through the source code, it appeared that connections older
> than the cutoff were evicted from the connections table, but the query
> depended on the connections table to find the packets on disk/ram.
> 
> On Tue, Oct 26, 2010 at 2:43 PM, Vern Paxson <vern at icir.org> wrote:
>>> I don't think you need conn_timeout set that high.
>>
>> Right.  conn_timeout is how long to keep internal state when a connection
>> is inactive; *not* how long to keep recorded connections lying around.
>>
>>                Vern
>>
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


-- 
Gregor Maier                                             gregor at icir.org
Int. Computer Science Institute (ICSI)          gregor at icsi.berkeley.edu
1947 Center St., Ste. 600                    http://www.icir.org/gregor/
Berkeley, CA 94704
USA