[Bro] Time Machine RAM usage question

[Martin Holste]

Thanks for looking into it.  I blew away all of my buffers and indexes
and restarted with more sane settings and queries seem to be behaving.
 I'm certainly not ruling out user error.  In any case, thanks for
your help, it's very appreciated.

On Tue, Oct 26, 2010 at 7:45 PM, Gregor Maier <gregor at icir.org> wrote:
> Hi,
>
> looking at the source code, it seems that the message 'not found in
> connection table' is related to subscriptions (i.e., the query request
> that all future packets for this connections should be included in the
> query results. And subscriptions only work for connections that are
> currently active). So this message is ok. (Actually it is commented out
> in the current svn-snapshot. Did you uncomment it or was it in your
> version of the TM source code).
>
> As others already pointed out the conn_timeout is indeed the idle time
> until a connection is expired from the connection table (we only use the
> timeout to expire connections). Setting this to high value is
> counter-productive: the memory consumption increases significantly.
> Furthermore, long timeouts will reduce visibility. No new packets will
> be recorded for connections (actually 5-tuples) that aren't expired but
> have exceeded the cutoff. So long timeouts can be problematic in the
> case of 5-tuple reuse.
>
> To check your current retention times, you can check the classes.tm.log
> file. mem_dt and disk_dt will tell you how many seconds of packet data
> are currently retained in memory and on disk. Can you check whether the
> packets you want to retrieve fall into this time-frame?
>
>
> cu
> Gregor
>
>
>
>
>
> On 10/26/10 14:04 , Martin Holste wrote:
>> That's what I originally thought.  What was throwing me was when I
>> would try to find packets any older than the cutoff, the queries would
>> come up empty, the log showing something like "query not found in
>> connection table."  So I ran "show conn sample" to see the connections
>> table, and the oldest connections were always at the cutoff.  When I
>> looked through the source code, it appeared that connections older
>> than the cutoff were evicted from the connections table, but the query
>> depended on the connections table to find the packets on disk/ram.
>>
>> On Tue, Oct 26, 2010 at 2:43 PM, Vern Paxson <vern at icir.org> wrote:
>>>> I don't think you need conn_timeout set that high.
>>>
>>> Right.  conn_timeout is how long to keep internal state when a connection
>>> is inactive; *not* how long to keep recorded connections lying around.
>>>
>>>                Vern
>>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> --
> Gregor Maier                                             gregor at icir.org
> Int. Computer Science Institute (ICSI)          gregor at icsi.berkeley.edu
> 1947 Center St., Ste. 600                    http://www.icir.org/gregor/
> Berkeley, CA 94704
> USA
>