[Bro] Time Machine RAM usage question

Gregor Maier gregor at icir.org
Tue Oct 26 19:52:51 PDT 2010


no worries.
Just let me know if the error pops up again.

cu
gregor

On 10/26/10 18:46 , Martin Holste wrote:
> Thanks for looking into it.  I blew away all of my buffers and indexes
> and restarted with more sane settings and queries seem to be behaving.
>  I'm certainly not ruling out user error.  In any case, thanks for
> your help, it's very appreciated.
> 
> On Tue, Oct 26, 2010 at 7:45 PM, Gregor Maier <gregor at icir.org> wrote:
>> Hi,
>>
>> looking at the source code, it seems that the message 'not found in
>> connection table' is related to subscriptions (i.e., the query request
>> that all future packets for this connections should be included in the
>> query results. And subscriptions only work for connections that are
>> currently active). So this message is ok. (Actually it is commented out
>> in the current svn-snapshot. Did you uncomment it or was it in your
>> version of the TM source code).
>>
>> As others already pointed out the conn_timeout is indeed the idle time
>> until a connection is expired from the connection table (we only use the
>> timeout to expire connections). Setting this to high value is
>> counter-productive: the memory consumption increases significantly.
>> Furthermore, long timeouts will reduce visibility. No new packets will
>> be recorded for connections (actually 5-tuples) that aren't expired but
>> have exceeded the cutoff. So long timeouts can be problematic in the
>> case of 5-tuple reuse.
>>
>> To check your current retention times, you can check the classes.tm.log
>> file. mem_dt and disk_dt will tell you how many seconds of packet data
>> are currently retained in memory and on disk. Can you check whether the
>> packets you want to retrieve fall into this time-frame?
>>
>>
>> cu
>> Gregor
>>
>>
>>
>>
>>
>> On 10/26/10 14:04 , Martin Holste wrote:
>>> That's what I originally thought.  What was throwing me was when I
>>> would try to find packets any older than the cutoff, the queries would
>>> come up empty, the log showing something like "query not found in
>>> connection table."  So I ran "show conn sample" to see the connections
>>> table, and the oldest connections were always at the cutoff.  When I
>>> looked through the source code, it appeared that connections older
>>> than the cutoff were evicted from the connections table, but the query
>>> depended on the connections table to find the packets on disk/ram.
>>>
>>> On Tue, Oct 26, 2010 at 2:43 PM, Vern Paxson <vern at icir.org> wrote:
>>>>> I don't think you need conn_timeout set that high.
>>>>
>>>> Right.  conn_timeout is how long to keep internal state when a connection
>>>> is inactive; *not* how long to keep recorded connections lying around.
>>>>
>>>>                Vern
>>>>
>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>> --
>> Gregor Maier                                             gregor at icir.org
>> Int. Computer Science Institute (ICSI)          gregor at icsi.berkeley.edu
>> 1947 Center St., Ste. 600                    http://www.icir.org/gregor/
>> Berkeley, CA 94704
>> USA
>>
> 


-- 
Gregor Maier                                             gregor at icir.org
Int. Computer Science Institute (ICSI)          gregor at icsi.berkeley.edu
1947 Center St., Ste. 600                    http://www.icir.org/gregor/
Berkeley, CA 94704
USA



More information about the Bro mailing list