From robin at icir.org  Fri Jul 15 17:22:37 2011
From: robin at icir.org (robin at icir.org)
Date: Fri, 15 Jul 2011 17:22:37 -0700
Subject: [TM] [svn] time-machine - r265 - trunk
Message-ID: <201107160022.p6G0MbMg015044@envoy.icir.org>

Author: robin
Date: 2011-07-15 17:22:37 -0700 (Fri, 15 Jul 2011)
New Revision: 265
Repository: svn.icir.org/time-machine

Modified:
   trunk/AUTHORS
Log:
Test commit.


Modified: trunk/AUTHORS
===================================================================
--- trunk/AUTHORS	2010-06-11 05:58:22 UTC (rev 264)
+++ trunk/AUTHORS	2011-07-16 00:22:37 UTC (rev 265)
@@ -17,5 +17,6 @@
 Fabian Schneider
 
 
+
 Hash Function (c) by Bob Jenkins, May 2006, Public Domain.
 ptmalloc (c) by Wolfram Gloger (see ptmalloc2/COPYRIGHT for details)


From gregor at icir.org  Sat Jul 16 09:17:30 2011
From: gregor at icir.org (Gregor Maier)
Date: Sat, 16 Jul 2011 09:17:30 -0700
Subject: [TM] [svn] time-machine - r266 - trunk
Message-ID: <201107161617.p6GGHUWn024624@envoy.icir.org>

Author: gregor
Date: 2011-07-16 09:17:30 -0700 (Sat, 16 Jul 2011)
New Revision: 266
Repository: svn.icir.org/time-machine

Modified:
   trunk/FifoDisk.cc
Log:
Change naming of classfiles on disk. We now use the timestamp (from packets) instead of a counter. Prevents a restarted TM from overwriting old class files. 


Modified: trunk/FifoDisk.cc
===================================================================
--- trunk/FifoDisk.cc	2011-07-16 00:22:37 UTC (rev 265)
+++ trunk/FifoDisk.cc	2011-07-16 16:17:30 UTC (rev 266)
@@ -104,6 +104,7 @@
 
 void FifoDisk::addPkt(const pkt_ptr p) {
 	if (size>0) {
+		newestTimestamp = to_tm_time(&(((struct pcap_pkthdr*)p)->ts));
 		if (files.empty() ||
 				files.back()->getCurFileSize()
 				+ sizeof(struct pcap_file_header)
@@ -130,16 +131,15 @@
 					}
 				}
 				file_number++;
-				const int strsz=classname.length()+1+8+1; // "_" + "number" + "\0"
+				const int strsz=classname.length()+30; 
 				char *new_file_name=(char*)malloc(strsz);
-				snprintf(new_file_name, strsz, "%s_%08x",
-						 classname.c_str(), file_number);
+				snprintf(new_file_name, strsz, "%s_%.6f",
+						 classname.c_str(), newestTimestamp);
 				files.push_back(new FifoDiskFile(new_file_name, pcap_handle));
 				free(new_file_name);
 			}
 			unlockQueryInProgress();
 		}
-		newestTimestamp = to_tm_time(&(((struct pcap_pkthdr*)p)->ts));
 		files.back()->addPkt(p);
 		if (oldestTimestamp < 1e-3)
 			oldestTimestamp = files.front()->getOldestTimestamp();


From gregor at icir.org  Sun Jul 17 10:26:22 2011
From: gregor at icir.org (Gregor Maier)
Date: Sun, 17 Jul 2011 10:26:22 -0700
Subject: [TM] [svn] time-machine - r267 - in trunk: . Documentation
Message-ID: <201107171726.p6HHQMWE005844@envoy.icir.org>

Author: gregor
Date: 2011-07-17 10:26:21 -0700 (Sun, 17 Jul 2011)
New Revision: 267
Repository: svn.icir.org/time-machine

Modified:
   trunk/Documentation/TM_HOWTO
   trunk/Index.cc
   trunk/Index.hh
   trunk/Storage.cc
   trunk/Storage.hh
   trunk/conf_parser.yy
   trunk/conf_scanner.ll
   trunk/config.h.in
   trunk/configure.in
   trunk/tm.conf
Log:
Make indexes configurable. One can now enable/disable indexes in the configuration file. One can also enable/disable disk indexes for a particular index. Needs some more extensive testing on live traffic. 


Modified: trunk/Documentation/TM_HOWTO
===================================================================
--- trunk/Documentation/TM_HOWTO	2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/Documentation/TM_HOWTO	2011-07-17 17:26:21 UTC (rev 267)
@@ -1,5 +1,7 @@
 $Id$
 
+# vim: expandtab
+
 Also have a look at the TM's homepage and at the papers in this
 directory. They give a good idea of what one can do with a
 Time Machine
@@ -56,8 +58,8 @@
     TCP port <port> via the Bro communication protocol. The Bro connection
     is established by the Time Machine upon TM startup. TM uses the
     Broccoli library for communcation with Bro.
-	This option is deprecated. The new method is that the TM listens for 
-	incomming connections from Bro. See the bro_listen_* options
+    This option is deprecated. The new method is that the TM listens for 
+    incomming connections from Bro. See the bro_listen_* options
 
   console 0|1
     Determine whether a command line interface should be displayed on the
@@ -66,7 +68,7 @@
 
   daemon 0|1
     Run the timemachine as a daemon in background mode. Incompatible with
-	console. 
+    console. 
 
   workdir "<path>"
     Determine the working directory where all class storage files and index
@@ -98,20 +100,20 @@
 
   rmtconsole 0|1
     Start a remote console listener. This will listen on a network socket
-	for incomming connections. When connected a remote user can issue 
-	timemachine commands as could be done from the locale console. I.e. use
-	telnet to connect. 
-	Please note, that this connection is not authenticated and not 
-	encrypted. Use with care.
+    for incomming connections. When connected a remote user can issue 
+    timemachine commands as could be done from the locale console. I.e. use
+    telnet to connect. 
+    Please note, that this connection is not authenticated and not 
+    encrypted. Use with care.
 
   rmtconsole_port <number>
     The local port to listen for incoming remote console connections.
-	Default is 42042.
-	
+    Default is 42042.
+    
   rmtconsole_listen_addr <ip address>
     The IP address to listen for incoming remote console connections.
-	Default is 127.0.0.1. Hostnames cannot be used. A value of 0.0.0.0 will
-	listen on all interfaces.
+    Default is 127.0.0.1. Hostnames cannot be used. A value of 0.0.0.0 will
+    listen on all interfaces.
 
   bro_listen 0|1 
     Start a Bro listener. This will listen on a network socket for
@@ -123,7 +125,7 @@
     
   bro_listen_port <number>
     The local port to listen for incoming Bro connections.
-	Default is 47757.
+    Default is 47757.
     
   bro_listen_addr <ip address>
     The IP address to listen for incoming Bro connections. Default
@@ -132,14 +134,23 @@
 
   tweak_capture_thread  priority|scope|no
     This option can only be used on FreeBSD. It will tweak thread scheduling
-	parameters for the capture thread (see Documentation/TUNIG)
-	NOTE NOTE NOTE: You will have to experiment with these settings, since 
-	different hardware platforms und FreeBSD releases behave quite differently!
+    parameters for the capture thread (see Documentation/TUNIG)
+    NOTE NOTE NOTE: You will have to experiment with these settings, since 
+    different hardware platforms und FreeBSD releases behave quite differently!
     priority ... Increase the capture thread's priority to realtime priority
-	scope ...... Set the capture thread's contentio scope to system scope.
-	no ......... Do tweak scheduling parameters  (default).
+    scope ...... Set the capture thread's contentio scope to system scope.
+    no ......... Do tweak scheduling parameters  (default).
 
+  index "<index_name>" [disk]
+    Enable the index named <index_name>. Currently supported indexes are
+    "connection4", "connection3", "connection2", and "ip". You need to
+    enable an index in order to use it for queries. The keyword disk enables
+    the disk index for this index. Only indexes with an enabled disk index
+    can perform on disk queries. 
+    Disabling indexes can safe significant CPU time and disabling disk indexes
+    can reduce disk usage. 
 
+
 * class section
 
   A class section in the configuration file is started by
@@ -185,8 +196,8 @@
 
   dyn_timeout <double>
     The timeout for dynamic classes. If a dynamic rule for an  IP Adresses
-	is pointing to this class, the dynamic rule be removed dyn_timeout
-	seconds after the rule has been set. 
+    is pointing to this class, the dynamic rule be removed dyn_timeout
+    seconds after the rule has been set. 
 
 
 INDEXES
@@ -247,32 +258,32 @@
   <queryspec> ::= index <indexname> "<key-specification>" 
   <query-flags> ::= start <timestamp> end <timestamp> |
                     mem_only | subscribe
-		The flags can be given in order and they can be combined. 
-	
-	Query the indexes for the given index key. The result of a query
-	can either be send to a remote bro system or to a file in the
-	local filesystem. 
-	The index to query is specified by the keyword index followed
-	by the name of the index. This name corresponds to the
-	name that is returned by the getIndexNameStatic() method. 
-	Examples are connection4, connection3, ip, etc.
-	Finally the key to search is specified by 
-	<key-specification>. The sematics of the key spec is defined
-	by the index itself. For example a valid spec for connection4 
-	would be "tcp 1.2.3.4:80 5.6.7.8:88"
-	The <query-flags> enables one to restrict the search or to set
-	a subsciption (see above). Currently only connection4 querys 
-	support the subscribe flag, other indexes will silently ignore
-	the flag. 
-	When mem_only is specified, only the index entries stored in RAM 
-	are searched and only packets from the memomry ringbuffer are 
-	returned. 
-	The timestamps enable one, to specify a timespan. Only packets
-	falling in this timespan will be returned. timestamps and mem_only
-	can be combined. The result will be the intersection of both 
-	(i.e. only packets from memory, that fall into the specified
-	timefragme).
-	The timespan has not been tested extensively.
+        The flags can be given in order and they can be combined. 
+    
+    Query the indexes for the given index key. The result of a query
+    can either be send to a remote bro system or to a file in the
+    local filesystem. 
+    The index to query is specified by the keyword index followed
+    by the name of the index. This name corresponds to the
+    name that is returned by the getIndexNameStatic() method. 
+    Examples are connection4, connection3, ip, etc.
+    Finally the key to search is specified by 
+    <key-specification>. The sematics of the key spec is defined
+    by the index itself. For example a valid spec for connection4 
+    would be "tcp 1.2.3.4:80 5.6.7.8:88"
+    The <query-flags> enables one to restrict the search or to set
+    a subsciption (see above). Currently only connection4 querys 
+    support the subscribe flag, other indexes will silently ignore
+    the flag. 
+    When mem_only is specified, only the index entries stored in RAM 
+    are searched and only packets from the memomry ringbuffer are 
+    returned. 
+    The timestamps enable one, to specify a timespan. Only packets
+    falling in this timespan will be returned. timestamps and mem_only
+    can be combined. The result will be the intersection of both 
+    (i.e. only packets from memory, that fall into the specified
+    timefragme).
+    The timespan has not been tested extensively.
 Examples
 # query to_file "file1.pcap" index connection4 "tcp 1.2.3.4:80 5.6.7.8:1025" subscribe
 # query to_file "file1.pcap" index connection4 "tcp 1.2.3.4:80 5.6.7.8:1025" 
@@ -290,7 +301,7 @@
 
   unsuspend_cutoff "<proto> <ip>:<port>  <ip>:<port>"
      Suspend resp. remove the supension of the cutoff for one connection. If a connection
-	 is supended, all packets will get recorded and the cutoff value is ignored
+     is supended, all packets will get recorded and the cutoff value is ignored
 
   suspend_timeout "<proto> <ip>:<port>  <ip>:<port>"
     Inhibit the eviction of the specified connection from the connection
@@ -310,9 +321,9 @@
     protocol. Not yet supported
 
   bro_connect
-  	DEPRECATED (see CUSTOMIZATION)
+    DEPRECATED (see CUSTOMIZATION)
     (re)connect to the Bro configured in the TM configuration file. Not
-	supported.
+    supported.
 
   show conn "tcp 1.2.3.4:80 7.8.9.1:1042"
     Display information available on the specified connection in the TM's
@@ -321,27 +332,27 @@
   show conn sample
     Display a sample of the newest and oldest connections from the TM's
     connection table (see CONNECTION TABLE above).
-	NOTE/TODO: this function reads the connection table without locking. 
-	This might result in race conditions and in the worst case to a 
-	segfault. Use with care!
+    NOTE/TODO: this function reads the connection table without locking. 
+    This might result in race conditions and in the worst case to a 
+    segfault. Use with care!
 
   set_dyn_class <ip> <classname> [orig|resp]
   unset_dyn_class <ip>
     Sets a rule for a dynamic class. Whenever a new connection with
-	<ip> is seen, the class for this connection will be <classname> and
-	not the class defined by the config file.
-	Dynamic class rules are automatically deleted after a certain time. 
-	The dyn_timeout option of a class specifies, how long a dynamic
-	class rule stays effective. 
-	If orig or resp are given, then only connection that originated from
-	<ip> (in the case of orig) respectively only connections that go to <ip> 
-	(in the case of resp) are assigned to the dynamic class.
-	If two rules (one with orig and one with resp) would match a new packet, 
-	the one with orig take precedence.
-	If several rules for the same IP are set, the latest rule will overwrite all
-	earlier rules.
-	unset_dyn_class can be used to remove a dynamic class rule before the timeout
-	expires.
+    <ip> is seen, the class for this connection will be <classname> and
+    not the class defined by the config file.
+    Dynamic class rules are automatically deleted after a certain time. 
+    The dyn_timeout option of a class specifies, how long a dynamic
+    class rule stays effective. 
+    If orig or resp are given, then only connection that originated from
+    <ip> (in the case of orig) respectively only connections that go to <ip> 
+    (in the case of resp) are assigned to the dynamic class.
+    If two rules (one with orig and one with resp) would match a new packet, 
+    the one with orig take precedence.
+    If several rules for the same IP are set, the latest rule will overwrite all
+    earlier rules.
+    unset_dyn_class can be used to remove a dynamic class rule before the timeout
+    expires.
 
 
 

Modified: trunk/Index.cc
===================================================================
--- trunk/Index.cc	2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/Index.cc	2011-07-17 17:26:21 UTC (rev 267)
@@ -65,10 +65,9 @@
  */
 
 template <class T>
-Index<T>::Index(tm_time_t d_t, uint32_t hash_size, Storage *storage):
+Index<T>::Index(tm_time_t d_t, uint32_t hash_size, bool do_disk_index, Storage *storage):
 		input_q(MyQueue(500000)),
 		cap_thread_iat(0), idx_thread_iat(0),
-		disk_index((std::string)conf_main_indexdir, "index_"+T::getIndexNameStatic()),
 		d_t(d_t),
 		last_rotated(0),
 		last_updated(0),
@@ -77,6 +76,10 @@
 		rotate_count(0){
 	cur = new IndexHash(hash_size);
 	old = new IndexHash(hash_size);
+	if (do_disk_index)
+		disk_index = new IndexFiles<T>((std::string)conf_main_indexdir, "index_"+T::getIndexNameStatic());
+	else
+		disk_index = NULL;
 	pthread_mutex_init(&hash_lock_mutex, NULL);
 	pthread_mutex_init(&queue_lock_mutex, NULL);
 	pthread_cond_init(&queue_cond, NULL);
@@ -103,6 +106,8 @@
 	// Destroy the hashes
 	delete cur;
 	delete old;
+	if (disk_index)
+		delete disk_index;
 }
 
 template <class T>
@@ -190,10 +195,15 @@
 			// Write the old hash to disk.
 			if (old->getNumEntries() != 0)
 			{ 
-				tmlog(TM_LOG_NOTE, T::getIndexNameStatic().c_str(), 
-						"Writing %d entries to disk.",  old->getNumEntries());
-				// writeIndex will delete the entries from the hash
-				disk_index.writeIndex(old);
+				if (disk_index) {
+					tmlog(TM_LOG_NOTE, T::getIndexNameStatic().c_str(), 
+							"Writing %d entries to disk.",  old->getNumEntries());
+					// writeIndex will delete the entries from the hash
+					disk_index->writeIndex(old);
+				} else {
+					// not disk writer
+					old->clear();
+				}
 #ifdef TM_HEAVY_DEBUG
 				tmlog(TM_LOG_DEBUG, T::getIndexNameStatic().c_str(), "Qlen now is %d", input_q.size());
 				assert(old->getNumEntries() == 0);
@@ -292,17 +302,20 @@
 
 template <class T>
 void Index<T>::lookupDisk(IntervalSet* set, IndexField* key, tm_time_t t0, tm_time_t t1) {
-	disk_index.lookup(set, key, t0, t1);
+	if (disk_index)
+		disk_index->lookup(set, key, t0, t1);
 }
 
 template <class T>
 void Index<T>::aggregate() {
+	if (!disk_index)
+		return;
 	tm_time_t oldestTimestampDisk; 
 	//FIXME: do we really have to lock the queue here??
 	lock_queue(); 
 	oldestTimestampDisk = this->idx_thread_oldestTimestampDisk;
 	unlock_queue();
-	disk_index.aggregate(oldestTimestampDisk);
+	disk_index->aggregate(oldestTimestampDisk);
 }
 
 /* Main method of the index maintainer thread

Modified: trunk/Index.hh
===================================================================
--- trunk/Index.hh	2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/Index.hh	2011-07-17 17:26:21 UTC (rev 267)
@@ -184,6 +184,10 @@
 	 * index entries from the packet and enqueues the keys as IndexQueueEntry
 	 * objects */
 	virtual void addPkt(const pcap_pkthdr* header, const u_char* packet)=0;
+	/** Set the storage class for this IndexType. 
+	 *  TODO: we might want to make storage a global singleton ... 
+	 */
+	virtual void setStorage(Storage *arg_storage)=0;
 	/** Aggregate / Merge disk index files into larger files */
 	virtual void aggregate() = 0;
 	virtual const std::string getIndexName()=0;
@@ -248,12 +252,15 @@
 public:
 	// rot_offset is a (small) offset to delay the rotation and thus the writing of
 	// the index to disk. This should be used 
-	Index(tm_time_t d_t, uint32_t hash_size, Storage * storage);
+	Index(tm_time_t d_t, uint32_t hash_size, bool do_disk_index, Storage * storage);
 	~Index();
 	void cancelThread();
 	void lookupMem(IntervalSet* set, IndexField* key);
 	void lookupDisk(IntervalSet* set, IndexField* key, tm_time_t t0, tm_time_t t1);
 	void addPkt(const pcap_pkthdr* header, const u_char* packet);
+	void setStorage(Storage *arg_storage) {
+		storage = arg_storage;
+	}
 	void aggregate();
 	void run();
 	virtual const std::string getIndexName() {
@@ -295,7 +302,7 @@
 	tm_time_t idx_thread_oldestTimestampMem;
 	tm_time_t idx_thread_iat; // InterArrivalTime 
 
-	IndexFiles<T> disk_index;
+	IndexFiles<T> *disk_index;
 	tm_time_t d_t;
 	tm_time_t last_rotated;
 	tm_time_t last_updated;  // last packet ts to be added to hash
@@ -331,6 +338,14 @@
 		for (std::list<IndexType*>::iterator i=begin(); i!=end(); i++)
 			delete (*i);
 	}
+	void setStorage(Storage *s) {
+		for (std::list<IndexType*>::iterator i=begin(); i!=end(); i++)
+			( *i)->setStorage(s);
+	}
+	void startThread() {
+		for (std::list<IndexType*>::iterator i=begin(); i!=end(); i++)
+			pthread_create(&((*i)->maintainer_thread), NULL, start_index_thread, (*i));
+	}
 	void cancelThread() {
 		for (std::list<IndexType*>::iterator i=begin(); i!=end(); i++)
 			( *i)->cancelThread();

Modified: trunk/Storage.cc
===================================================================
--- trunk/Storage.cc	2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/Storage.cc	2011-07-17 17:26:21 UTC (rev 267)
@@ -95,9 +95,19 @@
 }
 
 
+StorageConfig::StorageConfig() : 
+			filter(),
+			device(),
+			readtracefile(),
+			conn_timeout(1800),
+			max_subscriptions(10000),
+			indexes(new Indexes) 
+{
+	;
+}
+
 Storage::Storage(StorageConfig& conf):
 		snaplen(SNAPLEN),
-		indexes(new Indexes),
 		conns(1000000),
 		dynclasses(25000),
 tot_num_queries(0) {
@@ -109,26 +119,10 @@
 	conn_timeout = conf.conn_timeout;
 	conns.setMaxSubscriptions(conf.max_subscriptions);
 
-/* INDEX CONFIGRATION */
-	/* comment any of the following blocks if you want to disable the 
-	 * index */
-	IndexType *idx = new Index<ConnectionIF4>(30, int(250000), this);
-	pthread_create(&(idx->maintainer_thread), NULL, start_index_thread, idx);
-	indexes->addIndex(idx);
-	
-	idx = new Index<ConnectionIF3>(30, int(250000), this);
-	pthread_create(&(idx->maintainer_thread), NULL, start_index_thread, idx);
-	indexes->addIndex(idx);
+	indexes = conf.indexes;
+	indexes->setStorage(this);
+	indexes->startThread();
 
-	idx = new Index<ConnectionIF2>(30, int(250000), this);
-	pthread_create(&(idx->maintainer_thread), NULL, start_index_thread, idx);
-	indexes->addIndex(idx);
-
-	idx = new Index<IPAddress>(30, int(250000), this);
-	pthread_create(&(idx->maintainer_thread), NULL, start_index_thread, idx);
-	indexes->addIndex(idx);
-/* END INDEX CONFIGRATION */
-
 	// Get pcap handle 
 	errbuf[0] = '\0';
 	if (!conf.readtracefile.empty()) {

Modified: trunk/Storage.hh
===================================================================
--- trunk/Storage.hh	2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/Storage.hh	2011-07-17 17:26:21 UTC (rev 267)
@@ -41,6 +41,7 @@
 #include <pthread.h>
 #include <list>
 #include <string>
+#include <tr1/unordered_map>
 
 #include "Fifo.hh"
 #include "Connections.hh"
@@ -51,28 +52,27 @@
 // used in Index.hh at PktLinkList next/previous members
 // #define NUM_INDEXES 6
 
+// TODO: We should really make the Storage class a true global singleton instead 
+// of having to have a pointer to storage in a bunch of classes!
+
 //#include "Index.hh"
 class Indexes;
 
-
 void *capture_thread(void *arg);
 void callback(u_char *args, const struct pcap_pkthdr *header, const u_char *packet);
 
 // Abstracts the configuration for a Storage instance
 class StorageConfig {
 	public:
-		StorageConfig() : 
-			filter(),
-			device(),
-			readtracefile(),
-			conn_timeout(1800),
-			max_subscriptions(10000) {};
+		StorageConfig(); 
+
 		std::string filter;
 		std::string device;
 		std::string readtracefile;
 		std::list<Fifo *> fifos;
 		tm_time_t conn_timeout;
 		int max_subscriptions;
+		Indexes *indexes;
 };
 
 class Storage {

Modified: trunk/conf_parser.yy
===================================================================
--- trunk/conf_parser.yy	2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/conf_parser.yy	2011-07-17 17:26:21 UTC (rev 267)
@@ -52,6 +52,7 @@
 
   #include "Fifo.hh"
   #include "Storage.hh"
+  #include "Index.hh"
 
   // Work around a bug in the relation between bison and GCC 3.x:
   #if defined (__GNUC__) && 3 <= __GNUC__
@@ -70,6 +71,7 @@
   void new_class() { if (!newclass) newclass=new Fifo(); }
   void end_new_class() { assert(newclass); newclass=NULL; }
 
+  void conf_add_index(const char* name, bool do_disk_index);
 /*
   IndexType* newindex=NULL;
   void new_index() { if (!newindex) newindex=new IndexType(); }
@@ -108,6 +110,7 @@
 %token TOK_RMTCONSOLE TOK_RMTCONSOLE_PORT TOK_RMTCONSOLE_LISTEN_ADDR
 %token TOK_BRO_LISTEN TOK_BRO_LISTEN_PORT TOK_BRO_LISTEN_ADDR
 %token TOK_TWEAK_CAPTURE_THREAD TOK_SCOPE TOK_PRIORITY 
+%token TOK_INDEX
 
 %type <s> classname option
 %type <i64> size
@@ -324,6 +327,12 @@
 	| TOK_BRO_LISTEN_ADDR TOK_IPADDRESS {
 		conf_main_bro_listen_addr = $2;
 	}
+	| TOK_INDEX TOK_STRING TOK_DISK {
+		conf_add_index($2, true);
+	}
+	| TOK_INDEX TOK_STRING {
+		conf_add_index($2, false);
+	}
 	;
 	 
 %%
@@ -350,3 +359,35 @@
   return conf_parse_errors;
 }
 
+void conf_add_index(const char* name, bool do_disk_index) {
+	if (conf_parser_storageConf->indexes->getIndexByName(name)) {
+		char msg[2048];
+		snprintf(msg, sizeof(msg), "Index %s already configured\n", name);
+		conferror(msg);
+		return;
+	}
+	/* TODO: We really should do index configuration with a regitry that knows 
+	   of all potential IndexTypes .... */
+	if (ConnectionIF4::getIndexNameStatic() == name) {
+		IndexType *idx = new Index<ConnectionIF4>(30, int(250000), do_disk_index, NULL);
+		conf_parser_storageConf->indexes->addIndex(idx);
+	}
+	else if (ConnectionIF3::getIndexNameStatic() == name) {
+		IndexType *idx = new Index<ConnectionIF3>(30, int(250000), do_disk_index, NULL);
+		conf_parser_storageConf->indexes->addIndex(idx);
+	}
+
+	else if (ConnectionIF2::getIndexNameStatic() == name) {
+		IndexType *idx = new Index<ConnectionIF2>(30, int(250000), do_disk_index, NULL);
+		conf_parser_storageConf->indexes->addIndex(idx);
+	}
+	else if (IPAddress::getIndexNameStatic() == name) {
+		IndexType *idx = new Index<IPAddress>(30, int(250000), do_disk_index, NULL);
+		conf_parser_storageConf->indexes->addIndex(idx);
+	}
+	else {
+		char msg[2048];
+		snprintf(msg, sizeof(msg), "Don't know about index %s\n", name);
+		conferror(msg);
+	}
+}

Modified: trunk/conf_scanner.ll
===================================================================
--- trunk/conf_scanner.ll	2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/conf_scanner.ll	2011-07-17 17:26:21 UTC (rev 267)
@@ -98,6 +98,7 @@
 "workdir"	 return TOK_WORKDIR;
 "queryfiledir"	 return TOK_QUERYFILEDIR;
 "indexdir"	 return TOK_INDEXDIR;
+"index"      return TOK_INDEX;
 "logfile"	 return TOK_LOGFILE;
 "bro_connect_str" return TOK_BRO_CONNECT_STR;
 "pkts_to_disk"	 return TOK_PKTS_TO_DISK;

Modified: trunk/config.h.in
===================================================================
--- trunk/config.h.in	2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/config.h.in	2011-07-17 17:26:21 UTC (rev 267)
@@ -81,6 +81,9 @@
 /* Define to the one symbol short name of this package. */
 #undef PACKAGE_TARNAME
 
+/* Define to the home page for this package. */
+#undef PACKAGE_URL
+
 /* Define to the version of this package. */
 #undef PACKAGE_VERSION
 

Modified: trunk/configure.in
===================================================================
--- trunk/configure.in	2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/configure.in	2011-07-17 17:26:21 UTC (rev 267)
@@ -17,8 +17,8 @@
 AM_PROG_CC_C_O
 #AC_PROG_CXXCPP
 
-OUR_CFLAGS="-O2 -g -Wall -I/usr/local/include"
-LDFLAGS="${LDFLAGS} -L/usr/local/lib"
+OUR_CFLAGS="-O2 -g -Wall -I/usr/local/include -I/opt/local/include"
+LDFLAGS="${LDFLAGS} -L/usr/local/lib -L/opt/local/lib"
 
 CXXFLAGS="${CXXFLAGS} ${OUR_CFLAGS}"
 CFLAGS="${CXXFLAGS} ${OUR_CFLAGS}"

Modified: trunk/tm.conf
===================================================================
--- trunk/tm.conf	2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/tm.conf	2011-07-17 17:26:21 UTC (rev 267)
@@ -22,6 +22,13 @@
 	bro_listen 1;
 	bro_listen_port 47757;   # 47757 is default
 	bro_listen_addr  127.0.0.1;  # 127.0.0.1 is default
+
+	# Index configuration. 
+	# Default is to enable all indexes w/ disk index
+	index "connection4" disk;
+	index "connection3" disk;
+	index "connection2" disk;
+	index "ip" disk;
     
 	daemon 0;
 	# see Documentation/TUNING !!!


From jwelcher at lbl.gov  Tue Jul 19 16:58:54 2011
From: jwelcher at lbl.gov (James Welcher)
Date: Tue, 19 Jul 2011 16:58:54 -0700
Subject: [TM] r267 minor error
Message-ID: <CA+JE=JZxV6YArHJEE-COQj1r5Q=1XsvVBzM+kKP1=Uva8us-pg@mail.gmail.com>

I think a change added to r267 introduced a build problem. I don't
think this line is needed in Storage.hh:

#include <tr1/unordered_map>

(from?http://mailman.icsi.berkeley.edu/pipermail/time-machine/2011-July/000021.html)

It keeps time-machine from building, and there is no tr1 sub-directory.

BTW: thanks for the filename patch, I'm trying it out...

--
James Welcher <jwelcher at lbl.gov> 1.510.486.5543
Cyber Security, IT Division
Lawrence Berkeley National Laboratory - http://www.lbl.gov


From gregor at icir.org  Wed Jul 20 10:19:13 2011
From: gregor at icir.org (Gregor Maier)
Date: Wed, 20 Jul 2011 10:19:13 -0700
Subject: [TM] [svn] time-machine - r268 - trunk
Message-ID: <201107201719.p6KHJDYb007273@envoy.icir.org>

Author: gregor
Date: 2011-07-20 10:19:13 -0700 (Wed, 20 Jul 2011)
New Revision: 268
Repository: svn.icir.org/time-machine

Modified:
   trunk/Storage.hh
Log:
Remove leftover include


Modified: trunk/Storage.hh
===================================================================
--- trunk/Storage.hh	2011-07-17 17:26:21 UTC (rev 267)
+++ trunk/Storage.hh	2011-07-20 17:19:13 UTC (rev 268)
@@ -41,7 +41,6 @@
 #include <pthread.h>
 #include <list>
 #include <string>
-#include <tr1/unordered_map>
 
 #include "Fifo.hh"
 #include "Connections.hh"


From gregor at icir.org  Wed Jul 20 10:46:47 2011
From: gregor at icir.org (Gregor Maier)
Date: Wed, 20 Jul 2011 10:46:47 -0700
Subject: [TM] [svn] time-machine - r269 - trunk
Message-ID: <201107201746.p6KHklN6007669@envoy.icir.org>

Author: gregor
Date: 2011-07-20 10:46:47 -0700 (Wed, 20 Jul 2011)
New Revision: 269
Repository: svn.icir.org/time-machine

Modified:
   trunk/Connection.hh
   trunk/Connections.cc
   trunk/Fifo.cc
   trunk/Fifo.hh
   trunk/FifoDisk.cc
   trunk/FifoDisk.hh
   trunk/FifoMem.cc
   trunk/FifoMem.hh
   trunk/Storage.cc
   trunk/conf_parser.yy
   trunk/main.cc
   trunk/types.h
Log:
Cleaning up integer types and fixing printf() warnings. Need to cross check on a 32bit machine


Modified: trunk/Connection.hh
===================================================================
--- trunk/Connection.hh	2011-07-20 17:19:13 UTC (rev 268)
+++ trunk/Connection.hh	2011-07-20 17:46:47 UTC (rev 269)
@@ -271,7 +271,7 @@
 	tm_time_t getLastTs() {
 		return last_ts;
 	}
-	byte_count_t getTotPktbytes() {
+	uint64_t getTotPktbytes() {
 		return tot_pktbytes;
 	}
 	//  ConnectionID* getKey() { return key; }
@@ -320,8 +320,8 @@
 
 	//	bool tcp_syn;
 
-	pkt_count_t tot_pkts;
-	byte_count_t tot_pktbytes;
+	uint64_t tot_pkts;
+	uint64_t tot_pktbytes;
 
 	//  hash_t hash() const;
 	//  bool operator==(const Connection& other) const { return c_id==other.c_id; }

Modified: trunk/Connections.cc
===================================================================
--- trunk/Connections.cc	2011-07-20 17:19:13 UTC (rev 268)
+++ trunk/Connections.cc	2011-07-20 17:46:47 UTC (rev 269)
@@ -141,7 +141,7 @@
 	
 void Connections::printStats(FILE *fp) const {
 	fprintf(fp, "Connections Object Statistics\n"
-		   "%"PRINTF_UINT64" total connections stored\n",
+		   "%"PRIu64" total connections stored\n",
 		   getNumEntries()
 		  );
 }

Modified: trunk/Fifo.cc
===================================================================
--- trunk/Fifo.cc	2011-07-20 17:19:13 UTC (rev 268)
+++ trunk/Fifo.cc	2011-07-20 17:46:47 UTC (rev 269)
@@ -64,7 +64,7 @@
 }
 
 
-Fifo::Fifo(const std::string& classname, u_fifosize_t fifo_mem_sz, u_fifosize_t fifo_disk_sz, pcap_t* pcap_handle) {
+Fifo::Fifo(const std::string& classname, uint64_t fifo_mem_sz, uint64_t fifo_disk_sz, pcap_t* pcap_handle) {
 	init();
 	this->classname=classname;
 	this->fifo_mem_sz=fifo_mem_sz;
@@ -122,15 +122,15 @@
 }
 
 
-byte_count_t Fifo::pktEviction() {
-	byte_count_t n=0;
-	pkt_count_t i;
+uint64_t Fifo::pktEviction() {
+	uint64_t n=0;
+	uint64_t i;
 	for (i=0; i<pkts_to_disk && fm->getHeldPkts()>0; i++) {
 		fd->addPkt(fm->getS());
 		n+=fm->popPkt();
 	}
 	if (i<pkts_to_disk) {
-		tmlog(TM_LOG_WARN, "storage", "Strange, only evicted %"PRINTF_UINT64" packets", i);
+		tmlog(TM_LOG_WARN, "storage", "Strange, only evicted %"PRIu64" packets", i);
 	}
 	return n;
 }
@@ -164,15 +164,15 @@
 	char s[STR_SIZE];
 	if (started) {
 		snprintf(s, STR_SIZE, 
-				"%"PRINTF_UINT64" "
-				"%"PRINTF_UINT64" "
-				"%"PRINTF_UINT64" "
-				"%"PRINTF_UINT64" "
-				"%"PRINTF_UINT64" "
-				"%"PRINTF_UINT64" "
+				"%"PRIu64" "
+				"%"PRIu64" "
+				"%"PRIu64" "
+				"%"PRIu64" "
+				"%"PRIu64" "
+				"%"PRIu64" "
 				"%.2lf "
-				"%"PRINTF_UINT64" "
-				"%"PRINTF_UINT64" "
+				"%"PRIu64" "
+				"%"PRIu64" "
 				"%.2lf ",
 			 stored_bytes,
 			 stored_pkts,
@@ -217,9 +217,9 @@
 }
 
 
-pkt_count_t Fifo::query(QueryRequest *qreq, QueryResult *qres,
+uint64_t Fifo::query(QueryRequest *qreq, QueryResult *qres,
 				 IntervalSet *interval_set) {
-	pkt_count_t matches = 0;
+	uint64_t matches = 0;
 	FifoDiskFile *cur_file;
 
 	if (!qreq->isMemOnly()) {

Modified: trunk/Fifo.hh
===================================================================
--- trunk/Fifo.hh	2011-07-20 17:19:13 UTC (rev 268)
+++ trunk/Fifo.hh	2011-07-20 17:46:47 UTC (rev 269)
@@ -62,9 +62,9 @@
   }
 protected:
   std::string classname;
-  u_fifosize_t fifo_mem_sz;
-  u_fifosize_t fifo_disk_sz;
-  byte_count_t cutoff;
+  uint64_t fifo_mem_sz;
+  uint64_t fifo_disk_sz;
+  uint64_t cutoff;
   pcap_t* ph;
 }
 */
@@ -72,16 +72,16 @@
 class Fifo: FifoMemEvictionHandler {
 public:
 	Fifo();
-	Fifo(const std::string& classname, u_fifosize_t fifo_mem_sz,
-		 u_fifosize_t fifo_disk_sz, pcap_t*);
+	Fifo(const std::string& classname, uint64_t fifo_mem_sz,
+		 uint64_t fifo_disk_sz, pcap_t*);
 	void start();
 	virtual ~Fifo();
-	byte_count_t pktEviction();
+	uint64_t pktEviction();
 	bool addPkt(const struct pcap_pkthdr* header, const unsigned char* packet,
 				Connection*);
 	const FifoMem* getFm();
 	const FifoDisk* getFd();
-	void setCutoff(byte_count_t n) {
+	void setCutoff(uint64_t n) {
 		cutoff=n;
 	}
 	bool doCutoff() {
@@ -93,7 +93,7 @@
 	void disableCutoff() {
 		do_cutoff=false;
 	}
-	byte_count_t getCutoff() {
+	uint64_t getCutoff() {
 		return cutoff;
 	}
 	void setClassname(std::string s) {
@@ -102,13 +102,13 @@
 	std::string getClassname() {
 		return classname;
 	}
-	void setFifoMemSz(u_fifosize_t s) {
+	void setFifoMemSz(uint64_t s) {
 		fifo_mem_sz=s;
 	}
-	void setFifoDiskSz(u_fifosize_t s) {
+	void setFifoDiskSz(uint64_t s) {
 		fifo_disk_sz=s;
 	}
-	void setFifoDiskFileSz(u_fifosize_t s) {
+	void setFifoDiskFileSz(uint64_t s) {
 		fifo_disk_filesz=s;
 	}
 	void setPcapHandle(pcap_t* ph) {
@@ -133,42 +133,42 @@
 		return dynTimeout;
 	}
 	bool matchPkt(const struct pcap_pkthdr*, const unsigned char*);
-	byte_count_t getStoredBytes() {
+	uint64_t getStoredBytes() {
 		return stored_bytes;
 	}
-	byte_count_t getCutofBbytes() {
+	uint64_t getCutofBbytes() {
 		return cutoff_bytes;
 	}
-	pkt_count_t getStoredPkts()   {
+	uint64_t getStoredPkts()   {
 		return stored_pkts;
 	}
-	pkt_count_t getCutoffPkts()   {
+	uint64_t getCutoffPkts()   {
 		return cutoff_pkts;
 	}
 	std::string getStatsStr();
 	static std::string getStatsStrHeader();
-	pkt_count_t query(QueryRequest*, QueryResult*, IntervalSet*);
+	uint64_t query(QueryRequest*, QueryResult*, IntervalSet*);
 protected:
 	bool do_cutoff;
-	byte_count_t cutoff;
+	uint64_t cutoff;
 	std::string classname;
 	std::string filter;
-	u_fifosize_t fifo_mem_sz;
-	u_fifosize_t fifo_disk_sz;
-	u_fifosize_t fifo_disk_filesz;
+	uint64_t fifo_mem_sz;
+	uint64_t fifo_disk_sz;
+	uint64_t fifo_disk_filesz;
 	int precedence;
 	tm_time_t dynTimeout;
 	pcap_t* ph;
 	bool started;
 	struct bpf_program fp;
 
-	pkt_count_t pkts_to_disk;
+	uint64_t pkts_to_disk;
 
 	// statistics
-	byte_count_t stored_bytes;
-	byte_count_t cutoff_bytes;
-	pkt_count_t stored_pkts;
-	pkt_count_t cutoff_pkts;
+	uint64_t stored_bytes;
+	uint64_t cutoff_bytes;
+	uint64_t stored_pkts;
+	uint64_t cutoff_pkts;
 
 	FifoMem* fm;
 	FifoDisk* fd;

Modified: trunk/FifoDisk.cc
===================================================================
--- trunk/FifoDisk.cc	2011-07-20 17:19:13 UTC (rev 268)
+++ trunk/FifoDisk.cc	2011-07-20 17:46:47 UTC (rev 269)
@@ -81,8 +81,8 @@
  *    See tm.h 
  */
 
-FifoDisk::FifoDisk(const std::string& classname, u_fifosize_t size,
-				   u_fifosize_t file_size, pcap_t* pcap_handle):
+FifoDisk::FifoDisk(const std::string& classname, uint64_t size,
+				   uint64_t file_size, pcap_t* pcap_handle):
 		classname(classname), size(size), file_size(file_size),
 		tot_bytes(0), tot_pkts(0),
 		file_number(0), pcap_handle(pcap_handle),
@@ -158,10 +158,10 @@
 }
 
 /*
-pkt_count_t FifoDisk::query(QueryRequest *qreq, QueryResult *qres,
+uint64_t FifoDisk::query(QueryRequest *qreq, QueryResult *qres,
 					 IntervalSet *interval_set) {
 	FifoDiskFile *cur_file;
-	pkt_count_t matches=0;
+	uint64_t matches=0;
 	IntervalSet::iterator i_i=interval_set->begin();
 	std::list <FifoDiskFile*>::iterator f_i=files.begin();
 	while ( f_i!=files.end() && i_i != interval_set->end() ) {
@@ -236,8 +236,8 @@
 		   p+sizeof(struct pcap_pkthdr));  // packet
 }
 
-pkt_count_t FifoDiskFile::query( QueryRequest *qreq, QueryResult *qres, IntervalSet *set) {
-	pkt_count_t matches = 0;
+uint64_t FifoDiskFile::query( QueryRequest *qreq, QueryResult *qres, IntervalSet *set) {
+	uint64_t matches = 0;
 	unsigned scanned_packets=0;
 #ifdef HAVE_LIBPCAPNAV
 	ConnectionID4 *c_id;

Modified: trunk/FifoDisk.hh
===================================================================
--- trunk/FifoDisk.hh	2011-07-20 17:19:13 UTC (rev 268)
+++ trunk/FifoDisk.hh	2011-07-20 17:46:47 UTC (rev 269)
@@ -55,18 +55,18 @@
 
 class FifoDisk {
 public:
-	FifoDisk(const std::string& classname, u_fifosize_t size,
-			 u_fifosize_t file_size, pcap_t*);
+	FifoDisk(const std::string& classname, uint64_t size,
+			 uint64_t file_size, pcap_t*);
 	~FifoDisk();
 	//  void addPkt(const struct pcap_pkthdr *header, const unsigned char *packet);
 	void addPkt(const pkt_ptr p);
 	tm_time_t getStartTimestamp();
 	tm_time_t getOldestTimestamp() const;
 	tm_time_t getNewestTimestamp() const;
-	byte_count_t getHeldBytes() {
+	uint64_t getHeldBytes() {
 		return held_bytes;
 	}
-	pkt_count_t getHeldPkts() {
+	uint64_t getHeldPkts() {
 		return held_pkts;
 	};
 	std::list <FifoDiskFile*>::iterator filesBeginIt() {
@@ -106,14 +106,14 @@
 protected:
 	std::string classname;
 	std::list <FifoDiskFile*> files;
-	u_fifosize_t size;
-	u_fifosize_t file_size;
-	byte_count_t tot_bytes;
-	pkt_count_t tot_pkts;
+	uint64_t size;
+	uint64_t file_size;
+	uint64_t tot_bytes;
+	uint64_t tot_pkts;
 	uint32_t file_number;
 	pcap_t* pcap_handle;
-	byte_count_t held_bytes;
-	pkt_count_t held_pkts;
+	uint64_t held_bytes;
+	uint64_t held_pkts;
 	tm_time_t oldestTimestamp;
 	tm_time_t newestTimestamp;
 	pthread_mutex_t query_in_progress_mutex;
@@ -130,13 +130,13 @@
 	void remove();
 	void addPkt(const struct pcap_pkthdr *header, const unsigned char *packet);
 	void addPkt(pkt_ptr p);
-	fifosize_t getCurFileSize() {
+	int64_t getCurFileSize() {
 		return cur_file_size;
 	}
-	byte_count_t getHeldBytes() {
+	uint64_t getHeldBytes() {
 		return held_bytes;
 	}
-	pkt_count_t getHeldPkts() {
+	uint64_t getHeldPkts() {
 		return held_pkts;
 	}
 	tm_time_t getOldestTimestamp() {
@@ -153,14 +153,14 @@
 	}
 	/* iterator will be increased up to the first interval completeley
 	   not in file */
-	pkt_count_t query( QueryRequest*, QueryResult*, IntervalSet*);
+	uint64_t query( QueryRequest*, QueryResult*, IntervalSet*);
 protected:
 	std::string filename;
 	bool is_open;
 	pcap_dumper_t *pcap_dumper_handle;
-	fifosize_t cur_file_size;
-	byte_count_t held_bytes;
-	pkt_count_t held_pkts;
+	int64_t cur_file_size;
+	uint64_t held_bytes;
+	uint64_t held_pkts;
 	pcap_t *pcap_handle;
 	tm_time_t oldest_timestamp;
 	tm_time_t newest_timestamp;

Modified: trunk/FifoMem.cc
===================================================================
--- trunk/FifoMem.cc	2011-07-20 17:19:13 UTC (rev 268)
+++ trunk/FifoMem.cc	2011-07-20 17:46:47 UTC (rev 269)
@@ -46,7 +46,7 @@
 #include "Index.hh"
 
 
-FifoMem::FifoMem(u_fifosize_t size): size(size), oldestTimestamp(0), newestTimestamp(0) {
+FifoMem::FifoMem(uint64_t size): size(size), oldestTimestamp(0), newestTimestamp(0) {
 	start=(unsigned char *)malloc(size+MAXCAPLEN+sizeof(struct pcap_pkthdr)+1);
 	assert(start);
 	end=start+size;
@@ -62,7 +62,7 @@
 	align_num=(size+MAXCAPLEN+sizeof(struct pcap_pkthdr))/align_gran + 1;
 	align=(pkt_ptr*)malloc(align_num*sizeof(pkt_ptr));
 	// DEBUG
-	for (pkt_count_t i=0; i<align_num; i++) 
+	for (uint64_t i=0; i<align_num; i++) 
 		align[i]=start;
 	s=wp=lp=start;
 	a_s=a_lp=a_max=0;
@@ -123,7 +123,7 @@
 			/* previous call to addPkt wrapped wp 
 			   Must make sure that remainder of align array points to a valid addr */
 			assert(wp == start);
-			for (pkt_count_t i=a_lp+1; i<align_num; i++) {
+			for (uint64_t i=a_lp+1; i<align_num; i++) {
 				align[i] = lp;
 			}
 			/* wrap a_lp */ 
@@ -171,28 +171,28 @@
 }
 
 
-pkt_count_t FifoMem::getTotPkts() const {
+uint64_t FifoMem::getTotPkts() const {
 	return tot_pkts;
 }
 
 
-byte_count_t FifoMem::getTotPktbytes() const {
+uint64_t FifoMem::getTotPktbytes() const {
 	return tot_pktbytes;
 }
 
 
-pkt_count_t FifoMem::getTotLostPkts() const {
+uint64_t FifoMem::getTotLostPkts() const {
 	return tot_lost_pkts;
 }
 
 
-byte_count_t FifoMem::getTotLostPktbytes() const {
+uint64_t FifoMem::getTotLostPktbytes() const {
 	return tot_lost_pktbytes;
 }
 
 
-u_fifosize_t FifoMem::popPkt() {
-	u_fifosize_t n=0;
+uint64_t FifoMem::popPkt() {
+	uint64_t n=0;
 	if (held_pkts) {
 		n=((struct pcap_pkthdr *)s)->caplen;
 		s+=sizeof(struct pcap_pkthdr)+((struct pcap_pkthdr *)s)->caplen;
@@ -211,7 +211,7 @@
 	return n;
 }
 
-byte_count_t FifoMem::pktEviction() {
+uint64_t FifoMem::pktEviction() {
 	if (eviction_handler)
 		return eviction_handler->pktEviction();
 	else
@@ -239,8 +239,8 @@
 void FifoMem::debugPrint(FILE *fp) const {
 	fprintf(fp, "\nstart = %ld  s = %ld  wp = %ld  lp = %ld  end = %ld  held_pkts = %lu\n",
 		   (long int)(start-start), (long int)(s-start), (long int)(wp-start), (long int)(lp-start), (long int)(end-start), (long int)held_pkts);
-	fprintf(fp, "a_next = %ld  a_wp = XX  a_s = %lu  a_lp = %lu  a_max = %lu\n",
-		   (long int)(a_next-start), (unsigned long)a_s, (unsigned long)a_lp, (unsigned long)a_max);
+	fprintf(fp, "a_next = %ld  a_wp = XX  a_s = %"PRIu64" a_lp = %"PRIu64" a_max = %"PRIu64"\n",
+		   (long int)(a_next-start), a_s, a_lp, a_max);
 
 	/*
 	align[i] ?
@@ -282,9 +282,9 @@
 }
 
 int FifoMem::bin_search (pkt_ptr *p, tm_time_t t, bool floor) {
-	pkt_count_t a0=a_s, a1=a_lp, at;
+	uint64_t a0=a_s, a1=a_lp, at;
 	pkt_ptr pt;
-	pkt_count_t my_a0, my_a1;
+	uint64_t my_a0, my_a1;
 
 	my_a0 = a0<align_num ? a0 : a0-align_num;
 	my_a1 = a1<align_num ? a1 : a1-align_num; 
@@ -333,10 +333,10 @@
 //in the FifoMem, we will get problems!!!
 //
 //CALLER MUS HOLD THE LOCK ON FIFO MEM!!
-pkt_count_t FifoMem::query(QueryRequest *qreq, QueryResult *qres,
+uint64_t FifoMem::query(QueryRequest *qreq, QueryResult *qres,
 					IntervalSet *interval_set) {
 	pkt_ptr p, p_orig, p_old;
-	pkt_count_t matches = 0;
+	uint64_t matches = 0;
 	ConnectionID4 *c_id;
 	int intcnt=0, iter=0;
 	bool p_will_wrap;

Modified: trunk/FifoMem.hh
===================================================================
--- trunk/FifoMem.hh	2011-07-20 17:19:13 UTC (rev 268)
+++ trunk/FifoMem.hh	2011-07-20 17:46:47 UTC (rev 269)
@@ -57,29 +57,29 @@
 
 class FifoMem {
 public:
-	FifoMem(u_fifosize_t);
+	FifoMem(uint64_t);
 	~FifoMem();
 	void setEvictionHandler(FifoMemEvictionHandler *h);
 	void addPkt(const struct pcap_pkthdr *header, const unsigned char *packet);
-	u_fifosize_t popPkt();
+	uint64_t popPkt();
 	pkt_ptr getWp() const;
 	pkt_ptr getS() const;
-	u_fifosize_t getClearance() const;
-	pkt_count_t getTotPkts() const;
-	byte_count_t getTotPktbytes() const;
-	pkt_count_t getTotLostPkts() const;
-	byte_count_t getTotLostPktbytes() const;
-	pkt_count_t getHeldPkts() const {
+	uint64_t getClearance() const;
+	uint64_t getTotPkts() const;
+	uint64_t getTotPktbytes() const;
+	uint64_t getTotLostPkts() const;
+	uint64_t getTotLostPktbytes() const;
+	uint64_t getHeldPkts() const {
 		return held_pkts;
 	}
-	byte_count_t getHeldBytes() const {
+	uint64_t getHeldBytes() const {
 		return held_bytes;
 	}
 	tm_time_t getOldestTimestamp() const;
 	tm_time_t getNewestTimestamp() const;
 	void debugPrint() const;
 	void debugPrint(FILE *fp) const;
-	pkt_count_t query(QueryRequest*, QueryResult*, IntervalSet*);
+	uint64_t query(QueryRequest*, QueryResult*, IntervalSet*);
 	void lock();
 	void unlock();
 
@@ -92,17 +92,17 @@
 	//    e,                // e
 	s;                // beginning of first valid packet in block
 	pkt_ptr *align;     // list of positions where packets are aligned
-	u_fifosize_t align_num;  //
-	u_fifosize_t align_gran;  //
+	uint64_t align_num;  //
+	uint64_t align_gran;  //
 
-	pkt_count_t a_s, a_lp, a_max;
+	uint64_t a_s, a_lp, a_max;
 	pkt_ptr a_next;
-	u_fifosize_t size;  // size of block in bytes
-	pkt_count_t
+	uint64_t size;  // size of block in bytes
+	uint64_t
 	tot_pkts,     // packets entered into this Fifo
 	tot_lost_pkts,
 	held_pkts;    // packets currently held in block
-	byte_count_t
+	uint64_t
 	tot_pktbytes, // bytes entered into this Fifo (excluding pcap hdrs)
 	tot_lost_pktbytes,
 	held_bytes;
@@ -116,7 +116,7 @@
 	inline pkt_ptr block (pkt_ptr p);
 	int bin_search (pkt_ptr *p, tm_time_t t, bool floor);
 
-	byte_count_t pktEviction();
+	uint64_t pktEviction();
 
 };
 
@@ -126,7 +126,7 @@
 	virtual ~FifoMemEvictionHandler() { }
 	// stuff to do before a packet will get evicted from memory
 	// returns true if it removed a packet
-	virtual byte_count_t pktEviction() = 0;
+	virtual uint64_t pktEviction() = 0;
 };
 
 

Modified: trunk/Storage.cc
===================================================================
--- trunk/Storage.cc	2011-07-20 17:19:13 UTC (rev 268)
+++ trunk/Storage.cc	2011-07-20 17:46:47 UTC (rev 269)
@@ -385,13 +385,13 @@
 void Storage::debugPrint(FILE *fp) {
 	//  Fifo* f[]={&f_udp, &f_tcp, &f_other};
 	for (std::list<Fifo*>::iterator i=fifos.begin(); i!=fifos.end(); i++)
-		fprintf(fp, "%s totBytes: %"PRINTF_UINT64"\n", (*i)->getClassname().c_str(),
+		fprintf(fp, "%s totBytes: %"PRIu64"\n", (*i)->getClassname().c_str(),
 			   (*i)->getFm()->getTotPktbytes());
 	for (std::list<IndexType*>::iterator i=indexes->begin();
 			i!=indexes->end();
 			i++) {
 		//(*i)->lock();
-		fprintf(fp, "# %s index nodes RAM/Disk %"PRINTF_UINT64" %"PRINTF_UINT64"\n",
+		fprintf(fp, "# %s index nodes RAM/Disk %"PRIu64" %"PRIu64"\n",
 			   (*i)->getIndexName().c_str(),
 			   (*i)->getNumEntriesRAM(),
 			   (*i)->getNumEntriesDisk());

Modified: trunk/conf_parser.yy
===================================================================
--- trunk/conf_parser.yy	2011-07-20 17:19:13 UTC (rev 268)
+++ trunk/conf_parser.yy	2011-07-20 17:46:47 UTC (rev 269)
@@ -229,9 +229,9 @@
 	;
 option:
 	TOK_ID TOK_INTEGER {
-	  printf("option: ignored option %s int value %"PRINTF_INT64"\n", $1, $2);
+	  printf("option: ignored option %s int value %"PRIi64"\n", $1, $2);
 	  $$=(char*)malloc(31);
-	  snprintf($$, 30, "%"PRINTF_INT64, $2);
+	  snprintf($$, 30, "%"PRIi64, $2);
 	  free($1);
 	}
 	| TOK_ID TOK_STRING {

Modified: trunk/main.cc
===================================================================
--- trunk/main.cc	2011-07-20 17:19:13 UTC (rev 268)
+++ trunk/main.cc	2011-07-20 17:46:47 UTC (rev 269)
@@ -208,8 +208,8 @@
 	fprintf(outfp, "%u\tpkts received\t[pcap_stats]\n"
 		   "%u\tpkts dropped by kernel\t[pcap_stats]\n"
 		   "%.2f\t%% dropped/received\t[pcap_stats]\n"
-		   "%"PRINTF_UINT64"\tpkts total received by capture application\n"
-		   "%"PRINTF_UINT64"\tbytes total\n",
+		   "%"PRIu64"\tpkts total received by capture application\n"
+		   "%"PRIu64"\tbytes total\n",
 		   ps.ps_recv, ps.ps_drop,
 		   (float)ps.ps_drop/ps.ps_recv*100,
 		   tot_pkt_cnt, tot_bytes);
@@ -493,24 +493,24 @@
 
 		snprintf(s, LOG_STRING_SIZE,
 				 "%.2lf "
-				 "%"PRINTF_UINT64" "
+				 "%"PRIu64" "
 				 "%d " 
 				 "%u "
 				 "%u " 
 				 "%.2lf%% "
 				 "%u "
-				 "%"PRINTF_UINT64" "
-				 "%"PRINTF_UINT64" "
+				 "%"PRIu64" "
+				 "%"PRIu64" "
 				 "%.2lf "
-				 "%"PRINTF_UINT64" "
-				 "%"PRINTF_UINT64" "
+				 "%"PRIu64" "
+				 "%"PRIu64" "
 				 "%.2lf "
 				 "%.1lf "
 				 "%.1lf "
-				 "%"PRINTF_UINT64" "
-				 "%"PRINTF_UINT64" "
-				 "%"PRINTF_UINT64" "
-				 "%"PRINTF_UINT64" ",
+				 "%"PRIu64" "
+				 "%"PRIu64" "
+				 "%"PRIu64" "
+				 "%"PRIu64" ",
 				 to_tm_time(&now),
 				 storage->getConns().getNumEntries(),
 				 storage->getNumDynClasses(),
@@ -552,7 +552,7 @@
 			snprintf(s, LOG_STRING_SIZE, 
 					"%.2lf "
 					"%s "
-					"%"PRINTF_UINT64" "
+					"%"PRIu64" "
 					"%u ",
 					to_tm_time(&now),
 					(*i)->getIndexName().c_str(),

Modified: trunk/types.h
===================================================================
--- trunk/types.h	2011-07-20 17:19:13 UTC (rev 268)
+++ trunk/types.h	2011-07-20 17:46:47 UTC (rev 269)
@@ -39,29 +39,13 @@
 
 #include <sys/types.h>
 #include <sys/time.h>
+// Expose C99 functionality from inttypes.h, which would otherwise not be
+// available in C++.
+#define __STDC_FORMAT_MACROS
 #include <inttypes.h>
 
 #include "config.h"
 
-#if SIZEOF_VOIDP == 8
-// 64 bit systems
-#define PRINTF_UINT64 "lu"
-#define PRINTF_INT64 "ld"
-#define PRINTF_SIZE "lu"
-#else
-// 32 bit systems
-#define PRINTF_UINT64 "llu"
-#define PRINTF_INT64 "lld"
-#define PRINTF_SIZE "u"
-#endif
-
-
-typedef int64_t fifosize_t;
-typedef uint64_t u_fifosize_t;
-
-typedef uint64_t pkt_count_t;
-typedef uint64_t byte_count_t;
-
 typedef u_char* pkt_ptr;
 
 typedef double tm_time_t;


From gregor at icir.org  Wed Jul 20 11:56:46 2011
From: gregor at icir.org (Gregor Maier)
Date: Wed, 20 Jul 2011 11:56:46 -0700
Subject: [TM] [svn] time-machine - r270 - trunk
Message-ID: <201107201856.p6KIukxq008757@envoy.icir.org>

Author: gregor
Date: 2011-07-20 11:56:46 -0700 (Wed, 20 Jul 2011)
New Revision: 270
Repository: svn.icir.org/time-machine

Removed:
   trunk/config.h.in
Modified:
   trunk/Hash.cc
   trunk/tm.h
   trunk/types.h
Log:
config.h.in

Modified: trunk/Hash.cc
===================================================================
--- trunk/Hash.cc	2011-07-20 17:46:47 UTC (rev 269)
+++ trunk/Hash.cc	2011-07-20 18:56:46 UTC (rev 270)
@@ -38,6 +38,7 @@
 
 #include <algorithm>
 #include <assert.h>
+#include <stdio.h>
 
 #include "Hash.hh"
 

Deleted: trunk/config.h.in
===================================================================
--- trunk/config.h.in	2011-07-20 17:46:47 UTC (rev 269)
+++ trunk/config.h.in	2011-07-20 18:56:46 UTC (rev 270)
@@ -1,101 +0,0 @@
-/* config.h.in.  Generated from configure.in by autoheader.  */
-
-/* Define to 1 if you have the <broccoli.h> header file. */
-#undef HAVE_BROCCOLI_H
-
-/* Define to 1 if you have the <inttypes.h> header file. */
-#undef HAVE_INTTYPES_H
-
-/* Define to 1 if you have the `broccoli' library (-lbroccoli). */
-#undef HAVE_LIBBROCCOLI
-
-/* Define to 1 if you have the `pcap' library (-lpcap). */
-#undef HAVE_LIBPCAP
-
-/* Define to 1 if you have the `pcapnav' library (-lpcapnav). */
-#undef HAVE_LIBPCAPNAV
-
-/* Define to 1 if you have the `pcre' library (-lpcre). */
-#undef HAVE_LIBPCRE
-
-/* Define to 1 if you have the `pcrecpp' library (-lpcrecpp). */
-#undef HAVE_LIBPCRECPP
-
-/* Define to 1 if you have the `pthread' library (-lpthread). */
-#undef HAVE_LIBPTHREAD
-
-/* Define to 1 if you have the <memory.h> header file. */
-#undef HAVE_MEMORY_H
-
-/* Define to 1 if you have the <pcapnav.h> header file. */
-#undef HAVE_PCAPNAV_H
-
-/* Define to 1 if you have the <pcap.h> header file. */
-#undef HAVE_PCAP_H
-
-/* Define to 1 if you have the <pcrecpp.h> header file. */
-#undef HAVE_PCRECPP_H
-
-/* Define to 1 if you have the <pthread.h> header file. */
-#undef HAVE_PTHREAD_H
-
-/* Define to 1 if you have the <readline/readline.h> header file. */
-#undef HAVE_READLINE_READLINE_H
-
-/* Define to 1 if you have the <stdint.h> header file. */
-#undef HAVE_STDINT_H
-
-/* Define to 1 if you have the <stdlib.h> header file. */
-#undef HAVE_STDLIB_H
-
-/* Define to 1 if you have the <strings.h> header file. */
-#undef HAVE_STRINGS_H
-
-/* Define to 1 if you have the <string.h> header file. */
-#undef HAVE_STRING_H
-
-/* Define to 1 if you have the <sys/stat.h> header file. */
-#undef HAVE_SYS_STAT_H
-
-/* Define to 1 if you have the <sys/types.h> header file. */
-#undef HAVE_SYS_TYPES_H
-
-/* Define to 1 if you have the <unistd.h> header file. */
-#undef HAVE_UNISTD_H
-
-/* Define to 1 if your C compiler doesn't accept -c and -o together. */
-#undef NO_MINUS_C_MINUS_O
-
-/* Name of package */
-#undef PACKAGE
-
-/* Define to the address where bug reports for this package should be sent. */
-#undef PACKAGE_BUGREPORT
-
-/* Define to the full name of this package. */
-#undef PACKAGE_NAME
-
-/* Define to the full name and version of this package. */
-#undef PACKAGE_STRING
-
-/* Define to the one symbol short name of this package. */
-#undef PACKAGE_TARNAME
-
-/* Define to the home page for this package. */
-#undef PACKAGE_URL
-
-/* Define to the version of this package. */
-#undef PACKAGE_VERSION
-
-/* The size of `void*', as computed by sizeof. */
-#undef SIZEOF_VOIDP
-
-/* Define to 1 if you have the ANSI C header files. */
-#undef STDC_HEADERS
-
-/* Version number of package */
-#undef VERSION
-
-/* Define to 1 if `lex' declares `yytext' as a `char *' by default, not a
-   `char[]'. */
-#undef YYTEXT_POINTER

Modified: trunk/tm.h
===================================================================
--- trunk/tm.h	2011-07-20 17:46:47 UTC (rev 269)
+++ trunk/tm.h	2011-07-20 18:56:46 UTC (rev 270)
@@ -39,6 +39,8 @@
 
 #include <string>
 
+#include "types.h"
+
 // #define QUERY_RACE_PROTECT
 
 

Modified: trunk/types.h
===================================================================
--- trunk/types.h	2011-07-20 17:46:47 UTC (rev 269)
+++ trunk/types.h	2011-07-20 18:56:46 UTC (rev 270)
@@ -37,12 +37,12 @@
 #ifndef TM_TYPES_H
 #define TM_TYPES_H
 
-#include <sys/types.h>
-#include <sys/time.h>
 // Expose C99 functionality from inttypes.h, which would otherwise not be
 // available in C++.
 #define __STDC_FORMAT_MACROS
 #include <inttypes.h>
+#include <sys/types.h>
+#include <sys/time.h>
 
 #include "config.h"
 


From gregor at ICSI.Berkeley.EDU  Wed Jul 20 11:58:21 2011
From: gregor at ICSI.Berkeley.EDU (Gregor Maier)
Date: Wed, 20 Jul 2011 11:58:21 -0700
Subject: [TM] [svn] time-machine - r270 - trunk
In-Reply-To: <201107201856.p6KIukxq008757@envoy.icir.org>
References: <201107201856.p6KIukxq008757@envoy.icir.org>
Message-ID: <4E27254D.8020003@icsi.berkeley.edu>

grrmpf. Actually the log message was supposed to be: fixing some compile 
issues.

On 7/20/11 11:56 , Gregor Maier wrote:
> Author: gregor
> Date: 2011-07-20 11:56:46 -0700 (Wed, 20 Jul 2011)
> New Revision: 270
> Repository: svn.icir.org/time-machine
>
> Removed:
>     trunk/config.h.in
> Modified:
>     trunk/Hash.cc
>     trunk/tm.h
>     trunk/types.h
> Log:
> config.h.in



-- 
Gregor Maier
<gregor at icir.org>  <gregor at icsi.berkeley.edu>
Int. Computer Science Institute (ICSI)
1947 Center St., Ste. 600
Berkeley, CA 94704, USA
http://www.icir.org/gregor/

From gregor at icir.org  Wed Jul 20 17:54:46 2011
From: gregor at icir.org (Gregor Maier)
Date: Wed, 20 Jul 2011 17:54:46 -0700
Subject: [TM] [git] master: Test commit (14d706d)
Message-ID: <201107210054.p6L0skrS002454@bro-ids.icir.org>

Repository : ssh://git at bro-ids.icir.org/time-machine

On branch  : master

>---------------------------------------------------------------

commit 14d706d9c67ddbd80761b6ae7ea4e689a7b43f4d
Author: Gregor Maier <gregor at icir.org>
Date:   Wed Jul 20 17:54:32 2011 -0700

    Test commit


>---------------------------------------------------------------

 README |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/README b/README
index b724938..cbd4c0e 100644
--- a/README
+++ b/README
@@ -17,3 +17,4 @@ with a NIDS.
 
 ATTENTION FreeBSD Users: You really want to read the MEMORY
 ALLOCATOR section of the the TUNING file.
+


From time-machine at ICSI.Berkeley.EDU  Thu Jul 21 11:46:40 2011
From: time-machine at ICSI.Berkeley.EDU (Time Machine Tracker)
Date: Thu, 21 Jul 2011 18:46:40 -0000
Subject: [TM] [The Time Machine] Password reset for user: gregor
Message-ID: <201107211846.p6LIkefT011362@bro-ids.icir.org>

Password reset for user for user gregor

--
The Time Machine <http://tracker.icir.org/time-machine>
High-volume network traffic stream recorder.



From time-machine at ICSI.Berkeley.EDU  Thu Jul 21 11:49:00 2011
From: time-machine at ICSI.Berkeley.EDU (Time Machine Tracker)
Date: Thu, 21 Jul 2011 18:49:00 -0000
Subject: [TM] [The Time Machine] Password reset for user: gregor
Message-ID: <201107211849.p6LIn0Tm011725@bro-ids.icir.org>

Password reset for user for user gregor

--
The Time Machine <http://tracker.icir.org/time-machine>
High-volume network traffic stream recorder.



From time-machine at ICSI.Berkeley.EDU  Thu Jul 21 11:58:55 2011
From: time-machine at ICSI.Berkeley.EDU (Time Machine Tracker)
Date: Thu, 21 Jul 2011 18:58:55 -0000
Subject: [TM]
 =?utf-8?b?IzU6IOKAmHVpbnQ2NF904oCZIGRvZXMgbm90IG5hbWUgYSB0?=
 =?utf-8?q?ype?=
In-Reply-To: <054.1667e0880bcbce6c73b6f158c49e8099@icsi.berkeley.edu>
References: <054.1667e0880bcbce6c73b6f158c49e8099@icsi.berkeley.edu>
Message-ID: <069.f40f5d0ccf5fae10d01e86949b18c366@icsi.berkeley.edu>

#5: ?uint64_t? does not name a type
-------------------------+----------------------
  Reporter:  sroddy      |      Owner:  somebody
      Type:  defect      |     Status:  closed
  Priority:  trivial     |  Milestone:
 Component:  component1  |    Version:
Resolution:  fixed       |   Keywords:
-------------------------+----------------------
Changes (by gregor):

 * status:  new => closed
 * resolution:   => fixed


Comment:

 fixed in current master

-- 
Ticket URL: <http://tracker.icir.org/time-machine/ticket/5#comment:1>
The Time Machine <http://tracker.icir.org/time-machine>
High-volume network traffic stream recorder.


From time-machine at ICSI.Berkeley.EDU  Thu Jul 21 11:59:54 2011
From: time-machine at ICSI.Berkeley.EDU (Time Machine Tracker)
Date: Thu, 21 Jul 2011 18:59:54 -0000
Subject: [TM]
 =?utf-8?b?IzY6IOKAmHN0ZGVycuKAmSB3YXMgbm90IGRlY2xhcmVkIGlu?=
 =?utf-8?q?_this_scope?=
In-Reply-To: <054.acbe61dc3bbba0888e9797725559341a@icsi.berkeley.edu>
References: <054.acbe61dc3bbba0888e9797725559341a@icsi.berkeley.edu>
Message-ID: <069.cd30b02c5187a844d9bd03572fe2816f@icsi.berkeley.edu>

#6: ?stderr? was not declared in this scope
-------------------------+----------------------
  Reporter:  sroddy      |      Owner:  somebody
      Type:  defect      |     Status:  closed
  Priority:  minor       |  Milestone:
 Component:  component1  |    Version:
Resolution:  fixed       |   Keywords:
-------------------------+----------------------
Changes (by gregor):

 * status:  new => closed
 * resolution:   => fixed


Comment:

 fixed in current master

-- 
Ticket URL: <http://tracker.icir.org/time-machine/ticket/6#comment:1>
The Time Machine <http://tracker.icir.org/time-machine>
High-volume network traffic stream recorder.


From gregor at icir.org  Thu Jul 21 15:26:28 2011
From: gregor at icir.org (Gregor Maier)
Date: Thu, 21 Jul 2011 15:26:28 -0700
Subject: [TM] [git] master: Updating TODO list. (12029bb)
Message-ID: <201107212226.p6LMQSVm014664@bro-ids.icir.org>

Repository : ssh://git at bro-ids.icir.org/time-machine

On branch  : master

>---------------------------------------------------------------

commit 12029bb296c62c3595a878c653b358b4b6140bb5
Author: Gregor Maier <gregor at icir.org>
Date:   Thu Jul 21 15:13:56 2011 -0700

    Updating TODO list.
    
    Mostly adding ideas that have been floating around for while (but also
    add some newer ideas).


>---------------------------------------------------------------

 TODO |   99 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 files changed, 96 insertions(+), 3 deletions(-)

diff --git a/TODO b/TODO
index 8e10af4..50f78f8 100644
--- a/TODO
+++ b/TODO
@@ -1,6 +1,101 @@
 TODOs
 
-* ORDER AND CLEANUP THIS LIST ;-)
+A) The smaller items:
+=====================
+
+* Move build system to cmake. Can probably borrow quite some chunks from
+  Bro's new build system. 
+
+* TM restart. This is probably the most pressing issue!
+  Currently when the TM restarts (or crashes) it cannot use
+  the data it still has on its disk. It would be great if the restart
+  could take this data into account. There are several options to do
+  this:
+
+  + Re-read the stored files (they are in pcap format) and/or the index
+    files and rebuild the full internal state and continue after the
+    last file. This enables queries for stored data. However, when
+    large disk-storage is used re-reading the files might well take
+    several hours
+
+  + Do not re-read the stored files, but learn about them and include
+    them in buffer management. I.e., the TM starts building its internal
+    state only from data that newly arrives, but it knows that there are
+    old files lying around and it will delete the old files in order to
+    stay within the buffer budget. Then the old files can be searched
+    manually with tcpdump/tcpslice/whatever and restart is pretty much
+    instantaneous.
+
+* TM-cluster mode. This should be fairly easy. We would need a TM
+  cluster front-end. Bro would then communicate with the front-end. The
+  front-end sends the request to its workers (maybe with some
+  intelligence to only query the workers that see the traffic according
+  to the load balancing scheme) and gathers the results from the works,
+  sorts them (by time) and delivers them to the requesting Bro.
+
+* Use a directory/inventory for disk searches. Currently disk queries
+  are done using pcapnav to try to find the "right" location in a
+  file (probabilistically jump to an offset and try to see if the
+  offset is a valid start of a pcap-record).
+  It would be good if the TM could store a directory for each pcap file
+  it writes. The directory could then contain the file offset of
+  each n-th packet + timestamp. A query can then just check the
+  directory for the best location to jump to. No probabilist search.
+  (Maybe this should be part of (B) though)
+  We can then get rid of pcapnav as a dependency
+
+
+B) TM code rework
+=================
+
+In general some of the biggest problems of the TM are IMHO:
+
+* poor write performance. The memory buffer is not really used as elastic
+  storage (packets are only moved from memory to disk once the memory buffer
+  is full). Thus disk can block the capturing thread and thus lead to packet
+  loss.
+  Solution: have the most current packets in memory and on disk at the same
+  time by
+  + write to memory first. A second disk-writer thread will then read
+	packets from memory and write them to disk as soon as possible (TODO:
+	try to minimize lock/unlock operations
+  + write to memory and a to-disk-queue at the same time. A disk-writer
+    thread will then pick up the data from this to-disk-queue and write it
+	to disk. 
+
+
+* Index generation. Currently the capture thread generates for each stored
+  packet IndexField* (i.e., the index keys for this packet) and then places 
+  these pointers in per-Index queues. The index threads then pick up the 
+  IndexFields from these queues and store them. 
+  When we rework how packets are stored on disk (see above) it might be
+  worthwhile to also change the way the IndexFields are passed to the 
+  Index threads. E.g., if start using a disk-writer thread then this 
+  disk writer thread could generate the IndexField* and pass them on to
+  the Index threads. This would reduce the number of lock/unlocks the 
+  capture thread needs to do. 
+
+* inflexible, hard-coded indexes. Slow-ish lookup performance for on-disk
+  queries.
+  All possible key combinations (e.g.,
+  2-tuple, 5-tuple, etc.) have to be specified at compile-time. It would
+  be great if the TM could support queries for any combination of
+  IP,IP,port,port,transport.
+  Using fastbit and an indirection could help here. I have some
+  early ideas on how this could be done.
+
+* Keep flow records in addition to packet data and keep it *longer*.
+  The TM pretty implicitly keeps "flow" data for the connections it
+  has in its storage. We could extend this to actually write the flow
+  records to disk and assign a separate disk budget for such flow
+  records. This would allow us to store flow records for significantly
+  longer than just packet data. So it would increase the amount of time
+  we can "travel back", but with less information.
+
+
+-------------------------------
+UNSORTED ITEMS:
+
 
 FOR THE PAPER
 * Concurrent queries
@@ -30,8 +125,6 @@ intervals
 * Handle Queries with syntax errors
 * There's an awful mix of iostreams, Strings, char *, stdout, stderr .... ==> Solve this.
 * Make stats logfile configureable
-* There are heaps of different typedefs for sizes in the Fifos, but none is used
-consistently. 
 * held_bytes / stored_bytes / total_bytes / ... whatever they may be called are
 inconsistend. Some use caplen, some wirelen, some caplen+pcap_header, etc.
 


From gregor at icir.org  Thu Jul 21 15:34:06 2011
From: gregor at icir.org (Gregor Maier)
Date: Thu, 21 Jul 2011 15:34:06 -0700
Subject: [TM] TimeMachine website and repository have moved
Message-ID: <4E28A95E.5080502@icir.org>

Dear TimeMachine users,

in following up with the infrastructure changes for Bro the Time Machine 
also has a new website and repository now:

You can find the TimeMachine's website at:
http://tracker.bro-ids.org/time-machine

The code respository is now using git and can be found at
git://git.bro-ids.org/time-machine


-- 
Gregor Maier
<gregor at icir.org>  <gregor at icsi.berkeley.edu>
Int. Computer Science Institute (ICSI)
1947 Center St., Ste. 600
Berkeley, CA 94704, USA
http://www.icir.org/gregor/

From gregor at icir.org  Fri Jul 22 13:33:11 2011
From: gregor at icir.org (Gregor Maier)
Date: Fri, 22 Jul 2011 13:33:11 -0700
Subject: [TM] [git] master: Fixing autogen.sh which runs the autotools
	toolchain. (e1dd50d)
Message-ID: <201107222033.p6MKXBsg009991@bro-ids.icir.org>

Repository : ssh://git at bro-ids.icir.org/time-machine

On branch  : master

>---------------------------------------------------------------

commit e1dd50d82dbac6b25c70da3fce3b62c02bb005a1
Author: Gregor Maier <gregor at icir.org>
Date:   Fri Jul 22 13:32:29 2011 -0700

    Fixing autogen.sh which runs the autotools toolchain.


>---------------------------------------------------------------

 autogen.sh |   44 +++++++++++++++++++++++---------------------
 1 files changed, 23 insertions(+), 21 deletions(-)

diff --git a/autogen.sh b/autogen.sh
index 2fd814c..ce2399c 100755
--- a/autogen.sh
+++ b/autogen.sh
@@ -1,24 +1,26 @@
 
 # $Id: autogen.sh 161 2006-12-19 02:35:21Z gregor $
 
-if which aclocal-1.9 > /dev/null 2>/dev/null; then
-	ACLOCAL=aclocal-1.9
-elif which aclocal19 > /dev/null 2>/dev/null; then
-	ACLOCAL=aclocal19
-else 
-	echo "Could not found aclocal-1.9 or aclocal19. Exiting"
-	exit
-fi
-
-if which  automake-1.9 > /dev/null 2>/dev/null; then
-	AM=automake-1.9
-elif which  automake19 > /dev/null 2>/dev/null ; then
-	AM=automake19
-else 
-	echo "Could not found automake-1.9 or automake19. Exiting"
-	exit
-fi
+aclocal && autoheader && autoconf && automake --add-missing --copy
 
+#if which aclocal-1.9 > /dev/null 2>/dev/null; then
+#	ACLOCAL=aclocal-1.9
+#elif which aclocal19 > /dev/null 2>/dev/null; then
+#	ACLOCAL=aclocal19
+#else 
+#	echo "Could not found aclocal-1.9 or aclocal19. Exiting"
+#	exit
+#fi
+#
+#if which  automake-1.9 > /dev/null 2>/dev/null; then
+#	AM=automake-1.9
+#elif which  automake19 > /dev/null 2>/dev/null ; then
+#	AM=automake19
+#else 
+#	echo "Could not found automake-1.9 or automake19. Exiting"
+#	exit
+#fi
+#
 
 #aclocal-1.9 && autoheader && autoconf && automake-1.9 --add-missing --copy
 
@@ -26,7 +28,7 @@ fi
 # autoconf    depends on aclocal
 # configure depends on everything
 # no other dependencies
-${ACLOCAL} \
-	&& autoheader \
-	&&  autoconf \
-	&& ${AM} --add-missing --copy
+#${ACLOCAL} \
+#	&& autoheader \
+#	&&  autoconf \
+#	&& ${AM} --add-missing --copy