[TM] [svn] time-machine - r267 - in trunk: . Documentation
Gregor Maier
gregor at icir.org
Sun Jul 17 10:26:22 PDT 2011
Author: gregor
Date: 2011-07-17 10:26:21 -0700 (Sun, 17 Jul 2011)
New Revision: 267
Repository: svn.icir.org/time-machine
Modified:
trunk/Documentation/TM_HOWTO
trunk/Index.cc
trunk/Index.hh
trunk/Storage.cc
trunk/Storage.hh
trunk/conf_parser.yy
trunk/conf_scanner.ll
trunk/config.h.in
trunk/configure.in
trunk/tm.conf
Log:
Make indexes configurable. One can now enable/disable indexes in the configuration file. One can also enable/disable disk indexes for a particular index. Needs some more extensive testing on live traffic.
Modified: trunk/Documentation/TM_HOWTO
===================================================================
--- trunk/Documentation/TM_HOWTO 2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/Documentation/TM_HOWTO 2011-07-17 17:26:21 UTC (rev 267)
@@ -1,5 +1,7 @@
$Id$
+# vim: expandtab
+
Also have a look at the TM's homepage and at the papers in this
directory. They give a good idea of what one can do with a
Time Machine
@@ -56,8 +58,8 @@
TCP port <port> via the Bro communication protocol. The Bro connection
is established by the Time Machine upon TM startup. TM uses the
Broccoli library for communcation with Bro.
- This option is deprecated. The new method is that the TM listens for
- incomming connections from Bro. See the bro_listen_* options
+ This option is deprecated. The new method is that the TM listens for
+ incomming connections from Bro. See the bro_listen_* options
console 0|1
Determine whether a command line interface should be displayed on the
@@ -66,7 +68,7 @@
daemon 0|1
Run the timemachine as a daemon in background mode. Incompatible with
- console.
+ console.
workdir "<path>"
Determine the working directory where all class storage files and index
@@ -98,20 +100,20 @@
rmtconsole 0|1
Start a remote console listener. This will listen on a network socket
- for incomming connections. When connected a remote user can issue
- timemachine commands as could be done from the locale console. I.e. use
- telnet to connect.
- Please note, that this connection is not authenticated and not
- encrypted. Use with care.
+ for incomming connections. When connected a remote user can issue
+ timemachine commands as could be done from the locale console. I.e. use
+ telnet to connect.
+ Please note, that this connection is not authenticated and not
+ encrypted. Use with care.
rmtconsole_port <number>
The local port to listen for incoming remote console connections.
- Default is 42042.
-
+ Default is 42042.
+
rmtconsole_listen_addr <ip address>
The IP address to listen for incoming remote console connections.
- Default is 127.0.0.1. Hostnames cannot be used. A value of 0.0.0.0 will
- listen on all interfaces.
+ Default is 127.0.0.1. Hostnames cannot be used. A value of 0.0.0.0 will
+ listen on all interfaces.
bro_listen 0|1
Start a Bro listener. This will listen on a network socket for
@@ -123,7 +125,7 @@
bro_listen_port <number>
The local port to listen for incoming Bro connections.
- Default is 47757.
+ Default is 47757.
bro_listen_addr <ip address>
The IP address to listen for incoming Bro connections. Default
@@ -132,14 +134,23 @@
tweak_capture_thread priority|scope|no
This option can only be used on FreeBSD. It will tweak thread scheduling
- parameters for the capture thread (see Documentation/TUNIG)
- NOTE NOTE NOTE: You will have to experiment with these settings, since
- different hardware platforms und FreeBSD releases behave quite differently!
+ parameters for the capture thread (see Documentation/TUNIG)
+ NOTE NOTE NOTE: You will have to experiment with these settings, since
+ different hardware platforms und FreeBSD releases behave quite differently!
priority ... Increase the capture thread's priority to realtime priority
- scope ...... Set the capture thread's contentio scope to system scope.
- no ......... Do tweak scheduling parameters (default).
+ scope ...... Set the capture thread's contentio scope to system scope.
+ no ......... Do tweak scheduling parameters (default).
+ index "<index_name>" [disk]
+ Enable the index named <index_name>. Currently supported indexes are
+ "connection4", "connection3", "connection2", and "ip". You need to
+ enable an index in order to use it for queries. The keyword disk enables
+ the disk index for this index. Only indexes with an enabled disk index
+ can perform on disk queries.
+ Disabling indexes can safe significant CPU time and disabling disk indexes
+ can reduce disk usage.
+
* class section
A class section in the configuration file is started by
@@ -185,8 +196,8 @@
dyn_timeout <double>
The timeout for dynamic classes. If a dynamic rule for an IP Adresses
- is pointing to this class, the dynamic rule be removed dyn_timeout
- seconds after the rule has been set.
+ is pointing to this class, the dynamic rule be removed dyn_timeout
+ seconds after the rule has been set.
INDEXES
@@ -247,32 +258,32 @@
<queryspec> ::= index <indexname> "<key-specification>"
<query-flags> ::= start <timestamp> end <timestamp> |
mem_only | subscribe
- The flags can be given in order and they can be combined.
-
- Query the indexes for the given index key. The result of a query
- can either be send to a remote bro system or to a file in the
- local filesystem.
- The index to query is specified by the keyword index followed
- by the name of the index. This name corresponds to the
- name that is returned by the getIndexNameStatic() method.
- Examples are connection4, connection3, ip, etc.
- Finally the key to search is specified by
- <key-specification>. The sematics of the key spec is defined
- by the index itself. For example a valid spec for connection4
- would be "tcp 1.2.3.4:80 5.6.7.8:88"
- The <query-flags> enables one to restrict the search or to set
- a subsciption (see above). Currently only connection4 querys
- support the subscribe flag, other indexes will silently ignore
- the flag.
- When mem_only is specified, only the index entries stored in RAM
- are searched and only packets from the memomry ringbuffer are
- returned.
- The timestamps enable one, to specify a timespan. Only packets
- falling in this timespan will be returned. timestamps and mem_only
- can be combined. The result will be the intersection of both
- (i.e. only packets from memory, that fall into the specified
- timefragme).
- The timespan has not been tested extensively.
+ The flags can be given in order and they can be combined.
+
+ Query the indexes for the given index key. The result of a query
+ can either be send to a remote bro system or to a file in the
+ local filesystem.
+ The index to query is specified by the keyword index followed
+ by the name of the index. This name corresponds to the
+ name that is returned by the getIndexNameStatic() method.
+ Examples are connection4, connection3, ip, etc.
+ Finally the key to search is specified by
+ <key-specification>. The sematics of the key spec is defined
+ by the index itself. For example a valid spec for connection4
+ would be "tcp 1.2.3.4:80 5.6.7.8:88"
+ The <query-flags> enables one to restrict the search or to set
+ a subsciption (see above). Currently only connection4 querys
+ support the subscribe flag, other indexes will silently ignore
+ the flag.
+ When mem_only is specified, only the index entries stored in RAM
+ are searched and only packets from the memomry ringbuffer are
+ returned.
+ The timestamps enable one, to specify a timespan. Only packets
+ falling in this timespan will be returned. timestamps and mem_only
+ can be combined. The result will be the intersection of both
+ (i.e. only packets from memory, that fall into the specified
+ timefragme).
+ The timespan has not been tested extensively.
Examples
# query to_file "file1.pcap" index connection4 "tcp 1.2.3.4:80 5.6.7.8:1025" subscribe
# query to_file "file1.pcap" index connection4 "tcp 1.2.3.4:80 5.6.7.8:1025"
@@ -290,7 +301,7 @@
unsuspend_cutoff "<proto> <ip>:<port> <ip>:<port>"
Suspend resp. remove the supension of the cutoff for one connection. If a connection
- is supended, all packets will get recorded and the cutoff value is ignored
+ is supended, all packets will get recorded and the cutoff value is ignored
suspend_timeout "<proto> <ip>:<port> <ip>:<port>"
Inhibit the eviction of the specified connection from the connection
@@ -310,9 +321,9 @@
protocol. Not yet supported
bro_connect
- DEPRECATED (see CUSTOMIZATION)
+ DEPRECATED (see CUSTOMIZATION)
(re)connect to the Bro configured in the TM configuration file. Not
- supported.
+ supported.
show conn "tcp 1.2.3.4:80 7.8.9.1:1042"
Display information available on the specified connection in the TM's
@@ -321,27 +332,27 @@
show conn sample
Display a sample of the newest and oldest connections from the TM's
connection table (see CONNECTION TABLE above).
- NOTE/TODO: this function reads the connection table without locking.
- This might result in race conditions and in the worst case to a
- segfault. Use with care!
+ NOTE/TODO: this function reads the connection table without locking.
+ This might result in race conditions and in the worst case to a
+ segfault. Use with care!
set_dyn_class <ip> <classname> [orig|resp]
unset_dyn_class <ip>
Sets a rule for a dynamic class. Whenever a new connection with
- <ip> is seen, the class for this connection will be <classname> and
- not the class defined by the config file.
- Dynamic class rules are automatically deleted after a certain time.
- The dyn_timeout option of a class specifies, how long a dynamic
- class rule stays effective.
- If orig or resp are given, then only connection that originated from
- <ip> (in the case of orig) respectively only connections that go to <ip>
- (in the case of resp) are assigned to the dynamic class.
- If two rules (one with orig and one with resp) would match a new packet,
- the one with orig take precedence.
- If several rules for the same IP are set, the latest rule will overwrite all
- earlier rules.
- unset_dyn_class can be used to remove a dynamic class rule before the timeout
- expires.
+ <ip> is seen, the class for this connection will be <classname> and
+ not the class defined by the config file.
+ Dynamic class rules are automatically deleted after a certain time.
+ The dyn_timeout option of a class specifies, how long a dynamic
+ class rule stays effective.
+ If orig or resp are given, then only connection that originated from
+ <ip> (in the case of orig) respectively only connections that go to <ip>
+ (in the case of resp) are assigned to the dynamic class.
+ If two rules (one with orig and one with resp) would match a new packet,
+ the one with orig take precedence.
+ If several rules for the same IP are set, the latest rule will overwrite all
+ earlier rules.
+ unset_dyn_class can be used to remove a dynamic class rule before the timeout
+ expires.
Modified: trunk/Index.cc
===================================================================
--- trunk/Index.cc 2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/Index.cc 2011-07-17 17:26:21 UTC (rev 267)
@@ -65,10 +65,9 @@
*/
template <class T>
-Index<T>::Index(tm_time_t d_t, uint32_t hash_size, Storage *storage):
+Index<T>::Index(tm_time_t d_t, uint32_t hash_size, bool do_disk_index, Storage *storage):
input_q(MyQueue(500000)),
cap_thread_iat(0), idx_thread_iat(0),
- disk_index((std::string)conf_main_indexdir, "index_"+T::getIndexNameStatic()),
d_t(d_t),
last_rotated(0),
last_updated(0),
@@ -77,6 +76,10 @@
rotate_count(0){
cur = new IndexHash(hash_size);
old = new IndexHash(hash_size);
+ if (do_disk_index)
+ disk_index = new IndexFiles<T>((std::string)conf_main_indexdir, "index_"+T::getIndexNameStatic());
+ else
+ disk_index = NULL;
pthread_mutex_init(&hash_lock_mutex, NULL);
pthread_mutex_init(&queue_lock_mutex, NULL);
pthread_cond_init(&queue_cond, NULL);
@@ -103,6 +106,8 @@
// Destroy the hashes
delete cur;
delete old;
+ if (disk_index)
+ delete disk_index;
}
template <class T>
@@ -190,10 +195,15 @@
// Write the old hash to disk.
if (old->getNumEntries() != 0)
{
- tmlog(TM_LOG_NOTE, T::getIndexNameStatic().c_str(),
- "Writing %d entries to disk.", old->getNumEntries());
- // writeIndex will delete the entries from the hash
- disk_index.writeIndex(old);
+ if (disk_index) {
+ tmlog(TM_LOG_NOTE, T::getIndexNameStatic().c_str(),
+ "Writing %d entries to disk.", old->getNumEntries());
+ // writeIndex will delete the entries from the hash
+ disk_index->writeIndex(old);
+ } else {
+ // not disk writer
+ old->clear();
+ }
#ifdef TM_HEAVY_DEBUG
tmlog(TM_LOG_DEBUG, T::getIndexNameStatic().c_str(), "Qlen now is %d", input_q.size());
assert(old->getNumEntries() == 0);
@@ -292,17 +302,20 @@
template <class T>
void Index<T>::lookupDisk(IntervalSet* set, IndexField* key, tm_time_t t0, tm_time_t t1) {
- disk_index.lookup(set, key, t0, t1);
+ if (disk_index)
+ disk_index->lookup(set, key, t0, t1);
}
template <class T>
void Index<T>::aggregate() {
+ if (!disk_index)
+ return;
tm_time_t oldestTimestampDisk;
//FIXME: do we really have to lock the queue here??
lock_queue();
oldestTimestampDisk = this->idx_thread_oldestTimestampDisk;
unlock_queue();
- disk_index.aggregate(oldestTimestampDisk);
+ disk_index->aggregate(oldestTimestampDisk);
}
/* Main method of the index maintainer thread
Modified: trunk/Index.hh
===================================================================
--- trunk/Index.hh 2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/Index.hh 2011-07-17 17:26:21 UTC (rev 267)
@@ -184,6 +184,10 @@
* index entries from the packet and enqueues the keys as IndexQueueEntry
* objects */
virtual void addPkt(const pcap_pkthdr* header, const u_char* packet)=0;
+ /** Set the storage class for this IndexType.
+ * TODO: we might want to make storage a global singleton ...
+ */
+ virtual void setStorage(Storage *arg_storage)=0;
/** Aggregate / Merge disk index files into larger files */
virtual void aggregate() = 0;
virtual const std::string getIndexName()=0;
@@ -248,12 +252,15 @@
public:
// rot_offset is a (small) offset to delay the rotation and thus the writing of
// the index to disk. This should be used
- Index(tm_time_t d_t, uint32_t hash_size, Storage * storage);
+ Index(tm_time_t d_t, uint32_t hash_size, bool do_disk_index, Storage * storage);
~Index();
void cancelThread();
void lookupMem(IntervalSet* set, IndexField* key);
void lookupDisk(IntervalSet* set, IndexField* key, tm_time_t t0, tm_time_t t1);
void addPkt(const pcap_pkthdr* header, const u_char* packet);
+ void setStorage(Storage *arg_storage) {
+ storage = arg_storage;
+ }
void aggregate();
void run();
virtual const std::string getIndexName() {
@@ -295,7 +302,7 @@
tm_time_t idx_thread_oldestTimestampMem;
tm_time_t idx_thread_iat; // InterArrivalTime
- IndexFiles<T> disk_index;
+ IndexFiles<T> *disk_index;
tm_time_t d_t;
tm_time_t last_rotated;
tm_time_t last_updated; // last packet ts to be added to hash
@@ -331,6 +338,14 @@
for (std::list<IndexType*>::iterator i=begin(); i!=end(); i++)
delete (*i);
}
+ void setStorage(Storage *s) {
+ for (std::list<IndexType*>::iterator i=begin(); i!=end(); i++)
+ ( *i)->setStorage(s);
+ }
+ void startThread() {
+ for (std::list<IndexType*>::iterator i=begin(); i!=end(); i++)
+ pthread_create(&((*i)->maintainer_thread), NULL, start_index_thread, (*i));
+ }
void cancelThread() {
for (std::list<IndexType*>::iterator i=begin(); i!=end(); i++)
( *i)->cancelThread();
Modified: trunk/Storage.cc
===================================================================
--- trunk/Storage.cc 2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/Storage.cc 2011-07-17 17:26:21 UTC (rev 267)
@@ -95,9 +95,19 @@
}
+StorageConfig::StorageConfig() :
+ filter(),
+ device(),
+ readtracefile(),
+ conn_timeout(1800),
+ max_subscriptions(10000),
+ indexes(new Indexes)
+{
+ ;
+}
+
Storage::Storage(StorageConfig& conf):
snaplen(SNAPLEN),
- indexes(new Indexes),
conns(1000000),
dynclasses(25000),
tot_num_queries(0) {
@@ -109,26 +119,10 @@
conn_timeout = conf.conn_timeout;
conns.setMaxSubscriptions(conf.max_subscriptions);
-/* INDEX CONFIGRATION */
- /* comment any of the following blocks if you want to disable the
- * index */
- IndexType *idx = new Index<ConnectionIF4>(30, int(250000), this);
- pthread_create(&(idx->maintainer_thread), NULL, start_index_thread, idx);
- indexes->addIndex(idx);
-
- idx = new Index<ConnectionIF3>(30, int(250000), this);
- pthread_create(&(idx->maintainer_thread), NULL, start_index_thread, idx);
- indexes->addIndex(idx);
+ indexes = conf.indexes;
+ indexes->setStorage(this);
+ indexes->startThread();
- idx = new Index<ConnectionIF2>(30, int(250000), this);
- pthread_create(&(idx->maintainer_thread), NULL, start_index_thread, idx);
- indexes->addIndex(idx);
-
- idx = new Index<IPAddress>(30, int(250000), this);
- pthread_create(&(idx->maintainer_thread), NULL, start_index_thread, idx);
- indexes->addIndex(idx);
-/* END INDEX CONFIGRATION */
-
// Get pcap handle
errbuf[0] = '\0';
if (!conf.readtracefile.empty()) {
Modified: trunk/Storage.hh
===================================================================
--- trunk/Storage.hh 2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/Storage.hh 2011-07-17 17:26:21 UTC (rev 267)
@@ -41,6 +41,7 @@
#include <pthread.h>
#include <list>
#include <string>
+#include <tr1/unordered_map>
#include "Fifo.hh"
#include "Connections.hh"
@@ -51,28 +52,27 @@
// used in Index.hh at PktLinkList next/previous members
// #define NUM_INDEXES 6
+// TODO: We should really make the Storage class a true global singleton instead
+// of having to have a pointer to storage in a bunch of classes!
+
//#include "Index.hh"
class Indexes;
-
void *capture_thread(void *arg);
void callback(u_char *args, const struct pcap_pkthdr *header, const u_char *packet);
// Abstracts the configuration for a Storage instance
class StorageConfig {
public:
- StorageConfig() :
- filter(),
- device(),
- readtracefile(),
- conn_timeout(1800),
- max_subscriptions(10000) {};
+ StorageConfig();
+
std::string filter;
std::string device;
std::string readtracefile;
std::list<Fifo *> fifos;
tm_time_t conn_timeout;
int max_subscriptions;
+ Indexes *indexes;
};
class Storage {
Modified: trunk/conf_parser.yy
===================================================================
--- trunk/conf_parser.yy 2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/conf_parser.yy 2011-07-17 17:26:21 UTC (rev 267)
@@ -52,6 +52,7 @@
#include "Fifo.hh"
#include "Storage.hh"
+ #include "Index.hh"
// Work around a bug in the relation between bison and GCC 3.x:
#if defined (__GNUC__) && 3 <= __GNUC__
@@ -70,6 +71,7 @@
void new_class() { if (!newclass) newclass=new Fifo(); }
void end_new_class() { assert(newclass); newclass=NULL; }
+ void conf_add_index(const char* name, bool do_disk_index);
/*
IndexType* newindex=NULL;
void new_index() { if (!newindex) newindex=new IndexType(); }
@@ -108,6 +110,7 @@
%token TOK_RMTCONSOLE TOK_RMTCONSOLE_PORT TOK_RMTCONSOLE_LISTEN_ADDR
%token TOK_BRO_LISTEN TOK_BRO_LISTEN_PORT TOK_BRO_LISTEN_ADDR
%token TOK_TWEAK_CAPTURE_THREAD TOK_SCOPE TOK_PRIORITY
+%token TOK_INDEX
%type <s> classname option
%type <i64> size
@@ -324,6 +327,12 @@
| TOK_BRO_LISTEN_ADDR TOK_IPADDRESS {
conf_main_bro_listen_addr = $2;
}
+ | TOK_INDEX TOK_STRING TOK_DISK {
+ conf_add_index($2, true);
+ }
+ | TOK_INDEX TOK_STRING {
+ conf_add_index($2, false);
+ }
;
%%
@@ -350,3 +359,35 @@
return conf_parse_errors;
}
+void conf_add_index(const char* name, bool do_disk_index) {
+ if (conf_parser_storageConf->indexes->getIndexByName(name)) {
+ char msg[2048];
+ snprintf(msg, sizeof(msg), "Index %s already configured\n", name);
+ conferror(msg);
+ return;
+ }
+ /* TODO: We really should do index configuration with a regitry that knows
+ of all potential IndexTypes .... */
+ if (ConnectionIF4::getIndexNameStatic() == name) {
+ IndexType *idx = new Index<ConnectionIF4>(30, int(250000), do_disk_index, NULL);
+ conf_parser_storageConf->indexes->addIndex(idx);
+ }
+ else if (ConnectionIF3::getIndexNameStatic() == name) {
+ IndexType *idx = new Index<ConnectionIF3>(30, int(250000), do_disk_index, NULL);
+ conf_parser_storageConf->indexes->addIndex(idx);
+ }
+
+ else if (ConnectionIF2::getIndexNameStatic() == name) {
+ IndexType *idx = new Index<ConnectionIF2>(30, int(250000), do_disk_index, NULL);
+ conf_parser_storageConf->indexes->addIndex(idx);
+ }
+ else if (IPAddress::getIndexNameStatic() == name) {
+ IndexType *idx = new Index<IPAddress>(30, int(250000), do_disk_index, NULL);
+ conf_parser_storageConf->indexes->addIndex(idx);
+ }
+ else {
+ char msg[2048];
+ snprintf(msg, sizeof(msg), "Don't know about index %s\n", name);
+ conferror(msg);
+ }
+}
Modified: trunk/conf_scanner.ll
===================================================================
--- trunk/conf_scanner.ll 2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/conf_scanner.ll 2011-07-17 17:26:21 UTC (rev 267)
@@ -98,6 +98,7 @@
"workdir" return TOK_WORKDIR;
"queryfiledir" return TOK_QUERYFILEDIR;
"indexdir" return TOK_INDEXDIR;
+"index" return TOK_INDEX;
"logfile" return TOK_LOGFILE;
"bro_connect_str" return TOK_BRO_CONNECT_STR;
"pkts_to_disk" return TOK_PKTS_TO_DISK;
Modified: trunk/config.h.in
===================================================================
--- trunk/config.h.in 2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/config.h.in 2011-07-17 17:26:21 UTC (rev 267)
@@ -81,6 +81,9 @@
/* Define to the one symbol short name of this package. */
#undef PACKAGE_TARNAME
+/* Define to the home page for this package. */
+#undef PACKAGE_URL
+
/* Define to the version of this package. */
#undef PACKAGE_VERSION
Modified: trunk/configure.in
===================================================================
--- trunk/configure.in 2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/configure.in 2011-07-17 17:26:21 UTC (rev 267)
@@ -17,8 +17,8 @@
AM_PROG_CC_C_O
#AC_PROG_CXXCPP
-OUR_CFLAGS="-O2 -g -Wall -I/usr/local/include"
-LDFLAGS="${LDFLAGS} -L/usr/local/lib"
+OUR_CFLAGS="-O2 -g -Wall -I/usr/local/include -I/opt/local/include"
+LDFLAGS="${LDFLAGS} -L/usr/local/lib -L/opt/local/lib"
CXXFLAGS="${CXXFLAGS} ${OUR_CFLAGS}"
CFLAGS="${CXXFLAGS} ${OUR_CFLAGS}"
Modified: trunk/tm.conf
===================================================================
--- trunk/tm.conf 2011-07-16 16:17:30 UTC (rev 266)
+++ trunk/tm.conf 2011-07-17 17:26:21 UTC (rev 267)
@@ -22,6 +22,13 @@
bro_listen 1;
bro_listen_port 47757; # 47757 is default
bro_listen_addr 127.0.0.1; # 127.0.0.1 is default
+
+ # Index configuration.
+ # Default is to enable all indexes w/ disk index
+ index "connection4" disk;
+ index "connection3" disk;
+ index "connection2" disk;
+ index "ip" disk;
daemon 0;
# see Documentation/TUNING !!!
More information about the Time-Machine
mailing list