From seth at icir.org Thu Jan 24 12:37:38 2013 From: seth at icir.org (Seth Hall) Date: Thu, 24 Jan 2013 12:37:38 -0800 Subject: [TM] [git] branch 'topic/seth/cmake-changeover' created Message-ID: <201301242037.r0OKbc7O009144@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine New branch : topic/seth/cmake-changeover Referencing: 6f90b9f0f1aebd60b8f1bea8f01b79730156f3e2 From seth at icir.org Thu Jan 24 12:37:40 2013 From: seth at icir.org (Seth Hall) Date: Thu, 24 Jan 2013 12:37:40 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: First step toward clean up. (29608cd) Message-ID: <201301242037.r0OKbe6V009155@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit 29608cd20f1be3a84c80de0db63e53e63a9fa1b3 Author: Seth Hall Date: Wed Jan 23 09:59:39 2013 -0500 First step toward clean up. >--------------------------------------------------------------- 29608cd20f1be3a84c80de0db63e53e63a9fa1b3 COPYING | 5 +- {Documentation => doc}/INSTALL | 0 {Documentation => doc}/TM_HOWTO | 0 {Documentation => doc}/TUNING | 0 {Documentation => doc}/imc2005-short-paper.pdf | Bin {Documentation => doc}/sigcomm2008-paper.pdf | Bin {Documentation => doc}/tm-architecture.txt | 0 human_time.pl | 9 - ptmalloc2/COPYRIGHT | 30 - ptmalloc2/ChangeLog | 186 - ptmalloc2/README | 198 - ptmalloc2/arena.c | 800 ---- ptmalloc2/hooks.c | 640 --- ptmalloc2/malloc-stats.c | 161 - ptmalloc2/malloc.c | 5439 ------------------------ ptmalloc2/malloc.h | 291 -- ptmalloc2/sysdeps/generic/atomic.h | 1 - ptmalloc2/sysdeps/generic/malloc-machine.h | 68 - ptmalloc2/sysdeps/generic/thread-st.h | 48 - ptmalloc2/sysdeps/pthread/malloc-machine.h | 131 - ptmalloc2/sysdeps/pthread/thread-st.h | 111 - BroccoliComm.cc => src/BroccoliComm.cc | 0 BroccoliComm.hh => src/BroccoliComm.hh | 0 Connection.cc => src/Connection.cc | 0 Connection.hh => src/Connection.hh | 0 Connections.cc => src/Connections.cc | 0 Connections.hh => src/Connections.hh | 0 DiskIndex.cc => src/DiskIndex.cc | 0 DiskIndex.hh => src/DiskIndex.hh | 0 DynClass.cc => src/DynClass.cc | 0 DynClass.hh => src/DynClass.hh | 0 Fifo.cc => src/Fifo.cc | 0 Fifo.hh => src/Fifo.hh | 0 FifoDisk.cc => src/FifoDisk.cc | 0 FifoDisk.hh => src/FifoDisk.hh | 0 FifoMem.cc => src/FifoMem.cc | 0 FifoMem.hh => src/FifoMem.hh | 0 Hash.cc => src/Hash.cc | 0 Hash.hh => src/Hash.hh | 0 Index.cc => src/Index.cc | 0 Index.hh => src/Index.hh | 0 IndexEntry.hh => src/IndexEntry.hh | 0 IndexField.cc => src/IndexField.cc | 0 IndexField.hh => src/IndexField.hh | 0 IndexHash.cc => src/IndexHash.cc | 0 IndexHash.hh => src/IndexHash.hh | 0 LogFile.cc => src/LogFile.cc | 0 LogFile.hh => src/LogFile.hh | 0 Query.cc => src/Query.cc | 0 Query.hh => src/Query.hh | 0 Queue.cc => src/Queue.cc | 0 Queue.hh => src/Queue.hh | 0 Storage.cc => src/Storage.cc | 0 Storage.hh => src/Storage.hh | 0 broSendPkts.cc => src/broSendPkts.cc | 0 bropipe.cc => src/bropipe.cc | 0 client.cc => src/client.cc | 0 cmd_parser.yy => src/cmd_parser.yy | 0 cmd_scanner.ll => src/cmd_scanner.ll | 0 conf.h => src/conf.h | 0 conf_parser.yy => src/conf_parser.yy | 0 conf_scanner.ll => src/conf_scanner.ll | 0 jhash3.h => src/jhash3.h | 0 main.cc => src/main.cc | 0 packet_headers.h => src/packet_headers.h | 0 querybm.cc => src/querybm.cc | 0 rmtconsole.cc => src/rmtconsole.cc | 0 tm.h => src/tm.h | 0 tm_q.cc => src/tm_q.cc | 0 types.h => src/types.h | 0 util.cc => src/util.cc | 0 util.h => src/util.h | 0 ylwrap | 222 - 73 files changed, 1 insertion(+), 8339 deletions(-) Diff suppressed because of size. To see it, use: git diff-tree --root --patch-with-stat --no-color --find-copies-harder --ignore-space-at-eol --cc 29608cd20f1be3a84c80de0db63e53e63a9fa1b3 From seth at icir.org Thu Jan 24 12:37:42 2013 From: seth at icir.org (Seth Hall) Date: Thu, 24 Jan 2013 12:37:42 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Added the cmake submodule. (a0623cd) Message-ID: <201301242037.r0OKbge0009164@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit a0623cd544549845feae007f2108a2cfcdbc27c5 Author: Seth Hall Date: Wed Jan 23 10:02:23 2013 -0500 Added the cmake submodule. >--------------------------------------------------------------- a0623cd544549845feae007f2108a2cfcdbc27c5 .gitmodules | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 0000000..4d3f285 --- /dev/null +++ b/.gitmodules @@ -0,0 +1,4 @@ +[submodule "cmake"] + path = cmake + url = git://git.bro-ids.org/cmake + From seth at icir.org Thu Jan 24 12:37:44 2013 From: seth at icir.org (Seth Hall) Date: Thu, 24 Jan 2013 12:37:44 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Updated cmake (6b01f7b) Message-ID: <201301242037.r0OKbiTE009173@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit 6b01f7b0f30cada59461920fad31b59dc061a7ef Author: Seth Hall Date: Wed Jan 23 10:04:40 2013 -0500 Updated cmake >--------------------------------------------------------------- 6b01f7b0f30cada59461920fad31b59dc061a7ef cmake | 1 + 1 file changed, 1 insertion(+) diff --git a/cmake b/cmake new file mode 160000 index 0000000..14537f5 --- /dev/null +++ b/cmake @@ -0,0 +1 @@ +Subproject commit 14537f56d66b18ab9d5024f798caf4d1f356fc67 From seth at icir.org Thu Jan 24 12:37:46 2013 From: seth at icir.org (Seth Hall) Date: Thu, 24 Jan 2013 12:37:46 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Checkpoint (d8fbaff) Message-ID: <201301242037.r0OKbkxB009182@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit d8fbaff20d75b8c03f5b041be624ab36446a91be Author: Seth Hall Date: Wed Jan 23 10:26:06 2013 -0500 Checkpoint >--------------------------------------------------------------- d8fbaff20d75b8c03f5b041be624ab36446a91be AUTHORS | 12 ------------ COPYING | 1 - autogen.sh | 34 ---------------------------------- tm.conf => etc/tm.conf | 0 4 files changed, 47 deletions(-) diff --git a/AUTHORS b/AUTHORS index a86de94..4184a6f 100644 --- a/AUTHORS +++ b/AUTHORS @@ -1,10 +1,3 @@ -Timemachine -(c) Technische Universitaet Muenchen -(c) Technische Universitaet Berlin -(c) The Regents of the University of California - -Contact: tm at lists.net.t-labs.tu-berlin.de - Written by: Stefan Kornexl Gregor Maier @@ -15,8 +8,3 @@ Anja Feldmann Vern Paxson Robin Sommer Fabian Schneider - - - -Hash Function (c) by Bob Jenkins, May 2006, Public Domain. -ptmalloc (c) by Wolfram Gloger (see ptmalloc2/COPYRIGHT for details) diff --git a/COPYING b/COPYING index b41760b..f3e6b8b 100644 --- a/COPYING +++ b/COPYING @@ -6,7 +6,6 @@ All rights reserved. Contact: tm at lists.net.t-labs.tu-berlin.de - Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: diff --git a/autogen.sh b/autogen.sh deleted file mode 100755 index ce2399c..0000000 --- a/autogen.sh +++ /dev/null @@ -1,34 +0,0 @@ - -# $Id: autogen.sh 161 2006-12-19 02:35:21Z gregor $ - -aclocal && autoheader && autoconf && automake --add-missing --copy - -#if which aclocal-1.9 > /dev/null 2>/dev/null; then -# ACLOCAL=aclocal-1.9 -#elif which aclocal19 > /dev/null 2>/dev/null; then -# ACLOCAL=aclocal19 -#else -# echo "Could not found aclocal-1.9 or aclocal19. Exiting" -# exit -#fi -# -#if which automake-1.9 > /dev/null 2>/dev/null; then -# AM=automake-1.9 -#elif which automake19 > /dev/null 2>/dev/null ; then -# AM=automake19 -#else -# echo "Could not found automake-1.9 or automake19. Exiting" -# exit -#fi -# - -#aclocal-1.9 && autoheader && autoconf && automake-1.9 --add-missing --copy - -# autoheader depends on aclocal -# autoconf depends on aclocal -# configure depends on everything -# no other dependencies -#${ACLOCAL} \ -# && autoheader \ -# && autoconf \ -# && ${AM} --add-missing --copy diff --git a/tm.conf b/etc/tm.conf similarity index 100% rename from tm.conf rename to etc/tm.conf From seth at icir.org Thu Jan 24 12:37:48 2013 From: seth at icir.org (Seth Hall) Date: Thu, 24 Jan 2013 12:37:48 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Checkpoint (96385c1) Message-ID: <201301242037.r0OKbmZA009191@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit 96385c1d02d6514a467a898ede255582656d6e78 Author: Seth Hall Date: Wed Jan 23 10:52:20 2013 -0500 Checkpoint >--------------------------------------------------------------- 96385c1d02d6514a467a898ede255582656d6e78 CMakeLists.txt | 70 ++++++++++++++++++++++++++++++ VERSION | 1 + configure | 123 +++++++++++++++++++++++++++++++++++++++++++++++++++++ src/CMakeLists.txt | 85 ++++++++++++++++++++++++++++++++++++ 4 files changed, 279 insertions(+) diff --git a/CMakeLists.txt b/CMakeLists.txt new file mode 100644 index 0000000..d2b3d82 --- /dev/null +++ b/CMakeLists.txt @@ -0,0 +1,70 @@ +project(TimeMachine C CXX) +cmake_minimum_required(VERSION 2.6.3 FATAL_ERROR) +include(cmake/CommonCMakeConfig.cmake) + +file(STRINGS "${CMAKE_CURRENT_SOURCE_DIR}/VERSION" TIMEMACHINE_VERSION LIMIT_COUNT 1) + +######################################################################## +## Dependency Configuration + +include(FindRequiredPackage) + +FindRequiredPackage(FLEX) +FindRequiredPackage(BISON) +FindRequiredPackage(PCAP) +FindRequiredPackage(Broccoli) + +set(USE_PCAPNAV false) +#find_package(PCAPNav) + +if (MISSING_PREREQS) + foreach (prereq ${MISSING_PREREQ_DESCS}) + message(SEND_ERROR ${prereq}) + endforeach () + message(FATAL_ERROR "Configuration aborted due to missing prerequisites") +endif () + +######################################################################## +## System Introspection + +configure_file(${CMAKE_CURRENT_SOURCE_DIR}/config.h.in + ${CMAKE_CURRENT_BINARY_DIR}/config.h) + +include_directories(BEFORE ${CMAKE_CURRENT_BINARY_DIR}) + +######################################################################## +## Recurse on sub-directories + +add_subdirectory(src) + +######################################################################## +## Build Summary + +if (TimeMachine_SKIP_INSTALL) + set(timemachine_install_summary "Install skipped") +else () + set(timemachine_install_summary "${CMAKE_INSTALL_PREFIX}") +endif () + +if (CMAKE_BUILD_TYPE) + string(TOUPPER ${CMAKE_BUILD_TYPE} BuildType) +endif () + +message( + "\n==================| TimeMachine Build Summary |====================" + "\n" + "\nInstall prefix: ${timemachine_install_summary}" + "\nDebug mode: ${ENABLE_DEBUG}" + "\n" + "\nCC: ${CMAKE_C_COMPILER}" + "\nCFLAGS: ${CMAKE_C_FLAGS} ${CMAKE_C_FLAGS_${BuildType}}" + "\nCXX: ${CMAKE_CXX_COMPILER}" + "\nCXXFLAGS: ${CMAKE_CXX_FLAGS} ${CMAKE_CXX_FLAGS_${BuildType}}" + "\nCPP: ${CMAKE_CXX_COMPILER}" + "\n" + "\nBroccoli: ${HAVE_BROCCOLI}" + "\n" + "\n================================================================\n" +) + +include(UserChangedWarning) diff --git a/VERSION b/VERSION new file mode 100644 index 0000000..ceab6e1 --- /dev/null +++ b/VERSION @@ -0,0 +1 @@ +0.1 \ No newline at end of file diff --git a/configure b/configure new file mode 100755 index 0000000..8c27e62 --- /dev/null +++ b/configure @@ -0,0 +1,123 @@ +#!/bin/sh +# Convenience wrapper for easily viewing/setting options that +# the project's CMake scripts will recognize +set -e +command="$0 $*" + +# check for `cmake` command +type cmake > /dev/null 2>&1 || { + echo "\ +This package requires CMake, please install it first, then you may +use this configure script to access CMake equivalent functionality.\ +" >&2; + exit 1; +} + +usage="\ +Usage: $0 [OPTION]... [VAR=VALUE]... + + Build Directory: + --builddir=DIR place build files in directory [build] + + Installation Directories: + --prefix=PREFIX installation directory [/usr/local] + + Optional Features: + --enable-debug compile in debugging mode + + Required Packages in Non-Standard Locations: + --with-broccoli=PATH path to libbroccoli install root + --with-pcap=PATH path to libpcap install root + --with-pcapnav=PATH path to libpcapnav install root + --with-flex=PATH path to flex executable + --with-bison=PATH path to bison executable + + Influential Environment Variables (only on first invocation + per build directory): + CC C compiler command + CFLAGS C compiler flags + CXX C++ compiler command + CXXFLAGS C++ compiler flags +" + +sourcedir="$( cd "$( dirname "$0" )" && pwd )" + +# Function to append a CMake cache entry definition to the +# CMakeCacheEntries variable +# $1 is the cache entry variable name +# $2 is the cache entry variable type +# $3 is the cache entry variable value +append_cache_entry () { + CMakeCacheEntries="$CMakeCacheEntries -D $1:$2=$3" +} + +# set defaults +builddir=build +CMakeCacheEntries="" +append_cache_entry CMAKE_INSTALL_PREFIX PATH /usr/local +append_cache_entry ENABLE_DEBUG BOOL false + +# parse arguments +while [ $# -ne 0 ]; do + case "$1" in + -*=*) optarg=`echo "$1" | sed 's/[-_a-zA-Z0-9]*=//'` ;; + *) optarg= ;; + esac + + case "$1" in + --help|-h) + echo "${usage}" 1>&2 + exit 1 + ;; + --builddir=*) + builddir=$optarg + ;; + --prefix=*) + append_cache_entry CMAKE_INSTALL_PREFIX PATH $optarg + ;; + --enable-debug) + append_cache_entry ENABLE_DEBUG BOOL true + ;; + --with-broccoli=*) + append_cache_entry Broccoli_ROOT_DIR PATH $optarg + ;; + --with-pcapnav=*) + append_cache_entry PCAPNAV_ROOT_DIR PATH $optarg + ;; + --with-pcap=*) + append_cache_entry PCAP_ROOT_DIR PATH $optarg + ;; + --with-flex=*) + append_cache_entry FLEX_EXECUTABLE PATH $optarg + ;; + --with-bison=*) + append_cache_entry BISON_EXECUTABLE PATH $optarg + ;; + *) + echo "Invalid option '$1'. Try $0 --help to see available options." + exit 1 + ;; + esac + shift +done + +if [ -d $builddir ]; then + # If build directory exists, check if it has a CMake cache + if [ -f $builddir/CMakeCache.txt ]; then + # If the CMake cache exists, delete it so that this configuration + # is not tainted by a previous one + rm -f $builddir/CMakeCache.txt + fi +else + # Create build directory + mkdir -p $builddir +fi + +echo "Build Directory : $builddir" +echo "Source Directory: $sourcedir" +cd $builddir +cmake $CMakeCacheEntries $sourcedir + +echo "# This is the command used to configure this build" > config.status +echo $command >> config.status +chmod u+x config.status diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt new file mode 100644 index 0000000..40ec4f7 --- /dev/null +++ b/src/CMakeLists.txt @@ -0,0 +1,85 @@ +bison_target(CMDParser cmd_parser.yy ${CMAKE_CURRENT_BINARY_DIR}/cmd_parser.cc + HEADER ${CMAKE_CURRENT_BINARY_DIR}/cmd_parser.h + VERBOSE ${CMAKE_CURRENT_BINARY_DIR}/cmd_parser.output + COMPILE_FLAGS "--debug") +flex_target(CMDScanner cmd_scanner.ll ${CMAKE_CURRENT_BINARY_DIR}/cmd_scanner.cc) +add_flex_bison_dependency(CMDScanner CMDParser) + +bison_target(CONFParser conf_parser.yy ${CMAKE_CURRENT_BINARY_DIR}/conf_parser.cc + HEADER ${CMAKE_CURRENT_BINARY_DIR}/conf_parser.h + VERBOSE ${CMAKE_CURRENT_BINARY_DIR}/conf_parser.output + COMPILE_FLAGS "--debug") +flex_target(CMDScanner conf_scanner.ll ${CMAKE_CURRENT_BINARY_DIR}/conf_scanner.cc) +add_flex_bison_dependency(CONFScanner CONFParser) + +include_directories(${CMAKE_CURRENT_SOURCE_DIR} + ${CMAKE_CURRENT_BINARY_DIR}) + +set(timemachine_SRCS + ${BISON_CMDParser_INPUT} + ${FLEX_CMDScanner_INPUT} + ${BISON_CMDParser_OUTPUTS} + ${FLEX_CMDScanner_OUTPUTS} + ${BISON_CONFParser_INPUT} + ${FLEX_CONFScanner_INPUT} + ${BISON_CONFParser_OUTPUTS} + ${FLEX_CONFScanner_OUTPUTS} + BroccoliComm.cc + BroccoliComm.hh + Connection.cc + Connection.hh + Connections.cc + Connections.hh + DiskIndex.cc + DiskIndex.hh + DynClass.cc + DynClass.hh + Fifo.cc + Fifo.hh + FifoDisk.cc + FifoDisk.hh + FifoMem.cc + FifoMem.hh + Hash.cc + Hash.hh + Index.cc + Index.hh + IndexEntry.hh + IndexField.cc + IndexField.hh + IndexHash.cc + IndexHash.hh + LogFile.cc + LogFile.hh + Query.cc + Query.hh + Queue.cc + Queue.hh + Storage.cc + Storage.hh + broSendPkts.cc + bropipe.cc + client.cc + conf.h + jhash3.h + main.cc + packet_headers.h + querybm.cc + rmtconsole.cc + tm.h + tm_q.cc + types.h + util.cc + util.h +) + +add_executable(timemachine ${timemachine_SRCS}) + +if (NOT TimeMachine_SKIP_INSTALL) + install(TARGETS timemachine DESTINATION bin) +endif () + +# This is set to assist superprojects that want to build BinPac +# from source and rely on it as a target +set(TimeMachine_EXE timemachine + CACHE STRING "TimeMachine executable" FORCE) From seth at icir.org Thu Jan 24 12:37:51 2013 From: seth at icir.org (Seth Hall) Date: Thu, 24 Jan 2013 12:37:51 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Clean up checkpoint. (7254a7f) Message-ID: <201301242037.r0OKbpPV009201@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit 7254a7f6cd9f3acd610f2858e22c34cb3294e678 Author: Seth Hall Date: Wed Jan 23 22:15:35 2013 -0500 Clean up checkpoint. - Removed PCRE dependency, added RE2 to the repository and ported the few PCRE calls to RE2 style. - Began removing copyright notices from all source files. - A little clean up to the example config. >--------------------------------------------------------------- 7254a7f6cd9f3acd610f2858e22c34cb3294e678 CMakeLists.txt | 1 + config.h.in | 75 + etc/tm.conf | 33 +- src/BroccoliComm.cc | 2 +- src/BroccoliComm.hh | 2 +- src/CMakeLists.txt | 37 +- src/Connection.cc | 41 +- src/Connection.hh | 40 +- src/DiskIndex.cc | 44 - src/DiskIndex.hh | 36 - src/FifoDisk.cc | 42 +- src/Index.cc | 38 - src/IndexField.cc | 56 +- src/IndexField.hh | 48 +- src/Query.cc | 6 +- src/Query.hh | 4 +- src/cmd_parser.yy | 48 +- src/cmd_scanner.ll | 42 - src/conf_scanner.ll | 44 - src/main.cc | 50 +- src/re2/CMakeLists.txt | 77 + src/re2/Makefile | 1 + src/re2/bitstate.cc | 378 +++ src/re2/compile.cc | 1139 +++++++++ src/re2/dfa.cc | 2132 ++++++++++++++++ src/re2/filtered_re2.cc | 102 + src/re2/filtered_re2.h | 101 + src/re2/make_perl_groups.pl | 110 + src/re2/make_unicode_casefold.py | 146 ++ src/re2/make_unicode_groups.py | 111 + src/re2/mimics_pcre.cc | 185 ++ src/re2/nfa.cc | 709 ++++++ src/re2/onepass.cc | 614 +++++ src/re2/parse.cc | 2214 ++++++++++++++++ src/re2/perl_groups.cc | 119 + src/re2/prefilter.cc | 715 ++++++ src/re2/prefilter.h | 105 + src/re2/prefilter_tree.cc | 398 +++ src/re2/prefilter_tree.h | 130 + src/re2/prog.cc | 341 +++ src/re2/prog.h | 376 +++ src/re2/re2.cc | 1209 +++++++++ src/re2/re2.h | 855 +++++++ src/re2/regexp.cc | 931 +++++++ src/re2/regexp.h | 633 +++++ src/re2/set.cc | 113 + src/re2/set.h | 55 + src/re2/simplify.cc | 393 +++ src/re2/stringpiece.h | 182 ++ src/re2/tostring.cc | 341 +++ src/re2/unicode.py | 297 +++ src/re2/unicode_casefold.cc | 469 ++++ src/re2/unicode_casefold.h | 75 + src/re2/unicode_groups.cc | 4851 ++++++++++++++++++++++++++++++++++++ src/re2/unicode_groups.h | 64 + src/re2/util/arena.cc | 168 ++ src/re2/util/arena.h | 103 + src/re2/util/atomicops.h | 79 + src/re2/util/benchmark.cc | 153 ++ src/re2/util/benchmark.h | 41 + src/re2/util/flags.h | 27 + src/{jhash3.h => re2/util/hash.cc} | 136 +- src/re2/util/logging.h | 86 + src/re2/util/mutex.h | 211 ++ src/re2/util/pcre.cc | 961 +++++++ src/re2/util/pcre.h | 679 +++++ src/re2/util/random.cc | 34 + src/re2/util/random.h | 29 + src/re2/util/rune.cc | 258 ++ src/re2/util/sparse_array.h | 453 ++++ src/re2/util/sparse_array_test.cc | 150 ++ src/re2/util/sparse_set.h | 179 ++ src/re2/util/stringpiece.cc | 87 + src/re2/util/stringprintf.cc | 78 + src/re2/util/strutil.cc | 97 + src/re2/util/test.cc | 39 + src/re2/util/test.h | 57 + src/re2/util/thread.cc | 44 + src/re2/util/thread.h | 26 + src/re2/util/utf.h | 43 + src/re2/util/util.h | 131 + src/re2/util/valgrind.cc | 18 + src/re2/util/valgrind.h | 4517 +++++++++++++++++++++++++++++++++ src/re2/variadic_function.h | 346 +++ src/re2/walker-inl.h | 244 ++ src/tm.h | 35 - src/types.h | 36 - 87 files changed, 30262 insertions(+), 643 deletions(-) Diff suppressed because of size. To see it, use: git diff-tree --root --patch-with-stat --no-color --find-copies-harder --ignore-space-at-eol --cc 7254a7f6cd9f3acd610f2858e22c34cb3294e678 From seth at icir.org Thu Jan 24 12:37:53 2013 From: seth at icir.org (Seth Hall) Date: Thu, 24 Jan 2013 12:37:53 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Added a .gitignore file to stop seeing build/ (1b300e2) Message-ID: <201301242037.r0OKbr8I009210@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit 1b300e27258302a289f36d220ba013b2f1a2d7a4 Author: Seth Hall Date: Wed Jan 23 22:17:53 2013 -0500 Added a .gitignore file to stop seeing build/ >--------------------------------------------------------------- 1b300e27258302a289f36d220ba013b2f1a2d7a4 .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..378eac2 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +build From seth at icir.org Thu Jan 24 12:37:55 2013 From: seth at icir.org (Seth Hall) Date: Thu, 24 Jan 2013 12:37:55 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Removing files. (a5c59e8) Message-ID: <201301242037.r0OKbtYJ009220@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit a5c59e86678c437487349f24301b29707010d70a Author: Seth Hall Date: Wed Jan 23 23:23:17 2013 -0500 Removing files. >--------------------------------------------------------------- a5c59e86678c437487349f24301b29707010d70a Makefile.am | 47 -- configure.in | 148 ------ depcomp | 522 --------------------- .../tm-synflood-discussion.txt | 0 filter.sh | 6 - filter_class.sh | 37 -- install-sh | 323 ------------- missing | 353 -------------- show_classes.sh | 11 - size-account.sh | 10 - testcli.c | 292 ------------ tm.sh | 5 - tm_q_files.pl | 70 --- 13 files changed, 1824 deletions(-) Diff suppressed because of size. To see it, use: git diff-tree --root --patch-with-stat --no-color --find-copies-harder --ignore-space-at-eol --cc a5c59e86678c437487349f24301b29707010d70a From seth at icir.org Thu Jan 24 12:37:57 2013 From: seth at icir.org (Seth Hall) Date: Thu, 24 Jan 2013 12:37:57 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Checkpoint. (b9fcabc) Message-ID: <201301242037.r0OKbvVt009230@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit b9fcabced1e171ff2543f9e51756d2f7deba1d50 Author: Seth Hall Date: Thu Jan 24 13:40:12 2013 -0500 Checkpoint. - Lots of small updates to build system and documentation. - Integrated libpcapnav into the build tree since it's not typically readily davailable as a package for many operating systems. Signed-off-by: Seth Hall >--------------------------------------------------------------- b9fcabced1e171ff2543f9e51756d2f7deba1d50 CMakeLists.txt | 17 +- COPYING | 5 +- INSTALL | 2 +- Makefile | 63 ++ README | 25 +- config.h.in | 68 +- doc/{TM_HOWTO => HOWTO} | 113 ++-- doc/INSTALL | 50 +- doc/TUNING | 171 ----- TODO => doc/papers-and-notes/TODO | 0 doc/{ => papers-and-notes}/imc2005-short-paper.pdf | Bin doc/{ => papers-and-notes}/sigcomm2008-paper.pdf | Bin doc/{ => papers-and-notes}/tm-architecture.txt | 0 .../tm-synflood-discussion.txt | 0 etc/timemachine.conf | 64 ++ etc/tm.conf | 49 -- src/BroccoliComm.cc | 146 ++--- src/BroccoliComm.hh | 38 +- src/CMakeLists.txt | 36 +- src/FifoDisk.cc | 41 +- src/Query.cc | 6 +- src/Query.hh | 45 +- src/broSendPkts.cc | 137 ---- src/bropipe.cc | 393 ------------ src/client.cc | 2 - src/cmd_parser.yy | 18 +- src/main.cc | 39 +- src/pcapnav/CMakeLists.txt | 32 + src/pcapnav/pcapnav.c | 713 +++++++++++++++++++++ src/pcapnav/pcapnav.h | 519 +++++++++++++++ src/pcapnav/pcapnav_append.c | 237 +++++++ src/pcapnav/pcapnav_append.h | 33 + src/pcapnav/pcapnav_buf.c | 220 +++++++ src/pcapnav/pcapnav_buf.h | 182 ++++++ src/pcapnav/pcapnav_debug.c | 85 +++ src/pcapnav/pcapnav_debug.h | 109 ++++ src/pcapnav/pcapnav_globals.c | 31 + src/pcapnav/pcapnav_header.c | 378 +++++++++++ src/pcapnav/pcapnav_header.h | 99 +++ src/pcapnav/pcapnav_macros.h | 104 +++ src/pcapnav/pcapnav_private.h | 74 +++ src/pcapnav/pcapnav_timestamp.h | 30 + src/pcapnav/pcapnav_trace.c | 536 ++++++++++++++++ src/pcapnav/pcapnav_trace.h | 128 ++++ src/pcapnav/pcapnav_util.c | 110 ++++ src/pcapnav/pcapnav_util.h | 83 +++ src/tm_q.cc | 37 +- src/{querybm.cc => tm_query_benchmark.cc} | 41 +- tm-query/README | 2 +- 49 files changed, 4021 insertions(+), 1290 deletions(-) Diff suppressed because of size. To see it, use: git diff-tree --root --patch-with-stat --no-color --find-copies-harder --ignore-space-at-eol --cc b9fcabced1e171ff2543f9e51756d2f7deba1d50 From seth at icir.org Thu Jan 24 12:37:59 2013 From: seth at icir.org (Seth Hall) Date: Thu, 24 Jan 2013 12:37:59 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: More documentation cleanup and removal of in-file right notices. (6f90b9f) Message-ID: <201301242037.r0OKbx97009240@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit 6f90b9f0f1aebd60b8f1bea8f01b79730156f3e2 Author: Seth Hall Date: Thu Jan 24 15:37:36 2013 -0500 More documentation cleanup and removal of in-file right notices. >--------------------------------------------------------------- 6f90b9f0f1aebd60b8f1bea8f01b79730156f3e2 doc/HOWTO | 357 ------------------------------- doc/INSTALL | 18 -- doc/howto.rst | 347 ++++++++++++++++++++++++++++++ doc/install.rst | 31 +++ doc/papers-and-notes/tm-architecture.txt | 8 +- etc/timemachine.conf | 38 ++-- src/CMakeLists.txt | 2 - src/Connections.cc | 34 --- src/Connections.hh | 36 ---- src/DynClass.cc | 36 ---- src/DynClass.hh | 35 --- src/Fifo.cc | 35 --- src/Fifo.hh | 36 ---- src/FifoDisk.hh | 36 ---- src/FifoMem.cc | 35 --- src/FifoMem.hh | 36 ---- src/Hash.cc | 35 --- src/Hash.hh | 36 ---- src/Index.hh | 36 ---- src/IndexEntry.hh | 36 ---- src/IndexHash.cc | 37 ---- src/IndexHash.hh | 36 ---- src/LogFile.cc | 36 ---- src/LogFile.hh | 36 ---- src/Query.cc | 35 --- src/Queue.cc | 36 ---- src/Queue.hh | 36 ---- src/Storage.cc | 38 ---- src/Storage.hh | 36 ---- src/cmd_parser.yy | 3 +- src/conf.h | 36 ---- src/conf_parser.yy | 34 --- src/jhash3.h | 2 - src/packet_headers.h | 36 ---- src/rmtconsole.cc | 38 ---- src/util.cc | 2 - src/util.h | 9 - 37 files changed, 398 insertions(+), 1351 deletions(-) Diff suppressed because of size. To see it, use: git diff-tree --root --patch-with-stat --no-color --find-copies-harder --ignore-space-at-eol --cc 6f90b9f0f1aebd60b8f1bea8f01b79730156f3e2 From seth at icir.org Thu Jan 24 13:08:11 2013 From: seth at icir.org (Seth Hall) Date: Thu, 24 Jan 2013 13:08:11 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Changing the cmake branch to get the readline module finder. (1fe58cf) Message-ID: <201301242108.r0OL8B8L013296@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit 1fe58cf97fb2e0103eb03a4cd1df8b3202309118 Author: Seth Hall Date: Thu Jan 24 16:07:51 2013 -0500 Changing the cmake branch to get the readline module finder. >--------------------------------------------------------------- 1fe58cf97fb2e0103eb03a4cd1df8b3202309118 cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake b/cmake index 14537f5..599c3fa 160000 --- a/cmake +++ b/cmake @@ -1 +1 @@ -Subproject commit 14537f56d66b18ab9d5024f798caf4d1f356fc67 +Subproject commit 599c3fa916a8d746a535666955ab4439bea20471 From seth at icir.org Thu Jan 24 13:08:13 2013 From: seth at icir.org (Seth Hall) Date: Thu, 24 Jan 2013 13:08:13 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Very minor documentation updates. (10df3c5) Message-ID: <201301242108.r0OL8D7J013305@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit 10df3c509d527c509d006d01dfc3474514a5e6c7 Author: Seth Hall Date: Thu Jan 24 16:08:15 2013 -0500 Very minor documentation updates. >--------------------------------------------------------------- 10df3c509d527c509d006d01dfc3474514a5e6c7 AUTHORS | 14 +++++++------- COPYING | 2 ++ INSTALL | 2 +- src/cmd_parser.yy | 2 +- 4 files changed, 11 insertions(+), 9 deletions(-) diff --git a/AUTHORS b/AUTHORS index 4184a6f..e5fa57b 100644 --- a/AUTHORS +++ b/AUTHORS @@ -1,10 +1,10 @@ Written by: -Stefan Kornexl -Gregor Maier + Stefan Kornexl + Gregor Maier Design and contributions: -Holger Dreger -Anja Feldmann -Vern Paxson -Robin Sommer -Fabian Schneider + Holger Dreger + Anja Feldmann + Vern Paxson + Robin Sommer + Fabian Schneider diff --git a/COPYING b/COPYING index 31af25d..75fcf6a 100644 --- a/COPYING +++ b/COPYING @@ -36,3 +36,5 @@ ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Hash Function (c) by Bob Jenkins, May 2006, Public Domain. +PCAPNav (c) by Christian Kreibich, May 2007, BSD Licensed. +RE2 (c) by Russ Cox, 2012, New BSD Licensed \ No newline at end of file diff --git a/INSTALL b/INSTALL index 912f674..6eed51b 100644 --- a/INSTALL +++ b/INSTALL @@ -1,2 +1,2 @@ -see doc/INSTALL +see doc/install.rst diff --git a/src/cmd_parser.yy b/src/cmd_parser.yy index fa3c611..52e4a83 100644 --- a/src/cmd_parser.yy +++ b/src/cmd_parser.yy @@ -135,7 +135,7 @@ cmd: } | TOK_HELP { if (cmd_parser_outfp) { - fprintf(cmd_parser_outfp, "Online help not yet implemented. See doc/HOWTO\n"); + fprintf(cmd_parser_outfp, "Online help not yet implemented. See doc/howto.rst\n"); } } /* From seth at icir.org Thu Jan 24 23:42:20 2013 From: seth at icir.org (Seth Hall) Date: Thu, 24 Jan 2013 23:42:20 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Added a "log_level" option for reducing the volume of logging. (94d45f5) Message-ID: <201301250742.r0P7gK46025792@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit 94d45f505c8284f2b22f26a8110d45562aae1615 Author: Seth Hall Date: Fri Jan 25 02:39:57 2013 -0500 Added a "log_level" option for reducing the volume of logging. >--------------------------------------------------------------- 94d45f505c8284f2b22f26a8110d45562aae1615 doc/howto.rst | 9 ++++++--- src/conf.h | 1 + src/conf_parser.yy | 5 ++++- src/conf_scanner.ll | 3 +++ src/main.cc | 7 ++++--- 5 files changed, 18 insertions(+), 7 deletions(-) diff --git a/doc/howto.rst b/doc/howto.rst index 733f958..e4002a6 100644 --- a/doc/howto.rst +++ b/doc/howto.rst @@ -72,6 +72,9 @@ Main section options Specify the frequency of statistical output to the logfile (log interval in seconds). + log_level + Specify the level at which to log. 10=DEBUG, 20=NOTE(default), 30=WARN, 40=ERROR + conn_timeout Determine the inactivity timeout in seconds for connections before they are deleted from the timemachine's connection table. @@ -239,7 +242,7 @@ Full query grammar:: The flags can be given in order and they can be combined. Query the indexes for the given index key. The result of a query -can either be send to a remote bro system or to a file in the +can either be sent to a remote Bro system or to a file in the local filesystem. The index to query is specified by the keyword index followed by the name of the index. This name corresponds to the @@ -254,13 +257,13 @@ a subsciption (see above). Currently only connection4 querys support the subscribe flag, other indexes will silently ignore the flag. When mem_only is specified, only the index entries stored in RAM -are searched and only packets from the memomry ringbuffer are +are searched and only packets from the memory ringbuffer are returned. The timestamps enable one, to specify a timespan. Only packets falling in this timespan will be returned. timestamps and mem_only can be combined. The result will be the intersection of both (i.e. only packets from memory, that fall into the specified -timefragme). +timeframe). The timespan has not been tested extensively. Examples:: diff --git a/src/conf.h b/src/conf.h index d6e8a44..ef76e01 100644 --- a/src/conf.h +++ b/src/conf.h @@ -10,6 +10,7 @@ */ extern int conf_main_log_interval; +extern int conf_main_log_level; extern const char* conf_main_workdir; extern const char* conf_main_indexdir; extern const char* conf_main_logfile_name; diff --git a/src/conf_parser.yy b/src/conf_parser.yy index 9bfee85..01a7ae1 100644 --- a/src/conf_parser.yy +++ b/src/conf_parser.yy @@ -66,7 +66,7 @@ %token TOK_DOUBLE %token TOK_ID %token TOK_IPADDRESS; -%token TOK_CLASS TOK_FILTER TOK_MAIN TOK_LOG_INTERVAL TOK_DEVICE +%token TOK_CLASS TOK_FILTER TOK_MAIN TOK_LOG_INTERVAL TOK_LOG_LEVEL TOK_DEVICE %token TOK_LOGFILE TOK_WORKDIR TOK_QUERYFILEDIR TOK_INDEXDIR %token TOK_READ_TRACEFILE TOK_BRO_CONNECT_STR %token TOK_MEM TOK_DISK TOK_K TOK_M TOK_G TOK_CUTOFF TOK_PRECEDENCE @@ -255,6 +255,9 @@ main_option: | TOK_LOG_INTERVAL TOK_INTEGER { conf_main_log_interval=$2; } + | TOK_LOG_LEVEL TOK_INTEGER { + conf_main_log_level=$2; + } | TOK_READ_TRACEFILE TOK_STRING { if (conf_parser_storageConf->readtracefile.empty()) conf_parser_storageConf->readtracefile.assign($2); diff --git a/src/conf_scanner.ll b/src/conf_scanner.ll index fcf44fa..81356e0 100644 --- a/src/conf_scanner.ll +++ b/src/conf_scanner.ll @@ -21,6 +21,8 @@ ID [a-zA-Z_]+[0-9]* INT [0-9]+ DBL [0-9]*\.[0-9]* IP [0-9]+\.[0-9]+\.[0-9]+\.[0-9]+ +HEX [0-9a-fA-F]+ +IP6 ("["({HEX}:){7}{HEX}"]")|("["0x{HEX}({HEX}|:)*"::"({HEX}|:)*"]")|("["({HEX}|:)*"::"({HEX}|:)*"]")|("["({HEX}|:)*"::"({HEX}|:)*({INT}"."){3}{INT}"]") WHITE [ \t]+ COMMENT \#.* NEWLINE \n @@ -48,6 +50,7 @@ NEWLINE \n "dyn_timeout" return TOK_DYN_TIMEOUT; "main" return TOK_MAIN; "log_interval" return TOK_LOG_INTERVAL; +"log_level" return TOK_LOG_LEVEL; "device" return TOK_DEVICE; "read_tracefile" return TOK_READ_TRACEFILE; "no" return TOK_NO; diff --git a/src/main.cc b/src/main.cc index c15da44..4bb123d 100644 --- a/src/main.cc +++ b/src/main.cc @@ -96,10 +96,11 @@ pthread_t StorageConfig storageConf; int conf_main_log_interval=60; +int conf_main_log_level=20; const char* conf_main_workdir="./"; const char* conf_main_indexdir="./"; const char* conf_main_queryfiledir="./"; -const char* conf_main_logfile_name="tm.log"; +const char* conf_main_logfile_name="timemachine.log"; const char* conf_main_bro_connect_str=NULL; int conf_main_console=0; int conf_main_daemon=0; @@ -115,7 +116,7 @@ struct in_addr conf_main_bro_listen_addr; // defualt value is set in main! /*************************************************************************** - * loggins functions + * logging functions */ #define MAX_MSG_LEN 1024 @@ -123,7 +124,7 @@ static void tmlog_backend(int severity, const char *ident, const char *msg) { if (severity == TM_LOG_ERROR && stderr_is_open) { fprintf(stderr, "tm: %s: %s\n", ident, msg); } - if (log_file) + if (log_file && conf_main_log_level <= severity) log_file->log(ident, msg); } From seth at icir.org Thu Jan 24 23:42:23 2013 From: seth at icir.org (Seth Hall) Date: Thu, 24 Jan 2013 23:42:23 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Some groundwork for IPv6 support. (1d269b3) Message-ID: <201301250742.r0P7gNdD025801@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit 1d269b3842611cf0bd88efa15275ad2f1e07cceb Author: Seth Hall Date: Fri Jan 25 02:42:22 2013 -0500 Some groundwork for IPv6 support. - Brought Bro's IPAddr class into TimeMachine. - Modified the command scanner to recognize IPv6 in commands. - Began changing some data structures to use the IPAddr class instead of uint32_t >--------------------------------------------------------------- 1d269b3842611cf0bd88efa15275ad2f1e07cceb etc/timemachine.conf | 2 +- src/CMakeLists.txt | 2 + src/Connection.cc | 85 ++++------- src/Connection.hh | 44 +++--- src/IPAddr.cc | 178 ++++++++++++++++++++++ src/IPAddr.hh | 415 +++++++++++++++++++++++++++++++++++++++++++++++++++ src/IndexField.cc | 77 ++++------ src/IndexField.hh | 39 ++++- src/cmd_scanner.ll | 3 +- src/types.h | 2 + 10 files changed, 708 insertions(+), 139 deletions(-) diff --git a/etc/timemachine.conf b/etc/timemachine.conf index ed2465e..53ec759 100644 --- a/etc/timemachine.conf +++ b/etc/timemachine.conf @@ -2,7 +2,7 @@ main { ## The device to sniff. - device "en2"; + device "en0"; ## Directories for packet captures, logs, and indexes. ## These directories must exist when timemachine starts! diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 22afc16..f779fd0 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -49,6 +49,8 @@ set(timemachine_SRCS IndexField.hh IndexHash.cc IndexHash.hh + IPAddr.cc + IPAddr.hh LogFile.cc LogFile.hh Query.cc diff --git a/src/Connection.cc b/src/Connection.cc index a0d1ef8..4eb1c1e 100644 --- a/src/Connection.cc +++ b/src/Connection.cc @@ -12,8 +12,15 @@ #include "Query.hh" #include "tm.h" -static std::string pattern_ip ("(\\d+\\.\\d+\\.\\d+\\.\\d+)"); -static std::string pattern_ipport ("(\\d+\\.\\d+\\.\\d+\\.\\d+):(\\d+)"); +static std::string pattern_ip4 ("(?:\\d+\\.\\d+\\.\\d+\\.\\d+)"); +static std::string pattern_ip6_expanded ("(?:(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4})"); +static std::string pattern_ip6_compressed_hex ("(?:(?:[0-9A-Fa-f]{1,4}(?::[0-9A-Fa-f]{1,4})*)?)::(?:(?:[0-9A-Fa-f]{1,4}(?::[0-9A-Fa-f]{1,4})*)?)"); +static std::string pattern_ip6_6hex4dec ("(?:(?:[0-9A-Fa-f]{1,4}:){6,6})(?:[0-9]+)\\.(?:[0-9]+)\\.(?:[0-9]+)\\.(?:[0-9]+)"); +static std::string pattern_ip6_compressed_6hex4dec ("(?:(?:[0-9A-Fa-f]{1,4}(?::[0-9A-Fa-f]{1,4})*)?)::(?:(?:[0-9A-Fa-f]{1,4}:)*)(?:[0-9]+)\\.(?:[0-9]+)\\.(?:[0-9]+)\\.(?:[0-9]+)"); +static std::string pattern_ip6 = pattern_ip6_expanded + "|" + pattern_ip6_compressed_hex + "|" + pattern_ip6_6hex4dec + "|" + pattern_ip6_compressed_6hex4dec; +static std::string pattern_ip = "(" + pattern_ip4 + "|" + pattern_ip6 + ")"; + +static std::string pattern_ipport = pattern_ip + ":(\\d+)"; inline uint32_t revert_uint32(uint32_t i) { uint32_t r; @@ -47,14 +54,15 @@ void ConnectionID4::init(proto_t proto, v.proto=proto; if (addr_port_canon_lt(s_ip,d_ip,s_port,d_port)) { // v.is_canonified=true; - v.ip1=d_ip; - v.ip2=s_ip; + v.ip1=IPAddr(IPv4, &d_ip, IPAddr::Network); + v.ip2=IPAddr(IPv4, &s_ip, IPAddr::Network); + v.port1=d_port; v.port2=s_port; } else { // v.is_canonified=false; - v.ip1=s_ip; - v.ip2=d_ip; + v.ip1=IPAddr(IPv4, &s_ip, IPAddr::Network); + v.ip2=IPAddr(IPv4, &d_ip, IPAddr::Network); v.port1=s_port; v.port2=d_port; } @@ -64,20 +72,20 @@ void ConnectionID3::init(proto_t proto, uint32_t ip1, uint32_t ip2, uint16_t port2) { v.proto=proto; - v.ip1=ip1; - v.ip2=ip2; + v.ip1=IPAddr(IPv4, &ip1, IPAddr::Network); + v.ip2=IPAddr(IPv4, &ip2, IPAddr::Network); v.port2=port2; } void ConnectionID2::init( uint32_t s_ip, uint32_t d_ip) { if (addr_port_canon_lt(s_ip,d_ip,0,0)) { // v.is_canonified=true; - v.ip1=d_ip; - v.ip2=s_ip; + v.ip1=IPAddr(IPv4, &d_ip, IPAddr::Network); + v.ip2=IPAddr(IPv4, &s_ip, IPAddr::Network); } else { // v.is_canonified=false; - v.ip1=s_ip; - v.ip2=d_ip; + v.ip1=IPAddr(IPv4, &s_ip, IPAddr::Network); + v.ip2=IPAddr(IPv4, &d_ip, IPAddr::Network); } } @@ -174,29 +182,16 @@ void ConnectionID2::getStr(char* s, int maxsize) const { } std::string ConnectionID4::getStr() const { -#define UCP(x) ((unsigned char *)&x) - std::stringstream ss; - uint32_t s_ip=v.ip1; //get_s_ip(); - uint32_t d_ip=v.ip2; //get_d_ip(); - ss << " ConnectionID4 " - /* - << " Proto " << 0+get_proto() - << " canonified " << get_is_canonified() << " " - */ - << (UCP(s_ip)[0] & 0xff) << "." - << (UCP(s_ip)[1] & 0xff) << "." - << (UCP(s_ip)[2] & 0xff) << "." - << (UCP(s_ip)[3] & 0xff) + << get_proto() << " " + // << " canonified " << get_is_canonified() << " " + << get_ip1()->AsString() << ":" << ntohs(get_port1()) << " - " - << (UCP(d_ip)[0] & 0xff) << "." - << (UCP(d_ip)[1] & 0xff) << "." - << (UCP(d_ip)[2] & 0xff) << "." - << (UCP(d_ip)[3] & 0xff) + << get_ip2()->AsString() << ":" << ntohs(get_port2()); return ss.str(); @@ -204,46 +199,24 @@ std::string ConnectionID4::getStr() const { std::string ConnectionID3::getStr() const { -#define UCP(x) ((unsigned char *)&x) - std::stringstream ss; - uint32_t s_ip=get_ip1();//get_s_ip(); - uint32_t d_ip=get_ip2();//get_d_ip(); - ss << " ConnectionID3 " - << (UCP(s_ip)[0] & 0xff) << "." - << (UCP(s_ip)[1] & 0xff) << "." - << (UCP(s_ip)[2] & 0xff) << "." - << (UCP(s_ip)[3] & 0xff) + << get_ip1()->AsString() << " - " - << (UCP(d_ip)[0] & 0xff) << "." - << (UCP(d_ip)[1] & 0xff) << "." - << (UCP(d_ip)[2] & 0xff) << "." - << (UCP(d_ip)[3] & 0xff) + << get_ip2()->AsString() << ":" << get_port(); return ss.str(); } std::string ConnectionID2::getStr() const { -#define UCP(x) ((unsigned char *)&x) - std::stringstream ss; - uint32_t s_ip=get_ip1();//get_s_ip(); - uint32_t d_ip=get_ip2();//get_d_ip(); - ss << " ConnectionID2 " - << (UCP(s_ip)[0] & 0xff) << "." - << (UCP(s_ip)[1] & 0xff) << "." - << (UCP(s_ip)[2] & 0xff) << "." - << (UCP(s_ip)[3] & 0xff) + << get_ip1()->AsString() << " - " - << (UCP(d_ip)[0] & 0xff) << "." - << (UCP(d_ip)[1] & 0xff) << "." - << (UCP(d_ip)[2] & 0xff) << "." - << (UCP(d_ip)[3] & 0xff); + << get_ip2()->AsString(); return ss.str(); } @@ -251,7 +224,7 @@ std::string ConnectionID2::getStr() const { // Static Member initialization std::string ConnectionID4::pattern_connection4 = "\\s*(\\w+)\\s+" - + pattern_ipport + "\\s+" + pattern_ipport + "\\s*"; + + pattern_ipport + "\\s+-?\\s*" + pattern_ipport + "\\s*"; RE2 ConnectionID4::re(ConnectionID4::pattern_connection4); ConnectionID4* ConnectionID4::parse(const char *str) { diff --git a/src/Connection.hh b/src/Connection.hh index d95847c..36ca605 100644 --- a/src/Connection.hh +++ b/src/Connection.hh @@ -5,6 +5,8 @@ #include "types.h" #include "packet_headers.h" +#include "IPAddr.hh" + #include "jhash3.h" #include "re2/re2.h" @@ -42,7 +44,7 @@ public: virtual ~ConnectionID4() {}; uint32_t hash() const { //TODO: initval - return hash3words(v.ip1, v.ip2^v.proto, v.port1 | ((v.port2)<<16), 0); + return hash3words(v.ip1.Hash(), v.ip2.Hash()^v.proto, v.port1 | ((v.port2)<<16), 0); } bool operator==(const ConnectionID& other) const { @@ -58,11 +60,11 @@ public: proto_t get_proto() const { return v.proto; } - uint32_t get_ip1() const { - return v.ip1; + const IPAddr* get_ip1() const { + return (const IPAddr*) &v.ip1; } - uint32_t get_ip2() const { - return v.ip2; + const IPAddr* get_ip2() const { + return (const IPAddr*) &v.ip2; } uint16_t get_port1() const { return v.port1; @@ -84,8 +86,8 @@ public: typedef struct { // time locality // uint32_t ts; - uint32_t ip1; - uint32_t ip2; + IPAddr ip1; + IPAddr ip2; uint16_t port1; uint16_t port2; proto_t proto; @@ -121,17 +123,17 @@ public: virtual ~ConnectionID3() {}; uint32_t hash() const { //TODO: initval - return hash3words(v.ip1, v.ip2, v.port2 | ((v.proto)<<16), 0); + return hash3words(v.ip1.Hash(), v.ip2.Hash(), v.port2 | ((v.proto)<<16), 0); } bool operator==(const ConnectionID& other) const; proto_t get_proto() const { return v.proto; } - uint32_t get_ip1() const { - return v.ip1; + const IPAddr* get_ip1() const { + return (const IPAddr*) &v.ip1; } - uint32_t get_ip2() const { - return v.ip2; + const IPAddr* get_ip2() const { + return (const IPAddr*) &v.ip2; } uint16_t get_port() const { return v.port2; @@ -146,8 +148,8 @@ public: typedef struct { // time locality // uint32_t ts; - uint32_t ip1; - uint32_t ip2; + IPAddr ip1; + IPAddr ip2; uint16_t port2; proto_t proto; // bool is_canonified; @@ -177,14 +179,14 @@ public: virtual ~ConnectionID2() {}; uint32_t hash() const { //TODO: initval - return hash2words(v.ip1, v.ip2, 0); + return hash2words(v.ip1.Hash(), v.ip2.Hash(), 0); } bool operator==(const ConnectionID& other) const; - uint32_t get_ip1() const { - return v.ip1; + const IPAddr* get_ip1() const { + return (const IPAddr*) &v.ip1; } - uint32_t get_ip2() const { - return v.ip2; + const IPAddr* get_ip2() const { + return (const IPAddr*) &v.ip2; } /* bool get_is_canonified() const { return v.is_canonified; } @@ -196,8 +198,8 @@ public: typedef struct { // time locality // uint32_t ts; - uint32_t ip1; - uint32_t ip2; + IPAddr ip1; + IPAddr ip2; // bool is_canonified; } __attribute__((packed)) v_t; diff --git a/src/IPAddr.cc b/src/IPAddr.cc new file mode 100644 index 0000000..1db1612 --- /dev/null +++ b/src/IPAddr.cc @@ -0,0 +1,178 @@ +#include +#include +#include "IPAddr.hh" + +const uint8_t IPAddr::v4_mapped_prefix[12] = { 0, 0, 0, 0, + 0, 0, 0, 0, + 0, 0, 0xff, 0xff }; + + +void IPAddr::Mask(int top_bits_to_keep) + { + if ( top_bits_to_keep < 0 || top_bits_to_keep > 128 ) + { + //reporter->Error("Bad IPAddr::Mask value %d", top_bits_to_keep); + return; + } + + uint32_t tmp[4]; + memcpy(tmp, in6.s6_addr, sizeof(in6.s6_addr)); + + int word = 3; + int bits_to_chop = 128 - top_bits_to_keep; + + while ( bits_to_chop >= 32 ) + { + tmp[word] = 0; + --word; + bits_to_chop -= 32; + } + + uint32_t w = ntohl(tmp[word]); + w >>= bits_to_chop; + w <<= bits_to_chop; + tmp[word] = htonl(w); + + memcpy(in6.s6_addr, tmp, sizeof(in6.s6_addr)); + } + +void IPAddr::ReverseMask(int top_bits_to_chop) + { + if ( top_bits_to_chop < 0 || top_bits_to_chop > 128 ) + { + //reporter->Error("Bad IPAddr::ReverseMask value %d", top_bits_to_chop); + return; + } + + uint32_t tmp[4]; + memcpy(tmp, in6.s6_addr, sizeof(in6.s6_addr)); + + int word = 0; + int bits_to_chop = top_bits_to_chop; + + while ( bits_to_chop >= 32 ) + { + tmp[word] = 0; + ++word; + bits_to_chop -= 32; + } + + uint32_t w = ntohl(tmp[word]); + w <<= bits_to_chop; + w >>= bits_to_chop; + tmp[word] = htonl(w); + + memcpy(in6.s6_addr, tmp, sizeof(in6.s6_addr)); + } + +void IPAddr::Init(const std::string& s) + { + if ( s.find(':') == std::string::npos ) // IPv4. + { + memcpy(in6.s6_addr, v4_mapped_prefix, sizeof(v4_mapped_prefix)); + + // Parse the address directly instead of using inet_pton since + // some platforms have more sensitive implementations than others + // that can't e.g. handle leading zeroes. + int a[4]; + int n = sscanf(s.c_str(), "%d.%d.%d.%d", a+0, a+1, a+2, a+3); + + if ( n != 4 || a[0] < 0 || a[1] < 0 || a[2] < 0 || a[3] < 0 || + a[0] > 255 || a[1] > 255 || a[2] > 255 || a[3] > 255 ) + { + //reporter->Error("Bad IP address: %s", s.c_str()); + memset(in6.s6_addr, 0, sizeof(in6.s6_addr)); + return; + } + + uint32_t addr = (a[0] << 24) | (a[1] << 16) | (a[2] << 8) | a[3]; + addr = htonl(addr); + memcpy(&in6.s6_addr[12], &addr, sizeof(uint32_t)); + } + + else + { + if ( inet_pton(AF_INET6, s.c_str(), in6.s6_addr) <=0 ) + { + //reporter->Error("Bad IP address: %s", s.c_str()); + memset(in6.s6_addr, 0, sizeof(in6.s6_addr)); + } + } + } + +std::string IPAddr::AsString() const + { + if ( GetFamily() == IPv4 ) + { + char s[INET_ADDRSTRLEN]; + + if ( ! inet_ntop(AF_INET, &in6.s6_addr[12], s, INET_ADDRSTRLEN) ) + return "> 24) & 0xff; + uint32_t a2 = (a >> 16) & 0xff; + uint32_t a1 = (a >> 8) & 0xff; + uint32_t a0 = a & 0xff; + snprintf(buf, sizeof(buf), "%u.%u.%u.%u.in-addr.arpa", a0, a1, a2, a3); + return buf; + } + else + { + static const char hex_digit[] = "0123456789abcdef"; + std::string ptr_name("ip6.arpa"); + uint32_t* p = (uint32_t*) in6.s6_addr; + + for ( unsigned int i = 0; i < 4; ++i ) + { + uint32_t a = ntohl(p[i]); + for ( unsigned int j = 1; j <=8; ++j ) + { + ptr_name.insert(0, 1, '.'); + ptr_name.insert(0, 1, hex_digit[(a >> (32-j*4)) & 0x0f]); + } + } + + return ptr_name; + } + } + diff --git a/src/IPAddr.hh b/src/IPAddr.hh new file mode 100644 index 0000000..27dab03 --- /dev/null +++ b/src/IPAddr.hh @@ -0,0 +1,415 @@ +// See the file "COPYING" in the main distribution directory for copyright. + +#ifndef IPADDR_H +#define IPADDR_H + +#include +#include +#include +#include "types.h" +#include "jhash3.h" + +typedef in_addr in4_addr; +/** + * Class storing both IPv4 and IPv6 addresses. + */ +class IPAddr +{ +public: + /** + * Address family. + */ + typedef IPFamily Family; + + /** + * Byte order. + */ + enum ByteOrder { Host, Network }; + + /** + * Constructs the unspecified IPv6 address (all 128 bits zeroed). + */ + IPAddr() + { + memset(in6.s6_addr, 0, sizeof(in6.s6_addr)); + } + + /** + * Constructs an address instance from an IPv4 address. + * + * @param in6 The IPv6 address. + */ + explicit IPAddr(const in4_addr& in4) + { + memcpy(in6.s6_addr, v4_mapped_prefix, sizeof(v4_mapped_prefix)); + memcpy(&in6.s6_addr[12], &in4.s_addr, sizeof(in4.s_addr)); + } + + /** + * Constructs an address instance from an IPv6 address. + * + * @param in6 The IPv6 address. + */ + explicit IPAddr(const in6_addr& arg_in6) : in6(arg_in6) { } + + /** + * Constructs an address instance from a string representation. + * + * @param s String containing an IP address as either a dotted IPv4 + * address or a hex IPv6 address. + */ + IPAddr(const std::string& s) + { + Init(s); + } + + /** + * Constructs an address instance from a string representation. + * + * @param s ASCIIZ string containing an IP address as either a + * dotted IPv4 address or a hex IPv6 address. + */ + IPAddr(const char* s) + { + Init(s); + } + + /** + * Constructs an address instance from a string representation. + * + * @param s String containing an IP address as either a dotted IPv4 + * address or a hex IPv6 address. + */ + IPAddr(const std::string s) + { + Init(s.c_str()); + } + + /** + * Constructs an address instance from a raw byte representation. + * + * @param family The address family. + * + * @param bytes A pointer to the raw byte representation. This must point + * to 4 bytes if \a family is IPv4, and to 16 bytes if \a family is + * IPv6. + * + * @param order Indicates whether the raw representation pointed to + * by \a bytes is stored in network or host order. + */ + IPAddr(Family family, const uint32_t* bytes, ByteOrder order); + + /** + * Copy constructor. + */ + IPAddr(const IPAddr& other) : in6(other.in6) { }; + + /** + * Destructor. + */ + ~IPAddr() { }; + + /** + * Returns the address' family. + */ + Family GetFamily() const + { + if ( memcmp(in6.s6_addr, v4_mapped_prefix, 12) == 0 ) + return IPv4; + else + return IPv6; + } + + /** + * Returns true if the address represents a loopback device. + */ + bool IsLoopback() const; + + /** + * Returns true if the address represents a multicast address. + */ + bool IsMulticast() const + { + if ( GetFamily() == IPv4 ) + return in6.s6_addr[12] == 224; + else + return in6.s6_addr[0] == 0xff; + } + + /** + * Returns true if the address represents a broadcast address. + */ + bool IsBroadcast() const + { + if ( GetFamily() == IPv4 ) + return ((in6.s6_addr[12] == 0xff) && (in6.s6_addr[13] == 0xff) + && (in6.s6_addr[14] == 0xff) && (in6.s6_addr[15] == 0xff)); + else + return false; + } + + /** + * Retrieves the raw byte representation of the address. + * + * @param bytes The pointer to which \a bytes points will be set to + * the address of the raw representation in network-byte order. + * The return value indicates how many 32-bit words are valid starting at + * that address. The pointer will be valid as long as the address instance + * exists. + * + * @return The number of 32-bit words the raw representation uses. This + * will be 1 for an IPv4 address and 4 for an IPv6 address. + */ + int GetBytes(const uint32_t** bytes) const + { + if ( GetFamily() == IPv4 ) + { + *bytes = (uint32_t*) &in6.s6_addr[12]; + return 1; + } + else + { + *bytes = (uint32_t*) in6.s6_addr; + return 4; + } + } + + /** + * Retrieves a copy of the IPv6 raw byte representation of the address. + * If the internal address is IPv4, then the copied bytes use the + * IPv4 to IPv6 address mapping to return a full 16 bytes. + * + * @param bytes The pointer to a memory location in which the + * raw bytes of the address are to be copied. + * + * @param order The byte-order in which the returned raw bytes are copied. + * The default is network order. + */ + void CopyIPv6(uint32_t* bytes, ByteOrder order = Network) const + { + memcpy(bytes, in6.s6_addr, sizeof(in6.s6_addr)); + + if ( order == Host ) + { + for ( unsigned int i = 0; i < 4; ++i ) + bytes[i] = ntohl(bytes[i]); + } + } + + /** + * Retrieves a copy of the IPv6 raw byte representation of the address. + * @see CopyIPv6(uint32_t) + */ + void CopyIPv6(in6_addr* arg_in6) const + { + memcpy(arg_in6->s6_addr, in6.s6_addr, sizeof(in6.s6_addr)); + } + + /** + * Retrieves a copy of the IPv4 raw byte representation of the address. + * The caller should verify the address is of the IPv4 family type + * beforehand. @see GetFamily(). + * + * @param in4 The pointer to a memory location in which the raw bytes + * of the address are to be copied in network byte-order. + */ + void CopyIPv4(in4_addr* in4) const + { + memcpy(&in4->s_addr, &in6.s6_addr[12], sizeof(in4->s_addr)); + } + + uint32_t Hash() const + { + const uint32_t *bytes; + int len = GetBytes(&bytes); + if ( len == 1 ) + return hash1words(bytes[0], 0); + else + // TODO: this is only hashing the latter 96bits of the address. + return hash3words(bytes[1], bytes[2], bytes[3], 0); + } + + /** + * Masks out lower bits of the address. + * + * @param top_bits_to_keep The number of bits \a not to mask out, + * counting from the highest order bit. The value is always + * interpreted relative to the IPv6 bit width, even if the address + * is IPv4. That means if compute ``192.168.1.2/16``, you need to + * pass in 112 (i.e., 96 + 16). The value must be in the range from + * 0 to 128. + */ + void Mask(int top_bits_to_keep); + + /** + * Masks out top bits of the address. + * + * @param top_bits_to_chop The number of bits to mask out, counting + * from the highest order bit. The value is always interpreted relative + * to the IPv6 bit width, even if the address is IPv4. So to mask out + * the first 16 bits of an IPv4 address, pass in 112 (i.e., 96 + 16). + * The value must be in the range from 0 to 128. + */ + void ReverseMask(int top_bits_to_chop); + + /** + * Assignment operator. + */ + IPAddr& operator=(const IPAddr& other) + { + // No self-assignment check here because it's correct without it and + // makes the common case faster. + in6 = other.in6; + return *this; + } + + /** + * Bitwise OR operator returns the IP address resulting from the bitwise + * OR operation on the raw bytes of this address with another. + */ + IPAddr operator|(const IPAddr& other) + { + in6_addr result; + for ( int i = 0; i < 16; ++i ) + result.s6_addr[i] = this->in6.s6_addr[i] | other.in6.s6_addr[i]; + + return IPAddr(result); + } + + /** + * Returns a string representation of the address. IPv4 addresses + * will be returned in dotted representation, IPv6 addresses in + * compressed hex. + */ + std::string AsString() const; + + /** + * Returns a string representation of the address suitable for inclusion + * in an URI. For IPv4 addresses, this is the same as AsString(), but + * IPv6 addresses are encased in square brackets. + */ + std::string AsURIString() const + { + if ( GetFamily() == IPv4 ) + return AsString(); + else + return std::string("[") + AsString() + "]"; + } + + /** + * Returns a host-order, plain hex string representation of the address. + */ + std::string AsHexString() const; + + /** + * Returns a string representation of the address. This returns the + * same as AsString(). + */ + operator std::string() const { return AsString(); } + + /** + * Returns a reverse pointer name associated with the IP address. + * For example, 192.168.0.1's reverse pointer is 1.0.168.192.in-addr.arpa. + */ + std::string PtrName() const; + + /** + * Comparison operator for IP address. + */ + friend bool operator==(const IPAddr& addr1, const IPAddr& addr2) + { + return memcmp(&addr1.in6, &addr2.in6, sizeof(in6_addr)) == 0; + } + + friend bool operator!=(const IPAddr& addr1, const IPAddr& addr2) + { + return ! (addr1 == addr2); + } + + /** + * Comparison operator IP addresses. This defines a well-defined order for + * IP addresses. However, the order does not necessarily correspond to + * their numerical values. + */ + friend bool operator<(const IPAddr& addr1, const IPAddr& addr2) + { + return memcmp(&addr1.in6, &addr2.in6, sizeof(in6_addr)) < 0; + } + + friend bool operator<=(const IPAddr& addr1, const IPAddr& addr2) + { + return addr1 < addr2 || addr1 == addr2; + } + + friend bool operator>=(const IPAddr& addr1, const IPAddr& addr2) + { + return ! ( addr1 < addr2 ); + } + + friend bool operator>(const IPAddr& addr1, const IPAddr& addr2) + { + return ! ( addr1 <= addr2 ); + } + +private: + friend class IPPrefix; + + /** + * Initializes an address instance from a string representation. + * + * @param s String containing an IP address as either a dotted IPv4 + * address or a hex IPv6 address. + */ + void Init(const std::string& s); + + in6_addr in6; // IPv6 or v4-to-v6-mapped address + + static const uint8_t v4_mapped_prefix[12]; // top 96 bits of v4-mapped-addr +}; + +inline IPAddr::IPAddr(Family family, const uint32_t* bytes, ByteOrder order) + { + if ( family == IPv4 ) + { + memcpy(in6.s6_addr, v4_mapped_prefix, sizeof(v4_mapped_prefix)); + memcpy(&in6.s6_addr[12], bytes, sizeof(uint32_t)); + + if ( order == Host ) + { + uint32_t* p = (uint32_t*) &in6.s6_addr[12]; + *p = htonl(*p); + } + } + + else + { + memcpy(in6.s6_addr, bytes, sizeof(in6.s6_addr)); + + if ( order == Host ) + { + for ( unsigned int i = 0; i < 4; ++ i) + { + uint32_t* p = (uint32_t*) &in6.s6_addr[i*4]; + *p = htonl(*p); + } + } + } + } + +inline bool IPAddr::IsLoopback() const + { + if ( GetFamily() == IPv4 ) + return in6.s6_addr[12] == 127; + + else + return ((in6.s6_addr[0] == 0) && (in6.s6_addr[1] == 0) + && (in6.s6_addr[2] == 0) && (in6.s6_addr[3] == 0) + && (in6.s6_addr[4] == 0) && (in6.s6_addr[5] == 0) + && (in6.s6_addr[6] == 0) && (in6.s6_addr[7] == 0) + && (in6.s6_addr[8] == 0) && (in6.s6_addr[9] == 0) + && (in6.s6_addr[10] == 0) && (in6.s6_addr[11] == 0) + && (in6.s6_addr[12] == 0) && (in6.s6_addr[13] == 0) + && (in6.s6_addr[14] == 0) && (in6.s6_addr[15] == 1)); + } + +#endif diff --git a/src/IndexField.cc b/src/IndexField.cc index dfc266f..1c3c8d1 100644 --- a/src/IndexField.cc +++ b/src/IndexField.cc @@ -11,7 +11,14 @@ #include "IndexField.hh" #include "tm.h" -static std::string pattern_ip ("(\\d+\\.\\d+\\.\\d+\\.\\d+)"); +static std::string pattern_ip4 ("(\\d+\\.\\d+\\.\\d+\\.\\d+)"); +static std::string pattern_ip6_expanded ("(([0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4})"); +static std::string pattern_ip6_compressed_hex ("(([0-9A-Fa-f]{1,4}(:[0-9A-Fa-f]{1,4})*)?)::(([0-9A-Fa-f]{1,4}(:[0-9A-Fa-f]{1,4})*)?)"); +static std::string pattern_ip6_6hex4dec ("(([0-9A-Fa-f]{1,4}:){6,6})([0-9]+)\\.([0-9]+)\\.([0-9]+)\\.([0-9]+)"); +static std::string pattern_ip6_compressed_6hex4dec ("(([0-9A-Fa-f]{1,4}(:[0-9A-Fa-f]{1,4})*)?)::(([0-9A-Fa-f]{1,4}:)*)([0-9]+)\\.([0-9]+)\\.([0-9]+)\\.([0-9]+)"); +static std::string pattern_ip6 = "(" + pattern_ip6_expanded + "|" + pattern_ip6_compressed_hex + "|" + pattern_ip6_6hex4dec + "|" + pattern_ip6_compressed_6hex4dec + ")"; +static std::string pattern_ip = "(" + pattern_ip4 + "|" + pattern_ip6 + ")"; + static std::string pattern_ipport ("(\\d+\\.\\d+\\.\\d+\\.\\d+):(\\d+)"); @@ -57,28 +64,16 @@ std::list IPAddress::genKeys(const u_char* packet) { } void IPAddress::getStr(char* s, int maxsize) const { - unsigned char *ucp = (unsigned char *)&ip_address; - - snprintf(s, maxsize, "%d.%d.%d.%d", - ucp[0] & 0xff, - ucp[1] & 0xff, - ucp[2] & 0xff, - ucp[3] & 0xff); + snprintf(s, maxsize, "%s", addr.AsString().c_str()); } std::string IPAddress::getStr() const { - unsigned char *ucp = (unsigned char *)&ip_address; - std::stringstream ss; - ss << (ucp[0] & 0xff) << "." - << (ucp[1] & 0xff) << "." - << (ucp[2] & 0xff) << "." - << (ucp[3] & 0xff); - - return ss.str(); + return addr.AsString(); } void IPAddress::getBPFStr(char *str, int max_str_len) const { int rc = snprintf(str, max_str_len, "host %s", getStr().c_str()); + if ( rc >= max_str_len ) tmlog(TM_LOG_ERROR, "query", "IPAddress::getBPFStr: %s truncated by %d characters", str, rc-max_str_len); @@ -227,7 +222,7 @@ void DstPort::getBPFStr(char *str, int max_str_len) const { **********************************************************************/ // Static Member initialization std::string ConnectionIF4::pattern_connection4 = "\\s*(\\w+)\\s+" - + pattern_ipport + "\\s+" + pattern_ipport + "\\s*"; + + pattern_ipport + "\\s+-?\\s*" + pattern_ipport + "\\s*"; RE2 ConnectionIF4::re(ConnectionIF4::pattern_connection4); std::list ConnectionIF4::genKeys(const u_char* packet) { @@ -261,30 +256,16 @@ IndexField* ConnectionIF4::parseQuery(const char *query) { } void ConnectionIF4::getBPFStr(char *str, int max_str_len) const { - - char s_ip_str[TM_IP_STR_SIZE]; - char d_ip_str[TM_IP_STR_SIZE]; - uint32_t s_port; - uint32_t d_port; - /* - if (c_id.get_is_canonified()) { - s_ip=c_id.get_ip2(); - d_ip=c_id.get_ip1(); - s_port=c_id.get_port2(); - d_port=c_id.get_port1(); - } else { - */ - ip_to_str(c_id.get_ip1(), s_ip_str, sizeof(s_ip_str)); - ip_to_str(c_id.get_ip2(), d_ip_str, sizeof(d_ip_str)); - s_port=c_id.get_port1(); - d_port=c_id.get_port2(); - /* } */ + const char *ip1_str = c_id.get_ip1()->AsString().c_str(); + const char *ip2_str = c_id.get_ip2()->AsString().c_str(); + uint32_t s_port = c_id.get_port1(); + uint32_t d_port = c_id.get_port2(); snprintf(str, max_str_len, "host %s and port %d and host %s and port %d", - s_ip_str, + ip1_str, ntohs(s_port), - d_ip_str, + ip2_str, ntohs(d_port)); } @@ -294,7 +275,7 @@ void ConnectionIF4::getBPFStr(char *str, int max_str_len) const { **********************************************************************/ // Static Member initialization std::string ConnectionIF3::pattern_connection3 = "\\s*(\\w+)\\s+" - + pattern_ip + "\\s+" + pattern_ip + ":" + + pattern_ip + "\\s+-?\\s*" + pattern_ip + ":" + "(\\d+)\\s*"; RE2 ConnectionIF3::re(ConnectionIF3::pattern_connection3); @@ -329,12 +310,8 @@ IndexField* ConnectionIF3::parseQuery(const char *query) { } void ConnectionIF3::getBPFStr(char *str, int max_str_len) const { - - char ip1_str[TM_IP_STR_SIZE]; - char ip2_str[TM_IP_STR_SIZE]; - - ip_to_str(c_id.get_ip1(), ip1_str, sizeof(ip1_str)); - ip_to_str(c_id.get_ip2(), ip2_str, sizeof(ip2_str)); + const char *ip1_str = c_id.get_ip1()->AsString().c_str(); + const char *ip2_str = c_id.get_ip2()->AsString().c_str(); snprintf(str, max_str_len, "(src host %s and dst host %s and dst port %d) or " @@ -349,7 +326,7 @@ void ConnectionIF3::getBPFStr(char *str, int max_str_len) const { **********************************************************************/ // Static Member initialization std::string ConnectionIF2::pattern_connection2 = - "\\s*" + pattern_ip + "\\s+" + pattern_ip + "\\s*"; + "\\s*" + pattern_ip + "\\s+-?\\s*" + pattern_ip + "\\s*"; RE2 ConnectionIF2::re(ConnectionIF2::pattern_connection2); std::list @@ -375,16 +352,12 @@ IndexField* ConnectionIF2::parseQuery(const char *query) { void ConnectionIF2::getBPFStr(char *str, int max_str_len) const { - char s_ip_str[TM_IP_STR_SIZE]; - char d_ip_str[TM_IP_STR_SIZE]; - - ip_to_str(c_id.get_ip1(), s_ip_str, sizeof(s_ip_str)); - ip_to_str(c_id.get_ip2(), d_ip_str, sizeof(d_ip_str)); - + const char *ip1_str = c_id.get_ip1()->AsString().c_str(); + const char *ip2_str = c_id.get_ip2()->AsString().c_str(); snprintf(str, max_str_len, "host %s and host %s", - s_ip_str, d_ip_str); + ip1_str, ip2_str); } diff --git a/src/IndexField.hh b/src/IndexField.hh index 7be649c..674244b 100644 --- a/src/IndexField.hh +++ b/src/IndexField.hh @@ -4,11 +4,14 @@ #include #include #include +#include +#include #include #include "re2/re2.h" #include "types.h" +#include "IPAddr.hh" #include "packet_headers.h" class IndexField; @@ -63,25 +66,43 @@ class SrcIPAddress; class DstIPAddress; class IPAddress: public IndexField { public: - IPAddress(uint32_t ip): ip_address(ip) {} - IPAddress(const char* s): ip_address(inet_addr(s)) {} + IPAddress(uint32_t ip) { + addr = IPAddr(IPv4, &ip, IPAddr::Network); + } + IPAddress(const char* s) { + addr = IPAddr(s); + } IPAddress(void *p) { memcpy((void*)getConstKeyPtr(), p, getKeySize()); } virtual ~IPAddress() {}; virtual uint32_t hash() const { // TODO: initval - return hash1words(ip_address, 0); + const uint32_t *bytes; + int len = addr.GetBytes(&bytes); + if ( len == 1 ) + return hash1words(bytes[0], 0); + else + // TODO: this is only hashing the latter 96bits of the address. + return hash3words(bytes[1], bytes[2], bytes[3], 0); } virtual uint32_t getInt() const { - return ip_address; + const uint32_t *bytes; + int len = addr.GetBytes(&bytes); + if ( len == 1 ) + return bytes[3]; + else + return bytes[3]; } virtual const char* getConstKeyPtr() const { - return (const char*)&ip_address; + const uint32_t *bytes; + addr.GetBytes(&bytes); + return (const char*) bytes; } - // char* getKeyPtr() { return (char*)&ip_address; } virtual const int getKeySize() const { - return sizeof(ip_address); + const uint32_t *bytes; + int len = addr.GetBytes(&bytes); + return len*4; } virtual void getStr(char* s, int maxsize) const; virtual std::string getStr() const; @@ -91,6 +112,7 @@ public: static const std::string getIndexNameStatic() { return "ip"; } + static std::list genKeys(const u_char* packet); static int keysPerPacket() { return 2; @@ -102,7 +124,8 @@ public: static IndexField* parseQuery(const char *query); virtual void getBPFStr(char *, int) const; private: - uint32_t ip_address; + IPAddr addr; + static std::string pattern; static RE2 re; }; diff --git a/src/cmd_scanner.ll b/src/cmd_scanner.ll index 51f0103..d554a45 100644 --- a/src/cmd_scanner.ll +++ b/src/cmd_scanner.ll @@ -25,7 +25,8 @@ DBL -?[0-9]*\.[0-9]* WHITE [ \t]+ COMMENT \#.* NEWLINE \n -IP [0-9]+\.[0-9]+\.[0-9]+\.[0-9]+ +HEX [0-9a-fA-F]+ +IP ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+|"["({HEX}:){7}{HEX}"]")|("["0x{HEX}({HEX}|:)*"::"({HEX}|:)*"]")|("["({HEX}|:)*"::"({HEX}|:)*"]")|("["({HEX}|:)*"::"({HEX}|:)*({INT}"."){3}{INT}"]") %% "{" return LBRACE; diff --git a/src/types.h b/src/types.h index 8163802..7362d13 100644 --- a/src/types.h +++ b/src/types.h @@ -23,4 +23,6 @@ typedef uint8_t proto_t; //typedef double hash_t; +typedef enum { IPv4, IPv6 } IPFamily; + #endif From seth at icir.org Fri Jan 25 00:04:49 2013 From: seth at icir.org (Seth Hall) Date: Fri, 25 Jan 2013 00:04:49 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Removed another old debug statement. (8ba90ee) Message-ID: <201301250804.r0P84now029115@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit 8ba90ee09949cf31b5a05144777d336a4cd509c0 Author: Seth Hall Date: Fri Jan 25 03:04:52 2013 -0500 Removed another old debug statement. >--------------------------------------------------------------- 8ba90ee09949cf31b5a05144777d336a4cd509c0 src/Storage.cc | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/Storage.cc b/src/Storage.cc index 4b5c6b9..92d5f68 100644 --- a/src/Storage.cc +++ b/src/Storage.cc @@ -413,8 +413,7 @@ tm_time_t Storage::getOldestTimestampDisk() { void Storage::query(QueryRequest *query_req, QueryResult *query_res) { struct timeval t_start, t_end; gettimeofday(&t_start, NULL); - fprintf(stderr, "Query ID: %d\n", query_res->getQueryID()); - + //fprintf(stderr, "Query ID: %d\n", query_res->getQueryID()); IndexType* idx=indexes->getIndexByName(query_req->getField()->getIndexName()); if (!idx) { From seth at icir.org Fri Jan 25 00:21:21 2013 From: seth at icir.org (Seth Hall) Date: Fri, 25 Jan 2013 00:21:21 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Added query logging at the INFO logging level (default). (670e0a6) Message-ID: <201301250821.r0P8LLVG030511@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit 670e0a68d07b62e3573afbefcd580169e6b8a71a Author: Seth Hall Date: Fri Jan 25 03:21:22 2013 -0500 Added query logging at the INFO logging level (default). >--------------------------------------------------------------- 670e0a68d07b62e3573afbefcd580169e6b8a71a src/cmd_parser.yy | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/cmd_parser.yy b/src/cmd_parser.yy index 52e4a83..a6f45a8 100644 --- a/src/cmd_parser.yy +++ b/src/cmd_parser.yy @@ -443,6 +443,10 @@ cmderror(const char *msg) { /* Don't forget to call cmd_parser_init(), before you call the parser */ int parse_cmd(const char* cmd, FILE *outfp, Storage* s, broccoli_worker_thread_data* thread) { + if ( strstr(cmd, "query") != NULL ) { + tmlog(TM_LOG_NOTE, "query", "Query submitted: %s", cmd); + } + pthread_mutex_lock(&cmd_parser_lock); // set variable parser_storage global to conf_parser.cc q_req = NULL; @@ -450,7 +454,7 @@ parse_cmd(const char* cmd, FILE *outfp, Storage* s, broccoli_worker_thread_data* q_mem_only = false; q_subscribe = false; q_start = 0; - q_end = 1e13; /* along time in the future */ + q_end = 1e13; /* a long time in the future */ q_bc_thread = thread; cmd_parser_storage=s; From seth at icir.org Fri Jan 25 09:02:51 2013 From: seth at icir.org (Seth Hall) Date: Fri, 25 Jan 2013 09:02:51 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Revert "Some groundwork for IPv6 support." (869cd99) Message-ID: <201301251702.r0PH2pbC023616@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit 869cd997fcdaf66a8e5e1b0f9a2b98eaead1ba7a Author: Seth Hall Date: Fri Jan 25 11:59:40 2013 -0500 Revert "Some groundwork for IPv6 support." This reverts commit 1d269b3842611cf0bd88efa15275ad2f1e07cceb. >--------------------------------------------------------------- 869cd997fcdaf66a8e5e1b0f9a2b98eaead1ba7a etc/timemachine.conf | 2 +- src/CMakeLists.txt | 2 - src/Connection.cc | 85 +++++++---- src/Connection.hh | 44 +++--- src/IPAddr.cc | 178 ---------------------- src/IPAddr.hh | 415 --------------------------------------------------- src/IndexField.cc | 77 ++++++---- src/IndexField.hh | 39 +---- src/cmd_scanner.ll | 3 +- src/types.h | 2 - 10 files changed, 139 insertions(+), 708 deletions(-) diff --git a/etc/timemachine.conf b/etc/timemachine.conf index 53ec759..ed2465e 100644 --- a/etc/timemachine.conf +++ b/etc/timemachine.conf @@ -2,7 +2,7 @@ main { ## The device to sniff. - device "en0"; + device "en2"; ## Directories for packet captures, logs, and indexes. ## These directories must exist when timemachine starts! diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index f779fd0..22afc16 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -49,8 +49,6 @@ set(timemachine_SRCS IndexField.hh IndexHash.cc IndexHash.hh - IPAddr.cc - IPAddr.hh LogFile.cc LogFile.hh Query.cc diff --git a/src/Connection.cc b/src/Connection.cc index 4eb1c1e..a0d1ef8 100644 --- a/src/Connection.cc +++ b/src/Connection.cc @@ -12,15 +12,8 @@ #include "Query.hh" #include "tm.h" -static std::string pattern_ip4 ("(?:\\d+\\.\\d+\\.\\d+\\.\\d+)"); -static std::string pattern_ip6_expanded ("(?:(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4})"); -static std::string pattern_ip6_compressed_hex ("(?:(?:[0-9A-Fa-f]{1,4}(?::[0-9A-Fa-f]{1,4})*)?)::(?:(?:[0-9A-Fa-f]{1,4}(?::[0-9A-Fa-f]{1,4})*)?)"); -static std::string pattern_ip6_6hex4dec ("(?:(?:[0-9A-Fa-f]{1,4}:){6,6})(?:[0-9]+)\\.(?:[0-9]+)\\.(?:[0-9]+)\\.(?:[0-9]+)"); -static std::string pattern_ip6_compressed_6hex4dec ("(?:(?:[0-9A-Fa-f]{1,4}(?::[0-9A-Fa-f]{1,4})*)?)::(?:(?:[0-9A-Fa-f]{1,4}:)*)(?:[0-9]+)\\.(?:[0-9]+)\\.(?:[0-9]+)\\.(?:[0-9]+)"); -static std::string pattern_ip6 = pattern_ip6_expanded + "|" + pattern_ip6_compressed_hex + "|" + pattern_ip6_6hex4dec + "|" + pattern_ip6_compressed_6hex4dec; -static std::string pattern_ip = "(" + pattern_ip4 + "|" + pattern_ip6 + ")"; - -static std::string pattern_ipport = pattern_ip + ":(\\d+)"; +static std::string pattern_ip ("(\\d+\\.\\d+\\.\\d+\\.\\d+)"); +static std::string pattern_ipport ("(\\d+\\.\\d+\\.\\d+\\.\\d+):(\\d+)"); inline uint32_t revert_uint32(uint32_t i) { uint32_t r; @@ -54,15 +47,14 @@ void ConnectionID4::init(proto_t proto, v.proto=proto; if (addr_port_canon_lt(s_ip,d_ip,s_port,d_port)) { // v.is_canonified=true; - v.ip1=IPAddr(IPv4, &d_ip, IPAddr::Network); - v.ip2=IPAddr(IPv4, &s_ip, IPAddr::Network); - + v.ip1=d_ip; + v.ip2=s_ip; v.port1=d_port; v.port2=s_port; } else { // v.is_canonified=false; - v.ip1=IPAddr(IPv4, &s_ip, IPAddr::Network); - v.ip2=IPAddr(IPv4, &d_ip, IPAddr::Network); + v.ip1=s_ip; + v.ip2=d_ip; v.port1=s_port; v.port2=d_port; } @@ -72,20 +64,20 @@ void ConnectionID3::init(proto_t proto, uint32_t ip1, uint32_t ip2, uint16_t port2) { v.proto=proto; - v.ip1=IPAddr(IPv4, &ip1, IPAddr::Network); - v.ip2=IPAddr(IPv4, &ip2, IPAddr::Network); + v.ip1=ip1; + v.ip2=ip2; v.port2=port2; } void ConnectionID2::init( uint32_t s_ip, uint32_t d_ip) { if (addr_port_canon_lt(s_ip,d_ip,0,0)) { // v.is_canonified=true; - v.ip1=IPAddr(IPv4, &d_ip, IPAddr::Network); - v.ip2=IPAddr(IPv4, &s_ip, IPAddr::Network); + v.ip1=d_ip; + v.ip2=s_ip; } else { // v.is_canonified=false; - v.ip1=IPAddr(IPv4, &s_ip, IPAddr::Network); - v.ip2=IPAddr(IPv4, &d_ip, IPAddr::Network); + v.ip1=s_ip; + v.ip2=d_ip; } } @@ -182,16 +174,29 @@ void ConnectionID2::getStr(char* s, int maxsize) const { } std::string ConnectionID4::getStr() const { +#define UCP(x) ((unsigned char *)&x) + std::stringstream ss; + uint32_t s_ip=v.ip1; //get_s_ip(); + uint32_t d_ip=v.ip2; //get_d_ip(); + ss << " ConnectionID4 " - << get_proto() << " " - // << " canonified " << get_is_canonified() << " " - << get_ip1()->AsString() + /* + << " Proto " << 0+get_proto() + << " canonified " << get_is_canonified() << " " + */ + << (UCP(s_ip)[0] & 0xff) << "." + << (UCP(s_ip)[1] & 0xff) << "." + << (UCP(s_ip)[2] & 0xff) << "." + << (UCP(s_ip)[3] & 0xff) << ":" << ntohs(get_port1()) << " - " - << get_ip2()->AsString() + << (UCP(d_ip)[0] & 0xff) << "." + << (UCP(d_ip)[1] & 0xff) << "." + << (UCP(d_ip)[2] & 0xff) << "." + << (UCP(d_ip)[3] & 0xff) << ":" << ntohs(get_port2()); return ss.str(); @@ -199,24 +204,46 @@ std::string ConnectionID4::getStr() const { std::string ConnectionID3::getStr() const { +#define UCP(x) ((unsigned char *)&x) + std::stringstream ss; + uint32_t s_ip=get_ip1();//get_s_ip(); + uint32_t d_ip=get_ip2();//get_d_ip(); + ss << " ConnectionID3 " - << get_ip1()->AsString() + << (UCP(s_ip)[0] & 0xff) << "." + << (UCP(s_ip)[1] & 0xff) << "." + << (UCP(s_ip)[2] & 0xff) << "." + << (UCP(s_ip)[3] & 0xff) << " - " - << get_ip2()->AsString() + << (UCP(d_ip)[0] & 0xff) << "." + << (UCP(d_ip)[1] & 0xff) << "." + << (UCP(d_ip)[2] & 0xff) << "." + << (UCP(d_ip)[3] & 0xff) << ":" << get_port(); return ss.str(); } std::string ConnectionID2::getStr() const { +#define UCP(x) ((unsigned char *)&x) + std::stringstream ss; + uint32_t s_ip=get_ip1();//get_s_ip(); + uint32_t d_ip=get_ip2();//get_d_ip(); + ss << " ConnectionID2 " - << get_ip1()->AsString() + << (UCP(s_ip)[0] & 0xff) << "." + << (UCP(s_ip)[1] & 0xff) << "." + << (UCP(s_ip)[2] & 0xff) << "." + << (UCP(s_ip)[3] & 0xff) << " - " - << get_ip2()->AsString(); + << (UCP(d_ip)[0] & 0xff) << "." + << (UCP(d_ip)[1] & 0xff) << "." + << (UCP(d_ip)[2] & 0xff) << "." + << (UCP(d_ip)[3] & 0xff); return ss.str(); } @@ -224,7 +251,7 @@ std::string ConnectionID2::getStr() const { // Static Member initialization std::string ConnectionID4::pattern_connection4 = "\\s*(\\w+)\\s+" - + pattern_ipport + "\\s+-?\\s*" + pattern_ipport + "\\s*"; + + pattern_ipport + "\\s+" + pattern_ipport + "\\s*"; RE2 ConnectionID4::re(ConnectionID4::pattern_connection4); ConnectionID4* ConnectionID4::parse(const char *str) { diff --git a/src/Connection.hh b/src/Connection.hh index 36ca605..d95847c 100644 --- a/src/Connection.hh +++ b/src/Connection.hh @@ -5,8 +5,6 @@ #include "types.h" #include "packet_headers.h" -#include "IPAddr.hh" - #include "jhash3.h" #include "re2/re2.h" @@ -44,7 +42,7 @@ public: virtual ~ConnectionID4() {}; uint32_t hash() const { //TODO: initval - return hash3words(v.ip1.Hash(), v.ip2.Hash()^v.proto, v.port1 | ((v.port2)<<16), 0); + return hash3words(v.ip1, v.ip2^v.proto, v.port1 | ((v.port2)<<16), 0); } bool operator==(const ConnectionID& other) const { @@ -60,11 +58,11 @@ public: proto_t get_proto() const { return v.proto; } - const IPAddr* get_ip1() const { - return (const IPAddr*) &v.ip1; + uint32_t get_ip1() const { + return v.ip1; } - const IPAddr* get_ip2() const { - return (const IPAddr*) &v.ip2; + uint32_t get_ip2() const { + return v.ip2; } uint16_t get_port1() const { return v.port1; @@ -86,8 +84,8 @@ public: typedef struct { // time locality // uint32_t ts; - IPAddr ip1; - IPAddr ip2; + uint32_t ip1; + uint32_t ip2; uint16_t port1; uint16_t port2; proto_t proto; @@ -123,17 +121,17 @@ public: virtual ~ConnectionID3() {}; uint32_t hash() const { //TODO: initval - return hash3words(v.ip1.Hash(), v.ip2.Hash(), v.port2 | ((v.proto)<<16), 0); + return hash3words(v.ip1, v.ip2, v.port2 | ((v.proto)<<16), 0); } bool operator==(const ConnectionID& other) const; proto_t get_proto() const { return v.proto; } - const IPAddr* get_ip1() const { - return (const IPAddr*) &v.ip1; + uint32_t get_ip1() const { + return v.ip1; } - const IPAddr* get_ip2() const { - return (const IPAddr*) &v.ip2; + uint32_t get_ip2() const { + return v.ip2; } uint16_t get_port() const { return v.port2; @@ -148,8 +146,8 @@ public: typedef struct { // time locality // uint32_t ts; - IPAddr ip1; - IPAddr ip2; + uint32_t ip1; + uint32_t ip2; uint16_t port2; proto_t proto; // bool is_canonified; @@ -179,14 +177,14 @@ public: virtual ~ConnectionID2() {}; uint32_t hash() const { //TODO: initval - return hash2words(v.ip1.Hash(), v.ip2.Hash(), 0); + return hash2words(v.ip1, v.ip2, 0); } bool operator==(const ConnectionID& other) const; - const IPAddr* get_ip1() const { - return (const IPAddr*) &v.ip1; + uint32_t get_ip1() const { + return v.ip1; } - const IPAddr* get_ip2() const { - return (const IPAddr*) &v.ip2; + uint32_t get_ip2() const { + return v.ip2; } /* bool get_is_canonified() const { return v.is_canonified; } @@ -198,8 +196,8 @@ public: typedef struct { // time locality // uint32_t ts; - IPAddr ip1; - IPAddr ip2; + uint32_t ip1; + uint32_t ip2; // bool is_canonified; } __attribute__((packed)) v_t; diff --git a/src/IPAddr.cc b/src/IPAddr.cc deleted file mode 100644 index 1db1612..0000000 --- a/src/IPAddr.cc +++ /dev/null @@ -1,178 +0,0 @@ -#include -#include -#include "IPAddr.hh" - -const uint8_t IPAddr::v4_mapped_prefix[12] = { 0, 0, 0, 0, - 0, 0, 0, 0, - 0, 0, 0xff, 0xff }; - - -void IPAddr::Mask(int top_bits_to_keep) - { - if ( top_bits_to_keep < 0 || top_bits_to_keep > 128 ) - { - //reporter->Error("Bad IPAddr::Mask value %d", top_bits_to_keep); - return; - } - - uint32_t tmp[4]; - memcpy(tmp, in6.s6_addr, sizeof(in6.s6_addr)); - - int word = 3; - int bits_to_chop = 128 - top_bits_to_keep; - - while ( bits_to_chop >= 32 ) - { - tmp[word] = 0; - --word; - bits_to_chop -= 32; - } - - uint32_t w = ntohl(tmp[word]); - w >>= bits_to_chop; - w <<= bits_to_chop; - tmp[word] = htonl(w); - - memcpy(in6.s6_addr, tmp, sizeof(in6.s6_addr)); - } - -void IPAddr::ReverseMask(int top_bits_to_chop) - { - if ( top_bits_to_chop < 0 || top_bits_to_chop > 128 ) - { - //reporter->Error("Bad IPAddr::ReverseMask value %d", top_bits_to_chop); - return; - } - - uint32_t tmp[4]; - memcpy(tmp, in6.s6_addr, sizeof(in6.s6_addr)); - - int word = 0; - int bits_to_chop = top_bits_to_chop; - - while ( bits_to_chop >= 32 ) - { - tmp[word] = 0; - ++word; - bits_to_chop -= 32; - } - - uint32_t w = ntohl(tmp[word]); - w <<= bits_to_chop; - w >>= bits_to_chop; - tmp[word] = htonl(w); - - memcpy(in6.s6_addr, tmp, sizeof(in6.s6_addr)); - } - -void IPAddr::Init(const std::string& s) - { - if ( s.find(':') == std::string::npos ) // IPv4. - { - memcpy(in6.s6_addr, v4_mapped_prefix, sizeof(v4_mapped_prefix)); - - // Parse the address directly instead of using inet_pton since - // some platforms have more sensitive implementations than others - // that can't e.g. handle leading zeroes. - int a[4]; - int n = sscanf(s.c_str(), "%d.%d.%d.%d", a+0, a+1, a+2, a+3); - - if ( n != 4 || a[0] < 0 || a[1] < 0 || a[2] < 0 || a[3] < 0 || - a[0] > 255 || a[1] > 255 || a[2] > 255 || a[3] > 255 ) - { - //reporter->Error("Bad IP address: %s", s.c_str()); - memset(in6.s6_addr, 0, sizeof(in6.s6_addr)); - return; - } - - uint32_t addr = (a[0] << 24) | (a[1] << 16) | (a[2] << 8) | a[3]; - addr = htonl(addr); - memcpy(&in6.s6_addr[12], &addr, sizeof(uint32_t)); - } - - else - { - if ( inet_pton(AF_INET6, s.c_str(), in6.s6_addr) <=0 ) - { - //reporter->Error("Bad IP address: %s", s.c_str()); - memset(in6.s6_addr, 0, sizeof(in6.s6_addr)); - } - } - } - -std::string IPAddr::AsString() const - { - if ( GetFamily() == IPv4 ) - { - char s[INET_ADDRSTRLEN]; - - if ( ! inet_ntop(AF_INET, &in6.s6_addr[12], s, INET_ADDRSTRLEN) ) - return "> 24) & 0xff; - uint32_t a2 = (a >> 16) & 0xff; - uint32_t a1 = (a >> 8) & 0xff; - uint32_t a0 = a & 0xff; - snprintf(buf, sizeof(buf), "%u.%u.%u.%u.in-addr.arpa", a0, a1, a2, a3); - return buf; - } - else - { - static const char hex_digit[] = "0123456789abcdef"; - std::string ptr_name("ip6.arpa"); - uint32_t* p = (uint32_t*) in6.s6_addr; - - for ( unsigned int i = 0; i < 4; ++i ) - { - uint32_t a = ntohl(p[i]); - for ( unsigned int j = 1; j <=8; ++j ) - { - ptr_name.insert(0, 1, '.'); - ptr_name.insert(0, 1, hex_digit[(a >> (32-j*4)) & 0x0f]); - } - } - - return ptr_name; - } - } - diff --git a/src/IPAddr.hh b/src/IPAddr.hh deleted file mode 100644 index 27dab03..0000000 --- a/src/IPAddr.hh +++ /dev/null @@ -1,415 +0,0 @@ -// See the file "COPYING" in the main distribution directory for copyright. - -#ifndef IPADDR_H -#define IPADDR_H - -#include -#include -#include -#include "types.h" -#include "jhash3.h" - -typedef in_addr in4_addr; -/** - * Class storing both IPv4 and IPv6 addresses. - */ -class IPAddr -{ -public: - /** - * Address family. - */ - typedef IPFamily Family; - - /** - * Byte order. - */ - enum ByteOrder { Host, Network }; - - /** - * Constructs the unspecified IPv6 address (all 128 bits zeroed). - */ - IPAddr() - { - memset(in6.s6_addr, 0, sizeof(in6.s6_addr)); - } - - /** - * Constructs an address instance from an IPv4 address. - * - * @param in6 The IPv6 address. - */ - explicit IPAddr(const in4_addr& in4) - { - memcpy(in6.s6_addr, v4_mapped_prefix, sizeof(v4_mapped_prefix)); - memcpy(&in6.s6_addr[12], &in4.s_addr, sizeof(in4.s_addr)); - } - - /** - * Constructs an address instance from an IPv6 address. - * - * @param in6 The IPv6 address. - */ - explicit IPAddr(const in6_addr& arg_in6) : in6(arg_in6) { } - - /** - * Constructs an address instance from a string representation. - * - * @param s String containing an IP address as either a dotted IPv4 - * address or a hex IPv6 address. - */ - IPAddr(const std::string& s) - { - Init(s); - } - - /** - * Constructs an address instance from a string representation. - * - * @param s ASCIIZ string containing an IP address as either a - * dotted IPv4 address or a hex IPv6 address. - */ - IPAddr(const char* s) - { - Init(s); - } - - /** - * Constructs an address instance from a string representation. - * - * @param s String containing an IP address as either a dotted IPv4 - * address or a hex IPv6 address. - */ - IPAddr(const std::string s) - { - Init(s.c_str()); - } - - /** - * Constructs an address instance from a raw byte representation. - * - * @param family The address family. - * - * @param bytes A pointer to the raw byte representation. This must point - * to 4 bytes if \a family is IPv4, and to 16 bytes if \a family is - * IPv6. - * - * @param order Indicates whether the raw representation pointed to - * by \a bytes is stored in network or host order. - */ - IPAddr(Family family, const uint32_t* bytes, ByteOrder order); - - /** - * Copy constructor. - */ - IPAddr(const IPAddr& other) : in6(other.in6) { }; - - /** - * Destructor. - */ - ~IPAddr() { }; - - /** - * Returns the address' family. - */ - Family GetFamily() const - { - if ( memcmp(in6.s6_addr, v4_mapped_prefix, 12) == 0 ) - return IPv4; - else - return IPv6; - } - - /** - * Returns true if the address represents a loopback device. - */ - bool IsLoopback() const; - - /** - * Returns true if the address represents a multicast address. - */ - bool IsMulticast() const - { - if ( GetFamily() == IPv4 ) - return in6.s6_addr[12] == 224; - else - return in6.s6_addr[0] == 0xff; - } - - /** - * Returns true if the address represents a broadcast address. - */ - bool IsBroadcast() const - { - if ( GetFamily() == IPv4 ) - return ((in6.s6_addr[12] == 0xff) && (in6.s6_addr[13] == 0xff) - && (in6.s6_addr[14] == 0xff) && (in6.s6_addr[15] == 0xff)); - else - return false; - } - - /** - * Retrieves the raw byte representation of the address. - * - * @param bytes The pointer to which \a bytes points will be set to - * the address of the raw representation in network-byte order. - * The return value indicates how many 32-bit words are valid starting at - * that address. The pointer will be valid as long as the address instance - * exists. - * - * @return The number of 32-bit words the raw representation uses. This - * will be 1 for an IPv4 address and 4 for an IPv6 address. - */ - int GetBytes(const uint32_t** bytes) const - { - if ( GetFamily() == IPv4 ) - { - *bytes = (uint32_t*) &in6.s6_addr[12]; - return 1; - } - else - { - *bytes = (uint32_t*) in6.s6_addr; - return 4; - } - } - - /** - * Retrieves a copy of the IPv6 raw byte representation of the address. - * If the internal address is IPv4, then the copied bytes use the - * IPv4 to IPv6 address mapping to return a full 16 bytes. - * - * @param bytes The pointer to a memory location in which the - * raw bytes of the address are to be copied. - * - * @param order The byte-order in which the returned raw bytes are copied. - * The default is network order. - */ - void CopyIPv6(uint32_t* bytes, ByteOrder order = Network) const - { - memcpy(bytes, in6.s6_addr, sizeof(in6.s6_addr)); - - if ( order == Host ) - { - for ( unsigned int i = 0; i < 4; ++i ) - bytes[i] = ntohl(bytes[i]); - } - } - - /** - * Retrieves a copy of the IPv6 raw byte representation of the address. - * @see CopyIPv6(uint32_t) - */ - void CopyIPv6(in6_addr* arg_in6) const - { - memcpy(arg_in6->s6_addr, in6.s6_addr, sizeof(in6.s6_addr)); - } - - /** - * Retrieves a copy of the IPv4 raw byte representation of the address. - * The caller should verify the address is of the IPv4 family type - * beforehand. @see GetFamily(). - * - * @param in4 The pointer to a memory location in which the raw bytes - * of the address are to be copied in network byte-order. - */ - void CopyIPv4(in4_addr* in4) const - { - memcpy(&in4->s_addr, &in6.s6_addr[12], sizeof(in4->s_addr)); - } - - uint32_t Hash() const - { - const uint32_t *bytes; - int len = GetBytes(&bytes); - if ( len == 1 ) - return hash1words(bytes[0], 0); - else - // TODO: this is only hashing the latter 96bits of the address. - return hash3words(bytes[1], bytes[2], bytes[3], 0); - } - - /** - * Masks out lower bits of the address. - * - * @param top_bits_to_keep The number of bits \a not to mask out, - * counting from the highest order bit. The value is always - * interpreted relative to the IPv6 bit width, even if the address - * is IPv4. That means if compute ``192.168.1.2/16``, you need to - * pass in 112 (i.e., 96 + 16). The value must be in the range from - * 0 to 128. - */ - void Mask(int top_bits_to_keep); - - /** - * Masks out top bits of the address. - * - * @param top_bits_to_chop The number of bits to mask out, counting - * from the highest order bit. The value is always interpreted relative - * to the IPv6 bit width, even if the address is IPv4. So to mask out - * the first 16 bits of an IPv4 address, pass in 112 (i.e., 96 + 16). - * The value must be in the range from 0 to 128. - */ - void ReverseMask(int top_bits_to_chop); - - /** - * Assignment operator. - */ - IPAddr& operator=(const IPAddr& other) - { - // No self-assignment check here because it's correct without it and - // makes the common case faster. - in6 = other.in6; - return *this; - } - - /** - * Bitwise OR operator returns the IP address resulting from the bitwise - * OR operation on the raw bytes of this address with another. - */ - IPAddr operator|(const IPAddr& other) - { - in6_addr result; - for ( int i = 0; i < 16; ++i ) - result.s6_addr[i] = this->in6.s6_addr[i] | other.in6.s6_addr[i]; - - return IPAddr(result); - } - - /** - * Returns a string representation of the address. IPv4 addresses - * will be returned in dotted representation, IPv6 addresses in - * compressed hex. - */ - std::string AsString() const; - - /** - * Returns a string representation of the address suitable for inclusion - * in an URI. For IPv4 addresses, this is the same as AsString(), but - * IPv6 addresses are encased in square brackets. - */ - std::string AsURIString() const - { - if ( GetFamily() == IPv4 ) - return AsString(); - else - return std::string("[") + AsString() + "]"; - } - - /** - * Returns a host-order, plain hex string representation of the address. - */ - std::string AsHexString() const; - - /** - * Returns a string representation of the address. This returns the - * same as AsString(). - */ - operator std::string() const { return AsString(); } - - /** - * Returns a reverse pointer name associated with the IP address. - * For example, 192.168.0.1's reverse pointer is 1.0.168.192.in-addr.arpa. - */ - std::string PtrName() const; - - /** - * Comparison operator for IP address. - */ - friend bool operator==(const IPAddr& addr1, const IPAddr& addr2) - { - return memcmp(&addr1.in6, &addr2.in6, sizeof(in6_addr)) == 0; - } - - friend bool operator!=(const IPAddr& addr1, const IPAddr& addr2) - { - return ! (addr1 == addr2); - } - - /** - * Comparison operator IP addresses. This defines a well-defined order for - * IP addresses. However, the order does not necessarily correspond to - * their numerical values. - */ - friend bool operator<(const IPAddr& addr1, const IPAddr& addr2) - { - return memcmp(&addr1.in6, &addr2.in6, sizeof(in6_addr)) < 0; - } - - friend bool operator<=(const IPAddr& addr1, const IPAddr& addr2) - { - return addr1 < addr2 || addr1 == addr2; - } - - friend bool operator>=(const IPAddr& addr1, const IPAddr& addr2) - { - return ! ( addr1 < addr2 ); - } - - friend bool operator>(const IPAddr& addr1, const IPAddr& addr2) - { - return ! ( addr1 <= addr2 ); - } - -private: - friend class IPPrefix; - - /** - * Initializes an address instance from a string representation. - * - * @param s String containing an IP address as either a dotted IPv4 - * address or a hex IPv6 address. - */ - void Init(const std::string& s); - - in6_addr in6; // IPv6 or v4-to-v6-mapped address - - static const uint8_t v4_mapped_prefix[12]; // top 96 bits of v4-mapped-addr -}; - -inline IPAddr::IPAddr(Family family, const uint32_t* bytes, ByteOrder order) - { - if ( family == IPv4 ) - { - memcpy(in6.s6_addr, v4_mapped_prefix, sizeof(v4_mapped_prefix)); - memcpy(&in6.s6_addr[12], bytes, sizeof(uint32_t)); - - if ( order == Host ) - { - uint32_t* p = (uint32_t*) &in6.s6_addr[12]; - *p = htonl(*p); - } - } - - else - { - memcpy(in6.s6_addr, bytes, sizeof(in6.s6_addr)); - - if ( order == Host ) - { - for ( unsigned int i = 0; i < 4; ++ i) - { - uint32_t* p = (uint32_t*) &in6.s6_addr[i*4]; - *p = htonl(*p); - } - } - } - } - -inline bool IPAddr::IsLoopback() const - { - if ( GetFamily() == IPv4 ) - return in6.s6_addr[12] == 127; - - else - return ((in6.s6_addr[0] == 0) && (in6.s6_addr[1] == 0) - && (in6.s6_addr[2] == 0) && (in6.s6_addr[3] == 0) - && (in6.s6_addr[4] == 0) && (in6.s6_addr[5] == 0) - && (in6.s6_addr[6] == 0) && (in6.s6_addr[7] == 0) - && (in6.s6_addr[8] == 0) && (in6.s6_addr[9] == 0) - && (in6.s6_addr[10] == 0) && (in6.s6_addr[11] == 0) - && (in6.s6_addr[12] == 0) && (in6.s6_addr[13] == 0) - && (in6.s6_addr[14] == 0) && (in6.s6_addr[15] == 1)); - } - -#endif diff --git a/src/IndexField.cc b/src/IndexField.cc index 1c3c8d1..dfc266f 100644 --- a/src/IndexField.cc +++ b/src/IndexField.cc @@ -11,14 +11,7 @@ #include "IndexField.hh" #include "tm.h" -static std::string pattern_ip4 ("(\\d+\\.\\d+\\.\\d+\\.\\d+)"); -static std::string pattern_ip6_expanded ("(([0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4})"); -static std::string pattern_ip6_compressed_hex ("(([0-9A-Fa-f]{1,4}(:[0-9A-Fa-f]{1,4})*)?)::(([0-9A-Fa-f]{1,4}(:[0-9A-Fa-f]{1,4})*)?)"); -static std::string pattern_ip6_6hex4dec ("(([0-9A-Fa-f]{1,4}:){6,6})([0-9]+)\\.([0-9]+)\\.([0-9]+)\\.([0-9]+)"); -static std::string pattern_ip6_compressed_6hex4dec ("(([0-9A-Fa-f]{1,4}(:[0-9A-Fa-f]{1,4})*)?)::(([0-9A-Fa-f]{1,4}:)*)([0-9]+)\\.([0-9]+)\\.([0-9]+)\\.([0-9]+)"); -static std::string pattern_ip6 = "(" + pattern_ip6_expanded + "|" + pattern_ip6_compressed_hex + "|" + pattern_ip6_6hex4dec + "|" + pattern_ip6_compressed_6hex4dec + ")"; -static std::string pattern_ip = "(" + pattern_ip4 + "|" + pattern_ip6 + ")"; - +static std::string pattern_ip ("(\\d+\\.\\d+\\.\\d+\\.\\d+)"); static std::string pattern_ipport ("(\\d+\\.\\d+\\.\\d+\\.\\d+):(\\d+)"); @@ -64,16 +57,28 @@ std::list IPAddress::genKeys(const u_char* packet) { } void IPAddress::getStr(char* s, int maxsize) const { - snprintf(s, maxsize, "%s", addr.AsString().c_str()); + unsigned char *ucp = (unsigned char *)&ip_address; + + snprintf(s, maxsize, "%d.%d.%d.%d", + ucp[0] & 0xff, + ucp[1] & 0xff, + ucp[2] & 0xff, + ucp[3] & 0xff); } std::string IPAddress::getStr() const { - return addr.AsString(); + unsigned char *ucp = (unsigned char *)&ip_address; + std::stringstream ss; + ss << (ucp[0] & 0xff) << "." + << (ucp[1] & 0xff) << "." + << (ucp[2] & 0xff) << "." + << (ucp[3] & 0xff); + + return ss.str(); } void IPAddress::getBPFStr(char *str, int max_str_len) const { int rc = snprintf(str, max_str_len, "host %s", getStr().c_str()); - if ( rc >= max_str_len ) tmlog(TM_LOG_ERROR, "query", "IPAddress::getBPFStr: %s truncated by %d characters", str, rc-max_str_len); @@ -222,7 +227,7 @@ void DstPort::getBPFStr(char *str, int max_str_len) const { **********************************************************************/ // Static Member initialization std::string ConnectionIF4::pattern_connection4 = "\\s*(\\w+)\\s+" - + pattern_ipport + "\\s+-?\\s*" + pattern_ipport + "\\s*"; + + pattern_ipport + "\\s+" + pattern_ipport + "\\s*"; RE2 ConnectionIF4::re(ConnectionIF4::pattern_connection4); std::list ConnectionIF4::genKeys(const u_char* packet) { @@ -256,16 +261,30 @@ IndexField* ConnectionIF4::parseQuery(const char *query) { } void ConnectionIF4::getBPFStr(char *str, int max_str_len) const { - const char *ip1_str = c_id.get_ip1()->AsString().c_str(); - const char *ip2_str = c_id.get_ip2()->AsString().c_str(); - uint32_t s_port = c_id.get_port1(); - uint32_t d_port = c_id.get_port2(); + + char s_ip_str[TM_IP_STR_SIZE]; + char d_ip_str[TM_IP_STR_SIZE]; + uint32_t s_port; + uint32_t d_port; + /* + if (c_id.get_is_canonified()) { + s_ip=c_id.get_ip2(); + d_ip=c_id.get_ip1(); + s_port=c_id.get_port2(); + d_port=c_id.get_port1(); + } else { + */ + ip_to_str(c_id.get_ip1(), s_ip_str, sizeof(s_ip_str)); + ip_to_str(c_id.get_ip2(), d_ip_str, sizeof(d_ip_str)); + s_port=c_id.get_port1(); + d_port=c_id.get_port2(); + /* } */ snprintf(str, max_str_len, "host %s and port %d and host %s and port %d", - ip1_str, + s_ip_str, ntohs(s_port), - ip2_str, + d_ip_str, ntohs(d_port)); } @@ -275,7 +294,7 @@ void ConnectionIF4::getBPFStr(char *str, int max_str_len) const { **********************************************************************/ // Static Member initialization std::string ConnectionIF3::pattern_connection3 = "\\s*(\\w+)\\s+" - + pattern_ip + "\\s+-?\\s*" + pattern_ip + ":" + + pattern_ip + "\\s+" + pattern_ip + ":" + "(\\d+)\\s*"; RE2 ConnectionIF3::re(ConnectionIF3::pattern_connection3); @@ -310,8 +329,12 @@ IndexField* ConnectionIF3::parseQuery(const char *query) { } void ConnectionIF3::getBPFStr(char *str, int max_str_len) const { - const char *ip1_str = c_id.get_ip1()->AsString().c_str(); - const char *ip2_str = c_id.get_ip2()->AsString().c_str(); + + char ip1_str[TM_IP_STR_SIZE]; + char ip2_str[TM_IP_STR_SIZE]; + + ip_to_str(c_id.get_ip1(), ip1_str, sizeof(ip1_str)); + ip_to_str(c_id.get_ip2(), ip2_str, sizeof(ip2_str)); snprintf(str, max_str_len, "(src host %s and dst host %s and dst port %d) or " @@ -326,7 +349,7 @@ void ConnectionIF3::getBPFStr(char *str, int max_str_len) const { **********************************************************************/ // Static Member initialization std::string ConnectionIF2::pattern_connection2 = - "\\s*" + pattern_ip + "\\s+-?\\s*" + pattern_ip + "\\s*"; + "\\s*" + pattern_ip + "\\s+" + pattern_ip + "\\s*"; RE2 ConnectionIF2::re(ConnectionIF2::pattern_connection2); std::list @@ -352,12 +375,16 @@ IndexField* ConnectionIF2::parseQuery(const char *query) { void ConnectionIF2::getBPFStr(char *str, int max_str_len) const { - const char *ip1_str = c_id.get_ip1()->AsString().c_str(); - const char *ip2_str = c_id.get_ip2()->AsString().c_str(); + char s_ip_str[TM_IP_STR_SIZE]; + char d_ip_str[TM_IP_STR_SIZE]; + + ip_to_str(c_id.get_ip1(), s_ip_str, sizeof(s_ip_str)); + ip_to_str(c_id.get_ip2(), d_ip_str, sizeof(d_ip_str)); + snprintf(str, max_str_len, "host %s and host %s", - ip1_str, ip2_str); + s_ip_str, d_ip_str); } diff --git a/src/IndexField.hh b/src/IndexField.hh index 674244b..7be649c 100644 --- a/src/IndexField.hh +++ b/src/IndexField.hh @@ -4,14 +4,11 @@ #include #include #include -#include -#include #include #include "re2/re2.h" #include "types.h" -#include "IPAddr.hh" #include "packet_headers.h" class IndexField; @@ -66,43 +63,25 @@ class SrcIPAddress; class DstIPAddress; class IPAddress: public IndexField { public: - IPAddress(uint32_t ip) { - addr = IPAddr(IPv4, &ip, IPAddr::Network); - } - IPAddress(const char* s) { - addr = IPAddr(s); - } + IPAddress(uint32_t ip): ip_address(ip) {} + IPAddress(const char* s): ip_address(inet_addr(s)) {} IPAddress(void *p) { memcpy((void*)getConstKeyPtr(), p, getKeySize()); } virtual ~IPAddress() {}; virtual uint32_t hash() const { // TODO: initval - const uint32_t *bytes; - int len = addr.GetBytes(&bytes); - if ( len == 1 ) - return hash1words(bytes[0], 0); - else - // TODO: this is only hashing the latter 96bits of the address. - return hash3words(bytes[1], bytes[2], bytes[3], 0); + return hash1words(ip_address, 0); } virtual uint32_t getInt() const { - const uint32_t *bytes; - int len = addr.GetBytes(&bytes); - if ( len == 1 ) - return bytes[3]; - else - return bytes[3]; + return ip_address; } virtual const char* getConstKeyPtr() const { - const uint32_t *bytes; - addr.GetBytes(&bytes); - return (const char*) bytes; + return (const char*)&ip_address; } + // char* getKeyPtr() { return (char*)&ip_address; } virtual const int getKeySize() const { - const uint32_t *bytes; - int len = addr.GetBytes(&bytes); - return len*4; + return sizeof(ip_address); } virtual void getStr(char* s, int maxsize) const; virtual std::string getStr() const; @@ -112,7 +91,6 @@ public: static const std::string getIndexNameStatic() { return "ip"; } - static std::list genKeys(const u_char* packet); static int keysPerPacket() { return 2; @@ -124,8 +102,7 @@ public: static IndexField* parseQuery(const char *query); virtual void getBPFStr(char *, int) const; private: - IPAddr addr; - + uint32_t ip_address; static std::string pattern; static RE2 re; }; diff --git a/src/cmd_scanner.ll b/src/cmd_scanner.ll index d554a45..51f0103 100644 --- a/src/cmd_scanner.ll +++ b/src/cmd_scanner.ll @@ -25,8 +25,7 @@ DBL -?[0-9]*\.[0-9]* WHITE [ \t]+ COMMENT \#.* NEWLINE \n -HEX [0-9a-fA-F]+ -IP ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+|"["({HEX}:){7}{HEX}"]")|("["0x{HEX}({HEX}|:)*"::"({HEX}|:)*"]")|("["({HEX}|:)*"::"({HEX}|:)*"]")|("["({HEX}|:)*"::"({HEX}|:)*({INT}"."){3}{INT}"]") +IP [0-9]+\.[0-9]+\.[0-9]+\.[0-9]+ %% "{" return LBRACE; diff --git a/src/types.h b/src/types.h index 7362d13..8163802 100644 --- a/src/types.h +++ b/src/types.h @@ -23,6 +23,4 @@ typedef uint8_t proto_t; //typedef double hash_t; -typedef enum { IPv4, IPv6 } IPFamily; - #endif From seth at icir.org Fri Jan 25 09:03:41 2013 From: seth at icir.org (Seth Hall) Date: Fri, 25 Jan 2013 09:03:41 -0800 Subject: [TM] [git] branch 'topic/seth/ipv6' created Message-ID: <201301251703.r0PH3fc3024220@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine New branch : topic/seth/ipv6 Referencing: 1d269b3842611cf0bd88efa15275ad2f1e07cceb From seth at icir.org Fri Jan 25 09:56:48 2013 From: seth at icir.org (Seth Hall) Date: Fri, 25 Jan 2013 09:56:48 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Config file updates. (88ae58b) Message-ID: <201301251756.r0PHum67008669@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit 88ae58b70f6654ab36ae5e20fedf750615edbffc Author: Seth Hall Date: Fri Jan 25 12:56:58 2013 -0500 Config file updates. - It's renamed to be more in line with other Bro-universe configs (broctl) - It's actually installed now. >--------------------------------------------------------------- 88ae58b70f6654ab36ae5e20fedf750615edbffc CMakeLists.txt | 11 +++++++++++ configure | 15 +++++++++++---- doc/howto.rst | 6 +++--- etc/{timemachine.conf => timemachine.cfg} | 9 +++++---- src/main.cc | 2 +- tm-query/README | 2 +- 6 files changed, 32 insertions(+), 13 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 48009e5..efbc8c1 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -4,6 +4,9 @@ include(cmake/CommonCMakeConfig.cmake) file(STRINGS "${CMAKE_CURRENT_SOURCE_DIR}/VERSION" VERSION LIMIT_COUNT 1) +set(PREFIX "${CMAKE_INSTALL_PREFIX}") +set(ETC "${BRO_ETC_INSTALL_DIR}") + ######################################################################## ## Dependency Configuration @@ -44,6 +47,14 @@ include_directories(BEFORE ${CMAKE_CURRENT_BINARY_DIR}) add_subdirectory(src) + +include(InstallPackageConfigFile) + +InstallPackageConfigFile( + ${CMAKE_CURRENT_SOURCE_DIR}/etc/timemachine.cfg + ${ETC} + timemachine.cfg) + ######################################################################## ## Build Summary diff --git a/configure b/configure index 8c27e62..a412a44 100755 --- a/configure +++ b/configure @@ -21,6 +21,7 @@ Usage: $0 [OPTION]... [VAR=VALUE]... Installation Directories: --prefix=PREFIX installation directory [/usr/local] + --conf-files-dir=DIR config files installation directory [PREFIX/etc] Optional Features: --enable-debug compile in debugging mode @@ -53,8 +54,9 @@ append_cache_entry () { # set defaults builddir=build +prefix=/usr/local CMakeCacheEntries="" -append_cache_entry CMAKE_INSTALL_PREFIX PATH /usr/local +append_cache_entry CMAKE_INSTALL_PREFIX PATH $prefix append_cache_entry ENABLE_DEBUG BOOL false # parse arguments @@ -75,15 +77,16 @@ while [ $# -ne 0 ]; do --prefix=*) append_cache_entry CMAKE_INSTALL_PREFIX PATH $optarg ;; + --conf-files-dir=*) + append_cache_entry BRO_ETC_INSTALL_DIR PATH $optarg + user_set_conffilesdir="true" + ;; --enable-debug) append_cache_entry ENABLE_DEBUG BOOL true ;; --with-broccoli=*) append_cache_entry Broccoli_ROOT_DIR PATH $optarg ;; - --with-pcapnav=*) - append_cache_entry PCAPNAV_ROOT_DIR PATH $optarg - ;; --with-pcap=*) append_cache_entry PCAP_ROOT_DIR PATH $optarg ;; @@ -101,6 +104,10 @@ while [ $# -ne 0 ]; do shift done +if [ "$user_set_conffilesdir" != "true" ]; then + append_cache_entry BRO_ETC_INSTALL_DIR PATH $prefix/etc +fi + if [ -d $builddir ]; then # If build directory exists, check if it has a CMake cache if [ -f $builddir/CMakeCache.txt ]; then diff --git a/doc/howto.rst b/doc/howto.rst index e4002a6..3a87e76 100644 --- a/doc/howto.rst +++ b/doc/howto.rst @@ -18,14 +18,14 @@ a number of user configuration options. This includes general options such as logfile names, capture device settings as well as the configuration of storage classes with the respective parameters. -By default, the configuration file is called 'timemachine.conf' and is read +By default, the configuration file is called 'timemachine.cfg' and is read from the current working directory. This can be overridden by the -c command line option (see `Command Line Options`_ section below). The configuration file is structured in sections. Options for a section are grouped by curly brackets and are separated by semicolons. There is exactly one main section, and a number of class configuration -sections. For better orientation, see the example timemachine.conf file +sections. For better orientation, see the example timemachine.cfg file in the timemachine package. Main section options @@ -345,6 +345,6 @@ line options override the according configuration file settings. -f Apply global BPF filter. Cf. 'filter' configuration directive in `Configuration File`_ section. -c Read configuration file (see `Configuration File`_ section - above) from specified file rather than from timemachine.conf + above) from specified file rather than from timemachine.cfg in the current working directory, which is the default. diff --git a/etc/timemachine.conf b/etc/timemachine.cfg similarity index 93% rename from etc/timemachine.conf rename to etc/timemachine.cfg index ed2465e..fa85cb1 100644 --- a/etc/timemachine.conf +++ b/etc/timemachine.cfg @@ -2,14 +2,17 @@ main { ## The device to sniff. - device "en2"; + device "eth0"; ## Directories for packet captures, logs, and indexes. ## These directories must exist when timemachine starts! - workdir "/tmp/tm"; + workdir "/var/timemachine"; indexdir "indexes/"; queryfiledir "queries/"; + ## Name for the log to be stored in + logfile "timemachine.log"; + ## Set this value to '1' if you'd like timemachine to ## start in daemon mode. daemon 0; @@ -17,8 +20,6 @@ main { ## MUST be set to '0' console 1; - ## Name for the log. - logfile "timemachine.log"; ## How often in seconds timemachine should log statistical ## information. diff --git a/src/main.cc b/src/main.cc index 4bb123d..1c8e7fc 100644 --- a/src/main.cc +++ b/src/main.cc @@ -589,7 +589,7 @@ void usage() { int main(int argc, char** argv) { - const char *conffile="timemachine.conf"; + const char *conffile="timemachine.cfg"; struct sigaction exit_action; int i; inet_aton("127.0.0.1", &conf_main_rmtconsole_listen_addr); diff --git a/tm-query/README b/tm-query/README index 577f0a6..4b9e243 100644 --- a/tm-query/README +++ b/tm-query/README @@ -17,7 +17,7 @@ The general usage is If the TM is running on a port other than 47757, you can append ":" to "". "" is the name of the destination file into which query matches will be written; the files will be -created in the TM's query directory as configured in "timemachine.conf". +created in the TM's query directory as configured in "timemachine.cfg". Currently, tm-query supports the following query types to extract traffic From seth at icir.org Fri Jan 25 13:41:54 2013 From: seth at icir.org (Seth Hall) Date: Fri, 25 Jan 2013 13:41:54 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Find the Thread packet through cmake. (3e6aaba) Message-ID: <201301252141.r0PLfsEa020767@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit 3e6aaba5ac84e909693433cf6f98f1e7b61d31fb Author: Seth Hall Date: Fri Jan 25 16:42:10 2013 -0500 Find the Thread packet through cmake. >--------------------------------------------------------------- 3e6aaba5ac84e909693433cf6f98f1e7b61d31fb CMakeLists.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/CMakeLists.txt b/CMakeLists.txt index efbc8c1..e429143 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -16,6 +16,7 @@ FindRequiredPackage(FLEX) FindRequiredPackage(BISON) FindRequiredPackage(PCAP) FindRequiredPackage(Readline) +FindRequiredPackage(Threads) # Optional Dependencies From seth at icir.org Fri Jan 25 18:10:57 2013 From: seth at icir.org (Seth Hall) Date: Fri, 25 Jan 2013 18:10:57 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Turned off and fixed some gcc compiler warnings. (76885a5) Message-ID: <201301260210.r0Q2AvuZ030077@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit 76885a5b24bb54df2e657f228c916d1d82993568 Author: Seth Hall Date: Fri Jan 25 21:10:59 2013 -0500 Turned off and fixed some gcc compiler warnings. >--------------------------------------------------------------- 76885a5b24bb54df2e657f228c916d1d82993568 src/FifoDisk.cc | 2 +- src/re2/CMakeLists.txt | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/FifoDisk.cc b/src/FifoDisk.cc index de8ce65..61ffeac 100644 --- a/src/FifoDisk.cc +++ b/src/FifoDisk.cc @@ -71,7 +71,7 @@ void FifoDisk::addPkt(const pkt_ptr p) { files.back()->getCurFileSize() + sizeof(struct pcap_file_header) + sizeof(struct pcap_pkthdr) - + ((struct pcap_pkthdr*)p)->caplen > (int)file_size) { /* Why do we have to be THAT precise?!?!? */ + + ((struct pcap_pkthdr*)p)->caplen > file_size) { /* Why do we have to be THAT precise?!?!? */ // Do not add or delete files while a query is in progress, because // the file iterator of the query might get fucked up. // XXX: This my starve the rotation of files or generate files that diff --git a/src/re2/CMakeLists.txt b/src/re2/CMakeLists.txt index d575939..594aa9d 100644 --- a/src/re2/CMakeLists.txt +++ b/src/re2/CMakeLists.txt @@ -1,3 +1,6 @@ +# GCC is giving warnings about this. +set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-sign-compare") + include_directories(BEFORE ${CMAKE_CURRENT_BINARY_DIR}) include_directories(${CMAKE_CURRENT_SOURCE_DIR} @@ -72,6 +75,3 @@ if (UNIX) set_target_properties(re2Static PROPERTIES OUTPUT_NAME re2) endif () target_link_libraries(re2Static) -#install(TARGETS re2Static DESTINATION lib) - -#install(FILES ${CMAKE_CURRENT_BINARY_DIR}/broccoli.h DESTINATION include) From seth at icir.org Fri Jan 25 18:55:23 2013 From: seth at icir.org (Seth Hall) Date: Fri, 25 Jan 2013 18:55:23 -0800 Subject: [TM] [git] topic/seth/cmake-changeover: Small improvement to config file handling. (1263dff) Message-ID: <201301260255.r0Q2tNGT032313@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/seth/cmake-changeover >--------------------------------------------------------------- commit 1263dffaa975edeed65880a702db5ca2d24cb249 Author: Seth Hall Date: Fri Jan 25 21:55:26 2013 -0500 Small improvement to config file handling. - timemachine.cfg is looked for in the config file location now instead of CWD. >--------------------------------------------------------------- 1263dffaa975edeed65880a702db5ca2d24cb249 CMakeLists.txt | 2 ++ doc/howto.rst | 8 ++++---- src/CMakeLists.txt | 5 ++++- src/{main.cc => main.cc.in} | 8 +++----- 4 files changed, 13 insertions(+), 10 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index e429143..c6f8f54 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -73,6 +73,8 @@ message( "\n==================| TimeMachine Build Summary |====================" "\n" "\nInstall prefix: ${timemachine_install_summary}" + "\nConfig File Dir: ${BRO_ETC_INSTALL_DIR}" + "\nDebug mode: ${ENABLE_DEBUG}" "\n" "\nCC: ${CMAKE_C_COMPILER}" diff --git a/doc/howto.rst b/doc/howto.rst index 3a87e76..58f1cc7 100644 --- a/doc/howto.rst +++ b/doc/howto.rst @@ -19,8 +19,8 @@ such as logfile names, capture device settings as well as the configuration of storage classes with the respective parameters. By default, the configuration file is called 'timemachine.cfg' and is read -from the current working directory. This can be overridden by the -c -command line option (see `Command Line Options`_ section below). +from the config file installation directory. This can be overridden by +the -c command line option (see `Command Line Options`_ section below). The configuration file is structured in sections. Options for a section are grouped by curly brackets and are separated by semicolons. @@ -345,6 +345,6 @@ line options override the according configuration file settings. -f Apply global BPF filter. Cf. 'filter' configuration directive in `Configuration File`_ section. -c Read configuration file (see `Configuration File`_ section - above) from specified file rather than from timemachine.cfg - in the current working directory, which is the default. + above) from specified file rather than from the default + file. diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 22afc16..9021cdf 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -15,6 +15,9 @@ add_flex_bison_dependency(CONFScanner CONFParser) include_directories(${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_SOURCE_DIR}/..) +configure_file(${CMAKE_CURRENT_SOURCE_DIR}/main.cc.in + ${CMAKE_CURRENT_BINARY_DIR}/main.cc) + set(timemachine_SRCS ${BISON_CMDParser_INPUT} ${FLEX_CMDScanner_INPUT} @@ -59,7 +62,7 @@ set(timemachine_SRCS Storage.hh conf.h jhash3.h - main.cc + ${CMAKE_CURRENT_BINARY_DIR}/main.cc packet_headers.h rmtconsole.cc tm.h diff --git a/src/main.cc b/src/main.cc.in similarity index 98% rename from src/main.cc rename to src/main.cc.in index 1c8e7fc..00582db 100644 --- a/src/main.cc +++ b/src/main.cc.in @@ -304,8 +304,6 @@ void *cli_console_thread(void *arg) { /* char buf[8192]; */ char *line, *lastline=NULL; // char *opt[2]; - - char prompt[64]; snprintf(prompt, 63, "tm@%s# ", hostname); @@ -432,7 +430,6 @@ void *statisticslog_thread(void* arg) { if (last_call_ts.tv_sec) { dt=to_tm_time(&now) - to_tm_time(&last_call_ts); - utime_perc = 100.*(ru_utime-last_ru_utime)/dt; stime_perc = 100.*(ru_stime-last_ru_stime)/dt; @@ -577,7 +574,8 @@ void *statisticslog_thread(void* arg) { */ void usage() { - fprintf(stderr, "usage: timemachine[-i interface] [-r file] [-f filter] [-c config file]\n"); + fprintf(stderr, "usage: timemachine [-i interface] [-r file] [-f filter] [-c config file]\n"); + fprintf(stderr, " default config file: @BRO_ETC_INSTALL_DIR@/timemachine.cfg\n"); exit(1); } @@ -589,7 +587,7 @@ void usage() { int main(int argc, char** argv) { - const char *conffile="timemachine.cfg"; + const char *conffile="@BRO_ETC_INSTALL_DIR@/timemachine.cfg"; struct sigaction exit_action; int i; inet_aton("127.0.0.1", &conf_main_rmtconsole_listen_addr); From time-machine at ICSI.Berkeley.EDU Sat Jan 26 18:16:16 2013 From: time-machine at ICSI.Berkeley.EDU (Time Machine Tracker) Date: Sun, 27 Jan 2013 02:16:16 -0000 Subject: [TM] [The Time Machine] Password reset for user: seth Message-ID: <201301270216.r0R2GGWT010636@bro-ids.icir.org> Password reset for user for user seth -- The Time Machine High-volume network traffic stream recorder. From time-machine at ICSI.Berkeley.EDU Sat Jan 26 18:21:52 2013 From: time-machine at ICSI.Berkeley.EDU (Time Machine Tracker) Date: Sun, 27 Jan 2013 02:21:52 -0000 Subject: [TM] #11: REST API Message-ID: <052.cd51c4683b25fdc776971a1b567b0d49@icsi.berkeley.edu> #11: REST API ------------------------+------------------ Reporter: seth | Owner: Type: enhancement | Status: new Priority: blocker | Version: master Keywords: | ------------------------+------------------ TimeMachine should probably move away from the existing console based query interface and toward an HTTP based REST-style API eventually. -- Ticket URL: The Time Machine High-volume network traffic stream recorder. From time-machine at ICSI.Berkeley.EDU Sat Jan 26 18:23:13 2013 From: time-machine at ICSI.Berkeley.EDU (Time Machine Tracker) Date: Sun, 27 Jan 2013 02:23:13 -0000 Subject: [TM] #12: Test suite Message-ID: <052.90c83e3aca46725a7061327f3fb642e8@icsi.berkeley.edu> #12: Test suite --------------------+--------------- Reporter: seth | Owner: Type: task | Status: new Priority: blocker | Version: Keywords: | --------------------+--------------- A btest based test suite needs to be created for timemachine. -- Ticket URL: The Time Machine High-volume network traffic stream recorder. From time-machine at ICSI.Berkeley.EDU Mon Jan 28 09:11:22 2013 From: time-machine at ICSI.Berkeley.EDU (Time Machine Tracker) Date: Mon, 28 Jan 2013 17:11:22 -0000 Subject: [TM] #13: Crash due to fprintf? Message-ID: <052.1515898bd78c85256d75ceee193a7282@icsi.berkeley.edu> #13: Crash due to fprintf? --------------------+---------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Low | Version: git/master Keywords: | --------------------+---------------------- You can recreate this crash by connecting to the remote console (port 42042) repeatedly and running the command "show conn sample" then disconnecting. {{{ Thread 11 (process 86162): #0 0x00007fff8b1776fe in write$NOCANCEL () #1 0x00007fff8e491857 in _swrite () #2 0x00007fff8e48cad8 in __sflush () #3 0x00007fff8e48f533 in __sfvwrite () #4 0x00007fff8e49cb33 in __vfprintf () #5 0x00007fff8e498337 in vfprintf_l () #6 0x00007fff8e48dcbf in fprintf () #7 0x0000000100004106 in cmderror (msg=0x10004bd5e "Illegal character in command string") at cmd_parser.yy:437 #8 0x00000001000045b7 in cmdlex () at cmd_scanner.ll:103 #9 0x0000000100002b63 in cmdparse () at cmd_parser.cc:1452 #10 0x0000000100004246 in parse_cmd (cmd=, outfp=0x7fff742d0300, s=0x10040a6e0, thread=0x0) at cmd_parser.yy:464 #11 0x0000000100021580 in rmtconsole_worker_thread () at rmtconsole.cc:59 #12 0x00007fff8e424742 in _pthread_start () #13 0x00007fff8e411181 in thread_start () }}} -- Ticket URL: The Time Machine High-volume network traffic stream recorder. From time-machine at ICSI.Berkeley.EDU Mon Jan 28 09:16:09 2013 From: time-machine at ICSI.Berkeley.EDU (Time Machine Tracker) Date: Mon, 28 Jan 2013 17:16:09 -0000 Subject: [TM] #13: Crash due to fprintf? In-Reply-To: <052.1515898bd78c85256d75ceee193a7282@icsi.berkeley.edu> References: <052.1515898bd78c85256d75ceee193a7282@icsi.berkeley.edu> Message-ID: <067.feb4d76d130b5ce95563f87cbaea74eb@icsi.berkeley.edu> #13: Crash due to fprintf? ----------------------+----------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Low | Version: git/master Resolution: | Keywords: ----------------------+----------------------- Description changed by seth: Old description: > You can recreate this crash by connecting to the remote console (port > 42042) repeatedly and running the command "show conn sample" then > disconnecting. > > {{{ > Thread 11 (process 86162): > #0 0x00007fff8b1776fe in write$NOCANCEL () > #1 0x00007fff8e491857 in _swrite () > #2 0x00007fff8e48cad8 in __sflush () > #3 0x00007fff8e48f533 in __sfvwrite () > #4 0x00007fff8e49cb33 in __vfprintf () > #5 0x00007fff8e498337 in vfprintf_l () > #6 0x00007fff8e48dcbf in fprintf () > #7 0x0000000100004106 in cmderror (msg=0x10004bd5e "Illegal character in > command string") at cmd_parser.yy:437 > #8 0x00000001000045b7 in cmdlex () at cmd_scanner.ll:103 > #9 0x0000000100002b63 in cmdparse () at cmd_parser.cc:1452 > #10 0x0000000100004246 in parse_cmd (cmd= due to optimizations>, outfp=0x7fff742d0300, s=0x10040a6e0, thread=0x0) > at cmd_parser.yy:464 > #11 0x0000000100021580 in rmtconsole_worker_thread () at rmtconsole.cc:59 > #12 0x00007fff8e424742 in _pthread_start () > #13 0x00007fff8e411181 in thread_start () > }}} New description: You can recreate this crash by connecting to the remote console (port 42042) repeatedly and running the command "show conn sample" then disconnecting. {{{ Program received signal SIGPIPE, Broken pipe. 0x00007fff8b177552 in __sigsuspend () Thread 11 (process 86162): #0 0x00007fff8b1776fe in write$NOCANCEL () #1 0x00007fff8e491857 in _swrite () #2 0x00007fff8e48cad8 in __sflush () #3 0x00007fff8e48f533 in __sfvwrite () #4 0x00007fff8e49cb33 in __vfprintf () #5 0x00007fff8e498337 in vfprintf_l () #6 0x00007fff8e48dcbf in fprintf () #7 0x0000000100004106 in cmderror (msg=0x10004bd5e "Illegal character in command string") at cmd_parser.yy:437 #8 0x00000001000045b7 in cmdlex () at cmd_scanner.ll:103 #9 0x0000000100002b63 in cmdparse () at cmd_parser.cc:1452 #10 0x0000000100004246 in parse_cmd (cmd=, outfp=0x7fff742d0300, s=0x10040a6e0, thread=0x0) at cmd_parser.yy:464 #11 0x0000000100021580 in rmtconsole_worker_thread () at rmtconsole.cc:59 #12 0x00007fff8e424742 in _pthread_start () #13 0x00007fff8e411181 in thread_start () Thread 1 (process 86162): #0 0x00007fff8b177552 in __sigsuspend () #1 0x00007fff8e4ae57a in pause () #2 0x0000000100021105 in main (argc=, argv=) at main.cc:804 }}} -- -- Ticket URL: The Time Machine High-volume network traffic stream recorder. From time-machine at ICSI.Berkeley.EDU Mon Jan 28 10:52:12 2013 From: time-machine at ICSI.Berkeley.EDU (Time Machine Tracker) Date: Mon, 28 Jan 2013 18:52:12 -0000 Subject: [TM] #13: Crash due to fprintf? In-Reply-To: <052.1515898bd78c85256d75ceee193a7282@icsi.berkeley.edu> References: <052.1515898bd78c85256d75ceee193a7282@icsi.berkeley.edu> Message-ID: <067.37072815522a014408fab9f33b995d96@icsi.berkeley.edu> #13: Crash due to fprintf? ----------------------+----------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Low | Version: git/master Resolution: | Keywords: ----------------------+----------------------- Comment (by seth): To **really** replicate the crash you just need to connect to the command socket (42042) and send a ctrl-q, then disconnect. That will immediately crash the timemachine process. -- Ticket URL: The Time Machine High-volume network traffic stream recorder. From time-machine at ICSI.Berkeley.EDU Tue Jan 29 09:52:34 2013 From: time-machine at ICSI.Berkeley.EDU (Time Machine Tracker) Date: Tue, 29 Jan 2013 17:52:34 -0000 Subject: [TM] #14: Failed assertion resulting in termination Message-ID: <052.3a419b8bf7f91e7e75bb99a4c58986c6@icsi.berkeley.edu> #14: Failed assertion resulting in termination --------------------+---------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Medium | Version: git/master Keywords: | --------------------+---------------------- I used the to_file query result to try and write to a non-existent directory with the query sent from Bro. It seemed like the assertion failure happened when I terminated Bro. Assertion failed: (is_open==true), function addPkt, file /Users/seth/bro /time-machine/src/FifoDisk.cc, line 184. -- Ticket URL: The Time Machine High-volume network traffic stream recorder.