From asharma at lbl.gov Mon Jun 15 17:09:27 2015 From: asharma at lbl.gov (Aashish Sharma) Date: Mon, 15 Jun 2015 17:09:27 -0700 Subject: [TM] [git/tm] topic/aashish/ipv6's head updated: This is my final commit before leaving lab for this year. (f984098) Message-ID: <201506160009.t5G09Rw1008444@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine Branch 'topic/aashish/ipv6' now includes: 29aa931 just a test file to commit be967c5 some code review 256b528 started some IPv6 implementation. Cannot display all the ipv6 packets yet, but does display the full ipv6 addresses in the class files. d70ea95 started implementation of hash function from bro. It crashes early still and can loop forever. This commit is only to save work so far. 5900dc0 fixed the silly bug in the lookup function (called itself within itself) and attempted a fix on the deletion of the entries in old hash table. It seems to run to completion, but I have not checked rigorously yet. Index files in the indexes directory don't seem to be correct. The class files may not be entirely correct either, seems to only get most of the packets rather than nearly all of the packets. I am committing because this seems to run without segfault e047c0b Some querying is working (with some adjust for VLAN tags) However, it does not take care of MPLS labels Also, the check for hash index conflicts has not been done The check that full sessions are being taken into account has not been done either 56b3e91 I seemed to have removed the major memory leaks. There are still some memory leaks, but they seem to come from the original Time Machine code, namely with the connections. d3183a0 It does not seem to seg fault anymore. I removed the bug where I overdeleted (did delete [] instead of delete) 7b03c40 made taking care of hash collisions more explicit 155a210 fixed some bugs mainly in Connection.cc and IndexField.cc runs with less dropped packets, still a sizeable amount of dropped packets though (10% with indexes) runs with less cpu usage, still rather large cpu usage (120% with indexes) f356165 making this commit to save latest work. This commit attempted to fix the Hash bugs cba1493 deleted two unnecessary files and made a change in IndexHash.cc to help avoid a segfault da9bca8 added some gperftools capabilities (taken from bro code) 9646fd4 added the foundation in the CMake for cpu profiling via gperftools 070782b saving my work to try to get rid of permissions 737c7e9 updated the foundation for the gperftools, cpu profiler 1ed43c2 gperftools with CPU profiler seems to be working. You can put the ProfilerStart() and ProfilerStop() wherever you want, but be wary that putting them relatively close to each other may result in 0 samples. f550318 Fixed one very hidden bug (in ConnectionIF3, IndexField.cc) 325409e removed tmlog by commenting them out (to better cpu performance) 5854443 Updated the README to include some instructions about using gperftools' cpu profiler d0e18fe Changed the key struct in Connection.cc and Connection.hh to be less convoluted. 00557be Fixed a small bug in IndexField.hh (had to do with memcpy), and tried to add some support for Mac OS 6d6db31 Removed the many instances of gettimeofday and used a counter instead to avoid system calls (Aashish's idea) 74b8a32 Changed the format of the configuration file to allow the class files to be placed in different directories based on their bucket type, a suggestion made by Partha Also fixed querying for IPv4 for conn2, conn3, and conn4. Please note that querying for IPv6 for conn2, conn3, conn4 is currently not working. fdf52f7 Fixed the implementation for querying for IPv6 for conn2/conn3/conn4 Fixed the regular expression so that it is more code friendly (less matching arguments needed). 40329f8 Fixed a bug in querying for longer IPv6 addresses. The regular expression for IPv6 addresses has been corrected 6fe29c1 Fixed getStr() method for IPv4 0175f2b Changed data structure for treating IPv4 addresses in Connection.hh for conn4 only. This change was made to help compete with original TM's simple comparisons for IPv4 addresses I will change it for conn3 and conn2 next. Also commented out unnecessary debug statements which take some CPU usage. 47cb41d Added the data structure for IPv4 for conn2 and conn3 to help compete with the original TM's method of comparing ip addresses c41a7a1 Got rid of some unnecessary variables and warnings about multi-line comments 01d751e Commented out some printf debugging statements, and updated version number. 418a79a Implemented Jim's Precedence. It improves CPU usage slightly. Basically, the buckets are sorted by precedence after the config file is parsed. Then, the packets go through the buckets and break when it meets a match. a6f204b Implemented Partha's indexdir and queryfiledir changes. Basically, if you forget to create the index and query directories, but they are on the config file, they will be created automatically, with a message that lets you know. ed402cd Created Aashish's -v command line parameter which outputs the version number of Time Machine, and also updated the version number to 2-0 b08480e Added a profilepath option in the configuration file to place the gperftools cpu profiler's .prof file, if wanted Hopefully corrected the code for Apple compilers (I don't have an Apple compiler). a29e64b Hopefully fixed the Apple compiler issues (I do not have an Apple compiler) Added some comments to code and change log 97e862b Updated some comments about prime number hash sizes and the use of the number in the counter when trying to write indxes to disk. f5f4c9d Fixed a Memory-Illegal access error found by Coverity. This error occurred in Index.cc, and was in the original tm-master code (the ipv4 only implementation). Basically, iqe had a chance to be deleted, and then was to be accessed after that. f984098 This is my final commit before leaving lab for this year. From asharma at lbl.gov Mon Jun 15 17:11:13 2015 From: asharma at lbl.gov (Aashish Sharma) Date: Mon, 15 Jun 2015 17:11:13 -0700 Subject: [TM] [git/tm] topic/aashish/ipv6: fixed vlan tag use with packet (72a32f8) Message-ID: <201506160011.t5G0BKar008560@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/aashish/ipv6 >--------------------------------------------------------------- commit 72a32f8b3e2be246c172e9b7d982b3a348ac2608 Author: Aashish Sharma Date: Mon Jun 15 17:11:13 2015 -0700 fixed vlan tag use with packet >--------------------------------------------------------------- 72a32f8b3e2be246c172e9b7d982b3a348ac2608 src/Storage.cc | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/Storage.cc b/src/Storage.cc index 8e57c12..491e1bf 100644 --- a/src/Storage.cc +++ b/src/Storage.cc @@ -390,6 +390,8 @@ void Storage::addPkt(const struct pcap_pkthdr *header, idxpacket += 4; // Skip the vlan header + packet += 4; // Skip the vlan header + //pkt_hdr_size = 0; // Check for 802.1ah (Q-in-Q) containing IP. @@ -397,8 +399,12 @@ void Storage::addPkt(const struct pcap_pkthdr *header, // stripping because there is no // specification that allows for deeper // nesting. + if ( ((idxpacket[2] << 8) + idxpacket[3]) == 0x0800 ) + { idxpacket += 4; + packet += 4; + } } // TODO: Cannot handle MPLS labels just yet