From leaf.pan at yahoo.com Thu Aug 9 05:09:55 2018 From: leaf.pan at yahoo.com (leaf pan) Date: Thu, 9 Aug 2018 12:09:55 +0000 (UTC) Subject: [TM] [Timemachine]cannot query previous data after timemachine restarting? References: <242106371.4356820.1533816595757.ref@mail.yahoo.com> Message-ID: <242106371.4356820.1533816595757@mail.yahoo.com> Dear Team, Sorry for bothering you:) I?ve launched timemachine in my server for one week and itworked well. But yesterday after my server(timemachine) rebooting, Ifound the data captured before server rebooting cannot be queried anymore. I tried many queries with parameters ?IP? or ?Connection?,the result always only contain the data captured from the timeserver/timemachine restarting. ? Could anyone tell if timemachine have this limitation? Orhow to work around? ? Thanks BR, Leaf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/time-machine/attachments/20180809/d573a376/attachment.html From asharma at lbl.gov Tue Aug 14 12:16:11 2018 From: asharma at lbl.gov (Aashish Sharma) Date: Tue, 14 Aug 2018 12:16:11 -0700 Subject: [TM] [Timemachine]cannot query previous data after timemachine restarting? In-Reply-To: <242106371.4356820.1533816595757@mail.yahoo.com> References: <242106371.4356820.1533816595757.ref@mail.yahoo.com> <242106371.4356820.1533816595757@mail.yahoo.com> Message-ID: <20180814191610.GF8565@MacPro-2331.local> Leaf, Greetings! [ Sorry for the delayed response ] What you bring is a very well know issue with time-machine. Indexes aren't persistent. So once you restart TM you'd loose the query capabilities. Sadly! My understanding is that indexes are implemented as AVL trees. Those are kept in memory and on the disk, but to be able re-read those after a new start is tricky. I know (I think) that Scott Sakai from SDSC has used some postgres/sqlite backend to store indexes. Its specific to Scott's setup so never pushed upstream to git repo. We/LBNL just use workarounds: 1) A virtual timestamp based directory structure called TMquick: Here it how it looks (basically a human readable timestamps appended to buckets. [/TMquick/TODAY]$ ls all-00:12:58 all-02:14:05 all-05:47:02 all-08:34:39 all-10:18:53 all-11:49:10 dns-03:04:48 dns-04:18:14 dns-06:30:58 dns-09:00:20 dns-10:36:25 dns-12:06:44 smtp-03:10:51 smtp-08:07:32 smtp-09:38:38 smtp-11:11:10 ssh-02:01:37 2) We have a tm-extract.sh script which if you give a bro log entry (conn, dns, http, smtp etc), will go and find the right TM-bucket and extract the pcaps. (1) is used by humans (2) is used by robot-controlled scripts. For (1) If interested, I can share the TMquick scripts (Partha has ownership of those) For (2) see extract-tm.sh here: https://github.com/initconf/timemachine-conf-scripts.git Let me know if you have followup questions. Aashish ps - if you are not already use topic/aashish/ipv6 branch - its stable, has IPv6 support and good - Its Naoki Eto's branch with some little modifications related to vlan tag stripping and FreeBSD 11 compilations. Aashish On Thu, Aug 09, 2018 at 12:09:55PM +0000, leaf pan wrote: > > Dear Team, > > Sorry for bothering you:) > > I?ve launched timemachine in my server for one week and itworked well. > > But yesterday after my server(timemachine) rebooting, Ifound the data captured before server rebooting cannot be queried anymore. > > I tried many queries with parameters ?IP? or ?Connection?,the result always only contain the data captured from the timeserver/timemachine restarting. > > ? > > Could anyone tell if timemachine have this limitation? Orhow to work around? > > ? > > Thanks > > BR, > > Leaf > _______________________________________________ > Time-Machine mailing list > Time-Machine at mailman.ICSI.Berkeley.EDU > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/time-machine From asharma at lbl.gov Tue Aug 14 16:03:32 2018 From: asharma at lbl.gov (Aashish Sharma) Date: Tue, 14 Aug 2018 16:03:32 -0700 Subject: [TM] [git/tm] topic/aashish/ipv6: now compiles with FreeBSD 11 (351ea23) Message-ID: <201808142314.w7ENEVxE015028@bro-ids.icir.org> Repository : ssh://git at bro-ids.icir.org/time-machine On branch : topic/aashish/ipv6 >--------------------------------------------------------------- commit 351ea23cb189cf3d3563176af2e81c77701bb708 Author: Aashish Sharma Date: Tue Aug 14 16:03:32 2018 -0700 now compiles with FreeBSD 11 >--------------------------------------------------------------- 351ea23cb189cf3d3563176af2e81c77701bb708 CMakeLists.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CMakeLists.txt b/CMakeLists.txt index 83a4457..ea37d0d 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -18,6 +18,9 @@ FindRequiredPackage(PCAP) FindRequiredPackage(Readline) FindRequiredPackage(Threads) +include_directories(BEFORE ${Readline_INCLUDE_DIR}) +target_link_libraries(-lreadline) + # Optional Dependencies set(USE_BROCCOLI false)