From leaf.pan at yahoo.com Wed Jun 13 01:21:46 2018 From: leaf.pan at yahoo.com (leaf pan) Date: Wed, 13 Jun 2018 08:21:46 +0000 (UTC) Subject: [TM] how to make time-machine write data to disk References: <1489165447.3012552.1528878106652.ref@mail.yahoo.com> Message-ID: <1489165447.3012552.1528878106652@mail.yahoo.com> Hi Guys, I've run time-machine in centos 7 platform, but found that no data written to disk Below are my configuration and some logs, would you please give some suggestions? [timemachine.cfg]main { ? ? ? ? ## The device to sniff.? ? ? ? device "ens160";? ? ? ? workdir "/usr/local/var/tm";? ? ? ? indexdir "indexes";? ? ? ? queryfiledir "queries"; ? ? ? ? logfile "timemachine.log";? ? ? ? daemon 1;? ? ? ? console 0;? ? ? ? log_interval 60;? ? ? ? conn_timeout 180;? ? ? ? rmtconsole? 1;? ? ? ? rmtconsole_listen_addr 127.0.0.1;? ? ? ? rmtconsole_port 42042;? ? ? ? bro_listen 1;? ? ? ? bro_listen_port 47757;? ? ? ? bro_listen_addr 127.0.0.1; ? ? ? ? index "connection4" disk;? ? ? ? index "connection3" disk;? ? ? ? index "connection2" disk;? ? ? ? index "ip" disk;} class "all" {? ? ? ? filter "";? ? ? ? precedence 10;? ? ? ? cutoff 10k;? ? ? ? disk 50g;? ? ? ? filesize 128m;? ? ? ? mem 1g;} [timemachine.log]1528773477.257997 main: TimeMachine version 0.1-41528773477.265212 main: Forking Daemon1528773487.213428 main: capture started, capture thread1528773487.244189 main: Index aggregation thread started1528773487.244268 main: WARNING: Broccoli support not compiled in. [classes.timemachine.log]timestamp class stored_bytes stored_pkts cut_bytes cut_pkts mem_bytes mem_pkts mem_dt disk_bytes disk_pkts disk_dt1528773487.24 class_all 0 0 0 0 0 0 0.00 0 0 0.001528774087.24 class_all 70733 530 250414 1556 70733 530 598.83 0 0 0.001528774687.25 class_all 121598 875 385579 2253 121598 875 1198.74 0 0 0.00 [indexes.timemachine.log]timestamp index entries_mem qlen1528773487.24 connection4 0 01528773487.24 connection3 0 01528773487.24 connection2 0 01528773487.24 ip 0 01528774087.24 connection4 61 101528774087.24 connection3 124 21528774087.24 connection2 40 101528774087.24 ip 42 4 [stats.timemachine.log]timestamp conns dyn_class pcap_recv pcap_drop drop_ratio d_drop totPkts totBytes link_rate_mbit uncutPkts uncutBytes uncut_rate_mbit utime stime mem_os mem queryPkts queryBytes1528773487.24 0 0 1 0 0.00% 0 1 60 0.00 0 0 0.00 0.0 0.0 1158610944 103200 0 01528774087.24 37 0 7226 0 0.00% 0 7226 629295 0.01 530 70733 0.00 0.0 0.0 1159421952 174816 0 01528774687.25 21 0 14608 0 0.00% 0 14608 1195527 0.01 875 121598 0.00 0.0 0.0 1159557120 197488 0 0 ThanksBest Regards,Leaf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/time-machine/attachments/20180613/ef3d8d9f/attachment.html From asharma at lbl.gov Wed Jun 13 05:19:17 2018 From: asharma at lbl.gov (Aashish Sharma) Date: Wed, 13 Jun 2018 05:19:17 -0700 Subject: [TM] how to make time-machine write data to disk In-Reply-To: <1489165447.3012552.1528878106652@mail.yahoo.com> References: <1489165447.3012552.1528878106652.ref@mail.yahoo.com> <1489165447.3012552.1528878106652@mail.yahoo.com> Message-ID: <20180613121916.GI11976@MacPro-2331.local> Leaf, > class "all" {? ? ? ? filter "";? ? ? ? precedence 10;? ? ? ? cutoff 10k;? ? ? ? disk 50g;? ? ? ? filesize 128m;? ? ? ? mem 1g;} You specified mem 1g; Which basically means that TM which cache 1g of pcaps in memory before start to write on the disk. This functionality was there to help bro talk to time-machine and do a speedy extraction for any notice/alert of interest. If you run TM standalone (ie without bro integration), I'd advice put mem 1k or so, and as soon as 1k buffer is filled, TM starts putting pcaps on disk. Hope this helps, Aashish On Wed, Jun 13, 2018 at 08:21:46AM +0000, leaf pan wrote: > Hi Guys, > I've run time-machine in centos 7 platform, but found that no data written to disk > > Below are my configuration and some logs, would you please give some suggestions? > [timemachine.cfg]main { > ? ? ? ? ## The device to sniff.? ? ? ? device "ens160";? ? ? ? workdir "/usr/local/var/tm";? ? ? ? indexdir "indexes";? ? ? ? queryfiledir "queries"; > ? ? ? ? logfile "timemachine.log";? ? ? ? daemon 1;? ? ? ? console 0;? ? ? ? log_interval 60;? ? ? ? conn_timeout 180;? ? ? ? rmtconsole? 1;? ? ? ? rmtconsole_listen_addr 127.0.0.1;? ? ? ? rmtconsole_port 42042;? ? ? ? bro_listen 1;? ? ? ? bro_listen_port 47757;? ? ? ? bro_listen_addr 127.0.0.1; > ? ? ? ? index "connection4" disk;? ? ? ? index "connection3" disk;? ? ? ? index "connection2" disk;? ? ? ? index "ip" disk;} > class "all" {? ? ? ? filter "";? ? ? ? precedence 10;? ? ? ? cutoff 10k;? ? ? ? disk 50g;? ? ? ? filesize 128m;? ? ? ? mem 1g;} > > [timemachine.log]1528773477.257997 main: TimeMachine version 0.1-41528773477.265212 main: Forking Daemon1528773487.213428 main: capture started, capture thread1528773487.244189 main: Index aggregation thread started1528773487.244268 main: WARNING: Broccoli support not compiled in. > [classes.timemachine.log]timestamp class stored_bytes stored_pkts cut_bytes cut_pkts mem_bytes mem_pkts mem_dt disk_bytes disk_pkts disk_dt1528773487.24 class_all 0 0 0 0 0 0 0.00 0 0 0.001528774087.24 class_all 70733 530 250414 1556 70733 530 598.83 0 0 0.001528774687.25 class_all 121598 875 385579 2253 121598 875 1198.74 0 0 0.00 > [indexes.timemachine.log]timestamp index entries_mem qlen1528773487.24 connection4 0 01528773487.24 connection3 0 01528773487.24 connection2 0 01528773487.24 ip 0 01528774087.24 connection4 61 101528774087.24 connection3 124 21528774087.24 connection2 40 101528774087.24 ip 42 4 > [stats.timemachine.log]timestamp conns dyn_class pcap_recv pcap_drop drop_ratio d_drop totPkts totBytes link_rate_mbit uncutPkts uncutBytes uncut_rate_mbit utime stime mem_os mem queryPkts queryBytes1528773487.24 0 0 1 0 0.00% 0 1 60 0.00 0 0 0.00 0.0 0.0 1158610944 103200 0 01528774087.24 37 0 7226 0 0.00% 0 7226 629295 0.01 530 70733 0.00 0.0 0.0 1159421952 174816 0 01528774687.25 21 0 14608 0 0.00% 0 14608 1195527 0.01 875 121598 0.00 0.0 0.0 1159557120 197488 0 0 > > > ThanksBest Regards,Leaf > _______________________________________________ > Time-Machine mailing list > Time-Machine at mailman.ICSI.Berkeley.EDU > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/time-machine From leaf.pan at yahoo.com Wed Jun 13 23:00:19 2018 From: leaf.pan at yahoo.com (leaf pan) Date: Thu, 14 Jun 2018 06:00:19 +0000 (UTC) Subject: [TM] how to make time-machine write data to disk In-Reply-To: <20180613121916.GI11976@MacPro-2331.local> References: <1489165447.3012552.1528878106652.ref@mail.yahoo.com> <1489165447.3012552.1528878106652@mail.yahoo.com> <20180613121916.GI11976@MacPro-2331.local> Message-ID: <97818293.104914.1528956019626@mail.yahoo.com> Hi Aashish, That's great! It works well under 1k-mem setting.Thanks for your kindly help. Best Regards,Leaf On Wednesday, June 13, 2018, 8:19:22 PM GMT+8, Aashish Sharma wrote: Leaf, > class "all" {? ? ? ? filter "";? ? ? ? precedence 10;? ? ? ? cutoff 10k;? ? ? ? disk 50g;? ? ? ? filesize 128m;? ? ? ? mem 1g;} You specified mem 1g; Which basically means that TM which cache 1g of pcaps in memory before start to write on the disk. This functionality was there to help bro talk to time-machine and do a speedy extraction for any notice/alert of interest. If you run TM standalone (ie without bro integration), I'd advice put mem 1k or so, and as soon as 1k buffer is filled, TM starts putting pcaps on disk. Hope this helps, Aashish On Wed, Jun 13, 2018 at 08:21:46AM +0000, leaf pan wrote: > Hi Guys, > I've run time-machine in centos 7 platform, but found that no data written to disk > > Below are my configuration and some logs, would you please give some suggestions? > [timemachine.cfg]main { > ? ? ? ? ## The device to sniff.? ? ? ? device "ens160";? ? ? ? workdir "/usr/local/var/tm";? ? ? ? indexdir "indexes";? ? ? ? queryfiledir "queries"; > ? ? ? ? logfile "timemachine.log";? ? ? ? daemon 1;? ? ? ? console 0;? ? ? ? log_interval 60;? ? ? ? conn_timeout 180;? ? ? ? rmtconsole? 1;? ? ? ? rmtconsole_listen_addr 127.0.0.1;? ? ? ? rmtconsole_port 42042;? ? ? ? bro_listen 1;? ? ? ? bro_listen_port 47757;? ? ? ? bro_listen_addr 127.0.0.1; > ? ? ? ? index "connection4" disk;? ? ? ? index "connection3" disk;? ? ? ? index "connection2" disk;? ? ? ? index "ip" disk;} > class "all" {? ? ? ? filter "";? ? ? ? precedence 10;? ? ? ? cutoff 10k;? ? ? ? disk 50g;? ? ? ? filesize 128m;? ? ? ? mem 1g;} > > [timemachine.log]1528773477.257997 main: TimeMachine version 0.1-41528773477.265212 main: Forking Daemon1528773487.213428 main: capture started, capture thread1528773487.244189 main: Index aggregation thread started1528773487.244268 main: WARNING: Broccoli support not compiled in. > [classes.timemachine.log]timestamp class stored_bytes stored_pkts cut_bytes cut_pkts mem_bytes mem_pkts mem_dt disk_bytes disk_pkts disk_dt1528773487.24 class_all 0 0 0 0 0 0 0.00 0 0 0.001528774087.24 class_all 70733 530 250414 1556 70733 530 598.83 0 0 0.001528774687.25 class_all 121598 875 385579 2253 121598 875 1198.74 0 0 0.00 > [indexes.timemachine.log]timestamp index entries_mem qlen1528773487.24 connection4 0 01528773487.24 connection3 0 01528773487.24 connection2 0 01528773487.24 ip 0 01528774087.24 connection4 61 101528774087.24 connection3 124 21528774087.24 connection2 40 101528774087.24 ip 42 4 > [stats.timemachine.log]timestamp conns dyn_class pcap_recv pcap_drop drop_ratio d_drop totPkts totBytes link_rate_mbit uncutPkts uncutBytes uncut_rate_mbit utime stime mem_os mem queryPkts queryBytes1528773487.24 0 0 1 0 0.00% 0 1 60 0.00 0 0 0.00 0.0 0.0 1158610944 103200 0 01528774087.24 37 0 7226 0 0.00% 0 7226 629295 0.01 530 70733 0.00 0.0 0.0 1159421952 174816 0 01528774687.25 21 0 14608 0 0.00% 0 14608 1195527 0.01 875 121598 0.00 0.0 0.0 1159557120 197488 0 0 > > > ThanksBest Regards,Leaf > _______________________________________________ > Time-Machine mailing list > Time-Machine at mailman.ICSI.Berkeley.EDU > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/time-machine -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/time-machine/attachments/20180614/1b740458/attachment.html From asharma at lbl.gov Thu Jun 14 17:14:24 2018 From: asharma at lbl.gov (Aashish Sharma) Date: Thu, 14 Jun 2018 17:14:24 -0700 Subject: [TM] how to make time-machine write data to disk In-Reply-To: <97818293.104914.1528956019626@mail.yahoo.com> References: <1489165447.3012552.1528878106652.ref@mail.yahoo.com> <1489165447.3012552.1528878106652@mail.yahoo.com> <20180613121916.GI11976@MacPro-2331.local> <97818293.104914.1528956019626@mail.yahoo.com> Message-ID: <20180615001423.GC17769@6948462.dhcp.lbl.gov> Hello Leaf, Couple of other notes: 1) I see you've enabled indexes. Are you using them ? If not, I'd advice disable indexes. These are implemented as AVL trees and we've noticed packets dropped when these indexes are rebalanced. If you are using indexes then only enable. 2) What branch of TM are you using ? We had a intern (Naoki Eto) work on IPv6 integration and he also did a huge performance improvements. if you use topic/aashish/ipv6 you'd see much better results - this is pretty much Naoki's branch with one 2 line change I made for vlan tag stripping. 3) We (LBNL) has a couple of scripts to (i) extract TM logs if you provide bro logs to the script (ii) create a 'human' timeline <-> class files for some 'manual' or bot-controlled searches. Both are to avoid problem of (1) we've disabled indexes. I don't quite recall but removing indexes improves performance by 20+% - old data point though. 4) we also have a concept of fat and shallow TMs. Let us know if you have any questions/exchange tools technology etc. Thanks, Aashish On Thu, Jun 14, 2018 at 06:00:19AM +0000, leaf pan wrote: > Hi Aashish, > That's great! It works well under 1k-mem setting.Thanks for your kindly help. > > Best Regards,Leaf > On Wednesday, June 13, 2018, 8:19:22 PM GMT+8, Aashish Sharma wrote: > > Leaf, > > > class "all" {? ? ? ? filter "";? ? ? ? precedence 10;? ? ? ? cutoff 10k;? ? ? ? disk 50g;? ? ? ? filesize 128m;? ? ? ? mem 1g;} > > You specified mem 1g; > > Which basically means that TM which cache 1g of pcaps in memory before start to > write on the disk. > > This functionality was there to help bro talk to time-machine and do a speedy > extraction for any notice/alert of interest. > > If you run TM standalone (ie without bro integration), I'd advice put mem 1k or > so, and as soon as 1k buffer is filled, TM starts putting pcaps on disk. > > Hope this helps, > Aashish > > > On Wed, Jun 13, 2018 at 08:21:46AM +0000, leaf pan wrote: > > Hi Guys, > > I've run time-machine in centos 7 platform, but found that no data written to disk > > > > Below are my configuration and some logs, would you please give some suggestions? > > [timemachine.cfg]main { > > ? ? ? ? ## The device to sniff.? ? ? ? device "ens160";? ? ? ? workdir "/usr/local/var/tm";? ? ? ? indexdir "indexes";? ? ? ? queryfiledir "queries"; > > ? ? ? ? logfile "timemachine.log";? ? ? ? daemon 1;? ? ? ? console 0;? ? ? ? log_interval 60;? ? ? ? conn_timeout 180;? ? ? ? rmtconsole? 1;? ? ? ? rmtconsole_listen_addr 127.0.0.1;? ? ? ? rmtconsole_port 42042;? ? ? ? bro_listen 1;? ? ? ? bro_listen_port 47757;? ? ? ? bro_listen_addr 127.0.0.1; > > ? ? ? ? index "connection4" disk;? ? ? ? index "connection3" disk;? ? ? ? index "connection2" disk;? ? ? ? index "ip" disk;} > > class "all" {? ? ? ? filter "";? ? ? ? precedence 10;? ? ? ? cutoff 10k;? ? ? ? disk 50g;? ? ? ? filesize 128m;? ? ? ? mem 1g;} > > > > [timemachine.log]1528773477.257997 main: TimeMachine version 0.1-41528773477.265212 main: Forking Daemon1528773487.213428 main: capture started, capture thread1528773487.244189 main: Index aggregation thread started1528773487.244268 main: WARNING: Broccoli support not compiled in. > > [classes.timemachine.log]timestamp class stored_bytes stored_pkts cut_bytes cut_pkts mem_bytes mem_pkts mem_dt disk_bytes disk_pkts disk_dt1528773487.24 class_all 0 0 0 0 0 0 0.00 0 0 0.001528774087.24 class_all 70733 530 250414 1556 70733 530 598.83 0 0 0.001528774687.25 class_all 121598 875 385579 2253 121598 875 1198.74 0 0 0.00 > > [indexes.timemachine.log]timestamp index entries_mem qlen1528773487.24 connection4 0 01528773487.24 connection3 0 01528773487.24 connection2 0 01528773487.24 ip 0 01528774087.24 connection4 61 101528774087.24 connection3 124 21528774087.24 connection2 40 101528774087.24 ip 42 4 > > [stats.timemachine.log]timestamp conns dyn_class pcap_recv pcap_drop drop_ratio d_drop totPkts totBytes link_rate_mbit uncutPkts uncutBytes uncut_rate_mbit utime stime mem_os mem queryPkts queryBytes1528773487.24 0 0 1 0 0.00% 0 1 60 0.00 0 0 0.00 0.0 0.0 1158610944 103200 0 01528774087.24 37 0 7226 0 0.00% 0 7226 629295 0.01 530 70733 0.00 0.0 0.0 1159421952 174816 0 01528774687.25 21 0 14608 0 0.00% 0 14608 1195527 0.01 875 121598 0.00 0.0 0.0 1159557120 197488 0 0 > > > > > > ThanksBest Regards,Leaf > > > _______________________________________________ > > Time-Machine mailing list > > Time-Machine at mailman.ICSI.Berkeley.EDU > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/time-machine > >