From seth at corelight.com Mon Oct 2 06:46:52 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 02 Oct 2017 09:46:52 -0400 Subject: [Bro] bro and pf_ring zc configuration success stories In-Reply-To: References: <212F32B7-0DAA-48D9-AD0F-4A308BD57271@illinois.edu> <0578457A-01D5-4D69-8D63-C02BFBA5DBAF@illinois.edu> <49EF131F-C806-4896-B721-2B50788604ED@illinois.edu> <1703643E-5A0D-48AF-B194-6F55A3780BB6@illinois.edu> <37EA893A-FCC7-467F-AC3F-ED8F2F52B785@illinois.edu> <553EEDF8-51CC-438D-A293-DFA6A5EEE9F1@illinois.edu> Message-ID: <5EBE8209-22D3-423D-B98E-1572829F3288@corelight.com> Hi radek, Would you like to test using NETMAP too? It's fairly easy to get going with it these days and I'd be more than happy to give you hand. Seems worthwhile as a comparison point at least and it shouldn't take very long to get it setup. .Seth On 29 Sep 2017, at 11:44, radek wrote: > Hi! > I'm back with results. I've created new test and ran 200 Mbits, 600 > Mbit, > 1Gbit then went all in with 8 Gbits. > 1. You were right with traffic generator, previous test had some > parameters > changed and was doing something funky with TCP. I've removed this and > above > issues are to some extent gone. > 2. With zbalance_ipc -n 20 and worker definition: > > [worker-0] > type=worker > host=localhost > interface=pf_ring::zc:27 at 0 > pin_cpus=1 > > I'm able to process 4.5 Gbit/s with all 20 cores loaded at 60 - 70 % > with > minimal drop at bro > > [BroControl] > netstats > > worker-0: 1506695586.298096 recvd=5465310 dropped=30118 > link=5465310 > > worker-1: 1506695586.497686 recvd=5438281 dropped=9041 link=5438281 > > worker-2: 1506695586.701504 recvd=5498208 dropped=8756 link=5498208 > > worker-3: 1506695586.901398 recvd=5457893 dropped=9326 link=5457893 > > worker-4: 1506695587.101722 recvd=5472315 dropped=8877 link=5472315 > > worker-5: 1506695587.301448 recvd=5541810 dropped=10604 > link=5541810 > > worker-6: 1506695587.501405 recvd=5556953 dropped=2022 link=5556953 > > worker-7: 1506695587.705590 recvd=5508997 dropped=2149 link=5508997 > > worker-8: 1506695587.905592 recvd=5526052 dropped=1955 link=5526052 > > worker-9: 1506695588.105445 recvd=5506942 dropped=2751 link=5506942 > > worker-10: 1506695588.305863 recvd=5597609 dropped=7534 link=5597609 > > worker-11: 1506695588.505499 recvd=5550657 dropped=4975 link=5550657 > > worker-12: 1506695588.705426 recvd=5578005 dropped=1152 link=5578005 > > worker-13: 1506695588.905554 recvd=5541178 dropped=90 link=5541178 > > worker-14: 1506695589.109446 recvd=5561273 dropped=3568 link=5561273 > > worker-15: 1506695589.309585 recvd=5552211 dropped=2850 link=5552211 > > worker-16: 1506695589.509799 recvd=5524173 dropped=7896 link=5524173 > > worker-17: 1506695589.709838 recvd=5565320 dropped=10923 > link=5565320 > > worker-18: 1506695589.910352 recvd=5632122 dropped=9169 link=5632122 > > worker-19: 1506695590.113969 recvd=5603647 dropped=10448 > link=5603647 > > > > this drop occured at the beginning of test and stayed like this until > end > (20 minutes) > > with zbalance_ipc - n 20 -r 0:dummy0 and so on for 20 workers defined > like > this: > > [worker-0] > type=worker > host=localhost > interface=pf_ring::dummy0 > pin_cpus=1 > > > I can process at around 3 Gbit/s and around 36 % of packets are > dropped at > zbalance_ipc ingress (ixgbe NIC) (so it seems that bottleneck here is > zc - >> dummy packets processing) > Core designated for zbalance_ipc is loaded 100% during test , I'll > look > into it next. > > So so far so good. > I'll be posting updates on my findings > > I'm very grateful for your help. > Thank you. > > Best regards > Rado > > > > On 28 September 2017 at 19:17, radek wrote: > >> Will do, I'll get back with results tomorrow as my day ended. Thanks >> for >> your help so far. >> >> On 28 September 2017 at 19:13, Azoff, Justin S >> wrote: >> >>> >>>> On Sep 28, 2017, at 12:55 PM, radek wrote: >>>> >>>>> Can you configure your traffic generator to send it "real" >>>>> traffic? >>>> >>>> that's the setup, it is even called Real-World Traffic (TM) by >>>> vendor. >>> currently that's the only way for me to have somewhat reproducible >>> test >>> results in my setup. >>> >>> >>> Can you set the rate to 200mbit then for a bit? You need to get >>> things >>> to a point where the workers are running properly without drops. >>> >>> Then once the configuration looks correct and bro is logging proper >>> connections you can start ramping the rate back up. >>> >>> Based on the "error: 99.17%, 7562 out of 7625 connections are half >>> duplex" from before, nothing was working properly... and 50% drops >>> alone >>> wouldn't cause that. >>> >>> ? >>> Justin Azoff >>> >>> >> > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * www.corelight.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171002/dd38b97a/attachment.html From seth at corelight.com Mon Oct 2 07:11:19 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 02 Oct 2017 10:11:19 -0400 Subject: [Bro] optimize running bro from PCAPs / advantage of cluster mode In-Reply-To: <20170929092224.5f61c79e@NB181106> References: <20170922131506.3883ecfa@NB181106> <20170929092224.5f61c79e@NB181106> Message-ID: On 29 Sep 2017, at 3:22, Frank Meier wrote: > My original question still stands: Are there any parsers which combine > the information seen by different workers in different flows? Yes, FTP (control and data channels). Also, there are some scripts that take global views of activity to create derived logs (may not matter so much in your use case?). .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From jdopheid at illinois.edu Mon Oct 2 07:13:19 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Mon, 2 Oct 2017 14:13:19 +0000 Subject: [Bro] Renaming the Bro Project: seeking proposed names from the community Message-ID: This year at BroCon we announced that the Bro Project will be changing its name. While ?Bro? was originally meant as an Orwellian reminder of the risk that any monitoring fundamentally entails, it has more recently gained a very different, and quite offensive, reputation (?Bro culture?). To avoid facing instant negative impressions with new users that aren?t aware of the history, the Leadership Team has decided to seek a name change. We are accepting proposed names from the community for two months (due Monday December 4th). The Leadership Team will review the list of possible names and narrow it down to 5 finalists. We will announce the finalists and take a second round of feedback from the community before making the final selection. We hope to announce the new name within the next major release. To submit a proposed name, fill out the form here: https://goo.gl/forms/qwR8s6Yd4H0Bu8Ca2 ------ Jeannette Dopheide Sr. Education, Outreach, and Training Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From philosnef at gmail.com Tue Oct 3 05:10:23 2017 From: philosnef at gmail.com (erik clark) Date: Tue, 3 Oct 2017 08:10:23 -0400 Subject: [Bro] zc and bro Message-ID: There is some relevant information regarding the zc issues with bro on the ntop list for anyone using pf_ring zc for bro. See: http://listgateway.unipi.it/mailman/private/ntop-misc/2017-October/006519.html for relevance. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171003/3f11dc69/attachment.html From radoslawc at gmail.com Tue Oct 3 07:00:58 2017 From: radoslawc at gmail.com (radek) Date: Tue, 3 Oct 2017 16:00:58 +0200 Subject: [Bro] bro and pf_ring zc configuration success stories In-Reply-To: <5EBE8209-22D3-423D-B98E-1572829F3288@corelight.com> References: <212F32B7-0DAA-48D9-AD0F-4A308BD57271@illinois.edu> <0578457A-01D5-4D69-8D63-C02BFBA5DBAF@illinois.edu> <49EF131F-C806-4896-B721-2B50788604ED@illinois.edu> <1703643E-5A0D-48AF-B194-6F55A3780BB6@illinois.edu> <37EA893A-FCC7-467F-AC3F-ED8F2F52B785@illinois.edu> <553EEDF8-51CC-438D-A293-DFA6A5EEE9F1@illinois.edu> <5EBE8209-22D3-423D-B98E-1572829F3288@corelight.com> Message-ID: Hi! Sure, this looks interesting, let me get back to you once setup is ready. Best regards Rado On 2 October 2017 at 15:46, Seth Hall wrote: > Hi radek, > > Would you like to test using NETMAP too? It's fairly easy to get going > with it these days and I'd be more than happy to give you hand. Seems > worthwhile as a comparison point at least and it shouldn't take very long > to get it setup. > > .Seth > > On 29 Sep 2017, at 11:44, radek wrote: > > Hi! > I'm back with results. I've created new test and ran 200 Mbits, 600 Mbit, > 1Gbit then went all in with 8 Gbits. > 1. You were right with traffic generator, previous test had some > parameters changed and was doing something funky with TCP. I've removed > this and above issues are to some extent gone. > 2. With zbalance_ipc -n 20 and worker definition: > > [worker-0] > type=worker > host=localhost > interface=pf_ring::zc:27 at 0 > pin_cpus=1 > > I'm able to process 4.5 Gbit/s with all 20 cores loaded at 60 - 70 % with > minimal drop at bro > > [BroControl] > netstats > > worker-0: 1506695586.298096 recvd=5465310 dropped=30118 link=5465310 > > worker-1: 1506695586.497686 recvd=5438281 dropped=9041 link=5438281 > > worker-2: 1506695586.701504 recvd=5498208 dropped=8756 link=5498208 > > worker-3: 1506695586.901398 recvd=5457893 dropped=9326 link=5457893 > > worker-4: 1506695587.101722 recvd=5472315 dropped=8877 link=5472315 > > worker-5: 1506695587.301448 recvd=5541810 dropped=10604 link=5541810 > > worker-6: 1506695587.501405 recvd=5556953 dropped=2022 link=5556953 > > worker-7: 1506695587.705590 recvd=5508997 dropped=2149 link=5508997 > > worker-8: 1506695587.905592 recvd=5526052 dropped=1955 link=5526052 > > worker-9: 1506695588.105445 recvd=5506942 dropped=2751 link=5506942 > > worker-10: 1506695588.305863 recvd=5597609 dropped=7534 link=5597609 > > worker-11: 1506695588.505499 recvd=5550657 dropped=4975 link=5550657 > > worker-12: 1506695588.705426 recvd=5578005 dropped=1152 link=5578005 > > worker-13: 1506695588.905554 recvd=5541178 dropped=90 link=5541178 > > worker-14: 1506695589.109446 recvd=5561273 dropped=3568 link=5561273 > > worker-15: 1506695589.309585 recvd=5552211 dropped=2850 link=5552211 > > worker-16: 1506695589.509799 recvd=5524173 dropped=7896 link=5524173 > > worker-17: 1506695589.709838 recvd=5565320 dropped=10923 link=5565320 > > worker-18: 1506695589.910352 recvd=5632122 dropped=9169 link=5632122 > > worker-19: 1506695590.113969 recvd=5603647 dropped=10448 link=5603647 > > > > this drop occured at the beginning of test and stayed like this until end > (20 minutes) > > with zbalance_ipc - n 20 -r 0:dummy0 and so on for 20 workers defined like > this: > > [worker-0] > type=worker > host=localhost > interface=pf_ring::dummy0 > pin_cpus=1 > > > I can process at around 3 Gbit/s and around 36 % of packets are dropped at > zbalance_ipc ingress (ixgbe NIC) (so it seems that bottleneck here is zc - > > dummy packets processing) > Core designated for zbalance_ipc is loaded 100% during test , I'll look > into it next. > > So so far so good. > I'll be posting updates on my findings > > I'm very grateful for your help. > Thank you. > > Best regards > Rado > > > > On 28 September 2017 at 19:17, radek wrote: > >> Will do, I'll get back with results tomorrow as my day ended. Thanks for >> your help so far. >> >> On 28 September 2017 at 19:13, Azoff, Justin S >> wrote: >> >>> >>> > On Sep 28, 2017, at 12:55 PM, radek wrote: >>> > >>> > > Can you configure your traffic generator to send it "real" traffic? >>> > >>> > that's the setup, it is even called Real-World Traffic (TM) by vendor. >>> currently that's the only way for me to have somewhat reproducible test >>> results in my setup. >>> >>> >>> Can you set the rate to 200mbit then for a bit? You need to get things >>> to a point where the workers are running properly without drops. >>> >>> Then once the configuration looks correct and bro is logging proper >>> connections you can start ramping the rate back up. >>> >>> Based on the "error: 99.17%, 7562 out of 7625 connections are half >>> duplex" from before, nothing was working properly... and 50% drops alone >>> wouldn't cause that. >>> >>> ? >>> Justin Azoff >>> >>> >> > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- > Seth Hall * Corelight, Inc * www.corelight.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171003/6d0a5ec3/attachment.html From eshelton at butler.net Tue Oct 3 12:25:03 2017 From: eshelton at butler.net (eshelton) Date: Tue, 3 Oct 2017 13:25:03 -0600 Subject: [Bro] 'cf' utility with Bro 2.5 Message-ID: I've started to experience an issue with the 'cf' utility, so I wanted to check and see if anyone else had ever experienced a similar issue: Initially, I started to think that either I was crazy, or something had changed. I was checking some alerts Monday morning from last Friday, when I started experiencing an issue zcatting an http.log.gz file piped into 'cf'. I've changed nothing on this management server for months, yet I'm seeing seg faults like the following: Oct 2 11:54:29 lamborghini kernel: cf[182679]: segfault at 30 ip 00007fb078511111 sp 00007ffc2c7b87b0 error 4 in libc-2.19.so [7fb078459000+1ba000] Oct 2 11:54:50 lamborghini kernel: cf[182694]: segfault at 30 ip 00007ff5d3004111 sp 00007ffc70d411b0 error 4 in libc-2.19.so [7ff5d2f4c000+1ba000] Oct 2 11:54:57 lamborghini kernel: cf[182698]: segfault at 30 ip 00007ff285fb6111 sp 00007ffd44bb72d0 error 4 in libc-2.19.so [7ff285efe000+1ba000] Oct 2 11:55:39 lamborghini kernel: cf[182737]: segfault at 30 ip 00007f12925da111 sp 00007ffffdb25410 error 4 in libc-2.19.so [7f1292522000+1ba000] Oct 2 11:55:48 lamborghini kernel: cf[182743]: segfault at 30 ip 00007f7a86b20111 sp 00007fff3eec2900 error 4 in libc-2.19.so [7f7a86a68000+1ba000] Oct 2 11:55:53 lamborghini kernel: cf[182748]: segfault at 30 ip 00007f7134340111 sp 00007fffb518fca0 error 4 in libc-2.19.so [7f7134288000+1ba000] Oct 2 11:56:40 lamborghini kernel: cf[182772]: segfault at 30 ip 00007f5569185111 sp 00007ffc02d9ecd0 error 4 in libc-2.19.so [7f55690cd000+1ba000] Oct 2 11:58:12 lamborghini kernel: cf[183017]: segfault at 30 ip 00007f7d1167c111 sp 00007ffef64635f0 error 4 in libc-2.19.so [7f7d115c4000+1ba000] Oct 2 11:58:49 lamborghini kernel: cf[183032]: segfault at 30 ip 00007fa016b4c111 sp 00007ffc80bbbc00 error 4 in libc-2.19.so [7fa016a94000+1ba000] Oct 2 11:59:40 lamborghini kernel: cf[183062]: segfault at 30 ip 00007f4b2bbec111 sp 00007ffd7d556c00 error 4 in libc-2.19.so [7f4b2bb34000+1ba000] Oct 2 11:59:58 lamborghini kernel: cf[183068]: segfault at 30 ip 00007f71ab8ad111 sp 00007ffe11c6a230 error 4 in libc-2.19.so [7f71ab7f5000+1ba000] Oct 2 12:00:59 lamborghini kernel: cf[183102]: segfault at 30 ip 00007f11db924111 sp 00007ffe814cdc40 error 4 in libc-2.19.so [7f11db86c000+1ba000] Oct 2 12:01:26 lamborghini kernel: cf[183126]: segfault at 30 ip 00007ff0fb745111 sp 00007fff28522010 error 4 in libc-2.19.so [7ff0fb68d000+1ba000] Oct 2 12:02:08 lamborghini kernel: cf[183323]: segfault at 30 ip 00007f66d0079111 sp 00007fff1466ded0 error 4 in libc-2.19.so [7f66cffc1000+1ba000] Oct 2 12:02:20 lamborghini kernel: cf[183345]: segfault at 30 ip 00007f61ebad1111 sp 00007ffec6a6af60 error 4 in libc-2.19.so [7f61eba19000+1ba000] Oct 2 12:04:54 lamborghini kernel: cf[183420]: segfault at 30 ip 00007fdb0084f111 sp 00007ffc6d02ce90 error 4 in libc-2.19.so [7fdb00797000+1ba000] Oct 2 13:36:04 lamborghini kernel: cf[191311]: segfault at 30 ip 00007f4b682d9111 sp 00007ffff14c5850 error 4 in libc-2.19.so [7f4b68221000+1ba000] Oct 2 13:37:27 lamborghini kernel: cf[191707]: segfault at 30 ip 00007fd2e1a9a111 sp 00007fffa4d628a0 error 4 in libc-2.19.so [7fd2e19e2000+1ba000] Oct 2 13:40:50 lamborghini kernel: cf[193145]: segfault at 30 ip 00007f7480dea111 sp 00007ffe3bdb5f70 error 4 in libc-2.19.so [7f7480d32000+1ba000] Oct 2 13:41:29 lamborghini kernel: cf[193171]: segfault at 30 ip 00007fbb45684111 sp 00007ffffcb81670 error 4 in libc-2.19.so [7fbb455cc000+1ba000] Oct 2 13:41:48 lamborghini kernel: cf[193383]: segfault at 30 ip 00007fc039d6f111 sp 00007ffc0ff665e0 error 4 in libc-2.19.so [7fc039cb7000+1ba000] Oct 2 13:54:12 lamborghini kernel: cf[195708]: segfault at 30 ip 00007fcea4675111 sp 00007ffeec7142f0 error 4 in libc-2.19.so [7fcea45bd000+1ba000] Oct 2 14:17:22 lamborghini kernel: cf[1272]: segfault at 30 ip 00007fe0331f1111 sp 00007fff74bba0d0 error 4 in libc-2.19.so [7fe033139000+1ba000] Oct 2 14:32:51 lamborghini kernel: cf[1791]: segfault at 30 ip 00007fc53a151111 sp 00007fff567376a0 error 4 in libc-2.19.so [7fc53a099000+1ba000] Oct 2 14:33:26 lamborghini kernel: cf[2413]: segfault at 30 ip 00007fa6f93b8111 sp 00007ffd778c7740 error 4 in libc-2.19.so [7fa6f9300000+1ba000] Oct 2 14:55:26 lamborghini kernel: cf[5664]: segfault at 30 ip 00007f0c18ebe111 sp 00007ffd844fb370 error 4 in libc-2.19.so [7f0c18e06000+1ba000] Oct 2 14:55:56 lamborghini kernel: cf[5696]: segfault at 30 ip 00007f3814019111 sp 00007ffc936a16a0 error 4 in libc-2.19.so [7f3813f61000+1ba000] Oct 2 14:56:17 lamborghini kernel: cf[5702]: segfault at 30 ip 00007f1bf94ac111 sp 00007fff2584d280 error 4 in libc-2.19.so [7f1bf93f4000+1ba000] I attempted to check to see if there was a newer version of 'cf', but I now notice the link on the bro.org website to the 'cf' utility appears to no longer be valid. Is 'cf' still being used/promoted, and if so, is it possible that it's getting a re-work right now, and as such the the download link for the old version is no longer valid? Respectfully, -Erin Shelton -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171003/1aaf7a5e/attachment.html From asharma at lbl.gov Tue Oct 3 12:38:09 2017 From: asharma at lbl.gov (Aashish Sharma) Date: Tue, 3 Oct 2017 12:38:09 -0700 Subject: [Bro] 'cf' utility with Bro 2.5 In-Reply-To: References: Message-ID: <20171003193807.GA26807@MacPro-2331.local> Eshelton, Here is link to cf utility: ftp://ftp.ee.lbl.gov/cf-1.2.5.tar.gz Try this one and see if you still encounter seg faults. Aashish On Tue, Oct 03, 2017 at 01:25:03PM -0600, eshelton wrote: > I've started to experience an issue with the 'cf' utility, so I wanted to > check and see if anyone else had ever experienced a similar issue: > > Initially, I started to think that either I was crazy, or something had > changed. I was checking some alerts Monday morning from last Friday, when I > started experiencing an issue zcatting an http.log.gz file piped into 'cf'. > I've changed nothing on this management server for months, yet I'm seeing > seg faults like the following: > > Oct 2 11:54:29 lamborghini kernel: cf[182679]: segfault at 30 ip > 00007fb078511111 sp 00007ffc2c7b87b0 error 4 in libc-2.19.so > [7fb078459000+1ba000] > Oct 2 11:54:50 lamborghini kernel: cf[182694]: segfault at 30 ip > 00007ff5d3004111 sp 00007ffc70d411b0 error 4 in libc-2.19.so > [7ff5d2f4c000+1ba000] > Oct 2 11:54:57 lamborghini kernel: cf[182698]: segfault at 30 ip > 00007ff285fb6111 sp 00007ffd44bb72d0 error 4 in libc-2.19.so > [7ff285efe000+1ba000] > Oct 2 11:55:39 lamborghini kernel: cf[182737]: segfault at 30 ip > 00007f12925da111 sp 00007ffffdb25410 error 4 in libc-2.19.so > [7f1292522000+1ba000] > Oct 2 11:55:48 lamborghini kernel: cf[182743]: segfault at 30 ip > 00007f7a86b20111 sp 00007fff3eec2900 error 4 in libc-2.19.so > [7f7a86a68000+1ba000] > Oct 2 11:55:53 lamborghini kernel: cf[182748]: segfault at 30 ip > 00007f7134340111 sp 00007fffb518fca0 error 4 in libc-2.19.so > [7f7134288000+1ba000] > Oct 2 11:56:40 lamborghini kernel: cf[182772]: segfault at 30 ip > 00007f5569185111 sp 00007ffc02d9ecd0 error 4 in libc-2.19.so > [7f55690cd000+1ba000] > Oct 2 11:58:12 lamborghini kernel: cf[183017]: segfault at 30 ip > 00007f7d1167c111 sp 00007ffef64635f0 error 4 in libc-2.19.so > [7f7d115c4000+1ba000] > Oct 2 11:58:49 lamborghini kernel: cf[183032]: segfault at 30 ip > 00007fa016b4c111 sp 00007ffc80bbbc00 error 4 in libc-2.19.so > [7fa016a94000+1ba000] > Oct 2 11:59:40 lamborghini kernel: cf[183062]: segfault at 30 ip > 00007f4b2bbec111 sp 00007ffd7d556c00 error 4 in libc-2.19.so > [7f4b2bb34000+1ba000] > Oct 2 11:59:58 lamborghini kernel: cf[183068]: segfault at 30 ip > 00007f71ab8ad111 sp 00007ffe11c6a230 error 4 in libc-2.19.so > [7f71ab7f5000+1ba000] > Oct 2 12:00:59 lamborghini kernel: cf[183102]: segfault at 30 ip > 00007f11db924111 sp 00007ffe814cdc40 error 4 in libc-2.19.so > [7f11db86c000+1ba000] > Oct 2 12:01:26 lamborghini kernel: cf[183126]: segfault at 30 ip > 00007ff0fb745111 sp 00007fff28522010 error 4 in libc-2.19.so > [7ff0fb68d000+1ba000] > Oct 2 12:02:08 lamborghini kernel: cf[183323]: segfault at 30 ip > 00007f66d0079111 sp 00007fff1466ded0 error 4 in libc-2.19.so > [7f66cffc1000+1ba000] > Oct 2 12:02:20 lamborghini kernel: cf[183345]: segfault at 30 ip > 00007f61ebad1111 sp 00007ffec6a6af60 error 4 in libc-2.19.so > [7f61eba19000+1ba000] > Oct 2 12:04:54 lamborghini kernel: cf[183420]: segfault at 30 ip > 00007fdb0084f111 sp 00007ffc6d02ce90 error 4 in libc-2.19.so > [7fdb00797000+1ba000] > Oct 2 13:36:04 lamborghini kernel: cf[191311]: segfault at 30 ip > 00007f4b682d9111 sp 00007ffff14c5850 error 4 in libc-2.19.so > [7f4b68221000+1ba000] > Oct 2 13:37:27 lamborghini kernel: cf[191707]: segfault at 30 ip > 00007fd2e1a9a111 sp 00007fffa4d628a0 error 4 in libc-2.19.so > [7fd2e19e2000+1ba000] > Oct 2 13:40:50 lamborghini kernel: cf[193145]: segfault at 30 ip > 00007f7480dea111 sp 00007ffe3bdb5f70 error 4 in libc-2.19.so > [7f7480d32000+1ba000] > Oct 2 13:41:29 lamborghini kernel: cf[193171]: segfault at 30 ip > 00007fbb45684111 sp 00007ffffcb81670 error 4 in libc-2.19.so > [7fbb455cc000+1ba000] > Oct 2 13:41:48 lamborghini kernel: cf[193383]: segfault at 30 ip > 00007fc039d6f111 sp 00007ffc0ff665e0 error 4 in libc-2.19.so > [7fc039cb7000+1ba000] > Oct 2 13:54:12 lamborghini kernel: cf[195708]: segfault at 30 ip > 00007fcea4675111 sp 00007ffeec7142f0 error 4 in libc-2.19.so > [7fcea45bd000+1ba000] > Oct 2 14:17:22 lamborghini kernel: cf[1272]: segfault at 30 ip > 00007fe0331f1111 sp 00007fff74bba0d0 error 4 in libc-2.19.so > [7fe033139000+1ba000] > Oct 2 14:32:51 lamborghini kernel: cf[1791]: segfault at 30 ip > 00007fc53a151111 sp 00007fff567376a0 error 4 in libc-2.19.so > [7fc53a099000+1ba000] > Oct 2 14:33:26 lamborghini kernel: cf[2413]: segfault at 30 ip > 00007fa6f93b8111 sp 00007ffd778c7740 error 4 in libc-2.19.so > [7fa6f9300000+1ba000] > Oct 2 14:55:26 lamborghini kernel: cf[5664]: segfault at 30 ip > 00007f0c18ebe111 sp 00007ffd844fb370 error 4 in libc-2.19.so > [7f0c18e06000+1ba000] > Oct 2 14:55:56 lamborghini kernel: cf[5696]: segfault at 30 ip > 00007f3814019111 sp 00007ffc936a16a0 error 4 in libc-2.19.so > [7f3813f61000+1ba000] > Oct 2 14:56:17 lamborghini kernel: cf[5702]: segfault at 30 ip > 00007f1bf94ac111 sp 00007fff2584d280 error 4 in libc-2.19.so > [7f1bf93f4000+1ba000] > > I attempted to check to see if there was a newer version of 'cf', but I now > notice the link on the bro.org website to the 'cf' utility appears to no > longer be valid. > > Is 'cf' still being used/promoted, and if so, is it possible that it's > getting a re-work right now, and as such the the download link for the old > version is no longer valid? > > Respectfully, > > -Erin Shelton > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From dopheide at gmail.com Tue Oct 3 12:48:08 2017 From: dopheide at gmail.com (Mike Dopheide) Date: Tue, 3 Oct 2017 14:48:08 -0500 Subject: [Bro] 'cf' utility with Bro 2.5 In-Reply-To: <20171003193807.GA26807@MacPro-2331.local> References: <20171003193807.GA26807@MacPro-2331.local> Message-ID: Or switch to using 'bro-cut -d' and not have to worry about keeping up with 'cf'. :) -Dop On Tue, Oct 3, 2017 at 2:38 PM, Aashish Sharma wrote: > Eshelton, > > Here is link to cf utility: ftp://ftp.ee.lbl.gov/cf-1.2.5.tar.gz > > Try this one and see if you still encounter seg faults. > > Aashish > > On Tue, Oct 03, 2017 at 01:25:03PM -0600, eshelton wrote: > > I've started to experience an issue with the 'cf' utility, so I wanted to > > check and see if anyone else had ever experienced a similar issue: > > > > Initially, I started to think that either I was crazy, or something had > > changed. I was checking some alerts Monday morning from last Friday, > when I > > started experiencing an issue zcatting an http.log.gz file piped into > 'cf'. > > I've changed nothing on this management server for months, yet I'm seeing > > seg faults like the following: > > > > Oct 2 11:54:29 lamborghini kernel: cf[182679]: segfault at 30 ip > > 00007fb078511111 sp 00007ffc2c7b87b0 error 4 in libc-2.19.so > > [7fb078459000+1ba000] > > Oct 2 11:54:50 lamborghini kernel: cf[182694]: segfault at 30 ip > > 00007ff5d3004111 sp 00007ffc70d411b0 error 4 in libc-2.19.so > > [7ff5d2f4c000+1ba000] > > Oct 2 11:54:57 lamborghini kernel: cf[182698]: segfault at 30 ip > > 00007ff285fb6111 sp 00007ffd44bb72d0 error 4 in libc-2.19.so > > [7ff285efe000+1ba000] > > Oct 2 11:55:39 lamborghini kernel: cf[182737]: segfault at 30 ip > > 00007f12925da111 sp 00007ffffdb25410 error 4 in libc-2.19.so > > [7f1292522000+1ba000] > > Oct 2 11:55:48 lamborghini kernel: cf[182743]: segfault at 30 ip > > 00007f7a86b20111 sp 00007fff3eec2900 error 4 in libc-2.19.so > > [7f7a86a68000+1ba000] > > Oct 2 11:55:53 lamborghini kernel: cf[182748]: segfault at 30 ip > > 00007f7134340111 sp 00007fffb518fca0 error 4 in libc-2.19.so > > [7f7134288000+1ba000] > > Oct 2 11:56:40 lamborghini kernel: cf[182772]: segfault at 30 ip > > 00007f5569185111 sp 00007ffc02d9ecd0 error 4 in libc-2.19.so > > [7f55690cd000+1ba000] > > Oct 2 11:58:12 lamborghini kernel: cf[183017]: segfault at 30 ip > > 00007f7d1167c111 sp 00007ffef64635f0 error 4 in libc-2.19.so > > [7f7d115c4000+1ba000] > > Oct 2 11:58:49 lamborghini kernel: cf[183032]: segfault at 30 ip > > 00007fa016b4c111 sp 00007ffc80bbbc00 error 4 in libc-2.19.so > > [7fa016a94000+1ba000] > > Oct 2 11:59:40 lamborghini kernel: cf[183062]: segfault at 30 ip > > 00007f4b2bbec111 sp 00007ffd7d556c00 error 4 in libc-2.19.so > > [7f4b2bb34000+1ba000] > > Oct 2 11:59:58 lamborghini kernel: cf[183068]: segfault at 30 ip > > 00007f71ab8ad111 sp 00007ffe11c6a230 error 4 in libc-2.19.so > > [7f71ab7f5000+1ba000] > > Oct 2 12:00:59 lamborghini kernel: cf[183102]: segfault at 30 ip > > 00007f11db924111 sp 00007ffe814cdc40 error 4 in libc-2.19.so > > [7f11db86c000+1ba000] > > Oct 2 12:01:26 lamborghini kernel: cf[183126]: segfault at 30 ip > > 00007ff0fb745111 sp 00007fff28522010 error 4 in libc-2.19.so > > [7ff0fb68d000+1ba000] > > Oct 2 12:02:08 lamborghini kernel: cf[183323]: segfault at 30 ip > > 00007f66d0079111 sp 00007fff1466ded0 error 4 in libc-2.19.so > > [7f66cffc1000+1ba000] > > Oct 2 12:02:20 lamborghini kernel: cf[183345]: segfault at 30 ip > > 00007f61ebad1111 sp 00007ffec6a6af60 error 4 in libc-2.19.so > > [7f61eba19000+1ba000] > > Oct 2 12:04:54 lamborghini kernel: cf[183420]: segfault at 30 ip > > 00007fdb0084f111 sp 00007ffc6d02ce90 error 4 in libc-2.19.so > > [7fdb00797000+1ba000] > > Oct 2 13:36:04 lamborghini kernel: cf[191311]: segfault at 30 ip > > 00007f4b682d9111 sp 00007ffff14c5850 error 4 in libc-2.19.so > > [7f4b68221000+1ba000] > > Oct 2 13:37:27 lamborghini kernel: cf[191707]: segfault at 30 ip > > 00007fd2e1a9a111 sp 00007fffa4d628a0 error 4 in libc-2.19.so > > [7fd2e19e2000+1ba000] > > Oct 2 13:40:50 lamborghini kernel: cf[193145]: segfault at 30 ip > > 00007f7480dea111 sp 00007ffe3bdb5f70 error 4 in libc-2.19.so > > [7f7480d32000+1ba000] > > Oct 2 13:41:29 lamborghini kernel: cf[193171]: segfault at 30 ip > > 00007fbb45684111 sp 00007ffffcb81670 error 4 in libc-2.19.so > > [7fbb455cc000+1ba000] > > Oct 2 13:41:48 lamborghini kernel: cf[193383]: segfault at 30 ip > > 00007fc039d6f111 sp 00007ffc0ff665e0 error 4 in libc-2.19.so > > [7fc039cb7000+1ba000] > > Oct 2 13:54:12 lamborghini kernel: cf[195708]: segfault at 30 ip > > 00007fcea4675111 sp 00007ffeec7142f0 error 4 in libc-2.19.so > > [7fcea45bd000+1ba000] > > Oct 2 14:17:22 lamborghini kernel: cf[1272]: segfault at 30 ip > > 00007fe0331f1111 sp 00007fff74bba0d0 error 4 in libc-2.19.so > > [7fe033139000+1ba000] > > Oct 2 14:32:51 lamborghini kernel: cf[1791]: segfault at 30 ip > > 00007fc53a151111 sp 00007fff567376a0 error 4 in libc-2.19.so > > [7fc53a099000+1ba000] > > Oct 2 14:33:26 lamborghini kernel: cf[2413]: segfault at 30 ip > > 00007fa6f93b8111 sp 00007ffd778c7740 error 4 in libc-2.19.so > > [7fa6f9300000+1ba000] > > Oct 2 14:55:26 lamborghini kernel: cf[5664]: segfault at 30 ip > > 00007f0c18ebe111 sp 00007ffd844fb370 error 4 in libc-2.19.so > > [7f0c18e06000+1ba000] > > Oct 2 14:55:56 lamborghini kernel: cf[5696]: segfault at 30 ip > > 00007f3814019111 sp 00007ffc936a16a0 error 4 in libc-2.19.so > > [7f3813f61000+1ba000] > > Oct 2 14:56:17 lamborghini kernel: cf[5702]: segfault at 30 ip > > 00007f1bf94ac111 sp 00007fff2584d280 error 4 in libc-2.19.so > > [7f1bf93f4000+1ba000] > > > > I attempted to check to see if there was a newer version of 'cf', but I > now > > notice the link on the bro.org website to the 'cf' utility appears to no > > longer be valid. > > > > Is 'cf' still being used/promoted, and if so, is it possible that it's > > getting a re-work right now, and as such the the download link for the > old > > version is no longer valid? > > > > Respectfully, > > > > -Erin Shelton > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171003/e87f033a/attachment-0001.html From BLMILLER at comerica.com Tue Oct 3 15:41:49 2017 From: BLMILLER at comerica.com (Miller, Brad L) Date: Tue, 3 Oct 2017 22:41:49 +0000 Subject: [Bro] Issue compiling from source on VM / Digital Ocean Message-ID: I noticed this started occurring with 2.5.x. Attempting to make from source. All dependencies met and following instructions from Digital Ocean. Whether from a git clone or pulling the source files in manually, always leads to an internal compiler error. System is Ubuntu 16.04, fully updated. Additionally, when attempting to install from the opensuse repo, there is an umet dependency error. I recall (not confirmed) that I attempted to build from source in a KVM guest on my own machine (at Brocon) and had the exact same compiler error. Is there a new incompatibility with virtual systems? Repo error Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: bro : Depends: bro-core (= 2.5.1-0) but it is not going to be installed Depends: broctl (= 2.5.1-0) but it is not going to be installed Compiler error [ 86%] Building CXX object src/CMakeFiles/bro.dir/Func.cc.o c++: internal compiler error: Killed (program cc1plus) Please submit a full bug report, with preprocessed source if appropriate. See for instructions. src/CMakeFiles/bro.dir/build.make:1222: recipe for target 'src/CMakeFiles/bro.dir/Func.cc.o' failed make[3]: *** [src/CMakeFiles/bro.dir/Func.cc.o] Error 4 make[3]: Leaving directory '/root/bro/build' CMakeFiles/Makefile2:828: recipe for target 'src/CMakeFiles/bro.dir/all' failed make[2]: *** [src/CMakeFiles/bro.dir/all] Error 2 make[2]: Leaving directory '/root/bro/build' Makefile:149: recipe for target 'all' failed make[1]: *** [all] Error 2 make[1]: Leaving directory '/root/bro/build' Makefile:15: recipe for target 'all' failed make: *** [all] Error 2 Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171003/6c26f7fd/attachment.html From johanna at icir.org Tue Oct 3 16:15:26 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 3 Oct 2017 17:15:26 -0600 Subject: [Bro] Issue compiling from source on VM / Digital Ocean In-Reply-To: References: Message-ID: <20171003231526.wn5xg76etqimksyp@Trafalgar.local> For the repo error (which I assume is when trying to install binary packages, not when trying to install from source) - could you give the exact steps that you perform to get that error (in the optimal case all commands starting from a clean installation). For the compilation error - is there a chance that your VM is running out of memory? Johanna On Tue, Oct 03, 2017 at 10:41:49PM +0000, Miller, Brad L wrote: > I noticed this started occurring with 2.5.x. Attempting to make from source. All dependencies met and following instructions from Digital Ocean. Whether from a git clone or pulling the source files in manually, always leads to an internal compiler error. System is Ubuntu 16.04, fully updated. > > Additionally, when attempting to install from the opensuse repo, there is an umet dependency error. > > I recall (not confirmed) that I attempted to build from source in a KVM guest on my own machine (at Brocon) and had the exact same compiler error. > > Is there a new incompatibility with virtual systems? > > Repo error > Some packages could not be installed. This may mean that you have > requested an impossible situation or if you are using the unstable > distribution that some required packages have not yet been created > or been moved out of Incoming. > The following information may help to resolve the situation: > > The following packages have unmet dependencies: > bro : Depends: bro-core (= 2.5.1-0) but it is not going to be installed > Depends: broctl (= 2.5.1-0) but it is not going to be installed > > Compiler error > > [ 86%] Building CXX object src/CMakeFiles/bro.dir/Func.cc.o > c++: internal compiler error: Killed (program cc1plus) > Please submit a full bug report, > with preprocessed source if appropriate. > See for instructions. > src/CMakeFiles/bro.dir/build.make:1222: recipe for target 'src/CMakeFiles/bro.dir/Func.cc.o' failed > make[3]: *** [src/CMakeFiles/bro.dir/Func.cc.o] Error 4 > make[3]: Leaving directory '/root/bro/build' > CMakeFiles/Makefile2:828: recipe for target 'src/CMakeFiles/bro.dir/all' failed > make[2]: *** [src/CMakeFiles/bro.dir/all] Error 2 > make[2]: Leaving directory '/root/bro/build' > Makefile:149: recipe for target 'all' failed > make[1]: *** [all] Error 2 > make[1]: Leaving directory '/root/bro/build' > Makefile:15: recipe for target 'all' failed > make: *** [all] Error 2 > > > Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From BLMILLER at comerica.com Tue Oct 3 19:11:15 2017 From: BLMILLER at comerica.com (Miller, Brad L) Date: Wed, 4 Oct 2017 02:11:15 +0000 Subject: [Bro] Issue compiling from source on VM / Digital Ocean In-Reply-To: References: Message-ID: Update: Does NOT occur with same source package, same dependencies, using Ubuntu 14.04 From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Miller, Brad L Sent: Tuesday, October 03, 2017 6:42 PM To: bro at bro.org Subject: [Bro] Issue compiling from source on VM / Digital Ocean I noticed this started occurring with 2.5.x. Attempting to make from source. All dependencies met and following instructions from Digital Ocean. Whether from a git clone or pulling the source files in manually, always leads to an internal compiler error. System is Ubuntu 16.04, fully updated. Additionally, when attempting to install from the opensuse repo, there is an umet dependency error. I recall (not confirmed) that I attempted to build from source in a KVM guest on my own machine (at Brocon) and had the exact same compiler error. Is there a new incompatibility with virtual systems? Repo error Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: bro : Depends: bro-core (= 2.5.1-0) but it is not going to be installed Depends: broctl (= 2.5.1-0) but it is not going to be installed Compiler error [ 86%] Building CXX object src/CMakeFiles/bro.dir/Func.cc.o c++: internal compiler error: Killed (program cc1plus) Please submit a full bug report, with preprocessed source if appropriate. See > for instructions. src/CMakeFiles/bro.dir/build.make:1222: recipe for target 'src/CMakeFiles/bro.dir/Func.cc.o' failed make[3]: *** [src/CMakeFiles/bro.dir/Func.cc.o] Error 4 make[3]: Leaving directory '/root/bro/build' CMakeFiles/Makefile2:828: recipe for target 'src/CMakeFiles/bro.dir/all' failed make[2]: *** [src/CMakeFiles/bro.dir/all] Error 2 make[2]: Leaving directory '/root/bro/build' Makefile:149: recipe for target 'all' failed make[1]: *** [all] Error 2 make[1]: Leaving directory '/root/bro/build' Makefile:15: recipe for target 'all' failed make: *** [all] Error 2 Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email. Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171004/7b78e1c2/attachment-0001.html From BLMILLER at comerica.com Tue Oct 3 19:15:55 2017 From: BLMILLER at comerica.com (Miller, Brad L) Date: Wed, 4 Oct 2017 02:15:55 +0000 Subject: [Bro] Issue compiling from source on VM / Digital Ocean In-Reply-To: <20171003231526.wn5xg76etqimksyp@Trafalgar.local> References: <20171003231526.wn5xg76etqimksyp@Trafalgar.local> Message-ID: The repo method was as described here: https://www.bro.org/download/packages.html (Ubuntu 16.04) It was newly provisioned VPS with Digital Ocean, running no other utilities at the time. It's unlikely it was running out of memory, but I destroyed it soon after I spun up Ubuntu 14.04 to test. As I mentioned, building from source the same exact way on Ubuntu 14.04 did not have the compiler error. I didn't try and install from repo on 14.04. -----Original Message----- From: Johanna Amann [mailto:johanna at icir.org] Sent: Tuesday, October 03, 2017 7:15 PM To: Miller, Brad L Cc: bro at bro.org Subject: Re: [Bro] Issue compiling from source on VM / Digital Ocean For the repo error (which I assume is when trying to install binary packages, not when trying to install from source) - could you give the exact steps that you perform to get that error (in the optimal case all commands starting from a clean installation). For the compilation error - is there a chance that your VM is running out of memory? Johanna On Tue, Oct 03, 2017 at 10:41:49PM +0000, Miller, Brad L wrote: > I noticed this started occurring with 2.5.x. Attempting to make from source. All dependencies met and following instructions from Digital Ocean. Whether from a git clone or pulling the source files in manually, always leads to an internal compiler error. System is Ubuntu 16.04, fully updated. > > Additionally, when attempting to install from the opensuse repo, there is an umet dependency error. > > I recall (not confirmed) that I attempted to build from source in a KVM guest on my own machine (at Brocon) and had the exact same compiler error. > > Is there a new incompatibility with virtual systems? > > Repo error > Some packages could not be installed. This may mean that you have > requested an impossible situation or if you are using the unstable > distribution that some required packages have not yet been created or > been moved out of Incoming. > The following information may help to resolve the situation: > > The following packages have unmet dependencies: > bro : Depends: bro-core (= 2.5.1-0) but it is not going to be installed > Depends: broctl (= 2.5.1-0) but it is not going to be installed > > Compiler error > > [ 86%] Building CXX object src/CMakeFiles/bro.dir/Func.cc.o > c++: internal compiler error: Killed (program cc1plus) > Please submit a full bug report, > with preprocessed source if appropriate. > See for instructions. > src/CMakeFiles/bro.dir/build.make:1222: recipe for target > 'src/CMakeFiles/bro.dir/Func.cc.o' failed > make[3]: *** [src/CMakeFiles/bro.dir/Func.cc.o] Error 4 > make[3]: Leaving directory '/root/bro/build' > CMakeFiles/Makefile2:828: recipe for target > 'src/CMakeFiles/bro.dir/all' failed > make[2]: *** [src/CMakeFiles/bro.dir/all] Error 2 > make[2]: Leaving directory '/root/bro/build' > Makefile:149: recipe for target 'all' failed > make[1]: *** [all] Error 2 > make[1]: Leaving directory '/root/bro/build' > Makefile:15: recipe for target 'all' failed > make: *** [all] Error 2 > > > Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email. From BLMILLER at comerica.com Tue Oct 3 19:58:16 2017 From: BLMILLER at comerica.com (Miller, Brad L) Date: Wed, 4 Oct 2017 02:58:16 +0000 Subject: [Bro] Issue compiling from source on VM / Digital Ocean In-Reply-To: References: <20171003231526.wn5xg76etqimksyp@Trafalgar.local> Message-ID: I destroyed that VM again and repeated the package install on Ubuntu 14.04, the method worked with no issues seen in 16.04. -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Miller, Brad L Sent: Tuesday, October 03, 2017 10:16 PM To: Johanna Amann Cc: bro at bro.org Subject: Re: [Bro] Issue compiling from source on VM / Digital Ocean The repo method was as described here: https://www.bro.org/download/packages.html (Ubuntu 16.04) It was newly provisioned VPS with Digital Ocean, running no other utilities at the time. It's unlikely it was running out of memory, but I destroyed it soon after I spun up Ubuntu 14.04 to test. As I mentioned, building from source the same exact way on Ubuntu 14.04 did not have the compiler error. I didn't try and install from repo on 14.04. -----Original Message----- From: Johanna Amann [mailto:johanna at icir.org] Sent: Tuesday, October 03, 2017 7:15 PM To: Miller, Brad L Cc: bro at bro.org Subject: Re: [Bro] Issue compiling from source on VM / Digital Ocean For the repo error (which I assume is when trying to install binary packages, not when trying to install from source) - could you give the exact steps that you perform to get that error (in the optimal case all commands starting from a clean installation). For the compilation error - is there a chance that your VM is running out of memory? Johanna On Tue, Oct 03, 2017 at 10:41:49PM +0000, Miller, Brad L wrote: > I noticed this started occurring with 2.5.x. Attempting to make from source. All dependencies met and following instructions from Digital Ocean. Whether from a git clone or pulling the source files in manually, always leads to an internal compiler error. System is Ubuntu 16.04, fully updated. > > Additionally, when attempting to install from the opensuse repo, there is an umet dependency error. > > I recall (not confirmed) that I attempted to build from source in a KVM guest on my own machine (at Brocon) and had the exact same compiler error. > > Is there a new incompatibility with virtual systems? > > Repo error > Some packages could not be installed. This may mean that you have > requested an impossible situation or if you are using the unstable > distribution that some required packages have not yet been created or > been moved out of Incoming. > The following information may help to resolve the situation: > > The following packages have unmet dependencies: > bro : Depends: bro-core (= 2.5.1-0) but it is not going to be installed > Depends: broctl (= 2.5.1-0) but it is not going to be installed > > Compiler error > > [ 86%] Building CXX object src/CMakeFiles/bro.dir/Func.cc.o > c++: internal compiler error: Killed (program cc1plus) > Please submit a full bug report, > with preprocessed source if appropriate. > See for instructions. > src/CMakeFiles/bro.dir/build.make:1222: recipe for target > 'src/CMakeFiles/bro.dir/Func.cc.o' failed > make[3]: *** [src/CMakeFiles/bro.dir/Func.cc.o] Error 4 > make[3]: Leaving directory '/root/bro/build' > CMakeFiles/Makefile2:828: recipe for target > 'src/CMakeFiles/bro.dir/all' failed > make[2]: *** [src/CMakeFiles/bro.dir/all] Error 2 > make[2]: Leaving directory '/root/bro/build' > Makefile:149: recipe for target 'all' failed > make[1]: *** [all] Error 2 > make[1]: Leaving directory '/root/bro/build' > Makefile:15: recipe for target 'all' failed > make: *** [all] Error 2 > > > Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email. _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email. From vikrambasu059 at gmail.com Wed Oct 4 05:54:59 2017 From: vikrambasu059 at gmail.com (Vikram Basu) Date: Wed, 4 Oct 2017 18:24:59 +0530 Subject: [Bro] Calling external scripts on extracted files Message-ID: Hi, I am using the bro file-extraction script from the bro-pkg manager and want to run a python script as soon as the file is completely extracted. Currently I am calling the script using the Bro Exec::run command after modifying the script but often times the script is running before the file has finished extracted and is failing as a result. How can I make it so that Bro calls the script after file has already been extracted? I thought maybe using file_state_remove would help but even in that case I am getting /Input::READER_RAW: Child process exited with non-zero return code 127 which I am assuming means the script was run before the file as truly extracted? Any advice would be much appreciated. Regards Vikram -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171004/2ab4636c/attachment.html From hosom at battelle.org Wed Oct 4 06:24:45 2017 From: hosom at battelle.org (Hosom, Stephen M) Date: Wed, 4 Oct 2017 13:24:45 +0000 Subject: [Bro] Calling external scripts on extracted files In-Reply-To: References: Message-ID: <2b99858caf264df4b105d724db28d435@battelle.org> Vikram, I'm the author of the package that you're using. Happy to help! I don't know why it is precisely that your script is not working, however, I have good examples of how to do this type of activity within the plugin. Check out the file store-files-by-md5.bro withing the plugins directory. This script uses the mv command to move files and rename them based on their hash once Bro finishes extracting them and is a good example of how to perform an action on a file once it has been extracted "the right way". Please let me know if you have any issues... You may find that I am more responsive to the issues page for the project on GitHub. Thanks, Stephen ________________________________ From: bro-bounces at bro.org on behalf of Vikram Basu Sent: Wednesday, October 4, 2017 8:54:59 AM To: bro at bro.org Subject: [Bro] Calling external scripts on extracted files Message received from outside the Battelle network. Carefully examine it before you open any links or attachments. Hi, I am using the bro file-extraction script from the bro-pkg manager and want to run a python script as soon as the file is completely extracted. Currently I am calling the script using the Bro Exec::run command after modifying the script but often times the script is running before the file has finished extracted and is failing as a result. How can I make it so that Bro calls the script after file has already been extracted? I thought maybe using file_state_remove would help but even in that case I am getting /Input::READER_RAW: Child process exited with non-zero return code 127 which I am assuming means the script was run before the file as truly extracted? Any advice would be much appreciated. Regards Vikram From vikrambasu059 at gmail.com Wed Oct 4 06:41:07 2017 From: vikrambasu059 at gmail.com (Vikram Basu) Date: Wed, 4 Oct 2017 19:11:07 +0530 Subject: [Bro] Calling external scripts on extracted files In-Reply-To: <2b99858caf264df4b105d724db28d435@battelle.org> References: <2b99858caf264df4b105d724db28d435@battelle.org> Message-ID: Awesome. Thanks, I'll check it out. On 04-Oct-2017 6:54 PM, "Hosom, Stephen M" wrote: > Vikram, > > > I'm the author of the package that you're using. Happy to help! > > > I don't know why it is precisely that your script is not working, however, > I have good examples of how to do this type of activity within the plugin. > > > Check out the file store-files-by-md5.bro withing the plugins directory. > > > This script uses the mv command to move files and rename them based on > their hash once Bro finishes extracting them and is a good example of how > to perform an action on a file once it has been extracted "the right way". > > > Please let me know if you have any issues... You may find that I am more > responsive to the issues page for the project on GitHub. > > > Thanks, > > > Stephen > > ________________________________ > From: bro-bounces at bro.org on behalf of Vikram Basu < > vikrambasu059 at gmail.com> > Sent: Wednesday, October 4, 2017 8:54:59 AM > To: bro at bro.org > Subject: [Bro] Calling external scripts on extracted files > > Message received from outside the Battelle network. Carefully examine it > before you open any links or attachments. > > Hi, > > I am using the bro file-extraction script from the bro-pkg manager and > want to run a python script as soon as the file is completely extracted. > > Currently I am calling the script using the Bro Exec::run command after > modifying the script but often times the script is running before the file > has finished extracted and is failing as a result. > > How can I make it so that Bro calls the script after file has already been > extracted? > > I thought maybe using file_state_remove would help but even in that case I > am getting > /Input::READER_RAW: Child process exited with non-zero return code 127 > which I am assuming means the script was run before the file as truly > extracted? > > Any advice would be much appreciated. > > Regards > > Vikram > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171004/c4c47558/attachment-0001.html From johanna at icir.org Wed Oct 4 07:57:57 2017 From: johanna at icir.org (Johanna Amann) Date: Wed, 4 Oct 2017 08:57:57 -0600 Subject: [Bro] Issue compiling from source on VM / Digital Ocean In-Reply-To: References: <20171003231526.wn5xg76etqimksyp@Trafalgar.local> Message-ID: <20171004145757.fve44pcsq6yjgk43@Trafalgar.local> Hi Brad, I sadly cannot reproduce this. I used a Ubuntu 16.04.03 x64 on Ubuntu ($20/mo standard instance). The binary installation worked flawlessly - asciinema recording of the whole process starting with a fresh VM at https://asciinema.org/a/zqwKSWyjuWkwMlsRcqfRjExc3. Similarly compiling the bro 2.5.1 source did not yield any problems during compilation. If you can still see this, please give exact step-by-step instructions on how to reproduce your problems and I will try to look into it. Thanks a lot, Johanna On Wed, Oct 04, 2017 at 02:58:16AM +0000, Miller, Brad L wrote: > I destroyed that VM again and repeated the package install on Ubuntu 14.04, the method worked with no issues seen in 16.04. > > -----Original Message----- > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Miller, Brad L > Sent: Tuesday, October 03, 2017 10:16 PM > To: Johanna Amann > Cc: bro at bro.org > Subject: Re: [Bro] Issue compiling from source on VM / Digital Ocean > > The repo method was as described here: https://www.bro.org/download/packages.html (Ubuntu 16.04) > > It was newly provisioned VPS with Digital Ocean, running no other utilities at the time. It's unlikely it was running out of memory, but I destroyed it soon after I spun up Ubuntu 14.04 to test. As I mentioned, building from source the same exact way on Ubuntu 14.04 did not have the compiler error. I didn't try and install from repo on 14.04. > > -----Original Message----- > From: Johanna Amann [mailto:johanna at icir.org] > Sent: Tuesday, October 03, 2017 7:15 PM > To: Miller, Brad L > Cc: bro at bro.org > Subject: Re: [Bro] Issue compiling from source on VM / Digital Ocean > > For the repo error (which I assume is when trying to install binary packages, not when trying to install from source) - could you give the exact steps that you perform to get that error (in the optimal case all commands starting from a clean installation). > > For the compilation error - is there a chance that your VM is running out of memory? > > Johanna > > On Tue, Oct 03, 2017 at 10:41:49PM +0000, Miller, Brad L wrote: > > I noticed this started occurring with 2.5.x. Attempting to make from source. All dependencies met and following instructions from Digital Ocean. Whether from a git clone or pulling the source files in manually, always leads to an internal compiler error. System is Ubuntu 16.04, fully updated. > > > > Additionally, when attempting to install from the opensuse repo, there is an umet dependency error. > > > > I recall (not confirmed) that I attempted to build from source in a KVM guest on my own machine (at Brocon) and had the exact same compiler error. > > > > Is there a new incompatibility with virtual systems? > > > > Repo error > > Some packages could not be installed. This may mean that you have > > requested an impossible situation or if you are using the unstable > > distribution that some required packages have not yet been created or > > been moved out of Incoming. > > The following information may help to resolve the situation: > > > > The following packages have unmet dependencies: > > bro : Depends: bro-core (= 2.5.1-0) but it is not going to be installed > > Depends: broctl (= 2.5.1-0) but it is not going to be installed > > > > Compiler error > > > > [ 86%] Building CXX object src/CMakeFiles/bro.dir/Func.cc.o > > c++: internal compiler error: Killed (program cc1plus) > > Please submit a full bug report, > > with preprocessed source if appropriate. > > See for instructions. > > src/CMakeFiles/bro.dir/build.make:1222: recipe for target > > 'src/CMakeFiles/bro.dir/Func.cc.o' failed > > make[3]: *** [src/CMakeFiles/bro.dir/Func.cc.o] Error 4 > > make[3]: Leaving directory '/root/bro/build' > > CMakeFiles/Makefile2:828: recipe for target > > 'src/CMakeFiles/bro.dir/all' failed > > make[2]: *** [src/CMakeFiles/bro.dir/all] Error 2 > > make[2]: Leaving directory '/root/bro/build' > > Makefile:149: recipe for target 'all' failed > > make[1]: *** [all] Error 2 > > make[1]: Leaving directory '/root/bro/build' > > Makefile:15: recipe for target 'all' failed > > make: *** [all] Error 2 > > > > > > Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email. > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > > Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email. > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > Please be aware that if you reply directly to this particular message, your reply may not be secure. Do not use email to send us communications that contain unencrypted confidential information such as passwords, account numbers or Social Security numbers. If you must provide this type of information, please visit comerica.com to submit a secure form using any of the ?Contact Us? forms. In addition, you should not send via email any inquiry or request that may be time sensitive. The information in this e-mail is confidential. It is intended for the individual or entity to whom it is addressed. If you have received this email in error, please destroy or delete the message and advise the sender of the error by return email. > From bhklimk at gmail.com Thu Oct 5 04:31:10 2017 From: bhklimk at gmail.com (Benjamin Klimkowski) Date: Thu, 5 Oct 2017 07:31:10 -0400 Subject: [Bro] Parsing Extension Mechanisms for DNS (EDNS0) Fields Message-ID: All, I am trying to analyze the client subnet option (RFC 7871) in some of the network traffic where it is set. It is not appear in dns.log. Also it appears to cause an issue in weird.log. Is this a known issue or bug? Thanks, Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171005/ded20bbf/attachment.html From jmellander at lbl.gov Thu Oct 5 12:22:29 2017 From: jmellander at lbl.gov (Jim Mellander) Date: Thu, 5 Oct 2017 12:22:29 -0700 Subject: [Bro] Parsing Extension Mechanisms for DNS (EDNS0) Fields In-Reply-To: References: Message-ID: Hi Ben: If you look at share/bro/base/protocols/dns/main.bro, you will find that the EDNS section is commented out, and labeled: TODO: figure out how to handle these So, its another area of Bro that Needs Work? Take care, Jim Mellander ESNet On Thu, Oct 5, 2017 at 4:31 AM, Benjamin Klimkowski wrote: > All, > > I am trying to analyze the client subnet option (RFC 7871) in some of the > network traffic where it is set. It is not appear in dns.log. Also it > appears to cause an issue in weird.log. > > Is this a known issue or bug? > > Thanks, > > Ben > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171005/19dcbac0/attachment.html From Hafiz.Ul-Asad.1 at city.ac.uk Fri Oct 6 04:27:19 2017 From: Hafiz.Ul-Asad.1 at city.ac.uk (Ul Asad, Hafiz) Date: Fri, 6 Oct 2017 11:27:19 +0000 Subject: [Bro] source ip and destination ip have been swaped in bro logs Message-ID: Hi, I have noticed in my bor notices.log, that a for a connection, the source_ip and destination_ip, as well as the corresponding ports, have been swaped. Is there any explaination for it somewhere and how to find that for which connection bro does this? Regards Asad -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171006/21befae2/attachment.html From seth at corelight.com Fri Oct 6 06:25:50 2017 From: seth at corelight.com (Seth Hall) Date: Fri, 06 Oct 2017 09:25:50 -0400 Subject: [Bro] source ip and destination ip have been swaped in bro logs In-Reply-To: References: Message-ID: <1A72AD35-552D-4B76-BAFF-660097FF4171@corelight.com> On 6 Oct 2017, at 7:27, Ul Asad, Hafiz wrote: > I have noticed in my bor notices.log, that a for a connection, the > source_ip and destination_ip, as well as the corresponding ports, have > been swaped. Is there any explaination for it somewhere and how to > find that for which connection bro does this? Bro will try to get the relationship between who "originated" and "responded" to the connection correct. Let's imagine the case that the initial syn packet for an http connection was dropped so the first packet that Bro saw was source port 80 and the dest port will be some arbitrary high number. Bro will look at the connection and make a guess that it may be looking at the connection backwards and flip it. The fact that the flip happened is also indicated in the "history" field in the conn log with the caret "^". There are a lot of other scenarios that could lead to the same behavior too. If you'd like to go further into the particular case you're encountering, you could send a conn log entry that looks problematic to the list (with IP addresses hidden) and we may be able to diagnose the particular problem you're seeing. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From Hafiz.Ul-Asad.1 at city.ac.uk Fri Oct 6 07:08:59 2017 From: Hafiz.Ul-Asad.1 at city.ac.uk (Ul Asad, Hafiz) Date: Fri, 6 Oct 2017 14:08:59 +0000 Subject: [Bro] source ip and destination ip have been swaped in bro logs In-Reply-To: <1A72AD35-552D-4B76-BAFF-660097FF4171@corelight.com> References: , <1A72AD35-552D-4B76-BAFF-660097FF4171@corelight.com> Message-ID: Thanks for this, this is really useful. Asad Get Outlook for Android ________________________________ From: Seth Hall Sent: Friday, October 6, 2017 2:25:50 PM To: Ul Asad, Hafiz Cc: bro at bro.org Subject: Re: [Bro] source ip and destination ip have been swaped in bro logs On 6 Oct 2017, at 7:27, Ul Asad, Hafiz wrote: > I have noticed in my bor notices.log, that a for a connection, the > source_ip and destination_ip, as well as the corresponding ports, have > been swaped. Is there any explaination for it somewhere and how to > find that for which connection bro does this? Bro will try to get the relationship between who "originated" and "responded" to the connection correct. Let's imagine the case that the initial syn packet for an http connection was dropped so the first packet that Bro saw was source port 80 and the dest port will be some arbitrary high number. Bro will look at the connection and make a guess that it may be looking at the connection backwards and flip it. The fact that the flip happened is also indicated in the "history" field in the conn log with the caret "^". There are a lot of other scenarios that could lead to the same behavior too. If you'd like to go further into the particular case you're encountering, you could send a conn log entry that looks problematic to the list (with IP addresses hidden) and we may be able to diagnose the particular problem you're seeing. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171006/97596b02/attachment-0001.html From haoscs at gmail.com Mon Oct 9 06:33:42 2017 From: haoscs at gmail.com (Shuai Hao) Date: Mon, 9 Oct 2017 09:33:42 -0400 Subject: [Bro] A lower level interface Message-ID: Hi All, I've seen many discussions referring to the Bro as an alternative of libnids. I wonder that can we use the similar lower-level interface similar to libnids in Bro (e.g., for the tcp assembly)? We would like to explore the string features of packets, while keeping to leverage Bro's high-level events. Regards, Shuai -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171009/4dbf9127/attachment.html From jlamps at sandia.gov Mon Oct 9 10:52:22 2017 From: jlamps at sandia.gov (Lamps, Jereme) Date: Mon, 9 Oct 2017 17:52:22 +0000 Subject: [Bro] Building Bro 2.5.1 with PF_Ring 6.6 configure/cmake not working Message-ID: Hello, I am trying to build Bro with PF_Ring support. I followed the instructions https://www.bro.org/sphinx/configuration/index.html#installing-pf-ring and built pfring to /opt/pfring-6.6. When I try to run: ?./configure --with-pcap=/opt/pfring-6.6? it does not seem to work. -- Found PCAP: /usr/local/lib/libpcap.so -- Performing Test PCAP_LINKS_SOLO -- Performing Test PCAP_LINKS_SOLO - Success -- Looking for pcap_get_pfring_id -- Looking for pcap_get_pfring_id - not found Digging around cmake Have a great day! Jereme -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171009/3b5b412e/attachment.html From jazoff at illinois.edu Mon Oct 9 11:30:23 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 9 Oct 2017 18:30:23 +0000 Subject: [Bro] Building Bro 2.5.1 with PF_Ring 6.6 configure/cmake not working In-Reply-To: References: Message-ID: <98081810-BDFC-4183-976E-AD920B874C00@illinois.edu> > On Oct 9, 2017, at 1:52 PM, Lamps, Jereme wrote: > > -- Found PCAP: /usr/local/lib/libpcap.so > Where did this file come from? ? Justin Azoff From dopheide at gmail.com Mon Oct 9 12:57:14 2017 From: dopheide at gmail.com (Mike Dopheide) Date: Mon, 9 Oct 2017 13:57:14 -0600 Subject: [Bro] myricom SNFv3 / kernel upgrade CentOS 7 Message-ID: Another myricom heads-up. Not sure how many of you are using or keeping your myri_snf rpm up to date, but the latest kernel update breaks if you're still using myri-snf 3.0.9 due to the removal of trans_start from net_device. You'll want to download the new 3.0.12 .tar.gz from myricom. The tarball contains the new rpm. "rpm -U blaa.rpm" should rebuild the module for you, then just run '/opt/snf/sbin/myri_start_stop start'. Good times. -Dop -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171009/a9b15c02/attachment.html From cchiaverini at bnl.gov Mon Oct 9 13:38:22 2017 From: cchiaverini at bnl.gov (Chiaverini, Christian) Date: Mon, 9 Oct 2017 20:38:22 +0000 Subject: [Bro] Building Bro 2.5.1 with PF_Ring 6.6 configure/cmake not working In-Reply-To: References: Message-ID: <967F0B5E-D1BD-48A6-8A59-62EECC2E4455@bnl.gov> Not sure how you installed pfring (assuming just the prefix for /opt/pfring-6.6 and no indicating the lib dir) but it may need to be: ./configure --with-pcap=/opt/pfring-6.6/lib If not, then you can do a find /opt/pfring-6.6 ?name libpcap.so The result with be the directory you need to put in the ?--with-pcap? part (exclude the ?libpcap.so?). -- Regards, Chris Chiaverini Cyber Security Operations Brookhaven National Laboratory Upton, New York 11973 From: on behalf of "Lamps, Jereme" Date: Monday, October 9, 2017 at 1:58 PM To: "bro at bro.org" Subject: [Bro] Building Bro 2.5.1 with PF_Ring 6.6 configure/cmake not working Hello, I am trying to build Bro with PF_Ring support. I followed the instructions https://www.bro.org/sphinx/configuration/index.html#installing-pf-ring and built pfring to /opt/pfring-6.6. When I try to run: ?./configure --with-pcap=/opt/pfring-6.6? it does not seem to work. -- Found PCAP: /usr/local/lib/libpcap.so -- Performing Test PCAP_LINKS_SOLO -- Performing Test PCAP_LINKS_SOLO - Success -- Looking for pcap_get_pfring_id -- Looking for pcap_get_pfring_id - not found Digging around cmake Have a great day! Jereme -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171009/6d1cbbd2/attachment.html From matthieu at treussart.com Mon Oct 9 14:11:38 2017 From: matthieu at treussart.com (matthieu) Date: Mon, 9 Oct 2017 23:11:38 +0200 Subject: [Bro] Check Syntax Bro scripts Message-ID: Hi Are there any Bro rules available via HTTP (like Open EmergingThreats) ? Thanks Matthieu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171009/d9793b53/attachment-0001.html From wren3 at illinois.edu Mon Oct 9 15:10:39 2017 From: wren3 at illinois.edu (Ren, Wenyu) Date: Mon, 9 Oct 2017 22:10:39 +0000 Subject: [Bro] Is there a way to intentionally delay Bro's reading of trace file for something else to finish first? Message-ID: Hello all, I am recently using pybroker to feed some event data to my python program. I use the auto_event to do that and read traffic from a pcap file. However, it takes some time for the broker to establish the connection with my python program but the processing of the traffic starts immediately. As a result, the first part of the traffic is always missing in my python program. The following is how I set up the connection and the utilize the auto_event. I am wondering if there is a way to intentionally delay Bro's processing of the pcap file so that the connection can be established before Bro start to process the traffic. event bro_init() &priority=5 { Broker::enable(); Broker::connect("127.0.0.1", broker_port, 1sec); Broker::auto_event("bro/event/packet_get", FlowLevel::packet_get); Broker::auto_event("bro/event/data_get", DataLevel::data_get); } Any help is appreciated. Thanks a lot. Best, Wenyu Wenyu Ren Ph.D. Candidate Department of Computer Science University of Illinois at Urbana-Champaign From jazoff at illinois.edu Mon Oct 9 15:33:20 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 9 Oct 2017 22:33:20 +0000 Subject: [Bro] Is there a way to intentionally delay Bro's reading of trace file for something else to finish first? In-Reply-To: References: Message-ID: <6624ABDA-7477-4BBD-8CF3-761425BC9ABB@illinois.edu> > On Oct 9, 2017, at 6:10 PM, Ren, Wenyu wrote: > > Hello all, > > I am recently using pybroker to feed some event data to my python program. I use the auto_event to do that and read traffic from a pcap file. However, it takes some time for the broker to establish the connection with my python program but the processing of the traffic starts immediately. As a result, the first part of the traffic is always missing in my python program. The following is how I set up the connection and the utilize the auto_event. I am wondering if there is a way to intentionally delay Bro's processing of the pcap file so that the connection can be established before Bro start to process the traffic. > > event bro_init() &priority=5 > { > Broker::enable(); > Broker::connect("127.0.0.1", broker_port, 1sec); > Broker::auto_event("bro/event/packet_get", FlowLevel::packet_get); > Broker::auto_event("bro/event/data_get", DataLevel::data_get); > } > > Any help is appreciated. Thanks a lot. > > Best, > Wenyu > You can try something like this, not sure if it will work though event resume() { continue_processing(); } event bro_init() &priority=5 { # your existing stuff suspend_processing(); schedule 10secs { resume() }; } You may want to look at the suggestion I wrote up here: http://mailman.icsi.berkeley.edu/pipermail/bro/2017-July/012355.html Having a 'pcapdir' pktsource plugin would solve a lot of problems like this. ? Justin Azoff From anthony.kasza at gmail.com Mon Oct 9 16:03:56 2017 From: anthony.kasza at gmail.com (anthony kasza) Date: Mon, 9 Oct 2017 17:03:56 -0600 Subject: [Bro] Is there a way to intentionally delay Bro's reading of trace file for something else to finish first? In-Reply-To: <6624ABDA-7477-4BBD-8CF3-761425BC9ABB@illinois.edu> References: <6624ABDA-7477-4BBD-8CF3-761425BC9ABB@illinois.edu> Message-ID: Here's a solution I hacked up a couple of years back while trying to scan pcaps for indicators. I believe it's very similar to what Justin replied with. https://github.com/anthonykasza/scratch_pad/blob/master/input_for_pcaps/README.md -AK On Oct 9, 2017 4:35 PM, "Azoff, Justin S" wrote: > > On Oct 9, 2017, at 6:10 PM, Ren, Wenyu wrote: > > > > Hello all, > > > > I am recently using pybroker to feed some event data to my python > program. I use the auto_event to do that and read traffic from a pcap file. > However, it takes some time for the broker to establish the connection with > my python program but the processing of the traffic starts immediately. As > a result, the first part of the traffic is always missing in my python > program. The following is how I set up the connection and the utilize the > auto_event. I am wondering if there is a way to intentionally delay Bro's > processing of the pcap file so that the connection can be established > before Bro start to process the traffic. > > > > event bro_init() &priority=5 > > { > > Broker::enable(); > > Broker::connect("127.0.0.1", broker_port, 1sec); > > Broker::auto_event("bro/event/packet_get", FlowLevel::packet_get); > > Broker::auto_event("bro/event/data_get", DataLevel::data_get); > > } > > > > Any help is appreciated. Thanks a lot. > > > > Best, > > Wenyu > > > > You can try something like this, not sure if it will work though > > event resume() > { > continue_processing(); > } > > event bro_init() &priority=5 > { > # your existing stuff > suspend_processing(); > schedule 10secs { resume() }; > } > > > You may want to look at the suggestion I wrote up here: > > http://mailman.icsi.berkeley.edu/pipermail/bro/2017-July/012355.html > > Having a 'pcapdir' pktsource plugin would solve a lot of problems like > this. > > ? > Justin Azoff > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171009/bfa3609c/attachment.html From fatema.bannatwala at gmail.com Tue Oct 10 05:36:01 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Tue, 10 Oct 2017 08:36:01 -0400 Subject: [Bro] Check Syntax Bro scripts Message-ID: Hi Matthieu, I am not aware of any source available for Bro signatures (rules, if that's what you meant), however, there used to be a script snort2bro that converted snort signatures/rules to corresponding Bro sigs, but not maintained anymore. Not sure what you are looking to solve, but if you know what you are searching for in your traffic, then you might want to take a look at the Bro's Signature Language, to write your own signatures. Here's the link: https://www.bro.org/sphinx/frameworks/signatures.html Hope this helps. -Fatema -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171010/fb8b2b64/attachment.html From jazoff at illinois.edu Tue Oct 10 06:16:47 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 10 Oct 2017 13:16:47 +0000 Subject: [Bro] Is there a way to intentionally delay Bro's reading of trace file for something else to finish first? In-Reply-To: References: <6624ABDA-7477-4BBD-8CF3-761425BC9ABB@illinois.edu> Message-ID: <9192FFDA-6916-47A7-99CE-D40AD2DBF53E@illinois.edu> > On Oct 9, 2017, at 10:31 PM, Ren, Wenyu wrote: > > Hi Anthony and Justin, > > Thanks a lot for your solutions. I think using the suspend and continue works. Actually, I have another question about using pybroker. I have a listener in my python program doing something as follows: > > epl = endpoint("listener") > mql = message_queue("bro/event", epl) > icsq = epl.incoming_connection_status() > > epl.listen(10007, "127.0.0.1") > select.select([icsq.fd()],[],[]) > msgs = icsq.want_pop() > > for m in msgs: > print("incoming connection", m.peer_name, m.status) > assert(m.peer_name == "connector") > assert(m.status == incoming_connection_status.tag_established) > > while True: > select.select([mql.fd()], [], []) > msgs = mql.want_pop() > for m in msgs: > raw_data_queue.put_nowait(m) > gevent.sleep(0) > > I put the listener inside a greenlet which is a coroutine I use for my own purpose. The problem is that I don't know a good way to terminate this python program as soon as the Bro part finishes processing all the trace file. If I just terminate by using Ctrl+C, the current port will not be released and that prevents me from using it in the future. Do you have any good idea about how I should stop this listener and free that port as soon as the Bro stops sending more events? > > Best, > Wenyu You could use the bro_done event to send a "EXIT" message to your python listener telling it that bro is done running and it should exit. The problem with the port sounds like something is not setting SO_REUSEADDR inside broker. ? Justin Azoff From hacecky at jlab.org Tue Oct 10 06:20:19 2017 From: hacecky at jlab.org (Eric Hacecky) Date: Tue, 10 Oct 2017 09:20:19 -0400 (EDT) Subject: [Bro] Check Syntax Bro scripts In-Reply-To: <2016806935.18119472.1507641607407.JavaMail.zimbra@jlab.org> References: Message-ID: <1154608474.18119562.1507641619787.JavaMail.zimbra@jlab.org> As a long time snort user, I asked a similar question when I first started getting into Bro. The short of it is no. If you're looking for a repository that is constantly updated, I don't know of anything like that available. However, if you want a similar function to ET rules available to you, Bro can do that. Based on my needs I decided to write some Bro scripts that perform a similar function to my most triggered ET rules. In the same vein you can find scripts/sigs on github that may be of interest to you. Snort2bro really is the kind of endgame for this and I have wanted to revive this project for a number of years. One of my guys has been working on it for about 6 months and has made some progress, however, he is leaving in December. I intend to continue development myself at that time. Regards, Eric From jdopheid at illinois.edu Tue Oct 10 07:01:30 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Tue, 10 Oct 2017 14:01:30 +0000 Subject: [Bro] Renaming the Bro Project: seeking proposed names from the community Message-ID: Friendly reminder that we are accepting proposed names until December 4th. ------ Jeannette Dopheide Sr. Education, Outreach, and Training Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign On 10/2/17, 9:13 AM, "bro-bounces at bro.org on behalf of Dopheide, Jeannette M" wrote: This year at BroCon we announced that the Bro Project will be changing its name. While ?Bro? was originally meant as an Orwellian reminder of the risk that any monitoring fundamentally entails, it has more recently gained a very different, and quite offensive, reputation (?Bro culture?). To avoid facing instant negative impressions with new users that aren?t aware of the history, the Leadership Team has decided to seek a name change. We are accepting proposed names from the community for two months (due Monday December 4th). The Leadership Team will review the list of possible names and narrow it down to 5 finalists. We will announce the finalists and take a second round of feedback from the community before making the final selection. We hope to announce the new name within the next major release. To submit a proposed name, fill out the form here: https://goo.gl/forms/qwR8s6Yd4H0Bu8Ca2 ------ Jeannette Dopheide Sr. Education, Outreach, and Training Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Tue Oct 10 09:48:51 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 10 Oct 2017 09:48:51 -0700 Subject: [Bro] Is there a way to intentionally delay Bro's reading of trace file for something else to finish first? In-Reply-To: <9192FFDA-6916-47A7-99CE-D40AD2DBF53E@illinois.edu> References: <6624ABDA-7477-4BBD-8CF3-761425BC9ABB@illinois.edu> <9192FFDA-6916-47A7-99CE-D40AD2DBF53E@illinois.edu> Message-ID: <20171010164851.u4k4bnz4heg3hqxj@Beezling.local> Just to point out one thing to prevent future annoyance on your part - broker is currently getting a re-write which includes changed python APIs. The new version of Broker will be used in Bro 2.6 and the old API will no longer work. The current state is not yet merged into master, but you can look at topic/actor-system of the broker repository; the best point probably are the tests in https://github.com/bro/broker/tree/topic/actor-system/tests/python It might be worth to take a short look at the new syntax just so that you know how you might have to adapt things in the future. Johanna On Tue, Oct 10, 2017 at 01:16:47PM +0000, Azoff, Justin S wrote: > > > > On Oct 9, 2017, at 10:31 PM, Ren, Wenyu wrote: > > > > Hi Anthony and Justin, > > > > Thanks a lot for your solutions. I think using the suspend and continue works. Actually, I have another question about using pybroker. I have a listener in my python program doing something as follows: > > > > epl = endpoint("listener") > > mql = message_queue("bro/event", epl) > > icsq = epl.incoming_connection_status() > > > > epl.listen(10007, "127.0.0.1") > > select.select([icsq.fd()],[],[]) > > msgs = icsq.want_pop() > > > > for m in msgs: > > print("incoming connection", m.peer_name, m.status) > > assert(m.peer_name == "connector") > > assert(m.status == incoming_connection_status.tag_established) > > > > while True: > > select.select([mql.fd()], [], []) > > msgs = mql.want_pop() > > for m in msgs: > > raw_data_queue.put_nowait(m) > > gevent.sleep(0) > > > > I put the listener inside a greenlet which is a coroutine I use for my own purpose. The problem is that I don't know a good way to terminate this python program as soon as the Bro part finishes processing all the trace file. If I just terminate by using Ctrl+C, the current port will not be released and that prevents me from using it in the future. Do you have any good idea about how I should stop this listener and free that port as soon as the Bro stops sending more events? > > > > Best, > > Wenyu > > You could use the bro_done event to send a "EXIT" message to your python listener telling it that bro is done running and it should exit. > > The problem with the port sounds like something is not setting SO_REUSEADDR inside broker. > > ? > Justin Azoff > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From johanna at icir.org Tue Oct 10 09:52:17 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 10 Oct 2017 09:52:17 -0700 Subject: [Bro] Is there a way to intentionally delay Bro's reading of trace file for something else to finish first? In-Reply-To: References: <6624ABDA-7477-4BBD-8CF3-761425BC9ABB@illinois.edu> Message-ID: <20171010165217.dfm4s45vtohv4mhi@Beezling.local> I just wanted to point out that the Bro unit tests themselves also use an approach like this. See for example https://github.com/bro/bro/blob/master/testing/btest/scripts/policy/frameworks/intel/seen/certs.bro Johanna On Mon, Oct 09, 2017 at 11:03:56PM +0000, anthony kasza wrote: > Here's a solution I hacked up a couple of years back while trying to scan > pcaps for indicators. I believe it's very similar to what Justin replied > with. > > https://github.com/anthonykasza/scratch_pad/blob/master/input_for_pcaps/README.md > > -AK > > On Oct 9, 2017 4:35 PM, "Azoff, Justin S" wrote: > > > > On Oct 9, 2017, at 6:10 PM, Ren, Wenyu wrote: > > > > > > Hello all, > > > > > > I am recently using pybroker to feed some event data to my python > > program. I use the auto_event to do that and read traffic from a pcap file. > > However, it takes some time for the broker to establish the connection with > > my python program but the processing of the traffic starts immediately. As > > a result, the first part of the traffic is always missing in my python > > program. The following is how I set up the connection and the utilize the > > auto_event. I am wondering if there is a way to intentionally delay Bro's > > processing of the pcap file so that the connection can be established > > before Bro start to process the traffic. > > > > > > event bro_init() &priority=5 > > > { > > > Broker::enable(); > > > Broker::connect("127.0.0.1", broker_port, 1sec); > > > Broker::auto_event("bro/event/packet_get", FlowLevel::packet_get); > > > Broker::auto_event("bro/event/data_get", DataLevel::data_get); > > > } > > > > > > Any help is appreciated. Thanks a lot. > > > > > > Best, > > > Wenyu > > > > > > > You can try something like this, not sure if it will work though > > > > event resume() > > { > > continue_processing(); > > } > > > > event bro_init() &priority=5 > > { > > # your existing stuff > > suspend_processing(); > > schedule 10secs { resume() }; > > } > > > > > > You may want to look at the suggestion I wrote up here: > > > > http://mailman.icsi.berkeley.edu/pipermail/bro/2017-July/012355.html > > > > Having a 'pcapdir' pktsource plugin would solve a lot of problems like > > this. > > > > ? > > Justin Azoff > > > > > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Tue Oct 10 10:01:44 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 10 Oct 2017 10:01:44 -0700 Subject: [Bro] A lower level interface In-Reply-To: References: Message-ID: <20171010170144.vyzk2f4igyfmxa3j@Beezling.local> Hi, > I've seen many discussions referring to the Bro as an alternative of > libnids. Out of curiosity - where did you see discussions like that? I do not really know much about libnids, but from the readme it seems that libnids is a library that mostly implements TCP reassembly. While this is a part of what Bro does, it only is a small part of it; obviously the main focus of Bro is on a different layer. > I wonder that can we use the similar lower-level interface similar > to libnids in Bro (e.g., for the tcp assembly)? I am not quite sure what you are looking for here, could you perhaps expand a bit on that? The lowest level access that you can get is probably by writing a custom (c++) analyzer that gets passed either the reassembled TCP payloads. (Or just the raw packets in case of UDP). > We would like to explore the string features of packets, while keeping > to leverage Bro's high-level events. You can use something like the new_packet events in Bro to get access to individual packet information. However, there is a performance penalty associated with this (script-level events are fairly expensive and usually there are a few per connection, not a few per packet). For anything carrying a significant amount of traffic that approach probably is not viable. It depends a bit on what you want to do - preprocessing in C++ and then bubbling up more high-level events should be a more realisitic choice. I hope this helps, Johanna From johanna at icir.org Tue Oct 10 10:06:43 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 10 Oct 2017 10:06:43 -0700 Subject: [Bro] Losing events associated with Signature Matching In-Reply-To: References: Message-ID: <20171010170643.vly7zqvjqyfzgkva@Beezling.local> Hi, Since it sounds that the same program runs well on a beefier machnine, I assume that this is a case of your fanless atom machine not able to keep up. Did you check if there is any capture loss? (Packet loss statistics should be added to notice.log by default). Johanna On Tue, Sep 26, 2017 at 03:38:20PM +0000, Shuai Hao wrote: > Hi All, > > My Bro program shows a wired behavior. We leverage the signature framework > to capture embedded components in HTTP replies (http-reply-body) as well as > the file download (tcp payload). However, we lose many events associated > with the signature (only around 1/3 shown). > > The exactly same program actually runs well on another desktop (capturing > all signature matching we issued). I would be appreciate if anyone can have > a clue on the problem. > > The machine running bro is fanless computer with Intel Atom and Ubuntu > 16.04. It is almost dedicated to the Bro monitoring so it shouldn't be > performance issue. > > The signature matching is quite straightforward: we define some simple > signature patterns, load those signatures to BroControl, and pull some > fields from corresponding log files via a broccoli python client. > > We do capture some signature matching events, but also lose many that > should be captured. Those events are not shown in signatures.log; it means > that they are either failure of capturing or dropped by Bro Control, rather > than the problem of python client. > > BTW, we use File Analysis to capture the file downloads, it works well as > expected. > > Thanks very much for any comments~ > > Cheers, > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Tue Oct 10 10:08:32 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 10 Oct 2017 10:08:32 -0700 Subject: [Bro] Netflow and bro In-Reply-To: References: Message-ID: <20171010170832.y3zumutyei2mqx4j@Beezling.local> Hi, > Is there a decoder for Netflow, such that one could use bro to collect and > log Netflow packets seen by a hardware tap, from multiple sources, in a > similar fashion to how Bro handles syslog? while there was support for this in the past, it was removed a while ago (I think the last version supporting this was 1.5, and even then it was not well tested and there were not scripts for it as far as I know). So - sadly the answer here is no. Johanna > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From haoscs at gmail.com Tue Oct 10 11:06:51 2017 From: haoscs at gmail.com (Shuai Hao) Date: Tue, 10 Oct 2017 14:06:51 -0400 Subject: [Bro] A lower level interface In-Reply-To: <20171010170144.vyzk2f4igyfmxa3j@Beezling.local> References: <20171010170144.vyzk2f4igyfmxa3j@Beezling.local> Message-ID: Thanks for your reply, Johanna. Basically I am looking for an interface by which we can examine and extract the features of byte stream (or strings) from the traffic (TCP payload), and then we will feed the stream to our analyzer (e.g., via BinPac). Currently I am looking at the tcp_contents; I think it might be sufficient so I don't have to use tcp_packet or new_packet. Although there is also a performance note for tcp_contents in the manual, I assume that it would be better than using tcp_packet or new_packet (is this true? I do care about the performance). Thank you very much! Cheers, Shuai On Tue, Oct 10, 2017 at 1:01 PM, Johanna Amann wrote: > Hi, > > > I've seen many discussions referring to the Bro as an alternative of > > libnids. > > Out of curiosity - where did you see discussions like that? > > I do not really know much about libnids, but from the readme it seems that > libnids is a library that mostly implements TCP reassembly. While this is > a part of what Bro does, it only is a small part of it; obviously the > main focus of Bro is on a different layer. > > > I wonder that can we use the similar lower-level interface similar > > to libnids in Bro (e.g., for the tcp assembly)? > > I am not quite sure what you are looking for here, could you perhaps > expand a bit on that? > > The lowest level access that you can get is probably by writing a custom > (c++) analyzer that gets passed either the reassembled TCP payloads. (Or > just the raw packets in case of UDP). > > > We would like to explore the string features of packets, while keeping > > to leverage Bro's high-level events. > > You can use something like the new_packet events in Bro to get access to > individual packet information. However, there is a performance penalty > associated with this (script-level events are fairly expensive and usually > there are a few per connection, not a few per packet). > > For anything carrying a significant amount of traffic that approach > probably is not viable. > > It depends a bit on what you want to do - preprocessing in C++ and then > bubbling up more high-level events should be a more realisitic choice. > > I hope this helps, > Johanna > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171010/4770ce16/attachment.html From johanna at icir.org Tue Oct 10 11:22:22 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 10 Oct 2017 11:22:22 -0700 Subject: [Bro] A lower level interface In-Reply-To: References: <20171010170144.vyzk2f4igyfmxa3j@Beezling.local> Message-ID: <20171010182218.hdpl23ay6sqlr7w4@Beezling.local> > Basically I am looking for an interface by which we can examine and extract > the features of byte stream (or strings) from the traffic (TCP payload), > and then we will feed the stream to our analyzer (e.g., via BinPac). > Currently I am looking at the tcp_contents; I think it might be sufficient > so I don't have to use tcp_packet or new_packet. I still don't quite get what you are planning to do here. Do you plan to do some kind of signature to figure out that something is a specific protocol (so match certain byte sequences)? Or do you really want to do something more complex that needs scripting? tcp_contents is probably less expensive than the other named choices, but it probably still is pretty heavyweight. Johanna From haoscs at gmail.com Tue Oct 10 11:41:46 2017 From: haoscs at gmail.com (Shuai Hao) Date: Tue, 10 Oct 2017 14:41:46 -0400 Subject: [Bro] A lower level interface In-Reply-To: <20171010182218.hdpl23ay6sqlr7w4@Beezling.local> References: <20171010170144.vyzk2f4igyfmxa3j@Beezling.local> <20171010182218.hdpl23ay6sqlr7w4@Beezling.local> Message-ID: The thing we are planing to do is similar to the [protocol reverse engineering + traffic pattern recognition]; so we consider that we may need the lower level interface to inspect the byte stream since the patterns that we want to identify (e.g., a serious of connection activities) would involve various protocols. We do have the signature part to accomplish the payload matching. But it may be not sufficient when we consider the traffic recognition (e.g., generating a signature that involve various protocols and network components). Thanks for your relies. It do helps much. On Tue, Oct 10, 2017 at 2:22 PM, Johanna Amann wrote: > > Basically I am looking for an interface by which we can examine and > extract > > the features of byte stream (or strings) from the traffic (TCP payload), > > and then we will feed the stream to our analyzer (e.g., via BinPac). > > Currently I am looking at the tcp_contents; I think it might be > sufficient > > so I don't have to use tcp_packet or new_packet. > > I still don't quite get what you are planning to do here. Do you plan to > do some kind of signature to figure out that something is a specific > protocol (so match certain byte sequences)? Or do you really want to do > something more complex that needs scripting? > > tcp_contents is probably less expensive than the other named choices, but > it probably still is pretty heavyweight. > > Johanna > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171010/df71cf88/attachment.html From matthieu at treussart.com Tue Oct 10 11:43:23 2017 From: matthieu at treussart.com (matthieu) Date: Tue, 10 Oct 2017 20:43:23 +0200 Subject: [Bro] Community source for rules In-Reply-To: References: Message-ID: <92EC931F-AD10-4AC9-84F5-E030656E8A6C@treussart.com> Hi Thank you for your reply. Yes I know snort2bro, but I use Snort or Suricata for this rules. I was hoping there was a Bro rules contribution available on the Internet. Generic rules that answer to the actuality like WannaCry (SMB) ? Matthieu > On 10 Oct 2017, at 14:36, fatema bannatwala wrote: > > Hi Matthieu, > > I am not aware of any source available for Bro signatures (rules, if that's what you meant), > however, there used to be a script snort2bro that converted snort signatures/rules to corresponding Bro sigs, but not maintained anymore. > > Not sure what you are looking to solve, but if you know what you are searching for in your traffic, > then you might want to take a look at the Bro's Signature Language, to write your own signatures. > Here's the link: https://www.bro.org/sphinx/frameworks/signatures.html > > Hope this helps. > > -Fatema -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171010/90613354/attachment.html From johanna at icir.org Tue Oct 10 11:50:59 2017 From: johanna at icir.org (Johanna Amann) Date: Tue, 10 Oct 2017 11:50:59 -0700 Subject: [Bro] A lower level interface In-Reply-To: References: <20171010170144.vyzk2f4igyfmxa3j@Beezling.local> <20171010182218.hdpl23ay6sqlr7w4@Beezling.local> Message-ID: <20171010185059.hnlzipha6hgvr6sq@Beezling.local> Hi again, > The thing we are planing to do is similar to the [protocol reverse > engineering + traffic pattern recognition]; so we consider that we may need > the lower level interface to inspect the byte stream since the patterns > that we want to identify (e.g., a serious of connection activities) would > involve various protocols. It sadly is really a bit hard to tell what exactly the best starting point is without knowing the exact problem. You mention connection activities - does that mean activities inside the same connection or activities within different connetions? If it is the latter - you could potentially use signatures to identify "interesting" connections and use Bro script level events to tie cross-connection information together. If signatures for some reason are not enough, it depends a bit on your traffic. If you only want this to run in rather low-traffic environments, it might be ok to use the low-level events like tcp_payload. If not - your only choice quickly becomes to write a C++ analyzer, which then once again can raise events. You even can write several analyzers, one which only tries to deduce if a connection is interesting, which then in turn can forward data to more specific analyzers if interesting data is found. I hope that helps; I am sorry that I am not more specific, but given that I still don't 100% understand what you are trying to do this is the best I can do. Johanna > > We do have the signature part to accomplish the payload matching. But it > may be not sufficient when we consider the traffic recognition (e.g., > generating a signature that involve various protocols and network > components). > > Thanks for your relies. It do helps much. > > > On Tue, Oct 10, 2017 at 2:22 PM, Johanna Amann wrote: > > > > Basically I am looking for an interface by which we can examine and > > extract > > > the features of byte stream (or strings) from the traffic (TCP payload), > > > and then we will feed the stream to our analyzer (e.g., via BinPac). > > > Currently I am looking at the tcp_contents; I think it might be > > sufficient > > > so I don't have to use tcp_packet or new_packet. > > > > I still don't quite get what you are planning to do here. Do you plan to > > do some kind of signature to figure out that something is a specific > > protocol (so match certain byte sequences)? Or do you really want to do > > something more complex that needs scripting? > > > > tcp_contents is probably less expensive than the other named choices, but > > it probably still is pretty heavyweight. > > > > Johanna > > From fatema.bannatwala at gmail.com Tue Oct 10 12:16:29 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Tue, 10 Oct 2017 15:16:29 -0400 Subject: [Bro] Community source for rules In-Reply-To: <92EC931F-AD10-4AC9-84F5-E030656E8A6C@treussart.com> References: <92EC931F-AD10-4AC9-84F5-E030656E8A6C@treussart.com> Message-ID: Then, I think you might want to look at the Bro scripting language, although still you have to script what you are looking for. Bro has started this awesome Bro-pkg manager project, which is similar to a central repository, for hosting the various Bro scripts that community can get benefit from: Here's the list of packages, available for the community to download and install: https://github.com/bro/packages Also, there are many individual Bro scripts available on github. If interested, there's this script from Fox-IT regarding ransomeware detection using SMB: https://github.com/fox-it/bro-scripts/tree/master/smb-ransomware -Fatema. On Tue, Oct 10, 2017 at 2:43 PM, matthieu wrote: > Hi > Thank you for your reply. > > Yes I know snort2bro, but I use Snort or Suricata for this rules. > I was hoping there was a Bro rules contribution available on the Internet. > Generic rules that answer to the actuality like WannaCry (SMB) ? > > Matthieu > > > > > On 10 Oct 2017, at 14:36, fatema bannatwala > wrote: > > Hi Matthieu, > > I am not aware of any source available for Bro signatures (rules, if > that's what you meant), > however, there used to be a script snort2bro that converted snort > signatures/rules to corresponding Bro sigs, but not maintained anymore. > > Not sure what you are looking to solve, but if you know what you are > searching for in your traffic, > then you might want to take a look at the Bro's Signature Language, to > write your own signatures. > Here's the link: https://www.bro.org/sphinx/frameworks/signatures.html > > Hope this helps. > > -Fatema > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171010/1823fd14/attachment.html From wren3 at illinois.edu Tue Oct 10 12:18:23 2017 From: wren3 at illinois.edu (Ren, Wenyu) Date: Tue, 10 Oct 2017 19:18:23 +0000 Subject: [Bro] Is there a way to intentionally delay Bro's reading of trace file for something else to finish first? In-Reply-To: <9192FFDA-6916-47A7-99CE-D40AD2DBF53E@illinois.edu> References: <6624ABDA-7477-4BBD-8CF3-761425BC9ABB@illinois.edu> <9192FFDA-6916-47A7-99CE-D40AD2DBF53E@illinois.edu> Message-ID: <4C902DFE-883D-46A4-AB61-87BB22378DC1@illinois.edu> Hi Justin, Thanks for the reply. That?s also what I planned to do. Do you have any idea what function is used at the python side to close the connection? The toy example in the test folder does not have that part included. Thanks a lot. Best, Wenyu On Oct 10, 2017, at 8:16 AM, Azoff, Justin S > wrote: On Oct 9, 2017, at 10:31 PM, Ren, Wenyu > wrote: Hi Anthony and Justin, Thanks a lot for your solutions. I think using the suspend and continue works. Actually, I have another question about using pybroker. I have a listener in my python program doing something as follows: epl = endpoint("listener") mql = message_queue("bro/event", epl) icsq = epl.incoming_connection_status() epl.listen(10007, "127.0.0.1") select.select([icsq.fd()],[],[]) msgs = icsq.want_pop() for m in msgs: print("incoming connection", m.peer_name, m.status) assert(m.peer_name == "connector") assert(m.status == incoming_connection_status.tag_established) while True: select.select([mql.fd()], [], []) msgs = mql.want_pop() for m in msgs: raw_data_queue.put_nowait(m) gevent.sleep(0) I put the listener inside a greenlet which is a coroutine I use for my own purpose. The problem is that I don't know a good way to terminate this python program as soon as the Bro part finishes processing all the trace file. If I just terminate by using Ctrl+C, the current port will not be released and that prevents me from using it in the future. Do you have any good idea about how I should stop this listener and free that port as soon as the Bro stops sending more events? Best, Wenyu You could use the bro_done event to send a "EXIT" message to your python listener telling it that bro is done running and it should exit. The problem with the port sounds like something is not setting SO_REUSEADDR inside broker. ? Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171010/36b1787c/attachment.html From haoscs at gmail.com Tue Oct 10 12:28:53 2017 From: haoscs at gmail.com (Shuai Hao) Date: Tue, 10 Oct 2017 15:28:53 -0400 Subject: [Bro] A lower level interface In-Reply-To: <20171010185059.hnlzipha6hgvr6sq@Beezling.local> References: <20171010170144.vyzk2f4igyfmxa3j@Beezling.local> <20171010182218.hdpl23ay6sqlr7w4@Beezling.local> <20171010185059.hnlzipha6hgvr6sq@Beezling.local> Message-ID: > > It sadly is really a bit hard to tell what exactly the best starting point > is without knowing the exact problem. You mention connection activities - > does that mean activities inside the same connection or activities within > different connetions? > > If it is the latter - you could potentially use signatures to identify > "interesting" connections and use Bro script level events to tie > cross-connection information together. > Johanna, thanks for your patient and detailed replies. I think it is the latter. For example, some external connections (let's say a HTTP Get/Post to server A, with a binary string with signature S1) will raise some activities of local network components B and C, where the traffic is associated with a signature S2. Then we would like to *learn* such a pattern (HTTP_Get/Post_A, S1, B, C, S2) as a pattern signature. So we consider "tokenize" the byte stream to extract and cluster the strings from raw payload. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171010/9c2f4d2f/attachment-0001.html From jazoff at illinois.edu Tue Oct 10 13:01:20 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 10 Oct 2017 20:01:20 +0000 Subject: [Bro] Is there a way to intentionally delay Bro's reading of trace file for something else to finish first? In-Reply-To: <4C902DFE-883D-46A4-AB61-87BB22378DC1@illinois.edu> References: <6624ABDA-7477-4BBD-8CF3-761425BC9ABB@illinois.edu> <9192FFDA-6916-47A7-99CE-D40AD2DBF53E@illinois.edu> <4C902DFE-883D-46A4-AB61-87BB22378DC1@illinois.edu> Message-ID: <1CEB041B-02BA-44BF-A93A-2AE4CD2A0486@illinois.edu> Just exiting your while loop and letting things get garbage collected should probably work.. failing that, it would be something like epl.close() or epl.disconnect() ? Justin Azoff > On Oct 10, 2017, at 3:18 PM, Ren, Wenyu wrote: > > Hi Justin, > > Thanks for the reply. That?s also what I planned to do. Do you have any idea what function is used at the python side to close the connection? The toy example in the test folder does not have that part included. > > Thanks a lot. > > Best, > Wenyu > >> On Oct 10, 2017, at 8:16 AM, Azoff, Justin S wrote: >> >> >> >>> On Oct 9, 2017, at 10:31 PM, Ren, Wenyu wrote: >>> >>> Hi Anthony and Justin, >>> >>> Thanks a lot for your solutions. I think using the suspend and continue works. Actually, I have another question about using pybroker. I have a listener in my python program doing something as follows: >>> >>> epl = endpoint("listener") >>> mql = message_queue("bro/event", epl) >>> icsq = epl.incoming_connection_status() >>> >>> epl.listen(10007, "127.0.0.1") >>> select.select([icsq.fd()],[],[]) >>> msgs = icsq.want_pop() >>> >>> for m in msgs: >>> print("incoming connection", m.peer_name, m.status) >>> assert(m.peer_name == "connector") >>> assert(m.status == incoming_connection_status.tag_established) >>> >>> while True: >>> select.select([mql.fd()], [], []) >>> msgs = mql.want_pop() >>> for m in msgs: >>> raw_data_queue.put_nowait(m) >>> gevent.sleep(0) >>> >>> I put the listener inside a greenlet which is a coroutine I use for my own purpose. The problem is that I don't know a good way to terminate this python program as soon as the Bro part finishes processing all the trace file. If I just terminate by using Ctrl+C, the current port will not be released and that prevents me from using it in the future. Do you have any good idea about how I should stop this listener and free that port as soon as the Bro stops sending more events? >>> >>> Best, >>> Wenyu >> >> You could use the bro_done event to send a "EXIT" message to your python listener telling it that bro is done running and it should exit. >> >> The problem with the port sounds like something is not setting SO_REUSEADDR inside broker. >> >> ? >> Justin Azoff > From cchiaverini at bnl.gov Tue Oct 10 13:10:27 2017 From: cchiaverini at bnl.gov (Chiaverini, Christian) Date: Tue, 10 Oct 2017 20:10:27 +0000 Subject: [Bro] myricom SNFv3 / kernel upgrade CentOS 7 In-Reply-To: References: Message-ID: <72865279-E791-450A-A8A3-3E42E4F09AAC@bnl.gov> I ran into this issue? do not use 3.0.11, use 3.0.12 which came out recently. 3.0.11 also broke setting the ring size in Bro?s node.cfg. I just installed SNF 3.0.12 and it works with kernel 3.10.0-693.2.2 (RH/CentOS 7.4) also addressing the node.cfg issue. -- Regards, Chris Chiaverini From: on behalf of Mike Dopheide Date: Monday, October 9, 2017 at 4:07 PM To: "bro at bro.org" Subject: [Bro] myricom SNFv3 / kernel upgrade CentOS 7 Another myricom heads-up. Not sure how many of you are using or keeping your myri_snf rpm up to date, but the latest kernel update breaks if you're still using myri-snf 3.0.9 due to the removal of trans_start from net_device. You'll want to download the new 3.0.12 .tar.gz from myricom. The tarball contains the new rpm. "rpm -U blaa.rpm" should rebuild the module for you, then just run '/opt/snf/sbin/myri_start_stop start'. Good times. -Dop -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171010/b07dca38/attachment.html From apumphrey at bricata.com Wed Oct 11 08:49:35 2017 From: apumphrey at bricata.com (Adam Pumphrey) Date: Wed, 11 Oct 2017 15:49:35 +0000 Subject: [Bro] Community source for rules In-Reply-To: References: <92EC931F-AD10-4AC9-84F5-E030656E8A6C@treussart.com> Message-ID: I also suggest looking at Bro?s Intelligence Framework, https://www.bro.org/sphinx-git/frameworks/intel.html. This is how Bro consumes and makes use of threat intel indicators, which is essentially what the ET rule feeds contain. There are many intel indicator sources available, some require more effort than others to integrate. As mentioned some tools exist that can help with that. If you?re looking for an indicator source(s), Criticalstack offers a free feed aggregation service that directly integrates with Bro?s Intel Framework. It?s easy to use and a good tool for quickly getting external indicator sources in. Worth a look if you?re exploring how threat intel, supplementary to ET rule feeds, can be used. Adam From: on behalf of fatema bannatwala Date: Tuesday, October 10, 2017 at 3:16 PM To: matthieu Cc: bro Subject: Re: [Bro] Community source for rules Then, I think you might want to look at the Bro scripting language, although still you have to script what you are looking for. Bro has started this awesome Bro-pkg manager project, which is similar to a central repository, for hosting the various Bro scripts that community can get benefit from: Here's the list of packages, available for the community to download and install: https://github.com/bro/packages Also, there are many individual Bro scripts available on github. If interested, there's this script from Fox-IT regarding ransomeware detection using SMB: https://github.com/fox-it/bro-scripts/tree/master/smb-ransomware -Fatema. On Tue, Oct 10, 2017 at 2:43 PM, matthieu > wrote: Hi Thank you for your reply. Yes I know snort2bro, but I use Snort or Suricata for this rules. I was hoping there was a Bro rules contribution available on the Internet. Generic rules that answer to the actuality like WannaCry (SMB) ? Matthieu On 10 Oct 2017, at 14:36, fatema bannatwala > wrote: Hi Matthieu, I am not aware of any source available for Bro signatures (rules, if that's what you meant), however, there used to be a script snort2bro that converted snort signatures/rules to corresponding Bro sigs, but not maintained anymore. Not sure what you are looking to solve, but if you know what you are searching for in your traffic, then you might want to take a look at the Bro's Signature Language, to write your own signatures. Here's the link: https://www.bro.org/sphinx/frameworks/signatures.html Hope this helps. -Fatema -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171011/fbece60d/attachment-0001.html From matthieu at treussart.com Wed Oct 11 12:34:40 2017 From: matthieu at treussart.com (matthieu) Date: Wed, 11 Oct 2017 21:34:40 +0200 Subject: [Bro] Community source for rules In-Reply-To: References: <92EC931F-AD10-4AC9-84F5-E030656E8A6C@treussart.com> Message-ID: <1E34C5B4-2504-45FB-A42D-9DC83F58AE65@treussart.com> Hi Hi Adam This is what I am looking for. I am trying the API Cristalstack. Thank you very much. Matthieu > On 11 Oct 2017, at 17:49, Adam Pumphrey wrote: > > I also suggest looking at Bro?s Intelligence Framework, https://www.bro.org/sphinx-git/frameworks/intel.html . This is how Bro consumes and makes use of threat intel indicators, which is essentially what the ET rule feeds contain. > > There are many intel indicator sources available, some require more effort than others to integrate. As mentioned some tools exist that can help with that. If you?re looking for an indicator source(s), Criticalstack offers a free feed aggregation service that directly integrates with Bro?s Intel Framework. It?s easy to use and a good tool for quickly getting external indicator sources in. Worth a look if you?re exploring how threat intel, supplementary to ET rule feeds, can be used. > > Adam > > From: on behalf of fatema bannatwala > Date: Tuesday, October 10, 2017 at 3:16 PM > To: matthieu > Cc: bro > Subject: Re: [Bro] Community source for rules > > Then, I think you might want to look at the Bro scripting language, > although still you have to script what you are looking for. > Bro has started this awesome Bro-pkg manager project, which is similar to a central repository, > for hosting the various Bro scripts that community can get benefit from: > > Here's the list of packages, available for the community to download and install: > https://github.com/bro/packages > > Also, there are many individual Bro scripts available on github. > If interested, there's this script from Fox-IT regarding ransomeware detection using SMB: > https://github.com/fox-it/bro-scripts/tree/master/smb-ransomware > > -Fatema. > > > On Tue, Oct 10, 2017 at 2:43 PM, matthieu > wrote: > Hi > Thank you for your reply. > > > Yes I know snort2bro, but I use Snort or Suricata for this rules. > I was hoping there was a Bro rules contribution available on the Internet. > Generic rules that answer to the actuality like WannaCry (SMB) ? > > > Matthieu > > > > On 10 Oct 2017, at 14:36, fatema bannatwala > wrote: > > Hi Matthieu, > > I am not aware of any source available for Bro signatures (rules, if that's what you meant), > however, there used to be a script snort2bro that converted snort signatures/rules to corresponding Bro sigs, but not maintained anymore. > > Not sure what you are looking to solve, but if you know what you are searching for in your traffic, > then you might want to take a look at the Bro's Signature Language, to write your own signatures. > Here's the link: https://www.bro.org/sphinx/frameworks/signatures.html > > Hope this helps. > > -Fatema > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171011/49c74e4f/attachment.html From shirkdog.bsd at gmail.com Thu Oct 12 04:27:32 2017 From: shirkdog.bsd at gmail.com (Michael Shirk) Date: Thu, 12 Oct 2017 07:27:32 -0400 Subject: [Bro] Community source for rules In-Reply-To: References: <92EC931F-AD10-4AC9-84F5-E030656E8A6C@treussart.com> Message-ID: It has been discussed several times about "signatures", and what it seems folks would love to do is take Snort/Suricata rules and throw them into Bro. I think the Intel framework would work be a good place to get quick wins with the conversion, but I have never used the signature features in Bro, as a the first rule of fight club is to never use it. I think there may be different avenues to achieve similar functionality to signature based IDS engines, just requires time to figure it out. -- Michael Shirk Daemon Security, Inc. https://www.daemon-security.com On Oct 11, 2017 14:58, "Adam Pumphrey" wrote: > I also suggest looking at Bro?s Intelligence Framework, > https://www.bro.org/sphinx-git/frameworks/intel.html. This is how Bro > consumes and makes use of threat intel indicators, which is essentially > what the ET rule feeds contain. > > > > There are many intel indicator sources available, some require more effort > than others to integrate. As mentioned some tools exist that can help with > that. If you?re looking for an indicator source(s), Criticalstack offers a > free feed aggregation service that directly integrates with Bro?s Intel > Framework. It?s easy to use and a good tool for quickly getting external > indicator sources in. Worth a look if you?re exploring how threat intel, > supplementary to ET rule feeds, can be used. > > > > Adam > > > > *From: * on behalf of fatema bannatwala < > fatema.bannatwala at gmail.com> > *Date: *Tuesday, October 10, 2017 at 3:16 PM > *To: *matthieu > *Cc: *bro > *Subject: *Re: [Bro] Community source for rules > > > > Then, I think you might want to look at the Bro scripting language, > > although still you have to script what you are looking for. > > Bro has started this awesome Bro-pkg manager project, which is similar to > a central repository, > > for hosting the various Bro scripts that community can get benefit from: > > > > Here's the list of packages, available for the community to download and > install: > > https://github.com/bro/packages > > > > Also, there are many individual Bro scripts available on github. > > If interested, there's this script from Fox-IT regarding ransomeware > detection using SMB: > > https://github.com/fox-it/bro-scripts/tree/master/smb-ransomware > > > > -Fatema. > > > > > > On Tue, Oct 10, 2017 at 2:43 PM, matthieu wrote: > > Hi > > Thank you for your reply. > > > > Yes I know snort2bro, but I use Snort or Suricata for this rules. > > I was hoping there was a Bro rules contribution available on the Internet. > > Generic rules that answer to the actuality like WannaCry (SMB) ? > > > > Matthieu > > > > > > > > On 10 Oct 2017, at 14:36, fatema bannatwala > wrote: > > > > Hi Matthieu, > > > > I am not aware of any source available for Bro signatures (rules, if > that's what you meant), > > however, there used to be a script snort2bro that converted snort > signatures/rules to corresponding Bro sigs, but not maintained anymore. > > > > Not sure what you are looking to solve, but if you know what you are > searching for in your traffic, > > then you might want to take a look at the Bro's Signature Language, to > write your own signatures. > > Here's the link: https://www.bro.org/sphinx/frameworks/signatures.html > > > > Hope this helps. > > > > -Fatema > > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171012/af2e911b/attachment-0001.html From jlamps at sandia.gov Thu Oct 12 12:16:02 2017 From: jlamps at sandia.gov (Lamps, Jereme) Date: Thu, 12 Oct 2017 19:16:02 +0000 Subject: [Bro] Looking up fa_file given FUID Message-ID: Hello, I was just wondering if it was possible to lookup fa_file or Files::Info records given a FUID. I have been looking through the built in functions but have not seen anything. Best, Jereme Lamps -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171012/7f37129e/attachment.html From promero at cenic.org Thu Oct 12 14:14:26 2017 From: promero at cenic.org (Philip Romero) Date: Thu, 12 Oct 2017 14:14:26 -0700 Subject: [Bro] PF_RING Help Needed Message-ID: <37ca1e5e-727a-b164-7cee-62469852fd8b@cenic.org> All, I've been meaning to get PF_RING going for a while and am now trying to focus on getting it working. Until now I've been running the "standalone" [bro] config at the top of my node.cfg output below. I've been thru the past threads and came across some info related to output that might confirm if pf_ring and bro were compiled together correctly. Below I've add the output of some of the commands suggested for input on troubleshooting the issue. I suspect an error in some part of my config or setup since I don't get any usable logs when the load-balance/pf_ring node.cfg setting are enabled. If I comment them out and do a broctl deploy usable logs immediately appear in my log directory. Any hints or suggestions as to why my pf_ring configuration is not working would be greatly appreciated. Let me know if any additional details I need to provide would help shed some light on my issue. [root at xxx-bro-1 etc]# *cat node.cfg* # Example BroControl node configuration. # # This example has a standalone node ready to go except for possibly changing # the sniffing interface. # This is a complete standalone configuration.? Most likely you will # only need to change the interface. #[bro] #type=standalone #host=localhost #interface=ens2f0 ## Below is an example clustered configuration. If you use this, ## remove the [bro] node above. #[logger] #type=logger #host=localhost # [manager] type=manager host=localhost # [proxy-1] type=proxy host=localhost # [worker-1] lb_method=pf_ring lb_procs=4 pin_cpus=4,5,6,7 type=worker host=localhost interface=ens2f0 # #[worker-2] #type=worker #host=localhost #interface=eth0 [root at xxx-bro-1 etc]# [root at xxx-bro-1 etc]# *broctl status* Name???????? Type??? Host???????????? Status??? Pid??? Started manager????? manager localhost??????? running?? 24982? 12 Oct 13:51:46 proxy-1????? proxy?? localhost??????? running?? 25040? 12 Oct 13:51:48 worker-1-1?? worker? localhost??????? running?? 25123? 12 Oct 13:51:49 worker-1-2?? worker? localhost??????? running?? 25126? 12 Oct 13:51:49 worker-1-3?? worker? localhost??????? running?? 25124? 12 Oct 13:51:49 worker-1-4?? worker? localhost??????? running?? 25125? 12 Oct 13:51:49 [root at xxx-bro-1 etc]# *broctl config | grep pfring* pfringclusterid = 21 pfringclustertype = 4-tuple pfringfirstappinstance = 0 [root at xxx-bro-1 etc]# *ldd /usr/local/bro/bin/bro* ??? linux-vdso.so.1 =>? (0x00007ffeaabf8000) ??? libpcap.so.1 => /opt/pfring/lib/libpcap.so.1 (0x00007f9c52470000) ??? libssl.so.10 => /lib64/libssl.so.10 (0x00007f9c521f5000) ??? libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f9c51d94000) ??? libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f9c51b7a000) ??? libz.so.1 => /lib64/libz.so.1 (0x00007f9c51963000) ??? libGeoIP.so.1 => /lib64/libGeoIP.so.1 (0x00007f9c51733000) ??? libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9c51517000) ??? libdl.so.2 => /lib64/libdl.so.2 (0x00007f9c51312000) ??? libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f9c5100a000) ??? libm.so.6 => /lib64/libm.so.6 (0x00007f9c50d08000) ??? libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f9c50af1000) ??? libc.so.6 => /lib64/libc.so.6 (0x00007f9c5072e000) ??? librt.so.1 => /lib64/librt.so.1 (0x00007f9c50526000) ??? libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f9c502d8000) ??? libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f9c4fff0000) ??? libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f9c4fdec000) ??? libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f9c4fbb8000) ??? /lib64/ld-linux-x86-64.so.2 (0x0000561284250000) ??? libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f9c4f9aa000) ??? libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f9c4f7a5000) ??? libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f9c4f57e000) ??? libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f9c4f31b000) [root at xxx-bro-1 etc]# *cat /proc/net/pf_ring/** Bound Device(s)??? : ens2f0 Active???????????? : 1 Breed????????????? : Standard Appl. Name???????? : bro-ens2f0 Socket Mode??????? : RX+TX Capture Direction? : RX+TX Sampling Rate????? : 1 IP Defragment????? : No BPF Filtering????? : Disabled Sw Filt Hash Rules : 0 Sw Filt WC Rules?? : 0 Hw Filt Rules????? : 0 Sw Filt Hash Match : 0 Sw Filt Hash Miss? : 0 Poll Pkt Watermark : 1 Num Poll Calls???? : 3940305 Channel Id Mask??? : 0xFFFFFFFFFFFFFFFF Cluster Id???????? : 21 Slot Version?????? : 16 [6.6.0] Min Num Slots????? : 32768 Bucket Len???????? : 8192 Slot Len?????????? : 8248 [bucket+header] Tot Memory???????? : 270282752 Tot Packets??????? : 468041 Tot Pkt Lost?????? : 0 Tot Insert???????? : 468041 Tot Read?????????? : 468041 Insert Offset????? : 53220976 Remove Offset????? : 53220976 Num Free Slots???? : 32768 TX: Send Ok??????? : 0 TX: Send Errors??? : 0 Reflect: Fwd Ok??? : 0 Reflect: Fwd Errors: 0 Bound Device(s)??? : ens2f0 Active???????????? : 1 Breed????????????? : Standard Appl. Name???????? : bro-ens2f0 Socket Mode??????? : RX+TX Capture Direction? : RX+TX Sampling Rate????? : 1 IP Defragment????? : No BPF Filtering????? : Disabled Sw Filt Hash Rules : 0 Sw Filt WC Rules?? : 0 Hw Filt Rules????? : 0 Sw Filt Hash Match : 0 Sw Filt Hash Miss? : 0 Poll Pkt Watermark : 1 Num Poll Calls???? : 3928875 Channel Id Mask??? : 0xFFFFFFFFFFFFFFFF Cluster Id???????? : 21 Slot Version?????? : 16 [6.6.0] Min Num Slots????? : 32768 Bucket Len???????? : 8192 Slot Len?????????? : 8248 [bucket+header] Tot Memory???????? : 270282752 Tot Packets??????? : 278361 Tot Pkt Lost?????? : 0 Tot Insert???????? : 278361 Tot Read?????????? : 278361 Insert Offset????? : 153697792 Remove Offset????? : 153697792 Num Free Slots???? : 32768 TX: Send Ok??????? : 0 TX: Send Errors??? : 0 Reflect: Fwd Ok??? : 0 Reflect: Fwd Errors: 0 Bound Device(s)??? : ens2f0 Active???????????? : 1 Breed????????????? : Standard Appl. Name???????? : bro-ens2f0 Socket Mode??????? : RX+TX Capture Direction? : RX+TX Sampling Rate????? : 1 IP Defragment????? : No BPF Filtering????? : Disabled Sw Filt Hash Rules : 0 Sw Filt WC Rules?? : 0 Hw Filt Rules????? : 0 Sw Filt Hash Match : 0 Sw Filt Hash Miss? : 0 Poll Pkt Watermark : 1 Num Poll Calls???? : 4036165 Channel Id Mask??? : 0xFFFFFFFFFFFFFFFF Cluster Id???????? : 21 Slot Version?????? : 16 [6.6.0] Min Num Slots????? : 32768 Bucket Len???????? : 8192 Slot Len?????????? : 8248 [bucket+header] Tot Memory???????? : 270282752 Tot Packets??????? : 497001 Tot Pkt Lost?????? : 0 Tot Insert???????? : 497001 Tot Read?????????? : 497001 Insert Offset????? : 217876744 Remove Offset????? : 217876744 Num Free Slots???? : 32768 TX: Send Ok??????? : 0 TX: Send Errors??? : 0 Reflect: Fwd Ok??? : 0 Reflect: Fwd Errors: 0 Bound Device(s)??? : ens2f0 Active???????????? : 1 Breed????????????? : Standard Appl. Name???????? : bro-ens2f0 Socket Mode??????? : RX+TX Capture Direction? : RX+TX Sampling Rate????? : 1 IP Defragment????? : No BPF Filtering????? : Disabled Sw Filt Hash Rules : 0 Sw Filt WC Rules?? : 0 Hw Filt Rules????? : 0 Sw Filt Hash Match : 0 Sw Filt Hash Miss? : 0 Poll Pkt Watermark : 1 Num Poll Calls???? : 3935048 Channel Id Mask??? : 0xFFFFFFFFFFFFFFFF Cluster Id???????? : 21 Slot Version?????? : 16 [6.6.0] Min Num Slots????? : 32768 Bucket Len???????? : 8192 Slot Len?????????? : 8248 [bucket+header] Tot Memory???????? : 270282752 Tot Packets??????? : 383337 Tot Pkt Lost?????? : 0 Tot Insert???????? : 383337 Tot Read?????????? : 383337 Insert Offset????? : 213239720 Remove Offset????? : 213239720 Num Free Slots???? : 32768 TX: Send Ok??????? : 0 TX: Send Errors??? : 0 Reflect: Fwd Ok??? : 0 Reflect: Fwd Errors: 0 cat: /proc/net/pf_ring/dev: Is a directory PF_RING Version????????? : 6.6.0 (unknown) Total rings????????????? : 4 Standard (non ZC) Options Ring slots?????????????? : 32768 Slot version???????????? : 16 Capture TX?????????????? : No [RX only] IP Defragment??????????? : No Socket Mode????????????? : Standard Cluster Fragment Queue?? : 0 Cluster Fragment Discard : 0 cat: /proc/net/pf_ring/stats: Is a directory [root at xxx-bro-1 etc]# -- Philip Romero, CISSP, CISA Sr. Information Security Analyst CENIC promero at cenic.org Phone: (714) 220-3430 Mobile: (562) 237-9290 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171012/6f87444f/attachment-0001.html From jazoff at illinois.edu Thu Oct 12 14:22:06 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 12 Oct 2017 21:22:06 +0000 Subject: [Bro] PF_RING Help Needed In-Reply-To: <37ca1e5e-727a-b164-7cee-62469852fd8b@cenic.org> References: <37ca1e5e-727a-b164-7cee-62469852fd8b@cenic.org> Message-ID: <1A4BB6F8-C32F-4A4A-8C48-F43DEAB05BF3@illinois.edu> > On Oct 12, 2017, at 5:14 PM, Philip Romero wrote: > > All, > > I've been meaning to get PF_RING going for a while and am now trying to focus on getting it working. Until now I've been running the "standalone" [bro] config at the top of my node.cfg output below. I've been thru the past threads and came across some info related to output that might confirm if pf_ring and bro were compiled together correctly. Below I've add the output of some of the commands suggested for input on troubleshooting the issue. > I suspect an error in some part of my config or setup since I don't get any usable logs when the load-balance/pf_ring node.cfg setting are enabled. If I comment them out and do a broctl deploy usable logs immediately appear in my log directory. Any hints or suggestions as to why my pf_ring configuration is not working would be greatly appreciated. Let me know if any additional details I need to provide would help shed some light on my issue. Can you clarify specifically which lines you are commenting out? Does the below configuration work? [manager] type=manager host=localhost [proxy-1] type=proxy host=localhost [worker-1] type=worker host=localhost interface=ens2f0 ? Justin Azoff From promero at cenic.org Thu Oct 12 14:31:13 2017 From: promero at cenic.org (Philip Romero) Date: Thu, 12 Oct 2017 14:31:13 -0700 Subject: [Bro] PF_RING Help Needed In-Reply-To: <1A4BB6F8-C32F-4A4A-8C48-F43DEAB05BF3@illinois.edu> References: <37ca1e5e-727a-b164-7cee-62469852fd8b@cenic.org> <1A4BB6F8-C32F-4A4A-8C48-F43DEAB05BF3@illinois.edu> Message-ID: I comment out all the [manger], [proxy-1], and [worker-1] lines (including the names) below and re-enable the below standalone [bro] lines. [bro] type=standalone host=localhost interface=ens2f0 On 10/12/17 2:22 PM, Azoff, Justin S wrote: >> On Oct 12, 2017, at 5:14 PM, Philip Romero wrote: >> >> All, >> >> I've been meaning to get PF_RING going for a while and am now trying to focus on getting it working. Until now I've been running the "standalone" [bro] config at the top of my node.cfg output below. I've been thru the past threads and came across some info related to output that might confirm if pf_ring and bro were compiled together correctly. Below I've add the output of some of the commands suggested for input on troubleshooting the issue. >> I suspect an error in some part of my config or setup since I don't get any usable logs when the load-balance/pf_ring node.cfg setting are enabled. If I comment them out and do a broctl deploy usable logs immediately appear in my log directory. Any hints or suggestions as to why my pf_ring configuration is not working would be greatly appreciated. Let me know if any additional details I need to provide would help shed some light on my issue. > Can you clarify specifically which lines you are commenting out? Does the below configuration work? > > [manager] > type=manager > host=localhost > > [proxy-1] > type=proxy > host=localhost > > [worker-1] > type=worker > host=localhost > interface=ens2f0 > > ? > Justin Azoff > > -- Philip Romero, CISSP, CISA Sr. Information Security Analyst CENIC promero at cenic.org Phone: (714) 220-3430 Mobile: (562) 237-9290 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171012/e09b2d68/attachment.html From promero at cenic.org Thu Oct 12 14:41:14 2017 From: promero at cenic.org (Philip Romero) Date: Thu, 12 Oct 2017 14:41:14 -0700 Subject: [Bro] PF_RING Help Needed In-Reply-To: <1A4BB6F8-C32F-4A4A-8C48-F43DEAB05BF3@illinois.edu> References: <37ca1e5e-727a-b164-7cee-62469852fd8b@cenic.org> <1A4BB6F8-C32F-4A4A-8C48-F43DEAB05BF3@illinois.edu> Message-ID: <1d8a9c50-3a7a-72ce-e765-2a256f5186fd@cenic.org> I just tried the below node.cfg setting and it did not seem to fix the issue. The weird thing is that I never really get a noticeable error during the startup process. It says stuff is running, but I just don't get any logs. I also am not able to pull a "broctl netstats" output when load-balancing is configured, but I can when it is not. On 10/12/17 2:22 PM, Azoff, Justin S wrote: >> On Oct 12, 2017, at 5:14 PM, Philip Romero wrote: >> >> All, >> >> I've been meaning to get PF_RING going for a while and am now trying to focus on getting it working. Until now I've been running the "standalone" [bro] config at the top of my node.cfg output below. I've been thru the past threads and came across some info related to output that might confirm if pf_ring and bro were compiled together correctly. Below I've add the output of some of the commands suggested for input on troubleshooting the issue. >> I suspect an error in some part of my config or setup since I don't get any usable logs when the load-balance/pf_ring node.cfg setting are enabled. If I comment them out and do a broctl deploy usable logs immediately appear in my log directory. Any hints or suggestions as to why my pf_ring configuration is not working would be greatly appreciated. Let me know if any additional details I need to provide would help shed some light on my issue. > Can you clarify specifically which lines you are commenting out? Does the below configuration work? > > [manager] > type=manager > host=localhost > > [proxy-1] > type=proxy > host=localhost > > [worker-1] > type=worker > host=localhost > interface=ens2f0 > > ? > Justin Azoff > > -- Philip Romero, CISSP, CISA Sr. Information Security Analyst CENIC promero at cenic.org Phone: (714) 220-3430 Mobile: (562) 237-9290 From jazoff at illinois.edu Thu Oct 12 15:01:56 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 12 Oct 2017 22:01:56 +0000 Subject: [Bro] PF_RING Help Needed In-Reply-To: <1d8a9c50-3a7a-72ce-e765-2a256f5186fd@cenic.org> References: <37ca1e5e-727a-b164-7cee-62469852fd8b@cenic.org> <1A4BB6F8-C32F-4A4A-8C48-F43DEAB05BF3@illinois.edu> <1d8a9c50-3a7a-72ce-e765-2a256f5186fd@cenic.org> Message-ID: > On Oct 12, 2017, at 5:41 PM, Philip Romero wrote: > > I just tried the below node.cfg setting and it did not seem to fix the issue. The weird thing is that I never really get a noticeable error during the startup process. It says stuff is running, but I just don't get any logs. I also am not able to pull a "broctl netstats" output when load-balancing is configured, but I can when it is not. OK! This is not a load balancing problem or a pf_ring problem at all. The different bro processes are unable to connect to each other. Check that 'localhost' resolves to 127.0.0.1 and that you don't have any iptables rules applied to the lo interface that would be preventing processes from reaching each other. ? Justin Azoff From promero at cenic.org Thu Oct 12 15:12:02 2017 From: promero at cenic.org (Philip Romero) Date: Thu, 12 Oct 2017 15:12:02 -0700 Subject: [Bro] PF_RING Help Needed In-Reply-To: References: <37ca1e5e-727a-b164-7cee-62469852fd8b@cenic.org> <1A4BB6F8-C32F-4A4A-8C48-F43DEAB05BF3@illinois.edu> <1d8a9c50-3a7a-72ce-e765-2a256f5186fd@cenic.org> Message-ID: <8a8bb23a-08b7-d35a-e655-2314b8d478fa@cenic.org> Success, for now. I put back my original lb node.cfg config and turned off the local firewall to see if stuff would work. It does. Now I need to get my system admin team to adjust their standard server firewall config to allow the bro processes to talk locally. Thanks for the help. Philip On 10/12/17 3:01 PM, Azoff, Justin S wrote: >> On Oct 12, 2017, at 5:41 PM, Philip Romero wrote: >> >> I just tried the below node.cfg setting and it did not seem to fix the issue. The weird thing is that I never really get a noticeable error during the startup process. It says stuff is running, but I just don't get any logs. I also am not able to pull a "broctl netstats" output when load-balancing is configured, but I can when it is not. > OK! This is not a load balancing problem or a pf_ring problem at all. > > The different bro processes are unable to connect to each other. Check that 'localhost' resolves to 127.0.0.1 and that you don't have any iptables rules applied to the lo interface that would be preventing processes from reaching each other. > > > ? > Justin Azoff > -- Philip Romero, CISSP, CISA Sr. Information Security Analyst CENIC promero at cenic.org Phone: (714) 220-3430 Mobile: (562) 237-9290 From jlay at slave-tothe-box.net Thu Oct 12 15:13:11 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 12 Oct 2017 16:13:11 -0600 Subject: [Bro] PF_RING Help Needed In-Reply-To: <1d8a9c50-3a7a-72ce-e765-2a256f5186fd@cenic.org> References: <37ca1e5e-727a-b164-7cee-62469852fd8b@cenic.org> <1A4BB6F8-C32F-4A4A-8C48-F43DEAB05BF3@illinois.edu> <1d8a9c50-3a7a-72ce-e765-2a256f5186fd@cenic.org> Message-ID: <6d8fb306f088cb231b7cb52175a1c5d5@localhost> On 2017-10-12 15:41, Philip Romero wrote: > I just tried the below node.cfg setting and it did not seem to fix the > issue. The weird thing is that I never really get a noticeable error > during the startup process. It says stuff is running, but I just don't > get any logs. I also am not able to pull a "broctl netstats" output > when > load-balancing is configured, but I can when it is not. > > > On 10/12/17 2:22 PM, Azoff, Justin S wrote: >>> On Oct 12, 2017, at 5:14 PM, Philip Romero wrote: >>> >>> All, >>> >>> I've been meaning to get PF_RING going for a while and am now trying >>> to focus on getting it working. Until now I've been running the >>> "standalone" [bro] config at the top of my node.cfg output below. >>> I've been thru the past threads and came across some info related to >>> output that might confirm if pf_ring and bro were compiled together >>> correctly. Below I've add the output of some of the commands >>> suggested for input on troubleshooting the issue. >>> I suspect an error in some part of my config or setup since I don't >>> get any usable logs when the load-balance/pf_ring node.cfg setting >>> are enabled. If I comment them out and do a broctl deploy usable logs >>> immediately appear in my log directory. Any hints or suggestions as >>> to why my pf_ring configuration is not working would be greatly >>> appreciated. Let me know if any additional details I need to provide >>> would help shed some light on my issue. >> Can you clarify specifically which lines you are commenting out? Does >> the below configuration work? >> >> [manager] >> type=manager >> host=localhost >> >> [proxy-1] >> type=proxy >> host=localhost >> >> [worker-1] >> type=worker >> host=localhost >> interface=ens2f0 >> >> ? >> Justin Azoff >> >> Make sure you've followed this: https://www.bro.org/documentation/load-balancing.html Also, your logs may no longer be in spool, but might be in manager or logger directories. James From daniel_aka_sniper_d at hotmail.com Mon Oct 16 09:49:10 2017 From: daniel_aka_sniper_d at hotmail.com (Sniper) Date: Mon, 16 Oct 2017 16:49:10 +0000 Subject: [Bro] Documentation and getting started. Message-ID: Hello Everyone, Is there reference page on all the default installation directory locations are by any chance? $PREFIX just makes it a very long process establishing where all the files are located. If not, I think this would be excellent for beginners like me. Also, I have created a bridge interface that I want to monitor using ubuntu/bro by connecting two hosts, for some reason I can't seem to generate any logs in /usr/local/bro/logs/ (no 'current' folder when bro is started as in the documentation). Is this even possible to monitor a bridge interface using bro on the same host? I have already changed node.cfg interface to br0. There are no tutorials nowhere on how to actually get started, tried to follow the instructions but still no luck, ive been wasting days on this. If someone could point me in the right direction i'll greatly appreciate it. Kind regards Daniel --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus From jmellander at lbl.gov Mon Oct 16 12:20:28 2017 From: jmellander at lbl.gov (Jim Mellander) Date: Mon, 16 Oct 2017 12:20:28 -0700 Subject: [Bro] Documentation and getting started. In-Reply-To: References: Message-ID: Hi Daniel: Check this link for info on ? ? the bro directory structure that may help you: https://www.bro.org/sphinx/ install/release-notes.html#script-organization ?As far as monitoring a bridged interface, there should be no problem, as long as bro can access the interface. ?If you're not running as root, see: https://www.bro.org/documentation/faq.html#how-can-i-capture-packets-as-an-unprivileged-user Does tcpdump provide expected output when run against br0? Hope this helps, Jim On Mon, Oct 16, 2017 at 9:49 AM, Sniper wrote: > Hello Everyone, > > Is there reference page on all the default installation directory > locations are by any chance? $PREFIX just makes it a very long process > establishing where all the files are located. If not, I think this would > be excellent for beginners like me. > > Also, I have created a bridge interface that I want to monitor using > ubuntu/bro by connecting two hosts, for some reason I can't seem to > generate any logs in /usr/local/bro/logs/ (no 'current' folder when bro > is started as in the documentation). Is this even possible to monitor a > bridge interface using bro on the same host? I have already changed > node.cfg interface to br0. > > There are no tutorials nowhere on how to actually get started, tried to > follow the instructions but still no luck, ive been wasting days on > this. If someone could point me in the right direction i'll greatly > appreciate it. > > Kind regards > > Daniel > > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171016/e30dbe29/attachment.html From fatema.bannatwala at gmail.com Mon Oct 16 12:58:43 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Mon, 16 Oct 2017 15:58:43 -0400 Subject: [Bro] The code for "weird" logging activity. Message-ID: Hey All, So, I was going through the weird.log file generated by bro every hour, and found lot of activity that I would like to suppress, and for some activity I would like to know the source (i.e. what part of bro code is raising those "weird" activity logs in the weird.log) to analyse whether it's legit or can be suppressed. For example, I would like to suppress "DNS_RR_unknown_type 46", as it's , I think, is not an unknown-type, it's defined in RFC 4034 as "RRSIG" (and some other similar weird activity.) Hence, wanted to see what code during packet analysis might have raised one of the *_weird events to log that connection. I was searching for the string "weird" in an effort to find the Bro scripts that either load weird or create a log stream in weird.log, but couldn't find the code/script that is responsible for those notices in weird.log P.S: I know about the weird.bro in notice framework, I am searching for part of the code that would *use* *_weird events to log weird activity in weird.log. Checked policy/base dirs : policy]$ find . -type f -exec cat {} + | grep "weird" ##! This script handles core generated connection related "weird" events to ##! push weird information about connections into the weird framework. # This is weird beause it would mean that someone didn't event conn_weird("smb_pipe_request_missing_uuid", c, ""); # This is weird: the inquirer must also be providing answers in Any pointers to the right direction would be really appreciated :) Thanks, Fatema. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171016/01ab4007/attachment.html From jazoff at illinois.edu Mon Oct 16 13:01:44 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 16 Oct 2017 20:01:44 +0000 Subject: [Bro] The code for "weird" logging activity. In-Reply-To: References: Message-ID: > On Oct 16, 2017, at 3:58 PM, fatema bannatwala wrote: > > Hey All, > > So, I was going through the weird.log file generated by bro every hour, > and found lot of activity that I would like to suppress, and for some > activity I would like to know the source (i.e. what part of bro code is raising those > "weird" activity logs in the weird.log) to analyse whether it's legit or can be suppressed. > > For example, I would like to suppress "DNS_RR_unknown_type 46", as it's , > I think, is not an unknown-type, it's defined in RFC 4034 as "RRSIG" (and some other similar weird activity.) > > Hence, wanted to see what code during packet analysis might have raised one of the *_weird events to log that connection. > > I was searching for the string "weird" in an effort to find the Bro scripts > that either load weird or create a log stream in weird.log, but couldn't find the code/script > that is responsible for those notices in weird.log Ah.. it's also 'Weird' inside of analyzers, so 'weird' would not have found it: $ git grep DNS_RR_unknown_type CHANGES: * DNS: Log the type number for the DNS_RR_unknown_type weird. (Vlad Grigorescu) scripts/base/frameworks/notice/weird.bro: ["DNS_RR_unknown_type"] = ACTION_LOG, src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_RR_unknown_type", fmt("%d", msg->atype)); testing/btest/Baseline/scripts.base.protocols.dns.duplicate-reponses/weird.log:1363716396.798286 CHhAvVGS1DHFjwGM9 55.247.223.174 27285 222.195.43.124 53 DNS_RR_unknown_type 46 F bro $ git grep 'analyzer->Weird' src/analyzer/protocol/dnp3/DNP3.cc: analyzer->Weird("dnp3_header_lacks_magic"); src/analyzer/protocol/dnp3/DNP3.cc: analyzer->Weird("dnp3_unexpected_flow_direction"); src/analyzer/protocol/dnp3/DNP3.cc: analyzer->Weird("dnp3_negative_or_zero_length_link_layer"); src/analyzer/protocol/dnp3/DNP3.cc: analyzer->Weird("dnp3_first_application_layer_chunk_missing"); src/analyzer/protocol/dnp3/DNP3.cc: analyzer->Weird(fmt("dnp3_corrupt_%s_checksum", where)); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_truncated_len_lt_hdr_len"); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_Conn_count_too_large"); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_truncated_quest_too_short"); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_truncated_ans_too_short"); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_truncated_RR_rdlength_lt_len"); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_RR_unknown_type", fmt("%d", msg->atype)); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_NAME_too_long"); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_label_forward_compress_offset"); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_label_len_gt_pkt"); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_label_too_long"); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_label_len_gt_name_len"); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_RR_length_mismatch"); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_RR_length_mismatch"); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_RR_length_mismatch"); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_RR_length_mismatch"); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_RR_bad_length"); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_AAAA_neg_length"); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_A6_neg_length"); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_TXT_char_str_past_rdlen"); src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_CAA_char_str_past_rdlen"); src/analyzer/protocol/http/HTTP.cc: analyzer->Weird(msg); src/analyzer/protocol/http/HTTP.cc: analyzer->Weird("illegal_%_at_end_of_URI"); src/analyzer/protocol/http/HTTP.cc: analyzer->Weird("partial_escape_at_end_of_URI"); src/analyzer/protocol/http/HTTP.cc: analyzer->Weird("double_%_in_URI"); src/analyzer/protocol/http/HTTP.cc: analyzer->Weird("unescaped_%_in_URI"); src/analyzer/protocol/ncp/NCP.cc: analyzer->Weird(e.msg().c_str()); src/analyzer/protocol/netbios/NetbiosSSN.cc: analyzer->Weird(fmt("unknown_netbios_type: 0x%x", type)); src/analyzer/protocol/netbios/NetbiosSSN.cc: analyzer->Weird(fmt("excess_netbios_hdr_len (%d > %d)", src/analyzer/protocol/netbios/NetbiosSSN.cc: analyzer->Weird("deficit_netbios_hdr_len"); src/analyzer/protocol/netbios/NetbiosSSN.cc: analyzer->Weird(fmt("excess_netbios_hdr_len (%d > %d)", src/analyzer/protocol/netbios/NetbiosSSN.cc: analyzer->Weird(fmt("deficit_netbios_hdr_len (%d < %d)", src/analyzer/protocol/netbios/NetbiosSSN.cc: analyzer->Weird("netbios_raw_session_msg"); src/analyzer/protocol/netbios/NetbiosSSN.cc: analyzer->Weird("no_smb_session_using_parsesambamsg"); src/analyzer/protocol/netbios/NetbiosSSN.cc: analyzer->Weird("netbios_server_session_request"); src/analyzer/protocol/netbios/NetbiosSSN.cc: analyzer->Weird("netbios_client_session_reply"); src/analyzer/protocol/netbios/NetbiosSSN.cc: analyzer->Weird("netbios_client_session_reply"); src/analyzer/protocol/netbios/NetbiosSSN.cc: analyzer->Weird("netbios_client_session_reply"); src/analyzer/protocol/rpc/RPC.cc: analyzer->Weird(msg); src/analyzer/protocol/tcp/TCP_Reassembler.cc: tcp_analyzer->Weird("above_hole_data_without_any_acks"); src/analyzer/protocol/tcp/TCP_Reassembler.cc: tcp_analyzer->Weird("excessive_data_without_further_acks"); src/analyzer/protocol/teredo/Teredo.h: { analyzer->Weird(name); } $ ? Justin Azoff From fatema.bannatwala at gmail.com Mon Oct 16 13:11:38 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Mon, 16 Oct 2017 16:11:38 -0400 Subject: [Bro] The code for "weird" logging activity. In-Reply-To: References: Message-ID: Hah, there's a reason we have -i option with grep *facepalm* :) (could have saved me lot of time). Thanks Justin for the quick response. Appreciate it! Yay! Fatema. On Mon, Oct 16, 2017 at 4:01 PM, Azoff, Justin S wrote: > > > On Oct 16, 2017, at 3:58 PM, fatema bannatwala < > fatema.bannatwala at gmail.com> wrote: > > > > Hey All, > > > > So, I was going through the weird.log file generated by bro every hour, > > and found lot of activity that I would like to suppress, and for some > > activity I would like to know the source (i.e. what part of bro code is > raising those > > "weird" activity logs in the weird.log) to analyse whether it's legit or > can be suppressed. > > > > For example, I would like to suppress "DNS_RR_unknown_type 46", as it's , > > I think, is not an unknown-type, it's defined in RFC 4034 as "RRSIG" > (and some other similar weird activity.) > > > > Hence, wanted to see what code during packet analysis might have raised > one of the *_weird events to log that connection. > > > > I was searching for the string "weird" in an effort to find the Bro > scripts > > that either load weird or create a log stream in weird.log, but couldn't > find the code/script > > that is responsible for those notices in weird.log > > Ah.. it's also 'Weird' inside of analyzers, so 'weird' would not have > found it: > > $ git grep DNS_RR_unknown_type > CHANGES: * DNS: Log the type number for the DNS_RR_unknown_type weird. > (Vlad Grigorescu) > scripts/base/frameworks/notice/weird.bro: > ["DNS_RR_unknown_type"] = ACTION_LOG, > src/analyzer/protocol/dns/DNS.cc: > analyzer->Weird("DNS_RR_unknown_type", fmt("%d", msg->atype)); > testing/btest/Baseline/scripts.base.protocols.dns. > duplicate-reponses/weird.log:1363716396.798286 CHhAvVGS1DHFjwGM9 > 55.247.223.174 27285 222.195.43.124 53 DNS_RR_unknown_type > 46 F bro > $ git grep 'analyzer->Weird' > src/analyzer/protocol/dnp3/DNP3.cc: > analyzer->Weird("dnp3_header_lacks_magic"); > src/analyzer/protocol/dnp3/DNP3.cc: > analyzer->Weird("dnp3_unexpected_flow_direction"); > src/analyzer/protocol/dnp3/DNP3.cc: > analyzer->Weird("dnp3_negative_or_zero_length_link_layer"); > src/analyzer/protocol/dnp3/DNP3.cc: > analyzer->Weird("dnp3_first_application_layer_chunk_missing"); > src/analyzer/protocol/dnp3/DNP3.cc: analyzer->Weird(fmt("dnp3_corrupt_%s_checksum", > where)); > src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_ > truncated_len_lt_hdr_len"); > src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_Conn_ > count_too_large"); > src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_ > truncated_quest_too_short"); > src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_ > truncated_ans_too_short"); > src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_ > truncated_RR_rdlength_lt_len"); > src/analyzer/protocol/dns/DNS.cc: > analyzer->Weird("DNS_RR_unknown_type", fmt("%d", msg->atype)); > src/analyzer/protocol/dns/DNS.cc: > analyzer->Weird("DNS_NAME_too_long"); > src/analyzer/protocol/dns/DNS.cc: > analyzer->Weird("DNS_label_forward_compress_offset"); > src/analyzer/protocol/dns/DNS.cc: > analyzer->Weird("DNS_label_len_gt_pkt"); > src/analyzer/protocol/dns/DNS.cc: > analyzer->Weird("DNS_label_too_long"); > src/analyzer/protocol/dns/DNS.cc: > analyzer->Weird("DNS_label_len_gt_name_len"); > src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_RR_ > length_mismatch"); > src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_RR_ > length_mismatch"); > src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_RR_ > length_mismatch"); > src/analyzer/protocol/dns/DNS.cc: analyzer->Weird("DNS_RR_ > length_mismatch"); > src/analyzer/protocol/dns/DNS.cc: > analyzer->Weird("DNS_RR_bad_length"); > src/analyzer/protocol/dns/DNS.cc: > analyzer->Weird("DNS_AAAA_neg_length"); > src/analyzer/protocol/dns/DNS.cc: > analyzer->Weird("DNS_A6_neg_length"); > src/analyzer/protocol/dns/DNS.cc: > analyzer->Weird("DNS_TXT_char_str_past_rdlen"); > src/analyzer/protocol/dns/DNS.cc: > analyzer->Weird("DNS_CAA_char_str_past_rdlen"); > src/analyzer/protocol/http/HTTP.cc: analyzer->Weird(msg); > src/analyzer/protocol/http/HTTP.cc: > analyzer->Weird("illegal_%_at_end_of_URI"); > src/analyzer/protocol/http/HTTP.cc: > analyzer->Weird("partial_escape_at_end_of_URI"); > src/analyzer/protocol/http/HTTP.cc: > analyzer->Weird("double_%_in_URI"); > src/analyzer/protocol/http/HTTP.cc: > analyzer->Weird("unescaped_%_in_URI"); > src/analyzer/protocol/ncp/NCP.cc: > analyzer->Weird(e.msg().c_str()); > src/analyzer/protocol/netbios/NetbiosSSN.cc: > analyzer->Weird(fmt("unknown_netbios_type: 0x%x", type)); > src/analyzer/protocol/netbios/NetbiosSSN.cc: > analyzer->Weird(fmt("excess_netbios_hdr_len (%d > %d)", > src/analyzer/protocol/netbios/NetbiosSSN.cc: > analyzer->Weird("deficit_netbios_hdr_len"); > src/analyzer/protocol/netbios/NetbiosSSN.cc: > analyzer->Weird(fmt("excess_netbios_hdr_len (%d > %d)", > src/analyzer/protocol/netbios/NetbiosSSN.cc: > analyzer->Weird(fmt("deficit_netbios_hdr_len (%d < %d)", > src/analyzer/protocol/netbios/NetbiosSSN.cc: > analyzer->Weird("netbios_raw_session_msg"); > src/analyzer/protocol/netbios/NetbiosSSN.cc: > analyzer->Weird("no_smb_session_using_parsesambamsg"); > src/analyzer/protocol/netbios/NetbiosSSN.cc: > analyzer->Weird("netbios_server_session_request"); > src/analyzer/protocol/netbios/NetbiosSSN.cc: > analyzer->Weird("netbios_client_session_reply"); > src/analyzer/protocol/netbios/NetbiosSSN.cc: > analyzer->Weird("netbios_client_session_reply"); > src/analyzer/protocol/netbios/NetbiosSSN.cc: > analyzer->Weird("netbios_client_session_reply"); > src/analyzer/protocol/rpc/RPC.cc: analyzer->Weird(msg); > src/analyzer/protocol/tcp/TCP_Reassembler.cc: > tcp_analyzer->Weird("above_hole_data_without_any_acks"); > src/analyzer/protocol/tcp/TCP_Reassembler.cc: > tcp_analyzer->Weird("excessive_data_without_further_acks"); > src/analyzer/protocol/teredo/Teredo.h: { analyzer->Weird(name); } > $ > > ? > Justin Azoff > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171016/62b814c1/attachment-0001.html From johanna at icir.org Mon Oct 16 15:11:19 2017 From: johanna at icir.org (Johanna Amann) Date: Mon, 16 Oct 2017 15:11:19 -0700 Subject: [Bro] Bro 2.5.2 & 2.4.2 release (security update) Message-ID: <20171016221119.egzusbmjzllc4x5t@user190.sys.ICSI.Berkeley.EDU> We announce the release of Bro v2.5.2. The new version is now available for download at: https://bro.org/download/index.html or directly at: https://www.bro.org/downloads/bro-2.5.2.tar.gz Binary packages for the new version are currently building and will be available in the next hours at: https://bro.org/download/packages.html This is a security release that fixes an out-of-bound write in the ContentLine analyzer. This issue can be used by remote attackers to crash Bro (i.e. a DoS attack). There also is a possibility this can be exploited in other ways. This bug was found by Frank Meier. A CVE has been requested for this bug. Bro 2.5.2 does not contain any other changes. We urge everyone to update their installation as quickly as possible. Due to the potential severity of this bug we also provide a patched version of Bro v2.4.2. The only difference to version v2.4.1 is this bugfix. Please note that we encourage users to use version 2.5.2 instead; we do generally not provide security updates for old releases; version 2.4.2 is missing a number of other bugfixes that were applied to v2.5.2. Version 2.4.2 is available for download at: https://www.bro.org/downloads/bro-2.4.2.tar.gz Johanna From roberixion at gmail.com Tue Oct 17 03:06:12 2017 From: roberixion at gmail.com (=?UTF-8?Q?Rober_Fern=C3=A1ndez?=) Date: Tue, 17 Oct 2017 12:06:12 +0200 Subject: [Bro] Payload Message-ID: Hi, How can I get the payload of each connection and print each payload in a different file? Regards -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171017/e1801c8b/attachment.html From daniel_aka_sniper_d at hotmail.com Tue Oct 17 06:38:29 2017 From: daniel_aka_sniper_d at hotmail.com (Sniper) Date: Tue, 17 Oct 2017 13:38:29 +0000 Subject: [Bro] Documentation and getting started. In-Reply-To: References: Message-ID: Hello Jim, Thanks a lot just what I needed, trying to search for everything just takes up too much time. Running as root is just sudo -s then broclt right or do I need to change it as stated in that link you sent me? No tcpdump does not work against br0 but it runs agains eth0 and eth1. I have assigned a IP address to br0, is this even required? I tried to use OpenBSD to accomplish a network tap but brconfig that configures the bridge is not in the operating system for some reason. I gett an error saying its not recognised, after many hours of searching I couldn't find a solution. Linux is a pain in the backside, it takes up soo much time trying to find solutions to problems. This is my layout, I have put everything on the same subnet to just to get things started. VM ethernet adapter(my PC) 192.168.10.5 - no gateway Ubuntu (Bro/Bridge) br0 192.168.10.1 - no gateway eth0 192.168.10.2 - no gateway eth1 192.168.10.3 - no gateway Ubuntu Victim 192.168.10.6 - gw 192.168.10.2 Linux Kali Attacker 192.168.10.7 - gw 192.168.10.3 Regards Daniel On 16/10/2017 20:20, Jim Mellander wrote: Hi Daniel: Check this link for info on ? ? the bro directory structure that may help you: https://www.bro.org/sphinx/install/release-notes.html#script-organization ?As far as monitoring a bridged interface, there should be no problem, as long as bro can access the interface. ?If you're not running as root, see: https://www.bro.org/documentation/faq.html#how-can-i-capture-packets-as-an-unprivileged-user Does tcpdump provide expected output when run against br0? Hope this helps, Jim On Mon, Oct 16, 2017 at 9:49 AM, Sniper > wrote: Hello Everyone, Is there reference page on all the default installation directory locations are by any chance? $PREFIX just makes it a very long process establishing where all the files are located. If not, I think this would be excellent for beginners like me. Also, I have created a bridge interface that I want to monitor using ubuntu/bro by connecting two hosts, for some reason I can't seem to generate any logs in /usr/local/bro/logs/ (no 'current' folder when bro is started as in the documentation). Is this even possible to monitor a bridge interface using bro on the same host? I have already changed node.cfg interface to br0. There are no tutorials nowhere on how to actually get started, tried to follow the instructions but still no luck, ive been wasting days on this. If someone could point me in the right direction i'll greatly appreciate it. Kind regards Daniel --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro [https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif] Virus-free. www.avast.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171017/7aa55d89/attachment.html From therencamureithi at gmail.com Tue Oct 17 07:34:12 2017 From: therencamureithi at gmail.com (Therenca Mureithi) Date: Tue, 17 Oct 2017 17:34:12 +0300 Subject: [Bro] Fwd: Other log files besides conn.log In-Reply-To: References: Message-ID: ---------- Forwarded message ---------- From: Therenca Mureithi Date: Tue, Oct 17, 2017 at 5:30 PM Subject: Other log files besides conn.log To: bro at bro.org Is there a way to add mac address to log files like http.log, ssl.log, ssh.log, especially when the ip addresses are dynamic. I have been able to add mac address to the conn.log file following bro related threads. I am not skilled at bro scripting but i would very much like to have this functionality. Why? Due to the fact that i want to track down users of the network and at one point their ip addresses do change, however rarely do mac address change unless ofcourse you have spoofed it. Kindly reply. Anyone. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171017/e9ceca46/attachment-0001.html From daniel_aka_sniper_d at hotmail.com Tue Oct 17 08:25:04 2017 From: daniel_aka_sniper_d at hotmail.com (Sniper) Date: Tue, 17 Oct 2017 15:25:04 +0000 Subject: [Bro] Documentation and getting started. In-Reply-To: <7bb8aa51-ee63-c856-58ca-8a73bbb7a889@hotmail.com> References: <7bb8aa51-ee63-c856-58ca-8a73bbb7a889@hotmail.com> Message-ID: Ok so I removed eth0/1 from network connections, the ethernet connection so br0 has br0 slave 1 and 2 which has removed the IP addresses and is now using MAC addresses on eth 0/1. Now when I ping the br0 192.168.10.1 I get activity using tcpdump, however, when I ping hosts Attacker and Victim from eachother there is no activity on br0. Regards Daniel On 17/10/2017 14:38, Daniel wrote: Hello Jim, Thanks a lot just what I needed, trying to search for everything just takes up too much time. Running as root is just sudo -s then broclt right or do I need to change it as stated in that link you sent me? No tcpdump does not work against br0 but it runs agains eth0 and eth1. I have assigned a IP address to br0, is this even required? I tried to use OpenBSD to accomplish a network tap but brconfig that configures the bridge is not in the operating system for some reason. I gett an error saying its not recognised, after many hours of searching I couldn't find a solution. Linux is a pain in the backside, it takes up soo much time trying to find solutions to problems. This is my layout, I have put everything on the same subnet to just to get things started. VM ethernet adapter(my PC) 192.168.10.5 - no gateway Ubuntu (Bro/Bridge) br0 192.168.10.1 - no gateway eth0 192.168.10.2 - no gateway eth1 192.168.10.3 - no gateway Ubuntu Victim 192.168.10.6 - gw 192.168.10.2 Linux Kali Attacker 192.168.10.7 - gw 192.168.10.3 Regards Daniel On 16/10/2017 20:20, Jim Mellander wrote: Hi Daniel: Check this link for info on ? ? the bro directory structure that may help you: https://www.bro.org/sphinx/install/release-notes.html#script-organization ?As far as monitoring a bridged interface, there should be no problem, as long as bro can access the interface. ?If you're not running as root, see: https://www.bro.org/documentation/faq.html#how-can-i-capture-packets-as-an-unprivileged-user Does tcpdump provide expected output when run against br0? Hope this helps, Jim On Mon, Oct 16, 2017 at 9:49 AM, Sniper > wrote: Hello Everyone, Is there reference page on all the default installation directory locations are by any chance? $PREFIX just makes it a very long process establishing where all the files are located. If not, I think this would be excellent for beginners like me. Also, I have created a bridge interface that I want to monitor using ubuntu/bro by connecting two hosts, for some reason I can't seem to generate any logs in /usr/local/bro/logs/ (no 'current' folder when bro is started as in the documentation). Is this even possible to monitor a bridge interface using bro on the same host? I have already changed node.cfg interface to br0. There are no tutorials nowhere on how to actually get started, tried to follow the instructions but still no luck, ive been wasting days on this. If someone could point me in the right direction i'll greatly appreciate it. Kind regards Daniel --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro [https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif] Virus-free. www.avast.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171017/1a2677c7/attachment.html From vikrambasu059 at gmail.com Tue Oct 17 10:49:13 2017 From: vikrambasu059 at gmail.com (Vikram Basu) Date: Tue, 17 Oct 2017 23:19:13 +0530 Subject: [Bro] Extracting files transferred over smb Message-ID: <59e6429a.1b67620a.dd244.bf51@mx.google.com> Hi, Using hosom?s excellent file-extraction module for Bro, I am able to extract files transferred over FTP and HTTP. I am left wondering if however there is a way to extract files transferred over SMB as well. Bro already can track smb files from what I understand. How difficult would it be to extract files transferred over smb currently ? Also I lack any accessible SMTP server at the moment so I have to ask can bro extract files transferred over SMTP as well ? Regards Vikram Basu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171017/7f5b4f23/attachment.html From hosom at battelle.org Tue Oct 17 11:48:07 2017 From: hosom at battelle.org (Hosom, Stephen M) Date: Tue, 17 Oct 2017 18:48:07 +0000 Subject: [Bro] Extracting files transferred over smb In-Reply-To: <59e6429a.1b67620a.dd244.bf51@mx.google.com> References: <59e6429a.1b67620a.dd244.bf51@mx.google.com> Message-ID: <3c49afb34f584f67975cafc590862f99@battelle.org> File extraction over SMB should be fairly trivial. In fact, there's nothing limiting the plugin from doing it currently. Any of the extracted filetypes will be extracted regardless of protocol or direction--so long as Bro sees a file and it matches the extraction 'policy' configured in the plugin. If you wanted to find files specifically being extracted from SMB, look in your files.log for entries where the source field is SMB and the extracted value isn't unset (which by default is "-"). If you're interested in a plugin that specifically targets files transferred over SMB... I could see the usefulness of that and would gladly write it sometime in the next couple nights. Thanks, Stephen ________________________________ From: bro-bounces at bro.org on behalf of Vikram Basu Sent: Tuesday, October 17, 2017 1:49:13 PM To: bro at bro.org Subject: [Bro] Extracting files transferred over smb Message received from outside the Battelle network. Carefully examine it before you open any links or attachments. Hi, Using hosom?s excellent file-extraction module for Bro, I am able to extract files transferred over FTP and HTTP. I am left wondering if however there is a way to extract files transferred over SMB as well. Bro already can track smb files from what I understand. How difficult would it be to extract files transferred over smb currently ? Also I lack any accessible SMTP server at the moment so I have to ask can bro extract files transferred over SMTP as well ? Regards Vikram Basu From jmellander at lbl.gov Tue Oct 17 13:37:55 2017 From: jmellander at lbl.gov (Jim Mellander) Date: Tue, 17 Oct 2017 13:37:55 -0700 Subject: [Bro] Documentation and getting started. In-Reply-To: References: <7bb8aa51-ee63-c856-58ca-8a73bbb7a889@hotmail.com> Message-ID: Assuming that you're just doing the bonding for monitoring purposes, you could also have bro monitor multiple interfaces, see: http://mailman.icsi.berkeley.edu/pipermail/bro/2014-January/006477.html Running bro as root is possible, but could be a security risk - the setcap method is safer & better. Hope this helps, Jim On Tue, Oct 17, 2017 at 8:25 AM, Sniper wrote: > Ok so I removed eth0/1 from network connections, the ethernet connection > so br0 has br0 slave 1 and 2 which has removed the IP addresses and is now > using MAC addresses on eth 0/1. > > Now when I ping the br0 192.168.10.1 I get activity using tcpdump, > however, when I ping hosts Attacker and Victim from eachother there is no > activity on br0. > Regards > Daniel > > > On 17/10/2017 14:38, Daniel wrote: > > Hello Jim, > > Thanks a lot just what I needed, trying to search for everything just > takes up too much time. > > Running as root is just sudo -s then broclt right or do I need to change > it as stated in that link you sent me? > No tcpdump does not work against br0 but it runs agains eth0 and eth1. I > have assigned a IP address to br0, is this even required? I tried to use > OpenBSD to accomplish a network tap but brconfig that configures the bridge > is not in the operating system for some reason. I gett an error saying its > not recognised, after many hours of searching I couldn't find a solution. > > Linux is a pain in the backside, it takes up soo much time trying to find > solutions to problems. > > This is my layout, I have put everything on the same subnet to just to get > things started. > > VM ethernet adapter(my PC) > 192.168.10.5 - no gateway > > Ubuntu (Bro/Bridge) > br0 192.168.10.1 - no gateway > eth0 192.168.10.2 - no gateway > eth1 192.168.10.3 - no gateway > > Ubuntu Victim > 192.168.10.6 - gw 192.168.10.2 > > Linux Kali Attacker > 192.168.10.7 - gw 192.168.10.3 > > Regards > Daniel > > On 16/10/2017 20:20, Jim Mellander wrote: > > Hi Daniel: > > Check this link for info on > ? ? > the bro directory structure that may help you: > https://www.bro.org/sphinx/install/release-notes.html#script-organization > > ?As far as monitoring a bridged interface, there should be no problem, as > long as bro can access the interface. ?If you're not running as root, see: > https://www.bro.org/documentation/faq.html#how- > can-i-capture-packets-as-an-unprivileged-user > > Does tcpdump provide expected output when run against br0? > > Hope this helps, > > Jim > > > > > On Mon, Oct 16, 2017 at 9:49 AM, Sniper > wrote: > >> Hello Everyone, >> >> Is there reference page on all the default installation directory >> locations are by any chance? $PREFIX just makes it a very long process >> establishing where all the files are located. If not, I think this would >> be excellent for beginners like me. >> >> Also, I have created a bridge interface that I want to monitor using >> ubuntu/bro by connecting two hosts, for some reason I can't seem to >> generate any logs in /usr/local/bro/logs/ (no 'current' folder when bro >> is started as in the documentation). Is this even possible to monitor a >> bridge interface using bro on the same host? I have already changed >> node.cfg interface to br0. >> >> There are no tutorials nowhere on how to actually get started, tried to >> follow the instructions but still no luck, ive been wasting days on >> this. If someone could point me in the right direction i'll greatly >> appreciate it. >> >> Kind regards >> >> Daniel >> >> >> >> --- >> This email has been checked for viruses by Avast antivirus software. >> https://www.avast.com/antivirus >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > > > > > Virus-free. > www.avast.com > > <#m_1024518197508363682_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171017/50acce81/attachment.html From jmellander at lbl.gov Tue Oct 17 15:55:46 2017 From: jmellander at lbl.gov (Jim Mellander) Date: Tue, 17 Oct 2017 15:55:46 -0700 Subject: [Bro] Fwd: Other log files besides conn.log In-Reply-To: References: Message-ID: ?Hi Therenca: You could add this to local.bro: @load policy/protocols/conn/mac-logging However, unless you're actually directly monitoring inside the border of a subnet, the host MAC address will not be seen, but the MAC addresses of the routers, so this may not be too useful. Depending on your network topology, dhcp.log might have some information on the mapping. You could also check your DHCP server's logs, which should have the information you need. Hope this helps, Jim On Tue, Oct 17, 2017 at 7:34 AM, Therenca Mureithi < therencamureithi at gmail.com> wrote: > > ---------- Forwarded message ---------- > From: Therenca Mureithi > Date: Tue, Oct 17, 2017 at 5:30 PM > Subject: Other log files besides conn.log > To: bro at bro.org > > > Is there a way to add mac address to log files like http.log, ssl.log, > ssh.log, especially when the ip addresses are dynamic. I have been able to > add mac address to the conn.log file following bro related threads. I am > not skilled at bro scripting but i would very much like to have this > functionality. Why? Due to the fact that i want to track down users of the > network and at one point their ip addresses do change, however rarely do > mac address change unless ofcourse you have spoofed it. Kindly reply. > Anyone. > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171017/2e0b397f/attachment.html From dotwayland at gmail.com Tue Oct 17 16:10:00 2017 From: dotwayland at gmail.com (Wayland Morgan) Date: Tue, 17 Oct 2017 23:10:00 +0000 Subject: [Bro] Fwd: Other log files besides conn.log In-Reply-To: References: Message-ID: I?m not the authority on Bro?s capabilities but http.log, ssl.log, ssh.log are all protocol specific and none have any notion of hardware addresses. If you?re looking to perform user attribution then I recommend pairing these logs with DHCP data to obtain a hardware address which you can in turn correlate with your lower layer information sources not processed by Bro such as ARP and switch port data. You may also get some mileage out of querying any domain specific authentication data where an explicit set of user credentials was used to authenticate from a device. If you?re doing any kind of centralized logging with something like ELK or Splunk you might be able to create a custom search that pulls hardware addresses into the logs you named, but as far as I know Bro won?t do this natively (nor should it). Hope this helps. Wayland On Tue, Oct 17, 2017 at 9:44 AM Therenca Mureithi < therencamureithi at gmail.com> wrote: > > ---------- Forwarded message ---------- > From: Therenca Mureithi > Date: Tue, Oct 17, 2017 at 5:30 PM > Subject: Other log files besides conn.log > To: bro at bro.org > > > Is there a way to add mac address to log files like http.log, ssl.log, > ssh.log, especially when the ip addresses are dynamic. I have been able to > add mac address to the conn.log file following bro related threads. I am > not skilled at bro scripting but i would very much like to have this > functionality. Why? Due to the fact that i want to track down users of the > network and at one point their ip addresses do change, however rarely do > mac address change unless ofcourse you have spoofed it. Kindly reply. > Anyone. > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Wayland -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171017/b3b0866d/attachment-0001.html From dotwayland at gmail.com Tue Oct 17 16:13:41 2017 From: dotwayland at gmail.com (Wayland Morgan) Date: Tue, 17 Oct 2017 23:13:41 +0000 Subject: [Bro] Fwd: Other log files besides conn.log In-Reply-To: References: Message-ID: I was unaware of the mac-logging option. Thanks for sharing. On Tue, Oct 17, 2017 at 6:04 PM Jim Mellander wrote: > ?Hi Therenca: > > You could add this to local.bro: > > @load policy/protocols/conn/mac-logging > > However, unless you're actually directly monitoring inside the border of a > subnet, the host MAC address will not be seen, but the MAC addresses of the > routers, so this may not be too useful. > > Depending on your network topology, dhcp.log might have some information > on the mapping. You could also check your DHCP server's logs, which should > have the information you need. > > Hope this helps, > > Jim > > > > > > On Tue, Oct 17, 2017 at 7:34 AM, Therenca Mureithi < > therencamureithi at gmail.com> wrote: > >> >> ---------- Forwarded message ---------- >> From: Therenca Mureithi >> Date: Tue, Oct 17, 2017 at 5:30 PM >> Subject: Other log files besides conn.log >> To: bro at bro.org >> >> >> Is there a way to add mac address to log files like http.log, ssl.log, >> ssh.log, especially when the ip addresses are dynamic. I have been able to >> add mac address to the conn.log file following bro related threads. I am >> not skilled at bro scripting but i would very much like to have this >> functionality. Why? Due to the fact that i want to track down users of the >> network and at one point their ip addresses do change, however rarely do >> mac address change unless ofcourse you have spoofed it. Kindly reply. >> Anyone. >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Wayland -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171017/7c7932db/attachment.html From roberixion at gmail.com Wed Oct 18 03:23:24 2017 From: roberixion at gmail.com (=?UTF-8?Q?Rober_Fern=C3=A1ndez?=) Date: Wed, 18 Oct 2017 12:23:24 +0200 Subject: [Bro] Share vector Message-ID: Hi, I would like to access to the same vector and modificate it in two different scripts, How can I do this? Regards -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171018/3d4a8070/attachment.html From daniel_aka_sniper_d at hotmail.com Thu Oct 19 08:39:15 2017 From: daniel_aka_sniper_d at hotmail.com (Sniper) Date: Thu, 19 Oct 2017 15:39:15 +0000 Subject: [Bro] Mininet Message-ID: Hello, Just wondering if anyone has tryed to use Bro on an openflow-based network using mininet? Kind regards Daniel --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus From bill.de.ping at gmail.com Thu Oct 19 09:03:35 2017 From: bill.de.ping at gmail.com (william de ping) Date: Thu, 19 Oct 2017 19:03:35 +0300 Subject: [Bro] Share vector In-Reply-To: References: Message-ID: Hi I guess you can put that vector in one script, and call it from the second script using the first script module name : script 1 : module S1; export { global vec: vector of string; }; ------- script 2 : module S2; event bro_init() { S1::vec[0]="hello"; } On Wed, Oct 18, 2017 at 1:23 PM, Rober Fern?ndez wrote: > Hi, > > I would like to access to the same vector and modificate it in two > different scripts, How can I do this? > > Regards > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171019/43f5ee28/attachment.html From roberixion at gmail.com Thu Oct 19 09:14:18 2017 From: roberixion at gmail.com (=?UTF-8?Q?Rober_Fern=C3=A1ndez?=) Date: Thu, 19 Oct 2017 18:14:18 +0200 Subject: [Bro] Share vector In-Reply-To: References: Message-ID: Hi I would like to do something like this script 1 : module S1; export { global vec: vector of string; }; event bro_init() { vec[0] = "hello" } And script2 can print vec ------- script 2 : module S2; event bro_init() { print S1::vec } Output: ["hello"] 2017-10-19 18:03 GMT+02:00 william de ping : > Hi > > I guess you can put that vector in one script, and call it from the second > script using the first script module name : > > script 1 : > > module S1; > > export { > global vec: vector of string; > }; > > > ------- > script 2 : > > module S2; > > event bro_init() > { > S1::vec[0]="hello"; > } > > > > On Wed, Oct 18, 2017 at 1:23 PM, Rober Fern?ndez > wrote: > >> Hi, >> >> I would like to access to the same vector and modificate it in two >> different scripts, How can I do this? >> >> Regards >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171019/b92ce797/attachment.html From wren3 at illinois.edu Thu Oct 19 15:01:11 2017 From: wren3 at illinois.edu (Ren, Wenyu) Date: Thu, 19 Oct 2017 22:01:11 +0000 Subject: [Bro] Use of suspend_processing and continue_processing messes up network_time Message-ID: Dear all, Have anyone using suspend_processing() and continue_processing() have problem with the network_time() function? I found that when those two functions are used, sometimes network_time() called for each packet all return the same time. Since network_time() returns the network time of the last packet processed, I guess this has something to do with the suspend_processing() and continue_processing() messing up the order of when the event for each packet is triggered. Any idea? Any help is appreciated. Best, Wenyu Wenyu Ren Ph.D. Candidate Department of Computer Science University of Illinois at Urbana-Champaign From wren3 at illinois.edu Thu Oct 19 15:09:00 2017 From: wren3 at illinois.edu (Ren, Wenyu) Date: Thu, 19 Oct 2017 22:09:00 +0000 Subject: [Bro] Use of suspend_processing and continue_processing messes up network_time In-Reply-To: References: Message-ID: To be more specific, I found sometimes (sometimes not) the network_time() will return the current wall time instead of the packet time if suspend_processing and continue_processing are used. Wenyu Ren Ph.D. Candidate Department of Computer Science University of Illinois at Urbana-Champaign ________________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Ren, Wenyu [wren3 at illinois.edu] Sent: Thursday, October 19, 2017 5:01 PM To: bro at bro.org Subject: [Bro] Use of suspend_processing and continue_processing messes up network_time Dear all, Have anyone using suspend_processing() and continue_processing() have problem with the network_time() function? I found that when those two functions are used, sometimes network_time() called for each packet all return the same time. Since network_time() returns the network time of the last packet processed, I guess this has something to do with the suspend_processing() and continue_processing() messing up the order of when the event for each packet is triggered. Any idea? Any help is appreciated. Best, Wenyu Wenyu Ren Ph.D. Candidate Department of Computer Science University of Illinois at Urbana-Champaign _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Thu Oct 19 15:38:38 2017 From: johanna at icir.org (Johanna Amann) Date: Thu, 19 Oct 2017 15:38:38 -0700 Subject: [Bro] Use of suspend_processing and continue_processing messes up network_time In-Reply-To: References: Message-ID: <20171019223838.h2xtvpqjyncc7ube@Beezling.local> This sounds like a bug - if this is easily reproducable, could you create a ticket on the tracker containing the steps on how to reproduce this? Thanks :) Johanna On Thu, Oct 19, 2017 at 10:09:00PM +0000, Ren, Wenyu wrote: > To be more specific, I found sometimes (sometimes not) the network_time() will return the current wall time instead of the packet time if suspend_processing and continue_processing are used. > > > Wenyu Ren > Ph.D. Candidate > Department of Computer Science > University of Illinois at Urbana-Champaign > > ________________________________________ > From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Ren, Wenyu [wren3 at illinois.edu] > Sent: Thursday, October 19, 2017 5:01 PM > To: bro at bro.org > Subject: [Bro] Use of suspend_processing and continue_processing messes up network_time > > Dear all, > > Have anyone using suspend_processing() and continue_processing() have problem with the network_time() function? I found that when those two functions are used, sometimes network_time() called for each packet all return the same time. Since network_time() returns the network time of the last packet processed, I guess this has something to do with the suspend_processing() and continue_processing() messing up the order of when the event for each packet is triggered. > > Any idea? Any help is appreciated. > > Best, > Wenyu > > Wenyu Ren > Ph.D. Candidate > Department of Computer Science > University of Illinois at Urbana-Champaign > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From dopheide at gmail.com Thu Oct 19 19:19:39 2017 From: dopheide at gmail.com (Mike Dopheide) Date: Thu, 19 Oct 2017 21:19:39 -0500 Subject: [Bro] Share vector In-Reply-To: References: Message-ID: So, have you tried it? :) Two things to consider: 1) event &priority and 2) If you're running Bro in standalone vs cluster mode. Correcting a couple syntax issues and adding a priority: Script1: module S1; export { global vec: vector of string; } event bro_init() &priority=1 { vec[0] = "hello"; } Script2: module S2; event bro_init() { print S1::vec; } $ /usr/local/bro/bin/bro -i en0 script1.bro script2.bro listening on en0 [hello] It gets more complicated if you're running in a cluster. -Dop On Thu, Oct 19, 2017 at 11:14 AM, Rober Fern?ndez wrote: > Hi > > I would like to do something like this > > script 1 : > > module S1; > > export { > global vec: vector of string; > }; > > event bro_init() { > vec[0] = "hello" > } > > > And script2 can print vec > ------- > script 2 : > > module S2; > > event bro_init() > { > print S1::vec > } > > Output: > ["hello"] > > 2017-10-19 18:03 GMT+02:00 william de ping : > >> Hi >> >> I guess you can put that vector in one script, and call it from the >> second script using the first script module name : >> >> script 1 : >> >> module S1; >> >> export { >> global vec: vector of string; >> }; >> >> >> ------- >> script 2 : >> >> module S2; >> >> event bro_init() >> { >> S1::vec[0]="hello"; >> } >> >> >> >> On Wed, Oct 18, 2017 at 1:23 PM, Rober Fern?ndez >> wrote: >> >>> Hi, >>> >>> I would like to access to the same vector and modificate it in two >>> different scripts, How can I do this? >>> >>> Regards >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >> >> > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171019/a21409d1/attachment.html From charles.fair at mac.com Sat Oct 21 22:50:47 2017 From: charles.fair at mac.com (Charles A. Fair) Date: Sun, 22 Oct 2017 00:50:47 -0500 Subject: [Bro] Other log files besides conn.log In-Reply-To: References: Message-ID: ---------- Forwarded message ---------- > From: Therenca Mureithi > Date: Tue, Oct 17, 2017 at 5:30 PM > Subject: Other log files besides conn.log > To: bro at bro.org > > > Is there a way to add mac address to log files like http.log, ssl.log, ssh.log, especially when the ip addresses are dynamic. I have been able to add mac address to the conn.log file following bro related threads. I am not skilled at bro scripting but i would very much like to have this functionality. Why? Due to the fact that i want to track down users of the network and at one point their ip addresses do change, however rarely do mac address change unless ofcourse you have spoofed it. Kindly reply. Anyone. Check out how the Bro logs are modified in ROCK: http://rocknsm.io We Have added in ASN's in each log along with the IP addresses. You could replicate, so the fields would be directly in the Bro logs of choice, but with the MAC addresses logged in the conn.log after enabling that policy script. Charles "Chuck" A. Fair From bill.de.ping at gmail.com Mon Oct 23 02:42:17 2017 From: bill.de.ping at gmail.com (william de ping) Date: Mon, 23 Oct 2017 12:42:17 +0300 Subject: [Bro] bro arithmetic issue - int, double, count Message-ID: Hi, I'm having a strange issue regarding arithmetic of large numbers : local a = 1.054929e+10; is the type of a = double ? thanks B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171023/e11474f3/attachment.html From dnthayer at illinois.edu Mon Oct 23 08:04:22 2017 From: dnthayer at illinois.edu (Daniel Thayer) Date: Mon, 23 Oct 2017 10:04:22 -0500 Subject: [Bro] bro arithmetic issue - int, double, count In-Reply-To: References: Message-ID: <1f83626b-e8d4-7743-e654-2196eb9ca2de@illinois.edu> On 10/23/17 4:42 AM, william de ping wrote: > Hi, > > I'm having a strange issue regarding arithmetic of large numbers : > > local a = 1.054929e+10; > > is the type of a = double ? > > thanks > B You can check the type by using the "type_name" built-in function: print type_name(a); From bluebike.sjlee at gmail.com Tue Oct 24 10:13:26 2017 From: bluebike.sjlee at gmail.com (SJ Lee) Date: Tue, 24 Oct 2017 13:13:26 -0400 Subject: [Bro] Question about disable lookup_addr Message-ID: Hello, Looking at reverse dns record, seeing a lot of record from the IDS sensor nodes. And found bro calling lookup_addr function in few files. I was trying to disable all lookup_addr function, but below files not able to disable due to dependency issue. Is here my question, is there any easy way to disable lookup_addr function? OR restrict internal dns db ONLY not want to hit external dns server, is there any way can do this? 1) /opt/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro: when ( local h1name = lookup_addr(h1) ) /opt/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro: when ( local h2name = lookup_addr(h2) ) /opt/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro: when ( local h2name_ = lookup_addr(h2) ) 2) /opt/bro/share/bro/base/bif/bro.bif.bro:global lookup_addr: function(host: addr ) : string ; Thanks, SJ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171024/177d5470/attachment.html From jazoff at illinois.edu Tue Oct 24 11:41:13 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 24 Oct 2017 18:41:13 +0000 Subject: [Bro] Question about disable lookup_addr In-Reply-To: References: Message-ID: <4A0F3B34-1D17-451C-A0B7-A9C0445D07DD@illinois.edu> > On Oct 24, 2017, at 1:13 PM, SJ Lee wrote: > > Hello, > > Looking at reverse dns record, seeing a lot of record from the IDS sensor nodes. > And found bro calling lookup_addr function in few files. Set the BRO_DNS_FAKE environment variable and bro will not use real dns.. > I was trying to disable all lookup_addr function, but below files not able to disable due to dependency issue. > > Is here my question, is there any easy way to disable lookup_addr function? > OR restrict internal dns db ONLY not want to hit external dns server, is there any way can do this? Bro will use whatever servers are configured in /etc/resolv.conf > 1) /opt/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro: when ( local h1name = lookup_addr(h1) ) > /opt/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro: when ( local h2name = lookup_addr(h2) ) > /opt/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro: when ( local h2name_ = lookup_addr(h2) ) >From that script: ##! Notice extension that mails out a pretty-printed version of alarm.log ##! in regular intervals, formatted for better human readability. If activated, ##! that replaces the default summary mail having the raw log output. @load base/frameworks/cluster @load ../main module Notice; export { ## Activate pretty-printed alarm summaries. const pretty_print_alarms = T &redef; So, that is easily disabled. ? Justin Azoff From bluebike.sjlee at gmail.com Tue Oct 24 11:59:17 2017 From: bluebike.sjlee at gmail.com (SJ Lee) Date: Tue, 24 Oct 2017 14:59:17 -0400 Subject: [Bro] Question about disable lookup_addr In-Reply-To: <4A0F3B34-1D17-451C-A0B7-A9C0445D07DD@illinois.edu> References: <4A0F3B34-1D17-451C-A0B7-A9C0445D07DD@illinois.edu> Message-ID: Hello Justin, Thank you for your quick response mail. Bro will use whatever servers are configured in /etc/resolv.conf => Good to know this. Thank you. export { ## Activate pretty-printed alarm summaries. const pretty_print_alarms = T &redef; => easily disabled, means instead of using T, I can set up F, than disable this feature? Thanks, SJ On Tue, Oct 24, 2017 at 2:41 PM, Azoff, Justin S wrote: > > > On Oct 24, 2017, at 1:13 PM, SJ Lee wrote: > > > > Hello, > > > > Looking at reverse dns record, seeing a lot of record from the IDS > sensor nodes. > > And found bro calling lookup_addr function in few files. > > Set the BRO_DNS_FAKE environment variable and bro will not use real dns.. > > > I was trying to disable all lookup_addr function, but below files not > able to disable due to dependency issue. > > > > Is here my question, is there any easy way to disable lookup_addr > function? > > OR restrict internal dns db ONLY not want to hit external dns server, > is there any way can do this? > > Bro will use whatever servers are configured in /etc/resolv.conf > > > > 1) /opt/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro: > when ( local h1name = lookup_addr(h1) ) > > /opt/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro: > when ( local h2name = lookup_addr(h2) ) > > /opt/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro: > when ( local h2name_ = lookup_addr(h2) ) > > From that script: > > ##! Notice extension that mails out a pretty-printed version of alarm.log > ##! in regular intervals, formatted for better human readability. If > activated, > ##! that replaces the default summary mail having the raw log output. > > @load base/frameworks/cluster > @load ../main > > module Notice; > > export { > ## Activate pretty-printed alarm summaries. > const pretty_print_alarms = T &redef; > > > So, that is easily disabled. > > ? > Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171024/4e593a4e/attachment.html From seth at corelight.com Tue Oct 24 12:45:44 2017 From: seth at corelight.com (Seth Hall) Date: Tue, 24 Oct 2017 15:45:44 -0400 Subject: [Bro] Question about disable lookup_addr In-Reply-To: References: Message-ID: That script should only run if you are turning some notices into alarms. I suspect that the look ups you are seeing are due to something else. The two primary scripts that are probably causing DNS lookups are: policy/protocols/ssh/interesting-hostnames.bro policy/frameworks/files/detect-MHR.bro .Seth On 24 Oct 2017, at 13:13, SJ Lee wrote: > Hello, > > Looking at reverse dns record, seeing a lot of record from the IDS > sensor > nodes. > And found bro calling lookup_addr function in few files. > > I was trying to disable all lookup_addr function, but below files not > able > to disable due to dependency issue. > > Is here my question, is there any easy way to disable lookup_addr > function? > OR restrict internal dns db ONLY not want to hit external dns > server, is > there any way can do this? > > > 1) /opt/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro: > when > ( local h1name = lookup_addr(h1) ) > /opt/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro: > when ( local h2name = lookup_addr(h2) ) > /opt/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro: > when ( local h2name_ = lookup_addr(h2) ) > > 2) /opt/bro/share/bro/base/bif/bro.bif.bro:global lookup_addr: > function(host: addr ) : string ; > > Thanks, > SJ > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * www.corelight.com From bluebike.sjlee at gmail.com Tue Oct 24 13:26:50 2017 From: bluebike.sjlee at gmail.com (SJ Lee) Date: Tue, 24 Oct 2017 16:26:50 -0400 Subject: [Bro] Question about disable lookup_addr In-Reply-To: References: Message-ID: Hello Seth, I checked policy/frameworks/files/detect-MHR.bro, but does not able to fine function for lookup_addr but seeing - when ( local MHR_result = lookup_hostname_txt(hash_domain) ) Is this also related with dns lookup? Thanks, SJ On Tue, Oct 24, 2017 at 3:45 PM, Seth Hall wrote: > That script should only run if you are turning some notices into alarms. > I suspect that the look ups you are seeing are due to something else. The > two primary scripts that are probably causing DNS lookups are: > policy/protocols/ssh/interesting-hostnames.bro > policy/frameworks/files/detect-MHR.bro > > .Seth > > > On 24 Oct 2017, at 13:13, SJ Lee wrote: > > Hello, >> >> Looking at reverse dns record, seeing a lot of record from the IDS sensor >> nodes. >> And found bro calling lookup_addr function in few files. >> >> I was trying to disable all lookup_addr function, but below files not able >> to disable due to dependency issue. >> >> Is here my question, is there any easy way to disable lookup_addr >> function? >> OR restrict internal dns db ONLY not want to hit external dns server, is >> there any way can do this? >> >> >> 1) /opt/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro: >> when >> ( local h1name = lookup_addr(h1) ) >> /opt/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro: >> when ( local h2name = lookup_addr(h2) ) >> /opt/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro: >> when ( local h2name_ = lookup_addr(h2) ) >> >> 2) /opt/bro/share/bro/base/bif/bro.bif.bro:global lookup_addr: >> function(host: addr ) : string ; >> >> Thanks, >> SJ >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > -- > Seth Hall * Corelight, Inc * www.corelight.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171024/2a67503f/attachment.html From roberixion at gmail.com Wed Oct 25 03:06:12 2017 From: roberixion at gmail.com (=?UTF-8?Q?Rober_Fern=C3=A1ndez?=) Date: Wed, 25 Oct 2017 12:06:12 +0200 Subject: [Bro] Origin data Message-ID: hi, when I try to capture the connection data, my origin data is not captured. In wireshark it works. What is the problem? some kind of bro configuration? -rw-r--r-- 1 183 oct 25 11:49 contents_X_resp.dat -rw-r--r-- 1 0 oct 25 11:49 contents_X_orig.dat <---- 0 bytes -rw-r--r-- 1 183 oct 25 11:49 contents_X_resp.dat -rw-r--r-- 1 0 oct 25 11:49 contents_X_orig.dat <---- 0 -rw-r--r-- 1 183 oct 25 11:49 contents_X_resp.dat -rw-r--r-- 1 0 oct 25 11:49 contents_X_orig.dat <---- 0 -rw-r--r-- 1 2,8K oct 25 11:50 contents_X_resp.dat Also, in http.log, doesn't appear the parameters, method, uri,host 1508923559.491507 CdS3HN1j2ou0LUObRb X 59772 X 80 1 - - - - 1.1 - 0 4685 200 OK - - (empty) - -- - - - F152dj2pHXhPN1wXng - image/jpeg -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171025/798f8128/attachment.html From seth at corelight.com Wed Oct 25 06:51:27 2017 From: seth at corelight.com (Seth Hall) Date: Wed, 25 Oct 2017 09:51:27 -0400 Subject: [Bro] Origin data In-Reply-To: References: Message-ID: <7A3BB742-9719-4A2C-89D0-FD318D33A1C5@corelight.com> On 25 Oct 2017, at 6:06, Rober Fern?ndez wrote: > when I try to capture the connection data, my origin data is not captured. > In wireshark it works. What is the problem? some kind of bro configuration? How are you declaring that you want to capture contents? .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From seth at corelight.com Wed Oct 25 06:52:30 2017 From: seth at corelight.com (Seth Hall) Date: Wed, 25 Oct 2017 09:52:30 -0400 Subject: [Bro] Question about disable lookup_addr In-Reply-To: References: Message-ID: <5FBE067C-4BE2-4B9F-8B05-6DB409D3DB71@corelight.com> On 24 Oct 2017, at 16:26, SJ Lee wrote: > I checked policy/frameworks/files/detect-MHR.bro, but does not able to fine > function for lookup_addr but seeing > - when ( local MHR_result = lookup_hostname_txt(hash_domain) ) > > Is this also related with dns lookup? Yep. All of the DNS related functions are: lookup_addr lookup_hostname_txt lookup_hostname .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From bluebike.sjlee at gmail.com Wed Oct 25 07:23:10 2017 From: bluebike.sjlee at gmail.com (SJ Lee) Date: Wed, 25 Oct 2017 10:23:10 -0400 Subject: [Bro] Question about disable lookup_addr In-Reply-To: <5FBE067C-4BE2-4B9F-8B05-6DB409D3DB71@corelight.com> References: <5FBE067C-4BE2-4B9F-8B05-6DB409D3DB71@corelight.com> Message-ID: Thank you Seth, this helps me a lot. --SJ On Wed, Oct 25, 2017 at 9:52 AM, Seth Hall wrote: > > > On 24 Oct 2017, at 16:26, SJ Lee wrote: > > > I checked policy/frameworks/files/detect-MHR.bro, but does not able to > fine > > function for lookup_addr but seeing > > - when ( local MHR_result = lookup_hostname_txt(hash_domain) ) > > > > Is this also related with dns lookup? > > Yep. All of the DNS related functions are: > lookup_addr > lookup_hostname_txt > lookup_hostname > > .Seth > > -- > Seth Hall * Corelight, Inc * www.corelight.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171025/bc2b2135/attachment-0001.html From jazoff at illinois.edu Wed Oct 25 08:04:08 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 25 Oct 2017 15:04:08 +0000 Subject: [Bro] Origin data In-Reply-To: References: Message-ID: <4A808361-D499-4237-8467-4E9BBB3B2202@illinois.edu> > On Oct 25, 2017, at 6:06 AM, Rober Fern?ndez wrote: > > hi, > when I try to capture the connection data, my origin data is not captured. In wireshark it works. What is the problem? some kind of bro configuration? Yes, https://www.bro.org/documentation/faq.html#why-isn-t-bro-producing-the-logs-i-expect-a-note-about-checksums ? Justin Azoff From hector.pena at relativity.com Wed Oct 25 14:40:11 2017 From: hector.pena at relativity.com (Hector Pena) Date: Wed, 25 Oct 2017 21:40:11 +0000 Subject: [Bro] Scanned Unique Host Message-ID: <8655B0F9-A35E-4546-B4C2-416CB114575F@contoso.com> Hi, Is there a way to view which host were scanned when receiving a notice for the scan.bro script? We have been receiving a lot of notices lately for ?x.x.x.x scanned at least X unique hosts on port X in Xtime?. I cannot seem to find a good way to determine which host were scanned by the host machine. Thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171025/109f1bb5/attachment.html From bill.de.ping at gmail.com Thu Oct 26 07:02:40 2017 From: bill.de.ping at gmail.com (william de ping) Date: Thu, 26 Oct 2017 17:02:40 +0300 Subject: [Bro] - Exec::run when reading a pcap Message-ID: Hi, I'm having troubles running a simple script on bro 2.5 reading a pcap file: @load base/utils/exec #redef exit_only_after_terminate=T; event bro_init() { print "Hello, World!"; local t="echo 123 ; echo 456"; local cmd=Exec::Command($cmd=t); when (local res = Exec::run(cmd)) { print "start !"; print res$stdout; print "finish !"; } } the exec commands runs successfully on try.bro.org, but prints nothing in the "when" bracket when running on local bro (v2.5). Is there a simple way of running a command with arguments and getting its return value right away ? system("ls"); works great but by design I can't get its results. any thoughts ? Thanks B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171026/28b2c991/attachment.html From seth at corelight.com Thu Oct 26 07:11:26 2017 From: seth at corelight.com (Seth Hall) Date: Thu, 26 Oct 2017 10:11:26 -0400 Subject: [Bro] - Exec::run when reading a pcap In-Reply-To: References: Message-ID: <98E6AFF5-84EC-411B-BC44-9E3C5BCA6CC3@corelight.com> On 26 Oct 2017, at 10:02, william de ping wrote: > #redef exit_only_after_terminate=T; Uncomment this and it works. Bro can't do anything by blocking script execution but if there is no input or you haven't explicitly told it to avoid shutting down then it will shutdown when there is no more input. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From bill.de.ping at gmail.com Thu Oct 26 07:36:57 2017 From: bill.de.ping at gmail.com (william de ping) Date: Thu, 26 Oct 2017 17:36:57 +0300 Subject: [Bro] - Exec::run when reading a pcap In-Reply-To: <98E6AFF5-84EC-411B-BC44-9E3C5BCA6CC3@corelight.com> References: <98E6AFF5-84EC-411B-BC44-9E3C5BCA6CC3@corelight.com> Message-ID: Hi Seth, after uncommenting this line, bro is stuck and won't terminate on its own.. On Thu, Oct 26, 2017 at 5:11 PM, Seth Hall wrote: > > > On 26 Oct 2017, at 10:02, william de ping wrote: > > #redef exit_only_after_terminate=T; >> > > Uncomment this and it works. Bro can't do anything by blocking script > execution but if there is no input or you haven't explicitly told it to > avoid shutting down then it will shutdown when there is no more input. > > .Seth > > -- > Seth Hall * Corelight, Inc * www.corelight.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171026/710f181a/attachment.html From dnthayer at illinois.edu Thu Oct 26 08:38:52 2017 From: dnthayer at illinois.edu (Daniel Thayer) Date: Thu, 26 Oct 2017 10:38:52 -0500 Subject: [Bro] - Exec::run when reading a pcap In-Reply-To: References: <98E6AFF5-84EC-411B-BC44-9E3C5BCA6CC3@corelight.com> Message-ID: On 10/26/17 9:36 AM, william de ping wrote: > Hi Seth, > > after uncommenting this line, bro is stuck and won't terminate on its own.. > Add this line immediately after the 'print "finish !";' line: terminate(); From r.bortolameotti at utwente.nl Fri Oct 27 07:33:01 2017 From: r.bortolameotti at utwente.nl (BortolameottiR) Date: Fri, 27 Oct 2017 16:33:01 +0200 Subject: [Bro] How information is stored in a set() and table() in bro Message-ID: <14f2b5e8-2ef4-14b8-8713-4954326da433@utwente.nl> Hi everyone, Some time ago I have dumped several log files using Bro. I had used the script in the attachment. Essentially, during the event http_all_headers I wanted to dump into the log the set of headers and values. I would like to re-use this dataset I have collected, however this time I would require the headers to be in the exact order as they are parsed. This information is not in logs, so I was wondering whether it is possible: given the script and the logs, to "reverse" the original order of the headers. For this purpose, I think I need to know:? 1) how data is stored in a "set [string]", because that's what I use to temporarily store the values; and 2) how data is stored in "hlist: mime_header_list" which is a table() in Bro. Can anyone help me? I already have a script that capture the headers in order with Bro, but this would require me to re-capture the data for long period of time. If I could reverse the process, it would save me quite some time. Thanks in advance, Riccardo -------------- next part -------------- @load /opt/bro/share/bro/base/protocols/http @load /opt/bro/share/bro/base/protocols/conn redef record HTTP::Info += { ## Write in the log ALL header names and their values header_values: set[string] &optional &log; ## Add the MAC address of origin of the connection mac_orig: string &optional &log; }; event bro_init() { local filter: Log::Filter = [$name="decanter_http", $path="decanter", $include=set("ts", "uid", "id.orig_h", "id.orig_p", "id.resp_h", "id.resp_p", "mac_orig", "method", "uri", "version", "request_body_len", "proxied", "orig_mime_types", "header_values")]; #filter$interv = 6 hr; Log::add_filter(HTTP::LOG, filter); Log::remove_filter(HTTP::LOG, "default"); Log::disable_stream(Conn::LOG); Log::disable_stream(Files::LOG); } event http_all_headers (c: connection, is_orig: bool, hlist: mime_header_list) { if (c?$http && is_orig ==T) { local header_set : set[string] = set(); print hlist; for (header in hlist) { print header; local concatenate : string; concatenate = hlist[header]$name + "||" + hlist[header]$value; add header_set[concatenate]; } c$http$header_values = header_set; } if (c$orig?$l2_addr) { c$http$mac_orig = c$orig$l2_addr; } } From johanna at icir.org Fri Oct 27 08:15:47 2017 From: johanna at icir.org (Johanna Amann) Date: Fri, 27 Oct 2017 08:15:47 -0700 Subject: [Bro] How information is stored in a set() and table() in bro In-Reply-To: <14f2b5e8-2ef4-14b8-8713-4954326da433@utwente.nl> References: <14f2b5e8-2ef4-14b8-8713-4954326da433@utwente.nl> Message-ID: <67FB3A4D-9307-4B6E-BF5A-3C6740F0CB29@icir.org> Hi Riccardo, > I would like to re-use this dataset I have collected, however this > time > I would require the headers to be in the exact order as they are > parsed. > This information is not in logs, so I was wondering whether it is > possible: given the script and the logs, to "reverse" the original > order > of the headers. > > For this purpose, I think I need to know:? 1) how data is stored in a > "set [string]", because that's what I use to temporarily store the > values; and 2) how data is stored in "hlist: mime_header_list" which > is > a table() in Bro. Sadly the order in sets is random (well, it depends on the random seed that was used at the time that you ran Bro to capture the data). As far as I am aware it is not possible to reverse this process; so as painful as it might be, you will have to re-capture the data. Johanna From r.bortolameotti at utwente.nl Fri Oct 27 08:36:23 2017 From: r.bortolameotti at utwente.nl (BortolameottiR) Date: Fri, 27 Oct 2017 17:36:23 +0200 Subject: [Bro] How information is stored in a set() and table() in bro In-Reply-To: <67FB3A4D-9307-4B6E-BF5A-3C6740F0CB29@icir.org> References: <14f2b5e8-2ef4-14b8-8713-4954326da433@utwente.nl> <67FB3A4D-9307-4B6E-BF5A-3C6740F0CB29@icir.org> Message-ID: <089b86ac-53e7-4439-433e-9b5475238b8c@utwente.nl> Hi Johanna, Thanks for your responsive reply, although I was hoping for a different answer :P . I will re-start the process again then. Thanks again! Riccardo On 10/27/2017 05:15 PM, Johanna Amann wrote: > Sadly the order in sets is random (well, it depends on the random seed > that was used at the time that you ran Bro to capture the data). As > far as I am aware it is not possible to reverse this process; so as > painful as it might be, you will have to re-capture the data. From asharma at lbl.gov Fri Oct 27 11:16:58 2017 From: asharma at lbl.gov (Aashish Sharma) Date: Fri, 27 Oct 2017 11:16:58 -0700 Subject: [Bro] How information is stored in a set() and table() in bro In-Reply-To: <089b86ac-53e7-4439-433e-9b5475238b8c@utwente.nl> References: <14f2b5e8-2ef4-14b8-8713-4954326da433@utwente.nl> <67FB3A4D-9307-4B6E-BF5A-3C6740F0CB29@icir.org> <089b86ac-53e7-4439-433e-9b5475238b8c@utwente.nl> Message-ID: May be try using vectors - they preserve the order ! https://www.bro.org/sphinx/script-reference/types.html#type-vector On Fri, Oct 27, 2017 at 8:36 AM, BortolameottiR wrote: > Hi Johanna, > > Thanks for your responsive reply, although I was hoping for a different > answer :P . I will re-start the process again then. > > Thanks again! > > Riccardo > > > On 10/27/2017 05:15 PM, Johanna Amann wrote: >> Sadly the order in sets is random (well, it depends on the random seed >> that was used at the time that you ran Bro to capture the data). As >> far as I am aware it is not possible to reverse this process; so as >> painful as it might be, you will have to re-capture the data. > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From bill.de.ping at gmail.com Sun Oct 29 07:20:52 2017 From: bill.de.ping at gmail.com (william de ping) Date: Sun, 29 Oct 2017 16:20:52 +0200 Subject: [Bro] - large double are not printed/written correctly Message-ID: Hi, I have this simple script : event bro_init() { local a=-3.019159e-8; print "a",a; local s=fmt("%e",a); print "s",s; } results : a, -0 s, -3.019159e-08 the printing of variable a is important because its -0 once written in a log file. is there anyway of printing\writing large doubles ? Thanks B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171029/e638fc59/attachment.html From lagoon7 at gmail.com Sun Oct 29 15:01:49 2017 From: lagoon7 at gmail.com (Ludwig Goon) Date: Sun, 29 Oct 2017 18:01:49 -0400 Subject: [Bro] expire-certs.bro can I get the expiry date too? Message-ID: Is there a way to also print in the notice.log the actual date the cert expires? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171029/6c1e81a3/attachment.html From vikrambasu059 at gmail.com Mon Oct 30 05:22:02 2017 From: vikrambasu059 at gmail.com (Vikram Basu) Date: Mon, 30 Oct 2017 17:52:02 +0530 Subject: [Bro] SMB copied files not showing in files.log Message-ID: <59f7196b.4955650a.ebea6.ec93@mx.google.com> Hi, So I am using the SMB plugin for Bro by loading in local.bro but it seems to be very inconsistent. Often times when I am copying files between two windows machines over the domain there is no corresponding file in the files.log. The smb_files.log itself seems to filled up with a lot of .ini files as well and they all seem to have the ?SMB::FILE_OPEN? action even when I haven?t opened any of them. I thought I would use files showing source as SMB in files.log to differentiate when files are actually copied over the network but often times Bro does not detect the same. Is there any particular way I need to share the files in windows to get the copied files to show up consistently in bro? Regards Vikram Basu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171030/8338d3a7/attachment.html From seth at corelight.com Mon Oct 30 07:41:40 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 30 Oct 2017 10:41:40 -0400 Subject: [Bro] expire-certs.bro can I get the expiry date too? In-Reply-To: References: Message-ID: <6680510D-E589-4B4B-8B28-6D3648CB6541@corelight.com> On 29 Oct 2017, at 18:01, Ludwig Goon wrote: > Is there a way to also print in the notice.log the actual date the > cert expires? If you're talking about the notice from the policy/protocols/ssl/expiring-certs.bro then the date should already be in there. For the three notices that script defines, you should get these messages... - fmt("Certificate %s isn't valid until %T", cert$subject, cert$not_valid_before) - fmt("Certificate %s expired at %T", cert$subject, cert$not_valid_after), - fmt("Certificate %s is going to expire at %T", cert$subject, cert$not_valid_after), .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From seth at corelight.com Mon Oct 30 07:47:54 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 30 Oct 2017 10:47:54 -0400 Subject: [Bro] - large double are not printed/written correctly In-Reply-To: References: Message-ID: If you are talking about logging through the logging framework, then that's a different code path for value serialization (I think we try and avoid scientific notation there). Generally just allowing Bro to serialize values how ever it feels like it as you are doing with the bare unformatted print statement can occasionally result in some undesirable behavior. Are you talking about having this trouble with the logging framework? .Seth On 29 Oct 2017, at 10:20, william de ping wrote: > Hi, > > I have this simple script : > > event bro_init() > { > local a=-3.019159e-8; > print "a",a; > local s=fmt("%e",a); > print "s",s; > } > > results : > a, -0 > s, -3.019159e-08 > > the printing of variable a is important because its -0 once written in > a > log file. > > is there anyway of printing\writing large doubles ? > > Thanks > B > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * www.corelight.com From seth at corelight.com Mon Oct 30 07:51:45 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 30 Oct 2017 10:51:45 -0400 Subject: [Bro] SMB copied files not showing in files.log In-Reply-To: <59f7196b.4955650a.ebea6.ec93@mx.google.com> References: <59f7196b.4955650a.ebea6.ec93@mx.google.com> Message-ID: SMB is a complicated protocol. Windows systems will frequently call open on remote files but not actually transfer any of the bytes of the file. I think there may be several scenarios where they do that and I may not understand them all completely yet unfortunately. Generally if some bytes of a file are transferred over SMB, that file will show up in files.log since files.log is meant to represent the actual transfer of files. The confusion arising from the smb_cmds.log file (where you saw the SMB::FILE_OPEN command) is one of the many reasons that that log is disabled by default too. Are you experiencing a case where you know that a file was actually transferred over SMB but you didn't see a corresponding entry in files.log? If that's true, then I would really appreciate a pcap of the problem! I would really like to know about any cases where that isn't working correctly. Thanks, .Seth On 30 Oct 2017, at 8:22, Vikram Basu wrote: > Hi, > > So I am using the SMB plugin for Bro by loading in local.bro but it > seems to be very inconsistent. > Often times when I am copying files between two windows machines over > the domain there is no corresponding file in the files.log. > The smb_files.log itself seems to filled up with a lot of .ini files > as well and they all seem to have the ?SMB::FILE_OPEN? action even > when I haven?t opened any of them. > I thought I would use files showing source as SMB in files.log to > differentiate when files are actually copied over the network but > often times Bro does not detect the same. > Is there any particular way I need to share the files in windows to > get the copied files to show up consistently in bro? > > Regards > > Vikram Basu > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * www.corelight.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171030/4cbb24d1/attachment.html From jlamps at sandia.gov Mon Oct 30 10:20:30 2017 From: jlamps at sandia.gov (Lamps, Jereme) Date: Mon, 30 Oct 2017 17:20:30 +0000 Subject: [Bro] &write_expire functionality Message-ID: If I have: global my_table: table[string] of vector of HTTP::Info &write_expire = 5mins; Will the timeout reset for a particular string entry if I modify a single element within one of the HTTP::Info records? Best, Jereme -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171030/f6e79ab9/attachment-0001.html From seth at corelight.com Mon Oct 30 10:38:41 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 30 Oct 2017 13:38:41 -0400 Subject: [Bro] &write_expire functionality In-Reply-To: References: Message-ID: Unfortunately I don't think it will. Mutable types break some assumptions in Bro and this is one of them. I will do some testing this afternoon to verify if that's actually true though. .Seth On 30 Oct 2017, at 13:20, Lamps, Jereme wrote: > If I have: > global my_table: table[string] of vector of HTTP::Info &write_expire = > 5mins; > Will the timeout reset for a particular string entry if I modify a > single element within one of the HTTP::Info records? > > Best, > > Jereme > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * www.corelight.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171030/a4661c68/attachment.html From lagoon7 at gmail.com Mon Oct 30 16:32:46 2017 From: lagoon7 at gmail.com (Ludwig Goon) Date: Mon, 30 Oct 2017 23:32:46 +0000 Subject: [Bro] expire-certs.bro can I get the expiry date too? In-Reply-To: <6680510D-E589-4B4B-8B28-6D3648CB6541@corelight.com> References: <6680510D-E589-4B4B-8B28-6D3648CB6541@corelight.com> Message-ID: Does that only apply to the variable number of days before expiry? So for instance if it set to 30 days all of those will fire within the 30 day window. Whereas everything else outside of the window will not fire. So if we want every cert we detect to fire should we set it to 0 or to like to 3650 days? I may have answered my own question but still wanna get your reponse. On Mon, Oct 30, 2017 at 10:41 Seth Hall wrote: > > > On 29 Oct 2017, at 18:01, Ludwig Goon wrote: > > > Is there a way to also print in the notice.log the actual date the > > cert expires? > > If you're talking about the notice from the > policy/protocols/ssl/expiring-certs.bro then the date should already be > in there. For the three notices that script defines, you should get > these messages... > > - fmt("Certificate %s isn't valid until %T", cert$subject, > cert$not_valid_before) > - fmt("Certificate %s expired at %T", cert$subject, > cert$not_valid_after), > - fmt("Certificate %s is going to expire at %T", cert$subject, > cert$not_valid_after), > > .Seth > > -- > Seth Hall * Corelight, Inc * www.corelight.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171030/a69bbc2b/attachment.html From seth at corelight.com Mon Oct 30 17:15:50 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 30 Oct 2017 20:15:50 -0400 Subject: [Bro] expire-certs.bro can I get the expiry date too? In-Reply-To: References: <6680510D-E589-4B4B-8B28-6D3648CB6541@corelight.com> Message-ID: <1C181D78-27E7-45FE-A0C4-9EEDA994F5BD@corelight.com> Oh, if you're just looking for when all certificates expire it sounds like you want the "not_valid_before" and "not_valid_after" timestamps in the x509 log. Is that what you wanted? .Seth On 30 Oct 2017, at 19:32, Ludwig Goon wrote: > Does that only apply to the variable number of days before expiry? So > for > instance if it set to 30 days all of those will fire within the 30 day > window. Whereas everything else outside of the window will not fire. > So if > we want every cert we detect to fire should we set it to 0 or to like > to > 3650 days? I may have answered my own question but still wanna get > your > reponse. > > On Mon, Oct 30, 2017 at 10:41 Seth Hall wrote: > >> >> >> On 29 Oct 2017, at 18:01, Ludwig Goon wrote: >> >>> Is there a way to also print in the notice.log the actual date the >>> cert expires? >> >> If you're talking about the notice from the >> policy/protocols/ssl/expiring-certs.bro then the date should already >> be >> in there. For the three notices that script defines, you should get >> these messages... >> >> - fmt("Certificate %s isn't valid until %T", cert$subject, >> cert$not_valid_before) >> - fmt("Certificate %s expired at %T", cert$subject, >> cert$not_valid_after), >> - fmt("Certificate %s is going to expire at %T", cert$subject, >> cert$not_valid_after), >> >> .Seth >> >> -- >> Seth Hall * Corelight, Inc * www.corelight.com >> -- Seth Hall * Corelight, Inc * www.corelight.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171030/7ae7feb6/attachment.html From seth at corelight.com Mon Oct 30 17:22:06 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 30 Oct 2017 20:22:06 -0400 Subject: [Bro] &write_expire functionality In-Reply-To: References: Message-ID: Whoops, I replied off list. For everyone else's sake, here's what I came to based on digging around a bit... It doesn't work that way. You may want to change the record and then save the record back to the table. The snippet would look like this ```bro local x = my_table[index_value]; x$fieldname += 1; my_table[index_value] = x; ``` That should make the timeouts work right. :) .Seth On 30 Oct 2017, at 13:38, Seth Hall wrote: > Unfortunately I don't think it will. Mutable types break some > assumptions in Bro and this is one of them. I will do some testing > this afternoon to verify if that's actually true though. > > .Seth > > On 30 Oct 2017, at 13:20, Lamps, Jereme wrote: > >> If I have: >> global my_table: table[string] of vector of HTTP::Info &write_expire >> = 5mins; >> Will the timeout reset for a particular string entry if I modify a >> single element within one of the HTTP::Info records? >> >> Best, >> >> Jereme > > >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- > Seth Hall * Corelight, Inc * www.corelight.com -- Seth Hall * Corelight, Inc * www.corelight.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171030/d41c3255/attachment.html From newfire.bw at gmail.com Tue Oct 31 02:28:34 2017 From: newfire.bw at gmail.com (Bowen Li) Date: Tue, 31 Oct 2017 17:28:34 +0800 Subject: [Bro] bro 2.5.2 not compatible with PF_RING 7.0.0 Message-ID: Hi all, Recently, I use bro-2.5.2 with PF_RING-7.0.0, but there is a problem when I do bro source code configure, it seems that bro cannot detect new version of PF_RING, here is the log: Looking for pcap_get_pfring_id - not found. I wonder if it is caused by the FindPCAP.cmake in cmake? What should I do to solve this problem? Any insight would be helpful. Bowen Li -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171031/7c1d5404/attachment.html From seth at corelight.com Tue Oct 31 05:03:02 2017 From: seth at corelight.com (Seth Hall) Date: Tue, 31 Oct 2017 08:03:02 -0400 Subject: [Bro] bro 2.5.2 not compatible with PF_RING 7.0.0 In-Reply-To: References: Message-ID: <784C9781-7736-4BF0-BA6B-1F8D01E292AE@corelight.com> On 31 Oct 2017, at 5:28, Bowen Li wrote: > Recently, I use bro-2.5.2 with PF_RING-7.0.0, but there is a > problem > when I do bro source code configure, it seems that bro cannot detect > new > version of PF_RING, here is the log: > > Looking for pcap_get_pfring_id - not found. I just checked their source code and it appears that that function still exists in their libpcap wrapper. I'm not sure that the correct libpcap is being checked for pf_ring support. What was your full configure command and where is pf_ring 7.0 installed? .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From jlamps at sandia.gov Tue Oct 31 07:09:12 2017 From: jlamps at sandia.gov (Lamps, Jereme) Date: Tue, 31 Oct 2017 14:09:12 +0000 Subject: [Bro] &expire_func functionality/verification with pcap Message-ID: <6B9CB7BB-2FE6-4446-B3B2-EBF181AE73FE@sandia.gov> Building a little off my previous question, I have a structure my_table defined: global my_table: table[string] of vector of HTTP::Info &write_expire = 30secs &expire_func=process; and my_table will get written to in the connection_state_remove event, which should then call the expire_func 30s later. I have tried triggering the functionality two ways: * Having Bro read in a 1GB test.pcap, waiting for minutes (with exit_only_after_terminate=T), then CTRL-C to exit * Having Bro listen on a dummy interface and tcpthrow the test.pcap against it, waiting for minutes then CTRL-C to exit It seems to work for a subset of the connections but not all of them. My hunch is that Bro?s connection state table has no strict time-based removal process, so the connection_state_remove event will not be triggered unless I throw more data at it. My second thought is that it does get triggered at the end for the CTRL-C, but then shuts down before the expire_func fires 30secs later. If my hunches correct please let me know, as then it should theoretically work with Bro on the wire as new data comes in. But for testing purposes, is there any way to either force flushes of the connection table or ensure that Bro waits long enough after the CTRL-C to handle the expire_func? Best, Jereme Lamps -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171031/e7851c8c/attachment.html From seth at corelight.com Tue Oct 31 07:42:35 2017 From: seth at corelight.com (Seth Hall) Date: Tue, 31 Oct 2017 10:42:35 -0400 Subject: [Bro] &expire_func functionality/verification with pcap In-Reply-To: <6B9CB7BB-2FE6-4446-B3B2-EBF181AE73FE@sandia.gov> References: <6B9CB7BB-2FE6-4446-B3B2-EBF181AE73FE@sandia.gov> Message-ID: On 31 Oct 2017, at 10:09, Lamps, Jereme wrote: > * Having Bro read in a 1GB test.pcap, waiting for minutes (with > exit_only_after_terminate=T), then CTRL-C to exit > * Having Bro listen on a dummy interface and tcpthrow the > test.pcap against it, waiting for minutes then CTRL-C to exit > > It seems to work for a subset of the connections but not all of them. > My hunch is that Bro?s connection state table has no strict > time-based removal process, so the connection_state_remove event will > not be triggered unless I throw more data at it I believe that's correct. The combination of the exit_only_after_terminate setting and reading a pcap is not particularly well supported because it's not needed for most circumstances. It's also conceptually hard to pull off cleanly because Bro's packet clock is driven by incoming packet timestamps. The reason you aren't seeing those connections expire is because as far as Bro is concerned time has stopped the moment that packets stop coming in. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From jlamps at sandia.gov Tue Oct 31 08:16:26 2017 From: jlamps at sandia.gov (Lamps, Jereme) Date: Tue, 31 Oct 2017 15:16:26 +0000 Subject: [Bro] [EXTERNAL] Re: &expire_func functionality/verification with pcap In-Reply-To: References: <6B9CB7BB-2FE6-4446-B3B2-EBF181AE73FE@sandia.gov> Message-ID: <3EF36010-9D08-431A-929E-48FDB63A2E8B@sandia.gov> Pulling the thread a little more: I waited a few minutes after tcpthrowing the test.pcap and then tcpthrew a small pcap consisting of three pings. This was unfortunately not enough to trigger the connection_state_remove event for any of the other flows. I also tried creating a pcap that consisted of test.pcap followed by 3 pings 4 minutes later and running that through Bro with no luck. Do you think there is something additional going on underneath the hood? In the conn.log I can verify the ?ts? jumped ~4minutes between the last flow from the first pcap and the starting flow the second pcap. Best, Jereme On 10/31/17, 10:42 AM, "Seth Hall" wrote: On 31 Oct 2017, at 10:09, Lamps, Jereme wrote: > * Having Bro read in a 1GB test.pcap, waiting for minutes (with > exit_only_after_terminate=T), then CTRL-C to exit > * Having Bro listen on a dummy interface and tcpthrow the > test.pcap against it, waiting for minutes then CTRL-C to exit > > It seems to work for a subset of the connections but not all of them. > My hunch is that Bro?s connection state table has no strict > time-based removal process, so the connection_state_remove event will > not be triggered unless I throw more data at it I believe that's correct. The combination of the exit_only_after_terminate setting and reading a pcap is not particularly well supported because it's not needed for most circumstances. It's also conceptually hard to pull off cleanly because Bro's packet clock is driven by incoming packet timestamps. The reason you aren't seeing those connections expire is because as far as Bro is concerned time has stopped the moment that packets stop coming in. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From bill.de.ping at gmail.com Tue Oct 31 09:47:18 2017 From: bill.de.ping at gmail.com (william de ping) Date: Tue, 31 Oct 2017 18:47:18 +0200 Subject: [Bro] - input table of regex patterns OR convert string to regex pattern Message-ID: Hi all, I am trying to read a csv file that has regex patterns in it. it seems that bro does not like reading a column into a regex type. anyway to accomplish that ? is there any function that converts string to regex ? Thanks a lot, B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171031/d94762f2/attachment.html From seth at corelight.com Tue Oct 31 12:01:04 2017 From: seth at corelight.com (Seth Hall) Date: Tue, 31 Oct 2017 15:01:04 -0400 Subject: [Bro] - input table of regex patterns OR convert string to regex pattern In-Reply-To: References: Message-ID: <2BADA848-1C3B-4C86-A3F6-6F3E86449163@corelight.com> On 31 Oct 2017, at 12:47, william de ping wrote: > anyway to accomplish that ? is there any function that converts string > to regex ? It's not possible to create patterns at runtime right now, sorry. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com