From haoscs at gmail.com Mon Jan 1 16:57:56 2018 From: haoscs at gmail.com (Shuai Hao) Date: Mon, 1 Jan 2018 19:57:56 -0500 Subject: [Bro] Calling external functions in binpac protocol parser In-Reply-To: <20171228203815.7iwbvz4z36a7xczn@Beezling.fritz.box> References: <20171228203815.7iwbvz4z36a7xczn@Beezling.fritz.box> Message-ID: Thanks for your detailed reply, Johanna! Ok, I will give a slightly long answer to this. First - assuming that the > test function is just a c++ function, chances are that you want to develop > it outside of the binpac files - Yes. We are developing a module written by C++ that analyzes the traffic pattern. you can e.g. move it into a separate > class that is accessible by everyone. Depending on the things that your > function does, it even might be possible to make it a static function. > Yes, as you mentioned, we do want a module that can be accessed and called by every protocol analyzer. Regarding the "separate class that is accessible by everyone", where we should implement this module? Do we need put the C++ files in bro/src? > The second answer is - since binpac files are only compiles to c++ you > have to do the same thing that the other protocol analyzers do - include > the headers using #include statements. I think you can just use relative > paths from where the files are located, in addition to absolute paths from > the bro base. So - doing something like #include "../your.h" might work > fine. Also note that you probably should not put your plugins into > bro-aux/plugin-support in the first case. Having them in a separate > directory is probably preferable - completely outside of the Bro source > tree. > Sorry I don't get the point of this part. In my understanding, wherever we develop the plugins, it will be complied into bro's library /bro/lib/.../plugin. What do you mean (and how) "outside of the Bro source tree"? Thanks a lot! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180101/677aea0b/attachment.html From johanna at icir.org Tue Jan 2 00:08:47 2018 From: johanna at icir.org (Johanna Amann) Date: Tue, 2 Jan 2018 09:08:47 +0100 Subject: [Bro] Bro 2.5.2 & 2.4.2 release (security update) In-Reply-To: <20171016221119.egzusbmjzllc4x5t@user190.sys.ICSI.Berkeley.EDU> References: <20171016221119.egzusbmjzllc4x5t@user190.sys.ICSI.Berkeley.EDU> Message-ID: <20180102080837.jzq2xuigq4h4psv4@Beezling.fritz.box> > This is a security release that fixes an out-of-bound write in the ContentLine > analyzer. This issue can be used by remote attackers to crash Bro (i.e. a DoS > attack). There also is a possibility this can be exploited in other ways. This has been assigned CVE-2017-1000458. Johanna From promero at cenic.org Tue Jan 2 11:39:41 2018 From: promero at cenic.org (Philip Romero) Date: Tue, 2 Jan 2018 11:39:41 -0800 Subject: [Bro] Triplicate Entries in CONN Log Message-ID: <989c6994-ca58-2c52-79fb-0373df9ee492@cenic.org> All, I did a quick search, but did not see any threads on this type of subject, so forgive me if this has already been discussed. We have a new bro server being stood up that looks to be creating multiple (3) entries for every conn log. Below is a sample of what I'm speaking of. We have 4 monitoring interfaces with varying numbers of CPU cores assigned to the 4 workers they are associated with. The number of entries appears to be related to the number pf_ring workers created because I changed the nodes from 3 lb_procs each to the below node.cfg config this morning and I am now seeing 1 to 5 entries for each log entry.? Would this be an indication that there is a problem with our pf_ring setup? How might we confirm what may be causing this? $ zcat /usr/local/logs/2018-01-01/conn.01\:00\:00-02\:00\:00.log.gz |bro-cut -d |grep '101.124.88.146' 2018-01-01T01:16:50-0800??? CxcveSg78Iow3jix??? 101.124.88.146??? 33589??? 137.164.29.67??? 53??? udp??? dns??? 0.000383??? 44??? 162??? SF??? F??? T??? 0??? Dd??? 72??? 1??? 190??? (empty) 2018-01-01T01:16:50-0800??? CbvYq642taiDEEPLwc??? 101.124.88.146??? 33589??? 137.164.29.67??? 53??? udp??? dns??? 0.000383??? 44??? 162??? SF??? F??? T??? 0??? Dd??? 72??? 1??? 190??? (empty) 2018-01-01T01:16:50-0800??? CEc8UA2AyEKYDSNiw9??? 101.124.88.146??? 33589??? 137.164.29.67??? 53??? udp??? dns??? 0.000383??? 44??? 162??? SF??? F??? T??? 0??? Dd??? 72??? 1??? 190??? (empty) 2018-01-01T01:17:44-0800??? Clg6KI2hLQBShRUxal??? 101.124.88.146??? 54683??? 137.145.204.10??? 53??? udp??? dns??? -??? -??? -??? S0??? F??? F??? 0??? D??? 1??? 72??? 0??? 0??? (empty) 2018-01-01T01:17:44-0800??? CN1WXh1ae3r9FzXA7d??? 101.124.88.146??? 54683??? 137.145.204.10??? 53??? udp??? dns??? -??? -??? -??? S0??? F??? F??? 0??? D??? 1??? 72??? 0??? 0??? (empty) 2018-01-01T01:17:44-0800??? C7s9Qp4LPByzC6Gm53??? 101.124.88.146??? 16779??? 137.145.204.10??? 53??? udp??? dns??? -??? -??? -??? S0??? F??? F??? 0??? D??? 1??? 82??? 0??? 0??? (empty) 2018-01-01T01:17:44-0800??? CHBMNSB9eOhekrAS6??? 101.124.88.146??? 54683??? 137.145.204.10??? 53??? udp??? dns??? -??? -??? -??? S0??? F??? F??? 0??? D??? 1??? 72??? 0??? 0??? (empty) 2018-01-01T01:17:44-0800??? Cmnrmh2ivDksWcJXLl??? 101.124.88.146??? 16779??? 137.145.204.10??? 53??? udp??? dns??? -??? -??? -??? S0??? F??? F??? 0??? D??? 1??? 82??? 0??? 0??? (empty) 2018-01-01T01:17:44-0800??? CTCK4qkRHhDz4jLqk??? 101.124.88.146??? 16779??? 137.145.204.10??? 53??? udp??? dns??? -??? -??? -??? S0??? F??? F??? 0??? D??? 1??? 82??? 0??? 0??? (empty) 2018-01-01T01:17:45-0800??? CZ90QS3RGjHqck8zvc??? 101.124.88.146??? 26774??? 137.145.204.10??? 53??? udp??? dns??? -??? -??? -??? S0??? F??? F??? 0??? D??? 1??? 82??? 0??? 0??? (empty) 2018-01-01T01:17:45-0800??? C96L6a1YTuUCsvbHRk??? 101.124.88.146??? 26774??? 137.145.204.10??? 53??? udp??? dns??? -??? -??? -??? S0??? F??? F??? 0??? D??? 1??? 82??? 0??? 0??? (empty) 2018-01-01T01:17:45-0800??? CnP2AnwrmyniFfDBe??? 101.124.88.146??? 26774??? 137.145.204.10??? 53??? udp??? dns??? -??? -??? -??? S0??? F??? F??? 0??? D??? 1??? 82??? 0??? 0??? (empty) 2018-01-01T01:17:46-0800??? CB5iF7f4hoqrQWiq2??? 101.124.88.146??? 25389??? 137.145.204.10??? 53??? udp??? dns??? -??? -??? -??? S0??? F??? F??? 0??? D??? 1??? 72??? 0??? 0??? (empty) 2018-01-01T01:17:46-0800??? CkZEAp0saM0cEaTSf??? 101.124.88.146??? 25389??? 137.145.204.10??? 53??? udp??? dns??? -??? -??? -??? S0??? F??? F??? 0??? D??? 1??? 72??? 0??? 0??? (empty) 2018-01-01T01:17:46-0800??? CQx4sVzjmGlPnAa51??? 101.124.88.146??? 25389??? 137.145.204.10??? 53??? udp??? dns??? -??? -??? -??? S0??? F??? F??? 0??? D??? 1??? 72??? 0??? 0??? (empty) $ cat /usr/local/etc/node.cfg # Example BroControl node configuration. # # This example has a standalone node ready to go except for possibly changing # the sniffing interface. # This is a complete standalone configuration.? Most likely you will # only need to change the interface. #[bro] #type=standalone #host=localhost #interface=ens2f0 ## Below is an example clustered configuration. If you use this, ## remove the [bro] node above. #[logger] #type=logger #host=localhost # [manager] type=manager host=localhost # [proxy-1] type=proxy host=localhost # [worker-1] lb_method=pf_ring lb_procs=1 #pin_cpus=2,3 type=worker host=localhost interface=ens2f0 # [worker-2] lb_method=pf_ring lb_procs=2 #pin_cpus=4,5 type=worker host=localhost interface=ens2f1 # [worker-3] lb_method=pf_ring lb_procs=4 #pin_cpus=2,3 type=worker host=localhost interface=ens2f2 # [worker-4] lb_method=pf_ring lb_procs=5 #pin_cpus=4,5 type=worker host=localhost interface=eno2 -- Philip Romero, CISSP, CISA Sr. Information Security Analyst CENIC promero at cenic.org Phone: (714) 220-3430 Mobile: (562) 237-9290 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180102/0cdea4e1/attachment.html From jazoff at illinois.edu Tue Jan 2 11:52:53 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 2 Jan 2018 19:52:53 +0000 Subject: [Bro] Triplicate Entries in CONN Log In-Reply-To: <989c6994-ca58-2c52-79fb-0373df9ee492@cenic.org> References: <989c6994-ca58-2c52-79fb-0373df9ee492@cenic.org> Message-ID: > On Jan 2, 2018, at 2:39 PM, Philip Romero wrote: > > All, > > I did a quick search, but did not see any threads on this type of subject, so forgive me if this has already been discussed. We have a new bro server being stood up that looks to be creating multiple (3) entries for every conn log. Below is a sample of what I'm speaking of. We have 4 monitoring interfaces with varying numbers of CPU cores assigned to the 4 workers they are associated with. The number of entries appears to be related to the number pf_ring workers created because I changed the nodes from 3 lb_procs each to the below node.cfg config this morning and I am now seeing 1 to 5 entries for each log entry. > Would this be an indication that there is a problem with our pf_ring setup? How might we confirm what may be causing this? You're probably not really using pf_ring. Bro-doctor was written to troubleshoot problems like this.. bro-pkg install bro-doctor broctl doctor.bro You're either not linked against pf_ring properly, or possible you installed bro and then pf_ring, in which case a (just fixed) bug in broctl will disable pf_ring and you need to add pfringclusterid = 11 to your broctl.cfg You should also look into use the native bro pf_ring plugin, which is a little harder to misconfigure. ? Justin Azoff From promero at cenic.org Tue Jan 2 13:42:00 2018 From: promero at cenic.org (Philip Romero) Date: Tue, 2 Jan 2018 13:42:00 -0800 Subject: [Bro] Triplicate Entries in CONN Log In-Reply-To: References: <989c6994-ca58-2c52-79fb-0373df9ee492@cenic.org> Message-ID: Justin, Thanks for the quick response. Our Systems team assures me the pf_ring is compiled correctly and provided the below output. We have some ACL's in place that makes it difficult to load the bro-doctor pkg you mention easily, but will work towards getting that tool in place. In the meantime, is there anything about the below output that looks out of place or missing?? We'll also be setting the pfringclusterid in the broctl.cfg to see if that fixes the issue. # ldd /usr/local/bin/bro | grep pcap libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x00007f5473516000) # strings /usr/local/lib/libpcap.so.1 | grep pfring | tail -n3 pfring_mod_open pfring_mod_get_bound_device_address pfring_hw_ft_remove_hw_rule Philip -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180102/68df8f41/attachment.html From seth at corelight.com Wed Jan 3 08:58:56 2018 From: seth at corelight.com (Seth Hall) Date: Wed, 03 Jan 2018 11:58:56 -0500 Subject: [Bro] Triplicate Entries in CONN Log In-Reply-To: <989c6994-ca58-2c52-79fb-0373df9ee492@cenic.org> References: <989c6994-ca58-2c52-79fb-0373df9ee492@cenic.org> Message-ID: <1EB358B9-B537-40EB-BE6D-FF83DB55AC75@corelight.com> Your node.cfg is slightly complicated since you're sniffing a number of interfaces. I wouldn't be surprised if you're accidentally seeing the same traffic on multiple interfaces. Add this little blurb into your local.bro... #### begin #### @load base/protocols/conn export { redef record Conn::Info += { ## The name of the node where this connection was analyzed. node: string &log &optional; }; } event connection_state_remove(c: connection) &priority=2 { c$conn$node = peer_description; } #### end #### That will let you see which node each of your conn log entries was being written by. Look at three of the same connections to see if they're coming from different workers and let us know. .Seth On 2 Jan 2018, at 14:39, Philip Romero wrote: > All, > > I did a quick search, but did not see any threads on this type of > subject, so forgive me if this has already been discussed. We have a > new > bro server being stood up that looks to be creating multiple (3) > entries > for every conn log. Below is a sample of what I'm speaking of. We have > 4 > monitoring interfaces with varying numbers of CPU cores assigned to > the > 4 workers they are associated with. The number of entries appears to > be > related to the number pf_ring workers created because I changed the > nodes from 3 lb_procs each to the below node.cfg config this morning > and > I am now seeing 1 to 5 entries for each log entry.? > > Would this be an indication that there is a problem with our pf_ring > setup? How might we confirm what may be causing this? > > $ zcat /usr/local/logs/2018-01-01/conn.01\:00\:00-02\:00\:00.log.gz > |bro-cut -d |grep '101.124.88.146' > 2018-01-01T01:16:50-0800??? CxcveSg78Iow3jix??? > 101.124.88.146??? > 33589??? 137.164.29.67??? 53??? udp??? dns??? > 0.000383??? 44??? 162??? > SF??? F??? T??? 0??? Dd??? 72??? 1??? 190??? > (empty) > 2018-01-01T01:16:50-0800??? CbvYq642taiDEEPLwc??? > 101.124.88.146??? > 33589??? 137.164.29.67??? 53??? udp??? dns??? > 0.000383??? 44??? 162??? > SF??? F??? T??? 0??? Dd??? 72??? 1??? 190??? > (empty) > 2018-01-01T01:16:50-0800??? CEc8UA2AyEKYDSNiw9??? > 101.124.88.146??? > 33589??? 137.164.29.67??? 53??? udp??? dns??? > 0.000383??? 44??? 162??? > SF??? F??? T??? 0??? Dd??? 72??? 1??? 190??? > (empty) > 2018-01-01T01:17:44-0800??? Clg6KI2hLQBShRUxal??? > 101.124.88.146??? > 54683??? 137.145.204.10??? 53??? udp??? dns??? -??? > -??? -??? S0??? F??? > F??? 0??? D??? 1??? 72??? 0??? 0??? (empty) > 2018-01-01T01:17:44-0800??? CN1WXh1ae3r9FzXA7d??? > 101.124.88.146??? > 54683??? 137.145.204.10??? 53??? udp??? dns??? -??? > -??? -??? S0??? F??? > F??? 0??? D??? 1??? 72??? 0??? 0??? (empty) > 2018-01-01T01:17:44-0800??? C7s9Qp4LPByzC6Gm53??? > 101.124.88.146??? > 16779??? 137.145.204.10??? 53??? udp??? dns??? -??? > -??? -??? S0??? F??? > F??? 0??? D??? 1??? 82??? 0??? 0??? (empty) > 2018-01-01T01:17:44-0800??? CHBMNSB9eOhekrAS6??? > 101.124.88.146??? > 54683??? 137.145.204.10??? 53??? udp??? dns??? -??? > -??? -??? S0??? F??? > F??? 0??? D??? 1??? 72??? 0??? 0??? (empty) > 2018-01-01T01:17:44-0800??? Cmnrmh2ivDksWcJXLl??? > 101.124.88.146??? > 16779??? 137.145.204.10??? 53??? udp??? dns??? -??? > -??? -??? S0??? F??? > F??? 0??? D??? 1??? 82??? 0??? 0??? (empty) > 2018-01-01T01:17:44-0800??? CTCK4qkRHhDz4jLqk??? > 101.124.88.146??? > 16779??? 137.145.204.10??? 53??? udp??? dns??? -??? > -??? -??? S0??? F??? > F??? 0??? D??? 1??? 82??? 0??? 0??? (empty) > 2018-01-01T01:17:45-0800??? CZ90QS3RGjHqck8zvc??? > 101.124.88.146??? > 26774??? 137.145.204.10??? 53??? udp??? dns??? -??? > -??? -??? S0??? F??? > F??? 0??? D??? 1??? 82??? 0??? 0??? (empty) > 2018-01-01T01:17:45-0800??? C96L6a1YTuUCsvbHRk??? > 101.124.88.146??? > 26774??? 137.145.204.10??? 53??? udp??? dns??? -??? > -??? -??? S0??? F??? > F??? 0??? D??? 1??? 82??? 0??? 0??? (empty) > 2018-01-01T01:17:45-0800??? CnP2AnwrmyniFfDBe??? > 101.124.88.146??? > 26774??? 137.145.204.10??? 53??? udp??? dns??? -??? > -??? -??? S0??? F??? > F??? 0??? D??? 1??? 82??? 0??? 0??? (empty) > 2018-01-01T01:17:46-0800??? CB5iF7f4hoqrQWiq2??? > 101.124.88.146??? > 25389??? 137.145.204.10??? 53??? udp??? dns??? -??? > -??? -??? S0??? F??? > F??? 0??? D??? 1??? 72??? 0??? 0??? (empty) > 2018-01-01T01:17:46-0800??? CkZEAp0saM0cEaTSf??? > 101.124.88.146??? > 25389??? 137.145.204.10??? 53??? udp??? dns??? -??? > -??? -??? S0??? F??? > F??? 0??? D??? 1??? 72??? 0??? 0??? (empty) > 2018-01-01T01:17:46-0800??? CQx4sVzjmGlPnAa51??? > 101.124.88.146??? > 25389??? 137.145.204.10??? 53??? udp??? dns??? -??? > -??? -??? S0??? F??? > F??? 0??? D??? 1??? 72??? 0??? 0??? (empty) > > $ cat /usr/local/etc/node.cfg > # Example BroControl node configuration. > # > # This example has a standalone node ready to go except for possibly > changing > # the sniffing interface. > > # This is a complete standalone configuration.? Most likely you will > # only need to change the interface. > > #[bro] > #type=standalone > #host=localhost > #interface=ens2f0 > > ## Below is an example clustered configuration. If you use this, > ## remove the [bro] node above. > > #[logger] > #type=logger > #host=localhost > # > [manager] > type=manager > host=localhost > # > [proxy-1] > type=proxy > host=localhost > # > [worker-1] > lb_method=pf_ring > lb_procs=1 > #pin_cpus=2,3 > type=worker > host=localhost > interface=ens2f0 > # > [worker-2] > lb_method=pf_ring > lb_procs=2 > #pin_cpus=4,5 > type=worker > host=localhost > interface=ens2f1 > # > [worker-3] > lb_method=pf_ring > lb_procs=4 > #pin_cpus=2,3 > type=worker > host=localhost > interface=ens2f2 > # > [worker-4] > lb_method=pf_ring > lb_procs=5 > #pin_cpus=4,5 > type=worker > host=localhost > interface=eno2 > > -- > Philip Romero, CISSP, CISA > Sr. Information Security Analyst > CENIC > promero at cenic.org > Phone: (714) 220-3430 > Mobile: (562) 237-9290 > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * www.corelight.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180103/ca151352/attachment.html From promero at cenic.org Wed Jan 3 09:39:20 2018 From: promero at cenic.org (Philip Romero) Date: Wed, 3 Jan 2018 09:39:20 -0800 Subject: [Bro] Triplicate Entries in CONN Log In-Reply-To: <1EB358B9-B537-40EB-BE6D-FF83DB55AC75@corelight.com> References: <989c6994-ca58-2c52-79fb-0373df9ee492@cenic.org> <1EB358B9-B537-40EB-BE6D-FF83DB55AC75@corelight.com> Message-ID: <0e12f9eb-7348-a198-0dc7-ab14b9bea18b@cenic.org> Seth, Thanks for the troubleshooting code. It looks like only one interface is getting the traffic, but all 4 cores assigned are processing the same traffic individually. I'm still working with my Systems team on the suggestion from Justin. ? $ cat /usr/local/logs/current/conn.log |grep -P '137.164.29.72' |grep -P '173.245.59.100' |grep '49519' 1515000332.481539??? CY3pub5wesleIxzhh??? 137.164.29.72??? 49519??? 173.245.59.100??? 53??? udp??? dns??? 0.001269??? 54??? 54??? SF??? T??? F??? 0??? Dd??? 1??? 82??? 1??? 82??? (empty)??? worker-4-2 1515000332.481539??? CkBMQVujDASjTbiZb??? 137.164.29.72??? 49519??? 173.245.59.100??? 53??? udp??? dns??? 0.001269??? 54??? 54??? SF??? T??? F??? 0??? Dd??? 1??? 82??? 1??? 82??? (empty)??? worker-4-1 1515000332.481539??? CMTY2G1ferRjOZtWz??? 137.164.29.72??? 49519??? 173.245.59.100??? 53??? udp??? dns??? 0.001269??? 54??? 54??? SF??? T??? F??? 0??? Dd??? 1??? 82??? 1??? 82??? (empty)??? worker-4-4 1515000332.481539??? CMj0iL2677DvhMHNue??? 137.164.29.72??? 49519??? 173.245.59.100??? 53??? udp??? dns??? 0.001269??? 54??? 54??? SF??? T??? F??? 0??? Dd??? 1??? 82??? 1??? 82??? (empty)??? worker-4-3 Philip On 1/3/18 8:58 AM, Seth Hall wrote: > > Your node.cfg is slightly complicated since you're sniffing a number > of interfaces. I wouldn't be surprised if you're accidentally seeing > the same traffic on multiple interfaces. Add this little blurb into > your local.bro... > > #### begin #### > @load base/protocols/conn > > export { > redef record Conn::Info += { > ## The name of the node where this connection was analyzed. > node: string &log &optional; > }; > } > > event connection_state_remove(c: connection) &priority=2 > { > c$conn$node = peer_description; > } > #### end #### > > That will let you see which node each of your conn log entries was > being written by. Look at three of the same connections to see if > they're coming from different workers and let us know. > > .Seth > > On 2 Jan 2018, at 14:39, Philip Romero wrote: > > All, > > I did a quick search, but did not see any threads on this type of > subject, so forgive me if this has already been discussed. We have > a new bro server being stood up that looks to be creating multiple > (3) entries for every conn log. Below is a sample of what I'm > speaking of. We have 4 monitoring interfaces with varying numbers > of CPU cores assigned to the 4 workers they are associated with. > The number of entries appears to be related to the number pf_ring > workers created because I changed the nodes from 3 lb_procs each > to the below node.cfg config this morning and I am now seeing 1 to > 5 entries for each log entry.? > > Would this be an indication that there is a problem with our > pf_ring setup? How might we confirm what may be causing this? > > $ zcat > /usr/local/logs/2018-01-01/conn.01\:00\:00-02\:00\:00.log.gz > |bro-cut -d |grep '101.124.88.146' > 2018-01-01T01:16:50-0800??? CxcveSg78Iow3jix??? 101.124.88.146??? > 33589??? 137.164.29.67??? 53??? udp??? dns??? 0.000383??? 44??? > 162??? SF??? F??? T??? 0??? Dd??? 72??? 1??? 190??? (empty) > 2018-01-01T01:16:50-0800??? CbvYq642taiDEEPLwc??? > 101.124.88.146??? 33589??? 137.164.29.67??? 53??? udp??? dns??? > 0.000383??? 44??? 162??? SF??? F??? T??? 0??? Dd??? 72??? 1??? > 190??? (empty) > 2018-01-01T01:16:50-0800??? CEc8UA2AyEKYDSNiw9??? > 101.124.88.146??? 33589??? 137.164.29.67??? 53??? udp??? dns??? > 0.000383??? 44??? 162??? SF??? F??? T??? 0??? Dd??? 72??? 1??? > 190??? (empty) > 2018-01-01T01:17:44-0800??? Clg6KI2hLQBShRUxal??? > 101.124.88.146??? 54683??? 137.145.204.10??? 53??? udp??? dns??? > -??? -??? -??? S0??? F??? F??? 0??? D??? 1??? 72??? 0??? 0??? (empty) > 2018-01-01T01:17:44-0800??? CN1WXh1ae3r9FzXA7d??? > 101.124.88.146??? 54683??? 137.145.204.10??? 53??? udp??? dns??? > -??? -??? -??? S0??? F??? F??? 0??? D??? 1??? 72??? 0??? 0??? (empty) > 2018-01-01T01:17:44-0800??? C7s9Qp4LPByzC6Gm53??? > 101.124.88.146??? 16779??? 137.145.204.10??? 53??? udp??? dns??? > -??? -??? -??? S0??? F??? F??? 0??? D??? 1??? 82??? 0??? 0??? (empty) > 2018-01-01T01:17:44-0800??? CHBMNSB9eOhekrAS6??? 101.124.88.146??? > 54683??? 137.145.204.10??? 53??? udp??? dns??? -??? -??? -??? > S0??? F??? F??? 0??? D??? 1??? 72??? 0??? 0??? (empty) > 2018-01-01T01:17:44-0800??? Cmnrmh2ivDksWcJXLl??? > 101.124.88.146??? 16779??? 137.145.204.10??? 53??? udp??? dns??? > -??? -??? -??? S0??? F??? F??? 0??? D??? 1??? 82??? 0??? 0??? (empty) > 2018-01-01T01:17:44-0800??? CTCK4qkRHhDz4jLqk??? 101.124.88.146??? > 16779??? 137.145.204.10??? 53??? udp??? dns??? -??? -??? -??? > S0??? F??? F??? 0??? D??? 1??? 82??? 0??? 0??? (empty) > 2018-01-01T01:17:45-0800??? CZ90QS3RGjHqck8zvc??? > 101.124.88.146??? 26774??? 137.145.204.10??? 53??? udp??? dns??? > -??? -??? -??? S0??? F??? F??? 0??? D??? 1??? 82??? 0??? 0??? (empty) > 2018-01-01T01:17:45-0800??? C96L6a1YTuUCsvbHRk??? > 101.124.88.146??? 26774??? 137.145.204.10??? 53??? udp??? dns??? > -??? -??? -??? S0??? F??? F??? 0??? D??? 1??? 82??? 0??? 0??? (empty) > 2018-01-01T01:17:45-0800??? CnP2AnwrmyniFfDBe??? 101.124.88.146??? > 26774??? 137.145.204.10??? 53??? udp??? dns??? -??? -??? -??? > S0??? F??? F??? 0??? D??? 1??? 82??? 0??? 0??? (empty) > 2018-01-01T01:17:46-0800??? CB5iF7f4hoqrQWiq2??? 101.124.88.146??? > 25389??? 137.145.204.10??? 53??? udp??? dns??? -??? -??? -??? > S0??? F??? F??? 0??? D??? 1??? 72??? 0??? 0??? (empty) > 2018-01-01T01:17:46-0800??? CkZEAp0saM0cEaTSf??? 101.124.88.146??? > 25389??? 137.145.204.10??? 53??? udp??? dns??? -??? -??? -??? > S0??? F??? F??? 0??? D??? 1??? 72??? 0??? 0??? (empty) > 2018-01-01T01:17:46-0800??? CQx4sVzjmGlPnAa51??? 101.124.88.146??? > 25389??? 137.145.204.10??? 53??? udp??? dns??? -??? -??? -??? > S0??? F??? F??? 0??? D??? 1??? 72??? 0??? 0??? (empty) > > $ cat /usr/local/etc/node.cfg > # Example BroControl node configuration. > # > # This example has a standalone node ready to go except for > possibly changing > # the sniffing interface. > > # This is a complete standalone configuration.? Most likely you will > # only need to change the interface. > > #[bro] > #type=standalone > #host=localhost > #interface=ens2f0 > > ## Below is an example clustered configuration. If you use this, > ## remove the [bro] node above. > > #[logger] > #type=logger > #host=localhost > # > [manager] > type=manager > host=localhost > # > [proxy-1] > type=proxy > host=localhost > # > [worker-1] > lb_method=pf_ring > lb_procs=1 > #pin_cpus=2,3 > type=worker > host=localhost > interface=ens2f0 > # > [worker-2] > lb_method=pf_ring > lb_procs=2 > #pin_cpus=4,5 > type=worker > host=localhost > interface=ens2f1 > # > [worker-3] > lb_method=pf_ring > lb_procs=4 > #pin_cpus=2,3 > type=worker > host=localhost > interface=ens2f2 > # > [worker-4] > lb_method=pf_ring > lb_procs=5 > #pin_cpus=4,5 > type=worker > host=localhost > interface=eno2 > > -- > Philip Romero, CISSP, CISA > Sr. Information Security Analyst > CENIC > promero at cenic.org > Phone: (714) 220-3430 > Mobile: (562) 237-9290 > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- > Seth Hall * Corelight, Inc * www.corelight.com > -- Philip Romero, CISSP, CISA Sr. Information Security Analyst CENIC promero at cenic.org Phone: (714) 220-3430 Mobile: (562) 237-9290 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180103/c25949ce/attachment-0001.html From seth at corelight.com Wed Jan 3 11:57:34 2018 From: seth at corelight.com (Seth Hall) Date: Wed, 03 Jan 2018 14:57:34 -0500 Subject: [Bro] Triplicate Entries in CONN Log In-Reply-To: <0e12f9eb-7348-a198-0dc7-ab14b9bea18b@cenic.org> References: <989c6994-ca58-2c52-79fb-0373df9ee492@cenic.org> <1EB358B9-B537-40EB-BE6D-FF83DB55AC75@corelight.com> <0e12f9eb-7348-a198-0dc7-ab14b9bea18b@cenic.org> Message-ID: On 3 Jan 2018, at 12:39, Philip Romero wrote: > Thanks for the troubleshooting code. It looks like only one interface > is > getting the traffic, but all 4 cores assigned are processing the same > traffic individually. I'm still working with my Systems team on the > suggestion from Justin. ? Could you try removing all of the worker configs from node.cfg except for worker-4? I'm curious if there is something we did that is causing trouble for PF_Ring if multiple interfaces are being sniffed like that. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From promero at cenic.org Wed Jan 3 12:49:31 2018 From: promero at cenic.org (Philip Romero) Date: Wed, 3 Jan 2018 12:49:31 -0800 Subject: [Bro] Triplicate Entries in CONN Log In-Reply-To: References: <989c6994-ca58-2c52-79fb-0373df9ee492@cenic.org> <1EB358B9-B537-40EB-BE6D-FF83DB55AC75@corelight.com> <0e12f9eb-7348-a198-0dc7-ab14b9bea18b@cenic.org> Message-ID: <6153848a-ece4-a841-3c02-55216eb52c2a@cenic.org> Seth and Justin, It looks to be working now. The latest change that was made was dropping the pfring version from 7.0.0 to 6.6.0. That in combination with using the "pfringclusterid = 11" setting in the broctl.cfg got it working correctly. We're no longer seeing any multiple entries for the same activity. Thanks for all the help. Philip On 1/3/18 11:57 AM, Seth Hall wrote: > > On 3 Jan 2018, at 12:39, Philip Romero wrote: > >> Thanks for the troubleshooting code. It looks like only one interface is >> getting the traffic, but all 4 cores assigned are processing the same >> traffic individually. I'm still working with my Systems team on the >> suggestion from Justin. ? > > Could you try removing all of the worker configs from node.cfg except > for worker-4?? I'm curious if there is something we did that is > causing trouble for PF_Ring if multiple interfaces are being sniffed > like that. > > ? .Seth > > > -- > Seth Hall * Corelight, Inc * www.corelight.com -- Philip Romero, CISSP, CISA Sr. Information Security Analyst CENIC promero at cenic.org Phone: (714) 220-3430 Mobile: (562) 237-9290 From moustafa.elbadry at oregonstate.edu Wed Jan 3 13:13:59 2018 From: moustafa.elbadry at oregonstate.edu (ElBadry Shaker, Moustafa) Date: Wed, 3 Jan 2018 21:13:59 +0000 Subject: [Bro] Bro logs from JSON to TSV Message-ID: Greetings, Hope my email finds you well. I was wondering if someone can help me figure out how to transform existing Bro logs from JSON format to TSV format. The TSV format is what Bro uses by default to write log files. Thanks in advance! Sincerely, Moustafa ElBadry, Information Security Analyst, Office of Information Security Oregon State University | Information Services | 541-737-4545 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180103/ee1af76a/attachment.html From jazoff at illinois.edu Wed Jan 3 13:25:01 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 3 Jan 2018 21:25:01 +0000 Subject: [Bro] Bro logs from JSON to TSV In-Reply-To: References: Message-ID: <3925859B-5857-47A0-B529-5DEE2E23DDB4@illinois.edu> Do you want the exact TSV format with the #fields and #types header, or just TSV in general? This is a somewhat strange thing to want to do - since working with the data in JSON format is generally easier.. What exactly are you trying to accomplish after you convert the logs? ? Justin Azoff > On Jan 3, 2018, at 4:13 PM, ElBadry Shaker, Moustafa wrote: > > Greetings, > > Hope my email finds you well. I was wondering if someone can help me figure out how to transform existing Bro logs from JSON format to TSV format. The TSV format is what Bro uses by default to write log files. Thanks in advance! > > Sincerely, > Moustafa ElBadry, Information Security Analyst, Office of Information Security > Oregon State University | Information Services | 541-737-4545 > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From moustafa.elbadry at oregonstate.edu Wed Jan 3 13:38:26 2018 From: moustafa.elbadry at oregonstate.edu (ElBadry Shaker, Moustafa) Date: Wed, 3 Jan 2018 21:38:26 +0000 Subject: [Bro] Bro logs from JSON to TSV In-Reply-To: <3925859B-5857-47A0-B529-5DEE2E23DDB4@illinois.edu> References: <3925859B-5857-47A0-B529-5DEE2E23DDB4@illinois.edu> Message-ID: <55C831FB-481D-4771-8846-905DD20B381A@oregonstate.edu> I want the exact TSV format. We currently have our Bro cluster writing logs in JSON. There are couple of network traffic analytics tools like RITA (Real Intelligence Threat Analytics) and some AWK scripts that we want to use. The problem is that the tools we want to use work only with Bro?s default TSV format. Moustafa On 1/3/18, 1:25 PM, "Azoff, Justin S" wrote: Do you want the exact TSV format with the #fields and #types header, or just TSV in general? This is a somewhat strange thing to want to do - since working with the data in JSON format is generally easier.. What exactly are you trying to accomplish after you convert the logs? ? Justin Azoff > On Jan 3, 2018, at 4:13 PM, ElBadry Shaker, Moustafa wrote: > > Greetings, > > Hope my email finds you well. I was wondering if someone can help me figure out how to transform existing Bro logs from JSON format to TSV format. The TSV format is what Bro uses by default to write log files. Thanks in advance! > > Sincerely, > Moustafa ElBadry, Information Security Analyst, Office of Information Security > Oregon State University | Information Services | 541-737-4545 > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jazoff at illinois.edu Thu Jan 4 12:35:43 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 4 Jan 2018 20:35:43 +0000 Subject: [Bro] Bro logs from JSON to TSV In-Reply-To: <55C831FB-481D-4771-8846-905DD20B381A@oregonstate.edu> References: <3925859B-5857-47A0-B529-5DEE2E23DDB4@illinois.edu> <55C831FB-481D-4771-8846-905DD20B381A@oregonstate.edu> Message-ID: <6A51750F-798D-44E1-8378-0E4364F98A44@illinois.edu> > On Jan 3, 2018, at 4:38 PM, ElBadry Shaker, Moustafa wrote: > > I want the exact TSV format. > > We currently have our Bro cluster writing logs in JSON. There are couple of network traffic analytics tools like RITA (Real Intelligence Threat Analytics) and some AWK scripts that we want to use. The problem is that the tools we want to use work only with Bro?s default TSV format. > > Moustafa Ah, I see now. You have a few of options here. You could just tell bro to write out the logs in both formats at the same time. For older logs there is only a script for bro that can re-log to json, but not the other way, most people have the opposite problem. There is an open issue for RITA to support json: https://github.com/ocmdev/rita/issues/146 A tool to convert the json logs back into the TSV format could be written, but ultimately that would be a waste of time. Better to update RITA to support json instead of writing more tools to work with the tsv format that only bro uses. For awk stuff you can swap out bro-cut for jq or https://github.com/JustinAzoff/json-cut json-cut it doesn't support all the options that bro-cut supports and may be a bit buggy, but it's easier to extract a few fields from a json log as TSV and 2x faster than jq. If I can find a nice, small json library for C we can probably update bro-cut to natively support the json logs. For now, to extract note and msg from a stream of notice logs with bro-cut and json-cut you just do zcat notice.* | bro-cut note msg | awk ... zcat notice.* | json-cut note msg | awk ... For jq you use something like zcat notice.* | jq -r '[.note, .msg]|@tsv' | awk ... If the awk scripts are hardcoding top level field numbers like $3 and $5 instead of using bro-cut... they should not do that :-) ? Justin Azoff From moustafa.elbadry at oregonstate.edu Fri Jan 5 08:34:58 2018 From: moustafa.elbadry at oregonstate.edu (ElBadry Shaker, Moustafa) Date: Fri, 5 Jan 2018 16:34:58 +0000 Subject: [Bro] Bro logs from JSON to TSV In-Reply-To: <6A51750F-798D-44E1-8378-0E4364F98A44@illinois.edu> References: <3925859B-5857-47A0-B529-5DEE2E23DDB4@illinois.edu> <55C831FB-481D-4771-8846-905DD20B381A@oregonstate.edu> <6A51750F-798D-44E1-8378-0E4364F98A44@illinois.edu> Message-ID: <94820346-DD4D-4331-9483-97D055D94D53@oregonstate.edu> Great. Thanks Justin for sharing this. Definitely helps us a lot. Moustafa On 1/4/18, 12:36 PM, "Azoff, Justin S" wrote: > On Jan 3, 2018, at 4:38 PM, ElBadry Shaker, Moustafa wrote: > > I want the exact TSV format. > > We currently have our Bro cluster writing logs in JSON. There are couple of network traffic analytics tools like RITA (Real Intelligence Threat Analytics) and some AWK scripts that we want to use. The problem is that the tools we want to use work only with Bro?s default TSV format. > > Moustafa Ah, I see now. You have a few of options here. You could just tell bro to write out the logs in both formats at the same time. For older logs there is only a script for bro that can re-log to json, but not the other way, most people have the opposite problem. There is an open issue for RITA to support json: https://github.com/ocmdev/rita/issues/146 A tool to convert the json logs back into the TSV format could be written, but ultimately that would be a waste of time. Better to update RITA to support json instead of writing more tools to work with the tsv format that only bro uses. For awk stuff you can swap out bro-cut for jq or https://github.com/JustinAzoff/json-cut json-cut it doesn't support all the options that bro-cut supports and may be a bit buggy, but it's easier to extract a few fields from a json log as TSV and 2x faster than jq. If I can find a nice, small json library for C we can probably update bro-cut to natively support the json logs. For now, to extract note and msg from a stream of notice logs with bro-cut and json-cut you just do zcat notice.* | bro-cut note msg | awk ... zcat notice.* | json-cut note msg | awk ... For jq you use something like zcat notice.* | jq -r '[.note, .msg]|@tsv' | awk ... If the awk scripts are hardcoding top level field numbers like $3 and $5 instead of using bro-cut... they should not do that :-) ? Justin Azoff From tfleury at illinois.edu Fri Jan 5 09:08:29 2018 From: tfleury at illinois.edu (Terry Fleury) Date: Fri, 5 Jan 2018 11:08:29 -0600 Subject: [Bro] Web viewer for Bro Packages Message-ID: * Hello Bro Community, Work has begun on a web-based view of packages available via the Bro Package Manager : https://bro-pkg-test.security.ncsa.illinois.edu/ In its current state, the website shows the packages listed at GitHub and has basic search capability. More features and improved look-and-feel are in the works. Feel free to look around**and leave feedback . Expect changes in the weeks to come! * -- Terry Fleury tfleury at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180105/65a562b6/attachment.html From seth at corelight.com Sun Jan 7 12:56:52 2018 From: seth at corelight.com (Seth Hall) Date: Sun, 7 Jan 2018 15:56:52 -0500 Subject: [Bro] Web viewer for Bro Packages In-Reply-To: References: Message-ID: <859860D4-3BAF-4345-8006-96A25403BB1C@corelight.com> Thanks for posting Terry! I'm really excited to see the continued development of the site! .Seth -- Seth Hall * Corelight, Inc * www.corelight.com > On Jan 5, 2018, at 12:08 PM, Terry Fleury wrote: > > Hello Bro Community, > > Work has begun on a web-based view of packages available via the Bro Package Manager: > > https://bro-pkg-test.security.ncsa.illinois.edu/ > > In its current state, the website shows the packages listed at GitHub and has basic search capability. More features and improved look-and-feel are in the works. > > Feel free to look around and leave feedback. Expect changes in the weeks to come! > -- > Terry Fleury > tfleury at illinois.edu > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180107/6f9b7585/attachment.html From seth at corelight.com Mon Jan 8 08:59:38 2018 From: seth at corelight.com (Seth Hall) Date: Mon, 08 Jan 2018 11:59:38 -0500 Subject: [Bro] Bay Area Bro Meetup Message-ID: Sorry for the late notice, but I wanted to make sure to ping everyone in and around San Francisco that there is a Bay Area Bro meet up scheduled for tomorrow evening. Quite a few Bro developers will be and many long-time Bro users will be there (me included!). I'm pretty sure it should be a lot of fun for everyone, hopefully we will have time to talk about everything. :) Here is the link for more information about the meet up: https://www.meetup.com/Bay-Area-Bro-Security-Meetup/events/245836249/ .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From slagell at illinois.edu Mon Jan 8 09:37:22 2018 From: slagell at illinois.edu (Slagell, Adam J) Date: Mon, 8 Jan 2018 17:37:22 +0000 Subject: [Bro] Update on the Bro renaming process Message-ID: <137321AB-4FF2-4AB8-8E65-8DD2EAFC6B9B@illinois.edu> Speaking for the Bro leadership team, I would like to update the community on our progress towards choosing a new name. First, I would like to thank all of the people who submitted their ideas in the past months. Many of you had even come up with the same ideas independently, and it was nice to see all the thoughtful responses. Before the holidays, we met as a team to try and narrow the list down to our 5 favorite candidate names, but we found it challenging to find names we all liked without trademark issues. This is unfortunately a very dense space for names: software and tech companies. As such, we have decided to use some of our funds from previous sponsorship to hire a professional company to help us come up with a new name. We will be giving this company all the data you have submitted to help seed the process to generate more ideas. They will also be interviewing leadership team members and other contributors. While I wish that we could give a timeline, this is uncharted territory for us as a project. What we can and will do though is communicate as this process continues and involve the community as possible and appropriate along the way. :Adam Slagell Chair of the Bro Project Leadership Team ------ Adam J. Slagell Director, Cybersecurity & Networking Division Chief Information Security Officer National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." From valerio.click at gmx.com Mon Jan 8 10:25:01 2018 From: valerio.click at gmx.com (Valerio) Date: Mon, 8 Jan 2018 19:25:01 +0100 Subject: [Bro] Best way to contribute to existing analyzer In-Reply-To: <20170613232832.GP10430@icir.org> References: <20170613232832.GP10430@icir.org> Message-ID: <39f5d71b-bae2-371a-93da-9b559b88aa00@gmx.com> Hi, after a few months I finally made to pack my contribution proposal as a pull request available at https://github.com/bro/bro/pull/121 The patch introduces new options types for DHCP protocol and extends dhcp event including new parameters that I believe are useful in network forensics analysis. The options are the following: 55 Parameters Request List; 58 Renewal time; 59 Rebinding time; 61 Client Identifier; 82 Relay Agent Information. while the following are the extended events: dhcp_discover exports client identifier and parameters request list; dhcp_request exports client_identifier and parameters request list; dhcp_ack exports rebinding time, renewal time and list of suboptions value of dhcp relay agent information option; dhcp_inform exports parameters request list. Looking forward to receving feedbacks! best, Valerio Il 14/06/2017 01:28, Robin Sommer ha scritto: > > > On Wed, Jun 14, 2017 at 01:04 +0200, Valerio wrote: > >> What would be the best procedure (and format) to submit such a patch? > > Easiest is to prepare a pull request on GitHub. We have some > guidelines here: > https://www.bro.org/development/contribute.html#submitting-patches > > Looking forward to your patches! > > Robin > From moustafa.elbadry at oregonstate.edu Mon Jan 8 11:27:26 2018 From: moustafa.elbadry at oregonstate.edu (ElBadry Shaker, Moustafa) Date: Mon, 8 Jan 2018 19:27:26 +0000 Subject: [Bro] Bro logs from JSON to TSV In-Reply-To: <94820346-DD4D-4331-9483-97D055D94D53@oregonstate.edu> References: <3925859B-5857-47A0-B529-5DEE2E23DDB4@illinois.edu> <55C831FB-481D-4771-8846-905DD20B381A@oregonstate.edu> <6A51750F-798D-44E1-8378-0E4364F98A44@illinois.edu> <94820346-DD4D-4331-9483-97D055D94D53@oregonstate.edu> Message-ID: <1FB3C7A8-C939-47F2-BE58-CA781F9FE806@oregonstate.edu> Hello, I have a follow up question on this. Justin, you mentioned that I could tell bro to write out the logs in both formats (TSV and JSON) at the same time. How can I do this? And can I have the TSV logs saved in one directory and the JSON logs saved in another directory? Is the ascii.bro file located at /usr/local/bro/share/bro/base/frameworks/logging/writers/ the right file where we can configure bro to write in two different formats? Thanks a lot for your help. I really appreciate it! Moustafa On 1/5/18, 8:34 AM, "ElBadry Shaker, Moustafa" wrote: Great. Thanks Justin for sharing this. Definitely helps us a lot. Moustafa On 1/4/18, 12:36 PM, "Azoff, Justin S" wrote: > On Jan 3, 2018, at 4:38 PM, ElBadry Shaker, Moustafa wrote: > > I want the exact TSV format. > > We currently have our Bro cluster writing logs in JSON. There are couple of network traffic analytics tools like RITA (Real Intelligence Threat Analytics) and some AWK scripts that we want to use. The problem is that the tools we want to use work only with Bro?s default TSV format. > > Moustafa Ah, I see now. You have a few of options here. You could just tell bro to write out the logs in both formats at the same time. For older logs there is only a script for bro that can re-log to json, but not the other way, most people have the opposite problem. There is an open issue for RITA to support json: https://github.com/ocmdev/rita/issues/146 A tool to convert the json logs back into the TSV format could be written, but ultimately that would be a waste of time. Better to update RITA to support json instead of writing more tools to work with the tsv format that only bro uses. For awk stuff you can swap out bro-cut for jq or https://github.com/JustinAzoff/json-cut json-cut it doesn't support all the options that bro-cut supports and may be a bit buggy, but it's easier to extract a few fields from a json log as TSV and 2x faster than jq. If I can find a nice, small json library for C we can probably update bro-cut to natively support the json logs. For now, to extract note and msg from a stream of notice logs with bro-cut and json-cut you just do zcat notice.* | bro-cut note msg | awk ... zcat notice.* | json-cut note msg | awk ... For jq you use something like zcat notice.* | jq -r '[.note, .msg]|@tsv' | awk ... If the awk scripts are hardcoding top level field numbers like $3 and $5 instead of using bro-cut... they should not do that :-) ? Justin Azoff From moustafa.elbadry at oregonstate.edu Tue Jan 9 08:38:21 2018 From: moustafa.elbadry at oregonstate.edu (ElBadry Shaker, Moustafa) Date: Tue, 9 Jan 2018 16:38:21 +0000 Subject: [Bro] Bro logs from JSON to TSV In-Reply-To: References: <3925859B-5857-47A0-B529-5DEE2E23DDB4@illinois.edu> <55C831FB-481D-4771-8846-905DD20B381A@oregonstate.edu> <6A51750F-798D-44E1-8378-0E4364F98A44@illinois.edu> <94820346-DD4D-4331-9483-97D055D94D53@oregonstate.edu> <1FB3C7A8-C939-47F2-BE58-CA781F9FE806@oregonstate.edu> Message-ID: Great. Thanks for sharing this. I really appreciate it! Moustafa On 1/8/18, 2:45 PM, "Kinkead, Tanner" wrote: Look at add-JSON: https://gist.github.com/J-Gras/f9f86828f9e9d9c0b8f0908bc3573bb0 That will log JSON output to the path you define in path_json, and should retain the standard logging as well. Add-JSON is also available as a bro package. I've been able to get the log rotation to work for this script, though. I ended up creating a cron job that stops bro once a day, purges the JSON logs, and restarts. -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of ElBadry Shaker, Moustafa Sent: Monday, January 8, 2018 11:27 AM To: Azoff, Justin S ; bro at bro.org Subject: Re: [Bro] Bro logs from JSON to TSV Hello, I have a follow up question on this. Justin, you mentioned that I could tell bro to write out the logs in both formats (TSV and JSON) at the same time. How can I do this? And can I have the TSV logs saved in one directory and the JSON logs saved in another directory? Is the ascii.bro file located at /usr/local/bro/share/bro/base/frameworks/logging/writers/ the right file where we can configure bro to write in two different formats? Thanks a lot for your help. I really appreciate it! Moustafa On 1/5/18, 8:34 AM, "ElBadry Shaker, Moustafa" wrote: Great. Thanks Justin for sharing this. Definitely helps us a lot. Moustafa On 1/4/18, 12:36 PM, "Azoff, Justin S" wrote: > On Jan 3, 2018, at 4:38 PM, ElBadry Shaker, Moustafa wrote: > > I want the exact TSV format. > > We currently have our Bro cluster writing logs in JSON. There are couple of network traffic analytics tools like RITA (Real Intelligence Threat Analytics) and some AWK scripts that we want to use. The problem is that the tools we want to use work only with Bro?s default TSV format. > > Moustafa Ah, I see now. You have a few of options here. You could just tell bro to write out the logs in both formats at the same time. For older logs there is only a script for bro that can re-log to json, but not the other way, most people have the opposite problem. There is an open issue for RITA to support json: https://github.com/ocmdev/rita/issues/146 A tool to convert the json logs back into the TSV format could be written, but ultimately that would be a waste of time. Better to update RITA to support json instead of writing more tools to work with the tsv format that only bro uses. For awk stuff you can swap out bro-cut for jq or https://github.com/JustinAzoff/json-cut json-cut it doesn't support all the options that bro-cut supports and may be a bit buggy, but it's easier to extract a few fields from a json log as TSV and 2x faster than jq. If I can find a nice, small json library for C we can probably update bro-cut to natively support the json logs. For now, to extract note and msg from a stream of notice logs with bro-cut and json-cut you just do zcat notice.* | bro-cut note msg | awk ... zcat notice.* | json-cut note msg | awk ... For jq you use something like zcat notice.* | jq -r '[.note, .msg]|@tsv' | awk ... If the awk scripts are hardcoding top level field numbers like $3 and $5 instead of using bro-cut... they should not do that :-) ? Justin Azoff _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro Notice: All email and instant messages (including attachments) sent to or from Franklin Templeton Investments (FTI) personnel may be retained, monitored and/or reviewed by FTI and its agents, or authorized law enforcement personnel, without further notice or consent. . From seth at corelight.com Wed Jan 10 10:54:25 2018 From: seth at corelight.com (Seth Hall) Date: Wed, 10 Jan 2018 10:54:25 -0800 Subject: [Bro] Looking up fa_file given FUID In-Reply-To: References: Message-ID: <9AAC75AF-114A-4DFE-A303-8C35E50E2DBA@corelight.com> On 12 Oct 2017, at 15:16, Lamps, Jereme wrote: > I was just wondering if it was possible to lookup fa_file or > Files::Info records given a FUID. I have been looking through the > built in functions but have not seen anything. Sorry about that oversight. I just filed a merge request for this feature and it will be in Bro 2.6 https://bro-tracker.atlassian.net/browse/BIT-1887 .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From dnj0496 at gmail.com Wed Jan 10 11:55:58 2018 From: dnj0496 at gmail.com (Dk Jack) Date: Wed, 10 Jan 2018 11:55:58 -0800 Subject: [Bro] conn.log question Message-ID: Hi, I am trying to make sense of a couple of fields in the conn.log. The fields in question are 'local_orig' and 'local_resp'. I read the comments (shown at the end of this email) in main.bro of conn directory but I still can't quiet follow what these fields mean. Do these fields mean that the request/response were initiated from the system where bro was running? I am performing analysis using bro and bro is receiving traffic over a span port. In the connection log both these fields are set to true for a connection and I am wondering why. Any further clarification is appreciated. Thanks. Dk. ## If the connection is originated locally, this value will be T. ## If it was originated remotely it will be F. In the case that ## the :bro:id:`Site::local_nets` variable is undefined, this ## field will be left empty at all times. local_orig: bool &log &optional; ## If the connection is responded to locally, this value will be T. ## If it was responded to remotely it will be F. In the case that ## the :bro:id:`Site::local_nets` variable is undefined, this ## field will be left empty at all times. local_resp: bool &log &optional; -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180110/8d1a38de/attachment.html From zeolla at gmail.com Wed Jan 10 12:22:53 2018 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Wed, 10 Jan 2018 20:22:53 +0000 Subject: [Bro] conn.log question In-Reply-To: References: Message-ID: I suggest you look more into local_nets and networks.cfg. Networks set in networks.cfg are those that bro will consider local, and those fields are not associated to traffic to/from the workers (excluding the traffic that they are monitoring). Think non-RFC 1918 (and associated RFCs) subnets that bro may be monitoring and you own/are associated with your systems - public IPs that you own. https://www.bro.org/sphinx/scripts/base/utils/site.bro.html Jon On Wed, Jan 10, 2018, 15:05 Dk Jack wrote: > Hi, > I am trying to make sense of a couple of fields in the conn.log. The > fields in question are 'local_orig' and 'local_resp'. I read the comments > (shown at the end of this email) in main.bro of conn directory but I still > can't quiet follow what these fields mean. Do these fields mean that the > request/response were initiated from the system where bro was running? > > I am performing analysis using bro and bro is receiving traffic over a > span port. In the connection log both these fields are set to true for a > connection and I am wondering why. Any further clarification is > appreciated. Thanks. > > Dk. > > > > ## If the connection is originated locally, this value will be T. > ## If it was originated remotely it will be F. In the case that > ## the :bro:id:`Site::local_nets` variable is undefined, this > ## field will be left empty at all times. > local_orig: bool &log &optional; > > ## If the connection is responded to locally, this value will be T. > ## If it was responded to remotely it will be F. In the case that > ## the :bro:id:`Site::local_nets` variable is undefined, this > ## field will be left empty at all times. > local_resp: bool &log &optional; > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180110/80a21abf/attachment.html From jlay at slave-tothe-box.net Wed Jan 10 13:39:50 2018 From: jlay at slave-tothe-box.net (James Lay) Date: Wed, 10 Jan 2018 14:39:50 -0700 Subject: [Bro] conn.log question In-Reply-To: References: Message-ID: <8632fb12dabf065445b841d6f4fa2976@localhost> I keep this one bookmarked: https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info James On 2018-01-10 13:22, Zeolla at GMail.com wrote: > I suggest you look more into local_nets and networks.cfg. Networks set in networks.cfg are those that bro will consider local, and those fields are not associated to traffic to/from the workers (excluding the traffic that they are monitoring). Think non-RFC 1918 (and associated RFCs) subnets that bro may be monitoring and you own/are associated with your systems - public IPs that you own. > > https://www.bro.org/sphinx/scripts/base/utils/site.bro.html > > Jon > > On Wed, Jan 10, 2018, 15:05 Dk Jack wrote: > >> Hi, >> I am trying to make sense of a couple of fields in the conn.log. The fields in question are 'local_orig' and 'local_resp'. I read the comments (shown at the end of this email) in main.bro of conn directory but I still can't quiet follow what these fields mean. Do these fields mean that the request/response were initiated from the system where bro was running? >> >> I am performing analysis using bro and bro is receiving traffic over a span port. In the connection log both these fields are set to true for a connection and I am wondering why. Any further clarification is appreciated. Thanks. >> >> Dk. >> >> ## If the connection is originated locally, this value will be T. >> ## If it was originated remotely it will be F. In the case that >> ## the :bro:id:`Site::local_nets` variable is undefined, this >> ## field will be left empty at all times. >> local_orig: bool &log &optional; >> >> ## If the connection is responded to locally, this value will be T. >> ## If it was responded to remotely it will be F. In the case that >> ## the :bro:id:`Site::local_nets` variable is undefined, this >> ## field will be left empty at all times. >> local_resp: bool &log &optional; >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- > > Jon > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180110/996031c9/attachment.html From dnj0496 at gmail.com Wed Jan 10 14:49:46 2018 From: dnj0496 at gmail.com (Dk Jack) Date: Wed, 10 Jan 2018 14:49:46 -0800 Subject: [Bro] http.log q. Message-ID: Hi, In a cluster environment, in the HTTP log, for the same connection-id i.e same 4-tuple and UID, is it ok if the transaction depth field value is lower than the ten-depth of some of the lines that came before it? for example, I am seeing txns as shown below... 1515542375.578187 CGR1kN3pynC8a3GXK1 10.20.11.1 7867 10.20.11.120 9453 79 POST ... 1515542387.701328 CGR1kN3pynC8a3GXK1 10.20.11.1 7867 10.20.11.120 9453 90 POST ... 1515542354.674611 CGR1kN3pynC8a3GXK1 10.20.11.1 7867 10.20.11.120 9453 55 POST ... 1515542382.015911 CGR1kN3pynC8a3GXK1 10.20.11.1 7867 10.20.11.120 9453 85 POST ... Is this normal? What is the explanation. Thanks. Dk. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180110/abe770fd/attachment.html From joe at onshore.com Fri Jan 12 00:21:42 2018 From: joe at onshore.com (Joseph Gresham) Date: Fri, 12 Jan 2018 02:21:42 -0600 Subject: [Bro] Using Bro in offline mode (pcap spooling) Message-ID: <937e1e32-f187-e703-5a00-63841a4e21ba@onshore.com> Greetings! I have seen previous posts about people wanting to use bro against full packet capture datastore's and wanted to share a solution I have found, some limitations encountered and a partial solution before finally asking for clarification on cluster support for reading PCAP's. Solution for consuming pcap's continously: Firstly to deal with the issue of sessions spanning multiple capture files without resorting to using tcpreplay I use this project from CIRCL https://github.com/CIRCL/pcapdj It works by using redis to instantiate a queue that holds paths to pcap's for sequential processing.? The initial PCAP is fed to a fifo with the header intact but at the end of that file the footer is not put on the spool causing the libpcap code to continue waiting for additional packets, the next and subsequent PCAP's have the header removed so to the reading application this appears as a continous packet stream.?? There are additional details regarding how the queue is managed (you have to authorize reading of the next file) that I won't go into here but I will add that I use inotify to trigger adding PCAP's to the queue after they? are closed by the writing application. Limitations: I went this route initially because I believed that processing PCAP's already present on these sensors due to fullpacket capture would be more efficient as it removed the realtime constraints associated with live operation and analysis.? This has held true on sensors analyzing single or multiple 1Gbps links however on a deployment that has 2x10Gbps and 1x1Gbps link being dumped using PF_RING multi (eth4,eth5,eth6) I have run into a few issues.? 1.? Timing issues between the 10Gbps and 1Gbps links lead to packet ordering issues that made me change the way the SPAN sessions were setup otherwise file extraction and other things did not happen properly.? Specifically whenever dealing with multiple capture links on switches in the same zone (especially when 2 or more switches are redundant) my practice is to setup an RX span on every port except those that interconnect the redundant chassis.?? This gets a copy of both directions of the traffic and prevents duplication. In this specific scenario the two 10Gbps links are to redundant Nexus (think server-core) switches (cross connect via port-channel) serving multiple leaf swtches where servers live and an additional port channel to the redundant user-core switches. User core has to transit server-core to get WAN/INET. The 1Gbps link is to a standalone switch that terminates the WAN routers and firewalls. So all ports had RX copied except the port-channels cross connecting the nexus's and each of their uplinks to the wan switch.? To work around the timing issues and maintain fidelity of capture for internet bound traffic i had to change that WAN switch to copy RX on every port and implement ACL's on the Nexus SPAN setup to prevent getting duplicate copies in the outbound direction for internet flows while still getting user->server and server->user traffic 2. I have found that first Bro is amazingly efficient provided you do not get stupid with scripts, and so far for non 10Gbps environments this spooling setup works great! However for the 10G environments we are falling quite far behind using a single process (12-14 hours) during peak usage periods.? As the evening goes on and usage dies down we start to catch up but I only expect this to get worse as this organization continues to grow.?? It's obvious to me that due to the nature of reading from a FIFO my options for parallelization are farely limited but I have found some somewhat cludgey solutions. a. use mbuffer to read from the single fifo and output the same stream to additional fifo's which are then read by individual bro instances. Use BPF's in the local.bro as detailed here ( http://ossectools.blogspot.com/2012/10/multi-node-bro-cluster-setup-howto.html ) to do load sharing based on 2-tuple. b. use tcpsplit ( https://github.com/pmcgleenon/tcpsplit ) to read from the single fifo and write to a number of additional fifo's which can then be read by multiple bro's.? TCPSplit supports VLAN headers and uses either an LFU table, or a hash on the 4-tuple (5 for vlan), hash on the 2-tuple or any of the previous methods but using only the high order 24 bits of the ip (split by /24).? In testing this was workable but lacks the elegance of bro-clustering and its state and metadate sharing.??? The standalone nature of the bro's would break things like SMTPUrl click analysis and other cool correlations we use. c. DPDK? has support for software virtual devices ( http://dpdk.org/doc/guides/nics/pcap_ring.html ) backed by pcap (librte_pmd_ring,librte_pmd_pcap) which I have not tested but look promising as a way to run a bro cluster against PCAP's. Now recently I was reading this list and came across this http://mailman.icsi.berkeley.edu/pipermail/bro/2014-September/007458.html where seth mentions using the process command in broctl.? I wanted to ask if that is still valid in a cluster environment, and if so how is the pcap distributed to workers???? I do recall someone else mentioning using packet briks as a PCAP broker which at that time was still in development.... Thanks and sorry if this is all TLDR. -- -- ======================= Joseph Gresham Jr. joe at onshore.com Network Security Engineer Onshore Networks 312-850-5200 x.116 Desk 312-208-1887 Cell From jsiwek at corelight.com Fri Jan 12 08:38:33 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Fri, 12 Jan 2018 10:38:33 -0600 Subject: [Bro] Using Bro in offline mode (pcap spooling) In-Reply-To: <937e1e32-f187-e703-5a00-63841a4e21ba@onshore.com> References: <937e1e32-f187-e703-5a00-63841a4e21ba@onshore.com> Message-ID: On Fri, Jan 12, 2018 at 2:21 AM, Joseph Gresham wrote: > Now recently I was reading this list and came across this > http://mailman.icsi.berkeley.edu/pipermail/bro/2014-September/007458.html > where seth mentions using the process command in broctl. I wanted to > ask if that is still valid in a cluster environment, and if so how is > the pcap distributed to workers? The process command only runs the pcap through a single Bro instance, so probably not what you need. There's more details on how it works in the docs [1], for reference. - Jon [1] https://www.bro.org/sphinx/components/broctl/README.html#command-reference From johanna at icir.org Fri Jan 12 08:55:34 2018 From: johanna at icir.org (Johanna Amann) Date: Fri, 12 Jan 2018 08:55:34 -0800 Subject: [Bro] http.log q. In-Reply-To: References: Message-ID: <20180112165534.kol55x2gd2jemgxe@user237.sys.ICSI.Berkeley.EDU> Hi, If you take a look at the timestamps in the log that you posted you will notice that the transaction depth value is in the correct order if you sort the log by timestamp. Bro log files are generally not guaranteed to be well-odered - though I am admittedly not 100% sure without looking into the http scripts why the http.log sent by a single worker would be reordered like that :) I hope this helps, Johanna On Wed, Jan 10, 2018 at 02:49:46PM -0800, Dk Jack wrote: > Hi, > In a cluster environment, in the HTTP log, for the same connection-id i.e > same 4-tuple and UID, is it ok if the transaction depth field value is > lower than the ten-depth of some of the lines that came before it? for > example, I am seeing txns as shown below... > > 1515542375.578187 CGR1kN3pynC8a3GXK1 10.20.11.1 7867 10.20.11.120 9453 > 79 POST ... > 1515542387.701328 CGR1kN3pynC8a3GXK1 10.20.11.1 7867 10.20.11.120 9453 > 90 POST ... > 1515542354.674611 CGR1kN3pynC8a3GXK1 10.20.11.1 7867 10.20.11.120 9453 > 55 POST ... > 1515542382.015911 CGR1kN3pynC8a3GXK1 10.20.11.1 7867 10.20.11.120 9453 > 85 POST ... > > Is this normal? What is the explanation. Thanks. > > Dk. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From seth at corelight.com Sat Jan 13 06:46:15 2018 From: seth at corelight.com (Seth Hall) Date: Sat, 13 Jan 2018 09:46:15 -0500 Subject: [Bro] http.log q. In-Reply-To: References: Message-ID: On 10 Jan 2018, at 17:49, Dk Jack wrote: > 1515542375.578187 CGR1kN3pynC8a3GXK1 10.20.11.1 7867 10.20.11.120 > 9453 > 79 POST ... > 1515542387.701328 CGR1kN3pynC8a3GXK1 10.20.11.1 7867 10.20.11.120 > 9453 > 90 POST ... > 1515542354.674611 CGR1kN3pynC8a3GXK1 10.20.11.1 7867 10.20.11.120 > 9453 > 55 POST ... > 1515542382.015911 CGR1kN3pynC8a3GXK1 10.20.11.1 7867 10.20.11.120 > 9453 > 85 POST ... Are these logs being written with the normal "ascii" log writer? If they are, I don't have a sensible explanation yet for why they would be out of order like that and I've never seen that behavior. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From dnj0496 at gmail.com Sat Jan 13 09:52:58 2018 From: dnj0496 at gmail.com (Dk Jack) Date: Sat, 13 Jan 2018 09:52:58 -0800 Subject: [Bro] http.log q. In-Reply-To: References: Message-ID: <9B167B1B-A9BD-456D-849E-AB00E75AD143@gmail.com> Yes, they are being written using the default ascii writer Bhasker. > On Jan 13, 2018, at 6:46 AM, Seth Hall wrote: > > > >> On 10 Jan 2018, at 17:49, Dk Jack wrote: >> >> 1515542375.578187 CGR1kN3pynC8a3GXK1 10.20.11.1 7867 10.20.11.120 9453 >> 79 POST ... >> 1515542387.701328 CGR1kN3pynC8a3GXK1 10.20.11.1 7867 10.20.11.120 9453 >> 90 POST ... >> 1515542354.674611 CGR1kN3pynC8a3GXK1 10.20.11.1 7867 10.20.11.120 9453 >> 55 POST ... >> 1515542382.015911 CGR1kN3pynC8a3GXK1 10.20.11.1 7867 10.20.11.120 9453 >> 85 POST ... > > Are these logs being written with the normal "ascii" log writer? If they are, I don't have a sensible explanation yet for why they would be out of order like that and I've never seen that behavior. > > .Seth > > -- > Seth Hall * Corelight, Inc * www.corelight.com From seth at corelight.com Sun Jan 14 07:38:01 2018 From: seth at corelight.com (Seth Hall) Date: Sun, 14 Jan 2018 10:38:01 -0500 Subject: [Bro] conn.log question In-Reply-To: <8632fb12dabf065445b841d6f4fa2976@localhost> References: <8632fb12dabf065445b841d6f4fa2976@localhost> Message-ID: There is also this... https://github.com/corelight/bro-cheatsheets/blob/master/Corelight-Bro-Cheatsheets-2.5.pdf .Seth On 10 Jan 2018, at 16:39, James Lay wrote: > I keep this one bookmarked: > > https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info > > > James > > On 2018-01-10 13:22, Zeolla at GMail.com wrote: > >> I suggest you look more into local_nets and networks.cfg. Networks >> set in networks.cfg are those that bro will consider local, and those >> fields are not associated to traffic to/from the workers (excluding >> the traffic that they are monitoring). Think non-RFC 1918 (and >> associated RFCs) subnets that bro may be monitoring and you own/are >> associated with your systems - public IPs that you own. >> >> https://www.bro.org/sphinx/scripts/base/utils/site.bro.html >> >> Jon >> >> On Wed, Jan 10, 2018, 15:05 Dk Jack wrote: >> >>> Hi, >>> I am trying to make sense of a couple of fields in the conn.log. The >>> fields in question are 'local_orig' and 'local_resp'. I read the >>> comments (shown at the end of this email) in main.bro of conn >>> directory but I still can't quiet follow what these fields mean. Do >>> these fields mean that the request/response were initiated from the >>> system where bro was running? >>> >>> I am performing analysis using bro and bro is receiving traffic over >>> a span port. In the connection log both these fields are set to true >>> for a connection and I am wondering why. Any further clarification >>> is appreciated. Thanks. >>> >>> Dk. >>> >>> ## If the connection is originated locally, this value will be T. >>> ## If it was originated remotely it will be F. In the case that >>> ## the :bro:id:`Site::local_nets` variable is undefined, this >>> ## field will be left empty at all times. >>> local_orig: bool &log &optional; >>> >>> ## If the connection is responded to locally, this value will be T. >>> ## If it was responded to remotely it will be F. In the case that >>> ## the :bro:id:`Site::local_nets` variable is undefined, this >>> ## field will be left empty at all times. >>> local_resp: bool &log &optional; >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> -- >> >> Jon >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * www.corelight.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180114/41612fcd/attachment.html From Izik.Birka at hot.net.il Tue Jan 16 06:20:33 2018 From: Izik.Birka at hot.net.il (Izik Birka) Date: Tue, 16 Jan 2018 14:20:33 +0000 Subject: [Bro] bro - intel critical stack Message-ID: <592228F4D0C8504187F2F76658040CB6E048A5E9@HOT-MAILBOX-02.HOT.NET.IL> Hi I install the Critical Stack Intel Client It is looks like it's working , I can see that the feeds are updated , but for some reason I don't have intel.log file I notice that the script was created in /opt/critical-stack/framework/intel There are 3 files in the folder : 1. __load__.bro 2. Feeds.bro 3. master-public.bro.dat the __load__ file contains : @load ./feeds.bro The feeds.bro file contains : @load base/frameworks/intel @load frameworks/intel/seen @load frameworks/intel/do_notice redef Intel::read_files += { "/opt/critical-stack/frameworks/intel/master-public.bro.dat" The master-public.bro.dat contains the intel : 109.229.210.250 Intel::ADDR from https://zeustracker.abuse.ch/blocklist.php?download=ipblocklist via intel.criticalstack.com F 124.110.195.160 Intel::ADDR from https://zeustracker.abuse.ch/blocklist.php?download=ipblocklist via intel.criticalstack.com F I try to load it in the local.bro but still it's not working .... I trying access those ip's to create log , but it is not working..... Any idea ? Thanks izik This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain materials protected by copyright or information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or agreement. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication by error, notify the sender immediately and delete this message immediately. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180116/2922b6e9/attachment.html From jlay at slave-tothe-box.net Wed Jan 17 11:46:32 2018 From: jlay at slave-tothe-box.net (James Lay) Date: Wed, 17 Jan 2018 12:46:32 -0700 Subject: [Bro] Intel framework not working as expected Message-ID: <9919629fe8d384497ee57537221c876e@localhost> So I have a current working intel framework via this: http://blog.bro.org/2014/01/intelligence-data-and-bro_4980.html this works great and the intel feeds fire off in intel.log. With a couple minor tweaks, I modded the info here to make a newdomain.intel file: https://isc.sans.edu/forums/diary/Tracking+Newly+Registered+Domains/23127/ >From my newdomain.intel (obfuscation added): #fields indicator indicator_type meta.source meta.url meta.do_notice meta.if_in 00009117[.]com Intel::DOMAIN newdomains - F - 0000dw[.]com Intel::DOMAIN newdomains - F - 0008[.]red Intel::DOMAIN newdomains - F - And my intel lines in local.bro: redef Intel::read_files += { "/opt/bro/share/bro/site/alienvault.intel", "/opt/bro/share/bro/site/meyhemic.intel", "/opt/bro/share/bro/site/malhosts.intel", "/opt/bro/share/bro/site/malips.intel", "/opt/bro/share/bro/site/newdomain.intel" }; ..... As I'm typing this I think I might have the answer, but now I have another question :D If a do a dns request for 0008[.]red I get: "2018-01-17T17:01:25+0000 Cn235WxlXKegS2qn4 x.x.x.x 61616 x.x.x.x 53 udp 4327 0.260124 000movies[.]com 1 C_INTERNET 1 A 0 NOERROR F F T T 0 x.x.x.x 14400.000000 F" but nothing in the intel.log. So...it appears that the intel framework is using just active connections? Which makes sense, but now, how would I get bro to, in layman's terms: "bounce dns requests off of the intel lists as well"? Please let me know if I haven't explained this well enough..thank you. James -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180117/fcaa8de0/attachment.html From jlay at slave-tothe-box.net Wed Jan 17 12:31:29 2018 From: jlay at slave-tothe-box.net (James Lay) Date: Wed, 17 Jan 2018 13:31:29 -0700 Subject: [Bro] Intel framework not working as expected In-Reply-To: References: <9d3c4d3008b4929c8a9edc8e15798a8e@localhost> Message-ID: Thanks Fatema...that was it..had to do the install as well :) James On 2018-01-17 13:17, fatema bannatwala wrote: > Not sure if the new changes are loaded when doing broctl stop and start, might want to try broctl install as well before doing broctl start? > > On Wed, Jan 17, 2018 at 3:12 PM, James Lay wrote: > > I did do a broctl stop and start thank you...and I was thinking an entry should show up in intel, but it does not. > > James > > On 2018-01-17 13:10, fatema bannatwala wrote: > Just a trivial question: > Did you restart Bro after adding your new intel file entry in local.bro? > If a DNS record is getting logged for the domain in your intel file, then I would think an entry should be created in intel.log. > > Fatema. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180117/cc8919b9/attachment.html From jazoff at illinois.edu Wed Jan 17 13:10:27 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 17 Jan 2018 21:10:27 +0000 Subject: [Bro] Intel framework not working as expected In-Reply-To: References: <9d3c4d3008b4929c8a9edc8e15798a8e@localhost> Message-ID: > On Jan 17, 2018, at 3:31 PM, James Lay wrote: > > Thanks Fatema...that was it..had to do the install as well :) broctl deploy exists because of this problem :-) If you are doing install/stop/start you should just use deploy, it does all the steps and in the correct order. ? Justin Azoff From jlay at slave-tothe-box.net Thu Jan 18 08:42:56 2018 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 18 Jan 2018 09:42:56 -0700 Subject: [Bro] A little more confusion with Intel Message-ID: So I'm testing something completely unrelated to this issue, but I've run into something interesting. First off following this works: https://www.bro.org/current/solutions/intel/index.html my test intel-1.bro: @load frameworks/intel/seen redef Intel::read_files += { fmt("%s/intel-1.dat", @DIR) }; my intel-1.dat file (whitespace=tab): #fields indicator indicator_type meta.source fetchback.com Intel::DOMAIN my_special_source yahoo.com Intel::DOMAIN testdomain I've carved out the dns request for fetchback.com from the exercise packet capture, which I'm including. Testing line below works just fine: bro -C -r exercise-traffic-fetch-dns.pcap intel-1.bro I see lot's of good stuff: conn.log 1258565309.806483 CmeOAzpOmlw26nOEi 192.168.1.103 53856 192.168.1.1 53 udp dns 0.200354 31 99 SF - - 0 Dd 1 59 1 127 (empty) dns.log 1258565309.806483 CVifWt1zc5YSG0Vhc9 192.168.1.103 53856 192.168.1.1 53 udp 4438 0.200354 fetchback.com 1 C_INTERNET 1 A 0 NOERROR F F TT 0 69.71.52.52 1800.000000 F intel.log 1258565309.806483 CmeOAzpOmlw26nOEi 192.168.1.103 53856 192.168.1.1 53 fetchback.com Intel::DOMAIN DNS::IN_REQUEST bro Intel::DOMAIN my_special_source - - - however running against the included yahoodns.pcap here's what I get: conn.log 1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 53 udp dns 0.003246 31 124 SF - - 0 Dd 1 59 1 152 (empty) dns.log 1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 53 udp 3285 0.003246 www.yahoo.com 1 C_INTERNET 1 A 0 NOERROR F F TT 0 atsv2-fp.wg1.b.yahoo.com,98.138.252.38,98.138.252.39,98.139.180.180,206.190.39.43 1320.000000,39.000000,39.000000,39.000000,39.000000 F and no intel.log. What's different here? Would love to know what I'm missing..thank you. James -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180118/8e89bdd0/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/vnd.tcpdump.pcap Size: 295 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180118/8e89bdd0/attachment.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/vnd.tcpdump.pcap Size: 274 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180118/8e89bdd0/attachment-0001.bin From shirkdog.bsd at gmail.com Thu Jan 18 09:16:34 2018 From: shirkdog.bsd at gmail.com (Michael Shirk) Date: Thu, 18 Jan 2018 12:16:34 -0500 Subject: [Bro] A little more confusion with Intel In-Reply-To: References: Message-ID: What do you have your local_nets set to? -- Michael Shirk Daemon Security, Inc. https://www.daemon-security.com On Jan 18, 2018 11:55, "James Lay" wrote: > So I'm testing something completely unrelated to this issue, but I've run > into something interesting. First off following this works: > > https://www.bro.org/current/solutions/intel/index.html > > my test intel-1.bro: > @load frameworks/intel/seen > > redef Intel::read_files += { > fmt("%s/intel-1.dat", @DIR) > }; > > my intel-1.dat file (whitespace=tab): > #fields indicator indicator_type meta.source > fetchback.com Intel::DOMAIN my_special_source > yahoo.com Intel::DOMAIN testdomain > > I've carved out the dns request for fetchback.com from the exercise > packet capture, which I'm including. Testing line below works just fine: > > bro -C -r exercise-traffic-fetch-dns.pcap intel-1.bro > > I see lot's of good stuff: > conn.log > 1258565309.806483 CmeOAzpOmlw26nOEi 192.168.1.103 53856 192.168.1.1 53 udp > dns 0.200354 31 99 SF - - 0 Dd 1 59 1 127 (empty) > > dns.log > 1258565309.806483 CVifWt1zc5YSG0Vhc9 192.168.1.103 53856 > 192.168.1.1 53 udp 4438 0.200354 fetchback.com > 1 C_INTERNET 1 A 0 NOERROR F F TT > 0 69.71.52.52 1800.000000 F > > intel.log > 1258565309.806483 CmeOAzpOmlw26nOEi 192.168.1.103 53856 192.168.1.1 53 > fetchback.com Intel::DOMAIN DNS::IN_REQUEST bro Intel::DOMAIN > my_special_source - - - > > > however running against the included yahoodns.pcap here's what I get: > conn.log > 1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 53 udp > dns 0.003246 31 124 SF - - 0 Dd 1 59 1 152 (empty) > > dns.log > 1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 53 udp > 3285 0.003246 www.yahoo.com 1 C_INTERNET 1 A 0 NOERROR F F TT 0 > atsv2-fp.wg1.b.yahoo.com,98.138.252.38,98.138.252.39,98.139.180.180,206.190.39.43 > 1320.000000,39.000000,39.000000,39.000000,39.000000 F > > and no intel.log. What's different here? Would love to know what I'm > missing..thank you. > > James > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180118/84fb6cd0/attachment-0001.html From jlay at slave-tothe-box.net Thu Jan 18 09:24:12 2018 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 18 Jan 2018 10:24:12 -0700 Subject: [Bro] A little more confusion with Intel In-Reply-To: References: Message-ID: <4e039ee09b0a272b140188bb1e0e320c@localhost> In this particular test I haven't set it for either run. Thanks Michael. James On 2018-01-18 10:16, Michael Shirk wrote: > What do you have your local_nets set to? > > -- > Michael Shirk > Daemon Security, Inc. > https://www.daemon-security.com > > On Jan 18, 2018 11:55, "James Lay" wrote: > >> So I'm testing something completely unrelated to this issue, but I've run into something interesting. First off following this works: >> >> https://www.bro.org/current/solutions/intel/index.html [1] >> >> my test intel-1.bro: >> @load frameworks/intel/seen >> >> redef Intel::read_files += { >> fmt("%s/intel-1.dat", @DIR) >> }; >> >> my intel-1.dat file (whitespace=tab): >> #fields indicator indicator_type meta.source >> fetchback.com [2] Intel::DOMAIN my_special_source >> yahoo.com [3] Intel::DOMAIN testdomain >> >> I've carved out the dns request for fetchback.com [2] from the exercise packet capture, which I'm including. Testing line below works just fine: >> >> bro -C -r exercise-traffic-fetch-dns.pcap intel-1.bro >> >> I see lot's of good stuff: >> conn.log >> 1258565309.806483 CmeOAzpOmlw26nOEi 192.168.1.103 53856 192.168.1.1 53 udp dns 0.200354 31 99 SF - - 0 Dd 1 59 1 127 (empty) >> >> dns.log >> 1258565309.806483 CVifWt1zc5YSG0Vhc9 192.168.1.103 53856 192.168.1.1 53 udp 4438 0.200354 fetchback.com [2] 1 C_INTERNET 1 A 0 NOERROR F F TT 0 69.71.52.52 1800.000000 F >> >> intel.log >> 1258565309.806483 CmeOAzpOmlw26nOEi 192.168.1.103 53856 192.168.1.1 53 fetchback.com [2] Intel::DOMAIN DNS::IN_REQUEST bro Intel::DOMAIN my_special_source - - - >> >> however running against the included yahoodns.pcap here's what I get: >> conn.log >> 1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 53 udp dns 0.003246 31 124 SF - - 0 Dd 1 59 1 152 (empty) >> >> dns.log >> 1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 53 udp 3285 0.003246 www.yahoo.com [4] 1 C_INTERNET 1 A 0 NOERROR F F TT 0 atsv2-fp.wg1.b.yahoo.com [5],98.138.252.38,98.138.252.39,98.139.180.180,206.190.39.43 1320.000000,39.000000,39.000000,39.000000,39.000000 F >> >> and no intel.log. What's different here? Would love to know what I'm missing..thank you. >> >> James >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro [6] Links: ------ [1] https://www.bro.org/current/solutions/intel/index.html [2] http://fetchback.com [3] http://yahoo.com [4] http://www.yahoo.com [5] http://atsv2-fp.wg1.b.yahoo.com [6] http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180118/e8095b0b/attachment.html From shirkdog.bsd at gmail.com Thu Jan 18 09:26:52 2018 From: shirkdog.bsd at gmail.com (Michael Shirk) Date: Thu, 18 Jan 2018 12:26:52 -0500 Subject: [Bro] A little more confusion with Intel In-Reply-To: <4e039ee09b0a272b140188bb1e0e320c@localhost> References: <4e039ee09b0a272b140188bb1e0e320c@localhost> Message-ID: Maybe tab formatting in the intel.dat file? I think you will get Reporter errors, so the first IOC works, but the second one does not because it cannot be parsed. -- Michael Shirk Daemon Security, Inc. https://www.daemon-security.com On Jan 18, 2018 12:24, "James Lay" wrote: > In this particular test I haven't set it for either run. Thanks Michael. > > James > > On 2018-01-18 10:16, Michael Shirk wrote: > > What do you have your local_nets set to? > > -- > Michael Shirk > Daemon Security, Inc. > https://www.daemon-security.com > > On Jan 18, 2018 11:55, "James Lay" wrote: > >> So I'm testing something completely unrelated to this issue, but I've run >> into something interesting. First off following this works: >> >> https://www.bro.org/current/solutions/intel/index.html >> >> my test intel-1.bro: >> @load frameworks/intel/seen >> >> redef Intel::read_files += { >> fmt("%s/intel-1.dat", @DIR) >> }; >> >> my intel-1.dat file (whitespace=tab): >> #fields indicator indicator_type meta.source >> fetchback.com Intel::DOMAIN my_special_source >> yahoo.com Intel::DOMAIN testdomain >> >> I've carved out the dns request for fetchback.com from the exercise >> packet capture, which I'm including. Testing line below works just fine: >> >> bro -C -r exercise-traffic-fetch-dns.pcap intel-1.bro >> >> I see lot's of good stuff: >> conn.log >> 1258565309.806483 CmeOAzpOmlw26nOEi 192.168.1.103 53856 192.168.1.1 53 >> udp dns 0.200354 31 99 SF - - 0 Dd 1 59 1 127 (empty) >> >> dns.log >> 1258565309.806483 CVifWt1zc5YSG0Vhc9 192.168.1.103 53856 >> 192.168.1.1 53 udp 4438 0.200354 fetchback.com >> 1 C_INTERNET 1 A 0 NOERROR F F TT >> 0 69.71.52.52 1800.000000 F >> >> intel.log >> 1258565309.806483 CmeOAzpOmlw26nOEi 192.168.1.103 53856 192.168.1.1 53 >> fetchback.com Intel::DOMAIN DNS::IN_REQUEST bro Intel::DOMAIN >> my_special_source - - - >> >> >> however running against the included yahoodns.pcap here's what I get: >> conn.log >> 1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 53 udp >> dns 0.003246 31 124 SF - - 0 Dd 1 59 1 152 (empty) >> >> dns.log >> 1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 53 udp >> 3285 0.003246 www.yahoo.com 1 C_INTERNET 1 A 0 NOERROR F F TT 0 >> atsv2-fp.wg1.b.yahoo.com,98.138.252.38,98.138.252.39,98.139.180.180,206.190.39.43 >> 1320.000000,39.000000,39.000000,39.000000,39.000000 F >> >> and no intel.log. What's different here? Would love to know what I'm >> missing..thank you. >> >> James >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180118/53dac0e2/attachment.html From jlay at slave-tothe-box.net Thu Jan 18 09:33:07 2018 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 18 Jan 2018 10:33:07 -0700 Subject: [Bro] A little more confusion with Intel In-Reply-To: References: <4e039ee09b0a272b140188bb1e0e320c@localhost> Message-ID: <740cf328421f4d46befd52a9dbe861d9@localhost> Thanks...both entries are tabbed formatted...still digging on my end via trace files. James On 2018-01-18 10:26, Michael Shirk wrote: > Maybe tab formatting in the intel.dat file? > > I think you will get Reporter errors, so the first IOC works, but the second one does not because it cannot be parsed. > > -- > Michael Shirk > Daemon Security, Inc. > https://www.daemon-security.com > > On Jan 18, 2018 12:24, "James Lay" wrote: > > In this particular test I haven't set it for either run. Thanks Michael. > > James > > On 2018-01-18 10:16, Michael Shirk wrote: > What do you have your local_nets set to? > > -- > Michael Shirk > Daemon Security, Inc. > https://www.daemon-security.com [1] > > On Jan 18, 2018 11:55, "James Lay" wrote: > > So I'm testing something completely unrelated to this issue, but I've run into something interesting. First off following this works: > > https://www.bro.org/current/solutions/intel/index.html [2] > > my test intel-1.bro: > @load frameworks/intel/seen > > redef Intel::read_files += { > fmt("%s/intel-1.dat", @DIR) > }; > > my intel-1.dat file (whitespace=tab): > #fields indicator indicator_type meta.source > fetchback.com [3] Intel::DOMAIN my_special_source > yahoo.com [4] Intel::DOMAIN testdomain > > I've carved out the dns request for fetchback.com [3] from the exercise packet capture, which I'm including. Testing line below works just fine: > > bro -C -r exercise-traffic-fetch-dns.pcap intel-1.bro > > I see lot's of good stuff: > conn.log > 1258565309.806483 CmeOAzpOmlw26nOEi 192.168.1.103 53856 192.168.1.1 53 udp dns 0.200354 31 99 SF - - 0 Dd 1 59 1 127 (empty) > > dns.log > 1258565309.806483 CVifWt1zc5YSG0Vhc9 192.168.1.103 53856 192.168.1.1 53 udp 4438 0.200354 fetchback.com [3] 1 C_INTERNET 1 A 0 NOERROR F F TT 0 69.71.52.52 1800.000000 F > > intel.log > 1258565309.806483 CmeOAzpOmlw26nOEi 192.168.1.103 53856 192.168.1.1 53 fetchback.com [3] Intel::DOMAIN DNS::IN_REQUEST bro Intel::DOMAIN my_special_source - - - > > however running against the included yahoodns.pcap here's what I get: > conn.log > 1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 53 udp dns 0.003246 31 124 SF - - 0 Dd 1 59 1 152 (empty) > > dns.log > 1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 53 udp 3285 0.003246 www.yahoo.com [5] 1 C_INTERNET 1 A 0 NOERROR F F TT 0 atsv2-fp.wg1.b.yahoo.com [6],98.138.252.38,98.138.252.39,98.139.180.180,206.190.39.43 1320.000000,39.000000,39.000000,39.000000,39.000000 F > > and no intel.log. What's different here? Would love to know what I'm missing..thank you. > > James > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro [7] Links: ------ [1] https://www.daemon-security.com [2] https://www.bro.org/current/solutions/intel/index.html [3] http://fetchback.com [4] http://yahoo.com [5] http://www.yahoo.com [6] http://atsv2-fp.wg1.b.yahoo.com [7] http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180118/14cd6154/attachment-0001.html From jazoff at illinois.edu Thu Jan 18 09:33:37 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 18 Jan 2018 17:33:37 +0000 Subject: [Bro] A little more confusion with Intel In-Reply-To: References: Message-ID: > On Jan 18, 2018, at 11:42 AM, James Lay wrote: > > ... > yahoo.com Intel::DOMAIN testdomain > ... > 1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 53 udp 3285 0.003246 www.yahoo.com 1 C_INTERNET 1 A 0 NOERROR F F TT 0 atsv2-fp.wg1.b.yahoo.com,98.138.252.38,98.138.252.39,98.139.180.180,206.190.39.43 1320.000000,39.000000,39.000000,39.000000,39.000000 F > > and no intel.log. What's different here? Would love to know what I'm missing..thank you. > www.yahoo.com is not yahoo.com You need an intel::seen even that uses https://github.com/sethhall/domain-tld to get that to match. I thought someone wrote a package that did this, but apparently not. ? Justin Azoff From jlay at slave-tothe-box.net Thu Jan 18 09:42:11 2018 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 18 Jan 2018 10:42:11 -0700 Subject: [Bro] A little more confusion with Intel In-Reply-To: References: Message-ID: <60a34e5669db124ce69305bc1232ab3c@localhost> Wait what? So....it looks like www.yahoo.com [1] matches, but yahoo.com doesn't? That kinda nukes the whole match any bad host with domain ;) Thank you can lend a hand Seth? Thanks. James On 2018-01-18 10:33, Azoff, Justin S wrote: >> On Jan 18, 2018, at 11:42 AM, James Lay wrote: >> >> ... > >> yahoo.com Intel::DOMAIN testdomain >> ... > >> 1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 53 udp 3285 0.003246 www.yahoo.com [1] 1 C_INTERNET 1 A 0 NOERROR F F TT 0 atsv2-fp.wg1.b.yahoo.com,98.138.252.38,98.138.252.39,98.139.180.180,206.190.39.43 1320.000000,39.000000,39.000000,39.000000,39.000000 F >> >> and no intel.log. What's different here? Would love to know what I'm missing..thank you. > > www.yahoo.com [1] is not yahoo.com > > You need an intel::seen even that uses > https://github.com/sethhall/domain-tld to get that to match. I > thought someone wrote a package that did this, but apparently not. > > -- > Justin Azoff Links: ------ [1] http://www.yahoo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180118/ade75315/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: 2018-01-18 10_35_16-fetchtrace.txt - Visual Studio Code.png Type: image/png Size: 32839 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180118/ade75315/attachment-0001.bin From fatema.bannatwala at gmail.com Thu Jan 18 09:42:36 2018 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Thu, 18 Jan 2018 12:42:36 -0500 Subject: [Bro] A little more confusion with Intel Message-ID: I see the dns request is for "www.yahoo.com", however the entry in your intel-1.dat is for "yahoo.com" Not sure if Bro intel framework works with the sub-domains lookup as well for intel. Try adding "www.yahoo.com" in your intel-1.dat , and see if intel.log triggers. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180118/c789022e/attachment.html From jlay at slave-tothe-box.net Thu Jan 18 09:46:22 2018 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 18 Jan 2018 10:46:22 -0700 Subject: [Bro] A little more confusion with Intel In-Reply-To: References: Message-ID: <1666153946a67ebf166097eef30fbf43@localhost> Ya we discovered that worked thanks Fatema...but that defeats the point of "domain" in the intel file :( James On 2018-01-18 10:42, fatema bannatwala wrote: > I see the dns request is for "www.yahoo.com [1]", however the entry in your intel-1.dat is for "yahoo.com [2]" > Not sure if Bro intel framework works with the sub-domains lookup as well for intel. > Try adding "www.yahoo.com [1]" in your intel-1.dat , and see if intel.log triggers. Links: ------ [1] http://www.yahoo.com [2] http://yahoo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180118/cbcb657b/attachment.html From jlay at slave-tothe-box.net Thu Jan 18 10:06:33 2018 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 18 Jan 2018 11:06:33 -0700 Subject: [Bro] A little more confusion with Intel In-Reply-To: <60a34e5669db124ce69305bc1232ab3c@localhost> References: <60a34e5669db124ce69305bc1232ab3c@localhost> Message-ID: So mystery #1 solved: Intel::DOMAIN != tld domain; cool Next up...with these same files only having this in the intel-1.dat file: 192.168.1.1 Intel::ADDR testip the above is tabbed formatted correctly. Now...running against both pcaps I get no intel hits. Here is the only entries in the trace file that show Intel::ADDR: }]', tpe = 'Input::EVENT_NEW', item = '[indicator=192.168.1.1, indicator_type=Intel::ADDR, meta=[source=testip, desc=, url=]]') 0.000000 /opt/bro/share/bro/base/frameworks/intel/./main.bro:469 function called: Intel::insert(item = '[indicator=192.168.1.1, indicator_type=Intel::ADDR, meta=[source=testip, desc=, url=]]') 0.000000 /opt/bro/share/bro/base/frameworks/intel/./main.bro:400 Builtin Function called: to_lower(str = '192.168.1.1') 0.000000 /opt/bro/share/bro/base/frameworks/intel/./main.bro:400 Function return: 192.168.1.1 0.000000 /opt/bro/share/bro/base/frameworks/intel/./main.bro:404 Builtin Function called: to_addr(ip = '192.168.1.1') 0.000000 /opt/bro/share/bro/base/frameworks/intel/./main.bro:404 Function return: 192.168.1.1 0.000000 /opt/bro/share/bro/base/frameworks/input/./main.bro:248 event called: Input::end_of_data(name = 'intel-/home/user/dev/bro/intel/./intel-1.dat', source = '/home/user/dev/bro/intel/./intel-1.dat') 0.000000 /opt/bro/share/bro/base/utils/./exec.bro:102 Builtin Function called: split_string1(str = 'intel-/home/user/dev/bro/intel/./intel-1.dat', re = '/^?(_)$?/') 0.000000 /opt/bro/share/bro/base/utils/./exec.bro:102 Function return: [intel-/home/user/dev/bro/intel/./intel-1.dat] 1516289219.143906 /opt/bro/share/bro/base/misc/find-checksum-offloading.bro:62 event called: ChecksumOffloading::check() 1516289219.143906 /opt/bro/share/bro/base/misc/find-checksum-offloading.bro:29 Builtin Function called: get_net_stats() 1516289219.143906 /opt/bro/share/bro/base/misc/find-checksum-offloading.bro:29 Function return: [pkts_recvd=1, pkts_dropped=0, pkts_link=0, bytes_recvd=73] 1516289219.143906 /opt/bro/share/bro/base/frameworks/packet-filter/./main.bro:157 event called: filter_change_tracking() 1516289219.143906 /opt/bro/share/bro/base/bif/event.bif.bro:88 event called: new_connection(c = '[id=[orig_h=192.168.1.2, orig_p=62196/udp, resp_h=192.168.1.1, resp_p=53/udp], orig=[size=31, state=1, num_pkts=0, num_bytes_ip=0, flow_label=0, l2_addr=48:4d:7e:a3:53:5e], resp=[size=0, state=0, num_pkts=0, num_bytes_ip=0, flow_label=0, l2_addr=00:08:e3:ff:fc:04], start_time=1516289219.143906, duration=0.0, service={ Here too, is there something I'm missing? In testing a different packet captures using TCP, I get intel...so does the Intel framework not support UDP? Thank you. James On 2018-01-18 10:42, James Lay wrote: > Wait what? So....it looks like www.yahoo.com [1] matches, but yahoo.com doesn't? > > That kinda nukes the whole match any bad host with domain ;) Thank you can lend a hand Seth? Thanks. > > James > > On 2018-01-18 10:33, Azoff, Justin S wrote: On Jan 18, 2018, at 11:42 AM, James Lay wrote: > > ... > yahoo.com Intel::DOMAIN testdomain > ... > 1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 53 udp 3285 0.003246 www.yahoo.com [1] 1 C_INTERNET 1 A 0 NOERROR F F TT 0 atsv2-fp.wg1.b.yahoo.com,98.138.252.38,98.138.252.39,98.139.180.180,206.190.39.43 1320.000000,39.000000,39.000000,39.000000,39.000000 F > > and no intel.log. What's different here? Would love to know what I'm missing..thank you. > www.yahoo.com [1] is not yahoo.com > > You need an intel::seen even that uses > https://github.com/sethhall/domain-tld to get that to match. I > thought someone wrote a package that did this, but apparently not. > > -- > Justin Azoff _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro Links: ------ [1] http://www.yahoo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180118/2fc16a25/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: 2018-01-18 10_35_16-fetchtrace.txt - Visual Studio Code.png Type: image/png Size: 32839 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180118/2fc16a25/attachment-0001.bin From jazoff at illinois.edu Thu Jan 18 10:13:11 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 18 Jan 2018 18:13:11 +0000 Subject: [Bro] A little more confusion with Intel In-Reply-To: References: <60a34e5669db124ce69305bc1232ab3c@localhost> Message-ID: <6D6D313F-3261-4590-AE60-7DFAB4CAD2A6@illinois.edu> > On Jan 18, 2018, at 1:06 PM, James Lay wrote: > > Here too, is there something I'm missing? In testing a different packet captures using TCP, I get intel...so does the Intel framework not support UDP? Thank you. > > James > The intel framework doesn't know anything about tcp or udp. The default scripts for connections only alert on tcp connections though: https://github.com/bro/bro/blob/master/scripts/policy/frameworks/intel/seen/conn-established.bro ? Justin Azoff From jlay at slave-tothe-box.net Thu Jan 18 10:15:27 2018 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 18 Jan 2018 11:15:27 -0700 Subject: [Bro] A little more confusion with Intel In-Reply-To: <6D6D313F-3261-4590-AE60-7DFAB4CAD2A6@illinois.edu> References: <60a34e5669db124ce69305bc1232ab3c@localhost> <6D6D313F-3261-4590-AE60-7DFAB4CAD2A6@illinois.edu> Message-ID: <64835b3c3307bcc75c2659f82473e516@localhost> Ah....Ok thanks again Justin. Seth should I put in a feature request for both TLD and UDP for the Intel framework? Thanks. James On 2018-01-18 11:13, Azoff, Justin S wrote: >> On Jan 18, 2018, at 1:06 PM, James Lay >> wrote: >> >> Here too, is there something I'm missing? In testing a different >> packet captures using TCP, I get intel...so does the Intel framework >> not support UDP? Thank you. >> >> James >> > > The intel framework doesn't know anything about tcp or udp. The > default scripts for connections only alert on tcp connections though: > > https://github.com/bro/bro/blob/master/scripts/policy/frameworks/intel/seen/conn-established.bro > > ? > Justin Azoff From jan.grashoefer at gmail.com Thu Jan 18 10:31:17 2018 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Thu, 18 Jan 2018 19:31:17 +0100 Subject: [Bro] A little more confusion with Intel In-Reply-To: <64835b3c3307bcc75c2659f82473e516@localhost> References: <60a34e5669db124ce69305bc1232ab3c@localhost> <6D6D313F-3261-4590-AE60-7DFAB4CAD2A6@illinois.edu> <64835b3c3307bcc75c2659f82473e516@localhost> Message-ID: <0f56fb05-f2cd-cd6b-2dc1-27ee078efea5@gmail.com> On 18/01/18 19:15, James Lay wrote: > Ah....Ok thanks again Justin. Seth should I put in a feature request > for both TLD and UDP for the Intel framework? Thanks. That's probably something that can be addressed with a package. In general you can have a look at https://github.com/bro/bro/tree/master/scripts/policy/frameworks/intel/seen to get an idea of how the intel framework gathers its information. Jan From shirkdog.bsd at gmail.com Thu Jan 18 10:45:29 2018 From: shirkdog.bsd at gmail.com (Michael Shirk) Date: Thu, 18 Jan 2018 13:45:29 -0500 Subject: [Bro] A little more confusion with Intel In-Reply-To: References: <60a34e5669db124ce69305bc1232ab3c@localhost> <6D6D313F-3261-4590-AE60-7DFAB4CAD2A6@illinois.edu> <64835b3c3307bcc75c2659f82473e516@localhost> <0f56fb05-f2cd-cd6b-2dc1-27ee078efea5@gmail.com> Message-ID: Yes this would be a nice to have. -- Michael Shirk Daemon Security, Inc. https://www.daemon-security.com On Jan 18, 2018 13:37, "Jan Grash?fer" wrote: On 18/01/18 19:15, James Lay wrote: > Ah....Ok thanks again Justin. Seth should I put in a feature request > for both TLD and UDP for the Intel framework? Thanks. That's probably something that can be addressed with a package. In general you can have a look at https://github.com/bro/bro/tree/master/scripts/policy/frameworks/intel/seen to get an idea of how the intel framework gathers its information. Jan _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180118/5dd0c31f/attachment.html From jan.grashoefer at gmail.com Thu Jan 18 12:48:58 2018 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Thu, 18 Jan 2018 21:48:58 +0100 Subject: [Bro] A little more confusion with Intel In-Reply-To: References: <60a34e5669db124ce69305bc1232ab3c@localhost> <6D6D313F-3261-4590-AE60-7DFAB4CAD2A6@illinois.edu> <64835b3c3307bcc75c2659f82473e516@localhost> <0f56fb05-f2cd-cd6b-2dc1-27ee078efea5@gmail.com> Message-ID: <89283fe8-4df2-9d87-dbea-b148f1d5eb13@gmail.com> > Yes this would be a nice to have. I put together a POC for effective TLDs but haven't tested deploying. During the weekend I should be able to polish it a bit. If someone already wants to give it a try: bro-pkg install https://github.com/J-Gras/intel-seen-more Jan From jazoff at illinois.edu Thu Jan 18 13:55:42 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 18 Jan 2018 21:55:42 +0000 Subject: [Bro] A little more confusion with Intel In-Reply-To: <89283fe8-4df2-9d87-dbea-b148f1d5eb13@gmail.com> References: <60a34e5669db124ce69305bc1232ab3c@localhost> <6D6D313F-3261-4590-AE60-7DFAB4CAD2A6@illinois.edu> <64835b3c3307bcc75c2659f82473e516@localhost> <0f56fb05-f2cd-cd6b-2dc1-27ee078efea5@gmail.com> <89283fe8-4df2-9d87-dbea-b148f1d5eb13@gmail.com> Message-ID: > On Jan 18, 2018, at 3:48 PM, Jan Grash?fer wrote: > >> Yes this would be a nice to have. > > I put together a POC for effective TLDs but haven't tested deploying. > During the weekend I should be able to polish it a bit. If someone > already wants to give it a try: > bro-pkg install https://github.com/J-Gras/intel-seen-more > > Jan That looks just like what I had in mind.. It makes sense that the type would be different, but I could see some people expecting it to just use the normal Intel::DOMAIN so existing feeds match. The more I think about this, there's also the similar calls to seen() for via HTTP::IN_HOST_HEADER, SSL::IN_SERVER_NAME, and X509::IN_CERT Maybe the intel framework itself needs to have an option to use the effective TLD when looking up Intel::DOMAINs inside of seen() ? Justin Azoff From seth at corelight.com Fri Jan 19 06:44:57 2018 From: seth at corelight.com (Seth Hall) Date: Fri, 19 Jan 2018 09:44:57 -0500 Subject: [Bro] A little more confusion with Intel In-Reply-To: <60a34e5669db124ce69305bc1232ab3c@localhost> References: <60a34e5669db124ce69305bc1232ab3c@localhost> Message-ID: <309A727C-26D8-4D26-975D-8FDD5158A82E@corelight.com> Hi James! :) Right now the Intel framework is only for doing complete strings matches for all of the string types (which Intel::DOMAIN is) so you don't get the substring matching like you want. Robin and I talked about this a couple of years ago as something that we wanted to address in Bro and Robin did a small prototype of a library that would make it possible by globbing. The idea was that you'd be able to have intelligence items that looked like this.. "*yahoo.com" or "www.*.yahoo.*". The initial implementation didn't perform acceptably well and we haven't had time to get back to that work yet. Right now if you are interested in looking for "www.yahoo.com" you will have to insert that specifically as an intelligence item. I'm not sure that the example you've given is something that people encounter in typical operational usage (although if I'm wrong, someone please let me know!). .Seth On 18 Jan 2018, at 12:42, James Lay wrote: > Wait what? So....it looks like www.yahoo.com [1] matches, but > yahoo.com > doesn't? > > That kinda nukes the whole match any bad host with domain ;) Thank > you > can lend a hand Seth? Thanks. > > James > > On 2018-01-18 10:33, Azoff, Justin S wrote: > >>> On Jan 18, 2018, at 11:42 AM, James Lay >>> wrote: >>> >>> ... >> >>> yahoo.com Intel::DOMAIN testdomain >>> ... >> >>> 1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 >>> 53 udp 3285 0.003246 www.yahoo.com [1] 1 C_INTERNET 1 A 0 NOERROR F >>> F TT 0 >>> atsv2-fp.wg1.b.yahoo.com,98.138.252.38,98.138.252.39,98.139.180.180,206.190.39.43 >>> 1320.000000,39.000000,39.000000,39.000000,39.000000 F >>> >>> and no intel.log. What's different here? Would love to know what >>> I'm missing..thank you. >> >> www.yahoo.com [1] is not yahoo.com >> >> You need an intel::seen even that uses >> https://github.com/sethhall/domain-tld to get that to match. I >> thought someone wrote a package that did this, but apparently not. >> >> -- >> Justin Azoff > > > Links: > ------ > [1] http://www.yahoo.com > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * www.corelight.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180119/7b2d888c/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: 2018-01-18 10_35_16-fetchtrace.txt - Visual Studio Code.png Type: image/png Size: 32839 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180119/7b2d888c/attachment-0001.bin From jan.grashoefer at gmail.com Fri Jan 19 07:26:02 2018 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Fri, 19 Jan 2018 16:26:02 +0100 Subject: [Bro] A little more confusion with Intel In-Reply-To: References: <60a34e5669db124ce69305bc1232ab3c@localhost> <6D6D313F-3261-4590-AE60-7DFAB4CAD2A6@illinois.edu> <64835b3c3307bcc75c2659f82473e516@localhost> <0f56fb05-f2cd-cd6b-2dc1-27ee078efea5@gmail.com> <89283fe8-4df2-9d87-dbea-b148f1d5eb13@gmail.com> Message-ID: <8677c462-9e74-a178-6edc-228d88259b36@gmail.com> On 18/01/18 22:55, Azoff, Justin S wrote: > It makes sense that the type would be different, but I could see some people expecting it to just use the normal Intel::DOMAIN so > existing feeds match. While that's certainly true, a couple of people might already rely on Intel::DOMAIN matching the complete domain. > The more I think about this, there's also the similar calls to seen() for via HTTP::IN_HOST_HEADER, SSL::IN_SERVER_NAME, and X509::IN_CERT Yep, I will just add corresponding scripts to the package. > Maybe the intel framework itself needs to have an option to use the effective TLD when looking up Intel::DOMAINs inside of seen() In that case the framework should report both: The effective and the complete domain. However, using a separate type would be more flexible as users could decide case by case or even add both. Given that the effective_domain function is already available as a package, I would vote for an additional package. In theory even the intel framework itself could be made a package. Jan From ben.mackcrane at corsa.com Fri Jan 19 08:09:25 2018 From: ben.mackcrane at corsa.com (Ben Mack-Crane) Date: Fri, 19 Jan 2018 10:09:25 -0600 Subject: [Bro] Advice on REST plugin structure Message-ID: Hi, I am a Bro newbie and, having run into an endless loop, I'm looking for advice. Background: I'm working on a NetControl plugin that will use a REST API to talk to the network device. I am trying to do this using ActiveHTTP (my web search having failed to find a REST plugin template). I am working from the Debug plugin and other examples I've found in the Bro 2.5.2 distribution. My endless loop: Using "when (local rest_resp = ActiveHTTP::request(rest_req))" in the plugin's init function leads to an endless loop (perhaps similar to type resolver loop mentioned in plugin.bro). I think the 'when' statement is involved since "WhenStmt::Exec" is in the early part of the (overflowing) stack. The loop starts with BroType::Serialize for NetControl::PluginState and returns there via BroType::Serialize for NetControl::Plugin after several steps. My plea: Can anyone offer advice about how I should structure my REST plugin to avoid this loop? Are there examples (templates) for the recommended structure? Regards, Ben Mack-Crane -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180119/76141d5f/attachment.html From vitaly.repin at gmail.com Sun Jan 21 22:50:33 2018 From: vitaly.repin at gmail.com (Vitaly Repin) Date: Mon, 22 Jan 2018 08:50:33 +0200 Subject: [Bro] User Agent parser in bro Message-ID: Hello, I am looking for a way to parse the User Agent string in bro. Is anybody aware of any bro scripts which are similar in functionality to something like ua-parser-js ( https://github.com/faisalman/ua-parser-js ) or ES user-agent ingest plugin ( https://www.elastic.co/guide/en/elasticsearch/plugins/master/ingest-user-agent.html )? Thanks in advance! -- WBR & WBW, Vitaly -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180122/2b37367d/attachment.html From vitaly.repin at gmail.com Sun Jan 21 22:57:08 2018 From: vitaly.repin at gmail.com (Vitaly Repin) Date: Mon, 22 Jan 2018 08:57:08 +0200 Subject: [Bro] User Agent parser in bro In-Reply-To: References: Message-ID: Hello, I have found this project: https://github.com/ua-parser/uap-core/blob/master/docs/specification.md It shall be possible to build bro ua-parser based on it. But I do not want to reinvent the wheel and prefer to use existing implementation if any... 2018-01-22 8:50 GMT+02:00 Vitaly Repin : > Hello, > > I am looking for a way to parse the User Agent string in bro. > > Is anybody aware of any bro scripts which are similar in functionality to > something like ua-parser-js ( https://github.com/faisalman/ua-parser-js ) > or ES user-agent ingest plugin ( https://www.elastic.co/guide/ > en/elasticsearch/plugins/master/ingest-user-agent.html )? > > Thanks in advance! > -- > WBR & WBW, Vitaly > -- WBR & WBW, Vitaly -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180122/d8c6f0ef/attachment.html From jan.grashoefer at gmail.com Mon Jan 22 02:32:38 2018 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Mon, 22 Jan 2018 11:32:38 +0100 Subject: [Bro] User Agent parser in bro In-Reply-To: References: Message-ID: On 22/01/18 07:57, Vitaly Repin wrote: > I have found this project: > https://github.com/ua-parser/uap-core/blob/master/docs/specification.md It > shall be possible to build bro ua-parser based on it. But I do not want to > reinvent the wheel and prefer to use existing implementation if any... The only thing I am aware of is the software log (see https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info). If you need more functionality, this would be a good candidate for a bro package. Jan From jlamps at sandia.gov Mon Jan 22 06:17:14 2018 From: jlamps at sandia.gov (Lamps, Jereme) Date: Mon, 22 Jan 2018 14:17:14 +0000 Subject: [Bro] Using the &persistent attribute Message-ID: I am a little confused as to how this attribute will work. Ideally I would like a state table that persists across numerous iterations of stopping/starting bro (broctl deploy). To get this to work do I simply need the &persistent attribute and Bro magically handles the rest? Or do I also need to use the functions checkpoint_state() (in bro_done) with rescan_state (in bro_init) to load the tables? Best, Jereme -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180122/575a1fa6/attachment.html From jlay at slave-tothe-box.net Mon Jan 22 08:30:54 2018 From: jlay at slave-tothe-box.net (James Lay) Date: Mon, 22 Jan 2018 09:30:54 -0700 Subject: [Bro] A little more confusion with Intel In-Reply-To: <309A727C-26D8-4D26-975D-8FDD5158A82E@corelight.com> References: <60a34e5669db124ce69305bc1232ab3c@localhost> <309A727C-26D8-4D26-975D-8FDD5158A82E@corelight.com> Message-ID: <3ccd76709d5eafabe321885b5acec101@localhost> Hi Seth, It's actually the inverse of what I'm seeing. In my tests if I have Intel::DOMAIN yahoo.com and I did a "dig www.yahoo.com", [2] the domain intel would not match because the dns request was for "www.yahoo.com", not yahoo.com. Does that make sense? Thank you. James On 2018-01-19 07:44, Seth Hall wrote: > Hi James! :) > > Right now the Intel framework is only for doing complete strings matches for all of the string types (which Intel::DOMAIN is) so you don't get the substring matching like you want. Robin and I talked about this a couple of years ago as something that we wanted to address in Bro and Robin did a small prototype of a library that would make it possible by globbing. The idea was that you'd be able to have intelligence items that looked like this.. "*yahoo.com" or "www.*.yahoo.*". The initial implementation didn't perform acceptably well and we haven't had time to get back to that work yet. > > Right now if you are interested in looking for "www.yahoo.com" you will have to insert that specifically as an intelligence item. I'm not sure that the example you've given is something that people encounter in typical operational usage (although if I'm wrong, someone please let me know!). > > .Seth > > On 18 Jan 2018, at 12:42, James Lay wrote: > > Wait what? So....it looks like www.yahoo.com [1] matches, but yahoo.com doesn't? > > That kinda nukes the whole match any bad host with domain ;) Thank you can lend a hand Seth? Thanks. > > James > > On 2018-01-18 10:33, Azoff, Justin S wrote: On Jan 18, 2018, at 11:42 AM, James Lay wrote: > > ... > yahoo.com Intel::DOMAIN testdomain > ... > 1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 53 udp 3285 0.003246 www.yahoo.com [1] 1 C_INTERNET 1 A 0 NOERROR F F TT 0 atsv2-fp.wg1.b.yahoo.com,98.138.252.38,98.138.252.39,98.139.180.180,206.190.39.43 1320.000000,39.000000,39.000000,39.000000,39.000000 F > > and no intel.log. What's different here? Would love to know what I'm missing..thank you. > www.yahoo.com [1] is not yahoo.com > > You need an intel::seen even that uses > https://github.com/sethhall/domain-tld to get that to match. I > thought someone wrote a package that did this, but apparently not. > > -- > Justin Azoff > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * www.corelight.com Links: ------ [1] http://www.yahoo.com [2] http://www.yahoo.com", -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180122/bf88d0cd/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: 2018-01-18 10_35_16-fetchtrace.txt - Visual Studio Code.png Type: image/png Size: 32839 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180122/bf88d0cd/attachment-0001.bin From robin at icir.org Tue Jan 23 19:34:40 2018 From: robin at icir.org (Robin Sommer) Date: Tue, 23 Jan 2018 22:34:40 -0500 Subject: [Bro] Using the &persistent attribute In-Reply-To: References: Message-ID: <20180124033440.GD47252@icir.org> On Mon, Jan 22, 2018 at 14:17 +0000, Lamps, Jereme wrote: > To get this to work do I simply > need the &persistent attribute and Bro magically handles the rest? Or > do I also need to use the functions checkpoint_state() (in bro_done) > with rescan_state (in bro_init) to load the tables? The former: at startup and termination checkpoint_state() and rescan_state are called automatically, respectively. You can in addition call them during operation for checkpointing/rereading the state on the fly. That said, please note that &persistent is going to go away, it will likely be deprecated starting with Bro 2.6. Broker's data stores are going to be the primary replacement. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From iangabriel0 at gmail.com Wed Jan 24 06:34:09 2018 From: iangabriel0 at gmail.com (Ian Gabriel) Date: Wed, 24 Jan 2018 09:34:09 -0500 Subject: [Bro] Bro Logging Error: count value too high for JSON Message-ID: I am having some trouble with a bro script we have. It is listening on the tcp_packet event and logging using the ASCII writer in JSON. As the subject indicates, I am getting the error as follows: error: conn/Log::WRITER_ASCII: count value too large for JSON: 184467440718605600000 >From the bro manual I understand that the `count` data type is an unsigned 64 bit int, while `int` is a signed 64bit int. From bro's git and my error message, I understand that we cannot print values to JSON larger than the signed int max. With my bro script, I printed out the `count` data types passed to me in the `tcp_packet` hook (being SEQ LEN ACK), and noticed that my SEQ numbers were the values that bro was having trouble serializing properly as they were bigger than the signed int maximum. This raised the eyebrows of a team member smarter than myself, as he reminded me that SEQ numbers are 32 bits in length in TCP packets. After changing the datatypes of the structs I am logging to `int` and "downcasting" the `count` values, I no longer run into this problem... but then I also get negative sequence numbers in my result : ) I wonder: A) Am I doing something wrong? B) There seems to be a related issue on the issue tracker: https://bro-tracker.atlassian.net/browse/BIT-1863 , but I am thinking there might be some intricacies with how bro generates sequence numbers for a given packet/pcap? Bro is passing these values directly to the tcp_event hook, and I am doing no manipulation before printing out these too large sequence numbers, which is why I am not attaching my broscript. Thanks in advance for your time, Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180124/3df53fc1/attachment.html From koshybibin3 at gmail.com Wed Jan 24 07:06:05 2018 From: koshybibin3 at gmail.com (Bibin Koshy) Date: Wed, 24 Jan 2018 15:06:05 +0000 Subject: [Bro] bro unrecognized character error! Message-ID: Hi, I am trying to run bro in my bash terminal. I have a "local.bro" file in a different directory so its not in default path. I am just trying to do a simple signature match, therefore i have created a "signature.sig" file in the directory. In my local.bro file i have tried using both ways "@load-sigs ./signature" and "redef signature_files += "signature.sig". The signature.sig file has the "signature my-first-sig" example from bro.org site. In my terminal when i try to execute this: bro -r traffic.pcap local i get an error message saying "line 27: unrecognized character -" and i have also tried doing bro -r traffic.pcap -s signature.sig which also give me the same unrecognized character error. Am i doing something wrong, please can you guide me to a solution ? Thank you Bibin K Bro Network Analyst (Beginner) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180124/6408bf33/attachment.html From dnj0496 at gmail.com Wed Jan 24 17:18:00 2018 From: dnj0496 at gmail.com (Dk Jack) Date: Wed, 24 Jan 2018 17:18:00 -0800 Subject: [Bro] conn. uid Message-ID: Hi, I am trying to include the uid that's shown in conn.log in the log messages I generate from my plugin. I want to do this so that I can correlate my log messages to the other log lines generated in the other logs. After looking into the bro code a little, I came up with the following based on EncapsulatingConn::EncapsulatingConn (src/TunnelEncapsulation.cc): Bro::UID uid = c->GetUID(); if (!uid) { uid.Set(bits_per_uid); c->SetUID(uid); uid = c->GetUID(); } std::string uid_str = uid.Base62("C"); My plugin is based on tcp::TCP_ApplicationAnalyzer 'c' is of type 'Connection'. Things seem to be working ok. I am getting a uid that looks similar to what I see in conn.log. However, there is one thing that's a bit puzzling though. Not all the UIDs that show up in my log are present in the conn.log. What could be the reason for this? Would appreciate any insight into this. Thanks. Dk. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180124/6c358f93/attachment.html From email4myth at gmail.com Thu Jan 25 08:18:07 2018 From: email4myth at gmail.com (Myth Ren) Date: Fri, 26 Jan 2018 00:18:07 +0800 Subject: [Bro] [BRO-ISSUE]: bro crash when so many Repoter::Error calls Message-ID: Hi all, I'm using bro 2.5.1 for network security monitoring , the message queue is kafka componment (the bro-to-kafka plugin version is v0.5.0, librdkafka version is v0.9.5). Now i have encountered an error when network traffic up to 1.6Gbps, the error message is segment fault from `src/Event.cc#90`, bro crashed. The following listed is our test environment informations: > CPU: 32 core Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz Memory: 64G NIC: Speed 10000Mb/s Storage: 2TB SATA, 100GB SSD Below listed information is backtrace from core dump. (more on gist ) > #0 SetNext (this=0x0, n=0x7fe292ebd490) at > /opt/download/bro/src/Event.h:21 #1 EventMgr::QueueEvent (this=0xc302c0 > , event=event at entry=0x7fe292ebd490) at > /opt/download/bro/src/Event.cc:90 #2 0x00000000005fe6a7 in QueueEvent > (obj=0x0, mgr=0x0, aid=0, src=0, vl=0x7fe2e2bedb80, h=..., this= out>) at /opt/download/bro/src/Event.h:88 #3 Reporter::DoLog > (this=0x29aabb0, prefix=prefix at entry=0x908cd7 "error", event=..., > out=0x0, conn=conn at entry=0x0, addl=addl at entry=0x0, location=location at entry=true, > time=time at entry=true, postfix=postfix at entry=0x0, fmt=fmt at entry=0x7fe36c719d70 > "Kafka send failed: %s", ap=ap at entry=0x7fe36aa3eaf8) at > /opt/download/bro/src/Reporter.cc:350 #4 0x00000000005fee8f in > Reporter::Error (this=, fmt=fmt at entry=0x7fe36c719d70 > "Kafka send failed: %s") at /opt/download/bro/src/Reporter.cc:76 #5 > 0x00007fe36c717fa9 in logging::writer::KafkaWriter::DoWrite > (this=0x6369270, num_fields=, fields=, > vals=0x69d2080) at > /opt/download/bro/aux/plugins/kafka/src/KafkaWriter.cc:156 #6 > 0x000000000089e495 in logging::WriterBackend::Write (this=0x6369270, > arg_num_fields=, num_writes=1000, vals=0x6dc7bf0) at > /opt/download/bro/src/logging/WriterBackend.cc:301 #7 0x0000000000662180 > in threading::MsgThread::Run (this=0x6369270) at > /opt/download/bro/src/threading/MsgThread.cc:371 #8 0x000000000065eaa8 in > threading::BasicThread::launcher (arg=0x6369270) at > /opt/download/bro/src/threading/BasicThread.cc:205 #9 0x00007fe36e8ce2b0 > in ?? () from /lib64/libstdc++.so.6 #10 0x00007fe36ed2ce25 in start_thread > () from /lib64/libpthread.so.0 #11 0x00007fe36e03634d in clone () from > /lib64/libc.so.6 Varibles on frame 1 (gdb) f 1 #1 EventMgr::QueueEvent (this=0xc302c0 , event=event at entry=0x7fe292ebd490) at /opt/download/bro/src/Event.cc:90 90 tail->SetNext(event); (gdb) info args this = 0xc302c0 event = 0x7fe292ebd490 (gdb) info locals done = (gdb) p head $1 = (Event *) 0x7fe3540c81c0 (gdb) p tail $2 = (Event *) 0x0 (gdb) p event > > $3 = (Event *) 0x7fe292ebd490 During test, the variable `tail` is NULL pointer always when bro crashed, however the variable `head` is NULL or not. on my research, in the huge network traffic scenario, KakfaWriter write log to kafka exceed the limit of configuration `queue.buffering.max.messages(default is 100000)` or `queue.buffering.max.kbytes(default is 4000000, 4G in other words)` in librdkafka, and QUEUE_FULL error raised by librdkafka, then KafkaWriter call Reporter::Error to report the runtime error. so KafkaWriter::DoWrite lead too many action to call Reporter::Error function. I guess the issue cause with concurrency access to the varible `tail` without lock, then it set to be a NULL pointer, but i don't know why. Then call the function `SetNext` with the NULL pointer, segment fault was raised. The above is my guesswork, maybe there is another reason. Wish someone could help. Best regards, Myth -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180126/24c53273/attachment.html From jazoff at illinois.edu Thu Jan 25 08:46:17 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 25 Jan 2018 16:46:17 +0000 Subject: [Bro] [BRO-ISSUE]: bro crash when so many Repoter::Error calls In-Reply-To: References: Message-ID: <56CD4701-71F3-46C7-8B04-41DD7A1F3D9A@illinois.edu> > On Jan 25, 2018, at 11:18 AM, Myth Ren wrote: > > then KafkaWriter call Reporter::Error to report the runtime error. This would be a problem if bro is configured to send the reporter.log to Kafka. Reporter::Error generates a reporter_error event which then calls Log::write(Reporter::LOG, [$ts=t, $level=ERROR, $message=msg, $location=location]); So you're probably also ending up in the situation where bro is trying to log to Kafka the fact that Kafka is broken. If you tell bro to not send the reporter.log to Kafka, does your problem go away? ? Justin Azoff From koshybibin3 at gmail.com Thu Jan 25 15:14:02 2018 From: koshybibin3 at gmail.com (Bibin Koshy) Date: Thu, 25 Jan 2018 23:14:02 +0000 Subject: [Bro] Fwd: bro unrecognized character error help In-Reply-To: References: <20180125182217.08C212C4129@rock.ICSI.Berkeley.EDU> Message-ID: ---------- Forwarded message ---------- From: Bibin Koshy Date: 25 January 2018 at 23:08 Subject: Re: bro unrecognized character error help To: Vern Paxson Hi sorry i forgot to attach the localv2.bro and signature.sig file Thank you Bibin On 25 January 2018 at 23:07, Bibin Koshy wrote: > Hi, > > Thank you for replying back. I thought i have posted this question on the > mailing list. I have attached the localv2.bro file which is the duplicate > local.bro file. i am using the localv2.bro file in my working directory so > not the default path and the signature.sig file is also in the same > directory which is also attached. I have included the signature file on > line 37 in localv2.bro. I have attached two jpg image.One is on the > directory and the files i am using, another is on the error i get using the > localv2.bro and also using the alternative of using -s signature. > > Thank you > Bibin > > On 25 January 2018 at 18:22, Vern Paxson wrote: > >> Send questions like this to the Bro mailing list, >> per https://www.bro.org/community/index.html and >> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro . >> You should send along your local.bro and signature.sig. >> >> Vern >> > > > > -- > *Bibin* > -- *Bibin* -- *Bibin* -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180125/4f271c71/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: localv2.bro Type: application/octet-stream Size: 3840 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180125/4f271c71/attachment.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.sig Type: application/octet-stream Size: 116 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180125/4f271c71/attachment-0001.obj From johanna at icir.org Fri Jan 26 08:59:00 2018 From: johanna at icir.org (Johanna Amann) Date: Fri, 26 Jan 2018 08:59:00 -0800 Subject: [Bro] Exam In-Reply-To: References: Message-ID: <20180126165900.szezxp6fyrbgmeyb@user237.sys.ICSI.Berkeley.EDU> Hi, this might be too late - but if you look on our documentation page, you find a list of all the teaching and training material that we have (https://www.bro.org/playground/index.html). The best way to start is probably http://try.bro.org. Johanna On Sat, Dec 16, 2017 at 12:03:22AM +0100, Neda Danilovic wrote: > Hello, > > I need to prepare for my exam Resilient Networks and I have a few questions > about Bro. Do you have smething like lectures? I would gladly pay for that, > that someone help me to prepare Bro for exam. > > Regards, > Neda Danilovi? > *www.kaficamagazin.rs * > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jazoff at illinois.edu Fri Jan 26 09:17:59 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 26 Jan 2018 17:17:59 +0000 Subject: [Bro] bro unrecognized character error help In-Reply-To: References: <20180125182217.08C212C4129@rock.ICSI.Berkeley.EDU> Message-ID: <9E680441-4CE5-4DCB-86F4-90CEDE0DF667@illinois.edu> Your files have dos line endings. Convert them to unix using dos2unix or whatever. ? Justin Azoff > On Jan 25, 2018, at 6:14 PM, Bibin Koshy wrote: > > > ---------- Forwarded message ---------- > From: Bibin Koshy > Date: 25 January 2018 at 23:08 > Subject: Re: bro unrecognized character error help > To: Vern Paxson > > > Hi > > sorry i forgot to attach the localv2.bro and signature.sig file > > Thank you > Bibin > > On 25 January 2018 at 23:07, Bibin Koshy wrote: > Hi, > > Thank you for replying back. I thought i have posted this question on the mailing list. I have attached the localv2.bro file which is the duplicate local.bro file. i am using the localv2.bro file in my working directory so not the default path and the signature.sig file is also in the same directory which is also attached. I have included the signature file on line 37 in localv2.bro. I have attached two jpg image.One is on the directory and the files i am using, another is on the error i get using the localv2.bro and also using the alternative of using -s signature. > > Thank you > Bibin > > On 25 January 2018 at 18:22, Vern Paxson wrote: > Send questions like this to the Bro mailing list, > per https://www.bro.org/community/index.html and > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro . > You should send along your local.bro and signature.sig. > > Vern > > > > -- > Bibin > > > > -- > Bibin > > > > -- > Bibin > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Fri Jan 26 09:41:08 2018 From: johanna at icir.org (Johanna Amann) Date: Fri, 26 Jan 2018 09:41:08 -0800 Subject: [Bro] How to get generated specific log files under DEFAULT path (e.g. notice.log) In-Reply-To: <3805979d-624b-1522-294f-88a51ed15bbf@gmx.de> References: <45a7a45d-3a81-639c-600a-fdd3877cda3f@gmx.de> <3805979d-624b-1522-294f-88a51ed15bbf@gmx.de> Message-ID: <20180126174108.gj7moywiskgc3wuh@user237.sys.ICSI.Berkeley.EDU> Hi, did you randomly already find a solution for this in the meantime? Just from reading over this, I am a bit at a loss for why this would happen. Basically - the on-disk-files are created the first time that something is written into then. So - it is normal that a notice.log might not show up immediately. However, it should show up after you call NOTICE. It is a bit weird that it would work after you put a full path into notice/main - I am not sure why this would change anything. How exactly did you try to generate your test notice? Note that if you are using cluster mode, a NOTICE in bro_init() will probably not show up in notice.log due to the fact that the logging connections are not set up yet when it is raised. Also - what Bro version are you using? Johanna On Sat, Dec 16, 2017 at 08:59:30AM +0100, Zick Zack wrote: > Hi Bro'ers > > I have a problem to get generated a notice.log file with it's DEFAULT path. > > Short description of my problem: > > * whenever I start Bro to do sth., I get generated some log-files > (e.g. communication, http, ...) in a folder named /var/log/bro > * however (also after a "deploy" command!), when I call e.g. > "NOTICE([$note=***, $msg="***"])", I get NOT generated a notice.log > file ANYWHERE on my VM > * I can somehow circumvent that by manipulating the > share/bro/base/frameworks/notice/main.bro file, when I explicitly > set the $path variables in there to my absolute path like > "/var/log/bro/notice" > > Some background I already found out: > > * it is said in the Bro documentation NOT to change any files in the > directories (and its sub-folders) from share/bro EXCEPT the > share/bro/site-folder > * I found out, all the modules for which the DEFAULT path log-file > generation is working somehow load (directly or indirectly) the > base/utils/paths or the base/utils/site modules > > What I want: > > * getting generated my notice.log file without specifiying an absolute > path; only the file-name (just like as it works for the other log > files in my /var/log/bro folder) > > Please help me to get my notice.log file WITHOUT manipulating files > which one should not touch! > > Thanks alot in advance! > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Fri Jan 26 09:50:48 2018 From: johanna at icir.org (Johanna Amann) Date: Fri, 26 Jan 2018 09:50:48 -0800 Subject: [Bro] Calling external functions in binpac protocol parser In-Reply-To: References: <20171228203815.7iwbvz4z36a7xczn@Beezling.fritz.box> Message-ID: <20180126175048.bfblsojvk5brmqmo@user237.sys.ICSI.Berkeley.EDU> Hi again, > > you can e.g. move it into a separate > > class that is accessible by everyone. Depending on the things that your > > function does, it even might be possible to make it a static function. > > > > Yes, as you mentioned, we do want a module that can be accessed and called > by every protocol analyzer. Regarding the "separate class that is > accessible by everyone", > where we should implement this module? Do we need put the C++ files in > bro/src? Thinking about it again - this actually is a good question. It would be neat if you could add functionality to your plugins without having to modify the base Bro installation (which is what you basically do when you move things to /bro/src and include them into the compile process there - which certainly is a theoretical possibility). But - it would be nicer if that was not necessary. So I think my first idea would be to just put the shared sources into a directory that is accessible by all your plugins, and compile it together with a plugin (in separate namespaces so that it does not conflict afterwards). This should be fine assuming that your shared functionality is not ultra huge.. > > Also note that you probably should not put your plugins into > > bro-aux/plugin-support in the first case. Having them in a separate > > directory is probably preferable - completely outside of the Bro source > > tree. > > > > Sorry I don't get the point of this part. In my understanding, wherever we > develop > the plugins, it will be complied into bro's library /bro/lib/.../plugin. > What do you mean > (and how) "outside of the Bro source tree"? Yes, your plugins will reside inside the Bro installation in the end. However you probably should keep your sourecode separate from the main Bro sourcecode - that's all what I meant :). Johanna From jsiwek at corelight.com Fri Jan 26 10:42:38 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Fri, 26 Jan 2018 12:42:38 -0600 Subject: [Bro] conn. uid In-Reply-To: References: Message-ID: On Wed, Jan 24, 2018 at 7:18 PM, Dk Jack wrote: > Not all the UIDs that show up in my log are present in the > conn.log. What could be the reason for this? If you were watching logs in real time, it could be that an entry just has not been written to conn.log yet since those are generated when connections end or are inactive for too long (5 mins is Bro's default timeout for TCP). Else, I'd try isolating an example pcap where you have something logged in your custom log but not in conn.log then stepping through with a debugger to find out what happens to the connections that are missing from conn.log. And if you can provide such a pcap and a minimal example plugin that shows the behavior, I can also help take a look. - Jon From dnj0496 at gmail.com Fri Jan 26 10:47:46 2018 From: dnj0496 at gmail.com (Dk Jack) Date: Fri, 26 Jan 2018 10:47:46 -0800 Subject: [Bro] conn. uid In-Reply-To: References: Message-ID: Hi Jon, Thanks for your insight. I think you and Mark are correct. I haven?t seen this when I use a pcap. I?ll continue to monitor. Thanks again. > On Jan 26, 2018, at 10:42 AM, Jon Siwek wrote: > >> On Wed, Jan 24, 2018 at 7:18 PM, Dk Jack wrote: >> >> Not all the UIDs that show up in my log are present in the >> conn.log. What could be the reason for this? > > If you were watching logs in real time, it could be that an entry just > has not been written to conn.log yet since those are generated when > connections end or are inactive for too long (5 mins is Bro's default > timeout for TCP). > > Else, I'd try isolating an example pcap where you have something > logged in your custom log but not in conn.log then stepping through > with a debugger to find out what happens to the connections that are > missing from conn.log. And if you can provide such a pcap and a > minimal example plugin that shows the behavior, I can also help take a > look. > > - Jon From jsiwek at corelight.com Fri Jan 26 11:27:51 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Fri, 26 Jan 2018 13:27:51 -0600 Subject: [Bro] Bro Logging Error: count value too high for JSON In-Reply-To: References: Message-ID: On Wed, Jan 24, 2018 at 8:34 AM, Ian Gabriel wrote: > I printed out the `count` data types > passed to me in the `tcp_packet` hook (being SEQ LEN ACK), and noticed that > my SEQ numbers were the values that bro was having trouble serializing > properly as they were bigger than the signed int maximum. This raised the > eyebrows of a team member smarter than myself, as he reminded me that SEQ > numbers are 32 bits in length in TCP packets. The sequence numbers in that event are actually "relative" to the starting sequence number. Bro internally tracks how many times the 32-bit sequence space may have wrapped around and so expands any given sequence number in a packet out into where it would be located in a larger 64-bit space before passing that value along into the `tcp_packet` event. > After changing the datatypes of the structs I am logging to `int` and > "downcasting" the `count` values, I no longer run into this problem... but > then I also get negative sequence numbers in my result : ) > > I wonder: > > A) Am I doing something wrong? Maybe it's partly yes, partly no :) I think one way you might want to get around the JSON limitation in this case would be logging two different numbers based on the result of the sequence number modulo INT64_MAX. E.g. if you have both the quotient and remainder of that, then you can generate back the full, relative sequence number in the 64-bit sequence space. > error: conn/Log::WRITER_ASCII: count value too large for JSON: > 184467440718605600000 Though in this case, it looks close to the upper bound for an unsigned 64-bit integer and I doubt a sequence number actually wrapped around the 32-bit sequence space enough times to get that high. A guess would be that Bro has gotten the starting sequence number wrong and so incorrectly wrapped the full 64-bit space going backwards. If you can provide an example pcap that produces sequence numbers like that, it would be interesting to take a look and see if something needs to be fixed/improved in Bro. - Jon From koshybibin3 at gmail.com Fri Jan 26 13:16:30 2018 From: koshybibin3 at gmail.com (Bibin Koshy) Date: Fri, 26 Jan 2018 21:16:30 +0000 Subject: [Bro] bro unrecognized character error help In-Reply-To: <9E680441-4CE5-4DCB-86F4-90CEDE0DF667@illinois.edu> References: <20180125182217.08C212C4129@rock.ICSI.Berkeley.EDU> <9E680441-4CE5-4DCB-86F4-90CEDE0DF667@illinois.edu> Message-ID: Thank you so much Justin! This has done the trick! :) On 26 January 2018 at 17:17, Azoff, Justin S wrote: > Your files have dos line endings. Convert them to unix using dos2unix or > whatever. > > ? > Justin Azoff > > > > On Jan 25, 2018, at 6:14 PM, Bibin Koshy wrote: > > > > > > ---------- Forwarded message ---------- > > From: Bibin Koshy > > Date: 25 January 2018 at 23:08 > > Subject: Re: bro unrecognized character error help > > To: Vern Paxson > > > > > > Hi > > > > sorry i forgot to attach the localv2.bro and signature.sig file > > > > Thank you > > Bibin > > > > On 25 January 2018 at 23:07, Bibin Koshy wrote: > > Hi, > > > > Thank you for replying back. I thought i have posted this question on > the mailing list. I have attached the localv2.bro file which is the > duplicate local.bro file. i am using the localv2.bro file in my working > directory so not the default path and the signature.sig file is also in the > same directory which is also attached. I have included the signature file > on line 37 in localv2.bro. I have attached two jpg image.One is on the > directory and the files i am using, another is on the error i get using the > localv2.bro and also using the alternative of using -s signature. > > > > Thank you > > Bibin > > > > On 25 January 2018 at 18:22, Vern Paxson wrote: > > Send questions like this to the Bro mailing list, > > per https://www.bro.org/community/index.html and > > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro . > > You should send along your local.bro and signature.sig. > > > > Vern > > > > > > > > -- > > Bibin > > > > > > > > -- > > Bibin > > > > > > > > -- > > Bibin > > ________________________________ > _______________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- *Bibin* -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180126/0793807d/attachment.html From krasinski at cines.fr Mon Jan 29 02:45:17 2018 From: krasinski at cines.fr (Nicolas KRASINSKI) Date: Mon, 29 Jan 2018 11:45:17 +0100 (CET) Subject: [Bro] arp_main.bro : "Reporter::ERROR no such index" Message-ID: <32009600.710678.1517222717490.JavaMail.zimbra@cines.fr> Hello, I try to use the script arp_main.bro, that I found here : https://gist.github.com/grigorescu/a28b814a8fb626e2a7b4715d278198aa I can load this script without error, but I have an error in my reporter.log : Reporter::ERROR no such index (ARP::arp_states[ARP::SHA]) /usr/local/bro/spool/installed-scripts-do-not-touch/site/arp_main.bro, line 186 In the script, the error is on the line 186 : 181 event arp_request(mac_src: string, mac_dst: string, SPA: addr, SHA: string, TPA: addr, THA: string) 182 { 183 mac_addr_association(SHA, SPA); 184 185 local arp_state: State; 186 arp_state = arp_states[SHA]; I don't understand how to solve this problem, could you help me ? Thanks in advance, Nicolas -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180129/ec426fc7/attachment.html From seth at corelight.com Mon Jan 29 08:11:48 2018 From: seth at corelight.com (Seth Hall) Date: Mon, 29 Jan 2018 11:11:48 -0500 Subject: [Bro] User Agent parser in bro In-Reply-To: References: Message-ID: Hi Vitaly! I've been wanting to port one of these type of things to Bro for a long time. That would be a great contribution if you wanted to take that on. I'm sure that a number of people would find it valuable. I don't know of anyone in the community that has already done it. .Seth On 22 Jan 2018, at 1:50, Vitaly Repin wrote: > Hello, > > I am looking for a way to parse the User Agent string in bro. > > Is anybody aware of any bro scripts which are similar in functionality > to > something like ua-parser-js ( > https://github.com/faisalman/ua-parser-js ) > or ES user-agent ingest plugin ( > https://www.elastic.co/guide/en/elasticsearch/plugins/master/ingest-user-agent.html > )? > > Thanks in advance! > -- > WBR & WBW, Vitaly > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * www.corelight.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180129/06323698/attachment.html From seth at corelight.com Mon Jan 29 08:14:33 2018 From: seth at corelight.com (Seth Hall) Date: Mon, 29 Jan 2018 11:14:33 -0500 Subject: [Bro] A little more confusion with Intel In-Reply-To: <3ccd76709d5eafabe321885b5acec101@localhost> References: <60a34e5669db124ce69305bc1232ab3c@localhost> <309A727C-26D8-4D26-975D-8FDD5158A82E@corelight.com> <3ccd76709d5eafabe321885b5acec101@localhost> Message-ID: <3A565BA7-E885-420B-BB4A-3689FFE9150F@corelight.com> On 22 Jan 2018, at 11:30, James Lay wrote: > It's actually the inverse of what I'm seeing.? In my tests if I have > Intel::DOMAIN yahoo.com and I did a > "dig?[www.yahoo.com",]()?the domain intel > would not match because the dns request was for "www.yahoo.com", not > yahoo.com.? Does that make sense?? Thank you. Yeah, if we had a more comprehensive matcher for the intel framework then you'd have a lot of options open for you. I suppose that my main point was that at the moment you will have to just include the exact domain that you want to match on. Do you have a large list where you'd like to watch for any hits on the effective second level domain like you're describing here? .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From jsiwek at corelight.com Mon Jan 29 09:02:28 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Mon, 29 Jan 2018 11:02:28 -0600 Subject: [Bro] arp_main.bro : "Reporter::ERROR no such index" In-Reply-To: <32009600.710678.1517222717490.JavaMail.zimbra@cines.fr> References: <32009600.710678.1517222717490.JavaMail.zimbra@cines.fr> Message-ID: On Mon, Jan 29, 2018 at 4:45 AM, Nicolas KRASINSKI wrote: > I try to use the script arp_main.bro, that I found here : > https://gist.github.com/grigorescu/a28b814a8fb626e2a7b4715d278198aa > I can load this script without error, but I have an error in my reporter.log > : > > Reporter::ERROR no such index (ARP::arp_states[ARP::SHA]) > /usr/local/bro/spool/installed-scripts-do-not-touch/site/arp_main.bro, line > 186 > > In the script, the error is on the line 186 : > > 181 event arp_request(mac_src: string, mac_dst: string, SPA: addr, SHA: > string, TPA: addr, THA: string) > 182 { > 183 mac_addr_association(SHA, SPA); > 184 > 185 local arp_state: State; > 186 arp_state = arp_states[SHA]; > > I don't understand how to solve this problem, could you help me ? To get rid of that error, the code needs to be fixed/rewritten. It might be as simple as putting 'if ( SHA !in arp_states ) return;' after line 183 or it might not be, one would have to read/understand the code more closely to determine how to fix it. If you don't get more of a response in this list, you could try contacting the author of the script directly (their name/email is at the top of it) to see if they have any changes/updates, else you could try fixing the code yourself. - Jon From shanem at vt.edu Tue Jan 30 07:50:39 2018 From: shanem at vt.edu (Shane Mullins) Date: Tue, 30 Jan 2018 10:50:39 -0500 Subject: [Bro] Critical Stack intel feeds and cluster question Message-ID: <4fde39cc-ccf4-25d1-6e40-4184cf8420c4@vt.edu> Good morning everyone, Does anyone use the Critical Stack intel feeds in with a Bro cluster?? Or does anyone know if the Critical Stack client is supported in a cluster environment? Thanks Shane From liam.randall at gmail.com Tue Jan 30 08:47:45 2018 From: liam.randall at gmail.com (Liam Randall) Date: Tue, 30 Jan 2018 11:47:45 -0500 Subject: [Bro] Critical Stack intel feeds and cluster question In-Reply-To: <4fde39cc-ccf4-25d1-6e40-4184cf8420c4@vt.edu> References: <4fde39cc-ccf4-25d1-6e40-4184cf8420c4@vt.edu> Message-ID: It should work just fine on a cluster; just install it on the manager. The intel framework itself checks to see if it's running on a cluster and then distributes the intel accordingly. https://github.com/bro/bro/blob/master/scripts/base/frameworks/intel/input.bro Liam On Tue, Jan 30, 2018 at 10:50 AM, Shane Mullins wrote: > Good morning everyone, > > Does anyone use the Critical Stack intel feeds in with a Bro cluster? > Or does anyone know if the Critical Stack client is supported in a > cluster environment? > > Thanks > Shane > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180130/e0f1e291/attachment.html From dotwayland at gmail.com Tue Jan 30 11:41:35 2018 From: dotwayland at gmail.com (Wayland Morgan) Date: Tue, 30 Jan 2018 13:41:35 -0600 Subject: [Bro] Critical Stack intel feeds and cluster question In-Reply-To: References: <4fde39cc-ccf4-25d1-6e40-4184cf8420c4@vt.edu> Message-ID: I have it running on the manager in my home network. It was a painless set up. Wayland On Tue, Jan 30, 2018 at 10:47 AM, Liam Randall wrote: > It should work just fine on a cluster; just install it on the manager. > The intel framework itself checks to see if it's running on a cluster and > then distributes the intel accordingly. > > https://github.com/bro/bro/blob/master/scripts/base/ > frameworks/intel/input.bro > > Liam > > On Tue, Jan 30, 2018 at 10:50 AM, Shane Mullins wrote: > >> Good morning everyone, >> >> Does anyone use the Critical Stack intel feeds in with a Bro cluster? >> Or does anyone know if the Critical Stack client is supported in a >> cluster environment? >> >> Thanks >> Shane >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180130/d44e87d1/attachment.html From jlamps at sandia.gov Tue Jan 30 12:07:29 2018 From: jlamps at sandia.gov (Lamps, Jereme) Date: Tue, 30 Jan 2018 20:07:29 +0000 Subject: [Bro] Bro-2.5.2 and PF_RING 6.7 not load balancing properly Message-ID: It appears PF_RING is not properly load balancing between Bro instances. For example, I have a single Bro node with 5 bro procs. Every entry in http.log is duplicated 5 times (exact timestamp and all fields are identical except the UID). My conclusion is pf_ring is not splitting the traffic and that all procs are seeing all the traffic. my node.cfg: [bro-worders] type=worker host=localhost interface=eth5 lb_method=pf_ring lb_procs=5 pf_ring was loaded with: enable_tx_capture=0 min_num_slots=32768 Bro is correctly linked to libpcap libraries: ldd /usr/local/bro/bin/bro | grep pcap libpcap.so.1 => /opt/pfring-6.6/lib/libpcap.so.1 (0x00007effe684d000) pf_ring info: [root at bro-box]# cat /proc/net/pf_ring/info PF_RING Version : 6.7.0 (dev:9b0e7c81718edb0ff6d070793bc868e3c3456bd5) Total rings : 6 Standard (non ZC) Options Ring slots : 32768 Slot version : 16 Capture TX : No [RX only] IP Defragment : No Socket Mode : Standard Cluster Fragment Queue : 0 Cluster Fragment Discard : 0 I am not sure where to go from here. Does anyone have any suggestions? Jereme Lamps? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180130/d39d28de/attachment.html From ben.bt.wood at gmail.com Wed Jan 31 07:49:11 2018 From: ben.bt.wood at gmail.com (Benjamin Wood) Date: Wed, 31 Jan 2018 10:49:11 -0500 Subject: [Bro] Bro-2.5.2 and PF_RING 6.7 not load balancing properly In-Reply-To: References: Message-ID: The default load balancing for bro pf_ring is to use 4-tuple. If you have a lot of asymmetric traffic (hot IP/port combo like a syslog or something), you'll see some "buckets" with more packets. You may want to try a different load balancing scheme as outlined here: https://www.bro.org/sphinx/components/broctl/README.html#pfringclustertype On Tue, Jan 30, 2018 at 3:07 PM, Lamps, Jereme wrote: > It appears PF_RING is not properly load balancing between Bro instances. > For example, I have a single Bro node with 5 bro procs. Every entry in > http.log is duplicated 5 times (exact timestamp and all fields are > identical except the UID). My conclusion is pf_ring is not splitting the > traffic and that all procs are seeing all the traffic. > > *my node.cfg: * > [bro-worders] > type=worker > host=localhost > interface=eth5 > lb_method=pf_ring > lb_procs=5 > > *pf_ring was loaded with: * > enable_tx_capture=0 min_num_slots=32768 > > *Bro is correctly linked to libpcap libraries:* > ldd /usr/local/bro/bin/bro | grep pcap > libpcap.so.1 => /opt/pfring-6.6/lib/libpcap.so.1 > (0x00007effe684d000) > > *pf_ring info:* > [root at bro-box]# cat /proc/net/pf_ring/info > PF_RING Version : 6.7.0 (dev:9b0e7c81718edb0ff6d070793bc868 > e3c3456bd5) > Total rings : 6 > Standard (non ZC) Options > Ring slots : 32768 > Slot version : 16 > Capture TX : No [RX only] > IP Defragment : No > Socket Mode : Standard > Cluster Fragment Queue : 0 > Cluster Fragment Discard : 0 > > I am not sure where to go from here. Does anyone have any suggestions? > > Jereme Lamps? > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180131/9b4f8f78/attachment.html From jazoff at illinois.edu Wed Jan 31 08:25:37 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 31 Jan 2018 16:25:37 +0000 Subject: [Bro] Bro-2.5.2 and PF_RING 6.7 not load balancing properly In-Reply-To: References: Message-ID: <3A7060C0-DB4B-47FD-A39E-C2C3EF49970C@illinois.edu> > On Jan 30, 2018, at 3:07 PM, Lamps, Jereme wrote: > > It appears PF_RING is not properly load balancing between Bro instances. For example, I have a single Bro node with 5 bro procs. Every entry in http.log is duplicated 5 times (exact timestamp and all fields are identical except the UID). My conclusion is pf_ring is not splitting the traffic and that all procs are seeing all the traffic. You may be running into an issue that was recently fixed in broctl and will be resolved in the next release. Depending on the order you install things in, pf_ring load balancing can end up disabled. What does the following output for your host? [root at bro-dev ~]# broctl config | grep pfring pfringclusterid = 21 pfringclustertype = 4-tuple ringfirstappinstance = 0 if you have pfringclusterid set to 0, that's the problem that was just fixed. You can easily workaround that by adding PFRINGClusterID = 21 to your /usr/local/bro/etc/broctl.cfg Once that is there, a broctl deploy should get everything working. ? Justin Azoff From jlay at slave-tothe-box.net Wed Jan 31 15:21:58 2018 From: jlay at slave-tothe-box.net (James Lay) Date: Wed, 31 Jan 2018 16:21:58 -0700 Subject: [Bro] A little more confusion with Intel In-Reply-To: <3A565BA7-E885-420B-BB4A-3689FFE9150F@corelight.com> References: <60a34e5669db124ce69305bc1232ab3c@localhost> <309A727C-26D8-4D26-975D-8FDD5158A82E@corelight.com> <3ccd76709d5eafabe321885b5acec101@localhost> <3A565BA7-E885-420B-BB4A-3689FFE9150F@corelight.com> Message-ID: Thanks Seth, I basically modified this for bro use: https://isc.sans.edu/forums/diary/Tracking+Newly+Registered+Domains/23127/ It's basically a list of domain names that have been newly registered. Does that help? James On 2018-01-29 09:14, Seth Hall wrote: > On 22 Jan 2018, at 11:30, James Lay wrote: > >> It's actually the inverse of what I'm seeing.? In my tests if I have >> Intel::DOMAIN yahoo.com and I did a >> "dig?[www.yahoo.com",]()?the domain intel >> would not match because the dns request was for "www.yahoo.com", not >> yahoo.com.? Does that make sense?? Thank you. > > Yeah, if we had a more comprehensive matcher for the intel framework > then you'd have a lot of options open for you. I suppose that my main > point was that at the moment you will have to just include the exact > domain that you want to match on. > > Do you have a large list where you'd like to watch for any hits on the > effective second level domain like you're describing here? > > .Seth > > -- > Seth Hall * Corelight, Inc * www.corelight.com