From jan.grashoefer at gmail.com Thu Dec 1 02:24:01 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Thu, 1 Dec 2016 11:24:01 +0100 Subject: [Bro] Intelligence framework and bro logs In-Reply-To: References: Message-ID: <3750554e-9f75-ef9e-4eb9-06a63feb528d@gmail.com> Hi N, > -I am updating the IOCs regularly and the only way I found to reload IOCs > in bro is to restart the service with broctl, is there any better way? > (like just reloading the configuration and not restarting everything) using Bro 2.5 you can use the new expiration feature of the intel framework. There might be a blog post explaining the details. I will check that. Best regards Jan From leonardo.mokarzel.falcon at gmail.com Thu Dec 1 02:55:49 2016 From: leonardo.mokarzel.falcon at gmail.com (Leonardo Mokarzel Falcon) Date: Thu, 1 Dec 2016 11:55:49 +0100 Subject: [Bro] DNS request type empty Message-ID: <02EBE069-3DC3-4B02-87D6-721FB2D270EC@gmail.com> Hi all! I've installed and configured Bro 2.5 in my Raspberry Pi and it works pretty well but some fields in the DNS log file are empty, like the query code and query type. My Pi it's configured to be DHCP and gateway for my test network. The DHCP configures Google's public DNS servers for new clients. Can anybody advice or share his experience with similar issues? Thanks! Kind regards, Leonardo Mokarzel Falcon @LMokarzel From philosnef at gmail.com Thu Dec 1 04:56:29 2016 From: philosnef at gmail.com (erik clark) Date: Thu, 1 Dec 2016 07:56:29 -0500 Subject: [Bro] odd large swing in capture loss Message-ID: I have been running Bro with af_packet for some time now, but this morning I noticed I have a massive capture loss, on the order of 10-30% capture loss across the workers. Could this be because I did not have checksums turned off in local.bro? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161201/470613ce/attachment.html From philosnef at gmail.com Thu Dec 1 05:47:02 2016 From: philosnef at gmail.com (erik clark) Date: Thu, 1 Dec 2016 08:47:02 -0500 Subject: [Bro] huge weird.log/conn.log Message-ID: I have two bro sensors. One is running 2.5, one is running 2.4.1. Both are running on the same link off the tap. The weird.log on the 2.5 box is 6 times bigger than the weird.log on the 2.4.1 log. Any idea why this might be? How can I troubleshoot this. My conn.log is 3 times bigger. For reference: conn.log -> 2.5 (45 minutes) 17 gig conn.log -> 2.4.1 (45 min) 5.5 gig weird.log -> 2.5 (45 minutes) 11 gig weird.log -> 2.4.1 (45 minutes) 1.2 gig These numbers seem to be WAY off. I have no idea how to even try and parse this to see what is going on. Packet loss on 2.4.1 is 6% Packet loss on 2.5 is 1%. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161201/57a1706c/attachment.html From jazoff at illinois.edu Thu Dec 1 06:46:57 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 1 Dec 2016 14:46:57 +0000 Subject: [Bro] DNS request type empty In-Reply-To: <02EBE069-3DC3-4B02-87D6-721FB2D270EC@gmail.com> References: <02EBE069-3DC3-4B02-87D6-721FB2D270EC@gmail.com> Message-ID: <321FA86E-5816-4CB0-B1C5-47D442731893@illinois.edu> > On Dec 1, 2016, at 4:55 AM, Leonardo Mokarzel Falcon wrote: > > Hi all! > > I've installed and configured Bro 2.5 in my Raspberry Pi and it works pretty well but some fields in the DNS log file are empty, like the query code and query type. My Pi it's configured to be DHCP and gateway for my test network. The DHCP configures Google's public DNS servers for new clients. > > Can anybody advice or share his experience with similar issues? Sounds like https://www.bro.org/documentation/faq.html#why-isn-t-bro-producing-the-logs-i-expect-a-note-about-checksums -- - Justin Azoff From vladg at illinois.edu Thu Dec 1 08:14:25 2016 From: vladg at illinois.edu (Vlad Grigorescu) Date: Thu, 01 Dec 2016 10:14:25 -0600 Subject: [Bro] huge weird.log/conn.log In-Reply-To: References: Message-ID: Can you take a look at what weirds, specifically, you're getting? Something like: > cat weird.log | bro-cut name| sort | uniq -c | sort -n --Vlad erik clark writes: > I have two bro sensors. One is running 2.5, one is running 2.4.1. Both are > running on the same link off the tap. > > The weird.log on the 2.5 box is 6 times bigger than the weird.log on the > 2.4.1 log. Any idea why this might be? How can I troubleshoot this. > > My conn.log is 3 times bigger. For reference: > > conn.log -> 2.5 (45 minutes) 17 gig > conn.log -> 2.4.1 (45 min) 5.5 gig > > weird.log -> 2.5 (45 minutes) 11 gig > weird.log -> 2.4.1 (45 minutes) 1.2 gig > > These numbers seem to be WAY off. I have no idea how to even try and parse > this to see what is going on. > > Packet loss on 2.4.1 is 6% > Packet loss on 2.5 is 1%. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 800 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161201/70f6610e/attachment.bin From philosnef at gmail.com Thu Dec 1 08:22:09 2016 From: philosnef at gmail.com (erik clark) Date: Thu, 1 Dec 2016 11:22:09 -0500 Subject: [Bro] huge weird.log/conn.log In-Reply-To: References: Message-ID: Hm. It looks like this may be related to af_packet and bro2.5 in general. I did a subset of production weird and took a subset of development weird, sorted it out and compared the two. From the looks of things, the ratio of items in the files take in identical number of events is pretty close to identical. This leads me to believe that I am just not dropping traffic either at Bro or the interface on the dev box. Right now I have dropped only 70k packets out of 49TiB of traffic according to ifconfig, and bro is reporting packet loss of ~1%. The 2.4.1 production box on the other hand is seeing 2-5% packet loss and some packet loss at the interface. The services (http, dns, so on so forth) on the dev box all have equal or more than the number of events on the production box. All I can think of right now is that tuned af_packet on rh7 w/ 2.5 is so much better than tuned pf_ring on rh61 w/ 2.4.1 that it has been noticeable. Also, memory consumption on 2.5 is a significant fraction less than on the production box with the same link. Wish I could say why this is, but it really impresses me. Load is still high though at ~16, but MEH. On Thu, Dec 1, 2016 at 11:14 AM, Vlad Grigorescu wrote: > Can you take a look at what weirds, specifically, you're getting? > Something like: > > > cat weird.log | bro-cut name| sort | uniq -c | sort -n > > --Vlad > > erik clark writes: > > > I have two bro sensors. One is running 2.5, one is running 2.4.1. Both > are > > running on the same link off the tap. > > > > The weird.log on the 2.5 box is 6 times bigger than the weird.log on the > > 2.4.1 log. Any idea why this might be? How can I troubleshoot this. > > > > My conn.log is 3 times bigger. For reference: > > > > conn.log -> 2.5 (45 minutes) 17 gig > > conn.log -> 2.4.1 (45 min) 5.5 gig > > > > weird.log -> 2.5 (45 minutes) 11 gig > > weird.log -> 2.4.1 (45 minutes) 1.2 gig > > > > These numbers seem to be WAY off. I have no idea how to even try and > parse > > this to see what is going on. > > > > Packet loss on 2.4.1 is 6% > > Packet loss on 2.5 is 1%. > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161201/54b4dff7/attachment-0001.html From philosnef at gmail.com Thu Dec 1 09:24:16 2016 From: philosnef at gmail.com (erik clark) Date: Thu, 1 Dec 2016 12:24:16 -0500 Subject: [Bro] huge weird.log/conn.log In-Reply-To: References: Message-ID: Hmm. I note that I am actually, in a given hour, getting 25-30% less logs from http.log. Are there any guides to tuning Bro to work with af_packet? On Thu, Dec 1, 2016 at 11:22 AM, erik clark wrote: > Hm. It looks like this may be related to af_packet and bro2.5 in general. > I did a subset of production weird and took a subset of development weird, > sorted it out and compared the two. From the looks of things, the ratio of > items in the files take in identical number of events is pretty close to > identical. > > This leads me to believe that I am just not dropping traffic either at Bro > or the interface on the dev box. Right now I have dropped only 70k packets > out of 49TiB of traffic according to ifconfig, and bro is reporting packet > loss of ~1%. > > The 2.4.1 production box on the other hand is seeing 2-5% packet loss and > some packet loss at the interface. The services (http, dns, so on so forth) > on the dev box all have equal or more than the number of events on the > production box. All I can think of right now is that tuned af_packet on rh7 > w/ 2.5 is so much better than tuned pf_ring on rh61 w/ 2.4.1 that it has > been noticeable. > > Also, memory consumption on 2.5 is a significant fraction less than on the > production box with the same link. Wish I could say why this is, but it > really impresses me. Load is still high though at ~16, but MEH. > > > On Thu, Dec 1, 2016 at 11:14 AM, Vlad Grigorescu > wrote: > >> Can you take a look at what weirds, specifically, you're getting? >> Something like: >> >> > cat weird.log | bro-cut name| sort | uniq -c | sort -n >> >> --Vlad >> >> erik clark writes: >> >> > I have two bro sensors. One is running 2.5, one is running 2.4.1. Both >> are >> > running on the same link off the tap. >> > >> > The weird.log on the 2.5 box is 6 times bigger than the weird.log on the >> > 2.4.1 log. Any idea why this might be? How can I troubleshoot this. >> > >> > My conn.log is 3 times bigger. For reference: >> > >> > conn.log -> 2.5 (45 minutes) 17 gig >> > conn.log -> 2.4.1 (45 min) 5.5 gig >> > >> > weird.log -> 2.5 (45 minutes) 11 gig >> > weird.log -> 2.4.1 (45 minutes) 1.2 gig >> > >> > These numbers seem to be WAY off. I have no idea how to even try and >> parse >> > this to see what is going on. >> > >> > Packet loss on 2.4.1 is 6% >> > Packet loss on 2.5 is 1%. >> > _______________________________________________ >> > Bro mailing list >> > bro at bro-ids.org >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161201/1e025da3/attachment.html From jazoff at illinois.edu Thu Dec 1 09:48:31 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 1 Dec 2016 17:48:31 +0000 Subject: [Bro] huge weird.log/conn.log In-Reply-To: References: Message-ID: <7A21A0B1-8E74-4F3D-AA07-F8197E7EE7C1@illinois.edu> > On Dec 1, 2016, at 11:24 AM, erik clark wrote: > > Hmm. I note that I am actually, in a given hour, getting 25-30% less logs from http.log. > > Are there any guides to tuning Bro to work with af_packet? > Step 1: ensure that you can use af_packet in the first place: https://github.com/JustinAzoff/can-i-use-afpacket-fanout/ It looks like your current setup is not working. -- - Justin Azoff From philosnef at gmail.com Thu Dec 1 10:13:14 2016 From: philosnef at gmail.com (erik clark) Date: Thu, 1 Dec 2016 13:13:14 -0500 Subject: [Bro] huge weird.log/conn.log In-Reply-To: <7A21A0B1-8E74-4F3D-AA07-F8197E7EE7C1@illinois.edu> References: <7A21A0B1-8E74-4F3D-AA07-F8197E7EE7C1@illinois.edu> Message-ID: Sorry this was supposed to go to the list as well: Hmm. I see FAIL: saw flow {tcp $ip $num $ip $num} on workers $num and $num. This is on RHEL7 with the latest kernel. How can I address what I am assuming is a failure of the kernel? On Thu, Dec 1, 2016 at 12:48 PM, Azoff, Justin S wrote: > > > On Dec 1, 2016, at 11:24 AM, erik clark wrote: > > > > Hmm. I note that I am actually, in a given hour, getting 25-30% less > logs from http.log. > > > > Are there any guides to tuning Bro to work with af_packet? > > > Step 1: ensure that you can use af_packet in the first place: > > https://github.com/JustinAzoff/can-i-use-afpacket-fanout/ > > It looks like your current setup is not working. > > -- > - Justin Azoff > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161201/1722ed9a/attachment.html From jazoff at illinois.edu Thu Dec 1 10:46:46 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 1 Dec 2016 18:46:46 +0000 Subject: [Bro] huge weird.log/conn.log In-Reply-To: References: <7A21A0B1-8E74-4F3D-AA07-F8197E7EE7C1@illinois.edu> Message-ID: > On Dec 1, 2016, at 12:13 PM, erik clark wrote: > > Sorry this was supposed to go to the list as well: > > Hmm. I see > > FAIL: saw flow {tcp $ip $num $ip $num} on workers $num and $num. > > This is on RHEL7 with the latest kernel. How can I address what I am assuming is a failure of the kernel? > Yeah, that kernel does not work. I believe Michal said that if you upgrade the ixgb driver to the latest from intel and mess around with ethtool settings you can get it to work. -- - Justin Azoff From philosnef at gmail.com Thu Dec 1 12:25:13 2016 From: philosnef at gmail.com (erik clark) Date: Thu, 1 Dec 2016 15:25:13 -0500 Subject: [Bro] huge weird.log/conn.log In-Reply-To: References: <7A21A0B1-8E74-4F3D-AA07-F8197E7EE7C1@illinois.edu> Message-ID: Justin, any chance you might have that commentary stored anywhere? I spent the past 3 hours trying to find ethtool settings for this, but have had no success. The only thing I did find was --set-priv-flags $iface rss-symmetric off but I get no private flags found (for ixgbe nic). Other than that, I can find nothing anywhere concerning this issue that seems to be here. I did see: https://media.readthedocs.org/pdf/jasonish-suricata/latest/jasonish-suricata.pdf , specifically: --- Some NIC?s allow you to set it into a symmetric mode. The Intel X(L)710 card can do this in theory, but the drivers aren?t capable of enabling this yet (work is underway to try to address this). Another way to address is by setting a special ?Random Secret Key? that will make the RSS symmetrical. See http://www.ndsl.kaist.edu/~kyoungsoo/papers/TR-symRSS.pdf (PDF). In most scenario?s however, the optimal solution is to reduce the number of RSS queues to 1: --- The pdf pointed to in this link is abstract and totally not useful in any sort of practical way. That pdf (TR-symRSS.pdf) is purely academic and has little, if any, use to us in this situation. Setting the RSS queues to 1 doesn't seem like a very good solution to this problem. The alternative is to just go back to pf_ring, which I am loathe to do. On Thu, Dec 1, 2016 at 1:46 PM, Azoff, Justin S wrote: > > > On Dec 1, 2016, at 12:13 PM, erik clark wrote: > > > > Sorry this was supposed to go to the list as well: > > > > Hmm. I see > > > > FAIL: saw flow {tcp $ip $num $ip $num} on workers $num and $num. > > > > This is on RHEL7 with the latest kernel. How can I address what I am > assuming is a failure of the kernel? > > > > Yeah, that kernel does not work. I believe Michal said that if you > upgrade the ixgb driver to the latest from intel and mess around with > ethtool settings you can get it to work. > > -- > - Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161201/f05c45e3/attachment.html From philosnef at gmail.com Fri Dec 2 05:42:34 2016 From: philosnef at gmail.com (erik clark) Date: Fri, 2 Dec 2016 08:42:34 -0500 Subject: [Bro] broctly deploy hangs on checking scripts Message-ID: If I have bro running and try a broctl deploy, it hangs on checking scripts. I have confirmed there are no files owned by a user other than the one running bro. If I stop bro first, do an install, then do a check, it does it very fast and cleanly. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161202/50c585da/attachment.html From jazoff at illinois.edu Fri Dec 2 06:22:44 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 2 Dec 2016 14:22:44 +0000 Subject: [Bro] broctly deploy hangs on checking scripts In-Reply-To: References: Message-ID: <130EB9E2-81B9-4643-9FBD-2F048014C7EE@illinois.edu> > On Dec 2, 2016, at 7:42 AM, erik clark wrote: > > If I have bro running and try a broctl deploy, it hangs on checking scripts. I have confirmed there are no files owned by a user other than the one running bro. If I stop bro first, do an install, then do a check, it does it very fast and cleanly. I used to see that occasionally when the manager was extremely heavily loaded... but it's been a while (I thought Daniel fixed it?). It was some sort of race condition in the command/script that check runs that causes it to not exit. all the check script itself does is start bro with event bro_init() &priority=-10 { terminate_communication(); } Are you still seeing this on 2.5? -- - Justin Azoff From philosnef at gmail.com Fri Dec 2 07:11:07 2016 From: philosnef at gmail.com (erik clark) Date: Fri, 2 Dec 2016 10:11:07 -0500 Subject: [Bro] broctly deploy hangs on checking scripts In-Reply-To: <130EB9E2-81B9-4643-9FBD-2F048014C7EE@illinois.edu> References: <130EB9E2-81B9-4643-9FBD-2F048014C7EE@illinois.edu> Message-ID: Yes, seeing this in 2.5. On Fri, Dec 2, 2016 at 9:22 AM, Azoff, Justin S wrote: > > On Dec 2, 2016, at 7:42 AM, erik clark wrote: > > > > If I have bro running and try a broctl deploy, it hangs on checking > scripts. I have confirmed there are no files owned by a user other than the > one running bro. If I stop bro first, do an install, then do a check, it > does it very fast and cleanly. > > I used to see that occasionally when the manager was extremely heavily > loaded... but it's been a while (I thought Daniel fixed it?). It was some > sort of race condition in the command/script that check runs that causes it > to not exit. > > all the check script itself does is start bro with > > event bro_init() &priority=-10 > { > terminate_communication(); > } > > Are you still seeing this on 2.5? > > -- > - Justin Azoff > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161202/6cad2769/attachment.html From dopheide at gmail.com Fri Dec 2 08:10:08 2016 From: dopheide at gmail.com (Mike Dopheide) Date: Fri, 2 Dec 2016 10:10:08 -0600 Subject: [Bro] broctly deploy hangs on checking scripts In-Reply-To: References: <130EB9E2-81B9-4643-9FBD-2F048014C7EE@illinois.edu> Message-ID: If you have exit_only_after_terminate set, usually for debugging, we've had that hang the 'bro check'. -Dop On Fri, Dec 2, 2016 at 9:11 AM, erik clark wrote: > Yes, seeing this in 2.5. > > On Fri, Dec 2, 2016 at 9:22 AM, Azoff, Justin S > wrote: > >> > On Dec 2, 2016, at 7:42 AM, erik clark wrote: >> > >> > If I have bro running and try a broctl deploy, it hangs on checking >> scripts. I have confirmed there are no files owned by a user other than the >> one running bro. If I stop bro first, do an install, then do a check, it >> does it very fast and cleanly. >> >> I used to see that occasionally when the manager was extremely heavily >> loaded... but it's been a while (I thought Daniel fixed it?). It was some >> sort of race condition in the command/script that check runs that causes it >> to not exit. >> >> all the check script itself does is start bro with >> >> event bro_init() &priority=-10 >> { >> terminate_communication(); >> } >> >> Are you still seeing this on 2.5? >> >> -- >> - Justin Azoff >> >> >> >> > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161202/6cb7119b/attachment.html From jazoff at illinois.edu Fri Dec 2 15:38:22 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 2 Dec 2016 23:38:22 +0000 Subject: [Bro] broctly deploy hangs on checking scripts In-Reply-To: References: <130EB9E2-81B9-4643-9FBD-2F048014C7EE@illinois.edu> Message-ID: <9B6973B0-ACA8-41DF-8DF8-63CEA8F24E04@illinois.edu> Yeah, that would do it.. but it would break like that 100% of the time, not only when bro was running. Daniel, do you remember why we use terminate_communication() instead of exit_only_after_terminate + terminate() ? I vaguely remember something about terminate_communication being misused somewhere else that had issues with the switch to broker. -- - Justin Azoff > On Dec 2, 2016, at 10:10 AM, Mike Dopheide wrote: > > If you have exit_only_after_terminate set, usually for debugging, we've had that hang the 'bro check'. > > -Dop > > On Fri, Dec 2, 2016 at 9:11 AM, erik clark wrote: > Yes, seeing this in 2.5. > > On Fri, Dec 2, 2016 at 9:22 AM, Azoff, Justin S wrote: > > On Dec 2, 2016, at 7:42 AM, erik clark wrote: > > > > If I have bro running and try a broctl deploy, it hangs on checking scripts. I have confirmed there are no files owned by a user other than the one running bro. If I stop bro first, do an install, then do a check, it does it very fast and cleanly. > > I used to see that occasionally when the manager was extremely heavily loaded... but it's been a while (I thought Daniel fixed it?). It was some sort of race condition in the command/script that check runs that causes it to not exit. > > all the check script itself does is start bro with > > event bro_init() &priority=-10 > { > terminate_communication(); > } > > Are you still seeing this on 2.5? > > -- > - Justin Azoff > > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From dnthayer at illinois.edu Sat Dec 3 10:13:27 2016 From: dnthayer at illinois.edu (Daniel Thayer) Date: Sat, 3 Dec 2016 12:13:27 -0600 Subject: [Bro] broctly deploy hangs on checking scripts In-Reply-To: References: <130EB9E2-81B9-4643-9FBD-2F048014C7EE@illinois.edu> Message-ID: <6b0f2b64-b1f0-a2bc-90df-d85108e9db3b@illinois.edu> Does this happen every time, or only sometimes? On 12/2/16 9:11 AM, erik clark wrote: > Yes, seeing this in 2.5. > > On Fri, Dec 2, 2016 at 9:22 AM, Azoff, Justin S > wrote: > > > On Dec 2, 2016, at 7:42 AM, erik clark > wrote: > > > > If I have bro running and try a broctl deploy, it hangs on checking scripts. I have confirmed there are no files owned by a user other than the one running bro. If I stop bro first, do an install, then do a check, it does it very fast and cleanly. > > I used to see that occasionally when the manager was extremely > heavily loaded... but it's been a while (I thought Daniel fixed > it?). It was some sort of race condition in the command/script that > check runs that causes it to not exit. > > all the check script itself does is start bro with > > event bro_init() &priority=-10 > { > terminate_communication(); > } > > Are you still seeing this on 2.5? > > -- > - Justin Azoff > > > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From philosnef at gmail.com Sat Dec 3 14:46:24 2016 From: philosnef at gmail.com (erik clark) Date: Sat, 3 Dec 2016 17:46:24 -0500 Subject: [Bro] broctly deploy hangs on checking scripts In-Reply-To: <6b0f2b64-b1f0-a2bc-90df-d85108e9db3b@illinois.edu> References: <130EB9E2-81B9-4643-9FBD-2F048014C7EE@illinois.edu> <6b0f2b64-b1f0-a2bc-90df-d85108e9db3b@illinois.edu> Message-ID: Some times. I can't seem to get a consistently repeatable behavior. When it does happen though, no matter how many times I kill the broctl attempt and rerun deploy, it won't check. I have to issue a stop first. On Sat, Dec 3, 2016 at 1:13 PM, Daniel Thayer wrote: > Does this happen every time, or only sometimes? > > > On 12/2/16 9:11 AM, erik clark wrote: > >> Yes, seeing this in 2.5. >> >> On Fri, Dec 2, 2016 at 9:22 AM, Azoff, Justin S > > wrote: >> >> > On Dec 2, 2016, at 7:42 AM, erik clark > > wrote: >> > >> > If I have bro running and try a broctl deploy, it hangs on checking >> scripts. I have confirmed there are no files owned by a user other than the >> one running bro. If I stop bro first, do an install, then do a check, it >> does it very fast and cleanly. >> >> I used to see that occasionally when the manager was extremely >> heavily loaded... but it's been a while (I thought Daniel fixed >> it?). It was some sort of race condition in the command/script that >> check runs that causes it to not exit. >> >> all the check script itself does is start bro with >> >> event bro_init() &priority=-10 >> { >> terminate_communication(); >> } >> >> Are you still seeing this on 2.5? >> >> -- >> - Justin Azoff >> >> >> >> >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161203/86652ad8/attachment.html From dnthayer at illinois.edu Sat Dec 3 21:58:25 2016 From: dnthayer at illinois.edu (Daniel Thayer) Date: Sat, 3 Dec 2016 23:58:25 -0600 Subject: [Bro] broctly deploy hangs on checking scripts In-Reply-To: References: <130EB9E2-81B9-4643-9FBD-2F048014C7EE@illinois.edu> <6b0f2b64-b1f0-a2bc-90df-d85108e9db3b@illinois.edu> Message-ID: <4efb22d6-3401-12c3-271b-f1dcb732d0ba@illinois.edu> OK, would you be willing to try the attached patch? I'd like to know if it prevents "broctl deploy" (or "broctl check") from hanging. To apply the patch, you don't need to re-install Bro, you can just patch the installed copy (replace $PREFIX with your Bro install prefix directory, such as /usr/local/bro): cd $PREFIX patch -p0 -b < check.patch -------------- next part -------------- --- lib/broctl/BroControl/control.py.orig 2016-12-03 23:36:57.000000000 -0600 +++ lib/broctl/BroControl/control.py 2016-12-03 23:34:34.000000000 -0600 @@ -694,6 +694,8 @@ cmd = os.path.join(self.config.scriptsdir, "check-config") + " %s %s %s %s" % (installed_policies, print_scripts, cwd, " ".join(_make_bro_params(node, False))) cmd += " broctl/check" + if not list_scripts: + cmd += " -a" cmds += [((node, cwd), cmd, env, None)] From deshmukh at slac.stanford.edu Sun Dec 4 18:29:01 2016 From: deshmukh at slac.stanford.edu (Deshmukh, Andy) Date: Mon, 5 Dec 2016 02:29:01 +0000 Subject: [Bro] No notice.log after switch upgrade/downgrade Message-ID: <40935193-70B0-4501-862C-08476B5E0B4F@slac.stanford.edu> Hello, We are having some issues with the BRO cluster here @SLAC. I am kind of a noob with respect to BRO and the setup we have @SLAC. Please excuse me and my ignorance. We have Cisco 3k switch running on tap aggregation mode and it also load-balances traffic to the BRO cluster. We tried to upgrade the switch to the newer NX-OS version but we had some problems and we had to revert to the original version with the exact same configuration. However, there are no notice.log being generated since the upgrade/downgrade incident. On splunk, the BRO traffic event counts have decreased 1/7th after the incident. I am sure there are things that I am missing after the upgrade/downgrade and I am unable to figure out. One of the colleague suggested, it might be related to asymmetric flow of forward and reverse packets to the worker nodes which is why BRO is failing to analyze the traffic. So, on the switch, I checked if there is load-balance symmetry command; which is on the switch and I performed tcpdump on bro-worker node and the traffic is communication with bro-manager node. Planning to involve cisco support tomorrow and to capture traffic from the switchport to Bro worker node and see if I can figure out what?s going on. Any thoughts? Thanks, Andy -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161205/9c54d9ff/attachment.html From jazoff at illinois.edu Sun Dec 4 19:22:28 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 5 Dec 2016 03:22:28 +0000 Subject: [Bro] No notice.log after switch upgrade/downgrade In-Reply-To: <40935193-70B0-4501-862C-08476B5E0B4F@slac.stanford.edu> References: <40935193-70B0-4501-862C-08476B5E0B4F@slac.stanford.edu> Message-ID: <9C43110E-6B07-4893-9BBA-C4EC06F4551C@illinois.edu> > On Dec 4, 2016, at 8:29 PM, Deshmukh, Andy wrote: > > Hello, > > We are having some issues with the BRO cluster here @SLAC. I am kind of a noob with respect to BRO and the setup we have @SLAC. Please excuse me and my ignorance. > > We have Cisco 3k switch running on tap aggregation mode and it also load-balances traffic to the BRO cluster. We tried to upgrade the switch to the newer NX-OS version but we had some problems and we had to revert to the original version with the exact same configuration. > However, there are no notice.log being generated since the upgrade/downgrade incident. On splunk, the BRO traffic event counts have decreased 1/7th after the incident. I am sure there are things that I am missing after the upgrade/downgrade and I am unable to figure out. > > One of the colleague suggested, it might be related to asymmetric flow of forward and reverse packets to the worker nodes which is why BRO is failing to analyze the traffic. So, on the switch, I checked if there is load-balance symmetry command; which is on the switch and I performed tcpdump on bro-worker node and the traffic is communication with bro-manager node. > Planning to involve cisco support tomorrow and to capture traffic from the switchport to Bro worker node and see if I can figure out what?s going on. > > Any thoughts? > > Thanks, > Andy Sounds like you are on the right track. You can tell from conn.log entries if you are getting asymmetric flow distribution. Instead of seeing a single connection between a and b with orig_pkts and resp_pkts and a history like ShADadFf, you'll see two connections one from a to b with orig_pkts and no resp_pkts with a history of SAD.. (SAD IS BAD) one from b to a with resp_pkts and no orig_pkts with a history of had.. What I would do is check for this, then reproduce it by using tcpdump directly, that way you can take the evidence to cisco and they can't blame Bro. -- - Justin Azoff From philosnef at gmail.com Mon Dec 5 05:33:25 2016 From: philosnef at gmail.com (erik clark) Date: Mon, 5 Dec 2016 08:33:25 -0500 Subject: [Bro] pf_ring vrs PF_RING::$iface Message-ID: So, having built bro with the pf_ring plugin and pf_ring (libpcap pfring), I have found that the plugin does not seem to be working as expected. When I run interface=$iface lb_method=pf_ring lb_procs=18 I get much better performace and less "weird" stuff like rapidly growing conn and weird logs. When I use interface=pf_ring::$iface lb_method-=(pf_ring or custom, doesnt matter which I choose) lb_procs=18 my conn logs go crazy. Additionally, some logs which normally grow at 1 to 2 meg a second grow at 1/10th of that. Is there something undocumented about the native pf_ring plugin that I am unaware of which would lead to this behavioral discrepency? Is this also rooted in RHEL7 kernel land issues? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161205/37f476b8/attachment.html From philosnef at gmail.com Mon Dec 5 05:34:57 2016 From: philosnef at gmail.com (erik clark) Date: Mon, 5 Dec 2016 08:34:57 -0500 Subject: [Bro] broctly deploy hangs on checking scripts In-Reply-To: <4efb22d6-3401-12c3-271b-f1dcb732d0ba@illinois.edu> References: <130EB9E2-81B9-4643-9FBD-2F048014C7EE@illinois.edu> <6b0f2b64-b1f0-a2bc-90df-d85108e9db3b@illinois.edu> <4efb22d6-3401-12c3-271b-f1dcb732d0ba@illinois.edu> Message-ID: I will try to get this applied this morning and will get back to you. On Sun, Dec 4, 2016 at 12:58 AM, Daniel Thayer wrote: > OK, would you be willing to try the attached patch? > I'd like to know if it prevents "broctl deploy" (or "broctl check") > from hanging. > > To apply the patch, you don't need to re-install Bro, > you can just patch the installed copy (replace $PREFIX > with your Bro install prefix directory, such as /usr/local/bro): > cd $PREFIX > patch -p0 -b < check.patch > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161205/6cff9502/attachment.html From jdopheid at illinois.edu Mon Dec 5 07:21:24 2016 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Mon, 5 Dec 2016 15:21:24 +0000 Subject: [Bro] Bro4Pros 2017: Seeking sponsors for happy hour Message-ID: <630EA303-D78D-41C3-9408-C1C3219A0A28@illinois.edu> Attention Bro Community, Since Bro4Pros 2017 attendee registration is free we are operating on a tight catering budget. Salesforce has been very generous to provide the event venue and food during the day. However, we will need additional sponsorship funds if we want to offer a happy hour afterwards (drinks and light hors d?oeuvre). https://www.bro.org/community/bro4pros2017.html If you or your company are interested, please contact us at info at bro.org. And don?t forget to register and submit your CFP proposal. Thanks, The Bro Project ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From jdopheid at illinois.edu Mon Dec 5 08:31:55 2016 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Mon, 5 Dec 2016 16:31:55 +0000 Subject: [Bro] Guest blog: The Intelligence Framework Update Message-ID: Guest blogger, Jan Grash?fer, has written a blog about updates to the Bro Intelligence Framework. You can read the full post here, below is a summary: http://blog.bro.org/2016/12/the-intelligence-framework-update.html Summary This blog post discusses the data model of Bro's intelligence framework and the new remove function. Furthermore the intelligence expiration and match extension mechanisms are explained. Finally the new type for subnets and the changes to the do_notice.bro script are reviewed. I hope this post could shed some light on the ideas behind Bro's intelligence framework. Have fun integrating the framework into your Bro deployment! Thanks Jan for your contribution! ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From deshmukh at slac.stanford.edu Mon Dec 5 12:03:19 2016 From: deshmukh at slac.stanford.edu (Deshmukh, Andy) Date: Mon, 5 Dec 2016 20:03:19 +0000 Subject: [Bro] No notice.log after switch upgrade/downgrade In-Reply-To: <9C43110E-6B07-4893-9BBA-C4EC06F4551C@illinois.edu> References: <40935193-70B0-4501-862C-08476B5E0B4F@slac.stanford.edu> <9C43110E-6B07-4893-9BBA-C4EC06F4551C@illinois.edu> Message-ID: <17289a01ed634e53b01971e5a6c987c5@exch13-mail04.win.slac.stanford.edu> Thanks Justin for your help. I was able to look at connection log and I listed the connections by increasing order of duration, i.e., the longest connections at the end. Here?s the snippet of connections at the end of this sort : [cid:image002.jpg at 01D24EEF.90BBC220] I do see there is no equal amount of packets in orig_pkts and resp_pkts but it seems like symmetry is not broken considering in the conn.log there are packets recorded in both directions. However, there are many small connections logged which do not have resp_pkts. Thoughts? Thanks, Andy -----Original Message----- From: Azoff, Justin S [mailto:jazoff at illinois.edu] Sent: Sunday, December 4, 2016 7:22 PM To: Deshmukh, Andy Cc: bro at bro.org Subject: Re: [Bro] No notice.log after switch upgrade/downgrade > On Dec 4, 2016, at 8:29 PM, Deshmukh, Andy wrote: > > Hello, > > We are having some issues with the BRO cluster here @SLAC. I am kind of a noob with respect to BRO and the setup we have @SLAC. Please excuse me and my ignorance. > > We have Cisco 3k switch running on tap aggregation mode and it also load-balances traffic to the BRO cluster. We tried to upgrade the switch to the newer NX-OS version but we had some problems and we had to revert to the original version with the exact same configuration. > However, there are no notice.log being generated since the upgrade/downgrade incident. On splunk, the BRO traffic event counts have decreased 1/7th after the incident. I am sure there are things that I am missing after the upgrade/downgrade and I am unable to figure out. > > One of the colleague suggested, it might be related to asymmetric flow of forward and reverse packets to the worker nodes which is why BRO is failing to analyze the traffic. So, on the switch, I checked if there is load-balance symmetry command; which is on the switch and I performed tcpdump on bro-worker node and the traffic is communication with bro-manager node. > Planning to involve cisco support tomorrow and to capture traffic from the switchport to Bro worker node and see if I can figure out what?s going on. > > Any thoughts? > > Thanks, > Andy Sounds like you are on the right track. You can tell from conn.log entries if you are getting asymmetric flow distribution. Instead of seeing a single connection between a and b with orig_pkts and resp_pkts and a history like ShADadFf, you'll see two connections one from a to b with orig_pkts and no resp_pkts with a history of SAD.. (SAD IS BAD) one from b to a with resp_pkts and no orig_pkts with a history of had.. What I would do is check for this, then reproduce it by using tcpdump directly, that way you can take the evidence to cisco and they can't blame Bro. -- - Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161205/f426373e/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 55713 bytes Desc: image002.jpg Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161205/f426373e/attachment-0001.jpg From francois.pennaneach at free.fr Mon Dec 5 12:57:00 2016 From: francois.pennaneach at free.fr (=?UTF-8?Q?Fran=c3=a7ois_Pennaneach?=) Date: Mon, 5 Dec 2016 21:57:00 +0100 Subject: [Bro] Custom configuration and installation of btest package Message-ID: <0f5eb550-b217-6016-a481-34ac51e94d0d@free.fr> Hi, I have a question wrt installing the btest package. From the README file, installation command is : python setup.py install Is there a way to customize the installation as in "./configure --prefix" option ? I would like to install btest in my personal directories, not /usr/local/bin and other system directories ? Thank you. Fran?ois From dnthayer at illinois.edu Mon Dec 5 13:28:27 2016 From: dnthayer at illinois.edu (Daniel Thayer) Date: Mon, 5 Dec 2016 15:28:27 -0600 Subject: [Bro] Custom configuration and installation of btest package In-Reply-To: <0f5eb550-b217-6016-a481-34ac51e94d0d@free.fr> References: <0f5eb550-b217-6016-a481-34ac51e94d0d@free.fr> Message-ID: <925d4ae7-523e-f01f-4282-205267486e31@illinois.edu> Have you tried this: python setup.py install --prefix=/some/directory On 12/5/16 2:57 PM, Fran?ois Pennaneach wrote: > Hi, > > I have a question wrt installing the btest package. > > > From the README file, installation command is : > > python setup.py install > > Is there a way to customize the installation as in "./configure > --prefix" option ? > > I would like to install btest in my personal directories, not > /usr/local/bin and other system directories ? > > > Thank you. > > Fran?ois > From francois.pennaneach at free.fr Mon Dec 5 13:34:43 2016 From: francois.pennaneach at free.fr (=?UTF-8?Q?Fran=c3=a7ois_Pennaneach?=) Date: Mon, 5 Dec 2016 22:34:43 +0100 Subject: [Bro] Custom configuration and installation of btest package In-Reply-To: <925d4ae7-523e-f01f-4282-205267486e31@illinois.edu> References: <0f5eb550-b217-6016-a481-34ac51e94d0d@free.fr> <925d4ae7-523e-f01f-4282-205267486e31@illinois.edu> Message-ID: <0d03471d-fd36-8d76-9381-1aad20085fd4@free.fr> Thank you :-) I missed the point when reading the output of "python setup --help" ! Oops, sorry ! Thanks ! Fran?ois Le 05/12/2016 ? 22:28, Daniel Thayer a ?crit : > Have you tried this: > python setup.py install --prefix=/some/directory > > > On 12/5/16 2:57 PM, Fran?ois Pennaneach wrote: >> Hi, >> >> I have a question wrt installing the btest package. >> >> >> From the README file, installation command is : >> >> python setup.py install >> >> Is there a way to customize the installation as in "./configure >> --prefix" option ? >> >> I would like to install btest in my personal directories, not >> /usr/local/bin and other system directories ? >> >> >> Thank you. >> >> Fran?ois >> From vladg at illinois.edu Tue Dec 6 07:52:00 2016 From: vladg at illinois.edu (Vlad Grigorescu) Date: Tue, 06 Dec 2016 09:52:00 -0600 Subject: [Bro] BinPAC analyzer name In-Reply-To: <124db383-ead5-a669-d7a4-54ff95ea9619@googlemail.com> References: <58210681.4000508@googlemail.com> <20161128230555.bkkqrgeuedtebexa@Beezling.local> <1ab7114a-2307-62d6-ddda-d79374db8044@googlemail.com> <20161128231525.otgkhyf3pnccemcj@Beezling.local> <124db383-ead5-a669-d7a4-54ff95ea9619@googlemail.com> Message-ID: Hmm. Could be a capitalization issue? binpac_quickstart does do some uppercasing/lowercasing in a couple of the templates. For example: Dane Wullen writes: > function proc_NAME_message(msg: NAME_PDU) : bool > ... > BifEvent::generate_NAME_event(...); > std::cout << "Name PDU" << endl; # for debugging > ... NAME here will always be lowercase (since that matches the naming scheme for events). I poked around with this a bit, and couldn't duplicate it. Can you put the contents of src/analyzer/protocol/ams somewhere I could see it? Thanks, --Vlad -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 800 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161206/47d62dcd/attachment.bin From albertociolini92 at gmail.com Tue Dec 6 09:38:08 2016 From: albertociolini92 at gmail.com (Alberto Ciolini) Date: Tue, 6 Dec 2016 18:38:08 +0100 Subject: [Bro] Project help Message-ID: Good evening everyone. I am a computer engineering student at the University of Florence and I was assigned a project for my thesis. Basically I have 2 devices, udoo board and OpenMote, which must communicate with each other. OpenMote is my sensor network and udoo is my base station. So when OpenMote detects something odd in the network sends a signal to the base station. My problem is that OpenMote is a small device with low memory and little RAM which requires flash to load programs written in C exclusively. So as you understand I cannot install completely Bro but I have to work through the Broccoli library and I don't know how to write the program that does what I wrote above. Do you have any idea? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161206/9b1e0f8f/attachment.html From bro at pingtrip.com Tue Dec 6 11:48:52 2016 From: bro at pingtrip.com (Dave Crawford) Date: Tue, 6 Dec 2016 14:48:52 -0500 Subject: [Bro] Log File Modifications Message-ID: Is it possible (via scripts vs code modifications) to rename existing columns in a log file? The logging documentation has examples for filtering out specific events, or adding additional columns, but I couldn't find a reference for renaming. Thanks, -Dave From dnthayer at illinois.edu Tue Dec 6 17:13:48 2016 From: dnthayer at illinois.edu (Daniel Thayer) Date: Tue, 6 Dec 2016 19:13:48 -0600 Subject: [Bro] Log File Modifications In-Reply-To: References: Message-ID: You can do something like this: redef Log::default_field_name_map = { ["id.orig_h"] = "src", ["id.orig_p"] = "src_port", ["id.resp_h"] = "dst", ["id.resp_p"] = "dst_port", }; On 12/6/16 1:48 PM, Dave Crawford wrote: > Is it possible (via scripts vs code modifications) to rename existing columns in a log file? The logging documentation has examples for filtering out specific events, or adding additional columns, but I couldn't find a reference for renaming. > > Thanks, > -Dave From bro at pingtrip.com Tue Dec 6 18:16:50 2016 From: bro at pingtrip.com (Dave Crawford) Date: Tue, 6 Dec 2016 21:16:50 -0500 Subject: [Bro] Log File Modifications In-Reply-To: References: Message-ID: Thanks for the pointer Daniel! I was able to find the documentation here: https://www.bro.org/sphinx-git/scripts/base/frameworks/logging/main.bro.html#id-Log::default_field_name_map I also have this working at the filter level now as well and helps me reduce overhead on the Splunk side. -Dave > On Dec 6, 2016, at 8:13 PM, Daniel Thayer wrote: > > You can do something like this: > > redef Log::default_field_name_map = { > ["id.orig_h"] = "src", > ["id.orig_p"] = "src_port", > ["id.resp_h"] = "dst", > ["id.resp_p"] = "dst_port", > }; > > > On 12/6/16 1:48 PM, Dave Crawford wrote: >> Is it possible (via scripts vs code modifications) to rename existing columns in a log file? The logging documentation has examples for filtering out specific events, or adding additional columns, but I couldn't find a reference for renaming. >> >> Thanks, >> -Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161206/8f1f3b11/attachment.html From albertociolini92 at gmail.com Wed Dec 7 00:03:50 2016 From: albertociolini92 at gmail.com (Alberto Ciolini) Date: Wed, 7 Dec 2016 09:03:50 +0100 Subject: [Bro] Communications with Broccoli Message-ID: Good evening everyone. I am a computer engineering student at the University of Florence and I was assigned a project for my thesis. Basically I have 2 devices, udoo board and OpenMote, which must communicate with each other. OpenMote is my sensor network and udoo is my base station. So when OpenMote detects something odd in the network sends a signal to the base station. My problem is that OpenMote is a small device with low memory and little RAM which requires flash to load programs written in C exclusively. So as you understand I cannot install completely Bro but I have to work through the Broccoli library and I don't know how to write the program that does what I wrote above. Do you have any idea? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161207/c4b0ef4a/attachment.html From philosnef at gmail.com Wed Dec 7 04:34:11 2016 From: philosnef at gmail.com (erik clark) Date: Wed, 7 Dec 2016 07:34:11 -0500 Subject: [Bro] RHEL7 and AF_PACKET Message-ID: Short answer: No, AF_PACKET will not work with RHEL7. The long of it, from RedHat direct, is: --- The eb70db875 fix is included in upstream v4.7 so if you need this feature at the cost of everything else, you could use ELRepo's kernel-ml package of v4.7 or later. I tested their kernel-ml-4.8.12-1.el7.elrepo, recompiled the go application, and the test passes fine. --- I am working with RH on this, and maybe with luck it will make it into RH7.4. I will keep everyone posted. :) Erik -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161207/f024c0b4/attachment.html From philosnef at gmail.com Wed Dec 7 05:32:10 2016 From: philosnef at gmail.com (erik clark) Date: Wed, 7 Dec 2016 08:32:10 -0500 Subject: [Bro] one bro manager, multiple node.cfgs Message-ID: Is it possible to push multiple node.cfgs out to separate clusters from one manager? I want to have different loggers for different sets of hosts in different clusters, and having one manager for each set of hosts in each cluster would be ideal. E.g.: cluster 1 proxy 1 worker 1 proxy 2 worker 2 logger 1 cluster 2 proxy 1 worker 1 proxy 2 worker 2 logger 2 This would be with a single manager. The reason for this is because I would be managing hosts in different clusters with vastly different traffic/traffic profiles, and I don't want to comingle it on a single logger. If they log separately, then I can winnow out what traffic came from which tap. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161207/1bee1d18/attachment.html From jackycsie at gmail.com Wed Dec 7 08:24:38 2016 From: jackycsie at gmail.com (=?UTF-8?B?6buD6aao5bmz?=) Date: Thu, 8 Dec 2016 00:24:38 +0800 Subject: [Bro] About some of the Bro related issues. Message-ID: Hello I am a graduate student in Taiwan. I encountered problem. I tried to build simple-client.py in the following URL. https://github.com/bro/bro-netcontrol/tree/master/test It seems like some packages of the version are different. enum -> python 3.5.2 pybroker -> python 2.7.12 Can you get me some solutions to build this file? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161208/5f32d0b7/attachment.html From johanna at icir.org Wed Dec 7 12:37:14 2016 From: johanna at icir.org (Johanna Amann) Date: Wed, 7 Dec 2016 12:37:14 -0800 Subject: [Bro] one bro manager, multiple node.cfgs In-Reply-To: References: Message-ID: <20161207203710.dy5exyhzdmosgkxf@Beezling.local> Sorry, no, it is currently not. There have been plans for clusters with more complex hierarchies for a while, but that is not done yet. If you really just need to mark the source of data, you always could add a column to each logfile that shows which worker node it originated from. Johanna On Wed, Dec 07, 2016 at 08:32:10AM -0500, erik clark wrote: > Is it possible to push multiple node.cfgs out to separate clusters from one > manager? I want to have different loggers for different sets of hosts in > different clusters, and having one manager for each set of hosts in each > cluster would be ideal. E.g.: > > cluster 1 > proxy 1 > worker 1 > proxy 2 > worker 2 > logger 1 > > cluster 2 > proxy 1 > worker 1 > proxy 2 > worker 2 > logger 2 > > This would be with a single manager. The reason for this is because I would > be managing hosts in different clusters with vastly different > traffic/traffic profiles, and I don't want to comingle it on a single > logger. If they log separately, then I can winnow out what traffic came > from which tap. > > Thanks! > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Wed Dec 7 12:39:41 2016 From: johanna at icir.org (Johanna Amann) Date: Wed, 7 Dec 2016 12:39:41 -0800 Subject: [Bro] About some of the Bro related issues. In-Reply-To: References: Message-ID: <20161207203941.rrrud2gpxbu7hquc@Beezling.local> Hi, I am not quite sure if I 100% understand the question; in any case, I only tested the netcontrol python things using python 2; so you should probably use that... the Enum type usd is the https://pypi.python.org/pypi/enum library for python 2. Johanna On Thu, Dec 08, 2016 at 12:24:38AM +0800, ??? wrote: > Hello > I am a graduate student in Taiwan. > I encountered problem. > I tried to build simple-client.py in > the following URL. > https://github.com/bro/bro-netcontrol/tree/master/test > > It seems like some packages of the version are different. > enum -> python 3.5.2 > pybroker -> python 2.7.12 > > Can you get me some solutions to build this file? > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Wed Dec 7 12:44:02 2016 From: johanna at icir.org (Johanna Amann) Date: Wed, 7 Dec 2016 12:44:02 -0800 Subject: [Bro] Communications with Broccoli In-Reply-To: References: Message-ID: <20161207204402.vtjutjlkgg5brdi6@Beezling.local> Hi Albert, Broccoli is only a communication library for talking with Bro; it itself does not do any detection. So even if you could compile broccoli on your OpenMote (which I doubt a little bit), it would not do any traffic detection. Apart from that I doubt we are supporting the protocols spoken in such networks (assuming it uses zigbee or sth. like that). So - you probably have to write something yourself here, unless I misunderstand your use-case. Johanna On Wed, Dec 07, 2016 at 09:03:50AM +0100, Alberto Ciolini wrote: > Good evening everyone. I am a computer engineering student at the > University of Florence and I was assigned a project for my thesis. > Basically I have 2 devices, udoo board and OpenMote, which must communicate > with each other. OpenMote is my sensor network and udoo is my base station. > So when OpenMote detects something odd in the network sends a signal to the > base station. My problem is that OpenMote is a small device with low memory > and little RAM which requires flash to load programs written in C > exclusively. So as you understand I cannot install completely Bro but I > have to work through the Broccoli library and I don't know how to write the > program that does what I wrote above. Do you have any idea? > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From charles.fair at mac.com Wed Dec 7 14:50:33 2016 From: charles.fair at mac.com (Charles A. Fair) Date: Wed, 07 Dec 2016 17:50:33 -0500 Subject: [Bro] Project help In-Reply-To: References: Message-ID: Alberto, How are the Udoo system and OpenMode system communicating? I would guess you are using Wifi for the connectivity? If you were going to install Bro on one of those two devices, it probably should be the Udoo. More practically, since the Udoo has an ethernet port, sniff the ethernet traffic from an inexpensive port mirroring switch, such as any Mikrotik switch, or a Dualcomm ?tap? which is actually also a port mirroring switch, with dedicated ports. You would then install Bro on a 3rd system and monitor from there. Chuck > On Dec 6, 2016, at 12:38 PM, Alberto Ciolini wrote: > > Good evening everyone. I am a computer engineering student at the University of Florence and I was assigned a project for my thesis. > Basically I have 2 devices, udoo board and OpenMote, which must communicate with each other. OpenMote is my sensor network and udoo is my base station. So when OpenMote detects something odd in the network sends a signal to the base station. My problem is that OpenMote is a small device with low memory and little RAM which requires flash to load programs written in C exclusively. So as you understand I cannot install completely Bro but I have to work through the Broccoli library and I don't know how to write the program that does what I wrote above. Do you have any idea? > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From gary.w.weasel2.civ at mail.mil Thu Dec 8 06:44:34 2016 From: gary.w.weasel2.civ at mail.mil (Weasel, Gary W Jr CIV DISA RE (US)) Date: Thu, 8 Dec 2016 14:44:34 +0000 Subject: [Bro] Bro 2.5 and FIPS Message-ID: <0C34D9CA9B9DBB45B1C51871C177B4B285CF81D7@UMECHPA68.easf.csd.disa.mil> Hello, I'm attempting to run Bro 2.5 on a system that is in FIPS 140-2 compliance mode. However, any time that I attempt to run anything Bro related, I end up with MD5 Digest errors, such as: md5_dgst.c(80): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode! Is there any configuration in Bro somewhere that I can change to solve this, where Bro is compatible with a system that's FIPS enabled? Is that something I would only be able to deal with when compiling Bro from source, or is there a way to run Bro at all in FIPS mode? Thanks, - Gary From hosom at battelle.org Thu Dec 8 10:15:45 2016 From: hosom at battelle.org (Hosom, Stephen M) Date: Thu, 8 Dec 2016 18:15:45 +0000 Subject: [Bro] Bro 2.5 and FIPS In-Reply-To: <0C34D9CA9B9DBB45B1C51871C177B4B285CF81D7@UMECHPA68.easf.csd.disa.mil> References: <0C34D9CA9B9DBB45B1C51871C177B4B285CF81D7@UMECHPA68.easf.csd.disa.mil> Message-ID: The problem is caused by the fact that Bro needs to process certs that make use of md5 and in order to do that it uses the portions of OpenSSL that handle md5...which have been disabled. As for the fix? I'm not actually sure. No matter how you swing it, you really do want to be able to use those portions of the library for network monitoring purposes. The only thing I can think of that might get you out of this is to link against an alternate version of OpenSSL that you use solely for Bro that disables FIPS mode... that way you have it enabled for most applications, but disabled for Bro. I didn't have long to look into how FIPS mode is set, but it looks like in your case it may have been a build time option. -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Weasel, Gary W Jr CIV DISA RE (US) Sent: Thursday, December 8, 2016 9:45 AM To: 'bro at bro.org' Subject: [Bro] Bro 2.5 and FIPS Importance: High Hello, I'm attempting to run Bro 2.5 on a system that is in FIPS 140-2 compliance mode. However, any time that I attempt to run anything Bro related, I end up with MD5 Digest errors, such as: md5_dgst.c(80): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode! Is there any configuration in Bro somewhere that I can change to solve this, where Bro is compatible with a system that's FIPS enabled? Is that something I would only be able to deal with when compiling Bro from source, or is there a way to run Bro at all in FIPS mode? Thanks, - Gary _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Thu Dec 8 11:09:27 2016 From: johanna at icir.org (Johanna Amann) Date: Thu, 08 Dec 2016 11:09:27 -0800 Subject: [Bro] Bro 2.5 and FIPS In-Reply-To: References: <0C34D9CA9B9DBB45B1C51871C177B4B285CF81D7@UMECHPA68.easf.csd.disa.mil> Message-ID: Actually, MD5 certificates don't really happen anymore in practice, and OpenSSL would do the verification itself, which probably won't give an assertion. While we don't support a configuration of Bro out of the Box that does not use MD5, I think you might actually be able to accomplish this without changing too much. I would try downloading the source, going into scripts/base and commenting all calls that look like... Files::add_analyzer(f, Files::ANALYZER_MD5); There only are a few places that do that (mainly certificates are hashed by default); however, I don't think we really need that. You probably also need to stay away from using bloom filters. But - that might be good enough to eliminate all traditional digest MD5 calls in the base configuration. Johanna On 8 Dec 2016, at 10:15, Hosom, Stephen M wrote: > The problem is caused by the fact that Bro needs to process certs that > make use of md5 and in order to do that it uses the portions of > OpenSSL that handle md5...which have been disabled. As for the fix? > I'm not actually sure. No matter how you swing it, you really do want > to be able to use those portions of the library for network monitoring > purposes. The only thing I can think of that might get you out of this > is to link against an alternate version of OpenSSL that you use solely > for Bro that disables FIPS mode... that way you have it enabled for > most applications, but disabled for Bro. I didn't have long to look > into how FIPS mode is set, but it looks like in your case it may have > been a build time option. > > -----Original Message----- > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of > Weasel, Gary W Jr CIV DISA RE (US) > Sent: Thursday, December 8, 2016 9:45 AM > To: 'bro at bro.org' > Subject: [Bro] Bro 2.5 and FIPS > Importance: High > > Hello, > > I'm attempting to run Bro 2.5 on a system that is in FIPS 140-2 > compliance mode. However, any time that I attempt to run anything Bro > related, I end up with MD5 Digest errors, such as: > > md5_dgst.c(80): OpenSSL internal error, assertion failed: Digest MD5 > forbidden in FIPS mode! > > Is there any configuration in Bro somewhere that I can change to solve > this, where Bro is compatible with a system that's FIPS enabled? Is > that something I would only be able to deal with when compiling Bro > from source, or is there a way to run Bro at all in FIPS mode? > > Thanks, > - Gary > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From gc355804 at ohio.edu Thu Dec 8 17:28:15 2016 From: gc355804 at ohio.edu (Clark, Gilbert) Date: Fri, 9 Dec 2016 01:28:15 +0000 Subject: [Bro] Project help In-Reply-To: References: , Message-ID: If I had to do this as described, I think I'd build some custom code on the OpenMote sensor that would take the events it detected, pack them into a custom protocol, and shoot those events to an ingest process running on the Udoo via an appropriate link. If Bro is desired, then one can build the ingest process on the Udoo as a Bro plugin. This plugin would listen for the special incoming event packets that came from the sensor, read them, and use those event packets to generate / log / etc. custom bro events on the Udoo. The principle would be similar to broccoli, but the idea would be to keep unnecessary overhead on the OpenMote to a minimum. My guess is that the sensor is probably going to need to devote the vast majority of its resources to both classifying "weird" traffic and running the stack necessary to communicate over whatever protocol it might use (802.15.4 / Zigbee?) in the first place. I really doubt the sensor is going to be able to do much classification, either ... For any kind of more significant analysis, I'd either use a passive receiver or, if one of those weren't available and / or the sensor were doing something weird, something like a RTL-SDR or HackRF to capture raw I/Q and demodulate / decode into relevant packets that could be passed along to a bro worker running on a more powerful host. That approach, though, would indeed most likely involve writing custom code to parse Zigbee / 802.15.4 / whatever. Just a few thoughts, Gilbert Clark ________________________________ From: bro-bounces at bro.org on behalf of Charles A. Fair Sent: Wednesday, December 7, 2016 5:50:33 PM To: Alberto Ciolini Cc: bro at bro-ids.org Subject: Re: [Bro] Project help Alberto, How are the Udoo system and OpenMode system communicating? I would guess you are using Wifi for the connectivity? If you were going to install Bro on one of those two devices, it probably should be the Udoo. More practically, since the Udoo has an ethernet port, sniff the ethernet traffic from an inexpensive port mirroring switch, such as any Mikrotik switch, or a Dualcomm "tap" which is actually also a port mirroring switch, with dedicated ports. You would then install Bro on a 3rd system and monitor from there. Chuck > On Dec 6, 2016, at 12:38 PM, Alberto Ciolini wrote: > > Good evening everyone. I am a computer engineering student at the University of Florence and I was assigned a project for my thesis. > Basically I have 2 devices, udoo board and OpenMote, which must communicate with each other. OpenMote is my sensor network and udoo is my base station. So when OpenMote detects something odd in the network sends a signal to the base station. My problem is that OpenMote is a small device with low memory and little RAM which requires flash to load programs written in C exclusively. So as you understand I cannot install completely Bro but I have to work through the Broccoli library and I don't know how to write the program that does what I wrote above. Do you have any idea? > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro Bro Info Page - University of California, Berkeley mailman.icsi.berkeley.edu This is a mailing list for users and developers of the Bro intrusion detection system. See bro.org for general information about Bro. To see the collection of prior ... _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro Bro Info Page - University of California, Berkeley mailman.icsi.berkeley.edu This is a mailing list for users and developers of the Bro intrusion detection system. See bro.org for general information about Bro. To see the collection of prior ... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161209/61e22dc3/attachment.html From gary.w.weasel2.civ at mail.mil Fri Dec 9 06:40:08 2016 From: gary.w.weasel2.civ at mail.mil (Weasel, Gary W Jr CIV DISA RE (US)) Date: Fri, 9 Dec 2016 14:40:08 +0000 Subject: [Bro] [Non-DoD Source] Re: Bro 2.5 and FIPS In-Reply-To: References: <0C34D9CA9B9DBB45B1C51871C177B4B285CF81D7@UMECHPA68.easf.csd.disa.mil> Message-ID: <0C34D9CA9B9DBB45B1C51871C177B4B285CFE2EC@UMECHPA68.easf.csd.disa.mil> Unfortunately it doesn't seem to be that simple. Commenting out all the references to ANALYZER_MD5 in the scripts didn't make any difference in attempting to run the program, and it seems anything that uses the openssl md5 wrapper probably is what gets stopped. I'm going through the code trying to see what happens if I just try to remove all the md5 usage for that wrapper, but honestly my current expectation is that this won't even succeed in make. v/r Gary W. Weasel, Jr. | Computer Engineer Incident Response and Recovery Team, RE62 COM: 717.267.5777 -----Original Message----- From: Johanna Amann [mailto:johanna at icir.org] Sent: Thursday, December 8, 2016 2:09 PM To: Hosom, Stephen M Cc: Weasel, Gary W Jr CIV DISA RE (US) ; bro at bro.org Subject: [Non-DoD Source] Re: [Bro] Bro 2.5 and FIPS All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. ---- Actually, MD5 certificates don't really happen anymore in practice, and OpenSSL would do the verification itself, which probably won't give an assertion. While we don't support a configuration of Bro out of the Box that does not use MD5, I think you might actually be able to accomplish this without changing too much. I would try downloading the source, going into scripts/base and commenting all calls that look like... Files::add_analyzer(f, Files::ANALYZER_MD5); There only are a few places that do that (mainly certificates are hashed by default); however, I don't think we really need that. You probably also need to stay away from using bloom filters. But - that might be good enough to eliminate all traditional digest MD5 calls in the base configuration. Johanna On 8 Dec 2016, at 10:15, Hosom, Stephen M wrote: > The problem is caused by the fact that Bro needs to process certs that > make use of md5 and in order to do that it uses the portions of > OpenSSL that handle md5...which have been disabled. As for the fix? > I'm not actually sure. No matter how you swing it, you really do want > to be able to use those portions of the library for network monitoring > purposes. The only thing I can think of that might get you out of this > is to link against an alternate version of OpenSSL that you use solely > for Bro that disables FIPS mode... that way you have it enabled for > most applications, but disabled for Bro. I didn't have long to look > into how FIPS mode is set, but it looks like in your case it may have > been a build time option. > > -----Original Message----- > From: bro-bounces at bro.org [Caution-mailto:bro-bounces at bro.org] On Behalf Of > Weasel, Gary W Jr CIV DISA RE (US) > Sent: Thursday, December 8, 2016 9:45 AM > To: 'bro at bro.org' > Subject: [Bro] Bro 2.5 and FIPS > Importance: High > > Hello, > > I'm attempting to run Bro 2.5 on a system that is in FIPS 140-2 > compliance mode. However, any time that I attempt to run anything Bro > related, I end up with MD5 Digest errors, such as: > > md5_dgst.c(80): OpenSSL internal error, assertion failed: Digest MD5 > forbidden in FIPS mode! > > Is there any configuration in Bro somewhere that I can change to solve > this, where Bro is compatible with a system that's FIPS enabled? Is > that something I would only be able to deal with when compiling Bro > from source, or is there a way to run Bro at all in FIPS mode? > > Thanks, > - Gary > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > Caution-http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > Caution-http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From hosom at battelle.org Fri Dec 9 10:06:30 2016 From: hosom at battelle.org (Hosom, Stephen M) Date: Fri, 9 Dec 2016 18:06:30 +0000 Subject: [Bro] [Non-DoD Source] Re: Bro 2.5 and FIPS In-Reply-To: <0C34D9CA9B9DBB45B1C51871C177B4B285CFE2EC@UMECHPA68.easf.csd.disa.mil> References: <0C34D9CA9B9DBB45B1C51871C177B4B285CF81D7@UMECHPA68.easf.csd.disa.mil> <0C34D9CA9B9DBB45B1C51871C177B4B285CFE2EC@UMECHPA68.easf.csd.disa.mil> Message-ID: Gary, Is creating a policy exception possible for you? Honestly, I work in similar environments and it I would advise against making major modifications to Bro's source code. Document the fact that in the event of a vulnerability in your system you won't be able to be as responsive to the vulnerability. Significant source code modifications will hamper your ability to pull a patch in quickly. Since Bro doesn't use md5 anywhere that this really matters... I would hope that your compliance team is willing to be reasonable about this. Thanks, Stephen -----Original Message----- From: Weasel, Gary W Jr CIV DISA RE (US) [mailto:gary.w.weasel2.civ at mail.mil] Sent: Friday, December 9, 2016 9:40 AM To: 'johanna at icir.org' ; Hosom, Stephen M Cc: 'bro at bro.org' Subject: RE: [Non-DoD Source] Re: [Bro] Bro 2.5 and FIPS Unfortunately it doesn't seem to be that simple. Commenting out all the references to ANALYZER_MD5 in the scripts didn't make any difference in attempting to run the program, and it seems anything that uses the openssl md5 wrapper probably is what gets stopped. I'm going through the code trying to see what happens if I just try to remove all the md5 usage for that wrapper, but honestly my current expectation is that this won't even succeed in make. v/r Gary W. Weasel, Jr. | Computer Engineer Incident Response and Recovery Team, RE62 COM: 717.267.5777 -----Original Message----- From: Johanna Amann [mailto:johanna at icir.org] Sent: Thursday, December 8, 2016 2:09 PM To: Hosom, Stephen M Cc: Weasel, Gary W Jr CIV DISA RE (US) ; bro at bro.org Subject: [Non-DoD Source] Re: [Bro] Bro 2.5 and FIPS All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. ---- Actually, MD5 certificates don't really happen anymore in practice, and OpenSSL would do the verification itself, which probably won't give an assertion. While we don't support a configuration of Bro out of the Box that does not use MD5, I think you might actually be able to accomplish this without changing too much. I would try downloading the source, going into scripts/base and commenting all calls that look like... Files::add_analyzer(f, Files::ANALYZER_MD5); There only are a few places that do that (mainly certificates are hashed by default); however, I don't think we really need that. You probably also need to stay away from using bloom filters. But - that might be good enough to eliminate all traditional digest MD5 calls in the base configuration. Johanna On 8 Dec 2016, at 10:15, Hosom, Stephen M wrote: > The problem is caused by the fact that Bro needs to process certs that > make use of md5 and in order to do that it uses the portions of > OpenSSL that handle md5...which have been disabled. As for the fix? > I'm not actually sure. No matter how you swing it, you really do want > to be able to use those portions of the library for network monitoring > purposes. The only thing I can think of that might get you out of this > is to link against an alternate version of OpenSSL that you use solely > for Bro that disables FIPS mode... that way you have it enabled for > most applications, but disabled for Bro. I didn't have long to look > into how FIPS mode is set, but it looks like in your case it may have > been a build time option. > > -----Original Message----- > From: bro-bounces at bro.org [Caution-mailto:bro-bounces at bro.org] On > Behalf Of Weasel, Gary W Jr CIV DISA RE (US) > Sent: Thursday, December 8, 2016 9:45 AM > To: 'bro at bro.org' > Subject: [Bro] Bro 2.5 and FIPS > Importance: High > > Hello, > > I'm attempting to run Bro 2.5 on a system that is in FIPS 140-2 > compliance mode. However, any time that I attempt to run anything Bro > related, I end up with MD5 Digest errors, such as: > > md5_dgst.c(80): OpenSSL internal error, assertion failed: Digest MD5 > forbidden in FIPS mode! > > Is there any configuration in Bro somewhere that I can change to solve > this, where Bro is compatible with a system that's FIPS enabled? Is > that something I would only be able to deal with when compiling Bro > from source, or is there a way to run Bro at all in FIPS mode? > > Thanks, > - Gary > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > Caution-http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > Caution-http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From tcarpenter604 at gmail.com Fri Dec 9 10:22:15 2016 From: tcarpenter604 at gmail.com (Todd Carpenter) Date: Fri, 9 Dec 2016 10:22:15 -0800 Subject: [Bro] Best set up practice Message-ID: <1500FC08-38ED-4394-8469-65387DA0E8F0@gmail.com> Hi all, Just joined the list and had a question ? that I apparently sent to customer support ..oops. anyways Im building a freebsd server and was wondering what the best practice / placement for bro would be Essentially It?s a forward facing firewall based on freebsd. SO I was wondering if its best to deploy on the host OS, or create a jail or two and funnel traffic through that? I also wanted to know if there were any special considerations with jails / setup. some options I came up with .. internet > firewall > lan/dmz internet > firewall > nginx proxy > lan/dmz internet > firewall > dmz jail > NO lan internet > firewall > bro jail > proxy jail > lan/dmz Thanks! From derek.ditch at criticalstack.com Fri Dec 9 15:57:27 2016 From: derek.ditch at criticalstack.com (Ditch, Derek) Date: Fri, 9 Dec 2016 23:57:27 +0000 Subject: [Bro] RHEL7 and AF_PACKET In-Reply-To: References: Message-ID: Erik, Is this in response to your earlier post regarding AF_PACKET plugin on 6.8? I use AF_PACKET w/ Bro 2.5 on CentOS 7 and RHEL 7.2 every day, in production, using the default production kernel of 3.10. While kernel 3.10 is the minimum to support functional AF_PACKET, more recent patches have improved performance, fixed bugs, etc. The often-misunderstood notion, however, is that RHEL uses an (relatively) ancient kernel in 3.10. However, Red Hat backports patches and have been pretty responsive to my interactions with them. To be clear, AF_PACKET on RHEL7 and CentOS7 work extremely well w/ Bro 2.5 and the af_packet plugin. It will not, however, work under RHEL 6 because it uses the 2.x kernel. -Derek From: on behalf of erik clark Date: Wednesday, December 7, 2016 at 06:34 To: Bro-IDS Subject: [Bro] RHEL7 and AF_PACKET Short answer: No, AF_PACKET will not work with RHEL7. The long of it, from RedHat direct, is: --- The eb70db875 fix is included in upstream v4.7 so if you need this feature at the cost of everything else, you could use ELRepo's kernel-ml package of v4.7 or later. I tested their kernel-ml-4.8.12-1.el7.elrepo, recompiled the go application, and the test passes fine. --- I am working with RH on this, and maybe with luck it will make it into RH7.4. I will keep everyone posted. :) Erik ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161209/de0e0321/attachment.html From jazoff at illinois.edu Fri Dec 9 16:02:55 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Sat, 10 Dec 2016 00:02:55 +0000 Subject: [Bro] RHEL7 and AF_PACKET In-Reply-To: References: Message-ID: > On Dec 9, 2016, at 5:57 PM, Ditch, Derek wrote: > > To be clear, AF_PACKET on RHEL7 and CentOS7 work extremely well w/ Bro 2.5 and the af_packet plugin. It will not, however, work under RHEL 6 because it uses the 2.x kernel. Is this with a single worker or multiple workers? A single worker would work fine, but as far as I can tell hash based fanout is broken. If bro is working for you, any ideas why https://github.com/JustinAzoff/can-i-use-afpacket-fanout/ fails to run properly on Centos 7? -- - Justin Azoff From baxter.milliwew at gmail.com Fri Dec 9 16:48:49 2016 From: baxter.milliwew at gmail.com (Baxter Milliwew) Date: Fri, 9 Dec 2016 16:48:49 -0800 Subject: [Bro] Best, set up, practice Message-ID: Try byhve ? On Fri, Dec 9, 2016 at 10:22 AM, Todd Carpenter wrote: > Hi all, > > Just joined the list and had a question ? that I apparently sent to > customer support ..oops. > > anyways Im building a freebsd server and was wondering what the best > practice / placement for bro would be > > Essentially It?s a forward facing firewall based on freebsd. SO I was > wondering if its best to deploy on the host OS, or create a jail or two and > funnel traffic through that? I also wanted to know if there were any > special considerations with jails / setup. > > some options I came up with .. > > internet > firewall > lan/dmz > internet > firewall > nginx proxy > lan/dmz > internet > firewall > dmz jail > NO lan > internet > firewall > bro jail > proxy jail > lan/dmz > > Thanks! > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161209/232fb8ef/attachment.html From derek.ditch at criticalstack.com Fri Dec 9 19:23:50 2016 From: derek.ditch at criticalstack.com (Ditch, Derek) Date: Sat, 10 Dec 2016 03:23:50 +0000 Subject: [Bro] RHEL7 and AF_PACKET In-Reply-To: References: Message-ID: Justin, I haven?t used your tool before. That?s interesting?I tested in my ROCK NSM dev VM and it failed. When I switched to the El Repo kernel it had no problem. On production sensors w/ AF_PACKET I get ~ 0.06% packet loss. I?ll have to dig deeper on this. Your go app fails on my production sensor too, but I never had sufficient packet loss to dig into it. Have you submitted an issue with Red Hat to get the fix backported? If so, can you post the bug tracker number? -Derek On 12/9/16, 18:02, "Azoff, Justin S" wrote: > On Dec 9, 2016, at 5:57 PM, Ditch, Derek wrote: > > To be clear, AF_PACKET on RHEL7 and CentOS7 work extremely well w/ Bro 2.5 and the af_packet plugin. It will not, however, work under RHEL 6 because it uses the 2.x kernel. Is this with a single worker or multiple workers? A single worker would work fine, but as far as I can tell hash based fanout is broken. If bro is working for you, any ideas why https://github.com/JustinAzoff/can-i-use-afpacket-fanout/ fails to run properly on Centos 7? -- - Justin Azoff ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer. From shirkdog.bsd at gmail.com Sat Dec 10 08:49:13 2016 From: shirkdog.bsd at gmail.com (Michael Shirk) Date: Sat, 10 Dec 2016 11:49:13 -0500 Subject: [Bro] Best set up practice In-Reply-To: <1500FC08-38ED-4394-8469-65387DA0E8F0@gmail.com> References: <1500FC08-38ED-4394-8469-65387DA0E8F0@gmail.com> Message-ID: In the FreeBSD sense, jail all the things. You will be able to find some write-ups for Snort, but not so much for Bro, which I will look to create and blog about. The main thing is that when you setup the jail, make sure the jail is configured for the interface you wish to monitor. You world normally monitor the LAN side, but you could have a separate jail configured to monitor the external side in a separate jail looking for threats and traffic making it in and out of your firewall. A couple of additional items I myself have not had the chance to play with but should be possible in Bro 2.5 is the ability to interact with ipfw/pf with the NetControl Framework to use update the firewall on the fly, also for shunting flows. As far as logging, I normally stick to the standard Bro log files, and you can run tools from the host OS to process the log files in the jail if you want. -- Michael Shirk Daemon Security, Inc. http://www.daemon-security.com On Dec 9, 2016 13:31, "Todd Carpenter" wrote: > Hi all, > > Just joined the list and had a question ? that I apparently sent to > customer support ..oops. > > anyways Im building a freebsd server and was wondering what the best > practice / placement for bro would be > > Essentially It?s a forward facing firewall based on freebsd. SO I was > wondering if its best to deploy on the host OS, or create a jail or two and > funnel traffic through that? I also wanted to know if there were any > special considerations with jails / setup. > > some options I came up with .. > > internet > firewall > lan/dmz > internet > firewall > nginx proxy > lan/dmz > internet > firewall > dmz jail > NO lan > internet > firewall > bro jail > proxy jail > lan/dmz > > Thanks! > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161210/4b598386/attachment.html From cgaylord at vt.edu Sat Dec 10 09:16:44 2016 From: cgaylord at vt.edu (Clark Gaylord) Date: Sat, 10 Dec 2016 12:16:44 -0500 Subject: [Bro] Linux vs FreeBSD In-Reply-To: References: Message-ID: Yeah this is probably a faq but thought I'd see if, especially with newer bro, there's a prevailing wind. I've been running single 1gig Linux bro box for a little over a year and it just hums along. For my Christmas project, I'm going to upgrade to 10 gig (probably only single) and will likely rebuild the box while I'm at it. A second 10 gig is possible in the future. I'm comfortable with both FreeBSD and Linux and will have Myricom 10 gig NIC. Thoughts/suggestions regarding implementation choices? -- Clark Gaylord cgaylord at vt.edu ... autocorrect may have improved this message brevity should not be interpreted as curtness ... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161210/757ecf41/attachment.html From shirkdog.bsd at gmail.com Sat Dec 10 10:05:47 2016 From: shirkdog.bsd at gmail.com (Michael Shirk) Date: Sat, 10 Dec 2016 13:05:47 -0500 Subject: [Bro] Linux vs FreeBSD In-Reply-To: References: Message-ID: There are at least 2 large FreeBSD installations monitoring 10-50Gbps (one that can scale to 100Gbps). I believe both utilize Myricom cards so from a performance perspective, you can get the necessary performance out of FreeBSD. My focus has been on the low end of this (1-10Gbps) getting FreeBSD to scale on commodity hardware, as FreeBSD 12 will have updated netmap code for better packet I/O. Though I live and die by BSD, it is up to you what you feel more comfortable with, for maintenance and operations. With management tools like Salt/Ansible/Puppet, it really is a personal/organizational preference as to which one you use. FreeBSD 11 has shown better network performance compared to some Linux distros in a recent test with netperf/iperf, but the test did not cover packet monitoring: https://www.phoronix.com/scan.php?page=article&item=netperf-bsd-linux&num=3 -- Michael Shirk Daemon Security, Inc. http://www.daemon-security.com On Dec 10, 2016 12:31, "Clark Gaylord" wrote: Yeah this is probably a faq but thought I'd see if, especially with newer bro, there's a prevailing wind. I've been running single 1gig Linux bro box for a little over a year and it just hums along. For my Christmas project, I'm going to upgrade to 10 gig (probably only single) and will likely rebuild the box while I'm at it. A second 10 gig is possible in the future. I'm comfortable with both FreeBSD and Linux and will have Myricom 10 gig NIC. Thoughts/suggestions regarding implementation choices? -- Clark Gaylord cgaylord at vt.edu ... autocorrect may have improved this message brevity should not be interpreted as curtness ... _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161210/c3f514e4/attachment.html From abdulrahmanmusallam at gmail.com Sun Dec 11 12:17:53 2016 From: abdulrahmanmusallam at gmail.com (abdulrahman musallam) Date: Sun, 11 Dec 2016 22:17:53 +0200 Subject: [Bro] id.orig_h and id.resp_h fields. Message-ID: hi, i've noticed that in conn.log under id.orig_h and id.resp_h fields it does not show a normal IPv4 address insted i get somthing similar to this ff02::fb or this fe80::219:e3ff:fee7:5d23 can anyone please explain what do these IDs mean? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161211/0a234b07/attachment.html From tgdesrochers at gmail.com Sun Dec 11 12:28:12 2016 From: tgdesrochers at gmail.com (Tim Desrochers) Date: Sun, 11 Dec 2016 15:28:12 -0500 Subject: [Bro] id.orig_h and id.resp_h fields. In-Reply-To: References: Message-ID: That is ipv6 addresding On Dec 11, 2016 15:26, "abdulrahman musallam" wrote: > hi, > i've noticed that in conn.log under id.orig_h and id.resp_h fields it does > not show a normal IPv4 address insted i get somthing similar to this ff02::fb > or this fe80::219:e3ff:fee7:5d23 > can anyone please explain what do these IDs mean? > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161211/503b90ad/attachment.html From philosnef at gmail.com Sun Dec 11 16:56:42 2016 From: philosnef at gmail.com (erik clark) Date: Sun, 11 Dec 2016 19:56:42 -0500 Subject: [Bro] RHEL7 and AF_PACKET In-Reply-To: References: Message-ID: I have a bug report with RH. It is being worked on. It MAY make it into 7.4. The solution from RH is to use the elrepo kernel. I haven't been back to work yet, but I may be getting a test kernel to work with to help get this into the main branch earlier than 7.4. Per RH, the permanent fix isn't that bad, it just touches on a bunch of things at once making it undesireable to push into production immediately. On Fri, Dec 9, 2016 at 10:23 PM, Ditch, Derek wrote: > Justin, > > I haven?t used your tool before. That?s interesting?I tested in my ROCK > NSM dev VM and it failed. When I switched to the El Repo kernel it had no > problem. On production sensors w/ AF_PACKET I get ~ 0.06% packet loss. I?ll > have to dig deeper on this. Your go app fails on my production sensor too, > but I never had sufficient packet loss to dig into it. > > Have you submitted an issue with Red Hat to get the fix backported? If so, > can you post the bug tracker number? > > -Derek > > On 12/9/16, 18:02, "Azoff, Justin S" wrote: > > > > On Dec 9, 2016, at 5:57 PM, Ditch, Derek < > derek.ditch at criticalstack.com> wrote: > > > > To be clear, AF_PACKET on RHEL7 and CentOS7 work extremely well w/ > Bro 2.5 and the af_packet plugin. It will not, however, work under RHEL 6 > because it uses the 2.x kernel. > > Is this with a single worker or multiple workers? > > A single worker would work fine, but as far as I can tell hash based > fanout is broken. > > If bro is working for you, any ideas why https://github.com/ > JustinAzoff/can-i-use-afpacket-fanout/ fails to run properly on Centos 7? > > -- > - Justin Azoff > > > > ________________________________________________________ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161211/463c6f60/attachment.html From philosnef at gmail.com Sun Dec 11 16:58:05 2016 From: philosnef at gmail.com (erik clark) Date: Sun, 11 Dec 2016 19:58:05 -0500 Subject: [Bro] RHEL7 and AF_PACKET In-Reply-To: References: Message-ID: Also, I get very low packet loss when using AF_PACKET on 7.3, BUT, conn and weird logs go absolutely bonkers, and long term conns are trashed because traffic goes out one worker but back in on a different one. This is a big issue for me, as we were going to go AF_PAcKET with suricata as well. On Sun, Dec 11, 2016 at 7:56 PM, erik clark wrote: > I have a bug report with RH. It is being worked on. It MAY make it into > 7.4. The solution from RH is to use the elrepo kernel. I haven't been back > to work yet, but I may be getting a test kernel to work with to help get > this into the main branch earlier than 7.4. Per RH, the permanent fix isn't > that bad, it just touches on a bunch of things at once making it > undesireable to push into production immediately. > > On Fri, Dec 9, 2016 at 10:23 PM, Ditch, Derek < > derek.ditch at criticalstack.com> wrote: > >> Justin, >> >> I haven?t used your tool before. That?s interesting?I tested in my ROCK >> NSM dev VM and it failed. When I switched to the El Repo kernel it had no >> problem. On production sensors w/ AF_PACKET I get ~ 0.06% packet loss. I?ll >> have to dig deeper on this. Your go app fails on my production sensor too, >> but I never had sufficient packet loss to dig into it. >> >> Have you submitted an issue with Red Hat to get the fix backported? If >> so, can you post the bug tracker number? >> >> -Derek >> >> On 12/9/16, 18:02, "Azoff, Justin S" wrote: >> >> >> > On Dec 9, 2016, at 5:57 PM, Ditch, Derek < >> derek.ditch at criticalstack.com> wrote: >> > >> > To be clear, AF_PACKET on RHEL7 and CentOS7 work extremely well w/ >> Bro 2.5 and the af_packet plugin. It will not, however, work under RHEL 6 >> because it uses the 2.x kernel. >> >> Is this with a single worker or multiple workers? >> >> A single worker would work fine, but as far as I can tell hash based >> fanout is broken. >> >> If bro is working for you, any ideas why >> https://github.com/JustinAzoff/can-i-use-afpacket-fanout/ fails to run >> properly on Centos 7? >> >> -- >> - Justin Azoff >> >> >> >> ________________________________________________________ >> >> The information contained in this e-mail is confidential and/or >> proprietary to Capital One and/or its affiliates and may only be used >> solely in performance of work or services for Capital One. The information >> transmitted herewith is intended only for use by the individual or entity >> to which it is addressed. If the reader of this message is not the intended >> recipient, you are hereby notified that any review, retransmission, >> dissemination, distribution, copying or other use of, or taking of any >> action in reliance upon this information is strictly prohibited. If you >> have received this communication in error, please contact the sender and >> delete the material from your computer. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161211/7fe9bc34/attachment.html From jholmes at psu.edu Mon Dec 12 13:28:03 2016 From: jholmes at psu.edu (Jason Holmes) Date: Mon, 12 Dec 2016 16:28:03 -0500 Subject: [Bro] SSH Geodata Lookup Failures in 2.5 Message-ID: <5808bf4f-bf74-266f-0d81-dfb9c435a8ac@psu.edu> Hi, Since upgrading to Bro 2.5, we've seen some odd behavior with the geodata lookups in the SSH logs. In particular, the remote_location.* fields in the SSH logs are always missing the geodata when auth_success is true. For example, here are stats for a day running 2.4-709 and a day running 2.5: Bro version, auth_success, country_code logged, country_code not logged ----------------------------------------------------------------------- 2.4-709, T, 22169, 26 2.4-709, F, 167400, 10 2.5, T, 0, 23120 2.5, F, 247183, 16 Can anyone confirm that they are also seeing this behavior? I.e., that with 2.5 there is no geodata for successful SSH connections? To confound matters, I looked in the policy/protocols/ssh/geo-data.bro files and I see that when auth_success is true, it's not only supposed to try to log the geodata information, it's also supposed to print a entry in the notice log if the country code that is looked up matches a country code code in the watch list. Here's an example where a notice was logged but the SSH log still doesn't have geodata in it. Based on the code in geo-data.bro, the country code would have had to have been looked up for the notice to be printed, so this seems to indicate that the lookup is successful but it's just not making it to the ssh log. ssh.log ------- 1481518954.665457 CknPAX2R85O0gumn 159.226.238.72 50972 128.XXX.XXX.XXX 22 2 T 1 INBOUND SSH-2.0-PuTTY_Snapshot_2016_11_20.09b7497 SSH-2.0-OpenSSH_5.3 aes128-ctr hmac-md5 none diffie-hellman-group-exchange-sha256 ssh-rsa b6:65:5c:8d:8b:8d:dc:bb:05:58:0d:9e:25:1e:da:37 - - - - - notice.log ---------- 1481519053.725294 CknPAX2R85O0gumn 159.226.238.72 50972 128.XXX.XXX.XXX 22 - - - tcp SSH::Watched_Country_Login SSH login from watched country: CN - 159.226.238.72 128.XXX.XXX.XXX 22 - worker-3-11 Notice::ACTION_LOG 3600.000000 F - - - - - Thanks, -- Jason Holmes From philosnef at gmail.com Tue Dec 13 08:27:28 2016 From: philosnef at gmail.com (erik clark) Date: Tue, 13 Dec 2016 11:27:28 -0500 Subject: [Bro] broctl cron not rotating logs? Message-ID: On my 2.5 manager/logger, I see that broctl cron is not rotating the logs in /data/bro/logs/current? How can I fix this. Right now they are growing rather large because they arent getting rotated out and compressed. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161213/3b351f7d/attachment.html From philosnef at gmail.com Tue Dec 13 08:32:02 2016 From: philosnef at gmail.com (erik clark) Date: Tue, 13 Dec 2016 11:32:02 -0500 Subject: [Bro] broctl cron not rotating logs? In-Reply-To: References: Message-ID: Sorry, ignore this. json format logs are much bigger than tsv, so my comparision of logged data was incorrect. They are in fact being rotated properly. On Tue, Dec 13, 2016 at 11:27 AM, erik clark wrote: > On my 2.5 manager/logger, I see that broctl cron is not rotating the logs > in /data/bro/logs/current? How can I fix this. Right now they are growing > rather large because they arent getting rotated out and compressed. Thanks! > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161213/d283ecbc/attachment.html From albertociolini92 at gmail.com Tue Dec 13 14:50:10 2016 From: albertociolini92 at gmail.com (Alberto Ciolini) Date: Tue, 13 Dec 2016 23:50:10 +0100 Subject: [Bro] Syntax and Semantics message Broccoli Message-ID: I have to develop a program, using Broccoli, that communicates with Bro which installed on raspberry pi. My question is if there is a standard for syntax and semantics for message that Broccoli and Bro send each other. How messages are sent from broccoli to bro? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161213/c430b0da/attachment.html From seth at icir.org Wed Dec 14 06:35:39 2016 From: seth at icir.org (Seth Hall) Date: Wed, 14 Dec 2016 09:35:39 -0500 Subject: [Bro] SSH Geodata Lookup Failures in 2.5 In-Reply-To: <5808bf4f-bf74-266f-0d81-dfb9c435a8ac@psu.edu> References: <5808bf4f-bf74-266f-0d81-dfb9c435a8ac@psu.edu> Message-ID: > On Dec 12, 2016, at 4:28 PM, Jason Holmes wrote: > > Bro version, auth_success, country_code logged, country_code not logged > ----------------------------------------------------------------------- > 2.4-709, T, 22169, 26 > 2.4-709, F, 167400, 10 > 2.5, T, 0, 23120 > 2.5, F, 247183, 16 > > Can anyone confirm that they are also seeing this behavior? I.e., that > with 2.5 there is no geodata for successful SSH connections? I'm curious if you have Bro built against libGeoIP correctly? What you are seeing would indicate to me that it isn't. It's also possible that you don't have the geoip database installed. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jholmes at psu.edu Wed Dec 14 07:57:03 2016 From: jholmes at psu.edu (Jason Holmes) Date: Wed, 14 Dec 2016 10:57:03 -0500 Subject: [Bro] SSH Geodata Lookup Failures in 2.5 In-Reply-To: References: <5808bf4f-bf74-266f-0d81-dfb9c435a8ac@psu.edu> Message-ID: <2f21d4af-fc59-9f45-3974-8364f468744b@psu.edu> On 12/14/16 9:35 AM, Seth Hall wrote: > >> On Dec 12, 2016, at 4:28 PM, Jason Holmes wrote: >> >> Bro version, auth_success, country_code logged, country_code not logged >> ----------------------------------------------------------------------- >> 2.4-709, T, 22169, 26 >> 2.4-709, F, 167400, 10 >> 2.5, T, 0, 23120 >> 2.5, F, 247183, 16 >> >> Can anyone confirm that they are also seeing this behavior? I.e., that >> with 2.5 there is no geodata for successful SSH connections? > > I'm curious if you have Bro built against libGeoIP correctly? What you are seeing would indicate to me that it isn't. It's also possible that you don't have the geoip database installed. Hi Seth, Thanks for your response. GeoIP lookups are working for our HTTP logs (code we added) and the SSH logs when auth_success==F. It's only not working with SSH when auth_success==T, and in this case it apparently is partially working since there are watched country entries in the notice log for successful SSH connections, but the SSH log does not contain the geodata for these successful connections it's altering on (see the two log lines I had in my initial mail for evidence of this). -- Jason Holmes From dnj0496 at gmail.com Wed Dec 14 13:30:56 2016 From: dnj0496 at gmail.com (Dk Jack) Date: Wed, 14 Dec 2016 13:30:56 -0800 Subject: [Bro] deleting table entries Message-ID: Hi, Does bro scripting allow table entries to be deleted while iterating over the entries of the table? My test script below shows a weird behavior. Is there a way to clear the entire contents of the table? I am collecting sum-stats (epoch is 60s) in a table during the epoch_result callback. Then in the epoch_finished callback, I'd like to write the summary results to a log. After that, I'd like to clear the table so that its ready for the next iteration. Iterating and deleting individual elements seems to have a problem as shown in my test script. Any help is appreciated. Thanks. Dk. global test_table: table[count] of count = table(); event bro_init() { local c: count = 0; while (c < 100) { test_table[c] = c; c += 1; } print fmt("test_table size: %d", |test_table|); } event bro_done() { while (|test_table| > 0) { print fmt("test_table size: %d", |test_table|); # this loop should walk all entries in one go. But it doesn't # because of the delete. for (c in test_table) { delete test_table[c]; } } print fmt("test_table size: %d", |test_table|); } #test_table size: 100 #test_table size: 100 #test_table size: 43 #test_table size: 14 #test_table size: 1 #test_table size: 0 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161214/669666b1/attachment.html From philosnef at gmail.com Thu Dec 15 07:01:44 2016 From: philosnef at gmail.com (erik clark) Date: Thu, 15 Dec 2016 10:01:44 -0500 Subject: [Bro] deep cluster documentation & status Message-ID: Is there any additional documentation on the deep cluster as noted here: https://www.bro.org/development/projects/deep-cluster.html I would like to contribute to this, but the status of this project is unclear from the documentation, and there are some requirements that need to be laid out in Bro itself to make this work, such as logging the hostname associated with a given worker node in every log file in order to track node health. The @stats option gives you incremental information for all node types, BUT, that is all it does. Determining from incremental counters when Bro fails or loses capture through a network connectivity issue becomes impossible when all the data in the logger node is intermingled. Having the hostname in all the logs means you can simply track the event count rate (non-incremental) in your visualization tool of choice, like ELK or Splunk. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161215/aea671c4/attachment.html From abdulrahmanmusallam at gmail.com Thu Dec 15 09:45:47 2016 From: abdulrahmanmusallam at gmail.com (abdulrahman musallam) Date: Thu, 15 Dec 2016 19:45:47 +0200 Subject: [Bro] Smart Phone!! Message-ID: Is there any smart phone application that supports bro(shows network statistics, notifications on detection)?? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161215/0b3219a7/attachment.html From dnj0496 at gmail.com Thu Dec 15 13:32:51 2016 From: dnj0496 at gmail.com (Dk Jack) Date: Thu, 15 Dec 2016 13:32:51 -0800 Subject: [Bro] deleting table entries In-Reply-To: References: Message-ID: All, I performed on more experiment. If I iterate over table entries without removing any entries while iterating then it iterates over all entries. However, if I remove one or more entries during iteration, then the loop does not iterate over all the elements. Some elements could be skipped. See the code and the result from my experiment below. Could one of the experts chime. Clearly, this is not a desired behavior. Thanks. Dk. global test_table: table[count] of count = table(); event bro_init() { local c: count = 0; while (c < 100) { test_table[c] = c; c += 1; } print fmt("test_table size: %d", |test_table|); } event bro_done() { local lcount: count = 0; for (c in test_table) { ++lcount; } print fmt("Entries iterated: %d", lcount); while (|test_table| > 0) { lcount = 0; print fmt("Table table size before deletion: %d", |test_table|); for (c in test_table) { delete test_table[c]; ++lcount; } print fmt("Entries deleted: %d", lcount); } print fmt("test_table size: %d", |test_table|); } test_table size: 100 Entries iterated: 100 Table table size before deletion: 100 Entries deleted: 59 Table table size before deletion: 41 Entries deleted: 30 Table table size before deletion: 11 Entries deleted: 11 test_table size: 0 On Wed, Dec 14, 2016 at 1:30 PM, Dk Jack wrote: > Hi, > Does bro scripting allow table entries to be deleted while iterating over > the entries of the table? > My test script below shows a weird behavior. > > Is there a way to clear the entire contents of the table? I am collecting > sum-stats (epoch is 60s) > in a table during the epoch_result callback. Then in the epoch_finished > callback, I'd like to write > the summary results to a log. After that, I'd like to clear the table so > that its ready for the next > iteration. Iterating and deleting individual elements seems to have a > problem as shown in my > test script. Any help is appreciated. Thanks. > > Dk. > > global test_table: table[count] of count = table(); > > event bro_init() > { > local c: count = 0; > > while (c < 100) { > test_table[c] = c; > c += 1; > } > > print fmt("test_table size: %d", |test_table|); > } > > event bro_done() > { > while (|test_table| > 0) { > print fmt("test_table size: %d", |test_table|); > > # this loop should walk all entries in one go. But it doesn't > # because of the delete. > for (c in test_table) { > delete test_table[c]; > } > } > > print fmt("test_table size: %d", |test_table|); > } > > #test_table size: 100 > #test_table size: 100 > #test_table size: 43 > #test_table size: 14 > #test_table size: 1 > #test_table size: 0 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161215/5223e1e5/attachment.html From francois.pennaneach at free.fr Thu Dec 15 14:10:18 2016 From: francois.pennaneach at free.fr (=?UTF-8?Q?Fran=c3=a7ois_Pennaneach?=) Date: Thu, 15 Dec 2016 23:10:18 +0100 Subject: [Bro] deleting table entries In-Reply-To: References: Message-ID: <14ac8483-b04a-a012-04d8-3d80fb0e896d@free.fr> Hi, I think this is the expected (at least, documented) behavior : https://www.bro.org/sphinx/script-reference/statements.html#keyword-for Currently, modifying a container?s membership while iterating over it may result in undefined behavior, so do not add or remove elements inside the loop. Fran?ois Le 15/12/2016 ? 22:32, Dk Jack a ?crit : > All, > I performed on more experiment. If I iterate over table entries > without removing any entries > while iterating then it iterates over all entries. However, if I > remove one or more entries > during iteration, then the loop does not iterate over all the > elements. Some elements > could be skipped. See the code and the result from my experiment below. > > Could one of the experts chime. Clearly, this is not a desired > behavior. Thanks. > > Dk. > > > global test_table: table[count] of count = table(); > > event bro_init() > { > local c: count = 0; > > while (c < 100) { > test_table[c] = c; > c += 1; > } > > print fmt("test_table size: %d", |test_table|); > } > > event bro_done() > { > local lcount: count = 0; > for (c in test_table) { > ++lcount; > } > > print fmt("Entries iterated: %d", lcount); > while (|test_table| > 0) { > lcount = 0; > > print fmt("Table table size before deletion: %d", |test_table|); > for (c in test_table) { > delete test_table[c]; > ++lcount; > } > print fmt("Entries deleted: %d", lcount); > } > > print fmt("test_table size: %d", |test_table|); > } > > > test_table size: 100 > Entries iterated: 100 > Table table size before deletion: 100 > Entries deleted: 59 > Table table size before deletion: 41 > Entries deleted: 30 > Table table size before deletion: 11 > Entries deleted: 11 > test_table size: 0 > > On Wed, Dec 14, 2016 at 1:30 PM, Dk Jack > wrote: > > Hi, > Does bro scripting allow table entries to be deleted while > iterating over the entries of the table? > My test script below shows a weird behavior. > > Is there a way to clear the entire contents of the table? I am > collecting sum-stats (epoch is 60s) > in a table during the epoch_result callback. Then in the > epoch_finished callback, I'd like to write > the summary results to a log. After that, I'd like to clear the > table so that its ready for the next > iteration. Iterating and deleting individual elements seems to > have a problem as shown in my > test script. Any help is appreciated. Thanks. > > Dk. > > global test_table: table[count] of count = table(); > > event bro_init() > { > local c: count = 0; > > while (c < 100) { > test_table[c] = c; > c += 1; > } > > print fmt("test_table size: %d", |test_table|); > } > > event bro_done() > { > while (|test_table| > 0) { > print fmt("test_table size: %d", |test_table|); > > # this loop should walk all entries in one go. But it doesn't > # because of the delete. > for (c in test_table) { > delete test_table[c]; > } > } > > print fmt("test_table size: %d", |test_table|); > } > > #test_table size: 100 > #test_table size: 100 > #test_table size: 43 > #test_table size: 14 > #test_table size: 1 > #test_table size: 0 > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161215/f74bf3fb/attachment-0001.html From jedwards2728 at gmail.com Thu Dec 15 19:09:09 2016 From: jedwards2728 at gmail.com (John Edwards) Date: Fri, 16 Dec 2016 14:09:09 +1100 Subject: [Bro] specific logging per worker Message-ID: Hi all, If i have a cluster that contains 2 workers among a proxy and logger etc, Worker 1 watches and logs everything, Is there a way i can tell worker 2 to only log a specific protocol and not watch everything the Worker 1? thanks John -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161216/3a5737bc/attachment.html From jdopheid at illinois.edu Fri Dec 16 06:59:59 2016 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Fri, 16 Dec 2016 14:59:59 +0000 Subject: [Bro] Bro4Pros 2017: submit CFP proposal Message-ID: Hello Bro Community, If you?re attending Bro4Pros 2017 on Feb. 2nd, consider submitting a CFP proposal. We?re looking for presentations that advanced users can apply to their day-to-day activities as a security professional. Send abstracts (max 500 words) to: info at bro.org Subject: Bro4Pros 2017 Call for Presentations Submission due date: January 6th, 2017 Every presentation is limited to 45 minutes including questions and discussion. Feel free to reach out to us if you have questions. https://www.bro.org/community/bro4pros2017.html Thanks, Jeannette Dopheide ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From jlay at slave-tothe-box.net Fri Dec 16 08:35:31 2016 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 16 Dec 2016 09:35:31 -0700 Subject: [Bro] Quick af_packet question Message-ID: Love the plugin thanks...quick question for cli...does af_packet need -i for multiple interfaces, or can it be used like snort with af_packet::eth0:eth1? Thank you. James From jlay at slave-tothe-box.net Fri Dec 16 08:51:30 2016 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 16 Dec 2016 09:51:30 -0700 Subject: [Bro] Quick af_packet question In-Reply-To: References: Message-ID: So far my testing says yes: 09:30:56 @tester:/opt/bro/spool$] sudo bro -C -i af_packet::eth0:wlan0 listening on eth0:wlan0 eth0 Link encap:Ethernet HWaddr 00:1f:f3:46:62:ca inet addr:192.168.1.7 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::21f:f3ff:fe46:62ca/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:434251 errors:0 dropped:59 overruns:0 frame:0 TX packets:261164 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:600874115 (600.8 MB) TX bytes:70240696 (70.2 MB) Interrupt:16 wlan0 Link encap:Ethernet HWaddr 00:23:6c:7b:29:1d inet addr:192.168.1.60 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::223:6cff:fe7b:291d/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:74 errors:0 dropped:0 overruns:0 frame:0 TX packets:12 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:10726 (10.7 KB) TX bytes:1820 (1.8 KB) ssh.log: 1481906017.175240 CWWs1B3RQhgUy1QqT2 192.168.1.2 45480 192.168.1.7 22 2 T 1 - SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1 chacha20-poly1305 at openssh.com umac-64-etm at openssh.com none curve25519-sha256 at libssh.org ssh-rsa 1481906687.051242 CfvBJT3Gs2r7YAX2n1 192.168.1.2 34956 192.168.1.60 22 2 T 1 - SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1 chacha20-poly1305 at openssh.com umac-64-etm at openssh.com none curve25519-sha256 at libssh.org ssh-rsa but wanting to verify. Thank you. James On 2016-12-16 09:35, James Lay wrote: > Love the plugin thanks...quick question for cli...does af_packet need > -i > for multiple interfaces, or can it be used like snort with > af_packet::eth0:eth1? Thank you. > > James > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jlay at slave-tothe-box.net Fri Dec 16 10:03:44 2016 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 16 Dec 2016 11:03:44 -0700 Subject: [Bro] Quick af_packet question In-Reply-To: References: Message-ID: Does not appear to decode pppoe however :( On 2016-12-16 09:51, James Lay wrote: > So far my testing says yes: > > 09:30:56 @tester:/opt/bro/spool$] sudo bro -C -i af_packet::eth0:wlan0 > listening on eth0:wlan0 > > eth0 Link encap:Ethernet HWaddr 00:1f:f3:46:62:ca > inet addr:192.168.1.7 Bcast:192.168.1.255 > Mask:255.255.255.0 > inet6 addr: fe80::21f:f3ff:fe46:62ca/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:434251 errors:0 dropped:59 overruns:0 frame:0 > TX packets:261164 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:600874115 (600.8 MB) TX bytes:70240696 (70.2 MB) > Interrupt:16 > > wlan0 Link encap:Ethernet HWaddr 00:23:6c:7b:29:1d > inet addr:192.168.1.60 Bcast:192.168.1.255 > Mask:255.255.255.0 > inet6 addr: fe80::223:6cff:fe7b:291d/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:74 errors:0 dropped:0 overruns:0 frame:0 > TX packets:12 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:10726 (10.7 KB) TX bytes:1820 (1.8 KB) > > ssh.log: > 1481906017.175240 CWWs1B3RQhgUy1QqT2 192.168.1.2 45480 > 192.168.1.7 22 2 T 1 - > SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 SSH-2.0-OpenSSH_7.2p2 > Ubuntu-4ubuntu2.1 chacha20-poly1305 at openssh.com > umac-64-etm at openssh.com none curve25519-sha256 at libssh.org ssh-rsa > > 1481906687.051242 CfvBJT3Gs2r7YAX2n1 192.168.1.2 34956 > 192.168.1.60 22 2 T 1 - > SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 SSH-2.0-OpenSSH_7.2p2 > Ubuntu-4ubuntu2.1 chacha20-poly1305 at openssh.com > umac-64-etm at openssh.com none curve25519-sha256 at libssh.org ssh-rsa > > but wanting to verify. Thank you. > > James > > On 2016-12-16 09:35, James Lay wrote: >> Love the plugin thanks...quick question for cli...does af_packet need >> -i >> for multiple interfaces, or can it be used like snort with >> af_packet::eth0:eth1? Thank you. >> >> James >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From renaud.luca at gmail.com Fri Dec 16 15:16:20 2016 From: renaud.luca at gmail.com (Luca Renaud) Date: Fri, 16 Dec 2016 23:16:20 +0000 Subject: [Bro] Detection of backdoors with Bro. Message-ID: I noticed that the bro script Backdoor.bro has been deprecated with Bro 2.5.So,what is now the script or group of scripts (or method) used to deal with this kind of problem.As a use Bro mainly to read tcpdump pcaps of my desktop Internet/browser sessions and malware installed this way is a concern. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161216/1723b492/attachment.html From brot212 at googlemail.com Sat Dec 17 06:00:46 2016 From: brot212 at googlemail.com (Dane Wullen) Date: Sat, 17 Dec 2016 15:00:46 +0100 Subject: [Bro] BinPAC analyzer name In-Reply-To: References: <58210681.4000508@googlemail.com> <20161128230555.bkkqrgeuedtebexa@Beezling.local> <1ab7114a-2307-62d6-ddda-d79374db8044@googlemail.com> <20161128231525.otgkhyf3pnccemcj@Beezling.local> <124db383-ead5-a669-d7a4-54ff95ea9619@googlemail.com> Message-ID: <96b7c945-1c20-9352-a7d7-a57f7650530f@googlemail.com> Hey, sorry, I was kind of busy the last days and couldn't answer. http://pastebin.com/6Z9fykTT <-- all .pac files http://pastebin.com/VBUM5CwE <- all .CC and .h files This is the binPAC code. Like I said, only the standard code, untouched (except the std::cout line). The same code with Test instead of AMS works fine. I don't know, maybe I'm doing something wrong, I've tested it on 3 VMs and on a Raspberry Pi, but the result are always the same. Thanks Dane Am 06.12.2016 um 16:52 schrieb Vlad Grigorescu: > Hmm. Could be a capitalization issue? binpac_quickstart does do some > uppercasing/lowercasing in a couple of the templates. For example: > > Dane Wullen writes: > >> function proc_NAME_message(msg: NAME_PDU) : bool >> ... >> BifEvent::generate_NAME_event(...); >> std::cout << "Name PDU" << endl; # for debugging >> ... > NAME here will always be lowercase (since that matches the naming scheme > for events). > > I poked around with this a bit, and couldn't duplicate it. Can you put > the contents of src/analyzer/protocol/ams somewhere I could see it? > > Thanks, > > --Vlad From ed.sealing at sealingtech.org Sat Dec 17 09:02:49 2016 From: ed.sealing at sealingtech.org (Ed Sealing) Date: Sat, 17 Dec 2016 12:02:49 -0500 Subject: [Bro] CPU usage with no traffic on Bro 2.5 with AF_PACKET Message-ID: I'm seeing ~6% CPU utilization on workers, with no traffic. Is that expected? Is there any way to minimize the CPU load? Using AF_PACKET plugin. The cores are isolated using "isolcpus", so nothing else should be running on them. Workers are pinned to the CPUs in the [worker-1] type=worker host=localhost interface=af_packet::eth1 lb_method=custom lb_procs=14 pin_cpus=1,2,3,4,5,6,7,9,10,11,12,13,14,15 ~Ed -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161217/ee43567d/attachment.html From zeolla at gmail.com Sat Dec 17 09:16:22 2016 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Sat, 17 Dec 2016 17:16:22 +0000 Subject: [Bro] CPU usage with no traffic on Bro 2.5 with AF_PACKET In-Reply-To: References: Message-ID: I have a fork of 2.5 that may help. It's intended to minimize CPU load on sensors that see low volume/sensor-local traffic. Check out the most recent commits, shout out to Justin for the basis of the tweaks. https://github.com/JonZeolla/bro/tree/topic/jonzeolla/low-volume Jon On Sat, Dec 17, 2016, 12:05 Ed Sealing wrote: > I'm seeing ~6% CPU utilization on workers, with no traffic. Is that > expected? Is there any way to minimize the CPU load? > > Using AF_PACKET plugin. The cores are isolated using "isolcpus", so > nothing else should be running on them. Workers are pinned to the CPUs in > the > > [worker-1] > type=worker > host=localhost > interface=af_packet::eth1 > lb_method=custom > lb_procs=14 > pin_cpus=1,2,3,4,5,6,7,9,10,11,12,13,14,15 > > > ~Ed > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Jon Sent from my mobile device -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161217/47b1750a/attachment.html From ed.sealing at sealingtech.org Sat Dec 17 09:54:58 2016 From: ed.sealing at sealingtech.org (Ed Sealing) Date: Sat, 17 Dec 2016 12:54:58 -0500 Subject: [Bro] CPU usage with no traffic on Bro 2.5 with AF_PACKET In-Reply-To: References: Message-ID: Thanks Jon. I'll take a look. I should clarify. I'm working on a multi-tenant solution with Bro, Docker, and SR-IOV. The plan is to support 10Gbps+, with VLANs as the dividers for tenants. The containerized Bro is working and I'm able to run multiple Bro instances for multiple tenants. However, when I start additional Bro containers, they each consume 6% CPU (12% for 2, 18% for 3, etc). Would the usleep patch still apply to a high-throughput solution? On Sat, Dec 17, 2016 at 12:16 PM, Zeolla at GMail.com wrote: > I have a fork of 2.5 that may help. It's intended to minimize CPU load on > sensors that see low volume/sensor-local traffic. Check out the most > recent commits, shout out to Justin for the basis of the tweaks. > https://github.com/JonZeolla/bro/tree/topic/jonzeolla/low-volume > > Jon > > On Sat, Dec 17, 2016, 12:05 Ed Sealing wrote: > >> I'm seeing ~6% CPU utilization on workers, with no traffic. Is that >> expected? Is there any way to minimize the CPU load? >> >> Using AF_PACKET plugin. The cores are isolated using "isolcpus", so >> nothing else should be running on them. Workers are pinned to the CPUs in >> the >> >> [worker-1] >> type=worker >> host=localhost >> interface=af_packet::eth1 >> lb_method=custom >> lb_procs=14 >> pin_cpus=1,2,3,4,5,6,7,9,10,11,12,13,14,15 >> >> >> ~Ed >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- > > Jon > > Sent from my mobile device > -- R/S *Ed Sealing President / CEO* *CISSP, CEH, RHCSA* 7226 Lee Deforest Dr. Columbia, MD 21046 Mobile: (301) 885-6947 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161217/ae4def8d/attachment.html From jan.grashoefer at gmail.com Sat Dec 17 14:08:26 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Sat, 17 Dec 2016 23:08:26 +0100 Subject: [Bro] Quick af_packet question In-Reply-To: References: Message-ID: Hi James, to be honest, I don't know that interfaceA:interfaceB notation at all. Doing a quick search it seems related to running snort inline. Actually, I don't think AF_Packet can be used to capture from two different interfaces using a single instance of Bro. But, running a cluster one could setup a worker per interface using AF_Packet. The latest version of the plugin contains an additional broctl-plugin to allow specifying the necessary parameters (workers will need different fanout_ids, see https://bro-tracker.atlassian.net/browse/BIT-1747). The README was extended as well to provide some information on how to setup Bro and AF_Packet using broctl. Hope that helps, Jan > Does not appear to decode pppoe however :( > > On 2016-12-16 09:51, James Lay wrote: >> So far my testing says yes: >> >> 09:30:56 @tester:/opt/bro/spool$] sudo bro -C -i af_packet::eth0:wlan0 >> listening on eth0:wlan0 >> >> eth0 Link encap:Ethernet HWaddr 00:1f:f3:46:62:ca >> inet addr:192.168.1.7 Bcast:192.168.1.255 >> Mask:255.255.255.0 >> inet6 addr: fe80::21f:f3ff:fe46:62ca/64 Scope:Link >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:434251 errors:0 dropped:59 overruns:0 frame:0 >> TX packets:261164 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:1000 >> RX bytes:600874115 (600.8 MB) TX bytes:70240696 (70.2 MB) >> Interrupt:16 >> >> wlan0 Link encap:Ethernet HWaddr 00:23:6c:7b:29:1d >> inet addr:192.168.1.60 Bcast:192.168.1.255 >> Mask:255.255.255.0 >> inet6 addr: fe80::223:6cff:fe7b:291d/64 Scope:Link >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:74 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:12 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:1000 >> RX bytes:10726 (10.7 KB) TX bytes:1820 (1.8 KB) >> >> ssh.log: >> 1481906017.175240 CWWs1B3RQhgUy1QqT2 192.168.1.2 45480 >> 192.168.1.7 22 2 T 1 - >> SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 SSH-2.0-OpenSSH_7.2p2 >> Ubuntu-4ubuntu2.1 chacha20-poly1305 at openssh.com >> umac-64-etm at openssh.com none curve25519-sha256 at libssh.org ssh-rsa >> >> 1481906687.051242 CfvBJT3Gs2r7YAX2n1 192.168.1.2 34956 >> 192.168.1.60 22 2 T 1 - >> SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 SSH-2.0-OpenSSH_7.2p2 >> Ubuntu-4ubuntu2.1 chacha20-poly1305 at openssh.com >> umac-64-etm at openssh.com none curve25519-sha256 at libssh.org ssh-rsa >> >> but wanting to verify. Thank you. >> >> James >> >> On 2016-12-16 09:35, James Lay wrote: >>> Love the plugin thanks...quick question for cli...does af_packet need >>> -i >>> for multiple interfaces, or can it be used like snort with >>> af_packet::eth0:eth1? Thank you. >>> >>> James >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From jlay at slave-tothe-box.net Sun Dec 18 05:04:40 2016 From: jlay at slave-tothe-box.net (James Lay) Date: Sun, 18 Dec 2016 06:04:40 -0700 Subject: [Bro] Quick af_packet question In-Reply-To: References: Message-ID: <1482066280.2598.0.camel@slave-tothe-box.net> Thanks Jan....those were my final results as well. James On Sat, 2016-12-17 at 23:08 +0100, Jan Grash?fer wrote: > Hi James, > > to be honest, I don't know that interfaceA:interfaceB notation at > all. > Doing a quick search it seems related to running snort inline. > Actually, > I don't think AF_Packet can be used to capture from two different > interfaces using a single instance of Bro. But, running a cluster one > could setup a worker per interface using AF_Packet. The latest > version > of the plugin contains an additional broctl-plugin to allow > specifying > the necessary parameters (workers will need different fanout_ids, see > https://bro-tracker.atlassian.net/browse/BIT-1747). The README was > extended as well to provide some information on how to setup Bro and > AF_Packet using broctl. > > Hope that helps, > Jan > > > > > Does not appear to decode pppoe however :( > > > > On 2016-12-16 09:51, James Lay wrote: > > > > > > So far my testing says yes: > > > > > > 09:30:56 @tester:/opt/bro/spool$] sudo bro -C -i > > > af_packet::eth0:wlan0 > > > listening on eth0:wlan0 > > > > > > eth0??????Link encap:Ethernet??HWaddr 00:1f:f3:46:62:ca > > > ???????????inet addr:192.168.1.7??Bcast:192.168.1.255?? > > > Mask:255.255.255.0 > > > ???????????inet6 addr: fe80::21f:f3ff:fe46:62ca/64 Scope:Link > > > ???????????UP BROADCAST RUNNING MULTICAST??MTU:1500??Metric:1 > > > ???????????RX packets:434251 errors:0 dropped:59 overruns:0 > > > frame:0 > > > ???????????TX packets:261164 errors:0 dropped:0 overruns:0 > > > carrier:0 > > > ???????????collisions:0 txqueuelen:1000 > > > ???????????RX bytes:600874115 (600.8 MB)??TX bytes:70240696 (70.2 > > > MB) > > > ???????????Interrupt:16 > > > > > > wlan0?????Link encap:Ethernet??HWaddr 00:23:6c:7b:29:1d > > > ???????????inet addr:192.168.1.60??Bcast:192.168.1.255 > > > Mask:255.255.255.0 > > > ???????????inet6 addr: fe80::223:6cff:fe7b:291d/64 Scope:Link > > > ???????????UP BROADCAST RUNNING MULTICAST??MTU:1500??Metric:1 > > > ???????????RX packets:74 errors:0 dropped:0 overruns:0 frame:0 > > > ???????????TX packets:12 errors:0 dropped:0 overruns:0 carrier:0 > > > ???????????collisions:0 txqueuelen:1000 > > > ???????????RX bytes:10726 (10.7 KB)??TX bytes:1820 (1.8 KB) > > > > > > ssh.log: > > > 1481906017.175240???????CWWs1B3RQhgUy1QqT2??????192.168.1.2???454 > > > 80 > > > 192.168.1.7?????22??????2???????T???????1???????- > > > SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8???????SSH-2.0- > > > OpenSSH_7.2p2 > > > Ubuntu-4ubuntu2.1 chacha20-poly1305 at openssh.com > > > umac-64-etm at openssh.com none????curve25519-sha256 at libssh.org????s > > > sh-rsa > > > > > > 1481906687.051242???????CfvBJT3Gs2r7YAX2n1??????192.168.1.2???349 > > > 56 > > > 192.168.1.60????22??????2???????T???????1???????- > > > SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8???????SSH-2.0- > > > OpenSSH_7.2p2 > > > Ubuntu-4ubuntu2.1 chacha20-poly1305 at openssh.com > > > umac-64-etm at openssh.com none????curve25519-sha256 at libssh.org????s > > > sh-rsa > > > > > > but wanting to verify.??Thank you. > > > > > > James > > > > > > On 2016-12-16 09:35, James Lay wrote: > > > > > > > > Love the plugin thanks...quick question for cli...does > > > > af_packet need > > > > -i > > > > for multiple interfaces, or can it be used like snort with > > > > af_packet::eth0:eth1???Thank you. > > > > > > > > James > > > > _______________________________________________ > > > > Bro mailing list > > > > bro at bro-ids.org > > > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > > > Bro mailing list > > > bro at bro-ids.org > > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161218/26d40e7f/attachment.html From hovsep.sanjay.levi at gmail.com Mon Dec 19 13:26:17 2016 From: hovsep.sanjay.levi at gmail.com (Hovsep Levi) Date: Mon, 19 Dec 2016 21:26:17 +0000 Subject: [Bro] Bro cluster requirements and manager logging backlog bug Message-ID: Hello all, We are still having a problem with our Bro cluster and logging. During peak times the manager will slowly consume all available memory while the logs sent to disk are delayed by an hour or more. Does anyone know the official bug ID for this within bro-tracker.atlassian.net ? I've tracked this problem for a while now and tried all variations of the proposed fixes: the flare patch, the no-flare patch, segmented cluster with one manager per box, and an architecture change from Linux+PF_RING to FreeBSD+Myricom. Currently we are using a standard build of bro-2.5-beta in a cluster configuration with one dedicated manager and three dedicated sensors, each using both ports of a Myricom card with 22 workers attached to each port. ( 1 manager, 1 logger, 12 proxies, 6 worker nodes (22 procs each, 132 total). Restarting the cluster on a regular basis is much easier without PF_RING but that's only partially curing the symptom. In that regard the last proposed solution is the most expensive, using faster CPUs which will reduce the worker count. But will that really solve the problem ? I'm more interested in defining what the problem actually is. FWIW there's some text below to illustrate, the dates are somewhat old but it's still a representative example. 21:05 UTC - Manager node is near out of memory.. 2800 Mb left - Workers have moderate CPU usage, 60% - Logs on manager node are 25 minutes behind.. - 21:05 vs 20:40 - Initiated cluster restart at 21:06, completed at 21:11. 21:26 UTC - Workers have moderate CPU usage. - Logs are 16 minutes behind Earlier the logs were roughly two hours behind. [bro at mgr /opt/bro]$ date -r 1471373408 (most recent conn.log timestamp) Tue Aug 16 18:50:08 UTC 2016 [bro at mgr /opt/bro]$ date Tue Aug 16 20:43:45 UTC 2016 Bro manager process is using 70G of memory and the system is swapping: last pid: 96557; load averages: 46.37, 53.09, 54.88 up 0+18:06:24 21:25:17 55 processes: 8 running, 47 sleeping CPU: 7.7% user, 2.1% nice, 68.1% system, 0.2% interrupt, 21.9% idle Mem: 103G Active, 2412M Inact, 19G Wired, 549M Cache, 331M Free ARC: 15G Total, 89M MFU, 15G MRU, 29M Anon, 68M Header, 211M Other Swap: 12G Total, 12G Used, 85M Free, 99% Inuse, 9248K In PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 7305 bro 34 20 0 40121M 39498M uwait 10 31.7H 280.27% bro 7337 bro 1 96 5 70653M 61577M CPU36 36 868:45 59.96% bro Currently in this state the logs over two hours behind the current time. bro at mgr:~ % date -r 1471374952 (most recent conn.log timestamp) Tue Aug 16 19:15:52 UTC 2016 bro at mgr:~ % date Tue Aug 16 21:27:04 UTC 2016 Memory usage over the past week: -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161219/ff9a0e58/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: memory-week.png Type: image/png Size: 28315 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161219/ff9a0e58/attachment-0001.bin From jazoff at illinois.edu Mon Dec 19 14:47:38 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 19 Dec 2016 22:47:38 +0000 Subject: [Bro] Bro cluster requirements and manager logging backlog bug In-Reply-To: References: Message-ID: <39591A79-2357-45E1-AFE5-7009E764326E@illinois.edu> > On Dec 19, 2016, at 4:26 PM, Hovsep Levi wrote: > > Hello all, > > > We are still having a problem with our Bro cluster and logging. During peak times the manager will slowly consume all available memory while the logs sent to disk are delayed by an hour or more. You're saying "the manager" but do you mean "the manager node" or "the manager process"? With the added logger process the manager process does not have anything to do with logs. The last time you mentioned these issues the logger node capability did not exist yet. A lot has changed since then but the logs you show are from 4 months ago. We need to see what this command outputs when your cluster is having log issues: broctl top manager logger Also, you've never mentioned the actual rate of logs you are seeing at these peak times Running this in your log directory would help: du -ms;cat *|wc -l;sleep 60;du -ms;cat *|wc -l -- - Justin Azoff From hovsep.sanjay.levi at gmail.com Tue Dec 20 10:56:36 2016 From: hovsep.sanjay.levi at gmail.com (Hovsep Levi) Date: Tue, 20 Dec 2016 18:56:36 +0000 Subject: [Bro] Bro cluster requirements and manager logging backlog bug In-Reply-To: <39591A79-2357-45E1-AFE5-7009E764326E@illinois.edu> References: <39591A79-2357-45E1-AFE5-7009E764326E@illinois.edu> Message-ID: On Mon, Dec 19, 2016 at 10:47 PM, Azoff, Justin S wrote: > > > On Dec 19, 2016, at 4:26 PM, Hovsep Levi > wrote: > > > > Hello all, > > > > > > We are still having a problem with our Bro cluster and logging. During > peak times the manager will slowly consume all available memory while the > logs sent to disk are delayed by an hour or more. > > > You're saying "the manager" but do you mean "the manager node" or "the > manager process"? > The manager node. > > With the added logger process the manager process does not have anything > to do with logs. The last time you mentioned these issues the logger node capability did not > exist yet. A lot has changed since then but the logs you show are from 4 > months ago. > The email was a saved draft accidentally sent before finishing the edit. I've been using the logger process since October. As of yesterday I'm using the latest Bro-2.5 with a default local.bro file. > We need to see what this command outputs when your cluster is having log > issues: > > broctl top manager logger > > Ok. Today I find this: [bro at mgr /opt/bro]$ bin/broctl top manager logger Name Type Host Pid Proc VSize Rss Cpu Cmd logger logger 169.232.234.36 52852 parent 109G 100G 0% bro logger logger 169.232.234.36 52867 child 837M 498M 0% bro manager manager 169.232.234.36 52935 child 485M 17M 0% bro manager manager 169.232.234.36 52892 parent 2G 557M 0% bro In this condition all the workers are at 100% CPU and the worker nodes have all 128GB RAM used. The manager node had to be rebooted as "killall -9 bro" had no effect. This is what happens if Bro isn't restarted every 30 minutes. > Also, you've never mentioned the actual rate of logs you are seeing at > these peak times > > Running this in your log directory would help: > > du -ms;cat *|wc -l;sleep 60;du -ms;cat *|wc -l > > [bro at mgr /opt/bro_data/logs/current]$ du -ms;cat *|wc -l;sleep 60;du -ms;cat *|wc -l 56 . 789695 220 . 2801719 Bro was started @ Tue Dec 20 18:39:26 UTC 2016, the command below was run a minute later. [bro at mgr /opt/bro]$ bin/broctl top manager logger Name Type Host Pid Proc VSize Rss Cpu Cmd logger logger 169.232.234.36 18832 parent 289M 139M 194% bro logger logger 169.232.234.36 18874 child 485M 78M 36% bro manager manager 169.232.234.36 18905 parent 530M 384M 72% bro manager manager 169.232.234.36 18947 child 510M 231M 51% bro @ Tue Dec 20 18:46:48 UTC 2016 already the logger has 5G memory: [bro at mgr /opt/bro]$ bin/broctl top manager logger Name Type Host Pid Proc VSize Rss Cpu Cmd logger logger 169.232.234.36 18832 parent 5G 5G 192% bro logger logger 169.232.234.36 18874 child 1G 1G 58% bro manager manager 169.232.234.36 18947 child 510M 255M 55% bro manager manager 169.232.234.36 18905 parent 11G 1G 25% bro [bro at mgr /opt/bro_data/logs/current]$ du -ms;cat *|wc -l;sleep 60;du -ms;cat *|wc -l 593 . 7117478 809 . 9573974 @ Tue Dec 20 18:51:05 UTC 2016 the logger has 10G memory and the manager has increased by 5G as well [bro at mgr /opt/bro]$ bin/broctl top manager logger Name Type Host Pid Proc VSize Rss Cpu Cmd logger logger 169.232.234.36 18832 parent 10G 10G 222% bro logger logger 169.232.234.36 18874 child 3G 3G 64% bro manager manager 169.232.234.36 18947 child 510M 255M 65% bro manager manager 169.232.234.36 18905 parent 16G 1G 23% bro [bro at mgr /opt/bro_data/logs/current]$ du -ms;cat *|wc -l;sleep 60;du -ms;cat *|wc -l 1346 . 15357570 1623 . 18708418 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161220/cbded41c/attachment.html From hovsep.sanjay.levi at gmail.com Tue Dec 20 12:18:58 2016 From: hovsep.sanjay.levi at gmail.com (Hovsep Levi) Date: Tue, 20 Dec 2016 20:18:58 +0000 Subject: [Bro] Bro cluster requirements and manager logging backlog bug In-Reply-To: References: <39591A79-2357-45E1-AFE5-7009E764326E@illinois.edu> Message-ID: Back from lunch and the cluster is essentially crashed. @ Tue Dec 20 19:03:48 UTC 2016 [bro at mgr /opt/bro]$ bin/broctl top manager logger Name Type Host Pid Proc VSize Rss Cpu Cmd logger logger 169.232.234.36 18832 parent 23G 22G 171% bro logger logger 169.232.234.36 18874 child 11G 10G 58% bro manager manager 169.232.234.36 18947 child 510M 255M 54% bro manager manager 169.232.234.36 18905 parent 23G 2G 19% bro [bro at mgr /opt/bro_data/logs/current]$ du -ms;cat *|wc -l;sleep 60;du -ms;cat *|wc -l 1096 . 17483890 1393 . 20633538 It's about ~55K EPS, but probably more since logs are buffering in memory. @ Tue Dec 20 20:12:47 UTC 2016 [bro at mgr /opt/bro]$ bin/broctl top manager logger Name Type Host Pid Proc VSize Rss Cpu Cmd logger logger 169.232.234.36 18832 parent 67G 20G 0% bro logger logger 169.232.234.36 18874 child 44G 24G 0% bro manager manager 169.232.234.36 18947 child 510M 249M 99% bro manager manager 169.232.234.36 18905 parent 18G 798M 0% bro last pid: 42312; load averages: 4.50, 5.40, 20.50 up 0+02:27:46 20:12:58 63 processes: 5 running, 53 sleeping, 1 zombie, 4 waiting CPU: 2.3% user, 0.2% nice, 13.7% system, 0.0% interrupt, 83.8% idle Mem: 104G Active, 3328M Inact, 17G Wired, 340M Cache, 180M Free ARC: 15G Total, 7820M MFU, 7921M MRU, 16K Anon, 46M Header, 44M Other Swap: 12G Total, 12G Used, K Free, 100% Inuse PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 18947 bro 1 108 5 510M 249M CPU40 40 55:54 100.00% bro 18832 bro 36 20 0 69454M 21145M uwait 44 129:19 0.00% bro 18874 bro 1 52 5 45265M 25543M pfault 1 42:47 0.00% bro 18905 bro 7 20 0 19346M 798M uwait 13 16:27 0.00% bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161220/ca182891/attachment.html From jazoff at illinois.edu Tue Dec 20 12:40:37 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 20 Dec 2016 20:40:37 +0000 Subject: [Bro] Bro cluster requirements and manager logging backlog bug In-Reply-To: References: <39591A79-2357-45E1-AFE5-7009E764326E@illinois.edu> Message-ID: <4A126664-FC0D-465B-896C-0BD93809CE33@illinois.edu> > On Dec 20, 2016, at 1:56 PM, Hovsep Levi wrote: > > > [bro at mgr /opt/bro]$ bin/broctl top manager logger > Name Type Host Pid Proc VSize Rss Cpu Cmd > logger logger 169.232.234.36 52852 parent 109G 100G 0% bro > logger logger 169.232.234.36 52867 child 837M 498M 0% bro > manager manager 169.232.234.36 52935 child 485M 17M 0% bro > manager manager 169.232.234.36 52892 parent 2G 557M 0% bro > > In this condition all the workers are at 100% CPU and the worker nodes have all 128GB RAM used. The manager node had to be rebooted as "killall -9 bro" had no effect. This is what happens if Bro isn't restarted every 30 minutes. This output with the cpu at 0 is kind of odd, unless it was already swapping or something. > > Also, you've never mentioned the actual rate of logs you are seeing at these peak times > > Running this in your log directory would help: > > du -ms;cat *|wc -l;sleep 60;du -ms;cat *|wc -l > > [bro at mgr /opt/bro_data/logs/current]$ du -ms;cat *|wc -l;sleep 60;du -ms;cat *|wc -l > 56 . > 789695 > 220 . > 2801719 So this shows only 33k logs/sec and 3MB/sec > > > @ Tue Dec 20 18:46:48 UTC 2016 already the logger has 5G memory: > > [bro at mgr /opt/bro]$ bin/broctl top manager logger > Name Type Host Pid Proc VSize Rss Cpu Cmd > logger logger 169.232.234.36 18832 parent 5G 5G 192% bro > logger logger 169.232.234.36 18874 child 1G 1G 58% bro > manager manager 169.232.234.36 18947 child 510M 255M 55% bro > manager manager 169.232.234.36 18905 parent 11G 1G 25% bro This shows that your logger process seems to just have issues keeping up with the volume... > > [bro at mgr /opt/bro_data/logs/current]$ du -ms;cat *|wc -l;sleep 60;du -ms;cat *|wc -l > 593 . > 7117478 > 809 . > 9573974 but based on this you are only doing 40k logs/sec and 4 MB/sec and shouldn't really be having issues. We have users doing over 200k/sec. Can you check the following: after bro has been running for a bit: wc -l *.log | sort -n to show which log files are the largest the output of this command: top -b -n 1 -H -o TIME |grep bro:|head -n 20 or just run top and press H. That should show all the bro logging threads (it works on linux at least) They may show up truncated but it's enough to tell them apart. What model/count CPU does your manager have? Are you writing out logs as the default ascii or using json? -- - Justin Azoff From hovsep.sanjay.levi at gmail.com Tue Dec 20 13:09:18 2016 From: hovsep.sanjay.levi at gmail.com (Hovsep Levi) Date: Tue, 20 Dec 2016 21:09:18 +0000 Subject: [Bro] Bro cluster requirements and manager logging backlog bug In-Reply-To: <4A126664-FC0D-465B-896C-0BD93809CE33@illinois.edu> References: <39591A79-2357-45E1-AFE5-7009E764326E@illinois.edu> <4A126664-FC0D-465B-896C-0BD93809CE33@illinois.edu> Message-ID: > > In this condition all the workers are at 100% CPU and the worker nodes > have all 128GB RAM used. The manager node had to be rebooted as "killall > -9 bro" had no effect. This is what happens if Bro isn't restarted every > 30 minutes. > > This output with the cpu at 0 is kind of odd, unless it was already > swapping or something. Yes, it was already swapping. > Can you check the following: > > after bro has been running for a bit: > > wc -l *.log | sort -n > > to show which log files are the largest > > [bro at mgr /opt/bro_data/logs/current]$ wc -l *.log | sort -n 1 stderr.log 4 stdout.log 8 packet_filter.log 9 stats.log 24 intel.log 37 pe.log 38 kerberos.log 50 irc.log 53 dce_rpc.log 78 radius.log 104 ftp.log 332 mysql.log 355 rfb.log 416 snmp.log 548 reporter.log 841 notice.log 1202 dpd.log 2090 tunnel.log 5515 known_certs.log 7808 rdp.log 15737 communication.log 16393 smtp.log 19713 known_services.log 26794 ssh.log 38487 sip.log 101981 known_hosts.log 106543 software.log 492146 x509.log 714576 ssl.log 795818 dns.log 886360 http.log 985519 weird.log 1936147 files.log 5121874 conn.log 11277601 total > the output of this command: > > top -b -n 1 -H -o TIME |grep bro:|head -n 20 > > or just run top and press H. That should show all the bro logging threads > (it works on linux at least) They may show up truncated but it's enough to > tell them apart. > > [bro at mgr /opt/bro_data/logs/current]$ top -n -H -o time | grep bro 5672 bro 100 10 21076K 2796K RUN 19 6:35 62.79% gzip 5858 bro 95 5 510M 257M RUN 8 3:26 61.38% bro 5785 bro 95 5 2373M 2058M CPU11 11 3:20 59.08% bro 5743 bro 87 0 7897M 7743M RUN 14 3:19 52.78% bro{bro} 5743 bro 88 0 7897M 7743M CPU0 0 3:18 55.18% bro{bro} 5816 bro 40 0 5298M 1158M nanslp 23 1:59 23.29% bro{bro} 5743 bro 37 0 7897M 7743M uwait 4 1:32 23.58% bro{bro} > What model/count CPU does your manager have? > > Four of these with 32 GB per NUMA node. Processor Information Socket Designation: CPU1 Type: Central Processor Family: Opteron 6200 Manufacturer: AMD ID: 12 0F 60 00 FF FB 8B 17 Signature: Family 21, Model 1, Stepping 2 Flags: (...) Version: AMD Opteron(TM) Processor 6238 Voltage: 1.2 V External Clock: 3200 MHz Max Speed: 3600 MHz Current Speed: 2600 MHz Status: Populated, Enabled Upgrade: Socket AM3 L1 Cache Handle: 0x0700 L2 Cache Handle: 0x0701 L3 Cache Handle: 0x0702 Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Core Count: 12 Core Enabled: 12 Thread Count: 12 > Are you writing out logs as the default ascii or using json? > > The default Ascii. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161220/9cb67c48/attachment.html From hovsep.sanjay.levi at gmail.com Tue Dec 20 13:11:10 2016 From: hovsep.sanjay.levi at gmail.com (Hovsep Levi) Date: Tue, 20 Dec 2016 21:11:10 +0000 Subject: [Bro] Bro cluster requirements and manager logging backlog bug In-Reply-To: References: <39591A79-2357-45E1-AFE5-7009E764326E@illinois.edu> <4A126664-FC0D-465B-896C-0BD93809CE33@illinois.edu> Message-ID: > > > [bro at mgr /opt/bro_data/logs/current]$ top -n -H -o time | grep bro > 5672 bro 100 10 21076K 2796K RUN 19 6:35 62.79% gzip > 5858 bro 95 5 510M 257M RUN 8 3:26 61.38% bro > 5785 bro 95 5 2373M 2058M CPU11 11 3:20 59.08% bro > 5743 bro 87 0 7897M 7743M RUN 14 3:19 52.78% bro{bro} > 5743 bro 88 0 7897M 7743M CPU0 0 3:18 55.18% bro{bro} > 5816 bro 40 0 5298M 1158M nanslp 23 1:59 23.29% bro{bro} > 5743 bro 37 0 7897M 7743M uwait 4 1:32 23.58% bro{bro} > > Here are the associated mappings for the PIDs: [bro at f01 /opt/bro]$ bin/broctl top manager logger Name Type Host Pid Proc VSize Rss Cpu Cmd logger logger 169.231.234.36 5743 parent 9G 9G 194% bro logger logger 169.231.234.36 5785 child 3G 2G 60% bro manager manager 169.231.234.36 5858 child 510M 257M 63% bro manager manager 169.231.234.36 5816 parent 5G 1G 18% bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161220/a99dbc7e/attachment-0001.html From jazoff at illinois.edu Tue Dec 20 13:46:25 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 20 Dec 2016 21:46:25 +0000 Subject: [Bro] Bro cluster requirements and manager logging backlog bug In-Reply-To: References: <39591A79-2357-45E1-AFE5-7009E764326E@illinois.edu> <4A126664-FC0D-465B-896C-0BD93809CE33@illinois.edu> Message-ID: > On Dec 20, 2016, at 4:09 PM, Hovsep Levi wrote: > > > > Can you check the following: > > after bro has been running for a bit: > > wc -l *.log | sort -n > > to show which log files are the largest > > [bro at mgr /opt/bro_data/logs/current]$ wc -l *.log | sort -n > 101981 known_hosts.log > 106543 software.log > 492146 x509.log > 714576 ssl.log > 795818 dns.log > 886360 http.log > 985519 weird.log > 1936147 files.log > 5121874 conn.log > 11277601 total Your weird.log is 1/5 the size of your conn.log and larger than your http log.. You should look into which weird name is showing up so much, you may have a serious problem with your tap configuration. That is not normal at all. > > the output of this command: > > top -b -n 1 -H -o TIME |grep bro:|head -n 20 > > or just run top and press H. That should show all the bro logging threads (it works on linux at least) They may show up truncated but it's enough to tell them apart. > > > > [bro at mgr /opt/bro_data/logs/current]$ top -n -H -o time | grep bro > 5672 bro 100 10 21076K 2796K RUN 19 6:35 62.79% gzip > 5858 bro 95 5 510M 257M RUN 8 3:26 61.38% bro > 5785 bro 95 5 2373M 2058M CPU11 11 3:20 59.08% bro > 5743 bro 87 0 7897M 7743M RUN 14 3:19 52.78% bro{bro} > 5743 bro 88 0 7897M 7743M CPU0 0 3:18 55.18% bro{bro} > 5816 bro 40 0 5298M 1158M nanslp 23 1:59 23.29% bro{bro} > 5743 bro 37 0 7897M 7743M uwait 4 1:32 23.58% bro{bro} Ah, I guess that doesn't work on freebsd, It would have output thread names like bro: conn/Log bro: dns/Log It looks like the bulk of your logging load is coming from files+conn and weird though, so what you can do is cut down the volume of those logs to get your cpu to be happy. > > What model/count CPU does your manager have? > > > Four of these with 32 GB per NUMA node. > > Processor Information > Socket Designation: CPU1 > Type: Central Processor > Family: Opteron 6200 Ah!!! This is part of your problem. Every site we have worked with in the past year or so that was having serious manager performance issues was using the crazy high core count AMD systems. While they perform well when you have a heavily threaded task (And I bet they do, we have an entire supercomputer filled with 40,000 of them) the bro logger only has few heavyweight threads and just does not work well on these processors. That said, you can probably get this working acceptably though. There are two options for this: * filter some noisy log lines, which will cause them to not be logged at all. * split heavy streams into multiple log files, which will let the logger process dedicate a logging thread to each part. I would start by trying some of these config fragments that split log files apart: # Split files log into files and files_certs log event bro_init() { Log::remove_default_filter(Files::LOG); Log::add_filter(Files::LOG, [ $name = "files-split", $path_func(id: Log::ID, path: string, rec: Files::Info) = { if (rec?$mime_type && rec$mime_type == "application/pkix-cert") return "files_certs"; return "files"; } ]); } (you can probably just ignore those lines completely since the x509 log is more useful) #Split conn into conn and conn_dns event bro_init() { Log::remove_default_filter(Conn::LOG); Log::add_filter(Conn::LOG, [ $name = "conn-split", $path_func(id: Log::ID, path: string, rec: Conn::Info) = { if (rec?$service && "dns" in rec$service) return "conn_dns"; return "conn"; } ]); } #Split http.log into directions event bro_init() { Log::remove_default_filter(HTTP::LOG); Log::add_filter(HTTP::LOG, [ $name = "http-directions", $path_func(id: Log::ID, path: string, rec: HTTP::Info) = { local l = Site::is_local_addr(rec$id$orig_h); local r = Site::is_local_addr(rec$id$resp_h); if(l && r) return "http_internal"; if (l) return "http_outbound"; else return "http_inbound"; } ]); } You could also do the directions thing for the conn.log as well. If your network is anything like ours, your conn.log is 90% scan attempts to tcp port 23 from IoT devices, splitting that out to a separate log file of filtering it entirely would probably help more than anything. You can also take a look at the filter* scripts that are at https://github.com/michalpurzynski/bro-gramming -- - Justin Azoff From hovsep.sanjay.levi at gmail.com Tue Dec 20 14:06:53 2016 From: hovsep.sanjay.levi at gmail.com (Hovsep Levi) Date: Tue, 20 Dec 2016 22:06:53 +0000 Subject: [Bro] Bro cluster requirements and manager logging backlog bug In-Reply-To: References: <39591A79-2357-45E1-AFE5-7009E764326E@illinois.edu> <4A126664-FC0D-465B-896C-0BD93809CE33@illinois.edu> Message-ID: > Your weird.log is 1/5 the size of your conn.log and larger than your http > log.. You should look into which weird name is showing up so much, you may > have a serious problem with your tap configuration. That is not normal at > all. I'm going to guess that the messages relate to packet loss on the workers which is currently 3%-10% per worker process. Adding another worker node should fix it. AFAIK the tap is working fine. In my experience the only weird message that corresponded to tap issues was "possible_split_routing". [bro at mgr /opt/bro_data/logs/current]$ awk '{print $7}' weird.log | sort | uniq -c | sort -rn | head -25 243077 dns_unmatched_msg 205444 SYN_seq_jump 183857 bad_HTTP_request 54153 window_recision 42960 bad_UDP_checksum 20681 dns_unmatched_reply 14280 data_before_established 13048 possible_split_routing 9717 DNS_RR_unknown_type 4310 line_terminated_with_single_CR 4121 active_connection_reuse 1676 TCP_ack_underflow_or_misorder 1490 connection_originator_SYN_ack 1226 SYN_with_data 925 TCP_seq_underflow_or_misorder 862 above_hole_data_without_any_acks 727 Teredo_bubble_with_payload 706 HTTP_version_mismatch 685 data_after_reset 677 unknown_HTTP_method 626 unexpected_multiple_HTTP_requests 584 inappropriate_FIN 566 empty_http_request 511 bad_TCP_checksum 373 inflate_failed > > It looks like the bulk of your logging load is coming from files+conn and > weird though, so what you can do is cut down the volume of those logs to > get your cpu to be happy. > > > > > What model/count CPU does your manager have? > > > > > > Four of these with 32 GB per NUMA node. > > > > Processor Information > > Socket Designation: CPU1 > > Type: Central Processor > > Family: Opteron 6200 > > Ah!!! This is part of your problem. Every site we have worked with in the > past year or so that was having serious manager performance issues was > using the crazy high core count AMD systems. While they perform well when > you have a heavily threaded task (And I bet they do, we have an entire > supercomputer filled with 40,000 of them) the bro logger only has few > heavyweight threads and just does not work well on these processors. > > That said, you can probably get this working acceptably though. There are > two options for this: > > * filter some noisy log lines, which will cause them to not be logged at > all. > * split heavy streams into multiple log files, which will let the logger > process dedicate a logging thread to each part. > > I would start by trying some of these config fragments that split log > files apart: > > Thanks for the help, I'm going to give your suggestions a try. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161220/ba1f2beb/attachment.html From zeolla at gmail.com Wed Dec 21 09:18:17 2016 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Wed, 21 Dec 2016 17:18:17 +0000 Subject: [Bro] broctl unable to find peers Message-ID: I'm seeing an issue using bro 2.4.1 where when I run `./broctl status` it hangs on 'Getting peer status ...'. When I run the same command specifying manager, any of the proxies, or any of the individual workers it has no issue. Has anybody seen this before? This is a 5 node cluster (1 manager, 4 sensors) running on Ubuntu 14.04. I am in the process of upgrading to 2.5, but before I do so I'm adding 2 additional sensor machines (bringing it to 7 nodes) to the cluster because we sorely need the additional processing power. After the upgrade to 2.5 I will be adding another node and splitting the logger function onto it, making it an 8 node cluster. Here's an example of me running `./broctl status` and it failing after 3 1/2 minutes, then it goes on to successfully get the status for every component/instance specifically, however the Peers section returns "???". $ time ./broctl status || time ./broctl status manager;time for proxy in {1..5}; do ./broctl status proxy-${proxy}; done;for svr in {1..4}; do for instance in {1..20}; do ./broctl status worker-${svr}-${instance}; done; done removing stale lock Getting process status ... Getting peer status ... Killed real 3m35.233s user 0m0.126s sys 0m0.119s waiting for lock (owned by PID 22222) ... Getting process status ... Getting peer status ... Name Type Host Status Pid Peers Started manager manager A.B.C.D running 11111 ??? 18 Dec 03:24:38 Jon -- Jon Sent from my mobile device -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161221/9f91abf6/attachment.html From jazoff at illinois.edu Wed Dec 21 09:39:07 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 21 Dec 2016 17:39:07 +0000 Subject: [Bro] broctl unable to find peers In-Reply-To: References: Message-ID: > On Dec 21, 2016, at 12:18 PM, Zeolla at GMail.com wrote: > > I'm seeing an issue using bro 2.4.1 where when I run `./broctl status` it hangs on 'Getting peer status ...'. When I run the same command specifying manager, any of the proxies, or any of the individual workers it has no issue. Has anybody seen this before? You likely have iptables enabled on your hosts and it is preventing broctl from connecting to bro on the workers. https://www.bro.org/sphinx/components/broctl/README.html#bro-communication -- - Justin Azoff From zeolla at gmail.com Wed Dec 21 09:49:18 2016 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Wed, 21 Dec 2016 17:49:18 +0000 Subject: [Bro] broctl unable to find peers In-Reply-To: References: Message-ID: I've tested with iptables stopped and have the same issue. We do typically run with iptables up but have openings for all the required communication as far as I'm aware. This additional context may be helpful: $ ./broctl status Getting process status ... Getting peer status ... Killed $ Traceback (most recent call last): File "", line 1, in File "", line 23, in File "/usr/lib/python2.7/json/__init__.py", line 338, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded Jon On Wed, Dec 21, 2016 at 12:39 PM Azoff, Justin S wrote: > > > On Dec 21, 2016, at 12:18 PM, Zeolla at GMail.com wrote: > > > > I'm seeing an issue using bro 2.4.1 where when I run `./broctl status` > it hangs on 'Getting peer status ...'. When I run the same command > specifying manager, any of the proxies, or any of the individual workers it > has no issue. Has anybody seen this before? > > > You likely have iptables enabled on your hosts and it is preventing broctl > from connecting to bro on the workers. > > > > https://www.bro.org/sphinx/components/broctl/README.html#bro-communication > -- > - Justin Azoff > > -- Jon Sent from my mobile device -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161221/0db29279/attachment.html From jazoff at illinois.edu Wed Dec 21 09:56:56 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 21 Dec 2016 17:56:56 +0000 Subject: [Bro] broctl unable to find peers In-Reply-To: References: Message-ID: > On Dec 21, 2016, at 12:49 PM, Zeolla at GMail.com wrote: > > I've tested with iptables stopped and have the same issue. We do typically run with iptables up but have openings for all the required communication as far as I'm aware. Are you sure? That's always what this is. If you run tcpdump at the same time you should see the manager try (and probably fail) to connect to the other nodes. It's probably working when you do one at a time because only one has to timeout instead of all of them. -- - Justin Azoff From zeolla at gmail.com Wed Dec 21 10:29:31 2016 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Wed, 21 Dec 2016 18:29:31 +0000 Subject: [Bro] broctl unable to find peers In-Reply-To: References: Message-ID: I could be wrong, but I don't think that's the issue. `tcpdump -nn -i ${interface} "dst net ${Worker_Subnet}/24 and src host ${Manager}"` shows plenty of valid traffic between the manager the cluster members, and everything else in the cluster appears to be functioning normally. I modified the iptables to allow all tcp ports between members of the cluster, restarted iptables, verified the new rules were effective across all systems, and tested `./broctl status` again, but it failed the same way as before. Jon On Wed, Dec 21, 2016 at 12:56 PM Azoff, Justin S wrote: > > On Dec 21, 2016, at 12:49 PM, Zeolla at GMail.com wrote: > > > > I've tested with iptables stopped and have the same issue. We do > typically run with iptables up but have openings for all the required > communication as far as I'm aware. > > > Are you sure? That's always what this is. > > If you run tcpdump at the same time you should see the manager try (and > probably fail) to connect to the other nodes. > > It's probably working when you do one at a time because only one has to > timeout instead of all of them. > > > -- > - Justin Azoff > > > -- Jon Sent from my mobile device -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161221/f90409ba/attachment.html From dnthayer at illinois.edu Wed Dec 21 10:54:43 2016 From: dnthayer at illinois.edu (Daniel Thayer) Date: Wed, 21 Dec 2016 12:54:43 -0600 Subject: [Bro] broctl unable to find peers In-Reply-To: References: Message-ID: <99915535-8401-a52c-60c0-04c1ba7d4e8e@illinois.edu> What happens if you run "broctl peerstatus"? (after starting the cluster, of course) On 12/21/16 11:18 AM, Zeolla at GMail.com wrote: > I'm seeing an issue using bro 2.4.1 where when I run `./broctl status` > it hangs on 'Getting peer status ...'. When I run the same command > specifying manager, any of the proxies, or any of the individual workers > it has no issue. Has anybody seen this before? > > This is a 5 node cluster (1 manager, 4 sensors) running on Ubuntu > 14.04. I am in the process of upgrading to 2.5, but before I do so I'm > adding 2 additional sensor machines (bringing it to 7 nodes) to the > cluster because we sorely need the additional processing power. After > the upgrade to 2.5 I will be adding another node and splitting the > logger function onto it, making it an 8 node cluster. > > Here's an example of me running `./broctl status` and it failing after 3 > 1/2 minutes, then it goes on to successfully get the status for every > component/instance specifically, however the Peers section returns "???". > > $ time ./broctl status || time ./broctl status manager;time for proxy in > {1..5}; do ./broctl status proxy-${proxy}; done;for svr in {1..4}; do > for instance in {1..20}; do ./broctl status worker-${svr}-${instance}; > done; done > > removing stale lock > > Getting process status ... > > Getting peer status ... > > Killed > > > real3m35.233s > > user0m0.126s > > sys0m0.119s > > waiting for lock (owned by PID 22222) ... > > Getting process status ... > > Getting peer status ... > > Name Type Host Status Pid Peers Started > > manager manager A.B.C.D running 11111 ??? 18 Dec 03:24:38 > > > > > Jon > -- > > Jon > > Sent from my mobile device > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From zeolla at gmail.com Wed Dec 21 11:43:57 2016 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Wed, 21 Dec 2016 19:43:57 +0000 Subject: [Bro] broctl unable to find peers In-Reply-To: <99915535-8401-a52c-60c0-04c1ba7d4e8e@illinois.edu> References: <99915535-8401-a52c-60c0-04c1ba7d4e8e@illinois.edu> Message-ID: I get a similar failure with broctl peerstatus when the cluster is up. It sits for a few minutes then kills itself. $ time ./broctl peerstatus Killed real 6m48.594s user 0m0.102s sys 0m0.111s I have tried adding a log line to my iptables so it will log right before getting dropped, but after reviewing the log over a 10 minute period I wasn't able to find anything from any members of my bro cluster getting dropped. While the logging was on I tried multiple ./broctl commands, including directly hitting the server using ./broctl status worker-1-1 and a more general ./broctl status or ./broctl peerstatus. Jon On Wed, Dec 21, 2016 at 1:54 PM Daniel Thayer wrote: > What happens if you run "broctl peerstatus"? (after starting > the cluster, of course) > > > On 12/21/16 11:18 AM, Zeolla at GMail.com wrote: > > I'm seeing an issue using bro 2.4.1 where when I run `./broctl status` > > it hangs on 'Getting peer status ...'. When I run the same command > > specifying manager, any of the proxies, or any of the individual workers > > it has no issue. Has anybody seen this before? > > > > This is a 5 node cluster (1 manager, 4 sensors) running on Ubuntu > > 14.04. I am in the process of upgrading to 2.5, but before I do so I'm > > adding 2 additional sensor machines (bringing it to 7 nodes) to the > > cluster because we sorely need the additional processing power. After > > the upgrade to 2.5 I will be adding another node and splitting the > > logger function onto it, making it an 8 node cluster. > > > > Here's an example of me running `./broctl status` and it failing after 3 > > 1/2 minutes, then it goes on to successfully get the status for every > > component/instance specifically, however the Peers section returns "???". > > > > $ time ./broctl status || time ./broctl status manager;time for proxy in > > {1..5}; do ./broctl status proxy-${proxy}; done;for svr in {1..4}; do > > for instance in {1..20}; do ./broctl status worker-${svr}-${instance}; > > done; done > > > > removing stale lock > > > > Getting process status ... > > > > Getting peer status ... > > > > Killed > > > > > > real3m35.233s > > > > user0m0.126s > > > > sys0m0.119s > > > > waiting for lock (owned by PID 22222) ... > > > > Getting process status ... > > > > Getting peer status ... > > > > Name Type Host Status Pid Peers Started > > > > manager manager A.B.C.D running 11111 ??? 18 Dec 03:24:38 > > > > > > > > > > Jon > > -- > > > > Jon > > > > Sent from my mobile device > > > > > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > -- Jon Sent from my mobile device -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161221/7c3e1b36/attachment-0001.html From dnthayer at illinois.edu Wed Dec 21 12:19:35 2016 From: dnthayer at illinois.edu (Daniel Thayer) Date: Wed, 21 Dec 2016 14:19:35 -0600 Subject: [Bro] broctl unable to find peers In-Reply-To: References: <99915535-8401-a52c-60c0-04c1ba7d4e8e@illinois.edu> Message-ID: <90a28281-6849-848a-b7a0-fc80202c7704@illinois.edu> One simple workaround for the status command being too slow is to edit your etc/broctl.cfg file and look for the option "StatusCmdShowAll". Change it to this: StatusCmdShowAll = 0 However, this doesn't solve the problem of Bro processes not being able to communicate with each other. On 12/21/16 1:43 PM, Zeolla at GMail.com wrote: > I get a similar failure with broctl peerstatus when the cluster is up. > It sits for a few minutes then kills itself. > > $ time ./broctl peerstatus > > Killed > > > real6m48.594s > > user0m0.102s > > sys0m0.111s > > > I have tried adding a log line to my iptables so it will log right > before getting dropped, but after reviewing the log over a 10 minute > period I wasn't able to find anything from any members of my bro cluster > getting dropped. While the logging was on I tried multiple ./broctl > commands, including directly hitting the server using ./broctl status > worker-1-1 and a more general ./broctl status or ./broctl peerstatus. > > Jon > > On Wed, Dec 21, 2016 at 1:54 PM Daniel Thayer > wrote: > > What happens if you run "broctl peerstatus"? (after starting > the cluster, of course) > > > On 12/21/16 11:18 AM, Zeolla at GMail.com wrote: > > I'm seeing an issue using bro 2.4.1 where when I run `./broctl status` > > it hangs on 'Getting peer status ...'. When I run the same command > > specifying manager, any of the proxies, or any of the individual > workers > > it has no issue. Has anybody seen this before? > > > > This is a 5 node cluster (1 manager, 4 sensors) running on Ubuntu > > 14.04. I am in the process of upgrading to 2.5, but before I do > so I'm > > adding 2 additional sensor machines (bringing it to 7 nodes) to the > > cluster because we sorely need the additional processing power. After > > the upgrade to 2.5 I will be adding another node and splitting the > > logger function onto it, making it an 8 node cluster. > > > > Here's an example of me running `./broctl status` and it failing > after 3 > > 1/2 minutes, then it goes on to successfully get the status for every > > component/instance specifically, however the Peers section returns > "???". > > > > $ time ./broctl status || time ./broctl status manager;time for > proxy in > > {1..5}; do ./broctl status proxy-${proxy}; done;for svr in {1..4}; do > > for instance in {1..20}; do ./broctl status worker-${svr}-${instance}; > > done; done > > > > removing stale lock > > > > Getting process status ... > > > > Getting peer status ... > > > > Killed > > > > > > real3m35.233s > > > > user0m0.126s > > > > sys0m0.119s > > > > waiting for lock (owned by PID 22222) ... > > > > Getting process status ... > > > > Getting peer status ... > > > > Name Type Host Status Pid Peers Started > > > > manager manager A.B.C.D running 11111 ??? 18 Dec 03:24:38 > > > > > > > > > > Jon > > -- > > > > Jon > > > > Sent from my mobile device > > > > > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > > -- > > Jon > > Sent from my mobile device > From zeolla at gmail.com Wed Dec 21 12:53:05 2016 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Wed, 21 Dec 2016 20:53:05 +0000 Subject: [Bro] broctl unable to find peers In-Reply-To: <90a28281-6849-848a-b7a0-fc80202c7704@illinois.edu> References: <99915535-8401-a52c-60c0-04c1ba7d4e8e@illinois.edu> <90a28281-6849-848a-b7a0-fc80202c7704@illinois.edu> Message-ID: Awesome, thank you. So, I worked with Justin on IRC and we did find this: $ ./broctl print foo worker-1-1 worker-1-1 However, when I ran tcpdump on WORKER I saw a clean connection setup, data transfer, and teardown from the manager. I also turned logging on for the manager's iptables, ran `./broctl status` assuming it would hit the manager first, and I didn't see any DROPs or REJECTs that would be relevant (looking at eth0, 127.0.0.1, and 127.0.1.1). Per Justin's suggestion I'm going to look into enabling debugging in broccoli tomorrow. Jon On Wed, Dec 21, 2016 at 3:19 PM Daniel Thayer wrote: > One simple workaround for the status command being too slow is to > edit your etc/broctl.cfg file and look for the option > "StatusCmdShowAll". Change it to this: > > StatusCmdShowAll = 0 > > However, this doesn't solve the problem of Bro processes > not being able to communicate with each other. > > > On 12/21/16 1:43 PM, Zeolla at GMail.com wrote: > > I get a similar failure with broctl peerstatus when the cluster is up. > > It sits for a few minutes then kills itself. > > > > $ time ./broctl peerstatus > > > > Killed > > > > > > real6m48.594s > > > > user0m0.102s > > > > sys0m0.111s > > > > > > I have tried adding a log line to my iptables so it will log right > > before getting dropped, but after reviewing the log over a 10 minute > > period I wasn't able to find anything from any members of my bro cluster > > getting dropped. While the logging was on I tried multiple ./broctl > > commands, including directly hitting the server using ./broctl status > > worker-1-1 and a more general ./broctl status or ./broctl peerstatus. > > > > Jon > > > > On Wed, Dec 21, 2016 at 1:54 PM Daniel Thayer > > wrote: > > > > What happens if you run "broctl peerstatus"? (after starting > > the cluster, of course) > > > > > > On 12/21/16 11:18 AM, Zeolla at GMail.com wrote: > > > I'm seeing an issue using bro 2.4.1 where when I run `./broctl > status` > > > it hangs on 'Getting peer status ...'. When I run the same command > > > specifying manager, any of the proxies, or any of the individual > > workers > > > it has no issue. Has anybody seen this before? > > > > > > This is a 5 node cluster (1 manager, 4 sensors) running on Ubuntu > > > 14.04. I am in the process of upgrading to 2.5, but before I do > > so I'm > > > adding 2 additional sensor machines (bringing it to 7 nodes) to the > > > cluster because we sorely need the additional processing power. > After > > > the upgrade to 2.5 I will be adding another node and splitting the > > > logger function onto it, making it an 8 node cluster. > > > > > > Here's an example of me running `./broctl status` and it failing > > after 3 > > > 1/2 minutes, then it goes on to successfully get the status for > every > > > component/instance specifically, however the Peers section returns > > "???". > > > > > > $ time ./broctl status || time ./broctl status manager;time for > > proxy in > > > {1..5}; do ./broctl status proxy-${proxy}; done;for svr in {1..4}; > do > > > for instance in {1..20}; do ./broctl status > worker-${svr}-${instance}; > > > done; done > > > > > > removing stale lock > > > > > > Getting process status ... > > > > > > Getting peer status ... > > > > > > Killed > > > > > > > > > real3m35.233s > > > > > > user0m0.126s > > > > > > sys0m0.119s > > > > > > waiting for lock (owned by PID 22222) ... > > > > > > Getting process status ... > > > > > > Getting peer status ... > > > > > > Name Type Host Status Pid Peers > Started > > > > > > manager manager A.B.C.D running 11111 ??? 18 Dec > 03:24:38 > > > > > > > > > > > > > > > Jon > > > -- > > > > > > Jon > > > > > > Sent from my mobile device > > > > > > > > > > > > _______________________________________________ > > > Bro mailing list > > > bro at bro-ids.org > > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > < > https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.ICSI.Berkeley.EDU_mailman_listinfo_bro&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Bi5qPBnY0NmYPqnRTPj_AfXQKpfQTZUpCzpfFBcawv0&m=jpt8TXKljrs0LwDVNY1QHlYBJ0kWtZsyM3QUo0ee46M&s=DLU_e8vfR1vSmBwUN8TMkF012iVQWkEVPZXC6elvBLE&e= > > > > > > > > > -- > > > > Jon > > > > Sent from my mobile device > > > -- Jon Sent from my mobile device -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161221/5eb8aa8b/attachment.html From jlay at slave-tothe-box.net Thu Dec 22 05:49:47 2016 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 22 Dec 2016 06:49:47 -0700 Subject: [Bro] Bro 2.5 and log rotation Message-ID: <1482414587.2534.7.camel@slave-tothe-box.net> I guess I'm in this boat as well. ?Since my upgrade, bro will stop rotating logs at some point. ?I'm not running bro via broctl. ?Here's my process for log rotation: local.bro: ? ? ? ? redef Log::default_rotation_interval = 86400 secs; ????????redef Log::default_rotation_postprocessor_cmd = "archive-log"; broctl.cfg: ? ? ? ? LogRotationInterval = 86400 sudo /usr/local/bro/bin/broctl install sudo ln -s /usr/local/bro/share/broctl/scripts/archive-log /usr/local/bin/ sudo ln -s /usr/local/bro/share/broctl/scripts/broctl-config.sh /usr/local/bin/ sudo ln -s /usr/local/bro/share/broctl/scripts/make-archive-name /usr/local/bin/ sudo ln -s /usr/local/bro/share/broctl/scripts/expire-logs /usr/local/bin/ sudo ln -s /usr/local/bro/share/broctl/scripts/delete-log /usr/local/bin/ sudo ln -s /usr/local/bro/share/broctl/scripts/cflow-stats /usr/local/bin/ sudo ln -s /usr/local/bro/share/broctl/scripts/stats-to-csv /usr/local/bin/ This will work for a while. ?But at some point it stops: at the core I believe it's because bro, after sometime, won't respond to a "normal" kill command. ?A "sudo killall bro" will do nothing. ?Usually I'll "sudo killall bro", wait a minute, and then my spool directory will be empty, I'll have an email with stats, and I'll have my new archive directory. ?I'll have to -9 it in order to get it to stop, ?I've restarted this morning and will see how many days it will go. ?Thank you. James -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161222/ce4362ce/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2016-12-22 05-58-45.png Type: image/png Size: 59878 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161222/ce4362ce/attachment-0001.bin From hovsep.sanjay.levi at gmail.com Thu Dec 22 09:29:33 2016 From: hovsep.sanjay.levi at gmail.com (Hovsep Levi) Date: Thu, 22 Dec 2016 17:29:33 +0000 Subject: [Bro] Bro cluster requirements and manager logging backlog bug In-Reply-To: References: <39591A79-2357-45E1-AFE5-7009E764326E@illinois.edu> <4A126664-FC0D-465B-896C-0BD93809CE33@illinois.edu> Message-ID: > > Thanks for the help, I'm going to give your suggestions a try. > > Unfortunately I wasn't able to stabilize the cluster. I tried splitting the conn log into six different types, inbound(dns,http,other} and outbound{dns,http,other} in addition to the http inbound/outbound split but the logger process continues to buffer about 1G of memory per minute. Short of a re-write of the logging process the only option is to upgrade CPUs ? I tried running more than one logger but that doesn't seem to work. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161222/82abf272/attachment.html From hovsep.sanjay.levi at gmail.com Thu Dec 22 09:32:32 2016 From: hovsep.sanjay.levi at gmail.com (Hovsep Levi) Date: Thu, 22 Dec 2016 17:32:32 +0000 Subject: [Bro] Bro cluster requirements and manager logging backlog bug In-Reply-To: References: <39591A79-2357-45E1-AFE5-7009E764326E@illinois.edu> <4A126664-FC0D-465B-896C-0BD93809CE33@illinois.edu> Message-ID: Maybe streaming logs via Kafka and disabling writing to disk has a chance. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161222/d4a6e939/attachment.html From jazoff at illinois.edu Thu Dec 22 09:55:48 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 22 Dec 2016 17:55:48 +0000 Subject: [Bro] Bro cluster requirements and manager logging backlog bug In-Reply-To: References: <39591A79-2357-45E1-AFE5-7009E764326E@illinois.edu> <4A126664-FC0D-465B-896C-0BD93809CE33@illinois.edu> Message-ID: <6EDF7C86-443B-4441-8A74-00C09A653845@illinois.edu> > On Dec 22, 2016, at 12:29 PM, Hovsep Levi wrote: > > Thanks for the help, I'm going to give your suggestions a try. > > > Unfortunately I wasn't able to stabilize the cluster. I tried splitting the conn log into six different types, inbound(dns,http,other} and outbound{dns,http,other} in addition to the http inbound/outbound split but the logger process continues to buffer about 1G of memory per minute. > > Short of a re-write of the logging process the only option is to upgrade CPUs ? I tried running more than one logger but that doesn't seem to work. There may be some inefficiencies in the thread queuing code the logger uses, but the only people that seem to have these major issues have the slow AMD cpus. Multiple loggers is something we hope to add once broker is integrated. There's a few places I hope to be able to do some sort of consistent ring hashing to scale out different tasks. Many tasks in bro are easily partitioned, like logging and sumstats. > Maybe streaming logs via Kafka and disabling writing to disk has a chance. Ah! if that is your end goal, you could try looking into having your workers write directly to kafka and bypass the manager entirely. -- - Justin Azoff From hovsep.sanjay.levi at gmail.com Thu Dec 22 10:07:59 2016 From: hovsep.sanjay.levi at gmail.com (Hovsep Levi) Date: Thu, 22 Dec 2016 18:07:59 +0000 Subject: [Bro] Bro cluster requirements and manager logging backlog bug In-Reply-To: <6EDF7C86-443B-4441-8A74-00C09A653845@illinois.edu> References: <39591A79-2357-45E1-AFE5-7009E764326E@illinois.edu> <4A126664-FC0D-465B-896C-0BD93809CE33@illinois.edu> <6EDF7C86-443B-4441-8A74-00C09A653845@illinois.edu> Message-ID: > > There may be some inefficiencies in the thread queuing code the logger > uses, but the only people that seem to have these major issues have the > slow AMD cpus. > > Multiple loggers is something we hope to add once broker is integrated. > There's a few places I hope to be able to do some sort of consistent ring > hashing to scale out different tasks. Many tasks in bro are easily > partitioned, like logging and sumstats. > > I wasn't implying poor code just code not optimized for our deployment. Maybe the multiple logger approach would do it but in the meanwhile I'm looking for a quick fix. > > Maybe streaming logs via Kafka and disabling writing to disk has a > chance. > > Ah! if that is your end goal, you could try looking into having your > workers write directly to kafka and bypass the manager entirely. > > I thought there was some degree of normalization that occurred at the manager node ? Would having workers write directly to Kafka limit any features of Bro ? What you are saying sounds like using Kafka on the manager isn't going to fix anything as it will encounter the same resource bottleneck. Here's the config I was going to use: # Kafka output #@load logs-to-kafka.bro #redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG etc...); #redef Kafka::kafka_conf = table( # ["metadata.broker.list"] = "10.1.1.1:9092" #); -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161222/ac995842/attachment.html From jazoff at illinois.edu Thu Dec 22 10:23:14 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 22 Dec 2016 18:23:14 +0000 Subject: [Bro] Bro cluster requirements and manager logging backlog bug In-Reply-To: References: <39591A79-2357-45E1-AFE5-7009E764326E@illinois.edu> <4A126664-FC0D-465B-896C-0BD93809CE33@illinois.edu> <6EDF7C86-443B-4441-8A74-00C09A653845@illinois.edu> Message-ID: > On Dec 22, 2016, at 1:07 PM, Hovsep Levi wrote: > > There may be some inefficiencies in the thread queuing code the logger uses, but the only people that seem to have these major issues have the slow AMD cpus. > > Multiple loggers is something we hope to add once broker is integrated. There's a few places I hope to be able to do some sort of consistent ring hashing to scale out different tasks. Many tasks in bro are easily partitioned, like logging and sumstats. > > > I wasn't implying poor code just code not optimized for our deployment. Maybe the multiple logger approach would do it but in the meanwhile I'm looking for a quick fix. Yeah.. mostly it's just not designed or optimized for systems that have 48 slow cores.. It's possible that broker may end up performing a bit better (since it uses threads for more things) but at some point we would still need to scale out to multiple machines anyway. > > > Maybe streaming logs via Kafka and disabling writing to disk has a chance. > > Ah! if that is your end goal, you could try looking into having your workers write directly to kafka and bypass the manager entirely. > > > > I thought there was some degree of normalization that occurred at the manager node ? Would having workers write directly to Kafka limit any features of Bro ? Not really, the main thing it does is log aggregation and rotation, which if you are sending to kafka you don't really need. > What you are saying sounds like using Kafka on the manager isn't going to fix anything as it will encounter the same resource bottleneck. Correct, it would still go through the same single receiving process. It's probably not that much work (for kafka use cases at least) to be able to run two(or four) logger processes. It's really just a matter of running more of them on different ports and updating the cluster layout so half the workers have one port and half the workers have the other port (broctl already does this kind of thing when you configure multiple proxy nodes). It's a little more complicated for non kafka uses cases because you would end up logger-1/conn.log logger-2/conn.log and log rotation wouldn't work right. I think getting it to work for the kafka use cases may only require a few lines of code to be changed (basically do for logger nodes what broctl already does for proxy nodes) -- - Justin Azoff From jdopheid at illinois.edu Thu Dec 22 12:07:08 2016 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Thu, 22 Dec 2016 20:07:08 +0000 Subject: [Bro] =?utf-8?q?BroCon_=E2=80=9916_videos?= Message-ID: <671C1D35-2A73-41A7-B528-17BBF9AE826C@illinois.edu> Bro Community, The last of the BroCon ?16 videos have been posted to our YouTube channel. I apologize for the delay. The full playlist is available here: https://www.youtube.com/playlist?list=PL2EYTX8UVCMikdxa7U7GXBIblKBlfrslF Thanks, Jeannette Dopheide ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From hovsep.sanjay.levi at gmail.com Thu Dec 22 15:12:22 2016 From: hovsep.sanjay.levi at gmail.com (Hovsep Levi) Date: Thu, 22 Dec 2016 23:12:22 +0000 Subject: [Bro] Bro cluster requirements and manager logging backlog bug In-Reply-To: References: <39591A79-2357-45E1-AFE5-7009E764326E@illinois.edu> <4A126664-FC0D-465B-896C-0BD93809CE33@illinois.edu> <6EDF7C86-443B-4441-8A74-00C09A653845@illinois.edu> Message-ID: > > I think getting it to work for the kafka use cases may only require a few > lines of code to be changed (basically do for logger nodes what broctl > already does for proxy nodes) > > I don't know how to do that. I think I'm confused between what I can do at the script layer vs modifying something in the base. Are you referring to base/frameworks/cluster/setup-connections.bro ? Are you suggesting I override the single logger limit and have each logger export via Kafka ? I guess one logger process for each worker node used by all workers local to that node. Do I need to build a custom cluster-layout to do this ? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161222/26b1cd33/attachment.html From ysrivas at ncsu.edu Thu Dec 22 15:28:27 2016 From: ysrivas at ncsu.edu (Yagyesh Srivastava) Date: Thu, 22 Dec 2016 18:28:27 -0500 Subject: [Bro] Modification of bro source code Message-ID: hi , Does anyone know how to debug in bro other than using lldb? lldb just gives the frame variables of that particular frame, while making modifications in bro source code, require knowing the values of data members of some other class defined somewhere else. One way i can think of is by individually going and checking every variable when its getting populated, but that seems a tedious task considering the multiple inheritance going on. Is there another way to debug? Please let me know, thanks!! Regards, Yagyesh -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161222/1488e8dd/attachment.html From hovsep.sanjay.levi at gmail.com Thu Dec 22 15:42:11 2016 From: hovsep.sanjay.levi at gmail.com (Hovsep Levi) Date: Thu, 22 Dec 2016 23:42:11 +0000 Subject: [Bro] Bro cluster requirements and manager logging backlog bug In-Reply-To: References: <39591A79-2357-45E1-AFE5-7009E764326E@illinois.edu> <4A126664-FC0D-465B-896C-0BD93809CE33@illinois.edu> <6EDF7C86-443B-4441-8A74-00C09A653845@illinois.edu> Message-ID: I can see a number of knobs that could make it happen but I don't know how to go about scripting it. I think it would: - disable logging to manager (done automatically by having a logger node) - bypass the single logger limit - configure each logger to have a writer::kafka - disable other writers if necessary - check if the local worker is part of the same node for the local logger (based on IP address I guess) and use that as a filter for the worker2logger events Starting from bro_init () I don't know how to do this or if it can be done in conjunction with node.cfg or a custom-layout.bro. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161222/0b51920b/attachment.html From hovsep.sanjay.levi at gmail.com Thu Dec 22 16:22:16 2016 From: hovsep.sanjay.levi at gmail.com (Hovsep Levi) Date: Fri, 23 Dec 2016 00:22:16 +0000 Subject: [Bro] Bro cluster requirements and manager logging backlog bug In-Reply-To: References: <39591A79-2357-45E1-AFE5-7009E764326E@illinois.edu> <4A126664-FC0D-465B-896C-0BD93809CE33@illinois.edu> <6EDF7C86-443B-4441-8A74-00C09A653845@illinois.edu> Message-ID: Or via local-logger.bro ? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161223/6abb2ffa/attachment.html From jazoff at illinois.edu Thu Dec 22 20:42:39 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 23 Dec 2016 04:42:39 +0000 Subject: [Bro] Bro cluster requirements and manager logging backlog bug In-Reply-To: References: <39591A79-2357-45E1-AFE5-7009E764326E@illinois.edu> <4A126664-FC0D-465B-896C-0BD93809CE33@illinois.edu> <6EDF7C86-443B-4441-8A74-00C09A653845@illinois.edu> Message-ID: > > On Dec 22, 2016, at 6:12 PM, Hovsep Levi wrote: > > Do I need to build a custom cluster-layout to do this ? For a proof of concept this is really the only thing that needs to be done. I think broctl has one check to prevent you from configuring two loggers. If that was removed and the code that generates cluster layout was amended to use more than one (like it does for proxies) I think it would mostly just work. -- - Justin Azoff From philosnef at gmail.com Fri Dec 23 08:47:32 2016 From: philosnef at gmail.com (erik clark) Date: Fri, 23 Dec 2016 11:47:32 -0500 Subject: [Bro] AF_PACKET and RHEL7 Message-ID: A patch for ixgbe has been pushed upstream. Estimated fix is q3 2017 in RH7.4. It may make it earlier. Those interested in running AF_PACKET in the meantime are free to send me an offlist email on how to get this running. Performance exceeds that of pf_ring from test data on a 6 Gb/s link, so we internally will at least be moving to it. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161223/cbbc65e3/attachment-0001.html From rleonar7 at uoregon.edu Fri Dec 23 11:08:55 2016 From: rleonar7 at uoregon.edu (Ryan Leonard) Date: Fri, 23 Dec 2016 11:08:55 -0800 Subject: [Bro] Bro 2.5 Logger crash --> Broken Log Directory naming Message-ID: <03a501d25d50$0221ebb0$0665c310$@uoregon.edu> Hello all, I've recently come to be responsible for a Bro server and am doing my best to keep everything running smoothly at the moment. We are running a cluster configuration on a single physical machine. Recently we updated to Bro 2.5 from 2.4. Additionally, we modified our cluster configuration to enable a 1 logger process alongside 16 workers, 3 proxies, and 1 manager process (prior we were running without the logger and were seeing the manager crashing regularly due to memory constraints). The output log file structure has had a strange file naming for a short period of time around 2am last night. It seems that the incorrect file naming may correspond to the logger having crashed. It seems when the Logger process is being brought back online by the Broctl Cron task, the logger logs to a strange directory naming for some short period of time. Strange log directory naming: [/bro/logs]$ du -h 20* 1.7G 2000-00- 1.7G 2000-59- 3.3G 2010-00- 67G 2016-12-21 160G 2016-12-22 84G 2016-12-23 1.9G 2020-00- 1.6G 2021-16- 5.1G 2030-00- 8.0K 2030-16- 3.2G 2040-00- 1.9G 2040-10- 1.7G 2050-00- 1.9G 2050-05- How can I ensure that when the logger comes online after a crash that it won't use a strange directory naming? Thanks for any thoughts or help! Best Regards, -Ryan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161223/6ec05caf/attachment.html -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: LoggerCrashReport.txt Url: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161223/6ec05caf/attachment.txt From center.mnt at gmail.com Sun Dec 25 02:56:46 2016 From: center.mnt at gmail.com (sec-x sec-x) Date: Sun, 25 Dec 2016 12:56:46 +0200 Subject: [Bro] Extract Executables Message-ID: Hi, I recently used bro IDS - Default Policy (GetTraffic from TAP on the network) and i want to analysis Files. 1- extract all Executables Files from all traffic (http,smb and others protocols). 2- md5 of all files that passed in the traffic. How can i do it? Thanks, CM. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161225/a547cab4/attachment.html From blackhole.em at gmail.com Sun Dec 25 04:56:15 2016 From: blackhole.em at gmail.com (Joe Blow) Date: Sun, 25 Dec 2016 07:56:15 -0500 Subject: [Bro] Extract Executables In-Reply-To: Message-ID: <585fc1f1.956e240a.a606e.fcfa@mx.google.com> An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161225/513037d8/attachment.html From bro at pingtrip.com Tue Dec 27 08:31:19 2016 From: bro at pingtrip.com (Dave Crawford) Date: Tue, 27 Dec 2016 11:31:19 -0500 Subject: [Bro] AF_Packet Activation Message-ID: What is the proper way to activate the AF_Packet plugin? I believe its installed and configured correctly in nodes.cfg $ bro -N Bro::AF_Packet Bro::AF_Packet - Packet acquisition via AF_Packet (dynamic, version 1.0) [lan] type=worker host=192.168.20.141 interface=af_packet::eth1 lb_method=custom lb_procs=2 #af_packet_fanout_id=11 #af_packet_fanout_mode=FANOUT_HASH But if I uncomment the options for fanout ID and mode I receive the errors: Warning: ignoring unrecognized node config option 'af_packet_fanout_id' given for node 'lan' Warning: ignoring unrecognized node config option 'af_packet_fanout_mode' given for node ?lan' Should I be explicitly loading the init.bro script, and if so what would the syntax be to reference the bro/lib/bro/plugins/Bro_AF_Packet/scripts/directory? -Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161227/a23398b3/attachment.html From jan.grashoefer at gmail.com Tue Dec 27 13:53:16 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Tue, 27 Dec 2016 22:53:16 +0100 Subject: [Bro] AF_Packet Activation In-Reply-To: References: Message-ID: Hi Dave, > What is the proper way to activate the AF_Packet plugin? I believe its installed and configured correctly in nodes.cfg in general, using the af_packet:: interface prefix activates the AF_Packet plugin. > $ bro -N Bro::AF_Packet > Bro::AF_Packet - Packet acquisition via AF_Packet (dynamic, version 1.0) > > > [lan] > type=worker > host=192.168.20.141 > interface=af_packet::eth1 > lb_method=custom > lb_procs=2 > #af_packet_fanout_id=11 > #af_packet_fanout_mode=FANOUT_HASH > > But if I uncomment the options for fanout ID and mode I receive the errors: > > Warning: ignoring unrecognized node config option 'af_packet_fanout_id' given for node 'lan' > Warning: ignoring unrecognized node config option 'af_packet_fanout_mode' given for node ?lan' According to your bro -N output you are using the plugin in version 1.0. The additional options have been added in version 1.1. In any case, default values should be set automatically. If you want to set the fanout mode manually, you need to use something like "af_packet_fanout_mode=AF_Packet::FANOUT_HASH" (forgot the prefix in the README, sorry). Hope that helps, Jan From jan.grashoefer at gmail.com Wed Dec 28 03:46:54 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Wed, 28 Dec 2016 12:46:54 +0100 Subject: [Bro] AF_Packet Activation In-Reply-To: <0D0C82F9-A69D-43EF-A232-037827F44888@pingtrip.com> References: <0D0C82F9-A69D-43EF-A232-037827F44888@pingtrip.com> Message-ID: <6647202c-11f0-6948-a617-7c38611a08b0@gmail.com> Hi Dave, > The fanout_id option no longer causes a warning message but setting the fanout mode option as shown in the README file causes an error: > > af_packet_fanout_mode=FANOUT_HASH > > > error in /data/bro/spool/tmp/check-config-lan-1/broctl-config.bro, line 19: unknown identifier FANOUT_HASH, at or near ?FANOUT_HASH" > > >> On Dec 27, 2016, at 4:53 PM, Jan Grash?fer wrote: >> >> If you want to set the fanout mode manually, you need to use something like >> "af_packet_fanout_mode=AF_Packet::FANOUT_HASH" (forgot the prefix in the >> README, sorry). ;) Best regards, Jan From bro at pingtrip.com Wed Dec 28 04:33:37 2016 From: bro at pingtrip.com (Dave Crawford) Date: Wed, 28 Dec 2016 07:33:37 -0500 Subject: [Bro] AF_Packet Activation In-Reply-To: <6647202c-11f0-6948-a617-7c38611a08b0@gmail.com> References: <0D0C82F9-A69D-43EF-A232-037827F44888@pingtrip.com> <6647202c-11f0-6948-a617-7c38611a08b0@gmail.com> Message-ID: <34B3D5D3-0184-417E-A6A3-1EFC34960529@pingtrip.com> Thanks Jan! I misread your response and thought you were referring me to the README when you were actually pointing out the typo. > On Dec 28, 2016, at 6:46 AM, Jan Grash?fer wrote: > > Hi Dave, > >> The fanout_id option no longer causes a warning message but setting the fanout mode option as shown in the README file causes an error: >> >> af_packet_fanout_mode=FANOUT_HASH >> >> >> error in /data/bro/spool/tmp/check-config-lan-1/broctl-config.bro, line 19: unknown identifier FANOUT_HASH, at or near ?FANOUT_HASH" >> >> >>> On Dec 27, 2016, at 4:53 PM, Jan Grash?fer wrote: >>> >>> If you want to set the fanout mode manually, you need to use something like >>> "af_packet_fanout_mode=AF_Packet::FANOUT_HASH" (forgot the prefix in the >>> README, sorry). > > ;) > > Best regards, > Jan From siberkartal at gmail.com Wed Dec 28 06:11:44 2016 From: siberkartal at gmail.com (=?UTF-8?B?QmV5YXogxZ5hcGth?=) Date: Wed, 28 Dec 2016 16:11:44 +0200 Subject: [Bro] Mime-type issues (text/plain and application/x-msdownload) Message-ID: Hi all, I have two questions for the following pcap. Bro says the mime-type as "text/plain" for the response of first HTTP GET request. However, at least, wireshark (and also CapTipper) says it is "text/html". The correct one is text/html, it is clear. I think, bro does not look only Content-Type (maybe due to malicious manipulation), but makes some heuristics. But there should be some issues for this case. The other one is that, there are 3 binary files in this pcap. Bro extracts them pretty fine. However again there are some issues about content-type. While their content type is application/x-msdownload, the http.log and files.log says dash dash (not found). In relation to this issue, I have a local file extract bro script, although I have definition for application/x-msdownload extension, I am not able to set its extension as exe. Since meta$mime_type returns empty. The sample: http://www.malware-traffic-analysis.net/2016/12/13/2016-12-13-pseudoDarkleech-Rig-V-sends-Cerber-ransomware.pcap.zip Thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161228/6a1a8733/attachment.html From michelleacrawley at gmail.com Wed Dec 28 18:58:03 2016 From: michelleacrawley at gmail.com (Michelle Crawley) Date: Wed, 28 Dec 2016 21:58:03 -0500 Subject: [Bro] Fwd: Sending Bro Logs to a Remote Syslog Server In-Reply-To: References: Message-ID: Hello, I am very new to Bro. I have an external Syslog server in my environment that I am trying to send logs to from Bro. I have been searching everywhere and following different tutorials/hints, but I am still having no luck. How should I go about doing this? Thanks, Michelle -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161228/fdaced1e/attachment.html From finalstatic at gmail.com Wed Dec 28 23:06:58 2016 From: finalstatic at gmail.com (Carl Forsythe) Date: Wed, 28 Dec 2016 23:06:58 -0800 Subject: [Bro] Fwd: Sending Bro Logs to a Remote Syslog Server In-Reply-To: References: Message-ID: The easiest way I've found to date is to use rsyslog to pick them up off the file system. A good template/starting point can be found at https://github.com/lruppert/bro-scripts/blob/master/rsyslog/bro-ids.conf Hope this helps. -Carl On Wed, Dec 28, 2016 at 6:58 PM, Michelle Crawley < michelleacrawley at gmail.com> wrote: > Hello, > > I am very new to Bro. I have an external Syslog server in my environment > that I am trying to send logs to from Bro. I have been searching everywhere > and following different tutorials/hints, but I am still having no luck. How > should I go about doing this? > > Thanks, > Michelle > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161228/83d2d4e3/attachment.html From dhoelzer at enclaveforensics.com Thu Dec 29 02:36:32 2016 From: dhoelzer at enclaveforensics.com (=?UTF-8?Q?David_Hoelzer?=) Date: Thu, 29 Dec 2016 10:36:32 +0000 Subject: [Bro] Fwd: Sending Bro Logs to a Remote Syslog Server In-Reply-To: References: Message-ID: <010001594a271718-85eb3a01-b9de-4e39-98fb-4240088ad41d-000000@email.amazonses.com> rsyslogd forwarding the logs with file monitoring. > On Dec 28, 2016, at 9:58 PM, Michelle Crawley wrote: > > Hello, > > I am very new to Bro. I have an external Syslog server in my environment that I am trying to send logs to from Bro. I have been searching everywhere and following different tutorials/hints, but I am still having no luck. How should I go about doing this? > > Thanks, > Michelle > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From ryanstillions at hotmail.com Fri Dec 30 08:00:35 2016 From: ryanstillions at hotmail.com (Ryan Stillions) Date: Fri, 30 Dec 2016 16:00:35 +0000 Subject: [Bro] SHA256 Hash File Analyzer In-Reply-To: References: , Message-ID: I'm curious if anyone has this turned on at scale, on production systems? If so, can you speak to the performance impacts Seth mentioned below? Seth, any thoughts if this would be the same with 2.5 as it was when you originally posted? I didn't see anything specific about it in release notes, so would we be correct to assume the SHA256 analyzer would probably perform the same as what you saw back in Feb 16? Thanks, ryan Sent from Outlook ________________________________ From: bro-bounces at bro.org on behalf of Shawn Homan Sent: Thursday, February 11, 2016 5:39 PM To: Seth Hall Cc: bro at bro.org Subject: Re: [Bro] SHA256 Hash File Analyzer Thanks for the information. I have it turned on in my offline system, but not sure how to measure performance. On Thu, Feb 11, 2016 at 10:40 AM, Seth Hall > wrote: > On Feb 10, 2016, at 4:55 PM, Shawn Homan > wrote: > > I was wondering if anyone can tell me why the sha256 hash functionality isn't turned on by default for the files log. > > I am working on something and needed to turn it on. I normally only use Bro to process pcap files offline and have never used it on a live network. > > Does it cause performance issues? When I was setting the default behavior a few years ago, I did some very weak testing and noticed that if I had md5 and sha1 turned on, the performance impact was ~1%, but it jumped up somewhere between 3-4% when I enabled SHA256. That measurement should be revisited sometime soon though and perhaps even better measurements done to see if that performance impact is still there. Generally though, there is nothing in place which is stopping you from enabling SHA256 file hashes. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161230/40c71e1a/attachment.html From jazoff at illinois.edu Fri Dec 30 08:27:31 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 30 Dec 2016 16:27:31 +0000 Subject: [Bro] SHA256 Hash File Analyzer In-Reply-To: References: Message-ID: <8D113EA8-5546-4AF5-A000-B2A2F4CB6559@illinois.edu> > > On Dec 30, 2016, at 11:00 AM, Ryan Stillions wrote: > > I'm curious if anyone has this turned on at scale, on production systems? If so, can you speak to the performance impacts Seth mentioned below? > > Seth, > any thoughts if this would be the same with 2.5 as it was when you originally posted? I didn't see anything specific about it in release notes, so would we be correct to assume the SHA256 analyzer would probably perform the same as what you saw back in Feb 16? The analyzer really just delegates to openssl to do all the hashing, so you should be able to use openssl to gauge the performance impact: $ openssl speed md5 sha1 sha256 Doing md5 for 3s on 16 size blocks: 6879766 md5's in 3.00s Doing md5 for 3s on 64 size blocks: 5066897 md5's in 3.00s Doing md5 for 3s on 256 size blocks: 2814019 md5's in 3.00s Doing md5 for 3s on 1024 size blocks: 1016906 md5's in 3.00s Doing md5 for 3s on 8192 size blocks: 147949 md5's in 3.00s Doing sha1 for 3s on 16 size blocks: 7763902 sha1's in 3.00s Doing sha1 for 3s on 64 size blocks: 5420584 sha1's in 3.00s Doing sha1 for 3s on 256 size blocks: 2965390 sha1's in 3.00s Doing sha1 for 3s on 1024 size blocks: 1054003 sha1's in 3.00s Doing sha1 for 3s on 8192 size blocks: 147866 sha1's in 3.00s Doing sha256 for 3s on 16 size blocks: 4896135 sha256's in 3.00s Doing sha256 for 3s on 64 size blocks: 2682706 sha256's in 3.00s Doing sha256 for 3s on 256 size blocks: 1131865 sha256's in 3.00s Doing sha256 for 3s on 1024 size blocks: 342980 sha256's in 3.00s Doing sha256 for 3s on 8192 size blocks: 45549 sha256's in 3.00s OpenSSL 1.0.1e-fips 11 Feb 2013 built on: Tue Sep 27 13:37:25 UTC 2016 options:bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -DTERMIO -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -Wa,--noexecstack -DPURIFY -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes md5 36692.09k 108093.80k 240129.62k 347103.91k 403999.40k sha1 41407.48k 115639.13k 253046.61k 359766.36k 403772.76k sha256 26112.72k 57231.06k 96585.81k 117070.51k 124379.14k On a different machine with a different distribution and newer CPUs I get type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes md5 50302.28k 175259.63k 373751.13k 536014.85k 632668.98k sha1 62768.79k 170994.71k 358746.20k 509927.77k 569868.29k sha256 50775.24k 110530.33k 188262.14k 241865.05k 270240.43k The 1024 byte block size and below would be the most relevant for bro. Unless you're using jumbo frames bro shouldn't be doing much with blocks larger than 1500. -- - Justin Azoff