From alajal at gmail.com Sun Mar 3 23:20:02 2019 From: alajal at gmail.com (Mustafa Qasim) Date: Mon, 4 Mar 2019 18:20:02 +1100 Subject: [Zeek] Mixed logging format and corrupt gz files Message-ID: Hi, I've a bro 2.6.1 instance runnign on CentOS 7. I encountered the following error while trying to consume the rotated logs from last couple weeks. I've json logging applied and all the current hour logs in the "logs/current" are in json format. I tried uncompressing few some logs files from last week and observed that only few files at random are in json and reset are in standard format. There is no order in how the json logs appear. Second, I'm unable to uncompress some logs and get the error " invalid compressed data--format violated". This error is also random and doesn't hint at a limited time overall system glitch. Has anyone encountered such an issue? ------ *Mustafa Qasim* PGP: C57E0A7C -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190304/b77c5966/attachment.html From undicizeri at gmail.com Mon Mar 4 00:45:35 2019 From: undicizeri at gmail.com (Federico Foschini) Date: Mon, 4 Mar 2019 09:45:35 +0100 Subject: [Zeek] Fatal error: out of memory in new with version 2.6.1 Message-ID: Hello, I've recently updated Zeek from version `2.5.6` to version `2.6.1`, and sometime (a couple times a day) Zeek crashes with the following error: ``` out of memory in new. 1551685295.400057 fatal error: out of memory in new. ``` I'm managing about 130 deployment of Zeek using the same plugins and configuration and only 3 of them are crashing. What can I do to investigate the problem further? Thanks --Federico Foschini. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190304/64fb52dc/attachment.html From vlad at es.net Mon Mar 4 07:48:12 2019 From: vlad at es.net (Vlad Grigorescu) Date: Mon, 4 Mar 2019 15:48:12 +0000 Subject: [Zeek] Detection of packets with no TCP flags set In-Reply-To: References: Message-ID: This seems like a job for Bro's signature engine: https://docs.zeek.org/en/stable/frameworks/signatures.html Here's an example: http://try.bro.org/#/trybro/saved/303957 The signature I used is visible in the tcp.sig tab, copying it here for posterity: signature tcp-syn-no-flag { > ip-proto == tcp > header tcp[12:2] & 4095 == 2 > event "Flag-less SYN" > } > Someone should double-check my logic there. --Vlad On Thu, Feb 28, 2019 at 7:07 PM Jim Mellander wrote: > Zeek is mainly connection oriented, rather than packet oriented. However, > you *could* write a policy that allows for detection of these packets using > the raw_packet, new_packet, or tcp packet events, bearing in mind the > caveats in the documentation, particularly the expense of triggering events > at the packet level. > > If there is a particular concern about these packets (covert communication > channel, perhaps?), it would be of interest. > > Hope this helps, > > Jim > > > On Thu, Feb 28, 2019 at 9:02 AM anthony kasza > wrote: > >> I tried feeding Zeek two pcap files. >> >> The first was a single TCP SYN packet with the flags nulled out. Zeek >> complained that the pcap only contained TCP control packets. The single >> entry in the conn.log file had a conn_state of OTH. >> >> The second was a single TLS connection over TCP. I nulled out the TCP >> flags of a single encrypted data packet (after the TCP and TLS handshakes >> had completed) and ran it through Zeek. Zeek processed the stream normally, >> with correct files, conn, x509, and ssl log entries, as if the packet I >> changed had the appropriate flags. >> >> Could you say more about the null-flag packets you are referring to? Do >> you know what they are generated from? >> >> -AK >> >> >> On Wed, Feb 27, 2019, 20:51 eshelton wrote: >> >>> Good evening, >>> >>> My Google-fu is failing me right now, so I wanted to reach out to the >>> list to see if anyone has ever attempted to use Zeek to detect packets with >>> no TCP flags set? >>> >>> In Snort land, a signature would look something like this: >>> >>> alert tcp $HOME_NET any -> $EXTERNAL_NET 443 (msg:"LOCAL Port 443 and no >>> TCP flags set"; flags:0; classtype:misc-activity; sid:7;) >>> >>> Before anyone asks, I'll just ahead and state that "yes Virginia, these >>> packets do really exist in the real world..." (though rare). >>> >>> Thanks in advance, >>> >>> -E >>> _______________________________________________ >>> Zeek mailing list >>> zeek at zeek.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek >> >> _______________________________________________ >> Zeek mailing list >> zeek at zeek.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190304/1ef20e1d/attachment.html From bill.de.ping at gmail.com Mon Mar 4 12:26:55 2019 From: bill.de.ping at gmail.com (william de ping) Date: Mon, 4 Mar 2019 22:26:55 +0200 Subject: [Zeek] - Writer for FIFO files In-Reply-To: References: Message-ID: Hi, Eventually I managed to do it. If anyone is interested feel free to contact me on how B On Thu, Feb 28, 2019 at 11:54 PM william de ping wrote: > Hi everyone, > > I was wondering if anyone knows any way for Bro ASCII writer to output > directly to FIFO file ? > I wish to output logs to FIFO file and have a reader app listening to it, > without the need for file postprocessor actions. > > Thanks > B > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190304/291808d3/attachment.html From anthony.kasza at gmail.com Mon Mar 4 10:17:30 2019 From: anthony.kasza at gmail.com (anthony kasza) Date: Mon, 4 Mar 2019 10:17:30 -0800 Subject: [Zeek] Detection of packets with no TCP flags set In-Reply-To: References: Message-ID: I agree with Vlad. His response made me realize I only sent my response to Erin. Sending to the list for everyone's benefit. Try looking for conn.log lines with a conn$state of "OTH" and a conn$history value containing a 'D' or a 'd', indicating the connection carried data. -AK On Mon, Mar 4, 2019, 08:04 Vlad Grigorescu wrote: > This seems like a job for Bro's signature engine: > https://docs.zeek.org/en/stable/frameworks/signatures.html > > Here's an example: http://try.bro.org/#/trybro/saved/303957 > > The signature I used is visible in the tcp.sig tab, copying it here for > posterity: > > signature tcp-syn-no-flag { >> ip-proto == tcp >> header tcp[12:2] & 4095 == 2 >> event "Flag-less SYN" >> } >> > > Someone should double-check my logic there. > > --Vlad > > On Thu, Feb 28, 2019 at 7:07 PM Jim Mellander wrote: > >> Zeek is mainly connection oriented, rather than packet oriented. >> However, you *could* write a policy that allows for detection of these >> packets using the raw_packet, new_packet, or tcp packet events, bearing in >> mind the caveats in the documentation, particularly the expense of >> triggering events at the packet level. >> >> If there is a particular concern about these packets (covert >> communication channel, perhaps?), it would be of interest. >> >> Hope this helps, >> >> Jim >> >> >> On Thu, Feb 28, 2019 at 9:02 AM anthony kasza >> wrote: >> >>> I tried feeding Zeek two pcap files. >>> >>> The first was a single TCP SYN packet with the flags nulled out. Zeek >>> complained that the pcap only contained TCP control packets. The single >>> entry in the conn.log file had a conn_state of OTH. >>> >>> The second was a single TLS connection over TCP. I nulled out the TCP >>> flags of a single encrypted data packet (after the TCP and TLS handshakes >>> had completed) and ran it through Zeek. Zeek processed the stream normally, >>> with correct files, conn, x509, and ssl log entries, as if the packet I >>> changed had the appropriate flags. >>> >>> Could you say more about the null-flag packets you are referring to? Do >>> you know what they are generated from? >>> >>> -AK >>> >>> >>> On Wed, Feb 27, 2019, 20:51 eshelton wrote: >>> >>>> Good evening, >>>> >>>> My Google-fu is failing me right now, so I wanted to reach out to the >>>> list to see if anyone has ever attempted to use Zeek to detect packets with >>>> no TCP flags set? >>>> >>>> In Snort land, a signature would look something like this: >>>> >>>> alert tcp $HOME_NET any -> $EXTERNAL_NET 443 (msg:"LOCAL Port 443 and >>>> no TCP flags set"; flags:0; classtype:misc-activity; sid:7;) >>>> >>>> Before anyone asks, I'll just ahead and state that "yes Virginia, these >>>> packets do really exist in the real world..." (though rare). >>>> >>>> Thanks in advance, >>>> >>>> -E >>>> _______________________________________________ >>>> Zeek mailing list >>>> zeek at zeek.org >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek >>> >>> _______________________________________________ >>> Zeek mailing list >>> zeek at zeek.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek >> >> _______________________________________________ >> Zeek mailing list >> zeek at zeek.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190304/1bb97645/attachment-0001.html From Brett.Warrick at sensato.co Tue Mar 5 08:19:04 2019 From: Brett.Warrick at sensato.co (Brett Warrick) Date: Tue, 5 Mar 2019 16:19:04 +0000 Subject: [Zeek] Detection Capabilities and Extensions Message-ID: Hi everyone! I'm working on a project, and would like the tap the collective expertise and knowledge of the community. I'm seeking to use Zeek to detect certain behaviors, listed below. My questions are as follows: (1) Is Zeek capable of detecting this behavior? (2) Are there, to the best of your knowledge, any additional scripts I could run to extend Zeek's abilities in this regard? (3) Are there any recommendations on how to test that these attacks can be detected? (4) Are there any recommendations for other attacks or scripts we should add besides the default? Your thoughts and advice are most appreciated. Thanks in advance for your time and assistance! General - OpenDNS DNSCrypt - Session Traversal Utilities for NAT (STUN Binding Request) - Internal Host Retrieving External IP Address (ifconfig. me) - GNU/Linux APT User-Agent Outbound likely related to package management - Common 0a0a0a0a Heap Spray String - EXE IsDebuggerPresent (Used in Malware Anti-Debugging) - Observed SSL Cert (IP Lookup - ipify .org) - Delivery & Attack ? Malicious website ? Phishing activity Exploitation & Installation ? Sandworm - CVE-2014-4114 - KuaiZip Adware CnC Checkin Sipvicious - Sipvicious User-Agent Detected (friendly-scanner) - Sipvicious Scan SMB2 - SMB2 NT Create AndX Request For a DLL File - Possible Lateral Movement - SMB2 NT Create AndX Request For a .bat File - SMB2 NT Create AndX Request For an Executable File HTTP - Outgoing Basic Auth Base64 HTTP Password detected unencrypted - Http Client Body contains passphrase= in cleartext - Served Attached HTTP - PE EXE or DLL Windows file download HTTP Applications - Skype User-Agent detected - TeamViewer Dyngate User-Agent Communication - Potential SSH Scan OUTBOUND - MS Remote Desktop Request RDP - RDP connection confirm Environmental Awareness ? Configuration Changed ? Cisco Device ? Desktop Software - Chat Client ? IRC ? Desktop Software - BitCoin ? BitCoin client ? Network Anomaly - Protocol on Unexpected Port ? HTTP on HTTPS ? Confidential Data - Password in Cleartext ? HTTP Reconnaissance & Probing ? Portscan ? Nmap ? Service discovery ? SIPvicious VOIP tool ? Service discovery ? Microsoft Remote Desktop System Compromise ? C&C Communication ? ipinfo.io SSL Certificate ? Trojan infection ? BlackCarat ? Worm infection ? Internal Host scanning ? Trojan infection ? Commonly Abused File Sharing Site Domain ? Suspicious Behaviour ? Suspicious user-agent detected Vulnerable/Outdated Version - Java 1.6.x Detected - Java 1.7.x Detected - Java 1.8.x Detected - Flash Version M1 - Flash Version M2 Android Device Connectivity Check - Android Device (KitKat OS) - Android Device (Marshmallow OS) Kindest Regards, Brett J. Warrick -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190305/1ed0b1b7/attachment.html From konrad.weglowski at gmail.com Tue Mar 5 11:38:23 2019 From: konrad.weglowski at gmail.com (Konrad Weglowski) Date: Tue, 5 Mar 2019 14:38:23 -0500 Subject: [Zeek] Zeek 2.6.1 - packet_filter - unable to filter out traffic Message-ID: Hello, I have added the following to local.bro in order to test filtering out certain traffic from inspection: redef restrict_filters += { ["Exclude Test"] = "not (src host 192.168.204.2 and dst host 8.8.8.8 and dst port 53)" }; Bro restarts no problem but the filter is not taking effect. If I look at packet_filter.log it is not showing up there either: more packet_filter.log #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path packet_filter #open 2019-03-05-19-16-49 #fields ts node filter init success #types time string string bool bool I also tried adding the filter in broctl.cfg as per below: broargs = -f 'not (src host 192.168.204.2 and dst host 8.8.8.8 and dst port 53)' Argument would show up in the bro process list but again filter would not take affect. I have the same configuration in local.bro on another instance running Bro v2.5.5 and it works. Below you can see output from packet_filter.log: more packet_filter.log #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path packet_filter #open 2019-03-05-19-09-48 #fields ts node filter init success #types time string string bool bool 1551812988.372400 bro (ip or not ip) and (not (src host 192.168.204.2 and dst host 8.8.8.8 and dst port 53)) T T Do you know what could be the issue here? Also is there another way of filtering out traffic? For example have a file with bunch of BPF filters that could be passed on to Bro at startup instead of modifying the local.bro file each time traffic needs to be excluded? Thank You Konrad -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190305/f262914f/attachment.html From fmclane at clemson.edu Mon Mar 11 07:56:35 2019 From: fmclane at clemson.edu (Foster McLane) Date: Mon, 11 Mar 2019 14:56:35 +0000 Subject: [Zeek] Bro/Zeek Script Support in Highlight.js Message-ID: Hello all! I've put together a language file for the popular Highlight.js library for the Bro/Zeek script language. Highlight.js is the syntax highlighting library Discord and Slack use and is useful for websites that include automatic client-side syntax highlighting. Let me know if you have any comments or suggestions! If you don't, a thumbs-up on the pull request would be appreciated to gauge how much people want this feature in Highlight.js. GitHub PR Link: https://github.com/highlightjs/highlight.js/pull/1997 Thanks! Foster From michalpurzynski1 at gmail.com Thu Mar 14 02:43:18 2019 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Thu, 14 Mar 2019 02:43:18 -0700 Subject: [Zeek] Replacing the &synchronized attribute in 2.6 Message-ID: Hey, do we have any example how to replace the old &synchronized attribute in the new Broker-powered world? I looked at the documentation (it's extremely verbose) and found nothing that I could relate to. Here is the pattern I'm trying to port to 2.6. It's basically a code that uses the Input Framework, reads some lists, stores them in a simple set and keeps this set synchronized. The problem might be coming from a not obvious behavior of the &synchronized attribute - I had no idea what it did, I knew that it was supposed to be there. The old 2.5.5 documentation mentions this attribute briefly https://docs.zeek.org/en/stable/script-reference/attributes.html &synchronized - synchronize a variable across nodes <- what nodes are we talking about? what's the communication flow here? Who is the producer, who is the consumer? It is impossible to move my scripts over to Broker without answering those questions. type Idx: record { whitelist_ip: subnet; }; global whitelist_scan_ip: set[subnet] = {} *&synchronized*; <-- the boo-boo is here event bro_init() { Input::add_table([$source="scan_ip.txt", $name="whitelist_scan_ip", $idx=Idx, $destination=whitelist_scan_ip, $mode=Input::REREAD]); } -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190314/8449a259/attachment.html From jan.grashoefer at gmail.com Thu Mar 14 03:02:35 2019 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Thu, 14 Mar 2019 11:02:35 +0100 Subject: [Zeek] Replacing the &synchronized attribute in 2.6 In-Reply-To: References: Message-ID: <68a38da4-2590-a406-d503-18404ef8f5c6@gmail.com> On 14/03/2019 10:43, Micha? Purzy?ski wrote: > do we have any example how to replace the old &synchronized attribute in > the new Broker-powered world? I looked at the documentation (it's extremely > verbose) and found nothing that I could relate to. https://docs.zeek.org/en/stable/frameworks/broker.html#porting-guide I guess data stores are the way to go. Jan From hosom at battelle.org Thu Mar 14 05:58:32 2019 From: hosom at battelle.org (Hosom, Stephen M) Date: Thu, 14 Mar 2019 12:58:32 +0000 Subject: [Zeek] Replacing the &synchronized attribute in 2.6 In-Reply-To: <68a38da4-2590-a406-d503-18404ef8f5c6@gmail.com> References: , <68a38da4-2590-a406-d503-18404ef8f5c6@gmail.com> Message-ID: Michal, For the use case in your email, the best option available to you is the Configuration Framework. https://docs.zeek.org/en/stable/frameworks/configuration.html # First file: module TestModule; export { option whitelist_scan_ip: set[subnet] = {}; redef Config::config_files += { "/path/to/my/config.dat" }; } # /path/to/my/config.dat: TestModule::whitelist_scan_ip = 10.1.2.0/24,10.1.3.0/24,10.1.4.0/24 Thanks, Stephen ________________________________ From: zeek-bounces at zeek.org on behalf of Jan Grash?fer Sent: Thursday, March 14, 2019 6:02:35 AM To: zeek at zeek.org Subject: Re: [Zeek] Replacing the &synchronized attribute in 2.6 Message received from outside the Battelle network. Carefully examine it before you open any links or attachments. On 14/03/2019 10:43, Micha? Purzy?ski wrote: > do we have any example how to replace the old &synchronized attribute in > the new Broker-powered world? I looked at the documentation (it's extremely > verbose) and found nothing that I could relate to. https://docs.zeek.org/en/stable/frameworks/broker.html#porting-guide I guess data stores are the way to go. Jan _______________________________________________ Zeek mailing list zeek at zeek.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek From michalpurzynski1 at gmail.com Thu Mar 14 11:15:55 2019 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Thu, 14 Mar 2019 11:15:55 -0700 Subject: [Zeek] Replacing the &synchronized attribute in 2.6 In-Reply-To: References: <68a38da4-2590-a406-d503-18404ef8f5c6@gmail.com> Message-ID: Thanks, using the configuration framework is easier indeed. Just for the sake of discussing some broker code - do we have examples how people replace the &synchronized attribute? On Thu, Mar 14, 2019 at 6:00 AM Hosom, Stephen M wrote: > Michal, > > > For the use case in your email, the best option available to you is the > Configuration Framework. > > > https://docs.zeek.org/en/stable/frameworks/configuration.html > > > # First file: > > module TestModule; > > export { > > option whitelist_scan_ip: set[subnet] = {}; > > redef Config::config_files += { "/path/to/my/config.dat" }; > > } > > > # /path/to/my/config.dat: > > TestModule::whitelist_scan_ip = 10.1.2.0/24,10.1.3.0/24,10.1.4.0/24 > > > > Thanks, > > Stephen > > ________________________________ > From: zeek-bounces at zeek.org on behalf of Jan > Grash?fer > Sent: Thursday, March 14, 2019 6:02:35 AM > To: zeek at zeek.org > Subject: Re: [Zeek] Replacing the &synchronized attribute in 2.6 > > Message received from outside the Battelle network. Carefully examine it > before you open any links or attachments. > > On 14/03/2019 10:43, Micha? Purzy?ski wrote: > > do we have any example how to replace the old &synchronized attribute in > > the new Broker-powered world? I looked at the documentation (it's > extremely > > verbose) and found nothing that I could relate to. > > https://docs.zeek.org/en/stable/frameworks/broker.html#porting-guide > > I guess data stores are the way to go. > Jan > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190314/25e54aa1/attachment.html From soehlert at es.net Thu Mar 14 11:25:11 2019 From: soehlert at es.net (Samuel Oehlert) Date: Thu, 14 Mar 2019 13:25:11 -0500 Subject: [Zeek] Replacing the &synchronized attribute in 2.6 In-Reply-To: References: <68a38da4-2590-a406-d503-18404ef8f5c6@gmail.com> Message-ID: Mike Dopheide wrote a blog post (on the Zeek blog) about that exact topic not too long ago. He had spent a lot of time at work fixing a bug with one of our policies and had this deep dive in the process. It's a good read. https://blog.zeek.org/2018/07/broker-is-coming-part-2-replacing.html - Sam On Thu, Mar 14, 2019 at 1:19 PM Micha? Purzy?ski wrote: > Thanks, using the configuration framework is easier indeed. > > Just for the sake of discussing some broker code - do we have examples how > people replace the &synchronized attribute? > > On Thu, Mar 14, 2019 at 6:00 AM Hosom, Stephen M > wrote: > >> Michal, >> >> >> For the use case in your email, the best option available to you is the >> Configuration Framework. >> >> >> https://docs.zeek.org/en/stable/frameworks/configuration.html >> >> >> # First file: >> >> module TestModule; >> >> export { >> >> option whitelist_scan_ip: set[subnet] = {}; >> >> redef Config::config_files += { "/path/to/my/config.dat" }; >> >> } >> >> >> # /path/to/my/config.dat: >> >> TestModule::whitelist_scan_ip = 10.1.2.0/24,10.1.3.0/24,10.1.4.0/24 >> >> >> >> Thanks, >> >> Stephen >> >> ________________________________ >> From: zeek-bounces at zeek.org on behalf of Jan >> Grash?fer >> Sent: Thursday, March 14, 2019 6:02:35 AM >> To: zeek at zeek.org >> Subject: Re: [Zeek] Replacing the &synchronized attribute in 2.6 >> >> Message received from outside the Battelle network. Carefully examine it >> before you open any links or attachments. >> >> On 14/03/2019 10:43, Micha? Purzy?ski wrote: >> > do we have any example how to replace the old &synchronized attribute in >> > the new Broker-powered world? I looked at the documentation (it's >> extremely >> > verbose) and found nothing that I could relate to. >> >> https://docs.zeek.org/en/stable/frameworks/broker.html#porting-guide >> >> I guess data stores are the way to go. >> Jan >> _______________________________________________ >> Zeek mailing list >> zeek at zeek.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek >> >> _______________________________________________ >> Zeek mailing list >> zeek at zeek.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek >> > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190314/6877d4e6/attachment-0001.html From mnmblair at hotmail.com Thu Mar 14 15:28:52 2019 From: mnmblair at hotmail.com (COLIN BLAIR) Date: Thu, 14 Mar 2019 22:28:52 +0000 Subject: [Zeek] zbalance_ipc and Zeek Message-ID: Hi All, Does anyone have a success story using zbalance_ipc and Zeek. We are getting very high packet loss using zbalance_ipc. When we remove zbalance_ipc, Zeek performs well on pf_ring zero copy with RSS. Any advice is appreciated. R, CB -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190314/cb00f98b/attachment.html From michalpurzynski1 at gmail.com Thu Mar 14 19:42:56 2019 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Thu, 14 Mar 2019 19:42:56 -0700 Subject: [Zeek] Replacing the &synchronized attribute in 2.6 In-Reply-To: References: <68a38da4-2590-a406-d503-18404ef8f5c6@gmail.com> Message-ID: Thanks - this is exactly what I was Googling for (and could not find). On Thu, Mar 14, 2019 at 11:25 AM Samuel Oehlert wrote: > Mike Dopheide wrote a blog post (on the Zeek blog) about that exact topic > not too long ago. He had spent a lot of time at work fixing a bug with one > of our policies and had this deep dive in the process. It's a good read. > > https://blog.zeek.org/2018/07/broker-is-coming-part-2-replacing.html > > - Sam > > On Thu, Mar 14, 2019 at 1:19 PM Micha? Purzy?ski < > michalpurzynski1 at gmail.com> wrote: > >> Thanks, using the configuration framework is easier indeed. >> >> Just for the sake of discussing some broker code - do we have examples >> how people replace the &synchronized attribute? >> >> On Thu, Mar 14, 2019 at 6:00 AM Hosom, Stephen M >> wrote: >> >>> Michal, >>> >>> >>> For the use case in your email, the best option available to you is the >>> Configuration Framework. >>> >>> >>> https://docs.zeek.org/en/stable/frameworks/configuration.html >>> >>> >>> # First file: >>> >>> module TestModule; >>> >>> export { >>> >>> option whitelist_scan_ip: set[subnet] = {}; >>> >>> redef Config::config_files += { "/path/to/my/config.dat" }; >>> >>> } >>> >>> >>> # /path/to/my/config.dat: >>> >>> TestModule::whitelist_scan_ip = 10.1.2.0/24,10.1.3.0/24,10.1.4.0/24 >>> >>> >>> >>> Thanks, >>> >>> Stephen >>> >>> ________________________________ >>> From: zeek-bounces at zeek.org on behalf of Jan >>> Grash?fer >>> Sent: Thursday, March 14, 2019 6:02:35 AM >>> To: zeek at zeek.org >>> Subject: Re: [Zeek] Replacing the &synchronized attribute in 2.6 >>> >>> Message received from outside the Battelle network. Carefully examine it >>> before you open any links or attachments. >>> >>> On 14/03/2019 10:43, Micha? Purzy?ski wrote: >>> > do we have any example how to replace the old &synchronized attribute >>> in >>> > the new Broker-powered world? I looked at the documentation (it's >>> extremely >>> > verbose) and found nothing that I could relate to. >>> >>> https://docs.zeek.org/en/stable/frameworks/broker.html#porting-guide >>> >>> I guess data stores are the way to go. >>> Jan >>> _______________________________________________ >>> Zeek mailing list >>> zeek at zeek.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek >>> >>> _______________________________________________ >>> Zeek mailing list >>> zeek at zeek.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek >>> >> _______________________________________________ >> Zeek mailing list >> zeek at zeek.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190314/335cb77d/attachment.html From bill.de.ping at gmail.com Sun Mar 17 06:08:50 2019 From: bill.de.ping at gmail.com (william de ping) Date: Sun, 17 Mar 2019 15:08:50 +0200 Subject: [Zeek] zbalance_ipc and Zeek In-Reply-To: References: Message-ID: Hi Colin, Can you please clarify your deployment ? (node.cfg file, NIC type, PF_RING version, zbalance_ipc parameters and the ingress line rate ) Thanks B On Fri, Mar 15, 2019 at 12:38 AM COLIN BLAIR wrote: > Hi All, > > Does anyone have a success story using zbalance_ipc and Zeek. We are > getting very high packet loss using zbalance_ipc. When we remove > zbalance_ipc, Zeek performs well on pf_ring zero copy with RSS. Any advice > is appreciated. > > R, > CB > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190317/8ad05fad/attachment.html From mnmblair at hotmail.com Sun Mar 17 05:34:04 2019 From: mnmblair at hotmail.com (C Blair) Date: Sun, 17 Mar 2019 12:34:04 +0000 Subject: [Zeek] zbalance_ipc and Zeek Message-ID: Hi Bill, Thank you for the assist. Currently, Zeek cannot reliably capture more than 300Mbps with this configuration. When I remove zbalance_ipc and use RSS with num_rss_queues=lb_procs Zeek can capture up to 2Gbps. I need to use zbalance_ipc because I use a single capture interface with multiple consuming applications, i.e. Zeek and Snort. It seems obvious that a software load balancer will perform less than hardware, however, I don't see the same significant performance drop with other consuming applications like Snort. Ingress Line speed: I am using a traffic generator so I can regulate up to 10Gbps. ZEEK node.cfg [manager] type=manager host=localhost [logger] type=logger host=localhost [proxy-1] type=proxy host=localhost [worker-1] type=worker host=localhost interface=zc:99 lb_method=pf_ring lb_procs=10 pin_cpus=1,2,3,4,5,6,7,8,9,10 ZBALANCE_IPC run config zbalance_ipc -i zc:eth0 -c 99 -n 10 -m 4 -g 15 -S 0 PFRING-ZC INFO PF_RING Version : 7.5.0 (unknown) Total rings : 22 Standard (non ZC) Options Ring slots : 65536 Slot version : 17 Capture TX : No [RX only] IP Defragment : No Socket Mode : Standard Cluster Fragment Queue : 0 Cluster Fragment Discard : 0 Name : eth? Index : 40 Address : XX:XX:XX:XX:XX:XX Polling Mode : NAPI/ZC Type : Ethernet Family : ixgbe TX Queues : 1 RX Queues : 1 Num RX Slots : 32768 Num TX Slots : 32768 System Specs: Xeon D-1587 16 cores, 32 logical, 1.7 Ghz, 2.3 Ghz turbo, 20M Cache 128GB DDR4 2133Mhz 8TB SSD Intel 10GBase-T X557 ixgbe On Mar 17, 2019, at 9:08 AM, william de ping wrote: Hi Colin, Can you please clarify your deployment ? (node.cfg file, NIC type, PF_RING version, zbalance_ipc parameters and the ingress line rate ) Thanks B On Fri, Mar 15, 2019 at 12:38 AM COLIN BLAIR > wrote: Hi All, Does anyone have a success story using zbalance_ipc and Zeek. We are getting very high packet loss using zbalance_ipc. When we remove zbalance_ipc, Zeek performs well on pf_ring zero copy with RSS. Any advice is appreciated. R, CB _______________________________________________ Zeek mailing list zeek at zeek.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190317/b8475049/attachment.html From bill.de.ping at gmail.com Sun Mar 17 10:38:26 2019 From: bill.de.ping at gmail.com (william de ping) Date: Sun, 17 Mar 2019 19:38:26 +0200 Subject: [Zeek] zbalance_ipc and Zeek In-Reply-To: References: Message-ID: Hi, I would check the followings : - Numa node configuration - This server should have 2 CPU sockets, if you pinned zbalance_ipc to a numa node which is not directly connected to the PCI bus hosting the NIC all traffic will go through the QPI and that could explain why it will be slower. I would check that the zbalance_ipc app is pinned to the CPU socket that is closer to the PCI NIC to avoid this - Check line rate on each virtual interface using PF_RING/userland/examples/pfcount. check on : zc:99@[0,1,...,9] after using zbalance_ipc and without zbalance_ipc using the RSS. This should give you a clue if there is a specific worker instance that is receiving significantly more traffic than others (RSS and zbalance_ipc LB might differ). It really depends on the type of traffic, but I assume that on a 2.3Ghz processor, a single bro worker can process anything between 150-400mbps. - Run a single instance of bro with local configuration and dump-events.bro script (you can redef include_args=F to get only events name without parameters). Output, sort, uniq -c it to get a clue on what event occur more often. Some analyzers might be turned off to save CPU cycles. Let me know if it helps B On Sun, Mar 17, 2019 at 2:34 PM C Blair wrote: > Hi Bill, > Thank you for the assist. Currently, Zeek cannot reliably capture more > than 300Mbps with this configuration. When I remove zbalance_ipc and use > RSS with num_rss_queues=lb_procs Zeek can capture up to 2Gbps. I need to > use zbalance_ipc because I use a single capture interface with multiple > consuming applications, i.e. Zeek and Snort. It seems obvious that a > software load balancer will perform less than hardware, however, I don't > see the same significant performance drop with other consuming applications > like Snort. > > Ingress Line speed: > I am using a traffic generator so I can regulate up to 10Gbps. > > ZEEK node.cfg > > [manager] > type=manager > host=localhost > > [logger] > type=logger > host=localhost > > [proxy-1] > type=proxy > host=localhost > > [worker-1] > type=worker > host=localhost > interface=zc:99 > lb_method=pf_ring > lb_procs=10 > pin_cpus=1,2,3,4,5,6,7,8,9,10 > > > ZBALANCE_IPC run config > > zbalance_ipc -i zc:eth0 -c 99 -n 10 -m 4 -g 15 -S 0 > > > PFRING-ZC INFO > > PF_RING Version : 7.5.0 (unknown) > Total rings : 22 > Standard (non ZC) Options > Ring slots : 65536 > Slot version : 17 > Capture TX : No [RX only] > IP Defragment : No > Socket Mode : Standard > Cluster Fragment Queue : 0 > Cluster Fragment Discard : 0 > Name : eth? > Index : 40 > Address : XX:XX:XX:XX:XX:XX > Polling Mode : NAPI/ZC > Type : Ethernet > Family : ixgbe > TX Queues : 1 > RX Queues : 1 > Num RX Slots : 32768 > Num TX Slots : 32768 > > > System Specs: > Xeon D-1587 16 cores, 32 logical, 1.7 Ghz, 2.3 Ghz turbo, 20M Cache > 128GB DDR4 2133Mhz > 8TB SSD > Intel 10GBase-T X557 ixgbe > > > On Mar 17, 2019, at 9:08 AM, william de ping > wrote: > > Hi Colin, > > Can you please clarify your deployment ? (node.cfg file, NIC type, PF_RING > version, zbalance_ipc parameters and the ingress line rate ) > > Thanks > B > > On Fri, Mar 15, 2019 at 12:38 AM COLIN BLAIR wrote: > > Hi All, > > Does anyone have a success story using zbalance_ipc and Zeek. We are > getting very high packet loss using zbalance_ipc. When we remove > zbalance_ipc, Zeek performs well on pf_ring zero copy with RSS. Any advice > is appreciated. > > R, > CB > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190317/381311e4/attachment-0001.html From m.dariuz at gmail.com Mon Mar 18 02:49:32 2019 From: m.dariuz at gmail.com (Dario Mohaddes) Date: Mon, 18 Mar 2019 05:49:32 -0400 Subject: [Zeek] Can Zeek be installed as in-line IPS? Message-ID: I'm starting a comparison paper about inline Network IPS. I was looking for an opensource anomaly-based detection engine with IPS capabilities. The easiest choice seemed Zeek but from the website user-manual it doesn't look like it actually supports packets dropping, instead can only work as IDS. Digging a bit online I found a lot of confusion and contradictions with people asserting either that is possible or not but none giving a practical example. I have scraped a multitude of academic and research papers but they haven?t help... I was wondering if anyone can tell me if is feasible before wasting hours trying to do something that is not. Any help or insight is much appreciated. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190318/49cd4817/attachment.html From blackhole.em at gmail.com Mon Mar 18 04:55:03 2019 From: blackhole.em at gmail.com (Joe Blow) Date: Mon, 18 Mar 2019 07:55:03 -0400 Subject: [Zeek] Can Zeek be installed as in-line IPS? In-Reply-To: Message-ID: <7sohv27mv67a7ffecqs28p54.1552910103918@gmail.com> An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190318/edbce1c8/attachment.html From richard at corelight.com Mon Mar 18 08:09:23 2019 From: richard at corelight.com (Richard Bejtlich) Date: Mon, 18 Mar 2019 11:09:23 -0400 Subject: [Zeek] Can Zeek be installed as in-line IPS? In-Reply-To: References: Message-ID: JB's answer was great. I'd only add that I don't think of Zeek as an IDS. Zeek is a network security monitor. It's designed to describe what's happening on your network in a mostly neutral way. It's up to the analyst to use that data for a variety of purposes, one of which could be intrusion detection. Suricata and Snort are more characteristic of an "IDS" because they make judgements about what they see, although Suricata has been integrating ever more NSM functionality by logging DNS, HTTP, etc. as Zeek does. Aside from web application firewalls, I think the IPS market is fairly dead anyway with the ubiquity of encrypted north-south network traffic. Sincerely, Richard On Mon, Mar 18, 2019 at 6:04 AM Dario Mohaddes wrote: > I'm starting a comparison paper about inline Network IPS. I was looking > for an opensource anomaly-based detection engine with IPS capabilities. The > easiest choice seemed Zeek but from the website user-manual it doesn't look > like it actually supports packets dropping, instead can only work as IDS. > Digging a bit online I found a lot of confusion and contradictions with > people asserting either that is possible or not but none giving a practical > example. I have scraped a multitude of academic and research papers but > they haven?t help... I was wondering if anyone can tell me if is feasible > before wasting hours trying to do something that is not. Any help or > insight is much appreciated. Thank you. > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -- Richard Bejtlich Principal Security Strategist, Corelight -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190318/4902640f/attachment.html From patrick.kelley at criticalpathsecurity.com Mon Mar 18 08:30:57 2019 From: patrick.kelley at criticalpathsecurity.com (Patrick Kelley) Date: Mon, 18 Mar 2019 11:30:57 -0400 Subject: [Zeek] Can Zeek be installed as in-line IPS? In-Reply-To: References: Message-ID: Had me all the way until... "Aside from web application firewalls, I think the IPS market is fairly dead anyway with the ubiquity of encrypted north-south network traffic.". I still see the same issues we had on networks 10 years ago. It is reduced, due to HTTPS and some SMTP, sure. Dead... not really. On Mon, Mar 18, 2019 at 11:19 AM Richard Bejtlich wrote: > JB's answer was great. I'd only add that I don't think of Zeek as an IDS. > Zeek is a network security monitor. It's designed to describe what's > happening on your network in a mostly neutral way. It's up to the analyst > to use that data for a variety of purposes, one of which could be intrusion > detection. Suricata and Snort are more characteristic of an "IDS" because > they make judgements about what they see, although Suricata has been > integrating ever more NSM functionality by logging DNS, HTTP, etc. as Zeek > does. > > Aside from web application firewalls, I think the IPS market is fairly > dead anyway with the ubiquity of encrypted north-south network traffic. > > Sincerely, > > Richard > > On Mon, Mar 18, 2019 at 6:04 AM Dario Mohaddes wrote: > >> I'm starting a comparison paper about inline Network IPS. I was looking >> for an opensource anomaly-based detection engine with IPS capabilities. The >> easiest choice seemed Zeek but from the website user-manual it doesn't look >> like it actually supports packets dropping, instead can only work as IDS. >> Digging a bit online I found a lot of confusion and contradictions with >> people asserting either that is possible or not but none giving a practical >> example. I have scraped a multitude of academic and research papers but >> they haven?t help... I was wondering if anyone can tell me if is feasible >> before wasting hours trying to do something that is not. Any help or >> insight is much appreciated. Thank you. >> _______________________________________________ >> Zeek mailing list >> zeek at zeek.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > > > > -- > Richard Bejtlich > Principal Security Strategist, Corelight > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -- *Patrick Kelley, CISSP, C|EH, ITIL* *CTO* patrick.kelley at criticalpathsecurity.com (o) 770-224-6482 *The limit to which you have accepted being comfortable is the limit to which you have grown. Accept new challenges as an opportunity to enrich yourself and not as a point of potential failure.* -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190318/f25d68fb/attachment-0001.html From phatbuckett at gmail.com Mon Mar 18 09:16:50 2019 From: phatbuckett at gmail.com (Darren S.) Date: Mon, 18 Mar 2019 09:16:50 -0700 Subject: [Zeek] Can Zeek be installed as in-line IPS? In-Reply-To: References: Message-ID: Yes, exactly. We need to be careful with our messaging on this as a community because the number of threats still seen (and more generally, the amount of metadata from traffic that can be successfully logged to support NSM) is still significant. Richard said "fairly dead" but casual readers and the tech press tend to take that as a soundbyte and parrot it out as "it's basically all encrypted, don't worry about it." I have had customers that have refuse an option to deploy a network sensor like Zeek or Suricata in their environment in the role of NSM sensors because of this erroneous belief (and a convenient chance to save some capex not buying more hardware). It's disappointing because we see a lot of success detecting badness in other environments so these customers willfully put themselves at a disadvantage to attackers who still operate over cleartext protocols. - Darren On Mon, Mar 18, 2019 at 8:46 AM Patrick Kelley < patrick.kelley at criticalpathsecurity.com> wrote: > Had me all the way until... > > "Aside from web application firewalls, I think the IPS market is fairly > dead anyway with the ubiquity of encrypted north-south network traffic.". > > I still see the same issues we had on networks 10 years ago. It is > reduced, due to HTTPS and some SMTP, sure. Dead... not really. > > On Mon, Mar 18, 2019 at 11:19 AM Richard Bejtlich > wrote: > >> JB's answer was great. I'd only add that I don't think of Zeek as an IDS. >> Zeek is a network security monitor. It's designed to describe what's >> happening on your network in a mostly neutral way. It's up to the analyst >> to use that data for a variety of purposes, one of which could be intrusion >> detection. Suricata and Snort are more characteristic of an "IDS" because >> they make judgements about what they see, although Suricata has been >> integrating ever more NSM functionality by logging DNS, HTTP, etc. as Zeek >> does. >> >> Aside from web application firewalls, I think the IPS market is fairly >> dead anyway with the ubiquity of encrypted north-south network traffic. >> >> Sincerely, >> >> Richard >> >> On Mon, Mar 18, 2019 at 6:04 AM Dario Mohaddes >> wrote: >> >>> I'm starting a comparison paper about inline Network IPS. I was looking >>> for an opensource anomaly-based detection engine with IPS capabilities. The >>> easiest choice seemed Zeek but from the website user-manual it doesn't look >>> like it actually supports packets dropping, instead can only work as IDS. >>> Digging a bit online I found a lot of confusion and contradictions with >>> people asserting either that is possible or not but none giving a practical >>> example. I have scraped a multitude of academic and research papers but >>> they haven?t help... I was wondering if anyone can tell me if is feasible >>> before wasting hours trying to do something that is not. Any help or >>> insight is much appreciated. Thank you. >>> _______________________________________________ >>> Zeek mailing list >>> zeek at zeek.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek >> >> >> >> -- >> Richard Bejtlich >> Principal Security Strategist, Corelight >> _______________________________________________ >> Zeek mailing list >> zeek at zeek.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > > > > -- > > *Patrick Kelley, CISSP, C|EH, ITIL* > *CTO* > patrick.kelley at criticalpathsecurity.com > (o) 770-224-6482 > > *The limit to which you have accepted being comfortable is the limit to > which you have grown. Accept new challenges as an opportunity to enrich > yourself and not as a point of potential failure.* > > > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -- Darren Spruell phatbuckett at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190318/91c871d8/attachment.html From seth at corelight.com Mon Mar 18 10:02:20 2019 From: seth at corelight.com (Seth Hall) Date: Mon, 18 Mar 2019 13:02:20 -0400 Subject: [Zeek] Can Zeek be installed as in-line IPS? In-Reply-To: References: Message-ID: <8A83DD10-3A09-4FB8-9274-0B87552C545F@corelight.com> On 18 Mar 2019, at 11:30, Patrick Kelley wrote: > I still see the same issues we had on networks 10 years ago. It is > reduced, due to HTTPS and some SMTP, sure.? Dead... not really.? To be fair, he did say IPS. In my opinion IPS has always been in a weird spot where the definition isn't terribly clear (block a single packet in-flight? block a connection after a determination is made? ...etc). I think everyone here will agree that the visibility provided by Zeek is useful even on modern networks and that tail of completely unencrypted traffic is awfully long. :) .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From seth at corelight.com Mon Mar 18 10:05:10 2019 From: seth at corelight.com (Seth Hall) Date: Mon, 18 Mar 2019 13:05:10 -0400 Subject: [Zeek] zbalance_ipc and Zeek In-Reply-To: References: Message-ID: <8B6F0549-B3D7-4E80-AC2F-72EC322B016D@corelight.com> On 14 Mar 2019, at 18:28, COLIN BLAIR wrote: > Does anyone have a success story using zbalance_ipc and Zeek. We are > getting very high packet loss using zbalance_ipc. When we remove > zbalance_ipc, Zeek performs well on pf_ring zero copy with RSS. Any > advice is appreciated. I tried it long ago and got it working correctly. One thing you may need to keep in mind is that if a single one of your workers dies or is restarted, you need to shut them all down and possibly even restart zbalance_ipc and then bring up all of the workers again. I don't know if this is still the behavior of pf_ring anymore but it was at one point. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From seth at corelight.com Mon Mar 18 10:08:17 2019 From: seth at corelight.com (Seth Hall) Date: Mon, 18 Mar 2019 13:08:17 -0400 Subject: [Zeek] Bro/Zeek Script Support in Highlight.js In-Reply-To: References: Message-ID: <5AED843D-B8D6-4DB9-B4E9-8AAF275683BA@corelight.com> On 11 Mar 2019, at 10:56, Foster McLane wrote: > I've put together a language file for the popular Highlight.js library > for the Bro/Zeek script language. Highlight.js is the syntax > highlighting library Discord and Slack use and is useful for websites > that include automatic client-side syntax highlighting. Nice! Thanks! I love as all of these highlighter supports show up because it means that the language keeps being highlighted correctly on all of these services. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From jlay at slave-tothe-box.net Mon Mar 18 10:16:54 2019 From: jlay at slave-tothe-box.net (James Lay) Date: Mon, 18 Mar 2019 11:16:54 -0600 Subject: [Zeek] Can Zeek be installed as in-line IPS? In-Reply-To: <8A83DD10-3A09-4FB8-9274-0B87552C545F@corelight.com> References: <8A83DD10-3A09-4FB8-9274-0B87552C545F@corelight.com> Message-ID: <9d36cd35005ab80212bf0b06c73dda14@slave-tothe-box.net> On 2019-03-18 11:02, Seth Hall wrote: > On 18 Mar 2019, at 11:30, Patrick Kelley wrote: > >> I still see the same issues we had on networks 10 years ago. It is >> reduced, due to HTTPS and some SMTP, sure.? Dead... not really.? > > To be fair, he did say IPS. In my opinion IPS has always been in a > weird spot where the definition isn't terribly clear (block a single > packet in-flight? block a connection after a determination is made? > ...etc). > > I think everyone here will agree that the visibility provided by Zeek > is > useful even on modern networks and that tail of completely unencrypted > traffic is awfully long. :) > > .Seth > > -- > Seth Hall * Corelight, Inc * www.corelight.com > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek Concur. Zeek on the perimeter is great for metadata about encrypted sessions. Zeek internally from client/server or Windows Client/Windows Domain Controller will open your eyes to a LOT of traffic you may not have expected. James From richard at corelight.com Mon Mar 18 10:40:06 2019 From: richard at corelight.com (Richard Bejtlich) Date: Mon, 18 Mar 2019 13:40:06 -0400 Subject: [Zeek] Can Zeek be installed as in-line IPS? In-Reply-To: <8A83DD10-3A09-4FB8-9274-0B87552C545F@corelight.com> References: <8A83DD10-3A09-4FB8-9274-0B87552C545F@corelight.com> Message-ID: Yes, as Seth said, I said IPS. Is anyone really deploying IPS now? I only see Palo Alto firewalls, etc. Sincerely, Richard On Mon, Mar 18, 2019 at 1:02 PM Seth Hall wrote: > > > On 18 Mar 2019, at 11:30, Patrick Kelley wrote: > > > I still see the same issues we had on networks 10 years ago. It is > > reduced, due to HTTPS and some SMTP, sure. Dead... not really. > > To be fair, he did say IPS. In my opinion IPS has always been in a > weird spot where the definition isn't terribly clear (block a single > packet in-flight? block a connection after a determination is made? > ...etc). > > I think everyone here will agree that the visibility provided by Zeek is > useful even on modern networks and that tail of completely unencrypted > traffic is awfully long. :) > > .Seth > > -- > Seth Hall * Corelight, Inc * www.corelight.com > -- Richard Bejtlich Principal Security Strategist, Corelight -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190318/5dbbca4c/attachment.html From patrick.kelley at criticalpathsecurity.com Mon Mar 18 10:49:50 2019 From: patrick.kelley at criticalpathsecurity.com (Patrick Kelley) Date: Mon, 18 Mar 2019 13:49:50 -0400 Subject: [Zeek] Can Zeek be installed as in-line IPS? In-Reply-To: References: <8A83DD10-3A09-4FB8-9274-0B87552C545F@corelight.com> Message-ID: Yes. Many. PCI-DSS 11.4 comes up quite often. Whether we have consensus on the validity and utility of an IPS or not, it comes up in every single PCI audit. On Mon, Mar 18, 2019 at 1:40 PM Richard Bejtlich wrote: > Yes, as Seth said, I said IPS. Is anyone really deploying IPS now? I only > see Palo Alto firewalls, etc. > > Sincerely, > > Richard > > On Mon, Mar 18, 2019 at 1:02 PM Seth Hall wrote: > >> >> >> On 18 Mar 2019, at 11:30, Patrick Kelley wrote: >> >> > I still see the same issues we had on networks 10 years ago. It is >> > reduced, due to HTTPS and some SMTP, sure. Dead... not really. >> >> To be fair, he did say IPS. In my opinion IPS has always been in a >> weird spot where the definition isn't terribly clear (block a single >> packet in-flight? block a connection after a determination is made? >> ...etc). >> >> I think everyone here will agree that the visibility provided by Zeek is >> useful even on modern networks and that tail of completely unencrypted >> traffic is awfully long. :) >> >> .Seth >> >> -- >> Seth Hall * Corelight, Inc * www.corelight.com >> > > > -- > Richard Bejtlich > Principal Security Strategist, Corelight > -- *Patrick Kelley, CISSP, C|EH, ITIL* *CTO* patrick.kelley at criticalpathsecurity.com (o) 770-224-6482 *The limit to which you have accepted being comfortable is the limit to which you have grown. Accept new challenges as an opportunity to enrich yourself and not as a point of potential failure.* -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190318/7d0aa693/attachment.html From richard at corelight.com Mon Mar 18 11:28:16 2019 From: richard at corelight.com (Richard Bejtlich) Date: Mon, 18 Mar 2019 14:28:16 -0400 Subject: [Zeek] Can Zeek be installed as in-line IPS? In-Reply-To: References: <8A83DD10-3A09-4FB8-9274-0B87552C545F@corelight.com> Message-ID: The PCI requirement is for IDS or IPS, which is unfortunate because they are totally different. I'm surprised IPS is even a market segment anymore. At this point it's really just a firewall feature. There's so much more that can be done with a passive observation platform like Zeek, when you don't have to worry about making line-speed judgements. Sincerely, Richard On Mon, Mar 18, 2019 at 1:50 PM Patrick Kelley < patrick.kelley at criticalpathsecurity.com> wrote: > Yes. Many. > > PCI-DSS 11.4 comes up quite often. Whether we have consensus on the > validity and utility of an IPS or not, it comes up in every single PCI > audit. > > On Mon, Mar 18, 2019 at 1:40 PM Richard Bejtlich > wrote: > >> Yes, as Seth said, I said IPS. Is anyone really deploying IPS now? I only >> see Palo Alto firewalls, etc. >> >> Sincerely, >> >> Richard >> >> On Mon, Mar 18, 2019 at 1:02 PM Seth Hall wrote: >> >>> >>> >>> On 18 Mar 2019, at 11:30, Patrick Kelley wrote: >>> >>> > I still see the same issues we had on networks 10 years ago. It is >>> > reduced, due to HTTPS and some SMTP, sure. Dead... not really. >>> >>> To be fair, he did say IPS. In my opinion IPS has always been in a >>> weird spot where the definition isn't terribly clear (block a single >>> packet in-flight? block a connection after a determination is made? >>> ...etc). >>> >>> I think everyone here will agree that the visibility provided by Zeek is >>> useful even on modern networks and that tail of completely unencrypted >>> traffic is awfully long. :) >>> >>> .Seth >>> >>> -- >>> Seth Hall * Corelight, Inc * www.corelight.com >>> >> >> >> -- >> Richard Bejtlich >> Principal Security Strategist, Corelight >> > > > -- > > *Patrick Kelley, CISSP, C|EH, ITIL* > *CTO* > patrick.kelley at criticalpathsecurity.com > (o) 770-224-6482 > > *The limit to which you have accepted being comfortable is the limit to > which you have grown. Accept new challenges as an opportunity to enrich > yourself and not as a point of potential failure.* > > > -- Richard Bejtlich Principal Security Strategist, Corelight -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190318/caef4133/attachment.html From craig.edgmand at okstate.edu Mon Mar 18 11:39:51 2019 From: craig.edgmand at okstate.edu (Edgmand, Craig) Date: Mon, 18 Mar 2019 18:39:51 +0000 Subject: [Zeek] Can Zeek be installed as in-line IPS? In-Reply-To: References: <8A83DD10-3A09-4FB8-9274-0B87552C545F@corelight.com> Message-ID: Didn?t IDS die circa 2005? ? From: zeek-bounces at zeek.org On Behalf Of Richard Bejtlich Sent: Monday, March 18, 2019 1:28 PM To: Patrick Kelley Cc: zeek at zeek.org Subject: Re: [Zeek] Can Zeek be installed as in-line IPS? **External Email - Please verify sender email address before responding.** The PCI requirement is for IDS or IPS, which is unfortunate because they are totally different. I'm surprised IPS is even a market segment anymore. At this point it's really just a firewall feature. There's so much more that can be done with a passive observation platform like Zeek, when you don't have to worry about making line-speed judgements. Sincerely, Richard On Mon, Mar 18, 2019 at 1:50 PM Patrick Kelley > wrote: Yes. Many. PCI-DSS 11.4 comes up quite often. Whether we have consensus on the validity and utility of an IPS or not, it comes up in every single PCI audit. On Mon, Mar 18, 2019 at 1:40 PM Richard Bejtlich > wrote: Yes, as Seth said, I said IPS. Is anyone really deploying IPS now? I only see Palo Alto firewalls, etc. Sincerely, Richard On Mon, Mar 18, 2019 at 1:02 PM Seth Hall > wrote: On 18 Mar 2019, at 11:30, Patrick Kelley wrote: > I still see the same issues we had on networks 10 years ago. It is > reduced, due to HTTPS and some SMTP, sure. Dead... not really. To be fair, he did say IPS. In my opinion IPS has always been in a weird spot where the definition isn't terribly clear (block a single packet in-flight? block a connection after a determination is made? ...etc). I think everyone here will agree that the visibility provided by Zeek is useful even on modern networks and that tail of completely unencrypted traffic is awfully long. :) .Seth -- Seth Hall * Corelight, Inc * www.corelight.com -- Richard Bejtlich Principal Security Strategist, Corelight -- Patrick Kelley, CISSP, C|EH, ITIL CTO patrick.kelley at criticalpathsecurity.com (o) 770-224-6482 The limit to which you have accepted being comfortable is the limit to which you have grown. Accept new challenges as an opportunity to enrich yourself and not as a point of potential failure. [https://drive.google.com/a/criticalpathsecurity.com/uc?id=0B8pLF9KsqY6YVy1zb3FUUkpmTHM&export=download] -- Richard Bejtlich Principal Security Strategist, Corelight -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190318/4424c619/attachment-0001.html From patrick.kelley at criticalpathsecurity.com Mon Mar 18 11:49:19 2019 From: patrick.kelley at criticalpathsecurity.com (Patrick Kelley) Date: Mon, 18 Mar 2019 14:49:19 -0400 Subject: [Zeek] Can Zeek be installed as in-line IPS? In-Reply-To: References: <8A83DD10-3A09-4FB8-9274-0B87552C545F@corelight.com> Message-ID: Somedays I wish... As I sit here reviewing the deployment and Change Management notes for a ASA/FirePower, two SourceFire 7120's, two 8250's, and two VM FirePowers. Orgs are still trying to get ROI on some of this stuff. It's not AI/ML or Blockchain, but it's still running. On Mon, Mar 18, 2019 at 2:39 PM Edgmand, Craig wrote: > Didn?t IDS die circa 2005? J > > > > *From:* zeek-bounces at zeek.org *On Behalf Of *Richard > Bejtlich > *Sent:* Monday, March 18, 2019 1:28 PM > *To:* Patrick Kelley > *Cc:* zeek at zeek.org > *Subject:* Re: [Zeek] Can Zeek be installed as in-line IPS? > > > > **External Email - Please verify sender email address before responding.** > > The PCI requirement is for IDS or IPS, which is unfortunate because they > are totally different. I'm surprised IPS is even a market segment anymore. > At this point it's really just a firewall feature. There's so much more > that can be done with a passive observation platform like Zeek, when you > don't have to worry about making line-speed judgements. > > > > Sincerely, > > > > Richard > > > > On Mon, Mar 18, 2019 at 1:50 PM Patrick Kelley < > patrick.kelley at criticalpathsecurity.com> wrote: > > Yes. Many. > > > > PCI-DSS 11.4 comes up quite often. Whether we have consensus on the > validity and utility of an IPS or not, it comes up in every single PCI > audit. > > > > On Mon, Mar 18, 2019 at 1:40 PM Richard Bejtlich > wrote: > > Yes, as Seth said, I said IPS. Is anyone really deploying IPS now? I only > see Palo Alto firewalls, etc. > > > > Sincerely, > > > > Richard > > > > On Mon, Mar 18, 2019 at 1:02 PM Seth Hall wrote: > > > > On 18 Mar 2019, at 11:30, Patrick Kelley wrote: > > > I still see the same issues we had on networks 10 years ago. It is > > reduced, due to HTTPS and some SMTP, sure. Dead... not really. > > To be fair, he did say IPS. In my opinion IPS has always been in a > weird spot where the definition isn't terribly clear (block a single > packet in-flight? block a connection after a determination is made? > ...etc). > > I think everyone here will agree that the visibility provided by Zeek is > useful even on modern networks and that tail of completely unencrypted > traffic is awfully long. :) > > .Seth > > -- > Seth Hall * Corelight, Inc * www.corelight.com > > > > > > -- > > Richard Bejtlich > > Principal Security Strategist, Corelight > > > > > -- > > > > *Patrick Kelley, CISSP, C|EH, ITIL* > > *CTO* > > patrick.kelley at criticalpathsecurity.com > > (o) 770-224-6482 > > > > *The limit to which you have accepted being comfortable is the limit to > which you have grown. Accept new challenges as an opportunity to enrich > yourself and not as a point of potential failure.* > > > > > > > -- > > Richard Bejtlich > > Principal Security Strategist, Corelight > -- *Patrick Kelley, CISSP, C|EH, ITIL* *CTO* patrick.kelley at criticalpathsecurity.com (o) 770-224-6482 *The limit to which you have accepted being comfortable is the limit to which you have grown. Accept new challenges as an opportunity to enrich yourself and not as a point of potential failure.* -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190318/1f0c35c3/attachment.html From patrick.kelley at criticalpathsecurity.com Mon Mar 18 12:01:49 2019 From: patrick.kelley at criticalpathsecurity.com (Patrick Kelley) Date: Mon, 18 Mar 2019 15:01:49 -0400 Subject: [Zeek] Can Zeek be installed as in-line IPS? In-Reply-To: References: <8A83DD10-3A09-4FB8-9274-0B87552C545F@corelight.com> Message-ID: @James Lay Couldn't agree more about the metadata and convergence of E/W traffic. Additionally, we've used SIP analyzers to validate the implementation of Zeek as a security platform, as it could provide greater visibility into call center traffic. Showing that it could increase efficiency, while providing a better security posture was a good win. On Mon, Mar 18, 2019 at 2:49 PM Patrick Kelley < patrick.kelley at criticalpathsecurity.com> wrote: > Somedays I wish... > > As I sit here reviewing the deployment and Change Management notes for a > ASA/FirePower, two SourceFire 7120's, two 8250's, and two VM FirePowers. > > Orgs are still trying to get ROI on some of this stuff. It's not AI/ML or > Blockchain, but it's still running. > > On Mon, Mar 18, 2019 at 2:39 PM Edgmand, Craig > wrote: > >> Didn?t IDS die circa 2005? J >> >> >> >> *From:* zeek-bounces at zeek.org *On Behalf Of *Richard >> Bejtlich >> *Sent:* Monday, March 18, 2019 1:28 PM >> *To:* Patrick Kelley >> *Cc:* zeek at zeek.org >> *Subject:* Re: [Zeek] Can Zeek be installed as in-line IPS? >> >> >> >> **External Email - Please verify sender email address before responding.** >> >> The PCI requirement is for IDS or IPS, which is unfortunate because they >> are totally different. I'm surprised IPS is even a market segment anymore. >> At this point it's really just a firewall feature. There's so much more >> that can be done with a passive observation platform like Zeek, when you >> don't have to worry about making line-speed judgements. >> >> >> >> Sincerely, >> >> >> >> Richard >> >> >> >> On Mon, Mar 18, 2019 at 1:50 PM Patrick Kelley < >> patrick.kelley at criticalpathsecurity.com> wrote: >> >> Yes. Many. >> >> >> >> PCI-DSS 11.4 comes up quite often. Whether we have consensus on the >> validity and utility of an IPS or not, it comes up in every single PCI >> audit. >> >> >> >> On Mon, Mar 18, 2019 at 1:40 PM Richard Bejtlich >> wrote: >> >> Yes, as Seth said, I said IPS. Is anyone really deploying IPS now? I only >> see Palo Alto firewalls, etc. >> >> >> >> Sincerely, >> >> >> >> Richard >> >> >> >> On Mon, Mar 18, 2019 at 1:02 PM Seth Hall wrote: >> >> >> >> On 18 Mar 2019, at 11:30, Patrick Kelley wrote: >> >> > I still see the same issues we had on networks 10 years ago. It is >> > reduced, due to HTTPS and some SMTP, sure. Dead... not really. >> >> To be fair, he did say IPS. In my opinion IPS has always been in a >> weird spot where the definition isn't terribly clear (block a single >> packet in-flight? block a connection after a determination is made? >> ...etc). >> >> I think everyone here will agree that the visibility provided by Zeek is >> useful even on modern networks and that tail of completely unencrypted >> traffic is awfully long. :) >> >> .Seth >> >> -- >> Seth Hall * Corelight, Inc * www.corelight.com >> >> >> >> >> >> -- >> >> Richard Bejtlich >> >> Principal Security Strategist, Corelight >> >> >> >> >> -- >> >> >> >> *Patrick Kelley, CISSP, C|EH, ITIL* >> >> *CTO* >> >> patrick.kelley at criticalpathsecurity.com >> >> (o) 770-224-6482 >> >> >> >> *The limit to which you have accepted being comfortable is the limit to >> which you have grown. Accept new challenges as an opportunity to enrich >> yourself and not as a point of potential failure.* >> >> >> >> >> >> >> -- >> >> Richard Bejtlich >> >> Principal Security Strategist, Corelight >> > > > -- > > *Patrick Kelley, CISSP, C|EH, ITIL* > *CTO* > patrick.kelley at criticalpathsecurity.com > (o) 770-224-6482 > > *The limit to which you have accepted being comfortable is the limit to > which you have grown. Accept new challenges as an opportunity to enrich > yourself and not as a point of potential failure.* > > > -- *Patrick Kelley, CISSP, C|EH, ITIL* *CTO* patrick.kelley at criticalpathsecurity.com (o) 770-224-6482 *The limit to which you have accepted being comfortable is the limit to which you have grown. Accept new challenges as an opportunity to enrich yourself and not as a point of potential failure.* -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190318/f1d8a0c9/attachment.html From jsiwek at corelight.com Mon Mar 18 18:35:29 2019 From: jsiwek at corelight.com (Jon Siwek) Date: Mon, 18 Mar 2019 18:35:29 -0700 Subject: [Zeek] Request for Feedback - Zeek Process Supervision Model Message-ID: I just published some design thoughts related to a major new Zeek feature that's planned/upcoming: a process supervision model that may act as an alternative (successor) to BroControl. Find that here: https://blog.zeek.org/2019/03/beyond-brocontrol-new-process.html Feel free to use this mailing list / thread to provide feedback, thanks. - Jon From anthony.kasza at gmail.com Mon Mar 18 18:55:44 2019 From: anthony.kasza at gmail.com (anthony kasza) Date: Mon, 18 Mar 2019 19:55:44 -0600 Subject: [Zeek] Request for Feedback - Zeek Process Supervision Model In-Reply-To: References: Message-ID: I'm excited to see this. I think it's a great design choice. This sentence is my favorite, "We need to make it easy to test, from the command-line, using just PCAP files, a complete cluster deployment (scaled down) as it would work in production." I'm looking forward to it! -AK On Mon, Mar 18, 2019, 19:44 Jon Siwek wrote: > I just published some design thoughts related to a major new Zeek > feature that's planned/upcoming: a process supervision model that may > act as an alternative (successor) to BroControl. Find that here: > > https://blog.zeek.org/2019/03/beyond-brocontrol-new-process.html > > Feel free to use this mailing list / thread to provide feedback, thanks. > > - Jon > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190318/4f004f72/attachment.html From zander.work at oregonstate.edu Tue Mar 19 08:31:23 2019 From: zander.work at oregonstate.edu (Zander Work) Date: Tue, 19 Mar 2019 08:31:23 -0700 Subject: [Zeek] Request for Feedback - Zeek Process Supervision Model In-Reply-To: References: Message-ID: <97e48cb7-97e0-a78c-95fe-1aba9a46afb8@oregonstate.edu> This would be awesome to have, especially in a cluster environment. Testing new scripts before we push them to production is a bit challenging sometimes, so being able to reliably and repeatably test them in a clustered environment would be awesome. Another thing that wasn't mentioned in the blog post that I want to make sure makes it into the new process is something like "broctl cron". It does a great job of restarting processes when they crash, but also monitoring disk usage and other potential issues for us, which is helpful in ensuring the stability of our cluster. I see the "revive zombie nodes" piece in the Google Doc, which covers the first bit, but other host-level monitoring/alerting components would be nice to see in the new process. -Zander On 3/18/19 6:55 PM, anthony kasza wrote: > I'm excited to see this. I think it's a great design choice. This > sentence is my favorite, "We need to make it easy to test, from the > command-line, using just PCAP files, a complete cluster deployment > (scaled down) as it would work in production." > > I'm looking forward to it! > > -AK > > On Mon, Mar 18, 2019, 19:44 Jon Siwek > wrote: > > I just published some design thoughts related to a major new Zeek > feature that's planned/upcoming: a process supervision model that may > act as an alternative (successor) to BroControl.? Find that here: > > https://blog.zeek.org/2019/03/beyond-brocontrol-new-process.html > > Feel free to use this mailing list / thread to provide feedback, > thanks. > > - Jon > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > > > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190319/d75e7972/attachment-0001.html From asavran at layerxtech.com Tue Mar 19 08:38:55 2019 From: asavran at layerxtech.com (Arda Savran) Date: Tue, 19 Mar 2019 11:38:55 -0400 Subject: [Zeek] Bro scripts to write logs to Kafka fails Message-ID: Hello folks: I just installed zeek from source on a CentOS7.5 platform. Everything works fine. I can confirm that the logs are getting populated with http, dns etc. data. I am trying to write the data to a remote Kafka topic. I believe I compiled everything correctly based on the following link: *https://packages.zeek.org/packages/view/7388aa77-4fb7-11e8-88be-0a645a3f3086 * And I am getting the expected return when I execute: * $ bro -N Apache::Kafka* * Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3)* Next, I edited /usr/local/bro/share/bro/site/local.bro and appended the following: *@load packages/metron-bro-plugin-kafka/Apache/Kafka redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG); redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "XX.XX.XX.XX:9092" );* Unfortunately, this part doesn't work: *[root at localhost scripts]# broctl check* *Warning: broctl config has changed (run the broctl "deploy" command)* *bro scripts failed.* *fatal error in /usr/local/bro/share/bro/site/local.bro, line 7: can't find packages/metron-bro-plugin-kafka/Apache/Kafka* Has this happened to anyone before? Is there a fix for this? Thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190319/6ee3db90/attachment.html From patrick.kelley at criticalpathsecurity.com Tue Mar 19 09:17:39 2019 From: patrick.kelley at criticalpathsecurity.com (Patrick Kelley) Date: Tue, 19 Mar 2019 12:17:39 -0400 Subject: [Zeek] Bro scripts to write logs to Kafka fails In-Reply-To: References: Message-ID: Had some issues with this last week when building our ARM/ICS/IOT sensor. Let me see if I can run down some notes. On Tue, Mar 19, 2019 at 11:57 AM Arda Savran wrote: > Hello folks: > > I just installed zeek from source on a CentOS7.5 platform. Everything > works fine. I can confirm that the logs are getting populated with http, > dns etc. data. > > I am trying to write the data to a remote Kafka topic. I believe I > compiled everything correctly based on the following link: *https://packages.zeek.org/packages/view/7388aa77-4fb7-11e8-88be-0a645a3f3086 > * > > And I am getting the expected return when I execute: > > * $ bro -N Apache::Kafka* > * Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3)* > > Next, I edited /usr/local/bro/share/bro/site/local.bro and appended the > following: > > *@load packages/metron-bro-plugin-kafka/Apache/Kafka redef > Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG); redef Kafka::kafka_conf = > table( ["metadata.broker.list"] = "XX.XX.XX.XX:9092" );* > > Unfortunately, this part doesn't work: > > *[root at localhost scripts]# broctl check* > *Warning: broctl config has changed (run the broctl "deploy" command)* > *bro scripts failed.* > *fatal error in /usr/local/bro/share/bro/site/local.bro, line 7: can't > find packages/metron-bro-plugin-kafka/Apache/Kafka* > > Has this happened to anyone before? Is there a fix for this? > > Thanks, > > > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -- *Patrick Kelley, CISSP, C|EH, ITIL* *CTO* patrick.kelley at criticalpathsecurity.com (o) 770-224-6482 *The limit to which you have accepted being comfortable is the limit to which you have grown. Accept new challenges as an opportunity to enrich yourself and not as a point of potential failure.* -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190319/ba44c7c6/attachment.html From jawren at cisco.com Tue Mar 19 12:52:51 2019 From: jawren at cisco.com (Jay Wren (jawren)) Date: Tue, 19 Mar 2019 19:52:51 +0000 Subject: [Zeek] Access the encrypted TLS payload In-Reply-To: References: Message-ID: After some time, I returned to this and learned how binpac actually works. proc_ciphertext_record is called when the record is parsed and before it is added to the container which holds it. To get access to it, pass `this` to the function. e.g. function proc_ciphertext_record(rec : SSLRecord, ct : CiphertextRecord) and refine typeattr CiphertextRecord += &let { proc : bool = $context.connection.proc_ciphertext_record(rec, this); } Thanks, -- Jay ________________________________ From: zeek-bounces at zeek.org on behalf of Jay Wren (jawren) Sent: Thursday, February 28, 2019 11:15 AM To: zeek at zeek.org Subject: [Zeek] Access the encrypted TLS payload Hello, Apologies for my ignorant question, my C++ is worse than rusty and I'm completely new to binpac. I'm trying to access the CiphertextRecord restofdata here: https://github.com/zeek/zeek/blob/master/src/analyzer/protocol/ssl/ssl-dtls-analyzer.pac#L59 I'm expecting SSLRecord to have the data in the rec vector, based on how SSLRecord is defined. I must be misunderstanding something: https://github.com/jrwren/zeek/blob/6f7b2973bd23690b6cac65b4d8c0f8fa64e72758/src/analyzer/protocol/ssl/ssl-dtls-analyzer.pac#L61 The RecordText vector is always empty. How can I get at the encrypted data? Thanks, -- Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190319/fc6098a3/attachment.html From soehlert at es.net Tue Mar 19 13:06:31 2019 From: soehlert at es.net (Samuel Oehlert) Date: Tue, 19 Mar 2019 15:06:31 -0500 Subject: [Zeek] Request for Feedback - Zeek Process Supervision Model In-Reply-To: <97e48cb7-97e0-a78c-95fe-1aba9a46afb8@oregonstate.edu> References: <97e48cb7-97e0-a78c-95fe-1aba9a46afb8@oregonstate.edu> Message-ID: Personally, I think it would be poor design to rebuild host OS monitoring inside the Zeek supervisor. I think that should be left up to the many other projects specifically designed to monitor disk usage, etc. That being said, exposing some metrics about Zeek the application layer sounds like it would be a win. That being said, that might be outside the scope of a supervisor as well. Overall, I'm in agreement with what i'm reading in these responses as well the design docs. I think this is much needed and I'm glad it is getting the focus it deserves. - Sam On Tue, Mar 19, 2019 at 10:58 AM Zander Work wrote: > This would be awesome to have, especially in a cluster environment. > Testing new scripts before we push them to production is a bit challenging > sometimes, so being able to reliably and repeatably test them in a > clustered environment would be awesome. > Another thing that wasn't mentioned in the blog post that I want to make > sure makes it into the new process is something like "broctl cron". It does > a great job of restarting processes when they crash, but also monitoring > disk usage and other potential issues for us, which is helpful in ensuring > the stability of our cluster. I see the "revive zombie nodes" piece in the > Google Doc, which covers the first bit, but other host-level > monitoring/alerting components would be nice to see in the new process. > > -Zander > > > On 3/18/19 6:55 PM, anthony kasza wrote: > > I'm excited to see this. I think it's a great design choice. This sentence > is my favorite, "We need to make it easy to test, from the command-line, > using just PCAP files, a complete cluster deployment (scaled down) as it > would work in production." > > I'm looking forward to it! > > -AK > > On Mon, Mar 18, 2019, 19:44 Jon Siwek wrote: > >> I just published some design thoughts related to a major new Zeek >> feature that's planned/upcoming: a process supervision model that may >> act as an alternative (successor) to BroControl. Find that here: >> >> https://blog.zeek.org/2019/03/beyond-brocontrol-new-process.html >> >> Feel free to use this mailing list / thread to provide feedback, thanks. >> >> - Jon >> _______________________________________________ >> Zeek mailing list >> zeek at zeek.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek >> > > _______________________________________________ > Zeek mailing listzeek at zeek.orghttp://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190319/2f08ebe4/attachment.html From jlay at slave-tothe-box.net Tue Mar 19 13:16:23 2019 From: jlay at slave-tothe-box.net (James Lay) Date: Tue, 19 Mar 2019 14:16:23 -0600 Subject: [Zeek] Request for Feedback - Zeek Process Supervision Model In-Reply-To: <97e48cb7-97e0-a78c-95fe-1aba9a46afb8@oregonstate.edu> References: <97e48cb7-97e0-a78c-95fe-1aba9a46afb8@oregonstate.edu> Message-ID: <031ea64a3698c0f0ba3e12b62ab273d3@slave-tothe-box.net> High resource usage of broctrl prevents me from running it at home...so hopefully that can be improved upon. James On 2019-03-19 09:31, Zander Work wrote: > This would be awesome to have, especially in a cluster environment. > Testing new scripts before we push them to production is a bit > challenging sometimes, so being able to reliably and repeatably test > them in a clustered environment would be awesome. > Another thing that wasn't mentioned in the blog post that I want to > make sure makes it into the new process is something like "broctl > cron". It does a great job of restarting processes when they crash, > but also monitoring disk usage and other potential issues for us, > which is helpful in ensuring the stability of our cluster. I see the > "revive zombie nodes" piece in the Google Doc, which covers the first > bit, but other host-level monitoring/alerting components would be nice > to see in the new process. > > -Zander > > On 3/18/19 6:55 PM, anthony kasza wrote: > > I'm excited to see this. I think it's a great design choice. This > sentence is my favorite, "We need to make it easy to test, from the > command-line, using just PCAP files, a complete cluster deployment > (scaled down) as it would work in production." > > I'm looking forward to it! > > -AK > > On Mon, Mar 18, 2019, 19:44 Jon Siwek wrote: > > I just published some design thoughts related to a major new Zeek > feature that's planned/upcoming: a process supervision model that > may > act as an alternative (successor) to BroControl. Find that here: > > https://blog.zeek.org/2019/03/beyond-brocontrol-new-process.html > > Feel free to use this mailing list / thread to provide feedback, > thanks. > > - Jon > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek _______________________________________________ Zeek mailing list zeek at zeek.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190319/f151e19f/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: 2019-03-19 14_14_19-2fa.png Type: image/png Size: 13284 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190319/f151e19f/attachment-0001.bin From zeolla at gmail.com Tue Mar 19 13:30:48 2019 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Tue, 19 Mar 2019 16:30:48 -0400 Subject: [Zeek] Bro scripts to write logs to Kafka fails In-Reply-To: References: Message-ID: Did you install it manually or with bro-pkg? Also, is this master or a specific release? - Jon Zeolla Zeolla at GMail.Com On Tue, Mar 19, 2019 at 12:25 PM Patrick Kelley < patrick.kelley at criticalpathsecurity.com> wrote: > Had some issues with this last week when building our ARM/ICS/IOT sensor. > > Let me see if I can run down some notes. > > On Tue, Mar 19, 2019 at 11:57 AM Arda Savran > wrote: > >> Hello folks: >> >> I just installed zeek from source on a CentOS7.5 platform. Everything >> works fine. I can confirm that the logs are getting populated with http, >> dns etc. data. >> >> I am trying to write the data to a remote Kafka topic. I believe I >> compiled everything correctly based on the following link: *https://packages.zeek.org/packages/view/7388aa77-4fb7-11e8-88be-0a645a3f3086 >> * >> >> And I am getting the expected return when I execute: >> >> * $ bro -N Apache::Kafka* >> * Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3)* >> >> Next, I edited /usr/local/bro/share/bro/site/local.bro and appended the >> following: >> >> *@load packages/metron-bro-plugin-kafka/Apache/Kafka redef >> Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG); redef Kafka::kafka_conf = >> table( ["metadata.broker.list"] = "XX.XX.XX.XX:9092" );* >> >> Unfortunately, this part doesn't work: >> >> *[root at localhost scripts]# broctl check* >> *Warning: broctl config has changed (run the broctl "deploy" command)* >> *bro scripts failed.* >> *fatal error in /usr/local/bro/share/bro/site/local.bro, line 7: can't >> find packages/metron-bro-plugin-kafka/Apache/Kafka* >> >> Has this happened to anyone before? Is there a fix for this? >> >> Thanks, >> >> >> _______________________________________________ >> Zeek mailing list >> zeek at zeek.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > > > > -- > > *Patrick Kelley, CISSP, C|EH, ITIL* > *CTO* > patrick.kelley at criticalpathsecurity.com > (o) 770-224-6482 > > *The limit to which you have accepted being comfortable is the limit to > which you have grown. Accept new challenges as an opportunity to enrich > yourself and not as a point of potential failure.* > > > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190319/24d363e3/attachment.html From michalpurzynski1 at gmail.com Wed Mar 20 00:49:20 2019 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Wed, 20 Mar 2019 00:49:20 -0700 Subject: [Zeek] Request for Feedback - Zeek Process Supervision Model In-Reply-To: References: Message-ID: Thanks a lot for doing this. Those who don't want to replace broctl shall do a triple back salto. No one? I see. I have only one request so far, still reading the proposal. Can we make sure that we support a configuration where in a stable state (after initialization has been done) there is only one worker process per core, without all those run-bro scripts and the like? 1 process per core = timer ticking disabled = trips to kernel and back minimized, no partial cache flushing, no partial TLB flushing and higher performance. On Mon, Mar 18, 2019 at 6:44 PM Jon Siwek wrote: > I just published some design thoughts related to a major new Zeek > feature that's planned/upcoming: a process supervision model that may > act as an alternative (successor) to BroControl. Find that here: > > https://blog.zeek.org/2019/03/beyond-brocontrol-new-process.html > > Feel free to use this mailing list / thread to provide feedback, thanks. > > - Jon > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190320/bcf90dd7/attachment.html From jeffrey.bencteux at ssi.gouv.fr Wed Mar 20 00:53:42 2019 From: jeffrey.bencteux at ssi.gouv.fr (Jeffrey Bencteux) Date: Wed, 20 Mar 2019 08:53:42 +0100 Subject: [Zeek] Fatal error: out of memory in new with version 2.6.1 In-Reply-To: References: Message-ID: <49ffddcf-4c02-f72a-59d0-1f80384c065a@ssi.gouv.fr> Could you try the following commands in broctl : > cleanup --all > deploy And tell us if it solve your problem ? Les donn?es ? caract?re personnel recueillies et trait?es dans le cadre de cet ?change, le sont ? seule fin d?ex?cution d?une relation professionnelle et s?op?rent dans cette seule finalit? et pour la dur?e n?cessaire ? cette relation. Si vous souhaitez faire usage de vos droits de consultation, de rectification et de suppression de vos donn?es, veuillez contacter contact.rgpd at sgdsn.gouv.fr. Si vous avez re?u ce message par erreur, nous vous remercions d?en informer l?exp?diteur et de d?truire le message. The personal data collected and processed during this exchange aims solely at completing a business relationship and is limited to the necessary duration of that relationship. If you wish to use your rights of consultation, rectification and deletion of your data, please contact: contact.rgpd at sgdsn.gouv.fr. If you have received this message in error, we thank you for informing the sender and destroying the message. From michalpurzynski1 at gmail.com Wed Mar 20 21:45:25 2019 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Wed, 20 Mar 2019 21:45:25 -0700 Subject: [Zeek] The deduplication during the script's loading Message-ID: Hey! Credit for discovery of this issue goes to Justin Azoff. During module loading, Zeek checks the inode number of each script file and silently ignores a file with the same inode as loaded previously. For example, if /usr/share/bro/base/frameworks/input/__load__.bro /usr/share/bro/base/frameworks/dpd/__load__.bro share the same inode (are hardlinked) only the first one will load and the second one will not. This breaks the workflow for me. The ClearLinux packaging mechanism, by design, hardlinks identical files (if they have identical checksum). The net result is that a correctly packaged Zeek is left in a non-functioning state after installation. Justin suggested that I can break hardlinks with rsync, like this /usr/bin/rsync -aP /usr/share/bro/ /somewhere /usr/bin/rsync -aP /somewhere /usr/share/bro/ That works - and I have a working Zeek cluster, with af_packet plugin, just by installing a single "network-monitoring-node" bundle. Do we want to do something about it? Or is this expected? Can we, maybe, make this configurable? Should I live with this workaround? -- M. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190320/3dcd0be5/attachment.html From chencheng.wang93 at gmail.com Thu Mar 21 05:15:32 2019 From: chencheng.wang93 at gmail.com (=?UTF-8?B?546L6L6w5oiQ?=) Date: Thu, 21 Mar 2019 20:15:32 +0800 Subject: [Zeek] File data loss in stream event Message-ID: Hi, I'm sorry to bother you. During using Zeek I met some problem. Could you help me? I found a bro package named credit-card-exposure(link ), and imitated the bro script in this package for detect some sensitive info. Part of the code is as follows [image: ttt.png] I printed the fields named seen_bytes and total_bytes of all the files, and found that many data of files have not entered the handler of stream event. I can also find in files.log? the seen_bytes was far less than total_bytes. what can I do to solve this problem? Yours respectfully -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190321/9e72d9ed/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: ttt.png Type: image/png Size: 55370 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190321/9e72d9ed/attachment-0001.bin From jsiwek at corelight.com Thu Mar 21 09:38:14 2019 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 21 Mar 2019 09:38:14 -0700 Subject: [Zeek] The deduplication during the script's loading In-Reply-To: References: Message-ID: On Wed, Mar 20, 2019 at 9:54 PM Micha? Purzy?ski wrote: > During module loading, Zeek checks the inode number of each script file and silently ignores a file with the same inode as loaded previously. That's intended/expected. > This breaks the workflow for me. The ClearLinux packaging mechanism, by design, hardlinks identical files (if they have identical checksum). That's interesting/unfortunate. > Do we want to do something about it? Or is this expected? Can we, maybe, make this configurable? Should I live with this workaround? IMO, it seems reasonable for us to have an intended file system structure/layout of the stuff we install, so for a given packaging system to come in and start changing that without knowing whether the packaged-software actually relies on it seems like that's their own fault/problem. However, if a patch/pull-request were made that provides the same functionality (prevent loading the same script via alternate, relative path or symlink) and does not rely on inode, then that would likely be accepted. - Jon From seth at corelight.com Fri Mar 22 07:19:25 2019 From: seth at corelight.com (Seth Hall) Date: Fri, 22 Mar 2019 10:19:25 -0400 Subject: [Zeek] Bro scripts to write logs to Kafka fails In-Reply-To: References: Message-ID: <47027D30-4E77-49F6-94A0-4BA418C288B5@corelight.com> An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190322/40c36198/attachment.html From seth at corelight.com Fri Mar 22 07:21:01 2019 From: seth at corelight.com (Seth Hall) Date: Fri, 22 Mar 2019 10:21:01 -0400 Subject: [Zeek] Request for Feedback - Zeek Process Supervision Model In-Reply-To: <031ea64a3698c0f0ba3e12b62ab273d3@slave-tothe-box.net> References: <97e48cb7-97e0-a78c-95fe-1aba9a46afb8@oregonstate.edu> <031ea64a3698c0f0ba3e12b62ab273d3@slave-tothe-box.net> Message-ID: <5C8E2F2C-3016-4AD5-94F1-A5567CC94197@corelight.com> On 19 Mar 2019, at 16:16, James Lay wrote: > High resource usage of broctrl prevents me from running it at > home...so hopefully that can be improved upon. That's planned... https://github.com/zeek/zeek/issues/264 .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From jlay at slave-tothe-box.net Fri Mar 22 07:23:58 2019 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 22 Mar 2019 08:23:58 -0600 Subject: [Zeek] Request for Feedback - Zeek Process Supervision Model In-Reply-To: <5C8E2F2C-3016-4AD5-94F1-A5567CC94197@corelight.com> References: <97e48cb7-97e0-a78c-95fe-1aba9a46afb8@oregonstate.edu> <031ea64a3698c0f0ba3e12b62ab273d3@slave-tothe-box.net> <5C8E2F2C-3016-4AD5-94F1-A5567CC94197@corelight.com> Message-ID: On 2019-03-22 08:21, Seth Hall wrote: > On 19 Mar 2019, at 16:16, James Lay wrote: > >> High resource usage of broctrl prevents me from running it at >> home...so hopefully that can be improved upon. > > That's planned... > https://github.com/zeek/zeek/issues/264 > > .Seth > > -- > Seth Hall * Corelight, Inc * www.corelight.com Excellent..looking forward to it! James From seth at corelight.com Fri Mar 22 07:30:11 2019 From: seth at corelight.com (Seth Hall) Date: Fri, 22 Mar 2019 10:30:11 -0400 Subject: [Zeek] Request for Feedback - Zeek Process Supervision Model In-Reply-To: References: <97e48cb7-97e0-a78c-95fe-1aba9a46afb8@oregonstate.edu> Message-ID: <6C42FA6F-7828-4C88-A3D1-C2488F4190B7@corelight.com> On 19 Mar 2019, at 16:06, Samuel Oehlert wrote: > Personally, I think it would be poor design to rebuild host OS > monitoring > inside the Zeek supervisor. I think that should be left up to the many > other projects specifically designed to monitor disk usage, etc. That > being > said, exposing some metrics about Zeek the application layer sounds > like it > would be a win. That being said, that might be outside the scope of a > supervisor as well. Just tell yourself that all of the processes that are being spawned and supervised are just threads and then you may think about this project differently. The fact that we will be spawning and monitoring child processes is merely an implementation detail. If we chose to offset the responsibility for starting and managing all of the process to something like systemd then it would specifically tie us to systemd (and we definitely don't want to maintain compatibility with multiple supervisors). The benefit to this approach is that from the OS perspective it's easy to run under any system supervisor and in Docker since it effectively has the same model of "run in the foreground and monitor that the process is still alive". There is an additional benefit too because we've been discussing doing an "early fork" of the supervisor process so that they all derive from the same binary (same initial memory image) which you can think of like a stem cell so the supervisor can tell it to fork again and specialize into a particular cluster process. This has the benefit of being sure that all of the processes are the same. Otherwise, if systemd restarted one of the workers and the binary on disk had changed in the intervening time it would end up being a different process (different version of Zeek?). I know it's a somewhat contrived example but it's always surprising to see the problems that will be encountered in the real world so the more potential problems we can avoid up front in the design is probably better. Another benefit to this approach is that a full cluster can be started from the command line really easily and will run in the foreground. It's been really fascinating using the prototype as it is. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From seth at corelight.com Fri Mar 22 07:33:48 2019 From: seth at corelight.com (Seth Hall) Date: Fri, 22 Mar 2019 10:33:48 -0400 Subject: [Zeek] Request for Feedback - Zeek Process Supervision Model In-Reply-To: References: Message-ID: On 20 Mar 2019, at 3:49, Micha? Purzy?ski wrote: > I have only one request so far, still reading the proposal. Can we > make > sure that we support a configuration where in a stable state (after > initialization has been done) there is only one worker process per > core, > without all those run-bro scripts and the like? All of that optimization should be possible. It wasn't included in the proposal because it should be possible to build on top of whatever this ends up looking like in the end and I don't think any of us quite know what that actual config would look like. If you have some suggestions about how the configuration should look or work or even thoughts about the mechanism it should use feel free to speak up. :) .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From dopheide at gmail.com Fri Mar 22 08:11:18 2019 From: dopheide at gmail.com (Mike Dopheide) Date: Fri, 22 Mar 2019 11:11:18 -0400 Subject: [Zeek] Request for Feedback - Zeek Process Supervision Model In-Reply-To: References: Message-ID: One thing I haven't seen specifically called out yet (perhaps I missed it) was making sure we keep the functionality for broctl commands that aren't really about managing processes. Like 'check', 'print', 'diag', etc. I could be them being part of a separate tool still, but I find them extremely valuable for debugging. -Dop On Mon, Mar 18, 2019 at 9:44 PM Jon Siwek wrote: > I just published some design thoughts related to a major new Zeek > feature that's planned/upcoming: a process supervision model that may > act as an alternative (successor) to BroControl. Find that here: > > https://blog.zeek.org/2019/03/beyond-brocontrol-new-process.html > > Feel free to use this mailing list / thread to provide feedback, thanks. > > - Jon > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190322/37d9d5ac/attachment.html From seth at corelight.com Fri Mar 22 14:09:48 2019 From: seth at corelight.com (Seth Hall) Date: Fri, 22 Mar 2019 17:09:48 -0400 Subject: [Zeek] Request for Feedback - Zeek Process Supervision Model In-Reply-To: References: Message-ID: On 22 Mar 2019, at 11:11, Mike Dopheide wrote: > One thing I haven't seen specifically called out yet (perhaps I missed > it) > was making sure we keep the functionality for broctl commands that > aren't > really about managing processes. Like 'check', 'print', 'diag', etc. > I > could be them being part of a separate tool still, but I find them > extremely valuable for debugging. We haven't specified or deeply discussed what some of the extra tooling will look like yet, but one goal we have is to simplify everything and cut out features that aren't utterly critical or can't be done better by other system tools (and obviously watching for community feedback and discussion on what stays and goes as we keep moving forward!) .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From mnmblair at hotmail.com Sat Mar 23 05:58:24 2019 From: mnmblair at hotmail.com (C Blair) Date: Sat, 23 Mar 2019 12:58:24 +0000 Subject: [Zeek] zbalance_ipc and Zeek In-Reply-To: References: , Message-ID: Hi Bill, I just wanted to follow up. I have had success after disabling hyper-threading. I have also isolated the cores for the queue consumers. Zeek and Snort now reliably process over 2Gbps simultaneously with zbalance_ipc. The traffic is a vanilla enterprise profile generated by a traffic generator. I will look into tuning the Zeek analyzers. Thank you for the assist. CB On Mar 18, 2019, at 6:41 AM, william de ping wrote: Hi Colin Have you seen any difference in traffic rate of virtual NICs between zbalance_ipc and RSS LB ? Can you send htop when bro workers are running ? Drops should mean that a worker reaches more than 100% CPU usage, if this is the case, I would dive into the world of cpuset. With this pseudo directory you can view what other processes are running on a core in addition to bro's instance so you could make the core exclusive for bro and the OS will use other available cpus for the rest of the processes. I would first resort to the difference between RSS and zbalance_ipc prior to making cpus exclusive. There are many tweaks in bro, but it really depends on the type of traffic and what you do with it. using dump-events script you get a sense of the most active events. grep these events and search for the bro scripts that registered them, it could very well be the case that no log file will be generated in that script. Such scripts could be irrelevant so you can switch off their analyzer (comment loading them in init-default.bro) Let me know how its working out for you B ________________________________ From: C Blair Sent: Sunday, March 17, 2019 6:09 PM To: bill.de.ping at gmail.com Subject: Re: [Zeek] zbalance_ipc and Zeek Hi Bill, The server is a single socket. Attached is my lstopo output. I have run zbalance_ipc with the -p option. This sends the packet per queue data to stdout and you can view what is happening in real time. The queues receive with zero drops and then Zeek drops packets equally. I have pinned zbalance_ipc to logical core 15 and the Bro workers are pinned to 1-10 logical cores. I have reserved core 0 for packet time stamping. I let CentOS schedule the remaining logical cores. I have not tried turning off Hyper-threading. Can you recommend core affinity for my given hw? Are there certain analyzers you recommend turning off and how do I accomplish that? Thanks again, CB On Mar 17, 2019, at 1:38 PM, william de ping wrote: Hi, I would check the followings : * Numa node configuration - This server should have 2 CPU sockets, if you pinned zbalance_ipc to a numa node which is not directly connected to the PCI bus hosting the NIC all traffic will go through the QPI and that could explain why it will be slower. I would check that the zbalance_ipc app is pinned to the CPU socket that is closer to the PCI NIC to avoid this * Check line rate on each virtual interface using PF_RING/userland/examples/pfcount. check on : zc:99@[0,1,...,9] after using zbalance_ipc and without zbalance_ipc using the RSS. This should give you a clue if there is a specific worker instance that is receiving significantly more traffic than others (RSS and zbalance_ipc LB might differ). It really depends on the type of traffic, but I assume that on a 2.3Ghz processor, a single bro worker can process anything between 150-400mbps. * Run a single instance of bro with local configuration and dump-events.bro script (you can redef include_args=F to get only events name without parameters). Output, sort, uniq -c it to get a clue on what event occur more often. Some analyzers might be turned off to save CPU cycles. Let me know if it helps B ________________________________ From: C Blair Sent: Sunday, March 17, 2019 8:34 AM To: bill.de.ping at gmail.com Cc: zeek at zeek.org Subject: Re: [Zeek] zbalance_ipc and Zeek Hi Bill, Thank you for the assist. Currently, Zeek cannot reliably capture more than 300Mbps with this configuration. When I remove zbalance_ipc and use RSS with num_rss_queues=lb_procs Zeek can capture up to 2Gbps. I need to use zbalance_ipc because I use a single capture interface with multiple consuming applications, i.e. Zeek and Snort. It seems obvious that a software load balancer will perform less than hardware, however, I don't see the same significant performance drop with other consuming applications like Snort. Ingress Line speed: I am using a traffic generator so I can regulate up to 10Gbps. ZEEK node.cfg [manager] type=manager host=localhost [logger] type=logger host=localhost [proxy-1] type=proxy host=localhost [worker-1] type=worker host=localhost interface=zc:99 lb_method=pf_ring lb_procs=10 pin_cpus=1,2,3,4,5,6,7,8,9,10 ZBALANCE_IPC run config zbalance_ipc -i zc:eth0 -c 99 -n 10 -m 4 -g 15 -S 0 PFRING-ZC INFO PF_RING Version : 7.5.0 (unknown) Total rings : 22 Standard (non ZC) Options Ring slots : 65536 Slot version : 17 Capture TX : No [RX only] IP Defragment : No Socket Mode : Standard Cluster Fragment Queue : 0 Cluster Fragment Discard : 0 Name : eth? Index : 40 Address : XX:XX:XX:XX:XX:XX Polling Mode : NAPI/ZC Type : Ethernet Family : ixgbe TX Queues : 1 RX Queues : 1 Num RX Slots : 32768 Num TX Slots : 32768 System Specs: Xeon D-1587 16 cores, 32 logical, 1.7 Ghz, 2.3 Ghz turbo, 20M Cache 128GB DDR4 2133Mhz 8TB SSD Intel 10GBase-T X557 ixgbe On Mar 17, 2019, at 9:08 AM, william de ping wrote: Hi Colin, Can you please clarify your deployment ? (node.cfg file, NIC type, PF_RING version, zbalance_ipc parameters and the ingress line rate ) Thanks B On Fri, Mar 15, 2019 at 12:38 AM COLIN BLAIR > wrote: Hi All, Does anyone have a success story using zbalance_ipc and Zeek. We are getting very high packet loss using zbalance_ipc. When we remove zbalance_ipc, Zeek performs well on pf_ring zero copy with RSS. Any advice is appreciated. R, CB _______________________________________________ Zeek mailing list zeek at zeek.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190323/bbd13049/attachment-0001.html From mauro.palumbo at aizoon.it Mon Mar 25 04:24:09 2019 From: mauro.palumbo at aizoon.it (Palumbo Mauro) Date: Mon, 25 Mar 2019 11:24:09 +0000 Subject: [Zeek] btest configuration Message-ID: Hi Zeek-devs, I am trying to use the btest framework to add some additional tests to Zeek to test my own configuration and scripts. I would rather keep these tests separate from the default Zeek's ones to avoid messing up. I have then installed my tests, baseline files, and btest config file in a separate folder. However, I cannot manage to configure btest to load the default zeek's scripts (init-bare.bro, etc.). I tried to modify the SCRIPTS variable in the btest.cfg file but it does not seem to help. Any suggestioni is most welcome! Thanks in advance. Mauro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190325/77356a62/attachment.html From jsiwek at corelight.com Mon Mar 25 09:16:44 2019 From: jsiwek at corelight.com (Jon Siwek) Date: Mon, 25 Mar 2019 09:16:44 -0700 Subject: [Zeek] btest configuration In-Reply-To: References: Message-ID: On Mon, Mar 25, 2019 at 4:26 AM Palumbo Mauro wrote: > However, I cannot manage to configure btest to load the default zeek?s scripts (init-bare.bro, etc.). I tried to modify the SCRIPTS variable in the btest.cfg file but it does not seem to help. Try setting BROPATH, that's the one that modifies the search paths for .bro scripts. - Jon From Zach.Rogers at oregonstate.edu Mon Mar 25 12:33:44 2019 From: Zach.Rogers at oregonstate.edu (Rogers, Zach) Date: Mon, 25 Mar 2019 19:33:44 +0000 Subject: [Zeek] tcmalloc large alloc Message-ID: <365A9752-7077-4A07-BE85-74F42D778748@contoso.com> Hello, We have been seeing some crash reports on some of our nodes, regarding a tcmalloc error. I was wondering if anyone else has seen this before and if anyone has any suggestions on what the cause might be. We are running Zeek 2.6. Here is an example stderr.log output from one of these crashes: ==== stderr.log Myricom: Local timesource listening on p2p2 tcmalloc: large alloc 1329594368 bytes == 0xc701c000 @ 0x7f72a12604ef 0x7f72a1280d56 0x9623cf 0x9623ff 0x8d8c90 0x8d1b79 0x928352 0x92895f 0x928a71 0x9242bd 0x7b5908 0x7ff59f 0x7b535d 0x7b555f 0x7b3a98 0x8c422e 0x8c3a70 0x95d49e 0x95dc16 0x8c33cc 0x8c36f9 0x8c323f 0x8c18be 0x8bef32 0x95d352 0x5c61dd 0x676f75 0x677f1c 0x648a0f 0x914669 0x648ec5 tcmalloc: large alloc 1661992960 bytes == 0x11641c000 @ 0x7f72a12604ef 0x7f72a1280dad 0x9623cf 0x9623ff 0x8d8c90 0x8d1b79 0x928352 0x92895f 0x928a71 0x9242bd 0x7b5908 0x7ff59f 0x7b535d 0x7b555f 0x7b3a98 0x8c422e 0x8c3a70 0x95d49e 0x95dc16 0x8c33cc 0x8c36f9 0x8c323f 0x8c18be 0x8bef32 0x95d352 0x5c61dd 0x676f75 0x677f1c 0x648a0f 0x914669 0x648ec5 /usr/local/bro/share/broctl/scripts/run-bro: line 110: 138751 Killed nohup "$mybro" "$@" Thanks! -- Zach Rogers Lead Security Analyst Security and Network Monitoring Oregon Research & Teaching Security Operations Center (ORTSOC) Phone: 541.737.7723 GPG Fingerprint: ECC5 03A6 7E91 17C6 50C6 8FAC D6A0 8001 2869 BD52 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190325/dfe1890a/attachment.html From jeffrey.bencteux at ssi.gouv.fr Tue Mar 26 00:42:47 2019 From: jeffrey.bencteux at ssi.gouv.fr (Jeffrey Bencteux) Date: Tue, 26 Mar 2019 08:42:47 +0100 Subject: [Zeek] tcmalloc large alloc In-Reply-To: <365A9752-7077-4A07-BE85-74F42D778748@contoso.com> References: <365A9752-7077-4A07-BE85-74F42D778748@contoso.com> Message-ID: Hi, I experienced the same, I think it might be related to corrupted temporary files created by workers. An idea I had is that the corrupted files are read with some wrong value and an allocation depend on it. The following command solved the problem for me: > cleanup --all > deploy Regards, Les donn?es ? caract?re personnel recueillies et trait?es dans le cadre de cet ?change, le sont ? seule fin d?ex?cution d?une relation professionnelle et s?op?rent dans cette seule finalit? et pour la dur?e n?cessaire ? cette relation. Si vous souhaitez faire usage de vos droits de consultation, de rectification et de suppression de vos donn?es, veuillez contacter contact.rgpd at sgdsn.gouv.fr. Si vous avez re?u ce message par erreur, nous vous remercions d?en informer l?exp?diteur et de d?truire le message. The personal data collected and processed during this exchange aims solely at completing a business relationship and is limited to the necessary duration of that relationship. If you wish to use your rights of consultation, rectification and deletion of your data, please contact: contact.rgpd at sgdsn.gouv.fr. If you have received this message in error, we thank you for informing the sender and destroying the message. From Brett.Warrick at sensato.co Tue Mar 26 14:43:10 2019 From: Brett.Warrick at sensato.co (Brett Warrick) Date: Tue, 26 Mar 2019 21:43:10 +0000 Subject: [Zeek] Projected Throughput Message-ID: I've built a 1U box (Xeon Bronze-3104 / 16 GB RAM / 10GBase-T ports with Intel X557) and I'm wondering if it's able to manage a certain level of traffic; in this case, a sustained daily rate of 10MBps, spiking at 15MBps (please note, MBps, not Mbps - I know I could easily handle a sustained 15 Mbps). I'll be analyzing traffic on a large corporate network. What do you think? Is it underpowered? Way overboard? Any best guesses about the max level of throughput it could handle? Thanks in advance for your time and your thoughts! BJW -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190326/17dd3cab/attachment.html From mfernandez at mitre.org Wed Mar 27 06:02:08 2019 From: mfernandez at mitre.org (Fernandez, Mark I) Date: Wed, 27 Mar 2019 13:02:08 +0000 Subject: [Zeek] Bro/Zeek ATT&CK-based Analytics and Reporting (BZAR), by MITRE Message-ID: All, MITRE has created a set of Bro/Zeek scripts to detect ATT&CK-like adversarial activity. The project is called BZAR - Bro/Zeek ATT&CK-based Analytics and Reporting. MITRE ATT&CK is a publicly-available, curated knowledge base for cyber adversary behavior, reflecting the various phases of the adversary lifecycle and the platforms they are known to target. The ATT&CK model includes behaviors of numerous threats groups. BZAR is a set of Bro/Zeek scripts utilizing the SMB and DCE-RPC protocol analyzers and the File Extraction Framework to detect ATT&CK-like activity, correlate certain techniques, and write to the Notice Log. BZAR is publicly released as open source, under MITRE case number 18-2489. It is available for download at the following URL: * https://github.com/mitre-attack/car/tree/master/implementations/bzar For more information on MITRE ATT&CK, visit https://attack.mitre.org. Mark I. Fernandez The MITRE Corporation mfernandez at mitre.org P.S. It does not yet support the Bro/Zeek Package Manager (this is on the todo list). -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190327/8c49f2ec/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5063 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190327/8c49f2ec/attachment-0001.bin From zeolla at gmail.com Wed Mar 27 06:37:04 2019 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Wed, 27 Mar 2019 09:37:04 -0400 Subject: [Zeek] Bro/Zeek ATT&CK-based Analytics and Reporting (BZAR), by MITRE In-Reply-To: References: Message-ID: Nice work, thanks for sharing! - Jon Zeolla Zeolla at GMail.Com On Wed, Mar 27, 2019 at 9:09 AM Fernandez, Mark I wrote: > All, > > > > MITRE has created a set of Bro/Zeek scripts to detect ATT&CK-like > adversarial activity. The project is called BZAR ? Bro/Zeek ATT&CK-based > Analytics and Reporting. > > > > MITRE ATT&CK is a publicly-available, curated knowledge base for cyber > adversary behavior, reflecting the various phases of the adversary > lifecycle and the platforms they are known to target. The ATT&CK model > includes behaviors of numerous threats groups. > > > > BZAR is a set of Bro/Zeek scripts utilizing the SMB and DCE-RPC protocol > analyzers and the File Extraction Framework to detect ATT&CK-like activity, > correlate certain techniques, and write to the Notice Log. > > > > BZAR is publicly released as open source, under MITRE case number > 18-2489. It is available for download at the following URL: > > - https://github.com/mitre-attack/car/tree/master/implementations/bzar > > > > For more information on MITRE ATT&CK, visit https://attack.mitre.org. > > > > > > *Mark I. Fernandez* > > The MITRE Corporation > > mfernandez at mitre.org > > > > P.S. It does not yet support the Bro/Zeek Package Manager (this is on the > todo list). > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190327/3f135825/attachment.html From seth at corelight.com Wed Mar 27 07:44:46 2019 From: seth at corelight.com (Seth Hall) Date: Wed, 27 Mar 2019 10:44:46 -0400 Subject: [Zeek] Bro/Zeek ATT&CK-based Analytics and Reporting (BZAR), by MITRE In-Reply-To: References: Message-ID: Seconded! This is great, thanks for sharing Mark! Are guys planning on turning this into a package and adding it to the package manager? https://bro-package-manager.readthedocs.io/en/stable/package.html .Seth On 27 Mar 2019, at 9:37, Zeolla at GMail.com wrote: > Nice work, thanks for sharing! > > - Jon Zeolla > Zeolla at GMail.Com > > > On Wed, Mar 27, 2019 at 9:09 AM Fernandez, Mark I > > wrote: > >> All, >> >> >> >> MITRE has created a set of Bro/Zeek scripts to detect ATT&CK-like >> adversarial activity. The project is called BZAR ? Bro/Zeek >> ATT&CK-based >> Analytics and Reporting. >> >> >> >> MITRE ATT&CK is a publicly-available, curated knowledge base for >> cyber >> adversary behavior, reflecting the various phases of the adversary >> lifecycle and the platforms they are known to target. The ATT&CK >> model >> includes behaviors of numerous threats groups. >> >> >> >> BZAR is a set of Bro/Zeek scripts utilizing the SMB and DCE-RPC >> protocol >> analyzers and the File Extraction Framework to detect ATT&CK-like >> activity, >> correlate certain techniques, and write to the Notice Log. >> >> >> >> BZAR is publicly released as open source, under MITRE case number >> 18-2489. It is available for download at the following URL: >> >> - >> https://github.com/mitre-attack/car/tree/master/implementations/bzar >> >> >> >> For more information on MITRE ATT&CK, visit https://attack.mitre.org. >> >> >> >> >> >> *Mark I. Fernandez* >> >> The MITRE Corporation >> >> mfernandez at mitre.org >> >> >> >> P.S. It does not yet support the Bro/Zeek Package Manager (this is >> on the >> todo list). >> _______________________________________________ >> Zeek mailing list >> zeek at zeek.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -- Seth Hall * Corelight, Inc * www.corelight.com From seth at corelight.com Wed Mar 27 07:48:43 2019 From: seth at corelight.com (Seth Hall) Date: Wed, 27 Mar 2019 10:48:43 -0400 Subject: [Zeek] tcmalloc large alloc In-Reply-To: <365A9752-7077-4A07-BE85-74F42D778748@contoso.com> References: <365A9752-7077-4A07-BE85-74F42D778748@contoso.com> Message-ID: <4D68CE36-22B1-4B57-85A7-6A0D8095A4E0@corelight.com> On 25 Mar 2019, at 15:33, Rogers, Zach wrote: > We have been seeing some crash reports on some of our nodes, regarding > a tcmalloc error. I was wondering if anyone else has seen this before > and if anyone has any suggestions on what the cause might be. We are > running Zeek 2.6. Here is an example stderr.log output from one of > these crashes: We've seen evidence before that there is a file analyzer freaking out with particular files and attempting to do these very large allocations. Unfortuantely we still don't have concrete indications about exactly what is causing the problem. It would be helpful for us if you converted those offsets into symbolic procedure names. You can do it this way (just specify the correct location for your binary)... addr2line -e /usr/local/bro/bin/bro 0x7f72a12604ef 0x7f72a1280d56 0x9623cf 0x9623ff 0x8d8c90 0x8d1b79 0x928352 0x92895f 0x928a71 0x9242bd 0x7b5908 0x7ff59f 0x7b535d 0x7b555f 0x7b3a98 0x8c422e 0x8c3a70 0x95d49e 0x95dc16 0x8c33cc 0x8c36f9 0x8c323f 0x8c18be 0x8bef32 0x95d352 0x5c61dd 0x676f75 0x677f1c 0x648a0f 0x914669 0x648ec5 .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From mfernandez at mitre.org Wed Mar 27 07:48:52 2019 From: mfernandez at mitre.org (Fernandez, Mark I) Date: Wed, 27 Mar 2019 14:48:52 +0000 Subject: [Zeek] [EXT] Re: Bro/Zeek ATT&CK-based Analytics and Reporting (BZAR), by MITRE In-Reply-To: <8619_1553697890_5C9B8C62_8619_2_3_DF23CA52-E96C-4DFA-815B-EF7DF7F3D7D3@corelight.com> References: <8619_1553697890_5C9B8C62_8619_2_3_DF23CA52-E96C-4DFA-815B-EF7DF7F3D7D3@corelight.com> Message-ID: Hi Seth, yes, that is on the todo list. Hopefully, I'll have a package for it and add it to the package-manager soon. Mark -----Original Message----- From: Seth Hall Sent: Wednesday, March 27, 2019 10:45 AM To: Fernandez, Mark I Cc: zeek at zeek.org; Zeolla at GMail.com Subject: [EXT] Re: [Zeek] Bro/Zeek ATT&CK-based Analytics and Reporting (BZAR), by MITRE Seconded! This is great, thanks for sharing Mark! Are guys planning on turning this into a package and adding it to the package manager? https://bro-package-manager.readthedocs.io/en/stable/package.html .Seth On 27 Mar 2019, at 9:37, Zeolla at GMail.com wrote: > Nice work, thanks for sharing! > > - Jon Zeolla > Zeolla at GMail.Com > > > On Wed, Mar 27, 2019 at 9:09 AM Fernandez, Mark I > > wrote: > >> All, >> >> >> >> MITRE has created a set of Bro/Zeek scripts to detect ATT&CK-like >> adversarial activity. The project is called BZAR ? Bro/Zeek >> ATT&CK-based Analytics and Reporting. >> >> >> >> MITRE ATT&CK is a publicly-available, curated knowledge base for >> cyber adversary behavior, reflecting the various phases of the >> adversary lifecycle and the platforms they are known to target. The >> ATT&CK model includes behaviors of numerous threats groups. >> >> >> >> BZAR is a set of Bro/Zeek scripts utilizing the SMB and DCE-RPC >> protocol analyzers and the File Extraction Framework to detect >> ATT&CK-like activity, correlate certain techniques, and write to the >> Notice Log. >> >> >> >> BZAR is publicly released as open source, under MITRE case number >> 18-2489. It is available for download at the following URL: >> >> - >> https://github.com/mitre-attack/car/tree/master/implementations/bzar >> >> >> >> For more information on MITRE ATT&CK, visit https://attack.mitre.org. >> >> >> >> >> >> *Mark I. Fernandez* >> >> The MITRE Corporation >> >> mfernandez at mitre.org >> >> >> >> P.S. It does not yet support the Bro/Zeek Package Manager (this is >> on the todo list). >> _______________________________________________ >> Zeek mailing list >> zeek at zeek.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -- Seth Hall * Corelight, Inc * www.corelight.com From akgraner at corelight.com Wed Mar 27 08:14:19 2019 From: akgraner at corelight.com (Amber Graner) Date: Wed, 27 Mar 2019 10:14:19 -0500 Subject: [Zeek] Bro/Zeek ATT&CK-based Analytics and Reporting (BZAR), by MITRE In-Reply-To: References: Message-ID: Mark, Thank you so much for sharing this. ~Amber On Wed, Mar 27, 2019 at 8:09 AM Fernandez, Mark I wrote: > All, > > > > MITRE has created a set of Bro/Zeek scripts to detect ATT&CK-like > adversarial activity. The project is called BZAR ? Bro/Zeek ATT&CK-based > Analytics and Reporting. > > > > MITRE ATT&CK is a publicly-available, curated knowledge base for cyber > adversary behavior, reflecting the various phases of the adversary > lifecycle and the platforms they are known to target. The ATT&CK model > includes behaviors of numerous threats groups. > > > > BZAR is a set of Bro/Zeek scripts utilizing the SMB and DCE-RPC protocol > analyzers and the File Extraction Framework to detect ATT&CK-like activity, > correlate certain techniques, and write to the Notice Log. > > > > BZAR is publicly released as open source, under MITRE case number > 18-2489. It is available for download at the following URL: > > - https://github.com/mitre-attack/car/tree/master/implementations/bzar > > > > For more information on MITRE ATT&CK, visit https://attack.mitre.org. > > > > > > *Mark I. Fernandez* > > The MITRE Corporation > > mfernandez at mitre.org > > > > P.S. It does not yet support the Bro/Zeek Package Manager (this is on the > todo list). > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -- *Amber Graner* Director of Community Corelight, Inc 828.582.9469 * Ask me about how you can participate in the Zeek (formerly Bro) community. * Remember - ZEEK AND YOU SHALL FIND!! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190327/ce40d02b/attachment-0001.html From zander.work at oregonstate.edu Wed Mar 27 08:54:28 2019 From: zander.work at oregonstate.edu (Zander Work) Date: Wed, 27 Mar 2019 08:54:28 -0700 Subject: [Zeek] tcmalloc large alloc In-Reply-To: <4D68CE36-22B1-4B57-85A7-6A0D8095A4E0@corelight.com> References: <4D68CE36-22B1-4B57-85A7-6A0D8095A4E0@corelight.com> Message-ID: <070160D7-12A1-4D7F-B28B-3D216C647DC2@getmailspring.com> I work with Zach, here are the symbols: $ addr2line -e /usr/local/bro/bin/bro 0x7f72a12604ef 0x7f72a1280d56 0x9623cf 0x9623ff 0x8d8c90 0x8d1b79 0x928352 0x92895f 0x928a71 0x9242bd 0x7b5908 0x7ff59f 0x7b535d 0x7b555f 0x7b3a98 0x8c422e 0x8c3a70 0x95d49e 0x95dc16 0x8c33cc 0x8c36f9 0x8c323f 0x8c18be 0x8bef32 0x95d352 0x5c61dd 0x676f75 0x677f1c 0x648a0f 0x914669 0x648ec5 ??:0 ??:0 /home/zeek/bro-2.6.1/aux/binpac/lib/binpac_buffer.cc:119 /home/zeek/bro-2.6.1/aux/binpac/lib/binpac_buffer.cc:529 /home/zeek/bro-2.6.1/build/src/file_analysis/analyzer/pe/pe_pac.cc:1900 /home/zeek/bro-2.6.1/src/file_analysis/analyzer/pe/PE.cc:26 /home/zeek/bro-2.6.1/src/file_analysis/File.cc:440 /home/zeek/bro-2.6.1/src/file_analysis/File.cc:481 /home/zeek/bro-2.6.1/src/file_analysis/File.cc:540 /home/zeek/bro-2.6.1/src/file_analysis/Manager.cc:167 /usr/include/c++/4.8.2/bits/basic_string.h:583 (discriminator 3) /home/zeek/bro-2.6.1/src/analyzer/protocol/mime/MIME.cc:1230 /home/zeek/bro-2.6.1/src/analyzer/protocol/http/HTTP.cc:217 /home/zeek/bro-2.6.1/src/analyzer/protocol/http/HTTP.cc:161 /home/zeek/bro-2.6.1/src/analyzer/protocol/http/HTTP.cc:947 /home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/ContentLine.cc:174 /home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/ContentLine.cc:110 /home/zeek/bro-2.6.1/src/analyzer/Analyzer.cc:245 /home/zeek/bro-2.6.1/src/analyzer/Analyzer.cc:331 /home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/TCP_Reassembler.cc:621 /home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/TCP_Reassembler.cc:375 /home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/TCP_Reassembler.cc:460 /home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/TCP_Endpoint.cc:210 /home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/TCP.cc:989 /home/zeek/bro-2.6.1/src/analyzer/Analyzer.cc:222 /home/zeek/bro-2.6.1/src/Conn.cc:271 /home/zeek/bro-2.6.1/src/Sessions.cc:769 /home/zeek/bro-2.6.1/src/IP.h:382 /home/zeek/bro-2.6.1/src/Net.cc:272 /home/zeek/bro-2.6.1/src/iosource/PktSrc.cc:263 /home/zeek/bro-2.6.1/src/Net.cc:315 The first two showing ??:0 makes sense b/c those are memory addresses. It looks like the PE analyzer might be the culprit but I'm not sure. Thanks for your help! Zander Work | Security Analyst | Oregon Research & Teaching Security Operations Center (ORTSOC) A008 Kerr Admin Bldg | Corvallis, OR 97331 | Phone: 541-737-9800 On Mar 27 2019, at 7:48 am, Seth Hall wrote: > > On 25 Mar 2019, at 15:33, Rogers, Zach wrote: > > We have been seeing some crash reports on some of our nodes, regarding > > a tcmalloc error. I was wondering if anyone else has seen this before > > and if anyone has any suggestions on what the cause might be. We are > > running Zeek 2.6. Here is an example stderr.log output from one of > > these crashes: > > > We've seen evidence before that there is a file analyzer freaking out > with particular files and attempting to do these very large allocations. > Unfortuantely we still don't have concrete indications about exactly > what is causing the problem. It would be helpful for us if you > converted those offsets into symbolic procedure names. You can do it > this way (just specify the correct location for your binary)... > > addr2line -e /usr/local/bro/bin/bro 0x7f72a12604ef 0x7f72a1280d56 > 0x9623cf 0x9623ff 0x8d8c90 0x8d1b79 0x928352 0x92895f 0x928a71 0x9242bd > 0x7b5908 0x7ff59f 0x7b535d 0x7b555f 0x7b3a98 0x8c422e 0x8c3a70 0x95d49e > 0x95dc16 0x8c33cc 0x8c36f9 0x8c323f 0x8c18be 0x8bef32 0x95d352 0x5c61dd > 0x676f75 0x677f1c 0x648a0f 0x914669 0x648ec5 > > .Seth > -- > Seth Hall * Corelight, Inc * www.corelight.com > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190327/48245cf8/attachment.html From seth at corelight.com Wed Mar 27 10:56:57 2019 From: seth at corelight.com (Seth Hall) Date: Wed, 27 Mar 2019 13:56:57 -0400 Subject: [Zeek] tcmalloc large alloc In-Reply-To: <070160D7-12A1-4D7F-B28B-3D216C647DC2@getmailspring.com> References: <4D68CE36-22B1-4B57-85A7-6A0D8095A4E0@corelight.com> <070160D7-12A1-4D7F-B28B-3D216C647DC2@getmailspring.com> Message-ID: <94D2E9E4-F4F0-4EE2-A3EC-5A9A94E7B0C2@corelight.com> On 27 Mar 2019, at 11:54, Zander Work wrote: > The first two showing ??:0 makes sense b/c those are memory addresses. > It looks like the PE analyzer might be the culprit but I'm not sure. Yep, I knew the first two would look like that. It's ASLR being applied to glibc function (which is fine and not what I was interested in anyway). It did end up showing what I expected it to. I'll look around a little bit and see if anything makes sense. Thanks! .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From Zach.Rogers at oregonstate.edu Wed Mar 27 10:58:10 2019 From: Zach.Rogers at oregonstate.edu (Rogers, Zach) Date: Wed, 27 Mar 2019 17:58:10 +0000 Subject: [Zeek] tcmalloc large alloc In-Reply-To: <94D2E9E4-F4F0-4EE2-A3EC-5A9A94E7B0C2@corelight.com> References: <4D68CE36-22B1-4B57-85A7-6A0D8095A4E0@corelight.com> <070160D7-12A1-4D7F-B28B-3D216C647DC2@getmailspring.com>, <94D2E9E4-F4F0-4EE2-A3EC-5A9A94E7B0C2@corelight.com> Message-ID: Thanks Seth, much appreciated! -- Zach Rogers Lead Security Analyst Security and Network Monitoring Oregon Research & Teaching Security Operations Center (ORTSOC) Phone: 541.737.7723 GPG Fingerprint: ECC5 03A6 7E91 17C6 50C6 8FAC D6A0 8001 2869 BD52 On Wed, Mar 27, 2019 at 10:57 AM -0700, "Seth Hall" > wrote: On 27 Mar 2019, at 11:54, Zander Work wrote: > The first two showing ??:0 makes sense b/c those are memory addresses. > It looks like the PE analyzer might be the culprit but I'm not sure. Yep, I knew the first two would look like that. It's ASLR being applied to glibc function (which is fine and not what I was interested in anyway). It did end up showing what I expected it to. I'll look around a little bit and see if anything makes sense. Thanks! .Seth -- Seth Hall * Corelight, Inc * www.corelight.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190327/cb413a47/attachment.html From gary.w.weasel2.civ at mail.mil Wed Mar 27 11:35:36 2019 From: gary.w.weasel2.civ at mail.mil (Weasel, Gary W CIV DISA RE (US)) Date: Wed, 27 Mar 2019 18:35:36 +0000 Subject: [Zeek] [Non-DoD Source] Bro/Zeek ATT&CK-based Analytics and Reporting (BZAR), by MITRE In-Reply-To: References: Message-ID: <0C34D9CA9B9DBB45B1C51871C177B4B291C073EA@UMECHPA68.easf.csd.disa.mil> Mark, Is this developed for Bro/Zeek 2.5.5? I'm getting errors when attempting to load this in Bro/Zeek 2.6.1. v/r Gary W. Weasel, Jr. -----Original Message----- From: zeek-bounces at zeek.org On Behalf Of Fernandez, Mark I Sent: Wednesday, March 27, 2019 9:02 AM To: zeek at zeek.org Subject: [Non-DoD Source] [Zeek] Bro/Zeek ATT&CK-based Analytics and Reporting (BZAR), by MITRE All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. ________________________________ All, MITRE has created a set of Bro/Zeek scripts to detect ATT&CK-like adversarial activity. The project is called BZAR - Bro/Zeek ATT&CK-based Analytics and Reporting. MITRE ATT&CK is a publicly-available, curated knowledge base for cyber adversary behavior, reflecting the various phases of the adversary lifecycle and the platforms they are known to target. The ATT&CK model includes behaviors of numerous threats groups. BZAR is a set of Bro/Zeek scripts utilizing the SMB and DCE-RPC protocol analyzers and the File Extraction Framework to detect ATT&CK-like activity, correlate certain techniques, and write to the Notice Log. BZAR is publicly released as open source, under MITRE case number 18-2489. It is available for download at the following Caution-url: * Caution-https://github.com/mitre-attack/car/tree/master/implementations/bzar < Caution-https://github.com/mitre-attack/car/tree/master/implementations/bzar > For more information on MITRE ATT&CK, visit Caution-https://attack.mitre.org. Mark I. Fernandez The MITRE Corporation mfernandez at mitre.org < Caution-mailto:mfernandez at mitre.org > P.S. It does not yet support the Bro/Zeek Package Manager (this is on the todo list). From mfernandez at mitre.org Wed Mar 27 12:03:25 2019 From: mfernandez at mitre.org (Fernandez, Mark I) Date: Wed, 27 Mar 2019 19:03:25 +0000 Subject: [Zeek] Bro/Zeek ATT&CK-based Analytics and Reporting (BZAR), by MITRE Message-ID: Hi Gary, >> Is this developed for Bro/Zeek 2.5.5? I'm getting errors when attempting to load this in Bro/Zeek 2.6.1. Yes, I used v2.5.x. What types of errors are you getting? Is it @load errors with SMB, by chance? One thing I know changed with v2.6 is that the SMB analyzer was previously disabled by default in v2.5.x and I believe it is enable by default in v2.6. In main.bro line 10: @load policy/protocols/smb. This should be backward compatible with older versions of Bro/Zeek. But if you are getting @laod SMB errors, you could try changing line 10 to this: @load base/protocols/smb. Mark -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5063 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190327/b0be2a12/attachment.bin From gary.w.weasel2.civ at mail.mil Wed Mar 27 12:52:30 2019 From: gary.w.weasel2.civ at mail.mil (Weasel, Gary W CIV DISA RE (US)) Date: Wed, 27 Mar 2019 19:52:30 +0000 Subject: [Zeek] Bro/Zeek ATT&CK-based Analytics and Reporting (BZAR), by MITRE In-Reply-To: References: Message-ID: <0C34D9CA9B9DBB45B1C51871C177B4B291C07418@UMECHPA68.easf.csd.disa.mil> I did change the SMB load directive but continued getting errors such as Error in /opt/bro/share/bro/base/gif/plugins/./Bro_DCE_RPC.events.bif.bro, line 125 and /opt/bro/share/bro/policy/bzar/./bzar_dce-rpc.bro, line 224: incompatible types (event(c:connection; fid:count; ctx_id:count; opnum:count; stub_len:count;) and event(c:connection: fid:count; opnum:count; stud_len:count)) And after trying to reconcile that Error in /opt/bro/share/bro/policy/bzar/./bzar_smb.bro, line 39: "redef" used but not previously defined (SMB::write_cmd_log) Error in /opt/bro/share/bro/base/bif/plugins/./Bro_SMB.smb2_com_create.bif.bro, line 17 and /opt/bro/share/bro/policy/bzar/./bzar_smb.bro, line 252: incompatible types (event(c:connection; hdr:SMB2::Header; request:SMB2::CreateRequest;) and event(c:connection; hdr:SMB2::Header; name:string;)) I stopped attempting to resolve on my own at that point. v/r Gary W. Weasel, Jr. | Computer Engineer Incident Response and Recovery Team, RE62 COM: 717.267.5777 -----Original Message----- From: Fernandez, Mark I Sent: Wednesday, March 27, 2019 3:03 PM To: zeek at zeek.org; Weasel, Gary W CIV DISA RE (US) Subject: [Non-DoD Source] RE: [Zeek] Bro/Zeek ATT&CK-based Analytics and Reporting (BZAR), by MITRE Hi Gary, >> Is this developed for Bro/Zeek 2.5.5? I'm getting errors when attempting to load this in Bro/Zeek 2.6.1. Yes, I used v2.5.x. What types of errors are you getting? Is it @load errors with SMB, by chance? One thing I know changed with v2.6 is that the SMB analyzer was previously disabled by default in v2.5.x and I believe it is enable by default in v2.6. In main.bro line 10: @load policy/protocols/smb. This should be backward compatible with older versions of Bro/Zeek. But if you are getting @laod SMB errors, you could try changing line 10 to this: @load base/protocols/smb. Mark From mfernandez at mitre.org Wed Mar 27 13:16:25 2019 From: mfernandez at mitre.org (Fernandez, Mark I) Date: Wed, 27 Mar 2019 20:16:25 +0000 Subject: [Zeek] [EXT] RE: Bro/Zeek ATT&CK-based Analytics and Reporting (BZAR), by MITRE In-Reply-To: <26670_1553716418_5C9BD4C2_26670_314_1_0C34D9CA9B9DBB45B1C51871C177B4B291C07418@UMECHPA68.easf.csd.disa.mil> References: <26670_1553716418_5C9BD4C2_26670_314_1_0C34D9CA9B9DBB45B1C51871C177B4B291C07418@UMECHPA68.easf.csd.disa.mil> Message-ID: Gary, I see the problem. There are rather significant changes between v2.5x and v2.6.x, as follows: DCE-RPC Event Differences: v2.5.x: event dce_rpc_response(c: connection, fid: count, opnum: count, stub_len: count); v2.6.x: event dce_rpc_response(c: connection, fid: count, ctx_id: count, opnum: count, stub_len: count); SMB Event Differences: v2.5.x: event smb2_create_request(c: connection, hdr: SMB2::Header, name: string); v2.6.x: event smb2_create_request(c: connection, hdr: SMB2::Header, request: SMB2::CreateRequest); Mark -----Original Message----- From: Weasel, Gary W CIV DISA RE (US) Sent: Wednesday, March 27, 2019 3:53 PM To: Fernandez, Mark I ; 'zeek at zeek.org' Subject: [EXT] RE: [Zeek] Bro/Zeek ATT&CK-based Analytics and Reporting (BZAR), by MITRE I did change the SMB load directive but continued getting errors such as Error in /opt/bro/share/bro/base/gif/plugins/./Bro_DCE_RPC.events.bif.bro, line 125 and /opt/bro/share/bro/policy/bzar/./bzar_dce-rpc.bro, line 224: incompatible types (event(c:connection; fid:count; ctx_id:count; opnum:count; stub_len:count;) and event(c:connection: fid:count; opnum:count; stud_len:count)) And after trying to reconcile that Error in /opt/bro/share/bro/policy/bzar/./bzar_smb.bro, line 39: "redef" used but not previously defined (SMB::write_cmd_log) Error in /opt/bro/share/bro/base/bif/plugins/./Bro_SMB.smb2_com_create.bif.bro, line 17 and /opt/bro/share/bro/policy/bzar/./bzar_smb.bro, line 252: incompatible types (event(c:connection; hdr:SMB2::Header; request:SMB2::CreateRequest;) and event(c:connection; hdr:SMB2::Header; name:string;)) I stopped attempting to resolve on my own at that point. v/r Gary W. Weasel, Jr. | Computer Engineer Incident Response and Recovery Team, RE62 COM: 717.267.5777 -----Original Message----- From: Fernandez, Mark I Sent: Wednesday, March 27, 2019 3:03 PM To: zeek at zeek.org; Weasel, Gary W CIV DISA RE (US) Subject: [Non-DoD Source] RE: [Zeek] Bro/Zeek ATT&CK-based Analytics and Reporting (BZAR), by MITRE Hi Gary, >> Is this developed for Bro/Zeek 2.5.5? I'm getting errors when >> attempting to load this in Bro/Zeek 2.6.1. Yes, I used v2.5.x. What types of errors are you getting? Is it @load errors with SMB, by chance? One thing I know changed with v2.6 is that the SMB analyzer was previously disabled by default in v2.5.x and I believe it is enable by default in v2.6. In main.bro line 10: @load policy/protocols/smb. This should be backward compatible with older versions of Bro/Zeek. But if you are getting @laod SMB errors, you could try changing line 10 to this: @load base/protocols/smb. Mark -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5063 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190327/2d40b19c/attachment-0001.bin From mfernandez at mitre.org Wed Mar 27 13:25:43 2019 From: mfernandez at mitre.org (Fernandez, Mark I) Date: Wed, 27 Mar 2019 20:25:43 +0000 Subject: [Zeek] Bro/Zeek ATT&CK-based Analytics and Reporting (BZAR), by MITRE Message-ID: Gary, >> bzar_smb.bro, line 39: "redef" used but not previously defined (SMB::write_cmd_log) Looks like "SMB::write_cmd_log" is removed from v2.6.x. Mark -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5063 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190327/590df196/attachment.bin From al.kefallonitis at gmail.com Wed Mar 27 13:47:49 2019 From: al.kefallonitis at gmail.com (Alex Kefallonitis) Date: Wed, 27 Mar 2019 22:47:49 +0200 Subject: [Zeek] Bro/Zeek ATT&CK-based Analytics and Reporting (BZAR), by MITRE In-Reply-To: References: Message-ID: Hi Mark, Thanks for sharing these gj! Just two questions: - Is the repository going to be maintain and updated e.g new attacks and categories techniques ? - Second isn't possible to detect pth attack throught *bzar_smb.bro ?* Alex Kefallonitis ???? ???, 27 ??? 2019 ???? 10:34 ?.?., ?/? Fernandez, Mark I < mfernandez at mitre.org> ??????: > Gary, > > >> bzar_smb.bro, line 39: "redef" used but not previously defined > (SMB::write_cmd_log) > > Looks like "SMB::write_cmd_log" is removed from v2.6.x. > > Mark > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190327/b4e78c5d/attachment.html From mfernandez at mitre.org Thu Mar 28 06:27:07 2019 From: mfernandez at mitre.org (Fernandez, Mark I) Date: Thu, 28 Mar 2019 13:27:07 +0000 Subject: [Zeek] : Bro/Zeek ATT&CK-based Analytics and Reporting (BZAR), by MITRE Message-ID: Alex, >> - Is the repository going to be maintain and updated >> e.g new attacks and categories techniques ? To be determined. We may do some small updates in the near future. Contributions from the Zeek community are welcome, and I believe we?ll be able to incorporate community contributions. >>- Second isn't possible to detect pth attack throught >> bzar_smb.bro ? Pass-the-Hash (pth) was not in the initial scope of the BZAR work. I think it would be great to add it, but I haven?t done a market survey to see if anyone else has already developed pth detection for Zeek. Cheers, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190328/93de5bec/attachment-0001.html From regisu85 at gmail.com Thu Mar 28 06:42:47 2019 From: regisu85 at gmail.com (=?UTF-8?Q?=C5=81ukasz_Biedka?=) Date: Thu, 28 Mar 2019 14:42:47 +0100 Subject: [Zeek] Sniffing on active/active firewalls Message-ID: Hello, I have a cluster of two active/active nodes of firewall. Each node of this firewall is in separate datacenter. Every node of this cluster have a Zeek server that is sniffing traffic from it through TAP. Each Zeek server works as a separate node - they are not clustered togheter. Problem is that I see a lot of "gaps" and percent_loss(from 30 to 70%) in capute_loss.log. broctl netstats also shows drops. Someone told me that this may be a problem with this active/active cluster and the method how it works - both nodes of this firewall receive traffic but only one of them sends responses back based on his load etc. As far as I know capture_loss and broctl netstats stats are based on data that they get from TCP sessions. So if I think correctly if Zeek server sees only part of the TCP session then he will log loss and dropped packets. Does anybody had similar problem and have some tips how to solve this? Best regards, ?ukasz -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190328/2466f2ac/attachment.html From patrick.kelley at criticalpathsecurity.com Thu Mar 28 06:55:39 2019 From: patrick.kelley at criticalpathsecurity.com (Patrick Kelley) Date: Thu, 28 Mar 2019 09:55:39 -0400 Subject: [Zeek] : Bro/Zeek ATT&CK-based Analytics and Reporting (BZAR), by MITRE In-Reply-To: References: Message-ID: We'll try to crack something out around PTH, if nothing exists already. We'll post it here when done. We have the pcaps from the lab and live engagements. Should be able to knock that out. On Thu, Mar 28, 2019 at 9:35 AM Fernandez, Mark I wrote: > Alex, > > > > >> - Is the repository going to be maintain and updated > > >> e.g new attacks and categories techniques ? > > > > To be determined. We may do some small updates in the near future. > Contributions from the Zeek community are welcome, and I believe we?ll be > able to incorporate community contributions. > > > > >>- Second isn't possible to detect pth attack throught > > >> *bzar_smb.bro ?* > > > > Pass-the-Hash (pth) was not in the initial scope of the BZAR work. I > think it would be great to add it, but I haven?t done a market survey to > see if anyone else has already developed pth detection for Zeek. > > > > Cheers, > > Mark > _______________________________________________ > Zeek mailing list > zeek at zeek.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek -- *Patrick Kelley, CISSP, C|EH, ITIL* *CTO* patrick.kelley at criticalpathsecurity.com (o) 770-224-6482 *The limit to which you have accepted being comfortable is the limit to which you have grown. Accept new challenges as an opportunity to enrich yourself and not as a point of potential failure.* -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190328/f3e56a5a/attachment.html