From Jawad.Rajput at hq.doe.gov Thu Jan 3 08:01:28 2019 From: Jawad.Rajput at hq.doe.gov (Rajput, Jawad (CONTR)) Date: Thu, 3 Jan 2019 16:01:28 +0000 Subject: [Zeek-Dev] Bro 2.5.4 Message-ID: Hello All, Is there a way to add Bro server hostname field into all the Bro log types? We have 5 Bro servers capturing traffic on different network nodes, we are trying to add each server/sensor hostname into all the log types so analyst can identify where the logs are coming from. v/r Jawad Rajput From vern at corelight.com Thu Jan 3 15:03:38 2019 From: vern at corelight.com (Vern Paxson) Date: Thu, 03 Jan 2019 15:03:38 -0800 Subject: [Zeek-Dev] adding credits to the schema for package metadata Message-ID: <20190103230338.981632C4EA1@rock.ICSI.Berkeley.EDU> A few months ago at (what was then called) BroCon, in the Community Session I put up a list of newly contributed packages along with my best guess as to authorship / whom to credit for the contribution. A couple of contributors came up to me afterwards to discuss adjusting how they were credited; and, more generally, the notion of adding an explicit "credits" field to the info associated with a package. This could look like: [package] credit = Originally written by A. Sacker . JSON support added by W00ter (Acme Corporation). As suggested by the example, the field would be free-form. Here, the original author decided to include their email address, and the additional contributor their company affiliation. We wouldn't have any apriori rules about who can update the "credit" field, but rather rely on the community to do that reasonably (and I imagine go back to the original contributor if a dispute arose). How does that sound? Vern From vallentin at icir.org Fri Jan 4 07:50:49 2019 From: vallentin at icir.org (Matthias Vallentin) Date: Fri, 4 Jan 2019 16:50:49 +0100 Subject: [Zeek-Dev] adding credits to the schema for package metadata In-Reply-To: <20190103230338.981632C4EA1@rock.ICSI.Berkeley.EDU> References: <20190103230338.981632C4EA1@rock.ICSI.Berkeley.EDU> Message-ID: <20190104155049.GR63703@tenzir.com> > [package] > credit = Originally written by A. Sacker . JSON > support added by W00ter (Acme Corporation). I like this idea. To support incremental growth of the credit field, we could call it "credits" and make it a list of strings: credits = A. Sacker ., JSON support added by W00ter (Acme Corporation) This may make it easier to grow the credits, simply by adding another credit string to the end. Matthias From jsiwek at corelight.com Fri Jan 4 09:02:27 2019 From: jsiwek at corelight.com (Jon Siwek) Date: Fri, 4 Jan 2019 11:02:27 -0600 Subject: [Zeek-Dev] adding credits to the schema for package metadata In-Reply-To: <20190103230338.981632C4EA1@rock.ICSI.Berkeley.EDU> References: <20190103230338.981632C4EA1@rock.ICSI.Berkeley.EDU> Message-ID: On Thu, Jan 3, 2019 at 5:04 PM Vern Paxson wrote: > [package] > credit = Originally written by A. Sacker . JSON > support added by W00ter (Acme Corporation). So first I have to question the use-case: who or what will need this information and how often? This is not information I think users of the command-line tool ever care about. E.g. I've never used other package-management tools to look into credit/author info. It's maybe something packages.zeek.org would show, but I don't see how that would be better than looking at the "contributors" stats already compiled by GitHub from author info encoded directly into git commits. It's definitely info we've needed/used ourselves, like for the BroCon presentation. Is that going to be a regular thing? What was insufficient about using implicit author info in the git commit log? It's left up to the contributor to self-report (via their git config) the name/email info as they wish to be recognized. I'm not sure how we can get any more accurate than that. Introducing manually-maintained data can only make that info less reliable. Alternatively, why would it help to have more free-form "credit" information specifically in the metadata file versus in README, CONTRIBUTORS, or AUTHORS files, which are already a common convention in open-source projects? Are there other use-cases I missed? I'm not so much opposed to standardizing on a new metadata field (people can already add a 'credit' field right now if they want, there's no code changes to bro-pkg needed since it won't use it for anything), but if it's only optional, not sure it will be adopted/maintained widely and so not solve the problem as intended. - Jon From vern at corelight.com Fri Jan 4 11:36:50 2019 From: vern at corelight.com (Vern Paxson) Date: Fri, 04 Jan 2019 11:36:50 -0800 Subject: [Zeek-Dev] adding credits to the schema for package metadata In-Reply-To: <20190104155049.GR63703@tenzir.com> (Fri, 04 Jan 2019 16:50:49 +0100). Message-ID: <20190104193650.3924B2C5268@rock.ICSI.Berkeley.EDU> > To support incremental growth of the credit field, we could call it > "credits" and make it a list of strings: Good thought - makes extracting & formatting it easier. Vern From vern at corelight.com Fri Jan 4 11:48:26 2019 From: vern at corelight.com (Vern Paxson) Date: Fri, 04 Jan 2019 11:48:26 -0800 Subject: [Zeek-Dev] adding credits to the schema for package metadata In-Reply-To: (Fri, 04 Jan 2019 11:02:27 CST). Message-ID: <20190104194826.DE0D4740DE2@rock.ICSI.Berkeley.EDU> > So first I have to question the use-case: who or what will need this > information and how often? The uses I have in mind are (1) displayed by the Web UI when browsing packages, and (2) shout-outs at our annual conference. > This is not information I think users of the command-line tool ever > care about. E.g. I've never used other package-management tools to > look into credit/author info. Agreed. > It's maybe something packages.zeek.org would show, but I don't see how > that would be better than looking at the "contributors" stats already > compiled by GitHub from author info encoded directly into git commits. The problem with those stats is (1) they're removed from the Web UI, (2) they're not in a coherent form. #2 was the issue that came up at BroCon. The git commits don't necessarily identify the author like they would want for public recognition; can be missing co-authors for instances where one author does the git end of publishing the package even though several people worked on it; and can make it unclear whether a given git contributor merits public credit. A primary goals here is to encourage contributors in terms of gaining public visibility for themselves / their group / their affiliation. I think there are enough degrees of freedom in doing so that we won't be able to simply infer the correct way to do it based on the GitHub activity. > It's definitely info we've needed/used ourselves, like for the BroCon > presentation. Is that going to be a regular thing? I would like it to be. It helps convey a sense of our active developer community. > What was > insufficient about using implicit author info in the git commit log? See the above. Those shortcomings are what led to the contributors coming up to me afterwards. > Alternatively, why would it help to have more free-form "credit" > information specifically in the metadata file versus in README, > CONTRIBUTORS, or AUTHORS files, which are already a common convention > in open-source projects? There are 3 possible advantages. (1) We know where to look for it. (2) The Web UI can display it. (3) Contributors can know to think about it up-front in terms of what ought to be displayed publicly, which could be a bit different than what might go into one of those files. > I'm not so much opposed to standardizing on a new metadata field > (people can already add a 'credit' field right now if they want, > there's no code changes to bro-pkg needed since it won't use it for > anything), but if it's only optional, not sure it will be > adopted/maintained widely and so not solve the problem as intended. Can we make it non-optional by having it be part of the contribution process, just like (I believe) the need to clarify licensing currently is? Vern From jsiwek at corelight.com Fri Jan 4 14:05:02 2019 From: jsiwek at corelight.com (Jon Siwek) Date: Fri, 4 Jan 2019 16:05:02 -0600 Subject: [Zeek-Dev] adding credits to the schema for package metadata In-Reply-To: <20190104194826.DE0D4740DE2@rock.ICSI.Berkeley.EDU> References: <20190104194826.DE0D4740DE2@rock.ICSI.Berkeley.EDU> Message-ID: On Fri, Jan 4, 2019 at 1:48 PM Vern Paxson wrote: > > It's maybe something packages.zeek.org would show, but I don't see how > > that would be better than looking at the "contributors" stats already > > compiled by GitHub from author info encoded directly into git commits. > > The problem with those stats is (1) they're removed from the Web UI, Could just link to it or else gather stats ourselves for display there if it's important enough. > (2) they're not in a coherent form. That seem like it's up to the committer to get right -- if they don't care enough to use coherent git user information, then that seems like an indication that they don't care how others recognize their contributions. > The git commits don't necessarily identify the author like they > would want for public recognition But that's completely under their own control to change however they want. > can be missing co-authors for instances There's a common git convention for co-authors they should then use: https://help.github.com/articles/creating-a-commit-with-multiple-authors/ > can make it unclear whether a given git contributor merits public credit. I think I get that point, but who gets chosen still seems arbitrary, which is why I feel the focus should be on getting the git data accurate since that can speak for itself. But I can see how it may be important to have alternate credit mechanisms in cases where historical git data is not correct and can't be changed. > > Alternatively, why would it help to have more free-form "credit" > > information specifically in the metadata file versus in README, > > CONTRIBUTORS, or AUTHORS files, which are already a common convention > > in open-source projects? > > There are 3 possible advantages. (1) We know where to look for it. > (2) The Web UI can display it. (3) Contributors can know to think about > it up-front in terms of what ought to be displayed publicly, which could > be a bit different than what might go into one of those files. (1) and (2) are just about choosing a standardized location and documenting it. That can be anywhere, but I was more just pointing out that while adding it to the package metadata does work, it's also currently only hosting data that serves a functional purpose for the command-line tool. Credit information would not be used in any functional way by the command-line tool, so that's why I was suggesting alternatives that would put this issue more in the "good conventions/practices for open-source project management" camp rather than anything specific to bro-pkg. (Thinking it's generally a good idea to limit our involvement in how people choose to maintain their own work). > Can we make it non-optional by having it be part of the contribution > process, just like (I believe) the need to clarify licensing currently is? It could, but I'd say it's not great to add requirements to the contribution process unless really needed. We're also not going to be able to enforce whether people keep that field updated properly and begs what to do about existing packages that don't promptly add a credits field to their metadata. I'm still cool with documenting an optional "credits" field for the package metadata, but just making sure, given all the caveats, that it solves the problem sufficiently? I'd probably word the docs like: "If you have particular requirements or concerns regarding how authors or contributors for your package are credited in public listings, you may explicitly provide the text that should be used to name or describe such people in this field". And then also provide your example. - Jon From mauropalumbo75 at gmail.com Thu Jan 10 00:33:24 2019 From: mauropalumbo75 at gmail.com (Mauro Palumbo) Date: Thu, 10 Jan 2019 09:33:24 +0100 Subject: [Zeek-Dev] CIFS/SMB protocol analyzer Message-ID: <9609fd58-ca3c-622e-4eca-d8df1d83b4b8@gmail.com> Hi everybody, ??? I am new to zeek/bro. For an internship which will complete a master course I have been attending, I will work with zeek and specifically with the CIFS/SMB analyzer. After looking at the documentation and the code, it seems to me that the this analyzer (as available in zeek github master branch) was written in BinPac language and only the most used protocol commands are implemented. I could possibly work on extending the current implementation of the protocol. Do you have any thoughts/suggestions about this? Is anyone already doing (or planning to do) it? Best wishes, Mauro From johanna at icir.org Thu Jan 10 06:39:49 2019 From: johanna at icir.org (Johanna Amann) Date: Thu, 10 Jan 2019 06:39:49 -0800 Subject: [Zeek-Dev] CIFS/SMB protocol analyzer In-Reply-To: <9609fd58-ca3c-622e-4eca-d8df1d83b4b8@gmail.com> References: <9609fd58-ca3c-622e-4eca-d8df1d83b4b8@gmail.com> Message-ID: <3A12967E-FC59-4631-9B2F-248761695186@icir.org> Hi Mauro, the right person to answer this question is probably seth (added directly to cc so it will pop up more prominently for him). Johanna On 10 Jan 2019, at 0:33, Mauro Palumbo wrote: > Hi everybody, > > ??? I am new to zeek/bro. For an internship which will complete a > master course I have been attending, I will work with zeek and > specifically with the CIFS/SMB analyzer. After looking at the > documentation and the code, it seems to me that the this analyzer (as > available in zeek github master branch) was written in BinPac language > and only the most used protocol commands are implemented. I could > possibly work on extending the current implementation of the protocol. > > Do you have any thoughts/suggestions about this? Is anyone already > doing > (or planning to do) it? > > Best wishes, > > Mauro > > _______________________________________________ > zeek-dev mailing list > zeek-dev at zeek.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev From jsiwek at corelight.com Thu Jan 10 17:33:30 2019 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 10 Jan 2019 19:33:30 -0600 Subject: [Zeek-Dev] docs.zeek.org Message-ID: FYI, https://docs.zeek.org is now live with the Zeek manual hosted by a Read the Docs custom domain (originally was hosted at www.zeek.org/sphinx-git). The broker and bro-pkg docs are also now available as sub-projects there: https://docs.zeek.org/projects/broker https://docs.zeek.org/projects/package-manager Let me know of any problems. - Jon From robin at corelight.com Fri Jan 11 07:04:10 2019 From: robin at corelight.com (Robin Sommer) Date: Fri, 11 Jan 2019 07:04:10 -0800 Subject: [Zeek-Dev] docs.zeek.org In-Reply-To: References: Message-ID: <20190111150410.GD73709@corelight.com> This is very cool! It's also nice that there's an "edit" button in the sidebar that leads directly to GitHub (not for Broker, though). Can we keep old links into the documentation working through redirects? Robin On Thu, Jan 10, 2019 at 19:33 -0600, Jonathan Siwek wrote: > FYI, https://docs.zeek.org is now live with the Zeek manual hosted by > a Read the Docs custom domain (originally was hosted at > www.zeek.org/sphinx-git). > > The broker and bro-pkg docs are also now available as sub-projects there: > > https://docs.zeek.org/projects/broker > https://docs.zeek.org/projects/package-manager > > Let me know of any problems. > > - Jon > _______________________________________________ > zeek-dev mailing list > zeek-dev at zeek.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev -- Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com From jsiwek at corelight.com Fri Jan 11 08:20:48 2019 From: jsiwek at corelight.com (Jon Siwek) Date: Fri, 11 Jan 2019 10:20:48 -0600 Subject: [Zeek-Dev] docs.zeek.org In-Reply-To: <20190111150410.GD73709@corelight.com> References: <20190111150410.GD73709@corelight.com> Message-ID: On Fri, Jan 11, 2019 at 9:04 AM Robin Sommer wrote: > > Can we keep old links into the documentation working through > redirects? Yes, those should already be set up, but let me know if I missed anything (note that the "release" version of the manual still lives on zeek.org as there's not yet any release that can be build on RTD). - Jon From jsiwek at corelight.com Fri Jan 11 11:26:23 2019 From: jsiwek at corelight.com (Jon Siwek) Date: Fri, 11 Jan 2019 13:26:23 -0600 Subject: [Zeek-Dev] adding credits to the schema for package metadata In-Reply-To: References: <20190104194826.DE0D4740DE2@rock.ICSI.Berkeley.EDU> Message-ID: On Fri, Jan 4, 2019 at 4:05 PM Jon Siwek wrote: > I'm still cool with documenting an optional "credits" field for the > package metadata I went ahead and documented it [1] as I was updating other things there. Feel free to PR any language tweaks. - Jon [1] https://docs.zeek.org/projects/package-manager/en/stable/package.html#credits-field From robin at corelight.com Fri Jan 11 15:03:32 2019 From: robin at corelight.com (Robin Sommer) Date: Fri, 11 Jan 2019 15:03:32 -0800 Subject: [Zeek-Dev] docs.zeek.org In-Reply-To: References: <20190111150410.GD73709@corelight.com> Message-ID: <20190111230332.GH69335@corelight.com> On Fri, Jan 11, 2019 at 10:20 -0600, Jonathan Siwek wrote: > Yes, those should already be set up, but let me know if I missed > anything (note that the "release" version of the manual still lives on > zeek.org as there's not yet any release that can be build on RTD). Would it be worth aiming to do that update with the next 2.6.x patch release? Would be nice to get the modern look for the release version, too. Robin -- Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com From jsiwek at corelight.com Fri Jan 11 17:08:04 2019 From: jsiwek at corelight.com (Jon Siwek) Date: Fri, 11 Jan 2019 19:08:04 -0600 Subject: [Zeek-Dev] docs.zeek.org In-Reply-To: <20190111230332.GH69335@corelight.com> References: <20190111150410.GD73709@corelight.com> <20190111230332.GH69335@corelight.com> Message-ID: On Fri, Jan 11, 2019 at 5:03 PM Robin Sommer wrote: > Would it be worth aiming to do that update with the next 2.6.x patch > release? Would be nice to get the modern look for the release version, > too. Not sure how worth it -- feel a bit odd/unmotivated preparing a pure-cosmetic feature for a patch release which I usually think more strictly in the Semantic Versioning sense as being bug fixes only, not new features. Maybe a good segue to a separate discussion on improving release/version planning so there's more appropriate and frequent times to release smallish features like this? - Jon From seth at corelight.com Wed Jan 16 13:41:19 2019 From: seth at corelight.com (Seth Hall) Date: Wed, 16 Jan 2019 16:41:19 -0500 Subject: [Zeek-Dev] CIFS/SMB protocol analyzer In-Reply-To: <9609fd58-ca3c-622e-4eca-d8df1d83b4b8@gmail.com> References: <9609fd58-ca3c-622e-4eca-d8df1d83b4b8@gmail.com> Message-ID: Hi Mauro! Sorry for the late response, I know we've been communicating offlist, but I thought I'd respond here so that others could see too. I'm not actively working on the SMB analyzer and I don't know of anyone else actively working on it so it's unlikely that you will have any interence with merging your code. I can't wait to find out more about what you're interested in implementing and where you'd like to take the analyzer! Thanks, .Seth On 10 Jan 2019, at 3:33, Mauro Palumbo wrote: > Hi everybody, > > ??? I am new to zeek/bro. For an internship which will complete a > master course I have been attending, I will work with zeek and > specifically with the CIFS/SMB analyzer. After looking at the > documentation and the code, it seems to me that the this analyzer (as > available in zeek github master branch) was written in BinPac language > and only the most used protocol commands are implemented. I could > possibly work on extending the current implementation of the protocol. > > Do you have any thoughts/suggestions about this? Is anyone already > doing > (or planning to do) it? > > Best wishes, > > Mauro > > _______________________________________________ > zeek-dev mailing list > zeek-dev at zeek.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev -- Seth Hall * Corelight, Inc * www.corelight.com From jsiwek at corelight.com Thu Jan 17 13:39:05 2019 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 17 Jan 2019 15:39:05 -0600 Subject: [Zeek-Dev] docs.zeek.org In-Reply-To: References: <20190111150410.GD73709@corelight.com> Message-ID: On Fri, Jan 11, 2019 at 10:20 AM Jon Siwek wrote: > (note that the "release" version of the manual still lives on > zeek.org as there's not yet any release that can be build on RTD). Both release and master versions are on RTD (docs.zeek.org) now and build from a new zeek-docs repo on GitHub. The 2.6.1 version was manually imported with a few tweaks to get it working, so don't think we have to consider backporting anything into a 2.6.x -- probably those won't change the generated docs anyway so we can get away with just bumping a new git tag. - Jon From robin at corelight.com Fri Jan 18 09:22:17 2019 From: robin at corelight.com (Robin Sommer) Date: Fri, 18 Jan 2019 09:22:17 -0800 Subject: [Zeek-Dev] docs.zeek.org In-Reply-To: References: <20190111150410.GD73709@corelight.com> Message-ID: <20190118172217.GB47795@corelight.com> On Thu, Jan 17, 2019 at 15:39 -0600, Jonathan Siwek wrote: > Both release and master versions are on RTD (docs.zeek.org) now and > build from a new zeek-docs repo on GitHub. Very cool! Robin -- Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com From seth at corelight.com Thu Jan 24 10:10:50 2019 From: seth at corelight.com (Seth Hall) Date: Thu, 24 Jan 2019 13:10:50 -0500 Subject: [Zeek-Dev] Bro 2.5.4 In-Reply-To: References: Message-ID: On 3 Jan 2019, at 11:01, Rajput, Jawad (CONTR) wrote: > Is there a way to add Bro server hostname field into all the Bro log > types? We have 5 Bro servers capturing traffic on different network > nodes, we are trying to add each server/sensor hostname into all the > log types so analyst can identify where the logs are coming from. Yes! We added a log extension mecahnism a while ago. Here's a snippet you could start from... ```bro option my_server_name = ""; type MyLogExtension: record { server_name: string &log; }; function add_my_log_extension(path: string): MyLogExtension { return MyLogExtension($server_name = my_server_name); } redef Log::default_ext_func = add_my_log_extension; ``` .Seth -- Seth Hall * Corelight, Inc * www.corelight.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190124/53a9aecd/attachment.html From Jawad.Rajput at hq.doe.gov Sat Jan 26 08:23:44 2019 From: Jawad.Rajput at hq.doe.gov (Rajput, Jawad (CONTR)) Date: Sat, 26 Jan 2019 16:23:44 +0000 Subject: [Zeek-Dev] [EXTERNAL] Re: Bro 2.5.4 In-Reply-To: References: Message-ID: Thanks a lot Seth, it worked out. v/r Jawad From: Seth Hall [mailto:seth at corelight.com] Sent: Thursday, January 24, 2019 1:11 PM To: Rajput, Jawad (CONTR) Cc: bro-dev at bro.org Subject: [EXTERNAL] Re: [Zeek-Dev] Bro 2.5.4 On 3 Jan 2019, at 11:01, Rajput, Jawad (CONTR) wrote: Is there a way to add Bro server hostname field into all the Bro log types? We have 5 Bro servers capturing traffic on different network nodes, we are trying to add each server/sensor hostname into all the log types so analyst can identify where the logs are coming from. Yes! We added a log extension mecahnism a while ago. Here's a snippet you could start from... option my_server_name = ""; type MyLogExtension: record { server_name: string &log; }; function add_my_log_extension(path: string): MyLogExtension { return MyLogExtension($server_name = my_server_name); } redef Log::default_ext_func = add_my_log_extension; .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ******************************************************************** This message does not originate from a known Department of Energy email system. Use caution if this message contains attachments, links or requests for information. ******************************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190126/f73d70c3/attachment.html From jsiwek at corelight.com Thu Jan 31 15:21:44 2019 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 31 Jan 2019 17:21:44 -0600 Subject: [Zeek-Dev] support for event handlers using a subset of parameters Message-ID: Just wanted to draw more attention/feedback to a proof-of-concept patch I just made: https://github.com/zeek/zeek/pull/262 Basically lets us do this: global my_event: event(a: count, b: string); event my_event(b: string) { print "my_event", b; } Which is good because: * user doesn't care about parameter 'a', so they shouldn't have to list it * it makes it easier for to deprecate/change event parameters I didn't see any drawbacks to this change, but maybe I missed something, let me know. - Jon From vern at corelight.com Thu Jan 31 16:29:07 2019 From: vern at corelight.com (Vern Paxson) Date: Thu, 31 Jan 2019 16:29:07 -0800 Subject: [Zeek-Dev] support for event handlers using a subset of parameters In-Reply-To: (Thu, 31 Jan 2019 17:21:44 CST). Message-ID: <20190201002907.0699B74B264@rock.ICSI.Berkeley.EDU> > global my_event: event(a: count, b: string); > > event my_event(b: string) > { print "my_event", b; } > > Which is good because: > > * user doesn't care about parameter 'a', so they shouldn't have to list it > * it makes it easier for to deprecate/change event parameters This seems like a pretty niche pair of benefits. Is there a compelling use-case that's motivating this change? One thing I initially wondered was whether this was going to tie our hands in the future if we want to introduce C++-style overloading. However, it looks like you've implemented this based on matching the names in the declaration rather than the types, so that should be okay. Vern