[Zeek] Avro Schema Availability for Zeek Logs

[Phil Rzewski]

Joseph,

Just chiming in late here with a thought...

If you're using Zeek and Avro with Kafka and Schema Registry <https://github.com/confluentinc/schema-registry> (as I suspect many are), my org put up an open source project up that might be of interest. It's called "zinger" (https://github.com/brimsec/zinger <https://github.com/brimsec/zinger>) and is closely related to "zq" (https://github.com/brimsec/zq <https://github.com/brimsec/zq>), which is another open source project we announced here recently (link <http://mailman.icsi.berkeley.edu/pipermail/zeek/2020-February/015012.html>).

What zinger does is post Avro schemas to the registry dynamically based on what it sees in the Zeek data, in addition to putting Avro-encoded Zeek events onto a Kafka queue. In this regard it's even an improvement over just having a current "stock" Zeek v3 schema, since it will dynamically reflect extra fields/logs it sees in your Zeek data that might have come about as a result of customizations, scripts, and/or plugins you've installed.

If you're in test mode with this stuff, we've got yet another repo called "kavro-demo" (https://github.com/brimsec/kavro-demo <https://github.com/brimsec/kavro-demo>) which creates a simple end-to-end setup with Kafka & Schema Registry along with test producer/consumer scripts, which might be helpful for when you're getting acquainted with zinger.

The zinger project is effectively just a prototype at this point, but we're open to working with people that are testing or starting to use this Kafka/Avro stuff in production. If you or anyone else reading this starts tinkering with it, feel free to ping me via email with questions... or better yet, come find us on our public Slack system <https://join.slack.com/t/brimsec/shared_invite/enQtOTMwMDczODg2ODgyLTk1NTdjOTQxNmI0OGYwOThiYzNlNDc5OWI5NjczZDljNDdmZGZjNGI3NTNiOWRiNzJkMzg4OTEwZWM0Y2NiYWQ> where we're also fielding inquiries about zq.

Good luck!

--
Phil Rzewski
Brim Security


> On Mar 9, 2020, at 11:12 AM, Joseph C Marion <humantargetjoe at hotmail.com> wrote:
> 
> In an effort to save myself a lot of typing, and potentially missing fields, does anyone have a current set of Avro schemas that map to the current built-in logs in Zeek version (3.0.0+)?
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org <mailto:zeek at zeek.org>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek <http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200318/e57309e8/attachment-0001.html