[Zeek] Deprecation of &persistent

[Michał Purzyński]

I second David's opinion that some form of a quick-path stores or a new
implementation of &persistent should be implemented.

The solution we offer right now makes people write tens of lines of cluster
code, learning all details of cluster communication and dealing with a
product that's pretty much impossible to debug (broker). Rinse, repeat,
guess why it does not work.

I communicated internally we are not upgrading to 2.6 for now. I scoped the
upgrade project to take me half a year (or we would have to take a
detection impact).

While it is understood the old &synchronized attribute was an enormous
hack, broker gives us the ability to do it right.
An easy to use, transparent attribute or some kind of wrapper is something
we should consider to offer.



On Sat, Apr 20, 2019 at 7:05 PM David Hoelzer <dhoelzer at enclaveforensics.com>
wrote:

> Hello all!
>
> TLDR:
> I'd like to ask that there be some thought given to the deprecation and
> eventual removal of the &persistent option in favor of Broker data
> stores.  IMHO, there are uses cases where the &persistent attribute is
> much more attractive and lower overhead than the data store approach.
>
> Longer:
> As you are likely aware, &persistent is now marked deprecated and we
> expect it to disappear in the next version or two.  The recommendation
> for replacement is the much more robust, SQLite backed, Broker data store.
>
> The data store solution is very elegant, though it does seem to require
> more fiddling than it ought to to get a data store set up.  In the long
> term and when dealing with large amounts of data that must be persistent
>
> and synchronized across nodes, this really is a wonderful solution.
>
> That said, there seem to me to be some use cases where that is a massive
>
> hammer to swing at some very small problems.  For example, we have one
> analysis script that is tracking successful external DNS resolutions.
> Specifically, it is keeping track of all IPv4 and IPv6 addresses
> resolved in the last 7 days (&read_expire 7 days) in a set.  For all
> outbound connection attempts, this script generates a notice when the
> connection involves an external host that never appeared in a DNS answer
>
> record.  This is quite handy when it comes to locating unauthorized
>
> outbound scanning, some C2 behaviors that do not rely on DNS/fast flux
> sorts of things, fragile configurations of enterprise services, etc.
> This has been performing quite well for several years now in more than
> one relatively decent sized networks (100,000+ hosts).
>
> For this problem (and others that I can imagine that would take a
> similar tack - i.e., only storing a set, vector, or other single
> primitive, rather than a massive record in a table or a table of
> tables), the &persistent is perfectly "sized."
>
> Am I alone in thinking that this feature should be retained *along side
> of* Broker data stores and potentially documented as recommended for
> simple primitive data persistence?
>
> Thanks!
>
> --
> ----
> David Hoelzer
> Chief of Operations
> Enclave Forensics, Inc.
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190420/e7a8d292/attachment-0001.html