<div dir="ltr">I second David's opinion that some form of a quick-path stores or a new implementation of &persistent should be implemented.<div><br></div><div>The solution we offer right now makes people write tens of lines of cluster code, learning all details of cluster communication and dealing with a product that's pretty much impossible to debug (broker). Rinse, repeat, guess why it does not work.</div><div><br></div><div>I communicated internally we are not upgrading to 2.6 for now. I scoped the upgrade project to take me half a year (or we would have to take a detection impact).</div><div><br></div><div>While it is understood the old &synchronized attribute was an enormous hack, broker gives us the ability to do it right.</div><div>An easy to use, transparent attribute or some kind of wrapper is something we should consider to offer.</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Apr 20, 2019 at 7:05 PM David Hoelzer <<a href="mailto:dhoelzer@enclaveforensics.com">dhoelzer@enclaveforensics.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello all!<br>
<br>
TLDR:<br>
I'd like to ask that there be some thought given to the deprecation and<br>
eventual removal of the &persistent option in favor of Broker data<br>
stores. IMHO, there are uses cases where the &persistent attribute is<br>
much more attractive and lower overhead than the data store approach.<br>
<br>
Longer:<br>
As you are likely aware, &persistent is now marked deprecated and we<br>
expect it to disappear in the next version or two. The recommendation<br>
for replacement is the much more robust, SQLite backed, Broker data store.<br>
<br>
The data store solution is very elegant, though it does seem to require<br>
more fiddling than it ought to to get a data store set up. In the long<br>
term and when dealing with large amounts of data that must be persistent<br>
<br>
and synchronized across nodes, this really is a wonderful solution.<br>
<br>
That said, there seem to me to be some use cases where that is a massive<br>
<br>
hammer to swing at some very small problems. For example, we have one<br>
analysis script that is tracking successful external DNS resolutions. <br>
Specifically, it is keeping track of all IPv4 and IPv6 addresses<br>
resolved in the last 7 days (&read_expire 7 days) in a set. For all<br>
outbound connection attempts, this script generates a notice when the<br>
connection involves an external host that never appeared in a DNS answer<br>
<br>
record. This is quite handy when it comes to locating unauthorized<br>
<br>
outbound scanning, some C2 behaviors that do not rely on DNS/fast flux<br>
sorts of things, fragile configurations of enterprise services, etc. <br>
This has been performing quite well for several years now in more than<br>
one relatively decent sized networks (100,000+ hosts).<br>
<br>
For this problem (and others that I can imagine that would take a<br>
similar tack - i.e., only storing a set, vector, or other single<br>
primitive, rather than a massive record in a table or a table of<br>
tables), the &persistent is perfectly "sized."<br>
<br>
Am I alone in thinking that this feature should be retained *along side<br>
of* Broker data stores and potentially documented as recommended for<br>
simple primitive data persistence?<br>
<br>
Thanks!<br>
<br>
-- <br>
----<br>
David Hoelzer<br>
Chief of Operations<br>
Enclave Forensics, Inc.<br>
<br>
_______________________________________________<br>
Zeek mailing list<br>
<a href="mailto:zeek@zeek.org" target="_blank">zeek@zeek.org</a><br>
<a href="http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek" rel="noreferrer" target="_blank">http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek</a></blockquote></div>