[Zeek-Dev] Zeek Table Cluster distribution using broker ready for testing

[Johanna Amann]

Hello everyone,

If you followed last year’s Zeek Week, you might be aware that we have 
been working on a new way to more easily distribute Zeek Table content 
in a cluster setup. We now have a working prototype - and I would be 
happy for feedback if someone wants to start playing with it.

We tried to make this feature as easy to use as possible. In the case 
that you just want to distribute a table over an entire Zeek-cluster, 
you only have to add &backend=Broker::MEMORY to the table definition.

So - for example:

global table_to_share: table[string] of count &backend=Broker::MEMORY;

This will automatically synchronized the table over the entire cluster. 
In the background, a Broker store (in this case a memory-backed store) 
is created and used for the actual data synchronization. Changes to the 
table are automatically sent to the broker store and distributed over 
the cluster.

We also support persistent broker stores. At the moment you need to 
specify the path in which the database should be stored for this 
feature. Example:

redef Broker::auto_store_db_directory = "[path]";
global table_to_share: table[string] of count &backend=Broker::SQLITE;

Data that is stored in the table will be persistent across restarts of 
Zeek.

Current limitations:
  * there is no conflict resolution. Simultaneous inserts for the same 
key will probably lead to a divergent state over the cluster. This is by 
design - if you need to be absolutely sure that you do not loose any 
data, or if you want conflict resolution for multiple inserts, you will 
still have to roll your own script-level logic using events.
  * tables only can have a single index, multi-indexed tables (like 
table[string, count] of X) are not yet supported
  * tables only can have simple values. Tables that store records, 
tables, sets, vectors are not supported. The reason for this is that we 
cannot track table-changes in these cases.
  * &expire_func cannot be used simultaneously. Normal expiry should 
work correctly.
  * documentation is basically still completely missing - I will write 
it over the next days.

If you want to try this you have to compile the 
topic/johanna/table-changes branch of the Zeek repository. To check out 
this branch into a new directory, use something like:

git clone https://github.com/zeek/zeek --branch 
topic/johanna/table-changes --recursive [target-directory]

Please let me know if you have any feedback/questions/problems :)

Johanna