[Zeek-Dev] Log archival (Re: Zeek Supervisor: designing client and log archival) behavior

[Robin Sommer]

On Tue, Jun 30, 2020 at 01:39 -0700, Jon Siwek wrote:

> * https://github.com/zeek/zeek/wiki/Zeek-Supervisor-Log-Handling

This overall sounds good to me. Some notes & questions:

> Log Rotation

> To help bridge/replace Step (4) and (5), suggest adding a new option:
> Log::default_rotation_dir. The Log::rotation_format_func() will use
> this as part of its default return value.

Seems we should then set this to "." by default, and have the cluster
framework override it.

> The log_mgr will attempt to create necessary dirs just-in-time,
> failing to do so emits an error, but otherwise continues with rotation
> using working directory instead.

I'd extend this to any error case: if moving from current location to
Log::default_rotation_dir fails (e.g., because the latter is a on
different file system), continue with new name inside the current
working directory (and report the error).

Once moved, I suppose we would continue to optionally run a
post-processor, right? For a supervised cluster, we wouldn't use that
and suggest that people go with "zeek-archive" instead; but with
ZeekControl we'd keep the current behavior of gzipping behavior so
that we don't break any setups.

We can implement that distinction through the post-processer function:
the new default function would just do the rename according to the new
scheme, and a separate legacy function for ZeekControl spawns the
"archive-log" script.

> zeek-archiver

I like making this a standard tool, but seems like something we could
postpone doing right now and prioritize getting the Zeek-side
infrastructure in place.

> We can potentially have the Zeek Supervisor process configurable to
> auto-start and keep a zeek-archiver child alive. 

I'd say that's a job for systemd (or whatever service manager). I know
Seth disagress. :-)

> Leftover Log Rotation

> The rotation for such a leftover log file uses the metadata in the
> shadowfile to help try to go through the exact rotation that it should
> have occurred, including running the postprocessor function.

Not sure it's worth retaining the information about the post-processor
function, and it could to potentially lead to trouble if the function
changed somehow in between (or disppeared). We could instead just run
the leftovers through whatever the restarted config says to do with
files.

Do we even need any other meta data at all in the new scheme? I'm
wondering if we could simplify this all to: "If at open() time, X.log
exists, first rotate it away through the currently configured
postprocessor function". If we did that, we should probably have an
global boolean that allows to choose between that and just overwriting
existing files. The latter would be the default to retain current
command-line behavior, and the cluster framework would enable leftover
recovery.

Hmm, actually, there's a piece of meta that we'll need: the opening
timestamp, so that one can incorporate that into the name of the
rotated file (assuming we want to retain that capability). Unless we
parsed that out of the X.log itself ...

Robin

-- 
Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com