[Zeek-Dev] Proposal: Improve Zeek's log-writing system with batch support and better status reporting

[Robin Sommer]

On Thu, Jul 09, 2020 at 18:19 -0700, Bob Murphy wrote:

> Proposed Solution: Add a new optional API for writing a batch all at once, while
> still supporting older log writers that don't need to write batches.

That sounds good to me, a PR with the proposed API would be great.

> a. For non-batching log writers, change the "false" status to just mean
>    "There was an error writing a log record". The log writing system will then
>    report those failures to other Zeek components such as plug-ins, so they can
>    monitor a log writer's health, and make more sophisticated decisions about
>    whether a log writer can continue running or needs to be shut down.

Not quite sure what this would look like. Right now we just shut down
the thread on error, right? Can you elaborate how "report those
failures to other Zeek components" and "make more sophisticated
decisions" would look like?

Could we just change the boolean result into a tri-state (1) all good;
(2) recoverable error, and (3) fatal error? Here, (2) would mean that
the writer failed with an individual write, but remains prepared to
receive further messages for output. We could the also implicitly
treat a current "false" as (3), so that existing writers wouldn't even
notice the difference (at the source code level at least).

> b. Batching log writers will have a new API anyway, so that will let log
>    writers report more detail about write failures, including suggestions about
>    possible ways to recover.

Similar question here: how would these "suggestions" look like?

Robin

-- 
Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com