[Zeek-Dev] Proposal: Improve Zeek's log-writing system with batch support and better status reporting

[Seth Hall]

On 15 Jul 2020, at 20:45, Bob Murphy wrote:

>> On Jul 15, 2020, at 1:09 AM, Robin Sommer <robin at corelight.com> 
>> wrote:
>>
>> Not quite sure what this would look like. Right now we just shut down
>> the thread on error, right? Can you elaborate how "report those
>> failures to other Zeek components" and "make more sophisticated
>> decisions" would look like?
>
> Yes, right now, any writer error just shuts down the entire thread.
>
> That’s a good solution for destinations like a disk, because if a 
> write fails, something really bad has probably happened. But Seth Hall 
> pointed out that some log destinations can recover, and it’s not a 
> good solution for those.

More or less this is the same sort of thing that I'm always pushing for 
to move more functionality into scripts.  If I got an event in 
scriptland I might be able to determine what resulting action to take in 
the script and whether or not to shut down the writer or to let it keep 
going.

> For batching, I was thinking of having a way to send back a 
> std::vector of structs that would be something like this:
>
> struct failure_info {
>     uint32_t index_in_batch;
>     uint16_t failure_type;
>     uint16_t recovery_suggestion;
> };

This is almost starting to sound a bit more complicated than is worth 
it.  We may need to discuss this a bit more to figure out something 
simpler.  The immediate problem that springs to mind is that as a 
developer, I don't think I'd have any clue what failure_types and 
recovery_suggestions could be common among export destinations.

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com