[Zeek-Dev] Proposal: Make Zeek's debug logging thread-safe

[Bob Murphy]

Hi Johanna,

I wasn’t aware of that call, but it also wouldn’t have done what I needed.

If I understand the code correctly, each MsgThread has a FIFO queue that it pushes messages onto. Later on, the main thread occasionally runs a loop where it handles all the queued messages from the first MsgThread, then all the queued messages from the second MsgThread, etc.

The development I was doing sometimes required me to examine the debug messages from different threads in the chronological order they were generated. But if I understand it correctly, the threading framework’s logging doesn’t maintain that ordering.

Also, that work sometimes generated a LOT of debug messages - thousands or millions of lines of them - when only a tiny fraction of them were interesting. To cut down on the garbage, I used the DebugLogger class’s member functions to selectively enable and disable individual streams when particular conditions occurred. However, those member functions have immediate effect, and because the threading framework’s Debug member function emits log lines after a delay, it seems likely I would have not seen debug output I wanted to see, and seeing debug output I didn’t want to see.

Best regards,
Bob

> On Jul 9, 2020, at 7:16 PM, Johanna Amann <johanna at corelight.com> wrote:
> 
> On 9 Jul 2020, at 16:57, Bob Murphy wrote:
> 
>> Right now, if you try to use Zeek's debug logging facilities in DebugLogger.h concurrently from multiple threads, the contents of debug.log can get mixed up and look like like "word salad".
> 
> Is there a reason why you didn’t just use the Debug call of the threading framework (which goes through the message queues then ends up in debug.log)?
> 
> Johanna