[Bro-Dev] New logging architecture

[Gilbert Clark]

On 7/4/2011 4:45 PM, Jonathan Siwek wrote:
>> I figure the LogMgr would need to be able to generate (at a minimum)
>> the following types of messages:
>>
>> *) EnableStream
>> *) DisableStream
>> *) StreamInit
>> *) StreamFinish
>> *) RotateLog
>> *) LogMessage
> Does that mean the LogMgr would send messages to Stream threads even if they're disabled?  Couldn't the LogMgr itself keep track of the Stream enabled/disabled state and just not send messages to disabled Streams?
>

Good point.  Yes :)

Thinking about it now, I'm not sure it would benefit the client to know 
whether or not a stream was disabled; assuming the client has 
subscribed, it simply won't get any messages to log from the publisher.  
Logic dictates it should also be up to the client to determine when to 
rotate logs.

So, let's drop everything but LogMessage, and enable an additional 
heartbeat message that gets broadcast once every 15 seconds (in the 
absence of a LogMessage).  If the subscriber misses two consecutive 
heartbeats, it's safe to assume the publisher is no longer broadcasting, 
and the subscriber shuts down.  The stream is initialized when the first 
LogMessage is received, and is torn down when two heartbeats are missed 
OR the subscriber is terminated locally (e.g. CTRL-C).

>> anything that spoke the correct message format could act as a logger for Bro.
> That's a neat idea if you're hinting at something like creating Streams such that the LogMgr binds its 0MQ socket to a tcp port, allowing 0MQ sockets connected from other hosts (not necessarily running a full Bro process) to receive logs.

Yup.  That's the idea.

In the longer-term, I've been thinking about what would happen if 
someone tried to turn Bro into a set of well-defined, focused utilities 
all built around a shared protocol.  It's a fun thought-experiment :)

> But I think some 0MQ-specific caveats of that might be:
>
> * the same PUSH/PULL pattern may not work well because it's possible for the pusher to block on a call to zmq_send().  Maybe PUB/SUB is better when working w/ Streams that aren't inproc?

Another good point.  Is there a good reason to keep inproc streams PUSH 
/ PULL if inter-process communications use PUB / SUB?

> * 0MQ doesn't (currently) provide a good framework for securing messages[1] or being exposed to the public Internet[2].  (I'm about to send another mail about an experiment I did with the former topic, but basically I just ended up coming to the same conclusions that they/we already talked about).

I wasn't really anticipating internet deployment; instead, I was 
thinking of limited use-cases within a Bro cluster / local environment.

I think the new message scheme would be a lot more robust than what I 
proposed initially, but we'd still want to implement some kind of 
message signing (at a minimum) if we wanted to support logging across an 
untrusted network; if we didn't, it'd be pretty simple to flood the 
remote logger with a stream of invalid events.

--Gilbert