[Bro-Dev] New logging architecture
gc355804 at ohio.edu
Mon Jul 4 15:40:00 PDT 2011
On 7/4/2011 4:45 PM, Jonathan Siwek wrote:
>> I figure the LogMgr would need to be able to generate (at a minimum)
>> the following types of messages:
>> *) EnableStream
>> *) DisableStream
>> *) StreamInit
>> *) StreamFinish
>> *) RotateLog
>> *) LogMessage
> Does that mean the LogMgr would send messages to Stream threads even if they're disabled? Couldn't the LogMgr itself keep track of the Stream enabled/disabled state and just not send messages to disabled Streams?
Good point. Yes :)
Thinking about it now, I'm not sure it would benefit the client to know
whether or not a stream was disabled; assuming the client has
subscribed, it simply won't get any messages to log from the publisher.
Logic dictates it should also be up to the client to determine when to
So, let's drop everything but LogMessage, and enable an additional
heartbeat message that gets broadcast once every 15 seconds (in the
absence of a LogMessage). If the subscriber misses two consecutive
heartbeats, it's safe to assume the publisher is no longer broadcasting,
and the subscriber shuts down. The stream is initialized when the first
LogMessage is received, and is torn down when two heartbeats are missed
OR the subscriber is terminated locally (e.g. CTRL-C).
>> anything that spoke the correct message format could act as a logger for Bro.
> That's a neat idea if you're hinting at something like creating Streams such that the LogMgr binds its 0MQ socket to a tcp port, allowing 0MQ sockets connected from other hosts (not necessarily running a full Bro process) to receive logs.
Yup. That's the idea.
In the longer-term, I've been thinking about what would happen if
someone tried to turn Bro into a set of well-defined, focused utilities
all built around a shared protocol. It's a fun thought-experiment :)
> But I think some 0MQ-specific caveats of that might be:
> * the same PUSH/PULL pattern may not work well because it's possible for the pusher to block on a call to zmq_send(). Maybe PUB/SUB is better when working w/ Streams that aren't inproc?
Another good point. Is there a good reason to keep inproc streams PUSH
/ PULL if inter-process communications use PUB / SUB?
> * 0MQ doesn't (currently) provide a good framework for securing messages or being exposed to the public Internet. (I'm about to send another mail about an experiment I did with the former topic, but basically I just ended up coming to the same conclusions that they/we already talked about).
I wasn't really anticipating internet deployment; instead, I was
thinking of limited use-cases within a Bro cluster / local environment.
I think the new message scheme would be a lot more robust than what I
proposed initially, but we'd still want to implement some kind of
message signing (at a minimum) if we wanted to support logging across an
untrusted network; if we didn't, it'd be pretty simple to flood the
remote logger with a stream of invalid events.
More information about the bro-dev