[Bro-Dev] persistent storage location

[Seth Hall]

The input framework is about to shake up some assumptions that we make in the logging framework now.  The assumption now is that our logs are dead and only write once then ignore.  The input framework makes it possible to use logs as a persistent storage mechanism but in my opinion much better than the old &persistent attribute because it gives us a way to provide an interface between the storage mechanism (database, log file, etc) and the way the data is stored in Bro.  It gets around the problems I've had with iterative development of Bro scripts and safety of maintaining state which led me to avoid &persistent like the plague.

I don't like the model of keeping persistent state (that's how I'll refer to files/databases used with the input framework and logging framework) in the CWD in a hidden .state directory either.  Ultimately that ends up putting persistent state in the spool/ directory when using broctl which seems very wrong and unsafe to me since "spool" implies log spooling intended for eventual rotation.

I propose we add another field to logging filters which indicates what "type" of log is being written.  The default type could be "LOG" which would do the normal rotation and write to the normal logging location (whatever that means for the plugin being used for writing).  Optionally we could use the "PERSISTENT_STATE" type (better names?) which would store to whatever output plugin is configured for the filter in a more appropriate location and not do the normal file rotation and other log maintenance.  

Ultimately being able to store persistent state with a script level defined interface on how to write to the store and read from it using the logging framework and input framework we could pull off a lot of stuff that is now either difficult or impossible.

Thoughts on changes to the logging framework to fit this model better?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/