[Bro-Dev] persistent storage location

[G. Clark]

I would suggest that, rather than trying to attach this functionality to 
the logging framework, we wrap the input / output frameworks into a 
single unified set of script functions that handles generic K/V data 
I/O, e.g.:

ds_create(datastore_name, datastore_key, datastore_value)
ds_read(datastore_name, datastore_key)
ds_update(datastore_name, datastore_key, datastore_value)
ds_delete(datastore_name, datastore_key)

Unfortunately, logging currently only does create, and input only does 
read, so we have update and delete to somehow work into the mix.  For 
store operations, I believe those two operations are relatively 
important.  Since not every log writer is suited for CRUD, it might make 
sense to support C & R on all formats, and only support U & D on those 
formats meant to be used for store operations (e.g. redis, memcache, 
whatever).

Anyway, I'd imagine the logging framework would need support for U & D 
thrown in to really make this model work well.  We'd also need to ensure 
that ds_update / ds_delete failed in a sane way for formats on which it 
was not supported.

Just a thought.

--Gilbert

On 2/27/12 2:43 PM, Seth Hall wrote:
> The input framework is about to shake up some assumptions that we make in the logging framework now.  The assumption now is that our logs are dead and only write once then ignore.  The input framework makes it possible to use logs as a persistent storage mechanism but in my opinion much better than the old&persistent attribute because it gives us a way to provide an interface between the storage mechanism (database, log file, etc) and the way the data is stored in Bro.  It gets around the problems I've had with iterative development of Bro scripts and safety of maintaining state which led me to avoid&persistent like the plague.
>
> I don't like the model of keeping persistent state (that's how I'll refer to files/databases used with the input framework and logging framework) in the CWD in a hidden .state directory either.  Ultimately that ends up putting persistent state in the spool/ directory when using broctl which seems very wrong and unsafe to me since "spool" implies log spooling intended for eventual rotation.
>
> I propose we add another field to logging filters which indicates what "type" of log is being written.  The default type could be "LOG" which would do the normal rotation and write to the normal logging location (whatever that means for the plugin being used for writing).  Optionally we could use the "PERSISTENT_STATE" type (better names?) which would store to whatever output plugin is configured for the filter in a more appropriate location and not do the normal file rotation and other log maintenance.
>
> Ultimately being able to store persistent state with a script level defined interface on how to write to the store and read from it using the logging framework and input framework we could pull off a lot of stuff that is now either difficult or impossible.
>
> Thoughts on changes to the logging framework to fit this model better?
>
>    .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro-ids.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev