[Bro-Dev] Broker data store use case and questions

[Azoff, Justin S]

> On May 11, 2018, at 10:13 AM, Jon Siwek <jsiwek at corelight.com> wrote:
> 
> 
> There's no check against the local cache to first see if the key exists 
> as going down that path leads to race conditions.

What sort of race conditions?

Right now I see a lot of events going around so it seems like there may be a bit of overhead in this area.

For example, in about a minute one node in a two node cluster sent 2104 Software::register events (so likely 4k total)
In that time, only 7 new entries were logged to software.log.

It's always good to reduce memory usage, but I think especially for things like known hosts which are generally kept at LOCAL_HOSTS
the small amount of memory used for caching already seen hosts saves more resources than would be spent sending redundant events around.
Especially if those events end up queued and buffered in ram anyway.

Things are a bit better off now in that we can use a short lived cache, since the cache doesn't need to be the actual data store anymore like the old known hosts set was.



— 
Justin Azoff