[Bro-Dev] [JIRA] (BIT-1319) topic/jsiwek/broker

Jon Siwek (JIRA) jira at bro-tracker.atlassian.net
Wed Mar 4 11:29:00 PST 2015

    [ https://bro-tracker.atlassian.net/browse/BIT-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=19910#comment-19910 ] 

Jon Siwek commented on BIT-1319:

Question: could we skip the queue of operations for sync operations? We'd tell people: "look, you can use this, but you might get inconsistent results in exchange for a less cumbersome interface". My guess is that often, it'll be good enough. So the sync store operations would always go directly to the local cache.

Yes, should be possible to give these operations high priority when talking to a data store clone.

Another problem at the moment is that queries against a clone store will wait to be processed until the clone has performed an initial synchronization with the master store.  Without doing that wait, it seems prone to a lot of bad results at startup.  But it also means if you do a synchronous query and a corresponding master store just doesn't exist yet, you can still end up blocking for an indeterminate amount of time, possibly indefinitely.  What should be the behavior/solution here?

* don't distinguish high priority queries from low-priority queries, but change blocking query API to require timeout parameters
* distinguish high-priority queries and they are still held up by the wait-for-initial-sync, but all methods in the blocking query API require a timeout parameter (this may make synchronous queries less affected by overload, but then maybe it opens up some starvation possibilities if one sends high-priority queries too often)
* high-priority (synchronous) queries bypass the wait-for-initial-sync, but low-priority (asynchronous) queries do not
* all queries bypass the wait-for-initial-sync, but updates/modifications wait for master to become available instead of just dropping them
* get rid of wait-for-initial-sync behavior entirely

The first option seems to have fewest drawbacks and may be least complex to implement.  And you'd get a less cumbersome API in Bro for synchronous queries: you don't have to use a when statement, but do need to specify the maximum time you want to allow the call to block.

> topic/jsiwek/broker
> -------------------
>                 Key: BIT-1319
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1319
>             Project: Bro Issue Tracker
>          Issue Type: New Feature
>          Components: Bro
>            Reporter: Jon Siwek
>            Assignee: Jon Siwek
>             Fix For: 2.4
> The "topic/jsiwek/broker" branch is in the bro and cmake repos to add the initial support for Broker.
> Notes/Disclaimers/Caveats:
> - Bro has a --enable-broker configure flag.
> - requires actor-framework "develop" branch.  When version 0.13 is out, I will put that as a requirement in the README and have CMake check for that.
> - no C bindings yet
> - no Python bindings yet
> - other than checking compilation that the new unit tests pass on Linux/FreeBSD/Mac, I've not done must extensive of testing, profiling, optimization etc.

This message was sent by Atlassian JIRA

More information about the bro-dev mailing list