[Bro-Dev] [JIRA] (BIT-1319) topic/jsiwek/broker

[Jon Siwek (JIRA)]

    [ https://bro-tracker.atlassian.net/browse/BIT-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=19910#comment-19910 ] 

Jon Siwek commented on BIT-1319:
--------------------------------

{quote}
Question: could we skip the queue of operations for sync operations? We'd tell people: "look, you can use this, but you might get inconsistent results in exchange for a less cumbersome interface". My guess is that often, it'll be good enough. So the sync store operations would always go directly to the local cache.
{quote}

Yes, should be possible to give these operations high priority when talking to a data store clone.

Another problem at the moment is that queries against a clone store will wait to be processed until the clone has performed an initial synchronization with the master store.  Without doing that wait, it seems prone to a lot of bad results at startup.  But it also means if you do a synchronous query and a corresponding master store just doesn't exist yet, you can still end up blocking for an indeterminate amount of time, possibly indefinitely.  What should be the behavior/solution here?

* don't distinguish high priority queries from low-priority queries, but change blocking query API to require timeout parameters
* distinguish high-priority queries and they are still held up by the wait-for-initial-sync, but all methods in the blocking query API require a timeout parameter (this may make synchronous queries less affected by overload, but then maybe it opens up some starvation possibilities if one sends high-priority queries too often)
* high-priority (synchronous) queries bypass the wait-for-initial-sync, but low-priority (asynchronous) queries do not
* all queries bypass the wait-for-initial-sync, but updates/modifications wait for master to become available instead of just dropping them
* get rid of wait-for-initial-sync behavior entirely

The first option seems to have fewest drawbacks and may be least complex to implement.  And you'd get a less cumbersome API in Bro for synchronous queries: you don't have to use a when statement, but do need to specify the maximum time you want to allow the call to block.

> topic/jsiwek/broker
> -------------------
>
>                 Key: BIT-1319
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1319
>             Project: Bro Issue Tracker
>          Issue Type: New Feature
>          Components: Bro
>            Reporter: Jon Siwek
>            Assignee: Jon Siwek
>             Fix For: 2.4
>
>
> The "topic/jsiwek/broker" branch is in the bro and cmake repos to add the initial support for Broker.
> Notes/Disclaimers/Caveats:
> - Bro has a --enable-broker configure flag.
> - requires actor-framework "develop" branch.  When version 0.13 is out, I will put that as a requirement in the README and have CMake check for that.
> - no C bindings yet
> - no Python bindings yet
> - other than checking compilation that the new unit tests pass on Linux/FreeBSD/Mac, I've not done must extensive of testing, profiling, optimization etc.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)