[Bro-Dev] Broker raw throughput

[Matthias Vallentin]

> You could additionally try to tweak
> caf#middleman.max_consecutive_reads, which configures how many
> new_data_msg messages a broker receives from the backend in a single
> shot. It makes sense to have the two separated, because one configures
> fairness in the scheduling and the other fairness of connection
> multiplexing.

Good to know about this tuning knob. I played with a few values, from 1
to 1K, but could not find an improvement by tweaking this value alone.
Have you already performed some measurements to find the optimal
combination of parameters?

> The 27.9% is accumulating all load down the path, isn't it?

Yeah, right, I must have confused the absolute vs. cumulative numbers in
this case. :-/

> By using many small messages, you basically maximize the messaging
> overhead.

Exactly. That is the worse-case scenario I'm trying to benchmark :-).

> Tackling the "many small messages problem" isn't going to be easy. CAF
> could try to wrap multiple messages from the network into a single
> heap-allocated storage that is then shipped to an actor as a whole,
> but this optimization would have a high complexity.

A common strategy to reduce high heap pressure involves custom
allocators, and memory pools in particular. Assuming that a single actor
produces a fixed number of message types (e.g., <= 10), one could create
one memory pool for each message type. What do you think about such a
strategy?

    Matthias