[Bro-Dev] Fw: Broker raw throughput

[Clark, Gilbert]

Hi:

Forwarding reply to the bro-dev list: original was mistakenly posted elsewhere (sorry about that).  Leaving original message content inline for context.
________________________________________
From: Matthias Vallentin <matthias at vallentin.net> on behalf of Matthias Vallentin <vallentin at icir.org>
Sent: Thursday, March 10, 2016 1:46 PM
To: Clark, Gilbert
Subject: Re: [Bro-Dev] Broker raw throughput

> > Sorry if this is a stupid question, but what are the performance
> > requirements for broker, exactly?
>
>We have too little experience to tell what we need 

Right, this was my question.  The less that broker's requirements are documented and understood, the more difficult it becomes to evaluate whether or not broker will fit with an intended use-case, I think.  To me, the performance numbers themselves don't matter as much as managing expectations does: should I *expect* to be able to pass all of my events through broker?

>and where we hit bottlenecks. 
> 
> This is why we compare a worst-case scenario across
>multiple libraries and see where we find low-hanging fruits for
>optimization potential.

Got it.  I think that's what confused me ...

>> That's *good enough* for most things, but it's also still quite
>> possible to do better: I've built DSP applications on top of DPDK that
>> do fine with ~14 Mpps, which is itself pretty slow when compared to
>> results reported by e.g. [2].
>
>It's not about going as fast as possible. We're looking to achieve good
>performance in the common case, *without* reducing a high level of
>abstraction.

... because some of those messaging libraries operate at different levels of abstraction than others, which is going to drive the performance to some extent ...

>
>> Realistically, depending on what broker is intended to support, maybe
>> 200k messages / second is fine:
>
>Agreed. At this point, this rate is certainly fine for our
>worst-case-single-int-per-message benchmarks.
>
>> TL/DR: I'm of the opinion that optimization is fun, but I also would
>> feel kind of bad watching CAF go too far down a (very scary) rabbit
>> hole just to support one (albeit very large and rather cool)
>> application ...
>
>I don't think we want to do that either, this must be a
>misunderstanding. The optimizations we've been looking at are
>application-independent. 

... so it looks like it was indeed a misunderstanding on my part.  Sorry about that.

Trying to express things a slightly different way, I was concerned that the different numbers from the different libraries were being interpreted as an apples-to-apples comparison.  Modifying CAF to achieve the same results as e.g. 0mq would, at some point and in some way, eventually require modifying CAF to be more like 0mq.  I don't think that would be good, because 0mq and CAF aren't (and shouldn't be, in my humble opinion) the same thing.

> We have looked at a very specific workload to
>bound worst-case performance. I'm very happy with the recent
>improvements that will ship with CAF 0.15.

Definitely.  Performance improvements are always good :)

Cheers,
Gilbert