[Xorp-hackers] Xrl/Thrift: Technical implementation details
Bruce Simpson
bms at incunabulum.net
Mon Nov 2 11:01:43 PST 2009
So far, the following has happened in the development branch:
* I have pretty much read all of libxipc and thrift, and digested most
of it. That has been most time consuming to date.
* I reverse engineered what Thrift was generating for typical RPC
calls, and compared it with XRL's behaviour.
* Once it was clear there was a fit, work proceeded further.
* libxipc has been cut down to the bare minimum API surface required
for routing processes to link against.
* This involves cutting out a lot of what will now be dead wood in
libxipc.
* We have in fact more than one RPC protocol in libxipc, if you
include the textual Finder protocol, and more than two if you include
more than just XrlPFSTCP.
* XRL keeps a lot of state for in-flight RPC calls around, as C++
objects, in an intermediate representation, because of this Finder
protocol split.
* A lot of the protocol machinery in libxipc exists to deal with that
conceptual split.
* The Router Manager has been cut out of the dev branch whilst the work
is ongoing.
* Reason for this is, as we've seen in other recent threads on this
list, that it makes use of XRL in very different ways from the rest of
the tree, and uses a number of libxipc APIs which the rest of the tree
never touches.
* In xrl/scripts, clnt-gen and tgt-gen change to generate the
equivalent Thrift binary blobs.
* They use Thrift's TProtocol interface, instead of building an Xrl
(with help from XrlArgs / XrlAtom / XrlAtomList etc.) as they currently
do in mainline XORP.
* Thrift methods can potentially return structs.
* XRL methods don't return structs; instead, return arguments are
listed after the '->' token in the *.xif IDL file.
* The approach used is to turn the XRL return arguments into a Thrift
struct, which can be accessed from Thrift using a compatible 'struct {}'
declaration in a *.thrift IDL file.
* This is transparent inside the XRL/Thrift port, and only exposed in
thrift-gen, see below.
In XRL clients, method dispatch is still pseudo-asynchronous:-
* The trickiest part is dealing with asynchronous method call resolution.
* In XRL, this is per-method, and pushes a lot of state around.
* In Thrift, this can happen per-service.
* In the old XRL world, the XrlFooClient::send_foo() stubs build up an
XRL, which is then sent using the XrlSender interface.
* This send may happen over a variety of destination transports,
because of the Finder protocol vs XrlPF* split.
* The XRL send MAY be deferred if the method can't be resolved (it's
pending Finder target resolution).
* In the new XRL/Thrift world, these stubs marshal the body of a
binary T_CALL message directly into a binary buffer, and then call into
XrlRouter (via the XrlSender interface) to ship it off to the correct
destination.
* Because RPC endpoints can come and go, the FinderClient needs to
track the endpoints based on what it learns from the Finder.
* Because we're tied to TCP (for now) as a network transport, method
dispatch is not fully asynchronous (nor do want it to be, for now).
In XRL targets, method dispatch is still synchronous, and doesn't change
[yet]:-
* Request comes in, libxipc parses it from the server's RPC endpoint,
and will ship it off to the Thrifted XRL target stubs (handle_*()).
* In the Thrift case, instead of flipping class Xrl instances around,
the handle*() callbacks read directly from a buffered Thrift binary blob
(T_CALL message) we just read from the transport.
* No plans to break this up further at this time. See below re
scalability.
* If the process's method handler returns XrlCmdError::OKAY(), then
we marshall the result out using a Thrift T_REPLY.
* Otherwise, XrlCmdErrors will get translated into Thrift T_EXCEPTION
reply messages by the XRL target.
* This translation doesn't happen in the stubs themselves, rather,
we preserve the existing XRL APIs and deal with it in libxipc.
A few words on 3rd party process interop:
* XRL, at wire level, is conceptually a subset of Thrift's wire-level
protocol.
* A thrift-gen translator has been written which takes a XORP *.xif XRL
IDL file, and generates a compatible *.thrift IDL file.
* Whilst Thrift has language-level exception support, we don't use
this here. It requires an additional struct in the T_REPLY. It made more
sense to keep things simple, as this also bloats the translated Thrift
service definitions.
* Whilst thrift-gen is not useful immediately (see other message re
using Thrift to talk to XORP), it'll be needed for writing Thrifted code
to talk to XORP components directly later on.
* It made sense to do this first, to get familiar with the Xif parser
in Python, and get more of a feel for the Thrift syntax, as well as
identifying the conceptual overlap with XRL.
Some performance issues exist with XRL/Thrift which require modifying
Thrift itself:
* Thrift's TProtocol binary read/write methods take only std::string as
arguments.
* They also do some legwork to avoid allocating intermediate storage on
the stack, but instead from the C++ runtime heap.
* There is no clean way to cast a vector<uint8_t>, or similar scoped
array types, to std::string, without introducing an intermediate copy on
the stack.
* It isn't something which needs to be resolved immediately, as the
only affected XRL method calls are those which ship packet payloads.
* All other XIF native types should cast cleanly into their new Thrift
representation, without significant intermediate copies on the stack;
apart from libxorp's Mac, which is just a 6 byte binary quantity.
* [Note re scalability: unmarshalling need only happen at the point of
dispatch, assuming buffering mechanisms are in place.]
* The likely resolution is that I'll send the Thrift developers a patch
for TProtocol to implement a (void *, size_t) overload for the
TProtocol::readBinary() and TProtocol::writeBinary() methods.
On the subject of invocation through the Finder, something which isn't
possible/relevant in Thrift:
* The current libxipc supports the notion of tunneled XRL method calls.
* These calls are routed directly to the Finder, which is then
responsible for dispatching them to the target.
* In Thrifted libxipc, this changes; the Finder just acts as a naming
service.
* This is an essential mechanism for the Router Manager, which invokes
XRL methods based on the same textual representation of them used by the
current Finder protocol.
* The plan is to ditch the XRLdb (or keep it around purely for testing
purposes); refactor XrlAction and XorpClient in the Router Manager to
generate Thrift method calls, based on the textual XRL method call
description; and execute those calls using the same lookup mechanism as
we will use in the new libxipc.
thanks,
BMS
More information about the Xorp-hackers
mailing list