[Xorp-hackers] Xrl/Thrift: Technical implementation details

[Bruce Simpson]

So far, the following has happened in the development branch:

 * I have pretty much read all of libxipc and thrift, and digested most 
of it. That has been most time consuming to date.
  * I reverse engineered what Thrift was generating for typical RPC 
calls, and compared it with XRL's behaviour.
  * Once it was clear there was a fit, work proceeded further.

 * libxipc has been cut down to the bare minimum API surface required 
for routing processes to link against.
  * This involves cutting out a lot of what will now be dead wood in 
libxipc.
  * We have in fact more than one RPC protocol in libxipc, if you 
include the textual Finder protocol, and more than two if you include 
more than just XrlPFSTCP.
  * XRL keeps a lot of state for in-flight RPC calls around, as C++ 
objects, in an intermediate representation, because of this Finder 
protocol split.
  * A lot of the protocol machinery in libxipc exists to deal with that 
conceptual split.

 * The Router Manager has been cut out of the dev branch whilst the work 
is ongoing.
   * Reason for this is, as we've seen in other recent threads on this 
list, that it makes use of XRL in very different ways from the rest of 
the tree, and uses a number of libxipc APIs which the rest of the tree 
never touches.

 * In xrl/scripts, clnt-gen and tgt-gen change to generate the 
equivalent Thrift binary blobs.
  * They use Thrift's TProtocol interface, instead of building an Xrl 
(with help from XrlArgs / XrlAtom / XrlAtomList etc.) as they currently 
do in mainline XORP.
  * Thrift methods can potentially return structs.
  * XRL methods don't return structs; instead, return arguments are 
listed after the '->' token in the *.xif IDL file.
  * The approach used is to turn the XRL return arguments into a Thrift 
struct, which can be accessed from Thrift using a compatible 'struct {}' 
declaration in a *.thrift IDL file.
  * This is transparent inside the XRL/Thrift port, and only exposed in 
thrift-gen, see below.

In XRL clients, method dispatch is still pseudo-asynchronous:-
  * The trickiest part is dealing with asynchronous method call resolution.
    * In XRL, this is per-method, and pushes a lot of state around.
    * In Thrift, this can happen per-service.

  * In the old XRL world, the XrlFooClient::send_foo() stubs build up an 
XRL, which is then sent using the XrlSender interface.
   * This send may happen over a variety of destination transports, 
because of the Finder protocol vs XrlPF* split.
   * The XRL send MAY be deferred if the method can't be resolved (it's 
pending Finder target resolution).

  * In the new XRL/Thrift world, these stubs marshal the body of a 
binary T_CALL message directly into a binary buffer, and then call into 
XrlRouter (via the XrlSender interface) to ship it off to the correct 
destination.
   * Because RPC endpoints can come and go, the FinderClient needs to 
track the endpoints based on what it learns from the Finder.
   * Because we're tied to TCP (for now) as a network transport, method 
dispatch is not fully asynchronous (nor do want it to be, for now).

In XRL targets, method dispatch is still synchronous, and doesn't change 
[yet]:-
   * Request comes in, libxipc parses it from the server's RPC endpoint, 
and will ship it off to the Thrifted XRL target stubs (handle_*()).
   * In the Thrift case, instead of flipping class Xrl instances around, 
the handle*() callbacks read directly from a buffered Thrift binary blob 
(T_CALL message) we just read from the transport.
   * No plans to break this up further at this time. See below re 
scalability.
   * If the process's method handler returns XrlCmdError::OKAY(), then 
we marshall the result out using a Thrift T_REPLY.
   * Otherwise, XrlCmdErrors will get translated into Thrift T_EXCEPTION 
reply messages by the XRL target.
    * This translation doesn't happen in the stubs themselves, rather, 
we preserve the existing XRL APIs and deal with it in libxipc.

A few words on 3rd party process interop:
 * XRL, at wire level, is conceptually a subset of Thrift's wire-level 
protocol.
 * A thrift-gen translator has been written which takes a XORP *.xif XRL 
IDL file, and generates a compatible *.thrift IDL file.
  * Whilst Thrift has language-level exception support, we don't use 
this here. It requires an additional struct in the T_REPLY. It made more 
sense to keep things simple, as this also bloats the translated Thrift 
service definitions.
  * Whilst thrift-gen is not useful immediately (see other message re 
using Thrift to talk to XORP), it'll be needed for writing Thrifted code 
to talk to XORP components directly later on.
  * It made sense to do this first, to get familiar with the Xif parser 
in Python, and get more of a feel for the Thrift syntax, as well as 
identifying the conceptual overlap with XRL.

Some performance issues exist with XRL/Thrift which require modifying 
Thrift itself:
 * Thrift's TProtocol binary read/write methods take only std::string as 
arguments.
 * They also do some legwork to avoid allocating intermediate storage on 
the stack, but instead from the C++ runtime heap.
 * There is no clean way to cast a vector<uint8_t>, or similar scoped 
array types, to std::string, without introducing an intermediate copy on 
the stack.
   * It isn't something which needs to be resolved immediately, as the 
only affected XRL method calls are those which ship packet payloads.
  * All other XIF native types should cast cleanly into their new Thrift 
representation, without significant intermediate copies on the stack; 
apart from libxorp's Mac, which is just a 6 byte binary quantity.
  * [Note re scalability: unmarshalling need only happen at the point of 
dispatch, assuming buffering mechanisms are in place.]
 * The likely resolution is that I'll send the Thrift developers a patch 
for TProtocol to implement a (void *, size_t) overload for the 
TProtocol::readBinary() and TProtocol::writeBinary() methods.

On the subject of invocation through the Finder, something which isn't 
possible/relevant in Thrift:
 * The current libxipc supports the notion of tunneled XRL method calls.
 * These calls are routed directly to the Finder, which is then 
responsible for dispatching them to the target.
 * In Thrifted libxipc, this changes; the Finder just acts as a naming 
service.
 * This is an essential mechanism for the Router Manager, which invokes 
XRL methods based on the same textual representation of them used by the 
current Finder protocol.
 * The plan is to ditch the XRLdb (or keep it around purely for testing 
purposes); refactor XrlAction and XorpClient in the Router Manager to 
generate Thrift method calls, based on the textual XRL method call 
description; and execute those calls using the same lookup mechanism as 
we will use in the new libxipc.

thanks,
BMS