[Xorp-hackers] XRL/Thrift: using Thrift to talk to XORP processes (future)

[Bruce Simpson]

A few words on using Thrift externally to talk to XORP processes, and 
what shape that is likely to take.

 * The intention behind the current effort, is to replace XRL with the 
Thrift protocol.
 * In Thrift-speak, every XORP process advertising a service, will have 
the equivalent of a TServerSocket endpoint open for reaching that 
service and sending it RPC requests.
 * This will use the TFramedTransport with the TBinaryProtocol.

There are going to be some issues with this in the beginning which need 
to be resolved. Rather than try to resolve them myself upfront, I'm 
going to document them as we go along.

 * If you look at the tutorials shipped with Thrift, they're very simple:
  * The Calculator client example opens a single socket to a known 
Thrift service port.
  * No service discovery (RPC name resolution) is performed.

Now because XORP processes need to participate in the Finder protocol to 
discover where the components are (and in some cases, as we've seen, 
even communicate with them), it isn't a simple matter of just 
instantiating a TSocket as Calculator does.

 * It would be unreasonable to expect developers to clone all of the 
logic in XrlRouter for naming/discovery, so XrlRouter is still going to 
be needed in these situations.
 * There may be situations where developers need to implement an XRL 
target in their 3rd party code, without necessarily getting involved in 
the sync/async split.
 * It is one very constrained scenario where threading could be useful.

  * In Thrift clients, as they are currently implemented, calls are 
purely blocking (synchronous), but language native.
  * However: Thrift's client stubs always pass a 0 sequence number for 
their T_CALL messages.
   * This is absolutely fine if you only ever have one request in flight 
from the client to the server in a session, but it pretty much kills any 
possible parallelism or asynchrony.

 In XORP's integration of Thrift, we *will* be relying on the ability to 
tell requests from the same client apart.
   * We could delve into the blob and rewrite it, but we can only do 
that if we have control of the transport. We have no such control in the 
Thrift libraries as shipped.
    * Also if we're using ring buffers, that's gnarly. We would ideally 
like to hit a send() or writev() once and be done with it, to avoid 
increasing syscall overhead in situations where we need high performance.

   * As we've seen, the use of a TCP stream (where in-order delivery is 
guaranteed) forces serialization of RPC calls, but it isn't evident at 
API level.
   * This may break message-oriented transports, and it's difficult to 
mix-and-match libxipc's pseudo-async i/o mode with a Thrift client's 
sync i/o mode without tripping up over it.

  * The assumption in their programming model seems to be that threads 
will be used in a Java-like way: ie. cheap, don't share much state, and 
what they do is explicitly 'synchronized'.
  * So we'd have to be very careful about how libxipc gets entered. The 
code is not currently thread-safe.

 * So what we'll probably have to do, is support one model or the other, 
but not both at once.
  * Add a blocking API for service lookup to XrlRouter -- we can't rely 
on the EventLoop being run whilst we're in blocking Thrift APIs.
  * The 0 cseqid is only really an issue if we have more than one RPC 
in-flight on the same transport from the same client.
  * Either force clients to build their own TTransport to connect to it, 
or add a hook to XrlRouter to place the I/O streams in a blocking mode.

Alternatively, we could add an API to Thrift internally to request a new 
cseqid for each outgoing request, but that is something which needs to 
be discussed with the Thrift developers, through the Apache JIRA process.

thanks,
BMS