[Xorp-hackers] XRL performance: UNIX domain sockets vs TCP sockets

[Bruce Simpson]

Bruce Simpson wrote:
> What's likely to give a performance boost, in the short term, is cutting 
> over to UNIX domain stream sockets for XRL. These are likely to function 
> like pipes. At least in FreeBSD, pipes are handled as a fifofs vnode, 
> which shims directly to the UNIX domain stream socket I/O path; these 
> are zero-copy inside the kernel, because the I/O is strictly local.
>   

UNIX domain sockets can be used in XRL as it stands, by passing 'env 
XORP_PF=x', without any patches.

One cosmetic change I'd make to that, is to put some sort of identifier 
into the name of the UNIX domain socket created, so we can easily tell 
where it came from.

In the pipelined XRL run (the default test mode for 
test_xrl_sender/test_xrl_receiver), you can see there is a *very* 
statistically significant difference in performance between TCP and UNIX 
domain sockets:

%%%
anglepoise:/tmp % ministat y x
x y
+ x
+------------------------------------------------------------------------------+
|        
x                                                                  +  |
|        
x                                                                  +  |
|       
xx                                                                  +  |
|       
xx                                                                  ++ |
|x      xx                                                            
+   ++++ |
|x  x  xxxx                                                  +     +  
++  +++++|
|    |__AM|                                                           
|___A_M_||
+------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  19      106000.7     109398.42     108716.92     108415.02     927.42543
+  19     127581.94     133547.01     132840.94     132133.92     1516.5559
Difference at 95.0% confidence
        23718.9 +/- 827.063
        21.8779% +/- 0.762868%
        (Student's t, pooled s = 1256.99)
%%%

This is without any change to the existing libxipc code, other than 
what's in SVN trunk right now.

It is good practice to throw away the first run, to account for page faults.

Forcing CPU affinity can help also, in eliminating additional context 
switch overhead due to CPU migration. Although schedulers are meant to 
be smart enough to figure this out on their own, sometimes a hint from 
the app helps.

You might have noticed I moved the bench_ipc* scripts into 
libxipc/tests/ today. These scripts are actually bit-rotted, and assume 
that the sender logs the meaningful statistics.

There is a small problem with test_xrl_receiver in that it assumes 
stdout is OK to output to. If we're running it and then killing it, we  
lose output. So it needs a bit of review and patching before it can be 
used again to compare runs automatically.

Also, the scripts assume they can pass XORP_PF=t and get tcp 
meaningfully. This hasn't worked for a long time, by the looks of SVN  
history. The patch is trivial.

Normally the finder is built as a library, and linked directly into the 
Router Manager (there can be only one!). This has the side-effect that 
we usually can't run it from the $BUILDDIR.

The xorp_finder binary probably also needs to get built as part of a 
check run, rather than the regular production build, so the RPATH magic 
can be performed for it when used for testing the code.

cheers,
BMS