[Xorp-hackers] Taking benefit of parallel processing in BGP

Bruce Simpson bms at incunabulum.net
Wed Dec 16 05:53:25 PST 2009


Hi Simon,

Good to have you on board. I visited Belgium briefly in August, and it 
was great to be able to pick up a keg of Leffe with friends without 
paying a hefty import premium for it :-)

I'm in a process of winding things down for the end of year and winter 
break, so I may not be as active on these lists as I usually am for at 
least the next month.  So I'm endeavouring to answer your questions now.

Simon van der Linden wrote:
> Actually, in my mind, the dispatcher would mainly talk "BGP" with its 
> processes: it'd slice the updates by prefix and send them to the 
> processes in charge as BGP updates over TCP. The BGP processes shouldn't 
> need to communicate to each other. This way, we should be able to take 
> benefit of multiple processors.

That is one way to achieve it. BGP itself as a protocol isn't too 
arduous, the big hit is in all the state.

You can probably prototype this without worrying too much about runtime 
linkage. For production refinement, linkage has to be carefully 
considered, to avoid blowing up the binary footprint.

This is critically important for deployment in embedded systems. As of 
bleeding edge SVN, XORP will fit into a FreeBSD package of 22MB on disk, 
on x86-64, with a full feature set.

In the community branch, BGP hasn't been fully DLL-ized.  There are a 
few issues with C++ templates and shared libraries which you should be 
aware of. This may be a good text to examine:
http://www.amazon.co.uk/Linkers-Kaufmann-Software-Engineering-Programming/dp/1558604960/ref=sr_1_1?ie=UTF8&qid=1260970906&sr=8-1-fkmr0

GCC has an '-frepo' mode. I normally hack XORP on a FreeBSD box. 
Unfortunately, 'collect2' is missing in the system compiler; this can be 
worked around by installing GCC as a package.

For most C++ use, GCC will put template instantiations in a 'linkonce' 
segment, and GNU ld is able to chew this on its own. That's fine for 
statically linked C++ binaries; shared libraries bring their own set of 
challenges.

'collect2' contains all the intelligence about doing another pass over 
potentially shared code, looking for template instantiations, and 
allowing them to be placed / referenced at runtime, in ELF sections.

GCC itself doesn't have link-time optimization (LTO) as such. LTO is one 
of LLVM's 'big wins' over GCC's SSA tree compiler design, which is 
showing a bit of age now. Compiler design is a difficult space to 
innovate in, and the industry has made big strides towards 
object-oriented scripting languages for web apps.

The nature of Apple's business has meant that they've hit a roadblock 
with GCC itself, so they have been actively funding LLVM compiler 
research and development as an alternative foundation for what they do.

There have been ongoing discussions in the GCC camp about actually 
taking LLVM's design on board for GCC 5.x, however they are subject to 
the GNU political flavours.

>  What I don't know yet though is whether 
> the RIB/FEA would become a bottleneck, whenever all the BGP processes 
> send their new decisions by XRL.
>   

Definitely.

XORP has been designed from the ground up to be a router control plane, 
capable of being distributed across multiple nodes. This manifests in 
libxipc as the Finder component used to discover service endpoints, and 
the XRL classes used to actually communicate with them.

In the community XORP SVN trunk, the code has been changed to use UNIX 
domain sockets by default, which allows us to benefit from page sharing 
on socket buffers inside the kernel, and can boost IPC performance by up 
to 30% on x86 cycles alone.

This is fine for the default use case of a single node router. It 
however has the side-effect that users have to explicitly ask for XORP 
components to use TCP sockets, currently the only other supported XRL 
transport protocol.

AMQP is worth looking into as a better way of building distributed 
systems. It is actively being positioned as an alternative to similar 
systems in commercial banking deployment, e.g. TIBCO SmartSockets.

The other XRL transport protocols (UDP, UNIX signals) weren't used by 
any components, or weren't reliable for control plane use, so have been 
removed from the shipping code.

An interesting task would be:
    The ability to modify the existing clnt-gen and tgt-gen XRL IDL 
generators, to wrap shared XORP components, as shared libraries, without 
modifying the existing architecture.
    The BGP-RIB-FEA costs could be mitigated, by exporting the existing 
data structures in Boost.Interprocess shared memory.
    The Finder can then be used to 'drop in' additions which can use 
shared memory as a fast path.

Good luck! Feel free to holler if you have questions.

cheers,
BMS



More information about the Xorp-hackers mailing list