[Xorp-hackers] Taking benefit of parallel processing in BGP
Bruce Simpson
bms at incunabulum.net
Wed Dec 16 05:53:25 PST 2009
Hi Simon,
Good to have you on board. I visited Belgium briefly in August, and it
was great to be able to pick up a keg of Leffe with friends without
paying a hefty import premium for it :-)
I'm in a process of winding things down for the end of year and winter
break, so I may not be as active on these lists as I usually am for at
least the next month. So I'm endeavouring to answer your questions now.
Simon van der Linden wrote:
> Actually, in my mind, the dispatcher would mainly talk "BGP" with its
> processes: it'd slice the updates by prefix and send them to the
> processes in charge as BGP updates over TCP. The BGP processes shouldn't
> need to communicate to each other. This way, we should be able to take
> benefit of multiple processors.
That is one way to achieve it. BGP itself as a protocol isn't too
arduous, the big hit is in all the state.
You can probably prototype this without worrying too much about runtime
linkage. For production refinement, linkage has to be carefully
considered, to avoid blowing up the binary footprint.
This is critically important for deployment in embedded systems. As of
bleeding edge SVN, XORP will fit into a FreeBSD package of 22MB on disk,
on x86-64, with a full feature set.
In the community branch, BGP hasn't been fully DLL-ized. There are a
few issues with C++ templates and shared libraries which you should be
aware of. This may be a good text to examine:
http://www.amazon.co.uk/Linkers-Kaufmann-Software-Engineering-Programming/dp/1558604960/ref=sr_1_1?ie=UTF8&qid=1260970906&sr=8-1-fkmr0
GCC has an '-frepo' mode. I normally hack XORP on a FreeBSD box.
Unfortunately, 'collect2' is missing in the system compiler; this can be
worked around by installing GCC as a package.
For most C++ use, GCC will put template instantiations in a 'linkonce'
segment, and GNU ld is able to chew this on its own. That's fine for
statically linked C++ binaries; shared libraries bring their own set of
challenges.
'collect2' contains all the intelligence about doing another pass over
potentially shared code, looking for template instantiations, and
allowing them to be placed / referenced at runtime, in ELF sections.
GCC itself doesn't have link-time optimization (LTO) as such. LTO is one
of LLVM's 'big wins' over GCC's SSA tree compiler design, which is
showing a bit of age now. Compiler design is a difficult space to
innovate in, and the industry has made big strides towards
object-oriented scripting languages for web apps.
The nature of Apple's business has meant that they've hit a roadblock
with GCC itself, so they have been actively funding LLVM compiler
research and development as an alternative foundation for what they do.
There have been ongoing discussions in the GCC camp about actually
taking LLVM's design on board for GCC 5.x, however they are subject to
the GNU political flavours.
> What I don't know yet though is whether
> the RIB/FEA would become a bottleneck, whenever all the BGP processes
> send their new decisions by XRL.
>
Definitely.
XORP has been designed from the ground up to be a router control plane,
capable of being distributed across multiple nodes. This manifests in
libxipc as the Finder component used to discover service endpoints, and
the XRL classes used to actually communicate with them.
In the community XORP SVN trunk, the code has been changed to use UNIX
domain sockets by default, which allows us to benefit from page sharing
on socket buffers inside the kernel, and can boost IPC performance by up
to 30% on x86 cycles alone.
This is fine for the default use case of a single node router. It
however has the side-effect that users have to explicitly ask for XORP
components to use TCP sockets, currently the only other supported XRL
transport protocol.
AMQP is worth looking into as a better way of building distributed
systems. It is actively being positioned as an alternative to similar
systems in commercial banking deployment, e.g. TIBCO SmartSockets.
The other XRL transport protocols (UDP, UNIX signals) weren't used by
any components, or weren't reliable for control plane use, so have been
removed from the shipping code.
An interesting task would be:
The ability to modify the existing clnt-gen and tgt-gen XRL IDL
generators, to wrap shared XORP components, as shared libraries, without
modifying the existing architecture.
The BGP-RIB-FEA costs could be mitigated, by exporting the existing
data structures in Boost.Interprocess shared memory.
The Finder can then be used to 'drop in' additions which can use
shared memory as a fast path.
Good luck! Feel free to holler if you have questions.
cheers,
BMS
More information about the Xorp-hackers
mailing list