[Xorp-hackers] FEA performance improvements: only 'pull' active interfaces.
greearb at candelatech.com
Fri Mar 14 01:03:38 PDT 2008
Pavlin Radoslavov wrote:
> Ben Greear <greearb at candelatech.com> wrote:
>> As previously mentioned, I notice that fea pays attention to all
>> interfaces, not just those
>> it's configured to use.
>> To help improve scalability, especially in a virtualized environement, I
>> am attempting
>> the following:
>> 1) Only have the _pulled_config pull information for devices stored in
>> the _local_config
>> tree. This means asking netlink for specific if-index values instead of
>> the entire tree.
> The problem with this solution is that it will work only for Netlink
> on Linux. The (majoriyty of the) other mechanisms for obtaining the
> network interface information (getifaddrs(3), ioctl(2), syssctl(3),
> etc) don't allow the granularity for asking only the information for
> a specific interface.
> In addition, if you have lots of configured interfaces, it might
> actually be much more expensive to ask separately for each of them
> (i.e., to use a system call per interface) instead of using a single
> system call to obtain the information for all interfaces.
Yes, this may only be useful for my scenario where I'm using a small
number of interfaces per xorp instance, with large numbers of total interfaces.
Only linux can virtualize routing tables, as far as I know, so this performance
gain is only really important on Linux.
>> 2) The netlink observer will ignore anything not in the _local_config,
>> and will remove
>> interfaces from _local_config if it observes them unregistering from the
>> 3) When adding an interface (though XRL), the ifconfig object will add
>> it to _local_config,
>> tell the pulled_config to pull it from the system, and if found, will
>> save it in the _original_config
>> as well in case we want to roll back to the original system state. Once
>> added, nothing is removed
>> from the _original_config.
> You could keep track in _pulled_config of only those interfaces that
> are configured in XORP (i.e., those in _local_config).
> However, this could add more complexity to all the interface-related
> machinery (IfTree, IfConfig, etc).
> Also, given the extra search/complexity you need to do to maintain
> that state it becomes questionable how much performance gain there
> will be.
> Last but not least, don't forget that if there is any performance
> gain it will show only in the case where there are lots of
> interfaces in the system, but the XORP instance is configured with a
> very small number of that interfaces.
My assumption is that anything not in local_config can be ignored
in the pull_config logic, and anything that searches for a device not
in local config should not expect to find it. There will be a bit of
special cases for adding new devices to the local_config, but not much else
So far, it seems the code complexity will not be great, it's mostly lots of
small boring API changes. But, it's not working yet, so it may get more
>> 4) There is an XRL method to configure all interfaces from the system.
>> I am hoping this isn't
>> actually needed and can be removed, as it would require reading the
>> entire set of interfaces. I
>> can (re)add code to support this if needed, but maybe it isn't really
>> useful and could be removed?
> Currently this method is used by some of the FEA test programs.
Ok, I'll (re)add it...it's not difficult since that is the default behaviour
of the existing pull_config logic.
>> I am only implementing the optimizations for the netlink related
>> portions. The remainder of the iftree-get/set logic
>> will use the current method of reading all interfaces regardless of
>> local config.
>> I believe this will go a long way towards helping fea scale to 1000+
>> interfaces, but don't have performance
>> numbers or working code quite yet.
> As usual, I don't want to commit to any optimizations before I see
> numbers that justify the extra complexity :)
> Independent from the above, as you have noticed already the FEA
> calls pull_config() several times so the alternative solution would
> be to try to reduce that number. There is more than one
> pull_config() for both technical and historical reasons (see below).
My problem is that each xorp is only interested in ~15 interfaces, but
I have 30+ xorp instances (and would like 100+). This means that decreasing pull_config calls by
a small number will not be nearly as important making the method scale
linearly with *configured* devices as opposed to existing devices.
I did look at the code you describe, and didn't immediately get any ideas for
being able to remove calls. But, I'll look at all of that again when I get
the logic to pull only configured devices functioning...
Ben Greear <greearb at candelatech.com>
Candela Technologies Inc http://www.candelatech.com
More information about the Xorp-hackers