[Xorp-hackers] FEA performance improvements: only 'pull' active interfaces.

Pavlin Radoslavov pavlin at ICSI.Berkeley.EDU
Thu Mar 13 14:58:39 PDT 2008


Ben Greear <greearb at candelatech.com> wrote:

> As previously mentioned, I notice that fea pays attention to all 
> interfaces, not just those
> it's configured to use.
> 
> To help improve scalability, especially in a virtualized environement, I 
> am attempting
> the following:
> 
> 1)  Only have the _pulled_config pull information for devices stored in 
> the _local_config
> tree.  This means asking netlink for specific if-index values instead of 
> the entire tree.

The problem with this solution is that it will work only for Netlink
on Linux. The (majoriyty of the) other mechanisms for obtaining the
network interface information (getifaddrs(3), ioctl(2), syssctl(3),
etc) don't allow the granularity for asking only the information for
a specific interface.

In addition, if you have lots of configured interfaces, it might
actually be much more expensive to ask separately for each of them
(i.e., to use a system call per interface) instead of using a single
system call to obtain the information for all interfaces.


> 2)  The netlink observer will ignore anything not in the _local_config, 
> and will remove
> interfaces from _local_config if it observes them unregistering from the 
> system.
> 
> 3)  When adding an interface (though XRL), the ifconfig object will add 
> it to _local_config,
> tell the pulled_config to pull it from the system, and if found, will 
> save it in the _original_config
> as well in case we want to roll back to the original system state.  Once 
> added, nothing is removed
> from the _original_config.

You could keep track in _pulled_config of only those interfaces that
are configured in XORP (i.e., those in _local_config).
However, this could add more complexity to all the interface-related
machinery (IfTree, IfConfig, etc).
Also, given the extra search/complexity you need to do to maintain
that state it becomes questionable how much performance gain there
will be.

Last but not least, don't forget that if there is any performance
gain it will show only in the case where there are lots of
interfaces in the system, but the XORP instance is configured with a
very small number of that interfaces.

> 4)  There is an XRL method to configure all interfaces from the system.  
> I am hoping this isn't
> actually needed and can be removed, as it would require reading the 
> entire set of interfaces.  I
> can (re)add code to support this if needed, but maybe it isn't really 
> useful and could be removed?

Currently this method is used by some of the FEA test programs.

> I am only implementing the optimizations for the netlink related 
> portions.  The remainder of the iftree-get/set logic
> will use the current method of reading all interfaces regardless of 
> local config.
> 
> I believe this will go a long way towards helping fea scale to 1000+ 
> interfaces, but don't have performance
> numbers or working code quite yet.

As usual, I don't want to commit to any optimizations before I see
numbers that justify the extra complexity :)


Independent from the above, as you have noticed already the FEA
calls pull_config() several times so the alternative solution would
be to try to reduce that number. There is more than one
pull_config() for both technical and historical reasons (see below).

On startup the FEA calls pull_config() to get the original interface
configuration (in case it needs to restore it on shutdown). This
pull_config() would have to stay.

However, each IfConfig::commit_transaction() triggers probably four
pull_config():

 * Once inside IfConfigSet::push_config() right after the
  interfaces/vifs have been created into the system (e.g., vlans).
  This pull_config() is needed to read information such as interface
  index that needs to be used right after that.
  Getting rid of this might add extra complexity to the IfConfigSet
  plugin API which should be avoided.
  This pull_config() might have to stay (or at least it shouldn't be
  the first one to optimize for).

 * In the beginning of IfConfig::commit_transaction() there is
   pull_config() to guarantee that we start the interface
   reconfiguration with what is currently in the system.
   This might be removed for systems that have a mechanism/plugin to
   observe the asynchronous changes to the system interfaces (e.g.,
   BSD and Linux), but would have to stay for systems that don't
   have it (e.g., Windows).

* Inside method IfConfig::push_config() there is another
  pull_config(). This one is probably redundant when push_config()
  is called from IfConfig::commit_transaction() and a good candidate
  for elimination. Of course, before doing so the rest of the code
  where push_config() is called needs to be examined to see what
  is the safe thing to do.

* Toward the end of IfConfig::commit_transaction() there is another
  pull_config() right after the interface configuration was pushed.
  The reason for that is to make sure there is no mismatch with what
  actually went into the kernel and to aligh the current
  configuration.
  Optimizing this one could be tricky, because eliminating it could
  result in error prone interface configuration.

I hope the above make is clear why there is more than one
pull_config() and what could be done to eliminate or get around some
of them.

Regards,
Pavlin



More information about the Xorp-hackers mailing list