[Xorp-hackers] FEA performance improvements: only 'pull' active interfaces.

[Ben Greear]

Pavlin Radoslavov wrote:
> Ben Greear <greearb at candelatech.com> wrote:
> 
>> As previously mentioned, I notice that fea pays attention to all 
>> interfaces, not just those
>> it's configured to use.
>>
>> To help improve scalability, especially in a virtualized environement, I 
>> am attempting
>> the following:
>>
>> 1)  Only have the _pulled_config pull information for devices stored in 
>> the _local_config
>> tree.  This means asking netlink for specific if-index values instead of 
>> the entire tree.
> 
> The problem with this solution is that it will work only for Netlink
> on Linux. The (majoriyty of the) other mechanisms for obtaining the
> network interface information (getifaddrs(3), ioctl(2), syssctl(3),
> etc) don't allow the granularity for asking only the information for
> a specific interface.
> 
> In addition, if you have lots of configured interfaces, it might
> actually be much more expensive to ask separately for each of them
> (i.e., to use a system call per interface) instead of using a single
> system call to obtain the information for all interfaces.

Yes, this may only be useful for my scenario where I'm using a small
number of interfaces per xorp instance, with large numbers of total interfaces.
Only linux can virtualize routing tables, as far as I know, so this performance
gain is only really important on Linux.

>> 2)  The netlink observer will ignore anything not in the _local_config, 
>> and will remove
>> interfaces from _local_config if it observes them unregistering from the 
>> system.
>>
>> 3)  When adding an interface (though XRL), the ifconfig object will add 
>> it to _local_config,
>> tell the pulled_config to pull it from the system, and if found, will 
>> save it in the _original_config
>> as well in case we want to roll back to the original system state.  Once 
>> added, nothing is removed
>> from the _original_config.
> 
> You could keep track in _pulled_config of only those interfaces that
> are configured in XORP (i.e., those in _local_config).
> However, this could add more complexity to all the interface-related
> machinery (IfTree, IfConfig, etc).
> Also, given the extra search/complexity you need to do to maintain
> that state it becomes questionable how much performance gain there
> will be.
> 
> Last but not least, don't forget that if there is any performance
> gain it will show only in the case where there are lots of
> interfaces in the system, but the XORP instance is configured with a
> very small number of that interfaces.

My assumption is that anything not in local_config can be ignored
in the pull_config logic, and anything that searches for a device not
in local config should not expect to find it.  There will be a bit of
special cases for adding new devices to the local_config, but not much else
I think.

So far, it seems the code complexity will not be great, it's mostly lots of
small boring API changes.  But, it's not working yet, so it may get more
complex...

> 
>> 4)  There is an XRL method to configure all interfaces from the system.  
>> I am hoping this isn't
>> actually needed and can be removed, as it would require reading the 
>> entire set of interfaces.  I
>> can (re)add code to support this if needed, but maybe it isn't really 
>> useful and could be removed?
> 
> Currently this method is used by some of the FEA test programs.

Ok, I'll (re)add it...it's not difficult since that is the default behaviour
of the existing pull_config logic.

> 
>> I am only implementing the optimizations for the netlink related 
>> portions.  The remainder of the iftree-get/set logic
>> will use the current method of reading all interfaces regardless of 
>> local config.
>>
>> I believe this will go a long way towards helping fea scale to 1000+ 
>> interfaces, but don't have performance
>> numbers or working code quite yet.
> 
> As usual, I don't want to commit to any optimizations before I see
> numbers that justify the extra complexity :)
> 
> 
> Independent from the above, as you have noticed already the FEA
> calls pull_config() several times so the alternative solution would
> be to try to reduce that number. There is more than one
> pull_config() for both technical and historical reasons (see below).

My problem is that each xorp is only interested in ~15 interfaces, but
I have 30+ xorp instances (and would like 100+).  This means that decreasing pull_config calls by
a small number will not be nearly as important making the method scale
linearly with *configured* devices as opposed to existing devices.

I did look at the code you describe, and didn't immediately get any ideas for
being able to remove calls.  But, I'll look at all of that again when I get
the logic to pull only configured devices functioning...

Thanks,
Ben

-- 
Ben Greear <greearb at candelatech.com>
Candela Technologies Inc  http://www.candelatech.com