[Xorp-hackers] RFC: Use one socket per interface for receiving packets in the FEA.

[Ben Greear]

Bruce M. Simpson wrote:
> Ben,
>
> I've only briefly glimpsed at this patch, but I have a few comments 
> based on your description of the patch.
>
> I'll leave it to Pavlin to comment on the MFEA specific scope of the 
> patch.
>
> Ben Greear wrote:
>> Here's a first attempt at using one rx socket per device, and binding 
>> to that particular
>> device.  This keeps us from receiving multicast traffic not destined 
>> for us when we are running
>> multiple instances of xorp on the same system.
>>
>> This code appears to work, but it does not properly clean up sockets 
>> when devices
>> are un-configured.  I'll be working on that next.
>>
>> This code will be less efficient than the old way if the OS doesn't 
>> support
>> SO_BINDTODEVICE, so I'll also add some code to mimic the old behaviour
>> in that case (ie, windows).
>>
>> There's also some other cruft in there to deal with races around 
>> removing interfaces
>> and removing the OSPF multicast groups.  These changes have nothing 
>> in particular
>> to do with per-interface rx sockets.
>>
>> Suggestions for improvements are welcome.
>
> Right now, using a socket per interface is actually REQUIRED by 
> protocols which rely on IP layer broadcasts, i.e. OLSR (which is in 
> production deployment in places), BCAST (which was only ever 
> experimental) and the old RIPv1 which didn't use IP layer multicast 
> (which is more widespread than you'd hope for).
>
> Pavlin and I have had some discussion about this. He quite rightly 
> states that using link-scoped multicasts is "the right thing" to do, 
> unfortunately the way that deployment of these protocols has played 
> out operationally, they are using all-ones and network broadcasts.
>
> The xorp_olsr code, which has not yet been committed publicly, opens a 
> single socket per interface via XRL in a very similar way as xorp_rip 
> does to handle RIPv1.
>
> It does this as it's the only consistent way of receiving IP 
> broadcasts on multiple interfaces in an OS portable way.
>
> The lack of SO_BINDTODEVICE on a host platform is actually not that 
> big a deal.
>
> To be sure, it's a Linux specific hack to deal with the ambiguity we 
> are presented with by the legacy BSD socket behaviour, that is, 
> broadcasts are NOT delivered to sockets which are bound in the laddr 
> tuple by bind().
>
> If anything, I speculate that the overall cost of each additional 
> socket on the host platform is negligible, compared to the cost of 
> doing dispatch/fan-out in userland for a large number of such sockets. 
> I don't have hard data, but that is my gut feeling based on exposure.
>
> In a "more deeply embedded" situation, the possibility exists that you 
> implement the IP layer in the FEA process anyway, so tight control 
> over the resource use of the XRL socket APIs exists at the cost of 
> having to code your own IP.
>
> I still feel SO_BINDTODEVICE isn't the right way forward for solving 
> the problems it was introduced to solve, it's a case of immediatist 
> pragmatism, as it introduces a number of layering violations.
I just want to make sure that I can bind a socket to an interface, and 
only have packets *from* that interface
show up on the socket.  This means that the app will not receive crap on 
interfaces that it is not (logically)
listening on.  The current CVS tree of Xorp receives on all interfaces, 
and then filters in the fea user-space logic.

When using 30 Xorp instances, this is about 900 wasted process wakes per 
second for bogus packets
(assuming a conservative one multicast pkt tx per xorp per second).

It scales N^2 as well, so gets much worse as the numbers of Xorp 
instances increase.

One problem with binding to local IPs is that if the IP addr of the 
interface changes, then
you'd have to rebind the socket.  With BINDTODEVICE that shouldn't be a 
problem.

Anyway, I've done some more work to make this handle interfaces 
leaving.  I guess it could
still fail to register a new socket if nothing tries to bind a multicast 
addr to it.  With OSPF
that isn't a problem, but maybe other routing protocols don't use 
multicast and would need
some explicit logic to allow fea to detect addition of new interfaces.

The new patch is attached.

Thanks,
Ben

-- 
Ben Greear <greearb at candelatech.com> 
Candela Technologies Inc  http://www.candelatech.com


-------------- next part --------------
A non-text attachment was scrubbed...
Name: fea2.patch
Type: text/x-patch
Size: 45678 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/xorp-hackers/attachments/20080302/17bc3709/attachment-0001.bin