[Xorp-users] Question on supporting multiple routing tables.

[Pavlin Radoslavov]

Ben Greear <greearb at candelatech.com> wrote:

> Pavlin Radoslavov wrote:
> >> This sounds like a good first step, and it sounds like it meets my 
> >> immediate needs.  Instead
> >>     
> >
> > OK, I just committed to CVS the changes for configuring the table
> > ID. You can do it with configuration like:
> >
> >       fea {
> >           unicast-forwarding4 {
> >               table-id: 20
> >           }
> >           unicast-forwarding6 {
> >               table-id: 20
> >           }
> >       }
> >
> > Obviously, you won't need the unicast-forwarding6 section if you
> > don't need IPv6.
> >
> > I just did some simple testing and it appears to work.
> > Please get the latest code from anon. CVS and let us know whether it
> > works for you:
> >
> > http://www.xorp.org/cvs.html
> >   
> Thank you!  This was a bit faster than I had hoped, so it may take a day 
> or two before
> we can try it out!  I'm going to have to read up on how to configure 
> Xorp, as I haven't
> configured an OSPF router or set of routers before.
> 
> In case you have time & inclination, do you have two simple config files 
> to make this route with OSPF:
> 
> VR1:  table 100, eth0, eth1
> VR2:  table 101, eth2, eth3
> 
> Assume eth1 and eth2 are connected to each other via cross-over cable 
> and some separate PCs
> are connected to eth0 and eth3.

The "Getting Started" Web page has quick startup information for
configuring XORP. However, it reflects only the latest 1.4 release,
so here is a sample config with the table IDs for VR1:

interfaces {
    interface eth0 {
        vif eth0 {
            address 10.0.0.1 {
                prefix-length: 24
            }
        }
    }
    interface eth1 {
        vif eth1 {
            address 10.1.1.1 {
                prefix-length: 24
            }
        }
    }
}

fea {
    unicast-forwarding4 {
        disable: false
        table-id: 100
    }
}

protocols {
    ospf4 {
        router-id: 10.0.0.1
	area 0.0.0.0 {
	    interface eth0 {
		vif eth0 {
		    address 10.0.0.1 {
		    }
		}
	    }
	    interface eth1 {
		vif eth1 {
		    address 10.1.1.1 {
		    }
		}
	    }
	}
    }
}

For VR2 just replace eth0 and eth1 with eth2 and eth3 respectively,
change their IP addresses, and set table-id to 101.

> In order to make the messages go over the external interfaces, the Xorp 
> code will probably
> need an option to bind to the local IP and device.  For TCP sockets that
> can do this, my code looks like this (I don't know if this applies to 
> IPv6 or not..I've only
> tried with IPv4.)

When we transmit multicast packets (e.g., raw IP packets used by
OSPF or UDP used by RIP), we explicitly set the outgoing interface
by using setsockopt(IP_MULTICAST_IF).

I could be wrong, but for transmitting unicast packets (if we
exclude the Linux-specific SO_BINDTODEVICE) I don't think we (can)
explicitly specify the outgoing interface. I believe it will be
automatically chosen by the kernel (based on the matching forwarding
entry toward the destination). Only if the destination is on the same
subnet, we could use SO_DONTROUTE socket option or MSG_DONTROUTE
flag for send*(); currently we don't which we might need to change.

Could you confirm that SO_BINDTODEVICE has impact when choosing the
outgoing interface for transmitting packets?
The socket(7) Linux manual page describes SO_BINDTODEVICE's impact 
only for receiving packets, but the manual page itself might be
incomplete:

      If a socket is bound to an interface, only packets received from
      that particular interface are processed by the socket.

> int createTcpSocket(unsigned int ip_addr, int ip_port, const char* 
> dev_to_bind_to) {
>  
>    int s = socket(AF_INET, SOCK_STREAM, 0);  
> 
>    if (s < 0) {
>       cerr << "ERROR: tcp socket:  " << LFSTRERROR << endl;
>            return s;
>    }
> 
>    close_exec(s);
> 
>    reuseaddr_socket(s);
> 
>    if (dev_to_bind_to) {
>       // Bind to specific device.
> #ifndef __WIN32__
>       if (setsockopt(s, SOL_SOCKET, SO_BINDTODEVICE,
>                      dev_to_bind_to, DEV_NAME_LEN + 1)) {
>          VLOG_ERR(VLOG << "ERROR:  tcp-connect, setsockopt 
> (BINDTODEVICE):  "
>                   << LFSTRERROR << "  Not fatal in most 
> cases..continuing...\n");
>       }
> #endif
>    }//if
> 
>    struct sockaddr_in my_ip_addr;
>    memset(&my_ip_addr, 0, sizeof(my_ip_addr));
>   
>    my_ip_addr.sin_family = AF_INET;
>    my_ip_addr.sin_addr.s_addr = htonl(ip_addr);
>    my_ip_addr.sin_port = htons(ip_port);
>   
>    int r; //retval
>    r = bind(s, (struct sockaddr*)(&my_ip_addr), sizeof(my_ip_addr));
>    if (r < 0) {
>       //system("netstat -an");
>       cerr << "ERROR: tcp bind:  " << LFSTRERROR << endl;
>       closesocket(s);
>       return r;
>    }
>    else {
>       VLOG_INF(VLOG << "Successfully bound to IP: " << 
> toStringIP(ip_addr) << " port: "
>                << ip_port << endl);
>    }
> 
>    nonblock(s, "create tcp socket");
>    return s;
> }
> 
> 
> 
> >> of an environment variable, I'd prefer a cmd-line variable and/or 
> >> something in
> >> the xorp router config file.  However, I know almost nothing about how Xorp
> >> is written, so if it needs to be an environ variable, that is OK.
> >>     
> >
> > Eventually, in the future we should add support for configuring the
> > Finder's address and port inside the XORP confuguration file, but
> > such change probably won't be trivial.
> >
> > BTW, if you set the environmental variable before starting the XORP
> > rtrmgr, you need to set that variable as well before starting the
> > corresponding xorpsh CLI.
> >   
> So, the environ variable is still needed, in addition to the 
> unicast-forwarding { } sections above?

Yes. The environmental variable is needed so you can run more than
one XORP instance on the same machine.

> >> I do not need this to support multicast at the moment.
> >>
> >> Eventually, I'd like to scale to 100s of VRs, so it would probably be 
> >> more efficient to run a
> >> smaller number of Xorp processes, each able to handle a lot of routing 
> >> tables, but
> >> that can be dealt with later.  And, it may turn out that running 100+ 
> >> processes
> >> is not such a big deal considering all of the cores on newer systems...
> >>     
> >
> > Probably you will be the first one to try 100s of XORP instances, so
> > we would be very interested to know how it goes.
> >
> > BTW, in Linux you cannot have more than 255 or so forwarding tables,
> > and some of those tables shouldn't be touched, so ~250 is the
> > max. number of table IDs and XORP instances you can try.
> >   
> I worked with someone to get this fixed recently, and have tested 2000+ 
> tables,
> so this limit no longer exists since kernel 2.6.20 or so.  You need the 
> latest 'ip'
> tool as well...

Thanks for the info. You are right.
Originally I was looking inside file "include/linux/rtnetlink.h" I
saw that field rtm_table inside "struct rtmsg" is "unsigned char"
which explained the 255 limit.
After looking into the latest source code for iproute2 I see that
the > 255 support is done by using the new RTA_TABLE attribute.

For those of you who might be interested in the technical semantics
of setting the table ID, it is:

If the table ID is <= 0xff, then we set it in rtmsg->rtm_table,
otherwise we set rtmsg->rtm_table to RT_TABLE_UNSPEC and add the
real value as an RTA_TABLE attribute.

Anyway, I fixed the XORP code and now it supports table ID larger
than 255.

Thanks,
Pavlin