[Xorp-hackers] Route with bad next-hop added with OSPF with redundant-link configuration.
Atanu Ghosh
atanu at ICSI.Berkeley.EDU
Wed Oct 24 00:42:54 PDT 2007
Hi,
If you think that routes are being incorrectly computed could you send
me the LSA database as well as the routes that you expect to see.
The LSA database can be extracted with the print_lsas command.
$ print_lsas -S saved.lsas
Atanu.
>>>>> "Ben" == Ben Greear <greearb at candelatech.com> writes:
Ben> I think this problem may be triggered by the high latency, as another
Ben> team tried to reproduce
Ben> this on two separate machines configured similarly and connected with
Ben> fast ethernet, but it
Ben> worked for them. Full logs with lots of debug enabled are available if
Ben> you want them. This
Ben> problem is 100% reproducible on my system, so I can also enable more
Ben> debugging and
Ben> generate new logs if that helps.
Ben> I have two (virtual) routers, each with 3 interfaces. Two pairs
Ben> connect to each other, the other pair connects
Ben> to an external network (not running OSPF). The interfaces are named as
Ben> A.B.C
Ben> Router 1
Ben> Router 2
Ben> 1.3.1 IP: 99.1.1.1/24 { high latency (1.5s RTT) network } 2.3.2 IP:
Ben> 99.1.1.2/24
Ben> 1.2.1 IP: 10.1.2.1/24 { 15ms RTT }
Ben> 1.2.2 IP: 10.1.2.2/24
Ben> eth1 IP: 10.1.1.1/24 -- not connected
Ben> -- eth2 IP: 10.2.2.2/24
Ben> The routing tables for router 2 use the 1.2.2 interface's IP as next
Ben> hop, instead of 1.2.1 on the peer:
Ben> Router-1 table looks OK:
Ben> [root at lf1016-55 lanforge]# ip route show table 10001
Ben> 99.1.1.0/24 dev 1.3.1 scope link
Ben> 10.2.2.0/24 via 10.1.2.2 dev 1.2.1 proto xorp metric 2 notify
Ben> 10.1.1.0/24 dev eth1 scope link
Ben> 10.1.2.0/24 dev 1.2.1 scope link
Ben> unreachable default proto xorp metric 1 notify
Ben> [root at lf1016-55 lanforge]#
Ben> Router-2 uses wrong next-hop for the xorp route:
Ben> [root at lf1016-55 lanforge]# ip route show table 10002
Ben> 99.1.1.0/24 dev 2.3.2 scope link
Ben> 10.2.2.0/24 dev eth2 scope link
Ben> 10.1.1.0/24 via 10.1.2.2 dev 1.2.2 proto xorp metric 2 notify
Ben> 10.1.2.0/24 dev 1.2.2 scope link
Ben> unreachable default proto xorp metric 1 notify
Ben> [root at lf1016-55 lanforge]#
Ben> Config files at end of email.
Ben> Logs leading up to where I think the problem lies:
Ben> [ 6317 +5582 area_router.cc routing_router_lsaV2 ] Vertex OSPFv2 Router
Ben> 127.1.0.1(0x7f010001) 0.0.0.0(0)
Ben> Router-LSA:
Ben> LS age 1 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x1 Link
Ben> State ID 127.1.0.1 Advertising Router 127.1.0.1 LS sequence number
Ben> 0x80000002 LS checksum 0x7cb1 length 60
Ben> bit Nt false
Ben> bit V false
Ben> bit E false
Ben> bit B false
Ben> Type 2 Transit network IP address of Designated router 99.1.1.2
Ben> Routers interface address 99.1.1.1 Metric 10
Ben> Type 3 Stub network Subnet number 10.1.1.0 Mask 255.255.255.0
Ben> Metric 1
Ben> Type 2 Transit network IP address of Designated router 10.1.2.2
Ben> Routers interface address 10.1.2.1 Metric 1
Ben> [ 6317 +5562 area_router.cc update_edge ] src OSPFv2 Router
Ben> 127.1.0.1(0x7f010001) 0.0.0.0(0) metric 10 dst OSPFv2 Network
Ben> 99.1.1.2(0x63010102) 0.0.0.0(0)
Ben> [ 6317 +5562 area_router.cc update_edge ] src OSPFv2 Network
Ben> 99.1.1.2(0x63010102) 0.0.0.0(0) metric 0 dst OSPFv2 Router
Ben> 127.1.0.1(0x7f010001) 0.0.0.0(0)
Ben> [ 6317 +5562 area_router.cc update_edge ] src OSPFv2 Router
Ben> 127.1.0.1(0x7f010001) 0.0.0.0(0) metric 1 dst OSPFv2 Network
Ben> 10.1.2.2(0xa010202) 0.0.0.0(0)
Ben> [ 6317 +5562 area_router.cc update_edge ] src OSPFv2 Network
Ben> 10.1.2.2(0xa010202) 0.0.0.0(0) metric 0 dst OSPFv2 Router
Ben> 127.1.0.1(0x7f010001) 0.0.0.0(0)
Ben> [ 6317 +51 routing_table.cc begin ] area 0.0.0.0
Ben> [ 6317 +675 routing_table.cc clear_area ] Clearing area 0.0.0.0
Ben> [ 6317 +72 routing_table.cc begin ] ire Area: 0.0.0.0 RouteEntry:
Ben> Network Address 10.1.1.0 Area 0.0.0.0 intra area cost 11 nexthop
Ben> 99.1.1.1 advertising router 127.1.0.1 Network-LSA:
Ben> LS age 0 Options 0 DC: 0 EA: 0 N/P: 0 MC: 0 E: 0 LS type 0x2 Link
Ben> State ID 10.1.1.0 Advertising Router 127.1.0.1 LS sequence number
Ben> 0x80000001 LS checksum 0 length 0
Ben> Network Mask 0xffffff00 winner
Ben> [ 6317 +81 routing_table.cc begin ] empty ire only this area was present
Ben> [ 6317 +72 routing_table.cc begin ] ire Area: 0.0.0.0 RouteEntry:
Ben> Network direct Address 10.1.2.2 Area 0.0.0.0 intra area cost 1 nexthop
Ben> 10.1.2.2 advertising router 127.1.0.2 Network-LSA:
Ben> LS age 1 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x2 Link
Ben> State ID 10.1.2.2 Advertising Router 127.1.0.2 LS sequence number
Ben> 0x80000001 LS checksum 0x665e length 32
Ben> Network Mask 0xffffff00
Ben> Attached Router 127.1.0.2
Ben> Attached Router 127.1.0.1 winner
Ben> [ 6317 +81 routing_table.cc begin ] empty ire only this area was present
Ben> [ 6317 +72 routing_table.cc begin ] ire Area: 0.0.0.0 RouteEntry:
Ben> Network direct Address 10.2.2.0 Area 0.0.0.0 intra area cost 1 nexthop
Ben> 0.0.0.0 advertising router 127.1.0.2 Network-LSA:
Ben> LS age 0 Options 0 DC: 0 EA: 0 N/P: 0 MC: 0 E: 0 LS type 0x2 Link
Ben> State ID 10.2.2.0 Advertising Router 127.1.0.2 LS sequence number
Ben> 0x80000001 LS checksum 0 length 0
Ben> Network Mask 0xffffff00 winner
Ben> [ 6317 +81 routing_table.cc begin ] empty ire only this area was present
Ben> [ 6317 +72 routing_table.cc begin ] ire Area: 0.0.0.0 RouteEntry:
Ben> Network direct Address 99.1.1.2 Area 0.0.0.0 intra area cost 10 nexthop
Ben> 99.1.1.2 advertising router 127.1.0.2 Network-LSA:
Ben> LS age 0 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x2 Link
Ben> State ID 99.1.1.2 Advertising Router 127.1.0.2 LS sequence number
Ben> 0x80000001 LS checksum 0xe784 length 32
Ben> Network Mask 0xffffff00
Ben> Attached Router 127.1.0.2
Ben> Attached Router 127.1.0.1 winner
Ben> [ 6317 +81 routing_table.cc begin ] empty ire only this area was present
Ben> [ 6317 +907 ../libproto/spt.hh set_adjacent_weights ] Node: OSPFv2
Ben> Network 10.1.2.2(0xa010202) 10.1.2.2(0x202010a)
Ben> [ 6317 +907 ../libproto/spt.hh set_adjacent_weights ] Node: OSPFv2
Ben> Network 10.2.2.255(0xa0202ff) 0.0.0.0(0)
Ben> [ 6317 +907 ../libproto/spt.hh set_adjacent_weights ] Node: OSPFv2
Ben> Network 99.1.1.2(0x63010102) 99.1.1.2(0x2010163)
Ben> [ 6317 +907 ../libproto/spt.hh set_adjacent_weights ] Node: OSPFv2
Ben> Router 127.1.0.1(0x7f010001) 99.1.1.1(0x1010163)
Ben> [ 6317 +751 ../libproto/spt.hh dijkstra ] Previous: OSPFv2(Origin)
Ben> Router 127.1.0.2(0x7f010002) 0.0.0.0(0)
Ben> [ 6317 +755 ../libproto/spt.hh dijkstra ] Permanent: OSPFv2 Network
Ben> 10.2.2.255(0xa0202ff) 0.0.0.0(0) distance 1 next hop OSPFv2 Network
Ben> 10.2.2.255(0xa0202ff) 0.0.0.0(0)
Ben> [ 6317 +751 ../libproto/spt.hh dijkstra ] Previous: OSPFv2(Origin)
Ben> Router 127.1.0.2(0x7f010002) 0.0.0.0(0)
Ben> [ 6317 +755 ../libproto/spt.hh dijkstra ] Permanent: OSPFv2 Network
Ben> 10.1.2.2(0xa010202) 10.1.2.2(0x202010a) distance 1 next hop OSPFv2
Ben> Network 10.1.2.2(0xa010202) 10.1.2.2(0x202010a)
Ben> [ 6317 +907 ../libproto/spt.hh set_adjacent_weights ] Node: OSPFv2
Ben> Router 127.1.0.1(0x7f010001) 99.1.1.1(0x1010163)
Ben> [ 6317 +907 ../libproto/spt.hh set_adjacent_weights ] Node:
Ben> OSPFv2(Origin) Router 127.1.0.2(0x7f010002) 0.0.0.0(0)
Ben> [ 6317 +751 ../libproto/spt.hh dijkstra ] Previous: OSPFv2 Network
Ben> 10.1.2.2(0xa010202) 10.1.2.2(0x202010a)
Ben> [ 6317 +755 ../libproto/spt.hh dijkstra ] Permanent: OSPFv2 Router
Ben> 127.1.0.1(0x7f010001) 99.1.1.1(0x1010163) distance 1 next hop OSPFv2
Ben> Network 10.1.2.2(0xa010202) 10.1.2.2(0x202010a)
Ben> [ 6317 +907 ../libproto/spt.hh set_adjacent_weights ] Node: OSPFv2
Ben> Network 10.1.1.255(0xa0101ff) 0.0.0.0(0)
Ben> [ 6317 +907 ../libproto/spt.hh set_adjacent_weights ] Node: OSPFv2
Ben> Network 10.1.2.2(0xa010202) 10.1.2.2(0x202010a)
Ben> [ 6317 +907 ../libproto/spt.hh set_adjacent_weights ] Node: OSPFv2
Ben> Network 99.1.1.2(0x63010102) 99.1.1.2(0x2010163)
Ben> [ 6317 +751 ../libproto/spt.hh dijkstra ] Previous: OSPFv2 Router
Ben> 127.1.0.1(0x7f010001) 99.1.1.1(0x1010163)
Ben> [ 6317 +755 ../libproto/spt.hh dijkstra ] Permanent: OSPFv2 Network
Ben> 10.1.1.255(0xa0101ff) 0.0.0.0(0) distance 2 next hop OSPFv2 Network
Ben> 10.1.2.2(0xa010202) 10.1.2.2(0x202010a)
Ben> [ 6317 +751 ../libproto/spt.hh dijkstra ] Previous: OSPFv2(Origin)
Ben> Router 127.1.0.2(0x7f010002) 0.0.0.0(0)
Ben> [ 6317 +755 ../libproto/spt.hh dijkstra ] Permanent: OSPFv2 Network
Ben> 99.1.1.2(0x63010102) 99.1.1.2(0x2010163) distance 10 next hop OSPFv2
Ben> Network 99.1.1.2(0x63010102) 99.1.1.2(0x2010163)
Ben> [ 6317 +907 ../libproto/spt.hh set_adjacent_weights ] Node: OSPFv2
Ben> Router 127.1.0.1(0x7f010001) 99.1.1.1(0x1010163)
Ben> [ 6317 +907 ../libproto/spt.hh set_adjacent_weights ] Node:
Ben> OSPFv2(Origin) Router 127.1.0.2(0x7f010002) 0.0.0.0(0)
Ben> [ 6317 +3919 area_router.cc routing_total_recomputeV2 ] Add route: Node:
Ben> OSPFv2 Network 10.1.1.255(0xa0101ff) 0.0.0.0(0) -> Nexthop OSPFv2
Ben> Network 10.1.2.2(0xa010202) 10.1.2.2(0x202010a)
Ben> [ 6317 +231 routing_table.cc lookup_entry ] 10.1.1.0/24
Ben> [ 6317 +96 routing_table.cc add_entry ] area 0.0.0.0 10.1.1.0/24
Ben> RouteEntry: Network Address 10.1.1.0 Area 0.0.0.0 intra area cost 2
Ben> nexthop 10.1.2.2 advertising router 127.1.0.1 Network-LSA:
Ben> LS age 0 Options 0 DC: 0 EA: 0 N/P: 0 MC: 0 E: 0 LS type 0x2 Link
Ben> State ID 10.1.1.0 Advertising Router 127.1.0.1 LS sequence number
Ben> 0x80000001 LS checksum 0 length 0
Ben> Network Mask 0xffffff00 msg: void
Ben> AreaRouter<A>::routing_total_recomputeV2() [with A = IPv4]
Ben> [ 6317 +3919 area_router.cc routing_total_recomputeV2 ] Add route: Node:
Ben> OSPFv2 Network 10.1.2.2(0xa010202) 10.1.2.2(0x202010a) -> Nexthop OSPFv2
Ben> Network 10.1.2.2(0xa010202) 10.1.2.2(0x202010a)
Ben> [ 6317 +231 routing_table.cc lookup_entry ] 10.1.2.0/24
Ben> [ 6317 +96 routing_table.cc add_entry ] area 0.0.0.0 10.1.2.0/24
Ben> RouteEntry: Network direct Address 10.1.2.2 Area 0.0.0.0 intra area cost
Ben> 1 nexthop 10.1.2.2 advertising router 127.1.0.2 Network-LSA:
Ben> LS age 1 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x2 Link
Ben> State ID 10.1.2.2 Advertising Router 127.1.0.2 LS sequence number
Ben> 0x80000001 LS checksum 0x665e length 32
Ben> Network Mask 0xffffff00
Ben> Attached Router 127.1.0.2
Ben> Attached Router 127.1.0.1 msg: void
Ben> AreaRouter<A>::routing_total_recomputeV2() [with A = IPv4]
Ben> [ 6317 +3919 area_router.cc routing_total_recomputeV2 ] Add route: Node:
Ben> OSPFv2 Network 10.2.2.255(0xa0202ff) 0.0.0.0(0) -> Nexthop OSPFv2
Ben> Network 10.2.2.255(0xa0202ff) 0.0.0.0(0)
Ben> [ 6317 +231 routing_table.cc lookup_entry ] 10.2.2.0/24
Ben> [ 6317 +96 routing_table.cc add_entry ] area 0.0.0.0 10.2.2.0/24
Ben> RouteEntry: Network direct Address 10.2.2.0 Area 0.0.0.0 intra area cost
Ben> 1 nexthop 0.0.0.0 advertising router 127.1.0.2 Network-LSA:
Ben> LS age 0 Options 0 DC: 0 EA: 0 N/P: 0 MC: 0 E: 0 LS type 0x2 Link
Ben> State ID 10.2.2.0 Advertising Router 127.1.0.2 LS sequence number
Ben> 0x80000001 LS checksum 0 length 0
Ben> Network Mask 0xffffff00 msg: void
Ben> AreaRouter<A>::routing_total_recomputeV2() [with A = IPv4]
Ben> [ 6317 +3919 area_router.cc routing_total_recomputeV2 ] Add route: Node:
Ben> OSPFv2 Network 99.1.1.2(0x63010102) 99.1.1.2(0x2010163) -> Nexthop
Ben> OSPFv2 Network 99.1.1.2(0x63010102) 99.1.1.2(0x2010163)
Ben> [ 6317 +231 routing_table.cc lookup_entry ] 99.1.1.0/24
Ben> [ 6317 +96 routing_table.cc add_entry ] area 0.0.0.0 99.1.1.0/24
Ben> RouteEntry: Network direct Address 99.1.1.2 Area 0.0.0.0 intra area cost
Ben> 10 nexthop 99.1.1.2 advertising router 127.1.0.2 Network-LSA:
Ben> LS age 0 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x2 Link
Ben> State ID 99.1.1.2 Advertising Router 127.1.0.2 LS sequence number
Ben> 0x80000001 LS checksum 0xe784 length 32
Ben> Network Mask 0xffffff00
Ben> Attached Router 127.1.0.2
Ben> Attached Router 127.1.0.1 msg: void
Ben> AreaRouter<A>::routing_total_recomputeV2() [with A = IPv4]
Ben> [ 6317 +3919 area_router.cc routing_total_recomputeV2 ] Add route: Node:
Ben> OSPFv2 Router 127.1.0.1(0x7f010001) 99.1.1.1(0x1010163) -> Nexthop
Ben> OSPFv2 Network 10.1.2.2(0xa010202) 10.1.2.2(0x202010a)
Ben> [ 2007/10/23 21:39:47 TRACE xorp_ospfv2 OSPF ] Checking for virtual
Ben> links V2, count(rid): 0 Router-LSA:
Ben> LS age 1 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x1 Link
Ben> State ID 127.1.0.1 Advertising Router 127.1.0.1 LS sequence number
Ben> 0x80000002 LS checksum 0x7cb1 length 60
Ben> bit Nt false
Ben> bit V false
Ben> bit E false
Ben> bit B false
Ben> Type 2 Transit network IP address of Designated router 99.1.1.2
Ben> Routers interface address 99.1.1.1 Metric 10
Ben> Type 3 Stub network Subnet number 10.1.1.0 Mask 255.255.255.0
Ben> Metric 1
Ben> Type 2 Transit network IP address of Designated router 10.1.2.2
Ben> Routers interface address 10.1.2.1 Metric 1
Ben> [ 6317 +276 routing_table.cc end ]
Ben> [ 6317 +442 routing_table.cc replace_route ] REPLACE ROUTE area 0.0.0.0
Ben> 10.1.1.0/24
Ben> [ 6317 +417 routing_table.cc delete_route ] DELETE ROUTE area 0.0.0.0
Ben> 10.1.1.0/24 filtered false
Ben> [ 2007/10/23 21:39:47 TRACE xorp_ospfv2 OSPF ] Delete route Net 10.1.1.0/24
Ben> [ 6317 +389 routing_table.cc add_route ] ADD ROUTE area 0.0.0.0 net
Ben> 10.1.1.0/24 nexthop 10.1.2.2 metric 2
Ben> [ 6317 +522 routing_table.cc do_filtering ] [OSPF] Running filter:
Ben> Import on route: 10.1.1.0/24
Ben> [ 2007/10/23 21:39:47 TRACE xorp_ospfv2 OSPF ] [OSPF] Running filter:
Ben> Import on route: 10.1.1.0/24
Ben> [ 6317 +540 routing_table.cc do_filtering ] [OSPF] Running filter:
Ben> Export-SourceMatch on route: 10.1.1.0/24
Ben> [ 2007/10/23 21:39:47 TRACE xorp_ospfv2 OSPF ] [OSPF] Running filter:
Ben> Export-SourceMatch on route: 10.1.1.0/24
Ben> [ 2007/10/23 21:39:47 TRACE xorp_ospfv2 OSPF ] Add route Net 10.1.1.0/24
Ben> Nexthop 10.1.2.2 metric 2 equal false discard false policy
Ben> [ 2007/10/23 21:39:47 TRACE xorp_ospfv2 OSPF ] Interface 2.3.2 Vif 2.3.2
Ben> dst 224.0.0.5 src 99.1.1.1 data 0x8394730 len 88
Ben> router-1 config:
Ben> /* For Virtual-Router: Router-1 */
Ben> interfaces {
Ben> interface my_discard {
Ben> unreachable: true
Ben> vif my_discard {
Ben> }
Ben> }
Ben> interface 1.3.1 {
Ben> vif 1.3.1 {
Ben> address 99.1.1.1 {
Ben> prefix-length: 24
Ben> }
Ben> }
Ben> }
Ben> interface eth1 {
Ben> vif eth1 {
Ben> address 10.1.1.1 {
Ben> prefix-length: 24
Ben> }
Ben> }
Ben> }
Ben> interface 1.2.1 {
Ben> vif 1.2.1 {
Ben> address 10.1.2.1 {
Ben> prefix-length: 24
Ben> }
Ben> }
Ben> }
Ben> }
Ben> fea {
Ben> unicast-forwarding4 {
Ben> disable: false
Ben> table-id: 10001
Ben> }
Ben> }
Ben> protocols {
Ben> static {
Ben> interface-route 0.0.0.0/0 {
Ben> next-hop-interface: "my_discard"
Ben> next-hop-vif: "my_discard"
Ben> }
Ben> }
Ben> ospf4 {
Ben> router-id: 127.1.0.1
Ben> area 0.0.0.0 {
Ben> interface 1.3.1 {
Ben> vif 1.3.1 {
Ben> address 99.1.1.1 {
Ben> interface-cost: 10
Ben> }
Ben> }
Ben> }
Ben> interface eth1 {
Ben> vif eth1 {
Ben> address 10.1.1.1 {
Ben> interface-cost: 1
Ben> }
Ben> }
Ben> }
Ben> interface 1.2.1 {
Ben> vif 1.2.1 {
Ben> address 10.1.2.1 {
Ben> interface-cost: 1
Ben> }
Ben> }
Ben> }
Ben> }
Ben> traceoptions {
Ben> flag all {
Ben> disable: false
Ben> }
Ben> }
Ben> }
Ben> }
Ben> /* End of Config File */
Ben> router-2 config:
Ben> /* For Virtual-Router: Router-2 */
Ben> interfaces {
Ben> interface my_discard {
Ben> unreachable: true
Ben> vif my_discard {
Ben> }
Ben> }
Ben> interface 2.3.2 {
Ben> vif 2.3.2 {
Ben> address 99.1.1.2 {
Ben> prefix-length: 24
Ben> }
Ben> }
Ben> }
Ben> interface eth2 {
Ben> vif eth2 {
Ben> address 10.2.2.2 {
Ben> prefix-length: 24
Ben> }
Ben> }
Ben> }
Ben> interface 1.2.2 {
Ben> vif 1.2.2 {
Ben> address 10.1.2.2 {
Ben> prefix-length: 24
Ben> }
Ben> }
Ben> }
Ben> }
Ben> fea {
Ben> unicast-forwarding4 {
Ben> disable: false
Ben> table-id: 10002
Ben> }
Ben> }
Ben> protocols {
Ben> static {
Ben> interface-route 0.0.0.0/0 {
Ben> next-hop-interface: "my_discard"
Ben> next-hop-vif: "my_discard"
Ben> }
Ben> }
Ben> ospf4 {
Ben> router-id: 127.1.0.2
Ben> area 0.0.0.0 {
Ben> interface 2.3.2 {
Ben> vif 2.3.2 {
Ben> address 99.1.1.2 {
Ben> interface-cost: 10
Ben> }
Ben> }
Ben> }
Ben> interface eth2 {
Ben> vif eth2 {
Ben> address 10.2.2.2 {
Ben> interface-cost: 1
Ben> }
Ben> }
Ben> }
Ben> interface 1.2.2 {
Ben> vif 1.2.2 {
Ben> address 10.1.2.2 {
Ben> interface-cost: 1
Ben> }
Ben> }
Ben> }
Ben> }
Ben> traceoptions {
Ben> flag all {
Ben> disable: false
Ben> }
Ben> }
Ben> }
Ben> }
Ben> /* End of Config File */
Ben> --
Ben> Ben Greear <greearb at candelatech.com>
Ben> Candela Technologies Inc http://www.candelatech.com
Ben> _______________________________________________
Ben> Xorp-hackers mailing list
Ben> Xorp-hackers at icir.org
Ben> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/xorp-hackers
More information about the Xorp-hackers
mailing list