[Xorp-hackers] OSPF stuck in Loading state
Ben Greear
greearb at candelatech.com
Mon May 12 16:44:31 PDT 2008
I seem to have a repeatable OSPF problem where it gets stuck in
loading state.
The general scenario is that there is a dense mesh of 10 nodes
that slowly pull apart and drop connections. Near the end, node
4 is connected to node 9 and node-9 connects to most of the
rest of the network. Two nodes (2 and 7) are in their own group and not
reachable from the main group.
Both router 4 and router 9 show each other as peers and in
the Loading state. Nothing obviously wrong in the log files,
but the OSPF traffic between them seems to be stuck in some
sort of loop.
A pcap capture is here:
http://www.candelatech.com/oss/ospf_loading.pcap
I'm going to try this all again with the latest CVS (this build is a few weeks old)
and will enable debugging in OSPF if still no luck. Please let me know if you have
any suggestions or need more debugging info.
# Router 9:
lanforge at a440-qc> show ospf4 neighbor
created new heap in find_heap, ptr: 0x811ec30
created new heap in find_heap, ptr: 0x8120580
Address Interface State ID Pri Dead
10.3.9.3 3.9.9/3.9.9 Full 127.1.0.3 128 39
10.4.9.4 4.9.9/4.9.9 Loading 127.1.0.4 128 35
lanforge at a440-qc> show interfaces
created new heap in find_heap, ptr: 0x814bd10
created new heap in find_heap, ptr: 0x8150698
3.9.9/3.9.9: Flags:<ENABLED,BROADCAST,MULTICAST> mtu 1500
inet 10.3.9.9 subnet 10.3.9.0/24 broadcast 10.3.9.255
physical index 95
ether 0:26:50:3e:60:49
4.9.9/4.9.9: Flags:<ENABLED,BROADCAST,MULTICAST> mtu 1500
inet 10.4.9.9 subnet 10.4.9.0/24 broadcast 10.4.9.255
physical index 187
ether 0:4:13:84:8d:2e
br9/br9: Flags:<ENABLED,BROADCAST,MULTICAST> mtu 1500
inet 10.9.9.9 subnet 10.9.9.0/24 broadcast 10.9.9.255
physical index 44
ether 0:44:c3:f0:2f:46
my_discard/my_discard: Flags:<ENABLED,UNREACHABLE> mtu 0
physical index 0
lanforge at a440-qc>
[lanforge at a440-qc ~]$ tail -40 xorp-log-vr10009.txt
[ 2008/05/12 16:31:28 WARNING xorp_ospfv2:13291 OSPF area_router.cc:5759 routing_router_link_transitV2 ] LSA in database MaxAge
Network-LSA:
LS age 3600 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x2 Link State ID 10.2.5.5 Advertising Router 127.1.0.5 LS sequence number 0x80000001 LS checksum 0x3581 length 32
Network Mask 0xffffff00
Attached Router 127.1.0.5
Attached Router 127.1.0.2
[ 2008/05/12 16:31:49 WARNING xorp_fea XrlFeaTarget ] Handling method for raw_packet4/0.1/send failed: XrlCmdError 102 Command failed Error while sending to vif: 3.9.9:3.9.9 src: 10.3.9.9 dest: 224.0.0.5: sendmsg failed, error: Network is down socket: 57
[ 2008/05/12 16:31:49 ERROR xorp_ospfv2:13291 OSPF xrl_io.cc:188 send_cb ] Cannot send a packet on interface 3.9.9 vif 3.9.9: 102 Command failed Error while sending to vif: 3.9.9:3.9.9 src: 10.3.9.9 dest: 224.0.0.5: sendmsg failed, error: Network is down socket: 57
[ 2008/05/12 16:31:49 WARNING xorp_fea XrlFeaTarget ] Handling method for raw_packet4/0.1/send failed: XrlCmdError 102 Command failed Error while sending to vif: 3.9.9:3.9.9 src: 10.3.9.9 dest: 224.0.0.5: sendmsg failed, error: Network is down socket: 57
[ 2008/05/12 16:31:49 ERROR xorp_ospfv2:13291 OSPF xrl_io.cc:188 send_cb ] Cannot send a packet on interface 3.9.9 vif 3.9.9: 102 Command failed Error while sending to vif: 3.9.9:3.9.9 src: 10.3.9.9 dest: 224.0.0.5: sendmsg failed, error: Network is down socket: 57
[ 2008/05/12 16:31:56 WARNING xorp_ospfv2:13291 OSPF area_router.cc:5759 routing_router_link_transitV2 ] LSA in database MaxAge
Network-LSA:
LS age 3600 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x2 Link State ID 10.2.5.5 Advertising Router 127.1.0.5 LS sequence number 0x80000001 LS checksum 0x3581 length 32
Network Mask 0xffffff00
Attached Router 127.1.0.5
Attached Router 127.1.0.2
[ 2008/05/12 16:32:35 WARNING xorp_ospfv2:13291 OSPF area_router.cc:5759 routing_router_link_transitV2 ] LSA in database MaxAge
Network-LSA:
LS age 3600 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x2 Link State ID 10.2.5.5 Advertising Router 127.1.0.5 LS sequence number 0x80000001 LS checksum 0x3581 length 32
Network Mask 0xffffff00
Attached Router 127.1.0.5
Attached Router 127.1.0.2
[ 2008/05/12 16:33:09 WARNING xorp_ospfv2:13291 OSPF area_router.cc:5759 routing_router_link_transitV2 ] LSA in database MaxAge
Network-LSA:
LS age 3600 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x2 Link State ID 10.2.5.5 Advertising Router 127.1.0.5 LS sequence number 0x80000001 LS checksum 0x3581 length 32
Network Mask 0xffffff00
Attached Router 127.1.0.5
Attached Router 127.1.0.2
[ 2008/05/12 16:33:17 WARNING xorp_ospfv2:13291 OSPF area_router.cc:5759 routing_router_link_transitV2 ] LSA in database MaxAge
Network-LSA:
LS age 3600 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x2 Link State ID 10.2.5.5 Advertising Router 127.1.0.5 LS sequence number 0x80000001 LS checksum 0x3581 length 32
Network Mask 0xffffff00
Attached Router 127.1.0.5
Attached Router 127.1.0.2
[ 2008/05/12 16:33:45 WARNING xorp_ospfv2:13291 OSPF area_router.cc:5759 routing_router_link_transitV2 ] LSA in database MaxAge
Network-LSA:
LS age 3600 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x2 Link State ID 10.2.5.5 Advertising Router 127.1.0.5 LS sequence number 0x80000001 LS checksum 0x3581 length 32
Network Mask 0xffffff00
Attached Router 127.1.0.5
Attached Router 127.1.0.2
# Router 4
root at a440-qc> show interfaces
created new heap in find_heap, ptr: 0x814bd10
created new heap in find_heap, ptr: 0x8150698
4.9.4/4.9.4: Flags:<ENABLED,BROADCAST,MULTICAST> mtu 1500
inet 10.4.9.4 subnet 10.4.9.0/24 broadcast 10.4.9.255
physical index 185
ether 0:93:b5:69:ca:c4
br4/br4: Flags:<ENABLED,BROADCAST,MULTICAST> mtu 1500
inet 10.4.4.4 subnet 10.4.4.0/24 broadcast 10.4.4.255
physical index 24
ether 0:cc:d8:98:fa:2b
my_discard/my_discard: Flags:<ENABLED,UNREACHABLE> mtu 0
physical index 0
root at a440-qc> show ospf4 neighbor
created new heap in find_heap, ptr: 0x811ec30
created new heap in find_heap, ptr: 0x8120580
Address Interface State ID Pri Dead
10.4.9.9 4.9.4/4.9.4 Loading 127.1.0.9 128 36
root at a440-qc>
[root at a440-qc lanforge]# tail -40 xorp-log-vr10004.txt
LS age 3600 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x2 Link State ID 10.2.5.5 Advertising Router 127.1.0.5 LS sequence number 0x80000001 LS checksum 0x3581 length 32
Network Mask 0xffffff00
Attached Router 127.1.0.5
Attached Router 127.1.0.2
[ 2008/05/12 16:29:10 WARNING xorp_ospfv2:13215 OSPF area_router.cc:5759 routing_router_link_transitV2 ] LSA in database MaxAge
Network-LSA:
LS age 3600 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x2 Link State ID 10.2.5.5 Advertising Router 127.1.0.5 LS sequence number 0x80000001 LS checksum 0x3581 length 32
Network Mask 0xffffff00
Attached Router 127.1.0.5
Attached Router 127.1.0.2
[ 2008/05/12 16:31:27 WARNING xorp_ospfv2:13215 OSPF area_router.cc:5759 routing_router_link_transitV2 ] LSA in database MaxAge
Network-LSA:
LS age 3600 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x2 Link State ID 10.2.5.5 Advertising Router 127.1.0.5 LS sequence number 0x80000001 LS checksum 0x3581 length 32
Network Mask 0xffffff00
Attached Router 127.1.0.5
Attached Router 127.1.0.2
[ 2008/05/12 16:31:30 WARNING xorp_ospfv2:13215 OSPF area_router.cc:5759 routing_router_link_transitV2 ] LSA in database MaxAge
Network-LSA:
LS age 3600 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x2 Link State ID 10.2.5.5 Advertising Router 127.1.0.5 LS sequence number 0x80000001 LS checksum 0x3581 length 32
Network Mask 0xffffff00
Attached Router 127.1.0.5
Attached Router 127.1.0.2
[ 2008/05/12 16:32:06 WARNING xorp_ospfv2:13215 OSPF area_router.cc:5759 routing_router_link_transitV2 ] LSA in database MaxAge
Network-LSA:
LS age 3600 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x2 Link State ID 10.2.5.5 Advertising Router 127.1.0.5 LS sequence number 0x80000001 LS checksum 0x3581 length 32
Network Mask 0xffffff00
Attached Router 127.1.0.5
Attached Router 127.1.0.2
[ 2008/05/12 16:32:33 WARNING xorp_ospfv2:13215 OSPF area_router.cc:5759 routing_router_link_transitV2 ] LSA in database MaxAge
Network-LSA:
LS age 3600 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x2 Link State ID 10.2.5.5 Advertising Router 127.1.0.5 LS sequence number 0x80000001 LS checksum 0x3581 length 32
Network Mask 0xffffff00
Attached Router 127.1.0.5
Attached Router 127.1.0.2
[ 2008/05/12 16:32:49 WARNING xorp_ospfv2:13215 OSPF area_router.cc:5759 routing_router_link_transitV2 ] LSA in database MaxAge
Network-LSA:
LS age 3600 Options 0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x2 Link State ID 10.2.5.5 Advertising Router 127.1.0.5 LS sequence number 0x80000001 LS checksum 0x3581 length 32
Network Mask 0xffffff00
Attached Router 127.1.0.5
Attached Router 127.1.0.2
Thanks,
Ben
--
Ben Greear <greearb at candelatech.com>
Candela Technologies Inc http://www.candelatech.com
More information about the Xorp-hackers
mailing list