[Xorp-users] Error writing to routing socket: No such process

Edwin Schokkenbroek edwin at spacebugs.nl
Wed Jul 28 13:12:40 PDT 2010


On Jul 28, 2010, at 9:54 PM, Ben Greear wrote:

> On 07/28/2010 12:44 PM, Edwin Schokkenbroek wrote:
>> 
>> On Jul 28, 2010, at 9:35 PM, Ben Greear wrote:
>> 
>>> On 07/28/2010 12:18 PM, Edwin Schokkenbroek wrote:
>>>> Hello Ben,
>>>> 
>>>> Thank you for your quick response. I already started to experiment with xopr.ct. I ran into it when trying to solve the errors mentioned earlier.
>>>> 
>>>> 010/07/28 21:07:59  ERROR xorpsh:1807 XRL +468 xrl_router.cc get_sender ] Could not create XrlPFSender for protocol = "unix" address = ":var:tmp:xrl.POm5vR"
>>>> [ 2010/07/28 21:07:59  ERROR xorpsh:1807 LIBCOMM +831 comm_user.c comm_connect_unix ] Error connecting to unix socket.  Path: /var/tmp/xrl.cwKcRV.  Error: Permission denied
>>>> [ 2010/07/28 21:07:59  ERROR xorpsh:1807 XRL +63 xrl_pf_factory.cc create_sender ] XrlPFSenderFactory::create failed: XrlPFConstructorError from line 131 of xrl_pf_unix.cc: Could not connect to /var/tmp/xrl.cwKcRV
>>>> 
>>>> (These messages are going on forever.)
>>>> 
>>>> The message is clear, and easy to explain,  xorp_rtrmgr is started as the root user. However I start the xorpsh as a other user with less privileges.
>>>> When I change the ownership of the file in /var/tmp I'm able to login.
>>>> 
>>>> But when I reboot the machine or I start another vimage (or jail with vnet ) I ran into the same problem again. I guess xorp_rtrmgr is creating the files as root.
>>>> 
>>>> Since the messages appear very quickly I'm not able to see the messages when xorp_rtrmgr is started.
>>> 
>>> I could add an option to create the /var/tmp/ file as read/write by everyone (or maybe
>>> just the xorp group), I suppose.
>> 
>> I think a xorp group is the most clean way . read/write everyone could maybe introduce a security issue ?
>> 
>> The dir /var/tmp is already world writeable permissions (on my system):
>> 
>> drwxrwxrwt  3 beheer  wheel  2048 Jul 28 21:27 /var/tmp/
>> 
>>> 
>>> One thing I just noticed:  I have 126,000 or so xrl files in /var/tmp/
>>> on one of my test machines.
>>> 
>>> That can't be good :P
>> 
>> Nope....  hopefully the filesystem has enough inodes ;-)
>>> 
>>> Aside from the permissions issue, does xorp.ct fix that other bug you reported?
>> 
>> Well I managed to start one xopr_rtmgr in a vimage,  however the other xorp_rtr_mgr seems to hang.
>> 
>> I recieve a lot of messages :
> 
> The messages look OK to me (some seem scary, but I've been seeing them for years
> and they don't seem to cause harm, so I haven't bothered to track them down yet).
> 
> The one issue that might be a problem is the timer expiry warnings, but if the
> system is slow/virtual, that might be normal and it will probably work better
> once all the processes are started.

The FreeBSD system is running inside a virtual machine (vmware fusion). 

> 
> I don't know anything about vimages, but you might have to do some tricks
> to get two xorps to run on the same OS at the same time.  If the vimage looks
> like a separate machine (like vmware, xemu, etc), then it should be OK.

I don't know if you know jails on FreeBSD ? These "vimages" are jails , and they have a own ip stack. 
(For more info see : http://imunes.tel.fer.hr/virtnet/ )

I'm playing around with it since it appeared in FreeBSD 7. And I even use it in a production environment (by a dutch ISP)

> 
> For the process that hangs, what makes you think so?  Can you run 'strace' or
> whatever the BSD equiv is to see if it's really hung in a system call, or
> 'top' to see if it's spinning 100% CPU?

I just figured it out. the xorp_rtrmgr died, and the xorpsh was just waiting. So I killed the xoprsh and restarted xorp_rtrmgr. 
On the other xorp_rtrmgr I see the following messages appear: 

[ 2010/07/28 21:52:25.981458  ERROR xorp_fea:2274 FEA fea/data_plane/fibconfig/fibconfig_entry_set_routing_socket.cc:527 delete_entry ] Error writing to routing socket: No such process
[ 2010/07/28 21:52:25.983762  ERROR xorp_fea:2274 FEA fea/fibconfig_transaction.cc:80 operation_result ] FIB transaction commit failed on DeleteEntry4: net = 10.0.0.0/24 nexthop = 10.2.0.1 ifname = epair2b vifname = epair2b metric = 2 admin_distance = 110 xorp_route = true is_deleted = false is_unresolved = false is_connected_route = false
[ 2010/07/28 21:52:25.984558  WARNING xorp_fea:2274 XrlFeaTarget obj/x86_64-unknown-freebsd8.0/xrl/targets/fea_base.cc:3978 handle_redist_transaction4_0_1_commit_transaction ] Handling method for redist_transaction4/0.1/commit_transaction failed: XrlCmdError 102 Command failed DeleteEntry4: net = 10.0.0.0/24 nexthop = 10.2.0.1 ifname = epair2b vifname = epair2b metric = 2 admin_distance = 110 xorp_route = true is_deleted = false is_unresolved = false is_connected_route = false
[ 2010/07/28 21:52:25.985962  ERROR xorp_rib:2275 RIB rib/redist_xrl.cc:948 dispatch_complete ] Failed to commit transaction: 102 Command failed DeleteEntry4: net = 10.0.0.0/24 nexthop = 10.2.0.1 ifname = epair2b vifname = epair2b metric = 2 admin_distance = 110 xorp_route = true is_deleted = false is_unresolved = false is_connected_route = false



I started the xorp_rtrmgr as : 

 /usr/local/xorp/sbin/xorp_rtrmgr -m /usr/local/xorp/lib/xorp/sbin/ -b /root/chaos/vrd/edwin/router1/boot/config.boot -d


And I received the messages: 

Not doing pidfile...
[ 2010/07/28 21:51:54.616952  INFO xorp_rtrmgr:2330 RTRMGR rtrmgr/master_conf_tree.cc:234 execute ] Changed modules: interfaces, rtrmgr, firewall, fea, rib, policy, static_routes, ospf4
[ 2010/07/28 21:51:54.619137  WARNING xorp_rtrmgr:2330 RTRMGR rtrmgr/module_command.cc:327 startup_validation ] WARNING:  Using DelayValidation, module_name: rtrmgr
[ 2010/07/28 21:51:54.619200  WARNING xorp_rtrmgr:2330 RTRMGR rtrmgr/module_command.cc:348 config_validation ] WARNING:  Using DelayValidation, module_name: rtrmgr
[ 2010/07/28 21:51:54.620622  WARNING xorp_rtrmgr:2330 RTRMGR rtrmgr/module_command.cc:369 ready_validation ] WARNING:  Using DelayValidation, module_name: rtrmgr
[ 2010/07/28 21:51:54.620770  INFO xorp_rtrmgr:2330 RTRMGR rtrmgr/module_manager.cc:94 execute ] Executing module: interfaces (xorp_fea)
[ 2010/07/28 21:51:54.625817  WARNING xorp_rtrmgr:2330 XrlFinderTarget obj/x86_64-unknown-freebsd8.0/xrl/targets/finder_base.cc:482 handle_finder_0_2_resolve_xrl ] Handling method for finder/0.2/resolve_xrl failed: XrlCmdError 102 Command failed Target "fea" does not exist or is not enabled.
[ 2010/07/28 21:51:54.626844  WARNING xorp_rtrmgr:2330 RTRMGR rtrmgr/task.cc:215 xrl_done ] Failed to receive reply, code: 201 Resolve failed  retries: 0  max_retries: 30
[ 2010/07/28 21:51:55.656722  INFO xorp_fea:2331 MFEA fea/mfea_node.cc:317 enable ] MFEA enabled
[ 2010/07/28 21:51:55.657260  INFO xorp_fea:2331 MFEA fea/mfea_node_cli.cc:126 enable ] CLI enabled
[ 2010/07/28 21:51:55.659055  INFO xorp_fea:2331 MFEA fea/mfea_node_cli.cc:92 start ] CLI started
[ 2010/07/28 21:51:55.659268  INFO xorp_fea:2331 MFEA fea/mfea_node.cc:317 enable ] MFEA enabled
[ 2010/07/28 21:51:55.659364  INFO xorp_fea:2331 MFEA fea/mfea_node_cli.cc:126 enable ] CLI enabled
[ 2010/07/28 21:51:55.659809  INFO xorp_fea:2331 MFEA fea/mfea_node_cli.cc:92 start ] CLI started
[ 2010/07/28 21:51:55.695317  INFO xorp_rtrmgr:2330 RTRMGR rtrmgr/module_manager.cc:94 execute ] Executing module: rtrmgr ()
[ 2010/07/28 21:52:01.748221  INFO xorp_rtrmgr:2330 RTRMGR rtrmgr/module_manager.cc:94 execute ] Executing module: firewall (xorp_fea)
[ 2010/07/28 21:52:01.750650  INFO xorp_rtrmgr:2330 RTRMGR rtrmgr/module_manager.cc:94 execute ] Executing module: fea (xorp_fea)
[ 2010/07/28 21:52:01.752797  INFO xorp_rtrmgr:2330 RTRMGR rtrmgr/module_manager.cc:94 execute ] Executing module: rib (xorp_rib)
[ 2010/07/28 21:52:01.755686  WARNING xorp_rtrmgr:2330 XrlFinderTarget obj/x86_64-unknown-freebsd8.0/xrl/targets/finder_base.cc:482 handle_finder_0_2_resolve_xrl ] Handling method for finder/0.2/resolve_xrl failed: XrlCmdError 102 Command failed Target "rib" does not exist or is not enabled.
[ 2010/07/28 21:52:01.756492  WARNING xorp_rtrmgr:2330 RTRMGR rtrmgr/task.cc:215 xrl_done ] Failed to receive reply, code: 201 Resolve failed  retries: 0  max_retries: 30
[ 2010/07/28 21:52:02.795582  INFO xorp_rtrmgr:2330 RTRMGR rtrmgr/module_manager.cc:94 execute ] Executing module: policy (xorp_policy)
[ 2010/07/28 21:52:02.798833  WARNING xorp_rtrmgr:2330 XrlFinderTarget obj/x86_64-unknown-freebsd8.0/xrl/targets/finder_base.cc:482 handle_finder_0_2_resolve_xrl ] Handling method for finder/0.2/resolve_xrl failed: XrlCmdError 102 Command failed Target "policy" does not exist or is not enabled.
[ 2010/07/28 21:52:02.799463  WARNING xorp_rtrmgr:2330 RTRMGR rtrmgr/task.cc:215 xrl_done ] Failed to receive reply, code: 201 Resolve failed  retries: 0  max_retries: 30
[ 2010/07/28 21:52:03.862706  INFO xorp_rtrmgr:2330 RTRMGR rtrmgr/module_manager.cc:94 execute ] Executing module: static_routes (xorp_static_routes)
[ 2010/07/28 21:52:03.868031  WARNING xorp_rtrmgr:2330 XrlFinderTarget obj/x86_64-unknown-freebsd8.0/xrl/targets/finder_base.cc:482 handle_finder_0_2_resolve_xrl ] Handling method for finder/0.2/resolve_xrl failed: XrlCmdError 102 Command failed Target "static_routes" does not exist or is not enabled.
[ 2010/07/28 21:52:03.868473  WARNING xorp_rtrmgr:2330 RTRMGR rtrmgr/task.cc:215 xrl_done ] Failed to receive reply, code: 201 Resolve failed  retries: 0  max_retries: 30
[ 2010/07/28 21:52:04.903984  INFO xorp_rtrmgr:2330 RTRMGR rtrmgr/module_manager.cc:94 execute ] Executing module: ospf4 (xorp_ospfv2)
[ 2010/07/28 21:52:04.907514  WARNING xorp_rtrmgr:2330 XrlFinderTarget obj/x86_64-unknown-freebsd8.0/xrl/targets/finder_base.cc:482 handle_finder_0_2_resolve_xrl ] Handling method for finder/0.2/resolve_xrl failed: XrlCmdError 102 Command failed Target "ospfv2" does not exist or is not enabled.
[ 2010/07/28 21:52:04.908267  WARNING xorp_rtrmgr:2330 RTRMGR rtrmgr/task.cc:215 xrl_done ] Failed to receive reply, code: 201 Resolve failed  retries: 0  max_retries: 30
[ 2010/07/28 21:52:05.182318  WARNING xorp_ospfv2:2335 OSPF ospf/xrl_io.cc:948 updates_made ] XrlIO<IPv4>::updates_made, _iftree:
[ 2010/07/28 21:52:05.952538  WARNING xorp_ospfv2:2335 OSPF ospf/peer.cc:294 set_link_status ] Setting PeerOut link status to: 1  dbg: create_peer  vif: epair1a/epair1a  old-status: 0
[ 2010/07/28 21:52:05.952667  WARNING xorp_ospfv2:2335 OSPF ospf/peer.cc:304 peer_change ] PeerOut, peer_change on interface: epair1a/epair1a  running: 0  status: 0  link-status: 1
[ 2010/07/28 21:52:05.961775  WARNING xorp_ospfv2:2335 OSPF ospf/peer.cc:304 peer_change ] PeerOut, peer_change on interface: epair1a/epair1a  running: 0  status: 1  link-status: 1
[ 2010/07/28 21:52:05.961951  WARNING xorp_ospfv2:2335 OSPF ospf/xrl_io.cc:170 enable_interface_vif ] XRL-IO: Enable Interface epair1a Vif epair1a
[ 2010/07/28 21:52:05.963574  WARNING xorp_ospfv2:2335 OSPF ospf/peer.cc:294 set_link_status ] Setting PeerOut link status to: 1  dbg: create_peer  vif: epair3a/epair3a  old-status: 0
[ 2010/07/28 21:52:05.963602  WARNING xorp_ospfv2:2335 OSPF ospf/peer.cc:304 peer_change ] PeerOut, peer_change on interface: epair3a/epair3a  running: 0  status: 0  link-status: 1
[ 2010/07/28 21:52:05.973755  WARNING xorp_fea:2331 FEA fea/data_plane/io/io_ip_socket.cc:230 IoIpSocket ] Registering with iftree: pushed-config
[ 2010/07/28 21:52:05.976051  INFO xorp_fea:2331 FEA fea/data_plane/io/io_ip_socket.cc:523 join_multicast_group ] Joined IPv4 group: 224.0.0.5 on interface epair1a vif epair1a  socket: 45
[ 2010/07/28 21:52:05.976685  WARNING xorp_ospfv2:2335 OSPF ospf/peer.cc:304 peer_change ] PeerOut, peer_change on interface: epair3a/epair3a  running: 0  status: 1  link-status: 1
[ 2010/07/28 21:52:05.976928  WARNING xorp_ospfv2:2335 OSPF ospf/xrl_io.cc:170 enable_interface_vif ] XRL-IO: Enable Interface epair3a Vif epair3a
[ 2010/07/28 21:52:05.978695  INFO xorp_fea:2331 FEA fea/data_plane/io/io_ip_socket.cc:523 join_multicast_group ] Joined IPv4 group: 224.0.0.5 on interface epair3a vif epair3a  socket: 45
[ 2010/07/28 21:52:05.985697  INFO xorp_rtrmgr:2330 RTRMGR rtrmgr/task.cc:2242 run_task ] No more tasks to run
[root at router1 /]# [ 2010/07/28 21:52:07.70452  ERROR xorp_fea:2331 FEA fea/data_plane/fibconfig/fibconfig_entry_set_routing_socket.cc:527 delete_entry ] Error writing to routing socket: No such process
[ 2010/07/28 21:52:07.70660  ERROR xorp_fea:2331 FEA fea/fibconfig_transaction.cc:80 operation_result ] FIB transaction commit failed on DeleteEntry4: net = 0.0.0.0/0 nexthop = 172.19.0.2 ifname = epair4a vifname = epair4a metric = 1 admin_distance = 1 xorp_route = true is_deleted = false is_unresolved = false is_connected_route = false
[ 2010/07/28 21:52:07.70947  WARNING xorp_fea:2331 XrlFeaTarget obj/x86_64-unknown-freebsd8.0/xrl/targets/fea_base.cc:3978 handle_redist_transaction4_0_1_commit_transaction ] Handling method for redist_transaction4/0.1/commit_transaction failed: XrlCmdError 102 Command failed DeleteEntry4: net = 0.0.0.0/0 nexthop = 172.19.0.2 ifname = epair4a vifname = epair4a metric = 1 admin_distance = 1 xorp_route = true is_deleted = false is_unresolved = false is_connected_route = false
[ 2010/07/28 21:52:07.71468  ERROR xorp_rib:2332 RIB rib/redist_xrl.cc:948 dispatch_complete ] Failed to commit transaction: 102 Command failed DeleteEntry4: net = 0.0.0.0/0 nexthop = 172.19.0.2 ifname = epair4a vifname = epair4a metric = 1 admin_distance = 1 xorp_route = true is_deleted = false is_unresolved = false is_connected_route = false

[root at router1 /]# [ 2010/07/28 21:52:10.41438  ERROR xorp_fea:2331 FEA fea/data_plane/fibconfig/fibconfig_entry_set_routing_socket.cc:527 delete_entry ] Error writing to routing socket: No such process
[ 2010/07/28 21:52:10.41706  ERROR xorp_fea:2331 FEA fea/fibconfig_transaction.cc:80 operation_result ] FIB transaction commit failed on DeleteEntry4: net = 0.0.0.0/0 nexthop = 172.19.0.2 ifname = epair4a vifname = epair4a metric = 1 admin_distance = 1 xorp_route = true is_deleted = false is_unresolved = false is_connected_route = false
[ 2010/07/28 21:52:10.41868  WARNING xorp_fea:2331 XrlFeaTarget obj/x86_64-unknown-freebsd8.0/xrl/targets/fea_base.cc:3978 handle_redist_transaction4_0_1_commit_transaction ] Handling method for redist_transaction4/0.1/commit_transaction failed: XrlCmdError 102 Command failed DeleteEntry4: net = 0.0.0.0/0 nexthop = 172.19.0.2 ifname = epair4a vifname = epair4a metric = 1 admin_distance = 1 xorp_route = true is_deleted = false is_unresolved = false is_connected_route = false
[ 2010/07/28 21:52:10.42125  ERROR xorp_rib:2332 RIB rib/redist_xrl.cc:948 dispatch_complete ] Failed to commit transaction: 102 Command failed DeleteEntry4: net = 0.0.0.0/0 nexthop = 172.19.0.2 ifname = epair4a vifname = epair4a metric = 1 admin_distance = 1 xorp_route = true is_deleted = false is_unresolved = false is_connected_route = false
[ 2010/07/28 21:52:14.967225  INFO xorp_fea:2331 FEA fea/data_plane/io/io_ip_socket.cc:523 join_multicast_group ] Joined IPv4 group: 224.0.0.6 on interface epair1a vif epair1a  socket: 45
[ 2010/07/28 21:52:15.301767  INFO xorp_fea:2331 FEA fea/data_plane/io/io_ip_socket.cc:523 join_multicast_group ] Joined IPv4 group: 224.0.0.6 on interface epair3a vif epair3a  socket: 45

[root at router1 /]# [ 2010/07/28 21:52:20.2743 TRACE xorp_ospfv2 OSPF ] Rejecting LSA last one arrived less than 1 second(s) ago
Router-LSA:
LS age    1 Options  0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x1 Link State ID 10.1.0.2 Advertising Router 10.1.0.2 LS sequence number 0x80000002 LS checksum 0x6e84 length 48
        bit Nt false
        bit V false
        bit E false
        bit B false
        Type 2 Transit network IP address of Designated router 10.1.0.2 Routers interface address 10.1.0.2 Metric 1
        Type 2 Transit network IP address of Designated router 10.2.0.1 Routers interface address 10.2.0.2 Metric 1
Router-LSA:
LS age 1481 Options  0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x1 Link State ID 10.1.0.2 Advertising Router 10.1.0.2 LS sequence number 0x80000001 LS checksum 0xd42d length 48
        bit Nt false
        bit V false
        bit E false
        bit B false
        Type 3 Stub network Subnet number 10.1.0.0 Mask 255.255.255.0 Metric 1
        Type 2 Transit network IP address of Designated router 10.2.0.1 Routers interface address 10.2.0.2 Metric 1
[ 2010/07/28 21:52:20.4661 TRACE xorp_ospfv2 OSPF ] Rejecting LSA last one arrived less than 1 second(s) ago
Router-LSA:
LS age    2 Options  0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x1 Link State ID 10.1.0.2 Advertising Router 10.1.0.2 LS sequence number 0x80000002 LS checksum 0x6e84 length 48
        bit Nt false
        bit V false
        bit E false
        bit B false
        Type 2 Transit network IP address of Designated router 10.1.0.2 Routers interface address 10.1.0.2 Metric 1
        Type 2 Transit network IP address of Designated router 10.2.0.1 Routers interface address 10.2.0.2 Metric 1
Router-LSA:
LS age 1481 Options  0x2 DC: 0 EA: 0 N/P: 0 MC: 0 E: 1 LS type 0x1 Link State ID 10.1.0.2 Advertising Router 10.1.0.2 LS sequence number 0x80000001 LS checksum 0xd42d length 48
        bit Nt false
        bit V false
        bit E false
        bit B false
        Type 3 Stub network Subnet number 10.1.0.0 Mask 255.255.255.0 Metric 1
        Type 2 Transit network IP address of Designated router 10.2.0.1 Routers interface address 10.2.0.2 Metric 1



Tomorrow I'm going to reboot the whole machine and going to do it step by step.... To make sure I don't make stupid mistakes. 

Edwin


> 
> Thanks,
> Ben
> 
> 
> -- 
> Ben Greear <greearb at candelatech.com>
> Candela Technologies Inc  http://www.candelatech.com
> 


Edwin Schokkenbroek
edwin at spacebugs.nl

Engineers are full of great ideas. Making things better is what we're built to do. However, when it comes to engineers, there seems to be an inverse relationship between great ideas and great communication skills. This is what gets us into trouble.









More information about the Xorp-users mailing list