[已解决]Proxmox下丢失集群pveproxy[20850]: ipcc_send_rec[1] failed: Connection refused

问题描述

同之前的问题一样，但是之前的解决方案不再适用：

[已解决]Failed to start The Proxmox VE cluster filesystem | /etc/pve无法访问 | 集群丢失节点

文章目录[隐藏] 问题描述 排查 解决方案 问题描述 Proxmox集群单节点丢失，能够ping通，但是ssh通过key连接时卡住，使用密码正常。 使用ssh -vvv查看发现在验证key的时候无任何反馈，表明故障节点存public key的地方出现了问题。 排查 Proxmox存储集群public key的地方在/etc/pve，尝试cd进入失败。 /etc/pve路径使用的是corosync进行同步，查看状态无异常，能看到5个pve节点： root@PVE-EU-2 ~ # systemctl status corosync ● corosync.service - Corosync Cluster Engine Loaded: loaded (/lib/systemd/system/corosync.service; enabled; vendor preset: enabled Active: active (running) since Sun 2020-02-23 11:06:37 CET; 2min 55s ago Docs: man:corosync man:corosync.conf man:corosync_overview Main PID: 23596 (corosync) Tasks: 9 (limit: 4915) […]

root@PVE-EU-2 ~ # systemctl status pve-cluster pveproxy pvedaemon
● pve-cluster.service - The Proxmox VE cluster filesystem
   Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2020-02-24 09:59:19 CET; 2min 26s ago
  Process: 15018 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION)

Feb 24 09:59:19 PVE-EU-2 systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Feb 24 09:59:19 PVE-EU-2 systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Feb 24 09:59:19 PVE-EU-2 systemd[1]: Stopped The Proxmox VE cluster filesystem.
Feb 24 09:59:19 PVE-EU-2 systemd[1]: pve-cluster.service: Start request repeated too quickly.
Feb 24 09:59:19 PVE-EU-2 systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 24 09:59:19 PVE-EU-2 systemd[1]: Failed to start The Proxmox VE cluster filesystem.

● pveproxy.service - PVE API Proxy Server
   Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2020-01-26 09:29:32 CET; 4 weeks 1 days ago
  Process: 7829 ExecReload=/usr/bin/pveproxy restart (code=exited, status=0/SUCCESS)
 Main PID: 24944 (pveproxy)
    Tasks: 4 (limit: 4915)
   Memory: 169.4M
   CGroup: /system.slice/pveproxy.service
           ├─15701 pveproxy worker
           ├─20850 pveproxy worker
           ├─21560 pveproxy worker
           └─24944 pveproxy

Feb 24 09:56:08 PVE-EU-2 pveproxy[20850]: ipcc_send_rec[3] failed: Connection refused
Feb 24 10:00:07 PVE-EU-2 pveproxy[20850]: ipcc_send_rec[1] failed: Connection refused
Feb 24 10:00:07 PVE-EU-2 pveproxy[20850]: ipcc_send_rec[2] failed: Connection refused
Feb 24 10:00:07 PVE-EU-2 pveproxy[20850]: ipcc_send_rec[3] failed: Connection refused
Feb 24 10:01:03 PVE-EU-2 pveproxy[15701]: ipcc_send_rec[1] failed: Connection refused
Feb 24 10:01:03 PVE-EU-2 pveproxy[15701]: ipcc_send_rec[2] failed: Connection refused
Feb 24 10:01:03 PVE-EU-2 pveproxy[15701]: ipcc_send_rec[3] failed: Connection refused
Feb 24 10:01:10 PVE-EU-2 pveproxy[20850]: ipcc_send_rec[1] failed: Connection refused
Feb 24 10:01:10 PVE-EU-2 pveproxy[20850]: ipcc_send_rec[2] failed: Connection refused
Feb 24 10:01:10 PVE-EU-2 pveproxy[20850]: ipcc_send_rec[3] failed: Connection refused
● pvedaemon.service - PVE API Daemon
   Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2020-01-26 09:29:18 CET; 4 weeks 1 days ago
 Main PID: 24202 (pvedaemon)
    Tasks: 4 (limit: 4915)
   Memory: 147.7M
   CGroup: /system.slice/pvedaemon.service
           ├─10292 pvedaemon worker
           ├─17254 pvedaemon worker
           ├─24202 pvedaemon
           └─31808 pvedaemon worker

Feb 24 05:33:09 PVE-EU-2 pvedaemon[17254]: <root@pam> update VM 1563: -cpu kvm64
Feb 24 05:33:09 PVE-EU-2 pvedaemon[10292]: <root@pam> successful auth for user 'root@pam'
Feb 24 05:33:09 PVE-EU-2 pvedaemon[17254]: <root@pam> update VM 1563: -cores 1
Feb 24 05:33:09 PVE-EU-2 pvedaemon[10292]: <root@pam> update VM 1563: -boot dca
Feb 24 05:33:09 PVE-EU-2 pvedaemon[31808]: <root@pam> update VM 1563: -numa 0
Feb 24 05:33:09 PVE-EU-2 pvedaemon[31808]: <root@pam> update VM 1563: -onboot 1
Feb 24 05:33:09 PVE-EU-2 pvedaemon[17254]: <root@pam> update VM 1563: -name CR-200223-K41S.com
Feb 24 05:33:09 PVE-EU-2 pvedaemon[10292]: <root@pam> update VM 1563: -net0 rtl8139=00:16:3e:f8:8e:ae,bridge=vmbr1
Feb 24 05:33:09 PVE-EU-2 pvedaemon[10292]: <root@pam> update VM 1563: -bootdisk ide0
Feb 24 05:33:09 PVE-EU-2 pvedaemon[17254]: <root@pam> update VM 1563: -boot dca -bootdisk ide0

解决方案

重启pve服务

systemctl restart pveproxy pvedaemon
service pve-cluster restart