Proxmox Cluster Going Sluggish? Your Offline Node Has a Stale Config

You power on a node that's been offline for a while. Within seconds, the Proxmox web UI starts showing other nodes as dead. Management operations slow to a crawl. Nothing is obviously broken — all the nodes are still pinging — but something is clearly very wrong.

This is the stale corosync config problem, and it's easy to fix once you know what to look for.

What's Happening

Proxmox uses corosync to manage cluster membership. Every config change — adding a node, removing a node, changing votes — increments a config_version in /etc/corosync/corosync.conf. All cluster members must agree on this version.

When a node comes back online after missing several config changes, corosync starts up with an old config_version. The other nodes reject its packets. But corosync doesn't give up — it keeps retrying, flooding the network with rejected authentication attempts. This hammers pvedaemon on every node, causing the web UI to become sluggish and show phantom "dead" nodes even though the cluster itself is technically still quorate.

Diagnosis

First, confirm the cluster itself is still healthy from a node you trust:

pvecm status

If you see Quorate: Yes, the cluster is fine — the problem is the misbehaving node, not a genuine quorum loss. Note the Config Version value.

Then SSH into the suspect node and check:

ssh pve7 systemctl status corosync
ssh pve7 cat /etc/corosync/corosync.conf | grep config_version

Here's what it looks like when you've found the culprit:

● corosync.service - Corosync Cluster Engine
     Active: active (running) since Thu 2026-05-07 17:30:55 BST; 8min ago
   Main PID: 1146 (corosync)
     Memory: 155.9M (peak: 171.9M)
        CPU: 11.549s

May 07 17:39:36 pve7 corosync[1146]:   [KNET  ] rx: Packet rejected from 10.140.3.80:5405
May 07 17:39:37 pve7 corosync[1146]:   [KNET  ] rx: Packet rejected from 10.140.3.80:5405
May 07 17:39:38 pve7 corosync[1146]:   [KNET  ] rx: Packet rejected from 10.140.3.80:5405
May 07 17:39:43 pve7 corosync[1146]:   [QUORUM] Sync members[1]: 3
May 07 17:39:43 pve7 corosync[1146]:   [TOTEM ] A new membership (3.1a3c) was formed. Members
May 07 17:39:43 pve7 corosync[1146]:   [QUORUM] Members[1]: 3
May 07 17:39:43 pve7 corosync[1146]:   [MAIN  ] Completed service synchronization, ready to provide service.

  config_version: 10

Two red flags:
- "Packet rejected" — every node is refusing this node's traffic
- "Sync members[1]: 3" — the node has formed its own single-node pseudo-cluster with just itself (nodeid 3)
- config_version: 10 while the live cluster is at version 19 or higher

Fix

Step 1 — Stop corosync on the problem node

systemctl stop corosync

Verify it stopped:

systemctl status corosync

Expected output:

○ corosync.service - Corosync Cluster Engine
     Active: inactive (dead) since Thu 2026-05-07 17:40:45 BST; 48s ago

The web UI should recover almost immediately once the flood of rejected packets stops.

Step 2 — Check the corosync directory

ls -la /etc/corosync/

You'll likely see an authkey from the node's prior cluster membership:

drwxr-xr-x  3 root root 4096 Apr 25 17:21 .
-rw-r--r--  1 root root  256 Apr 25 16:14 authkey
-rw-r--r--  1 root root  639 Apr 25 17:21 corosync.conf

Don't manually delete it — pvecm add --force will handle it cleanly.

Step 3 — Rejoin the cluster

Run this from /tmp (pvecm refuses to run from inside /etc/pve/):

cd /tmp && pvecm add <lead-node-ip> --use_ssh --force

For example:

cd /tmp && pvecm add 10.140.3.10 --use_ssh --force

• --use_ssh — uses existing SSH key trust instead of the API password prompt
• --force — overrides warnings about existing config, authkey, and VMs (all expected for a rejoin)

You'll see output like:

detected the following error(s):
* authentication key '/etc/corosync/authkey' already exists
* cluster config '/etc/pve/corosync.conf' already exists
* this host already contains virtual guests

WARNING : detected error but forced to continue!

copy corosync auth key
stopping pve-cluster service
backup old database to '/var/lib/pve-cluster/backup/config-1778172132.sql.gz'
waiting for quorum...OK
(re)generate node files
generate new node certificate
merge authorized SSH keys
generated new node certificate, restart pveproxy and pvedaemon services
successfully added node 'pve7' to cluster.

Step 4 — Verify

pvecm status

Healthy output looks like:

Cluster information
-------------------
Name:             pve
Config Version:   20
Transport:        knet
Secure auth:      on

Quorum information
------------------
Nodes:            5
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   7
Total votes:      7
Quorum:           4
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          3 10.140.3.10
0x00000002          1 10.140.3.80
0x00000003          1 10.140.3.70 (local)
0x00000004          1 10.140.3.82
0x00000006          1 10.140.3.20

All nodes present, Quorate: Yes, config_version incremented by 2 (one increment per node during the join handshake — this is expected).

Why This Keeps Happening

Corosync is enabled by default and starts automatically on boot. If a node has been offline long enough to miss cluster config changes, it will always boot into this broken state. The node isn't malfunctioning — it's doing exactly what it's designed to do with the config it has. It's just that the config is stale.

Prevention: Before powering on a long-offline Proxmox node, check your current cluster's config_version with pvecm status. If it's significantly ahead of what the returning node last knew, plan for a rejoin rather than assuming it'll come back cleanly.

Quick Reference

Command	Purpose
pvecm status	Check cluster health and config version
systemctl status corosync	Check corosync state on a node
grep config_version /etc/corosync/corosync.conf	Check node's config version
systemctl stop corosync	Stop the misbehaving corosync
cd /tmp && pvecm add <ip> --use_ssh --force	Rejoin the cluster