09 July 2026

Getting 4× RTX 5060 Ti GPUs Working on Proxmox VE 9 (ASUS X99-E WS with PLX PCIe Switch)

Getting 4× RTX 5060 Ti GPUs Working on Proxmox VE 9 (ASUS X99-E WS with PLX PCIe Switch)

If you've tried to run NVIDIA Blackwell GPUs on an older workstation board with a PLX PCIe switch and hit cryptic GSP firmware failures — this post documents what went wrong and exactly how we fixed it.

The Setup

  • Host: Proxmox VE 9.1, kernel 6.14.11-7-pve
  • Board: ASUS X99-E WS — an older Intel LGA2011-v3 workstation board with 2× PLX PEX 8747 PCIe switch chips, enabling 4-way x16/x16/x16/x16 GPU slots
  • GPUs: 4× ASUS RTX 5060 Ti 16GB (Blackwell, GB206, PCI ID 10de:2d04)
  • Goal: Run all 4 GPUs for vLLM inference in an LXC container (64GB total VRAM)

The Problem

After installing the NVIDIA open kernel modules (required — the proprietary driver doesn't support GB206), nvidia-smi refused to show any GPU. dmesg showed the key error:

NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
NVRM: Xid (PCI:0000:06:00): 79, GPU has fallen off the bus.

And attempting modprobe nvidia caused a full kernel panic:

KERNEL PANIC! Please reboot your computer.
Timeout: Not all CPUs entered broadcast exception handler

Three sessions of diagnosis later, we had it fully working. Here's what we found.

Understanding GSP Firmware

Modern NVIDIA GPUs (Ampere onwards) use a GSP (GPU System Processor) — an ARM microcontroller inside the GPU that runs its own firmware. The CPU-side NVIDIA driver is a thin shim; all real GPU initialisation happens inside the GSP. When the driver loads, it bootstraps the GSP by uploading firmware and waiting for GSP_INIT_DONE.

The key implication: if GSP initialisation fails, the driver reports the GPU as lost. And anything that interferes with the DMA operations during that bootstrap will cause GSP to fail.

Root Cause 1: WPR2 Persistence

WPR2 (Write Protected Region 2) is a region of GPU SRAM that the GSP firmware locks during its bootstrap sequence. Critical properties:

  • WPR2 is locked when GSP begins running
  • It persists across warm reboots — the GPU stays powered via the ATX 12V rail
  • If GSP fails after locking WPR2, every subsequent driver load finds WPR2 already locked and refuses to proceed
  • Only cutting 12V power (a true cold cycle) clears WPR2

The root cause of our "WPR2 already up" loop: the nvidia module was baked into the initramfs and loading on every boot. It would attempt GSP init, fail, leave WPR2 locked, and every subsequent manual modprobe nvidia would hit the same locked state.

Fix: blacklist nvidia in /etc/modprobe.d/nvidia-blacklist.conf and rebuild initramfs:

cat > /etc/modprobe.d/nvidia-blacklist.conf << 'EOF'
blacklist nvidia
blacklist nvidia_drm
blacklist nvidia_modeset
blacklist nvidia_uvm
EOF
update-initramfs -u -k $(uname -r)

The update-initramfs step is essential — a modprobe.d blacklist alone doesn't affect modules already embedded in the initramfs image.

Root Cause 2: PCIe Completion Timeout Through PLX Switch

With WPR2 accumulation fixed, the actual GSP initialisation failure became visible. The culprit: PCIe completion timeout.

When the NVIDIA driver bootstraps GSP, the GSP ARM processor performs DMA transfers and RPC roundtrips. These operations take time — and when routed through a PLX PEX 8747 PCIe switch, they take more time than the default PCIe completion timeout (~50ms on X99 root ports).

When the timeout expires, the CPU-side driver marks the GPU as unreachable:
- CmpltTO+ flag in the GPU's PCIe AER status register
- NV_ERR_GPU_IS_LOST (0x0000000F) returned from RPC poll
- Xid 79 (GPU fallen off bus)

This was confirmed by lspci -vv on all GPUs showing UESta: CmpltTO+ after every failed init attempt.

Fix: disable PCIe completion timeout on all devices in the path — root ports, PLX bridges, and GPUs — by setting bit 4 of the PCIe Device Control 2 register (CAP_EXP+0x28):

for dev in 00:02.0 00:03.0 03:00.0 07:00.0 04:08.0 04:10.0 08:08.0 08:10.0 05:00.0 06:00.0 09:00.0 0a:00.0; do
    cur=$(setpci -s $dev CAP_EXP+0x28.w 2>/dev/null)
    [ -n "$cur" ] && setpci -s $dev CAP_EXP+0x28.w=$(printf '%04x' $((0x$cur | 0x0010)))
done

Devices in the chain:
- 00:02.0, 00:03.0 — X99 root ports
- 03:00.0, 07:00.0 — PLX PEX 8747 upstream ports
- 04:08.0, 04:10.0, 08:08.0, 08:10.0 — PLX downstream ports
- 05:00.0, 06:00.0, 09:00.0, 0a:00.0 — the 4 RTX 5060 Ti GPUs

After applying this, modprobe nvidia succeeded and all GPUs appeared in nvidia-smi.

Root Cause 3: Kernel Panic from PCIe AER → MCE Escalation

Without the completion timeout fix, the GSP failure caused a chain reaction:

  1. GSP DMA timeout → PCIe uncorrectable error
  2. Linux AER handler → escalates to Machine Check Exception
  3. MCE handler broadcasts NMI to all CPUs
  4. Some CPUs don't respond → kernel panic: "Not all CPUs entered broadcast exception handler"

Fix: add pci=noaer to kernel cmdline. This prevents AER from escalating to MCE, allowing the driver to fail gracefully instead of panicking the kernel.

Why FLR and SBR Don't Work Through PLX Switches

We tried the obvious software approaches to clear WPR2 between reboots:

  • PCIe FLR (Function Level Reset): echo 1 > /sys/bus/pci/devices/.../reset — does not propagate PERST# through the PLX PEX 8747. WPR2 remains.
  • Secondary Bus Reset (SBR): toggling bit 6 of BRIDGE_CONTROL via setpci — the PLX switch absorbs the reset signal rather than forwarding PERST# downstream. GPUs disappear from lspci during reset and re-enumerate, but WPR2 is still locked when the driver loads.

The PLX 8747 is designed for PCIe topology expansion, not to forward physical reset signals. Only a true cold power cycle (cutting ATX 12V) reliably clears WPR2.

The Bonus Issue: KVM on HDMI Causes Xid 119

After the PCIe fix, 3 of 4 GPUs came up cleanly. GPU 1 showed:

|   1  NVIDIA GeForce RTX 5060 Ti  Off | 00000000:06:00.0 N/A | ERR! ERR! ERR!  N/A/N/A |

This was not a boot failure. dmesg showed Xid 119 — a GSP RPC timeout during normal operation triggered when nvidia-smi queried the display subsystem. Root cause: a KVM switch was connected to that GPU's HDMI port. The GPU initialised fine but then got stuck on a display-related RPC call.

Fix: unplug the KVM HDMI cable from that GPU. After a cold cycle, all 4 GPUs came up clean.

Making It Persistent: systemd Service

The setpci changes are volatile — lost on every reboot. We created a systemd oneshot service to apply the fix and load the driver on every boot:

/usr/local/bin/nvidia-pcie-fix.sh:

#!/bin/bash
for dev in 00:02.0 00:03.0 03:00.0 07:00.0 04:08.0 04:10.0 08:08.0 08:10.0 05:00.0 06:00.0 09:00.0 0a:00.0; do
    cur=$(setpci -s $dev CAP_EXP+0x28.w 2>/dev/null)
    [ -n "$cur" ] && setpci -s $dev CAP_EXP+0x28.w=$(printf '%04x' $((0x$cur | 0x0010)))
done
modprobe nvidia

/etc/systemd/system/nvidia-pcie-fix.service:

[Unit]
Description=Disable PCIe completion timeout and load NVIDIA driver
After=sysinit.target systemd-udev-settle.service
Before=nvidia-persistenced.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/nvidia-pcie-fix.sh

[Install]
WantedBy=multi-user.target

systemctl enable nvidia-pcie-fix.service

Final Working State

| NVIDIA-SMI 575.64.05    Driver Version: 575.64.05    CUDA Version: 12.9     |
|   0  NVIDIA GeForce RTX 5060 Ti  Off | 00000000:05:00.0 Off |   0%  32C  P0  19W/180W |   0MiB/16311MiB |
|   1  NVIDIA GeForce RTX 5060 Ti  Off | 00000000:06:00.0 Off |   0%  33C  P0  20W/180W |   0MiB/16311MiB |
|   2  NVIDIA GeForce RTX 5060 Ti  Off | 00000000:09:00.0 Off |   0%  27C  P0  21W/180W |   0MiB/16311MiB |
|   3  NVIDIA GeForce RTX 5060 Ti  Off | 00000000:0A:00.0 Off |   0%  29C  P0  26W/180W |   0MiB/16311MiB |

64GB VRAM, all healthy.

Complete Working Configuration

Kernel cmdline (/etc/default/grub):

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_aspm=off pci=realloc,noaer"

/etc/modprobe.d/nvidia.conf:

softdep nouveau pre: nvidia
options nvidia NVreg_EnablePCIeGen3=1 NVreg_EnableGpuFirmwareLogs=1 NVreg_RegistryDwords=RmGspBootRetryAttempts=1;RMPcieFlrDevinitTimeout=4

Note: NVreg_RegistryDwords requires semicolons between key=value pairs — spaces silently fail.

/etc/modprobe.d/nvidia-blacklist.conf:

blacklist nvidia
blacklist nvidia_drm
blacklist nvidia_modeset
blacklist nvidia_uvm

Followed by update-initramfs -u -k $(uname -r).

Driver: NVIDIA open kernel modules 575.64.05 (earliest GB206-capable version available at time of writing).

Summary: What to Do If You Hit This

  1. If you see unexpected WPR2 already up: blacklist nvidia in initramfs, do a cold power cycle (12V off for 30s), then load the driver manually.
  2. If you see Xid 79 with CmpltTO+ in lspci -vv: disable PCIe completion timeout on the full root-to-GPU path via setpci CAP_EXP+0x28.
  3. If loading nvidia causes a kernel panic: add pci=noaer to the kernel cmdline.
  4. If one GPU shows ERR! in nvidia-smi: check for a display cable (HDMI/DP) connected to that GPU and disconnect it.
  5. Never do rmmod nvidia + modprobe nvidia without a cold cycle in between — rmmod re-locks WPR2.
Posted by Fixer at 10:58

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)