I had a Proxmox server sitting on a desk at work (let's call it NERIC), no IPMI, no remote anything, and a narrow window in which I was physically next to it. The goal was simple to state and, as it turned out, a mite tricky to achieve: get NERIC permanently reachable from home, over a university network that is locked down by a corporate firewall and has no obvious route in or out.
This is the story of how that went — including the several dead ends, because the dead ends are where the actual learning lives.
A note on the clock: I was at work with maybe an hour, and my laptop — the only thing on the same wired segment as NERIC — was going home with me at the end of it. So a recurring theme below is "make the right decision given that I'm about to lose my only local access." That pressure shaped a lot of the choices.
Step 0: I couldn't even reach it
First problem: NERIC (192.168.100.2) wouldn't answer a ping or SSH from my laptop, despite both being plugged into the same physical network. ARP resolved fine, though — which is the tell.
arp -n | grep 192.168.100.2
192.168.100.2 ether 04:d4:c4:xx:xx:xx C enx1ce192b3ec51
ARP works at layer 2, so the wire is fine. ICMP and TCP timing out while ARP succeeds is the classic signature of asymmetric routing: my packets reach NERIC, but NERIC can't get its replies back. A quick tcpdump confirmed it — NERIC received each ping and immediately ARPed for its gateway (192.168.100.1) trying to route the reply, got no answer, and gave up. My laptop was sourcing from 192.168.0.143 (a /16), which isn't on NERIC's local /24, so NERIC needed a gateway that was dead.
The fix is local and reversible — give my laptop a second IP on NERIC's subnet so the replies come straight back over ARP, no gateway required:
sudo ip addr add 192.168.100.50/24 dev enx1ce192b3ec51
ssh -b 192.168.100.50 root@192.168.100.2
And I was in. (Interestingly, ICMP stayed firewalled even after this — but TCP worked, which is all I needed. Don't let a failed ping convince you the path is dead.)
Step 1: a rabbit hole — "why does NERIC have no internet?"
Inside, the picture was bleak: vmbr0 statically set to 192.168.100.2/24, gateway 192.168.100.1, and that gateway stone dead. No internet, DNS broken, nothing.
I spotted that the live router on the wire was actually 192.168.0.1 (my laptop's own gateway, healthy at ~2ms), on a different subnet to NERIC entirely. So I did the clever thing — an onlink default route so NERIC would ARP for an off-subnet gateway directly:
ip route replace default via 192.168.0.1 dev vmbr0 onlink
NERIC could now reach 192.168.0.1... and traceroute died at exactly one hop. The router took the packets and refused to forward them. I burned a little time giving NERIC a temporary in-subnet IP to make it NAT-eligible — still nothing.
Then the penny dropped, and it's embarrassing in hindsight: my laptop's own internet was coming over wifi, not that wired segment at all (the wired default route had a much higher metric). I'd never actually established that 192.168.0.1 had a path to the internet — and it didn't. The whole wired segment was an island. I'd been trying to route through a door that opened onto a wall.
Lesson re-learned: verify your assumptions about the upstream before building anything on top of them. curl to a real host beats a ping to the gateway every time.
At this point the owner of the network kindly reconfigured the router to actually serve 192.168.100.0/24 with a working gateway — but testing from both NERIC and the laptop showed the gateway answered on the LAN yet still wouldn't pass DNS or TCP to the outside. The segment simply had no usable WAN. Dead end, cleanly confirmed.
Step 2: Tailscale — and why it was never going to work here
The plan had been Tailscale. I installed it, ran tailscale up... and it sat there refusing to produce a login URL. The daemon log told me exactly why, and it's worth quoting because Tailscale ships a detector for precisely this:
tlsdial: server cert seen while dialing "controlplane.tailscale.com"
looks like "Fortinet" equipment (could be blocking Tailscale)
Received error: fetch control key: ... x509: certificate signed by unknown authority
A quick look at the certificate confirmed it:
issuer=C=US, O=Fortinet, OU=Certificate Authority, CN=<appliance-serial-redacted>
subject=CN=controlplane.tailscale.com
There's a FortiGate doing SSL deep-inspection, and it's specifically intercepting Tailscale's coordination server with its own (untrusted) certificate. login.tailscale.com and tailscale.com passed through untouched — this is targeted. Tailscale validates that control-plane cert against the system trust store, the Fortinet CA isn't in it, the handshake fails, and you never get a login URL. There is no client-side fix; only the network team can exempt it.
So Tailscale was out. Not misconfigured — blocked. Good to know definitively rather than fiddle with it for an hour.
The useful question then became: what isn't blocked? I checked the certificates on a few endpoints:
| Endpoint | Cert issuer | Verdict |
|---|---|---|
controlplane.tailscale.com |
Fortinet | intercepted |
api.cloudflare.com |
Google Trust Services | clean |
update.argotunnel.com |
Google Trust Services | clean |
github.com |
Sectigo | clean |
And region1.v2.argotunnel.com:7844 (Cloudflare's tunnel port) connected fine. Cloudflare tunnels sail straight through this firewall. I already run cloudflared for another Proxmox box at home, so the decision made itself: cloudflared, not Tailscale.
Step 3: but first, NERIC needs its own internet — enter the wifi dongle
A tunnel is no use if the box can't get online once I've left with the laptop. The owner plugged in a USB wifi dongle, and this turned into its own little saga.
It enumerated as a TP-Link "802.11ac WLAN Adapter", USB ID 2357:0120 — which a quick search identifies as a TP-Link Archer T2U Plus, chipset Realtek RTL8821AU. No driver bound, no wl* interface, and (naturally) no in-kernel support on the PVE 6.8 kernel. The mainline rtl8xxxu lists 0107/0108/...0126 but not 0120; rtw88 has the 8821cu/8822bu families but, again, not this ID.
So it's the out-of-tree 8821au DKMS driver (morrownr's maintained fork). That needs kernel headers, which on a no-subscription PVE host means adding the no-subscription repo to get pve-headers-6.8.12-4-pve. Build tools and the driver went in fine:
git clone https://github.com/morrownr/8821au-20210708.git
cd 8821au-20210708 && ./install-driver.sh NoPrompt
DKMS reported rtl8821au/5.12.5.2 ... installed and the module loaded — but still no interface. The reason was entirely self-inflicted: while experimenting earlier I'd modprobe'd rtw88_8822bu and it had grabbed the device via a runtime new_id override, so the correct driver couldn't claim it. Unbinding it hung — the unbind write blocked in uninterruptible (D) state, which is about as stuck as Linux gets.
A reboot was the clean way out. Except the reboot also hung — on the very same wedged process. The console showed systemd-shutdown waiting 90 seconds for it, failing to remount root read-only ("Device or resource busy"), and finally forcing the reboot. A hard power-cycle finished the job. (No data loss — ext4 journalled-recovered cleanly on the way back up.)
On the next boot, with the conflicting drivers blacklisted:
# /etc/modprobe.d/neric-wifi-dongle.conf
blacklist rtw88_8822bu
blacklist rtw88_8821cu
blacklist rtl8xxxu
...the right driver won, and wlxXXXXXXXXXXXX appeared. Lesson: when you've been poking at new_id overrides, a reboot clears the runtime cruft and a blacklist makes the result stick.
Step 4: the captive portal — authenticating a headless server
The dongle associated to the open WiFi-Guest SSID and got a DHCP lease immediately. But any HTTP request bounced to a familiar wall:
Location: https://wifi.sky.com/login?client_mac=XXXXXXXXXXXX&reason=Un-Auth-Captive...
It's Sky WiFi / "The Cloud" — a per-MAC captive portal, which is precisely the wrong thing to put in front of a headless server. There's no browser to click "Continue", and the portal authorises the MAC that hits it — so I couldn't just log in from my laptop's browser, because my laptop was on a different network (eduroam, as it happens) with a different MAC.
The trick that solved it is the bit I'm most pleased with. I turned NERIC into a SOCKS proxy over the SSH connection I already had:
ssh -D 1080 -N -b 192.168.100.50 root@192.168.100.2
The SSH control channel rides the stable wired path; anything sent into that proxy is originated by NERIC and exits its wifi connection. So I pointed a headless Playwright browser at the proxy, and from the portal's point of view it was NERIC knocking — correct MAC, correct session. I accepted the terms, registered a Cloud account against a disposable mailbox, signed in, and:
(The throwaway address, in case it's handy: neric1780499778@wshu.net, from the rather useful mail.tm. It has an API, which is what makes it scriptable for catching a verification email — though The Cloud didn't end up demanding one. It'll lapse before long, so do generate your own rather than relying on mine.)
generate_204 -> 204 (no longer walled)
github.com -> 200
api.cloudflare.com -> 200
We were online. (The authorisation lives at the gateway keyed to NERIC's MAC, so once done I could tear the browser and proxy down — the box stays connected on its own.)
Step 5: making it survive a reboot (and me leaving)
A working setup that evaporates on reboot is not a setup. Two services make this persistent.
A wifi watchdog (neric-wifi.service) that re-associates, re-DHCPs, and — crucially — forces the default route over wifi, overriding the dead wired gateway that /etc/network/interfaces still installs at boot. I deliberately did not edit interfaces (no appetite for breaking PVE's bridge config remotely); the watchdog just out-prioritises the dead route at runtime, every boot, and re-checks every 30 seconds.
cloudflared, installed as a service, publishing two hostnames off my existing tunnel zone:
ingress:
- hostname: neric-pve.example.com # SSH
service: ssh://localhost:22
- hostname: neric-pve-ui.example.com # Proxmox web UI
service: https://localhost:8006
originRequest:
noTLSVerify: true
disableChunkedEncoding: true
- service: http_status:404
(noTLSVerify and disableChunkedEncoding because Proxmox uses a self-signed cert internally and doesn't play nicely with chunked encoding through a tunnel — learned that on the previous box.)
And because The Cloud's session can eventually expire, the watchdog calls a headless Playwright re-auth script on NERIC when it detects a walled state — replaying the same "accept terms, sign in" flow that worked manually.
Does it actually work?
Tested from the public internet, with the laptop's scaffolding (NAT, SOCKS, the IP alias) all torn down — so this is genuinely NERIC standing on its own:
| Test | Result |
|---|---|
Internet on NERIC (generate_204) |
204 |
| Default route | via 10.223.88.1 dev wlxXXXXXXXXXXXX (wifi) |
neric-wifi.service |
active, enabled |
cloudflared.service |
active, enabled (4 QUIC conns to LHR edge) |
SSH via cloudflared access ssh --hostname neric-pve.example.com |
returns hostname pve ✅ |
PVE UI https://neric-pve-ui.example.com |
HTTP 200, "pve - Proxmox Virtual Environment" ✅ |
| Re-auth script dry run | already online, exit 0 ✅ |
So: a Proxmox box, online via a USB wifi dongle, behind a captive portal, past a FortiGate that blocks Tailscale — and reachable from anywhere over SSH and the web UI. Et voilà.
A word on decisions under time pressure
A few calls were made specifically because the clock was running:
- Pre-stage over perfect. I got cloudflared up and tested before the laptop left, because cloudflared is my only way back in afterwards. Everything else can be refined remotely.
- Watchdog over editing
interfaces. Forcing the route at runtime is uglier than a clean config edit, but it can't brick the host's networking — and I wouldn't be there to fix it if it did. - Re-auth on the host for now. The neater home for the headless browser is an LXC (keep the hypervisor clean), but downloading a template and wiring its networking through the host's wifi was too much to finish in the window. Host now, container later — over the tunnel.
Not the choices I'd make with unlimited time, but the right ones for the time I had.
Next steps
- Migrate the portal re-auth browser into an LXC and strip Playwright/Chromium off the hypervisor.
- Tidy the now-unused DNS record left over from a rename.
- The dormant Tailscale install stays put — it costs nothing, and if the network team ever exempts
*.tailscale.comfrom inspection, it'll just connect.
If you're ever trying to do something similar — a headless box on a hostile corporate network — the two ideas worth stealing are the SOCKS-proxy-through-the-server trick for satisfying a per-MAC captive portal, and checking certificate issuers to find out what your firewall is and isn't tampering with before you commit to a tunnelling strategy.
Ta ta for now, and I hope you found this helpful.
No comments:
Post a Comment