I have a Proxmox server at the University of Salford — a box with an RTX 4090 and a 7B LLM already loaded, sitting in a room I can't get to outside working hours. Campus blocks Tailscale (FortiGate does SSL inspection on controlplane.tailscale.com). No IPMI, no KVM-over-IP. The only remote control is whatever you set up before the door closes.
The goal: get that GPU reachable from home as a real API endpoint — something LiteLLM, Open WebUI, and Claude Code can all hit as casually as a local Ollama instance. http://claude-hub.lan:11434, backed by a Salford GPU, no 1.33 GB binary installs on the home side.
It took longer than expected.
The setup
The university LAN (NERIC) is an isolated 192.168.100.0/24 segment with two Proxmox nodes. The main one (neric-pve, .2) has a Cloudflare tunnel and runs a captive portal reauthentication script. The second (NERIC-4090, .10) boots from a USB stick and hosts three LXC containers:
- CT100 —
ollama: Ollama 0.30.10 on192.168.100.50:11434, plus a vLLM instance serving Qwen3.6-27B-AWQ on port 8000. - CT101 — Ubuntu 24.04, general purpose.
- CT102 — Alpine 3.22, running
cloudflaredfor a dedicated tunnel.
Back home, claude-hub (Proxmox CT107, 10.140.3.202) is an always-on LXC I use for persistent Claude Code sessions. Everything on the home LAN reaches it at claude-hub.lan.
The architecture I ended up with:
Home LAN tools, LiteLLM, Open WebUI
|
| http://claude-hub.lan:11434
▼
[claude-hub, 10.140.3.202]
neric-ollama-forward.service
ssh -N -L 0.0.0.0:11434:192.168.100.50:11434 neric-4090
|
| Cloudflare Access SSH proxy (service token, non-interactive)
▼
[Cloudflare edge — neric-4090 tunnel]
|
▼
[NERIC-4090 host, 192.168.100.10]
|
▼
[CT100 — Ollama, 192.168.100.50:11434]
What I tried first — and why it didn't work
The HTTP Ollama tunnel
CT102 already exposes Ollama via an HTTP tunnel: https://neric-4090-ollama.mattsouthgate.co.uk → http://192.168.100.50:11434. Neat and simple, in theory.
The problem is Cloudflare's HTTP origin timeout, which sits at around 100 seconds and can't be increased on the free or Pro plan. A cold load of qwen2.5-coder:7b from a spinning HDD takes considerably longer than that. The result is a reliable 502 before the model finishes loading.
SSH tunnels have no such limit — SSH is a raw TCP stream; Cloudflare's edge doesn't interpret its contents and doesn't impose an application-level timeout on it. That's the path.
"Just install the ollama binary"
My first instinct was to install the Ollama CLI on claude-hub so I could type ollama list and ollama run directly. The tarball is 1.33 GB — because it bundles GPU inference libraries for CUDA and ROCm. claude-hub only needs to talk HTTP to a remote Ollama; installing a full GPU inference runtime to act as a thin HTTP client is absurd. A forwarded port and curl does the same job.
Getting there
Step 1: The tunnel was down
The neric-pve Cloudflare tunnel had been down for two days. No alarm, no obvious symptom — curl -sI returns 302 whether the tunnel is up (Cloudflare Access gating you) or down (Cloudflare returning its own 302 from the Access policy). The only way to know is to query the Cloudflare API directly, or look for 530 responses on unprotected paths.
Root cause: the cloudflared service had Restart=on-failure. It received a SIGTERM during a shutdown attempt, hung during teardown, got SIGKILL'd at the 90-second timeout, and systemd logged Failed with result 'timeout'. Because the stop was considered managed, on-failure didn't trigger. Fixed with Restart=always.
Lesson: Restart=on-failure is wrong for a service that must never go down. A managed-but-failed stop looks "clean" to systemd. Use Restart=always.
Step 2: The 4090 host had no internet
NERIC-4090's bridge (vmbr0) had a dead gateway hardcoded: gateway 192.168.100.1. That address belongs to a rogue DHCP server on the segment (a documented hazard from a dual-DHCP situation there). Every packet trying to leave the host was silently black-holed.
The fix was switching vmbr0 to DHCP. A DHCP reservation in dnsmasq on .2 ensures the host reliably gets 192.168.100.10 and gateway .2 automatically.
One gotcha: ifdown vmbr0 && ifup vmbr0 resets the bridge and detaches every container's veth interface from it. After the cycle, none of the three containers had connectivity — not even to the local gateway. Had to restart all three LXCs to let Proxmox re-attach them.
Rule: Never ifdown/ifup a Proxmox bridge while containers are running on it.
Step 3: The SSH port-forward service on claude-hub
# /etc/systemd/system/neric-ollama-forward.service
[Unit]
Description=SSH port-forward: NERIC-4090 Ollama → 0.0.0.0:11434
After=network-online.target
Wants=network-online.target
[Service]
User=user
Environment=HOME=/home/user
ExecStart=/usr/bin/ssh -N -o ExitOnForwardFailure=yes -o ServerAliveInterval=15 -o ServerAliveCountMax=3 -o BatchMode=yes -L 0.0.0.0:11434:192.168.100.50:11434 neric-4090
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
The neric-4090 SSH alias in ~/.ssh/config uses a Cloudflare Access service token for non-interactive authentication — no browser, no prompt. Binding to 0.0.0.0:11434 means any machine on the home LAN can hit claude-hub.lan:11434 and reach CT100's Ollama.
systemctl enable --now neric-ollama-forward
curl http://claude-hub.lan:11434/api/tags
# {"models":[{"name":"qwen2.5-coder:7b",...}]}
That worked on the first try. The next bit didn't.
The generate endpoint mystery
/api/tags returned results instantly. /api/generate returned nothing — not a timeout error, not a 500, just silence. The model was warm (confirmed by /api/ps). What was happening?
Discovery 1: Dropped connections leave Ollama's queue stuck
Earlier in the session, a warm-up attempt ran through the Cloudflare SSH tunnel. The SSH Access proxy kills individual forwarded TCP channels after roughly 100 seconds of no data. The model was loading when the channel died. Ollama's request didn't cancel — it kept generating into a dead socket, blocking the queue. Every subsequent request waited behind the zombie.
Fix:
pct exec 100 -- systemctl restart ollama
Rule: If you drop a connection mid-generation, restart Ollama. It does not self-recover.
Discovery 2: vLLM and VRAM
After fixing the queue and retrying, inference was still broken — but differently. The generate response came back after 62 seconds for one token. That's 0.016 tok/s.
pct exec 100 -- nvidia-smi --query-compute-apps=pid,used_memory,name --format=csv,noheader
# 2076143, 22786 MiB, VLLM::EngineCore
# 2333325, 592 MiB, /usr/lib/ollama/llama-server
CT100 runs both vLLM (Qwen3.6-27B-AWQ) and Ollama. vLLM pre-allocates 97% of the 24 GB VRAM at startup and holds it. Ollama gets 592 MB — enough to open a CUDA context but not enough to load a 4.7 GB model into GPU memory. The model ends up in CPU RAM, nominally backed by CUDA but practically blocked waiting in vLLM's queue.
Fix: force Ollama to CPU-only mode via the systemd override:
[Service]
Environment="CUDA_VISIBLE_DEVICES="
Environment="OLLAMA_NUM_GPU=0"
Discovery 3: 8 GB RAM and a model in swap
CT100 was allocated 8 GB of RAM. The model (qwen2.5-coder:7b Q4_K_M) is 4.7 GB. The container already had 4.2 GB in use. The overflow landed in swap — and swap on a spinning-disk-backed LXC volume means every model weight access is a random seek on an HDD.
In Proxmox, memory can be increased on a running container without a restart:
pct set 100 -memory 16384 -swap 2048
After the increase, the model loaded cleanly into RAM.
Discovery 4: 24 threads on 8 CPUs (this was the main one)
With the model in RAM and GPU out of the picture, inference should run at whatever CPU speed the i9-13900K can manage on a Q4_K_M 7B — somewhere in the range of 3–10 tok/s.
Instead: 0.11 tok/s.
The answer was in the Ollama journal:
system_info: n_threads = 24 (n_threads_batch = 24) / 8
Ollama's bundled llama-server detects the host CPU count — 24 logical cores on the i9-13900K. But the container's cpuset limits it to 8. The result: 24 threads competing for 8 CPU cores. Constant context switching, cache thrashing. A 100× slowdown from thread over-subscription alone.
The OLLAMA_NUM_THREAD environment variable — added to the systemd override — didn't propagate to llama-server in Ollama 0.30.10. The fix that actually works is baking it into an Ollama Modelfile:
FROM qwen2.5-coder:7b
PARAMETER num_thread 8
PARAMETER num_ctx 4096
OLLAMA_MODELS=/mnt/models/ollama ollama create qwen25coder7b-8t -f /tmp/Modelfile
With qwen25coder7b-8t as the active model:
generate: 5.14 tok/s
prefill: 146.23 tok/s
From 0.11 to 5.14 tok/s — a 47× improvement from fixing the thread count. The Modelfile is the reliable way to set it; the environment variable is not (at least in this version).
LiteLLM wiring
In the litellm Incus container at 10.140.20.63, /etc/litellm/config.yaml:
# ── NERIC-4090 (Salford, CPU-mode Ollama) ────────────────────────────────
# neric-ollama-forward.service on claude-hub (10.140.3.202:11434)
# → neric-4090 cloudflared tunnel → CT100 (192.168.100.50:11434)
- model_name: neric-qwen2.5-coder:7b
litellm_params:
model: ollama/qwen25coder7b-8t
api_base: http://10.140.3.202:11434
request_timeout: 120
Two notes: request_timeout: 120 is needed because a cold load from HDD takes ~90 seconds. And claude-hub.lan doesn't resolve inside Incus containers (they use a different DNS resolver) — use the raw IP 10.140.3.202.
End-to-end test via LiteLLM:
{"choices":[{"message":{"content":"NERIC online","role":"assistant"}}]}
Keeping it warm
The warm-pinger runs on the NERIC-4090 host itself (192.168.100.10) — inside the LAN, no Cloudflare in the path. It hits CT100's Ollama directly every 4 minutes:
#!/bin/sh
curl -s --max-time 290 -X POST http://192.168.100.50:11434/api/generate \
-H "Content-Type: application/json" \
-d '{"model":"qwen25coder7b-8t","stream":false,"keep_alive":"5m","options":{"num_predict":1},"prompt":"ping"}' \
-o /tmp/ollama-warm-last.json 2>/dev/null
A systemd timer fires it every 4 minutes. With the model warm, requests through the port-forward complete in 2–3 seconds.
Current state
| Test | Result |
|---|---|
claude-hub.lan:11434/api/tags |
✅ Returns model list |
| Warm inference via port-forward | ✅ 2.4s, 5.14 tok/s |
LiteLLM neric-qwen2.5-coder:7b |
✅ Confirmed |
| Port-forward service | ✅ Active, enabled, Restart=always |
| Warm-pinger timer | ✅ Running |
Performance in CPU-only mode (vLLM holds the GPU):
| Phase | Speed |
|---|---|
| Cold load from HDD | ~90 seconds |
| Prefill | 146 tok/s |
| Generate | 5.1 tok/s |
Lessons
The three surprises worth saving:
Thread over-subscription is catastrophic and invisible. 24 threads on 8 CPUs gave 0.11 tok/s. 8 threads gave 5.14 tok/s. CPU usage still reports 700% and nvidia-smi is silent (no GPU in play). The only clue is in the Ollama startup log: n_threads = 24 ... / 8.
llama-server reads the host CPU count, not the LXC cpuset. This isn't an Ollama bug exactly — it's llama.cpp using the host's nproc rather than the container's visible count. OLLAMA_NUM_THREAD in the systemd environment didn't fix it in Ollama 0.30.10. The Modelfile PARAMETER num_thread N did.
Two inference services on one GPU don't share nicely. vLLM's 97% reservation leaves nothing for Ollama. CPU-only mode with CUDA_VISIBLE_DEVICES="" and OLLAMA_NUM_GPU=0 is actually cleaner in this scenario — the GPU isn't involved at all, so there's no CUDA queue contention.
The port-forward pattern is genuinely useful for any remote Ollama behind a corporate firewall. SSH is already available everywhere for administrative access, and it sidesteps Cloudflare's HTTP timeout completely. The 1.33 GB CLI install really is unnecessary for API use.
Ta ta for now.
References
- Cloudflare Tunnel — zero-trust tunnelling; HTTP timeout is ~100s and can't be raised on free/Pro
- Cloudflare Access SSH — SSH proxy with service tokens for non-interactive auth
- Ollama v0.30.10 — local LLM runner;
OLLAMA_NUM_THREADenv var not reliably passed to llama-server in this version - llama.cpp — the inference backend Ollama bundles as
llama-server;num_threadModelfile parameter is the reliable override - vLLM v0.21.0 — high-throughput LLM inference server; pre-allocates 97% VRAM by default
- LiteLLM — unified LLM proxy;
request_timeoutparam essential for cold-loading models - systemd.service(5) —
Restart=alwaysvsRestart=on-failuredistinction matters for cloudflared - Proxmox
pct—pct set <id> -memory Napplies live to running containers
No comments:
Post a Comment